Nemotron 3.5 Transcribe – NVIDIA's Open-Source Speech Large Model

1. What is Nemotron 3.5 Transcribe

Nemotron 3.5 Transcribe is an open-source speech large model introduced by NVIDIA. It employs a speech encoder combined with a MoE architecture, featuring approximately 4.3 billion active parameters, and supports 40 languages. The model can process recordings with up to 8 speakers lasting as long as 1 hour, covering functionalities such as word-by-word transcription, speaker diarization, and speech translation. Through the SALM (Speech-Augmented Language Model) architecture, the speech encoder and a frozen speaker separation model are integrated into a large language model, enabling transcription with speaker annotations. Nemotron 3.5 Transcribe supports customizing output formats using natural language instructions, and can be fine-tuned to adapt to specialized domains. It is available for commercial use at no cost.

Technical positioning and domain: This model belongs to the intersection of speech recognition and natural language processing, targeting deep offline transcription scenarios such as meeting minutes, interview archiving, classroom playback, and multilingual localization. Its unique positioning lies in unifying speaker diarization and transcription tasks within a single LLM architecture, breaking through the performance limitations of traditional cascaded systems, and establishing itself as a technical benchmark among open-weight models.

Development background: Leveraging its deep expertise in GPU computing and AI infrastructure, NVIDIA has extended the Nemotron series language model technology into the speech domain. The development of this model fully utilizes NVIDIA's systematic advantages in the NeMo toolkit, TensorRT-LLM inference framework, and RTX GPU hardware ecosystem, demonstrating its full-stack R&D capabilities from the underlying hardware to the upper-layer models.

Core value: This model addresses three major pain points in traditional speech transcription systems: speaker diarization, long audio processing, and domain adaptation. Its instruction-based output capabilities allow users to switch between different output formats—such as verbatim transcripts, summarized notes, and translations—without retraining, significantly reducing the complexity of deployment and usage. By open-sourcing the weights and allowing commercial use, NVIDIA provides enterprises and developers with a professional-grade speech solution that can be deployed independently.

2. Key Features

• Multilingual Transcription: Supports voice transcription in 40 languages, with the model trained on globally mainstream languages. It can automatically identify the language spoken in colloquial speech and output the corresponding textual transcription. Unlike traditional ASR systems that require a pre-specified language, this model can automatically switch recognition in mixed-language meeting scenarios, reducing the usage barrier.

• Speaker Diarization: Can distinguish up to 8 speakers and label each sentence with its speaker. This capability first surpasses dedicated cascaded systems in overlapping multi-speaker scenarios. This is achieved by injecting activity signals from the frozen Nemotron 3 speaker separation model into the ASR encoder representations, allowing the model to perform transcription with speaker labels under a single LLM architecture.

• Long Audio Processing: Can process audio recordings up to approximately 1 hour in length, breaking through the context window limitations of traditional ASR models. For complete meeting records, lectures, or long interviews, users no longer need to split and upload the audio in segments; the model can directly process the full recording end-to-end, reducing errors from concatenation.

• Instruction-Based Transcription: Specify the output format using natural language, allowing flexible switching between different forms such as verbatim transcripts, written summaries, removal of filler words, and terminology normalization, without requiring retraining or deploying multiple models. For example, users can simply input "retain verbatim speech and label speakers" or "remove filler words and translate to English," and the model will understand and execute accordingly.

• Speech Translation: Can translate between 31 languages and English, directly converting non-English audio recordings into English transcripts. This feature is tailored for cross-border meetings, international interviews, and video subtitle creation. Combined with instruction-based output capabilities, users can obtain both the original transcription and the translated result in a single inference.

• Word-Level Timestamps: Returns the audio time position corresponding to each word, facilitating playback and retrieval. After decoding, a lightweight CTC adapter aligns the generated text with the audio, producing word-level timestamps with speaker labels for long audio and heavily overlapping conversations, without adding a burden to the main model.

• Domain Fine-Tuning: Based on the NeMo Speech toolkit, it can quickly adapt to specialized scenarios using a small amount of data on a single GPU. Practical tests show that fine-tuning takes approximately 0.5 to 2.3 hours, enabling the model to adapt to user-specific accents, industry-specific terminology, and custom output formats, meeting the customization needs of vertical fields such as healthcare, legal, and education.

3. Application Scenarios

• Meeting Minutes and Decision Extraction: In business meeting scenarios, the model automatically distinguishes speakers and retains the original dialogue, extracting meeting conclusions and action items based on instructions. Combined with word-level timestamps, participants can directly jump to the discussion moments of specific viewpoints, quickly reviewing key decision-making processes.

• Media Interviews and News Production: Journalists and editors can use this model to retain the interviewee's exact words and annotate speakers, ensuring accurate quotations. Instruction-based output allows for the on-demand removal of filler words or verbal tics, and the output can be directly used for published transcripts, significantly reducing manual editing time.

• Classroom and Training Recordings: Educational institutions can submit course recordings to the model for processing, generating transcriptions with word-level timestamps. This enables students to search for keywords and jump directly to the moment the teacher explained that concept. This capability greatly enhances the retrieval efficiency of long video courses, supporting a more refined learning experience in online education.

• Medical and Professional Field Documentation: Combined with a terminology list and domain-specific fine-tuning, patient-doctor conversations can be transcribed into medical records in a standardized format. In practical tests, providing a terminology list reduced named entity error rates by six times, eliminating the need to guess at professional terms, and offering a reliable automated documentation solution for high-precision scenarios such as healthcare and legal fields.

• Multilingual Content Localization: Audio recordings in 40 languages can be directly translated into English transcripts for use in international meetings, interviews, or video subtitles. The model's ability to translate between 31 languages and English enables global teams to bypass third-party translation services, completing localization during the transcription process.

4. How to Use

1. Hardware Preparation: The BF16 version requires approximately 80GB of GPU memory, suitable for professional GPUs such as NVIDIA A100 or higher. Optional quantized versions (FP8, NVFP4, or GGUF) can lower the memory requirements. The GGUF version can run locally on consumer-grade GPUs.

2. Obtain the Model: Search for NVIDIA-Nemotron-3.5-Transcribe-30B-A4B on Hugging Face or ModelScope, and download the weights under the OpenMDW-1.1 license. This license permits commercial use, but users must comply with the usage terms outlined in the model card.

3. Set Up the Environment: Choose one of three deployment options. Using the Nemotron Transcribe NIM (NVIDIA official container) provides the most optimized performance and convenient management. Using the vLLM framework supports MTP speculative decoding acceleration. Using NeMo-Speech.cpp is ideal for lightweight local execution with GGUF format models.

4. Prepare Audio: Upload a recording file up to one hour in length, converting the format according to the requirements of the selected framework. It is recommended to use common formats such as WAV, MP3, or FLAC, and align the sampling rate with the standard used during model training.

5. Write Prompts: Describe the task in natural language, such as "retain verbatim speech and annotate speakers" or "remove filler words and translate to English." The clarity of the prompt directly affects the accuracy of the output format. It is advisable to explicitly state whether to retain filler words, perform translation, and whether speaker labels are required.

6. Run Inference: Submit the audio and prompt via the API interface. The model will return the transcribed text, optionally including speaker labels and word-level timestamps. For long audio files, it is recommended to use NIM or vLLM deployment to ensure stable inference performance.

7. Online Quick Experience: Users who do not wish to deploy the model locally can first try the nvidia/nemotron-transcribe demo space on Hugging Face. This allows verification of the results before deciding on local deployment.

8. Domain Fine-tuning (Optional): Use the NeMo Speech toolkit to prepare annotated data. With a single GPU, domain fine-tuning can take approximately 0.5 to 2.3 hours to adapt to proprietary accents, terminology, and formatting. The amount of fine-tuning data does not need to be large; a small number of high-quality annotated samples can significantly improve performance.

5. Pros and Cons Analysis

6. Comparative Analysis with Similar Tools

Selection Recommendations: For professional scenarios requiring long-duration recordings, multi-speaker transcription, and speaker diarization, Nemotron 3.5 Transcribe is a strong choice among open-weight models, especially suitable for high-demand fields such as legal, healthcare, and media that require precise attribution and customized terminology. Its instruction-based output and domain fine-tuning capabilities offer long-term value that can offset the higher initial hardware investment.

For low-latency scenarios requiring streaming real-time subtitles or live transcription, the streaming capabilities of OpenAI GPT-Realtime-Whisper are more suitable. However, the cost of the closed-source API and dependency on cloud services should be considered. For Chinese-specific scenarios, especially those requiring domestic deployment, Aliyun Paraformer has a clear advantage in Chinese transcription accuracy and the Chinese ecosystem. For individual developers or lightweight applications with limited budgets, Whisper Large-v3 remains the most accessible choice due to its MIT license and large community support.

7. Editor's Summary

Nemotron 3.5 Transcribe represents a significant direction in the evolution of speech transcription technology, transitioning from a cascaded architecture to a unified model. NVIDIA has integrated the speech encoder, speaker separation model, and large language model seamlessly through the SALM architecture, enabling collaborative processing of transcription, speaker labeling, and translation within a single model. This architectural innovation not only simplifies the deployment pipeline but also allows the model to leverage the context understanding capabilities of the LLM for terminology correction and format customization—something traditional cascaded systems struggle to achieve. In terms of practical performance, the English Open ASR leaderboard's 4.71% WER and the multilingual leaderboard's 4.62% results have already reached the leading level among open-weight models. Moreover, its superior performance in overlapping speech scenarios highlights the real-world value of this architectural innovation.

From a practical standpoint, the most notable advantage of this model is its ability to lower the barrier for downstream tasks. The instruction-based output enables non-technical users to flexibly control the transcription format, while domain-specific fine-tuning capabilities allow enterprises to obtain specialized models tailored to their business needs at a relatively low cost. The OpenMDW-1.1 license permits commercial use, providing a compliant pathway for companies requiring private deployment. It is important to recognize, however, that high hardware requirements and the absence of streaming real-time capabilities remain constraints limiting its broader adoption. Currently, the model is positioned for offline deep processing scenarios, making it highly suitable for use cases such as meeting notes, interview archiving, course replays, and professional domain recordings. For teams with access to professional GPU resources and vertical industry users with customization needs, this model is a worthy open-weight solution to include in technical evaluations. As the NeMo ecosystem continues to mature and community contributions grow, the application boundaries of this model in the speech domain are expected to expand further.

8. FAQ

Q: What are the total parameter count and the activated parameter count for NVIDIA-Nemotron-3.5-Transcribe?
A: The model has approximately 34 billion total parameters and employs a hybrid Mamba-Transformer MoE architecture. Due to the MoE design, only about 4.3 billion parameters are activated during each inference. This sparse activation mechanism enables the model to achieve the semantic understanding capabilities of a large model with minimal computational overhead, achieving a high throughput of transcribing 2,238 seconds of audio per second on a single RTX PRO 6000 Blackwell GPU.

Q: Which languages does the model support? What is its translation capability?
A: The transcription supports 40 languages, and the model can identify the language type in spoken language and perform mixed recognition. Speech translation supports mutual translation between 31 languages and English. It should be noted that direct translation between non-English languages is not yet supported—for example, direct translation from Chinese to Japanese is not currently possible and requires translation to English first, followed by a second translation.

Q: What hardware configuration is required for deployment?
A: The BF16 version requires approximately 80GB of GPU memory, and it is recommended to use an NVIDIA A100 (80GB) or higher GPU configuration. To lower the usage threshold, the official provides quantized versions in FP8, NVFP4, and GGUF formats. The GGUF version can run on consumer-grade GPUs, but there will be some loss in precision. When deploying in practice, it is recommended to choose the appropriate version based on the audio length and throughput requirements.

Q: How can natural language instructions be used to control the output format?
A: Simply specify the requirement in the prompt. For example, inputting "retain the exact words and label the speaker" will generate a verbatim transcript with speaker labels; inputting "remove filler words and translate to English" will produce an English translation with filler words removed; inputting "organize in meeting minutes format, including conclusions and action items" will yield a structured meeting minutes output. No retraining or model replacement is required.

Q: Does the model support commercial use?
A: Yes. The model's weights are released under the OpenMDW-1.1 license, allowing for commercial use and self-deployment. For specific terms, it is recommended to refer to the complete license information in the model card. Compared to the pay-per-use model of closed-source APIs, the self-deployment model offers more cost control when used at scale over the long term.

Q: How can the model be fine-tuned for specific domains?
A: Use the NeMo Speech toolkit to prepare annotated data, and adaptation can be completed in approximately 0.5 to 2.3 hours on a single GPU. Fine-tuning does not require a large amount of data—only a small number of high-quality annotated samples are needed to adapt the model to in-house accents, industry-specific terminology, and custom output formats. After providing a list of entities such as terms and names, the named entity error rate can be reduced by six times.

9. Project Links

• NVIDIA Official Website NIM Product Page: https://build.nvidia.com/nvidia/nemotron-3-5-transcribe
• Hugging Face Model Library: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
• Hugging Face Online Experience Space: https://huggingface.co/spaces/nvidia/nemotron-transcribe
• ModelScope Model Library: https://www.modelscope.cn/models/NVIDIA/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
• Technical Paper (Technical Report): https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Transcribe-Technical-Report.pdf