NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt

What it claims

Nemotron 3.5 Transcribe is one open-weight checkpoint for transcription (40 languages), translation (31 languages to or from English), speaker attribution (13 languages, up to eight speakers) and audio Q&A. It handles recordings up to an hour long. Language support differs by task, and audio Q&A is trained only in English. NVIDIA reports first place among open-weight models on the Hugging Face Open ASR Leaderboard (English, 5.33% average WER) and the lowest error on two diarization benchmarks. The post also describes the Granary v2 curation pipeline and a medical fine-tuning recipe that took about 14 GPU-hours on one GB300.

How it works

A speech encoder feeds a hybrid Mamba-Transformer mixture-of-experts language model, with about 4B of 30B parameters active per token. Sortformer speaker cues reach the language model alongside acoustic features, so speaker turns are tagged during generation rather than attached afterwards. Output is steered by natural-language prompts: language, script, verbatim or clean style, term lists ("Use these terms only when supported by the audio"), and the task itself. Word-level timestamps are supported.

Granary v2 runs in four NeMo Curator stages. Its main branch is in ASR and acceptance:

flowchart LR
  A[Segmented audio] --> B[Primary ASR]
  B -->|fails hallucination check| C[Recovery ASR]
  B -->|passes| D[Transcript variants]
  C --> D
  H[Human references] --> D
  D --> E{Rate, hallucination, agreement checks}
  E -->|accepted| F[Biasing terms + translations]
  E -->|rejected| X[Dropped]
  F --> G[HiFi filter]

The audio-preparation stage was skipped for this release because the sources were already segmented. The HiFi stage removes noisy, clipped or multi-speaker segments. On VoxPopuli it reduced 354k hours to 111k hours. Each record carries three transcript forms (tn_raw, itn_raw, itn_clean), entity hints, instruction pairs, provenance and a quality flag. The release covers 463,062 hours, 48 languages and 16 corpora, up from 25 languages in v1. This is not Transcribe's full training mixture.

Evidence

• Open ASR Leaderboard: English 5.33% average WER, first among open-weight models with API models excluded. Multilingual mean WER is 4.62% over seven languages, and long-form is 10.98%. Both are lowest in the model card's open-weight comparison.
• Artificial Analysis: 2.8% AA-WER, tied for first among open-weight models with a model of 24B active parameters. On AA-AgentTalk it scores 1.8% WER against 2.3% for the next open-weight model.
• David AI DAI-ASR-I18N: 18.26% DER across 21 languages, lowest of 14 systems. NVIDIA's dedicated Nemotron 3 Diarization model is second at 18.38%. On the combined transcription-plus-attribution track, it is second overall.
• VoiceArena Diarization-Bench v1: 32.4% cpWER, lowest of 14 systems, covering English sessions with 2 to 8 speakers. Deletions are 8.32% of reference words, against a median of about 31%.
• Throughput: 100 hours of audio in under three minutes on one RTX PRO 6000 Blackwell Server Edition with NVFP4. This was measured on 30-second clips at high concurrency.

The medical case study used Eka Care's benchmark of Indian clinicians: 3,619 clips, split by speaker into 855 tuning clips and 2,764 held-out clips. Naming English and Latin script in the prompt cut full-set WER from 30.8% to 23.8%. Fine-tuning brought it to 11.8%. On held-out speakers, scored with KARMA, WER fell from 22.44% (best prompt) to 12.17%. Keyword WER fell from 9.2% to 6.5%. The tuned model beats an open medical-specialist model (13.79%) and a general-purpose ASR model (14.92%). Neither comparison model is named.

How to use it

Weights are on Hugging Face and ModelScope. Serving options are vLLM or NeMo-Speech.cpp for local use. Hosted endpoints are available from Baseten, DeepInfra and Simplismart. Any single NVIDIA GPU from Jetson AGX Thor and DGX Spark up to Blackwell data-center parts works. The model card holds prompt presets and a task-by-language matrix.

The fine-tuning recipe trains the speech encoder and projection and applies LoRA to the language model. Its data mix is about 153 h of synthetic TTS speech, 29 h of accented Common Voice English and 63 h of MultiMed. Synthetic text separates label from spoken form ("HbA1c" vs "H B A one C"). Prompts include near-miss distractor terms. Two runs use different mixtures; their adapters are merged and selected checkpoints are averaged. Selection uses decoded dev-set WER, not training loss. NVIDIA recommends at least 100 real labeled clips for selection. A coding-agent skill and ms-swift on ModelScope also support fine-tuning. The page gives no commands inline.

Limits

• The model is for recorded audio and does not support live streaming.
• Fine-tuning raised LibriSpeech test-clean WER from 1.54% to 1.69%.
• The medical specialist model still leads on exact brand-name drug and diagnostic matches.
• Granary source audio must be downloaded separately under each corpus's licence.
• The page does not state the model's licence.
• NVIDIA advises spot-checking names, speaker changes and Q&A answers against the recording.