480ms latency, Whisper-level accuracy, an open recipe, Nvidia's Canary redefines real-time ASR

Whisper has been the standard for transcription since 2022, but its offline design means waiting for the full utterance. Nvidia's Nemo Canary models, released open source this week, claim to match Whisper's accuracy while streaming with 480ms delay.

Architecture: why streaming no longer hurts

Canary uses a FastConformer transducer with a multi-token decoder that predicts six tokens per step. This design, common in image generation but rare in ASR, lets the model emit sub-word units faster without waiting for the full audio window. A promptable decoder lets users control output style, punctuation, capitalization, timestamp prediction, without retraining.

The family includes three sizes: Canary-180M, Canary-1B, and Canary-1B-Code. The 1B variants support four languages (English, German, French, Spanish) and include a code-switching mode for bilingual streams. The 180M model trades some accuracy for speed, targeting edge devices, a pattern seen in other edge-targeted models like Mistral Nano.

Benchmarks: matching Whisper head-to-head

On LibriSpeech clean, Canary-1B scores a 1.9% word error rate, identical to Whisper medium and within 0.3% of Whisper large. On Common Voice English, the gap narrows further: Canary-1B at 5.8% WER versus Whisper large at 5.6%. The streaming latency, measured end-to-end, is 480 milliseconds, roughly the time it takes to say the word "hello". Whisper large needs 3.2 seconds on the same GPU, a gap that highlights trends where smaller models rival larger ones through iterative refinement.

Reproducibility as a differentiator

The accuracy numbers are impressive. What matters more for the community is that Nvidia published the full training recipe inside the NeMo framework, including dataset preprocessing steps. Whisper's paper omitted those details, making it hard for the community to verify or improve upon. Canary's training is reproducible from scratch, a point that aligns with efforts to simplify deployment infrastructure.

"We wanted to show that state-of-the-art ASR doesn't require black magic," the PDF states. "Every configuration file is in NeMo." That claim, if verified, reduces the upfront work for teams that need domain-specific fine-tuning, medical, legal, or accented speech, without starting from zero.

What's missing

Canary covers only four languages. Whisper supports 99. The code-switching mode is limited to English paired with one other language, not a true multilingual mix. And the 480ms latency assumes a single GPU with batch size 1; production deployments on shared hardware often stretch those numbers.

Still, the open source community now has a credible streaming alternative to Whisper, one that matches the leader on accuracy while cutting latency by a factor of six. The next test will be whether the community adopts and extends it, following the path of other open-weight models that reached mass adoption.