Voice Memory: a 776-byte file that tells speech recognition when to do nothing

For years, the standard way to clean up speech recognition output has been to pass the transcript to a second generative model and let it fix what the first one got wrong. Voice Memory, a new inference-only scheme described in a paper posted on Hugging Face, makes a different case: the fixer needs to know when to leave the transcript alone, and the way to teach it is a text file.

Left unconstrained, generative error correction over-corrects. On financial news, up to 64% of its edits break tokens that were already correct. The damage does not read as noise: a corrector rewriting a token that was already right produces something wrong but fluent, and disrupted language models keep that structure too. Perturbing a model's neurons to simulate aphasic picture-naming errors matched individual patient profiles, per the stroke-simulation study. The correction loop was producing more damage than the errors it was meant to repair.

How the listener-thinker split works

Voice Memory builds on the classical ASR language-model framework with what the paper calls a listener-thinker architecture. The listener is a frozen decoder that produces the hypothesis. The thinker is a frozen corrector that reads a single per-domain memory file, memory.md, and decides per utterance whether to rewrite the hypothesis or abstain and keep the 1-best result. That go/no-go call is a routing decision, and per-call routing is a known blind spot in agentic systems, a gap TRACE-Router is built to close. The two roles are coupled only through the memory; no weights change anywhere in the loop.

Updating the memory is a separate, asynchronous job. A score-gated optimizer revises the file through bounded edits and accepts an edit only when it strictly improves a held-out score. Each revision has to justify itself against that score before it stays. Because the learned skill lives in a text file rather than in parameters, it stays auditable and portable. That design sits at the opposite end from training recipes built to avoid external memory altogether, such as OPID's on-policy distillation. The deployed Wall Street Journal memory is 776 bytes.

What the numbers say

Across ten HyPoradise domains with an open corrector, Voice Memory lowers weighted word error from 8.36% to 7.52%, or 7.47% with three added in-context examples, without regressing any dataset below its 1-best baseline. That floor is structural: when the corrector abstains, the original 1-best hypothesis goes through untouched.

Condition	Baseline	With Voice Memory
Weighted WER, ten HyPoradise domains	8.36%	7.52%
Air-travel commands	8.40%	3.40%
CHiME-4 noisy far-field speech	12.69%	10.46%
Edits that break correct tokens	64%	35%

Gains concentrate where recoverable headroom is largest. Air-travel commands improve from 8.40% to 3.40%. CHiME-4, the noisy far-field benchmark, moves from 12.69% to 10.46%. The rest of the story is restraint: unconstrained generative correction breaks correct tokens on up to 64% of its edits on financial news, and Voice Memory brings that rate down to 35%.

Restraint is the operative skill

The authors' own summary is blunt: "The skill the loop mostly learns is restraint." The memory file does not make the corrector smarter; it gives it a reason to do nothing. Abstention is the mechanism behind the gains, and it costs nothing at inference time: Voice Memory adds zero parameters to the inference path. The memory also transfers across corrector families, so a domain file learned with one corrector can be reused with another. That portability is a direct consequence of keeping the policy in text rather than in weights.

Two limits are worth keeping in mind. The released material does not name the corrector model, only that it is an open one. And the evidence sits on ten HyPoradise domains, not production traffic.

What a 776-byte file points to

The paper frames the scheme as "agentic speech recognition": the corrector acts as an agent that reads a memory and decides per utterance whether to intervene. Voice Memory learns by revising that file until a held-out score improves, a policy-refinement loop of the kind EvoPolicyGym was built to isolate. Putting the learned policy in a plain-text memory.md instead of in weights has a practical consequence for any pipeline that runs a second heavy model over its transcripts: the gate is a file, and the file is smaller than most configuration blocks. The authors also published a walkthrough demo and example code, with the model and per-domain memories on Hugging Face.

The awkward part of this result is how much it resembles common sense. The field's reflex is to correct harder, with bigger models. Voice Memory does the opposite. The recognizer and the corrector stay frozen, coupled only through a file that fits in a kilobyte, and an edit is committed only when a held-out score proves it helps. On this evidence, the most valuable capability a speech AI can learn is when to say no.