Memory Decoder: a 6.9B bolt-on memory makes a 410M model beat Pythia-12B

The most useful number in the Memory Decoder at Scale paper, posted to arXiv on July 30, is not a benchmark score. It is 39, the percentage of total parameters you apparently no longer need if you stop spending your whole budget on a bigger base model and spend some of it on a pretrained memory module instead.

Decoder-only language models fold long-term memory and reasoning into one parameter set, the authors argue, which makes memory capacity hard to grow on its own. Memory Decoder, a parametric long-term memory module from earlier work, separates the two. The earlier version was only studied at small scale, though. The new paper scales it: memory modules up to 6.9B parameters, pretrained on 300B tokens, attached to base models that run from 410M to 14B parameters.

A 410M model with a memory upgrade beats a 12B one

The flagship result is a giant-killer pairing. On 17 benchmarks, a 6.9B general memory module attached to Pythia-410M lifts its average score from 29.86 to 37.34. Pythia-12B, the much larger model, averages 37.24. The memory-equipped 410M model comes out ahead with roughly 7.3B total parameters against Pythia-12B's 12B, a difference the paper states as 39% fewer parameters.

The margin is 0.10 points, which is not nothing on a 17-benchmark average, and the point lands on the cost side: the smaller total parameter bill buys equal or better scores.

The same pattern shows up in a second model family. For Qwen3 Base models ranging from 0.6B to 14B parameters, 1.7B domain memories improved the average score by more than 9 points at every scale tested. At every scale, the memory upgrade paid off better than the parameter spend it replaced. Qwen has also been part of a broader push by Chinese labs to match or beat US frontier models on benchmarks, according to our roundup of that benchmark push.

The retrieval wall

Scaling memory stops being a pure parameter question once the pretraining run reaches 300B tokens. The combined cost of indexing and search makes a standard Faiss pipeline infeasible at that data scale, the authors write. Their fix is a distributed pipeline for Faiss indexing and retrieval, combined with sparse, batch-wise loading of kNN distributions.

That infrastructure detail is worth spelling out because it is the real price of the architecture. A memory module only improves a model if retrieval can keep up with it. Anyone who wants to reproduce the results, or build on them, inherits that engineering problem instead of a simple path of downloading more parameters. Other AI workloads hit the same wall: Ai2's OlmoEarth had to treat satellite inference as a distributed-systems problem for the same reason.

The numbers in one place

The table below is the comparison the paper leans on.

The ~7.3B figure is arithmetic from the paper's own numbers: 0.41B base plus 6.9B memory. That is where the 39% claim comes from.

What it means for scaling budgets

The broader claim is architectural: memory and reasoning should not have to fight for the same parameters. If a large memory can be pretrained once and reused across smaller base models, improving a model becomes a matter of upgrading the memory instead of rebuilding the whole stack. The idea that specialized modules can beat a single fat parameter set has other examples in the field; Ai2's EMO lets expert modules emerge from data rather than hand-designed routing.

Open questions outnumber answers. The memory modules stop at 6.9B parameters and the pretraining run at 300B tokens, so whether the tradeoff keeps favoring memory past those limits is untested. The scores are averages over 17 benchmarks, which can hide where memory helps and where it doesn't. Public benchmarks have a way of flattering models, as the gap between SWE-bench and private enterprise code showed. And a memory-equipped model carries retrieval infrastructure that parameter counts do not capture, a deployment cost that only shows up in production.

The project page has drawn 48 upvotes on HuggingFace, a modest signal that the scaling crowd has noticed. None of this retires base-model scaling. It hands labs a second dial, and the paper's numbers suggest that dial has been underpriced.