BM25 beats agentic RAG when corpora pass 10 million tokens

Agentic search is what most RAG stacks quietly aspire to be: a model that walks a corpus, decides what to open, and reads until it finds the answer. A scaling study posted to arXiv in late July argues the instinct has the order backwards. Once corpora cross about 10 million tokens, BM25, a plain lexical retriever, beats agentic search on accuracy at every larger corpus size the study tested. Claims like this need the kind of fixed evaluation the authors built; agent benchmarks have a track record of uneven results, as a 957,253-record audit of agent benchmarks showed.

The paper comes from researchers at muset-ai. It compares lexical retrieval, dense retrieval, graph-based indexing, and agentic search across 28 strictly nested corpus tiers spanning roughly 450-fold. Questions and a fixed bedrock of relevant and adversarial documents stay unchanged. One reader model and a single judging protocol score every paradigm, and the authors log accuracy, construction tokens, query tokens, and latency. Because the tiers are nested, each larger corpus contains every smaller one, so rankings stay comparable across sizes instead of drifting with the test set. That shared setup is the whole point: most RAG paradigms get benchmarked on different corpora at a single size, which has left their accuracy-cost scaling unclear. It applies the same standardized-yardstick discipline that GLUE brought to language models in 2018 (the GLUE story).

The crossover at 10 million tokens

The headline result is a scale-dependent crossover, not an unconditional winner. A File-System Agent leads at the smallest shared tiers, where walking a small tree is cheap. As the search space grows, that sequential strategy becomes less effective, and around 10 million corpus tokens BM25 overtakes it. From there BM25 leads at every larger tier, with a margin approaching 20 accuracy points at full scale.

Read together, the two curves tell a simple story. Global candidate ranking scores the whole corpus in one pass and lets the strongest matches rise to the top. Sequential exploration scales badly because every document the agent opens costs tokens a ranker would never spend. Lexical retrieval is the strongest scalable default the study finds; agentic reasoning is a second step, not a first one.

Agentic search burns tokens before it fades

The cost curve makes the point sharper. At the bedrock tier, the File-System Agent's sequential exploration costs 39 times more query tokens. On a corpus big enough to need retrieval at all, that gap decides whether the retrieval step costs pennies or reshapes the whole inference bill.

The authors put it plainly: agentic reasoning works best after ranked discovery, rather than in place of it. Keep a cheap global ranker as the front door, then let the agent read the shortlist. The agent's judgment still matters. It just should not be doing the search itself. The same line of thinking runs through agent routing, where TRACE-Router identifies per-call routing as the blind spot.

Dense and graph retrieval: efficient, or unfinished

Dense retrieval stays efficient but less accurate, the study finds. Graph-based RAG hits construction walls before deployment scale, and even its scalable variants remain below BM25 at the shared tiers. BM25 also anchors the low-cost end of the Pareto frontier with no LLM-based construction required. There are no embeddings to build and no graph to maintain just to rank candidates.

The authors claim no universal winner, and the small-scale result is the genuine caveat: on the smallest corpora, the agent leads. But the crossover near 10 million tokens sits inside the size range where real deployments already live. Rankings that hold on one benchmark can flip when the ground moves, as SWE-bench Verified's 96%-to-23% drop on private enterprise code showed. For teams wiring up retrieval pipelines, the practical read cuts against the hype: start with the boring index, and spend agent tokens on the few documents the ranker trusts.

The paper had 41 upvotes on Hugging Face when the v2 revision landed on 30 July.