957,253 records can't hide the gap: AI agents surge in coding but stall where enterprises need them

For years, anyone trying to compare AI agents across benchmarks faced a mess of incompatible tasks, custom scaffolds, one-off verifiers, and scoring rules designed in isolation. A single agent might ace one leaderboard and flop on another without any way to tell whether the difference was real capability or just evaluation design. That fragmentation made it nearly impossible to answer the only question that matters: where are agents actually getting better? As recent critiques have shown.

The Messier corpus, released on arXiv on July 28, 2026, tries to answer that question at scale. It consolidates 957,253 records spanning 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, and annotated with occupational and industry codes. The result is the first high-resolution cross-benchmark look at what AI agents can and cannot do, and the picture is not uniformly bright.

The fragmentation problem

The existing evaluation landscape is famously Balkanized. Different research groups create bespoke environments, define their own success criteria, and run agents on incompatible scaffolds. Even when two benchmarks test similar skills, the results are rarely comparable because the harness, verifier, or aggregation rule shifts the score in ways that have nothing to do with the agent itself. A recent benchmark, OpenRTAG, organized quality issues in graph data into a 3×3 taxonomy and found that different classes of model families (traditional GNNs vs. LLM-enhanced GNNs vs. graph foundation models) show different sensitivity patterns, another reminder that benchmark design choices deeply color results , a problem that has plagued even stalwarts like MMLU. Meanwhile, Mistral's own audit of its OCR 4 revealed that standard benchmarks can penalize output that is factually correct but formatted outside the expected schema, creating noise that obscures real gains. And the MosaicLeaks study showed that deep research agents can leak private data through web queries even when no single query is incriminating, a capability concern that no standard benchmark currently measures.

Uneven frontier

The Messier corpus confirms that the field's progress is dramatically uneven across benchmark categories. The paper identifies three distinct zones:

• Function calling, saturated. Agent performance on function-calling benchmarks has plateaued, suggesting that current models have learned the mechanics of tool invocation as well as existing tests can measure.
• Programming, surging. Coding benchmarks see the fastest improvement, driven by advances in code generation and repository-level editing. This aligns with a separate July 2024 study by Alibaba's HSCodeComp team, which found that top agents still hit only 49.4% accuracy on tariff classification versus human experts' 95%, but that ceiling is structural, not simply a matter of compute. Programming tasks that require reasoning about real-world rules remain hard.
• Enterprise workflows, lagging. Benchmark categories that simulate multi-step business processes, approval chains, and domain-specific tool use remain the most challenging. This finding matches Alibaba's broader strategy, which focuses on locking enterprises into an integrated AI stack rather than winning individual model benchmarks, and Gartner's assessment that enterprise AI coding agents like Cursor lead on completeness of vision but that the 70% Fortune 500 adoption statistic still leaves out how many of those deployments are limited-scale pilots , a gap highlighted by Claude Opus 5's deliberately capped skills.

The all-pass trap

Perhaps the most consequential finding in the Messier paper is methodological. When a task involves multiple verifiers, say, checking that an agent retrieved the right document, extracted the correct field, and formatted it according to a schema, many benchmarks use a strict all-pass aggregation rule: the agent gets credit only if it passes every single verifier. The Messier team ran counterfactual rescoring exercises and found that this all-pass rule systematically underestimates progress, especially on harder multi-verifier tasks, and can artificially alter agent rankings. An agent that passes nine out of ten checks looks identical to one that passes zero. This echoes findings from the earlier PawBench study, which showed that harness design can swing scores by over 11 points for smaller models, exposing a blind spot in how the field evaluates agent pipelines , consistent with evidence that harness design itself can inflate performance.

Capability scales that hold up

If the scores from different benchmarks cannot be directly compared, can we at least derive a general capability scale? The Messier team did exactly that. From the standardized records, they computed capability scales and validated them against Epoch's Evaluation Capability Index. The alignment is strong: a Spearman rank correlation of 0.81. This suggests that the corpus captures something real about relative agent capability, even across diverse evaluation settings , a welcome contrast to the known pitfalls of individual benchmarks like GSM8K. The scales can also be specialized by domain, occupation, action space, or verifier type, making them useful for targeted audits, for instance, asking whether agents are better at software engineering than legal reasoning, or whether small models are closing the gap in certain tool-use categories.

What Messier means for the next generation of benchmarks

The Messier corpus is not itself a benchmark. It is a meta-evaluation infrastructure, a way to audit existing benchmarks for bias, dead zones, and scoring artifacts. The paper pitches it as "a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures." Its release arrives alongside a growing recognition that the field needs shared standards. The recent Do Agent Benchmarks Measure Capability? paper argued that protocol validity, whether the evaluation protocol actually tests the intended capability, requires explicit evidence, and that without it, scores risk being meaningless. Messier provides exactly the cross-benchmark evidence base needed to make those protocol-validity checks possible. For enterprise buyers trying to decide which agent to deploy, or researchers trying to decide where to invest, Messier offers something the field has been missing: a single, standardized map of AI agent capability, warts and all.