AI agents run flawless statistics and still draw the wrong conclusions

LLM agents are being pointed at one of science's most load-bearing jobs: hypothesis testing. They inspect datasets, generate code, and produce analyses end-to-end. A preprint posted to arXiv on 7 Aug 2026 warns that this automation has a blind spot. Agents frequently draw incorrect conclusions even when the analysis itself runs correctly. The errors are inferential, not mechanical: a statistical method the data does not support, a p-value that does not hold under the data's assumptions. Most existing benchmarks never see these failures, because they grade outputs, not the validity of the inference underneath. The same gap shows up in video understanding, where Gemini 3.6 Flash counts state changes but misses blinks.

Evaluation suites rarely ask whether a reported p-value is statistically valid under the data's assumptions. That blind spot matters because LLM agents are increasingly used to automate hypothesis testing, and a clean run can look like a trustworthy result. Earlier work on coding agents documented the same inversion: verifying a generated solution is harder than producing one. Tool-using models have their own version of the problem; a KlingTeam study finds a tool can make an easy question harder. The new paper carries that lesson into statistics.

The failure mode that benchmarks miss

To measure the gap, the team built P-Bench, a benchmark of 425 open-ended hypothesis-testing tasks spanning economics, biology, and medicine. Each task gives an agent only a scientific hypothesis and a dataset. The agent must select a statistical method, compute a p-value, and draw a conclusion. Scoring then does what the paper says existing evals do not: it checks whether the reported p-value is statistically valid given the assumptions underlying the data.

What reinforcement learning bought

The proposed fix is Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing with synthetic tasks and reinforcement learning. The design choice that matters is the reward: statistical verification rather than agreement with a reference answer. Reward design is a known failure point; RDPO documents reward signals that work against each other. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeek-V4-Pro. The headline number is a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, rising to 26% on the most challenging tasks.

The caveats behind a 21% gap

The result deserves the standard preprint caveats. It is not peer-reviewed, and P-Bench is the authors' own construction, so the baselines were judged on a benchmark its creators designed, a reason to read the margins with care. Single-trial success is one metric. An agent allowed to check its work might close or widen the gap; this paper does not say.

What is harder to dismiss is the diagnosis, which lands in a field already wrestling with evaluation reliability. The Messier corpus, a July 2026 effort to unify agent evaluation, counted 957,253 records across 30 benchmarks and 714 agents and argued that most of that empirical record is incomparable. P-Bench is a narrower answer to the same problem: a benchmark built to measure one specific failure mode rather than another aggregate score. The worry that metrics miss what matters has appeared elsewhere too, as when a Picbreeder reboot found VLMs missed the boldness of human-made archives.

For teams handing hypothesis testing to agents, the practical lesson is cheap to state and costly to learn: a clean run is not a valid inference, and a p-value tells you little if the assumptions behind it were never checked.