LLMs can describe data. They cannot reason through it. A new benchmark proves the gap is real.

Large language models have been celebrated for their ability to write code, summarize literature, and even generate plausible-looking scientific workflows. But a new paper, published on arXiv on July 13, 2026, argues that the bar for scientific discovery is much higher, and that current benchmarks have been measuring the wrong thing. The problem echoes what researchers have found with sandbox evaluations, where agents look impressive until you test them on the fundamentals, see the gap between sandbox and real-world agent failures.

SDABench, introduced by researchers led by Chuhan Shi and Xiaoquan Ren, reorganizes evaluation around six distinct capabilities: descriptive, exploratory, inferential, predictive, causal, and mechanistic. Instead of rewarding models for stitching together a workflow that ends with a correct number, the benchmark tests whether models understand which analytical method fits the question, whether they can model unobserved variables, and whether they can reason about causation, not just correlation.

The benchmark contains 527 real-data instances (SDA-Real) and 6,000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, built through an automated pipeline. The domains span Biology, Chemistry, Environment, Geography, and Physics.

Descriptive mastery, inferential collapse

When the researchers evaluated 15 representative LLMs, the pattern was stark. Models handled descriptive analysis, summarizing means, variances, and distributions, with solid performance. But as soon as the task demanded assumption selection, latent-process modeling, or mechanistic reasoning, performance degraded sharply. This is not just a scaling problem: throwing more parameters at the task does not fix the gap between pattern matching and scientific reasoning, much like how more training data alone doesn't solve the hardest problems for AI coding agents, as measured by human hour savings.

"Models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning," the authors write. This suggests that LLMs are currently better suited to being data assistants than autonomous scientists.

A five-stage error analysis framework

One of the most useful contributions of SDABench may be its five-stage error analysis framework, which pinpoints exactly where LLMs fail. The stages are: scope identification, variable identification, procedure selection, relationship modeling, and conclusion drawing.

The findings are nuanced. More advanced models, such as the latest variants from OpenAI, Google DeepMind, and Anthropic, reliably identify the relevant scope and variables for a scientific question. But they still struggle to select appropriate analytical procedures, model the relationships between variables, and draw valid conclusions that align with the assumptions of the chosen method. This asymmetry, strong on description, weak on inference, mirrors what educators have seen in the classroom: AI tools can produce outputs, but they don't necessarily build understanding, a pattern documented in the cognitive debt study.

This is not a simple scaling story: throwing more parameters at the problem does not fix the fundamental gap between pattern matching and scientific reasoning.

What the benchmark reveals about AI science

The research has immediate implications for the growing field of "AI scientists", automated systems that claim to carry out end-to-end literature review, hypothesis generation, experimentation, and paper writing. Shi and Ren's work suggests that these systems may produce convincing-looking outputs while making fundamental logical errors that a human reviewer would catch.

SDABench also exposes a deeper issue: the conflation of "being able to generate an output" with "understanding the appropriate method." A model that can write Python code to run a t-test does not necessarily understand that a t-test assumes normality and independence, nor can it choose between a t-test and a Mann-Whitney U test based on data characteristics.

Beyond correlation into causation

The paper's focus on causal and mechanistic reasoning is particularly timely. The past year has seen a surge of interest in causal inference within the AI community, but most benchmarks do not explicitly test a model's ability to distinguish between correlation and causation. SDABench does, and the results suggest that this remains a frontier capability that even frontier models have not conquered. This mirrors the challenge seen in other domains: models can match patterns but struggle with deeper causal understanding, a limitation that even top-performing agents face, as noted in the MiniMax coding benchmark analysis.

Mechanistic reasoning, understanding the underlying processes that generate data, is even more challenging. None of the evaluated models performed well on this axis.

Implications for the future of AI in science

SDABench does not argue that LLMs are useless for science. On the contrary, it provides a fine-grained diagnostic tool that can guide development. A researcher designing a better scientific LLM can now target specific error types, such as procedure selection or relationship modeling, and track improvements.

The paper also raises a critical question for the AI safety and alignment community: if an agent cannot reliably select the correct statistical test or reason about cause and effect, should it be allowed to autonomously design experiments or draw conclusions from data? The benchmark provides a concrete testbed for answering that question. This kind of diagnostic granularity is what makes benchmarks like SDABench valuable, not just ranking models, but revealing where they break, a principle that also drives the cost-versus-capability shift in frontier models.

As the field of AI-driven scientific discovery matures, SDABench offers a rigorous, multidimensional measure of progress, and a sobering reminder that describing a dataset is not the same as understanding it.