LLMs can describe data. They cannot reason through it. A new benchmark proves the gap is real.

Large language models can write code, summarize literature, and produce plausible-looking scientific workflows. A new paper, published on arXiv on July 13, 2026, argues that the bar for scientific discovery sits much higher, and that current benchmarks have been measuring the wrong thing. The problem echoes what researchers found with sandbox evaluations, where agents look impressive until you test them on fundamentals, as the gap between sandbox and real-world agent failures shows.

SDABench, introduced by researchers led by Chuhan Shi and Xiaoquan Ren, reorganizes evaluation around six capabilities: descriptive, exploratory, inferential, predictive, causal, and mechanistic. Instead of rewarding models for stitching together a workflow that ends with the correct number, it tests whether they actually understand which analytical method fits the question, can model unobserved variables, and can reason about causation rather than correlation.

The benchmark contains 527 real-data instances (SDA-Real) and 6,000 synthetic instances (SDA-Synth), each in both multiple-choice and open-ended formats, built through an automated pipeline. The domains span Biology, Chemistry, Environment, Geography, and Physics.

Descriptive mastery, inferential collapse

When the researchers evaluated 15 representative LLMs, the pattern was stark. Models handled descriptive analysis, summarizing means, variances, and distributions, with solid performance. But the moment a task demanded choosing the right assumptions, modeling unobserved variables, or mechanistic reasoning, performance fell off sharply. It is not a scaling problem. More parameters do not close the gap between pattern matching and scientific reasoning, any more than more training data fixes the gap between benchmark scores and real work for coding agents, where SWE-bench Verified scores collapse on private enterprise code.

"Models handle descriptive analysis well but degrade sharply on tasks requiring assumption selection, latent-process modeling, or mechanistic reasoning," the authors write. That puts LLMs closer to data assistants than autonomous scientists.

A five-stage error analysis framework

One of the most useful contributions of SDABench may be its five-stage error analysis framework, which pinpoints exactly where LLMs fail. The stages are: scope identification, variable identification, procedure selection, relationship modeling, and conclusion drawing.

The findings are nuanced. Advanced models, such as the latest variants from OpenAI, Google DeepMind, and Anthropic, reliably identify the relevant scope and variables for a scientific question. But they still struggle to select appropriate analytical procedures, model the relationships between variables, and draw valid conclusions that fit the assumptions of the chosen method. That asymmetry, strong on description and weak on inference, is familiar to educators: AI tools can produce outputs without building understanding, a pattern documented in how AI tutors miss the way math concepts build on each other.

What the benchmark reveals about AI science

The research bears directly on the growing field of "AI scientists," automated systems that claim to carry out end-to-end literature review, hypothesis generation, experimentation, and paper writing. Shi and Ren's work suggests these systems can produce convincing outputs while making fundamental logical errors that a human reviewer would catch.

SDABench also flags a subtler problem: the conflation of generating an output with understanding the appropriate method. A model that can write Python code to run a t-test does not necessarily understand that a t-test assumes normality and independence, nor can it choose between a t-test and a Mann-Whitney U test based on data characteristics.

Beyond correlation into causation

The paper's focus on causal and mechanistic reasoning is timely. Causal inference has drawn increasing interest in the AI community over the past year, but most benchmarks never explicitly test whether a model can tell correlation from causation. SDABench does, and the results suggest this remains a frontier capability, one that even leading models have not conquered. The same pattern shows up in other domains: models match patterns but struggle as tasks demand deeper understanding, a limitation documented in why even the best VLMs fail at 3D action tasks.

Mechanistic reasoning, understanding the underlying processes that generate data, is even harder. None of the evaluated models performed well on that axis.

Where AI scientists go from here

SDABench does not argue that LLMs are useless for science. It provides a fine-grained diagnostic instead. A researcher designing a better scientific LLM can now target specific error types, such as procedure selection or relationship modeling, and track improvements.

The paper also raises a critical question for the AI safety and alignment community: if an agent cannot reliably select the correct statistical test or reason about cause and effect, should it be allowed to autonomously design experiments or draw conclusions from data? The benchmark provides a concrete testbed for answering that question. Diagnostic granularity of this kind reveals where models break instead of just ranking them, the same logic behind the Messier audit of agent capability across 30 benchmarks.

As AI-driven discovery matures, SDABench offers a rigorous measure of progress and a sobering reminder: describing a dataset is not the same as understanding it.