GPT-5.6 Sol almost cracked a physics benchmark built so AI couldn't cheat

CritPt was built by Argonne National Laboratory and the University of Illinois Urbana-Champaign to kill a specific kind of cheating. Once a problem and its answer go public, models can absorb them during training and look like they're reasoning their way to a solution they've actually just memorized. CritPt sidesteps that by drawing on unpublished, graduate-level research problems contributed by more than 60 researchers across over 30 institutions worldwide. The models never saw the answers before test time. On this front, the benchmark shares motivation with efforts like the M3 team's evolutionary verifier approach: finding ways to stop AI from faking understanding.

On Artificial Analysis's independent scoring, GPT-5.6 Sol (max) leads the field with roughly a 32% score, about 5 points ahead of its predecessor, GPT-5.5 (xhigh), and roughly 4 points ahead of Anthropic's Claude Fable 5. The rest of the field drops off sharply, with most frontier models scoring in the single digits to low twenties. These results follow a pattern visible across other benchmark evaluations, like Sonnet 4.6's surprising efficiency gains that reshaped who can afford frontier performance.

Even at the top, a score near 32% is a reminder of how hard this benchmark is by design. These aren't textbook exercises. They're problems drawn from active physics research, and no model is close to solving most of them. Artificial Analysis framed the result as an early signal of frontier models' potential usefulness in real scientific research, not evidence that any model can meaningfully assist with it yet. The gap between prototype performance and production utility remains stubbornly wide, as investigations into vibe coding's hidden costs have also shown.