On SWE-bench Verified, Top Models Hit 96%. On Private Enterprise Code, They Barely Clear 23%

SWE-bench evaluates a model by handing it an actual bug report or feature request pulled from a real GitHub repository, then checking whether the patch it writes makes the repository's existing test suite pass. No synthetic function stubs, no docstrings written to be easy. Just an issue, a codebase, and a pass/fail gate defined by tests that were already there before the model showed up.

On SWE-bench Verified, a curated, human-checked subset of public issues, the results now look almost solved. GPT-5.6 Sol clears 96.2%. Claude Mythos 5 hits 95.5%. Claude Fable 5 reaches 95.0%, GPT-5.6 Luna 93.0%, and even mid-pack models like Claude Opus 4.8 and Kimi K3 sit around 88% to 89%. Read in isolation, those numbers say frontier models have essentially mastered real-world bug fixing.

Then comes SWE-bench Pro

SWE-bench Pro exists to test exactly the assumption that number invites: does this generalize, or did the models just get very good at this specific, well-trodden public dataset? Pro swaps in issues from private, recently created enterprise repositories, codebases with no chance of having leaked into pretraining data the way years-old public GitHub issues might have.

The results are not close to the Verified numbers. GPT-5 and Claude Opus 4.1, models that were topping the charts on public SWE-bench, drop to between 23.1% and 23.3% on SWE-bench Pro. Restrict the test further to a strictly confidential holdout subset, repositories no model provider could plausibly have seen in any form, and performance falls again, down to a range of 14.9% to 17.8%.

What a 70-point gap is actually measuring

That's not a small calibration difference. It's a different capability. A model that solves 96% of public, well-documented, previously-discussed GitHub issues and 15% of private enterprise ones isn't failing at coding in some abstract sense, it's revealing how much of its apparent skill was shaped by prior exposure to the specific patterns, discussions, and even prior fixes attached to public repositories. Strip that familiarity away, and what's left is a much rougher measure of whether the model can actually reason through an unfamiliar codebase.

This is the sharpest illustration in current benchmarking of a pattern that shows up everywhere once you look for it: apparent mastery on a public, well-known test frequently does not survive contact with genuinely novel, private conditions. For engineering leaders deciding whether to trust an agent with a real production codebase, the SWE-bench Pro number, not the Verified one, is the one that matters. It's also the harder number for a lab to put on a slide.