HumanEval Measured Whether AI Could Code. It Never Asked Whether the Code Was Real Work

HumanEval asks a model to complete a Python function from a docstring, then checks the output against unit tests. It's a clean setup: 164 problems, unambiguous pass/fail, no human grader needed. That cleanliness is why it became the reference point for "can this model code" claims throughout the early wave of code-generating LLMs.

It's also why it stopped being useful. The problems are short, self-contained, and old enough to have been copied, discussed, and solved in public repositories thousands of times over. Research into code-model memorization has found a direct relationship between model size and the ability to reproduce known solutions verbatim, the same contamination pattern that undermined MMLU and GSM8K, just applied to code instead of trivia. A model can pass HumanEval by having memorized close variants of exactly these 164 problems, without that skill transferring to a single line of code it hasn't seen before.

The real gap: toy functions versus real repositories

The deeper issue isn't contamination, though. It's scope. HumanEval problems are the kind of thing you'd see in an early coding interview: write a function that reverses a string, write a function that checks if a number is prime. Writing software professionally looks nothing like that. It means understanding an existing codebase, tracing a bug across multiple files, respecting an API someone else designed, and not breaking tests you didn't write.

SWE-bench was built to close exactly that gap, evaluating models on real GitHub issues instead of synthetic functions. The results were not kind to the HumanEval-era narrative. Models that looked strong on isolated function completion saw their success rates collapse once asked to resolve an actual ticket in an actual repository, especially on private, recently created codebases where memorization can't help. Where top models clear 70% to 95% on public, vetted SWE-bench problems, that same tier of model drops to roughly 23% on SWE-bench Pro's private repositories, and further still on its confidential holdout set. That collapse is the real story HumanEval was never built to tell.

What HumanEval is still good for

As a five-minute smoke test, it still works: if a model can't complete a reverse-a-string function, something is badly wrong. But as evidence that a model can be trusted with a production codebase, it never measured the thing everyone assumed it measured. The industry's shift toward SWE-bench, and toward agentic benchmarks that test tool use and multi-file editing rather than single-function completion, is a tacit admission that HumanEval answered a much narrower question than the one people were asking.