Harness evolution looks good until you run a fair test

The paper, available on arXiv at 2607.12227, is a blunt methodological critique. It revisits how automatic harness evolution, the practice of using unit test feedback to iteratively search for better harness configurations, is typically evaluated, and finds the standard protocol fundamentally broken. This kind of scrutiny is overdue, especially as the agent ecosystem faces a reproducibility crisis, as noted in reports on agent memory failures.

The two core concerns are simple enough to state. First, harness evolution is itself a search procedure that consumes inference compute and task feedback. To know whether its gains come from better harness design or just from the extra search, you have to compare it to equally expensive baselines, parallel sampling, sequential refinement, that get the same feedback but don't change the harness. The paper argues this baseline is almost never run. That parallel with the AgentScope team's earlier work is telling: if harnesses matter, then isolating their effect from search artifact matters too.

Second, if you search the same benchmark you report on, you are measuring how well the harness can adapt to that specific set of tasks, not how transferable the improvement is. The paper calls this out bluntly: the reported gains risk overfitting to that specific task set.

What the experiments actually found

On Terminal-Bench 2.1, using GPT-5.4 and Claude Opus 4.6, the researchers compared harness evolution against simple test-time scaling methods under matched inference budgets. The results are not kind to the prevailing narrative.

Automatic harness evolution did not consistently outperform simple parallel sampling or sequential refinement, even when unit test feedback was available. When the search and evaluation tasks were separated, the evolved harness tested on held-out tasks it had not seen during the evolution, the improvements were marginal at best. This echoes findings from analysis of parallel agent systems: concurrency alone doesn't guarantee better results.

The authors are careful not to overstate. Their takeaway is not that harness evolution is ineffective, but that its benefits need to be assessed using fair setups, strong baselines with comparable budgets, and benchmarks that are genuinely sensitive to harness design.

This echoes related work from the AgentScope team, which found that harness design can swing scores by over 11 points for smaller models, but that earlier work was about evaluating harnesses, not evolving them. The new paper extends the same principle to the evolution process itself: if harnesses matter, then properly isolating their effect from search artifact matters too. It's a reminder that formal architectures for taming tool explosions are only as good as the evaluation that validates them.

Why this matters for the field

The paper lands at a time when the AI agent ecosystem is scrambling for reliable evaluation. The list of related papers flagged by Semantic Scholar is itself revealing. There is 'Do Agent Optimizers Compound?', 'SEAGym', 'MemoHarness', 'Harness Updating Is Not Harness Benefit', 'Evolving Agents in the Dark', 'Self-Harness', and 'TTHE: Test-Time Harness Evolution', a whole subfield is forming around the question of how to evaluate the evaluators. For context, similar debates are playing out in developer tooling reviews, where the gap between benchmark scores and real-world utility is a recurring theme.

The critique also raises a broader point about the agent evaluation pipeline. As noted in our earlier coverage, harness design can swing scores by over 11 points for smaller models. If evolution adds another layer of search that is not properly separated from measurement, the field risks building a house of cards where reported scores reflect search depth and benchmark familiarity rather than real agent capability.

The code is available at https://github.com/rethinking-harness-evolution for anyone who wants to replicate the experiments or extend them to other model families and task sets.