Your AI agent keeps failing? It might be the harness, not the brain

When an AI agent fails to complete a multi-step task, the natural instinct is to blame the model. But a new benchmark from the AgentScope team suggests the real problem might sit in the scaffolding around it, echoing prior critiques of how benchmarks can miss the full picture.

PawBench, released today as part of the OpenJudge ecosystem, gauges how large language models and the harnesses that orchestrate them work together as a unit. It runs 9 models across 3 harnesses on 150 curated tasks, producing 4,050 test cells that aim to untangle where agent failures actually start.

The model versus the harness

Most benchmarks evaluate models in isolation, feeding them prompts and scoring outputs. That is not how real-world agents get deployed. In practice, the model sets the upper bound on what an agent could do, and the harness decides whether that capability reliably translates into successful task execution. The gap between isolated tests and deployed agents has been documented as a known blind spot in agent evaluation.

PawBench formalizes this as a function: Agent Performance equals f(Model, Harness). The v1.0 release includes tasks drawn from six high-quality agent benchmarks: claweval, qwenclawbench, pinchbench, qwenpawbench, skillsbench, and wildclawbench. Each task gets tagged across five dimensions: application scenario, atomic capability, complexity, input modality, and runtime environment.

All tasks run inside Docker sandboxes with full traceability, which makes it possible to connect benchmark scores back to actual execution behavior. The final score combines automated graders, rule checks, and sub-assertions, with LLM-as-judge for more semantic outputs, following patterns seen in earlier benchmark audits.

Harness gaps are real and measurable

The central finding across the 4,050 cells is that harness design can introduce significant performance variation. Two extremes illustrate the point. Claude Opus 4.6 shows a spread of only 2.3 points across different harnesses. The Qwen3.6-35B-A3B model shifts by a full 11.5 points depending solely on the harness.

Trace analysis points to three common sources of failure: the model losing track of the current working directory, misjudging whether a file was actually written, or choosing the wrong first tool when the tool list gets too large. Larger models appear better at compensating for missing context; they infer paths, filter large tool lists, and check whether artifacts were actually produced. Smaller models are more brittle. This pattern aligns with the broader finding that orchestration now beats raw size in many agent setups.

The takeaway is not that small models are weak. It is that they lean more heavily on harness structure. A well-designed harness can shrink the gap considerably.

Skills and web search reveal new failure patterns

PawBench includes two task categories that expose distinct failure dynamics. The first is skill-related tasks, which simulate developers who store project-specific skills directly inside their workspace. Across all three harnesses, skill tasks were consistently more difficult than tool use, planning, or reasoning.

Two issues stand out: the harness must surface skills clearly