Your AI agent keeps failing? It might be the harness, not the brain

When an AI agent fails to complete a multi-step task, the natural instinct is to blame the model. But a new benchmark from the AgentScope team suggests the real problem might sit in the scaffolding around it. the-1115-problem-when-ai-writing-challenges-become-a-test-of-editorial-integrity

PawBench, released today as part of the OpenJudge ecosystem, gauges how large language models and the harnesses that orchestrate them work together as a unit. It runs 9 models across 3 harnesses on 150 curated tasks, producing 4,050 test cells that aim to untangle where agent failures actually start.

The model versus the harness

Most benchmarks evaluate models in isolation, feeding them prompts and scoring outputs. That is not how real-world agents get deployed. In practice, the model sets the upper bound on what an agent could do, and the harness decides whether that capability reliably translates into successful task execution. IBM's new open-source agent framework cuts the…

PawBench formalizes this as a function: Agent Performance equals f(Model, Harness). The v1.0 release includes tasks drawn from six high-quality agent benchmarks: claweval, qwenclawbench, pinchbench, qwenpawbench, skillsbench, and wildclawbench. Each task gets tagged along five dimensions: application scenario, atomic capability, complexity, input modality, and runtime environment.

All tasks run inside Docker sandboxes with full traceability, which makes it possible to connect benchmark scores back to actual execution behavior. The final score combines automated graders, rule checks, and sub-assertions, with LLM-as-judge for more semantic outputs. Ifbench reveals the instruction-following gap that…

Harness gaps are real and measurable

The headline finding across the 4,050 cells is that harness design can introduce significant performance variation. Two extremes illustrate the point. Claude Opus 4.6 shows a spread of only 2.3 points across different harnesses. The Qwen3.6-35B-A3B model shifts by a full 11.5 points depending solely on the harness.

Trace analysis points to three common sources of failure: the model losing track of the current working directory, misjudging whether a file was actually written, or choosing the wrong first tool when the tool list gets too large. Larger models appear better at compensating for missing context; they infer paths, filter large tool lists, and check whether artifacts were actually produced. Smaller models are more brittle.

The takeaway is not that small models are weak. It is that they lean more heavily on harness structure. A well-designed harness can shrink the gap considerably.

Skills and web search reveal new failure patterns

PawBench includes two task categories that expose distinct failure dynamics. The first is skill-related tasks, which simulate developers who store project-specific skills directly inside their workspace. Across all three harnesses, skill tasks were consistently more difficult than tool use, planning, or reasoning. MiniMax launches M2.7 model with strong software…

Two issues stand out: the harness must surface skills clearly, name, scope, usage, and the model must reliably decide to invoke them. If either side breaks, the model bypasses the skill and tries to solve the task with general reasoning, often failing.

Web search tasks test the model's ability to search the web, fetch content, and conduct deeper research. PawBench recreates the default developer experience, cloning a fixed version, adding the LLM key, and running, without assuming every search API key is pre-configured. Results showed that some models simply ignored broken tool outputs, while others fell into loops trying to rerun failed tools. Strong models can route around missing tools; weaker models depend on the harness keeping the search path stable and explicit.

Four principles for better harness design

Based on the benchmark results, the AgentScope team proposes four principles for effective harnesses. First, be explicit: models cannot act on information they do not have. Tell the model where it is, what resources exist, and what outputs are expected. Never assume the model will infer these details.

Second, keep tooling sufficient and efficient. Provide the tools that matter, make sure critical tools are usable by default, but avoid overwhelming the model with unnecessary options.

Third, verify, don't trust. Do not rely solely on what the model says. Verify artifacts, files, outputs, and execution results instead of just relying on agent self-reporting.

Fourth, build in recoverability. Many failures are recoverable if the framework provides useful feedback and retry opportunities. Provide critical information such as current state, missing requirements, and existing artifacts, and give the model a structured opportunity to recover. Microsoft's new platform gives scientists a governed…

The most important result from PawBench is not which model ranks first. It is that agent performance is not a property of the model alone. PawBench v1.0 is fully open source, and the team welcomes new harnesses, models, tasks, and contributions from the community.