The bottleneck holding back AI agents isn't exploration, it's evaluation

AI agents can already explore. They throw code at a browser, try a shell command, fail, retry. That part is fine. What they cannot do, not reliably, is tell whether they got it right. And without that signal, reinforcement learning stalls. Two papers published on Hugging Face recently confront this from opposite angles, and together they draw a map of where agent evaluation is headed. IBM's new open-source agent framework cuts the…

The first, AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation (arXiv 2604.18240), measures how well a judge agent can verify behavior by actively poking at the world, running queries, clicking buttons, scrolling results, before deciding if a step was correct. The second, HeavySkill: Heavy Thinking as the Inner Skill in Agentic Harness (arXiv 2605.02396), argues the opposite: the most reliable judge lives inside the model's own parameters, not on the internet.

The verification gap

Current practice relies on two crutches. Rule-based verifiers check a narrow set of conditions and miss everything else. LLM-as-a-Judge models produce qualitative verdicts without ever touching the environment. Both break as agents move from curated benchmarks to open-ended tasks that need grounded, factual answers. Ifbench reveals the instruction-following gap that…

AJ-Bench was built to measure how wide that gap actually is. Its authors, from Zhejiang University and collaborators, created 155 tasks across search, data, and GUI domains, with 516 annotated trajectories. The benchmark tests three capabilities: information acquisition (can the judge find the right context?), state verification (does it correctly assess the environment's condition?), and process verification (can it check each step?).

Results showed that agent-based judges, the ones that can run queries and interact with the environment, consistently beat static LLM baselines. But even the best agent judges still tripped up on process verification, especially in multi-step GUI tasks where intermediate states must be confirmed.

"A key bottleneck in scaling RL for AI agents is not only exploration, but evaluation," the authors write in a community note. "As agents operate in broader, more open-ended environments, we need judge agents that can use tools, verify environment states, and produce grounded feedback signals."

The same bottleneck shows up in other recent work. AI agents tasked with enterprise Java migration, for example, fail not because they cannot write code, but because they cannot judge whether the migration is complete without a build server telling them. AI agents can't tell when a Java migration is actually done

Reasoning as internal skill

HeavySkill takes a different approach. Instead of teaching the agent to reach for external tools, it proposes that the most scalable evaluation is a two-stage reasoning skill embedded directly in the model: parallel reasoning followed by summarization.

Under this framework, an agent facing a complex task first generates K independent reasoning trajectories in parallel, then synthesizes them into a final answer through critical analysis. The entire process is a single, learnable skill, one that can be improved through RL rather than orchestration of external tools. The paper's central insight is that many existing agentic harnesses obscure what actually drives performance. "The underlying mechanism that truly drives performance remains obscured behind intricate system designs," the authors argue.

HeavySkill consistently outperforms traditional Best-of-N strategies, with stronger LLMs approaching Pass@N performance. This echoes findings from other research: on-policy skill distillation, where models learn from their own completed trajectories, can boost agent training without relying on external memories. OPID feeds agents dense rewards from their own past, no…

Where the two paths converge

Despite their philosophical differences, both papers converge on the same point: better judges are the key to scaling. Whether the judge interacts with the environment or with its own internal reasoning, the fundamental need for grounded, process-aware verification is the same. HeavySkill's authors see a clear path: "The depth and width of heavy thinking, as a learnable skill, can be further scaled via reinforcement learning." AJ-Bench's contributors are equally direct: "Better judges expand the range of behaviors that can be evaluated reliably."

The tension between external verification and internalized reasoning may eventually resolve into a hybrid. HeavySkill's pipeline could be augmented with environment interaction at the summarization stage. Or AJ-Bench's benchmark could evaluate HeavySkill-style judges in dynamic settings, testing whether internalized reasoning holds up when the environment fights back.

This mirrors a broader pattern in agent research. Frameworks like SkillCoach already try to teach agents to evaluate their own skill selection and usage, and IBM's CUGA harness handles orchestration so agents can focus on judgment. Your AI agent passed by accident. SkillCoach grades the…

What comes next

Both papers are early-stage. Neither has been cited by other models on Hugging Face yet. The community is only beginning to recognize the verification bottleneck. But the direction is clear. As RL training for agents scales, the limiting factor will not be compute for exploration. It will be the quality of the evaluation signal. The race to build better judges has begun.

Data and code for AJ-Bench are available at the project website. HeavySkill's authors have not released their code yet, but the paper's framework is described in enough detail to be reproduced.