Why your AI model's self-review is structurally blind to visual defects

When an AI model generates a web page, a slide deck, or a scientific figure, how do we know it actually looks right? The field's default answer is to ask another model, a vision-language model (VLM) reading a screenshot of the output. A new paper from Pine AI and the University of Washington argues this approach is not just noisy, but structurally blind. The problem is reminiscent of how sandbox benchmarks hide how agents really fail, as a recent HKU study on agent failure modes showed.

The blind judge

The paper, Interaction Scaling: Grounding the Third Axis of Test-Time Compute, presents a striking demonstration. On a probe of 15 dense academic-figure renders, the VLM-on-screenshot judge rated 14 as 'perfect.' When the same artifacts were measured by a deterministic DOM geometry instrument, one reading every element's true bounding box, only three were actually clean. The rest had section titles printed on top of each other, labels spilling out of boxes, and content cropped off-frame.

'The failure is mechanical,' the authors write. Screenshots are captured at a fixed resolution; content overflowing the canvas is cropped off-frame before the image reaches the judge, and small in-frame overlaps fall below its visual acuity. This mirrors the logic behind why Alibaba's Qoder prioritizes visibility over magic in agent behavior.

Beyond reasoning and sampling

The paper's core contribution is reframing interaction as a third axis of test-time compute, distinct from reasoning (longer chains of thought) and sampling (best-of-N selection). The key variable is what the authors call grounding: feedback that comes from an instrument observing the artifact's actual form or behavior, not from a model voicing an opinion about it.

'A model re-reading its own output cannot learn anything it did not already know,' the authors explain. A critique from the same weights that produced the artifact is post-processing, adding no information about the correct answer. Grounded feedback, like a failing test traceback or a measured bounding box, imports a real observation each cycle, breaking through what they call the 'internal ceiling.' The same principle anchors how M3's math verifiers avoid cheating by grounding feedback in formal truth.

In controlled experiments, at a matched token budget, both reasoning-only and best-of-N with an oracle verifier plateaued below 87% pass rate on hard coding tasks. Every interaction strategy kept climbing. The proposer, reviewer harness reached a perfect 100% pass rate with zero seed variance.

Coverage is what matters

The framework introduces a coverage principle: a feedback channel helps exactly as far as its instrument's observational reach. This explains why, on the same code suite, adding a linter (grounded but observing only form) bought nothing over bare critique, while execution feedback converged roughly 2.5× cheaper and to a higher ceiling.

The decisive control came from swapping only the reviewer's instrument on visual modalities. When a VLM read a screenshot, its edits made geometry worse, increasing defects on slides, because it was blind to the true layout. A deterministic geometry reviewer, measuring actual bounding boxes, sharply reduced defects on both dense slides (73% reduction) and academic figures (74% reduction). This finding aligns with research showing that counting human hours reveals agent productivity more honestly than self-reported metrics.

'One reviewing pass, opposite sign,' the authors note, 'decided entirely by whether the signal covers the defect.'

Distributional variance is a budget

The paper also includes a cautionary finding about training. When the authors distilled the harness's interaction quality into an 8B parameter student via supervised fine-tuning, the student recovered about half the teacher's single-sample capability. But adding reinforcement fine-tuning, which improved per-turn consistency, actually hurt pass@k performance. The reason: variance in the output distribution is the very resource that sampling converts into quality. RFT spent from that budget.

This echoes a broader lesson about the hidden costs of optimization, as a 26,000-student study on cognitive debt in AI education found: squeezing variance can backfire.

Implications for the field

The practical lesson, the authors argue, is not 'add more loops' but 'ground the loop you add.' For visual artifacts, this means measuring the rendered DOM, never a screenshot, on both the reviewer and the scorer. The finding has immediate implications for the many papers measuring self-improving visual generation with VLM judges that cannot see the defects at issue.

'Interaction scaling is real, distinct from reasoning and sampling, and predictable from coverage,' the authors conclude. 'Once you ground the metric, it is plainly visible.'

The code and website are available at the links in the paper.