The three-stage rhythm that stops AI from seeing things that aren't there

Vision-language models can answer questions about images pretty well at this point. But when they hallucinate, describing objects that aren't there or getting spatial relationships wrong, it's often not because the visual information is missing. The model just lost track of it somewhere inside its own reasoning stack.

New research from Tongji University and Hong Kong Polytechnic University, published on arXiv, offers the most detailed mechanistic account yet of how this happens and what to do about it. The authors identify a three-stage internal rhythm across the transformer layers of VLMs, formalize the critical middle stage as the Visual Relay Window (VRW), and introduce a lightweight inference-time control framework called TRACE that adaptively schedules this window based on what each question needs. The approach addresses a similar core challenge seen in other modalities, where attention management determines whether agents succeed or fail, as noted by benchmark studies on agent failures.

The three-stage rhythm of multimodal attention

The researchers tracked two attention probes across layers of ten open-weight VLMs, including variants of Qwen-VL, InternVL, and LLaVA. The first probe measured how strongly answer tokens look back to the question. The second measured how strongly visual tokens attend to one another. They found a consistent pattern across all models: early layers emphasize question-conditioned organization, middle layers shift to intra-visual consolidation, and later layers return to answer formation.

That middle phase, the Visual Relay Window, is where visual evidence gets actively assembled before being handed off to the language side for reasoning. Its geometry is not fixed. Detail-heavy tasks like counting tend to need longer and later relay windows, while high-level reasoning tasks benefit from an earlier handoff. This adaptive scheduling principle echoes findings from inference scheduling research that treats efficiency as a systems problem.

Why unsupported answers emerge

The study demonstrates a causal link between relay mismatch and hallucination. Using the HaloQuest benchmark, the authors compared grounded and unsupported continuations generated from the same image-question pairs. Unsupported answers consistently showed narrower relay windows, earlier relay termination, and weaker visual anchoring after the handoff. In a patching experiment, replacing visual-token residual activations inside the estimated VRW with grounded-branch activations recovered correct responses in 41.4% of unsupported cases. For comparison, pre-relay patches recovered only 8.7%, and post-relay patches 11.2%.

Thinking variants of the same base models (e.g., Qwen3-VL-4B Instruct vs. Thinking) showed better relay alignment with task demands and stronger post-handoff anchoring. This suggests their advantage comes partly from more appropriate internal scheduling of visual evidence. The finding resonates with broader work on chain-of-thought monitoring, which shows that controlling internal reasoning steps is both promising and difficult.

TRACE: scheduling visual support without retraining the model

Building on these findings, the team built TRACE (Task-adaptive Relay Anchoring and Controlled Evidence Scheduling), a framework with only 199,244 trainable parameters for 4B models and 297,548 for 8B models. It works in three stages. A lightweight MLP predicts each layer's relay score from prefill-time attention statistics. A task-aware scheduler expands or contracts the relay window by applying a soft bias to visual-to-visual attention logits. A post-handoff anchoring module preserves access to the selected visual support during answer generation, with strength modulated by decoding uncertainty.

The modules are trained from pseudo-labels induced by the VRW estimator, keeping the base VLM frozen. Training takes roughly 8 hours on four NVIDIA RTX 5090 GPUs. The approach mirrors the philosophy behind tokenizer-free architectures, where architectural simplicity enables efficient adaptation without heavy retraining.

Results: grounding gains without trade-offs

TRACE was evaluated on four open-weight backbones (Qwen3-VL-4B/8B and InternVL3.5-4B/8B) across seven benchmarks covering document understanding, visual reasoning, and hallucination-sensitive settings. On grounding-sensitive benchmarks (HallusionBench, VlmsAreBlind, CountBench), it delivered an average improvement of 4.33 points, with gains up to 6.6 points on HallusionBench with InternVL3.5-8B. It also improved reasoning-heavy benchmarks like MathVista, suggesting the mechanism addresses both hallucination and logical reasoning through a single underlying control principle. Such efficiency gains are becoming increasingly important as synthetic data and training bottlenecks shift the focus from raw scale to smarter resource use.

Compared to eight baselines, including chain-of-thought prompting, visual contrastive decoding, and several attention-reallocation and activation-steering methods, TRACE achieved the highest average score on every backbone, with total gains of 2.84 to 3.18 points over vanilla inference across the full benchmark suite.

The team's code is available on GitHub.