SceneActBench: Why even the best VLMs fail at 3D action

Vision-language models can describe a room in vivid detail, the bookshelf, the lamp, the mug on the table, but ask them to place that mug onto the shelf, and performance tanks. That gap between describing and acting is what a new academic benchmark sets out to measure.

The missing dimension: Why 3D action matters

Existing 3D benchmarks either score textual responses about a scene or test single-object operations like picking up a cube. Neither captures what an agent must do in a full multi-object space: reason, plan, and execute through a tool interface. Several prior evaluations have already flagged how these tests miss real-world behavior when benchmarks misjudge real-world performance.

That gap matters because research labs are building VLM agents for 3D environments: warehouse robots, home assistants, simulation tools. The University of Oxford's GauntletBench, published earlier this year, noted that agents struggle with 3D reasoning tasks in professional applications like modelling. SceneActBench targets the same weakness. See also how VLM navigation agents can be derailed by simple image noise.

Inside SceneActBench: Design, tasks, and the agent loop

The benchmark covers five 3D tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Each task requires the agent to act on a 3D environment using PNG images or sampled video frames as input, sometimes with supplied 3D assets. The key design choice: every task runs through one fixed agent loop, keeping comparisons fair across configurations. The final output is evaluated against hidden ground truth using task-specific geometric metrics.

This unified loop prevents earlier problems where researchers tweaked the evaluation pipeline per model, making scores incomparable. SceneActBench takes the same modular pipeline approach as GauntletBench, aiming for transparent failure analysis rather than score inflation.

The numbers that tell the story: Scores from 38.6 to 50.2

The team tested eleven proprietary VLM configurations. Overall scores ranged from 38.6 to 50.2, meaning even the best model barely exceeds chance. No configuration performed consistently across all five tasks. The narrow spread suggests today's VLMs share a similar ceiling on 3D action, echoing the HSCodeComp benchmark which also found agents plateauing well below human performance.

The abstract does not break down scores per task, but the researchers analyse where and how failures manifest. Early hints point to object permanence, tracking objects that disappear behind others, and the ordering of events like scan then placement as common trouble spots. These failure modes echo findings in VLM visual hallucination studies.

What these failures mean for VLM research

These results align with other recent findings. A Sakana AI-led team replaced human users with VLMs on the Picbreeder platform and found that synthetic archives lacked the boldness of human-created ones; the models stalled on open-ended exploration the Picbreeder study. In a similar vein, lightweight trained modules improved grounding-sensitive benchmarks by several points, but the gap remains large, particularly in tasks requiring precise spatial action like mobile robot navigation.

What SceneActBench makes clear: the VLM field needs benchmarks that test action, not just perception. A model that scores well on image captioning or single-object manipulation can still fail when asked to coordinate vision, spatial reasoning, and sequential actions in a full scene. That blind spot will not fix itself by scaling alone. It needs evaluation suites that push on temporal ordering, multi-object reasoning, and precise spatial execution.