Video generation's dirty secret is finally on the record

Video generation benchmarks have mostly tested one thing: how well a model turns a sentence into a clip. KlingTeam's two new papers, released on the same day, look at harder problems instead. This kind of honest evaluation is rare, and it matters because production pipelines need more than one-shot clips, as a recent deep dive into agent observability reminded us.

The first, MultiRef-Compass, targets what the authors call multi-reference-to-audio-video generation, or MR2AV. Instead of one text prompt, the model gets several reference images plus instructions, and must produce synchronized audio and video that respects each reference. The benchmark includes 350 samples covering multi-view subject preservation, multi-entity binding, and human-object-scene composition.

Tests on eight representative MR2AV systems show none of them handle the task well. The paper defines four evaluation dimensions Basic Quality, Reference Consistency, Audio-Visual Consistency, and Instruction Following with 14 sub-metrics. The models consistently score lowest on reference consistency and instruction following, which are precisely the dimensions that matter most for practical use. This pattern of models failing at the hard parts echoes what researchers found when studying model routing and specialization.

The second benchmark, KeyFrame-Compass, tackles keyframe-conditioned video generation. Creators increasingly use storyboards a sequence of reference images to guide a video model. The benchmark includes 386 samples across three application domains, two video structures, two prompt granularities, two conditioning formats, and four keyframe densities.

The key finding here is a trade-off that turns out to be fundamental: models that execute keyframes faithfully produce less natural video, and models that produce natural video ignore keyframes. Performance degrades as keyframe density increases. Most open-source models also fail to interpret storyboard-grid inputs as temporally ordered sequences the models treat a grid of images as a single composite instead of a timeline. This echoes a broader problem in AI: agents struggle with temporal understanding and open-ended tasks.

Both benchmarks use a rejudging-enhanced MLLM-as-a-Judge evaluation framework. That means the judging itself is done by a large multimodal model, with a second pass that catches scoring errors. The approach is scalable and auditable, but it also means these benchmarks rely on the same technology they are testing other models against.

KlingTeam released the benchmarks on arXiv on July 15, 2026. The papers received 31 and 34 upvotes on Hugging Face respectively at the time of writing, which is modest interest for a research lab that has produced high-profile video models. The reception may reflect the fact that bad news for the field is less shareable than a new demo. Meanwhile, Nvidia's work on governed agentic systems shows how careful evaluation can still get attention when framed around real-world impact.

The results are consistent with a pattern that has been visible since the first Sora demos: controlled evaluation reveals failures that carefully chosen promotional clips hide. Text-to-video quality has improved dramatically over the last two years. Multi-reference and keyframe conditioning have not kept up.

For anyone building production pipelines on these models, the benchmarks suggest adjusting expectations. A model that produces impressive one-shot clips from a single prompt may produce garbage when asked to preserve a character's appearance across cuts, or to obey an exact sequence of storyboard frames. The trade-off between faithfulness and naturalness is not going away in the current generation of architectures. This is the kind of admission that training trick research is starting to address.