Qwen2.5-Omni outperforms Gemini-1.5-Pro on OmniBench, fits under 12GB

The Qwen team at Alibaba Cloud released Qwen2.5-Omni on March 26, 2025, and the numbers in its repository do most of the talking. The model is end-to-end multimodal: it takes text, images, audio and video as input, and responds with streaming text and natural speech. The 7B version hit the top of Hugging Face's trending list on April 2, a 3B variant followed on April 30, and the team published a technical report (arXiv 2503.20215) alongside the release.

The headline result is the benchmark gap. On OmniBench, which tests tasks that force a model to use several modalities at once, Qwen2.5-Omni-7B averaged 56.13% and the 3B scored 52.19%. Gemini-1.5-Pro posted 42.91%. These are the team's own published figures, and they are the reason this release deserves more than a skim. OmniBench belongs to a newer generation of integrated evaluations, the kind the field moved to after static benchmarks collapsed, tracked in what replaced the old benchmarks.

Audio reasoning tells a similar story. On MMAU, the 7B averaged 65.60%, ahead of the 54.90% recorded by Gemini-Pro-V1.5 in the same table. The team's June 9 update claims first on the MMAU leaderboard and the top open-source spot on MMAR. Three days later it reported first place among open-source models on MMSU, a spoken language understanding and reasoning benchmark.

Figures as published in the Qwen2.5-Omni repository. The MMAU table labels the Gemini result Gemini-Pro-V1.5; the OmniBench table uses Gemini-1.5-Pro.

The margins hold in single-modality tests too. On LibriSpeech test-other, the 7B's word error rate is 3.4, slightly better than Whisper-large-v3's 3.6. On Common Voice 15 it beats Whisper-large-v3 on all four language splits listed, including a 5.2 versus 12.8 gap on Chinese. The speech work is part of a wider Qwen audio push that has since expanded into dedicated ASR models, as the quiet expansion of the Qwen3 family shows.

Thinker-Talker and real-time speech

The architecture is the part most releases hide behind an API. Qwen2.5-Omni splits into a Thinker, which reasons across text, images, audio and video, and a Talker, which generates speech. A position embedding called TMRoPE aligns video timestamps with audio, so the model can hear what happens on screen while it watches. Real-time voice and video chat went live on Qwen Chat the same day as the release.

Speech output lands close to dedicated TTS systems, a space Alibaba is working on separately with Qwen-Audio-3.0-TTS, a production speech model. On the SEED benchmark's hard set, the RL-tuned 7B posts a 6.54 word error rate for content consistency, against 1.94 for Seed-TTS_RL and 6.83 for CosyVoice 2. Speaker similarity comes in at 0.752, just below Seed-TTS_RL's 0.782. For a model that also reads, watches and reasons, that is a respectable showing.

The 11.64GB build for consumer GPUs

The release that matters for people without a GPU cluster came on May 16. The team shipped 4-bit GPTQ-Int4 and AWQ versions of the 7B. A 15-second video input that needs 31.11GB of VRAM at BF16 drops to 11.64GB with GPTQ-Int4 and 11.77GB with AWQ. The README names RTX 3080, 4080 and 5070 as targets. Days earlier, MNN support had put the model on phones, with the 7B peaking at 5.8GB of memory on Snapdragon 8 Gen 1 and 8 Elite, and the 3B at 3.6GB. The lighter 3B, released April 30, exists so more platforms can run the model at all. The cost logic here is examined more fully in our analysis of the 7B's cost equation.

The trade-offs are visible in the same tables. Quantization costs some accuracy and a little speed: MMLU-Pro drops from 47.0 to 43.76 on the GPTQ build, while VideoMME holds at 72.0 on AWQ against 72.4 native. The team notes the VRAM figures are theoretical minimums, and real usage runs about 1.2 times higher. The 11.64GB number also covers just 15 seconds of video. At 60 seconds it climbs to 29.51GB. The under-12GB story has a short attention span.

Where the self-reported numbers wobble

Deployment has its catches. vLLM serve, the standard OpenAI-compatible serving path, only supports the Thinker, which means text output only; audio generation requires the end2end script. The DashScope API model qwen-omni-turbo is stream-only, and audio output depends on a specific system prompt, "You are Qwen, a virtual human...", without which speech may not be generated at all.

Every figure above comes from the team that built the model. Independent verification of omni models is thinner than for text-only LLMs, because the evaluation setups are newer, and the wider benchmarking shift is still settling, as a 2026 survey of AI benchmarking details. What holds up on inspection is the direction: an open-source 7B outscored a flagship closed model on integrated multimodal tasks, and a 4-bit build puts real-time voice interaction within reach of a 12GB GPU. Whether third-party testing confirms the margins is the question the next benchmark round should answer.