ViQ solved the trade-off that broke every other visual tokenizer

Representing images as discrete tokens, like language, has been a dream for multimodal AI since people started building it. The pitch is clean: a unified space for vision and text, simpler models, less compute. The problem is that making an image discrete costs information. You either keep the pixels and miss what they mean, or you capture the semantics and throw away the detail. A new paper from Tencent-Hunyuan, titled "ViQ: Text-Aligned Visual Quantized Representations at Any Resolution," proposes a framework that tries to hold on to both. The trick is splitting the problem into two stages, which is the kind of obvious-in-retrospect move that makes you wonder why nobody tried it this way before, as noted in DiScoFormer's similar two-stage unification approach.

The core challenge: quantization without sacrifice

The existing approaches to discrete visual representation sort into two camps. Reconstruction-first methods, like VQ-VAE and its offshoots, replicate pixel-level detail but miss the high-level concepts language models need. Contrastive pre-training, like CLIP, gives you semantic structure but compresses or discards visual granularity, making it useless for precise reconstruction. The state of the art has been stuck in this binary for years, as a 15-part series on AI generation recently documented in detail.

ViQ splits quantization learning into two stages. The first, text-aligned pre-training, exposes the visual encoder to semantic-rich supervision from a pretrained language model. Critically, it handles native-resolution inputs, no cropping, no resizing, no patchifying before encoding. The second stage, feature discretization, uses a novel proximal representation learning strategy to progressively compact the feature space, combined with a position-aware head-wise quantization mechanism that adapts to arbitrary resolutions. This modular emergence from data echoes what Ai2's EMO demonstrated for MoE models.

Technical innovation: proximal learning and head-wise quantization

The proximal representation learning strategy is the paper's key contribution. Instead of forcing the whole feature space into a fixed-size codebook in one pass, it applies iterative compression, gradually clustering and quantizing features while preserving neighborhoods from the continuous space. That keeps both local detail and global structure intact. The idea of preserving neighborhoods during compression has parallels in JetLong's bifocal attention mechanism, which tackles a similar trade-off in the context window domain.

The position-aware head-wise quantization mechanism adds flexibility. Standard vector quantization treats each feature position independently, which fails to capture spatial relationships in images of varying size. ViQ introduces multiple quantization heads, each attending to a different spatial scale, and weights their outputs based on positional context. The encoder can allocate more bits to semantically important regions without wasting capacity on uniform backgrounds.

Empirical performance: competitive quality with major speed gains

Extensive experiments across multimodal benchmarks, including image captioning, visual question answering, and text-to-image generation, show ViQ's discrete representations performing within 1 to 3 percentage points of state-of-the-art continuous vision encoders (such as SigLIP and DINOv2), while using fewer parameters and producing discrete tokens. On low-level reconstruction tasks, ViQ outperforms other discrete methods by a substantial margin. The ability to match continuous encoders while producing discrete tokens matters for agent architectures, as GPT-5.5's benchmark dominance shows what discrete representations enable at the frontier.

The training acceleration might be the most striking finding. When ViQ's quantized representations are used in multimodal LLM training, the authors report speedups of 20% to 70% depending on the base LLM size and training recipe. The acceleration comes from reduced dimensionality and simpler architecture of discrete tokens, which slash both memory bandwidth and matrix multiplication costs during forward passes.

"ViQ is the first framework to demonstrate that discrete visual representations can simultaneously match continuous encoders on multimodal tasks and offer dramatically faster training," the paper states. The authors attribute the efficiency gains to the compactness of quantized tokens and the removal of expensive continuous-to-discrete projection layers.

Implications for multimodal AI

If ViQ holds up in practice, it could accelerate the development of large-scale multimodal models that process images and text through a single token space. Unified representations are crucial for next-generation assistants, robotics perception, and real-time vision-language systems where latency and compute budget are constrained. The resolution-agnostic design also tackles what the unified solution for AI search's plumbing problem calls a foundational infrastructure challenge.

The ability to handle native-resolution inputs is particularly important. Many vision encoders require resizing images to a fixed resolution, discarding detail and introducing distortion. ViQ's resolution-agnostic design means it can process high-resolution documents, microscopy images, or satellite photos without pre-processing, retaining information that would otherwise be lost. This flexibility is crucial for robots adapting to novel environments without retraining, where input resolution varies dramatically.

The paper also notes that ViQ's discrete tokens are compatible with existing transformer architectures and can be fed directly into LLMs without architectural modifications, a practical advantage for teams integrating vision without rewriting their model stack.

Limitations and open questions

The study's primary limitation: all experiments ran on curated, clean datasets (COCO, Flickr30k, and a proprietary internal dataset). Performance on noisy, domain-shifted, or low-quality images, common in real-world deployments, remains untested. The codebook size and quantization overhead at inference time are not thoroughly analyzed, if the number of heads or codebook entries grows too large, the efficiency gains could erode. Real-world robustness is exactly what the new safety framework for critical systems is designed to stress-test.

The paper does not release a pre-trained model or code at time of publication, which limits reproducibility. External validation will be essential to confirm that the reported speedups translate to production workflows.

Conclusion

ViQ is a thoughtful step toward a unified representation space for vision and language, tackling the long-standing trade-off between semantic richness and visual detail in discrete tokenization. By combining text-aligned pre-training with progressive quantization and position-aware heads, Tencent-Hunyuan's team achieves competitive multimodal performance while delivering substantial training speedups. As multimodal models grow in scale and ambition, efficient and detail-preserving discrete representations may become a key enabler. Building such systems from scratch is exactly what the 2025 guide to AI writing pipelines covers for the text domain, ViQ does the same for vision.