ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Hugging Face Daily Papers Papers
visual-quantization multimodal-learning discrete-representations efficient-training text-alignment vision-encoder representation-learning

Summary

ViQ presents a visual quantization framework that balances semantic richness and detail preservation in discrete representations, enabling efficient multimodal training with native-resolution inputs by using text-aligned pre-training and proximal representation learning.

A unified representation for text and vision is a natural pursuit, as it enables simpler multimodal modeling and more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics in discrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, a Visual Quantized Representations framework, which is designed to balance semantics and details in discrete representations while supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages: text-aligned pre-training and feature discretization. With text-aligned pre-training, we enhance the visual encoder semantic-rich supervision from the pretrained language model and enable it to process native-resolution visual inputs. During discretization, we propose a proximal representation learning strategy to progressively compact the feature space, along with a position-aware head-wise quantization mechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision in low-level reconstruction. We also show that multimodal training with visual quantized representations largely improves efficiency, yielding up to 20\%-70\% acceleration with different base LLMs and training recipes.
Original Article
View Cached Full Text

Cached at: 06/26/26, 06:05 AM

Paper page - ViQ: Text-Aligned Visual Quantized Representations at Any Resolution

Source: https://huggingface.co/papers/2606.27313

Abstract

ViQ presents a visual quantization framework that balances semantic richness and detail preservation in discrete representations, enabling efficient multimodal training with native-resolution inputs.

A unified representation for text and vision is a natural pursuit, as it enables simplermultimodal modelingand more efficient training. However, representing images as discrete signals in the same way as text inevitably introduces severe information loss. Existing work struggles to balance low-level details and high-level semantics indiscrete representations: reconstruction-oriented representations often lack semantic information, whereas semantically stronger features typically suffer from severe loss of detail. We present ViQ, aVisual Quantized Representationsframework, which is designed to balance semantics and details indiscrete representationswhile supporting inputs at native resolutions, thereby enabling it to serve as a unified and general discrete representation for arbitrary visual inputs. Our approach structures quantization learning into two stages:text-aligned pre-trainingandfeature discretization. Withtext-aligned pre-training, we enhance thevisual encodersemantic-rich supervision from the pretrained language model and enable it to process native-resolution visual inputs. During discretization, we propose aproximal representation learningstrategy to progressively compact the feature space, along with aposition-aware head-wise quantizationmechanism that enables flexible processing of arbitrary resolutions. Extensive experiments on multimodal tasks demonstrate that ViQ achieves competitive performance compared to state-of-the-art multimodal vision encoders with continuous and high-dimensional visual features, while maintaining high precision inlow-level reconstruction. We also show that multimodal training withvisual quantized representationslargely improves efficiency, yielding up to 20\%-70\% acceleration with different base LLMs and training recipes.

View arXiv pageView PDFGitHub3Add to collection

Get this paper in your agent:

hf papers read 2606\.27313

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.27313 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.27313 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.27313 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Channel-wise Vector Quantization

Hugging Face Daily Papers

Channel-wise Vector Quantization (CVQ) replaces patch-wise tokens with channel-wise tokens for image tokenization, enabling a next-channel prediction framework (CAR) that generates images by progressively refining visual details, achieving strong reconstruction and text-to-image generation performance.

Qwen-Image-VAE-2.0 Technical Report

Hugging Face Daily Papers

Qwen-Image-VAE-2.0 is a high-compression Variational Autoencoder suite that improves reconstruction fidelity and diffusability through enhanced architecture, large-scale training, and semantic alignment strategies.

Continuous First, Discrete Later: VQ-VAEs Without Dimensional Collapse

arXiv cs.LG

This paper addresses the issue of dimensional collapse in VQ-VAEs, showing that representations often occupy a low-dimensional subspace. It proposes an 'AE Warm-Up' strategy that trains the model as an unquantized autoencoder first, which improves reconstruction quality and increases effective latent dimensionality.

UniSVQ: 2-bit Unified Scalar-Vector Quantization

arXiv cs.CL

UniSVQ proposes a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices, achieving state-of-the-art performance among scalar methods and matching vector methods with higher throughput.