Paper page - Can Discrete Tokens Serve as Semantically Lossless Representations of Visual Signals?

Source: https://huggingface.co/papers/2610.37662

Abstract

Factorized Vector Quantization improves discrete visual token quality by using parallel codebooks on complementary feature views, enabling discrete representations to match continuous ones in multimodal understanding.

Discrete visual tokensrepresent vision in a discrete form analogous to language, offering a native interface formultimodal modelingunder a sharednext-token predictionobjective. However,quantization erroris widely assumed to cause unavoidable semantic loss, fundamentally limiting the semantic fidelity ofdiscrete visual tokensand positioningcontinuous visual representationsas an intrinsic upper bound. Using multimodal understanding as a proxy for semantic fidelity, we revisit this assumption and findquantization errorunreliable, as discrete representations with substantially larger errors can achieve better understanding. Instead, the effective bit budget of discrete indices emerges as a key determinant of downstream performance. However, existing quantizers eventually saturate as additionalcodebookscontribute progressively less semantic information. As one approach to alleviating this saturation, we introduceFactorized Vector Quantization, which quantizes complementary, fixed-dimensional views of the visual feature with symmetric parallelcodebooks. It continues to improve at larger bit budgets, ultimately matchingcontinuous visual representationsin multimodal understanding. Trained on industrial-scale multimodal data, our model achieves state-of-the-art performance among discrete-representation models, outperforming UniAR and Emu3-Chat by 22.8% and 59.2%, respectively, on a nine-metric average. To our knowledge, it is also the first discrete-representation model competitive with strong continuous-representation models such as Qwen3-VL, including on fine-grained OCR tasks. We further identify the key principles and a practical recipe enabling this competitiveness. These results and insights support the potential of discrete visual representations to underpin large-scale native multimodal foundation models. Code and models are available at https://github.com/hustvl/FVQ

View arXiv pageView PDFProject pageGitHub8Add to collection

Get this paper in your agent:

hf papers read 2610\.37662

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.37662 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.37662 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.37662 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.