V-CoLA: Vision Token Compression with Linear Attention

Hugging Face Daily Papers Papers
vision-token-compression linear-attention vision-language-models token-merging efficiency multimodal

Summary

V-CoLA 提出一种面向线性注意力混合架构的免训练视觉 token 压缩框架，通过独特性感知的重要性准则与自适应 token 合并策略，在仅保留 50% 视觉 token 时达到 99.5% 原始性能，并实现 1.86x 至 6.15x 的预填充加速。

Vision-language models (VLMs) have demonstrated impressive capabilities but suffer from substantial computational overhead, as vision tokens dominate the input sequence. This motivates vision token compression as a key direction to alleviate the burden. However, with the emergence of hybrid architectures incorporating linear attention (\eg, Qwen3.5), prior methods designed for softmax attention struggle to generalize. Our analysis reveals that both attention- and similarity-based approaches suffer notable performance degradation, underscoring the urgent need for compression methods tailored to this regime. To this end, we propose V-CoLA, an efficient training-free token compression framework specifically designed for linear attention. V-CoLA introduces a novel uniqueness-aware importance criterion for identifying critical vision tokens, coupled with an adaptive token merging strategy that performs compression. All components are optimized at the implementation level to remain compatible with the chunk-wise parallelism of linear attention, ensuring strong practical value. Extensive experiments across multiple benchmarks demonstrate the superiority of V-CoLA: it achieves 99.5\% of the original performance with only 50.0\% of vision tokens, and over 88.0\% with as few as 12.5\%, while delivering a 1.86times to 6.15times prefill speedup.
Original Article
View Cached Full Text

Cached at: 10/09/26, 06:28 AM

Paper page - V-CoLA: Vision Token Compression with Linear Attention

Source: https://huggingface.co/papers/2610.11251 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Vision-languagemodels(VLMs)havedemonstratedimpressivecapabilitiesbutsufferfromsubstantialcomputationaloverhead,asvisiontokensdominatetheinputsequence.Thismotivatesvisiontokencompressionasakeydirectiontoalleviatetheburden.However,withtheemergenceofhybridarchitecturesincorporatinglinearattention(\eg,Qwen3.5),priormethodsdesignedforsoftmaxattentionstruggletogeneralize.Ouranalysisrevealsthatbothattention-andsimilarity-basedapproachessuffernotableperformancedegradation,underscoringtheurgentneedforcompressionmethodstailoredtothisregime.Tothisend,weproposeV-CoLA,anefficienttraining-freetokencompressionframeworkspecificallydesignedforlinearattention.V-CoLAintroducesanoveluniqueness-awareimportancecriterionforidentifyingcriticalvisiontokens,coupledwithanadaptivetokenmergingstrategythatperformscompression.Allcomponentsareoptimizedattheimplementationleveltoremaincompatiblewiththechunk-wiseparallelismoflinearattention,ensuringstrongpracticalvalue.ExtensiveexperimentsacrossmultiplebenchmarksdemonstratethesuperiorityofV-CoLA:itachieves99.5\%oftheoriginalperformancewithonly50.0\%ofvisiontokens,andover88.0\%withasfewas12.5\%,whiledeliveringa1.86timesto6.15timesprefillspeedup.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.11251

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.11251 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.11251 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.11251 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Vision Token Pruning via Query-Vision Interaction Decomposition

arXiv cs.LG

论文提出 QViD，一种免训练的查询感知视觉token剪枝方法，通过发现查询-视觉交互矩阵的低秩结构并分解出主导潜在交互模式，在图像与视频理解基准上以相同token预算持续提升性能，尤其在高压缩率下增益显著。

LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence

Hugging Face Daily Papers

LLaVA-OneVision-2 introduces codec-stream tokenization and windowed attention for efficient video understanding, achieving state-of-the-art performance across multiple multimodal benchmarks including video, spatial, and tracking tasks.