通过查询与视觉交互分解进行视觉词元剪枝

# Vision Token Pruning via Query–Vision Interaction Decomposition
Source: [https://arxiv.org/html/2610.33215](https://arxiv.org/html/2610.33215)
Harshithanjani Athi††thanks:Email addresses:ha26227\.my@utexas\.edu,sravan\.ankireddy@utexas\.edu,jianzhong\.z@samsung\.com,hyeji\.kim@utexas\.edu\.Jianzhong Charlie ZhangAffiliation:The University of Texas at Austin Samsung Research AmericaHyeji Kim

###### Abstract

Large vision\-language models \(VLMs\), which process both visual inputs and text queries, incur high inference costs due to the large number of visual tokens, making visual token pruning a natural approach for improving efficiency\. Recent work has leveraged query information to guide token selection, often through heuristics derived from attention patterns or token dynamics\. However, these approaches do not explicitly exploit the underlying structure of query–vision interactions\. We address this gap by observing that the query–vision interaction matrix, naturally induced by the query and key projections already present in transformer attention, is effectively low\-rank, with a small number of dominant latent interaction modes capturing query\-relevant semantics\. Motivated by this observation, we propose Query\-Vision Decomposition \(QViD\), a novel training\-free query\-aware visual token pruning method that exploits this structure\. Extensive experiments on both image and video understanding benchmarks show that QViD consistently improves performance under matched token budgets, with the largest gains appearing under aggressive compression regimes\.

## 1Introduction

Vision\-language models \(VLMs\) have demonstrated strong performance across a wide range of multimodal tasks, including visual question answering, optical character recognition, diagram understanding, and video comprehension\[[13](https://arxiv.org/html/2610.33215#bib.bib1),[19](https://arxiv.org/html/2610.33215#bib.bib3),[9](https://arxiv.org/html/2610.33215#bib.bib4)\]\. In a typical VLM, a vision encoder produces vision tokens from an image, and the language model jointly processes these vision tokens together with the text query to generate a final response\. However, this design also creates a major efficiency bottleneck: an image can produce hundreds to thousands of vision tokens, making multimodal inference expensive in both latency and memory\. Consequently, reducing the cost of processing vision tokens has become an important focus of recent work on efficient VLM inference\.

A direct remedy is to reduce the number of visual tokens passed to the language model\. Several*training\-free*token compression methods have been proposed for this purpose, and many of them use*attention\-guided*signals\. For example, FastV\[[4](https://arxiv.org/html/2610.33215#bib.bib6)\]and SparseVLM\[[21](https://arxiv.org/html/2610.33215#bib.bib7)\]use early\-layer attention patterns to score visual tokens, then prune tokens with low importance scores\. While intuitive and often effective, these methods can be biased: vision tokens that are closer to the text tokens in the sequence may receive higher attention scores even when they are not the most relevant to the query\. As a result, attention\-based scores do not always provide a reliable measure of true query relevance\.

The second paradigm is*attention\-free*token pruning\. A strong representative of this line of work is V2Drop\[[3](https://arxiv.org/html/2610.33215#bib.bib11)\], which shows that an attention\-free pruning signal can outperform earlier attention\-based methods\. Instead of using attention scores, it ranks vision tokens by how much their hidden states change across consecutive LLM layers, and drops tokens that change the least\. This avoids the positional bias of attention\-based scores\. Since visual and text tokens are processed jointly, the variation signal can still reflect some influence from the query, but only indirectly\. It captures generic token activity rather than explicitly measuring query–vision relevance\.

This reveals a gap between the two paradigms: attention\-guided methods are query\-aware but inherit attention\-score biases, while variation\-based methods avoid attention scores but do not explicitly model the query\. Our goal is to combine the strengths of both, enabling a query\-conditioned pruning signal that does not rely on attention weights\. This is particularly important because, in visual question answering and related tasks, the query determines what is relevant in the image\. For example, in an image of an animal wearing a hat, as in Figure[1](https://arxiv.org/html/2610.33215#S1.F1), tokens depicting the animal are highly relevant for identifying it, but less relevant for questions about the background or surrounding objects\. Ideally, a token selection method should therefore explicitly condition on the query, and more precisely on its underlying semantics\. This raises the question of how such semantics can be represented in a structured form\. This motivates our central question:

*Can a few dominant latent modes of query–vision interaction capture query\-relevant visual evidence without relying on attention weights?*

#### Our Approach\.

To answer this question, we proposeQuery\-VisionDecomposition \(QViD\), a training\-free method for visual\-token selection based on an explicit query–vision interaction matrix\. Our key observation is that query–vision interactions exhibit a strong low\-rank structure: a small number of dominant interaction modes capture the visual patterns most relevant to the query\.

Motivated by this observation, QViD constructs a query–vision interaction matrix at an early transformer layer and scores visual tokens according to how strongly they align with these dominant modes\. Unlike attention\-guided pruning, QViD does not rely on attention weights; unlike variation\-based pruning, it explicitly conditions on the input query\.

Figure[1](https://arxiv.org/html/2610.33215#S1.F1)qualitatively compares QViD with representative attention\-guided \(SparseVLM\) and attention\-free \(V2Drop\) approaches\. While these prior methods can highlight regions not directly relevant to the query, QViD focuses on image regions that provide direct evidence for the answer\.

Our main contributions are summarized as follows:

- •We show that query–vision interactions in VLMs, readily obtainable from standard transformer query–key projections, exhibit a strong low\-rank structure, where task\-relevant semantics concentrate into a small number of dominant interaction modes\.
- •We propose QViD, an SVD\-based query\-aware visual\-token pruning method that estimates token importance from the dominant singular structure of the query–vision interaction matrix\. Our analysis shows that isolating these dominant interaction modes is crucial for effective visual\-token selection\.
- •Experiments across multiple VLMs and image and video understanding benchmarks show that QViD consistently outperforms strong recent pruning methods across a wide range of token budgets, with particularly strong gains under aggressive compression\.

![Refer to caption](https://arxiv.org/html/2610.33215v1/comparison.png)Figure 1:Comparison of Visual\-Token Scoring\.Compared with prior methods such as V2Drop and SparseVLM, QViD concentrates high scores on query\-relevant image regions, leading to more effective visual\-token pruning\.

## 2Related Work

Modern vision\-language models \(VLMs\) process visual and textual tokens jointly, and recent advances toward higher\-resolution images and longer videos have significantly increased the number of visual tokens\[[14](https://arxiv.org/html/2610.33215#bib.bib2),[13](https://arxiv.org/html/2610.33215#bib.bib1),[19](https://arxiv.org/html/2610.33215#bib.bib3),[9](https://arxiv.org/html/2610.33215#bib.bib4),[12](https://arxiv.org/html/2610.33215#bib.bib5)\]\. This has led to a growing line of work on visual\-token compression for efficient VLM inference\. In this section, we review existing approaches to visual\-token compression and position our method within this landscape\. Our focus is ontraining\-free visual\-token pruning, which is complementary to improvements in the vision encoder, projector, or LLM backbone\.

#### Visual\-Token Compression\.

A common approach to reducing visual\-token cost is to prune or merge tokens based on importance scores\. Attention\-guided methods such as FastV\[[4](https://arxiv.org/html/2610.33215#bib.bib6)\]and SparseVLM\[[21](https://arxiv.org/html/2610.33215#bib.bib7)\]use attention patterns inside the LLM to identify important visual tokens, while PDrop\[[20](https://arxiv.org/html/2610.33215#bib.bib8)\]progressively reduces the token count across layers\. Token merging methods such as ToMe\[[2](https://arxiv.org/html/2610.33215#bib.bib12)\]and LLaVA\-PruMerge\[[17](https://arxiv.org/html/2610.33215#bib.bib10)\]reduce sequence length by combining similar visual tokens instead of discarding them\. While effective, attention\-based scoring requires access to attention matrices, making it less compatible with memory\-efficient implementations such as FlashAttention\[[5](https://arxiv.org/html/2610.33215#bib.bib13)\], and can suffer from positional bias in autoregressive LLMs\[[3](https://arxiv.org/html/2610.33215#bib.bib11)\]\.

#### Task\-Agnostic Versus Query\-Aware Pruning\.

V2Drop\[[3](https://arxiv.org/html/2610.33215#bib.bib11)\]avoids explicit attention scores by measuring how much each visual token changes across consecutive LLM layers\. Tokens with small hidden\-state variation are treated as “lazy tokens” and pruned\. This makes V2Drop naturally compatible with FlashAttention and avoids attention\-position bias\. However, the variation signal is largely task\-agnostic: it captures generic token dynamics rather than explicitly modeling query–vision relevance\. In contrast, QViD constructs a query–vision interaction matrix and ranks tokens based on their participation in its dominant low\-rank structure, yielding a training\-free pruning rule that is explicitly conditioned on the input query\.

![Refer to caption](https://arxiv.org/html/2610.33215v1/QVisSVD_Pipeline.png)Figure 2:QViD Pipeline\.QViD performs query\-aware visual\-token pruning by explicitly modeling query–vision interactions\. At pruning layerℓ\\ell, it forms the interaction matrixM=Qtxt​Kvis⊤M=Q\_\{\\text\{txt\}\}K\_\{\\text\{vis\}\}^\{\\top\}, extracts its dominant low\-rank modes with SVD, and scores each visual token by its participation in these modes\. The top\-KKtokens are retained for subsequent reasoning, preserving query\-relevant visual evidence while reducing sequence length\. Integer values in the illustrative matrices are used only for ease of presentation\.

## 3Method

### 3\.1Problem Setup

We consider a vision\-language model \(VLM\) that processes an input image and a text query jointly\. The image is first encoded into a sequence of visual tokens, which are then combined with text tokens and processed by a transformer\-based language model\. At transformer layerℓ\\ell, let

Hvis\(ℓ\)∈ℝNv×d,Htxt\(ℓ\)∈ℝNq×dH^\{\(\\ell\)\}\_\{\\mathrm\{vis\}\}\\in\\mathbb\{R\}^\{N\_\{v\}\\times d\},\\qquad H^\{\(\\ell\)\}\_\{\\mathrm\{txt\}\}\\in\\mathbb\{R\}^\{N\_\{q\}\\times d\}denote the hidden states corresponding to the visual tokens and user\-query tokens, respectively, whereNvN\_\{v\}andNqN\_\{q\}are the numbers of visual and query tokens, andddis the hidden dimension\.

Our goal is to prunevisual tokensin a query\-aware and training\-free manner by retaining those most relevant to the input query under a target token budget\. Our method can be applied at any layerℓ\\ell, and we study this design choice empirically\.

### 3\.2Query–Vision Interaction Matrix

Our method is based on a query–vision interaction matrix that measures the compatibility between query tokens and visual tokens\. We construct this matrix using the frozen query and key projections already present in the transformer\. For a given attention head at layerℓ\\ell, we compute

Qtxt=Htxt\(ℓ\)​WQ,Kvis=Hvis\(ℓ\)​WK,Q\_\{\\mathrm\{txt\}\}=H^\{\(\\ell\)\}\_\{\\mathrm\{txt\}\}W\_\{Q\},\\qquad K\_\{\\mathrm\{vis\}\}=H^\{\(\\ell\)\}\_\{\\mathrm\{vis\}\}W\_\{K\},\(1\)whereWQ,WK∈ℝd×dhW\_\{Q\},W\_\{K\}\\in\\mathbb\{R\}^\{d\\times d\_\{h\}\}are the query and key projection matrices, anddhd\_\{h\}is the attention\-head dimension\. We then define the query–vision interaction matrix

M=Qtxt​Kvis⊤∈ℝNq×Nv\.M=Q\_\{\\mathrm\{txt\}\}K\_\{\\mathrm\{vis\}\}^\{\\top\}\\in\\mathbb\{R\}^\{N\_\{q\}\\times N\_\{v\}\}\.\(2\)
The entryMi​jM\_\{ij\}measures the compatibility between query tokeniiand visual tokenjjin the model’s learned representation space\. Equivalently, thejj\-th column ofMMsummarizes how visual tokenjjinteracts with the full query\.

Advantages over Attention Scores\.This construction is closely related to the pre\-softmax attention computation, but differs in two important ways\. First, we compute the interaction before positional encoding\. In autoregressive LLMs, positional encoding can bias attention toward visual tokens that are closer to the query tokens in the sequence, regardless of their semantic relevance\. Since several prior pruning methods, such as SparseVLM and FastV, rely on attention scores for token selection, this positional bias can directly affect pruning decisions\. By removing positional effects,MMcaptures a purely content\-based query–vision interaction signal\.

Second, we avoid the normalization effects of softmax, where token relevance is mixed with competition among tokens\. As a result,MMpreserves the raw query–vision interaction structure without relying on explicit attention maps\. This is also advantageous in memory\-efficient attention implementations, where attention maps may be unavailable or costly to extract\.

From Interaction Matrix to Token Scores\.A natural way to derive visual\-token importance fromMMwould be to aggregate each column independently, for example by taking its norm\. This measures how strongly a visual token is compatible with all query tokens\. However, as we show later in the ablation study, naive column\-wise aggregation performs poorly as a pruning criterion\.

This is because not every query token, or word, is equally informative or relevant for the downstream task\. For example, in the question “What animal is wearing the hat?”, we would like to emphasize compatibility with meaningful words such asanimalandhat, rather than less informative words such asisorthe\.

More broadly, our goal is to capture visual evidence relevant to theoverall query semantics, rather than simply measuring isolated token\-level compatibility\. As such relevance may not be recoverable from pairwise query–vision interactions alone, we instead analyze the global structure of the interaction matrixMMitself\.

### 3\.3Low\-Rank Structure of Query–Vision Interactions and Dominant Interaction Modes

Interestingly, we observe thatMM, a quantity closely related to attention computation but not typically analyzed directly in prior pruning methods, exhibits a strong low\-rank structure across a wide range of queries and images\. Moreover, the dominant singular modes are not merely numerically significant; they consistently capture task\-relevant query–vision interaction patterns\.

![Refer to caption](https://arxiv.org/html/2610.33215v1/query_score_components_no_spectrum_main.png)

Figure 3:Qualitative Observations\.\(a\)The query–vision interaction matrix has a rapidly decaying singular\-value spectrum\.\(b\)Dominant SVD components emphasize query\-relevant tokens, while residual components are less aligned with the task\.#### Observation 1: Query–Vision Interactions Are Approximately Low\-Rank\.

Across images and queries, the singular spectrum ofMMis strongly concentrated in a small number of directions\. For each attention head, we compute the singular value decomposition \(SVD\)M=U​Σ​V⊤M=U\\Sigma V^\{\\top\}\.

Figure[3](https://arxiv.org/html/2610.33215#S3.F3)\(a\) shows the singular value spectrum ofMMfor LLaVA\-1\.5\-7B on the POPE dataset\[[11](https://arxiv.org/html/2610.33215#bib.bib19)\]\. We observe that the spectrum decays rapidly, with a small number of singular values explaining most of the query–vision interaction structure\. As further shown in Appendix[J\.1](https://arxiv.org/html/2610.33215#A10.SS1), this behavior appears consistently across different datasets, images, and queries, suggesting that the useful interaction between query and visual tokens often lies in a low\-dimensional subspace\.

#### Observation 2: Dominant Components Capture Task\-Relevant Structure\.

The low\-rank structure ofMMis not only a numerical property; it is also semantically meaningful\. In particular, we observe that task\-relevant semantics tend to concentrate into a small number of dominant interaction modes\.

For interpretability, we separate the leading singular directions, which capture the dominant interaction modes, from the remaining residual directions\. We then measure how strongly each query tokenjjparticipates in the dominant and residual subspaces through

sj\(dom\)=∑k=14Uj​k2,sj\(res\)=∑k\>4Uj​k2\.s\_\{j\}^\{\(\\mathrm\{dom\}\)\}=\\sum\_\{k=1\}^\{4\}U\_\{jk\}^\{2\},\\qquad s\_\{j\}^\{\(\\mathrm\{res\}\)\}=\\sum\_\{k\>4\}U\_\{jk\}^\{2\}\.\(3\)Figure[3](https://arxiv.org/html/2610.33215#S3.F3)\(b\) visualizes these two scores for the dog query–image example; additional qualitative examples are provided in Appendix[J\.2](https://arxiv.org/html/2610.33215#A10.SS2)\. We observe that semantically informative query tokens, such as*animal*and*hat*, tend to have high dominant contribution scores and low residual contribution scores\. In contrast, less informative words, such as*is*and*the*, exhibit the opposite behavior\.

Together, these observations suggest the following:

*Visual\-token importance should be estimated from the dominant interaction subspace ofMM, rather than from individual matrix entries or raw column magnitudes\.*

### 3\.4QViD Token Scoring and Pruning

We now describe how QViD converts the query–vision interaction matrix into visual\-token scores\.

Visual\-Token Scores\.For each attention head at layerℓ\\ell, we compute the singular value decomposition

M=U​Σ​V⊤\.M=U\\Sigma V^\{\\top\}\.\(4\)The right singular vectors inVVdescribe the visual\-token side of the query–vision interaction modes\. We score each visual tokenjjby its participation in the toprrright singular directions:

sj=∑k=1rVj​k2\.s\_\{j\}=\\sum\_\{k=1\}^\{r\}V\_\{jk\}^\{2\}\.\(5\)
Intuitively,Vj​k2V\_\{jk\}^\{2\}measures how strongly visual tokenjjparticipates in thekk\-th dominant interaction mode\. Therefore, a largersjs\_\{j\}indicates that the visual tokenjjcontributes more strongly to the dominant interaction subspace\.

For the multi\-head setting, we apply the same scoring rule independently to each attention head and average the resulting scores\. LetM\(h\)M^\{\(h\)\}denote the query–vision interaction matrix for headhh, and letV\(h\)V^\{\(h\)\}be its right singular vector matrix\. The final score for visual tokenjjis

sj=1H​∑h=1H∑k=1r\(Vj​k\(h\)\)2,s\_\{j\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}\\sum\_\{k=1\}^\{r\}\\left\(V^\{\(h\)\}\_\{jk\}\\right\)^\{2\},\(6\)whereHHis the number of attention heads\.

For short queries, the number of available nonzero singular directions may be smaller than the chosenrr\. In this case, we usereff=min⁡\(r,Nq,Nv\)r\_\{\\mathrm\{eff\}\}=\\min\(r,N\_\{q\},N\_\{v\}\), whereNqN\_\{q\}andNvN\_\{v\}denote the numbers of query and visual tokens, respectively, and replacerrbyreffr\_\{\\mathrm\{eff\}\}in the scoring rule\.

Visual\-Token Pruning\.The top\-KKvisual tokens with the largest scoressjs\_\{j\}are passed to the remaining LLM layers, withKKdetermined by the token budget\. Importantly, QViD is entirely training\-free, requires no architectural modification, and operates solely using the frozen projections already present in the model\.

## 4Experiments

### 4\.1Experimental Setup

We conduct comprehensive experiments across image understanding, video understanding, multimodal reasoning, hallucination\-sensitive evaluation, and captioning tasks under a wide range of token budgets\. In the main paper, we focus on the core image and video understanding results, while Appendices[G](https://arxiv.org/html/2610.33215#A7),[I](https://arxiv.org/html/2610.33215#A9)further demonstrate the robustness and generality of QViD across diverse settings\. Specifically, we consider the following models, benchmarks, and baselines\.

Models\.We evaluate QViD on four representative VLMs\. For image understanding, we use LLaVA\-1\.5\-7B\[[13](https://arxiv.org/html/2610.33215#bib.bib1)\]and Qwen2\-VL\-7B\[[19](https://arxiv.org/html/2610.33215#bib.bib3)\]\. For video understanding, we use Video\-LLaVA\-7B\[[12](https://arxiv.org/html/2610.33215#bib.bib5)\]and LLaVA\-OneVision\-7B\[[9](https://arxiv.org/html/2610.33215#bib.bib4)\]\.

Benchmarks\.For image understanding, we consider standard and diverse VLM benchmarks, including GQA\[[8](https://arxiv.org/html/2610.33215#bib.bib16)\], ScienceQA\[[16](https://arxiv.org/html/2610.33215#bib.bib14)\], TextVQA\[[18](https://arxiv.org/html/2610.33215#bib.bib15)\], POPE\[[11](https://arxiv.org/html/2610.33215#bib.bib19)\], MME\[[6](https://arxiv.org/html/2610.33215#bib.bib17)\], and MMBench\[[15](https://arxiv.org/html/2610.33215#bib.bib18)\]\. For video understanding, we evaluate on VideoMME\[[7](https://arxiv.org/html/2610.33215#bib.bib20)\]and MVBench\[[10](https://arxiv.org/html/2610.33215#bib.bib21)\]\.

Baselines\.V2Drop\[[3](https://arxiv.org/html/2610.33215#bib.bib11)\]serves as the primary baseline throughout our experiments, as it is the most recent and strongest existing visual\-token pruning method\. For LLaVA\-1\.5\-7B, we additionally compare against ToMe\[[2](https://arxiv.org/html/2610.33215#bib.bib12)\], FastV\[[4](https://arxiv.org/html/2610.33215#bib.bib6)\], HiRED\[[1](https://arxiv.org/html/2610.33215#bib.bib9)\], LLaVA\-PruMerge\[[17](https://arxiv.org/html/2610.33215#bib.bib10)\], SparseVLM\[[21](https://arxiv.org/html/2610.33215#bib.bib7)\], and PDrop\[[20](https://arxiv.org/html/2610.33215#bib.bib8)\]\. For these methods, we use the numbers reported in the V2Drop paper\. For V2Drop itself, we reproduce the results using the official implementation and confirm that they match the reported numbers\.

Implementation Details\.All experiments are conducted using Python 3\.11, PyTorch 2\.5\.1 \(CUDA 12\.1\)\. For LLaVA\-1\.5\-7B, Qwen2\-VL\-7B, Video\-LLaVA\-7B, and LLaVA\-OV\-7B, we follow the standard evaluation settings of the corresponding model and benchmark\. QViD is implemented as a separate scoring module at the pruning layer and does not modify the model architecture or attention kernel\. After token selection, all subsequent transformer layers process the retained sequence normally\. We use layerℓ=3\\ell=3,r=16r=16SVD components, and omit Rotary Positional Embedding \(RoPE\) when constructing the query–vision interaction matrix\.

### 4\.2QViD: Main Results

Across all evaluated VLMs and benchmarks, QViD consistently achieves stronger performance under matched token budgets\. The gains are particularly pronounced under aggressive compression regimes, where retaining only a small fraction of visual tokens makes query\-aware selection especially important\.

Compared with existing pruning methods, QViD more effectively preserves task\-relevant visual information, leading to improved downstream accuracy on both image and video understanding tasks\. This behavior is also reflectedqualitativelyin the retained visual tokens and score maps, as illustrated in Figure[1](https://arxiv.org/html/2610.33215#S1.F1)and additional visualizations provided in Appendix[K\.1](https://arxiv.org/html/2610.33215#A11.SS1)\.

#### LLaVA\-1\.5\-7B Image Understanding\.

Table[1](https://arxiv.org/html/2610.33215#S4.T1)compares QViD with prior visual\-token compression methods under matched token budgets\. We report the mean normalized performance relative to the full\-token baseline\. QViD achieves the best average performance at all three token budgets\. The gains are largest at 64 tokens, where QViD improves over V2Drop by3\.5%\\%in normalized average, with particularly strong gains on TextVQA \(3\.7%\\%\), POPE \(5\.7%\\%\), and MME \(83 points\)\. This suggests that explicit query conditioning is especially important under aggressive compression, where query\-agnostic signals may retain irrelevant visual tokens\.

Table 1:Image Understanding Results on LLaVA\-1\.5\-7B\.QViD achieves the best average performance at all retention ratios, with the largest gain under aggressive compression at 64 tokens\.Methods are sorted by average performance within each retention setting\.MethodsGQASQATextVQAPOPEMMEMMBAverageUpper Bound, 576 TokensLLaVA\-1\.5\-7B \(Liu et al\. 2024\)61\.969\.558\.285\.9186264\.6100\.0%Average Retain 192 Tokens \(↓\\downarrow66\.7%\)QViD \(ours\)59\.369\.057\.887\.4179363\.298\.1%V2Drop \[CVPR’26\]58\.569\.355\.685\.1182663\.797\.6%PDrop \[CVPR’25\]57\.168\.856\.182\.3176663\.296\.0%SparseVLM \[ICML’25\]57\.669\.156\.183\.6172162\.595\.9%HiRED \[AAAI’25\]58\.768\.447\.482\.8173762\.893\.6%LLaVA\-PruMerge \[ICCV’25\]54\.367\.954\.371\.3163259\.690\.3%ToMe \[ICLR’23\]54\.365\.252\.172\.4156360\.588\.8%FastV \[ECCV’24\]52\.767\.352\.564\.8161261\.288\.2%Average Retain 128 Tokens \(↓\\downarrow77\.8%\)QViD \(ours\)57\.768\.757\.185\.5172661\.996\.4%V2Drop \[CVPR’26\]56\.368\.853\.880\.9171261\.894\.0%PDrop \[CVPR’25\]56\.068\.354\.882\.3164461\.193\.6%SparseVLM \[ICML’25\]56\.067\.154\.980\.5169660\.093\.2%HiRED \[AAAI’25\]57\.268\.146\.179\.8171061\.591\.6%LLaVA\-PruMerge \[ICCV’25\]53\.367\.154\.367\.2155458\.187\.9%FastV \[ECCV’24\]49\.660\.250\.659\.6149056\.181\.7%ToMe \[ICLR’23\]52\.459\.649\.162\.8134353\.380\.4%Average Retain 64 Tokens \(↓\\downarrow88\.9%\)QViD \(ours\)52\.268\.355\.580\.8155356\.390\.4%V2Drop \[CVPR’26\]50\.568\.951\.875\.1147055\.286\.9%SparseVLM \[ICML’25\]52\.762\.251\.875\.1150556\.286\.5%LLaVA\-PruMerge \[ICCV’25\]51\.968\.154\.065\.3154955\.286\.5%PDrop \[CVPR’25\]41\.968\.645\.955\.9109233\.370\.1%FastV \[ECCV’24\]46\.151\.147\.848\.0125648\.071\.3%ToMe \[ICLR’23\]48\.650\.045\.352\.5113843\.769\.7%
#### Qwen2\-VL\-7B Image Understanding\.

Table[2](https://arxiv.org/html/2610.33215#S4.T2)shows that the gains of QViD generalize to Qwen2\-VL\-7B\. Across all retention ratios, QViD improves the average normalized performance over V2Drop\. The improvement is particularly large at 11\.1% retention, where QViD improves the average from 62\.7% to 81\.7%, indicating that query\-aware scoring is especially important when only a small fraction of visual tokens can be retained\.

Table 2:Image Understanding Results on Qwen2\-VL\-7B\.QViD improves average performance at all retention ratios, with the largest gains at 11\.1% retention\. Avg\. is mean normalized performance relative to the full\-token baseline\.
#### Video Understanding\.

Table[3](https://arxiv.org/html/2610.33215#S4.T3)\(a\) shows that QViD also extends effectively to video understanding on LLaVA\-OV\-7B\. QViD consistently outperforms V2Drop under matched token budgets, with a particularly large gain at 10% retention, where it preserves 91\.2% of full\-token performance compared to 69\.4% for V2Drop\. Additional results on Video\-LLaVA\-7B are provided in Appendix[H](https://arxiv.org/html/2610.33215#A8)\.

#### Additional Results\.

Beyond image and video understanding, we also evaluate whether the selected tokens preserve broader multimodal capabilities\. Appendix[G](https://arxiv.org/html/2610.33215#A7)reports additional results on multimodal reasoning benchmarks, while Appendix[I](https://arxiv.org/html/2610.33215#A9)evaluates hallucination\-sensitive performance and captioning quality\. These results show that QViD preserves strong performance beyond standard perception tasks, suggesting that its selected tokens retain the visual information needed for downstream reasoning and generation\.

Table 3:Video Understanding and Compute Comparison\.\(a\) Results on LLaVA\-OV\-7B show that QViD consistently outperforms V2Drop under matched token budgets, with the largest gains under aggressive compression\. Avg\. denotes the mean normalized performance relative to the full\-token baseline\. \(b\) Total TFLOPs on LLaVA\-1\.5\-7B include pruning overhead; reduction is computed relative to the full\-token baseline\.\(a\) Video Understanding on LLaVA\-OV\-7B

\(b\) Compute Comparison

Computational Complexity\.Table[3](https://arxiv.org/html/2610.33215#S4.T3)\(b\) reports total TFLOPs on LLaVA\-1\.5\-7B, including pruning overhead: the full\-token baseline costs8\.868\.86TFLOPs, while QViD costs2\.532\.53,3\.423\.42, and4\.324\.32TFLOPs at6464,128128, and192192retained tokens, reducing compute by71\.5%71\.5\\%,61\.4%61\.4\\%, and51\.3%51\.3\\%, respectively\. Compared with V2Drop, QViD spends modest extra compute on query\-aware scoring but retains more task\-relevant visual tokens; detailed latency, throughput, FLOPs accounting, and accuracy–compute tradeoffs are provided in Appendix[F](https://arxiv.org/html/2610.33215#A6)\.

### 4\.3Ablation Studies

We conduct ablation studies to examine the main design choices in QViD\. In particular, we study whether the SVD\-based decomposition is necessary, how many dominant SVD components are sufficient for token scoring, and how the choice of pruning layer affects performance\.

#### Dominant Low\-Rank Structure Is Important\.

To isolate the effect of SVD\-based decomposition and low\-rank interaction modeling, we compare QViD with a simple column\-norm baseline that uses the same query–vision interaction matrixM=Qtxt​Kvis⊤M=Q\_\{\\mathrm\{txt\}\}K\_\{\\mathrm\{vis\}\}^\{\\top\}, but scores each visual token by raw interaction magnitude:sjcol=∥M:,j∥22s\_\{j\}^\{\\mathrm\{col\}\}=\\\|M\_\{:,j\}\\\|\_\{2\}^\{2\}\. This baseline directly measures how strongly each visual token interacts with the query, but does not separate dominant query\-conditioned structure from residual interaction energy\.

Table[4](https://arxiv.org/html/2610.33215#S4.T4)\(a\) shows that simply using the query–vision interaction matrix is not sufficient\. Column\-norm scoring performs much worse than QViD across all token budgets, especially under aggressive compression\. The SVD step is crucial as it identifies the low\-rank interaction modes that provide a more reliable signal for visual\-token selection\.

#### Effect of the Number of SVD Components\.

We study how many dominant SVD components are needed for reliable token scoring\. Table[4](https://arxiv.org/html/2610.33215#S4.T4)\(b\) reports the component sweep at128128retained tokens\. We denote the number of SVD components used for scoring bynsvdn\_\{\\mathrm\{svd\}\}\. A small number of dominant components is sufficient to recover most of the benefit of QViD: performance improves sharply fromnsvd=1,2n\_\{\\mathrm\{svd\}\}=1,2tonsvd=4,8n\_\{\\mathrm\{svd\}\}=4,8, indicating that the most relevant query–vision structure is concentrated in a few dominant interaction modes\. Usingnsvd=16n\_\{\\mathrm\{svd\}\}=16provides additional robustness and achieves the best average performance at this budget\. The full sweep over all token budgets is provided in Appendix[C](https://arxiv.org/html/2610.33215#A3)\.

Table 4:Ablations on QViD Scoring with LLaVA\-1\.5\-7B\.\(a\) SVD scoring outperforms column\-norm scoring from the same query–vision interaction matrix\. \(b\) Using only a small number of dominant SVD components recovers most of the benefit of QViD, indicating that token relevance is concentrated in a few interaction modes\.\(a\) Effect of SVD\-Based Scoring

\(b\) Number of SVD Components at 128 Tokens

#### Choice of Pruning Layer and Other Design Choices\.

Unless otherwise stated, we prune at LLM layerℓ=3\\ell=3, which provides a strong accuracy–efficiency tradeoff in our layer\-selection study\. Detailed layer\-selection results for both LLaVA\-1\.5\-7B and Qwen2\-VL\-7B are provided in Appendix[B](https://arxiv.org/html/2610.33215#A2)\. In addition, Appendix[D](https://arxiv.org/html/2610.33215#A4)compares QViD with alternative scoring rules derived from the same query–vision interaction matrix, including column\-norm scoring, singular\-value\-weighted scoring, and per\-head max aggregation\.

## 5Conclusion

We presented QViD, a training\-free visual\-token pruning method that explicitly conditions on the query without relying on attention weights\. QViD constructs the query–vision interaction matrix induced by standard transformer query–key projections and scores visual tokens using its dominant low\-rank subspace, retaining tokens that participate in the main query\-relevant interaction modes\. Across image and video benchmarks and multiple VLMs, QViD improves performance under matched token budgets, with the largest gains under aggressive compression\. These results show that a few dominant SVD components are sufficient to capture much of the task\-relevant query–vision structure, establishing low\-rank query–vision decomposition as a simple, interpretable, and training\-free basis for efficient VLM inference\.

## Acknowledgements

This work was partially supported by Samsung Research America through the 6G@UT center within the Wireless Networking and Communications Group \(WNCG\) at the University of Texas at Austin, ARO Award W911NF2310062, ONR Award N000142412542, and NSF CAREER Award 2443857\.

## References

- \[1\]K\. H\. I\. Arif, J\. Yoon, D\. S\. Nikolopoulos, H\. Vandierendonck, D\. John, and B\. Ji\(2025\)HiRED: attention\-guided token dropping for efficient inference of high\-resolution vision\-language models in resource\-constrained environments\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.
- \[2\]D\. Bolya, C\. Fu, X\. Dai, P\. Zhang, C\. Feichtenhofer, and J\. Hoffman\(2023\)Token merging: your ViT but faster\.InProceedings of the International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.
- \[3\]J\. Chen, X\. Liu, Z\. Wen, Y\. Wang, S\. Huang, and H\. Chen\(2026\)Variation\-aware vision token dropping for faster large vision\-language models\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,Cited by:[§1](https://arxiv.org/html/2610.33215#S1.p3.1),[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.
- \[4\]L\. Chen, H\. Zhao, T\. Liu, S\. Bai, J\. Lin, C\. Zhou, and B\. Chang\(2024\)An image is worth 1/2 tokens after layer 2: plug\-and\-play inference acceleration for large vision\-language models\.InProceedings of the European Conference on Computer Vision,Cited by:[§1](https://arxiv.org/html/2610.33215#S1.p2.1),[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.
- \[5\]T\. Dao\(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InProceedings of the International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, Z\. Qiu, W\. Lin, J\. Yang, X\. Zheng,et al\.\(2023\)MME: a comprehensive evaluation benchmark for multimodal large language models\.arXiv preprint arXiv:2306\.13394\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[7\]C\. Fu, Y\. Dai, Y\. Luo, L\. Li, S\. Ren, R\. Zhang, Z\. Wang, C\. Zhou, Y\. Shen, M\. Zhang,et al\.\(2024\)Video\-MME: the first\-ever comprehensive evaluation benchmark of multi\-modal LLMs in video analysis\.arXiv preprint arXiv:2405\.21075\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[8\]D\. A\. Hudson and C\. D\. Manning\(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 6700–6709\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[9\]B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, Y\. Li, Z\. Liu, and C\. Li\(2024\)LLaVA\-OneVision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[§1](https://arxiv.org/html/2610.33215#S1.p1.1),[§2](https://arxiv.org/html/2610.33215#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p2.1)\.
- \[10\]K\. Li, Y\. Wang, Y\. He, Y\. Li, Y\. Wang, Y\. Liu, Z\. Wang, J\. Xu, G\. Chen, P\. Luo,et al\.\(2024\)MVBench: a comprehensive multi\-modal video understanding benchmark\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22195–22206\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[11\]Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen\(2023\)Evaluating object hallucination in large vision\-language models\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,pp\. 292–305\.Cited by:[§3\.3](https://arxiv.org/html/2610.33215#S3.SS3.SSS0.Px1.p2.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[12\]B\. Lin, B\. Zhu, Y\. Ye, M\. Ning, P\. Jin, and L\. Yuan\(2023\)Video\-LLaVA: learning united visual representation by alignment before projection\.arXiv preprint arXiv:2311\.10122\.Cited by:[§2](https://arxiv.org/html/2610.33215#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p2.1)\.
- \[13\]H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee\(2024\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26286–26296\.Cited by:[§1](https://arxiv.org/html/2610.33215#S1.p1.1),[§2](https://arxiv.org/html/2610.33215#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p2.1)\.
- \[14\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 34892–34916\.Cited by:[§2](https://arxiv.org/html/2610.33215#S2.p1.1)\.
- \[15\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu, K\. Chen, and D\. Lin\(2024\)MMBench: is your multi\-modal model an all\-around player?\.InProceedings of the European Conference on Computer Vision,pp\. 216–233\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[16\]P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan\(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems,pp\. 2507–2521\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[17\]Y\. Shang, M\. Cai, B\. Xu, Y\. J\. Lee, and Y\. Yan\(2025\)LLaVA\-PruMerge: adaptive token reduction for efficient large multimodal models\.InProceedings of the IEEE/CVF International Conference on Computer Vision,Cited by:[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.
- \[18\]A\. Singh, V\. Natarajan, M\. Shah, Y\. Jiang, X\. Chen, D\. Batra, D\. Parikh, and M\. Rohrbach\(2019\)Towards VQA models that can read\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 8317–8326\.Cited by:[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p3.1)\.
- \[19\]P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-VL: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§1](https://arxiv.org/html/2610.33215#S1.p1.1),[§2](https://arxiv.org/html/2610.33215#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p2.1)\.
- \[20\]L\. Xing, Q\. Huang, X\. Dong, J\. Lu, P\. Zhang, Y\. Zang, Y\. Cao, C\. He, J\. Wang, F\. Wu,et al\.\(2025\)PyramidDrop: accelerating your large vision\-language models via pyramid visual redundancy reduction\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.
- \[21\]Y\. Zhang, C\. Fan, J\. Ma, W\. Zheng, T\. Huang, K\. Cheng, D\. Gudovskiy, T\. Okuno, Y\. Nakata, K\. Keutzer, and S\. Zhang\(2025\)SparseVLM: visual token sparsification for efficient vision\-language model inference\.InProceedings of the International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2610.33215#S1.p2.1),[§2](https://arxiv.org/html/2610.33215#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.33215#S4.SS1.p4.1)\.

## Appendix ALimitations and Future Directions

Our evaluation is limited to public VLMs and standard image/video benchmarks, and may not fully capture behavior on closed\-source models, proprietary inference stacks, or specialized domains\. As a training\-free pruning method, QViD also inherits the biases and failure modes of the underlying VLM\. Practical speedups may vary with hardware, batching, sequence length, and attention\-kernel implementations\. Future work could extend query–vision decomposition to adaptive token budgets, dynamic selection of the number of SVD components and tighter integration with hardware\-aware inference systems\. It would also be valuable to study whether the same low\-rank interaction structure can guide token pruning during training, in longer video settings, or in broader multimodal architectures\.

## Appendix BLayer Selection Results

We provide additional ablations on the choice of pruning layerℓ\\ell\. For each model, we compute the QViD scores using intermediate representations from layerℓ\\elland evaluate the resulting compressed model\. This study tests whether token selection is more effective when computed from the raw input representations or after a few early layers of cross\-token processing\.

Table[5](https://arxiv.org/html/2610.33215#A2.T5)reports the layer sweep for LLaVA\-1\.5\-7B at 128 retained tokens\. Table[6](https://arxiv.org/html/2610.33215#A2.T6)reports the full Qwen2\-VL\-7B layer sweep across all three retention ratios\.

Table 5:Layer Selection Ablation on LLaVA\-1\.5\-7B\.We vary the scoring layerℓ\\ellwhile retaining 128 visual tokens\. MME denotes the sum of MME\-Perception and MME\-Cognition\. Avg\. denotes the mean normalized performance relative to the full\-token baseline across all reported metrics\.Bolddenotes the best value in each column\.For LLaVA\-1\.5\-7B, performance is relatively stable across early layers\. Layerℓ=3\\ell=3achieves the best average performance, but the gap between layers 1–5 is small, suggesting that QViD is not highly sensitive to the exact early layer used for scoring\. The mild drop at layer 6 suggests that pruning too late may begin to discard visual tokens after task\-relevant information has already been mixed across tokens\.

Table 6:Full Layer Sweep on Qwen2\-VL\-7B\.We vary the QViD scoring layerℓ\\ellacross three retention ratios\. Avg\. is computed as the mean normalized performance relative to the full\-token baseline\.Bolddenotes the best value in each column within a retention block\.For Qwen2\-VL\-7B, the choice of layer has a more visible effect, especially at lower retention ratios\. Layerℓ=3\\ell=3gives the best average performance at 22\.2% and 11\.1% retention, while remaining within 0\.2 percentage points of the best average at 33\.3% retention\. This suggests that a few early layers help refine the query–vision compatibility scores before pruning, while delaying pruning too much can reduce robustness under more aggressive compression\. Overall, these results support usingℓ=3\\ell=3as the default pruning layer in our experiments\.

## Appendix CEffect of the Number of SVD Components

Table 7:Ablation on the Number of SVD Components for LLaVA\-v1\.5\-7B\.nsvdn\_\{\\mathrm\{svd\}\}denotes the number of singular\-vector components used to compute the visual\-token importance scores\. We varynsvdn\_\{\\mathrm\{svd\}\}, the number of singular\-vector components used for QViD scoring, across three token budgets\. Avg\. denotes the mean normalized performance relative to the full\-token baseline across all reported metrics\.Here,nsvdn\_\{\\mathrm\{svd\}\}denotes the number of singular\-vector components used to compute the visual\-token importance scores\. The results show that QViD can capture useful query–vision structure with only a small number of components, but becomes more robust as more dominant interaction modes are included\. In particular, performance improves sharply fromnsvd=1,2n\_\{\\mathrm\{svd\}\}=1,2tonsvd=4,8n\_\{\\mathrm\{svd\}\}=4,8, showing that single\-mode scoring is insufficient under compression\. The default settingnsvd=16n\_\{\\mathrm\{svd\}\}=16achieves the best average performance at 128 and 192 retained tokens and remains close to the best setting at 64 tokens\. This justifies using multiple SVD components in the final scoring rule and shows that the dominant query–vision subspace provides a stronger pruning signal than lower\-rank alternatives\.

## Appendix DScoring Rule Ablation

We ablate different ways of converting the query–vision interaction matrix into visual\-token scores\. All variants use the same interaction matrixM=Qtxt​Kvis⊤M=Q\_\{\\mathrm\{txt\}\}K\_\{\\mathrm\{vis\}\}^\{\\top\}, but differ in how token importance is computed fromMM\.

#### Compared Scoring Rules\.

Column Normdirectly scores each visual token by theℓ2\\ell\_\{2\}\-norm of its corresponding column inMM, without using an SVD decomposition\. This measures the raw interaction magnitude between the visual token and the query tokens\.Sigma\-Weightedcomputes an SVD of the global interaction matrix and scores tokens by weighting each right singular\-vector component by the corresponding singular value\.Per\-Head Maxcomputes SVD\-based scores independently for each attention head, and then aggregates scores by taking the maximum across heads\. The default QViD scoring uses per\-head SVD scores with the fixed aggregation rule used in the main experiments\.

Table 8:Scoring\-Rule Ablation on LLaVA\-1\.5\-7B\.We compare different scoring rules derived from the query–vision interaction matrix at matched token budgets\. Avg\. denotes the mean normalized performance relative to the full\-token baseline across all reported metrics\.Table[8](https://arxiv.org/html/2610.33215#A4.T8)shows that raw interaction strength alone is not sufficient for reliable token selection\. Column\-norm and sigma\-weighted scoring perform identically because they induce the same token ordering\. Indeed, ifM=U​Σ​V⊤M=U\\Sigma V^\{\\top\}, then

M⊤​M=V​Σ2​V⊤\.M^\{\\top\}M=V\\Sigma^\{2\}V^\{\\top\}\.Therefore, for visual tokenjj,

∥M:,j∥22=\(M⊤M\)j​j=∑kσk2Vj​k2\.\\\|M\_\{:,j\}\\\|\_\{2\}^\{2\}=\(M^\{\\top\}M\)\_\{jj\}=\\sum\_\{k\}\\sigma\_\{k\}^\{2\}V\_\{jk\}^\{2\}\.Thus, sigma\-weighted scoring is exactly the squared column\-norm score\. Since squaring is monotone for nonnegative scores, both methods select the same top\-kkvisual tokens and give identical evaluation results\. Their poor performance shows that selecting tokens by raw interaction energy is much weaker than selecting tokens by their participation in the dominant query–vision subspace\.

In contrast, per\-head SVD\-based scoring performs substantially better, indicating that the main benefit comes from preserving head\-specific low\-rank query–vision structure rather than simply selecting tokens with large interaction magnitude\.

The per\-head max variant is competitive with the default QViD aggregation and performs slightly better on some metrics, especially at smaller token budgets\. This suggests that different attention heads may specialize in different query\-relevant visual cues\. In this work, we keep the default aggregation fixed for consistency across models and budgets\. Adaptive or query\-dependent head aggregation is an interesting direction for future work\.

## Appendix EEfficient QViD Scoring

QViD scores visual tokens using the dominant right singular directions of the query–vision interaction matrix\. For each attention headhh, we form

M\(h\)=Qtxt\(h\)​\(Kvis\(h\)\)⊤∈ℝNq×Nv,M^\{\(h\)\}=Q\_\{\\mathrm\{txt\}\}^\{\(h\)\}\\left\(K\_\{\\mathrm\{vis\}\}^\{\(h\)\}\\right\)^\{\\top\}\\in\\mathbb\{R\}^\{N\_\{q\}\\times N\_\{v\}\},whereNqN\_\{q\}is the number of query tokens andNvN\_\{v\}is the number of visual tokens\. A direct implementation could compute the SVD ofM\(h\)M^\{\(h\)\}and use its top right singular vectors for token scoring\. However, sinceNq≪NvN\_\{q\}\\ll N\_\{v\}, we can compute the same right\-singular subspace more efficiently using a QR decomposition\.

#### QR\-Based Reduction\.

Consider the transpose

\(M\(h\)\)⊤∈ℝNv×Nq,\\left\(M^\{\(h\)\}\\right\)^\{\\top\}\\in\\mathbb\{R\}^\{N\_\{v\}\\times N\_\{q\}\},which is tall and thin\. We first compute its thin QR decomposition:

\(M\(h\)\)⊤=Q^\(h\)​R\(h\),Q^\(h\)∈ℝNv×Nq,R\(h\)∈ℝNq×Nq,\\left\(M^\{\(h\)\}\\right\)^\{\\top\}=\\widehat\{Q\}^\{\(h\)\}R^\{\(h\)\},\\qquad\\widehat\{Q\}^\{\(h\)\}\\in\\mathbb\{R\}^\{N\_\{v\}\\times N\_\{q\}\},\\quad R^\{\(h\)\}\\in\\mathbb\{R\}^\{N\_\{q\}\\times N\_\{q\}\},where the columns ofQ^\(h\)\\widehat\{Q\}^\{\(h\)\}are orthonormal andR\(h\)R^\{\(h\)\}is upper triangular\. We then compute the SVD of the small matrix\(R\(h\)\)⊤\\left\(R^\{\(h\)\}\\right\)^\{\\top\}:

\(R\(h\)\)⊤=U~\(h\)​Σ\(h\)​\(V~\(h\)\)⊤\\left\(R^\{\(h\)\}\\right\)^\{\\top\}=\\widetilde\{U\}^\{\(h\)\}\\Sigma^\{\(h\)\}\\left\(\\widetilde\{V\}^\{\(h\)\}\\right\)^\{\\top\}
Substituting the QR factorization intoM\(h\)M^\{\(h\)\}, we obtain

M\(h\)\\displaystyle M^\{\(h\)\}=\(R\(h\)\)⊤​\(Q^\(h\)\)⊤\\displaystyle=\\left\(R^\{\(h\)\}\\right\)^\{\\top\}\\left\(\\widehat\{Q\}^\{\(h\)\}\\right\)^\{\\top\}=U~\(h\)​Σ\(h\)​\(V~\(h\)\)⊤​\(Q^\(h\)\)⊤\\displaystyle=\\widetilde\{U\}^\{\(h\)\}\\Sigma^\{\(h\)\}\\left\(\\widetilde\{V\}^\{\(h\)\}\\right\)^\{\\top\}\\left\(\\widehat\{Q\}^\{\(h\)\}\\right\)^\{\\top\}=U~\(h\)​Σ\(h\)​\(Q^\(h\)​V~\(h\)\)⊤\\displaystyle=\\widetilde\{U\}^\{\(h\)\}\\Sigma^\{\(h\)\}\\left\(\\widehat\{Q\}^\{\(h\)\}\\widetilde\{V\}^\{\(h\)\}\\right\)^\{\\top\}Therefore, the right singular vectors ofM\(h\)M^\{\(h\)\}are

V\(h\)=Q^\(h\)​V~\(h\)∈ℝNv×NqV^\{\(h\)\}=\\widehat\{Q\}^\{\(h\)\}\\widetilde\{V\}^\{\(h\)\}\\in\\mathbb\{R\}^\{N\_\{v\}\\times N\_\{q\}\}Thus, the QR\-based computation is mathematically equivalent to computing the right singular vectors ofM\(h\)M^\{\(h\)\}directly, but avoids applying SVD to the full rectangularNq×NvN\_\{q\}\\times N\_\{v\}matrix\.

#### Token Scoring\.

LetVreff\(h\)∈ℝNv×reffV^\{\(h\)\}\_\{r\_\{\\mathrm\{eff\}\}\}\\in\\mathbb\{R\}^\{N\_\{v\}\\times r\_\{\\mathrm\{eff\}\}\}denote the firstreffr\_\{\\mathrm\{eff\}\}columns ofV\(h\)V^\{\(h\)\}\. In our experiments, we setr=nsvdr=n\_\{\\mathrm\{svd\}\}and use

reff=min⁡\(nsvd,Nq,Nv\),r\_\{\\mathrm\{eff\}\}=\\min\(n\_\{\\mathrm\{svd\}\},N\_\{q\},N\_\{v\}\),so the score is well defined even for short queries\. The per\-head visual\-token score is the row\-leverage score

sj\(h\)=∑k=1reff\(Vj​k\(h\)\)2s\_\{j\}^\{\(h\)\}=\\sum\_\{k=1\}^\{r\_\{\\mathrm\{eff\}\}\}\\left\(V\_\{jk\}^\{\(h\)\}\\right\)^\{2\}We average these scores across attention heads:

sj=1H​∑h=1Hsj\(h\)s\_\{j\}=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}s\_\{j\}^\{\(h\)\}The top\-KKvisual tokens bysjs\_\{j\}are retained and passed to the remaining LLM layers\.

#### Complexity\.

For each head, the QR decomposition of\(M\(h\)\)⊤∈ℝNv×Nq\\left\(M^\{\(h\)\}\\right\)^\{\\top\}\\in\\mathbb\{R\}^\{N\_\{v\}\\times N\_\{q\}\}costsO⁡\(Nv​Nq2\)O\(N\_\{v\}N\_\{q\}^\{2\}\), and the SVD of the smallNq×NqN\_\{q\}\\times N\_\{q\}matrix costsO⁡\(Nq3\)O\(N\_\{q\}^\{3\}\)\. AcrossHHheads, the scoring complexity is

O⁡\(H​Nv​Nq2\+H​Nq3\)O\(HN\_\{v\}N\_\{q\}^\{2\}\+HN\_\{q\}^\{3\}\)SinceNqN\_\{q\}is typically much smaller thanNvN\_\{v\}, this cost is small compared with the cost of forwarding all visual tokens through the LLM\. In our LLaVA\-1\.5\-7B experiments, the full scoring step, including construction ofM=Qtxt​Kvis⊤M=Q\_\{\\mathrm\{txt\}\}K\_\{\\mathrm\{vis\}\}^\{\\top\}, costs less than5050MFLOPs\.

#### Implementation\.

The QR\-based scoring step can be implemented compactly as follows:

```
# M: (H, N_q, N_v), n_svd = number of SVD components
r_eff = min(n_svd, M.shape[-2], M.shape[-1])

Q_qr, R_qr = torch.linalg.qr(M.mT)
_, _, Vh = torch.linalg.svd(R_qr.mT, full_matrices=False)

# Right singular vectors of M
W = (Q_qr @ Vh.mT)[:, :, :r_eff]  # (H, N_v, r_eff)

# Visual-token leverage scores, averaged over heads
scores = (W * W).sum(dim=-1).mean(dim=0)  # (N_v,)
```

## Appendix FDetailed Efficiency Results

In this section, we provide detailed latency, throughput, and FLOPs measurements for LLaVA\-1\.5\-7B\. The main paper reports the high\-level compute comparison\. Here, we include the full efficiency breakdown used for the complexity analysis\.

#### Scoring Overhead\.

QViD adds a lightweight query–vision scoring pass before pruning: with the QR\-based implementation described in Appendix[E](https://arxiv.org/html/2610.33215#A5), the scoring cost isO⁡\(H​Nv​Nq2\+H​Nq3\)O\(HN\_\{v\}N\_\{q\}^\{2\}\+HN\_\{q\}^\{3\}\)forHHattention heads,NqN\_\{q\}query tokens, andNvN\_\{v\}visual tokens\. SinceNq≪NvN\_\{q\}\\ll N\_\{v\}in typical VLM inputs, this overhead is small relative to forwarding all visual tokens through the LLM and is amortized by pruning; detailed derivations and efficiency measurements are provided in Appendices[E](https://arxiv.org/html/2610.33215#A5)as well as Section[4\.2](https://arxiv.org/html/2610.33215#S4.SS2)\.

#### Compute Accounting\.

For the full\-token baseline, the model processes all576576visual tokens\. For V2Drop, the reported TFLOPs include its per\-layer token schedule, where early layers retain the full visual sequence before pruning\. For QViD, the reported TFLOPs include the early query\-aware scoring pass and the subsequent LLM computation using the retained visual tokens\. The QR/SVD scoring computation itself is small, costing less than5050MFLOPs including construction of the interaction matrix\.

From our FLOP measurements on LLaVA\-1\.5\-7B, removing one visual token from the remaining LLM computation saves roughly0\.010\.01TFLOPs\. Therefore, retaining6464,128128, and192192tokens instead of all576576visual tokens saves approximately7\.177\.17,6\.276\.27, and5\.385\.38TFLOPs, respectively, in the remaining LLM computation\. This shows that the QViD scoring overhead is much smaller than the compute saved by pruning\.

Table 9:Detailed Efficiency Results on LLaVA\-1\.5\-7B\.We report measured latency, throughput, speedup, and total TFLOPs for different pruning methods and retained\-token budgets\. TFLOPs include pruning overhead\.
#### Latency and Throughput\.

QViD has slightly higher latency than V2Drop at the same retained\-token budget because it computes query\-aware scores before pruning\. However, it still improves throughput over the full\-token baseline\. The full baseline runs at8\.608\.60images/s, while QViD runs at10\.4010\.40,10\.2710\.27, and10\.2610\.26images/s for6464,128128, and192192retained tokens, respectively\. Thus, QViD introduces modest additional overhead relative to V2Drop, but remains substantially more efficient than the full\-token baseline\.

Figure 4:Accuracy–Compute Tradeoff on LLaVA\-1\.5\-7B\.We compare V2Drop, and QViD at 64, 128, and 192 retained visual tokens on POPE, GQA, and TextVQA\. The gray dashed line and star indicate the full\-token baseline with 576 visual tokens\. QViD achieves a stronger accuracy–compute tradeoff, especially under aggressive pruning, by retaining more query\-relevant visual tokens at comparable TFLOPs\.
#### Implementation details\.

All experiments are conducted using Python 3\.11, PyTorch 2\.5\.1, and CUDA 12\.1 on a single NVIDIA A100\-80GB GPU\. For LLaVA\-1\.5\-7B, Qwen2\-VL\-7B, Video\-LLaVA\-7B, and LLaVA\-OV\-7B, we follow the standard evaluation settings of the corresponding model and benchmark\. Unless otherwise stated, QViD prunes at LLM layerℓ=3\\ell=3usingnsvd=16n\_\{\\mathrm\{svd\}\}=16dominant SVD components, and omits Rotary Positional Embedding \(RoPE\) when constructing the query–vision interaction matrix\. Reported TFLOPs include both the pruning overhead and the subsequent LLM computation on the retained visual tokens\.

## Appendix GAdditional Multimodal Benchmark Results

We further evaluate QViD on additional multimodal benchmarks beyond the main evaluation suite\. For LLaVA\-v1\.5\-7B, we report results on AI2D, ChartQA, MMMU, OCRBench, and SEEDBench\. For Qwen2\-VL\-7B, we additionally evaluate on MMMU\-Pro Standard, a more challenging multimodal reasoning benchmark with 10 answer options\.

Table 10:Additional Multimodal Benchmark Results on LLaVA\-v1\.5\-7B\.We compare QViD with V2Drop at matched token budgets on AI2D, ChartQA, MMMU, OCRBench, and SEEDBench\. Avg\. denotes the mean normalized performance relative to the full\-token baseline\.Table 11:MMMU\-Pro Standard Results on Qwen2\-VL\-7B\.We compare QViD with V2Drop under matched retention ratios on the 1,730\-sample standard\-format split\.Table[10](https://arxiv.org/html/2610.33215#A7.T10)shows that QViD consistently improves over V2Drop in average normalized performance across additional LLaVA\-v1\.5\-7B benchmarks\. The gains are especially large on OCRBench and ChartQA, suggesting that query\-aware token selection preserves visually grounded information beyond the main evaluation suite\.

Table[11](https://arxiv.org/html/2610.33215#A7.T11)reports results on MMMU\-Pro Standard for Qwen2\-VL\-7B\. In the standard\-format split, the model receives the question, answer choices, and associated image as structured multimodal input\. QViD remains close to V2Drop at 33\.3% and 22\.2% retention, while substantially improving performance at 11\.1% retention, from 23\.6% to 28\.8%\. This indicates that query\-aware token selection is especially beneficial under aggressive compression on challenging multimodal reasoning tasks\.

## Appendix HAdditional Video Understanding Results

Table[12](https://arxiv.org/html/2610.33215#A8.T12)reports additional video understanding results on Video\-LLaVA\-7B\. The trends are consistent with the LLaVA\-OV\-7B results in the main paper: QViD outperforms V2Drop at all retention ratios, with the largest gain under the most aggressive compression setting\.

Table 12:Video Understanding Results on Video\-LLaVA\-7B\.We compare QViD with V2Drop on MVBench and VideoMME under token compression\. Avg\. denotes the mean normalized performance over MVBench and VideoMME Overall relative to the full\-token baseline\.
## Appendix IHallucination and Captioning Results

We further evaluate QViD on HallusionBench and COCO Captioning to test whether token compression preserves both hallucination robustness and open\-ended generation quality\. HallusionBench is a hallucination\-sensitive benchmark and is reported using all\-pair accuracy \(aAcc\), figure accuracy \(fAcc\), and question accuracy \(qAcc\)\. COCO Captioning is evaluated using CIDEr, a caption\-quality metric based on consensus between generated captions and human reference captions\. CIDEr is not an accuracy metric; in our lmms\-eval setup it is reported on a normalized scale, where values around 1\.0 roughly correspond to a traditional CIDEr score around 100\. Higher is better for all metrics\.

Table 13:HallusionBench and COCO Captioning Results on LLaVA\-v1\.5\-7B\.We compare QViD with V2Drop at matched token budgets\. HallusionBench is reported using aAcc, fAcc, and qAcc, while COCO Captioning is reported using CIDEr\. QViD improves both hallucination\-related metrics and caption quality under visual\-token compression\.Table 14:HallusionBench and COCO Captioning Results on Qwen2\-VL\-7B\.We compare QViD with V2Drop at matched retention ratios where available\. HallusionBench is reported using aAcc, fAcc, and qAcc, while COCO Captioning is reported using CIDEr\. QViD preserves stronger hallucination robustness and substantially better captioning quality under compression\.Tables[13](https://arxiv.org/html/2610.33215#A9.T13)and[14](https://arxiv.org/html/2610.33215#A9.T14)show that QViD remains effective on hallucination\-sensitive and generative benchmarks\. On LLaVA\-v1\.5\-7B, QViD improves over V2Drop across all HallusionBench metrics and CIDEr at every token budget\. On Qwen2\-VL\-7B, QViD also improves HallusionBench performance and gives substantially higher CIDEr at matched retention ratios\. These results suggest that query\-aware token selection preserves visual information needed for both grounded recognition and open\-ended caption generation\.

## Appendix JAdditional Qualitative Analysis

### J\.1Low\-Rank Structure of Query–Vision Interactions

Figure 5:Low\-Rank Structure of Query–Vision Interactions\.We plot the normalized singular\-value spectrumσk/σ1\\sigma\_\{k\}/\\sigma\_\{1\}ofM=Qtxt​Kvis⊤M=Q\_\{\\mathrm\{txt\}\}K\_\{\\mathrm\{vis\}\}^\{\\top\}at the pruning layer of LLaVA\-1\.5\-7B\. Across datasets, the spectra decay rapidly, showing that query–vision interactions are concentrated in a few dominant directions\. Faint curves show individual samples, solid curves show the mean, and shaded bands indicate variation across heads\.A key motivation for QViD is that the query–vision interaction matrixMMhas a concentrated singular spectrum\. To verify this, we compute the singular values ofM=Qtxt​Kvis⊤M=Q\_\{\\mathrm\{txt\}\}K\_\{\\mathrm\{vis\}\}^\{\\top\}at the pruning layer across multiple evaluation datasets\. Figure[5](https://arxiv.org/html/2610.33215#A10.F5)plots the normalized spectrumσk/σ1\\sigma\_\{k\}/\\sigma\_\{1\}on a logarithmic scale\.

Across all datasets, the spectrum decays rapidly after the first few components, showing that most of the query–vision interaction energy is concentrated in a small number of dominant directions\. This supports our design choice of using only the top SVD components for token scoring\. The trend is consistent across POPE, GQA, TextVQA, MMBench, ScienceQA, and MME, suggesting that the low\-rank structure is not specific to a single benchmark or task type\. Although the exact decay rate varies across datasets, the dominant components consistently capture the strongest query\-conditioned interaction modes\.

### J\.2Qualitative Analysis of SVD Components

For the query\-token visualizations, we use the analogous left\-singular\-vector score\. IfM=U​Σ​V⊤M=U\\Sigma V^\{\\top\}andUrU\_\{r\}contains the toprrleft singular vectors, the query\-token score isqi=∑k=1rUi​k2\.q\_\{i\}=\\sum\_\{k=1\}^\{r\}U\_\{ik\}^\{2\}\.Figure[6](https://arxiv.org/html/2610.33215#A10.F6)compares scores from the dominant SVD components with scores from the residual components obtained after removing the top directions\. The dominant components assign high scores to task\-critical words, such as*country*,*coin*,*animal*, and*hat*, which are directly tied to the answer\. In contrast, the residual components place more weight on less informative or function words, indicating weaker alignment with the task\. The singular\-value spectra in the same figure also show rapid decay, suggesting that the main query–vision interaction structure is concentrated in a few dominant directions\. This supports our choice of using only the dominant SVD components for visual\-token scoring\.

![Refer to caption](https://arxiv.org/html/2610.33215v1/query_score_spectrum.png)Figure 6:Dominant SVD Components Capture Task\-Relevant Structure\.Top SVD components emphasize task\-critical query tokens, while residual components are less aligned with the question\. The spectra show that query–vision interactions are concentrated in a few dominant directions\.

## Appendix KConfidence Intervals

To assess the statistical reliability of the reported results, we provide 95% confidence intervals for Qwen2\-VL\-7B in Table[15](https://arxiv.org/html/2610.33215#A11.T15), computed over three independent random seeds\. Across different token retention ratios, QViD consistently outperforms FastV and V2Drop on GQA and TextVQA, with particularly large improvements at lower retention ratios\. At 5\.56% retention, QViD also achieves substantial gains across all six benchmarks compared to both baselines\.

Table 15:95% confidence intervals on Qwen2\-VL\-7B over three independent random seeds\.### K\.1Additional Token\-Score Visualizations

Figure[7](https://arxiv.org/html/2610.33215#A11.F7)shows additional token\-score maps comparing QViD with V2Drop and SparseVLM\. Across diverse image\-query pairs, QViD assigns high scores to localized regions that are relevant to the question, such as the snowboarder, bat, scoreboard, license plate, and player jersey\. In contrast, the baseline score maps are often more diffuse or place high scores on less query\-relevant regions\. These examples provide qualitative evidence that query–vision SVD scoring better aligns token importance with the visual evidence needed to answer the input query\.

![Refer to caption](https://arxiv.org/html/2610.33215v1/combined_appendix.png)Figure 7:Additional Qualitative Token\-Score Visualizations\.We compare token\-score maps from QViD, V2Drop, and SparseVLM across multiple image\-query pairs\. Brighter regions indicate higher token scores\. QViD tends to assign high scores to query\-relevant visual regions, while the baseline methods often produce more diffuse score maps\.
### K\.2Query\-Token Scores from Dominant and Residual Components

Figure[8](https://arxiv.org/html/2610.33215#A11.F8)provides additional examples comparing query\-token scores from the dominant SVD components and the residual components\. For each image\-query pair, we compute scores using the top SVD components and compare them with scores from the residual directions\. The dominant components tend to emphasize task\-relevant query words, such as*country*,*state*,*lions*,*score*,*batter*,*liquor*, and*jersey*\. In contrast, the residual components often assign larger scores to function words or generic question tokens\. This supports our observation that the dominant query–vision interaction modes capture semantic structure useful for answering the query\.

![Refer to caption](https://arxiv.org/html/2610.33215v1/query_score_components_no_spectrum_appendix.png)Figure 8:Additional Query\-Token Score Visualizations\.For each example, we compare query\-token scores obtained from the dominant SVD components with scores from the residual components\. The dominant components place higher weight on semantically informative query tokens, while residual components are less aligned with the task\-relevant words\.