Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Prunin

arXiv cs.AI 论文
visual-token-pruning multimodal-llm attention-distillation model-efficiency cross-modal research

摘要

This paper introduces a method to predict middle-layer attention in multimodal LLMs to prune visual tokens efficiently, using question-contrastive teacher selection and cross-modal attention distillation.

arXiv:2608.06411v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens. Visual token pruning can reduce this cost, but requires accurate token importance estimates. Recent studies have demonstrated that text-to-vision attention from middle language model layers can effectively guide visual token pruning, typically using attention from a predefined middle layer to select the visual tokens to retain. Two problems therefore remain. First, our analysis shows that the layer whose attention is most responsive to the question varies substantially across samples, making a fixed layer suboptimal. Second, obtaining attention from the appropriate middle layer requires processing numerous visual tokens through several language model layers, by which point considerable computation has already been spent. To address both problems, we propose Middle-layer Attention Prediction (MAP), which uses Question Contrastive Teacher Selection to identify a sample-specific teacher layer by contrasting attention under the original and reference questions, and distills attention from the selected layer into a lightweight predictor that estimates visual token importance from multi-modal input features. During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer. Thus, MAP requires no attention maps for pruning and remains compatible with existing inference acceleration techniques. Across ten benchmarks on LLaVA-NeXT-7B, MAP retains 97.5% of the unpruned model performance with only 5.56% of the visual tokens, yielding a 3.09x end-to-end speedup.
查看原文
查看缓存全文

缓存时间: 2026/08/10 07:58 

# Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning
Source: [https://arxiv.org/html/2608.06411](https://arxiv.org/html/2608.06411)
### Question Contrastive Teacher Selection

The utility of attention from different middle layers for visual token pruning varies across samples, making a fixed layer suboptimal as a universal teacher\. To construct teacher supervision tailored to each sample, we propose Question Contrastive Teacher Selection \(QCTS\), which compares the text\-to\-vision attention produced for the input question with that produced for a generic reference question on the same image and then selects the layer with the largest discrepancy as the teacher\. The selected teacher attention is expected to best capture visual regions relevant to the input question\.

Given an image and question pair, the image is processed by the vision encoder and multimodal projector to produceNNvisual token embeddings𝐕=\[𝐯1,…,𝐯N\]\\mathbf\{V\}=\[\\mathbf\{v\}\_\{1\},\\ldots,\\mathbf\{v\}\_\{N\}\], while theMMquestion tokens are mapped by the LLM embedding layer to text token embeddings𝐓=\[𝐭1,…,𝐭M\]\\mathbf\{T\}=\[\\mathbf\{t\}\_\{1\},\\ldots,\\mathbf\{t\}\_\{M\}\]\. Together,𝐕\\mathbf\{V\}and𝐓\\mathbf\{T\}constitute the multimodal input representations provided to the language model\. Letℒ=\{0,…,L−1\}\\mathcal\{L\}=\\\{0,\\ldots,L\-1\\\}denote the set of allLLlanguage model layers\. At each layerl∈ℒl\\in\\mathcal\{L\}, we extract text\-to\-vision attention and average it over allHHattention heads andMMquestion tokens for each visual token:

anl=1H​M​∑h=1H∑m=1MAh,m,nl\.a\_\{n\}^\{l\}=\\frac\{1\}\{HM\}\\sum\_\{h=1\}^\{H\}\\sum\_\{m=1\}^\{M\}A\_\{h,m,n\}^\{l\}\.\(1\)Here,Ah,m,nlA\_\{h,m,n\}^\{l\}denotes the attention from question token𝐭m\\mathbf\{t\}\_\{m\}to visual token𝐯n\\mathbf\{v\}\_\{n\}at headhhof layerll\. At each layer, we applyℓ1\\ell\_\{1\}normalization to the averaged attention scores over visual tokens, yielding the target attention distributions\{𝐩l\}l∈ℒ\\\{\\mathbf\{p\}^\{l\}\\\}\_\{l\\in\\mathcal\{L\}\}\.

We obtain the reference attention distributions by pairing the same image with the fixed reference question, “What is shown in this image?”, while keeping the remaining input unchanged\. Applying the attention aggregation and normalization procedure produces𝐩refl\\mathbf\{p\}\_\{\\mathrm\{ref\}\}^\{l\}at each layer, with attention averaged over theMrefM\_\{\\mathrm\{ref\}\}reference question tokens\. As the reference question does not specify a particular object, attribute, relation, or region, it provides a generic visual attention baseline\. The discrepancy between𝐩l\\mathbf\{p\}^\{l\}and𝐩refl\\mathbf\{p\}\_\{\\mathrm\{ref\}\}^\{l\}therefore quantifies how the input question redirects visual attention relative to this baseline at layerll\. Specifically, we quantify the discrepancy at each layer using the Jensen–Shannon divergence and select the layer with the largest value:

l⋆=arg​maxl∈ℒ⁡DJS​\(𝐩l,𝐩refl\)\.l^\{\\star\}=\\operatorname\*\{arg\\,max\}\_\{l\\in\\mathcal\{L\}\}D\_\{\\mathrm\{JS\}\}\\left\(\\mathbf\{p\}^\{l\},\\mathbf\{p\}\_\{\\mathrm\{ref\}\}^\{l\}\\right\)\.\(2\)The attention distribution𝐩l⋆\\mathbf\{p\}^\{l^\{\\star\}\}from the selected layer is then used as supervision to train the predictor\.

### Cross\-Modal Attention Distillation

After QCTS selects the teacher distribution𝐩l⋆\\mathbf\{p\}^\{l^\{\\star\}\}for each training sample, we train a lightweight predictor to reproduce it directly from the multimodal input representations of the LLM\. The predictor first transforms the text and visual representations using two modality\-specific MLPs,fTf\_\{T\}andfVf\_\{V\}, and then projects them into queries and keys using𝐖Q\\mathbf\{W\}\_\{Q\}and𝐖K\\mathbf\{W\}\_\{K\}, respectively\. Before computing the attention scores, we apply RoPE to the projected queries and keys to preserve positional information\. The predicted attention distribution over visual tokens is computed as

p^n=1M​∑m=1Msoftmaxn⁡\(⟨𝐖Q​fT​\(𝐭m\),𝐖K​fV​\(𝐯n\)⟩d\),\\hat\{p\}\_\{n\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}\\operatorname\{softmax\}\_\{n\}\\left\(\\frac\{\\left\\langle\\mathbf\{W\}\_\{Q\}f\_\{T\}\(\\mathbf\{t\}\_\{m\}\),\\mathbf\{W\}\_\{K\}f\_\{V\}\(\\mathbf\{v\}\_\{n\}\)\\right\\rangle\}\{\\sqrt\{d\}\}\\right\),\(3\)whereddis the projection dimension\. Since only attention weights are required, value and output projections are omitted\.

During offline training, the predictor takes the multimodal input embeddings as input and is optimized by minimizingDKL​\(𝐩l⋆∥𝐩^\)D\_\{\\mathrm\{KL\}\}\(\\mathbf\{p\}^\{l^\{\\star\}\}\\\|\\widehat\{\\mathbf\{p\}\}\)\. This objective distills the visual token importance from the selected teacher distribution into the predictor\.

### Inference\-Time Token Pruning

During inference, MAP treats the predicted attention𝐩^\\widehat\{\\mathbf\{p\}\}as visual token importance\. Given a target numberKKof visual tokens to retain, MAP first keeps the topK0K\_\{0\}tokens with the highest predicted attention to form the initial retained setℛ0\\mathcal\{R\}\_\{0\}, while the remaining tokens form the candidate set𝒞0\\mathcal\{C\}\_\{0\}\. Starting from this initialization, MAP greedily adds one candidate token at each iteration untilKKtokens are retained, jointly considering predicted importance and feature diversity\.

To promote diversity, MAP maintains an orthonormal basis constructed from the selected visual embeddings\. The initial basis𝐁0\\mathbf\{B\}\_\{0\}is obtained by orthonormalizing the embeddings inℛ0\\mathcal\{R\}\_\{0\}and is updated whenever a new token is retained\. At iterationtt, the next token is selected by

it=arg​maxi∈𝒞t−1⁡p^i​‖𝐯i−𝐁t−1​𝐁t−1⊤​𝐯i‖2\.i\_\{t\}=\\operatorname\*\{arg\\,max\}\_\{i\\in\\mathcal\{C\}\_\{t\-1\}\}\\widehat\{p\}\_\{i\}\\left\\\|\\mathbf\{v\}\_\{i\}\-\\mathbf\{B\}\_\{t\-1\}\\mathbf\{B\}\_\{t\-1\}^\{\\top\}\\mathbf\{v\}\_\{i\}\\right\\\|\_\{2\}\.\(4\)The importance term favors tokens related to the input question, while the orthogonal residual rewards features complementary to the current subspace\. Afteriti\_\{t\}is selected, it is moved from the candidate set to the retained set\. Its embedding𝐯it\\mathbf\{v\}\_\{i\_\{t\}\}is then orthogonalized against𝐁t−1\\mathbf\{B\}\_\{t\-1\}, and the normalized residual is appended to form𝐁t\\mathbf\{B\}\_\{t\}\. This procedure continues until the retained set containsKKtokens\. Before language model processing, the selected tokens are restored to their original order while preserving their positional indices and are then combined with the question tokens\.

## Experiments

MethodGQASQAIVQATPOPEMMEVQAv2MMBENMMBCNVizWizSEED\-IRel\.↑\\uparrow\\rowcolorgray\!12Upper Bound, All 2880 Tokens\(100\.0%\)Vanilla64\.270\.161\.386\.5185181\.867\.460\.657\.670\.2100\.0%\\rowcolorgray\!12Retain 640 Tokens\(↓\\downarrow77\.8%\)VisionZip \(CVPR2025\)61\.268\.159\.986\.0178779\.165\.858\.157\.166\.797\.1%DART \(EMNLP2025\)61\.368\.259\.585\.0172178\.364\.957\.157\.067\.996\.3%CDPruner \(NeurIPS2025\)62\.667\.958\.487\.3180079\.966\.357\.555\.668\.697\.3%MMTok \(ICLR2026\)62\.268\.458\.986\.7182979\.365\.256\.555\.867\.797\.0%ZOO\-Prune \(CVPR2026\)62\.267\.758\.086\.7178379\.665\.257\.255\.267\.996\.6%MAP \(Ours\)62\.868\.961\.088\.2183880\.468\.459\.657\.669\.599\.4%\\rowcolorgray\!12Retain 320 Tokens\(↓\\downarrow88\.9%\)VisionZip \(CVPR2025\)58\.967\.558\.882\.5170276\.263\.355\.656\.263\.493\.8%DART \(EMNLP2025\)59\.567\.557\.681\.0170575\.764\.255\.756\.864\.893\.9%CDPruner \(NeurIPS2025\)61\.667\.857\.487\.2180778\.465\.555\.755\.867\.196\.2%MMTok \(ICLR2026\)60\.967\.356\.985\.7179977\.664\.355\.855\.466\.295\.3%LearnPruner \(ICLR2026\)62\.268\.658\.4–184578\.366\.8–––97\.4%ZOO\-Prune \(CVPR2026\)61\.067\.257\.385\.5178778\.164\.956\.455\.166\.595\.5%MAP \(Ours\)61\.868\.660\.687\.8184279\.667\.159\.057\.568\.498\.5%\\rowcolorgray\!12Retain 160 Tokens\(↓\\downarrow94\.4%\)VisionZip \(CVPR2025\)55\.267\.955\.075\.8163071\.458\.650\.455\.558\.388\.5%DART \(EMNLP2025\)56\.867\.854\.975\.3161572\.562\.053\.656\.760\.690\.3%CDPruner \(NeurIPS2025\)60\.867\.555\.486\.8174976\.764\.253\.855\.265\.594\.3%MMTok \(ICLR2026\)60\.067\.954\.283\.8171575\.662\.954\.755\.764\.593\.3%LearnPruner \(ICLR2026\)58\.767\.655\.0–178476\.265\.3–––94\.0%ZOO\-Prune \(CVPR2026\)59\.967\.655\.483\.1173876\.164\.256\.355\.464\.193\.9%MAP \(Ours\)61\.368\.259\.787\.2181278\.566\.758\.657\.166\.997\.5%
Table 2:Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.Datasets and Models\.To empirically validate the effectiveness and generalizability of MAP, we conduct comprehensive experiments on three widely used MLLM backbones: LLaVA\-1\.5\-7B\(Liuet al\.[2024a](https://arxiv.org/html/2608.06411#bib.bib9)\), LLaVA\-NeXT\-7B\(Liuet al\.[2024b](https://arxiv.org/html/2608.06411#bib.bib10)\), and Qwen2\.5\-VL\-7B\-Instruct\(Baiet al\.[2025b](https://arxiv.org/html/2608.06411#bib.bib1)\)\. For LLaVA\-1\.5\-7B and LLaVA\-NeXT\-7B, we evaluate MAP on GQA\(Hudson and Manning[2019](https://arxiv.org/html/2608.06411#bib.bib39)\), SQA\(Luet al\.[2022](https://arxiv.org/html/2608.06411#bib.bib101)\), TextVQA\(Singhet al\.[2019](https://arxiv.org/html/2608.06411#bib.bib35)\), POPE\(Liet al\.[2023c](https://arxiv.org/html/2608.06411#bib.bib41)\), MME\(Fuet al\.[2023](https://arxiv.org/html/2608.06411#bib.bib72)\), VQAv2\(Goyalet al\.[2017](https://arxiv.org/html/2608.06411#bib.bib68)\), MMBench\(Liuet al\.[2024d](https://arxiv.org/html/2608.06411#bib.bib69)\), VizWiz\(Gurariet al\.[2018](https://arxiv.org/html/2608.06411#bib.bib40)\), and SEED\-Bench\(Liet al\.[2023a](https://arxiv.org/html/2608.06411#bib.bib71)\)\. For Qwen2\.5\-VL\-7B\-Instruct, we report results on AI2D\(Kembhaviet al\.[2016](https://arxiv.org/html/2608.06411#bib.bib89)\), MME, MMBench, TextVQA, and GQA\.

Compared Methods\.We compare MAP with representative token pruning methods\. FastV\(Chenet al\.[2024a](https://arxiv.org/html/2608.06411#bib.bib79)\)prunes visual tokens using early layer attention\. VisionZip\(Yanget al\.[2025](https://arxiv.org/html/2608.06411#bib.bib77)\)selects tokens using ViT attention\. DART\(Wenet al\.[2025](https://arxiv.org/html/2608.06411#bib.bib105)\)removes redundant tokens by similarity to pivot tokens\. CDPruner\(Zhanget al\.[2025b](https://arxiv.org/html/2608.06411#bib.bib80)\)balances relevance and diversity with an instruction\-conditioned DPP\. MMTok\(Donget al\.[2025](https://arxiv.org/html/2608.06411#bib.bib102)\)formulates token selection as a multimodal maximum coverage problem\. ZOO\-Prune\(Kimet al\.[2026](https://arxiv.org/html/2608.06411#bib.bib103)\)estimates token importance through sensitivity analysis\. LearnPruner\(Takezoeet al\.[2026](https://arxiv.org/html/2608.06411#bib.bib96)\)removes visual redundancy using learned importance and diversity, then prunes query\-irrelevant tokens based on middle\-layer text\-to\-vision attention\.

MethodAI2DMMEMMBENMMBCNVQATGQARel\.↑\\uparrow\\rowcolorgray\!12All 1280 Tokens\(100\.0%\)Vanilla83\.2232583\.780\.583\.061\.1100\.0%\\rowcolorgray\!12Retain 512 Tokens\(↓\\downarrow60%\)FastV78\.8231781\.579\.181\.458\.497\.3%VisionZip82\.7233781\.879\.780\.860\.198\.7%MAP82\.8235082\.979\.882\.260\.899\.6%\\rowcolorgray\!12Retain 256 Tokens\(↓\\downarrow80%\)FastV76\.2223878\.874\.476\.253\.692\.3%VisionZip79\.3220979\.776\.674\.257\.394\.0%MAP81\.0234882\.278\.079\.759\.997\.9%\\rowcolorgray\!12Retain 128 Tokens\(↓\\downarrow90%\)FastV68\.2174569\.565\.358\.352\.179\.4%VisionZip72\.5216876\.074\.560\.454\.087\.5%MAP79\.4229880\.376\.375\.158\.695\.2%
Table 3:Performance comparison on Qwen2\.5\-VL\-7B\.### Main Results\.

Tables[Method](https://arxiv.org/html/2608.06411#Sx4)and[Experiments](https://arxiv.org/html/2608.06411#Sx5)report comparisons on LLaVA\-1\.5\-7B and LLaVA\-NeXT\-7B, respectively\. MAP achieves the best average performance across all settings on both backbones\. When retaining only22\.2%,11\.1%, and5\.6%of the original visual tokens, MAP outperforms the strongest baselines by1\.1,1\.2, and1\.8points on LLaVA\-1\.5\-7B, and by2\.1,1\.1, and3\.2points on LLaVA\-NeXT\-7B\. The advantage becomes pronounced as the retention ratio decreases, indicating that MAP preserves question\-relevant visual evidence more effectively under aggressive pruning\. Its consistent gains on LLaVA\-NeXT\-7B demonstrate its effectiveness on longer visual sequences, where token selection is more challenging\.

We further evaluate MAP on Qwen2\.5\-VL\-7B\-Instruct in Table[Experiments](https://arxiv.org/html/2608.06411#Sx5)\. MAP consistently outperforms existing methods across all token retention ratios, improving the average relative score over the strongest baseline by0\.9points with 512 tokens,3\.9points with 256 tokens, and a substantially larger7\.7points with 128 tokens\. Notably, when restricted to only 128 tokens, existing methods degrade sharply, whereas MAP maintains an average relative score of95\.2%\. These results demonstrate that MAP better preserves question\-relevant visual evidence at high compression ratios\.

### Ablation of Teacher Selection for Direct Pruning

MethodCov\.↑\\uparrowROI\-NSS↑\\uparrowLayer Dist\.↓\\downarrowHit@1↑\\uparrowHit@3↑\\uparrowHit@5↑\\uparrowROI\-Max66\.2%2\.7280\.0100\.0%100\.0%100\.0%Random56\.5%1\.2106\.55\.8%16\.0%26\.3%Layer 1461\.3%2\.0564\.622\.8%42\.5%52\.7%QCTS64\.1%2\.4213\.347\.8%69\.8%79\.8%Table 4:Comparison of layer\-selection strategies\.QCTS is designed to identify a teacher layer whose attention best captures the visual regions relevant to each question\. To assess this ability, we use the same 1,000 image–question samples from the preceding analysis\. ROI\-Max selects the layer with the highest ROI\-NSS for each question\. Random uniformly selects one layer from Layers 8–24 for each question, while Layer 14 is the best\-performing fixed baseline\. For each method, we retain the top 128 visual tokens according to the attention scores from its identified layer\. Macro Coverage measures the average target\-region token coverage, ROI\-NSS measures the concentration of attention within the target region, Layer Dist\. and Hit@KKmeasure how closely the identified layer matches ROI\-Max\. As shown in Table[4](https://arxiv.org/html/2608.06411#Sx5.T4), QCTS consistently outperforms both baselines, achieving 64\.1% Macro Coverage, an ROI\-NSS of 2\.421, and a Layer Dist\. of 3\.3\. Its Hit@1/3/5 rates reach 47\.8%, 69\.8%, and 79\.8%, respectively\. The coverage and ROI\-NSS demonstrate that QCTS preserves question\-relevant visual information, while the low Layer Dist\. and high Hit@KKrates show that it reliably identifies layers close to ROI\-Max\.

We further evaluate whether the layers selected by QCTS provide effective signal for visual token pruning\. Figure[5](https://arxiv.org/html/2608.06411#Sx5.F5)compares it with attention from each fixed layer when only the top 11\.1% of visual tokens ranked by text\-to\-vision attention are retained\. The y\-axis represents the average relative performance across the six benchmarks in Table[Ablation of Teacher Selection for Predictor Training](https://arxiv.org/html/2608.06411#Sx5.SSx3)\. QCTS achieves average relative performance of 97\.26% and 98\.70% on LLaVA\-1\.5\-7B and LLaVA\-NeXT\-7B, exceeding the strongest fixed\-layer results by 0\.75 and 0\.99 percentage points, respectively\. These results show that attention from the layers identified by QCTS is more effective for visual token pruning than attention from any fixed layer\.

![Refer to caption](https://arxiv.org/html/2608.06411v1/x8.png)\(a\)LLaVA\-1\.5\-7B
![Refer to caption](https://arxiv.org/html/2608.06411v1/x9.png)\(b\)LLaVA\-NeXT\-7B

Figure 5:Pruning performance of different layers and QCTS\.### Ablation of Teacher Selection for Predictor Training

Teacher AttnGQATextVQAPOPEMMEMMBEN/CNRel\.Vanilla61\.958\.285\.9186264\.7/58\.3100\.0%\\rowcolorgray\!12Single\-layer teacher selectionFixed Layer 1158\.755\.987\.1175762\.5/56\.396\.6%Fixed Layer 1459\.056\.986\.8169862\.7/56\.296\.4%Fixed Layer 2458\.756\.686\.3174563\.1/55\.496\.5%QCTS59\.457\.087\.1174463\.2/57\.097\.4%\\rowcolorgray\!12Multi\-layer teacher fusionQCTS\-258\.856\.887\.1173562\.6/56\.396\.7%QCTS\-458\.956\.786\.9173763\.0/56\.496\.9%QCTS\-859\.156\.587\.3173963\.2/56\.096\.9%QCTS\-1658\.556\.387\.2172662\.5/55\.996\.3%

Table 5:Performance of predictors trained with different teacher\-selection strategies on LLaVA\-1\.5\-7B at 64 tokens\.We investigate the effect of teacher layer selection by training predictors with different strategies and comparing their pruning performance\. A fixed\-layer teacher uses attention from the same predefined layer as supervision for all training samples\. We report Layers 11, 14, and 24, as they are the three best performing fixed layer teachers\. As shown in Table[Ablation of Teacher Selection for Predictor Training](https://arxiv.org/html/2608.06411#Sx5.SSx3), QCTS achieves the best overall pruning performance, with a relative score of 97\.4%, surpassing the strongest fixed\-layer teacher by 0\.8 points\. This improvement suggests that QCTS can adaptively select a more suitable teacher layer for each sample, thereby providing more effective supervision for predictor training\. We further evaluate QCTS\-nn, which averages the attention from the topnnlayers selected by QCTS for each sample as teacher supervision\. However, multi\-layer fusion provides no additional gains and instead leads to a slight decrease in pruning performance\. These results show that QCTS outperforms both fixed\-layer selection and multi\-layer teacher fusion\. We therefore use QCTS to select the teacher layer for training the predictor in all experiments\.

SelectionTokensGQATextVQAPOPEMMEMMBEN/CNRel\.↑\\uparrowTop\-KK12860\.057\.387\.1178963\.3/56\.898\.0%\+Diversity12860\.157\.787\.3180063\.1/57\.098\.3%Top\-KK6459\.056\.487\.1172163\.7/56\.496\.9%\+Diversity6459\.457\.087\.1174463\.2/57\.097\.4%Top\-KK3257\.154\.387\.4164262\.8/53\.994\.2%\+Diversity3258\.155\.387\.4172662\.2/54\.695\.5%

Table 6:Effect of diversity\-aware selection with different numbers of retained visual tokens on LLaVA\-1\.5\-7B\.
### Effect of Diversity\-Aware Token Selection

Table[6](https://arxiv.org/html/2608.06411#Sx5.T6)studies the contribution of diversity\-aware selection under different token budgets\. The Top\-KKbaseline directly retains theKKtokens with the highest predicted attention scores, whereas the diversity\-aware variant additionally considers feature complementarity\. At 128 tokens, diversity\-aware selection provides only a marginal improvement, from 98\.0% to 98\.3%\. This result indicates that the predicted importance scores accurately identify the question\-relevant tokens and already cover the key visual evidence under a sufficient token budget, leaving limited room for diversity to help\. As the budget decreases, however, the improvement grows from 96\.9% to 97\.4% at 64 tokens and from 94\.2% to 95\.5% at 32 tokens\. Under aggressive pruning, redundancy among highly ranked tokens becomes more costly\. Diversity\-aware selection mitigates this issue by preserving complementary visual features, leading to larger gains at smaller token budgets\.

### Efficiency Analysis

MethodTokensPrefill \(ms\)End\-to\-End \(ms\)KV Cache \(MB\)\\rowcolorgray\!12LLaVA\-1\.5\-7BVanilla57658\.9101\.5336MAP3221\.5 \(2\.73×\\times\)67\.2 \(1\.51×\\times\)64 \(\-80\.95%\)\\rowcolorgray\!12LLaVA\-NeXT\-7BVanilla2880224\.5277\.91488MAP16030\.2 \(7\.44×\\times\)89\.8 \(3\.09×\\times\)128 \(\-91\.39%\)
Table 7:Average inference efficiency across ten benchmarks\.Table[Efficiency Analysis](https://arxiv.org/html/2608.06411#Sx5.SSx5)reports average efficiency across ten benchmarks\. MAP achieves up to 2\.73×\\timesprefill speedup and 80\.95% KV cache reduction on LLaVA\-1\.5\-7B, and 7\.44×\\timesprefill and 3\.09×\\timesend\-to\-end speedups with 91\.39% KV cache reduction on LLaVA\-NeXT\-7B\. Detailed results for all ten benchmarks and a runtime breakdown are provided in the appendix\.

## Conclusion

In this work, we show that the middle layer most responsive to the question varies across samples and its attention is costly to obtain\. MAP addresses both issues by using QCTS to select a sample\-specific teacher and distilling its attention into a lightweight predictor\. At inference, MAP combines predicted importance with feature diversity to prune visual tokens before the first LLM layer, without requiring attention maps\. Experiments on three MLLM backbones demonstrate the favorable accuracy–efficiency trade\-off of MAP\.

## References

- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds,et al\.\(2022\)Flamingo: a visual language model for few\-shot learning\.InNeurIPS,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025a\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang,et al\.\(2025b\)Qwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- An image is worth 1/2 tokens after layer 2: plug\-and\-play inference acceleration for large vision\-language models\.InECCV,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1),[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- Z\. Chen, W\. Wang, H\. Tian, S\. Ye, Z\. Gao, E\. Cui, W\. Tong, K\. Hu, J\. Luo, Z\. Ma,et al\.\(2024b\)How far are we to gpt\-4v? closing the gap to commercial multimodal models with open\-source suites\.arXiv preprint arXiv:2404\.16821\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1)\.
- W\. Dai, J\. Li, D\. Li, A\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. N\. Fung, and S\. Hoi \(2023\)Instructblip: towards general\-purpose vision\-language models with instruction tuning\.InNeurIPS,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)Flashattention: fast and memory\-efficient exact attention with io\-awareness\.NeurIPS\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p6.1)\.
- S\. Dong, J\. Hu, M\. Zhang, M\. Yin, Y\. Fu, and Q\. Qian \(2025\)Mmtok: multimodal coverage maximization for efficient inference of vlms\.arXiv preprint arXiv:2508\.18264\.Cited by:[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- C\. Fu, P\. Chen, Y\. Shen, Y\. Qin, M\. Zhang, X\. Lin, J\. Yang, X\. Zheng, K\. Li, X\. Sun,et al\.\(2023\)Mme: a comprehensive evaluation benchmark for multimodal large language models\.arXiv preprint arXiv:2306\.13394\.Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. Parikh \(2017\)Making the v in vqa matter: elevating the role of image understanding in visual question answering\.InCVPR,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- D\. Guo, F\. Wu, F\. Zhu, F\. Leng, G\. Shi, H\. Chen, H\. Fan, J\. Wang, J\. Jiang, J\. Wang,et al\.\(2025\)Seed1\. 5\-vl technical report\.arXiv preprint arXiv:2505\.07062\.Cited by:[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- D\. Gurari, Q\. Li, A\. J\. Stangl, A\. Guo, C\. Lin, K\. Grauman, J\. Luo, and J\. P\. Bigham \(2018\)Vizwiz grand challenge: answering visual questions from blind people\.InCVPR,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)Gqa: a new dataset for real\-world visual reasoning and compositional question answering\.InCVPR,Cited by:[Question\-Relevant Attention Across Layers](https://arxiv.org/html/2608.06411#Sx3.SSx1.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- A\. Kembhavi, M\. Salvato, E\. Kolve, M\. Seo, H\. Hajishirzi, and A\. Farhadi \(2016\)A diagram is worth a dozen images\.InECCV,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- Y\. Kim, Y\. Zhang, H\. Liu, A\. Jung, S\. Lee, and S\. Hong \(2026\)ZOO\-prune: training\-free token pruning via zeroth\-order gradient estimation in vision\-language models\.InCVPR,Cited by:[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, P\. Zhang, Y\. Li, Z\. Liu,et al\.\(2024\)Llava\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- B\. Li, R\. Wang, G\. Wang, Y\. Ge, Y\. Ge, and Y\. Shan \(2023a\)Seed\-bench: benchmarking multimodal llms with generative comprehension\.arXiv preprint arXiv:2307\.16125\.Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- J\. Li, D\. Li, S\. Savarese, and S\. Hoi \(2023b\)Blip\-2: bootstrapping language\-image pre\-training with frozen image encoders and large language models\.InICML,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, X\. Zhao, and J\. Wen \(2023c\)Evaluating object hallucination in large vision\-language models\.InEMNLP,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024a\)Improved baselines with visual instruction tuning\.InCVPR,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. Lee \(2024b\)Llava\-next: improved reasoning, ocr, and world knowledge\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.InNeurIPS,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.
- T\. Liu, L\. Shi, R\. Hong, Y\. Hu, Q\. Yin, and L\. Zhang \(2024c\)Multi\-stage vision token dropping: towards efficient multimodal large language model\.arXiv preprint arXiv:2411\.10803\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1)\.
- Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.\(2024d\)Mmbench: is your multi\-modal model an all\-around player?\.InECCV,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan \(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InNeurIPS,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- A\. Singh, V\. Natarajan, M\. Shah, Y\. Jiang, X\. Chen, D\. Batra, D\. Parikh, and M\. Rohrbach \(2019\)Towards vqa models that can read\.InCVPR,Cited by:[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.17)\.
- R\. Takezoe, Y\. Li, Z\. Bo, A\. Hou, M\. Guang, and K\. Long \(2026\)LearnPruner: rethinking attention\-based token pruning in vision language models\.arXiv preprint arXiv:2604\.23950\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1),[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- Z\. Wen, Y\. Gao, S\. Wang, J\. Zhang, Q\. Zhang, W\. Li, C\. He, and L\. Zhang \(2025\)Stop looking for “important tokens” in multimodal language models: duplication matters more\.InEMNLP,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1),[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- Z\. Wu, X\. Chen, Z\. Pan, X\. Liu, W\. Liu, D\. Dai, H\. Gao, Y\. Ma, C\. Wu, B\. Wang,et al\.\(2024\)Deepseek\-vl2: mixture\-of\-experts vision\-language models for advanced multimodal understanding\.arXiv preprint arXiv:2412\.10302\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1)\.
- S\. Yang, Y\. Chen, Z\. Tian, C\. Wang, J\. Li, B\. Yu, and J\. Jia \(2025\)Visionzip: longer is better but not necessary in vision language models\.InCVPR,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- W\. Ye, Q\. Wu, W\. Lin, and Y\. Zhou \(2025\)Fit and prune: fast and training\-free visual token pruning for multi\-modal large language models\.InAAAI,Cited by:[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1)\.
- Q\. Zhang, A\. Cheng, M\. Lu, R\. Zhang, Z\. Zhuo, J\. Cao, S\. Guo, Q\. She, and S\. Zhang \(2025a\)Beyond text\-visual attention: exploiting visual cues for effective token pruning in vlms\.InICCV,Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1),[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1)\.
- Q\. Zhang, M\. Liu, L\. Li, M\. Lu, Y\. Zhang, J\. Pan, Q\. She, and S\. Zhang \(2025b\)Beyond attention or similarity: maximizing conditional diversity for token pruning in mllms\.InNeurIPS,Cited by:[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1),[2Performance comparison on LLaVA\-NeXT\-7B with different numbers of retained visual tokens\.](https://arxiv.org/html/2608.06411#Sx5.16.18)\.
- Y\. Zhang, C\. Fan, J\. Ma, W\. Zheng, T\. Huang, K\. Cheng, D\. Gudovskiy, T\. Okuno, Y\. Nakata, K\. Keutzer,et al\.\(2024\)Sparsevlm: visual token sparsification for efficient vision\-language model inference\.arXiv preprint arXiv:2410\.04417\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p2.1),[Visual Token Pruning in MLLMs](https://arxiv.org/html/2608.06411#Sx2.SSx2.p1.1)\.
- J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao,et al\.\(2025\)Internvl3: exploring advanced training and test\-time recipes for open\-source multimodal models\.arXiv preprint arXiv:2504\.10479\.Cited by:[Introduction](https://arxiv.org/html/2608.06411#Sx1.p1.1),[Multimodal Large Language Models](https://arxiv.org/html/2608.06411#Sx2.SSx1.p1.1)\.

相似文章

使用ART微调多模态大语言模型：基于艺术强化训练

Hugging Face Daily Papers

ART（基于艺术强化训练）通过梯度反向传播优化原始视觉输入，实现对冻结的多模态大语言模型的参数高效微调，其性能与LoRA相当，同时支持为vLLM等高吞吐引擎预编译的计算图。

PreDiff-LM: 预训练离散掩码扩散语言建模与混合注意力

arXiv cs.AI

PreDiff-LM 提出了一种混合注意力机制，该机制对提示令牌保持因果注意力，对掩码目标令牌使用双向注意力，从而使得预训练自回归模型能够适应离散掩码扩散语言建模，在困惑度和下游任务上相较于先前的扩散基线取得了改进。

逐令牌关注多模态生成

Hugging Face Daily Papers

本文研究了多模态大语言模型在生成过程中的令牌级注意力转移，揭示了其中的一致模式，并提出了一种简单的测试时干预方法，显著提升了任务性能。

基于查询的跨模态投影器增强 Mamba 多模态大语言模型

arXiv cs.CL

本文提出了一种基于查询的跨模态投影器，通过交叉注意力机制对视觉标记进行压缩，以提升基于 Mamba 的多模态大语言模型的性能。该方法在视觉语言基准测试中同时提高了模型性能和吞吐量，并消除了手动设计二维扫描顺序的需求。