MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models

# MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Models
Source: [https://arxiv.org/html/2607.22586](https://arxiv.org/html/2607.22586)
Jinsong Shu1,,Chenyang Wu1,11footnotemark:1,Zhongle Xie1,✉,Baokun Wang2,Lidan Shou3,4 1Zhejiang University2Ant Group 3The State Key Laboratory of Blockchain and Data Security, Zhejiang University 4Hangzhou High\-Tech Zone \(Binjiang\) Institute of Blockchain and Data Security

###### Abstract

Key\-Value \(KV\) caching is essential for efficient inference in multimodal large language models \(MLLMs\), yet its memory footprint grows linearly with context length and becomes a major bottleneck due to the large number of visual tokens\. Recent prefill\-stage KV selection methods estimate KV importance from prefilling statistics, implicitly assuming that prefilling\-time queries are representative of those encountered during decoding\. We show that this assumption breaks down in multimodal inference, where decoding\-time queries exhibit substantially larger variance than prefilling\-stage representations, leading to unstable KV importance estimation under tight cache budgets\. As a result, small ranking errors can disproportionately discard semantically critical visual tokens and degrade grounding and reasoning performance\. We proposeMM\-ShiftKV, a training\-free, decode\-aware and strictly prefill\-only KV selection method\. MM\-ShiftKV approximates decoding\-time query behavior during prefilling by constructing variance\-expanded*query proxies*and estimates prompt KV importance based on their aggregated attention mass\. Experiments on multimodal benchmarks demonstrate that MM\-ShiftKV consistently outperforms existing methods under strict KV\-cache budgets\. Our code is available at[https://github\.com/zjuDBxAI/MM\-ShiftKV](https://github.com/zjuDBxAI/MM-ShiftKV)\.

MM\-ShiftKV: Decode\-Aware Prefill\-Stage KV Selection for Multimodal Large Language Models

![Refer to caption](https://arxiv.org/html/2607.22586v1/x1.png)\(a\)Feature mean/std \(OCRBench\)
![Refer to caption](https://arxiv.org/html/2607.22586v1/x2.png)\(b\)Attention mass coverage

Figure 1:Prefill–decode statistics and decode\-time attention coverage\.\(a\)Layer\-wise mean and variance of hidden\-state representations during prefill and decoding on OCRBench\.\(b\)Attention mass coverage \(retained prompt\), measured as the fraction of decode\-time attention probability mass assigned to prompt KV tokens retained after prefilling\.## 1Introduction

Multimodal large language models \(MLLMs\) extend text\-only language models with the ability to generate language grounded in visual inputs, enabling applications such as optical character recognition \(OCR\), document understanding, and visual question answering\(Li et al\.,[2024a](https://arxiv.org/html/2607.22586#bib.bib16); Bai et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib3)\)\. During inference, these models process short textual prompts together with high\-resolution visual inputs, which are encoded during prefilling into long multimodal sequences dominated by visual tokens\(Arif et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib2)\), followed by autoregressive decoding to generate textual outputs\. Efficient decoding relies on*Key–Value \(KV\) caching*, whose memory footprint grows linearly with the encoded sequence length, making KV cache size and access cost a primary bottleneck for memory consumption and decoding efficiency\.

To mitigate this bottleneck, recent work has proposed*prefill\-stage KV cache selection*, which retains a subset of KV states after prefilling and reuses them during decoding\(Xiao et al\.,[2023](https://arxiv.org/html/2607.22586#bib.bib36); Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17); Devoto et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib9); Park et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib25)\)\. Compared to decoding\-time cache eviction or adaptive cache compression\(Xiao et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib35)\), these prefill\-only approaches are attractive because they are training\-free\(Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17)\), introduce no decoding\-time intervention, and remain compatible with advanced attention kernels such as FlashAttention\(Dao,[2023](https://arxiv.org/html/2607.22586#bib.bib8)\)\. Most methods estimate KV importance from statistics observed during prefilling, implicitly assuming that prefill\-stage representations and attention behavior are representative of those encountered during decoding\.

This implicit assumption becomes fragile in multimodal inference due to the heterogeneity of multimodal inputs\. A large number of visually redundant tokens coexist with a small subset of semantically critical tokens\(Tao et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib31); Chen et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib6)\), such that small errors in KV ranking may disproportionately remove critical representations\. As a result, existing prefill\-stage KV selection methods often lead to degraded language grounding, unstable reasoning, and significant performance drops on multimodal tasks\(Devoto et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib9); Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17); Park et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib25); Devoto et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib10)\)\.

At a more fundamental level, prefilling and decoding correspond to distinct functional stages of multimodal inference\. Prefilling primarily emphasizes visual perception and cross\-modal alignment, whereas decoding shifts toward language generation and reasoning conditioned on previously generated tokens\. Although the two stages share identical model parameters, they may induce different distributions of hidden states and attention queries, a phenomenon that we systematically analyze in Section[2](https://arxiv.org/html/2607.22586#S2)\. Consequently, importance estimates derived solely from prefilling\-stage statistics can be misaligned with decoding\-time behavior, leading to unreliable KV selection under strict cache budgets\.

In this work, we proposeMM\-ShiftKV, a training\-free and strictly prefill\-only KV selection framework for multimodal inference\. Our core idea is to make prefill\-stage KV selection explicitly*decode\-aware*by calibrating importance estimates to reflect the distributional properties of decoding\-time queries, rather than relying solely on prefill\-stage statistics\. Decode\-aware here does not imply performing KV eviction or re\-ranking during decoding, but instead adjusts prefill\-based query proxies to better approximate decoding\-time behavior\. Concretely, MM\-ShiftKV performs one\-shot KV selection at the end of prefilling by sampling variance\-expanded query proxies from statistics computed over the current input and estimating KV importance based on aggregated attention mass\. The resulting compact prompt KV cache is reused unchanged during decoding, enabling efficient and robust multimodal inference under strict budgets\.

Contributions\.

- •We identify a persistent prefill–decode*scale mismatch*in multimodal inference, where decoding\-time*hidden\-state representations*exhibit substantially larger variance than those observed during prefilling\.
- •We show that this mismatch causes prefill\-based query proxies to be under\-scaled, leading to distorted query–key relevance estimation and unstable KV selection under strict KV\-cache budgets\.
- •We proposeMM\-ShiftKV, a training\-free and strictly*prefill\-only*KV selection framework that constructs variance\-expanded, decode\-aware*query proxies*to estimate prompt KV importance\.
- •We demonstrate improved accuracy–memory–latency trade\-offs on representative OCR, grounding, and long\-context VQA benchmarks under tight KV\-cache constraints\.

## 2Observation: Prefill–Decode Scale Mismatch in Multimodal Inference

Recent work has shown that activation statistics in large language models exhibit structured properties that can be exploited in a training\-free mannerLiu et al\. \([2024a](https://arxiv.org/html/2607.22586#bib.bib20)\)\. Inspired by the properties, most*prefill\-stage*KV selection methods assume that statistics observed during prefilling remain representative of KV usage during decoding\.

As discussed in Section[1](https://arxiv.org/html/2607.22586#S1), this assumption is critical for prefill\-stage KV selection, yet it has not been carefully examined in multimodal inference, and we provide additional theoretical analysis in Appendix[A](https://arxiv.org/html/2607.22586#A1)\. In this section, we show that the assumption is empirically violated and characterize a consistent*prefill\-decode statistical mismatch*onOCRBench\(Liu et al\.,[2023](https://arxiv.org/html/2607.22586#bib.bib21)\)andQwen2\.5\-VL\-7B\-Instruct\. Further results are provided in the Appendix[C](https://arxiv.org/html/2607.22586#A3)\. Specifically, we observe that hidden\-state representations during prefilling and decoding differ substantially in their statistical*scale*\. Figure[1\(a\)](https://arxiv.org/html/2607.22586#S0.F1.sf1)reports the layer\-wise mean and standard deviation of hidden features forQwen2\.5\-VL\-7B\-Instructevaluated onOCRBench\. Although prefilling and decoding share identical model parameters, decoding\-stage representations exhibit consistently larger variance across layers, while mean shifts remain relatively moderate\. This indicates that prefilling\-stage statistics systematically underestimate the scale of decoding\-time representations, a phenomenon that lacks dedicated research in prior work on prefill\-stage KV selection\.

This statistical mismatch has direct implications for the stability of prefill\-stage KV selection\. Existing methods typically rely on prefilling\-stage signals, such as local attention behavior, sequence\-level activation statistics, and KV similarity, to estimate the importance of the token\. When these signals are under\-scaled relative to true decoding\-time queries, the resulting importance estimates become distorted\. Figure[1\(b\)](https://arxiv.org/html/2607.22586#S0.F1.sf2)quantifies this effect using*attention mass coverage*, measured as the fraction of decoding\-time attention probability mass assigned to prompt KV tokens retained after prefilling\. Following the standard prefill\-only protocol, KV selection is performed once at the end of prefilling, and the compressed prompt KV cache is kept fixed during decoding\. Across methods, attention coverage is consistently reduced and exhibits high variance under tight cache budgets, indicating that KV sets selected from prefilling statistics fail to reliably capture decoding\-time KV usage\.

Taken together, these results demonstrate that prefill\-stage KV selection based solely on prefilling\-stage statistics is inherently brittle in multimodal settings\. The observed prefill\-decode statistical mismatch highlights the need to explicitly account for decoding\-time query behavior while remaining strictly within the prefill\-only regime\. This insight forms the basis for the decode\-aware prefill\-stage KV selection approach developed in the next section, which achieves the highest attention coverage shown in Figure[1\(b\)](https://arxiv.org/html/2607.22586#S0.F1.sf2)\.

![Refer to caption](https://arxiv.org/html/2607.22586v1/x3.png)Figure 2:Overview of MM\-ShiftKV\. The method computes statistics over the full prefill sequence of the current input and constructs variance\-expanded, decode\-aware*query proxies*during prefilling\. These query proxies are used to estimate prompt KV importance via attention\-mass aggregation and group\-wise voting with a last\-token anchor, yielding a compact KV cache under a fixed budget\.
## 3Method

##### Core Idea\.

MM\-ShiftKV addresses the instability of prefill\-stage KV selection in multimodal inference by explicitly approximating how prompt keys will be accessed by*future decoding queries*\. Since decoding\-time queries are unavailable during prefilling, the core idea is to construct a set of synthetic*query proxies*during the prefill stage that approximate the distributional properties of decoding queries\. Prompt keys that are consistently attended by these query proxies are more likely to be important during decoding and should therefore be retained under the limited KV\-cache budget\.

### 3\.1Problem Definition

Before describing our method, we formalize the prefill\-stage KV selection problem in multimodal inference\.

##### Input Sequence\.

Given a multimodal input sequence

X=\(x1,x2,…,xT\),X=\(x\_\{1\},x\_\{2\},\\ldots,x\_\{T\}\),\(1\)where tokens include both visual and textual modalities, andTTdenotes the sequence length of the*prefill stage*\.

##### Prompt KV cache\.

During prefilling, the model computes KV representations for all prompt tokens at each transformer layerℓ\\elland KV headhh\. The resulting set of KV pairs forms the*prompt KV cache*:

𝒞\(ℓ,h\)=\{\(kt\(ℓ,h\),vt\(ℓ,h\)\)\}t∈𝒯,\\mathcal\{C\}^\{\(\\ell,h\)\}=\\\{\(k\_\{t\}^\{\(\\ell,h\)\},v\_\{t\}^\{\(\\ell,h\)\}\)\\\}\_\{t\\in\\mathcal\{T\}\},\(2\)where𝒯=\{1,…,T\}\\mathcal\{T\}=\\\{1,\\ldots,T\\\}indexes prompt tokens\. These prompt KVs are repeatedly accessed during decoding and dominate memory consumption and attention computation in inference\.

##### Prefill\-stage KV selection\.

With a given cache budgetCℓ,hC\_\{\\ell,h\}for each layerℓ\\elland KV headhh, the objective of the prefill\-stage KV selection is to retain a subset

𝒞′\(ℓ,h\)⊆𝒞\(ℓ,h\),\|𝒞′\(ℓ,h\)\|≤Cℓ,h,\\mathcal\{C^\{\\prime\}\}^\{\(\\ell,h\)\}\\subseteq\\mathcal\{C\}^\{\(\\ell,h\)\},\\qquad\|\\mathcal\{C^\{\\prime\}\}^\{\(\\ell,h\)\}\|\\leq C\_\{\\ell,h\},\(3\)which is reused throughout the decoding to reduce memory usage and attention cost while preserving generation quality\.

In practice, the KV selection is performed independently for each layer and head, and executed once at the end of*prefilling*\. Note that the selected prompt KV cache remains fixed during decoding, while KV pairs from newly generated tokens are appended in a standard autoregressive manner\.

### 3\.2MM\-ShiftKV

Figure[2](https://arxiv.org/html/2607.22586#S2.F2)provides an overview of MM\-ShiftKV\. The method consists of three steps executed during prefilling: \(1\) constructing decode\-aware query proxies, \(2\) estimating prompt KV importance via mass\-based voting, and \(3\) performing budgeted KV selection based on aggregated importance scores\. All steps are strictly prefill\-only and introduce no decoding\-time intervention\. More implementation details can be found in Appendix[B](https://arxiv.org/html/2607.22586#A2)\.

#### 3\.2\.1Query Proxy Construction

To approximate decoding\-time query behavior, MM\-ShiftKV constructs a set of synthetic*query proxies*using prefilling\-stage statistics\.

##### Prefilling hidden\-state statistics\.

Leth\(ℓ\)∈ℝdh^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\}denote the hidden states produced at layerℓ\\ellduring prefilling\. We summarize these representations using element\-wise statistics:

μpre\(ℓ\)=mean​\(h\(ℓ\)\),σpre\(ℓ\)=std​\(h\(ℓ\)\),\\mu\_\{\\text\{pre\}\}^\{\(\\ell\)\}=\\mathrm\{mean\}\(h^\{\(\\ell\)\}\),\\qquad\\sigma\_\{\\text\{pre\}\}^\{\(\\ell\)\}=\\mathrm\{std\}\(h^\{\(\\ell\)\}\),\(4\)computed across prompt tokens for each feature dimension\. These statistics are computed separately for each input sample over its full prefill sequence, rather than shared across different samples\.

##### Variance\-expanded sampling\.

As shown in Section[2](https://arxiv.org/html/2607.22586#S2), decoding\-time queries exhibit substantially larger variance than prefilling\-stage representations\. To approximate this effect, we introduce a variance expansion factorγ\>1\\gamma\>1and sampleNN*proxy hidden states*\{H~i\(ℓ\)\}i=1N\\\{\\tilde\{H\}\_\{i\}^\{\(\\ell\)\}\\\}\_\{i=1\}^\{N\}from

H~i\(ℓ\)∼𝒩​\(μpre\(ℓ\),diag​\(\(γ​σpre\(ℓ\)\)2\)\)\.\\tilde\{H\}\_\{i\}^\{\(\\ell\)\}\\sim\\mathcal\{N\}\\\!\\left\(\\mu\_\{\\text\{pre\}\}^\{\(\\ell\)\},\\ \\mathrm\{diag\}\\\!\\left\(\(\\gamma\\sigma\_\{\\text\{pre\}\}^\{\(\\ell\)\}\)^\{2\}\\right\)\\right\)\.\(5\)

##### Query proxy projection and positioning\.

Each proxy hidden state is projected into the query space using the model’s query projection matrix:

q~i\(ℓ,h\)=WQ\(ℓ,h\)​H~i\(ℓ\),i=1,…,N\.\\tilde\{q\}\_\{i\}^\{\(\\ell,h\)\}=W\_\{Q\}^\{\(\\ell,h\)\}\\tilde\{H\}\_\{i\}^\{\(\\ell\)\},\\qquad i=1,\\ldots,N\.\(6\)The resulting vectors\{q~i\(ℓ,h\)\}\\\{\\tilde\{q\}\_\{i\}^\{\(\\ell,h\)\}\\\}are referred to as*query proxies*\. To reflect that decoding queries attend to prompt keys from future positions, query proxies are assigned synthetic future positions and encoded using the model’s rotary positional embedding \(RoPE\), while prompt keys retain their prefilling\-stage positional encodings\.

#### 3\.2\.2KV Importance Estimation

Given a set of query proxies, MM\-ShiftKV estimates the importance of each prompt key based on how consistently it attends across diverse proxies\.

##### Proxy\-induced attention\.

For each query proxyq~i\(ℓ,h\)\\tilde\{q\}\_\{i\}^\{\(\\ell,h\)\}, we compute its attention distribution over prompt keys:

at​\(q~i\(ℓ,h\)\)=softmax​\(\(q~i\(ℓ,h\)\)⊤​kt\(ℓ,h\)d\),t∈𝒯a\_\{t\}\\\!\\left\(\\tilde\{q\}\_\{i\}^\{\(\\ell,h\)\}\\right\)=\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\left\(\\tilde\{q\}\_\{i\}^\{\(\\ell,h\)\}\\right\)^\{\\top\}k\_\{t\}^\{\(\\ell,h\)\}\}\{\\sqrt\{d\}\}\\right\),t\\in\\mathcal\{T\}\(7\)

##### Mass aggregation and voting\.

TheNNquery proxies are partitioned intoGGdisjoint groups\{ℐg′\}g′=1G\\\{\\mathcal\{I\}\_\{g^\{\\prime\}\}\\\}\_\{g^\{\\prime\}=1\}^\{G\}\. For each groupg′g^\{\\prime\}, attention masses are aggregated as

a¯t\(g′\)=∑i∈ℐg′at​\(q~i\(ℓ,h\)\),t∈𝒯\\bar\{a\}\_\{t\}^\{\(g^\{\\prime\}\)\}=\\sum\_\{i\\in\\mathcal\{I\}\_\{g^\{\\prime\}\}\}a\_\{t\}\\\!\\left\(\\tilde\{q\}\_\{i\}^\{\(\\ell,h\)\}\\right\),\\quad t\\in\\mathcal\{T\}\(8\)Prompt keys are then ranked in descending order ofa¯t\(g′\)\\bar\{a\}\_\{t\}^\{\(g^\{\\prime\}\)\}, and we define𝒮g′⊆𝒯\\mathcal\{S\}\_\{g^\{\\prime\}\}\\subseteq\\mathcal\{T\}as the*smallest*set of keys whose cumulative mass satisfies

∑t∈𝒮g′a¯t\(g′\)≥τ​∑t∈𝒯a¯t\(g′\),\\sum\_\{t\\in\\mathcal\{S\}\_\{g^\{\\prime\}\}\}\\bar\{a\}\_\{t\}^\{\(g^\{\\prime\}\)\}\\;\\geq\\;\\tau\\sum\_\{t\\in\\mathcal\{T\}\}\\bar\{a\}\_\{t\}^\{\(g^\{\\prime\}\)\},\(9\)whereτ∈\(0,1\)\\tau\\in\(0,1\)is a predefined mass threshold\. Each selected key receives one vote, and the final importance score of keyttis given by

vote​\(t\)=∑g′=1G𝟏​\[t∈𝒮g′\]\.\\mathrm\{vote\}\(t\)=\\sum\_\{g^\{\\prime\}=1\}^\{G\}\\mathbf\{1\}\[t\\in\\mathcal\{S\}\_\{g^\{\\prime\}\}\]\.\(10\)

#### 3\.2\.3Budgeted KV Selection

KV selection is performed independently for each layerℓ\\elland KV headhh\. Among the prompt keys, we retain the most recent key to ensure decoding stability and select the remaining keys based on their aggregated importance scores\. To make ranking deterministic when vote counts tie, we use a last\-query anchor:

St\(ℓ,h\)=vote​\(t\)\+λ⋅at​\(qlast\(ℓ,h\)\),S\_\{t\}^\{\(\\ell,h\)\}=\\mathrm\{vote\}\(t\)\+\\lambda\\cdot a\_\{t\}\\\!\\left\(q\_\{\\mathrm\{last\}\}^\{\(\\ell,h\)\}\\right\),\(11\)whereat​\(qlast\(ℓ,h\)\)a\_\{t\}\(q\_\{\\mathrm\{last\}\}^\{\(\\ell,h\)\}\)is the attention score from the last real prefill query to keykt\(ℓ,h\)k\_\{t\}^\{\(\\ell,h\)\}, andλ\\lambdais a small constant \(λ=1\\lambda\{=\}1by default\)\. The ranking follows a lexicographical\-like priority: the discrete vote count is primary, while the anchor term is secondary\. Because the anchor weight does not exceed one integer vote step, it only acts as a tie\-breaker and does not override proxy voting\. Specifically, the topCℓ,h−1C\_\{\\ell,h\}\-1keys ranked bySt\(ℓ,h\)S\_\{t\}^\{\(\\ell,h\)\}are used to form the compressed prompt KV cache, and the most recent token is always kept\.

##### Overhead\.

Similar to existing prefill\-only approaches, MM\-ShiftKV introduces additional computation only during the*prefill stage*\. For each layer and KV head, it computes attention betweenNNquery proxies and\|𝒯\|\|\\mathcal\{T\}\|prompt keys once, followed by group\-wise aggregation\. The time complexity scales asO​\(N​\|𝒯\|\)O\(N\|\\mathcal\{T\}\|\), while the peak additional memory can be bounded byO​\(\|𝒯\|\)O\(\|\\mathcal\{T\}\|\)using streaming implementations\.

Table 1:Results on multimodal benchmarks under different per\-head KV\-cache budgets \(64/128/256/512\) usingQwen2\.5\-VL\-7B\-Instruct\. FullKV denotes standard inference without KV cache compression\. Avg is the arithmetic mean over DocVQA, OCRBench, TextVQA, ChartQA, TextCaps, and MMMU\.Table 2:Results on multimodal benchmarks under different per\-head KV\-cache budgets \(64/128/256/512\) usingLLaVA\-v1\.6\-Vicuna\-7B\. FullKV denotes standard inference without KV cache compression\. Avg is the arithmetic mean over DocVQA, OCRBench, TextVQA, ChartQA, TextCaps, and MMMU\.Table 3:Results on multimodal benchmarks with different KV\-cache*budget allocation*methods under various per\-head KV\-cache budgets \(64/128/256/512\) usingLLaVA\-v1\.6\-Vicuna\-7B\.\+oursapplies MM\-ShiftKV as a prefill\-only KV selection module on top of the original allocation strategy, without changing its budget\. Avg is the arithmetic mean over DocVQA, OCRBench, TextVQA, ChartQA, TextCaps, and MMMU\.

## 4Experiments

We conduct extensive experiments to evaluateMM\-ShiftKVunder memory\-constrained multimodal inference\. Following prior work on prompt KV cache compression \(e\.g\., SnapKV\(Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17)\), ExpectedAttn\(Devoto et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib9)\), and KEYDIFF\(Park et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib25)\)\), we focus on a*one\-shot prefilling*setting, where the KV cache is compressed once after prompt encoding and kept fixed throughout decoding\.

Our experiments aim to answer the following questions: \(i\) how MM\-ShiftKV compares with existing*prefill\-stage*KV compression baselines under strict KV\-cache budgets, \(ii\) how performance degrades as the cache budget decreases, and \(iii\) what accuracy\-memory\-latency trade\-offs can be achieved while remaining compatible with FlashAttention\(Dao,[2023](https://arxiv.org/html/2607.22586#bib.bib8)\)\-style decoding kernels\.

### 4\.1Experimental Setup

##### Models, Datasets, and Metrics\.

We evaluate MM\-ShiftKV on two representative multimodal large language models,Qwen2\.5\-VL\-7B\-Instruct\(Bai et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib3)\)andLLaVA\-v1\.6\-Vicuna\-7B\(Li et al\.,[2024a](https://arxiv.org/html/2607.22586#bib.bib16)\)\. Experiments are conducted on a diverse suite of multimodal benchmarks covering document understanding, OCR\-centric visual question answering, chart reasoning, and image captioning\. We employOCRBench\(Liu et al\.,[2023](https://arxiv.org/html/2607.22586#bib.bib21)\)using exact\-match accuracy to measure precise text recognition andDocVQA\(Mathew et al\.,[2021](https://arxiv.org/html/2607.22586#bib.bib24)\)using Average Normalized Levenshtein Similarity \(ANLS\) as the metric\. For chart reasoning and visual question answering, we use exact\-match accuracy onChartQA\(Masry et al\.,[2022](https://arxiv.org/html/2607.22586#bib.bib23)\),TextVQA\(Singh et al\.,[2019](https://arxiv.org/html/2607.22586#bib.bib29)\), andMMMU\(Yue et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib40)\)to measure the fraction of correctly answered questions\. For image captioning, we evaluateTextCaps\(Sidorov et al\.,[2020](https://arxiv.org/html/2607.22586#bib.bib28)\)using caption quality consensus with reference captions \(CIDEr\) as the metric\. The average number of input tokens for each benchmark is reported in Table[4](https://arxiv.org/html/2607.22586#S4.T4)\. Consistent with the observations inWang et al\. \([2025](https://arxiv.org/html/2607.22586#bib.bib34)\), we find that while text instructions remain concise, the inclusion of high\-resolution visual tokens significantly expands the sequence length, posing a primary bottleneck for KV cache memory\. For all metrics, higher values indicate better performance\.

##### Baselines\.

We compare against eviction\-free inference \(Full KV\) and strong prefilling\-stage KV compression baselines, includingSnapKV\(Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17)\),ExpectedAttn\(Devoto et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib9)\),StreamingLLM\(Xiao et al\.,[2023](https://arxiv.org/html/2607.22586#bib.bib36)\), andKEYDIFF\(Park et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib25)\)\(one\-shot variant\)\. All baselines are evaluated under identical KV\-cache budgets and decoding settings, using their recommended configurations\. More details are provided in the Appendix[F](https://arxiv.org/html/2607.22586#A6)\.

##### Prefill\-Only Evaluation Protocol and KV Budget\.

Following prior work , we adopt a one\-shot prefilling protocol\. Given an input prompt consisting of both visual and textual tokens, we first perform standard prefilling to compute the full KV cache\. Each method is then applied once at the end of prefilling to score and select a subset of KV pairs according to its selection policy, producing a compressed prompt KV cache that is kept fixed throughout decoding\. No decode\-time eviction, re\-ranking, or decoding\-time statistics are used\.

KV\-cache budgets are defined at the level of individual KV heads for each transformer layer, with per\-head budgetsC∈\{64,128,256,512\}C\\in\\\{64,128,256,512\\\}\. Unless otherwise specified, the same budget is applied uniformly across all layers and all KV heads\. For models with Grouped\-Query Attention \(GQA\)\(Ainslie et al\.,[2023](https://arxiv.org/html/2607.22586#bib.bib1)\)or Multi\-Query Attention \(MQA\)\(Komatsuzaki et al\.,[2022](https://arxiv.org/html/2607.22586#bib.bib15)\), where multiple query heads share one KV head, attention statistics from the corresponding query heads are aggregated, and the budget is applied at the head level\.

##### Implementation and Hyperparameters\.

Unless otherwise specified, we useN=512N\{=\}512decode\-aware query proxies, partitioned intoG=32G\{=\}32groups of sizeg=16g\{=\}16\(N=G​gN=Gg\), with variance expansion factorγ=10\\gamma\{=\}10, attention mass thresholdτ=0\.95\\tau\{=\}0\.95, and last\-token anchor weightλ=1\\lambda\{=\}1\. All hyperparameters are fixed globally and shared across models, datasets, and budgets\. We apply greedy decoding with a maximum generation length of 64\. All experiments are conducted on NVIDIA H100 80GB GPUs using CUDA 12\.8 and FlashAttention 2\.4\.1 with mixed\-precision \(fp16/bf16\)\. Additional empirical analysis are detailed in the Appendix[D](https://arxiv.org/html/2607.22586#A4)\. All results are averaged over 3 independent runs with different random seeds\.

Table 4:Average number of input tokens across benchmarks\. Text instructions are short, and visual tokens constitute the majority of the input sequence\.
##### Results on multimodal benchmarks\.

We evaluateMM\-ShiftKVon a diverse set of multimodal benchmarks under different per\-head KV\-cache budgets\. As shown in Table[1](https://arxiv.org/html/2607.22586#S3.T1)and Table[2](https://arxiv.org/html/2607.22586#S3.T2), MM\-ShiftKV consistently outperforms existing prefill\-stage KV selection baselines across both backbone models \(Qwen2\.5\-VL and LLaVA\-v1\.6\) and all tasks, with the performance gap becoming more pronounced as the KV budget decreases\.

Under extreme compression \(64 tokens per KV head\), baseline methods often suffer from performance degradation, particularly on OCR\- and grounding\-centric tasks\. Compared to strong prefill\-only baselines, MM\-ShiftKV achieves relative accuracy improvements on the order of20%20\\%–30%30\\%in representative document understanding benchmarks\. When compared to decode\-agnostic or heuristic KV selection methods, relative performance gains can exceed50%50\\%, highlighting the brittleness of decode\-unaware KV management under multimodal inputs\.

Consistent trends are also observed on generation\-oriented tasks\. Under the same extreme budget, MM\-ShiftKV yields relative improvements of approximately40%40\\%or more in generation quality \(e\.g\., TextCaps\) compared to prefill\-only baselines, indicating substantially better preservation of visually grounded information required for coherent multimodal generation\.

In addition to standalone performance, MM\-ShiftKV remains complementary to KV budget allocation strategies\. As shown in Table[3](https://arxiv.org/html/2607.22586#S3.T3), integrating MM\-ShiftKV with representative budget allocation methods \(e\.g\., PyramidKV and AdaKV\) leads to consistent relative improvements, typically in the range of10%10\\%–20%20\\%under the most restrictive budgets\. These results indicate that correcting the prefill–decode scale mismatch improves KV*selection quality*independently of how KV budgets are allocated across layers or heads\.

Overall, the results demonstrate that MM\-ShiftKV provides a robust and effective prefill\-only solution for multimodal inference under strict KV\-cache constraints, offering favorable accuracy–memory trade\-offs while remaining compatible with different KV budgeting schemes\.

Table 5:Ablation study under a fixed per\-head KV cache budget\.Samp\.,Var\. Exp\., andGVotedenote query proxy sampling, variance expansion, and group\-wise voting, respectively\.

### 4\.2Ablation Study

We analyze the contribution of individual components inMM\-ShiftKVthrough ablation experiments under a fixed per\-head KV cache budget\. All ablation experiments are conducted using theQwen2\.5\-VL\-7B\-Instructmodel, following the same prefill\-only evaluation protocol as in the main experiments\. We start from a lightweight*LastAttn*baseline, which incorporates only the attention induced by the last prefill query, and progressively add decode\-aware query sampling, variance expansion, and group\-wise voting\. Results are reported onOCRBenchandTextCapsin Table[5](https://arxiv.org/html/2607.22586#S4.T5)\.

##### Overall Effect\.

As shown in Table[5](https://arxiv.org/html/2607.22586#S4.T5), the*LastAttn*baseline achieves 52\.3 OCR accuracy and 40\.8 CIDEr on TextCaps, providing a simple but stable prefill\-only reference\. Enabling decode\-aware query sampling \(\+ Samp\.\) consistently improves performance, increasing OCR accuracy to 54\.5 and TextCaps CIDEr to 45\.5\. This indicates that sampling query proxies aligned with the decoding stage provides a more informative estimate of future attention behavior than relying solely on prefilling\-stage statistics\.

Adding variance expansion \(\+ Var\. Exp\.\) yields further gains, improving performance to 59\.6 on OCRBench and 48\.6 CIDEr on TextCaps\. This suggests that query proxies derived directly from prefilling statistics are systematically under\-scaled, and that scale calibration is critical for capturing the variability of decoding\-time queries in multimodal inference\.

Finally, incorporating group\-wise voting \(\+ GVote\) achieves the best overall results, reaching 68\.3 OCR accuracy and 50\.4 CIDEr on TextCaps\. By aggregating attention mass across groups of query proxies, group\-wise voting effectively reduces estimation variance and stabilizes KV ranking under tight cache budgets\.

Overall, these results demonstrate that decode\-aware query sampling, variance expansion, and group\-wise voting are complementary components\. Starting from a simple last\-query attention anchor, progressively introducing decode\-aware calibration and variance reduction is necessary to fully realize the performance gains ofMM\-ShiftKVin prefill\-only multimodal inference\.

![Refer to caption](https://arxiv.org/html/2607.22586v1/x4.png)Figure 3:End\-to\-end decoding latency and peak GPU memory usage under increasing input lengths\. Bars denote per\-token decoding latency and lines indicate peak GPU memory usage\. All methods are evaluated with a fixed per\-head KV budget of 256 and a maximum output length of 100 tokens\.

### 4\.3Efficiency Evaluation

##### Setup\.

We evaluate inference efficiency under long\-context multimodal settings by varying the input length in\{2​K,4​K,8​K,16​K,32​K\}\\\{2\\mathrm\{K\},4\\mathrm\{K\},8\\mathrm\{K\},16\\mathrm\{K\},32\\mathrm\{K\}\\\}while fixing the output length to 100 tokens\. Following the main configuration, we apply a per\-head KV cache budget of 256 after prefilling\-stage compression and report per\-token decoding latency and peak GPU memory usage\. All experiments use FlashAttention\-compatible decoding kernels\.

##### Decoding Latency\.

As shown in Figure[3](https://arxiv.org/html/2607.22586#S4.F3), MM\-ShiftKV consistently reduces per\-token decoding latency compared to FullKV, with the gap widening as the input length increases\. While FullKV exhibits steadily growing latency, MM\-ShiftKV maintains an almost constant latency profile, achieving up to a1\.9×1\.9\\timesspeedup at 32K input length\.

##### Memory Cost\.

By bounding the prompt KV cache before decoding, MM\-ShiftKV also substantially reduces peak GPU memory usage\. At 32K input length, peak memory consumption is reduced from approximately 32\.9 GB to 18\.6 GB, corresponding to a reduction of about 43%\. Overall, MM\-ShiftKV offers a more favorable latency–memory trade\-off than existing prefilling\-stage compression baselines\.

## 5Related Work

Current research on KV cache compression can be broadly categorized into two types, static pruning\(Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17); Jiang et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib14); Devoto et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib10),[2025](https://arxiv.org/html/2607.22586#bib.bib9); Park et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib25)\)and dynamic eviction\(Chen et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib6); Child et al\.,[2019](https://arxiv.org/html/2607.22586#bib.bib7)\), based on the intervention stage\. Recently, hybrid KV cache compression strategies have also emerged to balance efficiency across different inference stages\(Zeng et al\.,[2026](https://arxiv.org/html/2607.22586#bib.bib41)\)\. Static pruning methods are primarily deployed during the prefill stage, where the importance of each KV pair is evaluated using predefined metrics, and lower\-scoring KV pairs are pruned to reduce the initial memory footprint\. Dynamic eviction methods continuously discard tokens during the decode stage to maintain a fixed\-size cache\. There are three paradigms for KV cache eviction: first, attention\-based eviction\(Li et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib17); Devoto et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib9)\); second, value\-based analysis\(Devoto et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib10); Park et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib25)\); third, heuristic\-based structural eviction\(Cai et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib5); Xiao et al\.,[2023](https://arxiv.org/html/2607.22586#bib.bib36)\)\.

A key limitation of these methods lies in their implicit assumption of “distributional homogeneity”\(Devoto et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib9); Cai et al\.,[2024a](https://arxiv.org/html/2607.22586#bib.bib4)\)\. However, MM\-ShiftKV finds that in multimodal scenarios, there is a significant numerical distribution shift between these two stages\. MM\-ShiftKV explicitly analyzes the differences in numerical behavior between the prefill and decode stages in multimodal contexts\. By introducing a group voting mechanism, MM\-ShiftKV achieves more robust and accurate KV selection\.

Additionally, comprehensive studies outline efficient inference bottlenecks in large vision\-language models\(Zhang et al\.,[2026](https://arxiv.org/html/2607.22586#bib.bib42)\)\. Broadly, research covers quantization, early exiting, and speculative decoding\(Lin et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib18); Elhoushi et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib12); Child et al\.,[2019](https://arxiv.org/html/2607.22586#bib.bib7); Liu et al\.,[2024b](https://arxiv.org/html/2607.22586#bib.bib22); Xu et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib37); Su et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib30); Ji et al\.,[2026](https://arxiv.org/html/2607.22586#bib.bib13)\), alongside hardware and system\-level optimizations\(Dao,[2023](https://arxiv.org/html/2607.22586#bib.bib8); Zheng et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib45); Zhou et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib46)\)\. In multimodal scenarios, specialized strategies\(Lin et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib19)\)include dynamic attention head allocation\(Wang et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib34); Yang et al\.,[2025a](https://arxiv.org/html/2607.22586#bib.bib38); Wan et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib33)\)and visual token pruning\(Shen et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib27); Yang et al\.,[2025b](https://arxiv.org/html/2607.22586#bib.bib39); Zhang et al\.,[2025](https://arxiv.org/html/2607.22586#bib.bib44)\)\. MM\-ShiftKV is complementary and seamlessly integrates with these approaches for enhanced performance\.

## 6Conclusion

We studied a previously overlooked aspect of multimodal inference: the systematic mismatch between prefill and decoding stages\. By making*prefill\-stage*KV selection decode\-aware, MM\-ShiftKV provides a simple, training\-free solution that aligns prompt KV management with decoding\-time query distributions\. Our results highlight decode\-aware KV management as a key design principle for scalable multimodal inference\.

## 7Ethical Considerations

All experiments in our work are conducted using open\-source datasets and models\. Our research is solely aimed at enabling efficient inference of multimodal large models \(MLLMs\), and does not involve any human subject, sensitive data, or commercial applications\.

## 8Limitations

While MM\-ShiftKV exhibits prominent advantages in decoding latency optimization and inference accuracy performance under resource\-constrained conditions, it still has certain limitations: First, the additional computation introduced by the voting scoring for KV eviction processing in the prefilling phase incurs a certain overhead\. Although this overhead is negligible compared to the overall inference latency, there remains room for further acceleration in the prefilling phase\. Second, MM\-ShiftKV performs KV eviction only in the prefilling phase\. Although the KV cache in the prefilling phase accounts for the largest proportion in most visual question answering tasks, this design poses certain challenges in some tasks that require long\-context reasoning\. Third, our method primarily focuses on image and video benchmarks \(which generate a large amount of KV cache\), but it may face new challenges for other tasks and modalities \(e\.g\., audio\-based multimodal large models or long\-context reasoning scenarios\)\.

Nevertheless, MM\-ShiftKV is compatible with multiple future extension directions, including combination with attention head budget allocation methods, integration of offloading techniques in the decoding phase, and joint application with other compression paradigms \(e\.g\., model quantization, speculative decoding, etc\.\)\. We believe these directions will further enhance the generality and scalability of MM\-ShiftKV\.

## Acknowledgments

This work was supported by the Pioneer R&D Program of Zhejiang Province \(No\. 2024C01021\) and the Zhejiang Province Leading Talent of Technological Innovation Program \(No\. 2023R5214\)\.

## References

- Ainslie et al\. \(2023\)Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\. 2023\.[GQA: Training generalized multi\-query transformer models from multi\-head checkpoints](https://arxiv.org/abs/2305.13245)\.*Preprint*, arXiv:2305\.13245\.
- Arif et al\. \(2025\)Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S\. Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji\. 2025\.HiRED: Attention\-guided token dropping for efficient inference of high\-resolution vision\-language models\.In*Proceedings of the Thirty\-Ninth AAAI Conference on Artificial Intelligence*\.
- Bai et al\. \(2025\)Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others\. 2025\.[Qwen2\.5\-vl technical report](https://arxiv.org/abs/2502.13923)\.*Preprint*, arXiv:2502\.13923\.
- Cai et al\. \(2024a\)Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D\. Lee, Deming Chen, and Tri Dao\. 2024a\.[Medusa: Simple LLM inference acceleration framework with multiple decoding heads](https://arxiv.org/abs/2401.10774)\.*Preprint*, arXiv:2401\.10774\.
- Cai et al\. \(2024b\)Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao\. 2024b\.[PyramidKV: Dynamic KV cache compression based on pyramidal information funneling](https://arxiv.org/abs/2406.02069)\.*Preprint*, arXiv:2406\.02069\.
- Chen et al\. \(2025\)Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang\. 2025\.An image is worth 1/2 tokens after layer 2: Plug\-and\-play inference acceleration for large vision\-language models\.In*Computer Vision – ECCV 2024*, pages 19–35, Cham\. Springer Nature Switzerland\.
- Child et al\. \(2019\)Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever\. 2019\.[Generating long sequences with Sparse Transformers](https://arxiv.org/abs/1904.10509)\.*Preprint*, arXiv:1904\.10509\.
- Dao \(2023\)Tri Dao\. 2023\.[Flashattention\-2: Faster attention with better parallelism and work partitioning](https://arxiv.org/abs/2307.08691)\.*Preprint*, arXiv:2307\.08691\.
- Devoto et al\. \(2025\)Alessio Devoto, Maximilian Jeblick, and Simon Jégou\. 2025\.[Expected attention: Kv cache compression by estimating attention from future query distributions](https://arxiv.org/abs/2510.00636)\.*Preprint*, arXiv:2510\.00636\.
- Devoto et al\. \(2024\)Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini\. 2024\.[A simple and effectivel2l\_\{2\}norm\-based strategy for KV cache compression](https://arxiv.org/abs/2406.11430)\.*Preprint*, arXiv:2406\.11430\.
- El Maalouly \(2022\)Nicolas El Maalouly\. 2022\.[Exact matching: Algorithms and related problems](https://arxiv.org/abs/2203.13899)\.*Preprint*, arXiv:2203\.13899\.
- Elhoushi et al\. \(2024\)Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, Ahmed A\. Aly, Beidi Chen, and Carole\-Jean Wu\. 2024\.[LayerSkip: Enabling early exit inference and self\-speculative decoding](https://arxiv.org/abs/2404.16710)\.*Preprint*, arXiv:2404\.16710\.
- Ji et al\. \(2026\)Yicheng Ji, Jun Zhang, Jinpeng Chen, Cong Wang, Lidan Shou, Gang Chen, and Huan Li\. 2026\.[See the forest for the trees: Loosely speculative decoding via visual\-semantic guidance for efficient inference of video llms](https://arxiv.org/abs/2604.05650)\.*Preprint*, arXiv:2604\.05650\.
- Jiang et al\. \(2024\)Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H\. Abdi, Dongsheng Li, Chin\-Yew Lin, Yuqing Yang, and Lili Qiu\. 2024\.MInference 1\.0: Accelerating pre\-filling for long\-context LLMs via dynamic sparse attention\.In*Advances in Neural Information Processing Systems \(NeurIPS 2024\)*, volume 37, pages 52481–52515\.
- Komatsuzaki et al\. \(2022\)Aran Komatsuzaki, Joan Puigcerver, James Lee\-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby\. 2022\.[Sparse Upcycling: Training Mixture\-of\-Experts from dense checkpoints](https://arxiv.org/abs/2212.05055)\.*Preprint*, arXiv:2212\.05055\.
- Li et al\. \(2024a\)Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others\. 2024a\.[Llava\-onevision: Easy visual task transfer](https://arxiv.org/abs/2408.03326)\.*Preprint*, arXiv:2408\.03326\.
- Li et al\. \(2024b\)Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\. 2024b\.Snapkv: Llm knows what you are looking for before generation\.In*Advances in Neural Information Processing Systems*, volume 37\.
- Lin et al\. \(2024\)Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\. 2024\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, volume 6, pages 87–100\.
- Lin et al\. \(2025\)Junyan Lin, Haoran Chen, Yue Fan, Yingqi Fan, Xin Jin, Hui Su, Jinlan Fu, and Xiaoyu Shen\. 2025\.Multi\-layer visual feature fusion in multimodal LLMs: Methods, analysis, and best practices\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 4156–4166\.
- Liu et al\. \(2024a\)Jiahui Liu, Praveen Ponnusamy, Tianle Cai, Hanlin Guo, Yoon Kim, and Ben Athiwaratkun\. 2024a\.[Training\-free activation sparsity in large language models](https://arxiv.org/abs/2408.14690)\.*Preprint*, arXiv:2408\.14690\.
- Liu et al\. \(2023\)Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng\-lin Liu, Lianwen Jin, and Xiang Bai\. 2023\.[OCRBench: On the hidden mystery of OCR in large multimodal models](https://arxiv.org/abs/2305.07895)\.*Preprint*, arXiv:2305\.07895\.
- Liu et al\. \(2024b\)Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\. 2024b\.[KIVI: A tuning\-free asymmetric 2\-bit quantization for KV cache](https://arxiv.org/abs/2402.02750)\.*Preprint*, arXiv:2402\.02750\.
- Masry et al\. \(2022\)Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque\. 2022\.ChartQA: A benchmark for question answering about charts with visual and logical reasoning\.In*Findings of the Association for Computational Linguistics: ACL 2022*, pages 2263–2279, Dublin, Ireland\. Association for Computational Linguistics\.
- Mathew et al\. \(2021\)Minesh Mathew, Dimosthenis Karatzas, and C\.V\. Jawahar\. 2021\.DocVQA: A dataset for VQA on document images\.In*Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*, pages 2200–2209\.
- Park et al\. \(2025\)Junyoung Park, Dalton Jones, Matthew J\. Morse, Raghavv Goel, Mingu Lee, and Chris Lott\. 2025\.[Keydiff: Key similarity\-based kv cache eviction for long\-context llm inference in resource\-constrained environments](https://arxiv.org/abs/2504.15364)\.*Preprint*, arXiv:2504\.15364\.
- Peer et al\. \(2024\)David Peer, Philemon Schöpf, Volckmar Nebendahl, Alexander Rietzler, and Sebastian Stabinger\. 2024\.[ANLS\* – a universal document processing metric for generative large language models](https://arxiv.org/abs/2402.03848)\.*Preprint*, arXiv:2402\.03848\.
- Shen et al\. \(2024\)Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J\. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra\. 2024\.[LongVU: Spatiotemporal adaptive compression for long video\-language understanding](https://arxiv.org/abs/2410.17434)\.*Preprint*, arXiv:2410\.17434\.
- Sidorov et al\. \(2020\)Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh\. 2020\.TextCaps: A dataset for image captioning with reading comprehension\.In*Computer Vision – ECCV 2020*, pages 742–758, Cham\. Springer International Publishing\.
- Singh et al\. \(2019\)Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach\. 2019\.Towards VQA models that can read\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Su et al\. \(2025\)Zunhai Su, Zhe Chen, Wang Shen, Hanyu Wei, Linge Li, Huangqi Yu, and Kehong Yuan\. 2025\.[RotateKV: Accurate and robust 2\-bit KV cache quantization for LLMs via outlier\-aware adaptive rotations](https://arxiv.org/abs/2501.16383)\.*Preprint*, arXiv:2501\.16383\.
- Tao et al\. \(2025\)Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang\. 2025\.DyCoke: Dynamic compression of tokens for fast video large language models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 18992–19001\.
- Vedantam et al\. \(2014\)Ramakrishna Vedantam, C\. Lawrence Zitnick, and Devi Parikh\. 2014\.[CIDEr: Consensus\-based image description evaluation](https://arxiv.org/abs/1411.5726)\.*Preprint*, arXiv:1411\.5726\.
- Wan et al\. \(2024\)Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan\. 2024\.[LOOK\-M: Look\-once optimization in KV cache for efficient multimodal long\-context inference](https://arxiv.org/abs/2406.18139)\.*Preprint*, arXiv:2406\.18139\.
- Wang et al\. \(2025\)Jiahui Wang, Zuyan Liu, Yongming Rao, and Jiwen Lu\. 2025\.[Sparsemm: Head sparsity emerges from visual concept responses in mllms](https://arxiv.org/abs/2506.05344)\.*Preprint*, arXiv:2506\.05344\.
- Xiao et al\. \(2024\)Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun\. 2024\.InfLLM: Training\-free long\-context extrapolation for LLMs with an efficient context memory\.In*Proceedings of the 38th International Conference on Neural Information Processing Systems \(NeurIPS 2024\)*, Red Hook, NY, USA\. Curran Associates Inc\.
- Xiao et al\. \(2023\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\. 2023\.[Efficient streaming language models with attention sinks](https://arxiv.org/abs/2309.17453)\.*Preprint*, arXiv:2309\.17453\.
- Xu et al\. \(2025\)Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, and Guohao Dai\. 2025\.SpecEE: Accelerating large language model inference with speculative early exiting\.In*Proceedings of the 52nd Annual International Symposium on Computer Architecture \(ISCA ’25\)*, pages 467–481, New York, NY, USA\. Association for Computing Machinery\.
- Yang et al\. \(2025a\)Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, and Bo Yuan\. 2025a\.TopV: Compatible token pruning with inference time optimization for fast and low\-memory multimodal vision\-language model\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 19803–19813\.
- Yang et al\. \(2025b\)Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia\. 2025b\.VisionZip: Longer is better but not necessary in vision\-language models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 19792–19802\.
- Yue et al\. \(2024\)Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others\. 2024\.MMMU: A massive multi\-discipline multimodal understanding and reasoning benchmark for expert AGI\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 9556–9567\.
- Zeng et al\. \(2026\)Bowen Zeng, Feiyang Ren, Jun Zhang, Xiaoling Gu, Ke Chen, Lidan Shou, and Huan Li\. 2026\.[Hybridkv: Hybrid kv cache compression for efficient multimodal large language model inference](https://arxiv.org/abs/2604.05887)\.*Preprint*, arXiv:2604\.05887\.
- Zhang et al\. \(2026\)Jun Zhang, Yicheng Ji, Feiyang Ren, Yihang Li, Bowen Zeng, Zonghao Chen, Ke Chen, Lidan Shou, Gang Chen, and Huan Li\. 2026\.[Efficient inference for large vision\-language models: Bottlenecks, techniques, and prospects](https://arxiv.org/abs/2604.05546)\.*Preprint*, arXiv:2604\.05546\.
- Zhang et al\. \(2024\)Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu\. 2024\.[LMMs\-Eval: Reality check on the evaluation of large multimodal models](https://arxiv.org/abs/2407.12772)\.*Preprint*, arXiv:2407\.12772\.
- Zhang et al\. \(2025\)Yuan Zhang, Chun\-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang\. 2025\.[SparseVLM: Visual token sparsification for efficient vision\-language models inference](https://openreview.net/forum?id=1xG3MN1RRW)\.
- Zheng et al\. \(2025\)Simeng Zheng, Chih\-Hui Ho, Wenyu Peng, and Paul H\. Siegel\. 2025\.[Flash\-Gen: Spatio\-temporal generator for flash memory systems](https://doi.org/10.1109/TCOMM.2024.3442691)\.*IEEE Transactions on Communications*, 73\(2\):1100–1113\.
- Zhou et al\. \(2025\)Yuxin Zhou, Zheng Li, Jun Zhang, Jue Wang, Yiping Wang, Zhongle Xie, Ke Chen, and Lidan Shou\. 2025\.[Floe: On\-the\-fly moe inference on memory\-constrained gpu](https://proceedings.mlr.press/v267/)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*\. PMLR\.

## Appendix ATheoretical Analysis of Prefill–Decode Scale Mismatch

![Refer to caption](https://arxiv.org/html/2607.22586v1/x5.png)\(a\)Layer 15
![Refer to caption](https://arxiv.org/html/2607.22586v1/x6.png)\(b\)Layer 21

Figure 4:The figure shows the overall numerical distribution of hidden states across different layers of theLLaVA\-NeXT\-Vicuna\-7Bmodel during inference\. We flattened the hidden states to calculate their overall numerical distribution, where the x\-axis represents specific values and the y\-axis denotes distribution density \(with the calculation formula as follows:density​\(x\)=freq​\(x\)N⋅Δ​x\\mathrm\{density\}\(x\)=\\frac\{\\mathrm\{freq\}\(x\)\}\{N\\cdot\\Delta x\},wheredensity​\(x\)\\mathrm\{density\}\(x\): the empirical density \(height of the histogram\) at valuexx;freq​\(x\)\\mathrm\{freq\}\(x\): the number of samples falling into the interval centered at \(or starting from\)xx;NN: the total number of all samples;Δ​x\\Delta x: the width of the interval \(bin\)\.\)\.This appendix provides a concise theoretical explanation of the prefill–decode scale mismatch and motivates variance\-expanded, probe\-based KV selection as a principled decode\-aware approximation under prefill\-only execution\.

### A\.1Distributional Shift Between Prefilling and Decoding

As shown in Figure[4](https://arxiv.org/html/2607.22586#A1.F4), the overall numerical distribution of hidden states across layers is approximately Gaussian\-like at the projection input\. We therefore model hidden states at a given layer as Gaussian random variables for tractable approximation\. Empirically, decoding\-time hidden states and query projections exhibit larger variance than those observed during prefilling:

hpre∼𝒩​\(μ,Σpre\),hdec∼𝒩​\(μ,Σdec\)\\begin\{array\}\[\]\{l\}h\_\{\\mathrm\{pre\}\}\\sim\\mathcal\{N\}\(\\mu,\\Sigma\_\{\\mathrm\{pre\}\}\),\\\\ h\_\{\\mathrm\{dec\}\}\\sim\\mathcal\{N\}\(\\mu,\\Sigma\_\{\\mathrm\{dec\}\}\)\\end\{array\}\(12\)where

Σdec≻Σpre\.\\Sigma\_\{\\mathrm\{dec\}\}\\succ\\Sigma\_\{\\mathrm\{pre\}\}\.\(13\)Through linear query projection, this induces a corresponding variance gap in query distributions, causing prefilling\-based query proxies to underestimate the support of true decoding\-time queries\.

### A\.2Impact on Attention Estimation

For a fixed keykkand Gaussian queryq∼𝒩​\(μq,Σq\)q\\sim\\mathcal\{N\}\(\\mu\_\{q\},\\Sigma\_\{q\}\), the expected unnormalized attention score admits the closed form

𝔼​\[exp⁡\(q⊤​kd\)\]=exp⁡\(μq⊤​kd\+k⊤​Σq​k2​d\)\.\\mathbb\{E\}\\\!\\left\[\\exp\\\!\\left\(\\frac\{q^\{\\top\}k\}\{\\sqrt\{d\}\}\\right\)\\right\]=\\exp\\\!\\left\(\\frac\{\\mu\_\{q\}^\{\\top\}k\}\{\\sqrt\{d\}\}\+\\frac\{k^\{\\top\}\\Sigma\_\{q\}k\}\{2d\}\\right\)\.\(14\)
UnderestimatingΣq\\Sigma\_\{q\}therefore systematically underestimates expected attention mass, particularly for keys aligned with high\-variance query directions, leading to biased KV ranking under constrained budgets\.

### A\.3First\- and Second\-Order Interpretation

The closed\-form expression above provides a direct interpretation of the two design choices in MM\-ShiftKV:

- •Sample\-wise mean centering preserves the first\-order structure\(μq\\mu\_\{q\}\): the proxy distribution is centered using the prefill statistics of the current input sample, which provides an input\-adaptive semantic center for proxy construction\.
- •Variance expansion compensates the second\-order term\(Σq\\Sigma\_\{q\}\): inflating variance enlarges coverage along high\-variance directions so that semantically important outlier\-aligned keys are less likely to be underestimated\.

Therefore, MM\-ShiftKV does not explicitly align proxy means to decoding\-stage statistics\. Instead, it preserves the sample\-wise prefill mean structure for semantic stability and corrects the dominant prefill–decode discrepancy through variance expansion\.

### A\.4Variance\-Expanded and Probe\-Based Approximation

To compensate for this bias without accessing decoding\-time signals, we introduce a variance\-expanded proxy distribution

q~∼𝒩​\(μq,γ2​Σq,pre\),γ\>1\.\\tilde\{q\}\\sim\\mathcal\{N\}\(\\mu\_\{q\},\\gamma^\{2\}\\Sigma\_\{q,\\mathrm\{pre\}\}\),\\qquad\\gamma\>1\.\(15\)This expansion enlarges the support of prefilling\-based queries while preserving their mean structure\.

KV importance is then estimated by sampling a finite number of query probes, yielding a Monte Carlo approximation of expected attention mass\. Group\-wise aggregation further reduces estimator variance without materializing attention matrices\.

### A\.5Summary

This analysis explains why uncalibrated prefill\-based KV selection fails under distributional scale shift and why variance\-expanded, probe\-based attention estimation provides an effective and training\-free decode\-aware alternative for memory\-constrained multimodal inference\.

## Appendix BImplementation Details

Our method supports Grouped\-Query Attention \(GQA\) models such as Qwen2\.5\-VL\-7B\-Instruct and LLaVA\-v1\.6\-Vicuna\-7B\. In GQA, query states have shape\(B,L,Hq,d\)\(B,L,H\_\{q\},d\)and key\-value states stored in the KV cache have shape\(B,L,Hk​v,d\)\(B,L,H\_\{kv\},d\), withHq=Hk​v×GH\_\{q\}=H\_\{kv\}\\times G\. For attention computation, we repeat the key and value states along the head dimension to restore an MHA\-like layout, yielding attention scores of shape\(B,Hq,Lq,Lk\)\(B,H\_\{q\},L\_\{q\},L\_\{k\}\)\. Although attention scores are computed at the query\-head level, KV cache budgeting is performed at the key\-value head level by aggregating the scores of query heads belonging to the same key\-value head\. KV importance is estimated during prefilling using synthetic query probes sampled from statistics computed over the full prefill sequence, withN=512N=512samples per layer, generated from a diagonal Gaussian distribution parameterized by sample\-wise prefill statistics\. The probe queries are grouped intoGgroups=32G\_\{\\mathrm\{groups\}\}=32groups to stabilize estimation, and tokens are ranked by aggregated attention scores\. For each key\-value head, we select the smallest set of tokens whose cumulative attention mass exceeds a fixed threshold \(0\.95\), while always preserving the most recent token and incorporating the attention score from the last real query token\. The selected tokens are restored to their original temporal order, concatenated with the most recent token, and inserted into the KV cache using the standard update interface, ensuring full compatibility with FlashAttention\-style kernels\. All experiments are conducted in a purely inference\-time setting without additional training or fine\-tuning\.

Algorithm 1MM\-ShiftKV \(Prefill\-only, Decode\-aware KV Selection\)1:Prefill hidden states

\{ht\(ℓ\)\}t=1T\\\{h\_\{t\}^\{\(\\ell\)\}\\\}\_\{t=1\}^\{T\}, prompt KVs

\{\(kt\(ℓ,h\),vt\(ℓ,h\)\)\}t=1T\\\{\(k\_\{t\}^\{\(\\ell,h\)\},v\_\{t\}^\{\(\\ell,h\)\}\)\\\}\_\{t=1\}^\{T\}, budget

Cℓ,hC\_\{\\ell,h\}
2:Compressed KV cache

𝒞′\(ℓ,h\)\\mathcal\{C^\{\\prime\}\}^\{\(\\ell,h\)\}
3:Compute prefill statistics

\(μpre\(ℓ\),σpre\(ℓ\)\)\(\\mu\_\{\\text\{pre\}\}^\{\(\\ell\)\},\\sigma\_\{\\text\{pre\}\}^\{\(\\ell\)\}\)
4:Sample

N=G​gN\{=\}Gghidden states with variance expansion

γ\\gamma
5:Project samples to query proxies and apply future\-position RoPE

6:Partition query proxies into

GGgroups of size

gg
7:foreach group

g′g^\{\\prime\}do

8:Aggregate attention mass over prompt keys

9:Select minimal set covering fraction

τ\\tau
10:Vote selected tokens

11:endfor

12:Add last\-query anchor:

st←vote​\(t\)\+λ​at​\(qlast\(ℓ,h\)\)s\_\{t\}\\leftarrow\\mathrm\{vote\}\(t\)\+\\lambda a\_\{t\}\(q\_\{\\text\{last\}\}^\{\(\\ell,h\)\}\)
13:Always retain token

TTand select top\-

Cℓ,h−1C\_\{\\ell,h\}\-1tokens by

sts\_\{t\}
14:Restore temporal order and return

𝒞′\(ℓ,h\)\\mathcal\{C^\{\\prime\}\}^\{\(\\ell,h\)\}

![Refer to caption](https://arxiv.org/html/2607.22586v1/x7.png)\(a\)Per\-token decoding latency vs\. input length\.
![Refer to caption](https://arxiv.org/html/2607.22586v1/x8.png)\(b\)Prompt KV cache size vs\. input length\.

Figure 5:Visualization of decoding latency and prompt KV cache size under increasing input lengths\.![Refer to caption](https://arxiv.org/html/2607.22586v1/x9.png)\(a\)DocVQA
![Refer to caption](https://arxiv.org/html/2607.22586v1/x10.png)\(b\)SynthDog

Figure 6:Additional visualizations of the prefill–decode*scale mismatch*in multimodal inference across different datasets\.\(a\)Layer\-wise representation statistics onDocVQA, showing that decoding\-stage hidden states exhibit substantially larger variance than those observed during prefilling\.\(b\)The same prefill–decode variance mismatch observed onSynthDog, indicating that the scale mismatch is consistent across datasets\.
## Appendix CAdditional Visualization Results

We present additional visualizations to complement the statistical observations in Section[2](https://arxiv.org/html/2607.22586#S2)and to provide intuitive evidence of the prefill–decode*scale mismatch*in long\-context multimodal inference\. All visualizations are obtained usingQwen2\.5\-VL\-7B\-Instructon representative document understanding benchmarks, includingDocVQAandSynthDog\.

As shown in Figure[5](https://arxiv.org/html/2607.22586#A2.F5), FullKV exhibits steadily increasing per\-token decoding latency and prompt KV cache size as the input length grows\. In contrast, MM\-ShiftKV bounds the prompt KV cache after prefilling, resulting in near\-constant decoding latency and significantly reduced memory usage\. This visualization illustrates how prefill\-only KV selection decouples decoding cost from the original input length under long\-context multimodal inputs\.

Figure[6](https://arxiv.org/html/2607.22586#A2.F6)provides additional evidence of the prefill–decode*representation scale mismatch*observed in Section[2](https://arxiv.org/html/2607.22586#S2)\. Specifically, Figure[6\(a\)](https://arxiv.org/html/2607.22586#A2.F6.sf1)visualizes layer\-wise representation statistics onDocVQA, showing that decoding\-stage hidden states exhibit consistently larger variance than those observed during prefilling, despite sharing identical model parameters\.

Figure[6\(b\)](https://arxiv.org/html/2607.22586#A2.F6.sf2)shows that the same variance expansion effect persists onSynthDog, indicating that the prefill–decode scale mismatch is not specific to a single dataset but instead reflects a systematic property of multimodal inference under long contexts\. Together, these visualizations demonstrate that prefilling\-stage statistics systematically underestimate the scale of decoding\-time representations across different document understanding benchmarks\.

## Appendix DSensitivity Study on Hyperparameters

We conduct a sensitivity study to examine the robustness ofMM\-ShiftKVwith respect to its key hyperparameters\. Unless otherwise specified, all experiments in this section are performed onQwen2\.5\-VL\-7B\-Instructunder a fixed per\-head KV cache budget ofC=64C\{=\}64\. We report results on three representative multimodal benchmarks:OCRBenchandTextVQA\(accuracy\), andTextCaps\(CIDEr\)\. When analyzing one hyperparameter, all others are held fixed at their default values \(γ=10\\gamma\{=\}10,N=512N\{=\}512,G=32G\{=\}32,τ=0\.95\\tau\{=\}0\.95,λ=1\\lambda\{=\}1\)\.

### D\.1Variance Expansion Factorγ\\gamma

The variance expansion factorγ\\gammacontrols the scale calibration between prefilling\-stage query proxies and decoding\-time query distributions\. As discussed in Section[2](https://arxiv.org/html/2607.22586#S2), decoding\-time queries exhibit substantially larger variance than those observed during prefilling, motivating the use ofγ\>1\\gamma\>1\.

Table[6](https://arxiv.org/html/2607.22586#A4.T6)shows that settingγ=1\\gamma\{=\}1, corresponding to no variance expansion, leads to consistently degraded performance across all benchmarks\. Increasingγ\\gammasignificantly improves performance, indicating that scale calibration is critical for effective KV importance estimation\. Performance peaks aroundγ=10\\gamma\{=\}10, while further increasingγ\\gammayields diminishing returns\. Based on this observation, we fixγ=10\\gamma\{=\}10as the default value in all experiments\.

### D\.2Attention Mass Thresholdτ\\tau

The attention mass thresholdτ\\taudetermines the minimum cumulative attention mass preserved when selecting prompt KV tokens\. A smaller threshold may discard occasionally important tokens, whereas an overly large threshold approaches FullKV behavior and weakens compression\.

As shown in Table[7](https://arxiv.org/html/2607.22586#A4.T7),τ=0\.95\\tau\{=\}0\.95consistently achieves the best trade\-off between performance and compression across all evaluated benchmarks\. Lower thresholds result in noticeable performance drops, while higher thresholds provide limited additional benefit\. We therefore adoptτ=0\.95\\tau\{=\}0\.95as a stable default setting\.

### D\.3Number of Query ProxiesNN

The number of query proxiesNNcontrols the quality of the Monte Carlo approximation of expected attention mass\. LargerNNreduces estimator variance but increases prefilling\-stage computation\.

Results in Table[8](https://arxiv.org/html/2607.22586#A4.T8)show that performance improves asNNincreases from 128 to 512 and stabilizes atN=512N\{=\}512\. Using fewer proxies leads to noisier importance estimates, while larger values offer limited additional gains relative to the increased overhead\. We therefore setN=512N\{=\}512as a balanced choice between accuracy and efficiency\.

### D\.4Summary

Overall, these sensitivity studies demonstrate thatMM\-ShiftKVis robust to moderate variations in its hyperparameters\. The selected default values correspond to stable operating points that consistently balance accuracy, memory efficiency, and prefilling\-stage overhead across OCR\-centric, multimodal question answering, and image\-conditioned generation tasks\.

Table 6:Sensitivity study on the variance expansion factorγ\\gamma\. OCRBench and TextVQA are evaluated with accuracy, and TextCaps with CIDEr\. Avg denotes the macro\-average over the three benchmarks\. The per\-head KV cache budget isC=64C\{=\}64\. Higher is better\.Table 7:Sensitivity study on the attention mass thresholdτ\\tau\. Metrics and averaging follow Table[6](https://arxiv.org/html/2607.22586#A4.T6)\. The per\-head KV cache budget isC=64C\{=\}64\.Table 8:Sensitivity study on the number of query probesNN\. Metrics and averaging follow Table[6](https://arxiv.org/html/2607.22586#A4.T6)\. The per\-head KV cache budget isC=64C\{=\}64\.

## Appendix EAdditional Experiments

To provide a deeper understanding of the underlying mechanisms of MM\-ShiftKV, this appendix presents additional empirical evaluations\. We focus our analysis on three critical dimensions: the end\-to\-end efficiency trade\-off, the statistical robustness of our global hyperparameter strategy, and the dynamic decode\-awareness under shifting context lengths\.

### E\.1End\-to\-End Latency and Memory Breakdown

Table[9](https://arxiv.org/html/2607.22586#A5.T9)reports TTFT, end\-to\-end latency, and peak VRAM on a 32K\-token setting withLLaVA\-v1\.6\-7B\. While MM\-ShiftKV introduces a moderate TTFT increase \(approximately 0\.38s\) due to the one\-time proxy sampling in the prefill stage, this overhead accounts for only∼\\sim2\.2% of the total inference time\. This marginal initialization cost is heavily outweighed by a 42% reduction in total end\-to\-end latency and a 43\.3% decrease in peak VRAM compared to FullKV\. Since the proxy sampling complexity remains linear𝒪​\(N⋅K\)\\mathcal\{O\}\(N\\cdot K\)with respect to sequence length, the method provides a highly cost\-effective trade\-off for latency\-sensitive, long\-context applications without introducing any decoding\-time overhead\.

Table 9:Latency and memory summary on a 32K\-token setting\. TTFT denotes Time\-To\-First\-Token\. Lower is better for all metrics\.
### E\.2Global vs\. Layer\-Wise Expansion Strategies

Table[10](https://arxiv.org/html/2607.22586#A5.T10)compares our globally unified strategy against a refined layer\-wise variant\. Empirical analysis across diverse architectures \(e\.g\., MHA in LLaVA and GQA in Qwen2\.5\-VL\) reveals that variance expansion is an intrinsic property of MLLM inference, consistently expanding by an order of magnitude \(typically13×13\\timesto22×22\\times\) from prefill to decoding\. While actual variance fluctuates across layers, fine\-grained layer\-wise tuning often underestimates the required bandwidth for layers with smaller prefill deviations, leading to diversity collapse\. In contrast, a globally fixedγ\\gammaacts as a robust safety upper\-bound and broad search bandwidth, effectively preventing the loss of critical outliers without the need for ad\-hoc per\-model calibration\.

Table 10:Comparison of global and layer\-wise expansion strategies on Qwen2\.5\-VL\-7B\. Higher is better\.
### E\.3Mean\-Centering Strategy Ablation

Table[11](https://arxiv.org/html/2607.22586#A5.T11)compares different mean\-centering strategies for query proxy construction\. Since MM\-ShiftKV does not explicitly align proxy means to decoding\-stage statistics, we instead study how different prefill\-based centering schemes affect robustness\. In particular, we compare no mean centering, a globally shared prefill mean, and sample\-wise mean centering computed from the current input\. Sample\-wise mean centering performs best, indicating that adapting the proxy center to each input sample provides a better semantic anchor than a fixed global center\. The main gain of MM\-ShiftKV still comes from variance expansion, while sample\-wise centering further improves stability\.

Table 11:Ablation on mean\-centering strategies for query proxy construction \(Qwen2\.5\-VL\-7B, OCRBench\)\. Higher is better\.
### E\.4Sensitivity to Anchor Weightλ\\lambda

To clarify the role of the last\-query anchor, we varyλ\\lambdawhile keeping all other settings fixed \(Qwen2\.5\-VL\-7B, OCRBench,C=64C\{=\}64\)\. MM\-ShiftKV utilizes a lexicographical\-like hierarchical ranking mechanism rather than simple linear weighting\. Since proxy votes are discrete integers, settingλ=1\\lambda\{=\}1ensures the continuous anchor score acts strictly as a secondary tie\-breaker\. As shown in Table[12](https://arxiv.org/html/2607.22586#A5.T12), removing the anchor \(λ=0\\lambda\{=\}0\) forces unstable random dropping for tied tokens, degrading accuracy\. Conversely, an oversized weight \(λ=50\\lambda\{=\}50\) allows local heuristics to override the global proxy consensus, weakening the proxy\-vote dominance and leading to suboptimal context retention\.

Table 12:Sensitivity analysis of last\-query anchor weightλ\\lambda\(Qwen2\.5\-VL\-7B, OCRBench, budgetC=64C\{=\}64\)\. Higher is better\.
### E\.5Unified vs\. Modality\-Specific Processing

We further compare unified and modality\-specific statistic construction on OCRBench \(Qwen2\.5\-VL\-7B,C=64C\{=\}64\)\. In multimodal inference, text prompts serve as crucial anchors that guide the model to specific visual regions, exhibiting strong cross\-modal coupling\. Decoupling textual and visual modalities into disjoint proxy spaces disrupts unified attention scaling and fragments the evaluation pool\. Table[13](https://arxiv.org/html/2607.22586#A5.T13)confirms that unified processing preserves this critical cross\-modal alignment, significantly outperforming modality\-separated strategies and demonstrating the necessity of joint cross\-modal statistics for KV selection\.

Table 13:Unified vs\. modality\-specific processing under the same KV budget \(Qwen2\.5\-VL\-7B, OCRBench,C=64C\{=\}64\)\. Higher is better\.
### E\.6Distribution Shape Check at Sampling Entrance

To clarify the Gaussian modeling assumption, MM\-ShiftKV does not assume all internal activations are strictly Gaussian\. Instead, it accurately models the “layered distributional evolution\.” Due to Central Limit Theorem \(CLT\)\-like residual aggregation, accumulated hidden states at the*sampling entrance*\(before query projection\) maintain a highly stable, unimodal Gaussian\-like profile\. As shown in Figure[4](https://arxiv.org/html/2607.22586#A1.F4), this approximation is stable enough for proxy construction, providing a robust mathematical base\. The actual heavy\-tailed outliers emerge downstream during the projection phase, where weight matrices act as feature amplifiers\. Our method samples at this statistically stable entry point and relies on variance expansion to bridge the subsequent structural amplification\.

### E\.7Outlier Coverage Under Variance Expansion

To directly address the heavy\-tail concern, we measure whether sampled proxies can cover critical outlier channels after projection\. Following the rebuttal protocol, we track Top\-0\.1% systematic outlier dimensions\. Under narrow sampling \(γ=1\\gamma\{=\}1\), probes fail to perceive outliers at the long tails due to rapid normal distribution decay, resulting in “information blindness” \(12\.4% hit rate\)\. As shown in Table[14](https://arxiv.org/html/2607.22586#A5.T14), explicitly inflating the covariance via variance expansion mathematically fattens the sampling envelope, surging the outlier hit rate to 89\.7% and substantially improving coverage across heavy\-tailed directions\.

Table 14:Coverage of critical outlier channels under different sampling scales \(Qwen2\.5\-VL\-7B, budgetC=64C\{=\}64\)\. Higher is better\.

## Appendix FDetails of Baselines and Dataset

This appendix presents the implementation details and specific parameter configurations of the baselines, along with the detailed content and tasks of the datasets\.

### F\.1Details of Baselines

For our experiments, we use four methods, namely StreamingLLM, SnapKV, KeyDiff, and ExceptAttn, as our test baselines\. We also compare the performance differences between all these methods and FullKV under budget\-constrained conditions\.

StreamingLLM is a classic heuristic KV eviction method\. It introduces the concept of attention sink, retains the initial KV pairs statically, and leverages a sliding window mechanism to continuously preserve the KV pairs within the most recent window during the decoding stage\. This method discards a large number of redundant intermediate KV pairs\. It achieves favorable performance in text reasoning due to its streaming inference paradigm\. But in multimodal scenarios, it discards a large number of critical visual KV pairs, leading to performance collapse\. In the experiments on StreamingLLM, we adopt the optimal parameters specified in its original paper: to retain44attention sinks, set the window size tobudgets\-4, and to ensure fairness of comparative experiments, we do not perform KV eviction during the decoding stage\.

SnapKV is a strong baseline method for large language models\. It compares the similarity between the query attention scores of the final window in the prefilling stage and those in the decoding stage\. It uses the final window of the prefilling stage to score the prefix KV pairs based on attention, selects the KV pairs with high attention scores, and statically retains the final window to maintain the characteristics of streaming inference\. Following the optimal parameters provided by SnapKV, we set the window size to 32, the convolution size to 5, and the inter\-group pooling to average pooling\. SnapKV achieves outstanding performance on large language models and also has certain generality in multimodal scenarios, but it performs poorly under ultra\-low budget conditions\. We conduct a theoretical analysis of this issue: the statically retained window in SnapKV involves some waste and fails to truly evaluate the actually required queries, while reducing the window size leads to instability in KV selection\.

KeyDiff achieves state\-of\-the\-art metrics on large language models\. It evaluates the relationship between the cosine similarity of Keys in large language models and the magnitude of the attention scores they receive\. It proposes a query\-agnostic method that scores Keys based on the cosine similarity between them, and the KV pairs with low cosine similarity are retained\. In this experiment, since our dataset features streaming inference, we statically retain the last 1 token in accordance with KeyDiff’s handling of streaming inference and for the fairness of the experiment\. KeyDiff is also perturbed in multimodal scenarios\. The relationship between the cosine similarity of Keys for tokens in multimodal scenarios and the attention scores is inconsistent with that in text\-only unimodal scenarios, which is the main reason for the decrease in KeyDiff’s accuracy in multimodal scenarios\.

ExceptAttn also leverages the regularity of numerical distributions, but it assumes the distribution homogeneity between the prefilling and decoding stages\. However, in multimodal scenarios, there is a certain deviation between the distributions of these two stages, which results in relatively low performance\. In this experiment, we set its sampling count to 512 to ensure fairness, and we also configure the static retention of 4 attention sinks, consistent with the setup in its original paper\.

### F\.2Details of Dataset

To comprehensively evaluate MM\-ShiftKV , we utilize the lmms\-eval\(Zhang et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib43)\)evaluation framework\. We employ OCRbench to assess OCR tasks and cross\-modal text understanding tasks: this dataset covers multi\-scenario images such as document scans and street view recognition, and we adopt accuracy as the corresponding evaluation metric\. TextVQA, which spans real\-world scenarios including restaurant menus and street signs, is used to evaluate image\-text information question answering tasks, with performance measured by the Exact Match\(El Maalouly,[2022](https://arxiv.org/html/2607.22586#bib.bib11)\)metric\. Each image in TextCaps is annotated with 3\-5 reference descriptions that contain key text; we leverage this dataset to assess semantically consistent text\-image captioning tasks, with CIDEr\(Vedantam et al\.,[2014](https://arxiv.org/html/2607.22586#bib.bib32)\)serving as the performance metric\. ChartQA primarily composed of numerical charts is used to evaluate chart data understanding and reasoning\-based question answering tasks, with performance assessed using relative error and the Exact Match metric\. DocVQA covers a large volume of document images, and we use theANLS\(Peer et al\.,[2024](https://arxiv.org/html/2607.22586#bib.bib26)\)to evaluate MM\-ShiftKV’s performance on document image question answering tasks\. MMMU is a dataset for calculation, geometric proof, and logical reasoning tasks\. We use it to evaluate MM\-ShiftKV’s impact on reasoning capabilities and whether it generates hallucinations, with accuracy as the evaluation metric\.

## Appendix GCase Study

![Refer to caption](https://arxiv.org/html/2607.22586v1/x11.png)Figure 7:In the left example, the image contains two people wearinggreen shirtswith the printed text“Bossa Nova”\. While the full KV model produces an accurate caption, several baselines degrade significantly after KV compression\.SnapKVandKeyDiffidentify the two people but omit the color attribute, andStreamingLLMfurther loses the shirt\-related information\. In contrast,MM\-ShiftKVpreserves both the color and the textual content, yielding a caption consistent with the reference\.In the right example, the image shows aLone Star Beer canwith visible branding\.Under the same tight per\-head budget,SnapKVandKeyDiffmisclassify the object as a beer bottle or miss the embedded text, whereasMM\-ShiftKVcorrectly captures both the object type and textual content\. These examples demonstrate that decode\-aware KV selection enables MM\-ShiftKV to remain robust even under extreme KV\-cache compression, consistent with its quantitative gains on TextCaps\.We present a qualitative case study on theTextCapsdataset usingLLaVA\-v1\.6\-Vicuna\-7Bunder anextreme KV\-cache budget of 64 tokens per KV head, to illustrate the behavior of different prefill\-only KV selection methods under severe memory constraints\. Figure[7](https://arxiv.org/html/2607.22586#A7.F7)shows two representative examples requiring accurate recognition of visual attributes and embedded text\.Red denotes missing information, green indicates that key information is captured, and orange signifies that more information is captured compared to FullKV\.