HeadGuard：面向低比特 VLM KV-Cache 量化的选择性注意力头保护

# Selective Head Protectionfor Low-Bit VLM KV-Cache Quantization
Source: [https://arxiv.org/html/2609.35800](https://arxiv.org/html/2609.35800)
## HeadGuard: Selective Head Protection for Low\-Bit VLM KV\-Cache Quantization

###### Abstract

Low\-bit key–value \(KV\) cache quantization saves storage but can sharply degrade vision–language model \(VLM\) accuracy\. We introduceHeadGuard, a composable head\-protection method that augments a base KV\-cache quantizer with a fixed high\-precision mask\. Image\-sensitivity and output\-sensitivity scores select physical KV heads offline, with approximately1/81/8protected in the main experiments; their image keys and optionally values remain in bfloat16 \(BF16\), while the base quantizes unprotected image entries\. Across eight VLMs, three base quantizers, and eight benchmarks \(six discriminative and two generative\), HeadGuard recovers a substantial fraction of lost accuracy on weaker quantizers, with the strongest gains for Qwen and InternVL\. At22bits, the six\-task discriminative mean over eight models rises from0\.4360\.436to0\.5800\.580on the weakest base; protection can also improve generated answers and caption fidelity to BF16 outputs\. Mean accuracy gains persist across all three quantizers with both tested calibration datasets\. Keys\-only protection retains substantial recovery at lower modeled storage cost\. Evaluated through simulated quantization, HeadGuard offers a composable way to improve low\-bit VLM accuracy without replacing the underlying quantizer\.

## 1Introduction

Image tokens often dominate a vision–language model’s KV cache\. Low\-bit quantization reduces storage\([Liu et al\., 2024d](https://arxiv.org/html/2609.35800#bib.bib18);[Hooper et al\., 2024](https://arxiv.org/html/2609.35800#bib.bib9)\), but its effect on accuracy varies sharply\. Per\-token min–max, our simplest base quantizer, sets each token’s quantization range from its own minimum and maximum and rounds every entry to the nearest level in that range \(round\-to\-nearest\): on POPE, per\-token22\-bit min–max gives0\.8320\.832accuracy for LLaVA\-NeXT and0\.4960\.496for Qwen2\.5\-VL under the same evaluation protocol\.

Preserving a few entries in high precision is a common remedy\. We ask when protecting image KV heads helps, compared with changing the base quantizer or increasing precision\. Throughout, we quantize only the KV cache and keep model weights in bfloat16 \(BF16\)\. Our BF16 reference leaves the entire KV cache in BF16 as well, avoiding the rounding error that low\-bit cache quantization introduces\.

We introduceHeadGuard, a composable protection method, not a replacement quantizer\. It ranks heads offline using image\-sensitivity and output\-sensitivity scores, which measure how predictions respond to rescaling a head’s image contribution and its entire output, respectively\. Selected image keys—and optionally values—remain in BF16, while the base quantizer handles the remaining image entries\. The resulting mask is reused across tasks without weight updates or inference\-time gradients\.

Across eight VLMs, three base quantizers, and eight benchmarks, we examine when head protection helps and how its gains depend on precision, head budget, and calibration\. We also test whether image sensitivity adds value beyond output sensitivity and whether protecting keys alone retains recovery\.

## 2Related Work

KV\-cache quantization\.KIVI\([Liu et al\., 2024d](https://arxiv.org/html/2609.35800#bib.bib18)\)quantizes keys per channel and values per token; KVQuant\([Hooper et al\., 2024](https://arxiv.org/html/2609.35800#bib.bib9)\)combines low\-bit cache quantization with outlier handling\. AKVQ\-VL\([Su et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib22)\)uses attention\-based token saliency to allocate precision and Hadamard transforms to reduce outliers\. Our KIVI\-style and AKVQ\-style bases implement selected arithmetic components in a shared harness, not the complete published algorithms or serving kernels\.

Sensitivity\-based precision allocation\.Head pruning can use loss sensitivity to head masks\([Michel et al\., 2019](https://arxiv.org/html/2609.35800#bib.bib20)\)\. KVmix\([Li et al\., 2026](https://arxiv.org/html/2609.35800#bib.bib11)\)uses gradients of key/value projection weights to choose bit\-widths for each layer\. RateQuant\([Zuo et al\., 2026](https://arxiv.org/html/2609.35800#bib.bib29)\)chooses separate bit\-widths for each head’s keys and values under a total bit budget\. It uses squared gradients of KV projection outputs and fitted, quantizer\-specific curves that estimate quantization error at different bit\-widths\. Both target text LLMs\. HeadGuard also uses sensitivity to guide precision allocation, but targets image KV in VLMs\. It combines image\-sensitivity and output\-sensitivity scores to select a fixed set of top\-KKKV heads, preserving their image keys and optionally values in BF16\. Unlike RateQuant, it does not fit distortion curves; text KV remains BF16\.

Head\-based token retention\.HeadKV\([Fu et al\., 2025b](https://arxiv.org/html/2609.35800#bib.bib6)\)allocates token budgets from retrieval and reasoning importance; DuoAttention\([Xiao et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib26)\)keeps full context for retrieval heads and a short cache for streaming heads; and SparseMM\([Wang et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib23)\)uses OCR\-grounded attention scores for per\-head token budgets\. Our attention\-mass and visual\-response controls are simplified proxies, not AKVQ\-VL’s or SparseMM’s original scoring rules\. They let us compare head\-ranking signals using the same calibration data and BF16 protection budget\.

Visual token reduction\.DART\([Wen et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib25)\)removes visual tokens that carry similar information to other tokens\. Its experiments show that choosing which tokens to keep using attention scores can perform worse than choosing them at random\. Unlike such pruning, HeadGuard keeps all tokens and only changes their KV precision\.

## 3HeadGuard

HeadGuard operates on image\-token KV entries in the VLM’s language\-model attention layers\. HeadGuard takes a frozen VLM, a labeled calibration set𝒟\\mathcal\{D\}, a head budgetKK, and a base quantizer𝒬\\mathcal\{Q\}\. It scores heads offline, selects a fixed set, and bypasses low\-bit quantization for selected entries\.

Two sensitivity scores\.During calibration, we attach two differentiable scalar gates to each query head\(l,h\)\(l,h\), both evaluated at11\. The image gateγl,h\\gamma\_\{l,h\}scales the post\-softmax attention weights in columns corresponding to known image\-token positions in the input sequence, across all query positions in that head and*without renormalization*\. This scaling occurs before the attention weights are multiplied by the value matrix\. The output gateηl,h\\eta\_\{l,h\}instead scales the head’s entire output after this weighted sum and before the shared attention output projection\. Writing a head output asu=uimg\+utextu=u\_\{\\rm img\}\+u\_\{\\rm text\}, with contributions from image and text positions respectively, both gates are present in the same forward pass and give

u⁡\(γ,η\)=η⁡\(γ​uimg\+utext\)\.u\(\\gamma,\\eta\)=\\eta\(\\gamma u\_\{\\rm img\}\+u\_\{\\rm text\}\)\.Withη=1\\eta=1, the image gate rescales only the image contribution without reallocating attention to text\. Withγ=1\\gamma=1, the output gate scales the whole head output\. Unlike the output gate, the image gate leaves the text contribution unscaled\. We measure local sensitivity to these perturbations using mean absolute gradients\([Michel et al\., 2019](https://arxiv.org/html/2609.35800#bib.bib20)\):

gl,h=𝔼x∼𝒟​\|∂log⁡P⁡\(a⋆∣x\)∂γl,h\|,ol,h=𝔼x∼𝒟​\|∂ℒprompt​\(x\)∂ηl,h\|\.g\_\{l,h\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\left\|\\frac\{\\partial\\,\\log P\(a^\{\\star\}\\mid x\)\}\{\\partial\\gamma\_\{l,h\}\}\\right\|,\\qquad o\_\{l,h\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\left\|\\frac\{\\partial\\,\\mathcal\{L\}\_\{\\rm prompt\}\(x\)\}\{\\partial\\eta\_\{l,h\}\}\\right\|\.\(1\)Each expectation𝔼x∼𝒟\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}averages the per\-example absolute gradient over the calibration set𝒟\\mathcal\{D\}\. The image\-sensitivity scoregl,hg\_\{l,h\}measures the mean absolute sensitivity of the correct\-answer log\-probabilitylog⁡P⁡\(a⋆∣x\)\\log P\(a^\{\\star\}\\mid x\)to the image gate\. The output\-sensitivity scoreol,ho\_\{l,h\}measures the corresponding sensitivity ofℒprompt\\mathcal\{L\}\_\{\\rm prompt\}, the mean next\-token cross\-entropy on the text token positions, to the output gate; the image tokens stay in the input and are only excluded as prediction targets\. We pair these objectives to assess answer sensitivity to image contributions alongside whole\-head sensitivity across prompt\-text predictions\. The selector comparison does not isolate gate choice from objective choice\. Computing both scores for all heads uses one forward pass and two backward passes through the same graph per calibration example: one computes the answer\-log\-probability gradients with respect to the image gates, and the other computes the prompt\-loss gradients with respect to the output gates\. Neither gates nor model weights are updated\. Both are local \(first\-order\) sensitivity signals we use only to rank heads, not measures of faithful visual reasoning or of actual quantization error\.

Selecting physical KV heads\.Grouped\-query attention shares one KV head among several query heads\([Ainslie et al\., 2023](https://arxiv.org/html/2609.35800#bib.bib2)\), so we average the query\-head scores in each physical KV group to obtain image\-sensitivity and output\-sensitivity scoresg¯i\\bar\{g\}\_\{i\}ando¯i\\bar\{o\}\_\{i\}\. Hereiiindexes physical KV heads across all language\-model attention layers, andNKVN\_\{\\rm KV\}is their total count\. The two scores live on different scales, so we standardize each to zero mean and unit variance across the model’s KV heads \(z⁡\(⋅\)z\(\\cdot\)\), add them, and protect theKKheads with the largest sum:

si=z⁡\(g¯i\)\+z⁡\(o¯i\),𝒫=TopK⁡\(s,K\)\.s\_\{i\}=z\(\\bar\{g\}\_\{i\}\)\+z\(\\bar\{o\}\_\{i\}\),\\qquad\\mathcal\{P\}=\\operatorname\{TopK\}\(s,K\)\.\(2\)The head budgetKKis a parameter: our main K\+V and K\-only experiments useK=round⁡\(NKV/8\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/8\), and the capped budget sweep varies the fraction\. We choose this fraction to keep most image KV entries quantized while reserving a limited BF16 budget for sensitive heads\.

Applying protection\.With key\-and\-value \(K\+V\) protection, selected heads keep both image keys and values in BF16\. With keys\-only \(K\-only\) protection, selected image keys stay in BF16, while image values follow the base quantizer\. Text keys and values remain in BF16 in both cases\. A compatible base must allow quantization to be skipped for individual KV heads\.

## 4Evaluation Scope and Protocol

Models and tasks\.We evaluate LLaVA\-1\.5\-7B\([Liu et al\., 2023](https://arxiv.org/html/2609.35800#bib.bib14);[Liu et al\., 2024a](https://arxiv.org/html/2609.35800#bib.bib15)\), LLaVA\-NeXT\-Mistral\-7B\([Liu et al\., 2024b](https://arxiv.org/html/2609.35800#bib.bib16)\), Qwen2\.5\-VL\-7B\([Bai et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib3)\), Qwen2\-VL\-2B\([Wang et al\., 2024](https://arxiv.org/html/2609.35800#bib.bib24)\), Gemma\-3\-12B\([Gemma Team, 2025](https://arxiv.org/html/2609.35800#bib.bib7)\), Gemma\-4\-12B\([Gemma Team, 2026](https://arxiv.org/html/2609.35800#bib.bib8)\), Pixtral\-12B\([Agrawal et al\., 2024](https://arxiv.org/html/2609.35800#bib.bib1)\), and InternVL3\-8B\([Zhu et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib28)\)\. The six discriminative tasks are POPE\([Li et al\., 2023](https://arxiv.org/html/2609.35800#bib.bib12)\), MME\([Fu et al\., 2025a](https://arxiv.org/html/2609.35800#bib.bib5)\), MMBench\([Liu et al\., 2024c](https://arxiv.org/html/2609.35800#bib.bib17)\), image\-based ScienceQA\([Lu et al\., 2022](https://arxiv.org/html/2609.35800#bib.bib19)\), multiple\-choice MMMU\([Yue et al\., 2024](https://arxiv.org/html/2609.35800#bib.bib27)\), and SEED\-Bench\([Li et al\., 2024](https://arxiv.org/html/2609.35800#bib.bib10)\)\. TextVQA\([Singh et al\., 2019](https://arxiv.org/html/2609.35800#bib.bib21)\)and COCO captioning\([Chen et al\., 2015](https://arxiv.org/html/2609.35800#bib.bib4)\)add free\-form generation\. Unless noted otherwise, head scores are computed for each model and evaluation protocol from100100labeled POPE examples and then frozen across tasks and base quantizers\. The calibration comparison in §[5\.6](https://arxiv.org/html/2609.35800#S5.SS6)additionally computes scores from100100labeled MMBench examples\.

Evaluation protocols\.A preliminary check on Qwen2\-VL\-2B motivated the keys\-only evaluation: quantizing only the image keys collapsed accuracy, whereas quantizing only the image values left it near BF16\.

1. 1\.*K\+V protection:*22–33bits, a fixed1/81/8head budget, all eligible discriminative examples after holdouts\.
2. 2\.*K\-only evaluation:*22bits, the same head budget and evaluation scope, with only selected keys protected\.
3. 3\.*Budget/precision ablation:*22–44\-bit K\+V protection at nominal fractions1/161/16,1/121/12,1/81/8, and1/41/4, plus uniform, on capped discriminative subsets, also comparing POPE and MMBench calibration\.

The first two protocols give our main results; the third is a supporting ablation \(§[5\.6](https://arxiv.org/html/2609.35800#S5.SS6)\)\. For generation, the main K\+V and K\-only evaluations use10001000TextVQA examples and200200captioning examples; the capped sweeps use200200examples for each of these two tasks\.

Quantization setup\.We test per\-token min–max for keys and values; KIVI\-style, with per\-channel keys and per\-token values; and AKVQ\-style, with query/key rotation and per\-token quantization\. In each KV head, AKVQ\-style additionally keeps the keys and values of a nominal2%2\\%of image tokens in BF16, selecting those with the largest key norms\. This applies with or without HeadGuard\. KIVI\-style omits the original grouped/residual\-cache policy and recomputes key ranges over the supplied sequence, including text; text KV nevertheless remains in BF16\. Our large sweeps simulate low\-bit quantization inside attention: entries assigned low precision are rounded to the base quantizer’s levels and reconstructed as floating\-point values, rather than stored in a packed low\-bit cache\. These experiments measure the effect on accuracy, not the memory savings or speed of a packed cache\.

Head\-selection baselines\.To assess the contribution of the hybrid scoress, we compare against selectors that protect the same numberKKof physical KV heads\. The score\-based baselines use image sensitivityggalone; output sensitivityooalone; activation magnitude \(the squared head\-output norm\); image\-attention mass \(the sum of post\-softmax attention weights on image keys, an AKVQ\-inspired proxy\([Su et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib22)\)\); or visual\-response norm \(the norm of the head output contributed by image tokens, a proxy inspired by SparseMM’s visual\-head analysis\([Wang et al\., 2025](https://arxiv.org/html/2609.35800#bib.bib23)\)\)\. For the three non\-gradient signals, we average over query\-token positions and aggregate over calibration examples\. As in HeadGuard, all score\-based baselines then average scores across query heads sharing a physical KV head before selecting the top\-KKKV heads\. We also compare against random head selection on discriminative tasks\. Implementation details for all selectors, including score aggregation and head selection, are provided in the code supplement\.

Data splits, metrics, and averaging\.On POPE, head scores are calibrated once, on100100questions drawn in a fixed shuffled order\. For every base, main K\+V and K\-only results exclude those calibration questions and the same88questions reserved for preliminary diagnostic checks, leaving88928892\. Calibration and evaluation questions are disjoint\.

After filtering and holdouts, the numbers of examples used for main discriminative evaluation are: POPE88928892, MME23662366, MMBench43214321, ScienceQA20092009, MMMU839839, and SEED14,22514\{,\}225\. The capped budget sweep uses up to10001000evaluation questions per discriminative task and200200examples per generative task\. In the capped sweep, we evaluate on900900held\-out POPE questions when calibrating on POPE, and900900held\-out MMBench questions when calibrating on MMBench\. These evaluation sets exclude the100100questions used for calibration\. Within each setting, the held\-out set is identical across the three quantizer bases\. For the discriminative tasks we score each example from the model’s logits on the candidate answer tokens \(yes/no or the option letters\); the same scoring rule is used across models and bases\. TextVQA uses a VQA\-style match to the human answer annotations with case, punctuation, and article normalization and a token\-containment fallback \(not the official VQA scorer\)\. Captioning uses case\-sensitive whitespace\-token ROUGE\-L F1\([Lin, 2004](https://arxiv.org/html/2609.35800#bib.bib13)\)against BF16 output, not human captions, with no lowercasing or punctuation removal; BF16 scores11by convention\. We never average these generative scores with classification accuracy\. Greedy decoding generates at most1212new tokens for TextVQA and6464for captions, excluding input tokens and stopping earlier at an end\-of\-sequence token\.

For each model, we average accuracy across the six discriminative tasks; the overall mean then gives each model equal weight\. These averages use task scores rounded to three decimal places, with final rounding to the nearest reported digit \(ties rounded up\)\. Overall means and gains use the aggregates before this final rounding\.

Preprocessing\.We shrink each image so its longest side is at most336336pixels for LLaVA and768768for Qwen, Pixtral, and InternVL, turn off Gemma’s image\-cropping mode \(pan\-and\-scan\), and limit InternVL to one image tile; each image therefore yields a bounded number of tokens\. Gemma mixes short\-range and long\-range attention, so we cap the image length and do not test very long image inputs\.

## 5When Does Head Protection Help?

### 5\.1Recovery depends on the model and precision

Table[1](https://arxiv.org/html/2609.35800#S5.T1)reports K\+V protection on the per\-token min–max base, averaged over all six discriminative tasks for all eight models\. At22bits, HeadGuard lifts the eight\-model mean from0\.4360\.436to0\.5800\.580—a14\.414\.4\-point gain\. Qwen2\.5\-VL rises from0\.3600\.360to0\.7470\.747and InternVL3 from0\.3620\.362to0\.6880\.688, while Pixtral recovers only part of its loss \(from0\.5380\.538to0\.6420\.642; BF160\.7700\.770\)\. Neither Gemma model recovers from22\-bit min–max degradation under this mask\. At33bits, the unprotected mean is0\.5640\.564, but Qwen and InternVL remain fragile; protection raises the overall mean to0\.7240\.724\. Gemma\-3 also improves \(from0\.6510\.651to0\.6930\.693\), whereas LLaVA and Pixtral change little\.

Table 1:Head protection recovers low\-bit min–max accuracy, unevenly\.HeadGuard protects selected image keys and values on the per\-token min–max base, averaged over POPE/MME/MMBench/ScienceQA/MMMU/SEED\.K≈NKV/8K\\approx N\_\{\\rm KV\}/8\. BF16 is a reference, not an upper bound\.
### 5\.2Base quality matters more than recovery alone

Table[2](https://arxiv.org/html/2609.35800#S5.T2)compares all three bases on the same set of model–task cells\. At22bits, HeadGuard gains14\.3714\.37points on min–max,12\.7312\.73on AKVQ\-style, and0\.850\.85on KIVI\-style\. Average gains are larger on the weaker bases\. KIVI\-style is robust on most models, leaving limited headroom for further gains from protection\. A large gain does not guarantee the best absolute accuracy: protected min–max trails*unprotected*KIVI\-style by10\.6710\.67points at22bits, so a repaired weak base can still lose to a strong one\. Head protection is therefore a targeted repair, not a substitute for choosing a strong base\.

Table 2:K\+V protection: base accuracy and six\-task selector gains\.Uniform gives each base’s absolute six\-task accuracy; the remaining columns are percentage\-point gains over that uniform arm,K≈NKV/8K\\approx N\_\{\\rm KV\}/8\. Each row averages the six tasks and eight models equally, using full eligible task evaluation\. Image and Output use only the image\-sensitivity scoreggand output\-sensitivity scoreoo, respectively; HeadGuard combines them ass=z⁡\(g¯\)\+z⁡\(o¯\)s=z\(\\bar\{g\}\)\+z\(\\bar\{o\}\)\. Magn\., Attn\., and Vis\. use activation magnitude, image\-attention mass, and visual\-response norm\. Random head selection is reported for the min–max base only\. These are descriptive means, not significance tests; small differences can reflect rounding\.
### 5\.3The image\-sensitivity increment

Output sensitivity is already a strong selector: at22\-bit min–max it recovers13\.3413\.34points over uniform, versus1\.391\.39for random head selection \(Table[2](https://arxiv.org/html/2609.35800#S5.T2)\)\. Adding image sensitivity raises recovery to14\.3714\.37points, an additional1\.031\.03points averaged over all six tasks\. Image sensitivity alone recovers2\.572\.57points\.

On POPE with22\-bit min–max, HeadGuard improves accuracy over output\-only head selection by1\.81\.8–2\.92\.9percentage points for five of the eight models \(Table[7](https://arxiv.org/html/2609.35800#A3.T7)\)\. The95%95\\%confidence intervals for these five gains exclude zero\. We compute these intervals using10,00010\{,\}000paired image\-cluster bootstrap resamples of the full eligible POPE evaluation set\. Each resample compares the two methods’ saved predictions on the same questions\. The heads protected by each method are chosen before evaluation and are not selected again during bootstrapping \(Appendix[C](https://arxiv.org/html/2609.35800#A3)\)\. At33bits, hybrid and output\-only selection achieve comparable six\-task min–max recovery:15\.9615\.96and15\.9715\.97points, respectively\.

### 5\.4Keys\-only protection retains substantial recovery

Protecting only the selected image*keys*in BF16, while quantizing their values, retains substantial accuracy recovery\. It gives mean22\-bit gains of14\.5614\.56points on min–max,13\.9513\.95on AKVQ\-style, and0\.570\.57on KIVI\-style \(Table[3](https://arxiv.org/html/2609.35800#S5.T3)\)\. Min–max Qwen2\.5\-VL improves from0\.3660\.366to0\.7510\.751, while both Gemma models show little recovery \(per\-model results in Table[6](https://arxiv.org/html/2609.35800#A2.T6)\)\.

Table 3:K\-only results across classification and generation,22bits\.Uniform has no protection; HG\-K is keys\-only HeadGuard atK≈NKV/8K\\approx N\_\{\\rm KV\}/8\. Each metric averages all eight models equally; six\-task accuracy also weights the tasks equally\. Entries are scores, not gains, using the metrics in Section[4](https://arxiv.org/html/2609.35800#S4)\.For22\-bit min–max, we estimate image\-cache storage assuming equal128128\-dimensional K/V heads and a1/81/8protected fraction\. Including3232bits of quantization metadata per quantized key or value vector, the estimates are2\.252\.25,3\.973\.97, and3\.113\.11bits per element for uniform quantization, K\+V protection, and K\-only protection, respectively\. For the same selected heads, K\-only leaves their image values quantized, halving the additional storage required for protection and using about22%22\\%less modeled image\-cache storage than K\+V\. These estimates describe the storage needed for the image\-token KV cache, rather than measured peak memory usage during end\-to\-end inference\.

At22bits, K\-only and K\+V protection achieve similar mean accuracies across the six discriminative tasks and eight models for each base quantizer\.

### 5\.5Protection can also help generation

We also test generated answers on TextVQA and image captions\. K\+V protection tests cover all eight models and three base quantizers at22and33bits\. With22\-bit min–max, K\+V HeadGuard raises Qwen2\.5\-VL’s TextVQA score from0\.0150\.015to0\.6870\.687\. With33\-bit min–max, Qwen2\-VL’s score rises from0\.0220\.022to0\.7870\.787, close to its BF16 score of0\.7900\.790\. Even with K\+V HeadGuard, both Gemma models score poorly on TextVQA under22\-bit min–max quantization\. For captioning with22\-bit min–max, HeadGuard raises the eight\-model mean ROUGE\-L against BF16 output from0\.1710\.171to0\.3120\.312\.

Keys\-only protection is tested on all three bases at22bits\. For Qwen2\.5\-VL, it raises the TextVQA score from0\.0090\.009to0\.6900\.690on min–max, but lowers it from0\.7330\.733to0\.7240\.724on KIVI\-style\. Protection can therefore help generated answers, but not for every model and base\. Table[3](https://arxiv.org/html/2609.35800#S5.T3)summarizes K\-only results across all eight models\. Appendix[D](https://arxiv.org/html/2609.35800#A4)reports all results\.

### 5\.6Ablation: budget, precision, and calibration

We compare head budgets, precision, and calibration on capped task subsets\. For the precision and calibration comparisons, K\+V protection is evaluated at all four head budgets \(1/161/16,1/121/12,1/81/8, and1/41/4\)\. Uniform baselines protect no heads\. At the1/81/8budget, Appendix[E](https://arxiv.org/html/2609.35800#A5)reports six\-task accuracy and Appendix[F](https://arxiv.org/html/2609.35800#A6)reports TextVQA and captioning results\. Appendix[G](https://arxiv.org/html/2609.35800#A7)covers the other three head budgets\.

Figure[1](https://arxiv.org/html/2609.35800#S5.F1)uses all six tasks of the capped min–max sweep\. Larger protected fractions improve Qwen2\-VL and InternVL3\. For Gemma, increasing min–max precision from22to33bits is more effective than expanding the tested22\-bit head budget up to1/41/4\.

Increasing uniform min–max precision from22to33bits raises Gemma\-3’s six\-task mean from0\.3710\.371to0\.6560\.656and Gemma\-4’s from0\.3730\.373to0\.7080\.708\. For Qwen2\-VL, uniform44\-bit min–max still gives0\.3660\.366, versus0\.7350\.735with protection at the one\-eighth head budget \(Table[16](https://arxiv.org/html/2609.35800#A5.T16)\)\.

Figure 1:Accuracy as more heads are protected\.Selected image keys and values stay in BF16; the rest use22\-bit min–max\. Each line shows one model’s average accuracy across six tasks \(POPE, MME, MMBench, ScienceQA, MMMU, and SEED\-Bench\), using up to10001000questions per task \(900900for POPE;839839for MMMU\)\. HeadGuard’s ranking is fixed using POPE calibration\. Zero means no protection; fractions are rounded to whole head counts\. The vertical axis shows accuracy, not gains\. More protection helps some models, but both Gemma models remain close to their unprotected scores under this ranking\.At the one\-eighth head budget, K\+V HeadGuard on min–max raises the six\-task, eight\-model mean accuracy over uniform by14\.1114\.11versus12\.6512\.65percentage points at22bits and15\.7915\.79versus14\.3914\.39at33bits, for POPE versus MMBench calibration, respectively \(Tables[16](https://arxiv.org/html/2609.35800#A5.T16)and[17](https://arxiv.org/html/2609.35800#A5.T17)\)\. The image\-sensitivity contribution varies between conditions\.

## 6Discussion and Conclusion

HeadGuard shows that protecting a small subset of image KV heads can substantially recover low\-bit VLM accuracy without replacing the underlying quantizer\. Its offline sensitivity scores produce a fixed protection mask that is reused across tasks, retaining all tokens and requiring neither weight updates nor inference\-time gradients\. Recovery is particularly strong for Qwen and InternVL on weak bases\. Output sensitivity explains most recovery, while the additional benefit of image sensitivity depends on the model and calibration\.

The benefits extend beyond classification to generated answers and caption fidelity to BF16 outputs\. Keys\-only protection retains substantial recovery at a lower modeled storage cost, offering a more economical protection option\. These findings suggest a practical strategy: choose a strong base quantizer, then use selective head protection only when its accuracy benefit justifies the added modeled storage cost\. Overall, HeadGuard demonstrates that a small, targeted high\-precision allocation can improve the accuracy–storage trade\-off of low\-bit image KV quantization\.

Scope and limitations\.Results use fixed image\-size limits, custom scoring, and simplified KIVI\-style and AKVQ\-style bases\. We compare POPE and MMBench calibration, each with100100examples, but do not test larger calibration sets\. More calibration examples could further improve head selection and accuracy recovery, but this possibility remains untested\. Model comparisons do not isolate architectural causes of quantization sensitivity\. The main experiments simulate quantization and estimate storage; end\-to\-end memory use and speed with optimized packed caches remain future work\. Because text KV remains in BF16, total\-cache compression diminishes as the text fraction grows; long\-form and multi\-turn generation are not evaluated\.

## Reproducibility Statement

Full implementation details are provided in the anonymous code supplement, including experiment\-specific runners, evaluation and scoring code, frozen calibration scores, rounded reference results, aggregation/bootstrap code, pinned dependencies, and unit tests for the custom TextVQA and caption\-fidelity metrics\. Sections[3](https://arxiv.org/html/2609.35800#S3)and[4](https://arxiv.org/html/2609.35800#S4)specify the method, data splits, scoring, and averaging conventions\. Data shuffling uses a fixed seed, and the released analysis code reproduces the reported aggregates from the bundled reference results\.

Experiments were conducted on NVIDIA H100 and A100 GPUs\. Within each experimental protocol—main K\+V evaluation, K\-only evaluation, and the capped budget sweep—the same GPU was used for all models and measurements\. HeadGuard’s gains are measured within each evaluation by comparing the same model and KV\-cache quantization method with and without selective head protection\. Comparisons of absolute accuracy across protocols describe trends rather than quantify HeadGuard’s gains\.

## Ethics Statement

The study uses existing public models and benchmarks and collects no new human data\. Model and dataset licenses and privacy limitations still apply\. Quantized VLMs can hallucinate; improved benchmark accuracy does not establish safety in high\-stakes deployment\.

## AI Use Statement

The authors used generative AI assistants \(ChatGPT and Claude\) for a task with required disclosure: writing software code that implements methods \(scripts for evaluation, results aggregation, and figure rendering\)\. We additionally used these tools for recommended\-disclosure tasks: editing software code, creating figures, drafting parts of the manuscript \(including descriptions of results\), editing for grammar and readability, and formatting references\. Generative AI was not used to generate synthetic or experimental data, develop theoretical results or proofs, propose or refine hypotheses, or design the methodology\. All code, numerical results, figures, scientific claims, conclusions, and their interpretation were reviewed, validated, and finalized by the authors, who take full responsibility for the content of this article\.

## References

- Agrawal et al\. \(2024\)Pravesh Agrawal et al\. Pixtral 12B\.*arXiv preprint arXiv:2410\.07073*, 2024\.[https://arxiv\.org/abs/2410\.07073](https://arxiv.org/abs/2410.07073)\.
- Ainslie et al\. \(2023\)Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\. GQA: Training Generalized Multi\-Query Transformer Models from Multi\-Head Checkpoints\. In*EMNLP*, 2023\.[https://arxiv\.org/abs/2305\.13245](https://arxiv.org/abs/2305.13245)\.
- Bai et al\. \(2025\)Shuai Bai et al\. Qwen2\.5\-VL Technical Report\.*arXiv preprint arXiv:2502\.13923*, 2025\.[https://arxiv\.org/abs/2502\.13923](https://arxiv.org/abs/2502.13923)\.
- Chen et al\. \(2015\)Xinlei Chen et al\. Microsoft COCO Captions: Data Collection and Evaluation Server\.*arXiv preprint arXiv:1504\.00325*, 2015\.[https://arxiv\.org/abs/1504\.00325](https://arxiv.org/abs/1504.00325)\.
- Fu et al\. \(2025a\)Chaoyou Fu et al\. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models\. In*NeurIPS Datasets and Benchmarks Track*, 2025a\.[https://openreview\.net/forum?id=DgH9YCsqWm](https://openreview.net/forum?id=DgH9YCsqWm)\.
- Fu et al\. \(2025b\)Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao\. Not All Heads Matter: A Head\-Level KV Cache Compression Method with Integrated Retrieval and Reasoning\. In*ICLR*, 2025b\.[https://arxiv\.org/abs/2410\.19258](https://arxiv.org/abs/2410.19258)\.
- Gemma Team \(2025\)Gemma Team\. Gemma 3 Technical Report\.*arXiv preprint arXiv:2503\.19786*, 2025\.[https://arxiv\.org/abs/2503\.19786](https://arxiv.org/abs/2503.19786)\.
- Gemma Team \(2026\)Gemma Team\. Gemma 4 Technical Report\.*arXiv preprint arXiv:2607\.02770*, 2026\.[https://arxiv\.org/abs/2607\.02770](https://arxiv.org/abs/2607.02770)\.
- Hooper et al\. \(2024\)Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W\. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami\. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization\. In*NeurIPS*, 2024\.[https://arxiv\.org/abs/2401\.18079](https://arxiv.org/abs/2401.18079)\.
- Li et al\. \(2024\)Bohao Li et al\. SEED\-Bench: Benchmarking Multimodal Large Language Models\. In*CVPR*, 2024\.[https://openaccess\.thecvf\.com/content/CVPR2024/html/Li\_SEED\-Bench\_Benchmarking\_Multimodal\_Large\_Language\_Models\_CVPR\_2024\_paper\.html](https://openaccess.thecvf.com/content/CVPR2024/html/Li_SEED-Bench_Benchmarking_Multimodal_Large_Language_Models_CVPR_2024_paper.html)\.
- Li et al\. \(2026\)Fei Li, Song Liu, Weiguo Wu, Shiqiang Nie, and Jinyu Wang\. KVmix: Gradient\-Based Layer Importance\-Aware Mixed\-Precision Quantization for KV Cache\. In*AAAI*, 2026\.[https://arxiv\.org/abs/2506\.08018](https://arxiv.org/abs/2506.08018)\.
- Li et al\. \(2023\)Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji\-Rong Wen\. Evaluating Object Hallucination in Large Vision\-Language Models\. In*EMNLP*, 2023\.[https://arxiv\.org/abs/2305\.10355](https://arxiv.org/abs/2305.10355)\.
- Lin \(2004\)Chin\-Yew Lin\. ROUGE: A Package for Automatic Evaluation of Summaries\. In*Text Summarization Branches Out*, pp\. 74–81, 2004\.[https://aclanthology\.org/W04\-1013/](https://aclanthology.org/W04-1013/)\.
- Liu et al\. \(2023\)Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee\. Visual Instruction Tuning\. In*NeurIPS*, 2023\.[https://arxiv\.org/abs/2304\.08485](https://arxiv.org/abs/2304.08485)\.
- Liu et al\. \(2024a\)Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee\. Improved Baselines with Visual Instruction Tuning\. In*CVPR*, 2024a\.[https://arxiv\.org/abs/2310\.03744](https://arxiv.org/abs/2310.03744)\.
- Liu et al\. \(2024b\)Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee\. LLaVA\-NeXT: Improved reasoning, OCR, and world knowledge\. Technical blog, January 2024b\.[https://llava\-vl\.github\.io/blog/2024\-01\-30\-llava\-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/)\.
- Liu et al\. \(2024c\)Yuan Liu et al\. MMBench: Is Your Multi\-modal Model an All\-around Player? In*ECCV*, 2024c\.[https://arxiv\.org/abs/2307\.06281](https://arxiv.org/abs/2307.06281)\.
- Liu et al\. \(2024d\)Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\. KIVI: A Tuning\-Free Asymmetric 2bit Quantization for KV Cache\. In*ICML*, 2024d\.[https://arxiv\.org/abs/2402\.02750](https://arxiv.org/abs/2402.02750)\.
- Lu et al\. \(2022\)Pan Lu et al\. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering\. In*NeurIPS*, 2022\.[https://arxiv\.org/abs/2209\.09513](https://arxiv.org/abs/2209.09513)\.
- Michel et al\. \(2019\)Paul Michel, Omer Levy, and Graham Neubig\. Are Sixteen Heads Really Better than One? In*NeurIPS*, 2019\.[https://arxiv\.org/abs/1905\.10650](https://arxiv.org/abs/1905.10650)\.
- Singh et al\. \(2019\)Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach\. Towards VQA Models That Can Read\. In*CVPR*, 2019\.[https://arxiv\.org/abs/1904\.08920](https://arxiv.org/abs/1904.08920)\.
- Su et al\. \(2025\)Zunhai Su, Wang Shen, Linge Li, Zhe Chen, Hanyu Wei, Huangqi Yu, and Kehong Yuan\. AKVQ\-VL: Attention\-Aware KV Cache Adaptive 2\-Bit Quantization for Vision\-Language Models\. In*IEEE International Conference on Multimedia and Expo \(ICME\)*, pp\. 1–6, 2025\.[https://doi\.org/10\.1109/ICME59968\.2025\.11209367](https://doi.org/10.1109/ICME59968.2025.11209367)\.
- Wang et al\. \(2025\)Jiahui Wang, Zuyan Liu, Yongming Rao, and Jiwen Lu\. SparseMM: Head Sparsity Emerges from Visual Concept Responses in MLLMs\. In*ICCV*, 2025\.[https://arxiv\.org/abs/2506\.05344](https://arxiv.org/abs/2506.05344)\.
- Wang et al\. \(2024\)Peng Wang et al\. Qwen2\-VL: Enhancing Vision\-Language Model’s Perception of the World at Any Resolution\.*arXiv preprint arXiv:2409\.12191*, 2024\.[https://arxiv\.org/abs/2409\.12191](https://arxiv.org/abs/2409.12191)\.
- Wen et al\. \(2025\)Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang\. Stop Looking for “Important Tokens” in Multimodal Language Models: Duplication Matters More\. In*EMNLP*, pp\. 9961–9980, 2025\.[https://aclanthology\.org/2025\.emnlp\-main\.505/](https://aclanthology.org/2025.emnlp-main.505/)\.
- Xiao et al\. \(2025\)Guangxuan Xiao, Jiaming Tang, Jingwei Zuo, Junxian Guo, Shang Yang, Haotian Tang, Yao Fu, and Song Han\. DuoAttention: Efficient Long\-Context LLM Inference with Retrieval and Streaming Heads\. In*ICLR*, 2025\.[https://openreview\.net/forum?id=cFu7ze7xUm](https://openreview.net/forum?id=cFu7ze7xUm)\.
- Yue et al\. \(2024\)Xiang Yue et al\. MMMU: A Massive Multi\-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI\. In*CVPR*, 2024\.[https://arxiv\.org/abs/2311\.16502](https://arxiv.org/abs/2311.16502)\.
- Zhu et al\. \(2025\)Jinguo Zhu et al\. InternVL3: Exploring Advanced Training and Test\-Time Recipes for Open\-Source Multimodal Models\.*arXiv preprint arXiv:2504\.10479*, 2025\.[https://arxiv\.org/abs/2504\.10479](https://arxiv.org/abs/2504.10479)\.
- Zuo et al\. \(2026\)Fei Zuo, Zikang Zhou, Hao Cong, Xiaoyan Xi, and Ho Fai Leung\. RateQuant: Optimal Mixed\-Precision KV Cache Quantization via Rate\-Distortion Theory\.*arXiv preprint arXiv:2605\.06675*, 2026\.[https://arxiv\.org/abs/2605\.06675](https://arxiv.org/abs/2605.06675)\.

## Appendix

Detailed results follow the protocols in Section[4](https://arxiv.org/html/2609.35800#S4)\.

## Appendix AKey\-and\-Value Protection at Two and Three Bits

Tables[4](https://arxiv.org/html/2609.35800#A1.T4)and[5](https://arxiv.org/html/2609.35800#A1.T5)give per\-task min–max results\. U denotes uniform quantization; HG denotes HeadGuard\.

Table 4:K\+V protection, min–max at22bits, all six tasks\. Each pair is uniform/HG;K≈NKV/8K\\approx N\_\{\\rm KV\}/8\. Counts are8892/2366/4321/2009/839/142258892/2366/4321/2009/839/14225in column order\.Table 5:K\+V protection, min–max at33bits, same complete six\-task block and column conventions as Table[4](https://arxiv.org/html/2609.35800#A1.T4)\.
## Appendix BKeys\-only Discriminative Evaluation

Table[6](https://arxiv.org/html/2609.35800#A2.T6)gives the per\-model means for keys\-only protection on the full eligible discriminative splits, separately from K\+V evaluation\.

Table 6:K\-only evaluation at22bits, all six discriminative tasks\.K≈NKV/8K\\approx N\_\{\\rm KV\}/8\. Each pair gives accuracy without protection \(Uniform\) and with keys\-only HeadGuard \(HG\-K\), not gains\.
## Appendix CPOPE Uncertainty in Full Evaluation

POPE is used here as an evaluation benchmark, not only for calibration\. The bootstrap uses saved predictions on the full eligible POPE evaluation set, excluding calibration and diagnostic questions\.

Tables[7](https://arxiv.org/html/2609.35800#A3.T7)–[9](https://arxiv.org/html/2609.35800#A3.T9)use the same paired image\-cluster percentile bootstrap for95%95\\%intervals:10,00010\{,\}000draws with seed00, keeping each image’s questions together\. Entries are HeadGuard’s accuracy gains over the named baseline, in percentage points, with pointwise95%95\\%confidence intervals in brackets\. Positive values favor HeadGuard; negative values favor the baseline\. An interval spanning zero does not establish a clear difference\.

Table 7:K\+V protection,22\-bit min–max POPE: paired image\-cluster95%95\\%intervals,K≈NKV/8K\\approx N\_\{\\rm KV\}/8\. Gains and intervals are in percentage points, not absolute accuracies\.Table 8:K\+V protection,33\-bit min–max POPE: same paired image\-cluster bootstrap and head budget as Table[7](https://arxiv.org/html/2609.35800#A3.T7)\. Gains and intervals are in percentage points, not absolute accuracies\.Table 9:K\-only evaluation,22\-bit min–max POPE: same paired image\-cluster bootstrap and head budget as Table[7](https://arxiv.org/html/2609.35800#A3.T7)\. Gains and intervals are in percentage points, not absolute accuracies\. Contrasts compare selectors within K\-only evaluation, not K\-only against K\+V\.
## Appendix DGenerative Results

K\+V evaluation reports all model–base pairs for each generative task at both precisions\. K\-only evaluation reports TextVQA and caption pairs\. TextVQA uses10001000rows and captioning200200; scores are descriptive and use the custom metrics defined above\. U denotes no head protection, HG denotes HeadGuard, and HG\-K denotes keys\-only HeadGuard\. Entries are task scores, not gains: higher is better for TextVQA; higher caption fidelity means closer agreement with BF16 output, not necessarily better captions\.

Table 10:K\+V protection on TextVQA,22bits\.Uniform/HG pairs; all eight models are retained\.Table 11:K\+V protection on TextVQA,33bits\.Same conventions as Table[10](https://arxiv.org/html/2609.35800#A4.T10)\.Table 12:K\+V protection for captioning,22bits\.ROUGE\-L overlap with BF16 output, not human\-caption quality\. Each pair is uniform/HG on200200examples\.Table 13:K\+V protection for captioning,33bits\.Same scope and conventions as Table[12](https://arxiv.org/html/2609.35800#A4.T12)\.Table 14:K\-only TextVQA evaluation,22bits\.Each pair is uniform/HG\-K on10001000questions\.Table 15:K\-only caption evaluation,22bits\.200200examples per cell; each pair is uniform/HG\-K\.
## Appendix ECapped Budget Sweep

At22\-bit min–max with the one\-eighth head budget, HeadGuard improves mean accuracy over uniform by14\.1114\.11percentage points under POPE calibration and12\.6512\.65under MMBench calibration \(Tables[16](https://arxiv.org/html/2609.35800#A5.T16)and[17](https://arxiv.org/html/2609.35800#A5.T17)\)\.

In this section and Appendix[F](https://arxiv.org/html/2609.35800#A6), U = uniform; Out = output\-only selection; HG = hybrid HeadGuard\. Out uses only output sensitivity, whereas HG combines image\-sensitivity and output\-sensitivity scores\. Entries are accuracies or task scores, not gains\. Mean weights all eight models equally within each metric, base, and precision\.

Table 16:Capped sweep with POPE calibration\.Each score averages POPE, MME, MMBench, ScienceQA, MMMU, and SEED, using up to10001000questions per task \(900900for POPE;839839for MMMU\)\. K\+V protection usesK=round⁡\(NKV/8\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/8\)\. U = uniform; Out = output\-only selection; HG = hybrid HeadGuard\. Mean averages all eight models\. Values are accuracies, not gains; gains in the main text are calculated before final rounding\.Table 17:Capped sweep with MMBench calibration\.Same six evaluation tasks, one\-eighth head budget, and column definitions as Table[16](https://arxiv.org/html/2609.35800#A5.T16)\. Here MMBench uses900900held\-out questions, POPE10001000, and MMMU839839\. Mean averages all eight models\.
## Appendix FCapped Generative Evaluation

Tables[18](https://arxiv.org/html/2609.35800#A6.T18)–[21](https://arxiv.org/html/2609.35800#A6.T21)report TextVQA and captioning separately under both calibrations, using200200examples per task and K\+V protection atK=round⁡\(NKV/8\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/8\)\. Scores use the custom metrics in Section[4](https://arxiv.org/html/2609.35800#S4)and are descriptive; neither generative metric is pooled with the six\-task accuracy mean\.

Table 18:Capped TextVQA with POPE calibration\.Custom VQA\-style scores on200200questions, K\+V protection at the one\-eighth head budget\. Mean averages all eight models\.Table 19:Capped captioning with POPE calibration\.ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. K\+V protection uses the one\-eighth head budget\. Mean averages all eight models\.Table 20:Capped TextVQA with MMBench calibration\.Same200200\-question scope, metric, head budget, and columns as Table[18](https://arxiv.org/html/2609.35800#A6.T18)\. Mean averages all eight models\.Table 21:Capped captioning with MMBench calibration\.Same200200\-example scope, BF16\-output ROUGE\-L metric, head budget, and columns as Table[19](https://arxiv.org/html/2609.35800#A6.T19)\. Mean averages all eight models\.
## Appendix GAdditional Capped Head Budgets

The following tables extend the capped results to protected\-head fractions1/161/16,1/121/12, and1/41/4under POPE and MMBench calibration\. Six\-task accuracy, TextVQA, and caption fidelity are reported separately; each Mean uses all eight models within its metric, base, precision, and budget\. The fractions are nominal: the protected head count is rounded for each model\. For six\-task evaluation, the calibration benchmark uses900900held\-out questions, MMMU839839, and the other tasks10001000, consistently across bases\. Throughout these tables, U means no head protection, Out means output\-only selection, and HG means HeadGuard \(image\-sensitivity plus output\-sensitivity scores\)\. Entries are accuracies or task scores, not gains\.

Table 22:Six\-task accuracy,1/161/16budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/16\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/16\)\. Equal\-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to10001000questions/task \(900900for POPE;839839for MMMU\)\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 23:TextVQA,1/161/16budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/16\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/16\)\. Custom VQA\-style score on200200questions\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 24:Caption fidelity,1/161/16budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/16\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/16\)\. ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 25:Six\-task accuracy,1/161/16budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/16\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/16\)\. Equal\-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to10001000questions/task \(900900for MMBench;839839for MMMU\)\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 26:TextVQA,1/161/16budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/16\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/16\)\. Custom VQA\-style score on200200questions\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 27:Caption fidelity,1/161/16budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/16\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/16\)\. ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 28:Six\-task accuracy,1/121/12budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/12\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/12\)\. Equal\-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to10001000questions/task \(900900for POPE;839839for MMMU\)\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 29:TextVQA,1/121/12budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/12\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/12\)\. Custom VQA\-style score on200200questions\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 30:Caption fidelity,1/121/12budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/12\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/12\)\. ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 31:Six\-task accuracy,1/121/12budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/12\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/12\)\. Equal\-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to10001000questions/task \(900900for MMBench;839839for MMMU\)\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 32:TextVQA,1/121/12budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/12\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/12\)\. Custom VQA\-style score on200200questions\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 33:Caption fidelity,1/121/12budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/12\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/12\)\. ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 34:Six\-task accuracy,1/41/4budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/4\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/4\)\. Equal\-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to10001000questions/task \(900900for POPE;839839for MMMU\)\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 35:TextVQA,1/41/4budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/4\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/4\)\. Custom VQA\-style score on200200questions\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 36:Caption fidelity,1/41/4budget, POPE calibration\.K\+V protection withK=round⁡\(NKV/4\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/4\)\. ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 37:Six\-task accuracy,1/41/4budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/4\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/4\)\. Equal\-weight accuracy over POPE, MME, MMBench, ScienceQA, MMMU, and SEED; up to10001000questions/task \(900900for MMBench;839839for MMMU\)\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 38:TextVQA,1/41/4budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/4\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/4\)\. Custom VQA\-style score on200200questions\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.Table 39:Caption fidelity,1/41/4budget, MMBench calibration\.K\+V protection withK=round⁡\(NKV/4\)K=\\operatorname\{round\}\(N\_\{\\rm KV\}/4\)\. ROUGE\-L F1 against BF16 output on200200examples, not human\-caption quality\. U = uniform; Out = output\-only; HG = hybrid\. Mean averages all eight models\.