SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

# Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation
Source: [https://arxiv.org/html/2607.16213](https://arxiv.org/html/2607.16213)
Soumia Bouyahiaoui National School of Artificial Intelligence \(ENSIA\), Sidi Abdellah Campus, Algiers, Algeria soumia\.bouyahiaoui@ensia\.edu\.dz &Manel Kara laouar National School of Artificial Intelligence \(ENSIA\), Sidi Abdellah Campus, Algiers, Algeria manel\.karalaouar@ensia\.edu\.dz &Aicha Boutorh National School of Artificial Intelligence \(ENSIA\), Sidi Abdellah Campus, Algiers, Algeria aicha\.boutorh@ensia\.edu\.dz &Mohamed Hadj Ameur National School of Artificial Intelligence \(ENSIA\), Sidi Abdellah Campus, Algiers, Algeria mohamed\.hadj\.ameur@ensia\.edu\.dz

###### Abstract

Large Language Models \(LLMs\) generate text autoregressively, relying on a key\-value \(KV\) cache whose memory footprint grows linearly with context length, creating a major bottleneck\. Recent compression methods mitigate this cost via token merging; however, these approaches often rely on indiscriminate aggregation, which degrades representations and introduces*attention sag*, a mismatch where merged tokens receive the same softmax mass as individual tokens despite encoding multiple inputs\. We propose a training\-free, dual\-component framework for KV cache compression that addresses these limitations\. First, a*soft cosine gate*adaptively modulates merging decisions based on value\-vector similarity, suppressing or discarding dissimilar tokens to preserve semantic fidelity\. Second, we introduce an*attention\-ratio*compensation mechanism that applies a decoding\-time logit bias derived from prefill attention statistics, correcting the softmax imbalance induced by merging\. Evaluated on LongBench \(16 English datasets\) while retaining only 25% of the KV cache, our framework achieves strong compressed performance against representative one\-shot baselines\. It is especially robust on the evaluated grouped\-query attention \(GQA\) models, maintaining near\-lossless generation quality\. Furthermore, the method outperforms the full\-cache baseline on complex multi\-document QA tasks and delivers a 3\.3x decoding speedup at 100k tokens\.

*K*eywordsKV cache compression, token merging, attention compensation, large language models, inference efficiency

## 1Introduction

Large language models \(LLMs\) achieve strong performance across diverse tasks, but their autoregressive inference is bottlenecked by the key\-value \(KV\) cache, which stores past key and value vectors for attention computation\. For a model withLLlayers andHHattention heads of dimensiondd, the KV cache grows asO​\(L⋅H⋅d⋅n\)O\(L\\cdot H\\cdot d\\cdot n\)with sequence lengthnn, consuming multiple gigabytes even at moderate context lengths\. As applications demand longer contexts, KV cache memory has become a primary constraint on batch size, throughput, and deployment cost\.

Two families of methods have emerged for KV cache compression\.*Eviction*methods\[[19](https://arxiv.org/html/2607.16213#bib.bib6),[7](https://arxiv.org/html/2607.16213#bib.bib5),[15](https://arxiv.org/html/2607.16213#bib.bib4),[3](https://arxiv.org/html/2607.16213#bib.bib8)\]permanently discard low\-importance tokens, reducing cache size but irreversibly losing the information they carry\.*Merging*methods such as KVMerger\[[14](https://arxiv.org/html/2607.16213#bib.bib9)\], WeightedKV\[[17](https://arxiv.org/html/2607.16213#bib.bib10)\], D2O\[[12](https://arxiv.org/html/2607.16213#bib.bib11)\], and LOOK\-M\[[13](https://arxiv.org/html/2607.16213#bib.bib12)\]instead consolidate evicted tokens into retained cache entries, which can preserve more semantic content when information is distributed across many tokens\.

We focus on two limitations of current merging methods:

#### Problem 1: The uniform merge\-or\-drop decision\.

Most methods apply the same strategy to*all*evicted tokens: either merge all of them \(KVMerger, WeightedKV, LOOK\-M\) or drop all of them\[[19](https://arxiv.org/html/2607.16213#bib.bib6),[7](https://arxiv.org/html/2607.16213#bib.bib5),[15](https://arxiv.org/html/2607.16213#bib.bib4)\]\. D2O\[[12](https://arxiv.org/html/2607.16213#bib.bib11)\]partitions tokens into drop and merge sets based on attention patterns, but this is a binary decision that still merges every token assigned to the merge set at full weight\. No existing method modulates*how much*to merge on a continuous, per\-token basis using representation similarity\. When an evicted token’s value vector is dissimilar to its merge target, the weighted average can corrupt the retained representation\. Our experiments illustrate this trade\-off: uniform merging improves multi\-document QA by\+0\.92\+0\.92points on HotpotQA but degrades few\-shot classification by−4\.0\-4\.0points on TREC\. EMS\[[18](https://arxiv.org/html/2607.16213#bib.bib14)\]similarly notes that*“both over\-merging and over\-evicting yield sub\-optimal performance,”*yet does not provide a per\-token mechanism\.

#### Problem 2: Attention sag\.

When multiple tokens are merged into a single cache entry, that entry still receives roughly the same softmax attention as an unmerged token, despite representing several originals\. KeepKV\[[11](https://arxiv.org/html/2607.16213#bib.bib15)\]formalizes this as*attention sag*\(Theorem 3\.2\): a position representingmmmerged tokens is systematically under\-attended by a factor of∼\\simmm\. As a result, many existing merging methods likely underuse the information they preserve\[[14](https://arxiv.org/html/2607.16213#bib.bib9),[17](https://arxiv.org/html/2607.16213#bib.bib10),[12](https://arxiv.org/html/2607.16213#bib.bib11),[8](https://arxiv.org/html/2607.16213#bib.bib13)\]\.

We address both problems with SelKV \(Selective KV Cache Merging\), a training\-free framework with two complementary mechanisms that can be integrated into existing merging pipelines:

1. 1\.Soft cosine gate\(Problem 1\): For each evicted token, we compute the cosine similarity between its value vector and its merge target\. The gateg=max⁡\(cos​\_​sim,0\)g=\\max\(\\mathrm\{cos\\\_sim\},0\)then modulates merge intensity continuously: similar tokens merge fully \(g≈1g\\approx 1\), orthogonal tokens are dropped \(g=0g=0\), and intermediate cases merge partially\. No learned parameters or threshold tuning are required\.
2. 2\.Attention compensation\(Problem 2\): We add an attention\-ratio logit bias during decoding, using prefill attention weights to estimate how much attention mass each merged position should receive\. Unlike naïve count\-based compensation \(log⁡\(1\+Mi\)\\log\(1\+M\_\{i\}\)\), this formulation is naturally calibrated\.

Adaptive per\-token routing has precedent in vision transformers, where DiffRate\[[4](https://arxiv.org/html/2607.16213#bib.bib16)\]showed that learned merge\-or\-prune decisions can outperform uniform strategies\. Our soft cosine gate is a training\-free analogue for KV caches\.

We introduce SelKV \(Selective KV Cache Merging\), a training\-free framework with a*soft cosine gate*that adaptively modulates per\-token merge intensity and*attention\-ratio compensation*that recalibrates softmax attention toward merged positions\. We evaluate on LongBench\[[1](https://arxiv.org/html/2607.16213#bib.bib3)\]\(16 English datasets\) with three models spanning MHA and GQA architectures at a 25% KV retention budget\. SelKV is especially effective on*grouped\-query attention*\(GQA\), where it comes closest to full\-cache performance\. It also exceeds the full\-cache baseline on several multi\-document QA settings, suggesting that selective merging can act as an implicit attention filter\. At 100k tokens, the compressed cache decodes3\.3×3\.3\\timesfaster than the full cache\.

The remainder of this paper is organized as follows: Section[2](https://arxiv.org/html/2607.16213#S2)reviews related work, Section[3](https://arxiv.org/html/2607.16213#S3)presents the proposed methodology; Section[4](https://arxiv.org/html/2607.16213#S4)reports experimental evaluations and benchmarks against SOTA models; and Section[5](https://arxiv.org/html/2607.16213#S5)concludes with future directions \.

## 2Related Work

KV cache compression methods largely fall into three groups: token eviction, token merging, and quantization\. Quantization methods such as KIVI\[[9](https://arxiv.org/html/2607.16213#bib.bib21)\]and ZipCache\[[6](https://arxiv.org/html/2607.16213#bib.bib22)\]reduce precision rather than token count and are largely complementary to our method, so we focus here on eviction and merging\.

### 2\.1Token Eviction

Token eviction methods reduce the KV cache by permanently discarding entries deemed unimportant, retaining only a subset for subsequent decoding\.

Per\-step eviction: H2O\[[19](https://arxiv.org/html/2607.16213#bib.bib6)\]formulates cache eviction as a dynamic submodular problem, identifying*Heavy Hitter*tokens that accumulate disproportionate attention mass and evicting the rest at every decoding step while preserving a recent\-token window\. VATP\[[5](https://arxiv.org/html/2607.16213#bib.bib7)\]extends this idea by observing that attention weight alone is an incomplete importance proxy: a token’s actual contribution to the output is the product of its attention weight and its value magnitude\. While effective, per\-step methods re\-evaluate importance and reorganize the cache at*every*generation step, incurring substantial decode\-time overhead\.

One\-shot eviction: An alternative family compresses the cache once after prefill, then generates with a fixed compressed cache\. StreamingLLM\[[15](https://arxiv.org/html/2607.16213#bib.bib4)\]keeps only the first few*attention sink*tokens and a sliding recent window, requiring no attention\-based scoring but sacrificing all middle\-context information\. SnapKV\[[7](https://arxiv.org/html/2607.16213#bib.bib5)\]uses an observation window over the lastwwquery positions to identify important KV positions per head, smoothed by an average\-pooling kernel to preserve token clusters\. PyramidKV\[[3](https://arxiv.org/html/2607.16213#bib.bib8)\]applies a layer\-adaptive budget: lower layers retain more tokens, while upper layers rely more on recent context\. More recent one\-shot methods include RocketKV\[[2](https://arxiv.org/html/2607.16213#bib.bib23)\], EvolKV\[[16](https://arxiv.org/html/2607.16213#bib.bib24)\], and LAVa\[[10](https://arxiv.org/html/2607.16213#bib.bib26)\]\.

All eviction methods permanently destroy the information carried by discarded tokens\. This can be especially costly when important evidence is spread across many positions, as in multi\-document QA or long\-range summarization\.

### 2\.2Token Merging

Rather than discarding evicted tokens entirely, merging methods consolidate their information into retained cache entries\.

KVMerger\[[14](https://arxiv.org/html/2607.16213#bib.bib9)\]identifies sequences of tokens with similar key representations and fuses them via a Gaussian kernel\-weighted merging scheme\. D2O\[[12](https://arxiv.org/html/2607.16213#bib.bib11)\]partitions tokens into those that should be dropped and those that should be merged based on attention patterns, but applies the*same*strategy to all tokens within each partition\. LOOK\-M\[[13](https://arxiv.org/html/2607.16213#bib.bib12)\]merges tokens within a local sliding window to preserve spatial locality\. WeightedKV\[[17](https://arxiv.org/html/2607.16213#bib.bib10)\]applies attention\-weighted convex combinations during merging\. ChunkKV\[[8](https://arxiv.org/html/2607.16213#bib.bib13)\]treats fixed\-size semantic chunks as compression units rather than individual tokens\.

Most existing methods apply auniformmerge\-or\-drop decision: either*all*evicted tokens are merged at full weight \(KVMerger, WeightedKV, LOOK\-M\) or*all*are dropped \(H2O, SnapKV, StreamingLLM\)\. D2O partitions tokens into drop and merge groups using attention patterns, but tokens assigned to the merge group are still merged unconditionally\. EMS\[[18](https://arxiv.org/html/2607.16213#bib.bib14)\]explicitly observes that*“both over\-merging and over\-evicting yield sub\-optimal performance,”*yet proposes no continuous per\-token gating mechanism\. Our soft cosine gate fills this gap by modulating merge intensity on a\[0,1\]\[0,1\]spectrum based on value\-vector similarity\.

A further problem affects all merging methods\. When multiple tokens are consolidated into a single cache entry, the merged entry receives roughly the same softmax attention as a single unmerged token\. KeepKV\[[11](https://arxiv.org/html/2607.16213#bib.bib15)\]formalizes this as*attention sag*\(Theorem 3\.2\): merged positions are systematically under\-attended by a factor of∼\\simmm\. KeepKV proposes a logit bias oflog⁡\(1\+Mi\)\\log\(1\+M\_\{i\}\)based on raw merge counts as a correction\. However, raw merge counts can be extreme \(up to∼\\sim200 at 75% compression\), making count\-based compensation unstable\. Moreover, the majority of merging methods like KVMerger, WeightedKV, D2O, ChunkKV do not incorporate any compensation at all\.

## 3Method

We presentSelKV, a pluggable framework for selective KV cache compression that addresses the two problems identified in Section[1](https://arxiv.org/html/2607.16213#S1)\. Given a pretrained transformer withLLlayers,HHattention heads, and head dimensiondd, SelKV compresses the KV cache*once*after prefill and then decodes with the compressed cache\. The framework introduces two mechanisms: \(1\) a*soft cosine gate*that decides per token how much to merge, and \(2\)*attention compensation*that corrects the under\-attention of merged positions\. Both are training\-free and can be added to existing merging methods\.

Figure[1](https://arxiv.org/html/2607.16213#S3.F1)illustrates the full pipeline, which proceeds in six stages after prefill: importance scoring \(§[3\.2](https://arxiv.org/html/2607.16213#S3.SS2)\), token selection \(§[3\.3](https://arxiv.org/html/2607.16213#S3.SS3)\), merge\-target routing \(§[3\.4](https://arxiv.org/html/2607.16213#S3.SS4)\), selective merging with soft cosine gate \(§[3\.5](https://arxiv.org/html/2607.16213#S3.SS5)\), attention compensation \(§[3\.6](https://arxiv.org/html/2607.16213#S3.SS6)\), and RoPE repositioning \(§[3\.7](https://arxiv.org/html/2607.16213#S3.SS7)\)\. Token selection supports both*per\-head*mode \(each attention head independently selects its own tokens\) and*shared*mode \(all heads retain the same token set\), and attention compensation can be applied either averaged across layers or*per\-layer*via forward hooks\.

![Refer to caption](https://arxiv.org/html/2607.16213v1/images/pipeline.png)Figure 1:Overview of the selective KV cache compression pipeline\. After prefill, the method scores token importance, selects retained tokens, routes evicted tokens to merge targets, applies the soft cosine gate, compensates attention for merged positions, and repositions RoPE indices before decode\.### 3\.1Overview and Notation

Let𝐱=\(x1,…,xn\)\\mathbf\{x\}=\(x\_\{1\},\\dots,x\_\{n\}\)denote the input sequence of lengthnn\. We need \(i\) the full KV cache\{\(𝐊\(ℓ\),𝐕\(ℓ\)\)\}ℓ=1L\\\{\(\\mathbf\{K\}^\{\(\\ell\)\},\\mathbf\{V\}^\{\(\\ell\)\}\)\\\}\_\{\\ell=1\}^\{L\}, where𝐊\(ℓ\),𝐕\(ℓ\)∈ℝH×n×d\\mathbf\{K\}^\{\(\\ell\)\},\\mathbf\{V\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{H\\times n\\times d\}; and \(ii\) attention weights over an observation window ofWWquery positions for importance scoring and routing\. A naïve eager prefill would materializeO​\(n2\)O\(n^\{2\}\)attention, which is infeasible at long contexts\. Instead, we use a*two\-pass*strategy: \(1\) run SDPA prefill to build the full KV cache without materializing attention matrices, then \(2\) re\-run only the lastWWtokens with eager attention against the cached keys, producing attention matrices of sizeH×W×nH\\times W\\times nper layer\. This reduces prefill memory fromO​\(n2\)O\(n^\{2\}\)toO​\(W⋅n\)O\(W\\cdot n\)while yielding the same attention weights used by SnapKV\-style observation windows\.

Given a target retention ratioρ∈\(0,1\)\\rho\\in\(0,1\), we set the target cache length tom=⌊ρ⋅n⌋m=\\lfloor\\rho\\cdot n\\rfloor\. A*recent window*of the lastmr=16m\_\{r\}=16tokens is always retained to preserve local context\[[15](https://arxiv.org/html/2607.16213#bib.bib4)\]\. The remainingms=m−mrm\_\{s\}=m\-m\_\{r\}slots are filled by importance\-based selection\.

After compression, we switch to SDPA for decode, which does not require materializing full attention matrices and benefits from the reduced cache size\.

### 3\.2Importance Scoring

Prior methods score token importance by cumulative attention alone\[[19](https://arxiv.org/html/2607.16213#bib.bib6)\]\. Following the insight of VATP\[[5](https://arxiv.org/html/2607.16213#bib.bib7)\], we observe that attention weight alone is an incomplete proxy: a token may receive high attention but carry a near\-zero value vector, contributing little to the output\. Conversely, a moderately attended token with a large value vector may have substantial impact\.

We define the*contribution score*of tokenjjat layerℓ\\elland headhhas:

ch,j\(ℓ\)=∑i∈𝒲Ah,i,j\(ℓ\)⋅‖𝐯h,j\(ℓ\)‖2c\_\{h,j\}^\{\(\\ell\)\}=\\sum\_\{i\\in\\mathcal\{W\}\}A\_\{h,i,j\}^\{\(\\ell\)\}\\cdot\\\|\\mathbf\{v\}\_\{h,j\}^\{\(\\ell\)\}\\\|\_\{2\}\(1\)where𝒲\\mathcal\{W\}denotes the set of query positions used for scoring\. Following SnapKV\[[7](https://arxiv.org/html/2607.16213#bib.bib5)\], we restrict𝒲\\mathcal\{W\}to the lastW=32W=32positions and apply average\-pooling smoothing with kernel sizeK=5K=5to preserve token clusters\.

### 3\.3Token Selection

Different layers exhibit distinct attention patterns: lower layers tend to have broader, more diffuse attention, while upper layers focus more narrowly\[[3](https://arxiv.org/html/2607.16213#bib.bib8)\]\. We therefore perform token selection independently at each layer\.

#### Per\-head selection \(MHA models\)\.

Each headhhat layerℓ\\ellindependently selects its top\-msm\_\{s\}tokens by contribution score:

𝒮h\(ℓ\)=topms⁡\(\{ch,j\(ℓ\)\}j=1n−mr\)∪\{n−mr\+1,…,n\}\\mathcal\{S\}\_\{h\}^\{\(\\ell\)\}=\\operatorname\{top\}\\nolimits\_\{m\_\{s\}\}\\\!\\bigl\(\\\{c\_\{h,j\}^\{\(\\ell\)\}\\\}\_\{j=1\}^\{n\-m\_\{r\}\}\\bigr\)\\;\\cup\\;\\\{n\-m\_\{r\}\+1,\\dots,n\\\}\(2\)The complementary setℰh\(ℓ\)=\{1,…,n\}∖𝒮h\(ℓ\)\\mathcal\{E\}\_\{h\}^\{\(\\ell\)\}=\\\{1,\\dots,n\\\}\\setminus\\mathcal\{S\}\_\{h\}^\{\(\\ell\)\}contains the evicted tokens for headhh\.

#### Union selection \(GQA models\)\.

On GQA models, each KV head servesGGquery heads\. Each query head produces its own importance scorescg,jc\_\{g,j\}; we first average these across theGGquery heads sharing a KV head to obtain per\-KV\-head scores\. Each KV head then independently selects its top\-msm\_\{s\}tokens, and we take the*set union*across heads: a token is kept if*any*KV head selects it\. If the union set exceeds the layer budgetmm, we trim by the mean aggregate score across heads until the budget is met\. This ensures that tokens critical to even a single head are retained, at the cost of a slightly variable per\-layer cache size before trimming\.

### 3\.4Merge\-Target Routing

Each evicted tokenj∈ℰh\(ℓ\)j\\in\\mathcal\{E\}\_\{h\}^\{\(\\ell\)\}must be assigned a merge targetπh​\(j\)∈𝒮h\(ℓ\)\\pi\_\{h\}\(j\)\\in\\mathcal\{S\}\_\{h\}^\{\(\\ell\)\}\. We use*bucket\-attention routing*: the sequence is partitioned into positional buckets of sizeB=32B=32, and each evicted token is routed to the kept token within the same bucket that received the highest attention from it during prefill:

πh​\(j\)=argmaxi∈𝒮h\(ℓ\)∩ℬ​\(j\)Ah,j,i\(ℓ\)\\pi\_\{h\}\(j\)=\\operatorname\*\{argmax\}\_\{i\\in\\mathcal\{S\}\_\{h\}^\{\(\\ell\)\}\\cap\\mathcal\{B\}\(j\)\}A\_\{h,j,i\}^\{\(\\ell\)\}\(3\)This uses theH×W×nH\\times W\\times nattention matrix materialized during the observation\-window pass \(Section[3\.1](https://arxiv.org/html/2607.16213#S3.SS1)\): for each evicted tokenjj, we look up the attention that the observation\-window queries paid tojjand to the candidate targets within its bucket\. The complexity isO​\(H⋅\|ℰ\|⋅B\)O\(H\\cdot\|\\mathcal\{E\}\|\\cdot B\), reduced fromO​\(H⋅\|𝒮\|⋅\|ℰ\|⋅d\)O\(H\\cdot\|\\mathcal\{S\}\|\\cdot\|\\mathcal\{E\}\|\\cdot d\)for full cosine routing\.

### 3\.5Selective Merging with Soft Cosine Gate

Uniformly merging all evicted tokens is harmful when an evicted token’s value vector is dissimilar to its merge target’s value vector\. We need a mechanism that merges when similarity is high and drops when similarity is low, without any learned parameters\.

For each evicted tokenjjrouted to targetπh​\(j\)\\pi\_\{h\}\(j\)at headhh, we compute the cosine similarity between their value vectors:

simh,j=𝐯h,j\(ℓ\)⋅𝐯h,πh​\(j\)\(ℓ\)‖𝐯h,j\(ℓ\)‖2⋅‖𝐯h,πh​\(j\)\(ℓ\)‖2\\mathrm\{sim\}\_\{h,j\}=\\frac\{\\mathbf\{v\}\_\{h,j\}^\{\(\\ell\)\}\\cdot\\mathbf\{v\}\_\{h,\\pi\_\{h\}\(j\)\}^\{\(\\ell\)\}\}\{\\\|\\mathbf\{v\}\_\{h,j\}^\{\(\\ell\)\}\\\|\_\{2\}\\cdot\\\|\\mathbf\{v\}\_\{h,\\pi\_\{h\}\(j\)\}^\{\(\\ell\)\}\\\|\_\{2\}\}\(4\)The soft gate isgh,j=max⁡\(simh,j,0\)g\_\{h,j\}=\\max\(\\mathrm\{sim\}\_\{h,j\},\\;0\)\. It operates on a continuous spectrum:g=0g=0\(drop\) when vectors are dissimilar,0<g<10<g<1\(partial merge\) for moderate similarity, andg≈1g\\approx 1\(full merge\) for high similarity\.

The merged value for a kept tokeniiis:

𝐯~h,i\(ℓ\)=ah,i⋅𝐯h,i\+∑j:πh​\(j\)=igh,j⋅ah,j⋅𝐯h,jah,i\+∑j:πh​\(j\)=igh,j⋅ah,j\\tilde\{\\mathbf\{v\}\}\_\{h,i\}^\{\(\\ell\)\}=\\frac\{a\_\{h,i\}\\cdot\\mathbf\{v\}\_\{h,i\}\\;\+\\;\\sum\_\{j:\\pi\_\{h\}\(j\)=i\}g\_\{h,j\}\\cdot a\_\{h,j\}\\cdot\\mathbf\{v\}\_\{h,j\}\}\{a\_\{h,i\}\\;\+\\;\\sum\_\{j:\\pi\_\{h\}\(j\)=i\}g\_\{h,j\}\\cdot a\_\{h,j\}\}\(5\)whereah,i=∑q∈𝒲Ah,q,ia\_\{h,i\}=\\sum\_\{q\\in\\mathcal\{W\}\}A\_\{h,q,i\}is the cumulative attention over the observation window\. Wheng=0g=0for all evicted tokens, Eq\.[5](https://arxiv.org/html/2607.16213#S3.E5)reduces to pure drop; wheng=1g=1, it reduces to standard attention\-weighted merging\. The gate is training\-free, parameter\-free, and operates per token and per head\.

### 3\.6Attention Compensation

As established by KeepKV\[[11](https://arxiv.org/html/2607.16213#bib.bib15)\]\(Theorem 3\.2\), softmax attention systematically under\-weights merged positions\. KeepKV compensates by addinglog⁡\(1\+Mi\)\\log\(1\+M\_\{i\}\)to attention logits, whereMiM\_\{i\}is the raw merge count\. Under heavy compression \(≥\\geq75% eviction\), however, merge counts can reach∼\\sim200, so count\-based correction becomes less stable\. We therefore replace raw counts with an*attention\-ratio*derived from the prefill pass:

Rh,i=ah,ikept\+∑j→igh,j⋅ah,jdiscah,ikeptR\_\{h,i\}=\\frac\{a\_\{h,i\}^\{\\mathrm\{kept\}\}\+\\sum\_\{j\\to i\}g\_\{h,j\}\\cdot a\_\{h,j\}^\{\\mathrm\{disc\}\}\}\{a\_\{h,i\}^\{\\mathrm\{kept\}\}\}\(6\)On GQA models \(where all query heads sharing a KV head operate on the same retained token set\), we averageRh,iR\_\{h,i\}across KV heads and addα⋅log⁡\(Ri\)\\alpha\\cdot\\log\(R\_\{i\}\)\(withα=0\.5\\alpha=0\.5\) to the attention logits during decoding\. On MHA models with per\-head selection, the bias is applied per head without cross\-head averaging, since each head retains a different token set\. Unlike raw merge counts,RiR\_\{i\}stays bounded by the redistributed attention mass, so positions that absorbed negligible attention receive little or no boost\.

### 3\.7RoPE Repositioning

After compression, we re\-embed retained keys at contiguous positions by undoing the original RoPE and applying new position embeddings:

𝐤~i=RoPE​\(RoPE−1​\(𝐤i,piold\),pinew\)\\tilde\{\\mathbf\{k\}\}\_\{i\}=\\mathrm\{RoPE\}\\\!\\bigl\(\\mathrm\{RoPE\}^\{\-1\}\(\\mathbf\{k\}\_\{i\},\\,p\_\{i\}^\{\\mathrm\{old\}\}\),\\;p\_\{i\}^\{\\mathrm\{new\}\}\\bigr\)\(7\)In our ablation, disabling repositioning causes repetition loops in approximately 15% of generated outputs on short\-context models\. However, on long\-context models under heavy compression \(\>\>75% eviction\), repositioning can distort positional identity and harm retrieval tasks \(Section[4\.3](https://arxiv.org/html/2607.16213#S4.SS3)\)\. We therefore disable it by default for long\-context evaluation\.

### 3\.8Algorithm Summary

The full pipeline is presented in Algorithm[1](https://arxiv.org/html/2607.16213#alg1)\.

Selective KV Cache Compression Input:KV cache\{\(𝐊\(ℓ\),𝐕\(ℓ\)\)\}\\\{\(\\mathbf\{K\}^\{\(\\ell\)\},\\mathbf\{V\}^\{\(\\ell\)\}\)\\\}, attention\{𝐀\(ℓ\)\}\\\{\\mathbf\{A\}^\{\(\\ell\)\}\\\}, ratioρ\\rhoOutput:Compressed KV cache, attention bias vectorforeach layerℓ=1,…,L\\ell=1,\\dots,Ldo 1\. Score tokens:ch,j\(ℓ\)=∑i∈𝒲Ah,i,j⋅‖𝐯h,j‖c\_\{h,j\}^\{\(\\ell\)\}=\\sum\_\{i\\in\\mathcal\{W\}\}A\_\{h,i,j\}\\cdot\\\|\\mathbf\{v\}\_\{h,j\}\\\|\(Eq\.[1](https://arxiv.org/html/2607.16213#S3.E1)\) 2\. Select top\-msm\_\{s\}per head:𝒮h\(ℓ\)\\mathcal\{S\}\_\{h\}^\{\(\\ell\)\}; evictℰh\(ℓ\)\\mathcal\{E\}\_\{h\}^\{\(\\ell\)\}\(Eq\.[2](https://arxiv.org/html/2607.16213#S3.E2)\) foreach evictedj∈ℰh\(ℓ\)j\\in\\mathcal\{E\}\_\{h\}^\{\(\\ell\)\}do 3\. Route to targetπh​\(j\)\\pi\_\{h\}\(j\)via bucket\-attention\(Eq\.[3](https://arxiv.org/html/2607.16213#S3.E3)\) 4\. Gate:gh,j=max⁡\(simh,j,0\)g\_\{h,j\}=\\max\(\\mathrm\{sim\}\_\{h,j\},\\,0\)\(Eq\.[4](https://arxiv.org/html/2607.16213#S3.E4)\) end for 5\. Merge:𝐯~h,i←\\tilde\{\\mathbf\{v\}\}\_\{h,i\}\\leftarrowgated attn\-weighted avg\(Eq\.[5](https://arxiv.org/html/2607.16213#S3.E5)\) 6\. Compute attn ratioRh,iR\_\{h,i\}per kept position\(Eq\.[6](https://arxiv.org/html/2607.16213#S3.E6)\) 7\. Reposition keys via RoPE inverse \+ re\-embed\(Eq\.[7](https://arxiv.org/html/2607.16213#S3.E7)\) end for Decodewith compressed cache \+α⋅log⁡\(Ri\)\\alpha\\cdot\\log\(R\_\{i\}\)bias

Algorithm 1Pseudocode for selective KV cache compression\.

## 4Experiments

### 4\.1Setup

#### Models\.

We evaluate on three models spanning two attention architectures: LongChat\-7B\-v1\.5\-32k \(multi\-head attention, MHA, 32 KV heads\), LLaMA\-3\.1\-8B\-Instruct \(grouped\-query attention, GQA, 8 KV heads\), and Gemma\-2\-9B\-IT \(GQA, 8 KV heads with alternating sliding\-window attention\)\. LongChat uses 31,500\-token inputs, enabling evaluation under heavy compression \(75% eviction\); the remaining models use 3,500\-token inputs\. All experiments use FP16 inference on NVIDIA H100 80GB GPUs\.

#### Benchmark\.

We evaluate on the 16 English datasets of LongBench\[[1](https://arxiv.org/html/2607.16213#bib.bib3)\], spanning 6 task categories: single\-document QA \(NarrativeQA, Qasper, MultifieldQA\-en\), multi\-document QA \(HotpotQA, 2WikiMQA, MuSiQue\), summarization \(GovReport, QMSum, MultiNews\), few\-shot learning \(TREC, TriviaQA, SAMSum\), synthetic tasks \(PassageCount, PassageRetrieval\-en\), and code completion \(LCC, RepoBench\-P\)\. Each dataset is evaluated with its standard metric \(F1, ROUGE, accuracy, or code similarity\) on 200 samples \(500 for code tasks\)\.

#### Baselines\.

We compare against three one\-shot methods at a 25% KV retention budget: SnapKV\[[7](https://arxiv.org/html/2607.16213#bib.bib5)\]\(observation\-window eviction\), LOOK\-M\[[13](https://arxiv.org/html/2607.16213#bib.bib12)\]\(sliding\-window merging\), and PyramidKV\[[3](https://arxiv.org/html/2607.16213#bib.bib8)\]\(layer\-adaptive eviction\)\. Section[4\.4](https://arxiv.org/html/2607.16213#S4.SS4)reports decode throughput for SelKV against the full\-cache baseline\.

#### SelKV configuration\.

Contribution\-based importance scoring \(attention×\\timesvalue norm\), observation windowW=32W=32, smoothing kernelK=5K=5, bucket\-attention routing with bucket sizeB=32B=32, soft cosine gate enabled, attention compensation \(α=0\.5\\alpha\{=\}0\.5, attn\-ratio\), recent windowmr=16m\_\{r\}=16tokens, RoPE repositioning disabled\. On GQA models, each KV head independently selects its top\-msm\_\{s\}tokens and the set union across heads determines the kept set \(trimmed to budget by mean score if needed\)\.

### 4\.2Cross\-Model Results

We evaluate our approach SelKV and three one\-shot baselines across three LLMs—Gemma\-2\-9B\-IT, LLaMA\-3\.1\-8B\-Instruct, and LongChat\-7B\-v1\.5\-32k—covering six LongBench task categories: single\-document QA, multi\-document QA, summarization, few\-shot learning, synthetic tasks, and code completion\. Table[1](https://arxiv.org/html/2607.16213#S4.T1)reports the per\-dataset results for all methods and models, while the final Avg,Δ\\Delta, and Avg Rank columns summarize overall performance and consistency within each model\. Readers should focus on whether SelKV stays closest to the Full Cache baseline across architectures and where compression is most beneficial, especially on GQA models and multi\-document QA tasks\.

Table 1:LongBench results \(16 English datasets\) at a 25% KV retention budget across three models\.Δ\\Delta: gap to Full Cache within each model\. Avg Rank summarizes per\-dataset ranks among compressed methods within each model; it is the mean rank across datasets \(lower is better\), and ties receive the better rank\. Best compression method per task inbold\.Single\-Doc QAMulti\-Doc QASummarizationFew\-shotSyntheticCodeOverallLLMsMethodNrtQA Qasp MF\-en HpQA 2Wiki Musi GovR QMSm MNws TREC TrvQA SAMs PCnt PRet LCC RB\-P AvgΔ\\DeltaAvg RankGemma\-2Full Cache22\.9340\.7946\.6847\.2548\.7422\.8427\.6319\.4724\.8167\.5091\.9243\.567\.0033\.5072\.7870\.3342\.98––SnapKV22\.2240\.0146\.7846\.9046\.5422\.0024\.6319\.2822\.3066\.5091\.9242\.627\.0033\.5072\.6269\.7542\.16−\-0\.822\.06LOOK\-M22\.6636\.7545\.9346\.9746\.0522\.3423\.9819\.1822\.7355\.5092\.1742\.257\.0033\.5072\.0769\.8441\.18−\-1\.802\.56PyramidKV22\.4339\.7046\.7146\.3747\.3022\.4124\.4618\.9422\.1266\.5091\.9242\.367\.0033\.5072\.5369\.1042\.08−\-0\.902\.50SelKV\(ours\)23\.3240\.0146\.4446\.6746\.8422\.9024\.9419\.3322\.7967\.0091\.7642\.897\.0033\.5072\.2469\.3542\.31−\-0\.671\.69LLaMA\-3\.1Full Cache21\.6340\.2449\.5439\.0435\.2318\.8432\.3721\.0027\.1570\.0089\.0043\.683\.5834\.0064\.8758\.4140\.54––SnapKV21\.3339\.0049\.7139\.2535\.8618\.8827\.9721\.0423\.5967\.5089\.6342\.004\.0834\.0065\.0257\.7339\.79−\-0\.752\.06LOOK\-M20\.8433\.1844\.9238\.9635\.3617\.0128\.3321\.5924\.8465\.0089\.3541\.974\.0035\.0063\.8056\.3138\.78−\-1\.763\.00PyramidKV21\.6038\.0449\.2040\.0434\.7718\.6527\.6320\.7623\.3467\.0089\.7742\.374\.0834\.0063\.8357\.1939\.52−\-1\.022\.63SelKV\(ours\)21\.6038\.2550\.0139\.4435\.9119\.5527\.9421\.3723\.6367\.0089\.9542\.064\.0334\.5064\.3457\.3139\.81−\-0\.731\.75LongChatFull Cache21\.0129\.3143\.3733\.0823\.4414\.1530\.9122\.6726\.6066\.5084\.0922\.160\.0030\.5050\.0658\.1734\.75––SnapKV20\.7127\.3339\.4234\.7322\.6614\.1826\.5222\.5413\.2564\.0083\.7827\.680\.0029\.0043\.3557\.9632\.94−\-1\.812\.00LOOK\-M19\.8623\.1629\.0531\.7821\.5811\.9322\.5522\.3915\.2455\.5074\.3325\.930\.0023\.2547\.7656\.8630\.07−\-4\.683\.38PyramidKV20\.1728\.0041\.6834\.0422\.7613\.9727\.2923\.0722\.6064\.5082\.3423\.790\.0030\.5050\.8757\.5533\.95−\-0\.801\.63SelKV\(ours\)20\.4827\.7733\.6931\.0220\.9613\.1125\.6622\.2418\.6661\.0078\.0429\.540\.0038\.5048\.5056\.1832\.83−\-1\.922\.63

#### Architecture dependence\.

SelKV gives the top compressed result on both GQA models: Gemma\-2 \(42\.31,−\-0\.67\) and LLaMA\-3\.1 \(39\.81,−\-0\.73\)\. On the MHA model LongChat \(evaluated at 31,500 tokens vs\. 3,500 for the GQA models\), PyramidKV leads \(33\.95 vs\. 32\.83\)\. To control for context length, we also evaluate on LLaMA\-3\.1 at 31,500 tokens \(Appendix[E](https://arxiv.org/html/2607.16213#A5)\): SelKV \(48\.46\) and SnapKV \(48\.46\) are tied, both ahead of PyramidKV \(48\.26\) and LOOK\-M \(47\.16\), showing that the advantage on GQA models is not an artifact of the shorter evaluation context\.

#### Multi\-document QA\.

On several multi\-document QA settings, the compressed cache exceeds full\-cache scores \(e\.g\., LLaMA\-3\.1: HotpotQA 39\.44 vs\. 39\.04, MuSiQue 19\.55 vs\. 18\.84; Gemma\-2: MuSiQue 22\.90 vs\. 22\.84\)\. We interpret this as evidence that our approach can suppress distracting context while preserving salient evidence\.

![Refer to caption](https://arxiv.org/html/2607.16213v1/x1.png)Figure 2:Performance gap to Full Cache across models \(closer to 0 is better\)\. SelKV\(ours\) is the best for LLaMa and GemmaAt the category level on Gemma\-2, our approach SelKV leads on four of six categories \(Single\-Doc QA, Multi\-Doc QA, Summarization, and Few\-shot\) while SnapKV leads only on Code, where exact token identity matters \(full breakdown in Appendix[C](https://arxiv.org/html/2607.16213#A3), Table[4](https://arxiv.org/html/2607.16213#A3.T4)\)\.

### 4\.3Ablation and Compression Ratio Robustness

We perform a staged ablation on LLaMA\-3\.1\-8B \(GQA, 4:1 ratio\) at a 25% KV retention budget over all 16 datasets\. Starting from our token selection, we compare: \(i\)*eviction only*, which drops all evicted tokens; \(ii\)*merge all*, which merges without the soft cosine gate or attention compensation; \(iii\)*\+ gate*, which adds the soft cosine gate; and \(iv\)*\+ gate \+ comp*, which recovers the full method by further adding attention compensation\. To test portability, we also evaluate the gate on a 6\-dataset Gemma\-2 subset and as a plug\-in modification to LOOK\-M\. Finally, we test robustness to the retention budget by sweeping 10%–90% KV retention on LLaMA\-3\.1 and Gemma\-2, with full per\-ratio tables in Appendix[D](https://arxiv.org/html/2607.16213#A4)\.

Blind merging hurts on LLaMA\-3\.1 \(39\.01 vs\. 39\.69 for eviction only\), consistent with frequent value mismatch under 4:1 GQA\. The soft cosine gate recovers most of the loss \(39\.57,\+\+0\.56\), and adding compensation yields further improvement \(39\.81,\+\+0\.24\)\. On a 6\-dataset subset of Gemma\-2, compensation helps more than the gate \(\+\+0\.08 vs\.−\-0\.02\), suggesting that the two components matter differently across GQA ratios\. The gate is also portable: adding it to LOOK\-M improves Gemma\-2 from 41\.18 to 41\.35 and LLaMA\-3\.1 from 38\.78 to 38\.86\.

Table 2:Ablation on LLaMA\-3\.1\-8B \(16 datasets, 25% KV retention budget\)\. “Evict only” uses our token selection but drops evicted tokens without merging\.ConfigurationGateCompMergeAvg \(16 ds\)Full Cache–––40\.54Eviction only–––39\.69Merge all \(no gate\)––✓39\.01 \(−\-0\.68\)\+ gate✓–✓39\.57 \(\+0\.56\)\+ gate \+ comp✓✓✓39\.81\(\+0\.24\)#### RoPE repositioning\.

On LongChat\-7B \(31\.5k tokens\), repositioning harms PassageRetrieval\-en \(5\.0 vs\. 38\.5 without\) where paragraph identity depends on absolute position, so we disable it by default\.

#### Compression\-ratio robustness\.

We sweep the KV retention budget from 10% to 90% on LLaMA\-3\.1 and Gemma\-2 \(Figure[4](https://arxiv.org/html/2607.16213#A4.F4), Appendix[D](https://arxiv.org/html/2607.16213#A4)\) to test whether the relative ranking is stable across compression levels\. At 10%, SnapKV leads on both models because few merge targets remain\. From 25% onward, SelKV leads on Gemma\-2 and matches SnapKV on LLaMA\-3\.1; above 70%, all methods converge\. Notably, SelKV at an 80% retention budget on Gemma\-2 slightly exceeds the full cache \(43\.01 vs\. 42\.98\)\.

### 4\.4Latency Analysis

We next evaluate whether one\-shot compression translates into practical wall\-clock gains as prompt length grows\.

Figure[3](https://arxiv.org/html/2607.16213#S4.F3)shows wall\-clock time on LLaMA\-3\.1\-8B across prompt lengths from 512 to 100k tokens\. At short contexts \(≤\\leq8k\), throughput matches the full cache \(∼\\sim50 tok/s\)\. Beyond 16k tokens, the smaller KV cache makes decoding faster: at 100k tokens we observe 28 tok/s vs\. 8\.4 tok/s for the full cache \(3\.3×3\.3\\times\)\. Compression overhead remains below 5% of total time at 8k\+\.

![Refer to caption](https://arxiv.org/html/2607.16213v1/x2.png)Figure 3:Latency on LLaMA\-3\.1\-8B \(256 generated tokens, H100, 25% KV retention budget\)\. \(a\) Total time breakdown\. \(b\) Decode throughput\.

## 5Conclusion

We present SelKV, a training\-free KV\-cache compression framework with a soft cosine gate and attention\-ratio compensation\. On LongBench at a 25% KV retention budget, it gives the best compressed results on the two GQA models we study, stays within 0\.67 \- 0\.73 points of full cache on average, slightly exceeds full\-cache scores on some multi\-document QA settings, and yields3\.3×3\.3\\timesfaster decoding at 100k tokens\. These results suggest that merge quality, not just token selection, is an important design axis for KV compression\. The cosine gate spans the continuum between pure eviction \(g=0g\{=\}0\) and full merging \(g=1g\{=\}1\), and offers a simple interface for future learned or task\-adaptive policies\.

Limitations include reduced effectiveness on MHA models and code tasks, and at very aggressive compression \(10% budget\) where few merge targets remain\. We also evaluate only three models, and generation quality is validated on contexts up to 31,500 tokens; the 100k\-token latency benchmark measures throughput scaling only\. Future work includes broader model coverage, combinations with quantization, and extensions to larger\-context training\-time adaptations and longer\-context quality evaluation\.

## References

- \[1\]Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li\(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3119–3137\.Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p5.1),[§4\.1](https://arxiv.org/html/2607.16213#S4.SS1.SSS0.Px2.p1.1)\.
- \[2\]B\. Behnam, D\. Y\. Fu,et al\.\(2025\)RocketKV: hybrid two\-stage kv cache compression with sparse attention\.arXiv\.Cited by:[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p3.1)\.
- \[3\]Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu,et al\.\(2025\)PyramidKV: dynamic kv cache compression based on pyramidal information funneling\.InCOLM,Cited by:[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p3.1),[§3\.3](https://arxiv.org/html/2607.16213#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2607.16213#S4.SS1.SSS0.Px3.p1.1)\.
- \[4\]S\. Chen, Q\. Lou, C\. Li, and Y\. Lin\(2023\)DiffRate: differentiable compression rate for efficient vision transformers\.InICCV,Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p4.1)\.
- \[5\]Z\. Guo, H\. Kamigaito, and T\. Watanabe\(2024\)Attention score is not all you need for token importance indicator in kv cache reduction: value also matters\.InEMNLP,Cited by:[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2607.16213#S3.SS2.p1.1)\.
- \[6\]Y\. He, L\. Zhang, W\. Wu, J\. Liu, H\. Zhou, and B\. Zhuang\(2024\)ZipCache: accurate and efficient kv cache quantization with salient token identification\.InThirty\-eighth Conference on Neural Information Processing Systems \(NeurIPS\),Note:Available:[https://github\.com/ThisisBillhe/ZipCache/](https://github.com/ThisisBillhe/ZipCache/)Cited by:[§2](https://arxiv.org/html/2607.16213#S2.p1.1)\.
- \[7\]Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen\(2024\)SnapKV: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p3.1),[§3\.2](https://arxiv.org/html/2607.16213#S3.SS2.p2.7),[§4\.1](https://arxiv.org/html/2607.16213#S4.SS1.SSS0.Px3.p1.1)\.
- \[8\]X\. Liu, Z\. Tang, P\. Dong, Z\. Li, Y\. Liu, B\. Li, X\. Hu, and X\. Chu\(2025\)ChunkKV: semantic\-preserving kv cache compression for efficient long\-context llm inference\.arXiv preprint arXiv:2502\.00299\.Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p1.3),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p2.1)\.
- \[9\]Z\. Liuet al\.\(2024\)KIVI: a tuning\-free asymmetric 2bit quantization for kv cache\.InICML,Cited by:[§2](https://arxiv.org/html/2607.16213#S2.p1.1)\.
- \[10\]Y\. Shen, S\. Yuan, Z\. Zhang, X\. Wang, D\. Jiang, and C\. Nguyen\(2025\)LAVa: layer\-wise kv cache eviction with dynamic budget allocation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 13672–13692\.Cited by:[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p3.1)\.
- \[11\]Y\. Tian, Z\. Wang, Y\. Peng, A\. Yuan, Z\. Wang, B\. Yi, X\. Liu, Y\. Cui, and T\. Yang\(2024\)KeepKV: eliminating output perturbation in kv cache compression for efficient llms inference\.arXiv\.Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p1.3),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p4.4),[§3\.6](https://arxiv.org/html/2607.16213#S3.SS6.p1.4)\.
- \[12\]Z\. Wan, X\. Wu, Y\. Zhang, Y\. Xin, C\. Tao, Z\. Zhu, X\. Wang,et al\.\(2025\)D2O: dynamic discriminative operations for efficient long\-context inference of large language models\.InICLR,Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p1.3),[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p2.1)\.
- \[13\]Z\. Wan, Z\. Wu, C\. Liu, J\. Huang, Z\. Zhu, P\. Jin, L\. Wang, and L\. Yuan\(2024\)LOOK\-m: look\-once optimization in kv cache for efficient multimodal long\-context inference\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 4065–4078\.Note:Available:[https://github\.com/SUSTechBruce/LOOK\-M](https://github.com/SUSTechBruce/LOOK-M)Cited by:[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2607.16213#S4.SS1.SSS0.Px3.p1.1)\.
- \[14\]Z\. Wang, B\. Jin, Z\. Yu, and M\. Zhang\(2024\)Model tells you where to merge: adaptive kv cache merging for llms on long\-context tasks\.arXiv:2407\.08454\.Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p1.3),[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p2.1)\.
- \[15\]G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis\(2024\)Efficient streaming language models with attention sinks\.InICLR,Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p3.1),[§3\.1](https://arxiv.org/html/2607.16213#S3.SS1.p2.4)\.
- \[16\]B\. Yu and Y\. Chai\(2025\)EvolKV: evolutionary kv cache compression for llm inference\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 1673–1689\.Cited by:[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p3.1)\.
- \[17\]J\. Yuan, Z\. He, H\. Bai, J\. Leng, and B\. Jiang\(2025\)WeightedKV: attention scores weighted key\-value cache merging for large language models\.InICASSP 2025 – IEEE International Conference on Acoustics, Speech and Signal Processing,External Links:[Document](https://dx.doi.org/10.1109/ICASSP49660.2025.10889583)Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px2.p1.3),[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p2.1)\.
- \[18\]Y\. Zhanget al\.\(2024\)EMS: adaptive evict\-then\-merge strategy for head\-wise kv cache compression\.arXiv\.Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px1.p1.2),[§2\.2](https://arxiv.org/html/2607.16213#S2.SS2.p3.1)\.
- \[19\]Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Re, C\. Barrett, Z\. Wang, and B\. Chen\(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InNeurIPS,Cited by:[§1](https://arxiv.org/html/2607.16213#S1.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2607.16213#S1.p2.1),[§2\.1](https://arxiv.org/html/2607.16213#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2607.16213#S3.SS2.p1.1)\.

## Appendix AComputational Complexity

The dominant prefill cost is the SDPA forward pass,O​\(L⋅H⋅n2⋅d\)O\(L\\cdot H\\cdot n^\{2\}\\cdot d\), identical to any method that builds a KV cache\. The two\-pass eager re\-pass over the observation window addsO​\(L⋅H⋅W⋅n⋅d\)O\(L\\cdot H\\cdot W\\cdot n\\cdot d\), which is linear innnfor fixedWW\. The compression step itself is efficient:

- •Importance scoring:O​\(L⋅H⋅n⋅\(W\+d\)\)O\(L\\cdot H\\cdot n\\cdot\(W\+d\)\)attention summation over the observation window and value\-norm computation\.
- •Selection:O​\(L⋅Hkv⋅n​log⁡m\)O\(L\\cdot H\_\{\\mathrm\{kv\}\}\\cdot n\\log m\)per\-KV\-head top\-kkvia partial sort, followed by set union\.
- •Routing:O​\(L⋅H⋅\|ℰ\|⋅B\)O\(L\\cdot H\\cdot\|\\mathcal\{E\}\|\\cdot B\)whereBBis the bucket size \(default 32\)\.
- •Soft gate:O​\(L⋅H⋅\|ℰ\|⋅d\)O\(L\\cdot H\\cdot\|\\mathcal\{E\}\|\\cdot d\)one cosine similarity per evicted token\.
- •Merge:O​\(L⋅H⋅\|ℰ\|⋅d\)O\(L\\cdot H\\cdot\|\\mathcal\{E\}\|\\cdot d\)vectorized scatter\-add operations\.
- •RoPE repositioning:O​\(L⋅H⋅m⋅d\)O\(L\\cdot H\\cdot m\\cdot d\)\.
- •Attention compensation\(decode\):O​\(m\)O\(m\)per step add a scalar bias per position\.

The total compression overhead is 0\.04 s for a 2048\-token sequence on an H100 GPU, amortized over the entire decode phase\.

## Appendix BHyperparameter Sensitivity

Table[3](https://arxiv.org/html/2607.16213#A2.T3)varies the observation window sizeWWand smoothing kernel sizeKKon Gemma\-2 \(6\-dataset subset, 25% KV retention budget\)\.

Table 3:Sensitivity to observation windowWWand kernel sizeKKon Gemma\-2\-9B\-IT \(6 datasets, 25% KV retention budget\)\. Default:W=32W\{=\}32,K=5K\{=\}5\.WWKKQaspSAMsHpQALCCTRECPRetAvg16540\.1643\.0447\.4272\.2967\.0033\.0050\.4932540\.0142\.8946\.6772\.2467\.0033\.5050\.3964539\.7143\.2447\.5072\.0766\.5033\.0050\.3432340\.1542\.7446\.9071\.9870\.5933\.5050\.9832739\.8342\.7346\.5072\.5067\.5033\.0050\.34All configurations fall within a 0\.6\-point range \(50\.34–50\.98\), demonstrating that our method is robust to these hyperparameter choices\. The observation window has minimal effect \(W=16W\{=\}16andW=64W\{=\}64perform comparably toW=32W\{=\}32\), and kernel sizesK∈\{3,5,7\}K\\in\\\{3,5,7\\\}yield near\-identical results\.

## Appendix CCategory\-Level Analysis

Table[4](https://arxiv.org/html/2607.16213#A3.T4)summarizes per\-category averages on Gemma\-2\-9B\-IT \(25% KV retention budget\)\. Our method leads on four of six categories: Single\-Doc QA, Multi\-Doc QA, Summarization, and Few\-shot\. SnapKV leads only on Code, where precise token identity matters\.

Table 4:Category\-level averages on LongBench \(Gemma\-2\-9B\-IT, 25% KV retention budget\)\. Best one\-shot result per category inbold\.CategoryFullSelKV\(ours\)SnapKVLOOK\-MPyramidKVSingle\-Doc QA36\.8036\.5936\.3435\.1136\.28Multi\-Doc QA39\.6138\.8038\.4838\.4538\.69Summarization23\.9722\.3522\.0721\.9621\.84Few\-shot67\.6667\.2267\.0163\.3166\.93Synthetic20\.2520\.2520\.2520\.2520\.25Code71\.5670\.8071\.1970\.9670\.82
## Appendix DCompression Ratio Sweep

Tables[6](https://arxiv.org/html/2607.16213#A4.T6)and[6](https://arxiv.org/html/2607.16213#A4.T6)present per\-method scores at six compression ratios on LLaMA\-3\.1\-8B and Gemma\-2\-9B\-IT \(3\.5k tokens, 16 datasets each\)\. On LLaMA\-3\.1 \(Figure[4](https://arxiv.org/html/2607.16213#A4.F4)\), SnapKV leads at 10%; from 25% onward, SelKV matches or exceeds SnapKV; above 70%, all methods converge\. On Gemma\-2, SelKV leads at every budget from 25% onward, and at 80% it slightly exceeds the full cache \(43\.01 vs\. 42\.98\), suggesting that selective merging can act as implicit attention filtering under moderate compression\.

![Refer to caption](https://arxiv.org/html/2607.16213v1/x3.png)Figure 4:LongBench average at varying KV retention ratios \(3\.5k tokens, 16 datasets\)\. \(a\) LLaMA\-3\.1\-8B \(b\) Gemma\-2\-9B\-IT\.Table 5:LongBench average \(LLaMA\-3\.1\-8B, 3\.5k tokens\) at different KV retention ratios\.Method10%25%50%70%80%90%Full Cache40\.54SnapKV38\.0639\.7940\.2940\.4140\.4940\.46LOOK\-M36\.6538\.7840\.2740\.4040\.4940\.51SelKV\(ours\)37\.5639\.8140\.3440\.3840\.4240\.46

Table 6:LongBench average \(Gemma\-2\-9B\-IT, 3\.5k tokens\) at different KV retention ratios\. SelKV leads from 25% onward and exceeds Full Cache at 80%\.Method10%25%50%70%80%90%Full Cache42\.98SnapKV41\.2842\.1642\.6942\.8742\.9242\.96LOOK\-M39\.1141\.1842\.3742\.7342\.8442\.95SelKV\(ours\)41\.0542\.3142\.8042\.9343\.0142\.96

Figure[5](https://arxiv.org/html/2607.16213#A4.F5)illustrates why the adaptive gate matters by contrasting two datasets on Gemma\-2\. On Qasper \(single\-document QA\), eviction outperforms blind merging at low budgets\. SelKV tracks close to the better baseline at each budget and exceeds the full cache from 50% onward, the gate adapts by dropping dissimilar tokens where merging would hurt\. On MuSiQue \(multi\-document QA\), merging helps, and SelKV at 25% exceeds the full cache \(22\.90 vs\. 22\.84\), demonstrating that selective merging can act as implicit attention filtering\.

![Refer to caption](https://arxiv.org/html/2607.16213v1/x4.png)Figure 5:Per\-dataset ratio sweep on Gemma\-2\-9B\-IT\. Left: Qasper, where eviction outperforms blind merging but the gate protects quality by dropping dissimilar tokens\. Right: MuSiQue, where merging helps and SelKV exceeds the full cache at 25%\.
## Appendix ELong\-Context Evaluation \(31,500 Tokens\)

Table[7](https://arxiv.org/html/2607.16213#A5.T7)evaluates the same four compressed methods as Table[1](https://arxiv.org/html/2607.16213#S4.T1)on LLaMA\-3\.1\-8B at 31,500\-token inputs\. At this length, SelKV and SnapKV are tied and both outperform PyramidKV and LOOK\-M\.

Table 7:LongBench average \(LLaMA\-3\.1\-8B, 31,500 tokens, 25% KV retention\)\.MethodAvg \(16 ds\)Δ\\DeltaFull Cache49\.36–SnapKV48\.46−\-0\.90PyramidKV48\.26−\-1\.10LOOK\-M47\.16−\-2\.20SelKV \(ours\)48\.46−\-0\.90
## Appendix FImplementation Details

#### Hardware\.

All experiments were run on NVIDIA H100 80GB GPUs with CUDA 12\.1 and PyTorch 2\.4\.

#### Hyperparameters\.

Unless otherwise noted, we use observation windowW=32W=32, smoothing kernelK=5K=5, bucket sizeB=32B=32, recent windowr=16r=16tokens, compensation factorα=0\.5\\alpha=0\.5, and KV retention ratioρ=0\.25\\rho=0\.25\. For GQA models, each KV head independently selects its top\-msm\_\{s\}tokens; the final kept set is the union across heads, with any excess tokens trimmed by mean aggregate score\. For MHA models \(LongChat\), we use per\-head selection with the same observation\-window procedure\.

#### Evaluation\.

We follow the LongBench evaluation pipeline and use greedy decoding \(temperature 0, no sampling\)\. LongChat is evaluated with 31,500\-token inputs, whereas all other models use 3,500\-token inputs\. For code\-completion tasks \(LCC and RepoBench\-P\), we evaluate 500 samples; for all other tasks, we evaluate 200 samples\.

#### Runtime\.

A full 16\-dataset evaluation for one model takes approximately 1\-2 hours on a single H100 GPU, depending on context length\. The ratio\-sweep ablation \(6 ratios×\\times16 datasets×\\times3 methods per model\) requires approximately 8 GPU\-hours per model\.

## Appendix GGenerated Text Examples

We show qualitative examples of generated text from LLaMA\-3\.1\-8B\-Instruct under Full Cache and SelKV \(25% KV budget\)\. All outputs use greedy decoding \(temperature 0\)\.

#### Example 1: Qasper \(Single\-Document QA\)\.

A scientific article about characterizing political fake news on Twitter \(3,500 tokens\) is provided as context\. The question asks:*“How is the ground truth for fake news established?”*

Ground truth:Ground truth is not established in the paper\.

Full Cache:\[…\] who found that fake news items are more likely to be shared than legitimate articles coming from trusted sources, and that they tend to exhibit a larger level of polarization\.Conclusion:In this paper, we have presented a preliminary characterization of fake news in Twitter by looking into meta\-data embedded in tweets\. Our results show that there are specific pieces of meta\-data about tweets that may allow the identification of fake news\. In particular, we found that tweets containing fake news are shorter\-lived, have a larger proportion of friends/followers, and include more URLs \[…\]

SelKV \(25%\):\[…\] who found that fake news is more likely to be polarized than legit news\.Conclusion:In this paper, we have characterized political fake news on Twitter by its meta\-data\. We found that tweets containing fake news are more likely to be created by unverified accounts, are shorter\-lived, and are more likely to be polarized\. We also found that they are more likely to be spread by accounts with a larger proportion of friends/followers, and that they contain more URLs \[…\]

Our method produces coherent, topically consistent summaries of the article’s findings\. It preserve the key conclusions \(fake news is shorter\-lived, more polarized, spread by accounts with more friends/followers\) while differing only in phrasing\.

#### Example 2: SAMSum \(Dialogue Summarization\)\.

A dialogue dataset sample \(3,500 tokens of few\-shot examples\) is provided\. The target dialogue is between William and Emilia\.

Ground truth:Emilia is still angry\.

Full Cache:Vicki will come to the place at 21:30 tomorrow night\. They will be open until 22:00\. \[continues generating further dialogue\-summary pairs\]

SelKV \(25%\):Vicki is asking about the store hours for tomorrow night\. \[continues generating further dialogue\-summary pairs\]

Our method \(SelKV\) correctly follows the few\-shot pattern \(generating summary for subsequent dialogues\)\. It produced slightly more concise summary while preserving factual accuracy\.