ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression

# Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Source: [https://arxiv.org/html/2607.29591](https://arxiv.org/html/2607.29591)
###### Abstract

KV cache compression is essential for efficient long\-context inference\. Existing eviction methods permanently discard unselected tokens and consequently remove their aggregate contribution to attention\. Merging\-based alternatives preserve more information but can perturb retained keys and values that should remain exact\. We observe that the information omitted by cache eviction can be formulated as residual statistics in both the numerator and denominator of softmax attention\. Based on this observation, we proposeResKV, which divides a fixed KV budget into an exact main cache and a compact residual cache that reconstructs the contribution of omitted tokens\. ResKV lets main\-cache tokens and residual entries participate in the same softmax normalization, so residual entries restore both attention numerator and denominator mass rather than acting as a post\-hoc correction\. A construction\-time validation proxy determines residual allocation for each layer and KV head, while a decode\-time dynamic gate adjusts residual contributions for individual queries\. Comprehensive evaluations on LongBench and RULER, covering query\-aware and query\-agnostic settings, multiple backbones, cache budgets, and representative compression baselines, demonstrate broad improvements under the same retained KV budget while preserving the practical efficiency of compressed decoding, including peak memory usage and long\-context decode throughput\.

## Introduction

Long\-context large language models rely on a KV cache to avoid recomputing past keys and values during autoregressive generation\. While this cache makes decode efficient, it also grows linearly with the context length and quickly becomes a dominant source of memory use and memory traffic\[Kwonet al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib20), Shenget al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib21), Liuet al\.,[2024b](https://arxiv.org/html/2607.29591#bib.bib18), Hooperet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib17)\]\. This creates a central tension in long\-context inference: the model should retain enough historical information to answer future queries, but the system can only afford to keep a limited number of KV slots\.

KV cache compression methods commonly address this tension by deciding which tokens to keep\. Eviction\-based methods score cached tokens, retain a subset as exact keys and values, and permanently discard the rest\[Liuet al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib22), Zhanget al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib3), Adnanet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib23), Liet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib5), Orenet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib4), Tanget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib24), Fenget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib7), Zhouet al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib26), Guet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib8)\]\. This design is efficient, but it makes cache compression a binary decision: each token is either kept unchanged or removed together with its value contribution and softmax normalization mass\.

Merging\-based methods preserve information from evicted tokens by combining related cache states\. CaM folds evicted states into retained entries, KVMerger merges similar keys with Gaussian\-kernel weights, KeepKV compensates attention scores using merging history, and SemantiCache constructs semantic cores with proportional attention\[Zhanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib9), Wanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib10), Tianet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib11), Wuet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib12)\]\. These methods show that omitted information can remain useful after aggregation\. However, they fold omitted content into retained or merged cache states, which can perturb retained memories and erase the individual identities of folded tokens\.

Another line of recent work estimates the contribution of compressed or omitted tokens\. KVSculpt optimizes unconstrained virtual KV pairs to preserve layer attention behavior\[Jiang and Jin,[2026](https://arxiv.org/html/2607.29591#bib.bib13)\], while RESA uses a low\-rank logit prior and an online aggregator to compensate the output of sparse attention\[Yanget al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib14)\]\. However, these methods do not jointly address how to preserve selected tokens as exact memories and represent omitted tokens as cache\-resident entries under the same fixed budget\.

We instead view the removed tokens through the attention computation itself\. In the attention numerator–denominator form, the evicted side is a residual contribution: a pair of omitted softmax statistics over the tokens outside the main cache\. If these omitted statistics could be reconstructed, the model would not need to choose between deleting them outright, folding them into retained tokens, or replacing them with merged cache states\. The key question is therefore how to represent this residual contribution within the same fixed KV budget\.

We proposeResKV, a residual KV cache that splits a fixed cache budget into two parts: a main cache that stores selected tokens exactly, and a residual cache that represents the omitted side with compact residual entries\. The main cache keeps high\-priority tokens unchanged as exact memories, while the residual cache reintroduces structured information from evicted tokens without increasing the total number of cache slots\. In this way, ResKV changes the cache representation from a pure keep\-or\-drop decision into a main\-plus\-residual layout under the same retained KV budget\. This avoids the binary deletion of eviction, does not fold omitted information into exact retained memories, and keeps the omitted side as entries that can participate in attention\.

At decode time, ResKV combines the exact main cache and the residual cache through a shared\-softmax residual formulation\. Residual entries are not used as a post\-hoc output correction; instead, they participate in the same softmax normalization as the exact main\-cache tokens\. This lets each residual entry restore both numerator mass and denominator mass, making it an approximate attention participant rather than an external value update\. ResKV further uses adaptive residual control at two points\. During cache construction, a validation proxy selects the residual budget for each layer and KV head only when residual entries improve held\-out attention\-output reconstruction\. During decode, a dynamic gate scales the residual logits according to the sharpness of the main\-cache attention, allowing residual mass under diffuse attention while protecting sharp retrieval peaks\.

We evaluate ResKV across multiple long\-context benchmarks, backbones, and representative compression baselines\. We consider both query\-aware and query\-agnostic construction; the latter compresses the cache without access to the future query and therefore better matches realistic deployment\. Across the main tables, ResKV improves all 32 displayed LongBench configurations and 63 of the 64 displayed RULER configurations under the same retained KV budget, with especially clear benefits under tight cache budgets and on tasks that require distributed contextual evidence, with negligible additional memory overhead and stable throughput at long context lengths\. Ablations further confirm the roles of the validation proxy, dynamic gate, and shared softmax\.

Our contributions are threefold:

- •We introduce ResKV, a main\-plus\-residual KV cache representation that accounts for omitted attention mass under a fixed KV budget while keeping selected main\-cache entries exact\.
- •We develop adaptive residual control for ResKV, including a construction\-time validation proxy and a decode\-time dynamic gate\.
- •We show through comprehensive experiments that ResKV improves representative compression baselines across long\-context benchmarks, cache budgets, and construction settings, while preserving practical efficiency\.

## Related Work

##### KV cache eviction and budget allocation\.

KV cache eviction exploits attention sparsity by retaining a subset of past tokens\. Existing methods preserve attention sinks and recent tokens, select entries from accumulated or observed attention, adapt budgets across layers or heads, or use removal\-induced perturbation as saliency\[Xiaoet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib2), Liuet al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib22), Zhanget al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib3), Adnanet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib23), Liet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib5), Orenet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib4), Caiet al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib6), Tanget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib24), Wanget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib25), Fenget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib7), Zhouet al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib26), Guet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib8)\]\.

##### Representing omitted cache information\.

Several methods go beyond permanent deletion by encoding omitted content into compact representations\. CaM, KVMerger, KeepKV, and SemantiCache merge or aggregate evicted states using attention importance, key similarity, merging history, or semantic cores\[Zhanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib9), Wanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib10), Tianet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib11), Wuet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib12)\]\. KVSculpt distills compressed regions into unconstrained virtual KV pairs to preserve attention behavior\[Jiang and Jin,[2026](https://arxiv.org/html/2607.29591#bib.bib13)\]\. RESA estimates omitted contributions with a low\-rank logit prior and compensates sparse\-attention outputs online\[Yanget al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib14)\]\. LESS augments sparse attention with a learned recurrent residual state, and ClusterKV organizes cache access around semantic clusters\[Donget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib15), Liuet al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib16)\]\.

##### Orthogonal KV cache optimizations\.

KV cache quantization reduces per\-entry precision, while sparse\-attention and retrieval systems reduce the entries accessed at each step without necessarily discarding the full cache\[Hooperet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib17), Liuet al\.,[2024b](https://arxiv.org/html/2607.29591#bib.bib18), Kanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib27), Sharmaet al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib28), Tanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib19), Liuet al\.,[2024a](https://arxiv.org/html/2607.29591#bib.bib29)\]\.

## Preliminaries

### KV Cache

An autoregressive Transformer\[Vaswaniet al\.,[2017](https://arxiv.org/html/2607.29591#bib.bib1)\]processes a prompt in a*prefill*phase and then generates tokens one by one in a*decode*phase\. Prefill computes and stores the prompt keys and values in a*KV cache*; each decode step appends the new token’s key and value and attends with a single queryqqover the cached matrices,

o=softmax​\(q​K⊤/d\)​V,o=\\mathrm\{softmax\}\\\!\\big\(qK^\{\\top\}/\\sqrt\{d\}\\big\)\\,V,\(1\)whereK,V∈ℝs×dK,V\\in\\mathbb\{R\}^\{s\\times d\}are the cached keys and values\. The cache grows with the sequence length and, at long context, dominates the memory and bandwidth of inference\[Kwonet al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib20), Shenget al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib21), Liuet al\.,[2024b](https://arxiv.org/html/2607.29591#bib.bib18), Hooperet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib17)\]\.

### KV Cache Compression

Under compression ratioρ∈\(0,1\)\\rho\\in\(0,1\), onlyb=⌊s​\(1−ρ\)⌋b=\\lfloor s\(1\-\\rho\)\\rfloorKV slots are retained\.*Eviction*methods score cached tokens, keep a setM0M\_\{0\}of sizebb, and dropE0=S∖M0E\_\{0\}=S\\setminus M\_\{0\}, so decoding attends only over the pruned cache,

oevict=softmax​\(q​KM0⊤/d\)​VM0,o^\{\\mathrm\{evict\}\}=\\mathrm\{softmax\}\\\!\\big\(qK\_\{M\_\{0\}\}^\{\\top\}/\\sqrt\{d\}\\big\)\\,V\_\{M\_\{0\}\},\(2\)withKM0,VM0K\_\{M\_\{0\}\},V\_\{M\_\{0\}\}the retained rows\[Liuet al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib22), Zhanget al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib3), Adnanet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib23), Liet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib5), Orenet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib4), Tanget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib24), Guet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib8)\]\.*Merging*methods instead fold evicted states back into retained slots, so part of the omitted information remains within the same budget\[Zhanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib9), Wanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib10), Tianet al\.,[2026](https://arxiv.org/html/2607.29591#bib.bib11)\]\.

## Motivation

### Observation

##### Observation 1: hard eviction discards residual information\.

Hard eviction drops all terms from the evicted set, which is risky under diffuse attention because many low\-scoring tokens can still carry substantial aggregate mass and may become useful again for future decoding queries\. To make this explicit, let the rows ofKKandVVbekpk\_\{p\}andvpv\_\{p\}, and defineap=⟨q,kp⟩/da\_\{p\}=\\langle q,k\_\{p\}\\rangle/\\sqrt\{d\}\. Then

o\\displaystyle o=softmax​\(q​K⊤/d\)​V=softmax​\(\[a1,…,as\]\)​V\\displaystyle=\\mathrm\{softmax\}\\\!\\big\(qK^\{\\top\}/\\sqrt\{d\}\\big\)V=\\mathrm\{softmax\}\\\!\\big\(\[a\_\{1\},\\ldots,a\_\{s\}\]\\big\)V\(3\)=∑p=1seap∑p′=1seap′​vp=∑p=1seap​vp∑p′=1seap′\\displaystyle=\\sum\_\{p=1\}^\{s\}\\frac\{e^\{a\_\{p\}\}\}\{\\sum\_\{p^\{\\prime\}=1\}^\{s\}e^\{a\_\{p^\{\\prime\}\}\}\}\\,v\_\{p\}=\\frac\{\\sum\_\{p=1\}^\{s\}e^\{a\_\{p\}\}v\_\{p\}\}\{\\sum\_\{p^\{\\prime\}=1\}^\{s\}e^\{a\_\{p^\{\\prime\}\}\}\}=∑p∈M0eap​vp\+∑p∈E0eap​vp∑p′∈M0eap′\+∑p′∈E0eap′\.\\displaystyle=\\frac\{\\sum\_\{p\\in M\_\{0\}\}e^\{a\_\{p\}\}v\_\{p\}\+\\sum\_\{p\\in E\_\{0\}\}e^\{a\_\{p\}\}v\_\{p\}\}\{\\sum\_\{p^\{\\prime\}\\in M\_\{0\}\}e^\{a\_\{p^\{\\prime\}\}\}\+\\sum\_\{p^\{\\prime\}\\in E\_\{0\}\}e^\{a\_\{p^\{\\prime\}\}\}\}\.In this form, eviction deletes the twoE0E\_\{0\}sums from the numerator and denominator, then renormalizes using only the retained denominator overM0M\_\{0\}\. Thus, even if each evicted token has small individual attention, their aggregate numerator and normalization mass can still be missing from the decoding output\.

##### Observation 2: folding the residual into exact tokens can corrupt them\.

Equation[3](https://arxiv.org/html/2607.29591#Sx4.E3)also shows that the missing information is a separate pair of sums overE0E\_\{0\}\. Merging avoids outright deletion, but it writes these evicted terms into retained slots, changing the retained keys and values themselves\. This is undesirable for sharp retrieval: a token inM0M\_\{0\}that should remain an exact memory can be perturbed by unrelated evicted tokens\. These two failure modes suggest that the missing information should be accounted for separately, without modifying the retained cache\.

### The Residual

Equation[3](https://arxiv.org/html/2607.29591#Sx4.E3)shows that the information lost by eviction is exactly a pair of unnormalized softmax statistics\. For a token subset𝒯\\mathcal\{T\}and queryqq, withap​\(q\)=⟨q,kp⟩/da\_\{p\}\(q\)=\\langle q,k\_\{p\}\\rangle/\\sqrt\{d\}, define

Z𝒯​\(q\)=∑p∈𝒯eap​\(q\),N𝒯​\(q\)=∑p∈𝒯eap​\(q\)​vp,Z\_\{\\mathcal\{T\}\}\(q\)=\\sum\_\{p\\in\\mathcal\{T\}\}e^\{a\_\{p\}\(q\)\},\\qquad N\_\{\\mathcal\{T\}\}\(q\)=\\sum\_\{p\\in\\mathcal\{T\}\}e^\{a\_\{p\}\(q\)\}\\,v\_\{p\},\(4\)so that the full\-cache output isoS​\(q\)=NS​\(q\)/ZS​\(q\)o\_\{S\}\(q\)=N\_\{S\}\(q\)/Z\_\{S\}\(q\)and hard eviction gives the main\-only outputoM0​\(q\)=NM0​\(q\)/ZM0​\(q\)o\_\{M\_\{0\}\}\(q\)=N\_\{M\_\{0\}\}\(q\)/Z\_\{M\_\{0\}\}\(q\)\(Eqs\.[1](https://arxiv.org/html/2607.29591#Sx3.E1)and[2](https://arxiv.org/html/2607.29591#Sx3.E2)\)\.

Now consider a main cache that keeps a setMMexactly, with evicted complementE=S∖ME=S\\setminus M\. We call the part omitted by the main cache the*residual*ofMM:

ℛ​\(M;q\)\\displaystyle\\mathcal\{R\}\(M;q\)=\(NS​\(q\)−NM​\(q\),ZS​\(q\)−ZM​\(q\)\)\\displaystyle=\\big\(N\_\{S\}\(q\)\-N\_\{M\}\(q\),\\,Z\_\{S\}\(q\)\-Z\_\{M\}\(q\)\\big\)\(5\)=\(NE​\(q\),ZE​\(q\)\)\\displaystyle=\\big\(N\_\{E\}\(q\),\\,Z\_\{E\}\(q\)\\big\)=\(∑p∈Eeap​\(q\)​vp,∑p∈Eeap​\(q\)\)\.\\displaystyle=\\left\(\\sum\_\{p\\in E\}e^\{a\_\{p\}\(q\)\}v\_\{p\},\\ \\sum\_\{p\\in E\}e^\{a\_\{p\}\(q\)\}\\right\)\.Addingℛ​\(M;q\)\\mathcal\{R\}\(M;q\)back would recover full attention exactly, but it is not cacheable: the weightseap​\(q\)e^\{a\_\{p\}\(q\)\}depend on the future queryqq, so evaluating it needs the evicted keys and values\. The core idea of ResKV is to trade exactness for reconstruction—keep the few most important tokens exactly and use a compact residual cache to approximateℛ​\(M;q\)\\mathcal\{R\}\(M;q\)through attention\-output reconstruction\. Concretely, the budget is split asb=m\+rb=m\+r: a main cacheMMofmmexact KV slots and a residual cacheRRofrrslots, leaving the retained tokens untouched\. Conceptually, for fixed cachesMMandRR, the best residual reconstructionR^\\widehat\{R\}would make residual\-augmented attention close to full attention:

R^⋆=arg⁡minR^⁡𝔼q∼𝒬​‖oS​\(q\)−o^​\(q\)‖22,\\widehat\{R\}^\{\\star\}=\\arg\\min\_\{\\widehat\{R\}\}\\mathbb\{E\}\_\{q\\sim\\mathcal\{Q\}\}\\left\\\|o\_\{S\}\(q\)\-\\hat\{o\}\(q\)\\right\\\|\_\{2\}^\{2\},\(6\)where

oS​\(q\)=NS​\(q\)ZS​\(q\),o^​\(q\)=NM​\(q\)\+N^R^​\(q\)ZM​\(q\)\+Z^R^​\(q\)\.o\_\{S\}\(q\)=\\frac\{N\_\{S\}\(q\)\}\{Z\_\{S\}\(q\)\},\\qquad\\hat\{o\}\(q\)=\\frac\{N\_\{M\}\(q\)\+\\widehat\{N\}\_\{\\widehat\{R\}\}\(q\)\}\{Z\_\{M\}\(q\)\+\\widehat\{Z\}\_\{\\widehat\{R\}\}\(q\)\}\.\(7\)This form requires the residual contribution to share the same normalization as the main cache, rather than being added as a separate output correction\. The next section describes how ResKV instantiates this attention\-output reconstruction within a fixed KV budget\.

## Residual KV Cache \(ResKV\)

### Overview

ResKV turns the residual view from the previous section into a main\-plus\-residual cache layout\. For each layer and KV head, the fixed budget isb=⌊s​\(1−ρ\)⌋=m\+rb=\\lfloor s\(1\-\\rho\)\\rfloor=m\+r:mmexact main\-cache slots andrrresidual slots, keeping the same persistent KV cache footprint\. Figure[1](https://arxiv.org/html/2607.29591#Sx5.F1)summarizes this layout and its decode\-time combination\.

![Refer to caption](https://arxiv.org/html/2607.29591v1/x1.png)Figure 1:Overview of ResKV\. Given a fixed budgetb=m\+rb=m\+r, ResKV keeps selected tokens exactly in the main cacheMMand reconstructs the omitted sideEEas a residual cacheRR\. During decoding, the main and residual branches are combined by a dynamic gate\.Main entries remain exact, while residual entries summarize omitted tokens and re\-enter the shared softmax through their representative keys, values, and counts\. A validation proxy selects the residual allocation after prefill, and a dynamic gate modulates residual logits during decode\.

### Cache Split and Selection

ResKV first splits the fixed cache budgetbbinto two parts,b=m\+rb=m\+r\. The main cacheMMusesmmslots to store selected tokens with their original keys and values, so these entries remain exact during decoding\. The residual cacheRRuses the remainingrrslots for attention\-output reconstruction of the omitted contribution\. Given a per\-token importance scoresps\_\{p\}and a protected recent window𝒫\\mathcal\{P\}, the main cache is selected as

M=𝒫∪Topm−\|𝒫\|​\(s\),E=S∖M\.M=\\mathcal\{P\}\\cup\\mathrm\{Top\}\_\{\\,m\-\|\\mathcal\{P\}\|\}\(s\),\\qquad E=S\\setminus M\.\(8\)where the recent window protects local continuity in autoregressive decoding\. The remaining prompt tokens form the evicted sideEE, whose contribution will be represented by the residual cache and reintroduced during decoding\.

### Residual Cache Construction

Given the split\(M,E\)\(M,E\)and a residual budgetrr, ResKV approximatesℛ​\(M;q\)\\mathcal\{R\}\(M;q\)by reconstructing the omitted statistics ofEEfor a future queryqq, withap​\(q\)=⟨q,kp⟩/da\_\{p\}\(q\)=\\langle q,k\_\{p\}\\rangle/\\sqrt\{d\}:

ZE​\(q\)=∑p∈Eeap​\(q\),NE​\(q\)=∑p∈Eeap​\(q\)​vp\.Z\_\{E\}\(q\)=\\sum\_\{p\\in E\}e^\{a\_\{p\}\(q\)\},\\qquad N\_\{E\}\(q\)=\\sum\_\{p\\in E\}e^\{a\_\{p\}\(q\)\}v\_\{p\}\.\(9\)Because the weights depend on the future query, ResKV caches compact query\-agnostic summaries instead of the exact statistics\.

LetΠr=\{C1,…,Cr\}\\Pi\_\{r\}=\\\{C\_\{1\},\\ldots,C\_\{r\}\\\}be a partition ofEE\. Each residual entry represents one groupCjC\_\{j\}and stores three quantities:

k¯j=1cj​∑p∈Cjkp,v¯j=1cj​∑p∈Cjvp,cj=\|Cj\|,\\bar\{k\}\_\{j\}=\\frac\{1\}\{c\_\{j\}\}\\sum\_\{p\\in C\_\{j\}\}k\_\{p\},\\quad\\bar\{v\}\_\{j\}=\\frac\{1\}\{c\_\{j\}\}\\sum\_\{p\\in C\_\{j\}\}v\_\{p\},\\quad c\_\{j\}=\|C\_\{j\}\|,\(10\)namely a representative key, a representative value, and a population count\. If the tokens inCjC\_\{j\}have similar logits for a query,ap​\(q\)≈a¯j​\(q\)a\_\{p\}\(q\)\\approx\\bar\{a\}\_\{j\}\(q\)witha¯j​\(q\)=⟨q,k¯j⟩/d\\bar\{a\}\_\{j\}\(q\)=\\langle q,\\bar\{k\}\_\{j\}\\rangle/\\sqrt\{d\}, then the group’s omitted statistics can be reconstructed as

∑p∈Cjeap​\(q\)≈cj​ea¯j​\(q\),∑p∈Cjeap​\(q\)​vp≈cj​ea¯j​\(q\)​v¯j\.\\sum\_\{p\\in C\_\{j\}\}e^\{a\_\{p\}\(q\)\}\\approx c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\},\\qquad\\sum\_\{p\\in C\_\{j\}\}e^\{a\_\{p\}\(q\)\}v\_\{p\}\\approx c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\}\\bar\{v\}\_\{j\}\.\(11\)Thus the representative key and value reconstruct the group’s address and numerator, whilecjc\_\{j\}restores its softmax mass\. Whencj=1c\_\{j\}=1, a residual entry reduces to an exact retained token\.

Since future queries are unavailable at cache\-construction time, ResKV uses key\-space consistency as a query\-agnostic surrogate: nearby keys tend to induce similar logits and can share one residual entry\. We choose the partition by minimizing

Πr⋆=arg⁡minΠr​∑j=1r∑p∈Cj‖kp−k¯j‖22,Cj⊂E\.\\Pi\_\{r\}^\{\\star\}=\\arg\\min\_\{\\Pi\_\{r\}\}\\sum\_\{j=1\}^\{r\}\\sum\_\{p\\in C\_\{j\}\}\\left\\lVert k\_\{p\}\-\\bar\{k\}\_\{j\}\\right\\rVert\_\{2\}^\{2\},\\qquad C\_\{j\}\\subset E\.\(12\)We solve this objective with a few Lloyd iterations\[Lloyd,[1982](https://arxiv.org/html/2607.29591#bib.bib36)\], initialized from high\-scored tokens inEE, independently for each layer and KV head after prefill\. The resultingrrresidual entries occupyrrcache slots, so the total budget remainsb=m\+rb=m\+r\.

### Shared\-Softmax Residual Decode

At decode, exact main\-cache tokens and residual entries participate in one shared softmax rather than using the residual branch as a post\-hoc correction\. For a queryqq, define the main logits

ap=⟨q,kp⟩/d,p∈M,a\_\{p\}=\\langle q,k\_\{p\}\\rangle/\\sqrt\{d\},\\qquad p\\in M,\(13\)and the residual\-entry logits

a~j=⟨q,k¯j⟩/d\+log⁡cj\+log⁡g​\(q\),j∈R\.\\tilde\{a\}\_\{j\}=\\langle q,\\bar\{k\}\_\{j\}\\rangle/\\sqrt\{d\}\+\\log c\_\{j\}\+\\log g\(q\),\\qquad j\\in R\.\(14\)The termlog⁡cj\\log c\_\{j\}injects the population mass from Eq\.[11](https://arxiv.org/html/2607.29591#Sx5.E11), andg​\(q\)∈\[0,1\]g\(q\)\\in\[0,1\]is the dynamic gate defined below\.

We write the shared softmax in the same numerator–denominator form as the motivation:

Z^​\(q\)\\displaystyle\\widehat\{Z\}\(q\)=∑p∈Meap\+∑j∈Rea~j,\\displaystyle=\\sum\_\{p\\in M\}e^\{a\_\{p\}\}\+\\sum\_\{j\\in R\}e^\{\\tilde\{a\}\_\{j\}\},\(15\)N^​\(q\)\\displaystyle\\widehat\{N\}\(q\)=∑p∈Meap​vp\+∑j∈Rea~j​v¯j\.\\displaystyle=\\sum\_\{p\\in M\}e^\{a\_\{p\}\}v\_\{p\}\+\\sum\_\{j\\in R\}e^\{\\tilde\{a\}\_\{j\}\}\\bar\{v\}\_\{j\}\.\(16\)The output is then

o^​\(q\)=N^​\(q\)Z^​\(q\),\\hat\{o\}\(q\)=\\frac\{\\widehat\{N\}\(q\)\}\{\\widehat\{Z\}\(q\)\},\(17\)where residual terms contribute to both the numerator and denominator under the same normalization as the main cache\. Hard eviction is recovered whenR=∅R=\\varnothing, where the second sums vanish and attention normalizes overMMalone\.

### Adaptive Residual Control

ResKV makes residual reconstruction adaptive rather than fixed\. During cache construction, a validation proxy selects the residual budget with an attention\-output objective: residual entries are allocated only when they improve held\-out reconstruction of the attention output\. During decoding, the residual cache is used query\-adaptively\. Broad attention patterns receive more residual mass to recover aggregate omitted contributions, while sharp main\-cache peaks are explicitly protected by down\-weighting the residual branch\. These two controls align residual storage and residual usage with the attention\-output objective in Eq\.[6](https://arxiv.org/html/2607.29591#Sx4.E6)\. Figure[2](https://arxiv.org/html/2607.29591#Sx5.F2)illustrates the construction\-time validation proxy and the decode\-time dynamic gate\.

#### Construction\-Time Validation Proxy\.

The validation proxy selects the residual budgetrrafter prefill\. We split the observation window into a𝒯fit\\mathcal\{T\}\_\{\\mathrm\{fit\}\}part, used to compute token scores and select the main cache, and a disjoint validation part𝒯val\\mathcal\{T\}\_\{\\mathrm\{val\}\}, used only to judge the resulting cache\. For each candidaterrin a small budget grid, ResKV setsm=b−rm=b\-r, uses the fit scores to formMMunder this budget as in Eq\.[8](https://arxiv.org/html/2607.29591#Sx5.E8), constructs residual entries fromE=S∖ME=S\\setminus M, and evaluates the residual\-augmented outputo^\(r\)​\(qt\)\\hat\{o\}^\{\(r\)\}\(q\_\{t\}\)on validation queries\. The validation loss is the reconstruction error against the full\-cache output:

ℒrval=1\|𝒯val\|​∑t∈𝒯val∥o^\(r\)​\(qt\)−oS​\(qt\)∥22\.\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{\\mathrm\{val\}\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{\\mathrm\{val\}\}\}\\big\\lVert\\hat\{o\}^\{\(r\)\}\(q\_\{t\}\)\-o\_\{S\}\(q\_\{t\}\)\\big\\rVert\_\{2\}^\{2\}\.\(18\)The validation output uses the same residual\-attention rule as decoding, including the dynamic gate defined below\. We taker⋆=arg⁡minr\>0⁡ℒrvalr^\{\\star\}=\\arg\\min\_\{r\>0\}\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r\}and enable the residual only if it beats the pure\-main cache by a margin,

ℒr⋆val<\(1−δ\)​ℒ0val,\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r^\{\\star\}\}<\(1\-\\delta\)\\,\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{0\},\(19\)whereℒ0val\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{0\}is the validation loss of ther=0r=0cache that spends the whole budget on main tokens\. Otherwise, the layer/head falls back tor=0r=0\. This makes residual allocation layer\- and KV\-head\-adaptive: residual entries occupy budget only where they reduce held\-out attention\-output error\.

#### Decode\-Time Dynamic Gate\.

When a query has diffuse attention over the main cache, the residual entries can restore omitted aggregate mass\. When the main cache already forms a sharp peak, the averaged residual entries can add diffuse mass and dilute that peak\. ResKV therefore dynamically adjusts the residual contribution for each query\.

![Refer to caption](https://arxiv.org/html/2607.29591v1/x2.png)Figure 2:Adaptive residual control\. Left: the validation proxy selects the residual allocation\. Right: the dynamic gate scales the residual logits according to the main\-cache sharpness before the shared softmax\.Table 1:RULER average scores under different context lengths and retained KV budgets\. Bold marks the better average within each baseline–ResKV pair\. Complete task\-level results are provided in Appendix\.![Refer to caption](https://arxiv.org/html/2607.29591v1/x3.png)Figure 3:Analysis of ResKV efficiency and residual clusterability\. \(a\)–\(b\) compare peak allocated memory and decode throughput across context lengths\. \(c\)–\(d\) show PCA projections of evicted candidate keys from representative layers and KV heads after main\-cache selection\.We measure main\-cache sharpness by

pmaxM​\(q\)=maxp∈M⁡eap∑p′∈Meap′p^\{M\}\_\{\\max\}\(q\)=\\max\_\{p\\in M\}\\frac\{e^\{a\_\{p\}\}\}\{\\sum\_\{p^\{\\prime\}\\in M\}e^\{a\_\{p^\{\\prime\}\}\}\}\(20\)the largest main\-only attention weight\. The dynamic gate is

g~​\(q\)\\displaystyle\\tilde\{g\}\(q\)=sigmoid⁡\(τ−pmaxM​\(q\)Tg\),\\displaystyle=\\operatorname\{sigmoid\}\\\!\\left\(\\frac\{\\tau\-p^\{M\}\_\{\\max\}\(q\)\}\{T\_\{g\}\}\\right\),\(21\)g​\(q\)\\displaystyle g\(q\)=max⁡\(gmin,g~​\(q\)\),\\displaystyle=\\max\\\!\\left\(g\_\{\\min\},\\,\\tilde\{g\}\(q\)\\right\),\(22\)This dynamic gate enters attention as thelog⁡g​\(q\)\\log g\(q\)shift in \([14](https://arxiv.org/html/2607.29591#Sx5.E14)\)\. The temperatureTgT\_\{g\}controls the smoothness of the transition around the thresholdτ\\tau\. A largepmaxM​\(q\)p^\{M\}\_\{\\max\}\(q\)indicates that the main cache already provides a concentrated attention pattern, so the residual logits are down\-weighted\. A smallerpmaxM​\(q\)p^\{M\}\_\{\\max\}\(q\)indicates broader main\-cache attention, so the residual entries retain more mass and can recover omitted aggregate contribution\.

## Experiments

### Experimental Setup

##### Benchmarks\.

We evaluate ResKV on two long\-context benchmarks: LongBench\[Baiet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib30)\]and RULER\[Hsiehet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib31)\]\. LongBench covers 16 real\-world long\-context understanding tasks, including single\-document QA, multi\-document QA, summarization, few\-shot learning, synthetic retrieval, and code completion\. RULER provides controlled long\-context tests for retrieval and aggregation; we report results at both 4K and 32K context lengths over its 13 tasks\.

##### Implementation Details\.

We evaluate ResKV on two instruction\-tuned backbones, LLaMA\-3\.1\-8B\-Instruct\[Grattafiori and others,[2024](https://arxiv.org/html/2607.29591#bib.bib32)\]and Qwen\-2\.5\-7B\-Instruct\[Yang and others,[2024](https://arxiv.org/html/2607.29591#bib.bib33)\], under both query\-aware and query\-agnostic cache construction at compression ratiosρ∈\{0\.6,0\.7,0\.8,0\.9\}\\rho\\in\\\{0\.6,0\.7,0\.8,0\.9\\\}\. ResKV uses the same total KV\-slot budget as the corresponding baseline, split between the main and residual caches\. For all methods, compression is performed once after prefill\. Unless otherwise specified, we use candidate residual budget ratios\{0,0\.05,0\.10,0\.15,0\.20\}\\\{0,0\.05,0\.10,0\.15,0\.20\\\}, 96 fitting queries, 32 validation queries, four Lloyd iterations,δ=0\.01\\delta=0\.01, and enable the dynamic gate at decode time\.

Both backbones use grouped\-query attention \(GQA\)\[Ainslieet al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib34)\]; ResKV therefore constructs residual entries per KV head and shares them within the corresponding query\-head group\. For FlashAttention\-2\[Dao,[2024](https://arxiv.org/html/2607.29591#bib.bib35)\], we keep the main branch in the fused attention path and combine it with the residual branch through log\-sum\-exp statistics, yielding the same shared softmax over main and residual entries\. All experiments are run on a single NVIDIA A100 GPU with 40GB memory\.

##### Baselines\.

We compare ResKV against the standard full KV cache and representative KV cache compression baselines\. H2O\[Zhanget al\.,[2023](https://arxiv.org/html/2607.29591#bib.bib3)\]retains tokens with high accumulated attention mass; SnapKV\[Liet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib5)\]estimates token importance from an observation window; TOVA\[Orenet al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib4)\]scores tokens using recent attention; AdaKV\[Fenget al\.,[2025](https://arxiv.org/html/2607.29591#bib.bib7)\]allocates cache budgets across attention heads; and CaM\[Zhanget al\.,[2024](https://arxiv.org/html/2607.29591#bib.bib9)\]compresses the cache by merging evicted states into retained entries\. We report average tables for SnapKV and AdaKV below, with complete task\-level tables for other baselines and settings provided in Appendix\.

### Results on LongBench

Table 2:LongBench average scores under different retained KV budgets\. Bold marks the better average within each baseline–ResKV pair\. Complete task\-level results are provided in Appendix\.Table[2](https://arxiv.org/html/2607.29591#Sx6.T2)reports the main LongBench average scores across the evaluated settings\. Complete task\-level results and additional baselines are provided in Appendix\.

#### Overall Performance\.

ResKV improves all 32 displayed LongBench configurations, with an average gain of 1\.02 points on LLaMA and Qwen\. The gains are strongest under tight budgets: at 10% and 20% retained KV, ResKV improves the corresponding baseline by 1\.43 and 1\.17 points\. Improvements are also larger in the query\-agnostic setting, which better matches practical serving because the cache is compressed before the next user query is known\.

#### Task\-Level Behavior\.

Task\-level gains are most visible on code and retrieval\-oriented tasks, including RepoBench\-P, LCC, Passage Retrieval, and TREC\. These tasks often depend on distributed definitions, repeated identifiers, or scattered evidence, which can be missed by a small exact cache but recovered through residual entries\.

### Results on RULER

Table[1](https://arxiv.org/html/2607.29591#Sx5.T1)reports the main RULER average scores across the evaluated settings\. Complete task\-level results and additional baselines are provided in Appendix\.

#### Overall Performance\.

ResKV improves 63 of the 64 displayed RULER configurations, with an average gain of 3\.38 points across 4K and 32K contexts and both backbones\. The improvement is particularly clear in the query\-agnostic setting, where the cache is compressed before the future query is available: ResKV improves this setting by 4\.54 points on average, compared with 2\.22 points under query\-aware construction\.

#### Effect Under Tight Cache Budgets\.

Under constrained cache budgets, ResKV improves the corresponding baseline by 3\.47 and 3\.66 points at 10% and 20% retained KV\. Task\-level gains concentrate on controlled retrieval and aggregation behaviors, especially Variable Tracking, FWE, and multi\-key or single\-key retrieval tasks, where useful evidence is distributed across the context\.

### Comparison with Merging Methods

Table 3:Task\-level comparison under Qwen\-2\.5\-7B\-Instruct, 10% retained KV, and query\-agnostic cache construction\.Table[3](https://arxiv.org/html/2607.29591#Sx6.T3)compares SnapKV, CaM, and ResKV under 10% retained KV in the query\-agnostic setting\. CaM can recover useful omitted information on some tasks, but its gains are less stable\. ResKV gives stronger gains by storing omitted information in a separate residual cache while leaving the main cache unchanged\.

### Analysis

##### Peak Allocated Memory\.

Figure[3](https://arxiv.org/html/2607.29591#Sx5.F3)\(a\) shows that ResKV preserves the peak\-memory footprint of the compressed baseline\. Its peak allocated memory essentially overlaps with SnapKV across context lengths, indicating negligible additional memory overhead, while the full cache grows rapidly and runs out of memory \(OOM\) at the longest setting\.

##### Decode Throughput\.

Figure[3](https://arxiv.org/html/2607.29591#Sx5.F3)\(b\) shows that ResKV incurs a moderate throughput cost relative to SnapKV because each decode step evaluates the residual branch and applies the dynamic gate\. The throughput remains substantially higher and more stable than the full cache at long context lengths; full\-cache decode slows down sharply and eventually OOMs, while ResKV maintains a stable decode rate through 128K context\.

##### Clusterability of Evicted Keys\.

Figures[3](https://arxiv.org/html/2607.29591#Sx5.F3)\(c\)–\(d\) examine the keys evicted from the main cache under SnapKV\+ResKV\. The projected keys show clear and repeated geometric patterns rather than unstructured noise, showing that the omitted side still contains organized key\-space structure that residual entries can summarize\.

### Ablation Study

We conduct ablations with LLaMA\-3\.1\-8B\-Instruct using SnapKV as the base method under 10% retained KV and the query\-agnostic construction setting\. We evaluate representative tasks from RULER\-4K and LongBench, and remove one component at a time from complete ResKV\. Complete ResKV performs best on all evaluated tasks\.

Table 4:Ablation results on representative tasks\. Parentheses show drops from ResKV\.Removing the validation proxy reduces S1 and VT by 4\.00 and 7\.04 points, while removing the dynamic gate reduces them by 4\.80 and 4\.48 points\. Replacing the shared softmax with separate main and residual normalizations reduces RepoBench\-P by 3\.00 points\. These drops show that all three components contribute to the final performance\.

## Conclusion

We presented ResKV, a fixed\-budget KV cache representation that preserves selected tokens exactly in a main cache while reconstructing the aggregate attention contribution of omitted tokens with compact residual entries\. Its shared\-softmax residual decode places main\-cache tokens and residual entries under the same normalization, restoring residual numerator and denominator mass while keeping omitted information separate from exact main\-cache entries\. ResKV further controls residual usage with a construction\-time validation proxy and a decode\-time dynamic gate\. Comprehensive evaluations on LongBench and RULER, spanning multiple backbones, cache budgets, compression baselines, and both query\-aware and query\-agnostic settings, show that ResKV improves all 32 displayed LongBench configurations and 63 of the 64 displayed RULER configurations under the same retained KV budget\. The efficiency analysis further shows that these gains incur negligible additional peak\-memory overhead while maintaining stable decode throughput at long context lengths\.

## References

- M\. Adnan, A\. Arunkumar, G\. Jain, P\. J\. Nair, I\. Soloveychik, and P\. Kamath \(2024\)Keyformer: KV cache reduction through key tokens selection for efficient generative inference\.InProceedings of Machine Learning and Systems,Vol\.6\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6)\.
- J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. Sanghai \(2023\)GQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.298)Cited by:[Implementation Details\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px2.p2.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3119–3137\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.172)Cited by:[Benchmarks\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px1.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.InProceedings of the Second Conference on Language Modeling,Cited by:[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations,Cited by:[Implementation Details\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px2.p2.1)\.
- H\. Dong, X\. Yang, Z\. Zhang, Z\. Wang, Y\. Chi, and B\. Chen \(2024\)Get more with LESS: synthesizing recurrence with KV cache compression for efficient LLM inference\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 11437–11452\.Cited by:[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2025\)Ada\-KV: optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[Baselines\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px3.p1.1)\.
- A\. Grattafioriet al\.\(2024\)The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[Implementation Details\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px2.p1.3)\.
- Y\. Gu, X\. Liang, J\. Zhao, and E\. Diao \(2026\)OBCache: optimal brain KV cache pruning for efficient long\-context LLM inference\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length LLM inference with KV cache quantization\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p1.1),[Orthogonal KV cache optimizations\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px3.p1.1),[KV Cache](https://arxiv.org/html/2607.29591#Sx3.SSx1.p1.2)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.InProceedings of the First Conference on Language Modeling,Cited by:[Benchmarks\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px1.p1.1)\.
- B\. Jiang and S\. Jin \(2026\)KVSculpt: KV cache compression as distillation\.External Links:2603\.27819,[Link](https://arxiv.org/abs/2603.27819)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p4.1),[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient error reduction framework for KV cache compression in LLM inference\.InProceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop,Proceedings of Machine Learning Research, Vol\.262,pp\. 305–321\.Cited by:[Orthogonal KV cache optimizations\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px3.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p1.1),[KV Cache](https://arxiv.org/html/2607.29591#Sx3.SSx1.p1.2)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6),[Baselines\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px3.p1.1)\.
- D\. Liu, M\. Chen, B\. Lu, H\. Jiang, Z\. Han, Q\. Zhang, Q\. Chen, C\. Zhang, B\. Ding, K\. Zhang, C\. Chen, F\. Yang, Y\. Yang, and L\. Qiu \(2024a\)RetrievalAttention: accelerating long\-context LLM inference via vector retrieval\.External Links:2409\.10516,[Link](https://arxiv.org/abs/2409.10516)Cited by:[Orthogonal KV cache optimizations\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px3.p1.1)\.
- G\. Liu, C\. Li, J\. Zhao, C\. Zhang, and M\. Guo \(2025\)ClusterKV: manipulating LLM KV cache in semantic space for recallable compression\.InProceedings of the 62nd ACM/IEEE Design Automation Conference,pp\. 1–7\.Cited by:[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for LLM KV cache compression at test time\.External Links:2305\.17118,[Link](https://arxiv.org/abs/2305.17118)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024b\)KIVI: a tuning\-free asymmetric 2bit quantization for KV cache\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32332–32344\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p1.1),[Orthogonal KV cache optimizations\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px3.p1.1),[KV Cache](https://arxiv.org/html/2607.29591#Sx3.SSx1.p1.2)\.
- S\. P\. Lloyd \(1982\)Least squares quantization in PCM\.IEEE Transactions on Information Theory28\(2\),pp\. 129–137\.External Links:[Document](https://dx.doi.org/10.1109/TIT.1982.1056489)Cited by:[Residual Cache Construction](https://arxiv.org/html/2607.29591#Sx5.SSx3.p3.4)\.
- M\. Oren, M\. Hassid, N\. Yarden, Y\. Adi, and R\. Schwartz \(2024\)Transformers are multi\-state RNNs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 18724–18741\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.1043)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6),[Baselines\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px3.p1.1)\.
- A\. Sharma, H\. Ding, J\. Li, N\. Dani, and M\. Zhang \(2025\)MiniKV: pushing the limits of 2\-bit KV cache via compression and system co\-design for efficient long context inference\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 18506–18523\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.952)Cited by:[Orthogonal KV cache optimizations\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, B\. Chen, P\. Liang, C\. Ré, I\. Stoica, and C\. Zhang \(2023\)FlexGen: high\-throughput generative inference of large language models with a single GPU\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 31094–31116\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p1.1),[KV Cache](https://arxiv.org/html/2607.29591#Sx3.SSx1.p1.2)\.
- H\. Tang, Y\. Lin, J\. Lin, Q\. Han, D\. Ke, S\. Hong, Y\. Yao, and G\. Wang \(2025\)RazorAttention: efficient KV cache compression through retrieval heads\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context LLM inference\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 47901–47911\.Cited by:[Orthogonal KV cache optimizations\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Tian, Z\. Wang, Y\. Peng, A\. Yuan, Z\. Wang, B\. Yi, X\. Liu, Y\. Cui, and T\. Yang \(2026\)KeepKV: eliminating output perturbation in KV cache compression via attention\-aware merging\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33259–33267\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i39.40611)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p3.1),[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[KV Cache](https://arxiv.org/html/2607.29591#Sx3.SSx1.p1.1)\.
- Z\. Wang, B\. Jin, Z\. Yu, and M\. Zhang \(2024\)Model tells you where to merge: adaptive KV cache merging for LLMs on long\-context tasks\.External Links:2407\.08454,[Link](https://arxiv.org/abs/2407.08454)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p3.1),[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6)\.
- Z\. Wang, B\. Cui, and S\. Gan \(2025\)SqueezeAttention: 2D management of KV cache in LLM inference via layer\-wise optimal budget\.InInternational Conference on Learning Representations,Cited by:[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1)\.
- S\. Wu, H\. Lin, S\. Chen, T\. Lu, Y\. Zeng, S\. Zhan, H\. Zheng, and H\. Kim \(2026\)SemantiCache: efficient KV cache compression via semantic chunking and clustered merging\.InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing,External Links:[Document](https://dx.doi.org/10.1109/ICASSP55912.2026.11464823)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p3.1),[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Cited by:[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1)\.
- A\. Yanget al\.\(2024\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[Implementation Details\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px2.p1.3)\.
- W\. Yang, H\. Huang, N\. Li, S\. Wang, D\. Yang, Y\. Pan, W\. Xia, S\. Li, and X\. Zou \(2026\)RESA: bringing back what sparse attention ignores with residual estimation\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p4.1),[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Zhang, Y\. Du, G\. Luo, Y\. Zhong, Z\. Zhang, S\. Liu, and R\. Ji \(2024\)CaM: cache merging for memory\-efficient LLM inference\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 58840–58850\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p3.1),[Representing omitted cache information\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px2.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6),[Baselines\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px3.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1),[KV Cache Compression](https://arxiv.org/html/2607.29591#Sx3.SSx2.p1.6),[Baselines\.](https://arxiv.org/html/2607.29591#Sx6.SSx1.SSSx2.Px3.p1.1)\.
- X\. Zhou, W\. Wang, M\. Zeng, J\. Guo, X\. Liu, L\. Shen, M\. Zhang, and L\. Ding \(2025\)DynamicKV: task\-aware adaptive KV cache compression for long context LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,pp\. 8042–8057\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.426)Cited by:[Introduction](https://arxiv.org/html/2607.29591#Sx1.p2.1),[KV cache eviction and budget allocation\.](https://arxiv.org/html/2607.29591#Sx2.SS0.SSS0.Px1.p1.1)\.

## Appendix

## Appendix ATheoretical Analysis

We provide additional derivations for the main\-residual attention form used by ResKV\. The notation follows the preceding sections\. All derivations are written for one layer and one KV head unless otherwise stated\.

### A\.1Main\-Residual Decomposition

This subsection shows why omitted cache statistics should be represented inside the same softmax as retained tokens\. The decomposition below separates the full\-cache output into main\-cache and omitted\-side numerator and denominator terms\.

For a queryqq, letap​\(q\)=⟨q,kp⟩/da\_\{p\}\(q\)=\\langle q,k\_\{p\}\\rangle/\\sqrt\{d\}\. Following Eq\. \([4](https://arxiv.org/html/2607.29591#Sx4.E4)\), the unnormalized denominator and numerator over a token set𝒯\\mathcal\{T\}are denoted byZ𝒯​\(q\)Z\_\{\\mathcal\{T\}\}\(q\)andN𝒯​\(q\)N\_\{\\mathcal\{T\}\}\(q\)\. If the main cache isMMand the omitted side isE=S∖ME=S\\setminus M, full attention can be decomposed as

ZS​\(q\)=ZM​\(q\)\+ZE​\(q\),NS​\(q\)=NM​\(q\)\+NE​\(q\)\.Z\_\{S\}\(q\)=Z\_\{M\}\(q\)\+Z\_\{E\}\(q\),\\qquad N\_\{S\}\(q\)=N\_\{M\}\(q\)\+N\_\{E\}\(q\)\.\(23\)The exact full\-cache output is therefore

oS​\(q\)=NM​\(q\)\+NE​\(q\)ZM​\(q\)\+ZE​\(q\)\.o\_\{S\}\(q\)=\\frac\{N\_\{M\}\(q\)\+N\_\{E\}\(q\)\}\{Z\_\{M\}\(q\)\+Z\_\{E\}\(q\)\}\.\(24\)Hard eviction removes both omitted terms and returnsoM​\(q\)=NM​\(q\)/ZM​\(q\)o\_\{M\}\(q\)=N\_\{M\}\(q\)/Z\_\{M\}\(q\)\. The resulting output difference is

oS​\(q\)−oM​\(q\)=ZM​\(q\)​NE​\(q\)−ZE​\(q\)​NM​\(q\)ZM​\(q\)​\(ZM​\(q\)\+ZE​\(q\)\)\.o\_\{S\}\(q\)\-o\_\{M\}\(q\)=\\frac\{Z\_\{M\}\(q\)N\_\{E\}\(q\)\-Z\_\{E\}\(q\)N\_\{M\}\(q\)\}\{Z\_\{M\}\(q\)\\left\(Z\_\{M\}\(q\)\+Z\_\{E\}\(q\)\\right\)\}\.\(25\)

### A\.2Residual Entry Approximation

This subsection analyzes the approximation made by each residual entry\. The bound shows that a residual summary is accurate when grouped keys induce similar query logits, and it becomes an exact KV token when the group size is one\.

For a residual groupCj⊂EC\_\{j\}\\subset E, the exact omitted denominator and numerator are

Zj​\(q\)=∑p∈Cjeap​\(q\),Nj​\(q\)=∑p∈Cjeap​\(q\)​vp\.Z\_\{j\}\(q\)=\\sum\_\{p\\in C\_\{j\}\}e^\{a\_\{p\}\(q\)\},\\qquad N\_\{j\}\(q\)=\\sum\_\{p\\in C\_\{j\}\}e^\{a\_\{p\}\(q\)\}v\_\{p\}\.\(26\)ResKV stores the mean keyk¯j\\bar\{k\}\_\{j\}, mean valuev¯j\\bar\{v\}\_\{j\}, and countcj=\|Cj\|c\_\{j\}=\|C\_\{j\}\|\. Leta¯j​\(q\)=⟨q,k¯j⟩/d\\bar\{a\}\_\{j\}\(q\)=\\langle q,\\bar\{k\}\_\{j\}\\rangle/\\sqrt\{d\}and write the token logits in the group asap​\(q\)=a¯j​\(q\)\+ϵp​\(q\)a\_\{p\}\(q\)=\\bar\{a\}\_\{j\}\(q\)\+\\epsilon\_\{p\}\(q\)\. Then

Zj​\(q\)=ea¯j​\(q\)​∑p∈Cjeϵp​\(q\)\.Z\_\{j\}\(q\)=e^\{\\bar\{a\}\_\{j\}\(q\)\}\\sum\_\{p\\in C\_\{j\}\}e^\{\\epsilon\_\{p\}\(q\)\}\.\(27\)The residual approximation replaces the group\-level log\-sum withcj​ea¯j​\(q\)c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\}\. When\|ϵp​\(q\)\|≤η\|\\epsilon\_\{p\}\(q\)\|\\leq\\etafor allp∈Cjp\\in C\_\{j\},

\|Zj​\(q\)−cj​ea¯j​\(q\)\|≤cj​ea¯j​\(q\)​\(eη−1\)\.\\left\|Z\_\{j\}\(q\)\-c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\}\\right\|\\leq c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\}\\left\(e^\{\\eta\}\-1\\right\)\.\(28\)If the values in the same group satisfy‖vp‖2≤V\\\|v\_\{p\}\\\|\_\{2\}\\leq V, the numerator error is bounded by

‖Nj​\(q\)−cj​ea¯j​\(q\)​v¯j‖2≤cj​ea¯j​\(q\)​\(eη−1\)​V\.\\left\\\|N\_\{j\}\(q\)\-c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\}\\bar\{v\}\_\{j\}\\right\\\|\_\{2\}\\leq c\_\{j\}e^\{\\bar\{a\}\_\{j\}\(q\)\}\\left\(e^\{\\eta\}\-1\\right\)V\.\(29\)

### A\.3Validation Objective and Dynamic Gate

This subsection describes the two control mechanisms used by ResKV\. The validation proxy selects layer\- and KV\-head\-specific residual capacity under the fixed retained KV budget, while the dynamic gate controls residual influence for each decode query without changing main\-cache logits or values\.

For a candidate residual budgetrr, ResKV constructs a main cache and residual cache, computes the shared\-softmax outputo^r​\(q\)\\hat\{o\}\_\{r\}\(q\)on validation queries, and compares it against the full\-cache output:

ℒrval=1\|𝒯val\|​∑q∈𝒯val‖o^r​\(q\)−oS​\(q\)‖22\.\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{\\mathrm\{val\}\}\|\}\\sum\_\{q\\in\\mathcal\{T\}\_\{\\mathrm\{val\}\}\}\\\|\\hat\{o\}\_\{r\}\(q\)\-o\_\{S\}\(q\)\\\|\_\{2\}^\{2\}\.\(30\)The residual budget is enabled only when this validation loss improves over the pure\-main cache by the relative marginδ\\delta\.

During decode, the dynamic gate modulates only residual logits\. The main\-cache logits and values are left unchanged\. With main\-cache sharpnesspmaxM​\(q\)p^\{M\}\_\{\\max\}\(q\), the gate is

g​\(q\)=max⁡\(gmin,sigmoid⁡\(τ−pmaxM​\(q\)Tg\)\)\.g\(q\)=\\max\\\!\\left\(g\_\{\\min\},\\operatorname\{sigmoid\}\\left\(\\frac\{\\tau\-p^\{M\}\_\{\\max\}\(q\)\}\{T\_\{g\}\}\\right\)\\right\)\.\(31\)A sharp main\-cache distribution reduces residual mass, while a diffuse main\-cache distribution allows residual entries to contribute more strongly\.

## Appendix BExperimental Setup

All experiments are run on a single NVIDIA A100 GPU with 40GB memory\. The implementation is based on PyTorch, Transformers, and FlashAttention\-2\. We evaluate two instruction\-tuned GQA backbones, LLaMA\-3\.1\-8B\-Instruct and Qwen\-2\.5\-7B\-Instruct\. All reported experiments are run once with a fixed random seed of 42\.

For GQA, ResKV constructs the main cache and residual entries at the KV\-head granularity\. All query heads mapped to the same KV head share the selected main indices and residual entries, while their logits and dynamic gates are computed separately\. For FlashAttention\-2, the main\-cache branch remains in the fused attention path\. We request the main\-branch output and per\-query log\-sum\-exp normalizer, evaluate the residual branch over compact residual entries, and merge the two branches through log\-sum\-exp statistics to produce one shared softmax over main and residual entries\.

### B\.1Benchmarks

We evaluate on RULER and LongBench using their official benchmark instances\. The prompts, generation limits, answer formats, and metrics follow the corresponding benchmark definitions without additional data preprocessing\. Across both benchmarks, we use greedy decoding and follow the benchmark metadata for the number of generated tokens\. For LongBench, we use each backbone’s chat template except for code\-completion tasks, where the native completion format is kept\. For RULER, we use the same chat\-template format for all tasks\.

##### RULER\.

RULER evaluates controlled long\-context behavior through synthetic retrieval and aggregation tasks\. We evaluate all 13 RULER tasks at 4K and 32K context lengths\. The official task templates, answer format, and string\-match metric are used without modification\. The reported average is the unweighted mean over the 13 tasks\.

##### LongBench\.

LongBench evaluates real\-world long\-context understanding over 16 tasks, covering single\-document QA, multi\-document QA, summarization, few\-shot learning, synthetic retrieval, and code completion\. The benchmark prompts, answer prefixes, task\-specific maximum generation lengths, and metrics are kept unchanged\. We report the official task\-specific scores and use the unweighted mean over the 16 tasks as the average\.

### B\.2Baseline Setups

#### B\.2\.1Prefill Compression

This subsection fixes the common compression protocol used by all compared methods\. Compression is performed once after the prefill pass, and every paired comparison uses the same total retained KV\-slot budget under the same compression ratio\. It also defines the query\-aware and query\-agnostic settings, where the latter mimics prefix\-cache reuse before future user queries are known\.

We evaluate both query\-aware and query\-agnostic construction\. In the query\-aware setting, the downstream query is appended to the context before cache construction\. In the query\-agnostic setting, the cache is constructed from the reusable context alone, and the query is provided only after the compressed cache has been formed\.

We report compression ratiosρ∈\{0\.6,0\.7,0\.8,0\.9\}\\rho\\in\\\{0\.6,0\.7,0\.8,0\.9\\\}, corresponding to retaining 40%, 30%, 20%, and 10% of the original KV slots\. For each paired comparison, ResKV uses the same base scoring or allocation rule as the corresponding baseline and splits the same retained budget between main and residual entries\.

#### B\.2\.2Compared Methods

This subsection lists the token scoring or budget allocation rule used by each baseline\.

##### H2O\.

H2O retains tokens with large accumulated attention mass\. Following the prefill\-only protocol, scores are computed from observation queries after prefill, and the most recent 64 tokens are protected\.

##### TOVA\.

TOVA scores tokens using recent attention\. We use the last\-token observation window and protect the most recent token\. To keep the protocol consistent across methods, TOVA selection is performed once after prefill rather than updated during decode\.

##### SnapKV\.

SnapKV estimates token importance from an observation window near the end of the prefill sequence\. We use an observation window of 64 tokens, apply average pooling with kernel size 5, and protect the most recent 64 tokens\.

##### AdaKV\.

AdaKV allocates the retained budget across KV heads\. We use SnapKV scores as the token\-importance signal and set the safeguard coefficient to 0\.2\.

### B\.3ResKV Construction Settings

Unless otherwise specified, ResKV uses the same construction settings across benchmarks, backbones, baselines, cache budgets, and query settings\. The candidate residual budget ratios are\{0,0\.05,0\.10,0\.15,0\.20\}\\\{0,0\.05,0\.10,0\.15,0\.20\\\}\. For each layer and KV head, ResKV uses 128 observation queries, with 96 fitting queries and 32 validation queries\. The validation proxy enables residual entries only when the held\-out reconstruction loss improves over the pure\-main cache by at leastδ=0\.01\\delta=0\.01in relative terms\.

Residual construction uses four Lloyd iterations\. The clustering initialization is deterministic because, for each candidate residual budget, initial centers are the evicted keys with the highest main\-cache scores\. Each residual entry stores the mean key, mean value, and population count of one cluster\. The population count is included in the residual logit with exponentβ=1\.0\\beta=1\.0\. The dynamic gate is enabled at decode time with main\-cache sharpness thresholdτ=0\.25\\tau=0\.25, sigmoid slopeα=12\.0\\alpha=12\.0\(equivalently,Tg=1/αT\_\{g\}=1/\\alphain Eq\. \([22](https://arxiv.org/html/2607.29591#Sx5.E22)\)\), and minimum residual gategmin=0\.0g\_\{\\min\}=0\.0\. Residual construction is enabled for all layers\.

## Appendix CExperimental Results

The following tables provide the complete task\-level results corresponding to the average results reported earlier\. Each table includes all evaluated retained KV budgets and both construction settings\. For every baseline–ResKV pair, bold marks the better average score within the same backbone, benchmark, budget, and construction setting\.

##### RULER\.

Tables[5](https://arxiv.org/html/2607.29591#A3.T5),[6](https://arxiv.org/html/2607.29591#A3.T6),[7](https://arxiv.org/html/2607.29591#A3.T7), and[8](https://arxiv.org/html/2607.29591#A3.T8)provide the complete RULER results across the two evaluated context lengths and two backbones\. Each table reports all 13 RULER tasks and the unweighted average\.

Table 5:RULER\-4K results for LLaMA\-3\.1\-8B\-Instruct\. Bold marks the better average within each paired setting\.Table 6:RULER\-4K results for Qwen\-2\.5\-7B\-Instruct\. Bold marks the better average within each paired setting\.Table 7:RULER\-32K results for LLaMA\-3\.1\-8B\-Instruct\. Bold marks the better average within each paired setting\.Table 8:RULER\-32K results for Qwen\-2\.5\-7B\-Instruct\. Bold marks the better average within each paired setting\.
##### LongBench\.

Tables[9](https://arxiv.org/html/2607.29591#A3.T9)and[10](https://arxiv.org/html/2607.29591#A3.T10)provide the complete LongBench results for LLaMA\-3\.1\-8B\-Instruct and Qwen\-2\.5\-7B\-Instruct\. The tables report all 16 LongBench tasks and the unweighted average\.

Table 9:LongBench results for LLaMA\-3\.1\-8B\-Instruct\. Bold marks the better average within each paired setting\.Table 10:LongBench results for Qwen\-2\.5\-7B\-Instruct\. Bold marks the better average within each paired setting\.

## Appendix DImplementation of ResKV

We provide pseudo\-code for ResKV in Algorithms[1](https://arxiv.org/html/2607.29591#alg1)and[2](https://arxiv.org/html/2607.29591#alg2)\. Algorithm[1](https://arxiv.org/html/2607.29591#alg1)summarizes prefill\-time cache construction with residual\-budget validation\. Algorithm[2](https://arxiv.org/html/2607.29591#alg2)summarizes the decode\-time shared\-softmax computation with the dynamic residual gate\. Both algorithms are written for one layer and one KV head\.

Algorithm 1ResKV construction for one \(layer, KV head\)Input:K,VK,V; budgetbb; residual grid𝒢\\mathcal\{G\}; recent windowww; Lloyd stepsTT; marginδ\\delta Output: main cache\(KM,VM\)\(K\_\{M\},V\_\{M\}\)and residual entries𝒮\\mathcal\{S\}

1:form observation queries; split them into fit queries and

𝒯val\\mathcal\{T\}\_\{\\mathrm\{val\}\}
2:compute token scores

sson the fit queries; set recent protection

𝒫\\mathcal\{P\}
3:build the pure\-main cache \(

r=0r=0\) and compute

ℒ0val\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{0\}
4:foreach

r∈𝒢r\\in\\mathcal\{G\}with

r\>0r\>0do

5:

m←b−rm\\leftarrow b\-r; select

MrM\_\{r\}by \([8](https://arxiv.org/html/2607.29591#Sx5.E8)\); set

Er←S∖MrE\_\{r\}\\leftarrow S\\setminus M\_\{r\}
6:build

rrresidual entries from

ErE\_\{r\}by \([12](https://arxiv.org/html/2607.29591#Sx5.E12)\) and \([10](https://arxiv.org/html/2607.29591#Sx5.E10)\)

7:compute

ℒrval\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r\}by \([18](https://arxiv.org/html/2607.29591#Sx5.E18)\)

8:endfor

9:

r⋆←arg⁡minr\>0⁡ℒrvalr^\{\\star\}\\leftarrow\\arg\\min\_\{r\>0\}\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r\}
10:if

ℒr⋆val<\(1−δ\)​ℒ0val\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{r^\{\\star\}\}<\(1\-\\delta\)\\mathcal\{L\}^\{\\mathrm\{val\}\}\_\{0\}then

11:choose

r⋆r^\{\\star\}
12:else

13:choose

r=0r=0
14:endif

15:rebuild

\(M,𝒮\)\(M,\\mathcal\{S\}\)with the chosen

rr; return

\(KM,VM\),𝒮\(K\_\{M\},V\_\{M\}\),\\mathcal\{S\}

Algorithm 2ResKV decoding step for one \(layer, KV head\)Input: queryqq; main cache\(KM,VM\)\(K\_\{M\},V\_\{M\}\); residual entries𝒮\\mathcal\{S\}; dynamic\-gate parametersτ,Tg,gmin\\tau,T\_\{g\},g\_\{\\min\} Output: attention outputo^\\hat\{o\}

1:

ap←⟨q,kp⟩/da\_\{p\}\\leftarrow\\langle q,k\_\{p\}\\rangle/\\sqrt\{d\}for all

p∈Mp\\in M
2:

pmaxM←maxp∈M⁡eap/∑p′∈Meap′p^\{M\}\_\{\\max\}\\leftarrow\\max\_\{p\\in M\}e^\{a\_\{p\}\}/\\sum\_\{p^\{\\prime\}\\in M\}e^\{a\_\{p^\{\\prime\}\}\}
3:

g←max⁡\(gmin,sigmoid⁡\(\(τ−pmaxM\)/Tg\)\)g\\leftarrow\\max\\\!\\big\(g\_\{\\min\},\\,\\operatorname\{sigmoid\}\(\(\\tau\-p^\{M\}\_\{\\max\}\)/T\_\{g\}\)\\big\)
4:

Z←∑p∈MeapZ\\leftarrow\\sum\_\{p\\in M\}e^\{a\_\{p\}\};

N←∑p∈Meap​vpN\\leftarrow\\sum\_\{p\\in M\}e^\{a\_\{p\}\}v\_\{p\}
5:foreach residual entry

\(k¯j,v¯j,cj\)∈𝒮\(\\bar\{k\}\_\{j\},\\bar\{v\}\_\{j\},c\_\{j\}\)\\in\\mathcal\{S\}do

6:

a¯j←⟨q,k¯j⟩/d\+log⁡cj\+log⁡g\\bar\{a\}\_\{j\}\\leftarrow\\langle q,\\bar\{k\}\_\{j\}\\rangle/\\sqrt\{d\}\+\\log c\_\{j\}\+\\log g
7:

Z←Z\+ea¯jZ\\leftarrow Z\+e^\{\\bar\{a\}\_\{j\}\};

N←N\+ea¯j​v¯jN\\leftarrow N\+e^\{\\bar\{a\}\_\{j\}\}\\bar\{v\}\_\{j\}
8:endfor

9:return

o^←N/Z\\hat\{o\}\\leftarrow N/Z

## Appendix ELimitations and Future Work

##### Broader model and serving coverage\.

ResKV is evaluated on two instruction\-tuned GQA backbones and two long\-context benchmarks\. Extending the evaluation to larger models, additional architectures, and serving workloads with larger batch sizes would provide a broader view of its practical behavior\.

##### Residual updates during generation\.

The current construction is performed once after prefill, which matches reusable prefix\-cache serving\. It does not refresh residual entries during long generation\. Future work can study lightweight residual updates for generation\-heavy workloads\.

##### Residual summaries and kernels\.

ResKV uses mean key\-value summaries selected by a validation proxy\. Richer residual summaries may further improve the approximation of omitted tokens\. The current implementation relies on standard PyTorch and FlashAttention\-2 operations, while specialized kernels could further reduce the decode\-time overhead of the residual branch\.