QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
摘要
This paper introduces QEvict, a KV-cache management scheme for LLMs that uses recoverable quantized eviction to handle attention drift during long-context decoding, improving memory efficiency while preserving important historical context.
arXiv:2608.05326v1 Announce Type: new
Abstract: Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key-Value (KV) cache. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention-derived scores. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again. We show that this assumption is brittle during decoding. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full-cache model. To characterize this behaviour, we introduce Future Missed Mass and Global LIR, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions. We propose QEvict, a three-tier KV-cache management scheme that replaces binary retain-or-delete eviction with recoverable eviction. QEvict maintains high-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest-confidence windows. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full-precision. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions. Across long-context understanding, retrieval, and reasoning benchmarks, QEvict consistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention
查看缓存全文
缓存时间: 2026/08/07 07:49
# QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
Source: [https://arxiv.org/html/2608.05326](https://arxiv.org/html/2608.05326)
Ayushman Garg1,\*, Akshita Gupta1,\*, Shaswata Bhattacharya1,\*, Abhishek Gupta2,\*, Sandeep Kumar2,3, Manoj Kumar1
###### Abstract
Autoregressive large language model inference is increasingly constrained by the memory footprint of the Key\-Value \(KV\) cache\. A dominant line of work reduces this footprint by evicting tokens that appear unimportant under attention\-derived scores\. However, such policies make an implicit irreversible decision: once a token is evicted, it cannot become useful again\. We show that this assumption is brittle during decoding\. Token and window importance drift as generated queries evolve, causing standard eviction policies to permanently discard states that later receive substantial attention under the full\-cache model\. To characterize this behaviour, we introduce*Future Missed Mass*and*Global LIR*, two diagnostics that measure future attention assigned to discarded states and the reactivation of historically inactive regions\. We proposeQEvict, a three\-tier KV\-cache management scheme that replaces binary retain\-or\-delete eviction with*recoverable eviction*\.QEvictmaintains high\-confidence windows in full precision, stores intermediate windows in a quantized recoverable tier, and deletes only the lowest\-confidence windows\. During decoding, cumulative attention scores update window importance and when a quantized window becomes important again, it is dequantized and promoted to the full\-precision\. Under a fixed memory budget, this design preserves broader historical context while retaining exact full precision for the most important regions\. Across long\-context understanding, retrieval, and reasoning benchmarks,QEvictconsistently improves over representative eviction and quantization baselines, reducing missed attention and improving information retention\.
11footnotetext:Equal Contribution## 1Introduction
Transformer\-based large language models \(LLMs\) have achieved remarkable success across language understanding, generation, and reasoning\(Vaswaniet al\.[2017](https://arxiv.org/html/2608.05326#bib.bib1); Brownet al\.[2020](https://arxiv.org/html/2608.05326#bib.bib22); Touvronet al\.[2023](https://arxiv.org/html/2608.05326#bib.bib23)\)\. Their deployment in long\-context settings, however, is increasingly constrained by the key–value \(KV\) cache\(Popeet al\.[2023](https://arxiv.org/html/2608.05326#bib.bib26)\)\. Because the cache grows linearly with sequence length and batch size, its memory footprint can rival or exceed that of the model parameters at long context lengths\. This limits batch capacity, increases inference latency, and raises serving costs\. Hardware\-aware attention kernels improve computational efficiency\(Dao[2023](https://arxiv.org/html/2608.05326#bib.bib29)\), while memory\-management systems reduce allocation overhead\(Kwonet al\.[2023](https://arxiv.org/html/2608.05326#bib.bib21)\); neither changes the fundamental storage complexity of the KV cache\.
Table 1:Comparison ofQEvictwith representative KV\-cache compression methods\. A ✓ indicates the presence of a desirable property, while ✗ indicates its absence\.MethodSelectiveEvictionLow\-BitRetentionWindowRoutingDynamicRecoveryStreamingLLM\(Xiaoet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib30)\)✓✗✗✗SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib10)\)✓✗✗✗AdaKV\(Fenget al\.[2024](https://arxiv.org/html/2608.05326#bib.bib13)\)✓✗✗✗CriticalKV\(Fenget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib32)\)✓✗✗✗DefensiveKV\(Fenget al\.[2026](https://arxiv.org/html/2608.05326#bib.bib4)\)✓✗✗✗KIVI\(Liuet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib15)\)✗✓✗✗KVQuant\(Hooperet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib16)\)✗✓✗✗ZipCache\(Heet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib41)\)✗✓✗✗\\rowcolorgray\!15QEvict\(Ours\)✓✓✓✓
A substantial body of work therefore compresses the KV cache through eviction\. StreamingLLM\(Xiaoet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib30)\)retains a small set of attention sinks together with recent tokens, while H2O\(Zhanget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib8)\)exploits attention sparsity to preserve accumulated heavy hitters\. Subsequent methods refine the selection process through prompt\-time observation\(Liet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib10)\), adaptive budget allocation across attention heads\(Fenget al\.[2024](https://arxiv.org/html/2608.05326#bib.bib13)\), value\-aware importance estimation\(Fenget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib32)\), and robustness to uncertain future attention\(Fenget al\.[2026](https://arxiv.org/html/2608.05326#bib.bib4)\)\. In parallel, quantization methods such as KIVI\(Liuet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib15)\), KVQuant\(Hooperet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib16)\), and ZipCache\(Heet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib41)\)retain broader historical coverage by representing KV states at reduced precision\.
Despite this progress, prevailing eviction methods retain a fundamental limitation: cache management is typically formulated as a hard and irreversible token\-level decision\. Natural language is structured through local lexical cohesion, entity continuity, and discourse relations that extend across neighbouring tokens and sentences\(Grosz[1995](https://arxiv.org/html/2608.05326#bib.bib48); Hearst[1997](https://arxiv.org/html/2608.05326#bib.bib43); Passonneau[1997](https://arxiv.org/html/2608.05326#bib.bib49); Barzilay[2008](https://arxiv.org/html/2608.05326#bib.bib44); Koshorek[2018](https://arxiv.org/html/2608.05326#bib.bib50)\)\. Contextualized representations are also strongly shaped by their surrounding context\(Ethayarajh[2019](https://arxiv.org/html/2608.05326#bib.bib45)\), and Transformer attention captures linguistic dependencies such as syntax and coreference across related tokens\(Clarket al\.[2019](https://arxiv.org/html/2608.05326#bib.bib51)\)\. Consistent with this structure, explicit modelling of contiguous spans improves tasks that require question answering, coreference resolution, and relation extraction\(Joshiet al\.[2020](https://arxiv.org/html/2608.05326#bib.bib46)\)\. Independently selecting individual tokens can therefore fragment locally coherent evidence by retaining isolated high\-scoring states while discarding neighbouring states that contribute to their interpretation\.
A second limitation is that token importance changes throughout decoding\. The relevance of cached states is query\-dependent, and the subset required for attention can vary substantially across decoding steps\(Tanget al\.[2024](https://arxiv.org/html/2608.05326#bib.bib47); Ribaret al\.[2024](https://arxiv.org/html/2608.05326#bib.bib52)\)\. A state that appears unimportant under the current query may become critical later when the model resolves a reference, retrieves supporting evidence, or advances along a new reasoning trajectory\. Because eviction is irreversible, information removed during a period of low apparent relevance cannot be recovered when its importance re\-emerges\.
Quantization preserves broader historical coverage by representing cached states at reduced precision\. Existing methods, however, generally do not jointly manage cache residency and numerical precision\. They lack an explicit mechanism for dynamically moving historical states among full\-precision execution, compact recoverable storage, and permanent removal\. This leaves a central design gap:
> *How can the KV cache be compressed aggressively while preserving coherent historical regions whose relevance may emerge only later?*
To address this gap, we proposeQEvict, a recoverable three\-tier KV\-cache hierarchy over contiguous windows\.QEvictperiodically ranks historical windows using cumulative attention, retaining high\-importance windows in full precision, storing intermediate windows in a compact low\-bit tier, and evicting the remainder\. Quantized windows participate in attention and can be promoted when their importance re\-emerges, converting eviction from a one\-shot decision into a dynamic process of demotion, recovery, and promotion\.
Window\-level routing preserves local context and reduces sensitivity to transient token\-level fluctuations, while the recoverable tier expands historical coverage under a fixed memory budget\. To prevent error accumulation across repeated tier transitions,QEvictmaintains a persistent quantized backing store: each window is quantized only on its first demotion, and the resulting low\-bit representation is reused for all subsequent recoveries and promotions\. The implementation supports GQA\(Ainslieet al\.[2023](https://arxiv.org/html/2608.05326#bib.bib39)\)and FlashAttention\-2\(Dao[2023](https://arxiv.org/html/2608.05326#bib.bib29)\), while selectively materializing SDPA attention at routing steps to compute cumulative attention scores\.
Our design is motivated by a systematic study of decoding\-time cache dynamics\. We find that historical importance is highly concentrated but temporally unstable: a small subset of windows dominates attention at any given step, yet the identity of this subset changes throughout generation, and previously inactive windows can later become relevant\. To characterize this behaviour, we introduceFuture Missed MassandGlobal LIR, which measure future attention assigned to discarded states and the reactivation of historically inactive windows, respectively\. We additionally use*Selection Churn*to quantify instability between consecutive cache assignments\. Together, these observations motivate preserving uncertain historical states without allocating full\-precision memory to all of them\.
We evaluateQEvicton LongBench\(Baiet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib35)\), RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib36)\), and GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2608.05326#bib.bib55)\)across three instruction\-tuned LLMs and multiple KV\-memory budgets\. The evaluation covers realistic long\-context understanding, controlled retrieval and reasoning, and autoregressive multi\-step generation\. Across these settings,QEvictconsistently improves the quality–memory trade\-off over representative eviction and quantization baselines\. Our primary contributions are:
- •A systematic analysis of decoding\-time cache dynamics, including two new diagnostics,Future Missed MassandGlobal LIR, together with Selection Churn for measuring routing instability\.
- •QEvict, a recoverable window\-level KV\-cache hierarchy that dynamically routes historical context among full\-precision, low\-bit, and evicted states, supported by a write\-once quantization ledger\.
- •Extensive evaluation across three benchmarks and three model families, demonstrating improved quality–memory trade\-offs under stringent KV\-cache budgets\.
## 2Related Work
The KV cache grows linearly with sequence length and can dominate the memory cost of long\-context autoregressive inference\. Existing methods primarily reduce this overhead through eviction, which retains only selected historical states, or quantization, which stores a larger portion of the cache at lower precision\. Eviction offers selective retention under strict budgets, whereas quantization provides broader historical coverage\.QEvictcombines these advantages through a dynamic hierarchy in which low\-bit storage serves as a recoverable intermediate state\.
\(a\)FMM over decoding\.Window policies reduce attention assigned to discarded states\. The dashed curve treats the R3 tier as inaccessible, isolating the contribution of recoverable low\-bit windows\.
\(b\)Attention\-mass allocation at a matched cache budget\.The three\-tier policy preserves a broad intermediate region in INT2 and reduces the mass assigned to permanent eviction\.
Figure 1:Diagnostics motivating window\-level routing and recoverable low\-bit retention\.\(a\) Contiguous\-window routing reduces Future Missed Mass, while the gap between the solid and dashed R3 curves shows the future utility of the quantized tier\. \(b\) At the same KV\-cache budget, three\-tier allocation preserves substantially more ground\-truth attention mass than binary full\-precision retention and eviction\.#### Local Context and Dynamic Relevance\.
Natural language exhibits local lexical, entity, and discourse coherence across neighbouring positions\(Grosz[1995](https://arxiv.org/html/2608.05326#bib.bib48); Hearst[1997](https://arxiv.org/html/2608.05326#bib.bib43); Passonneau[1997](https://arxiv.org/html/2608.05326#bib.bib49); Barzilay[2008](https://arxiv.org/html/2608.05326#bib.bib44); Koshorek[2018](https://arxiv.org/html/2608.05326#bib.bib50)\)\. Contextual representations and Transformer attention similarly encode dependencies across related spans\(Ethayarajh[2019](https://arxiv.org/html/2608.05326#bib.bib45); Clarket al\.[2019](https://arxiv.org/html/2608.05326#bib.bib51); Joshiet al\.[2020](https://arxiv.org/html/2608.05326#bib.bib46)\), motivating windows as a coherent cache\-management unit\. Relevance is also query\-dependent: Quest\(Tanget al\.[2024](https://arxiv.org/html/2608.05326#bib.bib47)\)and SparQ Attention\(Ribaret al\.[2024](https://arxiv.org/html/2608.05326#bib.bib52)\)selectively access cache regions required by the current query\.QEvictcomplements these approaches by preserving historical states whose future relevance has not yet emerged\.
#### KV\-Cache Eviction\.
StreamingLLM\(Xiaoet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib30)\)retains initial attention sinks and a recent\-token window, while H2O\(Zhanget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib8)\)preserves heavy\-hitter tokens using accumulated attention\. SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib10)\)estimates prompt importance from an observation window, AdaKV\(Fenget al\.[2024](https://arxiv.org/html/2608.05326#bib.bib13)\)adapts cache allocation across attention heads, and CriticalKV\(Fenget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib32)\)incorporates value\-related information into cache selection\. DefensiveKV and Layer\-DefensiveKV\(Fenget al\.[2026](https://arxiv.org/html/2608.05326#bib.bib4)\)further account for uncertainty in future attention\. Although these methods improve scoring and budget allocation, they retain a binary cache state\. A token is either preserved or permanently removed\. Consequently, a state discarded during a period of low apparent importance cannot be recovered when its relevance re\-emerges\. Token\-level selection can also fragment related context by assigning different retention decisions to neighbouring states\.
#### KV\-Cache Quantization\.
KIVI\(Liuet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib15)\)applies asymmetric 2\-bit quantization with different granularities for keys and values\. KVQuant\(Hooperet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib16)\)combines pre\-RoPE key quantization, non\-uniform representations, and outlier handling for sub\-4\-bit compression\. ZipCache\(Heet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib41)\)assigns saliency\-dependent precision at token granularity\.
Prior eviction and quantization methods leave two coupled challenges unresolved: token\-level decisions can fragment context and fluctuate across routing steps, while static cache states cannot recover information whose importance changes during decoding\.QEvictaddresses these limitations by routing contiguous windows among full\-precision, recoverable low\-bit, and evicted states\. This design preserves broader historical coverage under a fixed memory budget, stabilizes cache selection through window\-level aggregation, and supports promotion and demotion as window importance evolves\. Section[3](https://arxiv.org/html/2608.05326#S3)empirically examines the attention concentration, selection instability, and importance revival that motivate this three\-tier hierarchy\.
Figure 2:Overall workflow of theQEvictframework\.Stage 1 partitions the input sequence into contiguous KV windows while preserving sink and recent tokens in full precision\. Stage 2 accumulates attention scores over each window to produce a stable importance ranking\. Stage 3 routes windows under a fixed byte budget into a full\-precision tier, a recoverable INT2 tier that remains available for attention and later promotion, or permanent eviction, with dynamic promotion and demotion as importance evolves\.
## 3Observations and Motivation forQEvict
Conventional KV\-cache eviction assumes that token\-level importance is sufficiently stable for permanent deletion and that a binary retain\-or\-evict decision adequately captures the utility of historical states\. We examine these assumptions using attention traces from an uncompressed FullKV reference, which exposes the future attention assigned to states already removed by a compressed policy\.
#### Diagnostic setup\.
We study Llama\-3\.1\-8B\-Instruct with a512512\-token prefill,256256generated tokens, a20%20\\%KV\-cache budget, and five protected sink tokens\. R1 denotes token\-level eviction with window sizeΩ=1\\Omega=1, R2 denotes two\-tier window eviction withΩ=8\\Omega=8, and R3 and R4 denote three\-tier policies withΩ=8\\Omega=8andΩ=32\\Omega=32, respectively\. R2 and R3 use the same window size and measured cache budget, providing a controlled comparison between binary eviction and recoverable low\-bit retention\. Complete experimental and metric definitions are provided in Appendix[I](https://arxiv.org/html/2608.05326#A9)\.
#### Observation I: Windows stabilize cache decisions\.
We introduceFuture Missed Mass\(FMM\), which measures the fraction of future FullKV attention assigned to states already discarded by a policy, and use*Selection Churn*, the Jaccard distance between historical sets retained at consecutive routing events\. As shown in Figure[1\(a\)](https://arxiv.org/html/2608.05326#S2.F1.sf1), the window\-based policies incur substantially lower FMM than token\-level eviction throughout decoding\. Historical\-set churn per routing event decreases from0\.0170\.017for R1 to0\.00120\.0012for R3 and0\.0180\.018for a larger window; R4\. Window aggregation therefore suppresses short\-lived selection changes, although larger windows introduce a coarser allocation trade\-off\.
#### Observation II: Binary eviction discards a useful middle region\.
Figure[1\(b\)](https://arxiv.org/html/2608.05326#S2.F1.sf2)compares R2 and R3 at the same window size and byte budget\. R2 retains33\.3%33\.3\\%of the FullKV attention mass in full precision and evicts64\.8%64\.8\\%\. R3 retains19\.7%19\.7\\%in full precision and preserves another42\.6%42\.6\\%in INT2, reducing the evicted share to35\.7%35\.7\\%\. The INT2 scores maintain a cosine agreement of0\.98240\.9824with the corresponding FullKV scores\. Moreover, disabling access to the R3 quantized tier produces the FMM curve in Figure[1\(a\)](https://arxiv.org/html/2608.05326#S2.F1.sf1), showing that these windows remain relevant to subsequent decoding\. The ranked cache therefore admits three operational actions: retain in full precision, preserve compactly, or evict\.
#### Observation III: Importance is persistent but not static\.
We further introduceGlobal LIR, which measures how often a window re\-enters the full\-precision set after sustained inactivity\. For R3, oracle top\-ranked windows exhibit a0\.98%0\.98\\%revival rate, confirming that the dominant region is largely persistent\. The deployed full\-precision policy exhibits a higher6\.18%6\.18\\%revival rate, while lagged transitions show that demotion is substantially more frequent than promotion\. Thus, most important windows remain stable, but a non\-negligible subset leaves and later re\-enters the active set\.
Together, these motivate three design choices: contiguous windows as the routing unit, a recoverable low\-bit tier between full\-precision retention and eviction, and bidirectional migration as importance evolves\. Section[4](https://arxiv.org/html/2608.05326#S4)formalizes the resulting byte\-constrained cache hierarchy\.
Table 2:RULER performance at 32K context length on Llama\-3\.1\-8B\-Instruct\.We compareQEvictwith representative KV\-cache eviction methods under a matched 20% cache budget and with quantization baselines at comparable memory footprints\. Scores are string\-match percentages, and KV memory is reported relative to the full\-precision cache\. Excluding Full\-KV, the best and second\-best distinct results within each comparison group are shown in bold and underlined, respectively\.MethodKVmemoryAggregationNeedle\-in\-a\-HaystackQuestion AnsweringTrackingCWEFWEMK\-1MK\-2MK\-3MQMVS\-1S\-2S\-3QA\-1QA\-2VT*Llama\-3\.1\-8B\-Instruct, RULER 32K*\\rowcolorfullkvgray Full\-KV100%45\.2294\.1399\.6099\.6099\.4098\.7599\.10100\.00100\.00100\.0079\.8054\.8099\.24Eviction: 20% KV\-Cache BudgetStreamingLLM20%0\.0493\.4023\.2019\.2023\.0020\.9020\.1022\.2017\.6020\.8023\.4039\.0030\.76SnapKV20%14\.5670\.4098\.4094\.4072\.8098\.8098\.00100\.0097\.6048\.0082\.4052\.0097\.92AdaKV20%19\.2075\.2099\.4093\.6088\.0099\.4099\.40100\.0099\.2056\.0077\.6051\.2099\.52CriticalKV20%26\.8088\.8091\.6029\.4019\.4095\.0093\.60100\.0099\.6042\.4040\.8040\.2097\.76DefensiveKV20%22\.9490\.0099\.8086\.8097\.0098\.6597\.90100\.00100\.0097\.4068\.8045\.8098\.76Layer\-Def\. KV20%17\.8690\.8099\.6099\.4099\.0098\.8598\.45100\.00100\.00100\.0073\.0047\.6098\.56\\rowcolorqevictyellowQEvict20%43\.9893\.6099\.8097\.0077\.8097\.4595\.90100\.00100\.0098\.8084\.6051\.2098\.84Quantization\\rowcolorqevictyellowQEvict20%43\.9893\.6099\.8097\.0077\.8097\.4595\.90100\.00100\.0098\.8084\.6051\.2098\.84KIVI\-2b∼\\sim20%20\.7691\.2098\.0090\.2021\.2092\.4096\.3599\.8098\.4090\.4075\.6050\.2081\.68ZipCache\-4b \(70%\)∼\\sim22%44\.8093\.5399\.8084\.206\.6097\.6095\.25100\.00100\.0080\.0076\.8050\.4097\.60KVQuant\-2b \(s1%\)∼\\sim15%64\.2768\.2161\.0552\.5580\.8876\.0675\.3061\.8262\.9889\.9998\.0095\.0065\.86KVQuant\-3b \(s1%\)∼\\sim22%64\.3870\.4963\.5163\.1589\.8375\.5675\.4663\.6463\.6490\.0097\.8096\.2070\.07
\(a\)Llama\-3\.1\-8B\-Instruct\.
\(b\)Mistral\-7B\-Instruct\-v0\.2\.
\(c\)Qwen2\.5\-7B\-Instruct\.
Figure 3:GSM8K accuracy under KV\-cache compression\.Accuracy–memory trade\-offs ofQEvictand representative eviction and quantization baselines across three instruction\-tuned language models\.Table 3:LongBench performance on Llama\-3\.1\-8B\-Instruct and Mistral\-7B\-Instruct\-v0\.2\.Each cell reports the Llama score followed by the Mistral score\. We compareQEvictwith representative KV\-cache eviction methods under matched20%20\\%,10%10\\%, and5%5\\%budgets, and with quantization baselines at comparable memory footprints\. KV memory is reported relative to the full\-precision cache\. Excluding Full\-KV, the best and second\-best distinct results for each model within every comparison group are shown in bold and underlined, respectively\.MethodKVmemorySingle\-Doc QAMulti\-Doc QASummarizationFew\-Shot LearningNar\.QAQasperMul\.QAHot\.QA2Wi\.QAMusiqueGov\.Re\.QMSumM\.NewsTRECTri\.QASAMSum*Llama\-3\.1\-8B\-Instruct / Mistral\-7B\-Instruct\-v0\.2*\\rowcolorfullkvgray Full\-KV100%30\.5/21\.045\.5/29\.455\.0/47\.156\.0/36\.545\.7/21\.831\.3/19\.135\.1/32\.625\.6/24\.027\.3/27\.173\.0/71\.091\.7/86\.243\.7/43\.0Eviction: 20% KV\-Cache BudgetStreamingLLM20%19\.9/15\.319\.6/13\.324\.4/23\.940\.5/27\.820\.3/14\.715\.1/11\.627\.8/27\.420\.3/20\.122\.3/21\.953\.5/43\.089\.7/76\.940\.0/40\.2SnapKV20%20\.9/16\.728\.3/17\.129\.4/31\.145\.7/30\.130\.0/16\.822\.8/11\.527\.3/26\.221\.6/21\.422\.6/22\.850\.0/51\.092\.2/87\.244\.6/42\.3AdaKV20%21\.7/17\.128\.4/17\.633\.5/32\.350\.8/30\.830\.4/17\.221\.7/13\.426\.4/25\.821\.6/21\.122\.9/22\.955\.0/53\.091\.4/87\.043\.5/43\.2CriticalKV20%29\.9/19\.631\.1/19\.533\.2/33\.851\.3/30\.734\.3/18\.224\.9/13\.428\.5/27\.622\.9/21\.723\.1/23\.355\.0/57\.591\.8/87\.244\.1/41\.8DefensiveKV20%27\.1/19\.240\.8/23\.147\.8/43\.755\.3/34\.338\.7/21\.727\.9/17\.329\.2/28\.623\.7/21\.723\.9/23\.766\.0/67\.091\.8/86\.743\.7/43\.8Layer\-Def\. KV20%27\.7/20\.544\.3/26\.050\.7/47\.053\.2/35\.340\.7/23\.327\.5/18\.030\.6/30\.324\.1/22\.724\.6/24\.769\.0/71\.091\.6/86\.643\.7/43\.5\\rowcolorqevictyellowQEvict20%29\.69/22\.642\.3/27\.755\.74/46\.957\.97/38\.048\.05/20\.832\.99/18\.333\.71/30\.825\.11/24\.126\.89/26\.871\.0/71\.091\.21/85\.541\.7/41\.1Eviction: 10% KV\-Cache BudgetStreamingLLM10%18\.2/15\.816\.6/10\.822\.8/22\.634\.7/23\.116\.6/14\.311\.6/10\.324\.7/24\.818\.8/19\.219\.8/18\.851\.0/32\.088\.8/72\.838\.3/38\.6SnapKV10%18\.9/14\.920\.8/12\.023\.6/25\.243\.0/26\.422\.6/13\.518\.2/10\.824\.1/24\.319\.6/20\.220\.4/20\.944\.5/44\.592\.9/87\.343\.0/40\.7AdaKV10%18\.3/16\.322\.7/13\.726\.1/27\.240\.8/29\.122\.2/15\.417\.8/11\.924\.3/24\.320\.2/19\.920\.9/20\.646\.5/48\.592\.2/87\.143\.3/42\.2CriticalKV10%25\.1/16\.523\.4/13\.926\.1/26\.440\.3/27\.625\.6/15\.619\.7/10\.525\.3/25\.421\.0/20\.620\.9/21\.545\.5/45\.592\.5/87\.642\.5/42\.0DefensiveKV10%20\.9/17\.026\.0/14\.834\.0/32\.648\.3/31\.730\.0/16\.719\.6/11\.625\.1/25\.122\.4/20\.821\.5/21\.654\.0/53\.091\.4/87\.443\.7/43\.1Layer\-Def\. KV10%22\.6/19\.229\.1/16\.737\.5/38\.649\.6/35\.630\.2/20\.621\.7/14\.725\.6/26\.822\.6/21\.421\.4/21\.952\.0/64\.091\.5/87\.444\.0/43\.4\\rowcolorqevictyellowQEvict10%28\.8/20\.134\.94/23\.349\.8/42\.658\.4/36\.546\.4/18\.732\.9/18\.032\.1/29\.624\.9/24\.024\.6/25\.167\.5/69\.089\.8/83\.440\.4/39\.5Eviction: 5% KV\-Cache BudgetStreamingLLM5%14\.3/13\.113\.9/9\.419\.7/20\.330\.1/18\.514\.7/12\.87\.9/8\.222\.2/21\.817\.6/18\.817\.3/15\.938\.5/19\.587\.8/68\.736\.2/37\.5SnapKV5%18\.3/14\.415\.7/11\.419\.8/21\.836\.3/24\.013\.4/14\.713\.2/8\.621\.6/21\.818\.3/19\.217\.8/17\.834\.0/38\.592\.0/86\.642\.1/39\.5AdaKV5%17\.0/15\.116\.4/10\.619\.6/22\.936\.7/24\.817\.1/14\.514\.4/9\.021\.7/21\.618\.7/19\.418\.4/18\.235\.5/41\.092\.8/87\.442\.0/39\.8CriticalKV5%20\.2/14\.717\.4/11\.019\.6/22\.936\.0/25\.815\.5/14\.315\.8/9\.222\.7/23\.018\.7/19\.418\.2/18\.234\.0/40\.092\.5/85\.742\.0/40\.9DefensiveKV5%20\.8/16\.118\.1/11\.121\.6/24\.339\.8/25\.719\.8/15\.414\.2/10\.523\.2/23\.119\.5/20\.018\.9/19\.043\.5/44\.093\.0/87\.143\.2/40\.9Layer\-Def\. KV5%19\.8/15\.620\.1/11\.821\.8/27\.040\.9/28\.019\.8/14\.917\.8/10\.322\.9/23\.919\.7/20\.418\.9/19\.241\.5/47\.093\.0/87\.643\.2/42\.6\\rowcolorqevictyellowQEvict5%27\.3/20\.028\.0/16\.843\.5/34\.656\.4/34\.145\.1/17\.132\.5/25\.128\.8/27\.623\.9/22\.719\.3/19\.963\.0/63\.589\.6/84\.638\.2/38\.2Quantization\\rowcolorqevictyellowQEvict10%28\.8/20\.134\.94/23\.349\.8/42\.658\.4/36\.546\.4/18\.732\.9/18\.032\.1/29\.624\.9/24\.024\.6/25\.167\.5/69\.089\.8/83\.440\.4/39\.5\\rowcolorqevictyellowQEvict20%29\.69/22\.642\.3/27\.755\.74/46\.957\.97/38\.048\.05/20\.832\.99/18\.333\.71/30\.825\.11/24\.126\.89/26\.871\.0/71\.091\.21/85\.541\.7/41\.1KIVI\-2b∼\\sim20%23\.7/20\.636\.8/28\.741\.1/44\.944\.4/35\.530\.0/20\.721\.9/18\.030\.5/32\.624\.4/23\.726\.0/26\.568\.0/71\.087\.8/86\.044\.7/43\.3ZipCache\-4b \(70%\)∼\\sim22%24\.1/20\.443\.3/28\.153\.9/45\.447\.4/33\.250\.8/20\.925\.2/16\.632\.6/29\.922\.9/22\.926\.5/26\.821\.0/39\.591\.0/80\.320\.4/37\.3KVQuant\-2b \(s1%\)∼\\sim15%22\.6/16\.036\.5/23\.447\.1/41\.843\.8/29\.334\.2/17\.323\.2/11\.631\.5/27\.522\.9/22\.625\.8/26\.065\.0/10\.886\.3/68\.041\.9/25\.1KVQuant\-3b \(s1%\)∼\\sim22%28\.6/20\.144\.3/28\.653\.4/47\.153\.9/36\.742\.7/19\.928\.0/17\.333\.4/30\.425\.0/23\.726\.5/26\.772\.5/10\.392\.0/77\.144\.6/37\.0
## 4Methodology
QEvictmanages historical KV states as contiguous windows and dynamically assigns each window to full\-precision, recoverable low\-bit, or evicted state under a fixed budget\.
### 4\.1Preliminaries and Problem Formulation
Consider transformer layerℓ\\ellwithHqH\_\{q\}query heads andHkvH\_\{kv\}KV heads\. At decoding steptt, queryqtℓ,hq^\{\\ell,h\}\_\{t\}attends over the currently accessible keys and values:
atℓ,h=softmax\(qtℓ,h\(Kacc,tℓ\)⊤dkv\),otℓ,h=atℓ,hVacc,tℓ\.a^\{\\ell,h\}\_\{t\}=\\operatorname\{softmax\}\\left\(\\frac\{q^\{\\ell,h\}\_\{t\}\\bigl\(K^\{\\ell\}\_\{\\mathrm\{acc\},t\}\\bigr\)^\{\\top\}\}\{\\sqrt\{d\_\{kv\}\}\}\\right\),\\,\\,o^\{\\ell,h\}\_\{t\}=a^\{\\ell,h\}\_\{t\}V^\{\\ell\}\_\{\\mathrm\{acc\},t\}\.
After reserving a protected sink prefix and recent region, we partition the remaining historical cache into contiguous windows𝒲t\\mathcal\{W\}\_\{t\}\. For each windowww,QEvictassigns a state
ztℓ\(w\)∈\{Full,Quantized,Evicted\}\.z^\{\\ell\}\_\{t\}\(w\)\\in\\left\\\{\\textsc\{Full\},\\textsc\{Quantized\},\\textsc\{Evicted\}\\right\\\}\.LetMf\(w\)M\_\{f\}\(w\)andMq\(w\)M\_\{q\}\(w\)denote the storage costs of a full\-precision and quantized window\. Assignment must satisfy
Bsink\+Blocal\+∑w∈𝒲t\[𝟏\[ztℓ\(w\)=Full\]Mf\(w\)\+\\displaystyle B\_\{\\mathrm\{sink\}\}\+B\_\{\\mathrm\{local\}\}\+\\sum\_\{w\\in\\mathcal\{W\}\_\{t\}\}\\Bigl\[\\mathbf\{1\}\\\!\\left\[z^\{\\ell\}\_\{t\}\(w\)=\\textsc\{Full\}\\right\]M\_\{f\}\(w\)\+𝟏\[ztℓ\(w\)=Quantized\]Mq\(w\)\]≤Btotal\.\\displaystyle\\mathbf\{1\}\\\!\\left\[z^\{\\ell\}\_\{t\}\(w\)=\\textsc\{Quantized\}\\right\]M\_\{q\}\(w\)\\Bigr\]\\leq B\_\{\\mathrm\{total\}\}\.
Thus,QEvictjointly determines cache residency and representation precision rather than selecting a single retained subset\.
### 4\.2Cumulative Window Scoring
Letaτℓ,h\(i\)a^\{\\ell,h\}\_\{\\tau\}\(i\)denote the attention probability assigned to tokeniiby query headhh\. At routing eventtt, the cumulative score of windowwwis updated as
S¯tℓ\(w\)\\displaystyle\\overline\{S\}^\{\\ell\}\_\{t\}\(w\)=S¯t−Ωℓ\(w\)\+1Hq∑h=1Hq∑τ=t−Ω\+1t∑i∈waτℓ,h\(i\),\\displaystyle=\\overline\{S\}^\{\\ell\}\_\{t\-\\Omega\}\(w\)\+\\frac\{1\}\{H\_\{q\}\}\\sum\_\{h=1\}^\{H\_\{q\}\}\\sum\_\{\\tau=t\-\\Omega\+1\}^\{t\}\\sum\_\{i\\in w\}a^\{\\ell,h\}\_\{\\tau\}\(i\),\(1\)t≡0\(modR\)\.\\displaystyle\\qquad t\\equiv 0\\pmod\{R\}\.
The resulting scores induce a layer\-wise ranking over historical windows\. Although we use cumulative attention as the default importance estimator, the hierarchy requires only an ordering and is compatible with alternative ranking functions\. More detailed discussion in Appendix[J](https://arxiv.org/html/2608.05326#A10)
QEvictoperates during both prefill and decoding\. During prefill, prompt attention initializes the window scores and the first tier assignment\. During decoding, newly generated states first enter the protected recent region, and historical windows are rescored and rerouted everyΩ\\Omegagenerated tokens\.
### 4\.3QEvict: Recoverable Cache Hierarchy
At routing eventtt, the cache at layerℓ\\ellis partitioned as:
𝒞tℓ=\(𝒮,ℒt,ℱt,𝒬t,ℰt\),\\mathcal\{C\}^\{\\ell\}\_\{t\}=\\left\(\\mathcal\{S\},\\mathcal\{L\}\_\{t\},\\mathcal\{F\}\_\{t\},\\mathcal\{Q\}\_\{t\},\\mathcal\{E\}\_\{t\}\\right\),\(2\)where𝒮\\mathcal\{S\}andℒt\\mathcal\{L\}\_\{t\}are the protected sink and recent regions and remain in full precision\. Historical windows are assigned to the full\-precision tierℱt\\mathcal\{F\}\_\{t\}, the recoverable low\-bit tier𝒬t\\mathcal\{Q\}\_\{t\}, or the evicted setℰt\\mathcal\{E\}\_\{t\}\.
#### Byte\-Constrained Tier Allocation\.
After reserving the protected regions, the remaining historical budgetBhistB\_\{\\mathrm\{hist\}\}is divided between the full\-precision and quantized tiers\. Given quantized\-tier fractionqq, their capacities are
Kf=⌊\(1−q\)BhistMf⌋,Kq=⌊qBhistMq⌋\.K\_\{f\}=\\left\\lfloor\\frac\{\(1\-q\)B\_\{\\mathrm\{hist\}\}\}\{M\_\{f\}\}\\right\\rfloor,\\qquad K\_\{q\}=\\left\\lfloor\\frac\{qB\_\{\\mathrm\{hist\}\}\}\{M\_\{q\}\}\\right\\rfloor\.\(3\)The per\-window costs include packed codes, quantization parameters, positions, and persistent indexing state\. Complete byte\-level accounting and the default configuration are provided in Appendix[B\.4](https://arxiv.org/html/2608.05326#A2.SS4)and Appendix[B\.5](https://arxiv.org/html/2608.05326#A2.SS5)\. Sensitivity toqq,Ω\\Omega, and quantization precision is studied in Appendix[G](https://arxiv.org/html/2608.05326#A7)\.
#### Migration\-Stable Quantization\.
When a window first enters𝒬t\\mathcal\{Q\}\_\{t\},QEvictapplies asymmetric low\-bit quantization\. Following the distinct distributions of keys and values\(Liuet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib15)\), keys are quantized per channel across the token dimension, whereas values are quantized per token across the channel dimension\. Keys are represented in the pre\-RoPE domain together with their original absolute positions, and RoPE is reapplied after dequantization\.
The codes and quantization parameters produced on the first demotion are retained and reused during later tier transitions\. Promotion reconstructs the same low\-bit approximation in the model’s execution datatype, while a subsequent demotion reuses the existing representation instead of requantizing the reconstructed values\. This prevents approximation error from compounding across repeated migrations\.
#### Dynamic Routing and Recovery\.
At each routing event, the candidate pool contains the current full\-precision and quantized windows together with windows that have exited the recent region:
𝒲tcand=ℱt−Ω∪𝒬t−Ω∪ℒtaged\.\\mathcal\{W\}^\{\\mathrm\{cand\}\}\_\{t\}=\\mathcal\{F\}\_\{t\-\\Omega\}\\cup\\mathcal\{Q\}\_\{t\-\\Omega\}\\cup\\mathcal\{L\}^\{\\mathrm\{aged\}\}\_\{t\}\.\(4\)
Candidates are ranked byS¯tℓ\(w\)\\overline\{S\}^\{\\ell\}\_\{t\}\(w\)\. Highest\-rankedKfK\_\{f\}windows are assigned toℱt\\mathcal\{F\}\_\{t\}, nextKqK\_\{q\}to𝒬t\\mathcal\{Q\}\_\{t\}, and rest toℰt\\mathcal\{E\}\_\{t\}:
ℱt\\displaystyle\\mathcal\{F\}\_\{t\}=TopKKf\(𝒲tcand;S¯tℓ\)\\displaystyle=\\operatorname\{TopK\}\_\{K\_\{f\}\}\\left\(\\mathcal\{W\}^\{\\mathrm\{cand\}\}\_\{t\};\\overline\{S\}^\{\\ell\}\_\{t\}\\right\)
𝒬t\\displaystyle\\mathcal\{Q\}\_\{t\}=TopKKq\(𝒲tcand∖ℱt;S¯tℓ\),\\displaystyle=\\operatorname\{TopK\}\_\{K\_\{q\}\}\\left\(\\mathcal\{W\}^\{\\mathrm\{cand\}\}\_\{t\}\\setminus\\mathcal\{F\}\_\{t\};\\overline\{S\}^\{\\ell\}\_\{t\}\\right\),\(5\)ℰt\\displaystyle\\mathcal\{E\}\_\{t\}=𝒲tcand∖\(ℱt∪𝒬t\)\.\\displaystyle=\\mathcal\{W\}^\{\\mathrm\{cand\}\}\_\{t\}\\setminus\\left\(\\mathcal\{F\}\_\{t\}\\cup\\mathcal\{Q\}\_\{t\}\\right\)\.
Quantized windows remain available to attention and continue accumulating scores\. A window can therefore be promoted when its importance increases or demoted when it declines:
Full⇄Quantized⟶Evicted\.\\textsc\{Full\}\\rightleftarrows\\textsc\{Quantized\}\\longrightarrow\\textsc\{Evicted\}\.\(6\)
Only windows outside the combined capacitiesKf\+KqK\_\{f\}\+K\_\{q\}are permanently removed\.
#### Cache Execution\.
Full\-precision and quantized windows are stored separately\. Before attention, active low\-bit windows are dequantized, their keys are re\-rotated at the original positions, and all accessible states are restored in chronological order\. Standard decoding uses FlashAttention\-2, while SDPA is invoked only at prefill initialization and routing events to expose the attention probabilities required for scoring\. Further execution details are provided in Appendix[H](https://arxiv.org/html/2608.05326#A8)\.
## 5Experiments
#### Setup\.
We evaluateQEvicton Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib24)\), Qwen2\.5\-7B\-Instruct\(Yanget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib54)\), and Mistral\-7B\-Instruct\-v0\.2\(Jianget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib3)\)using LongBench\(Baiet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib35)\), RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib36)\), and GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2608.05326#bib.bib55)\)\. We compare against two baseline families: permanent KV\-cache eviction methods and global KV\-cache quantization methods\. Eviction baselines are evaluated under matched measured\-memory budgets, while quantization baselines are compared at their corresponding performance–memory operating points\.
Unless otherwise stated,QEvictuses routing intervalΩ=8\\Omega=8, five sink tokens,3232recent tokens, an INT2 recoverable tier, and quantized\-tier fractionq=0\.70q=0\.70across LongBench, RULER, and GSM8K\. Complete baseline configurations, decoding protocols, and byte\-level memory accounting are provided in Appendix[B](https://arxiv.org/html/2608.05326#A2)\.
#### Long\-context understanding\.
We evaluate 12 LongBench tasks across single\-document QA, multi\-document QA, summarization, and few\-shot learning at5%5\\%,10%10\\%, and20%20\\%KV\-memory budgets\. Table[3](https://arxiv.org/html/2608.05326#S3.T3)reports results for Llama and Mistral, with Qwen results in Appendix[C](https://arxiv.org/html/2608.05326#A3)\.
QEvictachieves the highest macro\-average across all six model–budget settings\. At20%20\\%memory, it scores46\.446\.4on Llama and37\.837\.8on Mistral, compared with44\.044\.0and37\.437\.4for the strongest matched\-memory eviction baseline\. The gains increase at5%5\\%memory to9\.79\.7and4\.74\.7points, respectively, highlighting the value of recoverable low\-bit retention under tight budgets\.QEvictalso achieves the best20%20\\%macro\-average among quantization methods; at only10%10\\%memory, it remains within1\.21\.2points on Llama and1\.81\.8points on Mistral of the strongest quantization baseline despite using roughly half the KV memory\. Further analyses are provided in Appendix[D](https://arxiv.org/html/2608.05326#A4)\.
#### Long\-range retrieval\.
We evaluate RULER at a3232K context length and a20%20\\%KV\-cache budget on Llama\-3\.1\-8B\-Instruct\. Table[2](https://arxiv.org/html/2608.05326#S3.T2)reports all 13 aggregation, needle\-in\-a\-haystack, question\-answering, and tracking tasks\.QEvictobtains a macro\-average of87\.687\.6, exceeding the strongest matched\-memory eviction baseline, Layer\-DefensiveKV, by1\.21\.2points\. It also outperforms the strongest comparable\-memory quantization baseline by8\.68\.6points and remains within2\.42\.4points of the uncompressed Full\-KV reference\. The results show that broad low\-bit historical coverage is especially effective for retrieval\-intensive long\-context tasks\.
#### Reasoning under compression\.
On GSM8K, Figure[3](https://arxiv.org/html/2608.05326#S3.F3)reports accuracy–memory curves for Llama, Qwen, and Mistral over complete memory sweep\. This evaluates whether the recoverable hierarchy preserves multi\-step generation quality as the available cache budget decreases\. Complete numerical results and model\-specific comparisons with eviction and quantization baselines are provided in Appendix[F](https://arxiv.org/html/2608.05326#A6)\.
#### Ablations\.
We vary the routing intervalΩ\\Omega, quantized\-tier fractionqq, and recoverable precision\. The results favourΩ=8\\Omega=8andq=0\.70q=0\.70across benchmarks\. INT2 provides broader historical coverage under a fixed byte budget, while INT4 trades coverage for lower quantization error\. Full results are reported in Appendix[G](https://arxiv.org/html/2608.05326#A7)\.
### 5\.1Efficiency, Limitations, and Future Work
#### End\-to\-end efficiency\.
Table[4](https://arxiv.org/html/2608.05326#S5.T4)evaluatesQEvictunder matched model, batch, sequence\-length, and backend settings\. With eager SDPA,QEvictadds only0\.5%0\.5\\%TTFT overhead, reduces TPOT by9\.3%9\.3\\%, and improves decoding throughput by9\.8%9\.8\\%over Full\-KV\. Thus, reducing the full\-precision attended cache offsets the cost of low\-bit cache management in the eager implementation\.
With FlashAttention\-2,QEvictreduces peak GPU memory from29\.5429\.54to20\.7820\.78GB, a29\.7%29\.7\\%reduction\. Its current execution path, however, requires periodic attention\-score materialization together with dequantization and cache reconstruction, which lowers decoding throughput\. The resulting trade\-off is backend dependent:QEvictimproves eager decoding efficiency and substantially reduces FlashAttention\-2 memory, while fused low\-bit attention and routing kernels remain necessary to realize both benefits simultaneously\.
Table 4:End\-to\-end inference efficiency on Llama\-3\.1\-8B\-Instruct\.Results use a 256\-token prefill, 1024 generated tokens, batch size 32, and the defaultQEvictconfiguration\. Comparisons are made within each attention backend\.
#### Limitations and future work\.
The current implementation reconstructs active low\-bit windows before attention and selectively falls back to SDPA at routing events to expose attention probabilities\. These operations limit FlashAttention\-2 throughput and introduce temporary workspace overhead\. Future work will integrate dequantization, positional rotation, and mixed\-precision attention directly into fused kernels, and investigate lower\-cost importance estimators that avoid explicit attention\-score materialization\. The present evaluation also focuses on decoder\-only models and fixed\-size contiguous windows, leaving adaptive window boundaries and broader model architectures for future study\.
## 6Conclusion
We introducedQEvict, a byte\-constrained KV\-cache framework that routes historical windows across full\-precision, recoverable INT2, and evicted tiers\. Across LongBench, RULER, and GSM8K,QEvictimproves performance–memory trade\-offs over eviction and quantization baselines, while reducing peak GPU memory and improving eager decoding throughput\. These results show that recoverable low\-bit retention is a practical alternative to permanent eviction\.
## References
- Gqa: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p8.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[§B\.1](https://arxiv.org/html/2608.05326#A2.SS1.p2.1),[§1](https://arxiv.org/html/2608.05326#S1.p10.1),[§5](https://arxiv.org/html/2608.05326#S5.SS0.SSS0.Px1.p1.1)\.
- Barzilay \(2008\)Modeling local coherence: an entity\-based approach\.Computational Linguistics34\(1\),pp\. 1–34\.External Links:[Link](https://aclanthology.org/J08-1001/),[Document](https://dx.doi.org/10.1162/coli.2008.34.1.1)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1877–1901\.Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p1.1)\.
- K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. Manning \(2019\)What does BERT look at? an analysis of BERT’s attention\.InProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,T\. Linzen, G\. Chrupała, Y\. Belinkov, and D\. Hupkes \(Eds\.\),Florence, Italy,pp\. 276–286\.External Links:[Link](https://aclanthology.org/W19-4828/),[Document](https://dx.doi.org/10.18653/v1/W19-4828)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§B\.1](https://arxiv.org/html/2608.05326#A2.SS1.p2.1),[§1](https://arxiv.org/html/2608.05326#S1.p10.1),[§5](https://arxiv.org/html/2608.05326#S5.SS0.SSS0.Px1.p1.1)\.
- T\. Dao \(2023\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p1.1),[§1](https://arxiv.org/html/2608.05326#S1.p8.1)\.
- A\. Devoto, Y\. Zhao, S\. Scardapane, and P\. Minervini \(2024\)A simple and effective l\_2 norm\-based strategy for kv cache compression\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 18476–18499\.Cited by:[4th item](https://arxiv.org/html/2608.05326#A10.I1.i4.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[Appendix K](https://arxiv.org/html/2608.05326#A11.SS0.SSS0.Px2.p1.1),[§B\.1](https://arxiv.org/html/2608.05326#A2.SS1.p1.1),[§C\.2](https://arxiv.org/html/2608.05326#A3.SS2.p1.1),[§F\.1](https://arxiv.org/html/2608.05326#A6.SS1.p2.1),[§I\.1](https://arxiv.org/html/2608.05326#A9.SS1.p2.2),[§5](https://arxiv.org/html/2608.05326#S5.SS0.SSS0.Px1.p1.1)\.
- K\. Ethayarajh \(2019\)How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),K\. Inui, J\. Jiang, V\. Ng, and X\. Wan \(Eds\.\),Hong Kong, China,pp\. 55–65\.External Links:[Link](https://aclanthology.org/D19-1006/),[Document](https://dx.doi.org/10.18653/v1/D19-1006)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Feng, H\. Guo, J\. Lv, S\. K\. Zhou, and X\. Xie \(2026\)DefensiveKV: taming the fragility of kv cache eviction in llm inference\.InThe Fourteenth International Conference on Learning Representations, 2026a\. URL https://openreview\. net/forum,Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.6.6.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2024\)Ada\-kv: optimizing kv cache eviction by adaptive budget allocation for efficient llm inference\.Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.4.4.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Feng, J\. Lv, H\. Guo, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2025\)Identify critical kv cache in llm inference from an output perturbation perspective\.OpenReview preprint\.Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.5.5.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px2.p1.1)\.
- Grosz \(1995\)Centering: a framework for modeling the local coherence of discourse\.Computational Linguistics21\(2\),pp\. 203–225\.External Links:[Link](https://aclanthology.org/J95-2003/)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. He, L\. Zhang, W\. Wu, J\. Liu, H\. Zhou, and B\. Zhuang \(2024\)Zipcache: accurate and efficient kv cache quantization with salient token identification\.Advances in Neural Information Processing Systems37,pp\. 68287–68307\.Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.9.9.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px3.p1.1)\.
- M\. A\. Hearst \(1997\)Text tiling: segmenting text into multi\-paragraph subtopic passages\.Computational Linguistics23\(1\),pp\. 33–64\.External Links:[Link](https://aclanthology.org/J97-1003/)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length llm inference with kv cache quantization\.InNeurIPS,Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.8.8.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.InConference on Language Modeling,Cited by:[§B\.1](https://arxiv.org/html/2608.05326#A2.SS1.p2.1),[§E\.1](https://arxiv.org/html/2608.05326#A5.SS1.p1.2),[§1](https://arxiv.org/html/2608.05326#S1.p10.1),[§5](https://arxiv.org/html/2608.05326#S5.SS0.SSS0.Px1.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§B\.1](https://arxiv.org/html/2608.05326#A2.SS1.p1.1),[§C\.2](https://arxiv.org/html/2608.05326#A3.SS2.p1.1),[§F\.1](https://arxiv.org/html/2608.05326#A6.SS1.p2.1),[§5](https://arxiv.org/html/2608.05326#S5.SS0.SSS0.Px1.p1.1)\.
- M\. Joshi, D\. Chen, Y\. Liu, D\. S\. Weld, L\. Zettlemoyer, and O\. Levy \(2020\)SpanBERT: improving pre\-training by representing and predicting spans\.Transactions of the Association for Computational Linguistics8,pp\. 64–77\.External Links:[Link](https://aclanthology.org/2020.tacl-1.5/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00300)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- Koshorek \(2018\)Text segmentation as a supervised learning task\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 469–473\.External Links:[Link](https://aclanthology.org/N18-2075/),[Document](https://dx.doi.org/10.18653/v1/N18-2075)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InSOSP,Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: llm knows what you are looking for before generation\.InNeurIPS,Cited by:[3rd item](https://arxiv.org/html/2608.05326#A10.I1.i3.p1.3),[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.3.3.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.InNeurIPS,Cited by:[2nd item](https://arxiv.org/html/2608.05326#A10.I1.i2.p1.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024\)KIVI: a tuning\-free asymmetric 2bit quantization for kv cache\.Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.7.7.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2608.05326#S4.SS3.SSS0.Px2.p1.1)\.
- I\. Mirzadeh, K\. Alizadeh, H\. Shahrokhi, O\. Tuzel, S\. Bengio, and M\. Farajtabar \(2024\)Gsm\-symbolic: understanding the limitations of mathematical reasoning in large language models\.arXiv preprint arXiv:2410\.05229\.Cited by:[§F\.1](https://arxiv.org/html/2608.05326#A6.SS1.p1.1)\.
- Passonneau \(1997\)Discourse segmentation by human and automated means\.Computational Linguistics23\(1\),pp\. 103–139\.External Links:[Link](https://aclanthology.org/J97-1005/)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p3.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean \(2023\)Efficiently scaling transformer inference\.Proceedings of Machine Learning and Systems5,pp\. 606–624\.Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p1.1)\.
- L\. Ribar, I\. Chelombiev, L\. Hudlass\-Galley, C\. Blake, C\. Luschi, and D\. Orr \(2024\)SparQ attention: bandwidth\-efficient LLM inference\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 42558–42583\.External Links:[Link](https://proceedings.mlr.press/v235/ribar24a.html)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p4.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context LLM inference\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 47901–47911\.External Links:[Link](https://proceedings.mlr.press/v235/tang24l.html)Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p4.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2608.05326#S1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Cited by:[§B\.2](https://arxiv.org/html/2608.05326#A2.SS2.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2608.05326#S1.T1.3.1.2.2.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§B\.1](https://arxiv.org/html/2608.05326#A2.SS1.p1.1),[§C\.2](https://arxiv.org/html/2608.05326#A3.SS2.p1.1),[§F\.1](https://arxiv.org/html/2608.05326#A6.SS1.p2.1),[§5](https://arxiv.org/html/2608.05326#S5.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InNeurIPS,Cited by:[2nd item](https://arxiv.org/html/2608.05326#A10.I1.i2.p1.1),[§1](https://arxiv.org/html/2608.05326#S1.p2.1),[§2](https://arxiv.org/html/2608.05326#S2.SS0.SSS0.Px2.p1.1)\.
## Appendix AAlgorithm
Algorithm 1QEvictDynamic Window Routing0:Previous tiers
ℱt−Ω\\mathcal\{F\}\_\{t\-\\Omega\}and
𝒬t−Ω\\mathcal\{Q\}\_\{t\-\\Omega\}
0:Aged local windows
ℒaged\\mathcal\{L\}\_\{\\mathrm\{aged\}\}, scores
S¯t\(w\)\\bar\{S\}\_\{t\}\(w\), capacities
Kf,KqK\_\{f\},K\_\{q\}, ledger
Λ\\Lambda
1:
𝒞cand←ℱt−Ω∪𝒬t−Ω∪ℒaged\\mathcal\{C\}\_\{\\mathrm\{cand\}\}\\leftarrow\\mathcal\{F\}\_\{t\-\\Omega\}\\cup\\mathcal\{Q\}\_\{t\-\\Omega\}\\cup\\mathcal\{L\}\_\{\\mathrm\{aged\}\}
2:
ℱt←TopKKf\{S¯t\(w\):w∈𝒞cand\}\\mathcal\{F\}\_\{t\}\\leftarrow\\operatorname\{TopK\}\_\{K\_\{f\}\}\\\{\\bar\{S\}\_\{t\}\(w\):w\\in\\mathcal\{C\}\_\{\\mathrm\{cand\}\}\\\}
3:
𝒬t←TopKKq\{S¯t\(w\):w∈𝒞cand∖ℱt\}\\mathcal\{Q\}\_\{t\}\\leftarrow\\operatorname\{TopK\}\_\{K\_\{q\}\}\\\{\\bar\{S\}\_\{t\}\(w\):w\\in\\mathcal\{C\}\_\{\\mathrm\{cand\}\}\\setminus\\mathcal\{F\}\_\{t\}\\\}
4:
ℰt←𝒞cand∖\(ℱt∪𝒬t\)\\mathcal\{E\}\_\{t\}\\leftarrow\\mathcal\{C\}\_\{\\mathrm\{cand\}\}\\setminus\(\\mathcal\{F\}\_\{t\}\\cup\\mathcal\{Q\}\_\{t\}\)
5:for
w∈ℱtw\\in\\mathcal\{F\}\_\{t\}do
6:if
w∈𝒬t−Ωw\\in\\mathcal\{Q\}\_\{t\-\\Omega\}then
7:
\(K^w,V^w\)←Dequantize\(Λw\)\(\\hat\{K\}\_\{w\},\\hat\{V\}\_\{w\}\)\\leftarrow\\operatorname\{Dequantize\}\(\\Lambda\_\{w\}\)
8:mark
Λw\\Lambda\_\{w\}dormant
9:endif
10:endfor
11:for
w∈𝒬tw\\in\\mathcal\{Q\}\_\{t\}do
12:if
wwhas no ledger entrythen
13:quantize
wwonce and create
Λw\\Lambda\_\{w\}
14:elseif
Λw\\Lambda\_\{w\}is dormantthen
15:reactivate
Λw\\Lambda\_\{w\}
16:endif
17:endfor
18:for
w∈ℰtw\\in\\mathcal\{E\}\_\{t\}do
19:release
wwand delete
Λw\\Lambda\_\{w\}if present
20:endfor
21:reconstruct the effective cache in chronological order
22:return
\(ℱt,𝒬t,ℰt,Λ\)\(\\mathcal\{F\}\_\{t\},\\mathcal\{Q\}\_\{t\},\\mathcal\{E\}\_\{t\},\\Lambda\)
## Appendix BComplete Experimental Details
This section specifies the models, inference protocol, baselines, comparison regimes, memory accounting, and defaultQEvictconfiguration used throughout the evaluation\.
### B\.1Models and Inference Protocol
We evaluateQEvicton Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib24)\), Qwen2\.5\-7B\-Instruct\(Yanget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib54)\), and Mistral\-7B\-Instruct\-v0\.2\(Jianget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib3)\)\.QEvictis training\-free and modifies only KV\-cache management at inference time\. Within each benchmark, all methods use the same checkpoint, tokenizer, prompt template, input and generation limits, and answer post\-processing\. Unless stated otherwise, decoding is greedy\.
We evaluate three complementary capabilities: long\-context understanding on LongBench\(Baiet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib35)\), controlled retrieval and aggregation on RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib36)\), and multi\-step mathematical reasoning on GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2608.05326#bib.bib55)\)\.
### B\.2Baselines
We compare against eviction and quantization methods, whose technical details are reviewed in Section[2](https://arxiv.org/html/2608.05326#S2)\.
#### Eviction baselines\.
The eviction baselines are StreamingLLM\(Xiaoet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib30)\), SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib10)\), AdaKV\(Fenget al\.[2024](https://arxiv.org/html/2608.05326#bib.bib13)\), CriticalKV\(Fenget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib32)\), DefensiveKV, and Layer\-DefensiveKV\(Fenget al\.[2026](https://arxiv.org/html/2608.05326#bib.bib4)\)\. They cover sink\- and recency\-based retention, accumulated\-attention scoring, prompt observation, head\-adaptive allocation, value\-aware importance, and drift\-aware eviction\.
#### Quantization baselines\.
We compare against KIVI\(Liuet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib15)\), KVQuant\(Hooperet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib16)\), and ZipCache\(Heet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib41)\)\. These methods retain broad historical coverage at reduced precision, whereasQEvictjointly determines residency and precision through full\-precision, quantized, and evicted tiers\.
#### Uncompressed reference\.
Full\-KV stores the complete key and value cache in the model’s native inference datatype and serves as the uncompressed reference\.
### B\.3Comparison Regimes
Eviction and quantization methods are evaluated under separate protocols because nominal compression settings do not necessarily yield the same physical memory footprint\.
#### Eviction baselines\.
For every method𝒜\\mathcal\{A\}, we define the measured KV\-memory ratio asρ𝒜=M𝒜/MFullKV\\rho\_\{\\mathcal\{A\}\}=M\_\{\\mathcal\{A\}\}/M\_\{\\mathrm\{FullKV\}\}\. LongBench usesρ𝒜∈\{0\.05,0\.10,0\.20\}\\rho\_\{\\mathcal\{A\}\}\\in\\\{0\.05,0\.10,0\.20\\\}, RULER 32K usesρ𝒜=0\.20\\rho\_\{\\mathcal\{A\}\}=0\.20, and GSM8K usesρ𝒜∈\{0\.10,0\.20,0\.40,0\.60,0\.80,1\.0\}\\rho\_\{\\mathcal\{A\}\}\\in\\\{0\.10,0\.20,0\.40,0\.60,0\.80,1\.0\\\}for the eviction comparisons\.
#### Quantization baselines\.
Quantization methods are compared at their measured performance–memory operating points rather than by nominal bit width\.
### B\.4Memory Accounting
For a model withLLlayers,HkvH\_\{kv\}KV heads, head dimensiondhd\_\{h\}, sequence lengthSS, andcfc\_\{f\}bytes per element, the uncompressed KV\-cache footprint is
MFullKV=2LHkvdhScf,M\_\{\\mathrm\{FullKV\}\}=2LH\_\{kv\}d\_\{h\}Sc\_\{f\},\(7\)where the factor of two accounts for keys and values\.
All persistent cache\-related storage is included in the measured budget, including retained KV tensors, packed codes, quantization parameters, full\-precision residual regions, outliers, and indexing metadata when applicable\. ForQEvict,
MQEvict=\\displaystyle M\_\{\\mathrm\{QEvict\}\}=Msink\+Mrecent\+MFP\+MQ\+Mquant\-meta\+Mposition\+Mledger\.\\displaystyle M\_\{\\mathrm\{sink\}\}\+M\_\{\\mathrm\{recent\}\}\+M\_\{\\mathrm\{FP\}\}\+M\_\{\\mathrm\{Q\}\}\+M\_\{\\mathrm\{quant\\text\{\-\}meta\}\}\+M\_\{\\mathrm\{position\}\}\+M\_\{\\mathrm\{ledger\}\}\.\(8\)Here,MQM\_\{\\mathrm\{Q\}\}contains packed low\-bit keys and values,Mquant\-metaM\_\{\\mathrm\{quant\\text\{\-\}meta\}\}contains their scales and offsets, andMledgerM\_\{\\mathrm\{ledger\}\}contains persistent migration records\. Temporary workspaces are excluded from the KV\-cache budget and captured separately by peak GPU memory\.
### B\.5QEvictConfiguration
Unless explicitly varied,QEvictuses routing intervalΩ=8\\Omega=8, five sink tokens, and an INT2 recoverable tier\. After allocating the protected regions, a fractionq=0\.70q=0\.70of the remaining historical budget is assigned to the quantized tier and1−q1\-qto the full\-precision historical tier\. This configuration is used for LongBench, RULER, and GSM8K\. Sensitivity toΩ\\Omega,qq, and quantization precision is reported in Appendix[G](https://arxiv.org/html/2608.05326#A7)\.
## Appendix CLongBench Evaluation
### C\.1Task Families and Metrics
We evaluate 12 LongBench tasks spanning the four families summarized in Table[5](https://arxiv.org/html/2608.05326#A3.T5)and we store 128 recent tokens for this task\.
Table 5:LongBench task families used in our evaluation\.Table 6:LongBench performance on Qwen2\.5\-7B\-Instruct\.We compareQEvictwith representative KV\-cache eviction methods under matched20%20\\%,10%10\\%, and5%5\\%budgets, and with quantization baselines at comparable memory footprints\. KV memory is reported relative to the full\-precision cache\. Excluding Full\-KV, the best and second\-best distinct results within each comparison group are shown in bold and underlined, respectively\.MethodKVmemorySingle\-Doc QAMulti\-Doc QASummarizationFew\-Shot LearningNar\.QAQasperMul\.QAHot\.QA2Wi\.QAMusiqueGov\.Re\.QMSumM\.NewsTRECTri\.QASAMSum*Qwen2\.5\-7B\-Instruct*\\rowcolorfullkvgray Full\-KV100%29\.3346\.3650\.2755\.9642\.4927\.5533\.7324\.2725\.4173\.5086\.5141\.20Eviction: 20% KV\-Cache BudgetStreamingLLM20%18\.5620\.0523\.8335\.0227\.0914\.3428\.5018\.9921\.0557\.0078\.4942\.96SnapKV20%22\.2122\.4330\.6745\.8928\.7722\.0028\.9819\.9421\.2849\.0087\.6140\.20AdaKV20%23\.3522\.5329\.5744\.5728\.6521\.3728\.6120\.0921\.4553\.5087\.8940\.00CriticalKV20%22\.8226\.7231\.7446\.8537\.0924\.9529\.7520\.7322\.1553\.0088\.9445\.61DefensiveKV20%20\.8030\.3034\.6044\.3032\.0019\.2030\.4020\.6022\.4037\.5087\.2046\.10Layer\-Def\. KV20%20\.4630\.4539\.3446\.5436\.1621\.2831\.0321\.4922\.5061\.0087\.2345\.36\\rowcolorqevictyellowQEvict20%19\.9735\.7444\.2849\.1537\.4324\.3726\.6822\.0425\.2867\.0089\.6344\.17Eviction: 10% KV\-Cache BudgetStreamingLLM10%18\.2016\.0121\.2527\.3922\.8311\.3825\.8818\.3017\.8749\.5071\.0641\.03SnapKV10%19\.4314\.0624\.5536\.4823\.9817\.1326\.3518\.3418\.6944\.5087\.7139\.17AdaKV10%20\.7815\.2225\.6737\.2722\.1317\.0026\.2718\.0919\.0443\.5087\.1439\.67CriticalKV10%19\.9019\.6725\.3340\.7527\.3619\.2127\.0218\.8319\.9946\.5088\.6445\.18DefensiveKV10%19\.0020\.1026\.5034\.0031\.7016\.4028\.0019\.1020\.2027\.5087\.9043\.60Layer\-Def\. KV10%21\.1921\.9326\.8137\.5322\.3217\.4320\.4418\.8321\.9943\.5087\.5144\.62\\rowcolorqevictyellowQEvict10%23\.8834\.4637\.7149\.2237\.1323\.5629\.3522\.1023\.7264\.5083\.4342\.55Eviction: 5% KV\-Cache BudgetStreamingLLM5%11\.9812\.0220\.2828\.0522\.139\.8723\.4217\.1015\.8238\.5045\.6934\.11SnapKV5%18\.1714\.3519\.8529\.0921\.9714\.3024\.4617\.4616\.8029\.5086\.0041\.51AdaKV5%17\.4614\.4121\.0629\.6022\.8412\.8524\.4217\.3217\.3233\.0087\.8041\.54CriticalKV5%17\.8714\.4821\.7735\.3620\.3914\.3724\.3917\.3817\.0336\.0088\.4543\.11DefensiveKV5%16\.8014\.9021\.2030\.5024\.8014\.3025\.6017\.9017\.6027\.5087\.9043\.20Layer\-Def\. KV5%15\.8614\.3322\.1230\.5124\.5014\.9425\.2417\.1917\.0834\.0087\.9042\.05\\rowcolorqevictyellowQEvict5%23\.7328\.6133\.7849\.7735\.1622\.8928\.7421\.8518\.5459\.5084\.9341\.48Quantization\\rowcolorqevictyellowQEvict10%23\.8834\.4637\.7149\.2237\.1323\.5629\.3522\.1023\.7264\.5083\.4342\.55\\rowcolorqevictyellowQEvict20%19\.9735\.7444\.2849\.1537\.4324\.3726\.6822\.0425\.2867\.0089\.6344\.17KIVI\-2b∼\\sim20%24\.2328\.2542\.6445\.9038\.0724\.1525\.4122\.1722\.5565\.2576\.2143\.94ZipCache\-4b \(70%\)∼\\sim22%22\.8234\.5345\.8148\.9240\.0323\.5029\.1021\.6523\.4934\.5090\.9740\.15KVQuant\-2b \(s1%\)∼\\sim15%20\.9832\.5742\.5841\.5332\.2516\.0028\.9921\.9323\.0257\.2579\.5435\.96KVQuant\-3b \(s1%\)∼\\sim22%24\.1937\.0745\.1653\.9439\.4326\.2431\.3823\.5923\.8669\.0085\.4439\.33
We use the official metric for each dataset and report task\-level scores\.
### C\.2Complete Results
All baselines, configurations, memory\-accounting rules, and comparison protocols follow Appendix[B](https://arxiv.org/html/2608.05326#A2), particularly Appendix[B\.3](https://arxiv.org/html/2608.05326#A2.SS3)\. Results for LongBench on Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib24)\)and Mistral\-7B\-Instruct\-v0\.2\(Jianget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib3)\)are reported in the main\-paper Table[3](https://arxiv.org/html/2608.05326#S3.T3)and for Qwen2\.5\-7B\-Instruct\(Yanget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib54)\)is reported in Table[6](https://arxiv.org/html/2608.05326#A3.T6)\.
## Appendix DLongBench Performance–Memory Pareto Analysis
Fixed\-budget comparisons capture performance at only a small number of operating points and may obscure the overall trade\-off between task quality and KV\-cache memory\. We therefore analyse the performance–memory Pareto frontier for each LongBench task family, model, and baseline class\.
Let\(ρi,Si\)\(\\rho\_\{i\},S\_\{i\}\)denote an operating point, whereρi\\rho\_\{i\}is the measured KV\-memory ratio andSiS\_\{i\}is the corresponding task\-family score\. Operating pointiidominatesjjwhen
ρi≤ρj,Si≥Sj,\\rho\_\{i\}\\leq\\rho\_\{j\},\\qquad S\_\{i\}\\geq S\_\{j\},\(9\)with at least one strict inequality\. The Pareto frontier is the set of non\-dominated operating points\. Points closer to the upper\-left corner are preferable because they achieve higher task performance with lower KV\-memory usage\.
We construct separate comparisons against eviction and quantization baselines for Single\-Document QA, Multi\-Document QA, Summarization, and Few\-Shot Learning\. This produces eight panels per model and 24 panels across Llama\-3\.1\-8B\-Instruct, Mistral\-7B\-Instruct\-v0\.2, and Qwen2\.5\-7B\-Instruct\. Figures[4](https://arxiv.org/html/2608.05326#A4.F4),[5](https://arxiv.org/html/2608.05326#A4.F5), and[6](https://arxiv.org/html/2608.05326#A4.F6)report the corresponding Pareto plots\. The left column of each figure comparesQEvictwith eviction methods, while the right column compares it with quantization methods\. Horizontal dashed lines indicate the performance of the uncompressed FullKV reference\.
These plots complement the fixed\-budget tables by showing whether an advantage persists across multiple memory regimes rather than at a single selected budget\. They also distinguish methods that improve performance at matched memory from those that reduce memory at comparable performance\.
\(a\)Single\-Document QA: eviction\.
\(b\)Single\-Document QA: quantization\.
\(c\)Multi\-Document QA: eviction\.
\(d\)Multi\-Document QA: quantization\.
\(e\)Summarization: eviction\.
\(f\)Summarization: quantization\.
\(g\)Few\-Shot Learning: eviction\.
\(h\)Few\-Shot Learning: quantization\.
Figure 4:LongBench performance–memory Pareto analysis for Llama\-3\.1\-8B\-Instruct\.Rows correspond to Single\-Document QA, Multi\-Document QA, Summarization, and Few\-Shot Learning\. The left and right columns compareQEvictwith eviction and quantization baselines, respectively\. Dashed horizontal lines denote FullKV performance\.\(a\)Single\-Document QA: eviction\.
\(b\)Single\-Document QA: quantization\.
\(c\)Multi\-Document QA: eviction\.
\(d\)Multi\-Document QA: quantization\.
\(e\)Summarization: eviction\.
\(f\)Summarization: quantization\.
\(g\)Few\-Shot Learning: eviction\.
\(h\)Few\-Shot Learning: quantization\.
Figure 5:LongBench performance–memory Pareto analysis for Mistral\-7B\-Instruct\-v0\.2\.Rows correspond to Single\-Document QA, Multi\-Document QA, Summarization, and Few\-Shot Learning\. The left and right columns compareQEvictwith eviction and quantization baselines, respectively\. Dashed horizontal lines denote FullKV performance\.\(a\)Single\-Document QA: eviction\.
\(b\)Single\-Document QA: quantization\.
\(c\)Multi\-Document QA: eviction\.
\(d\)Multi\-Document QA: quantization\.
\(e\)Summarization: eviction\.
\(f\)Summarization: quantization\.
\(g\)Few\-Shot Learning: eviction\.
\(h\)Few\-Shot Learning: quantization\.
Figure 6:LongBench performance–memory Pareto analysis for Qwen2\.5\-7B\-Instruct\.Rows correspond to Single\-Document QA, Multi\-Document QA, Summarization, and Few\-Shot Learning\. The left and right columns compareQEvictwith eviction and quantization baselines, respectively\. Dashed horizontal lines denote FullKV performance\.
## Appendix ERULER Evaluation Details
### E\.1Dataset and Evaluation Protocol
RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib36)\)provides a controlled evaluation of long\-context retrieval, aggregation, and reasoning by varying the location, multiplicity, and dependency structure of relevant evidence\. We evaluate Llama\-3\.1\-8B\-Instruct at a32K32\\mathrm\{K\}context length under a20%20\\%KV\-cache budget\. The complete results for this setting are reported in the main paper in Table[2](https://arxiv.org/html/2608.05326#S3.T2)\. This section provides additional details about the benchmark and task composition\. For RULER, we use a protected recent region of 64 tokens\.
RULER contains 13 tasks spanning four complementary capabilities, summarized in Table[7](https://arxiv.org/html/2608.05326#A5.T7)\. These task families expose complementary failure modes of KV\-cache compression\. Needle\-in\-a\-haystack tasks test whether sparse evidence remains accessible when its relevance is revealed only at query time\. Aggregation requires retaining information distributed across the sequence, while variable tracking evaluates the preservation of dependent intermediate states\. The question\-answering tasks provide a more naturalistic assessment of long\-range evidence use\.
Table 7:RULER task families used in our evaluation\.
## Appendix FGSM8K Evaluation
### F\.1Evaluation Protocol
GSM8K\(Mirzadehet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib42)\)evaluates KV\-cache compression during autoregressive multi\-step reasoning, complementing long\-context benchmarks that primarily test retrieval from the input sequence\. Successful generation requires maintaining intermediate decoding states that may become relevant again during later calculations or final\-answer construction\.
We evaluate Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib24)\), Qwen2\.5\-7B\-Instruct\(Yanget al\.[2025](https://arxiv.org/html/2608.05326#bib.bib54)\), and Mistral\-7B\-Instruct\-v0\.2\(Jianget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib3)\)across multiple measured KV\-memory budgets\. All methods use the same prompts, greedy decoding procedure, generation limits, and answer\-extraction protocol\. Performance is reported as exact\-match accuracy, and memory is measured relative to the FullKV cache\. Eviction methods are compared at matched memory budgets, while quantization methods are reported at their measured operating points following Appendix[B\.3](https://arxiv.org/html/2608.05326#A2.SS3)\. Because GSM8K sequences are relatively short, we reserve 25% of the total persistent KV\-cache budget for recent tokens\.
Fig[3](https://arxiv.org/html/2608.05326#S3.F3)in the main paper presents the accuracy–memory curves for all three models\. Tables[8](https://arxiv.org/html/2608.05326#A6.T8),[9](https://arxiv.org/html/2608.05326#A6.T9),[10](https://arxiv.org/html/2608.05326#A6.T10)report the corresponding numerical results for every evaluated operating point\.
Table 8:GSM8K performance on Llama\-3\.1\-8B\-Instruct\.The left subtable compares KV\-cache eviction methods at different KV\-cache budgets\. The right subtable compares KV\-cache quantization methods at approximately matched memory\.\(a\)KV\-cache eviction
\(b\)KV\-cache quantization
Table 9:GSM8K performance on Mistral\-7B\-Instruct\-v0\.2\.The left subtable compares KV\-cache eviction methods at different KV\-cache budgets\. The right subtable compares KV\-cache quantization methods at approximately matched memory footprints\. Full\-KV is excluded from the best and second\-best rankings\.\(a\)KV\-cache eviction
\(b\)KV\-cache quantization
Table 10:GSM8K performance on Qwen2\.5\-7B\-Instruc\.The left subtable compares KV\-cache eviction methods at different KV\-cache budgets\. The right subtable compares KV\-cache quantization methods at approximately matched memory footprints\. Full\-KV is excluded from the best and second\-best rankings\.\(a\)KV\-cache eviction
\(b\)KV\-cache quantization
## Appendix GAblation Studies
We analyse the sensitivity ofQEvictto its three principal design parameters: routing window size, quantized\-tier allocation, and recoverable precision\. All experiments use Llama\-3\.1\-8B\-Instruct under a5%5\\%KV\-cache budget\. To keep the ablation cost manageable while covering the major LongBench capabilities, we select one task from each of its four task families: NarrativeQA for single\-document question answering, MuSiQue for multi\-document question answering, QMSum for summarization, and TriviaQA for few\-shot learning\. We vary one parameter at a time while keeping the remaining settings fixed to the default configuration\(Ω=8,q=0\.70,INT2\)\(\\Omega=8,\\ q=0\.70,\\ \\mathrm\{INT2\}\)\.
### G\.1Window Granularity
We vary the routing window size overΩ∈\{4,8,16,32\}\\Omega\\in\\\{4,8,16,32\\\}, while fixingq=0\.70q=0\.70and using INT2 for the recoverable tier\. Smaller windows enable finer\-grained allocation, whereas larger windows provide smoother and more stable importance estimates at the cost of coarser selection\. This ablation measures how routing granularity affects performance across the four representative tasks\.
Table 11:Effect of routing\-window size on LongBench\.Results use Llama\-3\.1\-8B\-Instruct under a5%5\\%KV\-cache budget withq=0\.70q=0\.70and INT2 quantization\. One representative task is selected from each LongBench family\. The best and second\-best results in each column are shown in bold and underlined, respectively\. The highlighted row denotes the default configuration used in the main experiments\.
### G\.2Quantized\-Tier Allocation
We vary the fraction of the historical budget assigned to the recoverable tier over
q∈\{0\.10,0\.30,0\.50,0\.70,0\.90\},q\\in\\\{0\.10,0\.30,0\.50,0\.70,0\.90\\\},while fixingΩ=8\\Omega=8and using INT2 quantization\. Increasingqqexpands low\-bit historical coverage but reduces the capacity of the full\-precision historical tier\. This sweep isolates the trade\-off between precise retention and broader recoverable context\.
Table 12:Effect of quantized\-tier allocation on LongBench\.Results use Llama\-3\.1\-8B\-Instruct under a5%5\\%KV\-cache budget withΩ=8\\Omega=8and INT2 quantization\. One representative task is selected from each LongBench family\. The best and second\-best results in each column are shown in bold and underlined, respectively\. The highlighted row denotes the default configuration used in the main experiments\.
### G\.3Quantization Precision
We compare INT2 and INT4 recoverable representations while fixingΩ=8\\Omega=8andq=0\.70q=0\.70\. Under the same total KV\-cache budget, INT2 retains more historical windows, whereas INT4 represents fewer windows with lower quantization error\. This comparison evaluates whether broader recoverable coverage or higher per\-window precision is more beneficial\.
Table 13:Effect of recoverable\-tier precision on LongBench\.Results use Llama\-3\.1\-8B\-Instruct under a5%5\\%KV\-cache budget withΩ=8\\Omega=8andq=0\.70q=0\.70\. One representative task is selected from each LongBench family\. The best and second\-best results in each column are shown in bold and underlined, respectively\. The highlighted row denotes the default configuration used in the main experiments\.
## Appendix HEfficiency and Systems Analysis
### H\.1Measurement Protocol
All efficiency measurements use identical hardware, model precision, batch size, sequence lengths, and execution backend within each comparison\. We run warm\-up iterations before measurement to initialize kernels and memory pools, synchronize the GPU around timed regions, and reset peak\-memory statistics immediately before each run\.
We report time to first token \(TTFT\), time per output token \(TPOT\), aggregate decoding throughput, and peak GPU memory\. For a run producingNoutN\_\{\\mathrm\{out\}\}tokens, with timestampststartt\_\{\\mathrm\{start\}\},tfirstt\_\{\\mathrm\{first\}\}, andtendt\_\{\\mathrm\{end\}\}, we compute
TPOT=tend−tfirstNout−1,Throughput=Nouttend−tstart\.\\operatorname\{TPOT\}=\\frac\{t\_\{\\mathrm\{end\}\}\-t\_\{\\mathrm\{first\}\}\}\{N\_\{\\mathrm\{out\}\}\-1\},\\qquad\\operatorname\{Throughput\}=\\frac\{N\_\{\\mathrm\{out\}\}\}\{t\_\{\\mathrm\{end\}\}\-t\_\{\\mathrm\{start\}\}\}\.\(10\)
For batched inference, throughput is aggregated across all generated tokens in the batch\. Peak GPU memory includes model parameters, persistent KV\-cache storage, attention workspaces, and temporary routing buffers\. The complete end\-to\-end results are reported in Table[4](https://arxiv.org/html/2608.05326#S5.T4)of the main paper\.
### H\.2Runtime Components
Compared with FullKV,QEvictintroduces periodic score aggregation, window ranking, tier assignment, and cache migration\. First\-time demotion also creates the persistent low\-bit representation, which is reused during subsequent tier transitions\. Since routing occurs everyΩ\\Omegagenerated tokens, its amortized per\-token cost is
Troute/token=Tscore\+Trank\+TmigrateΩ\.T\_\{\\mathrm\{route/token\}\}=\\frac\{T\_\{\\mathrm\{score\}\}\+T\_\{\\mathrm\{rank\}\}\+T\_\{\\mathrm\{migrate\}\}\}\{\\Omega\}\.\(11\)
At each decoding step, active quantized windows are dequantized and assembled with the full\-precision cache before attention\. Routing events additionally materialize attention probabilities for cumulative scoring\. These operations explain the backend\-dependent behaviour observed in the main paper: reducing the attended full\-precision cache benefits eager execution, while the current FlashAttention\-2 path incurs overhead from dequantization, cache reconstruction, and selective score materialization\. Fusing these operations with mixed\-precision attention remains the primary systems optimization opportunity\.
### H\.3Hardware Settings
All experiments were run on a node with 8 NVIDIA A100 80GB GPUs, with a subset additionally run on 1 NVIDIA GB10 128GB GPU\. Larger evaluation sweeps were parallelized by sharding the dataset round\-robin across GPUs, with one full model copy per GPU\. The software stack was PyTorch 2\.6\.0 \(CUDA 12\.4\), Transformers 4\.47\.1 and flash\-attn 2\.8\.3\.
## Appendix IAdditional Observation Details
This section provides the diagnostic protocol, formal metric definitions, and supporting results for the observations in Section[3](https://arxiv.org/html/2608.05326#S3)\.
### I\.1Diagnostic Protocol
We collect reference attention traces from an uncompressed FullKV execution because attention assigned to a discarded state cannot be recovered from the compressed execution itself\. Unless stated otherwise, diagnostics are computed independently for each layer and attention head and then aggregated across eligible routing events and traces\.
The analysis uses Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib24)\)with a 512\-token prefill, 256 generated tokens, a measured20%20\\%KV\-cache budget, and five protected sink tokens\. We evaluate two input articles across all 32 transformer layers, yielding 64 layer\-level traces\. Routing begins at decoding step zero and is repeated everyΩ\\Omegagenerated tokens\. We consider the following conditions:
- •R0: uncompressed FullKV reference;
- •R1: token\-level eviction withΩ=1\\Omega=1;
- •R2: two\-tier window eviction withΩ=8\\Omega=8;
- •R3: three\-tier routing withΩ=8\\Omega=8,Kf=7K\_\{f\}=7, andKq=28K\_\{q\}=28;
- •R4: three\-tier routing withΩ=32\\Omega=32,Kf=1K\_\{f\}=1, andKq=11K\_\{q\}=11\.
R2 and R3 use the same window size and measured byte budget, isolating the effect of replacing part of the full\-precision allocation with a recoverable INT2 tier\. R1, R3, and R4 are complete operating points with different routing granularities\.
LetAtℓ,h\(i\)A^\{\\ell,h\}\_\{t\}\(i\)denote the FullKV attention probability assigned at decoding steptt, layerℓ\\ell, and headhhto historical tokenii\. The cumulative reference score of windowwwis
Stℓ,h\(w\)=∑τ≤t∑i∈wAτℓ,h\(i\)\.S^\{\\ell,h\}\_\{t\}\(w\)=\\sum\_\{\\tau\\leq t\}\\sum\_\{i\\in w\}A^\{\\ell,h\}\_\{\\tau\}\(i\)\.\(12\)
### I\.2Observation I:Future Missed Massand Selection Churn
#### Future Missed Mass\(FMM\)\.
At routing eventrr, letℰrℓ,h\\mathcal\{E\}^\{\\ell,h\}\_\{r\}denote the historical positions made inaccessible by the evaluated policy\. We define FMM over a future horizonHHas
FMMHℓ,h\(r\)=∑τ=r\+1min\(r\+H,T\)∑i∈ℰrℓ,hi≤rAτℓ,h\(i\)∑τ=r\+1min\(r\+H,T\)∑i≤rAτℓ,h\(i\)\.\\mathrm\{FMM\}^\{\\ell,h\}\_\{H\}\(r\)=\\frac\{\\displaystyle\\sum\_\{\\tau=r\+1\}^\{\\min\(r\+H,T\)\}\\sum\_\{\\begin\{subarray\}\{c\}i\\in\\mathcal\{E\}^\{\\ell,h\}\_\{r\}\\\\ i\\leq r\\end\{subarray\}\}A^\{\\ell,h\}\_\{\\tau\}\(i\)\}\{\\displaystyle\\sum\_\{\\tau=r\+1\}^\{\\min\(r\+H,T\)\}\\sum\_\{i\\leq r\}A^\{\\ell,h\}\_\{\\tau\}\(i\)\}\.\(13\)
Only states that already existed at routing eventrrare included\. We useH=32H=32\. Lower FMM indicates that the routing decision permanently discards less information required by subsequent queries\.
#### Selection Churn\.
Letℛrℓ,h\\mathcal\{R\}^\{\\ell,h\}\_\{r\}denote the historical positions retained after routing eventrr\. Window selections are expanded to token positions before comparison\. Selection Churn is the Jaccard distance between retained sets at consecutive routing events:
Churnℓ,h\(r\)=1−\|ℛrℓ,h∩ℛr\+1ℓ,h\|\|ℛrℓ,h∪ℛr\+1ℓ,h\|\.\\mathrm\{Churn\}^\{\\ell,h\}\(r\)=1\-\\frac\{\\left\|\\mathcal\{R\}^\{\\ell,h\}\_\{r\}\\cap\\mathcal\{R\}^\{\\ell,h\}\_\{r\+1\}\\right\|\}\{\\left\|\\mathcal\{R\}^\{\\ell,h\}\_\{r\}\\cup\\mathcal\{R\}^\{\\ell,h\}\_\{r\+1\}\\right\|\}\.\(14\)
The protected recent region is excluded because its deterministic movement reflects the sliding local policy rather than instability in historical selection\.
Table 14:Future Missed Mass and historical\-set churn\.FMM is averaged over eligible routing events withH=32H=32\. Churn values match those reported in Section[3](https://arxiv.org/html/2608.05326#S3)\.Table[14](https://arxiv.org/html/2608.05326#A9.T14)shows that R3 reduces average FMM relative to token\-level R1 while lowering historical\-set churn by more than an order of magnitude\. R4 also reduces FMM relative to R1, but its coarser routing granularity does not improve churn in this measurement\. Window aggregation is therefore beneficial, although excessively large windows can sacrifice allocation granularity and need not yield more stable assignments\.
The R3 full\-precision\-only ablation treats quantized windows as inaccessible without reallocating their bytes\. Its substantially higher FMM isolates the future attention preserved by the recoverable tier\. This is a mechanistic ablation rather than an iso\-memory comparison\.
### I\.3Observation II: Tier\-Mass Distribution and Quantized\-Score Agreement
For tierz∈\{Full,Quantized,Evicted,Local\}z\\in\\\{\\textsc\{Full\},\\textsc\{Quantized\},\\textsc\{Evicted\},\\textsc\{Local\}\\\}, we define its cumulative attention\-mass share as
μt\(z\)=∑w:zt\(w\)=zSt\(w\)∑w∈𝒲tSt\(w\)\.\\mu\_\{t\}\(z\)=\\frac\{\\displaystyle\\sum\_\{w:z\_\{t\}\(w\)=z\}S\_\{t\}\(w\)\}\{\\displaystyle\\sum\_\{w\\in\\mathcal\{W\}\_\{t\}\}S\_\{t\}\(w\)\}\.\(15\)
Under the byte\-matched R2–R3 comparison, R2 assigns33\.3%33\.3\\%of the reference attention mass to full\-precision windows and64\.8%64\.8\\%to evicted windows\. R3 assigns19\.7%19\.7\\%to full precision and preserves another42\.6%42\.6\\%in INT2, reducing the evicted share to35\.7%35\.7\\%\. The protected local region contributes1\.9%1\.9\\%in both conditions\. The recoverable tier therefore reduces the reference attention mass assigned to permanent eviction by29\.129\.1percentage points\.
To determine whether quantized windows remain suitable for subsequent ranking, let𝐬rQ\\mathbf\{s\}^\{Q\}\_\{r\}denote their scores under INT2 execution and𝐬rFull\\mathbf\{s\}^\{\\mathrm\{Full\}\}\_\{r\}the corresponding FullKV scores\. We define Quantized\-Score Agreement as
QSA\(r\)=⟨𝐬rQ,𝐬rFull⟩‖𝐬rQ‖2‖𝐬rFull‖2\.\\mathrm\{QSA\}\(r\)=\\frac\{\\left\\langle\\mathbf\{s\}^\{Q\}\_\{r\},\\mathbf\{s\}^\{\\mathrm\{Full\}\}\_\{r\}\\right\\rangle\}\{\\left\\\|\\mathbf\{s\}^\{Q\}\_\{r\}\\right\\\|\_\{2\}\\left\\\|\\mathbf\{s\}^\{\\mathrm\{Full\}\}\_\{r\}\\right\\\|\_\{2\}\}\.\(16\)
We additionally measure the ratio between the total attention mass assigned to these windows under quantized and full\-precision execution:
RQ\(r\)=∑w∈𝒬rSrQ\(w\)∑w∈𝒬rSrFull\(w\)\.R\_\{Q\}\(r\)=\\frac\{\\displaystyle\\sum\_\{w\\in\\mathcal\{Q\}\_\{r\}\}S^\{Q\}\_\{r\}\(w\)\}\{\\displaystyle\\sum\_\{w\\in\\mathcal\{Q\}\_\{r\}\}S^\{\\mathrm\{Full\}\}\_\{r\}\(w\)\}\.\(17\)
Table 15:Agreement between INT2 and FullKV scores over R3 quantized windows\.The high cosine agreement in Table[15](https://arxiv.org/html/2608.05326#A9.T15)shows that INT2 largely preserves the relative ordering of quantized windows\. The lower mass ratio indicates attenuation in attention magnitude, so low\-bit execution is not numerically identical to FullKV\. Nevertheless, the ranking signal remains sufficiently stable for continued scoring and subsequent promotion\.
### I\.4Observation III:Global LIR
LetXr,w∈\{0,1\}X\_\{r,w\}\\in\\\{0,1\\\}indicate whether windowwwbelongs to a selected full\-precision set at routing eventrr\. We analyse two selections:
- •Oracle: the top\-KfK\_\{f\}windows according to the R0 FullKV ranking;
- •Policy FP: the full\-precision tier selected by R3 or R4\.
An inactive episode is a maximal sequence of routing events for whichXr,w=0X\_\{r,w\}=0\. Aftermmconsecutive inactive events, the episode becomes eligible\. It is rescued if the window re\-enters the selected set at any later routing event before generation terminates\. We define
GlobalLIR\(m\)=\#\{eligible inactive episodes that later re\-enter\}\#\{eligible inactive episodes\}\.\\mathrm\{GlobalLIR\}\(m\)=\\frac\{\\\#\\\{\\text\{eligible inactive episodes that later re\-enter\}\\\}\}\{\\\#\\\{\\text\{eligible inactive episodes\}\\\}\}\.\(18\)
Each maximal inactive episode contributes once\. Episodes that do not return before generation terminates remain in the denominator and are counted as not rescued\. We usem=3m=3by default and verify the same trend form∈\{1,2,4,8\}m\\in\\\{1,2,4,8\\\}\.
Table 16:Global LIRfor oracle and policy full\-precision selections\.Confidence intervals are95%95\\%\. Median time\-to\-return is measured in routing events\.The low oracle LIR in Table[16](https://arxiv.org/html/2608.05326#A9.T16)confirms that the true top\-ranked region is largely persistent\. Under R3, however, the deployed full\-precision tier exhibits a6\.18%6\.18\\%return rate, approximately6\.3×6\.3\\timesthe oracle rate\. Approximate scoring and constrained allocation therefore produce substantially greater re\-entry pressure than the oracle ranking alone suggests\.
The median policy return times are eight routing events for R3 and two for R4\. Given routing intervals of eight and 32 tokens, respectively, both correspond to approximately 64 decoding steps\. Relevance revival can therefore occur well beyond the immediately following routing event\.
We complementGlobal LIRwith the lagged transition probability
Pab\(Δ\)=Pr\[Xr\+Δ,w=b∣Xr,w=a\],a,b∈\{0,1\}\.P^\{\(\\Delta\)\}\_\{ab\}=\\Pr\\left\[X\_\{r\+\\Delta,w\}=b\\mid X\_\{r,w\}=a\\right\],\\qquad a,b\\in\\\{0,1\\\}\.\(19\)
For the R3 policy,P01\(Δ\)P^\{\(\\Delta\)\}\_\{01\}increases from0\.31%0\.31\\%atΔ=1\\Delta=1to2\.39%2\.39\\%atΔ=8\\Delta=8, whileP10\(Δ\)P^\{\(\\Delta\)\}\_\{10\}increases from3\.05%3\.05\\%to22\.12%22\.12\\%\. AtΔ=8\\Delta=8, departures from the full\-precision tier are approximately9\.3×9\.3\\timesmore frequent than returns\.
These dynamics are asymmetric\. Most oracle\-important windows remain active, many policy\-selected windows subsequently decline, and a smaller but systematic subset eventually returns\. This behaviour motivates frequent demotion together with recoverable low\-bit storage that preserves the possibility of later promotion\.
## Appendix JCompatibility with Alternative Ranking Functions
The three\-tier hierarchy inQEvictrequires only an ordering over candidate windows and is therefore not tied to cumulative attention as its importance estimator\. To evaluate this modularity, we replace the default ranking function with three representative alternatives while leaving the cache hierarchy, byte allocation, and migration policy unchanged\.
All experiments use Llama\-3\.1\-8B\-Instruct on LongBench under a5%5\\%KV\-cache budget withΩ=8\\Omega=8,q=0\.70q=0\.70, and an INT2 recoverable tier\. The evaluation covers the 12 tasks used in the main study together with PassageCount and PassageRetrieval\-en\. We compare the following scoring functions:
- •Cumulative attention, the default estimator used byQEvict\.
- •Exponentially decayed attention, which discounts older observations and places greater weight on recent queries\. This variant is motivated by historical\-attention and persistence\-based cache selection\(Zhanget al\.[2023](https://arxiv.org/html/2608.05326#bib.bib8); Liuet al\.[2023](https://arxiv.org/html/2608.05326#bib.bib9)\)\.
- •Attentionpp\-norm, which replaces additive aggregation with a peak\-sensitivepp\-norm over observed window scores\. We usep=3p=3\. This is a power\-norm generalization of the attention\-based ranking signals used by method such SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib10)\)\.
- •Key\-vector norm, an attention\-free estimator based on theL2L\_\{2\}\-norm of cached keys\. FollowingDevotoet al\.\([2024](https://arxiv.org/html/2608.05326#bib.bib53)\), lower key norms receive higher retention priority\.
Letutℓ\(w\)u\_\{t\}^\{\\ell\}\(w\)denote the attention received by windowwwsince the previous routing event:
utℓ\(w\)=1Hq∑h=1Hq∑τ=t−Ω\+1t∑i∈waτℓ,h\(i\)\.u\_\{t\}^\{\\ell\}\(w\)=\\frac\{1\}\{H\_\{q\}\}\\sum\_\{h=1\}^\{H\_\{q\}\}\\sum\_\{\\tau=t\-\\Omega\+1\}^\{t\}\\sum\_\{i\\in w\}a\_\{\\tau\}^\{\\ell,h\}\(i\)\.\(20\)
The cumulative and exponentially decayed scores are
St,cumℓ\(w\)\\displaystyle S\_\{t,\\mathrm\{cum\}\}^\{\\ell\}\(w\)=St−Ω,cumℓ\(w\)\+utℓ\(w\),\\displaystyle=S\_\{t\-\\Omega,\\mathrm\{cum\}\}^\{\\ell\}\(w\)\+u\_\{t\}^\{\\ell\}\(w\),\(21\)St,decayℓ\(w\)\\displaystyle S\_\{t,\\mathrm\{decay\}\}^\{\\ell\}\(w\)=λSt−Ω,decayℓ\(w\)\+utℓ\(w\),λ=0\.99\.\\displaystyle=\\lambda S\_\{t\-\\Omega,\\mathrm\{decay\}\}^\{\\ell\}\(w\)\+u\_\{t\}^\{\\ell\}\(w\),\\qquad\\lambda=0\.99\.\(22\)
For thepp\-norm variant, we aggregate the event\-level window scores as
St,pℓ\(w\)=\(∑r≤t\[urℓ\(w\)\]p\)1/p,p=3\.S\_\{t,p\}^\{\\ell\}\(w\)=\\left\(\\sum\_\{r\\leq t\}\\left\[u\_\{r\}^\{\\ell\}\(w\)\\right\]^\{p\}\\right\)^\{1/p\},\\qquad p=3\.\(23\)
The key\-norm estimator is defined as
Skeyℓ\(w\)=−1\|w\|∑i∈w‖𝐤iℓ‖2,S\_\{\\mathrm\{key\}\}^\{\\ell\}\(w\)=\-\\frac\{1\}\{\|w\|\}\\sum\_\{i\\in w\}\\left\\\|\\mathbf\{k\}\_\{i\}^\{\\ell\}\\right\\\|\_\{2\},\(24\)
where the negative sign converts the low\-norm preference into the higher\-is\-better convention used by the routing algorithm\.
Table 17:Compatibility ofQEvictwith alternative window\-ranking functions on LongBench\.All results use Llama\-3\.1\-8B\-Instruct under a5%5\\%KV\-cache budget withΩ=8\\Omega=8,q=0\.70q=0\.70, and INT2 quantization\. Only the ranking function is changed\. The average is computed over all 14 tasks\. Best and second\-best distinct results are shown in bold and underlined, respectively\.
## Appendix KEffect of Q→\\rightarrowF Residency Promotion
#### Objective\.
We examine whether allowing quantized windows to return to the full execution tier improves the behavior of the R3 three\-tier policy\. We compare the one\-way policy
Full→Quantized→Evicted\\textsc\{Full\}\\rightarrow\\textsc\{Quantized\}\\rightarrow\\textsc\{Evicted\}against the bidirectional policy
Full↔Quantized→Evicted\.\\textsc\{Full\}\\leftrightarrow\\textsc\{Quantized\}\\rightarrow\\textsc\{Evicted\}\.A promoted window is reconstructed from its persistent INT2 representation\. Promotion therefore restores full\-tier residency and execution handling rather than the original pre\-quantization KV values\.
#### Experimental setting\.
We use Llama\-3\.1\-8B\-Instruct\(Dubeyet al\.[2024](https://arxiv.org/html/2608.05326#bib.bib24)\)with a 256\-token prefill, 1024 decoding tokens, a measured 20% persistent KV\-cache budget, five protected sink tokens, and the R3 three\-tier configuration with window sizeΩ=8\\Omega=8\. The two variants use the same tier capacities, cumulative\-attention scoring rule, routing schedule, quantization configuration, and memory budget\. They differ only in whether a window already assigned to the quantized tier may later be promoted to the full execution tier\. Under the no\-promotion variant, quantized windows may subsequently be evicted but cannot return to the full tier\.
The current pilot conditions use different source articles\. The comparison should therefore be interpreted as directional evidence of the behavior enabled by promotion rather than as a paired estimate of its exact causal effect\.
#### Metrics\.
We reuse the diagnostics introduced in Appendix[I](https://arxiv.org/html/2608.05326#A9): Future Missed Mass \(FMM\), Quantized\-Score Agreement \(QSA\), the attention\-mass ratioRQR\_\{Q\}, and Global LIR\. We additionally report the number of observed Q→\\rightarrowF transitions as an auxiliary measure of promotion activity\.
For this analysis, FMM measures the fraction of future FullKV attention assigned to historical states that have become inaccessible under R3\. Lower FMM indicates that fewer future\-relevant states are permanently discarded\.
QSA measures the cosine agreement between the score vector of the windows currently assigned to the quantized tier and their corresponding FullKV reference scores:
QSA\(r\)=⟨𝐬rQ,𝐬rFull⟩‖𝐬rQ‖2‖𝐬rFull‖2\.\\operatorname\{QSA\}\(r\)=\\frac\{\\left\\langle\\mathbf\{s\}^\{Q\}\_\{r\},\\mathbf\{s\}^\{\\mathrm\{Full\}\}\_\{r\}\\right\\rangle\}\{\\left\\lVert\\mathbf\{s\}^\{Q\}\_\{r\}\\right\\rVert\_\{2\}\\left\\lVert\\mathbf\{s\}^\{\\mathrm\{Full\}\}\_\{r\}\\right\\rVert\_\{2\}\}\.
The attention\-mass ratio measures the fraction of the FullKV reference attention mass preserved over the windows currently assigned to the quantized tier:
RQ\(r\)=∑w∈QrSrQ\(w\)∑w∈QrSrFull\(w\)\.R\_\{Q\}\(r\)=\\frac\{\\sum\_\{w\\in Q\_\{r\}\}S^\{Q\}\_\{r\}\(w\)\}\{\\sum\_\{w\\in Q\_\{r\}\}S^\{\\mathrm\{Full\}\}\_\{r\}\(w\)\}\.We reportRQR\_\{Q\}as a percentage\. A value of100%100\\%indicates that quantized execution preserves the total FullKV attention mass over the current Q\-tier membership\.
Global LIR measures the percentage of eligible inactive episodes that later re\-enter the selected full\-tier set\. The transition count records the total number of Q→\\rightarrowF movements observed during decoding\.
Table 18:Effect of Q→\\rightarrowF residency promotion for the R3 three\-tier policy with window sizeΩ=8\\Omega=8\. QSA andRQR\_\{Q\}are computed over the windows currently assigned to the quantized tier\. Lower is better for FMM; higher is better for the remaining metrics\.MetricNo promotionWith promotionΔ\\DeltaR3 FMM↓\\downarrow16\.8%16\.1%−0\.7\-0\.7ppQuantized\-Score Agreement↑\\uparrow0\.9470\.985\+0\.038\+0\.038FullKV attention mass preserved,RQR\_\{Q\}↑\\uparrow88\.7%95\.5%\+6\.8\+6\.8ppGlobal LIR↑\\uparrow0\.0%6\.6%\+6\.6\+6\.6ppRecorded Q→\\rightarrowF transitions0639\+639\+639
#### Observation 1: Promotion is associated with lower future missed mass\.
R3 FMM decreases from 16\.8% under one\-way demotion to 16\.1% when Q→\\rightarrowF promotion is enabled\. This directional reduction suggests that bidirectional routing can reduce the premature permanent removal of windows whose importance later re\-emerges\.
#### Observation 2: Promotion improves policy\-level Q\-tier score agreement\.
QSA increases from 0\.947 to 0\.985, showing that the score vectors of the windows currently remaining in the quantized tier align more closely with their FullKV reference scores under the promotion\-enabled policy\.
#### Observation 3: Promotion preserves more FullKV attention mass in the Q tier\.
The attention\-mass ratioRQR\_\{Q\}increases from 88\.7% to 95\.5%, a gain of 6\.8 percentage points\. Thus, the current Q\-tier population under bidirectional routing preserves a larger fraction of its FullKV reference attention mass and moves closer to the ideal value of 100%\.
Because promotion changes which windows remain in the quantized tier, the improvements in QSA andRQR\_\{Q\}should be interpreted as evidence of better tier composition and policy\-level score agreement\. They do not imply that promotion reverses the quantization error of a fixed window\.
#### Observation 4: Promotion is actively exercised\.
Global LIR increases from 0% under the one\-way policy to 6\.6% when promotion is enabled\. The zero value without promotion holds by construction, whereas the nonzero value confirms that previously inactive windows re\-enter the full\-tier selection under bidirectional routing\.
#### Observation 5: Re\-entry is frequent rather than purely hypothetical\.
The promotion\-enabled run records 639 Q→\\rightarrowF transitions, compared with none under one\-way demotion\. This shows that promotion is an active component of the routing policy rather than only a permitted but unused state transition\.
#### Overall observation\.
Taken together, the lower FMM, higher QSA andRQR\_\{Q\}, and nonzero re\-entry activity provide consistent directional evidence that Q→\\rightarrowF promotion improves the dynamic allocation of historical windows across the full and quantized tiers\.相似文章
ReST-KV：基于逐层输出重构与时空平滑的鲁棒 KV Cache 驱逐方法
本文介绍了 ReST-KV，一种用于大型语言模型的新型鲁棒 KV Cache 驱逐方法。该方法利用逐层输出重构与时空平滑技术来提升效率，显著降低了解码延迟，并在 LongBench 和 RULER 等长上下文基准测试中超越了现有的最先进基线模型。
基于顿悟感知的KV缓存淘汰方法（无需注意力矩阵）
本文介绍了EpiKV，一种基于内部表征变化（顿悟分数）而非注意力权重来评估token重要性的KV缓存淘汰方法，无需具体化注意力矩阵。该方法在推理基准测试中取得了具有竞争力的性能，同时支持长达16倍的上下文长度。
KV Packet: 免重计算的上下文无关KV缓存用于大语言模型
KV Packet 提出了一种免重计算的缓存复用框架，用于大语言模型。该框架使用可训练的软标记适配器来弥合上下文不连续性，消除了开销，同时在 Llama-3.1 和 Qwen2.5 上的性能与完全重计算基线相当。
CONF-KV: 置信度感知的KV缓存淘汰与混合精度存储用于长视界大语言模型
CONF-KV 是一种KV缓存管理系统，利用模型不确定性动态调整缓存保留策略，从而提升长上下文大语言模型推理的内存效率，同时将困惑度控制在1.5-2.1个点以内。
CompressKV：语义检索引导的KV缓存压缩方法，用于资源高效的长上下文大语言模型推理
CompressKV针对基于GQA的大语言模型，提出了一种语义检索引导的KV缓存压缩方法，通过识别语义检索头来保留关键令牌。在LongBench任务中，仅使用3%的KV缓存即可实现超过97%的全缓存性能。