What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation

# What Matters for Aggressive Decoding-Time KV Eviction?Temporal Aggregation and Ranking Preservation
Source: [https://arxiv.org/html/2609.03515](https://arxiv.org/html/2609.03515)
Yu ZhaoAffiliation:Alibaba GroupEmail:[hanfeng\.wxt@alibaba\-inc\.comhttps://github\.com/BobTsang\-NLP/InertiaKV](mailto:[email protected]://github.com/BobTsang-NLP/InertiaKV)Yefeng LiuAffiliation:Ant GroupZhihong LuAffiliation:Ant GroupXuanfan NiAffiliation:Alibaba GroupXintong Wang††thanks:Corresponding author xintong\.wang@uni\-hamburg\.de\.Affiliation:Alibaba GroupAffiliation:Universität Hamburg

###### Abstract

Decoding\-time KV cache compression research focuses heavily on designing better token scoring functions, while the temporal rule that aggregates scores across decode steps is often treated as an implementation detail\. Under aggressive KV compression, we find that exponential\-moving\-average \(EMA\) aggregation makes approximately order\-preserving scorer modifications largely indistinguishable at the eviction\-set level\. Value\-norm and entropy variants remain highly correlated with attention and produce nearly unchanged retention sets, whereas KeyDiff, key norm, recency, and a learned scorer alter the ranking and degrade substantially\. We associate this stability with the evaluated aggregation, which couples layer weighting and temporal retention\. Building on this observation, we introduceInertiaKV, an EMA\-based decoding\-time eviction method, andInertiaKV\-Lazy, its periodic\-refresh variant, which yields 1\.34–1\.46×\\timesdecode throughput relative to full refresh InertiaKV\. We also study Score\-Free decoding as a separate empirical operating point: it scores the full context once at the first decode step, freezes that ranking, and incurs an average quality change of\+0\.03\+0\.03while removing all subsequent scoring\. Across six open\-weight backbones and the LongBench, LongBench\-v2, and RULER benchmarks, the results identify temporal aggregation and ranking preservation as distinct, consequential design factors; they do not imply that scoring quality is irrelevant in general\.

## 1Introduction

Decoding\-time KV cache compression\([Zhang et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib27);[Liu et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib16);[Oren et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib19)\)is a sequential decision problem: at each generation step, the compressor must decide which tokens to retain and which to permanently evict under a fixed memory budget\. This decision decomposes into two commonly discussed design axes: \(1\) a*per\-step scoring function*that estimates each token’s current importance, and \(2\) an*aggregation rule*that combines scores across decoder layers and decode steps into the running utility estimate that drives eviction\.

Existing methods focus primarily on the first axis: H2O\([Zhang et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib27)\)accumulates attention mass, TOVA\([Oren et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib19)\)ranks by last\-query attention, ScissorHands\([Liu et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib16)\)exploits attention persistence patterns, AdaKV\([Feng et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib8)\)adapts budgets across heads, FastGen\([Ge et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib9)\)uses model\-aware adaptive compression, KeyDiff\([Park et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib20)\)scores by key geometry\. Prefill\-time methods such as SnapKV\([Li et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib15)\)select tokens before generation begins; orthogonal techniques reduce per\-token cost via quantization\([Liu et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib17)\)or cross\-layer sharing\([Brandon et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib5)\)\. The aggregation axis \(single\-step, cumulative sum, or bounded\-memory smoothing\) is treated as an implementation detail rather than a first\-class design choice\.

At moderate compression rates, existing methods generally maintain acceptable quality\. Under aggressive compression, however, this no longer holds: on RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib10)\), as the KV budget shrinks from 50% to 10%, TOVA, SnapKV, and AdaKV all degrade substantially, albeit at different rates, despite employing fundamentally different scoring functions \(Table[1](https://arxiv.org/html/2609.03515#S1.T1)\)\. This shared degradation across diverse scorers suggests that scoring alone may not determine quality in this regime\. Prior work has rarely isolated scoring from temporal aggregation; they are typically evaluated as a coupled system, leaving open which axis governs quality at such aggressive budgets\.

Table 1:RULER score on Llama\-3\.1\-8B as KV budget decreases\.Methods with different scoring functions exhibit different degradation rates as the budget shrinks\.This motivates a shift in focus toward the neglected second axis: score aggregation\. We adopt exponential moving average \(EMA\) smoothing, which accumulates scores during decoding with bounded memory and a tunable coefficientα\\alpha\. In the evaluated implementation, the running state is updated sequentially as scores are obtained from the decoder layers and is retained across decode steps, thereby coupling layer weighting with temporal retention\. To isolate scorer variation under this fixed aggregation procedure, we conduct an initial experiment on Llama\-3\.1\-8B at 90% compression: we fix the sequential layer–temporal aggregation rule and systematically vary the scoring function\. Replacing the attention\-based scorer with value\-norm or entropy weighting produces near\-identical eviction sets \(Figure[1](https://arxiv.org/html/2609.03515#S1.F1); mean pairwise Jaccard=0\.97=0\.97\)\. After the running ranking stabilizes, approximately order\-preserving scorer modifications rarely change keep/evict membership\.

To assess the generality of this finding, we extend the experiment to Qwen2\.5\-7B and Llama\-3\.3\-70B on LongBench\([Bai et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib2)\), LongBench\-v2\([Bai et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib3)\), and RULER\. The pattern is consistent: perturbations that approximately preserve the utility ranking produce similar retention sets under the evaluated aggregation, while scorers that substantially reorder it degrade quality\. Under these aggressive\-compression settings, the evaluated sequential layer–temporal aggregation makes approximately order\-preserving scorer modifications largely indistinguishable at the eviction\-set level; this conclusion does not extend to scorers that substantially alter the utility ranking\.

We associate this with*ranking inertia*under the evaluated sequential layer–temporal aggregation: repeated updates are associated with more stable retention\-set membership \(§[5\.3](https://arxiv.org/html/2609.03515#S5.SS3)\), but the present experiments do not separate cross\-step memory from the recurrence’s implicit later\-layer weighting\. This stability comes at a cost: stronger smoothing delays adaptation to relevance shifts, producing an observed*boundary\-error vs\. lag\-error*tradeoff analyzed in §[3\.1](https://arxiv.org/html/2609.03515#S3.SS1)\. This persistence motivates InertiaKV\-Lazy, which periodically refreshes scores and yields1\.341\.34–1\.46×1\.46\\timesdecode throughput relative to full\-refresh InertiaKV\. As a separate operating point, Score\-Free computes scores across the decoder layers at the first decode step and freezes the resulting ranking; in our evaluation it eliminates all subsequent score updates with negligible average quality change \(Δ=\+0\.03\\Delta\{=\}\{\+\}0\.03points\)\. In summary, this paper makes three contributions:

- •We identify ranking inertia in sequential layer\-temporal aggregation across 7B–70B models on LongBench, LongBench\-v2, and RULER: approximately order\-preserving scorer changes leave the eviction set nearly unchanged, whereas ranking\-altering scorers can sharply degrade quality\.
- •We characterize the observed boundary\-error vs\. lag\-error tradeoff and provide a rank\-stability bound, explaining why stronger temporal smoothing can stabilize eviction decisions yet delay adaptation to relevance shifts\. Our negative result further shows that local noise and drift alone cannot reliably guide adaptive smoothing\.
- •We translate decode\-time ranking persistence into operating points: InertiaKV\-Lazy yields1\.341\.34–1\.46×1\.46\\timesdecode speedup over full\-refresh InertiaKV, while Score\-Free eliminates all subsequent score updates after first\-step initialization with negligible average quality change\. These results establish refresh frequency as explicit quality–efficiency control\.

Figure 1:EMA stabilizes the retention sets of three highly rank\-correlated scorers\.Each panel pairs per\-step scoring \(*left*\) with EMA\-smoothed scoring \(*right*\) for Attention, Value Norm, and Entropy\. Per\-step scores fluctuate, while EMA yields near\-identical retention sets for these approximately order\-preserving variants \(mean pairwise Jaccard=0\.97=0\.97over top\-BBtoken sets, averaged across decode steps and layers\)\. Navy cells indicate retained tokens; columns are sorted by reference utility \(high→\\tolow\); the dashed line marks the budget boundary\.
## 2Related Work

#### Decode\-time eviction methods\.

Decode\-time KV eviction has evolved rapidly around the design of better scoring functions\. H2O\([Zhang et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib27)\)introduced cumulative attention mass as a heavy\-hitter signal, establishing the template of score\-and\-evict under a fixed budget\. ScissorHands\([Liu et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib16)\)then observed that attention patterns persist across decode steps, enabling future importance to be predicted from past attention\. TOVA\([Oren et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib19)\)simplified the scoring to a single\-step last\-query signal, trading temporal information for lower overhead\. Subsequent work enriched the scoring axis further: AdaKV\([Feng et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib8)\)adapts budgets per attention head, KeyDiff\([Park et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib20)\)replaces attention weights with key\-vector geometry, and TaDA\([Joshi et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib11)\)adds mean\-centering to adaptive compression\. Recent work such as RocketKV\([Behnam et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib4)\)applies two\-stage coarse\-to\-fine scoring for acceleration, and EvolKV\([Yu and Chai, 2025](https://arxiv.org/html/2609.03515#bib.bib26)\)uses evolutionary search to optimize layer\-wise budget allocation\. Throughout this progression, the temporal rule that aggregates per\-step scores into eviction decisions \(cumulative sum, sliding window, or implicit single\-step\) has remained a background implementation choice rather than an object of study in its own right\.

#### Temporal perspectives\.

Few works touch on the temporal dimension of eviction\. ScissorHands’ persistence hypothesis implies temporal stability but frames it as a property of the attention signal rather than of the aggregation mechanism\. FAEDKV\([Li et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib14)\)identifies recency bias in existing methods and proposes frequency\-domain transforms for unbiased scoring across an infinite window—among the first efforts to treat temporal dynamics explicitly as a design problem\. However, FAEDKV contributes a debiasing technique that improves the scoring signal; it does not decouple scoring from aggregation or test how temporal smoothing interacts with scorer ranking\. Our finding is complementary: Under aggressive KV compression, we find that exponential\-moving\-average \(EMA\) aggregation makes approximately order\-preserving scorer modifications largely indistinguishable at the eviction\-set level, but does not rescue scorers whose rankings differ substantially from attention\.

#### Prefill\-time and sparse\-attention methods\.

SnapKV\([Li et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib15)\)selects tokens during prefill; StreamingLLM\([Xiao et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib25)\)retains attention sinks and a local window\. These operate before or outside the decode loop\. Sparse\-attention systems\([Tang et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib23);[Lee et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib13);[Sun et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib22)\)maintain the full cache off\-chip and load subsets per step—a fundamentally different regime from irreversible eviction\.

#### Orthogonal dimensions\.

Cache size can also be reduced through quantization\([Liu et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib17)\), cross\-layer sharing\([Brandon et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib5)\), layer\-wise budget shaping\([Cai et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib6)\), or reconstruction\-based compression\([Kim et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib12)\)\. Hardware\-efficient attention implementations\([Dao, 2024](https://arxiv.org/html/2609.03515#bib.bib7)\)reduce per\-step cost but do not address cache growth\. These are composable with eviction\-based methods but orthogonal to the question studied here\. To our knowledge, prior work rarely isolates scoring from temporal aggregation as independently controllable design axes\.

## 3Method

We study decode\-time KV cache compression under aggressive budgets\. Given a prefill cache of lengthL0L\_\{0\}and compression ratioρ\\rho, the decode\-side budget isB=⌊\(1−ρ\)​L0⌋B=\\lfloor\(1\{\-\}\\rho\)L\_\{0\}\\rfloor\. Unlike sparse attention, eviction is irreversible: removed KV pairs cannot be recovered\.

### 3\.1Temporal Utility Aggregation

At each decode steptt, we extract the last\-query attention weights\([Vaswani et al\., 2017](https://arxiv.org/html/2609.03515#bib.bib24)\)from allHHheads and compute a per\-token importance score by mean\-pooling across heads:

st\(i\)=1H∑h=1Hat\(h\)\(i\),i=1,…,Lt,s\_\{t\}\(i\)=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}a\_\{t\}^\{\(h\)\}\(i\),\\quad i=1,\\ldots,L\_\{t\},\(1\)whereat\(h\)​\(i\)a\_\{t\}^\{\(h\)\}\(i\)is headhh’s attention to cached positionii\. We maintain a utility vectormt∈ℝLtm\_\{t\}\\in\\mathbb\{R\}^\{L\_\{t\}\}, wheremt​\(i\)m\_\{t\}\(i\)estimates the aggregated importance of cached tokeniithrough steptt\. Whenever the cache exceedsBB, we evict the tokens with the lowestmt​\(i\)m\_\{t\}\(i\)\. The aggregation rule is the primary design choice:

Single\-step:mt\\displaystyle\\makebox\[55\.00008pt\]\[l\]\{\\text\{Single\-step:\}\}m\_\{t\}=st,\\displaystyle=s\_\{t\},\(2\)Cumulative:mt\\displaystyle\\makebox\[55\.00008pt\]\[l\]\{\\text\{Cumulative:\}\}m\_\{t\}=mt−1\+st,\\displaystyle=m\_\{t\-1\}\+s\_\{t\},\(3\)EMA:mt\\displaystyle\\makebox\[55\.00008pt\]\[l\]\{\\text\{EMA:\}\}m\_\{t\}=α​mt−1\+\(1−α\)​st\.\\displaystyle=\\alpha\\,m\_\{t\-1\}\+\(1\{\-\}\\alpha\)\\,s\_\{t\}\.\(4\)These three rules span the stability–adaptivity axis: single\-step is maximally adaptive but noisy; cumulative never forgets but adapts slowly to relevance shifts; EMA \(α∈\(0,1\)\\alpha\\in\(0,1\)\) interpolates between them with bounded memory\. Equations[2](https://arxiv.org/html/2609.03515#S3.E2)–[4](https://arxiv.org/html/2609.03515#S3.E4)present aggregation at the decode\-step level\. In our implementation, compression is applied at multiple transformer layers, and the EMA state is updated after each such layer; consequently,α\\alphacontrols both persistence across decode steps and the relative influence of successive layers\.

Under aggressive compression, the choice ofα\\alphaexposes a fundamental tradeoff between two error modes\.*Boundary errors*: noisy score fluctuations near the budget boundary permanently evict useful tokens\.*Lag errors*: an overly slow estimator retains stale tokens after relevance has shifted\. We express this as a two\-term risk:

ℒ⁡\(α\)=ℰboundary​\(α\)⏟ranking instability\+ℰlag​\(α\)⏟adaptation delay\.\\mathcal\{L\}\(\\alpha\)=\\underbrace\{\\mathcal\{E\}\_\{\\mathrm\{boundary\}\}\(\\alpha\)\}\_\{\\text\{ranking instability\}\}\+\\underbrace\{\\mathcal\{E\}\_\{\\mathrm\{lag\}\}\(\\alpha\)\}\_\{\\text\{adaptation delay\}\}\.\(5\)We treat this decomposition as a qualitative design lens rather than a formal objective; the quantitative bound on ranking stability is given in Proposition[1](https://arxiv.org/html/2609.03515#Thmproposition1)\. This framing guides the alpha\-sensitivity analysis \(§[5\.5](https://arxiv.org/html/2609.03515#S5.SS5)\) and explains why adaptive scheduling fails \(Appendix[H](https://arxiv.org/html/2609.03515#A8)\)\.

We instantiate this framework asInertiaKV: decode\-side EMA aggregation \(Eq\.[4](https://arxiv.org/html/2609.03515#S3.E4)\) withα=0\.8\\alpha\{=\}0\.8and a proportional budgetB=⌊\(1−ρ\)​L0⌋B\{=\}\\lfloor\(1\{\-\}\\rho\)L\_\{0\}\\rfloor\. The name reflects the core mechanism—EMA builds*ranking inertia*that absorbs per\-step scoring perturbations \(§[5\.3](https://arxiv.org/html/2609.03515#S5.SS3)\)\. Compression is applied during decoding \(not prefill\), making it naturally query\-aware but not a prefill memory reduction\. In one 79\-step diagnostic trace atα=0\.8\\alpha\{=\}0\.8and 90% compression, the top\-BBset remains identical to its first\-step set, although within\-set rankings continue to change \(Appendix[I](https://arxiv.org/html/2609.03515#A9)\)\.

### 3\.2Exploiting Ranking Stability

If utility ranking changes little between consecutive steps, the expensive query\-aware scorests\_\{t\}need not be recomputed every step\. We define two operating points that progressively exploit this property\.

#### Lazy refresh\.

InertiaKV\-Lazy refreshes scores at the first decode step and everyrrsteps thereafter\. Letℛr=\{t≥1:\(t−1\)≡0\(modr\)\}\\mathcal\{R\}\_\{r\}=\\\{t\\geq 1:\(t\-1\)\\equiv 0\\pmod\{r\}\\\}denote the set of refresh steps:

mt=\{α​mt−1\+\(1−α\)​st,t∈ℛr,mt−1,otherwise\.m\_\{t\}\{=\}\\begin\{cases\}\\alpha m\_\{t\-1\}\{\+\}\(1\{\-\}\\alpha\)s\_\{t\},&t\\in\\mathcal\{R\}\_\{r\},\\\\\[2\.0pt\] m\_\{t\-1\},&\\text\{otherwise\}\.\\end\{cases\}\(6\)Eviction still occurs whenever the cache exceedsBB; only the score observation is skipped on non\-refresh steps\. The discrepancy from full refresh can grow with both the refresh interval and the update weight\. This heuristic motivates moderate refresh intervals; the resulting quality–speed tradeoff is evaluated empirically rather than guaranteed by Proposition[1](https://arxiv.org/html/2609.03515#Thmproposition1)\.

#### Score\-Free decoding\.

Score\-Free is a separate empirical operating point rather than ther→∞r\\\!\\to\\\!\\inftyconsequence of Proposition[1](https://arxiv.org/html/2609.03515#Thmproposition1)\. EMA is inactive during prefill and the momentum state is initialized asm0=0m\_\{0\}=0\. At the first decode step, Score\-Free sequentially processes the eligible layer scores and freezes the resulting state\. No score is computed from step 2 onward, while eviction continues using this fixed ranking\. Thus Score\-Free is more precisely “score once, then never refresh\.” Its effectiveness is an empirical observation about the evaluated workloads, not a claim that the first\-step ranking is universally optimal\.

Score\-Free decodingInput:prefill KV cache𝒞0\\mathcal\{C\}\_\{0\}, budgetBB, EMA coefficientα\\alpha\.Initialize:m0​\(i\)←0m\_\{0\}\(i\)\\leftarrow 0for every prefill positionii\.First decode step:sequentially process the eligible full\-context layer scores; freeze the resulting state and evict the lowest\-ranked entries if\|𝒞1\|\>B\|\\mathcal\{C\}\_\{1\}\|\>B\.Later stepst≥2t\\geq 2:do not computests\_\{t\}and keep the prefill\-token ranking fixed bym1m\_\{1\}; initialize each new token to the current mean momentum, and evict whenever the cache exceedsBB\.

Controlled auxiliary ablations \(static reserve, first\-eviction rescue\) are reported in Appendix[B](https://arxiv.org/html/2609.03515#A2)\.

## 4Experimental Setup

### 4\.1Benchmarks

Our main evaluation uses LongBench\([Bai et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib2)\), LongBench\-v2\([Bai et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib3)\), and RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib10)\)\. LongBench covers extractive QA and generation tasks\. RULER comprises 13 synthetic subtasks spanning single\-key, multi\-key, multi\-query, and multi\-value retrieval, plus variable tracking and word extraction; multi\-evidence subtasks \(multikey, multiquery\) are the hardest for eviction methods because they require retaining multiple dispersed tokens simultaneously—the regime where aggressive compression most frequently fails\. We additionally include a needle\-in\-a\-haystack grid as an auxiliary retrieval stress test, a MoE stress check on Qwen3\-30B\-A3B, and architecture/scale stress tests in the appendix\. LongBench and RULER are on a 0–100 scale; LongBench\-v2 uses proportion correct \(0–1\) following its official protocol\.

### 4\.2Models

We evaluate on Llama\-3\.1\-8B\-Instruct\([Llama Team, AI at Meta, 2024](https://arxiv.org/html/2609.03515#bib.bib18)\)and Qwen2\.5\-7B\-Instruct\([Qwen Team, 2024](https://arxiv.org/html/2609.03515#bib.bib21)\)for the main results\. These two models represent widely used open\-weight families and use different grouped\-query attention \(GQA;[Ainslie et al\., 2023](https://arxiv.org/html/2609.03515#bib.bib1)\) configurations: Llama uses 8 KV heads with4×4\\timesquery\-head grouping, while Qwen uses 4 KV heads with7×7\\timesgrouping\. This provides coverage of two distinct GQA configurations\. We include Llama\-3\.3\-70B\-Instruct to test whether the same empirical pattern extends beyond 7–8B parameters, and Qwen3\-30B\-A3B\-Instruct as a minimal MoE stress check where sparse expert activation may interact differently with cache eviction\. Appendix stress tests use Mistral\-7B\-Instruct\-v0\.3 and Qwen2\.5\-14B\-Instruct to probe transfer limits rather than to support the main claim\. All experiments run on NVIDIA H100 and H200 GPUs\.

### 4\.3Compared Methods

Baselines span the landscape of KV cache compression: TOVA\([Oren et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib19)\)\(single\-step last\-query attention\), SnapKV\([Li et al\., 2024](https://arxiv.org/html/2609.03515#bib.bib15)\)\(prefill\-time observation\-window selection\), AdaKV \+ Expected Attention\([Feng et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib8)\)\(adaptive per\-head budgets\), KeyDiff\([Park et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib20)\)\(key\-vector geometry\), and KVzip\([Kim et al\., 2025](https://arxiv.org/html/2609.03515#bib.bib12)\)\(reconstruction\-based compression\)\. We report both the full\-refresh InertiaKV scorer and practical InertiaKV\-Lazy variants\. All comparisons use 90% compression—the aggressive regime in which temporal effects are most pronounced in our experiments \(Table[1](https://arxiv.org/html/2609.03515#S1.T1)\)\. We also include a cumulative\-attention baseline that replaces EMA \(Eq\.[4](https://arxiv.org/html/2609.03515#S3.E4)\) with the non\-decaying accumulator \(Eq\.[3](https://arxiv.org/html/2609.03515#S3.E3)\)\.

## 5Results

### 5\.1Order\-Preserving Scoring Perturbations Are Absorbed by EMA

Among the three aggregation rules defined in §[3\.1](https://arxiv.org/html/2609.03515#S3.SS1), EMA \(α=0\.8\\alpha\{=\}0\.8\) substantially outperforms single\-step on retrieval\-heavy settings \(\+1\.8\+1\.8on LongBench and\+21\.6\+21\.6on RULER on Llama\), while cumulative aggregation is competitive but mixed across models and benchmarks \(Appendix[G](https://arxiv.org/html/2609.03515#A7)\)\. Given this strong baseline, we now ask: are order\-preserving modifications to the scoring function absorbed by EMA momentum? We test two alternative scorers on Qwen2\.5\-7B across all eight LongBench tasks at 90% compression:VNorm, which multiplies each token’s attention weight by the mean\-normalizedℓ2\\ell\_\{2\}norm of its value state; andEntropy weighting, a per\-layer scalar that modulates the step’s overall influence without changing the relative ordering among tokens\. Both modifications are applied per step before EMA aggregation\.

Figure 2:Core findings\.\(a\)Approximately order\-preserving perturbations \(Entropy, VNorm\) are absorbed by EMA momentum \(avg\|Δ\|<0\.35\|\\Delta\|<0\.35\), while the evaluated learned scorer changes the ranking and diverges \(\|Δ\|=45\.06\|\\Delta\|=45\.06\)\.\(b\)Lazy refresh reduces scoring frequency with little quality change\. Score\-Free is shown as a separate endpoint that scores once at the first decode step and performs no subsequent refresh\.Figure[2](https://arxiv.org/html/2609.03515#S5.F2)\(a\) shows the result\. Entropy weighting produces*exactly*zero task\-score deviation in this evaluation\. Its per\-layer scalar preserves the within\-layer token ordering, although different layer weights can in principle alter the final aggregate ranking\. VNorm, a genuinely per\-token perturbation, produces small deviations \(avg\|Δ\|≤0\.33\|\\Delta\|\\leq 0\.33\), consistent with EMA momentum absorbing this token\-wise reweighting\. In contrast, theα=0\\alpha\{=\}0endpoint drops the average by 1\.33 points, and random eviction collapses to 12\.45\. Hereα=0\\alpha\{=\}0retains only the last processed layer, so the ablation is not purely temporal\. The same pattern holds on Llama\-3\.1\-8B \(VNorm≤0\.35\\leq 0\.35;α=0\\alpha\{=\}0drops from 43\.88 to 42\.11\)\.

#### Scale check: Llama\-3\.3\-70B\.

To test whether this absorption pattern extends beyond the 7–8B scale, we repeat the ablation on Llama\-3\.3\-70B\-Instruct at 90% compression \(Table[2](https://arxiv.org/html/2609.03515#S5.T2)\)\. At this scale, the per\-step perturbation produces small residual differences \(≤0\.67\\leq 0\.67points\), while theα=0\\alpha\{=\}0endpoint drops by 4–13 points across all tasks\. This contrast changes both layer weighting and temporal retention, rather than momentum alone\.

Table 2:Scoring\-function ablation on Llama\-3\.3\-70B\-Instruct at 90% compression\.VNorm deviates from EMA by at most 0\.67 points \(avgΔ=−0\.33\\Delta\{=\}\{\-\}0\.33\), whileα=0\\alpha\{=\}0drops by 7\.37 points on average\.
#### Ranking correlation defines the observed boundary\.

We measure each scorer against attention at layer 15 using Spearman correlation and top\-BBJaccard at a 10% budget\. VNorm and entropy remain highly correlated \(ρ≈0\.97\\rho\\\!\\approx\\\!0\.97\) and preserve quality\. KeyDiff changes the ordering \(ρ=0\.49\\rho\{=\}0\.49, Jaccard=0\.38=0\.38\) and loses roughly 9–14 LongBench points; key norm \(ρ=0\.00\\rho\{=\}0\.00\) and recency \(ρ=0\.12\\rho\{=\}0\.12\) collapse on retrieval\. The learned scorer is a 327,680\-parameter bilinear head,score​\(q,k\)=\(Wk​k\)⊤​\(Wq​q\)/64\\text\{score\}\(q,k\)=\(W\_\{k\}k\)^\{\\top\}\(W\_\{q\}q\)/\\sqrt\{64\}, trained by self\-distillation with MSE\+\+KL \(λ=0\.1\\lambda\{=\}0\.1,τ=1\.0\\tau\{=\}1\.0\)\. Its held\-out ranking is far from attention \(ρ≈0\.05\\rho\\\!\\approx\\\!0\.05, Jaccard≈0\.08\\approx 0\.08\); it loses 1\.94 points on LongBench and 45\.06 on RULER\. These results establish an empirical correlation spectrum, not a proof that every non\-attention scorer must fail: our evidence does not rule out a competitive alternative scorer with a different signal\.

### 5\.2Score Refresh Can Be Reduced or Eliminated

Because the decode\-time ranking changes slowly after EMA stabilization, frequent score updates may be redundant though scorer quality still matters\. We test this by reducing refresh frequency\.

#### Lazy refresh\.

InertiaKV\-Lazy4 is the default operating point: benchmark\-level quality deltas versus full refresh are small in magnitude\. LongBench confidence intervals include zero on both backbones; RULER intervals also include zero except for a small Llama regression \(Δ=−0\.151\\Delta\{=\}\{\-\}0\.151, 95% CI\[−0\.223,−0\.078\]\[\-0\.223,\-0\.078\]\)\. For Lazy4, benchmark deltas are below 0\.05 except for the Llama RULER regression reported above;r=4r\{=\}4provides a practical quality–speed compromise\. Lazy4 skips scoring on three out of every four steps, reducing layer\-scoring operations by4×4\\timesand yielding 1\.34×\\timesand 1\.46×\\timesdecode speedups on Llama and Qwen respectively at 64k context \(Figure[10](https://arxiv.org/html/2609.03515#A10.F10)in Appendix[J](https://arxiv.org/html/2609.03515#A10)\)\.

#### Score\-Free decoding\.

As specified in §[3\.2](https://arxiv.org/html/2609.03515#S3.SS2.SSS0.Px2), Score\-Free computes one full\-context score at the first decode step and then freezes the ranking\. Table[3](https://arxiv.org/html/2609.03515#S5.T3)reports Llama\-3\.1\-8B across all eight LongBench tasks at 90% compression\. The average deviation is\+0\.03\+0\.03, and the aggregate paired\-bootstrap 95% confidence interval spans zero\. Seven of eight tasks are statistically indistinguishable from full refresh\. MultiNews is a genuine small regression \(Δ=−0\.61\\Delta\{=\}\{\-\}0\.61, 95% CI\[−1\.01,−0\.21\]\[\-1\.01,\-0\.21\], pairedtt\-testp=0\.003p\{=\}0\.003\), whereas passage retrieval improves by\+0\.50\+0\.50but is not significant \(95% CI\[0\.00,1\.50\]\[0\.00,1\.50\],p=0\.32p\{=\}0\.32\)\. On Qwen2\.5\-7B, the average deviation is−0\.01\-0\.01\(max\|Δ\|=0\.47\|\\Delta\|\{=\}0\.47\)\.

Table 3:Score\-Free decoding on Llama\-3\.1\-8B across eight LongBench tasks at 90% compression\.Score\-Free computes attention once at the first decode step and never refreshes afterward\.After first\-step initialization, the score\-update machinery can therefore be removed from the decode loop, minimizing score\-update overhead among the evaluated InertiaKV variants\. Figure[3](https://arxiv.org/html/2609.03515#S5.F3)quantifies this cost: at 16k context on Llama\-3\.1\-8B, score updates account for 36% of Full Refresh decode time; Lazy4 reduces this share to 14%; Score\-Free removes subsequent score updates\. Controlled ablations show that auxiliary mechanisms \(static reserve, first\-eviction rescue\) do not improve this operating point \(Appendix[B](https://arxiv.org/html/2609.03515#A2)\)\.

Figure 3:Decode time composition at 16k context on Llama\-3\.1\-8B \(90% compression\)\. Score updates account for 36% of Full Refresh decode time; Lazy4 reduces this share to 14%; Score\-Free removes score updates after first\-step initialization\.

### 5\.3Why: Momentum Dominance

The scorer ablations and lazy\-refresh results are consistent with a common mechanism: EMA accumulation builds*ranking inertia*that can reduce the effect of large single\-step perturbations\. Even when per\-step score perturbations are large in relative L2 norm, the current update enters EMA with weight\(1−α\)=0\.2\(1\{\-\}\\alpha\)\{=\}0\.2; an 82% relative perturbation therefore contributes at roughly 16% relative scale before accounting for the accumulated state\. The empirical evidence for stabilization is the reduction in top\-BBchurn and high retention\-set persistence reported in Appendix[I](https://arxiv.org/html/2609.03515#A9); Figure[4](https://arxiv.org/html/2609.03515#S5.F4)illustrates the mechanism\. Score\-Free is intentionally excluded from this explanation: it freezes the first\-step ranking and is supported by the separate empirical test in §[5\.2](https://arxiv.org/html/2609.03515#S5.SS2)\.

For analysis, we consider an idealized fixed\-token process with one aggregate update per step, omitting token insertion, eviction, and the layer\-sequential updates used in the reported implementation\. We denote its state byμT\\mu\_\{T\}to distinguish it from the dynamic cache statemtm\_\{t\}\. Under EMA,μT​\(i\)=\(1−α\)​∑τ=1TαT−τ​sτ​\(i\)\+αT​μ0​\(i\)\\mu\_\{T\}\(i\)=\(1\{\-\}\\alpha\)\\sum\_\{\\tau=1\}^\{T\}\\alpha^\{T\-\\tau\}s\_\{\\tau\}\(i\)\+\\alpha^\{T\}\\mu\_\{0\}\(i\)\.

###### Proposition 1\(EMA Expected\-Gap and Rank\-Stability Bound\)\.

For each tokenii, assume that\{st​\(i\)\}t≥1\\\{s\_\{t\}\(i\)\\\}\_\{t\\geq 1\}is i\.i\.d\. over time with means¯​\(i\)\\bar\{s\}\(i\)and variance bounded byσ2\\sigma^\{2\}, with independence across tokens and deterministic initializationμ0​\(i\)=0\\mu\_\{0\}\(i\)=0\. For two tokensi,ji,jwith utility gapδi​j≜s¯​\(i\)−s¯​\(j\)\>0\\delta\_\{ij\}\\triangleq\\bar\{s\}\(i\)\-\\bar\{s\}\(j\)\>0:

1. \(a\)The expected momentum gap converges toδi​j\\delta\_\{ij\}\.
2. \(b\)Var⁡\[μT​\(i\)−μT​\(j\)\]≤2​\(1−α\)1\+α​σ2\\mathrm\{Var\}\[\\mu\_\{T\}\(i\)\-\\mu\_\{T\}\(j\)\]\\leq\\dfrac\{2\(1\-\\alpha\)\}\{1\+\\alpha\}\\,\\sigma^\{2\}\.
3. \(c\)In the stationary limit,lim supT→∞Pr\[μT\(i\)<μT\(j\)\]≤2​\(1−α\)​σ2\(1\+α\)​δi​j2\\displaystyle\\limsup\_\{T\\to\\infty\}\\Pr\[\\mu\_\{T\}\(i\)<\\mu\_\{T\}\(j\)\]\\leq\\dfrac\{2\(1\-\\alpha\)\\sigma^\{2\}\}\{\(1\+\\alpha\)\\,\\delta\_\{ij\}^\{2\}\}\.

The bound decreases with stronger smoothing and with a larger expected utility gap relative to score variance\.

###### Corollary 2\(Expected\-Ranking Preservation under EMA\)\.

If an alternative scorer applies a common strictly increasing transformation to expected utilities \(i\.e\.,𝔼⁡\[s~t​\(i\)\]=f⁡\(s¯​\(i\)\)\\mathbb\{E\}\[\\tilde\{s\}\_\{t\}\(i\)\]=f\(\\bar\{s\}\(i\)\)\), its expected EMA momentum has the same asymptotic ranking\. Uniform positive scaling is exactly invisible to ranking, while the misranking bound in Proposition[1](https://arxiv.org/html/2609.03515#Thmproposition1)depends on the transformed gaps and variances\.

Proof is in Appendix[A](https://arxiv.org/html/2609.03515#A1)\. On Llama\-3\.1\-8B, we observe the corresponding stabilization diagnostics: raw attention scores exhibit 34% top\-BBJaccard churn; EMA reduces this to 8%; and 75% of decode steps produce≤1\\leq 1token change in the eviction set\.

Figure 4:Two\-token illustration showing how EMA can suppress boundary flips\.\(a\) Raw per\-step scores of tokens near the rank\-BBboundary cross the threshold in 5 of 16 steps\. \(b\) EMA smoothing \(α=0\.8\\alpha=0\.8\) yields persistent separation without flips\. Appendix I provides empirical rank\-trace statistics\.Figure 5:Quality vs\. KV budget across three benchmarkson Llama\-3\.1\-8B \(top\) and Qwen2\.5\-7B \(bottom\) at compression ratios 50%, 75%, and 90%\. InertiaKV \(dark blue\) maintains consistently strong quality under aggressive compression, while lightweight baselines degrade sharply—especially on RULER\.
### 5\.4Comparison with Existing Methods

Having established the interaction between EMA and scorer ranking, we now position InertiaKV against existing methods\. Table[4](https://arxiv.org/html/2609.03515#S5.T4)reports quality under the same 90% compression protocol against TOVA, SnapKV, AdaKV \+ Expected Attention, KVzip, and KeyDiff\.

Table 4:Baseline comparison under the 90% compression\-ratio protocol\.LongBench\-v2 scores are on a 0–1 scale; all others are on 0–100\. Methods differ in compression timing, memory footprint, and computational profile\.InertiaKV and InertiaKV\-Lazy4 outperform all lightweight baselines \(TOVA, SnapKV, AdaKV, KeyDiff\) on most rows\. KVzip is strongest on Llama RULER \(\+10\.84 over InertiaKV\), and we therefore report the full quality–cost comparison rather than treating the methods as directly interchangeable\. In the released masking\-based KVzip mode that we evaluated, KVzip obtains 86\.44 RULER with 31\.54 s prefill, 10\.9 token/s decode, and 64,029 retained decode\-side tokens; InertiaKV obtains 75\.60 with 4\.86 s prefill, 39\.1 token/s decode, and 6,401 retained tokens\. Thus KVzip provides higher RULER quality, while InertiaKV is 6\.5×\\timesfaster in prefill, 3\.6×\\timesfaster in decode, and retains approximately 10×\\timesfewer decode\-side KV tokens\. Figure[6](https://arxiv.org/html/2609.03515#S5.F6)makes these tradeoffs explicit\. Figure[5](https://arxiv.org/html/2609.03515#S5.F5)shows quality across KV budgets for the remaining baselines\.

Figure 6:Quality–cost tradeoffs on Llama\-3\.1\-8B at 64k context: \(a\) RULER versus retained decode\-side KV tokens across compression ratios; \(b\) RULER versus decode throughput at 90% compression\. KVzip denotes released masking\-based mode evaluated\.
### 5\.5Temporal Sensitivity

The choice of smoothing strengthα\\alphacontrols the boundary\-error / lag\-error tradeoff introduced in Eq\.[5](https://arxiv.org/html/2609.03515#S3.E5)\. Table[5](https://arxiv.org/html/2609.03515#S5.T5)sweepsα\\alphafrom 0 \(single\-step\) to 0\.95 on Llama\-3\.1\-8B at 90% compression\.

Table 5:Alpha sensitivity of InertiaKV at 90% compression on Llama\-3\.1\-8B\.Stronger smoothing improves LongBench and RULER; LongBench\-v2 peaks at smallerα\\alpha\.The pattern directly reflects the two\-term risk: retrieval\-heavy benchmarks prefer strong temporal memory \(α=0\.90\\alpha\{=\}0\.90–0\.950\.95\), LongBench is robust around0\.800\.80–0\.900\.90, and LongBench\-v2 peaks atα=0\.30\\alpha\{=\}0\.30, consistent with an adaptation\-sensitive minority that penalizes excessive smoothing\. We useα=0\.80\\alpha\{=\}0\.80as a robustness compromise rather than a per\-task optimum\. Comparing EMA against cumulative \(no\-forgetting\) aggregation is consistent with this picture: cumulative attention gains on Llama RULER \(\+2\.01\) but drops on Qwen RULER \(−\-7\.66\), making EMA the more robust bounded\-memory compromise \(Appendix[G](https://arxiv.org/html/2609.03515#A7)\)\.

#### Why online adaptation did not help\.

We also tested a theory\-derived adaptive selector that estimates noise and drift during an initial warmup\. It almost always selectsα=0\.8\\alpha\{=\}0\.8or0\.950\.95and cannot recover the low\-α\\alphapreference of LongBench\-v2\. During warmup, the noise\-to\-drift ratio is large \(η=50\\eta\{=\}50–600600\), so the selector favors strong smoothing; useful relevance shifts arrive later\. A reactive rank\-disagreement trigger is also insufficient: disagreement correlates only 0\.14 with alpha sensitivity and does not identify whetherα\\alphashould increase or decrease\. Appendix[H](https://arxiv.org/html/2609.03515#A8)gives the full negative result\. This suggests that successful adaptation requires directional detection of relevance shifts, not only early noise/drift estimation\.

#### Short\-prefix check\.

Because Score\-Free does not accumulate EMA during prefill, prefix length is not an EMA\-convergence requirement\. On the needle\-in\-a\-haystack stress test with 1k, 2k, 4k, and 8k prefills, Score\-Free obtains ROUGE\-L\-F 0\.71 at every length, matching Full Refresh and Lazy4\. The more relevant failure mode is a later shift in token relevance: on LongBench\-v2, 97\.6% of examples are invariant across temporal rules, while the remaining 2\.4% are adaptation\-sensitive\.

### 5\.6Efficiency Analysis

The previous sections establish the quality behavior; we now examine efficiency\. At 64k context, 90% compression reduces the logical final decode\-side KV footprint from 64,029 to 6,401 retained tokens, an approximately10×10\\timesreduction in retained\-token count\. This logical reduction does not by itself establish a10×10\\timesserving\-concurrency gain, which additionally depends on allocator behavior, page reclamation, prefill peaks, and batching\. InertiaKV does not reduce prefill peak memory \(29\.91 GB, same as full cache\), unlike TOVA \(22\.88 GB\) which evicts during prefill, but achieves a similar final retained\-token count \(6,401 vs\. 6,428\)\. In the evaluated masking\-based mode, KVzip retains 64,029 decode\-side tokens and incurs6\.5×6\.5\\timesprefill cost \(31\.54 s vs\. 4\.86 s\)\. InertiaKV\-Lazy further reduces decode cost via lazy refresh, scoring only every fourth step and cutting layer\-scoring operations by4×4\\times\(Figure[10](https://arxiv.org/html/2609.03515#A10.F10)in Appendix[J](https://arxiv.org/html/2609.03515#A10)\)\.

## 6Conclusion

Under the tested aggressive\-compression settings, EMA makes approximately order\-preserving scorer modifications largely indistinguishable at the eviction\-set level, but this conclusion does not extend to scorers that substantially alter utility ranking\. This distinction motivates InertiaKV\-Lazy4, which offers 1\.34–1\.46×\\timesdecode throughput relative to full\-refresh InertiaKV with little quality change\. Score\-Free is a separate empirical endpoint: scoring once at the first decode step and freezing the ranking yields an averageΔ=\+0\.03\\Delta\{=\}\+0\.03, with one significant small regression on MultiNews\. The results redirect attention toward temporal aggregation without claiming that scoring quality is generally irrelevant\. Adaptiveα\\alpha, relevance\-shift detection, and competitive non\-attention scorers remain open directions\.

## Limitations

InertiaKV and InertiaKV\-Lazy are decode\-only compressors that target the decode/time\-between\-tokens bottleneck; they do not reduce prefill peak memory and should be composed with a prefill\-time compressor when prefill memory is binding\. Score\-Free uses only the first decode\-step ranking and therefore may fail when relevance shifts later in generation; its stable 1k–8k needle\-in\-a\-haystack result does not establish robustness for multi\-turn dialogue or short\-prompt, long\-generation workloads\. At 90% compression, Mistral\-7B and Qwen2\.5\-14B degrade by 17–27 RULER points mainly on multi\-needle and distant\-token retrieval\. Because InertiaKV\-Lazy4 closely tracks full refresh on these models, the observed limitation lies in the underlying scorer/task interaction rather than lazy temporal aggregation\. Because the implementation updates EMA per eligible layer,α\\alphacouples layer weighting and temporal retention, limiting purely temporal interpretation and potentially architecture transfer\. EMA is also not a universal wrapper: applying it to other scoring rules produces only marginal changes \(Appendix[D](https://arxiv.org/html/2609.03515#A4)\)\. Finally, our evaluation is English\-only and does not rule out a competitive non\-attention scorer whose ranking differs from attention while retaining task utility\.

## Acknowledgments

We thank the anonymous reviewers and the area chair for their valuable feedback and constructive suggestions\. This research was jointly supported by Alibaba Group and the Excellence Funds of Universität Hamburg\.

## References

- Ainslie et al\. \(2023\)Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai\. 2023\.[GQA: Training generalized multi\-query transformer models from multi\-head checkpoints](https://doi.org/10.18653/v1/2023.emnlp-main.298)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 4895–4901, Singapore\. Association for Computational Linguistics\.
- Bai et al\. \(2024\)Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li\. 2024\.[LongBench: A bilingual, multitask benchmark for long context understanding](https://doi.org/10.18653/v1/2024.acl-long.172)\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3119–3137, Bangkok, Thailand\. Association for Computational Linguistics\.
- Bai et al\. \(2025\)Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li\. 2025\.[LongBench v2: Towards deeper understanding and reasoning on realistic long\-context multitasks](https://doi.org/10.18653/v1/2025.acl-long.183)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 3639–3664, Vienna, Austria\. Association for Computational Linguistics\.
- Behnam et al\. \(2025\)Payman Behnam, Yaosheng Fu, Ritchie Zhao, Po\-An Tsai, Zhiding Yu, and Alexey Tumanov\. 2025\.[RocketKV: Accelerating long\-context LLM inference via two\-stage KV cache compression](https://proceedings.mlr.press/v267/behnam25a.html)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 3358–3392, Vancouver, British Columbia, Canada\. PMLR\.
- Brandon et al\. \(2024\)William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, and Jonathan Ragan\-Kelley\. 2024\.[Reducing transformer key\-value cache size with cross\-layer attention](https://proceedings.neurips.cc/paper_files/paper/2024/hash/9e23d020c18e4c40d81c6a0fc7a46f68-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 37*, pages 86927–86945, Vancouver, British Columbia, Canada\. Curran Associates, Inc\.
- Cai et al\. \(2025\)Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, and Wen Xiao\. 2025\.[PyramidKV: Dynamic KV cache compression based on pyramidal information funneling](https://openreview.net/forum?id=ayi7qezU87)\.In*Proceedings of the Second Conference on Language Modeling*, Montreal, Quebec, Canada\.
- Dao \(2024\)Tri Dao\. 2024\.[FlashAttention\-2: Faster attention with better parallelism and work partitioning](https://proceedings.iclr.cc/paper_files/paper/2024/hash/98ed250b203d1ac6b24bbcf263e3d4a7-Abstract-Conference.html)\.In*The Twelfth International Conference on Learning Representations*, pages 35549–35562, Vienna, Austria\.
- Feng et al\. \(2025\)Yuan Feng, Junlin Lv, Yukun Cao, Xike Xie, and S\. Kevin Zhou\. 2025\.[Ada\-KV: Optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference](https://proceedings.neurips.cc/paper_files/paper/2025/hash/a40ff56daab9f4808b1e18350c8a11ce-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 38*, pages 113152–113188, San Diego, California, USA; Mexico City, Mexico\. Curran Associates, Inc\.
- Ge et al\. \(2024\)Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao\. 2024\.[Model tells you what to discard: Adaptive KV cache compression for LLMs](https://proceedings.iclr.cc/paper_files/paper/2024/hash/639a9a172c044fbb64175b5fad42e9a5-Abstract-Conference.html)\.In*The Twelfth International Conference on Learning Representations*, pages 22975–22988, Vienna, Austria\.
- Hsieh et al\. \(2024\)Cheng\-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg\. 2024\.[RULER: What’s the real context size of your long\-context language models?](https://openreview.net/forum?id=kIoBbc76Sy)In*Proceedings of the First Conference on Language Modeling*, Philadelphia, Pennsylvania, USA\.
- Joshi et al\. \(2025\)Vinay Joshi, Pratik Prabhanjan Brahma, Zicheng Liu, and Emad Barsoum\. 2025\.[TaDA: Training\-free recipe for decoding with adaptive KV cache compression and mean\-centering](https://doi.org/10.18653/v1/2025.acl-industry.101)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\)*, pages 1435–1443, Vienna, Austria\. Association for Computational Linguistics\.
- Kim et al\. \(2025\)Jang\-Hyun Kim, Jinuk Kim, Sangwoo Kwon, Jae W\. Lee, Sangdoo Yun, and Hyun Oh Song\. 2025\.[KVzip: Query\-agnostic KV cache compression with context reconstruction](https://proceedings.neurips.cc/paper_files/paper/2025/hash/f4eaa4b8f2d08edb3f0af990d56134ea-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 38*, pages 167563–167591, San Diego, California, USA; Mexico City, Mexico\. Curran Associates, Inc\.
- Lee et al\. \(2024\)Wonbeom Lee, Jungi Lee, Junghwan Seo, and Jaewoong Sim\. 2024\.[InfiniGen: Efficient generative inference of large language models with dynamic KV cache management](https://www.usenix.org/conference/osdi24/presentation/lee)\.In*18th USENIX Symposium on Operating Systems Design and Implementation*, pages 155–172, Santa Clara, California, USA\. USENIX Association\.
- Li et al\. \(2025\)Runchao Li, Yao Fu, Mu Sheng, Xianxuan Long, Haotian Yu, and Pan Li\. 2025\.[FAEDKV: Infinite\-window fourier transform for unbiased KV cache compression](https://doi.org/10.18653/v1/2025.findings-emnlp.914)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 16856–16866, Suzhou, China\. Association for Computational Linguistics\.
- Li et al\. \(2024\)Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\. 2024\.[SnapKV: LLM knows what you are looking for before generation](https://proceedings.neurips.cc/paper_files/paper/2024/hash/28ab418242603e0f7323e54185d19bde-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 37*, pages 22947–22970, Vancouver, British Columbia, Canada\. Curran Associates, Inc\.
- Liu et al\. \(2023\)Zichang Liu, Aditya Desai, Fangshuo Liao, Weitao Wang, Victor Xie, Zhaozhuo Xu, Anastasios Kyrillidis, and Anshumali Shrivastava\. 2023\.[Scissorhands: Exploiting the persistence of importance hypothesis for LLM KV cache compression at test time](https://proceedings.neurips.cc/paper_files/paper/2023/hash/a452a7c6c463e4ae8fbdc614c6e983e6-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 36*, pages 52342–52364, New Orleans, Louisiana, USA\. Curran Associates, Inc\.
- Liu et al\. \(2024\)Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\. 2024\.[KIVI: A tuning\-free asymmetric 2bit quantization for KV cache](https://proceedings.mlr.press/v235/liu24bi.html)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 32332–32344, Vienna, Austria\. PMLR\.
- Llama Team, AI at Meta \(2024\)Llama Team, AI at Meta\. 2024\.[The Llama 3 herd of models](https://arxiv.org/abs/2407.21783)\.*arXiv preprint arXiv:2407\.21783*\.
- Oren et al\. \(2024\)Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz\. 2024\.[Transformers are multi\-state RNNs](https://doi.org/10.18653/v1/2024.emnlp-main.1043)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 18724–18741, Miami, Florida, USA\. Association for Computational Linguistics\.
- Park et al\. \(2025\)Junyoung Park, Dalton Jones, Matthew Morse, Raghavv Goel, Mingu Lee, and Christopher Lott\. 2025\.[KeyDiff: Key similarity\-based KV cache eviction for long\-context LLM inference in resource\-constrained environments](https://proceedings.neurips.cc/paper_files/paper/2025/hash/0907335ecf28faf15be54485dbcbe70e-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 38*, pages 5983–6019, San Diego, California, USA; Mexico City, Mexico\. Curran Associates, Inc\.
- Qwen Team \(2024\)Qwen Team\. 2024\.[Qwen2\.5 technical report](https://arxiv.org/abs/2412.15115)\.*arXiv preprint arXiv:2412\.15115*\.
- Sun et al\. \(2025\)Hanshi Sun, Li\-Wen Chang, Wenlei Bao, Size Zheng, Ningxin Zheng, Xin Liu, Harry Dong, Yuejie Chi, and Beidi Chen\. 2025\.[ShadowKV: KV cache in shadows for high\-throughput long\-context LLM inference](https://proceedings.mlr.press/v267/sun25b.html)\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 57355–57373, Vancouver, British Columbia, Canada\. PMLR\.
- Tang et al\. \(2024\)Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, and Song Han\. 2024\.[Quest: Query\-aware sparsity for efficient long\-context LLM inference](https://proceedings.mlr.press/v235/tang24h.html)\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 47901–47911, Vienna, Austria\. PMLR\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N\. Gomez, Łukasz Kaiser, and Illia Polosukhin\. 2017\.[Attention is all you need](https://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html)\.In*Advances in Neural Information Processing Systems 30*, pages 5998–6008, Long Beach, California, USA\. Curran Associates, Inc\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\. 2024\.[Efficient streaming language models with attention sinks](https://proceedings.iclr.cc/paper_files/paper/2024/hash/5e5fd18f863cbe6d8ae392a93fd271c9-Abstract-Conference.html)\.In*The Twelfth International Conference on Learning Representations*, pages 21875–21895, Vienna, Austria\.
- Yu and Chai \(2025\)Bohan Yu and Yekun Chai\. 2025\.[EvolKV: Evolutionary KV cache compression for LLM inference](https://doi.org/10.18653/v1/2025.findings-emnlp.88)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 1673–1689, Suzhou, China\. Association for Computational Linguistics\.
- Zhang et al\. \(2023\)Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\. 2023\.[H2O: Heavy\-hitter oracle for efficient generative inference of large language models](https://proceedings.neurips.cc/paper_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html)\.In*Advances in Neural Information Processing Systems 36*, pages 48951–48971, New Orleans, Louisiana, USA\. Curran Associates, Inc\.

## Appendix AProof of Proposition[1](https://arxiv.org/html/2609.03515#Thmproposition1)

###### Proof\.

\(a\)Expanding the EMA recurrence givesμT​\(i\)=\(1−α\)​∑τ=1TαT−τ​sτ​\(i\)\+αT​μ0​\(i\)\\mu\_\{T\}\(i\)=\(1\{\-\}\\alpha\)\\sum\_\{\\tau=1\}^\{T\}\\alpha^\{T\-\\tau\}s\_\{\\tau\}\(i\)\+\\alpha^\{T\}\\mu\_\{0\}\(i\)\. Taking expectations,𝔼⁡\[μT​\(i\)\]=\(1−αT\)​s¯​\(i\)\+αT​μ0​\(i\)→s¯​\(i\)\\mathbb\{E\}\[\\mu\_\{T\}\(i\)\]=\(1\-\\alpha^\{T\}\)\\bar\{s\}\(i\)\+\\alpha^\{T\}\\mu\_\{0\}\(i\)\\to\\bar\{s\}\(i\), so the expected gap converges toδi​j\\delta\_\{ij\}\.\(b\)By temporal independence,Var⁡\[μT​\(i\)\]≤\(1−α\)2​σ2​∑k=0T−1α2​k≤1−α1\+α​σ2\\mathrm\{Var\}\[\\mu\_\{T\}\(i\)\]\\leq\(1\{\-\}\\alpha\)^\{2\}\\sigma^\{2\}\\sum\_\{k=0\}^\{T\-1\}\\alpha^\{2k\}\\leq\\frac\{1\-\\alpha\}\{1\+\\alpha\}\\sigma^\{2\}; independence across tokens makes the gap variance at most twice this quantity\.\(c\)LetDT=μT​\(i\)−μT​\(j\)D\_\{T\}=\\mu\_\{T\}\(i\)\-\\mu\_\{T\}\(j\)\. By part \(a\),𝔼⁡\[DT\]→δi​j\>0\\mathbb\{E\}\[D\_\{T\}\]\\to\\delta\_\{ij\}\>0\. Applying Chebyshev’s inequality toDT−𝔼⁡\[DT\]D\_\{T\}\-\\mathbb\{E\}\[D\_\{T\}\]and then takinglim supT→∞\\limsup\_\{T\\to\\infty\}, together with part \(b\), yields the stated asymptotic bound\. The i\.i\.d\. and cross\-token independence assumptions are for tractability\. Real attention sequences are temporally dependent, so the proposition is an explanatory bound rather than a direct certificate for the empirical traces\. ∎

## Appendix BControlled Auxiliary Ablations

The main paper treats static reserve and first\-eviction rescue mechanisms as auxiliary design checks rather than as part of the method claim\. Table[6](https://arxiv.org/html/2609.03515#A2.T6)summarizes the controlled comparison\. The auxiliary variant uses the same temporal accumulation rule as InertiaKV, but additionally protects a static reserve and applies a first\-large\-eviction rescue rule\. These mechanisms were intended to protect coarse positional coverage and avoid early catastrophic eviction, but under the aggressive budgets studied here they do not reliably compose with the bounded\-memory utility estimator\.

Table 6:Mechanism\-level delta between the controlled auxiliary variant and the final InertiaKV scorer\. The auxiliary variant tests whether static reserve and first\-eviction rescue compose with EMA temporal utility aggregation; it is not treated as an external baseline\.A compression sweep shows that the gap between InertiaKV and the auxiliary variant grows as the budget becomes tighter on LongBench and RULER \(Table[7](https://arxiv.org/html/2609.03515#A2.T7)\)\. LongBench\-v2 remains mixed, which is consistent with the main paper’s interpretation that this benchmark exposes the lag side of temporal aggregation rather than supporting a universal design law\.

Table 7:Compression sweep on Llama\-3\.1\-8B\. The no\-reserve InertiaKV design becomes increasingly stronger than the controlled auxiliary variant as compression becomes more aggressive, especially on RULER\.Figure 7:Compression sweep visualization of Table[7](https://arxiv.org/html/2609.03515#A2.T7)\. The gap between InertiaKV and the Auxiliary variant widens monotonically on LongBench and RULER as compression becomes more aggressive; LongBench\-v2 remains mixed\.Table[8](https://arxiv.org/html/2609.03515#A2.T8)reports nearby operating\-point variants at 90% compression\. Among these controlled variants, the no\-reserve InertiaKV scorer is the strongest choice on LongBench and RULER\. Removing momentum is especially harmful on RULER, while LongBench\-v2 remains close across variants\.

Table 8:Auxiliary variants at 90% compression on Llama\-3\.1\-8B\. These results motivate the base operating point but are not the main contribution\.
## Appendix CAuxiliary Generalization Checks

Having established the core method configuration in Appendix[B](https://arxiv.org/html/2609.03515#A2), we now test generalization beyond the main LongBench/RULER evaluation\.

We use a needle\-in\-a\-haystack retrieval grid as an auxiliary stress test even though the main claims rely on averaged LongBench, LongBench\-v2, RULER, and profiling results\. ROUGE\-L\-F is the most useful metric in this setup; exact match and hit rate are not informative\. Figure[8](https://arxiv.org/html/2609.03515#A3.F8)reports a 12\-context\-length \(10k–120k\) by 9\-depth heatmap on Llama\-3\.1\-8B\. Full Cache is included as a reference, while compressed methods use the same 90% setting\. We use this grid as a fixed\-setting stress check rather than as a benchmark for selecting the best alpha or refresh interval\.

![Refer to caption](https://arxiv.org/html/2609.03515v1/fig3_needle_heatmap_panel.png)Figure 8:Auxiliary needle\-in\-a\-haystack heatmap on Llama\-3\.1\-8B using ROUGE\-L\-F \(12 context lengths×\\times9 depths\)\. Full Cache is shown as a reference; compressed methods use 90% compression\. InertiaKV and InertiaKV\-Lazy4 closely track the full\-cache pattern, while comparison baselines \(TOVA, SnapKV, AdaKV\+EA, KVzip, KeyDiff\) degrade under this aggressive retrieval stress test\.For a small non\-dense stress check, we additionally evaluate a Qwen3\-30B\-A3B MoE model \(Table[9](https://arxiv.org/html/2609.03515#A3.T9)\)\. This is not a full MoE benchmark suite\. The same auxiliary\-mechanism issue appears in this setting, although the method remains below full\-cache quality\.

Table 9:MoE stress check on Qwen3\-30B\-A3B at 90% compression\. The controlled auxiliary variant again underperforms InertiaKV\.Table[10](https://arxiv.org/html/2609.03515#A3.T10)reports two additional stress tests\. These are not part of the main claim\. They ask whether the same 90% compression operating point transfers to a different dense architecture and to a larger Qwen model\. The answer is mixed and mostly negative: InertiaKV\-Lazy4 continues to track the full\-refresh InertiaKV estimator, but the estimator itself can degrade substantially on RULER\. Subtask inspection shows that this degradation is concentrated in multi\-key, multi\-query, and context\-word retrieval cases, while many single\-key and QA subtasks remain much closer to full cache\. On Qwen2\.5\-14B, for example, the 4k RULER average falls from 95\.92 to 58\.80 becauseniah\_multikey\_3,niah\_multikey\_2, andniah\_multiquerydrop from 100\.0, 100\.0, and 99\.8 to 0\.0, 4\.8, and 10\.45; at 16k, the corresponding drops are 99\.8→\\rightarrow2\.0, 99\.4→\\rightarrow57\.8, and 99\.75→\\rightarrow56\.45\. InertiaKV\-Lazy4 differs from full refresh by at most 0\.65 points on these failed subtask averages, indicating that lazy refresh is not the primary source of the degradation\. H2O\-style cumulative scoring partially recovers some multi\-evidence cases, indicating that no\-forgetting accumulation can better preserve persistent evidence, but it still fails on the hardest multikey cases\. We therefore treat these results as evidence of a current scorer limitation rather than as positive architecture or scale generalization\. We omit the default DecodingPress StreamingLLM runs from this table because that wrapper uses a fixed decode interval and target size rather than the same 90% compression\-ratio protocol, and in these runs it often behaved as a near no\-op\.

Table 10:Architecture and scale stress tests at 90% compression\. InertiaKV\-Lazy4 remains close to full\-refresh InertiaKV, but the underlying scorer is not robust on RULER multi\-evidence retrieval for these backbones\.†Mistral LongBench\-v2 is evaluated with a 32k context cap because the uncapped run exceeded single\-GPU prefill memory\.
## Appendix DAdditional Transfer Results

Appendix[C](https://arxiv.org/html/2609.03515#A3)showed mixed transfer of the InertiaKV scorer across architectures, with substantial limitations on multi\-evidence retrieval\. A separate question is whether the EMA temporal wrapper itself transfers to other scoring rules\.

Our fixed\-budget transfer experiments test whether a generic EMA wrapper can improve other decoding\-time rules beyond the InertiaKV design\. These experiments use explicit target sizes rather than nominal compression ratios, since decoding\-time wrappers are controlled by budget and compression interval rather than by a prefill\-style top\-kkratio\.

Table[11](https://arxiv.org/html/2609.03515#A4.T11)shows the transfer results\. The overall picture is weak: EMA produces only small changes for TOVA, KeyDiff, and KNorm under fixed budgets\. This is not a contradiction of the main result; it reinforces our scope\. Temporal smoothing is useful in the studied TOVA\-style irreversible eviction setting, but it is not a universal wrapper that automatically improves every scoring rule\.

Table 11:Fixed\-budget EMA transfer results\. EMA smoothing yields only small changes across these decoding\-time rules, so we treat transfer as weak supporting evidence rather than as a primary contribution\.
## Appendix EImplementation Details

This appendix specifies key implementation choices that are not fully explicit in the main text\. InertiaKV and InertiaKV\-Lazy are decode\-only compressors: they do not reduce prefill peak memory, and their primary effect is on the final decode\-side cache size and decode\-time score\-update cost\.

#### Per\-step scoring\.

At each decode step, we extract the attention weights from the last query token to all cached positions:at∈ℝH×Lta\_\{t\}\\in\\mathbb\{R\}^\{H\\times L\_\{t\}\}, whereHHis the number of attention heads andLtL\_\{t\}is the current cache length\. The per\-step score is obtained by mean\-pooling across heads:st​\(i\)=1H​∑h=1Hat\(h\)​\(i\)s\_\{t\}\(i\)=\\frac\{1\}\{H\}\\sum\_\{h=1\}^\{H\}a\_\{t\}^\{\(h\)\}\(i\)\. For GQA architectures \(Llama\-3\.1\), each KV head’s attention is shared across its corresponding query\-head group before pooling\. Optional VNorm weighting multiplies each head’s attention by the mean\-normalizedℓ2\\ell\_\{2\}norm of the value states before pooling; entropy weighting multiplies each layer’s contribution by the normalized entropy of its attention distribution\.

#### Momentum initialization \(m0m\_\{0\}\)\.

The momentum state is initialized to zeros for all positions:m0​\(i\)=0m\_\{0\}\(i\)=0fori=1,…,L0i=1,\\ldots,L\_\{0\}\. This means that the first decode step’s score determines the initial ranking entirely, and the expected momentum begins moving toward its stationary mean from step 1\.

#### New token insertion during non\-refresh steps\.

When a new token is generated during a non\-refresh step \(i\.e\.,t∉ℛrt\\notin\\mathcal\{R\}\_\{r\}in InertiaKV\-Lazy\), its momentum score is initialized to the mean momentum of theLt−1L\_\{t\}\-1existing tokens:mt​\(new\)=1Lt−1​∑i=1Lt−1mt−1​\(i\)m\_\{t\}\(\\mathrm\{new\}\)=\\frac\{1\}\{L\_\{t\}\-1\}\\sum\_\{i=1\}^\{L\_\{t\}\-1\}m\_\{t\-1\}\(i\)\. This places the new token at an average ranking position, allowing it to be scored and ranked at the next refresh\. The new token can still be evicted on the same step if the budget is exceeded\.

#### Layer scoring\.

LetKKdenote the number of eligible layers\. Each eligible layer immediately updates the shared state:mt,ℓ=α​mt,ℓ−1\+\(1−α\)​st,ℓm\_\{t,\\ell\}=\\alpha m\_\{t,\\ell\-1\}\+\(1\-\\alpha\)s\_\{t,\\ell\}forℓ=1,…,K\\ell=1,\\ldots,K, withmt,0=mt−1,Km\_\{t,0\}=m\_\{t\-1,K\}\. Thus the prior step has weightαK\\alpha^\{K\}and layerℓ\\ellhas weight\(1−α\)​αK−ℓ\(1\-\\alpha\)\\alpha^\{K\-\\ell\}; layers are not uniformly averaged\. For Lazy refresh, the layer hooks are deactivated on non\-refresh steps, so no attention is captured and the momentum state remains frozen\.

#### Eviction on non\-refresh steps\.

Even when scores are not refreshed, eviction still occurs if the cache exceeds the budget \(e\.g\., after a new token is added\)\. The stale momentummt−1m\_\{t\-1\}is used for the eviction decision\. This ensures the budget constraint is always satisfied regardless of the refresh schedule\.

## Appendix FBenchmark\-Level Interpretation Guide

Each benchmark in our evaluation tests a different aspect of the temporal aggregation tradeoff\. We provide brief guidance on how to read results from each\.

#### LongBench\-v2\.

LongBench\-v2 is consistently more mixed than LongBench and RULER\. Alpha sensitivity, pairwise flip analysis, and the failure\-anatomy statistics indicate that lower smoothing is preferable on a small but meaningful subset of examples, especially on Qwen2\.5\-7B\. Most examples \(95–98%\) are invariant across temporal scoring choices\. We treat LongBench\-v2 as a boundary case that exposes the*lag*side of the tradeoff rather than as evidence that EMA is broadly harmful\.

#### Needle\-in\-a\-haystack stress test\.

ROUGE\-L\-F is the only informative metric in our setup; exact match and hit rate are not discriminative\. We use this grid as a qualitative retrieval stress check \(Figure[8](https://arxiv.org/html/2609.03515#A3.F8)\) rather than as a core averaged benchmark\.

#### MoE\.

The Qwen3\-30B\-A3B experiment \(Table[9](https://arxiv.org/html/2609.03515#A3.T9)\) is a minimal stress check\. Its role is to check whether the auxiliary\-mechanism pattern \(static reserve hurts under aggressive budgets\) also appears beyond dense backbones\.

#### Architecture and scale stress tests\.

The Mistral\-7B and Qwen2\.5\-14B results \(Table[10](https://arxiv.org/html/2609.03515#A3.T10)\) should be read as identifying a*scorer limitation*, not a temporal aggregation failure\. In both cases, InertiaKV\-Lazy4 remains faithful to the full\-refresh estimator \(within 0\.1 points\), showing that lazy refresh is not the primary degradation source\. The degradation is concentrated in RULER multi\-evidence retrieval and appears tied to the full\-refresh attention\-derived scorer/task interaction under the evaluated eviction framework\. On Mistral, increasingα\\alphafrom 0\.8 to 0\.9 improves RULER from 72\.76 to 76\.79, partially mitigating but not solving the issue\. These results motivate future work on architecture\-aware scoring and multi\-span budget allocation\.

## Appendix GTemporal Sensitivity Analysis

This appendix characterizes the stability–adaptivity tradeoff across the temporal aggregation family, addressing the central design question: how much temporal memory should a decode\-time compressor keep?

### G\.1Alpha Sensitivity

The main alpha sensitivity sweep is reported in §[5\.5](https://arxiv.org/html/2609.03515#S5.SS5)\(Table[5](https://arxiv.org/html/2609.03515#S5.T5)\)\. Figure[9](https://arxiv.org/html/2609.03515#A7.F9)visualizes the normalized tradeoff curve\.

Figure 9:Alpha sensitivity visualized\. Each benchmark’s score is normalized by its best value across theα\\alpharange\. RULER benefits strongly from highα\\alpha; LongBench is relatively flat aboveα=0\.3\\alpha\{=\}0\.3; LongBench\-v2 slightly prefers lower smoothing\. The dashed line marks the defaultα=0\.8\\alpha\{=\}0\.8\.Here we provide cross\-model validation\. On Qwen2\.5\-7B,α=0\.90\\alpha=0\.90is essentially tied withα=0\.80\\alpha=0\.80on LongBench \(46\.05 vs\. 46\.01\), but is worse on both LongBench\-v2 \(0\.247 vs\. 0\.254\) and RULER \(78\.28 vs\. 79\.43\)\. We therefore keepα=0\.80\\alpha=0\.80as the default: it is a selected operating point for the studied setting, not a tuned optimum\.

LongBench\-v2 provides a useful boundary case\. Comparingα=0\.30\\alpha=0\.30against higher smoothing, Qwen2\.5\-7B loses more examples than it gains: moving toα=0\.80\\alpha=0\.80loses 13 previously correct examples and recovers 7, while moving toα=0\.90\\alpha=0\.90loses 17 and recovers 7\. Domain\-level breakdowns are mixed, but Qwen shows broad declines in long structured data understanding, long\-dialogue history understanding, multi\-document QA, and single\-document QA\. This demonstrates the lag side of the tradeoff in the evaluated Qwen setting\.

### G\.2Cumulative vs\. EMA

Because Eq\.[4](https://arxiv.org/html/2609.03515#S3.E4)is a bounded\-memory filter, a natural alternative is the cumulative rule in Eq\.[3](https://arxiv.org/html/2609.03515#S3.E3), which never forgets past evidence\. Table[12](https://arxiv.org/html/2609.03515#A7.T12)tests whether the non\-decaying endpoint dominates the bounded\-memory endpoint once both are placed inside the same decode\-time budgeted framework\.

Table 12:Delta of H2O\-style cumulative attention over EMA InertiaKV at 90% compression\. Cumulative attention is competitive but does not dominate\.The result is not one\-sided\. On Llama, cumulative attention is slightly better on LongBench and RULER but worse on LongBench\-v2\. On Qwen2\.5\-7B, it drops substantially on RULER \(−7\.66\-7\.66\)\. Together with theα=0\\alpha=0rows in Table[5](https://arxiv.org/html/2609.03515#S5.T5), this completes the family picture: single\-step scoring is too brittle for retrieval; cumulative accumulation can outperform EMA when evidence persists but degrades when stale retention matters\. EMA is therefore not universally best, but it is the most robust bounded\-memory compromise\.

## Appendix HAdaptiveα\\alpha: A Negative Result

The alpha sensitivity in Table[5](https://arxiv.org/html/2609.03515#S5.T5)raises a natural question: canα\\alphabe selected automatically? We tested a theory\-derived adaptive selector that estimates noise \(σ\\sigma\) and drift \(δ\\delta\) during a warmup period, then selectsα\\alphato minimize the boundary\-error–lag tradeoff from Eq\.[5](https://arxiv.org/html/2609.03515#S3.E5)\.

The result is negative\. Across both backbones, the selector almost always choosesα=0\.8\\alpha=0\.8orα=0\.95\\alpha=0\.95, never selecting lowα\\alpha\. On RULER, this yields similar or slightly worse performance than the fixedα=0\.8\\alpha=0\.8default; on LongBench\-v2, where lowerα\\alphahelps, the selector cannot adapt\. The failure mechanism is interpretable: cross\-layer Jaccard disagreement \(σ≈0\.07\\sigma\\approx 0\.07\) is stable, but temporal drift \(δ≈0\.0003\\delta\\approx 0\.0003–0\.0010\.001\) is extremely small during the first 10 decode steps, producing noise\-to\-drift ratiosη=σ/δ\\eta=\\sigma/\\deltain the range 50–600 that the theory formula maps to highα\\alpha\. Relevance shifts that would benefit from lowα\\alphaoccur*after*the warmup window\.

This supports retainingα=0\.8\\alpha=0\.8as a pragmatic default for the evaluated settings and identifies the failure mode a future adaptive policy must address: online detection of relevance shifts during generation, not just noise/drift estimation from initial steps\. A reactive strategy—triggering lowerα\\alphawhen raw/EMA ranking disagreement exceeds a threshold—is also unreliable: rank\-trace diagnostics \(Table[13](https://arxiv.org/html/2609.03515#A9.T13)\) show that disagreement correlates with alpha\-sensitivity \(0\.14–0\.15\) but does not reliably identify the correct direction\.

## Appendix IMechanistic Diagnostics and Failure Anatomy

The preceding appendices characterize*what*EMA does at the benchmark level; this appendix probes possible mechanisms by shifting from family\-level averages to mechanism\-level observables\. We trace top\-kktoken sets during decoding on 100 examples per trace set, reporting raw top\-kkchurn, EMA top\-kkchurn, raw/EMA Jaccard overlap, the fraction of raw top\-kktokens evicted, and trace accuracy \(Table[13](https://arxiv.org/html/2609.03515#A9.T13)\)\.

We use churn as a proxy for the boundary\-variance term in Eq\.[5](https://arxiv.org/html/2609.03515#S3.E5): high churn indicates score fluctuations can flip keep/drop decisions\. Raw/EMA Jaccard and raw\-top evicted rate proxy for lag: divergence means the aggregated estimate no longer tracks the current query\-aware signal\.

Table 13:Rank\-trace diagnostics at 90% compression on Llama\-3\.1\-8B\. EMA top\-kkchurn is consistently lower than raw churn\. On LongBench\-v2, stronger smoothing diverges from raw attention without improving accuracy; on RULER, it improves accuracy\.EMA top\-kkchurn is far lower than raw churn, showing that temporal accumulation stabilizes rankings in these traces\. On LongBench\-v2, increasingα\\alphalowers raw/EMA agreement without improving traced accuracy \(consistent with reduced adaptivity\)\. On RULER, stronger smoothing improves traced accuracy \(consistent with persistent retrieval evidence\)\.

#### Failure anatomy\.

Across the full 503\-example LongBench\-v2 split, 97\.6% of Llama\-3\.1\-8B examples and 95\.0% of Qwen2\.5\-7B examples are invariant across temporal rules\. The sensitive subset is small: on Qwen, 17/503 examples are adaptation\-lag candidates \(lower smoothing correct\) versus 7/503 stability\-help cases\. On Llama, the counts are 3/503 and 1/503\. LongBench\-v2 thus exposes the lag side on a minority of examples rather than indicating that EMA is broadly harmful\.

A larger 300\-example Qwen pilot shows that raw/EMA ranking disagreement \(correlation 0\.14\) and raw\-top\-kkeviction rate \(0\.15\) are more informative observables for alpha\-sensitivity than boundary margin \(0\.03\) or score entropy \(0\.07\)\. However, they identify sensitivity better than the correct*direction*: both low\-α\\alpha\-wins and high\-α\\alpha\-wins appear in elevated\-disagreement bins\.

## Appendix JSupplementary Result Details

This appendix collects detailed breakdowns and efficiency measurements omitted from the main text for space\.

#### Learned scorer degradation by sequence length\.

The learned bilinear scorer’s failure worsens with sequence length\. On Llama\-3\.1\-8B RULER, the combined deviation isΔ=−38\.0\\Delta\{=\}\{\-\}38\.0at 4k andΔ=−52\.2\\Delta\{=\}\{\-\}52\.2at 16k, with needle\-in\-a\-haystack retrieval tasks collapsing from∼100\{\\sim\}100to single digits \(e\.g\.,niah\_multiquerydrops from 99\.2 to 3\.4 at 16k\)\. This length dependence suggests that longer sequences amplify the cost of a misaligned utility ranking, since more tokens compete for limited budget slots\.

#### Lazy refresh tradeoff visualization\.

Figure[10](https://arxiv.org/html/2609.03515#A10.F10)shows the wall\-clock decode speedup of lazy refresh variants\.

Figure 10:Lazy refresh decode speedup vs\. full\-refresh InertiaKV at 90% compression and 64k context\. InertiaKV\-Lazy4 yields1\.34×1\.34\\times\(Llama\) and1\.46×1\.46\\times\(Qwen\) wall\-clock speedup\. For Lazy4, benchmark deltas are below 0\.05 except for the small Llama RULER regression \(Δ=−0\.151\\Delta\{=\}\{\-\}0\.151; Table[14](https://arxiv.org/html/2609.03515#A10.T14)\)\.
#### Lazy4 per\-benchmark quality deltas and confidence intervals\.

Table[14](https://arxiv.org/html/2609.03515#A10.T14)reports the full per\-benchmark breakdown for InertiaKV\-Lazy4 versus full refresh\.

Table 14:InertiaKV\-Lazy4 quality delta versus full refresh at 90% compression\. LongBench\-v2 has a single aggregate score, so no CI is computed\. Both LongBench intervals and the Qwen RULER interval include zero; Llama RULER shows a small negative delta whose interval excludes zero\.Figure 11:Decode throughput scaling at 90% compression\. Each panel shows absolute throughput \(tok/s\) for Full Cache, Full Refresh InertiaKV, and InertiaKV\-Lazy4 at 16k, 32k, and 64k context\. Annotations show InertiaKV\-Lazy4 speedup over Full Refresh, which remains stable at 1\.36–1\.44×\\timesacross context lengths and models\.
#### Throughput scaling across context lengths\.

Figure[11](https://arxiv.org/html/2609.03515#A10.F11)reports absolute decode throughput and Lazy4 speedup across 16k, 32k, and 64k contexts\. Full Refresh throughput decreases with context length due to growing attention cost; Lazy4 maintains a 1\.36–1\.44×\\timesspeedup at each tested length, indicating that the measured gain is not confined to one context length\.

#### Hook\-level profiling: scoring bottleneck\.

Hook\-level profiling on Llama\-3\.1\-8B at 64k context shows that the largest measured per\-step cost is per\-layer query\-aware score reconstruction \(specifically, attention fallback computation\), not cache remapping or eviction logic\. This motivates the lazy refresh family: it targets the largest measured cost component while leaving the retention budget and utility estimator unchanged\.

## Appendix KReproducibility

Reported result directories were archived and re\-analyzed offline with duplicate metric identities treated as errors\. For the central InertiaKV\-Lazy4 comparison, we also reran full\-refresh InertiaKV and InertiaKV\-Lazy4 from fresh roots on both main backbones: eight LongBench tasks per backbone \(16 model–task combinations\), LongBench\-v2 0\-shot, and both reported RULER lengths\. These clean\-root reruns reproduce the task\-level values in Figure[10](https://arxiv.org/html/2609.03515#A10.F10), which we use as the canonical lazy\-refresh quality result\. Other experiments include architecture, scale, MoE, retrieval\-grid, compression\-sweep, and efficiency stress checks\.