VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

# VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
Source: [https://arxiv.org/html/2607.15498](https://arxiv.org/html/2607.15498)
###### Abstract

The key–value \(KV\) cache is the main memory bottleneck in long\-context large language model \(LLM\) inference\. Two leading training\-free families are both structurally limited: token\-*selection*methods \(SnapKV, Ada\-KV\) score importance from an observation window and*evict*low\-scoring tokens, but eviction is irreversible—so when the importance signal degrades under query\-agnostic reuse, accuracy collapses by 11–15 points; uniform*low\-rank*coding keeps every token but spends equal rank everywhere, wasting budget\. We observe that both failures share one cure: rank should be*allocated*, not evicted\. We presentVarRate, a training\-free KV codec that assigns each token a variable low\-rank budget by its query salience, keeping every token at a nonzero rank\. Comparable adaptive\-rank codecs reach this allocation only through training; VarRate requires none\. Because no token is dropped, it degrades by only 3\.5–5\.5 points where query\-aware selection collapses\. At a matched 20% budget on LongBench \(16 tasks\), VarRate stays within 0\.8 points of the uncompressed model on both Llama\-3\.1\-8B and Qwen2\.5\-7B\. Averaged over the two, it is the strongest matched\-memory compressor\. It significantly beats its uniform\-rank ablation on both models\. Against KVzip, a method purpose\-built for query\-agnostic reuse, it is accuracy\-equivalent in three of four settings and within a point overall, at about one\-eighth the prefill overhead\.

![Refer to caption](https://arxiv.org/html/2607.15498v1/x1.png)Figure 1:VarRate allocates rank by salience—it neither evicts tokens nor spreads rank uniformly\.Each token is a bar whose height is its allocated rank, given below it\.\(a\)Token selection evicts the low\-scoring tokens; the dropped KV is gone for good, so a mistaken decision cannot be undone\.\(b\)Uniform low\-rank coding keeps every token but spends the same rank on all of them—wasted on the easy tokens, starved on the critical ones\.\(c\)The uncompressed cache: every token at full rank, the accuracy target\.\(d\)VarRate varies rank with salience and holds every token above a nonzero floor \(rmin=16r\_\{\\min\}\\\!=\\\!16\), reaching the target at \(a\) and \(b\)’s budget\. Panels \(a\), \(b\), \(d\) use the same KV budget; \(c\) is uncompressed\. Ranks are schematic\.## 1Introduction

Large language models \(LLMs\) increasingly operate over long contexts—multi\-document question answering \(QA\), repository\-scale code, and book\-length summarization\(Bai et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib1)\)\. What makes autoregressive inference over such inputs practical is the key–value \(KV\) cache, which stores the attention keys and values\(Vaswani et al\.[2017](https://arxiv.org/html/2607.15498#bib.bib21)\)of past tokens so they need not be recomputed at each step\. The cache, however, grows linearly with context length, and at long context it—not the model’s weights—comes to dominate GPU memory at decode time\(Xiao et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib22)\)\. Compressing the KV cache while preserving accuracy has therefore become a central problem for efficient LLM serving\.

Two training\-free families dominate\.*Token\-selection*methods keep a fixed\-size cache and*evict*the rest \(Figure[1](https://arxiv.org/html/2607.15498#S0.F1)a\)\. The strongest of them—SnapKV\(Li et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib13)\)and the budget\-refining successors it inspired\(Cai et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib4); Feng et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib8)\)—score each token from a recent observation window, a signal that is essentially the current query; earlier variants select positionally or by cumulative attention\(Xiao et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib22); Zhang et al\.[2023](https://arxiv.org/html/2607.15498#bib.bib27)\)\. Query\-aware selection is excellent when the query is known at compression time\. But compress a document once and serve many queries against it—prefix caching, multi\-turn dialogue—and the signal no longer describes what will be asked; because eviction is irreversible, every token it misjudges is gone for good, and accuracy collapses\.*Uniform low\-rank coding*\(Chang et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib5)\)takes the opposite stance: it keeps every token but projects each onto the same reduced\-rank subspace \(Figure[1](https://arxiv.org/html/2607.15498#S0.F1)b\)\. It discards no token, yet spends an identical rank budget on trivial and on critical tokens—wasting it where it is not needed and starving it where it is\. Quantization\(Liu et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib15); Zandieh, Daliri, and Han[2025](https://arxiv.org/html/2607.15498#bib.bib24)\), a third axis, lowers numerical precision and is orthogonal to both\. The middle ground neither family occupies is the natural one:*keep every token, but give each the rank it deserves*\. Methods that do vary rank per token either train a gate for it\(Lu et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib17)\)or split it by fixed position\(Zhu et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib28)\)—never from the model’s own attention\.

A cache fails under reuse only when two conditions coincide: the importance signal has gone stale, and the method’s response to a low score is destructive\. Query\-aware selection has both\. One escape is to repair the signal—KVzip\(Kim et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib12)\)scores tokens by how well the cache reconstructs the context, a query\-agnostic criterion, and is robust accordingly—but it pays by re\-encoding the entire context with the full model, several times the cost of the prefill it compresses\. We take the other escape and remove the second condition:*rank should be allocated, not evicted*\. A graded, reversible rank budget keeps every token at some fidelity, so a stale signal merely spends the budget suboptimally—a misjudged token is coarsened, never discarded—and the signal may then stay cheap, because being wrong is no longer fatal\. We realize this asVarRate, a training\-free KV codec that gives each token a*variable*low\-rank budget in proportion to its query salience, via water\-filling over a shared low\-rank basis, holding salient tokens at high rank and the rest above a nonzero floor \(Figure[1](https://arxiv.org/html/2607.15498#S0.F1)d\)\. To our knowledge, VarRate is the first*training\-free*method to set the per\-token rank from a query\-salience signal—a graded budget, not a fixed positional split\.

This design pays off precisely where query\-aware selection is weakest\. On LongBench across two model families, at a matched 20% KV budget VarRate stays within a point of the uncompressed model and significantly outperforms its own uniform\-rank ablation on both\. Under query\-agnostic reuse, query\-aware selection loses 11–15 points while VarRate loses only 3\.5–5\.5, and it strictly dominates the published low\-rank codec Palu at every budget\. It comes within a point of KVzip’s accuracy—which remains marginally ahead on average—at roughly one\-eighth of the compression cost\. We do not claim to beat every baseline—against memory\-matched quantization VarRate ties—but it delivers the robustness of adaptive\-rank compression with none of its usual training cost\.

Contributions\.

- •We show that the two dominant training\-free families fail for complementary reasons—query\-aware selection because a stale signal meets an irreversible response, uniform coding because it spends rank without regard to salience—and that*allocating*rank by salience cures both \(Figure[1](https://arxiv.org/html/2607.15498#S0.F1)\)\.
- •We presentVarRate, the first training\-free variable\-rate low\-rank KV codec: per\-token rank set by query salience via water\-filling over a shared basis, floored so no token is dropped, with no training, fine\-tuning, or architecture change\.
- •Across two model families and 16 LongBench tasks, VarRate stays within a point of the uncompressed model at a 20% budget, significantly beats uniform\-rank coding, strictly dominates Palu, and—unlike query\-aware selection—stays robust under query\-agnostic reuse, coming within a point of the purpose\-built KVzip at∼8×\{\\sim\}8\\timeslower compression cost\.

## 2Related Work

Token selection \(eviction\)\.The dominant training\-free approach keeps a fixed\-size cache by scoring tokens and discarding the rest\. StreamingLLM\(Xiao et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib22)\)retains attention sinks and a recent window; H2O\(Zhang et al\.[2023](https://arxiv.org/html/2607.15498#bib.bib27)\)evicts by cumulative attention; SnapKV\(Li et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib13)\)scores importance from a recent observation window; and PyramidKV\(Cai et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib4)\)and Ada\-KV\(Feng et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib8)\)refine the eviction*budget*across layers and heads; SlimInfer\(Long et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib16)\)prunes tokens dynamically during the forward pass\. The accuracy these methods forfeit across long\-context tasks has been benchmarked systematically\(Yuan et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib23)\)\. In the query\-aware members—the strongest of them—two weaknesses compound: the importance signal is essentially the current query, so it goes stale once a cache is reused across queries, and eviction is*irreversible*, so the tokens that stale signal misjudges cannot be recovered\. VarRate reuses SnapKV’s salience signal, but to*allocate*rank rather than to evict—a misjudged token is coarsened to a low rank, never destroyed\.

Low\-rank KV compression\.A second family reduces the*dimension*of each token’s key and value rather than the number of tokens\. Palu\(Chang et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib5)\)and Eigen Attention\(Saxena et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib19)\)project the cache onto a fixed low\-rank subspace: they discard no token, but assign every token the same rank\. Codecs that*adapt*the rank instead do so mostly across layers and heads—MatryoshkaKV\(Lin et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib14)\)tunes this by training, whereas LoRC\(Zhang et al\.[2024a](https://arxiv.org/html/2607.15498#bib.bib25)\)sets it post\-hoc, layer by layer, without training\. Two methods vary rank*per token*: DynaKV\(Lu et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib17)\)learns a gating network by fine\-tuning, while OjaKV\(Zhu et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib28)\)is training\-free but allocates by position—full\-rank anchors and one low rank for the rest, over an online Oja\-adapted basis\. VarRate fills the remaining cell: to our knowledge it is the first*training\-free*method to set a*graded*per\-token rank from a query\-salience signal, by water\-filling over a shared basis, with every token held above a nonzero floor\.

Quantization\.An orthogonal axis lowers numerical precision instead of rank or token count\. KIVI\(Liu et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib15)\)uses per\-channel 2\-bit keys and per\-token values, while KVQuant\(Hooper et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib10)\)and QJL\(Zandieh, Daliri, and Han[2025](https://arxiv.org/html/2607.15498#bib.bib24)\)push to lower bit\-widths with reduced overhead\. Because it reduces precision rather than rank, quantization is complementary to VarRate; the two compose, which we confirm empirically\.

Query\-agnostic reuse and merging\.When a cache is compressed once and reused across many queries, a query\-dependent score fails by construction\. Two methods answer this by making the*signal*query\-agnostic: KVzip\(Kim et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib12)\)scores tokens by how well the cache reconstructs the context, and Expected Attention\(Devoto, Jeblick, and Jégou[2025](https://arxiv.org/html/2607.15498#bib.bib7)\)estimates future\-query attention in closed form\. KVzip is genuinely robust—and it*evicts*, which shows that eviction alone is not the culprit: the culprit is a stale signal met by an irreversible response\. Its price is re\-encoding the context with the full model, several times the prefill it compresses; Expected Attention is far cheaper but does not hold up in this regime in our evaluation\. VarRate removes the irreversible response instead of repairing the signal, coming within a point of KVzip’s accuracy at roughly one\-eighth of its cost\. A related line*merges*rather than drops evicted entries, as in CaM\(Zhang et al\.[2024b](https://arxiv.org/html/2607.15498#bib.bib26)\)and KeepKV\(Tian et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib20)\), avoiding hard loss but introducing attention\-distribution shifts that VarRate’s reconstruction\-based coding avoids\.

## 3VarRate

We present VarRate, a training\-free codec that compresses the KV cache by giving each token a*variable*low\-rank budget, allocated by query salience under a fixed memory budget\. The codec is applied independently at every layer with its own calibrated basis; we describe one layer and drop the layer index\. Figure[2](https://arxiv.org/html/2607.15498#S3.F2)gives the pipeline and Algorithm[1](https://arxiv.org/html/2607.15498#alg1)states it in full\.

![Refer to caption](https://arxiv.org/html/2607.15498v1/x2.png)Figure 2:VarRate, one layer\.\(1\)Keys are un\-rotated to pre\-RoPE and joined with the values into one per\-token vectorztz\_\{t\}, so a single basis serves every head\.\(2\)Strided anchors and the recent window are stored exactly; every other token is coded as a residualete\_\{t\}to the mean of itsccnearest anchors\.\(3\)SnapKV saliences^t\\hat\{s\}\_\{t\}is water\-filled into a per\-token rankrtr\_\{t\}at the flat codec’s budget\. Each bar*is*the allocation: a grey floorrminr\_\{\\min\}that every token keeps, plus what salience buys above it\. Because∑t∈𝒞rt=B\\sum\_\{t\\in\\mathcal\{C\}\}r\_\{t\}=B, the bar area above the uniform\-rank line equals the area below it—VarRate spends exactly the flat codec’s budget, only redistributed\.\(4\)A single projection onto the shared basis is truncated per token to its leadingrtr\_\{t\}coefficients; every column retains at leastrminr\_\{\\min\}, so no token is dropped\. The reconstructed keys are finally re\-rotated to post\-RoPE\. Ranks are schematic\.### 3\.1Preliminaries and Codec Setup

Attention and the KV cache\.Each layer hasHHquery heads andG≤HG\\leq Hkey–value heads of widthdhd\_\{h\}\(grouped\-query attention, GQA\)\. In headhh, a queryqqattends to the cached keys and values\{\(kth,vth\)\}\\\{\(k^\{h\}\_\{t\},v^\{h\}\_\{t\}\)\\\}of the tokens seen so far and returns∑tαq,th​vth\\sum\_\{t\}\\alpha^\{h\}\_\{q,t\}\\,v^\{h\}\_\{t\}, with attention weightsαq,th=softmaxt​\(⟨q,kth⟩/dh\)\\alpha^\{h\}\_\{q,t\}=\\mathrm\{softmax\}\_\{t\}\(\\langle q,k^\{h\}\_\{t\}\\rangle/\\sqrt\{d\_\{h\}\}\)\. Caching these keys and values avoids recomputing them at each step, but the cache grows linearly with the prompt lengthLLand comes to dominate memory at long context\. VarRate compresses it; we first fix its representation\.

Why allocate rank\.Token selection and uniform low\-rank coding are two extremes of a single choice—how much of each token to keep\. Selection keeps a few tokens in full and discards the rest; uniform coding keeps every token at one reduced rank\. Neither matches effort to importance\. A token’s value enters the attention output weighted by the attention it receives, and an error in its key perturbs that output largely through the same weight, so a coding error in a heavily attended token distorts the output far more than the same error in a diffuse one\. VarRate exploits this asymmetry: it spends the rank budget where attention concentrates, keeping query\-salient tokens near\-exact and coding the diffuse remainder coarsely\.

Joint per\-token vector\.We invert the rotary position embedding \(RoPE\), un\-rotating each post\-RoPE key to its pre\-RoPE form, and stack the pre\-RoPE keys and values of allGGheads into one per\-token vector:

zt=\[k~t;vt\]∈ℝD,D=2​G​dh,z\_\{t\}=\[\\,\\tilde\{k\}\_\{t\}\\,;\\,v\_\{t\}\\,\]\\in\\mathbb\{R\}^\{D\},\\qquad D=2Gd\_\{h\},\(1\)writingk~t\\tilde\{k\}\_\{t\}andvtv\_\{t\}for the all\-head stacks\. A single basis per layer therefore serves all heads jointly\. RoPE’s per\-position rotation inflates the numerical rank of the key cache, so coding is far more effective before it; prior low\-rank KV compression likewise operates on the pre\-RoPE representation\(Chang et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib5); DeepSeek\-AI[2024](https://arxiv.org/html/2607.15498#bib.bib6)\)\.

Residual codec\.We keep a strided set of anchor tokens𝒜=\{z1,z1\+Δ,z1\+2​Δ,…\}\\mathcal\{A\}=\\\{z\_\{1\},z\_\{1\+\\Delta\},z\_\{1\+2\\Delta\},\\dots\\\}exact and code every other token by its residual to a local anchor mean:

z¯t=1c​∑a∈𝒩c​\(t\)za,et=zt−z¯t,\\bar\{z\}\_\{t\}=\\tfrac\{1\}\{c\}\\\!\\\!\\sum\_\{a\\in\\mathcal\{N\}\_\{c\}\(t\)\}\\\!\\\!z\_\{a\},\\qquad e\_\{t\}=z\_\{t\}\-\\bar\{z\}\_\{t\},\(2\)where𝒩c​\(t\)\\mathcal\{N\}\_\{c\}\(t\)are theccanchors nearestztz\_\{t\}in Euclidean distance\. Following selection methods we also keep the most recentwwtokens exact\. Writeℰ\\mathcal\{E\}for this exact set,𝒞\\mathcal\{C\}for the coded set, andnref=\|ℰ\|n\_\{\\mathrm\{ref\}\}=\|\\mathcal\{E\}\|\.

Shared basis \(training\-free\)\.Offline, on a handful of unlabeled calibration contexts, we gather the residuals\{et\}\\\{e\_\{t\}\\\}at each layer and take the top\-RRright singular vectors by singular value decomposition \(SVD\)—a per\-layer principal component analysis \(PCA\) basis of the residuals—V=\[𝐯1;…;𝐯R\]∈ℝR×DV=\[\\mathbf\{v\}\_\{1\};\\dots;\\mathbf\{v\}\_\{R\}\]\\in\\mathbb\{R\}^\{R\\times D\}, ordered by decreasing singular value\. This needs no gradients and no fine\-tuning; VarRate runs on off\-the\-shelf models\. The basis is a model\-side, one\-time cost shared by every sequence—oneR×DR\\times Dmatrix per layer,134134MB across all layers in half precision for Llama\-3\.1\-8B—and, following the convention of prior low\-rank codecs\(Chang et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib5)\), it is not charged to the per\-sequence cache budget\.

Budget\.Letκ∈\(0,1\)\\kappa\\in\(0,1\)be the target KV budget \(the fraction of the full cache retained\)\. The exact set costsnref​Dn\_\{\\mathrm\{ref\}\}Dscalars, and a token coded at rankrrcostsrrcoefficients\. Matching total storage toκ​L​D\\kappa LDleaves a coded budget

B=κ​L​D−nref​D=D​\(κ​L−nref\),B=\\kappa LD\-n\_\{\\mathrm\{ref\}\}D=D\\,\(\\kappa L\-n\_\{\\mathrm\{ref\}\}\),\(3\)to be distributed as ranks over the coded tokens,∑t∈𝒞rt=B\\sum\_\{t\\in\\mathcal\{C\}\}r\_\{t\}=B\. Each coded token additionally stores the indices of itsccanchors, which we count in the reported footprint; they add0\.2%0\.2\\%of the cache\. A uniform\-rank codec spends the budget as a constantrt=B/\|𝒞\|r\_\{t\}=B/\|\\mathcal\{C\}\|for every token; everything to this point—the reference codec and the shared basis—is common to that baseline, and VarRate’s contribution is how it setsrtr\_\{t\}instead\.

### 3\.2Salience\-Guided Variable\-Rate Allocation

Salience\.We score each token by the attention it draws from a recent observation window—the signal introduced by SnapKV\(Li et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib13)\)\. Writing𝒲\\mathcal\{W\}for the queries of the lastwobsw\_\{\\mathrm\{obs\}\}tokens, we average the attention weightsαq,th\\alpha^\{h\}\_\{q,t\}that this window places on tokentt,

st=1\|𝒲\|​H​∑q∈𝒲∑h=1Hαq,th,t∈𝒞,s\_\{t\}=\\frac\{1\}\{\|\\mathcal\{W\}\|\\,H\}\\sum\_\{q\\in\\mathcal\{W\}\}\\ \\sum\_\{h=1\}^\{H\}\\alpha^\{h\}\_\{q,t\},\\qquad t\\in\\mathcal\{C\},\(4\)apply SnapKV’s length\-55moving average over adjacent tokens \(its clustering step\), and min–max normalize over the coded set tos^t∈\[0,1\]\\hat\{s\}\_\{t\}\\in\[0,1\]\. The window is simply the tail of the prompt: when the prompt carries the question—the standard setting—this is a sharp signal, but when a document is compressed*once*and reused across many queries, no question is present ands^\\hat\{s\}is only a stale proxy for what will be asked\. VarRate does not try to repair the signal; it makes a stale one survivable, coarsening a misjudged token to a low rank rather than deleting it\.

Water\-filling\.We allocateBBacross coded tokens in proportion to salience, above a floorrminr\_\{\\min\}and capped at the basis rankRR:

rt=clip​\(rmin\+λ​s^t,rmin,R\),t∈𝒞,r\_\{t\}=\\mathrm\{clip\}\\\!\\big\(r\_\{\\min\}\+\\lambda\\,\\hat\{s\}\_\{t\},\\;r\_\{\\min\},\\;R\\big\),\\qquad t\\in\\mathcal\{C\},\(5\)where the single multiplierλ\>0\\lambda\>0is set so that∑t∈𝒞rt=B\\sum\_\{t\\in\\mathcal\{C\}\}r\_\{t\}=B\. This is the standard water\-filling recursion \(Algorithm[1](https://arxiv.org/html/2607.15498#alg1)\): raise the “water level”λ\\lambda, clip any token that reachesRR, and redistribute its surplus, until the budget is met\.111Ranks are integers in the implementation, so realized storage exceedsBBby under0\.4%0\.4\\%\(measured\)\. The uniform\-rank codec rounds identically, so matched\-memory comparisons are unaffected\.Two properties hold*by construction*: \(i\) total storage equals the uniform\-rank codec’s at the sameκ\\kappa, so every comparison is at matched memory; and \(ii\) sincermin\>0r\_\{\\min\}\>0, every token keeps at least rankrminr\_\{\\min\}—*no token is ever dropped*, unlike eviction\. The two familiar regimes are its limits: an uninformative signal \(constants^\\hat\{s\}\) collapses Eq\.[5](https://arxiv.org/html/2607.15498#S3.E5)to the uniform\-rank codec,rt=B/\|𝒞\|r\_\{t\}=B/\|\\mathcal\{C\}\|, while eviction is the degeneratert=0r\_\{t\}=0that the floor forbids\. VarRate lies strictly between them—graded ranks, none of them zero\.

Algorithm 1VarRate compression \(one layer, prefill\)Input: post\-RoPE keys/values\{\(kt,vt\)\}t=1L\\\{\(k\_\{t\},v\_\{t\}\)\\\}\_\{t=1\}^\{L\}; basisV∈ℝR×DV\\in\\mathbb\{R\}^\{R\\times D\}; budgetκ\\kappa; floorrminr\_\{\\min\}; strideΔ\\Delta; windowsww,wobsw\_\{\\mathrm\{obs\}\} Output: compressed keys/values

1:un\-rotate keys to pre\-RoPE;

zt←\[k~t;vt\]z\_\{t\}\\leftarrow\[\\,\\tilde\{k\}\_\{t\}\\,;\\,v\_\{t\}\\,\]
2:

𝒜←\\mathcal\{A\}\\\!\\leftarrow\\\!tokens at stride

Δ\\Delta;

ℰ←𝒜∪\{last​w​tokens\}\\mathcal\{E\}\\\!\\leftarrow\\\!\\mathcal\{A\}\\cup\\\{\\text\{last \}w\\text\{ tokens\}\\\};

𝒞←\[L\]∖ℰ\\mathcal\{C\}\\\!\\leftarrow\\\!\[L\]\\setminus\\mathcal\{E\}
3:

z¯t←\\bar\{z\}\_\{t\}\\\!\\leftarrow\\\!mean of the

ccnearest anchors;

et←zt−z¯te\_\{t\}\\\!\\leftarrow\\\!z\_\{t\}\-\\bar\{z\}\_\{t\}
4:

st←s\_\{t\}\\\!\\leftarrow\\\!SnapKV score over a length\-

wobsw\_\{\\mathrm\{obs\}\}window;

s^t←\\hat\{s\}\_\{t\}\\\!\\leftarrow\\\!min–max over

𝒞\\mathcal\{C\}
5:

B←D​\(κ​L−\|ℰ\|\)B\\leftarrow D\(\\kappa L\-\|\\mathcal\{E\}\|\)
6:

rt←rmin​\(t∈𝒞\)r\_\{t\}\\leftarrow r\_\{\\min\}\\ \(t\\in\\mathcal\{C\}\);

𝒰←𝒞\\mathcal\{U\}\\leftarrow\\mathcal\{C\}\{water\-filling\}

7:repeat

8:

δ←B−∑u∈𝒞ru\\delta\\leftarrow B\-\\sum\_\{u\\in\\mathcal\{C\}\}r\_\{u\}
9:

rt←rt\+δ​s^t/∑u∈𝒰s^u\(t∈𝒰\)r\_\{t\}\\leftarrow r\_\{t\}\+\\delta\\,\\hat\{s\}\_\{t\}\\big/\\\!\\sum\_\{u\\in\\mathcal\{U\}\}\\hat\{s\}\_\{u\}\\ \\ \(t\\in\\mathcal\{U\}\)
10:

𝒮←\{t∈𝒰:rt\>R\}\\mathcal\{S\}\\leftarrow\\\{t\\in\\mathcal\{U\}:r\_\{t\}\>R\\\};

rt←R​\(t∈𝒮\)r\_\{t\}\\leftarrow R\\ \(t\\in\\mathcal\{S\}\);

𝒰←𝒰∖𝒮\\mathcal\{U\}\\leftarrow\\mathcal\{U\}\\setminus\\mathcal\{S\}
11:until

𝒮=∅\\mathcal\{S\}=\\emptyset
12:

𝐜t←V​et\\mathbf\{c\}\_\{t\}\\\!\\leftarrow\\\!Ve\_\{t\};

z^t←z¯t\+V⊤​\(𝟏​\[j≤rt\]⊙𝐜t\)​\(t∈𝒞\)\\hat\{z\}\_\{t\}\\\!\\leftarrow\\\!\\bar\{z\}\_\{t\}\+V^\{\\\!\\top\}\(\\mathbf\{1\}\[j\\\!\\leq\\\!r\_\{t\}\]\\odot\\mathbf\{c\}\_\{t\}\)\\ \(t\\in\\mathcal\{C\}\)
13:

z^t←zt​\(t∈ℰ\)\\hat\{z\}\_\{t\}\\leftarrow z\_\{t\}\\ \(t\\in\\mathcal\{E\}\)
14:split

z^t\\hat\{z\}\_\{t\}into key and value; re\-rotate the key to post\-RoPE

15:returncompressed keys/values

### 3\.3Reconstruction and Complexity

Each coded token is projected once onto the shared basis, then*truncated*to its rank\. With coefficients𝐜t=V​et∈ℝR\\mathbf\{c\}\_\{t\}=Ve\_\{t\}\\in\\mathbb\{R\}^\{R\}and a nested maskmt,j=𝟏​\[j≤rt\]m\_\{t,j\}=\\mathbf\{1\}\[\\,j\\leq r\_\{t\}\\,\],

z^t=z¯t\+∑j≤rtct,j​𝐯j=z¯t\+V⊤​\(mt⊙𝐜t\),t∈𝒞,\\hat\{z\}\_\{t\}=\\bar\{z\}\_\{t\}\+\\\!\\\!\\sum\_\{j\\leq r\_\{t\}\}\\\!\\\!c\_\{t,j\}\\,\\mathbf\{v\}\_\{j\}=\\bar\{z\}\_\{t\}\+V^\{\\\!\\top\}\(m\_\{t\}\\odot\\mathbf\{c\}\_\{t\}\),\\quad t\\in\\mathcal\{C\},\(6\)andz^t=zt\\hat\{z\}\_\{t\}=z\_\{t\}fort∈ℰt\\in\\mathcal\{E\}\. BecauseVVis ordered by singular value, its leading directions carry the most residual energy on the calibration distribution; and becauseVVis orthonormal, enlargingrtr\_\{t\}monotonically reduces the reconstruction error of the*same*projection—a nested, Matryoshka\-style truncation obtained without any training\. Finallyz^t\\hat\{z\}\_\{t\}is split into key and value and the key is re\-rotated to post\-RoPE\.

Cost\.The projectionV​etVe\_\{t\}is computed once per token at the full rankRRand then masked, so the variable rate adds no arithmetic over a fixed\-rank codec; its only overhead is keeping allRRrows of the basis resident, where a flat code needs only therrit uses\. Beyond this the codec adds, per layer, a nearest\-anchor search inO​\(L2​dh​G/Δ\)O\(L^\{2\}d\_\{h\}G/\\Delta\)and two rank\-RRprojections inO​\(L​D​R\)O\(LDR\), all inside the single prefill pass the model already runs—it never re\-encodes the context to score tokens\. Both are far smaller than the prefill’s own attention, and Section[4](https://arxiv.org/html/2607.15498#S4)measures the resulting overhead\.

Composability\.The coefficients𝐜t\\mathbf\{c\}\_\{t\}are themselves quantizable; standard group\-wisebb\-bit quantization of them composes VarRate with the orthogonal quantization axis, leaving the allocation unchanged\.

## 4Experiments

### 4\.1Setup

Models and benchmark\.We evaluate on Llama\-3\.1\-8B\-Instruct and Qwen2\.5\-7B\-Instruct, and cross\-check on Mistral\-7B\-Instruct\-v0\.2\. All results are on LongBench\(Bai et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib1)\): 16 tasks under their official metrics, spanning single\- and multi\-document QA, summarization, few\-shot learning, synthetic retrieval and code completion, withn=200n\{=\}200examples per task and greedy decoding\. Confidence intervals are taken over examples, so no seed averaging is required\. Comparisons between methods are paired over the 16 tasks; we report a 20,000\-sample paired bootstrap 95% confidence interval \(CI\) and an exact sign test, and count a claim as supported only when the CI excludes zero\.

Implementation\.VarRate is implemented in KVPress\. Unless stated otherwise we compress to a KV budget ofκ=0\.20\\kappa\{=\}0\.20\(a5×5\\timesreduction\), with floorrmin=16r\_\{\\min\}\{=\}16, basis rankR=1024R\{=\}1024, strideΔ=16\\Delta\{=\}16,c=4c\{=\}4nearest anchors, and windowsw=wobs=64w\{=\}w\_\{\\mathrm\{obs\}\}\{=\}64\. Each layer’s basis is calibrated once, offline, from six unlabeled LongBench contexts—no gradients, no labels, no fine\-tuning—and reused for every task\. Accuracy is measured on a single A100\.

Baselines\.We compare against the uncompressed cache and against one compression axis at a time: token selection—SnapKV\(Li et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib13)\), PyramidKV\(Cai et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib4)\), and Ada\-KV\(Feng et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib8)\); low\-rank coding—Palu\(Chang et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib5)\)and our own uniform\-rank ablation, “flat”; quantization—KIVI\(Liu et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib15)\)at 2 and 4 bits; and the two methods built for query\-agnostic reuse, KVzip\(Kim et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib12)\)and Expected Attention\(Devoto, Jeblick, and Jégou[2025](https://arxiv.org/html/2607.15498#bib.bib7)\)\. All run from one KVPress installation at a matched budget, and every footprint we report is*measured*rather than nominal \(Table[3](https://arxiv.org/html/2607.15498#S4.T3)\)\.

Table 1:LongBench \(16 tasks\) at a matched20%20\\%KV budget \(5×5\\timescompression\)\. Bold marks the best*matched\-memory*compressor on each model\.†KIVI\-4 stores31%31\\%of the cache \(1\.55×1\.55\\timesthe others\) and is therefore not a matched comparison\. Two\-model means at matched memory:VarRate 46\.85, SnapKV 46\.02, Ada\-KV 45\.91, KIVI\-2 45\.82, PyramidKV 45\.46, flat 36\.99\.![Refer to caption](https://arxiv.org/html/2607.15498v1/x3.png)Figure 3:Budget sweep against the published low\-rank codec \(Llama\-3\.1\-8B\)\. Palu was built for the keep\-5050–70%70\\%regime: in ours it is at essentially zero, and only becomes viable at keep\-50%50\\%\. The sweep also shows, honestly, where VarRate’s margin over uniform rank lives—largest where the budget is tightest, closing as it loosens\.
### 4\.2Accuracy at a Matched Budget

At5×5\\timescompression VarRate lands within0\.30\.3points of the uncompressed ceiling on Llama and0\.80\.8on Qwen \(Table[1](https://arxiv.org/html/2607.15498#S4.T1)\), and it has the best two\-model mean of any matched\-memory compressor\.

The per\-model picture is more nuanced than that average\. On Llama VarRate is*fourth of seven*compressors: PyramidKV \(46\.0246\.02\) and SnapKV \(45\.7945\.79\) sit nominally above it\. Paired over the 16 tasks, however, those gaps are ties \(−0\.33\-0\.33and−0\.10\-0\.10; both CIs straddle zero\)\. On Qwen the ordering reverses and the gaps become real: VarRate is first, beating PyramidKV by\+3\.10\+3\.10\(p=0\.001p\{=\}0\.001\) and SnapKV by\+1\.75\+1\.75\(p=0\.022p\{=\}0\.022\)\. Against memory\-matched quantization it*ties*KIVI\-2 on both models \(\+0\.86\+0\.86and\+1\.18\+1\.18, both n\.s\.\)—not a win\. KIVI\-4 does edge it on Qwen, but stores31%31\\%of the cache,1\.55×1\.55\\timeswhat every other method spends\. The defensible cross\-model claim is therefore*consistency*: VarRate is never significantly worse than any matched\-memory baseline on either model, and significantly better than several on Qwen\.

What the allocation buys\.The comparison that isolates our contribution is against our own uniform\-rank ablation, and it is the strongest result we have:\+2\.22\+2\.22on Llama \(p=0\.021p\{=\}0\.021, winning 13 of 16 tasks\) and\+17\.50\+17\.50on Qwen \(p=0\.001p\{=\}0\.001, 15 of 16\)\. Qwen’s four\-KV\-head cache is especially hard for uniform low\-rank coding—flat collapses to30\.5030\.50, losing two thirds of its multi\-document QA score—and spending the same budget by salience recovers almost all of it\.

Against the published low\-rank codec\.Palu was designed for the keep\-5050–70%70\\%regime, whereas VarRate operates at keep\-1515–25%25\\%; sweeping the budget \(Figure[3](https://arxiv.org/html/2607.15498#S4.F3)\) makes the gap stark\. At a20%20\\%budget Palu scores0\.40\.4on passage retrieval,0\.70\.7on MuSiQue and3\.43\.4on GovReport, against VarRate’s99\.599\.5,32\.032\.0and30\.430\.4; only by keep\-50%50\\%does it become viable\. Palu’s released checkpoint stores20\.41%20\.41\\%of the cache against our measured20\.22%20\.22\\%, so the comparison is memory\-matched\. The same sweep shows where VarRate’s margin over uniform rank lives: it is largest exactly where the budget is tightest, and closes as the budget loosens\.

Table 2:Query\-agnostic reuse: the document is compressed once, with the question absent from the compressed context\. Three\-task mean \(passage / MuSiQue / Qasper\)\. KVzip is given the same 64\-token pin as VarRate\. The uncompressed control barely moves, so the collapses are caused by the compression\.
### 4\.3Query\-Agnostic Reuse

Compressing a document once and serving many queries against it removes the question from the compressed context, and with it the signal every query\-aware method depends on\. The uncompressed control barely moves \(\+0\.08\+0\.08and\+0\.55\+0\.55\), so what follows is caused by the compression and not by the change of prompt\.

Selection collapses: SnapKV loses12\.9612\.96and14\.6614\.66points and Ada\-KV11\.0111\.01and15\.2815\.28\(Table[2](https://arxiv.org/html/2607.15498#S4.T2)\)\. Expected Attention, though built to be query\-agnostic, collapses hardest of all \(−21\.68\-21\.68\)—its estimate of the future query is drawn from a prefill that ordinarily contains the question\. VarRate degrades gracefully instead, by3\.533\.53and5\.505\.50\. Allocation rather than eviction is the reason: a token that the stale signal misjudges is coarsened to a low rank, never removed\.

A correction we owe KVzip\.When the query is known at compression time it is appended*into*the compressed context\. SnapKV pins that region so it can never be evicted, and VarRate inherits the pin because it scores with SnapKV; stock KVzip protects only its first four tokens, and can therefore evict the very question it is about to be asked\. We grant KVzip the identical 64\-token pin\. Its budget is unchanged by construction—the pin decides*which*entries are dropped, never*how many*—and it lifts KVzip by\+0\.89\+0\.89on Llama and\+3\.12\+3\.12on Qwen, erasing an apparent lead of ours\. Every KVzip number below is the pinned, stronger one\.

Against KVzip, the method built for this regime, the outcome is genuinely mixed\. Given the same pin, the two are accuracy\-equivalent in three of the four model×\\timesregime cells; the fourth—Qwen under reuse—goes to*KVzip*by4\.634\.63points, and KVzip leads the four\-cell mean by0\.770\.77\.We claim no accuracy advantage over KVzip anywhere\.What we claim is that VarRate reaches comparable robustness by keeping a cheap query\-aware signal and making it survivable, rather than by paying to replace it\.

Table 3:Measured cost, one instrument \(Llama\-3\.1\-8B, A100, 16k context\)\. KV is the measured cache footprint, prefill the overhead over the uncompressed model, decode in tok/s\.‡The maskers’ decode rate is a harness artifact—KVPress simulates eviction rather than physically pruning—not a property of those methods\.![Refer to caption](https://arxiv.org/html/2607.15498v1/x4.png)Figure 4:Accuracy under query\-agnostic reuse against compression cost\. Both axes are measured and the cost axis is logarithmic\. The dashed guide traces the trade\-off frontier: VarRate lies on it, coming within1\.91\.9points of KVzip at roughly one\-eighth of the prefill overhead\. Uniform rank and Expected Attention are*dominated*—cheaper methods reach higher accuracy than either\.
### 4\.4Cost

VarRate’s prefill overhead is\+44\.0%\+44\.0\\%\(Table[3](https://arxiv.org/html/2607.15498#S4.T3)\), against\+2\.0%\+2\.0\\%for selection and\+24\.4%\+24\.4\\%for the flat codec\. Allocation is not free\. The underlying*arithmetic*, however, is nearly free: the codec’s entire floating\-point operation \(FLOP\) count is2\.61%2\.61\\%of one forward pass, so what we measure is unfusedfp32implementation rather than a floor—a bf16 variant cuts the overhead by40%40\\%with accuracy unchanged \(max\|Δ\|=0\.07\|\\Delta\|\{=\}0\.07\)\. The contrast with KVzip is of a different kind: its\+372\.2%\+372\.2\\%is*structural*, since it re\-encodes the whole context with the full model in order to score it\. Figure[4](https://arxiv.org/html/2607.15498#S4.F4)places the two\. VarRate comes within1\.91\.9points of KVzip’s query\-agnostic accuracy at roughly one\-eighth of its cost, while the methods that are genuinely cheap sit at least ten points below both\.

Decoding speed is unchanged \(37\.437\.4tok/s, against the uncompressed37\.537\.5\), because the codec reconstructs a full\-length cache; realizing the saving at run time requires a fused kernel, as Palu’s does\. We therefore measure accuracy in the standard reconstruct\-in\-place mode and cost on a separate instrument, following the convention of this literature\.

Table 4:Both halves of the method matter \(Llama,20%20\\%budget\)\. The*signal*buys retrieval \(87\.5→99\.587\.5\\\!\\rightarrow\\\!99\.5\); the*graded*allocation is what preserves summarization—a binary top\-KKsplit ties on retrieval but falls to29\.029\.0, below even uniform rank\.
### 4\.5Ablations

The signal, and the shape of the allocation\.Two things could be doing the work: the salience signal, or the graded way the budget is spent\. Both do \(Table[4](https://arxiv.org/html/2607.15498#S4.T4)\)\. Replacing SnapKV salience with a random signal costs6\.56\.5points on retrieval \(99\.5→93\.099\.5\\\!\\rightarrow\\\!93\.0\), and uniform rank costs1212\(87\.587\.5\)—the signal matters\. Replacing water\-filling with a binary top\-KKsplit, winners atRRand everyone else at the floor, ties on retrieval \(99\.599\.5\) but*loses*on summarization \(29\.029\.0, below even flat’s30\.130\.1\): starving the unselected tokens to the floor destroys exactly the distributed context that low\-rank coding is good at\. The graded allocation, not merely the signal, is what lets VarRate hold coding’s summarization strength while gaining selection’s retrieval strength\.

How good is the signal?On passage retrieval, an oracle that allocates rank by the gold answer span reaches100\.0100\.0and attention salience reaches99\.099\.0, against93\.093\.0for uniform rank \(a separaten=100n\{=\}100harness\)\. The signal we can actually compute is within a point of optimal\.

Composability\.Quantizing the coefficients to 3 bits leaves accuracy essentially unchanged \(99\.5/31\.3/30\.499\.5/31\.3/30\.4against99\.5/32\.0/30\.499\.5/32\.0/30\.4on passage/MuSiQue/GovReport\) at9%9\\%of the cache: low\-rank coding and quantization compose, as the method predicts\.

Calibration, and a third model\.Rebuilding the basis from different examples, or from different tasks, moves every result by at most1\.81\.8points, inside the confidence intervals—the codec is not tuned to its calibration set\. The pattern also carries to a third family: on Mistral\-7B, uniform rank collapses on retrieval \(29\.129\.1at a15%15\\%budget\) while VarRate holds73\.673\.6, within two points of the uncompressed75\.675\.6\.

Robustness checks \(signal, floor, budget, long context\)\.A further set of controls, provided in the supplementary material, confirms the signal is content\-driven rather than merely graded, replicates both the signal and the allocation choice on a second model, shows the rank floor is slack rather than a lever, states VarRate’s viable operating range at aggressive budgets, and extends the accuracy result to128128K context\.

## 5Conclusion

We asked what a KV codec should do when its importance signal is wrong\. Token selection answers by deleting, and pays for it: under cache reuse, where the signal goes stale, it loses1111–1515points\. VarRate answers by*coarsening*instead—allocating a variable low\-rank budget by query salience over a shared basis, holding every token above a nonzero floor, and requiring no training at all\. The allocation is what carries the result: it beats its own uniform\-rank ablation by2\.222\.22and17\.5017\.50points on two model families, and the signal it uses lands within a point of an oracle\. Against KVzip, a method purpose\-built for the reuse regime, VarRate is accuracy\-equivalent in three of four settings and behind in the fourth—but it gets there at roughly one\-eighth of the cost\. Robustness under reuse, we conclude, can be bought by making a cheap signal survivable rather than by paying to replace it\.

## Acknowledgments

This work used NCSA Delta GPU at NCSA through allocation CIS240646 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program\(Boerner et al\.[2023](https://arxiv.org/html/2607.15498#bib.bib3)\), which is supported by National Science Foundation grants \#2138259, \#2138286, \#2138307, \#2137603, and \#2138296\.

## References

- Bai et al\. \(2024\)Bai, Y\.; Lv, X\.; Zhang, J\.; Lyu, H\.; Tang, J\.; Huang, Z\.; Du, Z\.; Liu, X\.; Zeng, A\.; Hou, L\.; Li, J\.; and Tang, J\. 2024\.LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\)*\.
- Bhatnagar et al\. \(2026\)Bhatnagar, P\.; Moradifirouzabadi, A\.; Yang, S\.\-H\.; Lee, S\.; Choi, J\.; and Kang, M\. 2026\.STAR\-KV: Low\-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Control\.In*International Conference on Machine Learning \(ICML\)*\.
- Boerner et al\. \(2023\)Boerner, T\. J\.; Deems, S\.; Furlani, T\. R\.; Knuth, S\. L\.; and Towns, J\. 2023\.ACCESS: Advancing Innovation: NSF’s Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support\.In*Practice and Experience in Advanced Research Computing \(PEARC\)*, 173–176\.
- Cai et al\. \(2025\)Cai, Z\.; Zhang, Y\.; Gao, B\.; Liu, Y\.; Li, Y\.; Liu, T\.; Lu, K\.; Xiong, W\.; Dong, Y\.; Hu, J\.; and Xiao, W\. 2025\.PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling\.In*Conference on Language Modeling \(COLM\)*\.
- Chang et al\. \(2025\)Chang, C\.\-C\.; Lin, W\.\-C\.; Lin, C\.\-Y\.; Chen, C\.\-Y\.; Hu, Y\.\-F\.; Wang, P\.\-S\.; Huang, N\.\-C\.; Ceze, L\.; Abdelfattah, M\. S\.; and Wu, K\.\-C\. 2025\.Palu: Compressing KV\-Cache with Low\-Rank Projection\.In*International Conference on Learning Representations \(ICLR\)*\.
- DeepSeek\-AI \(2024\)DeepSeek\-AI\. 2024\.DeepSeek\-V2: A Strong, Economical, and Efficient Mixture\-of\-Experts Language Model\.*arXiv preprint arXiv:2405\.04434*\.
- Devoto, Jeblick, and Jégou \(2025\)Devoto, A\.; Jeblick, M\.; and Jégou, S\. 2025\.Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution\.*arXiv preprint arXiv:2510\.00636*\.
- Feng et al\. \(2025\)Feng, Y\.; Lv, J\.; Cao, Y\.; Xie, X\.; and Zhou, S\. K\. 2025\.Ada\-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Hao et al\. \(2026\)Hao, J\.; Huang, Q\.; Wang, Y\.; Zhang, M\.; and Yu, J\. 2026\.DeltaKV: Residual\-Based KV Cache Compression via Long\-Range Similarity\.*arXiv preprint arXiv:2602\.08005*\.
- Hooper et al\. \(2024\)Hooper, C\.; Kim, S\.; Mohammadzadeh, H\.; Mahoney, M\. W\.; Shao, Y\. S\.; Keutzer, K\.; and Gholami, A\. 2024\.KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Hsieh et al\. \(2024\)Hsieh, C\.\-P\.; Sun, S\.; Kriman, S\.; Acharya, S\.; Rekesh, D\.; Jia, F\.; Zhang, Y\.; and Ginsburg, B\. 2024\.RULER: What’s the Real Context Size of Your Long\-Context Language Models?In*Conference on Language Modeling \(COLM\)*\.
- Kim et al\. \(2025\)Kim, J\.\-H\.; Kim, J\.; Kwon, S\.; Lee, J\. W\.; Yun, S\.; and Song, H\. O\. 2025\.KVzip: Query\-Agnostic KV Cache Compression with Context Reconstruction\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Li et al\. \(2024\)Li, Y\.; Huang, Y\.; Yang, B\.; Venkitesh, B\.; Locatelli, A\.; Ye, H\.; Cai, T\.; Lewis, P\.; and Chen, D\. 2024\.SnapKV: LLM Knows What You are Looking for Before Generation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Lin et al\. \(2025\)Lin, B\.; Zeng, Z\.; Xiao, Z\.; Kou, S\.; Hou, T\.; Gao, X\.; Zhang, H\.; and Deng, Z\. 2025\.MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection\.In*International Conference on Learning Representations \(ICLR\)*\.
- Liu et al\. \(2024\)Liu, Z\.; Yuan, J\.; Jin, H\.; Zhong, S\.; Xu, Z\.; Braverman, V\.; Chen, B\.; and Hu, X\. 2024\.KIVI: A Tuning\-Free Asymmetric 2bit Quantization for KV Cache\.In*International Conference on Machine Learning \(ICML\)*\.
- Long et al\. \(2026\)Long, L\.; Yang, R\.; Huang, Y\.; Hui, D\.; Zhou, A\.; and Yang, J\. 2026\.SlimInfer: Accelerating Long\-Context LLM Inference via Dynamic Token Pruning\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*\.
- Lu et al\. \(2026\)Lu, L\.; Qiu, K\.; Zhou, J\.; Kai, J\.; Zhang, H\.; Wang, H\.; Leng, J\.; He, Z\.; and Lin, Z\. 2026\.One Size Does Not Fit All: Token\-Wise Adaptive Compression for KV Cache\.*arXiv preprint arXiv:2603\.04411*\.
- Merity et al\. \(2017\)Merity, S\.; Xiong, C\.; Bradbury, J\.; and Socher, R\. 2017\.Pointer Sentinel Mixture Models\.In*International Conference on Learning Representations \(ICLR\)*\.
- Saxena et al\. \(2024\)Saxena, U\.; Saha, G\.; Choudhary, S\.; and Roy, K\. 2024\.Eigen Attention: Attention in Low\-Rank Space for KV Cache Compression\.In*Findings of the Association for Computational Linguistics: EMNLP*\.
- Tian et al\. \(2026\)Tian, Y\.; Wang, Z\.; Peng, Y\.; Yuan, A\.; Wang, Z\.; Yi, B\.; Liu, X\.; Cui, Y\.; and Yang, T\. 2026\.KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*\.
- Vaswani et al\. \(2017\)Vaswani, A\.; Shazeer, N\.; Parmar, N\.; Uszkoreit, J\.; Jones, L\.; Gomez, A\. N\.; Kaiser, L\.; and Polosukhin, I\. 2017\.Attention is All You Need\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Xiao et al\. \(2024\)Xiao, G\.; Tian, Y\.; Chen, B\.; Han, S\.; and Lewis, M\. 2024\.Efficient Streaming Language Models with Attention Sinks\.In*International Conference on Learning Representations \(ICLR\)*\.
- Yuan et al\. \(2024\)Yuan, J\.; Liu, H\.; Zhong, S\.; Chuang, Y\.\-N\.; Li, S\.; Wang, G\.; Le, D\.; Jin, H\.; Chaudhary, V\.; Xu, Z\.; Liu, Z\.; and Hu, X\. 2024\.KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches\.In*Findings of the Association for Computational Linguistics: EMNLP*\.
- Zandieh, Daliri, and Han \(2025\)Zandieh, A\.; Daliri, M\.; and Han, I\. 2025\.QJL: 1\-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*\.
- Zhang et al\. \(2024a\)Zhang, R\.; Wang, K\.; Liu, L\.; Wang, S\.; Cheng, H\.; Zhang, C\.; and Shen, Y\. 2024a\.LoRC: Low\-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy\.*arXiv preprint arXiv:2410\.03111*\.
- Zhang et al\. \(2024b\)Zhang, Y\.; Du, Y\.; Luo, G\.; Zhong, Y\.; Zhang, Z\.; Liu, S\.; and Ji, R\. 2024b\.CaM: Cache Merging for Memory\-efficient LLMs Inference\.In*International Conference on Machine Learning \(ICML\)*\.
- Zhang et al\. \(2023\)Zhang, Z\.; Sheng, Y\.; Zhou, T\.; Chen, T\.; Zheng, L\.; Cai, R\.; Song, Z\.; Tian, Y\.; Ré, C\.; Barrett, C\.; Wang, Z\.; and Chen, B\. 2023\.H2O: Heavy\-Hitter Oracle for Efficient Generative Inference of Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Zhu et al\. \(2025\)Zhu, Y\.; Yang, D\. H\.; Amiri, M\. M\.; Murugesan, K\.; Pedapati, T\.; and Chen, P\.\-Y\. 2025\.OjaKV: Context\-Aware Online Low\-Rank KV Cache Compression with Oja’s Rule\.*arXiv preprint arXiv:2509\.21623*\.

## Appendix AAdditional Experiments and Analyses

### A\.1Perplexity at Equal Budget

Every other result in this paper is a downstream task metric \(exact match, ROUGE, F1\), which conflates the quality of the compressed representation with the model’s ability to work around a degraded one\. As an independent sanity check, we measure next\-token perplexity on WikiText\-2\(Merity et al\.[2017](https://arxiv.org/html/2607.15498#bib.bib18)\), a plain causal\-language\-modeling corpus unrelated to LongBench, with VarRate and the flat codec compressed to exactly the same storage budget \(not merely the same nominalκ\\kappa, but the same realized byte count, so the comparison cannot be won by either method rounding differently\)\. VarRate reaches6\.956\.95PPL against flat’s7\.267\.26—a0\.310\.31\-point improvement in raw predictive likelihood, not a task\-specific score\. Because perplexity has no notion of “retrieval” or “summarization,” this isolates the claim to its purest form: salience\-weighted rank allocation reconstructs the underlying key–value geometry more faithfully than uniform rank allocation, at identical cost, independent of what downstream task later reads that cache\.

### A\.2VarRate Against the Selection Family Across Budgets

Table[5](https://arxiv.org/html/2607.15498#A1.T5)sweeps SnapKV, PyramidKV, and Ada\-KV against VarRate at keep\-15/20/25%15/20/25\\%on Llama\. VarRate ties the family on retrieval and multi\-hop QA \(differences inside CIs at every budget\) and wins summarization outright, by1\.01\.0–2\.82\.8points over the best selection method—significant at keep\-20/25%20/25\\%\. This is a Pareto improvement over selection specifically: VarRate matches it where it is strong and wins where uniform attention\-window scoring is structurally weak \(distributed, low\-salience summarization context\)\. The pattern holds at every budget in the sweep, not just at the20%20\\%operating point used in the main text: the summarization margin is present already at keep\-15%15\\%\(28\.628\.6vs\. the best selection method’s27\.827\.8\) and widens to32\.232\.2vs\.29\.529\.5at keep\-25%25\\%, while retrieval and multi\-hop stay statistically tied throughout\. This matters because it rules out an alternative explanation for Table[1](https://arxiv.org/html/2607.15498#S4.T1)’s summarization result—that it is a one\-off artifact of the specific20%20\\%budget chosen for the headline comparison\. Three independent selection methods \(SnapKV, PyramidKV, Ada\-KV\) share the same observation\-window scoring mechanism and inherit the same structural weakness on distributed, low\-salience context; VarRate’s advantage tracks that shared weakness rather than any one competitor’s idiosyncrasies, which is why it appears consistently across the whole family and across the whole budget range\.

Table 5:Selection\-family budget sweep, Llama\-3\.1\-8B\. VarRate ties on retrieval and multi\-hop QA at every budget and leads summarization at every budget\.
### A\.3Quantization: Where It Wins, and Where It Composes

KIVI \(2\- and 4\-bit\) is a strong orthogonal axis, not a method VarRate dominates\. Reproduced in\-pipeline at effective memory≈19%\\approx 19\\%\(KIVI\-2\) and≈31%\\approx 31\\%\(KIVI\-4\): on Llama, KIVI reaches35\.235\.2–35\.535\.5on GovReport against VarRate’s30\.430\.4, because quantization preserves*distributed*information across all tokens at low precision, while rank reduction discards information outright\. VarRate ties or edges KIVI on retrieval and multi\-hop QA\. The mechanism behind the split is worth stating explicitly: quantization keeps every token at full rank and only coarsens the numerical precision of each coordinate, so information that is spread thinly across many tokens—exactly what a long, diffuse document like GovReport demands—survives at low bit\-width\. Low\-rank coding does the opposite: it keeps full numerical precision but discards entire coordinate directions, which is fatal precisely for that same distributed information, no matter how the discarded rank is allocated across tokens\. This is a property of the compression*axis*, not of any one implementation, so we do not expect a better salience signal or a different water\-filling schedule to close the gap\. The honest claim is therefore “the best low\-rank codec, competitive with and composable with quantization”—not a universal win, and Table[1](https://arxiv.org/html/2607.15498#S4.T1)/[3](https://arxiv.org/html/2607.15498#S4.T3)already show the two compose cleanly \(VarRate\+\+3\-bit coefficients:9%9\\%of the cache, accuracy within noise of fp16 VarRate on all three tasks\)\. That composability is the practical resolution: rather than choosing between the two axes, a deployment can quantize VarRate’s coefficients and recover quantization’s footprint while keeping the allocation mechanism’s retrieval and multi\-hop strength\.

Table 6:VarRate vs\. the quantization axis \(in\-pipeline KIVI reproduction\)\. KIVI wins summarization \(gov\_report\) on both models; VarRate ties or edges it on retrieval/multi\-hop\.
### A\.4Composability at Multiple Bit\-Widths

The main paper reports that quantizing VarRate’s low\-rank coefficients to 3 bits leaves accuracy unchanged at9%9\\%of the cache; Table[7](https://arxiv.org/html/2607.15498#A1.T7)shows the full sweep behind that single number\. Accuracy degrades smoothly and only slightly as bit\-width drops from fp16 through 8\-, 4\-, and 3\-bit coefficients, and even at 3 bits every task is within noise of the fp16 codec \(passage exact, musique−0\.7\-0\.7, gov exact\), while the footprint more than halves \(20%→9%20\\%\\to 9\\%\)\. This is the same composability pattern the published low\-rank literature reports for coefficient quantization on top of a fixed\-rank codec \(Palu\+\+quantization\); the fact that it holds here too, on a codec whose rank is itself variable per token, indicates the two compression axes are genuinely orthogonal rather than coincidentally compatible at one specific setting\.

Table 7:Coefficient quantization sweep\. Accuracy is within noise of fp16 all the way down to 3\-bit coefficients, at less than half the memory\.
### A\.5Additional Baselines: Sanity Checks Behind the Comparisons

Two validation steps underlie the baselines used throughout this paper and the main text, neither of which is visible in the headline numbers themselves\. First, KIVI \(2\- and 4\-bit\) is reproduced in\-pipeline rather than taken from published numbers; before trusting its comparison, we validate the reproduction directly on uncompressed\-equivalent passage retrieval, where KIVI\-2 \(≈19%\\approx 19\\%memory\) reaches100\.0100\.0and KIVI\-4 \(≈31%\\approx 31\\%memory\) reaches99\.599\.5—both within noise of the uncompressed ceiling, confirming the quantization implementation is not silently degrading accuracy on its own before any compression\-specific comparison is drawn\. Second, Palu is run through allama3RoPE shim so its published codec can be evaluated on Llama\-3\.1 at all \(Palu’s original release targets an older RoPE convention\); the shim passes a sanity gate before any of its collapse numbers are trusted—an uncompressed forward pass through the shimmed model reaches99\.599\.5on passage retrieval, against the true uncompressed ceiling of100100, confirming the port itself is correct and that Palu’s subsequent collapse at aggressive budgets \(Appendix[A\.6](https://arxiv.org/html/2607.15498#A1.SS6)\) is a genuine regime effect of the method, not an artifact of an incorrect port\. We also note, for completeness, a family of recent trained low\-rank codecs\. MatryoshkaKV\(Lin et al\.[2025](https://arxiv.org/html/2607.15498#bib.bib14)\)and STAR\-KV\(Bhatnagar et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib2)\)adapt rank across layers, heads, and blocks; DeltaKV\(Hao et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib9)\)encodes residuals against long\-range references; and DynaKV\(Lu et al\.[2026](https://arxiv.org/html/2607.15498#bib.bib17)\)varies rank per token as VarRate does\. All learn their allocation through gradient training\. VarRate is, to our knowledge, the training\-free point in this space, and we do not attempt a direct empirical comparison against these methods here, since a training\-free/trained contrast is a difference in kind, not in accuracy at matched budget\.

### A\.6The Published Low\-Rank Codec: Full Budget Curve and Footprint Validation

The main paper’s budget sweep against Palu \(its Figure[3](https://arxiv.org/html/2607.15498#S4.F3)\) plots the same numbers tabulated in full here \(Table[8](https://arxiv.org/html/2607.15498#A1.T8)\) across four budgets and three tasks\. Palu was designed for the keep\-5050–70%70\\%regime; at keep\-2020–30%30\\%it collapses to near\-zero on passage and musique \(0\.00\.0–0\.70\.7\), and only becomes viable once the budget loosens to keep\-50%50\\%\(passage82\.582\.5\) and keep\-70%70\\%\(passage97\.097\.0\)—exactly its own paper’s operating range\. VarRate strictly dominates it at every budget tested, including keep\-70%70\\%, where Palu is closest to competitive\.

Table 8:Full Palu budget curve, Llama\-3\.1,n=200n\{=\}200\. Even our own uniform\-rank flat codec beats Palu at matched budget, isolating two separate gains: basis construction \(flat≫\\ggPalu\) and salience allocation \(VarRate≫\\ggflat\)\.That last observation is worth stating explicitly: our own flat, uniform\-rank codec already beats Palu at every matched budget in the table, before salience allocation enters the picture at all\. This separates two independent sources of VarRate’s advantage over Palu—one from how the shared low\-rank basis itself is constructed \(flat’s residual\-to\-anchor\-mean codec versus Palu’s direct low\-rank projection\), and a second, larger one from allocating rank by salience rather than uniformly \(the comparison the main paper’s own uniform\-rank ablation isolates\)\. Reporting Palu’s comparison at a memory\-matched budget also required knowing its*true*footprint rather than its nominal one: computed directly from its released checkpoint’s per\-layerhead\_wise\_ranks, Palu’sratio\-0\.2configuration stores20\.41%20\.41\\%of the cache \(keys7\.4%7\.4\\%, values33\.4%33\.4\\%—its values are far less compressible than its keys\), marginally*more*than VarRate’s measured20\.22%20\.22\\%, confirming the keep\-20%20\\%row above is memory\-matched in Palu’s favor if anything\. Palu cannot be run through the same measured\-cost instrument as the other baselines \(it ships its own repository, environment, and fused kernel\), but its released accuracy pipeline reconstructs full K/V exactly as this paper’s convention does—PaluLlamaForCausalLMprojects to a latent representation and then immediately reconstructs it before caching, with the latent\-only kernel reserved for Palu’s own separate latency microbenchmarks\. The same accuracy\-in\-Mode\-A, efficiency\-in\-a\-separate\-kernel split holds for KIVI and other released low\-rank/ quantization codecs we are aware of; it is the field’s standard methodology, not a limitation specific to this comparison\. Finally, we attempted to reproduce Palu’s own latency claim on the GQA models used throughout this paper and could not: its fused attention kernel assumes a multi\-head attention \(MHA\) weight tensor \(num\_heads×\\timesrank×\\timeshead\_dim\) and raises when given Llama\-3\.1\-8B’s GQA\-shaped tensor \(num\_kv\_heads×\\timesrank×\\timeshead\_dim\) instead\. Run in Palu’s own MHA configuration, the kernel does reproduce its claimed attention\-module speedup \(1\.28×1\.28\\timesat 8K context,1\.24×1\.24\\timesat 16K, at a50%50\\%footprint\)—so the failure is specific to GQA models, consistent with Palu’s paper benchmarking latency on an MHA model \(Llama\-2\-7B\) and using a GQA model only for its accuracy numbers\. Every model evaluated in this paper \(Llama\-3\.1, Qwen2\.5, Mistral\-7B\) is GQA, which is why Palu’s memory claim carries over here but its kernel\-level speedup claim does not\.

### A\.7A Third Model at Full Budget Sweep \(Mistral\-7B\)

The main text reports that VarRate holds73\.673\.6on Mistral\-7B\-Instruct\-v0\.2 retrieval against flat’s collapse to29\.129\.1, at a15%15\\%budget; Table[9](https://arxiv.org/html/2607.15498#A1.T9)gives the full three\-budget, seven\-method comparison behind that summary\. Mistral\-v0\.2 is included for cross\-validation and three\-model\-family breadth—it is Palu’s own LongBench evaluation model—but is a substantially weaker base model on these tasks than Llama\-3\.1 or Qwen2\.5 \(uncompressed passage75\.675\.6, musique16\.916\.9, against Llama’s100100and3232–3434\), so it is reported as a cross\-check, not folded into the two\-model headline claim\.

Table 9:Third\-model cross\-check, Mistral\-7B\-Instruct\-v0\.2,n=200n\{=\}200\. The same pattern generalizes: flat collapses on retrieval, VarRate recovers to near\-ceiling and matches or beats full on multi\-hop QA, and quantization again wins summarization\.The pattern already established on Llama and Qwen generalizes cleanly to a third, architecturally distinct and substantially weaker base model: the flat codec’s retrieval collapse \(as low as29\.129\.1at the most aggressive budget tested\) is not a Llama\- or Qwen\-specific artifact, VarRate’s recovery to near\-uncompressed accuracy on retrieval \(73\.373\.3–74\.074\.0, against a75\.675\.6ceiling\) holds, VarRate matches or exceeds the uncompressed model on multi\-hop QA \(up to17\.817\.8against a ceiling of16\.916\.9\), and quantization’s summarization advantage over low\-rank coding \(KIVI≈32\\approx 32against VarRate’s≈30\\approx 30\) replicates as well\.

### A\.8Calibration Robustness Across Three Independently Built Bases

The shared PCA basis each layer uses is built once, offline, from a handful of unlabeled calibration contexts; if the codec’s accuracy depended sensitively on which contexts happened to be chosen, that would undermine its status as training\-free and broadly applicable\. We rebuild the basis three separate times—the original calibration set \(v0\), a set drawn from different examples \(v1\), and a set drawn from different tasks entirely \(v2\)—and rerun both the flat codec and VarRate at KR20 on all three \(Table[10](https://arxiv.org/html/2607.15498#A1.T10)\)\.

Table 10:Calibration robustness, Llama\-3\.1, KR20\. Every spread across the three independently built bases is≤1\.8\\leq\\\!1\.8points, inside the confidence intervals reported elsewhere in this paper\.Every spread across the three bases, for both methods and all three tasks, is at most1\.81\.8points—inside the confidence intervals used throughout the rest of this paper’s significance testing \(Appendix[A\.13](https://arxiv.org/html/2607.15498#A1.SS13)\)\. The codec’s accuracy is therefore not tuned to, or dependent on, the specific calibration examples used to build its basis; any small, unlabeled sample of representative contexts produces an equivalent codec\. The later negative\-control, Qwen\-signal, and floor experiments \(Apps\.[A\.17](https://arxiv.org/html/2607.15498#A1.SS17),[A\.18](https://arxiv.org/html/2607.15498#A1.SS18), and[A\.19](https://arxiv.org/html/2607.15498#A1.SS19)\) use a separately\-built basis of this kind; VarRate’s absolute scores there therefore sit within this same spread of the main tables \(e\.g\. Llama GovReport30\.030\.0rather than30\.430\.4\), not exactly on them\.

### A\.9A Data\-Integrity Correction in the 16\-Task Breadth Table

The main paper’s 16\-task LongBench breadth table \(its Table[1](https://arxiv.org/html/2607.15498#S4.T1)\) is assembled from per\-model, per\-task result files by a small script rather than computed inline, and an earlier version of that assembler had a real bug worth disclosing for transparency\. The original assembler globbed result files by pattern across all models’ output directories; because Qwen’s and Mistral’s runs reuse the same\(method, task\)key names as Llama’s, and the glob’s file ordering was not model\-aware, later\-sorted files from a different model silently overwrote the correct Llama value for that key in1818of the table’s2424reused cells—for example, Mistral’skivi\_k2v2/passagescore of73\.0973\.09overwrote Llama’s true value of100\.0100\.0for the same nominal key\. The corrupted table happened to make VarRate look like the top compressor and quantization look like it collapsed on retrieval, which is precisely why it was caught rather than silently published: those two conclusions contradicted the per\-run JSON files each score was drawn from\. The fix was to rebuild the assembler from explicit, per\-model\-scoped source paths and to make it raise an error on any duplicate key rather than silently overwrite, and to regenerate the table from scratch\. The individual per\-run result files were never wrong—only the now\-corrected aggregation step was—and the earlier, narrower experiments in this appendix \(the three\-task tables throughout\) were assembled independently and were unaffected\.

### A\.10Query\-Agnostic Budget Curve, and a Validated Expected\-Attention Baseline

Figure[4](https://arxiv.org/html/2607.15498#S4.F4)’s keep\-20%20\\%comparison could in principle be an artifact of testing KVzip outside its own headline regime \(keep\-2525–33%33\\%\)\. Sweeping keep\-\{10,20,25,50\}%\\\{10,20,25,50\\\}\\%on Llama, memory\-matched to within±2%\\pm 2\\%at every point, VarRate is at or above KVzip at keep\-2020,2525, and50%50\\%, including KVzip’s own regime \(keep\-25%25\\%:58\.1358\.13vs\.56\.8356\.83\)\. Only at keep\-10%10\\%does the default stride fall below KVzip, collapsing to29\.9229\.92\(App\.[A\.12](https://arxiv.org/html/2607.15498#A1.SS12)shows this is a stride artifact, not a property of low\-rank coding: widening the stride to6464recovers48\.8848\.88, above KVzip’s42\.1242\.12\)\. Expected Attention is separately validated against its own paper’s claimed regime—we reproduce its reported near\-parity at keep\-50%50\\%query\-aware \(−1\.2\-1\.2mean\)—so its−21\.7\-21\.7collapse under query\-agnostic reuse is a genuine mechanistic finding \(its future\-query estimate is built from a prefill that normally contains the question\) and not a misconfiguration on our part\. We additionally check Expected Attention against the exact regime its own paper claims: at keep\-50%50\\%, query\-aware, it reports near\-parity with the uncompressed model, and we reproduce that directly \(passage100\.0→98\.0100\.0\\to 98\.0, musique32\.76→32\.5632\.76\\to 32\.56, qasper47\.2→45\.9447\.2\\to 45\.94, a−1\.2\-1\.2mean drop\)\. Its published evaluation also sweeps removal ratios of0/10/25/50/75/90%0/10/25/50/75/90\\%, which brackets our80%80\\%\-removed \(κ=0\.20\\kappa\{=\}0\.20\) operating point, so the−21\.7\-21\.7collapse we observe under query\-agnostic reuse is squarely inside its own tested range—not an extrapolation to an untested regime where a method could reasonably be expected to fail\. Taken together, this rules out the two easiest objections to Figure[4](https://arxiv.org/html/2607.15498#S4.F4)and Table[2](https://arxiv.org/html/2607.15498#S4.T2): that KVzip was tested outside its comparative advantage, and that Expected Attention’s weakness is an artifact of our harness rather than a property of the method itself\.

Table 11:Query\-agnostic budget curve, Llama\-3\.1, 3\-task mean\. VarRate≥\\geqKVzip at keep\-20/25/50%20/25/50\\%, including KVzip’s own 25% regime; at keep\-10% the default stride \(s16\) falls below, a stride artifact \(App\.[A\.12](https://arxiv.org/html/2607.15498#A1.SS12)\) that widening the stride to s64 recovers\.
### A\.11Where the Prefill Overhead Goes, and How Far It Compresses

Profilingcompress\(\)stage\-by\-stage \(CUDA events, summed over 32 layers, cross\-checked against wall\-clock to1\.8%1\.8\\%\) attributes the\+44\.0%\+44\.0\\%overhead almost entirely to implementation, not arithmetic: reference\-token search \(32\.5%32\.5\\%of compress time,0\.84%0\.84\\%of its FLOPs\), the PCA projection \(21\.8%21\.8\\%time,1\.67%1\.67\\%FLOPs\), reconstruction, salience scoring, and water\-filling together account for the rest, with a materialized fp32 distance matrix, thousands of small kernel launches from water\-filling, and a per\-layer CPU→\\toGPU basis re\-upload as the actual costs\. A separate, bit\-identity\-controlled implementation \(varrate\_fast\_press\.py\) isolates this on its own timing run—its reference reads\+41\.7%\+41\.7\\%, against Table[3](https://arxiv.org/html/2607.15498#S4.T3)’s\+44\.0%\+44\.0\\%: caching the basis on GPU and using an analytic reference count reproduces the reference exactly \(99\.50/31\.97/30\.4599\.50/31\.97/30\.45, bit\-identical\) while cutting it to\+40\.3%\+40\.3\\%; adding bf16 compute on top cuts it further to\+25\.1%\+25\.1\\%, with accuracy unchanged \(max\|Δ\|=0\.07\|\\Delta\|=0\.07, inside noise\)\. The remaining floor is\+2\.6%\+2\.6\\%of prefill in pure arithmetic—the reference implementation in the main tables is not close to that floor, and a fused kernel is the natural next step\. The profile itself is cross\-validated two ways: the sum of the six stage timings \(732\.3732\.3ms\) agrees with an independently measured wall\-clock delta between compressed and uncompressed runs \(719\.2719\.2ms\) to within1\.8%1\.8\\%, so the breakdown is not an artifact of how the instrumentation itself perturbs timing\. We also record two optimizations that were tried and explicitly rejected, because both are easy mistakes that silently corrupt accuracy rather than merely underperforming: computing the anchor\-distance matrix in bf16 \(via the∥a∥2\+∥b∥2−2​a​b\\lVert a\\rVert^\{2\}\+\\lVert b\\rVert^\{2\}\-2abidentity\) catastrophically cancels, producing a15\.5%15\.5\\%mean relative error in the reference means and is kept in fp32 for that reason; and an early bf16 water\-filling variant fed the solver fp32 salience scores where the shipped reference always uses bf16 scores \(matching SnapKV’s native precision\), which silently changed the resulting rank allocation and broke bit\-identity by0\.100\.10points on musique before being caught and fixed\. Both are recorded so a reimplementation does not rediscover them the expensive way\.

Table 12:Stage\-by\-stage profile ofcompress\(\)\(32 layers, 16K ctx, A100\)\.Table 13:Optimized implementations, bit\-identity controlled\.fast\_fp32is exactly reference;fast\_bf16cuts overhead40%40\\%with accuracy unchanged\.
### A\.12The Stride Ablation

![Refer to caption](https://arxiv.org/html/2607.15498v1/x5.png)Figure 5:Worked example of the per\-token allocation on a6464K needle\-in\-a\-haystack instance\.Each token’s saliences^t\\hat\{s\}\_\{t\}\(its SnapKV observation\-window score\) sets a per\-token rankrtr\_\{t\}by water\-filling the shared budget: a low\-salience*boring*token is coarsened to the floorrminr\_\{\\min\}, whereas the high\-salience*needle*token is raised toward the capRR, so that its key and value survive reconstruction\. This is the mechanism behind VarRate’s92\.0%92\.0\\%at the6464K cell, where SnapKV—having irreversibly evicted the needle once its stale window score misjudged it—falls to48\.0%48\.0\\%\(Table[24](https://arxiv.org/html/2607.15498#A1.T24)\)\. The salience values shown are illustrative\.VarRate stores strided reference tokens at full width \(the exact\-kept anchors in Figure[5](https://arxiv.org/html/2607.15498#A1.F5), which walks the full per\-token scheme through one needle\); at the default stride1616they are6\.25%6\.25\\%of the cache, so at an aggressive keep\-10%10\\%budget they alone consume∼66%\\sim\\\!66\\%of it\. Widening the stride re\-splits the \(stride\-invariant\) footprint toward coefficients: at keep\-10%10\\%, query\-agnostic, stride16→6416\\to 64swings\+18\.96\+18\.96points \(29\.92→48\.8829\.92\\to 48\.88\), turning a loss to KVzip into a win\. This does*not*generalize to the full 16\-task suite, however: at the main keep\-20%20\\%operating point, stride6464costs−0\.71\-0\.71overall \(AVG\-1644\.9844\.98vs\.45\.6945\.69\), concentrated in few\-shot \(−3\.1\-3\.1\) and code \(−2\.69\-2\.69\)—structured contexts whose exemplars and cross\-file references must survive verbatim, which quartering the exact token budget destroys\. Multi\-hop QA gains slightly \(\+0\.8\+0\.8\) because the freed budget raises average rank\. The trade is explicit: stride governs structural preservation versus reconstruction fidelity, and the optimum shifts with the budget and the task type\. We keep stride=16=16as the committed default and report the wider setting only as an ablation, never tuned on the eval tasks themselves\. The accept/reject methodology matters here as much as the numbers: the query\-agnostic 3\-task signal at keep\-20%20\\%\(\+1\.13\+1\.13, inside its own confidence interval\) would, on its own, have looked like a plausible case for switching the default—and would have led us to rerun the full 16\-task suite with the*worse*setting had that decision been made after seeing the smaller, favorable number\. The rule for when a hyperparameter ablation earns a full 256\-example, 16\-task rerun was fixed before that rerun’s result was known, precisely to prevent this kind of favorable\-looking\-in\-isolation number from silently becoming the shipped default\. Stride is reported here as exactly what it is: a real trade\-off between how many tokens are stored exactly and how much rank budget survives for everyone else, with no universally correct setting, rather than a knob tuned post hoc on the evaluation tasks\.

Table 14:Stride trades structural preservation against reconstruction fidelity: wider stride wins at an aggressive 10% budget but loses0\.710\.71on the full 16\-task suite \(few\-shot−3\.1\-3\.1, code−2\.69\-2\.69\), so stride=16=16stays the default\.
### A\.13Statistical Significance, Summarized

Every comparison in the main text is a paired bootstrap \(20,000 resamples, 95% CI\) over the 16 LongBench tasks, backed by an exact sign test; Table[15](https://arxiv.org/html/2607.15498#A1.T15)summarizes the verdicts referenced throughout\. The pattern that survives testing is: VarRate never loses significantly to a matched\-memory baseline on either model, wins significantly against selection only on Qwen, ties quantization on both models, and beats its own uniform\-rank ablation significantly on both—the strongest and most robust result in the paper\. We treat a comparison as supported only when the paired bootstrap CI excludes zero; a nominal mean difference in VarRate’s favor that does not clear that bar \(e\.g\. Ada\-KV on Llama,\+0\.36\+0\.36, or KIVI\-2 on either model\) is reported as a tie, not as a win, throughout the paper, and the reverse discipline applies equally—KIVI\-4’s nominal−0\.98\-0\.98on Qwen is a real, CI\-excluding loss, but at1\.55×1\.55\\timesthe memory of every other method in the comparison, so it is reported as significant but not memory\-matched rather than folded into an unqualified “VarRate loses to KIVI” claim\. This two\-sided discipline is what lets the single strongest number in the table—VarRate beating its own uniform\-rank ablation by\+2\.22\+2\.22\(Llama\) and\+17\.50\+17\.50\(Qwen\), both significant atp≤0\.021p\\leq 0\.021and winning13/1613/16and15/1615/16tasks respectively—stand as the paper’s central claim rather than as one cherry among several nominal wins\.

Table 15:Paired VarRate−\-baseline deltas over 16 tasks, 20,000\-sample bootstrap\.∗CI excludes zero\.
### A\.14Real Wall\-Clock Latency

Table[3](https://arxiv.org/html/2607.15498#S4.T3)’s prefill overhead is an accounted, not measured, estimate for this section’s purpose; we additionally time full end\-to\-endpipe\(\)calls \(n=18, 2 warmup runs discarded\) on one prefill\-dominated task \(short\-answer retrieval\) and one decode\-dominated task \(long\-generation summarization\)\. On retrieval, VarRate’s overhead is\+41\.8%\+41\.8\\%end\-to\-end, at the top of the accounted range, becausecompress\(\)is a fixed one\-time prefill cost and prefill dominates a short generation\. On summarization, the identical fixed cost dilutes to\+2\.6%\+2\.6\\%, within run\-to\-run noise \(±4\.36\\pm 4\.36s\)—indistinguishable from uncompressed\. SnapKV and KIVI stay near\-zero overhead on both tasks, as expected from their respective mechanisms: SnapKV’s scoring is a single additional attention read over the observation window, and KIVI’s quantization is a fixed\-cost per\-tensor quantize step, neither of which scales with the codec\-style projection and reconstruction work VarRate performs\. The practical reading: VarRate’s compute tax is concentrated in short\-generation, retrieval\-style workloads and disappears for long\-generation workloads, the inverse of where its accuracy advantage over selection is smallest \(Appendix[A\.2](https://arxiv.org/html/2607.15498#A1.SS2)shows the summarization win, which is exactly the long\-generation regime where the latency tax is noise\)\. A deployment sensitive to prefill latency on short\-answer workloads should weigh this tax against the accuracy and robustness gains reported elsewhere in this appendix; a deployment dominated by long\-generation summarization or multi\-turn dialogue pays close to nothing for the same benefits\.

Table 16:Real end\-to\-end wall\-clock, Llama\-3\.1\-8B, one A100 \(n=18n\{=\}18\)\. VarRate’s fixedcompress\(\)cost dominates on prefill\-heavy retrieval but is noise on decode\-heavy summarization\.
### A\.15A Second Long\-Context Suite \(RULER\), Limited Scope

As a task\-diversity cross\-check beyond the three tasks used throughout, we also ran RULER’s\(Hsieh et al\.[2024](https://arxiv.org/html/2607.15498#bib.bib11)\)synthetic suite \(niah / QA / multi\-hop / aggregation\) at ctx=16=16K,n=40n=40\(Table[17](https://arxiv.org/html/2607.15498#A1.T17)\)\. The gap between VarRate and SnapKV \(\+0\.8\+0\.8\) is within noise at thisnn; both compressors give up a real∼8\\sim\\\!8points against uncompressed on RULER’s harder mixed suite\. The available HF mirror only hosts 4K/8K/16K subsets, so this is a diversity check, not the long\-context scaling result—that claim is made instead by the controlled needle\-in\-a\-haystack \(NIAH\) sweep to128128K \(App\.[A\.21](https://arxiv.org/html/2607.15498#A1.SS21)\)\. RULER’s four task families are harder and more synthetically adversarial than the three real\-text tasks \(passage retrieval, multi\-hop QA, summarization\) used throughout the rest of this paper, so a consistent VarRate/SnapKV ordering here is evidence that the main results are not an artifact of always reusing the same three tasks\. We had originally scoped this as a1616K–128128K scaling stress test, matching the NIAH sweep in App\.[A\.21](https://arxiv.org/html/2607.15498#A1.SS21); that plan was dropped mid\-run once we confirmed, via the hosted repository’s own file listing rather than a failed download, that the HF mirror used here \(simonjegou/ruler\) simply does not host32768/65536/13107232768/65536/131072\-token subsets at all\. We report the1616K result that was already run rather than discard it, but flag explicitly that it does not substitute for a scaling claim\.

Table 17:RULER cross\-check, Llama\-3\.1\-8B,n=40n=40\. VarRate/SnapKV gap is within noise\.
### A\.16A Powered Re\-Gate of the KVzip Comparison

The main text’s KVzip verdicts rest on a33\-task mean, whose sign test cannot reach significance in principle \(floorp=0\.25p=0\.25atn=3n=3\)\. We additionally pair VarRate against KVzip*per example*, pooled over the three tasks \(∼600\\sim\\\!600paired examples/cell, verified by a gold\-answer fingerprint\), summarized in Table[18](https://arxiv.org/html/2607.15498#A1.T18)\. This confirms, with real statistics rather than an eyeballed CI band, that VarRate carries no accuracy advantage over KVzip and is significantly behind only in one regime\. In the same per\-example analysis, VarRate beats every selection/flat baseline in the query\-agnostic regime with significance \(vs\. SnapKV\+10\.2\+10\.2/\+9\.9\+9\.9, vs\. Ada\-KV\+8\.2\+8\.2/\+11\.6\+11\.6, vs\. flat\+14\.5\+14\.5/\+29\.3\+29\.3, Llama/Qwen\)\. The per\-example pairing is necessary rather than cosmetic: a33\-task mean has an exact two\-sided sign test whose smallest achievablepp\-value atn=3n\{=\}3is0\.250\.25, so no amount of margin on a 3\-task table can ever cross a conventional significance threshold, no matter how large or consistent the underlying effect is\. Pairing at the level of individual examples restores real statistical power without changing what is being measured, as long as the pairing itself is trustworthy; we verify that VarRate’s and KVzip’s per\-example scores are aligned to the same underlying question by matching on a fingerprint of the gold answer, rather than assuming the two evaluation runs iterated examples in the same order\. This is what allows the main text’s “no accuracy advantage over KVzip, advantage is cost” claim to rest on a real confidence interval rather than an eyeballed±2\.5\\pm 2\.5band on three numbers\.

Table 18:Per\-example paired bootstrap, VarRate vs\. KVzip, pooled over 3 tasks\.∗CI excludes zero\. 3 of 4 cells are powered ties; KVzip leads only in Qwen query\-agnostic reuse\.
### A\.17Is the Signal Content, or Just Shape?

Table[4](https://arxiv.org/html/2607.15498#S4.T4)’s random signal changes the rank*marginal*along with the alignment, leaving open whether a graded, recency\-like profile would do just as well without reading content at all\. Two further controls isolate this:*shuffle*permutes the real salience scores across coded tokens \(same marginal, alignment destroyed\), and*position*keeps only the recency\-and\-attention\-sink shape that Appendix[A\.19](https://arxiv.org/html/2607.15498#A1.SS19)’s rank logs reveal, with no content signal at all\. On Qwen, where low\-rank coding is least forgiving, position collapses on multi\-hop QA \(7\.77\.7vs\. mass’s28\.728\.7\) and shuffle drops*below*flat on both retrieval \(64\.764\.7vs\.75\.075\.0\) and multi\-hop QA \(7\.57\.5vs\.8\.58\.5\)—a non\-uniform allocation aimed at the wrong tokens is worse than a uniform one\. Content alignment, not gradedness alone, is what the signal contributes; recency\-with\-extra\-steps is refuted as an alternative explanation\.

Figure[6](https://arxiv.org/html/2607.15498#A1.F6)makes the mechanism concrete with a single traced token: a schematic “needle” sitting in the document’s boring middle \(Figure[7](https://arxiv.org/html/2607.15498#A1.F7)’s low\-rank zone\), where recency and attention\-sink effects are weakest and content alignment is the only thing that can still find it\.*mass*is the only condition that scores this token highest and recovers it;*shuffle*keeps the exact same set of scores but attaches them to the wrong tokens, so the needle gets whatever another token’s score happened to be;*position*never looks at content at all, so a needle sitting away from both edges is scored low regardless of how critical it is\. The bar heights are illustrative of the mechanism, not a plot of the ablation itself—Table[19](https://arxiv.org/html/2607.15498#A1.T19)below gives the real numbers behind each panel’s accuracy annotation\.

![Refer to caption](https://arxiv.org/html/2607.15498v1/x6.png)Figure 6:Why the signal has to be content\-aligned, not just graded\.One token \(★\\bigstar, outlined\) plays a query\-critical “needle” placed in the document’s boring middle, away from both the attention\-sink start and the query\-proximate end\.\(a\)Flat coding does not look at the needle at all\.\(b\)Mass \(ours\) scores it highest, because it is the only condition that reads content\.\(c\)Shuffle keeps the identical multiset of scores from \(b\) but reassigns them across tokens, so the needle inherits an arbitrary score instead of its own\.\(d\)Position reproduces the recency/attention\-sink*shape*from Figure[7](https://arxiv.org/html/2607.15498#A1.F7)with no content signal, so a needle in the middle is scored low purely by where it sits\. Only the content\-aligned condition \(b\) recovers the needle; a graded\-but\-content\-free score, however shaped, does not substitute for it\.Table 19:Negative controls, KR20,n=200n\{=\}200\.*shuffle*holds the rank marginal but destroys alignment;*position*is recency\+sink shape with no content\. Both underperform mass, and on Qwen both underperform flat\.
### A\.18Does the Signal Generalize Past One Heuristic and One Model?

Replacing SnapKV attention with a structurally different second\-order signal—attention squared, weighted by value\-distinctiveness throughWOW\_\{O\}, adapted and reindexed from a closed\-form Fisher\-sensitivity metric developed for offline basis whitening rather than online per\-token rank allocation—ties mass on Llama, and on Qwen \(where the signal axis bites hardest\) it ties again\. This second\-order signal is still computed from a single forward pass with no gradients, so the comparison is between two training\-free heuristics, not between a heuristic and a trained alternative\. The result is not an artifact of one particular salience heuristic: a signal built from a completely different mathematical quantity \(second\-order attention sensitivity rather than first\-order attention mass\) lands within the confidence interval of the default on every task, on both models, which is the strongest form of evidence available that ‘mass’ was not a lucky or overfit choice among many salience heuristics we could have picked\. The allocation axis replicates too: on Qwen, water\-filling beats a fixed binary split on every task that is not pure retrieval, so Llama’s one point in fixed\-split’s favor does not carry over \(Table[20](https://arxiv.org/html/2607.15498#A1.T20)\)—if anything, Qwen is the sharper test of the allocation choice, because its four\-KV\-head cache is far less forgiving of a wrong allocation shape than Llama’s, as the main uniform\-rank ablation already shows, making any advantage for fixed\-split easier to see if it existed\.

Table 20:Signal and allocation axes replicated on Qwen2\.5\-7B\. Fisher ties mass; on Qwen, unlike Llama, water\-filling beats fixed\-split on every non\-retrieval task\.
### A\.19Is the Rank Floor a Lever?

Settingrmin=0r\_\{\\min\}=0changes accuracy by less than a point on every task \(Table[21](https://arxiv.org/html/2607.15498#A1.T21)\), which could mean water\-filling rediscovers hard eviction once permitted to\. It does not: per\-token rank logs over33–4\.34\.3M coded tokens show0\.0%0\.0\\%of tokens at the floor under either setting, at both the main20%20\\%budget and, extending the check, at10%10\\%and15%15\\%\. The floor never binds; VarRate is not secretly falling back to selection\. The same logs surface a mechanistic finding instead \(Figure[7](https://arxiv.org/html/2607.15498#A1.F7)\): allocated rank is elevated at the very start of the document \(an attention\-sink effect\), dips through the middle, and climbs sharply toward the query at the end—the signal is dominated by recency and query proximity, with content alignment \(Appendix[A\.17](https://arxiv.org/html/2607.15498#A1.SS17)\) supplying the rest\. This was originally motivated by a stronger hypothesis: that water\-filling would naturally*rediscover*hard eviction for the least\-salient tokens once the floor no longer forbids it, which would make VarRate a strict generalization subsuming both flat low\-rank coding and token selection as special cases\. The rank logs refute that stronger claim directly—the floor sits at0\.0%0\.0\\%occupancy whether or not it is even present, so there is no mass of low\-salience tokens straining againstrmin=16r\_\{\\min\}\{=\}16that gets released to zero once the constraint is lifted\. The correct description is more modest but still useful: at this operating point, water\-filling’s natural allocation already keeps every coded token comfortably above the floor, so the floor functions as an unused safety margin rather than as an active mechanism, and the “strict generalization” framing should be dropped in favor of a claim about accuracy parity under floor removal\. A related diagnostic is worth flagging honestly: a pooled Pearson correlation between raw, un\-normalized salience scores and final allocated rank across all sampled \(layer, example\) traces comes out surprisingly low \(∼0\.05\\sim\\\!0\.05–0\.060\.06\)\. This is very likely a pooling artifact rather than evidence that salience weakly predicts rank—raw attention\-mass scale varies substantially across different forward passes, and water\-filling only enforces a monotonic salience\-rank relationship*within*a single pass, after per\-pass min–max normalization, not across pooled raw scores drawn from different passes\. We do not read it as contradicting the main text’s oracle comparison and signal ablation, both of which show that salience does predict rank well; it simply shows that this particular pooled statistic is the wrong instrument to confirm it, and a within\-pass rank correlation would be the correct follow\-up if this needs to be demonstrated more directly\.

Table 21:Removing the rank floor costs nothing, but per\-token logs show0\.0%0\.0\\%of coded tokens at the floor under either setting—the floor is slack, never binding\.![Refer to caption](https://arxiv.org/html/2607.15498v1/x7.png)Figure 7:Where does the rank budget actually go?Mean allocated rank by document\-position decile, pooled over300300sampled \(layer, example\) traces on gov\_report at KR20 \(0 = document start, 9 = the end, where the query sits after the query\-aware transform appends it\)\. Three regimes, annotated directly on the data: an*attention\-sink*bump at the very start \(a well\-documented effect where the first tokens draw disproportionate attention regardless of content\); a dip through the “boring middle,” where content alignment \(Appendix[A\.17](https://arxiv.org/html/2607.15498#A1.SS17)\) is doing most of the remaining work; and a sharp, monotonic climb toward the query at the end, where recency and query proximity dominate\. The dashed floor line is the samermin=16r\_\{\\min\}=16from Table[21](https://arxiv.org/html/2607.15498#A1.T21)—every decile’s*mean*sits1010–45×45\\timesabove it, visual confirmation that the floor is slack everywhere, not just on average\.
### A\.20Does the Advantage Grow at Aggressive Budgets?

The main results operate at keep\-1515–25%25\\%; a natural conjecture is that VarRate’s margin over selection grows as the budget tightens further\. Sweeping to keep\-88–12%12\\%\(Table[22](https://arxiv.org/html/2607.15498#A1.T22)\) falsifies this\. At keep\-8%8\\%, SnapKV matches or beats VarRate on both models, and on Qwen VarRate collapses to flat’s level: below a viability threshold, exact selection keeps the query\-relevant tokens on any budget, while a shared low\-rank basis cannot\. VarRate recovers parity with selection by keep\-10%10\\%\(Llama\) and keep\-12%12\\%\(Qwen\), and still beats the flat codec throughout\. This is the mirror image of the query\-agnostic result in the main text: which method wins is set by the*regime*—query\-aware versus reused—not simply by how aggressive the budget is, and we state VarRate’s operating range accordingly rather than claiming a universal aggressive\-budget win\. For reference, memory\-matched quantization \(KIVI\-2, fixed at≈19%\\approx 19\\%budget regardless of the nominal KR target\) still wins summarization even in this aggressive regime \(Llama100/35\.3/28\.9100/35\.3/28\.9, Qwen94\.4/33\.2/30\.594\.4/33\.2/30\.5at passage/gov/musique\), consistent with Appendix[A\.3](https://arxiv.org/html/2607.15498#A1.SS3)’s finding that the quantization axis’s advantage on distributed information does not depend on how tight the rank budget is\. The mechanism behind VarRate’s collapse at KR8 on Qwen is structural rather than a tuning failure: once the per\-token rank budget shrinks far enough, a shared low\-rank basis eventually cannot represent*any*coded token to a useful fidelity, and every token degrades together\. Exact selection’s failure mode is different in kind, not just in degree—it always keeps some tokens at perfect fidelity and simply drops more of them as the budget tightens, which is why it remains viable at budgets where a shared low\-rank basis is not\. Whether a deployment should prefer VarRate or SnapKV at a given budget is therefore a question of which failure mode is more tolerable for the workload, not a question of which method is unconditionally better\.

Table 22:Aggressive\-budget sweep \(passage/gov/musique, query\-aware,n=200n\{=\}200\)\. On Qwen at KR8, VarRate collapses to flat’s level—a viability floor selection does not have\. Parity returns by KR10 \(Llama\) / KR12 \(Qwen\)\.Crossing this with the stride/floor knobs at KR10 \(Table[23](https://arxiv.org/html/2607.15498#A1.T23)\) extends App\.[A\.19](https://arxiv.org/html/2607.15498#A1.SS19)’s finding:rmin=0r\_\{\\min\}\{=\}0still tiesrmin=16r\_\{\\min\}\{=\}16at an aggressive budget, and wider stride still helps, exactly as at KR20\.

Table 23:Floor and stride crossed at an aggressive KR10 budget\. Both App\.[A\.19](https://arxiv.org/html/2607.15498#A1.SS19)and App\.[A\.12](https://arxiv.org/html/2607.15498#A1.SS12)’s findings extend unchanged: the floor stays slack, wider stride still helps\.![Refer to caption](https://arxiv.org/html/2607.15498v1/x8.png)Figure 8:Query\-agnostic needle\-in\-a\-haystack, Llama\-3\.1\-8B at a matched20%20\\%budget, fixed needle set at every length \(n=75n\{=\}75/cell\)\. VarRate tracks the uncompressed ceiling; SnapKV’s accuracy is non\-monotonic and drops to48%48\\%at6464K\. The128128K recovery is genuine in the raw generations but its mechanism is not established \(below\)\.
### A\.21Long\-Context Retrieval Beyond the 16K Ceiling

Every accuracy result in the main text is measured at LongBench’s native length \(≲16\\lesssim 16K\)\. We extend the query\-agnostic setting to128128K on Llama using kvpress’s needle\-in\-a\-haystack harness, at the same20%20\\%budget \(Figure[8](https://arxiv.org/html/2607.15498#A1.F8)\)\. VarRate tracks the uncompressed ceiling at every length, degrading to92%92\\%at its single worst point \(6464K\) against SnapKV’s48%48\\%there \(Table[24](https://arxiv.org/html/2607.15498#A1.T24)\); Figure[5](https://arxiv.org/html/2607.15498#A1.F5)traces a single needle through the codec to show why\. SnapKV’s accuracy is also non\-monotonic—it recovers to100%100\\%at128128K—and a controlled re\-run with the identical needle set at every length shows this is not a sampling artifact: the pattern survives\. We report the6464K dip and128128K recovery as real and reproducible \(verified against the raw generations\) without a confirmed mechanism; the load\-bearing claim is that VarRate degrades far less than selection under stress at matched memory, which the control does not disturb\. Realizing the memory saving at128128K at run time still requires the fused kernel or a multi\-GPU shard, as in §[3\.3](https://arxiv.org/html/2607.15498#S3.SS3)\.

Two further checks separate the genuine effect from possible instrumentation artifacts\. First,128128K required sharding the model across two GPUs, because the reconstruct\-full\-K/V harness \(which every accuracy number in this paper uses, by convention with prior low\-rank work\) exhausts a single 40–48GB GPU’s memory at that length; a dedicated shard\-check confirmed that the sharded configuration reproduces the single\-GPU3232K/6464K numbers exactly, so the128128K row is a deployment consideration—realizing the accuracy requires either the fused kernel or a second GPU at that length—rather than a signal that sharding itself changes the accuracy result\. Second, the uncompressed model’s own rare misses \(∼1%\\sim\\\!1\\%\) are themselves a decoding quirk rather than a retrieval failure: greedy decoding occasionally truncates the six\-digit needle mid\-generation for the exact same needle value at different lengths and depths \(e\.g\. producing “415\.” in place of “915965”\), independent of any compression, which sets the honest noise floor against which VarRate’s and SnapKV’s compressed\-model drops should be read\. Examining the raw per\-depth predictions further shows that VarRate’s6464K dip is narrow—60%60\\%at a single depth,100%100\\%at every other depth tested—while SnapKV’s dip at the same length is broad, spanning three of five depths at0–27%27\\%each; both compressed methods return to100%100\\%at128128K, and inspection of the raw generations confirms these are genuine correct completions rather than a scoring artifact\.

Table 24:Query\-agnostic NIAH, fixed needle set at every length\. VarRate’s worst point \(64K, 92%\) far exceeds SnapKV’s \(48%\); the 128K recovery is genuine but unexplained\.