AnchorKV: Anchor-Residual KV Cache Compression

# AnchorKV: Anchor-Residual KV Cache Compression
Source: [https://arxiv.org/html/2608.02901](https://arxiv.org/html/2608.02901)
Malik Khalaf1\\equalcontrib, Yara Shamshoum1\\equalcontrib, Nitzan Hodos1, Yuval Sieradzki1, Assaf Schuster1

###### Abstract

The key\-value \(KV\) cache is the primary memory bottleneck in long\-context LLM inference\. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression\. We propose AnchorKV, a compression scheme that shrinks the cache by 20×\\timeswithout discarding a single token\. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model’s output\. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full\-cache score at the 70B scale, while keeping the entire context at a fraction of its cost\.

††footnotetext:Preprint\. Under review\.††footnotetext:Our code will be released upon acceptance\.## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.02901v1/x1.png)Figure 1:Overview of AnchorKV\. \(a\) The representation inℝ2\\mathbb\{R\}^\{2\}: anchors are stored exactly as\-is, and every other token is projected onto the span of its nearest anchor and encoded by an anchor index and one coefficientγi=⟨xi,xa​\(i\)⟩/‖xa​\(i\)‖2\\gamma\_\{i\}=\\langle x\_\{i\},x\_\{a\(i\)\}\\rangle/\\\|x\_\{a\(i\)\}\\\|^\{2\}, which absorbs the anchor’s norm\. Tokenx1x\_\{1\}aligns closely with its anchor and stores no residual;x2x\_\{2\}aligns poorly, receives a budgeted slot, and stores a 2\-bit residual, givingx^2=x~2\+r~2\\hat\{x\}\_\{2\}=\\tilde\{x\}\_\{2\}\+\\tilde\{r\}\_\{2\}\. \(b\) Bytes per token under three compressed regimes at a common budget, against the full cache\. Eviction spends everything on exactly\-stored tokens and so keeps more of them, while AnchorKV affords fewer anchors because the rest of its budget represents every remaining token\.Long\-context inference is now the common case: models read whole repositories, documents, and conversation histories, and context windows run to hundreds of thousands of tokens\. The limit is memory, not compute\. The key\-value \(KV\) cache stores one key and one value per token per layer, so it grows linearly with context and batch size and is read in full at every decoding step\. At 128K, a single Llama\-3\.1\-8B\(Team[2024](https://arxiv.org/html/2608.02901#bib.bib36)\)request holds a 16 GiB cache in bf16, as much as the model weights serving it\. The cache, not the model, decides how many requests a device holds and how fast each advances\.

Several lines of work attack this problem\. The dominant one is eviction, which scores tokens and keeps only a subset, dropping the rest\(Zhanget al\.[2023](https://arxiv.org/html/2608.02901#bib.bib12); Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15); Caiet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib16); Fenget al\.[2025](https://arxiv.org/html/2608.02901#bib.bib17)\)\. A dropped token costs nothing, so compression is unbounded in principle, and the highest compressions come from these works\. The difficulty is that scoring happens before the queries that will read the cache are known: a token the prompt’s last queries ignored is gone when a later query needs it, and our per\-task results show this is not rare but the family’s dominant failure mode \(Section[4\.2](https://arxiv.org/html/2608.02901#S4.SS2)\)\. Offloading and retrieval systems answer this by holding the evicted state in host memory and fetching the blocks a query needs\(Xiaoet al\.[2024a](https://arxiv.org/html/2608.02901#bib.bib44); Tanget al\.[2024](https://arxiv.org/html/2608.02901#bib.bib45)\)\. Reach is restored, but the full cache still lives somewhere, and the fetch and its bookkeeping now sit in the decode loop, trading a memory problem for a transfer and scheduling one\.

A second line quantizes, keeping every token resident in fewer bits\(Liuet al\.[2024b](https://arxiv.org/html/2608.02901#bib.bib9); Hooperet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib22); Zandiehet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib5)\)\. Nothing is ever out of reach, the property eviction gives up\. What this line gives up instead is depth: accuracy falls as the representation is pushed to one or two bits, so compression is bounded\. In our comparison, it stops near5×5\\times, short of eviction’s ratios and of the regime where the cache stops being the binding constraint\.

In this work we present AnchorKV, which keeps every token and still compresses aggressively, storing the cache in an anchor\-residual representation\. Each KV head stores a small set of anchor tokens exactly and represents every other token as a scaled copy of its closest anchor, at a cost of a few bytes instead of a full vector \(Figure[1](https://arxiv.org/html/2608.02901#S1.F1)\)\. Whatever the byte budget leaves buys quantized residuals for the tokens whose approximation costs the most attention\-output error\. No position leaves the softmax or is out of reach of a later query, so the full context survives at a fraction of its cost and the compressed cache is a perturbation of the exact one, not a truncation of it\.

In AnchorKV, compression runs once at the end of prefill, on a frozen model, leaving the attention arithmetic unchanged\. The user sets a single knob: the fraction of the cache to retain, and an exact byte accounting turns it into a storage plan that matches the requested budget; every other parameter is held fixed across all models and benchmarks\. Under that one setting, AnchorKV at20×20\\timesscores higher than every eviction baseline at10×10\\times, the same accuracy at half the memory, across three model scales and three long\-context benchmarks\. On Llama\-3\.1\-70B it retains 99\.3% of the uncompressed RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib37)\)score at20×20\\timesagainst 86\.8% for the strongest baseline, and retention improves with scale, the regime where the cache is most expensive to hold\.

Section[2](https://arxiv.org/html/2608.02901#S2)reviews prior compression methods, Section[3](https://arxiv.org/html/2608.02901#S3)presents AnchorKV, and Section[4](https://arxiv.org/html/2608.02901#S4)evaluates it at matched byte budgets against eviction and quantization baselines, ablates each component, and reports runtime and memory\.

## 2Related Work

Long\-context inference has made the KV cache a standard target for compression, and the work on it falls into a few recognizable families\. We review three: token eviction, quantization and low\-rank projection, and shared representations\.

#### Token Eviction\.

The dominant approach keeps a subset of tokens and drops the rest, using attention sinks with a recency window\(Xiaoet al\.[2024b](https://arxiv.org/html/2608.02901#bib.bib11)\), accumulated attention mass\(Zhanget al\.[2023](https://arxiv.org/html/2608.02901#bib.bib12); Liuet al\.[2023](https://arxiv.org/html/2608.02901#bib.bib13); Orenet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib14)\), or an observation window that scores the prompt at prefill, such as SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15)\)\. Later work keeps the rule and improves the budget, distributing capacity across layers\(Caiet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib16)\)or heads\(Fenget al\.[2025](https://arxiv.org/html/2608.02901#bib.bib17)\), or ranking tokens by their change in attention output rather than by attention score alone\(Fenget al\.[2026](https://arxiv.org/html/2608.02901#bib.bib18); Guet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib19)\)\. AnchorKV uses the same prefill\-time scoring machinery, the SnapKV observation window in particular, but spends it on a different decision: how many bytes a token receives, not whether it is kept\. The binary decision of eviction is irreversible, and the cost is visible in our per\-task results, where the eviction baselines collapse on exactly the tasks whose answer tokens the observation window did not score highly \(Section[4\.2](https://arxiv.org/html/2608.02901#S4.SS2)\)\.

#### Quantization and Low\-Rank Projection\.

An orthogonal line keeps every token and compresses its representation, either lowering the bit width of each entry through per\-channel and outlier\-aware schemes\(Liuet al\.[2024b](https://arxiv.org/html/2608.02901#bib.bib9); Hooperet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib22); Kanget al\.[2024](https://arxiv.org/html/2608.02901#bib.bib23); Saxena and Roy[2025](https://arxiv.org/html/2608.02901#bib.bib24)\), or projecting keys and values onto a low\-rank subspace across layers\(Liuet al\.[2024a](https://arxiv.org/html/2608.02901#bib.bib28); Changet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib29)\)or along the hidden dimension\(Changet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib30)\)\. AnchorKV shares the commitment to the full context, and departs on how far compression can be pushed: accuracy in this line falls off as the representation approaches one or two bits, which caps it near5×5\\timesin our comparison\. AnchorKV instead varies how many bytes a token receives, and reaches20×20\\times\. The two axes are complementary: our residual codec already uses a randomized rotation of the kind this line introduced\.

#### Shared Representations\.

A third family represents several tokens together, merging evicted states into retained neighbors\(Zhanget al\.[2024](https://arxiv.org/html/2608.02901#bib.bib25); Wanet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib26)\), grouping similar tokens\(Wanget al\.[2024](https://arxiv.org/html/2608.02901#bib.bib27)\), or coding tokens against a shared set of centroids or codebook entries\(Liet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib32); Huet al\.[2026](https://arxiv.org/html/2608.02901#bib.bib33); Liuet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib34); Hooperet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib35)\)\. AnchorKV is closest to this family, since an anchor is a reference vector and most tokens are coded against one\. What differs is what survives the coding: when tokens collapse onto a shared representation, the softmax can no longer separate them\. AnchorKV gives every token its own coefficient along its anchor, so no two become identical, and adds a residual for those whose approximation costs the most output error\.

## 3Method

AnchorKV compresses the KV cache without removing positions from attention\. At the end of prefill, each KV head selectskkanchorpositions and stores their keys and values exactly\. Every other vector is approximated by its projection onto the span of a single anchor \(Figure[1](https://arxiv.org/html/2608.02901#S1.F1)\), costing only an anchor index and a scalar coefficient\. Tokens whose projection costs the most attention\-output error also store a quantizedresidual\. The user fixesθ\\theta, the fraction of the uncompressed cache to retain, and an exact byte accounting turns it into the number of residuals a layer can store\.

Throughout,SSis the context length,DDthe per\-head dimension,HHandHq≥HH\_\{q\}\\geq Hthe number of KV and query heads,WWthe recency window, andkkthe per\-head anchor budget, which includes the window\. All operations are per layer and per KV head, whose keys and values we write asK,V∈ℝS×DK,V\\in\\mathbb\{R\}^\{S\\times D\}\.

### 3\.1Anchor\-Residual Representation

The representation applies independently toKKand toVV, so we describe it for generic vectorsx1,…,xS∈ℝDx\_\{1\},\\dots,x\_\{S\}\\in\\mathbb\{R\}^\{D\}standing for the rows of either\. A subset𝒜\\mathcal\{A\}of them, theanchors, is stored exactly, and each anchor provides a direction along which every other vector is represented\. Each non\-anchor vector is assigned to the anchor whose direction is closest to its own,

a​\(i\)=arg​maxa∈𝒜⁡\|⟨xi,xa⟩\|∥xi∥​∥xa∥,a\(i\)=\\operatorname\*\{arg\\,max\}\_\{a\\in\\mathcal\{A\}\}\\frac\{\\left\|\\langle x\_\{i\},x\_\{a\}\\rangle\\right\|\}\{\\lVert x\_\{i\}\\rVert\\lVert x\_\{a\}\\rVert\},\(1\)and approximated by its projection onto that anchor,

γi=⟨xi,xa​\(i\)⟩∥xa​\(i\)∥2,x~i=γi​xa​\(i\),ri=xi−x~i,\\gamma\_\{i\}=\\frac\{\\langle x\_\{i\},x\_\{a\(i\)\}\\rangle\}\{\\lVert x\_\{a\(i\)\}\\rVert^\{2\}\},\\qquad\\tilde\{x\}\_\{i\}=\\gamma\_\{i\}x\_\{a\(i\)\},\\qquad r\_\{i\}=x\_\{i\}\-\\tilde\{x\}\_\{i\},\(2\)wherex~i\\tilde\{x\}\_\{i\}is the orthogonal projection ofxix\_\{i\}onto the span ofxa​\(i\)x\_\{a\(i\)\}andrir\_\{i\}is the residual the projection discards\.

Thus, every non\-anchor vector is represented by an anchor indexa​\(i\)a\(i\)and a scalar coefficientγi\\gamma\_\{i\}\. For a selected subset of positionsℛ\\mathcal\{R\}, AnchorKV additionally stores a quantized residualr~i\\tilde\{r\}\_\{i\}\. The reconstructed vector is

x^i=x~i\+𝟏​\[i∈ℛ\]​r~i\.\\hat\{x\}\_\{i\}=\\tilde\{x\}\_\{i\}\+\\mathbf\{1\}\[i\\in\\mathcal\{R\}\]\\,\\tilde\{r\}\_\{i\}\.\(3\)Without a stored residual, the vector is reconstructed only from its anchor projection\. Storing the residual restores the token\-specific component that lies outside the anchor direction\.

We apply this representation independently to keys and values\. They share the same anchor positions, but use separate assignments, projection coefficients, and residual setsℛK\\mathcal\{R\}\_\{K\}andℛV\\mathcal\{R\}\_\{V\}\.

#### Residual quantization\.

Stored residuals are quantized at two bits per coordinate\. Before quantization, a randomized Hadamard transform spreads the residual energy across coordinates, reducing sensitivity to outliers\(Ailon and Chazelle[2009](https://arxiv.org/html/2608.02901#bib.bib2); Tsenget al\.[2024](https://arxiv.org/html/2608.02901#bib.bib3); Ashkbooset al\.[2024](https://arxiv.org/html/2608.02901#bib.bib4)\)\. Each rotated residual is normalized using a per\-token absmax scale and quantized with the four\-level Lloyd–Max codebook for a unit Gaussian source\(Lloyd[1982](https://arxiv.org/html/2608.02901#bib.bib7); Roe[1964](https://arxiv.org/html/2608.02901#bib.bib8)\)\. The codes are packed four per byte, so each stored residual requiresD/4D/4bytes for its coordinates plus one scale\. Appendix[A\.1](https://arxiv.org/html/2608.02901#A1.SS1)specifies the codec and storage layout\.

### 3\.2AnchorKV Cache Compression

We now describe how AnchorKV uses this representation to compress the cache\. Compression runs once, at the end of prefill, in three steps\. Each KV head first selects itskkanchor positions\. Every other position is then assigned to an anchor and projected onto it, separately on the key and the value side\. Finally, a subset of positions receives a quantized residual, allocated where it recovers the most output error\. Heads select and assign on their own anchors, but compete for a single residual budget per layer \(Section[3\.4](https://arxiv.org/html/2608.02901#S3.SS4)\)\. Anchors fix both what is stored exactly and the directions everything else is represented along, so we begin there\.

#### Anchor selection\.

The finalWWpositions are always stored exactly and count against the anchor budget\. They serve three purposes at once: the recency window, the observation queries that score earlier positions, and anchor directions that cost no extra slots\. Each earlier position is scored as in SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15)\), by the attention it receives from those queries averaged over the query heads of its KV head, pooled over a positional kernel of widthκ\\kappaso a sharp peak also promotes its neighbors \(Appendix[A\.2](https://arxiv.org/html/2608.02901#A1.SS2)\)\. A fractionρ\\rhoof the remainingk−Wk\-Wslots goes to the highest scores, and the rest is sampled uniformly from the context\. Attention\-based selection identifies positions that the observation queries use directly\. However, an anchor also serves as a direction onto which other vectors are projected, and high\-attention tokens do not necessarily provide good directional coverage of the cache\. The uniformly sampled anchors improve this coverage\.

Key projection is performed before RoPE because positional rotation weakens the directional similarity used for anchor assignment\. Anchor scoring still uses post\-RoPE keys, so it reflects the attention computed by the model\. RoPE\(Suet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib10)\)is a per\-position linear mapRtR\_\{t\}, so it distributes over the decomposition,Rt​Kt=Rt​K~t\+Rt​rtKR\_\{t\}K\_\{t\}=R\_\{t\}\\tilde\{K\}\_\{t\}\+R\_\{t\}r^\{K\}\_\{t\}: thus keys can be stored before rotation and rotated during decoding, and values are unaffected\.

### 3\.3Attention\-Output\-Aware Residual Scoring

Neither the size of a residual nor the attention a position receives determines how much storing it helps: a large residual on a barely attended position changes the output little, and so does heavy attention on a position its anchor already represents well\. AnchorKV therefore ranks residuals by their first\-order effect on the attention output\.

Given a layer, a KV head and a queryqq,

st=q⊤​KtD,α=softmax⁡\(s\),y=∑tαt​Vts\_\{t\}=\\frac\{q^\{\\top\}K\_\{t\}\}\{\\sqrt\{D\}\},\\qquad\\alpha=\\operatorname\{softmax\}\(s\),\\qquad y=\\sum\_\{t\}\\alpha\_\{t\}V\_\{t\}\(4\)denote the exact logits, attention weights, and output\. Writeα^\\hat\{\\alpha\}andy^\\hat\{y\}for the same quantities computed from the reconstructed cache, and setΔ​Kt=K^t−Kt\\Delta K\_\{t\}=\\hat\{K\}\_\{t\}\-K\_\{t\},Δ​Vt=V^t−Vt\\Delta V\_\{t\}=\\hat\{V\}\_\{t\}\-V\_\{t\}, andΔ​st=q⊤​Δ​Kt/D\\Delta s\_\{t\}=q^\{\\top\}\\Delta K\_\{t\}/\\sqrt\{D\}\. For a token without a stored residual,Δ​Kt=−Rt​rtK\\Delta K\_\{t\}=\-R\_\{t\}r\_\{t\}^\{K\}andΔ​Vt=−rtV\\Delta V\_\{t\}=\-r\_\{t\}^\{V\}; otherwise, only quantization error remains\. LetΔ​s¯=∑tαt​Δ​st\\overline\{\\Delta s\}=\\sum\_\{t\}\\alpha\_\{t\}\\Delta s\_\{t\}\. Applying the softmax Jacobian gives

Δ​y≈∑tαt​\(Δ​st−Δ​s¯\)​\(Vt−y\)⏟K channel\+∑tαt​Δ​Vt⏟V channel\.\\Delta y\\approx\\underbrace\{\\sum\_\{t\}\\alpha\_\{t\}\\left\(\\Delta s\_\{t\}\-\\overline\{\\Delta s\}\\right\)\\left\(V\_\{t\}\-y\\right\)\}\_\{\\text\{K channel\}\}\+\\underbrace\{\\sum\_\{t\}\\alpha\_\{t\}\\Delta V\_\{t\}\}\_\{\\text\{V channel\}\}\.\(5\)See Appendix[A\.3](https://arxiv.org/html/2608.02901#A1.SS3)for details\. Key errors act through the attention weights and are modulated byVt−yV\_\{t\}\-y, whereas value errors act directly on the output\. We therefore score key and value residuals separately\.

Under an incoherence approximation in which per\-token contributions are zero\-mean and independent, the expected squared norm of Eq\.[5](https://arxiv.org/html/2608.02901#S3.E5)decomposes into additive token\-wise terms\. Since residuals on each side have equal cost, selecting the largest terms minimizes this approximation for a fixed budget\. A position’s utility is the term it contributes when its residual is dropped, estimated with them=W​\(Hq/H\)m=W\(H\_\{q\}/H\)observation queries of its KV head:

utK\\displaystyle u\_\{t\}^\{K\}=1m​∑w=1m\(αt\(w\)\)2​\(q\(w\)⊤​Rt​rtK\)2D​‖Vt−y\(w\)‖2,\\displaystyle=\\frac\{1\}\{m\}\\sum\_\{w=1\}^\{m\}\\left\(\\alpha\_\{t\}^\{\(w\)\}\\right\)^\{2\}\\frac\{\\left\(q^\{\(w\)\\top\}R\_\{t\}r\_\{t\}^\{K\}\\right\)^\{2\}\}\{D\}\\left\\lVert V\_\{t\}\-y^\{\(w\)\}\\right\\rVert^\{2\},\(6\)utV\\displaystyle u\_\{t\}^\{V\}=1m​∑w=1m\(αt\(w\)\)2​‖rtV‖2\.\\displaystyle=\\frac\{1\}\{m\}\\sum\_\{w=1\}^\{m\}\\left\(\\alpha\_\{t\}^\{\(w\)\}\\right\)^\{2\}\\left\\lVert r\_\{t\}^\{V\}\\right\\rVert^\{2\}\.Here,q\(w\)q^\{\(w\)\},α\(w\)\\alpha^\{\(w\)\}, andy\(w\)y^\{\(w\)\}are the query, exact attention distribution, and exact output for observation queryww\. Utilities are comparable across KV heads within the same side, enabling the pooled allocation in Section[3\.4](https://arxiv.org/html/2608.02901#S3.SS4)\. All required quantities are available at the end of prefill, so utility estimation requires a single pass over the cache\.

Like observation\-window eviction methods\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15); Caiet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib16); Fenget al\.[2025](https://arxiv.org/html/2608.02901#bib.bib17)\), the estimator assumes that the positions recent queries use are the positions decoding will need\. What differs is the consequence of a mistaken estimate: a position AnchorKV prices low is stored coarsely and remains available to later queries\.

#### Why retain every token?

The utility scores determine where residuals are spent, but the anchor\-residual representation provides a separate guarantee that holds regardless of the allocation\. Since no position leaves the softmax, the compressed cache is a perturbation of the exact one, and adding and subtracting∑tα^t​Vt\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}V\_\{t\}splits the output error into

∥y−y^∥≤EK\+EV,\\lVert y\-\\hat\{y\}\\rVert\\leq E\_\{K\}\+E\_\{V\},\(7\)withEK=∥∑t\(αt−α^t\)​Vt∥E\_\{K\}=\\lVert\\sum\_\{t\}\(\\alpha\_\{t\}\-\\hat\{\\alpha\}\_\{t\}\)V\_\{t\}\\rVertinduced by the reconstructed keys andEV=∑tα^t​∥Vt−V^t∥E\_\{V\}=\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\lVert V\_\{t\}\-\\hat\{V\}\_\{t\}\\rVertthe attention\-weighted value error\. Anchors are exact, so the perturbation is driven entirely by non\-anchor positions\. Appendix[A\.4](https://arxiv.org/html/2608.02901#A1.SS4)boundsEKE\_\{K\}by the largest key\-induced logit perturbation, so key error is controlled by a worst case while value error is averaged under attention mass\.

On the key side, retention is not automatically an advantage: a poorly reconstructed key can attract attention mass that its exact counterpart would not receive, which eviction avoids by removing the position outright\. A fixed number of residuals cannot control a maximum over positions, so this term is controlled by the representation instead\. Every key keeps its own projection, so no key error exceeds that position’s own residual norm\.

The value side is where the comparison with eviction is sharp\. Write∥Vt−V^t∥=βV​\(t\)​∥rtV∥\\lVert V\_\{t\}\-\\hat\{V\}\_\{t\}\\rVert=\\beta\_\{V\}\(t\)\\lVert r\_\{t\}^\{V\}\\rVert, withβV​\(t\)=1\\beta\_\{V\}\(t\)=1when no residual is stored and, by Jensen’s inequality,𝔼​\[βV​\(t\)\]≤η\\mathbb\{E\}\[\\beta\_\{V\}\(t\)\]\\leq\\sqrt\{\\eta\}when one is, whereη\\etais the normalized mean\-squared distortion of the quantizer\. Then

EV≤\(maxt∉𝒜⁡βV​\(t\)​∥rtV∥∥Vt∥\)​∑t∉𝒜α^t​∥Vt∥\.E\_\{V\}\\leq\\left\(\\max\_\{t\\notin\\mathcal\{A\}\}\\beta\_\{V\}\(t\)\\frac\{\\lVert r\_\{t\}^\{V\}\\rVert\}\{\\lVert V\_\{t\}\\rVert\}\\right\)\\sum\_\{t\\notin\\mathcal\{A\}\}\\hat\{\\alpha\}\_\{t\}\\lVert V\_\{t\}\\rVert\.\(8\)The sum on the right is the value mass outside the anchor set, the term an eviction method with the same kept set pays for discarding those positions\. AnchorKV pays it scaled by the largest relative projection error left unrepaired, a factor at most one, and one that shrinks as non\-anchor positions gain nearby anchors or receive residuals\. The uniform share of anchor selection buys exactly this coverage\. The comparison is not exact, since the weights in Eq\.[8](https://arxiv.org/html/2608.02901#S3.E8)are the reconstructedα^\\hat\{\\alpha\}and eviction additionally renormalizes the softmax over a smaller support\.

### 3\.4Byte\-Budgeted Residual Allocation

The retained fractionθ\\thetafixes the byte budget of every layer\. Anchors and per\-token metadata are committed first, and whatever remains buys residuals:

N=max⁡\{0,⌊θ​Mfull−Mbasebres⌋\},N=\\max\\left\\\{0,\\,\\left\\lfloor\\frac\{\\theta M\_\{\\mathrm\{full\}\}\-M\_\{\\mathrm\{base\}\}\}\{b\_\{\\mathrm\{res\}\}\}\\right\\rfloor\\right\\\},\(9\)whereMfullM\_\{\\mathrm\{full\}\}is the uncompressed cache of the layer,MbaseM\_\{\\mathrm\{base\}\}the bytes held by its anchors and metadata, andbresb\_\{\\mathrm\{res\}\}the bytes of one quantized residual\. Appendix[A\.1](https://arxiv.org/html/2608.02901#A1.SS1)gives the complete accounting, which charges every stored tensor, so the compressed footprint never exceeds the budget\. Keys receive⌊N/2⌋\\lfloor N/2\\rfloorslots and values the rest\. Within each side, we select the highest\-utility residuals according to Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6), pooling candidates across all KV heads in the layer\. This allows heads with larger estimated output errors to receive more of the residual budget\. The retained fractionθ\\thetais the only user\-facing compression parameter\. We set the anchor budgetkkas a fixed fraction of the context length and keepWW,ρ\\rho, andκ\\kappafixed across layers and experiments; their values are reported in Section[4](https://arxiv.org/html/2608.02901#S4)\.

#### Storage and Decoding

Anchors are stored in bf16, keys before RoPE\. Every other position stores an anchor index and a coefficient per side, and positions inℛK\\mathcal\{R\}\_\{K\}orℛV\\mathcal\{R\}\_\{V\}add a packed residual and its scale, laid out sparsely so positions without one cost nothing\. The denseK,VK,Vare discarded and never rebuilt: keys are reconstructed by Eq\.[3](https://arxiv.org/html/2608.02901#S3.E3)and rotated byRiR\_\{i\}, values by Eq\.[3](https://arxiv.org/html/2608.02901#S3.E3)alone, and generated tokens are appended exactly\. Reconstruction is fused into a FlashAttention\-style tiled kernel\(Dao[2024](https://arxiv.org/html/2608.02901#bib.bib39)\), so each tile reconstructs only the keys and values it consumes and the dense cache is never materialized: the runtime footprint is the compressed footprint\. Section[4\.4](https://arxiv.org/html/2608.02901#S4.SS4)reports memory measurements and decoding throughput\.

## 4Experiments

We evaluate AnchorKV across three model scales and three long\-context benchmarks against eviction and quantization baselines at matched byte budgets \(Sections[4\.1](https://arxiv.org/html/2608.02901#S4.SS1)and[4\.2](https://arxiv.org/html/2608.02901#S4.SS2)\), ablate each design choice and analyze the internal behavior of the compressed cache \(Section[4\.3](https://arxiv.org/html/2608.02901#S4.SS3)\), and measure the throughput and memory cost of the method \(Section[4\.4](https://arxiv.org/html/2608.02901#S4.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x2.png)Figure 2:AnchorKV against eviction and quantization baselines at matched byte budgets\. Rows are Llama\-3\.1\-8B, Mistral\-Small\-3\.1\-24B and Llama\-3\.1\-70B; columns are RULER\-32K, RULER\-64K and LongBench\. At both10×10\\timesand20×20\\times, AnchorKV is the highest\-scoring compressed cache in every panel, and its score at20×20\\timesexceeds that of every baseline at10×10\\times, so the same accuracy is reached with half the memory\. TurboQuant is measured at 3\.5 bits and stays close to the uncompressed cache, but its compression is bounded below5×5\\times\.### 4\.1Setup

#### Models and benchmarks\.

We evaluate three instruction\-tuned models spanning an order of magnitude in scale: Llama\-3\.1\-8B\-Instruct and Llama\-3\.1\-70B\-Instruct\(Team[2024](https://arxiv.org/html/2608.02901#bib.bib36)\), and Mistral\-Small\-3\.1\-24B\-Instruct\(Mistral AI[2025](https://arxiv.org/html/2608.02901#bib.bib41)\), all from public checkpoints with their default chat templates\. RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib37)\)supplies 13 synthetic tasks that evaluate long\-context capabilities, which we run at 32K and 64K\. LongBench\(Baiet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib38)\)covers various long\-context tasks with real documents \(e\.g\., summarization\), leaving less redundancy to exploit\. As a stress test we sweep context length and needle depth on Needle\-in\-a\-Haystack retrieval\(Kamradt[2023](https://arxiv.org/html/2608.02901#bib.bib42)\), using a 64\-digit passkey since shorter ones are close to saturated for every method we compare against\.

#### Baselines\.

We compare AnchorKV against the uncompressed bf16 cache \(FullKV\), against three eviction methods, SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15)\), PyramidKV\(Caiet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib16)\)and AdaKV\(Fenget al\.[2025](https://arxiv.org/html/2608.02901#bib.bib17)\), which allocate a fixed, layer\-adaptive and head\-adaptive budget respectively, and against TurboQuant\(Zandiehet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib5)\), a rotation\-based quantizer that, like AnchorKV, keeps every token\. The eviction baselines run through KVPress\(Devotoet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib43)\)with its default settings\. The two families expose incomparable knobs, a token budget and a bit width, so we convert both to bytes and match all methods at the same per\-layer footprint\. An eviction method keepingBBtokens stores4​B​H​D4BHDbytes, since kept tokens remain in bf16 and discarded ones cost nothing\. For AnchorKV the footprint is the accounting of Eq\.[9](https://arxiv.org/html/2608.02901#S3.E9), which charges every stored tensor including all per\-token metadata\. All ratios are relative to the bf16 cache and run up to20×20\\times\.

#### Implementation\.

AnchorKV uses one configuration for all models and all tasks: windowW=32W=32, anchor budgetk=S/128k=S/128, scored fractionρ=0\.7\\rho=0\.7, pooling kernelκ=7\\kappa=7, anchors in bf16 and residuals at 2 bits\. Compression runs once at the end of prefill and decoding is greedy\. Experiments use NVIDIA A100\-80GB GPUs at batch size 1\. Baseline configurations, and benchmark details are in Appendix[B](https://arxiv.org/html/2608.02901#A2)\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x3.png)Figure 3:Per\-task RULER accuracy on Llama\-3\.1\-8B\-Instruct at20×20\\times, averaged over RULER\-32K and 64K and given as a percentage of the uncompressed score\. AnchorKV matches or exceeds the best eviction baseline on most tasks and never falls below 60%, while the baselines drop under 16% on the tasks that require retrieval against distractors or aggregation over the whole context\.

### 4\.2Main Results

Figure[2](https://arxiv.org/html/2608.02901#S4.F2)compares all methods at matched byte budgets across nine model–benchmark settings: three model scales evaluated on RULER\-32K, RULER\-64K, and LongBench\. Two observations hold in every panel\. First, AnchorKV stays near the uncompressed cache as compression increases, while the eviction baselines degrade sharply\. Second, AnchorKV at20×20\\timesoutperforms every eviction baseline at10×10\\times: the same accuracy is reached with half the cache memory\. TurboQuant also tracks FullKV closely, but its fixed 3\.5\-bit representation caps it near5×5\\times, well short of the regime AnchorKV operates in\. The cost of compression also falls with model scale\. On RULER\-32K at20×20\\times, AnchorKV retains93\.5%93\.5\\%,95\.3%95\.3\\%, and99\.3%99\.3\\%of the FullKV score on Llama\-8B, Mistral\-24B, and Llama\-70B respectively; at 70B the compressed cache scores94\.494\.4against95\.095\.0uncompressed, while the strongest eviction baseline retains only86\.8%86\.8\\%\. The trend is not simply a property of stronger models: on Mistral\-24B the best eviction baseline stays under67%67\\%at both context lengths, no better than at 8B\. LongBench shows the same pattern on real documents, with AnchorKV retaining94\.1%94\.1\\%at 8B and98\.4%98\.4\\%at 70B, ahead of every baseline at every scale\. AnchorKV is therefore most accurate precisely where the cache is most expensive to hold\. Complete per\-dataset results are reported in Appendix[C](https://arxiv.org/html/2608.02901#A3)\.

#### Per\-task consistency\.

Figure[3](https://arxiv.org/html/2608.02901#S4.F3)breaks RULER down by task at20×20\\times\. AnchorKV matches or exceeds the strongest eviction baseline on twelve of thirteen tasks and trails it by a single point on the thirteenth\. Its floor across the suite is60%60\\%of FullKV, while the strongest baseline drops below70%70\\%on four tasks and below16%16\\%on three\. The largest margins appear exactly where eviction’s irreversible decision is most costly, retrieval against distractors and aggregation over the whole context: AnchorKV retains78%78\\%,92%92\\%, and60%60\\%oncwe,single\_3, andmultikey\_3, exceeding the strongest eviction baseline by6767,8888, and4545percentage points, respectively\. Additional task\-level results are given in Appendix[C](https://arxiv.org/html/2608.02901#A3)\.

#### Needle\-in\-a\-haystack\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x4.png)Figure 4:Needle\-in\-a\-haystack retrieval with a 64\-digit passkey on Llama\-3\.1\-8B\-Instruct at5×5\\times, over context lengths from 16K to 128K and needle depths from 0% to 100%, with mean accuracy above each panel\. AnchorKV recovers the needle across nearly the whole grid at 0\.94 against 0\.99 for the uncompressed cache, while the best eviction baseline \(AdaKV\) averages 0\.18\. All methods are exact along the bottom row, where the needle falls inside the recency window they store verbatim\.Figure[4](https://arxiv.org/html/2608.02901#S4.F4)sweeps context length and needle depth at5×5\\times\. AnchorKV averages 0\.94 against 0\.99 for the uncompressed cache and recovers the needle across nearly the whole grid, while the strongest eviction baseline, AdaKV, averages 0\.18 and fails almost everywhere: a needle the observation window did not score is simply gone\. AnchorKV’s few soft cells sit at shallow depths in the longest contexts, where a needle placed early is furthest from the queries that score the anchors\. Full grids at5×5\\timesand10×10\\timesfor all baselines are in Appendix[C](https://arxiv.org/html/2608.02901#A3)\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x5.png)Figure 5:Ablation Study\. Evaluated on Llama\-3\.1\-8B, RULER\-32K, at ×5/×10/×20 \(one variable changed at a time, byte budget fixed\)\. Left: how residuals are ranked; Right: component removals\. AnchorKV leads every variant, and the gaps widen as the budget tightens\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x6.png)Figure 6:Efficiency Analysis\. We profile AnchorKV at a20×20\\timestarget \(Llama\-3\.1\-8B\-Instruct, bf16, one 80 GiB A100; baseline is the full cache under PyTorch SDPA, memory excluding weights\)\. \(a\) Decode peak memory per request, log scale:1717–19×19\\timessmaller\. \(b\) Decode latency\. The fused kernel costs1\.3×1\.3\\timesat 32K, reaches parity near 64K, and is22–3%3\\%faster at 96K–128K \(shaded\)\. \(c\) Throughput against batch size at 64K, each arm walked to its memory ceiling;×\\timesmarks the first batch that fails\. AnchorKV sustainsB=6B\{=\}6againstB=3B\{=\}3, a1\.26×1\.26\\timesgain between frontiers\. \(d\) Largest batch that fits\. The band is the set of configurations only AnchorKV can serve:2×2\\timesthe concurrency from 64K upward\.

### 4\.3Ablations and Analysis

We now isolate the two decisions that define AnchorKV: how anchors are chosen and how residuals are spent\. All ablations use Llama\-3\.1\-8B on RULER\-32K, change one choice at a time, and hold the byte budget fixed\. Each variant runs at5×5\\times,10×10\\timesand20×20\\times, so Figure[5](https://arxiv.org/html/2608.02901#S4.F5)shows both the effect of a choice and how that effect grows as the budget tightens\.

#### Residual Ranking\.

We measure the effect of the metric that decides which tokens receive a residual, while leaving the anchors and the budget untouched\. In place of the utility of Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6)we rank by \(i\) cosine similarity to the nearest anchor, a measure of alignment, \(ii\) residual norm, a measure of approximation error, \(iii\) attention score, a measure of how heavily a token is used, and \(iv\) at random, a check that targeting is needed at all\. The utility leads at every ratio and random ranking is the worst everywhere\. At5×5\\timesthe choice barely matters, since the budget reaches almost every token and there is little left to choose between\. The gaps open as the budget tightens and are widest at20×20\\times, where random placement falls far behind the rest\. What separates the alternatives there is how much of the utility each one recovers\. Each alternative prices one factor and ignores the other\. Attention sees how heavily a token is used but not how badly its anchor represents it, the residual norm sees the reverse, and cosine sees neither, only the angle between the two\. The utility is a first\-order estimate of the attention output error, and Appendix[D\.1](https://arxiv.org/html/2608.02901#A4.SS1)measures that error directly, so the quantity the ranking is designed to control can be read off on its own rather than only through downstream accuracy\.

#### Component Removal\.

We remove one component at a time and measure the change in average score\. We \(i\) select all anchors by attention score, dropping the random share, \(ii\) give every head an equal residual budget, dropping the pooling across heads, \(iii\) store no residuals, spending the whole budget on anchors, which leaves projection onto the dictionary alone, and \(iv\) quantize residuals at 2 bits without the Hadamard rotation and the Lloyd–Max codebook\. Every removal costs accuracy\. Naive quantization is the largest loss and stays roughly constant across ratios, since it degrades every stored residual rather than changing how many are stored\. Storing no residuals costs accuracy at all three ratios even though the freed bytes buy more anchors: a larger dictionary does not replace what a residual restores\. Equal per\-head budgets and attention\-only anchors cost little at5×5\\timesand more as the budget tightens, the same pattern the ranking metrics show\. Appendix[D\.2](https://arxiv.org/html/2608.02901#A4.SS2)maps the share of residuals each head receives: the shares are far from uniform and shift with the input, so an equal split cannot follow them\. Both halves of the figure point the same way: slack hides these choices, and each matters in the regime the method targets, where every byte must go where it buys the most accuracy\.

Effect of Pre\-RoPE\.Finally, we ask if keys should be projected before or after RoPE\. Figure[7](https://arxiv.org/html/2608.02901#S4.F7)reports the cosine to the nearest anchor in each case: the median is0\.270\.27lower once the rotation is applied\. This matters because a token with no stored residual survives only as a scaled copy of its anchor, so its error is set by how closely an anchor direction aligns with it\. The projection is orthogonal, so a cosineccleaves a residual of relative norm1−c2\\sqrt\{1\-c^\{2\}\}: the shift moves the quantity that governs every unrepaired token, both in the utilities of Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6)and the relative value residual of Eq\.[8](https://arxiv.org/html/2608.02901#S3.E8)\. The ordering holds in all eight depth bands and four RULER categories, so it reflects the rotation rather than a particular layer or task\. Fitting before the rotation is thus better: every unrepaired token sits closer to its anchor, so the same dictionary covers more of the cache at no extra cost\. AnchorKV therefore projects keys pre\-RoPE and rotates them at decoding\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x7.png)Figure 7:Pre\-RoPE vs\. Post\-RoPE Key Projection\. We measure the per\-token cosine to the nearest anchor on Llama\-3\.1\-8B at RULER 32K, before and after RoPE\. The two labeled curves pool four tasks, one per RULER category \(multikey\_3,vt,fwe,qa\_2\), with 20 samples each; the faint curves split the same data into eight bands of four consecutive layers\. RoPE lowers the median cosine by0\.270\.27, and the ordering is the same in every band\.

### 4\.4Efficiency and Memory

AnchorKV decodes directly from the compressed representation rather than reconstructing the dense KV cache\. Our efficient Triton implementation follows a FlashAttention\-style tiled execution pattern\(Dao[2024](https://arxiv.org/html/2608.02901#bib.bib39)\): each key tile is reconstructed only when consumed, rotated with RoPE, and contracted with the query in registers, after which only its attention scores are retained\. Values are accumulated directly from their anchor projections and sparse residuals; by linearity of the inverse Hadamard transform, the residual contribution is transformed once after accumulation rather than once per token\. Reconstructed keys and values are therefore never written to global memory, yielding an approximately19×19\\timesreduction in steady\-state decode peak memory at a20×20\\timestarget \(Figure[6](https://arxiv.org/html/2608.02901#S4.F6)a\)\. Appendix[E](https://arxiv.org/html/2608.02901#A5)provides the kernel and memory\-accounting details\. On Llama\-3\.1\-8B\-Instruct, decode latency is higher at short contexts and comparable from 64K upward \(Figure[6](https://arxiv.org/html/2608.02901#S4.F6)b\)\. The smaller footprint also buys concurrency: at 64K it doubles the maximum batch size and, with each method at its memory frontier, improves throughput by1\.26×1\.26\\times\(Figure[6](https://arxiv.org/html/2608.02901#S4.F6)c\-d\)\. Overall, AnchorKV pairs a much smaller decode footprint with competitive long\-context latency and higher throughput\.

## 5Conclusion

We introduced AnchorKV, a KV cache compression method that keeps every token at a fraction of the memory\. Each token is stored in an anchor\-residual form, and residuals go where they most preserve the attention output\. At20×20\\times, AnchorKV outscores every eviction baseline at10×10\\times, reaching higher accuracy with half the memory\. At the 70B scale it retains up to 99\.3% of the uncompressed accuracy\. A tiled implementation carries these savings into decoding\. Under a fixed memory budget, it supports higher throughput and more concurrent requests\. Together, these results show that aggressive compression, full\-context reach and practical serving efficiency are not in conflict\.

## References

- TurboQuant: community reproduction\.Note:https://github\.com/0xsero/turboquantAccessed: 2026\-07\-25Cited by:[§B\.5](https://arxiv.org/html/2608.02901#A2.SS5.p2.1)\.
- N\. Ailon and B\. Chazelle \(2009\)The fast johnson–lindenstrauss transform and approximate nearest neighbors\.SIAM J\. Comput\.39\(1\),pp\. 302–322\.External Links:[Link](https://doi.org/10.1137/060673096),[Document](https://dx.doi.org/10.1137/060673096)Cited by:[§A\.1](https://arxiv.org/html/2608.02901#A1.SS1.SSS0.Px1.p1.7),[§3\.1](https://arxiv.org/html/2608.02901#S3.SS1.SSS0.Px1.p1.1)\.
- AmesianX \(2026\)TurboQuant\.Note:https://github\.com/AmesianX/TurboQuantAccessed: 2026\-07\-25Cited by:[§B\.5](https://arxiv.org/html/2608.02901#A2.SS5.p2.1)\.
- S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. Hensman \(2024\)QuaRot: outlier\-free 4\-bit inference in rotated llms\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html)Cited by:[§3\.1](https://arxiv.org/html/2608.02901#S3.SS1.SSS0.Px1.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: A bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2024, Bangkok, Thailand, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),pp\. 3119–3137\.External Links:[Link](https://doi.org/10.18653/v1/2024.acl-long.172),[Document](https://dx.doi.org/10.18653/V1/2024.ACL-LONG.172)Cited by:[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px1.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, B\. Chang, J\. Hu, and W\. Xiao \(2024\)PyramidKV: dynamic KV cache compression based on pyramidal information funneling\.CoRRabs/2406\.02069\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.02069),[Document](https://dx.doi.org/10.48550/ARXIV.2406.02069),2406\.02069Cited by:[§B\.4](https://arxiv.org/html/2608.02901#A2.SS4.p1.2),[§1](https://arxiv.org/html/2608.02901#S1.p2.1),[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2608.02901#S3.SS3.p4.1),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px2.p1.3)\.
- C\. Chang, C\. Lin, Y\. Akhauri, W\. Lin, K\. Wu, L\. Ceze, and M\. S\. Abdelfattah \(2025\)XKV: cross\-layer SVD for kv\-cache compression\.CoRRabs/2503\.18893\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.18893),[Document](https://dx.doi.org/10.48550/ARXIV.2503.18893),2503\.18893Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, and K\. Wu \(2024\)Palu: compressing kv\-cache with low\-rank projection\.CoRRabs/2407\.21118\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.21118),[Document](https://dx.doi.org/10.48550/ARXIV.2407.21118),2407\.21118Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- T\. Dao \(2024\)FlashAttention\-2: faster attention with better parallelism and work partitioning\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=mZn2Xyh9Ec)Cited by:[§3\.4](https://arxiv.org/html/2608.02901#S3.SS4.SSS0.Px1.p1.4),[§4\.4](https://arxiv.org/html/2608.02901#S4.SS4.p1.3)\.
- A\. Devoto, M\. Jeblick, and S\. Jégou \(2025\)Expected attention: KV cache compression by estimating attention from future queries distribution\.External Links:2510\.00636Cited by:[§B\.4](https://arxiv.org/html/2608.02901#A2.SS4.p1.2),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px2.p1.3)\.
- Y\. Feng, J\. Lv, Y\. Cao, X\. Xie, and S\. K\. Zhou \(2025\)Ada\-kv: optimizing KV cache eviction by adaptive budget allocation for efficient LLM inference\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2025/hash/a40ff56daab9f4808b1e18350c8a11ce-Abstract-Conference.html)Cited by:[§B\.4](https://arxiv.org/html/2608.02901#A2.SS4.p1.2),[§1](https://arxiv.org/html/2608.02901#S1.p2.1),[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2608.02901#S3.SS3.p4.1),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px2.p1.3)\.
- Y\. Feng, J\. Lv, H\. Guo, Y\. Cao, S\. K\. Zhou, and X\. Xie \(2026\)CriticalKV: optimizing kv cache eviction from an output perturbation perspective\.External Links:2502\.03805,[Link](https://arxiv.org/abs/2502.03805)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Gu, X\. Liang, J\. Zhao, and E\. Diao \(2025\)OBCache: optimal brain KV cache pruning for efficient long\-context LLM inference\.CoRRabs/2510\.07651\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.07651),[Document](https://dx.doi.org/10.48550/ARXIV.2510.07651),2510\.07651Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length LLM inference with KV cache quantization\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/028fcbcf85435d39a40c4d61b42c99a4-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.02901#S1.p3.1),[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- C\. R\. C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. Maheswaran, S\. Zhao, J\. Paik, M\. W\. Mahoney, K\. Keutzer, and A\. Gholami \(2025\)Squeezed attention: accelerating long context length LLM inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2025, Vienna, Austria, July 27 \- August 1, 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),pp\. 32631–32652\.External Links:[Link](https://doi.org/10.18653/v1/2025.acl-long.1568),[Document](https://dx.doi.org/10.18653/V1/2025.ACL-LONG.1568)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.CoRRabs/2404\.06654\.External Links:[Link](https://doi.org/10.48550/arXiv.2404.06654),[Document](https://dx.doi.org/10.48550/ARXIV.2404.06654),2404\.06654Cited by:[§B\.2](https://arxiv.org/html/2608.02901#A2.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.02901#S1.p5.3),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px1.p1.1)\.
- J\. Hu, S\. Wang, Y\. He, P\. Gong, J\. Yi, J\. Zhang, Y\. Bai, R\. Chen, G\. Zhang, C\. Li, and K\. Yuan \(2026\)CentroidKV: efficient long\-context LLM inference via KV cache clustering\.Trans\. Mach\. Learn\. Res\.2026\.External Links:[Link](https://openreview.net/forum?id=T3EeupQhGj)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Kamradt \(2023\)Needle in a haystack: pressure testing llms\.Note:https://github\.com/gkamradt/LLMTest\_NeedleInAHaystackAccessed: 2026\-07\-25Cited by:[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px1.p1.1)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient KV cache compression recipe for near\-lossless generative inference of LLM\.CoRRabs/2403\.05527\.External Links:[Link](https://doi.org/10.48550/arXiv.2403.05527),[Document](https://dx.doi.org/10.48550/ARXIV.2403.05527),2403\.05527Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- J\. Li, Y\. Zhang, M\. Y\. Hassan, T\. Chafekar, T\. Cai, Z\. Ren, P\. Guo, F\. Karimzadeh, C\. Reed, C\. Wang, and C\. Gan \(2025\)CommVQ: commutative vector quantization for KV cache compression\.InForty\-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13\-19, 2025,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267\.External Links:[Link](https://proceedings.mlr.press/v267/li25du.html)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/28ab418242603e0f7323e54185d19bde-Abstract-Conference.html)Cited by:[§A\.2](https://arxiv.org/html/2608.02901#A1.SS2.p1.1),[§B\.2](https://arxiv.org/html/2608.02901#A2.SS2.SSS0.Px2.p1.1),[§B\.4](https://arxiv.org/html/2608.02901#A2.SS4.p1.2),[§1](https://arxiv.org/html/2608.02901#S1.p2.1),[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2608.02901#S3.SS2.SSS0.Px1.p1.4),[§3\.3](https://arxiv.org/html/2608.02901#S3.SS3.p4.1),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px2.p1.3)\.
- A\. Liu, J\. Liu, Z\. Pan, Y\. He, R\. Haffari, and B\. Zhuang \(2024a\)MiniCache: KV cache compression in depth dimension for large language models\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/fd0705710bf01b88a60a3d479ea341d9-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- G\. Liu, C\. Li, J\. Zhao, C\. Zhang, and M\. Guo \(2025\)ClusterKV: manipulating LLM KV cache in semantic space for recallable compression\.In62nd ACM/IEEE Design Automation Conference, DAC 2025, San Francisco, CA, USA, June 22\-25, 2025,pp\. 1–7\.External Links:[Link](https://doi.org/10.1109/DAC63849.2025.11132479),[Document](https://dx.doi.org/10.1109/DAC63849.2025.11132479)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for LLM KV cache compression at test time\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/a452a7c6c463e4ae8fbdc614c6e983e6-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. \(\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024b\)KIVI: A tuning\-free asymmetric 2bit quantization for KV cache\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 32332–32344\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bz.html)Cited by:[§B\.2](https://arxiv.org/html/2608.02901#A2.SS2.SSS0.Px3.p1.1),[§C\.1](https://arxiv.org/html/2608.02901#A3.SS1.p1.2),[§1](https://arxiv.org/html/2608.02901#S1.p3.1),[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- S\. P\. Lloyd \(1982\)Least squares quantization in PCM\.IEEE Trans\. Inf\. Theory28\(2\),pp\. 129–136\.External Links:[Link](https://doi.org/10.1109/TIT.1982.1056489),[Document](https://dx.doi.org/10.1109/TIT.1982.1056489)Cited by:[§A\.1](https://arxiv.org/html/2608.02901#A1.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.02901#S3.SS1.SSS0.Px1.p1.1)\.
- Mistral AI \(2025\)Mistral small 3\.1\.Note:https://mistral\.ai/news/mistral\-small\-3\-1/Accessed: 2026\-07\-25Cited by:[§B\.1](https://arxiv.org/html/2608.02901#A2.SS1.p1.3),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px1.p1.1)\.
- A\. Mohtashami and M\. Jaggi \(2023\)Landmark attention: random\-access infinite context length for transformers\.External Links:2305\.16300,[Link](https://arxiv.org/abs/2305.16300)Cited by:[§B\.2](https://arxiv.org/html/2608.02901#A2.SS2.SSS0.Px3.p1.1)\.
- M\. Oren, M\. Hassid, Y\. Nir, Y\. Adi, and R\. Schwartz \(2024\)Transformers are multi\-state rnns\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 18724–18741\.External Links:[Link](https://doi.org/10.18653/v1/2024.emnlp-main.1043),[Document](https://dx.doi.org/10.18653/V1/2024.EMNLP-MAIN.1043)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1)\.
- G\. M\. Roe \(1964\)Quantizing for minimum distortion \(corresp\.\)\.IEEE Trans\. Inf\. Theory10\(4\),pp\. 384–385\.External Links:[Link](https://doi.org/10.1109/TIT.1964.1053693),[Document](https://dx.doi.org/10.1109/TIT.1964.1053693)Cited by:[§A\.1](https://arxiv.org/html/2608.02901#A1.SS1.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.02901#S3.SS1.SSS0.Px1.p1.1)\.
- U\. Saxena and K\. Roy \(2025\)KVLinC : KV cache quantization with hadamard rotation and linear correction\.CoRRabs/2510\.05373\.External Links:[Link](https://doi.org/10.48550/arXiv.2510.05373),[Document](https://dx.doi.org/10.48550/ARXIV.2510.05373),2510\.05373Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px2.p1.2)\.
- J\. Su, M\. H\. M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu \(2024\)RoFormer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.External Links:[Link](https://doi.org/10.1016/j.neucom.2023.127063),[Document](https://dx.doi.org/10.1016/J.NEUCOM.2023.127063)Cited by:[§3\.2](https://arxiv.org/html/2608.02901#S3.SS2.SSS0.Px1.p2.2)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)QUEST: query\-aware sparsity for efficient long\-context LLM inference\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 47901–47911\.External Links:[Link](https://proceedings.mlr.press/v235/tang24l.html)Cited by:[§1](https://arxiv.org/html/2608.02901#S1.p2.1)\.
- L\. Team \(2024\)The llama 3 herd of models\.CoRRabs/2407\.21783\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.21783),[Document](https://dx.doi.org/10.48550/ARXIV.2407.21783),2407\.21783Cited by:[§B\.1](https://arxiv.org/html/2608.02901#A2.SS1.p1.3),[§1](https://arxiv.org/html/2608.02901#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px1.p1.1)\.
- TheTom \(2026\)llama\-cpp\-turboquant\.Note:https://github\.com/TheTom/llama\-cpp\-turboquantAccessed: 2026\-07\-25Cited by:[§B\.5](https://arxiv.org/html/2608.02901#A2.SS5.p2.1)\.
- A\. Tseng, J\. Chee, Q\. Sun, V\. Kuleshov, and C\. D\. Sa \(2024\)QuIP\#: even better LLM quantization with hadamard incoherence and lattice codebooks\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 48630–48656\.External Links:[Link](https://proceedings.mlr.press/v235/tseng24a.html)Cited by:[§3\.1](https://arxiv.org/html/2608.02901#S3.SS1.SSS0.Px1.p1.1)\.
- vLLM \(2026\)TurboQuant in vLLM\.Note:https://vllm\.ai/blog/2026\-05\-11\-turboquantAccessed: 2026\-07\-25Cited by:[§B\.5](https://arxiv.org/html/2608.02901#A2.SS5.p2.1)\.
- Z\. Wan, X\. Wu, Y\. Zhang, Y\. Xin, C\. Tao, Z\. Zhu, X\. Wang, S\. Luo, J\. Xiong, and M\. Zhang \(2024\)D2O: dynamic discriminative operations for efficient generative inference of large language models\.CoRRabs/2406\.13035\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.13035),[Document](https://dx.doi.org/10.48550/ARXIV.2406.13035),2406\.13035Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Wang, B\. Jin, Z\. Yu, and M\. Zhang \(2024\)Model tells you where to merge: adaptive KV cache merging for llms on long\-context tasks\.CoRRabs/2407\.08454\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.08454),[Document](https://dx.doi.org/10.48550/ARXIV.2407.08454),2407\.08454Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Xiao, P\. Zhang, X\. Han, G\. Xiao, Y\. Lin, Z\. Zhang, Z\. Liu, and M\. Sun \(2024a\)InfLLM: training\-free long\-context extrapolation for llms with an efficient context memory\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2024/hash/d842425e4bf79ba039352da0f658a906-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.02901#S1.p2.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024b\)Efficient streaming language models with attention sinks\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=NG7sS51zVF)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Zandieh, M\. Daliri, M\. Hadian, and V\. Mirrokni \(2025\)TurboQuant: online vector quantization with near\-optimal distortion rate\.CoRRabs/2504\.19874\.External Links:[Link](https://doi.org/10.48550/arXiv.2504.19874),[Document](https://dx.doi.org/10.48550/ARXIV.2504.19874),2504\.19874Cited by:[§B\.5](https://arxiv.org/html/2608.02901#A2.SS5.p1.1),[§1](https://arxiv.org/html/2608.02901#S1.p3.1),[§4\.1](https://arxiv.org/html/2608.02901#S4.SS1.SSS0.Px2.p1.3)\.
- Y\. Zhang, Y\. Du, G\. Luo, Y\. Zhong, Z\. Zhang, S\. Liu, and R\. Ji \(2024\)CaM: cache merging for memory\-efficient llms inference\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 58840–58850\.External Links:[Link](https://proceedings.mlr.press/v235/zhang24n.html)Cited by:[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. W\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2608.02901#S1.p2.1),[§2](https://arxiv.org/html/2608.02901#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AMethod \- Further details

### A\.1Residual Codec and Storage Layout

This appendix gives the codec summarized in Section[3\.1](https://arxiv.org/html/2608.02901#S3.SS1)and the byte accounting behind Eq\.[9](https://arxiv.org/html/2608.02901#S3.E9)\.

#### Rotation\.

Residuals are rotated before quantization by

U=1D​ℋD​diag​\(s\),s∈\{±1\}D,U=\\tfrac\{1\}\{\\sqrt\{D\}\}\\mathcal\{H\}\_\{D\}\\,\\mathrm\{diag\}\(s\),\\qquad s\\in\\\{\\pm 1\\\}^\{D\},\(10\)withℋD\\mathcal\{H\}\_\{D\}the Walsh\-Hadamard matrix andssa sign pattern drawn once from a fixed seed\.UUis orthogonal, so it preserves norms and is inverted by its transpose, and it applies inO​\(D​log⁡D\)O\(D\\log D\)by the fast Walsh\-Hadamard transform\. Each coordinate ofU​riUr\_\{i\}is a signed average of the coordinates ofrir\_\{i\}, so unless the residual energy sits in a few coordinates, each rotated coordinate is approximately Gaussian with variance‖ri‖2/D\\\|r\_\{i\}\\\|^\{2\}/D\(Ailon and Chazelle[2009](https://arxiv.org/html/2608.02901#bib.bib2)\)\. The seed is shared by every layer and head, so the transform is not stored\.

#### Codebook\.

Each rotated coordinate is divided by the per\-token absmax scale and quantized with the four\-level Lloyd\-Max codebook for a unit Gaussian source\(Lloyd[1982](https://arxiv.org/html/2608.02901#bib.bib7); Roe[1964](https://arxiv.org/html/2608.02901#bib.bib8)\), which matches the marginals the rotation produces\. We write𝒞\\mathcal\{C\}for the encoder and

η=𝔼​\[‖ri−r~i‖2\]/‖ri‖2\\eta=\\mathbb\{E\}\[\\\|r\_\{i\}\-\\tilde\{r\}\_\{i\}\\\|^\{2\}\]/\\\|r\_\{i\}\\\|^\{2\}\(11\)for the relative squared distortion the codec leaves, a property of the codec and not of the token\. A non\-anchor vector is stored as

\(a​\(i\),γi\)​for​i∉ℛ,\(a​\(i\),γi,𝐪i\)​for​i∈ℛ,\\big\(a\(i\),\\ \\gamma\_\{i\}\\big\)\\ \\text\{for \}i\\notin\\mathcal\{R\},\\qquad\\big\(a\(i\),\\ \\gamma\_\{i\},\\ \{\\mathbf\{q\}\}\_\{i\}\\big\)\\ \\text\{for \}i\\in\\mathcal\{R\},\(12\)with𝐪i\{\\mathbf\{q\}\}\_\{i\}the packed code, and decoded asr~i=U⊤​𝒞−1​\(𝐪i\)\\tilde\{r\}\_\{i\}=U^\{\\top\}\\mathcal\{C\}^\{\-1\}\(\{\\mathbf\{q\}\}\_\{i\}\)before Eq\.[3](https://arxiv.org/html/2608.02901#S3.E3)is applied\.

#### Layout\.

Table[1](https://arxiv.org/html/2608.02901#A1.T1)lists the tensors a compressed layer holds, charging every byte the decode kernel touches\. Anchor positions are shared by the two sides, so a selected token contributes one key vector and one value vector\. Because keys are stored before RoPE, original positions are kept explicitly: an int32 per non\-anchor token, shared across heads and sides, and an int64 per selected anchor\. Every non\-window token carries an index and a coefficient per side, anchors included \(γ=1\\gamma=1on themselves\), keeping the per\-token arrays uniform\. Residuals are held sparsely: a bitmask of 64\-bit words marks which tokens carry one, the codes are packed in position order, and per\-word prefix counts with per\-head offsets make the position→\\toslot lookupO​\(1\)O\(1\)\. WritingP=S−WP=S\-Wfor the non\-window positions and summing the data\-independent rows gives

Mbase=H​\[4​k​D\+8​\(k−W\)\+2​P​\(ba\+bγ\)\]\\displaystyle M\_\{\\mathrm\{base\}\}=H\\Big\[4kD\+8\(k\{\-\}W\)\+2P\(b\_\{a\}\{\+\}b\_\{\\gamma\}\)\\Big\]\+24​H​⌈P/64⌉\+8​\(H\+1\)\+4​P,\\displaystyle\+24H\\big\\lceil P/64\\big\\rceil\+8\(H\{\+\}1\)\+4P,\(13\)withbab\_\{a\}the anchor index width andbγb\_\{\\gamma\}the coefficient width in bytes\. A key residual costscK=D/4\+4c\_\{K\}=D/4\+4\(codes plus an fp32 scale\); a value residual adds a one\-byte chunk\-local position,cV=D/4\+5c\_\{V\}=D/4\+5, since the value pass visits only stored residuals and must map each slot back to its position, while the key pass walks all positions and reuses the bitmask as its index\. Eq\.[9](https://arxiv.org/html/2608.02901#S3.E9)divides by these costs\.

Table 1:Per\-layer storage, withP=S−WP=S\{\-\}Wthe non\-window positions\. The upper block is data independent and formsMbaseM\_\{\\mathrm\{base\}\}; the lower block is what the byte budget of Eq\.[9](https://arxiv.org/html/2608.02901#S3.E9)buys, atcK=D/4\+4c\_\{K\}=D/4\+4bytes per key residual andcV=D/4\+5c\_\{V\}=D/4\+5per value residual\.
#### Worked example\.

AtS=32S=32K,D=128D=128,H=8H=8,W=32W=32andk=S/128=256k=S/128=256, the uncompressed layer isMfull=4​S​H​D=134\.2M\_\{\\mathrm\{full\}\}=4SHD=134\.2MB\. Withba=bγ=2b\_\{a\}=b\_\{\\gamma\}=2, Eq\.[13](https://arxiv.org/html/2608.02901#A1.E13)commits3\.393\.39MB, or2\.5%2\.5\\%of the layer, leaving3\.323\.32MB atθ=0\.05\\theta=0\.05forN=91,056N=91\{,\}056residuals out of the2​H​P=523,7762HP=523\{,\}776token\-sides that hold an assignment\. The committed part grows withSSat a fixed rate per token, so the fraction of the budget available for residuals is close to constant across the context lengths we evaluate\.

### A\.2Anchor Selection

In this section we provide more details on how we choose anchors following SnapKV’s setup\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15)\)

score​\(t\)=apκ​\(1m​∑wsm​\(q\(w\)⊤​K⊤/D\)t\),\\mathrm\{score\}\(t\)=\\mathrm\{ap\}\_\{\\kappa\}\\\!\\Big\(\\tfrac\{1\}\{m\}\\textstyle\\sum\_\{w\}\\text\{sm\}\\big\(q^\{\(w\)\\top\}K^\{\\top\}/\\sqrt\{D\}\\big\)\_\{t\}\\Big\),\(14\)
whereapdenotes average pooling,smis the softmax function andm=W​\(Hq/H\)m=W\(H\_\{q\}/H\)whereHHandHq≥HH\_\{q\}\\geq Hare the number of KV and query heads andWWis the window size\.

### A\.3Derivation of the Utility

This appendix derives Eq\.[5](https://arxiv.org/html/2608.02901#S3.E5)and the additive estimate behind the utilities of Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6)\. Fix a layer, a KV head and a queryqq, and writest=q⊤​Kt/Ds\_\{t\}=q^\{\\top\}K\_\{t\}/\\sqrt\{D\},α=softmax​\(s\)\\alpha=\\mathrm\{softmax\}\(s\)andy=∑tαt​Vty=\\sum\_\{t\}\\alpha\_\{t\}V\_\{t\}\.

#### Two channels\.

Compression perturbsKt→Kt\+Δ​KtK\_\{t\}\\to K\_\{t\}\+\\Delta K\_\{t\}andVt→Vt\+Δ​VtV\_\{t\}\\to V\_\{t\}\+\\Delta V\_\{t\}, and hence the logits byΔ​st=q⊤​Δ​Kt/D\\Delta s\_\{t\}=q^\{\\top\}\\Delta K\_\{t\}/\\sqrt\{D\}\. The softmax Jacobian is∂αt/∂su=αt​\(δt​u−αu\)\\partial\\alpha\_\{t\}/\\partial s\_\{u\}=\\alpha\_\{t\}\(\\delta\_\{tu\}\-\\alpha\_\{u\}\), so to first order

Δ​αt=αt​\(Δ​st−Δ​s¯\),Δ​s¯=∑uαu​Δ​su,\\Delta\\alpha\_\{t\}=\\alpha\_\{t\}\\big\(\\Delta s\_\{t\}\-\\overline\{\\Delta s\}\\big\),\\qquad\\overline\{\\Delta s\}=\\textstyle\\sum\_\{u\}\\alpha\_\{u\}\\Delta s\_\{u\},\(15\)which shows that a uniform shift of every logit changes nothing and only the spread of logit errors matters\. Expandingy^=∑t\(αt\+Δ​αt\)​\(Vt\+Δ​Vt\)\\hat\{y\}=\\sum\_\{t\}\(\\alpha\_\{t\}\+\\Delta\\alpha\_\{t\}\)\(V\_\{t\}\+\\Delta V\_\{t\}\)and dropping the second\-order term givesΔ​y≈∑tΔ​αt​Vt\+∑tαt​Δ​Vt\\Delta y\\approx\\sum\_\{t\}\\Delta\\alpha\_\{t\}V\_\{t\}\+\\sum\_\{t\}\\alpha\_\{t\}\\Delta V\_\{t\}\. Since∑tΔ​αt=0\\sum\_\{t\}\\Delta\\alpha\_\{t\}=0, the values may be centered byyywithout changing the first sum, which yields Eq\.[5](https://arxiv.org/html/2608.02901#S3.E5)\. Centering is what makes the key channel read inVt−yV\_\{t\}\-y: a key error re\-weights tokenttagainst the output the query has already formed, so it is harmless on a token whose value agrees with that output\.

#### Additivity\.

Eq\.[5](https://arxiv.org/html/2608.02901#S3.E5)prices the whole cache jointly\. To turn it into a per\-token quantity we adopt an incoherence model: seen against the query and value geometry, per\-token errors are zero\-mean and independent across tokens, and key errors are independent of value errors\. The assumption is that residual directions of distinct tokens are incoherent in high dimension, which the randomized rotation of Eq\.[10](https://arxiv.org/html/2608.02901#A1.E10)supports for the stored residuals\. Under the model the cross terms between distinct tokens and between the two channels vanish in expectation, and the common modeΔ​s¯\\overline\{\\Delta s\}concentrates near zero, leaving

𝔼​‖Δ​y‖2≈∑tαt2​𝔼​\[Δ​st2\]​‖Vt−y‖2\+∑tαt2​𝔼​‖Δ​Vt‖2,\\mathbb\{E\}\\\|\\Delta y\\\|^\{2\}\\approx\\sum\_\{t\}\\alpha\_\{t\}^\{2\}\\,\\mathbb\{E\}\[\\Delta s\_\{t\}^\{2\}\]\\,\\\|V\_\{t\}\-y\\\|^\{2\}\+\\sum\_\{t\}\\alpha\_\{t\}^\{2\}\\,\\mathbb\{E\}\\\|\\Delta V\_\{t\}\\\|^\{2\},\(16\)an equality in expectation under the model rather than a bound\. Its value is additivity: each token contributes one term, storing a token’s residual scales that term and leaves every other term untouched, and every residual costs the same bytes, so minimizing the total under a fixed number of residuals is taking the largest terms\.

#### From the estimate to the utilities\.

A non\-anchor token with no stored residual carries the dropped residuals,Δ​Kt=−Rt​rtK\\Delta K\_\{t\}=\-R\_\{t\}r^\{K\}\_\{t\}andΔ​Vt=−rtV\\Delta V\_\{t\}=\-r^\{V\}\_\{t\}, so its two terms areαt2​\(q⊤​Rt​rtK\)2​‖Vt−y‖2/D\\alpha\_\{t\}^\{2\}\(q^\{\\top\}R\_\{t\}r^\{K\}\_\{t\}\)^\{2\}\\\|V\_\{t\}\-y\\\|^\{2\}/Dandαt2​‖rtV‖2\\alpha\_\{t\}^\{2\}\\\|r^\{V\}\_\{t\}\\\|^\{2\}\. Storing the residual replaces the dropped vector by the quantization error, of relative squared energyη\\eta, so the term becomesη\\etatimes as large and the slot gains\(1−η\)\(1\-\\eta\)of it\. The factor is common to every token and affects no ranking, so it is dropped from Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6)\. Anchors and window tokens are exact and contribute nothing, so they never compete for slots\. Averaging the two terms over them=W⋅\(Hq/H\)m=W\\cdot\(H\_\{q\}/H\)observation\-window rowsq\(w\)q^\{\(w\)\}of the head’s query group, in place of an expectation over queries that decoding has not yet issued, gives Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6)\.

### A\.4Output Error Bound Theorem

This appendix states and proves the worst\-case bound referenced in Section[3\.3](https://arxiv.org/html/2608.02901#S3.SS3)\. It holds for any reconstruction of the keys and values, so it does not depend on how anchors are selected or how residuals are allocated\.

Fix a layer, a KV head and a queryq∈ℝDq\\in\\mathbb\{R\}^\{D\}\. LetK,V∈ℝS×DK,V\\in\\mathbb\{R\}^\{S\\times D\}be the exact keys and values used by attention, with keys taken post\-RoPE, and letK^,V^\\hat\{K\},\\hat\{V\}be their AnchorKV reconstructions\. The exact and reconstructed logits, attention weights and outputs are

st\\displaystyle s\_\{t\}=q⊤​Kt/D,\\displaystyle=q^\{\\top\}K\_\{t\}/\\sqrt\{D\},α\\displaystyle\\alpha=softmax​\(s\),\\displaystyle=\\mathrm\{softmax\}\(s\),y\\displaystyle y=∑tαt​Vt,\\displaystyle=\\textstyle\\sum\_\{t\}\\alpha\_\{t\}V\_\{t\},s^t\\displaystyle\\hat\{s\}\_\{t\}=q⊤​K^t/D,\\displaystyle=q^\{\\top\}\\hat\{K\}\_\{t\}/\\sqrt\{D\},α^\\displaystyle\\hat\{\\alpha\}=softmax​\(s^\),\\displaystyle=\\mathrm\{softmax\}\(\\hat\{s\}\),y^\\displaystyle\\hat\{y\}=∑tα^t​V^t\.\\displaystyle=\\textstyle\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\hat\{V\}\_\{t\}\.Three constants control the bound: the largest exact value normVmax=maxt⁡‖Vt‖V\_\{\\max\}=\\max\_\{t\}\\\|V\_\{t\}\\\|, the largest key reconstruction errorΔK=maxt⁡‖Kt−K^t‖\\Delta\_\{K\}=\\max\_\{t\}\\\|K\_\{t\}\-\\hat\{K\}\_\{t\}\\\|, and the logit perturbation it induces,

μ=‖q‖​ΔKD\.\\mu=\\frac\{\\\|q\\\|\\,\\Delta\_\{K\}\}\{\\sqrt\{D\}\}\.
###### Theorem 1\(Output error of AnchorKV\)\.

The attention output error satisfies

‖y−y^‖≤∑tα^t​‖Vt−V^t‖⏟EV\+2​Vmax​tanh⁡\(μ\)⏟EK\.\\\|y\-\\hat\{y\}\\\|\\;\\leq\\;\\underbrace\{\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\,\\\|V\_\{t\}\-\\hat\{V\}\_\{t\}\\\|\}\_\{E\_\{V\}\}\\;\+\\;\\underbrace\{2\\,V\_\{\\max\}\\,\\tanh\(\\mu\)\}\_\{E\_\{K\}\}\.\(17\)Sincetanh⁡\(μ\)≤μ\\tanh\(\\mu\)\\leq\\mu, this implies the linear bound

‖y−y^‖≤∑tα^t​‖Vt−V^t‖\+2​Vmax​‖q‖D​ΔK\.\\\|y\-\\hat\{y\}\\\|\\;\\leq\\;\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\,\\\|V\_\{t\}\-\\hat\{V\}\_\{t\}\\\|\\;\+\\;\\frac\{2\\,V\_\{\\max\}\\,\\\|q\\\|\}\{\\sqrt\{D\}\}\\,\\Delta\_\{K\}\.\(18\)

The two terms differ in structure\.EVE\_\{V\}is an attention\-weighted average, so a value error on a token the query barely attends to contributes little\.EKE\_\{K\}depends only on the worst key error, since one badly reconstructed key can shift the softmax normalization for every token\.

Both terms are supported on non\-anchor tokens alone: anchors and window tokens are stored exactly, so their reconstruction errors vanish\. A non\-anchor token whose residual is dropped has key error exactly‖rtK‖\\\|r^\{K\}\_\{t\}\\\|and value error exactly‖rtV‖\\\|r^\{V\}\_\{t\}\\\|, and one whose residual is stored has errorη\\sqrt\{\\eta\}times these in expectation\. Every key error is therefore at most that token’s own residual norm, which is what boundsΔK\\Delta\_\{K\}\. Since RoPE is orthogonal,‖Kt−K^t‖\\\|K\_\{t\}\-\\hat\{K\}\_\{t\}\\\|is the same whether measured before or after the rotation\.

###### Proof\.

Add and subtract∑tα^t​Vt\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}V\_\{t\}:

y−y^=∑t\(αt−α^t\)​Vt\+∑tα^t​\(Vt−V^t\)\.y\-\\hat\{y\}=\\sum\_\{t\}\(\\alpha\_\{t\}\-\\hat\{\\alpha\}\_\{t\}\)\\,V\_\{t\}\+\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\,\(V\_\{t\}\-\\hat\{V\}\_\{t\}\)\.By the triangle inequality,

‖y−y^‖≤‖∑t\(αt−α^t\)​Vt‖\+‖∑tα^t​\(Vt−V^t\)‖\.\\\|y\-\\hat\{y\}\\\|\\leq\\\|\\sum\_\{t\}\(\\alpha\_\{t\}\-\\hat\{\\alpha\}\_\{t\}\)\\,V\_\{t\}\\\|\+\\\|\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\,\(V\_\{t\}\-\\hat\{V\}\_\{t\}\)\\\|\.The second term is bounded by

‖∑tα^t​\(Vt−V^t\)‖≤∑tα^t​‖Vt−V^t‖,\\\|\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\,\(V\_\{t\}\-\\hat\{V\}\_\{t\}\)\\\|\\leq\\sum\_\{t\}\\hat\{\\alpha\}\_\{t\}\\,\\\|V\_\{t\}\-\\hat\{V\}\_\{t\}\\\|,which givesEVE\_\{V\}\. For the first term,

‖∑t\(αt−α^t\)​Vt‖≤∑t\|αt−α^t\|​‖Vt‖≤Vmax​‖α−α^‖1\.\\\|\\sum\_\{t\}\(\\alpha\_\{t\}\-\\hat\{\\alpha\}\_\{t\}\)\\,V\_\{t\}\\\|\\leq\\sum\_\{t\}\|\\alpha\_\{t\}\-\\hat\{\\alpha\}\_\{t\}\|\\,\\\|V\_\{t\}\\\|\\leq V\_\{\\max\}\\,\\\|\\alpha\-\\hat\{\\alpha\}\\\|\_\{1\}\.It remains to control‖α−α^‖1\\\|\\alpha\-\\hat\{\\alpha\}\\\|\_\{1\}\. Letϵt=s^t−st\\epsilon\_\{t\}=\\hat\{s\}\_\{t\}\-s\_\{t\}\. By Cauchy\-Schwarz,

\|ϵt\|=\|q⊤​\(K^t−Kt\)\|D≤‖q‖​‖Kt−K^t‖D≤μ\.\|\\epsilon\_\{t\}\|=\\frac\{\|q^\{\\top\}\(\\hat\{K\}\_\{t\}\-K\_\{t\}\)\|\}\{\\sqrt\{D\}\}\\leq\\frac\{\\\|q\\\|\\,\\\|K\_\{t\}\-\\hat\{K\}\_\{t\}\\\|\}\{\\sqrt\{D\}\}\\leq\\mu\.Sinceα^t=αt​eϵt/Z\\hat\{\\alpha\}\_\{t\}=\\alpha\_\{t\}\\,e^\{\\epsilon\_\{t\}\}/ZwithZ=∑uαu​eϵuZ=\\sum\_\{u\}\\alpha\_\{u\}\\,e^\{\\epsilon\_\{u\}\}, andϵt∈\[−μ,μ\]\\epsilon\_\{t\}\\in\[\-\\mu,\\mu\], we haveZ∈\[e−μ,eμ\]Z\\in\[e^\{\-\\mu\},e^\{\\mu\}\]and therefore

α^tαt=eϵtZ∈\[e−2​μ,e2​μ\]\.\\frac\{\\hat\{\\alpha\}\_\{t\}\}\{\\alpha\_\{t\}\}=\\frac\{e^\{\\epsilon\_\{t\}\}\}\{Z\}\\in\[e^\{\-2\\mu\},e^\{2\\mu\}\]\.We use the following bound on the distance between two distributions with bounded likelihood ratio\. Ifp,p~p,\\tilde\{p\}satisfyp​\(t\)/p~​\(t\)∈\[e−2​μ,e2​μ\]p\(t\)/\\tilde\{p\}\(t\)\\in\[e^\{\-2\\mu\},e^\{2\\mu\}\]for everytt, then

‖p−p~‖1≤2​tanh⁡\(μ\)\.\\\|p\-\\tilde\{p\}\\\|\_\{1\}\\leq 2\\tanh\(\\mu\)\.\(19\)Indeed,φ​\(r\)=\(r−1\)/\(r\+1\)\\varphi\(r\)=\(r\-1\)/\(r\+1\)is increasing on\(0,∞\)\(0,\\infty\)withφ​\(e±2​μ\)=±tanh⁡\(μ\)\\varphi\(e^\{\\pm 2\\mu\}\)=\\pm\\tanh\(\\mu\), so\|r−1\|≤tanh⁡\(μ\)​\(r\+1\)\|r\-1\|\\leq\\tanh\(\\mu\)\\,\(r\+1\)for everyr∈\[e−2​μ,e2​μ\]r\\in\[e^\{\-2\\mu\},e^\{2\\mu\}\]\. Applying this withr=p​\(t\)/p~​\(t\)r=p\(t\)/\\tilde\{p\}\(t\), multiplying byp~​\(t\)\\tilde\{p\}\(t\)and summing overtt,

‖p−p~‖1=∑t\|p​\(t\)−p~​\(t\)\|\\displaystyle\\\|p\-\\tilde\{p\}\\\|\_\{1\}=\\sum\_\{t\}\|p\(t\)\-\\tilde\{p\}\(t\)\|≤tanh⁡\(μ\)​∑t\(p​\(t\)\+p~​\(t\)\)\\displaystyle\\leq\\tanh\(\\mu\)\\sum\_\{t\}\\big\(p\(t\)\+\\tilde\{p\}\(t\)\\big\)=2​tanh⁡\(μ\)\.\\displaystyle=2\\tanh\(\\mu\)\.Applying Eq\.[19](https://arxiv.org/html/2608.02901#A1.E19)withp=αp=\\alphaandp~=α^\\tilde\{p\}=\\hat\{\\alpha\}gives‖α−α^‖1≤2​tanh⁡\(μ\)\\\|\\alpha\-\\hat\{\\alpha\}\\\|\_\{1\}\\leq 2\\tanh\(\\mu\), hence

‖∑t\(αt−α^t\)​Vt‖≤2​Vmax​tanh⁡\(μ\),\\\|\\sum\_\{t\}\(\\alpha\_\{t\}\-\\hat\{\\alpha\}\_\{t\}\)\\,V\_\{t\}\\\|\\leq 2V\_\{\\max\}\\tanh\(\\mu\),which isEKE\_\{K\}\. Combining the two terms proves Eq\.[17](https://arxiv.org/html/2608.02901#A1.E17), and the linear bound follows fromtanh⁡\(μ\)≤μ\\tanh\(\\mu\)\\leq\\mu\. ∎

### A\.5The Algorithm

Algorithm[1](https://arxiv.org/html/2608.02901#alg1)provides an overview of the approach per layer

Algorithm 1AnchorKV prefill compression \(one layer\)0:

K,VK,Vper KV head; window

WW; anchor budget

kk; target ratio

θ\\theta
1:foreach KV head

hhdo

2:

𝒜h←\\mathcal\{A\}\_\{h\}\\leftarrowlast

WW∪\\cuptop

ρ​\(k−W\)\\rho\(k\{\-\}W\)by Eq\.[14](https://arxiv.org/html/2608.02901#A1.E14)

∪\\cuprandom rest

3:per side: assign

a​\(t\),γta\(t\),\\gamma\_\{t\}\(Eqs\.[1](https://arxiv.org/html/2608.02901#S3.E1),[2](https://arxiv.org/html/2608.02901#S3.E2)\); form

rtK,rtVr^\{K\}\_\{t\},r^\{V\}\_\{t\}
4:per side: score

utK,utVu^\{K\}\_\{t\},u^\{V\}\_\{t\}for

t∉𝒜ht\\notin\\mathcal\{A\}\_\{h\}\(Eq\.[6](https://arxiv.org/html/2608.02901#S3.E6)\)

5:endfor

6:

N←N\\leftarrowEq\.[9](https://arxiv.org/html/2608.02901#S3.E9);

NK,NV←⌊N/2⌋N\_\{K\},N\_\{V\}\\leftarrow\\lfloor N/2\\rfloor
7:per side

⋆\\star:

ℛ⋆←\\mathcal\{R\}\_\{\\star\}\\leftarrowtop

N⋆N\_\{\\star\}by

ut⋆u^\{\\star\}\_\{t\}pooled over heads; rotate, quantize, store \(App\.[A\.1](https://arxiv.org/html/2608.02901#A1.SS1)\)

8:discard dense

K,VK,V

## Appendix BExperiments \- Setup details

In this appendix, we provide further details regarding the experiments we reported in Section[4](https://arxiv.org/html/2608.02901#S4)\. All the experiments were conducted in NVIDIA A100\-80GB GPUs\.

### B\.1Models

We evaluate three instruction\-tuned models spanning an order of magnitude in scale, all with a 128K\-token context window: Llama\-3\.1\-8B\-Instruct and Llama\-3\.1\-70B\-Instruct\(Team[2024](https://arxiv.org/html/2608.02901#bib.bib36)\), and Mistral\-Small\-3\.1\-24B\-Instruct\(Mistral AI[2025](https://arxiv.org/html/2608.02901#bib.bib41)\)\. We apply each model’s native chat template and do not truncate inputs \(full\-context evaluation\), so sequence lengths match the benchmark specification exactly, and we apply the KV compression to all of the layers in each model\. Models are loaded inbfloat16with FlashAttention\-2 and run under Hugging Facetransformers/ PyTorch\. All generation is greedy \(temperature0, no sampling\) with batch size11, and every run uses a fixed seed \(4242\)\.

### B\.2Benchmarks

#### RULER

RULER is a synthetic long\-context benchmark based on NIAH\(Hsiehet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib37)\), but contains more challenging tasks\. Overall it has 13 tasks spanning four categories: retrieval, multi\-hop, aggregation and question answering\. We follow the setup of the official repository using the tasks default configuration to generate RULER 16K\-96K\. Each different context length has 1300 randomly generated samples, 100 samples per subtask\. Scoring follows RULER: a normalized substring match against the gold string\(s\); tasks with multiple gold items \(niah\_multiquery,niah\_multivalue,vt,cwe,fwe\) are scored as the fraction of gold items recovered, and the remainder as an exact recall indicator\.

#### LongBench

LongBench contains 16 tasks across six different categories: single document QA, multi document QA, summarization, few shot learning, synthetic reasoning and code completion\. We follow the setup of SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15)\), and do not perform input truncation\. For each task in the benchmark a task\-specific evaluation is used, Exact match for QA tasks, ROUGE for summarization, etc\. Finally, following the benchmark’s official script, we do not apply a chat template when evaluating on the subtasks TREC, TriviaQA, SAMSum, LCC and RepoBench\-P tasks\. We use all of the samples in each subtask\.

#### Needle\-in\-a\-haystack

We follow the template used in\(Liuet al\.[2024b](https://arxiv.org/html/2608.02901#bib.bib9)\), of the passkey retrieval prompt of\(Mohtashami and Jaggi[2023](https://arxiv.org/html/2608.02901#bib.bib40)\), but instead of using a 7\-digit\-passkey, we use a 64\-digit\-passkey and Paul Graham Essays as the background filler\. There is an important info hidden inside a lot of irrelevant text\. Find it and memorize them\. I will quiz you about the important information there\. <text from Paul Graham Essays\> The pass key is <64\-digit passkey\>\. remember it\. <64\-digit\-passkey\> is the pass key\. <more text from Paul Graham Essays\> What is the pass key? The pass key is:

#### Prompt formatting\.

To isolate the effect of KV compression, the answer cue and query prefix are placed after the chat template and the answer cue is excluded from compression \(two\_stage\_answer\_cue\); the same prompt construction is used for every method, including the uncompressed FullKV baseline, so all methods see an identical prompt and differ only in how the past\-token KV cache is stored\. For instance for NIAH, the "The pass key is:" is excluded from the compression step\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x8.png)Figure 8:Needle in a haystack at5×5\\timescompression\. Each cell is the mean score over the 10 samples at that context length and needle depth\.

### B\.3AnchorKV Hyperparameters

AnchorKV is applied entirely at inference time; there is no training, fine\-tuning, or offline calibration\. The method requires no gradient updates and no external calibration corpus: the anchor set, residual selection, and quantization statistics are computed on the fly from the prompt’s own key/value tensors during prefill\. Consequently there are no learning rates, optimizers, or training schedules to report\. The only configuration consists of the fixed inference\-time hyperparameters we now list\. For all experiments we use windowW=32W=32, anchor budgetk=S/128k=S/128, scored fractionρ=0\.7\\rho=0\.7, pooling kernelκ=7\\kappa=7, anchors in bf16 and residuals at 2 bits\. We swept the anchor budget overS/64S/64,S/128S/128andS/256S/256once, on a small subset of samples drawn from a few RULER subtasks with Llama\-3\.1\-8B, selected the single best\-performing budget, and fixed it for all experiments in the paper; no per\-task, per\-benchmark or per\-model tuning was performed\.

### B\.4Baseline Evaluation Setup

We compare against three training\-free KV\-cache eviction baselines,SnapKV\(Liet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib15)\),AdaKV\(adaptive per\-head budget allocation on top of SnapKV scores\)\(Fenget al\.[2025](https://arxiv.org/html/2608.02901#bib.bib17)\), andPyramidKV\(Caiet al\.[2024](https://arxiv.org/html/2608.02901#bib.bib16)\), all evaluated through the official KVPress framework\(Devotoet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib43)\)\. Every baseline is run on the*identical*prompts, compression budgets, decoding settings, and scoring code as our method; the only difference is the eviction rule\. All the default hyperparameters set by KVPress are used\.W=64W=64andκ=5\\kappa=5for the three methods\. Finally, all the methods were run under the query\-aware setting\.

### B\.5TurboQuant Reproduction Details

TurboQuant\(Zandiehet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib5)\)randomly rotates each key/value vector so its coordinates become Beta\-distributed, then applies per\-coordinate optimal scalar quantizers; the Prod variant adds a 1\-bit QJL residual transform for unbiased inner\-product estimation\. We evaluate the 3\.5\-bit\-per\-channel configuration, which the paper reports as quality\-neutral relative to the full cache\.\(Zandiehet al\.[2025](https://arxiv.org/html/2608.02901#bib.bib5)\)did not release public code at the time of this submission, and therefore we opted to reproduce their work relying on community resources\.

Our implementation builds on the main open\-source reproduction\(0xsero[2026](https://arxiv.org/html/2608.02901#bib.bib46)\), with outlier handling matching two other public reference implementations \-\(AmesianX[2026](https://arxiv.org/html/2608.02901#bib.bib47)\),\(TheTom[2026](https://arxiv.org/html/2608.02901#bib.bib48)\)\. Rather than the channel split described in the original text: per key vector we retain the top\-k highest\-magnitude channels in fp16 and apply a uniform b\-bit base quantizer to the remainder, plus a 128\-token fp16 recent window\. Concretely, TQ3\.5 uses b=3, k=4\. We found the fixed high\-/low\-bit channel split described in the paper text did not reproduce the reported near\-lossless quality in our setup \(∼\\sim7\-point LongBench drop vs\. full KV\), whereas the uniform\-base \+ fp16\-outlier scheme used by the reference implementations does\. We also point out that the vLLM implementation\(vLLM[2026](https://arxiv.org/html/2608.02901#bib.bib49)\)also found that deviations from the verbatim method are required to reproduce the reported numbers\. They chose to exclude the first two and last two transformer layers from quantization to match the reported results, at the cost of a lower compression rate\. Our choice avoids this, compressing all layers\.

## Appendix CExperiments \- Additional Results

### C\.1Needle in a Haystack

We follow the passkey retrieval protocol ofLiuet al\.\([2024b](https://arxiv.org/html/2608.02901#bib.bib9)\)\. Depth is the fractional position of the passkey in the context, so depth0places it at the beginning and depth100100immediately before the query\. We sweep context length and needle depth over 10 depths per length and 10 samples per depth, giving 100 samples per context length, and resample the passkey for every sample\. A sample is scored 1 if the recovered digit string matches the planted one exactly and 0 otherwise, so each cell of the heatmaps is the mean over its 10 samples\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x9.png)Figure 9:Needle in a haystack at10×10\\timescompression\. Each cell is the mean score over the 10 samples at that context length and needle depth\.Figures[8](https://arxiv.org/html/2608.02901#A2.F8)and[9](https://arxiv.org/html/2608.02901#A3.F9)give the full grids at5×5\\timesand10×10\\times\. AnchorKV recovers the needle across nearly the whole grid at both ratios, while the eviction baselines collapse to almost zero away from the most recent positions\. All methods are exact along the bottom row, where the needle falls inside the recency window they store verbatim\. That row is the one region where the two families agree, and it marks the limit of what a recency window alone can do: everything above it is retrieval the compressed cache has to support on its own\.

### C\.2RULER and LongBench \- Full Results

We provide tables with the full results on all the subtasks for all of the models and baselines in Tables[6](https://arxiv.org/html/2608.02901#A5.T6)\-[15](https://arxiv.org/html/2608.02901#A5.T15)\. Additionally, per\-task heatmaps for the three models on RULER\-32k and Longbench are displayed in Figures[13](https://arxiv.org/html/2608.02901#A5.F13)\-[18](https://arxiv.org/html/2608.02901#A5.F18)\.

## Appendix DAdditional Ablation Results and Analysis

![Refer to caption](https://arxiv.org/html/2608.02901v1/x10.png)Figure 10:Attention\-output fidelity at20×20\\timescompression, per \(layer, head, sample\) cell \(Llama\-3\.1\-8B\-Instruct, RULER 32K; logyy\-axis, dashed line atcos=0\.9\\cos=0\.9\)\. Below that threshold AnchorKV leaves0\.2%0\.2\\%of cells against8\.4%8\.4\\%for the best baseline on NIAH\-S1, and0\.3%0\.3\\%against24%24\\%on FWE\.### D\.1Attention\-Output Fidelity

Figure[10](https://arxiv.org/html/2608.02901#A4.F10)measures the quantity the method targets directly, rather than through downstream accuracy: how closely each compressed cache reproduces the attention output of the uncompressed one\. For every \(layer, head, sample\) cell we take the cosine similarity between the two outputs and plot the cumulative distribution on a log axis\. A method that reconstructs the output exactly places all of its mass atcos=1\\cos=1, so its curve stays flat and turns up only at the far right\. We markcos=0\.9\\cos=0\.9as a reference: mass to the left of that line is cells whose output has moved substantially\.

AnchorKV leaves0\.2%0\.2\\%of cells below the threshold on NIAH\-S1 and0\.3%0\.3\\%on FWE\. The strongest eviction baseline leaves8\.4%8\.4\\%and24%24\\%, forty to eighty times as many, and its mass is spread across the whole range rather than gathered near one\. The gap is widest on FWE, an aggregation task whose answer draws on many tokens spread through the context\. Scoring tokens once and discarding the rest damages a large share of the outputs there, whereas a coarsely stored token still enters the softmax and contributes what its anchor preserves\.

### D\.2Where the residual budget goes\.

Figure[11](https://arxiv.org/html/2608.02901#A4.F11)visualizes how AnchorKV distributes each layer’s residual budget across the eight KV heads\. Each cell gives the share of a layer’s residual tokens assigned to one head, so a naive per\-head split would leave every cell at the12\.5%12\.5\\%reference \(white\); red cells receive more than this share and blue cells less\. The allocation is strongly non\-uniform: within a single layer the budget concentrates heavily on a few heads, with individual cells reaching3535\-48%48\\%while others in the same row fall below5%5\\%\. This concentration is not tied to a fixed set of privileged heads, the head that receives the largest share changes from layer to layer, so no head dominates globally and only a mild preference \(head 5\) survives once the per\-layer structure is averaged away\. Together these observations show that the shared\-head mechanism is actively exercised rather than decorative: cross\-head competition continuously redirects budget toward the heads each layer relies on, which a uniform12\.5%12\.5\\%split could not do\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x11.png)Figure 11:Share of each layer’s residual budget taken by each KV head, AnchorKV at20×20\\timeson Llama\-3\.1\-8B at RULER 32K, averaged over 20 samples per task\. Rows sum to100%100\\%, so a per\-head split would fix every cell at12\.5%12\.5\\%\. As we can see the head sharing is being used differently across layers as well as across tasks\.

## Appendix EEfficiency and Memory

This appendix expands Section 4\.4\. All measurements use Llama\-3\.1\-8B\-Instruct in bf16 on one 80 GiB A100; the baseline is the uncompressed bf16 cache \(FullKV\), AnchorKV runs at a20×20\\timestarget with the hyperparameters of Appendix B\.3, memory excludes weights, and every cell is the median of five repeats in a fresh process\.

![Refer to caption](https://arxiv.org/html/2608.02901v1/x12.png)Figure 12:Prefill Overhead and Amortization\. Same setup as Figure 6\. \(a\) Prefill wall clock atB=1B\{=\}1: the one\-shot compression pass costs1\.30×1\.30\\timesat 32K and1\.12×1\.12\\timesat 128K, and its relative cost shrinks as context grows\. \(b\) End\-to\-endgenerate\(\)overhead against output length, at each context’s largest common batch \(32K/B=4B\{=\}4, 64K/B=3B\{=\}3, 96K/B=2B\{=\}2, 128K/B=1B\{=\}1\): the pass amortizes over decode steps, reaching\+4\.2%\+4\.2\\%at 32K,−0\.7%\-0\.7\\%at 64K and−0\.5%\-0\.5\\%at 96K for 2048 tokens\. The 128K curve sits above 64K only because memory caps it atB=1B\{=\}1, where the per\-request decode gap is largest\. \(c\) Overhead against batch size at 2048 generated tokens: batching converges the decode step times, from\+28\.7%\+28\.7\\%atB=1B\{=\}1to\+4\.2%\+4\.2\\%atB=4B\{=\}4\(32K\)\. \(d\) Peak memory per request atB=1B\{=\}1, log scale: prefill peaks \(dashed\) and decode peaks \(solid\)\. Compression cuts the prefill transient2\.0×2\.0\\timesand the decode state19×19\\times; AnchorKV’s prefill peak lands at FullKV’s decode peak\.Table 2:Decode latency \(ms/token\) in every cell where both arms fit\. Bold marks cells where AnchorKV decodes faster; from 64K upward it is at parity or better at every batch\.### E\.1Implementation

AnchorKV decodes with thedecode\_attention\_compactkernel, which reads the compressed tensors of Table 1 directly and never rebuilds the dense cache\. Anchors and the trailing window are held dense in bf16 with keys pre\-RoPE; every other position stores a nearest\-anchor index and one bf16 coefficientγt\\gamma\_\{t\}\(Eq\.[2](https://arxiv.org/html/2608.02901#S3.E2)\) per side, and the two\-bit residuals are compacted to the tokens that carry one, with a bitmask resolving position to slot, so a token without a residual costs one bit and no slot\.

The execution pattern is motivated by FlashAttention\-2 \(Dao 2024\): the context is walked in tiles under an online softmax, and nothing dense is ever written back\. Each decode step launches three Triton kernels per layer\. The score kernel reconstructs keys in 16\-token tiles entirely in registers, applies RoPE from the stored positions, contracts the tile against all four query rows of its GQA group in one tensor\-core op, and emits only fp32 logits: 16 bytes per token instead of a 256\-byte reconstructed key, a16×16\\timestraffic reduction\. The value kernel never reconstructs values at all: since the inverse Hadamard transform is linear, it accumulates the anchor\-projection and rotated\-residual terms separately and, at 17\.3% residual density, walks values residual\-major, visiting only the roughly 11 of 64 positions per chunk that store a residual\. The combine kernel reduces the partials, applies the inverse transform once per head per step rather than once per token, and attends the exact window \(recent prompt tokens plus every generated token, kept dense\) under the same online softmax\. The steady\-state residency is therefore the compressed state plus a few MB of workspace, and the measured decode peak tracks it:16\.2616\.26against0\.850\.85GiB at 128K, a19\.1×19\.1\\timesreduction \(Figure[12](https://arxiv.org/html/2608.02901#A5.F12)d\)\.

### E\.2Generated Tokens

Tokens produced during decoding are appended to the exact window, dense in bf16, and are never compressed: the append path is identical to a standard cache, decoding pays no per\-step compression work, and the compressed prefix and the exact suffix meet in the same online softmax inside the combine kernel\. Because the suffix is dense, the measured ratio drifts below the operating point as the output grows: the exact20×20\\timesat the moment of compression becomes19\.3×19\.3\\timesafter 60 decode steps at 32K, and a compressed\-prompt\-plus\-dense\-suffix model reproduces the measurement to three decimals, so nothing is lost in the compressor\. The drift is not specific to AnchorKV; any method that compresses a prompt and then decodes uncompressed has it\. We quote the at\-compression figure as the operating point\.

### E\.3Prefill

Compression runs once, per layer, at the end of prefill \(Algorithm 1\), and is the only compression work in the request\. The pass is built from batched matrix products and fast Hadamard transforms, remains a small fraction of prefill compute, and its relative cost shrinks with context: prefill takes1\.30×1\.30\\timesthe FullKV time at 32K,1\.15×1\.15\\timesat 96K and1\.12×1\.12\\timesat 128K atB=1B\{=\}1\(Figure[12](https://arxiv.org/html/2608.02901#A5.F12)a\), and stays between\+12%\+12\\%and\+30%\+30\\%over the grid\. The prefill peak itself is2\.0×2\.0\\timessmaller than FullKV’s, and since the prefill peak is what binds the batch, this is what roughly doubles the concurrency ceiling at every context \(Table[3](https://arxiv.org/html/2608.02901#A5.T3)\); all measurements in this appendix use unchunked prefill\.

Table 3:Concurrency ceiling on one 80 GiB A100: the largest batch each arm can serve, with aggregate throughput there\. Every bound is exact \(the largest success and the first failure are consecutive batch sizes\)\. AnchorKV roughly doubles the ceiling and improves throughput at each arm’s own frontier by 21–26%\.
### E\.4End\-to\-End Overhead

Tables[4](https://arxiv.org/html/2608.02901#A5.T4)and[5](https://arxiv.org/html/2608.02901#A5.T5)report the wall clock of fullmodel\.generate\(\)calls, covering prefill, compression, and all decode steps, over output lengths of 64 to 2048 tokens; positive overhead means AnchorKV is slower\. Because the compression pass is a one\-shot cost attached to prefill, the overhead shrinks along every axis that matters in practice\.

With output length, the pass amortizes over decode steps: at 32K,B=4B\{=\}4, the overhead falls from\+20\.7%\+20\.7\\%at 64 generated tokens to\+4\.2%\+4\.2\\%at 2048, and at 64K,B=3B\{=\}3, from\+13\.6%\+13\.6\\%to−0\.7%\-0\.7\\%, with−0\.5%\-0\.5\\%at 96K,B=2B\{=\}2: the compressed run finishes faster end to end \(Figure[12](https://arxiv.org/html/2608.02901#A5.F12)b\)\. With batch size, the decode steps of the two arms converge as the full cache becomes bandwidth\-bound while AnchorKV reads20×20\\timesfewer bytes: at 2048 tokens the overhead falls from\+28\.7%\+28\.7\\%atB=1B\{=\}1to\+4\.2%\+4\.2\\%atB=4B\{=\}4\(Figure[12](https://arxiv.org/html/2608.02901#A5.F12)c\), and at 32K,B=4B\{=\}4the AnchorKV decode step is already faster \(56\.1 against 57\.8 ms/token\)\. With context, both the prefill ratio and the decode gap close, and from 64K upward AnchorKV decodes at parity or better at every batch \(Table[2](https://arxiv.org/html/2608.02901#A5.T2)\)\. In the regime compression exists for, long contexts, full batches and non\-trivial outputs, the end\-to\-end price of keeping every token at20×20\\timescompression is under5%5\\%and occasionally negative, while the decode footprint is19×19\\timessmaller and the concurrency ceiling roughly doubles \(Table[3](https://arxiv.org/html/2608.02901#A5.T3)\)\.

Table 4:End\-to\-endgenerate\(\)overhead of AnchorKV at a20×20\\timestarget over FullKV, at each context’s largest batch where both arms fit\. Positive means AnchorKV is slower; bold marks the cell where the compressed run is faster end to end\.
### E\.5Prefill\-Decode Disaggregation

The two phases stress memory differently: the prefill peak is the activation transient, which compression cuts2\.0×2\.0\\times, while the decode residency is per\-request cache state, which it cuts19×19\\times\(Figure[12](https://arxiv.org/html/2608.02901#A5.F12)d\)\. Serving stacks that disaggregate prefill from decode split these cleanly\. Prefill nodes absorb the one\-shot compression pass at its1\.121\.12–1\.30×1\.30\\timescost and ship the compressed state, shrinking the KV transfer by20×20\\times; each decode node then holds19×19\\timesless state per request, so a replica that fits one FullKV request at 128K holds nineteen compressed ones, decoding at parity or better\. The overhead analyzed in this appendix is thus confined to the compute\-bound phase, while the memory\-bound phase receives the full reduction\.

Table 5:Full end\-to\-end generation grid, Llama\-3\.1\-8B\-Instruct on one 80 GiB A100\. Ours is AnchorKV at a20×20\\timestarget; each point is the wall clock of onemodel\.generate\(\)call, covering prefill, compression, and all decode steps\. Throughput is aggregate over the batch\. Bold marks cells where AnchorKV is faster end to end\.Table 6:RULER\-32K on Llama\-3\.1\-8B\-Instruct\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x13.png)Figure 13:Llama\-3\.1\-8B\-Instruct on RULER \(32k\)\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x14.png)Figure 14:Llama\-3\.1\-8B\-Instruct on LongBench\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x15.png)Figure 15:Mistral\-Small\-3\.1\-24B\-Instruct on RULER \(32k\)\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x16.png)Figure 16:Mistral\-Small\-3\.1\-24B\-Instruct on LongBench\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x17.png)Figure 17:Llama\-3\.1\-70B\-Instruct on RULER \(32k\)\.![Refer to caption](https://arxiv.org/html/2608.02901v1/x18.png)Figure 18:Llama\-3\.1\-70B\-Instruct on LongBench\.Table 7:RULER\-64K on Llama\-3\.1\-8B\-Instruct\.Table 8:RULER\-96K on Llama\-3\.1\-8B\-Instruct\.Table 9:LongBench on Llama\-3\.1\-8B\-Instruct\.Table 10:RULER at 32K on Mistral\-Small\-3\.1\-24B\-Instruct\.Table 11:RULER at 64K on Mistral\-Small\-3\.1\-24B\-Instruct\.Table 12:LongBench on Mistral\-Small\-3\.1\-24B\-Instruct\.Table 13:RULER at 32K on Llama\-3\.1\-70B\-Instruct\.Table 14:RULER at 64K on Llama\-3\.1\-70B\-Instruct\.Table 15:LongBench on Llama\-3\.1\-70B\-Instruct\.