Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

# KV Cache Vector Quantization with Attention-Preserving Transforms
Source: [https://arxiv.org/html/2608.04074](https://arxiv.org/html/2608.04074)
\\contribution

\[\*\]Equal contribution

## Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention\-Preserving Transforms

Amir ZiashahabiEduardo PavezAntonio OrtegaSalman AvestimehrDepartment of Electrical and Computer Engineering, University of Southern California[\{samuelf9, ziashaha\}@usc\.edu](https://arxiv.org/html/2608.04074v1/mailto:%7Bsamuelf9,%20ziashaha%[email protected])

###### Abstract

Long\-context LLM decoding reads the key\-value \(KV\) cache at every step\. Loading it takes longer than computing attention over it, so throughput is bandwidth\-bound\. Hence, reducing the cache size can raise both decoding speed and serving capacity\. The challenge is to reduce cache size while preserving the attention products, keeping reconstruction cheap, and using a fixed per\-token bit count\. At two bits per element, the most competitive methods rely on orthogonal transforms\. However, existing techniques are either data\-oblivious or use the query statistics without deriving the transform from a distortion criterion\. Moreover, they rely on transforms built on top of random or Hadamard rotations, which equalize variances across entries rather than compacting energy, and fixed\-width scalar quantizers, which are suboptimal at low rates\. In this paper, we formulate KV cache quantization as a transform coding problem in which distortion is the error in the attention products\. We derive closed\-form optimal transforms for keys and values from calibration statistics, under a high\-resolution model\. We show that the optimal key transform is not orthogonal and satisfies a generalized Parseval relation: the attention\-aware distortion becomes mean\-squared error \(MSE\) in the transform domain\. Thus, we can use MSE\-optimal vector quantizers applied directly to the transformed key coefficients\. To meet the fixed\-width layout requirement, we show that grouping coefficients into equal\-volume partitions makes equal\-size codebooks attain the variable\-rate optimum under the same high\-resolution model\. At two bits per element, our method, termed NOVA\-KV, recovers most of the long\-context retrieval accuracy lost by scalar quantization methods at comparable throughput\. The margin is widest on hybrid\-attention mixture\-of\-experts models, an increasingly common design: on GPT\-OSS\-20B, prior two\-bit transforms collapse at every context length, while NOVA\-KV remains effective\.

\\website

https://amir\-zsh\.github\.io/nova\-kv\\codehttps://github\.com/Amir\-zsh/nova\-kv\\correspondence

## 1Introduction

The key\-value \(KV\) cache of a large language model \(LLM\) stores, for each past token and attention head, a key vector and a value vector, to avoid recomputation during decoding\(pope2023efficiently\)\. The size of the cache is a central bottleneck in inference\(shazeer2019fast\): it grows with context length and batch size, yet every decoding step reads it from memory in full\(sadhukhan2025magicdec\)\(Fig\.[1](https://arxiv.org/html/2608.04074#S1.F1)\)\. Since kernels compute attention products faster than the KV cache can be loaded from memory\(dao2022flashattention\), decoding throughput is bound by memory bandwidth rather than by compute\(kwon2023efficient\)\. Thus,*KV cache compression*can reduce both per\-step memory traffic and footprint, raising decoding speed and serving capacity\.

![Refer to caption](https://arxiv.org/html/2608.04074v1/figs/fig_decode_qwen3_8b.png)Figure 1:Measured per\-step decode time for Qwen3\-8B \(8×8\\timesH100, BF16,S=16384S\{=\}16384\)\. At large batch sizes, KV cache loading dominates over parameter loading and compute\.Three requirements constrain the design of KV cache compressors\. First, task accuracy must be preserved\. Since capturing the effect of quantization on task accuracy is not straightforward, we can model the problem using classical rate\-distortion \(RD\) theory\(berger1971rate\), with the mean squared error \(MSE\) of the quantized values as the distortion metric\. However, the cache contents are used to compute attention products, so we should minimize errors in the products rather than MSE in the factors\(zandieh2025turboquant;zhou2026oscar\)\. Second, complexity on the read path\(williams2009roofline\)has to be kept low, because the cache is read at every decoding step, so any per\-element cost is incurred across the whole cache at every step\. Third, a fixed\-width layout must be used, so that tokens occupy a fixed number of bits and can be efficiently retrieved by serving engines\(kwon2023efficient\)\.

calibration datastatistics𝐌q,𝐒~k,𝐤¯,𝐌s\\mathbf\{M\}\_\{q\},\\ \\widetilde\{\\mathbf\{S\}\}\_\{k\},\\ \\bar\{\\mathbf\{k\}\},\\ \\mathbf\{M\}\_\{s\}transforms𝐑K,𝐑V\\mathbf\{R\}\_\{K\},\\ \\mathbf\{R\}\_\{V\}\(Thm\. 1, Cor\. 1\)codebooks𝒞\\mathcal\{C\},groupingπ\\pi\(Thm\. 2\)\(a\) Offline calibration𝐤t\\mathbf\{k\}\_\{t\}𝐯t\\mathbf\{v\}\_\{t\}transform\(𝐤t−𝐤¯\)​𝐑K\(\\mathbf\{k\}\_\{t\}\-\\bar\{\\mathbf\{k\}\}\)\\,\\mathbf\{R\}\_\{K\}transform𝐯t​𝐑V\\mathbf\{v\}\_\{t\}\\mathbf\{R\}\_\{V\}VQ encode𝒬K\+\\mathcal\{Q\}\_\{K\}^\{\+\}encode𝒬V\+\\mathcal\{Q\}\_\{V\}^\{\+\}paged cache \(fixed\-width\)lookup𝒬K−\\mathcal\{Q\}\_\{K\}^\{\-\}lookup𝒬V−\\mathcal\{Q\}\_\{V\}^\{\-\}inverse transform\(𝐪t​𝐑K−⊤\)​𝐊^⊤\(\\mathbf\{q\}\_\{t\}\\mathbf\{R\}\_\{K\}^\{\-\\top\}\)\\,\\widehat\{\\mathbf\{K\}\}^\{\\top\}inverse transform𝐕^​𝐑V⊤\\widehat\{\\mathbf\{V\}\}\\mathbf\{R\}\_\{V\}^\{\\top\}attend\(softmax\)𝐨t\\mathbf\{o\}\_\{t\}rotation\+\+stretchrotationinverseinversecells inℝg\\mathbb\{R\}^\{g\}pick centroidwrite: once per tokenread: every decoding step\(b\) Online inference![Refer to caption](https://arxiv.org/html/2608.04074v1/x1.png)![Refer to caption](https://arxiv.org/html/2608.04074v1/x2.png)\(c\) Non\-orthogonality

Figure 2:NOVA\-KV\. \(a\) Calibration yields the transforms, the volume\-equalized groupingπ\\pi, and the codebooks\. \(b\) Online, each new key and value is transformed and encoded once on the write path; the read path, paid at every decoding step, is a table lookup\. The dashed line separates the two paths\. \(c\) Top: keys, queries, and query covariance \(dashed\)\. Bottom: quantization error at equal rate; our transform steers the error away from the high\-energy query directions, preserving the attention logits\.Fixed\-width quantization meets the last two requirements: decoding is a scaling or table lookup\(gray2002quantization\), and each token occupies a fixed number of bits\. As the context grows, the KV cache dominates the memory footprint, so the reduction in inference time approaches the KV compression ratio itself\. However, to preserve task accuracy, direct quantization of KV entries requires at least 4 bits per element\(liu2024kivi;sheng2023flexgen\); at 2 bits, accuracy degrades\(zhou2026oscar\)\. Transform coding\(goyal2001theoretical\)has been proposed as an alternative to address this problem\(zandieh2025turboquant;zhou2026oscar\)\.

In classical transform coding, adata\-dependentlinear operator \(the transform\) decorrelates the inputs; scalar quantization is applied to the outputs, followed by entropy coding\. Because these transforms achieve energy compaction, bits can be allocated proportionally to the coefficient variances, achieving lower overall distortion for a given average rate compared to directly encoding without a transform\(goyal2001theoretical\)\. Existing transform\-based KV cache methods depart from this classical scheme\. In some cases, they use transforms, such as random or Hadamard rotations\(ashkboos2024quarot;zandieh2025turboquant\), that are not optimized for the data\. While data\-driven transforms have been proposed, these are assembled heuristically from second\-order statistics rather than as solutions to a specific rate\-distortion optimization problem\(zhou2026oscar\)\. Finally, existing designs use fixed\-rate scalar quantization, with transforms that*flatten*the variances so that the same rate can be applied for every vector entry; flattening removes the variance spread \(energy compaction\) that bit allocation exploits, so the fixed\-width layout can be achieved but at the expense of reduced coding efficiency\. Scalar quantization with variable\-length entropy coding is effective \(even at low rates\)\(goyal2001theoretical\), but requires sequential decoding, breaking the fixed\-width layout and the parallel read path requirements\.

In this paper, we formulate KV cache compression as a transform coding problem\(goyal2001theoretical\)where distortion is set to be the attention product error\. This error separates into two terms, one for the keys and one for the values, recovering the two criteria that prior work adopts separately\(zandieh2025turboquant;zhou2026oscar\)\. We derive, in closed form, the optimal transform for each distortion term in the high\-resolution regime\. Forvalues, the optimal transform is orthogonal and is obtained from the same covariance that prior work introduced as a heuristic\(zhou2026oscar\)\. Forkeys, our focus in this work, we show that the optimal transform is not orthogonal\. All prior designs used orthogonal transforms, which we show are suboptimal in practice when the goal is to minimize the key\-query inner product\.

The transform we derive for the keys satisfies a generalized Parseval relation: the MSE in the transform domain equals the key\-query inner\-product distortion in the original domain\. This relation reduces the design of the quantizer to a standard MSE problem, so any MSE\-optimal quantizer can act on the transform coefficients\. Based on this reduction, we propose vector quantization of the key coefficients with an optimized grouping of entries into vectors\. Transform coding would normally allocate more bits to high\-variance entries, resulting in codebooks of different sizes, which breaks the fixed\-width layout that serving engines require\. Instead, our grouping approximately equalizes group volumes, allowing equal\-size codebooks to approach the variable\-rate distortion bound \(cf\. Thm\.[2](https://arxiv.org/html/2608.04074#Thmtheorem2)\)\. This combines the advantage of flattening \(fixed bits per quantized input\), while benefiting from energy compaction \(fewer bits per input to achieve same distortion\)\.

We integrate our method, termed NOVA\-KV \(Non\-Orthogonal Vector\-quantized Attention for KV cache\), into a production serving stack\(zheng2024sglang\)with a fused decoding kernel\(tillet2019triton\)\(Fig\.[2](https://arxiv.org/html/2608.04074#S1.F2)\)\. At 2 bits per element, NOVA\-KV recovers most of the accuracy lost by scalar methods in long\-context retrieval\. In terms of speed, decoding is on par with the scalar baseline\(zhou2026oscar\)at long contexts\.

Our contributions are: 1\) data\-driven transforms that minimize a high\-resolution bound on the attention product distortion \(Sec\.[3](https://arxiv.org/html/2608.04074#S3)\), 2\) an analysis of vector quantization for the keys in the KV cache, including the optimal fixed\-rate grouping of transform coefficients under an independent\-Gaussian high\-resolution model \(Sec\.[4](https://arxiv.org/html/2608.04074#S4)\), and 3\) a serving\-compatible implementation with a fused decoding kernel, which we test against state\-of\-the\-art 2\-bit methods \(Sec\.[5](https://arxiv.org/html/2608.04074#S5)\)\.

## 2Preliminaries

Notation\.Uppercase and lowercase bold letters, such as𝐀\\mathbf\{A\}and𝐚\\mathbf\{a\}, denote matrices and vectors, respectively\. Thennth entry of𝐚\\mathbf\{a\}isana\_\{n\}, and the\(i,j\)\(i,j\)th entry of𝐀\\mathbf\{A\}isAi​jA\_\{ij\}\. Regular letters denote scalar values\. We use row\-vector notation and assume the eigenvalues are sorted in decreasing order\. We use\(⋅\)\+\(\\cdot\)^\{\+\}to denote the pseudoinverse\.

### 2\.1Attention and the KV Cache

Consider a single attention head\(vaswani2017attention\)acting on a sequence of hidden states\{𝐱t\}t=1T\\\{\\mathbf\{x\}\_\{t\}\\\}\_\{t=1\}^\{T\},𝐱t∈ℝ1×d\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{1\\times d\}, with projection weights𝐖Q,𝐖K,𝐖V∈ℝd×d\\mathbf\{W\}\_\{Q\},\\mathbf\{W\}\_\{K\},\\mathbf\{W\}\_\{V\}\\in\\mathbb\{R\}^\{d\\times d\}, wheredddenotes the head dimension\. Thequery,key, andvaluevectors are𝐪~t=𝐱t​𝐖Q\\tilde\{\\mathbf\{q\}\}\_\{t\}=\\mathbf\{x\}\_\{t\}\\mathbf\{W\}\_\{Q\},𝐤~t=𝐱t​𝐖K\\tilde\{\\mathbf\{k\}\}\_\{t\}=\\mathbf\{x\}\_\{t\}\\mathbf\{W\}\_\{K\},𝐯t=𝐱t​𝐖V\\mathbf\{v\}\_\{t\}=\\mathbf\{x\}\_\{t\}\\mathbf\{W\}\_\{V\}\. We use rotary position embeddings \(RoPE\)\(su2024roformer\): queries and keys are rotated by a position\-dependent block\-diagonal matrix𝚽t∈ℝd×d\\boldsymbol\{\\Phi\}\_\{t\}\\in\\mathbb\{R\}^\{d\\times d\}before the inner product,𝐪t=𝐪~t​𝚽t\\mathbf\{q\}\_\{t\}=\\tilde\{\\mathbf\{q\}\}\_\{t\}\\boldsymbol\{\\Phi\}\_\{t\}and𝐤t=𝐤~t​𝚽t\\mathbf\{k\}\_\{t\}=\\tilde\{\\mathbf\{k\}\}\_\{t\}\\boldsymbol\{\\Phi\}\_\{t\}, while values carry no positional encoding\. Symbols without a tilde denote post\-RoPE vectors\. These are stacked into𝐐=\[𝐪1;…;𝐪T\]\\mathbf\{Q\}=\[\\mathbf\{q\}\_\{1\};\\dots;\\mathbf\{q\}\_\{T\}\],𝐊=\[𝐤1;…;𝐤T\]\\mathbf\{K\}=\[\\mathbf\{k\}\_\{1\};\\dots;\\mathbf\{k\}\_\{T\}\],𝐕=\[𝐯1;…;𝐯T\]∈ℝT×d\\mathbf\{V\}=\[\\mathbf\{v\}\_\{1\};\\dots;\\mathbf\{v\}\_\{T\}\]\\in\\mathbb\{R\}^\{T\\times d\}\. The scores and outputs are:

𝐒=softmaxrow⁡\(𝐐𝐊⊤/d\)∈ℝT×T,𝐎=𝐒𝐕\.\\mathbf\{S\}=\\operatorname\{softmax\}\_\{\\mathrm\{row\}\}\\\!\\left\(\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}/\{\\sqrt\{d\}\}\\right\)\\in\\mathbb\{R\}^\{T\\times T\},\\quad\\mathbf\{O\}=\\mathbf\{S\}\\mathbf\{V\}\.\(1\)During autoregressive decoding, generating tokent\+1t\+1requires the query𝐪t\+1\\mathbf\{q\}\_\{t\+1\}to attend over all previous positions\. To avoid recomputing past projections, the*KV cache*stores in memory𝐊1:t=\[𝐤1;…;𝐤t\]\\mathbf\{K\}\_\{1:t\}=\[\\mathbf\{k\}\_\{1\};\\dots;\\mathbf\{k\}\_\{t\}\]and𝐕1:t=\[𝐯1;…;𝐯t\]\\mathbf\{V\}\_\{1:t\}=\[\\mathbf\{v\}\_\{1\};\\dots;\\mathbf\{v\}\_\{t\}\]\.

### 2\.2Transform coding

Classical transform coding\(goyal2001theoretical\)consists of three stages: a transform that produces uncorrelated coefficients, scalar quantization of each coefficient, and entropy coding of the indices\. Entropy decoding is sequential and cannot be easily parallelized, so it is not well\-suited for our problem, where the cache is read at every decoding step, and parallel decoding is needed to maintain high throughput\.

Transform\.By Parseval’s relation, an orthogonal𝐑\\mathbf\{R\}preserves the MSE,‖𝐱−𝐱^‖2=‖𝐱𝐑−𝒬​\(𝐱𝐑\)‖2\\\|\\mathbf\{x\}\-\\widehat\{\\mathbf\{x\}\}\\\|\_\{2\}=\\\|\\mathbf\{x\}\\mathbf\{R\}\-\\mathcal\{Q\}\(\\mathbf\{x\}\\mathbf\{R\}\)\\\|\_\{2\}, so the quantizer can be designed in the transform domain; for Gaussian sources in the high\-resolution regime the Karhunen\-Loève transform \(KLT, the eigenbasis of the covariance\), is optimal\(goyal2001theoretical;gersho1992vector\)\. A transform achieves*energy compaction*when only a few coefficients in the transform have high variance\. The KLT is optimal in the sense that its leadingppcoefficients capture the most variance\. Thus, we can find the KLT,𝐑⋆\\mathbf\{R\}^\{\\star\}, by optimizing the rank\-ppreconstruction error: given centered𝐱j\\mathbf\{x\}\_\{j\},j=1,…,Mj=1,\\dots,M, let

𝐑p⋆=arg​min𝐑∈ℝd×p​∑j=1M‖𝐱j−𝐱j​𝐑𝐑\+‖22\.\\mathbf\{R\}^\{\\star\}\_\{p\}=\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{R\}\\,\\in\\,\\mathbb\{R\}^\{d\\times p\}\}\\,\\sum\_\{j=1\}^\{M\}\\,\\bigl\\\|\\,\\mathbf\{x\}\_\{j\}\-\\mathbf\{x\}\_\{j\}\\,\\mathbf\{R\}\\mathbf\{R\}^\{\+\}\\bigr\\\|\_\{2\}^\{2\}\.\(2\)The KLT can be computed via eigendecomposition of the covariance matrix, and𝐑p⋆\\mathbf\{R\}^\{\\star\}\_\{p\}is the KLT truncated to theppeigenvectors corresponding to the dominant eigenvalues of the data covariance\. We will show in Sec\.[3\.1](https://arxiv.org/html/2608.04074#S3.SS1)that this is not optimal in our setting\.

Quantization\.A vector quantizer operates on vectors of dimensionggand with an average ratebbbits per entry, spendingg​bgbbits per vector\. An encoder𝒬\+:ℝ1×g→\{1,…,2g​b\}\\mathcal\{Q\}^\{\+\}:\\mathbb\{R\}^\{1\\times g\}\\to\\\{1,\\dots,2^\{gb\}\\\}assigns each subvector the index of its nearest codeword, and a decoder𝒬−\\mathcal\{Q\}^\{\-\}returns that codeword from a codebook𝒞⊂ℝ1×g\\mathcal\{C\}\\subset\\mathbb\{R\}^\{1\\times g\}of2g​b2^\{gb\}entries; their composition is𝒬\\mathcal\{Q\}\(lloyd1982least;gersho1992vector\)\.*Scalar quantization*\(SQ\),g=1g=1, acts on each coordinate independently\. For a given rate,*vector quantization*\(VQ\),g\>1g\>1, achieves distortion no larger than SQ, since the products of scalar codebooks are valid vector codebooks\(lookabaugh1989high\)\. Classical transform coding pairs SQ with variable\-length entropy coding; however, as discussed earlier, entropy coding prevents parallel decoding, reducing the efficiency of the read path and impacting throughput\.

As a summary, given an invertible𝐑∈ℝd×d\\mathbf\{R\}\\in\\mathbb\{R\}^\{d\\times d\}and a vector𝐱\\mathbf\{x\}, transform coding encodes it as𝒬\+​\(𝐱𝐑\)\\mathcal\{Q\}^\{\+\}\(\\mathbf\{x\}\\mathbf\{R\}\)and reconstructs it as𝐱^=𝒬​\(𝐱𝐑\)​𝐑−1\\widehat\{\\mathbf\{x\}\}=\\mathcal\{Q\}\(\\mathbf\{x\}\\mathbf\{R\}\)\\,\\mathbf\{R\}^\{\-1\}\.

### 2\.3Related work

Table 1:KV cache compressors\.*Transform*: linear map applied before quantization\.*Criterion*: the objective the transform is derived from\.*Target*: the resource reduced; bandwidth\-oriented methods \(bw\) keep the read path to a scaling or a lookup, whereas storage\-oriented methods \(sto\) pay a penalty in the read path\. Had\. stands for Hadamard, orth\. stands for orthogonal\.KV cache reductioncan be achieved by combining several complementary approaches: \(i\) token eviction, where cached tokens are dropped based on their estimated importance to future attention\(zhang2023h2o;li2024snapkv;xiao2024efficient;devoto2025expected\); \(ii\) low\-rank designs, which shrink the head dimension\(liu2024deepseek\); and \(iii\) quantization \(our approach\)\.

Scalar KV quantizationmethods quantize entries independently and do not consider the attention products\. Some designs perform per\-channel quantization of keys and per\-token quantization of values without accounting for input data statistics\(liu2024kivi\)\. Others fit the quantizers to the data, typically learning per\-channel codebooks from the cache statistics\(hooper2024kvquant;cai2025nqkv\), or boosting sensitive channels at the cost of per\-channel metadata and non\-uniform layouts\(xia2025kitty\)\.

Transform coding\.QuaRot\(ashkboos2024quarot\)uses random or Hadamard rotations, and TurboQuant\(zandieh2025turboquant\)debiases inner products\. SpinQuant\(liu2025spinquant\)and RotateKV\(su2025rotatekv\)learn the rotation on calibration data\.zhou2026oscarreweight the key error by query statistics\. All are orthogonal and paired with SQ\. We derive the transform from the RD problem using the attention\-product distortion and show that the optimal key transform is non\-orthogonal\.

Vector quantization\.Unlike model weights\(chee2023quip\), KV caches are generated online and read at every decoding step, which changes the VQ requirements\.zhang2024kvexplores joint codebooks across channels\. CommVQ\(li2025commvq\)targets memory*storage*using VQ on the keys with large codebooks whose reconstruction requires a matrix product paid over the whole cache at every step\. We target bandwidth\-bound serving by applying VQ to small groups of transform\-domain entries and restricting reconstruction to fixed\-width codebook lookups that can be fused into the attention kernel\. Thus, our design jointly provides an attention\-derived transform, fixed\-width VQ, and a lookup\-based reconstruction\. In summary, NOVA\-KV is the only method that targets attention products and optimizes both the transform and quantization for this purpose \(see Table[1](https://arxiv.org/html/2608.04074#S2.T1)\)\.

We aim to design transforms \(𝐑K,𝐑V\\mathbf\{R\}\_\{K\},\\mathbf\{R\}\_\{V\}\) and quantizers \(𝒬K​\(⋅\),𝒬V​\(⋅\)\\mathcal\{Q\}\_\{K\}\(\\cdot\),\\mathcal\{Q\}\_\{V\}\(\\cdot\)\) to compress keys and values

𝐊^=𝒬K​\(𝐊𝐑K\)​𝐑K−1,𝐕^=𝒬V​\(𝐕𝐑V\)​𝐑V−1,\\widehat\{\\mathbf\{K\}\}=\\mathcal\{Q\}\_\{K\}\(\\mathbf\{K\}\\mathbf\{R\}\_\{K\}\)\\,\\mathbf\{R\}\_\{K\}^\{\-1\},\\qquad\\widehat\{\\mathbf\{V\}\}=\\mathcal\{Q\}\_\{V\}\(\\mathbf\{V\}\\mathbf\{R\}\_\{V\}\)\\,\\mathbf\{R\}\_\{V\}^\{\-1\},\(3\)while minimizing the attention output error‖𝐒𝐕−𝐒^​𝐕^‖F2\\\|\\mathbf\{S\}\\mathbf\{V\}\-\\widehat\{\\mathbf\{S\}\}\\widehat\{\\mathbf\{V\}\}\\\|\_\{F\}^\{2\}, where𝐒^\\widehat\{\\mathbf\{S\}\}is obtained by replacing𝐊\\mathbf\{K\}by𝐊^\\widehat\{\\mathbf\{K\}\}in \([1](https://arxiv.org/html/2608.04074#S2.E1)\)\. In the high\-resolution regime, second order terms on the quantization errors vanish, and the target error can be upper\-bounded by the sum of two terms \(Appendix[11\.3](https://arxiv.org/html/2608.04074#S11.SS3)\):

‖𝐒𝐕−𝐒^​𝐕^‖F≲‖𝐒−𝐒^‖F​‖𝐕‖2\+‖𝐒​\(𝐕−𝐕^\)‖F\.\\\|\\mathbf\{S\}\\mathbf\{V\}\-\\widehat\{\\mathbf\{S\}\}\\widehat\{\\mathbf\{V\}\}\\\|\_\{F\}\\lesssim\\\|\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\\\|\_\{F\}\\\|\\mathbf\{V\}\\\|\_\{2\}\+\\\|\\mathbf\{S\}\(\\mathbf\{V\}\-\\widehat\{\\mathbf\{V\}\}\)\\\|\_\{F\}\.\(4\)We use the score error‖𝐒−𝐒^‖F\\\|\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\\\|\_\{F\}as a criterion to derive𝐑K\\mathbf\{R\}\_\{K\}and the value error weighted by the scores‖𝐒​\(𝐕−𝐕^\)‖F\\\|\\mathbf\{S\}\(\\mathbf\{V\}\-\\widehat\{\\mathbf\{V\}\}\)\\\|\_\{F\}as a criterion to derive𝐑V\\mathbf\{R\}\_\{V\}\.𝒬K\\mathcal\{Q\}\_\{K\}consists of VQs with fixed codebook sizes applied to subvectors \(Sec\.[4](https://arxiv.org/html/2608.04074#S4)\), while𝒬V\\mathcal\{Q\}\_\{V\}applies SQ to each transformed entry\(zhou2026oscar\)\.

### 3\.1Key transform

Since row\-wise softmax is1/21/2\-Lipschitz\(gao2017properties\), the key\-dependent term in \([4](https://arxiv.org/html/2608.04074#S3.E4)\),‖𝐒−𝐒^‖F\\\|\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\\\|\_\{F\}, can be replaced by the logit error:‖𝐒−𝐒^‖F≤1/\(2​d\)​‖𝐐𝐊⊤−𝐐​𝐊^⊤‖F\\\|\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\\\|\_\{F\}\\leq\{1\}/\(2\\sqrt\{d\}\)\\\|\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\-\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}\\\|\_\{F\}\. Then, defining𝐌q=𝐐⊤​𝐐\\mathbf\{M\}\_\{q\}=\\mathbf\{Q\}^\{\\top\}\\mathbf\{Q\}, our goal is to design a transform and a quantizer that minimize:

‖𝐐𝐊⊤−𝐐​𝐊^⊤‖F2=∑j=1M\(𝐤j−𝐤^j\)​𝐌q​\(𝐤j−𝐤^j\)⊤,\\bigl\\\|\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\-\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}\\bigr\\\|\_\{F\}^\{2\}\\\\ =\\sum\_\{j=1\}^\{M\}\\,\(\\mathbf\{k\}\_\{j\}\-\\widehat\{\\mathbf\{k\}\}\_\{j\}\)\\,\\mathbf\{M\}\_\{q\}\\,\(\\mathbf\{k\}\_\{j\}\-\\widehat\{\\mathbf\{k\}\}\_\{j\}\)^\{\\top\},\(5\)for a given rate\. Denoting‖𝐚‖𝐌2=𝐚​𝐌​𝐚⊤\\\|\\mathbf\{a\}\\\|\_\{\\mathbf\{M\}\}^\{2\}=\\mathbf\{a\}\\,\\mathbf\{M\}\\,\\mathbf\{a\}^\{\\top\}, the right side of \([5](https://arxiv.org/html/2608.04074#S3.E5)\) can be rewritten as∑j‖𝐤j−𝐤^j‖𝐌q2\\sum\_\{j\}\\\|\\mathbf\{k\}\_\{j\}\-\\hat\{\\mathbf\{k\}\}\_\{j\}\\\|^\{2\}\_\{\\mathbf\{M\}\_\{q\}\}, i\.e\., an input\-weighted quadratic distortion with*constant*sensitivity matrix𝐌q\\mathbf\{M\}\_\{q\}\(linder1999high\)\.

To understand how a𝐌q\\mathbf\{M\}\_\{q\}\-based distortion affects the design, consider quantization based on companding\. In the SQ case, this involves defining an invertible mappingh​\(x\)h\(x\), so that given𝒬u\\mathcal\{Q\}\_\{u\}, a uniform SQ,x^=h−1​\(𝒬u​\(h​\(x\)\)\)\\hat\{x\}=h^\{\-1\}\(\\mathcal\{Q\}\_\{u\}\(h\(x\)\)\)\. For vectors, this can be generalized by selecting an invertible matrix𝐔\\mathbf\{U\}, so thathU​\(𝐱\)=h​\(𝐱𝐔\)h\_\{U\}\(\\mathbf\{x\}\)=h\(\\mathbf\{x\}\\mathbf\{U\}\)\. Then, companding can be applied directly by using𝒬u\\mathcal\{Q\}\_\{u\}for each entry \(e\.g\., entry\-wise SQ\) and defining𝐱^=hU−1​\(𝒬u​\(hU​\(𝐱\)\)\)\\hat\{\\mathbf\{x\}\}=h\_\{U\}^\{\-1\}\(\\mathcal\{Q\}\_\{u\}\(h\_\{U\}\(\\mathbf\{x\}\)\)\)\. High\-resolution quantization theory shows that we can minimize the𝐌q\\mathbf\{M\}\_\{q\}distortion in \([5](https://arxiv.org/html/2608.04074#S3.E5)\) via companding and an entry\-wise SQ by selecting the optimalhU​\(⋅\)h\_\{U\}\(\\cdot\), i\.e\., the one that minimizes \([5](https://arxiv.org/html/2608.04074#S3.E5)\), satisfyinghU′​\(𝐱\)​hU′​\(𝐱\)⊤=c​𝐌qh\_\{U\}^\{\\prime\}\(\\mathbf\{x\}\)h\_\{U\}^\{\\prime\}\(\\mathbf\{x\}\)^\{\\top\}=c\\,\\mathbf\{M\}\_\{q\}, which, for linear companding characterized by a matrix𝐔\\mathbf\{U\}implies𝐔𝐔⊤=c​𝐌q\\mathbf\{U\}\\mathbf\{U\}^\{\\top\}=c\\,\\mathbf\{M\}\_\{q\}\(linder1999high\)\. Orthogonal transforms are suboptimal for the high\-resolution regime unless𝐌q∝𝐈\\mathbf\{M\}\_\{q\}\\propto\\mathbf\{I\}\.

The condition𝐔𝐔⊤=c​𝐌q\\mathbf\{U\}\\mathbf\{U\}^\{\\top\}=c\\,\\mathbf\{M\}\_\{q\}determines the transform only up to an orthogonal factor: if𝐔\\mathbf\{U\}satisfies it, so does𝐔𝐖\\mathbf\{U\}\\mathbf\{W\}for any orthogonal𝐖\\mathbf\{W\}, since𝐔𝐖𝐖⊤​𝐔⊤=𝐔𝐔⊤\\mathbf\{U\}\\mathbf\{W\}\\mathbf\{W\}^\{\\top\}\\mathbf\{U\}^\{\\top\}=\\mathbf\{U\}\\mathbf\{U\}^\{\\top\}\. We fix this factor by optimizing energy compaction, as the KLT does in \([2](https://arxiv.org/html/2608.04074#S2.E2)\), but under the𝐌q\\mathbf\{M\}\_\{q\}\-weighted cost of \([5](https://arxiv.org/html/2608.04074#S3.E5)\) rather than the Euclidean one\. AssumeMMtokens in the calibration set and let𝐤¯=1/M​∑j𝐤j\\bar\{\\mathbf\{k\}\}=\{1\}/\{M\}\\sum\_\{j\}\\mathbf\{k\}\_\{j\}; centering the keys has no cost to attention \(Appendix[12\.1](https://arxiv.org/html/2608.04074#S12.SS1)\)\. We write𝐑†=\(𝐑⊤​𝐌q−1​𝐑\)−1​𝐑⊤​𝐌q−1\\mathbf\{R\}^\{\\dagger\}=\(\\mathbf\{R\}^\{\\top\}\\mathbf\{M\}\_\{q\}^\{\-1\}\\mathbf\{R\}\)^\{\-1\}\\mathbf\{R\}^\{\\top\}\\mathbf\{M\}\_\{q\}^\{\-1\}for the𝐌q\\mathbf\{M\}\_\{q\}\-weighted pseudoinverse\.

###### Theorem 1\.

Let𝐤~j=𝐤j−𝐤¯\\widetilde\{\\mathbf\{k\}\}\_\{j\}=\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\},𝐒~k=∑j𝐤~j⊤​𝐤~j\\widetilde\{\\mathbf\{S\}\}\_\{k\}=\\sum\_\{j\}\\widetilde\{\\mathbf\{k\}\}\_\{j\}^\{\\top\}\\widetilde\{\\mathbf\{k\}\}\_\{j\}, and let

𝐑K,p⋆=arg​min𝐑∈ℝd×p​∑j=1M‖𝐤~j−𝐤~j​𝐑𝐑†‖𝐌q2\.\\mathbf\{R\}^\{\\star\}\_\{K,p\}=\\operatorname\*\{arg\\,min\}\_\{\\mathbf\{R\}\\,\\in\\,\\mathbb\{R\}^\{d\\times p\}\}\\,\\sum\_\{j=1\}^\{M\}\\,\\bigl\\\|\\,\\widetilde\{\\mathbf\{k\}\}\_\{j\}\-\\widetilde\{\\mathbf\{k\}\}\_\{j\}\\,\\mathbf\{R\}\\mathbf\{R\}^\{\\dagger\}\\bigr\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\}\.\(6\)Then𝐑K,p⋆=𝐌q1/2​𝐄1:p\\mathbf\{R\}^\{\\star\}\_\{K,p\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\mathbf\{E\}\_\{1:p\}, where𝐄​𝚲​𝐄⊤\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}is the eigendecomposition of𝐌q1/2​𝐒~k​𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\,\\mathbf\{M\}\_\{q\}^\{1/2\}\.

Proof: Appendix[12\.2](https://arxiv.org/html/2608.04074#S12.SS2)\. We construct𝐑K\\mathbf\{R\}\_\{K\}by computing the eigendecomposition of𝐌q1/2​𝐒~k​𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\,\\mathbf\{M\}\_\{q\}^\{1/2\}and then multiplying by𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\};𝐑K,p⋆\\mathbf\{R\}^\{\\star\}\_\{K,p\}corresponds to the matrix truncated by keeping the eigenvectors corresponding to the dominant eigenvalues of𝐌q1/2​𝐒~k​𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\,\\mathbf\{M\}\_\{q\}^\{1/2\}, i\.e\.,𝐑K=𝐌q1/2​𝐄\\mathbf\{R\}\_\{K\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\mathbf\{E\}\. The centered key is encoded as𝐫j=\(𝐤j−𝐤¯\)​𝐑K\\mathbf\{r\}\_\{j\}=\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)\\,\\mathbf\{R\}\_\{K\}and reconstructed as𝐤^j=𝐫j​𝐑K−1\+𝐤¯\\widehat\{\\mathbf\{k\}\}\_\{j\}=\\mathbf\{r\}\_\{j\}\\,\\mathbf\{R\}^\{\-1\}\_\{K\}\+\\bar\{\\mathbf\{k\}\}\. We have𝐑K​𝐑K⊤=𝐌q\\mathbf\{R\}\_\{K\}\\mathbf\{R\}\_\{K\}^\{\\top\}=\\mathbf\{M\}\_\{q\}, as demanded by\(linder1999high\), and the transform satisfies a generalized Parseval relation\(girault2018irregularity\):

###### Proposition 1\.

Let𝐫\\mathbf\{r\}and𝐫^\\hat\{\\mathbf\{r\}\}be any two vectors, and𝐤=𝐫𝐑K−1\\mathbf\{k\}=\\mathbf\{r\}\\mathbf\{R\}\_\{K\}^\{\-1\}and𝐤^=𝐫^​𝐑K−1\\hat\{\\mathbf\{k\}\}=\\hat\{\\mathbf\{r\}\}\\mathbf\{R\}\_\{K\}^\{\-1\}\. Then,‖𝐫−𝐫^‖22=‖𝐤−𝐤^‖𝐌q2\\bigl\\\|\\mathbf\{r\}\-\\hat\{\\mathbf\{r\}\}\\bigr\\\|\_\{2\}^\{2\}=\\bigl\\\|\\mathbf\{k\}\-\\hat\{\\mathbf\{k\}\}\\bigr\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\}\.

Prop\.[1](https://arxiv.org/html/2608.04074#Thmproposition1)is analogous to the Parseval relation in Sec\.[2\.2](https://arxiv.org/html/2608.04074#S2.SS2)for the𝐌q\\mathbf\{M\}\_\{q\}\-MSE: the transform converts the weighted objective into ordinary MSE\. Thus, we can run any off\-the\-shelf MSE\-optimal quantizer on the transform coefficients\. Since𝐑K\\mathbf\{R\}\_\{K\}satisfies the linear compander condition, it is optimal for the associated high\-resolution attention\-weighted companding problem\. Its orthogonal factor is chosen to optimize the weighted low\-rank approximation in Theorem 1\.

### 3\.2Value transform

We derive the value transform from‖𝐒𝐕−𝐒​𝐕^‖F2=∑j,j′\(𝐌s\)j​j′​\(𝐯j−𝐯^j\)​\(𝐯j′−𝐯^j′\)⊤\\\|\\mathbf\{S\}\\mathbf\{V\}\-\\mathbf\{S\}\\widehat\{\\mathbf\{V\}\}\\\|\_\{F\}^\{2\}=\\sum\_\{j,j^\{\\prime\}\}\(\\mathbf\{M\}\_\{s\}\)\_\{jj^\{\\prime\}\}\\,\(\\mathbf\{v\}\_\{j\}\-\\widehat\{\\mathbf\{v\}\}\_\{j\}\)\(\\mathbf\{v\}\_\{j^\{\\prime\}\}\-\\widehat\{\\mathbf\{v\}\}\_\{j^\{\\prime\}\}\)^\{\\top\}, and𝐌s=𝐒⊤​𝐒\\mathbf\{M\}\_\{s\}=\\mathbf\{S\}^\{\\top\}\\mathbf\{S\}, the second moment of the attention scores\.

###### Corollary 1\.

Let𝐨i=𝐬i​𝐕\\mathbf\{o\}\_\{i\}=\\mathbf\{s\}\_\{i\}\\mathbf\{V\}denote the attention outputs on calibration data, and𝐄​𝚲​𝐄⊤\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}the eigendecomposition of𝐌o=𝐕⊤​𝐌s​𝐕\\mathbf\{M\}\_\{o\}=\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}\. The minimizer of‖𝐒𝐕−𝐒​𝐕^‖F2\\\|\\mathbf\{S\}\\mathbf\{V\}\-\\mathbf\{S\}\\widehat\{\\mathbf\{V\}\}\\\|\_\{F\}^\{2\}over low\-rank approximations is𝐑V=𝐄1:p\\mathbf\{R\}\_\{V\}=\\mathbf\{E\}\_\{1:p\}, with codes𝐬j=𝐯j​𝐑V\\mathbf\{s\}\_\{j\}=\\mathbf\{v\}\_\{j\}\\mathbf\{R\}\_\{V\}and reconstruction𝐯^j=𝐬j​𝐑V⊤\\widehat\{\\mathbf\{v\}\}\_\{j\}=\\mathbf\{s\}\_\{j\}\\mathbf\{R\}\_\{V\}^\{\\top\}\.

The proof is in Appendix[12\.4](https://arxiv.org/html/2608.04074#S12.SS4)\. Unlike𝐑K\\mathbf\{R\}\_\{K\}, the value transform*is*orthogonal\. The term𝐕⊤​𝐌s​𝐕\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}is the covariance used in\(zhou2026oscar\)\.

## 4KV cache vector quantization

Our goal is to design quantizers to minimize distortion at a fixed rate, because serving imposes a fixed\-width constraint: tokens must occupy a fixed number of bits, and rate cannot be reallocated across entries\. We achieve this via VQ, in which groups of entries are jointly quantized\. This leaves the choice of partition, i\.e\., how to group the entries, as the only degree of freedom\. We show that under standard information\-theoretic assumptions, there is a criterion that identifies which groupings are rate\-distortion optimal when per\-group rate is fixed, and we show how to find them\.

Letπ\\pidenote a partition of\{1,…,d\}\\\{1,\\dots,d\\\}intoL=d/gL=d/ggroupsG1,…,GLG\_\{1\},\\dots,G\_\{L\}of equal sizegg, and let𝐫Gℓ\\mathbf\{r\}\_\{G\_\{\\ell\}\}be the transform coefficients indexed byGℓG\_\{\\ell\}\. We first assume that each group has its own codebook𝒞ℓ\\mathcal\{C\}\_\{\\ell\}withbℓb\_\{\\ell\}bits, subject to∑ℓg​bℓ=d​b\\sum\_\{\\ell\}gb\_\{\\ell\}=db\. Later, we enforce the constraint thatbℓ=bb\_\{\\ell\}=b\. By Proposition[1](https://arxiv.org/html/2608.04074#Thmproposition1), we can work with the MSE, which satisfiesD​\(π,\{bℓ\}\)=∑ℓDℓ​\(bℓ\)D\(\\pi,\\\{b\_\{\\ell\}\\\}\)=\\sum\_\{\\ell\}D\_\{\\ell\}\(b\_\{\\ell\}\)\. We model the coefficients as independent zero\-mean Gaussians with variancesσ12≥⋯≥σd2\\sigma\_\{1\}^\{2\}\\geq\\cdots\\geq\\sigma\_\{d\}^\{2\}\(independence is assumed for tractability\.\) Under the high\-resolution regime, the distortion\-rate function of the optimalgg\-dimensional quantizer for groupGℓG\_\{\\ell\}is\(zador1982asymptotic\):

Dℓ​\(bℓ\)=Cg​2−2​bℓ​\(∏i∈Gℓσi2\)1/g​\(1\+o​\(1\)\),D\_\{\\ell\}\(b\_\{\\ell\}\)=C\_\{g\}\\,2^\{\-2b\_\{\\ell\}\}\\Bigl\(\\prod\_\{i\\in G\_\{\\ell\}\}\\sigma\_\{i\}^\{2\}\\Bigr\)^\{1/g\}\\bigl\(1\+o\(1\)\\bigr\),\(7\)whereCgC\_\{g\}is common to all groups\. Theo​\(1\)o\(1\)term in \([7](https://arxiv.org/html/2608.04074#S4.E7)\) vanishes for high rates, and we assume continuous ratesbℓ∈ℝb\_\{\\ell\}\\in\\mathbb\{R\}\. The partition enters \([7](https://arxiv.org/html/2608.04074#S4.E7)\) only through the*volume*vℓ​\(π\)≐∏i∈Gℓσi2v\_\{\\ell\}\(\\pi\)\\doteq\\prod\_\{i\\in G\_\{\\ell\}\}\\sigma\_\{i\}^\{2\}\. We seek the partition and per\-group rates minimizingD​\(π,\{bℓ\}\)D\(\\pi,\\\{b\_\{\\ell\}\\\}\), assuming:*variable rates*, where\{bℓ\}\\\{b\_\{\\ell\}\\\}can take any value subject to the budget∑ℓbℓ=L​b\\sum\_\{\\ell\}b\_\{\\ell\}=Lb, and*fixed rates*, wherebℓ≡bb\_\{\\ell\}\\equiv bas required by fixed\-width packing\. Note that∏ℓvℓ​\(π\)=∏i=1dσi2\\prod\_\{\\ell\}v\_\{\\ell\}\(\\pi\)=\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}for everyπ\\pi\.

###### Theorem 2\.

Under the model \([7](https://arxiv.org/html/2608.04074#S4.E7)\), for any partitionπ\\pithe allocation minimizingD​\(π,\{bℓ\}\)D\(\\pi,\\\{b\_\{\\ell\}\\\}\)subject to∑ℓbℓ=L​b\\sum\_\{\\ell\}b\_\{\\ell\}=Lbis

bℓ∗​\(π\)=b\+12​g​log2⁡\(vℓ​\(π\)/∏mvm​\(π\)1/L\),b^\{\*\}\_\{\\ell\}\(\\pi\)\\;=\\;b\\;\+\\;\\frac\{1\}\{2g\}\\,\\log\_\{2\}\\Big\(\{v\_\{\\ell\}\(\\pi\)\}\\big/\{\\prod\_\{m\}v\_\{m\}\(\\pi\)^\{1/L\}\}\\Big\),\(8\)which spends more bits on groups of larger volume\. The optimal distortion,D∗​\(b\)=Cg​L​2−2​b​\(∏i=1dσi2\)1/dD^\{\*\}\(b\)=C\_\{g\}L\\,2^\{\-2b\}\\bigl\(\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}\\bigr\)^\{1/d\}, is the same for everyπ\\pi, assuming that rates can be the arbitrary real values from \([8](https://arxiv.org/html/2608.04074#S4.E8)\)\.

The proof is in Appendix[13\.1](https://arxiv.org/html/2608.04074#S13.SS1)\. The allocation \([8](https://arxiv.org/html/2608.04074#S4.E8)\) depends on the partition only through the volumes, and it is uniform,bℓ∗=bb^\{\*\}\_\{\\ell\}=bfor allℓ\\ell, only whenv1​\(π\)=⋯=vL​\(π\)v\_\{1\}\(\\pi\)=\\cdots=v\_\{L\}\(\\pi\)\. Hence, for a volume\-equalizing partition, the optimal allocation is already fixed\-width\. Instead, if the volumes are not equal and we apply the same number of bits,bℓ≡bb\_\{\\ell\}\\equiv b, the distortion will be suboptimal,2−2​b​∑ℓvℓ​\(π\)1/g≥D∗​\(b\)2^\{\-2b\}\\sum\_\{\\ell\}v\_\{\\ell\}\(\\pi\)^\{1/g\}\\geq D^\{\*\}\(b\), with equality iffv1​\(π\)=⋯=vL​\(π\)v\_\{1\}\(\\pi\)=\\cdots=v\_\{L\}\(\\pi\)\. We therefore group entries into equal\-volume groups and spend the same number of bits per group; within each group, the codebook places more resolution on the entries of higher variance in the query\-weighted transform domain\.

The distortion of a partition is proportional to the average of thevℓ​\(π\)1/gv\_\{\\ell\}\(\\pi\)^\{1/g\}, whileD∗​\(b\)D^\{\*\}\(b\)is proportional to their geometric mean, so the penalty grows with the spread of the group volumes\. Sorting the entries and grouping them consecutively maximizes spread, while a random partition averages log\-variances per group \(closer to balanced\)\. Balancing∑i∈Gℓlog⁡σi2\\sum\_\{i\\in G\_\{\\ell\}\}\\log\\sigma\_\{i\}^\{2\}across groups is NP\-hard \(reduces to33\-Partition\(garey2002computers\)\); we use a heuristic\. Since each vector hasggentries, we will haveL=d/gL=d/gvectors to quantize\. Then, we sort the entries by decreasing order oflog⁡σi2\\log\\sigma\_\{i\}^\{2\}and, to approximately equalize their volume, groupGℓG\_\{\\ell\}contains entries with indexi≡ℓ−1modLi\\equiv\\ell\-1\\mod L\. The partition is folded into the transform, so it costs nothing at inference\.

We apply VQ to the keys and keep SQ for the values since they are well approximated at this rate: replacing the value quantizer with VQ leaves accuracy almost unchanged \(Appendix[8](https://arxiv.org/html/2608.04074#S8)\)\.Complexity\.Write, once per token: centering, and a nearest\-neighbor search over centroids per group\. Read: a lookup into a codebook of2g​b​g2^\{gb\}gscalars per group, and one multiply to restore per\-token scale\.

## 5Experiments

We use1212A100 GPUs \(40GB\); for throughput, we use a single H100 \(details in Appendix[8](https://arxiv.org/html/2608.04074#S8)\)\. The high\-resolution model might not hold at22bits, we use the theory to guide our design rather than as a guarantee\. As models, we use Llama\-3\.1\-8B\(grattafiori2024llama\), Qwen3, both 8B and 4B\-Thinking\(yang2025qwen3\), and GPT\-OSS\(agarwal2025gpt\)\. For GPT\-OSS, we quantize the KV cache only in its full\-attention layers, leaving sliding\-window layers \(128\-token window\) unchanged\.

#### Transform analysis\.

We fix the quantizer: uniform SQ per coordinate with per\-coordinate entropy coding, fit on the calibration split and evaluated on held\-out data\. Entropy coding removes rate allocation as a confound, so the comparison reflects the basis alone; OSCAR’s transforms target fixed\-rate SQ and gain no compaction here\. We compare four bases: the KLT of the keys \(MSE\-optimal\), the eigenbasis of the query second moment used by OSCAR, the full OSCAR transform, and ours\. Fig\.[3](https://arxiv.org/html/2608.04074#S5.F3)shows the MSE, the query\-weighted distortion \([5](https://arxiv.org/html/2608.04074#S3.E5)\), and the top\-1 attention agreement, i\.e\., the fraction of queries whose highest\-scoring key survives quantization\. MSE does not determine attention fidelity: the KLT attains the lowest MSE at every rate \(a\), yet ours, with a higher MSE, preserves the top\-1 key more often \(c\)\.

![Refer to caption](https://arxiv.org/html/2608.04074v1/x3.png)Figure 3:Bases under a fixed quantizer\. The KLT minimizes MSE \(a\) but not top\-1 agreement \(c\); the𝐌q\\mathbf\{M\}\_\{q\}\-MSE \(b\) orders the bases like \(c\); our transform is best in \(b\-c\)\.Table 2:RULER NIAH mean accuracy \(%\) across context lengths, mean±\\pm95% CI over 3 evaluation rollouts\. BPE is the effective bits per KV element at 128K context\. Best compressed method per column inbold\.Table 3:KV\-cache quantization compared across models and benchmarks\. Entries are mean±\\pm95% CI with 5 samples per prompt\. BPE denotes effective bits per KV element at 128K context\. “Drop” is the gap in the Mean column to the BF16 reference; less negative is better\. Best compressed method per column inbold\. Naive INT2 yields0\.00\.0for all models and tasks\.ModelMethodBPEGPQAHumanELCB v6AIME25MATH500MeanDropQwen3\-4B\-Thinking\-2507BF1616\.0064\.9±\\pm0\.884\.8±\\pm0\.559\.2±\\pm0\.970\.0±\\pm1\.597\.5±\\pm0\.275\.3–TurboQuant3\.2544\.3±\\pm1\.531\.2±\\pm0\.41\.3±\\pm1\.52\.2±\\pm3\.856\.6±\\pm0\.727\.1−\-48\.2QuaRot2\.2512\.6±\\pm1\.76\.1±\\pm1\.33\.0±\\pm1\.20\.0±\\pm0\.032\.5±\\pm1\.310\.8−\-64\.5OSCAR2\.2862\.4±\\pm0\.983\.8±\\pm0\.257\.4±\\pm0\.863\.3±\\pm2\.196\.8±\\pm0\.272\.7−\-2\.6NOVA\-KV\(ours\)2\.2262\.9±\\pm1\.384\.3±\\pm0\.456\.6±\\pm0\.667\.3±\\pm1\.297\.4±\\pm0\.273\.7−\-1\.6Qwen3\-8BBF1616\.0057\.2±\\pm0\.991\.7±\\pm0\.557\.6±\\pm0\.770\.0±\\pm1\.797\.0±\\pm0\.174\.7–TurboQuant3\.2545\.6±\\pm6\.276\.0±\\pm1\.932\.7±\\pm2\.545\.6±\\pm5\.194\.3±\\pm0\.158\.8−\-16\.9QuaRot2\.2544\.2±\\pm1\.339\.0±\\pm3\.79\.0±\\pm2\.020\.0±\\pm3\.375\.2±\\pm0\.437\.5−\-37\.2OSCAR2\.2853\.5±\\pm1\.290\.9±\\pm0\.753\.6±\\pm0\.266\.2±\\pm1\.796\.7±\\pm0\.272\.2−\-2\.5NOVA\-KV\(ours\)2\.2255\.1±\\pm1\.592\.1±\\pm0\.255\.6±\\pm0\.664\.6±\\pm2\.596\.9±\\pm0\.272\.9−\-1\.8Llama\-3\.1\-8BBF1616\.0025\.5±\\pm2\.763\.5±\\pm2\.310\.5±\\pm1\.70\.0±\\pm0\.044\.2±\\pm1\.928\.7–TurboQuant3\.2522\.7±\\pm2\.356\.7±\\pm2\.811\.2±\\pm1\.70\.0±\\pm0\.037\.3±\\pm1\.225\.6−\-3\.2QuaRot2\.252\.8±\\pm1\.06\.7±\\pm1\.40\.7±\\pm0\.50\.7±\\pm1\.36\.0±\\pm0\.83\.4−\-25\.4OSCAR2\.2823\.1±\\pm2\.664\.4±\\pm2\.59\.4±\\pm1\.43\.3±\\pm4\.144\.8±\\pm1\.129\.0\+\+0\.3NOVA\-KV\(ours\)2\.2221\.3±\\pm2\.664\.5±\\pm1\.610\.2±\\pm1\.73\.3±\\pm2\.945\.8±\\pm1\.329\.0\+\+0\.3GPT\-OSS\-20BBF1616\.0050\.4±\\pm3\.190\.2±\\pm2\.976\.2±\\pm2\.374\.0±\\pm7\.091\.8±\\pm1\.176\.5–TurboQuant3\.5035\.2±\\pm3\.033\.4±\\pm3\.830\.2±\\pm4\.117\.3±\\pm3\.570\.9±\\pm1\.837\.4−\-39\.1QuaRot2\.500\.0±\\pm0\.00\.0±\\pm0\.00\.0±\\pm0\.00\.0±\\pm0\.00\.0±\\pm0\.00\.0−\-76\.5OSCAR2\.5325\.1±\\pm2\.75\.4±\\pm1\.60\.0±\\pm0\.00\.0±\\pm0\.039\.0±\\pm1\.913\.9−\-62\.6NOVA\-KV\(ours\)2\.4146\.5±\\pm3\.184\.6±\\pm3\.671\.9±\\pm2\.567\.3±\\pm7\.591\.8±\\pm1\.172\.4−\-4\.1
#### Calibration\.

As in\(zhou2026oscar\), the key transform and codebooks are fit offline from198198GPQA\-Diamond prompts, run in prefill only, with per\-\(layer, KV\-head\) statistics\. For GQA\(ainslie2023gqa\), we calibrate across heads sharing the same KV cache\. For the keys, we fit codebooks with256256centroids \(vector size44, average22bits/coordinate\) bykk\-means\. We concatenate these prompts into long\-context sequences of128128K so the data spans the RoPE positions at which the transform and codebooks are evaluated\. No label, gold answer, model generation, or task metric enters the fit\. For values, we use the SQ from\(zhou2026oscar\)\. Refitting the transform, grouping, and codebooks on MMLU instead, a disjoint domain, leaves RULER NIAH within2\.02\.0points and does not lower GPQA, so calibrating on the evaluation domain confers no advantage \(Appendix[9](https://arxiv.org/html/2608.04074#S9)\)\.

#### Baselines\.

As in\(xiao2024efficient;zhou2026oscar\), we keep the first6464tokens and the most recent256256in BF16 and quantize the rest\. We run QuaRot, TurboQuant, and OSCAR under the configuration of\(zhou2026oscar\): matching group sizes, per\-channel/per\-token axes, and BF16 sink\-plus\-recent band\. For OSCAR we evaluated both the released transform and one refit on our calibration split; the two are within1\.51\.5points on RULER\-NIAH, and we report the better of the two\. TurboQuant is included at 3\.25 BPE, following\(zhou2026oscar\);zandieh2025turboquantreport quality neutrality at 3\.5 bits per channel and marginal quality degradation at 2\.5\.

#### Long context\.

We use RULER NIAH\(hsieh2024ruler\), a benchmark for long context robustness, and sweep the context from88K to128128K\. We run under chunked prefill, as serving engines do to bound iteration latency: the prompt is processed in fixed\-size chunks and each chunk is quantized on write, so every subsequent chunk attends to an already quantized history\. Deferring quantization until the whole prompt is prefilled avoids this accumulation, but holds the cache in BF16 at its peak\. All methods run in the same harness with the same chunk size\. Numbers for prior methods might be lower than those reported in the original papers, which do not specify a chunked prefill\. Yet, our implementations remain comparable to their reported behavior on the reasoning and coding benchmarks \(cf\. Table[3](https://arxiv.org/html/2608.04074#S5.T3)\)\. We report the mean over 3 rollouts with a 95% CI in Table[2](https://arxiv.org/html/2608.04074#S5.T2)\. The baselines degrade with context while NOVA\-KV tracks BF16 more closely: on Qwen3\-8B at128128K, QuaRot and OSCAR reach0\.00\.0and25\.325\.3against83\.483\.4for BF16, where NOVA\-KV reaches75\.475\.4\. At128128K, NOVA\-KV exceeds OSCAR by50\.150\.1points \(McNemarp<10−37p<10^\{\-37\}\)\. The same ordering holds on Llama\-3\.1\-8B, where NOVA\-KV reaches92\.492\.4at6464K and63\.363\.3at128128K against76\.976\.9and36\.736\.7for OSCAR\.

The separation is starkest on GPT\-OSS\-20B, where both baselines fail rather than degrading with context: QuaRot scores0\.00\.0at every length, and OSCAR reaches0\.50\.5at88K and0\.00\.0from1616K on, so neither retrieves anything at any context we test\. NOVA\-KV instead retains89\.689\.6at88K and54\.054\.0at128128K, against95\.895\.8and80\.480\.4for BF16\. This is the only model on which the orthogonal\-transform baselines lose the task entirely, and it is also the one whose architecture is closest to current practice: GPT\-OSS\-20B interleaves sliding\-window and full\-attention layers and routes its feed\-forward blocks through a mixture of experts, a combination that recent open\-weight models have increasingly adopted\.

#### Accuracy comparisons\.

We test on GPQA\-Diamond\(rein2023gpqa\), HumanEval\(chen2021evaluating\), LiveCodeBench v6\(jain2025livecodebench\), AIME25\(maa2025aime\), and MATH500\(lightman2024let;hendrycks2021measuring\)\. We draw55samples per prompt \(Table[3](https://arxiv.org/html/2608.04074#S5.T3)\)\. Against BF16, NOVA\-KV shows no detectable degradation on either Qwen model \(McNemarp=0\.17p=0\.17,0\.240\.24\), while OSCAR is significantly below BF16 on Qwen3\-4B \(p=0\.002p=0\.002\)\. The two compressed methods are not separable there \(p=0\.46p=0\.46\), and on these two models the significant separation appears only on long\-context retrieval\. The remaining two models separate them directly\. On Llama\-3\.1\-8B every arm tracks BF16 within the confidence intervals, so the methods are hard to tell apart: the uncompressed model already scores low on these tasks\. On GPT\-OSS\-20B, by contrast, NOVA\-KV preserves most of the BF16 accuracy on every benchmark \(mean72\.472\.4against76\.576\.5\), whereas OSCAR collapses, scoring5\.45\.4on HumanEval and0\.00\.0on both LiveCodeBench v6 and AIME25 for a mean of13\.913\.9, and TurboQuant falls to37\.437\.4\. NOVA\-KV therefore stays close to BF16 on all four models, while the robustness of the orthogonal\-transform baselines varies considerably across architectures\.

#### Bit rate\.

At128128K context, the K cache stores one88bit index per group ofg=4g=4entries \(22BPE, as\|𝒞\|=256\|\\mathcal\{C\}\|\{=\}256\) together with a per\-token BF16 scale \(16/128=0\.12516/128=0\.125\)\. The V cache is INT2 with a BF16 scale and offset \(zero\-point\), giving2\.000\+32/128=2\.2502\.000\+32/128=2\.250\. Averaging over the256256elements gives2\.18752\.1875, and the BF16 sink\-plus\-recent band \(6464,256256\) adds0\.0340\.034, for2\.222\.22BPE\. OSCAR stores a zero\-point on both caches, giving2\.282\.28BPE; the0\.060\.06gap is the K\-side zero\-point\. Transforms and codebooks are stored per model\.

#### Throughput\.

We measure decode throughput in SGLang\(zheng2024sglang\)at input lengths of 30K, 60K and 90K \(Fig\.[4](https://arxiv.org/html/2608.04074#S5.F4)\)\. We include a warm\-up request, and throughput is computed over the decode window alone, excluding time to first token\. Cross\-request prefix sharing is disabled\. We report aggregate tokens per second across the batch\. At batch 1 and batch 4 the comparison is bandwidth\-dominated: NOVA\-KV reaches1\.61\.6–3\.1×3\.1\\timesBF16 on Qwen3\-8B and1\.71\.7–3\.4×3\.4\\timeson Qwen3\-4B, with the larger factors at the longer inputs\. The factor is smaller on GPT\-OSS\-20B,1\.11\.1–1\.5×1\.5\\times, since sliding\-window reduces the share of the cache being compressed\. Against OSCAR at the same rate, the two arms are within a few percent of each other throughout\. At 90K the BF16 KV pool admits at most𝚋𝟺\\mathtt\{b4\}on either Qwen model, so it has no batch\-16 configuration, whereas both 2\-bit arms serve one\.

![Refer to caption](https://arxiv.org/html/2608.04074v1/x4.png)Figure 4:Decode throughput, prefill excluded\. Top row: batch 1 against input length\. Bottom row: batch scaling at 90K input\. BF16\+Qwen does not admit the largest batch at 90K\.
#### Transform\-quantizer\.

Table[4](https://arxiv.org/html/2608.04074#S5.T4)crosses transform \(OSCAR, NOVA\-KV\) and quantizer\. Under SQ \(same number of bits to each coordinate\), our transform collapses: it compacts energy rather than flattening it, so a single rate cannot serve every coordinate\. Replacing SQ by VQ is worth7\.57\.5and37\.437\.4points under OSCAR’s basis; replacing OSCAR’s basis by ours, under VQ, is worth a further8\.38\.3and12\.712\.7\. Neither substitution alone reaches the full result\.

Table 4:Transform\-quantizer ablation on RULER NIAH, Qwen3\-8B\. Mean accuracy \(%\)±\\pm95% CI over 3 rollouts\.
#### Volume equalization\.

We fix the transform, the codebook size, and the rate \(2\.22 BPE\), varying only the partition:*variance\-sorted*groups consecutive entries in decreasing variance, the configuration with the most unbalanced volumes;*random*draws one fixed partition at random; and*equalizing*is the partition of Sec\.[4](https://arxiv.org/html/2608.04074#S4)\. Over 3 evaluation rollouts for Qwen3\-8B, variance\-sorted is the worst \(53\.2±\\pm0\.7at 64K, 37\.2±\\pm0\.1at 128K\), random recovers most of the gap \(74\.2±\\pm0\.5, 74\.4±\\pm0\.2\), and the equalizing partition is the highest \(76\.4±\\pm0\.4, 75\.4±\\pm0\.1\), matching Theorem[2](https://arxiv.org/html/2608.04074#Thmtheorem2)\.

## 6Conclusion

We formulated KV cache quantization as a transform coding problem whose distortion is the attention product error\. Under a high\-resolution model, the optimal key transform is not orthogonal, and satisfies a generalized Parseval relation, turning the attention\-aware criterion into MSE in the transform domain\. Under the independent\-Gaussian high\-resolution model, equal\-volume grouping makes fixed\-rate VQ attain the variable\-rate optimum, allowing a fixed\-width cache without an asymptotic distortion penalty\. At two bits per element, NOVA\-KV reduces the gap to BF16 relative to the 2\-bit state\-of\-the\-art at comparable decoding speed\. Joint quantization of keys and values is left for future work\.

## 7Notation and Definitions

This appendix collects the notation and definitions used throughout the paper\.

### 7\.1Conventions

Uppercase bold letters \(𝐀\\mathbf\{A\}\) denote matrices, lowercase bold letters \(𝐚\\mathbf\{a\}\) denote row vectors, and regular letters denote scalars\. Thennth entry of𝐚\\mathbf\{a\}isana\_\{n\}and the\(i,j\)\(i,j\)th entry of𝐀\\mathbf\{A\}isAi​jA\_\{ij\}\. All vectors are row vectors, so a linear map𝐑\\mathbf\{R\}acts on the right:𝐱𝐑\\mathbf\{x\}\\mathbf\{R\}\. Eigenvalues are indexed in decreasing order,λ1≥⋯≥λd\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{d\}, and𝐄1:p\\mathbf\{E\}\_\{1:p\}collects the eigenvectors associated with thepplargest eigenvalues\. For a symmetric positive definite \(p\.d\.\)𝐌\\mathbf\{M\}we let‖𝐚‖𝐌2=𝐚𝐌𝐚⊤\\\|\\mathbf\{a\}\\\|\_\{\\mathbf\{M\}\}^\{2\}=\\mathbf\{a\}\\mathbf\{M\}\\mathbf\{a\}^\{\\top\}, and𝐌1/2\\mathbf\{M\}^\{1/2\}denotes the symmetric square root, i\.e\., the unique symmetric p\.d\.𝐁\\mathbf\{B\}with𝐁𝐁=𝐌\\mathbf\{B\}\\mathbf\{B\}=\\mathbf\{M\}\. We write∥⋅∥F\\\|\\cdot\\\|\_\{F\}for the Frobenius norm and∥⋅∥2\\\|\\cdot\\\|\_\{2\}for the spectral norm of a matrix and theℓ2\\ell\_\{2\}norm of a vector\.

### 7\.2Attention and the cache

Notation for attention, RoPE, and the cache follows Sec\.[2\.1](https://arxiv.org/html/2608.04074#S2.SS1)of the main text\. We let𝐊^\\hat\{\\mathbf\{K\}\},𝐕^\\hat\{\\mathbf\{V\}\}be the reconstructed keys and values after quantization, and𝐒^\\hat\{\\mathbf\{S\}\}be the scores computed from𝐊^\\hat\{\\mathbf\{K\}\}\. Under grouped\-query attention \(GQA\)\(ainslie2023gqa\), the query heads are partitioned into groups, and each group shares a single key\-value projection pair\(𝐖K,𝐖V\)\(\\mathbf\{W\}\_\{K\},\\mathbf\{W\}\_\{V\}\); we call the shared projection pair, together with the cache entries it produces, a*KV head*\. Quantities are then formed per \(layer, KV head\); the query statistic𝐌q\\mathbf\{M\}\_\{q\}is accumulated over the query heads sharing the KV head \(Appendix[12\.5](https://arxiv.org/html/2608.04074#S12.SS5)\), and the transform, grouping, and codebooks are fit per KV head\.

### 7\.3Calibration statistics

LetMMbe the number of cached keys collected on calibration\. The calibration statistics are the key mean𝐤¯=1/M​∑j𝐤j\\bar\{\\mathbf\{k\}\}=\{1\}/\{M\}\\,\\sum\_\{j\}\\mathbf\{k\}\_\{j\}, the key covariance𝐒~k=∑j\(𝐤j−𝐤¯\)⊤​\(𝐤j−𝐤¯\)\\widetilde\{\\mathbf\{S\}\}\_\{k\}=\\sum\_\{j\}\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)^\{\\top\}\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\), the query second moment𝐌q=𝐐⊤​𝐐\\mathbf\{M\}\_\{q\}=\\mathbf\{Q\}^\{\\top\}\\mathbf\{Q\}, the score second moment𝐌s=𝐒⊤​𝐒\\mathbf\{M\}\_\{s\}=\\mathbf\{S\}^\{\\top\}\\mathbf\{S\}, and the output second moment𝐌o=𝐕⊤​𝐌s​𝐕\\mathbf\{M\}\_\{o\}=\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}\.𝐌q\\mathbf\{M\}\_\{q\}is positive semidefinite by construction; we assume it is positive definite for the analysis\. Numerically,𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\}and𝐌q−1/2\\mathbf\{M\}\_\{q\}^\{\-1/2\}are formed from its symmetric eigendecomposition after flooring the eigenvalues at10−3010^\{\-30\}; no additive ridge is applied\. For values, we use the default rotation matrix provided in\(zhou2026oscar\)\.

### 7\.4Transforms, quantization, and grouping

For the keys, the transform is𝐑K=𝐌q1/2​𝐄\\mathbf\{R\}\_\{K\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{E\}, with𝐄\\mathbf\{E\}the eigenvectors of𝐌q1/2​𝐒~k​𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\mathbf\{M\}\_\{q\}^\{1/2\}\(Theorem 1\); the transform coefficients of keyjjare𝐫j=\(𝐤j−𝐤¯\)​𝐑K\\mathbf\{r\}\_\{j\}=\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)\\mathbf\{R\}\_\{K\}, andσ12≥⋯≥σd2\\sigma\_\{1\}^\{2\}\\geq\\cdots\\geq\\sigma\_\{d\}^\{2\}denote their per\-coordinate variances\. The query\-weighted distortion, or*𝐌q\\mathbf\{M\}\_\{q\}\-MSE*, of a reconstruction is‖𝐤−𝐤^‖𝐌q2\\\|\\mathbf\{k\}\-\\hat\{\\mathbf\{k\}\}\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\}; summed over cached keys it equals the key\-query inner\-product error \(cf\. Eq\. \([10](https://arxiv.org/html/2608.04074#S11.E10)\)\)\.

A quantizer of group sizeggmaps subvectors ofggentries jointly; a partitionπ\\piof\{1,…,d\}\\\{1,\\dots,d\\\}intoL=d/gL=d/ggroupsG1,…,GLG\_\{1\},\\dots,G\_\{L\}of sizegghas group*volume*vℓ​\(π\)=∏i∈Gℓσi2v\_\{\\ell\}\(\\pi\)=\\prod\_\{i\\in G\_\{\\ell\}\}\\sigma\_\{i\}^\{2\}, and is*volume\-equalizing*whenv1​\(π\)=⋯=vL​\(π\)v\_\{1\}\(\\pi\)=\\cdots=v\_\{L\}\(\\pi\)\. Under*fixed rates*every group gets the same ratebℓ≡bb\_\{\\ell\}\\equiv b; under*variable rates*\{bℓ\}\\\{b\_\{\\ell\}\\\}is free subject to∑ℓbℓ=L​b\\sum\_\{\\ell\}b\_\{\\ell\}=Lb\.

In \([7](https://arxiv.org/html/2608.04074#S4.E7)\), we letCg=g​Zg​γgC\_\{g\}=g\\,Z\_\{g\}\\,\\gamma\_\{g\}, withZgZ\_\{g\}the normalized moment of inertia of the optimalgg\-dimensional cell shape andγg\\gamma\_\{g\}the source\-density functional of Zador’s theorem; under the independent\-Gaussian modelCgC\_\{g\}is common to all groups\(zador1982asymptotic\)\(no effect on the choice of partition\)\.

The groupings compared in this paper are:*variance\-sorted*,Gℓ=\{\(ℓ−1\)​g\+1,…,ℓ​g\}G\_\{\\ell\}=\\\{\(\\ell\-1\)g\+1,\\dots,\\ell g\\\}after sorting entries by decreasingσi2\\sigma\_\{i\}^\{2\};*random*, one partition drawn uniformly at random and then held fixed; and*equalizing*, the partition of Sec\.[4](https://arxiv.org/html/2608.04074#S4),Gℓ=\{i∈\{1,…,d\}:i−1≡ℓ−1​\(mod​L\)\}G\_\{\\ell\}=\\\{\\,i\\in\\\{1,\\dots,d\\\}:i\-1\\equiv\\ell\-1\\ \(\\mathrm\{mod\}\\ L\)\\,\\\}after the same sort\. Fig\.[5](https://arxiv.org/html/2608.04074#S7.F5)illustrates the three rules\.

\(a\)\(a\)01234567891011\(b\)01234567891011\(c\)01234567891011Figure 5:The three partitions in a zero\-indexed toy example \(d=12d=12,g=4g=4,L=3L=3\); color denotes group membership, entries ordered by decreasing varianceσi2\\sigma\_\{i\}^\{2\}from left to right\. \(a\)*Variance\-sorted*groups consecutive spectrum entries\. \(b\)*Random*mixes levels by chance\. \(c\)*Equalizing*deals coordinateiito groupimodLi\\bmod L, so each group receives one coordinate per level \(Theorem 2\)\.
### 7\.5Symbol table

We summarize important notations in Table[5](https://arxiv.org/html/2608.04074#S7.T5)\.

Table 5:Summary of notations used in this work\.
### 7\.6High\-level algorithm

In Algorithm[1](https://arxiv.org/html/2608.04074#alg1), we detail our pipeline both during calibration and inference\.

Algorithm 1Query\-aware key transform and quantizer1:Calibration \(offline\):

2:Collect queries

𝐐\\mathbf\{Q\}and keys

𝐊\\mathbf\{K\}on calibration data

3:

𝐤¯←∑j=1M𝐤j/M\\bar\{\\mathbf\{k\}\}\\leftarrow\\sum\_\{j=1\}^\{M\}\\mathbf\{k\}\_\{j\}/\{M\};

𝐌q←𝐐⊤​𝐐\\mathbf\{M\}\_\{q\}\\leftarrow\\mathbf\{Q\}^\{\\top\}\\mathbf\{Q\};

𝐒~k←∑j=1M\(𝐤j−𝐤¯\)⊤​\(𝐤j−𝐤¯\)\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\leftarrow\\sum\_\{j=1\}^\{M\}\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)^\{\\top\}\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)
4:Eigendecompose

𝐌q1/2​𝐒~k​𝐌q1/2=𝐄​𝚲​𝐄⊤\\mathbf\{M\}\_\{q\}^\{1/2\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\mathbf\{M\}\_\{q\}^\{1/2\}=\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}
5:

𝐑K←𝐌q1/2​𝐄\\mathbf\{R\}\_\{K\}\\leftarrow\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{E\};

𝐑K−⊤=𝐌q−1/2​𝐄\\mathbf\{R\}\_\{K\}^\{\-\\top\}=\\mathbf\{M\}\_\{q\}^\{\-1/2\}\\mathbf\{E\}
6:Find volume\-equalized groups

π\\pi\(Thm\. 2\), fold into

𝐑K\\mathbf\{R\}\_\{K\}
7:Train per\-group codebooks

\{𝒞ℓ\}\\\{\\mathcal\{C\}\_\{\\ell\}\\\}on

\(𝐊−𝟏​𝐤¯\)​𝐑K\(\\mathbf\{K\}\-\\mathbf\{1\}\\bar\{\\mathbf\{k\}\}\)\\mathbf\{R\}\_\{K\}
8:Inference \(online\):

9:Write: cache

𝐊^=𝒬\+​\(\(𝐤j−𝐤¯\)​𝐑K\)\\hat\{\\mathbf\{K\}\}=\\mathcal\{Q\}^\{\+\}\\bigl\(\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)\\mathbf\{R\}\_\{K\}\\bigr\)
10:Attention:

𝐒=softmax⁡\(𝐐​𝐑K−⊤​𝒬−​\(𝐊^\)⊤/d\)\\mathbf\{S\}=\\operatorname\{softmax\}\\bigl\(\\mathbf\{Q\}\\,\\mathbf\{R\}\_\{K\}^\{\-\\top\}\\,\\mathcal\{Q\}^\{\-\}\(\\hat\{\\mathbf\{K\}\}\)^\{\\top\}/\\sqrt\{d\}\\bigr\)

## 8Experimental Details

In this section, we detail our experimental setup, including metrics, decoding configuration, calibration, serving setup, benchmarks, and datasets\. We also ablate the VQ and SQ in the value arm to justify our deployed method\.

### 8\.1Reported quantities

#### Bits per element \(BPE\)\.

The average number of stored bits per cached scalar entry, over keys and values and accounting for the full context, including per\-token scales, zero\-points, and the BF16 sink\-plus\-recent band, and excluding the per\-model transforms and codebooks, whose contribution is reported separately\. The cache\-rate accounting \(2\.222\.22for ours,2\.282\.28for OSCAR\) is given in the main text\.

The per\-model transforms and codebooks are stored once and are independent of context and batch\. Per \(layer, KV head\) we store the threed×dd\\times dtransforms𝐑K,𝐑K−⊤,𝐑V\\mathbf\{R\}\_\{K\},\\mathbf\{R\}\_\{K\}^\{\-\\top\},\\mathbf\{R\}\_\{V\}, the mean𝐤¯\\bar\{\\mathbf\{k\}\}\(ddscalars\), andL=d/gL=d/gkey codebooks of2g​b2^\{gb\}centroids inℝg\\mathbb\{R\}^\{g\}, that is3​d2\+d\+L​2g​b​g3d^\{2\}\+d\+L\\,2^\{gb\}gscalars\. For instance, with Qwen3\-8B, ford=128d=128,g=4g=4,2g​b=2562^\{gb\}=256,L=32L=32this is49152\+128\+32768≈8\.2×10449152\+128\+32768\\approx 8\.2\\times 10^\{4\}scalars per \(layer, KV head\); at1616\-bit transforms and88\-bit codebooks it totals≈0\.30\\approx 0\.30Gbit for the full3636\-layer,88\-KV\-head model\. Divided by the2​nlayer​nkv​d​Lctx​B2\\,n\_\{\\text\{layer\}\}\\,n\_\{\\text\{kv\}\}\\,d\\,L\_\{\\text\{ctx\}\}\\,Bquantized cache scalars, aLctx=128L\_\{\\text\{ctx\}\}=128K sequence carries≈0\.03\\approx 0\.03bits per KV element; since the same transforms and codebooks serve the entire batch and all requests, the amortized contribution falls below0\.010\.01bits per KV element at the batch sizes we serve, negligible against the≈2\.2\\approx 2\.2cache rate\.

#### Top\-1 attention agreement\.

The fraction of queries whose highest\-scoring key under𝐊^\\hat\{\\mathbf\{K\}\}is the same as under𝐊\\mathbf\{K\}\. It measures whether quantization preserves the argmax of the logits, which the MSE does not constrain directly\.

#### Sink\-plus\-recent band\.

The firstnsinkn\_\{\\mathrm\{sink\}\}and the most recentnrecn\_\{\\mathrm\{rec\}\}tokens are held in BF16 and excluded from quantization;nsinkn\_\{\\mathrm\{sink\}\}andnrecn\_\{\\mathrm\{rec\}\}are fixed constants, so the band’s share of the cache decreases as the context grows \(see Fig\.[6](https://arxiv.org/html/2608.04074#S8.F6)\)\.

BF16quantized \(NOVA\-KV\)BF1611\\vphantom\{n\_\{\\mathrm\{sink\}\}\}nsinkn\_\{\\mathrm\{sink\}\}t−nrect\-n\_\{\\mathrm\{rec\}\}\\vphantom\{n\_\{\\mathrm\{sink\}\}\}tt\\vphantom\{n\_\{\\mathrm\{sink\}\}\}sinkhistory \(grows withtt\)recentFigure 6:Cache layout at decoding positiontt\. The firstnsinkn\_\{\\mathrm\{sink\}\}tokens \(attention sinks\) and the most recentnrecn\_\{\\mathrm\{rec\}\}tokens are held in BF16 and excluded from quantization; the tokens in between are stored in the NOVA\-KV format\.nsinkn\_\{\\mathrm\{sink\}\}andnrecn\_\{\\mathrm\{rec\}\}are fixed \(nsink=64n\_\{\\mathrm\{sink\}\}\{=\}64,nrec=256n\_\{\\mathrm\{rec\}\}\{=\}256\), so the BF16 band’s share of the cache vanishes as the context grows\.
#### Benchmarks and scoring\.

Table[10](https://arxiv.org/html/2608.04074#S8.T10)lists, for each benchmark, the number of items scored \(\# items\), the number of samples drawn per prompt \(Smp\.\), the generation cap in tokens \(Gen\), and the per\-item metric\. In the main paper, the five generation benchmarks draw55samples per prompt under a3232K cap, while RULER NIAH uses33samples with a128128\-token answer cap\. The per\-item metrics are as follows\.*RULER NIAH*: case\-insensitive substring match, with partial credit equal to the fraction of gold needles found\.*GPQA\-Diamond*: the answer letter is extracted with the simple\-evals multiple\-choice template and scored by exact match against the gold option \(A–D\)\.*HumanEval*and*LiveCodeBench v6*: pass@1, i\.e\., a sample is accepted only if it passes all reference unit tests under sandboxed execution\.*AIME25*and*MATH500*: symbolic\-equivalence checking of the final answer with themath\-verifylibrary\. Benchmark accuracy is the mean of this per\-item score over samples, reported with a bootstrap95%95\\%confidence interval\.

### 8\.2Decode\-time breakdown \(Fig\.[1](https://arxiv.org/html/2608.04074#S1.F1)\)

Figure[1](https://arxiv.org/html/2608.04074#S1.F1)reports measured GPU kernel time per decoding step for Qwen3\-8B served with SGLang on8×8\\timesH100 80 GB SXM GPUs using tensor parallelism of degree 8\. We use a fixed context length of1638416384and a BF16 KV cache\. Profiling covers decode steps 6–8, after the complete1638416384\-token prompt has been written to the cache, so every profiled step reads the full cache\. Decode runs under CUDA graphs\. Each bar is the median over the three steps of the summed CUDA\-profiler kernel durations; CPU launch overhead and gaps between kernels are excluded\.

We assign kernels to the five categories shown in Fig\.[1](https://arxiv.org/html/2608.04074#S1.F1)according to their names\.*Parameter load*contains the linear and MLP GEMM kernels, whose time is dominated by accessing the model parameters in this regime\.*Activations*contains the remaining normalization, elementwise, and activation\-processing kernels\.*KV load*contains the fused decode\-attention kernels, which read the cached keys and values and compute the query\-key products, softmax, and score\-value products\.*KV store*contains the cache\-write kernels, and*all\-reduce*contains the NCCL collectives\.

The model parameter footprint is independent of batch size: the model has15\.115\.1GB of parameters in aggregate, and the same parameters are accessed once per decoding step regardless of the number of sequences in flight\. Hence, the parameter\-load bar remains nearly constant across batch sizes\. KV\-cache traffic, in contrast, scales with both batch size and context length\. Each token contributes512512bytes per KV head in BF16, including its key and value\. Under tensor parallelism of degree 8, the per\-GPU cache footprint therefore grows from0\.300\.30GB at batch 1 to19\.3319\.33GB at batch 64\. Correspondingly, the KV\-load bar in Fig\.[1](https://arxiv.org/html/2608.04074#S1.F1)increases from2\.22\.2to9\.49\.4ms, or from51%51\\%to76%76\\%of the kernel time\.

The KV\-load category measures the duration of the fused attention kernels rather than memory traffic in isolation\. These kernels stream the KV cache while performing the attention arithmetic\. Accounting for reuse across grouped\-query heads, decode attention has an arithmetic intensity of approximately4\.04\.0FLOP/byte, far below the H100 BF16 roofline ridge point of approximately295295FLOP/byte\. The fused attention kernel is therefore bandwidth\-bound once sufficient parallelism is available\. At batch 1, its measured KV traffic corresponds to only approximately4%4\\%of the3\.353\.35TB/s peak HBM bandwidth because a single1616K\-token sequence provides insufficient parallelism to occupy all 132 SMs\. At batch 64, the achieved bandwidth rises to approximately61%61\\%of peak\. Thus, at large batch sizes, streaming the KV cache dominates the measured GPU kernel time\.

### 8\.3Calibration

Statistics are accumulated over the calibration prompts, per \(layer, KV head\), in prefill only\. The matrices𝐌q±1/2\\mathbf\{M\}\_\{q\}^\{\\pm 1/2\}are formed from the symmetric eigendecomposition of𝐌q\\mathbf\{M\}\_\{q\}with eigenvalues floored at10−3010^\{\-30\}before the±1/2\\pm 1/2power; no additive ridge is applied\. The bases𝐑K\\mathbf\{R\}\_\{K\},𝐑K−⊤\\mathbf\{R\}\_\{K\}^\{\-\\top\},𝐑V\\mathbf\{R\}\_\{V\}are stored in float64 \(the mean𝐤¯\\bar\{\\mathbf\{k\}\}in float32\) and cast to bfloat16 when loaded into the serving engine\.

Key codebooks are trained bykk\-means with random initialization, for at most2525iterations, on the131072131072transformed key subvectors per \(layer, KV head\) \(88calibration sequences×16384\\times\\,16384averaged positions\) after centering, transformation, and grouping; empty clusters are reseeded each iteration from the points farthest from their centroid\. There is one codebook per group per \(layer, KV head\),256256centroids atg=4g=4, stored in float16 and snapped to fp8 offline; each cached subvector is an88\-bit index\. Encoding is an exact nearest\-neighbor search in the Euclidean metric of the transform domain\. The grouping sorts entries bylog⁡σi2\\log\\sigma\_\{i\}^\{2\}, takingσi2\\sigma\_\{i\}^\{2\}from the calibration spectrum, and deals them to theLLgroups in turn; the procedure is deterministic\. We detail the algorithmic training steps in Algorithms[2](https://arxiv.org/html/2608.04074#alg2)and[3](https://arxiv.org/html/2608.04074#alg3)\.

Algorithm 2Query\-aware transform and grouping1:

MMtoken positions; at each position

jj, the post\-RoPE key

𝐤j\\mathbf\{k\}\_\{j\}of this KV head and the post\-RoPE queries, shared across

HHheads,

𝐪j\(1\),…,𝐪j\(H\)\\mathbf\{q\}\_\{j\}^\{\(1\)\},\\dots,\\mathbf\{q\}\_\{j\}^\{\(H\)\}
2:

𝐑K\\mathbf\{R\}\_\{K\},

𝐑K−⊤\\mathbf\{R\}\_\{K\}^\{\-\\top\},

𝐤¯\\bar\{\\mathbf\{k\}\}
3:

𝐌q←∑j=1M∑h=1H𝐪j\(h\)⊤​𝐪j\(h\)\\mathbf\{M\}\_\{q\}\\leftarrow\\sum\_\{j=1\}^\{M\}\\sum\_\{h=1\}^\{H\}\\mathbf\{q\}\_\{j\}^\{\(h\)\\top\}\\mathbf\{q\}\_\{j\}^\{\(h\)\}⊳\\trianglerightsummed over theHHshared query heads \(Appendix[12\.5](https://arxiv.org/html/2608.04074#S12.SS5)\)

4:

𝐤¯←1/M​∑j𝐤j\\bar\{\\mathbf\{k\}\}\\leftarrow\{1\}/\{M\}\\sum\_\{j\}\\mathbf\{k\}\_\{j\};

𝐒~k←∑j\(𝐤j−𝐤¯\)⊤​\(𝐤j−𝐤¯\)\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\leftarrow\\sum\_\{j\}\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)^\{\\top\}\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)
5:

𝐄​𝚲​𝐄⊤←eig⁡\(𝐌q1/2​𝐒~k​𝐌q1/2\)\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}\\leftarrow\\operatorname\{eig\}\(\\mathbf\{M\}\_\{q\}^\{1/2\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\mathbf\{M\}\_\{q\}^\{1/2\}\),

λ1≥⋯≥λd\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{d\}
6:

𝐑K←𝐌q1/2​𝐄\\mathbf\{R\}\_\{K\}\\leftarrow\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{E\};

𝐑K−⊤←𝐌q−1/2​𝐄\\mathbf\{R\}\_\{K\}^\{\-\\top\}\\leftarrow\\mathbf\{M\}\_\{q\}^\{\-1/2\}\\mathbf\{E\}
7:

π​\(r\)←rmodL\\pi\(r\)\\leftarrow r\\bmod L,

r=0,…,d−1r=0,\\dots,d\-1by decreasing

λ\\lambda
8:

𝐑K←𝐑K​\[:,π\]\\mathbf\{R\}\_\{K\}\\leftarrow\\mathbf\{R\}\_\{K\}\[:,\\pi\];

𝐑K−⊤←𝐑K−⊤​\[:,π\]\\mathbf\{R\}\_\{K\}^\{\-\\top\}\\leftarrow\\mathbf\{R\}\_\{K\}^\{\-\\top\}\[:,\\pi\]⊳\\trianglerightfolded in

Algorithm 3Codebook training \(per layer, KV head\)1:keys

\{𝐤j\}\\\{\\mathbf\{k\}\_\{j\}\\\},

𝐑K\\mathbf\{R\}\_\{K\},

𝐤¯\\bar\{\\mathbf\{k\}\}; rate

bb
2:codebooks

\{𝒞ℓ\}ℓ=1L\\\{\\mathcal\{C\}\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}
3:for

j=1,…,Mj=1,\\dots,Mdo

4:

𝐫j←\(𝐤j−𝐤¯\)​𝐑K\\mathbf\{r\}\_\{j\}\\leftarrow\(\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\)\\mathbf\{R\}\_\{K\}
5:

ρj←∥𝐫j∥2/d\\rho\_\{j\}\\leftarrow\\lVert\\mathbf\{r\}\_\{j\}\\rVert\_\{2\}/\\sqrt\{d\};

𝐫j←ρj−1​𝐫j\\mathbf\{r\}\_\{j\}\\leftarrow\\rho\_\{j\}^\{\-1\}\\mathbf\{r\}\_\{j\}⊳\\trianglerightNormalize

6:endfor

7:for

ℓ=1,…,L\\ell=1,\\dots,Ldo

8:

𝒞ℓ←k​\-means​\(\{𝐫j​\[Gℓ\]\}j,2g​b\)\\mathcal\{C\}\_\{\\ell\}\\leftarrow k\\text\{\-means\}\\bigl\(\\\{\\mathbf\{r\}\_\{j\}\[G\_\{\\ell\}\]\\\}\_\{j\},\\,2^\{gb\}\\bigr\)⊳\\trianglerightequal size for every group

9:endfor

The per\-token scaleρj\\rho\_\{j\}in Algorithm[3](https://arxiv.org/html/2608.04074#alg3)normalizes each transformed key before the codebooks are trained\. It is not part of the model of Theorem 2, which treats the entries as independent Gaussians with fixed variancesσi2\\sigma\_\{i\}^\{2\}\.

### 8\.4Serving and evaluation

We run in a research fork based on SGLang v0\.5\.10 with custom decoding kernels\. Software versions for the two hardware setups are listed in Table[9](https://arxiv.org/html/2608.04074#S8.T9)\. The accuracy evaluations use the Triton attention backend for both prefill and decode\. Chunked prefill uses40964096\-token chunks on the dense models and81928192on GPT\-OSS\. RULER NIAH uses88subtasks \(niah\_single\_1/2/3,niah\_multikey\_1/2/3,niah\_multivalue,niah\_multiquery\); item counts, samples per prompt, generation caps, and per\-item metrics are collected in Table[10](https://arxiv.org/html/2608.04074#S8.T10)\. Baselines use their own reference calibration procedure, under the same sink\-plus\-recent band and the same chunk size as ours\.*Naive INT2*denotes per\-token INT2 applied directly to keys and values with no transform and no sink/recent band; it collapses to0\.00\.0on every model and task \(main\-text Table[3](https://arxiv.org/html/2608.04074#S5.T3)\)\. For the OSCAR\-basis rows of the transform–quantizer ablation \(main\-text Table[4](https://arxiv.org/html/2608.04074#S5.T4)\), codebooks are refit on that basis with the same group size, codebook size, and training budget as ours; the KLT comparison \(main\-text Fig\.[3](https://arxiv.org/html/2608.04074#S5.F3)\) instead uses entropy\-coded scalar quantization\.

### 8\.5Decoding kernel and read path

The read path is a fused Triton kernel that never multiplies by the transform\. The synthesis𝐑K−⊤\\mathbf\{R\}\_\{K\}^\{\-\\top\}is folded into the query once per step: the engine forms the modified query𝐪~=𝐪​𝐑K−⊤\\tilde\{\\mathbf\{q\}\}=\\mathbf\{q\}\\,\\mathbf\{R\}\_\{K\}^\{\-\\top\}\(a singled×dd\\times dproduct, amortized over the whole cache\), so scoring a cached key reduces to the inner product of𝐪~\\tilde\{\\mathbf\{q\}\}with the dequantized code𝒬−​\(𝐤^\)\\mathcal\{Q\}^\{\-\}\(\\hat\{\\mathbf\{k\}\}\)\. For each of theLLgroups the stored88\-bit index selects agg\-dimensional centroid from that group’s codebook; the kernel accumulates the partial logit across groups and applies the per\-token scaleρj\\rho\_\{j\}recovered from metadata \(one multiply\)\. Because a shared mean shifts every logit of a query by a constant,𝐤¯\\bar\{\\mathbf\{k\}\}cancels in the softmax \(Sec\.[12\.1](https://arxiv.org/html/2608.04074#S12.SS1)\) and is not re\-added on the read path\. On the value side the INT2 code is dequantized in the OSCAR rotation basis, whose inverse is absorbed into the output projection, so it adds no online matmul either\. Thus the per\-element read cost is a table lookup and a scale, matching scalar dequantization\. We implement the read path as a two\-stage split\-KV decode kernel, with gather and dequantize fused with the attention math in stage 1 and stage\-2 split\-combine; the split count follows the engine’s occupancy heuristic\.

#### Value quantizer\.

We keep SQ for the values: replacing the value SQ with VQ barely changes downstream accuracy\. We compared two configurations, the decorrelating transform𝐔S\\mathbf\{U\}\_\{S\}alone \(as suggested by Cor\. 1\) and𝐔S\\mathbf\{U\}\_\{S\}followed by a Hadamard transform \(closest to the deployed SQ value path\)\. Both stay within22points of the deployed SQ \(Table[6](https://arxiv.org/html/2608.04074#S8.T6)\)\. While VQ is slightly more accurate at a slightly lower rate \(2\.162\.16vs\.2\.222\.22BPE\), we keep SQ for read\-path simplicity\.

Table 6:VQ on the values under two transforms, on RULER NIAH with Qwen3\-8B, at fixed group size, codebook size, and rate\. Mean accuracy \(%\) with CIs drawn from33evaluation rollouts\. SQ corresponds to our deployed method\. BPE=2\.222\.22for the SQ method,2\.162\.16for the VQ method\.
#### Implementation

We consider two implementations of the read path of Sec\.[8\.5](https://arxiv.org/html/2608.04074#S8.SS5)\. Both realize the same two\-stage split\-KV decode attention, in which stage 1 fuses the codebook lookup, the per\-token scale, and the attention arithmetic, and stage 2 combines the splits; they share the cache layout and the fp8 codebooks\. The*Triton*kernel is the reference implementation, with launch configurations selected offline per batch size and geometry\. The*CUDA*kernel is a hand\-written stage 1 with a warp\-tiled codebook gather and fp32 accumulation; it reduces the instruction count of the gather\-and\-dequantize loop, which lowers the attention read at batch 128 on Qwen3\-8B from48\.948\.9to36\.436\.4ms per step \(Table[7](https://arxiv.org/html/2608.04074#S8.T7)\)\. The CUDA kernel is the one measured in the main\-text throughput comparison \(main\-text Fig\.[4](https://arxiv.org/html/2608.04074#S5.F4)\)\.

#### Per\-step kernel profiling

Table 7:Per\-decoding\-step kernel time on Qwen3\-8B, one H100, 8192\-token contexts\. Entries are ms per step with the share of the step in parentheses \(%\)\. The BF16 pool admits at most 38 concurrent requests at this length\.Table 8:Per\-decoding\-step kernel time on GPT\-OSS\-20B, same protocol as Table[7](https://arxiv.org/html/2608.04074#S8.T7)\.We adopt the per\-step profiling protocol ofzhou2026oscar: one server per arm on a single H100, 8192\-token contexts, batches 1 to 128, and GPU kernel time per decoding step split by kernel name into*GEMM*\(projections, MLP, and MoE experts\),*Attn*\(the fused attention read, including dequantization and the split combine\),*Quant*\(the write path: codebook encoder, per\-step query preparation, OSCAR pack\), and*Other*\(norms, activations, rotary, sampling, BF16\-band writes\)\. Profiling starts in steady state, after every warmed request is decoding, and stops after exactly 64 forward steps; per\-step figures divide device time by the step count recovered from the trace\. Tables[7](https://arxiv.org/html/2608.04074#S8.T7)and[8](https://arxiv.org/html/2608.04074#S8.T8)report the breakdown\. The kernel sum matches the wall clock per step within1%1\\%in every cell, and unassigned kernels are below0\.2%0\.2\\%\. New tokens enter the BF16 recent band and are quantized in blocks of eight steps in every quantized arm, so the scalar arm’s Quant column is smaller than a per\-step fused quantizer would show\. The BF16 and OSCAR columns reproduce the profile reported byzhou2026oscarwithin about2%2\\%at every batch size, which puts the NOVA\-KV columns on a verified scale\.

#### Qwen3\-8B \(Table[7](https://arxiv.org/html/2608.04074#S8.T7)\)\.

GEMM is flat at about66ms for every arm: parameter load is unaffected by quantization, so the step is a fixed GEMM cost plus an attention read that grows with batch \(the regime of main\-text Fig\.[1](https://arxiv.org/html/2608.04074#S1.F1)\)\. From batch 16 on, every 2\-bit arm is faster wherever BF16 runs \(at batch 8 the Triton arm’s write\-path cost still outweighs the small read saving\), and only the 2\-bit arms serve batches 64 and 128\. At batch 128, NOVA\-KV \(CUDA\) pays1\.15×1\.15\\timesOSCAR’s attention read \(36\.436\.4against31\.631\.6ms\) plus5\.35\.3ms of write\-path cost, for a1\.26×1\.26\\timesstep \(49\.749\.7against39\.339\.3ms\); the write\-path term grows from0\.30\.3ms at batch 1 and is the larger contribution beyond batch 64\. This is the kernel\-level form of the main\-text statement that decoding is on par with the scalar baseline: the difference is confined to the read and the write path\.

#### GPT\-OSS\-20B \(Table[8](https://arxiv.org/html/2608.04074#S8.T8)\)\.

At this context no 2\-bit arm beats BF16 per step: only 12 of 24 layers are quantized, the global\-layer cache at 8192 tokens is small, and atd=64d\{=\}64the dequantizing read is bound by instruction issue rather than bandwidth, so the write\-path cost is not repaid\. The MoE GEMMs dominate the step and vary across arms by more than the totals differ \(routing depends on the token streams\)\. What quantization buys here is the batch itself: the 2\-bit arms serve batch 128, which the BF16 pool cannot hold\. The main\-text gains for this model come from longer contexts and larger admitted batches, not from per\-step speed at 8192 tokens\.

### 8\.6Sampling and statistical testing

Generation is stochastic\. For each prompt, we draw five samples on the generation benchmarks and three samples on RULER NIAH\. The compression parameters \(transforms, groupings, codebooks, thekk\-means initialization, and the random comparison partition of Sec\.[4](https://arxiv.org/html/2608.04074#S4)\) are fitted once, with fixed seeds, and held fixed across these samples; the repetitions therefore characterize generation and evaluation variability, not variability from refitting the compression method\. Confidence intervals are computed by bootstrap over samples \(1000010000resamples\)\.

McNemar’s test compares two methods evaluated on the same items when each item’s outcome is binary \(correct/incorrect\)\. For each item, the pair of outcomes falls into one of four cells: both methods correct, both incorrect, only method A correct, or only method B correct\. Items where the two methods agree carry no information about their difference, so the test uses only the discordant items: withn01n\_\{01\}items that only A solves andn10n\_\{10\}that only B solves, it tests the null hypothesis that a discordant item is equally likely to fall either way, i\.e\., that the two methods have the same per\-item error rate\. We use the exact binomial form: under the null,n01∼Binomial​\(n01\+n10,1/2\)n\_\{01\}\\sim\\mathrm\{Binomial\}\(n\_\{01\}\{\+\}n\_\{10\},\\,1/2\)\.

We form binary outcomes by majority voting over the samples drawn for a given item \(5 on the generation benchmarks, 3 on RULER NIAH\), and run the test over the items that both compared methods evaluate on\. For the generation benchmarks the five task sets are combined into a single test of992992paired items per model; on RULER NIAH the test is run at128128K on Qwen3\-8B over200200paired items\. Tests are two\-sided and reported without any further correction\.

### 8\.7Datasets and licenses

We evaluate on six public benchmarks\.*RULER NIAH*\(hsieh2024ruler\)is a synthetic needle\-in\-a\-haystack retrieval suite \(Apache\-2\.0\)\.*GPQA\-Diamond*\(rein2023gpqa\)is graduate\-level multiple\-choice science \(CC BY 4\.0\)\.*HumanEval*\(chen2021evaluating\)is Python code synthesis graded by execution \(MIT\)\.*LiveCodeBench v6*\(jain2025livecodebench\)is contamination\-controlled competitive programming \(benchmark released under MIT; problem statements originate from competition platforms\)\.*AIME25*consists of the 2025 AIME competition problems, copyright © Mathematical Association of America, used here for research evaluation\(maa2025aime\)\.*MATH500*\(lightman2024let;hendrycks2021measuring\)is a500500\-problem subset of the MATH dataset \(MIT\)\. The models are Llama\-3\.1\-8B \(Llama 3\.1 Community License\)\(grattafiori2024llama\), Qwen3\-8B / Qwen3\-4B\-Thinking\-2507 \(Apache\-2\.0\)\(yang2025qwen3\), and GPT\-OSS\-20B \(Apache\-2\.0\)\(agarwal2025gpt\)\. All datasets and models are used for research evaluation consistent with their licenses\.

### 8\.8Throughput protocol and tuning

The decode\-throughput comparison \(main\-text Fig\.[4](https://arxiv.org/html/2608.04074#S5.F4)\) runs one server per arm on a single H100\. Prompts are distinct random token streams with no shared prefix\. Before the measured pass, a warm\-up pass prefills every request’s own prompt into the cache; the measured pass then reuses these prefixes, so all requests enter decoding together and throughput is computed over the decode window alone\. Every arm runs its best configuration: OSCAR takes the best of eight decode\-kernel tile configurations swept per cell, NOVA\-KV \(the CUDA kernel, Sec\.[8\.5](https://arxiv.org/html/2608.04074#S8.SS5.SSS0.Px2)\) takes the best of its compiled variants per cell, and BF16 runs the engine’s dense read with an uncapped memory\-fraction pool\. All arms share the same chunk size, split\-KV budget, and fp32 accumulation in the attention read\.

### 8\.9Configuration summary

Tables[9](https://arxiv.org/html/2608.04074#S8.T9)and[10](https://arxiv.org/html/2608.04074#S8.T10)collect the infrastructure, serving, method, and per\-benchmark settings described above\.

Table 9:Infrastructure, serving, and hyperparameters\.Table 10:Per\-benchmark evaluation settings\. Samples = samples per prompt; Gen = max generated tokens\.

## 9Calibration ablation

We vary the calibration set and refit the transform, grouping, and codebooks \(Table[11](https://arxiv.org/html/2608.04074#S9.T11)\)\. Reducing from198198to3232GPQA\-Diamond prompts \(4040K to8\.98\.9K tokens\) changes RULER NIAH by1\.11\.1points at6464K and1\.11\.1at128128K, both upward\. Calibrating on MMLU instead, a disjoint domain with116116K tokens, gives74\.474\.4and75\.475\.4, within2\.02\.0and0\.00\.0points of the default\. GPQA accuracy under MMLU calibration is59\.359\.3, against57\.257\.2for BF16 \(Table[3](https://arxiv.org/html/2608.04074#S5.T3)\)\.

Table[12](https://arxiv.org/html/2608.04074#S9.T12)extends this comparison to the five reasoning and coding benchmarks\. The effect of the calibration domain is small and model\-dependent\. Relative to the deployed GPQA\-Diamond calibration, MMLU raises the Qwen3\-8B mean by2\.02\.0points but lowers the Qwen3\-4B\-Thinking mean by0\.70\.7\. Both stay within two points of the deployed calibration and within2\.32\.3points of BF16\. For consistency with\(zhou2026oscar\), we use GPQA\-Diamond in our main evaluations\.

Table 11:Calibration data ablation, Qwen3\-8B\. We vary the calibration set and token count; GPQA accuracy does not benefit from calibrating on GPQA\. Mean accuracy \(%\)±\\pm95% CI over 3 rollouts\.Table 12:Calibration domain ablation on NOVA\-KV, for Qwen3\-4B\-Thinking\-2507 and Qwen3\-8B\. Compressed rows are NOVA\-KV at2\.222\.22BPE;*GPQA\-198*is the deployed calibration, and*MMLU*a disjoint domain \(116116K tokens\)\. Entries are mean±\\pm95% CI with55samples per prompt; “Drop” is the gap in the Mean column to the BF16 reference\.
## 10Partition statistics

![Refer to caption](https://arxiv.org/html/2608.04074v1/x5.png)Figure 7:Distribution of group volumes under the three partitions, averaged over all \(layer, KV head\) pairs of Qwen3\-8B atg=4g=4\. Each count is one group; volumes are normalized byvequalv\_\{\\mathrm\{equal\}\}, the common value they would take under a perfectly volume\-equalizing partition \(dashed line\)\. The horizontal axis is logarithmic\.The fixed\-rate case of Theorem 2 bounds the distortion of a partitionπ\\pibyCg​2−2​b​∑ℓvℓ​\(π\)1/g≥D∗​\(b\)C\_\{g\}\\,2^\{\-2b\}\\sum\_\{\\ell\}v\_\{\\ell\}\(\\pi\)^\{1/g\}\\geq D^\{\*\}\(b\), with equality iff all group volumes coincide\. In theory, the gap is a property of the volume distribution \(cf\. Sec\.[14](https://arxiv.org/html/2608.04074#S14)\)\. Fig\.[7](https://arxiv.org/html/2608.04074#S10.F7)shows that distribution for the three partitions compared in Sec\.[5](https://arxiv.org/html/2608.04074#S5)\. Volumes are normalized by:

vequal=\(∏i=1dσi2\)g/d,v\_\{\\mathrm\{equal\}\}=\\left\(\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}\\right\)^\{g/d\},\(9\)the group volume under an exactly equalizing partition; hence, a partition that equalizes exactly places all mass on11\.

The three panels differ by orders of magnitude in spread\. Grouping consecutive entries in eigenvalue order \(a\) produces volumes spanning roughly 20 decades, since the leading group collects the largestσi2\\sigma\_\{i\}^\{2\}and the trailing group the smallest\. A random partition \(b\) is narrower, as each group draws entries from across the spectrum, but the spread remains close to 3 decades\. The volume\-aware partition \(c\) concentrates almost all groups with a small tail attributable to the greedy dealing rule\.

The ordering of the three spreads matches the ordering of the corresponding accuracies reported in Sec\.[5](https://arxiv.org/html/2608.04074#S5), which is consistent with Theorem 2\. We note that the theorem is derived under the high\-resolution model with independent Gaussian coefficients, so the agreement is evidence that the predicted ordering survives outside the regime of the derivation rather than a verification of the bound \(cf\. Sec\.[14](https://arxiv.org/html/2608.04074#S14)\)\.

## 11Analytical tools

### 11\.1Companding for non\-difference distortions

This subsection states the high\-resolution theory for distortion measuresρ​\(⋅,⋅\)\\rho\(\\cdot,\\cdot\)other than the MSE, following the companding framework oflinder1999high, which covers locally quadratic measures whose weight may be input\-dependent\. The criterion in \([5](https://arxiv.org/html/2608.04074#S3.E5)\), that is,

‖𝐐𝐊⊤−𝐐​𝐊^⊤‖F2=∑j=1M\(𝐤j−𝐤^j\)​𝐌q​\(𝐤j−𝐤^j\)⊤,\\bigl\\\|\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\-\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}\\bigr\\\|\_\{F\}^\{2\}=\\sum\_\{j=1\}^\{M\}\\,\(\\mathbf\{k\}\_\{j\}\-\\widehat\{\\mathbf\{k\}\}\_\{j\}\)\\,\\mathbf\{M\}\_\{q\}\\,\(\\mathbf\{k\}\_\{j\}\-\\widehat\{\\mathbf\{k\}\}\_\{j\}\)^\{\\top\},\(10\)is a special case of this theory, as we will show next\.

#### Locally quadratic distortions\.

Supposeρ​\(⋅,⋅\)\\rho\(\\cdot,\\cdot\)is three times continuously differentiable in its second argument, vanishes exactly when its second argument equals the first argument, and has positive definite second derivative at that point\. Define the*sensitivity matrix*

\[𝐌​\(𝐱\)\]i​j=12​∂2ρ​\(𝐱,𝐱^\)∂x^i​∂x^j\|𝐱^=𝐱\.\\bigl\[\\mathbf\{M\}\(\\mathbf\{x\}\)\\bigr\]\_\{ij\}=\\frac\{1\}\{2\}\\,\\frac\{\\partial^\{2\}\\rho\(\\mathbf\{x\},\\hat\{\\mathbf\{x\}\}\)\}\{\\partial\\hat\{x\}\_\{i\}\\,\\partial\\hat\{x\}\_\{j\}\}\\Big\|\_\{\\hat\{\\mathbf\{x\}\}=\\mathbf\{x\}\}\.\(11\)Because the gradient vanishes at𝐱^=𝐱\\hat\{\\mathbf\{x\}\}=\\mathbf\{x\}, a second\-order expansion gives

ρ​\(𝐱,𝐱^\)=\(𝐱−𝐱^\)​𝐌​\(𝐱\)​\(𝐱−𝐱^\)⊤\+O​\(‖𝐱−𝐱^‖3\),\\rho\(\\mathbf\{x\},\\hat\{\\mathbf\{x\}\}\)=\(\\mathbf\{x\}\-\\hat\{\\mathbf\{x\}\}\)\\,\\mathbf\{M\}\(\\mathbf\{x\}\)\\,\(\\mathbf\{x\}\-\\hat\{\\mathbf\{x\}\}\)^\{\\top\}\+O\\bigl\(\\\|\\mathbf\{x\}\-\\hat\{\\mathbf\{x\}\}\\\|^\{3\}\\bigr\),\(12\)so at high resolution the distortion is a quadratic form with a possibly input\-dependent weight\. An input\-weighted quadratic measure

ρ​\(𝐱,𝐱^\)=‖\(𝐱−𝐱^\)​𝐖​\(𝐱\)‖22,\\rho\(\\mathbf\{x\},\\hat\{\\mathbf\{x\}\}\)=\\\|\(\\mathbf\{x\}\-\\hat\{\\mathbf\{x\}\}\)\\mathbf\{W\}\(\\mathbf\{x\}\)\\\|\_\{2\}^\{2\},\(13\)is of this type, with𝐌​\(𝐱\)=𝐖​\(𝐱\)​𝐖​\(𝐱\)⊤\\mathbf\{M\}\(\\mathbf\{x\}\)=\\mathbf\{W\}\(\\mathbf\{x\}\)\\mathbf\{W\}\(\\mathbf\{x\}\)^\{\\top\}\. In our criterion \([5](https://arxiv.org/html/2608.04074#S3.E5)\), the weight does not depend on the input,𝐌​\(𝐱\)≡𝐌q\\mathbf\{M\}\(\\mathbf\{x\}\)\\equiv\\mathbf\{M\}\_\{q\}, and the expansion \([12](https://arxiv.org/html/2608.04074#S11.E12)\) is exact\.

#### Companding quantizers\.

A compander applies an invertible maphhto the source, quantizes with a lattice quantizer𝒬ℒ\\mathcal\{Q\}\_\{\\mathcal\{L\}\}, and inverts:

𝐱⟶h​\(𝐱\)⟶𝒬ℒ​\(h​\(𝐱\)\)⟶h−1⟶𝐱^\.\\mathbf\{x\}\\;\\longrightarrow\\;h\(\\mathbf\{x\}\)\\;\\longrightarrow\\;\\mathcal\{Q\}\_\{\\mathcal\{L\}\}\\bigl\(h\(\\mathbf\{x\}\)\\bigr\)\\;\\longrightarrow\\;h^\{\-1\}\\;\\longrightarrow\\;\\hat\{\\mathbf\{x\}\}\.\(14\)Write𝐉​\(𝐱\)\\mathbf\{J\}\(\\mathbf\{x\}\)for the Jacobian ofhh, so thatd​h=d​𝐱​𝐉​\(𝐱\)\\mathrm\{d\}h=\\mathrm\{d\}\\mathbf\{x\}\\,\\mathbf\{J\}\(\\mathbf\{x\}\)\. The compander is a structured vector quantizer: all adaptation to the source and to the distortion is carried byhh, while the cells in the companded domain are congruent, i\.e\., translates of the basic lattice cell, identical in shape and volume\. Mapped back throughh−1h^\{\-1\}, these identical cells become the non\-uniform cells of the effective quantizer in the source domain\. A linearh​\(𝐱\)=𝐱𝐑h\(\\mathbf\{x\}\)=\\mathbf\{x\}\\mathbf\{R\}, with𝐉≡𝐑\\mathbf\{J\}\\equiv\\mathbf\{R\}a typical transform coder\. In this case,d​h=d​𝐱​𝐑\\mathrm\{d\}h=\\mathrm\{d\}\\mathbf\{x\}\\,\\mathbf\{R\}\.

#### Asymptotic rate\.

LetZgZ\_\{g\}denote the normalized second moment of the basic cell of the lattice\.linder1999highshow that, for a source with finite differential entropyH​\(𝐱\)H\(\\mathbf\{x\}\), the rate of the compander, defined as the entropyℋ​\(D\)\\mathcal\{H\}\(D\)of the quantizer indices when the lattice is scaled to operate at distortionDD, satisfies

limD→0\(ℋ​\(D\)\+g/2​log2⁡D\)=H​\(𝐱\)\+𝔼​\[log2⁡\|det𝐉​\(𝐱\)\|\]\+g/2​log2⁡\(g​Zg​𝔼​\[tr⁡𝚪​\(𝐱\)\]\),\\lim\_\{D\\to 0\}\\Bigl\(\\mathcal\{H\}\(D\)\+\{g\}/\{2\}\\log\_\{2\}D\\Bigr\)=H\(\\mathbf\{x\}\)\+\\mathbb\{E\}\\bigl\[\\log\_\{2\}\|\\det\\mathbf\{J\}\(\\mathbf\{x\}\)\|\\bigr\]\+\{g\}/\{2\}\\log\_\{2\}\\Bigl\(g\\,Z\_\{g\}\\,\\mathbb\{E\}\\bigl\[\\operatorname\{tr\}\\boldsymbol\{\\Gamma\}\(\\mathbf\{x\}\)\\bigr\]\\Bigr\),\(15\)where

𝚪​\(𝐱\)=𝐉​\(𝐱\)−1​𝐌​\(𝐱\)​𝐉​\(𝐱\)−⊤,\\boldsymbol\{\\Gamma\}\(\\mathbf\{x\}\)=\\mathbf\{J\}\(\\mathbf\{x\}\)^\{\-1\}\\,\\mathbf\{M\}\(\\mathbf\{x\}\)\\,\\mathbf\{J\}\(\\mathbf\{x\}\)^\{\-\\top\},\(16\)measures the sensitivity as seen in the companded domain\.

#### The optimal compressor\.

Minimizing \([15](https://arxiv.org/html/2608.04074#S11.E15)\) overhhuses two inequalities\. Since𝚪​\(𝐱\)≻0\\boldsymbol\{\\Gamma\}\(\\mathbf\{x\}\)\\succ 0, the arithmetic–geometric mean inequality givestr⁡𝚪≥g​\(det𝚪\)1/g\\operatorname\{tr\}\\boldsymbol\{\\Gamma\}\\geq g\\,\(\\det\\boldsymbol\{\\Gamma\}\)^\{1/g\}, with equality iff the eigenvalues of𝚪\\boldsymbol\{\\Gamma\}are all equal; Jensen’s inequality then moves the expectation inside the logarithm\. Together\(linder1999high, Thm\. 2\):

limD→0\(ℋ​\(D\)\+g/2​log2⁡D\)≥H​\(𝐱\)\+g/2​log2⁡\(g​Zg\)\+1/2​𝔼​\[log2​det𝐌​\(𝐱\)\],\\lim\_\{D\\to 0\}\\Bigl\(\\mathcal\{H\}\(D\)\+\{g\}/\{2\}\\log\_\{2\}D\\Bigr\)\\geq H\(\\mathbf\{x\}\)\+\{g\}/\{2\}\\log\_\{2\}\\bigl\(g\\,Z\_\{g\}\\bigr\)\+\{1\}/\{2\}\\,\\mathbb\{E\}\\bigl\[\\log\_\{2\}\\det\\mathbf\{M\}\(\\mathbf\{x\}\)\\bigr\],\(17\)with equality if and only if

𝐉​\(𝐱\)​𝐉​\(𝐱\)⊤=c​𝐌​\(𝐱\)a\.e\., for some​c\>0\.\\mathbf\{J\}\(\\mathbf\{x\}\)\\,\\mathbf\{J\}\(\\mathbf\{x\}\)^\{\\top\}=c\\,\\mathbf\{M\}\(\\mathbf\{x\}\)\\quad\\text\{a\.e\., for some \}c\>0\.\(18\)Three consequences:

1. \(C1\)*The optimal compressor does not depend on the source distribution*, only on the distortion measure\. This is the analogue, for locally quadratic distortions, of the fact that the optimal entropy\-coded quantizer for MSE is uniform regardless of the source\.
2. \(C2\)*Condition \([18](https://arxiv.org/html/2608.04074#S11.E18)\) fixeshhonly up to an orthogonal factor on the right*, since replacing𝐉\\mathbf\{J\}by𝐉𝐎\\mathbf\{J\}\\mathbf\{O\}with𝐎\\mathbf\{O\}orthogonal leaves𝐉𝐉⊤\\mathbf\{J\}\\mathbf\{J\}^\{\\top\}unchanged\. The condition therefore constrains the stretch but not the rotation\.
3. \(C3\)*An orthogonalhhsatisfies \([18](https://arxiv.org/html/2608.04074#S11.E18)\) only when𝐌∝𝐈\\mathbf\{M\}\\propto\\mathbf\{I\}*, since𝐉𝐉⊤=𝐈\\mathbf\{J\}\\mathbf\{J\}^\{\\top\}=\\mathbf\{I\}then forces𝐌=c−1​𝐈\\mathbf\{M\}=c^\{\-1\}\\mathbf\{I\}\.

#### For the key transform\.

Take𝐌​\(𝐱\)≡𝐌q\\mathbf\{M\}\(\\mathbf\{x\}\)\\equiv\\mathbf\{M\}\_\{q\}, constant\. The compressor is then linear,h​\(𝐱\)=𝐱𝐑h\(\\mathbf\{x\}\)=\\mathbf\{x\}\\mathbf\{R\}, and \([18](https://arxiv.org/html/2608.04074#S11.E18)\) reads𝐑𝐑⊤=c​𝐌q\\mathbf\{R\}\\mathbf\{R\}^\{\\top\}=c\\,\\mathbf\{M\}\_\{q\}\. By[\(C3\)](https://arxiv.org/html/2608.04074#S11.I1.i3), orthogonal transforms are suboptimal for the attention\-aware criterion unless the queries are isotropic; by[\(C2\)](https://arxiv.org/html/2608.04074#S11.I1.i2), Theorem 1 must resolve the remaining orthogonal factor, which we do by minimizing the rank\-ppreconstruction error\.

### 11\.2Softmax Perturbation Bound

We prove that the logit error controls the attention weights \(beginning of Sec\.[3](https://arxiv.org/html/2608.04074#S3)\)\.

###### Proposition S1\.

Let𝐙,𝐙^∈ℝT×T\\mathbf\{Z\},\\widehat\{\\mathbf\{Z\}\}\\in\\mathbb\{R\}^\{T\\times T\},𝐒=softmaxrow⁡\(𝐙/d\)\\mathbf\{S\}=\\operatorname\{softmax\}\_\{\\mathrm\{row\}\}\(\\mathbf\{Z\}/\\sqrt\{d\}\), and𝐒^=softmaxrow⁡\(𝐙^/d\)\\widehat\{\\mathbf\{S\}\}=\\operatorname\{softmax\}\_\{\\mathrm\{row\}\}\(\\widehat\{\\mathbf\{Z\}\}/\\sqrt\{d\}\)\. Then,‖𝐒−𝐒^‖F≤1/\(2​d\)​‖𝐙−𝐙^‖F\\\|\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\\\|\_\{F\}\\leq\{1\}/\(\{2\\sqrt\{d\}\}\)\\,\\\|\\mathbf\{Z\}\-\\widehat\{\\mathbf\{Z\}\}\\\|\_\{F\}\.

###### Proof\.

Consider one row\. The Jacobian ofsoftmax\\operatorname\{softmax\}at𝐳\\mathbf\{z\}is𝐉​\(𝐳\)=diag⁡\(𝐬\)−𝐬⊤​𝐬\\mathbf\{J\}\(\\mathbf\{z\}\)=\\operatorname\{diag\}\(\\mathbf\{s\}\)\-\\mathbf\{s\}^\{\\top\}\\mathbf\{s\},𝐬=softmax⁡\(𝐳\)\\mathbf\{s\}=\\operatorname\{softmax\}\(\\mathbf\{z\}\), symmetric positive semidefinite with‖𝐉​\(𝐳\)‖2≤1/2\\\|\\mathbf\{J\}\(\\mathbf\{z\}\)\\\|\_\{2\}\\leq 1/2\(gao2017properties\)\. Along the segment𝐳​\(t\)=\(1−t\)​𝐳^\+t​𝐳\\mathbf\{z\}\(t\)=\(1\-t\)\\hat\{\\mathbf\{z\}\}\+t\\mathbf\{z\},

‖softmax⁡\(𝐳\)−softmax⁡\(𝐳^\)‖2≤supt∈\[0,1\]‖𝐉​\(𝐳​\(t\)\)‖2​‖𝐳−𝐳^‖2≤1/2​‖𝐳−𝐳^‖2\.\\\|\\operatorname\{softmax\}\(\\mathbf\{z\}\)\-\\operatorname\{softmax\}\(\\hat\{\\mathbf\{z\}\}\)\\\|\_\{2\}\\leq\\sup\_\{t\\in\[0,1\]\}\\\|\\mathbf\{J\}\(\\mathbf\{z\}\(t\)\)\\\|\_\{2\}\\,\\\|\\mathbf\{z\}\-\\hat\{\\mathbf\{z\}\}\\\|\_\{2\}\\leq 1/2\\,\\\|\\mathbf\{z\}\-\\hat\{\\mathbf\{z\}\}\\\|\_\{2\}\.\(19\)Applying this to each row of𝐙/d\\mathbf\{Z\}/\\sqrt\{d\},𝐙^/d\\widehat\{\\mathbf\{Z\}\}/\\sqrt\{d\}and summing squares over rows gives the claim\. ∎

With𝐙=𝐐𝐊⊤\\mathbf\{Z\}=\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}and𝐙^=𝐐​𝐊^⊤\\widehat\{\\mathbf\{Z\}\}=\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}, the attention\-weight error is bounded by1/\(2​d\)​‖𝐐𝐊⊤−𝐐​𝐊^⊤‖F1/\(2\\sqrt\{d\}\)\\,\\\|\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\-\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}\\\|\_\{F\}\.

### 11\.3Decoupling of the Output Error

We justify the decomposition of Sec\.[3](https://arxiv.org/html/2608.04074#S3)\. Let𝐒^=softmaxrow⁡\(𝐐​𝐊^⊤/d\)\\widehat\{\\mathbf\{S\}\}=\\operatorname\{softmax\}\_\{\\mathrm\{row\}\}\(\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}/\\sqrt\{d\}\)be the scores computed from reconstructed keys and𝐎^=𝐒^​𝐕^\\widehat\{\\mathbf\{O\}\}=\\widehat\{\\mathbf\{S\}\}\\widehat\{\\mathbf\{V\}\}the output under joint key and value quantization\.

###### Proposition S2\.

‖𝐎−𝐎^‖F≤‖𝐕‖2/\(2​d\)​‖𝐐𝐊⊤−𝐐​𝐊^⊤‖F\+‖𝐒​\(𝐕−𝐕^\)‖F\+‖\(𝐒−𝐒^\)​\(𝐕−𝐕^\)‖F\.\\\|\\mathbf\{O\}\-\\widehat\{\\mathbf\{O\}\}\\\|\_\{F\}\\leq\{\\\|\\mathbf\{V\}\\\|\_\{2\}\}/\(\{2\\sqrt\{d\}\}\)\\,\\bigl\\\|\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\-\\mathbf\{Q\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}\\bigr\\\|\_\{F\}\+\\bigl\\\|\\mathbf\{S\}\(\\mathbf\{V\}\-\\widehat\{\\mathbf\{V\}\}\)\\bigr\\\|\_\{F\}\+\\bigl\\\|\(\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\)\(\\mathbf\{V\}\-\\widehat\{\\mathbf\{V\}\}\)\\bigr\\\|\_\{F\}\.\(20\)

###### Proof\.

Adding and subtracting𝐒​𝐕^\\mathbf\{S\}\\widehat\{\\mathbf\{V\}\}gives the exact identity

𝐎−𝐎^=\(𝐒−𝐒^\)​𝐕\+𝐒​\(𝐕−𝐕^\)−\(𝐒−𝐒^\)​\(𝐕−𝐕^\)\.\\mathbf\{O\}\-\\widehat\{\\mathbf\{O\}\}=\(\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\)\\mathbf\{V\}\+\\mathbf\{S\}\(\\mathbf\{V\}\-\\widehat\{\\mathbf\{V\}\}\)\-\(\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\)\(\\mathbf\{V\}\-\\widehat\{\\mathbf\{V\}\}\)\.\(21\)The triangle inequality,‖\(𝐒−𝐒^\)​𝐕‖F≤‖𝐒−𝐒^‖F​‖𝐕‖2\\\|\(\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\)\\mathbf\{V\}\\\|\_\{F\}\\leq\\\|\\mathbf\{S\}\-\\widehat\{\\mathbf\{S\}\}\\\|\_\{F\}\\\|\\mathbf\{V\}\\\|\_\{2\}, and Proposition[S1](https://arxiv.org/html/2608.04074#ThmpropositionS1)give the claim\. ∎

The first term is controlled by the key objective \([5](https://arxiv.org/html/2608.04074#S3.E5)\); the second is the value objective of Corollary 1 \(Sec\.[12\.4](https://arxiv.org/html/2608.04074#S12.SS4)\); the third is a product of the two quantization errors, hence second order\. We do not claim joint optimality of the two\-step design for the combined objective\. Consistently with this decomposition, the value objective below treats the scores as computed from full\-precision keys; the discrepancy from using𝐒^\\widehat\{\\mathbf\{S\}\}instead is absorbed by the second\-order term\.

## 12Optimal transforms

### 12\.1Lemma S1: Softmax Offset Invariance

We prove in this section that adding a constant to all keys leaves the attention scores unaltered\.

###### Lemma S1\.

For any𝐜∈ℝ1×d\\mathbf\{c\}\\in\\mathbb\{R\}^\{1\\times d\}and any query𝐪\\mathbf\{q\},softmaxj⁡\(𝐪​\(𝐤j−𝐜\)⊤\)=softmaxj⁡\(𝐪𝐤j⊤\)\\operatorname\{softmax\}\_\{j\}\\bigl\(\\mathbf\{q\}\(\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}\)^\{\\top\}\\bigr\)=\\operatorname\{softmax\}\_\{j\}\\bigl\(\\mathbf\{q\}\\mathbf\{k\}\_\{j\}^\{\\top\}\\bigr\)\.

###### Proof\.

Replacing each key by𝐤j−𝐜\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}shifts every logit of the query𝐪\\mathbf\{q\}by the same amount𝐪𝐜⊤\\mathbf\{q\}\\mathbf\{c\}^\{\\top\}, which cancels:

exp⁡\(𝐪​\(𝐤j−𝐜\)⊤\)∑j′exp⁡\(𝐪​\(𝐤j′−𝐜\)⊤\)=exp⁡\(𝐪​𝐤j⊤\)∑j′exp⁡\(𝐪​𝐤j′⊤\)\.\\frac\{\\exp\\bigl\(\\mathbf\{q\}\\,\(\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}\)^\{\\top\}\\bigr\)\}\{\\sum\_\{j^\{\\prime\}\}\\exp\\bigl\(\\mathbf\{q\}\\,\(\\mathbf\{k\}\_\{j^\{\\prime\}\}\-\\mathbf\{c\}\)^\{\\top\}\\bigr\)\}=\\frac\{\\exp\\bigl\(\\mathbf\{q\}\\,\\mathbf\{k\}\_\{j\}^\{\\top\}\\bigr\)\}\{\\sum\_\{j^\{\\prime\}\}\\exp\\bigl\(\\mathbf\{q\}\\,\\mathbf\{k\}\_\{j^\{\\prime\}\}^\{\\top\}\\bigr\)\}\.\(22\)∎

By Lemma S1, replacing every key𝐤j\\mathbf\{k\}\_\{j\}by𝐤j−𝐜\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}leaves the attention unchanged, so we may quantize the shifted keys instead and minimize

min𝐜,𝐃,\{𝐫j\}​∑j=1M‖𝐤j−𝐜−𝐫j​𝐃‖𝐌q2,\\min\_\{\\mathbf\{c\},\\,\\mathbf\{D\},\\,\\\{\\mathbf\{r\}\_\{j\}\\\}\}\\sum\_\{j=1\}^\{M\}\\bigl\\\|\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}\-\\mathbf\{r\}\_\{j\}\\mathbf\{D\}\\bigr\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\},\(23\)which coincides with the original objective at𝐜=𝟎\\mathbf\{c\}=\\mathbf\{0\}; optimizing over𝐜\\mathbf\{c\}therefore never increases the error, and we choose𝐜\\mathbf\{c\}to make the keys cheaper to compress\.

### 12\.2Theorem 1: Key Transform

We find the transform that minimizes the low\-rank reconstruction𝐌q\\mathbf\{M\}\_\{q\}\-MSE given training examples\. We write𝐑†=\(𝐑⊤​𝐌q−1​𝐑\)−1​𝐑⊤​𝐌q−1\\mathbf\{R\}^\{\\dagger\}=\(\\mathbf\{R\}^\{\\top\}\\mathbf\{M\}\_\{q\}^\{\-1\}\\mathbf\{R\}\)^\{\-1\}\\mathbf\{R\}^\{\\top\}\\mathbf\{M\}\_\{q\}^\{\-1\}for the𝐌q\\mathbf\{M\}\_\{q\}\-weighted pseudoinverse, so that𝐑𝐑†\\mathbf\{R\}\\mathbf\{R\}^\{\\dagger\}is the projector onto the column space of𝐑\\mathbf\{R\}that is orthogonal in the𝐌q\\mathbf\{M\}\_\{q\}inner product\.

###### Theorem 1 \(restated\)\.

Let𝐤~j=𝐤j−𝐤¯\\widetilde\{\\mathbf\{k\}\}\_\{j\}=\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\},𝐒~k=∑j𝐤~j⊤​𝐤~j\\widetilde\{\\mathbf\{S\}\}\_\{k\}=\\sum\_\{j\}\\widetilde\{\\mathbf\{k\}\}\_\{j\}^\{\\top\}\\widetilde\{\\mathbf\{k\}\}\_\{j\}, and let

𝐑K,p⋆=arg​min𝐑∈ℝd×p​∑j=1M‖𝐤~j−𝐤~j​𝐑𝐑†‖𝐌q2\.\\mathbf\{R\}^\{\\star\}\_\{K,p\}=\\operatorname\*\{arg\\,min\\,\}\_\{\\mathbf\{R\}\\,\\in\\,\\mathbb\{R\}^\{d\\times p\}\}\\,\\sum\_\{j=1\}^\{M\}\\,\\bigl\\\|\\,\\widetilde\{\\mathbf\{k\}\}\_\{j\}\-\\widetilde\{\\mathbf\{k\}\}\_\{j\}\\,\\mathbf\{R\}\\mathbf\{R\}^\{\\dagger\}\\bigr\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\}\.Then𝐑K,p⋆=𝐌q1/2​𝐄1:p\\mathbf\{R\}^\{\\star\}\_\{K,p\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\mathbf\{E\}\_\{1:p\}, where𝐄​𝚲​𝐄⊤\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}is the eigendecomposition of𝐌q1/2​𝐒~k​𝐌q1/2\\mathbf\{M\}\_\{q\}^\{1/2\}\\,\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\,\\mathbf\{M\}\_\{q\}^\{1/2\}\.

We prove a slightly more general statement\. Rather than fixing the reconstruction to the𝐌q\\mathbf\{M\}\_\{q\}\-orthogonal projection𝐤~j​𝐑𝐑†\\widetilde\{\\mathbf\{k\}\}\_\{j\}\\mathbf\{R\}\\mathbf\{R\}^\{\\dagger\}, we leave the codes, the synthesis matrix, and a shared offset free, and minimize the logit error \([5](https://arxiv.org/html/2608.04074#S3.E5)\) directly\. In particular, we parameterize a rank\-pplinear reconstruction with codes𝐫j∈ℝ1×p\\mathbf\{r\}\_\{j\}\\in\\mathbb\{R\}^\{1\\times p\}, a shared synthesis matrix𝐃∈ℝp×d\\mathbf\{D\}\\in\\mathbb\{R\}^\{p\\times d\}of full row rank, and a shared offset𝐜∈ℝ1×d\\mathbf\{c\}\\in\\mathbb\{R\}^\{1\\times d\}:

𝐤^j=𝐫j​𝐃\+𝐜,\\widehat\{\\mathbf\{k\}\}\_\{j\}=\\mathbf\{r\}\_\{j\}\\mathbf\{D\}\+\\mathbf\{c\},\(24\)and minimize

L​\(𝐃,𝐜,\{𝐫j\}\)=∑i=1N∑j=1M\(𝐪i​𝐤j⊤−𝐪i​𝐤^j⊤\)2\.L\(\\mathbf\{D\},\\mathbf\{c\},\\\{\\mathbf\{r\}\_\{j\}\\\}\)=\\sum\_\{i=1\}^\{N\}\\sum\_\{j=1\}^\{M\}\\bigl\(\\mathbf\{q\}\_\{i\}\\mathbf\{k\}\_\{j\}^\{\\top\}\-\\mathbf\{q\}\_\{i\}\\widehat\{\\mathbf\{k\}\}\_\{j\}^\{\\top\}\\bigr\)^\{2\}\.\(25\)We show our result in 5 steps\.

#### Step 1: reorder to expose the query second moment\.

With𝜹j=𝐤j−𝐜−𝐫j​𝐃\\boldsymbol\{\\delta\}\_\{j\}=\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}\-\\mathbf\{r\}\_\{j\}\\mathbf\{D\},

∑i,j\(𝐪i​𝜹j⊤\)2=∑j=1M𝜹j​\(∑i=1N𝐪i⊤​𝐪i\)​𝜹j⊤=∑j=1M‖𝜹j‖𝐌q2,\\sum\_\{i,j\}\\bigl\(\\mathbf\{q\}\_\{i\}\\boldsymbol\{\\delta\}\_\{j\}^\{\\top\}\\bigr\)^\{2\}=\\sum\_\{j=1\}^\{M\}\\boldsymbol\{\\delta\}\_\{j\}\\Bigl\(\\sum\_\{i=1\}^\{N\}\\mathbf\{q\}\_\{i\}^\{\\top\}\\mathbf\{q\}\_\{i\}\\Bigr\)\\boldsymbol\{\\delta\}\_\{j\}^\{\\top\}=\\sum\_\{j=1\}^\{M\}\\\|\\boldsymbol\{\\delta\}\_\{j\}\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\},\(26\)with𝐌q=𝐐⊤​𝐐\\mathbf\{M\}\_\{q\}=\\mathbf\{Q\}^\{\\top\}\\mathbf\{Q\}, which is \([5](https://arxiv.org/html/2608.04074#S3.E5)\)\. Invertibility of𝐌q\\mathbf\{M\}\_\{q\}requires the calibration queries to spanℝd\\mathbb\{R\}^\{d\}\(in particularN≥dN\\geq d\), which holds for typical calibration sets; in practice we enforce positive\-definiteness as in Appendix[7\.3](https://arxiv.org/html/2608.04074#S7.SS3)\.

#### Step 2: optimal codes for fixed𝐃,𝐜\\mathbf\{D\},\\mathbf\{c\}\.

The loss decouples acrossjj\. Setting the gradient of‖𝜹j‖𝐌q2\\\|\\boldsymbol\{\\delta\}\_\{j\}\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\}in𝐫j\\mathbf\{r\}\_\{j\}to zero,

𝐫j​𝐃𝐌q​𝐃⊤=\(𝐤j−𝐜\)​𝐌q​𝐃⊤⟹𝐫j∗=\(𝐤j−𝐜\)​𝐌q​𝐃⊤​\(𝐃𝐌q​𝐃⊤\)−1,\\mathbf\{r\}\_\{j\}\\,\\mathbf\{D\}\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}=\(\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}\)\\,\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\\;\\Longrightarrow\\;\\mathbf\{r\}\_\{j\}^\{\*\}=\(\\mathbf\{k\}\_\{j\}\-\\mathbf\{c\}\)\\,\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\\bigl\(\\mathbf\{D\}\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\\bigr\)^\{\-1\},\(27\)a generalized least\-squares projection onto the row space of𝐃\\mathbf\{D\}, measured in the𝐌q\\mathbf\{M\}\_\{q\}inner product\.

#### Step 3: optimal offset\.

For fixed𝐃\\mathbf\{D\}and codes,∇𝐜L=−2​∑j𝜹j​𝐌q=𝟎\\nabla\_\{\\mathbf\{c\}\}L=\-2\\sum\_\{j\}\\boldsymbol\{\\delta\}\_\{j\}\\mathbf\{M\}\_\{q\}=\\mathbf\{0\}\. Since𝐌q≻0\\mathbf\{M\}\_\{q\}\\succ 0, the condition reduces to∑j𝜹j=𝟎\\sum\_\{j\}\\boldsymbol\{\\delta\}\_\{j\}=\\mathbf\{0\}, i\.e\.,𝐜∗=𝐤¯−𝐫¯​𝐃\\mathbf\{c\}^\{\*\}=\\bar\{\\mathbf\{k\}\}\-\\bar\{\\mathbf\{r\}\}\\mathbf\{D\}with𝐫¯=1/M​∑j𝐫j\\bar\{\\mathbf\{r\}\}=1/M\\sum\_\{j\}\\mathbf\{r\}\_\{j\}\. Any choice satisfying this equation is optimal; we pick the one that centers the codes,𝐫¯=𝟎\\bar\{\\mathbf\{r\}\}=\\mathbf\{0\}, so that no rate is spent encoding the code mean𝐜∗=𝐤¯\\mathbf\{c\}^\{\*\}=\\bar\{\\mathbf\{k\}\}\. We write𝐤~j=𝐤j−𝐤¯\\tilde\{\\mathbf\{k\}\}\_\{j\}=\\mathbf\{k\}\_\{j\}\-\\bar\{\\mathbf\{k\}\}\.

#### Step 4: optimal synthesis\.

Substituting \([27](https://arxiv.org/html/2608.04074#S12.E27)\),𝐫j∗​𝐃=𝐤~j​𝐏\\mathbf\{r\}\_\{j\}^\{\*\}\\mathbf\{D\}=\\tilde\{\\mathbf\{k\}\}\_\{j\}\\mathbf\{P\}with𝐏=𝐌q​𝐃⊤​\(𝐃𝐌q​𝐃⊤\)−1​𝐃\\mathbf\{P\}=\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\(\\mathbf\{D\}\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\)^\{\-1\}\\mathbf\{D\}, the oblique projector onto the row space of𝐃\\mathbf\{D\}that is orthogonal in the𝐌q\\mathbf\{M\}\_\{q\}inner product \(𝐏2=𝐏\\mathbf\{P\}^\{2\}=\\mathbf\{P\}, and𝐏𝐌q\\mathbf\{P\}\\mathbf\{M\}\_\{q\}is symmetric\)\. Using these two identities,\(𝐈−𝐏\)​𝐌q​\(𝐈−𝐏\)⊤=𝐌q−𝐏𝐌q\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{M\}\_\{q\}\(\\mathbf\{I\}\-\\mathbf\{P\}\)^\{\\top\}=\\mathbf\{M\}\_\{q\}\-\\mathbf\{P\}\\mathbf\{M\}\_\{q\}, and with𝐒~k=∑j𝐤~j⊤​𝐤~j\\widetilde\{\\mathbf\{S\}\}\_\{k\}=\\sum\_\{j\}\\tilde\{\\mathbf\{k\}\}\_\{j\}^\{\\top\}\\tilde\{\\mathbf\{k\}\}\_\{j\},

L∗​\(𝐃\)=tr⁡\[𝐌q​𝐒~k\]−tr⁡\[𝐏𝐌q​𝐒~k\]\.L^\{\*\}\(\\mathbf\{D\}\)=\\operatorname\{tr\}\\bigl\[\\mathbf\{M\}\_\{q\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\bigr\]\-\\operatorname\{tr\}\\bigl\[\\mathbf\{P\}\\mathbf\{M\}\_\{q\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\bigr\]\.\(28\)The first term is constant, so we maximizeT​\(𝐃\)=tr⁡\[𝐌q​𝐃⊤​\(𝐃𝐌q​𝐃⊤\)−1​𝐃​𝐌q​𝐒~k\]T\(\\mathbf\{D\}\)=\\operatorname\{tr\}\\bigl\[\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\(\\mathbf\{D\}\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}\)^\{\-1\}\\mathbf\{D\}\\,\\mathbf\{M\}\_\{q\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\bigr\]\.

#### Step 5: change of variables and Ky Fan\.

Let𝐁=𝐃𝐌q1/2∈ℝp×d\\mathbf\{B\}=\\mathbf\{D\}\\mathbf\{M\}\_\{q\}^\{1/2\}\\in\\mathbb\{R\}^\{p\\times d\}, so𝐃𝐌q​𝐃⊤=𝐁𝐁⊤\\mathbf\{D\}\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}=\\mathbf\{B\}\\mathbf\{B\}^\{\\top\}and𝐌q​𝐃⊤=𝐌q1/2​𝐁⊤\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\\top\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{B\}^\{\\top\}\. ThenT=tr⁡\[𝚷𝐁​𝐀\],𝐀=𝐌q1/2​𝐒~k​𝐌q1/2,𝚷𝐁=𝐁⊤​\(𝐁𝐁⊤\)−1​𝐁T=\\operatorname\{tr\}\\bigl\[\\boldsymbol\{\\Pi\}\_\{\\mathbf\{B\}\}\\,\\mathbf\{A\}\\bigr\],\\quad\\mathbf\{A\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\widetilde\{\\mathbf\{S\}\}\_\{k\}\\mathbf\{M\}\_\{q\}^\{1/2\},\\quad\\boldsymbol\{\\Pi\}\_\{\\mathbf\{B\}\}=\\mathbf\{B\}^\{\\top\}\(\\mathbf\{B\}\\mathbf\{B\}^\{\\top\}\)^\{\-1\}\\mathbf\{B\}, where𝚷𝐁\\boldsymbol\{\\Pi\}\_\{\\mathbf\{B\}\}is a rank\-pporthogonal projector and𝐀⪰0\\mathbf\{A\}\\succeq 0\. Maximizingtr⁡\[𝚷​𝐀\]\\operatorname\{tr\}\[\\boldsymbol\{\\Pi\}\\mathbf\{A\}\]over rank\-pporthogonal projectors is the classical PCA problem, solved by the projector onto the top\-ppeigenvectors of𝐀\\mathbf\{A\}with maximum∑i≤pλi​\(𝐀\)\\sum\_\{i\\leq p\}\\lambda\_\{i\}\(\\mathbf\{A\}\)\(fan1949theorem\)\. With𝐀=𝐄​𝚲​𝐄⊤\\mathbf\{A\}=\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}and𝐄1:p\\mathbf\{E\}\_\{1:p\}the top\-ppeigenvectors, take𝐁∗=𝐄1:p⊤\\mathbf\{B\}^\{\*\}=\\mathbf\{E\}\_\{1:p\}^\{\\top\}, i\.e\.,

𝐃∗=𝐄1:p⊤​𝐌q−1/2=𝐑†\.\\mathbf\{D\}^\{\*\}=\\mathbf\{E\}\_\{1:p\}^\{\\top\}\\mathbf\{M\}\_\{q\}^\{\-1/2\}=\\mathbf\{R\}^\{\\dagger\}\.\(29\)Then𝐃∗​𝐌q​𝐃∗⊤=𝐈p\\mathbf\{D\}^\{\*\}\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\*\\top\}=\\mathbf\{I\}\_\{p\}, and \([27](https://arxiv.org/html/2608.04074#S12.E27)\) simplifies to

𝐫j∗=𝐤~j​𝐌q​𝐃∗⊤=𝐤~j​𝐌q1/2​𝐄1:p=𝐤~j​𝐑,\\mathbf\{r\}\_\{j\}^\{\*\}=\\tilde\{\\mathbf\{k\}\}\_\{j\}\\,\\mathbf\{M\}\_\{q\}\\mathbf\{D\}^\{\*\\top\}=\\tilde\{\\mathbf\{k\}\}\_\{j\}\\,\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{E\}\_\{1:p\}=\\tilde\{\\mathbf\{k\}\}\_\{j\}\\,\\mathbf\{R\},\(30\)with𝐑=𝐌q1/2​𝐄1:p=𝐑K,p⋆\\mathbf\{R\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{E\}\_\{1:p\}=\\mathbf\{R\}^\{\\star\}\_\{K,p\}\. The reconstruction is therefore𝐤^j=𝐤~j​𝐑𝐑†\+𝐤¯\\widehat\{\\mathbf\{k\}\}\_\{j\}=\\widetilde\{\\mathbf\{k\}\}\_\{j\}\\mathbf\{R\}\\mathbf\{R\}^\{\\dagger\}\+\\bar\{\\mathbf\{k\}\}, the projected form of the theorem, and the optimal cost is∑i\>pλi​\(𝐀\)\\sum\_\{i\>p\}\\lambda\_\{i\}\(\\mathbf\{A\}\)\. Forp=dp=d,𝐄\\mathbf\{E\}is orthogonal and𝐑𝐑†=𝐈\\mathbf\{R\}\\mathbf\{R\}^\{\\dagger\}=\\mathbf\{I\}\.■\\blacksquare

### 12\.3Proposition 1: Generalized Parseval Relation

Throughout,𝐑K=𝐑K,d⋆\\mathbf\{R\}\_\{K\}=\\mathbf\{R\}^\{\\star\}\_\{K,d\}denotes the full\-rank \(p=dp=d\) transform of Theorem 1, which is invertible\.

###### Proposition 1 \(restated\)\.

Let𝐫\\mathbf\{r\}and𝐫^\\hat\{\\mathbf\{r\}\}be any two vectors, and𝐤=𝐫𝐑K−1\\mathbf\{k\}=\\mathbf\{r\}\\mathbf\{R\}\_\{K\}^\{\-1\}and𝐤^=𝐫^​𝐑K−1\\hat\{\\mathbf\{k\}\}=\\hat\{\\mathbf\{r\}\}\\mathbf\{R\}\_\{K\}^\{\-1\}\. Then,‖𝐫−𝐫^‖22=‖𝐤−𝐤^‖𝐌q2\\bigl\\\|\\mathbf\{r\}\-\\hat\{\\mathbf\{r\}\}\\bigr\\\|\_\{2\}^\{2\}=\\bigl\\\|\\mathbf\{k\}\-\\hat\{\\mathbf\{k\}\}\\bigr\\\|\_\{\\mathbf\{M\}\_\{q\}\}^\{2\}\.

###### Proof\.

With𝐫−𝐫^=\(𝐤−𝐤^\)​𝐑\\mathbf\{r\}\-\\hat\{\\mathbf\{r\}\}=\(\\mathbf\{k\}\-\\hat\{\\mathbf\{k\}\}\)\\mathbf\{R\},

‖𝐫−𝐫^‖22=\(𝐤−𝐤^\)​𝐑𝐑⊤​\(𝐤−𝐤^\)⊤,\\\|\\mathbf\{r\}\-\\hat\{\\mathbf\{r\}\}\\\|\_\{2\}^\{2\}=\(\\mathbf\{k\}\-\\hat\{\\mathbf\{k\}\}\)\\,\\mathbf\{R\}\\mathbf\{R\}^\{\\top\}\(\\mathbf\{k\}\-\\hat\{\\mathbf\{k\}\}\)^\{\\top\},\(31\)and𝐑𝐑⊤=𝐌q1/2​𝐄𝐄⊤​𝐌q1/2=𝐌q\\mathbf\{R\}\\mathbf\{R\}^\{\\top\}=\\mathbf\{M\}\_\{q\}^\{1/2\}\\mathbf\{E\}\\mathbf\{E\}^\{\\top\}\\mathbf\{M\}\_\{q\}^\{1/2\}=\\mathbf\{M\}\_\{q\}, since𝐄\\mathbf\{E\}is orthogonal forp=dp=d\. ∎

### 12\.4Corollary 1: Value Transform

###### Corollary 1 \(restated\)\.

Let𝐨i=𝐬i​𝐕\\mathbf\{o\}\_\{i\}=\\mathbf\{s\}\_\{i\}\\mathbf\{V\}denote the attention outputs on calibration data, and𝐄​𝚲​𝐄⊤\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}the eigendecomposition of𝐌o=𝐕⊤​𝐌s​𝐕\\mathbf\{M\}\_\{o\}=\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}\. The minimizer of‖𝐒𝐕−𝐒​𝐕^‖F2\\\|\\mathbf\{S\}\\mathbf\{V\}\-\\mathbf\{S\}\\widehat\{\\mathbf\{V\}\}\\\|\_\{F\}^\{2\}over low\-rank approximations is𝐑V=𝐄1:p\\mathbf\{R\}\_\{V\}=\\mathbf\{E\}\_\{1:p\}, with codes𝐬j=𝐯j​𝐑V\\mathbf\{s\}\_\{j\}=\\mathbf\{v\}\_\{j\}\\mathbf\{R\}\_\{V\}and reconstruction𝐯^j=𝐬j​𝐑V⊤\\widehat\{\\mathbf\{v\}\}\_\{j\}=\\mathbf\{s\}\_\{j\}\\mathbf\{R\}\_\{V\}^\{\\top\}\.

Let𝜶i∈ℝ1×M\\boldsymbol\{\\alpha\}\_\{i\}\\in\\mathbb\{R\}^\{1\\times M\}denote theiith row of𝐒\\mathbf\{S\}, so the attention output for calibration queryiiis𝐨i=𝜶i​𝐕∈ℝ1×d\\mathbf\{o\}\_\{i\}=\\boldsymbol\{\\alpha\}\_\{i\}\\mathbf\{V\}\\in\\mathbb\{R\}^\{1\\times d\}\. We parameterize a rank\-pplinear reconstruction with codes𝐬j∈ℝ1×p\\mathbf\{s\}\_\{j\}\\in\\mathbb\{R\}^\{1\\times p\}and synthesis𝐃V∈ℝp×d\\mathbf\{D\}\_\{V\}\\in\\mathbb\{R\}^\{p\\times d\}of full row rank,𝐯^j=𝐬j​𝐃V\\widehat\{\\mathbf\{v\}\}\_\{j\}=\\mathbf\{s\}\_\{j\}\\mathbf\{D\}\_\{V\}, and minimize the output distortion

L​\(𝐃V,\{𝐬j\}\)=∑i=1N‖𝜶i​𝐕−𝜶i​𝐕^‖22\.L\(\\mathbf\{D\}\_\{V\},\\\{\\mathbf\{s\}\_\{j\}\\\}\)=\\sum\_\{i=1\}^\{N\}\\bigl\\\|\\boldsymbol\{\\alpha\}\_\{i\}\\mathbf\{V\}\-\\boldsymbol\{\\alpha\}\_\{i\}\\widehat\{\\mathbf\{V\}\}\\bigr\\\|\_\{2\}^\{2\}\.\(32\)
#### Step 1: reorder to expose the score second moment\.

With𝜹j=𝐯j−𝐬j​𝐃V\\boldsymbol\{\\delta\}\_\{j\}=\\mathbf\{v\}\_\{j\}\-\\mathbf\{s\}\_\{j\}\\mathbf\{D\}\_\{V\}and𝚫∈ℝM×d\\boldsymbol\{\\Delta\}\\in\\mathbb\{R\}^\{M\\times d\}stacking the𝜹j\\boldsymbol\{\\delta\}\_\{j\}as rows,

∑i‖∑jαi​j​𝜹j‖22=∑j,j′\(𝐌s\)j​j′​𝜹j​𝜹j′⊤=tr⁡\[𝚫⊤​𝐌s​𝚫\],\\sum\_\{i\}\\Bigl\\\|\\sum\_\{j\}\\alpha\_\{ij\}\\,\\boldsymbol\{\\delta\}\_\{j\}\\Bigr\\\|\_\{2\}^\{2\}=\\sum\_\{j,j^\{\\prime\}\}\(\\mathbf\{M\}\_\{s\}\)\_\{jj^\{\\prime\}\}\\,\\boldsymbol\{\\delta\}\_\{j\}\\boldsymbol\{\\delta\}\_\{j^\{\\prime\}\}^\{\\top\}=\\operatorname\{tr\}\\bigl\[\\boldsymbol\{\\Delta\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\boldsymbol\{\\Delta\}\\bigr\],\(33\)with𝐌s=𝐒⊤​𝐒=∑i𝜶i⊤​𝜶i\\mathbf\{M\}\_\{s\}=\\mathbf\{S\}^\{\\top\}\\mathbf\{S\}=\\sum\_\{i\}\\boldsymbol\{\\alpha\}\_\{i\}^\{\\top\}\\boldsymbol\{\\alpha\}\_\{i\}\. Attention reads values through score\-weighted sums \(mixing tokens\)\.

#### Step 2: optimal codes for fixed𝐃V\\mathbf\{D\}\_\{V\}\.

Writing𝚫=𝐕−𝐒c​𝐃V\\boldsymbol\{\\Delta\}=\\mathbf\{V\}\-\\mathbf\{S\}\_\{c\}\\mathbf\{D\}\_\{V\}with𝐒c∈ℝM×p\\mathbf\{S\}\_\{c\}\\in\\mathbb\{R\}^\{M\\times p\}stacking the codes gives

𝐌s​𝐒c​𝐃V​𝐃V⊤=𝐌s​𝐕𝐃V⊤\.\\mathbf\{M\}\_\{s\}\\,\\mathbf\{S\}\_\{c\}\\,\\mathbf\{D\}\_\{V\}\\mathbf\{D\}\_\{V\}^\{\\top\}=\\mathbf\{M\}\_\{s\}\\,\\mathbf\{V\}\\mathbf\{D\}\_\{V\}^\{\\top\}\.\(34\)A solution independent of𝐌s\\mathbf\{M\}\_\{s\}is

𝐒c∗=𝐕𝐃V⊤​\(𝐃V​𝐃V⊤\)−1,\\mathbf\{S\}\_\{c\}^\{\*\}=\\mathbf\{V\}\\mathbf\{D\}\_\{V\}^\{\\top\}\\bigl\(\\mathbf\{D\}\_\{V\}\\mathbf\{D\}\_\{V\}^\{\\top\}\\bigr\)^\{\-1\},\(35\)the ordinary Euclidean projection of each𝐯j\\mathbf\{v\}\_\{j\}onto the row space of𝐃V\\mathbf\{D\}\_\{V\}: the weight lives on the token axis, orthogonal to the head\-space axis along which the projection acts, so the projection geometry is standard\.111If𝐌s\\mathbf\{M\}\_\{s\}is invertible, \([35](https://arxiv.org/html/2608.04074#S12.E35)\) is the unique solution\. Invertibility is not required for what follows: the optimal basis depends on𝐌s\\mathbf\{M\}\_\{s\}only through thed×dd\\times dmatrix𝐌o=𝐕⊤​𝐌s​𝐕\\mathbf\{M\}\_\{o\}=\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}, which has rankmin⁡\(d,N\)\\min\(d,N\)regardless of the cache lengthMM\.

#### Step 3: optimal synthesis\.

Substituting,𝐒c∗​𝐃V=𝐕𝐏\\mathbf\{S\}\_\{c\}^\{\*\}\\mathbf\{D\}\_\{V\}=\\mathbf\{V\}\\mathbf\{P\}with𝐏=𝐃V⊤​\(𝐃V​𝐃V⊤\)−1​𝐃V\\mathbf\{P\}=\\mathbf\{D\}\_\{V\}^\{\\top\}\(\\mathbf\{D\}\_\{V\}\\mathbf\{D\}\_\{V\}^\{\\top\}\)^\{\-1\}\\mathbf\{D\}\_\{V\}, a standard rank\-pporthogonal projector\. Then

tr⁡\[\(𝐈−𝐏\)​𝐕⊤​𝐌s​𝐕​\(𝐈−𝐏\)\]=tr⁡\[𝐌o\]−tr⁡\[𝐏𝐌o\],\\operatorname\{tr\}\\bigl\[\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\,\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}\\,\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\bigr\]=\\operatorname\{tr\}\[\\mathbf\{M\}\_\{o\}\]\-\\operatorname\{tr\}\[\\mathbf\{P\}\\mathbf\{M\}\_\{o\}\],\(36\)with𝐌o=𝐕⊤​𝐌s​𝐕\\mathbf\{M\}\_\{o\}=\\mathbf\{V\}^\{\\top\}\\mathbf\{M\}\_\{s\}\\mathbf\{V\}\. Maximizingtr⁡\[𝐏𝐌o\]\\operatorname\{tr\}\[\\mathbf\{P\}\\mathbf\{M\}\_\{o\}\]over rank\-pporthogonal projectors is again the PCA problem of Sec\.[12\.2](https://arxiv.org/html/2608.04074#S12.SS2), Step 5: with𝐌o=𝐄​𝚲​𝐄⊤\\mathbf\{M\}\_\{o\}=\\mathbf\{E\}\\boldsymbol\{\\Lambda\}\\mathbf\{E\}^\{\\top\}, the maximizer is𝐏∗=𝐄1:p​𝐄1:p⊤\\mathbf\{P\}^\{\*\}=\\mathbf\{E\}\_\{1:p\}\\mathbf\{E\}\_\{1:p\}^\{\\top\}, i\.e\.,𝐃V∗=𝐄1:p⊤\\mathbf\{D\}\_\{V\}^\{\*\}=\\mathbf\{E\}\_\{1:p\}^\{\\top\}and, by \([35](https://arxiv.org/html/2608.04074#S12.E35)\),𝐬j∗=𝐯j​𝐄1:p\\mathbf\{s\}\_\{j\}^\{\*\}=\\mathbf\{v\}\_\{j\}\\mathbf\{E\}\_\{1:p\}\. This is Corollary 1 with𝐑V=𝐄1:p\\mathbf\{R\}\_\{V\}=\\mathbf\{E\}\_\{1:p\}; since𝐄1:p\\mathbf\{E\}\_\{1:p\}has orthonormal columns, the synthesis is the transpose and no normalization is needed\.

#### Second moment of the attention outputs\.

Although𝐌o\\mathbf\{M\}\_\{o\}was defined through theM×MM\\times Mmatrix𝐌s\\mathbf\{M\}\_\{s\}, it is the empirical second moment of the attention outputs:

𝐕⊤​\(∑i𝜶i⊤​𝜶i\)​𝐕=∑i=1N\(𝜶i​𝐕\)⊤​\(𝜶i​𝐕\)=∑i=1N𝐨i⊤​𝐨i\.\\mathbf\{V\}^\{\\top\}\\Bigl\(\\sum\_\{i\}\\boldsymbol\{\\alpha\}\_\{i\}^\{\\top\}\\boldsymbol\{\\alpha\}\_\{i\}\\Bigr\)\\mathbf\{V\}=\\sum\_\{i=1\}^\{N\}\(\\boldsymbol\{\\alpha\}\_\{i\}\\mathbf\{V\}\)^\{\\top\}\(\\boldsymbol\{\\alpha\}\_\{i\}\\mathbf\{V\}\)=\\sum\_\{i=1\}^\{N\}\\mathbf\{o\}\_\{i\}^\{\\top\}\\mathbf\{o\}\_\{i\}\.\(37\)We never form𝐌s\\mathbf\{M\}\_\{s\}on the token axis, which grows with context length; we accumulate outer products of attention\-output vectors in head space, ad×dd\\times dmatrix\.■\\blacksquare

### 12\.5Query Statistics under Grouped\-Query Attention

In grouped\-query attention \(GQA\)\(ainslie2023gqa\), one KV head is shared byHHquery heads with queries𝐐\(1\),…,𝐐\(H\)\\mathbf\{Q\}^\{\(1\)\},\\dots,\\mathbf\{Q\}^\{\(H\)\}\. The logit error for the shared keys is

∑h=1H‖𝐐\(h\)​𝐊⊤−𝐐\(h\)​𝐊^⊤‖F2=∑j=1M\(𝐤j−𝐤^j\)​\(∑h=1H𝐌q\(h\)\)​\(𝐤j−𝐤^j\)⊤,\\sum\_\{h=1\}^\{H\}\\bigl\\\|\\mathbf\{Q\}^\{\(h\)\}\\mathbf\{K\}^\{\\top\}\-\\mathbf\{Q\}^\{\(h\)\}\\widehat\{\\mathbf\{K\}\}^\{\\top\}\\bigr\\\|\_\{F\}^\{2\}=\\sum\_\{j=1\}^\{M\}\(\\mathbf\{k\}\_\{j\}\-\\widehat\{\\mathbf\{k\}\}\_\{j\}\)\\Bigl\(\\sum\_\{h=1\}^\{H\}\\mathbf\{M\}\_\{q\}^\{\(h\)\}\\Bigr\)\(\\mathbf\{k\}\_\{j\}\-\\widehat\{\\mathbf\{k\}\}\_\{j\}\)^\{\\top\},\(38\)with𝐌q\(h\)=𝐐\(h\)⊤​𝐐\(h\)\\mathbf\{M\}\_\{q\}^\{\(h\)\}=\\mathbf\{Q\}^\{\(h\)\\top\}\\mathbf\{Q\}^\{\(h\)\}\. This has the same form as \([5](https://arxiv.org/html/2608.04074#S3.E5)\) with𝐌q=∑h𝐌q\(h\)\\mathbf\{M\}\_\{q\}=\\sum\_\{h\}\\mathbf\{M\}\_\{q\}^\{\(h\)\}, so we accumulate the query second moment over the heads sharing each KV head\.

## 13Quantizer design: grouping and rate allocation

### 13\.1Theorem 2: Grouping

###### Theorem 2 \(restated\)\.

Under the model \([7](https://arxiv.org/html/2608.04074#S4.E7)\), for any partitionπ\\pithe allocation minimizingD​\(π,\{bℓ\}\)D\(\\pi,\\\{b\_\{\\ell\}\\\}\)subject to∑ℓbℓ=L​b\\sum\_\{\\ell\}b\_\{\\ell\}=Lbis

bℓ∗​\(π\)=b\+12​g​log2⁡\(vℓ​\(π\)/∏mvm​\(π\)1/L\),b^\{\*\}\_\{\\ell\}\(\\pi\)\\;=\\;b\\;\+\\;\\frac\{1\}\{2g\}\\,\\log\_\{2\}\\Big\(\{v\_\{\\ell\}\(\\pi\)\}\\big/\{\\prod\_\{m\}v\_\{m\}\(\\pi\)^\{1/L\}\}\\Big\),which spends more bits on groups of larger volume\. The resulting optimal distortion,D∗​\(b\)=Cg​L​2−2​b​\(∏i=1dσi2\)1/dD^\{\*\}\(b\)=C\_\{g\}L\\,2^\{\-2b\}\\bigl\(\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}\\bigr\)^\{1/d\}, is the same for everyπ\\pi, assuming that rates can be the arbitrary real values from \([8](https://arxiv.org/html/2608.04074#S4.E8)\)\.

We prove this statement together with the fixed\-rate claim that follows Theorem 2 in the main text: underbℓ≡bb\_\{\\ell\}\\equiv b, the distortion of a partitionπ\\piisCg​2−2​b​∑ℓvℓ​\(π\)1/g≥D∗​\(b\)C\_\{g\}\\,2^\{\-2b\}\\sum\_\{\\ell\}v\_\{\\ell\}\(\\pi\)^\{1/g\}\\geq D^\{\*\}\(b\), with equality iffv1​\(π\)=⋯=vL​\(π\)v\_\{1\}\(\\pi\)=\\cdots=v\_\{L\}\(\\pi\)\.

By the model \([7](https://arxiv.org/html/2608.04074#S4.E7)\) the total distortion isD​\(π,\{bℓ\}\)=Cg​∑ℓ2−2​bℓ​vℓ​\(π\)1/gD\(\\pi,\\\{b\_\{\\ell\}\\\}\)=C\_\{g\}\\sum\_\{\\ell\}2^\{\-2b\_\{\\ell\}\}v\_\{\\ell\}\(\\pi\)^\{1/g\}; the constantCg\>0C\_\{g\}\>0is common to all groups, so it leaves the minimizingπ\\piand\{bℓ\}\\\{b\_\{\\ell\}\\\}unchanged and only rescales the optimal value\. We therefore solve

minπ⁡min\{bℓ\}:∑ℓbℓ=L​b​∑ℓ=1L2−2​bℓ​vℓ​\(π\)1/g,\\min\_\{\\pi\}\\;\\min\_\{\\\{b\_\{\\ell\}\\\}:\\,\\sum\_\{\\ell\}b\_\{\\ell\}=Lb\}\\;\\sum\_\{\\ell=1\}^\{L\}2^\{\-2b\_\{\\ell\}\}\\,v\_\{\\ell\}\(\\pi\)^\{1/g\},\(39\)and restoreCgC\_\{g\}in the optimal distortion at the end\.

#### Optimal allocation and distortion\.

For fixedπ\\pi, each term2−2​bℓ​vℓ1/g2^\{\-2b\_\{\\ell\}\}v\_\{\\ell\}^\{1/g\}is convex inbℓb\_\{\\ell\}and the constraint is affine, so first\-order conditions are sufficient\. Stationarity of the Lagrangianℒ=∑ℓ2−2​bℓ​vℓ1/g\+μ​\(∑ℓbℓ−L​b\)\\mathcal\{L\}=\\sum\_\{\\ell\}2^\{\-2b\_\{\\ell\}\}v\_\{\\ell\}^\{1/g\}\+\\mu\(\\sum\_\{\\ell\}b\_\{\\ell\}\-Lb\)gives2​ln⁡2⋅2−2​bℓ​vℓ1/g=μ2\\ln 2\\cdot 2^\{\-2b\_\{\\ell\}\}v\_\{\\ell\}^\{1/g\}=\\mufor allℓ\\ell: the optimal allocation equalizes per\-group distortions\. Solving and enforcing the constraint,

bℓ∗=b\+12​g​log2⁡\(vℓ​\(π\)/∏mvm1/L​\(π\)\),b\_\{\\ell\}^\{\*\}=b\+\\frac\{1\}\{2g\}\\log\_\{2\}\\bigg\(\{v\_\{\\ell\}\(\\pi\)\}\\big/\\prod\_\{m\}v^\{1/L\}\_\{m\}\(\\pi\)\\bigg\),\(40\)which is \([8](https://arxiv.org/html/2608.04074#S4.E8)\)\. Substituting back, every term equals2−2​b​\(∏mvm​\(π\)\)1/\(L​g\)2^\{\-2b\}\\bigl\(\\prod\_\{m\}v\_\{m\}\(\\pi\)\\bigr\)^\{1/\(Lg\)\}, so the inner minimum is

L​2−2​b​\(∏ℓvℓ​\(π\)\)1/\(L​g\)=L​2−2​b​\(∏i=1dσi2\)1/d,L\\,2^\{\-2b\}\\Bigl\(\\prod\_\{\\ell\}v\_\{\\ell\}\(\\pi\)\\Bigr\)^\{1/\(Lg\)\}=L\\,2^\{\-2b\}\\Bigl\(\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}\\Bigr\)^\{1/d\},\(41\)where the last equality uses∏ℓvℓ​\(π\)=∏i=1dσi2\\prod\_\{\\ell\}v\_\{\\ell\}\(\\pi\)=\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}for every partition andL​g=dLg=d\. The value is independent ofπ\\pi; restoring the common factor, the optimal distortion isD∗=Cg​L​2−2​b​\(∏i=1dσi2\)1/dD^\{\*\}=C\_\{g\}\\,L\\,2^\{\-2b\}\\bigl\(\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}\\bigr\)^\{1/d\}, proving the theorem\.

#### Fixed rates\.

Underbℓ≡bb\_\{\\ell\}\\equiv b, the distortion isCg​2−2​b​∑ℓvℓ​\(π\)1/gC\_\{g\}\\,2^\{\-2b\}\\sum\_\{\\ell\}v\_\{\\ell\}\(\\pi\)^\{1/g\}\. By the arithmetic–geometric mean inequality applied to the nonnegative numbersvℓ1/gv\_\{\\ell\}^\{1/g\},

∑ℓ=1Lvℓ1/g≥L​\(∏ℓvℓ1/g\)1/L=L​\(∏i=1dσi2\)1/d,\\sum\_\{\\ell=1\}^\{L\}v\_\{\\ell\}^\{1/g\}\\geq L\\,\\Bigl\(\\prod\_\{\\ell\}v\_\{\\ell\}^\{1/g\}\\Bigr\)^\{1/L\}=L\\,\\Bigl\(\\prod\_\{i=1\}^\{d\}\\sigma\_\{i\}^\{2\}\\Bigr\)^\{1/d\},\(42\)with equality iff allvℓv\_\{\\ell\}coincide, so the fixed\-rate distortion is at leastCg​L​2−2​b​\(∏iσi2\)1/d=D∗C\_\{g\}\\,L\\,2^\{\-2b\}\(\\prod\_\{i\}\\sigma\_\{i\}^\{2\}\)^\{1/d\}=D^\{\*\}\. The equal\-rate distortion attains the unconstrained optimum on volume\-equalizing partitions\.■\\blacksquare

### 13\.2Low\-Rate Version of the Grouping Bound

The remark in Sec\.[4](https://arxiv.org/html/2608.04074#S4)removes the high\-resolution assumption for independent Gaussian groups\. Let𝐫Gℓ∈ℝ1×g\\mathbf\{r\}\_\{G\_\{\\ell\}\}\\in\\mathbb\{R\}^\{1\\times g\}have independent Gaussian entries with variances\{σi2\}i∈Gℓ\\\{\\sigma\_\{i\}^\{2\}\\\}\_\{i\\in G\_\{\\ell\}\}, and let𝒬ℓ\\mathcal\{Q\}\_\{\\ell\}be*any*quantizer withg​bℓgb\_\{\\ell\}bits\. The rate of a fixed\-rate quantizer upper\-bounds the mutual information between source and reconstruction, so the distortion is bounded below by the distortion\-rate function of the source\. For an independent Gaussian vector, reverse water\-filling\(berger1971rate\)gives, when the entry\-level distortionD/gD/gis belowmini∈Gℓ⁡σi2\\min\_\{i\\in G\_\{\\ell\}\}\\sigma\_\{i\}^\{2\},

R​\(D\)=∑i∈Gℓ1/2​log2⁡g​σi2D=g2​log2⁡g​vℓ1/gD\.R\(D\)=\\sum\_\{i\\in G\_\{\\ell\}\}1/2\\,\\log\_\{2\}\\frac\{g\\,\\sigma\_\{i\}^\{2\}\}\{D\}=\\frac\{g\}\{2\}\\log\_\{2\}\\frac\{g\\,v\_\{\\ell\}^\{1/g\}\}\{D\}\.\(43\)SettingR​\(D\)≤g​bℓR\(D\)\\leq gb\_\{\\ell\}and solving forDD,

𝔼​‖𝐫Gℓ−𝒬ℓ​\(𝐫Gℓ\)‖22≥g​vℓ1/g​2−2​bℓ\.\\mathbb\{E\}\\bigl\\\|\\mathbf\{r\}\_\{G\_\{\\ell\}\}\-\\mathcal\{Q\}\_\{\\ell\}\(\\mathbf\{r\}\_\{G\_\{\\ell\}\}\)\\bigr\\\|\_\{2\}^\{2\}\\geq g\\,v\_\{\\ell\}^\{1/g\}\\,2^\{\-2b\_\{\\ell\}\}\.\(44\)The dependence on the partition is only throughvℓ1/gv\_\{\\ell\}^\{1/g\}, so the fixed\-rate argument of Sec\.[13\.1](https://arxiv.org/html/2608.04074#S13.SS1)applies to this lower bound\.

## 14High\-Resolution Model: Empirical Test

High\-resolution quantization theory is often used as a design principle for compressors deployed at finite rates\(sullivan1998rate\)\. The accuracy of the approximation depends on whether the quantization cells are small relative to variations in the source density and distortion measure, which can hold even at modest nominal rates\. We emphasize, in any case, that our results are asymptotic rather than finite\-rate guarantees\. We test, at and around the deployed rate, the specific predictions that determine our design\.

We focus on Theorem 2, which rests on the high\-resolution distortion model \([7](https://arxiv.org/html/2608.04074#S4.E7)\),

Dℓ​\(b\)=Cg​2−2​b​vℓ​\(π\)1/g,D\_\{\\ell\}\(b\)=C\_\{g\}\\,2^\{\-2b\}\\,v\_\{\\ell\}\(\\pi\)^\{1/g\},\(45\)while the method operates atb=2b=2, where the asymptotics are not guaranteed to hold\. For a groupGℓG\_\{\\ell\}ofggtransform entries, a partitionπ\\pi, and a ratebb, we define the empirical distortion as

D^ℓ​\(b\)=𝔼test​\[‖𝐫Gℓ−𝒬ℓ​\(𝐫Gℓ\)‖22\]\.\\widehat\{D\}\_\{\\ell\}\(b\)=\\mathbb\{E\}\_\{\\mathrm\{test\}\}\\left\[\\left\\\|\\mathbf\{r\}\_\{G\_\{\\ell\}\}\-\\mathcal\{Q\}\_\{\\ell\}\(\\mathbf\{r\}\_\{G\_\{\\ell\}\}\)\\right\\\|\_\{2\}^\{2\}\\right\]\.\(46\)We estimate this quantity using a2g​b2^\{gb\}\-entrykk\-means codebook trained on training\-split coefficients and evaluated on held\-out coefficients\. We varybbunless otherwise stated and setg=4g=4\(matching the deployed method\)\. Here𝔼test\\mathbb\{E\}\_\{\\mathrm\{test\}\}denotes the empirical mean over held\-out coefficients, andD^ℓ,train\\widehat\{D\}\_\{\\ell,\\mathrm\{train\}\}denotes the same estimate on the training split\.

We evaluate \([45](https://arxiv.org/html/2608.04074#S14.E45)\) using parameters estimated from the training data\. The group volume is

vℓ​\(π\)=∏i∈Gℓσi2,v\_\{\\ell\}\(\\pi\)=\\prod\_\{i\\in G\_\{\\ell\}\}\\sigma\_\{i\}^\{2\},\(47\)where the coordinate variancesσi2\\sigma\_\{i\}^\{2\}are estimated on the training split\. Letb0=2\.5b\_\{0\}=2\.5\. We estimate the common constant once as

C^g=22​b0​medianℓ\(D^ℓ,train​\(b0\)/vℓ​\(π\)1/g\),\\widehat\{C\}\_\{g\}=2^\{2b\_\{0\}\}\\operatorname\*\{median\}\_\{\\ell\}\\left\(\{\\widehat\{D\}\_\{\\ell,\\mathrm\{train\}\}\(b\_\{0\}\)\}/\{v\_\{\\ell\}\(\\pi\)^\{1/g\}\}\\right\),\(48\)where the median is taken over the groups of the equalizing partition and all tested \(layer, KV head\) pairs\. We holdC^g\\widehat\{C\}\_\{g\}fixed across all held\-out rates, groups, and partitions\. In the empirical comparison, we therefore evaluate

Dℓ​\(b\)=C^g​2−2​b​vℓ​\(π\)1/g\.D\_\{\\ell\}\(b\)=\\widehat\{C\}\_\{g\}\\,2^\{\-2b\}\\,v\_\{\\ell\}\(\\pi\)^\{1/g\}\.\(49\)
We also report the volume\-normalized empirical distortionD^ℓ​\(b\)/vℓ​\(π\)1/g\{\\widehat\{D\}\_\{\\ell\}\(b\)\}/\{v\_\{\\ell\}\(\\pi\)^\{1/g\}\}, which the model predicts to be approximately common across groups at a given rate:

Dℓ​\(b\)/vℓ​\(π\)1/g=C^g​2−2​b\.\{D\_\{\\ell\}\(b\)\}/\{v\_\{\\ell\}\(\\pi\)^\{1/g\}\}=\\widehat\{C\}\_\{g\}\\,2^\{\-2b\}\.\(50\)Here the hat denotes an empirical estimate and is unrelated to the reconstruction notation of Appendix[7](https://arxiv.org/html/2608.04074#S7)\.

We test three predictions\. First, at each rate, the volume\-normalized distortionsD^ℓ​\(b\)/vℓ​\(π\)1/g\{\\widehat\{D\}\_\{\\ell\}\(b\)\}/\{v\_\{\\ell\}\(\\pi\)^\{1/g\}\}should approximately agree across groups\. Second, across rates, the volume\-normalized distortion should follow the scaling2−2​b2^\{\-2b\}\. Third, at the deployed rateb=2b=2, the measured total distortion∑ℓ=1LD^ℓ​\(b\)\\sum\_\{\\ell=1\}^\{L\}\\widehat\{D\}\_\{\\ell\}\(b\)and the theoretical partition criterion∑ℓ=1Lvℓ​\(π\)1/g\\sum\_\{\\ell=1\}^\{L\}v\_\{\\ell\}\(\\pi\)^\{1/g\}should order partitions in the same way\.

### 14\.1Setup

We used Qwen3\-8B, Qwen3\-4B\-Thinking\-2507, and Llama\-3\.1\-8B post\-RoPE keys at three \(layer, KV head\) pairs across the model depth \(layer 1, head 0; layer 17, head 3; layer 35, head 7 for the Qwen3 models, and layer 1, head 0; layer 15, head 3; layer 31, head 7 for the 32\-layer Llama\-3\.1\-8B\), giving3×L=963\\times L=96groups per partition per rate atd=128d=128,g=4g=4\. Coefficients are𝐫=\(𝐤−𝐤¯\)​𝐑K\\mathbf\{r\}=\(\\mathbf\{k\}\-\\bar\{\\mathbf\{k\}\}\)\\mathbf\{R\}\_\{K\}with the deployed transform\. Per\-token RMS normalization was off for this study; it isolates the unnormalized transform coefficients analyzed in Theorem 2\. Throughout,bbdenotes the rate for the key branch in bits per entry\. For Qwen3 we pooled three model captures \(GPQA\-198, MMLU, GPQA\-32\), each 8 sequences of 16384 tokens; examples 0\-5 of each capture formed the training pool, from which we retained218−1=2621432^\{18\}\-1=262143coefficient vectors, and examples 6\-7 formed the held\-out set \(98304 tokens\); the training and held\-out sets contain disjoint sequences from each capture\. For Llama\-3\.1\-8B, we used a single LongBench capture of 80 variable\-length prompts split 0\-23 / 24\-47 \(262144 training, 98304 held\-out vectors\)\. Codebooks were initialized by sampling2g​b2^\{gb\}codewords uniformly without replacement from the training vectors, followed by 25 Lloyd iterations; empty cells were reseeded to the training points worst served by the current codebook\. We consider ratesb∈\{1\.0,1\.5,2\.0,2\.5\}b\\in\\\{1\.0,1\.5,2\.0,2\.5\\\}, i\.e\.2g​b∈\{16,64,256,1024\}2^\{gb\}\\in\\\{16,64,256,1024\\\}codewords\. In testing the partitions, we consider the three methods in Appendix[7](https://arxiv.org/html/2608.04074#S7): equalizing \(deployed\), random, and variance\-sorted\.

### 14\.2Agreement across groups

At each rate, dividing each group’s distortion byvℓ1/gv\_\{\\ell\}^\{1/g\}removes most of the variation across the 96 groups \(Fig\.[8](https://arxiv.org/html/2608.04074#S14.F8)a–b\)\. The volume term accounts for the bulk of the per\-group differences, which is the component of the model that the partition argument of Theorem 2 relies on\.

### 14\.3Rate dependence

We fit the generalized rate law2−α​b2^\{\-\\alpha b\}to the volume\-normalized held\-out distortions atb∈\{1,1\.5,2,2\.5\}b\\in\\\{1,1\.5,2,2\.5\\\}and obtainα=1\.816\\alpha=1\.816for Qwen3\-8B,α=1\.828\\alpha=1\.828for Qwen3\-4B\-Thinking\-2507, andα=1\.878\\alpha=1\.878for Llama\-3\.1\-8B, compared with the high\-resolution valueα=2\\alpha=2\. If we compute the exponent between adjacent rates, which is more relevant for an asymptotic law, we obtain1\.7961\.796,1\.8131\.813, and1\.8401\.840on held\-out data for the intervals1→1\.51\{\\to\}1\.5,1\.5→21\.5\{\\to\}2, and2→2\.52\{\\to\}2\.5on Qwen3\-8B, increasing monotonically over the tested range, a trend consistent with approaching the asymptotic value22\. The fitted constants agree closely across the three models,C^g=1\.518\\widehat\{C\}\_\{g\}=1\.518,1\.5071\.507, and1\.4891\.489respectively, as expected for a quantity that should depend only onggand the source shape\. As we observe in Fig\.[8](https://arxiv.org/html/2608.04074#S14.F8)a,[8](https://arxiv.org/html/2608.04074#S14.F8)d, and[8](https://arxiv.org/html/2608.04074#S14.F8)g, withC^g\\widehat\{C\}\_\{g\}estimated from training data at the top rate \(b=2\.5b=2\.5\), the measured and predicted curves are closest at that rate and separate toward lower rates\.

Because these distortions are measured on held\-out coefficients, the reported deviation includes both finite\-rate effects and any residual suboptimality of the trained vector quantizers; we do not attempt to separate these effects\.

### 14\.4Partition comparison

Atb=2b=2, Table[13](https://arxiv.org/html/2608.04074#S14.T13)shows the measured and predicted per\-group distortion over the 96 groups for each partition\. The predicted criterion ranks the three partitions in the same order as the held\-out distortion, with the deployed equalizing partition performing best\. The model is quantitatively accurate for the equalizing and random partitions, but overestimates the distortion of the variance\-sorted partition by81\.5%81\.5\\%\.

Table 13:Partition comparison atb=2b=2for Qwen3\-8B\. Measured and predicted mean per\-group distortion over 96 groups\. Predictions use \([7](https://arxiv.org/html/2608.04074#S4.E7)\) withC^g=1\.518\\widehat\{C\}\_\{g\}=1\.518\.#### Residual dependence\.

We observe that our model provides worse predictions for the variance\-sorted partition\. Beyond existing finite\-rate approximation errors, we argue that this effect is consistent with a stronger violation of the independent\-Gaussian assumption\. The transform decorrelates the entry at second order but does not make them independent; the sorted partition places adjacent entries together, and those groups exhibit substantially greater dependence as measured by Shannon total correlationTC=∑jhj−hjoint\\mathrm\{TC\}=\\sum\_\{j\}h\_\{j\}\-h\_\{\\text\{joint\}\}\(Table[14](https://arxiv.org/html/2608.04074#S14.T14), Kozachenko–Leonenkokk\-NN joint\-entropy estimates andmm\-spacing marginal estimates on training coefficients\)\. We remark that total correlation indicates a violation of the independence assumption but does not quantify its effect on distortion\. Finite\-sample entropy estimates may yield slightly negative estimated total correlations, explaining why a reported mean can be below the corresponding median\.

Table 14:Total correlation per 4\-coordinate group for Qwen3\-8B on training coefficients\.![Refer to caption](https://arxiv.org/html/2608.04074v1/x6.png)Figure 8:High\-resolution model test, 96 groups pooled over three \(layer, KV head\) pairs of Qwen3\-8B \(a\-c\), Qwen3\-4B\-Thinking\-2507 \(d\-f\), and Llama\-3\.1\-8B \(g\-i\)\. \(a, d\) Median volume\-normalized distortionD^ℓ/vℓ1/g\\widehat\{D\}\_\{\\ell\}/v\_\{\\ell\}^\{1/g\}against rate, with the model curveC^g​2−2​b\\widehat\{C\}\_\{g\}2^\{\-2b\}tied to the median atb=2\.5b=2\.5; the curves are closest at the anchor and separate at low rate, where the local exponent is below 2\. \(b, e\) Per\-group measured distortionD^ℓ\\widehat\{D\}\_\{\\ell\}against the modelDℓD\_\{\\ell\}atb∈\{1\.5,2,2\.5\}b\\in\\\{1\.5,2,2\.5\\\}; the band is±25%\\pm 25\\%around the identity\. \(c, f\) Measured and predicted per\-group distortion for the three partitions atb=2b=2; the ordering is preserved\.

## 15Further evaluations

RULER NIAH measures exact\-match retrieval\. To test whether the comparison transfers beyond that setting, we evaluate on two suites: LongBench \(Sec\.[15\.1](https://arxiv.org/html/2608.04074#S15.SS1)\), a multitask suite of QA, summarization, and code completion, and LooGLE \(Sec\.[15\.2](https://arxiv.org/html/2608.04074#S15.SS2)\), long\-document QA and summarization with longer inputs and a larger item count\.

### 15\.1LongBench

We evaluate on LongBench\(bai2023longbench\)tasks \(Qasper, QMSum, MultiNews, TREC, TriviaQA, SAMSum, LCC, RepoBench\-P\) using the full test sets \(200 examples per task; 500 for LCC and RepoBench\-P\) and greedy decoding \(default for this benchmark\), so runs are deterministic\. Prompt KV entries are quantized during prefill at 2\.28 BPE for OSCAR and 2\.22 BPE for NOVA\-KV, while KV entries generated during decoding remain in BF16; layer 0 is kept in full precision for every arm\. OSCAR and NOVA\-KV differ only on the key quantizer: both share the same SQ value path\. NOVA\-KV transforms and codebooks are calibrated on the same GPQA\-Diamond subset we used in the main paper\.

#### Qwen3\-8B

We let both OSCAR and NOVA\-KV use the sink\-plus\-recent band \(nsink=64n\_\{\\mathrm\{sink\}\}\{=\}64,nrec=256n\_\{\\mathrm\{rec\}\}\{=\}256\)\. We also ablate the effect of this band: the half\-band rows reduce it tonsink=32n\_\{\\mathrm\{sink\}\}\{=\}32,nrec=128n\_\{\\mathrm\{rec\}\}\{=\}128for both methods\. Since decoding is deterministic, we assess separability with a paired bootstrap over items \(10000 resamples, percentile 95% CIs\); CIs on the mean row are computed by resampling items within each task and averaging the eight per\-task means with equal weight, so each task counts 1/8 regardless of test\-set size\. Per\-task intervals are uncorrected for multiple comparisons and are provided as descriptive analyses; the equal\-weight average is the primary comparison\.

Table 15:LongBench scores per task for Qwen3\-8B\. Full band is\(nsink,nrec\)=\(64,256\)\(n\_\{\\mathrm\{sink\}\},n\_\{\\text\{rec\}\}\)=\(64,256\); half band is\(32,128\)\(32,128\)\. BF16 is band\-independent\. Best compressed method per task and band setting inbold\.Table 16:Paired bootstrap comparison, gap between NOVA\-KV and OSCAR, per task and averaged\. Entries areΔ\\Deltawith 95% CI;∗marks CIs excluding zero\.Table 17:Paired bootstrap comparison against the BF16 reference, using equal\-weight average over the eight tasks\.∗marks CIs excluding zero\.LongBench contexts are shorter than the regime where the methods separate on RULER NIAH, so the differences in average score are small\. First, against BF16 \(Table[17](https://arxiv.org/html/2608.04074#S15.T17)\), we detect no significant difference between NOVA\-KV and the uncompressed reference at both band settings, while OSCAR is significantly below it at both\. Second, at the full band, NOVA\-KV’s\+0\.70\+0\.70advantage over OSCAR in average score is not separable from zero \(Table[16](https://arxiv.org/html/2608.04074#S15.T16)\)\. Third, at the half band the average\-score gap becomes separable \(\+3\.05\+3\.05\): halving the BF16 band costs OSCAR2\.02\.0points on average but leaves NOVA\-KV almost unchanged, indicating that NOVA\-KV is less dependent on protecting sink and recent tokens\.

#### GPT\-OSS\-20B\.

To check whether the OSCAR failure on GPT\-OSS \(cf\. Sec\.[16\.2](https://arxiv.org/html/2608.04074#S16.SS2)\) is specific to RULER NIAH’s exact\-match retrieval, we separately evaluate the same eight LongBench tasks on GPT\-OSS\-20B using the same configuration we used for Qwen3\-8B \(in this case, we use the full protection band,nsink=64,nrec=256n\_\{\\mathrm\{sink\}\}=64,n\_\{\\text\{rec\}\}=256\)\. Table[18](https://arxiv.org/html/2608.04074#S15.T18)reports every task\. OSCAR collapses on*every*task \(question answering, summarization, classification, and code completion alike\) for a mean of7\.517\.51against BF16’s36\.7636\.76, an80%80\\%relative drop\. NOVA\-KV stays within7%7\\%of BF16 \(34\.2434\.24\), consistent with the Qwen3\-8B result in Table[15](https://arxiv.org/html/2608.04074#S15.T15)\.

Table 18:LongBench\(bai2023longbench\)scores per task for GPT\-OSS\-20B\. Best compressed method per task inbold\.

### 15\.2LooGLE

We evaluate on LooGLE\(li2023loogle\)with Qwen3\-8B \(11011101items\), scored by ROUGE\-L, under the same serving configuration and band as the LongBench evaluation \(Sec\.[15\.1](https://arxiv.org/html/2608.04074#S15.SS1)\); decoding is greedy, so runs are deterministic\. Separability is assessed with the same paired bootstrap over items \(10000 resamples, percentile 95% CIs\)\.

Table 19:LooGLE ROUGE\-L for Qwen3\-8B overn=1101n\{=\}1101items, with paired bootstrap differences \(95% CIs;∗marks CIs excluding zero\)\.#### Discussion\.

NOVA\-KV scores2\.432\.43ROUGE\-L above OSCAR, separable from zero, and2\.692\.69below the BF16 reference, roughly half of OSCAR’s5\.125\.12\-point degradation\. Unlike on LongBench, where NOVA\-KV was statistically indistinguishable from BF16, the larger item count in this dataset resolves all pairs\. LooGLE complements RULER NIAH by measuring graded long\-document QA and summarization rather than exact\-match retrieval\.

## 16GPT\-OSS\-20B: Quantizing a Hybrid\-Attention Cache

As attention computation and KV\-cache size become the dominant cost of long\-context generation, recent architectures increasingly adopt hybrid designs in which only a fraction of the layers attend globally over the full context, and the remaining layers use a cheaper mechanism\. The mechanism varies across models: sliding\-window attention\(gemmateam2025gemma3;agarwal2025gpt\), gated delta networks\(yang2025gated\), or state\-space layers\(lieber2024jamba\)\. The consequence for KV quantization is the same in each case: only the global layers’ cache grows with context, so the cache that a quantizer sees is smaller, the information it holds is denser, and there is less redundancy, limiting potential compression gains\. GPT\-OSS\-20B\(agarwal2025gpt\)is such a model and differs from Llama\-3\.1\-8B and Qwen3\-8B in every property relevant to KV quantization \(Table[20](https://arxiv.org/html/2608.04074#S16.T20)\): only 12 of its 24 layers attend globally \(the other 12 see a 128\-token window\), heads are half as wide, keys are not bounded by QK\-norm, every head carries a learned sink logit that competes with tokens in the softmax, and the feed\-forward blocks route each token to 4 of 32 experts\. Its growing per\-token cache is55–6×6\\timessmaller than the dense models’\. We quantize only the 12 global\-attention layers; the sliding\-window layers stay BF16, since their cache is bounded by the window and does not grow with context\. In this setting, NOVA\-KV is the only 2\-bit method \(among the methods we tested\) that remains effective \(Table[2](https://arxiv.org/html/2608.04074#S5.T2)\)\. This section describes the serving changes the architecture requires, analyzes why the scalar baselines fail, and measures the rate at which scalar quantization recovers\.

Table 20:Architecture properties relevant to KV quantization\. “KV bytes/token” counts the BF16 cache that grows with context \(global\-attention layers\)\.### 16\.1Attention sinks in the quantized serving path

The learned sink enters the softmax as an extra logit per head that contributes to the normalizer but has no value vector\. Three changes are required in the quantized read path\.

#### Split\-KV normalization\.

In the two\-stage decode kernel \(Sec\.[8\.5](https://arxiv.org/html/2608.04074#S8.SS5)\), each KV split reduces independently\. The sink termexp⁡\(sh−m\)\\exp\(s\_\{h\}\-m\)is added to the normalizer exactly once, after the cross\-split combine in stage 2, not per split\.

#### Mean centering and offset invariance\.

NOVA\-KV stores centered keys and drops the constant𝐪⊤​𝐤¯\\mathbf\{q\}^\{\\top\}\\bar\{\\mathbf\{k\}\}from the logits, which is harmless under a softmax \(Lemma S1, Sec\.[12\.1](https://arxiv.org/html/2608.04074#S12.SS1)\)\. With a sink the invariance fails: token logits carry the offset but the learned sink logit does not, so the sink’s softmax share would be rescaled byexp⁡\(𝐪⊤​𝐤¯\)\\exp\(\\mathbf\{q\}^\{\\top\}\\bar\{\\mathbf\{k\}\}\)\. The kernel therefore shifts the sink logit by the same per\-query constant \(onedd\-dimensional dot product per KV head per step\), which restores exact equivalence\.

#### Chunked prefill\.

New chunks are scored against the dequantized prefix by a flash\-attention kernel\(dao2022flashattention\)with no sink support; its output and log\-sum\-exp are rescaled afterwards to fold the sink into the normalizer\.

### 16\.2On the OSCAR and QuaRot failures

On GPT\-OSS\-20B, OSCAR collapses on retrieval \(Table[2](https://arxiv.org/html/2608.04074#S5.T2)\) and QuaRot collapses on every task, while NOVA\-KV stays within a few points of BF16 at the same rate\. We test whether this is an integration artifact or a property of the model\.

#### Serving stack\.

We run the model under plain HuggingFace and overwrite the cache with the same codec the pool applies\. On a 10\-needle greedy retrieval probe at 1K context, GPT\-OSS drops from 4 needles \(its BF16 control on this harness\) to 1 with keys and values quantized, and to 0 with only keys quantized; Llama\-3\.1\-8B retrieves 10 of 10 both in BF16 and with OSCAR under the identical procedure\. Since the value\-side OSCAR codec is shared with NOVA\-KV, which is unaffected, the failure is specific to SQ of the keys\.

#### Reconstruction fidelity\.

We measure the OSCAR codec on real captured keys and queries in four bases \(Table[21](https://arxiv.org/html/2608.04074#S16.T21)\)\. GPT\-OSS keys*reconstruct better*than Llama’s in the served bases \(9\.19\.1vs\.8\.28\.2dB in the Hadamard basis,8\.88\.8vs\.8\.18\.1dB in the calibrated one\), and the rows are not outlier\-dominated there: the mean excess kurtosis of a key row is−0\.3\-0\.3on GPT\-OSS and−0\.1\-0\.1on Llama in the calibrated basis \(versus20\.820\.8and7\.27\.2in the identity basis, where both models are heavy\-tailed and both SNRs collapse\)\. The same key error nevertheless produces3\.53\.5–3\.9×3\.9\\timesthe relative attention\-logit error on GPT\-OSS, and the ratio persists across all four bases\. The architecture concentrates all long\-range information in 12 narrow\-head layers, so each cached element carries more task information and the logits tolerate less noise: one scale per \(token, head\) at 2 bits sits above GPT\-OSS’s noise tolerance and below Llama’s\. The MoE feed\-forward adds an amplification path: the perturbed attention output enters the router, whose top\-kkselection is a discrete function of the hidden state\. In a teacher\-forced probe \(24 decode steps at 1K context, production codec\), KV quantization changed the top\-4 expert set at70%70\\%of \(token, layer\) router decisions, an error mode a dense feed\-forward does not exhibit\.

Table 21:OSCAR key quantization on GPT\-OSS\-20B \(G\) and Llama\-3\.1\-8B \(L\): reconstruction SNR of the keys and relative attention\-logit error‖𝐪⊤​\(𝐊−𝐊^\)‖/‖𝐪⊤​𝐊‖\\\|\\mathbf\{q\}^\{\\top\}\(\\mathbf\{K\}\-\\hat\{\\mathbf\{K\}\}\)\\\|/\\\|\\mathbf\{q\}^\{\\top\}\\mathbf\{K\}\\\|, per basis\. Post\-RoPE keys and queries captured from GPQA prompts \(4 prompts for the SNR statistics, 8 for the logit error\), averaged over all quantized layers and KV heads; per\-\(token, head\) scale with the production percentile clip \(0\.960\.96\)\. “Calibrated” is each model’s served rotation\.
#### QuaRot\.

The QuaRot rows in Table[2](https://arxiv.org/html/2608.04074#S5.T2)already include the protective BF16 sink\-plus\-recent band and still sit at0\.00\.0\. Without the band, i\.e\. unclipped min–max QuaRot\-INT2 on every token from position 0, GPT\-OSS produces degenerate output at every prompt length we probed, from9090to80008000tokens \(greedy completions; served and reproduced in the HuggingFace simulation\), while Llama only degrades under the identical procedure\. On this model the unclipped scalar codec has no usable operating point, with or without the band\.

### 16\.3Scalar quantization recovers at higher rates

If information density is the correct explanation, a modest amount of additional rate should restore the scalar codec\. We simulate the OSCAR codec bit\-exactly on the write path of a BF16 server \(values are dequantized before storage; no memory is saved, and every token is quantized\) and vary the rate \(Table[22](https://arxiv.org/html/2608.04074#S16.T22)\)\. One additional bit takes the codec from collapse to functional, and 4 bits matches BF16 on retrieval\. Finer scale granularity at 2 bits does not substitute, despite a higher rate than INT3: the deficit is resolution per coordinate, not outlier range\. NOVA\-KV reaches the required fidelity at2\.222\.22BPE through vector quantization \(NIAH\-8K89\.689\.6, Table[2](https://arxiv.org/html/2608.04074#S5.T2)\), consistent with the main\-text result that, among the techniques we tested, it is the only 2\-bit method that remains effective on this model\.

Table 22:Scalar \(OSCAR\-codec\) rate ladder on GPT\-OSS\-20B, simulated bit\-exactly in serving\. “Bits” is the payload width of every stored key and value coordinate; “groups” is the number of min–max scale/zero\-point pairs per \(token, head\) row ofd=64d\{=\}64coordinates \(1 is the served configuration\)\. Single greedy rollout; NIAH\-8K on the 200\-prompt balanced subset \(25 per subtask\), MATH\-500 on 100 prompts\. BPE charges BF16 scale and zero\-point per group \(b\+0\.5​gb\+0\.5g\)\.BitsGroupsBPENIAH\-8KMATH\-500212\.50\.84244\.027\.015313\.582\.873414\.598\.385BF161699\.092

## 17A Chunked\-Prefill Dequantization Leak

Long prompts are prefilled in chunks rather than in one pass, bounding peak prefill memory and letting the scheduler interleave a long prompt’s prefill with other requests’ decoding\. In a mixed\-precision KV pool, each chunk’s keys and values are quantized as they are written, so a later chunk of the same prompt attends over earlier chunks’ dequantized rows instead of their original values – a leak that single\-pass prefill does not have\.

The leak can be closed by keeping an exact copy of a request’s own in\-flight rows until its final chunk completes, after which cached content matches the quantized\-on\-write baseline exactly\. This is not free: while a request’s own prefill is in progress, its in\-flight rows occupy close to BF16\-sized cache rather than the compressed footprint, trading part of the memory saving for correctness during that window\.

Table[23](https://arxiv.org/html/2608.04074#S17.T23)reports RULER NIAH with the leak present and closed\. It is real and grows with context \(Llama OSCAR:\+4\.5\+4\.5at88K,\+16\.6\+16\.6at128128K\), and closing it recovers most of the gap for NOVA\-KV and for Llama’s OSCAR\. GPT\-OSS’s OSCAR does not recover:22–1111across all five lengths versus a floor of0with the leak present\. The leak is real but does not explain the collapse \(Sec\.[16\.2](https://arxiv.org/html/2608.04074#S16.SS2)\)\.

Table 23:RULER NIAH accuracy with the chunked\-prefill leak present→\\toclosed\. One seed,2525prompts per NIAH subtask \(200200total\)\. OSCAR and NOVA\-KV on GPT\-OSS\-20B, and, as a control, Llama\-3\.1\-8B\.
## References