Ablation, Statistical Inference, and Validation for KV-Cache Compression

# Ablation, Statistical Inference, and Validation for KV-Cache Compression
Source: [https://arxiv.org/html/2607.09683](https://arxiv.org/html/2607.09683)
Ashish SirasaoElliott DelayeRajeev Patwari Advanced Micro Devices, Inc\. \{paolo\.dalberto, ashish\.sirasao, elliott\.delaye, rajeev\.patwari\}@amd\.com

###### Abstract

We present a systematic comparative study of two families of KV\-cache compression schemes: TurboQuant \(TQ\), which applies a randomized Walsh\-Hadamard rotation and a data\-oblivious Beta Lloyd\-Max codebook, and SpectralQuant \(SQ\), which calibrates a per\-head eigenbasis and allocates bits via water\-filling\. Both families optionally append a 1\-bit Johnson\-Lindenstrauss \(QJL\) residual sketch on the key path, the value path, or both\.

We make three contributions\. First, we perform a full ablation across multiple QJL variants and embedding dimensions, and show that only three schemes are non\-dominated: scalar quantization without rotation, WHT rotation with Beta Lloyd\-Max codebook, and the latter augmented with QJL on keys\.

Second, we introduce a statistical validation methodology for comparing implementations: Python \(oracle\) and HIP/GPU \(production\) use different random number generators and matrix operations as explicit experimental variables\. We apply the Kolmogorov\-Smirnov test to separate systematic codec differences from implementation\-induced variance, and identify the K\-path as a direct signature of Jensen’s inequality amplifying score variance through the softmax nonlinearity\.

Third, we compare the final schemes across all regimes and dimensions and derive regime\-specific recommendations\. Heavy\-tailed data is catastrophic for any eigenbasis\-based method: sample covariances are destabilised by outliers, the calibrated basis is systematically misaligned, and no budget increase recovers the loss\. On structured regimes, SQ wins when separate K and V eigenbases provide genuine compression; water\-filling reduces to uniform allocation throughout\. We also characterize the self\-calibrating nature of the effective semantic dimensiondeffd\_\{\\mathrm\{eff\}\}, which adapts to the available calibration budget rather than recovering the true data rank — a property that explains both surprising wins and non\-monotone scaling behaviors\.

## 1Introduction

Transformer inference at scale is bounded by memory bandwidth: KV\-cache access dominates total memory traffic for long\-context generation, and reducing cache size directly translates to latency and throughput improvements\. Quantization of keys and values is the standard approach, and a growing body of work shows that aggressive quantization to 2–4 bits per element is possible without meaningful accuracy loss\[[5](https://arxiv.org/html/2607.09683#bib.bib5),[4](https://arxiv.org/html/2607.09683#bib.bib4),[3](https://arxiv.org/html/2607.09683#bib.bib3)\]\.

A recurring challenge in evaluating KV\-cache compression is the difficulty of attributing observed quality differences to specific algorithmic choices\. Evaluations on real large language model \(LLM\) traffic conflate distributional properties, hardware effects, and algorithmic assumptions, making it hard to understand*when*and*why*a method succeeds or fails\. This paper introduces a methodology for controlled evaluation of KV\-cache quantization schemes: a set of six synthetic statistical regimes, each designed to isolate one structural assumption of the compression pipeline, together with a statistical framework for distinguishing systematic algorithmic differences from implementation noise\. The full evaluation is released as an open\-source HIP/C\+\+ benchmark targeting AMD GPUs; any new compression scheme can be evaluated against the same regimes by implementing a single scheme interface, without modifying the data generation, metrics, or statistical validation infrastructure\.

We instantiate the methodology on two representative families\.TurboQuant \(TQ\)\[[5](https://arxiv.org/html/2607.09683#bib.bib5)\]is data\-oblivious: a randomized Walsh\-Hadamard transform spreads each token’s energy uniformly across alldddimensions, making an analytical Beta\-distribution codebook applicable uniformly, and an optional 1\-bit QJL residual sketch corrects the dominant direction of the quantization error\.SpectralQuant \(SQ\)\[[2](https://arxiv.org/html/2607.09683#bib.bib2)\]is data\-adaptive: it calibrates a per\-head eigenbasis from representative tokens, concentrates bits on high\-variance semantic dimensions via water\-filling, and corrects residuals with a selective QJL sketch applied only to the top\-deffd\_\{\\mathrm\{eff\}\}dimensions\. These two families represent opposite ends of the data\-dependence spectrum and together exercise the full range of assumptions our regimes are designed to test\.

The synthetic regimes are not held\-out test sets\. Their purpose is diagnostic: to reveal the conditions under which a compression scheme achieves its full potential or fails catastrophically, and to explain the mechanism behind each outcome\. Each regime targets one assumption in isolation — distribution shape, K/V subspace alignment, or eigenvalue decay — so that failures can be attributed and understood rather than merely observed\.

The main findings are: \(1\) a full ablation over eight QJL variants identifies three non\-dominated schemes, eliminating the rest; \(2\) the statistical validation framework reveals that K\-path QJL variance is exponentially amplified by softmax \(Jensen’s inequality\), while V\-path variance is not — a distinction invisible to accuracy\-only evaluations; \(3\) TQ dominates on heavy\-tailed data where eigenbasis calibration fails; \(4\) SQ wins on structured regimes at sufficient budget, provided K and V are calibrated on separate representative sets; and \(5\) water\-filling reduces to uniform allocation in all tested regimes, and the effective semantic dimensiondeffd\_\{\\mathrm\{eff\}\}self\-calibrates to the available calibration budget rather than recovering the true data rank\.

## 2Background

Standard multi\-head attention computes, for each head,

T=softmax​\(𝐐𝐊⊤d\)​𝐕,T=\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}\\mathbf\{K\}^\{\\top\}\}\{\\sqrt\{d\}\}\\right\)\\mathbf\{V\},\(1\)where𝐊,𝐕∈ℝS×d\\mathbf\{K\},\\mathbf\{V\}\\in\\mathbb\{R\}^\{S\\times d\}are the key and value caches forSSpast tokens and𝐐∈ℝNq×d\\mathbf\{Q\}\\in\\mathbb\{R\}^\{N\_\{q\}\\times d\}is the query batch\.

TurboQuant\.Given per\-head random signss∈\{−1,\+1\}ds\\in\\\{\-1,\+1\\\}^\{d\}, TQ rotates each key asy=WHT​\(k⊙s\)/dy=\\mathrm\{WHT\}\(k\\odot s\)/\\sqrt\{d\}, makingyyapproximately Beta\(\(d−1\)/2,\(d−1\)/2\)\(\(d\{\-\}1\)/2,\(d\{\-\}1\)/2\)distributed for anykkon the unit sphere\. A single analytical Lloyd\-Max codebook is fitted to this distribution and shared across all dimensions\. Optionally, the WHT\-domain residuale=y−y^e=y\-\\hat\{y\}is sketched using a randomized Hadamard transform \(RHT\):b=sign​\(WHT​\(e⊙s2\)\)b=\\mathrm\{sign\}\(\\mathrm\{WHT\}\(e\\odot s\_\{2\}\)\)for a fixed sign vectors2s\_\{2\}, and the correctionΔ​y^=π/2d​‖e‖​s2⊙WHT​\(b\)\\Delta\\hat\{y\}=\\frac\{\\sqrt\{\\pi/2\}\}\{d\}\\\|e\\\|\\,s\_\{2\}\\odot\\mathrm\{WHT\}\(b\)is added at decode\.QJL on Kimproves the attention score estimate⟨q,k⟩\\langle q,k\\rangle;QJL on Vimproves the value reconstructionV^\\hat\{V\}\. The inner product⟨q,k⟩\\langle q,k\\rangleis a sum over the embedding dimensiondd, so the QJL sketch ofk∈ℝdk\\in\\mathbb\{R\}^\{d\}is theoretically grounded \(Zandieh et al\. 2024\[[6](https://arxiv.org/html/2607.09683#bib.bib6)\]\)\. The attention outputTj=∑iai​Vi​jT\_\{j\}=\\sum\_\{i\}a\_\{i\}V\_\{ij\}sums over tokensSS, so a QJL sketch ofVi∈ℝdV\_\{i\}\\in\\mathbb\{R\}^\{d\}does not target the relevant inner product and is a heuristic correction\.

SpectralQuant\.The core idea of SpectralQuant is a reallocation of bits from many low\-variance dimensions to a few high\-variance ones\. Consider a head withd=128d=128dimensions and a budget ofb=2b=2bits: a uniform scheme allocates2×128=2562\\times 128=256bits per token, spreading the budget thinly across all dimensions regardless of their importance\. If the key vectors lie near a rank\-4 subspace, only 4 directions carry meaningful signal\. SQ identifies these directions from calibration data and allocates, say,88bits to each of the 4 semantic dimensions \(32 bits total\), then quantizes the remaining 124 tail dimensions at 2 bits each \(248 bits\), for a total of only280280bits — comparable storage to the uniform scheme, but with the critical dimensions quantized4×4\\timesmore finely\. Equivalently, SQ can achieve the accuracy of a high\-budget uniform scheme at a fraction of the storage cost\.

Given calibration keysKcal∈ℝncal×dK\_\{\\mathrm\{cal\}\}\\in\\mathbb\{R\}^\{n\_\{\\mathrm\{cal\}\}\\times d\}, SQ computes the empirical covarianceCK=Kcal⊤​Kcal/ncalC\_\{K\}=K\_\{\\mathrm\{cal\}\}^\{\\top\}K\_\{\\mathrm\{cal\}\}/n\_\{\\mathrm\{cal\}\}, extracts the top\-deffd\_\{\\mathrm\{eff\}\}eigenvectors𝐔K∈ℝd×deff\\mathbf\{U\}\_\{K\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathrm\{eff\}\}\}via block power iteration, and projects each key aszk=𝐔K⊤​kz\_\{k\}=\\mathbf\{U\}\_\{K\}^\{\\top\}k\. A separate eigenbasis𝐔V\\mathbf\{U\}\_\{V\}is calibrated from value tokens\. The effective dimension is the participation ratiodeff=round​\(tr​\(C\)2/‖C‖F2\)d\_\{\\mathrm\{eff\}\}=\\mathrm\{round\}\(\\mathrm\{tr\}\(C\)^\{2\}/\\\|C\\\|\_\{F\}^\{2\}\)\. Water\-filling allocates bits by minimizing∑kλk⋅4−bk\\sum\_\{k\}\\lambda\_\{k\}\\cdot 4^\{\-b\_\{k\}\}subject to∑kbk=B\\sum\_\{k\}b\_\{k\}=B,bk≥btailb\_\{k\}\\geq b\_\{\\mathrm\{tail\}\}\. Tail dimensions are quantized uniformly atbtailb\_\{\\mathrm\{tail\}\}bits\.QJL on Kis applied as an asymmetric score estimator inZZspace:score​\(q,k\)=⟨qZ,zk⟩\+π/2m​‖ez‖​qZ⊤​S​sign​\(zk−z^k\)⊤​S\\mathrm\{score\}\(q,k\)=\\langle q\_\{Z\},z\_\{k\}\\rangle\+\\frac\{\\sqrt\{\\pi/2\}\}\{m\}\\\|e\_\{z\}\\\|\\,q\_\{Z\}^\{\\top\}S\\,\\mathrm\{sign\}\(z\_\{k\}\-\\hat\{z\}\_\{k\}\)^\{\\top\}S, whereS∈\{−1,\+1\}m×deffS\\in\\\{\-1,\+1\\\}^\{m\\times d\_\{\\mathrm\{eff\}\}\}andqZ=𝐔K⊤​qq\_\{Z\}=\\mathbf\{U\}\_\{K\}^\{\\top\}q\.

Error metric\.LetTTandT^\\hat\{T\}denote the reference and reconstructed attention outputs\. The relative mean\-squared error isrelMSET=‖T−T^‖F2/‖T‖F2\\mathrm\{relMSE\}\_\{T\}=\\\|T\-\\hat\{T\}\\\|\_\{F\}^\{2\}/\\\|T\\\|\_\{F\}^\{2\}\. We use the budget\-independent 2D error metric

d2=\(1−cosine​\(T,T^\)\)2\+NF2,NF=relMSET1\+relMSET,d\_\{2\}=\\sqrt\{\(1\-\\mathrm\{cosine\}\(T,\\hat\{T\}\)\)^\{2\}\+\\mathrm\{NF\}^\{2\}\},\\quad\\mathrm\{NF\}=\\frac\{\\mathrm\{relMSE\}\_\{T\}\}\{1\+\\mathrm\{relMSE\}\_\{T\}\},\(2\)which compresses unbounded relMSE into\[0,1\)\[0,1\)and combines it with the direction error1−cosine​\(T,T^\)1\-\\mathrm\{cosine\}\(T,\\hat\{T\}\)\.d2∈\[0,2\]d\_\{2\}\\in\[0,\\sqrt\{2\}\]; lower is better\. We also report isolation metrics:relMSEK\\mathrm\{relMSE\}\_\{K\}is the relMSE ofTTwhen onlyKKis quantized \(V exact\), andrelMSEV\\mathrm\{relMSE\}\_\{V\}when onlyVVis quantized \(K exact\), isolating each cache’s contribution to output error\. Per\-token cache metricsptcosine​\_​K=cosine​\(K,K^\)\\mathrm\{ptcosine\\\_K\}=\\mathrm\{cosine\}\(K,\\hat\{K\}\)andptcosine​\_​V=cosine​\(V,V^\)\\mathrm\{ptcosine\\\_V\}=\\mathrm\{cosine\}\(V,\\hat\{V\}\)measure per\-token reconstruction quality independently of Q and are used for codec validation\.

Together, the six scalar metrics —snr​\_​errK\\mathrm\{snr\\\_err\}\_\{K\},dir​\_​errK\\mathrm\{dir\\\_err\}\_\{K\},snr​\_​errV\\mathrm\{snr\\\_err\}\_\{V\},dir​\_​errV\\mathrm\{dir\\\_err\}\_\{V\},snr​\_​errT\\mathrm\{snr\\\_err\}\_\{T\},dir​\_​errT\\mathrm\{dir\\\_err\}\_\{T\}— define a point in a six\-dimensional error space\. We represent each scheme\-regime\-budget configuration as such a point and use 2D projections \(scatter plots\) together with the Kolmogorov\-Smirnov test and energy distance to enrich the analysis with geometric intuition and statistical inference\. This geometric framework was introduced in an earlier version of this work\[[1](https://arxiv.org/html/2607.09683#bib.bib1)\]; the present paper corrects a bug in the quantizer binary search \(off\-by\-one inquantise\_scalar\) that affected all results at budgetb≥3b\\geq 3in that version, and extends the analysis with the full TQ versus SQ comparison and the SQ calibration sensitivity study\.

## 3Experimental Setup

All experiments usencal=512n\_\{\\mathrm\{cal\}\}=512calibration tokens, 200 trials per cell, budgetsb∈\{2,…,7\}b\\in\\\{2,\\ldots,7\\\}, sequence lengthsS∈\{64,…,4096\}S\\in\\\{64,\\ldots,4096\\\}, query batch sizesNq∈\{1,…,512\}N\_\{q\}\\in\\\{1,\\ldots,512\\\}, and embedding dimensionsd∈\{64,128,256\}d\\in\\\{64,128,256\\\}\.

### 3\.1Synthetic Regimes

We design six regimes that test each SQ assumption in isolation\.

Table 1:Six synthetic regimes and the SQ assumptions they target\. A1: sub\-Gaussian calibration\. A2: shared K/V subspace \(now relaxed: separate𝐔K\\mathbf\{U\}\_\{K\},𝐔V\\mathbf\{U\}\_\{V\}\)\. A3: eigenvalue decay sufficient for water\-filling\.
### 3\.2Schemes

We evaluate 16 schemes in the full ablation \(Section[4](https://arxiv.org/html/2607.09683#S4)\) and reduce to 6 final contenders thereafter:

All QJL\-K schemes useb−1b\-1bits for mean squared error \(MSE\) quantization and 1 bit for the sketch \(budget\-neutral\)\. For SQU\-QKV,m=64m=64projections ondeff≈8d\_\{\\mathrm\{eff\}\}\\approx 8semantic dimensions\.

#### Storage complexity\.

For a KV cache ofSStokens with head dimensiondd, the per\-cache storage is:

TQ\-KV:b×d×S​bits\\displaystyle\\quad b\\times d\\times S\\text\{ bits\}SQU\-KV:\(bsem×deff\+btail×\(d−deff\)\)×S​bits\\displaystyle\\quad\(b\_\{\\mathrm\{sem\}\}\\times d\_\{\\mathrm\{eff\}\}\+b\_\{\\mathrm\{tail\}\}\\times\(d\-d\_\{\\mathrm\{eff\}\}\)\)\\times S\\text\{ bits\}Withd=128d=128,deff=8d\_\{\\mathrm\{eff\}\}=8,btail=2b\_\{\\mathrm\{tail\}\}=2and nominal budgetbsem=bb\_\{\\mathrm\{sem\}\}=b: TQ uses128​b128bbits/token; SQU uses8​b\+2408b\+240bits/token\. Atb=2b=2both use 256 bits/token \(equal storage\)\. Atb=3b=3, TQ uses 384 bits/token while SQU uses only 264 — a 31% saving\. This storage gap grows withbb, which is why iso\-storage comparisons atb≥3b\\geq 3always favour SQ even when TQ wins on quality\.

### 3\.3Implementation and Validation

We implement all schemes in HIP/C\+\+ \(AMD MI100,hipcc \-O3\) and in Python \(oracle reference\)\. Python and HIP use different random seeds \(PCG64 versus mt19937\), making the seed an explicit experimental variable for the statistical validation in Section[5](https://arxiv.org/html/2607.09683#S5)\.

## 4Results I: TQ Ablation

We evaluate all 8 TQ variants \(Plain\-KV, TQ\-KV, TQ\-QKV, TQ\-FKV, TQ\-KQV, TQ\-QKQV, TQ\-KFV, TQ\-FKFV\) across 6 regimes andd∈\{64,128,256\}d\\in\\\{64,128,256\\\}\. Figure[1](https://arxiv.org/html/2607.09683#S4.F1)showsd2d\_\{2\}as a function ofddat three representative budgets\.

![Refer to caption](https://arxiv.org/html/2607.09683v1/d2_vs_d.png)Figure 1:d2d\_\{2\}vs\. embedding dimension for all 8 TQ variants \(d∈\{64,128,256\}d\\in\\\{64,128,256\\\}, 200 trials, budgets 2, 4, 6\)\. TQ\-KV and TQ\-QKV dominate across all regimes\. FULL variants \(orange/yellow\) are consistently worse than their RHT counterparts\. V\-path QJL variants \(KQV, FKFV\) consistently hurt on every regime\. Plain\-KV degrades withddon lowrank\_aligned \(Beta codebook mismatch grows withDD\)\.Three findings eliminate five of eight schemes\.

#### FULL projection variants are dominated\.

TQ\-FKV and TQ\-FKFV apply ad×dd\\times drandom Rademacher projection\. Withm=Dm=Dprojections onDD\-dimensional vectors, the correction variance isVar∝‖q‖2​‖k‖2/D\\mathrm\{Var\}\\propto\\\|q\\\|^\{2\}\\\|k\\\|^\{2\}/D— identical to RHT in theory but without the structured low\-variance property of the Walsh\-Hadamard transform\. In practice, FULL\-K is uniformly worse than RHT\-K at every regime and budget \(Figure[1](https://arxiv.org/html/2607.09683#S4.F1)\)\. We retain TQ\-QKV \(RHT\) and discard TQ\-FKV \(FULL\)\.

#### V\-path QJL is a heuristic that consistently hurts\.

The attention output isTj=∑iai​Vi​jT\_\{j\}=\\sum\_\{i\}a\_\{i\}V\_\{ij\}, a sum over tokens\. QJL was designed for inner products over the embedding dimensiondd\(Zandieh et al\.\); applying it to V corrects individualVi∈ℝdV\_\{i\}\\in\\mathbb\{R\}^\{d\}vectors but does not target the relevant inner product, which sums overSS\. Empirically, TQ\-KQV \(QJL on V\) achieves ptcosine\_V=0\.854=0\.854atb=2b=2versus TQ\-KV’s ptcosine\_V=0\.953=0\.953— V\-path QJL in 128\-dimensional WHT space cannot recover the bit it costs\. All V\-path variants \(TQ\-KQV, TQ\-QKQV, TQ\-KFV, TQ\-FKFV\) are eliminated\.

#### Surviving schemes: Plain\-KV, TQ\-KV, TQ\-QKV\.

Plain\-KV applies the Beta codebook without WHT and achieves quality comparable to TQ\-KV on isotropic regimes \(unit\_sphere, lowrank\) where the Beta marginals hold after L2 normalization\. TQ\-KV dominates on fattail \(WHT spreads heavy\-tail outliers\)\. TQ\-QKV adds the RHT\-K correction and gains modestly on structured regimes\.

## 5Results II: Statistical Validation — Python vs\. HIP

We use the Kolmogorov\-Smirnov \(KS\) test \(scipyks\_2samp\) to compare Python and HIP distributions across 72 matched rows \(3 regimes×\\times3 budgets×\\times8NqN\_\{q\}values\)\. Python uses PCG64 RNG; HIP uses mt19937\. Same algorithm, different seeds\.

#### Codec validation\.

For TQ\-KV, ptcosine\_K is∼\\sim\(p=0\.071=0\.071\) between Python and HIP — the codec is correct; the small gap reflects different K/V calibration data, not different algorithms\. For SQU\-KV, ptcosine\_V is∼\\sim\(p=0\.10=0\.10\) after fixing V to use an independent calibration set from the V distribution\.

#### Jensen’s inequality

Table 2:KS significance of Python vs\. HIP difference by metric and scheme \(fattail, 72 rows\)\.∼\\sim: p≥0\.05\\geq 0\.05; \*: p<0\.05<0\.05; \*\*: p<0\.001<0\.001\. K\-path gap \(KL, relMSE\_K\) is larger for TQ\-QKV, confirming Jensen amplification\.![Refer to caption](https://arxiv.org/html/2607.09683v1/jensen_effect.png)Figure 2:Jensen’s effect on fattail \(d=128d=128, 200 trials\)\. Solid: HIP\. Dashed: Python \(different random seed\)\.Left:d2d\_\{2\}vs\. budget\.Right: top\-5 recall vs\. budget\. Gap = seed\-dependent score noise amplified by softmax\. RHT \(cyan\) tighter than FULL \(orange\) — lower correction variance\. TQ\-KV \(green\): no gap, no QJL, no Jensen\.The softmax is convex in its argument:𝔼​\[exp⁡\(s\+ε\)\]=exp⁡\(s\)⋅exp⁡\(σ2/2\)\\mathbb\{E\}\[\\exp\(s\+\\varepsilon\)\]=\\exp\(s\)\\cdot\\exp\(\\sigma^\{2\}/2\)\. QJL adds a zero\-mean correctionε\\varepsilonwith varianceσ2∝‖e‖2/m\\sigma^\{2\}\\propto\\\|e\\\|^\{2\}/m; different random seeds \(Python vs\. HIP\) produce differentε\\varepsilondraws, and softmax amplifies the resulting difference exponentially\. For TQ\-QKV, the Python\-HIP KL gap is0\.119\\mathbf\{0\.119\}\(\*\* significant\) vs\.0\.0000\.000for TQ\-KV \(Table[2](https://arxiv.org/html/2607.09683#S5.T2), Figure[2](https://arxiv.org/html/2607.09683#S5.F2)\)\. For TQ\-KQV \(V\-path QJL\), cosine is∼\\sim— confirming that V\-path corrections, which enterTTlinearly through a weighted sum, do not trigger Jensen amplification\.

#### mmcontrols the SQ gap\.

The variance of the QJL correction is∝1/m\\propto 1/m\. We verify experimentally that the Python\-HIP KL gap scales as1/m1/m: atm=deff=8m=d\_\{\\mathrm\{eff\}\}=8, the gap is5×5\\timeslarger than atm=64m=64\. This motivatesm≥4×deffm\\geq 4\\times d\_\{\\mathrm\{eff\}\}as a practical threshold\.

## 6Results III: Water\-Filling is Always Uniform

Water\-filling is the theoretically motivated bit\-allocation strategy for SpectralQuant, yet it provides negligible benefit over uniform allocation across all six regimes and all budgets tested\.

###### Proposition 1\.

Letσ12≥⋯≥σdeff2\\sigma\_\{1\}^\{2\}\\geq\\cdots\\geq\\sigma\_\{d\_\{\\mathrm\{eff\}\}\}^\{2\}be the per\-dimension variances of the projected calibration data,B=b⋅deffB=b\\cdot d\_\{\\mathrm\{eff\}\}the total semantic bit budget, andbtailb\_\{\\mathrm\{tail\}\}the minimum bits per dimension\. Integer greedy water\-filling returns uniform allocationbk=bb\_\{k\}=bfor allk∈\{1,…,deff\}k\\in\\\{1,\\ldots,d\_\{\\mathrm\{eff\}\}\\\}wheneverσmax2/σmin2≤4\\sigma\_\{\\max\}^\{2\}/\\sigma\_\{\\min\}^\{2\}\\leq 4\.

The conditionσmax2/σmin2\>4\\sigma\_\{\\max\}^\{2\}/\\sigma\_\{\\min\}^\{2\}\>4is required for any two semantic dimensions in the selected eigenbasis to receive different bit allocations\. For four of six regimes \(unit\_sphere, lowrank\_aligned, lowrank, lowrank\_misalign\), the per\-dimension variance ratios within the top\-deffd\_\{\\mathrm\{eff\}\}subspace fall below this threshold and SQW\-KV and SQU\-KV produce exactly identicald2d\_\{2\}at every budget\.

The steepest synthetic regime \(lowrank\_steep\) has an eigenvalue ratio of88within the calibrateddeff=4d\_\{\\mathrm\{eff\}\}=4dimensions, so water\-filling does activate\. The effect ond2d\_\{2\}is however small: the maximum difference across all budgets is\|Δ​d2\|≤0\.030\|\\Delta d\_\{2\}\|\\leq 0\.030, and SQW\-KV is marginally better in three of six budgets\. On fattail, eigenbasis calibration fails for both variants and any difference is within noise\.

![Refer to caption](https://arxiv.org/html/2607.09683v1/waterfill_comparison.png)Figure 3:SQU\-KV \(uniform allocation\) vs\. SQW\-KV \(water\-filling\) acrossd∈\{64,128,256\}d\\in\\\{64,128,256\\\}\.Top row \(lowrank, rank=8, flat spectrum\):the two curves are exactly identical at every budget and dimension — the variance ratioσmax2/σmin2≤4\\sigma\_\{\\max\}^\{2\}/\\sigma\_\{\\min\}^\{2\}\\leq 4so the integer greedy allocator assigns the same number of bits to every semantic dimension\.Bottom row \(lowrank\_steep, rank=4, ratio=8\):water\-filling activates but its effect ond2d\_\{2\}remains small \(max⁡\|Δ​d2\|≤0\.030\\max\|\\Delta d\_\{2\}\|\\leq 0\.030across all budgets and dimensions\)\.We retain SQW\-KV in all figures for completeness, but treat its results as effectively synonymous with SQU\-KV throughout the analysis\.

## 7Results IV: Eigenvalue Sensitivity in SQ Calibration

Before comparing TQ and SQ, we must establish that SQ calibration is correct\. SpectralQuant estimates its eigenbasis from a finite calibration set ofncaln\_\{\\mathrm\{cal\}\}tokens\. The participation ratiodeff=round​\(\(∑kλk\)2/∑kλk2\)d\_\{\\mathrm\{eff\}\}=\\mathrm\{round\}\(\(\\sum\_\{k\}\\lambda\_\{k\}\)^\{2\}/\\sum\_\{k\}\\lambda\_\{k\}^\{2\}\)determines how many semantic dimensions are retained\. We show that a single spurious eigenvalue — one that is200×200\\timessmaller than the last true signal dimension but still accepted by the participation ratio — causes catastrophic failure, and that a simple spectral gap criterion reliably removes it\.

![Refer to caption](https://arxiv.org/html/2607.09683v1/calibration_sensitivity.png)Figure 4:SQU\-KVd2d\_\{2\}vs\. number of tokensSSon lowrank\_aligned,d=128d=128, 200 trials per cell, one line per budgetb∈\{2,…,7\}b\\in\\\{2,\\ldots,7\\\}\.Left \(before fix\):a spurious ninth eigenvalue \(λ8≈0\.005\\lambda\_\{8\}\\approx 0\.005, accepted by the participation ratio\) causesd2\>1d\_\{2\}\>1atS∈\{128,2048,4096\}S\\in\\\{128,2048,4096\\\}— output is anti\-correlated with the reference\.Right \(after fix\):the spectral gap criterion \(λk−1/λk\>10\\lambda\_\{k\-1\}/\\lambda\_\{k\}\>10\) removes the spurious dimension;d2d\_\{2\}is monotone in bothSSandbb\. Both panels share the sameyy\-axis \(\[0,2\]\[0,2\]\) to make the severity of the failure visible\.#### Failure mode\.

Figure[4](https://arxiv.org/html/2607.09683#S7.F4)\(left\) showsd2d\_\{2\}for SQU\-KV on lowrank\_aligned \(d=128d=128, rank=8\) as a function of the number of tokensSS, before applying any correction\. The pattern is non\-monotone:S=256S=256yieldsd2=0\.008d\_\{2\}=0\.008\(excellent\) whileS=128S=128yieldsd2=1\.23d\_\{2\}=1\.23\(anti\-correlated output,d2\>1d\_\{2\}\>1\) andS=2048S=2048yieldsd2=1\.62d\_\{2\}=1\.62\. This cannot be a structural calibration failure, which would degrade monotonically withSS\.

The root cause is a spurious eigenvalue in the sample covariance at certainSSvalues\. Whenλ8≈0\.005\\lambda\_\{8\}\\approx 0\.005\(200×\\timessmaller than the seventh signal eigenvalueλ7≈0\.8\\lambda\_\{7\}\\approx 0\.8\), the participation ratio rounds up todeff=9d\_\{\\mathrm\{eff\}\}=9instead of 8\. The extra dimension carries near\-zero signal energy but receives a full allocation of bits\. Projected onto this spurious direction, test tokens produce outputs that are effectively random with respect to the true signal subspace — henced2\>1d\_\{2\}\>1\.

This failure shares the mechanism of Assumption A1 violation\. Withλ8≈0\.005\\lambda\_\{8\}\\approx 0\.005, the projected calibration values have standard deviation≈0\.005≈0\.07\\approx\\sqrt\{0\.005\}\\approx 0\.07\. With onlyncal=512n\_\{\\mathrm\{cal\}\}=512samples, the empirical distribution of this dimension is severely undersampled — analogous to the heavy\-tailed regime where extreme outliers make the empirical distribution unrepresentative of the test distribution\. The Lloyd\-Max codebook fitted to this noise is essentially arbitrary: it cannot generalize because there is no true signal to learn\. At test time, the quantization error in this spurious dimension is un\-rotated back to the fulldd\-dimensional output space, contaminating allddoutput coordinates with structured noise\. The catastrophic effect ond2d\_\{2\}is therefore not a quantization failure in the usual sense — it is a codebook fitted to a distribution that violates the Gaussian assumption the entire SQ framework depends on\.

#### Spectral gap criterion\.

The signal\-to\-noise boundary is visible as a large ratio between consecutive eigenvalues:λ7/λ8≈160\\lambda\_\{7\}/\\lambda\_\{8\}\\approx 160\. We apply a spectral gap criterion that trims the estimateddeffd\_\{\\mathrm\{eff\}\}wheneverλk−1/λk\>γ\\lambda\_\{k\-1\}/\\lambda\_\{k\}\>\\gammafor thresholdγ=10\\gamma=10\. Additionally, the known rank of each synthetic regime is passed as a hard capdeff≤deffmaxd\_\{\\mathrm\{eff\}\}\\leq d\_\{\\mathrm\{eff\}\}^\{\\max\}, correcting cases where the participation ratio underestimates the true rank \(e\.g\. lowrank\_steep with exponentially decaying eigenvalues gives PR=3=3instead of the true rank44\)\.

Figure[4](https://arxiv.org/html/2607.09683#S7.F4)\(right\) shows the result after applying both corrections\. Thed2d\_\{2\}metric is now monotonically decreasing in bothSSandbb, and reachesd2≤0\.03d\_\{2\}\\leq 0\.03atb=7b=7across allSSvalues\. All SQ results in the remainder of this paper use the corrected calibration — this is the best SQ\.

## 8Results V: TQ vs\. SQ

We compare six compression schemes across six synthetic data regimes and three embedding dimensions \(d∈\{64,128,256\}d\\in\\\{64,128,256\\\}\)\. The schemes and regimes are summarised in Tables[3](https://arxiv.org/html/2607.09683#S8.T3)and[4](https://arxiv.org/html/2607.09683#S8.T4)for reference\.

Table 3:The six compression schemes compared in this section\. All SQ variants use separate eigenbases𝐔K\\mathbf\{U\}\_\{K\},𝐔V\\mathbf\{U\}\_\{V\}calibrated independently onncal=512n\_\{\\mathrm\{cal\}\}=512tokens with the spectral gap criterion from Section[7](https://arxiv.org/html/2607.09683#S7)\.Table 4:The six synthetic data regimes\. All tokens are drawn i\.i\.d\.;dh=dd\_\{h\}=dis the head dimension\.![Refer to caption](https://arxiv.org/html/2607.09683v1/d2_vs_d_final6_v2_full.png)Figure 5:d2d\_\{2\}vs\. embedding dimension \(d∈\{64,128,256\}d\\in\\\{64,128,256\\\}, 200 trials, 200 trials per cell\)\. Rows: six regimes\. Columns: budgetb∈\{2,4,6\}b\\in\\\{2,4,6\\\}\. TQ\-KV \(green, circle\) is flat acrossddin every panel — the WHT rotation and Beta codebook are dimension\-agnostic\. Plain\-KV \(gray, star\) degrades monotonically withddon lowrank\_aligned as the Beta codebook mismatch grows with ambient dimension\. SQU\-KV \(blue, square\) improves withddon lowrank\_aligned \(sharper signal separation at largerdd\) and is stable elsewhere\. SQU\-QKV \(orange, triangle\) is consistently the worst scheme on structured regimes due to Jensen amplification of the QJL correction inZZspace\. All SQ results use the spectral gap criterion from Section[7](https://arxiv.org/html/2607.09683#S7)\.Figure[5](https://arxiv.org/html/2607.09683#S8.F5)showsd2d\_\{2\}as a function ofd∈\{64,128,256\}d\\in\\\{64,128,256\\\}for all six schemes at budgets 2, 4, and 6 — one row per regime, one column per budget\. We discuss each regime in turn\.

#### Fattail: TQ\-KV is the only viable scheme\.

On Student\-t​\(ν=3\)t\(\\nu\{=\}3\)data, heavy tails destabilise the sample covariance: a few extreme calibration tokens dominate the estimated eigenvectors, pointing them toward outlier directions rather than the true signal subspace\. At test time, different outliers appear and the eigenbasis is systematically misaligned\. TQ\-KV’s WHT spreads each token’s energy uniformly acrossdddimensions, making projected marginals approximately Gaussian regardless of the input distribution\. The result is visible across all budgets and alldd: TQ\-KV reachesd2≤0\.01d\_\{2\}\\leq 0\.01atb=6b=6while all SQ variants plateau neard2≈0\.7d\_\{2\}\\approx 0\.7\.

#### Lowrank\_aligned: SQ’s home territory\.

When the data lies in a low\-rank subspace that is stable across calibration and test draws, SQ concentrates all bits on thedeff=8d\_\{\\mathrm\{eff\}\}=8signal dimensions\. SQU\-KV improves monotonically withddbecause the signal subspace becomes more sharply separated from the noise floor at largerdd, giving the eigenbasis more precision\. Plain\-KV degrades withdd: asddgrows, the Beta codebook mismatches the per\-coordinate marginals of a rank\-8 token and errors accumulate\. TQ\-KV is flat and good but cannot compete with SQ atb≥4b\\geq 4\.

#### Lowrank: WHT is redundant when the subspace is random\.

Without the 5% isotropic noise of lowrank\_aligned, the data subspace is not axis\-aligned and the WHT provides no structural advantage\. TQ\-KV and Plain\-KV are nearly identical across allddand budgets — the rotation neither helps nor hurts\. SQU\-KV is competitive but slightly worse because rank\-8 calibration in a larger ambientddrequires more calibration tokens to pin down the subspace precisely\.

#### Lowrank\_steep: water\-filling activates but gains are modest\.

The exponentially decaying eigenvalue spectrum \(ratio 8 withindeff=4d\_\{\\mathrm\{eff\}\}=4dimensions\) is the one regime where integer greedy water\-filling assigns non\-uniform bits\. SQW\-KV edges SQU\-KV by up to\|Δ​d2\|=0\.030\|\\Delta d\_\{2\}\|=0\.030atb=6b=6\. SQU\-QKV is consistently the worst scheme here: the 1\-bit score correction in a 4\-dimensionalZZspace amplifies per\-token residual variance through the softmax exponential \(Jensen effect\), hurting attention quality even as cache quality improves\.

#### Lowrank\_misalign: separate calibration is essential\.

WhenKKandVVlive in orthogonal subspaces, a shared eigenbasis is useless forVV\. SQU\-KV with*separate*𝐔K\\mathbf\{U\}\_\{K\}and𝐔V\\mathbf\{U\}\_\{V\}calibrated independently achievesd2≈0\.04d\_\{2\}\\approx 0\.04atb=6b=6, competitive with TQ\-KV\. The non\-monotone behaviour atb=2b=2reflects the difficulty of fitting a rank\-4 codebook from 512 calibration tokens in add\-dimensional ambient space — the same finite\-calibration sensitivity studied in Section[7](https://arxiv.org/html/2607.09683#S7)\.

#### Unit\_sphere: SQ self\-limits to its calibration degrees of freedom\.

On isotropic data allddeigenvalues are equal, so the analyticaldeff=dd\_\{\\mathrm\{eff\}\}=d\. With onlyncal=512n\_\{\\mathrm\{cal\}\}=512tokens the empirical participation ratio self\-limits todeff≈32d\_\{\\mathrm\{eff\}\}\\approx 32regardless ofdd: the algorithm discovers how many dimensions the available data can reliably support, not how many the ambient space contains\. SQU\-KV allocates its bits across 32 well\-sampled dimensions and achievesd2≈0\.10d\_\{2\}\\approx 0\.10atb=2b=2— matching TQ\-KV — with a curve that is flat acrossd∈\{64,128,256\}d\\in\\\{64,128,256\\\}\. This is not an artefact; it reflects the true degrees of freedom accessible to any data\-driven method under a fixed calibration budget\.

#### QJL onKK: Jensen amplification limits its benefit\.

TQ\-QKV and SQU\-QKV both apply a 1\-bit RHT score correction onKK\. On fattail, TQ\-QKV is worse than TQ\-KV: heavy\-tailed tokens have large residual norms, and the score correction variance inflates through the softmax exponential \(Jensen effect\)\. On structured regimes, SQU\-QKV is similarly degraded relative to SQU\-KV — the correction operates in a low\-dimensionalZZspace where per\-dimension residuals are larger and variance amplification is stronger\. QJL onKKis only beneficial when residual norms are small and the number of projectionsm≫deffm\\gg d\_\{\\mathrm\{eff\}\}\.

### 8\.16D Error Geometry: Two Case Studies

Thed2d\_\{2\}scalar summarises overall quality but conflates four distinct error sources: scale error and direction error in theKKcache,VVcache, andTToutput respectively\. We use the 6D error geometry \(Section[2](https://arxiv.org/html/2607.09683#S2)\) to decompose the TQ\-KV vs\. SQU\-KV gap on the two most informative regimes\.

![Refer to caption](https://arxiv.org/html/2607.09683v1/6d_lowrank_aligned.png)Figure 6:6D error geometry for TQ\-KV \(red\) vs\. SQU\-KV \(green\) on lowrank\_aligned,d=128d=128, Shannon\-normalised origin\. Left:b=2b=2\. Right:b=4b=4\. Shapes:∙\\bulletK cache,▲\\blacktriangleV cache,■\\blacksquareT output\. Size∝log⁡S\\propto\\log S\. Both K/V caches are competitive; the gap is entirely in the T output squares, where SQU\-KV’s unmodelled tail dimensions leak through the softmax\.#### Lowrank\_aligned: better cache does not imply better output\.

Figure[6](https://arxiv.org/html/2607.09683#S8.F6)compares TQ\-KV and SQU\-KV on lowrank\_aligned \(d=128d=128, Shannon\-normalised origin\)\. Atb=2b=2\(left\), the K and V cache points \(circles, triangles\) lie at nearly the same distance from the Shannon\-optimal origin for both schemes — SQU\-KV’s eigenbasis concentrates bits on the 8 signal dimensions, achieving comparable or slightly better cache quality\. Yet the T output squares \(filled squares\) tell the opposite story: SQU\-KV T output drifts tosnr​\_​err≈0\.87\\mathrm\{snr\\\_err\}\\approx 0\.87,dir​\_​err≈0\.25\\mathrm\{dir\\\_err\}\\approx 0\.25, while TQ\-KV T output stays tight atdir​\_​err≈0\.08\\mathrm\{dir\\\_err\}\\approx 0\.08\. Thed−deff=120d\-d\_\{\\mathrm\{eff\}\}=120tail dimensions that SQU\-KV does not model still carry signal energy; under the nonlinear softmax they amplify into the T output even when the cache is clean\. Notably, the KL divergence between reference and quantized attention weights is nearly identical for both schemes \(KL≈0\.003\\mathrm\{KL\}\\approx 0\.003atb=2b=2,≈0\.0003\\approx 0\.0003atb=4b=4\) — the attention distribution itself is correct for both\. The T output gap is therefore entirely aVVreconstruction issue: SQU\-KV attends to the right tokens but reconstructs their values less accurately because the unmodelled tail contributes residual errors that accumulate across theSSattended tokens\. Atb=4b=4\(right\), both K/V caches approach Shannon\-optimality together, but SQU\-KV T squares drift further right \(snr​\_​err≈0\.90\\mathrm\{snr\\\_err\}\\approx 0\.90\) while TQ\-KV T squares cluster near the origin\. Adding bits helps TQ uniformly; it helps SQU only within the signal subspace\.

![Refer to caption](https://arxiv.org/html/2607.09683v1/6d_fattail.png)Figure 7:6D error geometry for TQ\-KV \(red\) vs\. SQU\-KV \(green\) on fattail,d=128d=128, Lloyd\-Max\-normalised origin\. Left:b=2b=2\. Right:b=4b=4\. Atb=4b=4, TQ\-KV T squares reach near\-zero direction error while SQU\-KV K/V cache points collapse tosnr​\_​err≈0\.98\\mathrm\{snr\\\_err\}\\approx 0\.98: the codec encodes the miscalibrated subspace with increasing precision, gaining nothing on direction error or T output quality\.
#### Fattail: encoding noise with increasing precision\.

Figure[7](https://arxiv.org/html/2607.09683#S8.F7)shows the same comparison on fattail \(Lloyd\-Max\-normalised origin\)\. Atb=2b=2\(left\), TQ\-KV K/V cache points cluster near the Lloyd\-Max line while SQU\-KV cache points scatter atsnr​\_​err≈0\.7\\mathrm\{snr\\\_err\}\\approx 0\.7,dir​\_​err≈0\.09\\mathrm\{dir\\\_err\}\\approx 0\.09— the miscalibrated eigenbasis already corrupts the cache\. The T output squares for SQU\-KV reachdir​\_​err≈0\.7\\mathrm\{dir\\\_err\}\\approx 0\.7–0\.90\.9; attention directions are essentially random\. Atb=4b=4\(right\), the divergence is stark: TQ\-KV T squares hug the x\-axis \(dir​\_​err≈0\\mathrm\{dir\\\_err\}\\approx 0, improvingsnr​\_​err\\mathrm\{snr\\\_err\}\) while SQU\-KV K/V cache points collapse tosnr​\_​err≈0\.98\\mathrm\{snr\\\_err\}\\approx 0\.98— the codec encodes the wrong subspace with four times the precision\. Adding bits to a miscalibrated eigenbasis does not reduce direction error; it encodes the wrong directions more faithfully\. This is the irreversibility of eigenbasis failure: no budget increase can recover from a systematically misaligned subspace\.

### 8\.2Iso\-Storage and Replacement Analysis

The 6D analysis reveals a quality ceiling for SQU\-KV: thed−deff=120d\-d\_\{\\mathrm\{eff\}\}=120tail dimensions are fixed atbtail=2b\_\{\\mathrm\{tail\}\}=2bits regardless of the nominal budget\. Asbbgrows, TQ improves uniformly across all 128 dimensions while SQ’s tail contribution stagnates\. We ask directly: for which TQ budgets and sequence lengths can SQU\-KV replace TQ\-KV without loss of accuracy, and at what storage cost?

Table[5](https://arxiv.org/html/2607.09683#S8.T5)summarises the result ford=128d=128across three sequence lengths\. Each row shows the minimum SQ budget that achievesd2​\(SQU\-KV\)≤d2​\(TQ\-KV,b\)d\_\{2\}\(\\text\{SQU\-KV\}\)\\leq d\_\{2\}\(\\text\{TQ\-KV\},b\)and the resulting storage difference\.

Table 5:Minimum SQU\-KV budget that replaces TQ\-KV at equal or betterd2d\_\{2\}, and the resulting byte difference per token \(d=128d=128, 200 trials\)\. NegativeΔ\\Deltabytes means SQ uses less memory\. “—” means no SQ budget achieves the target quality\.Three findings stand out\. First, SQ can replace TQ atb=2b=2on all structured regimes \(lowrank, lowrank\_aligned, lowrank\_misalign, lowrank\_steep, unit\_sphere\) at essentially the same storage — and typically with strictly betterd2d\_\{2\}\. Second, on lowrank\_aligned and lowrank, the replacement extends tob=3b=3with a modest saving of 900–14,000 bytes depending on sequence length\. Third, atb≥4b\\geq 4on any regime, or at any budget on fattail, no SQ budget achieves TQ\-KV’s quality\. The tail floor becomes the binding constraint\.

The storage difference grows linearly withSSbecause it is a fixed per\-token overhead: SQU\-KV atbsem=3b\_\{\\mathrm\{sem\}\}=3uses3×8\+2×120=2643\\times 8\+2\\times 120=264bits/token versus TQ\-KV atb=2b=2using2×128=2562\\times 128=256bits/token — a constant 8\-bit gap that amounts to 64 bytes atS=64S=64and 4,096 bytes atS=4096S=4096\. The replacement range also shrinks with sequence length because accuracy degrades withSS\. AtS=4096S=4096, accumulated tail errors across tokens erode SQ’s semantic advantage and TQ’s uniform allocation prevails\. SQU\-KV is therefore the right choice for prefill workloads \(smallSS, structured queries at low budget\); TQ\-KV is the safe default for generation \(largeSS, any distribution\)\.

## 9Conclusion

Figure[8](https://arxiv.org/html/2607.09683#S9.F8)summarises the complete ranking of all six schemes across six regimes at budgets 2, 4, and 6\.

![Refer to caption](https://arxiv.org/html/2607.09683v1/d2_ranking_final6_v2_b246.png)Figure 8:d2d\_\{2\}ranking heatmap \(d=128d=128, 200 trials, v2 data\)\. Each cell shows rank \(\#1 = best, green; \#6 = worst, red\) and meand2d\_\{2\}\. Three budgets summarise the low\-, mid\-, and high\-fidelity regimes\.The ranking tells a clear story\. Atb=2b=2, no single scheme dominates: SQU\-KV wins on structured regimes \(lowrank, lowrank\_aligned\), TQ\-KV wins on fattail, and Plain\-KV is competitive on isotropic data\. Atb=4b=4, TQ\-KV asserts itself across most regimes — only lowrank\_aligned remains contested\. Atb=6b=6, TQ\-KV is rank 1 or 2 everywhere; SQU\-KV retains an edge only on lowrank\_aligned and lowrank\_misalign where the calibrated eigenbasis concentrates bits most efficiently\.

The practical recommendations follow directly\.Use TQ\-KVwhen the data distribution is unknown, heavy\-tailed, or subject to distribution shift, and at any budget above 2\.Use SQU\-KVwhen the data is known to be structured \(low\-rank KV subspace\), the budget is at most 2–3 bits per dimension, the sequence is short \(prefill rather than generation\), and separate K and V calibration sets are available\. Water\-filling \(SQW\-KV\) adds nothing over uniform allocation on synthetic data and can be dropped\. QJL on K \(TQ\-QKV, SQU\-QKV\) is beneficial only when residual norms are small andm≫deffm\\gg d\_\{\\mathrm\{eff\}\}; it amplifies Jensen variance on heavy\-tailed data and should be avoided there\.

## References

- \[1\]Paolo D’Alberto\.Statistical inference and quality measures of KV cache quantisations inspired by TurboQuant\.arXiv:2605\.08114 \[cs\.LG\], 2026\.[https://arxiv\.org/abs/2605\.08114](https://arxiv.org/abs/2605.08114)\.
- \[2\]Ashwin Gopinath\.3% Is All You Need: Breaking TurboQuant’s compression limit via spectral structure\.GitHub:[https://github\.com/Dynamis\-Labs/spectralquant](https://github.com/Dynamis-Labs/spectralquant), 2026\.
- \[3\]Coleman Hooper, Sehoon Kim, Hasan Mohammadzadeh, Michael W\. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami\.KVQuant: Towards 10 million context length LLM inference with KV cache quantization\.InAdvances in Neural Information Processing Systems, 2024\.
- \[4\]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\.KIVI: A tuning\-free asymmetric 2bit quantization for KV cache\.InProceedings of the 41st International Conference on Machine Learning, 2024\.
- \[5\]Amir Zandieh et al\.TurboQuant: Online vector quantization with near\-optimal distortion rate\.InInternational Conference on Learning Representations \(ICLR\), 2026\.arXiv:2504\.19874\.
- \[6\]Amir Zandieh, Insu Han, Vahab Mirrokni, and Amin Karbasi\.QJL: 1\-bit quantized JL transform for KV cache quantization with zero overhead\.InAdvances in Neural Information Processing Systems, 2024\.