SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding

# SPECTRA: Pushing the KV Cache Beyond the 2-Bit Cliff via Spectral Transform Coding
Source: [https://arxiv.org/html/2608.07915](https://arxiv.org/html/2608.07915)
Jiamu Zhang1,2Liang Wu1Kelly Wan1Hanjie Chen2Liangjie Hong1 1Nokia2Rice University

###### Abstract

Large language models \(LLMs\) increasingly read very long inputs in the agentic era nowadays, from whole documents and codebases to conversations across many turns\. Their inference memory is then dominated by the key\-value \(KV\) cache, the running store of the attention keys and values of everything the model has read and generated\. Since the cache grows with the context length and is re\-read in full at every generated token, a longer context directly means more GPU memory, until the cache dominates what the hardware can hold\.

To reduce this cost, most existing methods compress the KV cache by lowering every stored value to the same low precision, a technique known as quantization\. They can push this to nearly two bits per value, but rarely further, because at this so\-called 2\-bit cliff quality drops sharply: four levels per value are too few for the cache’s outlier\-heavy values, where a few large entries consume the levels and collapse the many small ones into noise\. A natural remedy is to spend more bits on the channels that matter and fewer on the rest, but the raw cache offers no handle: its channels are strongly correlated, so none is clearly more important than another\.

To fill this gap, we conduct a comprehensive study and find that this handle appears once the cache is rotated into a coordinate system that is computed from its own statistics and removes these correlations\. There, the channels’ contributions are highly uneven: a small fraction carries almost all of the information, and spending the budget on those few is far more accurate than spreading it evenly\.

Guided by these observations, we develop SPECTRA, a training\-free, drop\-in codec that re\-encodes the cache into this coordinate system and concentrates the bit budget on the channels that carry the signal\. On Llama\-3\.1\-8B and Qwen2\.5\-7B over long\-context benchmarks, SPECTRA is near\-lossless at 4x compression, competitive at 8x where uniform quantization has collapsed, and reaches up to 12x, pushing usable compression past the 2\-bit cliff and letting the same GPU serve much longer contexts and larger batches at higher throughput\. Code:[https://github\.com/nokia\-applied\-research/SPECTRA](https://github.com/nokia-applied-research/SPECTRA)\.

## 1Introduction

Large language models \(LLMs\) now power a growing range of agentic applications\. A coding assistant may work over an entire repository, and a dialogue agent may run for many turns while calling external toolsYaoet al\.\([2023](https://arxiv.org/html/2608.07915#bib.bib30)\); Schicket al\.\([2023](https://arxiv.org/html/2608.07915#bib.bib31)\); Jimenezet al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib32)\); Yanget al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib33)\)\. What these applications have in common is length: the context, together with the model’s own generated reasoning, can reach hundreds of thousands of tokensTeamet al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib34)\); Hsiehet al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib35)\); Baiet al\.\([2024b](https://arxiv.org/html/2608.07915#bib.bib36)\)\. At this scale the main cost of serving is no longer the model weights but the key\-value \(KV\) cache, which stores the attention keys and values of past tokens so the model does not recompute them\. Because every new token attends to the whole cache, the cache must sit in fast memory and be reloaded at each step, and its size grows with both the context length and the number of concurrent requests\. As a result, at long context the KV cache can grow larger than the model weights themselvesKwonet al\.\([2023](https://arxiv.org/html/2608.07915#bib.bib37)\); Zhenget al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib38)\), and it becomes the main constraint on the context length and the number of concurrent requests that the serving hardware can support\.

Many methods compress the KV cache, and they reduce it along one of three axes\. Token eviction methods reduce the cache by discarding tokens they judge unimportant, such as H2OZhanget al\.\([2023](https://arxiv.org/html/2608.07915#bib.bib19)\)and SnapKVLiet al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib18)\), which keep the tokens that have received the most attention, or StreamingLLMXiaoet al\.\([2024b](https://arxiv.org/html/2608.07915#bib.bib4)\), which keeps the first and the most recent tokens\. Low\-rank methods reduce the cache by storing keys and values in fewer dimensions, such as PaluChanget al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib20)\)and Eigen AttentionSaxenaet al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib43)\), which project them onto a low\-rank subspace obtained from the model’s weights or activations\. Quantization methods reduce the cache by storing every value at lower precision, such as KIVILiuet al\.\([2024b](https://arxiv.org/html/2608.07915#bib.bib1)\)and KVQuantHooperet al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib2)\), which quantize keys per channel and values per token to contain outlier channels\. These families differ in what they remove, yet most share the same strategy: they fix the budget in advance, as a set of tokens, a target rank, or a bit\-width, and rarely adapt it to how much each part of the cache matters for the output\. This limitation is clearest for quantization\. Lowering every value to the same precision reaches about two bits, but rarely goes further, because at this so\-called 2\-bit cliffDettmers and Zettlemoyer \([2023](https://arxiv.org/html/2608.07915#bib.bib39)\); Zhouet al\.\([2026](https://arxiv.org/html/2608.07915#bib.bib40)\), accuracy drops sharply: with only four levels per value, a few large entries take up the range and the many small ones collapse into noiseXiaoet al\.\([2024a](https://arxiv.org/html/2608.07915#bib.bib41)\); Dettmerset al\.\([2022](https://arxiv.org/html/2608.07915#bib.bib42)\)\.

We take a different view\. Rather than fixing any of these choices in advance, we ask where each bit of the budget should go, and treat KV\-cache compression as one problem: how to spend a limited budget on the parts of the cache that matter most\. Seen this way, the three axes are settings of a single allocation, and two of them are really one\. Low\-rank projection removes channels, while quantization sets how many bits each channel keeps, and a removed channel is just a channel kept at zero bits\. Choosing a rank and choosing a precision are therefore the same decision: how many bits to give each channel, from several down to zero\. To make this decision well, we first need to know which channels matter, yet the raw cache does not reveal this, because its channels are correlated and none stands out on its own\. We find that the answer appears once the cache is rotated into its own principal directions, the orthogonal axes along which it varies the most, which we estimate from its own statistics\. In this basis the channels are uncorrelated, and a small set of them accounts for most of the cache\.

Building on this, we introduce SPECTRA, a training\-free and drop\-in codec that rotates the cache into its spectral basis, the principal directions along which it varies, and spends its bit budget on the channels that carry the signal\. SPECTRA needs only a small, one\-time calibration to estimate this basis, after which its channels are uncorrelated and ordered by how much they contribute\. At inference, it stores each key and value in this basis and gives every channel a bit\-width that matches its contribution: more bits to the few dominant channels, and fewer, down to zero, for the rest\. A channel given zero bits is simply removed, so the single rule that sets each channel’s precision also sets its rank\. Because the basis is a small fixed matrix that folds into attention, SPECTRA reconstructs the cache with little overhead and without fine\-tuning\. Our contributions are summarized as follows:

- •An analysis of the KV cache showing that its information is concentrated in a few of its principal directions, where a small set of channels carries most of the signal\.This recasts KV\-cache compression as spending a fixed bit budget on the channels that matter, and unifies low\-rank projection and quantization as the two ends of one control, since a channel given zero bits is simply dropped\.
- •SPECTRA, a training\-free and drop\-in KV\-cache codec that transforms the cache into these principal directions and allocates its bits to the channels that carry the signal\.We evaluate SPECTRA on Llama\-3\.1\-8B, Mistral\-7B, and Qwen2\.5\-7B across LongBench and RULER\. SPECTRA compresses the KV cache further than existing quantization and low\-rank methods at the same quality, and stays near\-lossless well past 8x compression rate at which uniform quantization breaks down\.

## 2Motivation: Where Should Each Bit Go?

We view the KV cache as a signal to be coded rather than a tensor to be trimmed, so as a result, compressing it is then a rate–distortion problem, in which the question is not how many bits to use or which tokens to keep, but where each bit should go\. This reframes KV\-cache compression from an engineering heuristic into a coding problem, and raises a concrete question: given a fixed budget, in which basis and by what measure of importance should each bit be allocated across channels and precision? We answer it by probing the KV cache of pretrained LLMs directly, scoring each choice by the error it induces in the attention output on held\-out text; the statistics that define our transform come from a small calibration set\.The three observations below answer this question in turn\.

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/motivation/fig_OB1_concentration.png)Figure 1:Observation 1\.*Left:*magnitude of the key channel\-correlation matrix; the lower triangle shows the raw channels \(mean off\-diagonal≈0\.2\\approx 0\.2\) and the upper triangle shows the channels after rotating the cache into a basis computed from its own statistics \(≈0\\approx 0\)\.*Right:*cumulative energy over channels ranked by energy; the top25%25\\%of channels hold96%96\\%of the key energy, and the value curve is shown for reference\.### 2\.1OB1: The cache’s importance is concentrated, but only in the right basis

In the raw channel basis the keys within a layer are strongly correlated, so no channel stands out as more important than another and there is no natural axis to allocate along, lower triangle: mean off\-diagonal correlation≈0\.2\\approx 0\.2\)\. Once we rotate the cache into a basis computed from its own statistics, the channels become uncorrelated \(upper triangle:≈0\\approx 0\), and their energy concentrates sharply: the top25%25\\%of channels carry96%96\\%of the key energy, and the rest are nearly redundant \(Figure[1](https://arxiv.org/html/2608.07915#S2.F1), right\)\. Importance is therefore separable and heavily skewed, but only after this transform, and this tells us to allocate in the transformed basis rather than the raw one\.

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/motivation/fig_OB2_significance.png)Figure 2:Observation 2\.*Left:*per\-layer reduction in attention\-output error from importance\-weighted allocation over uniform \(uniform error÷\\divimportance\-weighted error\) at a one\-bit and a two\-bit budget; the reduction exceeds1×1\\timesat almost every layer, with a median of1\.6×1\.6\\timesand up to5\.5×5\.5\\timesat one bit\.*Right:*attention\-output error versus average bits per channel at a representative layer; the gap is largest at one bit and closes as precision grows\. Channels given zero bits are dropped, so this allocation unifies rank reduction and quantization\.
### 2\.2OB2: Allocating bits by importance beats uniform precision, most sharply below two bits\.

Because the channels differ so much in energy, giving each the same number of bits wastes the budget\. When we instead give more bits to high\-energy channels and fewer, down to zero, to low\-energy ones, the attention\-output error drops at almost every layer \(Figure[2](https://arxiv.org/html/2608.07915#S2.F2), left\), by a median of1\.6×1\.6\\timesand up to5\.5×5\.5\\timesat one bit per channel\. The advantage is concentrated where uniform quantization fails: the gap is largest at one bit and closes as the bit\-width grows \(Figure[2](https://arxiv.org/html/2608.07915#S2.F2), right\), exactly the sub\-two\-bit regime that gives the 2\-bit cliff its name\. A channel assigned zero bits is simply dropped, so this single allocation spans both quantization and rank reduction, making low\-rank projection and quantization the two ends of one control\.

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/motivation/fig_OB3_gweighted.png)Figure 3:Observation 3\.Both panels compare an activation\-weighted transform \(from𝐆=𝔼​\[𝐡⊤​𝐡\]\\mathbf\{G\}=\\mathbb\{E\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\]\) with a weight\-only transform\.*Left:*KV reconstruction error versus the fraction of rank kept; at half the rank the activation\-weighted error is about13×13\\timeslower\.*Right:*attention\-output error versus average bits per channel; at a matched budget the activation\-weighted error is about1\.9×1\.9\\timeslower\.
### 2\.3OB3: A channel’s importance is set by the data, not its weights\.

Wo observe that how much a channel matters depends on how strongly the input activations drive it, not on the magnitude of its projection weights\. When we build the transform and rank the channels from the activation statistics𝐆=𝔼​\[𝐡⊤​𝐡\]\\mathbf\{G\}=\\mathbb\{E\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\], where𝐡\\mathbf\{h\}is the layer input to the key/value projection, rather than from the weight matrix alone, we keep far more of the cache at the same budget: at half the rank the reconstruction error is about13×13\\timeslower \(Figure[3](https://arxiv.org/html/2608.07915#S2.F3), left\), and at a matched bit\-rate the attention\-output error is about1\.9×1\.9\\timeslower \(Figure[3](https://arxiv.org/html/2608.07915#S2.F3), right\)\. The basis and the importance measure must therefore both be activation\-weighted, not weight\-only\.

Together these observations specify our method\. We encode the cache in an activation\-weighted transform that decorrelates and concentrates it \(Observations 1 and 3\), and we spend a fixed bit budget across the resulting channels by importance, letting the weakest channels fall to zero bits \(Observation 2\)\. Section[3](https://arxiv.org/html/2608.07915#S3)makes this precise as SPECTRA\.

## 3Method

Throughout this section, bold lowercase denotes row vectors \(a hidden state𝐡∈ℝ1×d\\mathbf\{h\}\\in\\mathbb\{R\}^\{1\\times d\}, a key𝐤∈ℝ1×dk​v\\mathbf\{k\}\\in\\mathbb\{R\}^\{1\\times d\_\{kv\}\}, a latent𝐜∈ℝ1×r\\mathbf\{c\}\\in\\mathbb\{R\}^\{1\\times r\}\) and bold uppercase denotes matrices \(the projection𝐖K∈ℝd×dk​v\\mathbf\{W\}\_\{K\}\\in\\mathbb\{R\}^\{d\\times d\_\{kv\}\}, the stacked cache𝐊=𝐇𝐖K∈ℝL×dk​v\\mathbf\{K\}=\\mathbf\{H\}\\mathbf\{W\}\_\{K\}\\in\\mathbb\{R\}^\{L\\times d\_\{kv\}\}, and the uncentered second\-moment matrix𝐆=𝔼𝐡​\[𝐡⊤​𝐡\]∈ℝd×d\\mathbf\{G\}=\\mathbb\{E\}\_\{\\mathbf\{h\}\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\]\\in\\mathbb\{R\}^\{d\\times d\}\);𝔼𝐡\\mathbb\{E\}\_\{\\mathbf\{h\}\}is the expectation over hidden states on calibration data\.

Based on the 3 observations mentioned in the previous section, our reframing casts KV\-cache compression as deciding where to spend a fixed budget, which leaves one design choice to make concrete: what the cache should store\. The answer follows from how redundant the produced keys and values are\. Measured in the metric of the model’s activations, most of their energy occupies a small subspace, so we cache a compact latent in place of𝐊\\mathbf\{K\}and𝐕\\mathbf\{V\}and recover them only when needed\. This single decision organizes the rest of the section\. We first build the latent through aGG\-weighted transform \(a KLT in theGG\-metric\) that is second\-moment\-orthogonal and energy\-ordered by construction \(Section[3\.1](https://arxiv.org/html/2608.07915#S3.SS1)\); we then quantize it at a rate–distortion optimum that this very structure makes well\-posed \(Section[3\.2](https://arxiv.org/html/2608.07915#S3.SS2)\); and we finally let the same latent turn its memory saving into a compute saving, through attention carried out directly in the latent space \(Section[3\.3](https://arxiv.org/html/2608.07915#S3.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2608.07915v1/x1.png)Figure 4:Spectramethod overview\.\(a\) Latent KV cache from aGG\-weighted SVD\.Each key/value projection is factorized from the whitened weightM=G1/2​WM=G^\{1/2\}W; keeping the top\-rrsingular directions \(Eckart–Young\) yieldsWdown,WupW\_\{\\text\{down\}\},W\_\{\\text\{up\}\}, so a length\-LLsequence is cached as anL×rL\\times renergy\-ordered latent instead of theL×dk​vL\\times d\_\{kv\}full\-precisionK,VK,V; the low\-energy directions are discarded by the low\-rank truncation \(rank reduction\)\.\(b\) Water\-filling bit allocation\.Reverse water\-filling against a levelθ\\thetaassigns more bits to high\-variance latent groups and drives the low\-energy tail to zero bits, i\.e\. rank reduction\.\(c\) Streaming decode \(Mode A\)\.Each new token is down\-projected, quantized per token with the group bit\-widths, and appended to the latent cache;K,VK,Vare reconstructed on the fly \(RoPE applied toKK\) for standard attention\.### 3\.1Caching the Latent: AGG\-weighted Transform

We now make the latent precise\. We seek a rank\-rrfactorization of each key projection,𝐖K≈𝐖down​𝐖up\\mathbf\{W\}\_\{K\}\\approx\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}with𝐖down∈ℝd×r\\mathbf\{W\}\_\{\\text\{down\}\}\\in\\mathbb\{R\}^\{d\\times r\}and𝐖up∈ℝr×dk​v\\mathbf\{W\}\_\{\\text\{up\}\}\\in\\mathbb\{R\}^\{r\\times d\_\{kv\}\}\(r≤dk​vr\\leq d\_\{kv\}\), so that the cache holds the latent𝐜=𝐡𝐖down\\mathbf\{c\}=\\mathbf\{h\}\\mathbf\{W\}\_\{\\text\{down\}\}and recovers a key as𝐤^=𝐜​𝐖up\\hat\{\\mathbf\{k\}\}=\\mathbf\{c\}\\,\\mathbf\{W\}\_\{\\text\{up\}\}only when it is needed\. Over a length\-LLsequence this replaces the conventionalL×dk​vL\\times d\_\{kv\}cache𝐊=𝐇𝐖K\\mathbf\{K\}=\\mathbf\{H\}\\mathbf\{W\}\_\{K\}with theL×rL\\times rlatent𝐂=𝐇𝐖down\\mathbf\{C\}=\\mathbf\{H\}\\mathbf\{W\}\_\{\\text\{down\}\}, reconstructing𝐊^=𝐂​𝐖up\\hat\{\\mathbf\{K\}\}=\\mathbf\{C\}\\,\\mathbf\{W\}\_\{\\text\{up\}\}on demand \(Figure[4](https://arxiv.org/html/2608.07915#S3.F4)\(a\)\); the low\-energy directions beyond rankrrare simply discarded by the truncation, so the discarded tail is a rank reduction rather than a separate pruning step\. The question is what “≈\\approx” should mean here\. It should*not*mean that𝐖K≈𝐖down​𝐖up\\mathbf\{W\}\_\{K\}\\approx\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}in the ordinary Frobenius sense, which treats every direction of the weight as equally important; what we actually care about is that the keys the model*produces*are preserved,𝐡𝐖K≈𝐡​𝐖down​𝐖up\\mathbf\{h\}\\mathbf\{W\}\_\{K\}\\approx\\mathbf\{h\}\\,\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}, on the activations𝐡\\mathbf\{h\}that actually occur\. The right objective is therefore an expectation over the model’s own activation distribution,

minrank⁡r⁡𝔼𝐡​‖𝐡​\(𝐖K−𝐖down​𝐖up\)‖2,\\min\_\{\\operatorname\{rank\}r\}\\ \\mathbb\{E\}\_\{\\mathbf\{h\}\}\\big\\\|\\mathbf\{h\}\\,\(\\mathbf\{W\}\_\{K\}\-\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}\)\\big\\\|^\{2\},\(1\)and identically for𝐖V\\mathbf\{W\}\_\{V\}\.

The expectation in Equation \([1](https://arxiv.org/html/2608.07915#S3.E1)\) has a convenient closed form\. WritingΔ​𝐖=𝐖K−𝐖down​𝐖up\\Delta\\mathbf\{W\}=\\mathbf\{W\}\_\{K\}\-\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}and using𝐆=𝔼𝐡​\[𝐡⊤​𝐡\]\\mathbf\{G\}=\\mathbb\{E\}\_\{\\mathbf\{h\}\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\], a short calculation \(Appendix[B](https://arxiv.org/html/2608.07915#A2)\) gives𝔼𝐡​‖𝐡​Δ​𝐖‖2=‖𝐆1/2​Δ​𝐖‖F2\\mathbb\{E\}\_\{\\mathbf\{h\}\}\\\|\\mathbf\{h\}\\,\\Delta\\mathbf\{W\}\\\|^\{2\}=\\\|\\mathbf\{G\}^\{1/2\}\\Delta\\mathbf\{W\}\\\|\_\{F\}^\{2\}, so the objective becomes aGG\-weighted low\-rank problem,

minrank⁡r⁡‖𝐆1/2​\(𝐖K−𝐖down​𝐖up\)‖F\.\\min\_\{\\operatorname\{rank\}r\}\\ \\big\\\|\\mathbf\{G\}^\{1/2\}\\,\(\\mathbf\{W\}\_\{K\}\-\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}\)\\big\\\|\_\{F\}\.\(2\)The factor𝐆1/2\\mathbf\{G\}^\{1/2\}simply re\-weights the reconstruction error by how often each input direction is actually used: a direction the model visits constantly is expensive to get wrong, while one it never visits costs nothing, regardless of how large the corresponding weights are\. This is the one placeSpectradeparts from plain low\-rank factorization of the cache, such as Palu\(Changet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib20)\)or SVD\-LLM\(Wanget al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib9)\): we minimize error in the metric of the activations, not in raw weight space\. This is exactly the distinction our experiments confirm at the level of end\-task accuracy\.

Problem \([2](https://arxiv.org/html/2608.07915#S3.E2)\) has a closed\-form solution\. Whitening the weight \(rescaling it by𝐆1/2\\mathbf\{G\}^\{1/2\}so that reconstruction error is measured in the activation metric\) as𝐌=𝐆1/2​𝐖K\\mathbf\{M\}=\\mathbf\{G\}^\{1/2\}\\mathbf\{W\}\_\{K\}and taking its SVD𝐌=𝐔​𝚺​𝐕⊤\\mathbf\{M\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\}, the optimal rank\-rrfactor is obtained by keeping the toprrcomponents \(Eckart–Young\(Eckart and Young,[1936](https://arxiv.org/html/2608.07915#bib.bib22)\), the classical result that a truncated SVD is the best low\-rank approximation, now in theGG\-metric\) and undoing the whitening with a*square\-root*split of the singular values:

𝐖down=𝐆−1/2​𝐔r​𝚺r1/2,𝐖up=𝚺r1/2​𝐕r⊤\.\\mathbf\{W\}\_\{\\text\{down\}\}=\\mathbf\{G\}^\{\-1/2\}\\,\\mathbf\{U\}\_\{r\}\\,\\mathbf\{\\Sigma\}\_\{r\}^\{1/2\},\\qquad\\mathbf\{W\}\_\{\\text\{up\}\}=\\mathbf\{\\Sigma\}\_\{r\}^\{1/2\}\\,\\mathbf\{V\}\_\{r\}^\{\\top\}\.\(3\)Any split of𝚺r\\mathbf\{\\Sigma\}\_\{r\}between the two factors gives the same reconstruction𝐖down​𝐖up\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}; what the square\-root split buys is the statistics of the cached quantity itself\. Substituting𝐜=𝐡𝐖down\\mathbf\{c\}=\\mathbf\{h\}\\mathbf\{W\}\_\{\\text\{down\}\}and using𝔼𝐡​\[𝐡⊤​𝐡\]=𝐆\\mathbb\{E\}\_\{\\mathbf\{h\}\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\]=\\mathbf\{G\},

𝔼​\[𝐜⊤​𝐜\]=𝐖down⊤​𝔼​\[𝐡⊤​𝐡\]​𝐖down=𝐖down⊤​𝐆𝐖down=𝚺r\(diagonal, descending\)\.\\mathbb\{E\}\[\\mathbf\{c\}^\{\\top\}\\mathbf\{c\}\]=\\mathbf\{W\}\_\{\\text\{down\}\}^\{\\top\}\\,\\mathbb\{E\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\]\\,\\mathbf\{W\}\_\{\\text\{down\}\}=\\mathbf\{W\}\_\{\\text\{down\}\}^\{\\top\}\\mathbf\{G\}\\mathbf\{W\}\_\{\\text\{down\}\}=\\mathbf\{\\Sigma\}\_\{r\}\\qquad\(\\text\{diagonal, descending\}\)\.\(4\)Thus, the latent coordinates areorthogonal in uncentered second momentandenergy\-ordered, with coordinatejjcarrying second moment\(𝚺r\)j​j\(\\mathbf\{\\Sigma\}\_\{r\}\)\_\{jj\}\. Equation \([4](https://arxiv.org/html/2608.07915#S3.E4)\) should not be read as a covariance identity when𝔼​\[𝐜\]≠0\\mathbb\{E\}\[\\mathbf\{c\}\]\\neq 0: before quantization, we subtract the measured coordinate means and estimate the centered marginal variances separately\. The resulting canonical ordering distinguishes the transform from a plain SVD, which does not use the activation metric, and from the random rotations used by TurboQuantZandiehet al\.\([2025](https://arxiv.org/html/2608.07915#bib.bib15)\)or QuaRotAshkbooset al\.\([2024](https://arxiv.org/html/2608.07915#bib.bib17)\), which impose no energy ordering\.111A pure whitening split \(𝐖down=𝐆−1/2​𝐔r\\mathbf\{W\}\_\{\\text\{down\}\}=\\mathbf\{G\}^\{\-1/2\}\\mathbf\{U\}\_\{r\},𝐖up=𝚺r​𝐕r⊤\\mathbf\{W\}\_\{\\text\{up\}\}=\\mathbf\{\\Sigma\}\_\{r\}\\mathbf\{V\}\_\{r\}^\{\\top\}\) spans the same subspace but yields a unit\-variance latent, discarding the energy ordering; the square\-root split is what couples the transform to the bit allocator of Section[3\.2](https://arxiv.org/html/2608.07915#S3.SS2)\.

Two refinements complete the transform\. First, the low\-rank factorization of the value path leaves a small, structured residual at the attention output; since the output projection𝐖O\\mathbf\{W\}\_\{O\}lies outside the cache, we absorb the dominant part of this residual with a closed\-form, training\-free reduced\-rank correction to𝐖O\\mathbf\{W\}\_\{O\}\(Appendix[C](https://arxiv.org/html/2608.07915#A3)\), which changes neither the cached object nor the cost of inference\. Second, the rank budgetBBis split asymmetrically between the two paths,rk=κ​Br\_\{k\}=\\kappa Bandrv=\(1−κ\)​Br\_\{v\}=\(1\-\\kappa\)B: keys’ produced energy concentrates faster than values’, so keys tolerate a smaller rank \(more compression\) and values receive the larger share of the rank budget \(κ<12\\kappa<\\tfrac\{1\}\{2\}\)\. Both factors𝐖down,𝐖up\\mathbf\{W\}\_\{\\text\{down\}\},\\mathbf\{W\}\_\{\\text\{up\}\}and the𝐖O\\mathbf\{W\}\_\{O\}correction are folded back into the model weights, so the deployed model is a drop\-in replacement of𝐖K\\mathbf\{W\}\_\{K\}and𝐖V\\mathbf\{W\}\_\{V\}with no extra runtime structure\.

Beyond shrinking the cache, the transform hands the next stage a canonical, energy\-ordered latent with measured coordinate means and variances\. This gives the bit allocator a principled axis of importance, which we turn to next\.

### 3\.2Rate–Distortion Quantization of the Latent

Because the latent coordinates are energy\-ordered, quantizing them all at one bit\-width is wasteful: a high\-variance coordinate and a near\-zero one would be given the same precision despite contributing very differently to the reconstruction\. After subtracting the measured coordinate means, we model the centered coordinates as independent Gaussian scalar sources with marginal variancesvjv\_\{j\}\. This independence assumption is an approximation, because theGG\-weighted transform guarantees diagonal uncentered second moments, not diagonal centered covariance\. Under the scalar\-source model, quantization atbjb\_\{j\}bits incurs distortion proportional tovj​2−2​bjv\_\{j\}\\,2^\{\-2b\_\{j\}\}at high rate\(Gersho and Gray,[1992](https://arxiv.org/html/2608.07915#bib.bib11); Gray and Neuhoff,[1998](https://arxiv.org/html/2608.07915#bib.bib23)\), and minimizing∑jvj​2−2​bj\\sum\_\{j\}v\_\{j\}2^\{\-2b\_\{j\}\}subject to1r​∑jbj=b¯\\tfrac\{1\}\{r\}\\sum\_\{j\}b\_\{j\}=\\bar\{b\}yields the classical*reverse water\-filling*allocation\(Huang and Schultheiss,[1963](https://arxiv.org/html/2608.07915#bib.bib24); Cover and Thomas,[2006](https://arxiv.org/html/2608.07915#bib.bib10)\)\(Figure[4](https://arxiv.org/html/2608.07915#S3.F4)\(b\)\)

bj=b¯\+12​log2⁡\(vj/GM​\(v\)\),GM​\(v\)=\(∏jvj\)1/r,b\_\{j\}=\\bar\{b\}\+\\tfrac\{1\}\{2\}\\log\_\{2\}\\\!\\Big\(v\_\{j\}\\,/\\,\\mathrm\{GM\}\(v\)\\Big\),\\qquad\\mathrm\{GM\}\(v\)=\\Big\(\\textstyle\\prod\_\{j\}v\_\{j\}\\Big\)^\{1/r\},\(5\)clamped to\[0,bmax\]\[0,b\_\{\\max\}\]\(derivation in Appendix[D](https://arxiv.org/html/2608.07915#A4)\)\. Half a bit is added for every doubling of variance above the geometric mean; the optimum equalizes the residual distortion left in each retained coordinate, not the bits spent on it\.

The transform supplies what the allocation needs operationally: a canonical energy ordering and measured marginal variances\. Raw𝐊,𝐕\\mathbf\{K\},\\mathbf\{V\}channels are correlated and unordered, while random rotations such as TurboQuant or QuaRot\(Zandiehet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib15); Ashkbooset al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib17)\)impose no data\-dependent ordering\. Together, theGG\-weighted transform of Section[3\.1](https://arxiv.org/html/2608.07915#S3.SS1)and the per\-coordinate allocation here follow the classical transform\-coding pattern\(Goyal,[2001](https://arxiv.org/html/2608.07915#bib.bib25)\): transform a source into an ordered coordinate system, then allocate rate across its coefficients\.

Two properties make the allocation practical \(Figure[8](https://arxiv.org/html/2608.07915#A4.F8), Appendix[D](https://arxiv.org/html/2608.07915#A4)\)\. First, a coordinate given zero bits is dropped entirely, so asb¯\\bar\{b\}falls the low\-energy tail is zeroed and quantization passes*continuously*into rank reduction: the effective rank is an output of the allocator, not a separate hyperparameter\. Second, because per\-coordinate bit\-widths do not compose with a streaming cache, we groupggadjacent energy\-ordered coordinates to share one bit\-width \(from Equation \([5](https://arxiv.org/html/2608.07915#S3.E5)\) via the group’s mean variance\) and quantize each group with per\-token min/max scales, keeping the cache append\-only and outlier\-robust \(Figure[4](https://arxiv.org/html/2608.07915#S3.F4)\(c\)\)\. Each group stores its payload plus one fp16 scale and zero\-point per token, both counted in the effective rate; zero\-bit groups store nothing\.

### 3\.3Operating Points and System Co\-design

#### From storage to compute\.

Caching the latent also opens a compute path closed to scalar quantizers: because the score is linear in the latent, the up\-projection folds into the query and attention runs directly in therr\-dimensional latent space \(Mode B\)\. All quality results, however, use the reconstruct\-then\-attend path \(Mode A, Figure[4](https://arxiv.org/html/2608.07915#S3.F4)\(c\)\): each token is down\-projected, quantized, and appended, and𝐊,𝐕\\mathbf\{K\},\\mathbf\{V\}are rebuilt on the fly, which is RoPE\-exact and training\-free\.

## 4Experiments

### 4\.1Settings

#### Models\.

We evaluateSpectraon Llama\-3\.1\-8B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib28)\), Mistral\-7B\-Instruct\-v0\.3\(Jianget al\.,[2023](https://arxiv.org/html/2608.07915#bib.bib29)\), and Qwen2\.5\-7B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib47)\)\. All three use*grouped\-query attention*\(GQA\), an attention layout in which several query heads share a single key–value head: Llama and Mistral have 32 query and 8 key–value heads \(a 1024\-dimensional per\-token KV cache\), while Qwen2\.5\-7B has 28 query and 4 key–value heads \(a 512\-dimensional per\-token KV cache\)\. We cap contexts at 31,500 tokens so that all methods share a common evaluation protocol\.Spectrais implemented on top of Hugging Face Transformers and applied post hoc, without any fine\-tuning\.

#### Tasks\.

We evaluate on two long\-context suites\.LongBench\(Baiet al\.,[2024a](https://arxiv.org/html/2608.07915#bib.bib27)\)contributes eight English tasks spanning single\-document QA \(Qasper\), query\-based and multi\-document summarization \(QMSum, MultiNews\), few\-shot classification \(TREC\), few\-shot QA \(TriviaQA\), dialogue summarization \(SAMSum\), and repository\-level code completion \(LCC, RepoBench\-P\); we use the first150150examples of each task\.Needle\-in\-a\-Haystack\(NIAH\), which hides a short target string in a long distractor context and asks the model to retrieve it, measures long\-context retrieval on a grid of six context lengths \(11k–3232k\) and seven needle depths, in both single\-needle and four\-needle variants\.

#### Evaluation protocol\.

We follow the official LongBench harness: official per\-task prompt templates and metrics \(token\-levelF1F\_\{1\}for QA, ROUGE\-L for summarization, classification accuracy for TREC, and edit similarity for code\), middle\-truncation of inputs longer than31,50031\{,\}500tokens \(equal\-length prefix and suffix\), the chat template applied to all tasks except the few\-shot and code tasks \(TREC/TriviaQA/SAMSum/LCC/RepoBench\-P, which the official protocol feeds without a chat wrapper\), and greedy decoding with the task\-specific generation limits\. Every method including dense,Spectra, and all baselines, is run through this*identical*harness, so the comparison is strictly apples\-to\-apples\. NIAH scores the fraction of planted magic codes recovered by greedy generation\.

#### Baselines\.

We compare against five KV\-cache compression methods, each run from its official implementation but evaluated under the harness above: KIVI\(Liuet al\.,[2024b](https://arxiv.org/html/2608.07915#bib.bib1)\), TurboQuant\(Zandiehet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib15)\), PolarQuant\(Hanet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib16)\), OTT\(Suet al\.,[2025a](https://arxiv.org/html/2608.07915#bib.bib45)\), and RotateKV\(Suet al\.,[2025b](https://arxiv.org/html/2608.07915#bib.bib46)\)\. Baselines are run on Llama\-3\.1\-8B and Mistral\-7B; for Qwen2\.5\-7B we reportSpectraagainst the dense model only\. For every method we report the best configuration per compression tier, and all reported ratios are effective, counting the metadata each method stores; the full sweep and configuration details are in Appendix[G](https://arxiv.org/html/2608.07915#A7)\.

### 4\.2Overall Results

Tables[1](https://arxiv.org/html/2608.07915#S4.T1)and[2](https://arxiv.org/html/2608.07915#S4.T2)report LongBench, and Figure[5](https://arxiv.org/html/2608.07915#S4.F5)plots the resulting quality–compression frontier\. Two things stand out in our experiment result: at low compressionSpectrais effectively lossless and competitive with the strongest baselines: at3\.5×3\.5\\timesit scores53\.5653\.56on Llama and53\.5053\.50on Mistral, above the dense model and ahead of KIVI, OTT, and PolarQuant in the same range of compression rate\. The decisive gap is at high compression\. By approximately8×8\\timesthe scalar and rotation quantization method fall off the22\-bit cliff of Section[2](https://arxiv.org/html/2608.07915#S2)— on Llama, TurboQuant and RotateKV drop to47\.1147\.11and45\.9745\.97from a dense53\.2453\.24, and none of the five baselines report a usable point past8×8\\times\.Spectradoes not collapse: it holds54\.0054\.00at7\.7×7\.7\\timesand51\.8951\.89at8\.84×8\.84\\timescompression, and even stays usable out to11×11\\times\(48\.848\.8\), tracing a single frontier no baseline reaches\. Retrieval behaves the same way —Spectrastays lossless on single\-needle NIAH out to12×12\\times, where the quantization method degrade \(Appendix[G\.2](https://arxiv.org/html/2608.07915#A7.SS2)\)\.

Table 1:LongBench per task — Llama\-3\.1\-8B\-Instruct\.One SPECTRA row per compression tier for head\-to\-head with baselines, plus SPECTRA’s high\-compression points \(\>\>8×\\times\) that no baseline reaches\. Within each tier thebestandsecondAvg are highlighted\.Table 2:LongBench per task — Mistral\-7B\-Instruct\-v0\.3\.One SPECTRA row per compression tier for head\-to\-head with baselines, plus SPECTRA’s high\-compression points \(\>\>8×\\times\) that no baseline reaches\. Within each tier thebestandsecondAvg are highlighted\.MethodComp\.QasperQMSumMNewsTRECTQASAMSumLCCRB\-PAvgbase \(fp16\)1\.00×1\.00\\times36\.9325\.9526\.2574\.0089\.6646\.5264\.4861\.7753\.20\\cellcolorblue\!18SPECTRA3\.56×3\.56\\times37\.7625\.6625\.8674\.6789\.9946\.0465\.0463\.01\\cellcolorblue\!1853\.50OTT3\.76×3\.76\\times36\.9525\.7826\.1974\.0090\.0646\.4162\.7960\.6752\.86KIVI3\.76×3\.76\\times35\.9025\.7526\.1874\.0090\.0646\.3762\.9959\.4452\.59RotateKV4\.00×4\.00\\times39\.5625\.4726\.5374\.6790\.4346\.2661\.0957\.8652\.73\\cellcolorblue\!7TurboQuant4\.00×4\.00\\times37\.6625\.3626\.3574\.0089\.7346\.0563\.3961\.71\\cellcolorblue\!753\.03\\cellcolorblue\!18SPECTRA4\.88×4\.88\\times37\.7426\.0026\.0574\.6790\.2344\.6163\.7761\.06\\cellcolorblue\!1853\.02\\cellcolorblue\!7TurboQuant5\.33×5\.33\\times35\.9225\.6026\.0375\.3391\.6146\.1158\.7559\.03\\cellcolorblue\!752\.30RotateKV5\.33×5\.33\\times34\.4825\.1926\.3775\.3389\.0645\.0762\.5359\.6952\.21\\cellcolorblue\!18SPECTRA6\.31×6\.31\\times37\.7025\.9325\.8275\.3389\.7044\.5162\.7961\.78\\cellcolorblue\!1852\.94KIVI7\.11×7\.11\\times36\.0924\.9425\.6774\.0089\.8446\.1761\.6056\.3651\.83\\cellcolorblue\!7OTT7\.11×7\.11\\times35\.6424\.6826\.1674\.0090\.0146\.2862\.6358\.47\\cellcolorblue\!752\.23\\cellcolorblue\!18SPECTRA7\.73×7\.73\\times36\.2925\.9525\.9772\.0089\.4145\.5658\.3855\.81\\cellcolorblue\!1851\.17\\cellcolorblue\!7RotateKV8\.00×8\.00\\times30\.2424\.7926\.2570\.0087\.3142\.8859\.8753\.64\\cellcolorblue\!749\.37TurboQuant8\.00×8\.00\\times34\.5422\.9225\.5368\.6787\.9444\.4054\.6250\.2348\.61\\cellcolorblue\!18SPECTRA8\.93×8\.93\\times32\.5925\.8425\.9974\.0088\.3945\.5656\.8554\.05\\cellcolorblue\!1850\.41\\cellcolorblue\!18SPECTRA11\.28×11\.28\\times33\.6124\.9625\.2870\.0087\.8744\.4252\.6152\.03\\cellcolorblue\!1848\.85

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/pareto/llama_pareto_full.png)Figure 5:LongBench overall vs\. KV\-cache compression — Llama\-3\.1\-8B\-Instruct\.Average LongBench score \(8 EN tasks\) as a function of the effective KV compression ratio \(higherxx= smaller cache\)\. SPECTRA \(red\) forms a single Pareto frontier that stays within∼\\sim1\.5 points of the fp16 base \(53\.24, dashed\) out to∼\\sim8×\\timesand remains usable at 11\.12×\\times\(48\.81\), while the scalar/rotation baselines \(KIVI, OTT, TurboQuant, RotateKV, PolarQuant\) fall off the 2\-bit cliff by∼\\sim8×\\times\(TurboQuant 47\.11, RotateKV 45\.97\) and none reach past 8×\\times\. SPECTRA is on or above the frontier across the entire range; point labels give SPECTRA’s compression ratio and best config per tier\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/systems/oom_context_vs_memory.png)Figure 6:More context on the same hardware \(Llama\-3\.1\-8B, batch 1\)\.Longest single\-sequence context that fits before out\-of\-memory versus GPU memory budget, for the dense fp16 KV cache andSpectraat representative compression ratios\. Storing a compressed latent multiplies the context that fits in a fixed budget; measured H200 OOM points match the analytical curve within1%1\\%\(Table[3](https://arxiv.org/html/2608.07915#A5.T3)\)\. This isolates cache capacity, not an optimized attention kernel\.
### 4\.3Why cache a transformed latent?

We cache a*transformed*latent rather than the raw cache, because the transform concentrates the produced KV energy into a few ordered directions \(Section[2](https://arxiv.org/html/2608.07915#S2)\), and that concentration is what letsSpectraspend its budget on two axes at once instead of one\. Figure[7](https://arxiv.org/html/2608.07915#S4.F7)makes the consequence concrete: holding compression fixed and sweeping the latent width, the33\- and44\-bit curves stay far above the22\-bit curve everywhere, so a given storage budget goes much further when it keeps more precision*and*drops more rank than when it forces a single axis down to two bits\.Spectratherefore never has to sit on the22\-bit cliff\. Since rank reduction and quantization are the two ends of one control \(Section[3](https://arxiv.org/html/2608.07915#S3)\), it moves along the upper envelope of these curves, trading rank for precision as the target tightens; the crossover where33bits overtakes44at high compression is that trade\-off at work, since once the budget is tight, buying back rank at slightly lower precision beats holding precision and cutting rank\.

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/ablation/ablation_bits_rank.png)Figure 7:Splitting the budget beats spending it all on bits\(Llama\-3\.1\-8B,200200samples\)\. LongBench \(mean over six graded tasks\) against effective compression, sweeping the latent widthBBwithin each bit\-width\. At every matched compression the33\- and44\-bit curves dominate the22\-bit curve: given a fixed storage budget, distributing it across two axes—more bits*and*more rank reduction—is far better than driving a single axis to the22\-bit cliff \(red\)\.Spectratherefore never operates on the cliff; it moves along the upper envelope by trading rank for precision\.This compression translates directly into serving capacity\. Figure[6](https://arxiv.org/html/2608.07915#S4.F6)sweeps the GPU memory budget and reports the longest single sequence that fits before running out of memory: on2424GB the dense fp16 cache caps a Llama\-3\.1\-8B sequence at6565k tokens, whereasSpectrafits793793k at12×12\\times, and the measured H200 out\-of\-memory points track the analytical footprint within1%1\\%\(Table[3](https://arxiv.org/html/2608.07915#A5.T3)\)\. We report capacity, not wall\-clock speed: the latent\-space decode path of Section[3](https://arxiv.org/html/2608.07915#S3)turns this saved memory into saved compute in principle, but we validate it only with an isolated microbenchmark and leave end\-to\-end kernel integration to future work\.

## 5Related Work

Two families dominate KV\-cache compression\.*Quantization*lowers the bit\-width of the storedK,VK,V, either directly\(Liuet al\.,[2024b](https://arxiv.org/html/2608.07915#bib.bib1); Hooperet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib2)\)or after a decorrelating rotation\(Ashkbooset al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib17); Liuet al\.,[2024a](https://arxiv.org/html/2608.07915#bib.bib3); Zandiehet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib15); Hanet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib16)\), and degrades sharply near22bits\.*Token eviction*keeps only salient tokens\(Zhanget al\.,[2023](https://arxiv.org/html/2608.07915#bib.bib19); Liet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib18); Xiaoet al\.,[2024b](https://arxiv.org/html/2608.07915#bib.bib4)\), which is lossy on aggregation and multi\-fact queries\. Both fix the cached object to rawK,VK,V;Spectrais orthogonal and changes that object instead\.

Closest to us are low\-rank methods that cache an SVD\-based latent\(Changet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib20); Zhang and others,[2024](https://arxiv.org/html/2608.07915#bib.bib8)\)\(with related weight low\-rank compression\(Wanget al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib9); Liuet al\.,[2026](https://arxiv.org/html/2608.07915#bib.bib21)\)\) and latent\-attention architectures that build a compressed latent into the model\(DeepSeek\-AI,[2024](https://arxiv.org/html/2608.07915#bib.bib5); Meng and others,[2025](https://arxiv.org/html/2608.07915#bib.bib6); Ji and others,[2025](https://arxiv.org/html/2608.07915#bib.bib7)\)\. We differ in two ways: \(i\) we factorize in the metric of the produced activations \(aGG\-weighted, KLT\-like transform\) rather than in weight space, retaining more produced energy at equal rank and yielding a latent that is second\-moment\-orthogonal and energy\-ordered; and \(ii\) we are post\-hoc and training\-free, unlike architectures that require pretraining or distillation\. This latent makesSpectraan instance of classical transform coding, a Karhunen–Loève transform followed by rate–distortion \(water\-filling\) bit allocation\(Goyal,[2001](https://arxiv.org/html/2608.07915#bib.bib25); Cover and Thomas,[2006](https://arxiv.org/html/2608.07915#bib.bib10)\), applied to the cache: unlike allocation over the token or layer axis, we allocate over the channels of a data\-optimal transform\. A fuller treatment with additional baselines and background appears in Appendix[A](https://arxiv.org/html/2608.07915#A1)\.

## 6Conclusion

We have argued that the main lever in KV\-cache compression is not how finely to quantize the keys and values, but what to store in the first place\. The keys and values a model produces are highly redundant, yet that redundancy becomes usable only once they are rotated into a coordinate system computed from the model’s own statistics, where a few channels carry almost all of the information and the budget can be spent where it matters\.Spectraturns this into a training\-free, drop\-in codec that re\-encodes the cache into this basis and gives each channel a bit\-width that matches its contribution, from several bits down to zero, so that low\-rank reduction and quantization become two settings of one control rather than separate tools\. Because the transform folds into the key and value projections, the model runs unchanged\.

This letsSpectrapush usable compression past the point where quantization alone collapses: it is near\-lossless at4×4\\times, stays competitive at8×8\\timeswhere uniform quantization has already fallen off the22\-bit cliff, and remains usable up to12×12\\times, so the same GPU can hold far longer contexts\. How far the cache compresses tracks how concentrated its energy is, so the gains vary from model to model, but they degrade gracefully rather than collapsing\. Turning this saved memory into a matching speedup is the natural next step: caching the latent already opens an attention path that runs in the compressed space, and we leave a fused, end\-to\-end kernel for it to future work\.

## References

- S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. Hensman \(2024\)QuaRot: outlier\-free 4\-bit inference in rotated llms\.External Links:2404\.00456,[Link](https://arxiv.org/abs/2404.00456)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px1.p1.4),[§3\.1](https://arxiv.org/html/2608.07915#S3.SS1.p3.13),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p2.2),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024a\)LongBench: a bilingual, multitask benchmark for long context understanding\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2308\.14508,[Link](https://arxiv.org/abs/2308.14508)Cited by:[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px2.p1.3)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024b\)LongBench: a bilingual, multitask benchmark for long context understanding\.External Links:2308\.14508,[Link](https://arxiv.org/abs/2308.14508)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- C\. Chang, W\. Lin, C\. Lin, C\. Chen, Y\. Hu, P\. Wang, N\. Huang, L\. Ceze, M\. S\. Abdelfattah, and K\. Wu \(2024\)Palu: compressing kv\-cache with low\-rank projection\.External Links:2407\.21118,[Link](https://arxiv.org/abs/2407.21118)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.07915#S1.p2.1),[§3\.1](https://arxiv.org/html/2608.07915#S3.SS1.p2.5),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- T\. M\. Cover and J\. A\. Thomas \(2006\)Elements of information theory\.Wiley\-Interscience\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[Appendix D](https://arxiv.org/html/2608.07915#A4.p1.6),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p1.6),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-V2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer \(2022\)LLM\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.External Links:2208\.07339,[Link](https://arxiv.org/abs/2208.07339)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p2.1)\.
- T\. Dettmers and L\. Zettlemoyer \(2023\)The case for 4\-bit precision: k\-bit inference scaling laws\.External Links:2212\.09720,[Link](https://arxiv.org/abs/2212.09720)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p2.1)\.
- C\. Eckart and G\. Young \(1936\)The approximation of one matrix by another of lower rank\.Psychometrika1\(3\),pp\. 211–218\.External Links:[Document](https://dx.doi.org/10.1007/BF02288367)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[Appendix B](https://arxiv.org/html/2608.07915#A2.SS0.SSS0.Px2.p1.7),[§3\.1](https://arxiv.org/html/2608.07915#S3.SS1.p3.6)\.
- A\. Gersho and R\. M\. Gray \(1992\)Vector quantization and signal compression\.Kluwer Academic Publishers\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p1.6)\.
- V\.K\. Goyal \(2001\)Theoretical foundations of transform coding\.IEEE Signal Processing Magazine18\(5\),pp\. 9–21\.External Links:[Document](https://dx.doi.org/10.1109/79.952802)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p2.2),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. Ma \(2024\)The llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px1.p1.1)\.
- R\.M\. Gray and D\.L\. Neuhoff \(1998\)Quantization\.IEEE Transactions on Information Theory44\(6\),pp\. 2325–2383\.External Links:[Document](https://dx.doi.org/10.1109/18.720541)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p1.6)\.
- I\. Han, P\. Kacham, A\. Karbasi, V\. Mirrokni, and A\. Zandieh \(2025\)PolarQuant: quantizing kv caches with polar transformation\.External Links:2502\.02617,[Link](https://arxiv.org/abs/2502.02617)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px1.p1.4),[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2024\)KVQuant: towards 10 million context length LLM inference with KV cache quantization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2608.07915#S1.p2.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.External Links:2404\.06654,[Link](https://arxiv.org/abs/2404.06654)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- J\. Huang and P\. Schultheiss \(1963\)Block quantization of correlated gaussian random variables\.IEEE Transactions on Communications Systems11\(3\),pp\. 289–296\.External Links:[Document](https://dx.doi.org/10.1109/TCOM.1963.1088759)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[Appendix D](https://arxiv.org/html/2608.07915#A4.p1.6),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p1.6)\.
- A\. J\. Izenman \(1975\)Reduced\-rank regression for the multivariate linear model\.Journal of Multivariate Analysis5\(2\),pp\. 248–264\.External Links:ISSN 0047\-259X,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0047-259X%2875%2990042-1),[Link](https://www.sciencedirect.com/science/article/pii/0047259X75900421)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px5.p1.3),[Appendix C](https://arxiv.org/html/2608.07915#A3.SS0.SSS0.Px2.p1.5)\.
- T\. Jiet al\.\(2025\)Towards economical inference: enabling DeepSeek’s multi\-head latent attention in any transformer\-based LLMs\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.External Links:2310\.06770,[Link](https://arxiv.org/abs/2310.06770)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: llm knows what you are looking for before generation\.External Links:2404\.14469,[Link](https://arxiv.org/abs/2404.14469)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.07915#S1.p2.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- S\. Liu, M\. Khadkevich, N\. C\. Fung, C\. Sakr, C\. H\. Yang, C\. Wang, S\. Muralidharan, H\. Yin, K\. Cheng, J\. Kautz, Y\. F\. Wang, P\. Molchanov, and M\. Chen \(2026\)EoRA: fine\-tuning\-free compensation for compressed llm with eigenspace low\-rank approximation\.External Links:2410\.21271,[Link](https://arxiv.org/abs/2410.21271)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. Blankevoort \(2024a\)SpinQuant: LLM quantization with learned rotations\.arXiv preprint arXiv:2405\.16406\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px1.p1.4),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024b\)KIVI: a tuning\-free asymmetric 2bit quantization for KV cache\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px1.p1.4),[§1](https://arxiv.org/html/2608.07915#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- F\. Menget al\.\(2025\)TransMLA: multi\-head latent attention is all you need\.arXiv preprint arXiv:2502\.07864\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px1.p1.1)\.
- U\. Saxena, G\. Saha, S\. Choudhary, and K\. Roy \(2024\)Eigen attention: attention in low\-rank space for kv cache compression\.External Links:2408\.05646,[Link](https://arxiv.org/abs/2408.05646)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p2.1)\.
- T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom \(2023\)Toolformer: language models can teach themselves to use tools\.External Links:2302\.04761,[Link](https://arxiv.org/abs/2302.04761)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- Y\. Su, Y\. Zhou, Q\. Qiu, J\. Li, Q\. Xia, P\. Li, X\. Duan, Z\. Wang, and M\. Zhang \(2025a\)Accurate kv cache quantization with outlier tokens tracing\.External Links:2505\.10938,[Link](https://arxiv.org/abs/2505.10938)Cited by:[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px4.p1.1)\.
- Z\. Su, Z\. Chen, W\. Shen, H\. Wei, L\. Li, H\. Yu, and K\. Yuan \(2025b\)RotateKV: accurate and robust 2\-bit kv cache quantization for llms via outlier\-aware adaptive rotations\.External Links:2501\.16383,[Link](https://arxiv.org/abs/2501.16383)Cited by:[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px4.p1.1)\.
- G\. Team, P\. Georgiev, V\. I\. Lei, R\. Burnell, L\. Bai, A\. Gulati, G\. Tanzer, D\. Vincent, Z\. Pan, S\. Wang, S\. Mariooryad, Y\. Ding, X\. Geng, F\. Alcober, R\. Frostig, M\. Omernick, L\. Walker, C\. Paduraru, C\. Sorokin, A\. Tacchetti, C\. Gaffney, S\. Daruki, O\. Sercinoglu, Z\. Gleicher, J\. Love, P\. Voigtlaender, R\. Jain, G\. Surita, K\. Mohamed, R\. Blevins, J\. Ahn, T\. Zhu, K\. Kawintiranon, O\. Firat, Y\. Gu, Y\. Zhang, M\. Rahtz, M\. Faruqui, N\. Clay, J\. Gilmer, J\. Co\-Reyes, I\. Penchev, R\. Zhu, N\. Morioka, K\. Hui, K\. Haridasan, V\. Campos, M\. Mahdieh, M\. Guo, S\. Hassan, K\. Kilgour, A\. Vezer, H\. Cheng, R\. de Liedekerke, S\. Goyal, P\. Barham, D\. Strouse, S\. Noury, J\. Adler, M\. Sundararajan, S\. Vikram, D\. Lepikhin, M\. Paganini, X\. Garcia, F\. Yang, D\. Valter, M\. Trebacz, K\. Vodrahalli, C\. Asawaroengchai, R\. Ring, N\. Kalb, L\. B\. Soares, S\. Brahma, D\. Steiner, T\. Yu, F\. Mentzer, A\. He, L\. Gonzalez, B\. Xu, R\. L\. Kaufman, L\. E\. Shafey, J\. Oh, T\. Hennigan, G\. van den Driessche, S\. Odoom, M\. Lucic, B\. Roelofs, S\. Lall, A\. Marathe, B\. Chan, S\. Ontanon, L\. He, D\. Teplyashin, J\. Lai, P\. Crone, B\. Damoc, L\. Ho, S\. Riedel, K\. Lenc, C\. Yeh, A\. Chowdhery, Y\. Xu, M\. Kazemi, E\. Amid, A\. Petrushkina, K\. Swersky, A\. Khodaei, G\. Chen, C\. Larkin, M\. Pinto, G\. Yan, A\. P\. Badia, P\. Patil, S\. Hansen, D\. Orr, S\. M\. R\. Arnold, J\. Grimstad, A\. Dai, S\. Douglas, R\. Sinha, V\. Yadav, X\. Chen, E\. Gribovskaya, J\. Austin, J\. Zhao, K\. Patel, P\. Komarek, S\. Austin, S\. Borgeaud, L\. Friso, A\. Goyal, B\. Caine, K\. Cao, D\. Chung, M\. Lamm, G\. Barth\-Maron, T\. Kagohara, K\. Olszewska, M\. Chen, K\. Shivakumar, R\. Agarwal, H\. Godhia, R\. Rajwar, J\. Snaider, X\. Dotiwalla, Y\. Liu, A\. Barua, V\. Ungureanu, Y\. Zhang, B\. Batsaikhan, M\. Wirth, J\. Qin, I\. Danihelka, T\. Doshi, M\. Chadwick, J\. Chen, S\. Jain, Q\. Le, A\. Kar, M\. Gurumurthy, C\. Li, R\. Sang, F\. Liu, L\. Lamprou, R\. Munoz, N\. Lintz, H\. Mehta, H\. Howard, M\. Reynolds, L\. Aroyo, Q\. Wang, L\. Blanco, A\. Cassirer, J\. Griffith, D\. Das, S\. Lee, J\. Sygnowski, Z\. Fisher, J\. Besley, R\. Powell, Z\. Ahmed, D\. Paulus, D\. Reitter, Z\. Borsos, R\. Joshi, A\. Pope, S\. Hand, V\. Selo, V\. Jain, N\. Sethi, M\. Goel, T\. Makino, R\. May, Z\. Yang, J\. Schalkwyk, C\. Butterfield, A\. Hauth, A\. Goldin, W\. Hawkins, E\. Senter, S\. Brin, O\. Woodman, M\. Ritter, E\. Noland, M\. Giang, V\. Bolina, L\. Lee, T\. Blyth, I\. Mackinnon, M\. Reid, O\. Sarvana, D\. Silver, A\. Chen, L\. Wang, L\. Maggiore, O\. Chang, N\. Attaluri, G\. Thornton, C\. Chiu, O\. Bunyan, N\. Levine, T\. Chung, E\. Eltyshev, X\. Si, T\. Lillicrap, D\. Brady, V\. Aggarwal, B\. Wu, Y\. Xu, R\. McIlroy, K\. Badola, P\. Sandhu, E\. Moreira, W\. Stokowiec, R\. Hemsley, D\. Li, A\. Tudor, P\. Shyam, E\. Rahimtoroghi, S\. Haykal, P\. Sprechmann, X\. Zhou, D\. Mincu, Y\. Li, R\. Addanki, K\. Krishna, X\. Wu, A\. Frechette, M\. Eyal, A\. Dafoe, D\. Lacey, J\. Whang, T\. Avrahami, Y\. Zhang, E\. Taropa, H\. Lin, D\. Toyama, E\. Rutherford, M\. Sano, H\. Choe, A\. Tomala, C\. Safranek\-Shrader, N\. Kassner, M\. Pajarskas, M\. Harvey, S\. Sechrist, M\. Fortunato, C\. Lyu, G\. Elsayed, C\. Kuang, J\. Lottes, E\. Chu, C\. Jia, C\. Chen, P\. Humphreys, K\. Baumli, C\. Tao, R\. Samuel, C\. N\. dos Santos, A\. Andreassen, N\. Rakićević, D\. Grewe, A\. Kumar, S\. Winkler, J\. Caton, A\. Brock, S\. Dalmia, H\. Sheahan, I\. Barr, Y\. Miao, P\. Natsev, J\. Devlin, F\. Behbahani, F\. Prost, Y\. Sun, A\. Myaskovsky, T\. S\. Pillai, D\. Hurt, A\. Lazaridou, X\. Xiong, C\. Zheng, F\. Pardo, X\. Li, D\. Horgan, J\. Stanton, M\. Ambar, F\. Xia, A\. Lince, M\. Wang, B\. Mustafa, A\. Webson, H\. Lee, R\. Anil, M\. Wicke, T\. Dozat, A\. Sinha, E\. Piqueras, E\. Dabir, S\. Upadhyay, A\. Boral, L\. A\. Hendricks, C\. Fry, J\. Djolonga, Y\. Su, J\. Walker, J\. Labanowski, R\. Huang, V\. Misra, J\. Chen, R\. Skerry\-Ryan, A\. Singh, S\. Rijhwani, D\. Yu, A\. Castro\-Ros, B\. Changpinyo, R\. Datta, S\. Bagri, A\. M\. Hrafnkelsson, M\. Maggioni, D\. Zheng, Y\. Sulsky, S\. Hou, T\. L\. Paine, A\. Yang, J\. Riesa, D\. Rogozinska, D\. Marcus, D\. E\. Badawy, Q\. Zhang, L\. Wang, H\. Miller, J\. Greer, L\. L\. Sjos, A\. Nova, H\. Zen, R\. Chaabouni, M\. Rosca, J\. Jiang, C\. Chen, R\. Liu, T\. Sainath, M\. Krikun, A\. Polozov, J\. Lespiau, J\. Newlan, Z\. Cankara, S\. Kwak, Y\. Xu, P\. Chen, A\. Coenen, C\. Meyer, K\. Tsihlas, A\. Ma, J\. Gottweis, J\. Xing, C\. Gu, J\. Miao, C\. Frank, Z\. Cankara, S\. Ganapathy, I\. Dasgupta, S\. Hughes\-Fitt, H\. Chen, D\. Reid, K\. Rong, H\. Fan, J\. van Amersfoort, V\. Zhuang, A\. Cohen, S\. S\. Gu, A\. Mohananey, A\. Ilic, T\. Tobin, J\. Wieting, A\. Bortsova, P\. Thacker, E\. Wang, E\. Caveness, J\. Chiu, E\. Sezener, A\. Kaskasoli, S\. Baker, K\. Millican, M\. Elhawaty, K\. Aisopos, C\. Lebsack, N\. Byrd, H\. Dai, W\. Jia, M\. Wiethoff, E\. Davoodi, A\. Weston, L\. Yagati, A\. Ahuja, I\. Gao, G\. Pundak, S\. Zhang, M\. Azzam, K\. C\. Sim, S\. Caelles, J\. Keeling, A\. Sharma, A\. Swing, Y\. Li, C\. Liu, C\. G\. Bostock, Y\. Bansal, Z\. Nado, A\. Anand, J\. Lipschultz, A\. Karmarkar, L\. Proleev, A\. Ittycheriah, S\. H\. Yeganeh, G\. Polovets, A\. Faust, J\. Sun, A\. Rrustemi, P\. Li, R\. Shivanna, J\. Liu, C\. Welty, F\. Lebron, A\. Baddepudi, S\. Krause, E\. Parisotto, R\. Soricut, Z\. Xu, D\. Bloxwich, M\. Johnson, B\. Neyshabur, J\. Mao\-Jones, R\. Wang, V\. Ramasesh, Z\. Abbas, A\. Guez, C\. Segal, D\. D\. Nguyen, J\. Svensson, L\. Hou, S\. York, K\. Milan, S\. Bridgers, W\. Gworek, M\. Tagliasacchi, J\. Lee\-Thorp, M\. Chang, A\. Guseynov, A\. J\. Hartman, M\. Kwong, R\. Zhao, S\. Kashem, E\. Cole, A\. Miech, R\. Tanburn, M\. Phuong, F\. Pavetic, S\. Cevey, R\. Comanescu, R\. Ives, S\. Yang, C\. Du, B\. Li, Z\. Zhang, M\. Iinuma, C\. H\. Hu, A\. Roy, S\. Bijwadia, Z\. Zhu, D\. Martins, R\. Saputro, A\. Gergely, S\. Zheng, D\. Jia, I\. Antonoglou, A\. Sadovsky, S\. Gu, Y\. Bi, A\. Andreev, S\. Samangooei, M\. Khan, T\. Kocisky, A\. Filos, C\. Kumar, C\. Bishop, A\. Yu, S\. Hodkinson, S\. Mittal, P\. Shah, A\. Moufarek, Y\. Cheng, A\. Bloniarz, J\. Lee, P\. Pejman, P\. Michel, S\. Spencer, V\. Feinberg, X\. Xiong, N\. Savinov, C\. Smith, S\. Shakeri, D\. Tran, M\. Chesus, B\. Bohnet, G\. Tucker, T\. von Glehn, C\. Muir, Y\. Mao, H\. Kazawa, A\. Slone, K\. Soparkar, D\. Shrivastava, J\. Cobon\-Kerr, M\. Sharman, J\. Pavagadhi, C\. Araya, K\. Misiunas, N\. Ghelani, M\. Laskin, D\. Barker, Q\. Li, A\. Briukhov, N\. Houlsby, M\. Glaese, B\. Lakshminarayanan, N\. Schucher, Y\. Tang, E\. Collins, H\. Lim, F\. Feng, A\. Recasens, G\. Lai, A\. Magni, N\. D\. Cao, A\. Siddhant, Z\. Ashwood, J\. Orbay, M\. Dehghani, J\. Brennan, Y\. He, K\. Xu, Y\. Gao, C\. Saroufim, J\. Molloy, X\. Wu, S\. Arnold, S\. Chang, J\. Schrittwieser, E\. Buchatskaya, S\. Radpour, M\. Polacek, S\. Giordano, A\. Bapna, S\. Tokumine, V\. Hellendoorn, T\. Sottiaux, S\. Cogan, A\. Severyn, M\. Saleh, S\. Thakoor, L\. Shefey, S\. Qiao, M\. Gaba, S\. Chang, C\. Swanson, B\. Zhang, B\. Lee, P\. K\. Rubenstein, G\. Song, T\. Kwiatkowski, A\. Koop, A\. Kannan, D\. Kao, P\. Schuh, A\. Stjerngren, G\. Ghiasi, G\. Gibson, L\. Vilnis, Y\. Yuan, F\. T\. Ferreira, A\. Kamath, T\. Klimenko, K\. Franko, K\. Xiao, I\. Bhattacharya, M\. Patel, R\. Wang, A\. Morris, R\. Strudel, V\. Sharma, P\. Choy, S\. H\. Hashemi, J\. Landon, M\. Finkelstein, P\. Jhakra, J\. Frye, M\. Barnes, M\. Mauger, D\. Daun, K\. Baatarsukh, M\. Tung, W\. Farhan, H\. Michalewski, F\. Viola, F\. de Chaumont Quitry, C\. L\. Lan, T\. Hudson, Q\. Wang, F\. Fischer, I\. Zheng, E\. White, A\. Dragan, J\. Alayrac, E\. Ni, A\. Pritzel, A\. Iwanicki, M\. Isard, A\. Bulanova, L\. Zilka, E\. Dyer, D\. Sachan, S\. Srinivasan, H\. Muckenhirn, H\. Cai, A\. Mandhane, M\. Tariq, J\. W\. Rae, G\. Wang, K\. Ayoub, N\. FitzGerald, Y\. Zhao, W\. Han, C\. Alberti, D\. Garrette, K\. Krishnakumar, M\. Gimenez, A\. Levskaya, D\. Sohn, J\. Matak, I\. Iturrate, M\. B\. Chang, J\. Xiang, Y\. Cao, N\. Ranka, G\. Brown, A\. Hutter, V\. Mirrokni, N\. Chen, K\. Yao, Z\. Egyed, F\. Galilee, T\. Liechty, P\. Kallakuri, E\. Palmer, S\. Ghemawat, J\. Liu, D\. Tao, C\. Thornton, T\. Green, M\. Jasarevic, S\. Lin, V\. Cotruta, Y\. Tan, N\. Fiedel, H\. Yu, E\. Chi, A\. Neitz, J\. Heitkaemper, A\. Sinha, D\. Zhou, Y\. Sun, C\. Kaed, B\. Hulse, S\. Mishra, M\. Georgaki, S\. Kudugunta, C\. Farabet, I\. Shafran, D\. Vlasic, A\. Tsitsulin, R\. Ananthanarayanan, A\. Carin, G\. Su, P\. Sun, S\. V, G\. Carvajal, J\. Broder, I\. Comsa, A\. Repina, W\. Wong, W\. W\. Chen, P\. Hawkins, E\. Filonov, L\. Loher, C\. Hirnschall, W\. Wang, J\. Ye, A\. Burns, H\. Cate, D\. G\. Wright, F\. Piccinini, L\. Zhang, C\. Lin, I\. Gog, Y\. Kulizhskaya, A\. Sreevatsa, S\. Song, L\. C\. Cobo, A\. Iyer, C\. Tekur, G\. Garrido, Z\. Xiao, R\. Kemp, H\. S\. Zheng, H\. Li, A\. Agarwal, C\. Ngani, K\. Goshvadi, R\. Santamaria\-Fernandez, W\. Fica, X\. Chen, C\. Gorgolewski, S\. Sun, R\. Garg, X\. Ye, S\. M\. A\. Eslami, N\. Hua, J\. Simon, P\. Joshi, Y\. Kim, I\. Tenney, S\. Potluri, L\. N\. Thiet, Q\. Yuan, F\. Luisier, A\. Chronopoulou, S\. Scellato, P\. Srinivasan, M\. Chen, V\. Koverkathu, V\. Dalibard, Y\. Xu, B\. Saeta, K\. Anderson, T\. Sellam, N\. Fernando, F\. Huot, J\. Jung, M\. Varadarajan, M\. Quinn, A\. Raul, M\. Le, R\. Habalov, J\. Clark, K\. Jalan, K\. Bullard, A\. Singhal, T\. Luong, B\. Wang, S\. Rajayogam, J\. Eisenschlos, J\. Jia, D\. Finchelstein, A\. Yakubovich, D\. Balle, M\. Fink, S\. Agarwal, J\. Li, D\. Dvijotham, S\. Pal, K\. Kang, J\. Konzelmann, J\. Beattie, O\. Dousse, D\. Wu, R\. Crocker, C\. Elkind, S\. R\. Jonnalagadda, J\. Lee, D\. Holtmann\-Rice, K\. Kallarackal, R\. Liu, D\. Vnukov, N\. Vats, L\. Invernizzi, M\. Jafari, H\. Zhou, L\. Taylor, J\. Prendki, M\. Wu, T\. Eccles, T\. Liu, K\. Kopparapu, F\. Beaufays, C\. Angermueller, A\. Marzoca, S\. Sarcar, H\. Dib, J\. Stanway, F\. Perbet, N\. Trdin, R\. Sterneck, A\. Khorlin, D\. Li, X\. Wu, S\. Goenka, D\. Madras, S\. Goldshtein, W\. Gierke, T\. Zhou, Y\. Liu, Y\. Liang, A\. White, Y\. Li, S\. Singh, S\. Bahargam, M\. Epstein, S\. Basu, L\. Lao, A\. Ozturel, C\. Crous, A\. Zhai, H\. Lu, Z\. Tung, N\. Gaur, A\. Walton, L\. Dixon, M\. Zhang, A\. Globerson, G\. Uy, A\. Bolt, O\. Wiles, M\. Nasr, I\. Shumailov, M\. Selvi, F\. Piccinno, R\. Aguilar, S\. McCarthy, M\. Khalman, M\. Shukla, V\. Galic, J\. Carpenter, K\. Villela, H\. Zhang, H\. Richardson, J\. Martens, M\. Bosnjak, S\. R\. Belle, J\. Seibert, M\. Alnahlawi, B\. McWilliams, S\. Singh, A\. Louis, W\. Ding, D\. Popovici, L\. Simicich, L\. Knight, P\. Mehta, N\. Gupta, C\. Shi, S\. Fatehi, J\. Mitrovic, A\. Grills, J\. Pagadora, T\. Munkhdalai, D\. Petrova, D\. Eisenbud, Z\. Zhang, D\. Yates, B\. Mittal, N\. Tripuraneni, Y\. Assael, T\. Brovelli, P\. Jain, M\. Velimirovic, C\. Akbulut, J\. Mu, W\. Macherey, R\. Kumar, J\. Xu, H\. Qureshi, G\. Comanici, J\. Wiesner, Z\. Gong, A\. Ruddock, M\. Bauer, N\. Felt, A\. GP, A\. Arnab, D\. Zelle, J\. Rothfuss, B\. Rosgen, A\. Shenoy, B\. Seybold, X\. Li, J\. Mudigonda, G\. Erdogan, J\. Xia, J\. Simsa, A\. Michi, Y\. Yao, C\. Yew, S\. Kan, I\. Caswell, C\. Radebaugh, A\. Elisseeff, P\. Valenzuela, K\. McKinney, K\. Paterson, A\. Cui, E\. Latorre\-Chimoto, S\. Kim, W\. Zeng, K\. Durden, P\. Ponnapalli, T\. Sosea, C\. A\. Choquette\-Choo, J\. Manyika, B\. Robenek, H\. Vashisht, S\. Pereira, H\. Lam, M\. Velic, D\. Owusu\-Afriyie, K\. Lee, T\. Bolukbasi, A\. Parrish, S\. Lu, J\. Park, B\. Venkatraman, A\. Talbert, L\. Rosique, Y\. Cheng, A\. Sozanschi, A\. Paszke, P\. Kumar, J\. Austin, L\. Li, K\. Salama, B\. Perz, W\. Kim, N\. Dukkipati, A\. Baryshnikov, C\. Kaplanis, X\. Sheng, Y\. Chervonyi, C\. Unlu, D\. de Las Casas, H\. Askham, K\. Tunyasuvunakool, F\. Gimeno, S\. Poder, C\. Kwak, M\. Miecnikowski, V\. Mirrokni, A\. Dimitriev, A\. Parisi, D\. Liu, T\. Tsai, T\. Shevlane, C\. Kouridi, D\. Garmon, A\. Goedeckemeyer, A\. R\. Brown, A\. Vijayakumar, A\. Elqursh, S\. Jazayeri, J\. Huang, S\. M\. Carthy, J\. Hoover, L\. Kim, S\. Kumar, W\. Chen, C\. Biles, G\. Bingham, E\. Rosen, L\. Wang, Q\. Tan, D\. Engel, F\. Pongetti, D\. de Cesare, D\. Hwang, L\. Yu, J\. Pullman, S\. Narayanan, K\. Levin, S\. Gopal, M\. Li, A\. Aharoni, T\. Trinh, J\. Lo, N\. Casagrande, R\. Vij, L\. Matthey, B\. Ramadhana, A\. Matthews, C\. Carey, M\. Johnson, K\. Goranova, R\. Shah, S\. Ashraf, K\. Dasgupta, R\. Larsen, Y\. Wang, M\. R\. Vuyyuru, C\. Jiang, J\. Ijazi, K\. Osawa, C\. Smith, R\. S\. Boppana, T\. Bilal, Y\. Koizumi, Y\. Xu, Y\. Altun, N\. Shabat, B\. Bariach, A\. Korchemniy, K\. Choo, O\. Ronneberger, C\. Iwuanyanwu, S\. Zhao, D\. Soergel, C\. Hsieh, I\. Cai, S\. Iqbal, M\. Sundermeyer, Z\. Chen, E\. Bursztein, C\. Malaviya, F\. Biadsy, P\. Shroff, I\. Dhillon, T\. Latkar, C\. Dyer, H\. Forbes, M\. Nicosia, V\. Nikolaev, S\. Greene, M\. Georgiev, P\. Wang, N\. Martin, H\. Sedghi, J\. Zhang, P\. Banzal, D\. Fritz, V\. Rao, X\. Wang, J\. Zhang, V\. Patraucean, D\. Du, I\. Mordatch, I\. Jurin, L\. Liu, A\. Dubey, A\. Mohan, J\. Nowakowski, V\. Ion, N\. Wei, R\. Tojo, M\. A\. Raad, D\. A\. Hudson, V\. Keshava, S\. Agrawal, K\. Ramirez, Z\. Wu, H\. Nguyen, J\. Liu, M\. Sewak, B\. Petrini, D\. Choi, I\. Philips, Z\. Wang, I\. Bica, A\. Garg, J\. Wilkiewicz, P\. Agrawal, X\. Li, D\. Guo, E\. Xue, N\. Shaik, A\. Leach, S\. M\. Khan, J\. Wiesinger, S\. Jerome, A\. Chakladar, A\. W\. Wang, T\. Ornduff, F\. Abu, A\. Ghaffarkhah, M\. Wainwright, M\. Cortes, F\. Liu, J\. Maynez, A\. Terzis, P\. Samangouei, R\. Mansour, T\. Kępa, F\. Aubet, A\. Algymr, D\. Banica, A\. Weisz, A\. Orban, A\. Senges, E\. Andrejczuk, M\. Geller, N\. D\. Santo, V\. Anklin, M\. A\. Merey, M\. Baeuml, T\. Strohman, J\. Bai, S\. Petrov, Y\. Wu, D\. Hassabis, K\. Kavukcuoglu, J\. Dean, and O\. Vinyals \(2024\)Gemini 1\.5: unlocking multimodal understanding across millions of tokens of context\.External Links:2403\.05530,[Link](https://arxiv.org/abs/2403.05530)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- X\. Wang, Y\. Zheng, Z\. Wan, and M\. Zhang \(2024\)SVD\-LLM: truncation\-aware singular value decomposition for large language model compression\.arXiv preprint arXiv:2403\.07378\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2608.07915#S3.SS1.p2.5),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. Han \(2024a\)SmoothQuant: accurate and efficient post\-training quantization for large language models\.External Links:2211\.10438,[Link](https://arxiv.org/abs/2211.10438)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p2.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024b\)Efficient streaming language models with attention sinks\.International Conference on Learning Representations \(ICLR\)\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.07915#S1.p2.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.External Links:2405\.15793,[Link](https://arxiv.org/abs/2405.15793)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- A\. Zandieh, M\. Daliri, M\. Hadian, and V\. Mirrokni \(2025\)TurboQuant: online vector quantization with near\-optimal distortion rate\.External Links:2504\.19874,[Link](https://arxiv.org/abs/2504.19874)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px1.p1.4),[§3\.1](https://arxiv.org/html/2608.07915#S3.SS1.p3.13),[§3\.2](https://arxiv.org/html/2608.07915#S3.SS2.p2.2),[§4\.1](https://arxiv.org/html/2608.07915#S4.SS1.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- R\. Zhanget al\.\(2024\)LoRC: low\-rank compression for LLMs KV cache with a progressive compression strategy\.arXiv preprint arXiv:2410\.03111\.Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2608.07915#S5.p2.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.External Links:2306\.14048,[Link](https://arxiv.org/abs/2306.14048)Cited by:[Appendix A](https://arxiv.org/html/2608.07915#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.07915#S1.p2.1),[§5](https://arxiv.org/html/2608.07915#S5.p1.3)\.
- L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez, C\. Barrett, and Y\. Sheng \(2024\)SGLang: efficient execution of structured language model programs\.External Links:2312\.07104,[Link](https://arxiv.org/abs/2312.07104)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p1.1)\.
- C\. Zhou, P\. Cao, J\. Li, B\. Yu, J\. Ye, J\. Zhao, and K\. Liu \(2026\)From signal degradation to computation collapse: uncovering the two failure modes of llm quantization\.External Links:2604\.19884,[Link](https://arxiv.org/abs/2604.19884)Cited by:[§1](https://arxiv.org/html/2608.07915#S1.p2.1)\.

## Appendix AExtended Related Work

#### KV\-cache quantization\.

The dominant approach to shrinking the cache lowers the bit\-width of the stored keys and values\. KIVI\(Liuet al\.,[2024b](https://arxiv.org/html/2608.07915#bib.bib1)\)quantizes keys per\-channel and values per\-token to22bits; KVQuant\(Hooperet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib2)\)adds sensitivity\-aware non\-uniform quantization and outlier handling\. A second line first*rotates*the cache to spread outliers before quantizing, using random or learned orthogonal transforms \(QuaRot\(Ashkbooset al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib17)\), SpinQuant\(Liuet al\.,[2024a](https://arxiv.org/html/2608.07915#bib.bib3)\), TurboQuant\(Zandiehet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib15)\)\) or a polar reparameterization \(PolarQuant\(Hanet al\.,[2025](https://arxiv.org/html/2608.07915#bib.bib16)\)\)\. All of these keep the cached object fixed to rawK,VK,Vand spend bits uniformly \(or by heuristic importance\) over its channels; quality degrades sharply near22bits \(the “22\-bit cliff”\)\.Spectrainstead quantizes a second\-moment\-orthogonal, energy\-ordered latent, where the measured marginal variances provide a canonical axis for rate allocation and low\-rank truncation carries compression that scalar quantizers can only reach at INT2\.

#### Token eviction and sparse attention\.

A complementary family reduces the*number*of cached tokens rather than the bits per token, keeping a budget of salient entries\. H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2608.07915#bib.bib19)\)evicts by accumulated attention \(heavy hitters\), SnapKV\(Liet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib18)\)selects tokens using a lookahead window, and StreamingLLM\(Xiaoet al\.,[2024b](https://arxiv.org/html/2608.07915#bib.bib4)\)retains attention sinks plus a sliding window\. These methods are lossy on aggregation and multi\-fact queries, where the discarded tokens are exactly the evidence a later query needs\.Spectrais orthogonal: it compresses*every*token’s representation rather than dropping tokens, and could be composed with eviction\.

#### Low\-rank and latent KV compression\.

Closest to us are methods that store a low\-rank projection of the cache\. Palu\(Changet al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib20)\)caches a low\-rank latent obtained from an SVD of the key/value projection weights; LoRC\(Zhang and others,[2024](https://arxiv.org/html/2608.07915#bib.bib8)\)applies a progressive low\-rank compression across layers; SVD\-LLM\(Wanget al\.,[2024](https://arxiv.org/html/2608.07915#bib.bib9)\)and EoRA\(Liuet al\.,[2026](https://arxiv.org/html/2608.07915#bib.bib21)\)use \(truncation\- or eigenspace\-aware\) low\-rank factorization for LLM weight compression\. The key difference is the*metric*: these factorize in weight space, ranking directions by weight energy, whereasSpectraminimizes error in the metric of the activations the model actually produces \(GG\-weighted / KLT\-like\), which we show retains substantially more produced energy at equal rank and, unlike a pure whitening or random rotation, yields a latent that is both second\-moment\-orthogonal and energy\-ordered\.

#### Latent attention as an architecture\.

Multi\-head latent attention \(MLA\) in DeepSeek\-V2\(DeepSeek\-AI,[2024](https://arxiv.org/html/2608.07915#bib.bib5)\)caches a compressed latent by design, and recent work retrofits existing models to MLA via architectural conversion and continued training/distillation \(TransMLA\(Meng and others,[2025](https://arxiv.org/html/2608.07915#bib.bib6)\), MHA2MLA\(Ji and others,[2025](https://arxiv.org/html/2608.07915#bib.bib7)\)\)\.Spectratargets the same latent\-cache idea but as a*post\-hoc, training\-free*transform on an already\-trained model: no pretraining, fine\-tuning, or distillation, and a drop\-in replacement of the key/value projections\.

#### Transform coding and rate–distortion foundations\.

Spectrafollows the classical transform\-coding pattern\(Goyal,[2001](https://arxiv.org/html/2608.07915#bib.bib25); Gray and Neuhoff,[1998](https://arxiv.org/html/2608.07915#bib.bib23)\): a data\-dependent transform followed by bit allocation over its coefficients\. The classical KLT diagonalizes the centered covariance of a source; ourGG\-weighted transform instead diagonalizes the uncentered latent second moment, after which we center the coefficients and allocate using their measured marginal variances\. Reverse water\-filling is rate–distortion optimal for parallel Gaussian sources\(Huang and Schultheiss,[1963](https://arxiv.org/html/2608.07915#bib.bib24); Cover and Thomas,[2006](https://arxiv.org/html/2608.07915#bib.bib10); Gersho and Gray,[1992](https://arxiv.org/html/2608.07915#bib.bib11)\); the optimal low\-rank factor in a weighted metric follows from Eckart–Young\(Eckart and Young,[1936](https://arxiv.org/html/2608.07915#bib.bib22)\), and our output\-side correction is a reduced\-rank regression\(Izenman,[1975](https://arxiv.org/html/2608.07915#bib.bib26)\)\. Our contribution is not these tools but their application to the KV cache: identifyingGG\(the activation Gram matrix\) as the right metric, and showing that transform coding in that metric turns the ill\-posed problem of quantizing rawK,VK,Vinto a well\-posed one\.

## Appendix BG\-Weighted SVD

#### Activation\-metric identity\.

The same calculation underlies the objective of Section[3\.1](https://arxiv.org/html/2608.07915#S3.SS1)\. For any matrixΔ​𝐖\\Delta\\mathbf\{W\}\(e\.g\. the factorization residual𝐖−𝐖down​𝐖up\\mathbf\{W\}\-\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}\), the expected error on produced activations is, with𝐡\\mathbf\{h\}a row vector,

𝔼𝐡​‖𝐡​Δ​𝐖‖2\\displaystyle\\mathbb\{E\}\_\{\\mathbf\{h\}\}\\\|\\mathbf\{h\}\\,\\Delta\\mathbf\{W\}\\\|^\{2\}=𝔼𝐡​tr​\(Δ​𝐖⊤​𝐡⊤​𝐡​Δ​𝐖\)\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{h\}\}\\,\\mathrm\{tr\}\\\!\\big\(\\Delta\\mathbf\{W\}^\{\\top\}\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\\,\\Delta\\mathbf\{W\}\\big\)\(6\)=tr​\(Δ​𝐖⊤​𝔼𝐡​\[𝐡⊤​𝐡\]​Δ​𝐖\)\\displaystyle=\\mathrm\{tr\}\\\!\\big\(\\Delta\\mathbf\{W\}^\{\\top\}\\,\\mathbb\{E\}\_\{\\mathbf\{h\}\}\[\\mathbf\{h\}^\{\\top\}\\mathbf\{h\}\]\\,\\Delta\\mathbf\{W\}\\big\)=tr​\(Δ​𝐖⊤​𝐆​Δ​𝐖\)\\displaystyle=\\mathrm\{tr\}\\\!\\big\(\\Delta\\mathbf\{W\}^\{\\top\}\\mathbf\{G\}\\,\\Delta\\mathbf\{W\}\\big\)=‖𝐆1/2​Δ​𝐖‖F2,\\displaystyle=\\big\\\|\\mathbf\{G\}^\{1/2\}\\Delta\\mathbf\{W\}\\big\\\|\_\{F\}^\{2\},using linearity of the trace and expectation and the symmetry𝐆1/2⊤​𝐆1/2=𝐆\\mathbf\{G\}^\{1/2\\top\}\\mathbf\{G\}^\{1/2\}=\\mathbf\{G\}\. Minimizing this over rank\-rr𝐖down​𝐖up\\mathbf\{W\}\_\{\\text\{down\}\}\\mathbf\{W\}\_\{\\text\{up\}\}is thus a plain low\-rank problem on the whitened weight𝐆1/2​𝐖\\mathbf\{G\}^\{1/2\}\\mathbf\{W\}, solved in closed form by Equation[3](https://arxiv.org/html/2608.07915#S3.E3)\.

#### GG\-weighted \(ours\)\.

Let𝐌=𝐆1/2​𝐖=𝐔​𝚺​𝐕⊤\\mathbf\{M\}=\\mathbf\{G\}^\{1/2\}\\mathbf\{W\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\}\. Truncating𝐌\\mathbf\{M\}to rankrris optimal in produced energy \(Eckart–Young\(Eckart and Young,[1936](https://arxiv.org/html/2608.07915#bib.bib22)\)in theGG\-metric\), so the retained fraction is∑j≤rσj2/∑jσj2\\sum\_\{j\\leq r\}\\sigma\_\{j\}^\{2\}/\\sum\_\{j\}\\sigma\_\{j\}^\{2\}, whereσj\\sigma\_\{j\}are the singular values of𝐌\\mathbf\{M\}\.

#### Plain SVD \(Palu\)\.

Let𝐖=𝐔w​𝚺w​𝐕w⊤\\mathbf\{W\}=\\mathbf\{U\}\_\{w\}\\mathbf\{\\Sigma\}\_\{w\}\\mathbf\{V\}\_\{w\}^\{\\top\}be the SVD of the weight itself, and keep its top\-rrleft singular directions\. Scored in the*same*produced\-energy metric, the retained fraction is

∑j≤rσw,j2​\(𝐮w,j⊤​𝐆​𝐮w,j\)‖𝐆1/2​𝐖‖F2,\\frac\{\\sum\_\{j\\leq r\}\\sigma\_\{w,j\}^\{2\}\\,\\big\(\\mathbf\{u\}\_\{w,j\}^\{\\top\}\\mathbf\{G\}\\,\\mathbf\{u\}\_\{w,j\}\\big\)\}\{\\\|\\mathbf\{G\}^\{1/2\}\\mathbf\{W\}\\\|\_\{F\}^\{2\}\},\(7\)which follows from‖𝐆1/2​𝐖r‖F2=∑j≤rσw,j2​\(𝐮w,j⊤​𝐆​𝐮w,j\)\\\|\\mathbf\{G\}^\{1/2\}\\mathbf\{W\}\_\{r\}\\\|\_\{F\}^\{2\}=\\sum\_\{j\\leq r\}\\sigma\_\{w,j\}^\{2\}\\,\(\\mathbf\{u\}\_\{w,j\}^\{\\top\}\\mathbf\{G\}\\,\\mathbf\{u\}\_\{w,j\}\)since the right singular vectors are orthonormal\. Because this basis orders directions by weight energyσw,j2\\sigma\_\{w,j\}^\{2\}rather than by produced energy, it cannot retain more produced energy than theGG\-weighted optimum at equal rank and generally retains less\.

#### Setup\.

𝐆\\mathbf\{G\}\(and the mean𝐡¯\\bar\{\\mathbf\{h\}\}\) are accumulated in fp64 over2424calibration windows of length20482048from WikiText\-2 train, via a forward hook on each layer’s attention input; curves are averaged over all layers\.

## Appendix CClosed\-form𝐖O\\mathbf\{W\}\_\{O\}healing

The low\-rank factorization of the value path perturbs the attention output, and the error reaches the residual stream through the output projection𝐖O\\mathbf\{W\}\_\{O\}\. Because𝐖O\\mathbf\{W\}\_\{O\}is not stored in the cache, we can compensate the dominant, structured part of this error by adding a small correctionΔ​𝐖\\Delta\\mathbf\{W\}to𝐖O\\mathbf\{W\}\_\{O\}, at no inference cost\. Crucially, the correction is fit on the*low\-rank but un\-quantized*model, so it heals the low\-rank error, not the quantization noise \(which is handled separately in Section[3\.2](https://arxiv.org/html/2608.07915#S3.SS2)\)\.

#### Objective\.

Let𝐘\\mathbf\{Y\}be the input to𝐖O\\mathbf\{W\}\_\{O\}\(the post\-attention context\) under the low\-rank model andΔ​𝐎=𝐎orig−𝐎lowrank\\Delta\\mathbf\{O\}=\\mathbf\{O\}\_\{\\text\{orig\}\}\-\\mathbf\{O\}\_\{\\text\{lowrank\}\}the resulting output error, collected over calibration tokens\. We want𝐘​\(𝐖O\+Δ​𝐖\)≈𝐘𝐖O\+Δ​𝐎\\mathbf\{Y\}\(\\mathbf\{W\}\_\{O\}\+\\Delta\\mathbf\{W\}\)\\approx\\mathbf\{Y\}\\mathbf\{W\}\_\{O\}\+\\Delta\\mathbf\{O\}, i\.e\.𝐘​Δ​𝐖≈Δ​𝐎\\mathbf\{Y\}\\,\\Delta\\mathbf\{W\}\\approx\\Delta\\mathbf\{O\}, withΔ​𝐖\\Delta\\mathbf\{W\}constrained to rankρ\\rho:

minrank⁡Δ​𝐖≤ρ⁡‖𝐘​Δ​𝐖−Δ​𝐎‖F2\.\\min\_\{\\operatorname\{rank\}\\Delta\\mathbf\{W\}\\leq\\rho\}\\ \\big\\\|\\mathbf\{Y\}\\,\\Delta\\mathbf\{W\}\-\\Delta\\mathbf\{O\}\\big\\\|\_\{F\}^\{2\}\.\(8\)

#### Closed form\.

Let𝐀=𝔼​\[𝐘⊤​𝐘\]\\mathbf\{A\}=\\mathbb\{E\}\[\\mathbf\{Y\}^\{\\top\}\\mathbf\{Y\}\]and𝐁=𝔼​\[𝐘⊤​Δ​𝐎\]\\mathbf\{B\}=\\mathbb\{E\}\[\\mathbf\{Y\}^\{\\top\}\\Delta\\mathbf\{O\}\]\(both accumulated as streaming second moments,𝒪​\(d2\)\\mathcal\{O\}\(d^\{2\}\)memory, no per\-token buffers\)\. The unconstrained least\-squares solution is𝐀−1​𝐁\\mathbf\{A\}^\{\-1\}\\mathbf\{B\}; the optimal rank\-ρ\\rhosolution whitens, truncates, and un\-whitens\(Izenman,[1975](https://arxiv.org/html/2608.07915#bib.bib26)\):

𝐌=𝐀−1/2​𝐁=𝐔​𝚺​𝐕⊤,𝐌ρ=𝐔:ρ​𝚺:ρ​𝐕:ρ⊤,\\mathbf\{M\}=\\mathbf\{A\}^\{\-1/2\}\\mathbf\{B\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\},\\qquad\\mathbf\{M\}\_\{\\rho\}=\\mathbf\{U\}\_\{:\\rho\}\\mathbf\{\\Sigma\}\_\{:\\rho\}\\mathbf\{V\}\_\{:\\rho\}^\{\\top\},\(9\)Δ​𝐖ρ=𝐀−1/2​𝐌ρ,𝐖O←𝐖O\+Δ​𝐖ρ,\\Delta\\mathbf\{W\}\_\{\\rho\}=\\mathbf\{A\}^\{\-1/2\}\\mathbf\{M\}\_\{\\rho\},\\qquad\\mathbf\{W\}\_\{O\}\\leftarrow\\mathbf\{W\}\_\{O\}\+\\Delta\\mathbf\{W\}\_\{\\rho\},\(10\)which is the optimal reduced\-rank regression in the𝐀\\mathbf\{A\}\-metric\. The SVD is computed once per layer, so sweepingρ\\rhois essentially free; in practiceρ=4\\rho=4recovers most of the lost quality at negligible parameter cost\. The keys/values are not healed this way: the value error enters𝐖O\\mathbf\{W\}\_\{O\}linearly and is compensable there, whereas the key error acts through the softmax and is not\.

## Appendix DReverse water\-filling derivation

We derive the allocation of Equation[5](https://arxiv.org/html/2608.07915#S3.E5), following classical optimal bit allocation\(Huang and Schultheiss,[1963](https://arxiv.org/html/2608.07915#bib.bib24); Cover and Thomas,[2006](https://arxiv.org/html/2608.07915#bib.bib10)\)\. Under the scalar\-source approximation of Section[3\.2](https://arxiv.org/html/2608.07915#S3.SS2), treat therrcentered latent coordinates as independent sources with measured marginal variancesv1≥⋯≥vrv\_\{1\}\\geq\\cdots\\geq v\_\{r\}\. Under a high\-rate scalar quantizer, coordinatejjquantized atbjb\_\{j\}bits incurs distortionDj=c​vj​2−2​bjD\_\{j\}=c\\,v\_\{j\}\\,2^\{\-2b\_\{j\}\}for a constantcccommon to all coordinates\. We minimize the total distortion at a fixed average rate:

minb1,…,br​∑j=1rvj​2−2​bjs\.t\.1r​∑j=1rbj=b¯\.\\min\_\{b\_\{1\},\\dots,b\_\{r\}\}\\ \\sum\_\{j=1\}^\{r\}v\_\{j\}\\,2^\{\-2b\_\{j\}\}\\quad\\text\{s\.t\.\}\\quad\\frac\{1\}\{r\}\\sum\_\{j=1\}^\{r\}b\_\{j\}=\\bar\{b\}\.\(11\)With a multiplierλ\\lambdafor the rate constraint, the Lagrangian isℒ=∑jvj​2−2​bj\+λ​\(∑jbj−r​b¯\)\\mathcal\{L\}=\\sum\_\{j\}v\_\{j\}2^\{\-2b\_\{j\}\}\+\\lambda\\big\(\\sum\_\{j\}b\_\{j\}\-r\\bar\{b\}\\big\), and∂ℒ/∂bj=−2​ln⁡2​vj​2−2​bj\+λ=0\\partial\\mathcal\{L\}/\\partial b\_\{j\}=\-2\\ln 2\\,v\_\{j\}2^\{\-2b\_\{j\}\}\+\\lambda=0gives

vj​2−2​bj=λ2​ln⁡2=θ\(a constant, the “water level”\),v\_\{j\}\\,2^\{\-2b\_\{j\}\}\\;=\\;\\frac\{\\lambda\}\{2\\ln 2\}\\;=\\;\\theta\\qquad\\text\{\(a constant, the \`\`water level''\)\},\(12\)so the optimum equalizes the residual distortionDjD\_\{j\}across all retained coordinates\. Solving,bj=12​log2⁡\(vj/θ\)b\_\{j\}=\\tfrac\{1\}\{2\}\\log\_\{2\}\(v\_\{j\}/\\theta\)\. Imposing the rate constraint,1r​∑j12​log2⁡\(vj/θ\)=b¯\\tfrac\{1\}\{r\}\\sum\_\{j\}\\tfrac\{1\}\{2\}\\log\_\{2\}\(v\_\{j\}/\\theta\)=\\bar\{b\}, i\.e\.log2⁡θ=1r​∑jlog2⁡vj−2​b¯=log2⁡GM​\(v\)−2​b¯\\log\_\{2\}\\theta=\\tfrac\{1\}\{r\}\\sum\_\{j\}\\log\_\{2\}v\_\{j\}\-2\\bar\{b\}=\\log\_\{2\}\\mathrm\{GM\}\(v\)\-2\\bar\{b\}withGM​\(v\)=\(∏jvj\)1/r\\mathrm\{GM\}\(v\)=\(\\prod\_\{j\}v\_\{j\}\)^\{1/r\}\. Substituting back,

bj=b¯\+12​log2⁡\(vj/GM​\(v\)\),b\_\{j\}=\\bar\{b\}\+\\tfrac\{1\}\{2\}\\log\_\{2\}\\\!\\big\(v\_\{j\}/\\mathrm\{GM\}\(v\)\\big\),\(13\)which is Equation[5](https://arxiv.org/html/2608.07915#S3.E5)\. The unconstrainedbjb\_\{j\}can be negative for low\-variance coordinates; clamping to\[0,bmax\]\[0,b\_\{\\max\}\]\(reverse water\-filling\) sets those to zero bits — the coordinate is dropped, which is the continuous bridge to rank reduction\. The integer, group\-level realization \(per\-group mean variance, integer rounding with a rate\-matching fix\-up\) is described in Section[3\.2](https://arxiv.org/html/2608.07915#S3.SS2)\.

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/fig_bitprofile.png)Figure 8:Reverse water\-filling on a real layer\(Llama\-3\.1\-8B, layer 16, keys\)\. Allocated bits \(solid\) track the latent energy spectrum \(shaded\): high\-energy groups get more bits than the uniform baseline \(dotted, KIVI’s implicit choice\), low\-energy groups fewer\. Asb¯=1\\bar\{b\}\{=\}1, tail groups hit zero bits and quantization becomes rank reduction \(reff≈768/1024r\_\{\\text\{eff\}\}\\approx 768/1024\)\.
## Appendix ELatent\-space attention and memory

Caching the latent opens a path that is closed to scalar quantizers, because the attention score is*linear*in the latent:

𝐪​𝐤t⊤=𝐪​\(𝐜t​𝐖up\)⊤=\(𝐪​𝐖up⊤\)​𝐜t⊤=𝐪′​𝐜t⊤,\\mathbf\{q\}\\,\\mathbf\{k\}\_\{t\}^\{\\top\}\\;=\\;\\mathbf\{q\}\\,\(\\mathbf\{c\}\_\{t\}\\mathbf\{W\}\_\{\\text\{up\}\}\)^\{\\top\}\\;=\\;\(\\mathbf\{q\}\\,\\mathbf\{W\}\_\{\\text\{up\}\}^\{\\top\}\)\\,\\mathbf\{c\}\_\{t\}^\{\\top\}\\;=\\;\\mathbf\{q\}^\{\\prime\}\\,\\mathbf\{c\}\_\{t\}^\{\\top\},\(14\)so the up\-projection can be folded into the query once per decoding step \(𝐪′=𝐪𝐖up⊤\\mathbf\{q\}^\{\\prime\}=\\mathbf\{q\}\\mathbf\{W\}\_\{\\text\{up\}\}^\{\\top\}, a cheap1×dk​v→1×r1\\times d\_\{kv\}\\\!\\to\\\!1\\times rmap\), and attention can then run*directly*in therr\-dimensional latent space without ever materializing full keys or values\. The value side isout=softmax​\(𝐪′​𝐂⊤\)​𝐂V​𝐖upV\\mathrm\{out\}=\\mathrm\{softmax\}\(\\mathbf\{q\}^\{\\prime\}\\mathbf\{C\}^\{\\top\}\)\\,\\mathbf\{C\}\_\{V\}\\,\\mathbf\{W\}\_\{\\text\{up\}\}^\{V\}, where only the final⋅𝐖upV\\cdot\\,\\mathbf\{W\}\_\{\\text\{up\}\}^\{V\}lifts back todk​vd\_\{kv\}\. The dominant matrix multiplies \(𝐪′​𝐂⊤\\mathbf\{q\}^\{\\prime\}\\mathbf\{C\}^\{\\top\}andsoftmax⋅𝐂V\\mathrm\{softmax\}\\cdot\\mathbf\{C\}\_\{V\}\) then act in dimensionrrrather thandk​vd\_\{kv\}, so the memory saving becomes a compute saving as well\. This is unavailable to KIVI/TurboQuant/PolarQuant, which cache𝐤,𝐯\\mathbf\{k\},\\mathbf\{v\}directly and have no up\-projection to absorb\.

#### From memory to compute \(future work\)\.

Because the dominant matrix multiplies act in dimensionrrrather thandk​vd\_\{kv\}, the memory saving can in principle become a compute saving\. Realizing it as a wall\-clock speedup, however, requires a fused latent\-space attention kernel with mixed\-bit unpacking and RoPE integration—an engineering effort we deliberately leave to future work\. This paper therefore quantifies the dimension the latent already delivers*without*new kernels: memory\.

#### Memory under constrained budgets\.

A smaller per\-token cache directly increases the context \(or batch\) a fixed accelerator can hold\. We sweep a range of GPU memory budgets and, for each, report the longest single\-sequence context that fits before out\-of\-memory, comparing the dense fp16 cache againstSpectraat representative compression ratios; the footprint used is the true compressed size \(latent payload plus per\-group scales and zero\-points\), not an optimized runtime\. Figure[6](https://arxiv.org/html/2608.07915#S4.F6)plots the result and Table[3](https://arxiv.org/html/2608.07915#A5.T3)lists the analytical capacities\. The measured validation pins a filler tensor with the fp16 model resident and grows the compressed KV footprint until CUDA OOM; measured H200 points match the analytical values within1%1\\%at every tested budget and ratio \(e\.g\.2424GB dense68\.368\.3k measured vs\.68\.868\.8k analytical;8080GBSpectra\-9\.48×9\.48\\times5\.005\.00M measured vs\.5\.005\.00M analytical\)\.

Table 3:Max single\-sequence context \(tokens\) vs\. GPU memory budget — Llama\-3\.1\-8B, batch 1\.Geometry:3232layers,88KV heads, head dim128128\(128128KiB/token dense fp16 KV\); fp16 weights16\.0616\.06GB\. Storage counts the true compressed footprint \(latent payload plus per\-group scales and zero\-points\)\. Real H200 validation tracks these values within1%1\\%\.

## Appendix FEvaluation and compression\-accounting details

#### Calibration\.

We estimate the activation Gram matrices and means from 2,048\-token windows drawn from the WikiText\-2 training split, using3232windows for the LongBench and NIAH evaluations\. Statistics are accumulated in fp64 from the input to each attention layer, and no evaluation examples are used to construct the transform\. Unless noted otherwise, the latent group size isg=64g=64and the output correction has rankρ=4\\rho=4;Spectra\-WF uses an equal rank split between keys and values \(κ=0\.5\\kappa=0\.5\), whileSpectra\-LR assigns one quarter of the total rank budget to keys \(κ=0\.25\\kappa=0\.25\)\.

#### Effective compression ratio\.

All reported compression ratios compare total KV\-cache storage with the fp16 cache for the same model and sequence length\. Ifbeffb\_\{\\mathrm\{eff\}\}denotes the effective number of stored bits per original KV element, including the payload, scales, and zero\-points, then the reported ratio is16/beff16/b\_\{\\mathrm\{eff\}\}\. ForSpectra\-WF,b¯\\bar\{b\}denotes the target average payload rate before metadata rather than a uniform channel bit\-width\. ForSpectra\-LR, the tables report the resulting effective ratio after both rank reduction and latent quantization\. Baselines are accounted the same way, counting their per\-group scales and zero\-points, so all ratios are comparable\.

#### Evaluation scope\.

LongBench uses the test split ofTHUDM/LongBenchand the eight English tasks in Tables[1](https://arxiv.org/html/2608.07915#S4.T1)–[2](https://arxiv.org/html/2608.07915#S4.T2), taking the first150150examples per task\. We use the official prompt templates and per\-task metrics, truncate inputs longer than31,50031\{,\}500tokens by keeping equal\-length prefix and suffix segments, apply the chat template to all tasks except the few\-shot and code tasks \(which the official protocol feeds without a wrapper\), and decode greedily with the task\-specific generation limits\. Needle\-in\-a\-Haystack plants unique magic codes on a grid of six context lengths \(11k–3232k\) and seven depths, in single\- and four\-needle variants, and scores exact recovery under greedy decoding\. Every method—dense,Spectra, and all baselines—is evaluated through this single harness\. The memory experiment \(Appendix[E](https://arxiv.org/html/2608.07915#A5)\) sweeps a range of GPU memory budgets and reports the longest single\-sequence context that fits before out\-of\-memory, using each method’s true compressed cache footprint while the model stays resident; it isolates the cache\-capacity dimension and does not run an optimized compressed\-attention kernel\.

## Appendix GFull Experiment Results

### G\.1LongBench: Per\-Task Results

Table 4:LongBench per task — Llama\-3\.1\-8B\-Instruct\.Full sweep: every SPECTRA operating point and all baselines, grouped by compression regime\. Higher is better; Avg is the mean over the eight tasks\. Within each group thebestandsecondAvg are highlighted\.Table 5:LongBench per task — Mistral\-7B\-Instruct\-v0\.3\.Full sweep: every SPECTRA operating point and all baselines, grouped by compression regime\. Higher is better; Avg is the mean over the eight tasks\. Within each group thebestandsecondAvg are highlighted\.Table 6:LongBench per task — Qwen2\.5\-7B\-Instruct\.Dense \(fp16\) andSpectra\(best configuration per compression tier; no baselines were run for this model\)\. Higher is better; Avg is the mean over the eight tasks\. ThebestandsecondSpectraAvg are highlighted\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/pareto/qwen_pareto_full.png)Figure 9:LongBench quality vs\. compression — Qwen2\.5\-7B\-Instruct\.Dense line andSpectrabest\-per\-tier curve \(no baselines run for this model\)\.Spectratracks the dense average out to∼5×\\sim\\\!5\\timesand degrades beyond; the whole sweep tops out near6×6\\times—lower than Llama’s12×12\\times—consistent with Qwen’s less concentrated produced\-KV energy\.
### G\.2Needle\-in\-a\-Haystack \(NIAH\)

![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/needles/llama/niah_combined.png)Figure 10:Single\-needle NIAH — Llama\-3\.1\-8B\-Instruct\.Retrieval accuracy across context length \(x\-axis\) and needle depth \(y\-axis\); green marks a retrieved needle, red a missed one, and the badge reports overall accuracy\. SPECTRA matches the dense fp16 model up to high compression, staying near\-perfect where 2\-bit baselines \(KIVI, TurboQuant, PolarQuant\) begin to fail\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/needles/llama/niah_multi.png)Figure 11:Multi\-needle NIAH — Llama\-3\.1\-8B\-Instruct\.Fraction ofkkneedles retrieved as a function of context length \(x\-axis\) and needle depth \(y\-axis\); color runs from red \(none retrieved\) to green \(all retrieved\), and the badge reports the mean fraction\. SPECTRA preserves multi\-needle retrieval across the depth–length grid while 2\-bit baselines degrade sharply, especially at long contexts\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/needles/mistral/niah_combined.png)Figure 12:Single\-needle NIAH — Mistral\-7B\-Instruct\-v0\.3\.Retrieval accuracy across context length \(x\-axis\) and needle depth \(y\-axis\); green marks a retrieved needle, red a missed one, and the badge reports overall accuracy\. SPECTRA matches the dense fp16 model up to high compression, staying near\-perfect where 2\-bit baselines \(KIVI, TurboQuant, PolarQuant\) begin to fail\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/needles/mistral/niah_multi.png)Figure 13:Multi\-needle NIAH — Mistral\-7B\-Instruct\-v0\.3\.Fraction ofkkneedles retrieved as a function of context length \(x\-axis\) and needle depth \(y\-axis\); color runs from red \(none retrieved\) to green \(all retrieved\), and the badge reports the mean fraction\. SPECTRA preserves multi\-needle retrieval across the depth–length grid while 2\-bit baselines degrade sharply, especially at long contexts\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/needles/qwen/niah_combined.png)Figure 14:Needle\-in\-a\-Haystack \(single\-needle\) — Qwen2\.5\-7B\-Instruct\.Retrieval accuracy over context length×\\timesdepth for dense andSpectra;Spectramatches dense \(100%100\\%\) at every compression tier tested\.![Refer to caption](https://arxiv.org/html/2608.07915v1/figs/needles/qwen/niah_multi.png)Figure 15:Needle\-in\-a\-Haystack \(multi\-needle, 4\) — Qwen2\.5\-7B\-Instruct\.Spectraretrieval under the harder four\-needle setting; it holds7979–82%82\\%, at or slightly above the dense model \(76%76\\%\)\.