RoLaQ：用于多头潜在注意力中 2 位 KV 缓存的联合旋转与范围校准

# Joint Rotation and Range Calibrationfor 2-Bit KV Caches in Multi-Head Latent Attention
Source: [https://arxiv.org/html/2610.20686](https://arxiv.org/html/2610.20686)
Shiyu JiAffiliation:Research Center for Social Computing and Interactive Robotics,Harbin Institute of Technology, ChinaEmail:[syji@ir\.hit\.edu\.cngithub\-althttps://github\.com/noforit/RoLaQ](mailto:[email protected]://github.com/noforit/RoLaQ)Peng LingAffiliation:Tsinghua University, ChinaEmail:[car@ir\.hit\.edu\.cngithub\-althttps://github\.com/noforit/RoLaQ](mailto:[email protected]://github.com/noforit/RoLaQ)Shupeng LiAffiliation:Baidu Inc\., ChinaYixuan WangAffiliation:Research Center for Social Computing and Interactive Robotics,Harbin Institute of Technology, ChinaQingfu ZhuAffiliation:Research Center for Social Computing and Interactive Robotics,Harbin Institute of Technology, ChinaWanxiang Che††thanks:Corresponding author\.Affiliation:Research Center for Social Computing and Interactive Robotics,Harbin Institute of Technology, China

###### Abstract

Multi\-head latent attention \(MLA\) compresses the content of a key–value cache into a shared latent per token, making low\-bit storage attractive for long\-context inference\. At two bits, errors in this representation perturb both attention matching and value readout, with effects that depend on error direction and reconstruction scale\. We introduceRoLaQ\(Rotation\-calibrated Latent Quantization\), an offline method for joint rotation and range calibration\.RoLaQrefines a shared rotation within subspaces defined by the frozen key and value projections, and jointly learns quantization ranges whose thresholds and reconstruction gains are coupled to control the shared signal’s response\. Prediction matching and attention\-output supervision guide calibration with model weights frozen\. Across three MLA models and six benchmarks,RoLaQachieves the highest average accuracy among the compared 2\-bit methods both without and with token protection\. On DeepSeek\-V2\-Lite, an SGLang implementation provides3\.96×3\.96\\timesthe BF16 KV capacity under the same memory budget and2\.51×2\.51\\timesits decode throughput with 256K input tokens\.

## 1Introduction

Figure 1:\(a\) MLA shares a content cache between keys and values\. \(b,c\) On one A100 80GB,RoLaQgives DeepSeek\-V2\-Lite3\.96×3\.96\\timesthe BF16 capacity at equal memory budgets and2\.51×2\.51\\timesits decode throughput with four concurrent requests and 256K inputs\.Long contexts and concurrent requests increase the memory and bandwidth costs of the key–value \(KV\) cache during autoregressive decoding\([Vaswani et al\., 2017](https://arxiv.org/html/2610.20686#bib.bib1);[Liu et al\., 2024c](https://arxiv.org/html/2610.20686#bib.bib20)\)\. Multi\-head latent attention \(MLA\), introduced in DeepSeek\-V2, reduces these costs by caching a compact content latent shared across heads\([Liu et al\., 2024a](https://arxiv.org/html/2610.20686#bib.bib2)\)\. MLA and its variants now underpin models including DeepSeek\-V3, GLM\-5\.3, and Hy4\([Liu et al\., 2024b](https://arxiv.org/html/2610.20686#bib.bib23);[GLM\-5\-Team et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib29);[tencent, 2026](https://arxiv.org/html/2610.20686#bib.bib30)\)\. Even with this architectural compression, the cache continues to grow with context length and serving concurrency\. KV cache quantization offers further savings by lowering the storage precision of cached states\([Liu et al\., 2024c](https://arxiv.org/html/2610.20686#bib.bib20);[Hooper et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib19);[Kang et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib35)\)\. We focus on quantizing MLA’s content latent to two bits while preserving its two roles in attention matching and value readout\.

KIVI exploits distinct K/V statistics with per\-channel key and per\-token value quantization\([Liu et al\., 2024c](https://arxiv.org/html/2610.20686#bib.bib20)\)\. KVQuant and GEAR further control distortion through outlier handling and residual error compensation\([Hooper et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib19);[Kang et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib35)\)\. Rotations make low\-bit representations easier to quantize by redistributing outliers\([Ashkboos et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib18);[Liu et al\., 2025b](https://arxiv.org/html/2610.20686#bib.bib4);[Su et al\., 2025](https://arxiv.org/html/2610.20686#bib.bib21)\)\. Attention\-aware calibration further adapts these transformations to the computations that consume the cache, as in OptR’s attention\-output reconstruction objective\([Yun et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib6)\)\. These methods primarily target multi\-head or grouped\-query attention with separate key and value caches\.

In MLA, every head derives both content keys and values from the same quantized latent through distinct projections \(Figure[1](https://arxiv.org/html/2610.20686#S1.F1)a\)\. A single perturbation can therefore change both the matching scores that select information and the values aggregated under those scores\. OSCAR provides covariance\-aware rotations for this shared cache\([Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\)\. These projection paths guide our joint calibration of the latent basis and quantization range\.

Two observations guide our design \(§[2](https://arxiv.org/html/2610.20686#S2)\)\. Joint K/V projection energy is unevenly distributed across latent directions, so equal\-magnitude errors can produce different distortions\. Reconstruction gain also affects prediction fidelity even when the rotation is fixed\. Because rotation changes the quantizer’s input distribution, we learn its boundaries alongside the latent basis\. Together, these observations motivate joint rotation and range learning with a controlled reconstruction response\.

We introduceRoLaQ, which jointly learns a shared rotation within K/V\-defined subspaces and a quantization threshold per layer, with model weights frozen\. Each threshold determines its reconstruction gain, allowing the quantization assignments to adapt while controlling the shared signal’s reconstruction response\. A BF16 teacher supplies next\-token distributions and attention outputs: prediction KL aligns the complete model, while attention\-output supervision directly guides the individual layers\. Calibration takes roughly 30 minutes on one A100 GPU\. At inference,RoLaQuses a packed 2\-bit content cache and absorbs read\-side rotations into MLA’s projections, eliminating standalone rotation multiplications when attending to the cached history\.

We evaluateRoLaQon DeepSeek\-V2\-Lite, Moonlight\-16B\-A3B, and MiniCPM3\-4B across six benchmarks\. It achieves the highest average accuracy among the compared 2\-bit methods both with and without token protection\. Without protection, it improves on the strongest baseline by 1\.1 to 3\.8 points \(Table[1](https://arxiv.org/html/2610.20686#S3.T1)\)\. Retaining selected tokens in BF16 further improves accuracy\. RULER retrieval at 8K and 16K shows that calibration gains extend beyond the 4K training context\. Figure[1](https://arxiv.org/html/2610.20686#S1.F1)b,c shows the SGLang gains for DeepSeek\-V2\-Lite:3\.96×3\.96\\timesthe BF16 cache capacity at equal memory budgets and2\.51×2\.51\\timesits decode throughput with 256K input on one A100 80GB\.

Our contributions are as follows:

- •We develop joint rotation and range calibration for MLA, using a compact K/V search space and coupled thresholds and gains to control the shared latent’s response\.
- •We demonstrate accuracy gains across three MLA models alongside long\-context retrieval improvements, with ablations explaining the roles of rotation geometry, quantizer response, and calibration objectives\.
- •We implement packed INT2 serving in SGLang with fused cache writes and optimized attention paths, delivering greater KV capacity, higher long\-context throughput, and lower first\-token tail latency\.

## 2MLA Cache Quantization: Structure and Challenges

### 2\.1One Cache, Two Roles

For each tokentt, MLA caches a content latentct∈ℝdc\_\{t\}\\in\\mathbb\{R\}^\{d\}and a decoupled RoPE keyktR∈ℝdrk^\{R\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{r\}\}\([Liu et al\., 2024a](https://arxiv.org/html/2610.20686#bib.bib2)\)\. Each headhhobtains its content key and value through frozen up\-projections,kt,h=ct​WK,h⊤k\_\{t,h\}=c\_\{t\}W\_\{K,h\}^\{\\top\}andvt,h=ct​WV,h⊤v\_\{t,h\}=c\_\{t\}W\_\{V,h\}^\{\\top\}\. Using row vectors, a query with content partqhq\_\{h\}and positional partqhRq^\{R\}\_\{h\}produces

αt,h\\displaystyle\\alpha\_\{t,h\}=softmaxt⁡\[τ⁡\(qh​kt,h⊤\+qhR​ktR⊤\)\],\\displaystyle=\\operatorname\{softmax\}\_\{t\}\\\!\\left\[\\tau\\big\(q\_\{h\}k\_\{t,h\}^\{\\top\}\+q^\{R\}\_\{h\}\{k^\{R\}\_\{t\}\}^\{\\top\}\\big\)\\right\],\(1\)y\\displaystyle y=∑h∑tαt,h​vt,h​WO,h,\\displaystyle=\\textstyle\\sum\_\{h\}\\sum\_\{t\}\\alpha\_\{t,h\}v\_\{t,h\}W\_\{O,h\},whereτ\\tauis the attention scale, softmax runs over causally visible positions, andWO,hW\_\{O,h\}is headhh’s output\-projection block\. The shared latent therefore determines both where attention looks through the keys and what it reads through the values\. Its quantization error propagates through both roles\.

We quantizectc\_\{t\}to two bits and retain the lower\-dimensional RoPE keyktRk^\{R\}\_\{t\}in BF16 because of its quantization sensitivity\([Zhang et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib17)\)\. An orthogonal rotationRRspreads outliers before quantization\([Ashkboos et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib18);[Liu et al\., 2025b](https://arxiv.org/html/2610.20686#bib.bib4)\)\. WritingQQfor quantization andDQ\\operatorname\{DQ\}for dequantization, every head usesc^t=DQ⁡\(Q⁡\(ct​R\)\)​R⊤\\hat\{c\}\_\{t\}=\\operatorname\{DQ\}\(Q\(c\_\{t\}R\)\)R^\{\\top\}for matching and readout\.

### 2\.2Projection\-Dependent Distortion

Letet=c^t−cte\_\{t\}=\\hat\{c\}\_\{t\}\-c\_\{t\}be the reconstruction error\. Its effects on the attention logit and value are

Δ​st,h=τ​qh​WK,h​et⊤,Δ​vt,h=et​WV,h⊤\.\\Delta s\_\{t,h\}=\\tau q\_\{h\}W\_\{K,h\}e\_\{t\}^\{\\top\},\\qquad\\Delta v\_\{t,h\}=e\_\{t\}W\_\{V,h\}^\{\\top\}\.\(2\)Errors of equal magnitude can affect the two projections differently depending on their direction\. LetWKW\_\{K\}andWVW\_\{V\}stack the corresponding up\-projections across heads\. ForX∈\{K,V\}X\\in\\\{K,V\\\}, the squared projection error is‖e​WX⊤‖2=e​WX⊤​WX​e⊤\\\|eW\_\{X\}^\{\\top\}\\\|^\{2\}=eW\_\{X\}^\{\\top\}W\_\{X\}e^\{\\top\}\. We define

HX=WX⊤​WXtr⁡\(WX⊤​WX\),H=HK\+HV\.H\_\{X\}=\\frac\{W\_\{X\}^\{\\top\}W\_\{X\}\}\{\\operatorname\{tr\}\(W\_\{X\}^\{\\top\}W\_\{X\}\)\},\\qquad H=H\_\{K\}\+H\_\{V\}\.\(3\)Trace normalization gives both projections equal total weight\. The quadratic forme​H​e⊤eHe^\{\\top\}sums normalized squared K/V errors\. Each eigenvalue ofHHmeasures this error along its unit eigenvector\.

Figure 2:Two observations on shared\-latent quantization\. \(a\) K/V energy in the leading half of the joint spectrum \(mean±\\pmlayer std\.; dashed: 50%\)\. \(b\) Prediction KL at 8K with OSCAR’s rotation and threshold fixed, normalized by its value atg0=1\.16g\_\{0\}=1\.16\(vertical dashed line\)\. DS, ML, and CPM denote DeepSeek\-V2\-Lite, Moonlight, and MiniCPM3\.Figure[2](https://arxiv.org/html/2610.20686#S2.F2)a shows that the leading half ofHH’s eigenvectors captures 62–75% of each projection’s energy across the three models, averaged over layers\. This common basis separates directions by their influence on both projections, guiding the shared rotation search\.

### 2\.3Sensitivity to Reconstruction Scale

Beyond error direction, reconstruction scale affects both matching and readout\. The threshold controls four\-level assignments, while the gain scales the reconstructed signal shared by both paths\.

Figure[2](https://arxiv.org/html/2610.20686#S2.F2)b isolates this effect by fixing OSCAR’s rotation and threshold and varying only its gain\. Values below one indicate lower KL than the baseline with fixedg0=1\.16g\_\{0\}=1\.16\. On 8K\-token documents, adjusting the gain alone reduces DeepSeek\-V2\-Lite’s next\-token KL to BF16 by 52% within the sweep\. Together, these observations motivate joint calibration of the shared rotation and quantization range\. Appendix[B\.1](https://arxiv.org/html/2610.20686#A2.SS1)details both experiments and the full spectra\.

## 3Method

RoLaQjointly calibrates the shared representation and its quantizer to preserve attention matching and value readout \(Figure[3](https://arxiv.org/html/2610.20686#S3.F3)\)\. The*K/V\-aware shared rotation*uses the joint projection geometry to organize its search space \(§[3\.1](https://arxiv.org/html/2610.20686#S3.SS1)\)\. The*learnable quantization range*adapts the decision boundary to this basis while preserving the centered signal’s linear response \(§[3\.2](https://arxiv.org/html/2610.20686#S3.SS2)\)\. Joint calibration uses prediction matching and attention\-output supervision to learn both components in the frozen model \(Figure[3](https://arxiv.org/html/2610.20686#S3.F3)\(d\); §[3\.3](https://arxiv.org/html/2610.20686#S3.SS3)\)\.

Figure 3:Overview ofRoLaQ\. \(a\) A shared 2\-bit content cache supplies both K/V paths\. The dashed box groups the read\-side rotation with the projections that absorb it\. \(b\)HK\+HVH\_\{K\}\+H\_\{V\}defines two spectral subspaces for learning the shared rotation\. \(c\) The learned threshold determines the four\-level assignments and gaing⁡\(T\)=1/C⁡\(T\)g\(T\)=1/C\(T\)\. \(d\) A frozen BF16 teacher guides joint calibration through prediction KL and attention\-output loss\.### 3\.1K/V\-Aware Shared Rotation

We useHHfrom Eq\.[3](https://arxiv.org/html/2610.20686#S2.E3)to divide rotation refinement between directions with high and low joint projection energy\. Refining within these subspaces preserves this distinction while allowing the quantization basis to adapt to calibration data\. Both subspaces contribute to the key and value paths\.

Starting from OSCAR’s covariance\-aware rotationR0R\_\{0\}\([Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\), we express the geometry in the initial rotated coordinates asR0⊤​H​R0=U​Λ​U⊤R\_\{0\}^\{\\top\}HR\_\{0\}=U\\Lambda U^\{\\top\}\. We sort the eigenvalues in descending order and splitU=\[U1,U2\]U=\[U\_\{1\},U\_\{2\}\]into two equal\-dimensional subspaces\. Within each, a learnable skew\-symmetric generator defines the refinement:

R\\displaystyle R=R0​U​diag⁡\(eS1,eS2\)​U⊤,\\displaystyle=R\_\{0\}U\\operatorname\{diag\}\\\!\\big\(e^\{S\_\{1\}\},e^\{S\_\{2\}\}\\big\)U^\{\\top\},\(4\)Si\\displaystyle S\_\{i\}=Ai−Ai⊤,Ai∈ℝd2×d2\.\\displaystyle=A\_\{i\}\-A\_\{i\}^\{\\top\},\\quad A\_\{i\}\\in\\mathbb\{R\}^\{\\frac\{d\}\{2\}\\times\\frac\{d\}\{2\}\}\.This parameterization keepsRRorthogonal throughout calibration; zero\-initializingA1,A2A\_\{1\},A\_\{2\}recoversR0R\_\{0\}\. The two generators used2/2d^\{2\}/2trainable parameters, half the number in a fulld×dd\\times dgenerator\. The calibration objective selects the rotation within this structured space\. Section[5\.2](https://arxiv.org/html/2610.20686#S5.SS2)evaluates this choice of rotation search space\.

### 3\.2Response\-Preserving Range Calibration

The range controls both the quantization assignments and the strength of the reconstructed signal\. We adapt the assignments to the learned rotation while anchoring the linear response of the shared signal to its full\-precision reference\. We standardize each group ofGGrotated coordinates using its meanμ\\muand standard deviationσ\\sigma\([Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\)\. For a coordinatezz, reconstruction takes the form

x=z−μσ,z^=μ\+σ​g​aT​\(x\)\.x=\\frac\{z\-\\mu\}\{\\sigma\},\\qquad\\hat\{z\}=\\mu\+\\sigma g\\,a\_\{T\}\(x\)\.\(5\)HereaTa\_\{T\}assignsxxto four ordered levels\{±ain,±aout\}\\\{\\pm a\_\{\\mathrm\{in\}\},\\pm a\_\{\\mathrm\{out\}\}\\\}using boundaries\(−T,0,T\)\(\-T,0,T\), withaout=3​aina\_\{\\mathrm\{out\}\}=3a\_\{\\mathrm\{in\}\}\. The thresholdTTcontrols assignment to the inner or outer levels, while the gainggscales reconstruction around the group mean\.

To separate systematic scale change from residual distortion, we usex∼𝒩⁡\(0,1\)x\\sim\\mathcal\{N\}\(0,1\)as a reference distribution\. The quantized signal decomposes asg​aT​\(x\)=κ​x\+ϵg\\,a\_\{T\}\(x\)=\\kappa x\+\\epsilon, where𝔼⁡\[x​ϵ\]=0\\mathbb\{E\}\[x\\epsilon\]=0\([Bussgang and others, 1952](https://arxiv.org/html/2610.20686#bib.bib27);[Demir and Bjornson, 2020](https://arxiv.org/html/2610.20686#bib.bib16)\)and

κ=g​C​\(T\),C⁡\(T\)=𝔼⁡\[x​aT​\(x\)\]\.\\kappa=gC\(T\),\\qquad C\(T\)=\\mathbb\{E\}\[x\\,a\_\{T\}\(x\)\]\.\(6\)The coefficientκ\\kappameasures the linear response to the input\. In the original group scale,z^−μ=κ⁡\(z−μ\)\+σ​ϵ\\hat\{z\}\-\\mu=\\kappa\(z\-\\mu\)\+\\sigma\\epsilon\. ChangingTTandggtherefore changes the response of the centered signal reused by both attention paths\. Appendix[B\.3](https://arxiv.org/html/2610.20686#A2.SS3)derivesC⁡\(T\)C\(T\)\.

To enforce unit response under the Gaussian reference, we set the gain from the threshold:

g⁡\(T\)=1C⁡\(T\)⟹κ=1\.g\(T\)=\\frac\{1\}\{C\(T\)\}\\quad\\Longrightarrow\\quad\\kappa=1\.\(7\)Each layer learnsTℓT\_\{\\ell\}\(Appendix[C\.3](https://arxiv.org/html/2610.20686#A3.SS3)\), and its gain follows from this constraint\. The assignments can therefore adapt while preserving unit response\. Section[5\.3](https://arxiv.org/html/2610.20686#S5.SS3)tests this response after calibration across context lengths, complementing the initial gain sweep in Figure[2](https://arxiv.org/html/2610.20686#S2.F2)b\.

Table 1:Downstream accuracy \(%\) with 2\-bit content caches\. Each pair reports*w\.o\. / w\. token protection*\(without / with 4 sink and 16 recent tokens retained in BF16\)\. BF16 is a shared reference; bold marks the best 2\-bit scores at the displayed precision within each setting\.
### 3\.3Joint Calibration and Inference

The rotation changes the distribution seen by the quantizer, so we learn the rotation generators and thresholds together across allLLlayers\. The backbone remains frozen\. For the same input text, the BF16 model supplies target next\-token distributions and attention outputs\. We optimize

ℒ\\displaystyle\\mathcal\{L\}=ℒKL\+λ​ℒattn,\\displaystyle=\\mathcal\{L\}\_\{\\mathrm\{KL\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{attn\}\},\(8\)ℒKL\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{KL\}\}=1\|ℐ\|∑t∈ℐKL\(ptBF16∥pt2​b​i​t\),\\displaystyle=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\}\\mathrm\{KL\}\\\!\\big\(p\_\{t\}^\{\\mathrm\{BF16\}\}\\,\\\|\\,p\_\{t\}^\{\\mathrm\{2bit\}\}\\big\),ℒattn\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{attn\}\}=1L​∑ℓ=1L‖Yℓ2​b​i​t−YℓBF16‖F2‖YℓBF16‖F2\.\\displaystyle=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}\\frac\{\\\|Y\_\{\\ell\}^\{\\mathrm\{2bit\}\}\-Y\_\{\\ell\}^\{\\mathrm\{BF16\}\}\\\|\_\{F\}^\{2\}\}\{\\\|Y\_\{\\ell\}^\{\\mathrm\{BF16\}\}\\\|\_\{F\}^\{2\}\}\.Hereℐ\\mathcal\{I\}contains the scored token positions,ptp\_\{t\}is the next\-token distribution, andYℓY\_\{\\ell\}collects attention outputs afterWOW\_\{O\}at those positions\. Prediction KL aligns the complete quantized model with its BF16 teacher\. The auxiliary output loss, weighted byλ\\lambda, supplies a direct signal to each layer through both attention matching and value readout, following output\-aware calibration\([Yun et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib6)\)\. Normalizing by teacher output energy balances layers with different scales\.

We initialize the rotation and threshold from OSCAR and set the gain using Eq\.[7](https://arxiv.org/html/2610.20686#S3.E7)\. The forward pass uses the hard quantizer in Eq\.[5](https://arxiv.org/html/2610.20686#S3.E5)\. A straight\-through estimator passes gradients to its inputs\([Bengio et al\., 2013](https://arxiv.org/html/2610.20686#bib.bib15)\), while a smooth threshold surrogate supplies gradients for bin assignment\. The threshold also receives gradients throughg⁡\(Tℓ\)g\(T\_\{\\ell\}\)\. Calibration settings are in §[4](https://arxiv.org/html/2610.20686#S4); quantizer constants and checkpoint selection are in Appendix[A\.3](https://arxiv.org/html/2610.20686#A1.SS3)\.

At inference, the learned rotation is exported as one dense matrix per layer, and each new latent is rotated once and stored as 2\-bit codes with its group statistics\. Forz^t=DQ⁡\(Q⁡\(ct​R\)\)\\hat\{z\}\_\{t\}=\\operatorname\{DQ\}\(Q\(c\_\{t\}R\)\), the read\-side rotation in Figure[3](https://arxiv.org/html/2610.20686#S3.F3)\(a\)’s dashed box folds into MLA’s absorbed projections:

q~h\\displaystyle\\widetilde\{q\}\_\{h\}=qh​WK,h​R,\\displaystyle=q\_\{h\}W\_\{K,h\}R,\(9\)W~O,h\\displaystyle\\widetilde\{W\}\_\{O,h\}=R⊤​WV,h⊤​WO,h,\\displaystyle=R^\{\\top\}W\_\{V,h\}^\{\\top\}W\_\{O,h\},y\\displaystyle y=∑h\(∑tαt,h​z^t\)​W~O,h\.\\displaystyle=\\textstyle\\sum\_\{h\}\\big\(\\sum\_\{t\}\\alpha\_\{t,h\}\\hat\{z\}\_\{t\}\\big\)\\widetilde\{W\}\_\{O,h\}\.The content logit isτ​q~h​z^t⊤\\tau\\widetilde\{q\}\_\{h\}\\hat\{z\}\_\{t\}^\{\\top\}, so both matching and readout operate in the rotated basis\. The calibrated rotation, thresholds, and gains remain fixed throughout serving\. Only the group statistics are computed from incoming tokens\.

## 4Experiments

### 4\.1Setup

#### Models and benchmarks\.

We evaluate DeepSeek\-V2\-Lite\([Liu et al\., 2024a](https://arxiv.org/html/2610.20686#bib.bib2)\), Moonlight\-16B\-A3B\([Liu et al\., 2025a](https://arxiv.org/html/2610.20686#bib.bib3)\), and MiniCPM3\-4B\([Hu et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib22)\)on six benchmarks: MMLU and CMMLU for knowledge\([Hendrycks et al\., 2020](https://arxiv.org/html/2610.20686#bib.bib10);[Li et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib11)\), TriviaQA for question answering\([Joshi et al\., 2017](https://arxiv.org/html/2610.20686#bib.bib31)\), GSM8K for mathematics\([Cobbe et al\., 2021](https://arxiv.org/html/2610.20686#bib.bib9)\), and HumanEval and MBPP for code generation\([Chen et al\., 2021](https://arxiv.org/html/2610.20686#bib.bib12);[Austin et al\., 2021](https://arxiv.org/html/2610.20686#bib.bib13)\)\. We also evaluate long\-context retrieval on the eight RULER retrieval tasks\([Hsieh et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib8)\)\.

#### Baselines and cache settings\.

We compare with round\-to\-nearest quantization \(RTN\), QuaRot\([Ashkboos et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib18)\), KVarN\([Muller et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib7)\), and OSCAR\([Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\)\. QuaRot and KVarN are applied to the shared MLA latent; OSCAR uses its official MLA recipe\. All comparisons use 2\-bit content caches and BF16 RoPE keys\. We evaluate every method both without token protection and with the first 4 sink tokens and most recent 16 tokens retained in BF16\([Xiao et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib24);[Liu et al\., 2024c](https://arxiv.org/html/2610.20686#bib.bib20);[Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\)\. Accuracy is evaluated with simulated quantization in Hugging Face Transformers, giving all baselines a common evaluation framework\.

#### Calibration\.

We calibrateRoLaQon 1\.43M Pile tokens\([Gao et al\., 2020](https://arxiv.org/html/2610.20686#bib.bib14)\)with the model frozen and select checkpoints by held\-out prediction KL\. Main results average three seeds; component, objective, and random\-subspace ablations use one\. Calibration takes about 22 minutes for the 16B models and 34 minutes for MiniCPM3 on one A100 GPU\. Appendix[A](https://arxiv.org/html/2610.20686#A1)provides model configurations, evaluation protocols, baseline implementations, and calibration hyperparameters\.

### 4\.2Main Results

RoLaQachieves the highest six\-task average on all three models under both protection settings \(Table[1](https://arxiv.org/html/2610.20686#S3.T1)\)\. We first examine the setting in which every content token is quantized, which directly tests how well each method preserves accuracy under 2\-bit storage\.RoLaQreaches 46\.7, 60\.1, and 59\.4 on DeepSeek\-V2\-Lite, Moonlight, and MiniCPM3, exceeding the strongest baseline by 3\.8, 2\.8, and 1\.1 points\. The larger margins without protection highlight the benefit of calibrating the shared representation and its quantization range when the entire content history passes through the quantizer\.

On DeepSeek\-V2\-Lite, GSM8K accuracy without protection improves from OSCAR’s 18\.3 to 32\.1\. Moreover,RoLaQwithout protection reaches a higher average than the strongest baseline with 20 protected tokens \(46\.7 vs\. 45\.3\)\.

Retaining 4 sink and 16 recent tokens in BF16 further improvesRoLaQ, which maintains the highest average accuracy on all three models\. The calibrated cache thus benefits from mixed\-precision token protection while retaining its advantage over the baselines\. Section[5\.4](https://arxiv.org/html/2610.20686#S5.SS4)examines protection budgets; Appendix[C\.1](https://arxiv.org/html/2610.20686#A3.SS1)reports all seeds and windows\.

Appendix[C\.4](https://arxiv.org/html/2610.20686#A3.SS4)illustrates these differences with arithmetic and retrieval examples, comparing the answers generated by BF16 and 2\-bit models\.

Table 2:Retrieval accuracy \(%\) on the eight RULER retrieval tasks with 4 sink and 16 recent tokens retained in BF16\. Avg\. denotes the task mean; bold marks the best 2\-bit scores at the displayed precision\.
### 4\.3Long\-Context Retrieval

We evaluate eight RULER retrieval tasks at 16K on DeepSeek\-V2\-Lite and 8K on Moonlight \(Table[2](https://arxiv.org/html/2610.20686#S4.T2)\), their longest evaluated contexts in this study\. Both exceed the maximum calibration length of 4K\.RoLaQleads the 2\-bit methods on both models, reaching 73\.7% on DeepSeek\-V2\-Lite, 6\.3 points above the strongest baseline\. On Moonlight, it reaches 94\.8%, compared with BF16’s 96\.2%\. DeepSeek\-V2\-Lite’s MK2 accuracy also improves over OSCAR from 7\.0 to 55\.0\. These results demonstrate transfer beyond the calibration length; Section[5\.3](https://arxiv.org/html/2610.20686#S5.SS3)examines response sensitivity up to 32K\.

## 5Ablations and Analysis

We examine how rotation, range, and calibration choices affect accuracy and serving efficiency\.

### 5\.1Calibration Ablations

Table 3:Ablations of calibration components and objectives\. \(a\) MMLU and MBPP accuracy \(%\) for different component configurations\. \(b\) Six\-task average accuracy \(%\) for different calibration objectives\.#### Calibration components\.

Joint calibration achieves the highest accuracy on both models \(Table[3](https://arxiv.org/html/2610.20686#S5.T3)a\)\. Allowing the range to adapt increases the gains from rotation calibration\. For example, rotation calibration improves DeepSeek\-V2\-Lite’s MBPP accuracy by 2\.2 points with a fixed range and by 4\.0 points with a learned range\. These results support adapting the quantizer jointly with the latent representation\.

#### Calibration objective\.

With both components enabled, the combined objective yields the highest six\-task mean on both models \(Table[3](https://arxiv.org/html/2610.20686#S5.T3)b\)\. Attention\-output supervision alone slightly outperforms KL alone, while combining them raises the means to 48\.1 on DeepSeek\-V2\-Lite and 62\.6 on Moonlight\. These gains support supervising both final predictions and per\-layer attention outputs\. We useλ=0\.1\\lambda=0\.1for the combined objective\.

### 5\.2Rotation Search Space

Table 4:Rotation search spaces\. Params\. counts trainable rotation parameters per layer;SSis validation KL relative to OSCAR\. Avg\. is six\-task mean accuracy \(%\)\.Figure 4:Effect of quantizer response on DeepSeek\-V2\-Lite\. Prediction KL to BF16 is normalized by its value atκ=1\\kappa=1for each context length\. Using a checkpoint calibrated with full rotation and unit response, we vary the gain at inference while keeping the rotation and thresholds fixed\.TheHK\+HVH\_\{K\}\+H\_\{V\}partition uses half the trainable rotation parameters of full exponential and Cayley rotations while matching or improving their six\-task accuracy \(Table[4](https://arxiv.org/html/2610.20686#S5.T4)\)\. It also reduces validation KL by 6\.3% and 9\.1% relative to full exponential rotation on DeepSeek\-V2\-Lite and Moonlight\. The random control preserves block sizes, initialization, and calibration budget but replaces the K/V eigenbasis with a random orthogonal basis\. The K/V basis improves both KL and accuracy over this control\. These comparisons support a compact rotation search guided by MLA’s two projection paths \(Appendix[B\.2](https://arxiv.org/html/2610.20686#A2.SS2)\)\. The learned blocks export as a dense rotation, allowing the compact search space to use the same inference kernels as full rotations\.

### 5\.3Quantizer Response

Range calibration controls the shared signal used by both attention paths\. Under the Gaussian reference, minimizing reconstruction error can favor attenuation \(Appendix[B\.3](https://arxiv.org/html/2610.20686#A2.SS3)\)\. Coupling gain to threshold improves MiniCPM3’s normalized validation KL from 0\.606 with free gain to 0\.527 within the sameHK\+HVH\_\{K\}\+H\_\{V\}search space \(Appendix[C\.2](https://arxiv.org/html/2610.20686#A3.SS2)\)\. The coupled setting also gives more consistent KL across the three calibration seeds\.

We then examine response sensitivity by fixing a DeepSeek\-V2\-Lite checkpoint calibrated withκ=1\\kappa=1and varying its gain at inference\. Figure[4](https://arxiv.org/html/2610.20686#S5.F4)shows the lowest prediction KL atκ=1\\kappa=1across contexts from 2K to 32K\. Amplification becomes more costly as context grows\. Together, these results support adapting quantization boundaries while controlling the reconstructed signal shared by attention matching and value readout\.

### 5\.4Token Protection

Figure 5:Effect of token protection on six\-task mean accuracy \(%\)\. The horizontal axis gives the number of sink \+ recent content tokens retained in BF16\.Figure 6:Serving performance of DeepSeek\-V2\-Lite on an A100 80GB GPU\. \(a\) KV token capacity\. \(b,c\) Decode throughput for long inputs and successive output intervals, with four concurrent requests\. \(d\) P99 time to first token \(TTFT\) under increasing concurrency\. OSCAR andRoLaQshare the INT2 kernels, so their overlapping curves use timings measured withRoLaQunder the same workloads\.The advantage ofRoLaQis largest without token protection: it exceeds the strongest baseline by 1\.1–3\.8 points in six\-task mean accuracy across the three models \(Table[1](https://arxiv.org/html/2610.20686#S3.T1)\)\. With every content token quantized, these gains highlight the accuracy preserved by the learned latent quantization itself\. We reuse the same calibrated checkpoints across protection budgets\. Retaining sink and recent tokens in BF16 further improvesRoLaQ, which maintains the highest mean across all three protection budgets \(Figure[5](https://arxiv.org/html/2610.20686#S5.F5)\)\. Latent calibration improves the low\-bit representation, while token protection retains selected tokens at high precision; their benefits can therefore be combined\. With only 4 sink and 8 recent tokens protected,RoLaQalready exceeds the strongest baseline using 4 sink and 16 recent tokens on each model \(Tables[8](https://arxiv.org/html/2610.20686#A3.T8)and[1](https://arxiv.org/html/2610.20686#S3.T1)\)\.

### 5\.5Serving Efficiency

We integrateRoLaQinto SGLang\([Zheng et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib32)\)with Triton kernels for packed INT2 cache writes and attention\. We fold read\-side rotations into projections and apply thresholds and gains during cache writes \(§[3\.3](https://arxiv.org/html/2610.20686#S3.SS3)\)\. We benchmark DeepSeek\-V2\-Lite on one A100 80GB against SGLang’s default BF16 implementation with identical workloads and memory budgets \(Appendix[D](https://arxiv.org/html/2610.20686#A4)\)\.

Under the same memory budget,RoLaQand OSCAR each accommodate 4\.56M KV tokens, compared with 1\.15M for BF16, providing3\.96×3\.96\\timesthe capacity \(Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(a\)\)\. Decode throughput reaches2\.51×2\.51\\timesBF16 with 256K input tokens and2\.43×2\.43\\timesduring the 128K–256K output interval \(Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(b,c\)\)\. At 256 concurrent requests, P99 TTFT falls by 37\.8%, from 588\.6 s to 366\.0 s \(Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(d\)\)\. These measurements demonstrate greater KV capacity, faster decoding, and lower first\-token tail latency in a practical serving system\.

## 6Related Work

#### KV cache quantization\.

KV cache quantization reduces the memory cost of long\-context inference by storing cached states at lower precision\. KIVI adapts quantization granularity to key and value statistics, while KVQuant combines non\-uniform quantization with outlier handling\([Liu et al\., 2024c](https://arxiv.org/html/2610.20686#bib.bib20);[Hooper et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib19)\)\. GEAR combines low\-bit quantization with low\-rank and sparse components to compensate errors\([Kang et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib35)\), while ZipCache assigns higher precision to salient tokens\([He et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib36)\)\. Retaining recent tokens or attention sinks in full precision provides another way to control error\([Liu et al\., 2024c](https://arxiv.org/html/2610.20686#bib.bib20);[Su et al\., 2025](https://arxiv.org/html/2610.20686#bib.bib21)\)\. SnapMLA combines FP8 MLA caches with hardware\-aware decoding while retaining RoPE keys in high precision\([Zhang et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib17)\)\.

#### Rotation and calibration\.

Beyond the quantizer itself, representation transforms help control outliers at low bitwidths while preserving full\-precision computation\. QuaRot uses Hadamard rotations, while SpinQuant learns rotations through calibration\([Ashkboos et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib18);[Liu et al\., 2025b](https://arxiv.org/html/2610.20686#bib.bib4)\)\. RotateKV and KVarN adapt KV caches through outlier\-aware rotations and variance normalization, respectively\([Su et al\., 2025](https://arxiv.org/html/2610.20686#bib.bib21);[Muller et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib7)\)\. OmniQuant learns clipping and equivalent transformations with pretrained weights frozen, motivating joint calibration of representations and quantization ranges\([Shao et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib33)\)\.

Recent methods further tailor calibration to how cache errors affect attention\. OSCAR derives rotations from attention\-aware covariance statistics and supports MLA\([Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\), while OptR learns per\-head key and value rotations through attention\-output reconstruction after the output projection\([Yun et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib6)\)\. QuantMLA learns offline\-fusible transformations for content and RoPE caches, using attention\-output and positional\-QK reconstruction objectives, respectively\([Su et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib34)\)\.RoLaQuses MLA’s K/V projection structure to guide joint calibration of the shared latent rotation and quantization range\.

## 7Conclusion

We presentedRoLaQfor joint calibration of MLA’s latent rotation and quantization range at two bits\. K/V projection geometry provides a compact rotation search space, while coupled thresholds and gains control reconstruction response\. Experiments show higher average accuracy across three MLA models and better long\-context retrieval, with further gains from token protection\. Our SGLang implementation increases KV capacity and throughput while reducing latency at high concurrency\.

## Limitations

Further compression of MLA caches must account for both content and position\.RoLaQquantizes the shared content latent while retaining the RoPE key in BF16, leaving positional keys and group metadata as a substantial share of the remaining cache\. Extending these savings calls for coordinated content and positional precision while preserving attention’s ability to distinguish positions\. The serving benefits of low\-bit caches also depend on hardware and workload\. Bandwidth savings are most valuable when reading a long history dominates decoding\. At shorter contexts, quantization and unpacking overheads can offset them\. Our serving measurements use DeepSeek\-V2\-Lite on an A100\. Extending these benefits calls for kernels tailored to the target hardware and workload\.

## References

- Ashkbooset al\.\(2024\)S\. Ashkboos, A\. Mohtashami, M\. L\. Croci, B\. Li, P\. Cameron, M\. Jaggi, D\. Alistarh, T\. Hoefler, and J\. HensmanQuarot: Outlier\-free 4\-bit inference in rotated llms\.Advances in Neural Information Processing Systems37,pp\. 100213–100240\.Cited by:[§A\.2](https://arxiv.org/html/2610.20686#A1.SS2.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.20686#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Bengioet al\.\(2013\)Y\. Bengio, N\. Léonard, and A\. CourvilleEstimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§3\.3](https://arxiv.org/html/2610.20686#S3.SS3.p2.1)\.
- Bidermanet al\.\(2024\)S\. Biderman, H\. Schoelkopf, L\. Sutawika, L\. Gao, J\. Tow, B\. Abbasi, A\. F\. Aji, P\. S\. Ammanamanchi, S\. Black, J\. Clive,et al\.Lessons from the trenches on reproducible evaluation of language models\.arXiv preprint arXiv:2405\.14782\.Cited by:[§A\.1](https://arxiv.org/html/2610.20686#A1.SS1.p1.1)\.
- Bussganget al\.\(1952\)J\. J\. Bussganget al\.Crosscorrelation functions of amplitude\-distorted Gaussian signals\.Technical reportResearch Laboratory of Electronics, Massachusetts Institute of Technology\.Cited by:[§B\.3](https://arxiv.org/html/2610.20686#A2.SS3.p1.2),[§3\.2](https://arxiv.org/html/2610.20686#S3.SS2.p2.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Demir and Bjornson \(2020\)O\. T\. Demir and E\. BjornsonThe Bussgang decomposition of nonlinear systems: Basic theory and MIMO extensions \[lecture notes\]\.IEEE Signal Processing Magazine38\(1\),pp\. 131–136\.Cited by:[§B\.3](https://arxiv.org/html/2610.20686#A2.SS3.p1.2),[§3\.2](https://arxiv.org/html/2610.20686#S3.SS2.p2.1)\.
- Gaoet al\.\(2020\)L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima,et al\.The pile: An 800gb dataset of diverse text for language modeling\.arXiv preprint arXiv:2101\.00027\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px3.p1.1)\.
- GLM\-5\-Teamet al\.\(2026\)GLM\-5\-Team, :, A\. Zeng, X\. Lv, Z\. Hou, Z\. Du, Q\. Zheng, B\. Chen, D\. Yin, C\. Ge, C\. Huang, C\. Xie, C\. Zhu, C\. Yin, C\. Wang, G\. Pan, H\. Zeng, H\. Zhang, H\. Wang, H\. Chen, J\. Zhang, J\. Jiao, J\. Guo, J\. Wang, J\. Du, J\. Wu, K\. Wang, L\. Li, L\. Fan, L\. Zhong, M\. Liu, M\. Zhao, P\. Du, Q\. Dong, R\. Lu, Shuang\-Li, S\. Cao, S\. Liu, T\. Jiang, X\. Chen, X\. Zhang, X\. Huang, X\. Dong, Y\. Xu, Y\. Wei, Y\. An, Y\. Niu, Y\. Zhu, Y\. Wen, Y\. Cen, Y\. Bai, Z\. Qiao, Z\. Wang, Z\. Wang, Z\. Zhu, Z\. Liu, Z\. Li, B\. Wang, B\. Wen, C\. Huang, C\. Cai, C\. Yu, C\. Li, C\. Hu, C\. Zhang, D\. Zhang, D\. Lin, D\. Yang, D\. Wang, D\. Ai, E\. Zhu, F\. Yi, F\. Chen, G\. Wen, H\. Sun, H\. Zhao, H\. Hu, H\. Zhang, H\. Liu, H\. Zhang, H\. Peng, H\. Tai, H\. Zhang, H\. Liu, H\. Wang, H\. Yan, H\. Ge, H\. Liu, H\. Chu, J\. Zhao, J\. Wang, J\. Zhao, J\. Ren, J\. Wang, J\. Zhang, J\. Gui, J\. Zhao, J\. Li, J\. An, J\. Li, J\. Yuan, J\. Du, J\. Liu, J\. Zhi, J\. Duan, K\. Zhou, K\. Wei, K\. Wang, K\. Luo, L\. Zhang, L\. Sha, L\. Xu, L\. Wu, L\. Ding, L\. Chen, M\. Li, N\. Lin, P\. Ta, Q\. Zou, R\. Song, R\. Yang, S\. Tu, S\. Yang, S\. Wu, S\. Zhang, S\. Li, S\. Li, S\. Fan, W\. Qin, W\. Tian, W\. Zhang, W\. Yu, W\. Liang, X\. Kuang, X\. Cheng, X\. Li, X\. Yan, X\. Hu, X\. Ling, X\. Fan, X\. Xia, X\. Zhang, X\. Zhang, X\. Pan, X\. Zou, X\. Zhang, Y\. Liu, Y\. Wu, Y\. Li, Y\. Wang, Y\. Zhu, Y\. Tan, Y\. Zhou, Y\. Pan, Y\. Zhang, Y\. Su, Y\. Geng, Y\. Yan, Y\. Tan, Y\. Bi, Y\. Shen, Y\. Yang, Y\. Li, Y\. Liu, Y\. Wang, Y\. Li, Y\. Wu, Y\. Zhang, Y\. Duan, Y\. Zhang, Z\. Liu, Z\. Jiang, Z\. Yan, Z\. Zhang, Z\. Wei, Z\. Chen, Z\. Feng, Z\. Yao, Z\. Chai, Z\. Wang, Z\. Zhang, B\. Xu, M\. Huang, H\. Wang, J\. Li, Y\. Dong, and J\. TangGLM\-5: from Vibe Coding to Agentic Engineering\.External Links:2602\.15763,[Link](https://arxiv.org/abs/2602.15763)Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1)\.
- Heet al\.\(2024\)Y\. He, L\. Zhang, W\. Wu, J\. Liu, H\. Zhou, and B\. ZhuangZipcache: Accurate and efficient kv cache quantization with salient token identification\.Advances in Neural Information Processing Systems37,pp\. 68287–68307\.Cited by:[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px1.p1.1)\.
- Hendryckset al\.\(2020\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Hooperet al\.\(2024\)C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. GholamiKvquant: Towards 10 million context length llm inference with kv cache quantization\.Advances in Neural Information Processing Systems37,pp\. 1270–1303\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1),[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px1.p1.1)\.
- Hsiehet al\.\(2024\)C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. GinsburgRULER: What’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Huet al\.\(2024\)S\. Hu, Y\. Tu, X\. Han, C\. He, G\. Cui, X\. Long, Z\. Zheng, Y\. Fang, Y\. Huang, W\. Zhao,et al\.MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies\.arXiv preprint arXiv:2404\.06395\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Joshiet al\.\(2017\)M\. Joshi, E\. Choi, D\. S\. Weld, and L\. ZettlemoyerTriviaqa: A large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1601–1611\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Kanget al\.\(2024\)H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. ZhaoGear: An efficient kv cache compression recipe for near\-lossless generative inference of llm\.arXiv preprint arXiv:2403\.05527\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1),[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)H\. Li, Y\. Zhang, F\. Koto, Y\. Yang, H\. Zhao, Y\. Gong, N\. Duan, and T\. BaldwinCmmlu: Measuring massive multitask language understanding in chinese\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 11260–11285\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024a\)A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo,et al\.Deepseek\-v2: A strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.20686#S2.SS1.p1.2),[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024b\)A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1)\.
- Liuet al\.\(2025a\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025b\)Z\. Liu, C\. Zhao, I\. Fedorov, B\. Soran, D\. Choudhary, R\. Krishnamoorthi, V\. Chandra, Y\. Tian, and T\. BlankevoortSpinquant: Llm quantization with learned rotations\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 92009–92032\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.20686#S2.SS1.p2.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024c\)Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. HuKivi: A tuning\-free asymmetric 2bit quantization for kv cache\.arXiv preprint arXiv:2402\.02750\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1),[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px1.p1.1)\.
- Lloyd \(1982\)S\. LloydLeast squares quantization in PCM\.IEEE transactions on information theory28\(2\),pp\. 129–137\.Cited by:[§B\.3](https://arxiv.org/html/2610.20686#A2.SS3.p1.1)\.
- Max \(1960\)J\. MaxQuantizing for minimum distortion\.IRE Transactions on Information Theory6\(1\),pp\. 7–12\.Cited by:[§B\.3](https://arxiv.org/html/2610.20686#A2.SS3.p1.1)\.
- Mulleret al\.\(2026\)L\. K\. Muller, P\. Bich, C\. Boretti, H\. Chang, J\. Zhuang, and L\. CavigelliKVarN: Variance\-Normalized KV\-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks\.arXiv preprint arXiv:2606\.03458\.Cited by:[§A\.2](https://arxiv.org/html/2610.20686#A1.SS2.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p1.1)\.
- Shaoet al\.\(2024\)W\. Shao, M\. Chen, Z\. Zhang, P\. Xu, L\. Zhao, Z\. Li, K\. Zhang, G\. Peng, Y\. Qiao, and P\. LuoOmniquant: Omnidirectionally calibrated quantization for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 45472–45496\.Cited by:[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p1.1)\.
- Suet al\.\(2025\)Z\. Su, Z\. Chen, W\. Shen, H\. Wei, L\. Li, H\. Yu, and K\. YuanRotatekv: Accurate and robust 2\-bit kv cache quantization for llms via outlier\-aware adaptive rotations\.arXiv preprint arXiv:2501\.16383\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p1.1)\.
- Suet al\.\(2026\)Z\. Su, Y\. Sun, J\. Tan, T\. Zhang, R\. Hu, Y\. Xie, X\. Cai, and N\. WongQuantMLA: Function\-Aligned Dual\-Path Quantization for Low\-Bit MLA KV Caching\.arXiv preprint arXiv:2609\.36760\.Cited by:[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p2.1)\.
- tencent \(2026\)tencentHy4\-preview\.Note:Official model documentationExternal Links:[Link](https://github.com/Tencent-Hunyuan/Hy4-preview)Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p1.1)\.
- Xiaoet al\.\(2024\)G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. LewisEfficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21875–21895\.Cited by:[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px2.p1.1)\.
- Yunet al\.\(2026\)V\. Yun, W\. Lim, M\. Cheong, S\. Lee, M\. Annavaram, S\. P\. Karimireddy, and S\. YooOutput\-Aware Rotation for INT2 KV\-Cache Quantization\.External Links:2608\.02691,[Link](https://arxiv.org/abs/2608.02691)Cited by:[§1](https://arxiv.org/html/2610.20686#S1.p2.1),[§3\.3](https://arxiv.org/html/2610.20686#S3.SS3.p1.3),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p2.1)\.
- Zhanget al\.\(2026\)Y\. Zhang, Z\. Su, S\. Hu, R\. Yang, W\. Wu, Y\. Qian, Y\. Xie, and X\. CaiSnapMLA: Efficient Long\-Context MLA Decoding via Hardware\-Aware FP8 Quantized Pipelining\.arXiv preprint arXiv:2602\.10718\.Cited by:[§2\.1](https://arxiv.org/html/2610.20686#S2.SS1.p2.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2024\)L\. Zheng, L\. Yin, Z\. Xie, C\. Sun, J\. Huang, C\. H\. Yu, S\. Cao, C\. Kozyrakis, I\. Stoica, J\. E\. Gonzalez,et al\.Sglang: Efficient execution of structured language model programs\.Advances in neural information processing systems37,pp\. 62557–62583\.Cited by:[§5\.5](https://arxiv.org/html/2610.20686#S5.SS5.p1.1)\.
- Zhouet al\.\(2026\)Z\. Zhou, D\. Zhuang, J\. Li, Z\. Chen, S\. L\. Song, B\. Athiwaratkun, and X\. WuOSCAR: Offline Spectral Covariance\-Aware Rotation for 2\-bit KV Cache Quantization\.arXiv preprint arXiv:2605\.17757\.Cited by:[§A\.2](https://arxiv.org/html/2610.20686#A1.SS2.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2610.20686#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.20686#S3.SS1.p2.2),[§3\.2](https://arxiv.org/html/2610.20686#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.20686#S4.SS1.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2610.20686#S6.SS0.SSS0.Px2.p2.1)\.

## Appendix AImplementation and Evaluation Details

### A\.1Models and Evaluation

The three models let us examine shared\-latent quantization across both mixture\-of\-experts and dense architectures \(Table[5](https://arxiv.org/html/2610.20686#A1.T5)\)\. DeepSeek\-V2\-Lite and Moonlight each have 16B total parameters; Moonlight follows the DeepSeek\-V3 architecture\. MiniCPM3 is a dense 4B model with a smaller latent and more layers\. We use lm\-evaluation\-harness\([Biderman et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib28)\)with the few\-shot settings in Table[6](https://arxiv.org/html/2610.20686#A1.T6)and greedy decoding for generation tasks\.

Table 5:Model configurations;ddanddrd\_\{r\}are the content\-latent and RoPE\-key dimensions\.Table 6:Downstream evaluation settings\.The benchmarks cover both answer selection and free\-form generation\. MMLU and CMMLU report accuracy weighted by subject size; TriviaQA uses exact match, and GSM8K uses flexible answer extraction\. HumanEval and MBPP report greedy pass@1\. MiniCPM3 uses its chat template, with HumanEval code extracted from the generated response\. For MBPP, we truncate code at the first new example or trailing assertion block and remove test calls before rescoring all methods with a common evaluator\.

Our long\-context evaluation focuses on RULER’s eight retrieval tasks: three single\-key, three multi\-key, one multi\-value, and one multi\-query task\. This set tests whether the cache preserves the information needed to recover a specified item as the surrounding context grows\. Each task contains 100 examples scored with the official evaluation functions\.

### A\.2Baseline Implementations

We compare the methods at the same point in the MLA computation: the shared content latent before the K/V up\-projections\. The RoPE key remains in BF16 for every method\. The group size is 128; RTN, QuaRot, and OSCAR group latent coordinates within each token, whereas KVarN groups tokens for channel\-wise quantization\. The formulas below describe the transformations used in our MLA evaluation\. We writeQ^=DQ∘Q\\widehat\{Q\}=\\operatorname\{DQ\}\\circ Qfor quantization followed by reconstruction\.

#### RTN\.

For a groupxx, letmx=mini⁡xim\_\{x\}=\\min\_\{i\}x\_\{i\}andMx=maxi⁡xiM\_\{x\}=\\max\_\{i\}x\_\{i\}\. With step sizeΔx=\(Mx−mx\)/3\\Delta\_\{x\}=\(M\_\{x\}\-m\_\{x\}\)/3, the affine min–max quantizer is

qx\\displaystyle q\_\{x\}=clip\[0,3\]⁡\(round⁡x−mxΔx\),\\displaystyle=\\operatorname\{clip\}\_\{\[0,3\]\}\\\!\\left\(\\operatorname\{round\}\\frac\{x\-m\_\{x\}\}\{\\Delta\_\{x\}\}\\right\),\(10\)Q^mm​\(x\)\\displaystyle\\widehat\{Q\}\_\{\\mathrm\{mm\}\}\(x\)=mx\+Δx​qx\.\\displaystyle=m\_\{x\}\+\\Delta\_\{x\}q\_\{x\}\.Constant groups are reconstructed exactly\. RTN appliesQ^mm\\widehat\{Q\}\_\{\\mathrm\{mm\}\}directly to each content\-latent group\.

#### QuaRot\.

Following the cache rotation in QuaRot\([Ashkboos et al\., 2024](https://arxiv.org/html/2610.20686#bib.bib18)\), we use a block\-diagonal matrixH128H\_\{128\}of normalized 128\-dimensional Hadamard transforms:

c^t=Q^mm​\(ct​H128\)​H128⊤\.\\hat\{c\}\_\{t\}=\\widehat\{Q\}\_\{\\mathrm\{mm\}\}\(c\_\{t\}H\_\{128\}\)H\_\{128\}^\{\\top\}\.\(11\)The Hadamard transform mixes coordinates before rounding, providing a data\-independent rotation reference with the backbone frozen\.

#### KVarN\.

KVarN\([Muller et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib7)\)combines a normalized Hadamard rotationHHwith variance balancing across tokens and latent channels\. For a tileXXof up to 128 tokens, letZ=\(X​H\)⊤Z=\(XH\)^\{\\top\}\. Diagonal scalesDcD\_\{c\}andDtD\_\{t\}balance its channel and token axes, respectively:

Z~\\displaystyle\\widetilde\{Z\}=Dc−1​Z​Dt−1,\\displaystyle=D\_\{c\}^\{\-1\}ZD\_\{t\}^\{\-1\},\(12\)X^\\displaystyle\\hat\{X\}=\(Dc​Q^mm​\(Z~\)​Dt\)⊤​H⊤\.\\displaystyle=\\bigl\(D\_\{c\}\\widehat\{Q\}\_\{\\mathrm\{mm\}\}\(\\widetilde\{Z\}\)D\_\{t\}\\bigr\)^\{\\top\}H^\{\\top\}\.We use 16 alternating variance\-normalization iterations and applyQ^mm\\widehat\{Q\}\_\{\\mathrm\{mm\}\}to each row ofZ~\\widetilde\{Z\}across tokens\. Padding is excluded from the statistics\. We also quantize partial tiles; single\-token updates use Hadamard rotation followed by RTN over groups of latent coordinates\.

#### OSCAR\.

We use the official MLA recipe of OSCAR\([Zhou et al\., 2026](https://arxiv.org/html/2610.20686#bib.bib5)\), with the per\-layer rotationR0R\_\{0\}fitted on Pile text by its calibration code\. LetQ^T0,g0\\widehat\{Q\}\_\{T\_\{0\},g\_\{0\}\}denote its group\-normalized four\-level reconstruction in Eq\.[5](https://arxiv.org/html/2610.20686#S3.E5), with fixed thresholdT0=0\.9810653T\_\{0\}=0\.9810653and gaing0=1\.16g\_\{0\}=1\.16:

c^t=Q^T0,g0​\(ct​R0\)​R0⊤\.\\hat\{c\}\_\{t\}=\\widehat\{Q\}\_\{T\_\{0\},g\_\{0\}\}\(c\_\{t\}R\_\{0\}\)R\_\{0\}^\{\\top\}\.\(13\)We also useR0R\_\{0\}to initializeRoLaQ\.

### A\.3Calibration and Storage

Calibration adapts the cache representation while leaving the language model unchanged\. All three models use the settings in Table[7](https://arxiv.org/html/2610.20686#A1.T7)\. The Pile training text is disjoint from the text used to fit OSCAR’s initial rotation\. We update only the rotation and thresholds, with the teacher and backbone frozen\. No tokens are protected during calibration, so the learned parameters are exposed to quantization throughout the sequence\. We reuse these checkpoints across all protection budgets\.

Table 7:Calibration hyperparameters\.#### Selecting a checkpoint\.

We select checkpoints by prediction fidelity on held\-out Pile text\. For each context setting,rrdenotes the teacher\-forced KL to BF16 divided by OSCAR’s KL on the same text\. The selection score is a weighted average of the KL ratios for short, 2K, and 4K contexts:

S=0\.5​rshort\+0\.25​r2​K\+0\.25​r4​K\.S=0\.5\\,r\_\{\\mathrm\{short\}\}\+0\.25\\,r\_\{\\mathrm\{2K\}\}\+0\.25\\,r\_\{\\mathrm\{4K\}\}\.\(14\)We keep the checkpoint with the lowestSS\. Thus,S<1S<1indicates lower prediction divergence than OSCAR, and selection reflects behavior across several context lengths\.

#### Quantization grid\.

We use groups ofG=128G=128coordinates and OSCAR’s four\-level affine grid, withaout=1\.5095586a\_\{\\mathrm\{out\}\}=1\.5095586andain=aout/3a\_\{\\mathrm\{in\}\}=a\_\{\\mathrm\{out\}\}/3\. The outer level and initial thresholdT0=0\.9810653T\_\{0\}=0\.9810653follow its Gaussian Lloyd–Max construction; the inner levels give an equally spaced reconstruction grid\. Each layer learns its threshold and derives the gain fromg=1/C⁡\(T\)g=1/C\(T\), starting atg≈1\.114g\\approx 1\.114for the initial threshold\.

#### Storage accounting\.

The cache stores three kinds of information: packed content codes, group statistics, and the RoPE key\. With two bits per content coordinate, two FP32 scalars per group, and a BF16 RoPE key, an unprotected token requiresd/4\+8​d/G\+2​drd/4\+8d/G\+2d\_\{r\}bytes per layer\. This gives 288 bytes for DeepSeek\-V2\-Lite and Moonlight, and 144 bytes for MiniCPM3\. Protected tokens additionally require BF16 content storage\. The allocation measurements in Appendix[D](https://arxiv.org/html/2610.20686#A4)include the packed buffers, group metadata, and token\-protection buffers\.

## Appendix BSupplementary Method Analysis

The method combines a choice of rotation space with control over quantizer response\. We first describe the observations behind these choices, then make the rotation variants explicit and derive the response constraint\.

### B\.1Motivating Observations

The two observations separate direction from scale\. Projection energy measures how strongly latent directions affect the K/V projections, while the gain intervention changes reconstruction strength without changing the rotation or the assignments\.

#### Projection energy\.

For each layer, we extract the native, head\-interleaved key and value blocks of the frozen up\-projection and formHK,HVH\_\{K\},H\_\{V\}as in Eq\.[3](https://arxiv.org/html/2610.20686#S2.E3)\. Letu1,…,udu\_\{1\},\\ldots,u\_\{d\}be the eigenvectors ofHK\+HVH\_\{K\}\+H\_\{V\}, ordered by descending eigenvalue\. ForX∈\{K,V\}X\\in\\\{K,V\\\}, the fraction of projection energy in the firstrrdirections is

FX​\(r\)=∑i=1r‖WX​ui‖22‖WX‖F2\.F\_\{X\}\(r\)=\\frac\{\\sum\_\{i=1\}^\{r\}\\\|W\_\{X\}u\_\{i\}\\\|\_\{2\}^\{2\}\}\{\\\|W\_\{X\}\\\|\_\{F\}^\{2\}\}\.\(15\)Figure[2](https://arxiv.org/html/2610.20686#S2.F2)a reports100​FX​\(d/2\)100F\_\{X\}\(d/2\), averaged equally over all 27 DeepSeek\-V2\-Lite layers, 27 Moonlight layers, and 62 MiniCPM3 layers\. The error bars are sample standard deviations across layers\. The same subspace is used for the key and value measurements within each layer\. All matrix products and eigendecompositions use FP64 on the original stored weights\. Figure[7](https://arxiv.org/html/2610.20686#A2.F7)shows the full cumulative energy profiles for both projections\.

Figure 7:Cumulative K/V projection energy in the common joint eigenbasis\. Solid and dashed curves denote keys and values, respectively; shading shows one standard deviation across layers\. The diagonal gives the isotropic reference\. The vertical line marks the equal\-dimensional split used in Figure[2](https://arxiv.org/html/2610.20686#S2.F2)a\.
#### Gain intervention\.

Figure[2](https://arxiv.org/html/2610.20686#S2.F2)b uses the OSCAR rotationR0R\_\{0\}before learned refinement and its thresholdT0=0\.981065T\_\{0\}=0\.981065in every layer\. We vary the gain uniformly across layers, holding bothR0R\_\{0\}andT0T\_\{0\}fixed\. The six evaluated gains are approximately\{1\.025,1\.069,1\.114,1\.160,1\.203,1\.248\}\\\{1\.025,1\.069,1\.114,1\.160,1\.203,1\.248\\\}; the original gain isg0=1\.16g\_\{0\}=1\.16\. All content tokens are quantized; RoPE keys remain in BF16\. We use 12 natural documents, two each from PG\-19, Wikipedia, EuroParl, HackerNews, arXiv, and Pile\-CC, with 8,192 tokens of context\. Each document contributes the meanDKL\(pBF16∥pINT2\)D\_\{\\mathrm\{KL\}\}\(p\_\{\\mathrm\{BF16\}\}\\\|p\_\{\\mathrm\{INT2\}\}\)over its final 1,024 next\-token predictions\. Each plotted value is the mean over documents divided by that model’s mean atg0g\_\{0\}; the curves connect the six measured settings\.

### B\.2Rotation Search Spaces

All variants start from the same covariance\-aware rotationR0R\_\{0\}\. They differ in which directions can mix during refinement and in how the orthogonal update is parameterized\. Writing these choices explicitly separates the effect of the K/V geometry from the size of the trainable search space\.

#### Full exponential rotation\.

A dense generatorA∈ℝd×dA\\in\\mathbb\{R\}^\{d\\times d\}defines

Rfull=R0​eΩ,Ω=A−A⊤\.R\_\{\\mathrm\{full\}\}=R\_\{0\}e^\{\\Omega\},\\qquad\\Omega=A\-A^\{\\top\}\.\(16\)Every latent direction can mix with every other direction\. This provides the unrestricted exponential reference for the structured variants\.

#### Cayley rotation\.

Using the same dense skew\-symmetric generator, Cayley replaces the matrix exponential with

RCayley=R0​\(I−Ω\)​\(I\+Ω\)−1\.R\_\{\\mathrm\{Cayley\}\}=R\_\{0\}\(I\-\\Omega\)\(I\+\\Omega\)^\{\-1\}\.\(17\)The trainable parameter count remainsd2d^\{2\}, so this comparison changes the orthogonal map while keeping the generator unrestricted\.

#### Joint\-energy blocks:HK\+HVH\_\{K\}\+H\_\{V\}\.

The structured variants useB=diag⁡\(eS1,eS2\)B=\\operatorname\{diag\}\(e^\{S\_\{1\}\},e^\{S\_\{2\}\}\), whereSi=Ai−Ai⊤S\_\{i\}=A\_\{i\}\-A\_\{i\}^\{\\top\}andAi∈ℝd/2×d/2A\_\{i\}\\in\\mathbb\{R\}^\{d/2\\times d/2\}\. For the joint\-energy basis,

R0⊤​\(HK\+HV\)​R0\\displaystyle R\_\{0\}^\{\\top\}\(H\_\{K\}\+H\_\{V\}\)R\_\{0\}=U\+​Λ\+​U\+⊤,\\displaystyle=U\_\{\+\}\\Lambda\_\{\+\}U\_\{\+\}^\{\\top\},\(18\)R\+\\displaystyle R\_\{\+\}=R0​U\+​B​U\+⊤\.\\displaystyle=R\_\{0\}U\_\{\+\}BU\_\{\+\}^\{\\top\}\.We order eigenvalues from largest to smallest and splitU\+U\_\{\+\}into two equal halves\. These blocks separate high and low joint projection energy, while each still contributes to both keys and values\.

#### K/V\-contrast blocks:HK−HVH\_\{K\}\-H\_\{V\}\.

Replacing the sum by the difference gives

R0⊤​\(HK−HV\)​R0\\displaystyle R\_\{0\}^\{\\top\}\(H\_\{K\}\-H\_\{V\}\)R\_\{0\}=U−​Λ−​U−⊤,\\displaystyle=U\_\{\-\}\\Lambda\_\{\-\}U\_\{\-\}^\{\\top\},\(19\)R−\\displaystyle R\_\{\-\}=R0​U−​B​U−⊤\.\\displaystyle=R\_\{0\}U\_\{\-\}BU\_\{\-\}^\{\\top\}\.Here the signed eigenvalues measure the relative strength of the key and value projections along each direction\. The leading half is more key\-dominant; the trailing half is more value\-dominant\.

#### Random subspaces\.

The random control uses the same block sizes but replaces the K/V basis with a fixed random orthogonal basis:

Rrand=R0​Urand​B​Urand⊤\.R\_\{\\mathrm\{rand\}\}=R\_\{0\}U\_\{\\mathrm\{rand\}\}BU\_\{\\mathrm\{rand\}\}^\{\\top\}\.\(20\)It therefore retains thed2/2d^\{2\}/2trainable parameters of the spectral variants while removing their K/V\-based grouping\. All generators are initialized at zero, recoveringR0R\_\{0\}, and all variants export a densed×dd\\times drotation for inference\. The random control uses seed 42; the other variants use three seeds\.

### B\.3Response of the Four\-Level Quantizer

Decision thresholds and reconstruction levels determine a scalar quantizer’s distortion\([Lloyd, 1982](https://arxiv.org/html/2610.20686#bib.bib25);[Max, 1960](https://arxiv.org/html/2610.20686#bib.bib26)\)\. To isolate the scale change introduced by quantization, we separate the part of the reconstruction aligned with the input from its residual\. This gives the linear response used in §[3\.2](https://arxiv.org/html/2610.20686#S3.SS2)\. Forx∼𝒩⁡\(0,1\)x\\sim\\mathcal\{N\}\(0,1\), letϕ\\phidenote the standard normal density\. Symmetry of the code in Eq\.[5](https://arxiv.org/html/2610.20686#S3.E5)gives

𝔼⁡\[x​aT​\(x\)\]=\\displaystyle\\mathbb\{E\}\[x\\,a\_\{T\}\(x\)\]=\{\}2​ain​∫0Tx​ϕ​\(x\)​𝑑x\\displaystyle 2a\_\{\\mathrm\{in\}\}\\\!\\int\_\{0\}^\{T\}\\\!x\\phi\(x\)\\,dx\(21\)\+2aout∫T∞xϕ\(x\)dx\\displaystyle\+2a\_\{\\mathrm\{out\}\}\\\!\\int\_\{T\}^\{\\infty\}\\\!x\\phi\(x\)\\,dx=\\displaystyle=\{\}2​ain​\[ϕ⁡\(0\)−ϕ⁡\(T\)\]\+2​aout​ϕ​\(T\),\\displaystyle 2a\_\{\\mathrm\{in\}\}\\big\[\\phi\(0\)\-\\phi\(T\)\\big\]\+2a\_\{\\mathrm\{out\}\}\\,\\phi\(T\),which isC⁡\(T\)C\(T\), using∫x​ϕ​\(x\)​𝑑x=−ϕ⁡\(x\)\\int x\\phi\(x\)\\,dx=\-\\phi\(x\)\. Since𝔼⁡\[x2\]=1\\mathbb\{E\}\[x^\{2\}\]=1, the least\-squares coefficient ofg​aT​\(x\)g\\,a\_\{T\}\(x\)onxxisκ=g​C​\(T\)\\kappa=g\\,C\(T\), and the residualϵ=g​aT​\(x\)−κ​x\\epsilon=g\\,a\_\{T\}\(x\)\-\\kappa xsatisfies𝔼⁡\[x​ϵ\]=0\\mathbb\{E\}\[x\\epsilon\]=0\([Bussgang and others, 1952](https://arxiv.org/html/2610.20686#bib.bib27);[Demir and Bjornson, 2020](https://arxiv.org/html/2610.20686#bib.bib16)\)\. Settingκ=1\\kappa=1preserves the component aligned with the input, whileϵ\\epsiloncaptures the remaining quantization noise\.

For a fixed threshold, the gain that minimizes the Gaussian reconstruction error isgMSE=C⁡\(T\)/𝔼⁡\[aT​\(x\)2\]g\_\{\\mathrm\{MSE\}\}=C\(T\)/\\mathbb\{E\}\[a\_\{T\}\(x\)^\{2\}\], which givesκMSE=C​\(T\)2/𝔼⁡\[aT​\(x\)2\]≈0\.88\\kappa\_\{\\mathrm\{MSE\}\}=C\(T\)^\{2\}/\\mathbb\{E\}\[a\_\{T\}\(x\)^\{2\}\]\\approx 0\.88nearT=1T=1\. Thus, reducing reconstruction error can favor a weaker reconstructed signal\. CouplingggtoTTthrough1/C⁡\(T\)1/C\(T\)instead anchors the aligned component while allowing the decision boundary to adapt\. Section[5\.3](https://arxiv.org/html/2610.20686#S5.SS3)tests how this response affects prediction fidelity across context lengths\.

We boundT∈\[0\.5,1\.25\]T\\in\[0\.5,1\.25\]so thatg=1/C⁡\(T\)≲1\.3g=1/C\(T\)\\lesssim 1\.3\. The official setting\(T,g\)=\(0\.981,1\.16\)\(T,g\)=\(0\.981,1\.16\)givesκ=1\.041\\kappa=1\.041, andκ=1\\kappa=1at the official threshold corresponds tog=1\.114g=1\.114\.

## Appendix CAdditional Results

These results expand the main comparisons along three dimensions: token\-protection budget, calibration seed, and range calibration\. We then examine the learned thresholds and concrete model responses to connect the aggregate scores with the behavior of the cache\.

### C\.1Token Protection and Calibration Seeds

The intermediate budget of 4 sink and 8 recent tokens fills the gap between the two settings in Table[1](https://arxiv.org/html/2610.20686#S3.T1)\. Table[8](https://arxiv.org/html/2610.20686#A3.T8)reports all six tasks at this budget, whereRoLaQagain has the highest mean among the compared 2\-bit methods on all three models\. Together with the main table, it shows how latent calibration and token protection combine across protection budgets\.

Table 8:Accuracy \(%\) with 4 sink and 8 recent BF16 tokens\.Tables[9](https://arxiv.org/html/2610.20686#A3.T9),[10](https://arxiv.org/html/2610.20686#A3.T10), and[11](https://arxiv.org/html/2610.20686#A3.T11)expand the method averages into seeds 42, 43, and 44 for the three protection settings\. Each seed corresponds to one calibrated checkpoint reused across protection budgets, so the tables show both calibration variation and the effect of protecting more tokens\. Standard deviations are computed across seeds for each task; for Avg\., they are computed from the three six\-task means\. DS, ML, and CPM denote DeepSeek\-V2\-Lite, Moonlight, and MiniCPM3\.

Table 9:Per\-seed accuracy \(%\) without protection\. Avg\. averages six tasks; std denotes sample standard deviation\.Table 10:Per\-seed accuracy \(%\) with 4 sink and 8 recent BF16 tokens\.Table 11:Per\-seed accuracy \(%\) with 4 sink and 16 recent BF16 tokens\.
### C\.2Range Calibration on MiniCPM3

With theHK\+HVH\_\{K\}\+H\_\{V\}search space fixed, Table[12](https://arxiv.org/html/2610.20686#A3.T12.fig1)compares freely learned gain with gain coupled to the threshold on MiniCPM3\. Coupling gain to threshold reduces the mean validation KL ratio from 0\.606 to 0\.527 and its variation across calibration seeds\. Thus, response control improves prediction fidelity within the same structured rotation space\.

Table 12:Range calibration withHK\+HVH\_\{K\}\+H\_\{V\}on MiniCPM3\.SSis validation KL relative to OSCAR \(lower is better\); values are mean±\\pmsample standard deviation over three seeds\.
### C\.3Learned Thresholds

The response constraint leaves each layer free to choose its quantization boundary\. Figure[8](https://arxiv.org/html/2610.20686#A3.F8)shows the resulting profiles across rotations and seeds\. Most layers move below the initial Lloyd–Max threshold, with thresholds concentrated aroundT≈0\.91T\\approx 0\.91, 0\.89, and 0\.85 for the three models\. Lower thresholds narrow the central assignment region, while the coupled gain preserves unit response\. The layerwise profiles therefore show how the quantizer adapts its assignments under a shared response constraint\.

Figure 8:Learned per\-layer thresholds withκ=1\\kappa=1for all rotations and seeds\. The dashed line is the Lloyd–Max threshold used by OSCAR\.
### C\.4Case Study

Tables[14](https://arxiv.org/html/2610.20686#A3.T14)and[14](https://arxiv.org/html/2610.20686#A3.T14)present four selected examples from DeepSeek\-V2\-Lite\. With the same token protection budget,RoLaQpreserves the correct BF16 answers while both QuaRot and OSCAR fail\.

In Table[14](https://arxiv.org/html/2610.20686#A3.T14)\(a\), both baselines divide the chips in one bag among five people, overlooking the second bag\.RoLaQcounts both bags and obtains 22 chips per person\. In \(b\), both baselines return the total requirement of 600 feet;RoLaQsubtracts the 6 feet already available and answers 594\. Across all 1319 GSM8K problems,RoLaQsolves 277 that OSCAR misses and misses 69 that OSCAR solves\.

In 16K retrieval \(Table[14](https://arxiv.org/html/2610.20686#A3.T14)\), QuaRot changes two digits of the value requested by a single\-key query, while OSCAR returns a different number\. For four queried keys, OSCAR repeats two values and omits the others; QuaRot repeats an unrelated number\.RoLaQretrieves the correct value or complete set in each case\.

Table 13:GSM8K examples on DeepSeek\-V2\-Lite\. \(a\) Both bags must be counted\. \(b\) The available line must be subtracted\. We show BF16 final answers and the quantized models’ solutions\. All quantized methods retain 4 sink and 16 recent tokens in BF16;RoLaQuses calibration seed 42\.Table 14:RULER examples from the 16K evaluation on DeepSeek\-V2\-Lite\. \(a\) A single key is queried among multiple keys\. \(b\) Four keys are queried together\. We show the relevant context sentences and each model’s first answer line, with the same protection budget and calibration seed as Table[14](https://arxiv.org/html/2610.20686#A3.T14)\.

## Appendix DServing Benchmarks

We evaluate serving from three perspectives: how much context fits in the cache, how quickly decoding proceeds as the cache grows, and how long requests wait for their first token under concurrency\. Tables[15](https://arxiv.org/html/2610.20686#A4.T15)–[17](https://arxiv.org/html/2610.20686#A4.T17)report the capacity, throughput, and latency measurements in Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\.

The experiments use the finalHK\+HVH\_\{K\}\+H\_\{V\},κ=1\\kappa=1checkpoint of DeepSeek\-V2\-Lite \(seed 42\) on one A100 80GB PCIe GPU\. Tensor parallelism is one and the static memory fraction is 0\.8\. The content cache uses two\-bit codes; RoPE keys, 4 sink tokens, and 16 recent tokens remain in BF16\. Both BF16 andRoLaQuse Triton prefill and decode kernels under the same workload and memory budget\. WikiText\-2 requests arrive together and generate fixed\-length outputs with greedy decoding\. Prefix reuse and early stopping are disabled\. We report one timed run per workload after warmup\.

OSCAR uses the same packed\-cache layout and kernels asRoLaQ\. Its curves in Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(b–d\) therefore reuse theRoLaQtiming values under this shared implementation\. The allocation runs give the same KV capacity for both methods\.

Table 15:Allocated KV capacity in Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(a\)\. GiB measures KV memory; ratios use BF16 as the reference\.Long\-input and long\-output workloads expose the growing cost of reading a larger history in two ways\. The former sweeps 1K–256K input tokens with 128 output tokens; the latter starts from 128 input tokens and generates 256K tokens\. Both use four concurrent requests, with K denoting 1,024 tokens\. For the input sweep, decode throughput is the aggregate token rate after all requests finish prefill and before the first completes\. For the output sweep, throughput is measured in successive output\-token intervals beginning with the first generated token; Figure[6](https://arxiv.org/html/2610.20686#S5.F6)c plots each rate at the upper end of its interval\.

Table 16:Decode throughput \(tok/s\) underlying Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(b,c\)\. Both workloads use four concurrent requests\. Speedups relative to BF16 are computed from unrounded measurements\.The concurrency workload instead fixes each request at 16K input and 1K output tokens and increases simultaneous arrivals from 1 to 256\. P99 TTFT is measured within each burst and includes queueing and prefill\. This captures the delay experienced by requests as the service becomes more heavily loaded\. Capacity counts the allocated KV token slots under the same memory budget and request\-slot limit, including group metadata and protected\-token buffers\.

Table 17:P99 TTFT \(s\) underlying Figure[6](https://arxiv.org/html/2610.20686#S5.F6)\(d\)\. Each request has 16K input and 1K output tokens\.
## Appendix EAI Assistance

We used AI assistants for code development, manuscript organization, and language editing\. The authors take responsibility for the final work\.