Output-Aware Rotation for INT2 KV-Cache Quantization

# Output-Aware Rotation for INT2 KV-Cache Quantization
Source: [https://arxiv.org/html/2608.02691](https://arxiv.org/html/2608.02691)
Vincent\-Daniel Yun1,†, Woosang Lim2,†, Minsoo Cheong2, Sunwoo Lee3 Murali Annavaram1, Sai Praneeth Karimireddy1, Sungjoo Yoo2

1University of Southern California \{yunjuyou, annavara, karimire\}@usc\.edu 2Seoul National University \{ftyg656512, icycle0409\}@snu\.ac\.kr 3Inha University \{sunwool\}@inha\.ac\.kr

†Equal Contribution

###### Abstract

The key\-value \(KV\) cache has become a major memory and bandwidth bottleneck in long\-context large language model inference, making ultra\-low\-bit quantization increasingly important\. However, existing rotation\-based INT2 methods optimize cache statistics or proxy errors before the complete attention readout, even though the model is ultimately affected by the error propagated through attention and the output projectionWOW\_\{O\}\. To address this mismatch, we proposeOptR, an output\-aware rotation method that minimizes post\-WOW\_\{O\}attention\-output error\. OptR decomposes the post\-WOW\_\{O\}attention\-output error into key\- and value\-induced terms and learns per\-head orthogonal corrections through the full INT2 quantization and attention path\. OptR further applies an attention\-equivalent key reparameterization to reduce large channel\-wise offsets without changing the softmax distribution\. Across three models and five reasoning and coding benchmarks, OptR consistently improves both QuaRot and OSCAR and strengthens long\-context retrieval, while preserving the paged KV\-cache format with negligible inference overhead\.

## 1Introduction

As large language models \(LLMs\) grow in model size and context length, the key\-value \(KV\) cache becomes a major bottleneck in long\-context inferenceDao et al\. \([2022](https://arxiv.org/html/2608.02691#bib.bib1)\); Ainslie et al\. \([2023](https://arxiv.org/html/2608.02691#bib.bib2)\)\. During autoregressive decoding, each layer stores the keys and values of all previous tokens and reads them at every generation step\. As a result, KV\-cache storage and memory traffic increase with context length, batch size, and model depth\. KV\-cache quantization reduces these costs by storing the cache at lower precision\. We focus on INT2 because it requires only1/81/8of the BF16 KV\-cache storage and1/21/2of that of INT4, enabling longer contexts or larger batches under the same memory budget\. Since all values in a quantization group share one scale, a few large values can expand the range represented by only four INT2 levels\. This increases rounding error for most values, while aggressive clipping introduces large errors in the outliers themselvesZhang et al\. \([2023](https://arxiv.org/html/2608.02691#bib.bib3)\); Hooper et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib4)\); Zhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\.

Rotation\-based methods reduce this error by spreading a few extreme channel values across dimensions, and make the cache easier to quantize\. Given an orthogonal matrixRR, a cache vectorzzis transformed toz​RzRbefore quantization and mapped back withR⊤R^\{\\top\}after dequantizationChee et al\. \([2023](https://arxiv.org/html/2608.02691#bib.bib6)\); Ashkboos et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib7)\)\. Rotation preserves the cache shape and regular memory layout, maintaining compatibility with paged KV\-cache systems and fused decoding kernelsKwon et al\. \([2023](https://arxiv.org/html/2608.02691#bib.bib8)\); Zheng et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib9)\); Zhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\. The main challenge is selectingRR\. Existing Rotation\-based INT2 KV cache pipelines rely on fixed transforms, or proxy objectives defined before the complete attention readoutAshkboos et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib7)\); Su et al\. \([2025](https://arxiv.org/html/2608.02691#bib.bib10)\); Zhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\.

![Refer to caption](https://arxiv.org/html/2608.02691v1/x1.png)Figure 1:AIME25 accuracy of Qwen3\-8B under BF16 and INT2 KV\-cache quantization\.\+\+OptR denotes applying key reparameterization and output\-aware rotation correction to the corresponding base rotation\. The dashed line marks the BF16 accuracy\.However, these proxy objectives do not directly measure the error passed to later layers\. KV quantization changes the attention readout, and the output projectionWOW\_\{O\}maps this change into the model hidden space\. The resulting output error enters the residual stream and propagates through subsequent layers, potentially affecting the final prediction\. Consequently, the rotation that best reconstructs the cached keys and values may not be the one that best preserves the post\-WOW\_\{O\}output\. This objective mismatch motivates optimizing rotations in output space\.

To address this mismatch, we proposeOptR, an output\-aware rotation method for INT2 KV\-cache quantization\. OptR first centers the keys before rotation and quantization\. This shifts all logits for a query by the same constant and therefore leaves the softmax distribution unchanged\. Since INT2 has only four quantization levels, outliers can lead to large quantization errors\. This reparameterization reduces their effect by narrowing the quantization range\. OptR then learns per\-head orthogonal corrections to any base rotation by minimizing the post\-WOW\_\{O\}attention\-output error through the INT2 attention path\. It optimizes the key rotation before the value rotation because the quantized keys determine the attention weights used for value aggregation\. Only the rotation parameters are optimized on calibration data\. Model weights remain frozen, and the learned rotations are fixed during inference\.

We integrate OptR into an SGLang\-based INT2 KV cache pipeline while retaining paged and prefix\-cache support with negligible runtime overheadZhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\. Figure[1](https://arxiv.org/html/2608.02691#S1.F1)shows that OptR improves AIME25 accuracy on Qwen3\-8B from 17\.33% to 66\.67% with QuaRot and from 54\.67% to 66\.00% with OSCAR, compared with 68\.00% for BF16\. The gains with both base rotations show that OptR does not depend on a specific rotation initialization\.

Our contributions are summarized as follows\.

- •We formulate INT2 KV\-cache quantization as an output\-space optimization problem and decompose the post\-WOW\_\{O\}attention\-output error into key\- and value\-induced terms\.
- •We proposeOptR, which first applies attention\-equivalent key reparameterization to reduce large channel\-wise offsets and then learns per\-head orthogonal corrections through the complete INT2 quantization and attention path\.
- •We show that OptR consistently improves existing rotation\-based INT2 KV\-cache pipelines while retaining their cache layout with negligible serving overhead\.

## 2Related Works

#### KV\-cache quantization\.

The KV cache grows with context length and is repeatedly read during decoding, making it a major memory and bandwidth bottleneck\. Prior work reduces this cost through fine\-grained quantization, mixed precision, and vector quantizationLiu et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib11)\); Hooper et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib4)\); Xia et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib12)\); Son et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib13)\)\. These designs often introduce residual buffers, channel\-wise metadata, promoted high\-precision channels, or specialized cache layouts, which complicate their integration with paged KV\-cache systems and fused decoding kernels\. In contrast, rotation\-based quantization transforms cached vectors into a quantization\-friendly basis without changing their tensor shape or regular cache layout, making it easier to deploy in existing inference systemsAshkboos et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib7)\); Zhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\.

#### Rotation\-based KV\-cache quantization\.

QuaRot uses Hadamard rotations for weights, activations, and KV cachesAshkboos et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib7)\), while RotateKV adapts rotations to head\-specific key outliers and protects attention sinksSu et al\. \([2025](https://arxiv.org/html/2608.02691#bib.bib10)\)\. OSCAR derives key and value rotations from offline attention\-aware covariance statisticsZhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\. Despite these differences, existing methods select rotations using fixed transforms, cache statistics, or proxy objectives defined before the complete attention readout\. Instead, OptR optimizes rotations against the post\-WOW\_\{O\}attention\-output error produced by the complete INT2 attention path\.

## 3Problem Formulation

#### Preliminaries\.

We consider a decoder\-only Transformer layerℓ\\ellwith grouped\-query attention \(GQA\)Vaswani et al\. \([2017](https://arxiv.org/html/2608.02691#bib.bib14)\); Ainslie et al\. \([2023](https://arxiv.org/html/2608.02691#bib.bib2)\)\. Leth∈\{1,…,Hkv\}h\\in\\\{1,\\ldots,H\_\{\\mathrm\{kv\}\}\\\}denote a KV head, and letGh⊆\{1,…,Hq\}G\_\{h\}\\subseteq\\\{1,\\ldots,H\_\{q\}\\\}denote the set of query heads that share this KV head\. For each query headj∈Ghj\\in G\_\{h\}, we writeqt,jℓ,ks,hℓ,vs,hℓ∈ℝ1×d,q\_\{t,j\}^\{\\ell\},k\_\{s,h\}^\{\\ell\},v\_\{s,h\}^\{\\ell\}\\in\\mathbb\{R\}^\{1\\times d\},wheres≤ts\\leq t,ttis the current decoding position, andssindexes a cached source token\. Equivalently, the cached keys and values for KV headhhup to positionttareK1:t,hℓ=\[k1,hℓ;…;kt,hℓ\]∈ℝt×d,K\_\{1:t,h\}^\{\\ell\}=\[k\_\{1,h\}^\{\\ell\};\\ldots;k\_\{t,h\}^\{\\ell\}\]\\in\\mathbb\{R\}^\{t\\times d\},andV1:t,hℓ=\[v1,hℓ;…;vt,hℓ\]∈ℝt×d\.V\_\{1:t,h\}^\{\\ell\}=\[v\_\{1,h\}^\{\\ell\};\\ldots;v\_\{t,h\}^\{\\ell\}\]\\in\\mathbb\{R\}^\{t\\times d\}\.The BF16 attention logits and probabilities are:

at,sℓ,j,h=⟨qt,jℓ,ks,hℓ⟩d,ptℓ,j,h=softmaxs≤t⁡\(at,sℓ,j,h\)a\_\{t,s\}^\{\\ell,j,h\}=\\frac\{\\langle q\_\{t,j\}^\{\\ell\},k\_\{s,h\}^\{\\ell\}\\rangle\}\{\\sqrt\{d\}\},\\quad p\_\{t\}^\{\\ell,j,h\}=\\operatorname\{softmax\}\_\{s\\leq t\}\\left\(a\_\{t,s\}^\{\\ell,j,h\}\\right\)\(1\)The corresponding attention output is

ot,jℓ=∑s≤tpt,sℓ,j,h​vs,hℓ=\(ptℓ,j,h\)⊤​V1:t,hℓ∈ℝ1×do\_\{t,j\}^\{\\ell\}=\\sum\_\{s\\leq t\}p\_\{t,s\}^\{\\ell,j,h\}v\_\{s,h\}^\{\\ell\}=\(p\_\{t\}^\{\\ell,j,h\}\)^\{\\top\}V\_\{1:t,h\}^\{\\ell\}\\in\\mathbb\{R\}^\{1\\times d\}\(2\)LetWO,jℓ∈ℝdmodel×dW\_\{O,j\}^\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{model\}\}\\times d\}be the output projection for query headjj\. The attention\-output contribution of this head is:

yt,jℓ=\(\(ptℓ,j,h\)⊤​V1:t,hℓ\)​\(WO,jℓ\)⊤∈ℝ1×dmodely\_\{t,j\}^\{\\ell\}=\\left\(\\left\(p\_\{t\}^\{\\ell,j,h\}\\right\)^\{\\top\}V\_\{1:t,h\}^\{\\ell\}\\right\)\(W\_\{O,j\}^\{\\ell\}\)^\{\\top\}\\in\\mathbb\{R\}^\{1\\times d\_\{\\mathrm\{model\}\}\}\(3\)Thus, the KV cache affects the model through the attention\-weighted readout\(ptℓ,j,h\)⊤​V1:t,hℓ\(p\_\{t\}^\{\\ell,j,h\}\)^\{\\top\}V\_\{1:t,h\}^\{\\ell\}and its projection byWO,jℓW\_\{O,j\}^\{\\ell\}\.

![Refer to caption](https://arxiv.org/html/2608.02691v1/x2.png)Figure 2:Key magnitude \(top\) and key\-induced attention\-output error by cached token \(bottom\) across five INT2 KV\-cache settings on Qwen3\-8B \(AIME25\): OSCAR only, plain INT2, reparameterization \(mean shift\), reparameterization with OSCAR rotation, and the full OptR pipeline\. Lower is better; example details are provided in the Appendix\.
### 3\.1Rotated INT2 KV\-Cache Quantization

We consider long\-context decoding where the long\-history KV cache is stored in INT2 and a small BF16 window is preserved\. LetQ2​\(⋅;c,G\)Q\_\{2\}\(\\cdot;c,G\)denote the INT2 quantize\-dequantize map with clipping ratioccand group sizeGG\. For an orthogonal rotationR∈O​\(d\)R\\in O\(d\), define

DR,c​\(z\)=Q2​\(z​R;c,G\)​R⊤D\_\{R,c\}\(z\)=Q\_\{2\}\(zR;c,G\)R^\{\\top\}\(4\)IfQ2Q\_\{2\}is replaced by the identity map, thenDR,c​\(z\)=zD\_\{R,c\}\(z\)=z\. Thus, the rotation only changes the coordinate system in which INT2 quantization error is introduced\. We denote byk~s,hℓ\\widetilde\{k\}\_\{s,h\}^\{\\ell\}andv~s,hℓ\\widetilde\{v\}\_\{s,h\}^\{\\ell\}the effective keys and values used by attention after rotated INT2 quantization and BF16 window restoration\.

### 3\.2Output\-Space Error Induced by KV Quantization

We now trace the effective INT2 cache through attention andWOW\_\{O\}and decompose the resulting post\-WOW\_\{O\}attention\-output error into key\- and value\-induced terms\. With INT2 keys, the attention logits and probabilities become

a~t,sℓ,j,h\\displaystyle\\widetilde\{a\}\_\{t,s\}^\{\\ell,j,h\}=⟨qt,jℓ,k~s,hℓ⟩d,p~tℓ,j,h=softmaxs≤t⁡\(a~t,sℓ,j,h\)\\displaystyle=\\frac\{\\langle q\_\{t,j\}^\{\\ell\},\\widetilde\{k\}\_\{s,h\}^\{\\ell\}\\rangle\}\{\\sqrt\{d\}\},\\quad\\widetilde\{p\}\_\{t\}^\{\\ell,j,h\}=\\operatorname\{softmax\}\_\{s\\leq t\}\\left\(\\widetilde\{a\}\_\{t,s\}^\{\\ell,j,h\}\\right\)\(5\)
LetΔ​ks,hℓ=k~s,hℓ−ks,hℓ\\Delta k\_\{s,h\}^\{\\ell\}=\\widetilde\{k\}\_\{s,h\}^\{\\ell\}\-k\_\{s,h\}^\{\\ell\}andΔ​vs,hℓ=v~s,hℓ−vs,hℓ\\Delta v\_\{s,h\}^\{\\ell\}=\\widetilde\{v\}\_\{s,h\}^\{\\ell\}\-v\_\{s,h\}^\{\\ell\}denote the key and value quantization errors\. Key errors first perturb the attention logits:

Δ​at,sℓ,j,h=a~t,sℓ,j,h−at,sℓ,j,h=⟨qt,jℓ,Δ​ks,hℓ⟩d\\Delta a\_\{t,s\}^\{\\ell,j,h\}=\\widetilde\{a\}\_\{t,s\}^\{\\ell,j,h\}\-a\_\{t,s\}^\{\\ell,j,h\}=\\frac\{\\langle q\_\{t,j\}^\{\\ell\},\\Delta k\_\{s,h\}^\{\\ell\}\\rangle\}\{\\sqrt\{d\}\}\(6\)The resulting attention error isΔ​ptℓ,j,h=p~tℓ,j,h−ptℓ,j,h\\Delta p\_\{t\}^\{\\ell,j,h\}=\\widetilde\{p\}\_\{t\}^\{\\ell,j,h\}\-p\_\{t\}^\{\\ell,j,h\}\.

Thus, the effect of a key error depends on the query and the softmax attention map, not only on‖Δ​k‖22\\\|\\Delta k\\\|\_\{2\}^\{2\}\.

Under INT2 keys and values, the attention\-output contribution becomes

y~t,jℓ=\(∑s≤tp~t,sℓ,j,h​v~s,hℓ\)​\(WO,jℓ\)⊤\\widetilde\{y\}\_\{t,j\}^\{\\ell\}=\\left\(\\sum\_\{s\\leq t\}\\widetilde\{p\}\_\{t,s\}^\{\\ell,j,h\}\\widetilde\{v\}\_\{s,h\}^\{\\ell\}\\right\)\(W\_\{O,j\}^\{\\ell\}\)^\{\\top\}\(7\)
Subtracting the BF16 readout gives the exact decomposition\. LetΔ​yt,jℓ:=y~t,jℓ−yt,jℓ\\Delta y\_\{t,j\}^\{\\ell\}:=\\widetilde\{y\}\_\{t,j\}^\{\\ell\}\-y\_\{t,j\}^\{\\ell\}\. Then

Δ​yt,jℓ=\(∑s≤tΔ​pt,sℓ,j,h​vs,hℓ\)​\(WO,jℓ\)⊤⏟key\-induced output error\+\(∑s≤tp~t,sℓ,j,h​Δ​vs,hℓ\)​\(WO,jℓ\)⊤⏟value\-induced output error\\displaystyle\\Delta y\_\{t,j\}^\{\\ell\}=\\underbrace\{\\left\(\\sum\_\{s\\leq t\}\\Delta p\_\{t,s\}^\{\\ell,j,h\}v\_\{s,h\}^\{\\ell\}\\right\)\\left\(W\_\{O,j\}^\{\\ell\}\\right\)^\{\\top\}\}\_\{\\text\{key\-induced output error\}\}\+\\underbrace\{\\left\(\\sum\_\{s\\leq t\}\\widetilde\{p\}\_\{t,s\}^\{\\ell,j,h\}\\Delta v\_\{s,h\}^\{\\ell\}\\right\)\\left\(W\_\{O,j\}^\{\\ell\}\\right\)^\{\\top\}\}\_\{\\text\{value\-induced output error\}\}\(8\)
We denote the two terms above byδ​yK,t,jℓ,h\\delta y\_\{K,t,j\}^\{\\ell,h\}andδ​yV,t,jℓ,h\\delta y\_\{V,t,j\}^\{\\ell,h\}, respectively\. Hereδ​yK,t,jℓ,h\\delta y\_\{K,t,j\}^\{\\ell,h\}is the output error induced by key quantization through the attention distribution, whileδ​yV,t,jℓ,h\\delta y\_\{V,t,j\}^\{\\ell,h\}is the value error after attention\-weighted aggregation and output projection\.

This decomposition shows why raw cache reconstruction is only a proxy\. A reconstruction\-based objective measures

Erec=‖K−K~‖F2\+‖V−V~‖F2E\_\{\\mathrm\{rec\}\}=\\\|K\-\\widetilde\{K\}\\\|\_\{F\}^\{2\}\+\\\|V\-\\widetilde\{V\}\\\|\_\{F\}^\{2\}\(9\)whereas the model observes the attention\-output error

Eout=‖y~t,jℓ−yt,jℓ‖22=‖δ​yK,t,jℓ,h\+δ​yV,t,jℓ,h‖22E\_\{\\mathrm\{out\}\}=\\left\\\|\\widetilde\{y\}\_\{t,j\}^\{\\ell\}\-y\_\{t,j\}^\{\\ell\}\\right\\\|\_\{2\}^\{2\}=\\left\\\|\\delta y\_\{K,t,j\}^\{\\ell,h\}\+\\delta y\_\{V,t,j\}^\{\\ell,h\}\\right\\\|\_\{2\}^\{2\}\(10\)Eq\. \([9](https://arxiv.org/html/2608.02691#S3.E9)\) and Eq\. \([10](https://arxiv.org/html/2608.02691#S3.E10)\) can favor different rotations because attention andWOW\_\{O\}reduce the effect of some cache errors while allowing others to affect the attention\-output\. For this reason, OptR usesEoutE\_\{\\mathrm\{out\}\}as its rotation optimization target\.

![Refer to caption](https://arxiv.org/html/2608.02691v1/x3.png)Figure 3:Overview of OptR\. OptR augments an existing rotation\-based KV\-cache quantization pipeline with channel\-wise key reparameterization and output\-aware correction rotations before INT2 quantization\. The resulting quantized KV cache is consumed by attention and projected throughWOW\_\{O\}, where OptR targets reduced post\-WOW\_\{O\}attention\-output error\.

## 4Method: Output\-Aware Rotation

Figure[5](https://arxiv.org/html/2608.02691#S5.F5)shows that INT2\-induced output error differs across KV heads\. OptR therefore learns a separate orthogonal correction to the base key and value rotations for each head\. During one\-time offline calibration, model weights remain frozen and only the corrections are optimized to reduce post\-WOW\_\{O\}attention\-output error through the INT2 attention path\. We optimize the key rotation first because the INT2 keys determine the attention distribution, and then optimize the value rotation under this distribution\. The resulting rotations are fixed during inference\. Figure[3](https://arxiv.org/html/2608.02691#S3.F3)summarizes the full pipeline\. We omit\(ℓ,h\)\(\\ell,h\)when the layer and KV head are clear\.

### 4\.1Rotated INT2 Cache Reparameterization

As shown in Figure[2](https://arxiv.org/html/2608.02691#S3.F2), large channel\-wise key offsets increase the dynamic range of group\-wise INT2 quantization\. Therefore, we reparameterize keys using the per\-channel calibration meanμ∈ℝd\\mu\\in\\mathbb\{R\}^\{d\}before rotation and quantization\. For key and value rotationsRKR\_\{K\}andRVR\_\{V\}, OptR defines

k¯s​\(RK\)\\displaystyle\\bar\{k\}\_\{s\}\(R\_\{K\}\)=DRK,cK​\(ks−μ\),v¯s​\(RV\)=DRV,cV​\(vs\)\\displaystyle=D\_\{R\_\{K\},c\_\{K\}\}\(k\_\{s\}\-\\mu\),\\quad\\bar\{v\}\_\{s\}\(R\_\{V\}\)=D\_\{R\_\{V\},c\_\{V\}\}\(v\_\{s\}\)
This key reparameterization is attention\-equivalent: subtracting the sameμ\\mufrom every key adds a query\-dependent constant to all logits and leaves the softmax distribution unchanged\. We apply it only to keys, since shifting values would alter the attention output\. Sink and recent tokens remain in BF16, and the same centering is applied to their keys\. The effective cache is denoted byk~s​\(RK\)\\widetilde\{k\}\_\{s\}\(R\_\{K\}\)andv~s​\(RV\)\\widetilde\{v\}\_\{s\}\(R\_\{V\}\)\.

### 4\.2Orthogonal Rotation Refinement

OptR does not rely on a specific rotation initialization\. LetRK0,RV0∈O​\(d\)R\_\{K\}^\{0\},R\_\{V\}^\{0\}\\in O\(d\)denote arbitrary orthogonal initializations for the key and value rotations, respectively\. Given these initial rotations, OptR applies the same output\-aware rotation procedure regardless of how they are constructed\.

For each KV head, OptR learns unconstrained matricesAK,AV∈ℝd×dA\_\{K\},A\_\{V\}\\in\\mathbb\{R\}^\{d\\times d\}and forms the skew\-symmetric generators

SK=AK−AK⊤,SV=AV−AV⊤S\_\{K\}=A\_\{K\}\-A\_\{K\}^\{\\top\},\\quad S\_\{V\}=A\_\{V\}\-A\_\{V\}^\{\\top\}\(11\)which define the corrected rotations

RK​\(AK\)=RK0​exp⁡\(SK\),RV​\(AV\)=RV0​exp⁡\(SV\)R\_\{K\}\(A\_\{K\}\)=R\_\{K\}^\{0\}\\exp\(S\_\{K\}\),\\quad R\_\{V\}\(A\_\{V\}\)=R\_\{V\}^\{0\}\\exp\(S\_\{V\}\)SinceSKS\_\{K\}andSVS\_\{V\}are skew\-symmetric, their matrix exponentials are orthogonal\. Therefore, the corrected rotations remain orthogonal throughout calibration\. The same formulation applies to different rotation initializations without modifying the calibration objective\.

### 4\.3Output\-Aware Key Calibration

The key rotation affects attention logits and the attention distribution next\. To isolate this path, we calibrate the key rotation while keeping values in BF16\. The induced key\-only attention distribution is

at,sK​\(RK\)=⟨qt,j,k~s​\(RK\)⟩d,ptK​\(RK\)=softmaxs≤t⁡\(at,sK​\(RK\)\)\\displaystyle a\_\{t,s\}^\{K\}\(R\_\{K\}\)=\\frac\{\\langle q\_\{t,j\},\\widetilde\{k\}\_\{s\}\(R\_\{K\}\)\\rangle\}\{\\sqrt\{d\}\},\\qquad p\_\{t\}^\{K\}\(R\_\{K\}\)=\\operatorname\{softmax\}\_\{s\\leq t\}\\left\(a\_\{t,s\}^\{K\}\(R\_\{K\}\)\\right\)\(12\)The key\-induced attention\-output error is

eK​\(t,j;RK\)=\[∑s≤t\(pt,sK​\(RK\)−pt,s\)​vs\]​\(WO,j\)⊤e\_\{K\}\(t,j;R\_\{K\}\)=\\left\[\\sum\_\{s\\leq t\}\\left\(p\_\{t,s\}^\{K\}\(R\_\{K\}\)\-p\_\{t,s\}\\right\)v\_\{s\}\\right\]\(W\_\{O,j\}\)^\{\\top\}\(13\)We optimize the key rotation with

LK​\(RK;D\)=𝔼\(t,j\)∈D​\[DKL​\(pt∥ptK​\(RK\)\)\+λK​‖eK​\(t,j;RK\)‖22dmodel\]\\displaystyle L\_\{K\}\(R\_\{K\};D\)=\\mathbb\{E\}\_\{\(t,j\)\\in D\}\\Bigg\[D\_\{\\mathrm\{KL\}\}\\left\(p\_\{t\}\\,\\\|\\,p\_\{t\}^\{K\}\(R\_\{K\}\)\\right\)\+\\lambda\_\{K\}\\frac\{\\\|e\_\{K\}\(t,j;R\_\{K\}\)\\\|\_\{2\}^\{2\}\}\{d\_\{\\mathrm\{model\}\}\}\\Bigg\]\(14\)The KL term preserves the attention distribution, while the second term measures the post\-WOW\_\{O\}attention\-output error caused by key\-induced attention changes\.

### 4\.4Output\-Aware Value Calibration

After optimizing the key rotation, we calibrate the value rotation under the selected quantized\-key attention path\. LetR^K\\widehat\{R\}\_\{K\}be the selected key rotation and letp^tK\\widehat\{p\}\_\{t\}^\{K\}be its induced attention distribution\.

The value\-induced attention\-output error is

eV​\(t,j;RV\)=\[∑s≤tp^t,sK​\(v~s​\(RV\)−vs\)\]​\(WO,j\)⊤e\_\{V\}\(t,j;R\_\{V\}\)=\\left\[\\sum\_\{s\\leq t\}\\widehat\{p\}\_\{t,s\}^\{K\}\\left\(\\widetilde\{v\}\_\{s\}\(R\_\{V\}\)\-v\_\{s\}\\right\)\\right\]\(W\_\{O,j\}\)^\{\\top\}\(15\)We optimize the value rotation with

LV​\(RV;D\)=𝔼\(t,j\)∈D​\[‖eV​\(t,j;RV\)‖22dmodel\]L\_\{V\}\(R\_\{V\};D\)=\\mathbb\{E\}\_\{\(t,j\)\\in D\}\\left\[\\frac\{\\\|e\_\{V\}\(t,j;R\_\{V\}\)\\\|\_\{2\}^\{2\}\}\{d\_\{\\mathrm\{model\}\}\}\\right\]\(16\)This objective measures value\-induced attention\-output error after attention\-weighted aggregation and output projection, rather than raw value\-cache reconstruction\.

### 4\.5INT2\-Aware Orthogonal Calibration

OptR calibrates the rotation generators through the full INT2 quantization path: rotation, clipping, grouping, INT2 rounding, dequantization, and inverse rotation\. For each layer and KV head, OptR learns the orthogonal correction:

AK⋆=arg⁡minA⁡LK​\(RK0​exp⁡\(A−A⊤\);Dtrain\),AV⋆=arg⁡minA⁡LV​\(RV0​exp⁡\(A−A⊤\);Dtrain\)\\displaystyle A\_\{K\}^\{\\star\}=\\arg\\min\_\{A\}L\_\{K\}\\left\(R\_\{K\}^\{0\}\\exp\(A\-A^\{\\top\}\);D\_\{\\mathrm\{train\}\}\\right\),\\quad A\_\{V\}^\{\\star\}=\\arg\\min\_\{A\}L\_\{V\}\\left\(R\_\{V\}^\{0\}\\exp\(A\-A^\{\\top\}\);D\_\{\\mathrm\{train\}\}\\right\)\(17\)All model weights remain frozen; only the per\-head rotation generators are calibrated\. Since INT2 rounding is non\-differentiable, gradients are estimated using a straight\-through estimator:

∂Q2​\(Z;c,G\)∂Z≈𝟏​\{\|Z\|≤τc\}\\frac\{\\partial Q\_\{2\}\(Z;c,G\)\}\{\\partial Z\}\\approx\\mathbf\{1\}\\\{\|Z\|\\leq\\tau\_\{c\}\\\}\(18\)whereτc\\tau\_\{c\}is the clipping threshold determined by the clip ratiocc\.

## 5Experimental Results

Models and Benchmarks\.We evaluate OptR on Qwen3\-4B\-Thinking\-2507Yang et al\. \([2025](https://arxiv.org/html/2608.02691#bib.bib15)\), Qwen3\-8BYang et al\. \([2025](https://arxiv.org/html/2608.02691#bib.bib15)\), and Phi4\-14B\-reasoning\-plusAbdin et al\. \([2025](https://arxiv.org/html/2608.02691#bib.bib16)\)\. We measure reasoning and coding accuracy on AIME24, AIME25MAA \([2025](https://arxiv.org/html/2608.02691#bib.bib17)\), GPQA\-DiamondRein et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib18)\), MBPP\+Liu et al\. \([2023](https://arxiv.org/html/2608.02691#bib.bib19)\), and LiveCodeBench v6Jain et al\. \([2025](https://arxiv.org/html/2608.02691#bib.bib20)\)\. We additionally evaluate long\-context retrieval using RULER\-NIAH \(Needle\-in\-a\-Haystack\)Hsieh et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib21)\), with context lengths up to 64K for the Qwen3 models and 32K for Phi4\-14B\-reasoning\-plus\. For all models, we use a temperature of 0\.6, top\-ppof 0\.95, and top\-kkof 20\. For the reasoning and coding benchmarks, we use maximum generation lengths of 32K for the Qwen3 models and 16K for Phi4\-14B\-reasoning\-plus\.

ModelMethodBPEAIME24AIME25GPQAMBPP\+\+LCB v6MeanQwen3\-4B\-Thinking\-2507BF161678\.00±6\.9178\.00\\pm 6\.9171\.33±3\.8071\.33\\pm 3\.8064\.55±2\.3064\.55\\pm 2\.3077\.31±0\.8777\.31\\pm 0\.8746\.71±2\.0546\.71\\pm 2\.0567\.5867\.58TurboQuant \(no MP\)3\.2510\.0010\.0016\.6716\.6741\.9241\.9223\.0223\.021\.711\.7118\.6618\.66QuaRot\-INT22\.320\.00±0\.000\.00\\pm 0\.000\.00±0\.000\.00\\pm 0\.006\.06±1\.186\.06\\pm 1\.185\.61±2\.365\.61\\pm 2\.361\.37±0\.311\.37\\pm 0\.312\.612\.61\\cellcolorgray\!15QuaRot\-INT2 \+OptR\\cellcolorgray\!152\.32\\cellcolorgray\!1569\.33±4\.3569\.33\\pm 4\.35\\cellcolorgray\!1560\.67±4\.3560\.67\\pm 4\.35\\cellcolorgray\!1562\.12±2\.9262\.12\\pm 2\.92\\cellcolorgray\!1578\.31±1\.23\\mathbf\{78\.31\\pm 1\.23\}\\cellcolorgray\!1537\.71±0\.8137\.71\\pm 0\.81\\cellcolorgray\!1561\.6361\.63OSCAR2\.3268\.67±2\.9868\.67\\pm 2\.9863\.33±4\.0863\.33\\pm 4\.0862\.02±1\.7362\.02\\pm 1\.7375\.66±0\.8475\.66\\pm 0\.8443\.89±0\.7543\.89\\pm 0\.7562\.7162\.71\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!152\.32\\cellcolorgray\!1572\.00±1\.83\\mathbf\{72\.00\\pm 1\.83\}\\cellcolorgray\!1570\.67±2\.79\\mathbf\{70\.67\\pm 2\.79\}\\cellcolorgray\!1562\.83±2\.22\\mathbf\{62\.83\\pm 2\.22\}\\cellcolorgray\!1576\.83±0\.9176\.83\\pm 0\.91\\cellcolorgray\!1544\.57±0\.90\\mathbf\{44\.57\\pm 0\.90\}\\cellcolorgray\!1565\.38\\mathbf\{65\.38\}Qwen3\-8BBF161676\.00±6\.4176\.00\\pm 6\.4168\.00±2\.9868\.00\\pm 2\.9855\.76±2\.1655\.76\\pm 2\.1679\.15±1\.4879\.15\\pm 1\.4849\.37±0\.6549\.37\\pm 0\.6565\.6665\.66TurboQuant \(no MP\)3\.2553\.3353\.3340\.0040\.0045\.9645\.9658\.9958\.9921\.1421\.1443\.8843\.88QuaRot\-INT22\.3216\.00±5\.4816\.00\\pm 5\.4817\.33±4\.3517\.33\\pm 4\.3542\.12±1\.5442\.12\\pm 1\.5449\.10±1\.0049\.10\\pm 1\.005\.26±1\.245\.26\\pm 1\.2425\.9625\.96\\cellcolorgray\!15QuaRot\-INT2 \+OptR\\cellcolorgray\!152\.32\\cellcolorgray\!1576\.00±4\.3576\.00\\pm 4\.35\\cellcolorgray\!1566\.67±3\.33\\mathbf\{66\.67\\pm 3\.33\}\\cellcolorgray\!1557\.58±2\.4557\.58\\pm 2\.45\\cellcolorgray\!1580\.63±1\.14\\mathbf\{80\.63\\pm 1\.14\}\\cellcolorgray\!1539\.77±1\.5439\.77\\pm 1\.54\\cellcolorgray\!1564\.1364\.13OSCAR2\.3272\.00±3\.8072\.00\\pm 3\.8054\.67±3\.8054\.67\\pm 3\.8056\.26±3\.2556\.26\\pm 3\.2578\.73±0\.4478\.73\\pm 0\.4445\.03±2\.2645\.03\\pm 2\.2661\.3461\.34\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!152\.32\\cellcolorgray\!1576\.67±2\.36\\mathbf\{76\.67\\pm 2\.36\}\\cellcolorgray\!1566\.00±4\.9466\.00\\pm 4\.94\\cellcolorgray\!1558\.99±2\.56\\mathbf\{58\.99\\pm 2\.56\}\\cellcolorgray\!1579\.21±0\.5579\.21\\pm 0\.55\\cellcolorgray\!1547\.54±0\.48\\mathbf\{47\.54\\pm 0\.48\}\\cellcolorgray\!1565\.68\\mathbf\{65\.68\}Phi4\-14B\-reasoning\-plusBF161670\.00±4\.7170\.00\\pm 4\.7160\.67±4\.9460\.67\\pm 4\.9445\.56±2\.0945\.56\\pm 2\.0977\.18±2\.1677\.18\\pm 2\.1639\.43±2\.0639\.43\\pm 2\.0658\.5758\.57TurboQuant \(no MP\)3\.2560\.0060\.0053\.3353\.3346\.9746\.9774\.6074\.6033\.1433\.1453\.6153\.61QuaRot\-INT22\.3261\.33±4\.4761\.33\\pm 4\.4746\.00±5\.4846\.00\\pm 5\.4846\.77±3\.4046\.77\\pm 3\.4075\.82±1\.1675\.82\\pm 1\.1634\.97±1\.4834\.97\\pm 1\.4852\.9852\.98\\cellcolorgray\!15QuaRot\-INT2 \+OptR\\cellcolorgray\!152\.32\\cellcolorgray\!1564\.00±3\.6564\.00\\pm 3\.65\\cellcolorgray\!1552\.00±2\.9852\.00\\pm 2\.98\\cellcolorgray\!1547\.58±2\.91\\mathbf\{47\.58\\pm 2\.91\}\\cellcolorgray\!1576\.70±0\.67\\mathbf\{76\.70\\pm 0\.67\}\\cellcolorgray\!1536\.00±0\.57\\mathbf\{36\.00\\pm 0\.57\}\\cellcolorgray\!1555\.2655\.26OSCAR2\.3262\.67±2\.7962\.67\\pm 2\.7949\.33±3\.6549\.33\\pm 3\.6543\.84±2\.7143\.84\\pm 2\.7173\.02±1\.4473\.02\\pm 1\.4434\.29±1\.0434\.29\\pm 1\.0452\.6352\.63\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!152\.32\\cellcolorgray\!1565\.33±5\.06\\mathbf\{65\.33\\pm 5\.06\}\\cellcolorgray\!1558\.00±5\.58\\mathbf\{58\.00\\pm 5\.58\}\\cellcolorgray\!1546\.16±1\.5046\.16\\pm 1\.50\\cellcolorgray\!1573\.44±0\.5573\.44\\pm 0\.55\\cellcolorgray\!1535\.09±1\.1135\.09\\pm 1\.11\\cellcolorgray\!1555\.60\\mathbf\{55\.60\}

Table 1:Comparison of INT2 KV\-cache quantization methods across three model configurations and five benchmarks\. Results are reported asμ±σ\\mu\\pm\\sigmaover five seeds\. OptR indicates that output\-aware rotation is applied to the corresponding baseline\. BPE denotes the effective number of bits per KV\-cache element\. TurboQuant is reported from a single run because repeated 32K evaluations are slow in its vLLM implementation\.ModelMethod4k8k16k32k64kQwen3\-4B\-Thinking\-2507BF1699\.90±0\.0899\.90\\pm 0\.0899\.60±0\.0799\.60\\pm 0\.0798\.19±0\.0898\.19\\pm 0\.0897\.13±0\.1197\.13\\pm 0\.1188\.76±0\.4788\.76\\pm 0\.47QuaRot\-INT23\.08±0\.193\.08\\pm 0\.198\.26±0\.858\.26\\pm 0\.850\.00±0\.000\.00\\pm 0\.006\.14±1\.116\.14\\pm 1\.110\.72±0\.160\.72\\pm 0\.16\\cellcolorgray\!15QuaRot\-INT2 \+OptR\\cellcolorgray\!1599\.78±0\.17\\mathbf\{99\.78\\pm 0\.17\}\\cellcolorgray\!1599\.02±0\.38\\mathbf\{99\.02\\pm 0\.38\}\\cellcolorgray\!1595\.59±0\.52\\mathbf\{95\.59\\pm 0\.52\}\\cellcolorgray\!1574\.69±0\.99\{74\.69\\pm 0\.99\}\\cellcolorgray\!1546\.86±1\.77\{46\.86\\pm 1\.77\}OSCAR99\.29±0\.3199\.29\\pm 0\.3197\.59±0\.5097\.59\\pm 0\.5093\.78±0\.3393\.78\\pm 0\.3377\.75±0\.6477\.75\\pm 0\.6455\.72±1\.4455\.72\\pm 1\.44\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!1599\.42±0\.27\{99\.42\\pm 0\.27\}\\cellcolorgray\!1598\.57±0\.17\{98\.57\\pm 0\.17\}\\cellcolorgray\!1595\.31±0\.10\{95\.31\\pm 0\.10\}\\cellcolorgray\!1584\.98±0\.45\\mathbf\{84\.98\\pm 0\.45\}\\cellcolorgray\!1569\.58±1\.31\\mathbf\{69\.58\\pm 1\.31\}Qwen3\-8BBF1699\.83±0\.1199\.83\\pm 0\.1199\.93±0\.0299\.93\\pm 0\.0299\.45±0\.0899\.45\\pm 0\.0898\.70±0\.4998\.70\\pm 0\.4984\.22±1\.4184\.22\\pm 1\.41QuaRot\-INT284\.97±0\.0984\.97\\pm 0\.0941\.73±5\.7541\.73\\pm 5\.7518\.16±1\.1418\.16\\pm 1\.1413\.03±1\.7113\.03\\pm 1\.710\.04±0\.070\.04\\pm 0\.07\\cellcolorgray\!15QuaRot\-INT2 \+OptR\\cellcolorgray\!1599\.40±0\.29\{99\.40\\pm 0\.29\}\\cellcolorgray\!1598\.76±0\.57\\mathbf\{98\.76\\pm 0\.57\}\\cellcolorgray\!1596\.14±0\.35\\mathbf\{96\.14\\pm 0\.35\}\\cellcolorgray\!1586\.37±0\.72\{86\.37\\pm 0\.72\}\\cellcolorgray\!1570\.02±1\.79\\mathbf\{70\.02\\pm 1\.79\}OSCAR99\.59±0\.1599\.59\\pm 0\.1597\.94±0\.2897\.94\\pm 0\.2894\.39±0\.2994\.39\\pm 0\.2983\.76±0\.5883\.76\\pm 0\.5857\.54±1\.8257\.54\\pm 1\.82\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!1599\.60±0\.11\\mathbf\{99\.60\\pm 0\.11\}\\cellcolorgray\!1598\.16±0\.63\{98\.16\\pm 0\.63\}\\cellcolorgray\!1595\.50±0\.65\{95\.50\\pm 0\.65\}\\cellcolorgray\!1586\.42±0\.55\\mathbf\{86\.42\\pm 0\.55\}\\cellcolorgray\!1568\.65±0\.80\{68\.65\\pm 0\.80\}Phi4\-14B\-reasoning\-plusBF1698\.50±0\.4598\.50\\pm 0\.4597\.54±0\.5997\.54\\pm 0\.5997\.76±0\.7097\.76\\pm 0\.7092\.13±1\.3492\.13\\pm 1\.34N/AQuaRot\-INT296\.28±0\.4496\.28\\pm 0\.4492\.28±1\.3192\.28\\pm 1\.3184\.54±0\.5284\.54\\pm 0\.5269\.65±0\.9469\.65\\pm 0\.94N/A\\cellcolorgray\!15QuaRot\-INT2 \+OptR\\cellcolorgray\!1597\.62±0\.21\\mathbf\{97\.62\\pm 0\.21\}\\cellcolorgray\!1594\.02±0\.80\\mathbf\{94\.02\\pm 0\.80\}\\cellcolorgray\!1589\.98±0\.52\\mathbf\{89\.98\\pm 0\.52\}\\cellcolorgray\!1576\.21±0\.53\\mathbf\{76\.21\\pm 0\.53\}\\cellcolorgray\!15N/AOSCAR95\.62±1\.0295\.62\\pm 1\.0290\.50±0\.2490\.50\\pm 0\.2484\.59±0\.8684\.59\\pm 0\.8670\.09±0\.5970\.09\\pm 0\.59N/A\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!1596\.30±0\.37\{96\.30\\pm 0\.37\}\\cellcolorgray\!1593\.42±0\.35\{93\.42\\pm 0\.35\}\\cellcolorgray\!1587\.64±0\.45\{87\.64\\pm 0\.45\}\\cellcolorgray\!1575\.36±0\.85\{75\.36\\pm 0\.85\}\\cellcolorgray\!15N/A

Table 2:RULER\-NIAH long\-context retrieval accuracy evaluated at context lengths ranging from 4k to 64k tokens\. Results are reported asμ±σ\\mu\\pm\\sigmaover three random seeds, with 800 examples evaluated per seed\. OptR indicates that output\-aware rotation is applied to the corresponding baseline\. Since Phi4\-14B\-reasoning\-plus supports a maximum context length of 32k, its evaluation is limited to 32k, and the 64k setting is reported as N/A\. Additional 128K results are in the Appendix\.#### Implementation Details\.

OptR is implemented on top of the official OSCAR codebaseZhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\. We use group\-wise affine INT2 quantization with a group size of 128, clipping ratios of 0\.96 for keys and 0\.92 for values, and retain 64 sink tokens and 256 recent tokens in BF16\.λK\\lambda\_\{K\}is fixed to 1\.0 across all models and benchmarks\. We report an effective cache cost of 2\.32 bits per element \(BPE\) at a 64K\-token context, including quantization metadata and the BF16 windows\. For each model, we collect a 30K\-token pool of BF16 GPQA QKV traces and use disjoint subsets for calibration and held\-out rotation selection\. For each layer and KV head, we optimize only the rotation corrections for 80 Adam steps with a learning rate of 0\.02Kingma and Ba \([2014](https://arxiv.org/html/2608.02691#bib.bib22)\)\. All model weights remain frozen\. All experiments were conducted on four NVIDIA A100 40GB GPUs\. Additional implementation details are provided in the Appendix\.

#### Baselines\.

We compare OptR with BF16, TurboQuantZandieh et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib23)\)without mixed precision \(no MP\), QuaRot\-INT2Ashkboos et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib7)\), and OSCARZhou et al\. \([2026](https://arxiv.org/html/2608.02691#bib.bib5)\)\. OptR is applied to both QuaRot and OSCAR to evaluate its effectiveness across fixed Hadamard rotations and the state\-of\-the\-art attention\-aware covariance rotation\. All rotation\-based methods use the same INT2 group size and BF16 sink and recent\-token windows\.

### 5\.1Main Results

#### Overall Accuracy\.

Table[1](https://arxiv.org/html/2608.02691#S5.T1)compares OptR with BF16 and existing INT2 KV\-cache quantization methods\. QuaRot\-INT2 often suffers from severe accuracy degradation, particularly on reasoning and coding tasks\. Applying OptR substantially improves both QuaRot and OSCAR at the same 2\.32 BPE across the evaluated models and benchmarks\.

![Refer to caption](https://arxiv.org/html/2608.02691v1/x4.png)Figure 4:Per\-layer errors under INT2 KV\-cache quantization on Qwen3\-4B\-Thinking\-2507\. Panels \(a,b\) show post\-WOW\_\{O\}attention\-output error on GPQA\-Diamond and AIME25 at each layer\. Panels \(c,d\) show the propagated residual\-stream error, measured as the normalized squared error between BF16 and INT2\-KV block\-output hidden states on the same BF16\-generated token sequences\. Legend values indicate the mean across layers\. Error values are shown on a logarithmic scale\. Detailed settings are provided in Appendix\.![Refer to caption](https://arxiv.org/html/2608.02691v1/x5.png)Figure 5:Output error by layer and KV head relative to naive INT2 \(Qwen3\-4B\-Thinking\-2507, GPQA\-Diamond\)\. Each cell reports the post\-WOW\_\{O\}attention\-output RMS error as a percentage of plain per\-group INT2 without rotation\. Lower values indicate smaller output error, with naive INT2 corresponding to100%100\\%\. Panels show \(a\) TurboQuant, \(b\) QuaRot, \(c\) QuaRot \+ OptR, \(d\) OSCAR, and \(e\) OSCAR \+ OptR\. Detailed settings are provided in Appendix\.
#### Long\-Context Robustness\.

Table[2](https://arxiv.org/html/2608.02691#S5.T2)reports RULER\-NIAHHsieh et al\. \([2024](https://arxiv.org/html/2608.02691#bib.bib21)\)retrieval accuracy across increasing context lengths\. QuaRot\-INT2 degrades rapidly as the context grows, whereas OptR preserves substantially stronger retrieval performance for both QuaRot and OSCAR\. The gains become more pronounced at longer contexts, indicating that OptR more effectively mitigates accumulated KV quantization error over long histories\.

#### Output\-Error Analysis\.

Figure[4](https://arxiv.org/html/2608.02691#S5.F4)shows that OptR reduces both post\-WOW\_\{O\}attention\-output error and propagated residual\-stream error relative to QuaRot and OSCAR on GPQA\-Diamond and AIME25\. Figure[5](https://arxiv.org/html/2608.02691#S5.F5)further shows that these reductions occur across most layers and KV heads rather than only a small subset\.

#### Ablation Study\.

All ablation results are reported asμ±σ\\mu\\pm\\sigmaover five seeds\. Table[4](https://arxiv.org/html/2608.02691#S5.T4)compares cache\-reconstruction, pre\-WOW\_\{O\}, and post\-WOW\_\{O\}calibration objectives on Qwen3\-4B\-Thinking\-2507 using the same OSCAR base rotation\. The post\-WOW\_\{O\}attention\-output objective achieves the highest AIME25 accuracy\. Table[3](https://arxiv.org/html/2608.02691#S5.T3)shows that the full OptR consistently achieves the strongest performance across both base rotations and models, with key reparameterization providing an additional source of improvement\. Additional sensitivity ablation results forλK\\lambda\_\{K\}tuning are provided in the Appendix\.

MethodBF1671\.33±3\.8071\.33\\pm 3\.8060\.67±4\.9460\.67\\pm 4\.94OSCAR63\.33±4\.0863\.33\\pm 4\.0849\.33±3\.6549\.33\\pm 3\.65OSCAR \+ Key Reparam66\.00±4\.7166\.00\\pm 4\.7154\.67±6\.9154\.67\\pm 6\.91\\cellcolorgray\!15OSCAR \+OptR\\cellcolorgray\!1570\.67±2\.79\\mathbf\{70\.67\\pm 2\.79\}\\cellcolorgray\!1558\.00±5\.58\\mathbf\{58\.00\\pm 5\.58\}Table 3:Component\-wise evaluation of OptR on AIME25 across Qwen3\-4B\-Thinking\-2507 and Phi4\-14B\-reasoning\-plus using the state\-of\-the\-art OSCAR rotation\.Key ObjectiveℒK\\mathcal\{L\}\_\{K\}Value ObjectiveℒV\\mathcal\{L\}\_\{V\}AIME25DK​L\+λK​ℰ​\(K~−K\)D\_\{KL\}\+\\lambda\_\{K\}\\mathcal\{E\}\\\!\\left\(\\widetilde\{K\}\-K\\right\)ℰ​\(V~−V\)\\mathcal\{E\}\\\!\\left\(\\widetilde\{V\}\-V\\right\)65\.33±6\.9165\.33\\pm 6\.91DK​L\+λK​ℰ​\(Δ​pK⊤​V\)D\_\{KL\}\+\\lambda\_\{K\}\\mathcal\{E\}\\\!\\left\(\\Delta p\_\{K\}^\{\\top\}V\\right\)ℰ​\(\(p^tK\)⊤​Δ​V\)\\mathcal\{E\}\\\!\\left\(\(\\widehat\{p\}\_\{t\}^\{K\}\)^\{\\top\}\\Delta V\\right\)62\.67±2\.7962\.67\\pm 2\.79\\cellcolorgray\!15DKL\+λK​ℰ​\(\(Δ​pK⊤​V\)​WO,j⊤\)D\_\{\\mathrm\{KL\}\}\+\\lambda\_\{K\}\\mathcal\{E\}\\\!\\left\(\\left\(\\Delta p\_\{K\}^\{\\top\}V\\right\)W\_\{O,j\}^\{\\top\}\\right\)\\cellcolorgray\!15ℰ​\(\(\(p^tK\)⊤​Δ​V\)​WO,j⊤\)\\mathcal\{E\}\\\!\\left\(\\left\(\(\\widehat\{p\}\_\{t\}^\{K\}\)^\{\\top\}\\Delta V\\right\)W\_\{O,j\}^\{\\top\}\\right\)\\cellcolorgray\!1570\.67±2\.79\\mathbf\{70\.67\\pm 2\.79\}Table 4:Objective ablation for OptR on Qwen3\-4B\-Thinking\-2507 with the OSCAR base rotation\.ℰ​\(X\)=mean⁡\(X2\)\\mathcal\{E\}\(X\)=\\operatorname\{mean\}\(X^\{2\}\)denotes the element\-wise mean squared error\. TheDK​LD\_\{KL\}term is fixed across all rows, and the three rows vary the error term using cache reconstruction, pre\-WOW\_\{O\}attention readout, and post\-WOW\_\{O\}attention\-output error, respectively\.![Refer to caption](https://arxiv.org/html/2608.02691v1/x6.png)Figure 6:Efficiency analysis of OptR integrated into the optimized rotated INT2 KV cache pipeline on an NVIDIA A100 40GB using Qwen3\-4B\-Thinking\-2507\. \(a\) Batch\-1 decode latency across context lengths from 1K to 128K\. \(b\) End\-to\-end output throughput, for a 2K\-input and 4K\-output workload at different batch sizes\. \(c\) Prefill time for 2K\-token inputs at representative batch sizes\. Analysis shows that OptR adds negligible inference overhead to the underlying pipeline\.

## 6Discussion

#### Efficiency Analysis\.

We integrate OptR into an optimized INT2 KV\-cache pipeline and extend its Triton cache\-write kernel to support per\-KV\-head key rotations and key reparameterization\. For each headhh, reparameterization before rotation can be rewritten as

\(k−μh\)​RK,h=k​RK,h−μh​RK,h\(k\-\\mu\_\{h\}\)R\_\{K,h\}=kR\_\{K,h\}\-\\mu\_\{h\}R\_\{K,h\}We therefore precompute the rotated meanμh​RK,h\\mu\_\{h\}R\_\{K,h\}once offline and subtract it immediately after rotating each key\. The same kernel then clips, quantizes, packs, and writes the reparameterized keys and rotated values to the INT2 cache in a single launch\. The value rotation and its inverse are absorbed into the value projection andWOW\_\{O\}, respectively, avoiding separate value\-side rotation kernels\. To match the per\-KV\-head key rotations under GQA, we implement an optimized Triton kernel that applies the corresponding rotation to each query head\.

Figure[6](https://arxiv.org/html/2608.02691#S5.F6)shows that OptR remains within 2% of the corresponding base pipeline in decode latency, end\-to\-end throughput, and prefill time across the evaluated settings, while retaining the same maximum batch sizes\. For the 2K\-token input and 4K\-token output workload in Figure[6](https://arxiv.org/html/2608.02691#S5.F6)\(b\), adding OptR atB=128B=128increases GPU memory by only 18 MiB, from 36,205 MiB to 36,223 MiB\. Thus, OptR preserves the efficiency and serving capacity of the underlying INT2 pipeline\.

## 7Conclusion

We introduced OptR, an output\-aware rotation method for INT2 KV\-cache quantization\. OptR combines attention\-equivalent key reparameterization with per\-head orthogonal corrections optimized through the complete quantized attention path, directly preserving the attention output observed by subsequent layers\. Across diverse models, reasoning and coding tasks, and long\-context retrieval settings, OptR consistently strengthens existing rotation\-based methods and reduces quantization\-induced output error\. Moreover, OptR retains compatibility with paged KV\-cache serving and introduces negligible inference overhead, demonstrating that output\-aware optimization is an effective and practical direction for ultra\-low\-bit KV\-cache quantization\.

## References

- Dao et al\. \[2022\]Tri Dao, Daniel Y\. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré\.Flashattention: fast and memory\-efficient exact attention with io\-awareness\.In*Proceedings of the 36th International Conference on Neural Information Processing Systems*, 2022\.
- Ainslie et al\. \[2023\]Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\.GQA: Training generalized multi\-query transformer models from multi\-head checkpoints\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 4895–4901, 2023\.
- Zhang et al\. \[2023\]Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\.H2o: heavy\-hitter oracle for efficient generative inference of large language models\.In*Proceedings of the 37th International Conference on Neural Information Processing Systems*, 2023\.
- Hooper et al\. \[2024\]Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W\. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami\.Kvquant: towards 10 million context length llm inference with kv cache quantization\.In*Proceedings of the 38th International Conference on Neural Information Processing Systems*, 2024\.
- Zhou et al\. \[2026\]Zhongzhu Zhou, Donglin Zhuang, Jisen Li, Ziyan Chen, Shuaiwen Leon Song, Ben Athiwaratkun, and Xiaoxia Wu\.Oscar: Offline spectral covariance\-aware rotation for 2\-bit kv cache quantization\.2026\.URL[https://arxiv\.org/abs/2605\.17757](https://arxiv.org/abs/2605.17757)\.
- Chee et al\. \[2023\]Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa\.QuIP: 2\-bit quantization of large language models with guarantees\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=xrk9g5vcXR](https://openreview.net/forum?id=xrk9g5vcXR)\.
- Ashkboos et al\. \[2024\]Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman\.Quarot: Outlier\-free 4\-bit inference in rotated llms\.*Advances in Neural Information Processing Systems*, 37:100213–100240, 2024\.
- Kwon et al\. \[2023\]Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica\.Efficient memory management for large language model serving with PagedAttention\.In*Proceedings of the 29th Symposium on Operating Systems Principles \(SOSP\)*, 2023\.
- Zheng et al\. \[2024\]Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E\. Gonzalez, Clark Barrett, and Ying Sheng\.Sglang: efficient execution of structured language model programs\.In*Proceedings of the 38th International Conference on Neural Information Processing Systems*, 2024\.
- Su et al\. \[2025\]Zunhai Su, Hanyu Wei, Zhe Chen, Wang Shen, Linge Li, Huangqi Yu, and Kehong Yuan\.Rotatekv: accurate and robust 2\-bit kv cache quantization for llms via outlier\-aware adaptive rotations\.In*Proceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence*, pages 6200–6208, 2025\.
- Liu et al\. \[2024\]Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\.Kivi: A tuning\-free asymmetric 2bit quantization for kv cache\.In*International Conference on Machine Learning*, pages 32332–32344\. PMLR, 2024\.
- Xia et al\. \[2026\]Haojun Xia, Xiaoxia Wu, Jisen Li, Tsai chuan Wu, Junxiong Wang, Jue WANG, Chenxi Li, Aman Singhal, Alay Dilipbhai Shah, Alpay Ariyak, Donglin Zhuang, Zhongzhu Zhou, Ben Athiwaratkun, Zhen Zheng, and Shuaiwen Leon Song\.Kitty: Accurate and efficient 2\-bit KV cache quantization with dynamic channel\-wise precision boost\.In*Ninth Conference on Machine Learning and Systems*, 2026\.URL[https://openreview\.net/forum?id=r3mQiuYKIN](https://openreview.net/forum?id=r3mQiuYKIN)\.
- Son et al\. \[2026\]Donghyun Son, Euntae Choi, and Sungjoo Yoo\.Nsnquant: A double normalization approach for calibration\-free low\-bit vector quantization of kv cache\.*Advances in Neural Information Processing Systems*, 38:43124–43159, 2026\.
- Vaswani et al\. \[2017\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.*Advances in neural information processing systems*, 30, 2017\.
- Yang et al\. \[2025\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Abdin et al\. \[2025\]Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al\.Phi\-4\-reasoning technical report\.*arXiv preprint arXiv:2504\.21318*, 2025\.
- MAA \[2025\]MAA\.Aime 2025: American invitational mathematics examination\.[https://maa\.org/math\-competitions/aime](https://maa.org/math-competitions/aime), 2025\.
- Rein et al\. \[2024\]David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R\. Bowman\.GPQA: A graduate\-level google\-proof q&a benchmark\.In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=Ti67584b98](https://openreview.net/forum?id=Ti67584b98)\.
- Liu et al\. \[2023\]Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang\.Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation\.In*Thirty\-seventh Conference on Neural Information Processing Systems*, 2023\.URL[https://openreview\.net/forum?id=1qvx610Cu7](https://openreview.net/forum?id=1qvx610Cu7)\.
- Jain et al\. \[2025\]Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.Livecodebench: Holistic and contamination free evaluation of large language models for code\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=chfJJYC3iL](https://openreview.net/forum?id=chfJJYC3iL)\.
- Hsieh et al\. \[2024\]Cheng\-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg\.RULER: What’s the real context size of your long\-context language models?In*First Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=kIoBbc76Sy](https://openreview.net/forum?id=kIoBbc76Sy)\.
- Kingma and Ba \[2014\]Diederik P Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.*arXiv preprint arXiv:1412\.6980*, 2014\.
- Zandieh et al\. \[2026\]Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni\.Turboquant: Online vector quantization with near\-optimal distortion rate\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=tO3ASKZlok](https://openreview.net/forum?id=tO3ASKZlok)\.

## Appendix ASupplementary Materials

Additional implementation details, experimental settings, and qualitative analyses are provided in the supplementary materials\.

#### OptR Algorithm\.

Algorithm[1](https://arxiv.org/html/2608.02691#alg1)summarizes the one\-time offline calibration of OptR\. For each layer and KV head, OptR estimates the key mean and optimizes only the key and value rotation corrections while keeping all model weights frozen\. The matricesAKA\_\{K\}andAVA\_\{V\}parameterize orthogonal corrections throughexp⁡\(A−A⊤\)\\exp\(A\-A^\{\\top\}\)\. We useK~\\widetilde\{K\}andV~\\widetilde\{V\}to denote the effective keys and values consumed by attention after rotated INT2 quantization\. Sink and recent tokens skip INT2 quantization; their keys retain the same centering, while their values remain unchanged in BF16\.

Algorithm 1OptR offline calibration1:Calibration traces

𝒟\\mathcal\{D\}, base rotations

RK0,RV0R\_\{K\}^\{0\},R\_\{V\}^\{0\}, key\-loss weight

λK\\lambda\_\{K\}, optimization steps

TT
2:

\{R^K,ℓ,h,R^V,ℓ,h,μℓ,h\}ℓ,h\\\{\\widehat\{R\}\_\{K,\\ell,h\},\\widehat\{R\}\_\{V,\\ell,h\},\\mu\_\{\\ell,h\}\\\}\_\{\\ell,h\}
3:for alllayers

ℓ\\elland KV heads

hhdo

4:Estimate the key mean

μℓ,h\\mu\_\{\\ell,h\}from

𝒟\\mathcal\{D\}
5:Initialize

AK,AV←0A\_\{K\},A\_\{V\}\\leftarrow 0
6:for

i=1,…,Ti=1,\\ldots,Tdo

7:

RK←RK0​exp⁡\(AK−AK⊤\)R\_\{K\}\\leftarrow R\_\{K\}^\{0\}\\exp\(A\_\{K\}\-A\_\{K\}^\{\\top\}\)
8:Construct effective keys

K~​\(RK\)\\widetilde\{K\}\(R\_\{K\}\)through key reparameterization and rotated INT2 quantization

9:

ptK←softmax⁡\(qt,j​K~1:t​\(RK\)⊤/d\)p\_\{t\}^\{K\}\\leftarrow\\operatorname\{softmax\}\\left\(q\_\{t,j\}\\widetilde\{K\}\_\{1:t\}\(R\_\{K\}\)^\{\\top\}/\\sqrt\{d\}\\right\)
10:

eK​\(t,j\)←\[\(ptK−pt\)⊤​V1:t\]​WO,j⊤e\_\{K\}\(t,j\)\\leftarrow\\left\[\(p\_\{t\}^\{K\}\-p\_\{t\}\)^\{\\top\}V\_\{1:t\}\\right\]W\_\{O,j\}^\{\\top\}
11:

ℒK←𝔼\(t,j\)∈𝒟​\[DKL​\(pt∥ptK\)\+λK​‖eK​\(t,j\)‖22dmodel\]\\mathcal\{L\}\_\{K\}\\leftarrow\\mathbb\{E\}\_\{\(t,j\)\\in\\mathcal\{D\}\}\\left\[D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|p\_\{t\}^\{K\}\)\+\\lambda\_\{K\}\\frac\{\\\|e\_\{K\}\(t,j\)\\\|\_\{2\}^\{2\}\}\{d\_\{\\mathrm\{model\}\}\}\\right\]
12:Update

AKA\_\{K\}using Adam and the INT2 STE

13:endfor

14:

R^K←RK0​exp⁡\(AK−AK⊤\)\\widehat\{R\}\_\{K\}\\leftarrow R\_\{K\}^\{0\}\\exp\(A\_\{K\}\-A\_\{K\}^\{\\top\}\)
15:Compute

p^tK\\widehat\{p\}\_\{t\}^\{K\}using

K~​\(R^K\)\\widetilde\{K\}\(\\widehat\{R\}\_\{K\}\)
16:for

i=1,…,Ti=1,\\ldots,Tdo

17:

RV←RV0​exp⁡\(AV−AV⊤\)R\_\{V\}\\leftarrow R\_\{V\}^\{0\}\\exp\(A\_\{V\}\-A\_\{V\}^\{\\top\}\)
18:Construct effective values

V~​\(RV\)\\widetilde\{V\}\(R\_\{V\}\)through rotated INT2 quantization

19:

eV​\(t,j\)←\[\(p^tK\)⊤​\(V~1:t​\(RV\)−V1:t\)\]​WO,j⊤e\_\{V\}\(t,j\)\\leftarrow\\left\[\(\\widehat\{p\}\_\{t\}^\{K\}\)^\{\\top\}\\left\(\\widetilde\{V\}\_\{1:t\}\(R\_\{V\}\)\-V\_\{1:t\}\\right\)\\right\]W\_\{O,j\}^\{\\top\}
20:

ℒV←𝔼\(t,j\)∈𝒟​\[‖eV​\(t,j\)‖22dmodel\]\\mathcal\{L\}\_\{V\}\\leftarrow\\mathbb\{E\}\_\{\(t,j\)\\in\\mathcal\{D\}\}\\left\[\\frac\{\\\|e\_\{V\}\(t,j\)\\\|\_\{2\}^\{2\}\}\{d\_\{\\mathrm\{model\}\}\}\\right\]
21:Update

AVA\_\{V\}using Adam and the INT2 STE

22:endfor

23:

R^V←RV0​exp⁡\(AV−AV⊤\)\\widehat\{R\}\_\{V\}\\leftarrow R\_\{V\}^\{0\}\\exp\(A\_\{V\}\-A\_\{V\}^\{\\top\}\)
24:Store

R^K,ℓ,h←R^K\\widehat\{R\}\_\{K,\\ell,h\}\\leftarrow\\widehat\{R\}\_\{K\}and

R^V,ℓ,h←R^V\\widehat\{R\}\_\{V,\\ell,h\}\\leftarrow\\widehat\{R\}\_\{V\}
25:endfor

26:return

\{R^K,ℓ,h,R^V,ℓ,h,μℓ,h\}ℓ,h\\\{\\widehat\{R\}\_\{K,\\ell,h\},\\widehat\{R\}\_\{V,\\ell,h\},\\mu\_\{\\ell,h\}\\\}\_\{\\ell,h\}

Here,ptp\_\{t\}denotes the BF16 attention distribution,ptKp\_\{t\}^\{K\}is computed using the current effective INT2 keys, andp^tK\\widehat\{p\}\_\{t\}^\{K\}is computed using the selected key rotationR^K\\widehat\{R\}\_\{K\}\. The valuesV1:tV\_\{1:t\}remain in BF16 during key calibration\. The projectionWO,jW\_\{O,j\}maps the output of query headjjinto the model hidden space\. Layer and KV\-head indices are omitted inside the algorithm when clear\.

#### Details of Figure[2](https://arxiv.org/html/2608.02691#S3.F2)\.

We use the same 4,096\-token AIME25 trace from Qwen3\-8B for all five settings\. The example is taken from layer 4, KV head 5, and query head 20\. The top row shows the absolute key values immediately before INT2 quantization\. The bottom row shows each cached key token’s contribution to the post\-WOW\_\{O\}error, aggregated over the final 64 query positions while keeping values in BF16 to isolate key\-induced error\. All panels use the same sample and INT2 configuration: group size 128, key clipping ratio 0\.96, 64 BF16 sink tokens, and 256 BF16 recent tokens\.

#### Calibration details\.

We collect Q/K/V activation statistics from all 198 GPQA\-Diamond prompts with one generated token, without using answer labels\. During collection, each dynamically formed prefill batch is stored as a chunk containing aligned Q, K, V, and sequence\-length tensors, yielding 14 chunks for each Qwen model and 15 for Phi\-14B\. From these dumps, we use chunks 3 and 4 for optimization and chunks 10 and 12 for held\-out selection\. After removing sequences shorter than 328 tokens, the calibration/held\-out sets contain 6/5 prompts for Qwen3\-4B, 7/6 for Qwen3\-8B, and 9/3 for Phi\-14B, corresponding to 2,554/2,526, 2,711/3,092, and 4,151/1,619 tokens, respectively\. We compute the objective over the final 64 query positions and select the best rotation for each KV head using the held\-out objective every 20 steps\. Key\-centering statistics are estimated from 28 full model\-generated traces\. The calibration and held\-out chunks do not overlap, although their source prompts come from the GPQA\-Diamond evaluation pool\.

SettingConfigurationKV\-cache precisionGroup\-wise affine INT2Quantization group size128Key clipping ratio0\.96Value clipping ratio0\.92BF16 sink window64 tokensBF16 recent window256 tokensCalibration dataGPQA decoding tracesOptimizerAdamKingma and Ba \[[2014](https://arxiv.org/html/2608.02691#bib.bib22)\]Optimization steps80Learning rate0\.02Table A:Default experimental settings for OptR\.
#### Experimental settings\.

We implement OptR on the official OSCAR codebase and adopt its best\-performing INT2 configurationZhou et al\. \[[2026](https://arxiv.org/html/2608.02691#bib.bib5)\]\. The default settings used throughout the main experiments and appendix are summarized in Table[A](https://arxiv.org/html/2608.02691#A1.T1)\. Unless explicitly stated, each ablation changes only the setting under investigation while keeping all others fixed\.

We measure efficiency on a single NVIDIA A100\-SXM4 40GB GPU using Qwen3\-4B\-Thinking\-2507 with tensor parallelism of one and an eight\-token page size\. Standard CUDA graphs are enabled for each evaluated batch size, while piecewise CUDA graphs are disabled\. Each process performs an initial untimed warm\-up\. For every end\-to\-end and prefill setting, we additionally discard one complete warm\-up run before recording five timed runs\. Decode latency is measured over 1,024 generated tokens and excludes prefill time\. End\-to\-end output throughput is computed as the number of generated tokens divided by the combined prefill and decoding time\. We report the mean over five runs in the main figure and omit error bars for readability\. The maximum standard deviations across all evaluated settings are 0\.104 ms per token for decode latency, 0\.412 tokens/s for end\-to\-end throughput, and 0\.0115 s for prefill time\. We use a custom SGLang implementation, PyTorch 2\.9\.1 with CUDA 12\.8, and Triton 3\.5\.1\.

#### Key\-mean estimation and application\.

The key mean is estimated from post\-RoPE keys in the BF16 calibration traces:

μℓ,h=1N​∑n,skn,s,hℓ,RoPE\\mu\_\{\\ell,h\}=\\frac\{1\}\{N\}\\sum\_\{n,s\}k\_\{n,s,h\}^\{\\ell,\\mathrm\{RoPE\}\}\(19\)At inference, the same mean is subtracted from the post\-RoPE keys before rotation and quantization\. The cache\-write kernel implements this operation as

\(ksRoPE−μℓ,h\)​RK,ℓ,h=ksRoPE​RK,ℓ,h−μℓ,h​RK,ℓ,h\\left\(k\_\{s\}^\{\\mathrm\{RoPE\}\}\-\\mu\_\{\\ell,h\}\\right\)R\_\{K,\\ell,h\}=k\_\{s\}^\{\\mathrm\{RoPE\}\}R\_\{K,\\ell,h\}\-\\mu\_\{\\ell,h\}R\_\{K,\\ell,h\}\(20\)We precomputeμℓ,h​RK,ℓ,h\\mu\_\{\\ell,h\}R\_\{K,\\ell,h\}once and subtract it in the rotated space\. BF16 sink and recent keys retain the same centering\.

#### Rotated\-query implementation\.

For clarity, the formulation in the main paper maps an effective quantized key back withRK⊤R\_\{K\}^\{\\top\}\. Let

kR,s=Q2​\(\(ks−μ\)​RK\)k\_\{R,s\}=Q\_\{2\}\\\!\\left\(\(k\_\{s\}\-\\mu\)R\_\{K\}\\right\)\(21\)The corresponding effective key isk~s=kR,s​RK⊤\\widetilde\{k\}\_\{s\}=k\_\{R,s\}R\_\{K\}^\{\\top\}\. Its attention logit satisfies

qt​k~s⊤=qt​RK​kR,s⊤q\_\{t\}\\widetilde\{k\}\_\{s\}^\{\\top\}=q\_\{t\}R\_\{K\}k\_\{R,s\}^\{\\top\}\(22\)Therefore, the implementation storeskR,sk\_\{R,s\}directly in the INT2 cache and appliesRKR\_\{K\}to the post\-RoPE query instead of explicitly applyingRK⊤R\_\{K\}^\{\\top\}to every dequantized key\. Under GQA, each query head uses the rotation of its corresponding KV head\. The two implementations produce identical attention logits\.

#### Equivalent Inference Implementation\.

Figure[3](https://arxiv.org/html/2608.02691#S3.F3)shows the baseline and output\-aware correction rotations as separate operations\. We denote their combined key and value rotations by

RK=Rk​Rk​cRV=Rv​Rv​cR\_\{K\}=R\_\{k\}R\_\{kc\}\\qquad R\_\{V\}=R\_\{v\}R\_\{vc\}\(23\)
For the key path, the sameRKR\_\{K\}is applied to the post\-RoPE query and centered key\. SinceRKR\_\{K\}is orthogonal, the full\-precision attention logit satisfies

\(qt​RK\)​\(\(ks−μ\)​RK\)⊤=qt​\(ks−μ\)⊤\(q\_\{t\}R\_\{K\}\)\\left\(\(k\_\{s\}\-\\mu\)R\_\{K\}\\right\)^\{\\top\}=q\_\{t\}\(k\_\{s\}\-\\mu\)^\{\\top\}\(24\)Thus, the key rotations cancel in the full\-precision inner product\. The remaining term−qt​μ⊤\-q\_\{t\}\\mu^\{\\top\}is constant across all cached tokensssand therefore leaves the softmax distribution unchanged\.

This equivalence holds before quantization\. Under INT2, the stored key is

kR,s=Q2​\(\(ks−μ\)​RK\)k\_\{R,s\}=Q\_\{2\}\\\!\\left\(\(k\_\{s\}\-\\mu\)R\_\{K\}\\right\)\(25\)and the attention logit becomes\(qt​RK\)​kR,s⊤\(q\_\{t\}R\_\{K\}\)k\_\{R,s\}^\{\\top\}Therefore, the rotation changes the coordinate system in which the INT2 error is introduced, although it does not change the full\-precision attention computation\. This implementation is equivalent to the inverse\-rotation formulation in the main paper:

qt​\(kR,s​RK⊤\)⊤=\(qt​RK\)​kR,s⊤q\_\{t\}\\left\(k\_\{R,s\}R\_\{K\}^\{\\top\}\\right\)^\{\\top\}=\(q\_\{t\}R\_\{K\}\)k\_\{R,s\}^\{\\top\}\(26\)We therefore storekR,sk\_\{R,s\}directly in the INT2 cache and applyRKR\_\{K\}to each query head using the rotation of its corresponding KV head\.

For the value path, Figure[3](https://arxiv.org/html/2608.02691#S3.F3)explicitly appliesRvR\_\{v\}andRv​cR\_\{vc\}before quantization and their inverses after attention\. WithRV=Rv​Rv​cR\_\{V\}=R\_\{v\}R\_\{vc\}, the rotated value and attention output are

vs′=vs​RVoj′=oj​RVv\_\{s\}^\{\\prime\}=v\_\{s\}R\_\{V\}\\qquad o\_\{j\}^\{\\prime\}=o\_\{j\}R\_\{V\}\(27\)The explicit inverse path recovers the original output:

oj′​Rv​c−1​Rv−1​WO,j⊤=oj​WO,j⊤o\_\{j\}^\{\\prime\}R\_\{vc\}^\{\-1\}R\_\{v\}^\{\-1\}W\_\{O,j\}^\{\\top\}=o\_\{j\}W\_\{O,j\}^\{\\top\}\(28\)
In the implementation, these value\-side transforms are folded into the projection weights\. For KV headhhand query headj∈Ghj\\in G\_\{h\}, we use

W¯V,h=RV,h⊤​WV,hW¯O,j=WO,j​RV,h\\overline\{W\}\_\{V,h\}=R\_\{V,h\}^\{\\top\}W\_\{V,h\}\\qquad\\overline\{W\}\_\{O,j\}=W\_\{O,j\}R\_\{V,h\}\(29\)These weights satisfy

x​W¯V,h⊤\\displaystyle x\\overline\{W\}\_\{V,h\}^\{\\top\}=\(x​WV,h⊤\)​RV,h,\\displaystyle=\(xW\_\{V,h\}^\{\\top\}\)R\_\{V,h\},\(30\)\(oj​RV,h\)​W¯O,j⊤\\displaystyle\(o\_\{j\}R\_\{V,h\}\)\\overline\{W\}\_\{O,j\}^\{\\top\}=oj​WO,j⊤\\displaystyle=o\_\{j\}W\_\{O,j\}^\{\\top\}\(31\)Hence, the value rotations and their inverses shown in Figure[3](https://arxiv.org/html/2608.02691#S3.F3)are mathematical operations rather than separate inference kernels\.

#### Models and Benchmarks\.

We evaluate OptR across three reasoning\-oriented language models that differ in model scale and architecture\. This setting allows us to examine whether the proposed method remains effective across both compact and larger models, as well as across distinct model families\.

- •Qwen3\-4B\-Thinking\-2507 and Qwen3\-8BYang et al\. \[[2025](https://arxiv.org/html/2608.02691#bib.bib15)\]: two Qwen3 models at different scales, used to assess whether the effectiveness of OptR is preserved as model scale increases\.
- •Phi4\-14B\-reasoning\-plusAbdin et al\. \[[2025](https://arxiv.org/html/2608.02691#bib.bib16)\]: a reasoning model from a different model family, included to evaluate cross\-architecture generalization\.

Together, these models cover parameter scales from 4B to 14B and include both within\-family scaling and cross\-family evaluation\.

We evaluate reasoning, coding, and long\-context retrieval capabilities to measure the effect of INT2 KV\-cache quantization across diverse inference workloads\.

- •AIME24 and AIME25MAA \[[2025](https://arxiv.org/html/2608.02691#bib.bib17)\]: challenging mathematical reasoning benchmarks that require multi\-step problem solving\.
- •GPQA\-DiamondRein et al\. \[[2024](https://arxiv.org/html/2608.02691#bib.bib18)\]: a graduate\-level scientific reasoning benchmark covering questions that require specialized knowledge and careful reasoning\.
- •MBPP\+Liu et al\. \[[2023](https://arxiv.org/html/2608.02691#bib.bib19)\]: a code\-generation benchmark with extended test cases for more rigorous functional\-correctness evaluation\.
- •LiveCodeBench v6Jain et al\. \[[2025](https://arxiv.org/html/2608.02691#bib.bib20)\]: a contamination\-resistant coding benchmark constructed from recent programming problems\.
- •RULER\-NIAHHsieh et al\. \[[2024](https://arxiv.org/html/2608.02691#bib.bib21)\]: a long\-context retrieval benchmark used to evaluate whether quantized KV caches preserve information over extended context lengths\.

The reasoning and coding benchmarks evaluate the quality of generated outputs, whereas RULER\-NIAH isolates long\-context retrieval performance under increasing context lengths\.

Generation Settings\.We use the same sampling configuration across all models and benchmarks to ensure a consistent comparison\. Specifically, we set the temperature to0\.60\.6, top\-ppto0\.950\.95, and top\-kkto2020\. For the reasoning and coding benchmarks, we use maximum generation lengths of3232K tokens for the Qwen3 models and1616K tokens for Phi4\-14B\-reasoning\-plus\.

## Appendix BOutput\-Error Measurement and Analysis

This section describes the measurement settings for the per\-layer line plots, layer\-averaged bar plots, and per\-head heatmaps\. The main paper reports the Qwen3\-4B\-Thinking\-2507 per\-layer analysis and GPQA\-Diamond heatmap in Figures[4](https://arxiv.org/html/2608.02691#S5.F4)and[5](https://arxiv.org/html/2608.02691#S5.F5), respectively\. Additional per\-layer and layer\-averaged results for Qwen3\-4B\-Thinking\-2507 and Qwen3\-8B are provided in Figures[B](https://arxiv.org/html/2608.02691#A2.F2)–[E](https://arxiv.org/html/2608.02691#A2.F5), while the complete per\-head heatmaps across both models and datasets are shown in Figure[A](https://arxiv.org/html/2608.02691#A2.F1)\. Unless otherwise specified, all analyses use the same INT2 quantization settings as the main experiments\.

#### Output\-space error\.

For each layerℓ\\ell, KV headhh, and query headj∈𝒢hj\\in\\mathcal\{G\}\_\{h\}, we use the last6464query positions of each sequence\. Letptℓ,j,hp\_\{t\}^\{\\ell,j,h\}andp~tℓ,j,h\\widetilde\{p\}\_\{t\}^\{\\ell,j,h\}denote the causal attention distributions obtained with full\-precision and INT2 keys, respectively\. We writeV1:t,hℓV\_\{1:t,h\}^\{\\ell\}andV~1:t,hℓ\\widetilde\{V\}\_\{1:t,h\}^\{\\ell\}for the full\-precision and effective INT2 value matrices, where the latter includes full\-precision sink\- and recent\-token restoration\. Following the notation in the main paper, the post\-WOW\_\{O\}output discrepancy is

Δ​yt,jℓ=\[\(p~tℓ,j,h\)⊤​V~1:t,hℓ−\(ptℓ,j,h\)⊤​V1:t,hℓ\]​\(WO,jℓ\)⊤\\Delta y\_\{t,j\}^\{\\ell\}=\\left\[\\bigl\(\\widetilde\{p\}\_\{t\}^\{\\ell,j,h\}\\bigr\)^\{\\top\}\\widetilde\{V\}\_\{1:t,h\}^\{\\ell\}\-\\bigl\(p\_\{t\}^\{\\ell,j,h\}\\bigr\)^\{\\top\}V\_\{1:t,h\}^\{\\ell\}\\right\]\\bigl\(W\_\{O,j\}^\{\\ell\}\\bigr\)^\{\\top\}This quantity is the total output errorΔ​yt,jℓ=y~t,jℓ−yt,jℓ\\Delta y\_\{t,j\}^\{\\ell\}=\\widetilde\{y\}\_\{t,j\}^\{\\ell\}\-y\_\{t,j\}^\{\\ell\}and jointly captures key\-induced changes in the attention distribution and value\-induced changes in the attention\-weighted output\. For each layer, we average‖Δ​yt,jℓ‖22\\\|\\Delta y\_\{t,j\}^\{\\ell\}\\\|\_\{2\}^\{2\}over query positions, query heads, KV heads, and sequences\. We apply the same group\-wise INT2 quantization and dequantization path used during inference to the KV cache of the measured layer while using full\-precision input activations\. The resulting values measure the immediate error introduced by each attention block\. The corresponding per\-layer results are reported in Figures[4](https://arxiv.org/html/2608.02691#S5.F4),[B](https://arxiv.org/html/2608.02691#A2.F2), and[D](https://arxiv.org/html/2608.02691#A2.F4)\.

Qwen3\-4B\-Thinking \- GPQA\-Diamond

![Refer to caption](https://arxiv.org/html/2608.02691v1/x7.png)

Qwen3\-8B \- GPQA\-Diamond

![Refer to caption](https://arxiv.org/html/2608.02691v1/x8.png)

Qwen3\-4B\-Thinking \- AIME\-25

![Refer to caption](https://arxiv.org/html/2608.02691v1/x9.png)

Qwen3\-8B \- AIME\-25

![Refer to caption](https://arxiv.org/html/2608.02691v1/x10.png)

Figure A:Output error by layer and KV head relative to naive INT2 across models and evaluation datasets\. Each cell reports the post\-WOW\_\{O\}attention\-output RMS error as a percentage of plain per\-group INT2 without rotation\. Lower values indicate smaller output error, with naive INT2 corresponding to100%100\\%\. Within each heatmap, panels show \(a\) TurboQuant, \(b\) QuaRot, \(c\) QuaRot \+ OptR, \(d\) OSCAR, and \(e\) OSCAR \+ OptR\.![Refer to caption](https://arxiv.org/html/2608.02691v1/x11.png)Figure B:Per\-layer error analysis under INT2 KV\-cache quantization on Qwen3\-4B\-Thinking\. Panels \(a,b\) report the post\-WOW\_\{O\}output\-space MSE on GPQA\-Diamond and AIME25, capturing the immediate error introduced by each attention block\. Panels \(c,d\) report the propagated residual\-stream error, computed as the normalized squared difference between full\-precision and INT2\-KV block\-output hidden states evaluated on the same full\-precision\-generated sequences\. Values in the legend denote averages across layers\. All errors are shown on a logarithmic scale\.![Refer to caption](https://arxiv.org/html/2608.02691v1/x12.png)Figure C:Layer\-averaged error under INT2 KV\-cache quantization on Qwen3\-4B\-Thinking\. Panels \(a,b\) report the mean post\-WOW\_\{O\}output\-space MSE across Transformer layers on GPQA\-Diamond and AIME25, respectively\. Panels \(c,d\) report the corresponding mean propagated residual\-stream error\. The values summarize the per\-layer results shown in Figure[B](https://arxiv.org/html/2608.02691#A2.F2)\. Lower values indicate smaller quantization\-induced error\.![Refer to caption](https://arxiv.org/html/2608.02691v1/x13.png)Figure D:Per\-layer error analysis under INT2 KV\-cache quantization on Qwen3\-8B\. Panels \(a,b\) report the post\-WOW\_\{O\}output\-space MSE on GPQA\-Diamond and AIME25, capturing the immediate error introduced by each attention block\. Panels \(c,d\) report the propagated residual\-stream error, computed as the normalized squared difference between full\-precision and INT2\-KV block\-output hidden states evaluated on the same full\-precision\-generated sequences\. Values in the legend denote averages across layers\. All errors are shown on a logarithmic scale\.![Refer to caption](https://arxiv.org/html/2608.02691v1/x14.png)Figure E:Layer\-averaged error under INT2 KV\-cache quantization on Qwen3\-8B\. Panels \(a,b\) report the mean post\-WOW\_\{O\}output\-space MSE across Transformer layers on GPQA\-Diamond and AIME25, respectively\. Panels \(c,d\) report the corresponding mean propagated residual\-stream error\. The values summarize the per\-layer results shown in Figure[D](https://arxiv.org/html/2608.02691#A2.F4)\. Lower values indicate smaller quantization\-induced error\.
#### Propagated residual\-stream error\.

To measure the accumulation of quantization error across depth, we run the complete model with INT2 KV caches in all Transformer layers\. For each of1212prompts per dataset, the full\-precision model first generates a reasoning trace greedily, with up to15361536new tokens and a maximum sequence length of30723072\. The full\-precision and INT2 models are then evaluated on the same generated token sequence\. This teacher\-forced comparison isolates representational drift from differences caused by autoregressive sampling\.

All model weights remain in BF16\. Long\-history keys and values are quantized and stored in INT2 at cache\-write time and dequantized during attention, while the configured sink\- and recent\-token windows remain in full precision\. Lethℓ,th\_\{\\ell,t\}andh~ℓ,t\\widetilde\{h\}\_\{\\ell,t\}denote the full\-precision and INT2\-KV residual streams after blockℓ\\ellat token positiontt\. We report the relative squared error

∑t‖h~ℓ,t−hℓ,t‖22∑t‖hℓ,t‖22\\frac\{\\sum\_\{t\}\\\|\\widetilde\{h\}\_\{\\ell,t\}\-h\_\{\\ell,t\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{t\}\\\|h\_\{\\ell,t\}\\\|\_\{2\}^\{2\}\}pooled over all tokens and prompts\. This normalization accounts for changes in the residual\-stream magnitude across depth and enables comparison across layers\. The propagated errors across model depth are shown in panels \(c,d\) of Figures[4](https://arxiv.org/html/2608.02691#S5.F4),[B](https://arxiv.org/html/2608.02691#A2.F2), and[D](https://arxiv.org/html/2608.02691#A2.F4)\.

#### Layer\-averaged summaries\.

The bar plots summarize the corresponding per\-layer line plots by reporting the arithmetic mean of each error metric across Transformer layers\. Panels \(a,b\) report the mean post\-WOW\_\{O\}output\-space MSE on GPQA\-Diamond and AIME25, while panels \(c,d\) report the corresponding mean propagated residual\-stream error\. The Qwen3\-4B\-Thinking\-2507 and Qwen3\-8B summaries are reported in Figures[C](https://arxiv.org/html/2608.02691#A2.F3)and[E](https://arxiv.org/html/2608.02691#A2.F5), respectively\. These plots provide an aggregate comparison between methods, whereas Figures[B](https://arxiv.org/html/2608.02691#A2.F2)and[D](https://arxiv.org/html/2608.02691#A2.F4)show how the same errors vary across model depth\.

#### Per\-head error relative to naive INT2\.

The heatmaps use the same post\-WOW\_\{O\}output errorΔ​yt,jℓ\\Delta y\_\{t,j\}^\{\\ell\}, but retain a separate value for each\(layer,KV head\)\(\\text\{layer\},\\text\{KV head\}\)pair rather than averaging over KV heads\. Leterrℓ,hmethod\\mathrm\{err\}\_\{\\ell,h\}^\{\\mathrm\{method\}\}denote the mean squared output error for a given cell and leterrℓ,hINT2\\mathrm\{err\}\_\{\\ell,h\}^\{\\mathrm\{INT2\}\}denote the corresponding error under plain per\-group INT2 without rotation or clipping\. Each cell reports100​errℓ,hmethod/errℓ,hINT2100\\sqrt\{\\mathrm\{err\}\_\{\\ell,h\}^\{\\mathrm\{method\}\}/\\mathrm\{err\}\_\{\\ell,h\}^\{\\mathrm\{INT2\}\}\}, representing the RMS output error as a percentage of naive INT2\. Thus, naive INT2 corresponds to100%100\\%, and lower values indicate smaller output error\. The heatmaps use a logarithmic scale ranging from3%3\\%to100%100\\%\. The GPQA\-Diamond and AIME25 heatmaps are shown in Figures[A](https://arxiv.org/html/2608.02691#A2.F1)\.

#### Results\.

Figures[B](https://arxiv.org/html/2608.02691#A2.F2)and[C](https://arxiv.org/html/2608.02691#A2.F3)show that OptR reduces both immediate post\-WOW\_\{O\}output error and propagated residual\-stream error on Qwen3\-4B\-Thinking\-2507\. On GPQA\-Diamond, OptR reduces the average post\-WOW\_\{O\}output MSE from0\.830\.83to0\.230\.23for QuaRot and from0\.410\.41to0\.230\.23for OSCAR\. On AIME25, the corresponding errors decrease from3\.413\.41to1\.911\.91and from2\.522\.52to1\.891\.89, respectively\. The propagated residual\-stream error decreases from0\.3660\.366to0\.1460\.146for QuaRot and from0\.1650\.165to0\.1070\.107for OSCAR on GPQA\-Diamond, and from0\.3360\.336to0\.1550\.155and from0\.1580\.158to0\.1160\.116on AIME25\.

The same trend holds for Qwen3\-8B, as shown in Figures[D](https://arxiv.org/html/2608.02691#A2.F4)and[E](https://arxiv.org/html/2608.02691#A2.F5)\. On GPQA\-Diamond, OptR reduces the average post\-WOW\_\{O\}output MSE from2\.312\.31to0\.900\.90for QuaRot and from1\.731\.73to0\.870\.87for OSCAR\. On AIME25, the corresponding errors decrease from8\.888\.88to4\.004\.00and from7\.027\.02to4\.294\.29\. The propagated residual\-stream error decreases from0\.1950\.195to0\.1270\.127for QuaRot and from0\.1470\.147to0\.1090\.109for OSCAR on GPQA\-Diamond\. On AIME25, it decreases from0\.2100\.210to0\.1310\.131for QuaRot and from0\.1350\.135to0\.1320\.132for OSCAR\.

Finally, Figure[A](https://arxiv.org/html/2608.02691#A2.F1)shows that the error reductions extend across most layers and KV heads on both GPQA\-Diamond and AIME25\. The improvements are therefore broadly distributed throughout the models rather than being concentrated in a small subset of layers or heads\.

TokensOSCAROSCAR \+ OptR3\.13\.1k62\.00±6\.5062\.00\\pm 6\.5069\.33±3\.6569\.33\\pm 3\.656\.66\.6k63\.33±4\.0863\.33\\pm 4\.0870\.67±2\.7970\.67\\pm 2\.7912\.612\.6k62\.67±5\.9662\.67\\pm 5\.9670\.00±5\.7770\.00\\pm 5\.7719\.219\.2k64\.00±3\.6564\.00\\pm 3\.6570\.67±4\.3570\.67\\pm 4\.3526\.026\.0k62\.67±6\.4162\.67\\pm 6\.4170\.00±4\.0870\.00\\pm 4\.08Table B:Effect of calibration size on AIME25 accuracy for Qwen3\-4B\-Thinking\-2507\. Both methods use the same GPQA calibration tokens and a BF16 window ofS=64/R=256S\{=\}64/R\{=\}256\. Results areμ±σ\\mu\\pm\\sigmaover 5 seeds\.
### B\.1Ablation on Calibration Size

Table[B](https://arxiv.org/html/2608.02691#A2.T2)evaluates the sensitivity of OSCAR and OptR to the number of calibration tokens\. OptR consistently improves OSCAR across all calibration sizes, while its performance remains stable from a few thousand tokens onward\. Increasing the calibration size beyond6\.66\.6k tokens provides no consistent additional gain, indicating that OptR requires only a modest calibration set\. We therefore use6\.66\.6k tokens as the default configuration\.

### B\.2Ablation onλK\\lambda\_\{K\}

Table[C](https://arxiv.org/html/2608.02691#A2.T3)studies the sensitivity of OptR to the key\-loss weightλK\\lambda\_\{K\}\. The best value depends on the base rotation: OSCAR \+ OptR performs best atλK=1\\lambda\_\{K\}=1, whereas QuaRot\-INT2 \+ OptR achieves its highest accuracy atλK=10\\lambda\_\{K\}=10\.

λK\\lambda\_\{K\}QuaRot\-INT2 \+ OptROSCAR \+ OptR0\.10\.160\.67±5\.4860\.67\\pm 5\.4866\.67±4\.0866\.67\\pm 4\.080\.30\.362\.00±4\.3562\.00\\pm 4\.3563\.33±5\.2763\.33\\pm 5\.27\\rowcolorgray\!15𝟏\\mathbf\{1\}60\.67±4\.3560\.67\\pm 4\.3570\.67±2\.79\\mathbf\{70\.67\\pm 2\.79\}3360\.67±4\.3560\.67\\pm 4\.3563\.33±4\.7163\.33\\pm 4\.71101063\.33±4\.08\\mathbf\{63\.33\\pm 4\.08\}60\.67±2\.7960\.67\\pm 2\.79Table C:Effect ofλK\\lambda\_\{K\}on AIME25 accuracy for Qwen3\-4B\-Thinking\-2507\. Results areμ±σ\\mu\\pm\\sigmaover five seeds, and the shaded row denotes the default setting\.This difference indicates that the appropriate balance between attention\-distribution preservation and post\-WOW\_\{O\}output\-error reduction depends on the rotation initialization\. To avoid base\-specific tuning, we useλK=1\\lambda\_\{K\}=1throughout the main experiments, which gives the strongest performance with OSCAR, our state\-of\-the\-art rotation baseline\.

ModelCalibration Time \(Minute\)Qwen3\-4B\-Thinking\-25076\.25Qwen3\-8B7\.05Phi4\-14B\-Reasoning\-Plus11\.03Table D:Calibration time of OptR with 80 optimization steps\. Measurements use a single NVIDIA A100 GPU and exclude BF16 trace collection\. All model weights remain frozen throughout calibration\.#### Offline calibration cost\.

Table[D](https://arxiv.org/html/2608.02691#A2.T4)reports the one\-time offline calibration cost of OptR\. Calibration requires only several minutes on a single GPU across the evaluated model sizes\. All model weights remain frozen, and the resulting rotations are reused across subsequent inference requests, amortizing this cost during deployment\.

ModelMethod4k8k16k32k64k128kQwen3\-8BBF1699\.83±0\.1199\.83\\pm 0\.1199\.93±0\.0299\.93\\pm 0\.0299\.45±0\.0899\.45\\pm 0\.0898\.70±0\.4998\.70\\pm 0\.4984\.22±1\.4184\.22\\pm 1\.4180\.89±0\.5880\.89\\pm 0\.58QuaRot\-INT284\.97±0\.0984\.97\\pm 0\.0941\.73±5\.7541\.73\\pm 5\.7518\.16±1\.1418\.16\\pm 1\.1413\.03±1\.7113\.03\\pm 1\.710\.04±0\.070\.04\\pm 0\.070\.38±0\.130\.38\\pm 0\.13QuaRot\-INT2 \+OptR99\.40±0\.2999\.40\\pm 0\.2998\.76±0\.57\\mathbf\{98\.76\\pm 0\.57\}96\.14±0\.35\\mathbf\{96\.14\\pm 0\.35\}86\.37±0\.7286\.37\\pm 0\.7270\.02±1\.79\\mathbf\{70\.02\\pm 1\.79\}41\.87±0\.96\\mathbf\{41\.87\\pm 0\.96\}OSCAR99\.59±0\.1599\.59\\pm 0\.1597\.94±0\.2897\.94\\pm 0\.2894\.39±0\.2994\.39\\pm 0\.2983\.76±0\.5883\.76\\pm 0\.5857\.54±1\.8257\.54\\pm 1\.8226\.44±0\.8726\.44\\pm 0\.87OSCAR \+OptR99\.60±0\.11\\mathbf\{99\.60\\pm 0\.11\}98\.16±0\.6398\.16\\pm 0\.6395\.50±0\.6595\.50\\pm 0\.6586\.42±0\.55\\mathbf\{86\.42\\pm 0\.55\}68\.65±0\.8068\.65\\pm 0\.8040\.43±1\.5340\.43\\pm 1\.53

Table E:RULER\-NIAH retrieval accuracy of Qwen3\-8B at context lengths up to 128K tokens\. Results areμ±σ\\mu\\pm\\sigmaover three seeds, with 800 examples per seed\. Bold denotes the best INT2 result at each context length\.
#### Long\-Context Results

Table[E](https://arxiv.org/html/2608.02691#A2.T5)extends the Qwen3\-8B long\-context evaluation to 128K tokens\. At this context length, QuaRot\-INT2 and OSCAR degrade substantially, whereas applying OptR retains considerably higher retrieval accuracy\. The consistent gains with both base rotations show that OptR remains effective beyond the 64K range reported in the main paper\.