KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
Summary
KVarN is a calibration-free KV-cache quantizer that uses Hadamard rotation and dual-scaling variance normalization to reduce error accumulation during autoregressive decoding in large language models, achieving state-of-the-art 2-bit precision on reasoning benchmarks.
View Cached Full Text
Cached at: 06/03/26, 03:38 PM
Paper page - KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks
Source: https://huggingface.co/papers/2606.03458
Abstract
KVarN is a calibration-free KV-cache quantizer that uses Hadamard rotation and dual-scaling variance normalization to reduce error accumulation during autoregressive decoding in large language models.
Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows.KV-cache quantizationcan help improve this, but current methods are evaluated under prefill-like settings and errors behave differently underautoregressive decoding. We show that in the latter regime, quantization errors accumulate across timesteps, driven primarily by incorrecttoken scales. We introduceKVarN, a calibration-free KV-cache quantizer that applies aHadamard rotationfollowed by adual-scaling variance normalizationacross both axes of the K and V matrices. We find that this combination fixes outlying token-scale errors and substantially reduceserror accumulationover existing baselines.KVarNestablishes a new state-of-theart forKV-cache quantizationon generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision. A vLLM implementation of theKVarNmethod is available at https://github.com/huawei-csl/KVarN
View arXiv pageView PDFProject pageGitHub25Add to collection
Get this paper in your agent:
hf papers read 2606\.03458
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.03458 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.03458 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.03458 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
KVarN: Native vLLM backend for KV-cache quantization by Huawei
Huawei CSL releases KVarN, a native vLLM attention backend for KV-cache quantization that delivers 3-5x more KV-cache capacity and up to ~1.3x the throughput of FP16, with no calibration required. It claims up to ~2.4x the throughput of TurboQuant while maintaining FP16-level accuracy on models like Qwen3-32B.
RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
This paper introduces RateQuant, a method for optimal mixed-precision KV cache quantization that uses rate-distortion theory to address distortion model mismatch. It significantly reduces perplexity compared to existing methods like KIVI and QuaRot with minimal calibration overhead.
Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant
This paper analyzes KV cache quantization schemes inspired by TurboQuant, using statistical inference and a new 6D error framework to evaluate quality measures like KL divergence and geometric error.
KV Cache Compression 900000x Beyond TurboQuant and Per-Vector Shannon Limit
A new paper proposes sequential KV cache compression using probabilistic language tries and predictive delta coding, achieving theoretical compression ratios of ~914,000× beyond TurboQuant by exploiting the sequential structure of language model tokens rather than treating vectors independently.
VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
Introduces VarRate, a training-free method for KV cache compression that allocates variable low-rank budget per token based on query salience, avoiding irreversible token eviction and outperforming uniform-rank methods at matched memory budgets on LongBench.