D-Quant: Driftable Entropy Coding for KV Cache Quantization

# D-Quant: Driftable Entropy Coding for KV Cache Quantization
Source: [https://arxiv.org/html/2609.19880](https://arxiv.org/html/2609.19880)
###### Abstract

The KV cache has become a major bottleneck in deploying LLMs, as its memory footprint grows linearly with sequence length and batch size, imposing substantial pressure on both memory capacity and bandwidth\. Among various KV cache compression techniques, quantization is particularly attractive due to its effectiveness and ease of deployment\. However, most existing methods rely on fixed\-width quantization, where abbbit representation is inherently limited to2b2^\{b\}quantization levels\. As the bit width decreases, the number of available levels shrinks exponentially, leading to severe information loss and rapid performance degradation\. We further observe that fixed\-width quantization fails to exploit the highly non\-uniform distribution of KV cache\. After rotation and normalization, KV values approximately follow a normal distribution, with most values concentrated near the center and only a small fraction appearing in the tails\. Nevertheless, fixed\-width coding allocates the same number of bits to frequent and rare symbols\. Entropy coding naturally exploits such non\-uniformity by assigning shorter codewords to frequent symbols and longer ones to rare symbols, substantially reducing the average number of bits required for representation\. However, its variable\-length output is not suited to highly parallel attention kernels, where efficient dequantization and computation rely on regular memory layouts and fixed\-stride accesses\. To bridge this gap, we proposeD\-Quant, a flexible KV cache quantization framework that introduces adriftmechanism to convert entropy\-coded representations of each token into fixed\-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels\. D\-Quant also provides greater flexibility than conventional quantization methods, allowing the bit budget and number of quantization levels to be configured more freely rather than being constrained by fixed bit widths and group sizes\. Furthermore, since the rotated and normalized KV values closely follow a normal distribution, the probability model for entropy coding can be directly derived from the standard normal distribution, making D\-Quant entirely calibration\-free\. Extensive experiments show that D\-Quant preserves near BF16 performance at only2\.262\.26bits per element while substantially outperforming existing baselines\. D\-Quant further achieves up to a7×7\\timesreduction in KV cache memory footprint and a3\.5×3\.5\\timesimprovement in decoding throughput\.

Tencent

brunosu@tencent\.comlucayu@tencent\.com

𝒌t\{\\bm\{k\}\}\_\{t\}Hadamardremovechannel meanaffinequantizationdriftrANScontainer𝒗t\{\\bm\{v\}\}\_\{t\}Hadamard\(offline\)affinequantizationdriftrANScontainerpreprocessingentropy coding

Figure 1:Overview of our method\.A stream is the Keys or Values of one token and shares a single scale and offset\. After preprocessing \(rotation, mean removal, affine grid\) and affine quantization, the symbol indices are entropy\-coded into a container of fixed byte length, and a length\-constrained assignment makes every stream fit that container exactly\.## 1Introduction

As Large Language Models \(LLMs\) become increasingly capable, they are being widely adopted in applications such as multi\-turn conversations, complex reasoning, and coding agents\(intro1;intro2;intro3\)\. These applications place growing demands on long\-context capabilities, leading to substantial resource overhead during deployment\. During auto\-regressive inference, the KV cache of every processed token must be retained for subsequent generation\. Unlike model weights, whose size remains fixed, the KV cache grows linearly with sequence length and batch size, imposing substantial pressure on both memory capacity and bandwidth\. Quantization, which has proven effective and deployment\-friendly for compressing model weights and activations\([Frantar et al\., 2023](https://arxiv.org/html/2609.19880#bib.bib7);[Xiao et al\., 2023](https://arxiv.org/html/2609.19880#bib.bib31);[Lin et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib16)\), therefore emerges as a promising solution to this growing KV cache bottleneck\.

Existing KV cache quantization methods primarily improve quantization granularity or preprocessing strategies, but typically encode quantized values using a fixed number of bits\([Liu et al\., 2024b](https://arxiv.org/html/2609.19880#bib.bib18);[Hooper et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib10);[Ashkboos et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib2);[Su et al\., 2026](https://arxiv.org/html/2609.19880#bib.bib25)\)\. Under this paradigm, each value is represented usingbbbits and can therefore take only2b2^\{b\}quantization levels\. As the bit width decreases, the number of available levels drops exponentially, substantially limiting representation capacity and ultimately degrading quantization accuracy\. Fixed\-width quantization also fails to exploit the non\-uniform distribution of KV cache\. Existing methods such as rotation\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib2);[Tseng et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib27)\)and normalization\([Su et al\., 2026](https://arxiv.org/html/2609.19880#bib.bib25)\)have developed effective preprocessing techniques to make KV caches more amenable to quantization\. We find that KV cache closely follow a normal distribution after preprossing, with most values concentrated near the center and only a small fraction appearing in the tails\. However, fixed\-width quantization cannot exploit this distributional structure because it assigns the same number of bits to frequent and rare symbols\. Entropy coding provides a natural way to exploit this non\-uniformity\. By assigning shorter codewords to frequent symbols and longer ones to rare symbols, it reduces the average number of bits required to represent a given sequence\. However, entropy coding produces variable\-length bitstreams, making it difficult to integrate with highly parallel attention kernels\. Efficient dequantization and computation in these kernels typically rely on regular data layouts and fixed\-stride accesses\.

To address this issue, we proposeD\-Quant, a flexible KV cache quantization framework that introduces adriftmechanism to convert entropy\-coded representations of each token into fixed\-size bitstreams, enabling regular memory access and parallel dequantization within attention kernels\. To satisfy this constraint with minimal accuracy loss, we formulate quantization as a length\-constrained assignment problem that minimizes reconstruction error under the target bit budget\. When the encoded stream exceeds the budget, a small fraction of values drift from high\-cost symbols to nearby lower\-cost ones until the length constraint is met\. In this way, D\-Quant adapts only the necessary values rather than uniformly reducing the precision of the entire representation\. We solve this minimization problem efficiently using a Lagrangian formulation with bisection\. This design also makes D\-Quant substantially more flexible than conventional fixed\-width quantization\. The storage budget is no longer tied to an integer bit width or a particular group size, and the number of quantization levels is no longer restricted to2b2^\{b\}\. For example, at a matched storage budget of2\.262\.26bits, D\-Quant uses eight levels for keys and six for values\. The fixed\-container formulation further allows the storage budget to be flexibly divided between keys and values according to their different sensitivity to quantization\. Moreover, entropy coding typically requires a probability model estimated from the online data\. However, rotation and normalization have made the KV distribution closely approximate a standard normal distribution\. Therefore, we directly derive the probability model from the standard normal distribution, eliminating the need for calibration while retaining the benefits of distribution\-aware entropy coding\.

Our contributions are summarized as follows:

- •We identify a fundamental limitation of fixed\-width KV cache quantization\. It couples the number of quantization levels to the bit width and fails to exploit the non\-uniform distribution of KV cache\.
- •We propose D\-Quant, which combines entropy coding with fixed\-size per\-token storage through a drift mechanism, enabling flexible bit budgets and quantization levels while preserving efficient parallel processing\. D\-Quant further derives its probability model directly from the standard normal distribution, requiring no calibration data\.
- •Extensive experiments show that D\-Quant maintains near BF16 performance at 2\.26 bits per value, substantially outperforming existing baselines while reducing KV cache memory by up to7×7\\timesand improving decoding throughput by up to3\.5×3\.5\\times\.

## 2Related Work

#### KV cache quantization\.

KV cache quantization has been widely explored to reduce the memory and bandwidth overhead of long\-context inference\. KIVI\([Liu et al\., 2024b](https://arxiv.org/html/2609.19880#bib.bib18)\)identifies distinct distributional properties of keys and values, motivating per\-channel quantization for keys and per\-token quantization for values\. KVQuant\([Hooper et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib10)\)improves low\-bit quantization through non\-uniform quantization and explicit outlier handling, while OTT\(su2025accurate\)preserves salient outlier tokens in higher precision to mitigate quantization error\. Atom\([Zhao et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib36)\)reorganizes channels to better isolate outliers, while QuaRot\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib2)\)applies Hadamard rotations to suppress outliers and improve quantizability\. More recent work continues to push KV cache quantization toward lower bit widths through improved preprocessing, quantization schemes, and dedicated system optimizations\([Su et al\., 2026](https://arxiv.org/html/2609.19880#bib.bib25)\)\. Complementary to these efforts, our work explores the largely overlooked coding format of quantized KV values\.

#### Entropy coding\.

For non\-uniform data, entropy coding reduces the expected code length by assigning shorter codewords to more frequent symbols and longer ones to less frequent symbols, with the source entropy providing a fundamental lower bound on the average number of bits required per symbol\([Shannon, 1948](https://arxiv.org/html/2609.19880#bib.bib23)\)\. Huffman coding\([Huffman, 1952](https://arxiv.org/html/2609.19880#bib.bib12)\)constructs prefix codes whose average code length is within one bit of the entropy bound\. Arithmetic coding\([Witten et al\., 1987](https://arxiv.org/html/2609.19880#bib.bib30)\)approaches this bound more closely by encoding an entire sequence into a single interval\. Asymmetric numeral systems \(ANS\)\([Duda, 2013](https://arxiv.org/html/2609.19880#bib.bib5)\)achieve compression efficiency comparable to arithmetic coding while supporting efficient table\-based implementations, making them well suited for high\-throughput decoding\. When entropy coding is combined with lossy quantization, the coding cost of each symbol becomes part of the quantization objective, commonly formulated as a optimization that balances reconstruction error against coding rate\. Such formulations have been extensively studied in image and video compression\([Ortega & Ramchandran, 1998](https://arxiv.org/html/2609.19880#bib.bib21);[Sullivan & Wiegand, 1998](https://arxiv.org/html/2609.19880#bib.bib26);[Marcellin & Fischer, 1990](https://arxiv.org/html/2609.19880#bib.bib19);[Marpe et al\., 2003](https://arxiv.org/html/2609.19880#bib.bib20)\)and neural network compression\([Wiedemann et al\., 2020](https://arxiv.org/html/2609.19880#bib.bib29)\)\. A standard approach uses a Lagrangian objective to jointly account for reconstruction error and code length when assigning quantization symbols\. Entropy coding has also been explored for KV cache compression\. CacheGen\([Liu et al\., 2024a](https://arxiv.org/html/2609.19880#bib.bib17)\)encodes KV caches into compact bitstreams to reduce communication overhead during KV cache streaming\. SplitZip\([Guo & Joshi, 2026](https://arxiv.org/html/2609.19880#bib.bib9)\)employs lightweight lossless compression for KV cache transfer in disaggregated inference\. These methods primarily target KV cache communication, where compressed streams can be fully decoded before attention computation\. In contrast, directly operating on entropy\-coded KV caches within attention kernels is more challenging, as variable\-length representations introduce irregular memory access and hinder efficient parallel decoding\([Jiang et al\., 2025](https://arxiv.org/html/2609.19880#bib.bib13)\)\.

## 3Preliminary

### 3\.1Background

#### Group Quantization\.

Let𝒙∈ℝn\{\\bm\{x\}\}\\in\\mathbb\{R\}^\{n\}denote the key or value vector of a single token at one layer\. Group\-wise quantization partitions𝒙\{\\bm\{x\}\}into groups ofggvalues, each sharing a scalessand zero pointoo\. Underbb\-bit affine quantization, each value is reconstructed as

x^i=o\+s​mi,mi∈\{0,…,2b−1\},\\hat\{x\}\_\{i\}=o\+sm\_\{i\},\\qquad m\_\{i\}\\in\\\{0,\\ldots,2^\{b\}\-1\\\},\(1\)wheremim\_\{i\}is the quantized*symbol*\. A fixed\-widthbb\-bit representation therefore provides exactly2b2^\{b\}reconstruction levels\. We report storage in bits per value \(BPV\), including quantized symbols and metadata\. For example, INT2 with an FP16 scale and zero point per128128values requires 2\.25 BPV\.

#### Entropy Coding\.

Entropy coding assigns shorter representations to more probable symbols\. For a symbolmmwith probabilitypmp\_\{m\}, the ideal coding cost isr⁡\(m\)=−log2⁡pmr\(m\)=\-\\log\_\{2\}p\_\{m\}, whose expectation is the entropyH⁡\(p\)H\(p\)\. Thus, an alphabet ofMMsymbols can require substantially fewer thanlog2⁡M\\log\_\{2\}Mbits per value when its distribution is non\-uniform, allowing more quantization levels under the same average storage budget\.

We use range asymmetric numeral systems \(rANS\)\([Duda, 2013](https://arxiv.org/html/2609.19880#bib.bib5)\)for efficient entropy coding\. Given precisionpp, each symbolmmis assigned an integer frequencyfmf\_\{m\}and cumulative frequencycmc\_\{m\}, with∑mfm=2p\\sum\_\{m\}f\_\{m\}=2^\{p\}andpm=fm/2pp\_\{m\}=f\_\{m\}/2^\{p\}\. Encoding and decoding update the integer statexxas

Enc⁡\(x,m\)\\displaystyle\\mathrm\{Enc\}\(x,m\)=⌊xfm⌋​2p\+\(xmodfm\)\+cm,\\displaystyle=\\left\\lfloor\\frac\{x\}\{f\_\{m\}\}\\right\\rfloor 2^\{p\}\+\(x\\bmod f\_\{m\}\)\+c\_\{m\},\(2\)Dec⁡\(x,m\)\\displaystyle\\mathrm\{Dec\}\(x,m\)=fm​⌊x2p⌋\+\(xmod2p\)−cm\.\\displaystyle=f\_\{m\}\\left\\lfloor\\frac\{x\}\{2^\{p\}\}\\right\\rfloor\+\(x\\bmod 2^\{p\}\)\-c\_\{m\}\.During decoding,z=xmod2pz=x\\bmod 2^\{p\}uniquely identifies the symbol whose interval\[cm,cm\+fm\)\[c\_\{m\},c\_\{m\}\+f\_\{m\}\)containszz\. This mapping can be implemented by a2p2^\{p\}\-entry lookup table; withp=8p=8, only256256entries are required\. We store the corresponding reconstruction values in the same table, allowing decoding and dequantization to share a lookup\. Since rANS decodes symbols in reverse order, each stream is encoded backwards so that decoding proceeds in its original order\.

Unlike fixed\-width coding, rANS allows coding costs to vary across symbols, decoupling the number of quantization levels from a fixed per\-value bit width\. The resulting stream length, however, varies with its symbols\. D\-Quant exploits the former while eliminating the latter by enforcing a fixed storage budget for every stream\.

### 3\.2Observations

#### Preprocessed KV caches exhibit predictable and compressible distributions\.

Entropy coding is effective when symbols follow non\-uniform distributions\. We investigate these properties on conventionally processed KV caches\. Specifically, we quantize each token into an eight\-level per\-token grid and measure three quantities: reconstruction error \(NMSE\), the potential entropy coding gain over fixed\-width coding \(Coding​Gain=log2⁡M−H\\mathrm\{Coding\\ Gain\}=\\log\_\{2\}M\-H\), and the excess coding rate of using a standard\-normal\-derived model instead of an empirical model \(Normal​Gap=CE−H\\mathrm\{Normal\\ Gap\}=\\mathrm\{CE\}\-H\)\.

Table 1:Effect of preprocessing on quantization error and entropy modeling\. Results are measured on captured Qwen3\-8B\. Each token is quantized into an eight\-level per\-token grid\. Normal Gap measures the excess rate of using a standard\-normal\-derived entropy model instead of an empirical model, while Coding Gain measures the potential saving over fixed\-width quantization\.Hadamard rotation spreads structured outliers across channels and makes KV distributions substantially closer to Gaussian \(Figure[2](https://arxiv.org/html/2609.19880#S3.F2)\)\. However, rotation alone does not remove the channel\-wise mean of keys, which cannot be captured by a token\-level scalar offset\. Removing this residual mean further reduces key NMSE to0\.48%0\.48\\%and, more importantly, reduces the Normal Gap to only5×10−45\\times 10^\{\-4\}bits/value\. This indicates that the probability model required by entropy coding can be accurately derived from a standard normal distribution without calibration data\. Meanwhile, the processed KV cache still retains substantial entropy coding potential, achieving0\.720\.72and0\.860\.86bits/value Coding Gain for keys and values, respectively\.

Figure 2:Distribution of KV values before and after preprocessing\. Raw KV tensors exhibit strong outliers and non\-Gaussian patterns\. Hadamard rotation substantially reduces distribution irregularity, and mean removal further centers key distributions\.
#### Entropy coding decouples quantization levels from storage budgets\.

Fixed\-width quantization inherently couples the representation capacity with the storage budget\. Abb\-bit representation provides exactly2b2^\{b\}quantization levels\. Reducing the bit width therefore inevitably reduces the available alphabet\. Entropy coding changes this constraint by allocating bits according to symbol probabilities, where the storage budget limits the total coding rate rather than the number of quantization levels\. To study this effect, we fix the storage budget of each token and vary the number of quantization levels\. For each alphabet size, we solve the corresponding rate\-distortion optimization under the same storage constraint and measure the resulting reconstruction error\.

Table 2:Reconstruction error under different alphabet sizes with fixed storage budgets\. The key and value containers are fixed to 2\.68 BPE and 1\.84 BPE, respectively\. Dashes indicate that the corresponding alphabet cannot satisfy the storage constraint\.Table[2](https://arxiv.org/html/2609.19880#S3.T2)shows that entropy coding provides substantially more flexibility under a fixed storage budget\. The number of quantization levels is no longer constrained by the bit width, but can be selected according to the distribution of KV values and the desired rate\-distortion trade\-off\. This decoupling enables more expressive representations while preserving a fixed memory layout\.

## 4Method

### 4\.1Overview

D\-Quant compresses each key or value tensor of one token into a fixed\-size container\. As illustrated in Figure[1](https://arxiv.org/html/2609.19880#S0.F1), the encoding pipeline consists of two stages\. First, we apply lightweight preprocessing to improve the distribution of KV values, including Hadamard rotation, mean removal for keys, and affine quantization\. Second, instead of representing quantized symbols with fixed\-width codes, D\-Quant applies entropy coding to exploit the non\-uniform symbol distribution\.

A direct application of entropy coding, however, produces variable\-length representations, which prevents fixed\-stride memory access and efficient parallel decoding in attention kernels\. D\-Quant addresses this issue by introducing a fixed\-size container for each stream and a drift mechanism that optimizes symbol assignments under the container constraint\. The resulting representation preserves the compression benefits of entropy coding while maintaining the regular memory layout required by efficient KV cache dequantization\.

### 4\.2Fixed\-size entropy coding for KV cache

Instead of assigning a fixed\-width codeword to each value, D\-Quant allocates a container ofCCbytes per stream and encodes its quantized symbols using entropy coding\. A stream denotes thennvalues of one key or value tensor corresponding to a single token\. The encoded stream satisfies

∑i=1nr⁡\(mi\)≤B,B=8​C−σ,\\sum\_\{i=1\}^\{n\}r\(m\_\{i\}\)\\leq B,\\qquad B=8C\-\\sigma,\(3\)wherer⁡\(mi\)r\(m\_\{i\}\)is the entropy\-coded length of symbolmim\_\{i\}andσ\\sigmadenotes the final state overhead of the entropy coder\. Since each stream occupies a predetermined storage region, the KV cache remains a regular array of fixed\-stride records, enabling efficient parallelism inside attention kernels\.

Before entropy coding, D\-Quant applies lightweight preprocessing to obtain a predictable symbol distribution\. Following prior KV cache quantization methods\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib2);[Liu et al\., 2024b](https://arxiv.org/html/2609.19880#bib.bib18)\), we apply Hadamard rotation to each attention head\. For values, the rotation can be folded into projection weights offline, while for keys it is applied online due to the non\-commutativity with RoPE\([Su et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib24)\)\. Keys further remove a windowed channel\-wise mean\. The processed stream is quantized with a token\-wise affine:

x^i=o\+s​mi,mi∈\{0,…,M−1\},\\hat\{x\}\_\{i\}=o\+sm\_\{i\},\\qquad m\_\{i\}\\in\\\{0,\\dots,M\-1\\\},\(4\)wheres=\(max⁡\(𝒙\)−min⁡\(𝒙\)\)/\(M−1\)s=\(\\max\(\{\\bm\{x\}\}\)\-\\min\(\{\\bm\{x\}\}\)\)/\(M\-1\)ando=min⁡\(𝒙\)o=\\min\(\{\\bm\{x\}\}\)\. The entropy model is analytically derived from Gaussian order statistics and requires no calibration data\. Due to the affine invariance of normalized positions, the same probability table can be shared across different tokens and layers\.

### 4\.3Drift: length\-constrained assignment

Although entropy coding provides flexible representations, the resulting code length depends on the assigned symbols and may exceed the predefined budget\. Instead of reducing the number of quantization levels, D\-Quant directly optimizes the symbol assignment under the length constraint:

min⁡∑i=1n𝒎∈\{0,…,M−1\}n⁡\(xi−\(o\+s​mi\)\)2s\.t\.∑i=1nr⁡\(mi\)≤B,\\min\_\{\{\\bm\{m\}\}\\in\\\{0,\\dots,M\-1\\\}^\{n\}\}\\sum\_\{i=1\}^\{n\}\\left\(x\_\{i\}\-\(o\+sm\_\{i\}\)\\right\)^\{2\}\\quad\\mathrm\{s\.t\.\}\\quad\\sum\_\{i=1\}^\{n\}r\(m\_\{i\}\)\\leq B,\(5\)wherer⁡\(mi\)r\(m\_\{i\}\)is the entropy\-coded length obtained from the probability model\. We solve this constrained optimization through a Lagrangian formulation:

min⁡∑i=1n𝒎⁡\[Di​\(mi\)\+λ​r​\(mi\)\],\\min\_\{\{\\bm\{m\}\}\}\\sum\_\{i=1\}^\{n\}\\left\[D\_\{i\}\(m\_\{i\}\)\+\\lambda r\(m\_\{i\}\)\\right\],\(6\)whereDi​\(mi\)D\_\{i\}\(m\_\{i\}\)denotes the reconstruction error of assigning valuexix\_\{i\}to levelmim\_\{i\}\. For a fixed multiplierλ\\lambda, the objective is separable across values, allowing each symbol assignment to be independently optimized\. We use bisection to find the multiplier that satisfies the code\-length constraint\.

Intuitively, D\-Quant does not uniformly reduce the representation precision when the encoded stream exceeds its budget\. Instead, only a small fraction of values assigned to expensive symbols are allowed to*drift*toward nearby lower\-cost symbols\. This progressively reduces the total code length with minimal reconstruction error, preserving the benefits of a large entropy\-coded alphabet\. The resulting assignment jointly optimizes reconstruction quality and coding cost, enabling every stream to satisfy its fixed container constraint\.

### 4\.4Practical refinements

#### Separate K/V configurations\.

Keys and values play different roles in attention and exhibit different sensitivities to quantization\. Errors in keys directly perturb the attention logits and can be amplified through the exponential operation in softmax, whereas errors in values are introduced after softmax and are aggregated by the attention weights\. Empirically, we find keys to be more sensitive to quantization and therefore allocate them a larger container budget\. D\-Quant further allows the number of quantization levels for keys and values to be configured independently\.

#### Distribution sharpening\.

The analytic probability model describes the symbol distribution before drift, whereas drift preferentially moves values toward lower\-cost, more probable symbols, making the final coded distribution more concentrated\. We account for this shift by sharpening the analytic probabilities aspm′∝pmαp\_\{m\}^\{\\prime\}\\propto p\_\{m\}^\{\\alpha\}withα\>1\\alpha\>1before constructing the entropy table\. A singleα\\alphais used across all models and layers\.

### 4\.5Kernels

#### Encoding\.

A straightforward implementation of the drift solver materializes ann×Mn\\times Mdistortion matrix at every bisection step, introducing substantial memory traffic for values that are inexpensive to recompute\. We instead assign one warp to each stream and keep its values on chip throughout the optimization\. At each bisection step, distortion is recomputed on the fly for theMMcandidate levels, while the resulting code length is reduced through warp\-level operations\. The final affine refit and rANS encoding are performed within the same warp, avoiding intermediate materialization and minimizing memory traffic\.

#### Decoding\.

Since each entropy\-coded stream spans all KV heads of a token, conventional head\-centric attention kernels would repeatedly traverse the same stream when different heads consume their corresponding values\. We therefore organize decoding around tokens: each thread block owns a block of tokens, decodes each stream only once, and directly feeds the decoded levels into the corresponding attention computation\. As the stream is decoded one KV head at a time, warps partition the output accumulator by channel rather than by query head, allowing all decoded values to be consumed without redundant decoding\. Moreover, the affine transformation is folded into the attention contraction, so decoded levels can be used directly without explicit element\-wise dequantization\. Each rANS table entry jointly stores the reconstruction level and state\-update information, allowing a single lookup to serve both entropy decoding and attention computation\.

## 5Experiments

### 5\.1Setup

#### Models and benchmarks\.

We evaluate D\-Quant on Qwen3\-8B\([Yang et al\., 2025](https://arxiv.org/html/2609.19880#bib.bib33)\)and Llama\-3\.1\-8B\-Instruct\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib8)\)\. We evaluate on RULER\([Hsieh et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib11)\)at six context lengths from 4K to 128K and on the English subsets of LongBench\-E\([Bai et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib3)\), using greedy decoding for all evaluations\. RULER results are averaged over 100 samples per task at each context length\. Since Qwen3\-8B is natively trained with a 40K context window, we apply static YaRN\([Peng et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib22)\)for evaluation at 64K and 128K\. Although length extrapolation degrades the absolute performance at these lengths, all KV cache quantization methods are evaluated under the same setting, ensuring a fair comparison\.

#### Baselines\.

We compare D\-Quant with KIVI\([Liu et al\., 2024b](https://arxiv.org/html/2609.19880#bib.bib18)\), QuaRot\([Ashkboos et al\., 2024](https://arxiv.org/html/2609.19880#bib.bib2)\), TurboQuant\([Zandieh et al\., 2026](https://arxiv.org/html/2609.19880#bib.bib34)\), and OScaR\([Su et al\., 2026](https://arxiv.org/html/2609.19880#bib.bib25)\), using BF16 FlashAttention\-2\([Dao, 2023](https://arxiv.org/html/2609.19880#bib.bib4)\)as the full\-precision reference\. We use the official implementations and recommended configurations whenever available\. All quantized baselines use a group size of128128\. KIVI, QuaRot and OScaR use asymmetric INT2 quantization\. TurboQuant applies Lloyd–Max quantization to keys and affine quantization to values\. For TurboQuant, we use its MSE\-only configuration, which is the applicable configuration at the evaluated storage budget\. KIVI, QuaRot, and TurboQuant retain the most recent128128tokens in BF16, while OScaR uses its recommended256256\-token residual window\. Their effective storage costs are2\.252\.25,2\.252\.25,2\.192\.19, and2\.262\.26bits per value, respectively, including quantization metadata such as scales and zero points\.

#### Implementation details\.

Unless otherwise specified, D\-Quant allocates331331\-byte and231231\-byte containers to keys and values, with88and66quantization levels, respectively\. This corresponds to an average storage cost of2\.262\.26bits per value\. Each token is quantized as a single group of10241024values, the entropy model uses a sharpening factor ofα=1\.4\\alpha=1\.4, and the most recent128128tokens are retained in BF16\. The same configuration is used across all models, layers, context lengths, and benchmarks\. All experiments are conducted on NVIDIA H20 96GB GPUs\.

### 5\.2Main Results

Table 3:Accuracy \(%\) on LongBench\-E and RULER\.Boldindicates the best method\.LongBench\-ERULERMethodBitSQAMQASumFSSynCodeAVG4K8K16K32K64K128KAVGQwen3\-8BBF1616\.0018\.7013\.0725\.7967\.9255\.3368\.5741\.5694\.6693\.6893\.2991\.2978\.7872\.1387\.31KIVI2\.2516\.8112\.2524\.3067\.0047\.7363\.8538\.6684\.2281\.1278\.3669\.4348\.3333\.8365\.88QuaRot2\.2516\.8412\.4223\.9967\.3255\.3166\.8440\.4590\.7090\.9788\.2984\.6464\.1856\.4779\.21TurboQuant2\.1914\.5712\.0220\.4361\.9155\.9553\.9736\.4876\.3269\.6363\.4965\.2831\.2721\.1854\.53OScaR2\.2617\.0412\.8324\.2067\.4855\.7466\.7040\.6789\.6088\.9186\.8283\.1262\.9357\.1478\.09D\-Quant1\.96\{\}\_\{\\text\{1\.96\}\}1\.9618\.0513\.1825\.3067\.2152\.3267\.7540\.6493\.9092\.3792\.8590\.1174\.3867\.6785\.21D\-Quant2\.2618\.3312\.9126\.0167\.7151\.5269\.5641\.0093\.7893\.2793\.1692\.1375\.7770\.9286\.50Llama\-3\.1\-8BBF1616\.0019\.9617\.6830\.6667\.2957\.3862\.0342\.5095\.6995\.8294\.7089\.5185\.9676\.4489\.69KIVI2\.2519\.9224\.1928\.9566\.3157\.5056\.8842\.2987\.2483\.2479\.6875\.3769\.1851\.8774\.43QuaRot2\.2518\.2615\.2925\.4964\.4146\.0747\.4536\.1680\.6876\.0171\.2465\.5360\.7438\.1565\.39TurboQuant2\.1918\.1515\.1827\.8066\.2756\.5256\.0540\.0090\.5287\.1685\.6977\.7671\.7853\.8477\.79OScaR2\.2619\.1817\.1230\.0067\.8157\.4859\.5841\.8793\.9893\.2790\.9785\.4080\.3066\.1585\.01D\-Quant1\.96\{\}\_\{\\text\{1\.96\}\}1\.9618\.9717\.4529\.1967\.0756\.6361\.2341\.7694\.3994\.1492\.7287\.0984\.5270\.7287\.26D\-Quant2\.2619\.0117\.2730\.4167\.7356\.5661\.5942\.1095\.5994\.7893\.9789\.1085\.5374\.6588\.94

#### D\-Quant consistently preserves long\-context performance\.

As shown in Table[3](https://arxiv.org/html/2609.19880#S5.T3), D\-Quant at2\.262\.26bits per value consistently preserves model quality across both models\. On RULER, D\-Quant achieves average scores of86\.5086\.50and88\.9488\.94on Qwen3\-8B and Llama\-3\.1\-8B, respectively, closely matching the BF16 references of87\.3187\.31and89\.6989\.69while substantially outperforming all low\-bit baselines\. On LongBench\-E, where the differences among compressed methods are considerably smaller, D\-Quant remains competitive and maintains performance comparable to BF16\.

#### D\-Quant remains robust as context length increases\.

The performance gap between D\-Quant and existing quantization methods becomes more pronounced at longer context lengths\. While most baselines perform reasonably well at short contexts, their accuracy deteriorates rapidly as the sequence length increases\. D\-Quant, in contrast, consistently tracks the BF16 reference across the entire context range\. At128128K, D\-Quant achieves70\.9270\.92and74\.6574\.65on Qwen3\-8B and Llama\-3\.1\-8B, respectively, while the strongest baselines reach only57\.1457\.14and66\.1566\.15\.

#### D\-Quant remains strong at a lower storage budget\.

D\-Quant can further reduce the storage budget without being constrained by conventional fixed bit widths\. At1\.961\.96bits per value, it still achieves RULER averages of85\.2185\.21on Qwen3\-8B and87\.2687\.26on Llama\-3\.1\-8B, outperforming all baselines despite using less storage\. This demonstrates that D\-Quant enables more flexible trade\-offs between storage and accuracy than fixed\-width quantization methods\.

### 5\.3Efficiency

We evaluate the end\-to\-end efficiency of D\-Quant on Qwen3\-8B using a single NVIDIA H20 96GB GPU\. All measurements use the actual compressed KV cache\. We consider two settings: \(1\) an88K prompt with increasing batch sizes for decoding throughput and peak GPU memory, and \(2\) batch size11with a11K prompt and continuous decoding to6464K tokens for long\-context KV cache memory\.

Figure 3:System efficiency on Qwen3\-8B\.Left: decoding throughput with increasing batch size for an88K prompt\. Middle: peak GPU memory under the same setting\. Right: KV cache memory when decoding from a11K prompt to6464K tokens at batch size11\.#### D\-Quant substantially reduces KV cache memory\.

As shown in Figure[3](https://arxiv.org/html/2609.19880#S5.F3), D\-Quant significantly reduces memory consumption in both high\-batch and long\-context settings\. With an88K prompt, BF16 runs out of memory at batch size6464, whereas D\-Quant scales to batch size128128with only49\.449\.4GiB peak memory\. During long\-context decoding, D\-Quant reduces the KV cache memory by approximately7×7\\timesat6464K\. The smaller memory footprint directly enables substantially higher serving capacity under a fixed GPU memory budget\.

#### Higher capacity translates into higher serving throughput\.

Entropy decoding introduces additional computation, making D\-Quant slower than BF16 at small batch sizes\. As the batch size increases, the reduced KV cache enables substantially higher throughput\. At their respective maximum feasible batch sizes, D\-Quant reaches563\.6563\.6tokens/s compared with159\.5159\.5tokens/s for BF16, yielding up to3\.5×3\.5\\timeshigher throughput\. Overall, D\-Quant trades additional decoding computation for substantially lower memory consumption and higher memory\-constrained serving throughput\.

### 5\.4Analysis

Figure 4:Ablation of D\-Quant’s main design choices on Qwen3\-8B at 128K\.Each panel varies one configuration while keeping the others fixed\. Left: quantization levels at a fixed2\.262\.26bits per value\. Middle: storage budget with8/68/6key/value levels\. Right: K/V budget allocation at a fixed total storage of2\.262\.26bits per value\. The star marks the default configuration\.We explore the main design choices of D\-Quant on RULER at128128K with Qwen3\-8B\.

#### Larger alphabets improve representation until drift becomes costly\.

In our main configuration, we allocate2\.682\.68bits to keys and1\.841\.84bits to values, which correspond to approximately66and33levels, respectively\. Since entropy coding compresses the non\-uniform symbol distribution more efficiently, D\-Quant can use larger alphabets under the same storage budget\. As shown in Figure[4](https://arxiv.org/html/2609.19880#S5.F4)\(left\), using88levels for keys and66for values provides a favorable operating point\. Further increasing either alphabet makes drift more frequent to satisfy the container constraint, eventually degrading model performance\.

#### Increasing the storage budget steadily approaches BF16 performance\.

Figure[4](https://arxiv.org/html/2609.19880#S5.F4)\(middle\) shows that increasing the container size consistently improves accuracy, with the gain gradually saturating as performance approaches the BF16 reference\. This provides D\-Quant with a flexible trade\-off between KV cache storage and model quality\.

#### Keys benefit from a moderately larger budget\.

Most KV cache quantization methods allocate the same storage budget to keys and values\. As shown in Figure[4](https://arxiv.org/html/2609.19880#S5.F4)\(right\), assigning moderately more capacity to keys improves performance, consistent with their higher sensitivity to quantization error\. However, overly skewing the budget toward keys eventually hurts accuracy as the value representation becomes too constrained\. We therefore allocate slightly more bits to keys in our method\.

## 6Conclusion

We present D\-Quant, an entropy\-coded KV cache quantization framework that combines the compression efficiency of entropy coding with the regular memory layout required for efficient inference\. D\-Quant encodes each KV stream into a fixed\-size container and uses length\-constrained drift to ensure that every stream fits its storage budget\. Experiments across models and long\-context benchmarks show that D\-Quant consistently preserves near\-BF16 performance while substantially reducing KV cache memory and improving serving throughput\.

## References

- Ainslie et al\. \(2023\)Joshua Ainslie, James Lee\-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai\.GQA: Training generalized multi\-query transformer models from multi\-head checkpoints\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, 2023\.
- Ashkboos et al\. \(2024\)Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L\. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman\.QuaRot: Outlier\-free 4\-bit inference in rotated LLMs\.In*Advances in Neural Information Processing Systems*, 2024\.
- Bai et al\. \(2024\)Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li\.LongBench: A bilingual, multitask benchmark for long context understanding\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics*, 2024\.
- Dao \(2023\)Tri Dao\.FlashAttention\-2: Faster attention with better parallelism and work partitioning\.*arXiv preprint arXiv:2307\.08691*, 2023\.
- Duda \(2013\)Jarek Duda\.Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding\.*arXiv preprint arXiv:1311\.2540*, 2013\.
- Efron & Tibshirani \(1994\)Bradley Efron and Robert J\. Tibshirani\.*An Introduction to the Bootstrap*\.Chapman and Hall/CRC, 1994\.
- Frantar et al\. \(2023\)Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh\.GPTQ: Accurate post\-training quantization for generative pre\-trained transformers\.In*International Conference on Learning Representations*, 2023\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al\.The Llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Guo & Joshi \(2026\)Yipin Guo and Siddharth Joshi\.SplitZip: Ultra fast lossless KV compression for disaggregated LLM serving\.*arXiv preprint arXiv:2605\.01708*, 2026\.
- Hooper et al\. \(2024\)Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Michael W\. Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami\.KVQuant: Towards 10 million context length LLM inference with KV cache quantization\.In*Advances in Neural Information Processing Systems*, 2024\.
- Hsieh et al\. \(2024\)Cheng\-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg\.RULER: What’s the real context size of your long\-context language models?*arXiv preprint arXiv:2404\.06654*, 2024\.
- Huffman \(1952\)David A\. Huffman\.A method for the construction of minimum\-redundancy codes\.*Proceedings of the IRE*, 40\(9\):1098–1101, 1952\.
- Jiang et al\. \(2025\)Bo Jiang, Taolue Yang, Youyuan Liu, Chengming Zhang, Xubin He, and Sian Jin\.KVComp: A high\-performance, LLM\-aware, lossy compression framework for KV cache\.*arXiv preprint arXiv:2509\.00579*, 2025\.
- Kang et al\. \(2024\)Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao\.GEAR: An efficient KV cache compression recipe for near\-lossless generative inference of LLM\.*arXiv preprint arXiv:2403\.05527*, 2024\.
- Li et al\. \(2024\)Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen\.SnapKV: LLM knows what you are looking for before generation\.In*Advances in Neural Information Processing Systems*, 2024\.
- Lin et al\. \(2024\)Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware weight quantization for on\-device LLM compression and acceleration\.In*Proceedings of Machine Learning and Systems*, 2024\.
- Liu et al\. \(2024a\)Yuhan Liu, Hanchen Li, Yihua Cheng, Siddhant Ray, Yuyang Huang, Qizheng Zhang, Kuntai Du, Jiayi Yao, Shan Lu, Ganesh Ananthanarayanan, Michael Maire, Henry Hoffmann, Ari Holtzman, and Junchen Jiang\.CacheGen: KV cache compression and streaming for fast large language model serving\.In*Proceedings of the ACM SIGCOMM 2024 Conference*, 2024a\.
- Liu et al\. \(2024b\)Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu\.KIVI: A tuning\-free asymmetric 2bit quantization for KV cache\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 32332–32344\. PMLR, 2024b\.
- Marcellin & Fischer \(1990\)Michael W\. Marcellin and Thomas R\. Fischer\.Trellis coded quantization of memoryless and Gauss\-Markov sources\.*IEEE Transactions on Communications*, 38\(1\):82–93, 1990\.
- Marpe et al\. \(2003\)Detlev Marpe, Heiko Schwarz, and Thomas Wiegand\.Context\-based adaptive binary arithmetic coding in the H\.264/AVC video compression standard\.*IEEE Transactions on Circuits and Systems for Video Technology*, 13\(7\):620–636, 2003\.
- Ortega & Ramchandran \(1998\)Antonio Ortega and Kannan Ramchandran\.Rate\-distortion methods for image and video compression\.*IEEE Signal Processing Magazine*, 15\(6\):23–50, 1998\.
- Peng et al\. \(2024\)Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole\.YaRN: Efficient context window extension of large language models\.In*International Conference on Learning Representations*, 2024\.
- Shannon \(1948\)Claude E\. Shannon\.A mathematical theory of communication\.*The Bell System Technical Journal*, 27\(3\):379–423, 1948\.
- Su et al\. \(2024\)Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568:127063, 2024\.
- Su et al\. \(2026\)Zunhai Su, Rui Yang, Chao Zhang, Yaxiu Liu, Yifan Zhang, Wei Wu, Jing Xiong, Dayou Du, Xialie Zhuang, Yulei Qian, Yuchen Xie, Yik\-Chung Wu, Hongxia Yang, and Ngai Wong\.OScaR: The occam’s razor for extreme KV cache quantization in LLMs and beyond\.*arXiv preprint arXiv:2605\.19660*, 2026\.
- Sullivan & Wiegand \(1998\)Gary J\. Sullivan and Thomas Wiegand\.Rate\-distortion optimization for video compression\.*IEEE Signal Processing Magazine*, 15\(6\):74–90, 1998\.
- Tseng et al\. \(2024\)Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa\.QuIP\#: Even better LLM quantization with hadamard incoherence and lattice codebooks\.In*Proceedings of the 41st International Conference on Machine Learning*, 2024\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, 2017\.
- Wiedemann et al\. \(2020\)Simon Wiedemann, Heiner Kirchhoffer, Stefan Matlage, Paul Haase, Arturo Marban, Talmaj Marinč, David Neumann, Tung Nguyen, Heiko Schwarz, Thomas Wiegand, Detlev Marpe, and Wojciech Samek\.DeepCABAC: A universal compression algorithm for deep neural networks\.*IEEE Journal of Selected Topics in Signal Processing*, 14\(4\):700–714, 2020\.
- Witten et al\. \(1987\)Ian H\. Witten, Radford M\. Neal, and John G\. Cleary\.Arithmetic coding for data compression\.*Communications of the ACM*, 30\(6\):520–540, 1987\.
- Xiao et al\. \(2023\)Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han\.SmoothQuant: Accurate and efficient post\-training quantization for large language models\.In*Proceedings of the 40th International Conference on Machine Learning*, 2023\.
- Xiao et al\. \(2024\)Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis\.Efficient streaming language models with attention sinks\.In*International Conference on Learning Representations*, 2024\.
- Yang et al\. \(2025\)An Yang, Anfeng Li, Baosong Yang, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Zandieh et al\. \(2026\)Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni\.TurboQuant: Online vector quantization with near\-optimal distortion rate\.In*International Conference on Learning Representations*, 2026\.
- Zhang et al\. \(2023\)Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Zhangyang Wang, and Beidi Chen\.H2O: Heavy\-hitter oracle for efficient generative inference of large language models\.In*Advances in Neural Information Processing Systems*, 2023\.
- Zhao et al\. \(2024\)Yilong Zhao, Chien\-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci\.Atom: Low\-bit quantization for efficient and accurate LLM serving\.In*Proceedings of Machine Learning and Systems*, 2024\.

## Appendix AEncoding and Decoding

This section provides the detailed encoding and decoding procedures omitted from the main text\. Algorithm[1](https://arxiv.org/html/2609.19880#alg1)gives the encoder for one stream\. Two implementation details are worth noting\. First, we bisect the exponent of the Lagrange multiplier rather than the multiplier itself, which provides stable convergence across the dynamic range of distortion values\. Second, the scale and offset are rounded to FP16 before the final assignment, so that the encoder optimizes against exactly the reconstruction grid used by the decoder\.

Algorithm 1Encoding one stream1:values

𝒙∈ℝn\{\\bm\{x\}\}\\in\\mathbb\{R\}^\{n\}, frequency table

𝒇\{\\bm\{f\}\}of size

MM, container

CCbytes

2:

B←8​C−σB\\leftarrow 8C\-\\sigma;

rm←−log2⁡\(fm/∑jfj\)r\_\{m\}\\leftarrow\-\\log\_\{2\}\(f\_\{m\}/\\textstyle\\sum\_\{j\}f\_\{j\}\);

s,o←FitMinMax​\(𝒙,M\)s,o\\leftarrow\\textsc\{FitMinMax\}\(\{\\bm\{x\}\},M\)
3:

𝒎←Drift​\(𝒙,s,o,𝒓,B\)\{\\bm\{m\}\}\\leftarrow\\textsc\{Drift\}\(\{\\bm\{x\}\},s,o,\{\\bm\{r\}\},B\)
4:

s,o←Refit​\(𝒙,𝒎\)s,o\\leftarrow\\textsc\{Refit\}\(\{\\bm\{x\}\},\{\\bm\{m\}\}\)
5:

s,o←fp16​\(s\),fp16​\(o\)s,o\\leftarrow\\textsc\{fp16\}\(s\),\\textsc\{fp16\}\(o\)
6:

𝒎←Drift​\(𝒙,s,o,𝒓,B\)\{\\bm\{m\}\}\\leftarrow\\textsc\{Drift\}\(\{\\bm\{x\}\},s,o,\{\\bm\{r\}\},B\)
7:return

rANSEncode​\(𝒎,𝒇,C\),s,o\\textsc\{rANSEncode\}\(\{\\bm\{m\}\},\{\\bm\{f\}\},C\),\\,s,\\,o
8:

9:functionDrift\(

𝒙,s,o,𝒓,B\{\\bm\{x\}\},s,o,\{\\bm\{r\}\},B\)

10:

Di​m←\(xi−\(o\+s​m\)\)2D\_\{im\}\\leftarrow\(x\_\{i\}\-\(o\+s\\,m\)\)^\{2\}
11:

hi←log2⁡\(8​\(max⁡D−min⁡D\)\)\\text\{hi\}\\leftarrow\\log\_\{2\}\\\!\\bigl\(8\(\\max D\-\\min D\)\\bigr\);

lo←hi−40\\text\{lo\}\\leftarrow\\text\{hi\}\-40
12:for

11to

TTdo⊳\\trianglerightT=12T=12; bisectlog2⁡λ\\log\_\{2\}\\lambda

13:

μ←\(lo\+hi\)/2\\mu\\leftarrow\(\\text\{lo\}\+\\text\{hi\}\)/2
14:

𝒎←arg⁡minm⁡\(Di​m\+2μ​rm\)\{\\bm\{m\}\}\\leftarrow\\arg\\min\_\{m\}\\left\(D\_\{im\}\+2^\{\\mu\}r\_\{m\}\\right\)
15:if

∑irmi≤B\\sum\_\{i\}r\_\{m\_\{i\}\}\\leq Bthen

hi←μ\\text\{hi\}\\leftarrow\\muelse

lo←μ\\text\{lo\}\\leftarrow\\mu
16:endfor

17:return

arg⁡minm⁡\(Di​m\+2hi​rm\)\\arg\\min\_\{m\}\\left\(D\_\{im\}\+2^\{\\text\{hi\}\}r\_\{m\}\\right\)
18:endfunction

Algorithm[2](https://arxiv.org/html/2609.19880#alg2)gives the corresponding rANS decoder\. The stream is encoded in reverse order so that decoding produces symbols in their original order\. Withp=8p=8, the low eight bits of the state directly index a256256\-entry table\. Each entry stores the reconstruction level together with the frequency and cumulative offset required for the state update, allowing entropy decoding and reconstruction to share a single lookup\.

Algorithm 2Decoding one stream1:payload of

CCbytes, table

T⁡\[z\]=\(vm,fm,cm\)T\[z\]=\(v\_\{m\},f\_\{m\},c\_\{m\}\)
2:

x←LittleEndian\(payload\[0:σ/8\]\)x\\leftarrow\\textsc\{LittleEndian\}\(\\text\{payload\}\[0\{:\}\\sigma/8\]\);

ptr←σ/8\\text\{ptr\}\\leftarrow\\sigma/8
3:for

i=1i=1to

nndo

4:

z←xmod2pz\\leftarrow x\\bmod 2^\{p\}
5:

\(vmi,fmi,cmi\)←T⁡\[z\]\(v\_\{m\_\{i\}\},f\_\{m\_\{i\}\},c\_\{m\_\{i\}\}\)\\leftarrow T\[z\]
6:

x←fmi​⌊x/2p⌋\+z−cmix\\leftarrow f\_\{m\_\{i\}\}\\lfloor x/2^\{p\}\\rfloor\+z\-c\_\{m\_\{i\}\}
7:while

x<Lx<Ldo

8:

x←\(x≪8\)\|payload​\[ptr\]x\\leftarrow\(x\\ll 8\)\\mathbin\{\|\}\\text\{payload\}\[\\text\{ptr\}\];

ptr←ptr\+1\\text\{ptr\}\\leftarrow\\text\{ptr\}\+1
9:endwhile

10:Consume\(i,vmi\)\(i,v\_\{m\_\{i\}\}\)

11:endfor

The decoded levels are consumed directly by the attention kernel rather than materialized as reconstructed KV tensors\. For a key stream of tokentt, the reconstructed channeljjis

k^t,j=st​vmt,j\+ot\+μj,\\hat\{k\}\_\{t,j\}=s\_\{t\}v\_\{m\_\{t,j\}\}\+o\_\{t\}\+\\mu\_\{j\},\(7\)where𝝁\{\\bm\{\\mu\}\}is the windowed channel\-wise mean\. Sincests\_\{t\}andoto\_\{t\}are constant across the channel contraction,

𝒒⊤​𝒌^t=st​∑jqj​vmt,j\+ot​∑jqj\+∑jqj​μj\.\{\\bm\{q\}\}^\{\\top\}\\hat\{\{\\bm\{k\}\}\}\_\{t\}=s\_\{t\}\\sum\_\{j\}q\_\{j\}v\_\{m\_\{t,j\}\}\+o\_\{t\}\\sum\_\{j\}q\_\{j\}\+\\sum\_\{j\}q\_\{j\}\\mu\_\{j\}\.\(8\)The inner product can therefore accumulate decoded reconstruction levels directly and apply the affine terms after the contraction\.

For values, the contraction is instead performed across tokens\. For output channeljj,

∑tat​v^t,j=∑t\(at​st\)​vmt,j\+∑tat​ot,\\sum\_\{t\}a\_\{t\}\\hat\{v\}\_\{t,j\}=\\sum\_\{t\}\(a\_\{t\}s\_\{t\}\)v\_\{m\_\{t,j\}\}\+\\sum\_\{t\}a\_\{t\}o\_\{t\},\(9\)whereata\_\{t\}is the attention weight for tokentt\. The per\-token scale is folded into the corresponding attention weight, while the offset is accumulated separately\. Thus, neither path requires explicit element\-wise dequantization or materialization of a BF16 KV cache\. Since every stream occupies exactlyCCbytes, its address is determined directly from the token index without an offset table or dynamic allocator\.

## Appendix BComparison with Fixed\-Width Coding at Matched Storage

This experiment isolates D\-Quant’s coding mechanism from its preprocessing\. We keep the preprocessing, grouping, and storage budget unchanged and compare the entropy\-coded container with a fixed\-width representation at matched storage\.

#### Setting\.

We use2727held\-out LongBench\-E documents per model, with three documents sampled from each of nine subsets\. Each document provides a2,0482\{,\}048\-token context\. We replace the cached keys and values of every layer with their reconstructed counterparts and measure the token\-wise forward KL divergence from the unquantized model over the following256256positions\. We report the geometric mean of the per\-document KL ratios, with95%95\\%confidence intervals obtained from20,00020\{,\}000paired bootstrap resamples\.

#### Compared configurations\.

Both configurations use identical Hadamard rotation, windowed INT8 channel\-mean removal for keys, token\-wise min–max affine grids, per\-token grouping, least\-squares refitting, and FP16 metadata\. D\-Quant uses the default331331/231231\-byte key/value containers with88/66levels, rANS, and drift\. The fixed\-width counterpart receives the same storage budget and is conservatively charged the ideallog2⁡M\\log\_\{2\}Mbits per symbol, which permits66key levels and33value levels; symbols are assigned by nearest\-neighbor rounding\.

Table 4:D\-Quant versus fixed\-width coding at matched storage\.Both use identical preprocessing and grouping\. KL is measured against the unquantized modelFigure 5:D\-Quant versus fixed\-width coding at matched storage\.Left: paired divergence at the default storage budget\. Right: divergence as the storage budget for each tensor increases\. Stars mark the default D\-Quant operating points, and arrows indicate the additional storage required by fixed\-width coding to reach comparable divergence\.
#### D\-Quant achieves lower distortion at the same storage\.

As shown in Figure[5](https://arxiv.org/html/2609.19880#A2.F5), D\-Quant approximately halves the divergence from the full\-precision model at matched storage, with fixed\-width coding performing worse on every evaluated document\. Conversely, matching D\-Quant’s divergence requires fixed\-width coding to spend an additional0\.2340\.234and0\.2670\.267bits per value on Qwen3\-8B and Llama\-3\.1\-8B, respectively\. This confirms that the improvement is not due to preprocessing or additional storage, but to using the available storage more effectively\.

## Appendix CEmpirical Analysis of the Coding Mechanism

Figure 6:Empirical analysis of D\-Quant’s coding mechanism on Qwen3\-8B\.Left: maximum feasible alphabet size under the entropy\-coded container and equal\-rate fixed\-width coding\. Middle: analytic probability model, sharpened coding table, and measured symbol distributions\. Right: distribution of the remaining container budget after drift; only streams near the storage limit require appreciable reassignment\. Results are collected from six layers spanning the model depth\.#### Entropy coding supports a substantially richer alphabet\.

Figure[6](https://arxiv.org/html/2609.19880#A3.F6)\(left\) shows the maximum number of reconstruction levels supported as the payload budget varies\. Fixed\-width coding increases the alphabet only when the available rate crosses the next code\-width threshold, whereas the entropy\-coded container can exploit the concentrated symbol distribution to support substantially more levels at the same rate\. The default8/68/6configuration lies comfortably within this feasible region, providing additional reconstruction resolution while leaving sufficient capacity for drift\.

#### The analytic entropy model closely matches the measured distribution\.

Figure[6](https://arxiv.org/html/2609.19880#A3.F6)\(middle\) compares the calibration\-free probability model with the measured symbol distributions\. Despite using no calibration data, the analytic model incurs only0\.00450\.0045and0\.00390\.0039additional bits per value for keys and values, respectively\. The final coding table is intentionally sharper than this model: drift tends to move symbols toward more probable central levels, and sharpening anticipates the resulting coded distribution\.

#### Drift makes only sparse, local corrections\.

Figure[6](https://arxiv.org/html/2609.19880#A3.F6)\(right\) shows that many streams already have substantial remaining budget, while drift becomes relevant for streams approaching the container limit\. Across all KV streams, only0\.75%0\.75\\%of key symbols and5\.00%5\.00\\%of value symbols are changed, and no changed symbol moves by more than one reconstruction level\. Thus, drift satisfies the hard per\-stream constraint through sparse local adjustments rather than globally reducing the quantization resolution\.