High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration

# High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
Source: [https://arxiv.org/html/2607.16248](https://arxiv.org/html/2607.16248)
###### Abstract

Long\-context large language model inference relies on the KV cache to avoid redundant attention computation, but incurs high memory and bandwidth overheads\. Low\-bit KV\-cache quantization reduces this cost, yet it severely degrade quality; particularly, one\-bit quantization reduces accuracy from 84\.2% to 47\.8% on Llama\-3\.1\-8B under RULER\.

Rather than common beliefs that absolute error of logits, we find that the root cause is*structured local misranking*, where the distribution of logits in top\-KKregion is drifted\. We thereby propose*local distribution restoration*, a new technique that detects steps with high local distribution risk from quantized\-logit features and restores only the selected top\-KKcandidate distribution before token selection\. We implement DGAP to achieve local distribution restoration, with efficient risk detcetors and correctors\. Expeirments show that on Llama\-3\.1\-8B, DGAP recovers K1V1 RULER accuracy from 47\.8% to 83\.2% and reduces distribution drift from 0\.38 to 0\.14; across Llama, Mistral, and Qwen models, it preserves the persistent low\-bit KV\-cache footprint with modest decode overhead\.

## 1Introduction

Large language models \(LLMs\) increasingly rely on long\-context inference for document understanding, retrieval\-augmented generation, reasoning, and code analysis\(Guoet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib40); OpenAI[2024](https://arxiv.org/html/2607.16248#bib.bib41)\)\. However, the KV cache grows linearly with context length and batch size; a Llama\-style 8B model already requires about 8 GB of full\-precision \(FP16\) KV cache for a 128K\-token sequence, making KV\-cache memory and bandwidth key serving bottlenecks\(Kwonet al\.[2023](https://arxiv.org/html/2607.16248#bib.bib15); Yanget al\.[2025b](https://arxiv.org/html/2607.16248#bib.bib20)\)\. KV\-cache quantization reduces this cost by storing cached keys and values at low precision, with recent methods improving quantization through asymmetric formats\(Liuet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib4)\), outlier handling\(Hooperet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib6)\), and mixed\-precision or error\-compensated designs\(Liuet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib16); Zandiehet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib17)\)\.

Despite these advances, our experiments show that aggressive low\-bit KV quantization can still severely degrade quality\. Under K1V1, where both keys and values are stored at 1 bit, RULER accuracy on Llama\-3\.1\-8B drops from 84\.2% to 47\.8%, with similar trends on Mistral\-7B and Qwen2\.5\-14B\. This motivates a closer analysis of how low\-bit KV quantization changes the local candidate distribution before token selection, beyond what memory usage, throughput, and final task scores reveal\.

We find that this degradation is not explained by logit\-magnitude error alone\. In scaled dot\-product attention, attention weights are computed assoftmax​\(Q​K⊤/dk\)​V\\mathrm\{softmax\}\(QK^\{\\top\}/\\sqrt\{d\_\{k\}\}\)V\(Vaswaniet al\.[2017](https://arxiv.org/html/2607.16248#bib.bib37); Xuet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib32); Taoet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib42)\)\. Therefore, key quantization can directly change query\-key scores, reorder attended cache positions, and alter the context mixture used to form next\-token logits\. A controlled analysis that separates score magnitude from candidate ranking shows that imposing the quantized ranking on FP16 scores causes a 31\.8\-point accuracy drop, whereas ranking\-preserving score changes remain within 1\.2 points of FP16\. These results suggest that aggressive low\-bit quantization mainly harms decoding by reordering the local high\-probability candidate distribution, rather than by uniformly shifting logit values\.

Importantly, this distribution shift is often locally recoverable\. Although quantization distorts probability mass and reorders high\-probability candidates, those candidates are often not removed from consideration; instead, they remain within a compact quantized top\-KKcandidate region\. We call this phenomenon*structured local misranking*\. It suggests a practical alternative to increasing persistent KV precision or correcting the full vocabulary: restore only the affected local candidate distribution before token selection\.

We proposeDGAP\(Disagreement\-GuidedAdaptivePrecision\), a lightweight layer for restoring local distribution fidelity in low\-bit KV\-cache decoding\. DGAP contains two trainable modules\. Adistribution\-risk detectorestimates local distribution risk from quantized\-logit features, while aselective top\-KKlogit correctorrestores the selected candidate distribution only when the risk exceeds a calibrated threshold\. This separation keeps runtime overhead low: the detector avoids unnecessary correction on stable steps, and the corrector updates only a compact candidate region rather than the full vocabulary\. The remaining logits and the persistent low\-bit KV cache are left unchanged\.

DGAP is an add\-on to existing low\-bit KV\-cache decoders and requires no base\-model retraining or attention\-kernel modification\. Across Llama\-3\.1\-8B, Mistral\-7B, and Qwen2\.5\-14B on LongBench, RULER, MMLU, and WikiText\-2, DGAP improves quality and local distribution fidelity across K1V1–K8V8 settings\. Under K1V1 on Llama\-3\.1\-8B, DGAP raises RULER accuracy from 47\.8% to 83\.2%, close to the 84\.2% FP16 result, reduces distribution drift from 0\.38 to 0\.14, and preserves the persistent 1\-bit KV\-cache footprint with only \(1\.06×1\.06\\times\) relative decode latency\.

This work provides a mechanistic analysis of aggressive low\-bit KV\-cache quantization at the local distribution level\. DGAP turns this analysis into a practical restoration mechanism, showing that recoverable candidate\-distribution reordering can be detected and corrected locally without reconstructing FP16 KV states or increasing persistent KV precision\.

In summary, this paper makes the following contributions:

- •We provide a mechanistic analysis of aggressive low\-bit KV\-cache quantization, showing that quality degradation is strongly associated with local candidate\-distribution reordering rather than logit\-magnitude error alone\.
- •We identify*structured local misranking*, where high\-probability candidates are reordered by quantization but often remain within a compact quantized top\-KKregion, making local distribution restoration possible\.
- •We proposeDGAP, a lightweight distribution\-restoration layer with a risk detector and selective top\-KKcorrector that restores local candidate distributions without changing the persistent low\-bit KV cache\.
- •We evaluate DGAP across Llama, Mistral, and Qwen models on long\-context and language benchmarks, showing improved quality and distribution fidelity across K1V1–K8V8 with modest decode overhead\.

## 2Related Work

#### Long\-context KV\-cache efficiency\.

Long\-context LLM inference is often constrained by KV\-cache memory and bandwidth, since cached keys and values grow linearly with sequence length and batch size\(Shenget al\.[2023](https://arxiv.org/html/2607.16248#bib.bib43)\)\. Prior work improves serving efficiency through paged cache management\(Kwonet al\.[2023](https://arxiv.org/html/2607.16248#bib.bib15); Yanget al\.[2025b](https://arxiv.org/html/2607.16248#bib.bib20)\), I/O\-aware attention kernels\(Daoet al\.[2022](https://arxiv.org/html/2607.16248#bib.bib13)\), and sparse KV retention or eviction based on attention sinks, heavy hitters, or prompt\-level importance\(Xiaoet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib7); Zhanget al\.[2023](https://arxiv.org/html/2607.16248#bib.bib1); Liet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib9); Caiet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib14)\)\. These methods reduce memory fragmentation, memory traffic, or the number of retained KV states\. However, they mainly optimize how KV states are stored, accessed, or selected, and do not directly analyze how aggressive low\-bit KV representation changes the local next\-token candidate distribution during decoding\.

#### KV\-cache quantization and distribution preservation\.

KV\-cache quantization reduces decoding memory and bandwidth by storing cached keys and values at low precision\. Existing methods improve low\-bit quality through asymmetric key/value quantization\(Liuet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib4)\), outlier\-aware and non\-uniform quantization\(Hooperet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib6)\), sparse or low\-rank error compensation\(Kanget al\.[2024](https://arxiv.org/html/2607.16248#bib.bib5)\), progressive mixed precision\(Liuet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib16)\), online vector quantization\(Zandiehet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib17)\), and hierarchical quantized caches for speculative decoding\(Tiwariet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib18)\)\. A related line of quantization\-aware or distribution\-aware methods uses calibration, loss\-aware objectives, rotations, or distillation to reduce mismatch between low\-precision and full\-precision models\(Liuet al\.[2023](https://arxiv.org/html/2607.16248#bib.bib10); Zhang and Shrivastava[2025](https://arxiv.org/html/2607.16248#bib.bib28); Shaoet al\.[2026](https://arxiv.org/html/2607.16248#bib.bib29)\)\. These approaches primarily improve the quantized representation during quantizer design, calibration, or training, while runtime local distribution drift and candidate reordering under aggressive low\-bit KV\-cache decoding remain less explored\.

![Refer to caption](https://arxiv.org/html/2607.16248v1/x1.png)Figure 1:Left: K1V1 sharply degrades RULER accuracy\.Right: controlled variants identify candidate reordering as the main cause\.

## 3Motivation

### 3\.1Accuracy Loss under Aggressive KV Quantization

Although KV\-cache quantization reduces memory footprint\(Liuet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib4); Sunet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib31)\), aggressive low\-bit settings can severely degrade model quality\.

#### Methodology\.

We evaluate Llama\-3\.1\-8B on RULER under FP16 and four KV precision settings: K8V8, K4V4, K2V2, and K1V1, where KbbVbbdenotesbb\-bit keys andbb\-bit values\. All experiments use NVIDIA A100\-80GB GPUs and the setup in Section[5](https://arxiv.org/html/2607.16248#S5)\.

#### Observation \#1: Aggressive low\-bit KV quantization sharply reduces task accuracy\.

As shown in Fig\.[1](https://arxiv.org/html/2607.16248#S2.F1)\(left\), K1V1 reduces RULER accuracy from 84\.2% under FP16 to 47\.8%\. Higher\-bit settings remain close to FP16, with K8V8, K4V4, and K2V2 achieving 84\.0%, 83\.7%, and 81\.4%, respectively\. Thus, task degradation becomes especially severe in the aggressive 1\-bit KV\-cache regime\.

![Refer to caption](https://arxiv.org/html/2607.16248v1/x2.png)Figure 2:Top\-KKrestoration\. Quantization distorts the local top\-KKdistribution, while restoration recovers the selected candidates\.

### 3\.2Decode\-Time Analysis of Accuracy Loss

The sharp task drop motivates a local decode\-time question: how does low\-bit KV quantization alter the high\-probability candidate distribution before token selection?

#### Analysis\.

We consider two possible causes\. First, quantization may change logit magnitudes while preserving the relative ordering of high\-probability candidates\. Second, it may distort probability mass and reorder the local candidate distribution\. Because attention weights are computed from query\-key scores, key quantization can directly change the ranking of attended cache positions, which in turn alters the context mixture used to form next\-token logits\. Thus, local candidate reordering can be more damaging than logit\-magnitude error alone\.

#### Methodology\.

To separate logit\-magnitude changes from candidate\-ranking changes, we construct two controlled variants from FP16 logits\. The first keeps the FP16 logit values but permutes them according to the quantized ranking, isolating the effect of candidate reordering\. The second applies a ranking\-preserving score transformation to FP16 logits\. In our control, this is implemented as a constant offset, which changes raw logit values but leaves the softmax distribution and token ranking unchanged\.

#### Observation \#2: Local distribution reordering dominates logit\-value changes\.

As shown in Fig\.[1](https://arxiv.org/html/2607.16248#S2.F1), the distribution\-preserving variant remains within 1\.2 points of FP16, while replacing the FP16 ranking with the quantized ranking causes a 31\.8\-point accuracy drop\. This indicates that quality loss under aggressive low\-bit KV quantization is driven mainly by local candidate\-distribution reordering, rather than logit\-magnitude changes alone\. Fig\.[2](https://arxiv.org/html/2607.16248#S3.F2)illustrates this structured local misranking: quantization reorders high\-probability FP16 candidates, but many of these candidates remain within a compact quantized top\-KKregion\. Therefore, preserving local candidate\-distribution fidelity and relative rank structure is crucial for aggressive low\-bit KV decoding\.

A simple pairwise analysis explains why such reordering changes the local distribution\. For two candidatesiiandjjwithzifp\>zjfpz\_\{i\}^\{\\mathrm\{fp\}\}\>z\_\{j\}^\{\\mathrm\{fp\}\}, the FP16 distribution assigns a larger probability toiithan tojj, since

pifppjfp=exp⁡\(zifp−zjfp\)\>1\.\\frac\{p\_\{i\}^\{\\mathrm\{fp\}\}\}\{p\_\{j\}^\{\\mathrm\{fp\}\}\}=\\exp\(z\_\{i\}^\{\\mathrm\{fp\}\}\-z\_\{j\}^\{\\mathrm\{fp\}\}\)\>1\.Under quantized logitsz^i=zifp\+ei\\hat\{z\}\_\{i\}=z\_\{i\}^\{\\mathrm\{fp\}\}\+e\_\{i\}, the corresponding probability ratio becomes

p^ip^j=exp⁡\(\(zifp−zjfp\)\+\(ei−ej\)\)\.\\frac\{\\hat\{p\}\_\{i\}\}\{\\hat\{p\}\_\{j\}\}=\\exp\\big\(\(z\_\{i\}^\{\\mathrm\{fp\}\}\-z\_\{j\}^\{\\mathrm\{fp\}\}\)\+\(e\_\{i\}\-e\_\{j\}\)\\big\)\.The local ordering reverses when

z^j\>z^i⇔ej−ei\>zifp−zjfp\.\\hat\{z\}\_\{j\}\>\\hat\{z\}\_\{i\}\\quad\\Leftrightarrow\\quad e\_\{j\}\-e\_\{i\}\>z\_\{i\}^\{\\mathrm\{fp\}\}\-z\_\{j\}^\{\\mathrm\{fp\}\}\.Thus, even moderate non\-uniform errors can redistribute probability mass and reorder candidates when the FP16 margin is small\. In contrast, adding a constant offsetccgives

softmax⁡\(𝐳fp\+c​𝟏\)=softmax⁡\(𝐳fp\),\\operatorname\{softmax\}\(\\mathbf\{z\}^\{\\mathrm\{fp\}\}\+c\\mathbf\{1\}\)=\\operatorname\{softmax\}\(\\mathbf\{z\}^\{\\mathrm\{fp\}\}\),so token probabilities and ranking remain unchanged\.

### 3\.3Key Idea: Local Distribution Restoration

The analysis above suggests that quality loss from aggressive low\-bit KV\-cache quantization can be mitigated by restoring the affected local top\-KKcandidate distribution before token selection\. A naive distribution\-restoration pipeline would involve three steps\. First, compare the quantized decoding distribution with an FP16 reference to identify whether the current step has drifted\. Second, search the full vocabulary to locate tokens whose probability mass or relative ordering has changed\. Third, adjust the affected logits so that the resulting distribution better matches the FP16 distribution\.

#### Challenges\.

Although this pipeline is conceptually straightforward, distribution restoration is costly inside autoregressive decoding\. First, drift detection cannot rely on FP16 logits or extra reference forward passes at runtime, since this would defeat the purpose of low\-bit decoding\. Second, full\-vocabulary restoration is impractical because searching and correcting tens of thousands of logits at every decoding step would add substantial compute and memory traffic\. Third, correction must remain local and predictable; falling back to higher persistent KV precision or applying dense vocabulary\-wide updates would undermine the memory and latency gains of quantization\. These challenges motivate a lightweight detector–corrector framework that identifies risky decoding steps from quantized logits and restores only a compact top\-KKcandidate distribution\.

![Refer to caption](https://arxiv.org/html/2607.16248v1/x3.png)Figure 3:DGAP runtime distribution\-restoration pipeline\. At each decoding step, DGAP estimates distribution risk from quantized logits𝐳^t\\hat\{\\mathbf\{z\}\}\_\{t\}and, only whenrt\>τr\_\{t\}\>\\tau, restores the selected top\-KKcandidate slice before token selection; otherwise, the original logits are used unchanged\.

## 4Methodology

### 4\.1DGAP Overview

We propose DGAP \(Disagreement\-GuidedAdaptivePrecision\), a lightweight local distribution\-restoration layer for low\-bit KV\-cache decoding\.

#### Online restoration framework\.

As shown in Fig\.[3](https://arxiv.org/html/2607.16248#S3.F3), DGAP operates on logits produced by an existing low\-bit KV\-cache decoder\. At decoding steptt, the quantized decoder outputs𝐳^t\(b\)∈ℝV\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\\in\\mathbb\{R\}^\{V\}under precisionbb\. For analysis, we write

𝐳^t\(b\)=𝐳tfp\+𝜹t,\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}=\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}\+\\boldsymbol\{\\delta\}\_\{t\},\(1\)where𝐳tfp\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}denotes the FP16 logits and𝜹t\\boldsymbol\{\\delta\}\_\{t\}denotes the quantization\-induced logit error\. DGAP does not attempt to estimate the full error vector𝜹t\\boldsymbol\{\\delta\}\_\{t\}\. Instead, Eq\. \([1](https://arxiv.org/html/2607.16248#S4.E1)\) is used only to describe the mismatch between FP16 and low\-bit decoding; the goal is to identify and restore the affected local top\-KKcandidate distribution\. During online inference, neither𝐳tfp\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}nor𝜹t\\boldsymbol\{\\delta\}\_\{t\}is available, so DGAP estimates distribution risk using only features derived from𝐳^t\(b\)\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\.

DGAP has two trainable modules: a*distribution\-risk detector*and a*selective top\-KKdistribution corrector*\. Given runtime featuresϕt=Φ​\(𝐳^t\(b\)\)\\boldsymbol\{\\phi\}\_\{t\}=\\Phi\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\), the detector predicts a local distribution\-risk scorertr\_\{t\}, indicating whether the quantized candidate distribution is likely to be reordered or distorted\. Ifrt≤τr\_\{t\}\\leq\\tau, DGAP uses𝐳^t\(b\)\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}directly for token selection\. Ifrt\>τr\_\{t\}\>\\tau, DGAP selects the active local candidate region

𝒮t\(b\)=TopK⁡\(𝐳^t\(b\),Kb\),\\mathcal\{S\}^\{\(b\)\}\_\{t\}=\\operatorname\{TopK\}\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\},K\_\{b\}\),\(2\)predicts updates only for tokens in𝒮t\(b\)\\mathcal\{S\}^\{\(b\)\}\_\{t\}, and merges the corrected slice back into the full logit vector before token selection\. Thus, the detector runs at every step, while the corrector is invoked only on high\-risk steps to restore the local candidate distribution and relative ordering\. Thus, the detector runs at every step, while the corrector is invoked only on high\-risk steps to restore the local candidate distribution and relative ordering\. The persistent low\-bit KV cache is never modified\.

#### Offline supervision\.

DGAP is trained offline from paired FP16 and quantized\-KV decoding traces constructed from the PG\-19 training split\(Raeet al\.[2019](https://arxiv.org/html/2607.16248#bib.bib38)\)\. We use PG\-19 because it contains naturally long documents, making it suitable for collecting long\-context KV\-cache traces without using examples from the downstream evaluation benchmarks\. The calibration sequences are split into trace\-training and validation subsets, both disjoint from LongBench, RULER, MMLU, and WikiText\-2 evaluation data\. To expose DGAP to different cache lengths, we sample prefixes at multiple context lengths, such as 8K, 16K, 32K, and 64K tokens, when supported by the backbone model\.

For each PG\-19 sequence, we run the same model twice with the same prefix and target tokens: once with an FP16 KV cache and once with the target low\-bit KV cache\. At each decoding steptt, we record the FP16 logits𝐳tfp\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}, the quantized logits𝐳^t\(b\)\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}, and the corresponding local candidate distributions\. These paired traces provide distribution\-risk labels for the detector and local top\-KKdistribution targets for the corrector\. After training, only the detectorfθf\_\{\\theta\}, correctorgψg\_\{\\psi\}, calibrated thresholdτ\\tau, and recovery\-window sizesKbK\_\{b\}are used at runtime; the FP16 teacher is not used during inference\.

#### Adaptive recovery\-window selection\.

The recovery window required to cover the affected local candidate distribution depends primarily on key precision\. Lower\-bit keys alters query\-key attention scores more strongly, increasing local candidate reordering and requiring larger recovery windows\. Value precision can affect the aggregated representation, but the dominant source of ranking instability comes from key\-induced attention\-score changes\. For each precisionb∈\{K1V1,K2V2,K4V4,K8V8\}b\\in\\\{\\mathrm\{K1V1\},\\mathrm\{K2V2\},\\mathrm\{K4V4\},\\mathrm\{K8V8\}\\\}, we calibrateKbK\_\{b\}from paired FP16 and quantized\-KV validation traces\. Let𝐩tfp=softmax​\(𝐳tfp\)\\mathbf\{p\}^\{\\mathrm\{fp\}\}\_\{t\}=\\mathrm\{softmax\}\(\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}\), and letπtfp\\pi^\{\\mathrm\{fp\}\}\_\{t\}denote the FP16 token order sorted by decreasing probability\. We define𝒞tfp​\(α\)\\mathcal\{C\}^\{\\mathrm\{fp\}\}\_\{t\}\(\\alpha\)as the smallest prefix of this FP16 order whose cumulative probability mass reachesα\\alpha:

𝒞tfp​\(α\)\\displaystyle\\mathcal\{C\}^\{\\mathrm\{fp\}\}\_\{t\}\(\\alpha\)=\{πtfp​\(m\)\}m=1Mt,\\displaystyle=\\\{\\pi^\{\\mathrm\{fp\}\}\_\{t\}\(m\)\\\}\_\{m=1\}^\{M\_\{t\}\},\(3\)Mt\\displaystyle M\_\{t\}=min⁡\{M:∑m=1Mpt,πtfp​\(m\)fp≥α\}\.\\displaystyle=\\min\\Bigg\\\{M:\\sum\_\{m=1\}^\{M\}p^\{\\mathrm\{fp\}\}\_\{t,\\pi^\{\\mathrm\{fp\}\}\_\{t\}\(m\)\}\\geq\\alpha\\Bigg\\\}\.We choose the smallestKbK\_\{b\}whose quantized top\-KKregion covers the target FP16 candidate massρK\\rho\_\{K\}:

Kb=min⁡\{K:𝔼t​\[∑i∈𝒞tfp​\(α\)∩TopK⁡\(𝐳^t\(b\),K\)pt,ifp\]≥ρK\}\.K\_\{b\}=\\min\\left\\\{K:\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\\mathcal\{C\}^\{\\mathrm\{fp\}\}\_\{t\}\(\\alpha\)\\cap\\operatorname\{TopK\}\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\},K\)\}p^\{\\mathrm\{fp\}\}\_\{t,i\}\\right\]\\geq\\rho\_\{K\}\\right\\\}\.\(4\)At runtime, the active precision selectsKbK\_\{b\}, and DGAP corrects only𝒮t\(b\)=TopK⁡\(𝐳^t\(b\),Kb\)\\mathcal\{S\}^\{\(b\)\}\_\{t\}=\\operatorname\{TopK\}\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\},K\_\{b\}\)\. Lower\-bit settings therefore use larger recovery windows, while higher\-bit settings use smaller ones, avoiding both full\-vocabulary correction and uniformly higher persistent KV precision\.

#### Why local candidate reordering occurs\.

The compact recovery region follows from Eq\.[1](https://arxiv.org/html/2607.16248#S4.E1)\. Since attention weights are computed from query\-key scores, key quantization can reorder attended cache positions and induce non\-uniform logit errors\. For candidatesi,ji,jwithzt,ifp\>zt,jfpz^\{\\mathrm\{fp\}\}\_\{t,i\}\>z^\{\\mathrm\{fp\}\}\_\{t,j\}, definemt,i​j=zt,ifp−zt,jfpm\_\{t,ij\}=z^\{\\mathrm\{fp\}\}\_\{t,i\}\-z^\{\\mathrm\{fp\}\}\_\{t,j\}\. Their order reverses when

δt,j−δt,i\>mt,i​j\.\\delta\_\{t,j\}\-\\delta\_\{t,i\}\>m\_\{t,ij\}\.\(5\)Thus, small\-margin high\-probability candidates are most vulnerable to reordering\. This matters because decoding depends on relative probabilities rather than absolute logit error: an order\-preserving shift may leave token selection nearly unchanged, whereas a non\-uniform error that reverses local candidates can move probability mass to a different mode and alter subsequent autoregressive states\. This motivates local top\-KKrestoration rather than full\-vocabulary correction\.

### 4\.2Distribution\-Risk Detector

#### Goal\.

During online inference, the FP16 logits𝐳tfp\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}and quantization\-induced error𝜹t\\boldsymbol\{\\delta\}\_\{t\}are unavailable\. The detector therefore predicts whether the current low\-bit decoding step is likely to exhibit recoverable local candidate\-distribution drift using only features from the quantized logits\. Given lightweight featuresϕt=Φ​\(𝐳^t\(b\)\)\\boldsymbol\{\\phi\}\_\{t\}=\\Phi\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\), it predicts

rt=σ​\(fθ​\(ϕt\)\),rt∈\[0,1\],r\_\{t\}=\\sigma\\\!\\left\(f\_\{\\theta\}\(\\boldsymbol\{\\phi\}\_\{t\}\)\\right\),\\qquad r\_\{t\}\\in\[0,1\],\(6\)wherefθf\_\{\\theta\}is the detector andσ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function\. Correction is triggered only whenrt\>τr\_\{t\}\>\\tau, withτ\\taucalibrated offline on validation traces\.

#### Runtime features\.

The detector uses compact distribution\-shape statistics from𝐳^t\(b\)\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}, including entropy, top\-KbK\_\{b\}probability mass, local gap statistics, probability\-mass concentration, and dispersion among high\-probability candidates\. These features capture uncertainty and local rank instability: small candidate gaps or dispersed probability mass make non\-uniform quantization errors more likely to reorder the local distribution\. All features are computed from quantized logits, without FP16 logits or extra forward passes\.

#### Offline supervision and calibration\.

We train the detector from paired FP16 and quantized\-KV traces\. Let𝐩tfp=softmax⁡\(𝐳tfp\)\\mathbf\{p\}^\{\\mathrm\{fp\}\}\_\{t\}=\\operatorname\{softmax\}\(\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t\}\)and𝐩^t\(b\)=softmax⁡\(𝐳^t\(b\)\)\\hat\{\\mathbf\{p\}\}^\{\(b\)\}\_\{t\}=\\operatorname\{softmax\}\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\)denote the FP16 and quantized next\-token distributions\. Let𝒞tfp​\(α\)\\mathcal\{C\}^\{\\mathrm\{fp\}\}\_\{t\}\(\\alpha\)be the smallest FP16 candidate set whose cumulative probability mass reachesα\\alpha, and let𝒮t\(b\)=TopK⁡\(𝐳^t\(b\),Kb\)\\mathcal\{S\}^\{\(b\)\}\_\{t\}=\\operatorname\{TopK\}\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\},K\_\{b\}\)be the active quantized recovery region\. We measure recoverable candidate coverage as

ct=∑i∈𝒞tfp​\(α\)∩𝒮t\(b\)pt,ifp\.c\_\{t\}=\\sum\_\{i\\in\\mathcal\{C\}^\{\\mathrm\{fp\}\}\_\{t\}\(\\alpha\)\\cap\\mathcal\{S\}^\{\(b\)\}\_\{t\}\}p^\{\\mathrm\{fp\}\}\_\{t,i\}\.\(7\)To measure local distribution distortion, we compare normalized probability vectors over𝒰t=𝒞tfp​\(α\)∪𝒮t\(b\)\\mathcal\{U\}\_\{t\}=\\mathcal\{C\}^\{\\mathrm\{fp\}\}\_\{t\}\(\\alpha\)\\cup\\mathcal\{S\}^\{\(b\)\}\_\{t\}:

dtloc=‖𝐩¯t,𝒰tfp−𝐩¯t,𝒰t\(b\)‖2,d^\{\\mathrm\{loc\}\}\_\{t\}=\\left\\\|\\bar\{\\mathbf\{p\}\}^\{\\mathrm\{fp\}\}\_\{t,\\mathcal\{U\}\_\{t\}\}\-\\bar\{\\mathbf\{p\}\}^\{\(b\)\}\_\{t,\\mathcal\{U\}\_\{t\}\}\\right\\\|\_\{2\},\(8\)where𝐩¯𝒰\\bar\{\\mathbf\{p\}\}\_\{\\mathcal\{U\}\}denotes the probability vector renormalized over𝒰\\mathcal\{U\}\. A step is labeled risky when its local candidate distribution is distorted but sufficient FP16 candidate mass remains recoverable:

yt=𝕀​\[dtloc\>ϵ∧ct≥ρc\]\.y\_\{t\}=\\mathbb\{I\}\\left\[d^\{\\mathrm\{loc\}\}\_\{t\}\>\\epsilon\\;\\land\\;c\_\{t\}\\geq\\rho\_\{c\}\\right\]\.\(9\)Distorted steps with low candidate coverage are not labeled positive because the FP16\-relevant candidates are largely absent from the quantized recovery window, making sparse local correction unreliable\. Here,ρK\\rho\_\{K\}is used to chooseKbK\_\{b\}, whileρc\\rho\_\{c\}determines whether a distorted step is recoverable enough for detector supervision\. The detector is trained with binary cross\-entropy:

ℒrisk=−∑t\[yt​log⁡rt\+\(1−yt\)​log⁡\(1−rt\)\]\.\\mathcal\{L\}\_\{\\mathrm\{risk\}\}=\-\\sum\_\{t\}\\left\[y\_\{t\}\\log r\_\{t\}\+\(1\-y\_\{t\}\)\\log\(1\-r\_\{t\}\)\\right\]\.\(10\)After training,τ\\tauis calibrated offline to trade off restoration quality and correction frequency\.

### 4\.3Selective Top\-KKDistribution Corrector

#### Goal\.

For high\-risk steps, the corrector restores the local candidate distribution before token selection\. Correcting all logits𝐳^t\(b\)∈ℝV\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\\in\\mathbb\{R\}^\{V\}would require full\-vocabulary computation and reduce the efficiency gains of low\-bit KV\-cache decoding\. Motivated by the local reordering analysis in Sec\.[3](https://arxiv.org/html/2607.16248#S3), DGAP corrects only the quantized top\-KbK\_\{b\}recovery window, which is calibrated to cover the high\-probability FP16 candidate mass\.

#### Local candidate correction\.

Whenrt\>τr\_\{t\}\>\\tau, DGAP selects the active recovery window

𝒮t\(b\)=TopK⁡\(𝐳^t\(b\),Kb\),Kb≪V\.\\mathcal\{S\}^\{\(b\)\}\_\{t\}=\\operatorname\{TopK\}\(\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\},K\_\{b\}\),\\qquad K\_\{b\}\\ll V\.\(11\)For tokens in𝒮t\(b\)\\mathcal\{S\}^\{\(b\)\}\_\{t\}, we construct candidate\-level features𝝃t,𝒮t\(b\)\\boldsymbol\{\\xi\}\_\{t,\\mathcal\{S\}^\{\(b\)\}\_\{t\}\}from local logits, ranks, margins, and probabilities\. The correctorgψg\_\{\\psi\}predicts sparse updates for this slice:

Δ​𝐳^t,𝒮t\(b\)=gψ​\(𝝃t,𝒮t\(b\),ϕt\),\\Delta\\hat\{\\mathbf\{z\}\}\_\{t,\\mathcal\{S\}^\{\(b\)\}\_\{t\}\}=g\_\{\\psi\}\(\\boldsymbol\{\\xi\}\_\{t,\\mathcal\{S\}^\{\(b\)\}\_\{t\}\},\\boldsymbol\{\\phi\}\_\{t\}\),\(12\)whereϕt\\boldsymbol\{\\phi\}\_\{t\}denotes the detector features\. The corrected logits are obtained by sparse merge:

z~t,j=\{z^t,j\(b\)\+Δ​z^t,j,j∈𝒮t\(b\)​and​rt\>τ,z^t,j\(b\),otherwise\.\\tilde\{z\}\_\{t,j\}=\\begin\{cases\}\\hat\{z\}^\{\(b\)\}\_\{t,j\}\+\\Delta\\hat\{z\}\_\{t,j\},&j\\in\\mathcal\{S\}^\{\(b\)\}\_\{t\}\\ \\text\{and\}\\ r\_\{t\}\>\\tau,\\\\ \\hat\{z\}^\{\(b\)\}\_\{t,j\},&\\text\{otherwise\}\.\\end\{cases\}\(13\)Thus, non\-selected logits remain unchanged; ifrt≤τr\_\{t\}\\leq\\tau, DGAP bypasses the corrector and uses𝐳~t=𝐳^t\(b\)\\tilde\{\\mathbf\{z\}\}\_\{t\}=\\hat\{\\mathbf\{z\}\}^\{\(b\)\}\_\{t\}\.

#### Training objective\.

The corrector is trained offline using the same paired FP16 and quantized\-KV traces as the detector\. For compactness, letS=𝒮t\(b\)S=\\mathcal\{S\}^\{\(b\)\}\_\{t\}\. For each selected window, we form:

𝐪t,Sfp\\displaystyle\\mathbf\{q\}^\{\\mathrm\{fp\}\}\_\{t,S\}=softmax​\(𝐳t,Sfp/T\),𝐪~t,S=softmax​\(𝐳~t,S/T\),\\displaystyle=\\mathrm\{softmax\}\(\\mathbf\{z\}^\{\\mathrm\{fp\}\}\_\{t,S\}/T\),\\quad\\tilde\{\\mathbf\{q\}\}\_\{t,S\}=\\mathrm\{softmax\}\(\\tilde\{\\mathbf\{z\}\}\_\{t,S\}/T\),\(14\)whereTTis a training temperature\. We match the corrected local distribution to the FP16 local distribution:

ℒdist=∑tytDKL\(𝐪t,Sfp∥𝐪~t,S\)\.\\mathcal\{L\}\_\{\\mathrm\{dist\}\}=\\sum\_\{t\}y\_\{t\}D\_\{\\mathrm\{KL\}\}\\left\(\\mathbf\{q\}^\{\\mathrm\{fp\}\}\_\{t,S\}\\middle\\\|\\tilde\{\\mathbf\{q\}\}\_\{t,S\}\\right\)\.\(15\)
To directly target structured local reordering, we add a pairwise rank\-consistency loss over candidates in the selected window\. Let

𝒫tfp=\{\(i,j\):i,j∈𝒮t\(b\),zt,ifp\>zt,jfp\}\\mathcal\{P\}^\{\\mathrm\{fp\}\}\_\{t\}=\\left\\\{\(i,j\):i,j\\in\\mathcal\{S\}^\{\(b\)\}\_\{t\},\\ z^\{\\mathrm\{fp\}\}\_\{t,i\}\>z^\{\\mathrm\{fp\}\}\_\{t,j\}\\right\\\}\(16\)be the set of FP16\-ordered candidate pairs\. We define

ℒrank=∑tyt​∑\(i,j\)∈𝒫tfpwt,i​j​log⁡\(1\+exp⁡\(−\(z~t,i−z~t,j\)\)\),\\mathcal\{L\}\_\{\\mathrm\{rank\}\}=\\sum\_\{t\}y\_\{t\}\\sum\_\{\(i,j\)\\in\\mathcal\{P\}^\{\\mathrm\{fp\}\}\_\{t\}\}w\_\{t,ij\}\\log\\left\(1\+\\exp\\left\(\-\(\\tilde\{z\}\_\{t,i\}\-\\tilde\{z\}\_\{t,j\}\)\\right\)\\right\),\(17\)wherewt,i​j=pt,ifp−pt,jfpw\_\{t,ij\}=p^\{\\mathrm\{fp\}\}\_\{t,i\}\-p^\{\\mathrm\{fp\}\}\_\{t,j\}weights pairwise ordering errors by their FP16 probability gap\. This pairwise term is used only during offline training and is restricted toKb≪VK\_\{b\}\\ll Vcandidates, so it does not affect decode\-time latency\. The final objective is

ℒcorr=ℒdist\+λrank​ℒrank\+λreg​∑tyt​‖Δ​𝐳^t,𝒮t\(b\)‖22\.\\mathcal\{L\}\_\{\\mathrm\{corr\}\}=\\mathcal\{L\}\_\{\\mathrm\{dist\}\}\+\\lambda\_\{\\mathrm\{rank\}\}\\mathcal\{L\}\_\{\\mathrm\{rank\}\}\+\\lambda\_\{\\mathrm\{reg\}\}\\sum\_\{t\}y\_\{t\}\\,\\left\\\|\\Delta\\hat\{\\mathbf\{z\}\}\_\{t,\\mathcal\{S\}^\{\(b\)\}\_\{t\}\}\\right\\\|\_\{2\}^\{2\}\.\(18\)Thus, the KL term restores the local probability shape, the pairwise rank\-consistency term preserves relative ordering among high\-probability candidates, and the regularizer keeps sparse updates stable\.

#### Why detector–corrector instead of end\-to\-end correction?

A single end\-to\-end corrector could directly map quantized logits to a restored distribution, but it would have to run at every decoding step, including stable steps where correction is unnecessary\. If such a model were made conditional to avoid correcting every token, it would require an internal gating mechanism, effectively reintroducing the detector\. This would increase compute and memory traffic and weaken the efficiency benefit of low\-bit KV\-cache decoding\. DGAP separates the cheap decision from the heavier restoration: the detector runs every step using compact distribution\-shape features, while the corrector is invoked only when the local candidate distribution is distorted but recoverable\. The thresholdτ\\tautherefore controls correction frequency, keeping runtime overhead low while preserving the persistent low\-bit KV\-cache footprint\.

Table 1:LongBench results across three instruction\-tuned models\. DGAP\-1bit substantially recovers the degradation of K1V1 while retaining the 1\-bit KV\-cache setting, surpassing KIVI\-2bit and TurboQuant\-2\.5bit on average and approaching KVQuant\-3bit\.Table 2:Generalization beyond LongBench\. DGAP\-1bit recovers K1V1 degradation across perplexity, knowledge, and controlled long\-context benchmarks\.

## 5Experimental Setup

#### Testbed\.

All experiments run on NVIDIA A100\-80GB GPUs using a unified PyTorch–CUDA implementation\. Unless stated otherwise, we use PyTorch SDPA and evaluate batch\-size\-1 decoding, following the long\-context setting where KV\-cache memory and decode latency are most critical\.

#### Implementation details\.

We use Python 3\.10, PyTorch 2\.5\.1\+cu121, Transformers 4\.45\.1, CUDA 12\.1, and PyTorch SDPA\. All methods use the same evaluation prompts, decoding settings, and random seed for each model–benchmark pair\. DGAP is trained and calibrated offline on PG\-19 traces, while all reported test results are measured on disjoint downstream benchmarks\. DGAP hyperparameters, includingτ\\tau,KbK\_\{b\}, temperatureTT, and loss weights, are selected on validation traces and fixed for test evaluation\. Latency is measured with CUDA synchronization, and we report p95 per\-token decode latency and throughput\.

#### Models and benchmarks\.

We evaluate three instruction\-tuned decoder\-only models: Llama\-3\.1\-8B\-Instruct\(Touvronet al\.[2023](https://arxiv.org/html/2607.16248#bib.bib24)\), Mistral\-7B\-Instruct\(Jianget al\.[2023](https://arxiv.org/html/2607.16248#bib.bib25)\), and Qwen2\.5\-14B\-Instruct\(Yanget al\.[2025a](https://arxiv.org/html/2607.16248#bib.bib36)\)\. Benchmarks include WikiText\-2\(Merityet al\.[2016](https://arxiv.org/html/2607.16248#bib.bib21)\)for perplexity, MMLU\(Hendryckset al\.[2021](https://arxiv.org/html/2607.16248#bib.bib27)\)for knowledge and reasoning, LongBench\(Baiet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib22)\)for practical long\-context tasks, and RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib26)\)for controlled long\-context evaluation\.

#### Baselines\.

We compare DGAP with FP16, uniform low\-bit KV baselines, and recent KV\-cache quantization methods\. FP16 is the full\-precision reference\. K1V1 is the storage\-equivalent aggressive low\-bit baseline, while K2V2, K4V4, and K8V8 show the effect of increasing persistent KV precision\. We further compare against KIVI\-2bit\(Liuet al\.[2024](https://arxiv.org/html/2607.16248#bib.bib4)\), KVQuant\-3bit\(Hooperet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib6)\), and TurboQuant\-2\.5bit\(Zandiehet al\.[2025](https://arxiv.org/html/2607.16248#bib.bib17)\), with KIVI\-2bit serving as the main strong low\-bit reference\.

#### Metrics\.

We report task score, perplexity, KV memory, compression ratio, fallback rate, p95 decode latency, and throughput\. To measure local distribution fidelity, we report local distribution drift from FP16\. These metrics directly measure whether low\-bit quantization reorders the local candidate distribution and whether DGAP restores it\.

Table 3:DGAP gains are largest under aggressive KV quantization and diminish as bitwidth increases\. Scores are averaged over NIAH\-2, NIAH\-3, MK\-2, MK\-3, and MV\.Table 4:DGAP improves local distribution fidelity under KV quantization, leading to consistent RULER accuracy gains across bitwidths\.
### 5\.1End\-task Quality Restoration

We first evaluate whether DGAP restores task quality under aggressive low\-bit KV\-cache quantization\. Tables[1](https://arxiv.org/html/2607.16248#S4.T1)and[2](https://arxiv.org/html/2607.16248#S4.T2)report results on LongBench, WikiText\-2, MMLU, and RULER\. On LongBench,DGAP\-1bitimproves over K1V1 by 36\.2%, 37\.3%, and 36\.7% on Llama\-3\.1\-8B, Mistral\-7B, and Qwen2\.5\-14B, respectively, and exceeds the KIVI\-2bit reference on average\. On WikiText\-2, MMLU, and RULER, DGAP consistently narrows the gap between K1V1 and FP16, reaching higher\-bit quality across models\. These results show that local distribution restoration recovers much of the quality lost under low\-bit KV quantization without increasing persistent KV precision\.

### 5\.2DGAP Gains Decrease with Increasing KV Precision

Table[3](https://arxiv.org/html/2607.16248#S5.T3)reports DGAP performance across KV bitwidths using averaged scores over NIAH\-2, NIAH\-3, MK\-2, MK\-3, and MV\. Across all models, DGAP yields its largest improvements under aggressive quantization \(K1V1\), where performance increases by 27\.9–32\.6 points over the base setting\. As KV precision increases, the gains decrease substantially, with only marginal improvements under K4V4 and K8V8\. This trend indicates that DGAP is most effective in correcting severe low\-bit distortion, where local candidate reordering is strongest, while having limited impact when higher\-bit KV caches already preserve much of the original decoding behavior\.

Table 5:Persistent KV\-cache memory \(GB\) across context lengths for Qwen2\.5\-14B\. DGAP\-1bit does not modify KV storage and maintains the same 1\-bit footprint as K1V1\-1bit, achieving 14\.2×\\timescompression over FP16\.
### 5\.3Decode\-Time Local Distribution Fidelity

We analyze whether DGAP improves decode\-time local distribution fidelity\. As shown in Table[4](https://arxiv.org/html/2607.16248#S5.T4), DGAP consistently reduces local distribution drift under KV quantization, leading to improved RULER accuracy\. Improvements become smaller at higher bitwidths because the base KV representations already preserve the FP16 local candidate structure\. These results indicate that DGAP restores local distribution fidelity by correcting quantization\-induced candidate reordering\.

### 5\.4Efficiency Analysis

#### KV\-Cache Memory Across Context Lengths\.

Table[5](https://arxiv.org/html/2607.16248#S5.T5)reports KV\-cache memory \(GB\) across context lengths for Qwen2\.5\-14B\. DGAP\-1bit maintains the same persistent KV footprint as K1V1\-1bit across all context lengths, since it does not modify the stored KV representation\. As a result, it preserves a constant 14\.2×\\timescompression over FP16\.

#### Decode latency and throughput\.

Table[6](https://arxiv.org/html/2607.16248#S5.T6)reports batch\-size\-1 per\-token decoding latency under K1V1\. DGAP introduces a small overhead of approximately 1\.06×\\times, consistent with Fig\.[4](https://arxiv.org/html/2607.16248#S5.F4)\. The corresponding throughput drop remains below 6%, confirming that DGAP preserves efficient decoding\.

#### Why overhead is small\.

The runtime cost of DGAP is limited to lightweight risk detection and selective top\-KKcorrection\. Since correction is applied only at distribution\-risky decoding steps and restricted to local candidate sets, the additional computation remains sparse and does not scale with sequence length\.

![Refer to caption](https://arxiv.org/html/2607.16248v1/x4.png)Figure 4:DGAP runtime overhead across KV precisions\. Lower\-bit settings trigger more corrections, while higher\-bit settings require fewer fallbacks and incur lower latency overhead\.Table 6:Per\-token decode latency of DGAP under K1V1 using Qwen2\.5\-14B\.Table 7:Ablation on restoration target\. DGAP\-1bit restores local top\-KKdistribution and surpasses higher\-bit KV precision while retaining K1V1 memory footprint\.

### 5\.5Ablation Studies

We analyze four key design choices in DGAP: \(i\) the restoration target, \(ii\) detector–corrector decomposition, and \(iii\) the risk thresholdτ\\taucontrolling adaptive correction\.

#### Restoration target: local distribution vs\. KV precision\.

We first examine what should be restored under low\-bit KV quantization\. Unlike methods that increase persistent KV precision, DGAP directly restores the local top\-KKcandidate distribution at decode time\. As shown in Table[7](https://arxiv.org/html/2607.16248#S5.T7), increasing KV precision \(K2V2, K4V4\) improves performance by reducing information loss, but DGAP\-1bit surpasses higher\-bit settings while retaining the K1V1 memory footprint\. This indicates that correcting local candidate reordering is more effective than uniformly increasing KV precision\.

#### Detector–corrector architecture\.

Fig\.[5](https://arxiv.org/html/2607.16248#S5.F5)compares Linear, MLP, and Transformer DGAP variants in terms of accuracy, latency, and model size\. Linear\-DGAP is efficient but underfits the distribution\-restoration objective, while Transformer\-DGAP gives slightly higher accuracy at higher latency and larger size\. MLP\-DGAP provides the best accuracy–efficiency trade\-off and is used as the default 18MB architecture\.

#### Effect of risk thresholdτ\\tau\.

Finally, we study the impact of the trigger thresholdτ\\tau\. As shown in Fig\.[6](https://arxiv.org/html/2607.16248#S5.F6), lowerτ\\tauincreases correction frequency and improves restoration quality, but introduces higher latency due to more frequent activation\. Higherτ\\taureduces overhead but leaves more local distribution errors uncorrected\. We setτ=0\.60\\tau=0\.60as the best trade\-off, achieving strong accuracy with minimal runtime overhead\.

![Refer to caption](https://arxiv.org/html/2607.16248v1/x5.png)Figure 5:Accuracy–latency tradeoff of DGAP architectures\.![Refer to caption](https://arxiv.org/html/2607.16248v1/x6.png)Figure 6:Effect of risk thresholdτ\\tauon K1V1\+DGAP\.

## 6Conclusion

We presented DGAP, a lightweight decode\-time restoration layer for low\-bit KV\-cache quantized LLMs\. Motivated by the observation that key quantization induces local top\-KKcandidate distribution reordering, DGAP detects recoverable distribution\-risky steps and selectively restores the affected candidate region\. Across benchmarks, DGAP recovers much of the performance loss under K1V1 while preserving the persistent low\-bit KV\-cache footprint and introducing only modest decode\-time overhead\. These results highlight the effectiveness of distribution\-aware local correction as an alternative to increasing KV precision\.

## 7Limitations and Future Work

DGAP focuses on algorithmic distribution restoration at decode time, while leaving system\-level serving optimizations as future work\. Its current implementation uses offline\-calibrated thresholds and recovery windows derived from paired FP16 and quantized traces\. Future extensions may explore online calibration and adaptive window selection for deployment\-specific workloads, as well as kernel\-level optimizations \(e\.g\., Triton/CUDA fusion\) to further reduce runtime overhead in large\-scale serving systems\.

## References

- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou, Y\. Dong, J\. Tang, and J\. Li \(2024\)LongBench: a bilingual, multitask benchmark for long context understanding\.External Links:2308\.14508,[Link](https://arxiv.org/abs/2308.14508)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu, and W\. Xiao \(2025\)PyramidKV: dynamic kv cache compression based on pyramidal information funneling\.External Links:2406\.02069,[Link](https://arxiv.org/abs/2406.02069)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Dao, D\. Y\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with io\-awareness\.External Links:2205\.14135,[Link](https://arxiv.org/abs/2205.14135)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi, X\. Zhang, X\. Yu, Y\. Wu, Z\. F\. Wu, Z\. Gou, Z\. Shao, Z\. Li,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.External Links:ISSN 1476\-4687,[Link](http://dx.doi.org/10.1038/s41586-025-09422-z),[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.External Links:2009\.03300,[Link](https://arxiv.org/abs/2009.03300)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- C\. Hooper, S\. Kim, H\. Mohammadzadeh, M\. W\. Mahoney, Y\. S\. Shao, K\. Keutzer, and A\. Gholami \(2025\)KVQuant: towards 10 million context length llm inference with kv cache quantization\.External Links:2401\.18079,[Link](https://arxiv.org/abs/2401.18079)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1),[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px4.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.External Links:2404\.06654,[Link](https://arxiv.org/abs/2404.06654)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- H\. Kang, Q\. Zhang, S\. Kundu, G\. Jeong, Z\. Liu, T\. Krishna, and T\. Zhao \(2024\)GEAR: an efficient kv cache compression recipe for near\-lossless generative inference of llm\.External Links:2403\.05527,[Link](https://arxiv.org/abs/2403.05527)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.External Links:2309\.06180,[Link](https://arxiv.org/abs/2309.06180)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1),[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)SnapKV: llm knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 22947–22970\.External Links:[Document](https://dx.doi.org/10.52202/079017-0722),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/28ab418242603e0f7323e54185d19bde-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Liu, S\. Li, J\. Yang, T\. Zhao, F\. Zhou, X\. Song, G\. Dai, S\. Yan, H\. Yang, and Y\. Wang \(2025\)PM\-kvq: progressive mixed\-precision kv cache quantization for long\-cot llms\.External Links:2505\.18610,[Link](https://arxiv.org/abs/2505.18610)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1),[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, B\. Oguz, C\. Zhao, E\. Chang, P\. Stock, Y\. Mehdad, Y\. Shi, R\. Krishnamoorthi, and V\. Chandra \(2023\)LLM\-qat: data\-free quantization aware training for large language models\.External Links:2305\.17888,[Link](https://arxiv.org/abs/2305.17888)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, J\. Yuan, H\. Jin, S\. Zhong, Z\. Xu, V\. Braverman, B\. Chen, and X\. Hu \(2024\)KIVI: a tuning\-free asymmetric 2bit quantization for kv cache\.External Links:2402\.02750,[Link](https://arxiv.org/abs/2402.02750)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1),[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2607.16248#S3.SS1.p1.1),[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px4.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.External Links:1609\.07843,[Link](https://arxiv.org/abs/1609.07843)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- OpenAI \(2024\)Introducing openai o1\.Note:https://openai\.com/o1/Accessed: 2026\-06\-25Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1)\.
- J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, and T\. P\. Lillicrap \(2019\)Compressive transformers for long\-range sequence modelling\.External Links:1911\.05507,[Link](https://arxiv.org/abs/1911.05507)Cited by:[§4\.1](https://arxiv.org/html/2607.16248#S4.SS1.SSS0.Px2.p1.1)\.
- Y\. Shao, Y\. Chen, P\. Wang, J\. Yu, J\. Lin, Y\. Yao, Z\. Wei, and J\. Cheng \(2026\)DartQuant: efficient rotational distribution calibration for LLM quantization\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=LfcfwlLCHM)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Sheng, L\. Zheng, B\. Yuan, Z\. Li, M\. Ryabinin, D\. Y\. Fu, Z\. Xie, B\. Chen, C\. Barrett, J\. E\. Gonzalez, P\. Liang, C\. Ré, I\. Stoica, and C\. Zhang \(2023\)FlexGen: high\-throughput generative inference of large language models with a single gpu\.External Links:2303\.06865,[Link](https://arxiv.org/abs/2303.06865)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Sun, L\. Chang, W\. Bao, S\. Zheng, N\. Zheng, X\. Liu, H\. Dong, Y\. Chi, and B\. Chen \(2025\)ShadowKV: KV cache in shadows for high\-throughput long\-context LLM inference\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 57355–57373\.External Links:[Link](https://proceedings.mlr.press/v267/sun25b.html)Cited by:[§3\.1](https://arxiv.org/html/2607.16248#S3.SS1.p1.1)\.
- Q\. Tao, W\. Yu, and J\. Zhou \(2024\)AsymKV: enabling 1\-bit quantization of kv cache with layer\-wise asymmetric quantization configurations\.External Links:2410\.13212,[Link](https://arxiv.org/abs/2410.13212)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p3.1)\.
- R\. Tiwari, H\. Xi, A\. Tomar, C\. Hooper, S\. Kim, M\. Horton, M\. Najibi, M\. W\. Mahoney, K\. Keutzer, and A\. Gholami \(2025\)QuantSpec: self\-speculative decoding with hierarchical quantized kv cache\.External Links:2502\.10424,[Link](https://arxiv.org/abs/2502.10424)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.External Links:2307\.09288,[Link](https://arxiv.org/abs/2307.09288)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p3.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2024\)Efficient streaming language models with attention sinks\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21875–21895\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5e5fd18f863cbe6d8ae392a93fd271c9-Paper-Conference.pdf)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Xu, Z\. Jie, H\. Dong, L\. Wang, X\. Lu, A\. Zhou, A\. Saha, C\. Xiong, and D\. Sahoo \(2025\)ThinK: thinner key cache by query\-driven pruning\.External Links:2407\.21018,[Link](https://arxiv.org/abs/2407.21018)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng,et al\.\(2025a\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px3.p1.1)\.
- S\. Yang, J\. Guo, H\. Tang, Q\. Hu, G\. Xiao, J\. Tang, Y\. Lin, Z\. Liu, Y\. Lu, and S\. Han \(2025b\)LServe: efficient long\-sequence llm serving with unified sparse attention\.External Links:2502\.14866,[Link](https://arxiv.org/abs/2502.14866)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1),[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Zandieh, M\. Daliri, M\. Hadian, and V\. Mirrokni \(2025\)TurboQuant: online vector quantization with near\-optimal distortion rate\.External Links:2504\.19874,[Link](https://arxiv.org/abs/2504.19874)Cited by:[§1](https://arxiv.org/html/2607.16248#S1.p1.1),[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2607.16248#S5.SS0.SSS0.Px4.p1.1)\.
- T\. Zhang and A\. Shrivastava \(2025\)LeanQuant: accurate and scalable large language model quantization with loss\-error\-aware grid\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ISqx8giekS)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.External Links:2306\.14048,[Link](https://arxiv.org/abs/2306.14048)Cited by:[§2](https://arxiv.org/html/2607.16248#S2.SS0.SSS0.Px1.p1.1)\.