I mapped the KLD of KV cache quantization for Qwen3.6-35B-A3B and Gemma4-E2B QAT
Summary
The author maps the Kullback-Leibler divergence of KV cache quantization for the Qwen3.6-35B-A3B and Gemma4-E2B QAT models.
Similar Articles
Gemma 4 QAT 31B responds better to KV cache quantization too
The Gemma 4 QAT 31B model demonstrates improved behavior with KV cache quantization, suggesting enhanced inference efficiency.
Here are my KV cache quantization benchmarks: TurboQuant is overrated but saved by TCQ, q5 deserves more attention, and symmetric q8 might be a waste of VRAM
A detailed benchmark comparing KV cache quantization methods (TurboQuant, TCQ, q4, q5, q8) using PPL and KLD metrics on Qwen 3.6 27B, finding that TCQ improves low-bit quantization, asymmetric KV beats symmetric at same size, and q8 is often overkill. Includes analysis and data in linked article.
Qwen3.8-27b q8 KV cache does seem to actually hurt model performance
The article discusses how on-the-fly KV cache quantization can reduce long-context model performance due to compounding errors, based on experiments with Qwen3.8-27B.
Statistical Inference and Quality Measures of KV Cache Quantisations Inspired by TurboQuant
This paper analyzes KV cache quantization schemes inspired by TurboQuant, using statistical inference and a new 6D error framework to evaluate quality measures like KL divergence and geometric error.
High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
This paper identifies that low-bit KV cache quantization degrades LLM accuracy due to structured local misranking of logits, and proposes DGAP, a method that restores the local distribution of top-K candidates, recovering RULER accuracy from 47.8% to 83.2% on Llama-3.1-8B with minimal overhead.