@no_stp_on_snek: Always start with uncompressed k and compressed V and go more aggressively from there. Model families have different se…
Summary
A tip on KV-cache compression for transformer models: start with uncompressed keys and compressed values, then adjust based on model family sensitivity; try asymmetric before symmetric compression.
View Cached Full Text
Cached at: 05/23/26, 06:11 PM
Always start with uncompressed k and compressed V and go more aggressively from there.
Model families have different sensitivities to K compression in particular.
Asym first over sym.
Similar Articles
The risk of KV cache compression
This paper theoretically characterizes the minimax risk of KV cache compression in transformers, providing design principles for accurate compression under causal masking, and instantiates them in a practical algorithm with promising results on LongBench.
Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Caches
Codec-Gauge learns small orthogonal channel transforms (gauges) for Transformer KV caches to improve compression fidelity at fixed bit rates, achieving significant KL divergence reduction across multiple models and backends.
@VukRosic99: Most KV-cache compression applies SVD to the keys alone, or embeds queries and keys jointly. Both miss the obvious targ…
KQ-SVD is a new method for KV-cache compression that directly approximates the attention matrix via optimal low-rank decomposition, achieving 5-10x lower error than key-only SVD on LLaMA and Mistral models.
Ablation, Statistical Inference, and Validation for KV-Cache Compression
This paper presents a systematic comparative study of KV-cache compression schemes (TurboQuant and SpectralQuant), introduces a statistical validation methodology, and offers regime-specific recommendations for efficient transformer inference.
KV Cache Compression 900000x Beyond TurboQuant and Per-Vector Shannon Limit
A new paper proposes sequential KV cache compression using probabilistic language tries and predictive delta coding, achieving theoretical compression ratios of ~914,000× beyond TurboQuant by exploiting the sequential structure of language model tokens rather than treating vectors independently.