Tag
This paper proposes ReCo, a reward-coordinated compression framework that uses a process-reward estimator to adaptively compress KV-cache, control reflection tokens, and enable early stopping, cutting generated tokens by 37–65% and latency by ~2x across reasoning models while preserving accuracy.
The author describes implementing TurboQuant KV-cache compression into Apple's mlx-swift-lm, achieving 2.7x compression with quality on par with 8-bit, and 3-4x decode speed improvements via a fused Metal kernel.
REAL introduces an Attention Behavior Matrix to analyze attention head behaviors in both success and failure cases, enabling more effective KV cache eviction. It achieves comparable accuracy to the strongest baseline while requiring 32x less space.
This paper unifies memory compaction techniques across LLMs and agents under a rate-distortion framework, proposing a taxonomy and benchmark for evaluating compression across different layers.
DepthWeave-KV is a token-adaptive cross-layer residual factorization method for compressing KV cache in long-context transformer inference, achieving 8.3x memory reduction and 72.8 tokens/s at 64K context while preserving near-full-cache task quality across benchmarks.
Introduces FreqDepthKV, a frequency-guided depth sharing method for KV cache compression in long-context LLM inference, which factorizes adjacent-layer KV states into shared low-frequency components and sparse high-frequency residuals, improving memory efficiency and throughput while preserving accuracy on benchmarks.
This paper proposes Kara, a sliding-window KV cache compression method for efficient serving of reasoning LLMs, addressing limitations in existing compression techniques by using bidirectional attention and a Token2Chunk module. The method is integrated into the KvLLM inference framework built upon vLLM, improving output throughput while maintaining performance.
CompressKV proposes a semantic-retrieval-guided KV-cache compression method for GQA-based LLMs, identifying Semantic Retrieval Heads to retain critical tokens. It achieves over 97% full-cache performance using only 3% of the KV cache on LongBench tasks.
PolyKV is a layer-wise KV cache compression framework that assigns heterogeneous eviction policies and non-uniform budgets per layer, significantly improving over uniform baselines on LongBench with LLaMA-3.1-8B and Qwen3-8B.
The paper introduces Tangram, a serving framework that statically resolves non-uniform KV cache compression for multi-turn LLM serving, achieving up to 2.6x throughput improvement over the full-KV baseline by eliminating runtime overheads.
Benchmarks of DFlash speculative decoding combined with KV cache compression on RTX 5090 show up to 3.26x speedup on Qwen3.6-27B with minimal perplexity degradation, with q4_0/turbo4 providing the best balance.
BeeLlama.cpp is a fork of llama.cpp that integrates DFlash speculative decoding, TurboQuant/TCQ KV-cache compression, and adaptive draft control, achieving up to 3x faster inference and 7.5x context expansion on the same hardware.
This paper proposes Shadow Mask Distillation (SMD) to solve the off-policy bias caused by KV cache compression during reinforcement learning post-training for large language models. It introduces a mechanism that ensures on-policy alignment and improves memory efficiency for long-context reasoning tasks.
This paper introduces Forcing-KV, a hybrid KV cache compression strategy for autoregressive video diffusion models that separates attention heads into static and dynamic categories, achieving up to 2.82x speedup at 1080P resolution while maintaining output quality.
OjaKV introduces a context-aware online low-rank KV cache compression framework that uses hybrid storage and Oja's algorithm for incremental subspace adaptation to reduce GPU memory bottlenecks in long-context LLM inference without model fine-tuning.