@JunchenJiang: Will this put an end to the debate that compressing KV caches might screw up LLM inference?
Summary
This tweet questions whether a new finding will resolve the debate on whether compressing KV caches harms LLM inference performance.
View Cached Full Text
Cached at: 07/05/26, 12:55 AM
🤔Will this put an end to the debate that compressing KV caches might screw up LLM inference?
Similar Articles
CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
CompressKV proposes a semantic-retrieval-guided KV-cache compression method for GQA-based LLMs, identifying Semantic Retrieval Heads to retain critical tokens. It achieves over 97% full-cache performance using only 3% of the KV cache on LongBench tasks.
@oneill_c: 1/ Can you actually get new facts into an LLM's weights without breaking the model? This question decides how we approa…
This thread presents research on whether new facts can be added to an LLM's weights without breaking the model, and finds that it breaks unexpectedly, making compressed KV caches and in-context learning more promising for continual learning.
KV Cache Is Becoming the Memory Hierarchy of Inference
The article discusses how the KV cache is evolving into a memory hierarchy for LLM inference, optimizing memory management during decoding.
@JunchenJiang: I was delighted to give three keynotes recently, one at the 3rd HotInfra workshop at ISCA'26 (hosted by Prof. Jian Huan…
Junchen Jiang delivered three keynotes arguing that KV cache is an underappreciated asset for LLM inference, enabling cost savings, latency reduction, and quality improvements, and should be treated as a core data layer in future inference infrastructure.
@yukangchen_: We are excited to share a new technical article “KV Cache Compression and Its Infra Problems.” https://research.nvidia.…
NVIDIA Research publishes a technical blog post examining KV cache compression techniques and their infrastructure problems, including how FlashAttention and paged attention create practical obstacles for production deployment of long-context LLMs, with a proposed geometric solution using RoPE.