Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Caches
Summary
A new semantic-adaptive eviction policy for LLM prefix caches that learns token reuse patterns across different token types, achieving 1.4x-2.7x TTFT improvement over existing policies.
Similar Articles
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
This paper introduces a learned global retention-based KV cache eviction method that improves long-context reasoning by selectively retaining useful tokens and reducing attention dilution, while significantly lowering memory usage.
TokenPilot: Cache-Efficient Context Management for LLM Agents
TokenPilot is a dual-granularity context management framework that reduces inference costs in long-horizon LLM sessions by stabilizing prompt prefixes and conservatively managing context segments, achieving 61-87% cost reduction on benchmarks while maintaining competitive performance.
Sparse Prefix Caching for Hybrid and Recurrent LLM Serving
This paper introduces sparse prefix caching for hybrid and recurrent LLMs, which stores recurrent states at a limited set of checkpoint positions to avoid dense caching while minimizing recomputation. The method outperforms standard heuristics on real-world data, especially when requests share substantial but non-identical prefixes.
LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
This paper introduces LKV, a method for end-to-end learning of head-wise budgets and token selection to optimize KV cache eviction in large language models, achieving state-of-the-art performance with high compression rates.
Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
This empirical study examines practical online KV cache compaction for LLM agents, comparing token eviction and attention matching methods under different proxy query sources. It finds that delaying compaction to use future agent queries recovers performance, and token eviction preserves accuracy while reducing KV cache by 80%.