I enjoyed the daily HF papers today
Summary
A tweet sharing three interesting papers from Hugging Face Daily Paper on advancements in KV cache compression for local LLMs and agent harness optimization.
Similar Articles
@songhan_mit: Explore our continued efforts on KV cache compression:
A tweet from Song Han highlights continued work on KV cache compression, featuring a blog by Weian Mao that discusses system-level aspects often overlooked in papers.
@yukangchen_: We are excited to share a new technical article “KV Cache Compression and Its Infra Problems.” https://research.nvidia.…
NVIDIA Research publishes a technical blog post examining KV cache compression techniques and their infrastructure problems, including how FlashAttention and paged attention create practical obstacles for production deployment of long-context LLMs, with a proposed geometric solution using RoPE.
@akshay_pachaar: https://x.com/akshay_pachaar/status/2074502882812952666
A practitioner's guide to KV cache management, introducing the open-source LMCache architecture that cuts input token costs by 90% and speeds up LLM inference by up to 14x by eliminating redundant context processing in agentic workflows.
Practical Online KV Cache Compaction for LLM Agents: An Empirical Study
This empirical study examines practical online KV cache compaction for LLM agents, comparing token eviction and attention matching methods under different proxy query sources. It finds that delaying compaction to use future agent queries recovers performance, and token eviction preserves accuracy while reducing KV cache by 80%.
ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
ReCache is a framework for efficient KV cache reuse and compression in tool-augmented LLM agents, achieving significant speedup and memory reduction while maintaining performance.