Tag
CacheScout is an agent-aware KV-cache runtime layer for multi-agent LLM serving that learns agent execution semantics online to guide cache eviction and prefetching, improving cache hit rate and reducing latency.
This paper proposes a KV cache eviction strategy that scores tokens by counter-causal surprise, removing past tokens that are well-predicted from future context. The method is training-free, in-distribution, and achieves competitive performance with a fast single-layer approximation.
LMCache is an open-source KV cache management layer that separates cache I/O from compute, plugging into vLLM, SGLang, and TensorRT-LLM to achieve up to 14x faster time-to-first-token and 4x faster decoding by parallelizing cache lookups and sharing GPU memory.