Anyone else tried out KV cache blending?
Summary
The author experimented with KV cache blending by splitting prompts into chunks with overlap, achieving a 3x boost in prefill speed without affecting retrieval tasks on Ling3-tiny.
Similar Articles
@akshay_pachaar: 90% of your KV cache never gets reused. (prompt caching was never meant to fix it) if your system prompt and tool defin…
CacheBlend, a EuroSys 2025 Best Paper, solves the problem that 90% of KV cache is never reused due to rigid prefix-matching in prompt caching. By selectively recomputing only boundary tokens between documents, it achieves 2-4x faster multi-document processing without quality loss, implemented in the open-source LMCache layer.
Enabling KV Caching of Shared Prefix for Diffusion Language Models
This paper proposes BiCache, a novel KV caching technique for shared prefixes in diffusion language models, which avoids accuracy collapse by dynamically reusing cached keys and values in shallow layers and achieves 36.3%–98.3% throughput improvement.
KVBoost: Chunk-Level Key-Value Cache Reuse with Deviation-Guided Recomputation for Efficient Large Language Model Inference
KVBoost is a chunk-level key-value cache reuse system for efficient large language model inference that achieves high cache hit rates and significant speedup in time-to-first-token without quality loss, using dual-hash keying and deviation-guided recomputation.
OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching
OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM by prefetching sparse, important KV blocks using lookahead tokens from speculative decoding, achieving up to 2.1x throughput gains over dense vLLM with minimal accuracy loss.
SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation
SelKV is a training-free framework for KV cache compression that uses a soft cosine gate for selective merging and an attention-ratio compensation mechanism to correct softmax imbalance, achieving near-lossless generation at 25% cache size and 3.3x decoding speedup on LongBench.