kv-caching

FastGuide 是一种自适应的并行与自回归混合解码方法，通过复用奖励模型反向传播计算、利用 KV 缓存和稀疏注意力重计算来加速扩散大语言模型的梯度奖励引导，在三个奖励基准上实现最高 4.4× 加速同时保持相近生成质量。

The article discusses how optimizing context structure for KV caching can significantly reduce the cost of running AI agents, based on insights from an OpenAI podcast during migration to GPT 5.6.

Flash-dLLM is a training-free inference acceleration framework for diffusion LLMs that uses IO-aware KV caching and parallel decoding to achieve significant speedups and memory efficiency improvements.

TierKV proposes a predictive multi-tier KV caching framework to optimize memory usage and throughput for long-context LLMs on mobile devices, achieving significant performance improvements with minimal accuracy degradation.

Introduces Archer, a training-free KV caching method for diffusion language models that adaptively reuses cached hidden states to reduce recomputation while preserving rollback capabilities, achieving up to 2.95x speedup and improved generation quality.

A tweet recommends an arXiv paper that explains the mathematical foundations of Transformers, covering tokenization, embeddings, multi-headed attention, and KV caching for applied mathematicians.

A detailed breakdown of why uploading a large PDF to ChatGPT doesn't significantly slow response times, covering techniques like streaming, KV caching, and parallel prefill phase.

Explains how KV caching speeds up LLM inference by eliminating redundant recomputation of attention keys and values, trading off speed for memory, and introduces production-scale cache management challenges.

A detailed thread explaining key concepts of LLM inference: attention, KV caching, chunked prefill, and batching techniques, including continuous batching used in vLLM and SGLang.

A new post highlights a drawback of diffusion LLMs: bidirectional attention causes keys and values to drift across steps, breaking KV caching. However, generation quality is robust to slight KV drift, and research has focused on maximizing stale KV reuse without quality degradation.

A developer shares an architectural pattern to manage context window bloat in continuous Anthropic agent loops, using KV caching, dynamic tool schema loading, and decoupling executor/advisor roles with Claude 3.5 Sonnet and Claude 3 Opus.

This paper proposes BiCache, a novel KV caching technique for shared prefixes in diffusion language models, which avoids accuracy collapse by dynamically reusing cached keys and values in shallow layers and achieves 36.3%–98.3% throughput improvement.

KV Packet proposes a recomputation-free cache reuse framework for LLMs that uses trainable soft-token adapters to bridge context discontinuities, eliminating overhead while maintaining performance comparable to full recomputation baselines on Llama-3.1 and Qwen2.5.

oMLX is a new open-source tool for optimized LLM inference on Apple Silicon Macs, featuring continuous batching and tiered KV caching managed via a menu bar app.