@nathanrs: New post! A drawback of diffusion LLMs is that bidirectional attention causes keys and values to drift across steps, br…
Summary
A new post highlights a drawback of diffusion LLMs: bidirectional attention causes keys and values to drift across steps, breaking KV caching. However, generation quality is robust to slight KV drift, and research has focused on maximizing stale KV reuse without quality degradation.
View Cached Full Text
Cached at: 06/29/26, 02:22 AM
New post! A drawback of diffusion LLMs is that bidirectional attention causes keys and values to drift across steps, breaking KV caching. But generation quality is robust to slight KV drift, and much work has gone into maximizing stale KV reuse without causing quality drops (1/3)
Most common ways to add noise to text are masking (replace clean tokens with [MASK] tokens) and uniform noise (swap with random tokens instead). Both have many different properties and it was unclear whether caching strategies for the former would work on the latter (2/3)
It turns out it does, for two reasons that are noise process agnostic: decoding prefers left-to-right (confidence is highest next to already-resolved tokens) and state transitions’ impact on KV drift is local. Thus, masking caching strategies seem to generalize to others (3/3)
Read the full post here:
Similar Articles
@TheTuringPost: Why KV cache is one of the main reasons LLMs are fast? KV cache is what connects attention mechanism with generation st…
KV cache stores previously computed key and value vectors during autoregressive generation, allowing models to avoid recomputing the entire sequence at each step, significantly speeding up inference at the cost of increased memory usage.
The interesting BDH question: What if LLM memory lived in the network weights instead of the ever-growing KV cache?
This article analyzes Jan Chorowski's BDH architecture proposal, which explores embedding LLM memory directly into network weights using sparse high-dimensional key-query spaces as an alternative to traditional KV caches.
Enabling KV Caching of Shared Prefix for Diffusion Language Models
This paper proposes BiCache, a novel KV caching technique for shared prefixes in diffusion language models, which avoids accuracy collapse by dynamically reusing cached keys and values in shallow layers and achieves 36.3%–98.3% throughput improvement.
@pallavishekhar_: KV Cache in LLMs Read here: https://outcomeschool.com/blog/kv-cache-in-llms…
This article explains the concept of KV Cache in Large Language Models, detailing how it optimizes text generation by storing and reusing key-value pairs to avoid redundant computations during inference.
Memory
Explains why LLM inference is increasingly memory-bandwidth bound due to the KV cache scaling with context length and concurrent users, and how systems like vLLM and PagedAttention improve memory utilization.