@alec_helbling: Diffusion LMs generate multiple tokens in parallel. However, iterative unmasking repeatedly updates token states, limit…
Summary
The post describes how diffusion language models face KV-cache reuse limitations due to iterative unmasking, and introduces Block Diffusion as a solution that decodes blocks left-to-right for efficient caching while generating in parallel.
View Cached Full Text
Cached at: 09/11/26, 02:41 PM
Diffusion LMs generate multiple tokens in parallel.
However, iterative unmasking repeatedly updates token states, limiting KV-cache reuse.
Block Diffusion restores efficient caching by decoding blocks left-to-right while generating each block in parallel. https://t.co/JXODydX5ng
Similar Articles
Multi-Block Diffusion Language Models
This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.
BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion
This paper introduces BitLM, a language model that uses bitwise continuous diffusion to generate multiple tokens in parallel, aiming to overcome the sequential bottleneck of traditional autoregressive generation while preserving causal structure.
@volokuleshov: New blog post: How to Build a Diffusion Language Model. Diffusion LLMs went from open problem to reality in 2 years (Me…
A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.
ALoDLM: Adaptively Looped Diffusion Language Models
ALoDLM introduces token-adaptive latent recurrence for diffusion language models, allocating extra computation to difficult tokens and using a conditional negative ELBO to jointly learn prediction and compute allocation, outperforming other DLMs and AR baselines at 1.7B and 8B scales across eleven benchmarks.
@nathanrs: New post! A drawback of diffusion LLMs is that bidirectional attention causes keys and values to drift across steps, br…
A new post highlights a drawback of diffusion LLMs: bidirectional attention causes keys and values to drift across steps, breaking KV caching. However, generation quality is robust to slight KV drift, and research has focused on maximizing stale KV reuse without quality degradation.