Multi-Block Diffusion Language Models
Summary
This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.
View Cached Full Text
Cached at: 07/01/26, 07:41 AM
Paper page - Multi-Block Diffusion Language Models
Source: https://huggingface.co/papers/2606.29215 Authors:
,
,
,
,
,
,
,
,
,
Abstract
Multi-Block Diffusion Language Models extend single-block diffusion to concurrent block decoding with improved training strategies and optimized decoding algorithms.
Block Diffusion Language Models(BD-LMs) improvediffusion-based text generationwith KV caching and flexible-length generation. A natural next step is to extend them from Single-Block Diffusion (SingleBD) toMulti-Block Diffusion(MultiBD), where a running-set of consecutive blocks is decoded concurrently forinter-block parallelism. However, existing BD-LMs are mostly trained underteacher forcing, where the model observes only one noisy block conditioned on a clean prefix. While the recentdiffusion forcingstrategy introduces visibility among multiple noisy blocks, its training states still differ from MultiBD inference, where decoding operates on a bounded running-set with heterogeneous slot-wise noise patterns. To bridge this gap, we proposeMulti-Block DiffusionLanguage Models (MBD-LMs), obtained by post-training BD-LMs withMulti-block Teacher Forcing(MultiTF). MultiTF integratesteacher forcinganddiffusion forcingby training on bounded noise-groups conditioned on clean prefixes, with randomizednoise-schedulersthat better match MultiBD inference states. To make MultiBD practically executable, we further introduce an optimized decoding algorithm based on theBlock Buffer mechanismthat preservesprefix-cache reuse, keeps input shapes static, and translates increased decoding parallelism intowall-clock acceleration. Empirically, MBD-LLaDA2-Mini increases averageTokens Per Forward pass(TPF) from 3.47 to 6.19 and improves average accuracy from 79.95% to 81.03%; when combined with DMax, MBD-LLaDA2-Mini-DMax reaches an average TPF of 9.34 with only a 1.02% accuracy drop on math and code benchmarks.
View arXiv pageView PDFProject pageGitHub15Add to collection
Get this paper in your agent:
hf papers read 2606\.29215
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper8
#### SJTU-DENG-Lab/MBD-Math-LLaDA2-mini-16B 16B• Updatedabout 2 hours ago • 19
#### SJTU-DENG-Lab/MBD-Math-LLaDA2-mini-DMax-16B 16B• Updatedabout 2 hours ago • 21
#### SJTU-DENG-Lab/MBD-Code-LLaDA2-mini-DMax-16B 16B• Updatedabout 2 hours ago • 10
#### SJTU-DENG-Lab/MBD-Code-LLaDA2-mini-16B 16B• Updatedabout 2 hours ago • 110
Browse 8 models citing this paper## Datasets citing this paper1
#### SJTU-DENG-Lab/MBD-LMs-MultiTF-Datasets Preview• Updatedabout 2 hours ago • 42
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.29215 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Training Hybrid Block Diffusion Language Models with Partial Bidirectionality
This paper proposes a hybrid Mamba-attention architecture for block diffusion language models that restricts reverse Mamba scans to the active denoising block, enabling exact caching across blocks and achieving high throughput for long-context generation.
Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling
The paper proposes 'Early-Bird Decoding,' a framework to accelerate diffusion large language models by using learnable block sizes and parallel sampling, achieving significant throughput improvements without modifying pretrained weights.
@alec_helbling: Diffusion LMs generate multiple tokens in parallel. However, iterative unmasking repeatedly updates token states, limit…
The post describes how diffusion language models face KV-cache reuse limitations due to iterative unmasking, and introduces Block Diffusion as a solution that decodes blocks left-to-right for efficient caching while generating in parallel.
ALoDLM: Adaptively Looped Diffusion Language Models
ALoDLM introduces token-adaptive latent recurrence for diffusion language models, allocating extra computation to difficult tokens and using a conditional negative ELBO to jointly learn prediction and compute allocation, outperforming other DLMs and AR baselines at 1.7B and 8B scales across eleven benchmarks.
Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising
Proposes neuromorphic masked diffusion language models (N-MDLMs) that integrate block diffusion with spike-based neuromorphic computation to improve throughput and energy efficiency by leveraging sparsity and generating multiple tokens per parameter access, analyzed via a roofline-inspired model.