Tag
This paper introduces block-hybrid attention, which combines exact softmax attention within active denoising blocks and linear attention over previous blocks, to accelerate inference in pretrained diffusion language models. The authors retrofit this hybrid attention into LLaDA 2.1, achieving up to 1.7x higher decoding throughput with minimal post-training.
This paper analyzes the embedding space of Masked Diffusion Language Models (MDLMs) and finds it is hyperspherical, making linear interpolation suboptimal. The authors introduce Spherical Soft-Masking (S-SM), which uses SLERP and Fréchet means on the hypersphere, improving MAUVE and perplexity over prior soft-masking approaches.
This paper introduces OPTD, an on-policy transition distillation method with consistency-guided adaptive compression for few-step diffusion language models, improving quality-efficiency trade-offs across four reasoning and code-generation benchmarks.
This paper proposes speculative correction, a training-free draft-then-refine decoding strategy for diffusion language models, showing quality-latency improvements using LLaDA2.1 models.
A research paper presenting scaling laws and design principles for mixture-of-experts diffusion language models, trained as LLaDA MoE v2 (30B-A3B) on 23.5T tokens, approaching Qwen3 on several benchmarks with fewer pretraining tokens.
This paper investigates how training-free acceleration can silently change generated content in diffusion-based multimodal large language models, and proposes paired diagnostics and consistency-control methods to mitigate content drift.
This paper introduces LATCH, a training-free candidate-aware early-exit framework for diffusion language models that separates when to stop from where to accelerate, achieving 9.3-17.8x speedups on short-answer tasks and 2-3.3x on long-reasoning tasks with minimal accuracy loss on LLaDA and Dream.
A paper proposing Trace-Based On-Policy Distillation (TOPD), a teacher-supervised framework for transferring reasoning abilities to masked diffusion language models without reward estimation, achieving comparable accuracy to RL-trained counterparts with significant compute speedup.
This paper proposes JUMP, a single-pass membership inference attack for fine-tuned discrete diffusion language models that exploits their any-order and parallel decodability to improve detection accuracy with fewer queries.
This paper presents a mechanistic analysis of induction in masked diffusion language models, identifying a bidirectional induction circuit and showing that these models use the global fraction of masked tokens as an implicit timestep.
Proposes AdaLook, an adaptive multi-step lookahead decoding framework for masked diffusion language models that dynamically determines rollout depth and branch expansion based on candidate-score variance, achieving better accuracy-decoding steps trade-off compared to existing one-step lookahead decoding methods.
A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.
TACG is a training-free decoder for diffusion language models that uses trajectory-aware signals to decide when to commit tokens, improving accuracy and efficiency on code and math benchmarks.
This paper introduces dOPSD, an on-policy self-distillation method for diffusion language models that leverages internal denoising trajectories to improve mathematical reasoning and code generation.
This paper demonstrates that diffusion language models (DLMs) internally represent denoising progress as a latent timestep signal in their residual streams, which can be probed and steered to modulate model confidence and entropy.
This paper reveals that the low generative perplexity (Gen-PPL) reported by continuous diffusion language models like ELF is misleading, as it rewards repetition; the authors identify a one-dimensional attractor in the self-conditioning loop as the cause and propose ACE, a simple fix that subtracts this direction to reduce repetition without sacrificing quality.
Introduces Multi-Token Residual Prediction (MRP), a technique that accelerates diffusion language model inference by predicting residuals between adjacent denoising steps, achieving up to 1.56x speedup in SGLang and recovering up to +16 accuracy points in aggressive decoding settings.
This paper adapts a mixture-of-experts diffusion language model, DiffusionGemma-26B, for interactive radiology report drafting, showing it matches or exceeds autoregressive models in medical VQA with 3.5-4.4x faster decoding and bidirectional infill capabilities.
iLLaDA is an 8B parameter masked diffusion language model with fully bidirectional attention, trained from scratch on 12T tokens. It shows broad improvements over LLaDA and remains competitive with Qwen2.5 7B on several benchmarks. The model and code are open-sourced.
This paper investigates the effectiveness of top-1 collapse rate as a stability monitor for short-horizon LoRA fine-tuning of discrete diffusion language models, finding it has zero precision, and proposes max gradient norm as a more reliable alternative with higher precision and F1 score on LLaDA-family models.