Tag
NeoMME presents a family of efficient multimodal encoders pretrained with masked discrete diffusion, achieving strong performance in visual document retrieval and high embedding compression while outperforming smaller models on benchmarks.
The paper introduces DCGC, a Masked Diffusion Model framework for globally correcting flawed reasoning traces in LLMs by conditioning on an imperfect draft. It improves accuracy on reasoning benchmarks without ground-truth failure labels.
This paper characterizes the serving behavior of masked diffusion language models (dLLMs) using real hardware measurements, identifying key differences from autoregressive models and deriving design principles for efficient inference systems.
This paper introduces Entropy-Valley, a training-free length selector for masked diffusion machine translation that uses predictive entropy to improve adequacy, showing that length choice matters more than unmasking order.
This paper investigates when classifier-free guidance (CFG) is actually necessary in masked diffusion language models, showing that guidance dependence is prompt-specific and can often be removed without losing constraint satisfaction, leading to a defined 'commitment horizon'.
The paper introduces Stoicheia, a 405M-parameter character-level masked diffusion encoder for Ancient Greek that unifies textual restoration, parsing, and metrical scansion in a single model, outperforming prior systems like Ithaca on benchmark tasks.
PreDiff-LM proposes a hybrid attention mechanism that preserves causal attention for prompt tokens and bidirectional attention for masked target tokens, enabling adaptation of pretrained autoregressive models for discrete masked diffusion language modeling, achieving improvements in perplexity and downstream tasks over prior diffusion baselines.
This paper introduces CaRE, a compute-aware evaluation protocol for masked diffusion language models that standardizes step counts, metrics, and stochasticity. It demonstrates that previous comparisons conflate algorithmic improvements with evaluation artifacts, showing temperature explains most MAUVE variance and compute-matched comparisons reverse published rankings.
MotifRole-Diff proposes a role-aware corruption schedule for masked discrete diffusion on molecular graphs, allocating masking rates based on denoising difficulty and graph-level perturbation impact, demonstrating improved validity and reduced FCD on QM9 and MOSES benchmarks.
A paper proposing Trace-Based On-Policy Distillation (TOPD), a teacher-supervised framework for transferring reasoning abilities to masked diffusion language models without reward estimation, achieving comparable accuracy to RL-trained counterparts with significant compute speedup.
This paper introduces a learnable control module trained via Group Relative Policy Optimization (GRPO) to optimize the generation order in multimodal masked diffusion models, achieving improvements in text-to-image alignment and multimodal understanding.
A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.
This paper reinterprets masked diffusion language model decoding as continuous clean-state prediction, introducing a flow-based framework where tokens are updated continuously and asynchronously based on confidence, achieving 97% of LLaDA's performance with 25% of the decoding budget.
This paper introduces Masked Language Flow Models (MLFMs), which incorporate masking into flow-based language models to enable continuous flow for conditional generation and allow pretrained Masked Diffusion Models to be converted. The authors propose a novel sampler that alternates continuous denoising with discrete unmasking, demonstrating for the first time that flow-based language models can scale to downstream reasoning and instruction-following tasks.
iLLaDA is an 8B parameter masked diffusion language model with fully bidirectional attention, trained from scratch on 12T tokens. It shows broad improvements over LLaDA and remains competitive with Qwen2.5 7B on several benchmarks. The model and code are open-sourced.
VoidPadding introduces a [VOID] token to handle padding in masked diffusion language models, allowing [EOS] to focus solely on semantic termination. This method significantly improves performance on reasoning and coding benchmarks while reducing decoding steps.
This paper proposes TIE, a knowledge fusion framework for masked diffusion language models that tracks confidence dynamics to identify reliable decoding trajectories and iteratively transfers partially denoised sequences between models, improving generation quality on reasoning tasks.
This paper introduces ADAS, a training-free reranking rule for parallel masked diffusion decoding that uses attention to discount tokens that strongly attend to uncertain positions, improving low-NFE performance on reasoning and code tasks with minimal runtime overhead.
The author built Joey, a 170M parameter masked diffusion language model from scratch, trained on FineWeb-Edu and fine-tuned on DailyDialog, achieving fluent but incoherent sentences due to capacity limitations. The project highlights the differences from autoregressive LLMs and the lessons learned from building and debugging the system.
Proposes learning the unmasking order in masked diffusion models using a lightweight policy network, with a weighted loss that outperforms heuristics on combinatorial tasks and protein design.