diffusion-language-models

Tag

Cards List
#diffusion-language-models

Retrofitting Linear Attention into Diffusion Language Models

arXiv cs.LG ↗ · 2026-08-10 Cached

This paper introduces block-hybrid attention, which combines exact softmax attention within active denoising blocks and linear attention over previous blocks, to accelerate inference in pretrained diffusion language models. The authors retrofit this hybrid attention into LLaDA 2.1, achieving up to 1.7x higher decoding throughput with minimal post-training.

0 favorites 0 likes
#diffusion-language-models

Lost in Interpolation: Why Predictive Feedback Fails in Diffusion Language Models

arXiv cs.CL ↗ · 2026-08-10 Cached

This paper analyzes the embedding space of Masked Diffusion Language Models (MDLMs) and finds it is hyperspherical, making linear interpolation suboptimal. The authors introduce Spherical Soft-Masking (S-SM), which uses SLERP and Fréchet means on the hypersphere, improving MAUVE and perplexity over prior soft-masking approaches.

0 favorites 0 likes
#diffusion-language-models

OPTD: On-Policy Transition Distillation with Consistency-Guided Adaptive Compression for Few-Step Diffusion Language Models

arXiv cs.CL ↗ · 2026-08-05 Cached

This paper introduces OPTD, an on-policy transition distillation method with consistency-guided adaptive compression for few-step diffusion language models, improving quality-efficiency trade-offs across four reasoning and code-generation benchmarks.

0 favorites 0 likes
#diffusion-language-models

Speculative Correction: Draft-then-Refine Decoding for Diffusion Language Models

arXiv cs.CL ↗ · 2026-08-05 Cached

This paper proposes speculative correction, a training-free draft-then-refine decoding strategy for diffusion language models, showing quality-latency improvements using LLaDA2.1 models.

0 favorites 0 likes
#diffusion-language-models

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

Hugging Face Daily Papers ↗ · 2026-08-04 Cached

A research paper presenting scaling laws and design principles for mixture-of-experts diffusion language models, trained as LLaDA MoE v2 (30B-A3B) on 23.5T tokens, approaching Qwen3 on several benchmarks with fewer pretraining tokens.

0 favorites 0 likes
#diffusion-language-models

Faster but Different: Diagnosing and Controlling Content Drift in Accelerated Multimodal Diffusion Language Models

arXiv cs.CL ↗ · 2026-08-03 Cached

This paper investigates how training-free acceleration can silently change generated content in diffusion-based multimodal large language models, and proposes paired diagnostics and consistency-control methods to mitigate content drift.

0 favorites 0 likes
#diffusion-language-models

Where and When to Commit: Candidate-Aware Decoding for Diffusion Language Models

arXiv cs.CL ↗ · 2026-07-31 Cached

This paper introduces LATCH, a training-free candidate-aware early-exit framework for diffusion language models that separates when to stop from where to accelerate, achieving 9.3-17.8x speedups on short-answer tasks and 2-3.3x on long-reasoning tasks with minimal accuracy loss on LLaDA and Dream.

0 favorites 0 likes
#diffusion-language-models

Trace-Based On-Policy Distillation for Masked Diffusion Language Models

arXiv cs.CL ↗ · 2026-07-21 Cached

A paper proposing Trace-Based On-Policy Distillation (TOPD), a teacher-supervised framework for transferring reasoning abilities to masked diffusion language models without reward estimation, achieving comparable accuracy to RL-trained counterparts with significant compute speedup.

0 favorites 0 likes
#diffusion-language-models

JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models

arXiv cs.AI ↗ · 2026-07-21 Cached

This paper proposes JUMP, a single-pass membership inference attack for fine-tuned discrete diffusion language models that exploits their any-order and parallel decodability to improve detection accuracy with fewer queries.

0 favorites 0 likes
#diffusion-language-models

Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

arXiv cs.CL ↗ · 2026-07-20 Cached

This paper presents a mechanistic analysis of induction in masked diffusion language models, identifying a bidirectional induction circuit and showing that these models use the global fraction of masked tokens as an implicit timestep.

0 favorites 0 likes
#diffusion-language-models

Adaptive Multi-Step Lookahead Decoding for Diffusion Language Models

arXiv cs.CL ↗ · 2026-07-20 Cached

Proposes AdaLook, an adaptive multi-step lookahead decoding framework for masked diffusion language models that dynamically determines rollout depth and branch expansion based on candidate-score variance, achieving better accuracy-decoding steps trade-off compared to existing one-step lookahead decoding methods.

0 favorites 0 likes
#diffusion-language-models

@volokuleshov: New blog post: How to Build a Diffusion Language Model. Diffusion LLMs went from open problem to reality in 2 years (Me…

X AI KOLs Timeline ↗ · 2026-07-08 Cached

A comprehensive blog post by Volodymyr Kuleshov's Cornell group explains how to build diffusion language models, covering core techniques like masked diffusion, iterative refinement, variable-length generation, controllable generation, fast samplers, and RL post-training, using open-source models such as Mercury, Gemma Diffusion, and Nemotron Diffusion as examples.

0 favorites 0 likes
#diffusion-language-models

TACG: Trajectory-Aware Commit Gating for Diffusion Language Model Decoding

arXiv cs.CL ↗ · 2026-07-07 Cached

TACG is a training-free decoder for diffusion language models that uses trajectory-aware signals to decide when to commit tokens, improving accuracy and efficiency on code and math benchmarks.

0 favorites 0 likes
#diffusion-language-models

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

Hugging Face Daily Papers ↗ · 2026-07-05 Cached

This paper introduces dOPSD, an on-policy self-distillation method for diffusion language models that leverages internal denoising trajectories to improve mathematical reasoning and code generation.

0 favorites 0 likes
#diffusion-language-models

Subliminal Clocks: Latent Time Modelling in Diffusion Language Models

arXiv cs.AI ↗ · 2026-07-03 Cached

This paper demonstrates that diffusion language models (DLMs) internally represent denoising progress as a latent timestep signal in their residual streams, which can be probed and steered to modulate model confidence and entropy.

0 favorites 0 likes
#diffusion-language-models

Low Perplexity is Repetition: A One-Dimensional Self-Conditioning Attractor in Continuous Diffusion LMs

arXiv cs.CL ↗ · 2026-07-02 Cached

This paper reveals that the low generative perplexity (Gen-PPL) reported by continuous diffusion language models like ELF is misleading, as it rewards repetition; the authors identify a one-dimensional attractor in the self-conditioning loop as the cause and propose ACE, a simple fix that subtracts this direction to reduce repetition without sacrificing quality.

0 favorites 0 likes
#diffusion-language-models

@_yucheng_lu: MTP makes autoregressive LLMs fast. Can the same trick work for diffusion LMs? Had a fun collaboration with @modal expl…

X AI KOLs Following ↗ · 2026-07-02 Cached

Introduces Multi-Token Residual Prediction (MRP), a technique that accelerates diffusion language model inference by predicting residuals between adjacent denoising steps, achieving up to 1.56x speedup in SGLang and recovering up to +16 accuracy points in aggressive decoding settings.

0 favorites 0 likes
#diffusion-language-models

Discrete Diffusion Language Models for Interactive Radiology Report Drafting

Hugging Face Daily Papers ↗ · 2026-07-01 Cached

This paper adapts a mixture-of-experts diffusion language model, DiffusionGemma-26B, for interactive radiology report drafting, showing it matches or exceeds autoregressive models in medical VQA with 3.5-4.4x faster decoding and bidirectional infill capabilities.

0 favorites 0 likes
#diffusion-language-models

Improved Large Language Diffusion Models

arXiv cs.CL ↗ · 2026-06-25 Cached

iLLaDA is an 8B parameter masked diffusion language model with fully bidirectional attention, trained from scratch on 12T tokens. It shows broad improvements over LLaDA and remains competitive with Qwen2.5 7B on several benchmarks. The model and code are open-sourced.

0 favorites 0 likes
#diffusion-language-models

When Top-1 Fails: Calibrating LoRA Monitors for Masked Diffusion LMs

arXiv cs.LG ↗ · 2026-06-24 Cached

This paper investigates the effectiveness of top-1 collapse rate as a stability monitor for short-horizon LoRA fine-tuning of discrete diffusion language models, finding it has zero precision, and proposes max gradient norm as a more reliable alternative with higher precision and F1 score on LLaDA-family models.

0 favorites 0 likes
← Previous
Next →
← Back to home

Submit Feedback