dOPSD: On-Policy Self-Distillation for Diffusion Language Models
Summary
This paper introduces dOPSD, an on-policy self-distillation method for diffusion language models that leverages internal denoising trajectories to improve mathematical reasoning and code generation.
View Cached Full Text
Cached at: 07/07/26, 06:42 AM
Paper page - dOPSD: On-Policy Self-Distillation for Diffusion Language Models
Source: https://huggingface.co/papers/2607.04428
Abstract
Diffusion large language models face challenges in reasoning enhancement through post-training, but a novel on-policy self-distillation method using internal denoising trajectories improves mathematical reasoning and code generation performance.
Diffusion large language models(dLLMs) generate text by iteratively denoising a masked sequence, offering a parallel alternative toautoregressive models, but eliciting strong reasoning through post-training remains difficult:supervised fine-tuningis off-policy and suffers fromexposure bias, whilereinforcement learninggives only sparse, sequence-level rewards and is hard to apply without tractable sequence likelihoods.On-policy self-distillation(OPSD) offers a promising alternative, using one model as both student and teacher to provide dense, token-level, on-policy supervision, but its effectiveness hinges on giving the teacherprivileged information(PI) - typically an instance-specific ground-truth reference unavailable at inference - so the student ends up distilling a weak PI-free consensus policy that yields little improvement on dLLM reasoning. We introduce dOPSD, which instead derives the teacher’s privilege directly from the student’s owndenoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher’s advantage emerges from the model’s own decoding process; on Dream and LLaDA, dOPSD improves bothin-domain math reasoningandout-of-domain code generation, outperforming supervised and on-policy baselines.
View arXiv pageView PDFGitHub2Add to collection
Get this paper in your agent:
hf papers read 2607\.04428
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.04428 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.04428 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.04428 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
D-OPSD: On-Policy Self-Distillation for Continuously Tuning Step-Distilled Diffusion Models
This paper introduces D-OPSD, a novel training paradigm for step-distilled diffusion models that enables on-policy self-distillation during supervised fine-tuning. It allows models to learn new concepts or styles without compromising their efficient few-step inference capabilities.
Learning from the Self-future: On-policy Self-distillation for dLLMs
Introduces d-OPSD, the first on-policy self-distillation framework for diffusion large language models, using suffix conditioning and step-level supervision to outperform RLVR and SFT baselines on reasoning benchmarks.
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
The paper introduces OPDLM, a method that transforms autoregressive language models into diffusion language models via on-policy distillation, requiring 15x to 7000x fewer training tokens while retaining knowledge from the original model.
GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
GDSD proposes a reinforcement learning method that directly distills denoisers from advantage-guided self-teachers for diffusion language models, avoiding biases from ELBO-based likelihood surrogates. It achieves up to +19.6% accuracy improvements on planning, math, and coding benchmarks over prior state-of-the-art methods.
DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models
DiffusionOPD proposes a multi-task training paradigm for diffusion models that uses online policy distillation to efficiently combine task-specific teachers into a unified student, achieving state-of-the-art results on all evaluated benchmarks.