@ethantsliu: some distillation techniques: OPD: log pi - log pi_theta OPSD: log pi_theta (• | x, y*) - log pi_theta (• | x) RLSD: A(…
Summary
This tweet discusses OPD, OPSD, RLSD, and OPD^2 distillation techniques for language models, emphasizing the use of log probability differences to isolate post-training improvements.
View Cached Full Text
Cached at: 07/20/26, 01:26 PM
some distillation techniques:
OPD: log pi - log pi_theta OPSD: log pi_theta (• | x, y*) - log pi_theta (• | x) RLSD: A(x, y) x opsd (advantage direction, opsd magnitude) OPD^2: log π_teacher − log π_teacher-base
Xiuyu Li (@sheriyuo): OPD normally copies everything the teacher prefers, including habits inherited before reasoning tuning.
OPD² instead uses the token-level logprob change between a reasoning teacher and its own base checkpoint, so the reward isolates what post-training actually added.
On-Policy
Similar Articles
OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification
OmniOPD introduces a logit-free on-policy distillation method that uses chunk-level semantic similarity and speculative verification to train student models with black-box teachers, achieving up to +28.64% improvement on math benchmarks over standard OPD.
On-Policy Delta Distillation
The paper introduces On-Policy Delta Distillation (OPD^2), a new distillation reward called the delta signal that captures the difference between a teacher model and its base model before reasoning tuning, providing a more direct signal for transferring reasoning capabilities. Experiments across math, science, and code benchmarks show OPD^2 consistently outperforms conventional on-policy distillation.
dOPSD: On-Policy Self-Distillation for Diffusion Language Models
This paper introduces dOPSD, an on-policy self-distillation method for diffusion language models that leverages internal denoising trajectories to improve mathematical reasoning and code generation.
On-policy distillation: one of the hottest terms on PapersWithCode [R]
Hugging Face's Niels introduces On-policy Distillation (OPD), a key post-training technique used in models like Qwen 3.6/3.7, GLM-5.1, and DeepSeek-V4, now featured on PapersWithCode with a linked whiteboard explanation by Sasha Rush and Dwarkesh Patel.
Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives
This UCLA paper theoretically analyzes on-policy distillation (OPD) versus off-policy/SFT-style distillation, showing how forward and reverse KL objectives aggregate multiple teachers into arithmetic vs geometric mixtures, establishing logarithmic regret bounds, and explaining both the benefits and fragility of OPD (e.g., sensitivity to low-probability teachers and incorrect-prefix bias).