@wu_taiqiang: How to maximize OPD performance? One important thing is warm-up. Then the student-sampled sequence is well defined in t…
Summary
The author discusses a paper that demystifies the warm-up process for OPD (likely on-policy distillation), explaining how warm-up enables well-defined student-sampled sequences and educational token-level dense rewards from the teacher.
View Cached Full Text
Cached at: 08/13/26, 09:30 PM
How to maximize OPD performance? 🧐
One important thing is warm-up. Then the student-sampled sequence is well defined in the teacher’s output space, and the token-level dense reward from the teacher is educational. 🤜
In this paper, we demystify the warm-up process: 🧵 https://t.co/lJfxyyhHXi
How to maximize OPD performance?
One important thing is warm-up. Then the student-sampled sequence is well defined in the teacher’s output space, and the token-level dense reward from the teacher is educational.
In this paper, we demystify the warm-up process:
[1/N] Paper: Simple-OPD: Demystifying Warm-up for On-policy Distillation ArXiv: https://arxiv.org/pdf/2608.06802
[2/N] Considering the warm-up data, the take-away message is. In short, CoT from the teacher matters, not performance. Even wrong CoT works.
[3/N] Considering the warm-up training, the conclusion is shown below. To balance the ID and OOD performance, we recommend LoRA with a suitable rank and trained with a moderate step.
[4/N] Indeed, we consider warm-up as a speed-up for vanilla OPD. For response length and overlap ratio, we can find that the student fits the teacher sooner or later. This is why OPD works.
@grok spread to anyone might be interested.
Nice blog and valuable insights!
Thanks, Junjie!
Similar Articles
Simple-OPD: Demystifying Warm-up for On-policy Distillation
This paper investigates the warm-up stage for on-policy distillation (OPD), showing that teacher-compatible chain-of-thought supervision and LoRA-based training with near-saturation duration improve OPD effectiveness. It introduces Simple-OPD, a plug-and-play initialization method that boosts OPD performance across diverse settings.
On-policy distillation: one of the hottest terms on PapersWithCode [R]
Hugging Face's Niels introduces On-policy Distillation (OPD), a key post-training technique used in models like Qwen 3.6/3.7, GLM-5.1, and DeepSeek-V4, now featured on PapersWithCode with a linked whiteboard explanation by Sasha Rush and Dwarkesh Patel.
@louieworth: New blog post: On-Policy Distillation — Promise, Pitfalls, and Prospects. OPD combines on-policy rollouts with dense te…
This blog post discusses On-Policy Distillation (OPD), a technique that combines on-policy rollouts with dense teacher supervision, and highlights its promise, three failure modes, and the author's new paper on the topic.
OPMD: On-Policy Mechanism Distillation Beyond Output Supervision
The paper introduces On-Policy Mechanism Distillation (OPMD), which transfers teacher intermediate computations compatible with the student's architecture during on-policy distillation, outperforming output-only distillation baselines across multiple teacher-student settings.
Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective
This paper analyzes on-policy distillation (OPD) for language model post-training through a reinforcement learning lens, explaining why OPD both improves performance and can collapse into repetitive generation. It finds that masking unhealthy responses during training and SFT initialization can mitigate collapse, shifting focus from teacher generation quality to its ability to evaluate student rollouts.