The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
Summary
This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.
View Cached Full Text
Cached at: 05/13/26, 04:12 AM
Paper page - The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
Source: https://huggingface.co/papers/2605.11182
Abstract
On-policy distillation and self-distillation methods for large language models exhibit varying effectiveness depending on teacher choice, loss formulation, and instance-specific privileged information availability, with identified failure mechanisms including distribution mismatch, optimization instability, and PI-free policy learning.
On-policy distillation(OPD) andon-policy self-distillation(OPSD) have emerged as promising post-training methods forlarge language models, offering densetoken-level supervisionon trajectories sampled from the modelโs own policy. However, existing results on their effectiveness remain mixed: while OP(S)D has shown promise in system prompt and knowledge internalization, recent studies also report instability and degradation. In this work, we present a comprehensive empirical study of when OPD and OPSD work, when they fail, and why. We find that OPD on mathematical reasoning is highly sensitive to teacher choice and loss formulation, whereas OPSD fails in our tested settings due to test-time absence of instance-specific privileged information (PI). In contrast, OPSD is effective when PI represents a shared latent rule, such as a system prompt or alignment preference. We identify three failure mechanisms: (1) distribution mismatch between teacher and student caused by conditioning on student-generated prefixes, (2) optimization instability from biasedTopKreverse-KL gradients, and (3) an OPSD-specific limitation where the student learns a PI-free policy that aggregates PI-conditioned teachers, which is insufficient when PI is instance-specific. We further show thatstop-gradientTopKobjectives,RLVR-adapted teachers, andSFT-stabilized students mitigate these failures.
View arXiv pageView PDFProject pageGitHub1Add to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.11182 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.11182 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.11182 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
This paper systematically studies on-policy distillation in LLM post-training, clarifying its role as an exploration catalyst and identifying pathologies like Student-Teacher Mismatch and Length Exploitation, proposing light-weight signal regulations.
On-Policy Distillation (5 minute read)
This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.
@BhavinJawade: ๐ข๐ป-๐ฝ๐ผ๐น๐ถ๐ฐ๐ ๐ฑ๐ถ๐๐๐ถ๐น๐น๐ฎ๐๐ถ๐ผ๐ป ๐ถ๐๐ป'๐ ๐ฎ ๐ณ๐ฟ๐ฒ๐ฒ-๐น๐๐ป๐ฐ๐ต On-policy distillation has become a defaultโฆ
Bhavin Jawade discusses several failure modes of on-policy distillation for training large language models, including early mistakes becoming uncorrectable, stronger teachers being worse, privileged information conditioning failing to transfer, and thinking collapse from dense supervision.
Weak-to-Strong Generalization via Direct On-Policy Distillation
Direct-OPD distills the policy shift from a small model's pre- and post-RL checkpoints to improve a larger student model via on-policy distillation, achieving significant gains without expensive RL on the student.
SFT, RL, and On-Policy Distillation Through a Distributional Lens (19 minute read)
This article analyzes post-training methods for language models through a distributional perspective, comparing how SFT, RL, and on-policy distillation reshape model distributions and impact phenomena like catastrophic forgetting.