@neural_avb: This article actually explains all the components of On Policy Distillation Loss functions (forward vs reverse KL), sup…
Summary
This article explains components of On Policy Distillation loss functions, including forward vs reverse KL divergence, supervision granularity, privilege types, and privilege advantage estimation. Additional resources from Thinking Machines, Hugging Face, and others are provided.
View Cached Full Text
Cached at: 07/11/26, 07:21 AM
This article actually explains all the components of On Policy Distillation
Loss functions (forward vs reverse KL), supervision granularity, types of privilege, and this concept of privilege advantage estimation.
Additional resources I like for OPD:
Thinking Machines/Reverse KL: http://thinkingmachines.ai/blog/on-policy…
TRL: http://huggingface.co/spaces/Hugging…
wh: http://x.com/nrehiew_/statu…
Will Brown: http://x.com/willccbb/statu…
Similar Articles
On-Policy Distillation (5 minute read)
This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.
@neural_avb: There is a really banger article on On-Policy Distillation. Came out on HF a few months back.
A tweet recommending an article on on-policy distillation published on Hugging Face.
The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.
@neural_avb: If yall are interested in On Policy Distillation, check this specific repo. Somebody put together a curated collection …
A curated collection of papers and tools for On Policy Distillation, organized and annotated with a getting-started section, shared via a GitHub repo.
Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
This paper systematically studies on-policy distillation in LLM post-training, clarifying its role as an exploration catalyst and identifying pathologies like Student-Teacher Mismatch and Length Exploitation, proposing light-weight signal regulations.