Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations

Hugging Face Daily Papers Papers
on-policy-distillation llm-post-training exploration-catalyst student-teacher-mismatch length-exploitation signal-regulation

Summary

This paper systematically studies on-policy distillation in LLM post-training, clarifying its role as an exploration catalyst and identifying pathologies like Student-Teacher Mismatch and Length Exploitation, proposing light-weight signal regulations.

On-policy distillation (OPD) has become a key paradigm in LLM post-training, yet its training dynamics remain poorly understood. We present a systematic study examining the role, pathologies, and regulations of OPD. We first clarify the role of OPD as an exploration catalyst: it steers the student toward correct reasoning paths via dense token-level guidance, without expanding capability ceiling. We confirm this by showing that prompt diversity matters more than per-problem sampling numbers, and critically, that the effectiveness of OPD hinges entirely on the quality of its guiding signal. This dependency exposes two pathologies that derail exploration. The Student-Teacher Mismatch occurs when a large teacher-student distributional gap causes the guiding signal to misalign with task correctness, steering exploration in counterproductive directions. Length Exploitation arises when the aggregated token-level objective creates length-dependent shortcuts, allowing the student to game the reward landscape through response truncation or redundant padding, exploring degenerate length modes rather than reasoning strategies. To tame these pathologies, we investigate lightweight signal regulations: advantage clipping and log-scale compression, ensuring exploration is guided by faithful signals. Experiments across seven benchmarks demonstrate that these regulations alleviate length exploitation and enable effective distillation, stably surpassing OPD variants and RLVR baselines, thereby confirming that well-regulated signal quality, rather than mere teacher scale, governs successful exploration in OPD.
Original Article
View Cached Full Text

Cached at: 07/20/26, 09:43 AM

Paper page - Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations

Source: https://huggingface.co/papers/2607.13399

Abstract

On-policydistillation(OPD)hasbecomeakeyparadigminLLMpost-training,yetitstrainingdynamicsremainpoorlyunderstood.Wepresentasystematicstudyexaminingtherole,pathologies,andregulationsofOPD.WefirstclarifytheroleofOPDasanexplorationcatalyst:itsteersthestudenttowardcorrectreasoningpathsviadensetoken-levelguidance,withoutexpandingcapabilityceiling.Weconfirmthisbyshowingthatpromptdiversitymattersmorethanper-problemsamplingnumbers,andcritically,thattheeffectivenessofOPDhingesentirelyonthequalityofitsguidingsignal.Thisdependencyexposestwopathologiesthatderailexploration.TheStudent-TeacherMismatchoccurswhenalargeteacher-studentdistributionalgapcausestheguidingsignaltomisalignwithtaskcorrectness,steeringexplorationincounterproductivedirections.LengthExploitationariseswhentheaggregatedtoken-levelobjectivecreateslength-dependentshortcuts,allowingthestudenttogametherewardlandscapethroughresponsetruncationorredundantpadding,exploringdegeneratelengthmodesratherthanreasoningstrategies.Totamethesepathologies,weinvestigatelightweightsignalregulations:advantageclippingandlog-scalecompression,ensuringexplorationisguidedbyfaithfulsignals.Experimentsacrosssevenbenchmarksdemonstratethattheseregulationsalleviatelengthexploitationandenableeffectivedistillation,stablysurpassingOPDvariantsandRLVRbaselines,therebyconfirmingthatwell-regulatedsignalquality,ratherthanmereteacherscale,governssuccessfulexplorationinOPD.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.13399

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.13399 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.13399 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.13399 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Hugging Face Daily Papers

This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.

On-Policy Distillation (5 minute read)

TLDR AI

This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Hugging Face Daily Papers

MOPD proposes a multi-teacher on-policy distillation paradigm for LLM post-training, enabling efficient integration of multiple domain capabilities by distilling specialized RL teachers into a student model using its own rollouts. It outperforms existing methods like Mix-RL and Cascade RL, and has been deployed in industrial-scale models.