When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Hugging Face Daily Papers Papers
distillation on-policy llm-training teacher-student efficiency

Summary

该论文研究了 on-policy distillation (OPD) 是否总是优于离线方法，提出 Semi-OPD——在初始学生的离线 rollout 上进行蒸馏。在 17 组师生模型对中，Semi-OPD 在 14 组中优于 OPD，最高提升 13.6% 准确率并实现 11.4 倍训练加速。

On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills from offline rollouts generated by the initial student, can often outperform OPD in both accuracy and training efficiency. Across 17 teacher-student pairs ranging from 1.5B to 235B parameters, Semi-OPD outperforms OPD in 14 cases, with up to +13.6% accuracy and 11.4x training speedup. We further find that the choice between OPD and Semi-OPD depends on the alignment between the initial teacher and student, quantified by an output-token overlap ratio: OPD is beneficial only when the two are highly aligned with high overlap ratios. Our deeper investigation suggests that effective distillation requires on-policyness w.r.t. both the student and the teacher. For misaligned pairs, student rollouts can become increasingly off-policy w.r.t. the teacher as context length grows, weakening the distillation signal. In contrast, Semi-OPD is often more stable, as it distills on shorter contexts while covering full trajectories and exposing the student to more teacher-preferred tokens. Beyond proposing Semi-OPD as an efficient alternative, our work motivates the community to rethink when to use OPD and to study stronger OPD variants with meaningful teacher-student pairs.
Original Article
View Cached Full Text

Cached at: 10/12/26, 10:39 AM

Paper page - When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

Source: https://huggingface.co/papers/2610.11291

Abstract

On-policydistillation(OPD)hasbecomeincreasinglypopularfortransferringteachercapabilitiestostudentmodels.Inthiswork,weaskacriticalresearchquestion:Ison-policysamplingalwaysbeneficialfordistillingarbitraryteacher-studentpairs?Weshowthatasimplealternative,Semi-OPD,whichdistillsfromofflinerolloutsgeneratedbytheinitialstudent,canoftenoutperformOPDinbothaccuracyandtrainingefficiency.Across17teacher-studentpairsrangingfrom1.5Bto235Bparameters,Semi-OPDoutperformsOPDin14cases,withupto+13.6%accuracyand11.4xtrainingspeedup.WefurtherfindthatthechoicebetweenOPDandSemi-OPDdependsonthealignmentbetweentheinitialteacherandstudent,quantifiedbyanoutput-tokenoverlapratio:OPDisbeneficialonlywhenthetwoarehighlyalignedwithhighoverlapratios.Ourdeeperinvestigationsuggeststhateffectivedistillationrequireson-policynessw.r.t.boththestudentandtheteacher.Formisalignedpairs,studentrolloutscanbecomeincreasinglyoff-policyw.r.t.theteacherascontextlengthgrows,weakeningthedistillationsignal.Incontrast,Semi-OPDisoftenmorestable,asitdistillsonshortercontextswhilecoveringfulltrajectoriesandexposingthestudenttomoreteacher-preferredtokens.BeyondproposingSemi-OPDasanefficientalternative,ourworkmotivatesthecommunitytorethinkwhentouseOPDandtostudystrongerOPDvariantswithmeaningfulteacher-studentpairs.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.11291

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.11291 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.11291 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.11291 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

arXiv cs.LG

这篇论文从知识蒸馏视角重新审视on-policy distillation（OPD），通过受控token翻转实验测量教师在学生生成轨迹上的优势来源，发现基于mass和KL的选择性干预比随机翻转高出15-18个百分点，主张在保持分布稳定性的同时学习更有价值的教师纠正。