On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

Hugging Face Daily Papers Papers
knowledge-distillation on-policy-learning kl-divergence fine-tuning llm reinforcement-learning

Summary

This paper systematically studies the role of rollout policy in strong-to-weak distillation, finding that token-level KL direction and learning rate matter more than on-policy data, challenging the assumption that on-policy rollouts are inherently preferable.

On-policy learning has been argued to reduce catastrophic forgetting, produce sparser parameter updates, and improve generalisation. However, existing comparisons between supervised fine-tuning and reinforcement learning vary many factors simultaneously, making the contribution of rollout policy difficult to isolate. We study the effect of rollout policy in a controlled strong-to-weak distillation setting, by independently varying rollout policy, token-level KL direction, and learning rate across the Llama3 and Qwen2.5 model families and reasoning tasks spanning scientific, medical, and arithmetic domains. Our analysis reveals a nuanced picture of distillation dynamics in which rollout policy does not necessarily play a central role. Instead, token-level KL direction more clearly shapes task performance and output coverage, while learning rate governs forgetting and update sparsity. Analysis of KL gradients and experiments along a continuous student-teacher rollout-policy spectrum explain this pattern: forward KL is remarkably robust to rollout policy, with its performance stable and strong despite changes to the rollout policy, whereas reverse KL is substantially more sensitive and favours student-generated rollouts. On-policy data nevertheless improves generalisation to harder variants of the Countdown arithmetic task under both KL directions, although this advantage does not reliably persist after subsequent RLVR. Our broader conclusions remain robust to removing gradient clipping, using sampled KL estimators, and training on tasks requiring longer reasoning chains. Overall, our results challenge the view that on-policy rollouts are inherently preferable and show that their value depends critically on the objective, evaluation setting, and optimisation hyperparameters.
Original Article
View Cached Full Text

Cached at: 10/02/26, 04:30 PM

Paper page - On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

Source: https://huggingface.co/papers/2609.35259

Abstract

On-policylearninghasbeenarguedtoreducecatastrophicforgetting,producesparserparameterupdates,andimprovegeneralisation.However,existingcomparisonsbetweensupervisedfine-tuningandreinforcementlearningvarymanyfactorssimultaneously,makingthecontributionofrolloutpolicydifficulttoisolate.Westudytheeffectofrolloutpolicyinacontrolledstrong-to-weakdistillationsetting,byindependentlyvaryingrolloutpolicy,token-levelKLdirection,andlearningrateacrosstheLlama3andQwen2.5modelfamiliesandreasoningtasksspanningscientific,medical,andarithmeticdomains.Ouranalysisrevealsanuancedpictureofdistillationdynamicsinwhichrolloutpolicydoesnotnecessarilyplayacentralrole.Instead,token-levelKLdirectionmoreclearlyshapestaskperformanceandoutputcoverage,whilelearningrategovernsforgettingandupdatesparsity.AnalysisofKLgradientsandexperimentsalongacontinuousstudent-teacherrollout-policyspectrumexplainthispattern:forwardKLisremarkablyrobusttorolloutpolicy,withitsperformancestableandstrongdespitechangestotherolloutpolicy,whereasreverseKLissubstantiallymoresensitiveandfavoursstudent-generatedrollouts.On-policydataneverthelessimprovesgeneralisationtohardervariantsoftheCountdownarithmetictaskunderbothKLdirections,althoughthisadvantagedoesnotreliablypersistaftersubsequentRLVR.Ourbroaderconclusionsremainrobusttoremovinggradientclipping,usingsampledKLestimators,andtrainingontasksrequiringlongerreasoningchains.Overall,ourresultschallengetheviewthaton-policyrolloutsareinherentlypreferableandshowthattheirvaluedependscriticallyontheobjective,evaluationsetting,andoptimisationhyperparameters.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.35259

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.35259 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.35259 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.35259 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

On-Policy Distillation (5 minute read)

TLDR AI

This paper introduces on-policy distillation, which trains a student model on its own trajectories with teacher token-level KL supervision to fix train-inference mismatch, unifying forward-KL, reverse-KL, and JSD losses, with reverse-KL favored for smaller students.

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

arXiv cs.LG

This UCLA paper theoretically analyzes on-policy distillation (OPD) versus off-policy/SFT-style distillation, showing how forward and reverse KL objectives aggregate multiple teachers into arithmetic vs geometric mixtures, establishing logarithmic regret bounds, and explaining both the benefits and fragility of OPD (e.g., sensitivity to low-probability teachers and incorrect-prefix bias).

The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes

Hugging Face Daily Papers

This paper presents a comprehensive empirical study on on-policy distillation for large language models, identifying failure mechanisms like distribution mismatch and optimization instability, and proposing fixes such as stop-gradient objectives and RLVR-adapted teachers.