The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation

Hugging Face Daily Papers Papers
on-policy-distillation reinforcement-learning knowledge-distillation representation-learning language-models model-student-training

Summary

The paper proposes RIDE (RL-Induced Direction Extrapolation), a method that extrapolates RL-induced representation residuals in hidden-state space rather than output space during on-policy distillation, enabling student models to approach or exceed RL-trained teachers across multiple model scales and architectures.

On-policy distillation (OPD) trains a student to match the teacher's next-token distributions on the student's own trajectories and has yielded substantial empirical gains. Generalized variants allow the student to surpass the teacher by extrapolating an implicit reward in output space. The language-model head, however, attenuates this change anisotropically: much of the change encoded in the teacher's hidden states reaches the logits at a small fraction of its weight, and the sampled-token log-probability ratios on which output-space extrapolation relies inject noise that the extrapolation amplifies, making training unstable. We observe that reinforcement learning (RL) shifts a model's internal representations relative to its base checkpoint, and that the direction of this shift can be measured at every layer. Motivated by this observation, we propose RIDE (RL-Induced Direction Extrapolation), which extrapolates the RL-induced change directly in representation space: at every layer and token position, RIDE computes the residual between the teacher and its pre-RL checkpoint and regresses the student's hidden states toward targets displaced beyond the teacher along this residual. Conditioned on a sampled trajectory, this regression is equivalent to maximizing a linear directional reward defined by the residual under a quadratic penalty centered at the teacher, which makes explicit how the objective moves the student along the RL-induced direction while limiting its deviation from the teacher. Across four base/RL-teacher pairs spanning different scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL-trained teacher on every pair and is the only method whose mean does so, and it consistently outperforms output-space extrapolation, which degrades the student whenever the teacher is close to its base. Project page: https://github.com/xixixixixxxx/RIDE.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:21 AM

Paper page - The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation

Source: https://huggingface.co/papers/2609.36484

Abstract

On-policydistillation(OPD)trainsastudenttomatchtheteacher’snext-tokendistributionsonthestudent’sowntrajectoriesandhasyieldedsubstantialempiricalgains.Generalizedvariantsallowthestudenttosurpasstheteacherbyextrapolatinganimplicitrewardinoutputspace.Thelanguage-modelhead,however,attenuatesthischangeanisotropically:muchofthechangeencodedintheteacher’shiddenstatesreachesthelogitsatasmallfractionofitsweight,andthesampled-tokenlog-probabilityratiosonwhichoutput-spaceextrapolationreliesinjectnoisethattheextrapolationamplifies,makingtrainingunstable.Weobservethatreinforcementlearning(RL)shiftsamodel’sinternalrepresentationsrelativetoitsbasecheckpoint,andthatthedirectionofthisshiftcanbemeasuredateverylayer.Motivatedbythisobservation,weproposeRIDE(RL-InducedDirectionExtrapolation),whichextrapolatestheRL-inducedchangedirectlyinrepresentationspace:ateverylayerandtokenposition,RIDEcomputestheresidualbetweentheteacheranditspre-RLcheckpointandregressesthestudent’shiddenstatestowardtargetsdisplacedbeyondtheteacheralongthisresidual.Conditionedonasampledtrajectory,thisregressionisequivalenttomaximizingalineardirectionalrewarddefinedbytheresidualunderaquadraticpenaltycenteredattheteacher,whichmakesexplicithowtheobjectivemovesthestudentalongtheRL-induceddirectionwhilelimitingitsdeviationfromtheteacher.Acrossfourbase/RL-teacherpairsspanningdifferentscales,architectures,andpre-traininglineages,RIDEapproachesorexceedstheRL-trainedteacheroneverypairandistheonlymethodwhosemeandoesso,anditconsistentlyoutperformsoutput-spaceextrapolation,whichdegradesthestudentwhenevertheteacherisclosetoitsbase.Projectpage:https://github.com/xixixixixxxx/RIDE.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.36484

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.36484 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.36484 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.36484 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OPRD: On-Policy Representation Distillation

Hugging Face Daily Papers

OPRD proposes a new knowledge distillation method that aligns student and teacher hidden states across layers during on-policy rollouts, eliminating sampling variance from token-space KL estimation. Empirically, OPRD outperforms output-space baselines on math reasoning benchmarks (AIME 2024/2025, AIMO) while being 1.44x faster and using 54% less memory.