Trust Region Policy Distillation

Hugging Face Daily Papers Papers
reinforcement-learning policy-distillation trust-region training-stability mathematical-reasoning

Summary

Trust Region Policy Distillation (TOP-D) stabilizes on-policy distillation by dynamically constructing a proximal teacher, providing theoretical convergence guarantees and empirical gains in mathematical reasoning tasks with zero additional overhead.

Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.
Original Article
View Cached Full Text

Cached at: 07/13/26, 07:50 AM

Paper page - Trust Region Policy Distillation

Source: https://huggingface.co/papers/2607.04751

Abstract

Biggoalsarehardtoachieveallatonce;breakingthemintosmallstepsiswiser.WepresentTrustRegionPolicyDistillation(TOP-D),whichtransformsthenotoriouslyunstable,high-varianceOn-PolicyDistillation(OPD)intoastabletrainingparadigmbydynamicallyconstructingaproximalteacher.Theoretically,weestablisharigorousframeworkdemonstratingthatTOP-Dinherentlycontrolsgradientvariance.Byprovidingaformalglobalconvergenceanalysisalongsideamonotonicimprovementbound,wemathematicallyformalizethereliabilityandstabilityoftheoveralltrainingdynamics.Empirically,TOP-Ddramaticallyenhancestrainingstability,sampleefficiency,andfinalperformanceonmathematicalreasoningtasks.Moreimportantly,TOP-Dintroduceszeroadditionalcomputationaloverhead,positioningitselfasapromisingalternativetothewell-establishedOPDparadigm.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.04751

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.04751 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.04751 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.04751 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

Trust Region On-Policy Distillation

Hugging Face Daily Papers

The paper proposes Trust Region On-Policy Distillation (TrOPD) to stabilize on-policy distillation of large language models by using trust regions, outlier estimation, and off-policy guidance, outperforming existing methods on reasoning and code generation benchmarks.

Trust-Region Behavior Blending for On-Policy Distillation

Hugging Face Daily Papers

Trust-Region behavior Blending (TRB) improves on-policy distillation by replacing poor early student rollouts with teacher-like behavior within a KL trust region during warmup, achieving stronger results on math-reasoning tasks.

Pass the Baton: Trajectory-Relayed On-Policy Distillation

arXiv cs.CL

Proposes Relay On-Policy Distillation (Relay-OPD) that addresses prefix failure in on-policy distillation by having the teacher briefly take over at detected handoff triggers to correct reasoning trajectories. Achieves consistent improvements on mathematical reasoning benchmarks and reduces training trajectory length by over 50%.

Trace-Based On-Policy Distillation for Masked Diffusion Language Models

arXiv cs.CL

A paper proposing Trace-Based On-Policy Distillation (TOPD), a teacher-supervised framework for transferring reasoning abilities to masked diffusion language models without reward estimation, achieving comparable accuracy to RL-trained counterparts with significant compute speedup.