Trust Region On-Policy Distillation
Summary
The paper proposes Trust Region On-Policy Distillation (TrOPD) to stabilize on-policy distillation of large language models by using trust regions, outlier estimation, and off-policy guidance, outperforming existing methods on reasoning and code generation benchmarks.
View Cached Full Text
Cached at: 06/03/26, 03:35 AM
Paper page - Trust Region On-Policy Distillation
Source: https://huggingface.co/papers/2606.01249
Abstract
Trust Region On-Policy Distillation (TrOPD) improves reliable token-level supervision in large language model distillation by using trust regions, outlier estimation, and off-policy guidance to address instability issues under distribution mismatch.
On-Policy Distillation(OPD) is a fundamental technique for efficient post-training of large language models (LLMs), with broad applications in agent learning, multi-task enhancement, and model compression. However, OPD training becomes unstable when the teacher and student distributions differ substantially, as teacher supervision on student-generated tokens may yield unreliable policy gradients and even cause optimization failure. This work addresses reliable on-policytoken-level supervisionthrough credit assignment strategies, and proposesTrust RegionOn-Policy Distillation, TrOPD. It features the following characteristics: 1) Trust-Region On-Policy Learning: TrOPD performs OPD only in regions where the teacher provides reliable supervision, mitigating the optimization difficulty of the K1reverse-KL estimatorunderdistribution mismatch. 2) Outlier Estimation: For outlier regions, we exploregradient clipping, masking, andforward-KL estimationto reduce the adverse effects of unreliable supervision. 3)Off-Policy Guidance: The student continues generation from teacher prefixes and uses forward KL to imitateoff-policy guidance, encouraging on-policy exploration toward reliable regions. Experiments show that TrOPD consistently outperforms SoTA OPD baselines, including OPD, EOPD, and REOPOLD, across mathematical reasoning, code generation, and general-domain benchmarks.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.01249
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.01249 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.01249 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.01249 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Trust Region Policy Distillation
Trust Region Policy Distillation (TOP-D) stabilizes on-policy distillation by dynamically constructing a proximal teacher, providing theoretical convergence guarantees and empirical gains in mathematical reasoning tasks with zero additional overhead.
Trust-Region Behavior Blending for On-Policy Distillation
Trust-Region behavior Blending (TRB) improves on-policy distillation by replacing poor early student rollouts with teacher-like behavior within a KL trust region during warmup, achieving stronger results on math-reasoning tasks.
Tail-Aware Top-$k$ On-Policy Distillation
This paper proposes Tail-Aware Top-k On-Policy Distillation (TA-OPD) to address the loss of tail probability in on-policy distillation for language models, improving downstream accuracy on benchmarks.
PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation
PowerOPD introduces a bounded power transformation to stabilize on-policy distillation for large language models, achieving significant gains in accuracy and sample efficiency while reducing computational cost.
On-policy Distillation with Verifiable Reward
The paper introduces OPDVR, a method that combines on-policy distillation with verifiable rewards using a ReLU-gated reformulation to enhance reasoning in large language models without extra hyperparameters, outperforming standard approaches on six benchmarks.