Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Summary
The paper introduces On-Policy Reverse Distillation (OPRD), a method that enables stronger AI models to exceed weaker supervisors by amplifying verifier-supported policy gradients along the teacher's shift direction, achieving higher performance with fewer updates in distillation scenarios.
View Cached Full Text
Cached at: 09/09/26, 04:28 AM
Paper page - Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Source: https://huggingface.co/papers/2609.08798
Abstract
On-Policy Reverse Distillation enables stronger models to exceed weak supervisors by amplifying verifier-supported policy gradients along the teacher’s shift direction, accelerating optimization without imposing capacity limits.
Weak-to-strong generalizationasks whether stronger models can learn from weaker supervisors and surpass them. This question is particularly important for successive model generations and multi-domain consolidation, where repeating frontier-scale post-training from scratch can be prohibitively expensive. Yet conventional distillation treats the weak teacher as an optimization target, potentially imposing its capacity ceiling on the student. We introduceOn-Policy Reverse Distillation(OPRD), which evaluates the teacher’s policy shift relative to its reference policy on student rollouts and amplifies the component of the student’s verifier-drivenpolicy gradientalong that direction. By rescaling only verifier-supported updates, OPRD preserves the stationary points of policy optimization while accelerating learning beyond the teacher. In both successive model transfer andmulti-teacher distillation, OPRD achieves higher performance with fewer student updates than existing RL and distillation approaches. Response-style analysis shows that OPRD students remain closer to models trained with verifier-based RL alone than to their weak teachers, suggesting that teacher guidance accelerates rather than redirects the student’s own optimization. Results in conventional strong-to-weak distillation further demonstrate that OPRD effectively combinesverifier-driven policy optimizationwith teacher guidance regardless of capacity ordering.
View arXiv pageView PDFGitHub3Add to collection
Get this paper in your agent:
hf papers read 2609\.08798
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2609.08798 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.08798 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2609.08798 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Weak-to-Strong Generalization via Direct On-Policy Distillation
Direct-OPD distills the policy shift from a small model's pre- and post-RL checkpoints to improve a larger student model via on-policy distillation, achieving significant gains without expensive RL on the student.
Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
The paper analyzes on-policy distillation, revealing it primarily suppresses low-probability tokens rather than relying on teacher guidance, and introduces OPSA, a supervision-free method that significantly enhances reasoning performance.
Weak-to-Strong On-Policy Distillation
Introduces Weak-to-Strong On-Policy Distillation (W2S-OPD), a framework that improves a strong language model by distilling from multiple weaker models using contrast pairs in logit space, consistently outperforming standard on-policy distillation on math and code benchmarks.
On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers
This paper introduces a reinforcement learning-based distillation framework for training compact instruction-following rerankers, using off-policy GRPO for teacher enhancement and on-policy distillation for student learning, demonstrating superior performance under distribution shift.
On-Policy Delta Distillation
The paper introduces On-Policy Delta Distillation (OPD^2), a new distillation reward called the delta signal that captures the difference between a teacher model and its base model before reasoning tuning, providing a more direct signal for transferring reasoning capabilities. Experiments across math, science, and code benchmarks show OPD^2 consistently outperforms conventional on-policy distillation.