Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Hugging Face Daily Papers Papers
on-policy-distillation self-improvement entropy-adaptive distillation-noise token-suppression reasoning-performance

Summary

The paper analyzes on-policy distillation, revealing it primarily suppresses low-probability tokens rather than relying on teacher guidance, and introduces OPSA, a supervision-free method that significantly enhances reasoning performance.

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.
Original Article
View Cached Full Text

Cached at: 09/01/26, 11:55 AM

Paper page - Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Source: https://huggingface.co/papers/2608.31046

Abstract

On-policy distillation relies mainly on suppressing low-probability tokens rather than teacher guidance, motivating a supervision-free entropy-adaptive method that substantially improves reasoning performance.

On-policy distillation(OPD) offers densetoken-level supervisionas an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherentlyoff-policyfor it, so the reliability of its supervision, and hence the source of the student’s improvement, remains unclear. We quantitatively analyzeteacher supervisionduring OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates onlow log-probability tokens, and using a single fixednegative advantagematches the performance of teacher-provided ones. This suggests that OPD works largely by suppressinglow log-probability tokens, which requires no teacher. These findings motivateOn-Policy Self-Adaptation(OPSA), a supervision-free method usingentropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2608\.31046

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper3

#### Tuwhy/Qwen3-1.7B-OPSA Text Generation• 2B• Updatedabout 9 hours ago • 357 #### Tuwhy/Qwen3-4B-OPSA Text Generation• 4B• Updatedabout 9 hours ago • 356 #### Tuwhy/Qwen3.5-9B-OPSA Image-Text-to-Text• 10B• Updatedabout 9 hours ago • 12

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.31046 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.31046 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

Spend Teacher Tokens Where They Matter: Success-Referenced On-Policy Distillation

arXiv cs.AI

The paper introduces Success-Referenced On-Policy Distillation (SR-OPD), which uses a student's own successful rollouts as references to selectively allocate teacher supervision, achieving comparable reasoning performance on math benchmarks using only 3.46–5.02% of the teacher-input tokens required by vanilla on-policy distillation.

When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

Hugging Face Daily Papers

This paper introduces SA-OPD, a spurious-signal-aware on-policy distillation framework that filters misleading token-level teacher supervision based on input-grounding and optimization impact, improving LLM and VLM distillation performance.

Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?

arXiv cs.LG

这篇论文从知识蒸馏视角重新审视on-policy distillation（OPD），通过受控token翻转实验测量教师在学生生成轨迹上的优势来源，发现基于mass和KL的选择性干预比随机翻转高出15-18个百分点，主张在保持分布稳定性的同时学习更有价值的教师纠正。

Understanding Off- vs On-Policy Distillation: A Tale of Distinct Training Objectives

arXiv cs.LG

This UCLA paper theoretically analyzes on-policy distillation (OPD) versus off-policy/SFT-style distillation, showing how forward and reverse KL objectives aggregate multiple teachers into arithmetic vs geometric mixtures, establishing logarithmic regret bounds, and explaining both the benefits and fragility of OPD (e.g., sensitivity to low-probability teachers and incorrect-prefix bias).