What Does Privileged Information Add to On-Policy Self-Distillation?

Hugging Face Daily Papers Papers
self-distillation privileged-information language-models mathematical-reasoning on-policy-learning arxiv

Summary

The paper investigates the contribution of privileged information in on-policy self-distillation for language models, finding that reference-free distillation accounts for most improvements, with limited additional benefits from privileged references.

On-policy self-distillation (OPSD) lets a language model learn from a frozen copy of itself that sees an answer or a worked solution. Giving the teacher this extra information seems to offer the student more to learn, but how much does it add beyond distillation itself? To isolate that contribution, we construct AMPLE-Math, a reusable suite of 5,319 mathematical problems with six reasoning views that share the same answer, and compare each view with matched reference-free distillation. With a thinking-enabled teacher supervising direct-response rollouts, reference-free distillation accounts for much of Qwen3-1.7B's improvement under thinking-enabled evaluation, both in domain and on external benchmarks. Evidence for an additional reference benefit is modest in Qwen, strongest for a polished solution, whereas complete traces add two percentage points in SmolLM3-3B at step 50. These benefits depend on the student being trained. At the same checkpoint, replacing short direct-response rollouts with long thinking-enabled rollouts turns gains into losses in both families while the problems, references, and evaluation stay fixed. Teacher profiles and matched loss interventions in Qwen further show that changing token-level supervision can leave student behavior largely unchanged. Together, these findings suggest that OPSD can improve access to existing reasoning capabilities through parameters shared by direct-response and thinking-enabled inference. The value of a privileged reference is what it adds to this cross-mode transfer, not how much of the solution it reveals.
Original Article
View Cached Full Text

Cached at: 09/18/26, 02:59 AM

Paper page - What Does Privileged Information Add to On-Policy Self-Distillation?

Source: https://huggingface.co/papers/2609.20612

Abstract

On-policyself-distillation(OPSD)letsalanguagemodellearnfromafrozencopyofitselfthatseesanansweroraworkedsolution.Givingtheteacherthisextrainformationseemstoofferthestudentmoretolearn,buthowmuchdoesitaddbeyonddistillationitself?Toisolatethatcontribution,weconstructAMPLE-Math,areusablesuiteof5,319mathematicalproblemswithsixreasoningviewsthatsharethesameanswer,andcompareeachviewwithmatchedreference-freedistillation.Withathinking-enabledteachersupervisingdirect-responserollouts,reference-freedistillationaccountsformuchofQwen3-1.7B’simprovementunderthinking-enabledevaluation,bothindomainandonexternalbenchmarks.EvidenceforanadditionalreferencebenefitismodestinQwen,strongestforapolishedsolution,whereascompletetracesaddtwopercentagepointsinSmolLM3-3Batstep50.Thesebenefitsdependonthestudentbeingtrained.Atthesamecheckpoint,replacingshortdirect-responserolloutswithlongthinking-enabledrolloutsturnsgainsintolossesinbothfamilieswhiletheproblems,references,andevaluationstayfixed.TeacherprofilesandmatchedlossinterventionsinQwenfurthershowthatchangingtoken-levelsupervisioncanleavestudentbehaviorlargelyunchanged.Together,thesefindingssuggestthatOPSDcanimproveaccesstoexistingreasoningcapabilitiesthroughparameterssharedbydirect-responseandthinking-enabledinference.Thevalueofaprivilegedreferenceiswhatitaddstothiscross-modetransfer,nothowmuchofthesolutionitreveals.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2609\.20612

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.20612 in a model README.md to link it from this page.

Datasets citing this paper1

#### xiuyuz/ample-math Viewer• Updated43 minutes ago • 5.32k • 4 • 1

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.20612 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Rethinking Privileged Information in On-Policy Self-Distillation

arXiv cs.LG

The paper investigates whether performance gains in on-policy self-distillation come from learning privileged reference information or recovering existing reasoning behavior, finding that the correct reference does not consistently benefit performance across various conditions.

DAPD: Dual-Anchored Policy Distillation

Hugging Face Daily Papers

This paper introduces Dual-Anchored Policy Distillation (DAPD), a framework to address privilege illusion in on-policy self-distillation for language models. DAPD improves Qwen3-4B by +2.00 points on average across tasks, with gains persisting at larger scales.