DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation

Hugging Face Daily Papers Papers
on-policy-distillation multi-task-learning knowledge-distillation llm-training teacher-student reinforcement-learning

Summary

DuoOPD 论文提出一种新的多任务在线策略蒸馏方法，利用教师-学生的联合结果来决定反馈方向与教师支持方式，在 Qwen3 和 Llama 上相比标准 OPD 平均宏准确率提升 2.58 和 5.98 个百分点。

On-policy distillation (OPD) trains a student on its own responses with token-level feedback from a stronger teacher, yet the teacher can fail on questions the student already answers correctly, and how often each model succeeds varies across tasks. OPD ignores these outcomes and, on average, pushes down even the student's correct responses; gating feedback by student correctness fixes the direction but uses the teacher in the same way whether or not it succeeded. We introduce DuoOPD, in which the student's outcome sets the direction of feedback and the joint teacher-student outcome decides how the teacher supports it: when only the teacher succeeds, its verified answer becomes context for scoring the student's failed response, and when only the student succeeds, a weight shared within the task reinforces the whole response. A single rule covers all four outcome combinations without task-specific settings. Across Qwen3 and Llama, DuoOPD outperforms all five baselines in mean macro accuracy, improving over OPD by 2.58 and 5.98 percentage points, and it also leads on two further task mixtures spanning scientific calculation, instruction following, and code generation. Ablations show that outcome-based direction alone stays near the gated baseline, while the joint-outcome designs supply most of the gain.
Original Article
View Cached Full Text

Cached at: 10/01/26, 04:21 AM

Paper page - DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation

Source: https://huggingface.co/papers/2609.33711

Abstract

On-policydistillation(OPD)trainsastudentonitsownresponseswithtoken-levelfeedbackfromastrongerteacher,yettheteachercanfailonquestionsthestudentalreadyanswerscorrectly,andhowofteneachmodelsucceedsvariesacrosstasks.OPDignorestheseoutcomesand,onaverage,pushesdowneventhestudent’scorrectresponses;gatingfeedbackbystudentcorrectnessfixesthedirectionbutusestheteacherinthesamewaywhetherornotitsucceeded.WeintroduceDuoOPD,inwhichthestudent’soutcomesetsthedirectionoffeedbackandthejointteacher-studentoutcomedecideshowtheteachersupportsit:whenonlytheteachersucceeds,itsverifiedanswerbecomescontextforscoringthestudent’sfailedresponse,andwhenonlythestudentsucceeds,aweightsharedwithinthetaskreinforcesthewholeresponse.Asinglerulecoversallfouroutcomecombinationswithouttask-specificsettings.AcrossQwen3andLlama,DuoOPDoutperformsallfivebaselinesinmeanmacroaccuracy,improvingoverOPDby2.58and5.98percentagepoints,anditalsoleadsontwofurthertaskmixturesspanningscientificcalculation,instructionfollowing,andcodegeneration.Ablationsshowthatoutcome-baseddirectionalonestaysnearthegatedbaseline,whilethejoint-outcomedesignssupplymostofthegain.

View arXiv pageView PDFGitHub3Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.33711 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.33711 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.33711 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Stochastic Teacher Intervention for Agentic On-Policy Distillation

arXiv cs.CL

提出STI-OPD框架，通过基于教师-学生策略差异的随机教师干预和重要性加权反向KL目标，解决多轮agent任务中on-policy蒸馏的轨迹漂移问题，在工具推理与长程交互基准上超越先前最强OPD基线。