Flux-OPD: On-Policy Distillation with Evolving Contexts

Hugging Face Daily Papers Papers
on-policy-distillation language-models evolving-contexts training knowledge-distillation

Summary

Flux-OPD proposes an on-policy distillation paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains, outperforming existing OPD paradigms.

Large language model training in open-ended domains lacks verifiable rewards, making task preferences difficult to formalize as effective supervision. Contexts can convey such preferences, yet provide little additional supervision once distilled into the student, motivating contexts that evolve with student performance. However, directly using evolving contexts as in-training supervision results in an unstable distillation target and conflicting distributions, requiring mechanisms to stabilize target and downweight conflicts. In this paper, we analyze the effect of contexts through a decomposition of the reverse KL objective, revealing two findings: the student is distilled toward the geometric mean of context-conditioned teachers, and the objective contains a conflict term that measures conflicts among these teachers. Based on this decomposition, we propose Flux-OPD, an OPD paradigm that uses evolving contexts as in-training supervision to capture task preferences in open-ended domains. Flux-OPD treats the differences between context-conditioned and context-free teachers as contextual difference signals, injects them as contextual corrections into the context-free teacher anchor, and weights their correction strength using the conflict term as an indicator. Experiments on open-ended tasks show that Flux-OPD outperforms existing OPD paradigms, highlighting the potential to combine teacher supervision with evolving contexts.
Original Article
View Cached Full Text

Cached at: 07/31/26, 05:53 AM

Paper page - Flux-OPD: On-Policy Distillation with Evolving Contexts

Source: https://huggingface.co/papers/2607.28022 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Largelanguagemodeltraininginopen-endeddomainslacksverifiablerewards,makingtaskpreferencesdifficulttoformalizeaseffectivesupervision.Contextscanconveysuchpreferences,yetprovidelittleadditionalsupervisiononcedistilledintothestudent,motivatingcontextsthatevolvewithstudentperformance.However,directlyusingevolvingcontextsasin-trainingsupervisionresultsinanunstabledistillationtargetandconflictingdistributions,requiringmechanismstostabilizetargetanddownweightconflicts.Inthispaper,weanalyzetheeffectofcontextsthroughadecompositionofthereverseKLobjective,revealingtwofindings:thestudentisdistilledtowardthegeometricmeanofcontext-conditionedteachers,andtheobjectivecontainsaconflicttermthatmeasuresconflictsamongtheseteachers.Basedonthisdecomposition,weproposeFlux-OPD,anOPDparadigmthatusesevolvingcontextsasin-trainingsupervisiontocapturetaskpreferencesinopen-endeddomains.Flux-OPDtreatsthedifferencesbetweencontext-conditionedandcontext-freeteachersascontextualdifferencesignals,injectsthemascontextualcorrectionsintothecontext-freeteacheranchor,andweightstheircorrectionstrengthusingtheconflicttermasanindicator.Experimentsonopen-endedtasksshowthatFlux-OPDoutperformsexistingOPDparadigms,highlightingthepotentialtocombineteachersupervisionwithevolvingcontexts.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.28022

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.28022 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.28022 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.28022 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Flow-OPD: On-Policy Distillation for Flow Matching Models

Hugging Face Daily Papers

Flow-OPD is a research paper introducing a two-stage on-policy distillation framework for Flow Matching text-to-image models, significantly improving generation quality and alignment metrics using Stable Diffusion 3.5 Medium.

On-policy distillation: one of the hottest terms on PapersWithCode [R]

Reddit r/MachineLearning

Hugging Face's Niels introduces On-policy Distillation (OPD), a key post-training technique used in models like Qwen 3.6/3.7, GLM-5.1, and DeepSeek-V4, now featured on PapersWithCode with a linked whiteboard explanation by Sasha Rush and Dwarkesh Patel.