Paper page - Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Source: https://huggingface.co/papers/2608.26872 Authors:

,

,

,

,

,

,

,

,

,

Abstract

Self-OPD eliminates task-specific teachers in flow matching by using self-explored stochastic branches and normalized advantages to optimize the velocity field for multi-objective alignment.

On-policy distillation(OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted toflow matchingmodels. However, this paradigm suffers from two major issues: First, training a separate, task-specific teacher for every new objective incurs high computational costs. Second, the discrepancy between teacher and student distributions often leads to compounding errors along the generation trajectory. In this paper, we introduceSelf-OPD, a teacher-free OPD framework forflow matchingmodels that turns the student’s own self-exploration into step-wise supervision. At each timestep,Self-OPDbranches the deterministic next-state prediction into K stochasticSDE candidates, rolls them out with theODE sampler, and compares their rewards against a deterministic self-reference baseline to obtainnormalized advantages. The velocity field is optimized with an all-branchpull-push objective, where high-advantage branches attract the student and low-advantage branches repel it underdirection-aware attenuationandSDE-variance normalization. Formulti-objective alignment,Self-OPDfuses normalized scores at the reward level, avoiding direct gradient conflict. Experiments on single and mixed reward benchmarks show thatSelf-OPDoutperforms prior RL and OPD methods without task-specific teachers.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.26872

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.26872 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.26872 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.26872 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.