on-policy

This paper proposes a hierarchical agent distillation framework that separates on-policiness into two levels—the policy generating interaction histories and the one generating action prefixes—finding that student-induced histories with teacher-generated actions are the strongest signal, and that mixtures and scheduling matter beyond overall on-policy data amounts.

NVIDIA的研究提出Semi-OPD替代方案，发现在初始学生的离线rollout上进行蒸馏通常优于on-policy蒸馏，在17组教师-学生配对中14次胜出，最高提升13.6%准确率和11.4倍训练加速。

该论文研究了 on-policy distillation (OPD) 是否总是优于离线方法，提出 Semi-OPD——在初始学生的离线 rollout 上进行蒸馏。在 17 组师生模型对中，Semi-OPD 在 14 组中优于 OPD，最高提升 13.6% 准确率并实现 11.4 倍训练加速。

OnlineQAT 采用两阶段框架：先通过块级 QAT 得到低位初始化，再在学生自生成的轨迹上用冻结的全精度教师进行反向 KL 的在线蒸馏，从而缓解固定补全训练与部署时学生自采样前缀之间的分布错配。在 Qwen3-1.7B 上，OnlineQAT 在 W3A16（57.28）和 W2A16（32.52）设置中取得对比方法中的最佳平均分，较 ReasoningQAT 分别提升 2.90 和 0.44 分。

UniEvo-VL 提出了一种用于多模态模型的在线自蒸馏训练方案，让单一模型同时充当教师与学生（教师条件于自生成的批评信息），以提升其图像生成能力，在 GenEval 上从 0.747 提升至 0.808。

Where-OPD proposes spatially guided on-policy self-distillation for MLLMs, where a teacher with privileged spatial text guidance is distilled into a student using procedurally generated synthetic scenes, yielding a 3.23-point average gain on real-world perception benchmarks via synthetic-to-real transfer.

MInTRL proposes minimal interventions in on-policy reinforcement learning to expand exploration and improve performance on math and code benchmarks, outperforming standard baselines.

FlowBalance is a verifier-grounded self-improvement method that calibrates on-policy reasoning experience to enhance AI model performance and stability in mathematical reasoning tasks.

SMOPD is a loss-only stabilization method for multi-turn on-policy self-distillation that uses selective token-entropy masking to improve accuracy in dirty-history settings, demonstrating improvements with Qwen3 models.

This paper proposes an adaptive supervised anchoring framework for on-policy self-distillation, addressing the problem of rollout-conditioned signal degradation in language model training. The method separates rollout-conditioned distribution matching from canonical-context supervision, improving task acquisition while preserving general capabilities.

This paper studies On-Policy Delta Distillation (OPD^2) for multilingual math reasoning in English, Korean, and Japanese, showing consistent improvements over standard OPD and narrowing language gaps.

This paper introduces OPTD, an on-policy transition distillation method with consistency-guided adaptive compression for few-step diffusion language models, improving quality-efficiency trade-offs across four reasoning and code-generation benchmarks.

Poly-OPD is a framework for distilling complementary strengths from heterogeneous text-to-image flow models into a single compact flow-matching student, using pixel bridges and gradient-compatible adapters. It improves GenEval and DrawBench scores while consolidating multiple teacher capabilities.

Any-OPD presents the first framework for on-policy distillation between arbitrary latent flow-matching generators, enabling distillation from a 12B FLUX model to a 2.5B SD3.5 model by bridging via a frozen vision representation. It improves the student's PickScore from 0.846 to 0.884, rivaling the teacher at a fifth of its size.

Introduces Visual Attribution Distillation (VAD), a counterfactual target-reconstruction method for multimodal on-policy distillation that estimates the visually-attributable part of teacher corrections. Outperforms existing approaches across fine-grained visual benchmarks at 4B and 9B scales.

Proposes NOPD, a self-distillation method that improves vision-language models without external supervision by leveraging prediction discrepancies between clean and corrupted inputs. Achieves significant gains on visual reasoning tasks, matching or exceeding RL and distillation from external models.

Introduces Byte-Prefix Marginalization (BPM) for cross-tokenizer on-policy distillation, re-expressing teacher token distributions over student vocabulary in a shared byte space, outperforming baselines on mathematics and programming benchmarks.

This paper identifies a failure mode in classifier-free guidance distillation called Negative Branch Asymmetry, where errors in the positive and negative CFG branches cancel out, and proposes Positive-Direction Matching to supervise branches separately for more robust distilled models.

VCSD removes the need for external teachers, privileged answers, or visual evidence in on-policy self-distillation by using content-erased control images to produce contrastive signals, consistently outperforming existing methods on vision-language benchmarks.

A paper proposing Trace-Based On-Policy Distillation (TOPD), a teacher-supervised framework for transferring reasoning abilities to masked diffusion language models without reward estimation, achieving comparable accuracy to RL-trained counterparts with significant compute speedup.