self-distillation

UniEvo-VL 提出了一种用于多模态模型的在线自蒸馏训练方案，让单一模型同时充当教师与学生（教师条件于自生成的批评信息），以提升其图像生成能力，在 GenEval 上从 0.747 提升至 0.808。

The paper introduces the concept of 'Mathematical Primitive' to probe structural mathematical understanding in LLMs, presents a benchmark evaluating reasoning along Discovery, Generation, Digestion, and Execution, and proposes a primitive-privileged self-distillation framework that improves mathematical reasoning across model scales.

Where-OPD proposes spatially guided on-policy self-distillation for MLLMs, where a teacher with privileged spatial text guidance is distilled into a student using procedurally generated synthetic scenes, yielding a 3.23-point average gain on real-world perception benchmarks via synthetic-to-real transfer.

ReSAIL 提出一种即插即用的增强方法，通过轨迹平衡选择性蒸馏与特权信息保留，缓解智能体迭代自蒸馏中的性能坍缩问题；在 ALFWorld 和 TextCraft 上使用 Qwen3-4B/8B 模型，三个部署周期后平均成功率比 SDPO/OEL 基线提升 22.5 个百分点。

OmniReasoning introduces OmniReasoningBench, a data engine (OmniQA) producing SFT and RL training sets, and a Modality-Factored Self-Distillation method, yielding a 30B-A3B model that outperforms Qwen3-Omni-30B-A3B-Thinking by double-digit margins on audio-visual joint reasoning benchmarks.

The paper introduces Neighborhood On-Policy Self-Distillation (N-OPSD), which exploits local parameter perturbations of a privileged teacher to pool complementary reference-aligned corrections and route them to student-visited states, improving math reasoning benchmarks (AIME 2024/2025, HMMT Feb 2025) across Qwen3 1.7B/4B/8B models.

UniEvo-VL introduces an on-policy self-distillation training recipe where a single multimodal model acts as both teacher (conditioned on its own privileged self-critiques) and student, improving image generation on Qwen-Image-2512 with gains from 0.747 to 0.808 on GenEval without external supervision.

提出特权表示在策略自蒸馏（PR-OPD），通过让策略为轨迹生成事后技能并以自身作为停止梯度教师对齐各层隐藏状态，在 ALFWorld 与 WebShop 上显著优于 GRPO，最高提升 4.7 和 14.0 个百分点。

This paper analyzes on-policy self-distillation (OPSD) for LLM reasoning and shows that unverified student scaffolds create an imitation gap that worsens with model scale, motivating OASIS, a method that supervises mostly verified by-label on-policy trajectories using model-generated attempts as teacher context, improving Qwen3 1.7B/4B/8B by 3.2-3.8 points on AIME/HMMT benchmarks.

AdviSD proposes a method where a small trainable advisor steers a frozen frontier LLM executor via targeted multi-turn self-distillation, pairing outcome-based RL with feedback-conditioned self-distillation that selectively uses corrections. Experiments with Qwen3-8B advisors guiding Gemini and Claude show 4-6 point gains on BFCL-v3 and EnvScaler with good out-of-domain generalization.

LIFT is a unified image-to-video generation framework that introduces layout-in-future control to specify future scene content and layout, improving quality and controllability under large viewpoint changes.

TISD introduces a method for on-policy self-distillation with trajectory intervention to improve large language model performance in coding and science tasks by allowing teacher-guided branching.

This paper introduces Decoupled Credit Self-Distillation (DCSD) to separate credit direction and magnitude for self-distillation, achieving improvements of 8.45 points in mathematical reasoning and 7.01 points in multimodal reasoning across 11 benchmarks.

SCOPD introduces a sparse-context on-policy self-distillation framework that enhances the efficiency of vision-language models by improving the use of pruned visual tokens, achieving higher performance retention without additional inference cost.

This paper proposes MT-OPSD, a non-policy self-distillation framework to address degradation in multi-turn image editing and introduces LME-Bench for evaluating long-horizon robustness.

X-Tree recovers reusable skill hierarchies directly from agent trajectories (no LLM calls) and integrates them into offline RL, online RLVR, and on-policy self-distillation, improving success rates on WebArena, ScienceWorld, and WebShop by up to 5.8% over standard training recipes at matched data and budget.

The paper proposes LT-OPD, an on-policy self-distillation framework for extreme visual token reduction in multimodal large language models, significantly improving performance under low token budgets while reducing KV-cache usage and FLOPs.

The paper proposes Privileged Self-Practice (PSP), a method that injects privileged information into prompts for multi-turn LLM agents, improving task completion rates on AppWorld and SWE-bench benchmarks compared to existing distillation techniques.

The paper introduces GUI-SD-v2, a two-stage on-policy self-distillation framework for multi-turn GUI agents that enhances privilege following and guidance, achieving superior performance on AndroidWorld and MobileWorld benchmarks.

The paper investigates privileged context design in on-policy self-distillation, demonstrating that intermediate levels of abstraction can improve model performance over full solutions while using fewer hint tokens.