self-distillation

The paper proposes Muon-RT, a momentum-free variant of the Muon optimizer that reuses the EMA teacher-student weight displacement in LLM self-distillation, eliminating separate momentum buffers while improving accuracy and reducing persistent training memory on Qwen2.5-7B and Olmo-3-7B.

SCOPE 提出选择性关键步骤的在线自蒸馏框架，通过错误定位与技能生成在 RL 训练中提供细粒度监督，在 ALFWorld 和 WebShop 上显著优于 GRPO 与 SEED。

The paper proposes Teacher-Evolution Self-Distillation (TESD), which improves on-policy self-distillation by continuously optimizing the self-teacher with reward-guided objectives and adaptively down-weighting samples with unreliable teacher guidance, achieving consistent gains over token-level and sequence-level baselines across datasets and model scales.

论文提出 WISD（Weakening-Induced On-Policy Self-Distillation），通过将大推理模型与量化的弱化版本进行对比生成训练信号，在无需特权数据的情况下实现自蒸馏，在数学与代码推理基准上平均提升 4.0–5.5 个百分点并达到最先进性能。

Soft-MTP 提出一种 KL 梯度恢复技巧，去除多token预测训练中的噪声残差梯度，在无额外开销下改进困惑度、投机解码与推理能力。

A plug-and-play repo implementing self-distillation with an EMA teacher that scales to large models by combining LoRA adapters with DeepSpeed ZeRO-3, avoiding the memory and latency overhead of holding separate full teacher and student models. Tested on Qwen3.5 and ToolAlpaca, it includes adapter switching, EMA checkpointing, and memory-optimized DeepSpeed configs.

SWE-Ladder 提出渐进式特权自蒸馏方法，将每个训练实例的验证修复转化为定位、复现、补丁三个阶段的专属证据，用于训练 SWE 智能体。在 SWE-bench Verified 上，Qwen3-4B 结合 R2E-Gym 框架的 pass@1 从 9.6% 提升至 16.4%，优于 GRPO 和 OPSD 基线。

ReMedI 采用基于 Metropolis–Hastings 的推理时搜索策略，让冻结的化学语言模型在多目标药物先导化合物优化中提出并筛选分子编辑，再将搜索轨迹自蒸馏回基础模型，在 C-MuMOInstruct 和 SMDD 基准上以约 7.8× 更少的训练数据达到约 2.8× 于 RL 基线 C-MORAL 的严格成功率。

Google researchers propose a data-free on-policy self-distillation framework where LLMs synthesize their own curriculum of queries and privileged reasoning traces from coarse topic descriptions, achieving self-improvement across text and multimodal benchmarks without external supervision.

The paper proposes DIAL-S2ST, a post-training method that learns inference-time latency control in simultaneous speech-to-speech translation models via self-distillation from lookahead predictions and a jointly trained LoRA adapter, demonstrating flexible latency control on French-to-English translation using Hibiki with only 10 hours of source speech.

该论文提出 Fin-SD（hindsight self-distillation），一种无需外部教师模型的本地化自蒸馏方法，用于提升小型开放权重模型的多步金融推理能力；实验发现增益主要来自监督位置而非提示内容，且与基于完整参考解答的自蒸馏精度相当但少生成 9.5% token。

AMBIT是一个统一框架，通过将自回归场景规划与rectified-flow渲染耦合，实现第一阶Ambisonics（FOA）空间音频的生成与编辑，并利用FOA感知的连续VAE与on-policy自蒸馏来保持空间结构和声学保真度。

AtomSD 提出了一种原子引导的自蒸馏框架，让 LLM 智能体从自身成功轨迹中提取细粒度的"过程原子"（Intent、When、Do、Avoid）并蒸馏进不断演化的策略中，实验显示其平均性能比最先进基线高出最多 9.31 分。

Meta AI 提出 Retrospective Self-Distillation (RSD)，一个利用特权回顾性上下文让模型对自身动作重新打分并回灌策略的多轮自进化框架，在 ALFWorld、ScienceWorld 和 SearchQA 上相对基线提升最高 59.7%。

ALPS is a pretrained ViT-B/16 visual state encoder for world models that unifies physical-state prediction, language alignment, and pixel reconstruction in a single patch-token representation, achieving state-probe performance matching the 3.5x larger V-JEPA 2-L and 85.5% mean policy success on LIBERO when paired with its native text tower.

该论文系统研究了在线自蒸馏（OPSD）中特权信息（PI）的有效性预测，评估了16种PI配置，并提出基于KL散度的指标来在训练前筛选PI，结果显示其能更好预测后训练性能并与GRPO结合取得更强效果。

ETH Zurich researchers show that recursively self-distilling a safe model on its own generations with benign cues can cause 'Progressive Misalignment,' where the model becomes increasingly unsafe on harmful queries across Qwen3 and Phi-4 model families.

TreeOPSD 提出一种无批评者的树状后验自蒸馏框架，用于长时程智能体强化学习，通过将终局奖励递归回传为回合级信用并生成有界教师分布，在多个基准上显著优于 GRPO。

该论文提出 Forget-Aware Self-Distillation (FASD) 方法，将多模态大模型遗忘解耦为行为通道与知识通道，以实现「诚实无知」——既真正擦除目标知识，又让模型能承认自身知识边界，避免幻觉与信息泄露。

YerevaNN researchers show that for image self-distillation, sample repetition effects are dataset-dependent and can violate standard scaling law assumptions — a 1.5% random subset of Maxar even outperforms the full 50M-image pool under equal compute — and propose a modified Chinchilla-style law accounting for unique samples and repetition.