Tag
AnoMix 提出一种多教师排序蒸馏方法（CoRank），将多个异构无监督时间序列异常检测器蒸馏为单一轻量学生模型（<80k 参数），在七个基准数据集上优于单个检测器和静态分数集成，其零样本变体还能在未见数据集上超越大型时间序列基础模型。
本文提出 CA-MOPD，通过合并强化学习专家并利用验证器估算的师生能力差距来调整多教师在线蒸馏的监督强度，在数学、代码生成和指令遵循六个基准上超越现有 MOPD 方法。
The paper introduces Latent-MOPD, the first representation-level multi-teacher on-policy distillation method for LLMs, which integrates multiple specialist teachers through both their output distributions and hidden states without extra teacher training, outperforming token-only, representation-only, and averaging baselines across nine math, code, and logic benchmarks.
This paper investigates multi-teacher on-policy distillation (MOPD) by analyzing how teacher signals translate into parameter changes when combining RL-trained teachers into Qwen3-1.7B and SmolLM3-3B students. It finds that loss averaging implicitly weights responses by length, Adam's first moment largely erases differences between teacher gradients (cosine similarity 0.83), BF16 rounding hides most weight updates, and top-64 intersection KL gradients can match full-vocabulary gradients depending on the averaging rule used.
PMOPD 提出一种几何感知的多教师在线策略蒸馏方法，通过构建任务子空间记忆并对梯度和优化器更新做投影来缓解能力跷跷板效应，并加入冲突探测与任务循环策略，在 Code/Reason/Math 任务上于 Qwen2.5-7B 和 Llama-3.1-8B 上分别平均提升 2.54 和 2.09 分。
Poly-OPD is a framework for distilling complementary strengths from heterogeneous text-to-image flow models into a single compact flow-matching student, using pixel bridges and gradient-compatible adapters. It improves GenEval and DrawBench scores while consolidating multiple teacher capabilities.
This paper introduces AMRD, an adaptive multi-teacher relational distillation method for compressing large self-supervised speech emotion recognition models into lightweight student models for edge devices. It addresses teacher reliability variation and relational structure loss, showing improvements on IEMOCAP and CREMA-D datasets.
This paper diagnoses and proposes SoftClamp, a calibration method that reduces tool-call boundary drift in multi-teacher on-policy distillation for agentic language models, decreasing over-calling while maintaining accuracy.
This paper introduces a compete-then-collaborate framework where multiple frontier AI teachers (Claude, Codex-GPT, Grok, Gemini) are ranked by execution-based tests and then collaborate to build a verifiable curriculum. It finds that imitation (SFT) on teacher solutions degrades a competent coder student, while using the same curriculum for reinforcement learning with verifiable rewards (RLVR) improves performance, particularly on competition problems.
An article surveying how frontier AI models in 2026 use distillation techniques, covering off-policy, on-policy, and self-distillation stages, with examples from Gemma, DeepSeek, GLM, Nemotron, and Qwen3.
CoRD is a collaborative multi-teacher decoding framework that synthesizes reasoning trajectories through predictive perplexity scoring and beam search, enabling efficient distillation of large reasoning models with high-quality outputs and generalized performance.