Tag
提出了一种基于LLM的Teacher-Student弱监督框架，用于网络安全推文分类，在无需大规模人工标注的情况下训练的PLM模型F1超过0.92，接近完全人工监督的性能。
这篇论文从知识蒸馏视角重新审视on-policy distillation（OPD），通过受控token翻转实验测量教师在学生生成轨迹上的优势来源，发现基于mass和KL的选择性干预比随机翻转高出15-18个百分点，主张在保持分布稳定性的同时学习更有价值的教师纠正。
论文提出 BranchOPD（Branching On-Policy Distillation），通过在学生不确定或与教师分歧的决策点进行分支采样，并共享前缀以降低成本，同时提出协议边界修复（PBR）来缓解重复生成。在数学推理与 ALFWorld/WebShop 代理任务上，BranchOPD 显著优于 Vanilla OPD 和 Multi-Rollout OPD。
该论文研究了 on-policy distillation (OPD) 是否总是优于离线方法，提出 Semi-OPD——在初始学生的离线 rollout 上进行蒸馏。在 17 组师生模型对中，Semi-OPD 在 14 组中优于 OPD，最高提升 13.6% 准确率并实现 11.4 倍训练加速。
This work introduces Negative-Policy On-Policy Distillation (NP-OPD), which supplements a stronger teacher's token-level supervision with rollouts from a lower-capability negative policy, providing an explicit negative signal that improves OPD across model scales, generation modes, and reasoning domains. Code will be released at https://github.com/naver-ai/np-opd.
提出了 Slow-Fast 多教师在线蒸馏方法（SF-MOPD），通过将直接更新学生的 fast 模型与学生参数的 EMA slow 模型耦合，在吸收领域专家知识的同时抑制能力干扰，从而更好地保留多模态基础大模型的通用能力。
该论文研究提示数量、来源与选择如何影响在线策略蒸馏（OPD）的迁移效果，发现提示效用取决于师生模型对与目标任务，且定向选择策略（如语义多样性、难度、师生分歧等）并未一致优于均匀随机采样。
The paper identifies an off-policy asymmetry in on-policy distillation, where the teacher must supervise student-generated prefixes it was not trained on, and proposes SCOUT, a co-training framework that adapts the teacher via RL with verifiable rewards, consistently improving distillation across configurations, scales, and reasoning domains.
DuoOPD 论文提出一种新的多任务在线策略蒸馏方法，利用教师-学生的联合结果来决定反馈方向与教师支持方式，在 Qwen3 和 Llama 上相比标准 OPD 平均宏准确率提升 2.58 和 5.98 个百分点。
This paper studies the interaction between prompt breadth and rollout refresh in on-policy distillation for mathematical reasoning, revealing that prompt efficiency depends on both the refresh rate of student policies and the inference budget.
The paper introduces Calibrated On-Policy Distillation (Cal-OPD), a method that estimates the teacher's self-deviation to calibrate teacher-student discrepancies, improving on-policy distillation for mathematical reasoning tasks.
A theoretical study shows that in overparameterized ReLU networks, a positive-homogeneous scaling gauge hidden in the initial parameters can deterministically control which duplicate neuron learns a teacher feature, affecting specialization time and pruning trajectories.
Proposes Progressive2, a teacher-student progressive co-evolving knowledge distillation method for substantial model compression, addressing large disparity between server and client capabilities. Introduces progressive teacher layer selection and student size reduction with a multi-feature fusion adapter.
An educational blog post explaining how knowledge distillation works, covering the teacher-student framework, soft labels, temperature, and distillation loss, with real examples.
This paper tests the assumption that attention weights reveal what a model actually depends on for its output, finding that attention and causal dependence often disagree. They propose using causal evidence sets obtained via intervention masking as supervision for sparse attention routers, achieving near-perfect accuracy on retrieval tasks where attention-distilled routers fail.
AgentBrew proposes a training-free method for distilling knowledge from strong teacher LLM agents to weak student agents, using a failure-triggered reflection loop and student-aware synthesis to create executable notes, improving performance on coding, math, and tool-use tasks without weight updates.
This paper investigates the position bias phenomenon in online distillation, finding that early tokens provide more useful supervision signals, and proposes the importance-weighted IW-OPD method to improve OPD training.
An educational overview of knowledge distillation, covering its history, core concepts like softmax and temperature, types, scaling laws, and practical examples including DeepSeek-R1.
Sign-Gated On-Policy Distillation (SG-OPD) enhances standard on-policy distillation by using a binary verifier as a trust signal for teacher supervision, improving performance on competition-level math reasoning benchmarks.
Z-Reward is a teacher-student framework that decouples complex reasoning from efficient reward deployment for text-to-image training. It achieves 89.6% human preference accuracy with a 27B teacher and 88.6% with a 9B student, outperforming prior methods.