knowledge-distillation

The paper proposes Stepwise-Querying, a method that elicits step-specific explanations from black-box teachers without accessing their internal chain-of-thought traces, assembling them into training data for supervised fine-tuning of smaller student models. It achieves the top score in 14 of 16 comparisons across two teachers, two students, and four math/code benchmarks, and introduces the SEIG diagnostic for measuring stepwise explanation information gain.

CAMEL introduces a cross-architecture distillation framework with an asymmetric masking strategy that transfers knowledge from large retinal foundation models like RETFound to ultra-lightweight student models, preserving accuracy while drastically reducing computational cost for resource-limited clinical deployment.

This paper applies mechanistic interpretability tools—including linear CKA, logit lens, attention decomposition, sparse autoencoders, and activation patching—to compare GPT-2 with DistilGPT-2, revealing a compression-collapse dichotomy where the IOI circuit fully collapses while the Greater-Than circuit survives via layer remapping.

本文提出 CARD（Cognition-Aligned Reasoning Distillation），一种基于 RL 的推理蒸馏框架，将大推理模型的求解轨迹转化为与学生模型认知能力对齐的教学轨迹，通过分布对齐奖励和逻辑连贯性奖励在不重新采样的情况下提升蒸馏效果。

The paper proposes Teacher-Evolution Self-Distillation (TESD), which improves on-policy self-distillation by continuously optimizing the self-teacher with reward-guided objectives and adaptively down-weighting samples with unreliable teacher guidance, achieving consistent gains over token-level and sequence-level baselines across datasets and model scales.

该论文研究如何通过联合训练策略，让预训练语言模型同时支持单次前向和循环推理两种路径，在保留单次推理质量的同时获得循环计算带来的性能增益。

该论文通过受控实验研究了不同意见混合比例的后训练语料如何在SFT、DPO和GRPO等五种后训练路径中放大LLM的价值偏好失衡，并发现这些偏好可通过蒸馏传递给学生模型，建议在开发和评估LLM时应同时考虑语料意见构成与后训练方法的选择。

提出 Boa Compressor，一个基于 Mamba 状态空间模型的即插即用在线上下文压缩器，通过单次线性时间编码任意长度上下文，并以蒸馏方式训练以匹配完整上下文下的下一词元分布，在相同压缩率下匹敌或超越离线压缩基线且延迟降低一个数量级。

提出REDAR：将类增量学习中的正则化强度选择建模为序列决策问题，用轻量级RL智能体根据当前性能与数据统计动态输出每个阶段的知识蒸馏系数，在CIFAR-100和Tiny-ImageNet上降低了遗忘并提升了前向迁移。

This paper proposes a hierarchical agent distillation framework that separates on-policiness into two levels—the policy generating interaction histories and the one generating action prefixes—finding that student-induced histories with teacher-generated actions are the strongest signal, and that mixtures and scheduling matter beyond overall on-policy data amounts.

ThinkBit 是一个严格标量的两比特量化框架，通过分层比特分配、Gumbel-softmax 搜索码字、知识蒸馏重校准和量化感知训练来保留推理能力，在 Qwen3-1.7B 和 14B 模型上以远小于 bfloat16 的体积取得接近原模型的 MATH-500 成绩。

This paper shows that regenerating target model responses shrinks the gains that draft-training objectives like knowledge distillation provide in speculative decoding, replicating findings across vision-language (Qwen3-VL-8B) and text (Qwen3-4B) settings with multiple controls.

The paper introduces Common-Support Distillation (CSD), a unified logits-based framework for cross-tokenizer knowledge transfer that constructs a shared support between teacher and student and matches induced distributions with a single objective, outperforming existing cross-tokenizer distillation methods across multiple teacher-student settings.

AnoMix 提出一种多教师排序蒸馏方法（CoRank），将多个异构无监督时间序列异常检测器蒸馏为单一轻量学生模型（<80k 参数），在七个基准数据集上优于单个检测器和静态分数集成，其零样本变体还能在未见数据集上超越大型时间序列基础模型。

The paper introduces Ask-to-Distill (A2D), a framework that RL-trains a student-initialized synthesizer to generate synthetic questions that elicit dense teacher supervision for on-policy distillation, without needing reference answers. It improves performance and teacher-gap recovery over unoptimized prompts and real-question banks across math, code, and agentic tasks.

论文提出 projected distillation，通过将学生的失败尝试最小编辑为正确目标，实现近似 on-policy 的学习信号，在数学、代码、科学和工具使用等领域显著优于 GRPO 和 on-policy distillation 等基线。

本文提出 CA-MOPD，通过合并强化学习专家并利用验证器估算的师生能力差距来调整多教师在线蒸馏的监督强度，在数学、代码生成和指令遵循六个基准上超越现有 MOPD 方法。

TraceForge 提出一种两阶段（监督+对抗）方法，从教师答案中生成匹配学生模型自身风格的推理轨迹，无需访问隐藏思维链，在 HumanEval+ 和 MBPP+ 等编码基准上显著超越使用教师原始思考内容的训练效果。

Ask-to-Distill 提出了一种优化合成问题的框架，通过教师-学生分布差异作为代理信号，在无需参考答案的情况下提升 on-policy 蒸馏效果，在数学推理、代码生成和智能体任务上优于未优化的问题集和真实问题库。

ECHOES 提出一种通过递归历史压缩与特权在线蒸馏来增强长时程 GUI 智能体状态保持能力的方法，无需在部署时保留历史截图即可取得有竞争力的性能，并发布了细粒度诊断基准 ECHOESBMK。