从求解轨迹到教学轨迹:认知对齐的推理蒸馏
摘要
本文提出 CARD(认知对齐推理蒸馏),一种基于强化学习的推理蒸馏框架,将大型推理模型的求解轨迹转化为与学生模型认知能力对齐的教学轨迹,通过分布对齐奖励和逻辑连贯性奖励,在不重新采样的情况下提升蒸馏效果。
arXiv:2610.36445v1 Announce Type: new
Abstract: Knowledge distillation from large reasoning models (LRMs) often uses raw teacher trajectories as supervision. However, this practice creates a mismatch between the goal of distillation and the nature of teacher trajectories. Advanced LRMs typically improve their reasoning capabilities through reward-driven post-training, such as RLHF. This training optimizes them as strong solvers, rather than as teachers that produce reasoning processes smaller models can easily learn. Since smaller models have different "cognitive capacities" compared to their larger teachers, directly imitating raw teacher trajectories can sometimes be ineffective and often requires a substantial number of high-quality samples. To bridge this mismatch, we propose Cognition-Aligned Reasoning Distillation (CARD), an RL-based framework that transforms solver trajectories into teaching trajectories aligned with the student's cognition. Avoiding expensive resampling, our framework directly transforms existing trajectories. We optimize CARD with two reward signals: (1) a Distribution Alignment Reward that matches the student's predictive distribution, and (2) a Logical Coherence Reward that reduces misleading failed reasoning steps. Comprehensive evaluations across multiple student models and settings demonstrate that CARD outperforms distillation from raw teacher trajectories in in-domain settings while achieving substantial gains in out-of-distribution performance. Code is available at https://github.com/LiteSSLHub/CARD.
查看缓存全文
缓存时间: 2026/10/28 07:56
# From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation Source: [https://arxiv.org/abs/2610.36445](https://arxiv.org/abs/2610.36445) [View PDF](https://arxiv.org/pdf/2610.36445)[HTML \(experimental\)](https://arxiv.org/html/2610.36445v1) > Abstract:Knowledge distillation from large reasoning models \(LRMs\) often uses raw teacher trajectories as supervision\. However, this practice creates a mismatch between the goal of distillation and the nature of teacher trajectories\. Advanced LRMs typically improve their reasoning capabilities through reward\-driven post\-training, such as RLHF\. This training optimizes them as strong solvers, rather than as teachers that produce reasoning processes smaller models can easily learn\. Since smaller models have different "cognitive capacities" compared to their larger teachers, directly imitating raw teacher trajectories can sometimes be ineffective and often requires a substantial number of high\-quality samples\. To bridge this mismatch, we propose Cognition\-Aligned Reasoning Distillation \(CARD\), an RL\-based framework that transforms solver trajectories into teaching trajectories aligned with the student's cognition\. Avoiding expensive resampling, our framework directly transforms existing trajectories\. We optimize CARD with two reward signals: \(1\) a Distribution Alignment Reward that matches the student's predictive distribution, and \(2\) a Logical Coherence Reward that reduces misleading failed reasoning steps\. Comprehensive evaluations across multiple student models and settings demonstrate that CARD outperforms distillation from raw teacher trajectories in in\-domain settings while achieving substantial gains in out\-of\-distribution performance\. Code is available at[this https URL](https://github.com/LiteSSLHub/CARD)\. ## Submission history From: Yashuo Luo \[[view email](https://arxiv.org/show-email/c87c87a6/2610.36445)\] **\[v1\]**Tue, 27 Oct 2026 00:46:23 UTC \(1,613 KB\)
相似文章
Residual Advantage:面向可验证奖励强化学习的学生相对教师指导
本文提出 Residual Advantage(RA)与 Co-RA,将师生概率之间的残差作为有界的单步奖励并进行居中处理,用于 RLVR 后训练阶段;在数学推理基准上,其性能一致优于 GRPO、REINFORCE++ 以及仅使用教师蒸馏的方法。
Waypoint-OPD:通过门控同策略蒸馏迁移强化学习诱导的策略偏移
Waypoint-OPD 提出一种门控在线策略蒸馏方法,通过比较教师模型强化学习训练早期与晚期策略变化的方向一致性来过滤端点策略偏移,从而在数学与代码推理任务上提升学生模型的蒸馏效果,且仅需额外一次冻结教师评分。
通过混合层蒸馏和关键信息的逐步注意力改进小模型的推理能力
本文提出一种新颖的思维链蒸馏框架,通过混合层模块的动态层对齐,将教师模型对关键信息的逐步注意力转移到学生模型中。该方法通过明确指导学生模型在推理过程中逐步聚焦关键信息,在数学和常识推理基准测试中实现了一致的性能提升。
学习如何修订推理:分段在线策略蒸馏
该论文提出 Seg-OPD(分段在线策略蒸馏),通过选择学生模型的不确定推理片段并用教师重写稿作为显式修订监督,训练学生学习如何修订推理步骤。实验显示在数学推理与竞赛编程任务上,相比基线平均相对提升 5.22% 的推理准确率。
LARK:基于可学习性的轨迹选择方法用于高效推理蒸馏
LARK提出了一种基于可学习性的推理轨迹选择方法,用于大语言模型蒸馏。该方法采用可学习性因子和χ²正则化选择策略,平衡效率与泛化能力,在多个模型和任务上持续优于基线方法。