从求解轨迹到教学轨迹:认知对齐的推理蒸馏

arXiv cs.AI 论文

摘要

本文提出 CARD(认知对齐推理蒸馏),一种基于强化学习的推理蒸馏框架,将大型推理模型的求解轨迹转化为与学生模型认知能力对齐的教学轨迹,通过分布对齐奖励和逻辑连贯性奖励,在不重新采样的情况下提升蒸馏效果。

arXiv:2610.36445v1 Announce Type: new Abstract: Knowledge distillation from large reasoning models (LRMs) often uses raw teacher trajectories as supervision. However, this practice creates a mismatch between the goal of distillation and the nature of teacher trajectories. Advanced LRMs typically improve their reasoning capabilities through reward-driven post-training, such as RLHF. This training optimizes them as strong solvers, rather than as teachers that produce reasoning processes smaller models can easily learn. Since smaller models have different "cognitive capacities" compared to their larger teachers, directly imitating raw teacher trajectories can sometimes be ineffective and often requires a substantial number of high-quality samples. To bridge this mismatch, we propose Cognition-Aligned Reasoning Distillation (CARD), an RL-based framework that transforms solver trajectories into teaching trajectories aligned with the student's cognition. Avoiding expensive resampling, our framework directly transforms existing trajectories. We optimize CARD with two reward signals: (1) a Distribution Alignment Reward that matches the student's predictive distribution, and (2) a Logical Coherence Reward that reduces misleading failed reasoning steps. Comprehensive evaluations across multiple student models and settings demonstrate that CARD outperforms distillation from raw teacher trajectories in in-domain settings while achieving substantial gains in out-of-distribution performance. Code is available at https://github.com/LiteSSLHub/CARD.
查看原文
查看缓存全文

缓存时间: 2026/10/28 07:56

# From Solver Trajectories to Teaching Trajectories: Cognition-Aligned Reasoning Distillation
Source: [https://arxiv.org/abs/2610.36445](https://arxiv.org/abs/2610.36445)
[View PDF](https://arxiv.org/pdf/2610.36445)[HTML \(experimental\)](https://arxiv.org/html/2610.36445v1)

> Abstract:Knowledge distillation from large reasoning models \(LRMs\) often uses raw teacher trajectories as supervision\. However, this practice creates a mismatch between the goal of distillation and the nature of teacher trajectories\. Advanced LRMs typically improve their reasoning capabilities through reward\-driven post\-training, such as RLHF\. This training optimizes them as strong solvers, rather than as teachers that produce reasoning processes smaller models can easily learn\. Since smaller models have different "cognitive capacities" compared to their larger teachers, directly imitating raw teacher trajectories can sometimes be ineffective and often requires a substantial number of high\-quality samples\. To bridge this mismatch, we propose Cognition\-Aligned Reasoning Distillation \(CARD\), an RL\-based framework that transforms solver trajectories into teaching trajectories aligned with the student's cognition\. Avoiding expensive resampling, our framework directly transforms existing trajectories\. We optimize CARD with two reward signals: \(1\) a Distribution Alignment Reward that matches the student's predictive distribution, and \(2\) a Logical Coherence Reward that reduces misleading failed reasoning steps\. Comprehensive evaluations across multiple student models and settings demonstrate that CARD outperforms distillation from raw teacher trajectories in in\-domain settings while achieving substantial gains in out\-of\-distribution performance\. Code is available at[this https URL](https://github.com/LiteSSLHub/CARD)\.

## Submission history

From: Yashuo Luo \[[view email](https://arxiv.org/show-email/c87c87a6/2610.36445)\] **\[v1\]**Tue, 27 Oct 2026 00:46:23 UTC \(1,613 KB\)

相似文章

通过混合层蒸馏和关键信息的逐步注意力改进小模型的推理能力

arXiv cs.CL

本文提出一种新颖的思维链蒸馏框架,通过混合层模块的动态层对齐,将教师模型对关键信息的逐步注意力转移到学生模型中。该方法通过明确指导学生模型在推理过程中逐步聚焦关键信息,在数学和常识推理基准测试中实现了一致的性能提升。

学习如何修订推理:分段在线策略蒸馏

arXiv cs.AI

该论文提出 Seg-OPD(分段在线策略蒸馏),通过选择学生模型的不确定推理片段并用教师重写稿作为显式修订监督,训练学生学习如何修订推理步骤。实验显示在数学推理与竞赛编程任务上,相比基线平均相对提升 5.22% 的推理准确率。