How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
Summary
This paper introduces TESSY, a teacher-student cooperative framework for fine-tuning reasoning models that generates on-policy SFT data by decoupling generation into capability tokens (from teacher) and style tokens (from student), addressing catastrophic forgetting issues when using off-policy teacher data.
View Cached Full Text
Cached at: 04/20/26, 08:28 AM
Paper page - How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
Source: https://huggingface.co/papers/2604.14164
https://huggingface.co/papers/2604.14164#%F0%9F%9A%80-motivation🚀 Motivation
Training reasoning models (e.g., Qwen3) is highly sensitive to the data distribution. We observe that:
❗ Using off‑policy data (e.g., directly from a strong teacher model) for SFT can lead tosevere catastrophic forgetting, especially for complex reasoning tasks.
https://huggingface.co/papers/2604.14164#%F0%9F%92%A1-key-idea💡 Key Idea
To address this critical issue, we proposeTESSY, a novelTeacher–Student Cooperative Data Synthesis frameworkdesigned to generateon‑policytraining data. Instead of relying on a teacher model to fully generate training samples, TESSYdecouples the generation process into two distinct parts:
- 🧠Teacher model→ specializes in generatingcapability tokens.
- ✍️Student model→ focuses on generatingstyle tokens(e.g., Hmm, Wait...).
This cooperative approach ensures:
- Alignment with student distribution (on‑policy): The synthesized data is tailored to the student model’s own generation patterns.
- Preservation of teacher reasoning quality: The teacher’s advanced reasoning capabilities are effectively leveraged and maintained.
Similar Articles
Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models
This paper studies transferring lessons about supervised fine-tuning (SFT) across alignment training, model organisms, and toy models, showing that techniques like training on reasons for behavior and mixing on-model data can improve generalization and capability preservation.
Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information
This paper proposes a novel Chain-of-Thought distillation framework that transfers teacher models' stepwise attention on key information to student models through a Mixture-of-Layers module for dynamic layer alignment. The method achieves consistent performance improvements on mathematical and commonsense reasoning benchmarks by explicitly guiding student models to progressively focus on critical information during reasoning.
Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning
The paper presents a method to distill reasoning into compact video-language models using synthetic chain-of-thought rationales and difficulty-aware fine-tuning, enabling smaller models to outperform larger ones with minimal compute.
Probing the Origins of Reasoning Performance: Representational Quality for Mathematical Problem-Solving in RL vs. SFT Fine-Tuned Models
This paper investigates internal representational differences between RL and SFT fine-tuned models on mathematical reasoning, finding that RL models exhibit more linearly separable hidden states and hierarchical layer importance. Token allocation variability under repeated sampling suggests training pipeline dependence rather than RL vs SFT alone.
Think Multilingual, Not Harder: A Data-Efficient Framework for Teaching Reasoning Models to Code-Switch
This paper introduces a data-efficient fine-tuning framework for teaching reasoning models to code-switch (mix languages) effectively, demonstrating that strategic code-switching can improve reasoning capabilities for lower-resource languages. The work analyzes code-switching behaviors in large language models across diverse languages, tasks, and domains, then develops interventions to promote beneficial code-switching patterns.