RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

# RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
Source: [https://arxiv.org/html/2606.00147](https://arxiv.org/html/2606.00147)
Yuduo Li1,2∗§, Xiaofeng Shi1,2∗†, Qian Kou1, Longbin Yu1, Hua Zhou1‡ 1Beijing Academy of Artificial Intelligence \(BAAI\) 2Beijing Jiaotong University \(BJTU\)

###### Abstract

Domain\-specific supervised fine\-tuning \(SFT\) often improves in\-domain performance at the cost of degrading a model’s general capabilities\. We view this degradation through two practical gaps in domain SFT: a supervision\-compatibility gap, where domain targets differ in style and reasoning format from the original model’s natural responses, and a trajectory\-preservation gap, where teacher\-forced SFT optimizes fixed target tokens without constraining the model’s behavior on its own generated prefixes\. This process fails to preserve the model’s original behavior\. We propose RAFT \(DataRefinement andAdaptive Distillation for DomainFine\-Tuning with Alleviated Forgetting\), a two\-stage framework that addresses both factors\. First, RAFT constructs model\-compatible supervision through self\-conditioned rewriting, semantic filtering, and answer fusion\. Second, RAFT performs Answer\-Conditioned On\-Policy Distillation, where the original instruction\-tuned model provides soft targets on student\-generated trajectories while being conditioned on the fused answer as helpful context\. We further introduce top\-K temperature distillation and EMA\-based adaptive loss balancing to stabilize the domain\-general trade\-off\. Across three instruction\-tuned backbones and five domains, RAFT improves average domain accuracy by 23\.2% over standard SFT, while recovering part of the SFT\-induced degradation on MS\-Bench and IFEval, with relative improvements of 18\.2% and 10\.2%, respectively\. These results show that coupling data refinement with trajectory\-level preservation provides an effective recipe for domain fine\-tuning with alleviated forgetting\.

††∗Equal contribution\.††§Work done during internship at BAAI\.†††Corresponding author\. Email:xfshi@baai\.ac\.cn††‡Project leader\.## 1Introduction

Large language models \(LLMs\) acquire broad capabilities through pre\-training and instruction tuning\(Zhanget al\.,[2026](https://arxiv.org/html/2606.00147#bib.bib28); Grattafioriet al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib13); Aboueleninet al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib15)\), yet domain\-specific supervised fine\-tuning \(SFT\) often causes*catastrophic forgetting*\(Luoet al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib29); Kothaet al\.,[2023](https://arxiv.org/html/2606.00147#bib.bib30)\)—general capabilities degrade substantially\. We focus on two practical factors that make domain SFT prone to forgetting: \(1\)*distribution mismatch*between ground\-truth supervision and the model’s own output distribution, forcing abrupt parameter shifts that overwrite pre\-trained knowledge\(Yanget al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib26)\); and \(2\) the absence of explicit mechanism to*preserve general capabilities*during optimization, which focuses exclusively on domain loss\.

Many existing methods primarily emphasize one side of this problem\. Self\-distillation \(SDFT\)\(Yanget al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib26)\)rewrites supervision to reduce distribution mismatch but lacks training\-level regularization\. On\-policy distillation\(Lu and Lab,[2025](https://arxiv.org/html/2606.00147#bib.bib10)\)introduces KL constraints on student trajectories, but heavily relies on the teacher model’s quality, making it sensitive to teacher\-bias\. Replay\-based methods\(Luet al\.,[2026](https://arxiv.org/html/2606.00147#bib.bib31); Luoet al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib29)\)mix general\-domain data but are sensitive to mixing ratios\. Regularization approaches such as L2SP\(Xuhonget al\.,[2018](https://arxiv.org/html/2606.00147#bib.bib33)\)and EWC\(Kirkpatricket al\.,[2017](https://arxiv.org/html/2606.00147#bib.bib32)\)add parameter\-level constraints that ignore functional behavior and are costly for LLMs\. Forgetting\-aware pruning\(Huanget al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib6)\)provides structural regularization alone\. In short, existing methods usually focus on either data quality or training constraints, but rarely both\. While self\-distillation and KL divergence are well\-established, how to effectively combine data refinement with training\-level regularization remains a gap in domain SFT\.

The key challenge is that these two aspects are coupled rather than independent\. Data refinement improves the reference answers used for cross\-entropy training, but does not constrain the model’s behavior on its own generated trajectories\. Conversely, on\-policy distillation regularizes student trajectories, but without target\-aware supervision it can act as a conservative anchor that limits domain adaptation\. RAFT addresses this coupling by using the fused answer both as model\-compatible SFT supervision and as helpful context for the teacher during on\-policy distillation\.

As illustrated in Figure[1](https://arxiv.org/html/2606.00147#S1.F1), RAFT combines data optimization and training optimization through offline distillation and adaptive on\-policy distillation\. In*offline distillation*, we first rewrite the ground\-truth answers conditioned on the model’s own distribution, and use cosine\-similarity filtering to determine whether to adopt the rewritten answers\. The selected data is combined with the original samples and input into a stronger fusion model to generate refined responses\. In*adaptive on\-policy distillation*, we use the fused dataset to train the model with explicit general capability preservation\. At each training step, the model first generates trajectories via autoregressive sampling\. The original instruction\-tuned model \(teacher\) then provides soft supervision through Answer\-Conditioned On\-Policy Distillation, where it is conditioned on the answer in the Fused Dataset as additional context for more informative guidance\. We further introduce a top\-KKtemperature distillation mechanism to focus KL divergence on the most informative tokens, which prevents over\-smoothing and improves output diversity\. In addition, we propose an EMA\-based adaptive balancing strategy that dynamically adjusts the SFT–distillation weight, eliminating the need for manual hyperparameter tuning\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/result_v8.png)Figure 1:Overview of theRAFTframework\.Left: Offline Distillationgenerates higher\-quality fused data by combining the model’s rewritten answers \(filtered by cosine similarity\) with the original data through a stronger fusion model\.Right: Adaptive on\-policy Distillationtrains the model with Answer\-Conditioned On\-Policy Distillation, distillation over softened probability distributions, and EMA\-based adaptive loss balancing\.We evaluate RAFT on three models \(SmolLM3\-3B, Llama\-3\.2\-3B\-Instruct, Phi\-4\-mini\-instruct\) across five domains\. Our contributions are:

- •We identify and target two practical factors associated with forgetting in domain SFT: supervision distribution mismatch and the lack of explicit preservation constraints\.
- •We introduce an offline distillation stage that uses cosine similarity filtering to select distribution\-consistent rewritten answers and fuses them with original data to improve response quality\.
- •We propose an adaptive on\-policy distillation method that combines Answer\-Conditioned On\-Policy Distillation, top\-KKtemperature distillation mechanism, and EMA\-based loss balancing\.
- •RAFT recovers part of SFT\-induced degradation, as measured by MS\-Bench \(18\.2%\) and IFEval \(10\.2%\), while also improving domain accuracy \(D\-Acc\) by 23\.2%\.

## 2Related Work

##### Mitigating forgetting of general capabilities\.

Fine\-tuning LLMs for domain\-specific tasks often leads to catastrophic forgetting of their general capabilities\(Qiet al\.,[2023](https://arxiv.org/html/2606.00147#bib.bib7)\)\. Existing mitigation methods can be broadly grouped into three categories\. \(1\) replay\-based methods\(Rolnicket al\.,[2019](https://arxiv.org/html/2606.00147#bib.bib1); Huanget al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib2)\)alleviate forgetting by mixing general\-domain data or replay samples with task\-specific training data, so that the model continues to rehearse its pretrained knowledge during adaptation\. \(2\) regularization\-based methods\(Chenet al\.,[2020](https://arxiv.org/html/2606.00147#bib.bib3); Zixuanet al\.,[2023](https://arxiv.org/html/2606.00147#bib.bib4)\)reduce destructive drift from the pretrained model by constraining parameter updates or output distributions\. \(3\) parameter\-space adjustment methods restrict adaptation to a limited subset of parameters or carefully modify model weights, for example through parameter\-efficient tuning such as LoRA\(Huet al\.,[2022](https://arxiv.org/html/2606.00147#bib.bib5)\), partial parameter freezing\(Huanget al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib6)\), thereby preserving more of the model’s original general abilities\.

##### On\-policy distillation\.

Online distillation aligns the student with teacher signals on trajectories generated by the current student policy, rather than relying on a fixed offline dataset\(Shenfeldet al\.,[2026](https://arxiv.org/html/2606.00147#bib.bib8)\)\. By matching supervision to the model’s own trajectories during training, it has shown strong potential for preserving general capabilities\(Hübotteret al\.,[2026](https://arxiv.org/html/2606.00147#bib.bib9); Lu and Lab,[2025](https://arxiv.org/html/2606.00147#bib.bib10)\)\. However, pure online distillation does not fully leverage the labeled target answers provided in standard supervised datasets\(Agarwalet al\.,[2023](https://arxiv.org/html/2606.00147#bib.bib11)\)\. In contrast, our approach combines SFT supervision with on\-policy distillation, allowing the model to learn domain\-specific answers accurately while alleviating catastrophic forgetting\.A conceptual comparison with these strategies is provided in Appendix[B](https://arxiv.org/html/2606.00147#A2)\.

## 3Methodology

### 3\.1Overview

We propose a unified framework for mitigating catastrophic forgetting in domain\-specific supervised fine\-tuning \(SFT\)\. As discussed in Section[1](https://arxiv.org/html/2606.00147#S1), catastrophic forgetting arises from both data stage and optimization stage causes\. Accordingly, our framework intervenes at two stages: In the data stage, constructing higher\-quality training data through offline distillation \(Section[3\.2](https://arxiv.org/html/2606.00147#S3.SS2)\); In the optimization stage, adding distillation constraints through adaptive on\-policy distillation \(Section[3\.3](https://arxiv.org/html/2606.00147#S3.SS3)\)\.

Given a domain dataset𝒟=\{\(xi,yi\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, wherexix\_\{i\}is the input instruction andyiy\_\{i\}is the ground truth answer\. Letθ\\thetadenote the parameters of the model being fine\-tuned\. Standard SFT optimizesθ\\thetaonly on domain ground truth answers, which improves domain performance but causes the model to deviate from the knowledge and behavioral patterns acquired during pre\-training and alignment\.

### 3\.2Offline Distillation

A direct cause of catastrophic forgetting is the distribution mismatch between the ground truth answers and the model’s output distribution before fine\-tuning: forcing the model to fit answers far from its natural expression style leads to abrupt parameter shifts that overwrite pre\-trained knowledge\. To address this, we construct training data that preserves factual content while adapting the expression style to the original instruction\-tuned model, proceeding through three stages: self\-conditioned rewriting, semantic filtering, and multi\-expert optimal selection over fused responses\.

Self\-Conditioned Rewriting\.For each sample\(xi,yi\)\(x\_\{i\},y\_\{i\}\), the original instruction\-tuned model generates a rewrite conditioned on the original answer:

y~i∼pθ\(⋅∣xi,yi\)\\tilde\{y\}\_\{i\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\_\{i\},y\_\{i\}\)\(1\)wherepθp\_\{\\theta\}denotes the output distribution of the original instruction\-tuned model parameterized byθ\\theta\. Conditioning onyiy\_\{i\}preserves key factual content while adapting the expression style to that of the original instruction\-tuned model’s own output distribution, following the self\-distillation paradigm\(Yanget al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib26)\)\. However, self\-rewriting without quality control can introduce semantic drift, where the rewritten answer deviates from the original meaning\.

Semantic Filtering\.To prevent semantic drift from self\-rewriting, we compute the cosine similarity between the original answer and the rewritten answer in a shared embedding space:

si=ϕ​\(yi\)⊤​ϕ​\(y~i\)‖ϕ​\(yi\)‖⋅‖ϕ​\(y~i\)‖s\_\{i\}=\\frac\{\\phi\(y\_\{i\}\)^\{\\top\}\\phi\(\\tilde\{y\}\_\{i\}\)\}\{\\\|\\phi\(y\_\{i\}\)\\\|\\cdot\\\|\\phi\(\\tilde\{y\}\_\{i\}\)\\\|\}\(2\)whereϕ:𝒱∗→ℝd\\phi:\\mathcal\{V\}^\{\*\}\\to\\mathbb\{R\}^\{d\}is a pre\-trained sentence encoder that maps text sequences todd\-dimensional embeddings\. Candidate rewrites are filtered through a threshold:

y¯i=\{y~i,si≥τyi,si<τ\\bar\{y\}\_\{i\}=\\begin\{cases\}\\tilde\{y\}\_\{i\},&s\_\{i\}\\geq\\tau\\\\ y\_\{i\},&s\_\{i\}<\\tau\\end\{cases\}\(3\)whereτ\\tauis the similarity threshold \(see Appendix[E\.1](https://arxiv.org/html/2606.00147#A5.SS1)for detailed analysis\)\. When the rewrite is semantically consistent with the original answer \(si≥τs\_\{i\}\\geq\\tau\),y¯i\\bar\{y\}\_\{i\}is retained for subsequent fusion withyiy\_\{i\}; otherwise, the original answeryiy\_\{i\}is used directly, preventing error accumulation in self\-reinforcement\. While semantic filtering ensures per\-sample quality, individual model outputs still exhibit variance due to the stochastic nature of generation—a single fusion attempt may not yield the best possible result\.

Multi\-expert optimal selection over fused responses\.Considering the stochastic nature of model outputs, we perform multiple rounds of fusion to improve robustness and output quality\. Specifically, for samples that pass semantic filtering \(si≥τs\_\{i\}\\geq\\tau\), we fuseyiy\_\{i\}andy¯i\\bar\{y\}\_\{i\}through a fusion model \(Qwen3\-3B\(Team,[2025](https://arxiv.org/html/2606.00147#bib.bib23)\)\)fmergef\_\{\\text\{merge\}\}to generate a higher\-quality answery^i\\hat\{y\}\_\{i\}\(see Appendix[C](https://arxiv.org/html/2606.00147#A3)for the prompt template\)\. To ensure the quality of the fused output, we employ multiple judge models to independently evaluate the consistency between the fused result and the original answer\. We perform multiple rounds of fusion and scoring, and select the result with the highest average evaluation score as the finaly^i\\hat\{y\}\_\{i\}\. Details of the evaluation are provided in Appendix[D](https://arxiv.org/html/2606.00147#A4)\. Finally, we construct the dataset:

𝒟^=\{\(xi,y^i\)\}i:si≥τ∪\{\(xi,yi\)\}i:si<τ\\hat\{\\mathcal\{D\}\}=\\\{\(x\_\{i\},\\hat\{y\}\_\{i\}\)\\\}\_\{i:s\_\{i\}\\geq\\tau\}\\cup\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i:s\_\{i\}<\\tau\}\(4\)which preserves Factual integrity while adapting the expression style to the instruction\-tuned model’s distribution, ensuring output quality through the multi\-model scoring mechanism\.

### 3\.3Adaptive On\-Policy Distillation

Although the fused data𝒟^\\hat\{\\mathcal\{D\}\}alleviates distribution mismatch in the offline distillation stage, SFT alone still suffers from catastrophic forgetting: optimizing solely on domain data causes the model to gradually lose its general capabilities\. To mitigate this fundamental limitation, we adopt the original instruction\-tuned model itself as the teacher and perform online distillation, ensuring that while the student model learns domain\-specific knowledge, it is also constrained to preserve the behavioral patterns of the original model\.

SFT\.We optimize the standard cross\-entropy loss on𝒟^\\hat\{\\mathcal\{D\}\}\. Let𝒜i\\mathcal\{A\}\_\{i\}denote the set of answer token positions in theii\-th sample, andy^i,t\\hat\{y\}\_\{i,t\}denote thett\-th token of the fused answery^i\\hat\{y\}\_\{i\}:

ℒSFT=−1N​∑i=1N1\|𝒜i\|​∑t∈𝒜ilog⁡pθ​\(y^i,t∣xi,y^i,<t\)\\mathcal\{L\}\_\{\\text\{SFT\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{1\}\{\|\\mathcal\{A\}\_\{i\}\|\}\\sum\_\{t\\in\\mathcal\{A\}\_\{i\}\}\\log p\_\{\\theta\}\(\\hat\{y\}\_\{i,t\}\\mid x\_\{i\},\\hat\{y\}\_\{i,<t\}\)\(5\)This loss injects the fused domain knowledge intoθ\\theta\. However, this loss alone provides no mechanism to preserve the student model’s general capabilities\. To address this, we introduce distillation constraints that constrain the student model’s behavior on its own generated trajectories\.

Online Policy Trajectory Generation\.For eachxix\_\{i\}, the student model generates a token sequence as a response trajectory through autoregressive sampling according to the current policy:

𝐳i=\(zi,1,…,zi,Li\)∼pθ\(⋅∣xi\)\\mathbf\{z\}\_\{i\}=\(z\_\{i,1\},\\ldots,z\_\{i,L\_\{i\}\}\)\\sim p\_\{\\theta\}\(\\cdot\\mid x\_\{i\}\)\(6\)where eachzi,t∈𝒱z\_\{i,t\}\\in\\mathcal\{V\}is the token sampled by the student model at steptt, andLiL\_\{i\}is the trajectory length\. Unlike imposing constraints on reference answer trajectories, online policy distillation ensures that the constraints cover the state space that the student model would access during actual inference\. Having generated the student’s trajectories, we next need to define how the teacher provides supervision on these trajectories\.

Answer\-Conditioned On\-Policy Distillation\.Since the student’s trajectories may deviate from what the original instruction\-tuned teacher would naturally produce, the teacher model is given the fused answer as additional context so that it can provide more informative guidance at each step\. The student model predicts the next token conditioned only onxix\_\{i\}, while the teacher model uses the fused answer as additional context\. At positionttof trajectory𝐳i\\mathbf\{z\}\_\{i\}:

pθS\(⋅∣xi,zi,<t\),pψT\(⋅∣r,y^i,xi,zi,<t\)p\_\{\\theta\}^\{S\}\(\\cdot\\mid x\_\{i\},z\_\{i,<t\}\),\\qquad p\_\{\\psi\}^\{T\}\(\\cdot\\mid r,\\hat\{y\}\_\{i\},x\_\{i\},z\_\{i,<t\}\)\(7\)whererris the reference prompt, and the teacher model takes as input the sequential concatenation ofrr,y^i\\hat\{y\}\_\{i\},xix\_\{i\}, andzi,<tz\_\{i,<t\}\. Both models predict on the same trajectory prefixzi,<tz\_\{i,<t\}, but the teacher can provide more informative soft targets at each step of the student’s trajectory by conditioning ony^i\\hat\{y\}\_\{i\}\. See Appendix[H](https://arxiv.org/html/2606.00147#A8)for the teacher model’s concatenation template\. With the teacher providing soft targets on the student’s own trajectories, we next define how the student’s distribution is guided toward the teacher’s distribution while maintaining output diversity\.

Top\-K Temperature Distillation\.To increase the diversity of model outputs and prevent the model from concentrating all probability mass on a single token, we apply temperature scaling with top\-KKselection to smooth the output distributions\. Let𝐮i,t,𝐯i,t∈ℝ\|𝒱\|\\mathbf\{u\}\_\{i,t\},\\mathbf\{v\}\_\{i,t\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}denote the logits of the teacher and student models at position\(i,t\)\(i,t\)respectively, andT\>0T\>0be the temperature parameter\. The temperature\-scaled distributions over vocabulary𝒱\\mathcal\{V\}are:

qi,t​\(a\)=exp⁡\(ui,t​\(a\)/T\)∑b∈𝒱exp⁡\(ui,t​\(b\)/T\),pi,t\(T\)​\(a\)=exp⁡\(vi,t​\(a\)/T\)∑b∈𝒱exp⁡\(vi,t​\(b\)/T\)q\_\{i,t\}\(a\)=\\frac\{\\exp\(u\_\{i,t\}\(a\)/T\)\}\{\\sum\_\{b\\in\\mathcal\{V\}\}\\exp\(u\_\{i,t\}\(b\)/T\)\},\\quad p\_\{i,t\}^\{\(T\)\}\(a\)=\\frac\{\\exp\(v\_\{i,t\}\(a\)/T\)\}\{\\sum\_\{b\\in\\mathcal\{V\}\}\\exp\(v\_\{i,t\}\(b\)/T\)\}\(8\)We select the top\-KKtokens with highest probability from the teacher distribution to form the set𝒦i,t=TopK​\(qi,t,K\)\\mathcal\{K\}\_\{i,t\}=\\text\{TopK\}\(q\_\{i,t\},K\), and restrict both distributions to this set with renormalization:

q~i,t​\(a\)=qi,t​\(a\)​1​\[a∈𝒦i,t\]∑b∈𝒦i,tqi,t​\(b\),p~i,t​\(a\)=pi,t\(T\)​\(a\)​1​\[a∈𝒦i,t\]∑b∈𝒦i,tpi,t\(T\)​\(b\)\\tilde\{q\}\_\{i,t\}\(a\)=\\frac\{q\_\{i,t\}\(a\)\\,\\mathbf\{1\}\[a\\in\\mathcal\{K\}\_\{i,t\}\]\}\{\\sum\_\{b\\in\\mathcal\{K\}\_\{i,t\}\}q\_\{i,t\}\(b\)\},\\quad\\tilde\{p\}\_\{i,t\}\(a\)=\\frac\{p\_\{i,t\}^\{\(T\)\}\(a\)\\,\\mathbf\{1\}\[a\\in\\mathcal\{K\}\_\{i,t\}\]\}\{\\sum\_\{b\\in\\mathcal\{K\}\_\{i,t\}\}p\_\{i,t\}^\{\(T\)\}\(b\)\}\(9\)This focuses distillation on theKKtokens that the teacher considers most important while reducing the computational cost of full\-vocabulary KL, and the temperature scaling ensures output diversity by smoothing the distribution rather than collapsing it to a single token\. The online policy distillation loss is:

ℒKL=T2N​∑i=1N1Li​∑t=1LiKL​\(q~i,t∥p~i,t\)\\mathcal\{L\}\_\{\\text\{KL\}\}=\\frac\{T^\{2\}\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{1\}\{L\_\{i\}\}\\sum\_\{t=1\}^\{L\_\{i\}\}\\text\{KL\}\(\\tilde\{q\}\_\{i,t\}\\\|\\tilde\{p\}\_\{i,t\}\)\(10\)whereT2T^\{2\}is the standard compensation factor for temperature scaling, used to compensate for the gradient magnitude reduction after temperature transformation\. Having defined both the SFT loss and the distillation loss, we now face the challenge of balancing them: sinceℒSFT\\mathcal\{L\}\_\{\\text\{SFT\}\}uses cross\-entropy andℒKL\\mathcal\{L\}\_\{\\text\{KL\}\}uses KL divergence, their magnitudes differ in scale and vary dynamically during training\.

EMA\-based Adaptive Balancing\.We want the two losses to remain stable and equally important throughout training, rather than having their relative contributions shift unpredictably as the data and model change\. To this end, we introduce an exponential moving average \(EMA\)\-based adaptive balancing mechanism\. At training stepss, with decay coefficientβ∈\(0,1\)\\beta\\in\(0,1\):

mk\(s\)=β​mk\(s−1\)\+\(1−β\)​ℒk\(s\),m^k\(s\)=mk\(s\)1−βs,k∈\{SFT,KL\}m\_\{k\}^\{\(s\)\}=\\beta\\,m\_\{k\}^\{\(s\-1\)\}\+\(1\-\\beta\)\\,\\mathcal\{L\}\_\{k\}^\{\(s\)\},\\quad\\hat\{m\}\_\{k\}^\{\(s\)\}=\\frac\{m\_\{k\}^\{\(s\)\}\}\{1\-\\beta^\{s\}\},\\quad k\\in\\\{\\text\{SFT\},\\text\{KL\}\\\}\(11\)wheremk\(s\)m\_\{k\}^\{\(s\)\}is the EMA estimate of the loss, andm^k\(s\)\\hat\{m\}\_\{k\}^\{\(s\)\}is bias\-corrected by dividing by1−βs1\-\\beta^\{s\}to address early underestimation caused by zero initialization\. The adaptive weight and final objective are:

λ\(s\)=m^SFT\(s\)m^KL\(s\)\+ϵ,ℒ\(s\)=ℒSFT\(s\)\+λ\(s\)​ℒKL\(s\)\\lambda^\{\(s\)\}=\\frac\{\\hat\{m\}\_\{\\text\{SFT\}\}^\{\(s\)\}\}\{\\hat\{m\}\_\{\\text\{KL\}\}^\{\(s\)\}\+\\epsilon\},\\qquad\\mathcal\{L\}^\{\(s\)\}=\\mathcal\{L\}\_\{\\text\{SFT\}\}^\{\(s\)\}\+\\lambda^\{\(s\)\}\\,\\mathcal\{L\}\_\{\\text\{KL\}\}^\{\(s\)\}\(12\)whereϵ\>0\\epsilon\>0prevents the denominator from becoming too small\. WhenℒKL\\mathcal\{L\}\_\{\\text\{KL\}\}is relatively small compared toℒSFT\\mathcal\{L\}\_\{\\text\{SFT\}\},λ\(s\)\\lambda^\{\(s\)\}increases to strengthen the distillation constraint; conversely, thus maintaining a dynamic balance between losses without manual tuning\.

Overall, RAFT couples the two stages through the fused answer\. The fused answer serves as a model\-compatible target for the SFT loss and as answer context for the teacher in on\-policy distillation\. This design turns the teacher from a purely conservative anchor into a target\-aware guide, allowing the student to acquire domain knowledge while remaining close to the behavioral patterns of the original instruction\-tuned model\.

## 4Experiments

### 4\.1Experimental Setup

##### Models\.

We evaluate RAFT on three open\-source instruction\-tuned language models: SmolLM3\-3B\(Bakouchet al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib14)\), Llama\-3\.2\-3B\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib13)\), and Phi\-4\-mini\-instruct\(Aboueleninet al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib15)\)\. SmolLM3\-3B is a compact yet capable model designed for efficient deployment, Llama\-3\.2\-3B\-Instruct is the instruction\-tuned variant of Meta’s Llama\-3\.2 series optimized for dialogue and reasoning, and Phi\-4\-mini\-instruct is Microsoft’s lightweight instruction\-following model that achieves strong performance relative to its size\. Evaluating across these diverse architectures allows us to verify the generalizability of our approach\.

##### Domain Data\.

We construct high\-quality domain\-specific QA pairs by extracting question\-answer pairs from the content of CCL4\.0\(Liuet al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib20)\)and Common Corpus\(Langlaiset al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib16)\)using large language models, followed by a multi\-model filtering process for quality assurance\. The resulting dataset spans five domains: Business and Industry\(B&I\), Law and Government\(L&G\), Open Culture, Open Science, and Open Web, with approximately 2,000 QA pairs per domain on average\.

##### Evaluation\.

We assess model performance from three complementary perspectives\. For domain capability, we measure domain accuracy \(D\-Acc\) on the held\-out test set of each domain\-specific QA dataset, evaluated via majority voting among three expert models \(gpt\-oss\-120b\(OpenAI,[2025](https://arxiv.org/html/2606.00147#bib.bib21)\), DeepSeek\-v3\.2\(DeepSeek\-AI,[2025](https://arxiv.org/html/2606.00147#bib.bib22)\), and Qwen3\-32B\(Team,[2025](https://arxiv.org/html/2606.00147#bib.bib23)\)\)\. For general capability, we adopt two benchmarks covering different aspects: MS\-Bench\(Liet al\.,[2023](https://arxiv.org/html/2606.00147#bib.bib12)\), a subjective question\-answering benchmark also evaluated via multi\-expert majority voting; and IFEval\(Zhouet al\.,[2023](https://arxiv.org/html/2606.00147#bib.bib19)\), an instruction\-following format evaluation\. Additionally, we evaluate objective reasoning capability using MMLU\(Hendryckset al\.,[2021b](https://arxiv.org/html/2606.00147#bib.bib17),[a](https://arxiv.org/html/2606.00147#bib.bib18)\), and find that all methods exhibit only minor fluctuations without significant changes \(see Appendix[F](https://arxiv.org/html/2606.00147#A6)for details\)\.

##### Training Details\.

We fine\-tune all models for 3 epochs with a batch size of 16 and a learning rate of5×10−65\\times 10^\{\-6\}\. We use the AdamW optimizer with a weight decay of 0\.01 and BF16 precision\. In our Adaptive on\-policy Distillation, we use a temperatureT=1\.5T=1\.5and Top\-K=512K=512for KL\-based distillation\. Further details are provided in Appendix[E\.2](https://arxiv.org/html/2606.00147#A5.SS2)and Section[E\.3](https://arxiv.org/html/2606.00147#A5.SS3)\. Following Adam\(Kingma and Ba,[2014](https://arxiv.org/html/2606.00147#bib.bib24)\), the EMA decay coefficientβ\\betais set to 0\.9\. All experiments are conducted on NVIDIA A100 GPUs\. The primary experiments required approximately 1,000 GPU hours, with the total computational time \(including preliminary trials\) not exceeding 2,000 hours\.

##### Baselines\.

We compare RAFT against four baselines, all using their recommended hyperparameters\.SFTdenotes standard supervised fine\-tuning on the original domain data\.SDFT\(Yanget al\.,[2024](https://arxiv.org/html/2606.00147#bib.bib26)\)is self\-distillation fine\-tuning, which rewrites supervision targets using the student model itself to improve compatibility\.on\-policy\(Lu and Lab,[2025](https://arxiv.org/html/2606.00147#bib.bib10)\)applies online on\-policy distillation, directly computing KL divergence between teacher and student on student\-generated trajectories\.FAPM\(Huanget al\.,[2025](https://arxiv.org/html/2606.00147#bib.bib6)\)mitigates forgetting through forgetting\-aware pruning that selectively removes parameters while preserving knowledge critical for previously learned tasks\.

### 4\.2Main Results

Table 1:Main results across five domains on three backbone models\. All values are percentages\.D\-Acc: domain accuracy on domain\-specific datasets\. Best results among fine\-tuning methods arebolded\. TheAvgrow reports the mean across all five domains for each metric\. In the Avg row, the relative improvement over SFT is shown in parentheses\.As shown in Table[1](https://arxiv.org/html/2606.00147#S4.T1), RAFT achieves the strongest average trade\-off across the three backbone models and five domains\. It does not dominate every individual model\-domain setting, but it consistently improves the averaged D\-Acc over standard SFT and recovers a substantial portion of the SFT\-induced degradation on MS\-Bench and IFEval\. Averaging the backbone\-level relative improvements, RAFT improves D\-Acc by 23\.2%, MS\-Bench by 18\.2%, and IFEval by 10\.2% over SFT\. We also report retention rates relative to the base model in Appendix[G](https://arxiv.org/html/2606.00147#A7)\. RAFT improves average MS\-Bench retention from 76\.8% to 89\.2% and IFEval retention from 91\.2% to 99\.5% compared with SFT, supporting that its gains come with better preservation of general capabilities\.

Among the baselines, each exhibits clear limitations\. FAPM maintains IFEval performance on SmolLM3\-3B \(31\.2%, on par with the base model\), but achieves only marginal domain gains \(D\-Acc of 10\.7%, even below SFT’s 12\.8%\), indicating that pruning\-based regularization alone is insufficient for effective domain adaptation\. SDFT and on\-policy distillation show mixed performance: while they occasionally outperform SFT on specific metrics \(e\.g\., SDFT on Phi\-4\-mini achieves \+1\.8% D\-Acc and \+6\.7% MS\-Bench\), such improvements are not consistent across models; for instance, both methods degrade D\-Acc on Llama\-3\.2\-3B\-Instruct\. These contrasting failure modes reveal that addressing only one dimension, training stage constraints or data refinement is insufficient\. RAFT selects the most model\-compatible samples to refine training data and adds distillation constraints to the objective\. This approach achieves a stronger average trade\-off between domain learning and forgetting prevention\.

### 4\.3Ablation Studies

We conduct controlled ablation studies to validate the key components of RAFT\. Specifically, fused supervision is shown to provide higher\-quality and more informative training signals compared to original supervision\. Adaptive on\-policy distillation plays a key role in improving general capability preservation during domain adaptation\. Furthermore, EMA\-based adaptive loss balancing is essential for dynamically resolving the trade\-off between domain learning and general capability retention that fixed coefficients cannot address\.

#### 4\.3\.1Data Ablation: Effect of Fused Supervision

To evaluate the contribution of our data fusion strategy, we compare standard SFT using original domain data versus using fused data\. Table[2](https://arxiv.org/html/2606.00147#S4.T2)presents the results on SmolLM3\-3B across three representative domains\.

Table 2:Ablation study on data fusion and adaptive on\-policy distillation on SmolLM3\-3B across three domains\. All values are percentages\. Best results among all settings arebolded\.Fused data consistently outperforms original data across all three domains\. The domain accuracy improves by 7\.0% on Law & Government, 20\.5% on Open Science, and 13\.0% on Open Culture, while MS\-Bench also improves by 4\.5%, 3\.2%, and 6\.4% respectively\. IFEval improves by 12\.7% on Law & Government and 15\.7% on Open Science, and remains unchanged on Open Culture\. These results demonstrate that fusing the original and self\-distilled answers via a strong teacher produces more informative and better\-aligned supervision signals, which help reduce distribution shift and mitigate catastrophic forgetting\.

#### 4\.3\.2Training Objective Ablation: Effect of Adaptive On\-Policy Distillation

To evaluate Adaptive On\-Policy Distillation, we compare standard SFT and the full RAFT method\. Both settings use the same fused data, ensuring that any performance difference can be attributed to the adaptive on\-policy distillation objective\. Table[2](https://arxiv.org/html/2606.00147#S4.T2)presents the results on SmolLM3\-3B\.

The adaptive on\-policy distillation objective provides substantial gains over fused\-data SFT alone\. Domain accuracy improves by 31\.4% on Law & Government, 33\.0% on Open Science, and 33\.1% on Open Culture\. MS\-Bench improves modestly by 0\.7%, 1\.6%, and 11\.1% respectively, while IFEval improves by 8\.5%, 20\.2%, and 18\.2%\. These results confirm that aligning the student with the teacher on its own generated trajectories, combined with EMA\-based adaptive weighting, effectively constrains the training process and further mitigates forgetting while enhancing domain performance\. The consistent improvements across all domains demonstrate that the training objective is complementary to the data fusion strategy\.

#### 4\.3\.3Impact of Adaptive Loss Balancing

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/ema_loss_plot_v7_new.png)\(a\)CE/KL loss ratio curves under different balancing strategies on SmolLM3\-3B in the Open Culture domain\. “Fixed” denotes using a constant KL coefficient \(λ=0\.1\\lambda=0\.1orλ=1\.0\\lambda=1\.0\), while “Ours” denotes the EMA\-based adaptive scheme\.
![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/EMA_bar_accuracy_plot.png)\(b\)Domain accuracy \(Open Culture datasets\) and general capability \(MS\-Bench\) of SmolLM3\-3B under different balancing strategies\. Standard SFT,λ=0\.1\\lambda=0\.1,λ=1\.0\\lambda=1\.0, and EMA\-based adaptive scheme \(Ours\) are compared\.

Figure 2:Effect of different balancing strategies on SmolLM3\-3B in the Open Culture domain\.A key design choice in RAFT is the EMA\-based adaptive balancing mechanism that dynamically adjusts the relative weightλ\(s\)\\lambda^\{\(s\)\}betweenℒSFT\\mathcal\{L\}\_\{\\text\{SFT\}\}andℒKL\\mathcal\{L\}\_\{\\text\{KL\}\}during training\. We compare our adaptive scheme against fixed KL coefficientsλ∈\{0\.1,1\.0\}\\lambda\\in\\\{0\.1,1\.0\\\}and standard SFT \(no distillation\) on SmolLM3\-3B in the Open Culture domain\.

##### Training stability\.

Figure[2\(a\)](https://arxiv.org/html/2606.00147#S4.F2.sf1)shows the CE/KL loss ratio curves under different balancing strategies\. With both fixed coefficients \(λ=0\.1\\lambda=0\.1andλ=1\.0\\lambda=1\.0\), the ratio fluctuates sharply, indicating that the relative weight between CE and KL losses is highly unstable\. In contrast, the EMA\-based adaptive scheme maintains a stable ratio throughout training, indicating that the CE loss and KL loss are well\-balanced and the optimization proceeds smoothly\.

##### Domain and general trade\-off\.

Figure[2\(b\)](https://arxiv.org/html/2606.00147#S4.F2.sf2)presents the domain accuracy and general capability under different balancing strategies\. With a fixed coefficient, there exists an inherent trade\-off: increasingλ\\lambdastrengthens regularization and better preserves general capability \(MS\-Bench rises from 57\.5% atλ=0\.1\\lambda=0\.1to 64\.5% atλ=1\.0\\lambda=1\.0\), but simultaneously suppresses domain adaptation \(D\-Acc drops from 16\.7% to 15\.0%\)\. Standard SFT without distillation achieves a D\-Acc of 14\.7% and MS\-Bench of 58\.5%, suffering from both limited domain adaptation and general capability degradation\. No single fixed coefficient can simultaneously optimize both objectives\. The adaptive scheme resolves this dilemma\. It dynamically adjusts the weight based on the relative magnitudes of the two losses\. It balances the distillation constraint by approximately equalizing the scalar magnitudes of the two loss terms, ensuringλ​ℒK​L\\lambda\\mathcal\{L\}\_\{KL\}remains comparable toℒS​F​T\\mathcal\{L\}\_\{SFT\}\. This yields the best domain accuracy of 19\.7% and the best MS\-Bench score of 65\.0%, outperforming all fixed\-coefficient baselines on domain adaptation and achieving a favorable balance on general capability preservation\.

These results demonstrate that EMA\-based adaptive balancing is crucial for RAFT: it eliminates the need for manual tuning of the distillation coefficient, stabilizes training, and enables a dynamic trade\-off that fixed coefficients cannot achieve\.

## 5Conclusions

We propose RAFT, a two\-stage framework for domain\-specific fine\-tuning with alleviated forgetting\. RAFT couples model\-compatible data refinement with answer\-conditioned trajectory\-level preservation: the offline stage constructs higher\-quality fused supervision, while the adaptive on\-policy distillation stage regularizes student\-generated trajectories using the original instruction\-tuned model as a target\-aware teacher\. Across three backbone models and five domains, RAFT achieves a stronger averaged trade\-off than standard SFT and the evaluated forgetting\-mitigation baselines\. Averaged across all settings, RAFT improves D\-Acc by 23\.2% over standard SFT, while improving MS\-Bench and IFEval by 18\.2% and 10\.2%, respectively\. These results suggest that coupling compatible supervision with trajectory\-level preservation provides a practical recipe for domain adaptation with better general capability retention\.

## 6Limitations

This work has two primary limitations\. Firstly, due to computational constraints, our evaluation of RAFT is focused on small\-scale language models\. While RAFT demonstrates consistent improvements across multiple architectures, its scalability to larger models warrants further investigation\. Secondly, the diversity of training data and domains is relatively constrained\. Although we evaluated RAFT across five distinct domains, its performance in specialized tasks such as code generation, multilingual contexts, or significantly larger training corpora remains to be explored\.

## References

- A\. Abouelenin, A\. Ashfaq, A\. Atkinson, H\. Awadalla, N\. Bach, J\. Bao, A\. Benhaim, M\. Cai, V\. Chaudhary, C\. Chen,et al\.\(2025\)Phi\-4\-mini technical report: compact yet powerful multimodal language models via mixture\-of\-loras\.arXiv preprint arXiv:2503\.01743\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px1.p1.1)\.
- Gkd: generalized knowledge distillation for auto\-regressive sequence models\.arXiv preprint arXiv:2306\.1364912\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Bakouch, L\. Ben Allal, A\. Lozhkov, N\. Tazi, L\. Tunstall, C\. M\. Patiño, E\. Beeching, A\. Roucher, A\. J\. Reedi, Q\. Gallouédec, K\. Rasul, N\. Habib, C\. Fourrier, H\. Kydlicek, G\. Penedo, H\. Larcher, M\. Morlon, V\. Srivastav, J\. Lochner, X\. Nguyen, C\. Raffel, L\. von Werra, and T\. Wolf \(2025\)SmolLM3: smol, multilingual, long\-context reasoner\.Note:[https://huggingface\.co/blog/smollm3](https://huggingface.co/blog/smollm3)Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px1.p1.1)\.
- S\. Chen, Y\. Hou, Y\. Cui, W\. Che, T\. Liu, and X\. Yu \(2020\)Recall and learn: fine\-tuning deep pretrained language models with less forgetting\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7870–7881\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-v3\.2: pushing the frontier of open large language models\.Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Critch, J\. Li, D\. Song, and J\. Steinhardt \(2021a\)Aligning ai with shared human values\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021b\)Measuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Huang, L\. Cui, A\. Wang, C\. Yang, X\. Liao, L\. Song, J\. Yao, and J\. Su \(2024\)Mitigating catastrophic forgetting in large language models with self\-synthesized rehearsal\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1416–1428\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1)\.
- W\. Huang, A\. Cheng, and Y\. Wang \(2025\)Mitigating catastrophic forgetting in large language models with forgetting\-aware pruning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 21853–21867\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p2.1),[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px5.p1.1)\.
- J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. K\. Buening, C\. Guestrin,et al\.\(2026\)Reinforcement learning via self\-distillation\.arXiv preprint arXiv:2601\.20802\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px2.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px4.p1.4)\.
- J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the national academy of sciences114\(13\),pp\. 3521–3526\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p2.1)\.
- S\. Kotha, J\. M\. Springer, and A\. Raghunathan \(2023\)Understanding catastrophic forgetting in language models via implicit inference\.arXiv preprint arXiv:2309\.10105\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p1.1)\.
- P\. Langlais, C\. R\. Hinostroza, M\. Nee, C\. Arnett, P\. Chizhov, E\. K\. Jones, I\. Girard, D\. Mach, A\. Stasenko, and I\. P\. Yamshchikov \(2025\)Common corpus: the largest collection of ethical data for llm pre\-training\.arXiv preprint arXiv:2506\.01732\.Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px2.p1.1)\.
- C\. Li, H\. Chen, M\. Yan, W\. Shen, H\. Xu, Z\. Wu, Z\. Zhang, W\. Zhou, Y\. Chen, C\. Cheng,et al\.\(2023\)Modelscope\-agent: building your customizable agent system with open\-source large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 566–578\.Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- G\. Liu, L\. Wang, J\. Li, Y\. Yu, Y\. Xu, J\. Chen, Y\. Bai, F\. Liao, and Y\. Lin \(2025\)CCI4\.0: a bilingual pretraining dataset for enhancing reasoning in large language models\.External Links:2506\.07463,[Link](https://arxiv.org/abs/2506.07463)Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px2.p1.1)\.
- K\. Lu and T\. M\. Lab \(2025\)On\-policy distillation\.Thinking Machines Lab: Connectionism\.Note:https://thinkingmachines\.ai/blog/on\-policy\-distillationExternal Links:[Document](https://dx.doi.org/10.64434/tml.20251026)Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p2.1),[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px5.p1.1)\.
- Y\. Lu, Y\. He, J\. Chen, and H\. Zha \(2026\)MSSR: memory\-aware adaptive replay for continual llm fine\-tuning\.arXiv preprint arXiv:2603\.09892\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p2.1)\.
- Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang \(2025\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.IEEE Transactions on Audio, Speech and Language Processing\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p1.1),[§1](https://arxiv.org/html/2606.00147#S1.p2.1)\.
- OpenAI \(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- X\. Qi, Y\. Zeng, T\. Xie, P\. Chen, R\. Jia, P\. Mittal, and P\. Henderson \(2023\)Fine\-tuning aligned language models compromises safety\.Even When Users Do Not Intend To\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Rolnick, A\. Ahuja, J\. Schwarz, T\. Lillicrap, and G\. Wayne \(2019\)Experience replay for continual learning\.Advances in neural information processing systems32\.Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1)\.
- I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. Agrawal \(2026\)Self\-distillation enables continual learning\.External Links:2601\.19897,[Link](https://arxiv.org/abs/2601.19897)Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Team \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3\.2](https://arxiv.org/html/2606.00147#S3.SS2.p4.6),[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- L\. Xuhong, Y\. Grandvalet, and F\. Davoine \(2018\)Explicit inductive bias for transfer learning with convolutional networks\.InInternational conference on machine learning,pp\. 2825–2834\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p2.1)\.
- Z\. Yang, T\. Pang, H\. Feng, H\. Wang, W\. Chen, M\. Zhu, and Q\. Liu \(2024\)Self\-distillation bridges distribution gap in language model fine\-tuning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1028–1043\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p1.1),[§1](https://arxiv.org/html/2606.00147#S1.p2.1),[§3\.2](https://arxiv.org/html/2606.00147#S3.SS2.p2.4),[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px5.p1.1)\.
- S\. Zhang, L\. Dong, X\. Li, S\. Zhang, X\. Sun, S\. Wang, J\. Li, R\. Hu, T\. Zhang, G\. Wang,et al\.\(2026\)Instruction tuning for large language models: a survey\.ACM Computing Surveys58\(7\),pp\. 1–36\.Cited by:[§1](https://arxiv.org/html/2606.00147#S1.p1.1)\.
- J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. Hou \(2023\)Instruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§4\.1](https://arxiv.org/html/2606.00147#S4.SS1.SSS0.Px3.p1.1)\.
- K\. Zixuan, S\. Yijia, L\. Haowei, K\. Tatsuya, K\. Gyuhak, L\. Bing,et al\.\(2023\)Continual pre\-training of language models\.InProceedings of The Eleventh International Conference on Learning Representations \(ICLR\-2023\),Cited by:[§2](https://arxiv.org/html/2606.00147#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AModel Similarity Analysis

To further quantify the degree of catastrophic forgetting at the parameter level, we measure theℓ2\\ell\_\{2\}distance between the fine\-tuned model weights and the original instruction\-tuned checkpoint\. A smaller distance indicates that the fine\-tuned model remains closer to the original model, suggesting that less pre\-trained knowledge has been overwritten during fine\-tuning\. Figure[3](https://arxiv.org/html/2606.00147#A1.F3)shows the distribution of relativeℓ2\\ell\_\{2\}distances for standard SFT and RAFT on SmolLM3\-3B\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/weight_l2_distance_distribution_SFT_RAFT_old.png)Figure 3:Distribution of relativeℓ2\\ell\_\{2\}weight distances from the pre\-trained SmolLM3\-3B model\. Blue: standard SFT \(mean = 0\.0524\)\. Red: RAFT \(mean = 0\.0458\)\. A smaller distance indicates less deviation from the pre\-trained model and thus less catastrophic forgetting\. RAFT produces weights that are consistently closer to the original model across all layers\.RAFT achieves a meanℓ2\\ell\_\{2\}distance of 0\.0458 compared to 0\.0524 for standard SFT, a 12\.6% relative reduction\. This provides parameter\-level evidence that RAFT produces more conservative updates than standard SFT\. Although weight distance isn’t the only factor in forgetting, this smaller deviation aligns with the better performance seen on MS\-Bench and IFEval\. The reduced weight deviation is consistent with RAFT’s better preservation of general capabilities observed in the main results, further corroborating the connection between parameter\-level conservation and functional\-level forgetting prevention\.

## Appendix BComparison with Related Fine\-Tuning Strategies

Table[3](https://arxiv.org/html/2606.00147#A2.T3)summarizes how RAFT differs from related forgetting\-mitigation strategies\. RAFT is designed to couple model\-compatible supervision with trajectory\-level preservation\. In contrast, data\-only methods improve the reference answers but do not constrain student\-generated trajectories, while optimization\-only methods regularize trajectories or parameters without explicitly improving domain supervision\.

Table 3:Conceptual comparison between RAFT and related domain fine\-tuning strategies\.
## Appendix CResponse Fusion Prompt Template

To enhance the quality of model\-generated responses, we use a response fusion strategy\. It combines an original answer and a model\-regenerated answer into a single higher\-quality response\. The fusion is guided by a carefully designed prompt\. It integrates complementary information from both responses while removing redundancy and preserving factual accuracy and logical coherence\.

The complete prompt template used for response fusion is presented below:

Youareaprofessionalknowledgeassistant\.Belowisauserquestionalongwithtwodifferentresponses\(Response1istheoriginalanswer,andResponse2isaregeneratedanswerbasedontheoriginal\)\.Pleaseintegratethestrengthsandknowledgefromboth,andgenerateahigher\-qualitynewresponse\.

Requirements:

\-Synthesizeinformationfrombothresponsestomakethecontentmorecomplete,accurate,andclearlyexpressed;

\-Avoidrepetitionandredundancy;

\-Preservefactualcorrectness,professionalism,andlogicalcoherence;

\-Outputonlythebestfusedansweryoucanproduce;

\-Strictlyoutputtheansweronly,withoutanyadditionalcontent\.

\-\-\-

Question:

\{query\}

Response1\(Original\):

\{answer1\}

Response2\(Model\-generated\):

\{answer2\}

Fusedfinalresponse:

## Appendix DQuality Scoring Prompt

In the fusion stage, we use a single expert to generate the fused output\. We then employ multiple judge models to independently evaluate its consistency with the original answer\. The detailed prompt used for quality scoring is as follows:

### D\.1Prompt Template

Youareastrict,fair,andhallucination\-awareautomaticscoringassistant\.

TaskDescription:

\-YouwillreceiveaQuestion,aReferenceAnswer,andaModelAnswer\.

\-YourtaskistodeterminewhethertheModelAnswercorrectlycoversthecorecontentoftheReferenceAnswer,andtocheckifthemodelexhibitshallucination\(i\.e\.,fabricatingcontentthatisirrelevant,contradictory,orinvalidwithrespecttothequestionorreferenceanswer\)\.

\-Ifthemodelcontainsadditionalinformation,aslongasitissemanticallyconsistentwiththequestionandreferenceanswerwithoutconflict,nopointsshouldbededucted\.

\-Ifthereisfabricationoffacts,logicalerrors,ordeviationfromthetopic,itshouldbeconsideredhallucinationandresultinalowerscore\.

ScoringCriteria:

\-1\.0:Completelycorrect\.Themodelansweraccuratelyexpressesthecoreinformationofthereferenceanswerwithnohallucination\.

\-0\.8\-0\.9:Mostlycorrect\.Theexpressionishighlyconsistentwiththereferenceanswerwithonlyminorwordingdifferences,andnoobvioushallucination\.

\-0\.6\-0\.7:Partiallycorrect\.Althoughsomecorecontentismentioned,thereareomissionsorminorinaccuracies,withoccasionalquestionableextensions\.

\-0\.3\-0\.5:Weaklyrelated\.Missingkeycontent,orcontainingobviouslyirrelevant/invaliddescriptions\(hallucination\)\.

\-0\.0\-0\.2:Incorrectorirrelevant\.Failstoexpressthemaincontentofthereferenceanswer,orcontainsseverehallucination,contradictions,orfabricatedinformation\.

\-\-\-

Question:

\{user\_question\}

ReferenceAnswer:

\{original\_answer\}

ModelAnswer:

\{fused\_answer\}

Pleaseonlyoutputascorefrom0to1

### D\.2Aggregation Strategy

For each fused output, we collect scores fromMMindependent judge models and compute the average scorec¯\\bar\{c\}\. We perform multiple rounds of fusion and scoring, and select the result with the highest average score as the final output\. Moreover, to reduce computational overhead, we adopt a threshold of0\.80\.8: if the average scorec¯≥0\.8\\bar\{c\}\\geq 0\.8, the fused output is considered to be of sufficiently high quality and is adopted immediately without further generation rounds, thereby reducing unnecessary resource consumption\. This multi\-model scoring mechanism ensures robust quality selection while tolerating individual judge variability\.

## Appendix EParametric Analysis

Considering the impact of different parameter values on RAFT, here we have conducted a detailed analysis of the relevant parameters\.

### E\.1Analysis of Cosine Similarity Thresholdτ\\tau

In the semantic filtering stage of fusion supervision construction \(Section[3\.2](https://arxiv.org/html/2606.00147#S3.SS2)\), the cosine similarity thresholdτ\\taucontrols the trade\-off between adopting the student model’s self\-rewritten answer and falling back to the original ground truth answer\. A lowerτ\\tauaccepts more rewrites, which better adapts the expression style to the student model but risks introducing semantic drift; a higherτ\\tauis more conservative, preserving factual accuracy but potentially missing opportunities to reduce distribution mismatch\. We analyze the effect ofτ∈\{0,0\.3,0\.5,0\.8,1\.0\}\\tau\\in\\\{0,0\.3,0\.5,0\.8,1\.0\\\}on SmolLM3\-3B in the Open Culture domain, where both domain accuracy and general capability are evaluated\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/cos_accuracy_plot.png)Figure 4:Effect of cosine similarity thresholdτ\\tauon domain accuracy \(D\-Acc\) on Open Culture and general capability \(MS\-Bench\) on SmolLM3\-3B\. Both metrics peak atτ=0\.8\\tau=0\.8, indicating that a moderately strict filtering threshold best balances semantic fidelity and distribution adaptation\.As shown in Figure[4](https://arxiv.org/html/2606.00147#A5.F4), both domain accuracy and general capability exhibit a non\-monotonic relationship withτ\\tau\. At the two extremes, the results are suboptimal for different reasons\. Whenτ=0\\tau=0, all rewrites are accepted without filtering, allowing semantic drift to corrupt the training data—this yields a D\-Acc of 13\.3% and an MS\-Bench of 60\.5%\. Whenτ=1\.0\\tau=1\.0, the threshold is so strict that virtually no rewrite can satisfy the condition \(si≥1\.0s\_\{i\}\\geq 1\.0requires exact identity\),this similarly yields a D\-Acc of 13\.1% and an MS\-Bench of 55\.0%\.

At intermediate thresholds, both metrics improve asτ\\tauincreases, reaching their joint optimum atτ=0\.8\\tau=0\.8with a D\-Acc of 19\.7% and an MS\-Bench of 65\.0%\. This result suggests that a moderately strict threshold effectively filters out semantically drifted rewrites while retaining those that successfully adapt the expression style to the student model’s distribution\. The improved training data quality, in turn, benefits both domain adaptation and general capability preservation\. The consistent improvement across both metrics atτ=0\.8\\tau=0\.8confirms that semantic filtering plays a critical role in the fusion supervision pipeline: it is not merely a safeguard against errors, but an active mechanism that enhances the overall quality of the training data\. We therefore adoptτ=0\.8\\tau=0\.8as the default throughout all experiments\.

### E\.2Temperature Ablation

The temperatureTTcontrols the smoothness of the teacher distribution in KL divergence computation\. We ablateTTfrom 1\.0 to 2\.0 on SmolLM3\-3B in the Business & Industry domain\. As shown in Figure[5](https://arxiv.org/html/2606.00147#A5.F5), D\-Acc and MS\-Bench exhibit different sensitivities toTT: D\-Acc rises steadily withTT\(from 17\.3% atT=1\.0T=1\.0to 22\.2% atT=1\.8T=1\.8\), while MS\-Bench peaks atT=1\.5T=1\.5\(71\.5%\) and declines beyond\. Excessively lowTTyields sharp teacher distributions with limited supervision, while excessively highTTover\-smooths and dilutes discriminative information\. We selectT=1\.5T=1\.5as the default, achieving the best MS\-Bench \(71\.5%\) with competitive D\-Acc \(17\.3%\)\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/T_ablation_v2.png)Figure 5:Effect of temperatureTTon D\-Acc and MS\-Bench on SmolLM3\-3B \(Business & Industry\)\. The shaded region indicates the recommended rangeT∈\[1\.5,1\.8\]T\\in\[1\.5,1\.8\]\.
### E\.3Detailed Analysis of Top\-KKSelection

In our distillation framework, the Top\-KKparameter restricts the KL divergence computation to theKKtokens with the highest teacher probabilities, serving a dual purpose: reducing the computational cost of full\-vocabulary KL and focusing distillation on the most informative tokens\. We analyze the effect of Top\-KKfrom two perspectives—probability coverage versus computational cost, and downstream task performance—to justify our choice ofK=512K=512\.

##### Probability Coverage and Computational Cost\.

Figure[6](https://arxiv.org/html/2606.00147#A5.F6)presents the probability coverage of the teacher distribution and the relative GPU time overhead under different Top\-KKvalues\. The teacher’s probability distribution is highly concentrated: Top\-256 already covers 99\.2% of the probability mass, and Top\-512 achieves 99\.6% coverage\. Beyond 512, the marginal coverage gain diminishes rapidly \(99\.8% at 1024, 99\.9% at 2048\), while GPU time continues to grow\. This indicates that the vast majority of the teacher’s knowledge is captured within the top 512 tokens, and increasingKKfurther yields negligible information gain at disproportionately higher computational cost\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/top_k_v4_bar_gpu_growth.png)Figure 6:Probability coverage and relative GPU time growth under different Top\-KKvalues\. The bars show the cumulative probability coverage of the teacher distribution captured by the Top\-KKtokens; the line shows the relative GPU time increase compared toK=32K=32\. AsKKincreases beyond 512, the marginal coverage gain diminishes rapidly while computational cost continues to grow\.
##### Effect of Top\-KKon Task Performance\.

To examine how Top\-KKaffects both domain adaptation and general capability preservation, we conduct experiments on SmolLM3\-3B in the Open Culture domain withK∈\{64,128,256,512,1024,2048\}K\\in\\\{64,128,256,512,1024,2048\\\}\. Figure[7](https://arxiv.org/html/2606.00147#A5.F7)shows the domain accuracy \(D\-Acc\) and general capability \(MS\-Bench\) under differentKKvalues\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/topk_accuracy_plot_v1.png)Figure 7:Domain accuracy \(D\-Acc\) on Open Culture and general capability \(MS\-Bench\) under different Top\-KKvalues on SmolLM3\-3B\. D\-Acc peaks atK=512K=512while MS\-Bench remains relatively stable across allKKvalues, indicating that the choice ofKKprimarily affects domain adaptation quality rather than general capability preservation\.As shown in Figure[7](https://arxiv.org/html/2606.00147#A5.F7), the two metrics exhibit markedly different sensitivities toKK\. MS\-Bench remains relatively stable across allKKvalues, fluctuating within a narrow range of 64\.5%–67\.5%, which suggests that general capability preservation is largely determined by the presence of the distillation constraint itself rather than the specific number of tokens involved\. In contrast, domain accuracy shows a clear non\-monotonic trend: D\-Acc stays at 13\.3% forK≤256K\\leq 256, rises sharply to 19\.7% atK=512K=512, and then drops to 11\.7% atK=1024K=1024before partially recovering to 13\.6% atK=2048K=2048\.

We attribute this non\-monotonic behavior to the trade\-off between information sufficiency and noise\. WhenKKis too small \(e\.g\.,K≤256K\\leq 256\), the restricted token set fails to capture enough of the teacher’s distribution to provide effective guidance for domain adaptation, resulting in suboptimal D\-Acc\. AtK=512K=512, the distilled knowledge is both sufficiently comprehensive and sufficiently focused, yielding the best domain performance\. WhenKKbecomes too large \(e\.g\.,K=1024K=1024\), low\-probability tokens that carry little discriminative information are included in the KL computation, which can introduce noise and dilute the distillation signal, leading to degraded D\-Acc\. The partial recovery atK=2048K=2048suggests that with a sufficiently largeKK, the full\-vocabulary KL approaches a stable regime where the noise effect is averaged out, though at significantly higher computational cost\.

##### Summary\.

Combining both analyses,K=512K=512represents the optimal operating point: it captures 99\.6% of the teacher’s probability mass with acceptable computational overhead, and achieves the highest domain accuracy while maintaining stable general capability\. We therefore adoptK=512K=512as the default throughout all experiments\.

## Appendix FMMLU Results

We further examine the impact of domain adaptation on objective reasoning through the MMLU benchmark on SmolLM3\-3B\. As shown in Figure[8](https://arxiv.org/html/2606.00147#A6.F8), we compare the MMLU accuracy of the model, standard SFT, and RAFT across five domains\. All three methods exhibit comparable MMLU performance, with values ranging from 50\.3% to 54\.3% across domains\. The differences among the base model, SFT, and RAFT are marginal \(within 2 percentage points in each domain\), indicating that domain adaptation via RAFT does not compromise objective reasoning capability\. RAFT achieves the highest or near\-highest scores on four out of five domains \(54\.0% on B&I, 54\.0% on L&G, 54\.3% on Sci, and 54\.3% on Web\), closely matching the base model’s 52\.7% and SFT’s 50\.3%–54\.0%\. These results demonstrate that RAFT effectively preserves objective reasoning while achieving substantial domain adaptation gains\.

![Refer to caption](https://arxiv.org/html/2606.00147v1/Ablation_image/MMLU_SmolLM3_5domains.png)Figure 8:MMLU accuracy of the base model, SFT, and RAFT across five domains on SmolLM3\-3B\. All methods show comparable MMLU performance, indicating that RAFT preserves objective reasoning capability during domain adaptation\.
## Appendix GGeneral Capability Retention Analysis

Table 4:General capability retention relative to the base model\. Retention is computed asRetention=Scoremethod/Scorebase×100%\\mathrm\{Retention\}=\\mathrm\{Score\}\_\{\\mathrm\{method\}\}/\\mathrm\{Score\}\_\{\\mathrm\{base\}\}\\times 100\\%\. The values are derived from the averaged results in Table[1](https://arxiv.org/html/2606.00147#S4.T1)\.To make the forgetting effect more explicit, we additionally report the general capability retention rate relative to the corresponding base model\. For a general benchmark scoreSS, the retention rate is defined as

Retention​\(method\)=SmethodSbase×100%\.\\mathrm\{Retention\}\(\\mathrm\{method\}\)=\\frac\{S\_\{\\mathrm\{method\}\}\}\{S\_\{\\mathrm\{base\}\}\}\\times 100\\%\.We compute this value only for the general capability benchmarks, namely MS\-Bench and IFEval, because D\-Acc measures domain adaptation rather than retention of the original model’s general ability\. Table[4](https://arxiv.org/html/2606.00147#A7.T4)shows that RAFT improves general capability retention over standard SFT on both MS\-Bench and IFEval across all three backbone models\. On MS\-Bench, the average retention increases from 76\.8% under SFT to 89\.2% under RAFT\. On IFEval, the average retention increases from 91\.2% to 99\.5%\. These results provide a complementary view of the main results: RAFT improves domain accuracy over SFT while better retaining the base model’s general capabilities on the evaluated benchmarks\.

## Appendix HTeacher Model Concatenation Template

In the Answer\-Conditioned On\-Policy Distillation, the teacher model conditions on the fused answery^i\\hat\{y\}\_\{i\}as additional context when computing logits on the student’s generated trajectory\. This appendix provides the detailed concatenation template\.

### H\.1Template Structure

The teacher model’s input sequence is constructed by concatenating the following components in order:

\[Prefix\]\+\[FusedAnswer\]\+\[Middle\]\+\[UserPrompt\]\+\[Suffix\]\+\[StudentTrajectory\]

where:

- •Prefix: “Background reference knowledge:”
- •Fused Answer:y^i\\hat\{y\}\_\{i\}, the high\-quality fused answer from Section 3\.2
- •Middle: “\\nAnswer the question based on the reference knowledge:\\n”
- •User Prompt:xix\_\{i\}, the original user instruction
- •Suffix: “\\nAnswer:”
- •Student Trajectory:𝐳i\\mathbf\{z\}\_\{i\}, the token sequence generated by the student model

### H\.2Complete Template Example

The complete input template for the teacher model is as follows:

Backgroundreferenceknowledge:\{fused\_answer\}

Answerthequestionbasedonthereferenceknowledge:

\{user\_prompt\}

Answer:\{student\_trajectory\}

In contrast, the student model only conditions on:

\{user\_prompt\}\{student\_trajectory\}

### H\.3Design Rationale

This asymmetric context design serves two purposes:

1. 1\.Informed guidance: By conditioning ony^i\\hat\{y\}\_\{i\}, the teacher model can provide more informative soft targets at each position of the student’s trajectory, as it “knows” the reference answer while evaluating the student’s generation\.
2. 2\.Distribution alignment: The teacher’s logits on the student’s trajectory reflect how a model with access to ground\-truth knowledge would assign probabilities, guiding the student toward better generalization while preserving its own generation style\.

### H\.4Implementation Details

In practice, the template tokens are encoded without special tokens and concatenated as tensor sequences:

teacher\_input\_ids=concat\(\[

encode\("Backgroundreferenceknowledge:"\),

fused\_answer\_ids,

encode\("\\nAnswerthequestionbasedon"

"thereferenceknowledge:\\n"\),

prompt\_ids,

encode\("\\nAnswer:"\),

trajectory\_ids

\],dim=1\)

The logits corresponding to the trajectory positions are then extracted for distillation loss computation, ensuring alignment between teacher and student predictions on the same token positions\.