SFT-as-Context缓解监督微调中的灾难性遗忘

# SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning
Source: [https://arxiv.org/html/2610.11132](https://arxiv.org/html/2610.11132)
Consistent with prior observations of forgetting after SFT\([Chen et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib8);[Huan et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib4)\), we find that the improvement of fine\-tuned capabilities comes at the expense of general capabilities\. As shown in[Section2\.2](https://arxiv.org/html/2610.11132#S2.SS2), SFT causes substantial forgetting of general capabilities, reducing general capability scores by 33\.2 percentage points on average\. For example, among the three nutrition SFT models, Qwen3\-8B shows the largest drop in acknowledging non\-food queries, from 99\.6% to 10\.7%, catastrophically forgetting the instruction\-following capability of its parent model\.

This forgetting phenomenon becomes more problematic when a single query requires both fine\-tuned and general capabilities, as the loss of either can cause the entire response to fail\. For example, each query in MathIF requires both the fine\-tuned capability of solving the math problem correctly and the general capability of following additional response constraints, such as using only lowercase letters\. The SFT model significantly improves math correctness, from 40\.6% to 65\.0%, but fails to follow the format constraints\. As a result, the hard instruction\-following \(Hard IF\) score falls from 50\.1% to 21\.3%\. In one example of a polynomial\-remainder problem \(Appendix[C](https://arxiv.org/html/2610.11132#A3)\), the SFT model answers correctly but uses the uppercase symbolRR, violating the lowercase requirement\. Similar problems occur on LiveCodeBenchIF and NutriBench\-Non\-English\.

Taken together, fine\-tuning substantially improves performance on the task the model is fine\-tuned on, but at the expense of forgetting general capabilities that the parent model already possesses, e\.g\., instruction\-following and multilingual understanding\.

![Refer to caption](https://arxiv.org/html/2610.11132v1/all_benchmarks_differences.png)Figure 2:Across individual model pairs, SFT\-as\-context maintains fine\-tuned capabilities and mitigates forgetting of general capabilities\.In each heatmap, the first group of columns shows fine\-tuned capabilities, and the second group of columns shows general capabilities\. Each domain has two heatmaps: one showing the performance difference between the SFT model and the parent model \(SFT vs\. Parent\), and the other showing the difference between SFT\-as\-context and the parent model \(SFT\-as\-context vs\. Parent\)\. Darker blue indicates a larger improvement over the parent model, whereas darker red indicates more severe forgetting relative to the parent model\. Color ranges are clamped to prevent large outliers from dominating the colors\. For NutriBench macro MAE, the values are reversed to consistently show greater improvement in darker blue\.
## 3SFT\-as\-Context Successfully Mitigates Forgetting

#### Method\.

To mitigate forgetting in SFT models, we propose SFT\-as\-context, which leverages the in\-context learning ability of the parent model by providing the SFT model’s response as context when generating the final response\. Specifically, during inference, SFT\-as\-context answers each query through two sequential passes\. In the first pass, the SFT model generates a response to the query\. In the second pass, we construct the parent model’s prompt from three parts: the original query, the SFT response from the first pass, and the SFT\-as\-context instructions\. These instructions ask the parent model to treat the SFT response as a helpful answer unless there are concrete errors\.

Formally, letxxdenote the input query,fSFTf\_\{\\text\{SFT\}\}the SFT model, andfparentf\_\{\\text\{parent\}\}its parent model\. The SFT model first generates:ySFT=fSFT​\(x\)\.y\_\{\\text\{SFT\}\}=f\_\{\\text\{SFT\}\}\(x\)\.We then construct the input to the parent model as\[x,ySFT,I\]\[x,y\_\{\\text\{SFT\}\},I\], whereIIdenotes the SFT\-as\-context instructions\. The SFT\-as\-context instructions are fixed for each domain \(math, coding, or nutrition\), agnostic to models and different fine\-tuned or general capability benchmarks \(Appendix[B\.2](https://arxiv.org/html/2610.11132#A2.SS2)\)\. The parent model generates the final response as:

ySAC=fparent​\(x,ySFT,I\)\.y\_\{\\text\{SAC\}\}=f\_\{\\text\{parent\}\}\(x,y\_\{\\text\{SFT\}\},I\)\.\(1\)Thus, SFT\-as\-context provides the SFT model’s response as context to the parent model, which then generates the final answer\. Crucially, SFT\-as\-context is a novel, training\-free approach to mitigating forgetting that requires neither access to model weights nor the fine\-tuning process\. It operates entirely through text input\-output interfaces, making it applicable even to closed\-source models\.

#### Evaluation Metric\.

We define theperformance recovery rateto quantify how much of the performance gap between the parent and SFT models is recovered by SFT\-as\-context for a given capability\. Letsparents\_\{\\text\{parent\}\},sSFTs\_\{\\text\{SFT\}\}, andsSACs\_\{\\text\{SAC\}\}denote the parent, SFT, and SFT\-as\-context scores \(e\.g\., accuracy or MAE\), respectively\. For metrics where higher is better, the recovery rateRRis defined as

R=\[sSAC−min⁡\(sparent,sSFT\)\]/\|sparent−sSFT\|×100%\.R=\[s\_\{\\text\{SAC\}\}\-\\min\(s\_\{\\text\{parent\}\},s\_\{\\text\{SFT\}\}\)\]/\|s\_\{\\text\{parent\}\}\-s\_\{\\text\{SFT\}\}\|\\times 100\\%\.\(2\)A recovery rate of0%0\\%indicates that SFT\-as\-context matches the worse\-performing model, while100%100\\%indicates that it matches the better\-performing model\. The recovery rate can exceed100%100\\%when SFT\-as\-context outperforms both models\. For metrics where lower is better, such as MAE, the recovery rate is alternatively defined as100%−R100\\%\-R\.

#### SFT\-as\-Context Combines the Strengths of Parent and SFT Models\.

Despite its simplicity, SFT\-as\-context achieves performance close to the parent model on general capabilities while remaining close to the SFT model on fine\-tuned capabilities, effectively combining the strengths of SFT and parent models, as shown in[Section2\.2](https://arxiv.org/html/2610.11132#S2.SS2)\. Across 19 parent\-fine\-tuned model pairs and 11 benchmarks, SFT\-as\-context preserves 92\.0% of the fine\-tuning gains on average, while recovering 94\.2% of the general\-capability performance gap between parent and SFT models\. More importantly, SFT\-as\-context can combine fine\-tuned and general capabilities within the same response, which neither the parent nor the SFT model can achieve alone\. We evaluate this setting on MathIF, LiveCodeBenchIF, and NutriBench\-Non\-English, where each query requires both fine\-tuned and general capabilities, and we report the performance on both for each dataset in[Section2\.2](https://arxiv.org/html/2610.11132#S2.SS2)\. For the same MathIF example discussed in[Section2\.2](https://arxiv.org/html/2610.11132#S2.SS2), the SFT model provides a correct answer but uses uppercase letters, whereas the SFT\-as\-context response preserves the correct answer while satisfying the lowercase formatting constraint \(Appendix[C\.1](https://arxiv.org/html/2610.11132#A3.SS1)\)\.

[Figure2](https://arxiv.org/html/2610.11132#S2.F2)shows that these trends extend across individual model pairs, including those with severe forgetting\. For OCR\-Nemotron\-1\.1\-14B, SFT reduces CoQA performance by 73\.4 percentage points relative to the parent model, indicating severe forgetting in conversational question answering\. SFT\-as\-context brings this performance back to near the parent model’s level, with a gap of only 0\.8 percentage points, while preserving most of the gains on LiveCodeBench \(39\.8 out of 41\.8 percentage points\)\. This highlights that SFT\-as\-context can mitigate severe forgetting while preserving substantial fine\-tuning gains\.

To understand the remaining performance gap, we zoom into MathIF, which shows the largest gap on the Hard IF metric \([Figure2](https://arxiv.org/html/2610.11132#S2.F2)\)\. The dominant failure mode is that the small parent model, Qwen2\.5\-7B\-Instruct \(the shared parent of all 7B math SFT models\), often returns only a boxed answer and thus violates formatting requirements of using specific words or sections \(Appendix[C\.2](https://arxiv.org/html/2610.11132#A3.SS2)\)\. This is effectively mitigated by a larger parent: the 32B parent exhibits this failure far less often, yielding a smaller remaining gap for the 32B models than for the 7B ones \(an average gap of−1\.8\-1\.8versus−7\.5\-7\.5\)\.

## 4Understanding the Mechanisms of SFT\-as\-Context

To understand why SFT\-as\-context can mitigate forgetting while preserving the fine\-tuned capabilities, we first analyze how LLMs combine capabilities from a theoretical perspective \([Section4\.1](https://arxiv.org/html/2610.11132#S4.SS1)\)\. Then, by analyzing attention patterns, we empirically show that in SFT\-as\-context, a parent LLM attends more to useful SFT responses and less to hallucinated context, suggesting a possible mechanism for recovering general capabilities while preserving fine\-tuned capabilities \([Section4\.2](https://arxiv.org/html/2610.11132#S4.SS2)\)\.

### 4\.1Theoretical Explanation

We explain the effectiveness of SFT\-as\-context using a Bayesian framework for in\-context learning\([Xie et al\., 2022](https://arxiv.org/html/2610.11132#bib.bib2);[Hu et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib3)\), building on earlier Bayesian models\([Baum and Petrie, 1966](https://arxiv.org/html/2610.11132#bib.bib5);[Blei et al\., 2003](https://arxiv.org/html/2610.11132#bib.bib6)\)\. Following[Xie et al\. \(2022\)](https://arxiv.org/html/2610.11132#bib.bib2), we model response generation as conditioned on a domain variableθ\\theta, with output distributionPr⁡\(Y∣x,θ\)\\operatorname\{Pr\}\(Y\\mid x,\\theta\)\. Each such variableθ\\thetacorresponds to a domain such as nutrition estimation, math, instruction\-following, and multilinguality\.

The parent model is trained using a large dataset across many different domains and therefore has many capabilities\. LetΘparent\\Theta\_\{\\text\{parent\}\}denote the parent model’s domain set\. We assume that the SFT model is fine\-tuned on a single domainθSFT\\theta\_\{\\text\{SFT\}\}and denote its query set byDSFTD\_\{\\text\{SFT\}\}\. Our analysis explains how SFT\-as\-context combines SFT expertise with the parent model’s general capabilities\.

For a queryxx, letPrparent\(⋅∣x\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\cdot\\mid x\)andPrSFT\(⋅∣x\)\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(\\cdot\\mid x\)denote the parent and SFT response distributions, respectively, and definePrSAC\(⋅∣x\):=Prparent\(⋅∣x,ySFT,I\)\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(\\cdot\\mid x\):=\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\cdot\\mid x,y\_\{\\text\{SFT\}\},I\), whereySFTy\_\{\\text\{SFT\}\}is the SFT response andIIdenotes the SFT\-as\-context instructions\. All variables are discrete\. For each domain, assume an optimal domain variableθ∗\\theta^\{\*\}inducing the optimal response lawPr∗\(⋅∣x\):=Pr\(⋅∣x,θ∗\)\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\):=\\operatorname\{Pr\}\(\\cdot\\mid x,\\theta^\{\*\}\)\. Form∈\{parent,SFT,SAC\}m\\in\\\{\\text\{parent\},\\text\{SFT\},\\text\{SAC\}\\\}, define the corresponding errorsℰm:=KL\(Pr∗\(⋅∣x\)∥Prm\(⋅∣x\)\)\.\\mathcal\{E\}\_\{m\}:=\\operatorname\{KL\}\\\!\\left\(\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\)\\,\\\|\\,\\operatorname\{Pr\}\_\{m\}\(\\cdot\\mid x\)\\right\)\.We model the parent and SFT\-as\-context distributions as mixtures:

Prparent⁡\(y∣x\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid x\)=∑θ∈ΘparentPrparent⁡\(y∣x,θ\)​Prparent​\(θ∣x\),\\displaystyle=\\sum\_\{\\theta\\in\\Theta\_\{\\text\{parent\}\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid x,\\theta\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\\mid x\),PrSAC⁡\(y∣x\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(y\\mid x\)=∑θ∈ΘparentPrparent⁡\(y∣x,θ\)​PrSAC​\(θ∣x,ySFT,I\)\.\\displaystyle=\\sum\_\{\\theta\\in\\Theta\_\{\\text\{parent\}\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid x,\\theta\)\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(\\theta\\mid x,y\_\{\\text\{SFT\}\},I\)\.Thus, conditioning on the SFT response and instructions changes the domain posterior while preserving the per\-domain conditional output laws\. We refer toPr⋅⁡\(θ\|x\)\\operatorname\{Pr\}\_\{\\cdot\}\(\\theta\|x\)as the posterior probability, which under the Bayesian framework is the probability of the domain variable conditioned on the query\. Intuitively, we model the LLM’s generation process as predicting a domain based on the query with some probability and then generating the response conditioned on the domain variable\([Xie et al\., 2022](https://arxiv.org/html/2610.11132#bib.bib2)\)\. We make the following assumptions for allxxwhere all logarithms are natural, and the total variation is defined asTV⁡\(P,Q\):=12​∑y\|P⁡\(y\)−Q⁡\(y\)\|\\operatorname\{TV\}\(P,Q\):=\\frac\{1\}\{2\}\\sum\_\{y\}\|P\(y\)\-Q\(y\)\|\. We present assumptions intuitively followed by the formalism\.

1. 1\.The posterior probability of the SFT domain variable is low for the parent model\. Formally,0<ax:=Prparent⁡\(θSFT∣x\)≤ϵ0<a\_\{x\}:=\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\_\{\\text\{SFT\}\}\\mid x\)\\leq\\epsilonfor the inputxxthat belongs to the SFT domain, whereθSFT\\theta\_\{\\text\{SFT\}\}is the optimal domain variable for the SFT domain\.
2. 2\.The posterior probability of the SFT domain variable given the SFT response isϵ′\\epsilon^\{\\prime\}\-close to 1\. Formally, under the parent model, the SFT responseySFTy\_\{\\text\{SFT\}\}satisfiesPrSAC⁡\(θSFT∣x,I,ySFT\)≥1−ϵ′\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\\\!\\left\(\\theta\_\{\\text\{SFT\}\}\\mid x,I,y\_\{\\text\{SFT\}\}\\right\)\\geq 1\-\\epsilon^\{\\prime\}, meaning that the posterior probability assigned to the optimal domain variableθSFT\\theta\_\{\\text\{SFT\}\}is withinϵ′\\epsilon^\{\\prime\}of 1\.
3. 3\.The optimal response distribution is separated from that of the parent model by some margin\([Xie et al\., 2022](https://arxiv.org/html/2610.11132#bib.bib2)\)\. LetQparentQ\_\{\\text\{parent\}\}denote the parent model’s response distribution conditioned on a non\-SFT domain:Qparent​\(y∣x\):=Prparent⁡\(y∣x,θ≠θSFT\)\.Q\_\{\\text\{parent\}\}\(y\\mid x\):=\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid x,\\theta\\neq\\theta\_\{\\text\{SFT\}\}\)\.There existsϵ′′\>0\\epsilon^\{\\prime\\prime\}\>0such that, for everyx∈DSFTx\\in D\_\{\\text\{SFT\}\}:TV\(Pr∗\(⋅∣x\),Qparent\(⋅∣x\)\)≥ϵ′′\\operatorname\{TV\}\\\!\\left\(\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\),Q\_\{\\text\{parent\}\}\(\\cdot\\mid x\)\\right\)\\geq\\epsilon^\{\\prime\\prime\}\.
4. 4\.The SFT model is an expert on the one domain it was fine\-tuned on\. Formally, the SFT domain variableθSFT\\theta\_\{\\text\{SFT\}\}is such thatPrSFT⁡\(y∣θSFT,x\)=Pr∗⁡\(y∣x\)\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\\mid\\theta\_\{\\text\{SFT\}\},x\)=\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\)\. Furthermore, there exists0≤η<10\\leq\\eta<1such that, for everyx∈DSFTx\\in D\_\{\\text\{SFT\}\},cx:=PrSFT⁡\(θSFT∣x\)≥1−η\.c\_\{x\}:=\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(\\theta\_\{\\text\{SFT\}\}\\mid x\)\\geq 1\-\\eta\.
5. 5\.The SFT\-as\-context domain posterior retains a minimum fraction of the corresponding parent posterior\. Formally, there exists0≤ρo<10\\leq\\rho\_\{o\}<1such that, for every out\-of\-domain query and realized context under consideration, PrSAC⁡\(θ∣x,I,ySFT\)≥\(1−ρo\)​Prparent​\(θ∣x\),∀θ∈Θparent\.\\operatorname\{Pr\}\\nolimits\_\{\\text\{SAC\}\}\(\\theta\\mid x,I,y\_\{\\text\{SFT\}\}\)\\geq\(1\-\\rho\_\{o\}\)\\operatorname\{Pr\}\\nolimits\_\{\\text\{parent\}\}\(\\theta\\mid x\),\\qquad\\forall\\theta\\in\\Theta\_\{\\text\{parent\}\}\.

Under these assumptions, we have the following results forxxthat is in domain for the SFT dataset:

###### Theorem 1\(Error guarantees for in\-domain queries\)\.

Suppose that the shared conditional\-output model and in\-domain realizability condition hold \(Appendix[K\.3](https://arxiv.org/html/2610.11132#A11.SS3)\)\. Let Assumptions 1–4 hold with0<ϵ<10<\\epsilon<1,0≤ϵ′<10\\leq\\epsilon^\{\\prime\}<1,0<ϵ′′≤10<\\epsilon^\{\\prime\\prime\}\\leq 1, and0≤η<10\\leq\\eta<1\. For every in\-domain queryx∈DSFTx\\in D\_\{\\text\{SFT\}\}and the realized context\(ySFT,I\)\(y\_\{\\text\{SFT\}\},I\)satisfying these assumptions, the following are true:

Improvement over the parent:ℰparent−ℰSAC≥2\(1−ϵ\)2\(ϵ′′\)2\+log\(1−ϵ′\)\.\\textbf\{Improvement over the parent: \}\\mathcal\{E\}\_\{\\text\{parent\}\}\-\\mathcal\{E\}\_\{\\text\{SAC\}\}\\geq 2\(1\-\\epsilon\)^\{2\}\(\\epsilon^\{\\prime\\prime\}\)^\{2\}\+\\log\(1\-\\epsilon^\{\\prime\}\)\.\(3\)Closeness to SFT:log\(1−ϵ′\)≤ℰSFT−ℰSAC≤−log\(1−η\)\.\\textbf\{Closeness to SFT: \}\\log\(1\-\\epsilon^\{\\prime\}\)\\leq\\mathcal\{E\}\_\{\\text\{SFT\}\}\-\\mathcal\{E\}\_\{\\text\{SAC\}\}\\leq\-\\log\(1\-\\eta\)\.\(4\)

The theorem identifies sufficient conditions under which SFT\-as\-context improves in\-domain performance compared to the parent model while limiting degradation compared to the SFT model\. Intuitively, Equation[3](https://arxiv.org/html/2610.11132#S4.E3)shows that the error of the parent for in\-domain queries is greater than the error of SFT\-as\-context under the Bayesian model of text generation\. Equation[4](https://arxiv.org/html/2610.11132#S4.E4)means that the errors of SFT and SFT\-as\-context are close to one another\. The proof can be found in Appendix[K](https://arxiv.org/html/2610.11132#A11)\. While these results support how SFT\-as\-context maintains fine\-tuned capabilities \([Figure2](https://arxiv.org/html/2610.11132#S2.F2)\), we also show how under this Bayesian framework it maintains general capabilities with the following theorem\.

###### Theorem 2\(Error guarantees for out\-of\-domain queries\)\.

Suppose that the shared conditional\-output model and out\-of\-domain realizability condition hold \(Appendix[K\.5](https://arxiv.org/html/2610.11132#A11.SS5)\)\. For anyout\-of\-domain queryx∈Dox\\in D\_\{o\}, whereθo∈Θparent\\theta\_\{o\}\\in\\Theta\_\{\\text\{parent\}\}andθo≠θSFT\\theta\_\{o\}\\neq\\theta\_\{\\text\{SFT\}\}, suppose that Assumption 5 holds with0≤ρo<10\\leq\\rho\_\{o\}<1and thatℰparent<∞\\mathcal\{E\}\_\{\\text\{parent\}\}<\\infty, with errors measured relative to the optimal output law onDoD\_\{o\}\. Then, for every realized context satisfying this assumption:ℰSAC−ℰparent≤−log⁡\(1−ρo\)\.\\mathcal\{E\}\_\{\\text\{SAC\}\}\-\\mathcal\{E\}\_\{\\text\{parent\}\}\\leq\-\\log\(1\-\\rho\_\{o\}\)\.

The inequality above shows that for out\-of\-domain queries the SFT\-as\-context method performs similarly to the parent model by upper bounding the difference in error\. We have thus shown the conditions under which the errors of SFT\-as\-context are bounded relative to the SFT model on in\-domain queries and relative to the parent model on out\-of\-domain queries\. Our theoretical results bolster our empirical claims: SFT\-as\-context remains close to the SFT model on fine\-tuned capabilities while staying close to the parent model on general capabilities \([Section2\.2](https://arxiv.org/html/2610.11132#S2.SS2)and[Figure2](https://arxiv.org/html/2610.11132#S2.F2)\)\.

### 4\.2Attention Analysis

SFT\-as\-context places the SFT model’s response in the parent model’s context\. However, these responses can provide useful task\-specific information or contain misleading hallucinations, as illustrated in[Figure1](https://arxiv.org/html/2610.11132#S1.F1)\. How does the parent model use these responses in SFT\-as\-context?

We investigate how the parent model uses SFT responses by analyzing attention weights, which characterize how strongly it attends to different prompt tokens during generation\([Clark et al\., 2019](https://arxiv.org/html/2610.11132#bib.bib56)\)\. We compare two settings: nutrition queries from NutriBench\-English, for which SFT responses provide useful context, and non\-food queries from NutriBench\-Non\-Food, for which SFT responses may contain hallucinated nutrition estimates\. The model pair is Gemma\-4\-E4B\-it and its SFT version\. We randomly sample 100 queries from each category and measure the attention allocated to the SFT responses, averaging over response tokens, layers, and attention heads \(Appendix[D](https://arxiv.org/html/2610.11132#A4)\)\.

[Figure1](https://arxiv.org/html/2610.11132#S1.F1)\(d\) shows that the parent model attends more to useful SFT responses than hallucinated ones, with a 22\.33 percentage point difference in the average share of attention allocated to the SFT responses\. This selective attention suggests that the parent model can use useful information from SFT responses through in\-context learning while limiting the influence of irrelevant context, providing a possible explanation for how SFT\-as\-context mitigates forgetting while preserving fine\-tuned capabilities\.

## 5Discussion

Here, we extend SFT\-as\-context to a strong proprietary model, showing that responses from open\-source SFT models can further improve its performance with only text input\-output access \([Section5\.1](https://arxiv.org/html/2610.11132#S5.SS1)\)\. We also provide ablation studies to show that the recovery of both fine\-tuned and general capabilities does not arise solely from the SFT\-as\-context instructions \([Section5\.2](https://arxiv.org/html/2610.11132#S5.SS2)\)\.

### 5\.1SFT\-as\-Context Further Improves Proprietary Models

Beyond combining the strengths of the parent and SFT models, SFT\-as\-context can further improve fine\-tuned capabilities when the final response is generated by a stronger model rather than the original parent model\. To be more specific, we fine\-tune an open\-source model such as Qwen3 and use its response as context for Gemini, which further improves Gemini’s performance\. This flexibility is unique to SFT\-as\-context because it requires only standard text input\-output access, without access to model weights or the fine\-tuning process\.

As shown in[Table2](https://arxiv.org/html/2610.11132#S5.T2), using responses from open\-source SFT models as context consistently improves the performance of a strong proprietary model on NutriBench\-English, achieving lower MAE than either model alone\. This result also shows that the proprietary model does not merely copy the SFT response during inference\. Instead, it can combine task\-specific information from the SFT response with its own capabilities to produce a more accurate final answer\.

### 5\.2Ablation Studies

To test whether the gains of SFT\-as\-context come simply from the SFT\-as\-context instructions \(II\) and an additional inference pass, we evaluate two control settings on NutriBench: the parent model using its own response as context and the SFT model using its own response as context\.

As shown in[Table3](https://arxiv.org/html/2610.11132#S5.T3), the parent model with its own context preserves strong general capabilities, but its English and non\-English macro MAE remain at 36\.6 and 77\.4, compared with 18\.4 and 63\.6 for SFT\-as\-context\. Conversely, the SFT model with its own context retains most of its nutrition estimation gains but fails to recover general capabilities\. Language consistency and acknowledgment of non\-food queries are only 1\.3% and 17\.9%, compared with 73\.7% and 98\.0% for SFT\-as\-context\. These results suggest that the gains of SFT\-as\-context cannot be explained solely by the SFT\-as\-context instructions or an additional inference pass\.

Table 2:A strong proprietary model can use context from open\-source SFT models to further improve fine\-tuned capabilities\.On NutriBench\-English, using the context from Qwen3\-4B achieves the best macro MAE\. Best results are bolded and second\-best results are underlined\.Table 3:SFT\-as\-context achieves the best balance between fine\-tuned and general capabilities\.Parent with parent context fails to achieve low English and non\-English macro MAE, and SFT with SFT context still fails on language consistency and acknowledging non\-food queries\. Best results are bolded and second\-best results are underlined\.## 6Related Work

#### Forgetting and Mitigation\.

Prior work mitigates forgetting during fine\-tuning through regularization toward pretrained behavior or parameters, as in Learning without Forgetting\([Li and Hoiem, 2017](https://arxiv.org/html/2610.11132#bib.bib47)\)and RecAdam\([Chen et al\., 2020](https://arxiv.org/html/2610.11132#bib.bib44)\), replay\-based methods such as LAMOL\([Sun et al\., 2020](https://arxiv.org/html/2610.11132#bib.bib48)\), constraints on parameter updates\([Lopez\-Paz and Ranzato, 2017](https://arxiv.org/html/2610.11132#bib.bib49);[Lin et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib20)\), or the choice of adaptation algorithm\([Biderman et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib46);[Chen et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib8)\)\. Recent work finds that on\-policy reinforcement learning preserves parent capabilities better than SFT\([Chen et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib8)\); we therefore focus on SFT\-induced forgetting, where forgetting is typically more severe\. In principle, the same inference\-time approach could be applied to a model fine\-tuned with reinforcement learning \(RLFT\) by providing its response as context to the parent model\.

#### Inference\-Time Model Combination\.

A line of work related to our setting explores combining pretrained and fine\-tuned models at inference time\. Emulated Fine\-Tuning \(EFT\)\([Mitchell et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib50)\)and Proxy\-Tuning\([Liu et al\., 2024a](https://arxiv.org/html/2610.11132#bib.bib51)\)combine model predictions through token\-level probabilities or logits\. They require access to output distributions during decoding and do not specifically target forgetting\.

More recently, concurrent work CPR\([Ki et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib52)\)mitigates forgetting by training a router to select between parent and fine\-tuned models during decoding\. In contrast, SFT\-as\-context requires only text input\-output access, using the fine\-tuned model’s response as context for the parent model\.

## 7Conclusion

In this paper, we propose SFT\-as\-context, a training\-free method that leverages the in\-context learning ability of LLMs to mitigate the forgetting of general capabilities in SFT models while preserving their fine\-tuned capabilities\. Comprehensive experiments across domains, tasks, and model sizes demonstrate the effectiveness of SFT\-as\-context\. We further analyze its underlying mechanism through a Bayesian view of in\-context learning and attention analysis, showing that useful SFT responses can steer the parent model toward the fine\-tuned domain while having less influence on unrelated queries, thereby preserving general capabilities\. Since the same framework could naturally extend beyond SFT to other post\-training methods, such as reinforcement learning, we leave this as a promising future direction to combine capabilities acquired through post\-training with the broad capabilities of pretrained models\.

### AI use statement

In this work, we used generative AI tools for generating synthetic datasets, implementing methods, assisting with translation, cleaning and reformatting datasets, supporting qualitative and thematic data analysis, and interpreting results\. We have not used generative AI tools for helping develop theoretical models or conceptual frameworks, formulating mathematical claims, providing critical ingredients for proving mathematical claims, assisting in the writing of proofs, proposing or refining hypotheses, and designing or providing feedback on research methodology or experiments\. Additionally, we used generative AI tools for creating or modifying scientific figures or images, creating or editing software code, summarizing or analyzing existing literature, editing a research paper to improve readability, and identifying relevant literature\.

We have reviewed all AI\-assisted work\. We manually verify synthetically generated meal descriptions for NutriBench\. We compare with officially released metric values for public models to ensure LLM\-generated code correctly implements dataset formatting, inference, and scoring\. We use AI agents to speed up the identification of failure patterns in the forgetting of general capabilities, and we then manually conduct qualitative and thematic data analysis\. We read through each of the suggested papers returned from AI\-assisted literature review\.

We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI\.

### Reproducibility statement

Most of our results rely on publicly available model checkpoints and benchmarks, with the following three exceptions\. First, we train our own models for nutrition estimation \(Appendix[A\.3](https://arxiv.org/html/2610.11132#A1.SS3)\)\. Second, we add multilingual queries to the NutriBench dataset for testing fine\-tuned and general capabilities \(Appendix[A\.1](https://arxiv.org/html/2610.11132#A1.SS1)\)\. Third, the proposed method requires manually written SFT\-as\-context instructions, which we include in Appendix[B\.2](https://arxiv.org/html/2610.11132#A2.SS2)\. Hence, we have provided sufficient details to allow full reproduction of the results\. For theoretical results and proofs, we include more details in Appendix[K](https://arxiv.org/html/2610.11132#A11)\.

## References

- Ahmadet al\.\(2025\)W\. U\. Ahmad, S\. Narenthiran, S\. Majumdar, A\. Ficek, S\. Jain, J\. Huang, V\. Noroozi, and B\. GinsburgOpenCodeReasoning: advancing data distillation for competitive coding\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=aykM7KUVJZ)Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.13.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.14.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.15.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.16.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.18.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.19.2)\.
- Anet al\.\(2025\)C\. An, Z\. Xie, X\. Li, L\. Li, J\. Zhang, S\. Gong, M\. Zhong, J\. Xu, X\. Qiu, M\. Wang, and L\. KongPOLARIS: a post\-training recipe for scaling reinforcement learning on advanced reasoning models\.External Links:[Link](https://hkunlp.github.io/blog/2025/Polaris)Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.12.1)\.
- Arditiet al\.\(2024\)A\. Arditi, O\. Obeso, A\. Syed, D\. Paleka, N\. Panickssery, W\. Gurnee, and N\. NandaRefusal in language models is mediated by a single direction\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 136037–136083\.External Links:[Document](https://dx.doi.org/10.52202/079017-4322),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/f545448535dfde4f9786555403ab7c49-Paper-Conference.pdf)Cited by:[Appendix F](https://arxiv.org/html/2610.11132#A6.p2.1)\.
- Baum and Petrie \(1966\)L\. E\. Baum and T\. PetrieStatistical inference for probabilistic functions of finite state Markov chains\.The Annals of Mathematical Statistics37\(6\),pp\. 1554–1563\.Cited by:[§4\.1](https://arxiv.org/html/2610.11132#S4.SS1.p1.1)\.
- Bespoke Labs \(2025\)Bespoke LabsBespoke\-Stratos\-7B\.Note:Hugging FaceAccessed: 2026\-08\-01External Links:[Link](https://huggingface.co/bespokelabs/Bespoke-Stratos-7B)Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.6.2)\.
- Bidermanet al\.\(2024\)D\. Biderman, J\. Portes, J\. J\. G\. Ortiz, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle, C\. Blakeney, and J\. P\. CunninghamLoRA learns less and forgets less\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=aloEru2qCG)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Bleiet al\.\(2003\)D\. M\. Blei, A\. Y\. Ng, and M\. I\. JordanLatent Dirichlet allocation\.Journal of Machine Learning Research3\(Jan\),pp\. 993–1022\.Cited by:[§4\.1](https://arxiv.org/html/2610.11132#S4.SS1.p1.1)\.
- Caiet al\.\(2026\)W\. Cai, C\. Wang, J\. Yan, J\. Huang, and X\. FangReasoning with OmniThought: a large CoT dataset with verbosity and cognitive difficulty annotations\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 8431–8450\.External Links:[Link](https://aclanthology.org/2026.acl-long.382/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.382),ISBN 979\-8\-89176\-390\-6Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.7.2)\.
- Chenet al\.\(2026\)H\. Chen, N\. Razin, K\. R\. Narasimhan, and D\. ChenRetaining by doing: the role of on\-policy data in mitigating forgetting\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=ODTM64azGa)Cited by:[Appendix E](https://arxiv.org/html/2610.11132#A5.p1.1),[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.11132#S2.SS2.tab1.6.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2020\)S\. Chen, Y\. Hou, Y\. Cui, W\. Che, T\. Liu, and X\. YuRecall and learn: fine\-tuning deep pretrained language models with less forgetting\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 7870–7881\.Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2025\)Z\. Chen, Y\. Min, B\. Zhang, J\. Chen, J\. Jiang, D\. Cheng, W\. X\. Zhao, Z\. Liu, X\. Miao, Y\. Lu,et al\.An empirical study on eliciting and improving R1\-like reasoning models\.arXiv preprint arXiv:2503\.04548\.Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.8.1)\.
- Chunget al\.\(2024\)H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p1.1)\.
- Clarket al\.\(2019\)K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. ManningWhat does BERT look at? An analysis of BERT’s attention\.InProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,T\. Linzen, G\. Chrupała, Y\. Belinkov, and D\. Hupkes \(Eds\.\),Florence, Italy,pp\. 276–286\.External Links:[Link](https://aclanthology.org/W19-4828/),[Document](https://dx.doi.org/10.18653/v1/W19-4828)Cited by:[§4\.2](https://arxiv.org/html/2610.11132#S4.SS2.p2.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p5.1)\.
- Dhaliwalet al\.\(2025\)M\. Dhaliwal, A\. Hua, L\. Pullela, R\. Burke, and Y\. QinNutriBench: a dataset for evaluating large language models in nutrition estimation from meal descriptions\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 95927–95950\.Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p12.1),[§A\.3](https://arxiv.org/html/2610.11132#A1.SS3.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2610.11132#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Fuet al\.\(2026\)T\. Fu, Y\. Li, J\. Gu, X\. Qu, and Y\. ChengScaling reasoning, losing control: evaluating instruction following in large reasoning models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 40445–40463\.External Links:[Link](https://aclanthology.org/2026.acl-long.1878/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1878),ISBN 979\-8\-89176\-390\-6Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p3.1),[1st item](https://arxiv.org/html/2610.11132#S2.I1.i1.p1.1)\.
- Fuet al\.\(2025\)W\. Fu, J\. Gao, X\. Shen, C\. Zhu, Z\. Mei, C\. He, S\. Xu, G\. Wei, J\. Mei, J\. Wang, T\. Yang, B\. Yuan, and Y\. WuAReaL: a large\-scale asynchronous reinforcement learning system for language reasoning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=X9diEuva9R)Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.7.1)\.
- Gaoet al\.\(2024\)L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. ZouThe language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p10.1)\.
- Gemma Team \(2026\)Gemma TeamGemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.Cited by:[§A\.3](https://arxiv.org/html/2610.11132#A1.SS3.SSS0.Px1.p1.1)\.
- Guhaet al\.\(2026\)E\. K\. Guha, R\. Marten, S\. Keh, N\. Raoof, G\. Smyrnis, H\. Bansal, M\. Nezhurina, J\. Mercat, T\. Vu, Z\. R\. Sprague, A\. Suvarna, B\. Feuer, L\. L\. Chen, Z\. Khan, E\. Frankel, S\. Grover, C\. Choi, N\. Muennighoff, S\. Su, W\. Zhao, J\. Yang, S\. Pimpalgaonkar, K\. Sharma, C\. C\. Ji, Y\. Deng, S\. M\. Pratt, V\. Ramanujan, J\. Saad\-Falcon, S\. Acharya, J\. Li, A\. Dave, A\. Albalak, K\. Arora, B\. Wulfe, C\. Hegde, G\. Durrett, S\. Oh, M\. Bansal, S\. Gabriel, A\. Grover, K\. Chang, V\. Shankar, A\. Gokaslan, M\. A\. Merrill, T\. Hashimoto, Y\. Choi, J\. Jitsev, R\. Heckel, M\. Sathiamoorthy, A\. Dimakis, and L\. SchmidtOpenThoughts: data recipes for reasoning models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=7xjoTuaNmN)Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.10.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.3.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.4.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.5.2)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-R1: incentivizing reasoning capability in LLMs via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[Appendix E](https://arxiv.org/html/2610.11132#A5.p1.1),[§1](https://arxiv.org/html/2610.11132#S1.p1.1)\.
- Heet al\.\(2024\)J\. He, H\. Guo, K\. Zhu, Z\. Zhao, M\. Tang, and J\. WangSEEKR: selective attention\-guided knowledge retention for continual learning of large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 3254–3266\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.190/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.190)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1)\.
- Heet al\.\(2025\)J\. He, J\. Liu, C\. Y\. Liu, R\. Yan, C\. Wang, P\. Cheng, X\. Zhang, F\. Zhang, J\. Xu, W\. Shen,et al\.Skywork Open Reasoner 1 technical report\.arXiv preprint arXiv:2505\.22312\.Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.11.1)\.
- Heet al\.\(2026\)Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang,et al\.DeepMath\-103K: a large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 138306–138322\.Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.6.1)\.
- Heinrichet al\.\(2022\)Q\. Heinrich, G\. Viaud, and W\. BelblidiaFQuAD2\.0: French question answering and learning when you don’t know\.InProceedings of the Thirteenth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 2205–2214\.External Links:[Link](https://aclanthology.org/2022.lrec-1.237/)Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p9.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Huet al\.\(2024\)X\. Hu, F\. Zhang, S\. Chen, and Z\. YangUnveiling the statistical foundations of chain\-of\-thought prompting methods\.arXiv preprint arXiv:2408\.14511\.Cited by:[§K\.1](https://arxiv.org/html/2610.11132#A11.SS1.p2.1),[§1](https://arxiv.org/html/2610.11132#S1.p6.1),[§4\.1](https://arxiv.org/html/2610.11132#S4.SS1.p1.1)\.
- Huaet al\.\(2025\)A\. Hua, K\. Tang, C\. Gu, J\. Gu, E\. Wong, and Y\. QinFlaw or artifact? Rethinking prompt sensitivity in evaluating LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 19889–19899\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1006/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1006),ISBN 979\-8\-89176\-332\-6Cited by:[§A\.5](https://arxiv.org/html/2610.11132#A1.SS5.p2.1)\.
- Huanet al\.\(2026\)M\. Z\. Huan, Y\. Li, T\. Zheng, X\. Xu, S\. Kim, M\. Du, R\. Poovendran, G\. Neubig, and X\. YueDoes math reasoning improve general LLM capabilities? Understanding transferability of LLM reasoning\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=GbOD25IA88)Cited by:[§2\.2](https://arxiv.org/html/2610.11132#S2.SS2.tab1.6.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 58791–58831\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/94074dd5a072d28ff75a76dabed43767-Paper-Conference.pdf)Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p6.1),[§1](https://arxiv.org/html/2610.11132#S1.p4.1),[2nd item](https://arxiv.org/html/2610.11132#S2.I1.i2.p1.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Kiet al\.\(2026\)K\. Ki, Y\. Nam, J\. Jeong, and J\. KimCPR for LLMs: critical\-point routing against catastrophic forgetting in domain adaptation\.arXiv preprint arXiv:2608\.30158\.Cited by:[Appendix H](https://arxiv.org/html/2610.11132#A8.p1.1),[Appendix H](https://arxiv.org/html/2610.11132#A8.p2.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px2.p2.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p1.1)\.
- Kokelet al\.\(2025\)H\. Kokel, M\. Katz, K\. Srinivas, and S\. SohrabiACPBench: reasoning about action, change, and planning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 26559–26568\.Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p11.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Liet al\.\(2023\)X\. L\. Li, A\. Holtzman, D\. Fried, P\. Liang, J\. Eisner, T\. Hashimoto, L\. Zettlemoyer, and M\. LewisContrastive decoding: open\-ended text generation as optimization\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 12286–12312\.External Links:[Link](https://aclanthology.org/2023.acl-long.687/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.687)Cited by:[Appendix H](https://arxiv.org/html/2610.11132#A8.p1.1)\.
- Li and Hoiem \(2017\)Z\. Li and D\. HoiemLearning without forgetting\.IEEE Transactions on Pattern Analysis and Machine Intelligence40\(12\),pp\. 2935–2947\.Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)J\. Lin, Z\. Wang, K\. Qian, T\. Wang, A\. Srinivasan, H\. Zeng, R\. Jiao, X\. Zhou, J\. Gesi, D\. Wang, Y\. Guo, K\. Zhong, W\. Zhang, S\. Sanghavi, C\. Chen, H\. Yun, and L\. LiSFT doesn’t always hurt general capabilities: revisiting domain\-specific fine\-tuning in LLMs\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 46954–46992\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/4e447acb68f57e29234bc0eb19896f11-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024a\)A\. Liu, X\. Han, Y\. Wang, Y\. Tsvetkov, Y\. Choi, and N\. A\. SmithTuning language models by proxy\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=dribhnhm1i)Cited by:[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024b\)C\. Liu, Y\. Kang, S\. Wang, L\. Qing, F\. Zhao, C\. Wu, C\. Sun, K\. Kuang, and F\. WuMore than catastrophic forgetting: integrating general capabilities for domain\-specific LLMs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7531–7548\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.429/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.429)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, Y\. Chen, M\. Shoeybi, B\. Catanzaro, and W\. PingAceMath: advancing frontier math reasoning with post\-training and reward modeling\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3993–4015\.External Links:[Link](https://aclanthology.org/2025.findings-acl.206/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.206),ISBN 979\-8\-89176\-256\-5Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.10.1)\.
- Lopez\-Paz and Ranzato \(2017\)D\. Lopez\-Paz and M\. RanzatoGradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2025\)M\. Luo, S\. Tan, J\. Wong, X\. Shi, W\. Y\. Tang, M\. Roongta, C\. Cai, J\. Luo, L\. E\. Li, R\. A\. Popa, and I\. StoicaDeepScaleR: surpassing o1\-preview with a 1\.5B model by scaling RL\.Note:[https://pretty\-radio\-b75\.notion\.site/DeepScaleR\-Surpassing\-O1\-Preview\-with\-a\-1\-5B\-Model\-by\-Scaling\-RL\-19681902c1468005bed8ca303013a4e2](https://pretty-radio-b75.notion.site/DeepScaleR-Surpassing-O1-Preview-with-a-1-5B-Model-by-Scaling-RL-19681902c1468005bed8ca303013a4e2)Notion BlogCited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.2.1)\.
- Mazeikaet al\.\(2024\)M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li, D\. Forsyth, and D\. HendrycksHarmBench: a standardized evaluation framework for automated red teaming and robust refusal\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 35181–35224\.External Links:[Link](https://proceedings.mlr.press/v235/mazeika24a.html)Cited by:[Appendix F](https://arxiv.org/html/2610.11132#A6.p2.1)\.
- Mitchellet al\.\(2024\)E\. Mitchell, R\. Rafailov, A\. Sharma, C\. Finn, and C\. D\. ManningAn emulator for fine\-tuning large language models using small language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Eo7kv0sllr)Cited by:[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px2.p1.1)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, Z\. Yang, W\. Shi, X\. L\. Li, L\. Fei\-Fei, H\. Hajishirzi, L\. Zettlemoyer, P\. Liang, E\. Candès, and T\. HashimotoS1: simple test\-time scaling\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 20275–20321\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1025/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1025),ISBN 979\-8\-89176\-332\-6Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.8.2)\.
- Penedoet al\.\(2025\)G\. Penedo, A\. Lozhkov, H\. Kydlíček, L\. B\. Allal, E\. Beeching, A\. P\. Lajarín, Q\. Gallouédec, N\. Habib, L\. Tunstall, and L\. von WerraOlympicCoder\.Hugging Face\.Note:[https://huggingface\.co/open\-r1/OlympicCoder\-7B](https://huggingface.co/open-r1/OlympicCoder-7B)Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p6.1),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.12.2),[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.17.2)\.
- Qwen Team \(2024a\)Qwen TeamQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.p1.1)\.
- Qwen Team \(2024b\)Qwen TeamQwen2\.5\-Coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.p1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§A\.3](https://arxiv.org/html/2610.11132#A1.SS3.SSS0.Px1.p1.1)\.
- Rajpurkaret al\.\(2018\)P\. Rajpurkar, R\. Jia, and P\. LiangKnow what you don’t know: unanswerable questions for SQuAD\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),I\. Gurevych and Y\. Miyao \(Eds\.\),Melbourne, Australia,pp\. 784–789\.External Links:[Link](https://aclanthology.org/P18-2124/),[Document](https://dx.doi.org/10.18653/v1/P18-2124)Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p8.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Reddyet al\.\(2019\)S\. Reddy, D\. Chen, and C\. D\. ManningCoQA: a conversational question answering challenge\.Transactions of the Association for Computational Linguistics7,pp\. 249–266\.Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p10.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Shiet al\.\(2023\)F\. Shi, M\. Suzgun, M\. Freitag, X\. Wang, S\. Srivats, S\. Vosoughi, H\. W\. Chung, Y\. Tay, S\. Ruder, D\. Zhou, D\. Das, and J\. WeiLanguage models are multilingual chain\-of\-thought reasoners\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=fR3wGCk-IXp)Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p5.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.
- Songet al\.\(2025\)M\. Song, M\. Zheng, Z\. Li, W\. Yang, and X\. LuoFastCuRL: curriculum reinforcement learning with stage\-wise context scaling for efficient training R1\-like reasoning models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 8856–8866\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.470/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.470),ISBN 979\-8\-89176\-335\-7Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.3.1),[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.4.1),[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.5.1)\.
- Sunet al\.\(2020\)F\. Sun, C\. Ho, and H\. LeeLAMOL: language modeling for lifelong language learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Skgxcn4YDS)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p3.1),[§6](https://arxiv.org/html/2610.11132#S6.SS0.SSS0.Px1.p1.1)\.
- Vuet al\.\(2022\)T\. Vu, A\. Barua, B\. Lester, D\. Cer, M\. Iyyer, and N\. ConstantOvercoming catastrophic forgetting in zero\-shot cross\-lingual generation\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 9279–9300\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.630/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.630)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p1.1)\.
- Xieet al\.\(2022\)S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. MaAn explanation of in\-context learning as implicit Bayesian inference\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RdJVFCHjUMI)Cited by:[§K\.1](https://arxiv.org/html/2610.11132#A11.SS1.p2.1),[§K\.3](https://arxiv.org/html/2610.11132#A11.SS3.p3.1),[§1](https://arxiv.org/html/2610.11132#S1.p6.1),[item 3](https://arxiv.org/html/2610.11132#S4.I1.i3.p1.1),[§4\.1](https://arxiv.org/html/2610.11132#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2610.11132#S4.SS1.p3.2)\.
- Yeet al\.\(2025\)Y\. Ye, Z\. Huang, Y\. Xiao, E\. Chern, S\. Xia, and P\. LiuLIMO: less is more for reasoning\.InSecond Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=T2TZ0RY4Zk)Cited by:[§A\.2](https://arxiv.org/html/2610.11132#A1.SS2.tab1.3.9.2)\.
- Yuet al\.\(2026\)C\. Yu, Y\. Wang, Z\. Guo, H\. Lin, S\. Xu, H\. Zang, Q\. Zhang, Y\. Wu, C\. Zhu, J\. Hu, Z\. Huang, M\. Wei, Y\. Xie, K\. Yang, B\. Dai, Z\. Xu, J\. Du, X\. Wang, X\. Fu, L\. Shi, Z\. Liu, K\. Chen, W\. Liu, G\. Liu, B\. Li, J\. Yang, Z\. Yang, G\. Dai, and Y\. WangRLinf: flexible and efficient large\-scale reinforcement learning via macro\-to\-micro flow transformation\.In20th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 26\),Seattle, WA,pp\. 829–846\.External Links:ISBN 978\-1\-939133\-55\-7,[Link](https://www.usenix.org/conference/osdi26/presentation/yu-chao)Cited by:[Table 5](https://arxiv.org/html/2610.11132#A5.T5.4.9.1)\.
- Zenget al\.\(2024\)A\. Zeng, M\. Liu, R\. Lu, B\. Wang, X\. Liu, Y\. Dong, and J\. TangAgentTuning: enabling generalized agent abilities for LLMs\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3053–3077\.External Links:[Link](https://aclanthology.org/2024.findings-acl.181/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.181)Cited by:[§1](https://arxiv.org/html/2610.11132#S1.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.Cited by:[§A\.1](https://arxiv.org/html/2610.11132#A1.SS1.p4.1),[§2\.1](https://arxiv.org/html/2610.11132#S2.SS1.p3.1)\.

## Appendix

## Appendix AExperiment Details

We provide more experiment details in this section\.

### A\.1Benchmarks

In this work, we use the following benchmarks:

American Invitational Mathematics Examination 2024\(AIME 2024\) contains 30 math questions\. Each question requires multi\-step reasoning to solve\. The answer to each question is an integer from 000 to 999\.

MathIF\([Fu et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib14)\)is a benchmark for evaluating the instruction\-following capability of reasoning models\. It consists of 420 questions drawn from various math datasets, and each instruction contains Python\-verifiable constraints from four categories \(length, lexical, format, affix\)\.

IFEval\([Zhou et al\., 2023](https://arxiv.org/html/2610.11132#bib.bib23)\)is a benchmark that evaluates the instruction\-following capability of language models\. The benchmark contains explicit, verifiable instructions, such as constraints on formatting, length, keywords, and response structure\.

MGSM\([Shi et al\., 2023](https://arxiv.org/html/2610.11132#bib.bib13)\)is a multilingual math benchmark that contains translated versions of the 250 grade\-school math problems from the GSM8K dataset\([Cobbe et al\., 2021](https://arxiv.org/html/2610.11132#bib.bib24)\)in 10 languages\.

LiveCodeBench\([Jain et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib15)\)is a coding benchmark that uses continuously updated questions to evaluate coding models\. In this work, we use thev4\_v5subset of LiveCodeBench, containing 268 questions\. This choice is consistent with the version originally used by the OlympicCoder\-7B and OlympicCoder\-32B model series\([Penedo et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib25)\)\.

LiveCodeBenchIFis a benchmark we constructed, in a similar fashion to MathIF, where we append a short formatting requirement to the end of the original questions from LiveCodeBench\. To use a Python\-verifiable constraint and to minimize the impact of the formatting instruction on the code structure, we only specify the number of comment lines to be 3 in the formatting instruction \(Appendix[B\.1](https://arxiv.org/html/2610.11132#A2.SS1)\)\.

SQuAD2\.0\([Rajpurkar et al\., 2018](https://arxiv.org/html/2610.11132#bib.bib16)\)is a reading comprehension benchmark\. The answer to every question is a segment of text from the corresponding reading passage, or the question might be unanswerable\. We use a random sample of 1,187 \(10% of the total size\) questions from the validation set\. Of the 1,187 questions, 594 are answerable, and 593 are unanswerable\.

FQuAD2\.0\([Heinrich et al\., 2022](https://arxiv.org/html/2610.11132#bib.bib17)\)is a French\-native reading comprehension benchmark, similar to SQuAD2\.0\. The benchmark contains 800 questions in its test set, with 400 answerable and 400 unanswerable questions\.

CoQA\([Reddy et al\., 2019](https://arxiv.org/html/2610.11132#bib.bib18)\)is a benchmark designed to evaluate a model’s ability to answer questions in the context of an ongoing conversation\. We use the 500 questions in the public validation dataset\. Following the implementation of the Language Model Evaluation Harness\([Gao et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib26)\), we evaluate the answer of the last dialogue turn\.

ACPBench\([Kokel et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib19)\)is a benchmark designed to evaluate the reasoning capabilities of large language models across action, change, and planning\. The benchmark consists of 7 reasoning tasks over 13 domains, requiring both single\-step and multi\-step reasoning capabilities\. We evaluate on the officially released version where the answer is a boolean value\.

We construct three evaluation subsets based on NutriBench\([Dhaliwal et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib22)\)\.

- •NutriBench\-Englishis sampled directly from the original English NutriBench data\. We randomly sample 2,000 English queries\.
- •NutriBench\-Non\-Englishis constructed using GPT\-4o translations based on food items from the WHO source used in NutriBench\. We sample 2,018 non\-English queries to maximize language coverage, randomly sampling up to 200 queries per language and including all available queries for languages with fewer than 200 queries\.
- •NutriBench\-Non\-Fooduses IFEval queries as a proxy for non\-food inputs\. To simulate deployment scenarios where users may provide non\-food queries, we insert IFEval queries into the nutrition prediction template\.

### A\.2List of SFT Models

[SectionA\.2](https://arxiv.org/html/2610.11132#A1.SS2)shows the pairs of parent and SFT models we use for our experiments\. For math and coding, the parent models are instruction\-tuned versions from the Qwen2\.5 family\([Qwen Team, 2024a](https://arxiv.org/html/2610.11132#bib.bib33)\)and Qwen2\.5\-Coder family\([Qwen Team, 2024b](https://arxiv.org/html/2610.11132#bib.bib34)\)\. More details for nutrition estimation model pairs are provided in Appendix[A\.3](https://arxiv.org/html/2610.11132#A1.SS3)\.

As a conceptual shorthand, we use “pretrained model” throughout our discussion to refer to the off\-the\-shelf instruction\-tuned model that serves as the starting point for the task\-specific supervised fine\-tuning\. We note that this differs from the stricter usage of*pretrained*to denote a base model prior to instruction tuning\.

Table 4:Parent and SFT model pairs used in math, coding, and nutrition experiments\.All parent models are instruction\-tuned\. Models are listed in the order of increasing number of parameters\. We use the abbreviation OCR for OpenCodeReasoning in the main paper, consistent with the choice of the original paper\.### A\.3Nutrition Estimation Model Training

#### Models\.

We fine\-tune three open\-weight instruction\-tuned models with full\-parameter SFT \(no LoRA or other adapters\):gemma\-4\-E4B\-it\([Gemma Team, 2026](https://arxiv.org/html/2610.11132#bib.bib54)\),Qwen3\-4B, andQwen3\-8B\([Qwen Team, 2025](https://arxiv.org/html/2610.11132#bib.bib55)\)\.

#### Training data\.

All runs use the same data from NutriBench\([Dhaliwal et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib22)\)\. We sampled 40,000 English\-only meal descriptions for training and 1,000 development examples\.

#### Optimization\.

We train in bfloat16 with gradient checkpointing on 8×\\timesNVIDIA A100 80 GB GPUs using FSDP\. The per\-device batch size is 2 with 16 gradient\-accumulation steps \(effective batch 256\), and we use AdamW with a learning rate of1×10−51\\times 10^\{\-5\}, no weight decay, a linear schedule, a warmup ratio of 0\.03, and 10 epochs, i\.e\., 1,570 optimizer steps\.

### A\.4Inference Settings

For all benchmarks, we use a permissive decoding budget to leave the models sufficient room to generate complete responses\. We set the maximum total sequence length, including both the prompt and response, to 32,768 tokens, substantially higher than the typical generation limits used for the selected benchmarks\. We adopt this large budget because the math and coding SFT models are predominantly fine\-tuned on long reasoning traces\. For NutriBench only, we additionally cap the maximum number of response tokens at 8,192\.

For each benchmark, we use the same decoding parameters across all models to ensure a fair comparison\. We use greedy decoding for most benchmarks\. For benchmarks where sampling is commonly used, we follow the officially recommended sampling parameters\. For these non\-greedy benchmarks, we report performance averaged over three random seeds\. For SFT\-as\-context, each of the three inference runs uses a different context generated by the SFT model\.

### A\.5Metrics

For most of the benchmarks, we use the official evaluation setup\. For IFEval, we report the prompt\-level strict accuracy according to the official evaluation\. For MathIF, we report Hard IF according to the official evaluation\.

For SQuAD2\.0, FQuAD2\.0, and CoQA, we use LLM\-as\-a\-judge to avoid the unknown biases introduced by logit\-based evaluation\([Hua et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib7)\)\. We use the LLM\-as\-a\-judge prompt recommended by[Hua et al\. \(2025\)](https://arxiv.org/html/2610.11132#bib.bib7)and Gemini 3\.5 Flash as the judge\.

For MGSM, we check for the rate of using the answer prefix in the correct language\. The prefix is the word for “answer” in each non\-English language\.

For NutriBench, macro MAE is defined as the mean absolute error \(MAE\) averaged across carbohydrates, fat, protein, and energy\. For language consistency and acknowledgment of non\-food queries, we heuristically check the presence of phrases such as “contribute about” and “contributes about” in the response, which indicates an English response and a hallucinated response for these two categories, respectively\.

## Appendix BPrompts

We provide the task prompts and SFT\-as\-context prompts in this section\.

### B\.1Task Prompts

Other than LiveCodeBenchIF and NutriBench, we use the official prompt template for evaluation of each benchmark\. The custom prompt for LiveCodeBenchIF is shown below, where the only addition other than the original task prompt is a simple sentence asking for 3 lines of comments\.

LiveCodeBenchIF Prompt Template\{original\_task\_prompt\} Use exactly three lines of comments in your code\.

The prompt template for NutriBench is shown below\. The meal description is inserted at \{meal\_description\}\. In real deployment, non\-food queries may be inserted at \{meal\_description\} as well, and the model is expected to respond with a human\_reasoning field that acknowledges that there is no food in the query\. The prompt template explicitly asks for language consistency and the acknowledgment of non\-food queries\. However, with this prompt for training and inference, SFT models fail to preserve these general capabilities\.

NutriBench Prompt TemplateYou are a nutrition estimation assistant\. Your task is to estimate the total nutritional content of the following food or meal description\. Food or meal description: “\{meal\_description\}” Before estimating, first determine what kind of input you received\. If the description is written in a language other than English, write the entire \`human\_reasoning\` in that same language without switching to English\. If the input is unrelated to food or nutrition, explicitly identify it as irrelevant, do not invent a meal, and set all four numeric fields to 0\. If it mentions food but provides no quantity or serving size, explicitly state in \`human\_reasoning\` that the amount is unspecified and that the numerical result requires an assumption; then provide your best estimate\. Output format: \- You may write normal reasoning before the final prediction, including any internal matching, portion, scaling, and summing rationale\. \- End your response with the final prediction JSON object wrapped exactly in these tags: <nutrition\_estimate\_json\> \{ “human\_reasoning”: “brief explanation for a human, in the same language as the meal description”, “carbohydrate\_g”: number, “fat\_g”: number, “protein\_g”: number, “energy\_kcal”: number \} </nutrition\_estimate\_json\> \- Keep “human\_reasoning” compact, but include the nutrition contribution of each main component\. Do not merely say that items were estimated and summed\. \- For multi\-component meals, use a style like: “The meal consists of A, B, and C\. A contributes about …; B contributes about …; C contributes about …\.” \- The per\-component nutrition in “human\_reasoning” should cover carbohydrate, fat, protein, and energy when possible, using the same units as the JSON fields\. Guidelines: \- Estimate the total amount for the entire described food or meal\. \- Use grams for carbohydrate, fat, and protein\. \- Use kilocalories for energy\.

### B\.2SFT\-as\-Context Prompts

For SFT\-as\-context, we use two versions of prompts, one for NutriBench, another for math and coding models\. In these prompts, the \{task\_prompt\} is the original query in the benchmarks\. The nutrition estimation JSON replaces \{nutrition\_estimate\_json\}\. The model response after stripping the thinking block replaces \{sft\_response\_thinking\_stripped\}\.

For math and coding, the original task prompt is repeated at the end of the SFT\-as\-context instructions\. This design choice is justified by an additional ablation study in Appendix[J](https://arxiv.org/html/2610.11132#A10)\.

NutriBench SFT\-as\-Context Prompt Template\{task\_prompt\}The candidate below usually gives a better nutrition estimate than an independent guess\. Treat it as the default answer\. Use its numeric values unless there is an obvious hard error: it is for a different food, it ignores a major listed component, it uses a quantity that conflicts with the original description, or its energy is impossible for the stated amount\. Do not adjust merely because your own rough estimate is different; only override a specific candidate value when you can point to the original text that makes it impossible\. \{nutrition\_estimate\_json\}

Math and Coding SFT\-as\-Context Prompt TemplateProduce a response that fully satisfies the original task\.<task\> \{task\_prompt\} </task\> <reference\> \{sft\_response\_thinking\_stripped\} </reference\> Treat the original task as the authoritative specification\. Treat the reference as a candidate final answer, not as a draft that needs improvement\. If the reference already satisfies the original task, return it unchanged\. Only change it when you identify a concrete correctness error or a violation of an explicit requirement in the original task\. In that case, make the smallest necessary correction and preserve everything else\. Preserve the reference’s scope, granularity, format, and working approach\. Do not add optional material, rewrite for style, broaden the answer, or replace a working solution merely because another solution seems preferable\. The final response must be self\-contained and must follow every requirement of the original task\. Return only that response without mentioning the reference or this process\. Again, the original task is: <task\> \{task\_prompt\} </task\> Use the reference unchanged unless a specific error or explicit requirement violation requires a minimal correction\. Return only the final response\.

Additionally, to prevent degenerate or excessively long SFT responses from exhausting the parent model’s available context window, we omit the SFT response if any of the following conditions holds:

1. 1\.The SFT model exhausts its allocated token budget without producing a final answer\.
2. 2\.The SFT response exhibits pathological repetition of token sequences\.
3. 3\.The combined length of the query, SFT response, and prompt template leaves no generation budget for the parent model\.

In these cases, we fall back to generating a response from the parent model using the original query alone\. The fallback rate is generally low under the permissive token budget\. For example, the fallback rate of MathIF is only 3\.73%, corresponding to 376 of 10,080 responses, across eight model pairs and seeds 0, 1, and 2\. The gaps on fine\-tuned and general capabilities are over 20 percentage points, much larger than 3\.73%\. Hence, the improved capabilities of SFT\-as\-context cannot be attributed to the fallback conditions\.

## Appendix CMore Examples of Forgetting and Mitigation

In this section, we first provide example queries and responses to demonstrate forgetting of SFT models and the mitigation by SFT\-as\-context \(Appendix[C\.1](https://arxiv.org/html/2610.11132#A3.SS1)\)\. Following the discussion in[Section3](https://arxiv.org/html/2610.11132#S3), we also provide examples from a case study of the largest remaining gap of general capability, which is from the MathIF benchmark \(Appendix[C\.2](https://arxiv.org/html/2610.11132#A3.SS2)\)\.

### C\.1MathIF and LiveCodeBenchIF General Failures

Here, we provide two examples, one from MathIF and one from LiveCodeBenchIF\. Each example contains four parts: the query, the parent model response, the SFT model response, and the SFT\-as\-context response\. In both examples, the parent models are weaker on fine\-tuned capabilities, providing incorrect answers\. The SFT models are weaker on general capabilities, ignoring the explicit formatting instructions\. The SFT\-as\-context method generates a response with the correct answer and formatting, outperforming both parent and SFT models\.

The MathIF example query, parent model response, SFT model response, and SFT\-as\-context response are shown below\. The parent model is Qwen2\.5\-32B\-Instruct, and the SFT model is OpenThinker2\-32B\. The parent uses the correct remainder\-theorem strategy but incorrectly computesf⁡\(1\)=−5f\(1\)=\-5andf⁡\(−1\)=11f\(\-1\)=11, leading to a wrong answer of−8​x\+3\-8x\+3\. Its response is lowercase, satisfying the formatting constraint\. The SFT model correctly computes the answer but uses uppercase letters both in prose and notations \(using the uppercaseRRfor remainder\)\. SFT\-as\-context preserves the correct derivation and result, while using lowercase prose and notationrr\.

MathIF Example QueryLetf⁡\(x\)=x10\+5​x9−8​x8\+7​x7−x6−12​x5\+4​x4−8​x3\+12​x2−5​x−5f\(x\)=x^\{10\}\+5x^\{9\}\-8x^\{8\}\+7x^\{7\}\-x^\{6\}\-12x^\{5\}\+4x^\{4\}\-8x^\{3\}\+12x^\{2\}\-5x\-5\. Without using long division \(which would be horribly nasty\!\), find the remainder whenf⁡\(x\)f\(x\)is divided byx2−1x^\{2\}\-1\. Let’s think step by step and output the final answer within \\boxed\{\}\. Your entire response should be in English, and in all lowercase letters\. No capital letters are allowed\.

Parent Responsewe start by recognizing that when we divide a polynomialf⁡\(x\)f\(x\)byx2−1x^\{2\}\-1, the remainderr⁡\(x\)r\(x\)will be a polynomial of degree less than 2\. thus, we can expressr⁡\(x\)r\(x\)asa​x\+bax\+b, whereaaandbbare constants to be determined\.sincef⁡\(x\)=\(x2−1\)​q​\(x\)\+r⁡\(x\)f\(x\)=\(x^\{2\}\-1\)q\(x\)\+r\(x\)for some quotientq⁡\(x\)q\(x\)and remainderr⁡\(x\)=a​x\+br\(x\)=ax\+b, we can substitutex=1x=1andx=−1x=\-1intof⁡\(x\)f\(x\)to solve foraaandbb\. this is becausex2−1x^\{2\}\-1becomes 0 whenx=1x=1orx=−1x=\-1, simplifying the equation to justr⁡\(x\)r\(x\)\.substitutingx=1x=1:f⁡\(1\)=\(1\)10\+5​\(1\)9−8​\(1\)8\+7​\(1\)7−\(1\)6f\(1\)=\(1\)^\{10\}\+5\(1\)^\{9\}\-8\(1\)^\{8\}\+7\(1\)^\{7\}\-\(1\)^\{6\}−12​\(1\)5\+4​\(1\)4−8​\(1\)3\+12​\(1\)2−5​\(1\)−5=a⁡\(1\)\+b\-12\(1\)^\{5\}\+4\(1\)^\{4\}\-8\(1\)^\{3\}\+12\(1\)^\{2\}\-5\(1\)\-5=a\(1\)\+bf⁡\(1\)=1\+5−8\+7−1−12\+4−8\+12−5−5=a\+bf\(1\)=1\+5\-8\+7\-1\-12\+4\-8\+12\-5\-5=a\+bf⁡\(1\)=−5=a\+bf\(1\)=\-5=a\+bsubstitutingx=−1x=\-1:f⁡\(−1\)=\(−1\)10\+5​\(−1\)9−8​\(−1\)8\+7​\(−1\)7−\(−1\)6−12​\(−1\)5f\(\-1\)=\(\-1\)^\{10\}\+5\(\-1\)^\{9\}\-8\(\-1\)^\{8\}\+7\(\-1\)^\{7\}\-\(\-1\)^\{6\}\-12\(\-1\)^\{5\}\+4​\(−1\)4−8​\(−1\)3\+12​\(−1\)2−5​\(−1\)−5=a⁡\(−1\)\+b\+4\(\-1\)^\{4\}\-8\(\-1\)^\{3\}\+12\(\-1\)^\{2\}\-5\(\-1\)\-5=a\(\-1\)\+bf⁡\(−1\)=1−5−8−7−1\+12\+4\+8\+12\+5−5=−a\+bf\(\-1\)=1\-5\-8\-7\-1\+12\+4\+8\+12\+5\-5=\-a\+bf⁡\(−1\)=11=−a\+bf\(\-1\)=11=\-a\+bnow we have a system of equations:a\+b=−5a\+b=\-5−a\+b=11\-a\+b=11adding these equations:\(a\+b\)\+\(−a\+b\)=−5\+11\(a\+b\)\+\(\-a\+b\)=\-5\+112​b=62b=6b=3b=3substitutingb=3b=3into one of the original equations:a\+3=−5a\+3=\-5a=−8a=\-8thus, the remainder whenf⁡\(x\)f\(x\)is divided byx2−1x^\{2\}\-1is−8​x\+3\-8x\+3\.−8​x\+3\\boxed\{\-8x\+3\}

SFT ResponseTo find the remainder whenf⁡\(x\)f\(x\)is divided byx2−1x^\{2\}\-1, we use the Remainder Theorem for quadratic divisors\. The remainder will be a linear polynomialR⁡\(x\)=a​x\+bR\(x\)=ax\+b\. Evaluatingf⁡\(x\)f\(x\)at the roots ofx2−1x^\{2\}\-1\(which arex=1x=1andx=−1x=\-1\) gives us:\{f⁡\(1\)=a⁡\(1\)\+b=a\+b=−10f⁡\(−1\)=a⁡\(−1\)\+b=−a\+b=16\\begin\{cases\}f\(1\)=a\(1\)\+b=a\+b=\-10\\\\ f\(\-1\)=a\(\-1\)\+b=\-a\+b=16\\end\{cases\}Solving this system:1\. Subtract the second equation from the first:\(a\+b\)−\(−a\+b\)=−10−16⟹2​a=−26⟹a=−13\(a\+b\)\-\(\-a\+b\)=\-10\-16\\implies 2a=\-26\\implies a=\-132\. Substitutea=−13a=\-13intoa\+b=−10a\+b=\-10:−13\+b=−10⟹b=3\-13\+b=\-10\\implies b=3Thus, the remainder isR⁡\(x\)=−13​x\+3R\(x\)=\-13x\+3\.−13​x\+3\\boxed\{\-13x\+3\}

SFT\-as\-Context Responseto find the remainder whenf⁡\(x\)f\(x\)is divided byx2−1x^\{2\}\-1, we use the remainder theorem for quadratic divisors\. the remainder will be a linear polynomialr⁡\(x\)=a​x\+br\(x\)=ax\+b\. evaluatingf⁡\(x\)f\(x\)at the roots ofx2−1x^\{2\}\-1\(which arex=1x=1andx=−1x=\-1\) gives us:\{f⁡\(1\)=a⁡\(1\)\+b=a\+b=−10f⁡\(−1\)=a⁡\(−1\)\+b=−a\+b=16\\begin\{cases\}f\(1\)=a\(1\)\+b=a\+b=\-10\\\\ f\(\-1\)=a\(\-1\)\+b=\-a\+b=16\\end\{cases\}solving this system:1\. subtract the second equation from the first:\(a\+b\)−\(−a\+b\)=−10−16⟹2​a=−26⟹a=−13\(a\+b\)\-\(\-a\+b\)=\-10\-16\\implies 2a=\-26\\implies a=\-132\. substitutea=−13a=\-13intoa\+b=−10a\+b=\-10:−13\+b=−10⟹b=3\-13\+b=\-10\\implies b=3thus, the remainder isr⁡\(x\)=−13​x\+3r\(x\)=\-13x\+3\.−13​x\+3\\boxed\{\-13x\+3\}

The LiveCodeBenchIF example query, parent model response, SFT model response, and SFT\-as\-context response are shown below\. The parent model is Qwen2\.5\-32B\-Instruct, and the SFT model is OpenCodeReasoning\-Nemotron\-32B\. In the example query, the last sentence specifying the number of comments is the only addition from us to the original problem description \(Appendix[A\.1](https://arxiv.org/html/2610.11132#A1.SS1)\)\. The parent response has three comment lines but sorts integers descending withnums\.sort\(reverse=True\)\. For \[1,2,3\], this gives 11\|10\|1 = 11101 \(29\), while the correct maximum is 11\|1\|10 = 11110 \(30\)\. The SFT response is correct but fully ignores the instruction of using 3 comments\. The SFT\-as\-context response keeps the same executable Python structure from the SFT context and adds exactly three comment lines\.

LiveCodeBenchIF Example QueryYou will be given a question \(problem specification\) and will generate a correct Python program that matches the specification and passes all tests\. Question: You are given an array of integers nums of size 3\. Return the maximum possible number whose binary representation can be formed by concatenating the binary representation of all elements in nums in some order\. Note that the binary representation of any number does not contain leading zeros\. Example 1: Input: nums = \[1,2,3\] Output: 30 Explanation: Concatenate the numbers in the order \[3, 1, 2\] to get the result “11110”, which is the binary representation of 30\. Example 2: Input: nums = \[2,8,16\] Output: 1296 Explanation: Concatenate the numbers in the order \[2, 8, 16\] to get the result “10100010000”, which is the binary representation of 1296\. Constraints: nums\.length == 3 1 <= nums\[i\] <= 127 You will use the following starter code to write the solution to the problem and enclose your code within delimiters\. \`\`\`python class Solution: def maxGoodNumber\(self, nums: List\[int\]\) \-\> int: \`\`\` Use exactly three lines of comments in your code\.

Parent Response

SFT Response

SFT\-as\-Context Response

### C\.2SFT\-as\-Context Failure Mode: Boxed Answer Only

We provide two examples where Qwen2\.5\-7B\-Instruct fails to follow the original task instructions during SFT\-as\-context and only outputs a boxed answer\.

In the first example below, only outputting the boxed answer violates the requirement of using the word “number\.”

Example Query 1Zeke’s baseball team has 7 more players than Carlton’s\. If Carlton’s team has 13 players, how many players are there in both teams combined? Let’s think step by step and output the final answer after “\#\#\#\#”\. Let’s think step by step and output the final answer within \\boxed\{\}\. In your response, the word “number” should appear at least 1 times\. In your entire response, refrain from the use of any commas\.

Parent ResponseCarlton’s team number 13 players; Zeke’s has 7 more making it 13\+7=20 players; Combined both teams have 13\+20=33 players \#\#\#\#33

SFT ResponseFirst find Zeke’s team’s number of players:13\+7=20\. Then add both teams:20\+13=33\.\#\#\#\#33\#\#\#\#

SFT\-as\-Context Response33\#\#\#\#

In the second example below, only outputting the boxed answer violates the requirement of highlighting at least two sections\.

Example Query 2A pet store currently has 5 dogs, 2 cats, and 10 birds\. How many legs in total do the pets in the store have? Let’s think step by step and output the final answer after “\#\#\#\#”\. Let’s think step by step and output the final answer within \\boxed\{\}\. Highlight at least 2 sections in your answer with markdown, i\.e\. \*highlighted section\*\.

Parent ResponseSure, let’s break this down step by step\. First, we need to determine the number of legs each type of pet has: \- Dogs have 4 legs each\. \- Cats have 4 legs each\. \- Birds have 2 legs each\. Next, we calculate the total number of legs for each type of pet: \- \*For the dogs\*: There are 5 dogs, and each dog has 4 legs\. Therefore, the total number of legs for the dogs is5×4=205\\times 4=20\. \- \*For the cats\*: There are 2 cats, and each cat has 4 legs\. Therefore, the total number of legs for the cats is2×4=82\\times 4=8\. \- \*For the birds\*: There are 10 birds, and each bird has 2 legs\. Therefore, the total number of legs for the birds is10×2=2010\\times 2=20\. Finally, we sum the total number of legs from all the pets: \- The total number of legs is20\+8\+20=4820\+8\+20=48\. Therefore, the total number of legs the pets in the store have is48\.

SFT ResponseThe pet store has three types of pets: dogs, cats, and birds\. \*\*1\. Calculating legs for dogs:\*\* There are 5 dogs, each with 4 legs\.5​dogs×4​legs/dog=20​legs5\\text\{ dogs\}\\times 4\\text\{ legs/dog\}=20\\text\{ legs\}\*\*2\. Calculating legs for cats:\*\* There are 2 cats, each with 4 legs\.2​cats×4​legs/cat=8​legs2\\text\{ cats\}\\times 4\\text\{ legs/cat\}=8\\text\{ legs\}\*\*3\. Calculating legs for birds:\*\* There are 10 birds, each with 2 legs\.10​birds×2​legs/bird=20​legs10\\text\{ birds\}\\times 2\\text\{ legs/bird\}=20\\text\{ legs\}\*\*Total legs:\*\* 20​\(dogs\)\+8​\(cats\)\+20​\(birds\)=48​legs20\\text\{ \(dogs\)\}\+8\\text\{ \(cats\)\}\+20\\text\{ \(birds\)\}=48\\text\{ legs\}\*\*Final Answer:\*\* 48

SFT\-as\-Context Response\#\#\#\# 48

## Appendix DAttention Visualization

To investigate how strongly the model attends to the provided context, we analyze the attention weights assigned to prompt tokens during response generation\. For each response token, we extract its attention weights over all prompt tokens \(which we refer to as “prompt\-directed attention”\), excluding attention to preceding response tokens\. We then normalize these weights over the prompt tokens so that they sum to 100%, with each prompt token’s share proportional to its attention weight\. Finally, we average the normalized prompt\-directed attention over response tokens, attention heads, and transformer layers\. Chat wrappers and special tokens are excluded from the calculation\.

Formally, the share of prompt\-directed attention is calculated as follows\.

For each exampleee, the prompt is split into three regions: the queryQQ, the SFT\-as\-context instructionsII, and the SFT contextCC\. LetAe,ℓ,h,i,jA\_\{e,\\ell,h,i,j\}denote the attention weight assigned to prompt tokeniiby headhhin layerℓ\\ell, for generating response tokenjj\. Letwe,i,R∈\{0,1\}w\_\{e,i,R\}\\in\\\{0,1\\\}be a weight that indicates which regionR∈\{Q,I,C\}R\\in\\\{Q,I,C\\\}a prompt tokeniibelongs to\. This weight is11for a token insideRRand00for a token outsideRR\.

We first sum the attention assigned to each region:

me,ℓ,h,j​\(R\)=∑iwe,i,R​Ae,ℓ,h,i,j,R∈\{Q,I,C\}\.m\_\{e,\\ell,h,j\}\(R\)=\\sum\_\{i\}w\_\{e,i,R\}\\,A\_\{e,\\ell,h,i,j\},\\qquad R\\in\\\{Q,I,C\\\}\.The total attention directed to the prompt is then

de,ℓ,h,j=me,ℓ,h,j​\(Q\)\+me,ℓ,h,j​\(I\)\+me,ℓ,h,j​\(C\)\.d\_\{e,\\ell,h,j\}=m\_\{e,\\ell,h,j\}\(Q\)\+m\_\{e,\\ell,h,j\}\(I\)\+m\_\{e,\\ell,h,j\}\(C\)\.The share of the attention weight for each prompt region, for each layer, head, and response tokenjjis then:

se,ℓ,h,j​\(R\)=me,ℓ,h,j​\(R\)de,ℓ,h,j\.s\_\{e,\\ell,h,j\}\(R\)=\\frac\{m\_\{e,\\ell,h,j\}\(R\)\}\{d\_\{e,\\ell,h,j\}\}\.
Next, letℒ\\mathcal\{L\}be the included attention layers,HHthe number of attention heads per layer, andJJthe set of response tokens\. The percentage share for each regionRRin one exampleeeis

Se​\(R\)=100\|ℒ\|​H​\|J\|​∑ℓ∈ℒ∑h=1H∑j∈Jse,ℓ,h,j​\(R\)\.S\_\{e\}\(R\)=\\frac\{100\}\{\|\\mathcal\{L\}\|\\,H\\,\|J\|\}\\sum\_\{\\ell\\in\\mathcal\{L\}\}\\sum\_\{h=1\}^\{H\}\\sum\_\{j\\in J\}s\_\{e,\\ell,h,j\}\(R\)\.We use all generated non\-special tokens, all eight heads \(H=8H=8\), and the seven full\-attention layersℒ=\{6,12,18,24,30,36,42\}\\mathcal\{L\}=\\\{6,12,18,24,30,36,42\\\}in Gemma\-4\-E4B\-it\.

Finally, for a category containingNNexamples, we average the per\-example percentages to obtain the final share of prompt\-directed attention for each region:

Scategory​\(R\)=1N​∑e=1NSe​\(R\)\.S\_\{\\mathrm\{category\}\}\(R\)=\\frac\{1\}\{N\}\\sum\_\{e=1\}^\{N\}S\_\{e\}\(R\)\.Here, the category is relevant or irrelevant, and the number of queries we sample for each category isN=100N=100\.

[Figure3](https://arxiv.org/html/2610.11132#A4.F3)shows the token\-level distribution of prompt\-directed attention\. For token\-level visualization, we treat each prompt token as an individual region, while keeping the calculation the same\. The parent model places greater attention on tokens in the original instruction specifying that non\-food queries should be flagged as irrelevant\. This observation provides a possible explanation for why SFT\-as\-context recovers general instruction\-following capabilities even when the context is generated by an SFT model that has lost these capabilities\.

![Refer to caption](https://arxiv.org/html/2610.11132v1/full_prompt_comparison.png)Figure 3:The parent model attends less to the SFT context when the context is hallucinated for non\-food queries\.These two examples show that the mean share of prompt\-directed attention is higher for useful context \(29\.95%\) than for hallucinated context \(15\.61%\)\. Words like “unrelated” and “irrelevant” are more attended to when the query does not contain a food, demonstrating that the parent model is not distracted by hallucinated context and adheres to the original task instructions\.
## Appendix ERL Models Forget Less

We have evaluated SFT\-as\-context in a controlled setting where the model providing the context is obtained through standard SFT\. Since SFT\-as\-context is agnostic to the post\-training method used to obtain the context\-providing model, we would like to further investigate whether it generalizes to model pairs produced using alternative post\-training methods\. Specifically, we consider the model pairs in[Table5](https://arxiv.org/html/2610.11132#A5.T5), where the parent models belong to the DeepSeek\-R1\-Distill\-Qwen family\([Guo et al\., 2025](https://arxiv.org/html/2610.11132#bib.bib9)\)and the child models are further trained with reinforcement learning \(RL\)\. However, consistent with prior work\([Chen et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib8)\), we find that RL post\-training induces negligible forgetting of instruction\-following capabilities, as measured by IFEval\. Hence, we do not apply SFT\-as\-context \(or RL\-as\-context in this case\) to close the gaps that are already small\.

Table 5:RL\-trained models do not show forgetting of instruction\-following capability compared to their parent, as measured by IFEval\.All RL models in this table share DeepSeek\-R1\-Distill\-Qwen of the corresponding size \(1\.5B or 7B\) as their parent model\. The metric is prompt\-level strict accuracy \(PSA\) of IFEval\.Δ\\Deltais RL minus parent in percentage points\.
## Appendix FA Case Study on Safety

An LLM could be adversarially fine\-tuned to remove safety guardrails\. Here, the fine\-tuned capability directly conflicts with its general capabilities\. This setting differs from the other experiments in this paper, where fine\-tuned and general capabilities are orthogonal\.

To examine whether SFT\-as\-context favors fine\-tuned or general capabilities under such a conflict, we use the widely used series of jailbroken models released by Huihui AI on Hugging Face\.222[https://huggingface\.co/huihui\-ai/models](https://huggingface.co/huihui-ai/models)Although these models use an abliteration process that is not SFT\([Arditi et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib35)\), we nevertheless simulate a setting where an SFT model provides harmful responses\. We test the parent and abliterated models on HarmBench\([Mazeika et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib58)\)\.[Table6](https://arxiv.org/html/2610.11132#A6.T6)shows that SFT\-as\-context largely preserves the harmful responses produced by the abliterated model\. In other words, when fine\-tuned and general capabilities conflict, the parent model follows the fine\-tuned capability under SFT\-as\-context\. One possible explanation is that our SFT\-as\-context instruction does not explicitly impose safety requirements\. Consequently, the parent model may focus on verifying whether the response satisfies the task requirements without first determining whether those requirements are safe to fulfill\. Future work could investigate how SFT\-as\-context instructions can be optimized to prioritize the recovery of specific general capabilities\.

Table 6:SFT\-as\-context fails to recover the safety capability of the parent model when the context is adversarial\.Abliterated models show far higher attack success rates \(ASR\) on HarmBench than their parent models\. SFT\-as\-context largely preserves the harmfulness in the context, with an average recovery rate of only 16\.44%\. The average recovery rate is calculated from the averaged scores\.
## Appendix GForgetting Cannot Be Mitigated by Oracle Routing

Given a pair of parent and SFT models, queries can be routed at inference time\. Queries requiring fine\-tuned capabilities can be answered by the SFT model, while those requiring general capabilities can be answered by the parent model\. However, real\-world queries often require both fine\-tuned and general capabilities simultaneously\. In such cases, routing between the two models may be insufficient, since each query is ultimately answered by only one model\.

To demonstrate this limitation, we compare SFT\-as\-context with an oracle router that selects the better of the responses generated by the parent and SFT models\. We evaluate performance using the joint success rate, which requires a response to satisfy both the fine\-tuned and general capabilities\. For MathIF, joint success requires both a correct answer and compliance with all constraints \(hard instruction\-following\)\. For LiveCodeBenchIF, it requires both Pass@1 and exactly three comment lines\. As shown in Tables[7](https://arxiv.org/html/2610.11132#A7.T7)and[8](https://arxiv.org/html/2610.11132#A7.T8), SFT\-as\-context outperforms even the oracle router\. These results highlight a fundamental limitation of routing\. Even with oracle selection between the parent and SFT models, neither model alone can reliably satisfy requirements that depend on both sets of capabilities\. In contrast, SFT\-as\-context enables the two capabilities to be combined within a single response\.

Table 7:MathIF joint\-success rates show that SFT\-as\-context outperforms the oracle router\.A response counts as a joint success when the answer is correct and Hard IF is satisfied\. Improvement is defined as SFT\-as\-context performance minus oracle performance\. Best results are bolded and second\-best results are underlined\.Table 8:LiveCodeBenchIF joint\-success rates show that SFT\-as\-context outperforms the oracle router\.A response counts as a joint success when the answer is correct and the number of comments is exactly three\. Improvement is defined as SFT\-as\-context performance minus oracle performance\. The only exception where the improvement is negative is OlympicCoder\-7B\. Best results are bolded and second\-best results are underlined\.
## Appendix HToken\-Level Manipulation Fails to Mitigate Forgetting for NutriBench

Concurrent work has shown that token\-level manipulation methods, such as ensembling and contrastive decoding, can mitigate forgetting on certain benchmarks\([Ki et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib52)\)\. However, we identify a failure case for these methods\. Following the implementation of[Ki et al\. \(2026\)](https://arxiv.org/html/2610.11132#bib.bib52), we use ensembling and contrastive decoding as two strong baselines\. For ensembling, we average the token probabilities of the parent and SFT models with equal weights\. For contrastive decoding, we treat the SFT model as the expert model and the parent model as the amateur model\([Li et al\., 2023](https://arxiv.org/html/2610.11132#bib.bib53)\)\. From NutriBench, we sample 100 English queries, 100 non\-English queries, and 100 non\-food queries, and compare these baseline methods with SFT\-as\-context across all three nutrition model pairs\.

[AppendixH](https://arxiv.org/html/2610.11132#A8)shows that, although both token\-level baselines sometimes outperform SFT\-as\-context on fine\-tuned capabilities, neither successfully mitigates forgetting of general capabilities\. This finding contrasts with prior results on other benchmarks\([Ki et al\., 2026](https://arxiv.org/html/2610.11132#bib.bib52)\), where both baselines can often fully mitigate forgetting\. We attribute this failure to the specific failure patterns exhibited by the SFT models\. For non\-English queries, the SFT model has a strong tendency to generate the English word “contributes” after the name of a food item\. The disproportionately high probability assigned to this token makes it more likely to be selected under both ensembling and contrastive decoding, resulting in poor language consistency\. Similarly, for non\-food queries, “contributes” tends to dominate the token distribution, leading the model to continue to generate hallucinated, non\-zero nutritional values\. These results suggest that SFT\-as\-context can address certain shifts in token probability distributions that are difficult to correct through token\-level manipulation\.

Table 9:Token\-level baselines \(ensemble and contrastive decoding\) fail to recover general capabilities\.The results for each inference setting are based on 100 English, 100 non\-English, and 100 non\-food queries randomly sampled from NutriBench\. Best results are bolded and second\-best results are underlined\.Fine\-Tuned CapabilitiesGeneral CapabilitiesInference SettingEnglish MacroMAE↓\\downarrowNon\-English MacroMAE↓\\downarrowLanguageConsistency \(%\)↑\\uparrowAcknowledgeNon\-Food \(%\)↑\\uparrow\[0pt\]\[0pt\]Gemma\-4\-E4B\-itParent35\.675\.49697SFT14\.954\.3437Ensemble16\.164\.41144Contrastive Decoding17\.363\.0124SFT\-as\-Context16\.460\.696100\[0pt\]\[0pt\]Qwen3\-4BParent39\.599\.77499SFT15\.579\.0018Ensemble56\.5155\.72326Contrastive Decoding17\.557\.108SFT\-as\-Context19\.480\.36697\[0pt\]\[0pt\]Qwen3\-8BParent35\.781\.566100SFT15\.657\.168Ensemble28\.492\.13611Contrastive Decoding17\.560\.706SFT\-as\-Context20\.054\.26796
## Appendix ISFT\-as\-Context Incurs Modest Overhead Compared to SFT

Since SFT\-as\-context requires two inference passes, we examine the additional generation overhead introduced by the second pass, averaged across all responses for each benchmark\.[AppendixI](https://arxiv.org/html/2610.11132#A9)shows that, for all math and coding model pairs, the second pass incurs only modest overhead relative to the typically long responses generated by the SFT model in the first pass\. Specifically, the ratio of second\-pass to first\-pass response tokens is at most 13\.0%\.

For the nutrition estimation task, however, the parent model generates relatively longer responses during the second pass, as it performs additional reasoning to compare the SFT\-generated reference with its own estimate\. Nevertheless, the absolute number of generated tokens remains comparable to that of the other tasks, ranging from a few hundred to slightly over one thousand tokens\. Overall, these results suggest that the additional generation overhead introduced by SFT\-as\-context is modest\.

Table 10:SFT\-as\-context incurs modest overhead compared with SFT\.Under each of the 3 domains, we list benchmarks for testing both fine\-tuned and general capabilities\. Ratio is the number of SFT\-as\-context response tokens divided by the number of SFT response tokens, reported as percentages\. Although SFT\-as\-context adds a second inference pass after the first inference pass of the SFT model, only a small number of additional tokens is generated in the second pass for most benchmarks\. Nutrition estimation is an outlier, where SFT\-as\-context generates more response tokens to compare the reference with its own estimation\.BenchmarkSFT Response TokensSFT\-as\-Context Response TokensRatio \(%\)\[0pt\]\[0pt\]MathematicsAIME 202413,8146985\.1MathIF6,8333425\.0IFEval7,46997213\.0MGSM5,8513966\.8\[0pt\]\[0pt\]CodingLiveCodeBench14,3473502\.4LiveCodeBenchIF14,8853872\.6IFEval10,8165234\.8SQuAD2\.010,79980\.1FQuAD2\.07,174390\.6CoQA13,021410\.3ACPBench10,6181961\.8\[0pt\]\[0pt\]NutritionNutriBench\-English415823198\.1NutriBench\-Non\-English8701,252143\.9NutriBench\-Non\-Food1,00030030\.0
## Appendix JAdditional Ablation Studies

#### Repeating task instruction helps SFT\-as\-context\.

For math and coding models, we repeat the original task instruction in the SFT\-as\-context prompt\. On MathIF, we perform an ablation study by removing the repeated instructions \(from “Again, the original task is” to the end of the SFT\-as\-context instructions\)\.[Table11](https://arxiv.org/html/2610.11132#A10.T11)shows that repeating the original task instructions significantly helps instruction\-following, while barely affecting the fine\-tuned math capability\.

Table 11:Repeating the original task prompt in the SFT\-as\-context instructions helps instruction\-following\.For the benchmark MathIF, repeating the instruction significantly helps the instruction\-following \(Hard IF and Soft IF\) accuracies of SFT\-as\-context\. All results are percentages, averaged over 8 math model pairs\.Δ\\Deltais SFT\-as\-context without repeating original task minus SFT\-as\-context, in percentage points\.
## Appendix KTheoretical Results and Proofs

Given the above model of sequential generation in Section[4\.1](https://arxiv.org/html/2610.11132#S4.SS1), we characterize the errors for both in\-domain and out\-of\-domain queries\. We first describe the optimal response distributions under the optimal domain variable using a variable:θ∗∈Θ\\theta^\{\*\}\\in\\Theta, whereΘ\\Thetais discrete\. As above, the parent model is assumed to be more general\-purpose, meaning it has a set of domain expertiseΘparent\\Theta\_\{\\text\{parent\}\}\. Let the optimal output, denoted byYY, be generated given the prompt in some domain with the domain expertise vectorθ∗\\theta^\{\*\}:

Y∼Pr⁡\(Y\|x,θ∗\)\.\\displaystyle Y\\sim\\operatorname\{Pr\}\(Y\|x,\\theta^\{\*\}\)\.\(5\)
We denote byPr∗\\operatorname\{Pr\}\_\{\*\}the above output distribution under optimal domain variable\.

### K\.1Additional Intuition for the Bayesian Framework

The domain variableθ\\thetacaptures differences in the query\-response distributions across domains\. For example, nutrition estimation and mathematical reasoning involve different response distributions, modeled throughPr⁡\(Y∣x,θ\)\\operatorname\{Pr\}\(Y\\mid x,\\theta\)\. A parent model trained across multiple domains can represent several such capabilities; for example, Qwen2\.5\-1\.5B\-Instruct supports mathematical reasoning, multilingual understanding, and code generation\. Fine\-tuning on query\-response pairs from a single domainθSFT\\theta\_\{\\text\{SFT\}\}aims to improve the corresponding domain\-specific capability\.

Under the Bayesian interpretation\([Xie et al\., 2022](https://arxiv.org/html/2610.11132#bib.bib2);[Hu et al\., 2024](https://arxiv.org/html/2610.11132#bib.bib3)\), the response distribution is obtained by marginalizing over the domain variable\. Conceptually, generation first selects a domain according to its posterior probability and then generates a response from the corresponding conditional output distribution\. This is a probabilistic interpretation rather than a claim that the model explicitly performs these two steps\. SFT\-as\-context supplies the SFT responseySFTy\_\{\\text\{SFT\}\}and instructionsIIas additional evidence for domain inference\. Under our shared conditional\-output model, this evidence changes the domain posterior while leaving the conditional response distributions fixed\. Thus, the modeled improvement arises from assigning greater probability to an existing domain capability\.

### K\.2Defining the Errors

#### Error of Parent Model:

The parent model has plenty of domains \(Θparent\\Theta\_\{\\text\{parent\}\}\) in its set of capabilities and it need not always generate output dependent on the optimal domainDD\. Its own output is therefore:

Yparent∼Prparent⁡\(Y\|x\),\\displaystyle Y\_\{\\text\{parent\}\}\\sim\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(Y\|x\),
and therefore the output distribution is obtained by marginalizing over all the domains:

Prparent⁡\(y\|x\)=∑θ∈ΘparentPrparent⁡\(y\|θ,x\)​Prparent​\(θ\|x\)\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\|x\)=\\sum\_\{\\theta\\in\\Theta\_\{\\text\{parent\}\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\|\\theta,x\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\|x\)\.
Therefore, the error for the parent model is the KL divergence between the optimal output distribution under the optimal reasoning law and the parent distribution:

ℰparent=KL\(Pr∗\(y\|x\)\|\|Prparent\(y\|x\)\)\.\\displaystyle\\mathcal\{E\}\_\{\\text\{parent\}\}=KL\(\\operatorname\{Pr\}\_\{\*\}\(y\|x\)\|\|\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\|x\)\)\.

#### Error of SFT Model:

We assume that the SFT model has expertise in one domain specificallyDSFTD\_\{\\text\{SFT\}\}with domain variableθSFT\\theta\_\{\\text\{SFT\}\}\. As discussed previously, due to forgetting, the model has a reduced set of domains, denoted byΘSFT\\Theta\_\{\\text\{SFT\}\}\. Therefore, the output of the SFT model is:

PrSFT⁡\(y\|x\)=∑θ∈ΘSFTPrSFT⁡\(y\|θ,x\)​PrSFT​\(θ\|x\)\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\|x\)=\\sum\_\{\\theta\\in\\Theta\_\{\\text\{SFT\}\}\}\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\|\\theta,x\)\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(\\theta\|x\)\.
The associated error for the SFT model is as follows:

ℰSFT=KL\(Pr∗\(y\|x\)\|\|PrSFT\(y\|x\)\)\.\\displaystyle\\mathcal\{E\}\_\{\\text\{SFT\}\}=KL\(\\operatorname\{Pr\}\_\{\*\}\(y\|x\)\|\|\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\|x\)\)\.

#### Error of SFT\-as\-Context:

Similarly, we define the latent trajectory under SFT\-as\-context \(abbreviated as SAC; similar to Equation[5](https://arxiv.org/html/2610.11132#A11.E5)\)\.

YSAC∼PrSAC⁡\(Y\|x,I,ySFT\)\.\\displaystyle Y\_\{\\text\{SAC\}\}\\sim\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(Y\|x,I,y\_\{\\text\{SFT\}\}\)\.
The output distribution is therefore obtained by marginalizing over all domainsDDunder the SAC distribution:

PrSAC⁡\(y\|x,I,ySFT\)=∑θ∈ΘparentPrparent⁡\(y\|θ,x\)​PrSAC​\(θ\|x,I,ySFT\)\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(y\|x,I,y\_\{\\text\{SFT\}\}\)=\\sum\_\{\\theta\\in\\Theta\_\{\\text\{parent\}\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\|\\theta,x\)\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(\\theta\|x,I,y\_\{\\text\{SFT\}\}\)\.
Therefore, the error here is:

ℰSAC=KL\(Pr∗\(y\|x\)\|\|PrSAC\(y\|x,I,ySFT\)\)\.\\displaystyle\\mathcal\{E\}\_\{\\text\{SAC\}\}=KL\(\\operatorname\{Pr\}\_\{\*\}\(y\|x\)\|\|\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(y\|x,I,y\_\{\\text\{SFT\}\}\)\)\.
Using these error formulations, we split Theorem[1](https://arxiv.org/html/2610.11132#Thmtheorem1)into two theorems and their corresponding proofs, and finally present Theorem[2](https://arxiv.org/html/2610.11132#Thmtheorem2)and its proof\. Each following subsection corresponds to one theorem and its proof\.

### K\.3SFT\-as\-Context In\-Domain Error

Suppose the SFT model is fine\-tuned on data from the SFT domain with optimal domain variableθSFT\\theta\_\{\\text\{SFT\}\}\. For an in\-domain queryx∈DSFTx\\in D\_\{\\text\{SFT\}\}, assume thatθSFT∈Θparent\\theta\_\{\\text\{SFT\}\}\\in\\Theta\_\{\\text\{parent\}\}and that the optimal output law is realizable:

Prparent⁡\(y∣θSFT,x\)=Pr∗⁡\(y∣x\)\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid\\theta\_\{\\text\{SFT\}\},x\)=\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\)\.\(6\)Thus, the parent model can represent the optimal output law, but may assign little probability to its domain\. All logarithms below are natural\.

###### Assumption K\.1\(Domain posterior bounds\)\.

There exist0<ϵ<10<\\epsilon<1and0≤ϵ′<10\\leq\\epsilon^\{\\prime\}<1such that, for each query and realized context under consideration,

0<ax\\displaystyle 0<a\_\{x\}:=Prparent⁡\(θSFT∣x\)≤ϵ,\\displaystyle:=\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\_\{\\text\{SFT\}\}\\mid x\)\\leq\\epsilon,\(7\)bx\\displaystyle b\_\{x\}:=PrSAC⁡\(θSFT∣x,I,ySFT\)≥1−ϵ′\.\\displaystyle:=\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(\\theta\_\{\\text\{SFT\}\}\\mid x,I,y\_\{\\text\{SFT\}\}\)\\geq 1\-\\epsilon^\{\\prime\}\.\(8\)

This assumption corresponds to Assumptions 1 and 2 in Section[4\.1](https://arxiv.org/html/2610.11132#S4.SS1), which we present together formally for ease of exposition in the proof below\.

###### Assumption K\.2\(Output separability\)\.

Define the parent model’s normalized output mixture over the remaining domains by

Qparent​\(y∣x\):=∑θ∈Θparentθ≠θSFTPrparent⁡\(y∣θ,x\)​Prparent​\(θ∣x\)1−ax\.\\displaystyle Q\_\{\\text\{parent\}\}\(y\\mid x\):=\\frac\{\\sum\_\{\\begin\{subarray\}\{c\}\\theta\\in\\Theta\_\{\\text\{parent\}\}\\\\ \\theta\\neq\\theta\_\{\\text\{SFT\}\}\\end\{subarray\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid\\theta,x\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\\mid x\)\}\{1\-a\_\{x\}\}\.\(9\)There existsϵ′′\>0\\epsilon^\{\\prime\\prime\}\>0such that, for everyx∈DSFTx\\in D\_\{\\text\{SFT\}\},

TV\(Pr∗\(⋅∣x\),Qparent\(⋅∣x\)\)≥ϵ′′,\\displaystyle\\operatorname\{TV\}\\\!\\left\(\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\),Q\_\{\\text\{parent\}\}\(\\cdot\\mid x\)\\right\)\\geq\\epsilon^\{\\prime\\prime\},\(10\)whereTV⁡\(P,Q\):=12​∑y\|P⁡\(y\)−Q⁡\(y\)\|\\operatorname\{TV\}\(P,Q\):=\\frac\{1\}\{2\}\\sum\_\{y\}\|P\(y\)\-Q\(y\)\|\.

This assumption is Assumption 3 in Section[4\.1](https://arxiv.org/html/2610.11132#S4.SS1)\. Separability is required for the mixture over the remaining domains\([Xie et al\., 2022](https://arxiv.org/html/2610.11132#bib.bib2)\)\. This gives us our first result\.

###### Theorem 3\(In\-domain error reduction\)\.

Under Equation[6](https://arxiv.org/html/2610.11132#A11.E6)and Assumptions[K\.1](https://arxiv.org/html/2610.11132#Thmassumption1)–[K\.2](https://arxiv.org/html/2610.11132#Thmassumption2), for each query and context satisfying these conditions,

ℰparent−ℰSAC≥2​\(1−ϵ\)2​\(ϵ′′\)2\+log⁡\(1−ϵ′\)\.\\displaystyle\\mathcal\{E\}\_\{\\text\{parent\}\}\-\\mathcal\{E\}\_\{\\text\{SAC\}\}\\geq 2\(1\-\\epsilon\)^\{2\}\(\\epsilon^\{\\prime\\prime\}\)^\{2\}\+\\log\(1\-\\epsilon^\{\\prime\}\)\.\(11\)In particular,ℰSAC<ℰparent\\mathcal\{E\}\_\{\\text\{SAC\}\}<\\mathcal\{E\}\_\{\\text\{parent\}\}whenever

2​\(1−ϵ\)2​\(ϵ′′\)2\>−log⁡\(1−ϵ′\)\.\\displaystyle 2\(1\-\\epsilon\)^\{2\}\(\\epsilon^\{\\prime\\prime\}\)^\{2\}\>\-\\log\(1\-\\epsilon^\{\\prime\}\)\.

###### Proof\.

Fixxxand the realized context\(I,ySFT\)\(I,y\_\{\\text\{SFT\}\}\)\. By realizability and the definition ofQparentQ\_\{\\text\{parent\}\},

Prparent⁡\(y∣x\)=ax​Pr∗​\(y∣x\)\+\(1−ax\)​Qparent​\(y∣x\)\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid x\)=a\_\{x\}\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\)\+\(1\-a\_\{x\}\)Q\_\{\\text\{parent\}\}\(y\\mid x\)\.Consequently \(using Assumption[K\.2](https://arxiv.org/html/2610.11132#Thmassumption2)\),

TV\(Pr∗\(⋅∣x\),Prparent\(⋅∣x\)\)\\displaystyle\\operatorname\{TV\}\\\!\\left\(\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\),\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\cdot\\mid x\)\\right\)=\(1−ax\)TV\(Pr∗\(⋅∣x\),Qparent\(⋅∣x\)\)\\displaystyle=\(1\-a\_\{x\}\)\\operatorname\{TV\}\\\!\\left\(\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\),Q\_\{\\text\{parent\}\}\(\\cdot\\mid x\)\\right\)≥\(1−ϵ\)​ϵ′′\.\\displaystyle\\geq\(1\-\\epsilon\)\\epsilon^\{\\prime\\prime\}\.The standard Pinsker’s inequality therefore gives

ℰparent≥2​\(1−ϵ\)2​\(ϵ′′\)2\.\\displaystyle\\mathcal\{E\}\_\{\\text\{parent\}\}\\geq 2\(1\-\\epsilon\)^\{2\}\(\\epsilon^\{\\prime\\prime\}\)^\{2\}\.\(12\)
For SFT\-as\-context, retaining the optimal\-domain term in its nonnegative mixture gives

PrSAC⁡\(y∣x,I,ySFT\)≥bx​Pr∗​\(y∣x\)\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(y\\mid x,I,y\_\{\\text\{SFT\}\}\)\\geq b\_\{x\}\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\)\.Hence,

ℰSAC=𝔼Y∼Pr∗\(⋅∣x\)\[logPr∗⁡\(Y∣x\)PrSAC⁡\(Y∣x,I,ySFT\)\]≤−log⁡bx≤−log⁡\(1−ϵ′\)\.\\displaystyle\\begin\{split\}\\mathcal\{E\}\_\{\\text\{SAC\}\}&=\\mathbb\{E\}\_\{Y\\sim\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\)\}\\left\[\\log\\frac\{\\operatorname\{Pr\}\_\{\*\}\(Y\\mid x\)\}\{\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(Y\\mid x,I,y\_\{\\text\{SFT\}\}\)\}\\right\]\\\\ &\\leq\-\\log b\_\{x\}\\leq\-\\log\(1\-\\epsilon^\{\\prime\}\)\.\\end\{split\}\(13\)Subtracting this upper bound from the lower bound onℰparent\\mathcal\{E\}\_\{\\text\{parent\}\}\(Equation[12](https://arxiv.org/html/2610.11132#A11.E12)\) gives us the result\. ∎

### K\.4In\-Domain Closeness of SFT\-as\-Context to SFT

We recall that the error for the SFT model’s response distribution is defined as:

ℰSFT:=DKL\(Pr∗\(⋅∣x\)∥PrSFT\(⋅∣x\)\)\.\\displaystyle\\mathcal\{E\}\_\{\\text\{SFT\}\}:=D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\)\\,\\\|\\,\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(\\cdot\\mid x\)\\right\)\.
###### Assumption K\.3\(In\-domain SFT realizability and concentration\)\.

The optimal domainθSFT\\theta\_\{\\text\{SFT\}\}belongs toΘSFT\\Theta\_\{\\text\{SFT\}\}, and

PrSFT⁡\(y∣θSFT,x\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\\mid\\theta\_\{\\text\{SFT\}\},x\)=Pr∗⁡\(y∣x\)\.\\displaystyle=\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\)\.\(14\)Furthermore, there exists0≤η<10\\leq\\eta<1such that, for everyx∈DSFTx\\in D\_\{\\text\{SFT\}\},

cx:=PrSFT⁡\(θSFT∣x\)≥1−η\.\\displaystyle c\_\{x\}:=\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(\\theta\_\{\\text\{SFT\}\}\\mid x\)\\geq 1\-\\eta\.\(15\)

###### Theorem 4\(Closeness to SFT\)\.

Under the preceding SAC realizability and posterior concentration assumptions, and Assumption[K\.3](https://arxiv.org/html/2610.11132#Thmassumption3), we have

log⁡\(1−ϵ′\)≤ℰSFT−ℰSAC≤−log⁡\(1−η\)\.\\displaystyle\\log\(1\-\\epsilon^\{\\prime\}\)\\leq\\mathcal\{E\}\_\{\\text\{SFT\}\}\-\\mathcal\{E\}\_\{\\text\{SAC\}\}\\leq\-\\log\(1\-\\eta\)\.\(16\)Consequently,

\|ℰSFT−ℰSAC\|≤max⁡\{−log⁡\(1−η\),−log⁡\(1−ϵ′\)\}\.\\displaystyle\\left\|\\mathcal\{E\}\_\{\\text\{SFT\}\}\-\\mathcal\{E\}\_\{\\text\{SAC\}\}\\right\|\\leq\\max\\left\\\{\-\\log\(1\-\\eta\),\-\\log\(1\-\\epsilon^\{\\prime\}\)\\right\\\}\.\(17\)

###### Proof\.

Once again, fix an in\-domain queryx∈DSFTx\\in D\_\{\\text\{SFT\}\}and a realized context\(I,ySFT\)\(I,y\_\{\\text\{SFT\}\}\)satisfying the assumptions\. Since the optimal domainθSFT\\theta\_\{\\text\{SFT\}\}is in the SFT model’s capabilitiesΘSFT\\Theta\_\{\\text\{SFT\}\}, we have:

PrSFT⁡\(y∣x\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\\mid x\)≥cx​PrSFT​\(y∣θSFT,x\)\\displaystyle\\geq c\_\{x\}\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(y\\mid\\theta\_\{\\text\{SFT\}\},x\)=cx​Pr∗​\(y∣x\)\.\\displaystyle=c\_\{x\}\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\)\.By nonnegativity of KL divergence and this inequality,

0≤ℰSFT\\displaystyle 0\\leq\\mathcal\{E\}\_\{\\text\{SFT\}\}=𝔼Y∼Pr∗\(⋅∣x\)\[logPr∗⁡\(Y∣x\)PrSFT⁡\(Y∣x\)\]\\displaystyle=\\mathbb\{E\}\_\{Y\\sim\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\)\}\\left\[\\log\\frac\{\\operatorname\{Pr\}\_\{\*\}\(Y\\mid x\)\}\{\\operatorname\{Pr\}\_\{\\text\{SFT\}\}\(Y\\mid x\)\}\\right\]≤−log⁡cx≤−log⁡\(1−η\)\.\\displaystyle\\leq\-\\log c\_\{x\}\\leq\-\\log\(1\-\\eta\)\.Similarly, the preceding SFT\-as\-context proof \(Equation[13](https://arxiv.org/html/2610.11132#A11.E13)\) gives us:

0≤ℰSAC≤−log⁡\(1−ϵ′\)\.\\displaystyle 0\\leq\\mathcal\{E\}\_\{\\text\{SAC\}\}\\leq\-\\log\(1\-\\epsilon^\{\\prime\}\)\.Combining these two intervals yields

log⁡\(1−ϵ′\)≤ℰSFT−ℰSAC≤−log⁡\(1−η\),\\displaystyle\\log\(1\-\\epsilon^\{\\prime\}\)\\leq\\mathcal\{E\}\_\{\\text\{SFT\}\}\-\\mathcal\{E\}\_\{\\text\{SAC\}\}\\leq\-\\log\(1\-\\eta\),which leads to the absolute\-error bound\. ∎

Unlike the improvement guarantee over the parent model, this closeness guarantee does not require output separability\. Together, Theorems[3](https://arxiv.org/html/2610.11132#Thmtheorem3)and[4](https://arxiv.org/html/2610.11132#Thmtheorem4)are presented as Theorem[1](https://arxiv.org/html/2610.11132#Thmtheorem1)in Section[4\.1](https://arxiv.org/html/2610.11132#S4.SS1)\.

### K\.5SFT\-as\-Context Out\-of\-Domain Error

Consider a domainDoD\_\{o\}with domain variableθo∈Θparent\\theta\_\{o\}\\in\\Theta\_\{\\text\{parent\}\}, whereθo≠θSFT\\theta\_\{o\}\\neq\\theta\_\{\\text\{SFT\}\}\. For queriesx∈Dox\\in D\_\{o\}, assume that the optimal output distribution is realizable in the parent model:

Prparent⁡\(y∣θo,x\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid\\theta\_\{o\},x\)=Pr∗⁡\(y∣x\),\\displaystyle=\\operatorname\{Pr\}\_\{\*\}\(y\\mid x\),\(18\)Prparent⁡\(θo∣x\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\_\{o\}\\mid x\)\>0\.\\displaystyle\>0\.\(19\)In simpler terms, this means that the model has a non\-zero probability of outputting the optimal response via the optimal task domain posterior probability distribution\. This is referred to as out\-of\-domain realizability\. Bothℰparent\\mathcal\{E\}\_\{\\text\{parent\}\}andℰSAC\\mathcal\{E\}\_\{\\text\{SAC\}\}are measured relative to this optimal out\-of\-domain law\.

###### Assumption K\.4\(Out\-of\-domain posterior retention\)\.

There exists0≤ρo<10\\leq\\rho\_\{o\}<1such that, for every out\-of\-domain query and realized context under consideration,

PrSAC⁡\(θ∣x,I,ySFT\)≥\(1−ρo\)​Prparent​\(θ∣x\),∀θ∈Θparent\.\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(\\theta\\mid x,I,y\_\{\\text\{SFT\}\}\)\\geq\(1\-\\rho\_\{o\}\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\\mid x\),\\qquad\\forall\\theta\\in\\Theta\_\{\\text\{parent\}\}\.\(20\)

This assumption limits how much the SFT context can suppress the parent model’s existing domain probabilities and corresponds to Assumption 5 in Section[4\.1](https://arxiv.org/html/2610.11132#S4.SS1)\. It is an additional condition; membership ofθo\\theta\_\{o\}inΘparent\\Theta\_\{\\text\{parent\}\}alone need not imply it\.

###### Theorem 5\(Error guarantees for out\-of\-domain queries\)\.

Under Equation[18](https://arxiv.org/html/2610.11132#A11.E18), Assumption[K\.4](https://arxiv.org/html/2610.11132#Thmassumption4), and the conditional\-output model described above,

ℰSAC−ℰparent≤−log⁡\(1−ρo\)\.\\displaystyle\\mathcal\{E\}\_\{\\text\{SAC\}\}\-\\mathcal\{E\}\_\{\\text\{parent\}\}\\leq\-\\log\(1\-\\rho\_\{o\}\)\.\(21\)

###### Proof\.

Fixx∈Dox\\in D\_\{o\}and a realized context\(I,ySFT\)\(I,y\_\{\\text\{SFT\}\}\)that satisfies the assumption\. Using posterior retention in the SAC mixture,

PrSAC⁡\(y∣x,I,ySFT\)\\displaystyle\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(y\\mid x,I,y\_\{\\text\{SFT\}\}\)=∑θ∈ΘparentPrparent⁡\(y∣θ,x\)​PrSAC​\(θ∣x,I,ySFT\)\\displaystyle\\quad=\\sum\_\{\\theta\\in\\Theta\_\{\\text\{parent\}\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid\\theta,x\)\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(\\theta\\mid x,I,y\_\{\\text\{SFT\}\}\)≥\(1−ρo\)​∑θ∈ΘparentPrparent⁡\(y∣θ,x\)​Prparent​\(θ∣x\)\\displaystyle\\quad\\geq\(1\-\\rho\_\{o\}\)\\sum\_\{\\theta\\in\\Theta\_\{\\text\{parent\}\}\}\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid\\theta,x\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(\\theta\\mid x\)=\(1−ρo\)​Prparent​\(y∣x\)\.\\displaystyle\\quad=\(1\-\\rho\_\{o\}\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(y\\mid x\)\.The realizability assumption and the positive posterior ofθo\\theta\_\{o\}ensure that both errors are finite\. Therefore,

ℰSAC\\displaystyle\\mathcal\{E\}\_\{\\text\{SAC\}\}=𝔼Y∼Pr∗\(⋅∣x\)\[logPr∗⁡\(Y∣x\)PrSAC⁡\(Y∣x,I,ySFT\)\]\\displaystyle=\\mathbb\{E\}\_\{Y\\sim\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\)\}\\left\[\\log\\frac\{\\operatorname\{Pr\}\_\{\*\}\(Y\\mid x\)\}\{\\operatorname\{Pr\}\_\{\\text\{SAC\}\}\(Y\\mid x,I,y\_\{\\text\{SFT\}\}\)\}\\right\]≤𝔼Y∼Pr∗\(⋅∣x\)\[logPr∗⁡\(Y∣x\)\(1−ρo\)​Prparent​\(Y∣x\)\]\\displaystyle\\leq\\mathbb\{E\}\_\{Y\\sim\\operatorname\{Pr\}\_\{\*\}\(\\cdot\\mid x\)\}\\left\[\\log\\frac\{\\operatorname\{Pr\}\_\{\*\}\(Y\\mid x\)\}\{\(1\-\\rho\_\{o\}\)\\operatorname\{Pr\}\_\{\\text\{parent\}\}\(Y\\mid x\)\}\\right\]=ℰparent−log⁡\(1−ρo\)\.\\displaystyle=\\mathcal\{E\}\_\{\\text\{parent\}\}\-\\log\(1\-\\rho\_\{o\}\)\.Rearranging proves the result\. ∎

Thus, when the SFT context only weakly suppresses the parent posterior, the additional out\-of\-domain error is small:−log⁡\(1−ρo\)=ρo\+O⁡\(ρo2\)\-\\log\(1\-\\rho\_\{o\}\)=\\rho\_\{o\}\+O\(\\rho\_\{o\}^\{2\}\)asρo→0\\rho\_\{o\}\\to 0\. This proves Theorem[2](https://arxiv.org/html/2610.11132#Thmtheorem2)from Section[4\.1](https://arxiv.org/html/2610.11132#S4.SS1)above\.