Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

# why RL preserves circuits better than SFT?
Source: [https://arxiv.org/html/2605.28860](https://arxiv.org/html/2605.28860)
## Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?

Viraj SawantNathan AllenNomgondalai AmgalanbaatarYannis ZongoVasu SharmaMaheep Chaudhary

###### Abstract

Fine\-tuning large language models \(LLMs\) frequently induces catastrophic forgetting of prior capabilities\. Recent work has shown that reinforcement learning \(RL\) retains prior capabilities more effectively than supervised fine\-tuning \(SFT\), attributing this to policy\-gradient updates remaining closer to the base policy\(Shenfeld et al\.,[2025](https://arxiv.org/html/2605.28860#bib.bib10)\)\. We extend this behavioral account to the mechanistic level and ask whether RL’s advantage is mirrored by stronger preservation of internal computational circuits\. We introduce differential circuit vulnerability, a head\-level measure of how much a circuit degrades under fine\-tuning, and use it to compare RL and SFT on Qwen2\.5\-3B\-Instruct adapted to scientific question\-answering\. We find a clear mechanistic trade\-off: SFT adapts more rapidly to the target task but produces substantially greater circuit disruption and forgetting of prior capabilities, whereas RL preserves a larger fraction of the base circuit at the cost of slower task adaptation\. These findings suggest that circuit preservation may help explain why RL is more robust to catastrophic forgetting\. We released our code here:[https://github\.com/rl\-sft\-circuit\-research/differential\-circuit\-vulnerability](https://github.com/rl-sft-circuit-research/differential-circuit-vulnerability)\.

Machine Learning, ICML

## 1Introduction

Adapting large language models \(LLMs\) to new downstream tasks frequently incurs catastrophic forgetting: gains on a new objective come at the expense of prior capabilities\. As models are increasingly expected to update continuously and adapt to new domains, mitigating such degradation has emerged as a central challenge of post\-training\. Prior work shows that the choice of adaptation objective strongly shapes this trade\-off: RL typically preserves prior capabilities more effectively than SFT, arguably because policy\-gradient updates remain closer to the pretrained model and reduce distributional drift, commonly measured by KL divergence\(Shenfeld et al\.,[2025](https://arxiv.org/html/2605.28860#bib.bib10)\)\.

![Refer to caption](https://arxiv.org/html/2605.28860v1/x1.png)Figure 1:Circuit retention trajectories during high\-NTS training\.Starting from100%100\\%base\-circuit retention, SFT \(orange\) and RL \(blue\) diverge sharply over the two training epochs that produce the high new\-task score models\. SFT drops to63\.5%63\.5\\%after Epoch 1 and continues declining to59\.0%59\.0\\%by Epoch 2, whereas RL falls to69\.8%69\.8\\%after Epoch 1 and*recovers*to72\.5%72\.5\\%by Epoch 2—a13\.513\.5percentage\-point advantage\. Footer values report per\-epoch circuit faithfulness \(both≈1\.0\\approx 1\.0\) and Differential Causal Mediation \(DCM\), with RL maintaining consistently higher DCM \(15\.815\.8vs\.10\.410\.4at Epoch 1;10\.610\.6vs\.6\.36\.3at Epoch 2\), indicating that RL’s circuit retains stronger causal engagement with model behavior throughout training\.In parallel, mechanistic interpretability has shown that model capabilities are implemented by internal computationalcircuitscomposed of attention heads, MLP layers, and residual pathways\. Fine\-tuning may therefore succeed or fail depending on whether it preserves these structures or disrupts them\(Prakash et al\.,[2024](https://arxiv.org/html/2605.28860#bib.bib8)\)\. This motivates our central question:can RL’s retention advantage be explained by stronger preservation of task\-relevant circuits relative to SFT?

We examine post\-training through the lens of circuit preservation\. We introducedifferential circuit vulnerability, the relative susceptibility of internal computational subgraphs to degradation under different training objectives\. Using this framework, we compare RL and SFT on matched adaptation tasks and identify which circuits are preserved or disrupted\. Figure[1](https://arxiv.org/html/2605.28860#S1.F1)previews the central finding: over the course of training the two objectives trace diverging circuit\-retention trajectories, with RL consistently preserving a larger fraction of the base circuit\. We characterize this trade\-off mechanistically in the sections that follow\.

We instantiate our analysis on Qwen2\.5\-3B\-Instruct using a two\-stage protocol: adaptation on scientific question answering, followed by retention evaluation across a broad benchmark suite spanning commonsense reasoning, factuality, and instruction following\. The results expose a consistent trade\-off: SFT adapts faster to the new objective but at the cost of greater circuit disruption, whereas RL preserves more of the base circuit and of prior capabilities, occasionally at the expense of under\-optimizing the new task\.

These results suggest that continual adaptation should be understood not merely as parameter optimization, but as theselective preservation and modificationof circuits encoding skills\. This perspective points toward a mechanistic account of why RL is more conservative than SFT during post\-training\.

## 2Related Work

#### Reinforcement Learning, Supervised Fine\-Tuning, and Forgetting\.

Supervised fine\-tuning \(SFT\) and reinforcement learning \(RL\) are the dominant post\-training methods for adapting foundation models, yet their effects on previously acquired capabilities remain incompletely understood\. SFT optimizes imitation on a target distribution, whereas RL updates behavior through reward optimization and often produces qualitatively different parameter changes\. Recent studies suggest that, under comparable settings, RL can preserve pretrained capabilities better than SFT, indicating that the training objective influences both adaptation and forgetting\(Shenfeld et al\.,[2025](https://arxiv.org/html/2605.28860#bib.bib10); Hu et al\.,[2025](https://arxiv.org/html/2605.28860#bib.bib6)\)\.

#### Mechanistic Interpretability and Circuit Analysis\.

Mechanistic interpretability explains model behavior through concrete components such as attention heads, MLP layers, and circuits\. Intervention\-based methods, including activation patching, path patching, and masking, have shown that many behaviors can be localized to sparse causal subnetworks\. Related work further suggests that fine\-tuning frequently modifies existing mechanisms rather than replacing them entirely\(Davies et al\.,[2023](https://arxiv.org/html/2605.28860#bib.bib3); Prakash et al\.,[2024](https://arxiv.org/html/2605.28860#bib.bib8)\)\.

#### Evaluating Retention Across Capabilities\.

Retention cannot be assessed on the adaptation task alone\. Models may improve on a new objective while degrading reasoning, factuality, instruction following, or code generation\. For this reason, prior work increasingly relies on diverse benchmark suites\. Common evaluations include MMLU, HellaSwag, and WinoGrande for reasoning and commonsense inference, and TruthfulQA, IFEval, and HumanEval for factuality, instruction following, and coding\(Hendrycks et al\.,[2021](https://arxiv.org/html/2605.28860#bib.bib5); Zellers et al\.,[2019](https://arxiv.org/html/2605.28860#bib.bib11); Lin et al\.,[2022](https://arxiv.org/html/2605.28860#bib.bib7)\)\.

#### Research Gap and Positioning\.

Despite progress in both post\-training and interpretability, the two literatures remain loosely connected\. Optimization studies typically report benchmark outcomes without explaining the internal causes of forgetting, while interpretability studies rarely compare learning objectives\. We address this gap by comparing RL and SFT through the lens of circuit preservation, asking which objective better maintains the causal structures underlying pretrained capabilities\.

## 3Methodology

We test the hypothesis that RL retains prior capabilities because it preserves task\-relevant internal circuits more effectively than SFT\. Our pipeline comprises three phases: \(I\) reproduce the known retention gap between SFT and RL; \(II\) identify circuits in each model; and \(III\) compare how post\-training reshapes these circuits\.

We begin from a pretrained modelπbase\\pi\_\{\\mathrm\{base\}\}\. We first train an SFT modelπSFT\\pi\_\{\\mathrm\{SFT\}\}via completion\-only supervision, then refine it with Dr\.GRPO to obtainπRL\\pi\_\{\\mathrm\{RL\}\}\. Thus, our comparison isolates the effect of continuing post\-training with RL beyond SFTπθ∈\{πSFT,πRL\}\\pi\_\{\\theta\}\\in\\\{\\pi\_\{\\mathrm\{SFT\}\},\\pi\_\{\\mathrm\{RL\}\}\\\}\.

To quantify behavioral drift from the base model, we compute the expected KL divergence on retention tasks, where lower values correspond to less distributional shift\.

𝔼x∼τ\[DKL\(πbase\(⋅\|x\)∥πθ\(⋅\|x\)\)\]\.\\mathbb\{E\}\_\{x\\sim\\tau\}\\left\[D\_\{\\mathrm\{KL\}\}\\left\(\\pi\_\{\\mathrm\{base\}\}\(\\cdot\|x\)\\,\\middle\\\|\\,\\pi\_\{\\theta\}\(\\cdot\|x\)\\right\)\\right\]\.\(1\)
### 3\.1Phase I: Reproducing Distributional Shift Effects

We first verify that RL preserves prior capabilities better than SFT while remaining closer to the pretrained policy\(Shenfeld et al\.,[2025](https://arxiv.org/html/2605.28860#bib.bib10)\)\. Models are trained on a downstream Task A and evaluated on a separate suite of retention benchmarks \(Task B\)\.

#### SFT\.

We fine\-tuneπbase\\pi\_\{\\mathrm\{base\}\}with a completion\-only cross\-entropy loss\.

#### RL\.

We refineπSFT\\pi\_\{\\mathrm\{SFT\}\}with Dr\.GRPO\. The model samples candidate completions, receives binary rewards, computes normalized group\-relative advantages, and updates the policy through a weighted log\-probability objective\. We use a group size of6464, two refinement steps \(μ=2\\mu=2\), and no explicit KL penalty\.

### 3\.2Phase II: Circuit Identification via Differential Binary Masking

We analyze circuits at the attention\-head level using Differential Binary Masking \(DBM\)\(Chaudhary & Geiger,[2024](https://arxiv.org/html/2605.28860#bib.bib1)\)\. DBM learns a mask over heads that interpolates between base and counterfactual activations:a~h=\(1−mh\)​ahbase\+mh​ahsource\\tilde\{a\}\_\{h\}=\(1\-m\_\{h\}\)a\_\{h\}^\{\\mathrm\{base\}\}\+m\_\{h\}a\_\{h\}^\{\\mathrm\{source\}\}

wheremh∈\[0,1\]m\_\{h\}\\in\[0,1\]\. Annealing pushes the masks toward binary selections, yielding sparse causal circuits\.

#### Triplets\.

For chemistry QA, we construct triplets\(xbase,xsource,ytarget\)\(x\_\{\\mathrm\{base\}\},x\_\{\\mathrm\{source\}\},y\_\{\\mathrm\{target\}\}\)corresponding to three counterfactual hypotheses: answer\-key swaps, molecule swaps, and task\-type swaps\.

#### Objective\.

Masks are optimized to increase the probability of the target answer while remaining sparse:ℒDBM=−log⁡P​\(ytarget\|x,a~\)\+λ​∑hmh\\mathcal\{L\}\_\{\\mathrm\{DBM\}\}=\-\\log P\(y\_\{\\mathrm\{target\}\}\|x,\\tilde\{a\}\)\+\\lambda\\sum\_\{h\}m\_\{h\}

#### Scoring\.

We score answers using the geometric mean of token probabilities

p​\(y\|x\)=exp⁡\(1T​∑i=1Tlog⁡P​\(yi\|x,y<i\)\)\.p\(y\|x\)=\\exp\\left\(\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}\\log P\(y\_\{i\}\|x,y\_\{<i\}\)\\right\)\.\(2\)
Circuit discovery is run independently forπbase\\pi\_\{\\mathrm\{base\}\},πSFT\\pi\_\{\\mathrm\{SFT\}\}, andπRL\\pi\_\{\\mathrm\{RL\}\}\.

### 3\.3Phase III: Cross\-Model Circuit Comparison

We assess how strongly the discovered circuits remain functional after post\-training\. Circuit faithfulness is defined as

Faithfulness​\(𝒞,M\)=F​\(𝒞\|M\)F​\(M\)\.\\mathrm\{Faithfulness\}\(\\mathcal\{C\},M\)=\\frac\{F\(\\mathcal\{C\}\|M\)\}\{F\(M\)\}\.\(3\)
with values close to11indicating that the circuit recovers most of the model’s behavior\.

For each headhh, we compare DBM mask values against those of the base model:

Δ​mh​\(M\)=mhM−mhbase,\\Delta m\_\{h\}\(M\)=m\_\{h\}^\{M\}\-m\_\{h\}^\{\\mathrm\{base\}\},\(4\)
whereM∈\{πSFT,πRL\}M\\in\\\{\\pi\_\{\\mathrm\{SFT\}\},\\pi\_\{\\mathrm\{RL\}\}\\\}\. This identifies heads that are preserved, amplified, or weakened by training\.

We definevulnerableheads as those more degraded under SFT than under RL:𝒞vuln=\{h:mhSFT<mhRL−δ\}\.\\mathcal\{C\}\_\{\\mathrm\{vuln\}\}=\\\{h:\\;m\_\{h\}^\{\\mathrm\{SFT\}\}<m\_\{h\}^\{\\mathrm\{RL\}\}\-\\delta\\\}\.

## 4Experiments

Our experiments evaluate the central claim of the paper: RL retains prior capabilities because it preserves task\-relevant circuits more effectively than SFT\. We address two questions: \(I\) Does RL induce less forgetting and lower behavioral drift than SFT; and \(II\) are these gains reflected in stronger circuit faithfulness, more stable head\-level contributions, and more distributed circuits?

### 4\.1Experimental Setup

All experiments use Qwen2\.5\-3B\-Instruct\. The fine\-tuning task \(Task A\) is scientific question answering\. Retention \(Task B\) is measured on benchmarks spanning commonsense reasoning, factuality, instruction following, and code generation, since forgetting can be capability\-specific\. We compare three systems: the pretrained base model, standard SFT, and RL with Dr\.GRPO\. Behavior is measured with downstream accuracy and KL divergence from the base model\. The mechanistic analysis uses circuit faithfulness, head\-level mask shiftsΔ​mh=mhM−mhbase\\Delta m\_\{h\}=m\_\{h\}^\{M\}\-m\_\{h\}^\{\\text\{base\}\}derived from cross\-model DBM mask comparison, and necessity/sufficiency interventions; necessity measures the log\-probability drop when a head is disrupted, while sufficiency measures how much behavior is recovered when only that head is kept\. Together, these metrics distinguish distributed contributors from critical bottlenecks\. Table[1](https://arxiv.org/html/2605.28860#A1.T1)summarizes the full setup\.

### 4\.2Results

DBM identifies a base circuit comprising 297 attention heads \(51\.6%51\.6\\%of all attention heads\)\. After adaptation, the SFT model exhibits structural compression to roughly 265 heads \(46\.0%46\.0\\%\), whereas the RL model retains approximately 296 heads \(51\.4%51\.4\\%\), close to the base model’s 297\. The same disparity is reflected in base\-circuit overlap: the RL model preserves about68%68\\%of base heads, substantially more than the SFT model’s52%52\\%\(Figure[4](https://arxiv.org/html/2605.28860#S4.F4)\)\.

Figure[1](https://arxiv.org/html/2605.28860#S1.F1)traces this divergence over the course of the training runs that yield the high new\-task score models\. Both objectives begin from full base\-circuit retention, but their trajectories separate immediately: after the first epoch SFT retains only63\.5%63\.5\\%of base heads compared to RL’s69\.8%69\.8\\%, and by the second epoch the gap widens to13\.513\.5percentage points \(59\.0%59\.0\\%vs\.72\.5%72\.5\\%\)\. Notably, RL’s retention*increases*between epochs, suggesting that continued RL training can recover previously disrupted circuit components rather than monotonically degrading them\. Throughout training, RL also maintains consistently higher DCM scores \(15\.815\.8vs\.10\.410\.4at Epoch 1;10\.610\.6vs\.6\.36\.3at Epoch 2\), indicating that RL’s preserved circuit carries substantially more causal signal on the answer\-key counterfactual\. The two objectives thus diverge along different axes of the same trade\-off: SFT collapses adaptation into a small specialist set that discards much of the base circuit, while RL spreads adaptation across a broader, more causally engaged subgraph that retains substantially more of the original computation\.

Circuit faithfulness exceeds1\.01\.0for all three models \(1\.021\.02base,1\.041\.04SFT,1\.121\.12RL\), confirming that the extracted subgraphs recover the full model’s behavior on the target task\.

![Refer to caption](https://arxiv.org/html/2605.28860v1/x2.png)Figure 2:Performance–preservation trade\-off across NT levels\.SFT \(dashed\) exhibits a sharp preservation drop in the high\-NTs regime, while RL \(solid\) declines gradually and preserves15\.815\.8percentage points more of the base circuit at peak new\-task performance\.![Refer to caption](https://arxiv.org/html/2605.28860v1/x3.png)Figure 3:Head Role Distribution Under Base, Supervised, and RL Training\.In our setup, SFT produces a cluster of “Critical Specialists”—heads with high necessity and sufficiency—while RL maintains a distributed architecture that overlaps closely with the base model, avoiding the structural compression and specialization observed under the supervised objective\.
### 4\.3Functional Importance and Circuit Drift

Beyond aggregate circuit size, we ask whether fine\-tuning preferentially modifies the heads that matter most to the base model\. Figure[3](https://arxiv.org/html/2605.28860#S4.F3)shows the necessity–sufficiency landscape: SFT produces a tight cluster of “critical specialists”—heads with both high necessity and high sufficiency—while RL maintains a flatter distribution that closely tracks the base model\. This is the mechanistic signature of compression: SFT consolidates new\-task behavior into a small set of high\-importance heads, whereas RL spreads adaptation across many components without any single one becoming indispensable\.

Figure[7](https://arxiv.org/html/2605.28860#A1.F7)probes this directly, plotting per\-head necessity against the mask shiftΔ​mh\\Delta m\_\{h\}\(Eq\.[4](https://arxiv.org/html/2605.28860#S3.E4)\)\. If either objective preferentially edited the most important heads, we would expect a positive correlation\. We do not observe one: SFT shows a weak negative trend \(r=−0\.125r\{=\}\{\-\}0\.125,p=0\.12p\{=\}0\.12\) and RL is essentially flat \(r=0\.022r\{=\}0\.022\)\. Read together with Figure[3](https://arxiv.org/html/2605.28860#S4.F3), the picture is consistent across both models: neither objective rewires the circuit by tracking base\-model necessity\. SFT instead reorganizes broadly while elevating a new set of specialists, whereas RL adjusts uniformly across heads without producing any such concentration\.

### 4\.4Discussion

Taken together, our findings indicate that SFT exhibits a mechanistic “breaking point\.” SFT and RL are comparable at low levels of adaptation, suggesting that both objectives can initially improve new\-task performance without substantially disrupting the base circuit\. In the high\-NTS regime, however, SFT shifts from conservative adaptation to aggressive circuit reconfiguration, while RL remains comparatively stable\. This pattern is consistent with RL preserving and reusing existing representations more effectively than SFT, which appears to overwrite portions of the base circuit under stronger new\-task pressure\. The epoch\-level trajectory in Figure[1](https://arxiv.org/html/2605.28860#S1.F1)reinforces this interpretation: SFT’s retention declines monotonically across epochs, whereas RL’s retention recovers after an initial drop, suggesting that RL can consolidate and restore circuit structure during continued training\. Confirming this interpretation will require experiments across additional models, tasks, and optimization settings\.

We also observe that the RL model maintains greater prior\-task retention and circuit preservation despite having a larger output\-space Kullback–Leibler \(KL\) divergence than the SFT model\. This suggests that output\-space metrics may not reliably predict internal forgetting in this setting\. In particular, RL may alter the model’s output distribution while preserving much of the underlying computation responsible for prior\-task performance\. By contrast, SFT can appear less divergent at the output level while still reorganizing the internal circuits more aggressively\.

![Refer to caption](https://arxiv.org/html/2605.28860v1/x4.png)Figure 4:Attention Head Overlap Between Base, SFT, and RL\.The plot shows the circuit overlap study for the base, SFT, and RL models\. The bars reflect the number of attention heads that are unique to one model, shared by two models, or present at all three levels of training\.The gap between the SFT and RL circuit sizes \(∼265\\sim 265vs\.∼295\\sim 295heads\), depicted in Figure[4](https://arxiv.org/html/2605.28860#S4.F4), supports a characterization of SFT as a “compressor” and RL as a “distributed adaptor\.” SFT concentrates new\-task behaviour into a smaller, more specialized circuit, while RL preserves a broader set of base\-model heads during adaptation\.

## 5Conclusion

We asked whether RL retains prior capabilities better than SFT because it better preserves internal circuits\. Our behavioral and mechanistic analyses on Qwen2\.5\-3B\-Instruct support this hypothesis: RL stays closer to the pretrained model, preserves more of the original circuit, and maintains stronger functional faithfulness, while SFT adapts faster but causes greater circuit reorganization and more forgetting in our experiments\.

These results point to a potential trade\-off in post\-training: rapid specialization versus stable reuse of existing mechanisms\. In our setup, SFT optimizes the new objective efficiently but can disrupt circuits supporting prior skills, while RL is more conservative, preserving those circuits at the cost of smaller task gains\. More broadly, future adaptation methods may benefit from combining efficient learning with selective circuit preservation, toward models that continue improving without forgetting\.

## 6Limitations and Future Work

This study examines only one model \(Qwen2\.5\-3B\-Instruct\), limiting the generalizability of our findings\. Future work should validate these trade\-offs across diverse architectures: Gemma, Mistral, Llama, and Pythia, and at varying parameter scales\. Our circuit analysis is further constrained to attention heads and a narrow task set\. Expanding to MLP layers, residual\-stream features, and broader capability domains: multilingual reasoning, factual recall, safety, and tool use, would yield a more complete mechanistic picture\.

## References

- Chaudhary & Geiger \(2024\)Chaudhary, M\. and Geiger, A\.Evaluating open\-source sparse autoencoders on disentangling factual knowledge in gpt\-2 small\.*arXiv preprint arXiv:2409\.04478*, 2024\.
- Chen et al\. \(2021\)Chen, M\., Tworek, J\., Jun, H\., Yuan, Q\., Pinto, H\. P\. d\. O\., Kaplan, J\., Edwards, H\., Burda, Y\., Joseph, N\., Brockman, G\., Ray, A\., Puri, R\., Krueger, G\., Petrov, M\., Khlaaf, H\., Sastry, G\., Mishkin, P\., Chan, B\., Gray, S\., Ryder, N\., Pavlov, M\., Power, A\., Kaiser, L\., Bavarian, M\., Winter, C\., Tillet, P\., Such, F\. P\., Cummings, D\., Plappert, M\., Chantzis, F\., Barnes, E\., Herbert\-Voss, A\., Guss, W\. H\., Nichol, A\., Paino, A\., Tezak, N\., Tang, J\., Babuschkin, I\., Balaji, S\., Jain, S\., Saunders, W\., Hesse, C\., Carr, A\. N\., Leike, J\., Achiam, J\., Misra, V\., Morikawa, E\., Radford, A\., Knight, M\., Brundage, M\., Murati, M\., Mayer, K\., Welinder, P\., McGrew, B\., Amodei, D\., McCandlish, S\., Sutskever, I\., and Zaremba, W\.Evaluating large language models trained on code\.*arXiv preprint arXiv:2107\.03374*, 2021\.
- Davies et al\. \(2023\)Davies, X\., Nadeau, M\., Prakash, N\., Shaham, T\. R\., and Bau, D\.Discovering variable binding circuitry with desiderata\.*arXiv preprint arXiv:2310\.02336*, 2023\.
- Feng et al\. \(2025\)Feng, K\., Shen, X\., Wang, W\., Zhuang, X\., Tang, Y\., Zhang, Q\., and Ding, K\.Sciknoweval: Evaluating multi\-level scientific knowledge of large language models\.*arXiv preprint*, 2025\.
- Hendrycks et al\. \(2021\)Hendrycks, D\., Burns, C\., Basart, S\., Zou, A\., Mazeika, M\., Song, D\., and Steinhardt, J\.Measuring massive multitask language understanding\.*arXiv preprint arXiv:2009\.03300*, 2021\.
- Hu et al\. \(2025\)Hu, J\., Zhang, Y\., Han, Q\., Jiang, D\., Zhang, X\., and Shum, H\.\-Y\.Open\-reasoner\-zero: An open source approach to scaling up reinforcement learning on the base model\.*arXiv preprint*, 2025\.
- Lin et al\. \(2022\)Lin, S\., Hilton, J\., and Evans, O\.Truthfulqa: Measuring how models mimic human falsehoods\.*arXiv preprint arXiv:2109\.07958*, 2022\.
- Prakash et al\. \(2024\)Prakash, N\., Shaham, T\. R\., Haklay, T\., Belinkov, Y\., and Bau, D\.Fine\-tuning enhances existing mechanisms: A case study on entity tracking\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Sakaguchi et al\. \(2020\)Sakaguchi, K\., Le Bras, R\., Bhagavatula, C\., and Choi, Y\.Winogrande: An adversarial winograd schema challenge at scale\.In*AAAI Conference on Artificial Intelligence*, 2020\.
- Shenfeld et al\. \(2025\)Shenfeld, I\., Pari, J\., and Agrawal, P\.Rl’s razor: Why online reinforcement learning forgets less\.*arXiv preprint arXiv:2509\.04259*, 2025\.
- Zellers et al\. \(2019\)Zellers, R\., Holtzman, A\., Bisk, Y\., Farhadi, A\., and Choi, Y\.Hellaswag: Can a machine really finish your sentence?In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2019\.
- Zhou et al\. \(2023\)Zhou, J\., Lu, T\., Mishra, S\., Brahma, S\., Basu, S\., Luan, Y\., Zhou, D\., and Hou, L\.Instruction\-following evaluation for large language models\.In*arXiv preprint arXiv:2311\.07911*, 2023\.

## Appendix AAppendix

Table 1:Experimental Setup![Refer to caption](https://arxiv.org/html/2605.28860v1/x5.png)Figure 5:Layer\-wise Circuit Retention in RL — Our RL model shows architectural stability across all 36 transformer layers, with a high count of retained heads and relatively few forgotten components throughout the network depth\.![Refer to caption](https://arxiv.org/html/2605.28860v1/x6.png)Figure 6:Layer\-wise Circuit Retention in SFT — Our SFT model shows broader structural change, with forgotten heads scattered throughout all layers and higher concentrations in the mid\-to\-late transformer layers\.![Refer to caption](https://arxiv.org/html/2605.28860v1/x7.png)Figure 7:Per\-Head Necessity vs\.Δ​mh\\Delta m\_\{h\}: SFT vs\. RL\.We plot per\-head necessity againstΔ​mh\\Delta m\_\{h\}\(Eq\.[4](https://arxiv.org/html/2605.28860#S3.E4)\) for both models\. The absence of a positive correlation under either objective indicates that mask shifts are not driven by head necessity alone\. SFT exhibits a weak negative trend \(r=−0\.125r\{=\}\{\-\}0\.125\), suggesting it suppresses heads irrespective of their functional role, while RL’s flat relationship \(r=0\.022r\{=\}0\.022\) is consistent with its more uniform, distributed adaptation pattern\.![Refer to caption](https://arxiv.org/html/2605.28860v1/x8.png)Figure 8:This graph illustrates the number of shared \(overlapping\) heads among the ’Base’, ’SFT’ \(Supervised Fine\-Tuning\), and ’RL’ \(Reinforcement Learning\) circuits\. Diagonal elements \(such as Base\-Base, SFT\-SFT, and RL\-RL\) indicate the entire size \(number of heads\) of each particular circuit\. Off\-diagonal elements \(e\.g\., Base\-SFT, SFT\-RL\) represent the number of heads shared between two separate circuits\. For example, the cell at \(Base, SFT\) displays how many heads are present in both the Base and SFT circuits\.