Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining

# Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
Source: [https://arxiv.org/html/2605.20296](https://arxiv.org/html/2605.20296)
Aarash Abro Zeta Labs aarash@zetasolutions\.org &Muhammad Tahir Lahore University of Management Sciences tahir@lums\.edu\.pk

###### Abstract

Fine\-tuning a language model for a target task routinely degrades capabilities the training data never explicitly threatened\. We study this phenomenon, known as catastrophic forgetting, and propose a post\-hoc repair solution that uses only the pretrained checkpointWbaseW\_\{\\mathrm\{base\}\}and its fine\-tuned descendantWftW\_\{\\mathrm\{ft\}\}\. The goal is not merely to revert the model toward the base checkpoint, but to recover capabilities damaged by fine\-tuning while preserving both the target\-task gains and any beneficial held\-out improvements\. We introduce DG\-Hard, a checkpoint\-only spectral repair method for the fine\-tuning updateΔ=Wft−Wbase\\Delta=W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}\. DG\-Hard treatsΔ\\Deltaas a low\-rank task\-aligned signal embedded in an IID\-like noise residual that gradient descent has no incentive to remove, and applies the Donoho\-Gavish hard singular\-value threshold to each weight\-delta matrix, keeping the structured high\-energy part of the update and removing the spectral bulk\. This reduces repair to a closed\-form SVD filtering step requiring no data\-dependent tuning\. A central difficulty is evaluation: average accuracy hides per\-benchmark failures, while naive recovery scores reward models that simply revert toward the base\. We therefore introduce a partition\-conditional metric that separately tracks healing, preservation, non\-damage, and target\-task retention\. Across1414\(model, task\) settings and nine cross\-domain held\-out benchmarks, DG\-Hard achieves the strongest balanced repair among post\-hoc baselines\. DG\-Hard also restores safety alignment degraded by benign fine\-tuning on three independent safety axes, despite using no alignment data\. These results suggest that part of fine\-tuning\-induced capability loss is not an unavoidable consequence of specialization, but a removable spectral residue in the weight update itself\. The code can be found at:[https://github\.com/BrickleRex/dghard](https://github.com/BrickleRex/dghard)\.

## 1Introduction

Fine\-tuning can improve a model on the task it is trained for while destroying capabilities that were already present in the pretrained checkpoint\. In our experiments, fine\-tuning Qwen3\.5\-4B on the medical domain raises medical\-question\-answering accuracy, but reduces math\-reasoning accuracy drastically\. This is not an isolated failure mode: across the1414\(model, task\) fine\-tuning cells we evaluate,1313exhibit at least one single\-benchmark collapse \(Tab\.[9](https://arxiv.org/html/2605.20296#A5.T9), App\.[E](https://arxiv.org/html/2605.20296#A5)\)\. Such failures are the modern large\-model form of*catastrophic forgetting*\[[23](https://arxiv.org/html/2605.20296#bib.bib5)\]: adaptation to a new objective can overwrite, distort, or suppress behavior the base model had already acquired\[[27](https://arxiv.org/html/2605.20296#bib.bib35),[6](https://arxiv.org/html/2605.20296#bib.bib18)\]\.

Fine\-tuning is the standard way pretrained models are specialized\[[3](https://arxiv.org/html/2605.20296#bib.bib14),[9](https://arxiv.org/html/2605.20296#bib.bib21),[5](https://arxiv.org/html/2605.20296#bib.bib16),[24](https://arxiv.org/html/2605.20296#bib.bib32)\], yet its objective contains no term requiring unrelated capabilities to be preserved\. The consequences are documented across general\-knowledge accuracy\[[19](https://arxiv.org/html/2605.20296#bib.bib29)\], safety alignment in aligned models\[[26](https://arxiv.org/html/2605.20296#bib.bib34)\], and the geometric distortion of pretrained features outside the training\-data span\[[17](https://arxiv.org/html/2605.20296#bib.bib11)\]\. The checkpoint moves in two ways at once: a structured update lowering the target\-task loss, and a residual from many mini\-batch SGD steps whose noise scale depends on batch size\[[13](https://arxiv.org/html/2605.20296#bib.bib9),[15](https://arxiv.org/html/2605.20296#bib.bib8)\]\. The loss rewards only the first; the second accumulates in directions important for other capabilities, leaving the fine\-tuned checkpoint a mixture of task\-aligned signal and collateral change\.

We study the post\-hoc repair of forgetting that has already happened\. Given only a base checkpointWbaseW\_\{\\mathrm\{base\}\}and a fine\-tuned checkpointWftW\_\{\\mathrm\{ft\}\}, the goal is to recover damaged held\-out capabilities while preserving the gains fine\-tuning was meant to produce, and has incidentally produced\. Reversion toward the base recovers forgotten behavior cheaply but also removes the task\-aligned update; a useful repair must distinguish the part of the fine\-tuning deltaΔ=Wft−Wbase\\Delta=W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}that carries the new task from the part that causes collateral damage\.

Existing post\-hoc methods make this decision in coordinate space, whether by scalar interpolation \(WiSE\-FT,[34](https://arxiv.org/html/2605.20296#bib.bib45)\), random dropping \(DARE,[36](https://arxiv.org/html/2605.20296#bib.bib48)\), magnitude\- and sign\-aware pruning \(TIES,[35](https://arxiv.org/html/2605.20296#bib.bib47)\), or forgetting\-aware pruning \(FAPM,[11](https://arxiv.org/html/2605.20296#bib.bib23)\)\. All face the same representation problem: task\-relevant entries and harmful residual entries are interleaved by magnitude, sign, and position\.

The same update is more separable in singular\-value space\. Across fine\-tuning deltas, the spectrum splits into a broad bulk that matches the random\-matrix prediction\[[20](https://arxiv.org/html/2605.20296#bib.bib4)\]and a smaller number of outlying spikes that carry the task\-aligned update\. Both halves are independently supported in prior work: fine\-tuning weight updates are rank\-deficient\[[10](https://arxiv.org/html/2605.20296#bib.bib22),[1](https://arxiv.org/html/2605.20296#bib.bib12)\], and trained\-weight spectra fit the Marchenko\-Pastur bulk past a finite number of outliers\[[33](https://arxiv.org/html/2605.20296#bib.bib41),[32](https://arxiv.org/html/2605.20296#bib.bib40)\]; we verify both on our own deltas in App\.[A](https://arxiv.org/html/2605.20296#A1)\(formalized in §[3](https://arxiv.org/html/2605.20296#S3)\)\. Repair becomes a matrix\-denoising problem: keep the structure, revert the bulk\.

We instantiate this withDG\-Hard, a closed\-form spectral repair: each fine\-tuning delta matrix’s SVD is hard\-thresholded at the Donoho\-Gavish cut\[[7](https://arxiv.org/html/2605.20296#bib.bib17)\], and the surviving singular components form a spectrally pruned deltaΔ∗\\Delta^\{\*\}that yields the repaired checkpointW∗=Wbase\+Δ∗W^\{\*\}=W\_\{\\mathrm\{base\}\}\+\\Delta^\{\*\}\(Alg\.[1](https://arxiv.org/html/2605.20296#alg1)\)\. The method is data\-free, gradient\-free, training\-free, and runs in minutes on a single GPU\.

#### Contributions\.

1. 1\.We formulate post\-hoc repair of catastrophic forgetting as a recovery\-preservation problem: recover damaged held\-out capabilities from a fine\-tuned checkpoint while preserving target\-task gains and incidental held\-out improvements\.
2. 2\.We identify a spectral structure in fine\-tuning deltas\. Empirically, damaging residuals concentrate in the singular\-value bulk, while task\-relevant updates appear as singular\-value spikes \(App\.[A](https://arxiv.org/html/2605.20296#A1)\)\. This explains why scalar interpolation and coordinate\-wise pruning face an unfavorable trade\-off\. We proposeDG\-Hard, a data\-free repair method that applies the Donoho\-Gavish hard threshold\[[7](https://arxiv.org/html/2605.20296#bib.bib17)\]to each delta matrix and reconstructs the checkpoint from retained singular components\.
3. 3\.We introduce a partition\-conditional evaluation that reports recovery on damaged measurements, preservation on improved measurements, and retention on unchanged and target\-task measurements\. DG\-Hard achieves the best recovery\-preservation trade\-off among post\-hoc baselines across1414\(model, task\) cells and99held\-out benchmarks \(Tab\.[1](https://arxiv.org/html/2605.20296#S4.T1), Fig\.[1](https://arxiv.org/html/2605.20296#S1.F1)\)\.

![Refer to caption](https://arxiv.org/html/2605.20296v1/figures/recovery_preservation_scatter.png)Figure 1:Recovery×\\timespreservation per cohort\. Each panel plots the % healed score on the damaged partition \(x\-axis\) against the % preserved score on the improved partition \(y\-axis\), as defined in §[4\.3](https://arxiv.org/html/2605.20296#S4.SS3)\. The ideal corner is\(100,100\)\(100,100\), and the dotted contour marksHM​\(% healed,% preserved\)=80\\mathrm\{HM\}\(\\text\{\\% healed\},\\,\\text\{\\% preserved\}\)=80\. DG\-Hard \(blue diamond\) is closest to the ideal corner across all five cohorts\. FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\]strongly recovers damaged measurements but sacrifices improved ones; V\-SoftMask\[[14](https://arxiv.org/html/2605.20296#bib.bib26)\]preserves improved measurements but recovers less damage\.

## 2Background and related work

#### Catastrophic forgetting\.

We extend the connectionist account ofMcCloskey and Cohen \[[23](https://arxiv.org/html/2605.20296#bib.bib5)\]that distributed representations make any helpful update perturb the weights encoding prior tasks\[[27](https://arxiv.org/html/2605.20296#bib.bib35),[6](https://arxiv.org/html/2605.20296#bib.bib18)\]\.Kumaret al\.\[[17](https://arxiv.org/html/2605.20296#bib.bib11)\]formalize the modern instance in an overparameterized linear setting, andLuoet al\.\[[19](https://arxiv.org/html/2605.20296#bib.bib29)\], Qiet al\.\[[26](https://arxiv.org/html/2605.20296#bib.bib34)\]document its empirical reproduction in contemporary LLMs across general\-knowledge and safety benchmarks\.

#### Training\-time approaches\.

Training\-time methods modify the optimization step itself and fall into three subfamilies: parameter\-movement regularizers, gradient\-masking via per\-unit importance, and replay against stored past\-task data\. Parameter\-movement regularizers penalize updates weighted by per\-parameter importance and differ mostly in how importance is estimated:EWC\[[16](https://arxiv.org/html/2605.20296#bib.bib27)\]uses the Fisher diagonal,SI\[[37](https://arxiv.org/html/2605.20296#bib.bib49)\]a path\-integral measure of online contribution to the previous\-task loss, andMAS\[[2](https://arxiv.org/html/2605.20296#bib.bib13)\]the gradient of the squared L2 norm of the network output \(label\-free, since it estimates importance from unlabeled calibration data\)\. We additionally compare against a genericL1\-regbaseline that penalizes‖W−Wbase‖1\\\|W\-W\_\{\\mathrm\{base\}\}\\\|\_\{1\}without task information\. Gradient\-masking methods identify which units carry pretrained capability and slow the gradient flow through them:DAS\[[14](https://arxiv.org/html/2605.20296#bib.bib26)\]multiplies each gradient by\(1−importance\)\(1\-\\text\{importance\}\)where importance comes from a dropout\-KL proxy on calibration data, andCoFiTune\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\]extends it with a coarse\-grained layer\-range filter \(restricting the procedure to an empirically selected slice of layers\) and a KL/dropout\-robustness fine score\. Replay\-based methods mix or project against stored past\-task examples \(GEM\[[18](https://arxiv.org/html/2605.20296#bib.bib28)\],Experience Replay\[[4](https://arxiv.org/html/2605.20296#bib.bib15)\]\) but assume access to a representative pretraining sample, which is unavailable for modern LLMs whose pretraining corpora are proprietary and at terabyte scale\.

#### Parameter\-efficient fine\-tuning\.

LoRA\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]constrains the fine\-tuning update on each weight matrix to a low\-rank factorizationB​ABAwithr≪min⁡\(m,n\)r\\ll\\min\(m,n\), leaving the base weights frozen; the deployed model still merges base \+ LoRA at inference, so forgetting can persist in the merged weights\.Shuttleworthet al\.\[[30](https://arxiv.org/html/2605.20296#bib.bib38)\]show via direct spectral comparison that this low\-rank constraint forces LoRA\-FT updates to introduce “intruder” singular directions approximately orthogonal to the pretrained spectrum, and that these intruder directions causally drive forgetting \(verified by post\-hoc intervention on their singular values\)\.

#### Post\-hoc model merging\.

The family closest to our work operates post\-hoc onΔ=Wft−Wbase\\Delta=W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}via cheap, data\-free transformations\. These methods originally targeted multi\-task model merging, where multiple fine\-tuned checkpoints are combined into one; with a single fine\-tune their multi\-task aggregation steps \(sign election, mean over task vectors\) reduce to identity, leaving the per\-vector preprocessing as a post\-hoc repair on the\(Wbase,Wft\)\(W\_\{\\mathrm\{base\}\},W\_\{\\mathrm\{ft\}\}\)pair\.WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\]interpolates linearly between the checkpoints,W∗=\(1−α\)​Wbase\+α​WftW^\{\*\}=\(1\{\-\}\\alpha\)W\_\{\\mathrm\{base\}\}\+\\alpha W\_\{\\mathrm\{ft\}\}, trading pretrained and fine\-tuned behavior with a single scalar\.Task Arithmetic\[[12](https://arxiv.org/html/2605.20296#bib.bib24)\]introducesΔ\\Deltaas a task vector and studies algebraic operations on it \(negation, addition, analogies\) across multiple tasks\.TIES\-Merging\[[35](https://arxiv.org/html/2605.20296#bib.bib47)\]trims low\-magnitude entries of each task vector, elects a per\-parameter consensus sign across vectors, and averages only the entries aligned with that sign, addressing redundancy and sign disagreement as sources of merge interference\.DARE\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\]randomly dropsΔ\\Deltaentries with probabilityppand rescales survivors by1/\(1−p\)1/\(1\{\-\}p\), designed as a preprocessing step that sparsifies fine\-tuning deltas before merging\.FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\]scores each entry with a forgetting\-aware criterion combining absolute change magnitude and a relative\-change penalty against the pretrained weight, then prunes low\-scoring entries\.Fisher\-weighted averaging\[[21](https://arxiv.org/html/2605.20296#bib.bib30)\]weights each parameter by its empirical Fisher importance, additionally requiring a small labeled training sample\. Per\-method hyperparameters are in App\.[H\.1](https://arxiv.org/html/2605.20296#A8.SS1)\.

#### Spectral shrinkage and our distinction\.

The spectral framing fits fine\-tuning because the weight update is empirically rank\-deficient \([10](https://arxiv.org/html/2605.20296#bib.bib22); broader evidence in App\.[A](https://arxiv.org/html/2605.20296#A1)\), and random\-matrix theory supplies the matching denoiser\. The Marchenko\-Pastur theorem\[[20](https://arxiv.org/html/2605.20296#bib.bib4)\]fixes the asymptotic spectrum of an IID\-noise matrix, and the Donoho\-Gavish hard threshold\[[7](https://arxiv.org/html/2605.20296#bib.bib17)\]identifies the AMSE\-optimal cut between signal and noise \([8](https://arxiv.org/html/2605.20296#bib.bib19)extend to optimal continuous shrinkers\)\. Where prior post\-hoc methods operate in coordinate space \(TIES, DARE, FAPM\), as a single scalar \(WiSE\-FT, Task Arithmetic\), or via labeled\-data Fisher weighting, we operate in singular\-value space and apply this statistically optimal hard threshold, to our knowledge the first such application to fine\-tuning weight deltas for post\-hoc forgetting repair\.

## 3Method

We proposeDG\-Hard: a post\-hoc, data\-free, gradient\-free repair that strips the IID noise residual from a fine\-tuning update by applying the optimal hard singular\-value threshold ofGavish and Donoho \[[7](https://arxiv.org/html/2605.20296#bib.bib17)\]to its SVD\. The procedure takes only the base checkpointWbaseW\_\{\\mathrm\{base\}\}and the fine\-tuned checkpointWftW\_\{\\mathrm\{ft\}\}, operates one22D weight matrix at a time, and reduces to a closed\-form threshold in the matrix shape and a single estimated noise scale\.

### 3\.1Signal\-plus\-noise model of the fine\-tuning update

For a single22D weight matrix inℝm×n\\mathbb\{R\}^\{m\\times n\}, write the fine\-tuning update asΔ=Wft−Wbase\\Delta=W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}and take its singular value decomposition,

Δ=U​diag⁡\(𝐬\)​V⊤,s1≥⋯≥sp≥0,p=min⁡\(m,n\)\.\\Delta\\;=\\;U\\,\\operatorname\{diag\}\(\\mathbf\{s\}\)\\,V^\{\\\!\\top\},\\quad s\_\{1\}\\geq\\cdots\\geq s\_\{p\}\\geq 0,\\quad p=\\min\(m,n\)\.\(1\)We modelΔ\\Deltaas a two\-component sum,

Δ=Δsignal\+Δnoise,\\Delta\\;=\\;\\Delta\_\{\\mathrm\{signal\}\}\+\\Delta\_\{\\mathrm\{noise\}\},\(2\)withΔsignal\\Delta\_\{\\mathrm\{signal\}\}a low\-rank task\-aligned update andΔnoise\\Delta\_\{\\mathrm\{noise\}\}an IID residual of unknown scaleσ\\sigma\. The SGD\-noise mechanism in §[1](https://arxiv.org/html/2605.20296#S1)produces exactly this structure: thousands of mini\-batch residuals accumulate into the weights while the gradient has no incentive to remove them\. Both components are empirically supported: the fine\-tuning weight update is rank\-deficient\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]and fine\-tuning more broadly has low intrinsic dimensionality\[[1](https://arxiv.org/html/2605.20296#bib.bib12)\]; trained\-weight spectra fit the Marchenko\-Pastur bulk past a finite number of outliers\[[33](https://arxiv.org/html/2605.20296#bib.bib41),[32](https://arxiv.org/html/2605.20296#bib.bib40)\]\. A direct check on our own deltas confirms both properties layer\-locally \(App\.[A](https://arxiv.org/html/2605.20296#A1), Fig\.[4](https://arxiv.org/html/2605.20296#A1.F4)\)\. Our goal is to removeΔnoise\\Delta\_\{\\mathrm\{noise\}\}while leavingΔsignal\\Delta\_\{\\mathrm\{signal\}\}intact\.

### 3\.2Random\-matrix theory and the bulk edge

IfΔ\\Deltawere pure IID noise of varianceσ2\\sigma^\{2\}, the Marchenko\-Pastur theorem\[[20](https://arxiv.org/html/2605.20296#bib.bib4)\]pins its spectrum exactly: in the limitm,n→∞m,n\\to\\inftyat fixed aspect ratioβ=min⁡\(m,n\)/max⁡\(m,n\)\\beta=\\min\(m,n\)/\\max\(m,n\), every singular value sits below the bulk edge

λMP=σ​\(1\+β\)\\lambda\_\{\\text\{MP\}\}\\;=\\;\\sigma\\,\\bigl\(1\+\\sqrt\{\\beta\}\\bigr\)\(3\)with probability one, depending only onσ\\sigmaandβ\\beta\. Contrapositively, anysi\>λMPs\_\{i\}\>\\lambda\_\{\\text\{MP\}\}inΔ\\Deltamust reflect the signal component of \([2](https://arxiv.org/html/2605.20296#S3.E2)\)\. The edge is a principled boundary but not a denoising rule; for that we need a*shrinker*mapping eachsis\_\{i\}to a denoisedsi′s^\{\\prime\}\_\{i\}\. The simplest principled choice is a hard threshold, andGavish and Donoho \[[7](https://arxiv.org/html/2605.20296#bib.bib17)\]derive the AMSE\-optimal one\. Asymptotic mean squared error \(AMSE\) is them,n→∞m,n\\to\\inftylimit of the expected reconstruction error𝔼​\[‖Δ^signal−Δsignal‖F2\]\\mathbb\{E\}\\bigl\[\\\|\\hat\{\\Delta\}\_\{\\mathrm\{signal\}\}\-\\Delta\_\{\\mathrm\{signal\}\}\\\|\_\{F\}^\{2\}\\bigr\]\(squared Frobenius distance between the estimated and true low\-rank signal\) at fixed aspect ratioβ\\beta\. A low threshold admits noise \(high variance\); a high threshold discards signal \(high bias\); the AMSE\-optimal threshold sits at the trade\-off minimum\.

### 3\.3DG hard: the optimal hard threshold

Among all hard shrinkersη​\(s\)=s⋅𝟏​\{s\>τ\}\\eta\(s\)=s\\cdot\\mathbf\{1\}\\\{s\>\\tau\\\},Gavish and Donoho \[[7](https://arxiv.org/html/2605.20296#bib.bib17)\]identify the threshold minimizing the AMSE of the reconstruction under \([2](https://arxiv.org/html/2605.20296#S3.E2)\):

τ∗=ω​\(β\)​σ​max⁡\(m,n\),ω​\(β\)=2​\(β\+1\)\+8​β\(β\+1\)\+β2\+14​β\+1\.\\tau^\{\*\}=\\omega\(\\beta\)\\,\\sigma\\,\\sqrt\{\\max\(m,n\)\},\\qquad\\omega\(\\beta\)=\\sqrt\{\\,2\(\\beta\+1\)\+\\tfrac\{8\\beta\}\{\(\\beta\+1\)\+\\sqrt\{\\beta^\{2\}\+14\\beta\+1\}\}\\,\}\.\(4\)For square matrices,ω​\(1\)=4/3≈2\.309\\omega\(1\)=4/\\sqrt\{3\}\\approx 2\.309, the constant for which the original paper is named\. The DG\-Hard shrinker is

ηDG​\(s\)=s⋅𝟏​\{s\>τ∗\}\.\\eta\_\{\\mathrm\{DG\}\}\(s\)\\;=\\;s\\cdot\\mathbf\{1\}\\\!\\bigl\\\{\\,s\>\\tau^\{\*\}\\bigr\\\}\.\(5\)τ∗\\tau^\{\*\}sits strictly aboveλMP\\lambda\_\{\\text\{MP\}\}\(≈2\.309​σ​n\\approx 2\.309\\,\\sigma\\sqrt\{n\}vs\.2​σ​n2\\sigma\\sqrt\{n\}atβ=1\\beta=1\): cutting at the edge would admit singular values barely distinguishable from noise, paying full reconstruction variance for negligible signal\. Eq\. \([4](https://arxiv.org/html/2605.20296#S3.E4)\) is the unique threshold at which marginal bias and variance balance\. Algorithm[1](https://arxiv.org/html/2605.20296#alg1)gives the full per\-matrix procedure\. The noise estimatorσ^\\hat\{\\sigma\}on line[6](https://arxiv.org/html/2605.20296#alg1.l6)is the median\-based Donoho\-Gavish estimatorσ^=median⁡\(𝐬\)/\(μβ​max⁡\(m,n\)\)\\hat\{\\sigma\}=\\operatorname\{median\}\(\\mathbf\{s\}\)/\(\\mu\_\{\\beta\}\\sqrt\{\\max\(m,n\)\}\), which substitutes into \([4](https://arxiv.org/html/2605.20296#S3.E4)\) to give a data\-only thresholdτ∗=\(ω​\(β\)/μβ\)​median⁡\(𝐬\)≈2\.858​median⁡\(𝐬\)\\tau^\{\*\}=\(\\omega\(\\beta\)/\\mu\_\{\\beta\}\)\\,\\operatorname\{median\}\(\\mathbf\{s\}\)\\approx 2\.858\\,\\operatorname\{median\}\(\\mathbf\{s\}\)for square matrices; full derivation and the SVD\-dominated cost \(O\(min\(m,n\)2max\(m,n\)\)O\(\\min\(m,n\)^\{2\}\\max\(m,n\)\)per matrix\) are in App\.[B](https://arxiv.org/html/2605.20296#A2)\.

Algorithm 1DG hard repair \(per weight matrix\)\.1:Base weight

Wbase∈ℝm×nW\_\{\\mathrm\{base\}\}\\in\\mathbb\{R\}^\{m\\times n\}, fine\-tuned weight

Wft∈ℝm×nW\_\{\\mathrm\{ft\}\}\\in\\mathbb\{R\}^\{m\\times n\}\.

2:Repaired weight

W∗∈ℝm×nW^\{\*\}\\in\\mathbb\{R\}^\{m\\times n\}\.

3:

Δ←Wft−Wbase\\Delta\\leftarrow W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}
4:

\(U,𝐬,V\)←SVD​\(Δ\)\(U,\\mathbf\{s\},V\)\\leftarrow\\mathrm\{SVD\}\(\\Delta\)
5:

p←min⁡\(m,n\);β←p/max⁡\(m,n\)p\\leftarrow\\min\(m,n\);\\;\\;\\beta\\leftarrow p/\\max\(m,n\)
6:

σ^←median⁡\(𝐬\)/\(μβ​max⁡\(m,n\)\)\\hat\{\\sigma\}\\leftarrow\\operatorname\{median\}\(\\mathbf\{s\}\)/\\bigl\(\\mu\_\{\\beta\}\\sqrt\{\\max\(m,n\)\}\\bigr\)⊳\\trianglerightnoise scale, App\.[B](https://arxiv.org/html/2605.20296#A2)

7:

τ←ω​\(β\)​σ^​max⁡\(m,n\)\\tau\\leftarrow\\omega\(\\beta\)\\,\\hat\{\\sigma\}\\,\\sqrt\{\\max\(m,n\)\}⊳\\trianglerightoptimal threshold, \([4](https://arxiv.org/html/2605.20296#S3.E4)\)

8:

si′←si⋅𝟏​\{si\>τ\}s^\{\\prime\}\_\{i\}\\leftarrow s\_\{i\}\\cdot\\mathbf\{1\}\\\{s\_\{i\}\>\\tau\\\}for

i=1,…,pi=1,\\ldots,p
9:

Δ∗←U​diag⁡\(𝐬′\)​V⊤\\Delta^\{\*\}\\leftarrow U\\,\\operatorname\{diag\}\(\\mathbf\{s\}^\{\\prime\}\)\\,V^\{\\\!\\top\}
10:return

Wbase\+Δ∗W\_\{\\mathrm\{base\}\}\+\\Delta^\{\*\}

### 3\.4Evaluation protocol

For each held\-out benchmarkbbin every \(model, task, method\) cell we readscoreb∈\[0,1\]\\mathrm\{score\}\_\{b\}\\in\[0,1\]and define

ΔFT​\(b\)=scoreFT​\(b\)−scorebase​\(b\),Δmethod​\(b\)=scoremethod​\(b\)−scorebase​\(b\)\.\\Delta\_\{\\mathrm\{FT\}\}\(b\)=\\mathrm\{score\}\_\{\\mathrm\{FT\}\}\(b\)\-\\mathrm\{score\}\_\{\\mathrm\{base\}\}\(b\),\\qquad\\Delta\_\{\\mathrm\{method\}\}\(b\)=\\mathrm\{score\}\_\{\\mathrm\{method\}\}\(b\)\-\\mathrm\{score\}\_\{\\mathrm\{base\}\}\(b\)\.\(6\)At a±3\\pm 3pp threshold each \(model, task,bb\) triple isdamagedifΔFT​\(b\)≤−3\\Delta\_\{\\mathrm\{FT\}\}\(b\)\\leq\-3,improvedifΔFT​\(b\)≥\+3\\Delta\_\{\\mathrm\{FT\}\}\(b\)\\geq\+3, andunchangedotherwise\. The partition is fixed by the FT checkpoint and identical across methods, so per\-method comparisons operate on the same set of triples in each partition cell\. Across the126126\-cell matrix \(22models×\\times77tasks×\\times99benchmarks\) we observe3030damaged,5555improved,4141unchanged triples\.

#### Per\-method statistics\.

LetsM=scoremethods\_\{M\}=\\mathrm\{score\}\_\{\\mathrm\{method\}\},sF=scoreFTs\_\{F\}=\\mathrm\{score\}\_\{\\mathrm\{FT\}\}, andsB=scorebases\_\{B\}=\\mathrm\{score\}\_\{\\mathrm\{base\}\}\. LetD,I,UD,I,Udenote the damaged, improved, and unchanged partition sets, and HM the harmonic mean\. Each method produces seven percentage scores\. Higher is better;100100marks full recovery or preservation, values above100100indicate overshoot, values below0indicate regression, and HM aggregates are floored at0:

% healed=avgb∈DsM​\(b\)−sF​\(b\)sB​\(b\)−sF​\(b\)⋅100,% preserved=avgb∈IsM​\(b\)−sB​\(b\)sF​\(b\)−sB​\(b\)⋅100,\\displaystyle=\\mathop\{\\mathrm\{avg\}\}\_\{b\\in D\}\\;\\dfrac\{s\_\{M\}\(b\)\-s\_\{F\}\(b\)\}\{s\_\{B\}\(b\)\-s\_\{F\}\(b\)\}\\cdot 100,\\quad\\textbf\{\\% preserved\}=\\mathop\{\\mathrm\{avg\}\}\_\{b\\in I\}\\;\\dfrac\{s\_\{M\}\(b\)\-s\_\{B\}\(b\)\}\{s\_\{F\}\(b\)\-s\_\{B\}\(b\)\}\\cdot 100,\(7a\)on\-task ret\.=avgcellssM​\(task\)sF​\(task\)⋅100,non\-damage=avgb∈U1​\{sB​\(b\)−sM​\(b\)<3\}⋅100,\\displaystyle=\\mathop\{\\mathrm\{avg\}\}\_\{\\text\{cells\}\}\\;\\dfrac\{s\_\{M\}\(\\mathrm\{task\}\)\}\{s\_\{F\}\(\\mathrm\{task\}\)\}\\cdot 100,\\quad\\textbf\{non\-damage\}=\\mathop\{\\mathrm\{avg\}\}\_\{b\\in U\}\\,\\mathbf\{1\}\\\{s\_\{B\}\(b\)\-s\_\{M\}\(b\)<3\\\}\\cdot 100,\(7b\)Clean\-up=HM​\(% healed,non\-damage\),Retention=HM​\(% preserved,on\-task ret\.\),\\displaystyle=\\mathrm\{HM\}\(\\text\{\\% healed\},\\,\\text\{non\-damage\}\),\\quad\\textbf\{Retention\}=\\mathrm\{HM\}\(\\text\{\\% preserved\},\\,\\text\{on\-task ret\.\}\),\(7c\)Combined=HM​\(Clean\-up,Retention\)\.\\textbf\{Combined\}=\\mathrm\{HM\}\(\\text\{Clean\-up\},\\,\\text\{Retention\}\)\.\(7d\)Each statistic measures one axis:% healed\(FT damage recovered onDD\),% preserved\(FT gain retained onII\),on\-task ret\.\(on\-task accuracy held vs\. FT\),non\-damage\(unchanged triples kept within33pp of base\), andClean\-up/Retention\(HM of the corresponding pair\)\. “Combined” is the ranking statistic and acts as a triple\-bottleneck harmonic mean: collapse on any sub\-statistic drives it toward0, demoting methods that win one axis by sacrificing another \(V\-SoftMask preserves but does not heal; FAPM heals but does not preserve\)\. Per\-cell scores substitute a mean\-ratio variant; both aggregations and a comparison to flat unified scores\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\]are in App\.[C](https://arxiv.org/html/2605.20296#A3)\.

## 4Experiments

A*cell*is one \(model, task, held\-out benchmark\) triple, and a*\(model, task\) cell*is one of the1414fine\-tuned checkpoints, each expanding into99benchmark cells for a126126\-cell matrix\. The headline finding across this matrix: DG\-Hard sits at the highest balanced point of the recovery\-preservation trade\-off, dominating on the reasoning model and matching the strongest baseline on the non\-reasoning model\. Full per\-method per\-cell scores are in App\.[H](https://arxiv.org/html/2605.20296#A8)\.

### 4\.1Setup at a glance

Two models span the reasoning / non\-reasoning split:Qwen3\.5\-4B\(run in thinking mode with<think\>blocks stripped at scoring\) andLlama\-3\.2\-3B\-Instruct\. Each is fine\-tuned under a uniform full\-parameter SFT configuration on seven tasks \(RTE, StrategyQA, ReClor, BoolQ, MedQA, WikiQA, Winogrande\) and evaluated on nine cross\-domain held\-out benchmarks, partitioned into a 3\-benchmark*Knowledge*cohort \(MMLU, TriviaQA, TruthfulQA\) and a 6\-benchmark*Cognition*cohort \(ARC\-Challenge, GSM8K, IFEval, Math\-500, MNLI, HellaSwag\)\. The nine compared methods fall into three groups: reference pointsBase\(post\-alignment, unfine\-tuned\) andFull\-SFT\(unrepaired fine\-tuned\); training\-time interventionsL1\-reg,V\-SoftMask\(DAS,[14](https://arxiv.org/html/2605.20296#bib.bib26)\),CoFi\-Tune\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\], andLoRA\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]\(rank1616\); and post\-hoc methodsWiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\],FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\], and ourDG\-Hard, the only spectral method in the set; element\-wise merging baselines \(TIES\[[35](https://arxiv.org/html/2605.20296#bib.bib47)\], DARE\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\], Task Arithmetic\[[12](https://arxiv.org/html/2605.20296#bib.bib24)\]\) appear in App\.[I\.3](https://arxiv.org/html/2605.20296#A9.SS3)\. Inference is identical across methods \(vLLM, greedy decoding, nomax\_tokenscap\), with a measured±1\\pm 1pp drift floor and a33pp significance threshold \(≈3×\\approx 3\\timesthe drift floor\) applied throughout\. HuggingFace identifiers, eval splits, hyperparameters, per\-model repair scope, method configurations, and inference details are in App\.[D](https://arxiv.org/html/2605.20296#A4),[H\.1](https://arxiv.org/html/2605.20296#A8.SS1), and[H\.2](https://arxiv.org/html/2605.20296#A8.SS2)\.

### 4\.2Per\-cell forgetting and repair winners

Fine\-tuning damages at least one held\-out benchmark in1313of1414cells, often catastrophically \(Qwen/MedQA’s GSM8K collapses from93\.0%93\.0\\%to1\.1%1\.1\\%\)\. Damage concentrates on a small number of benchmarks per cell rather than spreading uniformly, and the matrix contains5555improved triples against only3030damaged ones, so a cell\-level mean held\-out score smooths the catastrophic drops away and is the wrong instrument to detect forgetting\. The rest of the section therefore evaluates methods at the per\-benchmark level \(per\-cell breakdown in Tab\.[9](https://arxiv.org/html/2605.20296#A5.T9), App\.[E](https://arxiv.org/html/2605.20296#A5)\)\.

A per\-cell balance score,HM\\mathrm\{HM\}of the method’s mean held\-out ratio against base and its on\-task ratio against Full\-SFT \(×100\\times 100\), resolves the per\-cell winners\.DG\-Hard wins66of77Qwen cells \(plus11tie\); WiSE\-FT wins44of77Llama cells \(plus22ties\); on the1414\-cell union, DG\-Hard77, WiSE\-FT44, ties33\.The reasoning\-trained model is more sensitive to the Cognition\-side preservation gap that DG\-Hard’s spectral threshold widens against the linear average \(§[4\.6](https://arxiv.org/html/2605.20296#S4.SS6)\); per\-cell scores and full discussion in Tab\.[10](https://arxiv.org/html/2605.20296#A5.T10)\(App\.[E](https://arxiv.org/html/2605.20296#A5)\)\.

### 4\.3Recovery vs preservation \(population\-level\)

Recovery and preservation are coupled: revertingWftW\_\{\\mathrm\{ft\}\}towardWbaseW\_\{\\mathrm\{base\}\}trivially maximizes recovery but discards the held\-out and on\-task gains fine\-tuning produced\. Tab\.[1](https://arxiv.org/html/2605.20296#S4.T1)reports the population\-level Clean\-up, Retention, and Combined scores \(Eq\.[7](https://arxiv.org/html/2605.20296#S3.E7), §[3\.4](https://arxiv.org/html/2605.20296#S3.SS4)\) for every method\.

Table 1:Population\-level Clean\-up, Retention, and Combined scores per method\. DG\-Hard is the only method scoring≥80\\geq 80on both Clean\-up and Retention; WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\]comes closest but falls to71\.071\.0on Retention\. Every other baseline collapses on at least one axis\.#### Per\-method profiles\.

DG\-Hardis the only method scoring≥80\\geq 80on both axes \(Combined83\.383\.3\)\.FAPM\(91\.9/0\.091\.9/0\.0\) andV\-SoftMask\(24\.9/97\.124\.9/97\.1\) are the one\-axis extremes: FAPM’s90%90\\%sparsity reversion drags improved benchmarks−6\.2%\-6\.2\\%below base, while V\-SoftMask’s gradient damping preserves the FT distribution but does not actively heal\.LoRA\(85\.6/9\.085\.6/9\.0\) cleans up well but has no FT\-side gain to preserve because low\-rank adapters never produced FT’s incidental held\-out gains\. CoFi\-Tune, L1\-reg, and WiSE\-FT are middling on both\.

### 4\.4Cohort breakdown

![Refer to caption](https://arxiv.org/html/2605.20296v1/figures/combined_score_by_cohort.png)Figure 2:Population\-level Combined score per method, sliced by cohort\. Panel titles listn=\(damaged/improved/unchanged\)n=\(\\text\{damaged\}/\\text\{improved\}/\\text\{unchanged\}\)triple counts per cohort\. DG\-Hard tops Overall, Cognition, Reasoning, and Non\-reasoning; L1\-reg edges past DG\-Hard on the small\-nnKnowledge cohort, where its102\.0102\.0Clean\-up reflects % healed overshooting base on the55\-case damaged partition\. The per\-cell balance view in Tab\.[10](https://arxiv.org/html/2605.20296#A5.T10)resolves the Llama cohort into split wins between DG\-Hard and WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\]\. Methods that collapse on either Clean\-up or Retention drop to a low Combined via the harmonic mean’s bottlenecking property\.DG\-Hard wins Combined on four of five cohorts\(Overall, Cognition, Reasoning, Non\-reasoning\) by maintaining strong Clean\-up and strong Retention simultaneously \(Fig\.[2](https://arxiv.org/html/2605.20296#S4.F2); full per\-cohort Clean\-up and Retention in App\.[F](https://arxiv.org/html/2605.20296#A6)\)\. On the small\-nnKnowledge cohort \(55damaged triples\) L1\-reg overshoots base in healing and edges past on Combined; this advantage does not transfer to the larger Cognition cohort, where L1\-reg’s Retention collapses to13\.013\.0and its Combined falls to22\.322\.3\.DG\-Hard’s edge over WiSE\-FT comes mostly from Retention: Clean\-up scores are within∼3\\sim 3pp on most cohorts, but Retention is12\+12\{\+\}pp higher on Overall, Cognition, and Reasoning, driven by held\-out preservation since on\-task retention is essentially flat at∼97%\\sim 97\\%for both\.

### 4\.5Bucketed view

Table 2:Improvement\-bucket % preserved \(%, higher is better\)\. Buckets sum to the5555\(model, task, benchmark\) triples in the126126\-cell matrix\.
Table 3:Damage\-bucket % healed \(%, higher is better\)\. Buckets sum to the3030damaged \(model, task, benchmark\) triples in the126126\-cell matrix\.

![Refer to caption](https://arxiv.org/html/2605.20296v1/figures/pareto_cleanup_retention.png)

![Refer to caption](https://arxiv.org/html/2605.20296v1/figures/pareto_knowledge_cognition.png)

Figure 3:Trade\-off frontiers \(both axes0to100100, higher is better\)\.*Left*: Clean\-up vs Retention\. DG\-Hard sits in the upper\-right region where both axes are simultaneously high; V\-SoftMask\[[14](https://arxiv.org/html/2605.20296#bib.bib26)\]is the retention\-extreme \(top\-left\); FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\]is the cleanup\-extreme \(bottom\-right\)\.*Right*: Knowledge\-cohort Combined \(xx\) vs Cognition\-cohort Combined \(yy\)\. DG\-Hard sits high on both \(84\.884\.8and82\.182\.1\), the most balanced strong method; WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\]matches on Knowledge but loses ground on Cognition \(66\.866\.8\) because its preservation drops there; V\-SoftMask sits in the Knowledge\-favouring region \(67\.767\.7vs30\.930\.9\); CoFi\-Tune\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\], FAPM, and LoRA\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]collapse on Cognition due to non\-positive average preservation\.Bucketing damaged and improved triples by FT\-effect magnitude \(Tabs\.[3](https://arxiv.org/html/2605.20296#S4.T3),[3](https://arxiv.org/html/2605.20296#S4.T3); per\-cohort sub\-score breakdown in App\.[G](https://arxiv.org/html/2605.20296#A7)\) resolves where the DG\-Hard / WiSE\-FT trade\-off is actually decided\.

#### Damage\-recovery is roughly tied\.

Both methods heal most of the damage in every bucket \(Tab\.[3](https://arxiv.org/html/2605.20296#S4.T3)\); the trade\-off is not decided here\.

#### Preservation widens with FT\-gain magnitude\.

The preservation gap grows with the size of the FT lift \(Tab\.[3](https://arxiv.org/html/2605.20296#S4.T3)\): on the largest lifts WiSE\-FT loses about half the gain while DG\-Hard keeps four\-fifths\. This matches the spectral mechanism: DG\-Hard’s threshold retains the high\-singular\-value directions ofΔ\\Deltathat a linear weight average uniformly attenuates, so the more concentrated the FT signal, the wider DG\-Hard’s preservation lead\. The damage side runs the same direction in expectation: the mild bucket holds1919of the3030damaged triples and DG\-Hard wins it, so even where average healing reads as a tie the dominant mode of forgetting falls to DG\-Hard \(Tab\.[3](https://arxiv.org/html/2605.20296#S4.T3)\)\.

### 4\.6Cognition and the trade\-off frontier

The Knowledge cohort \(n=5n=5damaged\) is too easy to differentiate methods on healing: every method except V\-SoftMask recovers at or above base on the damaged Knowledge cases, with several overshooting \(per\-method Clean\-up and Retention per cohort in Tabs\.[11](https://arxiv.org/html/2605.20296#A6.T11),[12](https://arxiv.org/html/2605.20296#A6.T12), App\.[F](https://arxiv.org/html/2605.20296#A6); underlying % healed and % preserved in the corresponding panels of Fig\.[1](https://arxiv.org/html/2605.20296#S1.F1)\)\.

The Cognition cohort \(n=25n=25damaged\) is where the field separates\. Healing remains broadly comparable across non\-pathological baselines, but Retention collapses for every method except DG\-Hard and V\-SoftMask: DG\-Hard’s Cognition Retention of82\.582\.5is roughly2525pp above the next strongest baseline \(WiSE\-FT at57\.557\.5\) and substantially higher than L1\-reg \(13\.013\.0\), while CoFi\-Tune, FAPM, and LoRA collapse to zero because their average % preserved on the Cognition partition is non\-positive \(Tab\.[12](https://arxiv.org/html/2605.20296#A6.T12)\)\.The Cognition preservation gap, is what separates DG\-Hard from the rest of the field and is the most plausible mechanism behind its reasoning\-model lead\.V\-SoftMask preserves Cognition at near\-perfect levels but heals almost nothing on the same cohort, so it does not function as a usable repair for any downstream task that requires reasoning\.

The same separation appears geometrically in Fig\.[3](https://arxiv.org/html/2605.20296#S4.F3)\. Only DG\-Hard sits in the upper\-right of both panels; FAPM and V\-SoftMask anchor opposite corners \(extreme on one axis at the cost of the other\); WiSE\-FT trails DG\-Hard but stays closest among the baselines; L1\-reg, CoFi\-Tune, and LoRA spread along the lower edges \(low Retention on the left, low Cognition Combined on the right\)\. For repair tasks, where neither axis can be allowed to collapse, this rules out the corners and the lower edges alike\.

### 4\.7Safety alignment recovery

Both base models are safety aligned at the Instruct tier, and benign supervised fine\-tuning on knowledge or reasoning data is not designed to touch alignment\. The question is whether the FT updates we study erode safety as an incidental side effect, and whether DG\-Hard, tuned only on capability metrics, restores it without being asked to\. We score each\{Base,FT,DG\-Hard\}\\\{\\text\{Base\},\\text\{FT\},\\text\{DG\-Hard\}\\\}triple on three axes: HarmBench v1\.0\[[22](https://arxiv.org/html/2605.20296#bib.bib31)\]for refusal of harmful prompts, XSTest v2\[[28](https://arxiv.org/html/2605.20296#bib.bib36)\]for over\-refusal of safe prompts, and StrongREJECT v1\.0\[[31](https://arxiv.org/html/2605.20296#bib.bib39)\]for continuous\-rubric harmfulness on forbidden prompts \(Tab\.[4](https://arxiv.org/html/2605.20296#S4.T4)\)\.

Table 4:Per\-cell Base / FT / DG\-Hard scores on three safety axes: HarmBench refusal\[[22](https://arxiv.org/html/2605.20296#bib.bib31)\]\(↑\\uparrow\), XSTest over\-refusal\[[28](https://arxiv.org/html/2605.20296#bib.bib36)\]\(↓\\downarrow\), and StrongREJECT harm\[[31](https://arxiv.org/html/2605.20296#bib.bib39)\]\(↓\\downarrow\)\.Bold= DG\-Hard best among\{Base,FT,DG\}\\\{\\text\{Base\},\\text\{FT\},\\text\{DG\}\\\};underline= second\. Eval details in App\.[D\.3](https://arxiv.org/html/2605.20296#A4.SS3)\.The signal\-plus\-noise model predicts that safety alignment, encoded in pretrained directions Base already occupies, should survive a repair that removes only the IID noise residual\. The data is consistent with this prediction\. On every Llama cell where FT lowered HarmBench refusal or raised StrongREJECT harmfulness, DG\-Hard recovers the majority of the gap toward Base\. No cell shows large degradation, including the Qwen cells where FT incidentally improved a safety axis\.

Qwen\+wikiqa provides a more demanding test\. FT shifts alignment in opposite directions on the same model: refusal of harmful prompts saturates at100%100\\%, while XSTest over\-refusal of safe prompts rises from2\.82\.8to15\.615\.6\. DG\-Hard returns both axes to within small residuals of Base\. That a non\-safety\-specific spectral repair simultaneously corrects both opposite\-direction shifts on the same cell is consistent with the bidirectional damage sharing the noise\-residual structure DG\-Hard targets\.

StrongREJECT’s continuous rubric provides a finer\-grained signal than the binary refusal label: on Llama, non\-refusals are accompanied by greater per\-occurrence severity, and DG\-Hard recovers a substantial fraction of that severity\. The threshold was set without reference to alignment data, so the safety recovery appears to follow from the noise\-residual model the spectral cut is built around, rather than from a designed objective\.

The broader pattern across capability and safety axes is consistent\. DG\-Hard occupies a balanced position between Base and FT: damaged held\-out capabilities recover toward Base, on\-task gains track FT, and safety alignment is preserved as an apparent side effect of the same spectral cut\. The three axes appear to track one mechanism rather than three\.

## 5Limitations and conclusion

#### Limitations\.

DG\-Hard treats the task\-relevant update as concentrated in a small number of high\-singular\-value directions; small\-spectral\-energy components are zeroed regardless of semantic importance, which can fail when the useful signal is spectrally diffuse\. This assumption is consistent with prior work showing that fine\-tuning weight updates are rank\-deficient\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\], that the fine\-tuning objective has low intrinsic dimensionality\[[1](https://arxiv.org/html/2605.20296#bib.bib12)\], and that full\-FT updates carry higher effective rank than LoRA reparameterizations\[[30](https://arxiv.org/html/2605.20296#bib.bib38)\]\(App\.[A](https://arxiv.org/html/2605.20296#A1)\); the failure mode applies to a minority regime but is not ruled out a priori\.

The only operational consideration is the per\-matrix SVD cost,O\(min\(m,n\)2max\(m,n\)\)O\(\\min\(m,n\)^\{2\}\\,\\max\(m,n\)\)\. Even at frontier scale the dominant matrix grows only∼7×\\sim 7\\timesfrom Llama\-3\.2\-3B’s embedding \(128,256×3,072128\{,\}256\\times 3\{,\}072\) to Kimi\-K2\.6’s \(163,840×7,168163\{,\}840\\times 7\{,\}168\)\. At the∼1\\sim 1T\-parameter scale of Kimi\-K2\.6, the remaining mixture\-of\-experts components admit parallel SVD; a full repair completes in hours, a negligible fraction of the weeks of training that produced the checkpoints\.

#### Conclusion\.

Fine\-tuning weight deltas admit a clean two\-component decomposition: a low\-rank task\-aligned update and an IID\-like noise residual that gradient descent has no incentive to remove\. DG\-Hard strips the residual by applying the Donoho\-Gavish optimal hard SVD threshold post hoc, data\-free, and gradient\-free\. Across1414\(model, task\) cells on Qwen3\.5\-4B and Llama\-3\.2\-3B\-Instruct and nine cross\-domain held\-out benchmarks, DG\-Hard achieves the highest balanced point on the recovery\-preservation trade\-off under a partition\-conditional Combined metric, beating the next\-best post\-hoc baseline by\+6\.9\+6\.9pp, and restores safety alignment that benign fine\-tuning incidentally erodes on three independent safety axes without alignment data entering the procedure\. The capability and safety patterns appear to follow from a single mechanism rather than three\. Additional experiments supporting these design choices, including a comparison with element\-wise model\-merging baselines, are in App\.[I](https://arxiv.org/html/2605.20296#A9)\.

## References

- \[1\]\(2021\)Intrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/2012.13255)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px1.p1.7),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2605.20296#S1.p5.1),[§3\.1](https://arxiv.org/html/2605.20296#S3.SS1.p1.9),[§5](https://arxiv.org/html/2605.20296#S5.SS0.SSS0.Px1.p1.1)\.
- \[2\]R\. Aljundi, F\. Babiloni, M\. Elhoseiny, M\. Rohrbach, and T\. Tuytelaars\(2018\)Memory aware synapses: learning what \(not\) to forget\.InProceedings of the European Conference on Computer Vision,External Links:[Link](https://arxiv.org/abs/1711.09601)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2)\.
- \[3\]T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. Amodei\(2020\)Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2005.14165)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1)\.
- \[4\]A\. Chaudhry, M\. Rohrbach, M\. Elhoseiny, T\. Ajanthan, P\. K\. Dokania, P\. H\. S\. Torr, and M\. Ranzato\(2019\)On tiny episodic memories in continual learning\.arXiv preprint arXiv:1902\.10486\.External Links:[Link](https://arxiv.org/abs/1902.10486)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2)\.
- \[5\]J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova\(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,External Links:[Link](https://arxiv.org/abs/1810.04805)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1)\.
- \[6\]R\. M\. French\(1999\)Catastrophic forgetting in connectionist networks\.Trends in Cognitive Sciences3\(4\),pp\. 128–135\.External Links:[Document](https://dx.doi.org/10.1016/S1364-6613%2899%2901294-2)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p1.2),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]M\. Gavish and D\. L\. Donoho\(2014\)The optimal hard threshold for singular values is4/34/\\sqrt\{3\}\.IEEE Transactions on Information Theory60\(8\),pp\. 5040–5053\.External Links:[Link](https://arxiv.org/abs/1305.5870)Cited by:[Appendix B](https://arxiv.org/html/2605.20296#A2.p2.10),[item 2](https://arxiv.org/html/2605.20296#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2605.20296#S1.p6.2),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px5.p1.1),[§3\.2](https://arxiv.org/html/2605.20296#S3.SS2.p1.13),[§3\.3](https://arxiv.org/html/2605.20296#S3.SS3.p1.1),[§3](https://arxiv.org/html/2605.20296#S3.p1.3)\.
- \[8\]M\. Gavish and D\. L\. Donoho\(2017\)Optimal shrinkage of singular values\.IEEE Transactions on Information Theory63\(4\),pp\. 2137–2152\.External Links:[Link](https://arxiv.org/abs/1405.7511)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px5.p1.1)\.
- \[9\]J\. Howard and S\. Ruder\(2018\)Universal language model fine\-tuning for text classification\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/1801.06146)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1)\.
- \[10\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px1.p1.7),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1),[Table 14](https://arxiv.org/html/2605.20296#A8.T14),[Table 15](https://arxiv.org/html/2605.20296#A8.T15),[§1](https://arxiv.org/html/2605.20296#S1.p5.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px3.p1.2),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px5.p1.1),[§3\.1](https://arxiv.org/html/2605.20296#S3.SS1.p1.9),[Figure 3](https://arxiv.org/html/2605.20296#S4.F3),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4),[§5](https://arxiv.org/html/2605.20296#S5.SS0.SSS0.Px1.p1.1)\.
- \[11\]W\. Huang, A\. Cheng, and Y\. Wang\(2025\)Mitigating catastrophic forgetting in large language models with forgetting\-aware pruning \(FAPM\)\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://arxiv.org/abs/2509.08255)Cited by:[Table 10](https://arxiv.org/html/2605.20296#A5.T10),[Table 14](https://arxiv.org/html/2605.20296#A8.T14),[Table 15](https://arxiv.org/html/2605.20296#A8.T15),[Figure 1](https://arxiv.org/html/2605.20296#S1.F1),[§1](https://arxiv.org/html/2605.20296#S1.p4.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px4.p1.7),[Figure 3](https://arxiv.org/html/2605.20296#S4.F3),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4)\.
- \[12\]G\. Ilharco, M\. T\. Ribeiro, M\. Wortsman, S\. Gururangan, L\. Schmidt, H\. Hajishirzi, and A\. Farhadi\(2023\)Editing models with task arithmetic\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2212.04089)Cited by:[§I\.3](https://arxiv.org/html/2605.20296#A9.SS3.p1.12),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px4.p1.7),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4)\.
- \[13\]S\. Jastrzębski, Z\. Kenton, D\. Arpit, N\. Ballas, A\. Fischer, Y\. Bengio, and A\. Storkey\(2017\)Three factors influencing minima in SGD\.arXiv preprint arXiv:1711\.04623\.Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1)\.
- \[14\]Z\. Ke, Y\. Shao, H\. Lin, T\. Konishi, G\. Kim, and B\. Liu\(2023\)Continual pre\-training of language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2302.03241)Cited by:[§H\.1](https://arxiv.org/html/2605.20296#A8.SS1.p1.23),[Table 14](https://arxiv.org/html/2605.20296#A8.T14),[Table 15](https://arxiv.org/html/2605.20296#A8.T15),[Figure 1](https://arxiv.org/html/2605.20296#S1.F1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2),[Figure 3](https://arxiv.org/html/2605.20296#S4.F3),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4)\.
- \[15\]N\. S\. Keskar, D\. Mudigere, J\. Nocedal, M\. Smelyanskiy, and P\. T\. P\. Tang\(2017\)On large\-batch training for deep learning: generalization gap and sharp minima\.InInternational Conference on Learning Representations,External Links:1609\.04836Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1)\.
- \[16\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwińska, D\. Hassabis, C\. Clopath, D\. Kumaran, and R\. Hadsell\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.External Links:[Link](https://arxiv.org/abs/1612.00796)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2)\.
- \[17\]A\. Kumar, A\. Raghunathan, R\. Jones, T\. Ma, and P\. Liang\(2022\)Fine\-tuning can distort pretrained features and underperform out\-of\-distribution\.InInternational Conference on Learning Representations,External Links:2202\.10054Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px1.p1.1)\.
- \[18\]D\. Lopez\-Paz and M\. Ranzato\(2017\)Gradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/1706.08840)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2)\.
- \[19\]Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang\(2023\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.arXiv preprint arXiv:2308\.08747\.External Links:[Link](https://arxiv.org/abs/2308.08747)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]V\. A\. Marchenko and L\. A\. Pastur\(1967\)Distribution of eigenvalues for some sets of random matrices\.Mathematics of the USSR\-Sbornik1\(4\),pp\. 457–483\.Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p5.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px5.p1.1),[§3\.2](https://arxiv.org/html/2605.20296#S3.SS2.p1.4)\.
- \[21\]M\. S\. Matena and C\. A\. Raffel\(2022\)Merging models with fisher\-weighted averaging\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2111.09832)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px4.p1.7)\.
- \[22\]M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li, D\. Forsyth, and D\. Hendrycks\(2024\)HarmBench: a standardized evaluation framework for automated red teaming and robust refusal\.InProceedings of the 41st International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2402.04249)Cited by:[§D\.3](https://arxiv.org/html/2605.20296#A4.SS3.p2.3),[§4\.7](https://arxiv.org/html/2605.20296#S4.SS7.p1.1),[Table 4](https://arxiv.org/html/2605.20296#S4.T4)\.
- \[23\]M\. McCloskey and N\. J\. Cohen\(1989\)Catastrophic interference in connectionist networks: the sequential learning problem\.InPsychology of Learning and Motivation,Vol\.24,pp\. 109–165\.Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p1.2),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px1.p1.1)\.
- \[24\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1)\.
- \[25\]A\. Panigrahi, N\. Saunshi, H\. Zhao, and S\. Arora\(2023\)Task\-specific skill localization in fine\-tuned language models\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2302.06600)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px1.p1.7),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1)\.
- \[26\]X\. Qi, Y\. Zeng, T\. Xie, P\. Chen, R\. Jia, P\. Mittal, and P\. Henderson\(2024\)Fine\-tuning aligned language models compromises safety, even when users do not intend to\!\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2310.03693)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p2.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px1.p1.1)\.
- \[27\]R\. Ratcliff\(1990\)Connectionist models of recognition memory: constraints imposed by learning and forgetting functions\.Psychological Review97\(2\),pp\. 285–308\.External Links:[Document](https://dx.doi.org/10.1037/0033-295X.97.2.285)Cited by:[§1](https://arxiv.org/html/2605.20296#S1.p1.2),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px1.p1.1)\.
- \[28\]P\. Röttger, H\. R\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. Hovy\(2024\)XSTest: a test suite for identifying exaggerated safety behaviours in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/2308.01263)Cited by:[§D\.3](https://arxiv.org/html/2605.20296#A4.SS3.p2.3),[Table 7](https://arxiv.org/html/2605.20296#A4.T7.6.2.4.2.4),[§4\.7](https://arxiv.org/html/2605.20296#S4.SS7.p1.1),[Table 4](https://arxiv.org/html/2605.20296#S4.T4)\.
- \[29\]P\. Sharma, J\. T\. Ash, and D\. Misra\(2024\)The truth is in there: improving reasoning in language models with layer\-selective rank reduction\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2312.13558)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px2.p1.3),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1)\.
- \[30\]R\. Shuttleworth, J\. Andreas, A\. Torralba, and P\. Sharma\(2025\)LoRA vs full fine\-tuning: an illusion of equivalence\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2410.21228)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px1.p1.7),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px3.p1.2),[§5](https://arxiv.org/html/2605.20296#S5.SS0.SSS0.Px1.p1.1)\.
- \[31\]A\. Souly, Q\. Lu, D\. Bowen, T\. Trinh, E\. Hsieh, S\. Pandey, P\. Abbeel, J\. Svegliato, S\. Emmons, O\. Watkins, and S\. Toyer\(2024\)A StrongREJECT for empty jailbreaks\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),External Links:[Link](https://arxiv.org/abs/2402.10260)Cited by:[Table 7](https://arxiv.org/html/2605.20296#A4.T7.6.2.2.5),[§4\.7](https://arxiv.org/html/2605.20296#S4.SS7.p1.1),[Table 4](https://arxiv.org/html/2605.20296#S4.T4)\.
- \[32\]M\. Staats, M\. Thamm, and B\. Rosenow\(2023\)Boundary between noise and information applied to filtering neural network weight matrices\.Physical Review E108,pp\. L022302\.External Links:[Link](https://arxiv.org/abs/2206.03927)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px2.p1.3),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2605.20296#S1.p5.1),[§3\.1](https://arxiv.org/html/2605.20296#S3.SS1.p1.9)\.
- \[33\]M\. Thamm, M\. Staats, and B\. Rosenow\(2022\)Random matrix analysis of deep neural network weight matrices\.Physical Review E106,pp\. 054124\.External Links:[Link](https://arxiv.org/abs/2203.14661)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px2.p1.3),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2605.20296#S1.p5.1),[§3\.1](https://arxiv.org/html/2605.20296#S3.SS1.p1.9)\.
- \[34\]M\. Wortsman, G\. Ilharco, J\. W\. Kim, M\. Y\. Li, S\. Kornblith, R\. Roelofs, R\. G\. Lopes, H\. Hajishirzi, A\. Farhadi, H\. Namkoong, and L\. Schmidt\(2022\)Robust fine\-tuning of zero\-shot models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,External Links:[Link](https://arxiv.org/abs/2109.01903)Cited by:[Table 13](https://arxiv.org/html/2605.20296#A7.T13),[§H\.1](https://arxiv.org/html/2605.20296#A8.SS1.p1.23),[Table 14](https://arxiv.org/html/2605.20296#A8.T14),[Table 15](https://arxiv.org/html/2605.20296#A8.T15),[§I\.3](https://arxiv.org/html/2605.20296#A9.SS3.p1.12),[§1](https://arxiv.org/html/2605.20296#S1.p4.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px4.p1.7),[Figure 2](https://arxiv.org/html/2605.20296#S4.F2),[Figure 3](https://arxiv.org/html/2605.20296#S4.F3),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4),[Table 1](https://arxiv.org/html/2605.20296#S4.T1)\.
- \[35\]P\. Yadav, D\. Tam, L\. Choshen, C\. Raffel, and M\. Bansal\(2023\)TIES\-merging: resolving interference when merging models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2306.01708)Cited by:[§H\.1](https://arxiv.org/html/2605.20296#A8.SS1.p1.23),[§I\.3](https://arxiv.org/html/2605.20296#A9.SS3.p1.12),[§1](https://arxiv.org/html/2605.20296#S1.p4.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px4.p1.7),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4)\.
- \[36\]L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. Li\(2024\)Language models are super mario: absorbing abilities from homologous models as a free lunch\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2311.03099)Cited by:[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px2.p1.3),[Appendix A](https://arxiv.org/html/2605.20296#A1.SS0.SSS0.Px4.p1.1),[§H\.1](https://arxiv.org/html/2605.20296#A8.SS1.p1.23),[§I\.3](https://arxiv.org/html/2605.20296#A9.SS3.p1.12),[Table 20](https://arxiv.org/html/2605.20296#A9.T20),[Table 21](https://arxiv.org/html/2605.20296#A9.T21),[§1](https://arxiv.org/html/2605.20296#S1.p4.1),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px4.p1.7),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4)\.
- \[37\]F\. Zenke, B\. Poole, and S\. Ganguli\(2017\)Continual learning through synaptic intelligence\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/1703.04200)Cited by:[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2)\.
- \[38\]H\. Zhang, Y\. Wu, D\. Li, S\. Yang, R\. Zhao, Y\. Jiang, and F\. Tan\(2024\)Balancing speciality and versatility: a coarse to fine framework for mitigating catastrophic forgetting in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://arxiv.org/abs/2404.10306)Cited by:[Appendix C](https://arxiv.org/html/2605.20296#A3.SS0.SSS0.Px6.p1.1),[§H\.1](https://arxiv.org/html/2605.20296#A8.SS1.p1.23),[Table 14](https://arxiv.org/html/2605.20296#A8.T14),[Table 15](https://arxiv.org/html/2605.20296#A8.T15),[§2](https://arxiv.org/html/2605.20296#S2.SS0.SSS0.Px2.p1.2),[§3\.4](https://arxiv.org/html/2605.20296#S3.SS4.SSS0.Px1.p1.12),[Figure 3](https://arxiv.org/html/2605.20296#S4.F3),[§4\.1](https://arxiv.org/html/2605.20296#S4.SS1.p1.4)\.

## Appendix

Contents

A[Empirical evidence for the signal\-plus\-noise structure ofΔ\\Delta](https://arxiv.org/html/2605.20296#A1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[A](https://arxiv.org/html/2605.20296#A1) B[Noise\-scale derivation](https://arxiv.org/html/2605.20296#A2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[B](https://arxiv.org/html/2605.20296#A2) C[Metric discussion](https://arxiv.org/html/2605.20296#A3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[C](https://arxiv.org/html/2605.20296#A3) D[Detailed experimental setup](https://arxiv.org/html/2605.20296#A4)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D](https://arxiv.org/html/2605.20296#A4) D\.1[Fine\-tuning tasks](https://arxiv.org/html/2605.20296#A4.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.1](https://arxiv.org/html/2605.20296#A4.SS1) D\.2[Held\-out benchmarks](https://arxiv.org/html/2605.20296#A4.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.2](https://arxiv.org/html/2605.20296#A4.SS2) D\.3[Training hyperparameters](https://arxiv.org/html/2605.20296#A4.SS4)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.4](https://arxiv.org/html/2605.20296#A4.SS4) D\.4[Repair scope](https://arxiv.org/html/2605.20296#A4.SS5)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[D\.5](https://arxiv.org/html/2605.20296#A4.SS5) E[Per\-cell forgetting and balance scores](https://arxiv.org/html/2605.20296#A5)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[E](https://arxiv.org/html/2605.20296#A5) F[Per\-cohort Clean\-up and Retention](https://arxiv.org/html/2605.20296#A6)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[F](https://arxiv.org/html/2605.20296#A6) G[Full per\-cohort sub\-score breakdown](https://arxiv.org/html/2605.20296#A7)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[G](https://arxiv.org/html/2605.20296#A7) H[Per\-\(model, task, method, benchmark\) results](https://arxiv.org/html/2605.20296#A8)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[H](https://arxiv.org/html/2605.20296#A8) H\.1[Method\-specific configurations](https://arxiv.org/html/2605.20296#A8.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[H\.1](https://arxiv.org/html/2605.20296#A8.SS1) H\.2[Inference engine and decoding](https://arxiv.org/html/2605.20296#A8.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[H\.2](https://arxiv.org/html/2605.20296#A8.SS2) I[Other experiments and ablations](https://arxiv.org/html/2605.20296#A9)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[I](https://arxiv.org/html/2605.20296#A9) I\.1[Layer\-wise noise concentration](https://arxiv.org/html/2605.20296#A9.SS1)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[I\.1](https://arxiv.org/html/2605.20296#A9.SS1) I\.2[Layer\-mask causal test](https://arxiv.org/html/2605.20296#A9.SS2)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[I\.2](https://arxiv.org/html/2605.20296#A9.SS2) I\.3[Comparison with element\-wise merging baselines](https://arxiv.org/html/2605.20296#A9.SS3)\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.\.[I\.3](https://arxiv.org/html/2605.20296#A9.SS3)

## Appendix AEmpirical evidence for the signal\-plus\-noise structure ofΔ\\Delta

The central modeling assumption of selective spectral reversion is that, for each22D weight matrixWWin the network, the fine\-tuning deltaΔ=Wft−Wbase\\Delta=W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}\(equivalentlyΔ​W\\Delta Win much of the prior LoRA\-adjacent literature\) admits a two\-component decompositionΔ=Δsignal\+Δnoise\\Delta=\\Delta\_\{\\mathrm\{signal\}\}\+\\Delta\_\{\\mathrm\{noise\}\}, whereΔsignal\\Delta\_\{\\mathrm\{signal\}\}is a low\-rank task\-aligned update andΔnoise\\Delta\_\{\\mathrm\{noise\}\}is a residual that behaves statistically like an IID random matrix\. The phenomenological evidence the model is designed to explain is the per\-cell forgetting pattern in Tab\.[9](https://arxiv.org/html/2605.20296#A5.T9): every fine\-tune produces a small set of large per\-benchmark drops on a small set of held\-out benchmarks \(concentrated signal\-side damage\), while the remaining held\-out scores drift only mildly \(diffuse noise\-side damage\)\. Two independent threads of empirical evidence, one from the fine\-tuning literature and one from the random\-matrix\-theory literature, support this picture, and a direct check on our own checkpoints \(Fig\.[4](https://arxiv.org/html/2605.20296#A1.F4)\) completes the argument\.

![Refer to caption](https://arxiv.org/html/2605.20296v1/figures/spectral_unforgetting_two_views.png)Figure 4:Spectral unforgetting in two views, on Llama\-3\.2\-3Bmlp\.up\_projat layer1414, withΔ=Wft−Wbase∈ℝ8192×3072\\Delta=W\_\{\\mathrm\{ft\}\}\-W\_\{\\mathrm\{base\}\}\\in\\mathbb\{R\}^\{8192\\times 3072\}\(β=0\.375\\beta=0\.375\)\.\(a\)The fine\-tune delta has a sharp spectral cliff:3030singular values \(red\) lie above the DG\-Hard thresholdτ∗=ω​\(β\)​σ^\\tau^\{\\ast\}=\\omega\(\\beta\)\\hat\{\\sigma\}, while the remaining30423042\(gray\) sit at or below the Marchenko\-Pastur bulk edgeλMP\\lambda\_\{\\text\{MP\}\}\.Top row, \(b\) to \(d\): entry\-space viewof the additive identityΔFT=Δ∗\+\(Wft−W∗\)\\Delta\_\{\\mathrm\{FT\}\}=\\Delta^\{\\ast\}\+\(W\_\{\\mathrm\{ft\}\}\-W^\{\\ast\}\), with each pixel showingmax⁡\|Δi​j\|\\max\|\\Delta\_\{ij\}\|over a block of matrix entries\. The full FT delta \(b\) is uniform speckle; the rank\-3030repaired delta \(c\) reveals horizontal banding from the kept left\-singular vectorsuru\_\{r\}; the discarded component \(d\) is again uniform speckle, carrying no spatial structure\.Bottom row, \(e\) to \(g\): spectrum\-space viewof the same identity, with each pixel showingmax⁡σr​\|ur​\[i\]\|\\max\\sigma\_\{r\}\|u\_\{r\}\[i\]\|, the contribution of singular directionrrto output neuronii\. Panel \(e\) decomposes into the kept\-only panel \(f\), nonzero only in its leftmost3030columns, and the bulk\-only panel \(g\), nonzero everywhere except those columns; their pixel\-wise sum reproduces \(e\) exactly, because every singular direction belongs to exactly one set\. Random\-matrix\-fit chips above the top row report the percentage match between each matrix’s singular\-value distribution and the Marchenko\-Pastur prediction \(green: structured, non\-random; red: IID\-noise\-like\):ΔFT\\Delta\_\{\\mathrm\{FT\}\}scores93\.1%93\.1\\%, the rank\-3030repaired deltaΔ∗\\Delta^\{\\ast\}scores1\.0%1\.0\\%, and the discarded component scores94\.0%94\.0\\%, slightly more MP\-like thanΔFT\\Delta\_\{\\mathrm\{FT\}\}itself, confirming that DG\-Hard separates the two without leaving residual signal in the noise\.#### The signal side: the task\-aligned update is low\-dimensional\.

Aghajanyanet al\.\[[1](https://arxiv.org/html/2605.20296#bib.bib12)\]establish low intrinsic dimensionality for the fine\-tuning objective by showing that optimizing only∼200\\sim 200trainable parameters via random projection into the full parameter space recovers90%90\\%of full\-fine\-tune performance on MRPC with RoBERTa; this is a property of the optimization landscape rather than of the realized weight delta\.Huet al\.\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]provide the more direct evidence: in their intrinsic\-rank analysis, very low LoRA ranks \(often single\-digit\) match high\-rank LoRA on downstream tasks, and the learnedΔ​W\\Delta Wamplifies a small set of task\-specific directions, supporting their explicit “rank\-deficiency ofΔ​W\\Delta W” conclusion\.Shuttleworthet al\.\[[30](https://arxiv.org/html/2605.20296#bib.bib38)\]report that full fine\-tuning carries a higher effective rank than LoRA reparameterizations even at matched nominal rank, with full\-FT deltas modifying the existing pretrained singular structure and LoRA introducing a small number of additional “intruder” directions approximately orthogonal to that structure; the spike\-versus\-bulk reading of our signal\-plus\-noise model survives both regimes because it isolates the structured component above the MP bulk rather than committing to a strict low\-rank cap\.Panigrahiet al\.\[[25](https://arxiv.org/html/2605.20296#bib.bib33)\]provide a complementary parametric\-sparsity result:∼0\.01%\\sim 0\.01\\%of parameters carry\>95%\>95\\%of fine\-tune task performance when grafted back onto the base model\. Sparsity in coordinate space and concentration in singular\-value space are distinct mathematical properties; we cite this work as a parallel structural prior, not as direct evidence for low\-rankΔ\\Delta\.

#### The noise side: the residual past the spikes follows the Marchenko\-Pastur bulk\.

Random\-matrix theory is an empirically validated description of trained neural\-network weight matrices, not just an asymptotic abstraction\.Thammet al\.\[[33](https://arxiv.org/html/2605.20296#bib.bib41)\]analyze the singular spectra of trained deep\-network weight matrices and demonstrate that the bulk fits the MP density layer by layer, with a finite number of outliers carrying the learned signal\.Staatset al\.\[[32](https://arxiv.org/html/2605.20296#bib.bib40)\]operationalize the same picture: they apply MP\-edge filtering directly to neural\-network weight matrices, set the sub\-edge bulk to zero, and recover the trained network’s behavior, providing direct evidence that the bulk is functionally inert\. Two independent fine\-tuning\-side observations point in the same direction\.Sharmaet al\.\[[29](https://arxiv.org/html/2605.20296#bib.bib37)\]\(LASER\) show that aggressively zeroing the high\-rank tail of a trained transformer’s weight matrix can improve downstream performance, suggesting the high\-rank bulk encodes noise rather than usable signal, a result that transfers to the fine\-tune delta by linearity of SVD truncation\.Yuet al\.\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\]\(DARE\) show that the fine\-tune delta tolerates random pruning of9090–99%99\\%of its entries with rescaling, attributing this to “extreme redundancy” of small\-magnitude updates, consistent with most ofΔ\\Deltabeing redundant rather than informative\.

#### Direct check on our own checkpoints\.

Fig\.[4](https://arxiv.org/html/2605.20296#A1.F4)verifies the structure layer\-locally on Llama\-3\.2\-3B’smlp\.up\_projat layer1414\. Panel \(a\) shows the spectral cliff at the DG\-Hard thresholdτ∗\\tau^\{\\ast\}\. Panels \(b\) to \(g\) show that the additive identityΔFT=Δ∗\+\(Wft−W∗\)\\Delta\_\{\\mathrm\{FT\}\}=\\Delta^\{\\ast\}\+\(W\_\{\\mathrm\{ft\}\}\-W^\{\\ast\}\)holds simultaneously in entry space \(top row\) and in singular\-vector space \(bottom row\)\. The random\-matrix\-fit chips on the top row verify the decomposition quantitatively\. We define

MP​\-​fit​\(𝐬\)=\(1−KS​\(F𝐬,FMP​\(σ^,β\)\)\)⋅100,\\mathrm\{MP\\text\{\-\}fit\}\(\\mathbf\{s\}\)\\;=\\;\\bigl\(1\-\\mathrm\{KS\}\(F\_\{\\mathbf\{s\}\},\\,F\_\{\\mathrm\{MP\}\}\(\\hat\{\\sigma\},\\beta\)\)\\bigr\)\\cdot 100,\(8\)whereF𝐬F\_\{\\mathbf\{s\}\}is the empirical CDF of the matrix’s singular values,FMPF\_\{\\mathrm\{MP\}\}is the closed\-form Marchenko\-Pastur CDF at the noise\-scale estimateσ^\\hat\{\\sigma\}shared across the three matrices and the aspect ratioβ\\beta, andKS\\mathrm\{KS\}is the Kolmogorov\-Smirnov distance between them;MP​\-​fit=100\\mathrm\{MP\\text\{\-\}fit\}=100indicates that the empirical and theoretical CDFs coincide \(𝐬\\mathbf\{s\}is statistically indistinguishable from IID noise\), whileMP​\-​fit→0\\mathrm\{MP\\text\{\-\}fit\}\\to 0indicates a maximally non\-random spectrum\. Under this metric,ΔFT\\Delta\_\{\\mathrm\{FT\}\}scores93\.1%93\.1\\%, the rank\-3030repaired deltaΔ∗\\Delta^\{\\ast\}scores1\.0%1\.0\\%\(explicitly non\-random\), and the discarded component scores94\.0%94\.0\\%, slightly more MP\-like thanΔFT\\Delta\_\{\\mathrm\{FT\}\}itself, confirming that DG\-Hard extracts the structured component without leaving residual signal in the noise\.

#### Where this lands the central assumption\.

The signal\-side and noise\-side evidence jointly support the decomposition: the weight update is rank\-deficient\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]and the fine\-tuning objective has low intrinsic dimensionality\[[1](https://arxiv.org/html/2605.20296#bib.bib12)\], with related findings on the spectral structure of full\-FT versus LoRA deltas\[[30](https://arxiv.org/html/2605.20296#bib.bib38)\]and on parametric sparsity\[[25](https://arxiv.org/html/2605.20296#bib.bib33)\]; the residual is empirically MP\-bulk\-distributed\[[33](https://arxiv.org/html/2605.20296#bib.bib41),[32](https://arxiv.org/html/2605.20296#bib.bib40)\], and that bulk is functionally redundant under direct manipulation of trained weights and fine\-tune deltas\[[29](https://arxiv.org/html/2605.20296#bib.bib37),[36](https://arxiv.org/html/2605.20296#bib.bib48)\]\. Fig\.[4](https://arxiv.org/html/2605.20296#A1.F4)checks the bulk\-fit and additivity claims on the exactΔ\\Deltamatrix the method operates on\.

## Appendix BNoise\-scale derivation

§[3\.3](https://arxiv.org/html/2605.20296#S3.SS3)of the main paper introduces the Donoho\-Gavish noise estimator\. This appendix gives the full derivation and the implementation footprint\.

Equation \([4](https://arxiv.org/html/2605.20296#S3.E4)\) requires the noise scaleσ\\sigmaof the residualΔnoise\\Delta\_\{\\mathrm\{noise\}\}in \([2](https://arxiv.org/html/2605.20296#S3.E2)\)\. Ideally we would estimateσ\\sigmafromΔnoise\\Delta\_\{\\mathrm\{noise\}\}directly, but the decomposition in \([2](https://arxiv.org/html/2605.20296#S3.E2)\) is unobservable\.Gavish and Donoho \[[7](https://arxiv.org/html/2605.20296#bib.bib17)\]resolve this within the same MP framework\. Under the spike model, only the toprrsingular values carry signal contamination, withrrunknown but small relative topp; the remainingp−rp\-rare asymptotically distributed as the singular\-value MP density at aspectβ\\beta\. That density has a closed\-form medianμβ\\mu\_\{\\beta\},

μβ=median⁡\(fβ\),fβ​\(λ\)=\(λ\+−λ\)​\(λ−λ−\)2​π​β​λ,λ±=\(1±β\)2,\\mu\_\{\\beta\}\\;=\\;\\sqrt\{\\operatorname\{median\}\(f\_\{\\beta\}\)\},\\qquad f\_\{\\beta\}\(\\lambda\)\\;=\\;\\frac\{\\sqrt\{\(\\lambda\_\{\+\}\-\\lambda\)\(\\lambda\-\\lambda\_\{\-\}\)\}\}\{2\\pi\\,\\beta\\,\\lambda\},\\quad\\lambda\_\{\\pm\}=\(1\\pm\\sqrt\{\\beta\}\)^\{2\},\(9\)which we precompute by fine\-grid trapezoidal integration once per distinct aspect ratio in the network\. Becauser≪pr\\ll p, the median of the*empirical*singular valuesmedian⁡\(𝐬\)\\operatorname\{median\}\(\\mathbf\{s\}\)falls inside the noise bulk regardless of the precise value ofrr, and asymptotically tracksσ​max⁡\(m,n\)⋅μβ\\sigma\\sqrt\{\\max\(m,n\)\}\\cdot\\mu\_\{\\beta\}\. Solving forσ\\sigmagives the Donoho\-Gavish noise estimator,

σ^=median⁡\(𝐬\)μβ​max⁡\(m,n\)\.\\hat\{\\sigma\}\\;=\\;\\frac\{\\operatorname\{median\}\(\\mathbf\{s\}\)\}\{\\mu\_\{\\beta\}\\,\\sqrt\{\\max\(m,n\)\}\}\.\(10\)Equation \([10](https://arxiv.org/html/2605.20296#A2.E10)\) is internally consistent with the rest of the DG framework: the same MP density that determines the bulk edge \([3](https://arxiv.org/html/2605.20296#S3.E3)\) and the optimal threshold \([4](https://arxiv.org/html/2605.20296#S3.E4)\) also determinesμβ\\mu\_\{\\beta\}\. Substituting \([10](https://arxiv.org/html/2605.20296#A2.E10)\) into \([4](https://arxiv.org/html/2605.20296#S3.E4)\) eliminatesmax⁡\(m,n\)\\sqrt\{\\max\(m,n\)\}and yields the unconditional form of the DG hard threshold,

τ∗=ω​\(β\)μβ​median⁡\(𝐬\),\\tau^\{\*\}\\;=\\;\\frac\{\\omega\(\\beta\)\}\{\\mu\_\{\\beta\}\}\\,\\operatorname\{median\}\(\\mathbf\{s\}\),\(11\)which evaluates to≈2\.858⋅median⁡\(𝐬\)\\approx 2\.858\\cdot\\operatorname\{median\}\(\\mathbf\{s\}\)for square matrices and is computable directly from the empirical spectrum without any auxiliary statistic\.

#### Cost and implementation footprint\.

The per\-matrix dominant cost is the SVD on line[6](https://arxiv.org/html/2605.20296#alg1.l6)’s preceding step,O\(min\(m,n\)2max\(m,n\)\)O\(\\min\(m,n\)^\{2\}\\,\\max\(m,n\)\), completing in a few hundred milliseconds for the largest matrix in the model sizes we evaluate \(the Llama\-3\.2\-3B\-Instruct embedding,128,256×3,072128\{,\}256\\times 3\{,\}072\); full\-network repair completes in≤3\\leq 3minutes on a single A100100\-8080GB for the model sizes we evaluate\. The implementation loadsWbaseW\_\{\\mathrm\{base\}\}andWftW\_\{\\mathrm\{ft\}\}once, processes Algorithm[1](https://arxiv.org/html/2605.20296#alg1)tensor\-by\-tensor, and stitches repaired matrices back at their original keys\. The repair scope is restricted to parameters withndim≥2\\mathrm\{ndim\}\\geq 2and at least10241024elements; tensors withndim\>2\\mathrm\{ndim\}\>2\(e\.g\.,11D convolution kernels\) are reshaped to22D before SVD and reshaped back afterwards\.11D parameters \(biases, normalization scales\) are left at their fine\-tuned values since they admit no non\-trivial MP edge\. Peak GPU memory is approximately twice the model’s parameter footprint plus a working buffer for the batched SVD, well within a4040GB A100100\.

## Appendix CMetric discussion

#### % healed\.

Conventional held\-out accuracy averages across all benchmarks regardless of whether fine\-tuning damaged them, so a method that simply mirrors FT scores indistinguishably from one that actively repairs\. % healed restricts the average to the FT\-damaged partitionDDand reports how much of the FT\-to\-Base gap each method closes\. The numerator is the repaired score’s gain over FT; the denominator is the gap fine\-tuning opened, so0means no recovery and100100means full restoration to Base\.

#### % preserved\.

The analogous construction on the FT\-improved partitionII\. A flat held\-out average pools FT\-improved triples with FT\-unchanged ones, so a method that erases every incidental gain looks numerically similar to one that keeps them whenever most cells were FT\-untouched\. % preserved isolates the triples where fine\-tuning produced a real lift and reports the share of that lift the repaired model still carries\.

#### Non\-damage\.

Repairs that aggressively shrink the delta can introduce regressions on benchmarks fine\-tuning never touched\. A flat held\-out mean smooths these regressions into the average; non\-damage surfaces them\. The statistic is the fraction of FT\-unchanged triples on which the repaired score is within33pp of Base, so a method that bleeds outside the FT\-damaged cells loses points here independently of how well it heals\.

#### On\-task retention\.

Absolute on\-task accuracy is misleading: tasks vary in difficulty, and a flat report rewards methods that simply leave the FT model unchanged\. On\-task ret\. normalizes the method’s target\-task score by FT’s, so100100means every fine\-tuning gain on the target task is preserved and0means none is\.

#### Clean\-up and Retention\.

Clean\-up=HM​\(% healed,non\-damage\)=\\mathrm\{HM\}\(\\text\{\\% healed\},\\,\\text\{non\-damage\}\)summarizes the cleaning side: actively reverting FT\-damage on the FT\-damaged setDDand leaving the FT\-unchanged setUUalone\. Retention=HM​\(% preserved,on\-task ret\.\)=\\mathrm\{HM\}\(\\text\{\\% preserved\},\\,\\text\{on\-task ret\.\}\)summarizes the preservation side: keeping incidental held\-out gains on the FT\-improved setIIand the target\-task gain\. Each is a harmonic mean by design, so a method that maximizes one sub\-statistic by collapsing the other is bottlenecked rather than rewarded; Combined applies the same bottleneck across the two sides\.

#### Why not a flat unified average?

A common alternative aggregation in the continual\-learning literature, the Uni\. score ofZhanget al\.\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\], computes a flat average of held\-out scores and combines it with on\-task accuracy\. Its held\-out term is blind to what fine\-tuning did to each benchmark, so a method that actively pulls damaged benchmarks back toward base and a method that simply avoids perturbing the FT distribution can post numerically similar Uni\. scores even though only the first is doing the work the metric implicitly claims to measure\. Our scoring conditions every held\-out measurement on its FT classification \(\([6](https://arxiv.org/html/2605.20296#S3.E6)\)\): % healed is averaged only over damaged cases, % preserved only over improved ones, so*recovered\-the\-damage*and*never\-touched\-it*become two distinct statistics rather than two paths to the same number\. Combined with the harmonic\-mean bottleneck of §[3\.4](https://arxiv.org/html/2605.20296#S3.SS4), this gives the metric a guarantee an unconditional flat sum cannot match\.

#### Per\-cell aggregation\.

The Combined score in \([7d](https://arxiv.org/html/2605.20296#S3.E7.4)\) is for population\-level \(cohort\) comparisons\. Per\-cell scores in Tab\.[10](https://arxiv.org/html/2605.20296#A5.T10)of the main paper use a separate aggregation: the harmonic mean of\(s¯methodheld/s¯baseheld\)×100\(\\bar\{s\}\_\{\\mathrm\{method\}\}^\{\\mathrm\{held\}\}/\\bar\{s\}\_\{\\mathrm\{base\}\}^\{\\mathrm\{held\}\}\)\\times 100and\(smethodtask/sFTtask\)×100\(s\_\{\\mathrm\{method\}\}^\{\\mathrm\{task\}\}/s\_\{\\mathrm\{FT\}\}^\{\\mathrm\{task\}\}\)\\times 100, wheres¯held\\bar\{s\}^\{\\mathrm\{held\}\}denotes the mean held\-out score across the cell’s nine held\-out benchmarks\. Each ratio multiplies by100100so that values cleanly exceed100100when a method outperforms its reference \(e\.g\., DG\-Hard scores108\.5108\.5on Qwen3\.5\-4B \+ BoolQ\)\. Per\-cell mean ratios are used here in place of the continuous % healed / % preserved statistics because, onn≈9n\\approx 9benchmarks per cell, the binary damaged / improved / unchanged classification produces small\-sample sub\-statistics that are too noisy to read cell\-by\-cell\.

## Appendix DDetailed experimental setup

This appendix documents the reproducibility\-level configuration that §[4](https://arxiv.org/html/2605.20296#S4)of the main paper summarizes in prose\.

### D\.1Fine\-tuning tasks

Tab\.[5](https://arxiv.org/html/2605.20296#A4.T5)lists the seven fine\-tuning tasks forming the row axis of the experimental matrix, with full HuggingFace dataset identifiers and eval split sizes\. RTE, StrategyQA, ReClor, MedQA, and Winogrande use their full eval splits; BoolQ and WikiQA are subsetted to15001500items \(out of32703270and61656165respectively\) to keep the14×914\\times 9matrix wall\-clock tractable, with per\-method standard\-error well below11pp at this sample size\.

Table 5:The seven fine\-tuning tasks forming the row axis of the experimental matrix\. The eval split of each task becomes that cell’s on\-task benchmarktask\_\{name\}throughout the analysis\.
### D\.2Held\-out benchmarks

Tab\.[6](https://arxiv.org/html/2605.20296#A4.T6)lists the nine held\-out cross\-domain benchmarks evaluated identically across every \(model, task, method\) combination, together with their HuggingFace dataset identifiers, splits, sample sizes, shot counts, scoring metrics, and Knowledge / Cognition cohort assignment\. Five benchmarks \(ARC\-Challenge, GSM8K, IFEval, Math\-500, TruthfulQA\) use their full eval splits; HellaSwag, TriviaQA, and MNLI are subsetted as noted\. MMLU uses a stratified subset of3030questions per subject across all5757subjects, totalling17101710items, so each subject contributes equal weight regardless of its original size\.

Table 6:The nine held\-out cross\-domain benchmarks evaluated identically across every \(model, task, method\) combination\. The IFEval metric is the average of prompt×\\timesinstruction strict×\\timesloose accuracy\.
### D\.3Safety benchmarks and judges

The safety axes reported in Tab\.[4](https://arxiv.org/html/2605.20296#S4.T4)of the main paper are evaluated identically across every \(model, task, method\) cell, with no alignment data entering the repair procedure\. Generation uses the same policy as all other benchmarks: vLLM with nomax\_tokenscap \(capping would bias refusal length on either side\), greedy decoding, and the model’s default chat template; the prompt set and judge for each axis are summarized in Tab\.[7](https://arxiv.org/html/2605.20296#A4.T7)\.

Table 7:Safety benchmark configurations\. HarmBench refusal is reported as the share of harmful prompts the model declines \(higher is safer\)\. XSTest over\-refusal is reported on the safe half of v2 only \(the “looks unsafe but isn’t” subset\); lower is better\. StrongREJECT reports the rubric harmfulness score on a0\-11scale aggregated as the official convex combination of refusal, specificity, and convincingness; lower is better\.The HarmBench classifier follows the protocol ofMazeikaet al\.\[[22](https://arxiv.org/html/2605.20296#bib.bib31)\]: each \(prompt, response\) pair is classified as “refusal” or “harmful compliance” and the reported score is the refusal share\. XSTest uses the safe half of v2 \(250250prompts the model should comply with despite surface cues\); the judge follows the three\-class protocol ofRöttgeret al\.\[[28](https://arxiv.org/html/2605.20296#bib.bib36)\]and we count any non\-full\-compliance \(partial or refusal\) as over\-refusal\. StrongREJECT applies its0\-11rubric verbatim to every \(prompt, response\) pair; harmfulness is averaged over prompts\. No category\- or severity\-level reweighting is applied; all benchmarks use unweighted means over their prompt sets\.

### D\.4Training hyperparameters

Tab\.[8](https://arxiv.org/html/2605.20296#A4.T8)lists the uniform training configuration used for every \(model, task\) cell, so any difference in repair outcome is attributable to the repair method rather than to training\-time variance\.

Table 8:Uniform training hyperparameters used for every \(model, task\) cell\.
### D\.5Repair scope

The post\-hoc methods we evaluate \(DG\-Hard, WiSE\-FT, FAPM; with TIES and DARE\-Linear additionally evaluated in the App\.[I\.3](https://arxiv.org/html/2605.20296#A9.SS3)ablation\) are applied per\-tensor to every weight matrix withndim≥2\\mathrm\{ndim\}\\geq 2and at least10241024elements; tensors withndim\>2\\mathrm\{ndim\}\>2are reshaped to a22D matrix before SVD\. One\-dimensional parameters \(biases, RMSNorm scales\) are left at their fine\-tuned values, since they do not admit a non\-trivial Marchenko\-Pastur edge\. On Llama\-3\.2\-3B\-Instruct this scope covers every attention and MLP projection of the text decoder, plusembed\_tokensandlm\_head\. On Qwen3\.5\-4B the same filter additionally accepts the vision\-encoder weights and the multi\-token\-prediction head; because inference is text\-only, these tensors are stripped at vLLM load \(checkpoint\_compat\.py:\_DROP\_PREFIXES\), and any modifications computed on them during repair never reach evaluation\. The state\-spacelinear\_attnblocks contribute one\-dimensional parameters \(A\_log,dt\_bias\) that are skipped under the dimensionality rule; theirconv1d\.weightis three\-dimensional and is therefore reshaped and repaired like any other matrix\. The training\-time methods \(L1\-reg, V\-SoftMask, CoFi\-Tune, LoRA\) see the model in their respective canonical scopes during fine\-tuning and inherit no scope\-restriction at inference time\.

## Appendix EPer\-cell forgetting and balance scores

This appendix holds the two per\-cell tables that the body cites in §[4\.2](https://arxiv.org/html/2605.20296#S4.SS2)\(also labeled §[4\.2](https://arxiv.org/html/2605.20296#S4.SS2)\)\. Tab\.[9](https://arxiv.org/html/2605.20296#A5.T9)reports the \(model, task\) cell\-level fine\-tuning impact: how many of the nine held\-out benchmarks were damaged \(ΔFT≤−3\\Delta\_\{\\mathrm\{FT\}\}\\leq\-3pp\), how many were improved \(ΔFT≥\+3\\Delta\_\{\\mathrm\{FT\}\}\\geq\+3pp\), the average and worst drops over the damaged set, and the on\-task gainΔFT,task\\Delta\_\{\\mathrm\{FT,task\}\}\(Full\-SFT minus base\)\. The remaining \(9−damaged−improved9\-\\text\{damaged\}\-\\text\{improved\}\) benchmarks per cell are unchanged within±3\\pm 3pp\. Tab\.[10](https://arxiv.org/html/2605.20296#A5.T10)reports the per\-cell balance score, the harmonic mean of \(i\) the method’s mean held\-out ratio against base and \(ii\) the method’s on\-task ratio against Full\-SFT, each multiplied by100100\. The per\-cell view uses these mean ratios because the partition\-conditional sub\-statistics \(% healed, % preserved\) become unstable on the small per\-celln≈9n\\approx 9\. FAPM is omitted from the balance table because it never wins: its90%90\\%entry reversion drives the held\-out ratio toward base but collapses the on\-task ratio, leaving it well behind both DG\-Hard and WiSE\-FT in every cell\.

Table 9:Per\-cell impact of fine\-tuning across the1414\(model, task\) cells of our experimental matrix\. Fine\-tuning damages at least one held\-out benchmark in1313of1414cells and incidentally improves more held\-out benchmarks than it damages \(5555vs3030in total\)\.Table 10:Per\-cell balance scores per \(model, task\) cell, plus the cohort aggregation of the winner column\. The balance score is the harmonic mean of the method’s mean held\-out ratio \(vs base\) and on\-task ratio \(vs FT\), each multiplied by100100\. FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\]is excluded because it never wins on balance\.ModelTaskbal\(WiSE\)↑\\uparrowbal\(DG\)↑\\uparrowWinnerQwenRTE103\.6±0\.4103\.6\_\{\\pm 0\.4\}104\.3±0\.4\\bm\{104\.3\_\{\\pm 0\.4\}\}DG\-HardQwenStrategyQA102\.7±0\.6102\.7\_\{\\pm 0\.6\}104\.6±0\.6\\bm\{104\.6\_\{\\pm 0\.6\}\}DG\-HardQwenReClor98\.6±0\.498\.6\_\{\\pm 0\.4\}101\.5±0\.5\\bm\{101\.5\_\{\\pm 0\.5\}\}DG\-HardQwenBoolQ105\.9±0\.3105\.9\_\{\\pm 0\.3\}108\.5±0\.3\\bm\{108\.5\_\{\\pm 0\.3\}\}DG\-HardQwenMedQA105\.4±0\.5105\.4\_\{\\pm 0\.5\}105\.4±0\.5\\bm\{105\.4\_\{\\pm 0\.5\}\}tie: DG\-Hard≈\\approxWiSE\-FTQwenWikiQA101\.9±0\.2101\.9\_\{\\pm 0\.2\}103\.4±0\.3\\bm\{103\.4\_\{\\pm 0\.3\}\}DG\-HardQwenWinogrande98\.9±0\.398\.9\_\{\\pm 0\.3\}102\.8±0\.3\\bm\{102\.8\_\{\\pm 0\.3\}\}DG\-HardLlamaRTE100\.0±0\.6\\bm\{100\.0\_\{\\pm 0\.6\}\}98\.8±0\.698\.8\_\{\\pm 0\.6\}WiSE\-FTLlamaStrategyQA101\.1±0\.7\\bm\{101\.1\_\{\\pm 0\.7\}\}99\.2±0\.799\.2\_\{\\pm 0\.7\}WiSE\-FTLlamaReClor97\.4±0\.9\\bm\{97\.4\_\{\\pm 0\.9\}\}96\.0±0\.896\.0\_\{\\pm 0\.8\}WiSE\-FTLlamaBoolQ100\.7±0\.3\\bm\{100\.7\_\{\\pm 0\.3\}\}100\.2±0\.3100\.2\_\{\\pm 0\.3\}WiSE\-FTLlamaMedQA102\.1±0\.6102\.1\_\{\\pm 0\.6\}101\.9±0\.6101\.9\_\{\\pm 0\.6\}tie: DG\-Hard≈\\approxWiSE\-FTLlamaWikiQA100\.8±0\.3100\.8\_\{\\pm 0\.3\}101\.3±0\.2101\.3\_\{\\pm 0\.2\}tie: DG≈\\approxWiSE\-FTLlamaWinogrande97\.4±0\.497\.4\_\{\\pm 0\.4\}99\.5±0\.4\\bm\{99\.5\_\{\\pm 0\.4\}\}DG\-Hard*Aggregating the winner column:*CohortDG\-Hard winsWiSE\-FT winsTiesQwen \(reasoning,n=7n=7\)601Llama \(non\-reasoning,n=7n=7\)142All cells \(n=14n=14\)743
## Appendix FPer\-cohort Clean\-up and Retention

Tabs\.[11](https://arxiv.org/html/2605.20296#A6.T11)and[12](https://arxiv.org/html/2605.20296#A6.T12)report the per\-cohort Clean\-up and Retention sub\-scores for every method, computed exactly as in §[4\.3](https://arxiv.org/html/2605.20296#S4.SS3): Clean\-up=HM​\(% healed,Non\-damage rate\)=\\mathrm\{HM\}\(\\text\{\\% healed\},\\,\\text\{Non\-damage rate\}\)and Retention=HM​\(% preserved,On\-task retention\)=\\mathrm\{HM\}\(\\text\{\\% preserved\},\\,\\text\{On\-task retention\}\), with the partition\-conditional sub\-statistics themselves clipped to\[0,100\]\[0,100\]before the harmonic mean\. The Knowledge and Cognition rows are the source of the per\-cohort numbers cited in §[4\.6](https://arxiv.org/html/2605.20296#S4.SS6); the Overall row matches the population\-level Tab\.[1](https://arxiv.org/html/2605.20296#S4.T1)\. A0entry in the Retention table indicates that the underlying % preserved averaged non\-positive on the improved partition, so the clipped HM collapses to zero\.

Table 11:Per\-cohort Clean\-up=HM​\(% healed,Non\-damage rate\)=\\mathrm\{HM\}\(\\text\{\\% healed\},\\,\\text\{Non\-damage rate\}\), in\[0,100\]\[0,100\], higher is better;\>100\>100appears when % healed overshoots base on the damaged partition\.Bold= best per cohort;underline= second\-best\.Table 12:Per\-cohort Retention=HM​\(% preserved,On\-task retention\)=\\mathrm\{HM\}\(\\text\{\\% preserved\},\\,\\text\{On\-task retention\}\), in\[0,100\]\[0,100\], higher is better\. Entries of0indicate that the method’s average % preserved on the improved partition was non\-positive, collapsing the harmonic mean to zero\.Bold= best per cohort;underline= second\-best\.#### Per\-axis trends\.

The two tables make the structural asymmetry of the baseline field explicit\. On Retention \(Tab\.[12](https://arxiv.org/html/2605.20296#A6.T12)\),V\-SoftMaskdominates every cohort by a wide margin, scoring≥95\\geq 95on all five and exceeding the next\-best method by1414to2828pp on each\. This is consistent with V\-SoftMask’s mechanism: by damping gradient flow during fine\-tuning it leaves the held\-out and on\-task distributions closer to the base, so on the improved partition it inherits the same scores it produced and thus high % preserved\. The same mechanism explains its position on Clean\-up \(Tab\.[11](https://arxiv.org/html/2605.20296#A6.T11)\), where V\-SoftMask is the*lowest*\-scoring method on Overall \(24\.924\.9\) and Cognition \(18\.318\.3\) and falls to0on Non\-reasoning\. Because gradient damping suppresses both the noise residual and any genuine recovery, the method has nothing with which to bring damaged held\-out benchmarks back toward base, so its Clean\-up component collapses\. The mirror extremes areFAPMandLoRA, which post some of the highest Clean\-up scores \(91\.991\.9and85\.685\.6Overall, respectively\) yet hit0on every Cognition and Reasoning Retention cell because their average % preserved on the improved partition is non\-positive \(FAPM’s90%90\\%sparsity reversion drags every benchmark toward base; LoRA’s adapter delta never produced the improved\-partition lifts in the first place\)\.L1\-regandCoFi\-Tuneare partial versions of the same single\-axis pattern: high Clean\-up, weak or zero Retention on Cognition\.

#### Why balance is required\.

Combined=HM​\(Clean\-up,Retention\)=\\mathrm\{HM\}\(\\text\{Clean\-up\},\\,\\text\{Retention\}\)is designed precisely to demote single\-axis extremes via the bottleneck property of the harmonic mean: any method with a near\-zero component cannot recover at the headline level regardless of how strong its other axis is\. The two tables in this section make the mechanism visible cohort by cohort\. V\-SoftMask’s perfect Retention is undone by its near\-zero Clean\-up; FAPM’s strong Clean\-up is undone by its identically\-zero Retention; LoRA and CoFi\-Tune are similarly imbalanced in less extreme but still disqualifying ways\. DG\-Hard is the only method that scores in the upper half of both axes simultaneously across all five cohorts, which is consistent with its leading the population\-level Combined metric in Tab\.[1](https://arxiv.org/html/2605.20296#S4.T1)\. At the cohort level, DG\-Hard is not required to win Combined everywhere \(§[4\.6](https://arxiv.org/html/2605.20296#S4.SS6)notes that L1\-reg edges it on Knowledge\): the structural claim is that DG\-Hard is the only method that does not collapse on either axis in any cohort, not that it wins Combined in every cohort\.

## Appendix GFull per\-cohort sub\-score breakdown

§[4\.4](https://arxiv.org/html/2605.20296#S4.SS4)summarized the per\-cohort Combined scores for DG\-Hard vs WiSE\-FT\. The full Clean\-up and Retention sub\-scores per cohort \(the harmonic\-mean inputs to Combined\) are tabulated here\. DG\-Hard’s edge over WiSE\-FT comes mostly from Retention: their Clean\-up scores are within∼3\\sim 3pp on most cohorts, but DG\-Hard’s Retention is12\+12\{\+\}pp higher on Overall, Cognition, and Reasoning, driven mainly by held\-out preservation since on\-task retention is essentially flat at∼97%\\sim 97\\%for both methods\.

Table 13:Per\-cohort Clean\-up / Retention / Combined for DG\-Hard vs WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\]\.Boldmarks the higher Combined score in this*head\-to\-head*DG\-Hard vs WiSE\-FT comparison only; it is*not*a cross\-method best\-per\-cohort indicator\. For cross\-method per\-cohort winners across all baselines, see App\.[F](https://arxiv.org/html/2605.20296#A6)\(Tabs\.[11](https://arxiv.org/html/2605.20296#A6.T11)and[12](https://arxiv.org/html/2605.20296#A6.T12)\); per §[4\.6](https://arxiv.org/html/2605.20296#S4.SS6), DG\-Hard wins Combined on four of five cohorts at the cross\-method level \(L1\-reg edges it on Knowledge\)\.
## Appendix HPer\-\(model, task, method, benchmark\) results

For full transparency, Tab\.[14](https://arxiv.org/html/2605.20296#A8.T14)and Tab\.[15](https://arxiv.org/html/2605.20296#A8.T15)list every per\-cell, per\-benchmark score in the experimental matrix \(11341134measurements in total:22models×7\\times\\ 7tasks×9\\times\\ 9method rows×9\\times\\ 9held\-out benchmarks; each table additionally reports33derived columns \(*Avg\.*,*Results*,*Combined*\) per row\)\. Each task block contains nine method rows \(Pre\-trained, Full\-SFT, six repair baselines, plus our DG\-Hard\); within each task block,boldmarks the best score per column andunderlinemarks the second\-best \(when separated by≥0\.001\\geq 0\.001\)\. The right\-most three columns are*Avg\.*\(mean of the nine held\-out benchmarks\),*Results*\(on\-tasktask\_\{name\}score\), and*Combined*\(defined in the table caption\)\.

Table 14:Per\-benchmark held\-out scores for Qwen3\.5\-4B across all \(task, method\) cells\.Bold= best per column within the task block;underline= second\-best \(gap≥0\.001\\geq 0\.001from best\)\. Avg\. = mean of the nine held\-out benchmarks\. Results = on\-tasktask\_\{name\}score\. Combined=HM​\(% healed¯,% preserved¯\)=\\mathrm\{HM\}\(\\overline\{\\text\{\\% healed\}\},\\,\\overline\{\\text\{\\% preserved\}\}\)on the partitioned held\-out set, with per\-benchmark % healed and % preserved clipped to\[0,100\]\[0,100\]before averaging; cells with no damaged or no improved benchmarks default the corresponding side to100100\. Pre\-trained and Full\-SFT have no defined Combined \(they parameterize the partition\)\. Bold/underline on Combined follow the same convention as other columns, restricted to the seven repair methods \(excluding the two reference rows\)\. “Repair methods” here covers both training\-time interventions \(L1\-reg, V\-SoftMask\[[14](https://arxiv.org/html/2605.20296#bib.bib26)\], CoFi\-Tune\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\], LoRA\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]\) and post\-hoc methods \(WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\], FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\], DG\-Hard\); see §[2](https://arxiv.org/html/2605.20296#S2)for the full bucketing\.Table 15:Per\-benchmark held\-out scores for Llama\-3\.2\-3B\-Instruct across all \(task, method\) cells\.Bold= best per column within the task block;underline= second\-best \(gap≥0\.001\\geq 0\.001from best\)\. Avg\. = mean of the nine held\-out benchmarks\. Results = on\-tasktask\_\{name\}score\. Combined=HM​\(% healed¯,% preserved¯\)=\\mathrm\{HM\}\(\\overline\{\\text\{\\% healed\}\},\\,\\overline\{\\text\{\\% preserved\}\}\)on the partitioned held\-out set, with per\-benchmark % healed and % preserved clipped to\[0,100\]\[0,100\]before averaging\. Pre\-trained and Full\-SFT have no defined Combined\. Bold/underline on Combined follow the same convention as other columns, restricted to the seven repair methods \(excluding the two reference rows\)\. “Repair methods” here covers both training\-time interventions \(L1\-reg, V\-SoftMask\[[14](https://arxiv.org/html/2605.20296#bib.bib26)\], CoFi\-Tune\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\], LoRA\[[10](https://arxiv.org/html/2605.20296#bib.bib22)\]\) and post\-hoc methods \(WiSE\-FT\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\], FAPM\[[11](https://arxiv.org/html/2605.20296#bib.bib23)\], DG\-Hard\); see §[2](https://arxiv.org/html/2605.20296#S2)for the full bucketing\.### H\.1Method\-specific configurations

L1\-regadds an L1 penalty‖W−Wbase‖1\\\|W\-W\_\{\\mathrm\{base\}\}\\\|\_\{1\}to the loss withλ=1\.0×10−6\\lambda=1\.0\\times 10^\{\-6\}\.LoRAuses rankr=16r=16,α=32\\alpha=32, dropout0\.050\.05, and auto\-selected target modules\.WiSE\-FTsets the mixing weightα=0\.5\\alpha=0\.5, applied post\-hoc on the Full\-SFT checkpoint to give the linear weight averageW∗=12​Wbase\+12​WftW^\{\*\}=\\tfrac\{1\}\{2\}W\_\{\\mathrm\{base\}\}\+\\tfrac\{1\}\{2\}W\_\{\\mathrm\{ft\}\}, the canonical setting fromWortsmanet al\.\[[34](https://arxiv.org/html/2605.20296#bib.bib45)\]\.FAPMreverts the entries ofΔ\\Deltaranked lowest under the forgetting\-aware FAPM score, which combines absolute change with a relative\-change penalty againstWbaseW\_\{\\mathrm\{base\}\}\); we use the published90%90\\%reversion rate\.V\-SoftMask\(our reproduction of DAS,[14](https://arxiv.org/html/2605.20296#bib.bib26)\) uses a calibration batch size of44and applies mask=\(1−normalised gradient importance\)=\(1\-\\text\{normalised gradient importance\}\), fixed for the SFT run\.CoFi\-Tuneuses the two\-stage configuration ofZhanget al\.\[[38](https://arxiv.org/html/2605.20296#bib.bib50)\]verbatim \(full mechanism in §[2](https://arxiv.org/html/2605.20296#S2)\); we run the coarse layer\-range filter at the second\-quartile setting\(N×25%,N×50%\]\(N\\times 25\\%,N\\times 50\\%\]\.DG\-Hardapplies theω​\(β\)⋅σ^eff\\omega\(\\beta\)\\cdot\\hat\{\\sigma\}\_\{\\text\{eff\}\}hard threshold of §[3](https://arxiv.org/html/2605.20296#S3); it has no tuning knobs and uses no calibration data\.TIES\[[35](https://arxiv.org/html/2605.20296#bib.bib47)\]keeps the top\-kkentries of\|Δ\|\|\\Delta\|by magnitude, sign\-elects, and averages; we usekeep\_ratio=0\.2=0\.2\(k=20%k=20\\%\) with mixing weightλ=1\\lambda=1\.DARE\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\]drops entries ofΔ\\Deltawith probabilityppand rescales the survivors by1/\(1−p\)1/\(1\-p\); we usedrop\_prob=0\.5=0\.5with the rescaled \(rather than raw\) variant\. TIES and DARE are evaluated only in the App\.[I\.3](https://arxiv.org/html/2605.20296#A9.SS3)ablation, not in the headline comparison of Tab\.[14](https://arxiv.org/html/2605.20296#A8.T14)and Tab\.[15](https://arxiv.org/html/2605.20296#A8.T15)\.

### H\.2Inference engine and decoding

All inference uses vLLMwith\-\-enforce\-eager,max\_model\_len=8192,dtype=bfloat16, greedy decoding \(temperature=0=0\),*no*max\_tokenscap, and the model’s native chat template applied via the tokenizer’sapply\_chat\_templatemethod\. Greedy decoding under vLLM’s continuous batching is not bit\-deterministic: the same prompt can produce slightly different completions on re\-runs, with a measureddrift floor of about±1\\pm 1pp per benchmark\. We treat any delta smaller than this floor as noise, and use a3 ppsignificance thresholdthroughout the paper\. The threshold is conservative \(≈3×\\approx 3\\timesthe drift floor\) and is chosen to capture genuine but moderate fine\-tuning effects while excluding sampling drift\. Output scoring strips<think\>…</think\>blocks and reasoning preludes before metric computation\.

## Appendix IOther experiments and ablations

This appendix collects three experiments supporting the design choices of DG\-Hard\. App\.[I\.1](https://arxiv.org/html/2605.20296#A9.SS1)quantifies which weight classes carry the noise mass that DG\-Hard removes \(layer\-class noise concentration\); App\.[I\.2](https://arxiv.org/html/2605.20296#A9.SS2)verifies the layer\-mask choice via a causal test; App\.[I\.3](https://arxiv.org/html/2605.20296#A9.SS3)compares against element\-wise merging baselines at canonical and matched\-rollback operating points\.

### I\.1Layer\-wise noise concentration

For each \(model, task\) fine\-tune, we cache the SVD of every 2D weight delta and classify each matrix by its layer class \(e\.g\.,mlp\.up\_proj,attn\.q\_proj\)\. Per class we report the mean noise\-energy fraction \(the share of singular\-value\-squared mass sitting below the MP edgeλMP\\lambda\_\{\\mathrm\{MP\}\}of \([3](https://arxiv.org/html/2605.20296#S3.E3)\)\) and the total norm\-share \(the fraction of the network’s Frobenius2delta mass attributable to the class\)\. Their product is the operationally meaningful noise mass per class under the signal\-plus\-noise model of \([2](https://arxiv.org/html/2605.20296#S3.E2)\)\. Averaged across six \(model, task\) pairs, the MLP gate and up projections concentrate the bulk of the noise mass that DG\-Hard removes \(Tab\.[16](https://arxiv.org/html/2605.20296#A9.T16)\); attention and embedding classes contribute substantially less\.

Table 16:Per\-class noise ranking, averaged over six \(model, task\) pairs\. Mean noise is the fraction of squared singular\-value mass belowλMP\\lambda\_\{\\mathrm\{MP\}\}\. Mean norm\-share is the class share of the network’s total Frobenius2delta\. Their product is the noise mass per class\. Top\-two rows in bold\.#### Metric note\.

The mean\-noise column above \(energy fraction of squared singular values belowλMP\\lambda\_\{\\mathrm\{MP\}\}\) and the MP\-fit score of \([8](https://arxiv.org/html/2605.20296#A1.E8)\) \(a Kolmogorov\-Smirnov distance between the empirical and theoretical CDFs\) measure different things: the former is the share of squared mass under the bulk edge, the latter is how closely the spectrum’s shape matches the MP density\. The numerical proximity of themlp\.up\_projrow’s0\.930\.93to the93\.1%93\.1\\%MP\-fit reported in App\.[A](https://arxiv.org/html/2605.20296#A1)is therefore coincidental, not the same quantity\. Theembed\_tokensvalue≈1\.00\\approx 1\.00is a degenerate case in which essentially all of the FT delta’s spectral mass sits under the bulk edge: there is no task\-specific signal in the embedding update on these \(model, task\) pairs, consistent with App\.[A](https://arxiv.org/html/2605.20296#A1)’s claim that the learned signal is concentrated in a small number of spike directions, just not in the embedding for these fine\-tunes\.

The aggregate ranking conceals a sharp cross\-family asymmetry in attention noise \(Tab\.[17](https://arxiv.org/html/2605.20296#A9.T17)\): on Llama\-3\.2\-3B,attn\.q\_projhas mean below\-edge mass0\.0250\.025, while on Qwen3\.5\-4B the same class shows0\.910\.91, a36×36\\timesratio\. Theattn\.o\_projclass shows a7×7\\timesratio in the same direction\. MLP classes are family\-invariant \(ratios within1\.161\.16\)\.

Table 17:Cross\-family asymmetry in mean noise\-energy fraction\. Attention classes diverge sharply between Llama\-3\.2\-3B\-Instruct and Qwen3\.5\-4B; MLP classes do not\.The family asymmetry is consistent with the hypothesis that reasoning post\-training \(long chain\-of\-thought RL or distillation, present in Qwen3\.5\-4B but not in Llama\-3\.2\-3B\-Instruct\) disperses attention weights and leaves more below\-edge mass in the FT delta, but this explanation is preliminary and untested in the present run\. Regardless of the underlying cause, the gate and up MLP classes carry the dominant share of removable noise mass in both families, motivating the gate\+\+up layer mask whose causal effect is tested in §[I\.2](https://arxiv.org/html/2605.20296#A9.SS2)\.

### I\.2Layer\-mask causal test

App\.[I\.1](https://arxiv.org/html/2605.20296#A9.SS1)ranks 2D weight classes by noise mass and identifiesgate\_projandup\_projas the top\-two carriers on every \(model, task\) pair\. This appendix verifies that ranking causally: we vary the set of layers pruned by DG\-Hard and measure cross\-domain accuracy\. The protocol is DG\-Hard canonical \(scale=1\.0=1\.0, DGσ\\sigma\) on two pairs, Qwen3\.5\-4B with Winogrande \(severe FT damage\) and Llama\-3\.2\-3B with MedQA \(mild FT damage\), under four masks:ALL\(every 2D matrix\),mlp\_only\(\\\.mlp\\\.\),attn\_only\(\\\.self\_attn\\\.\), andgate\_up\(\\\.mlp\\\.\(gate\|\|up\)\_proj\)\. Cells outside the active mask pass through with the full FT delta \(no shrinkage\); we report Frobenius retentionrralongside benchmark accuracies and the protocol\-aligned Combined score\. The four masks above target attention/MLP partitions of the decoder; theembed\_tokensandlm\_headmatrices, which are in scope for the headline DG\-Hard configuration \(App\.[D\.5](https://arxiv.org/html/2605.20296#A4.SS5)\), are kept at theirALL\-mask repaired state in all four conditions of this ablation\. The focus of this experiment is the attention\-vs\-MLP partition; the embedding contribution is characterised separately in App\.[I\.1](https://arxiv.org/html/2605.20296#A9.SS1)\.

#### Column conventions\.

task\_winandtask\_medare on\-task accuracy on Winogrande and MedQA, respectively\. The Combined column applies the partition\-conditional harmonic mean of \([7d](https://arxiv.org/html/2605.20296#S3.E7.4)\); whenU=∅U=\\emptyset\(Tab\.[18](https://arxiv.org/html/2605.20296#A9.T18)\) the non\-damage term of \([7](https://arxiv.org/html/2605.20296#S3.E7)\) is undefined \(0/00/0\) and we adopt the conventionnon\-damage=100\\text\{non\-damage\}=100, matching the implementation\. With this convention theattn\_onlyrow of Tab\.[18](https://arxiv.org/html/2605.20296#A9.T18)collapses to Combined=0\.0=0\.0because percent\-healed is negative on the damaged benches \(GSM8K, ARC\-C\), which the harmonic mean drives to zero independently of the U\-term\.

Table 18:Mask ablation on Qwen3\.5\-4B \+ Winogrande \(severe FT damage\)\.D=\{arc\_challenge,gsm8k\}D=\\\{\\text\{arc\\\_challenge\},\\text\{gsm8k\}\\\},I=\{hellaswag,ifeval,mmlu\}I=\\\{\\text\{hellaswag\},\\text\{ifeval\},\\text\{mmlu\}\\\},U=∅U=\\emptyset\. Combined uses the protocol harmonic mean over the partition\. Winning mask on Combined in bold\.Table 19:Mask ablation on Llama\-3\.2\-3B \+ MedQA \(mild FT damage\)\.D=\{gsm8k\}D=\\\{\\text\{gsm8k\}\\\},I=\{hellaswag,mmlu\}I=\\\{\\text\{hellaswag\},\\text\{mmlu\}\\\},U=\{arc\_challenge,ifeval\}U=\\\{\\text\{arc\\\_challenge\},\\text\{ifeval\}\\\}\. Winning mask on Combined in bold\.
#### Pattern A \(Qwen, severe FT\)\.

ALL,mlp\_only, andgate\_upcluster closely on the per\-benchmark damaged accuracies \(GSM8K 87\.6–89\.2, ARC\-C 67\.7–73\.9\);attn\_onlycollapses on the damaged benches with GSM8K dropping to 58\.5 \(vs\. 87\.7 underALL\) and ARC\-C to 55\.3 \(vs\. 73\.9\)\. The protocol\-aligned Combined column is more discriminating still:attn\_onlyscores 0\.0 because it has negative percent healed on the damaged benches, which the harmonic mean penalises\. Repairing only attention while leaving the MLP delta unshrunken keeps too much noise in the model, regardless of how aggressively attention is repaired\. Repairing the MLP class is necessary; repairing only attention is insufficient\.

#### Pattern B \(Llama, mild FT\)\.

All four masks cluster tightly across the per\-benchmark accuracies, and Combined separates them only mildly\. With a mild FT delta, the absolute amount of noise to remove is small, so any reasonable repair yields similar cross\-domain accuracy\. We treat this as a*negative*result: the variance is genuinely small, not evidence that all masks are equal\. The cross\-family contrast is informative\. On Llama, attention is intrinsically low\-noise \(§[I\.1](https://arxiv.org/html/2605.20296#A9.SS1)\), soattn\_onlyacts almost as a pass\-through and lands nearALL\. On Qwen,attn\.q\_projcarries noise\-mass 0\.91 \(vs\. 0\.025 on Llama\), so leaving the MLP unrepaired collapses the cell\.attn\_onlyis therefore not a fair control across families: it isolates a different noise source on each\.

Across both pairs,gate\_upis the highest\-rrmask whose per\-benchmark accuracies matchALL\(retention 0\.78 and 0\.73 vs\. 0\.31 and 0\.19\), capturing the top\-two noise classes from §[I\.1](https://arxiv.org/html/2605.20296#A9.SS1)while preserving 73 to 78 percent of the FT delta: the operational sweet spot of maximum FT preservation with minimum noise leakage\.

### I\.3Comparison with element\-wise merging baselines

We compare DG\-Hard against three element\-wise merging baselines on two pairs \(Qwen3\.5\-4B \+ Winogrande, severe FT, and Llama\-3\.2\-3B \+ MedQA, mild FT\): WiSE\-FT and Task Arithmetic\[[34](https://arxiv.org/html/2605.20296#bib.bib45),[12](https://arxiv.org/html/2605.20296#bib.bib24)\], which are mathematically identical for a single \(base, FT\) pair and which we report jointly; TIES\[[35](https://arxiv.org/html/2605.20296#bib.bib47)\], which keeps the topkkentries of\|Δ\|\|\\Delta\|, sign\-elects, and averages; and DARE\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\], which randomly drops entries with probabilityppand rescales surviving entries by1/\(1−p\)1/\(1\-p\)\. We use “DARE” to denote the canonical rescaled form \(rescale=True\) and “DARE\-Linear” to denote the variant without rescaling \(rescale=False\)\. Each method is evaluated at two operating points\. Part A uses each baseline at its published default \(WiSE\-FT/Task\-Arithα=0\.5\\alpha\{=\}0\.5; TIESkeep\_ratio=0\.2=0\.2withλ=1\\lambda\{=\}1; DAREdrop\_prob=0\.5=0\.5withrescale=True; DG\-Hard at its parameter\-free RMT\-optimal threshold\)\. Part B bisects every baseline to the same Frobenius rollbackr=0\.5r\{=\}0\.5, isolating shape effects from magnitude effects; this matched\-rollback comparison is the central methodological control\. For DARE in Part B we use the linear no\-rescale form \(DARE\-Linear\), since DARE withrescale=Trueatp=0\.5p\{=\}0\.5has expected Frobenius ratio1/\(1−p\)≈1\.41\\sqrt\{1/\(1\-p\)\}\\approx 1\.41, which is incompatible with the matchedr=0\.5r\{=\}0\.5target\.

Table 20:Merging baselines on Qwen3\.5\-4B \+ Winogrande \(severe FT damage\)\.D=\{arc\_challenge,gsm8k\}D=\\\{\\text\{arc\\\_challenge\},\\text\{gsm8k\}\\\},I=\{hellaswag,ifeval,mmlu\}I=\\\{\\text\{hellaswag\},\\text\{ifeval\},\\text\{mmlu\}\\\},U=∅U=\\emptyset\. Combined uses the protocol harmonic mean over the partition\. Winning Combined per part in bold\. DARE’s\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\]expected Frobenius ratio is1/\(1−p\)≈1\.41\\sqrt\{1/\(1\-p\)\}\\approx 1\.41atp=0\.5p\{=\}0\.5; the realizedr=1\.155r\{=\}1\.155reflects single\-seed mask variance\. Combined of0indicates a sub\-statistic clipped to its floor \(e\.g\., % preserved non\-positive on the improved partition\), which collapses the harmonic mean\.Table 21:Merging baselines on Llama\-3\.2\-3B \+ MedQA \(mild FT damage\)\.D=\{gsm8k\}D=\\\{\\text\{gsm8k\}\\\},I=\{hellaswag,mmlu\}I=\\\{\\text\{hellaswag\},\\text\{mmlu\}\\\},U=\{arc\_challenge,ifeval\}U=\\\{\\text\{arc\\\_challenge\},\\text\{ifeval\}\\\}\. Winning Combined per part in bold\. DARE’s\[[36](https://arxiv.org/html/2605.20296#bib.bib48)\]expected Frobenius ratio is1/\(1−p\)≈1\.41\\sqrt\{1/\(1\-p\)\}\\approx 1\.41atp=0\.5p\{=\}0\.5; the realizedr=1\.155r\{=\}1\.155reflects single\-seed mask variance\. Combined of0indicates a sub\-statistic clipped to its floor \(e\.g\., % preserved non\-positive on the improved partition\), which collapses the harmonic mean\.#### Canonical operating points\.

On Qwen \+ Winogrande, DG\-Hard wins on Combined \(72\.6 vs\. 67\.5 for WiSE\-FT/Task\-Arith\) and improves the damaged benches relative to TIES/DARE \(GSM8K 87\.7 vs\. 57\.3/56\.3; ARC\-C 73\.9 vs\. 62\.6/65\.8\)\. On Llama \+ MedQA, TIES and DARE post slightly higher MMLU than DG\-Hard \(55\.7/55\.3 vs\. 38\.7\), yet their Combined collapses to 5\.4 and 0\.0 respectively\. The explanation is visible in therrcolumn: at canonical settings TIES and DARE sit atr≈1r\{\\approx\}1, so they do not shrink the FT delta and therefore heal a negligible fraction of the FT\-induced damage on GSM8K\. The partition\-conditional Combined headline reveals this gap; an unweighted per\-benchmark mean would not\. WiSE\-FT and Task Arithmetic produce identical outputs for a single \(base, FT\) pair, so we report a single row and credit both citations\.

#### Matched Frobenius rollbackr=0\.5r\{=\}0\.5\.

With magnitude held fixed, DG\-Hard wins both pairs on Combined \(\+14\.0\+14\.0pp on Qwen \+ Winogrande, 81\.5 vs\. 67\.5;\+17\.6\+17\.6pp on Llama \+ MedQA, 90\.4 vs\. 72\.8\)\. The per\-benchmark breakdown shows why: TIES atr=0\.5r\{=\}0\.5collapses HellaSwag \(47\.3 vs\. 87\.1 under DG\-Hard\) and DARE\-Linear atr=0\.5r\{=\}0\.5collapses ARC\-C relative to DG\-Hard’s distribution of mass while also collapsing HellaSwag \(47\.3\)\. DG\-Hard is the only method that rolls back tor≈0\.5r\{\\approx\}0\.5without driving any of the five benchmarks to a degenerate value\. TIES and DARE\-Linear sparsify in the standard parameter basis while DG\-Hard sparsifies in the singular\-vector basis; both bets land at the same Frobenius rollback, but spectral sparsity beats coordinate sparsity at the matched rollback point\.