在 LoRA B 矩阵内基于损失的遗忘任务中，静态梯度归因的表现不及密度匹配的随机掩码

# Static Gradient Attribution Underperforms aDensity-Matched Random Mask on Loss-Based Forgetting Within LoRA’s B-Matrix
Source: [https://arxiv.org/html/2610.12744](https://arxiv.org/html/2610.12744)
Yu Sun††thanks:Corresponding author: yu\.sun@alumni\.upenn\.eduJunwei ZhouAffiliation:Matter Innovation Inc\.Zuodong XiangAffiliation:University of California, DavisYike ZhangAffiliation:Johns Hopkins UniversityPengcheng XuAffiliation:University of California, IrvineXinyu GuanAffiliation:University of GlasgowRuoyun MaAffiliation:Stanford UniversityHailu XuAffiliation:California State University, Long Beach

###### Abstract

Parameter\-efficient continual learning updates a restricted subset of adapter parameters to incorporate sequential tasks while limiting catastrophic forgetting\. Gradient attribution is a common rule for selecting that subset, but it requires an additional backward pass over calibration data\. We identify a parameter\-pool confound in comparisons between initialization\-time gradient masks and unrestricted random masks under standard zero\-initialized LoRA \(B=0B=0\): the gradient with respect toAAis identically zero \(∂ℒ/∂A=0\\partial\\mathcal\{L\}/\\partial A=0\), so gradient masks select strictly from theBBmatrix whereas unrestricted random masks draw from bothAAandBB\. Such a comparison changes both the selection rule and the candidate parameter pool\. Isolating the selection criterion by restricting both approaches to the same parameter pool, we find a negative result: a uniform random mask consistently achieves lower loss\-based backward transfer than gradient attribution across seven seeds on Qwen2\.5\-3B\. The effect is concentrated on the medical task, which contributes 80% of the aggregate; code and math move in the same direction, and legal shows no reliable difference\. This advantage is consistent with a stability–plasticity trade\-off under the tested task sequence, in which random masking slightly reduces single\-task fitting\. On the two discrete\-option tasks \(Medical and Legal\), the mean accuracy\-BWT difference is not statistically distinguishable from zero\. The 95% interval extends to4\.804\.80percentage points in the gradient mask’s favour, so the pre\-specified 4\-point non\-inferiority criterion is not met\. On loss\-based forgetting, Random\-B consistently outperforms the density\-matched gradient mask across seven seeds; on accuracy, the evidence remains inconclusive atn=7n=7\. The attribution pass is therefore not justified by the loss\-based forgetting evidence, while its accuracy consequences require a better\-powered evaluation\.

## 1Introduction

Parameter\-efficient adapters such as LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12744#bib.bib4);[Dettmers et al\., 2023](https://arxiv.org/html/2610.12744#bib.bib5)\)make it possible to incorporate a sequence of tasks while updating only a small fraction of a language model\. Sequential updates can nevertheless overwrite capabilities acquired on earlier tasks, the classical problem of catastrophic forgetting\([McCloskey and Cohen, 1989](https://arxiv.org/html/2610.12744#bib.bib10);[French, 1999](https://arxiv.org/html/2610.12744#bib.bib9)\)\. Continual learning for LoRA adapters updates only a chosen subset of parameters, and gradient attribution is the common rule for choosing it: a backward pass over held\-out calibration data scores each parameter, and the highest\-scoring ones are updated\. Choosing the subset uniformly at random costs nothing instead, so whether the attribution pass earns its cost has a direct engineering answer\.

Standard zero\-initialized LoRA introduces a potential confound\. WithB=0B=0at initialization,∂ℒ/∂A=0\\partial\\mathcal\{L\}/\\partial A=0, so gradient attribution assigns zero importance to everylora\_Aparameter and a gradient mask selects only fromlora\_B, while an unrestricted random baseline draws from both factors together\. Any such comparison mixes the selection criterion with the parameter pool\. Attribution\-Guided CL\([Liu et al\., 2026b](https://arxiv.org/html/2610.12744#bib.bib24)\)is the closest prior use of gradient attribution in LLM continual learning, but it estimates importance from a separately trained model; becauseB≠0B\\neq 0at scoring time, the zero\-initialization confound studied here does not apply to its attribution procedure\.

We separate the two by restricting both arms to the same pool and parameter count\. On Qwen2\.5\-3B with rank\-16 LoRA over a five\-task sequence we compare a uniformly random mask onlora\_B\(Random\-B\) against a gradient mask renormalised to the same 10% oflora\_B\(Grad\-B\), so both train exactly the same parameters and differ only in how those parameters are chosen\. Across seven seeds the random mask achieves the lower loss\-based backward transfer, and the gap holds in every seed \(Table[2](https://arxiv.org/html/2610.12744#S4.T2), Figure[1](https://arxiv.org/html/2610.12744#S1.F1)\)\. The aggregate gap is concentrated on the medical task, with code and math moving in the same direction and legal showing no reliable difference, a pattern consistent with a stability–plasticity trade\-off: the random mask fits each task less well on its own diagonal yet retains earlier tasks better\. Above all, this work makes two key contributions:

\(a\) Method and comparison\.

\(b\) Key effect \(D1, seven seeds\)\.

Figure 1:The density\-matched comparison\. \(a\) Both arms draw 10% of the rank\-16lora\_Bpool, so they train exactly the same 1\.6M parameters and differ only in how those coordinates are chosen; becauseB=0B=0at initialization, the gradient score is zero for everylora\_Acoordinate, and any baseline that draws from both pools changes the pool as well as the criterion\. \(b\) Under that matched density the random mask achieves lower loss\-based BWT than the gradient mask in all seven seeds\.A gradient\-attribution confound and density\-matched evaluation\.

We show that under standard zero\-initialized LoRA, initialization\-time gradient attribution assigns zero importance tolora\_A, so an unrestricted gradient\-versus\-random comparison changes both the selection rule and the candidate parameter pool\. We resolve this confound with a within\-lora\_B, density\-matched design that holds the candidate pool and update budget fixed, leaving the selection criterion as the only varying factor\.

Practical validation and scope\.We quantify the cost of the attribution pass and evaluate the same comparison using accuracy\-based retention \(accuracy\-BWT\)\. The accuracy\-BWT interval spans zero and does not establish the pre\-specified 4\-point non\-inferiority criterion, so the engineering recommendation is scoped to the loss\-based evidence\. Removing attribution eliminates a one\-time 64\-sample calibration forward–backward pass \(64 steps\), equal to 12\.8% of one 500\-step task\-training budget, together with the calibration\-data requirement\.

## 2Background and Related Work

Continual learning and parameter importance\.Continual learning aims to reduce catastrophic forgetting as models are updated on sequential tasks\([McCloskey and Cohen, 1989](https://arxiv.org/html/2610.12744#bib.bib10);[French, 1999](https://arxiv.org/html/2610.12744#bib.bib9)\)\. Importance\-based methods such as EWC\([Kirkpatrick et al\., 2017](https://arxiv.org/html/2610.12744#bib.bib1)\)and SI\([Zenke et al\., 2017](https://arxiv.org/html/2610.12744#bib.bib2)\)protect parameters judged important to earlier tasks, while GPM\([Saha et al\., 2021](https://arxiv.org/html/2610.12744#bib.bib19)\), OGD\([Farajtabar et al\., 2020](https://arxiv.org/html/2610.12744#bib.bib18)\), and gradient\-based selective\-update methods\([Qiao et al\., 2025](https://arxiv.org/html/2610.12744#bib.bib30);[Wang et al\., 2025a](https://arxiv.org/html/2610.12744#bib.bib17)\)constrain updates using gradient or parameter\-importance structure\. Pruning methods pose a related selection problem: Wanda\([Sun et al\., 2024](https://arxiv.org/html/2610.12744#bib.bib3)\)uses activation\-aware saliency and SNIP\([Lee et al\., 2019](https://arxiv.org/html/2610.12744#bib.bib27)\)scores connections before training\. These lines motivate our question: once the candidate pool and update budget are controlled, does the*selection rule*itself improve continual\-learning retention\.

Random subspaces and intrinsic dimensionality\.A complementary view is that the update budget may matter more than the exact coordinates selected\. Random subnetworks can remain effective\([Frankle and Carbin, 2019](https://arxiv.org/html/2610.12744#bib.bib6);[Ramanujan et al\., 2020](https://arxiv.org/html/2610.12744#bib.bib8)\), fine\-tuning often occupies a low\-dimensional subspace\([Aghajanyan et al\., 2021](https://arxiv.org/html/2610.12744#bib.bib12)\), and large fractions of delta parameters can be removed\([Yu et al\., 2024](https://arxiv.org/html/2610.12744#bib.bib20)\)\. Related PEFT results point in the same direction: HAT\([Serrà et al\., 2018](https://arxiv.org/html/2610.12744#bib.bib26)\)learns task\-specific masks,[Xu and Zhang \(2024\)](https://arxiv.org/html/2610.12744#bib.bib14)show that random masking can recover strong PEFT subnetworks, and[Biderman et al\. \(2024\)](https://arxiv.org/html/2610.12744#bib.bib15)find that LoRA forgets less than full fine\-tuning\. Because LoRA already constrains updates to a low\-rank subspace, masking imposes a second restriction within it; our question is whether gradient attribution improves over a random restriction of the same size\.

#### LoRA\-specific continual learning and parameterization\.

LoRA\-FA\([Zhang et al\., 2023](https://arxiv.org/html/2610.12744#bib.bib29)\)freezeslora\_Aand trains onlylora\_B, whereas we freeze a random or gradient\-selected subset of a standard LoRA\. O\-LoRA\([Wang et al\., 2023](https://arxiv.org/html/2610.12744#bib.bib16)\)and InfLoRA\([Liang and Li, 2024](https://arxiv.org/html/2610.12744#bib.bib21)\)instead organize task updates into separated subspaces\. PiSSA\([Meng et al\., 2024](https://arxiv.org/html/2610.12744#bib.bib7)\)is important to the scope of our analysis because it initializesB≠0B\\neq 0, so the zero\-gradient confound we identify is specific to standard zero\-initialized LoRA\.

#### Closest and concurrent work\.

Most directly related, Attribution\-Guided CL\([Liu et al\., 2026b](https://arxiv.org/html/2610.12744#bib.bib24)\)uses gradient attribution in LLM continual learning, but estimates importance from a separately trained model and gates gradients continuously\. BecauseB≠0B\\neq 0at scoring time, our zero\-initialization confound does not apply to that procedure; it nevertheless motivates the broader question of whether attribution improves retention relative to simpler controls\. Concurrent results also point to substantial redundancy in adapter or sparse parameterizations:[Dieudonné et al\. \(2026\)](https://arxiv.org/html/2610.12744#bib.bib22)study low\-rank redundancy across task\-specific LoRA adapters, while[Adewuyi et al\. \(2026\)](https://arxiv.org/html/2610.12744#bib.bib23)find that random sparse subnetworks suffice for RLVR\. Our contribution is complementary: we isolate the selection criterion itself by matching both the LoRA parameter pool and the update budget\.

#### Adjacent applications of model adaptation and supervision\.

Beyond continual learning, efficient adaptation is relevant to resource\-constrained retrieval\-augmented language models, as discussed in a survey of energy\-efficient on\-device RAG with small language models\([Cheng et al\., 2026](https://arxiv.org/html/2610.12744#bib.bib31)\)\. Application\-focused studies have also explored in\-prompt process supervision for content moderation\([Liu et al\., 2026a](https://arxiv.org/html/2610.12744#bib.bib32)\), reasoning\-enhanced multimodal domain adaptation\([Wang et al\., 2025b](https://arxiv.org/html/2610.12744#bib.bib33)\), and audio\-enhanced vision–language representation learning with latent\-space broadening\([Sun et al\., 2025](https://arxiv.org/html/2610.12744#bib.bib34)\)\. These works concern different tasks and training settings; they provide broader context for adaptation and supervision, not prior evaluations of density\-matched LoRA gradient masking\.

## 3Methodology: Masking Mechanisms

Table 1:Notation\. Indices run over taskst=1,…,Tt=1,\\dots,TwithT=5T=5, and over prior tasksj∈𝒫=\{1,…,T−1\}j\\in\\mathcal\{P\}=\\\{1,\\dots,T\-1\\\}\.### 3\.1Problem Formulation

We consider the standard sequential fine\-tuning protocol for continual learning\. A base modelfθf\_\{\\theta\}is adapted to tasktton its training split𝒟t\\mathcal\{D\}\_\{t\}and then evaluated on all tasksj≤tj\\leq t\. Tasks arrive in a fixed order indexed byt=1,…,Tt=1,\\dots,T, withT=5T=5throughout this paper\.

We use LoRA adapters, which add a low\-rank updateΔ​W=B​A\\Delta W=BAto a frozen pretrained weight matrixW0W\_\{0\}, whereB∈ℝdout×rB\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\},A∈ℝr×dinA\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}, andr≪min⁡\(dout,din\)r\\ll\\min\(d\_\{\\mathrm\{out\}\},d\_\{\\mathrm\{in\}\}\):

W=W0\+α​B​A\.W=W\_\{0\}\+\\alpha BA\.\(1\)Table[1](https://arxiv.org/html/2610.12744#S3.T1)summarizes the notation used throughout\. A binary maskm∈\{0,1\}\|A\|\+\|B\|m\\in\\\{0,1\\\}^\{\|A\|\+\|B\|\}restricts optimization to coordinates for whichmi=1m\_\{i\}=1\. The mask is constructed once from the calibration set before the first task and remains fixed throughout the task sequence \(Algorithm[1](https://arxiv.org/html/2610.12744#alg1)\)\.

We compare two coordinate\-selection rules:*gradient attribution*, which ranks coordinates by their calibration\-set gradient magnitude, and*uniform random selection*\. For a fixed candidate parameter pool, both methods activate the same budgetkk; they therefore differ only in which coordinates are selected\.

We measure forgetting using loss\-based backward transfer:

BWT=1\|𝒫\|​∑j∈𝒫\(ℒj\(T\)−ℒj\(j\)\),\\mathrm\{BWT\}=\\frac\{1\}\{\|\\mathcal\{P\}\|\}\\sum\_\{j\\in\\mathcal\{P\}\}\\left\(\\mathcal\{L\}\_\{j\}^\{\(T\)\}\-\\mathcal\{L\}\_\{j\}^\{\(j\)\}\\right\),\(2\)where𝒫=\{1,…,T−1\}\\mathcal\{P\}=\\\{1,\\dots,T\-1\\\}\. This is the mean increase in evaluation loss on prior tasks after completing the full sequence; lower values indicate less forgetting\.

### 3\.2TheB=0B=0Gradient\-Attribution Confound

Standard LoRA initializesB=0B=0\. For

W=W0\+α​B​A,W=W\_\{0\}\+\\alpha BA,\(3\)the gradient with respect toAAis

∂ℒ∂A=α​B⊤​∂ℒ∂W\.\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\}=\\alpha B^\{\\top\}\\frac\{\\partial\\mathcal\{L\}\}\{\\partial W\}\.\(4\)Hence, at initialization,

B=0⟹∂ℒ∂A=0\.B=0\\quad\\Longrightarrow\\quad\\frac\{\\partial\\mathcal\{L\}\}\{\\partial A\}=0\.\(5\)Gradient attribution therefore assigns zero importance to everylora\_Acoordinate and selects exclusively fromlora\_B\. In contrast, an unrestricted random mask samples from the full LoRA parameter pool,lora\_A∪lora\_B\\texttt\{lora\\\_A\}\\cup\\texttt\{lora\\\_B\}\. A direct comparison between these masks consequently changes two factors at once: the*selection criterion*and the*candidate parameter pool*\.

To isolate the selection criterion, the candidate pool must therefore be matched\. Restricting gradient attribution explicitly tolora\_Bis a null operation under zero initialization, whereas restricting the random baseline tolora\_Bremoves the pool mismatch\. Our primary comparison accordingly evaluates gradient and random selection within the samelora\_Bpool; Section[5\.2](https://arxiv.org/html/2610.12744#S5.SS2)additionally matches their selection density\.

This scope excludes Attribution\-Guided CL\([Liu et al\., 2026b](https://arxiv.org/html/2610.12744#bib.bib24)\), which computes attribution from a separately trained model\. BecauseB≠0B\\neq 0at scoring time in that procedure, the zero\-initialization confound identified here does not apply to it\.

This confound is specific to standard zero\-initialized LoRA\. For example, PiSSA\([Meng et al\., 2024](https://arxiv.org/html/2610.12744#bib.bib7)\)initializes bothAAandBBfrom principal singular values, soB≠0B\\neq 0and Equation[4](https://arxiv.org/html/2610.12744#S3.E4)does not force the gradient with respect toAAto vanish\. LoRA\+\([Hayou et al\., 2024](https://arxiv.org/html/2610.12744#bib.bib25)\)addresses related asymmetric optimization dynamics between the two LoRA factors\. Whether gradient attribution is beneficial under non\-zero initialization remains an open question\.

### 3\.3Density\-Matched Mask Construction

Let𝒮\\mathcal\{S\}denote the candidate set of LoRA coordinates from which a mask is selected\. For gradient attribution, each coordinatei∈𝒮i\\in\\mathcal\{S\}is scored by its mean absolute gradient over the held\-out calibration set𝒟\\mathcal\{D\}:

gi=𝔼x∼𝒟​\[\|∂ℒ⁡\(x\)∂wi\|\]\.g\_\{i\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\left\[\\left\|\\frac\{\\partial\\mathcal\{L\}\(x\)\}\{\\partial w\_\{i\}\}\\right\|\\right\]\.\(6\)Given a selection budgetkk, the binary mask activates thekkhighest\-scoring coordinates:

mi=𝟏\[i∈TopK\(\{gj\}j∈𝒮,k\)\]\.m\_\{i\}=\\mathbf\{1\}\\left\[i\\in\\operatorname\{TopK\}\\left\(\\\{g\_\{j\}\\\}\_\{j\\in\\mathcal\{S\}\},\\,k\\right\)\\right\]\.\(7\)
For uniform random selection, we instead draw independent scores

ui∼Uniform⁡\(0,1\),i∈𝒮,u\_\{i\}\\sim\\mathrm\{Uniform\}\(0,1\),\\qquad i\\in\\mathcal\{S\},\(8\)and apply the same top\-kkoperator in Equation[7](https://arxiv.org/html/2610.12744#S3.E7)withuiu\_\{i\}in place ofgig\_\{i\}\. Thus, once the candidate pool𝒮\\mathcal\{S\}and budgetkkare fixed, the two methods differ only in the rule used to rank coordinates\.

#### Density\-matched comparison\.

Following the pool confound established in §[3\.2](https://arxiv.org/html/2610.12744#S3.SS2), our primary comparison restricts both selection rules to thelora\_Bcoordinate pool, denoted byℬ\\mathcal\{B\}\. Both headline arms activate exactly 10% of this pool:

kD1=1,651,508\.k\_\{\\mathrm\{D1\}\}=1\{,\}651\{,\}508\.\(9\)*Grad\-B*applies Equation[6](https://arxiv.org/html/2610.12744#S3.E6)withinℬ\\mathcal\{B\}and selects the topkD1k\_\{\\mathrm\{D1\}\}coordinates, whereas*Random\-B*selects the same number of coordinates uniformly at random fromℬ\\mathcal\{B\}\. The two arms therefore have the same candidate pool and the same number of trainable coordinates; the selection criterion is the only varying factor\.

#### Reference and control arms\.

For comparison with the conventional design,*Grad\-Full*applies gradient attribution with the full LoRA coordinate set𝒜∪ℬ\\mathcal\{A\}\\cup\\mathcal\{B\}as its nominal candidate pool and uses a 10% full\-pool budget of 2,993,357 coordinates\. Under standard zero initialization, its selected support nevertheless lies entirely withinℬ\\mathcal\{B\}by §[3\.2](https://arxiv.org/html/2610.12744#S3.SS2); because its budget is larger thankD1k\_\{\\mathrm\{D1\}\}, Grad\-Full is pool\-matched to Random\-B in effective support but not density\-matched\.*Random A\+B*draws the same 10% full\-pool budget uniformly from𝒜∪ℬ\\mathcal\{A\}\\cup\\mathcal\{B\}and therefore constitutes the conventional pool\-unmatched random baseline\.

We additionally use two auxiliary controls where relevant\.*No mask*is standard LoRA with all adapter coordinates updated\.*Uniform\-B*uses the samelora\_Bbudget as the matched comparison while distributing selections uniformly across LoRA modules; it is used only as a module\-concentration control in §[6\.2](https://arxiv.org/html/2610.12744#S6.SS2)\.

Algorithm[1](https://arxiv.org/html/2610.12744#alg1)summarizes mask construction and sequential training\.

## 4Experimental Setup

### 4\.1Models, Benchmarks & Training Details

Qwen2\.5\-3B\([Team, 2024](https://arxiv.org/html/2610.12744#bib.bib13)\)in 4\-bit NF4 \(bitsandbytes\), LoRA rank 16 \(α\\alpha=32\), AdamW \(lr2×10−42\\times 10^\{\-4\}, warmup 0\.03\), 1 epoch, 500 train samples per task, batch size 1, max length 512; calibration uses 64 Alpaca samples\. All methods operate on 29\.9M trainable LoRA parameters \(0\.96% of 3\.1B\); the full\-pool top\-10% selects2,993,3572\{,\}993\{,\}357, falling entirely withinlora\_B\(B=0B=0\) and activating∼\\sim20% of B, against 10% for Random\-B\. The comparison is matched on pool but not density; §[5\.2](https://arxiv.org/html/2610.12744#S5.SS2)equalizes it\.

Five domains are used in a fixed order: Code \(sahil2801/CodeAlpaca\-20k\), Medical \(GBaker/MedQA\-USMLE\-4\-options\), Legal \(lex\_glue, casehold subset\), Math \(openai/gsm8k\), Creative \(ChaoticNeutrals/Creative\_Writing\-ShareGPT, 6\.6K instruction\-response pairs\)\. Calibration: tatsu\-lab/alpaca \(64 samples\); 100 eval samples per task\.

### 4\.2Evaluation Metrics

We use Loss\-based BWT in preference to accuracy\-based BWT because loss is continuous and more sensitive to the small differences between masking strategies\. All reported forgetting metrics are loss\-based, reported as mean±\\pmstd over seven seeds \(42, 123, 456, 789, 1024, 2048, 4096\); part of the runs used a second GPU\. The accuracy side of the same comparison, together with the option log\-likelihood scorer and its scope, is reported separately in §[5\.4](https://arxiv.org/html/2610.12744#S5.SS4)\.

Figure 2:Per\-seed within\-lora\_Bdifferences \(Δ=Random\-B−Grad\-Full\\Delta=\\text\{Random\-B\}\-\\text\{Grad\-Full\}, loss\-based BWT, 7 seeds\)\. All seven are negative \(Random\-B forgets less\), meanΔ=−0\.072\\Delta=\-0\.072\(orange line\); seed 42 is the largest effect \(−0\.165\-0\.165\)\. Leave\-one\-out excluding seed 42 remains significant \(pairedt⁡\(5\)t\(5\),p=0\.015p=0\.015\)\.Table 2:Headline result \(D1, density\-matched\)\.Both arms select the same 1,651,508 parameters \(10% oflora\_B\); only the criterion differs\. Per\-seed loss\-based BWT andΔ=Random\-B−Grad\-B\\Delta=\\text\{Random\-B\}\-\\text\{Grad\-B\}\(negative = random\-B forgets less\), seven sessions, one seed each\. The last column is the density\-*unmatched*contrast from the same sessions, that is, Random\-B against the unrestricted gradient mask \(Δ=Random\-B−Grad\-Full\\Delta=\\text\{Random\-B\}\-\\text\{Grad\-Full\}\) rather than against Grad\-B\.

## 5Experiments and Main Results

### 5\.1Confounded and Density\-Matched Comparisons

A first comparison follows the usual design, in which each method draws from its own pool\. Under that design gradient attribution shows no advantage: its loss\-based BWT is0\.437±0\.0610\.437\\pm 0\.061against0\.428±0\.0190\.428\\pm 0\.019for a shared random mask over five seeds, a seed\-dependent difference that is not significant \(Table[5](https://arxiv.org/html/2610.12744#A4.T5), Appendix[D](https://arxiv.org/html/2610.12744#A4)\)\. Both methods still reduce forgetting relative to an unmasked adapter, but the comparison is uninformative about the selection rule, for the reason set out in §[3\.2](https://arxiv.org/html/2610.12744#S3.SS2): gradient attribution selects onlylora\_Bwhile the random baseline draws from both pools, so the design varies pool and criterion together\. The result is reported for completeness and is distinct from the seven\-seed comparison that follows, the two coming from different sessions whose absolute levels are not comparable \(Appendix[F](https://arxiv.org/html/2610.12744#A6)\)\.

Holding the pool fixed changes the picture\. We introduce a Random\-B control drawn fromlora\_Balone, matching the gradient mask’s pool, and compare it against a gradient mask restricted to the same pool\. Each seed is an independent session, so absolute BWT values drift between runs \(Appendix[F](https://arxiv.org/html/2610.12744#A6)\) and only within\-session paired observations are used\. This comparison is matched on pool but not on density, since the gradient mask activates nearly twice as manylora\_Bcoordinates as Random\-B\. Even so, the random mask achieves the lower loss\-based BWT in all seven seeds, with a mean difference of−0\.072\-0\.072that survives dropping the largest\-effect seed \(two\-sided pairedt⁡\(5\)=−3\.62t\(5\)=\-3\.62,p=0\.015p=0\.015; Figure[2](https://arxiv.org/html/2610.12744#S4.F2), Appendix[E](https://arxiv.org/html/2610.12744#A5)\)\. This value is the direct recomputation on the six retained paired contrasts and corrects thep=0\.029p=0\.029value reported in an earlier draft\. That the direction is unchanged when the pool is fixed is the first indication that the criterion, rather than the pool, is what the earlier comparison was measuring\.

### 5\.2Density\-Matched Control

Pool and criterion are separated cleanly only when density is matched as well\. The comparison above leaves the gradient arm activating about 20% of the pool against 10% for Random\-B, so the two arms still differ in how many coordinates they train\. D1 removes that difference, with both arms drawing exactly 10% oflora\_Band only the ranking rule differing\. At matched density the random mask again achieves the lower loss\-based BWT in every seed, by a larger margin than in the density\-unmatched comparison \(Table[2](https://arxiv.org/html/2610.12744#S4.T2)\)\. Under a common \[Grad\-B−\-Random\-B\] convention, the mean contrast is\+0\.155\+0\.155on the V100 stack and\+0\.152\+0\.152on the A800 stack\. Thus the contrast reproduces at a similar magnitude in a second hardware/software environment; in a separate three\-seed sensitivity check, reversing the execution order of the two arms does not reverse the Random\-B–Grad\-B contrast \(Appendix[F](https://arxiv.org/html/2610.12744#A6)\)\.

### 5\.3Task\-Level Decomposition

Figure 3:The density\-matched D1 per\-task decomposition, averaged over seven seeds\. For each prior task, the bars show the loss\-based BWT contribution of Random\-B and Grad\-B, with the corresponding values reported above each pair\. Because the task\-level contributions span roughly two orders of magnitude, the left panel uses a brokenyy\-axis: the slashes mark the omitted interval from 0\.52 to 0\.92, and the labels above the Medical bars report their full values \(\+0\.998\+0\.998and\+1\.493\+1\.493\) rather than truncated heights\. Creative is the final task and therefore has no backward\-transfer contribution\. The right panel shows the aggregate loss\-based BWT on a separate scale\. Random\-B exhibits less loss\-based forgetting on Code, Medical, and Math; the Legal difference is negligible and changes sign across seeds\. Most of the aggregate difference is concentrated on Medical\. Per\-seed results are reported in Appendix[Q](https://arxiv.org/html/2610.12744#A17)\.The aggregate gain is not spread evenly across the five tasks, and where it concentrates bears on how the effect should be read\. Medical carries about four fifths of it, while code and math contribute the remainder in the same direction; on legal the arms are indistinguishable\. Two explanations can be assessed from the same data\. First, the gradient arm’s loss on each task’s own diagonal is lower than the random arm’s in every seed and task, yet it subsequently forgets more, so the two arms differ in retention rather than initial learning capacity\. The same seven within\-session pairs also show lower mean final held\-out loss over the four prior tasks for Random\-B: the Grad\-B minus Random\-B endpoint difference is\+0\.0813\+0\.0813\(pairedt⁡\(6\)=3\.03t\(6\)=3\.03,p=0\.023p=0\.023; 6/7 seeds\)\. This endpoint result is less robust to leave\-one\-out deletion than the headline BWT contrast, but it shows that the effect is not merely a BWT normalization artifact \(Appendix[R](https://arxiv.org/html/2610.12744#A18)\)\. By contrast, restoring the gradient arm’s full budget produces seed\-dependent results: at seed 42 the density\-matched gradient arm outperforms the unrestricted one, whereas the ordering reverses at the seven\-seed mean\. It therefore does not provide a stable explanation of the headline effect\.

### 5\.4Accuracy\-Based Validation

Figure 4:Accuracy\-based validation\. Blue points are the seven per\-seed accuracy\-BWT differences \(Grad\-B minus Random\-B, percentage points\) on the two discrete\-option tasks, spanning−6\.98\-6\.98to\+6\.00\+6\.00pp; green is the seven\-seed mean with its 95% interval\[−2\.91,\+4\.80\]\[\-2\.91,\+4\.80\]pp\. The plotted quantity is a difference of accuracy\-BWT \(mean final\-minus\-learned accuracy over prior tasks\), a retention measure, not a difference of final accuracies\. The interval spans zero, so no statistically reliable accuracy difference is detected, and it reaches past the 4 pp non\-inferiority margin, so that criterion is not met\. Per\-seed values are in Appendix[N](https://arxiv.org/html/2610.12744#A14)\.Loss\-based BWT is a proxy for forgetting rather than the quantity of ultimate interest, so we evaluate the same masking comparison on task accuracy and ask whether the accuracy ordering agrees with the loss ordering\. The evaluation reuses the primary protocol and the corrected task evaluators over the same seven seeds and 100 evaluation items per task, retraining from the base model on an A800 rather than re\-scoring a checkpoint\. Accuracy is scored only for the two tasks with discrete options \(Medical, MedQA four options; Legal, CaseHOLD\), so every quantity below concerns those two\.

Across seven seeds, the accuracy\-BWT difference averages\+0\.94\+0\.94percentage points with a 95% interval of\[−2\.91,\+4\.80\]\[\-2\.91,\+4\.80\]pp that spans zero\. Here accuracy\-BWT is the mean final\-minus\-learned accuracy over the prior discrete\-option tasks, so it measures retention on the same footing as loss\-based BWT and is not a difference of final accuracies\. The individual seeds range from−6\.98\-6\.98to\+6\.00\+6\.00pp, and it is that spread rather than a large mean that makes the interval wide atn=7n=7\. A 4 pp non\-inferiority margin was specified in advance, and the measured upper bound exceeds it, so non\-inferiority is not established\.

On the loss side, the same masks are separated: Grad\-B forgets more in all seven seeds by a mean of\+0\.152\+0\.152on the A800 loss runs\. This agrees in mean to within0\.0030\.003with the V100 estimate after aligning sign conventions \(\+0\.155\+0\.155under \[Grad−\-Random\]\)\. Loss therefore resolves a difference the accuracy measurement does not; the accuracy interval bounds that difference rather than contradicting it\.

Per\-seed values and the interval construction are reported in Appendix[N](https://arxiv.org/html/2610.12744#A14)\. Math is excluded from the accuracy read\-out because its generation\-based score is confounded by truncation \(Appendix[O](https://arxiv.org/html/2610.12744#A15)\)\.

## 6Analysis and Extensions

### 6\.1Sparsity, Scale, and Cross\-Architecture Generalization

The headline comparison fixes the sparsity budget at 10%\. Sweeping it from 5% to 30% at a single seed, no tested level shows gradient attribution outperforming a shared random mask \(Figure[6](https://arxiv.org/html/2610.12744#A8.F6), Appendix[H](https://arxiv.org/html/2610.12744#A8)\)\. The sweep is preliminary and inherits the pool confound rather than isolating the budget, so we report it as a boundary check on the direction of the effect rather than as evidence for the sparsity\-constraint view\.

No single\-seed probe reverses the conclusion, but a reversed task order shifts BWT by more than any method difference, which is why every conclusion in this paper is scoped to the tested sequence \(Table[13](https://arxiv.org/html/2610.12744#A11.T13), Appendices[H](https://arxiv.org/html/2610.12744#A8)–[K](https://arxiv.org/html/2610.12744#A11.SS0.SSS0.Px2)\)\. On Phi\-2, an architecturally distinct family, we additionally run a clean density\-matched D1 replication: both arms select exactly 10% of the samelora\_Bpool and therefore differ only in the ranking rule\. Random\-B achieves lower loss\-based BWT in all four seeds, with meanΔ=Random\-B−Grad\-B=−0\.1152\\Delta=\\text\{Random\-B\}\-\\text\{Grad\-B\}=\-0\.1152\(SD0\.01730\.0173\), compared with−0\.1552\-0\.1552on Qwen2\.5\-3B\. The task\-level decomposition also reproduces the primary pattern: Medical contributes 67\.2% of the Phi\-2 aggregate contrast and is the largest contributor in every seed, versus 79\.9% on Qwen2\.5\-3B\. Because the replication has only four seeds, we treat its parametric test as supporting evidence and base the cross\-architecture claim primarily on the matched design and 4/4 directional replication; full per\-seed results and the sign\-test caveat are in Appendix[S](https://arxiv.org/html/2610.12744#A19)\.

### 6\.2Module Concentration and Calibration Robustness

The gradient mask does not select uniformly across module types: withinlora\_Bit concentrates on output projections and to a lesser degree onup\_proj, together accounting for a little over half of all selections, while down projections receive well below the uniform rate\. The pattern is consistent across calibration sources and peaks at middle layers \(Appendix[J](https://arxiv.org/html/2610.12744#A10)\)\. That concentration invites the reading that it causes the gradient mask’s higher BWT, and theuniform\_Bcontroller tests it by spreading the same budget across modules\. The controller contradicts the reading: three of four seeds land on the gradient side rather than the random side\. Because those offsets are taken against different session baselines \(Appendix[F](https://arxiv.org/html/2610.12744#A6)\), we treat the ablation as failing to establish the mechanism rather than as evidence against the concentration itself, and neither the within\-lora\_Bfinding nor its density\-matched control depends on it \(Figure[7](https://arxiv.org/html/2610.12744#A10.F7)\)\. The result is also insensitive to the calibration domain: pooling in\-domain rather than out\-of\-domain calibration changes 42% of the selected parameters, yet the loss\-based BWT moves by less than a tenth of the between\-seed spread \(Appendix[L](https://arxiv.org/html/2610.12744#A12)\)\.

## 7Conclusion

At equal parameter count within thelora\_Bpool, the random mask achieves lower loss\-based BWT in every seed, an advantage that reproduces in mean and direction in a second hardware/software environment and is not an artifact of density\. The same paired runs also show lower mean final held\-out loss on the prior tasks for Random\-B \(Grad\-B minus Random\-B=\+0\.0813=\+0\.0813,p=0\.023p=0\.023\), although this endpoint result is less leave\-one\-out robust than the BWT contrast\. On the loss\-based evidence, the attribution pass is not justified: removing it eliminates a one\-time 64\-sample calibration forward–backward pass \(64 steps\), equal to 12\.8% of one 500\-step task\-training budget, and removes the calibration\-data requirement\. Accuracy\-BWT remains inconclusive at seven seeds, with the pre\-specified non\-inferiority criterion unmet\. All conclusions are scoped to the tested task order and static masks; a per\-task adaptive mask and a better\-powered accuracy evaluation are the most informative next tests\.

#### Limitations\.

The within\-lora\_Bfinding is limited to static masks, a single primary task order, and rank\-16 LoRA under standard zero initialization\. Our primary conclusions are based on loss\-based BWT, with accuracy validation limited to the two discrete\-option tasks\. Absolute BWT also varies across sessions, so all statistical claims rely on within\-session paired comparisons \(Appendix[F](https://arxiv.org/html/2610.12744#A6)\)\.

## Reproducibility Statement

Experiments use Python 3\.12, PyTorch 2\.8\.0\+cu128 \(4\-bit NF4 with bf16 compute\), Transformers 5\.16\.1, PEFT 0\.20\.0, bitsandbytes 0\.50\.2, datasets 5\.0\.1, and CUDA 12\.8\. The primary seven\-seed within\-lora\_Bcomparison, scale runs, and ablations use a single NVIDIA A800 80GB PCIe GPU\. The D1 density\-matched control and the M2 uniform\-B control ran in a separate session on a Tesla V100\-PCIE\-32GB under an earlier environment \(Python 3\.12\.3, PyTorch 2\.3\.0\+cu121, Transformers 4\.46\.3, PEFT 0\.13\.2, bitsandbytes 0\.43\.3\); as both are within\-run paired comparisons, the hardware difference does not affect their paired results\. Primary results use Qwen2\.5\-3B \(seven seeds\); cross\-architecture within\-B results use Phi\-2 2\.78B \(four seeds\); the confounded Phi\-2 comparison uses five seeds under a matched 100\-eval\-sample protocol \(an additional seed\-42 observation used 20 eval samples and is reported separately\); scale observations use Qwen2\.5\-0\.5B/1\.5B/7B \(single seed\)\. LoRA is applied to all linear projection layers\.

Model training uses AdamW \(β1=0\.9\\beta\_\{1\}\{=\}0\.9,β2=0\.999\\beta\_\{2\}\{=\}0\.999, weight decay 0\.01\) at learning rate2×10−42\\times 10^\{\-4\}with a linear warmup of 0\.03 and cosine annealing over the per\-task step budget; LoRA dropout 0\.05, rank 16,α\\alpha=32, batch size 1, max length 512, one epoch per task, and top\-kk0\.1 masking\. Five sequential tasks are used: Code \(code\_alpaca\_20k\), Medical \(medqa, 4\-option MC\), Legal \(lexglue/case\_hold\), Math \(gsm8k\), and Creative Writing \(Creative\_Writing\-ShareGPT\)\. Each task uses 500 training and 100 evaluation samples; gradient masks use 64 out\-of\-domain Alpaca calibration samples\. Seeds are 42, 123, 456, 789, 1024, 2048, and 4096; approximately 60 GPU\-hours were used in total\.

Boundary analyses \(rank sweep, task order reversal, scale observations\) are single\-seed; a method\-*order*check over three seeds is reported in Appendix[F](https://arxiv.org/html/2610.12744#A6)\. An isolated\-process protocol equivalence check confirms that running each method in a separate process reproduces the multi\-method BWT to within9\.6×10−59\.6\\times 10^\{\-5\}\(10×\\timesmargin below the pre\-specified10−310^\{\-3\}threshold\)\. All reported forgetting metrics are loss\-based BWT\. Code and configuration files are released as anonymized supplementary material \(code\_anonymous\.zip\): training and attribution scripts, five\-task data preparation, mask generation, the analysis scripts computing BWT and the paired statistics, and pinned dependency manifests for both environments\.

## Ethics Statement

This work studies parameter masking for continual learning and does not involve human subjects or sensitive data\. It does not introduce new capabilities or datasets that raise ethical concerns\. On data sources and licensing: the Code task uses CodeAlpaca\-20k \(CC BY 4\.0\), the Math task uses GSM8K \(MIT\), and the Medical and Legal tasks use MedQA and CaseHOLD, which are distributed on Hugging Face with no affirmative license declaration, so we use them for research evaluation only\. The Creative task uses a synthetic creative\-writing set composed of model\-generated outputs rather than user conversations\. The Alpaca data used to calibrate the gradient masks is released under CC BY\-NC 4\.0, a non\-commercial license; we therefore make no commercial\-use claim for models trained with it\. The finding that random selection outperforms gradient attribution within LoRA’s B\-matrix may reduce computational cost in continual\-learning deployments by removing the need for gradient\-based parameter attribution\. We do not foresee negative societal consequences specific to this contribution\.

## AI Use Disclosure

AI coding assistants were used for experiment\-script development and debugging, and AI assistance aided literature search and manuscript drafting and revision\. The research design, experimental methodology, statistical analysis, and scientific conclusions are the authors’ own work\. All reported results were produced by the described experimental pipeline without AI\-generated synthetic data; all analyses and claims were verified by the authors\.

## References

- Adewuyiet al\.\(2026\)I\. Adewuyi, S\. Okibe, and V\. IvanovThe multiple ticket hypothesis: random sparse subnetworks suffice for RLVR\.arXiv preprint arXiv:2602\.01599\.Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px2.p1.1)\.
- Aghajanyanet al\.\(2021\)A\. Aghajanyan, S\. Gupta, and L\. ZettlemoyerIntrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,External Links:[Link](https://arxiv.org/abs/2012.13255)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- Aljundiet al\.\(2018\)R\. Aljundi, F\. Babiloni, M\. Elhoseiny, M\. Rohrbach, and T\. TuytelaarsMemory aware synapses: learning what \(not\) to forget\.InProceedings of the European Conference on Computer Vision,pp\. 139–154\.External Links:[Link](https://openaccess.thecvf.com/content_ECCV_2018/html/Rahaf_Aljundi_Memory_Aware_Synapses_ECCV_2018_paper.html)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.2.1.1.1)\.
- Bidermanet al\.\(2024\)D\. Biderman, J\. Portes, J\. J\. Gonzalez Ortiz, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle, C\. Blakeney, and J\. P\. CunninghamLoRA learns less and forgets less\.Transactions on Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2405.09673)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- Chenget al\.\(2026\)Z\. Cheng, L\. Lai, Y\. Liu, and Y\. SunToward sustainable on\-device intelligence: a survey on energy\-efficient rag systems with small language models\.Available at SSRN 6698538\.Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px3.p1.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: efficient finetuning of quantized LLMs\.arXiv preprint arXiv:2305\.14314\.External Links:[Link](https://arxiv.org/abs/2305.14314)Cited by:[§1](https://arxiv.org/html/2610.12744#S1.p1.1)\.
- Dieudonnéet al\.\(2026\)T\. Dieudonné, G\. Lanzillotta, E\. Simsar, L\. Barinka, and T\. HofmannWhen one adapter speaks for many: discovering low\-rank redundancy in continual fine\-tuning\.arXiv preprint arXiv:2606\.28117\.Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px2.p1.1)\.
- Farajtabaret al\.\(2020\)M\. Farajtabar, N\. Azizan, A\. Mott, and A\. LiOrthogonal gradient descent for continual learning\.InProceedings of the 23rd International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.108,pp\. 3762–3773\.External Links:[Link](https://arxiv.org/abs/1910.07104)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Frankle and Carbin \(2019\)J\. Frankle and M\. CarbinThe lottery ticket hypothesis: finding sparse, trainable neural networks\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1803.03635)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- French \(1999\)R\. M\. FrenchCatastrophic forgetting in connectionist networks\.Trends in Cognitive Sciences3\(4\),pp\. 128–135\.Cited by:[§1](https://arxiv.org/html/2610.12744#S1.p1.1),[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuLoRA\+: efficient low rank adaptation of large models\.arXiv preprint arXiv:2402\.12354\.External Links:[Link](https://arxiv.org/abs/2402.12354)Cited by:[§3\.2](https://arxiv.org/html/2610.12744#S3.SS2.p4.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§1](https://arxiv.org/html/2610.12744#S1.p1.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska, D\. Hassabis, C\. Clopath, D\. Kumaran, and R\. HadsellOvercoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences114\(13\),pp\. 3521–3526\.External Links:[Document](https://dx.doi.org/10.1073/pnas.1611835114),[Link](https://arxiv.org/abs/1612.00796)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.2.1.1.1),[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Leeet al\.\(2019\)N\. Lee, T\. Ajanthan, and P\. H\. S\. TorrSNIP: single\-shot network pruning based on connection sensitivity\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1810.02340)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.3.1.1.1),[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Liang and Li \(2024\)Y\. Liang and W\. LiInfLoRA: interference\-free low\-rank adaptation for continual learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23638–23647\.External Links:[Link](https://arxiv.org/abs/2404.00228)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2026a\)M\. Liu, Y\. Sun, R\. Sun, X\. Dong, X\. Shen, H\. Wang, H\. Xiong, and Y\. SongIPS: in\-prompt process supervision for short video content moderation\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 1277–1288\.External Links:[Link](https://aclanthology.org/2026.acl-industry.89/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-industry.89),ISBN 979\-8\-89176\-394\-4Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2026b\)Y\. Liu, Y\. Wan, R\. Xu, X\. Zhang, S\. Xie, and H\. XiongAttribution\-guided continual learning for large language models\.arXiv preprint arXiv:2605\.05285\.Cited by:[§1](https://arxiv.org/html/2610.12744#S1.p2.1),[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.12744#S3.SS2.p3.1)\.
- McCloskey and Cohen \(1989\)M\. McCloskey and N\. J\. CohenCatastrophic interference in connectionist networks: the sequential learning problem\.InPsychology of Learning and Motivation,Cited by:[§1](https://arxiv.org/html/2610.12744#S1.p1.1),[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 121038–121072\.External Links:[Link](https://arxiv.org/abs/2404.02948)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px1.p1.1),[§3\.2](https://arxiv.org/html/2610.12744#S3.SS2.p4.1)\.
- Qiaoet al\.\(2025\)J\. Qiao, Z\. Zhang, X\. Tan, Y\. Qu, W\. Zhang, Z\. Han, and Y\. XieGradient projection for continual parameter\-efficient tuning\.IEEE Transactions on Pattern Analysis and Machine Intelligence47\(10\),pp\. 9316–9329\.Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Ramanujanet al\.\(2020\)V\. Ramanujan, M\. Wortsman, A\. Kembhavi, A\. Farhadi, and M\. RastegariWhat’s hidden in a randomly weighted neural network?\.In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition,External Links:[Link](https://openaccess.thecvf.com/content_CVPR_2020/html/Ramanujan_Whats_Hidden_in_a_Randomly_Weighted_Neural_Network_CVPR_2020_paper.html)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- Sahaet al\.\(2021\)G\. Saha, I\. Garg, and K\. RoyGradient projection memory for continual learning\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2103.09762)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Serràet al\.\(2018\)J\. Serrà, D\. Surís, M\. Miron, and A\. KaratzoglouOvercoming catastrophic forgetting with hard attention to the task\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 4548–4557\.External Links:[Link](https://arxiv.org/abs/1801.01423)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- Sunet al\.\(2024\)M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. KolterA simple and effective pruning approach for large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PxoFut3dWW)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.4.1.1.1),[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Sunet al\.\(2025\)Y\. Sun, Y\. Li, R\. Sun, C\. Liu, F\. Zhou, Z\. Jin, L\. Wang, X\. Shen, Z\. Hao, and H\. XiongAudio\-enhanced vision\-language modeling with latent space broadening for high quality data expansion\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2,KDD ’25,pp\. 4872–4881\.External Links:ISBN 9798400714542,[Link](https://doi.org/10.1145/3711896.3737195),[Document](https://dx.doi.org/10.1145/3711896.3737195)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px3.p1.1)\.
- Team \(2024\)Q\. TeamQwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.1](https://arxiv.org/html/2610.12744#S4.SS1.p1.1)\.
- Wanget al\.\(2020\)C\. Wang, G\. Zhang, and R\. GrossePicking winning tickets before training by preserving gradient flow\.External Links:[Link](https://arxiv.org/abs/2002.07376)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.3.1.1.1)\.
- Wanget al\.\(2025a\)C\. Wang, Y\. Lyu, Z\. Sun, and L\. JingContinual gradient low\-rank projection fine\-tuning for LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,External Links:[Link](https://aclanthology.org/2025.acl-long.721/)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. HuangO\-LoRA: orthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10658–10671\.External Links:[Link](https://arxiv.org/abs/2310.14152)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)Z\. Wang, Y\. Sun, H\. Wang, B\. Jing, X\. Shen, X\. Dong, Z\. Hao, H\. Xiong, and Y\. SongReasoning\-enhanced domain\-adaptive pretraining of multimodal large language models for short video content governance\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track,pp\. 1104–1112\.External Links:[Link](https://aclanthology.org/2025.emnlp-industry.77/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-industry.77),ISBN 979\-8\-89176\-333\-3Cited by:[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px3.p1.1)\.
- Xu and Zhang \(2024\)J\. Xu and J\. ZhangRandom masking finds winning tickets for parameter efficient fine\-tuning\.InProceedings of the 41st International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2405.02596)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.7.1.1.1),[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- Yuet al\.\(2024\)L\. Yu, B\. Yu, H\. Yu, F\. Huang, and Y\. LiLanguage models are Super Mario: absorbing abilities from homologous models as a free lunch\.InProceedings of the 41st International Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2311.03099)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p2.1)\.
- Zenkeet al\.\(2017\)F\. Zenke, B\. Poole, and S\. GanguliContinual learning through synaptic intelligence\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 3987–3995\.External Links:[Link](https://proceedings.mlr.press/v70/zenke17a.html)Cited by:[§2](https://arxiv.org/html/2610.12744#S2.p1.1)\.
- Zhanget al\.\(2023\)L\. Zhang, L\. Zhang, S\. Shi, X\. Chu, and B\. LiLoRA\-FA: efficient and effective low rank representation fine\-tuning\.arXiv preprint arXiv:2308\.03303\.External Links:[Link](https://arxiv.org/abs/2308.03303)Cited by:[Table 18](https://arxiv.org/html/2610.12744#A16.T18.8.5.1.1.1),[§2](https://arxiv.org/html/2610.12744#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AThe End\-to\-End Procedure

The notation is collected in Table[1](https://arxiv.org/html/2610.12744#S3.T1)in the Methodology\. Figure[1](https://arxiv.org/html/2610.12744#S1.F1)\(a\) shows the same procedure as a diagram; Algorithm[1](https://arxiv.org/html/2610.12744#alg1)states it step by step\.

## Appendix BAttribution Cost

Algorithm 1Masked continual fine\-tuning with a frozen selection mask1:base model

fθf\_\{\\theta\}with frozen

W0W\_\{0\}; task stream

\{\(𝒟t\)\}t=1T\\\{\(\\mathcal\{D\}\_\{t\}\)\\\}\_\{t=1\}^\{T\}; calibration set

𝒟\\mathcal\{D\}; budget

kk; rule

∈\{gradient,uniform\}\\in\\\{\\text\{gradient\},\\text\{uniform\}\\\}
2:frozen mask

mm; adapted adapter

W=W0\+α​B​AW=W\_\{0\}\+\\alpha BA
3:

𝒮←\\mathcal\{S\}\\leftarrowthe

\(A,B\)\(A,B\)LoRA coordinates⊳\\trianglerighttrainable coordinates

4:ifrule

==gradientthen

5:foreach coordinate

i∈𝒮i\\in\\mathcal\{S\}do

6:

gi←𝔼x∼𝒟​\[\|∂ℒ⁡\(x\)/∂wi\|\]g\_\{i\}\\leftarrow\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\}\\left\[\\left\|\\partial\\mathcal\{L\}\(x\)/\\partial w\_\{i\}\\right\|\\right\]⊳\\trianglerightEq\. equation[6](https://arxiv.org/html/2610.12744#S3.E6)

7:endfor

8:else

9:

gi←Uniform⁡\(0,1\)g\_\{i\}\\leftarrow\\mathrm\{Uniform\}\(0,1\)for each

i∈𝒮i\\in\\mathcal\{S\}
10:endif

11:

m←𝟙\[gi≥τk\]m\\leftarrow\\mathbb\{1\}\[\\,g\_\{i\}\\geq\\tau\_\{k\}\\,\]⊳\\trianglerighttop\-kkrule, Eq\. equation[7](https://arxiv.org/html/2610.12744#S3.E7)

12:freeze

mm⊳\\trianglerightshared by all tasks; never updated

13:

W←W0W\\leftarrow W\_\{0\}⊳\\trianglerightstandard zero\-init LoRA,B=0B=0

14:for

t=1t=1to

TTdo

15:train the coordinates with

mi=1m\_\{i\}=1on

𝒟t\\mathcal\{D\}\_\{t\}, updating only those

16:evaluate

ℒj\(t\)\\mathcal\{L\}\_\{j\}^\{\(t\)\}for all

j≤tj\\leq t⊳\\trianglerightevery task seen so far

17:endfor

18:return

mm, and

BWT\\mathrm\{BWT\}from Eq\. equation[2](https://arxiv.org/html/2610.12744#S3.E2)over

𝒫=\{1,…,T−1\}\\mathcal\{P\}=\\\{1,\\dots,T\-1\\\}

Table 3:Attribution cost on Qwen2\.5\-3B \(BF16\)\. Both columns are from a single*full\-parameter*measurement run \(\-\-include\-frozen,\-\-full\-precision\); the exact correspondence and its limits are described below\. Peak GPU istorch\.cuda\.max\_memory\_allocatedconverted to decimal GB \(profiling fields\*\_peak\_alloc\_mb= 13168 / 7102 MiB, i\.e\. 13\.8 / 7\.4 GB\); Time is the per\-task attribution wall\-clock recorded by the run \(584\.9 s gradient / 12\.1 s Wanda\)\.†*Full\-parameter*attribution configuration \(\-\-include\-frozen, scoring all 3,085,938,688 base parameters\); no method in this paper uses this configuration\. The attribution used in the continual\-learning experiments records a∼\\sim0\.73 GB transient allocation above the resident base model during its separate calibration pass \(4\-bit NF4, seq 512, 64 calibration samples, gradient checkpointing enabled;overhead\_mb= 699\.43 MiB\)\. This is not an increase over the ordinary training backward peak; the removable costs are the additional calibration pass, its wall\-clock compute, and its data requirement\. ‡Zero by construction because no separate attribution pass is run, not a measured end\-to\-end peak\-memory saving\.

Both measured columns of Table[3](https://arxiv.org/html/2610.12744#A2.T3)come from a single full\-parameter attribution profiling run \(\-\-include\-frozen,\-\-full\-precision\), so they are internally consistent; that run scores all 3,085,938,688 base\-model parameters, a configuration no reported method uses\. We describe the memory figure first, then the time figure\.

Gradient attribution requires 13\.8 GB peak GPU memory for Qwen2\.5\-3B \(max\_memory\_allocated, 13168 MiB→\\to13\.8 GB in decimal units\)\. This figure is from the full\-parameter attribution configuration, which scores all 3,085,938,688 parameters of the base model\. The peak comprises four components \(the resident base\-model weights, the forward and backward activation graph over the full model, the full\-parameter gradient accumulator, and the CUDA context\), but these were not separately instrumented in the surviving run record, so we report the total rather than a measured decomposition\. The backward pass does traverse the base model: attribution computes∂ℒ/∂w\\partial\\mathcal\{L\}/\\partial wover calibration samples, so both the forward and backward graphs span the full parameter space rather than the 29\.9M\-parameter LoRA surface\. This configuration peaks at 13\.8 GB, of which∼\\sim90% is resident base\-model weights \(6\.23 GB in BF16\) plus the full\-model gradient buffer \(6\.23 GB\); these are costs the CL configuration does not pay\. It is therefore an upper bound on a configuration no method uses, not a measure of attribution’s own overhead\. The CL attribution pass records a∼\\sim0\.73 GB transient allocation above the resident base model, but ordinary training also performs backward passes; this is not an incremental end\-to\-end peak\-memory cost\. The random baseline removes the separate calibration pass and its time/data costs, not the training backward peak \(Table[5](https://arxiv.org/html/2610.12744#A4.T5)\)\.

The 584\.9 s per\-task figure is the wall\-clock of that same full\-parameter run \(full\_alora\_time\_s= 584\.9 s\)\. As a checkpointing\-off contrast, an earlier LoRA\-only attribution profiling run \(4\-bit, seq 1024, 256 calibration samples, gradient checkpointing disabled\) reports 3,512 MiB of overhead, a different regime \(checkpointing off, longer sequences\) that should not be combined with the 0\.73 GB figure above\.

## Appendix CPruning Saliency Is Orthogonal to Gradient Importance

![Refer to caption](https://arxiv.org/html/2610.12744v1/fig2_spearman.png)Figure 5:Spearmanρ=0\.086\\rho=0\.086between Wanda and gradient importance \(LoRA parameters\) on Qwen2\.5\-3B\. These metrics rank parameters nearly independently\.Spearman correlation between Wanda \(\|W\|×RMS​\(X\)\|W\|\\times\\text\{RMS\}\(X\)\) and gradient importance \(\|∂ℒ/∂w\|\|\\partial\\mathcal\{L\}/\\partial w\|\) across 2M sampled parameters: \(Figure[5](https://arxiv.org/html/2610.12744#A3.F5);ρ=0\.086\\rho=0\.086, 95% CI≈\[0\.084,0\.088\]\\approx\[0\.084,0\.088\],n=2n\{=\}2M; correlation\-only, measured outside a continual\-learning context\)\.

## Appendix DConfounded Null\-Result Statistics

Table 4:Per\-seed BWT breakdown \(Qwen2\.5\-3B, rank\-16 LoRA, 5 tasks\)\.Table 5:Confounded comparison \(Qwen2\.5\-3B, rank\-16 LoRA, 5 tasks,n=5n\{=\}5seeds\)\. The last column records whether an additional calibration forward–backward pass is required; its time and data costs are described in Appendix[B](https://arxiv.org/html/2610.12744#A2)\. Across all tables, bold marks the better value in its column\.A pairedtt\-test on per\-seed BWT \(Table[4](https://arxiv.org/html/2610.12744#A4.T4); random vs\. gradient: seeds 42/123/456/789/1024 yield differences\+0\.056\+0\.056,\+0\.046\+0\.046,−0\.026\-0\.026,−0\.059\-0\.059,\+0\.028\+0\.028\) givest⁡\(4\)=0\.41t\(4\)=0\.41,p=0\.70p=0\.70\(two\-tailed\), Cohen’sdz=0\.18d\_\{z\}=0\.18\(small effect\)\. The direction is seed\-dependent, consistent with no systematic advantage for either method\. Normalizing BWT per task by zero\-shot loss \(Δ​L/L0\\Delta L/L\_\{0\}\) does not change the method ranking\. Tables[5](https://arxiv.org/html/2610.12744#A4.T5)and[6](https://arxiv.org/html/2610.12744#A5.T6)report differentfull\_aloraBWT for the same seed because they come from independent runs; the absolute BWT level varies across sessions \(§[F](https://arxiv.org/html/2610.12744#A6)\)\.

## Appendix EWithin\-lora\_BComparison \(Density\-Unmatched\)

The within\-lora\_Bcomparison matches the parameter pool \(both arms draw fromlora\_B\) but*not*the density: the gradient mask activates approximately2×2\\timesthe B parameters of random\-B\-only\. It is therefore an intermediate, density\-unmatched step; the density\-matched control \(D1\) is the headline comparison in the main text\. All comparisons are within\-run \(same method ordering and RNG state\);Δ=random\_B\_only−Grad\-Full\\Delta=\\text\{random\\\_B\\\_only\}\-\\text\{Grad\-Full\}, negativeΔ\\Deltafavors random\-B\. Absolute BWT values carry session\-level variation and should not be compared across seeds \(§[F](https://arxiv.org/html/2610.12744#A6)\)\.

Table 6:Within\-lora\_Bcomparison \(Qwen2\.5\-3B, rank\-16 LoRA, 5 tasks, 7 seeds\)\. All comparisons are within\-run \(same method ordering and RNG state\)\.Δ=random\_B\_only−Grad\-Full\\Delta=\\text\{random\\\_B\\\_only\}\-\\text\{Grad\-Full\}; negativeΔ\\Deltaindicates random\-B achieves lower \(better\) BWT\.
## Appendix FVariation Across Runs and Method\-Order Variation

Three quantities in this paper are sometimes read as competing estimates of one noise level\. They are not: each measures a different thing at a different granularity, and the table below separates them\.

Table 7:Distinct quantities that are sometimes all called run\-to\-run*noise*\. The paired tests use the across\-seed SD of the within\-session contrasts; the protocol\-equivalence discrepancy is an implementation check and does not enter inference\. Position and cross\-session effects are reported as sensitivities rather than as interchangeable error estimates\.Table 8:Method\-ordering sensitivity \(D1 density\-matched, three seeds\)\.Δpos\\Delta\_\{\\text\{pos\}\}is the BWT difference between running at position 1 and position 2\. The last two columns express each arm’s positional shift as a fraction of the main D1 contrast \(\|Δ\|=0\.155\|\\Delta\|=0\.155\)\.#### \(a\) Same process, protocol split\.

Running each method in its own process reproduces the multi\-method BWT to within9\.6×10−59\.6\\times 10^\{\-5\}, an order of magnitude below the pre\-specified10−310^\{\-3\}threshold\. This is a check on the protocol implementation, not on run\-to\-run variation of the model: two code paths that should agree do agree\. The pairedtttests instead use the across\-seed standard deviation of the within\-session paired contrasts, SD\(Δs\)=0\.071\(\\Delta\_\{s\}\)=0\.071for D1\.

#### \(b\) Same session, arm order swapped\.

Within one D1 session the two arms run back to back, and reversing their order is a genuine perturbation of the run\. At seed 42 thegradient\_Barm’s BWT moves by 0\.064 between the two orders; at seeds 789 and 2048 both arms stay within1\.4×10−21\.4\\times 10^\{\-2\}\. This is a within\-session positional effect, and the within\-run contrast survives it at all three tested seeds\. Because this sensitivity analysis covers three rather than seven seeds, it does not establish the population distribution of the position effect\.

#### \(c\) Cross\-session, same seed\.

The absolute BWT of a fixed method and seed is not reproducible across sessions\. For seed 42,full\_alorarecords0\.4910\.491,0\.3780\.378, and0\.4980\.498in three separate runs \(spread0\.1200\.120\), whilerandom\_B\_onlyrecords0\.3260\.326,0\.3240\.324, and0\.2780\.278\(spread0\.0480\.048\)\. The spread is method\-dependent, so it does not act as a constant offset that cancels in a comparison\.

Method position can shift the absolute BWT of Gradient\-B, most notably at seed 42, but reversing the execution order does not reverse the Random\-B–Grad\-B contrast in any of the three tested seeds\. \(Table[8](https://arxiv.org/html/2610.12744#A6.T8)\)

Table 9:Absolute loss\-based BWT for the same seed 42 under identical method definitions, across sessions and hardware\. Run C is the D1 density\-matched session and Run D the uniform\-B suppression control \(the M2 controller\) session, both on a separate GPU and software stack;random\_A\+Bwas not run in either\. A Run\-D entry exists only for methods Run D ran, and it did not train a plainfull\_aloraarm, so that cell is\-\-\-and its gradient\-side arm isgradient\_B, entered on its own row\. Spread ismax−min\\max\-\\minacross the runs in which the method appears\.
#### Why this is not called NF4 non\-determinism\.

Quantization is the same in every run of Table[9](https://arxiv.org/html/2610.12744#A6.T9), so it cannot by itself explain a cross\-session spread while leaving a same\-process protocol split at9\.6×10−59\.6\\times 10^\{\-5\}\. The cross\-session spread is method\-specific rather than a common shift shared by all arms\. We therefore report it as a*method\-specific cross\-session offset of unknown origin*and do not attribute it to a mechanism\. Two consequences follow\. First, only within\-run comparisons are used for any claim in this paper\. Second, therandom\_B\_onlyvalue that enters Table[2](https://arxiv.org/html/2610.12744#S4.T2)is the D1 session’s own0\.2780\.278, taken from the same session as its partner arm, not an average across sessions\.

#### Attribution of the level \(c\) spread\.

The level \(c\) spread has no single cause\. Method position is not such a cause within a session:set\_seedis called before each method’s model load and no random numbers are consumed between iterations, so the seed state entering training does not depend on position\. Level \(b\) shows that position can still matter, and level \(c\) shows that session identity can\. Which of these dominates across a full re\-run is not tested here\.

Therefore, absolute BWT levels should not be compared across sessions; all statistical claims in the paper use within\-session paired contrasts\.

## Appendix GRRM Boundary Analysis

We investigated whether assigning each task a different random mask \(Rotating Random Masks, RRM\) provides additional benefit beyond shared random masking\. RRM generates a deterministic per\-task mask viami\(t\)=𝟏\[hash\(t∥i\)mod10<k\]m^\{\(t\)\}\_\{i\}=\\mathbf\{1\}\[\\text\{hash\}\(t\\\|i\)\\bmod 10<k\], reducing expected pairwise inter\-task parameter overlap from 100% \(shared mask\) tok2k^\{2\}\(1% at 10% sparsity\) \(Table[10](https://arxiv.org/html/2610.12744#A7.T10)\)\.

Table 10:RRM boundary analysis \(Qwen2\.5\-3B, single seed per condition except rank\-16 reference\)\.RRM shows a favorable paired difference at rank 16 only\. The overlap\-reduction hypothesis predicts a consistent RRM advantage; the data do not support this across conditions\. RRM’s effect appears condition\-dependent rather than robust\.

## Appendix HSparsity Sweep

We sweep the masking budget over 5%, 10%, 20%, and 30% \(single seed\) to test whether the within\-pool ordering is specific to the 10% sparsity used in the main experiments\. Across all four levels, shared random achieves numerically lower BWT than gradient attribution, with no cross\-over; the largest margin is at 30%\.

Figure 6:Sparsity sweep \(Qwen2\.5\-3B, single seed\)\. Shared random achieves numerically lower BWT than gradient attribution at all four tested levels\.Table 11:Sparsity sweep numerical values \(Qwen2\.5\-3B, single seed\)\. Random BWT≤\\leqGradient BWT at all tested levels\.
## Appendix ITraining Stability: Rank\-8 No\-Mask Divergence

The rank\-8 no\-mask ablation reveals a discrepancy between two endpoint\-based forgetting summaries\. Endpoint BWT is 0\.176; average forgetting is 0\.2837\. Training produced a severe loss spike during task 4 \(Math\): evaluation losses on three prior tasks exceeded 14, compared to values below 4 for all masked conditions at the same rank\. This spike subsequently recovered during task 5, yielding deceptively low endpoint BWT\. We do not interpret the rank\-8 no\-mask endpoint BWT of 0\.176 as evidence that no\-mask outperforms masking\.

The same signature recurs at rank 16 and is seed\-dependent: at seed 123 the unmasked arm spiked during task 4 with evaluation losses of 16\.2/16\.0/14\.7 on the three prior tasks \(code/medical/legal\), while every masked arm in that run stayed below2\.52\.5; at seed 42 the unmasked arm did*not*spike \(task\-4 evaluations≤1\.2\\leq 1\.2\)\. The instability is therefore a property of the unmasked \(all\-parameter\) configuration rather than of a particular rank, and it appears in a minority of seeds\.

## Appendix JMask Concentration by Module

The o\_proj suppression ablation and module\-concentration figures, discussed in §[6](https://arxiv.org/html/2610.12744#S6), are reproduced below together with the module\-concentration table for reference\.

Figure 7:The suppression control’s outcome is*seed\-dependent*\. All three arms train 1,651,508 parameters, which is 5\.5% of the full LoRA pool \(each arm draws 10% of thelora\_Bsub\-pool, andBBholds 55% of the pool’s parameters\), so this is the density\-matched setting throughout\. The uniform\-B controller redistributes the gradient mask’s budget uniformly across LoRA modules\. At seed 42 it sits in the gradient band \(0\.440 above gradient\-B’s 0\.423 and well above random\-B\-only’s 0\.278\); seeds 123 and 456 also land on the gradient side; at seed 789 it falls to the random band \(0\.355 below gradient\-B’s 0\.416, alongside random\-B\-only’s 0\.352\)\. Three of four seeds fall on the gradient side\. M2 is an independent session from D1, so the same\-seed gradient\-B values differ between them \(e\.g\. seed 42: 0\.423 here vs 0\.453 in the D1 table; seed 789: 0\.416 vs 0\.387; §[F](https://arxiv.org/html/2610.12744#A6)\)\. We therefore do not treat this ablation as establishing theo\_proj\-concentration mechanism \(§[6](https://arxiv.org/html/2610.12744#S6)\)\.Figure 8:Gradient\-mask concentration by LoRA module \(Qwen2\.5\-3B\)\. Gradient attribution selects 29% ofo\_projparameters \(2\.9×\\timesthe 10% uniform rate\) and is under\-represented indown\_proj\(4\.1%\); the dashed line marks uniform 10% selection\.Table 12:Gradient mask concentration by LoRA module \(Qwen2\.5\-3B\)\. Random masking distributes uniformly at 10% per module\. Gradient attribution selects exclusivelylora\_Bparameters \(alllora\_Areceive zero importance due toB=0B=0initialization\)\.
## Appendix KGeneralization and Boundary Checks

#### Scale Observations

We additionally observe the within\-pool comparison at three parameter scales \(0\.5B, 1\.5B, 7B; single seed each\)\. These are observational rather than multi\-seed, and at 0\.5B and 1\.5B only the shared\-random and no\-mask conditions were run, so no gradient attribution baseline is available for a direct comparison there\.

Table 13:BWT at additional scales \(single seed, observational\)\.
#### Task Ordering \(single seed\)

A reversed\-order experiment yields BWT≈0\.11\\approx 0\.11for all masking methods, compared to≈0\.42\\approx 0\.42under the standard order, a nearly4×4\{\\times\}reduction that far exceeds the method differences measured within this experiment \(within\-session method contrasts of order10−210^\{\-2\}\)\. This is a*single seed*, so the4×4\{\\times\}magnitude is a qualitative pointer, not a quantitative estimate; it is reported to show that task ordering dominates the forgetting landscape, not as a measured effect size\. The conclusions of this paper apply only to the tested Code→\\toMedical→\\toLegal→\\toMath→\\toCreative sequence; additional task orderings with multi\-seed replication are needed before the null result can be generalized\.

#### Cross\-Architecture Comparison

Both cross\-architecture designs are tabulated in Table[14](https://arxiv.org/html/2610.12744#A11.T14): the confounded design \(gradient selecting B\-only, random selecting A\+B\) and the within\-lora\_Bdesign \(both methods restricted tolora\_B\)\. The two designs are not comparable row\-for\-row because the Phi\-2 within\-B rows use a different LoRA target set; §[6\.1](https://arxiv.org/html/2610.12744#S6.SS1)discusses both\.

Table 14:Cross\-architecture comparison on Qwen2\.5\-3B and Phi\-2 \(2\.78B, GPT\-NeoX\-style\), with both arms within a row sharing one target set\. Qwen2\.5\-3B targetsq/k/v/o/gate/up/down\_proj\(pool29,933,56829\{,\}933\{,\}568\), whereas Phi\-2 uses GPT\-NeoX\-styleq,k,v,fc1,fc2,denseprojections \(pool23,592,96023\{,\}592\{,\}960\) in the confounded design\. In the within\-lora\_Bdesign, the Phi\-2 rows omitdense\(pool20,971,52020\{,\}971\{,\}520, 12\.5% smaller\), so the two Phi\-2 designs are not comparable row\-for\-row\. The within\-lora\_Brows are matched on pool but*not*density \(the gradient mask activates approximately2×2\\timesthe B parameters of Random\-B\), as on Qwen2\.5\-3B\.

## Appendix LIn\-Domain Calibration Ablation

The primary experiments use out\-of\-domain Alpaca calibration data \(64 samples\)\. To test whether this choice drives the null result, we repeat the gradient\-mask experiment with*pooled in\-domain*calibration: 60 samples drawn equally from the five CL tasks\.111⌊64/5⌋×5=60\\lfloor 64/5\\rfloor\\times 5=60; a 6\.25% size difference from integer division\.All other parameters are identical \(Qwen2\.5\-3B, seed 42, rank 16, 10% sparsity\)\.

Loss\-based BWT is 0\.495 vs 0\.491 with Alpaca calibration \(Δ=0\.004\\Delta=0\.004, 0\.8% relative\), well within the 5\-seed standard deviation of 0\.061\. Random and no\-mask controls are numerically identical across both runs, confirming infrastructure determinism\. The*cross\-calibration*overlap \(Alpaca vs pooled\) is Jaccard=0\.585=0\.585: changing the calibration domain alters 42% of selected parameters\. Despite this substantial mask difference, BWT changes by only 0\.004\. This robustness is of the*outcome*, not the mask: the mask does change, but*among gradient\-derived masks*the specific 10% selected has little influence on forgetting, whereas swapping the selection criterion entirely \(gradient for random\) does, as the density\-matched control shows \(§[5\.1](https://arxiv.org/html/2610.12744#S5.SS1)\)\.

## Appendix MB=0 Restriction Asymmetry

The within\-lora\_Bcomparison varies the selection criterion while holding the parameter pool fixed\. That the pool restriction is the right control to hold follows from an asymmetry between the two arms at seed 42 \(Table[15](https://arxiv.org/html/2610.12744#A13.T15)\)\. Restricting the gradient mask tolora\_Bchanges nothing, because underB=0B=0everylora\_Aimportance score is already zero and the restriction is a no\-op\. The same restriction applied to the random arm removes thelora\_Adilution and improves BWT by 0\.025\. A restriction that does nothing to one arm and helps the other is evidence that the two arms were drawing from different pools before the restriction was applied, which is what makes the pool\-matched comparison the informative one\.

Table 15:Effect of restricting the mask tolora\_B\(seed 42\)\. For gradient attribution, the restriction is a no\-op by construction \(B=0 collapses everylora\_Aimportance score to zero\); for random, restricting to B removeslora\_Adilution and improves BWT by 0\.025\. This asymmetry demonstrates that the within\-B comparison varies the selection criterion, not the parameter pool\.
## Appendix NPer\-Seed Accuracy and Matched Loss Contrast

The accuracy evaluation is reported in the main text, §[5\.4](https://arxiv.org/html/2610.12744#S5.SS4), together with the option\-scoring protocol; its per\-seed values and the matched loss contrast are tabulated below\. \(Table[16](https://arxiv.org/html/2610.12744#A14.T16)\)

The interval is wide because the per\-seed differences are wide, not because the mean is large\. With the observed across\-seed standard deviation of 4\.17 pp, the 95% interval half\-width is about 3\.9 pp atn=7n=7and would fall below 2\.0 pp only at roughly twenty seeds\.

Table 16:Per\-seed results, both columns in the direction \[Grad−\-Random\] \(positive = gradient forgets more\)\. Col\. 2 is the accuracy\-BWT difference: the difference in mean final\-minus\-learned accuracy over the prior discrete\-option tasks, not a difference of final accuracies\. It is therefore a retention quantity on the same footing as the loss column and does not mix learning with retention\. Col\. 2 is reported in percentage points and is measured in the A800 accuracy runs; Col\. 3 is the loss\-based BWT difference from the A800 loss runs\. Together they form a second environment relative to the original V100 loss runs\. Col\. 2 carries the 95%tt\-interval on the seven\-seed mean\. Col\. 3 has mean\+0\.152\+0\.152\(SD0\.07490\.0749,t=5\.37t=5\.37, CI\[\+0\.083,\+0\.221\]\[\+0\.083,\+0\.221\], 7/7 same direction\)\. The V100 loss contrast is\+0\.155\+0\.155under the same \[Grad−\-Random\] convention\. Across matched seeds, the A800\-minus\-V100 loss\-contrast difference has mean−0\.0033\-0\.0033and SD0\.04410\.0441, so individual seeds move between environments even though the average does not\.
## Appendix OThe Math Task: Truncation Audit

Loss on math decreases with training while accuracy on the same task collapses to zero for all arms: peak accuracy at math’s own training phase is0\.040\.04and every arm ends at0\.000\.00\. The numeric\-match function accepts a bare number, a number with surrounding text, and a complete solution, and the prompt prefixes are identical; nevertheless, the interaction between output format and the 256\-token limit makes the generation\-based score unreliable\. A generation audit reports 56–95% of generations across all tasks and arms reaching the cap without an end\-of\-sequence token, and for math specifically 81–90% of generations reach the cap\. The last number in a truncated solution is often an intermediate quantity, so the reference answer is absent from the value the scorer inspects\. Every arm shares the same cap and scorer, so the bias is common rather than arm\-specific\. We do not treat the math column as a forgetting result: under option\-based scoring math yields no score, and under generation scoring it is confounded with truncation\.

## Appendix PPositioning Against Gradient\-Attribution Families

Table[17](https://arxiv.org/html/2610.12744#A16.T17)places our contribution against the gradient\-attribution families we are positioned against, on the two axes that drive the study\.

Table 17:Positioning against gradient\-attribution families\. “B=0 pool” marks a method whose attribution is computed at a zero\-initialized LoRA, where the gradient with respect tolora\_Ais zero and the mask is confined tolora\_B; “within\-pool grad\-vs\-random” marks a method compared against a matched random baseline drawn from the same pool\. Attribution\-Guided CL computes its attribution from a separately trained model \(B≠0B\\neq 0at scoring time\) and gates gradients continuously rather than selecting a binary mask, so it is outside the first axis by construction and unmatched on the second\.Table[18](https://arxiv.org/html/2610.12744#A16.T18)lists the selection rules this paper is positioned against and marks which we train under one protocol\. Only the rules we train contribute empirical numbers here; the rest are compared by their published design, with none of their reported numbers presented as a measurement of ours\.

Table 18:Selection rules this paper is positioned against\. The last column marks the four rules we train and evaluate under one protocol \(§[3\.3](https://arxiv.org/html/2610.12744#S3.SS3)\); the rest are compared by design only, and none of their published numbers is cited as a result of ours\.
## Appendix QD1 Per\-Task and Per\-Seed Decomposition

Table[19](https://arxiv.org/html/2610.12744#A17.T19)gives the same quantity for each seed, so the task\-level pattern can be checked against the seed\-level one; the main text reports the same decomposition as a figure \(Figure[3](https://arxiv.org/html/2610.12744#S5.F3)\)\. Each entry isloss\_matrix​\[4\]​\[j\]−loss\_matrix​\[j\]​\[j\]\\text\{loss\\\_matrix\}\[4\]\[j\]\-\\text\{loss\\\_matrix\}\[j\]\[j\]for prior taskjj; the final column is the mean over the four tasks, which is the per\-seed aggregateΔ\\Deltaof Table[2](https://arxiv.org/html/2610.12744#S4.T2)\.

Table 19:Per\-seed, per\-task D1 decomposition \(Δ=random\-B−gradient\-B\\Delta=\\text\{random\-B\}\-\\text\{gradient\-B\}for each prior task\)\. Negative favours random\-B\. Legal is the only task whose sign is not consistent across seeds\.
## Appendix RRuling Out a Learning Deficit in the D1 Gap

The D1 density\-matched comparison \(both arms 10% oflora\_B, 1,651,508 parameters\) shows random\-B\-only achieving lower BWT than gradient\-B\. One rival reading is that gradient\-B’s top\-10% is the more concentrated half of the original 20% mask, so gradient\-B learns each task less well and the gap is a learning deficit rather than a retention difference\. The per\-task diagonal losses \(each task’s loss immediately after it is learned\) exclude this: gradient\-B’s diagonal loss is lower than random\-B\-only’s in*all*35 seed×\\timestask cells \(7 seeds×\\times5 tasks\), with no exceptions\.

Table 20:Diagonal loss \(mean±\\pmstd over the seven D1 seeds\), by task\. Lower is better\. gradient\-B is lower than random\-B\-only in every cell \(Table[20](https://arxiv.org/html/2610.12744#A18.T20)\); Grad\-Full is the uncontrolled reference\. Loss diagonals only; the D1 run records contain no accuracy matrix \(accuracy is measured separately in the accuracy branch, §[5\.4](https://arxiv.org/html/2610.12744#S5.SS4)\)\.A second rival reading is that BWT, being a difference of losses, is inflated for gradient\-B simply because it starts from a lower per\-task loss\. The endpoint closes this from both sides\. Because BWT is the mean over prior tasks of \(final loss−\-learned loss\) and the start is the mean learned loss over those same tasks, the endpoint is exactly the mean final held\-out loss over the first four tasks, read directly from the final row of the stored loss matrix rather than reconstructed\. gradient\-B starts*lower*yet ends*higher*in 6 of 7 seeds, so its larger BWT reflects greater forgetting rather than a lower baseline \(Table[21](https://arxiv.org/html/2610.12744#A18.T21)\)\. Measured on the same seven within\-session pairs, the endpoint difference averages\+0\.0813\+0\.0813\(SD0\.07100\.0710; pairedt⁡\(6\)=3\.03t\(6\)=3\.03,p=0\.023p=0\.023, 95% CI\[\+0\.016,\+0\.147\]\[\+0\.016,\+0\.147\], Cohen’sdz=1\.15d\_\{z\}=1\.15\)\. The second environment agrees in both mean and direction:\+0\.078\+0\.078\(t⁡\(6\)=2\.62t\(6\)=2\.62,p=0\.039p=0\.039, 6/7\), a mean separation of0\.0030\.003from the V100 estimate\. Thus Random\-B’s advantage is not confined to a BWT normalization: it also achieves lower final held\-out loss on the prior tasks\.

The endpoint significance is less robust than the headline BWT contrast to single\-seed deletion\. Three of the seven leave\-one\-out fits exceedp=0\.05p=0\.05: dropping seed 42 or 123 givesp=0\.058p=0\.058, and dropping seed 4096 givesp=0\.059p=0\.059, all with a mean of about\+0\.076\+0\.076\. The remaining four deletions givep≤0\.048p\\leq 0\.048\. We therefore treat the endpoint as a directional result supported by the paired design rather than as an effect that survives arbitrary single\-seed removal; the density\-matched BWT contrast on the same seven pairs is robust to every single\-seed deletion \(p≤0\.005p\\leq 0\.005throughout\)\.

Table 21:Mean final held\-out loss over the four prior tasks, per seed, read from the final row of the stored loss matrix\. Start is the same quantity at each task’s own training step; end is at the end of the sequence\. gradient\-B’s endpoint is higher in 6 of 7 seeds \(seed 789 is the exception\); pairedt⁡\(6\)=3\.03t\(6\)=3\.03,p=0\.023p=0\.023, mean\+0\.081\+0\.081on the V100, and\+0\.078\+0\.078\(p=0\.039p=0\.039\) on the A800\. Three of the seven single\-seed deletions fall abovep=0\.05p=0\.05\(seeds 42, 123, 4096\); see text\.Both checks are loss\-based \(accuracy is measured separately in the accuracy branch, §[5\.4](https://arxiv.org/html/2610.12744#S5.SS4)\), use a single task order, and cover Qwen2\.5\-3B only\. The endpoint is read directly from the final row of the stored matrices, so its per\-seed differences are load\-bearing rather than derived; the caveat above concerns then=7n=7fragility of the full\-set significance, not the definition of the quantity\.

## Appendix SCross\-Architecture Density\-Matched Replication \(Phi\-2\)

The density\-matched contrast of §[5\.1](https://arxiv.org/html/2610.12744#S5.SS1)is also run on Phi\-2 \(2\.78B, GPT\-NeoX\-style\), an architecturally distinct family from Qwen2\.5\-3B\. As in the primary D1 comparison, both arms select fromlora\_Balone and use the same density: each activates exactly 1,048,576 coordinates, which is 10% of the 10,485,760\-coordinatelora\_Bsub\-pool \(and 5\.0% of the 20,971,520 trainableA\+BA\{\+\}Bpool\)\. The candidate pool and update budget are therefore matched, leaving only the ranking rule different between Random\-B and Grad\-B\.

Table 22:Density\-matched D1 replication on Phi\-2, four seeds\. Both arms select the same number of coordinates from the samelora\_Bpool;Δ=Random\-B−Grad\-B\\Delta=\\text\{Random\-B\}\-\\text\{Grad\-B\}in loss\-based BWT, so negative values mean the random mask forgets less\.SeedRandom\-B BWT↓\\downarrowGrad\-B BWT↓\\downarrowΔ\\Delta420\.0397\\mathbf\{0\.0397\}0\.1661−0\.1264\-0\.12641230\.0363\\mathbf\{0\.0363\}0\.1273−0\.0910\-0\.09104560\.0461\\mathbf\{0\.0461\}0\.1749−0\.1289\-0\.128910240\.0480\\mathbf\{0\.0480\}0\.1624−0\.1144\-0\.1144Mean——−0\.1152\-0\.1152\(SD0\.01730\.0173\)The direction replicates in all four seeds \(Table[22](https://arxiv.org/html/2610.12744#A19.T22)\)\. The mean paired contrast is−0\.1152\-0\.1152\(SD0\.01730\.0173\), with pairedt⁡\(3\)=−13\.32t\(3\)=\-13\.32,p=0\.0009p=0\.0009, and 95% CI\[−0\.143,−0\.088\]\[\-0\.143,\-0\.088\]\. A two\-sided sign test over four concordant seeds givesp=0\.125p=0\.125; becausen=4n=4is too small for a strong distributional claim and the normality assumption behind the pairedtt\-test is weak, we treat thettstatistic as a supporting parametric summary rather than resting the replication claim on it\. The replication claim is instead based on the density\-matched design and the 4/4 directional agreement\.

Table 23:Density\-matched D1 contrast on both architectures under the same sign convention,Δ=Random\-B−Grad\-B\\Delta=\\text\{Random\-B\}\-\\text\{Grad\-B\}\(negative means Random\-B forgets less\)\. Phi\-2 is an architecturally distinct four\-seed replication of the seven\-seed Qwen2\.5\-3B result\.Table[23](https://arxiv.org/html/2610.12744#A19.T23)places the clean Phi\-2 replication beside the primary Qwen2\.5\-3B result under the same sign convention\. The task\-level decomposition also resembles the primary model rather than merely sharing the aggregate sign\. Averaged over the four Phi\-2 seeds, Medical contributes 67\.2% of the aggregate contrast, Math 24\.1%, Code 5\.8%, and Legal 2\.9%; Medical is the largest contributor in every seed\. On Qwen2\.5\-3B, Medical contributes 79\.9% of the matched contrast\. Thus the two architectures agree on where most of the effect is concentrated, although the concentration is somewhat less extreme on Phi\-2\.

The replication uses the same five task domains, LoRA rank, calibration set, and 10%lora\_Bbudget as D1\. It differs in model family, in the Phi\-2 LoRA target set \(which omitsdense, giving a 20,971,520\-parameter trainable pool\), and in the number of seeds\. The earlier Phi\-2 within\-B comparison in Appendix[K](https://arxiv.org/html/2610.12744#A11.SS0.SSS0.Px3)remains a density\-unmatched boundary check; the results here are the clean cross\-architecture counterpart to the headline density\-matched D1 comparison\.