Attribution-Guided Continual Learning for Large Language Models

# Attribution-Guided Continual Learning for Large Language Models
Source: [https://arxiv.org/html/2605.05285](https://arxiv.org/html/2605.05285)
Yazheng Liu1, Yuxuan Wan1, Rui Xu1, Xi Zhang2, Sihong Xie1, Hui Xiong1 1The Hong Kong University of Science and Technology \(Guangzhou\), Guangzhou, China 2The Beijing University of Posts and Telecommunications, Beijing, China

###### Abstract

Large language models \(LLMs\) often suffer from catastrophic forgetting in continual learning: after learning new tasks sequentially, they perform worse on earlier tasks\. Existing methods mitigate catastrophic forgetting by data replay, parameter freezing, or regularization\. However, these methods lack semantic awareness of internal knowledge distribution in LLMs\. As a result, they cannot distinguish parameters that should be preserved or updated\. We propose an attribution\-guided continual fine\-tuning framework for LLMs\. Our method estimates task\-specific, element\-wise parameter importance in each Transformer layer and uses these scores to modulate gradients\. Parameters important to previous tasks receive smaller updates, while less relevant ones remain plastic for learning new tasks\. Experiments on continual learning benchmarks show that our method consistently outperforms baselines, achieving better retention of old tasks while maintaining competitive performance on new tasks\.

## 1Introduction

Large Language Models \(LLMs\)Achiamet al\.\([2023](https://arxiv.org/html/2605.05285#bib.bib4)\); Touvronet al\.\([2023](https://arxiv.org/html/2605.05285#bib.bib5)\); Liuet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib6)\); Comaniciet al\.\([2025](https://arxiv.org/html/2605.05285#bib.bib17)\); Huiet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib18)\)have achieved exceptional performance across diverse tasks, such as multi\-step reasoningWeiet al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib7)\); Wanget al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib19)\), instruction followingOuyanget al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib20)\); Chunget al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib8)\), and code geneartionRoziereet al\.\([2023](https://arxiv.org/html/2605.05285#bib.bib21)\); Chenet al\.\([2021](https://arxiv.org/html/2605.05285#bib.bib22)\)\. However, LLMs are usually trained on static, broad\-domain data\. This can lead to performance degradation when the target domain changesChenet al\.\([2023](https://arxiv.org/html/2605.05285#bib.bib10)\); Dhingraet al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib11)\); Luet al\.\([2025a](https://arxiv.org/html/2605.05285#bib.bib12)\)\. To adapt LLMs to downstream tasks while preserving prior knowledge, researchers use continual learning methodVan de Venet al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib13)\); Wanget al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib14)\)\. Continual learning trains models on a sequence of tasks but faces a challenge known as catastrophic forgettingMcCloskey and Cohen \([1989](https://arxiv.org/html/2605.05285#bib.bib15)\); McClellandet al\.\([1995](https://arxiv.org/html/2605.05285#bib.bib16)\): fine\-tuning LLM on a new task can substantially degrade its performance on previously learned tasks\.

Existing approaches have been proposed to mitigate catastrophic forgetting in continual learning\. Replay methodsSunet al\.\([2019](https://arxiv.org/html/2605.05285#bib.bib25)\); Huanget al\.\([2024b](https://arxiv.org/html/2605.05285#bib.bib27)\); Scialomet al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib26)\); Abbeset al\.\([2025](https://arxiv.org/html/2605.05285#bib.bib28)\)retain examples from previous tasks and train on them together with current task data\. Regularization methodsZhanget al\.\([2023](https://arxiv.org/html/2605.05285#bib.bib32)\); Rebuffiet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib33)\); Li and Hoiem \([2017](https://arxiv.org/html/2605.05285#bib.bib34)\)penalize large deviations from the previous model in parameter space\. Freezing methods instead fix selected LLM parameters, often in early layers, to reduce forgettingZhenget al\.\([2025](https://arxiv.org/html/2605.05285#bib.bib35)\)\. However, these methods lack a mechanistic understanding of how knowledge and task\-specific behaviors are distributed across LLM parameters\.

To understand the internal mechanisms of LLMs in continual learning, we explore important parameters in LLMs on different tasks\. Letθ\\thetadenote the model parameters,𝒲\(l\)​\(θ\)\\mathcal\{W\}^\{\(l\)\}\(\\theta\)be the set of model parameters in itsll\-th layer, andW\(l\)∈𝒲\(l\)​\(θ\)W^\{\(l\)\}\\in\\mathcal\{W\}^\{\(l\)\}\(\\theta\)represent an individual model parameter\. For a task𝒯\\mathcal\{T\}, we compute the importance of each parameter with respect to the next\-token logits using the attribution method introduced in Section[4\.1\.2](https://arxiv.org/html/2605.05285#S4.SS1.SSS2)\. We then select the top\-KKmost important elements inW\(l\)W^\{\(l\)\}, denoted by𝒫K​\(W\(l\);θ,𝒯\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta,\\mathcal\{T\}\\big\)\. Given two tasks,𝒯1\\mathcal\{T\}\_\{1\}and𝒯2\\mathcal\{T\}\_\{2\}, we compare independent single\-task fine\-tuning with sequential continual fine\-tuning, as illustrated in Figure[3](https://arxiv.org/html/2605.05285#S4.F3)\(a,b\)\. In the single\-task setting, a pretrained LLM is fine\-tuned separately on each task, producing task\-specific modelsθ1′\\theta\_\{1\}^\{\\prime\}andθ2′\\theta\_\{2\}^\{\\prime\}\. We compute𝒫K​\(W\(l\);θ1′,𝒯1\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{1\}^\{\\prime\},\\mathcal\{T\}\_\{1\}\\big\)and𝒫K​\(W\(l\);θ2′,𝒯2\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{2\}^\{\\prime\},\\mathcal\{T\}\_\{2\}\\big\)\. In the continual setting, the pretrained model is fine\-tuned sequentially on𝒯1\\mathcal\{T\}\_\{1\}and𝒯2\\mathcal\{T\}\_\{2\}, resulting in final parametersθ2\\theta\_\{2\}\. We obtain𝒫K​\(W\(l\);θ2,𝒯1\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{2\},\\mathcal\{T\}\_\{1\}\\big\)and𝒫K​\(W\(l\);θ2,𝒯2\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{2\},\\mathcal\{T\}\_\{2\}\\big\)\. Finally, we quantify the similarity of𝒫K​\(W\(l\);θ1′,𝒯1\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{1\}^\{\\prime\},\\mathcal\{T\}\_\{1\}\\big\)and𝒫K​\(W\(l\);θ2′,𝒯2\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{2\}^\{\\prime\},\\mathcal\{T\}\_\{2\}\\big\),𝒫K​\(W\(l\);θ2,𝒯1\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{2\},\\mathcal\{T\}\_\{1\}\\big\)and𝒫K​\(W\(l\);θ2,𝒯2\)\\mathcal\{P\}\_\{K\}\\big\(W^\{\(l\)\};\\theta\_\{2\},\\mathcal\{T\}\_\{2\}\\big\)\.

![Refer to caption](https://arxiv.org/html/2605.05285v1/figs/importance_similarity_by_layer.png)Figure 1:Similarity of task important parameters in single task and continual learning\.Figure[1](https://arxiv.org/html/2605.05285#S1.F1)shows the results on Llama\-3\.2\-Instruct\-3BGrattafioriet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib37)\)with LoRAHuet al\.\([2022](https://arxiv.org/html/2605.05285#bib.bib36)\)fine\-tuning, where𝒯1\\mathcal\{T\}\_\{1\}is summary generation and𝒯2\\mathcal\{T\}\_\{2\}is code completion\. In the independent single\-task setting \(pink and yellow\), both the top\-KKoverlap and Spearman correlation remain low scores across layers, indicating that the important parameters vary across tasks\. By contrast, after sequential continual learning \(blue and purple\), both similarity metrics increase\. Consequently, updates for learning𝒯2\\mathcal\{T\}\_\{2\}are likely to modify parameters that are also critical to𝒯1\\mathcal\{T\}\_\{1\}, thereby inducing interference and potential forgetting\.

![Refer to caption](https://arxiv.org/html/2605.05285v1/x1.png)Figure 2:Motivation and overview of our proposed framework\. \(a\): In single\-task fine\-tuning, a pre\-trained LLM is separately adapted to Task 1 and Task 2, yielding task\-specific models with low overlap among their top\-KKimportant parameters\. This suggests different tasks rely on distinct parameter subsets\. \(b\): In continual learning, the important parameters for the two tasks become highly overlapped in the final model, indicating that learning Task 2 may overwrite parameters important to Task 1\. \(c\): Existing methods lack a mechanistic understanding of how knowledge are distributed across LLM parameters\. \(d\): Our framework obtain parameter importance and protects important parameters while allowing less important parameters to adapt to new tasks\.Motivated by this observation, we propose an attribution\-aware continual fine\-tuning framework that mitigates forgetting by using the importance of model parameters\. Instead of treating all parameters uniformly during continual learning process, our method first identifies parameters that are critical to previously learned tasks through an attribution\-based importance estimation procedure\. The resulting task\-specific priors are then used to guide subsequent fine\-tuning by selectively constraining updates to parameters that are important for prior knowledge, while leaving less critical parameters available for adaptation\. We evaluate our framework on large language models under both full supervised fine\-tuning and LoRA\-based adaptation settings, and it consistently achieves superior performance over existing continual fine\-tuning baselines\.

## 2Related work

Continual learning in LLMs\. Recent work on continual learning for LLMs adopts classical strategies to mitigate catastrophic forgetting, including regularization, replay, and parameter freezing\. Among regularization\-based methods, CLoRALuet al\.\([2025b](https://arxiv.org/html/2605.05285#bib.bib44)\)regularizes LoRA updates in a subspace to reduce interference without replay\. Replay\-based methods mitigate forgetting by reusing or synthesizing old\-task information\. SSRHuanget al\.\([2024a](https://arxiv.org/html/2605.05285#bib.bib43)\)generates synthetic data with LLMs, then uses another LLM to refine and select high\-quality samples for rehearsal\. KPIGHeet al\.\([2025](https://arxiv.org/html/2605.05285#bib.bib42)\)selects replay samples based on information gain\. Freezing\-based methods preserve prior knowledge by updating only a small subset of parameters or added modules\. SAPTZhaoet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib41)\)learns and selects parameter\-efficient blocks via shared attentive learning and selection module\. LoRAMoEDouet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib38)\)freezes the backbone and adds MoE\-style LoRA adapters, while this methodZhenget al\.\([2025](https://arxiv.org/html/2605.05285#bib.bib35)\)freeze lower LLM layers and fine\-tune upper layers\. However, existing methods rarely analyze continual learning from the mechanistic perspective of LLMs, leaving the task\-specific distribution of parameter importance largely unexplored\.

## 3Background

Background on LLMs\. Large Language Models \(LLMs\) can be formulated as deep compositions of functions that progressively transform token representationsAubryet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib1)\)\. Given an input prompt tokenized asx=\[x1,…,xm\]x=\[x\_\{1\},\\dots,x\_\{m\}\], each tokenxix\_\{i\}is mapped to add\-dimensional embedding and augmented with positional information, yielding the initial hidden statehi\(0\)∈ℝ1×dh\_\{i\}^\{\(0\)\}\\in\\mathbb\{R\}^\{1\\times d\}\. The resulting sequence representation isH\(0\)=\[h1\(0\);…;hm\(0\)\]∈ℝm×dH^\{\(0\)\}=\[h\_\{1\}^\{\(0\)\};\\dots;h\_\{m\}^\{\(0\)\}\]\\in\\mathbb\{R\}^\{m\\times d\}\.

The representations are then processed byLLTransformer blocks, each consisting of multi\-head self\-attention \(MHA\), a feed\-forward network \(FFN\), layer normalization \(LN\), and residual connections\. LetH\(l\)H^\{\(l\)\}denote the hidden states after thell\-th block\. The\(l\+1\)\(l\+1\)\-th block is defined as

f\(l\)\\displaystyle f^\{\(l\)\}=MHA​\(LN​\(H\(l\)\)\),g\(l\)=LN​\(H\(l\)\+f\(l\)\),H\(l\+1\)=H\(l\)\+f\(l\)\+FFN​\(g\(l\)\)\.\\displaystyle=\\mathrm\{MHA\}\\big\(\\mathrm\{LN\}\(H^\{\(l\)\}\)\\big\),\\quad g^\{\(l\)\}=\\mathrm\{LN\}\\big\(H^\{\(l\)\}\+f^\{\(l\)\}\\big\),\\quad H^\{\(l\+1\)\}=H^\{\(l\)\}\+f^\{\(l\)\}\+\\mathrm\{FFN\}\\big\(g^\{\(l\)\}\\big\)\.\(1\)
AfterLLlayers,H\(L\)H^\{\(L\)\}are projected onto the vocabulary space to obtain the logits:

Z=LN​\(H\(L\)\)​Wvocab∈ℝm×\|𝒱\|,Z=\\mathrm\{LN\}\(H^\{\(L\)\}\)W\_\{\\mathrm\{vocab\}\}\\in\\mathbb\{R\}^\{m\\times\|\\mathcal\{V\}\|\},\(2\)where𝒱\\mathcal\{V\}denotes the vocabulary,\|𝒱\|\|\\mathcal\{V\}\|is its size, andWvocab∈ℝd×\|𝒱\|W\_\{\\mathrm\{vocab\}\}\\in\\mathbb\{R\}^\{d\\times\|\\mathcal\{V\}\|\}is the output embedding matrix\. The next predicted token index isj^=arg⁡maxj∈\{1,…,\|𝒱\|\}⁡Zm,j\\hat\{j\}=\\arg\\max\_\{j\\in\\\{1,\\dots,\|\\mathcal\{V\}\|\\\}\}Z\_\{m,j\}, which corresponds to the predicted tokenx^m\+1=𝒱​\[j^\]\\hat\{x\}\_\{m\+1\}=\\mathcal\{V\}\[\\hat\{j\}\]\.

Background on continual learning\. In continual learning, the LLM is trained on a sequence of tasks\{𝒯1,𝒯2,…,𝒯T\}\\\{\\mathcal\{T\}\_\{1\},\\mathcal\{T\}\_\{2\},\\ldots,\\mathcal\{T\}\_\{T\}\\\}, whereTTdenotes the total number of tasks\. We uset∈\{1,…,T\}t\\in\\\{1,\\ldots,T\\\}to index the training stage and its corresponding task\. Each task𝒯t\\mathcal\{T\}\_\{t\}is associated with a dataset𝒟t=\{\(x\(k\),y\(k\)\)\}k=1Nt\\mathcal\{D\}\_\{t\}=\\\{\(x^\{\(k\)\},y^\{\(k\)\}\)\\\}\_\{k=1\}^\{N\_\{t\}\}, wherex\(k\)x^\{\(k\)\}andy\(k\)y^\{\(k\)\}denote the input prompt and target response of thekk\-th example, respectively\. Letθt\\theta\_\{t\}denote the model parameters after training on task𝒯t\\mathcal\{T\}\_\{t\}\. At stagett, the model is initialized fromθt−1\\theta\_\{t\-1\}and optimized on the current dataset𝒟t\\mathcal\{D\}\_\{t\}using the autoregressive next\-token prediction objective:

ℒt​\(θ\)=−𝔼\(x\(k\),y\(k\)\)∼𝒟t​\[1n​∑j=1nlog⁡pθ​\(yj\(k\)∣x\(k\),y<j\(k\)\)\]\.\\mathcal\{L\}\_\{t\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(x^\{\(k\)\},y^\{\(k\)\}\)\\sim\\mathcal\{D\}\_\{t\}\}\\left\[\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{n\}\\log p\_\{\\theta\}\\left\(y^\{\(k\)\}\_\{j\}\\mid x^\{\(k\)\},y^\{\(k\)\}\_\{<j\}\\right\)\\right\]\.\(3\)

## 4Method

We propose an importance\-guided fine\-tuning framework for continual learning in LLMs\. For each task, our method first estimates element\-wise importance scores for model parameters to measure their task\-specific relevance \(Section[4](https://arxiv.org/html/2605.05285#S4)\)\. During fine\-tuning stage, parameters with high importance receive smaller updates, whereas those with low importance are updated freely\. This allows the model to adapt to new tasks while reducing interference with previously acquired knowledge \(Section[4\.2](https://arxiv.org/html/2605.05285#S4.SS2)\)\.

### 4\.1Calculating element\-wise importance of LLM parameters

We apply the Layer\-wise Relevance Propagation \(LRP\)Bachet al\.\([2015](https://arxiv.org/html/2605.05285#bib.bib2)\)method to estimate the importance of LLM parameters in each layer\. We first review the standard LRP formulation and its extension to Transformer architectures\. Then, unlike prior applications that primarily attribute relevance to input features, we adapt LRP to quantify the contribution of parameters to the predicted next\-token logit\.

#### 4\.1\.1LRP\-based input attribution

Original LRP\. LRP attributes the prediction score to input neurons by propagating relevance backward through the network\. For a neuronal\+1=ψ​\(\[a1\(l\),…,an\(l\)\]\)a^\{l\+1\}=\\psi\(\[a^\{\(l\)\}\_\{1\},\\dots,a^\{\(l\)\}\_\{n\}\]\), whereψ\\psimay be a linear function or a composition of a linear function with a nonlinear activation, its relevanceRl\+1R^\{l\+1\}is attributed to neurons in the preceding layer as:

Ri\(l\)=ai\(l\)​wil∑i′ai′\(l\)​wi′l​R\(l\+1\),R^\{\(l\)\}\_\{i\}=\\frac\{a^\{\(l\)\}\_\{i\}w^\{l\}\_\{i\}\}\{\\sum\_\{i^\{\\prime\}\}a^\{\(l\)\}\_\{i^\{\\prime\}\}w^\{l\}\_\{i^\{\\prime\}\}\}R^\{\(l\+1\)\},\(4\)wherewilw^\{l\}\_\{i\}denotes weight from the neuronai\(l\)a^\{\(l\)\}\_\{i\}to the neurona\(l\+1\)a^\{\(l\+1\)\}\. For a multi\-layer network, Eq\. \([4](https://arxiv.org/html/2605.05285#S4.E4)\) is recursively applied from the output logitR\(L\)R^\{\(L\)\}to the input layer, producing input relevance scoresRi\(0\)R^\{\(0\)\}\_\{i\}\. TheRi\(0\)R^\{\(0\)\}\_\{i\}is computed as follows:

Ri\(0\)=∑j,…,kai\(0\)​wi0∑i′ai′\(0\)​wi′0​⋯​ak\(L−1\)​wkL−1∑k′ak′\(L−1\)​wk′L−1⋅R\(L\),∑iRi\(0\)=R\(L\)R^\{\(0\)\}\_\{i\}=\\sum\_\{j,\\dots,k\}\\frac\{a^\{\(0\)\}\_\{i\}w^\{0\}\_\{i\}\}\{\\sum\_\{i^\{\\prime\}\}a^\{\(0\)\}\_\{i^\{\\prime\}\}w^\{0\}\_\{i^\{\\prime\}\}\}\\cdots\\frac\{a^\{\(L\-1\)\}\_\{k\}w^\{L\-1\}\_\{k\}\}\{\\sum\_\{k^\{\\prime\}\}a^\{\(L\-1\)\}\_\{k^\{\\prime\}\}w^\{L\-1\}\_\{k^\{\\prime\}\}\}\\cdot R^\{\(L\)\},\\quad\\sum\_\{i\}R^\{\(0\)\}\_\{i\}=R^\{\(L\)\}\(5\)
LRP in Transformer\. The original LRP method can not used to multiplicative operations, such asa\(l\+1\)=∑i=1nai\(l\)⋅bi\(l\)a^\{\(l\+1\)\}=\\sum\_\{i=1\}^\{n\}a^\{\(l\)\}\_\{i\}\\cdot b^\{\(l\)\}\_\{i\}, when bothai\(l\)a^\{\(l\)\}\_\{i\}andbi\(l\)b^\{\(l\)\}\_\{i\}are inputs whose relevance needs to be attributed\.

However, in Transformer, matrix multiplications are extensively used in the self\-attention module\. To address this issue, AttnLRPAchtibatet al\.\([2024](https://arxiv.org/html/2605.05285#bib.bib3)\)treats matrix multiplication as a bilinear operation: the relevance ofai\(l\)a^\{\(l\)\}\_\{i\}is computed withbi\(l\)b^\{\(l\)\}\_\{i\}fixed, and vice versa\. Formally, given the output relevance scoreR​\(a\(l\+1\)\)R\(a^\{\(l\+1\)\}\), the relevance assigned toai\(l\)a^\{\(l\)\}\_\{i\}andbi\(l\)b^\{\(l\)\}\_\{i\}, together with the conservation property, is given by:

R​\(ai\(l\)\)=R​\(bi\(l\)\)=12​ai\(l\)​bi\(l\)∑j=1naj\(l\)​bj\(l\)​R​\(a\(l\+1\)\),∑i=1nR​\(ai\(l\)\)\+∑i=1nR​\(bi\(l\)\)=R​\(a\(l\+1\)\)\\displaystyle R\(a^\{\(l\)\}\_\{i\}\)=R\(b^\{\(l\)\}\_\{i\}\)=\\frac\{1\}\{2\}\\frac\{a^\{\(l\)\}\_\{i\}b^\{\(l\)\}\_\{i\}\}\{\\sum\_\{j=1\}^\{n\}a^\{\(l\)\}\_\{j\}b^\{\(l\)\}\_\{j\}\}R\(a^\{\(l\+1\)\}\),\\quad\\sum\_\{i=1\}^\{n\}R\(a^\{\(l\)\}\_\{i\}\)\+\\sum\_\{i=1\}^\{n\}R\(b^\{\(l\)\}\_\{i\}\)=R\(a^\{\(l\+1\)\}\)\(6\)

#### 4\.1\.2LRP\-based parameters attribution in LLMs

Existing LRP methods, including their Transformer extensions, primarily attribute predictions to inputs, while leaving model parameters unattributed\. We adopt the bilinear attribution principle of AttnLRP to the FFN and MHA modules and derive parameter\-level relevance scores for each LLM layer to the next token logitZm,j^Z\_\{m,\\hat\{j\}\}, wherej^=arg⁡maxj∈\{1,…,\|𝒱\|\}⁡Zm,j\\hat\{j\}=\\arg\\max\_\{j\\in\\\{1,\\dots,\|\\mathcal\{V\}\|\\\}\}Z\_\{m,j\}\. These relevance scores are then used to guide fine\-tuning process\.

###### Lemma 4\.1\.

For a linear transformationC=P​W\+BC=PW\+B, whereP∈ℝm×dinP\\in\\mathbb\{R\}^\{m\\times d\_\{\\mathrm\{in\}\}\},W∈ℝdin×doutW\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times d\_\{\\mathrm\{out\}\}\}, andB,C∈ℝm×doutB,C\\in\\mathbb\{R\}^\{m\\times d\_\{\\mathrm\{out\}\}\}\. Let⊙\\odotdenote element\-wise multiplication, and the division is applied element\-wise\. Given the output relevanceR​\(C\)∈ℝm×doutR\(C\)\\in\\mathbb\{R\}^\{m\\times d\_\{\\mathrm\{out\}\}\}, the relevance assigned toPP,WW, andBBunder the bilinear LRP rule is obtained from the following formula and satisfies the conservation property:

R​\(P\)=12​P⊙\[\(R​\(C\)C\)​W⊤\],R​\(W\)=12​W⊙\[P⊤​\(R​\(C\)C\)\],R​\(B\)=B⊙R​\(C\)C,\\displaystyle R\(P\)=\\frac\{1\}\{2\}P\\odot\\left\[\\left\(\\frac\{R\(C\)\}\{C\}\\right\)W^\{\\top\}\\right\],R\(W\)=\\frac\{1\}\{2\}W\\odot\\left\[P^\{\\top\}\\left\(\\frac\{R\(C\)\}\{C\}\\right\)\\right\],R\(B\)=B\\odot\\frac\{R\(C\)\}\{C\},∑R​\(P\)\+∑R​\(W\)\+∑R​\(B\)=∑R​\(C\),\\sum R\(P\)\+\\sum R\(W\)\+\\sum R\(B\)=\\sum R\(C\),\(7\)

Lemma[4\.1](https://arxiv.org/html/2605.05285#S4.Thmtheorem1)quantifies the contributions of input activations, model parameters, and biases in matrix multiplication\. We apply this result to derive parameter\-level relevance scores for the FFN and MHA modules in LLMs\.

LRP Attribution in FFN\. LetΔFFN\(l\)=FFN​\(g\(l\)\)\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}=\\mathrm\{FFN\}\\big\(g^\{\(l\)\}\\big\)and the FFN module is defined as

U\(l\)=g\(l\)​W1\+b1,S\(l\)=σ​\(U\(l\)\),ΔFFN\(l\)=S\(l\)​W2\+b2,U^\{\(l\)\}=g^\{\(l\)\}W\_\{1\}\+b\_\{1\},\\quad S^\{\(l\)\}=\\sigma\(U^\{\(l\)\}\),\\quad\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}=S^\{\(l\)\}W\_\{2\}\+b\_\{2\},whereW1∈ℝd×dffW\_\{1\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathrm\{ff\}\}\},b1∈ℝdffb\_\{1\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\},W2∈ℝdff×dW\_\{2\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{ff\}\}\\times d\}, andb2∈ℝdb\_\{2\}\\in\\mathbb\{R\}^\{d\}\. The intermediate activations satisfyU\(l\),S\(l\)∈ℝm×dffU^\{\(l\)\},S^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times d\_\{\\mathrm\{ff\}\}\}, and the FFN output satisfiesΔFFN\(l\)∈ℝm×d\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times d\}\. We then apply Lemma[4\.1](https://arxiv.org/html/2605.05285#S4.Thmtheorem1)sequentially to the FFN, yielding the following relevance propagation rules for the input and parameters\.

###### Proposition 4\.2\(FFN relevance propagation\)\.

Given the output relevanceR​\(ΔFFN\(l\)\)R\(\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\), and treating the activationσ​\(⋅\)\\sigma\(\\cdot\)as identity relevance propagation , i\.e\.,R​\(U\(l\)\)=R​\(S\(l\)\)R\(U^\{\(l\)\}\)=R\(S^\{\(l\)\}\)\.⊙\\odotdenotes element\-wise multiplication, division is applied element\-wise, and𝟏m∈ℝm×1\\mathbf\{1\}\_\{m\}\\in\\mathbb\{R\}^\{m\\times 1\}is an all\-one vector\. The relevance scores of the FFN input and parameters are given by

R​\(W2\(l\)\)\\displaystyle R\(W\_\{2\}^\{\(l\)\}\)=12​W2\(l\)⊙\[\(S\(l\)\)⊤​\(R​\(ΔFFN\(l\)\)ΔFFN\(l\)\)\],R​\(b2\(l\)\)=b2\(l\)⊙𝟏m⊤​\(R​\(ΔFFN\(l\)\)ΔFFN\(l\)\),\\displaystyle=\\frac\{1\}\{2\}W\_\{2\}^\{\(l\)\}\\odot\\left\[\\left\(S^\{\(l\)\}\\right\)^\{\\top\}\\left\(\\frac\{R\(\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\)\}\{\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\}\\right\)\\right\],\\quad R\(b\_\{2\}^\{\(l\)\}\)=b\_\{2\}^\{\(l\)\}\\odot\\mathbf\{1\}\_\{m\}^\{\\top\}\\left\(\\frac\{R\(\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\)\}\{\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\}\\right\),R​\(S\(l\)\)\\displaystyle R\(S^\{\(l\)\}\)=12​S\(l\)⊙\[\(R​\(ΔFFN\(l\)\)ΔFFN\(l\)\)​\(W2\(l\)\)⊤\],R​\(g\(l\)\)=12​g\(l\)⊙\[\(R​\(S\(l\)\)U\(l\)\)​\(W1\(l\)\)⊤\],\\displaystyle=\\frac\{1\}\{2\}S^\{\(l\)\}\\odot\\left\[\\left\(\\frac\{R\(\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\)\}\{\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\}\\right\)\\left\(W\_\{2\}^\{\(l\)\}\\right\)^\{\\top\}\\right\],\\quad R\(g^\{\(l\)\}\)=\\frac\{1\}\{2\}g^\{\(l\)\}\\odot\\left\[\\left\(\\frac\{R\(S^\{\(l\)\}\)\}\{U^\{\(l\)\}\}\\right\)\\left\(W\_\{1\}^\{\(l\)\}\\right\)^\{\\top\}\\right\],R​\(W1\(l\)\)\\displaystyle R\(W\_\{1\}^\{\(l\)\}\)=12​W1\(l\)⊙\[\(g\(l\)\)⊤​\(R​\(S\(l\)\)U\(l\)\)\],R​\(b1\(l\)\)=b1\(l\)⊙𝟏m⊤​\(R​\(S\(l\)\)U\(l\)\)\.\\displaystyle=\\frac\{1\}\{2\}W\_\{1\}^\{\(l\)\}\\odot\\left\[\\left\(g^\{\(l\)\}\\right\)^\{\\top\}\\left\(\\frac\{R\(S^\{\(l\)\}\)\}\{U^\{\(l\)\}\}\\right\)\\right\],\\quad R\(b\_\{1\}^\{\(l\)\}\)=b\_\{1\}^\{\(l\)\}\\odot\\mathbf\{1\}\_\{m\}^\{\\top\}\\left\(\\frac\{R\(S^\{\(l\)\}\)\}\{U^\{\(l\)\}\}\\right\)\.\(8\)

Proposition[4\.2](https://arxiv.org/html/2605.05285#S4.Thmtheorem2)provides relevance scores for both inputs and parameters in LLM FFN modules\. These scores further satisfy the following conservation property:

∑R​\(g\(l\)\)\+∑R​\(W1\(l\)\)\+∑R​\(b1\(l\)\)\+∑R​\(W2\(l\)\)\+∑R​\(b2\(l\)\)=∑R​\(ΔFFN\(l\)\)\.\\sum R\(g^\{\(l\)\}\)\+\\sum R\(W\_\{1\}^\{\(l\)\}\)\+\\sum R\(b\_\{1\}^\{\(l\)\}\)\+\\sum R\(W\_\{2\}^\{\(l\)\}\)\+\\sum R\(b\_\{2\}^\{\(l\)\}\)=\\sum R\(\\Delta\_\{\\mathrm\{FFN\}\}^\{\(l\)\}\)\.
LRP Attribution in MHA\. LetX\(l\)=LN​\(H\(l\)\)X^\{\(l\)\}=\\mathrm\{LN\}\(H^\{\(l\)\}\), whereH\(l\),X\(l\)∈ℝm×dH^\{\(l\)\},X^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times d\}\. Letdheadd\_\{\\mathrm\{head\}\}denote the dimension of each attention head\. Forr=1,…,d/dheadr=1,\\dots,d/d\_\{\\mathrm\{head\}\}, therr\-th head is computed as

Qr\(l\)\\displaystyle Q\_\{r\}^\{\(l\)\}=X\(l\)​WQ,r\(l\),Kr\(l\)=X\(l\)​WK,r\(l\),Vr\(l\)=X\(l\)​WV,r\(l\),Ar\(l\)=softmax​\(Qr\(l\)​\(Kr\(l\)\)⊤dhead\),\\displaystyle=X^\{\(l\)\}W\_\{Q,r\}^\{\(l\)\},\\quad K\_\{r\}^\{\(l\)\}=X^\{\(l\)\}W\_\{K,r\}^\{\(l\)\},\\quad V\_\{r\}^\{\(l\)\}=X^\{\(l\)\}W\_\{V,r\}^\{\(l\)\},\\quad A\_\{r\}^\{\(l\)\}=\\mathrm\{softmax\}\\\!\\left\(\\frac\{Q\_\{r\}^\{\(l\)\}\(K\_\{r\}^\{\(l\)\}\)^\{\\top\}\}\{\\sqrt\{d\_\{\\mathrm\{head\}\}\}\}\\right\),Or\(l\)\\displaystyle O\_\{r\}^\{\(l\)\}=Ar\(l\)​Vr\(l\),O\(l\)=Concat​\(O1\(l\),…,Od/dhead\(l\)\),f\(l\)=O\(l\)​WO\(l\)\.\\displaystyle=A\_\{r\}^\{\(l\)\}V\_\{r\}^\{\(l\)\},\\quad O^\{\(l\)\}=\\mathrm\{Concat\}\\left\(O\_\{1\}^\{\(l\)\},\\dots,O\_\{d/d\_\{\\mathrm\{head\}\}\}^\{\(l\)\}\\right\),\\quad f^\{\(l\)\}=O^\{\(l\)\}W\_\{O\}^\{\(l\)\}\.WhereWQ,r\(l\),WK,r\(l\),WV,r\(l\)∈ℝd×dheadW\_\{Q,r\}^\{\(l\)\},W\_\{K,r\}^\{\(l\)\},W\_\{V,r\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\mathrm\{head\}\}\},Qr\(l\),Kr\(l\),Vr\(l\),Or\(l\)∈ℝm×dheadQ\_\{r\}^\{\(l\)\},K\_\{r\}^\{\(l\)\},V\_\{r\}^\{\(l\)\},O\_\{r\}^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times d\_\{\\mathrm\{head\}\}\},Ar\(l\)∈ℝm×mA\_\{r\}^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times m\},WO\(l\)∈ℝd×dW\_\{O\}^\{\(l\)\}\\in\\mathbb\{R\}^\{d\\times d\}andf\(l\)∈ℝm×df^\{\(l\)\}\\in\\mathbb\{R\}^\{m\\times d\}\. Applying Lemma[4\.1](https://arxiv.org/html/2605.05285#S4.Thmtheorem1)to the MHA module gives the following relevance propagation rules for its inputs and parameters\.

###### Proposition 4\.3\(MHA relevance propagation\)\.

Given the output relevanceR​\(f\(l\)\)R\(f^\{\(l\)\}\)of the MHA module, let\[⋅\]r\[\\cdot\]\_\{r\}denote therr\-th column\-wise block corresponding to therr\-th attention head\. We treat the softmax operation as identity relevance propagation,Λf\(l\)=R​\(f\(l\)\)/f\(l\)\\Lambda\_\{f\}^\{\(l\)\}=R\(f^\{\(l\)\}\)/f^\{\(l\)\},R​\(Or\(l\)\)=\[12​O\(l\)⊙\(Λf\(l\)​\(WO\(l\)\)⊤\)\]rR\(O\_\{r\}^\{\(l\)\}\)=\\left\[\\frac\{1\}\{2\}O^\{\(l\)\}\\odot\\left\(\\Lambda\_\{f\}^\{\(l\)\}\(W\_\{O\}^\{\(l\)\}\)^\{\\top\}\\right\)\\right\]\_\{r\},ΛO,r\(l\)=R​\(Or\(l\)\)/Or\(l\)\\Lambda\_\{O,r\}^\{\(l\)\}=R\(O\_\{r\}^\{\(l\)\}\)/O\_\{r\}^\{\(l\)\}, andΦr\(l\)=\(Ar\(l\)⊙\[ΛO,r\(l\)​\(Vr\(l\)\)⊤\]\)/Er\(l\)\\Phi\_\{r\}^\{\(l\)\}=\\left\(A\_\{r\}^\{\(l\)\}\\odot\\left\[\\Lambda\_\{O,r\}^\{\(l\)\}\(V\_\{r\}^\{\(l\)\}\)^\{\\top\}\\right\]\\right\)/E\_\{r\}^\{\(l\)\}\. We further defineΘQ,r\(l\)=Φr\(l\)​Kr\(l\)\\Theta\_\{Q,r\}^\{\(l\)\}=\\Phi\_\{r\}^\{\(l\)\}K\_\{r\}^\{\(l\)\},ΘK,r\(l\)=\(Φr\(l\)\)⊤​Qr\(l\)\\Theta\_\{K,r\}^\{\(l\)\}=\(\\Phi\_\{r\}^\{\(l\)\}\)^\{\\top\}Q\_\{r\}^\{\(l\)\}, andΘV,r\(l\)=\(Ar\(l\)\)⊤​ΛO,r\(l\)\\Theta\_\{V,r\}^\{\(l\)\}=\(A\_\{r\}^\{\(l\)\}\)^\{\\top\}\\Lambda\_\{O,r\}^\{\(l\)\}\. Letα=18​dhead\\alpha=\\frac\{1\}\{8\\sqrt\{d\_\{\\mathrm\{head\}\}\}\}All divisions are applied element\-wise\. Then the parameters and onput relevance scores of the MHA module are:

R​\(WO\(l\)\)\\displaystyle R\(W\_\{O\}^\{\(l\)\}\)=12​WO\(l\)⊙\[\(O\(l\)\)⊤​Λf\(l\)\],R​\(WQ,r\(l\)\)=α​WQ,r\(l\)⊙\[\(X\(l\)\)⊤​ΘQ,r\(l\)\],\\displaystyle=\\frac\{1\}\{2\}W\_\{O\}^\{\(l\)\}\\odot\\left\[\\left\(O^\{\(l\)\}\\right\)^\{\\top\}\\Lambda\_\{f\}^\{\(l\)\}\\right\],\\quad R\(W\_\{Q,r\}^\{\(l\)\}\)=\\alpha W\_\{Q,r\}^\{\(l\)\}\\odot\\left\[\\left\(X^\{\(l\)\}\\right\)^\{\\top\}\\Theta\_\{Q,r\}^\{\(l\)\}\\right\],R​\(WV,r\(l\)\)\\displaystyle R\(W\_\{V,r\}^\{\(l\)\}\)=14​WV,r\(l\)⊙\[\(X\(l\)\)⊤​ΘV,r\(l\)\],R​\(WK,r\(l\)\)=α​WK,r\(l\)⊙\[\(X\(l\)\)⊤​ΘK,r\(l\)\],\\displaystyle=\\frac\{1\}\{4\}W\_\{V,r\}^\{\(l\)\}\\odot\\left\[\\left\(X^\{\(l\)\}\\right\)^\{\\top\}\\Theta\_\{V,r\}^\{\(l\)\}\\right\],\\quad R\(W\_\{K,r\}^\{\(l\)\}\)=\\alpha W\_\{K,r\}^\{\(l\)\}\\odot\\left\[\\left\(X^\{\(l\)\}\\right\)^\{\\top\}\\Theta\_\{K,r\}^\{\(l\)\}\\right\],R​\(X\(l\)\)\\displaystyle R\(X^\{\(l\)\}\)=X\(l\)⊙∑r=1d/dhead\[α​ΘQ,r\(l\)​\(WQ,r\(l\)\)⊤\+α​ΘK,r\(l\)​\(WK,r\(l\)\)⊤\+14​ΘV,r\(l\)​\(WV,r\(l\)\)⊤\]\.\\displaystyle=X^\{\(l\)\}\\odot\\sum\_\{r=1\}^\{d/d\_\{\\mathrm\{head\}\}\}\\left\[\\alpha\\Theta\_\{Q,r\}^\{\(l\)\}\\left\(W\_\{Q,r\}^\{\(l\)\}\\right\)^\{\\top\}\+\\alpha\\Theta\_\{K,r\}^\{\(l\)\}\\left\(W\_\{K,r\}^\{\(l\)\}\\right\)^\{\\top\}\+\\frac\{1\}\{4\}\\Theta\_\{V,r\}^\{\(l\)\}\\left\(W\_\{V,r\}^\{\(l\)\}\\right\)^\{\\top\}\\right\]\.\(9\)

Moreover, these relevance scores satisfy:

∑R​\(X\(l\)\)\+∑R​\(WO\(l\)\)\+∑r=1d/dhead\[∑R​\(WQ,r\(l\)\)\+∑R​\(WK,r\(l\)\)\+∑R​\(WV,r\(l\)\)\]=∑R​\(f\(l\)\)\.\\sum R\(X^\{\(l\)\}\)\+\\sum R\(W\_\{O\}^\{\(l\)\}\)\+\\sum\_\{r=1\}^\{d/d\_\{\\mathrm\{head\}\}\}\\left\[\\sum R\(W\_\{Q,r\}^\{\(l\)\}\)\+\\sum R\(W\_\{K,r\}^\{\(l\)\}\)\+\\sum R\(W\_\{V,r\}^\{\(l\)\}\)\\right\]=\\sum R\(f^\{\(l\)\}\)\.LoRA\-based attribution\. For LoRA\-adapted MHA, we write each projection asW~O\(l\)=WO\(l\)\+AO\(l\)​BO\(l\)\\widetilde\{W\}\_\{O\}^\{\(l\)\}=W\_\{O\}^\{\(l\)\}\+A\_\{O\}^\{\(l\)\}B\_\{O\}^\{\(l\)\}andW~⋆,r\(l\)=W⋆,r\(l\)\+A⋆,r\(l\)​B⋆,r\(l\)\\widetilde\{W\}\_\{\\star,r\}^\{\(l\)\}=W\_\{\\star,r\}^\{\(l\)\}\+A\_\{\\star,r\}^\{\(l\)\}B\_\{\\star,r\}^\{\(l\)\}, where⋆∈\{Q,K,V\}\\star\\in\\\{Q,K,V\\\}, andA,BA,Bare trainable LoRA parameters\. We compute relevance scores for the frozen backbone parameters, the trainable LoRA parameters and the input\.

LRP Attribution in Transformer Block\.We initialize the output relevance using the predicted next\-token logit,R​\(H\(L\)\)=Zm,j^R\(H^\{\(L\)\}\)=Z\_\{m,\\hat\{j\}\}, wherej^=arg⁡maxj∈\{1,…,\|𝒱\|\}⁡Zm,j\\hat\{j\}=\\arg\\max\_\{j\\in\\\{1,\\dots,\|\\mathcal\{V\}\|\\\}\}Z\_\{m,j\}\. We then extend relevance propagation from individual FFN and MHA modules to the standard Transformer block, as illustrated in Figure[3](https://arxiv.org/html/2605.05285#S4.F3)\. Figure[3](https://arxiv.org/html/2605.05285#S4.F3)\(a\) shows the standard forward pass in Transformer block in LLMs\. Figure[3](https://arxiv.org/html/2605.05285#S4.F3)\(b\) presents our relevance propagation procedure, which decomposes the output relevanceR​\(H\(l\+1\)\)R\(H^\{\(l\+1\)\}\)across the residual, FFN, and MHA branches, assigning relevance scores to both intermediate activations and model parameters\.

GivenR​\(H\(l\+1\)\)R\(H^\{\(l\+1\)\}\), the residual connection first distributes relevance to its additive inputs:

R​\(H\(l\+1\)\)=R​\(H\(l\)\)\+R​\(f\(l\)\)\+R​\(FFN​\(g\(l\)\)\)\.R\(H^\{\(l\+1\)\}\)=R\(H^\{\(l\)\}\)\+R\(f^\{\(l\)\}\)\+R\\big\(\\mathrm\{FFN\}\(g^\{\(l\)\}\)\\big\)\.
TheR​\(FFN​\(g\(l\)\)\)R\\big\(\\mathrm\{FFN\}\(g^\{\(l\)\}\)\\big\)is then propagated through the FFN module using the proposition[4\.2](https://arxiv.org/html/2605.05285#S4.Thmtheorem2):

R​\(FFN​\(g\(l\)\)\)→\{R​\(g\(l\)\),R​\(W1\(l\)\),R​\(W2\(l\)\),R​\(b1\(l\)\),R​\(b2\(l\)\)\}\.R\(\\mathrm\{FFN\}\(g^\{\(l\)\}\)\)\\;\\rightarrow\\;\\big\\\{R\(g^\{\(l\)\}\),\\;R\(W\_\{1\}^\{\(l\)\}\),\\;R\(W\_\{2\}^\{\(l\)\}\),\\;R\(b\_\{1\}^\{\(l\)\}\),\\;R\(b\_\{2\}^\{\(l\)\}\)\\big\\\}\.We treat layer normalization as identity relevance propagation, yieldingR​\(g\(l\)\)=R​\(H\(l\)\)\+R​\(f\(l\)\)R\(g^\{\(l\)\}\)=R\(H^\{\(l\)\}\)\+R\(f^\{\(l\)\}\)\. ThenR​\(f\(l\)\)R\(f^\{\(l\)\}\)is propagated through the MHA module, yielding both input and parameter relevance using the proposition[4\.3](https://arxiv.org/html/2605.05285#S4.Thmtheorem3):

R​\(f\(l\)\)→\{R​\(X\(l\)\),R​\(WO\(l\)\),\{R​\(WQ,r\(l\)\),R​\(WK,r\(l\)\),R​\(WV,r\(l\)\)\}r\},R\(f^\{\(l\)\}\)\\;\\rightarrow\\;\\big\\\{R\(X^\{\(l\)\}\),\\;R\(W\_\{O\}^\{\(l\)\}\),\\;\\\{R\(W\_\{Q,r\}^\{\(l\)\}\),R\(W\_\{K,r\}^\{\(l\)\}\),R\(W\_\{V,r\}^\{\(l\)\}\)\\\}\_\{r\}\\big\\\},whereX\(l\)=LN​\(H\(l\)\)X^\{\(l\)\}=\\mathrm\{LN\}\(H^\{\(l\)\}\), andR​\(X\(l\)\)=R​\(H\(l\)\)R\(X^\{\(l\)\}\)=R\(H^\{\(l\)\}\)\.

Combining all branches, we obtain relevance scores for both the inputH\(l\)H^\{\(l\)\}and all model parameters in thell\-th Transformer block\. We denote the parameter set by𝒲\(l\)=\{WO\(l\),\{WQ,r\(l\),WK,r\(l\),WV,r\(l\)\}r,W1\(l\),W2\(l\),b1\(l\),b2\(l\)\}\\mathcal\{W\}^\{\(l\)\}=\\Big\\\{W\_\{O\}^\{\(l\)\},\\\{W\_\{Q,r\}^\{\(l\)\},W\_\{K,r\}^\{\(l\)\},W\_\{V,r\}^\{\(l\)\}\\\}\_\{r\},W\_\{1\}^\{\(l\)\},W\_\{2\}^\{\(l\)\},b\_\{1\}^\{\(l\)\},b\_\{2\}^\{\(l\)\}\\Big\\\}, and their corresponding relevance scores byR​\(𝒲\(l\)\)=\{R​\(WO\(l\)\),\{R​\(WQ,r\(l\)\),R​\(WK,r\(l\)\),R​\(WV,r\(l\)\)\}r,R​\(W1\(l\)\),R​\(W2\(l\)\),R​\(b1\(l\)\),R​\(b2\(l\)\)\}R\(\\mathcal\{W\}^\{\(l\)\}\)=\\Big\\\{R\(W\_\{O\}^\{\(l\)\}\),\\\{R\(W\_\{Q,r\}^\{\(l\)\}\),R\(W\_\{K,r\}^\{\(l\)\}\),R\(W\_\{V,r\}^\{\(l\)\}\)\\\}\_\{r\},R\(W\_\{1\}^\{\(l\)\}\),R\(W\_\{2\}^\{\(l\)\}\),R\(b\_\{1\}^\{\(l\)\}\),R\(b\_\{2\}^\{\(l\)\}\)\\Big\\\}\.R​\(𝒲\(l\)\)R\(\\mathcal\{W\}^\{\(l\)\}\)is further used to guide parameter updates in the continual learning fine\-tuning process\.

![Refer to caption](https://arxiv.org/html/2605.05285v1/figs/NIPS_figure1.png)Figure 3:Forward propagation and LRP\-based parameter Attribution in an LLM Transformer Block\.

### 4\.2Attribution guided fine\-tuning for continual learning

Importance prior estimation for each task\. For each task, we first estimate parameter importance in a single\-task setting and use it as a prior for continual learning\. Given task𝒯t\\mathcal\{T\}\_\{t\}with dataset𝒟t=\{\(x\(k\),y\(k\)\)\}k=1Nt\\mathcal\{D\}\_\{t\}=\\\{\(x^\{\(k\)\},y^\{\(k\)\}\)\\\}\_\{k=1\}^\{N\_\{t\}\}, we train a task\-specific model with parametersθt′\\theta\_\{t\}^\{\\prime\}\. Let𝒲\(l,t\)\\mathcal\{W\}^\{\(l,t\)\}denote the parameters of thell\-th Transformer block, including attention projections, feed\-forward weights, and biases\. To obtain reliable attribution signals, we only consider samples correctly generated by the task\-specific model𝒞t\\mathcal\{C\}\_\{t\}and normalize each sample\-level relevance map as

R^\(k\)​\(W\(l,t\)\)=sign⁡\(R\(k\)​\(W\(l,t\)\)\)⊙log⁡\(1\+\|R\(k\)​\(W\(l,t\)\)\|\)max⁡log⁡\(1\+\|R\(k\)​\(W\(l,t\)\)\|\)\+ϵ\.\\widehat\{R\}^\{\(k\)\}\(W^\{\(l,t\)\}\)=\\operatorname\{sign\}\\\!\\left\(R^\{\(k\)\}\(W^\{\(l,t\)\}\)\\right\)\\odot\\frac\{\\log\\\!\\left\(1\+\\left\|R^\{\(k\)\}\(W^\{\(l,t\)\}\)\\right\|\\right\)\}\{\\max\\log\\\!\\left\(1\+\\left\|R^\{\(k\)\}\(W^\{\(l,t\)\}\)\\right\|\\right\)\+\\epsilon\}\.\(10\)Let𝒦u​\(W\(l,t\)\)=TopKk∈𝒞t⁡\(R^\(k\)​\(W\(l,t\)\)​\[u\]\)\\mathcal\{K\}\_\{u\}\(W^\{\(l,t\)\}\)=\\operatorname\{TopK\}\_\{k\\in\\mathcal\{C\}\_\{t\}\}\\left\(\\widehat\{R\}^\{\(k\)\}\(W^\{\(l,t\)\}\)\[u\]\\right\)and we aggregate sample\-level attributions into a task\-level importance prior by retaining the top\-KKnormalized relevance values :

R​\(W\(l,t\)\)​\[u\]=1K​∑k∈𝒦u​\(W\(l,t\)\)R^\(k\)​\(W\(l,t\)\)​\[u\]\.R\(W^\{\(l,t\)\}\)\[u\]=\\frac\{1\}\{K\}\\sum\_\{k\\in\\mathcal\{K\}\_\{u\}\(W^\{\(l,t\)\}\)\}\\widehat\{R\}^\{\(k\)\}\(W^\{\(l,t\)\}\)\[u\]\.\(11\)WhereTopK\\operatorname\{TopK\}returns the indices of theKKlargest values\. The element\-wise importanceR​\(W\(l,t\)\)R\(W^\{\(l,t\)\}\)quantifies which parameters are important in task𝒯t\\mathcal\{T\}\_\{t\}\.

Attribution\-guided continual learningDuring continual learning, for the tasktt, theR​\(W\(l,1\)\),…,R​\(W\(l,t−1\)\)R\(W^\{\(l,1\)\}\),\\dots,R\(W^\{\(l,t\-1\)\}\)are used to protect parameters that are critical to previously learned tasks\. When learning the current task𝒯t\\mathcal\{T\}\_\{t\}, we summarize historical importance for each parameter by taking the element\-wise maximum over all prior tasks:

R¯​\(W\(l,t\)\)=\[maxτ<t⁡R​\(W\(l,τ\)\)\]\+,R¯​\(W\(l,t\)\)∈\[0,1\],\\bar\{R\}\(W^\{\(l,t\)\}\)=\\left\[\\max\_\{\\tau<t\}R\(W^\{\(l,\\tau\)\}\)\\right\]\_\{\+\},\\quad\\bar\{R\}\(W^\{\(l,t\)\}\)\\in\[0,1\],\(12\)where\[x\]\+=max⁡\(x,0\)\[x\]\_\{\+\}=\\max\(x,0\)is applied element\-wise\. We incorporate this prior by modulating the gradient during fine\-tuning:

∇~W\(l,t\)​ℒt=\(𝟏−R¯​\(W\(l,t\)\)\)⊙∇W\(l,t\)ℒt\.\\widetilde\{\\nabla\}\_\{W^\{\(l,t\)\}\}\\mathcal\{L\}\_\{t\}=\\left\(\\mathbf\{1\}\-\\bar\{R\}\(W^\{\(l,t\)\}\)\\right\)\\odot\\nabla\_\{W^\{\(l,t\)\}\}\\mathcal\{L\}\_\{t\}\.\(13\)Here,\(𝟏−R¯​\(W\(l,t\)\)\)\(\\mathbf\{1\}\-\\bar\{R\}\(W^\{\(l,t\)\}\)\)serves as an element\-wise gradient gate\. Parameters that are important to any previous task have large historical relevance, making the gate close to zero and thus receiving smaller updates\. Parameters with low historical relevance keep gates close to one, so their gradients are largely unchanged when learning the current task\.

## References

- I\. Abbes, G\. Subbaraj, M\. Riemer, N\. Islah, B\. Therien, T\. Tabaru, H\. Kingetsu, S\. Chandar, and I\. Rish \(2025\)Revisiting replay and gradient alignment for continual pre\-training of large language models\.arXiv preprint arXiv:2508\.01908\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- Attnlrp: attention\-aware layer\-wise relevance propagation for transformers\.arXiv preprint arXiv:2402\.05602\.Cited by:[§4\.1\.1](https://arxiv.org/html/2605.05285#S4.SS1.SSS1.p3.5)\.
- M\. Aubry, H\. Meng, A\. Sugolov, and V\. Papyan \(2024\)Transformer block coupling and its correlation with generalization in llms\.arXiv preprint arXiv:2407\.07810\.Cited by:[§3](https://arxiv.org/html/2605.05285#S3.p1.5)\.
- S\. Bach, A\. Binder, G\. Montavon, F\. Klauschen, K\. Müller, and W\. Samek \(2015\)On pixel\-wise explanations for non\-linear classifier decisions by layer\-wise relevance propagation\.PloS one10\(7\),pp\. e0130140\.Cited by:[§4\.1](https://arxiv.org/html/2605.05285#S4.SS1.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- W\. Chen, Y\. Zhou, N\. Du, Y\. Huang, J\. Laudon, Z\. Chen, and C\. Cui \(2023\)Lifelong language pretraining with distribution\-specialized experts\.InInternational Conference on Machine Learning,pp\. 5383–5395\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- H\. W\. Chung, L\. Hou, S\. Longpre, B\. Zoph, Y\. Tay, W\. Fedus, Y\. Li, X\. Wang, M\. Dehghani, S\. Brahma,et al\.\(2024\)Scaling instruction\-finetuned language models\.Journal of Machine Learning Research25\(70\),pp\. 1–53\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- B\. Dhingra, J\. R\. Cole, J\. M\. Eisenschlos, D\. Gillick, J\. Eisenstein, and W\. W\. Cohen \(2022\)Time\-aware language models as temporal knowledge bases\.Transactions of the Association for Computational Linguistics10,pp\. 257–273\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- S\. Dou, E\. Zhou, Y\. Liu, S\. Gao, W\. Shen, L\. Xiong, Y\. Zhou, X\. Wang, Z\. Xi, X\. Fan,et al\.\(2024\)LoRAMoE: alleviating world knowledge forgetting in large language models via moe\-style plugin\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1932–1945\.Cited by:[§2](https://arxiv.org/html/2605.05285#S2.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p4.5)\.
- Y\. He, W\. Zhang, X\. Huang, P\. Zhang, L\. Meng, X\. Zhou, K\. Zeng, and X\. Cai \(2025\)Don’t half\-listen: capturing key\-part information in continual instruction tuning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 23649–23668\.External Links:[Link](https://aclanthology.org/2025.acl-long.1153/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1153),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.05285#S2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p4.5)\.
- J\. Huang, L\. Cui, A\. Wang, C\. Yang, X\. Liao, L\. Song, J\. Yao, and J\. Su \(2024a\)Mitigating catastrophic forgetting in large language models with self\-synthesized rehearsal\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 1416–1428\.External Links:[Link](https://aclanthology.org/2024.acl-long.77/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.77)Cited by:[§2](https://arxiv.org/html/2605.05285#S2.p1.1)\.
- J\. Huang, L\. Cui, A\. Wang, C\. Yang, X\. Liao, L\. Song, J\. Yao, and J\. Su \(2024b\)Mitigating catastrophic forgetting in large language models with self\-synthesized rehearsal\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1416–1428\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- B\. Hui, J\. Yang, Z\. Cui, J\. Yang, D\. Liu, L\. Zhang, T\. Liu, J\. Zhang, B\. Yu, K\. Lu,et al\.\(2024\)Qwen2\. 5\-coder technical report\.arXiv preprint arXiv:2409\.12186\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- Z\. Li and D\. Hoiem \(2017\)Learning without forgetting\.IEEE transactions on pattern analysis and machine intelligence40\(12\),pp\. 2935–2947\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- W\. Lu, R\. K\. Luu, and M\. J\. Buehler \(2025a\)Fine\-tuning large language models for domain adaptation: exploration of training strategies, scaling, model merging and synergistic capabilities\.npj Computational Materials11\(1\),pp\. 84\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- Y\. Lu, B\. Qian, C\. Yuan, H\. Jiang, and X\. Wang \(2025b\)Controlled low\-rank adaptation with subspace regularization for continued training on large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19165–19181\.External Links:[Link](https://aclanthology.org/2025.acl-long.940/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.940),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.05285#S2.p1.1)\.
- J\. L\. McClelland, B\. L\. McNaughton, and R\. C\. O’Reilly \(1995\)Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory\.\.Psychological review102\(3\),pp\. 419\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- M\. McCloskey and N\. J\. Cohen \(1989\)Catastrophic interference in connectionist networks: the sequential learning problem\.InPsychology of learning and motivation,Vol\.24,pp\. 109–165\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- S\. Rebuffi, A\. Kolesnikov, G\. Sperl, C\. H\. Lampert,et al\.\(2024\)Incremental classifier and representation learning\.InConference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 5533–5542\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- B\. Roziere, J\. Gehring, F\. Gloeckle, S\. Sootla, I\. Gat, X\. E\. Tan, Y\. Adi, J\. Liu, R\. Sauvestre, T\. Remez,et al\.\(2023\)Code llama: open foundation models for code\.arXiv preprint arXiv:2308\.12950\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- T\. Scialom, T\. Chakrabarty, and S\. Muresan \(2022\)Fine\-tuned language models are continual learners\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 6107–6122\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- F\. Sun, C\. Ho, and H\. Lee \(2019\)Lamol: language modeling for lifelong language learning\.arXiv preprint arXiv:1909\.03329\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- G\. M\. Van de Ven, T\. Tuytelaars, and A\. S\. Tolias \(2022\)Three types of incremental learning\.Nature Machine Intelligence4\(12\),pp\. 1185–1197\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024\)A comprehensive survey of continual learning: theory, method and application\.IEEE transactions on pattern analysis and machine intelligence46\(8\),pp\. 5362–5383\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p1.1)\.
- H\. Zhang, L\. Gui, Y\. Zhai, H\. Wang, Y\. Lei, and R\. Xu \(2023\)COPR: continual learning human preference through optimal policy regularization\.arXiv preprint arXiv:2310\.15694\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1)\.
- W\. Zhao, S\. Wang, Y\. Hu, Y\. Zhao, B\. Qin, X\. Zhang, Q\. Yang, D\. Xu, and W\. Che \(2024\)Sapt: a shared attention framework for parameter\-efficient continual learning of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11641–11661\.Cited by:[§2](https://arxiv.org/html/2605.05285#S2.p1.1)\.
- J\. Zheng, X\. Cai, S\. Qiu, and Q\. Ma \(2025\)Spurious forgetting in continual learning of language models\.arXiv preprint arXiv:2501\.13453\.Cited by:[§1](https://arxiv.org/html/2605.05285#S1.p2.1),[§2](https://arxiv.org/html/2605.05285#S2.p1.1)\.