Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning

# Relative Parameter Importance in Task-Agnostic Replay-Free Continual Learning
Source: [https://arxiv.org/html/2608.00630](https://arxiv.org/html/2608.00630)
11institutetext:Robert Gordon University11email:m\.suresh@rgu\.ac\.uk###### Abstract

Achieving continual learning \(CL\) with deep neural networks requires balancing stability and plasticity while enabling knowledge transfer\. In this work, we focus on offline learning algorithms under the constraints: \(I\) no access to training data from prior tasks \(II\) no access to task\-id at inference time\. We introduce a novel measure, therelative parameter\-importance, which measures the relative importance of each parameter with respect to both the current and past tasks\. Parameters with high relative importance are interpreted as more important for maintaining past\-task stability and thus heavily regularised, whereas parameters with low relative\-importance are allowed to be more freely updated\. Unlike existing methods, our approach allows the update of parameters with high past\-task importance when they have low relative\-importance, thus enabling backward knowledge transfer in addition to tackling the stability\-plasticity trade\-off\. We demonstrate improvements against state\-of\-the\-art CL methods on both class\-incremental and domain\-incremental learning text classification problems111Code available at:https://github\.com/itsmemala/LACLand provide insights for extending our method to text generation problems\.

## 1Introduction

Avoiding the Catastrophic Forgetting \(CF\) of learnt abilities when training on new data \(i\.e\. the stability\-plasticity trade\-off\) and enabling Knowledge Transfer \(KT\) between old and new data are crucial elements to bridging the gap between Continual Learning \(CL\) and Multi\-Task Learning \(MTL\) in deep neural networks\.Experience replay\[dautumn2019mbpa,buzzega2020der\+\+\]andparameter masking\[serra2018hat,ke2021ctr\]are two common CL paradigms but they pose practical challenges\. Replay requires storing data from prior tasks which incurs memory costs and violates data privacy, while parameter masking requires knowing which task \(i\.e\. a task\-id\) the inference sample originates from, which is often practically infeasible\. In this work, we focus on the challenging setting of CL with \(I\) no access to past\-task data \(replay\-free\) and \(II\) no access to task\-id \(task\-agnostic\) at inference time\.

Traditionalgradient projection\[zeng2019owm,saha2021gpm\]andregularisationmethods\[kirkpatrick2017ewc,Li2016LWF\]tackle CL under constraints I and II but focus mainly on preventing CF\. Gradient projection methods restrict gradient updates for new tasks so they do not overlap with directions important to past tasks \(using orthogonality constraints\), while regularisation methods restrict changes to model parameters or layer representations deemed important for past tasks \(using regularisation objectives\)\. These constraints restrict model plasticity and KT, particularly when new tasks are correlated with past tasks\. To address this, some approaches relax the constraint based on task correlation\[lin2022trgp,cheng2025adabop\], or perform a joint regularisation of the main network towards both the frozen past\-task model and an auxiliary model trained solely on the new task\[kim2023ancl\]\. The success of these methods, however, depends on selecting optimal orthogonality/regularisation hyper\-parameters\. In the replay\-free setting, it is difficult to estimate the impact of such hyper\-parameters on past\-task performance \(i\.e\. knowledge transfer versus forgetting\), which risks sub\-optimal performance in practice\. To address this concern, we propose a granular parameter\-level solution that identifies and enables the update of parameters that facilitate backward knowledge transfer while ensuring that parameters more likely to cause forgetting are regularised\.

Our proposed method consists of two training phases for each new task introduced to the continual learner \- \(i\) Look\-Ahead \(LA\) Phase: A temporary auxiliary model is initialised with the frozen past\-task model and trained solely on the new task data, after which importance estimates for each model parameter are derived using a novel notion ofrelative\-importance\. \(ii\) Main Continual Learning \(MCL\) Phase: The main continual learner is trained with a relative\-importance based regularisation objective\. Unlike the traditional parameter regularisation methods\[kirkpatrick2017ewc,aljundi2018mas,chaudhry2018CLintransigence\], that only estimate the importance with respect to past tasks, here we define therelative\-importanceof a parameter as the ratio of the past\-task importance to the sum of the past\-task importance and current\-task importance\. Parameters with high relative\-importance can thus be interpreted as more relevant for maintaining past\-task stability and therefore updates to these parameters can to be regularised to prevent CF\. Parameters with low relative\-importance, on the other hand, can be interpreted as contributing to the learning of new task as well as shared knowledge and can be less regularised for allowing plasticity and backward knowledge transfer\. Our goal is to perform aninformedregularisation of parameters in order to improve performance\.

Lastly, the rise in the use of pre\-trained generative models has led to the emergence of new CL areas such ascontinual pre\-training\[abbes2025continualpretraining\]andcontinual fine\-tuningto add new abilities\[sanyal2025upweighting\]and ensure alignment\[lu2024onlinemergingoptimizersboosting\]\. Research towards employing traditional CL techniques to generative models is still naive\. Moreover, lack of access to pre\-training data and the large overlap in core language/vision abilities across different tasks make replay, parameter\-masking and gradient projection difficult to implement\. In this work, we analyse how parameter importance based regularisation can be extended to generative language models and provide insights to help drive research in this direction\.

In summary, we make the following contributions:

1. 1\.We introduce a novel relative parameter\-importance measure for replay\-free, task\-id\-free continual learning, estimating each parameter’s significance for both current and past tasks to enable informed regularisation\.
2. 2\.We evaluate our proposed approach against state\-of\-the\-art CL methods and show that our method leads to improved performance on both domain\-incremental and class\-incremental learning text classification scenarios\.
3. 3\.We discuss insights for extending our method to text generation scenarios\.

## 2Background and Related Work

Replaymethods store some data from past tasks for re\-use to mitigate CF\[dautumn2019mbpa,lin2024DGC\]\. This incurs high memory costs over several tasks and also violates data privacy\. While some propose using generative models for synthetic replay\[sun2020lamol\], or using activations for replay\[sarfraz2025sarl\], it is hard to ensure generation accuracy and to apply similarity constraints on activations when old and new task domains contain same classes\.Parameter maskingapproaches\[serra2018hat,kang2022wsn,hu2024osn\]use masks during forward/backward propagation to prevent re\-use of parameters important to past tasks, resulting in task\-specific sub\-networks\. Some methods\[ke2021kan,ke2021ctr\]also allow parameter sharing between tasks\. In all these methods, to identify the right sub\-network at test time, a task\-id must be selected\. This trivializes the problem by identifying a subset of classes rather than classifying among all seen classes\[chaudhry2018CLintransigence\]\. While some methods\[wortsman2020supsup,rajasegaran2020itaml,Dekhovich2023cps\]predict a task\-id, they require a batch making them impractical when only a single test sample is presented\.Gradient projectionmethods\[lopez2017gem,zeng2019owm,saha2021gpm\]ensure that updates from a new task are orthogonal to the subspace spanned by previous task inputs to minimise CF\. However, this affects KT and plasticity as parameter updates are increasingly restricted to directions orthogonal to all previous tasks\. Though several methods\[lin2022trgp,yang2025flatnessawareOGP,cheng2025adabop\]propose to tackle this by relaxing the orthogonality constraint, they can still result in sub\-optimal learning when there is high overlap between tasks, as shown in our experiments\.Representation regularisationmethods\[Li2016LWF,Szatkowski2024TAKD\]employ regularisation objectives to tackle CF by preventing change in the model representations corresponding to old tasks when learning the new task\. To enable KT, some works\[ebrahimi2020ACL,luo2024accl\]decompose the representation space into shared and task\-specific features and use additional objectives to encourage similarity between shared features\. Extracting task\-specific features at test\-time, however, requires a task\-id\.

Parameter importance based regularisationmethods estimate the importance of each parameter\[kirkpatrick2017ewc,aljundi2018mas,chaudhry2018CLintransigence\]or network node\[ahn2019ucl,jung2020agscl\]and regularise the learning rate for parameters important to past tasks to minimise CF\. Some methods directly fuse past\-task and current\-task parameters with importance\-based weighting\[sun2024RP2F\], or perform importance\-based gradient updates\[elsayed2024UPGD\]\. In all these methods, parameters with high past\-task importance arealwaysprevented from being updated, which over\-constrains the network and affects plasticity and KT\. Our proposed method tackles this issue by introducing the notion ofrelative\-importance\. Unlike other auxiliary model based methods such as\[kim2023ancl\], that employ two regularisation terms, we derive a single importance estimate for each parameter and use onlyoneregularisation term\. This enables a moreinformedregularisation and avoids the complexity of balancing multiple objectives\.

## 3Methodology

##### Notations

Consider N sequential tasks\{T1,T2,…,TN\}\\\{T\_\{1\},T\_\{2\},\.\.\.,T\_\{N\}\\\}\. Each taskTkT\_\{k\}consists of a traint​rktr\_\{k\}, a validationvkv\_\{k\}and a testtkt\_\{k\}split\. We denote the model trained onTkT\_\{k\}asMkM\_\{k\}and the model parameters asθk\\theta\_\{k\}\. The goal is to trainMkM\_\{k\}while retaining its performance on tasks\{T1,…,Tk−1\}\\\{T\_\{1\},\.\.\.,T\_\{k\-1\}\\\}without access to their data\.Mkl​aM\_\{k\}^\{la\}denotes the model at the end of the look\-ahead phase for thekt​hk^\{th\}task\.

### 3\.1Overview: Continual Learning with Look\-Ahead \(LA\)

![Refer to caption](https://arxiv.org/html/2608.00630v1/figures/Overview_Fitted.png)Figure 1:Look\-Ahead \(LA\) continual learning\. The modelM1M\_\{1\}is trained on the first task using a cross\-entropy loss\. For each subsequent taskkk, model training is done in two phases \- \(i\) Look\-Ahead Phase: A temporary modelMkl​aM\_\{k\}^\{la\}is initialised usingMk−1M\_\{k\-1\}and trained solely on the new task with no regularisation\. At the end of this phase, the relative importance based regularisation weights \(αk′\\alpha\_\{k\}^\{\\prime\}\) are estimated\. \(ii\) Main Continual Learning Phase: The main modelMkM\_\{k\}is initialised usingMk−1M\_\{k\-1\}and trained with relative importance based regularisation\.Figure[1](https://arxiv.org/html/2608.00630#S3.F1)depicts the training process\. We assume a fixed capacity model\. Each task is trained using a task\-specific cross\-entropy loss \(Lc​eL\_\{ce\}\)\. At the end of the training for each taskkk, the importance of each parameterjjis estimated with respect to that task, denoted asαk,j\\alpha\_\{k,j\}\. Here, we use the method of Memory Aware Synapses \(MAS\)\[aljundi2018mas\], which computes the importance as the sensitivity of the learned output function to the parameter, as shown in Equation[1](https://arxiv.org/html/2608.00630#S3.E1), i\.e\. the gradient when back\-propagating theL2L\_\{2\}\-norm of the output logits\.

αk,j=∂Mk​\(t​rk\)∂θk,j\\alpha\_\{k,j\}=\\frac\{\\partial M\_\{k\}\(tr\_\{k\}\)\}\{\\partial\\theta\_\{k,j\}\}\(1\)
For simplicity,αk\\alpha\_\{k\}is used to denote the set of all parameter importance values\{αk,j\}\\\{\\alpha\_\{k,j\}\\\}for thekt​hk^\{th\}task\. The importance values are aggregated across tasks \(to avoid memory cost of storing values for each task\) by mean\-pooling, such thatαk=m​e​a​n​\{α0,α1,…,αk\}\\alpha\_\{k\}=mean\\\{\\alpha\_\{0\},\\alpha\_\{1\},\.\.\.,\\alpha\_\{k\}\\\}\. This ensures that parameters that continue to be important for multiple tasks retain higher importance values and are better protected through importance\-based regularisation\.222It is noted that mean\-pooling leads to loss of some granularity\. While max\-pooling is an alternative, we believe it could cause early capacity saturation through high importance for most parameters \- we leave further exploration to future work\.

For the second task onwards, training is conducted in two phases \- the look\-ahead training \(l​ala\) phase and the main continual learning \(m​c​lmcl\) phase\. During the look\-ahead phase, the model is optimised solely on the new task without any regularisation using Equation[2](https://arxiv.org/html/2608.00630#S3.E2)\.

Ll​a=Lt​a​s​k=Lc​eL\_\{la\}=L\_\{task\}=L\_\{ce\}\(2\)
The look\-ahead training phase acts as a performance baseline and also identifies the gradient update path taken by the new task\. At the end of this phase, the importance of parameters with respect to the new task are estimated and used to obtain the relative importance of each parameter with respect to past and current tasks\. This helps inform which parameters are likely to cause CF and which may help with KT\. Based on this, a modified weightα′\\alpha^\{\\prime\}is calculated, which allows parameters likely to help with KT to be easily updated, even if they were of high importance to past tasks\. During the main continual learning phase, this modified weight is used in the regularisation, as shown in equation[3](https://arxiv.org/html/2608.00630#S3.E3), to selectively control the parameter plasticity\. The regularisation in this phase thus focuses on balancing CF and KT, as opposed to merely avoiding CF\. The strength of the regularisation is controlled using the hyper\-parameterλ\\lambda\.

Lm​c​l=Lt​a​s​k\+λ2​∑jαk,j′​\(θk−1,j−θk,j\)2L\_\{mcl\}=L\_\{task\}\+\\dfrac\{\\lambda\}\{2\}\\sum\_\{j\}\\alpha^\{\\prime\}\_\{k,j\}\(\\theta\_\{k\-1,j\}\-\\theta\_\{k,j\}\)^\{2\}\(3\)
At the end of the main continual learning phase, the latest parameter importance with respect to the new taskαk\\alpha\_\{k\}is calculated and mean\-pooled for use in the next task regularisation\. Next, we discuss how the parameter importance values for the main continual learning phaseαk′\\alpha\_\{k\}^\{\\prime\}are estimated\.

### 3\.2Controlling Parameter Plasticity using Relative\-Importance

Letαkl​a\\alpha\_\{k\}^\{la\}denote the importance of parameters with respect to the new taskkkat the end of the look\-ahead training phase\. The relative importance of parametersαkr​e​l\\alpha\_\{k\}^\{rel\}is computed as shown in equation[4](https://arxiv.org/html/2608.00630#S3.E4)and is in the range\[0,1\]\[0,1\]\.ϵ\\epsilonis a small constant to ensure numerical stability, i\.e\. to ensureαkr​e​l=0\\alpha\_\{k\}^\{rel\}=0when bothαk−1=0\\alpha\_\{k\-1\}=0andαkl​a=0\\alpha\_\{k\}^\{la\}=0\.333Sinceα\\alphais computed as the gradient with respect to the parameter, based on the floating point precision used in practice, it can sometimes be zero\. We only observe this for a negligible number of parameters in our experiments and setϵ=1​e−10\\epsilon=1e^\{\-10\}to ensure its effect on the denominator is otherwise negligible\.

αkr​e​l=αk−1αk−1\+αkl​a\+ϵ\\alpha\_\{k\}^\{rel\}=\\dfrac\{\\alpha\_\{k\-1\}\}\{\\alpha\_\{k\-1\}\+\\alpha\_\{k\}^\{la\}\+\\epsilon\}\(4\)
Here, we interpret parameters with a highαkr​e​l\\alpha\_\{k\}^\{rel\}value as more important to preserving past task knowledge than for learning new task knowledge\. Therefore, changes to these parameters must be minimal to help prevent CF\. On the other hand, parameters with a lowαkr​e​l\\alpha\_\{k\}^\{rel\}value are interpreted as more important to new task knowledge accumulation\. Therefore, by allowing these parameters to be updated, we can maintain model plasticity, while potentially also allowing backward KT\.

The new constraintαk′\\alpha\_\{k\}^\{\\prime\}for each parameter is determined by weighting the originalαk−1\\alpha\_\{k\-1\}value usingαkr​e​l\\alpha\_\{k\}^\{rel\}, as shown in Equation[5](https://arxiv.org/html/2608.00630#S3.E5), whereταr​e​l\\tau\_\{\\alpha^\{rel\}\}is the cut\-off for differentiating high and low relative\-importance andλu​p\\lambda\_\{up\}andλd​o​w​n\\lambda\_\{down\}are scaling hyper\-parameters\. The first term focusses on CF avoidance, where the constraint on parameters with high relative\-importance is increased through the weightλu​p\.αkr​e​l\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}\.λu​p\\lambda\_\{up\}is lower bounded by1/ταr​e​l1/\\tau\_\{\\alpha^\{rel\}\}to ensureλu​p\.αkr​e​l\>1\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}\>1\(to mitigate CF\)\. The second term focuses on plasticity and KT, where the constraint on parameters with low relative\-importance is decreased through the weightλd​o​w​n\.αkr​e​l\\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}\.λd​o​w​n\\lambda\_\{down\}is upper bounded by 1 to ensureλd​o​w​n\.αkr​e​l<1\\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}<1\(to enable plasticity/KT\)\. See Appendix[0\.C](https://arxiv.org/html/2608.00630#Pt0.A3)for a detailed empirical analysis of alternative weighting strategies considered\.

αk′=\{λu​p\.αkr​e​l\.αk−1ifαkr​e​l\>ταr​e​lλd​o​w​n\.αkr​e​l\.αk−1otherwise\\alpha\_\{k\}^\{\\prime\}=\\begin\{cases\}\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}\.\\alpha\_\{k\-1\}&\\text\{if $\\alpha\_\{k\}^\{rel\}\>\\tau\_\{\\alpha^\{rel\}\}$\}\\\\ \\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}\.\\alpha\_\{k\-1\}&\\text\{otherwise\}\\end\{cases\}\(5\)
The importance weight in quadratic regularisers can be seen as regulating the learning rate for each parameter\[lubana2021ReguInstability,kim2023ancl\]\. Our approach adjusts this learning rate by taking into account the relative importance of a parameter for current vs past tasks \(throughαkr​e​l\\alpha\_\{k\}^\{rel\}\)\. Algorithm[1](https://arxiv.org/html/2608.00630#alg1)\(Appendix[0\.A](https://arxiv.org/html/2608.00630#Pt0.A1)\) summarises the look\-ahead method, including a replay\-free hyper\-parameter search procedure\.

## 4Text Classification Experiments

This section evaluates the proposed Look\-Ahead method on text classification\.

### 4\.1Experiment Setup

#### 4\.1\.1Datasets

##### 1\. Intent classification \(CIL\)

This involves classifying home assistant queries\[liu2021hwu64\]by their intent type \(e\.g\. ‘set alarm’, ‘play music’\)\. The most frequent intents are split to form five tasks of five intents each\. This dataset models the CIL setting, which is more challenging when not using a task\-id based method\[chaudhry2018CLintransigence\]\.

##### 2\. Behaviour classification \(DIL\)

This involves classifying client behaviour \(as change/neutral/sustain\) in motivational interviewing \(MI\) conversations between a client and a therapist\[wu2022annomi\]in different domains\. This task is important for automating MI conversations and evaluating MI effectiveness\[tavabi2021analysisMIcodes\]\. Data collection often involves focus\-groups of one domain \(e\.g\. alcohol/anxiety\) with high label annotation cost and privacy concerns preventing data sharing\. This motivates a CL approach\. We use the data from\[wu2022annomi\]to form six domains based on the conversation topic \(see Appendix[0\.B\.1](https://arxiv.org/html/2608.00630#Pt0.A2.SS1)\)\. The dataset forms an interesting challenge due to in\-domain class imbalance and cross\-domain sample imbalance\.

Class\-wise examples of each dataset are provided in Table[5](https://arxiv.org/html/2608.00630#Pt0.A2.T5)in Appendix[0\.B\.2](https://arxiv.org/html/2608.00630#Pt0.A2.SS2)\.

#### 4\.1\.2Performance Metrics

Macro\-averaged F1 scores on the test sets are used to measure performance on individual tasks\. LetFk,iF\_\{k,i\}denote the F1 score of the modelMkM\_\{k\}on task i\. The following metrics are used to report CL performance:

1. 1\.Overall \(O​vOv\): Average performance across all tasks at the end of sequence\. 1N​∑i=1NFN,i\\dfrac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}F\_\{N,i\}\(6\)
2. 2\.Catastrophic Forgetting \(C​FCF\): Average performance loss on previous tasks after sequentially learning all tasks\[huang2021idbr\], i\.e\. for each taskii, the performance difference between the final modelNNand the previous best modelkk\. 1N−1​∑i=1N−1\[maxi≤k≤N−1⁡\(Fk,i\)−FN,i\]\\dfrac\{1\}\{N\-1\}\\sum\_\{i=1\}^\{N\-1\}\[\{\\max\_\{i\\leq k\\leq N\-1\}\(F\_\{k,i\}\)\-F\_\{N,i\}\}\]\(7\)
3. 3\.Positive backward transfer \(B​W​T\+BWT^\{\+\}\): Average gain in performance on seen tasks after learning each new task\[diaz2018CLeval\], i\.e\. for each taskii, the gain in performance when going from the previous \(k−1k\-1\) to the current model \(kk\)\. 1\(N​\(N−1\)2\)​∑k=2N∑i=1k−1max⁡\(0,Fk,i−Fk−1,i\)\\dfrac\{1\}\{\(\\dfrac\{N\(N\-1\)\}\{2\}\)\}\\sum\_\{k=2\}^\{N\}\\sum\_\{i=1\}^\{k\-1\}\{\\max\(0,F\_\{k,i\}\-F\_\{k\-1,i\}\}\)\(8\)
4. 4\.Forward transfer \(F​W​TFWT\): Average gain in performance on each task when it is first introduced to the CL model, relative to naive sequential learning \(s​e​qseq\)\[ke2021kan\]\. Positive values imply good model plasticity and KT from prior tasks to current task, while negative values typically imply loss of plasticity\. 1N−1​∑i=2N\[Fi,i−Fi,is​e​q\]\\dfrac\{1\}\{N\-1\}\\sum\_\{i=2\}^\{N\}\[F\_\{i,i\}\-F\_\{i,i\}^\{seq\}\]\(9\)

#### 4\.1\.3Compared Methods

Our main baselines areAdaBOP\[cheng2025adabop\]\(task\-correlation based gradient projection\),UPGD\[elsayed2024UPGD\]\(parameter regularisation\), andRP2F\[sun2024RP2F\]andANCL\[kim2023ancl\]\(auxiliary network based regularisation\)\. We apply ANCL to parameter regularisationMAS\[aljundi2018mas\]and representation regularisationLWF\[Li2016LWF\]\. We report multi\-taskedMTLand naive sequential learningSEQbaselines\.

#### 4\.1\.4Implementation Details

The network backbone is Adapter\-BERT\[houlsby2019bertadapter\], where only the adapters, layer\-norm and classifier head parameters are trainable during CL\. Hyper\-parameter search is implemented withd​r​o​p=10drop=10,t​h​r=90thr=90for CIL andt​h​r=95thr=95for DIL \(per algorithm[1](https://arxiv.org/html/2608.00630#alg1)\)\.ταr​e​l\\tau\_\{\\alpha^\{rel\}\}is dynamically set to 80% of theαr​e​l\\alpha^\{rel\}distribution mean at each model component and layer \(per analysis in Appendix[0\.C](https://arxiv.org/html/2608.00630#Pt0.A3)and Appendix[0\.D\.2](https://arxiv.org/html/2608.00630#Pt0.A4.SS2)\)\. Further details \(task splits, etc\.\) are in Appendix[0\.B\.3](https://arxiv.org/html/2608.00630#Pt0.A2.SS3)\.

### 4\.2Results and Discussion

Table 1:Mean \(std\) performance at the end of all tasks, measured in F1 scores and averaged across three random task orderings\.↑\\uparrowdenotes higher is better,↓\\downarrowdenotes lower is better\. Best is indicated in bold, second best is underlined\.Table[1](https://arxiv.org/html/2608.00630#S4.T1)compares our Look\-Ahead \(LA\) approach to various CL baselines\. On the CIL problem, while some CL methods provide gains over the SEQ baseline, they all fall significantly short of the MTL upper bound, highlighting the difficulty of replay\-free and task\-agnostic CIL\. With the gradient projection based method AdaBOP, model plasticity \(FWT\) is significantly affected by the orthogonality constraint resulting in poor overall performance \(Ov\) even though forgetting \(CF\) is reduced on average compared to SEQ\. RP2F \(auxilliary network based method\) improves plasticity slightly on average compared to SEQ but incurs significant CF resulting in poor overall performance again\. UPGD \(parameter regularisation method\) improves plasticity significantly, resulting in an improved overall performance, though CF still remains high\. For ANCL\-MAS and ANCL\-LWF \(auxilliary network based methods\), since the regularisation hyper\-parameters control the stability\-plasticity trade\-off, a desired level of plasticity can be specified and hyper\-parameters selected accordingly to maximise stability\. The desired plasticity is specified as a percentage threshold of the best achievable performance for the current task \(i\.e\. naive fine\-tuning of the main network on current task\)\. At a high threshold of 90%, both ANCL baselines achieve high plasticity comparable to UPGD, but ANCL\-MAS achieves better CF avoidance leading to better overall performance\. Using the same threshold for hyper\-parameter selection, the LA method reduces CF while also improving backward KT at a similar plasticity level, resulting in the best overall performance\. Appendix[0\.D\.1](https://arxiv.org/html/2608.00630#Pt0.A4.SS1)compares the performance at different thresholds\.

On the DIL problem, with significant shared knowledge across tasks, the gap between MTL and SEQ is reduced\. SEQ already achieves good model plasticity and backward KT, with the CL methods mainly improving performance by minimising CF\. Several methods \(AdaBOP, UPGD, ANCL\-MAS\) reduce CF at the cost of loss to plasticity, resulting in poor overall performance\. RP2F reduces CF without significant impact to plasticity, resulting in a small improvement on average over SEQ\. ANCL\-LWF reduces CF significantly while also maintaining good plasticity and backward KT, resulting in a good overall performance\. The LA method reduces CF further to achieve the best overall performance\.

Appendix[0\.E](https://arxiv.org/html/2608.00630#Pt0.A5)compares the computation cost of the various CL methods\.

## 5Preliminary Analysis for Text Generation

This section performs preliminary analysis for extending our method to generative models\. We first extend the parameter importance computation defined for classification outputs in Section[3\.1](https://arxiv.org/html/2608.00630#S3.SS1)to sequence outputs, as follows: The importance is computed as the gradient when back\-propagating theL2L\_\{2\}\-norm of the output token logits,averaged across all output tokens of the sequence\. Our experiments \(1\) support the proposed importance estimation method and, \(2\) reveal insights for next steps in this direction\.

### 5\.1Experiment Setup

We use the pre\-trained model checkpoint for Llama\-3\.2\-1B\. Our target task is mathematical reasoning \- we fine\-tune on MetaMathQA\[yu2024metamath\]and evaluate the performance on GSM8K\[cobbe2021gsm8k\]\. We consider Python programming \(MBPP\[jacob2021mbpp\]\) as a core pre\-trained ability to retain after fine\-tuning on target task\. We use the evaluation setup of\[sanyal2025upweighting\]for reporting performance on both pre\-trained and target tasks\. We use epochs=2 and batch size=4 for fine\-tuning\.

### 5\.2Results and Discussion

First, we perform naive fine\-tuning on target task and measure the correlation between thechange in parameter magnitudeand theparameter importancewith respect to the target task, across model layers\. The high correlation values in Table[2](https://arxiv.org/html/2608.00630#S5.T2)indicate that our proposed importance estimation method works \- parameters that have changed to adapt to the task are indeed assigned higher importance\. We then compute parameter importance of the pre\-trained model with respect to MBPP \(αj\\alpha\_\{j\}\) and use this for fine\-tuning with regularisation to prevent CF\. We setαj′=αj\\alpha^\{\{\}^\{\\prime\}\}\_\{j\}=\\alpha\_\{j\}for preliminary analysis\. Results in Table[3](https://arxiv.org/html/2608.00630#S5.T3)indicate that the training is unstable for a wide range ofλ\\lambda\. This suggests that to implement parameter regularisation in generative models, a crucial next step for research is to effectively balance the cross\-entropy and regularisation objectives\.

Table 2:Statistics of correlation \(∈\[−1,1\]\\in\[\-1,1\]\) between absolute change in parameter magnitude \(\|Δ​θj\|\|\\Delta\\theta\_\{j\}\|\) and parameter importance values \(αj\\alpha\_\{j\}\), across model layers, after naive fine\-tuning \(FT\) a pre\-trained \(PT\) model for a target task\.Table 3:Results of parameter\-importance regularisation during fine\-tuning\.

## 6Conclusion

In this work, we proposed the Look\-Ahead \(LA\) method, which implements a novelrelative parameter\-importancemeasure for replay\-free and task\-agnostic CL\. Our method estimates the importance of parameters with respect to both the current and past tasks\. Unlike existing methods, we allow parameters with high past\-task importance to be updated when they have low relative\-importance\. Experiments demonstrated improvements using the LA method compared to state\-of\-the\-art baselines on both DIL and CIL text classification\. Further analysis revealed insights for extending our method to text generation\. Finally, we note that our approach can be used with any importance estimation method \(for compute\-accuracy trade\-off\), as well as with replay and task\-id based methods\.

\{credits\}

#### 6\.0\.1\\discintname

The authors have no competing interests to declare that are relevant to the content of this article\.

## References

## Appendix 0\.ALook Ahead Continual Learning with Continual Hyper\-Parameter Search

Selecting the optimal hyper\-parameters without access to past\-task data is challenging since the impact of the hyper\-parameter choice on model stability cannot be directly observed\. The LA approach minimises this challenge by identifying parameters with high relative importance as those that are most likely to impact model stability negatively\. However, suitable hyper\-parameter choices forλ\\lambda,λu​p\\lambda\_\{up\},λd​o​w​n\\lambda\_\{down\}andταr​e​l\\tau\_\{\\alpha^\{rel\}\}still need to be made\. In Appendix[0\.C](https://arxiv.org/html/2608.00630#Pt0.A3), a detailed analysis of hyper\-parameter choices is conducted to understand the impact of different hyper\-parameters on the continual learning performance\. Here, an overall framework is described for selecting the hyper\-parameters for each task in practice\.

The proposed hyper\-parameter selection approach is inspired by the Continual Hyper\-parameter Search Framework \(CHSF\)\[lange2019chsf\]\. At the start of each taskkk, the best achievable performance for the task,A​c​ckAcc\_\{k\}\(on the validation splitvkv\_\{k\}\), is recorded using only cross\-entropy and a grid search on the learning rate\. Then, fork\>1k\>1, hyper\-parameter search forλ\\lambdais performed first by training with equation[10](https://arxiv.org/html/2608.00630#Pt0.A1.E10)\.

Lr​e​g=Lt​a​s​k\+λ2​∑jαk−1,j​\(θk−1,j−θk,j\)2L\_\{reg\}=L\_\{task\}\+\\dfrac\{\\lambda\}\{2\}\\sum\_\{j\}\\alpha\_\{k\-1,j\}\(\\theta\_\{k\-1,j\}\-\\theta\_\{k,j\}\)^\{2\}\(10\)
λ\\lambdais initialised to a high value such that current task performance=0% and iteratively decreased byd​r​o​p%drop\\%until we reach at least threshold percentage \(t​h​r%thr\\%\) ofA​c​ckAcc\_\{k\}\(on the validation splitvkv\_\{k\}\)\. This ensures that the highest possibleλ\\lambdais chosen, ensuring maximum stability at the givent​h​r%thr\\%\. Then, in order to select the LA specific hyper\-parameters fork\>1k\>1, the CHSF framework is extended as follows\. To perform hyper\-parameter search forλd​o​w​n\\lambda\_\{down\}andλu​p\\lambda\_\{up\}\- we initialise first to their maximum values, which ensures maximum stability, and decreaseλd​o​w​n\\lambda\_\{down\}first byd​r​o​p%drop\\%followed byλu​p\\lambda\_\{up\}byd​r​o​p%drop\\%, until we either improve the current task performance or reach a flat performance slope overb​u​d​g​e​tbudgetiterations\.ταr​e​l\\tau\_\{\\alpha^\{rel\}\}is set dynamically for each task based on theαr​e​l\\alpha^\{rel\}distribution \(discussed in section[0\.C](https://arxiv.org/html/2608.00630#Pt0.A3)\)\. Following\[lange2019chsf\],λ\\lambdachosen from previous tasks are propagated to subsequent tasks\.

##### Boundingλu​p\\lambda\_\{up\}

Substitutingαk′\\alpha\_\{k\}^\{\\prime\}from equations[5](https://arxiv.org/html/2608.00630#S3.E5)in equation[3](https://arxiv.org/html/2608.00630#S3.E3), we see thatλu​p\\lambda\_\{up\}andλd​o​w​n\\lambda\_\{down\}act as scaling values forλ\\lambda\. While increasingλu​p\\lambda\_\{up\}amounts to increasing the regularisation constraint on the corresponding parameters \(i\.e\. those with high relative importance\), in practice the values forλ\\lambdacan vary widely based on the task, which can subsequently impact the choice ofλu​p\\lambda\_\{up\}\. Therefore, for simplicity, we derive an upper bound forλu​p\\lambda\_\{up\}asλm​a​x/λ\\lambda\_\{max\}/\\lambda, whereλm​a​x\\lambda\_\{max\}is the largest value ofλ\\lambdabeyond which there is zero plasticity on the current task \(using equation[10](https://arxiv.org/html/2608.00630#Pt0.A1.E10)\)\. The lower bound forλu​p\\lambda\_\{up\}is given by1/ταr​e​l1/\\tau\_\{\\alpha^\{rel\}\}\. Note thatλm​a​x\\lambda\_\{max\}relies only on access to current task data and can easily be determined using a coarse hyper\-parameter search\.

The full Look\-Ahead algorithm, including hyper\-parameter search, is shown in algorithm[1](https://arxiv.org/html/2608.00630#alg1)\.

Algorithm 1Continual Learning with Look\-Ahead1:Model:

θ0\\theta\_\{0\}, Hyper\-parameters:

t​h​r,d​r​o​p,λi​n​i​t,ταr​e​lthr,drop,\\lambda\_\{init\},\\tau\_\{\\alpha^\{rel\}\},

\{l​r1,l​r2,…\}\\\{lr\_\{1\},lr\_\{2\},\.\.\.\\\},

b​u​d​g​e​tbudget,

ϵ\\epsilon\.

2:Accuracy estimation:

A​c​c​\(\)Acc\(\), Parameter importance estimation:

I​m​p​\(\)Imp\(\)\.

3:fortask

k=1,2,…,Nk=1,2,\\dots,Ndo

4:for

l​r∈\{l​r1,l​r2,…\}lr\\in\\\{lr\_\{1\},lr\_\{2\},\.\.\.\\\}do⊳\\trianglerightGrid search for learning rate

5:Initialise

θk=θk−1\\theta\_\{k\}=\\theta\_\{k\-1\}\.

6:Train

θk\\theta\_\{k\}using

Lt​a​s​k=Lc​eL\_\{task\}=L\_\{ce\}and

l​rlr\.⊳\\trianglerightNo regularisation

7:endfor⊳\\trianglerightRecord best accuracyA​c​ckAcc\_\{k\}and learning ratel​rklr\_\{k\}

8:if

k=1k=1then

9:Initialise

θk=θk−1\\theta\_\{k\}=\\theta\_\{k\-1\}\.

10:Train

θk\\theta\_\{k\}using

Lt​a​s​k=Lc​eL\_\{task\}=L\_\{ce\}and

l​rklr\_\{k\}\.⊳\\trianglerightNo regularisation

11:else

12:Initialise

λ=λi​n​i​t\\lambda=\\lambda\_\{init\}\.

13:while

A​c​c​\(θk,vk\)<t​h​r\.A​c​ckAcc\(\\theta\_\{k\},v\_\{k\}\)<thr\.Acc\_\{k\}do⊳\\trianglerightSearch forλ\\lambda

14:Initialise

θk=θk−1\\theta\_\{k\}=\\theta\_\{k\-1\}\.

15:Train

θk\\theta\_\{k\}using eqn[10](https://arxiv.org/html/2608.00630#Pt0.A1.E10)and

l​rklr\_\{k\}\.

16:Update

λ=\(1−d​r​o​p\)\.λ\\lambda=\(1\-drop\)\.\\lambda\.⊳\\trianglerightRecordλ\\lambdawith Acc=0 asλm​a​x\\lambda\_\{max\}

17:endwhile⊳\\trianglerightRecord accuracy withλ\\lambdaasA​c​cλAcc\_\{\\lambda\}

18:Initialise

λi​n​i​t=λ\\lambda\_\{init\}=\\lambda,

θkl​a=θk−1\\theta\_\{k\}^\{la\}=\\theta\_\{k\-1\}\.

19:Train

θkl​a\\theta\_\{k\}^\{la\}using

Lt​a​s​k=Lc​eL\_\{task\}=L\_\{ce\}and

l​rklr\_\{k\}\.⊳\\trianglerightLA Phase

20:Calculate

αkl​a=I​m​p​\(θkl​a\)\\alpha\_\{k\}^\{la\}=Imp\(\\theta\_\{k\}^\{la\}\)and

αkr​e​l\\alpha\_\{k\}^\{rel\}using eqn[4](https://arxiv.org/html/2608.00630#S3.E4)\.

21:Initialise

λd​o​w​n=1\.0\\lambda\_\{down\}=1\.0,

λu​p=λm​a​x/λ\\lambda\_\{up\}=\\lambda\_\{max\}/\\lambda,

x=1x=1\.

22:while

A​c​c​\(θk,vk\)<A​c​cλAcc\(\\theta\_\{k\},v\_\{k\}\)<Acc\_\{\\lambda\}and

x<=b​u​d​g​e​tx<=budgetdo⊳\\trianglerightSearch forλd​o​w​n,λu​p\\lambda\_\{down\},\\lambda\_\{up\}

23:Calculate

αk′\\alpha\_\{k\}^\{\\prime\}using eqn[5](https://arxiv.org/html/2608.00630#S3.E5)\.

24:Train

θk\\theta\_\{k\}using eqn[3](https://arxiv.org/html/2608.00630#S3.E3)and

l​rklr\_\{k\}\.⊳\\trianglerightMCL Phase

25:ifSlope\(

A​c​c​\(θk,vk\)Acc\(\\theta\_\{k\},v\_\{k\}\)\)

<ϵ<\\epsilonthen⊳\\trianglerightPerformance plateaus

26:Update

λu​p=\(1−d​r​o​p\)\.λu​p\\lambda\_\{up\}=\(1\-drop\)\.\\lambda\_\{up\}\.

27:else

28:Update

λd​o​w​n=\(1−d​r​o​p\)\.λd​o​w​n\\lambda\_\{down\}=\(1\-drop\)\.\\lambda\_\{down\}\.

29:endif

30:

x=x\+1x=x\+1
31:endwhile

32:endif

33:Calculate

αk=I​m​p​\(θk\)\\alpha\_\{k\}=Imp\(\\theta\_\{k\}\)and

αk=m​e​a​n​p​o​o​l​\(αk,αk−1\)\\alpha\_\{k\}=meanpool\(\\alpha\_\{k\},\\alpha\_\{k\-1\}\)\.

34:endfor

## Appendix 0\.BExperiment Setup

### 0\.B\.1Behaviour Classification: CL Domain Splits

Table[4](https://arxiv.org/html/2608.00630#Pt0.A2.T4)shows the keywords used for segmenting the data into domains and the class\-split in each domain\.

Table 4:Domain Split
### 0\.B\.2Dataset Examples

Table[5](https://arxiv.org/html/2608.00630#Pt0.A2.T5)shows some examples from the two datasets\.

Table 5:Dataset Examples
### 0\.B\.3Implementation Details

##### Compared methods implementation

We implement RP2F with fisher parameter importance, which is more computationally efficient and only marginally worse than using the parameter\-perturbation based importance\.444Based on results provided by authors\.For UPGD, we use the second\-order utility function as proposed by authors\.

##### Code

Code for UPGD, RP2F, AdaBOP and ANCL are adapted for text classification with the Adapter\-BERT architecture using the description in the paper and code provided by the authors\.

##### Common training hyper\-parameters

Across all methods, for each task we conduct a grid search on the learning rate∈\\in\{0\.00003, 0\.0003, 0\.003, 0\.03\}\. Models are trained for a maximum of 50 epochs with early stopping based on decrease in validation data loss\.

##### Hyper\-parameter search for ANCL

ANCL requires setting two hyper\-parameters \-λ\\lambda, which controls the strength of the regularisation towards the frozen past task model \(i\.e\. controls stability\), andλa\\lambda\_\{a\}, which controls the strength of the regularisation towards the auxiliary network trained on the new task \(i\.e\. controls plasticity\)\. Similar to the LA method, the CHSF framework is used whereλ\\lambdais selected first usingd​r​o​p=10drop=10, followed byλa\\lambda\_\{a\}, which is initialised toλa=0\.01\\lambda\_\{a\}=0\.01and increased byi​n​c=10inc=10until current task performance is improved\. For comparison to the LA method, results are reported for the same thresholds\.

##### Hyper\-parameter search for RP2F

For selectingλ\\lambda, which controls the strength of the parameter robustness term in the training objective function, a grid search∈\{1​e−5,1​e−6,1​e−7\}\\in\\\{1e\-5,1e\-6,1e\-7\\\}is performed for each task, as suggested by the authors, using the current task validation set performance for selection\. Note that unlike the LA and ANCL methods, this hyper\-parameter does not correlate directly with model stability or plasticity\.

##### Hyper\-parameter search for AdaBOP

AdaBOP requires selecting four hyper\-parameters \-s​v​dt​h​r∈\[0,1\]svd\_\{thr\}\\in\[0,1\], which represents the % of bottom right singular vectors \(after using svd on the task feature space\) that are selected as the task null space,λs\\lambda\_\{s\}andλl\\lambda\_\{l\}, which control the strength of the orthogonality constraint \(when projecting new task gradients to previous task null space\) \(i\.e\. controls stability\) for parameters with high and low correlation to past tasks, respectively, andϵ\\epsilon, which is the threshold for differentiating high and low correlation\.s​v​dt​h​rsvd\_\{thr\}is set to 0\.01, \(λs\\lambda\_\{s\},λl\\lambda\_\{l\}\) are selected from a grid search∈\{\(0\.1,0\.3\),\(0\.03,0\.08\),\(0\.005,0\.01\)\}\\in\\\{\(0\.1,0\.3\),\(0\.03,0\.08\),\(0\.005,0\.01\)\\\}, andϵ\\epsilonis selected from a grid search∈\{0\.002,0\.004\}\\in\\\{0\.002,0\.004\\\}, based on values suggested by the authors, using the current task validation set performance for selection\.

##### DIL specific setups

For DIL, class\-weighting is applied to the cross\-entropy loss to tackle the heavy class imbalance\. Class weights are calculated as the scaled inverse frequency of classes in the full dataset\.

##### CIL specific setups

For CIL with single\-head setting, a relaxed\-balanced\-softmax is used per\[JODELET2022RBS\]with the suggested default hyper\-parameter value\.

Table 6:Task Orders; TG \- Toys and Games, MI \- Musical Instruments, DM \- Digital Music, CDV \- CDs and Vinyl, AUTO \- Automotive, AIV \- Amazon Instant Video; ALC \- Alcohol, SM \- Smoking, DR \- Drug, EX \- Exercise, MED \- Medicine, ANX \- Anxiety; SP0 \- \[music, quirky, factoid, remove, negate\], SP1 \- \[praise, sendemail, explain, repeat, affirm\], SP2 \- \[radio, confirm, post, definition, dontcare\], SP3 \- \[recipe, podcasts, currency, events, commandstop\], SP4 \- \[createoradd, stock, locations, hue\_lightoff, audiobook\], SP5 \- \[ticket, game, hue\_lightchange, querycontact, likeness\], SP6 \- \[music, sendemail, post, events, audiobook\], SP7 \- \[quirky, explain, definition, commandstop, ticket\], SP8 \- \[factoid, repeat, dontcare, createoradd, game\], SP10 \- \[negate, radio, podcasts, locations, querycontact\], SP11 \- \[praise, confirm, currency, hue\_lightoff, likeness\]
##### Task Ordering

Table[6](https://arxiv.org/html/2608.00630#Pt0.A2.T6)shows the sequences of task orders used in experiments for each dataset\.

## Appendix 0\.CAnalysis of Parameter Weighting Strategies with Relative\-Importance

To understand the effect of treating parameters with high and lowαkr​e​l\\alpha\_\{k\}^\{rel\}differently, as well as to motivate howαkr​e​l\\alpha\_\{k\}^\{rel\}should be used in the main continual learning phase, we explore three different weighting strategies, as shown in equations[11](https://arxiv.org/html/2608.00630#Pt0.A3.E11)to[13](https://arxiv.org/html/2608.00630#Pt0.A3.E13)\. In each of these equations, the new constraintαk′\\alpha\_\{k\}^\{\\prime\}for each parameter is determined by weighting the originalαk−1\\alpha\_\{k\-1\}value usingαkr​e​l\\alpha\_\{k\}^\{rel\}, whereταr​e​l\\tau\_\{\\alpha^\{rel\}\}is the cut\-off for differentiating high and low relative\-importance andλu​p\\lambda\_\{up\}andλd​o​w​n\\lambda\_\{down\}are scaling hyper\-parameters\. Equation[11](https://arxiv.org/html/2608.00630#Pt0.A3.E11)focuses on CF avoidance, where the constraint on parameters with high relative\-importance is increased through the weightλu​p\.αkr​e​l\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}, while the constraint on parameters with low relative\-importance is unmodified\.λu​p\\lambda\_\{up\}is lower bounded by1/ταr​e​l1/\\tau\_\{\\alpha^\{rel\}\}to ensureλu​p\.αkr​e​l\>1\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}\>1\(to mitigate CF\)\. Equation[12](https://arxiv.org/html/2608.00630#Pt0.A3.E12)focuses on plasticity and KT, where the constraint on parameters with low relative\-importance is decreased through the weightλd​o​w​n\.αkr​e​l\\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}, while the constraint on parameters with high relative\-importance is unmodified\.λd​o​w​n\\lambda\_\{down\}is upper bounded by 1 to ensureλd​o​w​n\.αkr​e​l<1\\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}<1\(to enable plasticity/KT\)\. Our final proposed approach, Equation[13](https://arxiv.org/html/2608.00630#Pt0.A3.E13), combines the above two to simultaneously modify the constraint for both sets of parameters\.

CF Focus:

αk′=\{λu​p\.αkr​e​l\.αk−1ifαkr​e​l\>ταr​e​lαk−1otherwise\\alpha\_\{k\}^\{\\prime\}=\\begin\{cases\}\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}\.\\alpha\_\{k\-1\}&\\text\{if $\\alpha\_\{k\}^\{rel\}\>\\tau\_\{\\alpha^\{rel\}\}$\}\\\\ \\alpha\_\{k\-1\}&\\text\{otherwise\}\\end\{cases\}\(11\)
KT Focus:

αk′=\{αk−1ifαkr​e​l\>ταr​e​lλd​o​w​n\.αkr​e​l\.αk−1otherwise\\alpha\_\{k\}^\{\\prime\}=\\begin\{cases\}\\alpha\_\{k\-1\}&\\text\{if $\\alpha\_\{k\}^\{rel\}\>\\tau\_\{\\alpha^\{rel\}\}$\}\\\\ \\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}\.\\alpha\_\{k\-1\}&\\text\{otherwise\}\\end\{cases\}\(12\)
CF\-KT Balance:

αk′=\{λu​p\.αkr​e​l\.αk−1ifαkr​e​l\>ταr​e​lλd​o​w​n\.αkr​e​l\.αk−1otherwise\\alpha\_\{k\}^\{\\prime\}=\\begin\{cases\}\\lambda\_\{up\}\.\\alpha\_\{k\}^\{rel\}\.\\alpha\_\{k\-1\}&\\text\{if $\\alpha\_\{k\}^\{rel\}\>\\tau\_\{\\alpha^\{rel\}\}$\}\\\\ \\lambda\_\{down\}\.\\alpha\_\{k\}^\{rel\}\.\\alpha\_\{k\-1\}&\\text\{otherwise\}\\end\{cases\}\(13\)
![Refer to caption](https://arxiv.org/html/2608.00630v1/figures/IntentSH_EKTO.png)\(a\)KT\-Focus
![Refer to caption](https://arxiv.org/html/2608.00630v1/figures/IntentSH_ACFO_smalllambup.png)\(b\)CF\-Focus

Figure 2:Impact of different weighting strategies \(CF\-Focus, KT\-Focus\) and look\-ahead hyper\-parameters on CL performance metrics after learning the second task, under class\-incremental learning\.λu​p\\lambda\_\{up\}is scaled byλm​a​x/λ\\lambda\_\{max\}/\\lambda\. At this step ANCL\-MAS results in \(CF=4%, FWT=31%\), and ANCL\-LWF in \(CF=42%, FWT=60%\)\.![Refer to caption](https://arxiv.org/html/2608.00630v1/figures/IntentSH_ORIG_elasdown0.01_smalllambup.png)\(a\)CF\-KT Balance:λd​o​w​n=0\.01\\lambda\_\{down\}=0\.01
![Refer to caption](https://arxiv.org/html/2608.00630v1/figures/IntentSH_ORIG_elasdown0.5_smalllambup.png)\(b\)CF\-KT Balance:λd​o​w​n=0\.5\\lambda\_\{down\}=0\.5
![Refer to caption](https://arxiv.org/html/2608.00630v1/figures/IntentSH_ORIG_elasdown1.0_smalllambup.png)\(c\)CF\-KT Balance:λd​o​w​n=1\\lambda\_\{down\}=1

Figure 3:Impact of the CF\-KT Balance strategy and look\-ahead hyper\-parameters on CL performance metrics after learning the second task, under class\-incremental learning\. x\-axis is scaled byλm​a​x/λ\\lambda\_\{max\}/\\lambda\. At this step ANCL\-MAS results in \(CF=4%, FWT=31%\), and ANCL\-LWF in \(CF=42%, FWT=60%\)\.Next, we empirically analyse the three weighting strategies proposed above and various hyper\-parameter choices\. For this, we consider the second task of the class\-incremental sequence\. Figures[2](https://arxiv.org/html/2608.00630#Pt0.A3.F2)and[3](https://arxiv.org/html/2608.00630#Pt0.A3.F3)show the impact of different weighting strategies and choices of hyper\-parameters forλd​o​w​n\\lambda\_\{down\},λu​p\\lambda\_\{up\}andταr​e​l\\tau\_\{\\alpha^\{rel\}\}on the continual learning performance\. Crucially, only the LA hyper\-parameters are varied, whileλ\\lambdaremains the same as for the baseline ANCL\-MAS\.

##### Stability\-plasticity trade\-off:

Comparing Figures[2\(a\)](https://arxiv.org/html/2608.00630#Pt0.A3.F2.sf1)and[2\(b\)](https://arxiv.org/html/2608.00630#Pt0.A3.F2.sf2), we see that the KT\-Focus formulation results in higher FWT scores indicating better model plasticity, while the CF\-Focus formulation results in much lesser CF albeit at the cost of model plasticity\. With the CF\-KT Balance formulation \(Figures[3\(a\)](https://arxiv.org/html/2608.00630#Pt0.A3.F3.sf1),[3\(b\)](https://arxiv.org/html/2608.00630#Pt0.A3.F3.sf2)and[3\(c\)](https://arxiv.org/html/2608.00630#Pt0.A3.F3.sf3)\), we can achieve a good balance between stability and plasticity, under the right hyper\-parameter choices\.

##### Impact ofταr​e​l\\tau\_\{\\alpha^\{rel\}\}:

In general, increasingταr​e​l\\tau\_\{\\alpha^\{rel\}\}threshold amounts to relaxing the regularisation constraint on more parameters, for all formulations considered\. When settingταr​e​l=0\.1\\tau\_\{\\alpha^\{rel\}\}=0\.1, effectively fewer parameters can be updated for learning new task data than at higher thresholds and accordingly we see that this setting results in least FWT and CF across all formulations\. Increasingταr​e​l\\tau\_\{\\alpha^\{rel\}\}beyond0\.30\.3often has identical results\. To understand this further, we analyse theταr​e​l\\tau\_\{\\alpha^\{rel\}\}distribution at various model layers and components\. Interestingly, we find that at most model layers and components, the mean of the distribution falls aroundταr​e​l=0\.3\\tau\_\{\\alpha^\{rel\}\}=0\.3\. This implies that as we increase theταr​e​l\\tau\_\{\\alpha^\{rel\}\}threshold from0\.10\.1to0\.30\.3, the number of parameters with relaxed constraint increases significantly, which explains the significant change in performance\. Increasing the threshold further however affects fewer additional parameters, resulting in either similar performance or minor effects\. This supports settingταr​e​l\\tau\_\{\\alpha^\{rel\}\}to a value slightly lower than the distribution mean to ensure a good stability\-plasticity trade\-off\. An ablation study of different choices is depicted in Table[8](https://arxiv.org/html/2608.00630#Pt0.A4.T8)\(Appendix[0\.D](https://arxiv.org/html/2608.00630#Pt0.A4)\)\.

##### Impact ofλd​o​w​n\\lambda\_\{down\}:

λd​o​w​n\\lambda\_\{down\}acts only on parameters with low relative\-importance \(i\.e\.αkr​e​l<=ταr​e​l\\alpha\_\{k\}^\{rel\}<=\\tau\_\{\\alpha^\{rel\}\}\) and decreasing it amounts to relaxing the regularisation constraint on these parameters\. Figure[2\(a\)](https://arxiv.org/html/2608.00630#Pt0.A3.F2.sf1)shows that under the KT\-Focus formulation, whenταr​e​l\>=0\.3\\tau\_\{\\alpha^\{rel\}\}\>=0\.3, decreasingλd​o​w​n\\lambda\_\{down\}at first steadily increases CF \(up to 40%\) until at very lowλd​o​w​n\\lambda\_\{down\}\(=0\.01\), there is a drastic increase in CF \(up to 60%\), while throughout FWT remains the same \(∼\\sim60%\)\. Forταr​e​l=0\.2\\tau\_\{\\alpha^\{rel\}\}=0\.2, the increase in CF is less drastic\. Then, forταr​e​l=0\.1\\tau\_\{\\alpha^\{rel\}\}=0\.1, decreasingλd​o​w​n\\lambda\_\{down\}only minimally impacts CF \(which remains at<<30%\), while FWT steadily improves \(upto 50%\)\. This validates our hypothesis that parameters with low relative\-importance \(ταr​e​l<0\.3\\tau\_\{\\alpha^\{rel\}\}<0\.3in this case\) contribute less to forgetting of past\-task knowledge and relaxing the regularisation constraint on these can help with model plasticity\. In the CF\-KT Balance formulation \(figure[3](https://arxiv.org/html/2608.00630#Pt0.A3.F3)\), parameters with high relative\-importance are additionally given increased regularisation\. Forταr​e​l\>=0\.3\\tau\_\{\\alpha^\{rel\}\}\>=0\.3, decreasingλd​o​w​n\\lambda\_\{down\}has similar trends as the KT\-Focus formulation\. Forταr​e​l<0\.3\\tau\_\{\\alpha^\{rel\}\}<0\.3, at allλd​o​w​n\\lambda\_\{down\}, CF is reduced further compared to the KT\-Focus formulation, while still maintaining high FWT \(∼\\sim60%\) whenταr​e​l\\tau\_\{\\alpha^\{rel\}\}is not too low \(=0\.2\)\. This further validates our relative\-importance hypothesis and indicates that the CF\-KT Balance formulation achieves superior stability\-plasticity trade\-off\.

##### Impact ofλu​p\\lambda\_\{up\}:

λu​p\\lambda\_\{up\}acts only on parameters with high relative\-importance \(i\.e\.αkr​e​l\>ταr​e​l\\alpha\_\{k\}^\{rel\}\>\\tau\_\{\\alpha^\{rel\}\}\) and increasing it amounts to stricter constraints on these parameters\. We see that in the CF\-KT Balance formulation \(figures[3](https://arxiv.org/html/2608.00630#Pt0.A3.F3)\), increasingλu​p\\lambda\_\{up\}can help reduce CF with minimal impact to FWT\.

##### Conclusion:

The above analysis provides an indication of the impact of different hyper\-parameters on the continual learning performance, while also validating the assumptions made with respect to high and low relative\-importance\. Moreover, the CF\-KT Balance formulation is found to be the most promising approach\. Based on this analysis, the following approach to setting the three LA hyper\-parameters is followed, as described in Appendix[0\.A](https://arxiv.org/html/2608.00630#Pt0.A1)\- \(1\)ταr​e​l\\tau\_\{\\alpha^\{rel\}\}is set slightly lower than the distribution mean at a given model component and layer, ensuring a good balance between stability and plasticity\. An ablation study of different choices is conducted in Appendix[0\.D\.2](https://arxiv.org/html/2608.00630#Pt0.A4.SS2)\. \(2\)λd​o​w​n\\lambda\_\{down\}andλu​p\\lambda\_\{up\}are set to their maximum values of 1 andλm​a​x/λ\\lambda\_\{max\}/\\lambda, respectively\.λd​o​w​n\\lambda\_\{down\}is decreased first, ensuring minimum forgetting on previous tasks\. \(3\) When further decrease inλd​o​w​n\\lambda\_\{down\}results in a performance plateau,λu​p\\lambda\_\{up\}is decreased next\.

## Appendix 0\.DAdditional Results

### 0\.D\.1Comparison of Stability\-Plasticity Trade\-Off and Backward Transfer Across Thresholds

Table 7:Comparison of Stability\-Plasticity Trade\-Off and Backward Transfer: CL performance at the end of all tasks in the sequence, at different thresholds, measured in F1 scores\. Best is indicated in bold\.↑\\uparrowindicates higher is better,↓\\downarrowindicates lower is better\. Results on the second class\-incremental sequence\.Table[7](https://arxiv.org/html/2608.00630#Pt0.A4.T7)depicts the continual learning performance of the LA and ANCL methods as the threshold is varied\. The results indicate that ANCL\-LWF in general struggles to balance stability and plasticity, with comparatively high forgetting at all considered thresholds, resulting in consistently poorer overall performance than ANCL\-MAS and LA\-MAS\. Interestingly, at very high threshold \(=90\), forgetting is significantly high with negligible to no backward knowledge transfer, while lowering the threshold \(=70\) increasesbothstability and backward knowledge transfer for all methods\. This may indicate that to achieve backward knowledge transfer, some model stability is also desirable\. In other words, at high threshold, when regularisation \(λ\\lambda\) is very low, model parameters can undergo significant changes to accommodate new task knowledge, causing forgetting rather than backward transfer\. When regularisation is moderate and model parameters are not significantly updated, backward transfer becomes feasible\. As the threshold is reduced further \(=60\), however, high regularisation affects backward transfer\. LA\-MAS, by allowing parameters with low relative importance to be less regularised even when they have high past\-task importance, achieves better backward transfer and maintains high plasticity even at lower thresholds, at the cost of some model stability\.

### 0\.D\.2Ablation ofταr​e​l\\tau\_\{\\alpha^\{rel\}\}choice

Table[8](https://arxiv.org/html/2608.00630#Pt0.A4.T8)shows the stability\-plasticity trade\-off asταr​e​l\\tau\_\{\\alpha^\{rel\}\}is reduced, as new tasks are introduced\. Reducingταr​e​l\\tau\_\{\\alpha^\{rel\}\}effectively reduces the number of parameters that are allowed to be easily updated\. When learning the second task, reducingταr​e​l\\tau\_\{\\alpha^\{rel\}\}steadily helps reduce forgetting, resulting in better stability\-plasticity trade\-off\. When learning subsequent tasks, network capacity begins to saturate\. Therefore, while reducingταr​e​l\\tau\_\{\\alpha^\{rel\}\}helps reduce forgetting initially, at lowerταr​e​l\\tau\_\{\\alpha^\{rel\}\}values, regularisation also needs to be lowered \(i\.e\. lowerλd​o​w​n\\lambda\_\{down\},λu​p\\lambda\_\{up\}\) to achieve comparable plasticity\. This in turn results in increased forgetting at lowerταr​e​l\\tau\_\{\\alpha^\{rel\}\}values\. A good trade\-off is achieved at intermediateταr​e​l\\tau\_\{\\alpha^\{rel\}\}values of 0\.8 to 0\.85\.

Table 8:Analysing the effect ofταr​e​l\\tau\_\{\\alpha^\{rel\}\}choice on the continual learning performance\. Best is indicated in bold\. Results on the second class\-incremental learning sequence\.

## Appendix 0\.EComputation Cost

Table[9](https://arxiv.org/html/2608.00630#Pt0.A5.T9)provides the computation cost estimate for each of the compared continual learning methods in terms of the number of forward and backward passes through the data, ignoring the cost of hyper\-parameter search\. UPGD, designed for online learning, computes the parameter\-utility at each update step using the gradient at that step and thus requires the same number of passes through the data as SEQ\. AdaBOP, which uses the gradient space to estimate task correlations, requires an additional pass through the data at the start of training each task, to collect the gradients for correlation estimation\. RP2F estimates parameter importance at each epoch\. Notably, RP2F with fisher parameter importance is more efficient \(comparable to ANCL\-MAS, LA\-MAS\) than perturbation based importance, which requires separate passes through the data for each parameter being perturbed at each epoch\. LA\-MAS and ANCL\-MAS have similar computation cost, where the overhead over SEQ consists of parameter importance estimation at the end of each task and auxiliary/LA network training\. ANCL\-LWF can be more costly as the data is passed through both frozen and current network during MCL\.

Table 9:Computation Cost: Number of forward and backward passes through the data when training on a sequence of two tasks,b1b\_\{1\}andb2b\_\{2\}represent the number of batches in the first and second task, respectively,EEdenotes the number of epochs taken to converge in each training phase andNpN\_\{p\}denotes the number of model parameters perturbed\.