AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

# AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
Source: [https://arxiv.org/html/2608.11758](https://arxiv.org/html/2608.11758)
Jinfan LvAffiliation:College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, ChinaGuoming WangAffiliation:School of Software Technology, Zhejiang University, Hangzhou, ChinaAffiliation:Equal contribution\.Correspondence:[NB21013@zju\.edu\.cn](mailto:[email protected])Jupeng ZhengAffiliation:School of Artificial Intelligence, Sun Yat\-Sen University, Guangzhou, ChinaLing XiaoAffiliation:Graduate School of Information Science, Hokkaido University, Sapporo, JapanSiliang TangAffiliation:School of Software Technology, Zhejiang University, Hangzhou, China

###### Abstract

Multimodal Large Language Models \(MLLMs\) exhibit strong generalization and reasoning abilities due to large\-scale multimodal pre\-training\. However, fine\-tuning these models on downstream tasks often leads to catastrophic forgetting, where newly learned task\-specific knowledge degrades previously acquired capabilities\. This issue arises because gradient updates for new tasks overwrite parameters critical to prior knowledge, limiting the practical deployment of MLLMs\. To address this challenge, we proposeActivation\-Weighted Adaptive REtention \(AWARe\), a fine\-tuning method that mitigates catastrophic forgetting by dynamically controlling parameter updates based on activation patterns\. AWARe assigns activation\-based importance scores to parameters, selectively freezing those essential for preserving prior capabilities while allowing less important parameters to adapt to new tasks\. Importantly, AWARe operateswithout modifying model architectures, ensuring compatibility with existing inference engines\. Extensive experiments demonstrate that AWARe effectively preserves upstream capabilities while achieving superior downstream performance compared to existing methods\. Code is available at[https://github\.com/kaln27/AWARe](https://github.com/kaln27/AWARe)\.

## 1Introduction

Figure 1:Overall Performance of AWARe\.Top: radar plots showing comparison between our approach and various baselines under Single\-Task Downstream Adaptation\. Bottom: results on the MLLM\-DCL continuous learning benchmark\.![Refer to caption](https://arxiv.org/html/2608.11758v1/MethodOverview_v2.png)Figure 2:Overview of AWARe\.Given a multimodal base model, AWARe identifies important neurons or weight groups using activation\-based saliency on calibration samples, then freezes high\-saliency weights and updates only low\-saliency weights during fine\-tuning\.Multimodal Large Language Models \(MLLMs\) have emerged as powerful versatile agents, capable of solving complex tasks that require understanding and reasoning across both inputs of vision and language[Zhu et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib71);[Dai et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib11);[Bai et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib1);[Liu et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib36);[Lin et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib34);[Chen et al\. 2024b](https://arxiv.org/html/2608.11758#bib.bib9);[Wang et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib58);[Zhu et al\. 2025a](https://arxiv.org/html/2608.11758#bib.bib73);[Wang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib59);[Bai et al\. 2025b](https://arxiv.org/html/2608.11758#bib.bib3);[Bai et al\. 2025a](https://arxiv.org/html/2608.11758#bib.bib2)\. Typical MLLMs are composed of a pre\-trained visual encoder, a large language model \(LLM\), and a connector module[Li et al\. 2022](https://arxiv.org/html/2608.11758#bib.bib31);[Dai et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib11);[Liu et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib36)\. These models facilitate strong generalization abilities acquired from large\-scale pre\-training\. Despite their impressive zero\-shot capabilities, fine\-tuning MLLMs on downstream tasks remains a standard practice to tailor models for specific domains or improve instruction\-following performance[Dai et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib11);[Bai et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib1);[Liu et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib36)\. However, this adaptation process often comes at a cost:catastrophic forgetting\(CF\), where the model’s proficiency in previously learned upstream tasks degrades significantly as it optimizes for new objectives[McCloskey and Cohen 1989](https://arxiv.org/html/2608.11758#bib.bib46);[McClelland et al\. 1995](https://arxiv.org/html/2608.11758#bib.bib45);[Kirkpatrick et al\. 2017](https://arxiv.org/html/2608.11758#bib.bib26);[Li et al\. 2024b](https://arxiv.org/html/2608.11758#bib.bib29)\. Nonetheless, this form of forgetting remains particularly severe in MLLMs due to the complex interplay between modalities and the high dimensionality of the parameter space[Sung et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib55);[Luo et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib42);[Luo et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib41);[Zhai et al\. 2024a](https://arxiv.org/html/2608.11758#bib.bib68);[Shen et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib52);[Jiang et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib24);[Zhai et al\. 2024b](https://arxiv.org/html/2608.11758#bib.bib69)\.

Existing approaches to mitigate catastrophic forgetting, such as experience replay[Riemer et al\. 2018](https://arxiv.org/html/2608.11758#bib.bib51);[Chaudhry et al\. 2019](https://arxiv.org/html/2608.11758#bib.bib6)or regularization\-based methods[Kirkpatrick et al\. 2017](https://arxiv.org/html/2608.11758#bib.bib26), often incur high computational overhead or struggle to scale to the billions of parameters in MLLMs\. Parameter\-Efficient Fine\-Tuning \(PEFT\) methods like LoRA[Hu et al\. 2022](https://arxiv.org/html/2608.11758#bib.bib19)reduce the trainable parameter count but do not inherently prevent the erosion of pre\-trained knowledge, as recent studies show they can still disrupt reliable upstream features[Biderman et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib4);[Zhu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib72)\. Consequently, a critical challenge arises:How can we enable MLLMs to adapt effectively to downstream tasks while robustly preserving their pre\-trained capabilities without prohibitive costs?

Our work is primarily inspired by the principle of activation\-aware parameter saliency, most notably demonstrated by Activation\-aware Weight Quantization \(AWQ\)[Lin et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib33)\. AWQ shows that a small fraction of weights associated with large activation magnitudes are disproportionately important for preserving model behavior under quantization\. Wanda reaches a complementary conclusion in pruning: static weight magnitude alone is insufficient for LLMs, while combining weights with input activation statistics gives a simple and effective estimate of functional importance[Sun et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib54)\. Recent model merging studies further suggest that activation\-guided consensus can reduce parameter interference when combining specialized models[Nobari et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib49);[Yao et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib63)\. Together, these findings indicate that activations provide a dynamic, data\-dependent view of which internal pathways support stable behavior, beyond what weight\-centric criteria such as magnitude pruning can capture[Han et al\. 2015](https://arxiv.org/html/2608.11758#bib.bib17);[Frankle and Carbin 2019](https://arxiv.org/html/2608.11758#bib.bib13)\. We therefore repurpose activation saliency from compression and merging to forgetting mitigation\.

Building on this motivation, we introduceAWARe, short forActivation\-WeightedAdaptiveREtention, a novel framework designed to surgically mitigate catastrophic forgetting\. AWARe has two phases:Knowledge Profiling, which estimates neuron saliency from a small calibration set, andConstrained Fine\-tuning, which freezes the most salient parameters during downstream training\. When upstream data are unavailable, we can use a general\-purpose calibration set such as MMMU[Yue et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib65); as shown by the AWARe \(MMMU\) results in[Table1](https://arxiv.org/html/2608.11758#S4.T1), this remains highly effective while adding only a negligible profiling cost\.

We validate our approach through comprehensive experiments on IconQA[Lu et al\. 2021](https://arxiv.org/html/2608.11758#bib.bib40), COCO\-Caption[Lin et al\. 2014](https://arxiv.org/html/2608.11758#bib.bib35), and MLLM\-DCL[Zhao et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib70)\(Continuous learning\) benchmarks\. AWARe achieves strong stability\-plasticity trade\-offs, and freezing the top 30% of self\-attention parameters is often sufficient to preserve upstream knowledge while keeping downstream performance competitive\.

Our main contributions are summarized as follows:

- ❶AWARe Framework\.We introduce Activation\-Weighted Adaptive REtention \(AWARe\), a simple activation\-based method for preserving critical upstream knowledge during downstream adaptation in MLLMs\.
- ❷Efficiency and Simplicity\.Our approach requires no architectural changes and updates only a small subset of parameters \(e\.g\.,∼17\.5%\\sim 17\.5\\%; see[AppendixC](https://arxiv.org/html/2608.11758#A3)\), avoiding replay buffers and extra modules\.
- ❸Comprehensive Validation\.We show through ablations and benchmarks that activation\-based saliency is effective and that AWARe preserves generalization across diverse settings\.

## 2Related Work

### 2\.1Catastrophic Forgetting and Continual Learning

Fine\-tuning serves as a pivotal mechanism for adapting Multimodal Large Language Models \(MLLMs\) to downstream tasks\. However, deep learning models often suffer from catastrophic forgetting[McCloskey and Cohen 1989](https://arxiv.org/html/2608.11758#bib.bib46);[McClelland et al\. 1995](https://arxiv.org/html/2608.11758#bib.bib45), where previously learned knowledge is lost when acquiring new skills\. Various continual learning algorithms have been proposed to address this issue, generally encompassing rehearsal\-based, regularization\-based[Kirkpatrick et al\. 2017](https://arxiv.org/html/2608.11758#bib.bib26);[Lopez\-Paz and Ranzato 2017](https://arxiv.org/html/2608.11758#bib.bib39), and architecture\-based approaches[Houlsby et al\. 2019](https://arxiv.org/html/2608.11758#bib.bib18);[Lester et al\. 2021](https://arxiv.org/html/2608.11758#bib.bib27)\. While established, many traditional methods rely on full\-model fine\-tuning or second\-order statistics, making them computationally prohibitive for large\-scale models\.

### 2\.2Parameter\-Efficient Fine\-Tuning for Mitigation

In the era of large foundation models, the challenge shifts towards maintaining the model’s strong generalization and "Open\-World Stabilization" after downstream adaptation[Ni et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib48);[Zhai et al\. 2024a](https://arxiv.org/html/2608.11758#bib.bib68)\. Recent efforts have explored parameter\-efficient solutions to mitigate forgetting\. Reparameterization methods such as DoRA[Liu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib37)and LoRA\-based extensions like LoRAMoE[Dou et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib12)aim to enhance learning capacity while addressing generalization loss, whereas LoRASculpt[Liang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib32)prunes redundant parameters via sparse updates\. Regularization techniques have also been adapted; for instance, Orth\-Reg[Hu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib20)encourages fine\-tuned features to remain orthogonal to pre\-trained representations\. CorDA[Yang et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib62)builds task\-aware adapters through context\-guided weight decomposition to preserve world knowledge\. Selective tuning approaches, such as SPIDER[Huang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib22)and Model Tailor[Zhu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib72), leverage gradient or saliency analysis to construct sparse updates, aiming to protect critical pre\-trained features\. However, these criteria often rely on static information or gradient magnitudes, overlooking the dynamic nature of neuronal activations during forward propagation\.

Recognizing these limitations, we propose AWARe\. Unlike additive methods that introduce extra parameters or selective tuning that relies on static priors, AWARe utilizes task\-induced activation distributions to measure parameter importance\. By adaptively constraining high\-activation neurons based on upstream statistics, AWARe effectively balances the preservation of core capabilities with the flexibility needed for downstream task adaptation\.

## 3Methodology

### 3\.1Problem Formulation

A Multimodal Large Language Model \(MLLM\)ℳθ\\mathcal\{M\}\_\{\\theta\}generally comprises three core components: a vision encoder𝒱\\mathcal\{V\}\(e\.g\., CLIP[Radford et al\. 2021](https://arxiv.org/html/2608.11758#bib.bib50)or SigLIP[Zhai et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib67)\), a large language modelℒ\\mathcal\{L\}\(e\.g\., LLaMA[Touvron et al\. 2023b](https://arxiv.org/html/2608.11758#bib.bib57);[Touvron et al\. 2023a](https://arxiv.org/html/2608.11758#bib.bib56)or Vicuna[Chiang et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib10)\), and a connector moduleρ\\rhothat bridges the visual and textual modalities\. The standard paradigm involves aligning the pre\-trained vision encoder representations with the LLM’s embedding space via the connector, enabling the model to process multimodal inputs effectively\.

Consider a Multimodal Large Language Model \(MLLM\)ℳθ\\mathcal\{M\}\_\{\\theta\}parameterized byθ\\theta\. We assume the model has been initially trained on a foundational set of upstream tasks𝒯b​a​s​e\\mathcal\{T\}\_\{base\}\. In a continual instruction tuning setting, the model is sequentially exposed to a stream of downstream tasks, denoted as𝒯s​e​q=\{𝒯1,𝒯2,…,𝒯N\}\\mathcal\{T\}\_\{seq\}=\\\{\\mathcal\{T\}\_\{1\},\\mathcal\{T\}\_\{2\},\\dots,\\mathcal\{T\}\_\{N\}\\\}\. Our objective is to incrementally learn each new task𝒯t\\mathcal\{T\}\_\{t\}at stepttby updatingθ\\theta, while mitigating thecatastrophic forgetting[McCloskey and Cohen 1989](https://arxiv.org/html/2608.11758#bib.bib46);[McClelland et al\. 1995](https://arxiv.org/html/2608.11758#bib.bib45)of both the foundational capabilities from𝒯b​a​s​e\\mathcal\{T\}\_\{base\}and the previously acquired knowledge from tasks\{𝒯1,…,𝒯t−1\}\\\{\\mathcal\{T\}\_\{1\},\\dots,\\mathcal\{T\}\_\{t\-1\}\\\}\. As illustrated in[Figure3](https://arxiv.org/html/2608.11758#S3.F3), we achieve this by partitioning the parameters into a fixed setθf​i​x​e​d\\theta\_\{fixed\}, which captures and preserves core established features, and a plastic setθa​c​t​i​v​e\\theta\_\{active\}used for acquiring new task\-specific knowledge\.

![Refer to caption](https://arxiv.org/html/2608.11758v1/Method.png)Figure 3:Overview of AWARe\. For each target linear layer, we profile the activation matrix produced by calibration samples, normalize the activation statistics across sequence and feature dimensions, and obtain a saliency score for each output channel\. High\-saliency channels are selected and frozen to preserve pretrained knowledge, while the remaining channels stay trainable for downstream adaptation\.
### 3\.2Activation\-Based Saliency Estimation

To identify important parameters for retention, we utilize a calibration set𝒮c​a​l\\mathcal\{S\}\_\{cal\}composed of representative samples from each upstream task’s train datasets \(𝒯u​p\\mathcal\{T\}\_\{up\}\)\. Specifically, we randomly sample several instances from each upstream task\. In scenarios where access to upstream data is restricted, we demonstrate that utilizing a comprehensive general benchmark, such as MMMU[Yue et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib65), as a proxy for𝒮c​a​l\\mathcal\{S\}\_\{cal\}is equally effective\. We empirically demonstrate the robustness of model performance to the calibration set size in our ablation studies at[Section4\.6](https://arxiv.org/html/2608.11758#S4.SS6)\. Crucially, the profiling incurs negligible overhead, requiring only a single forward pass on a small sample set\. For a target linear layer with weight matrix𝐖∈ℝdo​u​t×di​n\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{out\}\\times d\_\{in\}\}, we analyze its activation tensor𝐀∈ℝB×L×do​u​t\\mathbf\{A\}\\in\\mathbb\{R\}^\{B\\times L\\times d\_\{out\}\}, wherei∈\{1,…,B\}i\\in\\\{1,\\dots,B\\\},j∈\{1,…,L\}j\\in\\\{1,\\dots,L\\\}, andk∈\{1,…,do​u​t\}k\\in\\\{1,\\dots,d\_\{out\}\\\}index the batch size, context length and hidden dimension, respectively\.

The saliency estimation follows a three\-step aggregation and normalization process\. First, we compute theL2L\_\{2\}\-norm of activations along the sequence length dimension for each samplejjand neuronkk:

ai,k′=∑j=1Lai,j,k2a^\{\\prime\}\_\{i,k\}=\\sqrt\{\\textstyle\\sum\_\{j=1\}^\{L\}a\_\{i,j,k\}^\{2\}\}\(1\)Second, to ensure comparability across samples and prevent those with outlier\-scale activations from biasing the estimations, we apply per\-sampleL2L\_\{2\}\-normalization across the hidden dimension:

ai,k′′=ai,k′‖𝐚i′‖2=ai,k′∑k′=1do​u​tai,k′′2a^\{\\prime\\prime\}\_\{i,k\}=\\frac\{a^\{\\prime\}\_\{i,k\}\}\{\\\|\\mathbf\{a\}^\{\\prime\}\_\{i\}\\\|\_\{2\}\}=\\frac\{a^\{\\prime\}\_\{i,k\}\}\{\\sqrt\{\\textstyle\\sum\_\{k^\{\\prime\}=1\}^\{d\_\{out\}\}\{a^\{\\prime\}\_\{i,k^\{\\prime\}\}\}^\{2\}\}\}\(2\)where𝐚i′=\[ai,1′,…,ai,do​u​t′\]⊤\\mathbf\{a\}^\{\\prime\}\_\{i\}=\[a^\{\\prime\}\_\{i,1\},\\dots,a^\{\\prime\}\_\{i,d\_\{out\}\}\]^\{\\top\}is the vector of sequence\-aggregated activations for sampleii\. This normalization ensures that the saliency is determined by therelativeimportance of neurons within each context rather than absolute activation scale, which is crucial for identifying neurons that consistently capture structural features across diverse upstream tasks\. This refinement distinguishes our approach from simple magnitude\-based methods and explains the superior performance observed in[Table3](https://arxiv.org/html/2608.11758#S4.T3)\. Finally, the saliency score for each neuronkkis obtained by averaging across the batch:

sk=1B​∑i=1Bai,k′′s\_\{k\}=\\frac\{1\}\{B\}\\textstyle\\sum\_\{i=1\}^\{B\}a^\{\\prime\\prime\}\_\{i,k\}\(3\)The resulting saliency vector𝐬∈ℝdo​u​t\\mathbf\{s\}\\in\\mathbb\{R\}^\{d\_\{out\}\}reflects the relative importance of each output neuron in preserving upstream task capabilities\.

### 3\.3Adaptive Parameter Retention

Guided by the saliency scores, we introduce a retention ratioρ∈\(0,1\)\\rho\\in\(0,1\)as a hyperparameter to control the fraction of parameters to be frozen\. We rank output neurons across all target linear layers by their saliency scores and identify the global index setℐ\\mathcal\{I\}containing the top\-ρ\\rhofraction of neurons \(e\.g\., top30%30\\%or10%10\\%\)\.

Since these salient neurons characterize the core upstream knowledge, we freeze their corresponding parameters in the weight matrix\. Specifically, we construct a binary gradient mask𝐌g​r​a​d∈\{0,1\}do​u​t×di​n\\mathbf\{M\}\_\{grad\}\\in\\\{0,1\\\}^\{d\_\{out\}\\times d\_\{in\}\}such that:

\(Mg​r​a​d\)k,:\\displaystyle\(M\_\{grad\}\)\_\{k,:\}=𝟎ifk∈ℐ,\\displaystyle=\\mathbf\{0\}\\quad\\text\{if \}k\\in\\mathcal\{I\},\(4\)\(Mg​r​a​d\)k,:\\displaystyle\(M\_\{grad\}\)\_\{k,:\}=𝟏otherwise\\displaystyle=\\mathbf\{1\}\\quad\\text\{otherwise\}Mr​e​t\\displaystyle M\_\{ret\}=𝟏−Mg​r​a​d\\displaystyle=\\mathbf\{1\}\-M\_\{grad\}where𝟎\\mathbf\{0\}and𝟏\\mathbf\{1\}are row vectors of sizedi​nd\_\{in\}\. During downstream optimization, the parameters associated with salient regions are held constant via masked gradient updates:

𝐖\(t\+1\)=𝐖\(t\)−η⋅\(∇𝐖ℒd​o​w​n⊙𝐌g​r​a​d\)\\mathbf\{W\}^\{\(t\+1\)\}=\\mathbf\{W\}^\{\(t\)\}\-\\eta\\cdot\\left\(\\nabla\_\{\\mathbf\{W\}\}\\mathcal\{L\}\_\{down\}\\odot\\mathbf\{M\}\_\{grad\}\\right\)\(5\)whereη\\etais the learning rate and⊙\\odotdenotes the Hadamard product\. In practice, the PyTorch implementation achieves this by utilizing a training\-time wrapper,AwareLinear, which decomposes the linear layer into active and frozen components to efficiently manage gradient updates without altering the underlying operator’s logic\. This wrapper is removed post\-training, leaving the original model configuration intact\. This row\-wise freezing mechanism ensures that the most influential output features of the pre\-trained LLaVA model remain intact while allowing less critical regions to adapt to new tasks\. As shown in[Figure4](https://arxiv.org/html/2608.11758#S3.F4), the distribution of these salient neurons varies across transformer layers and projection types\.

\(a\)q\_proj\(b\)k\_proj\(c\)v\_proj
Figure 4:Layer\-wise distribution of high\-activation neurons across different projection types\. The heatmaps visualize indices identified for freezing \(top 30% global activation\)\. Notably,q\_projandk\_projshow high concentration in early layers, whereasv\_projexhibits a more uniform distribution, suggesting value information is distributed more evenly across the depth of the network\.
### 3\.4Selection of Target Linear

We specifically apply AWARe to the linear projection layers within the self\-attention mechanism \(i\.e\.,q\_proj,k\_proj,v\_proj\) and the linear layers within the multimodal projector \(mm\_projector\)\. This selection is informed by recent research[Zhu et al\. 2025b](https://arxiv.org/html/2608.11758#bib.bib74)which demonstrates that updating self\-attention projections tends to cause significantly less catastrophic forgetting of pre\-existing knowledge compared to the multilayer perceptron \(MLP\) blocks\. By focusing our retention strategy on those linear, we balance the preservation of core cross\-modal reasoning capabilities with the flexibility needed for downstream task adaptation\. An ablation analysis is provided in[Section4\.5](https://arxiv.org/html/2608.11758#S4.SS5)\. Consistent with this finding, all other components of the LLM, including MLP blocks and the output projection \(𝐖o\\mathbf\{W\}\_\{o\}\), are kept entirely frozen during adaptation\. This configuration not only maximizes parameter efficiency but also minimizes the risk of distorting the pre\-trained feature space in regions less critical for downstream alignment\. The complete procedure for AWARe is summarized in[Algorithm1](https://arxiv.org/html/2608.11758#alg1)\.

Algorithm 1AWARe1:Pre\-trained model

ℳθ\\mathcal\{M\}\_\{\\theta\}, Upstream task

𝒯u​p\\mathcal\{T\}\_\{up\}, Downstream task

𝒯d​o​w​n\\mathcal\{T\}\_\{down\}, Retention ratio

ρ\\rho
2:Fine\-tuned model

ℳθ\\mathcal\{M\}\_\{\\theta\}
3:Phase 1: Knowledge Profiling

4:Sample calibration set

𝒮c​a​l∼𝒯u​p\\mathcal\{S\}\_\{cal\}\\sim\\mathcal\{T\}\_\{up\}
5:foreach target layer

llin

ℳθ\\mathcal\{M\}\_\{\\theta\}do

6:Perform forward pass on

𝒮c​a​l\\mathcal\{S\}\_\{cal\}to obtain activations

𝐇\(l\)\\mathbf\{H\}^\{\(l\)\}
7:Calculate neuron saliency

𝐬\(l\)\\mathbf\{s\}^\{\(l\)\}using[Equations1](https://arxiv.org/html/2608.11758#S3.E1),[2](https://arxiv.org/html/2608.11758#S3.E2)and[3](https://arxiv.org/html/2608.11758#S3.E3)

8:endfor

9:

ℐ←GlobalTopIndices​\(\{𝐬\(l\)\}l,ρ\)\\mathcal\{I\}\\leftarrow\\text\{GlobalTopIndices\}\(\\\{\\mathbf\{s\}^\{\(l\)\}\\\}\_\{l\},\\rho\)
10:foreach target layer

llin

ℳθ\\mathcal\{M\}\_\{\\theta\}do

11:Extract layer\-specific retained indices

ℐ\(l\)\\mathcal\{I\}^\{\(l\)\}from

ℐ\\mathcal\{I\}
12:Generate row\-wise gradient mask

𝐌g​r​a​d\(l\)\\mathbf\{M\}\_\{grad\}^\{\(l\)\}via[Equation4](https://arxiv.org/html/2608.11758#S3.E4)

13:endfor

14:Phase 2: Constrained Fine\-tuning

15:whilenot converged on

𝒯d​o​w​n\\mathcal\{T\}\_\{down\}do

16:

ℒ←ComputeLoss​\(ℳθ,𝒯d​o​w​n\)\\mathcal\{L\}\\leftarrow\\text\{ComputeLoss\}\(\\mathcal\{M\}\_\{\\theta\},\\mathcal\{T\}\_\{down\}\)
17:Update parameters:

θ←θ−η⋅\(∇θℒ⊙𝐌g​r​a​d\)\\theta\\leftarrow\\theta\-\\eta\\cdot\(\\nabla\_\{\\theta\}\\mathcal\{L\}\\odot\\mathbf\{M\}\_\{grad\}\)
18:endwhile

## 4Experiments

Table 1:Comparison with State\-of\-the\-Art Fine\-Tuning Solutions on LLaVA\-v1\.5\-7Bfor IconQA and COCO\-Caption\. The optimal and sub\-optimal results are denoted by boldface and underlining\.↑means improved accuracy compared with the sub\-optimal results\. †Specifically, the near\-zero performance of Full\-FT on upstream tasks reflects extreme catastrophic forgetting due to overfitting to the target task\. More details about the metrics are provided in[Section4\.3](https://arxiv.org/html/2608.11758#S4.SS3)\. Training setting are detailed in[AppendixD](https://arxiv.org/html/2608.11758#A4)\.AvgLastMethodRS↑\\uparrowMed↑\\uparrowAD↑\\uparrowSci↑\\uparrowFin↑\\uparrowAverage↑\\uparrowRS↑\\uparrowMed↑\\uparrowAD↑\\uparrowSci↑\\uparrowFin↑\\uparrowAverage↑\\uparrowModalPrompt53\.1945\.7340\.7841\.8287\.8253\.8753\.6345\.6840\.7741\.8187\.8253\.94HiDe75\.4148\.7137\.0642\.4981\.5557\.0474\.3148\.9533\.2138\.5481\.5555\.31LoRA\-FT73\.3447\.9738\.6845\.1687\.4558\.5269\.6541\.5925\.4340\.8887\.4553\.00MoELoRA77\.1947\.0237\.3044\.4086\.7558\.5377\.5441\.8527\.6240\.1386\.7554\.78CL\-MoE73\.5149\.9738\.5545\.7388\.7459\.3071\.3446\.8426\.3341\.1788\.7454\.88O\-LoRA76\.1247\.3840\.9746\.0287\.1559\.5374\.6444\.4230\.0241\.4787\.1555\.54SEFE77\.3851\.4144\.2444\.9586\.8260\.9677\.2650\.3737\.2140\.8786\.8258\.51DISCO77\.4148\.0448\.7048\.6589\.2262\.4076\.4944\.4844\.8446\.6189\.2260\.33AWARe79\.8855\.4251\.3849\.7991\.4565\.58↑3\.1878\.9647\.0146\.9446\.1391\.4562\.10↑1\.77

Table 2:Detailed continuous instruction tuning performance on the MLLM\-DCL benchmark[Zhao et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib70)\.Avgreports the average score for each subtask across all later evaluation points after that subtask is learned, whileLastreports the final score on each subtask after training on the last subtask\.In this section, we evaluate the effectiveness of our proposed method by comparing it with several state\-of\-the\-art baselines on multimodal downstream tasks\. We also conduct extensive ablation studies to analyze the impact of key components and hyperparameters\.

### 4\.1Experimental Setup

Datasets and Tasks\.We use LLaVA\-v1\.5\-7b[Liu et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib36)as the base model for training\. The model has been pre\-trained on several upstream tasks, including OKVQA[Marino et al\. 2019](https://arxiv.org/html/2608.11758#bib.bib43), OCRVQA[Mishra et al\. 2019](https://arxiv.org/html/2608.11758#bib.bib47), GQA[Hudson and Manning 2019](https://arxiv.org/html/2608.11758#bib.bib23), and TextVQA[Singh et al\. 2019](https://arxiv.org/html/2608.11758#bib.bib53)\. For downstream evaluation, we use IconQA[Lu et al\. 2021](https://arxiv.org/html/2608.11758#bib.bib40)for VQA and COCO\-Caption[Lin et al\. 2014](https://arxiv.org/html/2608.11758#bib.bib35)for image captioning, where captioning performance is measured by CIDEr\. For continual instruction tuning, we evaluate LLaVA\-v1\.5\-7b on MLLM\-DCL[Zhao et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib70), a five\-task sequence covering remote sensing \(RS\), medical VQA \(Med\), autonomous driving \(AD\), science reasoning \(Sci\), and financial understanding \(Fin\)\. We provide additional experiments on Qwen2\.5\-VL\-7B[Bai et al\. 2025b](https://arxiv.org/html/2608.11758#bib.bib3)in[AppendixB](https://arxiv.org/html/2608.11758#A2)\.

Implementation Details\.During training, we freeze the vision tower to preserve the pre\-trained visual representations\. The complete training configuration and hyperparameter settings are provided in[AppendixD](https://arxiv.org/html/2608.11758#A4)\.

### 4\.2Baselines

We evaluate AWARe under two complementary settings\. First, in single\-task downstream adaptation, the model learns one new downstream task and is evaluated for both target\-task learning and forgetting on upstream tasks\. In this setting, we compare against established parameter\-efficient fine\-tuning and anti\-forgetting baselines, includingFull\-FT,LoRA[Hu et al\. 2022](https://arxiv.org/html/2608.11758#bib.bib19),DoRA[Liu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib37),DARE[Yu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib64),Orth\-Reg[Hu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib20),Model Tailor[Zhu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib72),LoRASculpt[Liang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib32), andSPIDER[Huang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib22)\. Second, in continual instruction tuning, the model learns a sequence of subtasks and is evaluated after each training stage\. For this setting, we compare against continual learning baselines includingModalPrompt[Zeng et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib66),SEFE[Chen et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib8),HiDe\-LLaVA[Guo et al\. 2025a](https://arxiv.org/html/2608.11758#bib.bib15),CL\-MoE[Huai et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib21),O\-LoRA[Wang et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib60),MoELoRA[Chen et al\. 2024a](https://arxiv.org/html/2608.11758#bib.bib7), andDISCO[Guo et al\. 2025b](https://arxiv.org/html/2608.11758#bib.bib16)\. Detailed descriptions for all baselines are provided in[AppendixE](https://arxiv.org/html/2608.11758#A5)\.

### 4\.3Single\-Task Downstream Adaptation

We evaluate the performance of all methods by decomposing continual learning capabilities into two core components:Knowledge Retention \(ℛ\\mathcal\{R\}\)andLearning Efficiency \(ℰ\\mathcal\{E\}\)\. To provide a balanced assessment, we report their Harmonic Mean \(ℋ\\mathcal\{H\}\):

ℋ=2⋅ℛ⋅ℰℛ\+ℰ\\mathcal\{H\}=\\frac\{2\\cdot\\mathcal\{R\}\\cdot\\mathcal\{E\}\}\{\\mathcal\{R\}\+\\mathcal\{E\}\}\(6\)The components are defined as follows:

- •ℛ\\mathcal\{R\}:A stability metric measuring the preservation of performance on upstream tasks after fine\-tuning\. It is calculated as the average ratio of the fine\-tuned accuracy to the base \(zero\-shot\) accuracy: ℛ=1\|𝒯u​p\|​∑i∈𝒯u​pA​c​ci,finalA​c​ci,base\\mathcal\{R\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{up\}\|\}\\sum\_\{i\\in\\mathcal\{T\}\_\{up\}\}\\frac\{Acc\_\{i,\\text\{final\}\}\}\{Acc\_\{i,\\text\{base\}\}\}\(7\)whereA​c​ci,baseAcc\_\{i,\\text\{base\}\}is the model’s initial performance before fine\-tuning\. A sharp drop inℛ\\mathcal\{R\}indicates catastrophic forgetting\.
- •ℰ\\mathcal\{E\}:A plasticity metric measuring the model’s adaptation to the new downstream task\. To normalize against model capacity, it is defined relative to the Full Fine\-Tuning \(Full\-FT\) baseline: ℰ=A​c​cd​o​w​n,MethodA​c​cd​o​w​n,Full\-FT\\mathcal\{E\}=\\frac\{Acc\_\{down,\\text\{Method\}\}\}\{Acc\_\{down,\\text\{Full\-FT\}\}\}\(8\)whereA​c​cd​o​w​n,Full\-FTAcc\_\{down,\\text\{Full\-FT\}\}represents the performance upper bound achievable by Full\-FT\.

As shown in[Table1](https://arxiv.org/html/2608.11758#S4.T1), our method achieves a superior harmonic balanceℋ\\mathcal\{H\}between plasticity and stability\. Compared to other methods, our approach maintains high retention ratesℛ\\mathcal\{R\}while achieving efficiencyℰ\\mathcal\{E\}\. Notably, when upstream task data is restricted, we used the MMMU general dataset as a calibration dataset and still achieved relatively good results, outperforming both LoRASculpt and SPIDER\. We further verify that AWARe is not limited to the LLaVA\-v1\.5 backbone by conducting an additional Qwen2\.5\-VL base\-model experiment; these results are reported in[AppendixB](https://arxiv.org/html/2608.11758#A2)\.

### 4\.4Continual Instruction Tuning on MLLM\-DCL

To further demonstrate the effectiveness of our proposed method in sequential learning scenarios, we evaluate AWARe on the MLLM\-DCL benchmark[Zhao et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib70)\. This benchmark is specifically designed to assess continual instruction tuning for multimodal large language models\. As shown in[Table2](https://arxiv.org/html/2608.11758#S4.T2), AWARe achieves the highest average per\-task performance \(Avg\) and the best final performance after the last task \(Last\), improving the overall averages by 3\.18 and 1\.77 points over the strongest baseline, respectively\. The detailed CL results are provided in[AppendixA](https://arxiv.org/html/2608.11758#A1)\. Overall, AWARe effectively balances plasticity and stability, confirming its superiority in continual instruction tuning settings\.

### 4\.5Ablation Studies

We conduct ablation studies to investigate the contribution of different components and hyperparameter choices in our framework\.

Effectiveness of Activation\-based Selection\.We compare our activation\-based parameter selection strategy against weight norm\-based selection, a hybrid strategy \(0\.3×Weight\+0\.7×Activation0\.3\\times\\text\{Weight\}\+0\.7\\times\\text\{Activation\}\), and a baseline using random parameter selection at the same ratio \(30%\)\. As shown in[Table3](https://arxiv.org/html/2608.11758#S4.T3), random selection significantly fails to protect the upstream knowledge, with the average performance on upstream tasks \(Source Avg\) dropping to 49\.94 on the IconQA task\. In contrast, our activation\-based selection consistently achieves the best balance, outperforming both random and weight\-based selection in bothSource AvgandTargetscores\.

Table 3:Ablation Study on Parameter Selection Strategy\.We compare different methods for selecting parameters to retain: random selection \(Random Selection\), weight magnitude \(Weight Norm\), activation magnitude \(Activation\), and a hybrid approach \(0\.3​W\+0\.7​A0\.3W\+0\.7A\)\.Impact of Selection Ratio and Strategy\.We explore the effect of varying the selection ratio \(from 1% to 90%\) and comparing Layer\-Balanced versus Global\-Highest selection strategies\. As shown in[Table4](https://arxiv.org/html/2608.11758#S4.T4), the Global\-Highest selection strategy at a 30% ratio achieves the optimal balance between stability and plasticity, yielding the highest harmonic mean \(ℋ=103\.2\\mathcal\{H\}=103\.2\) and superior knowledge retention \(ℛ=98\.4\\mathcal\{R\}=98\.4\)\.

RatioOKVQAOCRVQAGQATextVQATargetℛ\\mathcal\{R\}ℰ\\mathcal\{E\}ℋ\\mathcal\{H\}Layer balanced selection1%55\.3364\.0060\.1754\.3982\.4495\.7102\.999\.210%55\.3365\.2059\.6455\.1686\.8496\.3108\.3102\.030%56\.2862\.7560\.6256\.1385\.2996\.5106\.4101\.240%56\.4763\.8560\.4656\.9486\.1397\.3107\.5102\.150%57\.3363\.9560\.8057\.1986\.2197\.9107\.6102\.570%57\.6964\.2061\.7757\.5485\.6798\.7106\.9102\.690%57\.6563\.7062\.0157\.9284\.0798\.7104\.9101\.7Global highest selection1%55\.2663\.4059\.8954\.9682\.6595\.6103\.199\.210%55\.8863\.2559\.5955\.3986\.1395\.8107\.5101\.330%56\.9565\.1060\.8357\.6686\.9298\.4108\.4103\.240%57\.6664\.3561\.4757\.7786\.6498\.7108\.1103\.250%57\.7164\.7561\.8757\.8685\.9799\.1107\.3103\.070%57\.7864\.3562\.0357\.8480\.6499\.0100\.699\.890%58\.1365\.5561\.8958\.3354\.6299\.868\.181\.0

Table 4:Ablation Study on Selection Ratio\.We evaluate the performance of our method with different ratios of parameters selected for retention on the IconQA task\.Calibration Dataset Composition\.We examine how the composition of the calibration dataset influences the model’s resistance to catastrophic forgetting on the IconQA task\. As shown in[Table5](https://arxiv.org/html/2608.11758#S4.T5), using any individual upstream task \(e\.g\., OKVQA, OCRVQA, GQA, or TextVQA\) as the calibration source consistently preserves knowledge, with relatively small performance variations across different configurations\. However, our comprehensive calibration approach \(ALL\), which integrates samples from all upstream sources, yields the most balanced and superior overall performance \(ℋ=103\.2\\mathcal\{H\}=103\.2\)\. This demonstrates that while AWARe is robust to the specific choice of calibration task, a diverse mixture of upstream data ensures the most effective protection across the model’s entire functional landscape\.

Table 5:Ablation Study on Calibration Dataset Composition\.We investigate how using different subsets of upstream data for activation statistics calculation affects performance on the IconQA task\.Training Targets\.We investigate the impact of applying our method to different components within the model architecture\. As illustrated in[Figure5](https://arxiv.org/html/2608.11758#S4.F5), targeting both the self\-attention projections and the multimodal \(mm\_projector\) yields the most favorable balance between plasticity and stability\. Specifically, themm\_projectoris essential for learning new downstream concepts; Locking it limits adaptation and causes a roughly 10\-point decline in target performance\. Furthermore, we observe that fine\-tuningMLPblocks results in a severe performance collapse on upstream tasks \(dropping below 30\-point\), even with our adaptive protection\. This reinforces the hypothesis that MLP weights house the bulk of the model’s fundamental general knowledge, whereas self\-attention projections provide a more flexible landscape for specialized task adaptation\.

Figure 5:Ablation Study on Training Target\.Performance comparison across different architecture components on the IconQA and upstream tasks\. Applying AWARe to bothself\-attnandmm\_projectorachieves the best stable\-plastic trade\-off\.
### 4\.6Sensitivity to Calibration Datasets

To assess the statistical significance and robustness of our results, we evaluate the sensitivity of AWARe to the randomness of the calibration set\. We repeat the Knowledge Profiling phase over three independent runs with different random seeds for sample selection\. As shown in[Table6](https://arxiv.org/html/2608.11758#S4.T6), the performance metrics exhibit minimal standard deviation across runs, demonstrating that the identified salient neurons are consistent and statistically stable artifacts of the model structure, rather than noise from specific data samples\. Moreover, increasing the calibration size from 200 to 400 samples yields negligible performance differences, confirming that 200 samples are statistically sufficient to robustly estimate parameter importance\.

Table 6:Sensitivity Analysis on Calibration Datasets\.We evaluate the performance of AWARe on IconQA task across different calibration set sizes \(200 and 400\) and random sample 3 times\.

## 5Conclusion

In this paper, we introduce AWARe, an activation\-weighted framework for mitigating catastrophic forgetting in Multimodal Large Language Models\. AWARe first profiles task\-induced activations to identify salient neurons, then freezes these regions during downstream fine\-tuning to preserve pre\-trained capabilities\. This design is architecture\-agnostic, requires no additional adapter modules, and updates only a small fraction of parameters\. Across downstream adaptation and MLLM\-DCL experiments, AWARe consistently improves the stability\-plasticity trade\-off, and remains effective even when using a general\-purpose calibration set such as MMMU\.

## 6Limitations

AWARe relies on a calibration set to estimate activation\-based neuron importance\. Although our experiments show that a general\-purpose dataset such as MMMU can provide a practical substitute when upstream task data is unavailable, the quality and coverage of the calibration data may still affect the selected frozen regions\. In addition, our evaluation focuses on representative MLLM backbones and multimodal tasks; broader validation on larger models, more diverse domains, and longer continual learning sequences would further clarify the generality of the approach\. Finally, because AWARe explicitly freezes salient parameters to preserve prior capabilities, extremely large distribution shifts may still require tuning the retention ratio to balance stability and plasticity\.

## References

- Bai et al\. \(2023\)Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou\. 2023\.Qwen\-vl: A versatile vision\-language model for understanding, localization, text reading, and beyond\.*arXiv preprint arXiv:2308\.12966*\.
- Bai et al\. \(2025a\)Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others\. 2025a\.Qwen3\-vl technical report\.*arXiv preprint arXiv:2511\.21631*\.
- Bai et al\. \(2025b\)Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others\. 2025b\.[Qwen2\.5\-vl technical report](https://arxiv.org/abs/2502.13923)\.*Preprint*, arXiv:2502\.13923\.
- Biderman et al\. \(2024\)Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, and 1 others\. 2024\.Lora learns less and forgets less\.*arXiv preprint arXiv:2405\.09673*\.
- Burgess et al\. \(2025\)James Burgess, Jeffrey J Nirschl, Laura Bravo\-Sánchez, Alejandro Lozano, Sanket Rajan Gupte, Jesus G Galaz\-Montoya, Yuhui Zhang, Yuchang Su, Disha Bhowmik, Zachary Coman, and 1 others\. 2025\.Microvqa: A multimodal reasoning benchmark for microscopy\-based scientific research\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 19552–19564\.
- Chaudhry et al\. \(2019\)Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc’Aurelio Ranzato\. 2019\.On tiny episodic memories in continual learning\.*arXiv preprint arXiv:1902\.10486*\.
- Chen et al\. \(2024a\)Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Jingkuan Song, and Lianli Gao\. 2024a\.Coin: A benchmark of continual instruction tuning for multimodel large language models\.*Advances in Neural Information Processing Systems*, 37:57817–57840\.
- Chen et al\. \(2025\)Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ip, and Sam Kwong\. 2025\.Sefe: Superficial and essential forgetting eliminator for multimodal continual instruction tuning\.In*ICML*\.
- Chen et al\. \(2024b\)Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others\. 2024b\.Internvl: Scaling up vision foundation models and aligning for generic visual\-linguistic tasks\.In*CVPR*, pages 24185–24198\.
- Chiang et al\. \(2023\)Wei\-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E\. Gonzalez, Ion Stoica, and Eric P\. Xing\. 2023\.[Vicuna: An open\-source chatbot impressing gpt\-4 with 90%\* chatgpt quality](https://lmsys.org/blog/2023-03-30-vicuna/)\.
- Dai et al\. \(2023\)Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi\. 2023\.Instructblip: Towards general\-purpose vision\-language models with instruction tuning\.In*NeurIPS*\.
- Dou et al\. \(2024\)Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, and 1 others\. 2024\.Loramoe: Alleviating world knowledge forgetting in large language models via moe\-style plugin\.In*ACL*, pages 1932–1945\.
- Frankle and Carbin \(2019\)Jonathan Frankle and Michael Carbin\. 2019\.The lottery ticket hypothesis: Finding sparse, trainable neural networks\.In*ICLR*\.
- Guan et al\. \(2024\)Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, and 1 others\. 2024\.Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision\-language models\.In*Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pages 14375–14385\.
- Guo et al\. \(2025a\)Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da\-Han Wang, Xu\-Yao Zhang, and Cheng\-Lin Liu\. 2025a\.Hide\-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 13572–13586\.
- Guo et al\. \(2025b\)Haiyang Guo, Fanhu Zeng, Fei Zhu, Wenzhuo Liu, Da\-Han Wang, Jian Xu, Xu\-Yao Zhang, and Cheng\-Lin Liu\. 2025b\.Federated continual instruction tuning\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 1325–1335\.
- Han et al\. \(2015\)Song Han, Huizi Mao, and William J Dally\. 2015\.Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding\.*arXiv preprint arXiv:1510\.00149*\.
- Houlsby et al\. \(2019\)Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly\. 2019\.Parameter\-efficient transfer learning for nlp\.In*ICML*, pages 2790–2799\. PMLR\.
- Hu et al\. \(2022\)Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\. 2022\.Lora: Low\-rank adaptation of large language models\.In*ICLR*\.
- Hu et al\. \(2024\)Jiajun Hu, Jian Zhang, Lei Qi, Yinghuan Shi, and Yang Gao\. 2024\.Learn to preserve and diversify: Parameter\-efficient group with orthogonal regularization for domain generalization\.In*European Conference on Computer Vision*, pages 198–216\. Springer\.
- Huai et al\. \(2025\)Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He\. 2025\.Cl\-moe: Enhancing multimodal large language model with dual momentum mixture\-of\-experts for continual visual question answering\.In*Proceedings of the computer vision and pattern recognition conference*, pages 19608–19617\.
- Huang et al\. \(2025\)Wenke Huang, Jian Liang, Zekun Shi, Didi Zhu, Guancheng Wan, He Li, Bo Du, Dacheng Tao, and Mang Ye\. 2025\.Learn from downstream and be yourself in multimodal large language models fine\-tuning\.In*ICML*\.
- Hudson and Manning \(2019\)Drew A Hudson and Christopher D Manning\. 2019\.Gqa: A new dataset for real\-world visual reasoning and compositional question answering\.In*CVPR*, pages 6700–6709\.
- Jiang et al\. \(2024\)Yao Jiang, Xinyu Yan, Ge\-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng\-Ping Fan, and Fahad Shahbaz Khan\. 2024\.Effectiveness assessment of recent large vision\-language models\.*Visual Intelligence*, 2\(1\):17\.
- Kembhavi et al\. \(2016\)Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi\. 2016\.A diagram is worth a dozen images\.In*European conference on computer vision*, pages 235–251\. Springer\.
- Kirkpatrick et al\. \(2017\)James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska\-Barwinska, and 1 others\. 2017\.Overcoming catastrophic forgetting in neural networks\.*Proceedings of the national academy of sciences*, 114\(13\):3521–3526\.
- Lester et al\. \(2021\)Brian Lester, Rami Al\-Rfou, and Noah Constant\. 2021\.The power of scale for parameter\-efficient prompt tuning\.*arXiv preprint arXiv:2104\.08691*\.
- Li et al\. \(2024a\)Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li\. 2024a\.Llava\-onevision: Easy visual task transfer\.*arXiv preprint arXiv:2408\.03326*\.
- Li et al\. \(2024b\)Hongyu Li, Liang Ding, Meng Fang, and Dacheng Tao\. 2024b\.Revisiting catastrophic forgetting in large language model tuning\.*arXiv preprint arXiv:2406\.04836*\.
- Li et al\. \(2023\)Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi\. 2023\.Blip\-2: Bootstrapping language\-image pre\-training with frozen image encoders and large language models\.In*ICML*, pages 19730–19742\. PMLR\.
- Li et al\. \(2022\)Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi\. 2022\.Blip: Bootstrapping language\-image pre\-training for unified vision\-language understanding and generation\.In*ICML*, pages 12888–12900\. PMLR\.
- Liang et al\. \(2025\)Jian Liang, Wenke Huang, Guancheng Wan, Qu Yang, and Mang Ye\. 2025\.Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models\.In*Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 26170–26180\.
- Lin et al\. \(2024\)Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\. 2024\.Awq: Activation\-aware weight quantization for on\-device llm compression and acceleration\.*Proceedings of machine learning and systems*, 6:87–100\.
- Lin et al\. \(2023\)Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han\. 2023\.Vila: On pre\-training for visual language models\.In*CVPR*\.
- Lin et al\. \(2014\)Tsung\-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick\. 2014\.Microsoft coco: Common objects in context\.In*ECCV*, pages 740–755\.
- Liu et al\. \(2023\)Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee\. 2023\.Improved baselines with visual instruction tuning\.In*CVPR*\.
- Liu et al\. \(2024\)Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\. 2024\.Dora: Weight\-decomposed low\-rank adaptation\.In*ICML*\.
- Liu et al\. \(2018\)Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song\. 2018\.Learning towards minimum hyperspherical energy\.*Advances in neural information processing systems*, 31\.
- Lopez\-Paz and Ranzato \(2017\)David Lopez\-Paz and Marc’Aurelio Ranzato\. 2017\.Gradient episodic memory for continual learning\.*NeurIPS*, 30\.
- Lu et al\. \(2021\)Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song\-Chun Zhu\. 2021\.Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning\.In*NeurIPS*\.
- Luo et al\. \(2023\)Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang\. 2023\.An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.*arXiv preprint arXiv:2308\.08747*\.
- Luo et al\. \(2025\)Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang\. 2025\.An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.*IEEE Transactions on Audio, Speech and Language Processing*\.
- Marino et al\. \(2019\)Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi\. 2019\.Ok\-vqa: A visual question answering benchmark requiring external knowledge\.In*CVPR*\.
- Mathew et al\. \(2021\)Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar\. 2021\.Docvqa: A dataset for vqa on document images\.In*Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pages 2200–2209\.
- McClelland et al\. \(1995\)James L McClelland, Bruce L McNaughton, and Randall C O’Reilly\. 1995\.Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory\.*Psychological review*, 102\(3\):419\.
- McCloskey and Cohen \(1989\)Michael McCloskey and Neal J Cohen\. 1989\.Catastrophic interference in connectionist networks: The sequential learning problem\.In*Psychology of learning and motivation*, volume 24, pages 109–165\. Elsevier\.
- Mishra et al\. \(2019\)Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty\. 2019\.Ocr\-vqa: Visual question answering by reading text in images\.In*ICDAR*, pages 947–952\. IEEE\.
- Ni et al\. \(2023\)Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang\. 2023\.Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models\.*arXiv preprint arXiv:2311\.08011*\.
- Nobari et al\. \(2025\)Amin Heyrani Nobari, Kaveh Alimohammadi, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, and Navid Azizan\. 2025\.Activation\-informed merging of large language models\.
- Radford et al\. \(2021\)Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others\. 2021\.Learning transferable visual models from natural language supervision\.In*ICML*, pages 8748–8763\.
- Riemer et al\. \(2018\)Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro\. 2018\.Learning to learn without forgetting by maximizing transfer and minimizing interference\.*arXiv preprint arXiv:1810\.11910*\.
- Shen et al\. \(2024\)Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang\. 2024\.Multimodal instruction tuning with conditional mixture of lora\.*arXiv preprint arXiv:2402\.15896*\.
- Singh et al\. \(2019\)Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach\. 2019\.Towards vqa models that can read\.In*CVPR*, pages 8317–8326\.
- Sun et al\. \(2024\)Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter\. 2024\.A simple and effective pruning approach for large language models\.In*International Conference on Learning Representations*, volume 2024, pages 4942–4964\.
- Sung et al\. \(2023\)Yi\-Lin Sung, Jaehong Yoon, and Mohit Bansal\. 2023\.Ecoflap: Efficient coarse\-to\-fine layer\-wise pruning for vision\-language models\.*arXiv preprint arXiv:2310\.02998*\.
- Touvron et al\. \(2023a\)Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others\. 2023a\.Llama: Open and efficient foundation language models\.*arXiv preprint arXiv:2302\.13971*\.
- Touvron et al\. \(2023b\)Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others\. 2023b\.Llama 2: Open foundation and fine\-tuned chat models\.*arXiv preprint arXiv:2307\.09288*\.
- Wang et al\. \(2024\)Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others\. 2024\.Qwen2\-vl: Enhancing vision\-language model’s perception of the world at any resolution\.*arXiv preprint arXiv:2409\.12191*\.
- Wang et al\. \(2025\)Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others\. 2025\.Internvl3\.5: Advancing open\-source multimodal models in versatility, reasoning, and efficiency\.*arXiv preprint arXiv:2508\.18265*\.
- Wang et al\. \(2023\)Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuan\-Jing Huang\. 2023\.Orthogonal subspace learning for language model continual learning\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 10658–10671\.
- Wu and Xie \(2024\)Penghao Wu and Saining Xie\. 2024\.V\*: Guided visual search as a core mechanism in multimodal llms\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 13084–13094\.
- Yang et al\. \(2024\)Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem\. 2024\.Corda: Context\-oriented decomposition adaptation of large language models\.*arXiv preprint arXiv:2406\.05223*\.
- Yao et al\. \(2025\)Yuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li, Mingyang Liu, Xiongwei Han, Zhijiang Guo, Han Wu, and Linqi Song\. 2025\.Activation\-guided consensus merging for large language models\.*NeurIPS*\.
- Yu et al\. \(2024\)Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li\. 2024\.Language models are super mario: Absorbing abilities from homologous models as a free lunch\.In*ICML*\.
- Yue et al\. \(2024\)Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, and 3 others\. 2024\.Mmmu: A massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.In*Proceedings of CVPR*\.
- Zeng et al\. \(2025\)Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu\-Yao Zhang, and Cheng\-Lin Liu\. 2025\.Modalprompt: Towards efficient multimodal continual instruction tuning with dual\-modality guided prompt\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 12137–12152\.
- Zhai et al\. \(2023\)Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer\. 2023\.Sigmoid loss for language image pre\-training\.In*ICCV*, pages 11975–11986\.
- Zhai et al\. \(2024a\)Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma\. 2024a\.Investigating the catastrophic forgetting in multimodal large language model fine\-tuning\.pages 202–227\. PMLR\.
- Zhai et al\. \(2024b\)Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma\. 2024b\.Investigating the catastrophic forgetting in multimodal large language model fine\-tuning\.In*CPAL*, pages 202–227\. PMLR\.
- Zhao et al\. \(2025\)Hongbo Zhao, Fei Zhu, Haiyang Guo, Meng Wang, Rundong Wang, Gaofeng Meng, and Zhaoxiang Zhang\. 2025\.Mllm\-cl: Continual learning for multimodal large language models\.*arXiv preprint arXiv:2506\.05453*\.
- Zhu et al\. \(2023\)Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny\. 2023\.Minigpt\-4: Enhancing vision\-language understanding with advanced large language models\.*arXiv preprint arXiv:2304\.10592*\.
- Zhu et al\. \(2024\)Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu\. 2024\.Model tailor: Mitigating catastrophic forgetting in multi\-modal large language models\.In*ICML*\.
- Zhu et al\. \(2025a\)Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others\. 2025a\.Internvl3: Exploring advanced training and test\-time recipes for open\-source multimodal models\.*arXiv preprint arXiv:2504\.10479*\.
- Zhu et al\. \(2025b\)Zhen Zhu, Yiming Gong, Yao Xiao, Yaoyao Liu, and Derek Hoiem\. 2025b\.[How to teach large multimodal models new skills](https://arxiv.org/abs/2510.08564)\.

## Appendix AMLLM\-DCL Evaluation Details

We follow the MLLM\-DCL benchmark[Zhao et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib70)and report four standard metrics for continuous instruction tuning\. LetTTbe the total number of sequential tasks, and letAt,iA\_\{t,i\}denote the test accuracy on taskiiafter the model has finished training on tasktt\.

- •Mean Fine\-tune Accuracy \(MFT\)\.MFT measures plasticity, i\.e\., how well the model adapts to each task immediately after learning it: MFT=1T​∑i=1TAi,i\.\\mathrm\{MFT\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}A\_\{i,i\}\.\(9\)A higher MFT indicates stronger immediate adaptation to the downstream distribution of each new task\.
- •Mean Final Accuracy \(MFN\)\.MFN measures the model’s average performance on all tasks after the entire training sequence: MFN=1T​∑i=1TAT,i\.\\mathrm\{MFN\}=\\frac\{1\}\{T\}\\sum\_\{i=1\}^\{T\}A\_\{T,i\}\.\(10\)Unlike MFT, MFN accounts for later performance degradation, so a high MFN requires both learning new tasks and retaining previous ones\.
- •Mean Average Accuracy \(MAA\)\.MAA summarizes the accuracy trajectory throughout continual learning by averaging the cumulative accuracy over observed tasks at each training step: MAA=1T​∑t=1T\(1t​∑i=1tAt,i\)\.\\mathrm\{MAA\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\left\(\\frac\{1\}\{t\}\\sum\_\{i=1\}^\{t\}A\_\{t,i\}\\right\)\.\(11\)MAA rewards methods that maintain consistently high accuracy across all known tasks during the learning process\.
- •Backward Transfer \(BWT\)\.BWT quantifies the average final change in performance on previously learned tasks relative to their accuracy immediately after learning: BWT=1T−1​∑i=1T−1\(AT,i−Ai,i\)\.\\mathrm\{BWT\}=\\frac\{1\}\{T\-1\}\\sum\_\{i=1\}^\{T\-1\}\\left\(A\_\{T,i\}\-A\_\{i,i\}\\right\)\.\(12\)A less negative or positive BWT indicates less forgetting, while a negative BWT reflects degradation on earlier tasks\.

Table 7:Detailed CL metric summary on MLLM\-DCL\.For the detailed AWARe run in[Table8](https://arxiv.org/html/2608.11758#A1.T8), we use 200 MMMU samples as the calibration dataset when training the first RS task\. For each subsequent task, the calibration dataset consists of MMMU together with 100 samples from each previous task, which are used to recompute the activation distributions\. Rows indicate the latest task after which the model is fine\-tuned, and columns indicate evaluation tasks\.

Table 8:Detailed AWARe accuracy results on MLLM\-DCL\.
## Appendix BAdditional Model Qwen2\.5\-VL

To examine whether AWARe remains effective on a stronger recent backbone, we additionally evaluate Qwen2\.5\-VL[Bai et al\. 2025b](https://arxiv.org/html/2608.11758#bib.bib3)on two downstream tasks from MLLM\-DCL: Med and AD\. We use HallusionBench[Guan et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib14), Micro\-VQA[Burgess et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib5), DocVQA[Mathew et al\. 2021](https://arxiv.org/html/2608.11758#bib.bib44), VStarBench[Wu and Xie 2024](https://arxiv.org/html/2608.11758#bib.bib61), and AI2D[Kembhavi et al\. 2016](https://arxiv.org/html/2608.11758#bib.bib25)as upstream evaluation tasks to measure retained general multimodal capabilities after task\-specific tuning\. The zero\-shot row reports the original Qwen2\.5\-VL performance before downstream adaptation\.

Table 9:Qwen2\.5\-VL results on the MLLM\-DCL Med task\.The downstream column reports DCL\-Med adaptation performance, while the remaining columns report upstream task retention\. Boldface marks the best downstream result and the best upstream retention among fine\-tuned methods\.Table 10:Qwen2\.5\-VL results on the MLLM\-DCL AD task\.The downstream column reports DCL\-AD adaptation performance, while the remaining columns report upstream task retention\. Boldface marks the best downstream result and the best upstream retention among fine\-tuned methods\.Across both downstream tasks, SFT reaches the highest target\-task accuracy but causes severe degradation on upstream capabilities, especially on DocVQA, VStarBench, and AI2D\. In contrast, AWARe consistently preserves the strongest upstream performance among fine\-tuned methods while retaining competitive downstream adaptation, indicating that activation\-weighted parameter freezing transfers to Qwen2\.5\-VL and provides a robust stability\-plasticity trade\-off\.

## Appendix CParameter Efficiency Analysis

In this section, we provide a detailed calculation of the trainable parameter ratio in our optimal setting\. Consider a standard Transformer block in LLaMA architecture with hidden dimensionhh\. The parameters can be categorized into Self\-Attention and Feed\-Forward Network \(FFN\) modules:

- •Self\-Attention:Consists of four projection matrices:𝐖q,𝐖k,𝐖v,𝐖o∈ℝh×h\\mathbf\{W\}\_\{q\},\\mathbf\{W\}\_\{k\},\\mathbf\{W\}\_\{v\},\\mathbf\{W\}\_\{o\}\\in\\mathbb\{R\}^\{h\\times h\}\. The total parameter count is4​h24h^\{2\}\.
- •FFN \(SwiGLU\):Comprises three matrices:𝐖u​p,𝐖g​a​t​e∈ℝhi​n​t​e​r×h\\mathbf\{W\}\_\{up\},\\mathbf\{W\}\_\{gate\}\\in\\mathbb\{R\}^\{h\_\{inter\}\\times h\}and𝐖d​o​w​n∈ℝh×hi​n​t​e​r\\mathbf\{W\}\_\{down\}\\in\\mathbb\{R\}^\{h\\times h\_\{inter\}\}\. In LLaMA, the intermediate dimension is typically set tohi​n​t​e​r≈83​hh\_\{inter\}\\approx\\frac\{8\}\{3\}h\. The total parameter count is approximately3×83​h×h=8​h23\\times\\frac\{8\}\{3\}h\\times h=8h^\{2\}\.

Thus, the total number of parameters per block is approximately12​h212h^\{2\}\.

In our best experimental setting, we apply the AWARe strategy specifically to the query, key, and value projection layers \(𝐖q,𝐖k,𝐖v\\mathbf\{W\}\_\{q\},\\mathbf\{W\}\_\{k\},\\mathbf\{W\}\_\{v\}\) with a retention ratio ofρ=30%\\rho=30\\%, while freezing the remaining components\. The number of trainable parameters is calculated as:

Nt​r​a​i​n=3×\(1−ρ\)×h2=3×0\.7×h2=2\.1​h2N\_\{train\}=3\\times\(1\-\\rho\)\\times h^\{2\}=3\\times 0\.7\\times h^\{2\}=2\.1h^\{2\}\(13\)Consequently, the ratio of trainable parameters to the total parameter count is:

Ratio=2\.1​h212​h2≈17\.5%\\text\{Ratio\}=\\frac\{2\.1h^\{2\}\}\{12h^\{2\}\}\\approx 17\.5\\%\(14\)This analysis demonstrates that our method effectively updates only a quarter of the model parameters, significantly reducing computational overhead while preserving generalization capabilities\.

## Appendix DExperimental Configuration

We summarize the hyperparameter configuration that achieves the optimal performance reported in our main results in[Table11](https://arxiv.org/html/2608.11758#A4.T11), specifically under theGlobal\-Highest 30%setting\. All experimental results are averaged over three random runs\.

Table 11:Detailed Hyperparameter Settings for the Optimal AWARe Configuration\.
## Appendix EBaseline Details

Single\-task downstream adaptation\.We calculate baselines using the following configurations:

- •Full\-FT: Full fine\-tuning of the model parameters\.
- •LoRA\[ICLR’22\][Hu et al\. 2022](https://arxiv.org/html/2608.11758#bib.bib19): Low\-Rank Adaptation, which injects trainable rank decomposition matrices\.
- •DoRA\[ICML’24\][Liu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib37): Enhances LoRA’s learning capacity and training stability by decomposing weights into magnitude and direction components\.
- •DARE\[ICML’24\][Yu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib64): Parameters from the fine\-tuned model are randomly selected and rescaled to preserve both generalization and specialization capabilities\.
- •Orth\-Reg\[ECCV’24\][Hu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib20): Encourages fine\-tuned features to remain orthogonal to pretrained features, thereby preserving model generalization\.
- •Model Tailor\[ICML’24\][Zhu et al\. 2024](https://arxiv.org/html/2608.11758#bib.bib72): Pre\-trained parameters are preserved while a small fraction \(e\.g\., 10%\) of fine\-tuned parameters is replaced based on salience and sensitivity analysis\.
- •LoRASculpt\[CVPR’25\][Liang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib32): The method prunes redundant LoRA parameters via sparse updates guided by weight importance and conflict\-aware regularization\.
- •SPIDER\[ICML’25\][Huang et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib22): Updates parameters only where specialization knowledge is more critical than generalization knowledge\.

Continual instruction tuning\.For the MLLM\-DCL experiments, we compare against continual learning baselines that are designed for sequential task adaptation:

- •LoRA\-FT\[ICLR’22\][Hu et al\. 2022](https://arxiv.org/html/2608.11758#bib.bib19): Sequential LoRA fine\-tuning on each new subtask without an explicit forgetting\-mitigation mechanism\.
- •O\-LoRA\[EMNLP’23\][Wang et al\. 2023](https://arxiv.org/html/2608.11758#bib.bib60): Learns new tasks in orthogonal subspaces to reduce interference with previously acquired knowledge\.
- •MoELoRA\[NIPS’24\][Chen et al\. 2024a](https://arxiv.org/html/2608.11758#bib.bib7): Uses a mixture\-of\-LoRA\-experts design for continual instruction tuning\.
- •ModalPrompt\[EMNLP’25\][Zeng et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib66): Introduces dual\-modality guided prompts to preserve multimodal knowledge during continual learning\.
- •CL\-MoE\[CVPR’25\][Huai et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib21): Uses a dual\-momentum mixture\-of\-experts mechanism for continual multimodal visual question answering\.
- •HiDe\-LLaVA\[ACL’25\][Guo et al\. 2025a](https://arxiv.org/html/2608.11758#bib.bib15): Decouples multimodal continual instruction tuning hierarchically to mitigate forgetting\.
- •SEFE\[ICML’25\][Chen et al\. 2025](https://arxiv.org/html/2608.11758#bib.bib8): Separates superficial and essential forgetting to preserve important multimodal capabilities\.
- •DISCO\[ICCV’25\][Guo et al\. 2025b](https://arxiv.org/html/2608.11758#bib.bib16): Applies federated continual instruction tuning for sequential instruction adaptation\.

\*