Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning

# Decomposing the Basic Abilities of Large Language Models: Mitigating Cross-Task Interference in Multi-Task Instruct-Tuning
Source: [https://arxiv.org/html/2605.05676](https://arxiv.org/html/2605.05676)
###### Abstract

Recently, the prominent performance of large language models \(LLMs\) has been largely driven by multi\-task instruct\-tuning\. Unfortunately, this training paradigm suffers from a key issue, named cross\-task interference, due to conflicting gradients over shared parameters among different tasks\. Some previous methods mitigate this issue by isolating task\-specific parameters, e\.g\., task\-specific neuron selection and mixture\-of\-experts\. In this paper, we empirically reveal that the cross\-task interference still exists for the existing solutions because of many parameters also shared by different tasks, and accordingly, we propose a novel solution, namely Basic Abilities Decomposition for multi\-task Instruct\-Tuning \(Badit\)\. Specifically, we empirically find that certain parameters are consistently co\-activated, and that co\-activated parameters naturally organize into base groups\. This motivates us to analogize that LLMs encode several orthogonal basic abilities, and that any task can be represented as a linear combination of these abilities\. Accordingly, we proposeBaditthat decomposes LLM parameters into orthogonal high\-singular\-value LoRA experts representing basic abilities, and dynamically enforces their orthogonality during training via spherical clustering of rank\-1 components\. We conduct extensive experiments on theSuperNIbenchmark with 6 LLMs, and empirical results demonstrate thatBaditcan outperform SOTA methods and mitigate the degree of cross\-task interference\.

Machine Learning, ICML

## 1Introduction

Recently, the community has witnessed the rapid development of*large language models*\(LLMs\) in tackling a variety of downstream tasks, e\.g\., reasoning\(Yanget al\.,[2025a](https://arxiv.org/html/2605.05676#bib.bib77)\)and Q&A\(Huet al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib86)\)\. This remarkable capability is primarily attributed to the supervised fine\-tuning across massive data of diverse tasks, namelymulti\-task instruct\-tuning\(Wanget al\.,[2023b](https://arxiv.org/html/2605.05676#bib.bib16); Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1); Shiet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib21)\)\.

Despite the success of LLMs, their multi\-task instruct\-tuning faces a key challenge referred to ascross\-task interference, that is, training multiple tasks simultaneously inevitably results in conflicting gradients from different task data over shared LLM parameters, thereby degrading performance on each task\(McCloskey and Cohen,[1989](https://arxiv.org/html/2605.05676#bib.bib12); Luoet al\.,[2023](https://arxiv.org/html/2605.05676#bib.bib14)\)\. To handle this, a basic assumption in multi\-task instruct\-tuning is thatdifferent tasks consistently activate distinct subsets of LLM parameters\(Leng and Xiong,[2025](https://arxiv.org/html/2605.05676#bib.bib71); Tanget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib72)\)\. Accordingly, some previous methods identify and isolate task‑specific parameters\(Tanget al\.,[2021](https://arxiv.org/html/2605.05676#bib.bib73); Zhaoet al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib38)\)\. For example,Leng and Xiong \([2025](https://arxiv.org/html/2605.05676#bib.bib71)\)locates task\-specific neurons in LLMs through gradient attribution and selectively trains only those parameters\. Another line of research applies*mixture\-of\-experts*\(MoE\) to separate the parameters associated with different tasks\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17); Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\), incrementally introducing multiple experts to explicitly isolate task\-specific parameters, while enforcing orthogonality constraints among the parameters of different experts\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)\.

In this paper, we empirically reveal that these existing solutions still suffer from cross\-task interference because of many parameters still shared by different tasks, and accordingly, we propose a novel solution to further mitigate it\. Specifically, we perform empirical evaluations with standard methods on the benchmark dataset with 15 different tasks\. The experimental results in Figs\.[1](https://arxiv.org/html/2605.05676#S1.F1)and[2](https://arxiv.org/html/2605.05676#S1.F2)reveal the significant overlap among the task\-specific parameters cross different tasks\. We analyze 15 tasks\(Wanget al\.,[2022b](https://arxiv.org/html/2605.05676#bib.bib67)\), and illustrate how they activate different neurons, parameters, and MoE experts across various LLMs, respectively\. The experimental results show that the majority of neurons, parameters, and MoE experts are simultaneously activated by multiple tasks\. Therefore, prior multi\-task instruct\-tuning methods fail to fully isolate task\-specific parameters, leaving a substantial number of shared parameters that still exhibit conflicting gradients across tasks during optimization\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x1.png)Figure 1:Shared task\-specific neurons and parameters across different tasks\.![Refer to caption](https://arxiv.org/html/2605.05676v1/x2.png)Figure 2:Activated experts in MoE\-based LLMs by different tasks\.To further mitigate cross\-task interference, we propose a new method, namely*Basic Abilities Decomposition for multi\-task Instruct\-Tuning*\(Badit\)\. Specifically,Baditis inspired by an empirical observation in Fig\.[1](https://arxiv.org/html/2605.05676#S1.F1)that certain neurons and parameters are always co\-activated across tasks, and these co\-activated ones can be naturally grouped into several base groups\. This motivates us todecompose LLM parameters into a set of orthogonal base groups, namedbasic ability, and formulate each ability as an MoE expert, thereby mitigating cross\-task interference by isolating these abilities, instead of tasks\. To implement this idea,Baditinvolves two key strategies\. First, we decompose the LLM parameters via singular value decomposition\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)into a linear combination of multiple large\-singular\-valuelow\-rank adaptation\(LoRA\) experts\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)and a residual term comprising low\-singular\-value ones\. Since the decomposed LoRA experts are naturally orthogonal, they serve as initial basic abilities\. Second, during multi\-task instruct\-tuning, to ensure that each LoRA expert consistently represents a distinct basic ability, we introduce a dynamic grouping strategy based on spherical clustering\. It enforces orthogonality among the optimization gradients of different LoRA experts while maintaining similar angles among the rank\-1 components within each LoRA\.

To evaluate the performance ofBadit, we conduct experiments on 6 different LLMs usingSuperNI\(Wanget al\.,[2022b](https://arxiv.org/html/2605.05676#bib.bib67)\), a multi\-task dataset encompassing 15 diverse tasks\. Generally,Baditoutperforms the state\-of\-the\-art multi\-task instruct\-tuning approach, GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\), by an average of 2\.68Rougescores, and quantitatively demonstrates reduced cross\-task interference\.Our source code has been released in the repository[https://github\.com/wangbing1416/BADIT](https://github.com/wangbing1416/BADIT)\.

Our contributions can be summarized as three\-fold:

- •Our experiments reveal that different tasks activate shared parameter subsets, and certain shared ones are always co\-activated across tasks\. We analogize the co\-activated subsets as the LLM’s basic abilities\.
- •We proposeBadit, which decomposes the basic abilities from the LLM’s parameters as MoE experts and represents any task as a combination of these abilities\.
- •Extensive experiments are conducted to demonstrate thatBaditcan alleviate cross\-task interference\.

## 2Preliminary Experiments

In this section, we present preliminary experiments demonstrating that, during the multi\-task instruct\-tuning of LLMs, their activated parameters can be organized as multiple base groups \(we refer to them asbasic abilities\)\.

### 2\.1Experimental Settings

FollowingZhaoet al\.\([2024b](https://arxiv.org/html/2605.05676#bib.bib38)\)andLianget al\.\([2025](https://arxiv.org/html/2605.05676#bib.bib82)\), we conduct an empirical study using 15 different tasks fromSuperNI\(Wanget al\.,[2022b](https://arxiv.org/html/2605.05676#bib.bib67)\), a multi\-task instruct\-tuning dataset\. Our investigation analyzes activation patterns of neurons and parameters in Llama3\-3B\(AI@Meta,[2024](https://arxiv.org/html/2605.05676#bib.bib3)\)and Qwen3\-4B\(Yanget al\.,[2025a](https://arxiv.org/html/2605.05676#bib.bib77)\), and evaluates their task\-shared activations\. Furthermore, we integrate a lightweight MoE architecture, termed LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\), into these LLMs to examine how different tasks activate these experts when each task is trained independently\. We fix the number of LoRA experts at 8 and select the top\-4 LoRA experts with the highest routing weights\.

How to identify activated neurons and parameters across tasks?To investigate how different task datasets influence LLM parameters, we begin by analyzing the sharedneuronsactivated by these tasks\. Formally, given a dataset𝒟t\\mathcal\{D\}\_\{t\}for thett\-th task, an activated neuron is defined as one hidden state whose output in a forward pass is non\-zero or significantly different from zero\. For example, consider the average activation in a feed\-forward layer computed over the task dataset𝐡=1\|𝒟t\|​∑𝐱∈𝒟tσ​\(𝐖𝐱\+𝐛\)\\mathbf\{h\}=\\frac\{1\}\{\|\\mathcal\{D\}\_\{t\}\|\}\\sum\_\{\\mathbf\{x\}\\in\\mathcal\{D\}\_\{t\}\}\\sigma\(\\mathbf\{W\}\\mathbf\{x\}\+\\mathbf\{b\}\), a neuronhih\_\{i\}is deemed activated if\|hi\|\>ϵ\|h\_\{i\}\|\>\\epsilon, withϵ\\epsilonbeing a small positive threshold\. Additionally, followingSonget al\.\([2024](https://arxiv.org/html/2605.05676#bib.bib70)\), we also examine activated LLM parameters throughgradientsand assess the task\-shared parameters\. Specifically, given𝒟t\\mathcal\{D\}\_\{t\}, we calculate the Fisher information matrix\(Wuet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib80)\)towards the gradient of a specific parameter matrix𝐖i\\mathbf\{W\}\_\{i\}in full LLM parameters𝜽\\boldsymbol\{\\theta\}\. We then use the diagonal entries of this matrix, each representing the expected variance of the score function for the corresponding parameter, as a measure of parameter activation as follows:

Fi​i=𝔼𝐱∼𝒟t​\[\(∂∂𝐖i​log⁡p𝜽​\(y\|𝐱\)\)2\],𝐖i∈𝜽,F\_\{ii\}=\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\mathcal\{D\}\_\{t\}\}\\left\[\\left\(\\frac\{\\partial\}\{\\partial\\mathbf\{W\}\_\{i\}\}\\log p\_\{\\boldsymbol\{\\theta\}\}\(y\|\\mathbf\{x\}\)\\right\)^\{2\}\\right\],\\ \\mathbf\{W\}\_\{i\}\\in\\boldsymbol\{\\theta\},where we implementlog⁡p𝜽​\(y\|𝐱\)\\log p\_\{\\boldsymbol\{\\theta\}\}\(y\|\\mathbf\{x\}\)by the supervised fine\-tuning loss\. Prior work suggests that LLMs primarily store knowledge in their feed\-forward networks\(Gevaet al\.,[2021](https://arxiv.org/html/2605.05676#bib.bib76); Daiet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib75)\), particularly within their gating layers\(Songet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib70)\)\. Consequently, our subsequent experiments examine the gating layer, and the results for other components, e\.g\., self\-attention layers, and additional analyses across more LLMs, are provided in Appendix[H\.1](https://arxiv.org/html/2605.05676#A8.SS1)\.

### 2\.2Task\-Specific Activations Are Still Shared

The activated neurons and parameters of the two LLMs are visualized in Fig\.[1](https://arxiv.org/html/2605.05676#S1.F1)\. The two left subplots depict neuron activation patterns, where each row corresponds to the set of neurons activated for a specific task\. The two right subplots display parameter\-level activations, where each column represents a specific task and indicates the number of activated parameters in the corresponding row of the parameter matrix for that task\. Additionally, Fig\.[2](https://arxiv.org/html/2605.05676#S1.F2)illustrates how different tasks activate the 8 MoE experts, based on the routing weights produced by the gate module in the final layer of the LLMs\. In all figures described above, activated components are marked in yellow\.

The experimental results in Fig\.[1](https://arxiv.org/html/2605.05676#S1.F1)show that,for each task, most activated neurons or parameters are shared with those of other tasks\. Some neurons and parameters are even shared across 15 tasks simultaneously\. Notably, in Qwen3\-4B, the activated neurons and parameters are predominantly shared among 15 tasks at lower IDs\. The results in Fig\.[2](https://arxiv.org/html/2605.05676#S1.F2)also reveal that even with a relatively advanced MoE architecture, a substantial number of experts are jointly activated across tasks\. These shared experts still suffer from interference due to conflicting gradients arising from different tasks\.

### 2\.3LLMs Can be Decomposed as Basic Abilities

Based on the results presented in Fig\.[1](https://arxiv.org/html/2605.05676#S1.F1), we can draw several preliminary observations\. First, whether considering neurons or LLM parameters, activations are consistently concentrated within a small subset, while the vast majority remain inactive\. Second, we further observe that certain neurons and parameters are consistently co\-activated across multiple tasks\. These co\-activated units tend to form a small number of recurring base activation patterns\. For example, in both Llama3\-3B and Qwen3\-4B, there exist specific neurons and parameters that are simultaneously activated by exactly 15 tasks\. These observations suggest that such co\-activated neurons and parameters likely fulfill shared functional roles—akin to the model possessing a repertoire of basic, reusable abilities, with each complex task being realized through a composition of these underlying primitives\. This insight motivates our core approach:decomposing the distinct, orthogonal basic abilitiesinherently encoded in LLMs\. By isolating these basic units, we aim to disentangle task\-specific interference\.

## 3The ProposedBaditMethod

In this section, we first present the technical preliminaries in Sec\.[3\.1](https://arxiv.org/html/2605.05676#S3.SS1)\. We then provide an overview of our proposedBaditmethod in Sec\.[3\.2](https://arxiv.org/html/2605.05676#S3.SS2)\. The two core steps ofBaditare described in detail in Secs\.[3\.3](https://arxiv.org/html/2605.05676#S3.SS3)and[3\.4](https://arxiv.org/html/2605.05676#S3.SS4), respectively\.

### 3\.1Technical Preliminaries

Multi\-task instruct\-tuning\.GivenTTnatural language tasks\{𝒟t\}t=1T\\\{\\mathcal\{D\}\_\{t\}\\\}\_\{t=1\}^\{T\},⋂t=1T𝒟t=∅\\bigcap\_\{t=1\}^\{T\}\\mathcal\{D\}\_\{t\}=\\varnothing, where𝒟t=\{\(𝐱it,yit\)\}i=1\|𝒟t\|\\mathcal\{D\}\_\{t\}=\\\{\(\\mathbf\{x\}\_\{i\}^\{t\},y\_\{i\}^\{t\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{D\}\_\{t\}\|\}represents task\-specific instruct samples, the goal of multi\-task instruct\-tuning is to tune a pre\-trained LLMℱ​\(⋅;𝜽0\)\\mathcal\{F\}\(\\cdot\\ ;\\boldsymbol\{\\theta\}^\{0\}\)across tasks while mitigating cross\-task interference\. Formally, for the task𝒟t\\mathcal\{D\}\_\{t\}, we optimize LLM parameters as

𝜽t=arg⁡min𝜽⁡1\|𝒟t\|​∑\(𝐱it,yit\)∈𝒟t\[ℒ​\(ℱ​\(𝐱it;𝜽\),yit\)\],\\boldsymbol\{\\theta\}^\{t\}=\\arg\\min\\limits\_\{\\boldsymbol\{\\theta\}\}\\frac\{1\}\{\|\\mathcal\{D\}\_\{t\}\|\}\\sum\\nolimits\_\{\(\\mathbf\{x\}\_\{i\}^\{t\},y\_\{i\}^\{t\}\)\\in\\mathcal\{D\}\_\{t\}\}\\Big\[\\mathcal\{L\}\\big\(\\mathcal\{F\}\(\\mathbf\{x\}\_\{i\}^\{t\};\\boldsymbol\{\\theta\}\),y\_\{i\}^\{t\}\\big\)\\Big\],\(1\)whereℒ​\(⋅,⋅\)\\mathcal\{L\}\(\\cdot,\\cdot\)is a supervised fine\-tuning loss\.

Low\-rank adaptation \(LoRA\)\.LoRA is a parameter\-efficient fine\-tuning method that injects trainable low\-rank matrices into each LLM weight matrix\. During fine\-tuning, the LLM weight matrix𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}\(e\.g\., MLP layers\) remains frozen, and the adapted forward pass becomes

𝐡=\(𝐖\+Δ​𝐖\)​𝐱=\(𝐖\+𝐀𝐁\)​𝐱,\\mathbf\{h\}=\(\\mathbf\{W\}\+\\Delta\\mathbf\{W\}\)\\mathbf\{x\}=\(\\mathbf\{W\}\+\\mathbf\{A\}\\mathbf\{B\}\)\\mathbf\{x\},\(2\)where𝐀∈ℝm×r\\mathbf\{A\}\\in\\mathbb\{R\}^\{m\\times r\}and𝐁∈ℝr×n\\mathbf\{B\}\\in\\mathbb\{R\}^\{r\\times n\}are trainable low\-rank matrices,r≪min⁡\(m,n\)r\\ll\\min\(m,n\)controls the rank w\.r\.t\. parameter efficiency\. Additionally,𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}are respectively initialized by theKaiminginitialization\(Heet al\.,[2015](https://arxiv.org/html/2605.05676#bib.bib31)\)and a zero matrix to keep the initialΔ​𝐖𝐱=𝟎\\Delta\\mathbf\{W\}\\mathbf\{x\}=\\mathbf\{0\}, preserving the pre\-trained LLM’s behavior at the start of fine\-tuning\.

Singular value decomposition \(SVD\)\.SVD is a classical matrix factorization algorithm, and any matrix𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}can be decomposed into a product of matrices as follows:

𝐖=𝐔​𝚺​𝐕⊤,\\mathbf\{W\}=\\mathbf\{U\}\\boldsymbol\{\\Sigma\}\\mathbf\{V\}^\{\\top\},\(3\)where𝐔∈ℝm×m\\mathbf\{U\}\\in\\mathbb\{R\}^\{m\\times m\}and𝐕∈ℝn×n\\mathbf\{V\}\\in\\mathbb\{R\}^\{n\\times n\}are orthogonal matrices whose columns correspond to the left and right singular vectors of𝐖\\mathbf\{W\}, respectively, and𝚺\\boldsymbol\{\\Sigma\}is a diagonal matrix containing the singular valuesσ1,…,σmin⁡\(m,n\)\\sigma\_\{1\},\\ldots,\\sigma\_\{\\min\(m,n\)\}arranged in descending order\. Previous works, e\.g\.,principal singular values and singular vectors adaptation\(PiSSA\)\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5); Yanget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib35)\), apply SVD in Eq\. \([3](https://arxiv.org/html/2605.05676#S3.E3)\) to the LLM weight matrix𝐖\\mathbf\{W\}to identify its principal components, and formulate a LoRA analogous to Eq\. \([2](https://arxiv.org/html/2605.05676#S3.E2)\) as follows:

𝐡=𝐖𝐱→SVD𝐔​𝚺​𝐕⊤​𝐱→truncate\(𝐀𝐁\+𝐖^\)​𝐱,\\mathbf\{h\}=\\mathbf\{W\}\\mathbf\{x\}\\xrightarrow\{\\text\{SVD\}\}\\mathbf\{U\}\\boldsymbol\{\\Sigma\}\\mathbf\{V\}^\{\\top\}\\mathbf\{x\}\\xrightarrow\{\\text\{truncate\}\}\(\\mathbf\{A\}\\mathbf\{B\}\+\\mathbf\{\\widehat\{W\}\}\)\\mathbf\{x\},\(4\)where𝐖^\\mathbf\{\\widehat\{W\}\},𝐀\\mathbf\{A\}, and𝐁\\mathbf\{B\}are respectively the residual matrix and two low\-rank matrices as follows:

𝐀=𝐔\[:r\]diag\(\\displaystyle\\mathbf\{A\}=\\mathbf\{U\}\_\{\[:r\]\}\\,\\mathrm\{diag\}\\big\(𝚺\[:r\]\),𝐁=diag\(𝚺\[:r\]\)𝐕\[:r\]⊤,\\displaystyle\\sqrt\{\\boldsymbol\{\\Sigma\}\_\{\[:r\]\}\}\\big\),\\ \\mathbf\{B\}=\\mathrm\{diag\}\\big\(\\sqrt\{\\boldsymbol\{\\Sigma\}\_\{\[:r\]\}\}\\big\)\\mathbf\{V\}\_\{\[:r\]\}^\{\\top\},\(5\)𝐖^\\displaystyle\\mathbf\{\\widehat\{W\}\}=𝐔\[r:\]​diag​\(𝚺\[r:\]\)​𝐕\[r:\]⊤\.\\displaystyle=\\mathbf\{U\}\_\{\[r:\]\}\\,\\mathrm\{diag\}\\big\(\\boldsymbol\{\\Sigma\}\_\{\[r:\]\}\\big\)\\mathbf\{V\}\_\{\[r:\]\}^\{\\top\}\.wherediag​\(⋅\)\\text\{diag\}\(\\cdot\)denotes the diagonal matrix whose values are the singular values of the matrix\.

### 3\.2Method Overview

The primary idea ofBaditis to decompose the basic abilities embedded within the LLM parameters and leverage these abilities to represent any downstream task\. Generally, ourBaditmethod consists of two main steps:Basic ability decomposition\(BAD\) disentangles an initial set of orthogonal LoRA experts from the LLM’s parameter matrices, where each expert corresponds to a different basic ability;Dynamically orthogonal grouping\(DOG\) dynamically groups rank\-1 components within LoRA during training to preserve orthogonality among these basic abilities\. The overall framework ofBaditis depicted in Fig\.[3](https://arxiv.org/html/2605.05676#S3.F3)\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x3.png)Figure 3:Comparison with prior multi\-task instruct\-tuning methods\. We decouple key LLM parameters into basic abilities \(i\.e\., LoRA experts\) and dynamically group their rank\-1 components, enforcing orthogonality to represent distinct basic abilities\.Specifically, given a pre\-trained LLMℱ​\(⋅;𝜽0\)\\mathcal\{F\}\(\\cdot\\ ;\\boldsymbol\{\\theta\}^\{0\}\), the BAD step begins by decomposing each pre\-trained weight matrix𝐖0∈𝜽0\\mathbf\{W\}^\{0\}\\in\\boldsymbol\{\\theta\}^\{0\}into a weighted combination of multiple LoRA components, as𝐖0=∑k=1Kαk​𝐀k​𝐁k\+𝐖^,\\mathbf\{W\}^\{0\}=\\sum\\nolimits\_\{k=1\}^\{K\}\\alpha\_\{k\}\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}\+\\mathbf\{\\widehat\{W\}\},using the extended version of the SVD method in Eq\. \([4](https://arxiv.org/html/2605.05676#S3.E4)\), whereKKrepresents the number of LoRA experts,αk\\alpha\_\{k\}is a learnable routing weight, and𝐖^\\mathbf\{\\widehat\{W\}\}represents the residual term\. By the decomposition, we interpret each LoRA component𝐀k​𝐁k\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}as an initial basic ability embedded within the pre\-trained weights\. During the supervised fine\-tuning process in Eq\. \([1](https://arxiv.org/html/2605.05676#S3.E1)\), we freeze the residual weights𝐖^\\mathbf\{\\widehat\{W\}\}and solely update the LoRA parameters\.

To further encourage each LoRA expert to specialize in a different basic ability during fine\-tuning, our method follows the hypothesis thatthe training gradients of different basic abilities should be mutually orthogonal\(Linet al\.,[2022a](https://arxiv.org/html/2605.05676#bib.bib65),[b](https://arxiv.org/html/2605.05676#bib.bib66)\)\. To implement this principle, the DOG step introduces a dynamic adaptation mechanism that progressively groups the rank\-1 components of the initialized LoRA experts\{𝐀k​𝐁k\}k=1K\\\{\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}\\\}\_\{k=1\}^\{K\}during the fine\-tuning of the LLM\. This grouping strategy actively enforces orthogonality among the training gradients of different LoRA modules\. We provide detailed descriptions of the BAD and DOG steps in Secs\.[3\.3](https://arxiv.org/html/2605.05676#S3.SS3)and[3\.4](https://arxiv.org/html/2605.05676#S3.SS4), respectively\.

### 3\.3Basic Ability Decomposition

The BAD step aims to decompose the pre\-trained weight matrices∀𝐖0∈𝜽0\\forall\\ \\mathbf\{W\}^\{0\}\\in\\boldsymbol\{\\theta\}^\{0\}of the LLM into multiple LoRA experts, each intended to capture a different basic ability implicitly encoded in𝐖0\\mathbf\{W\}^\{0\}\. Inspired by previous SVD methods on LLM parameters\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5); Yanget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib35)\), we decompose𝐖0\\mathbf\{W\}^\{0\}as follows:

𝐖0→SVD\\displaystyle\\mathbf\{W\}^\{0\}\\xrightarrow\{\\text\{SVD\}\}𝐔\[:r​K\]​𝚺\[:r​K\]​𝐕\[:r​K\]⊤\+𝐔\[rK:\]​𝚺\[rK:\]​𝐕\[rK:\]⊤\\displaystyle\\mathbf\{U\}\_\{\[:rK\]\}\\boldsymbol\{\\Sigma\}\_\{\[:rK\]\}\\mathbf\{V\}\_\{\[:rK\]\}^\{\\top\}\+\\mathbf\{U\}\_\{\[rK:\]\}\\boldsymbol\{\\Sigma\}\_\{\[rK:\]\}\\mathbf\{V\}\_\{\[rK:\]\}^\{\\top\}=\\displaystyle=\(𝐔\[:r​K\]​diag​\(𝚺\[:r​K\]\)\)​\(diag​\(𝚺\[:r​K\]\)​𝐕\[:r​K\]⊤\)\\displaystyle\\Big\(\\mathbf\{U\}\_\{\[:rK\]\}\\text\{diag\}\\big\(\\sqrt\{\\boldsymbol\{\\Sigma\}\_\{\[:rK\]\}\}\\big\)\\Big\)\\Big\(\\text\{diag\}\\big\(\\sqrt\{\\boldsymbol\{\\Sigma\}\_\{\[:rK\]\}\}\\big\)\\mathbf\{V\}\_\{\[:rK\]\}^\{\\top\}\\Big\)\+𝐔\[rK:\]​𝚺\[rK:\]​𝐕\[rK:\]⊤\\displaystyle\\hskip 0\.0pt\+\\mathbf\{U\}\_\{\[rK:\]\}\\boldsymbol\{\\Sigma\}\_\{\[rK:\]\}\\mathbf\{V\}\_\{\[rK:\]\}^\{\\top\}\(6\)→truncate\\displaystyle\\xrightarrow\{\\text\{truncate\}\}∑k=1K\(𝐔\[r​\(k−1\):r​k\]​diag​\(𝚺\[r​\(k−1\):r​k\]\)\)\\displaystyle\\sum\\nolimits\_\{k=1\}^\{K\}\\Big\(\\mathbf\{U\}\_\{\[r\(k\-1\):rk\]\}\\text\{diag\}\\big\(\\sqrt\{\\boldsymbol\{\\Sigma\}\_\{\[r\(k\-1\):rk\]\}\}\\big\)\\Big\)\(diag​\(𝚺\[r​\(k−1\):r​k\]\)​𝐕\[r​\(k−1\):r​k\]⊤\)\\displaystyle\\hskip 31\.86694pt\\Big\(\\text\{diag\}\\big\(\\sqrt\{\\boldsymbol\{\\Sigma\}\_\{\[r\(k\-1\):rk\]\}\}\\big\)\\mathbf\{V\}\_\{\[r\(k\-1\):rk\]\}^\{\\top\}\\Big\)\+𝐔\[rK:\]​𝚺\[rK:\]​𝐕\[rK:\]⊤=∑k=1K𝐀k​𝐁k\+𝐖^,\\displaystyle\\hskip 0\.0pt\+\\mathbf\{U\}\_\{\[rK:\]\}\\boldsymbol\{\\Sigma\}\_\{\[rK:\]\}\\mathbf\{V\}\_\{\[rK:\]\}^\{\\top\}=\\sum\\nolimits\_\{k=1\}^\{K\}\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}\+\\mathbf\{\\widehat\{W\}\},where𝐀k​𝐁k\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}denotes thekk\-th LoRA expert withrrranks, which encapsulates thekk\-th basic ability\. These LoRA modules correspond to the singular vectors associated with the topr​KrKlargest singular values obtained via SVD, i\.e\., they capture the most salient components of the original parameter matrix\. Meanwhile,𝐖^\\mathbf\{\\widehat\{W\}\}represents a residual term that accounts for the less important parameters in the pre\-trained weights \(those associated with relatively small singular values\), and we freeze its gradients during the subsequent fine\-tuning process\. In fact, following the aforementioned SVD\-based decoupling, the parameters of different LoRA experts are inherently orthogonal\. We provide a formal proof of this property in Appendix[A](https://arxiv.org/html/2605.05676#A1)\. Consequently, these orthogonal LoRA experts naturally represent mutually orthogonal basic abilities\. Based on this decomposition, we obtain a group of initialized LoRA experts\. To further align this model with effective MoE\-based architectures\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1); Liet al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib2)\), we introduce learnable routing weights for each expert as follows:

𝐖0→SVD∑k=1Kαk​𝐀k​𝐁k\+𝐖^,\\mathbf\{W\}^\{0\}\\xrightarrow\{\\text\{SVD\}\}\\sum\\nolimits\_\{k=1\}^\{K\}\\alpha\_\{k\}\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}\+\\mathbf\{\\widehat\{W\}\},\(7\)where the routing weightαk\\alpha\_\{k\},k∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}are initialized to 1 to keep it equal to Eq\. \([3\.3](https://arxiv.org/html/2605.05676#S3.Ex2)\) and are optimized across different tasks\. The routing weights are crucial for enabling the LLM to adaptively select the corresponding basic abilities required for inference on new tasks\.

### 3\.4Dynamically Orthogonal Grouping

Since LoRA experts inevitably lose their orthogonality advantage after multi\-task instruct\-tuning, their nature to represent distinct basic abilities is compromised\. To address it, the DOG step is introduced to dynamically regroup the rank\-1 components w\.r\.t\. ther​KrKsingular values from SVD of each LoRA during training, ensuring that their gradients remain mutually orthogonal throughout the optimization\.

Formally, given the dataset\(𝐱t,yt\)\(\\mathbf\{x\}^\{t\},y^\{t\}\)from thett\-th task, the training gradient of the globally\-indexedii\-th rank\-1 LoRA component\[𝐚i;𝐛i⊤\],i∈\{1,…,r​K\}\\big\[\\mathbf\{a\}\_\{i\};\\mathbf\{b\}\_\{i\}^\{\\top\}\\big\],i\\in\\\{1,\\ldots,rK\\\}is formulated as

𝐠i=\[∂ℒ​\(ℱ​\(𝐱t;𝜽\),yt\)∂𝐚i;∂ℒ​\(ℱ​\(𝐱t;𝜽\),yt\)∂𝐛i⊤\],\\mathbf\{g\}\_\{i\}=\\left\[\\frac\{\\partial\\mathcal\{L\}\\big\(\\mathcal\{F\}\(\\mathbf\{x\}^\{t\};\\boldsymbol\{\\theta\}\),y^\{t\}\\big\)\}\{\\partial\\mathbf\{a\}\_\{i\}\};\\frac\{\\partial\\mathcal\{L\}\\big\(\\mathcal\{F\}\(\\mathbf\{x\}^\{t\};\\boldsymbol\{\\theta\}\),y^\{t\}\\big\)\}\{\\partial\\mathbf\{b\}\_\{i\}^\{\\top\}\}\\right\],\(8\)where𝐚i\\mathbf\{a\}\_\{i\}and𝐛i\\mathbf\{b\}\_\{i\}denote a specific column of𝐀\\mathbf\{A\}and a specific row of𝐁\\mathbf\{B\}, respectively, corresponding to a particular expert amongKKexperts and associated with one of therrranks\. Given the gradients of these rank\-1 components, our goal is to derive a dynamic grouping strategy𝚷∈\(0,1\)r​K×K\\boldsymbol\{\\Pi\}\\in\(0,1\)^\{rK\\times K\}that reassigns ther​KrKrank\-1 components intoKKnew rank\-rrLoRA experts, where eachπi,k=1/0\\pi\_\{i,k\}=1/0denotes whether the originalii\-th LoRA component is assigned / not assigned to the newkk\-th expert\. We aim to optimize the strategy𝚷\\boldsymbol\{\\Pi\}so that the gradient directions of the resultingKKnew experts are as mutually orthogonal as possible, while the gradient directions of the rank\-1 components within each expert are similar\. This design ensures that each newly formed expert still retains a distinct and orthogonal basic ability\.

More specifically, since we focus solely on the directions of gradients as the grouping criterion, we first normalize all gradients onto the unit sphere as follows:

𝐠^i=𝐠i/‖𝐠i‖2\.\\mathbf\{\\widehat\{g\}\}\_\{i\}=\\mathbf\{g\}\_\{i\}\\ /\\ \\\|\\mathbf\{g\}\_\{i\}\\\|\_\{2\}\.\(9\)Then, to minimize the angular deviation among the rank\-1 components’ gradients within each LoRA expert, we adopt the following objective:

max𝚷​∑k=1K∑i,j=1r​Kπi,k​πj,k​⟨𝐠^i,𝐠^j⟩=max𝚷​∑k=1K‖∑i=1r​Kπi,k​𝐠^i‖2\.\\max\_\{\\boldsymbol\{\\Pi\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i,j=1\}^\{rK\}\\pi\_\{i,k\}\\pi\_\{j,k\}\\left\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{\\widehat\{g\}\}\_\{j\}\\right\\rangle=\\max\_\{\\boldsymbol\{\\Pi\}\}\\sum\_\{k=1\}^\{K\}\\left\\\|\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\}\.\(10\)Actually, optimizing this objective also inherently encourages larger, potentially even orthogonal, angular deviation between the gradients of different LoRA experts\. A detailed derivation of this objective function and its natural advantages are provided in Appendix[B](https://arxiv.org/html/2605.05676#A2)\. To implement this objective and further enforce a stronger orthogonality constraint, we design an iterative optimization algorithm\. First, we obtain an initial grouping strategy𝚷\(0\)\\boldsymbol\{\\Pi\}^\{\(0\)\}by performing sphericalKK\-means clustering by optimizing

max𝚷\(0\),𝒞\(0\)​∑k=1K∑i=1r​Kπi,k​⟨𝐠^i,𝐜k\(0\)⟩,‖𝐜k\(0\)‖=1,\\max\_\{\\boldsymbol\{\\Pi\}^\{\(0\)\},\\mathcal\{C\}^\{\(0\)\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\left\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{c\}\_\{k\}^\{\(0\)\}\\right\\rangle,\\ \\left\\\|\\mathbf\{c\}\_\{k\}^\{\(0\)\}\\right\\\|=1,\(11\)where𝒞\(0\)=\{𝐜k\(0\)\}k=1K\\mathcal\{C\}^\{\(0\)\}=\\\{\\mathbf\{c\}\_\{k\}^\{\(0\)\}\\\}\_\{k=1\}^\{K\}denotes the unit center direction of each cluster\. Then, theτ\\tau\-th iteration of our update algorithm to obtain the optimal𝚷\\boldsymbol\{\\Pi\}consists of the following three steps:

Step 1: Compute the current centroid vector for each cluster\.

𝐜k\(τ\)=∑i=1r​Kπi,k\(τ\)​𝐠^i,k∈\{1,…,K\}\.\\mathbf\{c\}\_\{k\}^\{\(\\tau\)\}=\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}^\{\(\\tau\)\}\\mathbf\{\\widehat\{g\}\}\_\{i\},\\ k\\in\\\{1,\\ldots,K\\\}\.\(12\)Step 2: To orthogonalize these cluster centroids, we perform SVD on the matrix formed by these centroids as

𝐂\(τ\)=\[𝐜1\(τ\),…,𝐜K\(τ\)\]=𝐔c​𝚺c​𝐕c⊤,\\mathbf\{C\}^\{\(\\tau\)\}=\\big\[\\mathbf\{c\}\_\{1\}^\{\(\\tau\)\},\\ldots,\\mathbf\{c\}\_\{K\}^\{\(\\tau\)\}\\big\]=\\mathbf\{U\}\_\{c\}\\boldsymbol\{\\Sigma\}\_\{c\}\\mathbf\{V\}\_\{c\}^\{\\top\},\(13\)and denote the resulting orthogonal basis as𝐐\(τ\)=𝐔c​𝐕c⊤\\mathbf\{Q\}^\{\(\\tau\)\}=\\mathbf\{U\}\_\{c\}\\mathbf\{V\}\_\{c\}^\{\\top\}\. Accordingly, the orthogonalized cluster directions, given by the columns\{𝐪k\(τ\)\}k=1K=𝐐\(τ\)\\\{\\mathbf\{q\}\_\{k\}^\{\(\\tau\)\}\\\}\_\{k=1\}^\{K\}=\\mathbf\{Q\}^\{\(\\tau\)\}, satisfy the required orthogonality constraint\(𝐪k\(τ\)\)⊤​𝐪k\(τ\)=1\\big\(\\mathbf\{q\}\_\{k\}^\{\(\\tau\)\}\\big\)^\{\\top\}\\mathbf\{q\}\_\{k\}^\{\(\\tau\)\}=1and\(𝐪k\(τ\)\)⊤​𝐪l≠k\(τ\)=0\\big\(\\mathbf\{q\}\_\{k\}^\{\(\\tau\)\}\\big\)^\{\\top\}\\mathbf\{q\}\_\{l\\neq k\}^\{\(\\tau\)\}=0\.

Step 3: Compute the similarity⟨𝐠^i,𝐪k\(τ\)⟩\\big\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{q\}\_\{k\}^\{\(\\tau\)\}\\big\\ranglebetween the gradient and the orthogonalized cluster directions, and solve the following integer optimization problem to obtain the updated policy𝚷\(τ\)\\boldsymbol\{\\Pi\}^\{\(\\tau\)\}:

max𝚷\(τ\)​∑k=1K∑i=1r​Kπi,k\(τ\)​⟨𝐠^i,𝐪k\(τ\)⟩,∑i=1r​Kπi,k\(τ\)=r\.\\max\_\{\\boldsymbol\{\\Pi\}^\{\(\\tau\)\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}^\{\(\\tau\)\}\\big\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{q\}\_\{k\}^\{\(\\tau\)\}\\big\\rangle,\\ \\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}^\{\(\\tau\)\}=r\.\(14\)
The iterative algorithm terminates when the strategy𝚷\\boldsymbol\{\\Pi\}stabilizes \(i\.e\., no longer changes\) or after a maximum of 10 optimization steps\. Using the final strategy𝚷\\boldsymbol\{\\Pi\}, we regroup the rank\-1 LoRA components accordingly\. Notably, the output of the MoE architecture remains invariant after our regrouping, even though each expert possesses its own routing weightsα\\alpha, a fact that is proven in Appendix[C](https://arxiv.org/html/2605.05676#A3)\.

## 4Experiments

In this section, we implement experiments to demonstrate the performance of our proposedBadit\. More experimental results, including intra\- and inter\-expert gradient angles, sensitivity analysis, and detailed performance across different tasks, are provided in Appendix[H](https://arxiv.org/html/2605.05676#A8)\.

Table 1:Experimental results ofBaditunder two implementation setting: mixed training and sequential training, respectively\. We repeat experiments with 5 different seeds, and report their average scores and standard deviations\.MethodMixed TrainingSequential TrainingRouge↑\\uparrowForward↑\\uparrowRouge↑\\uparrowForget Rate↓\\downarrowForward↑\\uparrowBackward↑\\uparrowQwen3\-8BLoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)54\.22±\\pm0\.72\-1\.66±\\pm0\.7347\.08±\\pm1\.959\.21±\\pm2\.24\-0\.11±\\pm0\.83\-8\.11±\\pm2\.19OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)54\.53±\\pm0\.46\-0\.45±\\pm0\.4648\.57±\\pm3\.158\.73±\\pm3\.18\-1\.15±\\pm0\.39\-6\.78±\\pm2\.95LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)54\.64±\\pm0\.71\-0\.57±\\pm0\.7148\.07±\\pm0\.798\.47±\\pm0\.59\-1\.26±\\pm0\.44\-6\.23±\\pm0\.55PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)53\.57±\\pm0\.35\-2\.30±\\pm0\.3547\.69±\\pm2\.527\.50±\\pm2\.80\-0\.06±\\pm0\.79\-6\.83±\\pm2\.32GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)54\.33±\\pm0\.73\-1\.78±\\pm0\.7348\.44±\\pm1\.028\.96±\\pm0\.95\-0\.73±\\pm1\.53\-6\.42±\\pm0\.71⋆\\boldsymbol\{\\star\}Badit\(ours\)55\.87±\\pm0\.350\.92±\\pm0\.3550\.86±\\pm1\.426\.95±\\pm1\.570\.44±\\pm0\.53\-5\.43±\\pm1\.47Qwen3\-4BLoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)53\.31±\\pm0\.92\-1\.24±\\pm0\.9248\.54±\\pm1\.638\.53±\\pm1\.58\-1\.31±\\pm0\.37\-6\.68±\\pm1\.34OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)53\.50±\\pm0\.760\.15±\\pm0\.7647\.83±\\pm1\.548\.78±\\pm2\.90\-1\.77±\\pm0\.61\-6\.47±\\pm2\.66LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)53\.63±\\pm0\.25\-0\.68±\\pm0\.2548\.58±\\pm1\.798\.50±\\pm1\.59\-0\.74±\\pm0\.44\-6\.25±\\pm1\.55PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)52\.87±\\pm0\.70\-0\.45±\\pm0\.7048\.26±\\pm1\.798\.05±\\pm1\.44\-0\.64±\\pm0\.67\-6\.12±\\pm1\.64GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)53\.24±\\pm0\.72\-1\.33±\\pm0\.7248\.07±\\pm1\.708\.47±\\pm1\.50\-1\.26±\\pm0\.44\-6\.23±\\pm1\.75⋆\\boldsymbol\{\\star\}Badit\(ours\)55\.22±\\pm0\.420\.91±\\pm0\.4250\.74±\\pm2\.275\.95±\\pm2\.120\.62±\\pm0\.70\-4\.21±\\pm2\.27Llama3\-8BLoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)52\.58±\\pm0\.79\-1\.02±\\pm0\.7944\.88±\\pm3\.6112\.41±\\pm3\.49\-1\.90±\\pm1\.48\-10\.23±\\pm3\.63OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)52\.89±\\pm0\.89\-0\.41±\\pm0\.8945\.25±\\pm2\.8411\.45±\\pm3\.040\.00±\\pm0\.77\-10\.01±\\pm2\.88LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)52\.95±\\pm0\.47\-1\.96±\\pm0\.4745\.47±\\pm3\.6811\.78±\\pm3\.250\.27±\\pm0\.93\-9\.71±\\pm3\.04PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)52\.88±\\pm0\.97\-0\.72±\\pm0\.9745\.23±\\pm2\.8112\.54±\\pm3\.07\-0\.63±\\pm0\.61\-10\.67±\\pm3\.04GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)52\.71±\\pm1\.07\-2\.13±\\pm1\.0745\.04±\\pm2\.7812\.70±\\pm2\.63\-1\.17±\\pm0\.85\-10\.94±\\pm3\.07⋆\\boldsymbol\{\\star\}Badit\(ours\)54\.75±\\pm0\.690\.79±\\pm0\.6948\.83±\\pm1\.898\.57±\\pm1\.751\.71±\\pm1\.53\-3\.49±\\pm2\.67Llama3\-3BLoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)50\.07±\\pm0\.97\-0\.23±\\pm0\.9645\.14±\\pm3\.059\.79±\\pm2\.17\-0\.17±\\pm0\.90\-4\.91±\\pm2\.91OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)49\.83±\\pm0\.70\-0\.47±\\pm0\.6945\.59±\\pm3\.1010\.07±\\pm3\.01\-0\.74±\\pm0\.91\-5\.95±\\pm2\.52LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)49\.72±\\pm0\.52\-0\.59±\\pm0\.5146\.20±\\pm2\.908\.58±\\pm2\.82\-0\.59±\\pm1\.38\-3\.51±\\pm2\.63PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)49\.64±\\pm0\.60\-0\.27±\\pm0\.5945\.49±\\pm2\.959\.13±\\pm2\.34\-0\.93±\\pm1\.00\-5\.27±\\pm2\.39GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)49\.60±\\pm0\.60\-0\.23±\\pm0\.6045\.34±\\pm3\.149\.67±\\pm2\.63\-0\.78±\\pm1\.23\-5\.66±\\pm3\.24⋆\\boldsymbol\{\\star\}Badit\(ours\)51\.45±\\pm0\.720\.61±\\pm0\.7248\.13±\\pm2\.266\.25±\\pm2\.450\.71±\\pm0\.85\-2\.57±\\pm2\.86Gemma2\-9BLoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)55\.16±\\pm0\.49\-1\.35±\\pm0\.4946\.80±\\pm2\.0110\.46±\\pm2\.05\-0\.32±\\pm0\.79\-9\.38±\\pm2\.10OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)54\.78±\\pm0\.81\-1\.05±\\pm0\.8146\.05±\\pm1\.5011\.20±\\pm1\.64\-0\.69±\\pm0\.69\-13\.57±\\pm1\.28LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)54\.78±\\pm0\.64\-0\.91±\\pm0\.6246\.98±\\pm2\.8911\.99±\\pm2\.990\.17±\\pm0\.45\-10\.11±\\pm2\.76PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)54\.20±\\pm1\.07\-1\.78±\\pm1\.0744\.93±\\pm3\.0912\.27±\\pm3\.26\-2\.24±\\pm0\.27\-14\.80±\\pm3\.10GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)54\.29±\\pm0\.45\-1\.88±\\pm0\.4546\.51±\\pm2\.2212\.62±\\pm2\.77\-0\.62±\\pm0\.48\-10\.80±\\pm2\.67⋆\\boldsymbol\{\\star\}Badit\(ours\)57\.35±\\pm0\.670\.54±\\pm0\.6748\.34±\\pm2\.1110\.31±\\pm2\.220\.39±\\pm0\.80\-8\.45±\\pm1\.46Gemma2\-2BLoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)51\.72±\\pm0\.96\-1\.30±\\pm0\.9643\.16±\\pm3\.0613\.21±\\pm3\.42\-0\.03±\\pm1\.77\-11\.28±\\pm2\.94OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)51\.82±\\pm0\.71\-0\.87±\\pm0\.7144\.93±\\pm3\.1210\.94±\\pm3\.36\-1\.03±\\pm0\.31\-10\.55±\\pm2\.61LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)51\.36±\\pm0\.36\-1\.05±\\pm0\.3743\.88±\\pm2\.6811\.66±\\pm3\.12\-0\.86±\\pm0\.17\-9\.95±\\pm2\.69PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)50\.96±\\pm0\.44\-1\.28±\\pm0\.4442\.57±\\pm2\.9012\.74±\\pm3\.59\-1\.65±\\pm0\.62\-10\.57±\\pm3\.37GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)51\.26±\\pm0\.76\-1\.70±\\pm0\.7644\.16±\\pm2\.3611\.81±\\pm2\.38\-0\.92±\\pm0\.57\-10\.27±\\pm2\.63⋆\\boldsymbol\{\\star\}Badit\(ours\)53\.60±\\pm0\.360\.70±\\pm0\.3646\.76±\\pm2\.169\.30±\\pm2\.980\.25±\\pm0\.49\-7\.65±\\pm2\.86### 4\.1Experimental Settings

Datasets and evaluation\.We evaluate ourBaditmethod onSuperNI\(Wanget al\.,[2022b](https://arxiv.org/html/2605.05676#bib.bib67)\), a multi\-task instruct\-tuning dataset comprising 15 distinct natural language tasks, e\.g\., sentiment analysis, each accompanied by its own training and evaluation sets\. Detailed statistics and descriptions of the dataset are provided in Appendix[G\.1](https://arxiv.org/html/2605.05676#A7.SS1)\. During training, we assess our approach under two multi\-task training paradigms:mixed trainingandsequential training\. In mixed training, we mix the training sets of all 15 tasks and train them simultaneously\. In sequential training, we train the tasks sequentially according to a prescribed order, and we report the average performance across five different task orders to mitigate the influence of ordering\.

Foundation LLMs\.We validate our method on 6 instruct\-tuned LLMs spanning different LLM families and sizes, including Qwen3\-8B, Qwen3\-4B\(Yanget al\.,[2025a](https://arxiv.org/html/2605.05676#bib.bib77)\), Llama3\-8B, Llama3\-3B\(AI@Meta,[2024](https://arxiv.org/html/2605.05676#bib.bib3)\), Gemma2\-9B, and Gemma2\-2B\(Rivièreet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib81)\)\. Their detailed model cards are provided in Appendix[G\.2](https://arxiv.org/html/2605.05676#A7.SS2)\.

Baselines\.We compare our approach against two LoRA series methods: LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)and PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\), and three MoE\-based methods specifically designed for multi\-task instruct\-tuning: OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\), LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\), and GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)\. Detailed descriptions of these methods are provided in Appendix[G\.3](https://arxiv.org/html/2605.05676#A7.SS3)\.

In addition to the experimental setup described above, implementation details of our method and the precise formulations of the evaluation metrics we use can be found in Appendices[G\.4](https://arxiv.org/html/2605.05676#A7.SS4)and[G\.5](https://arxiv.org/html/2605.05676#A7.SS5), respectively\.

### 4\.2Main Results

Our main experimental results are presented in Table[1](https://arxiv.org/html/2605.05676#S4.T1), where each experiment is repeated 5 times with different random seeds, and we report the average performance and its corresponding standard deviations\. Notably, under the sequential training setting, each random seed also determines a different task order; consequently, the reported standard deviations in this setting are generally larger due to the added sensitivity to task sequencing\. Among the evaluation metrics we employ,Rougedenotes the average performance across all tasks\. Forward measure the performance gain achieved by multi\-task training compared to training each task individually\. Forget Rate and Backward quantify the impact on previously learned tasks when training on new tasks for the sequential training setting\. The formal definitions of these metrics are provided in Appendix[G\.5](https://arxiv.org/html/2605.05676#A7.SS5)\.

Generally, our method consistently outperforms existing multi\-task instruct\-tuning approaches across all foundation LLMs and evaluation metrics\. For example,Baditachieves an average improvement of approximately 2\.68 points over the state\-of\-the\-art method, GainLoRA, on theRougemetric\. This not only demonstrates that our approach effectively enhances the multi\-task performance of LLMs but also shows its ability to substantially mitigate interference among tasks and alleviate catastrophic forgetting in the sequential training scenario\.

### 4\.3Ablation Study

We conduct an ablation study in Table[2](https://arxiv.org/html/2605.05676#S4.T2)to investigate the impact of removing two key components: BAD and DOG\. Specifically,w/o BADmeans we start from a LoRAMoE initialized with zero vectors and enforce orthogonality among rank\-1 LoRA components only during training, without the SVD initialization\.w/o DOGmeans we apply SVD\-based decoupled initialization but do not enforce orthogonality among experts during training\.

The experiments are carried out on Qwen3\-8B, Llama3\-8B, and Gemma\-9B, and we reportRougescores under both mixed and sequential training settings\. Generally, removing either component consistently degrades LLM performance, underscoring the critical role both modules play in enhancing multi\-task capability and mitigating interference among tasks\. Moreover, we observe that the DOG module generally contributes more substantially: while SVD\-based initialization alone ensures that distinct LoRA components remain orthogonal, thus represent diverse basic abilities, only during the early stages of training, this orthogonality gradually erodes as training progresses\. Consequently, dynamically preserving orthogonality throughout training \(as enabled by DOG\) proves essential for sustaining performance gains and robustness in two multi\-task settings\.

### 4\.4Intra\- and Inter\-Expert Gradient Angles

To investigate whether our method can dynamically promote orthogonality among expert gradients through clustering, we visualize in Fig\.[4](https://arxiv.org/html/2605.05676#S4.F4)the evolution of two angular metrics across training epochs: \(1\) the inter\-expert gradient angle, i\.e\., the angle between gradients of different experts, and \(2\) the intra\-expert gradient angle, i\.e\., the angle between gradients of different rank\-1 components within the same expert\. The results reveal that, in contrast to LoRAMoE, which lacks dynamic regrouping, our method consistently maintains inter\-expert gradient angles near 90° throughout training, indicating sustained orthogonality between experts\. Meanwhile, the intra\-expert gradient angles in our approach remain around 60°, approximately 20° lower than those observed in LoRAMoE\. Therefore, these findings demonstrate that our method effectively preserves orthogonality among expert gradients across the entire training process, thereby enabling a more precise and disentangled representation of basic abilities\. Additional results across more LLMs are provided in Appendix[H\.2](https://arxiv.org/html/2605.05676#A8.SS2)\.

### 4\.5Computation Budgets

In this section, we analyze the computational overhead of our proposed two\-stage method\. A formal𝒪\\mathcal\{O\}time complexity analysis is provided in Appendix[E](https://arxiv.org/html/2605.05676#A5)\. Specifically, Table[3](https://arxiv.org/html/2605.05676#S4.T3)reports, across 6 LLMs, the relative increase in computational time cost of our method compared to LoRAMoE, under identical settings of expert count and LoRA rank\. Overall, our method incurs approximately 1\.22×\\timesthe training time of LoRAMoE, with the majority of this additional cost attributed to the DOG stage\. This observation leads to two key insights: First, despite involving an SVD\-based decoupling step, our BAD stage does not increase, and may even reduce, computational time due to faster convergence during training\. Second, the DOG stage introduces extra overhead because it performs sphericalKK\-means clustering and integer optimization on parameter gradients, which are computationally intensive operations primarily executed on CPUs\. Although our approach entails a modest increase in training time, the resulting substantial gains in model performance render this trade\-off well justified and acceptable\.

Table 2:Ablation study of the BAD and DOG methods inBadit\. ST R\., ST F\., and MT R\. indicate theRougeand forget rate under the sequential training, and theRougeunder the mixed training, respectively\. Three LLMs use their 8B and 9B versions\.MethodST R\.ST F\.MT R\.𝚫\\boldsymbol\{\\Delta\}Qwen3Badit50\.866\.9555\.87\-w/o BAD50\.077\.3455\.022\.03w/o DOG49\.708\.2054\.853\.43w/o BAD & DOG48\.078\.4754\.645\.54Llama3Badit48\.838\.5754\.75\-w/o BAD47\.929\.3054\.022\.37w/o DOG47\.339\.7453\.743\.68w/o BAD & DOG45\.4711\.7852\.958\.37Gemma2Badit48\.3410\.3157\.35\-w/o BAD47\.7810\.5756\.711\.46w/o DOG47\.6011\.0756\.032\.82w/o BAD & DOG46\.9811\.9954\.785\.61![Refer to caption](https://arxiv.org/html/2605.05676v1/x4.png)Figure 4:Intra\- and inter\-expert gradient angles across epochs\.Table 3:Time cost of different methods across 6 LLMs under the sequential training setting\.MethodLoRALoRAMoEPiSSABadit\- DOG\- BADQwen3\-8B0\.92×\\times1×\\times1\.18×\\times1\.25×\\times1\.14×\\times1\.24×\\timesQwen3\-4B0\.75×\\times1×\\times0\.86×\\times1\.18×\\times0\.96×\\times1\.20×\\timesLlama3\-8B0\.68×\\times1×\\times0\.84×\\times1\.16×\\times0\.95×\\times1\.15×\\timesLlama3\-3B0\.71×\\times1×\\times0\.61×\\times1\.27×\\times0\.96×\\times1\.27×\\timesGemma2\-9B0\.74×\\times1×\\times1\.08×\\times1\.25×\\times1\.02×\\times1\.24×\\timesGemma2\-2B0\.79×\\times1×\\times0\.69×\\times1\.24×\\times1\.01×\\times1\.24×\\timesAverage0\.76×\\times1×\\times0\.87×\\times1\.22×\\times1\.00×\\times1\.22×\\times

## 5Related Work

Cross\-task interference is a critical challenge in the multi\-task instruct\-tuning of LLMs\(Luoet al\.,[2023](https://arxiv.org/html/2605.05676#bib.bib14); Janget al\.,[2023](https://arxiv.org/html/2605.05676#bib.bib37)\)\. Generally, existing approaches can be broadly classified into three categories\(Wanget al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib20); Zhouet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib25)\):rehearsal\-basedmethods\(Huanget al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib32); Wanget al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib34)\)store and replay historical data samples;optimization\-basedmethods\(Liet al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib33); Jianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib36)\)introduce training regularizations, e\.g\., orthogonal constraints\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\), to guide the optimization direction; andarchitecture\-basedmethods\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1); Zhaoet al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib24); Yanget al\.,[2025b](https://arxiv.org/html/2605.05676#bib.bib83)\)employ parameter\-efficient modules designed to isolate task\-specific knowledge and mitigate interference across tasks\.

In this section, we focus primarily on architecture\-based methods, and more comprehensive literature reviews can be seen in Appendix[F](https://arxiv.org/html/2605.05676#A6)\. Specifically, the basic idea of architecture\-based methods is to mitigate cross\-task interference by explicitly assigning task\-specific, independent trainable parameters\. For example,Wanget al\.\([2023b](https://arxiv.org/html/2605.05676#bib.bib16)\)allocate a small set of private parameters, e\.g\., soft prompts\(Li and Liang,[2021](https://arxiv.org/html/2605.05676#bib.bib39)\), and train them alongside a shared pre\-trained model\. Similarly, SAPT\(Zhaoet al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib38)\)addresses knowledge transfer among private parameters by introducing a shared attention module\. More recently, several works have alleviated interference by combining LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)with MoE architectures\(Shazeeret al\.,[2017](https://arxiv.org/html/2605.05676#bib.bib13)\)\. These methods design a LoRA\-based expert for each task within an MoE framework and employ a routing mechanism to automatically select one or more appropriate experts for incoming tasks\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1); Maet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib27); Fenget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib26); Liet al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib2); Liuet al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib29); Chenet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib47)\)\. Our approach builds upon this paradigm by dynamically adapting the expert models so that each expert encapsulates a distinct and orthogonal basic ability\.

## 6Conclusion

In this work, we identify a fundamental limitation in existing multi\-task instruct\-tuning methods: despite efforts to isolate task\-specific parameters, substantial parameter sharing persists across tasks, leading to unresolved cross\-task interference\. Through empirical analysis, we reveal that LLMs consistently co\-activate groups of parameters, suggesting that LLMs encode a small set of orthogonal basic abilities, each reusable across multiple tasks\. Motivated by this insight, we proposeBadit, which decomposes LLM parameters into high\-singular\-value LoRA experts representing these basic abilities\. By dynamically enforcing orthogonality among these experts via spherical clustering of their rank\-1 components during training,Baditeffectively mitigates gradient conflict while enabling flexible task representation as linear combinations of shared abilities\. Extensive experiments on theSuperNIbenchmark across six LLMs demonstrate thatBaditoutperforms state\-of\-the\-art methods by an average of 2\.68Rougepoints and exhibits significantly reduced cross\-task interference\.

## Acknowledgements

This work was supported in part by the National Natural Science Foundation of China \(No\.62276113\) and JST ASPIRE Grant Number JPMJAP2405\.

## Impact Statement

We propose a novel approach for multi\-task instruct\-tuning of LLMs, motivated by the observation that prior methods still rely on shared parameters, which can lead to conflicting gradients across tasks\. To address this, we decompose the basic abilities inherent in the LLM itself to represent any task\. We validate our method using publicly available datasets spanning 15 NLP tasks, e\.g\., sentiment analysis, none of which contain harmful, private, or sensitive content\. Furthermore, our experiments are conducted on open\-source instruct\-tuned LLMs, thereby avoiding any copyright risks\.

## References

- AI@Meta \(2024\)Llama 3 model card\.External Links:[Link](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)Cited by:[§G\.2](https://arxiv.org/html/2605.05676#A7.SS2.p1.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p2.1)\.
- A\. Bian, W\. Li, H\. Yuan, C\. Yu, M\. Wang, Z\. Zhao, A\. Lu, P\. Ji, and T\. Feng \(2024\)Make continual learning stronger via c\-flat\.InAdvances in Neural Information Processing Systems,Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1)\.
- C\. Chen, L\. Gao, P\. Zeng, M\. Cao, and H\. T\. Shen \(2026\)Towards redundancy\-free sub\-networks in continual learning\.Pattern Recognition171,pp\. 112020\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1)\.
- Z\. Chen, Z\. Wang, Z\. Wang, H\. Liu, Z\. Yin, S\. Liu, L\. Sheng, W\. Ouyang, and J\. Shao \(2024\)Octavius: mitigating task interference in mllms via lora\-moe\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. Wei \(2022\)Knowledge neurons in pretrained transformers\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 8493–8502\.Cited by:[§H\.1](https://arxiv.org/html/2605.05676#A8.SS1.p3.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p2.10)\.
- C\. de Masson d’Autume, S\. Ruder, L\. Kong, and D\. Yogatama \(2019\)Episodic memory in lifelong language learning\.InAdvances in Neural Information Processing Systems,pp\. 13122–13131\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized llms\.InAdvances in Neural Information Processing Systems,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- S\. Dou, E\. Zhou, Y\. Liu, S\. Gao, W\. Shen, L\. Xiong, Y\. Zhou, X\. Wang, Z\. Xi, X\. Fan, S\. Pu, J\. Zhu, R\. Zheng, T\. Gui, Q\. Zhang, and X\. Huang \(2024\)LoRAMoE: alleviating world knowledge forgetting in large language models via moe\-style plugin\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 1932–1945\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1),[4th item](https://arxiv.org/html/2605.05676#A7.I2.i4.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2605.05676#S3.SS3.p1.9),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.135.135.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.172.172.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.209.209.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.24.24.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.61.61.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.98.98.8.1.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- C\. Fan, Z\. Lu, S\. Liu, C\. Gu, X\. Qu, W\. Wei, and Y\. Cheng \(2025\)Make lora great again: boosting lora with adaptive singular values and mixture\-of\-experts optimization alignment\.InInternational Conference on Machine Learning,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- W\. Feng, C\. Hao, Y\. Zhang, Y\. Han, and H\. Wang \(2024\)Mixture\-of\-loras: an efficient multitask tuning method for large language models\.InInternational Conference on Computational Linguistics,pp\. 11371–11380\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- Z\. Gao, P\. Liu, W\. X\. Zhao, Z\. Lu, and J\. Wen \(2022\)Parameter\-efficient mixture\-of\-experts architecture for pre\-trained language models\.InInternational Conference on Computational Linguistics,pp\. 3263–3273\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.InConference on Empirical Methods in Natural Language Processing,pp\. 5484–5495\.Cited by:[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p2.10)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2015\)Delving deep into rectifiers: surpassing human\-level performance on imagenet classification\.InIEEE International Conference on Computer Vision,pp\. 1026–1034\.Cited by:[§3\.1](https://arxiv.org/html/2605.05676#S3.SS1.p2.7)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1),[1st item](https://arxiv.org/html/2605.05676#A7.I2.i1.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p4.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.12.12.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.123.123.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.160.160.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.197.197.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.49.49.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.86.86.8.1.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- N\. Hu, J\. Chen, Y\. Wu, G\. Qi, H\. Wang, S\. Bi, Y\. Chen, T\. Wu, and J\. Z\. Pan \(2025\)Can llms evaluate complex attribution in qa? automatic benchmarking using knowledge graphs\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 17096–17118\.Cited by:[§1](https://arxiv.org/html/2605.05676#S1.p1.1)\.
- J\. Huang, L\. Cui, A\. Wang, C\. Yang, X\. Liao, L\. Song, J\. Yao, and J\. Su \(2024a\)Mitigating catastrophic forgetting in large language models with self\-synthesized rehearsal\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 1416–1428\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- Q\. Huang, Z\. An, N\. Zhuang, M\. Tao, C\. Zhang, Y\. Jin, K\. Xu, L\. Chen, S\. Huang, and Y\. Feng \(2024b\)Harder task needs more experts: dynamic routing in moe models\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 12883–12895\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- X\. Huang, Rishabh, G\. Franke, Z\. Yang, J\. Bai, W\. Bai, J\. Bi, Z\. Ding, Y\. Duan, C\. Fan,et al\.\(2025\)Loong: synthesize long chain\-of\-thoughts at scale through verifiers\.CoRRabs/2509\.03059\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- J\. Jang, S\. Kim, S\. Ye, D\. Kim, L\. Logeswaran, M\. Lee, K\. Lee, and M\. Seo \(2023\)Exploring the benefits of training expert language models over instruction tuning\.InInternational Conference on Machine Learning,Vol\.202,pp\. 14702–14729\.Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. de Las Casas, E\. B\. Hanna, F\. Bressand,et al\.\(2024\)Mixtral of experts\.CoRRabs/2401\.04088\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- G\. Jiang, C\. Jiang, Z\. Li, S\. Xue, J\. Zhou, L\. Song, D\. Lian, and Y\. Wei \(2025\)Unlocking the power of function vectors for characterizing and mitigating catastrophic forgetting in continual instruction tuning\.InInternational Conference on Learning Representations,Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- Z\. Kou, H\. Xuan, J\. Zhu, H\. Wang, M\. Xie, C\. Wang, J\. Wang, Y\. Jia, and X\. Geng \(2026\)Tail\-aware reconstruction of incomplete label distributions with low\-rank and sparse modeling\.IEEE Transactions on Circuits and Systems for Video Technology36\(2\),pp\. 1571–1586\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- Y\. Leng and D\. Xiong \(2025\)Towards understanding multi\-task learning \(generalization\) of llms via detecting and exploring task\-specific neurons\.InInternational Conference on Computational Linguistics,pp\. 2969–2987\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p5.1),[§1](https://arxiv.org/html/2605.05676#S1.p2.1)\.
- D\. Li, Y\. Ma, N\. Wang, Z\. Cheng, L\. Duan, J\. Zuo, C\. Yang, and M\. Tang \(2024a\)MixLoRA: enhancing large language models fine\-tuning with lora based mixture of experts\.CoRRabs/2404\.15159\.Cited by:[§3\.3](https://arxiv.org/html/2605.05676#S3.SS3.p1.9),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- D\. Li, Z\. Sun, X\. Hu, B\. Hu, and M\. Zhang \(2025\)CMT: A memory compression method for continual knowledge learning of large language models\.InAAAI Conference on Artificial Intelligence,pp\. 24413–24421\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1)\.
- H\. Li, L\. Ding, M\. Fang, and D\. Tao \(2024b\)Revisiting catastrophic forgetting in large language model tuning\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 4297–4308\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- X\. L\. Li and P\. Liang \(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 4582–4597\.Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- Y\. Liang, J\. Chen, and W\. Li \(2025\)Gated integration of low\-rank adaptation for continual learning of large language models\.InAdvances in Neural Information Processing Systems,Cited by:[5th item](https://arxiv.org/html/2605.05676#A7.I2.i5.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.110.110.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.147.147.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.184.184.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.221.221.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.36.36.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.73.73.8.1.1)\.
- S\. Lin, L\. Yang, D\. Fan, and J\. Zhang \(2022a\)Beyond not\-forgetting: continual learning with backward knowledge transfer\.InAdvances in Neural Information Processing Systems,Cited by:[§3\.2](https://arxiv.org/html/2605.05676#S3.SS2.p3.1)\.
- S\. Lin, L\. Yang, D\. Fan, and J\. Zhang \(2022b\)TRGP: trust region gradient projection for continual learning\.InInternational Conference on Learning Representations,Cited by:[§3\.2](https://arxiv.org/html/2605.05676#S3.SS2.p3.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024a\)DeepSeek\-v3 technical report\.CoRRabs/2412\.19437\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- K\. Liu, S\. Yan, R\. Miao, B\. Wang, C\. Shen, J\. Zhang, and J\. Ye \(2026\)Where did this sentence come from? tracing provenance in LLM reasoning distillation\.InInternational Conference on Learning Representations,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- Q\. Liu, X\. Wu, X\. Zhao, Y\. Zhu, D\. Xu, F\. Tian, and Y\. Zheng \(2024b\)When MOE meets llms: parameter efficient fine\-tuning for multi\-task medical applications\.InInternational ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 1104–1114\.Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024c\)DoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- X\. Liu, C\. Wu, M\. Menta, L\. Herranz, B\. Raducanu, A\. D\. Bagdanov, S\. Jui, and J\. van de Weijer \(2020\)Generative feature replay for class\-incremental learning\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 915–924\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1)\.
- D\. Lopez\-Paz and M\. Ranzato \(2017\)Gradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems,pp\. 6467–6476\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1),[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1),[§G\.5](https://arxiv.org/html/2605.05676#A7.SS5.p1.1)\.
- Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang \(2023\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.CoRRabs/2308\.08747\.Cited by:[§1](https://arxiv.org/html/2605.05676#S1.p2.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- J\. Ma, Z\. Zhao, X\. Yi, J\. Chen, L\. Hong, and E\. H\. Chi \(2018\)Modeling task relationships in multi\-task learning with multi\-gate mixture\-of\-experts\.InACM SIGKDD International Conference on Knowledge Discovery & Data Mining,pp\. 1930–1939\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- Y\. Ma, Z\. Liang, H\. Dai, B\. Chen, D\. Gao, Z\. Ran, Z\. Wang, L\. Jin, W\. Jiang, G\. Zhang, X\. Cai, and L\. Yang \(2024\)MoDULA: mixture of domain\-specific and universal lora for multi\-task learning\.InConference on Empirical Methods in Natural Language Processing,pp\. 2758–2770\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- M\. McCloskey and N\. J\. Cohen \(1989\)Catastrophic interference in connectionist networks: the sequential learning problem\.Psychology of Learning and Motivation\-Advances in Research and Theory24\(C\),pp\. 109–165\.Cited by:[§1](https://arxiv.org/html/2605.05676#S1.p2.1)\.
- F\. Meng, Z\. Wang, and M\. Zhang \(2024\)PiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1),[2nd item](https://arxiv.org/html/2605.05676#A7.I2.i2.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p4.1),[§3\.1](https://arxiv.org/html/2605.05676#S3.SS1.p3.7),[§3\.3](https://arxiv.org/html/2605.05676#S3.SS3.p1.3),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.104.104.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.141.141.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.178.178.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.215.215.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.30.30.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.67.67.8.1.1)\.
- M\. Rivière, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé, J\. Ferret,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.CoRRabs/2408\.00118\.Cited by:[§G\.2](https://arxiv.org/html/2605.05676#A7.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p2.1)\.
- G\. Saha, I\. Garg, and K\. Roy \(2021\)Gradient projection memory for continual learning\.InInternational Conference on Learning Representations,Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. V\. Le, G\. E\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- H\. Shi, Z\. Xu, H\. Wang, W\. Qin, W\. Wang, Y\. Wang, Z\. Wang, S\. Ebrahimi, and H\. Wang \(2026\)Continual learning of large language models: A comprehensive survey\.ACM Computing Surveys58\(5\),pp\. 120:1–120:42\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p1.1)\.
- H\. Shin, J\. K\. Lee, J\. Kim, and J\. Kim \(2017\)Continual learning with deep generative replay\.InAdvances in Neural Information Processing Systems,pp\. 2990–2999\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1)\.
- R\. Song, S\. He, S\. Jiang, Y\. Xian, S\. Gao, K\. Liu, and Z\. Yu \(2024\)Does large language model contain task\-specific neurons?\.InConference on Empirical Methods in Natural Language Processing,pp\. 7101–7113\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p5.1),[§H\.1](https://arxiv.org/html/2605.05676#A8.SS1.p3.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p2.10),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p2.9)\.
- F\. Sun, C\. Ho, and H\. Lee \(2020\)LAMOL: language modeling for lifelong language learning\.InInternational Conference on Learning Representations,Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1)\.
- S\. Tang, D\. Chen, J\. Zhu, S\. Yu, and W\. Ouyang \(2021\)Layerwise optimization by gradient decomposition for continual learning\.InIEEE Conference on Computer Vision and Pattern Recognition,pp\. 9634–9643\.Cited by:[§1](https://arxiv.org/html/2605.05676#S1.p2.1)\.
- X\. Tang, Z\. Lv, X\. Cheng, J\. Li, W\. X\. Zhao, Z\. Wen, Z\. Zhang, and J\. Zhou \(2025\)Enhancing cross\-task transfer of large language models via activation steering\.CoRRabs/2507\.13236\.Cited by:[§1](https://arxiv.org/html/2605.05676#S1.p2.1)\.
- C\. Wang, Y\. Lyu, Z\. Sun, and L\. Jing \(2025\)Continual gradient low\-rank projection fine\-tuning for llms\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 14815–14829\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1)\.
- L\. Wang, B\. Lei, Q\. Li, H\. Su, J\. Zhu, and Y\. Zhong \(2022a\)Triple\-memory networks: A brain\-inspired method for continual learning\.IEEE Transactions on Neural Networks and Learning Systems33\(5\),pp\. 1925–1934\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1)\.
- L\. Wang, X\. Zhang, H\. Su, and J\. Zhu \(2024a\)A comprehensive survey of continual learning: theory, method and application\.IEEE Transactions on Pattern Analysis and Machine Intelligence46\(8\),pp\. 5362–5383\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p1.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. Huang \(2023a\)Orthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 10658–10671\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1),[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1),[3rd item](https://arxiv.org/html/2605.05676#A7.I2.i3.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p3.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.129.129.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.166.166.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.18.18.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.203.203.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.55.55.8.1.1),[Table 1](https://arxiv.org/html/2605.05676#S4.T1.92.92.8.1.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- Y\. Wang, Y\. Liu, C\. Shi, H\. Li, C\. Chen, H\. Lu, and Y\. Yang \(2024b\)InsCL: A data\-efficient continual learning paradigm for fine\-tuning large language models with instructions\.InConference of the North American Chapter of the Association for Computational Linguistics,pp\. 663–677\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p2.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022b\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ NLP tasks\.InConference on Empirical Methods in Natural Language Processing,pp\. 5085–5109\.Cited by:[Table 4](https://arxiv.org/html/2605.05676#A6.T4),[Table 4](https://arxiv.org/html/2605.05676#A6.T4.3.2),[§G\.1](https://arxiv.org/html/2605.05676#A7.SS1.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p3.1),[§1](https://arxiv.org/html/2605.05676#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p1.1)\.
- Z\. Wang, Y\. Liu, T\. Ji, X\. Wang, Y\. Wu, C\. Jiang, Y\. Chao, Z\. Han, L\. Wang, X\. Shao, and W\. Zeng \(2023b\)Rehearsal\-free continual language learning via efficient parameter isolation\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 10933–10946\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1),[§1](https://arxiv.org/html/2605.05676#S1.p1.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- X\. Wu, W\. Yu, C\. Zhang, and P\. C\. Woodland \(2024\)An improved empirical fisher approximation for natural gradient descent\.InAdvances in Neural Information Processing Systems,Cited by:[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p2.9)\.
- M\. Xie, J\. Xiao, Z\. Kou, Z\. Li, G\. Niu, and M\. Sugiyama \(2026\)Are multimodal large language models good annotators for image tagging?\.CoRRabs/2602\.20972\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- S\. Yan, K\. Liu, C\. Shen, B\. Wang, S\. Fan, J\. Zhang, Y\. Wu, Z\. Wang, and J\. Ye \(2026\)Distribution\-aligned sequence distillation for superior long\-cot reasoning\.CoRRabs/2601\.09088\.Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.CoRRabs/2505\.09388\.Cited by:[§G\.2](https://arxiv.org/html/2605.05676#A7.SS2.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p1.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.05676#S4.SS1.p2.1)\.
- S\. Yang, K\. Ning, Y\. Liu, J\. Yao, Y\. Tian, Y\. Song, and L\. Yuan \(2025b\)Is parameter collision hindering continual learning in llms?\.InInternational Conference on Computational Linguistics,pp\. 4243–4259\.Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p4.1),[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- Y\. Yang, X\. Li, Z\. Zhou, S\. Song, J\. Wu, L\. Nie, and B\. Ghanem \(2024\)CorDA: context\-oriented decomposition adaptation of large language models for task\-aware parameter\-efficient fine\-tuning\.InAdvances in Neural Information Processing Systems,Cited by:[§F\.2](https://arxiv.org/html/2605.05676#A6.SS2.p1.1),[§3\.1](https://arxiv.org/html/2605.05676#S3.SS1.p3.7),[§3\.3](https://arxiv.org/html/2605.05676#S3.SS3.p1.3)\.
- X\. Yuan, C\. Shen, S\. Yan, kaiyuan liu, X\. Zhang, S\. Fan, L\. Xie, W\. Wang, R\. Guan, Y\. Wang, and J\. Ye \(2026\)Differential fine\-tuning large language models towards better diverse reasoning abilities\.InInternational Conference on Learning Representations,Cited by:[§F\.1](https://arxiv.org/html/2605.05676#A6.SS1.p3.1)\.
- L\. Zhao, X\. Zhang, K\. Yan, S\. Ding, and W\. Huang \(2024a\)SAFE: slow and fast parameter\-efficient tuning for continual learning with pre\-trained models\.InAdvances in Neural Information Processing Systems,Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.
- W\. Zhao, S\. Wang, Y\. Hu, Y\. Zhao, B\. Qin, X\. Zhang, Q\. Yang, D\. Xu, and W\. Che \(2024b\)SAPT: A shared attention framework for parameter\-efficient continual learning of large language models\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 11641–11661\.Cited by:[§G\.1](https://arxiv.org/html/2605.05676#A7.SS1.p1.1),[§G\.5](https://arxiv.org/html/2605.05676#A7.SS5.p1.1),[§1](https://arxiv.org/html/2605.05676#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.05676#S2.SS1.p1.1),[§5](https://arxiv.org/html/2605.05676#S5.p2.1)\.
- D\. Zhou, H\. Sun, J\. Ning, H\. Ye, and D\. Zhan \(2024\)Continual learning with pre\-trained models: A survey\.InInternational Joint Conference on Artificial Intelligence,pp\. 8363–8371\.Cited by:[§5](https://arxiv.org/html/2605.05676#S5.p1.1)\.

## Appendix AProof of the Orthogonality of SVD Initialization

In this section, we formally prove that the proposed SVD\-based initialization of BAD yields mutually orthogonal low\-rank subspaces across experts at initialization\.

###### Theorem A\.1\.

Let𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}be a pre\-trained LLM weight matrix, and let

𝐖=𝐔​𝚺​𝐕⊤=∑i=1min⁡\(m,n\)σi​𝐮i​𝐯i⊤\\mathbf\{W\}=\\mathbf\{U\}\\boldsymbol\{\\Sigma\}\\mathbf\{V\}^\{\\top\}=\\sum\_\{i=1\}^\{\\min\(m,n\)\}\\sigma\_\{i\}\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\}be its compact singular value decomposition, whereσ1≥σ2≥…≥σmin⁡\(m,n\)\\sigma\_\{1\}\\geq\\sigma\_\{2\}\\geq\\ldots\\geq\\sigma\_\{\\min\(m,n\)\}are singular values, and𝐮i∈ℝm\\mathbf\{u\}\_\{i\}\\in\\mathbb\{R\}^\{m\},𝐯i∈ℝn\\mathbf\{v\}\_\{i\}\\in\\mathbb\{R\}^\{n\}are the corresponding orthonormal left and right singular vectors\.

Then the rank\-11matrices𝐮i​𝐯i⊤\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\}form an orthogonal basis under the Frobenius inner product as

⟨𝐮i​𝐯i⊤,𝐮j​𝐯j⊤⟩F\\displaystyle\\left\\langle\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\},\\mathbf\{u\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\}\\right\\rangle\_\{F\}=Tr⁡\(\(𝐮i​𝐯i⊤\)⊤​𝐮j​𝐯j⊤\)\\displaystyle=\\operatorname\{Tr\}\\left\(\\left\(\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\}\\right\)^\{\\top\}\\mathbf\{u\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\}\\right\)\(15\)=\(𝐮i⊤​𝐮j\)​\(𝐯i⊤​𝐯j\)=δi​j,\\displaystyle=\(\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{u\}\_\{j\}\)\\,\(\\mathbf\{v\}\_\{i\}^\{\\top\}\\mathbf\{v\}\_\{j\}\)=\\delta\_\{ij\},whereδi​j=1\\delta\_\{ij\}=1ifi=ji=jand0otherwise\.

###### Proof\.

The orthonormality of𝐮i\\mathbf\{u\}\_\{i\}and𝐯i\\mathbf\{v\}\_\{i\}from the SVD ensures𝐮i⊤​𝐮j=δi​j\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{u\}\_\{j\}=\\delta\_\{ij\}and𝐯i⊤​𝐯j=δi​j\\mathbf\{v\}\_\{i\}^\{\\top\}\\mathbf\{v\}\_\{j\}=\\delta\_\{ij\}\. Substituting into the definition of Frobenius inner product directly yields Eq\. \([15](https://arxiv.org/html/2605.05676#A1.E15)\), proving that𝐮i​𝐯i⊤\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\}are orthogonal in the Frobenius sense\. ∎

###### Proposition A\.2\(Orthogonal LoRA Expert Initialization\)\.

LetR=r​K≪min⁡\(m,n\)R=rK\\ll\\min\(m,n\)be the total SVD rank used for initializingKKexperts, each of rankrr\. Partition the firstRRSVD components intoKKcontiguous, non\-overlapping blocks: thekk\-th expert \(k∈\{0,…,K−1\}k\\in\\\{0,\\ldots,K\-1\\\}\) receives indices fromr​krktor​\(k\+1\)−1r\(k\+1\)\-1\. Define

𝐀k=𝐔k​𝚺kα,𝐁k=𝚺kα​𝐕k⊤,\\mathbf\{A\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\sqrt\{\\frac\{\\boldsymbol\{\\Sigma\}\_\{k\}\}\{\\alpha\}\},\\quad\\mathbf\{B\}\_\{k\}=\\sqrt\{\\frac\{\\boldsymbol\{\\Sigma\}\_\{k\}\}\{\\alpha\}\}\\mathbf\{V\}\_\{k\}^\{\\top\},where𝐔k∈ℝm×r\\mathbf\{U\}\_\{k\}\\in\\mathbb\{R\}^\{m\\times r\}and𝐕k∈ℝn×r\\mathbf\{V\}\_\{k\}\\in\\mathbb\{R\}^\{n\\times r\}contain the assigned singular vectors and𝚺k=diag​\(σr​k,…,σr​\(k\+1\)−1\)\\boldsymbol\{\\Sigma\}\_\{k\}=\\mathrm\{diag\}\(\\sigma\_\{rk\},\\dots,\\sigma\_\{r\(k\+1\)\-1\}\)\. The LoRA scaling factor isα\\alpha\(scalings=α/rs=\\alpha/r\)\. The initialization update for expertkkis

Δ​𝐖k=s​𝐀k​𝐁k=𝐔k​𝚺k​𝐕k⊤=∑i=r​kr​\(k\+1\)−1σi​𝐮i​𝐯i⊤\.\\Delta\\mathbf\{W\}\_\{k\}=s\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\boldsymbol\{\\Sigma\}\_\{k\}\\mathbf\{V\}\_\{k\}^\{\\top\}=\\sum\_\{i=rk\}^\{r\(k\+1\)\-1\}\\sigma\_\{i\}\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\}\.

###### Proof\.

For two distinct expertsk≠lk\\neq l, their updates have disjoint index sets as

Δ​𝐖k=∑i∈ℐkσi​𝐮i​𝐯i⊤,Δ​𝐖l=∑j∈ℐlσj​𝐮j​𝐯j⊤,\\Delta\\mathbf\{W\}\_\{k\}=\\sum\_\{i\\in\\mathcal\{I\}\_\{k\}\}\\sigma\_\{i\}\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\},\\quad\\Delta\\mathbf\{W\}\_\{l\}=\\sum\_\{j\\in\\mathcal\{I\}\_\{l\}\}\\sigma\_\{j\}\\mathbf\{u\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\},whereℐk∩ℐl=∅\\mathcal\{I\}\_\{k\}\\cap\\mathcal\{I\}\_\{l\}=\\varnothing\. By Theorem[A\.1](https://arxiv.org/html/2605.05676#A1.Thmtheorem1), fori∈ℐki\\in\\mathcal\{I\}\_\{k\},j∈ℐlj\\in\\mathcal\{I\}\_\{l\}, we have⟨𝐮i​𝐯i⊤,𝐮j​𝐯j⊤⟩F=0\\langle\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\},\\mathbf\{u\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\}\\rangle\_\{F\}=0\. Therefore,

⟨Δ​𝐖k,Δ​𝐖l⟩F=∑i∈Ik∑j∈Ilσi​σj​⟨𝐮i​𝐯i⊤,𝐮j​𝐯j⊤⟩F=0\.\\langle\\Delta\\mathbf\{W\}\_\{k\},\\Delta\\mathbf\{W\}\_\{l\}\\rangle\_\{F\}=\\sum\_\{i\\in I\_\{k\}\}\\sum\_\{j\\in I\_\{l\}\}\\sigma\_\{i\}\\sigma\_\{j\}\\langle\\mathbf\{u\}\_\{i\}\\mathbf\{v\}\_\{i\}^\{\\top\},\\mathbf\{u\}\_\{j\}\\mathbf\{v\}\_\{j\}^\{\\top\}\\rangle\_\{F\}=0\.Thus, all initialized experts are mutually orthogonal under the Frobenius inner product\. ∎

## Appendix BDetailed Regrouping Objective

Given gradients\{𝐠^i\}i=1r​K\\\{\\mathbf\{\\widehat\{g\}\}\_\{i\}\\\}\_\{i=1\}^\{rK\}, the grouping aims to optimize the strategy𝚷∈\(0,1\)r​K×K\\boldsymbol\{\\Pi\}\\in\(0,1\)^\{rK\\times K\}, where eachπi,k=1/0\\pi\_\{i,k\}=1/0denotes whether the originalii\-th component is assigned / not assigned to the newkk\-th expert\. For these fixed gradients, we aim to maximize the sum of inner products within groups while simultaneously minimizing the sum of inner products between groups\. To this end, we formulate the following objective function:

max𝚷⁡ℒintra−λ​ℒinter,\\max\_\{\\boldsymbol\{\\Pi\}\}\\mathcal\{L\}\_\{\\text\{intra\}\}\-\\lambda\\mathcal\{L\}\_\{\\text\{inter\}\},\(16\)ℒintra\\displaystyle\\mathcal\{L\}\_\{\\text\{intra\}\}=∑k=1K∑i,j=1r​Kπi,k​πj,k​⟨𝐠^i,𝐠^j⟩,\\displaystyle=\\sum\_\{k=1\}^\{K\}\\sum\_\{i,j=1\}^\{rK\}\\pi\_\{i,k\}\\pi\_\{j,k\}\\left\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{\\widehat\{g\}\}\_\{j\}\\right\\rangle,ℒinter\\displaystyle\\mathcal\{L\}\_\{\\text\{inter\}\}=∑l≠k∑i,j=1r​Kπi,k​πj,l​⟨𝐠^i,𝐠^j⟩,\\displaystyle=\\sum\_\{l\\neq k\}\\sum\_\{i,j=1\}^\{rK\}\\pi\_\{i,k\}\\pi\_\{j,l\}\\left\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{\\widehat\{g\}\}\_\{j\}\\right\\rangle,whereλ\>0\\lambda\>0is a trade\-off hyper\-parameter\. The first term of the objective function encourages smaller angular differences \(i\.e\., higher cosine distance\) among gradients within each expert, while the second term promotes larger angular separation \(i\.e\., lower cosine distance\) between gradients of different experts\. Meanwhile, due to

∑i,j=1r​Kπi,k​πj,k​⟨𝐠^i,𝐠^j⟩=‖∑i=1r​Kπi,k​𝐠^i‖2\.\\sum\_\{i,j=1\}^\{rK\}\\pi\_\{i,k\}\\pi\_\{j,k\}\\left\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{\\widehat\{g\}\}\_\{j\}\\right\\rangle=\\left\\\|\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\}\.Therefore, the objective is equivalent to

ℒintra\\displaystyle\\mathcal\{L\}\_\{\\text\{intra\}\}=∑k=1K‖∑i=1r​Kπi,k​𝐠^i‖2,\\displaystyle=\\sum\_\{k=1\}^\{K\}\\left\\\|\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\},ℒinter\\displaystyle\\mathcal\{L\}\_\{\\text\{inter\}\}=‖∑i=1r​K𝐠^i‖2−∑k=1K‖∑i=1r​Kπi,k​𝐠^i‖2\.\\displaystyle=\\left\\\|\\sum\_\{i=1\}^\{rK\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\}\-\\sum\_\{k=1\}^\{K\}\\left\\\|\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\}\.
Moreover, regardless of how the gradients are grouped, the total inner product across all pairs of gradients remains constant as

∑i,j=1r​Kπi,k​πj,k​⟨𝐠^i,𝐠^j⟩=‖∑i=1r​K𝐠^i‖2=ℒintra\+ℒinter=C\.\\sum\_\{i,j=1\}^\{rK\}\\pi\_\{i,k\}\\pi\_\{j,k\}\\left\\langle\\mathbf\{\\widehat\{g\}\}\_\{i\},\\mathbf\{\\widehat\{g\}\}\_\{j\}\\right\\rangle=\\left\\\|\\sum\_\{i=1\}^\{rK\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\}=\\mathcal\{L\}\_\{\\text\{intra\}\}\+\\mathcal\{L\}\_\{\\text\{inter\}\}=C\.Accordingly, the objective in Eq\. \([16](https://arxiv.org/html/2605.05676#A2.E16)\) can be formulated as:

max𝚷⁡ℒintra−λ​\(C−ℒintra\)=\(1\+λ\)​ℒintra−λ​C,\\displaystyle\\max\_\{\\boldsymbol\{\\Pi\}\}\\mathcal\{L\}\_\{\\text\{intra\}\}\-\\lambda\(C\-\\mathcal\{L\}\_\{\\text\{intra\}\}\)=\(1\+\\lambda\)\\mathcal\{L\}\_\{\\text\{intra\}\}\-\\lambda C,⇒\\displaystyle\\Rightarrowmax𝚷⁡\(1\+λ\)​∑k=1K‖∑i=1r​Kπi,k​𝐠^i‖2−λ​C,\\displaystyle\\max\_\{\\boldsymbol\{\\Pi\}\}\(1\+\\lambda\)\\sum\_\{k=1\}^\{K\}\\left\\\|\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\}\-\\lambda C,⇒\\displaystyle\\Rightarrowmax𝚷​∑k=1K‖∑i=1r​Kπi,k​𝐠^i‖2,\\displaystyle\\max\_\{\\boldsymbol\{\\Pi\}\}\\sum\_\{k=1\}^\{K\}\\left\\\|\\sum\_\{i=1\}^\{rK\}\\pi\_\{i,k\}\\mathbf\{\\widehat\{g\}\}\_\{i\}\\right\\\|^\{2\},which is equivalent to the objective in Eq\. \([10](https://arxiv.org/html/2605.05676#S3.E10)\)\.

## Appendix CProof of Regrouping Invariance

### C\.1Invariance of Rank\-1 Regrouping

We consider a LoRAMoE withKKexperts, each expertkkparameterized by a low\-rank adaptation of rankrr\. Denote by𝐀k∈ℝm×r\\mathbf\{A\}\_\{k\}\\in\\mathbb\{R\}^\{m\\times r\}and𝐁k∈ℝr×n\\mathbf\{B\}\_\{k\}\\in\\mathbb\{R\}^\{r\\times n\}the LoRA factors for expertkk, so that the LoRA update for expertkkis given by

Δ​𝐖k=𝐀k​𝐁k=∑i=1r𝐚k​i​𝐛k​i⊤,\\Delta\\mathbf\{W\}\_\{k\}=\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}=\\sum\_\{i=1\}^\{r\}\\mathbf\{a\}\_\{ki\}\\mathbf\{b\}\_\{ki\}^\{\\top\},where𝐚k​i∈ℝm\\mathbf\{a\}\_\{ki\}\\in\\mathbb\{R\}^\{m\}is theii\-th column of𝐀k\\mathbf\{A\}\_\{k\}and𝐛k​i∈ℝn\\mathbf\{b\}\_\{ki\}\\in\\mathbb\{R\}^\{n\}is theii\-th row of𝐁k\\mathbf\{B\}\_\{k\}\.

Let the residual weight matrix of the layer be𝐖^\\mathbf\{\\widehat\{W\}\}\. The total weight matrix of the LoRAMoE layer is

𝐖=𝐖^\+∑k=1KΔ​𝐖k\.\\mathbf\{W\}=\\mathbf\{\\widehat\{W\}\}\+\\sum\_\{k=1\}^\{K\}\\Delta\\mathbf\{W\}\_\{k\}\.
Define the set of allr​KrKrank\-1 components as

ℛ=\{𝐑i=𝐚i​𝐛i⊤\|i=1,2,…,r​K\},\\mathcal\{R\}=\\\{\\mathbf\{R\}\_\{i\}=\\mathbf\{a\}\_\{i\}\\mathbf\{b\}\_\{i\}^\{\\top\}\|\\ i=1,2,\\dots,rK\\\},where each𝐑i∈ℝm×n\\mathbf\{R\}\_\{i\}\\in\\mathbb\{R\}^\{m\\times n\}is exactly one rank\-1 term from some expert’s LoRA factors\.

###### Theorem C\.1\(Invariance of Rank\-1 Regrouping\)\.

Suppose the collectionℛ\\mathcal\{R\}is fixed element\-wise \(i\.e\., each𝐑i\\mathbf\{R\}\_\{i\}is unchanged in value\), and that the setℛ\\mathcal\{R\}is partitioned intoKKdisjoint groups of sizerrto form the per\-expert LoRA updates\. If the partition is changed to any other partition intoKKdisjoint groups of sizerrwithout altering the individual𝐑i\\mathbf\{R\}\_\{i\}values, then the total LoRAMoE weight matrix𝐖\\mathbf\{W\}remains unchanged\.

###### Proof\.

Let the original partition of indices\{1,…,r​K\}\\\{1,\\dots,rK\\\}be𝒢1old,…,𝒢Kold\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{1\},\\dots,\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{K\}, with\|𝒢kold\|=r\|\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{k\}\|=rfor allkkand⋃k=1K𝒢kold=\{1,…,r​K\}\\bigcup\_\{k=1\}^\{K\}\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{k\}=\\\{1,\\dots,rK\\\},𝒢kold∩𝒢jold=∅\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{k\}\\cap\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{j\}=\\varnothingfork≠jk\\neq j\.

The original total LoRA update is

Δ​𝐖old=∑k=1K∑i∈𝒢kold𝐑i=∑i=1r​K𝐑i\.\\Delta\\mathbf\{W\}\_\{\\mathrm\{old\}\}=\\sum\_\{k=1\}^\{K\}\\sum\_\{i\\in\\mathcal\{G\}^\{\\mathrm\{old\}\}\_\{k\}\}\\mathbf\{R\}\_\{i\}=\\sum\_\{i=1\}^\{rK\}\\mathbf\{R\}\_\{i\}\.\(17\)
Consider any new partition𝒢1new,…,𝒢Knew\\mathcal\{G\}^\{\\mathrm\{new\}\}\_\{1\},\\dots,\\mathcal\{G\}^\{\\mathrm\{new\}\}\_\{K\}of\{1,…,r​K\}\\\{1,\\dots,rK\\\}, with the same cardinality constraint\|𝒢knew\|=r\|\\mathcal\{G\}^\{\\mathrm\{new\}\}\_\{k\}\|=rand forming a disjoint union of the entire index set\.

The new total LoRA update is

Δ​𝐖new=∑k=1K∑i∈𝒢knew𝐑i=∑i=1r​K𝐑i\.\\Delta\\mathbf\{W\}\_\{\\mathrm\{new\}\}=\\sum\_\{k=1\}^\{K\}\\sum\_\{i\\in\\mathcal\{G\}^\{\\mathrm\{new\}\}\_\{k\}\}\\mathbf\{R\}\_\{i\}=\\sum\_\{i=1\}^\{rK\}\\mathbf\{R\}\_\{i\}\.\(18\)
Comparing Eqs\. \([17](https://arxiv.org/html/2605.05676#A3.E17)\) and \([18](https://arxiv.org/html/2605.05676#A3.E18)\) yields

Δ​𝐖new=Δ​𝐖old\.\\Delta\\mathbf\{W\}\_\{\\mathrm\{new\}\}=\\Delta\\mathbf\{W\}\_\{\\mathrm\{old\}\}\.Therefore, since𝐖=𝐖^\+Δ​𝐖\\mathbf\{W\}=\\mathbf\{\\widehat\{W\}\}\+\\Delta\\mathbf\{W\}, we have𝐖new=𝐖old\\mathbf\{W\}^\{\\mathrm\{new\}\}=\\mathbf\{W\}^\{\\mathrm\{old\}\}\. ∎

### C\.2Invariance of Gated Rank\-1 Regrouping

For a given input𝐱\\mathbf\{x\}, a gating network produces scalar coefficientsgk​\(𝐱\)∈ℝg\_\{k\}\(\\mathbf\{x\}\)\\in\\mathbb\{R\}fork∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}\. The output of the MoE layer is

𝐡​\(𝐱\)=∑k=1Kgk​\(𝐱\)​\[\(𝐖^\+Δ​𝐖k\)​𝐱\]\.\\mathbf\{h\}\(\\mathbf\{x\}\)=\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\mathbf\{x\}\)\\left\[\\left\(\\mathbf\{\\widehat\{W\}\}\+\\Delta\\mathbf\{W\}\_\{k\}\\right\)\\mathbf\{x\}\\right\]\.We consider the LoRAMoE setting and notation from the previous theorem as

Δ​𝐖k=∑i∈𝒢k𝐑i,𝐑i=𝐚i​𝐛i⊤,\\Delta\\mathbf\{W\}\_\{k\}=\\sum\_\{i\\in\\mathcal\{G\}\_\{k\}\}\\mathbf\{R\}\_\{i\},\\quad\\mathbf\{R\}\_\{i\}=\\mathbf\{a\}\_\{i\}\\mathbf\{b\}\_\{i\}^\{\\top\},where expertkkhas gating coefficientgk​\(𝐱\)g\_\{k\}\(\\mathbf\{x\}\)for input𝐱\\mathbf\{x\}\.

Suppose, for a given𝐱\\mathbf\{x\}, we transfer a single rank\-1 component𝐌=𝐑i∗\\mathbf\{M\}=\\mathbf\{R\}\_\{i^\{\*\}\}from its original expert𝐮=𝐮​\(i∗\)\\mathbf\{u\}=\\mathbf\{u\}\(i^\{\*\}\)to a new expert𝐯=𝐯​\(i∗\)\\mathbf\{v\}=\\mathbf\{v\}\(i^\{\*\}\), with𝐮≠𝐯\\mathbf\{u\}\\neq\\mathbf\{v\}\. We rescale it by

γ​\(𝐱\)=gu​\(𝐱\)gv​\(𝐱\),assuming​gv​\(𝐱\)≠0\.\\gamma\(\\mathbf\{x\}\)=\\frac\{g\_\{u\}\(\\mathbf\{x\}\)\}\{g\_\{v\}\(\\mathbf\{x\}\)\},\\quad\\text\{assuming \}g\_\{v\}\(\\mathbf\{x\}\)\\neq 0\.
Define the modified expert LoRA updates \(for this𝐱\\mathbf\{x\}\) as

Δ​𝐖^u=Δ​𝐖u−𝐌,\\displaystyle\\Delta\\mathbf\{\\widehat\{W\}\}\_\{u\}=\\Delta\\mathbf\{W\}\_\{u\}\-\\mathbf\{M\},Δ​𝐖^v=Δ​𝐖v\+γ​\(𝐱\)​𝐌,\\displaystyle\\Delta\\mathbf\{\\widehat\{W\}\}\_\{v\}=\\Delta\\mathbf\{W\}\_\{v\}\+\\gamma\(\\mathbf\{x\}\)\\,\\mathbf\{M\},Δ​𝐖^k=Δ​𝐖k​for​k∉\{𝐮,𝐯\}\.\\displaystyle\\Delta\\mathbf\{\\widehat\{W\}\}\_\{k\}=\\Delta\\mathbf\{W\}\_\{k\}\\ \\text\{for \}k\\notin\\\{\\mathbf\{u\},\\mathbf\{v\}\\\}\.
###### Theorem C\.3\(Invariance of Gated Rank\-1 Regrouping\)\.

Under the above modification, the total gated low\-rank adaptation for thisxxsatisfies

∑k=1Kgk​\(𝐱\)​Δ​𝐖^k=∑k=1Kgk​\(𝐱\)​Δ​𝐖k\.\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{\\widehat\{W\}\}\_\{k\}=\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{W\}\_\{k\}\.

###### Proof\.

Consider the modified total \(for fixed𝐱\\mathbf\{x\}\)

∑k=1K\\displaystyle\\sum\_\{k=1\}^\{K\}gk​\(𝐱\)​Δ​𝐖^k=gu​\(𝐱\)​Δ​𝐖^u\+gv​\(𝐱\)​Δ​𝐖^v\\displaystyle g\_\{k\}\(\\mathbf\{x\}\)\\Delta\\mathbf\{\\widehat\{W\}\}\_\{k\}=g\_\{u\}\(\\mathbf\{x\}\)\\Delta\\mathbf\{\\widehat\{W\}\}\_\{u\}\+g\_\{v\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{\\widehat\{W\}\}\_\{v\}\+∑k∉\{𝐮,𝐯\}gk​\(𝐱\)​Δ​𝐖k\\displaystyle\\quad\\quad\+\\sum\\nolimits\_\{k\\notin\\\{\\mathbf\{u\},\\mathbf\{v\}\\\}\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{W\}\_\{k\}=gu​\(𝐱\)​\(Δ​𝐖u−𝐌\)\+gv​\(𝐱\)​\(Δ​𝐖v\+γ​\(𝐱\)​𝐌\)\\displaystyle=g\_\{u\}\(\\mathbf\{x\}\)\\,\\big\(\\Delta\\mathbf\{W\}\_\{u\}\-\\mathbf\{M\}\\big\)\+g\_\{v\}\(\\mathbf\{x\}\)\\,\\big\(\\Delta\\mathbf\{W\}\_\{v\}\+\\gamma\(\\mathbf\{x\}\)\\,\\mathbf\{M\}\\big\)\+∑k∉\{𝐮,𝐯\}gk​\(𝐱\)​Δ​𝐖k\\displaystyle\\quad\\quad\+\\sum\\nolimits\_\{k\\notin\\\{\\mathbf\{u\},\\mathbf\{v\}\\\}\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{W\}\_\{k\}=\[∑k=1Kgk​\(𝐱\)​Δ​𝐖k\]\+\[−gu​\(𝐱\)\+gv​\(𝐱\)​γ​\(𝐱\)\]​𝐌\.\\displaystyle=\\left\[\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{W\}\_\{k\}\\right\]\+\\left\[\-g\_\{u\}\(\\mathbf\{x\}\)\+g\_\{v\}\(\\mathbf\{x\}\)\\,\\gamma\(\\mathbf\{x\}\)\\right\]\\mathbf\{M\}\.Now substituteγ​\(𝐱\)=gu​\(𝐱\)/gv​\(𝐱\)\\gamma\(\\mathbf\{x\}\)=g\_\{u\}\(\\mathbf\{x\}\)/g\_\{v\}\(\\mathbf\{x\}\)

−gu​\(𝐱\)\+gv​\(𝐱\)​gu​\(𝐱\)gv​\(𝐱\)=−gu​\(𝐱\)\+gu​\(𝐱\)=0\.\-g\_\{u\}\(\\mathbf\{x\}\)\+g\_\{v\}\(\\mathbf\{x\}\)\\,\\frac\{g\_\{u\}\(\\mathbf\{x\}\)\}\{g\_\{v\}\(\\mathbf\{x\}\)\}=\-g\_\{u\}\(\\mathbf\{x\}\)\+g\_\{u\}\(\\mathbf\{x\}\)=0\.Thus the extra term vanishes, and

∑k=1Kgk​\(𝐱\)​Δ​𝐖^k=∑k=1Kgk​\(𝐱\)​Δ​𝐖k\.\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{\\widehat\{W\}\}\_\{k\}=\\sum\_\{k=1\}^\{K\}g\_\{k\}\(\\mathbf\{x\}\)\\,\\Delta\\mathbf\{W\}\_\{k\}\.This proves invariance for this fixed𝐱\\mathbf\{x\}\. ∎

###### Corollary C\.4\(Necessary and sufficient condition for the regrouping invariance\)\.

If gating coefficientsgk​\(𝐱\)g\_\{k\}\(\\mathbf\{x\}\)vary with𝐱\\mathbf\{x\}and the transfer/rescaling is fixed in the model definition, identical outputs for*all*𝐱\\mathbf\{x\}occur if and only if the ratiogu​\(𝐱\)/gv​\(𝐱\)g\_\{u\}\(\\mathbf\{x\}\)/g\_\{v\}\(\\mathbf\{x\}\)is constant \(over all𝐱\\mathbf\{x\}\) for the transferred component\. In that case, the constant definesγ\\gamma, and the above proof applies to every𝐱\\mathbf\{x\}without change\. If the ratio is input\-dependent, the exact invariance rule can still be applied pointwise by settingγ​\(𝐱\)\\gamma\(\\mathbf\{x\}\)according togu​\(𝐱\)/gv​\(𝐱\)g\_\{u\}\(\\mathbf\{x\}\)/g\_\{v\}\(\\mathbf\{x\}\)for each forward computation\.

## Appendix DComplete Algorithm

In Alg[1](https://arxiv.org/html/2605.05676#alg1), we present the complete optimization procedure of our methodBadit\.

Algorithm 1The overall training pipeline ofBadit0:Pre\-trained LLM weights

𝜽0\\boldsymbol\{\\theta\}^\{0\}, number of experts

KK, rank per expert

rr
0:Fine\-tuned model with orthogonal basic abilities

⊳\\trianglerightBasic Ability Decomposition

foreach weight matrix

𝐖0∈𝜽0\\mathbf\{W\}^\{0\}\\in\\boldsymbol\{\\theta\}^\{0\}do

Decompose

𝐖0→∑k=1K𝐀k​𝐁k\+𝐖^\\mathbf\{W\}^\{0\}\\rightarrow\\sum\\nolimits\_\{k=1\}^\{K\}\\mathbf\{A\}\_\{k\}\\mathbf\{B\}\_\{k\}\+\\mathbf\{\\widehat\{W\}\}using Eq\. \([3\.3](https://arxiv.org/html/2605.05676#S3.Ex2)\)

Freeze residual

𝐖^\\widehat\{\\mathbf\{W\}\}during fine\-tuning

endfor

⊳\\trianglerightDynamically Orthogonal Grouping

foreach training batch

\(𝐱t,yt\)\(\\mathbf\{x\}^\{t\},y^\{t\}\)do

Compute gradients

𝐠^i\\widehat\{\\mathbf\{g\}\}\_\{i\}using Eqs\. \([8](https://arxiv.org/html/2605.05676#S3.E8)\) and \([9](https://arxiv.org/html/2605.05676#S3.E9)\)

Initialize grouping via spherical

KK\-means using Eq\. \([11](https://arxiv.org/html/2605.05676#S3.E11)\)

for

τ=1\\tau=1to

1010do

Compute cluster centroids using Eq\. \([12](https://arxiv.org/html/2605.05676#S3.E12)\)

Orthogonalize centroids via SVD using Eq\. \([13](https://arxiv.org/html/2605.05676#S3.E13)\)

Update assignment

𝚷\(τ\)\\boldsymbol\{\\Pi\}^\{\(\\tau\)\}using Eq\. \([14](https://arxiv.org/html/2605.05676#S3.E14)\)

if

𝚷\(τ\)\\boldsymbol\{\\Pi\}^\{\(\\tau\)\}is unchangedthen

break

endif

endfor

Regroup LoRA components with

𝚷\\boldsymbol\{\\Pi\}
endfor

## Appendix EComputational Complexity

In this section, we give a formal computational complexity analysis of our proposed basic ability decomposition and dynamic regrouping\.

### E\.1Complexity of Basic Ability Decomposition

We consider a single MLP layer weight matrix𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}, wheremmis the output dimension andnnis the input dimension\. Our goal is to decompose𝐖\\mathbf\{W\}intoKKrank\-rrLoRA experts\. The specific process in question performs one truncated SVD of𝐖\\mathbf\{W\}, keeping the topr​KrKsingular vectors/values, partitioning them intoKKgroups ofrrvectors, one group per expert, and using the remaining singular vectors as the residual𝐖^\\mathbf\{\\widehat\{W\}\}\.

We often only need the topr​KrKsingular components, wherer​K≪min⁡\(m,n\)rK\\ll\\min\(m,n\)\. Using algorithms such as Randomized SVD, we can compute only the firstr​KrKsingular values and vectors without forming the entire decomposition\. In truncated SVD, the main operations consist ofr​KrKiterations of matrix–vector multiplication, followed by orthogonalization\. Both methods have a complexity of𝒪​\(m​n​r​K\)\\mathcal\{O\}\(mnrK\)\.

### E\.2Complexity of Dynamic Rank\-1 Regrouping

LetKKbe the number of experts,rrthe LoRA rank per expert, andm×nm\\times nthe shape of each rank\-1 component\. The total number of components isr​KrK\. The dynamic regrouping consists of:

Similarity computation\.We compute pairwise similarities between allr​KrKcomponents:𝒪​\(m\+n\)\\mathcal\{O\}\(m\+n\)\. There arer2​K2r^\{2\}K^\{2\}pairs, hence𝒯sim=𝒪​\(r2​K2​\(m\+n\)\)\\mathcal\{T\}\_\{\\text\{sim\}\}=\\mathcal\{O\}\(r^\{2\}K^\{2\}\(m\+n\)\)and space𝒪​\(r2​K2\)\\mathcal\{O\}\(r^\{2\}K^\{2\}\)to store the similarity matrix\.

Group assignment\.Given the similarity matrix, finding the optimal partition with capacity constraints can be done by greedy search:𝒪​\(r2​K2\)\\mathcal\{O\}\(r^\{2\}K^\{2\}\)time\. Space is dominated by the similarity matrix:𝒪​\(r2​K2\)\\mathcal\{O\}\(r^\{2\}K^\{2\}\)\.

Therefore, the overall time complexity is𝒪​\(r2​K2​\(m\+n\)\)\\mathcal\{O\}\(r^\{2\}K^\{2\}\(m\+n\)\), and space complexity is𝒪​\(r2​K2\)\\mathcal\{O\}\(r^\{2\}K^\{2\}\)\.

## Appendix FMore Literature Reviews

In this section, we provide a comprehensive organization and overview of existing methods for multi\-task instruct\-tuning and parameter\-efficient fine\-tuning of LLMs\.

### F\.1LLM Multi\-Task Instruct\-Tuning

According to recent surveys\(Wanget al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib20); Shiet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib21)\), existing multi\-task instruct\-tuning approaches can be broadly categorized into three main paradigms:rehearsal\-based,optimization\-based, andarchitecture\-basedmethods\. Although multi\-task learning has been extensively explored in the computer vision community, its application to LLMs remains in its early stages\. Accordingly, this section provides a brief overview of general multi\-task instruct\-tuning techniques and their variants in LLMs\.

Rehearsal\-based methodsdirectly store a portion of old data and re\-train the model alongside new data when learning new tasks, thus preserving old knowledge\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2605.05676#bib.bib49); Liet al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib85)\)\. This approach is straightforward and effective, yet it encounters challenges such as storage costs\(de Masson d’Autumeet al\.,[2019](https://arxiv.org/html/2605.05676#bib.bib50); Huanget al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib32)\)\. To tackle the issue of storage costs, several generative rehearsal\-based methods employ generative models to generate data samples relevant to old tasks, thereby replacing the stored old task data\. For example, they utilize variational auto\-encoders\(Shinet al\.,[2017](https://arxiv.org/html/2605.05676#bib.bib19)\)and generative adversarial networks\(Liuet al\.,[2020](https://arxiv.org/html/2605.05676#bib.bib54); Wanget al\.,[2022a](https://arxiv.org/html/2605.05676#bib.bib53)\), to learn the data distribution of old tasks and generate data by sampling from this distribution in new tasks\. On this basis, most existing efforts to address LLM cross\-task interference involve synthesizing high\-quality data for rehearsal using untrained proxy LLMs\(Sunet al\.,[2020](https://arxiv.org/html/2605.05676#bib.bib56); Wanget al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib34)\)\. For example, SSR\(Huanget al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib32)\)uses an untrained base model to generate task samples, and then employs the latest trained model to produce responses, thereby constructing rehearsal data pairs\.

Optimization\-based methodsprimarily focus on the gradients between tasks during model training, aiming to guarantee the optimization gradients across tasks do not interfere with each other\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17); Yuanet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib87)\)\. To achieve this, most existing approaches employ orthogonal optimization, aiming to make the gradients between tasks mutually orthogonal\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2605.05676#bib.bib49); Sahaet al\.,[2021](https://arxiv.org/html/2605.05676#bib.bib57)\)or to identify flat minima in the loss landscape during alternative task training\(Liet al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib33); Bianet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib58)\)\. For example, O\-LoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)learns distinct tasks in different low\-rank subspaces while keeping orthogonality between these subspaces; GORP\(Wanget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib84)\)restricts gradient updates to a low\-rank subspace and ensures, via a projection operation, that this subspace is orthogonal to those used by previous tasks\.

Architecture\-based methodsinvolve isolating parameters between tasks, and allocating a new set of learnable parameters for each new task’s data\(Wanget al\.,[2023b](https://arxiv.org/html/2605.05676#bib.bib16),[a](https://arxiv.org/html/2605.05676#bib.bib17); Yanget al\.,[2025b](https://arxiv.org/html/2605.05676#bib.bib83)\)or extracting task\-specific sub\-networks from the pre\-trained model\(Chenet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib60)\)\. Subsequently, inference for new tasks is performed using techniques such as automatic routing\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1); Maet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib27); Fenget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib26)\)\. For example, LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)designs a LoRA\-based expert model for each task and employs a trained routing function for expert selection\.

In this work, we also attempt to explain the mechanism of parameter overlap across different tasks, which has received relatively little attention\.Songet al\.\([2024](https://arxiv.org/html/2605.05676#bib.bib70)\)identify the existence of task‑specific neurons in LLMs and propose using special tokens to locate these neurons; Building on this,Leng and Xiong \([2025](https://arxiv.org/html/2605.05676#bib.bib71)\)further demonstrate that the overlap among such task‑specific parameters is strongly correlated with the cross‑task generalization\.

### F\.2Parameter\-Efficient Fine\-Tuning of LLMs

LLMs and their multimodal variants have demonstrated remarkable performance in diverse downstream tasks, e\.g\., mathematical reasoning\(Liuet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib91); Huanget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib88)\)\. Nevertheless, their training costs remain prohibitively high\(Liuet al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib43); Yanet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib92); Xieet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib90)\)\. Therefore, cutting\-edge works proposeparameter\-efficient fine\-tuning\(PEFT\) methods, with LoRA and MoE series receiving the most attention\. Specifically, LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6); Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5); Kouet al\.,[2026](https://arxiv.org/html/2605.05676#bib.bib89)\)is built on the assumption that the parameter changes during model adaptation to new tasks exhibit low\-rank properties, and it possesses plug\-and\-play characteristics\. For example, DoRA\(Liuet al\.,[2024c](https://arxiv.org/html/2605.05676#bib.bib40)\)decomposes pre\-trained weights into magnitude and directional components; QLoRA\(Dettmerset al\.,[2023](https://arxiv.org/html/2605.05676#bib.bib41)\)further quantizes LoRA, significantly reducing the fine\-tuning costs of LLMs\. Additionally, the MoE architecture\(Shazeeret al\.,[2017](https://arxiv.org/html/2605.05676#bib.bib13); Gaoet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib10)\), due to its sparsity and selective activation characteristics, has become the preferred choice for several open\-source foundation LMs, e\.g\., Mixtral\(Jianget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib42)\)and DeepSeek\(Liuet al\.,[2024a](https://arxiv.org/html/2605.05676#bib.bib43)\), and also serves as the base structure for numerous multi\-task models\(Maet al\.,[2018](https://arxiv.org/html/2605.05676#bib.bib46); Huanget al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib45)\)\. More recently, partial works attempt to re\-initialize LoRAs by select key dimensions in pre\-trained weights through SVD\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5); Yanget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib35); Fanet al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib59)\), and this paper draws inspiration from these works to propose decomposing the basic abilities of LLMs\.

Table 4:Details of the multi\-task datasets SuperNI\(Wanget al\.,[2022b](https://arxiv.org/html/2605.05676#bib.bib67)\)\.Task IDDataset NameTask\#Train\#EvalMetricsT002task002\_quoref\_answer\_generationquestion answering1,000100Rouge\-LT073task073\_commonsenseqa\_answer\_generationquestion answering975100Rouge\-LT591task591\_sciq\_answer\_generationquestion answering1,000100Rouge\-LT181task181\_outcome\_extractioninformation extraction33843Rouge\-LT748task748\_glucose\_reverse\_cause\_event\_detectioninformation extraction1,000100Rouge\-LT1510task1510\_evalution\_relation\_extractioninformation extraction1,000100Rouge\-LT363task363\_sst2\_polarity\_classificationsentiment analysis1,000100AccuracyT875task875\_emotion\_classificationsentiment analysis1,000100AccuracyT1687task1687\_sentiment140\_classificationsentiment analysis1,000100AccuracyT511task511\_reddit\_tifu\_long\_text\_summarizationtext summarization1,000100Rouge\-LT1290task1290\_xsum\_summarizationtext summarization1,000100Rouge\-LT1572task1572\_samsum\_summarytext summarization16020Rouge\-LT639task639\_multi\_woz\_user\_utterance\_generationdialogue generation14218Rouge\-LT1590task1590\_diplomacy\_text\_generationdialogue generation12616Rouge\-LT1729task1729\_personachat\_generate\_nextdialogue generation1,000100Rouge\-LTable 5:Five different random seeds correspond to five distinct task orderings under the sequential training setting\.SeedTask Ordering1T591→T073→T1687→T875→T1572→T639→T1510→T1590→T511→T181→T363→T1729→T002→T1290→T748\\text\{T591\}\\rightarrow\\text\{T073\}\\rightarrow\\text\{T1687\}\\rightarrow\\text\{T875\}\\rightarrow\\text\{T1572\}\\rightarrow\\text\{T639\}\\rightarrow\\text\{T1510\}\\rightarrow\\text\{T1590\}\\rightarrow\\text\{T511\}\\rightarrow\\text\{T181\}\\rightarrow\\text\{T363\}\\rightarrow\\text\{T1729\}\\rightarrow\\text\{T002\}\\rightarrow\\text\{T1290\}\\rightarrow\\text\{T748\}2T1290→T1510→T1572→T002→T073→T511→T1590→T363→T748→T639→T181→T1729→T1687→T591→T875\\text\{T1290\}\\rightarrow\\text\{T1510\}\\rightarrow\\text\{T1572\}\\rightarrow\\text\{T002\}\\rightarrow\\text\{T073\}\\rightarrow\\text\{T511\}\\rightarrow\\text\{T1590\}\\rightarrow\\text\{T363\}\\rightarrow\\text\{T748\}\\rightarrow\\text\{T639\}\\rightarrow\\text\{T181\}\\rightarrow\\text\{T1729\}\\rightarrow\\text\{T1687\}\\rightarrow\\text\{T591\}\\rightarrow\\text\{T875\}3T002→T073→T1572→T181→T591→T1687→T363→T1729→T511→T875→T639→T748→T1590→T1290→T1510\\text\{T002\}\\rightarrow\\text\{T073\}\\rightarrow\\text\{T1572\}\\rightarrow\\text\{T181\}\\rightarrow\\text\{T591\}\\rightarrow\\text\{T1687\}\\rightarrow\\text\{T363\}\\rightarrow\\text\{T1729\}\\rightarrow\\text\{T511\}\\rightarrow\\text\{T875\}\\rightarrow\\text\{T639\}\\rightarrow\\text\{T748\}\\rightarrow\\text\{T1590\}\\rightarrow\\text\{T1290\}\\rightarrow\\text\{T1510\}4T1290→T875→T639→T1590→T591→T073→T1687→T002→T748→T511→T1572→T181→T1729→T363→T1510\\text\{T1290\}\\rightarrow\\text\{T875\}\\rightarrow\\text\{T639\}\\rightarrow\\text\{T1590\}\\rightarrow\\text\{T591\}\\rightarrow\\text\{T073\}\\rightarrow\\text\{T1687\}\\rightarrow\\text\{T002\}\\rightarrow\\text\{T748\}\\rightarrow\\text\{T511\}\\rightarrow\\text\{T1572\}\\rightarrow\\text\{T181\}\\rightarrow\\text\{T1729\}\\rightarrow\\text\{T363\}\\rightarrow\\text\{T1510\}5T748→T1510→T875→T002→T591→T1729→T1572→T511→T1687→T1590→T073→T639→T181→T363→T1290\\text\{T748\}\\rightarrow\\text\{T1510\}\\rightarrow\\text\{T875\}\\rightarrow\\text\{T002\}\\rightarrow\\text\{T591\}\\rightarrow\\text\{T1729\}\\rightarrow\\text\{T1572\}\\rightarrow\\text\{T511\}\\rightarrow\\text\{T1687\}\\rightarrow\\text\{T1590\}\\rightarrow\\text\{T073\}\\rightarrow\\text\{T639\}\\rightarrow\\text\{T181\}\\rightarrow\\text\{T363\}\\rightarrow\\text\{T1290\}

## Appendix GMore Experimental Details

In this section, we provide a detailed description of our experimental setup, including dataset statistics, specifications of the LLMs used, baseline methods for comparison, implementation details, and formal evaluation metrics\.

### G\.1Evaluation Benchmarks

SuperNI\(Wanget al\.,[2022b](https://arxiv.org/html/2605.05676#bib.bib67)\)is a benchmark comprising a diverse set of NLP tasks paired with expert\-written instructions, designed to enable rigorous evaluation of multi\-task instruct\-tuning methods for LLMs\. Specifically, we followZhaoet al\.\([2024b](https://arxiv.org/html/2605.05676#bib.bib38)\)to select three tasks each from five categories, including dialogue generation, information extraction, question answering, summarization, and sentiment analysis, resulting in a sequence of 15 tasks used to evaluate the various multi\-task approaches\. For each individual task, we truncate its training set to 1,000 samples and its validation set to 100 samples; for tasks with fewer samples than these thresholds, we retain their original sizes\. Detailed statistics, domains, and evaluation metrics for these datasets are summarized in Table[4](https://arxiv.org/html/2605.05676#A6.T4)\. Additionally, in Table[5](https://arxiv.org/html/2605.05676#A6.T5)we show the task orderings corresponding to the sequential learning setting when the random seed is set to\{1,2,3,4,5\}\\\{1,2,3,4,5\\\}, respectively\.

### G\.2Model Cards

We employ three families of contemporary open\-source: Qwen3\(Yanget al\.,[2025a](https://arxiv.org/html/2605.05676#bib.bib77)\), Llama3\(AI@Meta,[2024](https://arxiv.org/html/2605.05676#bib.bib3)\), Gemma2\(Rivièreet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib81)\), represented by two parameter scales to evaluate performance across model sizes\.

- •Qwen3\-4B111[https://huggingface\.co/Qwen/Qwen3\-4B](https://huggingface.co/Qwen/Qwen3-4B)/ 8B222[https://huggingface\.co/Qwen/Qwen3\-8B](https://huggingface.co/Qwen/Qwen3-8B)are dense\-architecture LLMs, which are distilled variants derived via off\-policy and on\-policy knowledge distillation from larger teacher models, aiming to preserve strong reasoning and instruction\-following capabilities while reducing computational cost\.
- •
- •

### G\.3Baselines

We compare our approach against a representative set of parameter\-efficient fine\-tuning baselines, including two low\-rank adaptation variants and three recent MoE\-inspired methods tailored for multi\-task instruct\-tuning:

- •LoRA\(Huet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib6)\)is a foundational PEFT method that freezes the pre\-trained model weights and injects trainable low\-rank decomposition matrices into attention layers, drastically reducing the number of trainable parameters while preserving performance close to full fine\-tuning\.
- •PiSSA\(Menget al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib5)\)improves upon LoRA by reinitializing the low\-rank adapters using the principal components of the pre\-trained weight matrix via SVD\. Instead of random or zero initialization, PiSSA decomposes the original weight as𝐖=𝐔​𝚺​𝐕⊤\\mathbf\{W\}=\\mathbf\{U\}\\boldsymbol\{\\Sigma\}\\mathbf\{V\}^\{\\top\}and initializes the adapter to capture the dominant singular directions, leading to faster convergence and higher downstream accuracy with the same parameter budget\.
- •OLoRA\(Wanget al\.,[2023a](https://arxiv.org/html/2605.05676#bib.bib17)\)enhances LoRA’s optimization stability and isolates different LoRA gradients by enforcing orthogonality in the low\-rank adapter matrices\. This structured initialization reduces gradient interference and accelerates training, particularly in low\-data or multi\-task regimes\.
- •LoRAMoE\(Douet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib1)\)integrates the MoE paradigm into LoRA by deploying multiple parallel LoRA adapters per layer, each acting as an expert, along with a trainable router that selects or blends experts based on input tokens\. Crucially, it partitions experts into two groups, one dedicated to preserving world knowledge from pretraining and the other to learning task\-specific instructions, thereby mitigating catastrophic forgetting during supervised fine\-tuning\.
- •GainLoRA\(Lianget al\.,[2025](https://arxiv.org/html/2605.05676#bib.bib82)\)addresses continual and multi\-task learning by dynamically gating the contribution of task\-specific LoRA branches during inference\. It introduces a learnable gate module that modulates how much each LoRA adapter influences the output, with constraints applied during training to suppress interference from newly added adapters on previously learned tasks, thus improving knowledge retention across sequential instruct\-tuning\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x5.png)Figure 5:Shared task\-specific neurons of MLP gate layers across different tasks\.
### G\.4Implementation Details

We implement our method based on the above 6 LLMs, using the Hugging Face Transformers library\. All experiments are conducted on 8 NVIDIA 4090 GPUs with DeepSpeed ZeRO\-2 enabled to support efficient distributed training without CPU offloading, and we enable FlashAttention\-2 for accelerated attention computation\. For parameter\-efficient fine\-tuning, we inject a MoE\-style LoRA architecture into the gate projection of each transformer MLP block\. The base LoRA configuration uses rankr=4r=4, scaling factorα=32\\alpha=32\(yielding an effective scaling ofα/r=8\\alpha/r=8\), and dropout rate of 0\.05\. Our method employs 8 experts, with a top\-k=4k=4routing strategy that activates four experts per token during training and inference\. The expert selection and combination are learned end\-to\-end alongside the router parameters\. Training is performed for 10 epochs on theSuperNIdataset, with a maximum sequence length of 1024 tokens and a target output length capped at 50 tokens\. We use a per\-device batch size of 1 and accumulate gradients over 1 step, resulting in an effective global batch size of 8\. Optimization is carried out using the AdamW optimizer with a cosine learning rate scheduler, peak learning rate of2×10−42\\times 10^\{\-4\}, warmup ratio of 0\.03, and no weight decay\. Mixed\-precision training is enabled via bf16\. We fix the random seed to the values\{1,2,3,4,5\}\\\{1,2,3,4,5\\\}to ensure reproducibility\.

### G\.5Evaluation Metrics

Following standard practice in previous literature\(Lopez\-Paz and Ranzato,[2017](https://arxiv.org/html/2605.05676#bib.bib49); Zhaoet al\.,[2024b](https://arxiv.org/html/2605.05676#bib.bib38)\), we adopt the following four metrics to comprehensively assess model performance:

- •Rougemeasures the overall effectiveness after learning the entire sequence of tasks as Rouge=1T​∑t=1Tat,T,\\textsc\{Rouge\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}a\_\{t,T\},where denotes the model’s performance ontt\-th task after training on theTT\-th task\. This reflects the final model’s ability to perform well across all seen tasks\.
- •Forwardquantifies how much prior knowledge acquired from previously seen tasks helps \(or hinders\) the learning of a new task\. It is measured relative to a baseline where each task is learned in isolation\. Letat,0a\_\{t,0\}denote the model’s performance on thett\-th task when trained alone \(i\.e\., without any other tasks\), Forward is defined as Forward=\{1T​∑t=1T\(at,T−at,0\),mixed training,1T​∑t=1T\(at,t−at,0\),sequential training\.\\text\{Forward\}=\\begin\{cases\}\\dfrac\{1\}\{T\}\\sum\\limits\_\{t=1\}^\{T\}\\left\(a\_\{t,T\}\-a\_\{t,0\}\\right\),&\\text\{mixed training\},\\\\ \\dfrac\{1\}\{T\}\\sum\\limits\_\{t=1\}^\{T\}\\left\(a\_\{t,t\}\-a\_\{t,0\}\\right\),&\\text\{sequential training\}\.\\end\{cases\}A positive Forward value indicates that leveraging prior experience improves performance compared to isolated training\.
- •Forget Ratequantifies the extent of catastrophic forgetting on previously learned tasks under the sequential training setting as Forget Rate=1T−1​∑t=1T−1\(at,t−at,T\)\.\\text\{Forget Rate\}=\\frac\{1\}\{T\-1\}\\sum\_\{t=1\}^\{T\-1\}\(a\_\{t,t\}\-a\_\{t,T\}\)\.A higher value indicates more severe forgetting, as it captures the performance drop from the peak \(after learningtt\-th task\) to the end \(after learning allTTtasks\)\.
- •Backwardmeasures the influence of learning subsequent tasks on the performance of earlier tasks under the sequential training setting as Backward=1T−1​∑t=1T−1\(at,T−at,t\)\.\\text\{Backward\}=\\frac\{1\}\{T\-1\}\\sum\_\{t=1\}^\{T\-1\}\(a\_\{t,T\}\-a\_\{t,t\}\)\.Backward can be positive if later tasks improve earlier ones \(positive backward transfer\), making it a more general measure\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x6.png)Figure 6:Shared task\-specific parameters of MLP gate layers across different tasks\.![Refer to caption](https://arxiv.org/html/2605.05676v1/x7.png)Figure 7:Shared task\-specific parameters of query matrices in self\-attention layers across different tasks\.![Refer to caption](https://arxiv.org/html/2605.05676v1/x8.png)Figure 8:Shared task\-specific parameters of key matrices in self\-attention layers across different tasks\.![Refer to caption](https://arxiv.org/html/2605.05676v1/x9.png)Figure 9:Shared task\-specific parameters of value matrices in self\-attention layers across different tasks\.

## Appendix HMore Experimental Results

In this section, we present additional experimental investigations and the results of our evaluation studies, including gradient angles, sensitivity analysis, and performance across different tasks\.

### H\.1More Investigation Results

In this section, we employ the same experimental setup as Sec\.[2](https://arxiv.org/html/2605.05676#S2)to investigate neuron and parameter sharing behaviors across different tasks in a broader range of modules\. Specifically, we conduct experiments on both Llama3\-3B and Qwen3\-4B and report their cross\-task sharing performance in key components, including the gate projections in the MLP blocks and the query, key, and value matrices in the self\-attention modules, in Figs\.[5](https://arxiv.org/html/2605.05676#A7.F5),[6](https://arxiv.org/html/2605.05676#A7.F6),[7](https://arxiv.org/html/2605.05676#A7.F7),[8](https://arxiv.org/html/2605.05676#A7.F8), and[9](https://arxiv.org/html/2605.05676#A7.F9)\. Furthermore, we examine these modules across Transformer layers at three distinct depths: low, middle, and high\. For Llama3\-3B, which comprises 28 layers in total, we select layers 1, 14, and 27 to represent the low, middle, and high levels, respectively\. Similarly, for Qwen3\-4B, which has 36 layers, we analyze layers 1, 18, and 35 as representatives of the corresponding depth categories\.

Overall, our findings are consistent with the conclusions drawn in Sec\.[2](https://arxiv.org/html/2605.05676#S2): a substantial portion of neurons and parameters remains inactive across all tasks, while those that are co\-activated continue to naturally organize into base compositional groups\. Additionally, Qwen3\-4B again exhibits the phenomenon where the majority of activated neurons are concentrated at lower neuron IDs\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x10.png)Figure 10:The number of parameters that activate each parameter column with positive or negative gradients across all tasks, respectively\.Comparing sharing behaviors across different layers, we first observe that, for the gate projections in Figs\.[5](https://arxiv.org/html/2605.05676#A7.F5)and[6](https://arxiv.org/html/2605.05676#A7.F6), activation patterns show no significant variation across layers, e\.g\., activations in Qwen3\-4B consistently cluster at lower neuron IDs regardless of depth\. In contrast, for the query, key, and value matrices in the self\-attention modules, we identify systematic shifts in activation patterns from lower to higher layers\. Specifically, activated parameters tend to migrate from higher to lower parameter IDs as depth increases, for instance, in the query matrix of Llama3\-3B in Fig\.[7](https://arxiv.org/html/2605.05676#A7.F7), or transition from scattered, diffuse activations toward more concentrated clusters, as seen in the value matrix of Qwen3\-4B in Fig\.[9](https://arxiv.org/html/2605.05676#A7.F9)\. Although these trends indicate that different layers exhibit distinct sharing behaviors, they all remain consistent with our core hypothesis: the parameter matrices in each layer can be decomposed into a set of basic, reusable abilities\. Notably, prior work has suggested that only the gate projections require decoupling\(Daiet al\.,[2022](https://arxiv.org/html/2605.05676#bib.bib75); Songet al\.,[2024](https://arxiv.org/html/2605.05676#bib.bib70)\)\. Given that the sharing patterns of gate matrices are largely consistent across layers in Fig\.[6](https://arxiv.org/html/2605.05676#A7.F6), applying the same decoupling strategy uniformly across all layers remains justified and effective\.

Moreover, a potential concern arises from our primary focus on the magnitude of parameter activation, i\.e\., the absolute value of gradients, across tasks\. Specifically, if different tasks induce gradients of opposite signs on the same parameter, they may exhibit conflicting optimization directions despite similar activation magnitudes\. To investigate this, we present an additional experiment in Fig\.[10](https://arxiv.org/html/2605.05676#A8.F10), which visualizes, for each parameter, the number of tasks that produce positive versus negative gradients\. In this figure, the vertical axis represents the total number of activated parameters across 15 distinct tasks for each parameter row \(i\.e\., the sum of tasks that activate that particular parameter, regardless of gradient sign\)\. This analysis helps reveal whether conflicting gradient directions are prevalent and provides further insight into the compatibility of task\-specific updates at the parameter level\. As shown in Fig\.[10](https://arxiv.org/html/2605.05676#A8.F10), the majority of parameters are consistently activated with gradients of the same sign across different tasks; only a small fraction exhibit both positive and negative gradients simultaneously\. This result demonstrates that, even when multiple tasks activate the same parameters, their gradient directions are largely aligned in most cases\. Consequently, this further supports our central analogy: such shared activation patterns can be interpreted as the reuse of basic abilities across tasks\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x11.png)Figure 11:Intra\- and inter\-expert gradient angles ofBaditon 6 LLMs across epochs\.![Refer to caption](https://arxiv.org/html/2605.05676v1/x12.png)Figure 12:Sensitivity analysis of the parametersKKandrr\.
### H\.2More Gradient Angles

In this section, we extend the intra\-expert and inter\-expert gradient angle results of our methodBadit, originally presented in Fig\.[4](https://arxiv.org/html/2605.05676#S4.F4), to a broader set of LLMs, with the extended results shown in Fig\.[11](https://arxiv.org/html/2605.05676#A8.F11)\. We conduct experiments across all 6 LLMs and evaluate two different task orders:Task Order AandTask Order B, which correspond to the training sequences using random seeds 1 and 2, respectively, as specified in Table[5](https://arxiv.org/html/2605.05676#A6.T5)\. The experimental results align with the findings in Fig\.[4](https://arxiv.org/html/2605.05676#S4.F4): our method successfully maintains inter\-expert gradient angles near 90 degrees, indicating approximate orthogonality\. In some cases, such as Qwen3\-4B under Task Order B, the inter\-expert angles remain consistently clustered around 90 degrees throughout training\. Meanwhile, intra\-expert gradient angles are consistently kept at relatively small values\. These results collectively demonstrate the effectiveness of our DOG method in preserving orthogonality between experts while encouraging coherent updates within each expert\.

### H\.3Sensitivity Analysis

To investigate the impact of the number of LoRA expertsKKand the LoRA rankrron model performance, we conduct a sensitivity analysis across three LLMs: Qwen3\-8B, Llama3\-8B, and Gemma2\-9B, and present the results in Fig\.[12](https://arxiv.org/html/2605.05676#A8.F12)\. Generally, under both mixed training and sequential training settings, our method consistently achieves optimal performance whenK=8K=8andr=4r=4\. Deviating from these values, either increasing or decreasingKKorrr, leads to degraded performance\.

In our method,KKcorresponds to the number of basic abilities decoupled byBadit\. The empirical results indicate that decomposing the model into exactly 8 such abilities yields the best performance\. WhenKKis too small, the set of decoupled abilities is insufficient to adequately represent all 15 tasks\. Conversely, whenKKis too large, redundant abilities emerge: this not only increases training overhead but also interferes with the routing mechanism by diluting its focus on the most relevant abilities\. Similarly, the LoRA rankrrrepresents the granularity of each decoupled unit\. Ifrris too small, individual experts lack sufficient capacity to fully encode a distinct basic ability\. On the other hand, an excessively largerr, much like an oversizedKK, imposes greater computational costs during both training and the DOG process\. Moreover, a largerrrintroduces more candidate features during grouping, which complicates the clean separation of basic abilities and undermines the effectiveness of the grouping strategy\.

### H\.4Visualizing Basic Abilities

To further explore whether these decomposed LoRAs represent distinct skills, we conduct a toy experiment onSuperNI\(K=8, top\-4 experts activated\) and evaluate expert activation on 20MMLUtasks using our proposedBadit\. Their activated patterns are shown in Fig\.[13](https://arxiv.org/html/2605.05676#A8.F13)\.

![Refer to caption](https://arxiv.org/html/2605.05676v1/x13.png)Figure 13:Visualizing basic abilities\.Our qualitative analysis \(where 1=activated, 0=inactive\) reveals intriguing functional patterns: Experts A and F consistently activate across nearly all tasks, suggesting they encode general skills; Experts A, C, D, and F concurrently activate for factual knowledge and logical reasoning; Expert H exhibits a strong preference for history and sociology; Expert E is predominantly triggered by governance, business, and law\-related tasks\. These results suggest that expert activations follow identifiable functional patterns\.

### H\.5Performance Across Different Tasks

We also provide detailed evaluation results under the sequential training setting, showing the performance of the LLMs on all 15 tasks after each new task is learned\. We compare LoRAMoE against ourBaditmethod under two specific configurations: \(1\) training Qwen3\-8B on the task sequence corresponding to random seed 1, and \(2\) training Gemma2\-9B on the task sequence corresponding to random seed 3\. The results for these experiments are presented in Tables[6](https://arxiv.org/html/2605.05676#A8.T6),[7](https://arxiv.org/html/2605.05676#A8.T7),[8](https://arxiv.org/html/2605.05676#A8.T8), and[9](https://arxiv.org/html/2605.05676#A8.T9), respectively\.

The results clearly reveal the presence of catastrophic forgetting during sequential training: as the model learns new tasks, its performance on previously acquired tasks consistently degrades\. For instance, Tasks 591 and 073, the first two tasks in the sequence, exhibit a steady decline in performance as training progresses\. In this challenging setting, our method consistently outperforms the baseline LoRAMoE across all evaluation metrics\. This not only demonstrates that our approach achieves superior overall multi\-task performance but also provides strong evidence that it effectively mitigates task interference and alleviates catastrophic forgetting in sequential learning scenarios\.

Table 6:Evaluation performance of the model across different tasks during sequential training, using theQwen3\-8Bmodel with theLoRAMoEmethod \(Rouge = 48\.43, Forget Rate = 8\.37, Backward = \-5\.71\)\.Evaluation Task ID591073168787515726391510159051118136317290021290748Training Task ID59162\.283\.079\.045\.037\.62\.987\.211\.416\.039\.692\.013\.575\.420\.643\.107368\.182\.779\.043\.038\.14\.579\.513\.615\.327\.690\.014\.475\.719\.541\.6168769\.378\.081\.037\.033\.33\.673\.312\.616\.023\.891\.016\.077\.919\.742\.487566\.674\.073\.069\.038\.14\.244\.711\.415\.421\.878\.016\.078\.720\.240\.7157267\.369\.068\.072\.043\.05\.279\.29\.816\.722\.186\.016\.875\.719\.739\.763960\.670\.063\.067\.036\.512\.377\.811\.715\.816\.677\.013\.976\.719\.539\.8151062\.971\.078\.069\.043\.115\.6100\.011\.814\.836\.586\.016\.677\.820\.243\.0159058\.469\.072\.066\.041\.412\.4100\.011\.815\.739\.584\.015\.275\.520\.644\.051153\.468\.075\.069\.035\.26\.4100\.011\.915\.637\.885\.010\.069\.822\.641\.418151\.364\.074\.068\.032\.98\.099\.711\.012\.869\.271\.013\.373\.321\.942\.236353\.363\.076\.062\.039\.68\.1100\.010\.114\.171\.287\.012\.274\.021\.445\.0172951\.369\.079\.060\.040\.95\.2100\.011\.711\.675\.488\.015\.677\.120\.743\.400252\.161\.082\.056\.039\.75\.1100\.011\.612\.862\.488\.014\.077\.221\.942\.0129046\.659\.075\.055\.018\.28\.199\.07\.38\.468\.085\.012\.371\.125\.734\.274843\.462\.075\.061\.031\.46\.998\.37\.912\.070\.086\.011\.876\.224\.759\.9Table 7:Evaluation performance of the model across different tasks during sequential training, using theQwen3\-8Bmodel with theBaditmethod \(Rouge = 49\.96, Forget Rate = 7\.27, Backward = \-5\.35\)\.Evaluation Task ID591073168787515726391510159051118136317290021290748Training Task ID59169\.278\.080\.044\.041\.85\.190\.814\.116\.350\.791\.013\.777\.720\.141\.707370\.679\.576\.048\.038\.95\.596\.511\.415\.740\.891\.017\.280\.521\.343\.0168769\.477\.086\.043\.037\.16\.791\.711\.616\.333\.887\.015\.881\.320\.640\.187569\.776\.083\.073\.039\.73\.647\.411\.117\.729\.281\.014\.672\.719\.738\.4157267\.975\.081\.075\.039\.76\.376\.812\.015\.930\.979\.013\.680\.420\.643\.363961\.171\.082\.075\.037\.412\.863\.814\.314\.330\.281\.014\.073\.520\.341\.0151065\.974\.081\.076\.039\.811\.7100\.010\.815\.434\.583\.014\.180\.319\.342\.2159063\.472\.083\.074\.035\.87\.999\.711\.415\.034\.382\.011\.176\.919\.542\.751156\.675\.083\.073\.032\.58\.9100\.08\.615\.329\.284\.010\.571\.922\.237\.318155\.875\.082\.073\.028\.56\.9100\.010\.315\.570\.883\.011\.076\.121\.641\.936360\.673\.081\.070\.033\.99\.7100\.010\.315\.871\.191\.011\.176\.621\.343\.9172960\.472\.080\.068\.034\.710\.799\.711\.215\.165\.992\.012\.877\.721\.144\.200254\.674\.080\.062\.035\.811\.0100\.011\.415\.364\.190\.015\.177\.219\.740\.8129052\.776\.079\.058\.027\.58\.4100\.09\.710\.660\.684\.012\.677\.926\.028\.974853\.575\.081\.056\.029\.17\.699\.011\.812\.755\.389\.013\.676\.424\.465\.1Table 8:Evaluation performance of the model across different tasks during sequential training, using theGemma2\-9Bmodel with theLoRAMoEmethod \(Rouge = 47\.05, Forget Rate = 11\.94, Backward = \-10\.11\)\.Evaluation Task ID002073157218159116873631729511875639748159012901510Training Task ID00283\.149\.034\.626\.467\.773\.087\.212\.615\.645\.04\.725\.47\.521\.289\.007383\.266\.037\.132\.471\.473\.082\.012\.015\.143\.03\.622\.75\.421\.184\.8157279\.876\.030\.065\.766\.875\.093\.015\.514\.946\.05\.639\.911\.721\.197\.018183\.283\.033\.347\.063\.575\.094\.012\.114\.744\.03\.727\.79\.520\.796\.059181\.777\.041\.050\.463\.972\.792\.014\.914\.946\.08\.241\.212\.421\.492\.9168781\.582\.038\.343\.963\.972\.092\.012\.916\.147\.08\.329\.713\.821\.695\.136381\.778\.040\.277\.461\.674\.086\.014\.216\.441\.03\.728\.112\.120\.394\.8172982\.881\.041\.173\.569\.668\.090\.011\.815\.342\.03\.722\.910\.020\.792\.551183\.469\.037\.570\.569\.870\.087\.08\.214\.440\.01\.221\.44\.218\.983\.387578\.736\.032\.972\.772\.372\.090\.07\.613\.845\.00\.819\.62\.618\.677\.063984\.469\.037\.266\.871\.086\.081\.08\.912\.446\.01\.119\.913\.720\.788\.474879\.943\.030\.665\.771\.987\.088\.06\.913\.445\.00\.818\.82\.919\.283\.3159081\.758\.040\.357\.272\.178\.092\.012\.812\.735\.00\.416\.111\.518\.980\.3129077\.962\.034\.756\.574\.480\.089\.011\.111\.740\.01\.620\.011\.819\.872\.5151079\.957\.038\.562\.470\.577\.091\.017\.413\.435\.06\.418\.710\.419\.491\.4Table 9:Evaluation performance of the model across different tasks during sequential training, using theGemma2\-9Bmodel with theBaditmethod \(Rouge = 50\.23, Forget Rate = 7\.72, Backward = \-6\.39\)\.Evaluation Task ID002073157218159116873631729511875639748159012901510Training Task ID00282\.570\.034\.028\.371\.673\.089\.012\.215\.743\.04\.530\.18\.021\.893\.007382\.781\.030\.853\.368\.074\.092\.014\.615\.445\.06\.941\.08\.721\.097\.3157282\.378\.040\.545\.070\.073\.091\.013\.915\.046\.09\.434\.911\.921\.896\.118176\.077\.039\.868\.466\.274\.090\.013\.415\.340\.05\.430\.013\.520\.787\.859177\.773\.036\.064\.172\.569\.090\.08\.313\.841\.02\.622\.14\.219\.283\.8168776\.568\.041\.062\.272\.885\.090\.08\.011\.342\.02\.416\.64\.918\.185\.836378\.377\.037\.562\.670\.781\.092\.010\.69\.544\.02\.215\.611\.519\.280\.3172977\.575\.040\.769\.569\.080\.094\.014\.612\.439\.06\.516\.711\.120\.288\.251177\.458\.025\.768\.671\.581\.092\.014\.016\.938\.03\.226\.112\.221\.878\.787576\.145\.031\.266\.768\.260\.078\.014\.516\.084\.01\.314\.89\.921\.250\.463976\.654\.019\.563\.463\.858\.078\.014\.315\.984\.09\.119\.412\.121\.161\.774875\.953\.036\.771\.263\.377\.079\.016\.716\.685\.010\.462\.812\.519\.659\.6159074\.653\.034\.271\.563\.672\.080\.015\.016\.682\.010\.663\.312\.621\.560\.3129073\.764\.017\.361\.061\.558\.584\.013\.311\.279\.07\.761\.211\.227\.655\.6151075\.768\.015\.564\.160\.975\.083\.013\.611\.182\.04\.962\.611\.226\.0100\.0