大型语言模型持续微调中的未知预训练梯度估计与正交化

# Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models
Source: [https://arxiv.org/html/2609.30935](https://arxiv.org/html/2609.30935)
Bing WangAffiliation:College of Computer Science and Technology, Jilin UniversityAffiliation:Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin UniversityEmail:[wangbing1416@gmail\.com](mailto:)Changchun LiAffiliation:College of Computer Science and Technology, Jilin UniversityAffiliation:Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin UniversityEmail:[changchunli93@gmail\.com](mailto:)Xin\-Qiang CaiLin Yuanbo WuAffiliation:University of WarwickXiming Li††thanks:Corresponding authorAffiliation:College of Computer Science and Technology, Jilin UniversityAffiliation:Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin UniversityAffiliation:RIKEN Center for Advanced Intelligence ProjectGang Niu, Masashi SugiyamaAffiliation:RIKEN Center for Advanced Intelligence ProjectAffiliation:Graduate School of Frontier Sciences, University of Tokyo

###### Abstract

Continual fine\-tuning is essential for large language models \(LLMs\) to dynamically adapt to real\-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs’ general\-purpose knowledge\. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine\-tuning tasks, they fundamentally fail to preserve pre\-training LLMs’ inherent general\-purpose knowledge because the originaldata and gradients of off\-the\-shelf pre\-training LLMs required by these methods are strictly unknown and highly diverse\. To bridge this critical gap, we proposeEoupCT, a novel framework designed toEstimate and Orthogonalize Unknown Pre\-training gradients for Continual LLM fine\-Tuning\. Specifically,EoupCTestimates pre\-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel\-Softmax relaxation\. Furthermore, we formulate a multi\-objective optimization problem and introduce a first\-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre\-training gradients\. Extensive experiments across multiple LLMs demonstrate thatEoupCTeffectively preserves both task\-specific proficiency and inherent general\-purpose knowledge, successfully mitigating the catastrophic forgetting\.

## 1Introduction

Recently, the community has witnessed the rapid development oflarge language models\(LLMs\) in solving a variety of downstream tasks,*e\.g\.,*scientific Q&A\[[26](https://arxiv.org/html/2609.30935#bib.bib16)\]and mathematical reasoning\[[4](https://arxiv.org/html/2609.30935#bib.bib18),[35](https://arxiv.org/html/2609.30935#bib.bib17)\]\. However, in dynamic real\-world environments, the ceaseless influx of knowledge across emerging tasks makes it difficult for off\-the\-shelf pre\-training LLMs to timely adapt to new information\. To bridge this gap,continual LLM fine\-tuningon a stream of new tasks presents a pivotal solution\[[3](https://arxiv.org/html/2609.30935#bib.bib1),[29](https://arxiv.org/html/2609.30935#bib.bib13)\]\.

Despite the continuous acquisition of new task\-specific knowledge, the primary challenge in continual LLM fine\-tuning remainscatastrophic forgetting, that is, training on new tasks leads to a sustained performance decline on previously learned tasks and a degradation of the pre\-training LLM’s inherent general\-purpose knowledge\[[20](https://arxiv.org/html/2609.30935#bib.bib5),[18](https://arxiv.org/html/2609.30935#bib.bib6),[23](https://arxiv.org/html/2609.30935#bib.bib31),[27](https://arxiv.org/html/2609.30935#bib.bib34)\]\. To address this issue, a typical solution in the continual learning community is theorthogonal gradient optimizationapproach\[[31](https://arxiv.org/html/2609.30935#bib.bib7),[29](https://arxiv.org/html/2609.30935#bib.bib13),[17](https://arxiv.org/html/2609.30935#bib.bib21)\]\. Generally, this method involves storing the gradients of previously learned tasks and projecting the training gradients of the new task onto the orthogonal subspace spanned by these stored gradients, thereby mitigating interference between parameter updates for different tasks\. For instance,orthogonal low\-rank adaptation\(OLoRA\)\[[31](https://arxiv.org/html/2609.30935#bib.bib7)\]allocates a distinct set of LoRA parameters\[[7](https://arxiv.org/html/2609.30935#bib.bib3)\]to each new task and enforces the orthogonality across the parameters of different task modules\.

Although these orthogonalization methods are effective in mitigating catastrophic forgetting across various downstream tasks,they fail to account for the preservation of the pre\-training LLMs’ inherent general\-purpose knowledge,*e\.g\.,*commonsense knowledge and logic reasoning\. In practice, maintaining pre\-trained general\-purpose knowledge is also computationally or architecturally infeasible for existing methods\. Specifically, asthe training data and gradients of off\-the\-shelf pre\-training LLMs are typically unknown, employing techniques such as gradient orthogonalization is rendered unrealistic\. Furthermore,the tasks involved in pre\-training LLM training are highly diverse\[[16](https://arxiv.org/html/2609.30935#bib.bib36)\], even if recent techniques could estimate the underlying pre\-training gradients by generating pseudo training data\[[2](https://arxiv.org/html/2609.30935#bib.bib9),[10](https://arxiv.org/html/2609.30935#bib.bib8),[21](https://arxiv.org/html/2609.30935#bib.bib29)\], identifying which gradients are most beneficial remains a critical challenge\.

To address these challenges, we propose a novel method toEstimate and Orthogonalize Unknown Pre\-training gradients for Continual LLM fine\-Tuning, namelyEoupCT\. Specifically,EoupCTinvolves two key steps\.First, we estimate pre\-training gradients by generating pseudo data\[[2](https://arxiv.org/html/2609.30935#bib.bib9),[38](https://arxiv.org/html/2609.30935#bib.bib28)\]through a learnable soft prompt\[[13](https://arxiv.org/html/2609.30935#bib.bib19)\]utilized as the prefix to directly prompt the pre\-training LLM\. Since the direct auto\-regressive generation of discrete tokens of the pseudo data is non\-differentiable w\.r\.t\. the soft prompt, we apply a Gumbel\-Softmax relaxation\[[9](https://arxiv.org/html/2609.30935#bib.bib15)\]to derive continuous token representations\.Second, we formulate a multi\-objective optimization problem to jointly optimize the soft prompt and the task\-specific LLM parameters\. Specifically, to optimize the LLM parameters, we not only perform continual fine\-tuning on the new tasks but also enforce orthogonality between task\-specific gradients, as well as between new task gradients and the estimated pre\-training gradients\. To optimize the soft prompt, we aim to generate pseudo data that is most susceptible to forgetting for each new task, while ensuring orthogonality among the estimated pre\-training gradients to represent diverse pre\-training tasks\. To solve this complex multi\-objective optimization problem, we further propose a first\-order efficient Pareto optimizer, avoiding the computational burden of solving second\-order optimization problems\.

To evaluate the performance ofEoupCT, we sample 15 natural language tasks from theSuperNIdataset\[[33](https://arxiv.org/html/2609.30935#bib.bib12)\]\. We evaluate task\-specific proficiency using theSuperNItest subset and assess general\-purpose knowledge via theMMLUbenchmark\[[6](https://arxiv.org/html/2609.30935#bib.bib4)\]\. Generally,EoupCTachieves a 3\.5% improvement in average accuracy and a 3\.7% reduction in forgetting rate against SOTA results across 6 different pre\-training LLMs, demonstrating that our approach consistently mitigates the catastrophic forgetting\.Our source code is released in[https://github\.com/wangbing1416/EoupCT](https://github.com/wangbing1416/EoupCT)\.

In summary, our contributions can be summarized as the following three\-folds:

- •We observe that while orthogonal gradient projection approaches mitigate interference between fine\-tuning tasks, they struggle to preserve the pre\-training LLMs’ general\-purpose knowledge, primarily due to the unknown and diverse nature of the pre\-training gradients\.
- •To address unknown and diverse pre\-training gradients during continual fine\-tuning, we proposeEoupCT, which estimates and orthogonalizes these pre\-training gradients\.
- •Extensive experiments demonstrate thatEoupCTsignificantly preserves both task\-specific proficiency and general\-purpose knowledge, thereby mitigating the catastrophic forgetting\.

## 2Related Work

In this section, we briefly summarize the related literature about recent continual LLM fine\-tuning methods\. Generally, existing research primarily mitigates the catastrophic forgetting issue during the continual LLM fine\-tuning through several series of methods:orthogonal gradient projection, which restricts parameter updates to lie within non\-interfering parameter directions\[[31](https://arxiv.org/html/2609.30935#bib.bib7),[37](https://arxiv.org/html/2609.30935#bib.bib22),[17](https://arxiv.org/html/2609.30935#bib.bib21),[29](https://arxiv.org/html/2609.30935#bib.bib13)\]\.Data rehearsalinvolves continuously storing training samples from historical tasks and training them concurrently with the new task to mitigate their catastrophic forgetting\[[24](https://arxiv.org/html/2609.30935#bib.bib30),[32](https://arxiv.org/html/2609.30935#bib.bib27),[8](https://arxiv.org/html/2609.30935#bib.bib26),[5](https://arxiv.org/html/2609.30935#bib.bib38)\]\. For example, self\-synthesized rehearsal \(SSR\)\[[8](https://arxiv.org/html/2609.30935#bib.bib26),[30](https://arxiv.org/html/2609.30935#bib.bib37)\]utilizes a pre\-training LLM to synthesize queries and employs a state\-of\-the\-art LLM to generate corresponding responses as historical task data, which is then utilized for retraining during new task iterations\. However, this approach remains unable to access pre\-training data for the purpose of experience replay\.Parameter isolation,*e\.g\.,*low\-rank adaptation\-based mixture\-of\-experts \(LoRAMoE\)\[[3](https://arxiv.org/html/2609.30935#bib.bib1),[19](https://arxiv.org/html/2609.30935#bib.bib33),[40](https://arxiv.org/html/2609.30935#bib.bib11),[36](https://arxiv.org/html/2609.30935#bib.bib39)\], which allocates independent parameters for each task to isolate the influence between them\[[14](https://arxiv.org/html/2609.30935#bib.bib10),[12](https://arxiv.org/html/2609.30935#bib.bib32),[28](https://arxiv.org/html/2609.30935#bib.bib35),[22](https://arxiv.org/html/2609.30935#bib.bib20),[39](https://arxiv.org/html/2609.30935#bib.bib40)\]\. While this approach decouples pre\-trained weights from new task parameters, it necessitates maintaining a number of historical LLMs for routing during inference\. Furthermore, achieving a balance between general\-purpose knowledge and task\-specific expertise remains challenging; as a unified model, even with parameter isolation, the integration of new tasks inevitably induces forgetting of general\-purpose knowledge\.

Among these, our work focuses on orthogonal gradient projection, a promising direction that ensures optimization directions do not interfere with general\-purpose knowledge in pre\-training weights\. However, existing methods face two critical limitations\.First, many current approaches do not operate directly on the gradients; instead, they often apply constraints to parameter weights,*e\.g\.,*OLoRA\[[31](https://arxiv.org/html/2609.30935#bib.bib7)\]or hidden feature subspaces,*e\.g\.,*InfLoRA\[[15](https://arxiv.org/html/2609.30935#bib.bib23)\], which may not optimally represent the dynamic optimization landscape\.Second, and more importantly, existing methods only consider the gradients of known sequential tasks within the training stream\[[29](https://arxiv.org/html/2609.30935#bib.bib13),[17](https://arxiv.org/html/2609.30935#bib.bib21),[34](https://arxiv.org/html/2609.30935#bib.bib24)\], which typically ignore the pre\-training gradients because the pre\-training data is unknown\. This neglect leads to the erosion of the LLM’s general\-purpose knowledge during continual adaptation\. To address these issues, we proposeEoupCTthat estimates the missing pre\-training gradients, thereby enabling a more comprehensive orthogonal projection that protects both general\-purpose and task\-specific knowledge\.

## 3Our ProposedEoupCT

In this section, we introduce our proposed methodEoupCTthat orthogonally optimize LLMs\.

Task definition of continual LLM fine\-tuning\.Letℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}denote a pre\-training LLM, which is pre\-trained on a set of datasets𝒯0\\mathcal\{T\}\_\{0\}encompassing general\-purpose knowledge\. The goal of continual fine\-tuning is to adapt this LLM to a sequence of downstream tasks\. Specifically, we consider a sequence of datasets\[𝒯1,𝒯2,…,𝒯T\]\[\\mathcal\{T\}\_\{1\},\\mathcal\{T\}\_\{2\},\\dots,\\mathcal\{T\}\_\{T\}\]representingTTdifferent tasks, where each task𝒯t=\{\(𝐱it,𝐲it\)\}i=1\|𝒯t\|\\mathcal\{T\}\_\{t\}=\\\{\(\\mathbf\{x\}\_\{i\}^\{t\},\\mathbf\{y\}\_\{i\}^\{t\}\)\\\}\_\{i=1\}^\{\|\\mathcal\{T\}\_\{t\}\|\}consists of input prompts𝐱it\\mathbf\{x\}\_\{i\}^\{t\}and target outputs𝐲it\\mathbf\{y\}\_\{i\}^\{t\}\. The LLM, parameterized by𝜽\\boldsymbol\{\\theta\}and initialized as𝜽0\\boldsymbol\{\\theta\}^\{0\}, is then fine\-tuned across this sequence by minimizing the empirical loss\. A critical issue in this process is the mitigation of catastrophic forgetting\. This necessitates the preservation of both the general\-purpose knowledge inherent inℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}, represented by𝒯0\\mathcal\{T\}\_\{0\}, and the task\-specific knowledge acquired from preceding tasks𝒯1:t−1\\mathcal\{T\}\_\{1:t\-1\}during the fine\-tuning of the current task dataset𝒯t\\mathcal\{T\}\_\{t\}\.

Formulation of orthogonal gradient projection methods\.To fundamentally mitigate the interference between the current task𝒯t\\mathcal\{T\}\_\{t\}and previous tasks𝒯1:t−1\\mathcal\{T\}\_\{1:t\-1\}, orthogonal gradient projection restricts parameter updates to the orthogonal subspace spanned by previous inputs\. Let𝜽\\boldsymbol\{\\theta\}denote the trainable LLM parameters\. During the fine\-tuning on𝒯t\\mathcal\{T\}\_\{t\}, we first compute the standard empirical gradient𝐠t=∇𝜽ℒ​\(𝒯t\)\\mathbf\{g\}\_\{t\}=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\(\\mathcal\{T\}\_\{t\}\)\. Instead of applying this gradient directly, we project it using an orthogonal projection matrix𝐇⟂\(𝒯1:t−1\)\\mathbf\{H\}\_\{\\perp\}\(\\mathcal\{T\}\_\{1:t\-1\}\), which projects𝐠t\\mathbf\{g\}\_\{t\}onto the null space of the gradients𝐠1:t−1\\mathbf\{g\}\_\{1:t\-1\}from all previous data𝒯1:t−1\\mathcal\{T\}\_\{1:t\-1\}\. The parameter update rule is thus formulated as𝜽←𝜽−η𝐠t𝐇⟂\(𝒯1:t−1\)\\boldsymbol\{\\theta\}\\leftarrow\\boldsymbol\{\\theta\}\-\\eta\\mathbf\{g\}\_\{t\}\\mathbf\{H\}\_\{\\perp\}\(\\mathcal\{T\}\_\{1:t\-1\}\), whereη\\etais the learning rate\. By mathematically ensuring that the projected gradient yields zero activation for any historical input𝐱∈𝒯1:t−1\\mathbf\{x\}\\in\\mathcal\{T\}\_\{1:t\-1\}, this method theoretically guarantees that the new parameter updates act as a null operator on previously acquired knowledge, therefore preventing catastrophic forgetting\.

### 3\.1Method Overview

Although the orthogonal gradient projection methods prevent interference between the gradients of the current task𝒯t\\mathcal\{T\}\_\{t\}and previous tasks𝒯1:t−1\\mathcal\{T\}\_\{1:t\-1\}, they fail to account for orthogonality relative to the gradients of the pre\-training task𝒯0\\mathcal\{T\}\_\{0\}, as the latter remainunknown\. Therefore, it is challenging to enforce orthogonality between the gradients of new tasks and those of pre\-training tasks, which leads to a degradation of general\-purpose knowledge\. To address this, the basic idea ofEoupCTis toestimate pre\-training gradients from the pre\-training LLMℱ𝛉0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}, and orthogonally optimize the gradients of tasks𝒯1:T\\mathcal\{T\}\_\{1:T\}against the pre\-training gradients\. Specifically,EoupCTis an end\-to\-end training strategy that consists of two pivotal modules:pre\-training gradient estimation, which directly prompts the pre\-training LLM to generate pseudo data and estimates pre\-training gradients, andPareto orthogonal optimization, which formulates a multi\-objective optimization problem to ensure that the gradients of new tasks remain orthogonal to the estimated pre\-training gradients\.

Formally, given an off\-the\-shelf pre\-training LLMℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}, and a trainable LLM parameterized by𝜽\\boldsymbol\{\\theta\},pre\-training gradient estimationintroduces a continuousLL\-length soft prompt𝐏t\\mathbf\{P\}\_\{t\}for the current task𝒯t\\mathcal\{T\}\_\{t\}into the pre\-training LLMℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}to generate pseudo data of the general\-purpose knowledge distribution𝒯0\\mathcal\{T\}\_\{\\text\{0\}\}\. Since discrete auto\-regressive generation is non\-differentiable during orthogonal optimization, we employ a Gumbel\-Softmax relaxation to generate a continuous, fully differentiable surrogate sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\. This continuous generation maintains an unbroken computational graph, which is mathematically essential for the subsequent backpropagation optimization of the prompt𝐏t\\mathbf\{P\}\_\{t\}\. Accordingly, we define a knowledge distillation lossℒpre​\(𝐒​\(𝐏t\)\)\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\)between the pre\-training and trained LLMs on the pseudo data, and estimate the pre\-training gradient𝐠pre​\(𝐏t\)=∇𝜽ℒpre\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\)=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\.

Given the estimated gradients,Pareto orthogonal optimizationformulates the continual LLM fine\-tuning process as an end\-to\-end multi\-objective optimization problem\. The objective is to jointly optimize the soft prompt𝐏t\\mathbf\{P\}\_\{t\}and the LLM parameters𝜽\\boldsymbol\{\\theta\}to balance new task plasticity and general\-purpose knowledge stability\. Specifically, to achieve this goal, we present the followingConstraints\. Given a new task𝒯t\\mathcal\{T\}\_\{t\},\[C1\] orthogonality:typically, addressing the forgetting requires ensuring gradient orthogonality not only between the new tasks during the optimization of𝜽\\boldsymbol\{\\theta\}, but also between the new tasks and the estimated pre\-training gradients\. Meanwhile, during the training of𝐏t\\mathbf\{P\}\_\{t\}, orthogonality must be kept between the current and previous estimated pre\-training gradients, to ensure that the pseudo data generated by different soft prompts represent distinct pre\-training tasks\.\[C2\] Vulnerability:we aim for the generated pseudo data sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)to effectively expose the general\-purpose knowledge that is most susceptible to being forgotten during the training of the current new task,*i\.e\.,*𝐦𝐚𝐱𝐏t⁡ℒnew​\(𝒯t\)\\boldsymbol\{\\max\}\_\{\\mathbf\{P\}\_\{t\}\}\\mathcal\{L\}\_\{\\text\{new\}\}\(\\mathcal\{T\}\_\{t\}\)\.\[C3\] Plasticity:given the generated pseudo data sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)and new task data𝒯t\\mathcal\{T\}\_\{t\}, we perform supervised fine\-tuning on𝜽\\boldsymbol\{\\theta\}using these datasets,*i\.e\.,*𝐦𝐢𝐧𝜽⁡ℒnew​\(𝒯t\),ℒpre​\(𝐒⁡\(𝐏t\)\)\\boldsymbol\{\\min\}\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{new\}\}\(\\mathcal\{T\}\_\{t\}\),\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\)\.

Figure 1:Framework ofEoupCTwith pre\-training gradient estimation and Pareto orthogonal optimization\.Finally, our formulation in Appendix[B\.1](https://arxiv.org/html/2609.30935#A2.SS1)demonstrates that the optimization of the aforementioned objective necessitates the computation of a second\-order Hessian matrix, rendering it computationally intractable\. Therefore, we propose a first\-order Pareto optimizer to efficiently address this multi\-objective optimization problem\. For clarity,the overall framework ofEoupCTis depicted in Fig\.[1](https://arxiv.org/html/2609.30935#S3.F1), and the complete algorithm ofEoupCTis described in Appendix[C](https://arxiv.org/html/2609.30935#A3)\.In the following subsections, we provide a detailed introduction to pre\-training gradient estimation, Pareto optimization objectives, and the efficient Pareto optimizer, respectively\.

### 3\.2Pre\-training Gradient Estimation

As motivated in Sec\.[1](https://arxiv.org/html/2609.30935#S1), approximating the orthogonal complement of the general\-purpose knowledge subspace requires access to the pre\-training data distribution𝒯0\\mathcal\{T\}\_\{\\text\{0\}\}, which is unknown during continual LLM fine\-tuning\. To address this bottleneck, we propose adifferentiable generationmechanism that treats the off\-the\-shelf pre\-training LLMℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}itself as an implicit generative model of𝒯0\\mathcal\{T\}\_\{\\text\{0\}\}\[[2](https://arxiv.org/html/2609.30935#bib.bib9),[38](https://arxiv.org/html/2609.30935#bib.bib28)\]\.

Specifically, for the current fine\-tuning task𝒯t\\mathcal\{T\}\_\{t\}, we introduce a learnableLL\-length soft prompt𝐏t∈ℝL×D\\mathbf\{P\}\_\{t\}\\in\\mathbb\{R\}^\{L\\times D\}, whereDDindicates the hidden size, that is prepended toℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}to steer it toward generating sequences of the general\-purpose knowledge distribution\. However, standard auto\-regressive decoding relies on discrete token selection,*e\.g\.,*top\-kksampling, which is non\-differentiable and would sever the gradient flow∇𝐏t\\nabla\_\{\\mathbf\{P\}\_\{t\}\}essential for the optimization of the prompt𝐏t\\mathbf\{P\}\_\{t\}\. To maintain a fully differentiable computational graph, we adopt the Gumbel\-Softmax relaxation\[[9](https://arxiv.org/html/2609.30935#bib.bib15)\]for differentiable decoding\. Let𝐖emb∈ℝ\|𝒱\|×D\\mathbf\{W\}\_\{\\text\{emb\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times D\}denote the token embedding matrix, where𝒱\\mathcal\{V\}is the token vocabulary\. We auto\-regressively generate a sequence ofMMsoft token embeddings𝐒⁡\(𝐏t\)=\[𝐞1,𝐞2,…,𝐞M\]\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)=\[\\mathbf\{e\}\_\{1\},\\mathbf\{e\}\_\{2\},\\ldots,\\mathbf\{e\}\_\{M\}\]\. At generation stepm∈\{1,…,M\}m\\in\\\{1,\\ldots,M\\\}, the pre\-training LLMℱ𝜽0\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}processes the sequence\[𝐏t,𝐞1,…,𝐞m−1\]\[\\mathbf\{P\}\_\{t\},\\mathbf\{e\}\_\{1\},\\ldots,\\mathbf\{e\}\_\{m\-1\}\]to produce the next\-token logits𝐳m∈ℝ\|𝒱\|\\mathbf\{z\}\_\{m\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}\. Instead of selecting a discrete token from𝐳m\\mathbf\{z\}\_\{m\}, we compute a continuous probability vector via the Gumbel\-Softmax relaxation as

πm,i=exp⁡\(\(zm,i\+εm,i\)/τ\)∑j=1\|𝒱\|exp⁡\(\(zm,j\+εm,j\)/τ\),\\pi\_\{m,i\}=\\frac\{\\exp\\big\(\(z\_\{m,i\}\+\\varepsilon\_\{m,i\}\)/\\tau\\big\)\}\{\\sum\_\{j=1\}^\{\|\\mathcal\{V\}\|\}\\exp\\big\(\(z\_\{m,j\}\+\\varepsilon\_\{m,j\}\)/\\tau\\big\)\},\(1\)whereτ\>0\\tau\>0is the temperature parameter \(asτ→0\\tau\\to 0,𝝅m\\boldsymbol\{\\pi\}\_\{m\}converges to a one\-hot vector\), andεm,i=−log⁡\(−log⁡\(um,i\)\)\\varepsilon\_\{m,i\}=\-\\log\(\-\\log\(u\_\{m,i\}\)\)withum,i∼Uniform​\(0,1\)u\_\{m,i\}\\sim\\text\{Uniform\}\(0,1\)are Gumbel noise samples\. The corresponding soft token embedding is obtained by computing the expected embedding under𝝅m\\boldsymbol\{\\pi\}\_\{m\}:

𝐞m=𝝅m​𝐖emb=∑i=1\|𝒱\|πm,i​\(𝐖emb\)i∈ℝD,\\mathbf\{e\}\_\{m\}=\\boldsymbol\{\\pi\}\_\{m\}\\mathbf\{W\}\_\{\\text\{emb\}\}=\\sum\\nolimits\_\{i=1\}^\{\|\\mathcal\{V\}\|\}\\pi\_\{m,i\}\(\\mathbf\{W\}\_\{\\text\{emb\}\}\)\_\{i\}\\in\\mathbb\{R\}^\{D\},\(2\)which is appended to the prefix for the next decoding step\. AfterMMiterations, this process yields the pseudo data𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\. Accordingly, every operation from the initial soft prompt𝐏t\\mathbf\{P\}\_\{t\}to the final sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)is strictly continuous and differentiable, making𝐏t\\mathbf\{P\}\_\{t\}optimizable\. Accordingly, we define a knowledge distillation loss based on themean squared errorbetween the latent representations of the pre\-training and trained LLMs𝜽0\\boldsymbol\{\\theta\}^\{0\}and𝜽\\boldsymbol\{\\theta\}on the pseudo data:

ℒpre​\(𝜽,𝐒⁡\(𝐏t\)\)=ℓMSE​\(ℱ𝜽0​\(𝐒⁡\(𝐏t\)\),ℱ𝜽​\(𝐒⁡\(𝐏t\)\)\),\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\)=\\ell\_\{\\text\{MSE\}\}\\left\(\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}^\{0\}\}\\big\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\\big\),\\;\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}\}\\big\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\\big\)\\right\),\(3\)with the corresponding estimated pre\-training gradient𝐠pre​\(𝐏t\)=∇𝜽ℒpre\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\)=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\.

### 3\.3Multi\-Objective Formulation of Pareto Orthogonal Optimization

Given the pseudo data sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)in Sec\.[3\.2](https://arxiv.org/html/2609.30935#S3.SS2)and the new task data𝒯t\\mathcal\{T\}\_\{t\}, Pareto orthogonal optimization formalizes the three conditions\[C1\-C3\]outlined in Sec\.[3\.1](https://arxiv.org/html/2609.30935#S3.SS1)into a unifiedmulti\-objective optimizationproblem to jointly optimize the soft prompt𝐏t\\mathbf\{P\}\_\{t\}and the LLM parameter𝜽\\boldsymbol\{\\theta\}\. Specifically, we formulate the following objectives\.

⋆\\boldsymbol\{\\star\}Plasticity objective\[C3\]\.To learn the new task𝒯t\\mathcal\{T\}\_\{t\}, we minimize the standard supervised fine\-tuning loss over the task dataset\. Formally, the plasticity objective is defined as the negative log\-likelihood:

ℒnew\(𝜽;𝒯t\)=−1\|𝒯t\|∑i=1\|𝒯t\|∑k=1\|𝐲i\|logP𝜽\(yi,kt∣𝐱it,𝐲i,<kt\),\(𝐱it,𝐲it\)∈𝒯t,\\mathcal\{L\}\_\{\\text\{new\}\}\(\\boldsymbol\{\\theta\};\\mathcal\{T\}\_\{t\}\)=\-\\frac\{1\}\{\|\\mathcal\{T\}\_\{t\}\|\}\\sum\\nolimits\_\{i=1\}^\{\|\\mathcal\{T\}\_\{t\}\|\}\\sum\\nolimits\_\{k=1\}^\{\|\\mathbf\{y\}\_\{i\}\|\}\\log P\_\{\\boldsymbol\{\\theta\}\}\\left\(y\_\{i,k\}^\{t\}\\mid\\mathbf\{x\}\_\{i\}^\{t\},\\mathbf\{y\}\_\{i,<k\}^\{t\}\\right\),\\quad\\left\(\\mathbf\{x\}\_\{i\}^\{t\},\\mathbf\{y\}\_\{i\}^\{t\}\\right\)\\in\\mathcal\{T\}\_\{t\},\(4\)whereP𝜽​\(yi,kt∣𝐱it,𝐲i,<kt\)P\_\{\\boldsymbol\{\\theta\}\}\\big\(y\_\{i,k\}^\{t\}\\mid\\mathbf\{x\}\_\{i\}^\{t\},\\mathbf\{y\}\_\{i,<k\}^\{t\}\\big\)denotes the probability of generating thekk\-th tokenyi,kty\_\{i,k\}^\{t\}conditioned on the input𝐱it\\mathbf\{x\}\_\{i\}^\{t\}\. This yields the task\-specific gradient𝐠new=∇𝜽ℒnew\\mathbf\{g\}\_\{\\text\{new\}\}=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{new\}\}\.

⋆\\boldsymbol\{\\star\}Stability objective\[C3\]\.We minimizeℒpre​\(𝜽,𝐒⁡\(𝐏t\)\)\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\)in Eq\. \([3](https://arxiv.org/html/2609.30935#S3.E3)\) to keep the general\-purpose knowledge stability\. To ensure that the pseudo data in Sec\.[3\.2](https://arxiv.org/html/2609.30935#S3.SS2)is informative and that the parameter update is orthogonal across previous gradients, we further impose the following three constraints\.

⋆\\boldsymbol\{\\star\}Vulnerability\[C2\]\.The soft prompt𝐏t\\mathbf\{P\}\_\{t\}must dynamically expose general\-purpose knowledge that is most susceptible to being forgotten during the current task𝒯t\\mathcal\{T\}\_\{t\}\. Instead of arbitrary estimation, we optimize𝐏t\\mathbf\{P\}\_\{t\}by maximizingℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}on avirtually updated𝜽virt=𝜽−α​∇𝜽ℒnew\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}=\\boldsymbol\{\\theta\}\-\\alpha\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{new\}\}\. This virtual step simulates the destructive impact of the new task’s gradient, forcing𝐏t\\mathbf\{P\}\_\{t\}to generate pseudo data that specifically exposes the general\-purpose knowledge boundaries most damaged by the current task\.

⋆\\boldsymbol\{\\star\}Pre\-training & new gradients orthogonality\[C1\]\.To resolve the forgetting, the new task gradient must not interfere with the currently estimated pre\-training gradient𝐠pre​\(𝐏t\)\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\)\. This is inherently formulated as: finding an update direction that minimizes the new task loss without increasing the loss on the optimal𝐒⁡\(𝐏t∗\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\), effectively enforcing orthogonality when gradient conflicts arise\.

⋆\\boldsymbol\{\\star\}Pre\-training & pre\-training gradients orthogonality\[C1\]\.To ensure diverse coverage of the general\-purpose knowledge subspace, current pseudo data must not redundantly overlap with previous generations\. We penalize the squared cosine similarity between the current protection gradient𝐠pre​\(𝐏t\)\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\)and historical pre\-training gradients\. This explicitly drives⟨𝐠pre\(𝐏t\),𝐠pre\(𝐏1:t−1\)⟩→0\\langle\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\),\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{1:t\-1\}\)\\rangle\\to 0, protecting distinct regions of the pre\-training distribution\.

⋆\\boldsymbol\{\\star\}New & new gradients orthogonality\[C1\]\.To prevent forgetting of previous downstream tasks, the actual parameter update𝜽t\+1−𝜽t\\boldsymbol\{\\theta\}\_\{t\+1\}\-\\boldsymbol\{\\theta\}\_\{t\}must reside entirely within the orthogonal null space ofℳnew=\{𝐠new1,𝐠new2,…,𝐠newt−1\}\\mathcal\{M\}\_\{\\text\{new\}\}=\\\{\\mathbf\{g\}\_\{\\text\{new\}\}^\{1\},\\mathbf\{g\}\_\{\\text\{new\}\}^\{2\},\\ldots,\\mathbf\{g\}\_\{\\text\{new\}\}^\{t\-1\}\\\}\. This mathematically guarantees⟨𝜽t\+1−𝜽t,𝐠newi⟩=0\\langle\\boldsymbol\{\\theta\}\_\{t\+1\}\-\\boldsymbol\{\\theta\}\_\{t\},\\mathbf\{g\}\_\{\\text\{new\}\}^\{i\}\\rangle=0for all historical gradients𝐠newi∈ℳnew\\mathbf\{g\}\_\{\\text\{new\}\}^\{i\}\\in\\mathcal\{M\}\_\{\\text\{new\}\}, ensuring absolute zero interference\.

Integrating these objectives and constraints, the entire continual LLM fine\-tuning process is cast as the following unified multi\-objective optimization formulation:

𝐦𝐢𝐧𝜽\\displaystyle\\mathop\{\\boldsymbol\{\\min\}\}\\limits\_\{\\boldsymbol\{\\theta\}\}\\\(ℒnew​\(𝜽,𝒯t\)⏟New task plasticity,ℒpre​\(𝜽,𝐒⁡\(𝐏t∗\)\)⏟General\-purpose stability\)\\displaystyle\\Big\(\\underbrace\{\\mathcal\{L\}\_\{\\text\{new\}\}\(\\boldsymbol\{\\theta\};\\,\\mathcal\{T\}\_\{t\}\)\}\_\{\\text\{New task plasticity\}\},\\;\\;\\underbrace\{\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\\big\)\}\_\{\\text\{General\-purpose stability\}\}\\Big\)\(5\)s\.t\.𝐏t∗=𝐚𝐫𝐠𝐦𝐚𝐱𝐏t\[ℒpre​\(𝜽virt,𝐒⁡\(𝐏t\)\)⏟Vulnerability−λ​∑i=1t−1cos2⁡\(𝐠pre​\(𝐏t\),𝐠pre​\(𝐏i\)\)⏟Pre\-training & pre\-training gradients orthogonality\],\\displaystyle\\mathbf\{P\}\_\{t\}^\{\*\}=\\boldsymbol\{\\arg\}\\mathop\{\\boldsymbol\{\\max\}\}\\limits\_\{\\mathbf\{P\}\_\{t\}\}\\Big\[\\underbrace\{\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\\big\)\}\_\{\\text\{Vulnerability\}\}\-\\underbrace\{\\lambda\\sum\\nolimits\_\{i=1\}^\{t\-1\}\\cos^\{2\}\\\!\\big\(\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\),\\,\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{i\}\)\\big\)\}\_\{\\text\{Pre\-training \\& pre\-training gradients orthogonality\}\}\\Big\],\(6\)⟨𝜽t\+1−𝜽t,∇𝜽ℒpre​\(𝜽,𝐒⁡\(𝐏t∗\)\)⟩≤0⏟Pre\-training & new gradients orthogonality,⟨𝜽t\+1−𝜽t,𝐠new1:t−1⟩=0⏟New & new gradients orthogonality\.\\displaystyle\\underbrace\{\\left\\langle\\boldsymbol\{\\theta\}\_\{t\+1\}\-\\boldsymbol\{\\theta\}\_\{t\},\\;\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\\big\)\\right\\rangle\\leq 0\}\_\{\\text\{Pre\-training \\& new gradients orthogonality\}\},\\;\\underbrace\{\\left\\langle\\boldsymbol\{\\theta\}\_\{t\+1\}\-\\boldsymbol\{\\theta\}\_\{t\},\\;\\mathbf\{g\}\_\{\\text\{new\}\}^\{1:\{t\-1\}\}\\right\\rangle=0\}\_\{\\text\{New \\& new gradients orthogonality\}\}\.\(7\)This formulation encodes the goals of continual fine\-tuning into a tractable mathematical structure\. We next describe an efficient first\-order optimizer for this second\-order optimization problem\.

### 3\.4Efficient Pareto Orthogonal Optimizer

Directly solving the multi\-objective optimization problem defined in Eqs\. \([5](https://arxiv.org/html/2609.30935#S3.E5)\), \([6](https://arxiv.org/html/2609.30935#S3.E6)\), and \([7](https://arxiv.org/html/2609.30935#S3.E7)\) is computationally intractable\. This stems from the natural trade\-off between vulnerability and stability; furthermore, explicitly coupling adversarial prompt optimization with new\-task gradients would necessitate the computation of second\-order Hessian matrices\. Therefore, we resolve this by decoupling the optimizer into four efficient first\-order steps, corresponding directly to our defined constraints\. The complete algorithm of this efficient optimizer is shown in Appendix[C](https://arxiv.org/html/2609.30935#A3)\.

Step 1:To satisfy both the vulnerability and the pre\-training & pre\-training gradients orthogonality in Eq\. \([6](https://arxiv.org/html/2609.30935#S3.E6)\), we optimize the soft prompt𝐏t\\mathbf\{P\}\_\{t\}\. A naïve approach, maximizingℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}at the current weights𝜽t\\boldsymbol\{\\theta\}\_\{t\}, only finds knowledge that the current model already struggles to preserve, completely ignoring the destructive effect of new tasks\. The correct objective is to maximize the gradient conflict,*i\.e\.,*𝐦𝐚𝐱𝐏t⁡⟨−𝐠new,∇𝜽ℒpre⟩\\boldsymbol\{\\max\}\_\{\\mathbf\{P\}\_\{t\}\}\\langle\-\\mathbf\{g\}\_\{\\text\{new\}\},\\,\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\rangle; however, computing its gradient w\.r\.t\.𝐏t\\mathbf\{P\}\_\{t\}requires an expensive Hessian matrix\. We circumvent this via thevirtual step approximation: by a first\-order Taylor expansion around the current weights𝜽t\\boldsymbol\{\\theta\}\_\{t\}, taking a virtual step𝜽virt=𝜽t−α​𝐠new\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}=\\boldsymbol\{\\theta\}\_\{t\}\-\\alpha\\,\\mathbf\{g\}\_\{\\text\{new\}\}changes the loss as

ℒpre​\(𝜽virt\)≈ℒpre​\(𝜽t\)−α⁡⟨𝐠new,∇𝜽ℒpre​\(𝜽t\)⟩\.\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}\)\\approx\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{t\}\)\-\\alpha\\big\\langle\\mathbf\{g\}\_\{\\text\{new\}\},\\,\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{t\}\)\\big\\rangle\.\(8\)Hence, maximizingℒpre​\(𝜽virt,𝐒⁡\(𝐏t\)\)\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\)over𝐏t\\mathbf\{P\}\_\{t\}is provably equivalent to maximizing the gradient conflict\. Simultaneously, to enforce the orthogonality among estimated pre\-training distributions, we subtract the historical independence penaltyλ​∑i=1t−1cos2⁡\(𝐠pre​\(𝐏t\),𝐠pre​\(𝐏i\)\)\\lambda\\sum\_\{i=1\}^\{t\-1\}\\cos^\{2\}\\\!\\big\(\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{t\}\),\\,\\mathbf\{g\}\_\{\\text\{pre\}\}\(\\mathbf\{P\}\_\{i\}\)\\big\)from this objective during the backpropagation of𝐏t\\mathbf\{P\}\_\{t\}\. Once the optimal prompt𝐏t∗\\mathbf\{P\}\_\{t\}^\{\*\}is obtained via this joint optimization, the true protection gradient is computed as𝐠pre∗=∇𝜽ℒpre​\(𝜽t,𝐒⁡\(𝐏t∗\)\)\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\}=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\}\_\{t\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\\big\)\.

Step 2:With both𝐠new\\mathbf\{g\}\_\{\\text\{new\}\}and𝐠pre∗\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\}in hand, we strictly enforce the orthogonality with new gradient in Eq\. \([7](https://arxiv.org/html/2609.30935#S3.E7)\) by constructing the null\-space projection matrix of the historical new\-task gradientsℳnew\\mathcal\{M\}\_\{\\text\{new\}\}:

𝚷new=𝐈−𝐌new​\(𝐌new⊤​𝐌new\)−1​𝐌new⊤,\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}=\\mathbf\{I\}\-\\mathbf\{M\}\_\{\\text\{new\}\}\\big\(\\mathbf\{M\}\_\{\\text\{new\}\}^\{\\top\}\\mathbf\{M\}\_\{\\text\{new\}\}\\big\)^\{\-1\}\\mathbf\{M\}\_\{\\text\{new\}\}^\{\\top\},\(9\)where𝐌new\\mathbf\{M\}\_\{\\text\{new\}\}denotes the matrix by concatenating historical task gradientsℳnew\\mathcal\{M\}\_\{\\text\{new\}\}as column vectors, and𝐈\\mathbf\{I\}is the identity matrix\. Then, we project both raw gradients into this historically safe subspace:

𝐠~new=𝚷new​𝐠new,𝐠~pre=𝚷new​𝐠pre∗\.\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}=\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}\\,\\mathbf\{g\}\_\{\\text\{new\}\},\\quad\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}=\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}\\,\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\}\.\(10\)This guarantees that any subsequent parameter update built from these projected gradients will have exactly zero inner product with any previously learned task\.

Step 3:To satisfy the pre\-training & new gradients orthogonality in Eq\. \([7](https://arxiv.org/html/2609.30935#S3.E7)\), we must ensure the new task update does not increase the loss on the generated pseudo data\. We compute the alignment score, if⟨𝐠~new,𝐠~pre⟩≥0\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\,\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\\geq 0, the two gradients are naturally aligned\. If⟨𝐠~new,𝐠~pre⟩<0\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\,\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle<0, updating along𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}would damage the adversarially identified general knowledge\. We resolve this by applying a Pareto\-optimal projection that removes the conflicting component of𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}along the direction of𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}:

𝐠~new∗=𝐠~new−⟨𝐠~new,𝐠~pre⟩‖𝐠~pre‖2​𝐠~pre\.\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}=\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\-\\frac\{\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\,\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\}\{\\\|\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\\|^\{2\}\}\\,\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\.\(11\)A direct calculation confirms⟨𝐠~new∗,𝐠~pre⟩=0\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\},\\,\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle=0, mathematically guaranteeing that the modified task gradient avoids any interference with the currently estimated pre\-training gradient\.

Step 4:With the interference\-free gradients established, the LLM parameters are updated as

𝜽τ\+1=𝜽τ−η⁡\(𝐠~new∗\+𝐠~pre\),\\boldsymbol\{\\theta\}\_\{\\tau\+1\}=\\boldsymbol\{\\theta\}\_\{\\tau\}\-\\eta\\big\(\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}\+\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\big\),\(12\)whereη\>0\\eta\>0represents the learning rate\. Because𝐠~new∗\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}and𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}reside in the null space ofℳnew\\mathcal\{M\}\_\{\\text\{new\}\}\(by Step 2\) and do not conflict with each other \(by Step 3\), this update direction𝜽τ\+1−𝜽τ\\boldsymbol\{\\theta\}\_\{\\tau\+1\}\-\\boldsymbol\{\\theta\}\_\{\\tau\}simultaneously satisfies Eq\. \([7](https://arxiv.org/html/2609.30935#S3.E7)\), efficiently breaking the catastrophic forgetting\.

## 4Experimental Evaluation

In this section, we present the experimental performance of ourEoupCTmethod\.

Table 1:Results ofEoupCTunder two task types: task\-specific knowledge bySuperNIand general\-purpose knowledge byMMLU\(Rou\.↑\\uparrow, Acc\.↑\\uparrow, Fgt\.↓\\downarrow\)\. The bold results represent the best scores\.SuperNIMMLU\(average three task orders\)MethodOrder 1Order 2Order 3STEMHumanityOtherRou\.Fgt\.Rou\.Fgt\.Rou\.Fgt\.Acc\.Fgt\.Acc\.Fgt\.Acc\.Fgt\.Qwen3\-8B\+ LoRA50\.17\.748\.29\.849\.98\.562\.63\.571\.37\.672\.54\.4\+ LoRAMoE50\.86\.349\.48\.748\.98\.461\.64\.571\.37\.572\.04\.9\+ OLoRA51\.26\.747\.49\.450\.17\.663\.32\.972\.06\.972\.94\.0\+ GainLoRA50\.06\.649\.78\.049\.28\.162\.73\.472\.76\.272\.44\.4\+ CLoRA50\.97\.249\.69\.149\.27\.663\.32\.872\.56\.472\.24\.7\+EoupCT52\.94\.452\.16\.051\.55\.764\.71\.475\.93\.075\.51\.4Qwen3\-4B\+ LoRA45\.113\.740\.816\.748\.69\.256\.44\.564\.08\.664\.25\.7\+ LoRAMoE48\.68\.843\.213\.948\.18\.857\.43\.665\.17\.465\.74\.1\+ OLoRA48\.99\.046\.010\.647\.99\.658\.12\.965\.47\.165\.93\.9\+ GainLoRA48\.49\.044\.912\.147\.89\.655\.95\.064\.87\.865\.34\.6\+ CLoRA47\.89\.945\.411\.347\.210\.057\.73\.364\.18\.464\.35\.6\+EoupCT50\.76\.050\.36\.150\.56\.260\.20\.767\.45\.268\.01\.9Llama3\-8B\+ LoRA35\.522\.033\.923\.740\.816\.839\.07\.058\.011\.163\.19\.3\+ LoRAMoE42\.914\.736\.720\.537\.520\.240\.35\.759\.110\.064\.38\.1\+ OLoRA44\.612\.843\.712\.845\.612\.243\.62\.462\.27\.065\.76\.7\+ GainLoRA40\.214\.536\.818\.637\.417\.440\.95\.159\.010\.065\.37\.1\+ CLoRA41\.614\.737\.218\.241\.513\.542\.13\.962\.07\.064\.97\.5\+EoupCT51\.46\.549\.07\.850\.27\.445\.01\.064\.54\.669\.62\.8Llama3\-3B\+ LoRA37\.617\.739\.314\.443\.710\.738\.91\.756\.06\.159\.54\.2\+ LoRAMoE40\.514\.240\.116\.239\.416\.139\.41\.256\.06\.160\.83\.7\+ OLoRA41\.613\.536\.420\.036\.718\.339\.21\.455\.66\.560\.02\.9\+ GainLoRA38\.016\.338\.318\.143\.011\.638\.42\.255\.86\.359\.34\.3\+ CLoRA38\.715\.339\.215\.639\.913\.239\.41\.255\.46\.759\.83\.9\+EoupCT44\.010\.545\.19\.246\.58\.241\.4\-0\.857\.24\.961\.91\.7Gemma2\-9B\+ LoRA41\.217\.741\.316\.840\.019\.141\.711\.560\.713\.964\.312\.8\+ LoRAMoE48\.810\.236\.623\.045\.513\.746\.87\.465\.78\.968\.78\.4\+ OLoRA50\.08\.844\.914\.440\.717\.042\.910\.361\.812\.867\.79\.5\+ GainLoRA43\.312\.940\.717\.042\.214\.444\.38\.965\.69\.067\.99\.2\+ CLoRA42\.813\.340\.815\.542\.512\.745\.18\.164\.89\.864\.812\.3\+EoupCT53\.44\.853\.95\.554\.34\.552\.30\.971\.82\.771\.75\.4Gemma2\-2B\+ LoRA34\.720\.835\.520\.428\.527\.331\.08\.345\.111\.752\.914\.0\+ LoRAMoE36\.020\.934\.422\.238\.317\.233\.06\.348\.97\.959\.67\.3\+ OLoRA39\.716\.536\.219\.640\.315\.533\.16\.247\.98\.958\.08\.9\+ GainLoRA41\.714\.441\.414\.943\.412\.434\.84\.550\.16\.759\.47\.5\+ CLoRA37\.517\.739\.216\.838\.817\.134\.94\.447\.49\.358\.18\.8\+EoupCT46\.710\.348\.18\.450\.94\.638\.50\.854\.42\.563\.03\.9Experimental settings\.To evaluate the performance of our method, we conduct experiments usingSuperNI\[[33](https://arxiv.org/html/2609.30935#bib.bib12)\], a large\-scale multi\-task instruction tuning dataset\. Following[Zhao et al\. \[40\]](https://arxiv.org/html/2609.30935#bib.bib11), we select 15 tasks organized into 3 different sequential orderings for continual LLM fine\-tuning\. Upon the completion of each task’s fine\-tuning phase, we evaluate the LLM on the correspondingSuperNItest subsets to assess task\-specific performance\. Furthermore, we utilize 9 tasks across 3 categories from theMMLUdataset\[[6](https://arxiv.org/html/2609.30935#bib.bib4)\]to benchmark the preservation of the LLM’s general\-purpose knowledge\. In our implementation, we utilize 6 pre\-training LLMs across 3 distinct families, including Qwen3, Llama3, and Gemma2, employinglow\-rank adaptation\(LoRA\)\[[7](https://arxiv.org/html/2609.30935#bib.bib3)\]with a rank ofr=4r=4for parameter\-efficient fine\-tuning\. We compare our approach against 5 baselines, encompassing a variety of continual fine\-tuning strategies\. Detailed dataset statistics, descriptions of the LLMs and baseline methods, implementation specifics, and formal evaluation metrics are provided in Appendix[D](https://arxiv.org/html/2609.30935#A4)\.

### 4\.1Main Results

Table[1](https://arxiv.org/html/2609.30935#S4.T1)presents the overall performance ofEoupCTcompared with a diverse set of continual LLM fine\-tuning baselines across multiple LLM families and scales\. We conduct fine\-tuning across three distinct task sequences in Table[6](https://arxiv.org/html/2609.30935#A4.T6)generated using 3 different random seeds, reporting the task\-specific performance onSuperNIfor each individual ordering\. Meanwhile, the general\-purpose knowledge performance onMMLUis reported as the average across these three task sequences\.

Generally, we can observe thatEoupCTconsistently achieves superior performance in both task\-specific learning and general\-purpose knowledge preservation\. OnSuperNI, our method yields the highest averageRougescores under all task orders while significantly reducing forgetting\. For instance, on Qwen3\-4B,EoupCTimproves theRougeof OLoRA from 48\.9 to 50\.7 under Order 1, and simultaneously reduces the forgetting from 9\.0 to 6\.0\. Similar improvements are consistently observed across different task sequences, indicating that the proposed method is robust to the order of incoming tasks\. More importantly,EoupCTdemonstrates a substantial advantage in mitigating catastrophic forgetting\. Across all model families, the forgetting values are consistently lower than those of competing methods\. This verifies that explicitly enforcing orthogonality with estimated pre\-training gradients effectively prevents destructive interference during continual updates\.

Meanwhile,EoupCTalso shows remarkable improvements in preserving general\-purpose knowledge onMMLU\. Compared to existing methods, which often suffer from severe degradation in general\-purpose knowledge,EoupCTachieves both higher accuracy and lower forgetting across all categories\. For example, on Gemma2\-9B,EoupCTimproves the STEM accuracy of OLoRA from 42\.9 to 52\.3 while reducing forgetting from 10\.3 to 0\.9\. Notably, these gains are consistently observed across different domains, including Humanity and Other, suggesting that the estimated pre\-training gradients effectively capture diverse general knowledge\.

### 4\.2Sensitivity Analysis

Figure 2:Sensitivity analysis of soft prompt lengthLLand surrogate sequence lengthMM\.We further investigate the sensitivity ofEoupCTto two key hyper\-parameters: the soft prompt lengthLLand the surrogate sequence lengthMM\. Specifically, we conduct experiments on Qwen3\-4B and Llama3\-3B, and report the performance in terms ofRougeon theSuperNIbenchmark under Order 1\. BothLLandMMare selected from the set\{4,8,16,32\}\\\{4,8,16,32\\\}to explore their impact on model performance\.

The results show a consistent trend across both LLMs\. WhenLLandMMare small,*e\.g\.,*4 or 8, the model exhibits relatively weaker performance, suggesting that limited prompt capacity and short generated sequences are insufficient to adequately cover diverse pre\-training knowledge\. AsLLandMMincrease, the performance improves steadily, indicating that a richer soft prompt and longer surrogate sequences enable more expressive and representative reconstruction of the underlying pre\-training distribution\. However, whenLLandMMbecome excessively large,*e\.g\.,*32, the performance does not continue to improve and may even slightly degrade\.

### 4\.3Ablation Experiments

We further conduct ablation studies to analyze the contribution of each component inEoupCT, as shown in Table[2](https://arxiv.org/html/2609.30935#S4.T2)\. Removing any component leads to a consistent degradation in both task\-specific performance and general knowledge preservation, highlighting the necessity of the full design\. In particular, removing thevulnerability objective,*i\.e\.,*w/o C2results in a noticeable drop inRougeand a sharp increase in forgetting\. For example, on Qwen3\-4B, theRougedecreases from 50\.7 to 47\.6 while the forgetting increases from 6\.0 to 10\.3, indicating that identifying the most vulnerable pre\-training knowledge is crucial for effective protection\.

Similarly,removing the orthogonality constraints between pre\-training gradients,*i\.e\.,*w/o P&POleads to further performance degradation\. On Llama3\-3B, theRougedrops from 44\.0 to 42\.1 and forgetting increases from 10\.5 to 12\.2, demonstrating that enforcing orthogonality with estimated pre\-training gradients plays a key role in preserving general\-purpose knowledge\. Whenremoving the orthogonality between new\-task and pre\-training gradients,*i\.e\.,*w/o N&PO, the LLM suffers from increased interference, resulting in higher forgetting across all settings\. Moreover,removing the orthogonality among new\-task gradients,*i\.e\.,*w/o N&NOleads to the most severe degradation, as it directly breaks the protection against inter\-task interference\. This confirms that maintaining orthogonality across sequential tasks is essential for stable continual learning\. Across all ablations, we also observe consistent declines onMMLUperformance, indicating that each component contributes to preserving general\-purpose knowledge\.

Table 2:Ablation experiments of different constraints inEoupCT\.AblationQwen3\-4BLlama3\-3BGemma2\-2BSuperNIMMLUSuperNIMMLUSuperNIMMLURou\.Fgt\.Acc\.Fgt\.Rou\.Fgt\.Acc\.Fgt\.Rou\.Fgt\.Acc\.Fgt\.EoupCT50\.76\.060\.20\.744\.010\.541\.4\-0\.846\.710\.338\.50\.8w/o C247\.610\.359\.02\.042\.313\.040\.40\.244\.810\.136\.92\.3w/o P&PO47\.39\.257\.83\.142\.112\.240\.30\.343\.513\.036\.92\.4w/o N&PO47\.48\.757\.93\.041\.912\.240\.00\.644\.213\.336\.62\.6w/o N&NO46\.910\.356\.94\.141\.511\.739\.61\.043\.213\.535\.73\.6
### 4\.4Forgetting Dynamics

Figure 3:Forgetting dynamics of Qwen3\-4B \(left\) and Llama3\-3B \(right\) acrossMMLU\.To further analyze the forgetting behavior during continual fine\-tuning, we track the evolution of general\-purpose knowledge by evaluating the accuracy on threeMMLUcategories as theSuperNItasks are sequentially learned\. We compareEoupCTwith the recent method CLoRA, focusing on how the performance changes throughout the training process\.

The results reveal a clear advantage ofEoupCTin terms of both the rate and extent of forgetting\. As the number of learned tasks increases, CLoRA exhibits a steady decline inMMLUaccuracy, indicating progressive degradation of general\-purpose knowledge\. In contrast,EoupCTdemonstrates a much slower decrease, maintaining consistently higher accuracy across all stages of training\. This confirms that our method effectively mitigates the accumulation of forgetting over time\. More interestingly, we observe thatEoupCTnot only preserves knowledge but can even improve performance in certain stages, where theMMLUaccuracy temporarily surpasses that of the initial pre\-trained LLM before any fine\-tuning\. This phenomenon suggests that the generated pseudo data and the associated orthogonal optimization not only prevent knowledge erosion but may also reinforce and refine useful representations within the LLM\.

## 5Conclusion

In this paper, we address the challenge of catastrophic forgetting in the continual LLM fine\-tuning, where existing methods fail to preserve inherent general\-purpose knowledge due to the unknown and highly diverse nature of original pre\-training data and gradients\. To overcome this fundamental barrier, we proposeEoupCT, a novel framework that dynamically estimates and orthogonalizes these unknown pre\-training representations\. Specifically, our approach leverages a learnable soft prompt equipped with a Gumbel\-Softmax relaxation to generate the pseudo pre\-training data most susceptible to forgetting\. Furthermore, we formulate a multi\-objective optimization problem solved by an efficient first\-order Pareto optimizer, which jointly optimizes the soft prompt and task\-specific LLM parameters to rigorously enforce gradient orthogonality without the burden of second\-order computations\. Extensive experiments across multiple LLMs demonstrate thatEoupCTeffectively maintains task\-specific proficiency while preserving the models’ broad general\-purpose capabilities, successfully offering a robust solution to the catastrophic forgetting in continual LLM fine\-tuning\.

## Acknowledgements

This work was supported in part by the National Natural Science Foundation of China \(No\.62276113, 62372150\) and JST ASPIRE Grant Number JPMJAP2405\.

## References

- \[1\]AI@Meta\(2024\)Llama 3 model card\.External Links:[Link](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)Cited by:[2nd item](https://arxiv.org/html/2609.30935#A4.I1.i2.p1.1),[§D\.1](https://arxiv.org/html/2609.30935#A4.SS1.p1.1)\.
- \[2\]N\. Carlini, F\. Tramèr, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. B\. Brown, D\. Song, Ú\. Erlingsson, A\. Oprea, and C\. Raffel\(2021\)Extracting training data from large language models\.InUSENIX Security Symposium,pp\. 2633–2650\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p3.1),[§1](https://arxiv.org/html/2609.30935#S1.p4.1),[§3\.2](https://arxiv.org/html/2609.30935#S3.SS2.p1.1)\.
- \[3\]S\. Dou, E\. Zhou, Y\. Liu, S\. Gao, W\. Shen, L\. Xiong, Y\. Zhou, X\. Wang, Z\. Xi, X\. Fan, S\. Pu, J\. Zhu, R\. Zheng, T\. Gui, Q\. Zhang, and X\. Huang\(2024\)LoRAMoE: alleviating world knowledge forgetting in large language models via moe\-style plugin\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 1932–1945\.Cited by:[§B\.3](https://arxiv.org/html/2609.30935#A2.SS3.p1.1),[2nd item](https://arxiv.org/html/2609.30935#A4.I2.i2.p1.1),[§1](https://arxiv.org/html/2609.30935#S1.p1.1),[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[4\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p1.1)\.
- \[5\]Y\. He, W\. Zhang, X\. Huang, P\. Zhang, L\. Meng, X\. Zhou, K\. Zeng, and X\. Cai\(2025\)Don’t half\-listen: capturing key\-part information in continual instruction tuning\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 23649–23668\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[6\]D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt\(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,Cited by:[§D\.2](https://arxiv.org/html/2609.30935#A4.SS2.p2.1),[§1](https://arxiv.org/html/2609.30935#S1.p5.1),[§4](https://arxiv.org/html/2609.30935#S4.p2.1)\.
- \[7\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[1st item](https://arxiv.org/html/2609.30935#A4.I2.i1.p1.1),[§1](https://arxiv.org/html/2609.30935#S1.p2.1),[§4](https://arxiv.org/html/2609.30935#S4.p2.1)\.
- \[8\]J\. Huang, L\. Cui, A\. Wang, C\. Yang, X\. Liao, L\. Song, J\. Yao, and J\. Su\(2024\)Mitigating catastrophic forgetting in large language models with self\-synthesized rehearsal\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 1416–1428\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[9\]E\. Jang, S\. Gu, and B\. Poole\(2017\)Categorical reparameterization with gumbel\-softmax\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2609.30935#A1.SS1.p3.3),[§1](https://arxiv.org/html/2609.30935#S1.p4.1),[§3\.2](https://arxiv.org/html/2609.30935#S3.SS2.p2.1)\.
- \[10\]A\. M\. Kassem, O\. Mahmoud, N\. Mireshghallah, H\. Kim, Y\. Tsvetkov, Y\. Choi, S\. Saad, and S\. Rana\(2025\)ALPACA AGAINST VICUNA: using llms to uncover memorization of llms\.InConference of the Nations of the Americas Chapter of the Association for Computational Linguistics,pp\. 8296–8321\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p3.1)\.
- \[11\]G\. Khromov and S\. P\. Singh\(2024\)Some fundamental aspects about lipschitz continuity of neural networks\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2609.30935#A1.SS1.p3.1)\.
- \[12\]Y\. Leng and D\. Xiong\(2025\)Towards understanding multi\-task learning \(generalization\) of llms via detecting and exploring task\-specific neurons\.InInternational Conference on Computational Linguistics,pp\. 2969–2987\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[13\]B\. Lester, R\. Al\-Rfou, and N\. Constant\(2021\)The power of scale for parameter\-efficient prompt tuning\.InConference on Empirical Methods in Natural Language Processing,pp\. 3045–3059\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p4.1)\.
- \[14\]Y\. Liang, J\. Chen, and W\. Li\(2025\)Gated integration of low\-rank adaptation for continual learning of large language models\.InAdvances in Neural Information Processing Systems,Cited by:[4th item](https://arxiv.org/html/2609.30935#A4.I2.i4.p1.1),[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[15\]Y\. Liang and W\. Li\(2024\)InfLoRA: interference\-free low\-rank adaptation for continual learning\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23638–23647\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p2.1)\.
- \[16\]C\. Liao, R\. Xie, X\. Sun, H\. Sun, and Z\. Kang\(2025\)Exploring forgetting in large language model pre\-training\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 2112–2127\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p3.1)\.
- \[17\]Y\. Lu, B\. Qian, C\. Yuan, H\. Jiang, and X\. Wang\(2025\)Controlled low\-rank adaptation with subspace regularization for continued training on large language models\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 19165–19181\.Cited by:[§B\.3](https://arxiv.org/html/2609.30935#A2.SS3.p1.1),[5th item](https://arxiv.org/html/2609.30935#A4.I2.i5.p1.1),[§1](https://arxiv.org/html/2609.30935#S1.p2.1),[§2](https://arxiv.org/html/2609.30935#S2.p1.1),[§2](https://arxiv.org/html/2609.30935#S2.p2.1)\.
- \[18\]Y\. Luo, Z\. Yang, F\. Meng, Y\. Li, J\. Zhou, and Y\. Zhang\(2023\)An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.CoRRabs/2308\.08747\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p2.1)\.
- \[19\]Y\. Ma, Z\. Liang, H\. Dai, B\. Chen, D\. Gao, Z\. Ran, Z\. Wang, L\. Jin, W\. Jiang, G\. Zhang, X\. Cai, and L\. Yang\(2024\)MoDULA: mixture of domain\-specific and universal lora for multi\-task learning\.InConference on Empirical Methods in Natural Language Processing,pp\. 2758–2770\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[20\]M\. McCloskey and N\. J\. Cohen\(1989\)Catastrophic interference in connectionist networks: the sequential learning problem\.Psychology of Learning and Motivation\-Advances in Research and Theory24\(C\),pp\. 109–165\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p2.1)\.
- \[21\]M\. Nasr, J\. Rando, N\. Carlini, J\. Hayase, M\. Jagielski, A\. F\. Cooper, D\. Ippolito, C\. A\. Choquette\-Choo, F\. Tramèr, and K\. Lee\(2025\)Scalable extraction of training data from aligned, production language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p3.1)\.
- \[22\]F\. Qiao and M\. Mahdavi\(2026\)Merge before forget: A single lora continual learning via continual merging\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[23\]H\. Shi, Z\. Xu, H\. Wang, W\. Qin, W\. Wang, Y\. Wang, Z\. Wang, S\. Ebrahimi, and H\. Wang\(2026\)Continual learning of large language models: A comprehensive survey\.ACM Computing Surveys58\(5\),pp\. 120:1–120:42\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p2.1)\.
- \[24\]F\. Sun, C\. Ho, and H\. Lee\(2020\)LAMOL: language modeling for lifelong language learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[25\]G\. Team\(2024\)Gemma 2: improving open language models at a practical size\.CoRRabs/2408\.00118\.Cited by:[3rd item](https://arxiv.org/html/2609.30935#A4.I1.i3.p1.1),[§D\.1](https://arxiv.org/html/2609.30935#A4.SS1.p1.1)\.
- \[26\]Q\. Team\(2025\)Qwen3 technical report\.CoRRabs/2505\.09388\.Cited by:[1st item](https://arxiv.org/html/2609.30935#A4.I1.i1.p1.1),[§D\.1](https://arxiv.org/html/2609.30935#A4.SS1.p1.1),[§1](https://arxiv.org/html/2609.30935#S1.p1.1)\.
- \[27\]B\. Wang, X\. Li, C\. Li, J\. Chi, G\. Niu, and M\. Sugiyama\(2026\)Decomposing the basic abilities of large language models: mitigating cross\-task interference in multi\-task instruct\-tuning\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p2.1)\.
- \[28\]B\. Wang, X\. Li, M\. Ye, C\. Li, B\. Fu, J\. Qu, and L\. Y\. Wu\(2025\)Remember past, anticipate future: learning continual multimodal misinformation detectors\.InACM International Conference on Multimedia,pp\. 5490–5498\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[29\]C\. Wang, Y\. Lyu, Z\. Sun, and L\. Jing\(2025\)Continual gradient low\-rank projection fine\-tuning for llms\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 14815–14829\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p1.1),[§1](https://arxiv.org/html/2609.30935#S1.p2.1),[§2](https://arxiv.org/html/2609.30935#S2.p1.1),[§2](https://arxiv.org/html/2609.30935#S2.p2.1)\.
- \[30\]J\. Wang, L\. Ding, S\. Wang, H\. Li, Y\. Luo, H\. Zhao, H\. Hu, and B\. Du\(2025\)Self\-evolving pseudo\-rehearsal for catastrophic forgetting with task similarity in llms\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[31\]X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. Huang\(2023\)Orthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 10658–10671\.Cited by:[3rd item](https://arxiv.org/html/2609.30935#A4.I2.i3.p1.1),[§1](https://arxiv.org/html/2609.30935#S1.p2.1),[§2](https://arxiv.org/html/2609.30935#S2.p1.1),[§2](https://arxiv.org/html/2609.30935#S2.p2.1)\.
- \[32\]Y\. Wang, Y\. Liu, C\. Shi, H\. Li, C\. Chen, H\. Lu, and Y\. Yang\(2024\)InsCL: A data\-efficient continual learning paradigm for fine\-tuning large language models with instructions\.InConference of the North American Chapter of the Association for Computational Linguistics,pp\. 663–677\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[33\]Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ NLP tasks\.InConference on Empirical Methods in Natural Language Processing,pp\. 5085–5109\.Cited by:[§D\.2](https://arxiv.org/html/2609.30935#A4.SS2.p1.1),[Table 5](https://arxiv.org/html/2609.30935#A4.T5),[§1](https://arxiv.org/html/2609.30935#S1.p5.1),[§4](https://arxiv.org/html/2609.30935#S4.p2.1)\.
- \[34\]Y\. Xiong and X\. Xie\(2026\)OPLoRA: orthogonal projection lora prevents catastrophic forgetting during parameter\-efficient fine\-tuning\.InAAAI Conference on Artificial Intelligence,pp\. 34088–34096\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p2.1)\.
- \[35\]S\. Yan, K\. Liu, C\. Shen, B\. Wang, S\. Fan, J\. Zhang, Y\. Wu, Z\. Wang, and J\. Ye\(2026\)Distribution\-aligned sequence distillation for superior long\-cot reasoning\.CoRRabs/2601\.09088\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p1.1)\.
- \[36\]L\. Yang, Y\. Liao, Y\. Wang, and Y\. Wang\(2026\)SLoRA: balancing plasticity and forgetting in large language models for continual learning\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 5437–5454\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[37\]S\. Yang, K\. Ning, Y\. Liu, J\. Yao, Y\. Tian, Y\. Song, and L\. Yuan\(2025\)Is parameter collision hindering continual learning in llms?\.InInternational Conference on Computational Linguistics,pp\. 4243–4259\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[38\]W\. Yu, T\. Pang, Q\. Liu, C\. Du, B\. Kang, Y\. Huang, M\. Lin, and S\. Yan\(2023\)Bag of tricks for training data extraction from language models\.InInternational Conference on Machine Learning,pp\. 40306–40320\.Cited by:[§1](https://arxiv.org/html/2609.30935#S1.p4.1),[§3\.2](https://arxiv.org/html/2609.30935#S3.SS2.p1.1)\.
- \[39\]Y\. Zhang, X\. Guo, Y\. Zhao, X\. Sui, B\. Zhou, X\. Qian, and X\. Yuan\(2026\)Grow\-on\-demand: sparse and adaptive expert expansion for continual instruction tuning\.InAAAI Conference on Artificial Intelligence,pp\. 28474–28482\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1)\.
- \[40\]W\. Zhao, S\. Wang, Y\. Hu, Y\. Zhao, B\. Qin, X\. Zhang, Q\. Yang, D\. Xu, and W\. Che\(2024\)SAPT: A shared attention framework for parameter\-efficient continual learning of large language models\.InAnnual Meeting of the Association for Computational Linguistics,pp\. 11641–11661\.Cited by:[§2](https://arxiv.org/html/2609.30935#S2.p1.1),[§4](https://arxiv.org/html/2609.30935#S4.p2.1)\.

## Appendix ATheoretical Proofs

In this section, we address the following three questions by providing theoretical proofs\.

RQ1\.From the data perspective, can continuous Gumbel\-Softmax relaxation be utilized to estimate the pre\-training distribution𝒯0\\mathcal\{T\}\_\{0\}composed of discrete token sequences?

RQ2\.From the local optimization perspective, can our proposed orthogonal projection method theoretically resolve the catastrophic forgetting?

RQ3\.From the global generalization perspective, can maintaining only a soft prompt and its generated surrogate sequences effectively preserve the vast repository of pre\-trained knowledge?

### A\.1Can Gumbel\-Softmax Relaxation Estimate Discrete Pre\-training Sequences?

In Sec\.[3\.2](https://arxiv.org/html/2609.30935#S3.SS2), we utilize a continuous soft prompt𝐏t\\mathbf\{P\}\_\{t\}to generate a sequence of soft token embeddings𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\. However, the authentic pre\-training distribution𝒯0\\mathcal\{T\}\_\{0\}inherently resides in a discrete text space\. We must mathematically guarantee that the stability objectiveℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}evaluated on our differentiable soft sequence effectively bounds the expected loss of a true discrete text sequence, and that this approximation gap strictly vanishes as the Gumbel\-Softmax temperatureτ\\tauapproaches zero\.

Theorem 1\.As formulated in Sec\.[3\.2](https://arxiv.org/html/2609.30935#S3.SS2), let𝐖emb∈ℝ\|𝒱\|×D\\mathbf\{W\}\_\{\\text\{emb\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times D\}be the token embedding matrix\. At decoding stepmm, let𝛑m​\(τ\)∈ℝ\|𝒱\|\\boldsymbol\{\\pi\}\_\{m\}\(\\tau\)\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}be the continuous probability vector generated by the Gumbel\-Softmax function at temperatureτ\\tau\. Let𝐲m∈\{0,1\}\|𝒱\|\\mathbf\{y\}\_\{m\}\\in\\\{0,1\\\}^\{\|\\mathcal\{V\}\|\}be the true discrete one\-hot vector obtained by an ideal𝐚𝐫𝐠⁡𝐦𝐚𝐱\\boldsymbol\{\\arg\}\\boldsymbol\{\\max\}operation on the logits, e\.g\., greedy sampling\. Assume the knowledge distillation loss functionℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}isLL\-Lipschitz continuous with respect to the input embeddings\. The expected gap between the loss evaluated on the soft embedding and the discrete embedding is strictly bounded by𝒪⁡\(τ\)\\mathcal\{O\}\(\\tau\), i\.e\., a linear correlation\.

Proof\.We define the continuous soft token embedding as𝐞soft=𝝅m​\(τ\)​𝐖emb\\mathbf\{e\}\_\{\\text\{soft\}\}=\\boldsymbol\{\\pi\}\_\{m\}\(\\tau\)\\mathbf\{W\}\_\{\\text\{emb\}\}and the corresponding true discrete token embedding as𝐞discrete=𝐲m​𝐖emb\\mathbf\{e\}\_\{\\text\{discrete\}\}=\\mathbf\{y\}\_\{m\}\\mathbf\{W\}\_\{\\text\{emb\}\}\. By the definition ofLL\-Lipschitz continuity\[[Khromov and Singh, 2024](https://arxiv.org/html/2609.30935#bib.bib25)\], the magnitude of the difference in the loss function is bounded by the Euclidean distance between the input vectors multiplied by the Lipschitz constantLL, which gives

\|ℒpre​\(𝐞soft\)−ℒpre​\(𝐞discrete\)\|≤L​‖𝐞soft−𝐞discrete‖2\.\|\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{e\}\_\{\\text\{soft\}\}\)\-\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{e\}\_\{\\text\{discrete\}\}\)\|\\leq L\\\|\\mathbf\{e\}\_\{\\text\{soft\}\}\-\\mathbf\{e\}\_\{\\text\{discrete\}\}\\\|\_\{2\}\.We expand this distance metric by substituting the definitions of the respective embeddings:

‖𝐞soft−𝐞discrete‖2=‖\(𝝅m​\(τ\)−𝐲m\)​𝐖emb‖2≤‖𝐖emb‖max⋅‖𝝅m​\(τ\)−𝐲m‖1,\\\|\\mathbf\{e\}\_\{\\text\{soft\}\}\-\\mathbf\{e\}\_\{\\text\{discrete\}\}\\\|\_\{2\}=\\\|\(\\boldsymbol\{\\pi\}\_\{m\}\(\\tau\)\-\\mathbf\{y\}\_\{m\}\)\\mathbf\{W\}\_\{\\text\{emb\}\}\\\|\_\{2\}\\leq\\\|\\mathbf\{W\}\_\{\\text\{emb\}\}\\\|\_\{\\max\}\\cdot\\\|\\boldsymbol\{\\pi\}\_\{m\}\(\\tau\)\-\\mathbf\{y\}\_\{m\}\\\|\_\{1\},where‖𝐖emb‖max\\\|\\mathbf\{W\}\_\{\\text\{emb\}\}\\\|\_\{\\max\}denotes the maximumL2L\_\{2\}\-norm of any single token embedding vector within the vocabulary𝒱\\mathcal\{V\}\. A foundational property of the Gumbel\-Softmax relaxation\[[Jang et al\., 2017](https://arxiv.org/html/2609.30935#bib.bib15)\]is that the expectedL1L\_\{1\}distance between the soft probability distribution𝝅m​\(τ\)\\boldsymbol\{\\pi\}\_\{m\}\(\\tau\)and the hard categorical one\-hot vector𝐲m\\mathbf\{y\}\_\{m\}scales linearly with the temperatureτ\\tauasτ\\taubecomes small\. Consequently, we have𝔼⁡\[‖𝝅m​\(τ\)−𝐲m‖1\]≤C⋅τ\\mathbb\{E\}\[\\\|\\boldsymbol\{\\pi\}\_\{m\}\(\\tau\)\-\\mathbf\{y\}\_\{m\}\\\|\_\{1\}\]\\leq C\\cdot\\tau, whereCCis a distribution\-dependent constant\. Taking the expectation on both sides of our Lipschitz inequality, we conclude:

𝔼⁡\[\|ℒpre​\(𝐞soft\)−ℒpre​\(𝐞discrete\)\|\]≤L⋅‖𝐖emb‖max⋅C⋅τ=𝒪⁡\(τ\)\.\\mathbb\{E\}\\big\[\|\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{e\}\_\{\\text\{soft\}\}\)\-\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{e\}\_\{\\text\{discrete\}\}\)\|\\big\]\\leq L\\cdot\\\|\\mathbf\{W\}\_\{\\text\{emb\}\}\\\|\_\{\\max\}\\cdot C\\cdot\\tau=\\mathcal\{O\}\(\\tau\)\.This mathematically proves that by annealing the temperatureτ→0\\tau\\to 0during the generative replay phase, our fully differentiable soft sequence seamlessly and accurately approximates the true discrete text distribution, introducing negligible approximation error while perfectly preserving the gradient flow for optimizing𝐏t\\mathbf\{P\}\_\{t\}\.□\\square

### A\.2CanEoupCTTheoretically Resolve the Catastrophic Forgetting?

We show that our proposedEoupCT, which builds upon the principles of orthogonal gradient projection, effectively resolves the forgetting as well\. In Step 3 of our Pareto orthogonal optimizer in Sec\.[3\.4](https://arxiv.org/html/2609.30935#S3.SS4), when the historically safe new task gradient𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}heavily conflicts with the protection gradient𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}, we project𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}onto the normal plane of𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\. We must guarantee that this specific projection is not a heuristic, but the exact geometric optimal solution that minimizes the new task loss while strictly prohibiting any increase in the general knowledge distillation loss\.

Theorem 2\.Let𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}and𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}be the task and protection gradients as Eq\. \([10](https://arxiv.org/html/2609.30935#S3.E10)\), respectively, both already projected into the historical null space via𝚷new\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}\. If a gradient conflict is detected, i\.e\.,⟨𝐠~new,𝐠~pre⟩<0\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle<0, the proposed update direction𝐠~new∗=𝐠~new−⟨𝐠~new,𝐠~pre⟩‖𝐠~pre‖2​𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}=\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\-\\frac\{\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\}\{\\\|\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\\|^\{2\}\}\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}is the exact analytical solution to the constrained optimization problem: maximize the alignment with the new task gradient, subject to absolute zero interference with the estimated pre\-training general\-purpose knowledge\.

Proof\.We formulate the constrained gradient adjustment problem by seeking a modified update direction𝐝\\mathbf\{d\}that is geometrically as close as possible to the ideal new task gradient𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\(minimizing their Euclidean distance\), subject to the hard constraint that moving along𝐝\\mathbf\{d\}will not increase the distillation lossℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}\. To achieve this,𝐝\\mathbf\{d\}must be strictly orthogonal to the protection gradient𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\. Mathematically, this is expressed as𝐝∗=𝐚𝐫𝐠⁡𝐦𝐢𝐧𝐝⁡\(‖𝐝−𝐠~new‖2/2\)\\mathbf\{d\}^\{\*\}=\\boldsymbol\{\\arg\}\\boldsymbol\{\\min\}\_\{\\mathbf\{d\}\}\(\\\|\\mathbf\{d\}\-\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\\\|^\{2\}/2\)subject to⟨𝐝,𝐠~pre⟩=0\\langle\\mathbf\{d\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle=0\. To solve this constrained objective, we construct the Lagrangian by introducing a Lagrange multiplierλ∈ℝ\\lambda\\in\\mathbb\{R\}for the equality constraint:

ℒ⁡\(𝐝,λ\)=12​‖𝐝−𝐠~new‖2\+λ⁡⟨𝐝,𝐠~pre⟩\.\\mathcal\{L\}\(\\mathbf\{d\},\\lambda\)=\\frac\{1\}\{2\}\\\|\\mathbf\{d\}\-\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\\\|^\{2\}\+\\lambda\\langle\\mathbf\{d\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\.To locate the minimum, we compute the derivative of the Lagrangianℒ\\mathcal\{L\}with respect to𝐝\\mathbf\{d\}and set it to the zero vector, yielding∇𝐝ℒ​\(𝐝,λ\)=\(𝐝−𝐠~new\)\+λ​𝐠~pre=𝟎\\nabla\_\{\\mathbf\{d\}\}\\mathcal\{L\}\(\\mathbf\{d\},\\lambda\)=\(\\mathbf\{d\}\-\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\)\+\\lambda\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}=\\mathbf\{0\}, which rearranges to𝐝=𝐠~new−λ​𝐠~pre\\mathbf\{d\}=\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\-\\lambda\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\. We then substitute this expression for𝐝\\mathbf\{d\}back into the strict hard orthogonality constraint⟨𝐝,𝐠~pre⟩=0\\langle\\mathbf\{d\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle=0:

⟨𝐠~new−λ​𝐠~pre,𝐠~pre⟩=0⟹⟨𝐠~new,𝐠~pre⟩−λ​‖𝐠~pre‖2=0\.\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\-\\lambda\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle=0\\implies\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\-\\lambda\\\|\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\\|^\{2\}=0\.Solving this algebraic equation for the Lagrange multiplier givesλ=⟨𝐠~new,𝐠~pre⟩‖𝐠~pre‖2\\lambda=\\frac\{\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\}\{\\\|\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\\|^\{2\}\}\. Finally, substitutingλ\\lambdaback into the expression for𝐝\\mathbf\{d\}yields exactly our proposed projection update rule from Sec\.[3\.4](https://arxiv.org/html/2609.30935#S3.SS4):

𝐝∗=𝐠~new−⟨𝐠~new,𝐠~pre⟩‖𝐠~pre‖2​𝐠~pre≡𝐠~new∗\.\\mathbf\{d\}^\{\*\}=\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\-\\frac\{\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle\}\{\\\|\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\\|^\{2\}\}\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\equiv\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}\.This derivation proves that our projection mechanism is the rigorous geometric Pareto optimum\. It maximally preserves the plasticity required for learning𝒯t\\mathcal\{T\}\_\{t\}while mathematically enforcing absolute zero catastrophic forgetting of the estimated pre\-training distribution\.□\\square

### A\.3Can Maintaining Only a Soft Prompt Preserve Pre\-trained Knowledge?

EoupCTdoes not blindly replay the entire infinite pre\-training distribution𝒯0\\mathcal\{T\}\_\{0\}\. Instead, we employ adversarial optimization in Eq\. \([6](https://arxiv.org/html/2609.30935#S3.E6)\) to locate a worst\-case boundary sequence𝐒⁡\(𝐏t∗\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)and strictly protect it\. A critical theoretical question emerges: why does protecting this specific, adversarially selected subset generated by only a soft prompt guarantee the protection of the vast, global pre\-training distribution? We mathematically validate this using minimax generalization bounds\.

Theorem 3\.Let𝒯0\\mathcal\{T\}\_\{0\}denote the true global distribution of general\-purpose knowledge\. Let𝒫\\mathcal\{P\}define the expressive continuous space of the soft prompt𝐏t\\mathbf\{P\}\_\{t\}\. Assuming our adversarial solver successfully identifies the worst\-case prompt𝐏t∗=𝐚𝐫𝐠⁡𝐦𝐚𝐱𝐏t∈𝒫​ℒpre​\(Δ​𝐖virt,𝐒⁡\(𝐏t\)\)\\mathbf\{P\}\_\{t\}^\{\*\}=\\boldsymbol\{\\arg\}\\boldsymbol\{\\max\}\_\{\\mathbf\{P\}\_\{t\}\\in\\mathcal\{P\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\Delta\\mathbf\{W\}\_\{\\text\{virt\}\};\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\), and our Pareto\-orthogonal update bounds this worst\-case loss such thatℒpre​\(Δ​𝐖t\+1,𝐒⁡\(𝐏t∗\)\)≤ϵ\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\Delta\\mathbf\{W\}\_\{t\+1\};\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\)\\leq\\epsilon, then the expected catastrophic forgetting over the entire true pre\-training distribution is strictly bounded from above\.

Proof\.We formalize the total catastrophic forgetting across the pre\-training LLM’s general\-purpose knowledge as the expected loss over the true pre\-training distribution:Forgettingglobal=𝔼𝐱∼𝒯0​\[ℒpre​\(Δ​𝐖t\+1,𝐱\)\]\\text\{Forgetting\}\_\{\\text\{global\}\}=\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\mathcal\{T\}\_\{0\}\}\[\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\Delta\\mathbf\{W\}\_\{t\+1\};\\mathbf\{x\}\)\]\. We operate under the standard assumption that the soft prompt space𝒫\\mathcal\{P\}is sufficiently expressive such that the generated continuous sequences can smoothly cover the support of the true distribution𝒯0\\mathcal\{T\}\_\{0\}\. Consequently, for any arbitrary actual data sample𝐱∈𝒯0\\mathbf\{x\}\\in\\mathcal\{T\}\_\{0\}, there exists some prompt configuration𝐏𝐱∈𝒫\\mathbf\{P\}\_\{\\mathbf\{x\}\}\\in\\mathcal\{P\}capable of approximating it with an approximation error bounded by a small constantδ\\delta, yielding the inequalityℒpre​\(𝐱\)≤ℒpre​\(𝐒⁡\(𝐏𝐱\)\)\+δ\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{x\}\)\\leq\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{\\mathbf\{x\}\}\)\)\+\\delta\. By the mathematical definition of the maximization operator in our vulnerability objective, the loss computed on any specific generated sequence𝐒⁡\(𝐏𝐱\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{\\mathbf\{x\}\}\)must be less than or equal to the loss evaluated on the adversarially optimized, worst\-case sequence𝐒⁡\(𝐏t∗\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\):

∀𝐱∈𝒯0,ℒpre​\(𝐒⁡\(𝐏𝐱\)\)≤max𝐏t∈𝒫⁡ℒpre​\(𝐒⁡\(𝐏t\)\)=ℒpre​\(𝐒⁡\(𝐏t∗\)\)\.\\forall\\mathbf\{x\}\\in\\mathcal\{T\}\_\{0\},\\quad\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{\\mathbf\{x\}\}\)\)\\leq\\max\_\{\\mathbf\{P\}\_\{t\}\\in\\mathcal\{P\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\)=\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\)\.By chaining these two inequalities, we elegantly bound the loss of any genuine data point𝐱\\mathbf\{x\}:

ℒpre​\(𝐱\)≤ℒpre​\(𝐒⁡\(𝐏t∗\)\)\+δ\.\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{x\}\)\\leq\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\)\+\\delta\.Taking the expectation over𝐱∼𝒯0\\mathbf\{x\}\\sim\\mathcal\{T\}\_\{0\}strictly preserves this inequality, leading directly to𝔼𝐱∼𝒯0​\[ℒpre​\(𝐱\)\]≤ℒpre​\(𝐒⁡\(𝐏t∗\)\)\+δ\\mathbb\{E\}\_\{\\mathbf\{x\}\\sim\\mathcal\{T\}\_\{0\}\}\[\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{x\}\)\]\\leq\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\)\+\\delta\. Because our Pareto\-orthogonal solver \(Sec\.[3\.4](https://arxiv.org/html/2609.30935#S3.SS4)\) explicitly enforces that the worst\-case loss does not increase \(*i\.e\.,*ℒpre​\(𝐒⁡\(𝐏t∗\)\)≤ϵ\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\)\\leq\\epsilon\), the global expected forgetting is naturally bounded by:

Forgettingglobal≤ϵ\+δ\.\\text\{Forgetting\}\_\{\\text\{global\}\}\\leq\\epsilon\+\\delta\.This mathematically proves that by explicitly hunting for and strictly protecting the vulnerability boundaries \(the adversarial “hard negatives”\),EoupCTcasts a robust, overarching protective net across the entire global distribution𝒯0\\mathcal\{T\}\_\{0\}\.□\\square

Remark 3\.1\.A potential critique of Theorem 3 is the assumption that the continuous prompt space𝒫\\mathcal\{P\}can densely cover the vast support of the true pre\-training distribution𝒯0\\mathcal\{T\}\_\{0\}\. We clarify that our bound does not require global coverage\. Instead, we rely on the concept oflocal adversarial representativeness\. For a given fine\-tuning task at steptt, the catastrophic forgetting is primarily driven by the parameter sub\-space aligned with the task gradient\. Therefore, optimizing𝐏t\\mathbf\{P\}\_\{t\}via the virtual step naturally forces the generated sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)to act as an adversarial coreset\. This coreset only needs to cover the specific local manifold of𝒯0\\mathcal\{T\}\_\{0\}that is most vulnerable to the current orthogonal update, bounded by a local approximation errorδl​o​c​a​l≪δg​l​o​b​a​l\\delta\_\{local\}\\ll\\delta\_\{global\}\. As a result, the theoretical upper bound on forgetting remains tight and non\-vacuous within the task\-specific projection subspace\.

## Appendix BComplexity Analysis ofEoupCT

In this section, we analyze the computational efficiency of theEoupCTframework\. Appendix[B\.1](https://arxiv.org/html/2609.30935#A2.SS1)mathematically demonstrates why directly solving the multi\-objective optimization problem in Eqs\. \([5](https://arxiv.org/html/2609.30935#S3.E5)\), \([6](https://arxiv.org/html/2609.30935#S3.E6)\), and \([7](https://arxiv.org/html/2609.30935#S3.E7)\) inherently requires intractable second\-order Hessian matrices, and how our efficient Pereto optimizer in Sec\.[3\.4](https://arxiv.org/html/2609.30935#S3.SS4)reduces this to a first\-order optimization; Appendix[B\.2](https://arxiv.org/html/2609.30935#A2.SS2)provides a step\-by\-step time complexity comparison between continual fine\-tuning baselines andEoupCT; Appendix[B\.3](https://arxiv.org/html/2609.30935#A2.SS3)empirically shows the computational budgets ofEoupCT\.

### B\.1Efficient First\-Order Pareto Optimization

The core of our vulnerability objective\[C2\]is to explicitly find a soft prompt𝐏t\\mathbf\{P\}\_\{t\}that generates data maximizing the gradient conflict with the new task\. Mathematically, this means maximizing the negative inner product between the new task gradient𝐠new\\mathbf\{g\}\_\{\\text\{new\}\}and the protection gradient∇𝜽ℒpre\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\. We now demonstrate why directly optimizing this requires a Hessian, and how we avoid it\.

Second\-order optimization bottleneck\.Let us formalize the naive objective\. To find the worst\-case prompt𝐏t\\mathbf\{P\}\_\{t\}, we want to solve:

𝐦𝐚𝐱𝐏t𝒥⁡\(𝐏t\)=⟨−𝐠new,∇𝜽ℒpre​\(𝜽,𝐒⁡\(𝐏t\)\)⟩=−𝐠new⊤​∇𝜽ℒpre\.\\mathop\{\\boldsymbol\{\\max\}\}\\limits\_\{\\mathbf\{P\}\_\{t\}\}\\mathcal\{J\}\(\\mathbf\{P\}\_\{t\}\)=\\big\\langle\-\\mathbf\{g\}\_\{\\text\{new\}\},\\,\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\\big\)\\big\\rangle=\-\\mathbf\{g\}\_\{\\text\{new\}\}^\{\\top\}\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\.To optimize this via gradient ascent, we must compute the derivative of𝒥⁡\(𝐏t\)\\mathcal\{J\}\(\\mathbf\{P\}\_\{t\}\)with respect to the prompt parameters𝐏t\\mathbf\{P\}\_\{t\}\. Since the new task gradient𝐠new\\mathbf\{g\}\_\{\\text\{new\}\}is computed on𝒯t\\mathcal\{T\}\_\{t\}and is strictly independent of𝐏t\\mathbf\{P\}\_\{t\}, it acts as a constant during this differentiation\. Applying the chain rule yields:

∇𝐏t𝒥​\(𝐏t\)=∇𝐏t\[−𝐠new⊤​∇𝜽ℒpre\]=−\(∇𝐏t,𝜽2ℒpre\)​𝐠new\.\\nabla\_\{\\mathbf\{P\}\_\{t\}\}\\mathcal\{J\}\(\\mathbf\{P\}\_\{t\}\)=\\nabla\_\{\\mathbf\{P\}\_\{t\}\}\\Big\[\-\\mathbf\{g\}\_\{\\text\{new\}\}^\{\\top\}\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\Big\]=\-\\Big\(\\nabla^\{2\}\_\{\\mathbf\{P\}\_\{t\},\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\Big\)\\mathbf\{g\}\_\{\\text\{new\}\}\.The term∇𝐏t,𝜽2ℒpre\\nabla^\{2\}\_\{\\mathbf\{P\}\_\{t\},\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}is amixed second\-order derivative matrix,*i\.e\.,*a Hessian cross\-derivative\. Computing a Hessian\-vector product through the massive layers of an LLM requires unrolling the entire computational graph of the first gradient backward\. This triggers an𝒪⁡\(N2\)\\mathcal\{O\}\(N^\{2\}\)memory explosion and massive computational latency, making it absolutely intractable for LLMs\.

First\-order resolution ofEoupCT\.Instead of differentiating the inner product,EoupCTemploys avirtual step approximation\. According to the first\-order Taylor expansion ofℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}around the current weights𝜽t\\boldsymbol\{\\theta\}\_\{t\}with a step sizeα\\alpha:

ℒpre​\(𝜽t−α​𝐠new\)≈ℒpre​\(𝜽t\)−α​𝐠new⊤​∇𝜽ℒpre​\(𝜽t\)\.\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{t\}\-\\alpha\\mathbf\{g\}\_\{\\text\{new\}\}\)\\approx\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{t\}\)\-\\alpha\\mathbf\{g\}\_\{\\text\{new\}\}^\{\\top\}\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\}\_\{t\}\)\.Notice that the term−𝐠new⊤​∇𝜽ℒpre\-\\mathbf\{g\}\_\{\\text\{new\}\}^\{\\top\}\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}is exactly our desired gradient conflict objective𝒥⁡\(𝐏t\)\\mathcal\{J\}\(\\mathbf\{P\}\_\{t\}\)\. By defining the virtual weights as𝜽virt=𝜽t−α​𝐠new\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}=\\boldsymbol\{\\theta\}\_\{t\}\-\\alpha\\mathbf\{g\}\_\{\\text\{new\}\}, the maximization problem becomes:

𝐦𝐚𝐱𝐏tℒpre​\(𝜽virt,𝐒⁡\(𝐏t\)\)\.\\mathop\{\\boldsymbol\{\\max\}\}\\limits\_\{\\mathbf\{P\}\_\{t\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\\big\)\.To optimize this new objective, we simply take the derivative w\.r\.t\.𝐏t\\mathbf\{P\}\_\{t\}:

∇𝐏tℒpre​\(𝜽virt,𝐒⁡\(𝐏t\)\)\.\\nabla\_\{\\mathbf\{P\}\_\{t\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\\big\(\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\};\\,\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)\\big\)\.Because𝜽virt\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}is treated as a fixed parameter matrix during this step, this operation is astandard, first\-order backpropagation\. We completely bypass the Hessian matrix, reducing the spatial and temporal complexity from𝒪⁡\(N2\)\\mathcal\{O\}\(N^\{2\}\)to𝒪⁡\(N\)\\mathcal\{O\}\(N\), rendering the adversarial training highly scalable\.

### B\.2Time Complexity Compared with Standard LoRA Baselines

We now quantify the exact computational overhead introduced byEoupCTcompared to standard LoRA fine\-tuning\. We define the time complexity of a standard LLM forward pass as𝒪⁡\(FPLLM\)\\mathcal\{O\}\(\\text\{FP\}\_\{\\text\{LLM\}\}\), a backward pass to update LoRA weights as𝒪⁡\(BPLoRA\)\\mathcal\{O\}\(\\text\{BP\}\_\{\\text\{LoRA\}\}\), and a backward pass to update the soft prompt as𝒪⁡\(BPPrompt\)\\mathcal\{O\}\(\\text\{BP\}\_\{\\text\{Prompt\}\}\)\. Note thatBPLoRA≈BPPrompt≪BPFull\_Model\\text\{BP\}\_\{\\text\{LoRA\}\}\\approx\\text\{BP\}\_\{\\text\{Prompt\}\}\\ll\\text\{BP\}\_\{\\text\{Full\\\_Model\}\}\.

In a standard LoRA fine\-tuning iteration, the model processes a batch of new task data, computes the loss, and backpropagates to update the LoRA parameters,*i\.e\.,*𝒪⁡\(FPLLM\)\+𝒪⁡\(BPLoRA\)\\mathcal\{O\}\(\\text\{FP\}\_\{\\text\{LLM\}\}\)\+\\mathcal\{O\}\(\\text\{BP\}\_\{\\text\{LoRA\}\}\)\.

Turn to analyzeEoupCT, a single iteration ofEoupCTperforms the following sequential steps:

1\. New task gradient \(Step 1\):Compute𝐠new\\mathbf\{g\}\_\{\\text\{new\}\}on𝒯t\\mathcal\{T\}\_\{t\},*i\.e\.,*𝒪⁡\(FPLLM\)\+𝒪⁡\(BPLoRA\)\\mathcal\{O\}\(\\text\{FP\}\_\{\\text\{LLM\}\}\)\+\\mathcal\{O\}\(\\text\{BP\}\_\{\\text\{LoRA\}\}\);

2\. Pre\-training gradient estimation \(Sec\.[3\.2](https://arxiv.org/html/2609.30935#S3.SS2)\):Auto\-regressively generating a soft sequence of lengthMMusing Gumbel\-Softmax requires generatingMMtokens iteratively, so the cost isM×𝒪⁡\(FPLLM\)M\\times\\mathcal\{O\}\(\\text\{FP\}\_\{\\text\{LLM\}\}\);

3\. Adversarial prompt update \(Step 1\):Evaluateℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}using the generated sequence on𝜽virt\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}, and backpropagate to update𝐏t\\mathbf\{P\}\_\{t\},*i\.e\.,*𝒪⁡\(FPLLM\)\+𝒪⁡\(BPPrompt\)\\mathcal\{O\}\(\\text\{FP\}\_\{\\text\{LLM\}\}\)\+\\mathcal\{O\}\(\\text\{BP\}\_\{\\text\{Prompt\}\}\);

4\. Protection gradient calculation \(Step 1\):Evaluateℒpre\\mathcal\{L\}\_\{\\text\{pre\}\}again on the true weights𝜽t\\boldsymbol\{\\theta\}\_\{t\}to obtain𝐠pre∗\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\},*i\.e\.,*𝒪⁡\(FPLLM\)\+𝒪⁡\(BPLoRA\)\\mathcal\{O\}\(\\text\{FP\}\_\{\\text\{LLM\}\}\)\+\\mathcal\{O\}\(\\text\{BP\}\_\{\\text\{LoRA\}\}\);

5\. Pareto orthogonal projection \(Steps 2 & 3\):Project the gradients using the historical memory matrix𝐌new\\mathbf\{M\}\_\{\\text\{new\}\}, which containst−1t\-1vectors of dimensionDLoRAD\_\{\\text\{LoRA\}\}\. This involves basic matrix\-vector multiplications on the parameter space, which is negligible compared to passing tensors through the LLM layers\. Therefore, its cost is𝒪⁡\(DLoRA⋅t\)≈𝒪⁡\(1\)\\mathcal\{O\}\(D\_\{\\text\{LoRA\}\}\\cdot t\)\\approx\\mathcal\{O\}\(1\)relative to LLM passes\.

Summing the costs, a single step ofEoupCTrequires𝒪⁡\(\(M\+3\)​FPLLM\+2​BPLoRA\+BPPrompt\)\\mathcal\{O\}\\big\(\(M\+3\)\\text\{FP\}\_\{\\text\{LLM\}\}\+2\\text\{BP\}\_\{\\text\{LoRA\}\}\+\\text\{BP\}\_\{\\text\{Prompt\}\}\\big\)\. Because the length of the surrogate sequenceMMis typically a small constant, the overall complexity ofEoupCTscaleslinearlywith the LLM size, identically to standard LoRA\. We successfully mitigate the catastrophic forgetting by introducing only asmall constant\-factor computational overhead, completely avoiding the exponential𝒪⁡\(N2\)\\mathcal\{O\}\(N^\{2\}\)scaling that traditionally plagues orthogonal gradient alignment and adversarial generation in LLMs\.

### B\.3Empirical Computational Budget

In Table[3](https://arxiv.org/html/2609.30935#A2.T3), we present an empirical comparison of the computational time required by our proposed method,EoupCT, against standard LoRA and other advanced continual LLM fine\-tuning baselines, including LoRAMoE\[[Dou et al\., 2024](https://arxiv.org/html/2609.30935#bib.bib1)\]and CLoRA\[[Lu et al\., 2025](https://arxiv.org/html/2609.30935#bib.bib21)\]\. All experiments are measured in running minutes using an 8×\\timesGPUs distributed training setup across various representative LLM architectures \(Qwen3, Llama3, and Gemma2 families\)\. As expected, standard LoRA serves as the lower bound for training time \(averaging 60 minutes, denoted as1×1\\times\) due to its straightforward low\-rank weight injection\. However, methods designed to enhance model capacity or mitigate catastrophic forgetting inherently introduce substantial computational overhead\. For instance, CLoRA aims to reduce output variations and preserve previous knowledge by applying subspace regularization, which imposes strict mathematical constraints on the direction of the updating matrix’s null space\. While effective for continual learning, this regularization mechanism significantly burdens the training process, resulting in an average computational cost of2\.6×2\.6\\timescompared to standard LoRA\. Similarly, LoRAMoE incurs the highest computational budget \(averaging3\.2×3\.2\\times\) due to the complex routing mechanisms and multiple expert networks involved during the forward and backward passes\.

In contrast, our proposedEoupCTdemonstrates superior computational efficiency among the advanced fine\-tuning methods\. Across all evaluated model scales,EoupCTconsistently requires considerably less training time than both CLoRA and LoRAMoE\. On average,EoupCTonly takes2\.0×2\.0\\timesthe time of standard LoRA, saving roughly 23% and 37% of the training time compared to CLoRA and LoRAMoE, respectively\. Notably, the efficiency ofEoupCTbecomes even more pronounced on relatively compact models; for example, on Llama3\-3B and Gemma2\-2B, the relative computational cost drops to just1\.8×1\.8\\timesand1\.5×1\.5\\times\. These results indicate thatEoupCTsuccessfully bypasses the severe computational bottlenecks typically associated with strict orthogonal gradient projection \(as seen in CLoRA\) or dense mixture\-of\-experts routing \(as in LoRAMoE\)\. Consequently,EoupCToffers a highly practical and scalable solution, striking an optimal balance between advanced model adaptability and empirical training efficiency\.

Table 3:Empirical computational time \(minutes\) on 8×\\timesGPUs\.LLMLoRALoRAMoE\(\#expert=4\)LoRAMoE\(\#expert=8\)OLoRACLoRAEoupCTQwen3\-8B88 \(1×\\times\)223 \(2\.5×\\times\)225 \(2\.5×\\times\)237 \(2\.7×\\times\)191 \(2\.2×\\times\)162 \(1\.8×\\times\)Qwen3\-4B62 \(1×\\times\)194 \(3\.1×\\times\)195 \(3\.1×\\times\)204 \(3\.3×\\times\)155 \(2\.5×\\times\)129 \(2\.1×\\times\)Llama3\-8B60 \(1×\\times\)197 \(3\.3×\\times\)199 \(3\.3×\\times\)211 \(3\.5×\\times\)156 \(2\.6×\\times\)130 \(2\.1×\\times\)Llama3\-3B35 \(1×\\times\)144 \(4\.1×\\times\)144 \(4\.1×\\times\)151 \(4\.3×\\times\)114 \(3\.2×\\times\)65 \(1\.8×\\times\)Gemma2\-9B76 \(1×\\times\)260 \(3\.4×\\times\)262 \(3\.4×\\times\)271 \(3\.5×\\times\)210 \(2\.7×\\times\)178 \(2\.3×\\times\)Gemma2\-2B44 \(1×\\times\)140 \(3\.2×\\times\)141 \(3\.2×\\times\)149 \(3\.4×\\times\)116 \(2\.6×\\times\)69 \(1\.5×\\times\)Average60 \(1×\\times\)193 \(3\.2×\\times\)194 \(3\.2×\\times\)203 \(3\.4×\\times\)157 \(2\.6×\\times\)122 \(2\.0×\\times\)

## Appendix CAlgorithm and Notations

In this section, we summarize the notations used in our proposed method and provide the complete algorithmic workflow ofEoupCT\. Table[4](https://arxiv.org/html/2609.30935#A3.T4)summarizes the primary mathematical notations and their corresponding descriptions used throughout the formulation of our method\. As analyzed in Sec\.[3\.4](https://arxiv.org/html/2609.30935#S3.SS4), directly solving the multi\-objective optimization problem for continual fine\-tuning is computationally intractable due to the natural adversariality and the requirement of second\-order Hessian matrices\. To address this,EoupCTelegantly decouples the complex optimization into an efficient first\-order bilevel training paradigm\. The overall training procedure is outlined in Alg\.[1](https://arxiv.org/html/2609.30935#alg1)\. For each new task, we first utilize a virtual step approximation to adversarially optimize the soft prompt, extracting the most vulnerable pre\-training data distribution\. Subsequently, we leverage null\-space projection and Pareto\-optimal orthogonal projection to eliminate gradient conflicts\. This ensures that the newly updated large model efficiently acquires new task plasticity while strictly maintaining general\-purpose knowledge stability and retaining previous downstream task abilities\.

Table 4:Summary of primary notations used inEoupCT\.NotationDescriptionℱ𝜽,𝜽\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}\},\\boldsymbol\{\\theta\}Frozen pre\-training LLM and trainable LLM weights𝒯t,𝒯0\\mathcal\{T\}\_\{t\},\\mathcal\{T\}\_\{0\}tt\-th new downstream task and original pre\-training distribution𝐏t,𝐒⁡\(𝐏t\)\\mathbf\{P\}\_\{t\},\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)Continuous soft prompt and generated surrogate sequenceℒnew,ℒpre\\mathcal\{L\}\_\{\\text\{new\}\},\\mathcal\{L\}\_\{\\text\{pre\}\}Plasticity and stability objective functions𝐠new,𝐠pre\\mathbf\{g\}\_\{\\text\{new\}\},\\mathbf\{g\}\_\{\\text\{pre\}\}Task\-specific and protection gradients w\.r\.t\.𝜽\\boldsymbol\{\\theta\}𝜽virt\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}Virtually updated weights simulating new task impactℳnew,𝚷new\\mathcal\{M\}\_\{\\text\{new\}\},\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}Historical gradient memory bank and its null\-space projection matrix𝐠~new,𝐠~pre\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}Gradients projected into historically safe null space𝐠~new∗\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}Pareto\-optimal modified new task gradientη,α,λ\\eta,\\alpha,\\lambdalearning rate, virtual step size, and orthogonal penalty coefficientAlgorithm 1Training Process ofEoupCT0:Frozen pre\-training LLM

ℱ𝜽\\mathcal\{F\}\_\{\\boldsymbol\{\\theta\}\}, task sequence

\{𝒯1,…,𝒯T\}\\\{\\mathcal\{T\}\_\{1\},\\dots,\\mathcal\{T\}\_\{T\}\\\}, learning rates

η,α\\eta,\\alpha, penalty coefficient

λ\\lambda\.

0:Optimized LLM parameters

𝜽\\boldsymbol\{\\theta\}\.

1:Initialize trainable LLM parameters

𝜽\\boldsymbol\{\\theta\}with

𝜽0\\boldsymbol\{\\theta\}^\{0\}\.

2:Initialize historical gradient memory sets

ℳnew←∅\\mathcal\{M\}\_\{\\text\{new\}\}\\leftarrow\\varnothing,

ℳpre←∅\\mathcal\{M\}\_\{\\text\{pre\}\}\\leftarrow\\varnothing\.

3:foreach new task

t=1,2,…,Tt=1,2,\\dots,Tdo

4:Initialize the learnable soft prompt

𝐏t\\mathbf\{P\}\_\{t\}\.

5:foreach training iterationdo

6:Sample a mini\-batch of data from

𝒯t\\mathcal\{T\}\_\{t\}\.

7:Compute new task gradient

𝐠new=∇𝜽ℒnew\\mathbf\{g\}\_\{\\text\{new\}\}=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{new\}\}\.

8:⊳\\trianglerightStep 1: Vulnerability & pre\-training gradient estimation

9:Perform virtual update

𝜽virt\\boldsymbol\{\\theta\}\_\{\\text\{virt\}\}via Eq\. \([8](https://arxiv.org/html/2609.30935#S3.E8)\)\.

10:Optimize

𝐏t\\mathbf\{P\}\_\{t\}by maximizing Eq\. \([6](https://arxiv.org/html/2609.30935#S3.E6)\) to generate

𝐒⁡\(𝐏t∗\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\.

11:Compute protection gradient

𝐠pre∗=∇𝜽ℒpre​\(𝜽,𝐒⁡\(𝐏t∗\)\)\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\}=\\nabla\_\{\\boldsymbol\{\\theta\}\}\\mathcal\{L\}\_\{\\text\{pre\}\}\(\\boldsymbol\{\\theta\};\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}^\{\*\}\)\)\.

12:⊳\\trianglerightStep 2: Null\-space projection

13:Construct projection matrix

𝚷new\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}via Eq\. \([9](https://arxiv.org/html/2609.30935#S3.E9)\) using

ℳnew\\mathcal\{M\}\_\{\\text\{new\}\}\.

14:Obtain safe gradients

𝐠~new=𝚷new​𝐠new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}=\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}\\mathbf\{g\}\_\{\\text\{new\}\}and

𝐠~pre=𝚷new​𝐠pre∗\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}=\\boldsymbol\{\\Pi\}\_\{\\text\{new\}\}\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\}\.

15:⊳\\trianglerightStep 3: Pareto orthogonal optimization

16:if

⟨𝐠~new,𝐠~pre⟩<0\\langle\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\},\\tilde\{\\mathbf\{g\}\}\_\{\\text\{pre\}\}\\rangle<0then

17:Compute interference\-free gradient

𝐠~new∗\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}via Pareto projection in Eq\. \([11](https://arxiv.org/html/2609.30935#S3.E11)\)\.

18:else

19:Set

𝐠~new∗=𝐠~new\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}^\{\*\}=\\tilde\{\\mathbf\{g\}\}\_\{\\text\{new\}\}\.

20:endif

21:⊳\\trianglerightStep 4: Parameter update

22:Update LLM parameters

𝜽\\boldsymbol\{\\theta\}following the direction in Eq\. \([12](https://arxiv.org/html/2609.30935#S3.E12)\)\.

23:endfor

24:Update memory:

ℳnew←ℳnew∪\{𝐠new\}\\mathcal\{M\}\_\{\\text\{new\}\}\\leftarrow\\mathcal\{M\}\_\{\\text\{new\}\}\\cup\\\{\\mathbf\{g\}\_\{\\text\{new\}\}\\\},

ℳpre←ℳpre∪\{𝐠pre∗\}\\mathcal\{M\}\_\{\\text\{pre\}\}\\leftarrow\\mathcal\{M\}\_\{\\text\{pre\}\}\\cup\\\{\\mathbf\{g\}\_\{\\text\{pre\}\}^\{\*\}\\\}\.

25:endfor

26:return

𝜽\\boldsymbol\{\\theta\}

## Appendix DExperimental Setups

To rigorously evaluateEoupCTunder a continual fine\-tuning setting, we conduct experiments on a broad suite of multi\-task benchmarks and multiple open\-source pre\-training LLMs\. In this section, we describe the pre\-training LLMs, datasets, baseline methods, evaluation metrics, and implementation details used throughout our experiments\.

### D\.1Pre\-training LLMs

To comprehensively evaluate the effectiveness of our method, we conduct experiments on three prominent families of open\-weight decoder\-only LLMs, selecting two parameter scales for each family:Qwen3\(4Band8B\)\[[Team, 2025](https://arxiv.org/html/2609.30935#bib.bib16)\],Llama3\(3Band8B\)\[[AI@Meta, 2024](https://arxiv.org/html/2609.30935#bib.bib2)\], andGemma2\(2Band9B\)\[[Team, 2024](https://arxiv.org/html/2609.30935#bib.bib14)\]\. These LLM families provide a diverse and representative testbed spanning different pre\-training corpora, architectural variations, and capacity regimes:

- •The Qwen3 family\[[Team, 2025](https://arxiv.org/html/2609.30935#bib.bib16)\]is renowned for its exceptional multilingual capabilities and robust alignment with human instructions\. We include the 4B and 8B variants to investigate continual learning dynamics, which distill from their larger\-scale variant,*e\.g\.,*Qwen3\-32B, by the off\-policy and on\-policy distillation\.
- •The Llama3 family\[[AI@Meta, 2024](https://arxiv.org/html/2609.30935#bib.bib2)\]is pre\-trained on a massive and highly curated corpus of over 9 trillion tokens\. It features an optimized Transformer architecture with grouped\-query attention and an expanded vocabulary size\. We utilize the 3B and 8B variants to represent highly capable models at accessible scales\.
- •The Gemma2 family\[[Team, 2024](https://arxiv.org/html/2609.30935#bib.bib14)\]introduces architectural innovations such as interleaved local and global attention mechanisms and logit soft\-capping\. The 2B and 9B models are selected to evaluate our framework’s stability on highly optimized, dense architectures\.

By evaluating across these distinct families and scales, we aim to demonstrate that our continual fine\-tuning framework is robust and generally applicable regardless of the underlying LLM architecture\.

Table 5:Overview of selected multi\-task datasets fromSuperNI\[[Wang et al\., 2022](https://arxiv.org/html/2609.30935#bib.bib12)\]\.IDDatasetTask TypeTrainEvalt1290xsum\_summarizationtext summarization1,000100t511reddit\_tifu\_long\_text\_summarizationtext summarization1,000100t1572samsum\_summarytext summarization16020t1729personachat\_generate\_nextdialogue generation1,000100t639multi\_woz\_user\_utterance\_generationdialogue generation14218t1590diplomacy\_text\_generationdialogue generation12616t1687sentiment140\_classificationsentiment analysis1,000100t363sst2\_polarity\_classificationsentiment analysis1,000100t875emotion\_classificationsentiment analysis1,000100t748glucose\_reverse\_cause\_event\_detectioninformation extraction1,000100t1510evalution\_relation\_extractioninformation extraction1,000100t181outcome\_extractioninformation extraction33843t591sciq\_answer\_generationquestion answering1,000100t002quoref\_answer\_generationquestion answering1,000100t073commonsenseqa\_answer\_generationquestion answering975100Table 6:Three distinct task orderings under the continual LLM fine\-tuning\.SeedTask Ordering1t591→t073→t1687→t875→t1572→t639→t1510→t1590→t511→t181→t363→t1729→t002→t1290→t748\\text\{t591\}\\rightarrow\\text\{t073\}\\rightarrow\\text\{t1687\}\\rightarrow\\text\{t875\}\\rightarrow\\text\{t1572\}\\rightarrow\\text\{t639\}\\rightarrow\\text\{t1510\}\\rightarrow\\text\{t1590\}\\rightarrow\\text\{t511\}\\rightarrow\\text\{t181\}\\rightarrow\\text\{t363\}\\rightarrow\\text\{t1729\}\\rightarrow\\text\{t002\}\\rightarrow\\text\{t1290\}\\rightarrow\\text\{t748\}2t1290→t1510→t1572→t002→t073→t511→t1590→t363→t748→t639→t181→t1729→t1687→t591→t875\\text\{t1290\}\\rightarrow\\text\{t1510\}\\rightarrow\\text\{t1572\}\\rightarrow\\text\{t002\}\\rightarrow\\text\{t073\}\\rightarrow\\text\{t511\}\\rightarrow\\text\{t1590\}\\rightarrow\\text\{t363\}\\rightarrow\\text\{t748\}\\rightarrow\\text\{t639\}\\rightarrow\\text\{t181\}\\rightarrow\\text\{t1729\}\\rightarrow\\text\{t1687\}\\rightarrow\\text\{t591\}\\rightarrow\\text\{t875\}3t002→t073→t1572→t181→t591→t1687→t363→t1729→t511→t875→t639→t748→t1590→t1290→t1510\\text\{t002\}\\rightarrow\\text\{t073\}\\rightarrow\\text\{t1572\}\\rightarrow\\text\{t181\}\\rightarrow\\text\{t591\}\\rightarrow\\text\{t1687\}\\rightarrow\\text\{t363\}\\rightarrow\\text\{t1729\}\\rightarrow\\text\{t511\}\\rightarrow\\text\{t875\}\\rightarrow\\text\{t639\}\\rightarrow\\text\{t748\}\\rightarrow\\text\{t1590\}\\rightarrow\\text\{t1290\}\\rightarrow\\text\{t1510\}
### D\.2Benchmark Datasets

We evaluate our continual LLM fine\-tuning framework on two complementary benchmarks,SuperNIandMMLU, aiming to jointly assess task\-specific adaptation and general\-purpose knowledge preservation under a continual training stream\. For continual task learning, we adopt a subset of theSuperNIbenchmark\[[Wang et al\., 2022](https://arxiv.org/html/2609.30935#bib.bib12)\], consisting of 15 diverse tasks spanning five categories, including question answering, information extraction, sentiment analysis, text summarization, and dialogue generation, as summarized in Table[5](https://arxiv.org/html/2609.30935#A4.T5)\. Each task is associated with its own training and evaluation splits, and the LLM is trained sequentially over the task stream, where each task is observed only once\. After completing the training on each task, we evaluate the LLM on the evaluation sets of all tasks, enabling a unified assessment of their accuracy and forgetting rate on previously learned tasks\. To mitigate the bias induced by a specific task sequence, we further consider three distinct task orderings generated from different random seeds, as shown in Table[6](https://arxiv.org/html/2609.30935#A4.T6), and conduct all experiments independently on each ordering to ensure robustness against ordering effects\.

Table 7:Statistic ofMMLUsubjects\.SubjectCategory\#Evalanatomyother135global\_factsother100marketingother234philosophyhumanities311formal\_logichumanities283prehistoryhumanities324high\_school\_mathSTEM270high\_school\_physicsSTEM154high\_school\_CSSTEM287In addition to task\-specific performance, we evaluate the LLM’s general\-purpose knowledge using a subset of theMMLUbenchmark\[[Hendrycks et al\., 2021](https://arxiv.org/html/2609.30935#bib.bib4)\], which covers a broad range of domains including STEM, humanities, and other disciplines, as listed in Table[7](https://arxiv.org/html/2609.30935#A4.T7)\. Following the standardMMLUprotocol, the development split is used to construct 5\-shot contexts, while performance is measured on the test split\. Importantly, consistent with our continual learning setup, we interleave evaluation with training by testing the LLM on the fullMMLUbenchmark after each task update\. This design allows us to track the evolution of general\-purpose knowledge throughout the continual fine\-tuning process, thereby providing a comprehensive view of the trade\-off between task specialization and generalization in LLMs\.

### D\.3Baselines Methods

We compare our method with several representative continual LLM fine\-tuning approaches as follows:

- •LoRA\[[Hu et al\., 2022](https://arxiv.org/html/2609.30935#bib.bib3)\]is a widely adopted method that isolates and reparameterizes the weight updates of a pre\-trained LLM into low\-rank matrices\. By freezing the backbone parameters and only optimizing the low\-rank adapters, LoRA significantly reduces the number of trainable parameters\.
- •LoRAMoE\[[Dou et al\., 2024](https://arxiv.org/html/2609.30935#bib.bib1)\]extends LoRA by incorporating a mixture\-of\-experts mechanism into the low\-rank adaptation process\. Specifically, multiple LoRA experts are maintained, and a routing strategy dynamically selects or combines these experts for each input, thereby improving model capacity and adaptability across tasks\.
- •O\-LoRA\[[Wang et al\., 2023](https://arxiv.org/html/2609.30935#bib.bib7)\]introduces orthogonality constraints on the LoRA update matrices to mitigate interference between tasks\. By encouraging the updates of different tasks to lie in orthogonal subspaces, O\-LoRA reduces catastrophic forgetting while preserving the efficiency of low\-rank adaptation\.
- •GainLoRA\[[Liang et al\., 2025](https://arxiv.org/html/2609.30935#bib.bib10)\]further enhances the LoRA\-based continual learning paradigm by introducing a gated integration mechanism over multiple LoRA branches\. For each new task, a new LoRA branch is added, and learnable gating modules are used to control the contribution of each branch\. This design effectively suppresses the influence of newly learned parameters on previous tasks, thereby alleviating forgetting without requiring task identity during inference\.
- •CLoRA[Lu et al\. \[2025\]](https://arxiv.org/html/2609.30935#bib.bib21)is a learning\-based method that imposes subspace regularization on LoRA updates\. Instead of only constraining the rank of the update, CLoRA further restricts the direction of the update matrix by enforcing orthogonality with respect to a predefined subspace\. This strategy reduces the magnitude of output perturbations while maintaining sufficient model capacity, achieving a better trade\-off between adaptation and forgetting\.

These baselines collectively cover standard LoRA fine\-tuning variants and state\-of\-the\-art continual fine\-tuning strategies, providing a comprehensive comparison for evaluating our method\.

### D\.4Evaluation Metrics

To rigorously assess both task\-specific adaptation and the preservation of previously acquired general\-purpose knowledge during continual fine\-tuning, we define specific performance and forgetting metrics for theSuperNIandMMLUbenchmarks\.

SuperNImetrics\.Given thatSuperNIcomprises diverse generation tasks, we utilize the standardRougescore \(specificallyRouge\-L\) to measure the generation quality by computing the longest common subsequence overlap between the LLM’s prediction and the ground\-truth reference\. To evaluate the stability of the LLM against catastrophic forgetting on sequential tasks, we introduce theforgetting rate\(Fgt\.\)\. For a continually fine\-tuned LLM, letRt,tR\_\{t,t\}denote theRougescore evaluated on tasktt’s test set immediately after the LLM finishes training on tasktt\. LetRT,tR\_\{T,t\}denote theRougescore on taskttafter the LLM has sequentially finished training on allTTtasks\. The forgetting rate for taskttis defined as the performance drop between these two stages:

FRt=Rt,t−RT,t\.\\mathrm\{FR\}\_\{t\}=R\_\{t,t\}\-R\_\{T,t\}\.The overall forgetting rate onSuperNIis calculated as the average ofFRt\\mathrm\{FR\}\_\{t\}across allTTlearned tasks:FRSuperNI=1T​∑t=1TFRt\\mathrm\{FR\}\_\{\\textit\{SuperNI\}\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathrm\{FR\}\_\{t\}\. A lower forgetting rate indicates that the model better retains the knowledge of previously learned tasks\.

MMLUmetrics\.SinceMMLUis a multi\-choice selection dataset designed to test general\-purpose knowledge, we useaccuracyas the primary metric\. The overall accuracy is computed by pooling all evaluated questions across the selected subjects\. To measure the degradation of the LLM’s pre\-trained knowledge after adapting to the sequence of downstream tasks, we define theMMLUforgetting rate\. LetA0A\_\{0\}denote theMMLUaccuracy of the original pre\-training LLM before any continual fine\-tuning\. LetATA\_\{T\}denote the MMLU accuracy of the LLM after it has been trained on allTTtasks in the continual learning stream\. The general\-purpose knowledge forgetting rate is defined as:

FRMMLU=A0−AT\.\\mathrm\{FR\}\_\{\\textit\{MMLU\}\}=A\_\{0\}\-A\_\{T\}\.This metric explicitly quantifies the trade\-off between acquiring new task\-specific skills and preserving the pre\-training model’s broad, general\-purpose knowledge\.

### D\.5Implementation Details

Our experiments are implemented in PyTorch based on Hugging FaceTransformersandPEFT\. We fine\-tune the backbone LLMs with LoRA while keeping the original pre\-training LLM parameters frozen\. During fine\-tuning, gradient checkpointing is enabled to lower activation memory consumption, and the input sequence length is truncated to 1024 tokens with the maximum generation length set to 50 tokens\. We optimize the trainable parameters with AdamW using an initial learning rate of2×10−42\\times 10^\{\-4\}\. To ensure reproducibility, the task order is shuffled once per run under a fixed random seed, and evaluation is performed after each task to track both task acquisition and forgetting over the course of continual learning\. ForMMLUevaluation, we follow the standard 5\-shot protocol and set the maximum context length to 2048 tokens\. If the composed prompt exceeds the context budget, the number of demonstrations is reduced until the input fits within the limit\. During inference, the LLM predicts among the four candidate answers by computing the logits at the final token position and selecting the option with the highest probability\. We use batch size 1 for MMLU to preserve the integrity of the few\-shot prompt, and this evaluation is run in the same continual\-learning pipeline asSuperNIto obtain a consistent estimate of both in\-domain adaptation and out\-of\-domain retention\.

Stable optimization of pre\-training gradient estimation\.As introduced in Sec\.[3\.2](https://arxiv.org/html/2609.30935#S3.SS2),EoupCTemploys the Gumbel\-Softmax relaxation to maintain a differentiable computational graph across auto\-regressive decoding steps\. However, applying Gumbel\-Softmax over the entire vocabulary𝒱\\mathcal\{V\}\(typically over 30k tokens for modern LLMs\) across a long sequence can lead to severe memory consumption and gradient instability \(*e\.g\.,*vanishing or exploding gradients\)\. To address this, we introduce two engineering adaptations\. First, we restrict the surrogate sequence length to a moderate size \(*e\.g\.,*M=16M=16or3232\)\. Empirical observations suggest that the model does not need to generate excessively long context to expose general\-purpose knowledge vulnerabilities; a short, highly\-targeted surrogate sequence𝐒⁡\(𝐏t\)\\mathbf\{S\}\(\\mathbf\{P\}\_\{t\}\)optimized via the virtual step is entirely sufficient to anchor the general\-purpose knowledge boundaries\. Second, to alleviate the memory overhead during the softmax expectation, we apply a dynamic vocabulary truncation strategy: at each generation stepmm, we only compute the Gumbel\-Softmax over the top\-50 highest probability tokens in𝐳m\\mathbf\{z\}\_\{m\}, effectively zeroing out the long\-tail noise and drastically reducing the size of the differentiable computational graph\.

## Appendix EPerformance Dynamics onSuperNI

In Tables[8](https://arxiv.org/html/2609.30935#A5.T8)and[9](https://arxiv.org/html/2609.30935#A5.T9), we compare the performance dynamics of our approach against the baseline methods across each task inSuperNI, recorded sequentially after the training of every task\. Our results demonstrate that our method outperforms the baselines in the vast majority of scenarios\.

Table 8:Performance dynamics during continual fine\-tuning, using theQwen3\-4Bmodel with theEoupCTmethod \(Rouge= 50\.7, Forget Rate = 6\.0\)\.Evaluation Task ID591073168787515726391510159051118136317290021290748Training Task ID59167\.280\.142\.541\.032\.68\.224\.211\.214\.813\.960\.610\.677\.118\.842\.907368\.981\.061\.445\.032\.07\.215\.313\.714\.313\.892\.010\.578\.219\.142\.8168766\.877\.086\.042\.032\.47\.354\.413\.614\.411\.691\.010\.977\.618\.743\.987562\.977\.085\.081\.035\.26\.674\.812\.013\.423\.884\.010\.273\.820\.743\.3157261\.067\.086\.074\.040\.77\.085\.910\.514\.829\.787\.014\.173\.518\.244\.563952\.459\.082\.078\.035\.111\.673\.512\.314\.129\.384\.012\.869\.418\.941\.1151060\.765\.082\.078\.038\.18\.5100\.014\.014\.536\.386\.013\.773\.119\.643\.0159054\.170\.082\.074\.028\.84\.199\.710\.813\.729\.380\.010\.069\.618\.341\.851152\.275\.076\.073\.024\.54\.599\.36\.914\.929\.987\.06\.970\.020\.540\.218160\.070\.083\.070\.018\.91\.699\.77\.815\.271\.384\.06\.772\.618\.941\.436358\.873\.082\.072\.023\.36\.298\.27\.715\.372\.187\.08\.674\.519\.441\.8172957\.874\.082\.070\.024\.18\.997\.811\.113\.668\.989\.014\.675\.421\.441\.200259\.871\.080\.068\.029\.76\.298\.89\.315\.266\.190\.015\.378\.619\.944\.3129059\.166\.078\.070\.018\.68\.494\.27\.510\.369\.087\.014\.477\.123\.535\.474858\.870\.074\.068\.030\.57\.892\.811\.915\.669\.288\.014\.171\.923\.364\.8Table 9:Performance dynamics during continual fine\-tuning, using theQwen3\-4Bmodel with theGainLoRAmethod \(Rouge= 48\.4, Forget Rate = 9\.0\)\.Evaluation Task ID591073168787515726391510159051118136317290021290748Training Task ID59167\.879\.078\.043\.034\.34\.095\.712\.814\.612\.388\.011\.977\.119\.543\.607366\.378\.075\.042\.035\.29\.420\.210\.314\.914\.291\.010\.476\.919\.243\.3168766\.780\.086\.038\.034\.76\.389\.911\.414\.916\.189\.011\.976\.818\.943\.587561\.166\.082\.083\.037\.25\.392\.714\.013\.96\.279\.011\.475\.218\.342\.6157255\.269\.080\.077\.044\.84\.393\.39\.114\.528\.282\.015\.976\.518\.743\.663952\.663\.077\.074\.036\.010\.584\.114\.213\.436\.180\.013\.677\.317\.043\.2151054\.965\.075\.076\.039\.712\.398\.213\.414\.226\.683\.013\.973\.117\.543\.9159051\.666\.076\.077\.031\.59\.699\.012\.012\.627\.885\.010\.972\.616\.943\.051150\.966\.081\.072\.028\.34\.898\.37\.014\.929\.784\.08\.773\.520\.237\.518152\.864\.077\.073\.030\.95\.599\.05\.613\.975\.380\.09\.776\.219\.435\.936350\.865\.074\.070\.021\.04\.399\.75\.513\.864\.191\.011\.176\.018\.936\.7172951\.166\.079\.064\.027\.86\.599\.08\.012\.467\.389\.013\.777\.918\.725\.200243\.471\.078\.062\.029\.47\.999\.08\.913\.661\.490\.014\.778\.818\.428\.0129050\.369\.075\.060\.015\.06\.792\.29\.58\.367\.990\.013\.276\.124\.622\.374848\.167\.076\.054\.025\.97\.599\.011\.910\.862\.489\.013\.971\.424\.365\.3
## Appendix FLimitations

WhileEoupCTeffectively addresses the catastrophic forgetting in continual LLM fine\-tuning without requiring access to original pre\-training data, it possesses a few limitations that warrant future exploration\. First, the differentiable generation mechanism naturally introduces additional computational overhead during the training phase\. AlthoughEoupCTcompletely circumvents the massive storage burden of maintaining historical pre\-training datasets, auto\-regressively generating pseudo data via Gumbel\-Softmax increases the latency of the forward and backward passes compared to standard LoRA methods\. We provide a comprehensive computational complexity analysis in Sec\.[B](https://arxiv.org/html/2609.30935#A2)and introduce several implementation refinements to minimize the overhead in Sec\.[D\.5](https://arxiv.org/html/2609.30935#A4.SS5)\.

Second, the representational capacity of the generated pseudo data sequence is inherently bounded by the predefined length of the soft promptsLLand generation stepsMM, and we provide a sensitivity experiment in Sec\.[4\.2](https://arxiv.org/html/2609.30935#S4.SS2)\. While our adversarial virtual step approximation guarantees that these sequences precisely target the most vulnerable boundaries of general\-purpose knowledge, dealing with extremely long and highly heterogeneous task sequences might eventually saturate this representational capacity\. Future work could explore dynamically expanding prompt lengths or incorporating hierarchical prompt routing to further scale the generation mechanism\.

## Appendix GLLM Usage

LLMs were utilized solely for grammatical refinement and linguistic polishing of the manuscript; the core ideas, methodology, and results were developed independently by the authors\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification:Our primary motivation is the observation that orthogonal gradient projection methods fail to preserve general\-purpose knowledge due to unknown pre\-training gradients\. Therefore, we introduceEoupCTto estimate and orthogonalize these unknown pre\-training gradients\. These contributions are detailed extensively in our Abstract and Introduction\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification:In Appendix[F](https://arxiv.org/html/2609.30935#A6), we provide a thorough discussion regarding the potential limitations of this work, focusing primarily on computational efficiency, which remains a critical area for prospective investigation\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification:In Appendix[A](https://arxiv.org/html/2609.30935#A1), we provide rigorous theoretical proofs for the claims and underlying assumptions, specifically addressing the three primary theoretical questions, presented throughout this work\.
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification:In Sec\.[4](https://arxiv.org/html/2609.30935#S4), we present comprehensive experimental results and analyses\. Furthermore, Appendix[D](https://arxiv.org/html/2609.30935#A4)is dedicated to detailing the essential information of our experiments, including LLM architectures, baseline methods, datasets, and implementation details\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
24. Justification:We have provided the source code for our implementation, accessible via the link included in Sec\.[1](https://arxiv.org/html/2609.30935#S1)\.
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification:Comprehensive details regarding the experimental setup are provided in Appendix[D](https://arxiv.org/html/2609.30935#A4)\. Furthermore, we perform a sensitivity analysis in Sec\.[4\.2](https://arxiv.org/html/2609.30935#S4.SS2)to delineate the selection process for key hyper\-parameters,*e\.g\.,*LLandMM\.
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification:The experiments presented in Table[1](https://arxiv.org/html/2609.30935#S4.T1)were conducted across three independent trials using distinct random seeds, each corresponding to a different task order\. We report the aggregated results from these trials, which consistently demonstrate statistical significance\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification:Appendix[B\.3](https://arxiv.org/html/2609.30935#A2.SS3)provides a comprehensive breakdown of the computational runtime measured across our 8\-GPU cluster\.
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification:We have thoroughly reviewed the Code of Ethics and certify that our research adheres to all prescribed ethical standards and institutional guidelines\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[N/A\]
49. Justification:This research focuses on the continual LLM fine\-tuning, utilizing exclusively open\-source instruction\-tuning datasets, such as those for sentiment classification and text summarization\. Consequently, this work does not present direct negative societal impacts regarding data privacy or sensitive information\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification:Our experiments are conducted using exclusively publicly available datasets and open\-source LLMs\. Consequently, this work presents no foreseeable risk regarding the misuse of models or datasets, as all resources are utilized in accordance with their respective open\-source licenses and intended research purposes\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification:All datasets and LLMs employed in this study are open\-source, and their respective origins have been duly cited\. Consequently, there are no unresolved licensing issues, as our use of these resources strictly adheres to their original distribution terms\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[Yes\]
64. Justification:We have open\-sourced the implementation code for our experiments and provided comprehensive documentation regarding its usage\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification:Our experiments do not require any additional crowdsourced data collection; all analyses are conducted using existing, publicly available benchmarks\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification:Our experiments do not require any additional crowdsourced data collection; all analyses are conducted using existing, publicly available benchmarks\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[Yes\]
79. Justification:In Appendix[G](https://arxiv.org/html/2609.30935#A7), we disclose that LLMs were employed strictly for grammatical proofreading and linguistic refinement to enhance the readability of the manuscript\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.