ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning

# ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Source: [https://arxiv.org/html/2607.07719](https://arxiv.org/html/2607.07719)
###### Abstract

Parameter\-efficient fine\-tuning adapts a large language model to one task cheaply, but across a task sequence LoRA\-style methods keep stacking low\-rank updates on the same frozen weight, so each new task tends to overwrite the previous ones\. We present*ReCoLoRA*\(Recursive Consolidation of Low\-Rank Adapters\), a spectrum\-aware framework for continual fine\-tuning: adapters are initialized from a randomized SVD of the pretrained weight, per\-layer effective ranks are selected by an elbow criterion, and the principal subspace is adapted before residual capacity is opened\. Before each new task, ReCoLoRA re\-decomposes the*current*effective weight, rather than the original one, into a frozen residual, a slowly updated principal component, and a fresh adapter \(recursive consolidation\), so every task starts from the model that has already absorbed its predecessors\. On a six\-task continual GLUE sequence over four 7–8B backbones, ReCoLoRA attains the best final average score on three of the four backbones against rank\-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines while training fewer parameters; an oracle\-routed task\-bank variant serves as an upper bound under full task isolation\. Code:[https://github\.com/bhqy666/ReCoLoRA](https://github.com/bhqy666/ReCoLoRA)\.

## 1Introduction

Large language models \(LLMs\) are usually trained on static corpora, but deployed systems must adapt to changing tasks, domains, and user requirements\. Re\-training a full LLM whenever new data arrives is often infeasible, so downstream adaptation is commonly performed with PEFT\. LoRA\[[1](https://arxiv.org/html/2607.07719#bib.bib1)\]is a representative PEFT method: it freezes the pretrained weight matrixW0W\_\{0\}and learns a low\-rank update

Weff=W0\+Δ​W,Δ​W=B​A,W\_\{\\mathrm\{eff\}\}=W\_\{0\}\+\\Delta W,\\qquad\\Delta W=BA,\(1\)whereB∈ℝdo​u​t×rB\\in\\mathbb\{R\}^\{d\_\{out\}\\times r\},A∈ℝr×di​nA\\in\\mathbb\{R\}^\{r\\times d\_\{in\}\}, andr≪min⁡\(do​u​t,di​n\)r\\ll\\min\(d\_\{out\},d\_\{in\}\)\. LoRA greatly reduces trainable parameters and is easy to deploy because the adapter can be merged into the base weight after training\.

However, the usual single\-task PEFT assumption is too weak for continual use\. In continual learning, data or tasks arrive sequentially, and the learner must acquire new knowledge while retaining useful behavior from previous stages\[[2](https://arxiv.org/html/2607.07719#bib.bib2),[3](https://arxiv.org/html/2607.07719#bib.bib3)\]\. This creates the classical stability\-plasticity dilemma: a model with high plasticity quickly adapts to a new task but may forget old tasks, while a highly stable model preserves old behavior but may underfit new tasks\. Recent LLM studies show that catastrophic forgetting can also occur during continual instruction tuning and fine\-tuning, including degradation in domain knowledge, reasoning, and reading\-comprehension capabilities\[[4](https://arxiv.org/html/2607.07719#bib.bib4),[5](https://arxiv.org/html/2607.07719#bib.bib5)\]\. Thus, PEFT does not automatically solve forgetting; low\-rank adapters can still over\-specialize to the latest task\.

Existing LoRA variants address related issues\. AdaLoRA\[[6](https://arxiv.org/html/2607.07719#bib.bib6)\]dynamically reallocates rank budgets during training\. PiSSA\[[7](https://arxiv.org/html/2607.07719#bib.bib7)\]initializes adapters from principal singular vectors of pretrained weights, improving convergence over random LoRA initialization\. DoRA\[[8](https://arxiv.org/html/2607.07719#bib.bib8)\]separates weight magnitude and direction to improve adapter expressiveness\. Orthogonal LoRA methods such as O\-LoRA\[[9](https://arxiv.org/html/2607.07719#bib.bib9)\]explicitly reduce interference by assigning new tasks to orthogonal low\-rank subspaces\. These methods motivate our central question: can we use the pretrained weight spectrum itself to decide which subspaces should change first, and which should be recovered later?

We propose ReCoLoRA \(Recursive Consolidation of Low\-Rank Adapters\), a spectral and staged variant of LoRA for continual fine\-tuning\. ReCoLoRA decomposes each pretrained weight into a principal and a residual subspace, initializes the principal adapter with randomized SVD, selects an effective rank per layer with an elbow criterion, and trains in two stages: the first stage adapts the principal subspace, which carries the dominant pretrained structure, and the second stage gradually opens up residual capacity when the principal subspace alone cannot fit the task\. The harder question is what to do across a sequence of tasks\. A natural first attempt, which we tried, is to protect old knowledge inside a single shared adapter by freezing the ranks it has already used or by anchoring it in parameter space; our experiments show this is unreliable, because strong protection suppresses plasticity while weak protection lets the old behavior drift\. The core of ReCoLoRA is therefore a different mechanism, recursive consolidation\. Instead of forever adding new updates on top of the original frozen weightW0W\_\{0\}, ReCoLoRA folds each finished task back into the model: before the next task it re\-decomposes the current effective weight into a frozen residual, a slowly trainable elbow\-compressed principal component, and a fresh fast adapter\. Each task thus departs from the consolidated model, and the knowledge of earlier tasks is preserved in the slow principal subspace rather than left exposed to overwriting\. The result is a single evolving model that can be deployed directly\. To measure how much an explicit memory mechanism can recover, we also study ReCoLoRA\-TaskBank, which trains one isolated spectrum\-initialized ReCoLoRA branch per task, freezes finished branches, and routes each evaluation to its own branch with an oracle\. TaskBank removes overwriting by construction; we read it as an upper bound rather than a deployable, task\-agnostic solution\.

Our contributions are:

- •We formulate ReCoLoRA, a spectrum\-aware PEFT framework for continual fine\-tuning that initializes adapters from a randomized SVD of the pretrained weight, selects an effective rank per layer with an elbow criterion, and adapts the principal subspace before the residual one\.
- •We introduce recursive consolidation as the continual mechanism of ReCoLoRA: each finished task is folded back into the model by re\-decomposing the current effective weight into a frozen residual, a slowly trainable principal component, and a fresh adapter\. This produces a single deployable model and, in our experiments, retains old tasks more reliably than freezing or anchoring ranks inside one shared adapter\.
- •We introduce ReCoLoRA\-TaskBank, an oracle\-routed task\-incremental variant that isolates one spectrum\-initialized branch per task\. It removes overwriting by construction and serves as an upper bound on retention\.
- •On a six\-task continual GLUE sequence over Qwen3\-8B, Llama\-3\.1\-8B\-Instruct, Mistral\-7B\-v0\.3, and InternLM2\.5\-7B\-Chat, and against rank\-swept baselines, ReCoLoRA achieves the best final average score on three of the four backbones with far fewer trainable parameters than the strongest rank\-256 baseline, while ReCoLoRA\-TaskBank attains0\.8957±0\.00260\.8957\\pm 0\.0026final average score with0\.0000±0\.00000\.0000\\pm 0\.0000average forgetting on Qwen3\-8B across three seeds under oracle routing\.

## 2Background and Related Work

### 2\.1Continual Learning Settings and Metrics

Continual learning studies models that receive data in a sequence rather than as a fixed independent and identically distributed training set\. General surveys distinguish several common settings\[[2](https://arxiv.org/html/2607.07719#bib.bib2)\]: instance\-incremental learning changes the input stream within a task; domain\-incremental learning changes input distributions while keeping label semantics similar; task\-incremental learning introduces a sequence of tasks, often with task identity available at evaluation; class\-incremental learning introduces new labels without task identity at test time; online continual learning restricts training to a single or small number of passes over streaming data\. LLM continual learning is often organized by training stage: continual pretraining updates facts, domains, or languages; continual instruction tuning updates tasks, skills, and domains; continual alignment updates preferences or values\[[5](https://arxiv.org/html/2607.07719#bib.bib5)\]\.

This paper focuses on task\-incremental continual fine\-tuning\. We sequentially train one model on a stream of downstream language understanding tasks\. The model is evaluated after each stage on all tasks seen so far\. This setting is close to continual instruction\-style adaptation because each classification task is converted into a natural\-language prompt and solved with a causal language model\. It does not use replay data from previous tasks, and all baseline methods share the same task order and evaluation protocol\.

Typical NLP/LLM continual learning benchmarks include GLUE and SuperGLUE\-style task sequences\[[10](https://arxiv.org/html/2607.07719#bib.bib10),[11](https://arxiv.org/html/2607.07719#bib.bib11)\], lifelong language learning streams such as the five\-task LAMOL setting\[[12](https://arxiv.org/html/2607.07719#bib.bib12)\], instruction\-tuning collections such as T0/FLAN\-style task mixtures\[[13](https://arxiv.org/html/2607.07719#bib.bib13)\], and capability\-retention suites that evaluate domain knowledge, reasoning, and reading comprehension after fine\-tuning\[[4](https://arxiv.org/html/2607.07719#bib.bib4)\]\. For the first complete implementation we choose GLUE because it is public, lightweight enough for repeated 8B\-model runs, and gives heterogeneous task formats: sentiment classification, paraphrase detection, entailment, question answering as entailment, duplicate\-question detection, and multi\-genre inference\.

Letai,ja\_\{i,j\}be the score on taskjjafter the model has trained through taskii\. The final average score afterTTtasks is

FinalAvg=1T​∑j=1TaT,j\.\\mathrm\{FinalAvg\}=\\frac\{1\}\{T\}\\sum\_\{j=1\}^\{T\}a\_\{T,j\}\.\(2\)A standard forgetting measure compares the final score on each task with the best score observed earlier in the sequence\[[2](https://arxiv.org/html/2607.07719#bib.bib2)\]:

Fj=maxi∈\{j,…,T\}⁡ai,j−aT,j,AvgForget=1T​∑j=1TFj\.F\_\{j\}=\\max\_\{i\\in\\\{j,\\ldots,T\\\}\}a\_\{i,j\}\-a\_\{T,j\},\\qquad\\mathrm\{AvgForget\}=\\frac\{1\}\{T\}\\sum\_\{j=1\}^\{T\}F\_\{j\}\.\(3\)Note thatFT=0F\_\{T\}=0by construction: the last task in the sequence is evaluated only once, immediately after its own training, so it cannot register forgetting regardless of method\. AvgForget therefore depends not only on the method but also on which task occupies the last position\. We report FinalAvg and AvgForget, which capture the two dimensions most relevant to this paper: final utility and memory stability\.

### 2\.2PEFT and Low\-Rank Adaptation

LoRA\[[1](https://arxiv.org/html/2607.07719#bib.bib1)\]freezes the backbone and learns low\-rank updates\. Subsequent work improves LoRA from several directions\. AdaLoRA\[[6](https://arxiv.org/html/2607.07719#bib.bib6)\]allocates rank adaptively by estimating component importance\. DyLoRA\[[14](https://arxiv.org/html/2607.07719#bib.bib14)\]supports multiple ranks from one trained adapter\. DoRA\[[8](https://arxiv.org/html/2607.07719#bib.bib8)\]decomposes pretrained weights into magnitude and direction\. VeRA\[[15](https://arxiv.org/html/2607.07719#bib.bib15)\]reduces trainable parameters by sharing random projections\. LoRA\+\[[16](https://arxiv.org/html/2607.07719#bib.bib16)\]improves optimization by using different learning rates for LoRA factors\.

### 2\.3Continual Learning for NLP and LLMs

Continual learning in NLP differs from vision because tasks may have different input\-output formats, label semantics, and instruction styles\[[3](https://arxiv.org/html/2607.07719#bib.bib3)\]\. LAMOL\[[12](https://arxiv.org/html/2607.07719#bib.bib12)\]is an early language lifelong learning method that trains a language model to solve tasks and generate pseudo\-samples for replay\. Continual\-T0\[[13](https://arxiv.org/html/2607.07719#bib.bib13)\]studies continual instruction learning over many tasks and argues that instruction\-tuned language models can retain broad skills under appropriate training\. Progressive Prompts\[[17](https://arxiv.org/html/2607.07719#bib.bib17)\]keeps the base model frozen and learns a new soft prompt for each task, concatenating prompts sequentially to reduce forgetting\.

For LoRA\-style continual learning, O\-LoRA\[[9](https://arxiv.org/html/2607.07719#bib.bib9)\]learns new tasks in orthogonal low\-rank subspaces and fixes the old ones to limit interference, while C\-LoRA\[[18](https://arxiv.org/html/2607.07719#bib.bib18)\]uses a routing matrix to manage parameter updates across tasks so that learned subspaces can be reused without maintaining many independent adapters\. ReCoLoRA shares the view that subspace interference is the central problem, but addresses it differently: it starts from the pretrained weight spectrum and, after each task, recomputes a slow principal subspace from the current effective weight, so earlier knowledge is carried forward inside one evolving model\. The TaskBank variant sits at the opposite end, close to adapter\-isolation and routing methods: it allocates one spectrum\-initialized branch per task and, in this paper, uses oracle routing as an upper bound before a learned, input\-conditioned router is introduced\.

## 3Method

ReCoLoRA modifies LoRA along three axes: spectral initialization, adaptive effective rank, and staged residual recovery\. Figure[1](https://arxiv.org/html/2607.07719#S3.F1)summarizes the pipeline\.

The method is designed around two observations\. First, pretrained transformer weights are not isotropic: their singular spectra often show a small number of dominant directions followed by a long tail\. Second, catastrophic forgetting in sequential fine\-tuning is caused not only by parameter count, but also by where the update is applied\. Random low\-rank factors can use their capacity in directions that fit the current task while interfering with previous behavior\. ReCoLoRA therefore asks the optimizer to first use directions already important to the pretrained model, and only later use residual capacity\.

![Refer to caption](https://arxiv.org/html/2607.07719v1/x1.png)Figure 1:ReCoLoRA pipeline\. Each target pretrained weight is decomposed approximately with randomized SVD\. An elbow heuristic chooses the effective rank and induces a rank mask\. Training first adapts the principal subspace, then gradually recovers residual capacity through adapter parameters rather than unfreezing the full backbone\.### 3\.1Spectral View of LoRA Updates

Consider a pretrained matrixW0∈ℝdo​u​t×di​nW\_\{0\}\\in\\mathbb\{R\}^\{d\_\{out\}\\times d\_\{in\}\}with singular value decomposition

W0=U​Σ​V⊤=Ur​Σr​Vr⊤\+U⟂​Σ⟂​V⟂⊤\.W\_\{0\}=U\\Sigma V^\{\\top\}=U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\+U\_\{\\perp\}\\Sigma\_\{\\perp\}V\_\{\\perp\}^\{\\top\}\.\(4\)The first term contains the leadingrrsingular directions and is called the principal component\. The remaining term is the residual component\. Standard LoRA does not explicitly use this decomposition; its low\-rank factors are randomly initialized and must discover useful directions through task gradients\. PiSSA uses the principal singular directions for initialization\[[7](https://arxiv.org/html/2607.07719#bib.bib7)\]\. ReCoLoRA follows the same spectral motivation but adds two continual\-learning mechanisms: adaptive rank masking and residual recovery\.

ReCoLoRA represents the effective weight as

Weff​\(t\)=W0\+Δ​Wp​\(t\)\+γ​\(t\)​Δ​Wr​\(t\),W\_\{\\mathrm\{eff\}\}\(t\)=W\_\{0\}\+\\Delta W\_\{p\}\(t\)\+\\gamma\(t\)\\Delta W\_\{r\}\(t\),\(5\)whereΔ​Wp\\Delta W\_\{p\}is the principal adapter,Δ​Wr\\Delta W\_\{r\}is the residual adapter, andγ​\(t\)∈\[0,1\]\\gamma\(t\)\\in\[0,1\]controls when residual updates affect the forward pass\. The backboneW0W\_\{0\}remains frozen\. This is important: ReCoLoRA is not full fine\-tuning of residual weights\. It is still a PEFT method, but it separates the geometry of adapter updates into principal and residual components\.

### 3\.2Randomized SVD Initialization

Direct full SVD is expensive for transformer weights\. ReCoLoRA uses randomized SVD\[[19](https://arxiv.org/html/2607.07719#bib.bib19)\]\. For a target rankrrand oversampling parameterpp, we sample a Gaussian matrixΩ∈ℝdi​n×\(r\+p\)\\Omega\\in\\mathbb\{R\}^\{d\_\{in\}\\times\(r\+p\)\}and form

Y=W0​Ω\.Y=W\_\{0\}\\Omega\.\(6\)After orthonormalizingYYintoQQ, we project the weight into the smaller subspace,

C=Q⊤​W0,C=Q^\{\\top\}W\_\{0\},\(7\)compute an SVD ofCC, and reconstruct approximate leading singular vectors ofW0W\_\{0\}\. Optional power iterations improve accuracy when the singular spectrum decays slowly\. The cost is roughlyO​\(do​u​t​di​n​r\)O\(d\_\{out\}d\_\{in\}r\), much smaller than full SVD whenr≪min⁡\(do​u​t,di​n\)r\\ll\\min\(d\_\{out\},d\_\{in\}\)\.

Given approximateUr,Σr,VrU\_\{r\},\\Sigma\_\{r\},V\_\{r\}, the principal adapter is initialized so that the initial low\-rank update is aligned withUr​Σr​Vr⊤U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\. In practice, the adapter factors are scaled according to the LoRA convention so that the effective model remains numerically stable at the start of fine\-tuning\. This initialization gives the optimizer directions that already explain high\-energy structure in the pretrained weight, reducing the need to discover them from random factors\.

### 3\.3Elbow\-Based Rank Selection and Rank Masks

A fixed rank applies the same capacity to every target matrix, even though transformer layers can have different spectral profiles\. ReCoLoRA selects an effective rank from the singular values\. Letσ1≥σ2≥⋯\\sigma\_\{1\}\\geq\\sigma\_\{2\}\\geq\\cdotsbe the singular values of a target matrix\. We search for an elbow point where the marginal spectral contribution drops sharply\. A practical criterion combines a ratio test and an energy constraint:

r∗=arg⁡maxr⁡σrσr\+1,∑i=1r∗σi2∑i=1rmax\+rr​e​sσi2≥ρ,r^\{\*\}=\\arg\\max\_\{r\}\\frac\{\\sigma\_\{r\}\}\{\\sigma\_\{r\+1\}\},\\qquad\\frac\{\\sum\_\{i=1\}^\{r^\{\*\}\}\\sigma\_\{i\}^\{2\}\}\{\\sum\_\{i=1\}^\{r\_\{\\max\}\+r\_\{res\}\}\\sigma\_\{i\}^\{2\}\}\\geq\\rho,\(8\)with clipping to a minimum and maximum rank\. The energy ratio is normalized against the truncated spectral budget that randomized SVD actually computes \(rmax\+rr​e​sr\_\{\\max\}\+r\_\{res\}singular values\), not the full matrix spectrum: for large transformer weight matrices, the energy contained in the toprmaxr\_\{\\max\}singular values is a vanishing fraction of the full\-matrix energy, so a full\-matrix normalization would makeρ\\rhounreachable and collapser∗r^\{\*\}tormaxr\_\{\\max\}for every layer\. The exact thresholds are implementation hyperparameters; in the current experiments the single\-adapter setting uses a rank\-16 budget withρ=0\.8\\rho=0\.8, which yields per\-layer principal ranks in\[8,16\]\[8,16\], while the TaskBank branches use a rank\-32 allocated budget with a smaller active elbow rank per task\.

![Refer to caption](https://arxiv.org/html/2607.07719v1/x2.png)Figure 2:Elbow/energy\-based rank selection \(Eq\.[8](https://arxiv.org/html/2607.07719#S3.E8)\) applied to a real Qwen3\-8B weight matrix \(model\.layers\.0\.self\_attn\.q\_proj,d=4096d=4096\), using the same randomized SVD \(truncated rankrmax\+rr​e​s=20r\_\{\\max\}\+r\_\{res\}=20, oversample 5, seed 42\) and selection hyperparameters \(ρ=0\.8\\rho=0\.8, elbow ratio1\.51\.5,rmax=16r\_\{\\max\}=16,rmin=2r\_\{\\min\}=2\) as the main experiments\. \(a\) Raw \(unnormalized\) singular\-value spectrum on a log\-index/linear\-value axis, in the classic “knee” style:σi\\sigma\_\{i\}drops sharply over the first∼\\sim10 indices and then flattens across the remainingd−20≈4076d\-20\\approx 4076dimensions\. The orange band marks the truncated computation budgetrmax\+rr​e​s=20r\_\{\\max\}\+r\_\{res\}=20that randomized SVD actually computes, and the selected rankr∗=8r^\{\*\}=8\(withσr∗=6\.54\\sigma\_\{r^\{\*\}\}=6\.54\) falls well inside it \(r∗≪dr^\{\*\}\\ll d\), visually confirming that only a handful of directions carry most of the matrix’s spectral mass\. \(b\) Cumulative energy ratio within the truncated budget;r∗r^\{\*\}is the smallest index at which this ratio reachesρ=0\.8\\rho=0\.8\. The rank mask keeps principal dimensions1,…,r∗1,\\ldots,r^\{\*\}active and treats the remaining allocated dimensions as residual capacity\.The chosen rank is implemented as a rank mask over the allocated adapter dimensions\. During the principal stage, only selected principal directions are active\. During residual recovery, the mask and residual ratio can be updated to expose additional capacity\. This is a lightweight mechanism for controlling plasticity without changing the backbone or allocating a new adapter for every task\.

### 3\.4Two\-Stage Principal\-to\-Residual Training

ReCoLoRA’s training schedule is designed for the stability\-plasticity trade\-off\. Stage 1 adapts only the principal adapter branch\. Because these directions correspond to high\-energy pretrained components, early updates remain close to the pretrained representation and are less likely to perturb low\-energy directions that may encode fragile but useful behavior\. Stage 2 activates residual recovery with a smaller residual learning rate or a scheduled residual ratio\. This gives the model additional degrees of freedom after the principal branch has learned the broad task structure\.

The schedule can be written as

γ​\(t\)=\{0,t<τ,g​\(t\),t≥τ,\\gamma\(t\)=\\begin\{cases\}0,&t<\\tau,\\\\ g\(t\),&t\\geq\\tau,\\end\{cases\}\(9\)whereτ\\tauis the stage boundary andg​\(t\)g\(t\)increases the residual contribution\. A simple step schedule is sufficient in the current implementation, but smoother schedules are compatible with the same parameterization\. We use separate parameter groups so that residual parameters can receive a lower learning rate\. This avoids a sudden high\-variance update when residual capacity is introduced\.

### 3\.5Recursive Consolidation

The mechanisms described so far still treat the original pretrained weightW0W\_\{0\}as a permanent reference\. This is a useful PEFT constraint, but it is a poor model of continual adaptation: after a task is learned, the next task should start from the consolidated effective model rather than repeatedly attaching new updates to the same untouchedW0W\_\{0\}\. ReCoLoRA therefore decomposes the effective weight before each task into a frozen residual, a slowly trainable principal component, and a fresh fast adapter:

Weff\(t\)=Wres\(t\)\+Wslow\(t\)\+Δ​Wfast\(t\)\.W\_\{\\mathrm\{eff\}\}^\{\(t\)\}=W\_\{\\mathrm\{res\}\}^\{\(t\)\}\+W\_\{\\mathrm\{slow\}\}^\{\(t\)\}\+\\Delta W\_\{\\mathrm\{fast\}\}^\{\(t\)\}\.\(10\)For the first task, letWeff\(0\)=W0W\_\{\\mathrm\{eff\}\}^\{\(0\)\}=W\_\{0\}\. Before tasktt, we compute an elbow rankktk\_\{t\}from the spectrum of the previous effective weight and define

Wslow\(t\)=topSVD​\(Weff\(t−1\),kt\),Wres\(t\)=Weff\(t−1\)−Wslow\(t\)\.W\_\{\\mathrm\{slow\}\}^\{\(t\)\}=\\mathrm\{topSVD\}\(W\_\{\\mathrm\{eff\}\}^\{\(t\-1\)\},k\_\{t\}\),\\qquad W\_\{\\mathrm\{res\}\}^\{\(t\)\}=W\_\{\\mathrm\{eff\}\}^\{\(t\-1\)\}\-W\_\{\\mathrm\{slow\}\}^\{\(t\)\}\.\(11\)The slow component is stored as two low\-rank factors and is updated with a much smaller effective learning rate or a delayed ramp, so dominant pretrained and previously consolidated directions change only gradually\. The fast component is initialized near zero and receives most short\-term task plasticity\. Its rank is tied to the layer\-specific elbow rank, for examplekt\+3k\_\{t\}\+3or a small percentage increase clipped by the maximum\-rank budget\.

At the task boundary, the trained weight

W^\(t\)=Wres\(t\)\+Wslow\(t\)\+Δ​Wfast\(t\)\\widehat\{W\}^\{\(t\)\}=W\_\{\\mathrm\{res\}\}^\{\(t\)\}\+W\_\{\\mathrm\{slow\}\}^\{\(t\)\}\+\\Delta W\_\{\\mathrm\{fast\}\}^\{\(t\)\}\(12\)is consolidated by SVD again\. The next task receives a new pair\(Wres\(t\+1\),Wslow\(t\+1\)\)\(W\_\{\\mathrm\{res\}\}^\{\(t\+1\)\},W\_\{\\mathrm\{slow\}\}^\{\(t\+1\)\}\)fromW^\(t\)\\widehat\{W\}^\{\(t\)\}, andΔ​Wfast\(t\+1\)\\Delta W\_\{\\mathrm\{fast\}\}^\{\(t\+1\)\}is reinitialized\. Thus, low\-value rank directions can be pruned, layers whose spectra require more capacity can grow, and the model no longer accumulates all tasks as unrelated deltas on the same original backbone\. This variant remains parameter\-efficient during each task, but it explicitly allows the consolidated effective weight to drift slowly over a task sequence\.

### 3\.6ReCoLoRA\-TaskBank: Adapter Isolation with Oracle Routing

Recursive consolidation lets one model evolve, but it still merges every task into a shared set of weights\. When task identity is available at evaluation, a stronger guarantee is possible: give each task its own parameters so that later tasks cannot touch earlier ones\. Approximating this inside a single shared adapter—by freezing or anchoring old ranks \(Appendix[A](https://arxiv.org/html/2607.07719#A1)\)—is unreliable, because hard freezing reduces plasticity, soft freezing lets old ranks drift, and parameter\-space anchoring does not preserve old\-task outputs\. ReCoLoRA\-TaskBank instead prevents overwriting by construction\.

For a task sequence\{D1,…,DT\}\\\{D\_\{1\},\\ldots,D\_\{T\}\\\}, ReCoLoRA\-TaskBank trains a separate adapter branchϕt\\phi\_\{t\}for each task:

ft​\(x\)=fθ0,ϕt​\(x\),t∈\{1,…,T\},f\_\{t\}\(x\)=f\_\{\\theta\_\{0\},\\phi\_\{t\}\}\(x\),\\qquad t\\in\\\{1,\\ldots,T\\\},\(13\)where the backboneθ0\\theta\_\{0\}is frozen and each branchϕt\\phi\_\{t\}is initialized with the same ReCoLoRA procedure: randomized SVD, elbow\-based effective rank, and principal\-to\-residual two\-stage training\. After taskttis trained,ϕt\\phi\_\{t\}is frozen\. Later tasks train only their own branches, so their updates cannot overwrite the parameters used by earlier tasks\.

In this manuscript we evaluate the task\-bank with oracle task routing: when evaluating taskjj, the model uses branchϕj\\phi\_\{j\}\. This matches the task\-incremental continual\-learning assumption where task identity is available at test time\. It should be read as an upper\-bound experiment rather than a complete task\-agnostic deployment solution\. The next step is to replace the oracle with a learned router or gated mixture that selects branches from the input prompt\.

Table 1:Decision\-level ReCoLoRA training procedure\. The backbone is frozen in the static and TaskBank variants; the default ReCoLoRA recursive consolidation instead freezes only the residual part of the previous consolidated effective weight while slowly updating the compressed principal component and training a fresh fast adapter\.

## 4Experimental Design

### 4\.1Continual GLUE Protocol

We evaluate causal language models using a sequential GLUE task stream\[[10](https://arxiv.org/html/2607.07719#bib.bib10)\]\. The main experiments use Qwen3\-8B, and the same protocol is repeated on Llama\-3\.1\-8B\-Instruct as a backbone validation\. A focused Llama\-3\-8B study \(Appendix[B](https://arxiv.org/html/2607.07719#A2)\) tests the task\-wise rank\-freezing mechanism after we observe that simple rank expansion alone does not reliably reduce forgetting\. Finally, we evaluate ReCoLoRA against LoRA, PiSSA, AdaLoRA, and DoRA on Mistral\-7B\-v0\.3 and InternLM2\.5\-7B\-Chat to test whether recursive consolidation transfers beyond the Qwen/Llama experiments\. The task order is

SST​\-​2→MRPC→QNLI→RTE→QQP→MNLI\.\\mathrm\{SST\\mbox\{\-\}2\}\\rightarrow\\mathrm\{MRPC\}\\rightarrow\\mathrm\{QNLI\}\\rightarrow\\mathrm\{RTE\}\\rightarrow\\mathrm\{QQP\}\\rightarrow\\mathrm\{MNLI\}\.\(14\)Each classification task is converted into a prompt with verbalized candidate labels\. During training, the model is optimized to generate the correct label token sequence\. During evaluation, each candidate label is scored by sequence loss and the lowest\-loss label is selected\. After training on taskii, the model is evaluated on all tasks1,…,i1,\\ldots,i, giving a lower\-triangular evaluation matrix\.

The resulting matrix has the form shown in Table[2](https://arxiv.org/html/2607.07719#S4.T2)\. Entries above the diagonal are intentionally blank because future tasks are not evaluated before they are introduced in the current protocol\. The diagonal measures immediate plasticity on the newly learned task, while changes down each column measure forgetting or backward transfer\.

Table 2:Task\-by\-stage evaluation matrix used in the continual GLUE protocol\. FinalAvg uses the last row; AvgForget compares each column’s final value with its best previous value\.SST\-2MRPCQNLIRTEQQPMNLIStage 1Stage 2Stage 3Stage 4Stage 5Stage 6After each stage, evaluate on all tasks seen so far and update the task\-by\-stage matrixai,ja\_\{i,j\}\.Figure 3:Continual fine\-tuning protocol\. The model sees one downstream task at a time\. After every stage, all previously seen tasks are evaluated, which allows us to compute final average score, average forgetting, and transfer\-related metrics\.
### 4\.2Baselines

We compare the following PEFT methods:

- •LoRA: standard fixed\-rank low\-rank adaptation with random initialization\.
- •PiSSA: principal singular vector initialization for LoRA adapters\[[7](https://arxiv.org/html/2607.07719#bib.bib7)\]\.
- •AdaLoRA: adaptive rank allocation during training\[[6](https://arxiv.org/html/2607.07719#bib.bib6)\]\.
- •DoRA: weight\-decomposed LoRA with magnitude\-direction separation\[[8](https://arxiv.org/html/2607.07719#bib.bib8)\]\.
- •ReCoLoRA: the full method—randomized SVD initialization, elbow\-based rank control, staged principal\-to\-residual training, and recursive consolidation, which re\-decomposes the effective weight at each task boundary into a frozen residual, an elbow\-compressed slow principal subspace, and a fresh fast adapter\.
- •Static ReCoLoRA: ReCoLoRA without recursive consolidation \(a single shared adapter whose updates always sit on the originalW0W\_\{0\}\); used as an ablation \(Appendix[C](https://arxiv.org/html/2607.07719#A3)\)\.
- •ReCoLoRA\-TaskBank: an oracle\-routed task\-incremental variant that trains an isolated ReCoLoRA branch for each task and freezes old branches\.

All methods use the same task order and random seeds\. For Qwen, Llama, and Mistral backbones, adapters target the query and value projection modules; for InternLM2\.5\-7B\-Chat, the adapters target the fused attention projection and output projection modules used by the local model implementation\. Each task is trained for 200 optimizer steps, with maximum sequence length 256 and at most 1000 evaluation samples per task\. We report mean and standard deviation over seeds 42, 43, and 44\. TaskBank uses oracle task identity at evaluation, so it is reported as a task\-incremental upper bound rather than as a task\-agnostic router\.

## 5Results

### 5\.1Main Continual Fine\-Tuning Results

Table[3](https://arxiv.org/html/2607.07719#S5.T3)reports the Qwen3\-8B comparison of single\-adapter PEFT methods, O\-LoRA \(a representative continual\-learning LoRA baseline\), and the oracle\-routed TaskBank upper bound over three seeds\. Among the single\-adapter methods, static ReCoLoRA obtains the highest final average score and reduces forgetting relative to LoRA, PiSSA, DoRA, and O\-LoRA, confirming that spectral initialization with staged training improves single\-adapter quality\. O\-LoRA, which trains one frozen LoRA branch per task with inter\-task orthogonality regularization, achieves intermediate performance: it outperforms LoRA on forgetting \(0\.0257 vs 0\.0414\) but falls short of ReCoLoRA on final average score \(0\.8722 vs 0\.8821\) using substantially more stored parameters \(∼\\sim23M vs 7\.67M for six tasks\)\. ReCoLoRA\-TaskBank then eliminates measured forgetting under oracle routing, isolating branch isolation as the component that structurally prevents overwriting\. The deployable ReCoLoRA with recursive consolidation is evaluated across backbones in Table[5](https://arxiv.org/html/2607.07719#S5.T5)\.

Table 3:Qwen3\-8B PEFT continual fine\-tuning results on the GLUE task sequence SST\-2→\\rightarrowMRPC→\\rightarrowQNLI→\\rightarrowRTE→\\rightarrowQQP→\\rightarrowMNLI\. Results are mean±\\pmstandard deviation over seeds 42, 43, and 44\. ReCoLoRA\-TaskBank uses oracle task routing and should be interpreted as a task\-incremental upper bound; it stores one branch per task\. O\-LoRA stores one low\-rank adapter per task at rank 8, so memory scales with task count; the value shown is approximate for six tasks\.∗Static ReCoLoRA’s trainable parameter count varies slightly by seed \(6,848,640–6,851,712\) because the elbow/energy rank selector \(ρ≥0\.8\\rho\\geq 0\.8of the truncated\-spectrum energy,rmax=16r\_\{\\max\}=16\) assigns a different principal rankr∈\[8,16\]r\\in\[8,16\]to each projection matrix; the value reported is the across\-seed average\.
### 5\.2Comparison with Continual Learning Baselines

Table[4](https://arxiv.org/html/2607.07719#S5.T4)extends the single\-adapter comparison to Mistral\-7B\-v0\.3 using the same continual\-GLUE protocol\. This enables a backbone\-level robustness check for static ReCoLoRA against O\-LoRA, a method designed explicitly for continual learning\. On Mistral, the two methods show different relative standings: O\-LoRA achieves higher final average score and lower forgetting \(0\.8255 / 0\.0675 vs 0\.8031 / 0\.0912\), while both outperform standard LoRA \(0\.8044 / 0\.0959\)\. This result illustrates why ReCoLoRA research shifted toward recursive consolidation: the static two\-stage schedule, while effective on Qwen3\-8B, does not generalize robustly across backbones\. The recursive variant \(evaluated in Table[5](https://arxiv.org/html/2607.07719#S5.T5)\) is designed to mitigate this sensitivity by consolidating and recomputing the task\-specific schedule after each task\.

Table 4:Mistral\-7B\-v0\.3 single\-adapter PEFT comparison on the six\-task continual GLUE sequence\. Results are mean±\\pmstandard deviation over seeds 42, 43, and 44\. ReCoLoRA uses the same configuration as the Qwen3\-8B main results \(elbow rank selection withρ=0\.8\\rho=0\.8,rmax=16r\_\{\\max\}=16, two\-stage training\)\. O\-LoRA trains one rank\-8 adapter per task with orthogonality regularization\. The higher forgetting for static ReCoLoRA on Mistral \(0\.0912\) compared to Qwen3\-8B \(0\.0240\) suggests that the static schedule is backbone\-sensitive; recursive consolidation \(Table[5](https://arxiv.org/html/2607.07719#S5.T5)\) is designed to address this\.†ReCoLoRA parameter count varies by seed \(∼\\sim10\.4–11\.2M\); value shown is approximate\.
### 5\.3Recursive Consolidation on Multiple Backbones

Table[5](https://arxiv.org/html/2607.07719#S5.T5)evaluates ReCoLoRA against rank\-swept baselines on four 7B/8B\-class backbones\. To address the possibility that ReCoLoRA benefits only from using a larger adaptive rank, each baseline is swept over ranks 64, 128, and 256, and the table reports the best final\-average configuration for each method\. ReCoLoRA is adaptive: each layer chooses an elbow\-compressed slow rank with minimum rank 8 and maximum rank 256, then adds a fresh fast update for the next task\.

The result is favorable but not universal\. Under the seed\-42 comparison reported in the table, ReCoLoRA obtains the best final average score on Qwen3\-8B, Mistral\-7B\-v0\.3, and InternLM2\.5\-7B\-Chat, with its largest apparent gain on Mistral \(0\.0654 final\-average points over the best baseline\) while using fewer trainable parameters than all rank\-64 LoRA\-family baselines\. On Qwen3\-8B and InternLM2\.5\-7B\-Chat the gains over the best LoRA\-family baseline are smaller but still positive, and Llama\-3\.1\-8B\-Instruct appears to be a negative case, with LoRA at rank 64 ahead on final average score\.

Because the baseline sweeps in Table[5](https://arxiv.org/html/2607.07719#S5.T5)use a single seed, we additionally reran ReCoLoRA \(only\) with seeds 43 and 44 on all four backbones to check how robust these single\-seed comparisons are\. The three\-seed means \(population standard deviation\) for final average score are: Qwen3\-8B0\.8863±0\.00040\.8863\\pm 0\.0004, Mistral\-7B\-v0\.30\.8324±0\.03030\.8324\\pm 0\.0303, InternLM2\.5\-7B\-Chat0\.8746±0\.00490\.8746\\pm 0\.0049, and Llama\-3\.1\-8B\-Instruct0\.8465±0\.01040\.8465\\pm 0\.0104\. Qwen3\-8B is essentially seed\-independent, and InternLM2\.5\-7B\-Chat’s gain over LoRA narrows from 0\.0055 to roughly parity but remains consistent with the seed\-42 ranking\. Mistral\-7B\-v0\.3 is the most seed\-sensitive backbone: the seed\-42 gain of 0\.0654 over the best baseline \(LoRA at rank 64, 0\.7979\) shrinks to about 0\.0345 on average across seeds, and one seed \(0\.7913\) falls slightly below that baseline on final average score; ReCoLoRA’s forgetting nevertheless remains the lowest in the table on average \(0\.0590±0\.03990\.0590\\pm 0\.0399, versus0\.10070\.1007–0\.11090\.1109for the Mistral baselines\), so we present the Mistral result as favorable but seed\-sensitive rather than a robust large gain\. Conversely, Llama\-3\.1\-8B\-Instruct is less of a negative case than the single seed suggests: LoRA at rank 64 remains nominally ahead on final average score \(0\.8522 versus a three\-seed mean of0\.8465±0\.01040\.8465\\pm 0\.0104\), but two of the three seeds \(0\.8522 and 0\.8554\) match or exceed LoRA, and ReCoLoRA’s three\-seed average forgetting \(0\.0453±0\.01250\.0453\\pm 0\.0125\) is lower than LoRA’s \(0\.0551\)\. We therefore interpret ReCoLoRA as a promising single\-evolving\-model mechanism whose advantage is robust on two of the four backbones \(Qwen3\-8B, InternLM2\.5\-7B\-Chat\), favorable but seed\-sensitive on a third \(Mistral\-7B\-v0\.3\), and close to a tie rather than a clear loss on the fourth \(Llama\-3\.1\-8B\-Instruct\), rather than a universally dominant adapter\.

Table 5:ReCoLoRA and rank\-swept PEFT baselines on the six\-task continual GLUE sequence\. For each baseline method and backbone, ranks 64, 128, and 256 are swept and the row reports the rank with the highest final average score\. Results in this table use seed 42 for all methods; a three\-seed robustness check for ReCoLoRA is discussed in the text below the table\. AdaLoRA at rank 256 exceeded our GPU memory budget for Qwen3\-8B and InternLM2\.5\-7B\-Chat, so the best completed rank is reported for those backbones instead\. ReCoLoRA uses elbow\-only consolidation with minimum rank 8 and maximum rank 256\.
### 5\.4Interpreting the Stability\-Plasticity Trade\-Off

The Qwen3\-8B table shows why continual PEFT should not be judged by a single metric\. Among single\-adapter methods, static ReCoLoRA gives the strongest final adaptation result while keeping forgetting low\. Compared with standard LoRA and PiSSA on Qwen3\-8B, static ReCoLoRA improves both final score and forgetting, which supports the claim that principal initialization should be paired with a staged residual strategy rather than used only as a static initialization trick\. However, the TaskBank result changes the main interpretation: explicit adapter isolation is much more effective than trying to preserve all tasks inside one shared adapter\. With oracle routing, ReCoLoRA\-TaskBank obtains0\.8957±0\.00260\.8957\\pm 0\.0026final average score and zero measured forgetting across all three seeds\.

The recursive results sharpen the interpretation\. Keeping all learning as a delta on the originalW0W\_\{0\}is not the only PEFT\-compatible design\. By recomputing the slow principal component from the previous task’s effective weight, ReCoLoRA gives the next task a consolidated starting point while still pruning low\-value rank directions\. The rank sweep is important: simply increasing LoRA\-family ranks to 128 or 256 does not consistently improve continual learning, and in several cases it worsens final average score or forgetting\. ReCoLoRA is strongest on Qwen3\-8B, Mistral\-7B\-v0\.3, and InternLM2\.5\-7B\-Chat, but Llama\-3\.1\-8B\-Instruct remains a boundary case where standard LoRA is better\. The strongest current evidence therefore points to spectrum\-aware adaptation plus an explicit memory mechanism, either branch isolation for task\-incremental upper\-bound evaluation or recursive consolidation for a single evolving model, while also showing that the recursive schedule is backbone\-sensitive\.

## 6Discussion

### 6\.1Relation to Replay, Prompt, and Orthogonal\-Subspace Methods

Replay\-based methods such as LAMOL reduce forgetting by regenerating or storing approximations of old data\[[12](https://arxiv.org/html/2607.07719#bib.bib12)\], and prompt\-based methods such as Progressive Prompts freeze the backbone and add per\-task prompt parameters\[[17](https://arxiv.org/html/2607.07719#bib.bib17)\]\. Closer to ReCoLoRA, O\-LoRA reserves orthogonal low\-rank subspaces for different tasks\[[9](https://arxiv.org/html/2607.07719#bib.bib9)\]and C\-LoRA routes continual parameter updates\[[18](https://arxiv.org/html/2607.07719#bib.bib18)\]\. ReCoLoRA differs in what it controls: the spectral organization of each pretrained weight\. Its recursive consolidation keeps a single evolving model and folds each finished task into a slow principal subspace, so old knowledge is carried forward rather than protected by an external buffer or router\. ReCoLoRA\-TaskBank sits at the isolation end of the same spectrum, giving each task its own spectrum\-initialized branch, and bounds how much retention is achievable\. The two views separate the roles our experiments find necessary: spectral initialization improves how well each task is learned, while an explicit memory structure–a consolidated slow weight or an isolated branch–is what prevents overwriting\.

### 6\.2Limitations

Our scope is intentionally bounded\. First, ReCoLoRA\-TaskBank relies on oracle task routing and stores one branch per task: it is a valid task\-incremental upper bound, but task\-agnostic deployment and long task streams will require a learned router and some form of branch compression or merging\. Second, the deployable recursive model is backbone\-sensitive\. Its advantage is robust on Qwen3\-8B and InternLM2\.5\-7B\-Chat, favorable but seed\-sensitive on Mistral\-7B\-v0\.3, and close to a tie on Llama\-3\.1\-8B\-Instruct, where rank\-64 LoRA remains nominally ahead; we therefore do not claim a universally dominant adapter, and the SVD rank thresholds, residual learning rates, and slow\-update schedule may need to be tuned per model family\. Third, the task sequence is GLUE\-style classification evaluated on a single task order; because AvgForget assigns zero forgetting to whichever task is evaluated last \(Eq\.[3](https://arxiv.org/html/2607.07719#S2.E3)\), longer instruction\-following and reasoning streams, together with more task permutations and order\-robust metrics, are needed before drawing order\-independent conclusions\.

## 7Conclusion

We presented ReCoLoRA, a spectrum\-aware PEFT framework for continual LLM fine\-tuning\. ReCoLoRA initializes adapters from a randomized SVD of the pretrained weight, selects an effective rank per layer with an elbow criterion, and adapts the principal subspace before the residual one; for continual use it consolidates each task recursively, re\-decomposing the current effective weight into a frozen residual, a slow principal component, and a fresh adapter so that the model evolves as a single deployable network instead of a growing stack of updates on the original weight\. We found that the simpler alternative, protecting old ranks inside one shared adapter through freezing or parameter\-space anchoring, only trades plasticity for stability and does not reliably stop forgetting\. On a six\-task continual GLUE sequence over four 7–8B backbones, ReCoLoRA attains the best final average score on Qwen3\-8B, Mistral\-7B\-v0\.3, and InternLM2\.5\-7B\-Chat with far fewer trainable parameters than the strongest rank\-256 baseline, with Llama\-3\.1\-8B\-Instruct a close negative case\. As an isolation upper bound under oracle routing, ReCoLoRA\-TaskBank reaches0\.8957±0\.00260\.8957\\pm 0\.0026final average score with0\.0000±0\.00000\.0000\\pm 0\.0000average forgetting on Qwen3\-8B, the ceiling that the single evolving model approaches\. The picture that emerges is that spectral initialization improves how well each task is learned, while an explicit memory mechanism–recursive consolidation in one model, or branch isolation as a ceiling–is what keeps earlier tasks from being erased\. Future work will add a learned router to remove the oracle assumption, branch compression for long streams, and evaluation on more task orders and longer instruction\-following sequences\.

*Note\.*Throughout the appendix, “ReCoLoRA” without qualification denotes the static single\-adapter variant \(*Static ReCoLoRA*in the main text\); the deployable recursive method is the main\-text ReCoLoRA\.

## Appendix ASingle\-Adapter Rank Freezing

The original staged schedule controls when residual capacity is introduced, but it does not by itself prevent later tasks from modifying principal ranks that were useful for earlier tasks\. To better match the continual\-learning objective, we add an optional task\-wise rank\-freezing rule\. At the beginning of tasktt, each target layer recomputes an elbow rank on its current effective weight and expands the active rank mask when additional capacity is needed\. Letmℓ,t∈\{0,1\}rm​a​xm\_\{\\ell,t\}\\in\\\{0,1\\\}^\{r\_\{max\}\}denote the active rank mask of layerℓ\\ellafter taskttis introduced, and letgℓ,tg\_\{\\ell,t\}denote the gradient mask\. Fort\>1t\>1, old active ranks remain in the forward pass but are excluded from gradient updates:

mℓ,t⊇mℓ,t−1,gℓ,t,k=\{1,mℓ,t,k=1​and​mℓ,t−1,k=0,0,otherwise\.m\_\{\\ell,t\}\\supseteq m\_\{\\ell,t\-1\},\\qquad g\_\{\\ell,t,k\}=\\begin\{cases\}1,&m\_\{\\ell,t,k\}=1\\ \\mathrm\{and\}\\ m\_\{\\ell,t\-1,k\}=0,\\\\ 0,&\\mathrm\{otherwise\}\.\\end\{cases\}\(15\)In implementation, this is applied by masking gradients of the corresponding rows ofAAand columns ofBB\. The forward computation still uses all active ranks, so previous task subspaces remain available at inference time\. New tasks are therefore encouraged to use newly opened dimensions instead of overwriting old ones\. This mechanism is related in spirit to orthogonal\-subspace continual learning, but it does not require a separate adapter per task; it grows a shared rank budget inside the same adapter\.

#### Forward\-preserving rank growth\.

Naively expanding the active rank mask can change the model function even before optimization if the frozen base weight already contains the SVD directions that are later reactivated through the adapter\. To avoid this drift, the forward\-preserving variant allocates a maximum SVD rank budget at initialization, subtracts the full allocated principal component from the frozen base, and keeps all allocated SVD directions present in the forward pass\. The active\-rank schedule then controls only which dimensions receive gradients\. In a local sanity check, naive growth from rank 4 to rank 8 changed layer outputs by8\.8×10−18\.8\\times 10^\{\-1\}, while forward\-preserving growth kept the output difference below2×10−72\\times 10^\{\-7\}\. This distinction is important for continual learning because rank expansion should add trainable capacity without silently perturbing previously learned behavior\.

## Appendix BSingle\-Adapter Validation on Llama Backbones

Table[6](https://arxiv.org/html/2607.07719#A2.T6)reports the Llama\-3\.1\-8B\-Instruct validation under the same task sequence, seeds, and training budget\. The result is intentionally reported as a robustness check rather than as a confirmation\-only experiment\. On this backbone, LoRA obtains the best final average score, with DoRA close behind\. The original ReCoLoRA configuration improves over PiSSA and AdaLoRA in final average score but has higher forgetting\. The forward\-preserving frozen\-rank variant reverses this behavior: it obtains the lowest forgetting in the table, but sacrifices final average score\. This cross\-backbone result narrows the claim: ReCoLoRA\-style subspace preservation can reduce forgetting, but the current implementation is not yet a universally strongest PEFT method across all 8B backbones\.

Table 6:Llama\-3\.1\-8B\-Instruct PEFT continual fine\-tuning results on the same GLUE task sequence\. Results are mean±\\pmstandard deviation over seeds 42, 43, and 44\. ReCoLoRA\-FP freeze denotes the forward\-preserving task\-wise rank\-freezing variant\. It lowers forgetting but also reduces final average performance, exposing the stability\-plasticity trade\-off\.∗ReCoLoRA’s trainable parameter count varies slightly by seed \(6,151,168–6,175,744\) because the elbow/energy rank selector \(ρ≥0\.8\\rho\\geq 0\.8of the truncated\-spectrum energy,rmax=16r\_\{\\max\}=16\) assigns a different principal rankr∈\[8,16\]r\\in\[8,16\]to each projection matrix; the value reported is the across\-seed average\.Table 7:Llama\-3\-8B ReCoLoRA rank\-allocation study on the continual GLUE sequence\. All variants use randomized SVD initialization and task\-wise elbow recomputation\. Task\-wise elbow expands the active rank mask across tasks\. Layer\-wise bonus adds projection\-type and depth\-dependent rank bonuses\. Frozen old ranks keeps previously active ranks in the forward pass but masks their gradients\. Forward\-preserving frozen ranks additionally keeps allocated inactive SVD directions present in the forward computation, so rank growth changes trainability rather than the initial function\. Results are mean±\\pmstandard deviation over seeds 42, 43, and 44\.The Llama\-3\.1\-8B\-Instruct validation adds an important caveat\. LoRA and DoRA remain stronger in final average score, but the forward\-preserving frozen\-rank variant has the lowest average forgetting\. Compared with the earlier ReCoLoRA implementation, it reduces forgetting from 0\.0719 to 0\.0378, while reducing final average score from 0\.8310 to 0\.8214\. This suggests that the spectral schedule is sensitive to backbone spectra, tokenizer\-label interactions, or optimization hyperparameters, and it exposes a clear stability\-plasticity trade\-off\.

Table[7](https://arxiv.org/html/2607.07719#A2.T7)clarifies the Llama\-side failure mode\. Simply recomputing elbow ranks per task gives moderate final performance but still leaves substantial forgetting\. Adding more rank capacity by hand through layer\-wise and depth\-wise bonuses is not sufficient; it actually increases forgetting in this run\. Freezing old ranks changes the behavior: the final average score improves from 0\.8095 to 0\.8178 relative to task\-wise elbow, and average forgetting drops from 0\.0631 to 0\.0283\. The forward\-preserving version obtains a similar result, with slightly higher final average score \(0\.8193\) and slightly higher forgetting \(0\.0340\)\. This supports the subspace\-preservation interpretation of ReCoLoRA: later tasks should not merely receive more low\-rank capacity; they should be prevented from overwriting rank dimensions already used by previous tasks, and rank growth should not introduce unintended function drift\.

## Appendix CAblations and Robustness Checks

The ablations isolate the design choices in ReCoLoRA\. The Qwen3\-8B ablations use randomized SVD initialization, elbow\-based rank control, and a static two\-stage principal\-to\-residual transition as the final baseline\.Dynamic recoveryadds the optional residual/rank recovery allocator during Stage 2\.One\-stageremoves the principal\-to\-residual split and activates recovery from the beginning\.Fixed\-rankbypasses the elbow rank selector and forces a uniform rank of 16\.Random\-initreplaces SVD initialization with random LoRA\-style initialization while keeping the dynamic recovery setting\. On Llama\-3\-8B, we further compare task\-wise elbow rank expansion, layer/depth bonus expansion, and task\-wise rank freezing\. These variants test whether continual robustness comes merely from adding rank capacity or from preventing old task subspaces from being overwritten\.

### C\.1Ablation Results

![Refer to caption](https://arxiv.org/html/2607.07719v1/x3.png)Figure 4:Distribution of the selected principal rankr∗r^\{\*\}across all 72q\_proj/v\_projmatrices in Qwen3\-8B under the elbow/energy criterion of Eq\.[8](https://arxiv.org/html/2607.07719#S3.E8)\(ρ=0\.8\\rho=0\.8,rmax=16r\_\{\\max\}=16, seed 42\)\. Every matrix selectsr∗∈\[8,16\]r^\{\*\}\\in\[8,16\]\(mean14\.5414\.54\), confirming ther∈\[8,16\]r\\in\[8,16\]range used throughout the paper from the actual model weights rather than an assumed range\.Table[8](https://arxiv.org/html/2607.07719#A3.T8)reports the completed ablations\. The final ReCoLoRA configuration is the static two\-stage variant, which uses SVD initialization, elbow/energy rank control \(ρ=0\.8\\rho=0\.8over the truncated spectral budget,r∈\[8,16\]r\\in\[8,16\]per projection matrix; Figure[4](https://arxiv.org/html/2607.07719#A3.F4)shows the empirical distribution ofr∗r^\{\*\}across all targeted Qwen3\-8B matrices\), and the principal\-to\-residual stage transition\. This variant achieves the best final average score and the best forgetting among all variants in the table, while also using fewer trainable parameters \(6,850,688 on average\) than the fixed\-rank variant \(7,668,864\)\.

The strongest negative result is the one\-stage variant\. Removing the principal\-to\-residual schedule reduces the final average score from 0\.8821 to 0\.6548 and increases forgetting from 0\.0240 to 0\.2331\. This strongly supports the central scheduling hypothesis: activating the residual/recovery mechanism too early makes training unstable in the continual setting\.

SVD initialization remains useful: the random\-init variant \(0\.8497/0\.0455\) is worse than ReCoLoRA \(ours\) \(0\.8821/0\.0240\) in both final average score and forgetting, even though both use the same elbow\-selected per\-layer ranks, which suggests that aligning the adapter with pretrained principal directions improves continual robustness independently of rank allocation\. The fixed\-rank comparison is informative for the first time: forcing a uniform rank of 16 at every layer \(7,668,864 trainable parameters\) yields both a lower final average score and higher forgetting \(0\.8658/0\.0407\) than the elbow\-selected per\-layer ranks of ReCoLoRA \(ours\) \(6,850,688 parameters on average,r∈\[8,16\]r\\in\[8,16\]\), despite using more parameters\. This supports the elbow/energy rank selector as a genuine accuracy and stability improvement rather than merely a parameter\-reduction heuristic\. The optional dynamic recovery allocator \(0\.8566/0\.0466\) remains weaker than static two\-stage ReCoLoRA, so we continue to treat it as an extension that requires further tuning rather than a core component\.

Table 8:Qwen3\-8B ReCoLoRA ablations on the continual GLUE sequence\. Results are mean±\\pmstandard deviation over seeds 42, 43, and 44\. Static two\-stage ReCoLoRA is the final method; the optional dynamic recovery allocator is weaker in this setting\.∗Trainable parameters vary slightly by seed \(6,848,640–6,851,712; average shown\) because the elbow/energy rank selector assigns a different principal rankr∈\[8,16\]r\\in\[8,16\]per projection matrix \(see Table[3](https://arxiv.org/html/2607.07719#S5.T3)\)\. Fixed rank forcesr=16r=16uniformly and is therefore unaffected\.
### C\.2Robustness Check: Energy\-Threshold Sensitivity

The main results \(Table[3](https://arxiv.org/html/2607.07719#S5.T3)\) use a single energy thresholdρ=0\.8\\rho=0\.8\. We report a Qwen3\-8B check, using the same training budget, hyperparameters, and seeds \(42, 43, 44\) as Table[3](https://arxiv.org/html/2607.07719#S5.T3), that probes the sensitivity of the main results to this choice\.

#### Energy\-threshold sensitivity\.

Figure[5](https://arxiv.org/html/2607.07719#A3.F5)sweeps the energy thresholdρ∈\{0\.6,0\.7,0\.8,0\.9,0\.95\}\\rho\\in\\\{0\.6,0\.7,0\.8,0\.9,0\.95\\\}in Eq\.[8](https://arxiv.org/html/2607.07719#S3.E8), holding the randomized SVD fixed \(same 72q\_proj/v\_projmatrices and seed as Figure[4](https://arxiv.org/html/2607.07719#A3.F4)\) and recomputingr∗r^\{\*\}for eachρ\\rho\. Asρ→0\.95\\rho\\to 0\.95, every matrix saturates atr∗=rmax=16r^\{\*\}=r\_\{\\max\}=16, so ReCoLoRA’s principal adapter becomes parameter\-equivalent to the Fixed\-rank ablation in Table[8](https://arxiv.org/html/2607.07719#A3.T8); asρ→0\.6\\rho\\to 0\.6, the mean selected rank drops to9\.769\.76\(range\[4,12\]\[4,12\]\), roughly65%65\\%of theρ=0\.8\\rho=0\.8adapter size\. Table[9](https://arxiv.org/html/2607.07719#A3.T9)reports downstream continual fine\-tuning atρ∈\{0\.6,0\.9\}\\rho\\in\\\{0\.6,0\.9\\\}on the main task order\. Final average score and forgetting both improve monotonically asρ\\rhoincreases over\{0\.6,0\.8,0\.9\}\\\{0\.6,0\.8,0\.9\\\}, andρ=0\.9\\rho=0\.9slightly outperforms the defaultρ=0\.8\\rho=0\.8on both metrics, though the two are within one standard deviation of each other\. Notably, even atρ=0\.9\\rho=0\.9\(meanr∗=15\.93r^\{\*\}=15\.93,7,627,9047\{,\}627\{,\}904trainable parameters, within1%1\\%of LoRA’s7,667,7127\{,\}667\{,\}712\), ReCoLoRA still outperforms LoRA by a wide margin on both axes \(0\.88490\.8849vs\.0\.86340\.8634final average,0\.01970\.0197vs\.0\.04140\.0414forgetting\)\. This suggests that ReCoLoRA’s gain over LoRA comes mainly from*per\-matrix adaptive rank and spectral initialization*rather than from aggressively shrinking the adapter, and thatρ=0\.8\\rho=0\.8is a reasonable but not provably optimal default; within this range,ρ=0\.9\\rho=0\.9is a plausible alternative when forgetting is the priority\.

![Refer to caption](https://arxiv.org/html/2607.07719v1/x4.png)Figure 5:Sensitivity of elbow/energy rank selection \(Eq\.[8](https://arxiv.org/html/2607.07719#S3.E8)\) to the energy thresholdρ\\rho, computed from the same 72 Qwen3\-8Bq\_proj/v\_projmatrices and SVD \(seed 42\) as Figure[4](https://arxiv.org/html/2607.07719#A3.F4)\. \(a\) Mean selected rankr∗r^\{\*\}\(line\) and per\-matrix min/max range \(shaded band\) versusρ\\rho; atρ=0\.95\\rho=0\.95every matrix saturates atrmax=16r\_\{\\max\}=16\. \(b\) Resulting principal\-adapter parameter count versusρ\\rho, with LoRA’sr=16r=16parameter count shown for reference\.Table 9:ReCoLoRA energy\-thresholdρ\\rhosensitivity on Qwen3\-8B, seeds 42, 43, 44 \(theρ=0\.8\\rho=0\.8row reproduces Table[3](https://arxiv.org/html/2607.07719#S5.T3)\)\. Meanr∗r^\{\*\}is averaged over the 72q\_proj/v\_projmatrices used to build Figure[5](https://arxiv.org/html/2607.07719#A3.F5)\.∗Trainable parameters vary slightly by seed \(6,848,640–6,851,712; average shown\)\.

## References

- \[1\]Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.Lora: Low\-rank adaptation of large language models\.InProc\. ICLR, 2022\.
- \[2\]Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu\.A comprehensive survey of continual learning: Theory, method and application\.arXiv preprint arXiv:2302\.00487, 2024\.
- \[3\]Zixuan Ke and Bing Liu\.Continual learning of natural language processing tasks: A survey\.arXiv preprint arXiv:2211\.12701, 2023\.
- \[4\]Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang\.An empirical study of catastrophic forgetting in large language models during continual fine\-tuning\.arXiv preprint arXiv:2308\.08747, 2023\.
- \[5\]Tongtong Wu, Linhao Luo, Yuan\-Fang Li, Shirui Pan, Thuy\-Trang Vu, and Gholamreza Haffari\.Continual learning for large language models: A survey\.arXiv preprint arXiv:2402\.01364, 2024\.
- \[6\]Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao\.Adaptive budget allocation for parameter\-efficient fine\-tuning \(adalora\)\.InProc\. ICLR, 2023\.
- \[7\]Fanxu Meng, Zhaohui Wang, and Muhan Zhang\.Pissa: Principal singular values and singular vectors adaptation of large language models\.arXiv preprint arXiv:2404\.02948, 2024\.
- \[8\]Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\.Dora: Weight\-decomposed low\-rank adaptation\.InProc\. ICML, 2024\.
- \[9\]Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang\.Orthogonal subspace learning for language model continual learning\.InFindings of EMNLP, 2023\.
- \[10\]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R\. Bowman\.Glue: A multi\-task benchmark and analysis platform for nlu\.InEMNLP Workshop BlackboxNLP, 2019\.
- \[11\]Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R\. Bowman\.Superglue: A stickier benchmark for general\-purpose language understanding systems\.InNeurIPS, 2019\.
- \[12\]Fan\-Keng Sun, Cheng\-Hao Ho, and Hung\-Yi Lee\.LAMOL: LAnguage MOdeling for lifelong language learning\.arXiv preprint arXiv:1909\.03329, 2019\.
- \[13\]Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan\.Fine\-tuned language models are continual learners\.InProc\. EMNLP, 2022\.
- \[14\]Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi\.Dylora: Parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProc\. EACL, 2023\.
- \[15\]Dawid J\. Kopiczko, Tijmen Blankevoort, and Yuki M\. Asano\.Vera: Vector\-based random matrix adaptation\.InProc\. ICLR, 2024\.
- \[16\]Soufiane Hayou, Nikhil Ghosh, and Bin Yu\.Lora\+: Efficient low rank adaptation of large models\.arXiv preprint arXiv:2402\.12354, 2024\.
- \[17\]Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi\.Progressive prompts: Continual learning for language models\.InProc\. ICLR, 2023\.
- \[18\]Xin Zhang, Liang Bai, Xian Yang, and Jiye Liang\.C\-LoRA: Continual low\-rank adaptation for pre\-trained models\.arXiv preprint arXiv:2502\.17920, 2025\.
- \[19\]Nathan Halko, Per\-Gunnar Martinsson, and Joel A\. Tropp\.Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions\.SIAM Review, 53\(2\):217–288, 2011\.