MuLoRA：用于持续学习的谱均衡低秩适配方法

# MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning
Source: [https://arxiv.org/html/2610.02283](https://arxiv.org/html/2610.02283)
###### Abstract

Low\-rank adaptation \(LoRA\) provides a parameter\-efficient approach to continual learning, but its nominal rank can conceal a loss of effective adaptation capacity\. We identify*spectral plasticity collapse*: during sequential adaptation, update energy becomes concentrated in a small subset of singular modes, leaving much of the available low\-rank space underutilized\. This exposes a limitation of interference avoidance alone: protecting historical representations does not ensure that the remaining adaptation capacity is responsive to new tasks or effectively utilized\. To address this problem, we proposeMuLoRA, which jointly controls capacity allocation and utilization\. First, historical whitening identifies input directions with strong current\-task response relative to accumulated historical response, yielding a task\-adaptive basis that remains fixed during training\. Second, approximate polar orthogonalization of momentum updates reduces spectral concentration within the selected space\. An orthonormal basis connects these mechanisms by transferring the factor\-update spectrum exactly to the induced weight update\. We establish a max–min characterization of exact subspace selection and derive cumulative spectral bounds under controlled cross\-step anisotropy\. Across five class\-incremental benchmarks and eight incremental settings,MuLoRAachieves the highest mean accuracy in 15 of 16 reported metrics\.

## 1Introduction

Continual learning \(CL\) requires models to acquire knowledge from sequential tasks while retaining performance on previous ones[Kirkpatrick et al\. \(2017\)](https://arxiv.org/html/2610.02283#bib.bib17);[Qiu et al\. \(2025b\)](https://arxiv.org/html/2610.02283#bib.bib27);[Wang et al\. \(2021\)](https://arxiv.org/html/2610.02283#bib.bib18)\. With the emergence of foundation models, this challenge increasingly concerns adapting pretrained representations under limited computation and memory budgets[Radford et al\. \(2021\)](https://arxiv.org/html/2610.02283#bib.bib21);[Cheng et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib30)\. Low\-Rank Adaptation \(LoRA\)[Hu et al\. \(2022\)](https://arxiv.org/html/2610.02283#bib.bib11)provides an efficient parameterization by expressing each task\-specific weight update asΔ​W=B​A\\Delta W=BAwhile freezing the backbone within each task\. However, a fixed adaptation rank does not ensure that all available directions are effectively used\. An update can retain its algebraic rank while concentrating most of its energy in only a few singular modes\. This distinction matters in continual adaptation, where a limited low\-rank space must support learning across successive tasks\.

We observe a phenomenon that exposes this limitation:*spectral plasticity collapse*\. During continual LoRA adaptation, the learned updateΔ​W\\Delta Wcan exhibit a highly concentrated spectrum, leaving many available modes weakly utilized \(Figure[1](https://arxiv.org/html/2610.02283#S1.F1)and Figure[6](https://arxiv.org/html/2610.02283#S4.F6)\)\. We examine this behavior through four complementary diagnostics: the singular\-value distribution, effective rank, the nonzero condition numberκ\+\\kappa\_\{\+\}, and stable rank\. The former reveals how update energy is distributed across modes, while the latter quantifies the effective number of modes carrying that energy\. Here, collapse refers to the concentration of update energy in a few singular directions, leading to lower effective and stable ranks without necessarily reducing the algebraic rank\. Thus, maintaining the nominal LoRA rank does not guarantee effective use of its adaptation capacity\.

Muon[Jordan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib35)offers a natural starting point for addressing this spectral concentration\. Through approximate polar orthogonalization of momentum, it reshapes the update spectrum toward balanced nonzero singular values[Bernstein \(2025\)](https://arxiv.org/html/2610.02283#bib.bib36)\. This motivates using spectral balancing to improve update utilization within a fixed\-rank adapter\. However, spectral balancing of LoRA factors does not automatically translate into spectral balancing of the weight update\. When both factors are trained, their changes induceδ​W=δ​B​A\+B​δ​A\+δ​B​δ​A\\delta W=\\delta BA\+B\\delta A\+\\delta B\\delta A\. Even if the factor steps are individually balanced, multiplication by the factors and interaction between terms can produce an uneven weight\-step spectrum\. The challenge is to make Muon’s spectral control carry to update of the model\.

We address this issue by fixing a row\-orthonormal basisAAand updating onlyBBwith Muon\. SinceA​A⊤=IrAA^\{\\top\}=I\_\{r\}, each weight stepδ​W=δ​B​A\\delta W=\\delta BAhas the same nonzero singular values asδ​B\\delta B\. Muon’s spectral balancing therefore transfers directly to the weight step\. Likewise,B​ABApreserves the nonzero singular values ofBB, so the basis introduces no spectral distortion\.

FixingAAmakes the choice of its subspace important, because the model can only update within that space\. An orthonormal basis preserves the update spectrum, but its directions may not be useful for the new task\. Methods such as InfLoRA[Liang and Li \(2024\)](https://arxiv.org/html/2610.02283#bib.bib7)avoid a retained historical feature subspace to reduce interference in Figure[2](https://arxiv.org/html/2610.02283#S1.F2)\. However, shared directions can also support new learning, and directions with little historical response may still be unhelpful for the current task\. We therefore select directions with strong current\-task response relative to their historical response, providing a task\-aware space for spectral balancing\.

These observations suggest two connected requirements\. The adaptation space should contain directions with strong current\-task response relative to historical exposure, and its parameterization should preserve spectral control from factor space to weight space\. Together, they lead to following question:

How can Muon’s spectral balancing be transferred to LoRA weight updates within a subspace suited to continual adaptation?

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/spectrum_task01_comparison.png)\(a\)Task 1
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/spectrum_task20_comparison.png)\(b\)Task 20
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/imbalance_balance.png)\(c\)Spectral Capacity Utilization

Figure 1:Spectral concentration limits effective low\-rank capacity\.\(a,b\) Singular\-value distributions \(Frequency vs\.σ\\sigma\) of learned LoRA updatesΔ​W\\Delta Wat Tasks 1 and 20, with and without spectral balancing\. More balanced spectra distribute update energy across a larger number of modes\. \(c\) Conceptual illustration of concentrated versus balanced spectra and their relationship to effective rank\. A fixed nominal rank can correspond to markedly different levels of spectral utilization\.We proposeMuonLow\-RankAdaptation \(MuLoRA\), which connects Relative\-Response Spectral Allocation \(RRS\) with Subspace Spectral Balancing \(SSB\)\. For each task,MuLoRAconstructs a row\-orthonormal input basisAtA\_\{t\}, keeps it fixed, and applies Muon\-style updates to the trainable up\-projectionBtB\_\{t\}\. The central design principle is to select task\-responsive directions using historical information and preserve the update spectrum through the LoRA parameterization\.

For subspace selection, we maximize current\-task feature response under a regularized historical\-response budget\. Historical whitening converts this objective into a standard spectral selection problem\. The selected directions are mapped back to the input space and orthonormalized to constructAtA\_\{t\}\. Historical information thus provides a graded selection metric: previously active directions remain eligible when their relative current\-task response is strong\. With exact spectral selection, the resulting subspace maximizes the worst\-direction relative response among all subspaces of the same dimension\. This provides a subspace\-wide characterization of the directions available for adaptation\.

For spectral balancing, we apply approximate polar orthogonalization to momentum updates ofBtB\_\{t\}\. The ideal transformation equalizes all nonzero singular values, maximizing spectral participation on the active update support\. SinceAt​At⊤=IrA\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\}, the resulting factor step and its induced weight step have identical nonzero singular values and effective rank\. This connects the two components: RRS determines where adaptation occurs and supplies an orthonormal basis, while SSB controls how energy is distributed in each weight step\.

Our analysis distinguishes this stepwise control from the behavior of accumulated updates\. Balanced steps need not produce a balanced sum, because successive steps can reinforce or cancel different directions\. We derive cumulative condition\-number and spectral participation bounds under controlled cross\-step anisotropy, identifying conditions under which stepwise balancing extends to accumulated adaptation\. Empirically, we examine singular\-value distributions and participation\-based effective rank alongside continual\-learning accuracy\. Across five benchmarks and eight incremental settings,MuLoRAachieves the highest mean accuracy in 15 of 16 metrics\. Ablations support jointly selecting task\-responsive subspaces and balancing updates within them\.

Our main contributionscan be summarized as follows: ∙\\bulletWe identify*spectral plasticity collapse*in continual LoRA adaptation: update energy concentrates in a few singular modes, leaving the effective rank well below the nominal adapter rank\. We further highlight a challenge in applying Muon: balancing the two factor steps separately does not guarantee a spectral balanced weight step\. ∙\\bulletWe proposeMuLoRA, a history\-aware low\-rank parameterization that transfers Muon’s spectral balancing directly to weight space\. It fixes an orthonormal input basis selected by current\-task response relative to historical exposure and applies Muon updates to the trainable factor\. This design preserves the nonzero singular values of each factor step in the induced weight step, enabling spectral control within a task\-responsive adaptation space\. ∙\\bulletWe show that exact subspace selection maximizes the worst\-direction relative response and derive cumulative spectral bounds under controlled cross\-step anisotropy\. Across five benchmarks and eight incremental settings,MuLoRAachieves the highest mean accuracy in 15 of 16 reported metrics, with ablations supporting the complementary benefits of RRS and SSB\.

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/fig_compare4.png)Figure 2:Comparison of different LoRA\-based CL paradigms\.\(a\) Seq\-LoRA employs independent adapters, \(b\) InfLoRA preserve previous knowledge by restricting updates to isolated directions, \(c\)MuLoRAfixes a task\-responsive orthonormal basis and applies Muon to the trainable factor, transferring spectral balancing directly to weight steps\.
## 2Related Works

PEFT\-based Continual Learning\.Parameter\-efficient fine\-tuning \(PEFT\) enables continual adaptation of pretrained models through lightweight trainable modules\. Prompt\-based methods[Smith et al\. \(2023\)](https://arxiv.org/html/2610.02283#bib.bib3);[Gao et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib6)learn and select prompts to accommodate sequential tasks, while PGP[Qiao et al\. \(2023\)](https://arxiv.org/html/2610.02283#bib.bib9)and VPT\-NSP[Lu et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib8)derive projection constraints for mitigating interference\. Adapter\-based methods[Li et al\. \(2025\)](https://arxiv.org/html/2610.02283#bib.bib22);[Tan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib5);[He et al\. \(2025\)](https://arxiv.org/html/2610.02283#bib.bib32);[Wang et al\. \(2026\)](https://arxiv.org/html/2610.02283#bib.bib31)preserve and integrate task knowledge through lightweight modules, representation alignment, or feature compensation\. LoRA\-based approaches further structure adaptation within low\-rank parameter spaces\. SD\-LoRA[Wu et al\. \(2025b\)](https://arxiv.org/html/2610.02283#bib.bib28)decouples direction and magnitude learning; O\-LoRA[Wang et al\. \(2023\)](https://arxiv.org/html/2610.02283#bib.bib29)and InfLoRA[Liang and Li \(2024\)](https://arxiv.org/html/2610.02283#bib.bib7)restrict overlap with historical subspaces to reduce interference\. SplitLoRA[Qiu et al\. \(2025a\)](https://arxiv.org/html/2610.02283#bib.bib33), BiLoRA[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.02283#bib.bib24), and PLAN[Wang et al\. \(2025\)](https://arxiv.org/html/2610.02283#bib.bib25)further explore subspace decomposition, orthogonality, and allocation to balance stability and plasticity\. Complementing these approaches, EWC\-LoRA[Zheng et al\. \(2026\)](https://arxiv.org/html/2610.02283#bib.bib38)regularizes a shared low\-rank update using parameter importance estimated in the full weight space\. Closely related to our work, EBLoRA[Gu et al\. \(2026\)](https://arxiv.org/html/2610.02283#bib.bib37)links spectral imbalance to forgetting and enforces equal nonzero singular values through orthonormal factors and a shared scalar magnitude\.MuLoRAaddresses the stability–plasticity trade\-off by selecting subspaces with strong current\-task response relative to historical exposure and balancing momentum steps within them\.

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/main_fig5.png)Figure 3:Relative\-response subspace selection and spectrally controlled optimization\.\(A\) Historical whitening identifies directions with strong current\-task response relative to accumulated historical response, from which we construct a row\-orthonormal basisAtA\_\{t\}\. \(B1\) We freeze the backbone andAtA\_\{t\}, and train onlyBtB\_\{t\}\. \(B2\) Approximate polar orthogonalization reshapes the momentum spectrum, balancing individual optimization steps within the selected subspace\. \(C\) After each task, the learned update is merged into the backbone asWt=Wt−1\+Bt​AtW\_\{t\}=W\_\{t\-1\}\+B\_\{t\}A\_\{t\}\.
## 3Methodology

Problem Definition\.In CL, there areTTsequential tasks with non\-overlapping classes\. The datasets are defined as\{𝒟1,⋯,𝒟t,⋯,𝒟T\\\{\\mathcal\{D\}^\{1\},\\cdots,\\mathcal\{D\}^\{t\},\\cdots,\\mathcal\{D\}^\{T\}\}\.𝒟t=\{xit,yit\}i=1Nt\\mathcal\{D\}^\{t\}=\\\{x^\{t\}\_\{i\},y^\{t\}\_\{i\}\\\}\_\{i=1\}^\{N^\{t\}\}denotes thett\-th task’s dataset, wherexit∈ℝH×W×Cx^\{t\}\_\{i\}\\in\\mathbb\{R\}^\{H\\times W\\times C\}is theii\-th input andyit∈𝒞ty^\{t\}\_\{i\}\\in\\mathcal\{C\}^\{t\}is its class label\.𝒞t\\mathcal\{C\}^\{t\}denotes the label space of thett\-th task, and𝒞t∩𝒞t′=∅\\mathcal\{C\}^\{t\}\\cap\\mathcal\{C\}^\{t^\{\\prime\}\}=\\emptysetfort≠t′t\\neq t^\{\\prime\}\. The objective of CL is to train a modelf⁡\(w,⋅\)f\(w,\\cdot\)sequentially onTTtasks and performs well on all seen classes𝒞1∪⋯∪𝒞T\\mathcal\{C\}^\{1\}\\cup\\cdots\\cup\\mathcal\{C\}^\{T\}\. We consider the task\-agnostic class\-incremental learning setting, where task identity is unavailable during inference\.

Overall framework\.Figure[3](https://arxiv.org/html/2610.02283#S2.F3)illustrates the proposed framework\. Following existing PEFT\-based methods[Wang et al\. \(2022b\)](https://arxiv.org/html/2610.02283#bib.bib1);[Tan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib5), we adopt a pretrained Vision Transformer \(ViT\)[Dosovitskiy et al\. \(2021\)](https://arxiv.org/html/2610.02283#bib.bib10)as the backbone\. For tasktt, each adapted weight matrixWt−1∈ℝn×mW\_\{t\-1\}\\in\\mathbb\{R\}^\{n\\times m\}remains frozen and is augmented with a rank\-rrupdateBt​AtB\_\{t\}A\_\{t\}, whereAt∈ℝr×mA\_\{t\}\\in\\mathbb\{R\}^\{r\\times m\}andBt∈ℝn×rB\_\{t\}\\in\\mathbb\{R\}^\{n\\times r\}\. Historical whitening identifies a task\-adaptive subspace with strong current\-task response relative to accumulated historical response\. We construct a row\-orthonormal basisAtA\_\{t\}of this subspace, satisfyingAt​At⊤=IrA\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\}, and keep it fixed throughout the task\. Starting fromBt=0B\_\{t\}=0, we optimizeBtB\_\{t\}using Muon\-style approximate polar orthogonalization of momentum updates\. The orthonormal basis preserves the nonzero singular values between each factor update and its induced weight update, connecting subspace selection with stepwise spectral control\. For an input featurex∈ℝmx\\in\\mathbb\{R\}^\{m\}, the adapted layer computesy=\(Wt−1\+Bt​At\)​x∈ℝny=\(W\_\{t\-1\}\+B\_\{t\}A\_\{t\}\)x\\in\\mathbb\{R\}^\{n\}\. After training, we merge the learned update into the backbone asWt=Wt−1\+Bt​AtW\_\{t\}=W\_\{t\-1\}\+B\_\{t\}A\_\{t\}and construct a new basis for the next task\.

### 3\.1Relative\-Response Spectral Allocation

Motivation: shared feature directions also can support new learning\.Isolation\-based methods such as InfLoRA restrict adaptation to directions orthogonal to a retained historical feature subspace to mitigate interference\. However, historical and current tasks may rely on shared directions: excluding them can discard useful adaptation capacity, while low historical response alone does not ensure relevance to the incoming task\. We therefore treat historical response as a graded metric for subspace selection\. A previously active direction remains eligible when its current\-task response is sufficiently strong relative to its historical exposure\. Our guiding principle is:*select directions by their current\-to\-history response, then balance updates within the selected subspace\.*

Relative\-Response Spectral Allocation \(RRS\)\.LetXt∈ℝ\(nt​nb\)×mX\_\{t\}\\in\\mathbb\{R\}^\{\(n\_\{t\}n\_\{b\}\)\\times m\}collect the input features of tasktt, wherentn\_\{t\}is the number of samples andnbn\_\{b\}is the number of patch tokens per sample\. LetMt=\(Xt\)⊤​XtM\_\{t\}=\(X\_\{t\}\)^\{\\top\}X\_\{t\}denote the current\-task second\-moment statistics andM1:t−1=∑i=1t−1MiM\_\{1:t\-1\}=\\sum\_\{i=1\}^\{t\-1\}M\_\{i\}the accumulated historical statistics\. We define the regularized historical metric asHt−1=M1:t−1\+ϵIH\_\{t\-1\}=M\_\{1:t\-1\}\+\\epsilon I, whereϵ\>0\\epsilon\>0\. For a directionv≠0v\\neq 0, we measure its relative response byℛt​\(v\)=v⊤​Mt​vv⊤​Ht−1​v=‖Xt​v‖22∑i<t‖Xi​v‖22\+ϵ​‖v‖22\.\\mathcal\{R\}\_\{t\}\(v\)=\\frac\{v^\{\\top\}M\_\{t\}v\}\{v^\{\\top\}H\_\{t\-1\}v\}=\\frac\{\\\|X\_\{t\}v\\\|\_\{2\}^\{2\}\}\{\\sum\_\{i<t\}\\\|X\_\{i\}v\\\|\_\{2\}^\{2\}\+\\epsilon\\\|v\\\|\_\{2\}^\{2\}\}\.As illustrated in Figure[4](https://arxiv.org/html/2610.02283#S3.F4), this criterion favors directions with strong current\-task response relative to their historical response\. Directions shared with previous tasks remain available when they are sufficiently responsive to the current task\. To select a rank\-rrsubspace, we solve

maxZ∈ℝm×r⁡tr⁡\(Z⊤​Mt​Z\)s\.t\.Z⊤​Ht−1​Z=Ir\.\\max\_\{Z\\in\\mathbb\{R\}^\{m\\times r\}\}\\operatorname\{tr\}\(Z^\{\\top\}M\_\{t\}Z\)\\quad\\text\{s\.t\.\}\\quad Z^\{\\top\}H\_\{t\-1\}Z=I\_\{r\}\.\(1\)The constraint gives each basis direction the same regularized historical\-response budget and makes the directions orthogonal under the historical metric\. Within this budget, the objective selects a subspace with the largest total current\-task response\. This provides a feature\-based criterion for allocating adaptation capacity, without directly estimating task loss or forgetting\.

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/whiten.png)Figure 4:Allocating LoRA capacity using current\-task response relative to historical feature statistics\.Solving the subspace selection problem with whitening\.We solve Eq\. \([1](https://arxiv.org/html/2610.02283#S3.E1)\) by transforming the historical\-metric constraint into a standard orthogonality constraint\. LetHt−1=Lt−1​Lt−1⊤H\_\{t\-1\}=L\_\{t\-1\}L\_\{t\-1\}^\{\\top\}be the Cholesky factorization ofHt−1≻0H\_\{t\-1\}\\succ 0\. With the change of variablesU=Lt−1⊤​ZU=L\_\{t\-1\}^\{\\top\}Z, or equivalentlyZ=Lt−1−⁣⊤​UZ=L\_\{t\-1\}^\{\-\\top\}U, Eq\. \([1](https://arxiv.org/html/2610.02283#S3.E1)\) becomes

maxU⊤​U=Ir⁡tr⁡\(U⊤​M~t​U\),M~t=Lt−1−1​Mt​Lt−1−⁣⊤\.\\max\_\{U^\{\\top\}U=I\_\{r\}\}\\operatorname\{tr\}\\left\(U^\{\\top\}\\widetilde\{M\}\_\{t\}U\\right\),\\widetilde\{M\}\_\{t\}=L\_\{t\-1\}^\{\-1\}M^\{t\}L\_\{t\-1\}^\{\-\\top\}\.Hence, the optimal whitened subspace is spanned by the leadingrreigenvectors ofM~t\\widetilde\{M\}\_\{t\}, which correspond to directions with the strongest current\-task response relative to historical response\.

In practice, we approximate this dominant eigenspace using randomized SVD[Halko et al\. \(2011\)](https://arxiv.org/html/2610.02283#bib.bib34):U~t,Σt,V~t=RSVD​\(M~t,r\),\\widetilde\{U\}\_\{t\},\\Sigma\_\{t\},\\widetilde\{V\}\_\{t\}=\\texttt\{RSVD\}\\bigl\(\\widetilde\{M\}\_\{t\},r\\bigr\),whereU~t∈ℝm×r\\widetilde\{U\}\_\{t\}\\in\\mathbb\{R\}^\{m\\times r\}contains orthonormal approximate leading spectral directions in the whitened coordinates\. We then map them back to the original feature space asZt=Lt−1−⁣⊤​U~t\.Z\_\{t\}=L\_\{t\-1\}^\{\-\\top\}\\widetilde\{U\}\_\{t\}\.Because the mapped basisZtZ\_\{t\}is generally not Euclidean\-orthonormal, we perform a thin QR factorizationZt=Qt​Rt,Qt⊤​Qt=Ir,Z\_\{t\}=Q\_\{t\}R\_\{t\},Q\_\{t\}^\{\\top\}Q\_\{t\}=I\_\{r\},and define the fixed LoRA down\-projection asAt=Qt⊤\.A\_\{t\}=Q\_\{t\}^\{\\top\}\.SinceRtR\_\{t\}is nonsingular wheneverZtZ\_\{t\}has full column rank,range⁡\(Qt\)=range⁡\(Zt\),\\operatorname\{range\}\(Q\_\{t\}\)=\\operatorname\{range\}\(Z\_\{t\}\),so the QR step changes only the basis representation and preserves the selected relative\-response subspace exactly\. Meanwhile,At​At⊤=IrA\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\}, which ensures that every factor updateδ​Bt\\delta B\_\{t\}and its induced weight updateδ​Bt​At\\delta B\_\{t\}A\_\{t\}have identical nonzero singular values\. Thus, RRS preserves the generalized\-eigen subspace selected by the relative\-response criterion while providing the Euclidean\-orthonormal parameterization required for exact spectral transfer in the subsequent SSB optimization\. For the first task, no historical statistics are available andH0=ϵ​IH\_\{0\}=\\epsilon I\. Therefore,M~1=ϵ−1​M1\\widetilde\{M\}\_\{1\}=\\epsilon^\{\-1\}M\_\{1\}, and RRS reduces to selecting the dominant current\-task feature subspace\.

### 3\.2Subspace Spectral Balancing

Motivation: spectral collapse limits capacity utilization\.Even within a task\-responsive subspace, adaptation updates can concentrate most of their energy in a few singular modes, leaving the remaining directions weakly utilized\. We refer to this concentration as*spectral collapse*: the effective rank of the update can be much lower than the nominal LoRA rank, even without a loss of algebraic rank\. Subspace selection alone therefore does not ensure that the available capacity is effectively used\. To address this issue, we apply Muon\-style spectral balancing to the optimization step, reducing the dominance of a few modes within the selected subspace\.

Connecting factor steps to weight steps\.For tasktt, we parameterize the adapted weight asWt,k=Wt−1\+Bt,k​At,At​At⊤=Ir,W\_\{t,k\}=W\_\{t\-1\}\+B\_\{t,k\}A\_\{t\},A\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\},wherekkindexes optimization steps\. We freezeAtA\_\{t\}and trainBt,kB\_\{t,k\}fromBt,0=0B\_\{t,0\}=0\. Unlike updating both factors, this yields a weight stepSk=δ​Bk​AtS\_\{k\}=\\delta B\_\{k\}A\_\{t\}, whereδ​Bk=Bt,k\+1−Bt,k\\delta B\_\{k\}=B\_\{t,k\+1\}\-B\_\{t,k\}, satisfyingSk​Sk⊤=δ​Bk​At​At⊤​δ​Bk⊤=δ​Bk​δ​Bk⊤\.S\_\{k\}S\_\{k\}^\{\\top\}=\\delta B\_\{k\}A\_\{t\}A\_\{t\}^\{\\top\}\\delta B\_\{k\}^\{\\top\}=\\delta B\_\{k\}\\delta B\_\{k\}^\{\\top\}\.Thus, the factor step and the weight step have identical nonzero singular values\. Controlling the spectrum ofδ​Bk\\delta B\_\{k\}therefore controls the spectrum of the actual weight step, without distortion fromAtA\_\{t\}\.

Subspace Spectral Balancing \(SSB\)\.LetGk∈ℝn×rG\_\{k\}\\in\\mathbb\{R\}^\{n\\times r\}denote the momentum matrix used to updateBt,kB\_\{t,k\}\. ForGk≠0G\_\{k\}\\neq 0, write its compact SVD asGk=Uk​Σk​Vk⊤,Σk=diag⁡\(σ1,…,σq\),G\_\{k\}=U\_\{k\}\\Sigma\_\{k\}V\_\{k\}^\{\\top\},\\Sigma\_\{k\}=\\operatorname\{diag\}\(\\sigma\_\{1\},\\ldots,\\sigma\_\{q\}\),whereq=rank⁡\(Gk\)q=\\operatorname\{rank\}\(G\_\{k\}\)andσj\>0\\sigma\_\{j\}\>0\. The ideal polar transformation preserves the active singular directions and sets their singular values to one:

Dk=𝒪⁡\(Gk\)=Uk​Vk⊤,Bt,k\+1=Bt,k−ηk​𝒪​\(Gk\)\.D\_\{k\}=\\mathcal\{O\}\(G\_\{k\}\)=U\_\{k\}V\_\{k\}^\{\\top\},\\\>B\_\{t,k\+1\}=B\_\{t,k\}\-\\eta\_\{k\}\\mathcal\{O\}\(G\_\{k\}\)\.\(2\)This distributes the step energy equally across all active modes in Figure[5](https://arxiv.org/html/2610.02283#S3.F5)\. SSB thus maximizes spectral participation on the existing update support; it does not create additional modes whenGkG\_\{k\}is rank deficient\. Together, RRS and SSB determine which input directions are available for adaptation and how evenly each step uses its active singular modes\.

Efficient approximate implementation\.Computing an SVD at every step is unnecessary\. Following Muon[Jordan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib35);[Bernstein \(2025\)](https://arxiv.org/html/2610.02283#bib.bib36), we approximate the polar transformation using Newton–Schulz polynomial iterations\. Starting fromY0=Gk/\(‖Gk‖F\+δ\)Y\_\{0\}=G\_\{k\}/\(\\\|G\_\{k\}\\\|\_\{F\}\+\\delta\), withδ\>0\\delta\>0, we compute

Kj=Yj⊤​Yj,Yj\+1=a​Yj\+b​Yj​Kj\+c​Yj​Kj2,K\_\{j\}=Y\_\{j\}^\{\\top\}Y\_\{j\},\\\>Y\_\{j\+1\}=aY\_\{j\}\+bY\_\{j\}K\_\{j\}\+cY\_\{j\}K\_\{j\}^\{2\},\(3\)forj=0,…,J−1j=0,\\ldots,J\-1, and updateBt,k\+1=Bt,k−ηk​YJB\_\{t,k\+1\}=B\_\{t,k\}\-\\eta\_\{k\}Y\_\{J\}\. Each iteration reshapes the singular values through a low\-degree polynomial\. SinceKj∈ℝr×rK\_\{j\}\\in\\mathbb\{R\}^\{r\\times r\}, the computation exploits the small LoRA dimension\. Finite iterations provide approximate spectral balancing, while spectral transfer from the factor step to the induced weight step remains exact\. The same spectrum\-preserving identity holds betweenBt,kB\_\{t,k\}and the accumulated adapterBt,k​AtB\_\{t,k\}A\_\{t\}\. However, balanced individual steps need not yield a balanced sum, because successive steps can reinforce or cancel different directions\. Our theoretical analysis characterizes cumulative spectral behavior under controlled cross\-step anisotropy\.

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/balance.png)Figure 5:Spectral control of LoRA updates\. An orthonormal input basis transfers the singular spectrum of each factor update to the induced weight update without distortion\.

## 4Theoretical Analysis

#### Subspace\-wide relative\-response optimality\.

RRS first identifies a dominant subspace in the historically whitened coordinates and then maps it back to the original feature space\. The following result shows that this mapped subspace is max–min optimal for the original relative\-response criterion, while Euclidean re\-orthogonalization preserves the guarantee\.

###### Proposition 4\.1\(Subspace\-Wide Relative\-Response Optimality\)\.

LetHt−1=Lt−1​Lt−1⊤≻0H\_\{t\-1\}=L\_\{t\-1\}L\_\{t\-1\}^\{\\top\}\\succ 0andMt⪰0M\_\{t\}\\succeq 0, and defineM~t=Lt−1−1​Mt​Lt−1−⁣⊤\.\\widetilde\{M\}\_\{t\}=L\_\{t\-1\}^\{\-1\}M^\{t\}L\_\{t\-1\}^\{\-\\top\}\.Letλ1≥⋯≥λm≥0\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{m\}\\geq 0be the eigenvalues ofM~t\\widetilde\{M\}\_\{t\}, and letUr∈ℝm×rU\_\{r\}\\in\\mathbb\{R\}^\{m\\times r\}contain its leadingrrorthonormal eigenvectors\. Define the corresponding subspace in the original feature space as𝒮t=range⁡\(Lt−1−⁣⊤​Ur\)\.\\mathcal\{S\}\_\{t\}=\\operatorname\{range\}\\\!\\left\(L\_\{t\-1\}^\{\-\\top\}U\_\{r\}\\right\)\.Then every nonzerov∈𝒮tv\\in\\mathcal\{S\}\_\{t\}satisfiesλr≤v⊤​Mt​vv⊤​Ht−1​v≤λ1\.\\lambda\_\{r\}\\leq\\frac\{v^\{\\top\}M^\{t\}v\}\{v^\{\\top\}H\_\{t\-1\}v\}\\leq\\lambda\_\{1\}\.Moreover,maxdim\(𝒮\)=r⁡minv∈𝒮∖\{0\}​v⊤​Mt​vv⊤​Ht−1​v=λr,\\max\_\{\\dim\(\\mathcal\{S\}\)=r\}\\;\\min\_\{v\\in\\mathcal\{S\}\\setminus\\\{0\\\}\}\\frac\{v^\{\\top\}M^\{t\}v\}\{v^\{\\top\}H\_\{t\-1\}v\}=\\lambda\_\{r\},and the maximum is attained by𝒮t\\mathcal\{S\}\_\{t\}\. IfLt−1−⁣⊤​Ur=Qt​RtL\_\{t\-1\}^\{\-\\top\}U\_\{r\}=Q\_\{t\}R\_\{t\}is a thin QR factorization andAt=Qt⊤A\_\{t\}=Q\_\{t\}^\{\\top\}, thenAt​At⊤=Ir,row⁡\(At\)=𝒮t\.A\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\},\\operatorname\{row\}\(A\_\{t\}\)=\\mathcal\{S\}\_\{t\}\.Hence, Euclidean re\-orthogonalization preserves the max–min\-optimal RRS subspace while providing the row\-orthonormal basis required for exact spectral transfer\.

Proposition[4\.1](https://arxiv.org/html/2610.02283#S4.Thmtheorem1)provides a uniform guarantee over the entire LoRA adaptation space: even its least responsive direction achieves relative response at leastλr\\lambda\_\{r\}, and no otherrr\-dimensional subspace can attain a larger worst\-direction response\. The result follows from the change of variablesu=Lt−1⊤​vu=L\_\{t\-1\}^\{\\top\}v, under whichv⊤​Mt​vv⊤​Ht−1​v=u⊤​M~t​uu⊤​u,\\frac\{v^\{\\top\}M\_\{t\}v\}\{v^\{\\top\}H\_\{t\-1\}v\}=\\frac\{u^\{\\top\}\\widetilde\{M\}\_\{t\}u\}\{u^\{\\top\}u\},together with the Courant–Fischer theorem\. Importantly, the subsequent QR factorization changes only the basis of the selected subspace, not the subspace itself\. Thus, RRS simultaneously preserves its relative\-response optimality and yieldsAt​At⊤=IrA\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\}, directly connecting subspace allocation to the spectrum\-preserving LoRA parameterization used by SSB\.

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/BA_effective_rank_avg2.png)\(a\)Effective rank
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/BA_kappa_plus_avg2.png)\(b\)κ\+\\kappa\_\{\+\}
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/BA_stable_rank_avg2.png)\(c\)Stable rank
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/cnn_top1_curve_avg1.png)\(d\)Overall Test Acc

Figure 6:Spectral balancing prevents rank collapse\.We track effective rank,κ\+\\kappa\_\{\+\}, stable rank ofΔ​W\\Delta W, and overall test accuracy across tasks, comparingMuLoRAwith and without spectral balancing\. SSB maintains high effective rank and stable rank while keeping the nonzero spectrum well conditioned throughout sequential adaptation\. Importantly, this geometric advantage persists over the same tasks for which overall class\-incremental accuracy is better preserved, suggesting that spectral utilization is closely associated with retained adaptation capacity\.
### 4\.1Cumulative spectral balance

Stepwise orthogonalization balances individual updates, but their accumulation also depends on cross\-step interactions\. We characterize how cumulative anisotropy controls the condition number, stable rank, and entropy\-based effective rank of the accumulated update\. For a nonzero matrixBBwith positive singular valuesσ1​\(B\),…,σq​\(B\)\\sigma\_\{1\}\(B\),\\ldots,\\sigma\_\{q\}\(B\), definesr\(B\)=‖B‖F2‖B‖22,erank\(B\)=exp\(−∑j=1qpjlogpj\),pj=σj​\(B\)∑i=1qσi​\(B\)\.\\operatorname\{sr\}\(B\)=\\frac\{\\\|B\\\|\_\{F\}^\{2\}\}\{\\\|B\\\|\_\{2\}^\{2\}\},\\operatorname\{erank\}\(B\)=\\exp\\\!\\left\(\-\\sum\_\{j=1\}^\{q\}p\_\{j\}\\log p\_\{j\}\\right\),p\_\{j\}=\\frac\{\\sigma\_\{j\}\(B\)\}\{\\sum\_\{i=1\}^\{q\}\\sigma\_\{i\}\(B\)\}\.Herelog\\logdenotes the natural logarithm\. For a full\-column\-rank matrixB∈ℝn×rB\\in\\mathbb\{R\}^\{n\\times r\}, letκ⁡\(B\)=σ1​\(B\)/σr​\(B\)\\kappa\(B\)=\\sigma\_\{1\}\(B\)/\\sigma\_\{r\}\(B\)\. For a general nonzero matrix,κ\+\\kappa\_\{\+\}denotes the ratio of its largest to smallest positive singular value in Figure[6](https://arxiv.org/html/2610.02283#S4.F6)\.

###### Theorem 4\.2\(Cumulative Spectral Balance\)\.

LetK≥1K\\geq 1,n≥r≥1n\\geq r\\geq 1,B0=0B\_\{0\}=0, andBK=−∑k=1KηkDk,ηk\>0,Dk∈ℝn×r,Dk⊤Dk=Ir\.B\_\{K\}=\-\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}D\_\{k\},\\eta\_\{k\}\>0,D\_\{k\}\\in\\mathbb\{R\}^\{n\\times r\},D\_\{k\}^\{\\top\}D\_\{k\}=I\_\{r\}\.DefineCK=∑1≤i<j≤Kηi​ηj​\(Di⊤​Dj\+Dj⊤​Di\),C\_\{K\}=\\sum\_\{1\\leq i<j\\leq K\}\\eta\_\{i\}\\eta\_\{j\}\\left\(D\_\{i\}^\{\\top\}D\_\{j\}\+D\_\{j\}^\{\\top\}D\_\{i\}\\right\),aK=∑k=1Kηk2\+tr⁡\(CK\)r,EK=CK−tr⁡\(CK\)r​Ir\.a\_\{K\}=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\+\\frac\{\\operatorname\{tr\}\(C\_\{K\}\)\}\{r\},E\_\{K\}=C\_\{K\}\-\\frac\{\\operatorname\{tr\}\(C\_\{K\}\)\}\{r\}I\_\{r\}\.SupposeaK\>0a\_\{K\}\>0and‖EK‖2≤ρ​aK\\\|E\_\{K\}\\\|\_\{2\}\\leq\\rho a\_\{K\}for someρ∈\[0,1\)\\rho\\in\[0,1\)\. ThenBKB\_\{K\}has full column rank, andaK​\(1−ρ\)≤σj\(BK\)≤aK​\(1\+ρ\),j=1,…,r\.\\sqrt\{a\_\{K\}\(1\-\\rho\)\}\\leq\\sigma\_\{j\}\(B\_\{K\}\)\\leq\\sqrt\{a\_\{K\}\(1\+\\rho\)\},j=1,\\ldots,r\.Moreover,κ⁡\(BK\)=aK\+λmax​\(EK\)aK\+λmin​\(EK\)≤1\+ρ1−ρ,sr⁡\(BK\)=r​aKaK\+λmax​\(EK\)≥r1\+ρ,erank⁡\(BK\)≥r1\+‖EK‖F2/\(r​aK2\)≥r1\+ρ2\.\\kappa\(B\_\{K\}\)=\\sqrt\{\\frac\{a\_\{K\}\+\\lambda\_\{\\max\}\(E\_\{K\}\)\}\{a\_\{K\}\+\\lambda\_\{\\min\}\(E\_\{K\}\)\}\}\\leq\\sqrt\{\\frac\{1\+\\rho\}\{1\-\\rho\}\},\\operatorname\{sr\}\(B\_\{K\}\)=\\frac\{ra\_\{K\}\}\{a\_\{K\}\+\\lambda\_\{\\max\}\(E\_\{K\}\)\}\\geq\\frac\{r\}\{1\+\\rho\},\\operatorname\{erank\}\(B\_\{K\}\)\\geq\\frac\{r\}\{1\+\\\|E\_\{K\}\\\|\_\{F\}^\{2\}/\(ra\_\{K\}^\{2\}\)\}\\geq\\frac\{r\}\{1\+\\rho^\{2\}\}\.IfA∈ℝr×mA\\in\\mathbb\{R\}^\{r\\times m\}satisfiesA​A⊤=IrAA^\{\\top\}=I\_\{r\}, thenκ\+​\(BK​A\)=κ⁡\(BK\),sr⁡\(BK​A\)=sr⁡\(BK\),erank⁡\(BK​A\)=erank⁡\(BK\)\.\\kappa\_\{\+\}\(B\_\{K\}A\)=\\kappa\(B\_\{K\}\),\\operatorname\{sr\}\(B\_\{K\}A\)=\\operatorname\{sr\}\(B\_\{K\}\),\\operatorname\{erank\}\(B\_\{K\}A\)=\\operatorname\{erank\}\(B\_\{K\}\)\.

Theorem[4\.2](https://arxiv.org/html/2610.02283#S4.Thmtheorem2)follows from the decompositionBK⊤​BK=aK​Ir\+EKB\_\{K\}^\{\\top\}B\_\{K\}=a\_\{K\}I\_\{r\}\+E\_\{K\}\. The scalaraK=‖BK‖F2/ra\_\{K\}=\\\|B\_\{K\}\\\|\_\{F\}^\{2\}/rdetermines the average squared singular value, whereasEK/aKE\_\{K\}/a\_\{K\}measures relative spectral anisotropy\. A smaller admissibleρ\\rhotightens the condition\-number upper bound and raises the stable\-rank and effective\-rank lower bounds\. WhenEK=0E\_\{K\}=0, all singular values are equal, givingκ⁡\(BK\)=1\\kappa\(B\_\{K\}\)=1andsr⁡\(BK\)=erank⁡\(BK\)=r\\operatorname\{sr\}\(B\_\{K\}\)=\\operatorname\{erank\}\(B\_\{K\}\)=r\. These guarantees are conditional on cumulative anisotropy: stepwise orthogonalization alone does not ensure that the condition holds\.

###### Corollary 4\.3\(Robust Cumulative Spectral Balance\)\.

LetK≥1K\\geq 1,n≥r≥1n\\geq r\\geq 1, andBK=−∑k=1KηkDk,ηk\>0,Dk∈ℝn×r,B\_\{K\}=\-\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}D\_\{k\},\\eta\_\{k\}\>0,D\_\{k\}\\in\\mathbb\{R\}^\{n\\times r\},where‖Dk⊤​Dk−Ir‖2≤εk,εk≥0\.\\\|D\_\{k\}^\{\\top\}D\_\{k\}\-I\_\{r\}\\\|\_\{2\}\\leq\\varepsilon\_\{k\},\\varepsilon\_\{k\}\\geq 0\.DefineCKC\_\{K\},EKE\_\{K\}, andaKa\_\{K\}as in Theorem[4\.2](https://arxiv.org/html/2610.02283#S4.Thmtheorem2), and letδK=∑k=1Kηk2​εk\.\\delta\_\{K\}=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\\varepsilon\_\{k\}\.SupposeaK\>0a\_\{K\}\>0and‖EK‖2≤ρ​aK\\\|E\_\{K\}\\\|\_\{2\}\\leq\\rho a\_\{K\}for someρ∈\[0,1\)\\rho\\in\[0,1\)\. ThenaK\(1−ρ\)−δK≤σj\(BK\)2≤aK\(1\+ρ\)\+δK,j=1,…,r\.a\_\{K\}\(1\-\\rho\)\-\\delta\_\{K\}\\leq\\sigma\_\{j\}\(B\_\{K\}\)^\{2\}\\leq a\_\{K\}\(1\+\\rho\)\+\\delta\_\{K\},j=1,\\ldots,r\.IfδK<aK​\(1−ρ\)\\delta\_\{K\}<a\_\{K\}\(1\-\\rho\), defineτK=δK/aK\\tau\_\{K\}=\\delta\_\{K\}/a\_\{K\}\. ThenBKB\_\{K\}has full column rank, andκ⁡\(BK\)≤1\+ρ\+τK1−ρ−τK,sr⁡\(BK\)≥r⁡\(1−τK\)1\+ρ\+τK,erank⁡\(BK\)≥r1\+\(ρ\+τK1−τK\)2\.\\kappa\(B\_\{K\}\)\\leq\\sqrt\{\\frac\{1\+\\rho\+\\tau\_\{K\}\}\{1\-\\rho\-\\tau\_\{K\}\}\},\\operatorname\{sr\}\(B\_\{K\}\)\\geq\\frac\{r\(1\-\\tau\_\{K\}\)\}\{1\+\\rho\+\\tau\_\{K\}\},\\operatorname\{erank\}\(B\_\{K\}\)\\geq\\frac\{r\}\{1\+\\left\(\\frac\{\\rho\+\\tau\_\{K\}\}\{1\-\\tau\_\{K\}\}\\right\)^\{2\}\}\.IfA∈ℝr×mA\\in\\mathbb\{R\}^\{r\\times m\}satisfiesA​A⊤=IrAA^\{\\top\}=I\_\{r\}, the same bounds hold forκ\+​\(BK​A\)\\kappa\_\{\+\}\(B\_\{K\}A\),sr⁡\(BK​A\)\\operatorname\{sr\}\(B\_\{K\}A\), anderank⁡\(BK​A\)\\operatorname\{erank\}\(B\_\{K\}A\), respectively\.

Corollary[4\.3](https://arxiv.org/html/2610.02283#S4.Thmtheorem3)separates cross\-step anisotropyρ\\rhofrom the relative accumulated orthogonalization errorτK\\tau\_\{K\}\. The conditionρ\+τK<1\\rho\+\\tau\_\{K\}<1ensures a positive lower bound on the smallest squared singular value\. WhenτK=0\\tau\_\{K\}=0, the bounds reduce to those in Theorem[4\.2](https://arxiv.org/html/2610.02283#S4.Thmtheorem2)\. Proofs are provided in the appendix\.

Table 1:Experimental results for 10 incremental tasks on ImageNetR, ImageNetA, CIFAR100, and CUB\. We report the averaged results over 3 random seeds\. Best and second\-best results areboldandunderlined\.MethodVenue10S\-ImageNetR10S\-ImageNetA10S\-CIFAR10010S\-CUB𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrow𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrow𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrow𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrowL2PCVPR’2272\.34±0\.1772\.34\_\{\\pm 0\.17\}77\.36±0\.6477\.36\_\{\\pm 0\.64\}44\.04±0\.9344\.04\_\{\\pm 0\.93\}51\.24±2\.2651\.24\_\{\\pm 2\.26\}84\.06±0\.8884\.06\_\{\\pm 0\.88\}88\.26±1\.3488\.26\_\{\\pm 1\.34\}67\.02±1\.9067\.02\_\{\\pm 1\.90\}79\.62±1\.6079\.62\_\{\\pm 1\.60\}DualPromptECCV’2269\.10±0\.6269\.10\_\{\\pm 0\.62\}74\.28±0\.6674\.28\_\{\\pm 0\.66\}53\.19±0\.7453\.19\_\{\\pm 0\.74\}64\.59±0\.0864\.59\_\{\\pm 0\.08\}86\.93±0\.2486\.93\_\{\\pm 0\.24\}91\.13±0\.3291\.13\_\{\\pm 0\.32\}68\.48±0\.4768\.48\_\{\\pm 0\.47\}80\.59±1\.5080\.59\_\{\\pm 1\.50\}CODAPromptCVPR’2373\.31±0\.5073\.31\_\{\\pm 0\.50\}78\.47±0\.5378\.47\_\{\\pm 0\.53\}52\.08±0\.1252\.08\_\{\\pm 0\.12\}63\.92±0\.1263\.92\_\{\\pm 0\.12\}83\.21±3\.3983\.21\_\{\\pm 3\.39\}87\.71±3\.1787\.71\_\{\\pm 3\.17\}77\.23±1\.1277\.23\_\{\\pm 1\.12\}81\.90±0\.8581\.90\_\{\\pm 0\.85\}InfLoRACVPR’2474\.75±0\.6474\.75\_\{\\pm 0\.64\}80\.67±0\.5580\.67\_\{\\pm 0\.55\}49\.20±1\.1249\.20\_\{\\pm 1\.12\}60\.92±0\.6160\.92\_\{\\pm 0\.61\}86\.75±0\.3586\.75\_\{\\pm 0\.35\}91\.72±0\.1591\.72\_\{\\pm 0\.15\}70\.82±0\.2370\.82\_\{\\pm 0\.23\}81\.39±0\.1481\.39\_\{\\pm 0\.14\}SD\-LoRAICLR’2577\.34±0\.3577\.34\_\{\\pm 0\.35\}82\.04±0\.2482\.04\_\{\\pm 0\.24\}55\.96±0\.7355\.96\_\{\\pm 0\.73\}64\.95±1\.6364\.95\_\{\\pm 1\.63\}88\.01±0\.3188\.01\_\{\\pm 0\.31\}92\.54±0\.1892\.54\_\{\\pm 0\.18\}77\.48±0\.2077\.48\_\{\\pm 0\.20\}85\.59±0\.4485\.59\_\{\\pm 0\.44\}Bi\-LoRACVPR’2577\.95±0\.1477\.95\_\{\\pm 0\.14\}81\.52±0\.2681\.52\_\{\\pm 0\.26\}––87\.46±0\.7687\.46\_\{\\pm 0\.76\}92\.50±0\.6292\.50\_\{\\pm 0\.62\}––PLANICCV’2575\.25±0\.4275\.25\_\{\\pm 0\.42\}80\.41±0\.5680\.41\_\{\\pm 0\.56\}––87\.54±0\.3187\.54\_\{\\pm 0\.31\}92\.21±0\.3592\.21\_\{\\pm 0\.35\}––LoRA\-P&MNeurIPS’2579\.95±0\.1879\.95\_\{\\pm 0\.18\}85\.29±0\.9385\.29\_\{\\pm 0\.93\}56\.57±0\.7856\.57\_\{\\pm 0\.78\}65\.35±1\.8165\.35\_\{\\pm 1\.81\}88\.45±0\.3588\.45\_\{\\pm 0\.35\}92\.89±1\.1392\.89\_\{\\pm 1\.13\}78\.29±0\.5078\.29\_\{\\pm 0\.50\}83\.39±0\.6183\.39\_\{\\pm 0\.61\}CoSONeurIPS’2581\.10¯±0\.39\\underline\{81\.10\}\_\{\\pm 0\.39\}85\.56±0\.1385\.56\_\{\\pm 0\.13\}––88\.77±0\.1688\.77\_\{\\pm 0\.16\}92\.99±0\.2392\.99\_\{\\pm 0\.23\}––SplitLoRAICLR’2681\.00±0\.1781\.00\_\{\\pm 0\.17\}85\.84¯±0\.62\\underline\{85\.84\}\_\{\\pm 0\.62\}––90\.33±0\.73\\mathbf\{90\.33\_\{\\pm 0\.73\}\}93\.70¯±0\.32\\underline\{93\.70\}\_\{\\pm 0\.32\}––EBLoRAICML’2679\.56±0\.6879\.56\_\{\\pm 0\.68\}82\.68±0\.8682\.68\_\{\\pm 0\.86\}61\.56¯±0\.54\\underline\{61\.56\}\_\{\\pm 0\.54\}69\.28¯±0\.56\\underline\{69\.28\}\_\{\\pm 0\.56\}88\.98±0\.6188\.98\_\{\\pm 0\.61\}92\.86±0\.4292\.86\_\{\\pm 0\.42\}78\.92¯±0\.38\\underline\{78\.92\}\_\{\\pm 0\.38\}88\.64¯±0\.52\\underline\{88\.64\}\_\{\\pm 0\.52\}EWC\-LoRAICLR’2672\.86±0\.7972\.86\_\{\\pm 0\.79\}78\.95±0\.8678\.95\_\{\\pm 0\.86\}59\.89±0\.2659\.89\_\{\\pm 0\.26\}68\.33±0\.6768\.33\_\{\\pm 0\.67\}87\.91±0\.5787\.91\_\{\\pm 0\.57\}92\.27±0\.3992\.27\_\{\\pm 0\.39\}72\.62±0\.3272\.62\_\{\\pm 0\.32\}83\.52±0\.5483\.52\_\{\\pm 0\.54\}MuLoRA \(Ours\)–81\.77±0\.25\\mathbf\{81\.77\_\{\\pm 0\.25\}\}87\.28±0\.28\\mathbf\{87\.28\_\{\\pm 0\.28\}\}62\.23±0\.52\\mathbf\{62\.23\_\{\\pm 0\.52\}\}71\.27±1\.02\\mathbf\{71\.27\_\{\\pm 1\.02\}\}90\.03¯±0\.26\\underline\{90\.03\}\_\{\\pm 0\.26\}93\.81±1\.42\\mathbf\{93\.81\_\{\\pm 1\.42\}\}80\.27±0\.25\\mathbf\{80\.27\_\{\\pm 0\.25\}\}89\.47±0\.42\\mathbf\{89\.47\_\{\\pm 0\.42\}\}

Table 2:Experimental results for 5 and 20 incremental sessions on ImageNetR, 20 sessions on ImageNetA, and 10 sessions on DomainNet\. The highest results are inbold, and the second highest results areunderlined\.MethodVenue5S\-ImageNetR20S\-ImageNetR20S\-ImageNetA10S\-DomainNet𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrow𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrow𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrow𝒜L​a​s​t↑\\mathcal\{A\}\_\{Last\}\\uparrow𝒜A​v​g↑\\mathcal\{A\}\_\{Avg\}\\uparrowL2PCVPR’2270\.83±0\.5870\.83\_\{\\pm 0\.58\}78\.34±0\.4778\.34\_\{\\pm 0\.47\}69\.64±0\.4269\.64\_\{\\pm 0\.42\}75\.28±0\.5775\.28\_\{\\pm 0\.57\}40\.48±1\.7840\.48\_\{\\pm 1\.78\}49\.62±1\.4649\.62\_\{\\pm 1\.46\}81\.17±0\.8381\.17\_\{\\pm 0\.83\}87\.43±0\.9587\.43\_\{\\pm 0\.95\}DualPromptECCV’2273\.05±0\.5073\.05\_\{\\pm 0\.50\}79\.47±0\.4079\.47\_\{\\pm 0\.40\}66\.61±0\.5866\.61\_\{\\pm 0\.58\}72\.45±0\.3772\.45\_\{\\pm 0\.37\}42\.28±1\.9442\.28\_\{\\pm 1\.94\}53\.39±1\.6453\.39\_\{\\pm 1\.64\}81\.70±0\.7881\.70\_\{\\pm 0\.78\}87\.80±0\.9987\.80\_\{\\pm 0\.99\}CODAPromptCVPR’2374\.91±0\.3374\.91\_\{\\pm 0\.33\}79\.25±0\.5379\.25\_\{\\pm 0\.53\}69\.96±0\.5069\.96\_\{\\pm 0\.50\}75\.34±0\.8575\.34\_\{\\pm 0\.85\}44\.62±1\.9244\.62\_\{\\pm 1\.92\}54\.86±0\.5054\.86\_\{\\pm 0\.50\}80\.04±0\.7980\.04\_\{\\pm 0\.79\}86\.27±0\.8286\.27\_\{\\pm 0\.82\}InfLoRACVPR’2477\.52±0\.3777\.52\_\{\\pm 0\.37\}82\.01±0\.1282\.01\_\{\\pm 0\.12\}71\.01±0\.4571\.01\_\{\\pm 0\.45\}77\.28±0\.4577\.28\_\{\\pm 0\.45\}46\.81±2\.3046\.81\_\{\\pm 2\.30\}57\.13±0\.9557\.13\_\{\\pm 0\.95\}81\.45±0\.6881\.45\_\{\\pm 0\.68\}87\.55±0\.5787\.55\_\{\\pm 0\.57\}SD\-LoRAICLR’2579\.15±0\.2079\.15\_\{\\pm 0\.20\}83\.01±0\.4283\.01\_\{\\pm 0\.42\}75\.26±0\.3775\.26\_\{\\pm 0\.37\}80\.22±0\.7280\.22\_\{\\pm 0\.72\}49\.12±1\.7849\.12\_\{\\pm 1\.78\}59\.63±1\.5759\.63\_\{\\pm 1\.57\}81\.01±0\.4281\.01\_\{\\pm 0\.42\}86\.85±0\.4086\.85\_\{\\pm 0\.40\}Bi\-LoRACVPR’2578\.32±0\.3778\.32\_\{\\pm 0\.37\}83\.31±0\.5283\.31\_\{\\pm 0\.52\}72\.41±0\.6572\.41\_\{\\pm 0\.65\}79\.28±0\.7679\.28\_\{\\pm 0\.76\}––76\.56±0\.4176\.56\_\{\\pm 0\.41\}83\.20±0\.3583\.20\_\{\\pm 0\.35\}LoRA\-P&MNeurIPS’2581\.47±0\.5681\.47\_\{\\pm 0\.56\}85\.96±0\.5285\.96\_\{\\pm 0\.52\}76\.37±0\.0976\.37\_\{\\pm 0\.09\}82\.77±0\.3182\.77\_\{\\pm 0\.31\}52\.27±1\.8452\.27\_\{\\pm 1\.84\}62\.15±1\.3362\.15\_\{\\pm 1\.33\}84\.35¯±0\.48\\underline\{84\.35\}\_\{\\pm 0\.48\}89\.43¯±0\.27\\underline\{89\.43\}\_\{\\pm 0\.27\}CoSONeurIPS’2582\.10±0\.1382\.10\_\{\\pm 0\.13\}86\.38¯±0\.07\\underline\{86\.38\}\_\{\\pm 0\.07\}78\.19±0\.2878\.19\_\{\\pm 0\.28\}83\.69±0\.1283\.69\_\{\\pm 0\.12\}––––SplitLoRAICLR’2681\.92±0\.2981\.92\_\{\\pm 0\.29\}85\.85±0\.5585\.85\_\{\\pm 0\.55\}78\.82¯±0\.28\\underline\{78\.82\}\_\{\\pm 0\.28\}84\.57¯±0\.44\\underline\{84\.57\}\_\{\\pm 0\.44\}––84\.31±0\.2384\.31\_\{\\pm 0\.23\}88\.99±0\.5788\.99\_\{\\pm 0\.57\}EBLoRAICML’2682\.56¯±0\.32\\underline\{82\.56\}\_\{\\pm 0\.32\}86\.11±0\.2586\.11\_\{\\pm 0\.25\}77\.45±0\.2377\.45\_\{\\pm 0\.23\}83\.76±0\.6583\.76\_\{\\pm 0\.65\}52\.52¯±1\.01\\underline\{52\.52\}\_\{\\pm 1\.01\}62\.16¯±0\.82\\underline\{62\.16\}\_\{\\pm 0\.82\}83\.67±0\.1483\.67\_\{\\pm 0\.14\}89\.27±0\.2189\.27\_\{\\pm 0\.21\}EWC\-LoRAICLR’2676\.36±0\.2576\.36\_\{\\pm 0\.25\}82\.12±0\.2882\.12\_\{\\pm 0\.28\}70\.18±0\.1570\.18\_\{\\pm 0\.15\}78\.56±0\.7478\.56\_\{\\pm 0\.74\}48\.58±1\.1548\.58\_\{\\pm 1\.15\}59\.06±0\.8259\.06\_\{\\pm 0\.82\}83\.56±0\.2883\.56\_\{\\pm 0\.28\}89\.25±0\.1289\.25\_\{\\pm 0\.12\}MuLoRA \(Ours\)–83\.21±0\.24\\mathbf\{83\.21\}\_\{\\pm 0\.24\}87\.56±0\.32\\mathbf\{87\.56\}\_\{\\pm 0\.32\}78\.93±0\.18\\mathbf\{78\.93\}\_\{\\pm 0\.18\}85\.02±0\.21\\mathbf\{85\.02\}\_\{\\pm 0\.21\}55\.86±0\.61\\mathbf\{55\.86\}\_\{\\pm 0\.61\}65\.61±0\.64\\mathbf\{65\.61\}\_\{\\pm 0\.64\}85\.61±0\.10\\mathbf\{85\.61\}\_\{\\pm 0\.10\}90\.21±0\.32\\mathbf\{90\.21\}\_\{\\pm 0\.32\}

## 5Experiments

Table 3:Ablation studies on ImageNetR and ImageNetA under 10 and 20 incremental sessions, evaluating the individual and combined contributions of Relative\-Response Spectral Allocation \(RRS\) and Subspace Spectral Balancing \(SSB\)\. Idx 1 uses a fixed random orthonormalAAand trains onlyBB\.IdxRRSSSB10S\-ImageNetR10S\-ImageNetA20S\-ImageNetR20S\-ImageNetAAL​a​s​t↑A\_\{Last\}\\uparrowAA​v​g↑A\_\{Avg\}\\uparrowAL​a​s​t↑A\_\{Last\}\\uparrowAA​v​g↑A\_\{Avg\}\\uparrowAL​a​s​t↑A\_\{Last\}\\uparrowAA​v​g↑A\_\{Avg\}\\uparrowAL​a​s​t↑A\_\{Last\}\\uparrowAA​v​g↑A\_\{Avg\}\\uparrow1––76\.23±0\.1876\.23\_\{\\pm 0\.18\}82\.05±0\.6282\.05\_\{\\pm 0\.62\}57\.48±1\.4757\.48\_\{\\pm 1\.47\}67\.34±1\.8667\.34\_\{\\pm 1\.86\}76\.21±0\.1876\.21\_\{\\pm 0\.18\}80\.26±0\.5180\.26\_\{\\pm 0\.51\}48\.62±1\.2848\.62\_\{\\pm 1\.28\}60\.52±1\.2660\.52\_\{\\pm 1\.26\}2✓–79\.23±0\.2879\.23\_\{\\pm 0\.28\}85\.34±0\.3685\.34\_\{\\pm 0\.36\}59\.77±0\.5659\.77\_\{\\pm 0\.56\}68\.91±1\.0868\.91\_\{\\pm 1\.08\}77\.86±0\.2077\.86\_\{\\pm 0\.20\}83\.06±0\.3283\.06\_\{\\pm 0\.32\}51\.92±0\.6651\.92\_\{\\pm 0\.66\}62\.42±0\.5162\.42\_\{\\pm 0\.51\}3–✓80\.74±0\.2880\.74\_\{\\pm 0\.28\}86\.31±0\.3586\.31\_\{\\pm 0\.35\}61\.86±0\.5561\.86\_\{\\pm 0\.55\}70\.26±0\.9870\.26\_\{\\pm 0\.98\}78\.50±0\.2678\.50\_\{\\pm 0\.26\}84\.25±0\.3684\.25\_\{\\pm 0\.36\}54\.32±0\.6454\.32\_\{\\pm 0\.64\}64\.53±0\.5164\.53\_\{\\pm 0\.51\}4✓✓81\.77±0\.25\\mathbf\{81\.77\}\_\{\\pm 0\.25\}87\.28±0\.28\\mathbf\{87\.28\}\_\{\\pm 0\.28\}62\.23±0\.52\\mathbf\{62\.23\}\_\{\\pm 0\.52\}71\.27±1\.02\\mathbf\{71\.27\}\_\{\\pm 1\.02\}78\.93±0\.18\\mathbf\{78\.93\}\_\{\\pm 0\.18\}85\.02±0\.21\\mathbf\{85\.02\}\_\{\\pm 0\.21\}55\.86±0\.61\\mathbf\{55\.86\}\_\{\\pm 0\.61\}65\.61±0\.64\\mathbf\{65\.61\}\_\{\\pm 0\.64\}

![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/iamgeneta_all.png)\(a\)20S\-ImageNetA \(all\)
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/imageneta_new.png)\(b\)20S\-ImageNetA \(new\)
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/imagenetr_all.png)\(c\)20S\-ImageNetR \(all\)
![Refer to caption](https://arxiv.org/html/2610.02283v1/Figures/imagenetr_new.png)\(d\)20S\-ImageNetR \(new\)

Figure 7:Stability–plasticity dynamics across incremental tasks\. Overall and new\-task accuracy on 20S\-ImageNetA and 20S\-ImageNetR\.MuLoRAbetter maintains overall performance while preserving strong adaptation to newly introduced classes\.Datasets and Evaluation Metrics\.We evaluate our method on five datasets, namely ImageNetR[Hendrycks et al\. \(2021a\)](https://arxiv.org/html/2610.02283#bib.bib12), ImageNetA[Hendrycks et al\. \(2021b\)](https://arxiv.org/html/2610.02283#bib.bib13), CIFAR\-100[Krizhevsky et al\. \(2009\)](https://arxiv.org/html/2610.02283#bib.bib14), CUB[Wah et al\. \(2011\)](https://arxiv.org/html/2610.02283#bib.bib19)and DomainNet[Peng et al\. \(2019\)](https://arxiv.org/html/2610.02283#bib.bib15)\. ImageNet\-R contains 30,000 images from 200 categories and diverse styles\. ImageNet\-A includes 7,500 real\-world adversarial examples, spanning 200 classes\. CIFAR\-100 is a standard CL benchmark, including 60,000 32×32 images from 100 classes\. DomainNet consists of images from 345 classes across 6 visual domains\. Following prior works[Gao et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib6), we select the top 200 most populous classes for evaluation\. Following existing PEFT\-based methods[Tan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib5);[Wu et al\. \(2025a\)](https://arxiv.org/html/2610.02283#bib.bib20), we report results using 2 metrics: \(1\) Last accuracy \(Al​a​s​tA\_\{\{last\}\}\), defined as the accuracy over all classes after the final session and \(2\) Average accuracy \(AA​v​gA\_\{Avg\}\), defined as the mean accuracy across all incremental sessions\.

Implementation Details\.All experiments are conducted on a single NVIDIA RTX 4090 GPU\. We use ViT\-B/16[Dosovitskiy et al\. \(2021\)](https://arxiv.org/html/2610.02283#bib.bib10)pre\-trained on ImageNet\-21K[Russakovsky et al\. \(2015\)](https://arxiv.org/html/2610.02283#bib.bib16)as the backbone\. The model is optimized with the Muon[Jordan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib35)with a Cosine Annealing scheduler\. We train each task for 20 epochs with a batch size of 48\. The initial learning rate is set to 0\.0001 for ImageNetR, and 0\.0002 for others\. Following[Wu et al\. \(2025a\)](https://arxiv.org/html/2610.02283#bib.bib20), we insertMuLoRAinto all attention blocks with rank 32, and employ a cosine classifier for the classification task\.

### 5\.1Comparison with Baseline Methods

We compareMuLoRAwith representative prompt\-based, adapter\-based, and LoRA\-based continual learning methods[Wang et al\. \(2022b\)](https://arxiv.org/html/2610.02283#bib.bib1);[Wang et al\. \(2022a\)](https://arxiv.org/html/2610.02283#bib.bib2);[Jiao et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib23);[Tan et al\. \(2024\)](https://arxiv.org/html/2610.02283#bib.bib5);[Gao et al\. \(2023\)](https://arxiv.org/html/2610.02283#bib.bib4);[Liang and Li \(2024\)](https://arxiv.org/html/2610.02283#bib.bib7);[Wu et al\. \(2025b\)](https://arxiv.org/html/2610.02283#bib.bib28);[Wang et al\. \(2025\)](https://arxiv.org/html/2610.02283#bib.bib25);[Cheng et al\. \(2025\)](https://arxiv.org/html/2610.02283#bib.bib26);[Qiu et al\. \(2025b\)](https://arxiv.org/html/2610.02283#bib.bib27)\. As shown in Tables[1](https://arxiv.org/html/2610.02283#S4.T1)and[2](https://arxiv.org/html/2610.02283#S4.T2),MuLoRAachieves the highest mean accuracy in 15 of 16 reported metrics across five benchmarks and eight incremental settings\. These results support jointly addressing low\-rank capacity allocation and spectral utilization\. On 10S\-ImageNetR and 10S\-ImageNetA,MuLoRAachieves81\.77%81\.77\\%and62\.23%62\.23\\%AL​a​s​tA\_\{Last\}, respectively, exceeding the strongest reported baseline for each setting by0\.670\.67percentage points\. The correspondingAA​v​gA\_\{Avg\}gains are1\.441\.44and1\.991\.99points, indicating improvements across the learning sequence as well as at its endpoint\. The largest gains occur on 20S\-ImageNetA:MuLoRAreaches55\.86%55\.86\\%AL​a​s​tA\_\{Last\}and65\.61%65\.61\\%AA​v​gA\_\{Avg\}, improving over LoRA\-P&M by3\.593\.59and3\.463\.46points, respectively\. On ImageNetR, it also obtains the highest means under both 5\- and 20\-session protocols, although the 20\-sessionAL​a​s​tA\_\{Last\}margin is modest \(0\.110\.11points\)\. The benefits extend to CUB and DomainNet, whereMuLoRAimprovesAL​a​s​tA\_\{Last\}over the strongest reported baselines by1\.351\.35and1\.261\.26points, respectively\. On CIFAR\-100, it achieves the highestAA​v​gA\_\{Avg\}\(93\.81%93\.81\\%\), while itsAL​a​s​tA\_\{Last\}\(90\.03%90\.03\\%\) is0\.300\.30points below SplitLoRA\. Overall, the results demonstrate broad empirical benefits from combining task\-responsive subspace selection with spectrally controlled updates, without replaying past examples or historical features\.

### 5\.2Ablation Studies

#### Effectiveness of RRS and SSB\.

Table[3](https://arxiv.org/html/2610.02283#S5.T3)and Figure[7](https://arxiv.org/html/2610.02283#S5.F7)evaluate the individual and combined effects of RRS and SSB on ImageNetR and ImageNetA under 10\- and 20\-session protocols\. Both components improve performance across all four settings\. RRS alone increasesAL​a​s​tA\_\{Last\}by1\.651\.65–3\.303\.30percentage points over the baseline, supporting the benefit of selecting adaptation directions based on current\-task response relative to historical exposure\. SSB alone improvesAL​a​s​tA\_\{Last\}by2\.292\.29–5\.705\.70points, yielding larger gains than RRS alone in each setting and supporting the effectiveness of spectral balancing within the adaptation space\. Combining RRS and SSB achieves the highest mean accuracy on all eight reported metrics\. Compared with SSB alone, adding RRS further improvesAL​a​s​tA\_\{Last\}by0\.370\.37–1\.541\.54points andAA​v​gA\_\{Avg\}by0\.770\.77–1\.081\.08points\. On 20S\-ImageNetA, the full model reaches55\.86%55\.86\\%AL​a​s​tA\_\{Last\}and65\.61%65\.61\\%AA​v​gA\_\{Avg\}, exceeding the baseline by7\.247\.24and5\.095\.09points, respectively\. These results support the complementary benefits of task\-responsive subspace selection and spectral balancing: their combination consistently outperforms either component alone\.

## 6Conclusion

We introducedMuLoRAto address*spectral plasticity collapse*that can leave low\-rank adaptation capacity underutilized in continual learning\.MuLoRAcouples history\-aware subspace selection with Muon\-style spectral balancing through a fixed row\-orthonormal basis, transferring factor\-step spectra exactly to weight space\. Our analysis establishes the max–min optimality of exact subspace selection and provides conditional spectral guarantees for accumulated updates\. Experiments across five benchmarks and eight incremental settings support a central design principle: effective adaptation requires both selecting task\-responsive directions and controlling how updates use them\.

## References

- Bernstein \(2025\)J\. BernsteinDeriving Muon\.Note:Technical blog postExternal Links:[Link](https://jeremybernste.in/writing/deriving-muon)Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.02283#S3.SS2.p4.1)\.
- Chenget al\.\(2024\)D\. Cheng, Z\. Xu, X\. Jiang, N\. Wang, D\. Li, and X\. GaoDisentangled prompt representation for domain generalization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23595–23604\.Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p1.1)\.
- Chenget al\.\(2025\)Q\. Cheng, Y\. Wan, L\. Wu, C\. Hou, and L\. ZhangContinuous subspace optimization for continual learning\.InConference on Neural Information Processing Systems,Cited by:[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Dosovitskiyet al\.\(2021\)A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. HoulsbyAn image is worth 16x16 words: transformers for image recognition at scale\.ICLR\.Cited by:[§3](https://arxiv.org/html/2610.02283#S3.p2.1),[§5](https://arxiv.org/html/2610.02283#S5.p2.1)\.
- Gaoet al\.\(2023\)Q\. Gao, C\. Zhao, Y\. Sun, T\. Xi, G\. Zhang, B\. Ghanem, and J\. ZhangA unified continual learning framework with general parameter\-efficient tuning\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 11483–11493\.Cited by:[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Gaoet al\.\(2024\)Z\. Gao, J\. Cen, and X\. ChangConsistent prompting for rehearsal\-free continual learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 28463–28473\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1),[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Guet al\.\(2026\)H\. Gu, M\. Luo, Z\. Zhou, H\. Zhang, M\. Zhang, and T\. WeiSpectral imbalance causes forgetting in low\-rank continual adaptation\.arXiv preprint arXiv:2602\.00722\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.00722),[Link](https://arxiv.org/abs/2602.00722)Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Halkoet al\.\(2011\)N\. Halko, P\. Martinsson, and J\. A\. TroppFinding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions\.SIAM Review53\(2\),pp\. 217–288\.Cited by:[§3\.1](https://arxiv.org/html/2610.02283#S3.SS1.p4.1)\.
- Heet al\.\(2025\)L\. He, D\. Cheng, Z\. Ma, H\. Wang, D\. Zhang, N\. Wang, and X\. GaoCkaa: cross\-subspace knowledge alignment and aggregation for robust continual learning\.arXiv preprint arXiv:2507\.09471\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Hendryckset al\.\(2021a\)D\. Hendrycks, S\. Basart, N\. Mu, S\. Kadavath, F\. Wang, E\. Dorundo, R\. Desai, T\. Zhu, S\. Parajuli, M\. Guo,et al\.The many faces of robustness: a critical analysis of out\-of\-distribution generalization\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 8340–8349\.Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Hendryckset al\.\(2021b\)D\. Hendrycks, K\. Zhao, S\. Basart, J\. Steinhardt, and D\. SongNatural adversarial examples\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 15262–15271\.Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p1.1)\.
- Jiaoet al\.\(2024\)L\. Jiao, Q\. Lai, Y\. Li, and Q\. XuVector quantization prompting for continual learning\.NeurIPS\.Cited by:[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Jordanet al\.\(2024\)K\. Jordan, Y\. Jin, V\. Boza, J\. You, F\. Cesista, L\. Newhouse, and J\. BernsteinMuon: an optimizer for hidden layers in neural networks, 2024\.URL https://kellerjordan\. github\. io/posts/muon6\(3\),pp\. 4\.External Links:[Link](https://kellerjordan.github.io/posts/muon/)Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p3.1),[§3\.2](https://arxiv.org/html/2610.02283#S3.SS2.p4.1),[§5](https://arxiv.org/html/2610.02283#S5.p2.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.Overcoming catastrophic forgetting in neural networks\.Proceedings of the national academy of sciences114\(13\),pp\. 3521–3526\.Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p1.1)\.
- Krizhevskyet al\.\(2009\)A\. Krizhevsky G\. Hintonet al\.Learning multiple layers of features from tiny images\.Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Liet al\.\(2025\)J\. Li, S\. Wang, B\. Qian, Y\. He, X\. Wei, Q\. Wang, and Y\. GongDynamic integration of task\-specific adapters for class incremental learning\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 30545–30555\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Liang and Li \(2024\)Y\. Liang and W\. LiInfLoRA: interference\-free low\-rank adaptation for continual learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23638–23647\.Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p5.1),[§2](https://arxiv.org/html/2610.02283#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Luet al\.\(2024\)Y\. Lu, S\. Zhang, D\. Cheng, Y\. Xing, N\. Wang, P\. Wang, and Y\. ZhangVisual Prompt Tuning in Null Space for Continual Learning\.InConference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Penget al\.\(2019\)X\. Peng, Q\. Bai, X\. Xia, Z\. Huang, K\. Saenko, and B\. WangMoment matching for multi\-source domain adaptation\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 1406–1415\.Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Qiaoet al\.\(2023\)J\. Qiao, X\. Tan, C\. Chen, Y\. Qu, Y\. Peng, Y\. Xie,et al\.Prompt gradient projection for continual learning\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Qiuet al\.\(2025a\)H\. Qiu, M\. Zhang, Z\. Qiao, W\. Guan, M\. Zhang, and L\. NieSplitLoRA: balancing stability and plasticity in continual learning through gradient space splitting\.arXiv preprint arXiv:2505\.22370\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Qiuet al\.\(2025b\)H\. Qiu, M\. Zhang, Z\. Qiao, and L\. NieTrain with perturbation, infer after merging: a two\-stage framework for continual learning\.InConference on Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InInternational conference on machine learning,pp\. 8748–8763\.Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p1.1)\.
- Russakovskyet al\.\(2015\)O\. Russakovsky, J\. Deng, H\. Su, J\. Krause, S\. Satheesh, S\. Ma, Z\. Huang, A\. Karpathy, A\. Khosla, M\. Bernstein,et al\.Imagenet large scale visual recognition challenge\.International journal of computer vision115,pp\. 211–252\.Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p2.1)\.
- Smithet al\.\(2023\)J\. S\. Smith, L\. Karlinsky, V\. Gutta, P\. Cascante\-Bonilla, D\. Kim, A\. Arbelle, R\. Panda, R\. Feris, and Z\. KiraCoda\-prompt: continual decomposed attention\-based prompting for rehearsal\-free continual learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11909–11919\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Tanet al\.\(2024\)Y\. Tan, Q\. Zhou, X\. Xiang, K\. Wang, Y\. Wu, and Y\. LiSemantically\-shifted incremental adapter\-tuning is a continual vitransformer\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23252–23262\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1),[§3](https://arxiv.org/html/2610.02283#S3.p2.1),[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1),[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Wahet al\.\(2011\)C\. Wah, S\. Branson, P\. Welinder, P\. Perona, and S\. BelongieThe caltech\-ucsd birds\-200\-2011 dataset\.Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p1.1)\.
- Wanget al\.\(2026\)H\. Wang, D\. Cheng, L\. He, Y\. Li, J\. Li, N\. Wang, and X\. GaoEKPC: elastic knowledge preservation and compensation for class\-incremental learning: h\. wang et al\.\.International Journal of Computer Vision134\(5\),pp\. 238\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Wanget al\.\(2021\)S\. Wang, X\. Li, J\. Sun, and Z\. XuTraining networks in null space of feature covariance for continual learning\.InProceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition,pp\. 184–193\.Cited by:[§1](https://arxiv.org/html/2610.02283#S1.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, T\. Chen, Q\. Ge, H\. Xia, R\. Bao, R\. Zheng, Q\. Zhang, T\. Gui, and X\. HuangOrthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10658–10671\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, Z\. Zhuang, and Y\. ZhangPLAN: proactive low\-rank allocation for continual learning\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 2909–2918\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Wanget al\.\(2022a\)Z\. Wang, Z\. Zhang, S\. Ebrahimi, R\. Sun, H\. Zhang, C\. Lee, X\. Ren, G\. Su, V\. Perot, J\. Dy,et al\.Dualprompt: complementary prompting for rehearsal\-free continual learning\.InEuropean Conference on Computer Vision,pp\. 631–648\.Cited by:[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Wanget al\.\(2022b\)Z\. Wang, Z\. Zhang, C\. Lee, H\. Zhang, R\. Sun, X\. Ren, G\. Su, V\. Perot, J\. Dy, and T\. PfisterLearning to prompt for continual learning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 139–149\.Cited by:[§3](https://arxiv.org/html/2610.02283#S3.p2.1),[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Wuet al\.\(2025a\)F\. Wu, L\. Cheng, S\. Tang, X\. Zhu, C\. Fang, D\. Zhang, and M\. WangNavigating semantic drift in task\-agnostic class\-incremental learning\.InForty\-second International Conference on Machine Learning,Cited by:[§5](https://arxiv.org/html/2610.02283#S5.p1.1),[§5](https://arxiv.org/html/2610.02283#S5.p2.1)\.
- Wuet al\.\(2025b\)Y\. Wu, H\. Piao, L\. Huang, R\. Wang, W\. Li, H\. Pfister, D\. Meng, K\. Ma, and Y\. WeiSD\-loRA: scalable decoupled low\-rank adaptation for class incremental learning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5U1rlpX68A)Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.02283#S5.SS1.p1.1)\.
- Zhenget al\.\(2026\)Y\. Zheng, Y\. Zhang, J\. van de Weijer, G\. M\. van de Ven, S\. Du, X\. Zhang, and Z\. TianRevisiting weight regularization for low\-rank continual learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=pZj2DhfaVD)Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.
- Zhuet al\.\(2025\)H\. Zhu, Y\. Zhang, J\. Dong, and P\. KoniuszBiLoRA: almost\-orthogonal parameter spaces for continual learning\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 25613–25622\.Cited by:[§2](https://arxiv.org/html/2610.02283#S2.p1.1)\.

## Ethics Statement

This work studies parameter\-efficient continual learning using existing image classification benchmarks: ImageNet\-R, ImageNet\-A, CIFAR\-100, CUB, and DomainNet\. Our experiments use benchmark datasets and pretrained models rather than collecting new data from human participants\. Although MuLoRA aims to improve adaptation across sequential tasks, it does not provide formal guarantees of privacy, fairness, or robustness\. Models may inherit biases from pretraining and downstream datasets, and aggregate classification accuracy does not establish reliable performance across demographic groups or deployment conditions\. Deployment in sensitive applications therefore requires appropriate privacy safeguards, bias assessment, and application\-specific evaluation\.

## Reproducibility Statement

The paper describes MuLoRA’s task\-adaptive subspace construction and momentum orthogonalization, together with the assumptions underlying its theoretical results\. The appendix provides detailed proofs of the subspace selection characterization, conditional cumulative spectral bounds, and preservation of nonzero singular values under a row\-orthonormal basis\. The experimental section specifies the benchmark datasets, incremental settings, evaluation metrics, pretrained ViT\-B/16 backbone, adapter placement and rank, optimizer, learning rates, learning\-rate schedule, training epochs, and batch size\. Experiments are conducted on a single NVIDIA RTX 4090 GPU, and the main comparison reports results averaged over three random seeds\. We will publicly release the implementation and experiment scripts upon acceptance\.

## AI Use Statement

We used large language models to assist with language editing, including sentence rephrasing, grammar correction, and readability improvements\. We did not use these tools for research ideation, methodology or experiment design, theoretical derivations, method implementation, data generation or processing, or interpretation of results\. The authors reviewed and revised all AI\-assisted text and take full responsibility for the final manuscript, including its claims, analyses, and conclusions\.

## Appendix Contents

## Appendix AProofs of the Theoretical Results

We prove the exact relative\-response guarantee and the cumulative spectral bounds stated in the main text\. All matrices are real, and all logarithms are natural\. Singular values are ordered nonincreasingly\. For the cumulative analysis,n≥rn\\geq r; allrrsingular values ofB∈ℝn×rB\\in\\mathbb\{R\}^\{n\\times r\}are retained, including zeros\. The cumulative analysis concerns optimization steps within one task, during which the input basis is fixed\.

### A\.1Two spectral preliminaries

#### Spectral quantities\.

For a nonzero matrixBBwith positive singular valuess1≥⋯≥sq\>0s\_\{1\}\\geq\\cdots\\geq s\_\{q\}\>0, we use

sr\(B\)=∑j=1qsj2s12,κ\+\(B\)=s1sq,erank\(B\)=exp\(−∑j=1qpjlogpj\),pj=sj∑isi\.\\operatorname\{sr\}\(B\)=\\frac\{\\sum\_\{j=1\}^\{q\}s\_\{j\}^\{2\}\}\{s\_\{1\}^\{2\}\},\\qquad\\kappa\_\{\+\}\(B\)=\\frac\{s\_\{1\}\}\{s\_\{q\}\},\\qquad\\operatorname\{erank\}\(B\)=\\exp\\\!\\left\(\-\\sum\_\{j=1\}^\{q\}p\_\{j\}\\log p\_\{j\}\\right\),\\quad p\_\{j\}=\\frac\{s\_\{j\}\}\{\\sum\_\{i\}s\_\{i\}\}\.\(4\)Entropy\-based effective rank differs from the energy participation ratio\(∑jsj2\)2/∑jsj4\(\\sum\_\{j\}s\_\{j\}^\{2\}\)^\{2\}/\\sum\_\{j\}s\_\{j\}^\{4\}; the next lemma relates them by inequalities\. Neither is the algebraic rank\. In particular,κ\+\\kappa\_\{\+\}excludes zero singular values and must be interpreted together with rank information\. For the zero matrix, normalized spectral statistics are undefined\.

###### Lemma A\.1\(Preservation of the nonzero singular spectrum\)\.

LetB∈ℝn×rB\\in\\mathbb\{R\}^\{n\\times r\}andA∈ℝr×mA\\in\\mathbb\{R\}^\{r\\times m\}satisfyA​A⊤=IrAA^\{\\top\}=I\_\{r\}\. ThenB​ABAandBBhave the same nonzero singular values, including multiplicities, and the same rank\. Consequently, forB≠0B\\neq 0, their stable ranks, entropy\-based effective ranks, and nonzero condition numbers coincide\. IfBBhas full column rank, thenκ\+​\(B​A\)=κ⁡\(B\)\\kappa\_\{\+\}\(BA\)=\\kappa\(B\)\.

###### Proof\.

The left Gram matrices coincide:

\(B​A\)​\(B​A\)⊤=B​A​A⊤​B⊤=B​B⊤\.\(BA\)\(BA\)^\{\\top\}=BAA^\{\\top\}B^\{\\top\}=BB^\{\\top\}\.\(5\)The positive eigenvalues of a left Gram matrix are precisely the squares of the nonzero singular values of the corresponding matrix\. Thus the nonzero singular spectra, their multiplicities, and the ranks agree\. Each stated spectral statistic is determined entirely by these nonzero singular values, which proves the remaining claims\. ∎

###### Lemma A\.2\(Entropy effective rank and energy participation\)\.

LetB≠0B\\neq 0have positive singular valuess1,…,sqs\_\{1\},\\ldots,s\_\{q\}, and set

Sℓ=∑j=1qsjℓ,pj=sjS1,erank\(B\)=exp\(−∑j=1qpjlogpj\)\.S\_\{\\ell\}=\\sum\_\{j=1\}^\{q\}s\_\{j\}^\{\\ell\},\\qquad p\_\{j\}=\\frac\{s\_\{j\}\}\{S\_\{1\}\},\\qquad\\operatorname\{erank\}\(B\)=\\exp\\left\(\-\\sum\_\{j=1\}^\{q\}p\_\{j\}\\log p\_\{j\}\\right\)\.\(6\)Then

erank⁡\(B\)≥S12S2≥S22S4=\(tr⁡\(B⊤​B\)\)2tr⁡\(\(B⊤​B\)2\)\.\\operatorname\{erank\}\(B\)\\geq\\frac\{S\_\{1\}^\{2\}\}\{S\_\{2\}\}\\geq\\frac\{S\_\{2\}^\{2\}\}\{S\_\{4\}\}=\\frac\{\\bigl\(\\operatorname\{tr\}\(B^\{\\top\}B\)\\bigr\)^\{2\}\}\{\\operatorname\{tr\}\\bigl\(\(B^\{\\top\}B\)^\{2\}\\bigr\)\}\.\(7\)The last quantity is the participation rank of the squared singular values; it is generally distinct fromerank⁡\(B\)\\operatorname\{erank\}\(B\)\. If all positive singular values are equal, all quantities in equation[7](https://arxiv.org/html/2610.02283#A1.E7)equalqq\.

###### Proof\.

By concavity of the logarithm, using the positive weightspjp\_\{j\},

∑j=1qpj​log⁡pj≤log⁡\(∑j=1qpj2\)\.\\sum\_\{j=1\}^\{q\}p\_\{j\}\\log p\_\{j\}\\leq\\log\\left\(\\sum\_\{j=1\}^\{q\}p\_\{j\}^\{2\}\\right\)\.\(8\)Negating and exponentiating gives

erank⁡\(B\)≥1∑jpj2=S12S2\.\\operatorname\{erank\}\(B\)\\geq\\frac\{1\}\{\\sum\_\{j\}p\_\{j\}^\{2\}\}=\\frac\{S\_\{1\}^\{2\}\}\{S\_\{2\}\}\.\(9\)Next, Hölder’s inequality with conjugate exponents3/23/2and33yields

S2=∑j=1qsj2/3​sj4/3≤\(∑j=1qsj\)2/3​\(∑j=1qsj4\)1/3=S12/3​S41/3\.S\_\{2\}=\\sum\_\{j=1\}^\{q\}s\_\{j\}^\{2/3\}s\_\{j\}^\{4/3\}\\leq\\left\(\\sum\_\{j=1\}^\{q\}s\_\{j\}\\right\)^\{2/3\}\\left\(\\sum\_\{j=1\}^\{q\}s\_\{j\}^\{4\}\\right\)^\{1/3\}=S\_\{1\}^\{2/3\}S\_\{4\}^\{1/3\}\.\(10\)ThereforeS23≤S12​S4S\_\{2\}^\{3\}\\leq S\_\{1\}^\{2\}S\_\{4\}\. SinceS2,S4\>0S\_\{2\},S\_\{4\}\>0, division givesS12/S2≥S22/S4S\_\{1\}^\{2\}/S\_\{2\}\\geq S\_\{2\}^\{2\}/S\_\{4\}\. Finally,tr⁡\(B⊤​B\)=S2\\operatorname\{tr\}\(B^\{\\top\}B\)=S\_\{2\}andtr⁡\(\(B⊤​B\)2\)=S4\\operatorname\{tr\}\(\(B^\{\\top\}B\)^\{2\}\)=S\_\{4\}, including whenBBis rank deficient\. Equal positive singular values givepj=1/qp\_\{j\}=1/qand equality throughout\. ∎

### A\.2Exact relative\-response subspace selection

This subsection proves Proposition[4\.1](https://arxiv.org/html/2610.02283#S4.Thmtheorem1)\.

###### Proof\.

Assume1≤r≤m1\\leq r\\leq m, and abbreviateH=Ht−1H=H\_\{t\-1\},L=Lt−1L=L\_\{t\-1\},M=MtM=M^\{t\}, andM~=L−1​M​L−⁣⊤\\widetilde\{M\}=L^\{\-1\}ML^\{\-\\top\}\. SinceH=L​L⊤≻0H=LL^\{\\top\}\\succ 0,LLis invertible\. Moreover,M~\\widetilde\{M\}is symmetric positive semidefinite\.

#### Step 1: Transform the relative response\.

For everyv≠0v\\neq 0, setu=L⊤​vu=L^\{\\top\}v, so thatv=L−⁣⊤​uv=L^\{\-\\top\}u\. Then

v⊤​H​v=u⊤​u,v⊤​M​v=u⊤​M~​u\.v^\{\\top\}Hv=u^\{\\top\}u,\\qquad v^\{\\top\}Mv=u^\{\\top\}\\widetilde\{M\}u\.\(11\)Consequently,

v⊤​M​vv⊤​H​v=u⊤​M~​uu⊤​u\.\\frac\{v^\{\\top\}Mv\}\{v^\{\\top\}Hv\}=\\frac\{u^\{\\top\}\\widetilde\{M\}u\}\{u^\{\\top\}u\}\.\(12\)Thus historical whitening converts the generalized Rayleigh quotient into an ordinary Rayleigh quotient through an invertible change of coordinates\.

#### Step 2: Bound every direction in the selected subspace\.

LetΛr=diag⁡\(λ1,…,λr\)\\Lambda\_\{r\}=\\operatorname\{diag\}\(\\lambda\_\{1\},\\ldots,\\lambda\_\{r\}\)\. SinceUr⊤​Ur=IrU\_\{r\}^\{\\top\}U\_\{r\}=I\_\{r\}andM~​Ur=Ur​Λr\\widetilde\{M\}U\_\{r\}=U\_\{r\}\\Lambda\_\{r\}, every nonzerov∈𝒮tv\\in\\mathcal\{S\}\_\{t\}can be written uniquely asv=L−⁣⊤​Ur​cv=L^\{\-\\top\}U\_\{r\}cfor somec∈ℝr∖\{0\}c\\in\\mathbb\{R\}^\{r\}\\setminus\\\{0\\\}\. Eq\. equation[12](https://arxiv.org/html/2610.02283#A1.E12)gives

v⊤​M​vv⊤​H​v=c⊤​Λr​cc⊤​c=∑j=1rλj​cj2∑j=1rcj2∈\[λr,λ1\]\.\\frac\{v^\{\\top\}Mv\}\{v^\{\\top\}Hv\}=\\frac\{c^\{\\top\}\\Lambda\_\{r\}c\}\{c^\{\\top\}c\}=\\frac\{\\sum\_\{j=1\}^\{r\}\\lambda\_\{j\}c\_\{j\}^\{2\}\}\{\\sum\_\{j=1\}^\{r\}c\_\{j\}^\{2\}\}\\in\[\\lambda\_\{r\},\\lambda\_\{1\}\]\.\(13\)Choosingccto be therr\-th coordinate vector attains the lower endpoint\. Hence the minimum response over𝒮t∖\{0\}\\mathcal\{S\}\_\{t\}\\setminus\\\{0\\\}is exactlyλr\\lambda\_\{r\}\.

#### Step 3: Establish global max–min optimality\.

The map𝒮↦L⊤​𝒮\\mathcal\{S\}\\mapsto L^\{\\top\}\\mathcal\{S\}is a bijection on the set ofrr\-dimensional subspaces\. Therefore

maxdim\(𝒮\)=r⁡minv∈𝒮∖\{0\}​v⊤​M​vv⊤​H​v\\displaystyle\\max\_\{\\dim\(\\mathcal\{S\}\)=r\}\\min\_\{v\\in\\mathcal\{S\}\\setminus\\\{0\\\}\}\\frac\{v^\{\\top\}Mv\}\{v^\{\\top\}Hv\}=maxdim\(𝒰\)=r⁡minu∈𝒰∖\{0\}​u⊤​M~​uu⊤​u=λr\.\\displaystyle=\\max\_\{\\dim\(\\mathcal\{U\}\)=r\}\\min\_\{u\\in\\mathcal\{U\}\\setminus\\\{0\\\}\}\\frac\{u^\{\\top\}\\widetilde\{M\}u\}\{u^\{\\top\}u\}=\\lambda\_\{r\}\.\(14\)For completeness, the last equality can also be verified directly\. Extend the columns ofUrU\_\{r\}to an orthonormal eigenbasisu1,…,umu\_\{1\},\\ldots,u\_\{m\}ofM~\\widetilde\{M\}, ordered by eigenvalue\. Everyrr\-dimensional subspace𝒰\\mathcal\{U\}intersectsspan⁡\{ur,…,um\}\\operatorname\{span\}\\\{u\_\{r\},\\ldots,u\_\{m\}\\\}nontrivially, since their dimensions sum tom\+1m\+1\. A nonzero vector in the intersection has Rayleigh quotient at mostλr\\lambda\_\{r\}\. Thus no such𝒰\\mathcal\{U\}has minimum Rayleigh quotient greater thanλr\\lambda\_\{r\}\. The subspacerange⁡\(Ur\)\\operatorname\{range\}\(U\_\{r\}\)attainsλr\\lambda\_\{r\}by equation[13](https://arxiv.org/html/2610.02283#A1.E13)\. This is the relevant Courant–Fischer characterization\.

#### Step 4: Verify preservation under Euclidean QR\.

The matrixZ=L−⁣⊤​UrZ=L^\{\-\\top\}U\_\{r\}has full column rank\. In its thin QR factorizationZ=Qt​RtZ=Q\_\{t\}R\_\{t\},Qt⊤​Qt=IrQ\_\{t\}^\{\\top\}Q\_\{t\}=I\_\{r\}andRtR\_\{t\}is invertible\. Hence

range⁡\(Qt\)=range⁡\(Z\)=𝒮t\.\\operatorname\{range\}\(Q\_\{t\}\)=\\operatorname\{range\}\(Z\)=\\mathcal\{S\}\_\{t\}\.\(15\)DefiningAt=Qt⊤A\_\{t\}=Q\_\{t\}^\{\\top\}therefore gives

At​At⊤=Ir,row⁡\(At\)=range⁡\(At⊤\)=𝒮t,A\_\{t\}A\_\{t\}^\{\\top\}=I\_\{r\},\\qquad\\operatorname\{row\}\(A\_\{t\}\)=\\operatorname\{range\}\(A\_\{t\}^\{\\top\}\)=\\mathcal\{S\}\_\{t\},\(16\)where row vectors are identified with their transposes\. Thus QR preserves the max–min\-optimal subspace and provides the orthonormal parameterization required by Lemma[A\.1](https://arxiv.org/html/2610.02283#A1.Thmtheorem1)\. ∎

#### Scope of the subspace guarantee\.

The proposition concerns exact leading eigenvectors\. Repeated eigenvalues may make the optimal subspace nonunique, but do not affect the result\. With randomized SVD, QR still preserves the computed subspace exactly; global max–min optimality is not automatic for that approximate subspace\. Also, QR need not preserve the constraintZ⊤​H​Z=IrZ^\{\\top\}HZ=I\_\{r\}: it preserves the subspace and its relative\-response guarantee, while replacing the basis by a Euclidean\-orthonormal one\.

### A\.3Cumulative spectral balance with exact steps

This subsection proves Theorem[4\.2](https://arxiv.org/html/2610.02283#S4.Thmtheorem2)\. For clarity, the quantities used below are

CK\\displaystyle C\_\{K\}=∑1≤i<j≤Kηi​ηj​\(Di⊤​Dj\+Dj⊤​Di\),\\displaystyle=\\sum\_\{1\\leq i<j\\leq K\}\\eta\_\{i\}\\eta\_\{j\}\(D\_\{i\}^\{\\top\}D\_\{j\}\+D\_\{j\}^\{\\top\}D\_\{i\}\),\(17\)aK\\displaystyle a\_\{K\}=∑k=1Kηk2\+tr⁡\(CK\)r,EK=CK−tr⁡\(CK\)r​Ir\.\\displaystyle=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\+\\frac\{\\operatorname\{tr\}\(C\_\{K\}\)\}\{r\},\\qquad E\_\{K\}=C\_\{K\}\-\\frac\{\\operatorname\{tr\}\(C\_\{K\}\)\}\{r\}I\_\{r\}\.\(18\)The assumptions areB0=0B\_\{0\}=0,ηk\>0\\eta\_\{k\}\>0,Dk⊤​Dk=IrD\_\{k\}^\{\\top\}D\_\{k\}=I\_\{r\},aK\>0a\_\{K\}\>0, and‖EK‖2≤ρ​aK\\\|E\_\{K\}\\\|\_\{2\}\\leq\\rho a\_\{K\}for0≤ρ<10\\leq\\rho<1\.

###### Proof\.

For brevity, writeB=BKB=B\_\{K\},C=CKC=C\_\{K\},a=aKa=a\_\{K\}, andE=EKE=E\_\{K\}\.

#### Step 1: Expand the cumulative Gram matrix\.

SinceB=−∑k=1KηkDkB=\-\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}D\_\{k\}andDk⊤​Dk=IrD\_\{k\}^\{\\top\}D\_\{k\}=I\_\{r\},

B⊤​B\\displaystyle B^\{\\top\}B=∑k=1Kηk2​Dk⊤​Dk\+∑1≤i<j≤Kηi​ηj​\(Di⊤​Dj\+Dj⊤​Di\)\\displaystyle=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}D\_\{k\}^\{\\top\}D\_\{k\}\+\\sum\_\{1\\leq i<j\\leq K\}\\eta\_\{i\}\\eta\_\{j\}\(D\_\{i\}^\{\\top\}D\_\{j\}\+D\_\{j\}^\{\\top\}D\_\{i\}\)=\(∑k=1Kηk2\)​Ir\+C=a​Ir\+E\.\\displaystyle=\\left\(\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\\right\)I\_\{r\}\+C=aI\_\{r\}\+E\.\(19\)By definitionEEis symmetric andtr⁡\(E\)=0\\operatorname\{tr\}\(E\)=0\. Taking traces gives

‖B‖F2=tr⁡\(B⊤​B\)=r​a\.\\\|B\\\|\_\{F\}^\{2\}=\\operatorname\{tr\}\(B^\{\\top\}B\)=ra\.\(20\)

#### Step 2: Bound singular values and the condition number\.

Lete1≥⋯≥ere\_\{1\}\\geq\\cdots\\geq e\_\{r\}be the eigenvalues ofEE\. Eq\. equation[19](https://arxiv.org/html/2610.02283#A1.E19)impliesσj​\(B\)2=a\+ej\\sigma\_\{j\}\(B\)^\{2\}=a\+e\_\{j\}\. Because\|ej\|≤‖E‖2≤ρ​a\|e\_\{j\}\|\\leq\\\|E\\\|\_\{2\}\\leq\\rho a,

a\(1−ρ\)≤σj\(B\)2≤a\(1\+ρ\),j=1,…,r\.a\(1\-\\rho\)\\leq\\sigma\_\{j\}\(B\)^\{2\}\\leq a\(1\+\\rho\),\\qquad j=1,\\ldots,r\.\(21\)The lower bound is strictly positive becausea\>0a\>0andρ<1\\rho<1\. ThusBBhas full column rank, and taking square roots proves the singular\-value bounds\. In particular,

κ⁡\(B\)=a\+λmax​\(E\)a\+λmin​\(E\)≤1\+ρ1−ρ\.\\kappa\(B\)=\\sqrt\{\\frac\{a\+\\lambda\_\{\\max\}\(E\)\}\{a\+\\lambda\_\{\\min\}\(E\)\}\}\\leq\\sqrt\{\\frac\{1\+\\rho\}\{1\-\\rho\}\}\.\(22\)

#### Step 3: Bound the stable rank\.

Using equation[20](https://arxiv.org/html/2610.02283#A1.E20)and‖B‖22=a\+λmax​\(E\)\\\|B\\\|\_\{2\}^\{2\}=a\+\\lambda\_\{\\max\}\(E\)yields

sr⁡\(B\)=r​aa\+λmax​\(E\)≥r1\+ρ\.\\operatorname\{sr\}\(B\)=\\frac\{ra\}\{a\+\\lambda\_\{\\max\}\(E\)\}\\geq\\frac\{r\}\{1\+\\rho\}\.\(23\)

#### Step 4: Bound the entropy\-based effective rank\.

SinceEEis symmetric and traceless,

tr⁡\(\(B⊤​B\)2\)\\displaystyle\\operatorname\{tr\}\\bigl\(\(B^\{\\top\}B\)^\{2\}\\bigr\)=tr⁡\(\(a​Ir\+E\)2\)\\displaystyle=\\operatorname\{tr\}\\bigl\(\(aI\_\{r\}\+E\)^\{2\}\\bigr\)=r​a2\+2​a​tr⁡\(E\)\+tr⁡\(E2\)=r​a2\+‖E‖F2\.\\displaystyle=ra^\{2\}\+2a\\operatorname\{tr\}\(E\)\+\\operatorname\{tr\}\(E^\{2\}\)=ra^\{2\}\+\\\|E\\\|\_\{F\}^\{2\}\.\(24\)Lemma[A\.2](https://arxiv.org/html/2610.02283#A1.Thmtheorem2)therefore gives

erank⁡\(B\)≥\(r​a\)2r​a2\+‖E‖F2=r1\+‖E‖F2/\(r​a2\)\.\\operatorname\{erank\}\(B\)\\geq\\frac\{\(ra\)^\{2\}\}\{ra^\{2\}\+\\\|E\\\|\_\{F\}^\{2\}\}=\\frac\{r\}\{1\+\\\|E\\\|\_\{F\}^\{2\}/\(ra^\{2\}\)\}\.\(25\)Finally,‖E‖F2≤r​‖E‖22≤r​ρ2​a2\\\|E\\\|\_\{F\}^\{2\}\\leq r\\\|E\\\|\_\{2\}^\{2\}\\leq r\\rho^\{2\}a^\{2\}, so

erank⁡\(B\)≥r1\+ρ2\.\\operatorname\{erank\}\(B\)\\geq\\frac\{r\}\{1\+\\rho^\{2\}\}\.\(26\)This argument bounds the entropy effective rank through energy participation; it does not equate the two definitions\.

#### Step 5: Transfer the guarantees to the weight update\.

IfA​A⊤=IrAA^\{\\top\}=I\_\{r\}, Lemma[A\.1](https://arxiv.org/html/2610.02283#A1.Thmtheorem1)implies

κ\+​\(B​A\)=κ⁡\(B\),sr⁡\(B​A\)=sr⁡\(B\),erank⁡\(B​A\)=erank⁡\(B\)\.\\kappa\_\{\+\}\(BA\)=\\kappa\(B\),\\qquad\\operatorname\{sr\}\(BA\)=\\operatorname\{sr\}\(B\),\\qquad\\operatorname\{erank\}\(BA\)=\\operatorname\{erank\}\(B\)\.\(27\)Thus all stated bounds transfer without spectral distortion\. ∎

### A\.4Cumulative spectral balance with approximate steps

This subsection proves Corollary[4\.3](https://arxiv.org/html/2610.02283#S4.Thmtheorem3)\. Here‖Dk⊤​Dk−Ir‖2≤εk\\\|D\_\{k\}^\{\\top\}D\_\{k\}\-I\_\{r\}\\\|\_\{2\}\\leq\\varepsilon\_\{k\}andδK=∑k=1Kηk2​εk\\delta\_\{K\}=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\\varepsilon\_\{k\}\. The definitions ofCK,aK,EKC\_\{K\},a\_\{K\},E\_\{K\}are unchanged and use the actual steps\. We assumeaK\>0a\_\{K\}\>0and‖EK‖2≤ρ​aK\\\|E\_\{K\}\\\|\_\{2\}\\leq\\rho a\_\{K\},0≤ρ<10\\leq\\rho<1; the condition\-number and rank bounds additionally requireδK<aK​\(1−ρ\)\\delta\_\{K\}<a\_\{K\}\(1\-\\rho\)\.

###### Proof\.

WriteB=BKB=B\_\{K\},C=CKC=C\_\{K\},a=aKa=a\_\{K\},E=EKE=E\_\{K\},δ=δK\\delta=\\delta\_\{K\}, and, when the strict positivity condition holds,τ=δ/a\\tau=\\delta/a\.

#### Step 1: Isolate the accumulated orthogonalization error\.

Define symmetric matrices

Fk=Dk⊤​Dk−Ir,F=∑k=1Kηk2​Fk\.F\_\{k\}=D\_\{k\}^\{\\top\}D\_\{k\}\-I\_\{r\},\\qquad F=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}F\_\{k\}\.\(28\)The assumed stepwise error bounds imply

‖F‖2≤∑k=1Kηk2​‖Fk‖2≤∑k=1Kηk2​εk=δ\.\\\|F\\\|\_\{2\}\\leq\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\\\|F\_\{k\}\\\|\_\{2\}\\leq\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\\varepsilon\_\{k\}=\\delta\.\(29\)Expanding the Gram matrix as before now gives

B⊤​B=a​Ir\+E\+F\.B^\{\\top\}B=aI\_\{r\}\+E\+F\.\(30\)The cross\-step matricesC,EC,Eand scalaraaare defined using the actual, possibly nonorthogonalDkD\_\{k\}\.

#### Step 2: Bound the spectrum and establish full rank\.

For every unit vectorx∈ℝrx\\in\\mathbb\{R\}^\{r\}, symmetry and equation[29](https://arxiv.org/html/2610.02283#A1.E29)imply

a−ρ​a−δ≤x⊤​B⊤​B​x≤a\+ρ​a\+δ\.a\-\\rho a\-\\delta\\leq x^\{\\top\}B^\{\\top\}Bx\\leq a\+\\rho a\+\\delta\.\(31\)Hence every eigenvalue ofB⊤​BB^\{\\top\}Bsatisfies

a\(1−ρ\)−δ≤σj\(B\)2≤a\(1\+ρ\)\+δ,j=1,…,r\.a\(1\-\\rho\)\-\\delta\\leq\\sigma\_\{j\}\(B\)^\{2\}\\leq a\(1\+\\rho\)\+\\delta,\\qquad j=1,\\ldots,r\.\(32\)This bound remains valid even if its lower endpoint is nonpositive\. For the remaining claims, assumeδ<a⁡\(1−ρ\)\\delta<a\(1\-\\rho\)\. Thenρ\+τ<1\\rho\+\\tau<1, and all singular values are positive\. Dividing the extreme bounds in equation[32](https://arxiv.org/html/2610.02283#A1.E32)gives

κ⁡\(B\)≤1\+ρ\+τ1−ρ−τ\.\\kappa\(B\)\\leq\\sqrt\{\\frac\{1\+\\rho\+\\tau\}\{1\-\\rho\-\\tau\}\}\.\(33\)

#### Step 3: Bound the stable rank\.

Becausetr⁡\(E\)=0\\operatorname\{tr\}\(E\)=0and\|tr⁡\(F\)\|≤r​‖F‖2≤r​δ\|\\operatorname\{tr\}\(F\)\|\\leq r\\\|F\\\|\_\{2\}\\leq r\\delta,

‖B‖F2=r​a\+tr⁡\(F\)≥r⁡\(a−δ\)=r​a​\(1−τ\)\.\\\|B\\\|\_\{F\}^\{2\}=ra\+\\operatorname\{tr\}\(F\)\\geq r\(a\-\\delta\)=ra\(1\-\\tau\)\.\(34\)Combining this with‖B‖22≤a⁡\(1\+ρ\)\+δ=a⁡\(1\+ρ\+τ\)\\\|B\\\|\_\{2\}^\{2\}\\leq a\(1\+\\rho\)\+\\delta=a\(1\+\\rho\+\\tau\)yields

sr⁡\(B\)≥r⁡\(1−τ\)1\+ρ\+τ\.\\operatorname\{sr\}\(B\)\\geq\\frac\{r\(1\-\\tau\)\}\{1\+\\rho\+\\tau\}\.\(35\)

#### Step 4: Center the Gram matrix at its actual mean\.

In the approximate case,aaneed not equal‖B‖F2/r\\\|B\\\|\_\{F\}^\{2\}/r\. Define

f=tr⁡\(F\)r,μ=a\+f=‖B‖F2r,T=E\+F−f​Ir\.f=\\frac\{\\operatorname\{tr\}\(F\)\}\{r\},\\qquad\\mu=a\+f=\\frac\{\\\|B\\\|\_\{F\}^\{2\}\}\{r\},\\qquad T=E\+F\-fI\_\{r\}\.\(36\)Then

B⊤​B=μ​Ir\+T,tr⁡\(T\)=0,μ≥a−δ=a⁡\(1−τ\)\>0\.B^\{\\top\}B=\\mu I\_\{r\}\+T,\\qquad\\operatorname\{tr\}\(T\)=0,\\qquad\\mu\\geq a\-\\delta=a\(1\-\\tau\)\>0\.\(37\)To boundTT, use the Frobenius norm\. Direct expansion gives

‖F−f​Ir‖F2=‖F‖F2−\(tr⁡\(F\)\)2r≤‖F‖F2≤r​δ2\.\\\|F\-fI\_\{r\}\\\|\_\{F\}^\{2\}=\\\|F\\\|\_\{F\}^\{2\}\-\\frac\{\\bigl\(\\operatorname\{tr\}\(F\)\\bigr\)^\{2\}\}\{r\}\\leq\\\|F\\\|\_\{F\}^\{2\}\\leq r\\delta^\{2\}\.\(38\)Thus, by the triangle inequality,

‖T‖F\\displaystyle\\\|T\\\|\_\{F\}≤‖E‖F\+‖F−f​Ir‖F\\displaystyle\\leq\\\|E\\\|\_\{F\}\+\\\|F\-fI\_\{r\}\\\|\_\{F\}≤r​ρ​a\+r​δ=r​a​\(ρ\+τ\)\.\\displaystyle\\leq\\sqrt\{r\}\\,\\rho a\+\\sqrt\{r\}\\,\\delta=\\sqrt\{r\}\\,a\(\\rho\+\\tau\)\.\(39\)The centering estimate is made in the Frobenius norm; no contraction of the spectral norm under trace subtraction is required\.

#### Step 5: Bound the entropy\-based effective rank\.

Using equation[37](https://arxiv.org/html/2610.02283#A1.E37), symmetry and tracelessness give

tr⁡\(\(B⊤​B\)2\)=r​μ2\+‖T‖F2\.\\operatorname\{tr\}\\bigl\(\(B^\{\\top\}B\)^\{2\}\\bigr\)=r\\mu^\{2\}\+\\\|T\\\|\_\{F\}^\{2\}\.\(40\)Lemma[A\.2](https://arxiv.org/html/2610.02283#A1.Thmtheorem2)and equation[39](https://arxiv.org/html/2610.02283#A1.E39)therefore imply

erank⁡\(B\)\\displaystyle\\operatorname\{erank\}\(B\)≥\(r​μ\)2r​μ2\+‖T‖F2=r1\+‖T‖F2/\(r​μ2\)\\displaystyle\\geq\\frac\{\(r\\mu\)^\{2\}\}\{r\\mu^\{2\}\+\\\|T\\\|\_\{F\}^\{2\}\}=\\frac\{r\}\{1\+\\\|T\\\|\_\{F\}^\{2\}/\(r\\mu^\{2\}\)\}≥r1\+\(ρ\+τ1−τ\)2\.\\displaystyle\\geq\\frac\{r\}\{1\+\\left\(\\dfrac\{\\rho\+\\tau\}\{1\-\\tau\}\\right\)^\{2\}\}\.\(41\)The final inequality follows fromμ≥a⁡\(1−τ\)\>0\\mu\\geq a\(1\-\\tau\)\>0and‖T‖F2≤r​a2​\(ρ\+τ\)2\\\|T\\\|\_\{F\}^\{2\}\\leq ra^\{2\}\(\\rho\+\\tau\)^\{2\}\.

#### Step 6: Transfer the bounds to the weight update\.

Apply Lemma[A\.1](https://arxiv.org/html/2610.02283#A1.Thmtheorem1)toBBandAA\. SinceBBhas full column rank under the strict positivity condition, the condition\-number, stable\-rank, and entropy\-effective\-rank bounds hold unchanged forB​ABA, usingκ\+​\(B​A\)\\kappa\_\{\+\}\(BA\)for its nonzero condition number\. ∎

## Appendix BApproximate Computation and Scope of the Guarantees

### B\.1Interpreting historical response

RRS usesMt=\(Xt\)⊤​XtM^\{t\}=\(X\_\{t\}\)^\{\\top\}X\_\{t\}andHt−1=∑i<tMi\+ϵ​IH\_\{t\-1\}=\\sum\_\{i<t\}M^\{i\}\+\\epsilon I, withϵ\>0\\epsilon\>0\. These unnormalized statistics weight observations according to their sample and token counts\. Replacing them by per\-task averages changes the relative weighting and is a different design choice\. Because the backbone evolves across tasks, stored historical statistics summarize features observed at their respective collection stages\. They serve as a surrogate for historical response, rather than an exact recomputation of all past features under the current backbone\. Positive definiteness ofHt−1H\_\{t\-1\}ensures that the spectral selection problem remains well defined; it does not imply that this surrogate exactly measures historical loss or forgetting\. The method retains aggregate second\-moment matrices, even though it does not replay past examples or individual historical features\.

### B\.2A computable characterization of an approximate RRS subspace

LetM~=L−1​M​L−⁣⊤⪰0\\widetilde\{M\}=L^\{\-1\}ML^\{\-\\top\}\\succeq 0and letU^∈ℝm×r\\widehat\{U\}\\in\\mathbb\{R\}^\{m\\times r\}have orthonormal columns, as returned by an approximate eigensolver\. Define

𝒮^=range⁡\(L−⁣⊤​U^\),Tr=U^⊤​M~​U^,θr=λmin​\(Tr\)\.\\widehat\{\\mathcal\{S\}\}=\\operatorname\{range\}\(L^\{\-\\top\}\\widehat\{U\}\),\\qquad T\_\{r\}=\\widehat\{U\}^\{\\top\}\\widetilde\{M\}\\widehat\{U\},\\qquad\\theta\_\{r\}=\\lambda\_\{\\min\}\(T\_\{r\}\)\.\(42\)Forv=L−⁣⊤​U^​cv=L^\{\-\\top\}\\widehat\{U\}cwithc≠0c\\neq 0, the same change of coordinates as in the exact proof yields

v⊤​M​vv⊤​H​v=c⊤​Tr​cc⊤​c\.\\frac\{v^\{\\top\}Mv\}\{v^\{\\top\}Hv\}=\\frac\{c^\{\\top\}T\_\{r\}c\}\{c^\{\\top\}c\}\.\(43\)Consequently, the worst\-direction relative response of the computed subspace is exactlyθr\\theta\_\{r\}\. Euclidean QR ofL−⁣⊤​U^L^\{\-\\top\}\\widehat\{U\}preserves this value by preserving the subspace\. Ifλr\\lambda\_\{r\}is therr\-th largest eigenvalue ofM~\\widetilde\{M\}, max–min optimality gives0≤θr≤λr0\\leq\\theta\_\{r\}\\leq\\lambda\_\{r\}\. This characterizes the approximate subspace without assuming exact RSVD optimality\.

A sufficient approximation certificate is available fromM^=U^​Tr​U^⊤\\widehat\{M\}=\\widehat\{U\}T\_\{r\}\\widehat\{U\}^\{\\top\}\. If‖M~−M^‖2≤ζ\\\|\\widetilde\{M\}\-\\widehat\{M\}\\\|\_\{2\}\\leq\\zeta, then

max⁡\{0,λr−ζ\}≤θr≤λr\.\\max\\\{0,\\lambda\_\{r\}\-\\zeta\\\}\\leq\\theta\_\{r\}\\leq\\lambda\_\{r\}\.\(44\)Indeed,M^\\widehat\{M\}is positive semidefinite, itsrr\-th largest eigenvalue isθr\\theta\_\{r\}\(including zeros\), and Weyl’s eigenvalue inequality bounds its difference fromλr\\lambda\_\{r\}byζ\\zeta\. This certificate is sufficient and may be conservative; no numerical value ofζ\\zetais assumed here\. A small invariant\-subspace residual alone does not establish that the computed subspace is the leading one\.

### B\.3Interpretation and scope of the cumulative guarantees

#### Why a cross\-step condition is necessary\.

Individually orthogonal steps can cancel along selected directions\. For example, taken=r=2n=r=2,K=2K=2,η1=η2=1\\eta\_\{1\}=\\eta\_\{2\}=1, and

D1=\(1001\),D2=\(100−1\)\.D\_\{1\}=\\begin\{pmatrix\}1&0\\\\ 0&1\\end\{pmatrix\},\\qquad D\_\{2\}=\\begin\{pmatrix\}1&0\\\\ 0&\-1\\end\{pmatrix\}\.\(45\)Both satisfyDk⊤​Dk=I2D\_\{k\}^\{\\top\}D\_\{k\}=I\_\{2\}, but

B2=−\(D1\+D2\)=\(−2000\)B\_\{2\}=\-\(D\_\{1\}\+D\_\{2\}\)=\\begin\{pmatrix\}\-2&0\\\\ 0&0\\end\{pmatrix\}\(46\)is rank deficient\. Herea2=2a\_\{2\}=2andE2=diag⁡\(2,−2\)E\_\{2\}=\\operatorname\{diag\}\(2,\-2\), so‖E2‖2/a2=1\\\|E\_\{2\}\\\|\_\{2\}/a\_\{2\}=1\. The strict anisotropy assumption correctly excludes this example\. The theorem therefore provides a conditional spectral guarantee, not an unconditional consequence of stepwise orthogonalization\.

#### Exact and approximate orthogonalization\.

IfτK=0\\tau\_\{K\}=0, thenδK=0\\delta\_\{K\}=0and the accumulated error matrixFFvanishes\. The corollary then recovers the theorem’s bounds in terms ofρ\\rho; retaining the exact Frobenius norm ofEKE\_\{K\}also recovers its sharper effective\-rank bound\. A finite polynomial orthogonalization does not by itself guaranteeρ\+τK<1\\rho\+\\tau\_\{K\}<1\. Similarly, a rank\-deficient step necessarily has‖Dk⊤​Dk−Ir‖2≥1\\\|D\_\{k\}^\{\\top\}D\_\{k\}\-I\_\{r\}\\\|\_\{2\}\\geq 1\. The strict cumulative condition must be checked or justified for the actual sequence of updates\.

#### Within\-task versus across\-task accumulation\.

The analysis fixes a single input basisAAand studiesBKA=−∑k=1KηkDkAB\_\{K\}A=\-\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}D\_\{k\}Awithin that task\. It does not directly apply to the across\-task sum∑t=1TBt​At\\sum\_\{t=1\}^\{T\}B\_\{t\}A\_\{t\}when the basesAtA\_\{t\}vary\. The results characterize update geometry; they do not independently establish a bound on task loss, classification error, or forgetting\.

### B\.4Checking the cumulative conditions without pairwise step storage

The cumulative conditions can be evaluated from the actual factor steps\. SetDk=−ΔBk/ηkD\_\{k\}=\-\\Delta B\_\{k\}/\\eta\_\{k\}for an additive updateΔ​Bk=Bk−Bk−1\\Delta B\_\{k\}=B\_\{k\}\-B\_\{k\-1\}withB0=0B\_\{0\}=0andηk\>0\\eta\_\{k\}\>0\. All step scaling must be included consistently; if the implementation contains additional terms, they cannot silently be omitted\. Accumulate

SK=∑k=1Kηk2​Dk⊤​Dk,sK=∑k=1Kηk2,δK=∑k=1Kηk2​‖Dk⊤​Dk−Ir‖2\.S\_\{K\}=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}D\_\{k\}^\{\\top\}D\_\{k\},\\quad s\_\{K\}=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\},\\quad\\delta\_\{K\}=\\sum\_\{k=1\}^\{K\}\\eta\_\{k\}^\{2\}\\\|D\_\{k\}^\{\\top\}D\_\{k\}\-I\_\{r\}\\\|\_\{2\}\.\(47\)The identity

CK=BK⊤​BK−SKC\_\{K\}=B\_\{K\}^\{\\top\}B\_\{K\}\-S\_\{K\}\(48\)recovers exactly the cross\-step matrix in the theorem without retaining all pairs of steps\. ComputeaK=sK\+tr⁡\(CK\)/ra\_\{K\}=s\_\{K\}\+\\operatorname\{tr\}\(C\_\{K\}\)/randEK=CK−tr⁡\(CK\)​Ir/rE\_\{K\}=C\_\{K\}\-\\operatorname\{tr\}\(C\_\{K\}\)I\_\{r\}/r\. ForaK\>0a\_\{K\}\>0, define

ρ^K=‖EK‖2aK,τ^K=δKaK\.\\widehat\{\\rho\}\_\{K\}=\\frac\{\\\|E\_\{K\}\\\|\_\{2\}\}\{a\_\{K\}\},\\qquad\\widehat\{\\tau\}\_\{K\}=\\frac\{\\delta\_\{K\}\}\{a\_\{K\}\}\.\(49\)The strict certificate holds wheneverρ^K\+τ^K<1\\widehat\{\\rho\}\_\{K\}\+\\widehat\{\\tau\}\_\{K\}<1; the corollary can then be evaluated using these quantities\. Cases withaK≤0a\_\{K\}\\leq 0or a failed strict inequality must be counted as uncertified rather than discarded\. Failure of this sufficient certificate does not imply poor accuracy or rank deficiency\. These expressions specify a diagnostic procedure; the accuracy comparisons below do not themselves establish that the certificate holds along training trajectories\.

## Appendix CMechanism Comparisons on 20S\-ImageNet\-A

### C\.1Definitions of the compared variants

The comparison separates choices of input subspace from the location at which spectral control is applied\. A fixed basis is held constant within each task and reconstructed for the next task\.

#### TrainableA,BA,B\+ Muon\.

Both LoRA factors are optimized using Muon\. The induced weight step isΔ​B​A\+B​Δ​A\+Δ​B​Δ​A\\Delta BA\+B\\Delta A\+\\Delta B\\Delta A, so controlling the spectra of the two factor steps separately does not ensure spectral control of their combined weight step\. This variant also has more trainable adapter parameters:r⁡\(m\+n\)r\(m\+n\)rather thann​rnrper adapted matrix\.

#### Fixed random orthonormalAA\+ Muon\.

A random row\-orthonormal basis satisfiesA​A⊤=IrAA^\{\\top\}=I\_\{r\}and onlyBBis trained\. This retains exact factor\-to\-weight spectral transfer without data\-dependent subspace selection\.

#### Fixed current\-task PCAAA\+ Muon\.

LetUrU\_\{r\}contain the leadingrreigenvectors ofMtM^\{t\}and setA=Ur⊤A=U\_\{r\}^\{\\top\}\. This selects the strongest current\-task second\-moment directions without historical information\. Here “PCA” denotes uncentered second\-moment spectral selection, consistent withMtM^\{t\}; centering the features would define a different variant\.

#### Fixed historical\-isolationAA\+ Muon\.

LetUHU\_\{H\}be an orthonormal basis for the retained historical feature subspace, and letQHQ\_\{H\}span its orthogonal complement\. IfVrV\_\{r\}contains the leadingrreigenvectors ofQH⊤​Mt​QHQ\_\{H\}^\{\\top\}M^\{t\}Q\_\{H\}, setA=\(QH​Vr\)⊤A=\(Q\_\{H\}V\_\{r\}\)^\{\\top\}\. This maximizes current\-task response subject to hard historical isolation and requiresdim\(range⁡\(QH\)\)≥r\\dim\(\\operatorname\{range\}\(Q\_\{H\}\)\)\\geq r\. The retained historical dimension determines the feasible space\.

#### EBLoRA\.

EBLoRA constrains the adapter’s nonzero singular values through orthonormal factors and a shared scalar magnitude\. The distinction is the object of spectral control: an adapter\-spectrum constraint versus MuLoRA’s control of individual momentum steps\.

#### MuLoRA \(RRS \+ SSB\)\.

RRS selects directions by current\-task response relative to historical response, followed by Euclidean orthonormalization\. SSB updates onlyBBwithin this fixed basis\. It does not require the accumulated adapter to have equal singular values, and thus retains spectral freedom across optimization steps\.

### C\.2Results and interpretation

Table 4:Comparison of subspace construction and spectral control on ImageNet\-A with 20 incremental sessions\.ALastA\_\{\\mathrm\{Last\}\}andAAvgA\_\{\\mathrm\{Avg\}\}are reported in %\.MethodALast↑A\_\{\\mathrm\{Last\}\}\\uparrowAAvg↑A\_\{\\mathrm\{Avg\}\}\\uparrowTrainableA,BA,B\+ Muon51\.34±0\.3551\.34\\pm 0\.3562\.46±0\.3362\.46\\pm 0\.33Fixed random orthonormalAA\+ Muon54\.32±0\.6454\.32\\pm 0\.6464\.53±0\.5164\.53\\pm 0\.51Fixed current\-task PCAAA\+ Muon52\.45±0\.3452\.45\\pm 0\.3463\.56±0\.6763\.56\\pm 0\.67Fixed historical\-isolationAA\+ Muon54\.67±0\.2554\.67\\pm 0\.2564\.76±0\.4764\.76\\pm 0\.47EBLoRA52\.52±1\.0152\.52\\pm 1\.0162\.16±0\.8262\.16\\pm 0\.82MuLoRA \(RRS \+ SSB\)55\.86±0\.61\\mathbf\{55\.86\}\\pm 0\.6165\.61±0\.64\\mathbf\{65\.61\}\\pm 0\.64Table[4](https://arxiv.org/html/2610.02283#A3.T4)reports the mechanism comparison\. MuLoRA attains the highest mean final and average accuracy\. Relative to the fixed random orthonormal basis, RRS improvesALastA\_\{\\mathrm\{Last\}\}andAAvgA\_\{\\mathrm\{Avg\}\}by 1\.54 and 1\.08 percentage points, respectively\. Its gains over current\-task PCA are 3\.41 and 2\.05 points, suggesting that strong current\-task feature response alone is insufficient to select an effective continual adaptation space in this setting\. Relative to historical isolation, the gains are 1\.19 and 0\.85 points, supporting graded relative\-response selection over strictly excluding retained historical directions\.

MuLoRA exceeds jointly trainable factors with Muon by 4\.52 and 3\.15 points and EBLoRA by 3\.34 and 3\.45 points\. These comparisons support the effectiveness of the complete design\. They do not establish that fixed orthonormal parameterization is necessary, or isolate a causal effect of spectral balancing: trainable\-factor counts, optimization dynamics, and subspace choice can differ across variants\. Together with the main\-text RRS/SSB component ablation, the results support combining history\-aware subspace selection and stepwise spectral balancing\. The reported mean differences are not claims of statistical significance; paired runs are needed to quantify their uncertainty\. Overall and average accuracy alone do not identify whether a gain arises from improved new\-task learning or reduced forgetting\.

## Appendix DReproducibility Details and Limitations

### D\.1Reported experimental configuration

The main experimental configuration uses a ViT\-B/16 backbone pretrained on ImageNet\-21K, adapter rankr=32r=32, a cosine classifier, and a single NVIDIA RTX 4090 GPU\. Each task is trained for 20 epochs with batch size 48 and a cosine learning\-rate schedule\. The reported initial learning rate is10−410^\{\-4\}on ImageNet\-R and2×10−42\\times 10^\{\-4\}on the other benchmarks, including ImageNet\-A\. The main comparison reports averages over three random seeds\.

### D\.2Memory and computational accounting

For an adapted matrix of sizen×mn\\times m, the trainable factorB∈ℝn×rB\\in\\mathbb\{R\}^\{n\\times r\}hasn​rnrparameters, while the fixed basis requiresm​rmrstored entries during a task\. A dense historical second\-moment matrix requiresO⁡\(m2\)O\(m^\{2\}\)persistent storage per distinct tracked input representation\. Thus absence of replay does not imply absence of historical state\. Statistics may be shared only where the corresponding input features are identical\.

ForNtN\_\{t\}collected feature vectors, directly formingMtM^\{t\}costsO⁡\(Nt​m2\)O\(N\_\{t\}m^\{2\}\)operations and can be accumulated in batches without retaining all feature vectors\. A dense Cholesky factorization costsO⁡\(m3\)O\(m^\{3\}\); triangular solves, approximate eigenspace extraction, and QR contribute additional task\-initialization costs\. ForB∈ℝn×rB\\in\\mathbb\{R\}^\{n\\times r\}, each polynomial orthogonalization iteration based onY⊤​YY^\{\\top\}YcostsO⁡\(n​r2\+r3\)O\(nr^\{2\}\+r^\{3\}\), withr≤nr\\leq n\. These operation counts do not establish a wall\-clock advantage\. End\-to\-end efficiency depends on feature collection and subspace construction as well as optimizer steps\.

### D\.3Scope of empirical and theoretical conclusions

The spectral\-transfer identity is exact for a row\-orthonormal basis, whereas the implemented polynomial orthogonalization and eigenspace selection are approximate\. Cumulative guarantees depend on measured or otherwise justified cross\-step conditions, and apply within a task with a fixed basis\. They are not bounds on cross\-task forgetting, classification error, or generalization\.

The mechanism comparison covers one 20\-session benchmark setting\. Broader main\-text accuracy results do not replace matched mechanism comparisons across additional settings\. A norm\-matched momentum control would help distinguish update\-scale effects from spectral reshaping\. Separate old\-class and new\-class measurements, with the prediction label space specified, would further distinguish retention from adaptation\. Spectral plots should identify the update matrix, layer aggregation, normalization, and numerical rank threshold;κ\+\\kappa\_\{\+\}should not be interpreted without rank\-sensitive statistics\.