PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Hugging Face Daily Papers Papers
continual-learning lora catastrophic-forgetting parameter-efficient-tuning null-space-projection svd image-classification

Summary

PaLoRA introduces a paced low-rank adaptation method for LoRA-based continual learning, deriving an optimal adaptive pacing law that scales gradient step magnitude with the effective rank of accumulated knowledge. The method combines adaptive SVD truncation, null-space gradient projection, and rank-aware pacing, achieving consistent gains including 4% accuracy improvements on 50-task ImageNet-A and ImageNet-R benchmarks.

LoRA-based continual learning methods mitigate catastrophic forgetting through various mechanisms, yet nearly all complement these with small learning rates as a heuristic to restrict gradient scaling magnitude. Such fixed heuristics lack theoretical guidance on how the strength of this restriction should evolve as tasks accumulate. We reveal that even under directional constraints such as nullspace projection, finite-precision updates inevitably leak into the subspace of accumulated prior knowledge along multiple directions. While small learning rates attenuate such leakage, they cannot prevent the accumulated forgetting from intensifying as the effective rank of historical knowledge grows. We show that the optimal magnitude restriction should adaptively increase with this effective rank to balance stability and plasticity, i.e., preservation of previous knowledge and acquisition of new task information. Under an anisotropic leakage model, we derive a pacing law s^*=R/c that characterizes the optimal scaling of gradient steps, i.e., the magnitude restriction itself, where R is the effective rank of past updates. Based on this insight, we propose PaLoRA, which compresses historical knowledge via adaptive SVD truncation, projects gradients onto the nullspace of prior tasks, and applies rank-aware adaptive pacing. Experiments demonstrate consistent improvements over prior methods, with particularly strong performance in long-horizon settings, achieving substantial gains of 4% accuracy on challenging 50-task ImageNet-A and ImageNet-R benchmarks.
Original Article
View Cached Full Text

Cached at: 10/06/26, 04:53 AM

Paper page - PaLoRA: Paced Low-Rank Adaptation for Continual Learning

Source: https://huggingface.co/papers/2610.04226

Abstract

LoRA-basedcontinuallearningmethodsmitigatecatastrophicforgettingthroughvariousmechanisms,yetnearlyallcomplementthesewithsmalllearningratesasaheuristictorestrictgradientscalingmagnitude.Suchfixedheuristicslacktheoreticalguidanceonhowthestrengthofthisrestrictionshouldevolveastasksaccumulate.Werevealthatevenunderdirectionalconstraintssuchasnullspaceprojection,finite-precisionupdatesinevitablyleakintothesubspaceofaccumulatedpriorknowledgealongmultipledirections.Whilesmalllearningratesattenuatesuchleakage,theycannotpreventtheaccumulatedforgettingfromintensifyingastheeffectiverankofhistoricalknowledgegrows.Weshowthattheoptimalmagnituderestrictionshouldadaptivelyincreasewiththiseffectiveranktobalancestabilityandplasticity,i.e.,preservationofpreviousknowledgeandacquisitionofnewtaskinformation.Underananisotropicleakagemodel,wederiveapacinglaws^*=R/cthatcharacterizestheoptimalscalingofgradientsteps,i.e.,themagnituderestrictionitself,whereRistheeffectiverankofpastupdates.Basedonthisinsight,weproposePaLoRA,whichcompresseshistoricalknowledgeviaadaptiveSVDtruncation,projectsgradientsontothenullspaceofpriortasks,andappliesrank-awareadaptivepacing.Experimentsdemonstrateconsistentimprovementsoverpriormethods,withparticularlystrongperformanceinlong-horizonsettings,achievingsubstantialgainsof4%accuracyonchallenging50-taskImageNet-AandImageNet-Rbenchmarks.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2610\.04226

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.04226 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.04226 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.04226 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

arXiv cs.LG

MuLoRA introduces a spectrally balanced low-rank adaptation method for continual learning that identifies 'spectral plasticity collapse' in LoRA updates and addresses it via historical whitening and momentum-based orthogonalization, achieving the best mean accuracy in 15 of 16 reported metrics across class-incremental benchmarks.

Normalized Low-Rank Adaptation

Hugging Face Daily Papers

Normalized Low-Rank Adaptation (NoRA) stabilizes LoRA training by normalizing down-projection matrices, accelerating convergence and improving performance without extra parameters or inference cost.