Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation?
Summary
This paper investigates singular components in LoRA and proposes SCLoRA, a method that uses spectral clipping to improve task adaptation while reducing catastrophic forgetting of pre-trained knowledge.
View Cached Full Text
Cached at: 08/14/26, 09:24 AM
# Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? Source: [https://arxiv.org/abs/2608.12332](https://arxiv.org/abs/2608.12332) [View PDF](https://arxiv.org/pdf/2608.12332) > Abstract:In recent years, low\-rank adaptation \(LoRA\) has emerged as a significant paradigm that freezes pre\-trained weights and introduces small, learnable adapters instead of fine\-tuning the full set of parameters\. In this work, we uncover several key insights regarding the singular components of network parameters based on Singular Value Decomposition \(SVD\)\. Firstly, the principal singular components with large singular values in pre\-trained network parameters can be effectively reused during fine\-tuning, whereas the minor components with smaller singular values are more task\-specific and require substantial adaptation\. Secondly, we first establish the theoretical connection that the uncontrolled growth of singular values in LoRA adapters leads to the forgetting of pre\-trained knowledge \-\- a well\-known issue referred to as catastrophic forgetting\. Building on these observations, we propose SCLoRA, which injects parameterized singular components with spectral clipping into the pre\-trained model in a way that is aware of the spectral distribution of the pre\-trained model\. SCLoRA effectively adapts to new tasks by focusing updates on components that require adaptation, while simultaneously alleviating catastrophic forgetting\. We conduct extensive experiments and demonstrate that SCLoRA not only improves downstream performance but also effectively retains pre\-trained knowledge\. ## Submission history From: Hyowon Wi \[[view email](https://arxiv.org/show-email/e9e722fd/2608.12332)\] **\[v1\]**Tue, 2 Jun 2026 12:49:42 UTC \(941 KB\)
Similar Articles
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
This paper proposes sparsity-induced adaptations to LoRA, including Cheap LoRA (cLA) and a chained circulant variant (c³LA), and provides theoretical generalization bounds along with empirical evaluations showing up to 10% training time reduction and 15% peak GPU memory savings while maintaining competitive performance.
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
SOS-LoRA extends LoRA by decomposing the rank budget into static orthogonal low-rank experts with fixed multi-scale scaling, improving fine-tuning on reasoning, NLU, and math benchmarks without adding inference cost.
Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
Hybrid-LoRA proposes a framework that selectively applies full fine-tuning to a small subset of modules while using LoRA for the rest, achieving performance near full fine-tuning with significantly lower computational cost. Experiments show improvements of up to 5.65% over existing parameter-efficient baselines.
CERSA: Cumulative Energy-Retaining Subspace Adaptation for Memory-Efficient Fine-Tuning
The paper introduces CERSA, a novel parameter-efficient fine-tuning method that uses singular value decomposition to retain principal components, significantly reducing memory usage while outperforming existing methods like LoRA.
BaLoRA: Bayesian Low-Rank Adaptation of Large Scale Models
BaLoRA introduces a Bayesian extension to Low-Rank Adaptation (LoRA) that provides calibrated uncertainty estimates and improves prediction accuracy by narrowing the gap with full fine-tuning.