Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization
Summary
Proposes FoLoRA, a forgetting-aware optimization framework for fine-tuning foundation models that balances task utility and forgetting penalty via generalized Rayleigh-quotient optimization, achieving better preservation of non-target capabilities.
View Cached Full Text
Cached at: 06/02/26, 03:39 PM
# Foundation-Preserving Adaptation via Generalized Rayleigh-Quotient Optimization Source: [https://arxiv.org/abs/2606.00132](https://arxiv.org/abs/2606.00132) [View PDF](https://arxiv.org/pdf/2606.00132) > Abstract:While finetuning effectively adapts foundation models to specialized downstream tasks, it can degrade nontarget capabilities acquired during pretraining\. Existing forgetting aware methods typically seek safer updates through specialized initialization or fixed constraints, but do not regulate the adaptation preservation trade\-off during training\. We propose Foundation Preserving LoRA \(FoLoRA\), a forgetting aware optimization framework\. Guided by a first order preservation condition, FoLoRA defines a forgetting penalty over pretraining\-proxy activations and a task utility over downstream task activations\. It then scores update directions by task utility per unit forgetting penalty via a generalized Rayleigh quotient\. The resulting spectral coordinate system enables direction wise gated Adam updates, attenuating low utility to penalty directions during training\. To estimate the forgetting penalty, FoLoRA constructs pretraining proxy calibration data by sampling from the pretrained model rather than relying on a single proxy dataset\. Experiments on math, code, and instruction following adaptation show that FoLoRA achieves the strongest preservation adaptation balance over baselines, improving target task performance with best aggregate preservation of non target capabilities\. ## Submission history From: Dongjun Kim \[[view email](https://arxiv.org/show-email/3a8d65cc/2606.00132)\] **\[v1\]**Thu, 28 May 2026 21:22:31 UTC \(1,246 KB\)
Similar Articles
Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation
This paper proposes a Mixture of LoRA and Full (MoLF) fine-tuning framework that uses gradient-guided optimizer routing to adaptively switch between LoRA and full fine-tuning. It aims to overcome the structural limitations of relying solely on static adaptation methods by combining the plasticity of full tuning with the regularization of LoRA.
FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
FoRA introduces a parameter-efficient fine-tuning method that selects task-informative layers via Fisher scores and trains LoRA down-projections on the Stiefel manifold, reducing parameters while preserving accuracy.
Hybrid-LoRA: Bridging Full Fine-Tuning and Low-Rank Adaptation for Post-Training
Hybrid-LoRA proposes a framework that selectively applies full fine-tuning to a small subset of modules while using LoRA for the rest, achieving performance near full fine-tuning with significantly lower computational cost. Experiments show improvements of up to 5.65% over existing parameter-efficient baselines.
RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
RAFT is a two-stage framework for domain-specific fine-tuning of LLMs that addresses catastrophic forgetting by refining supervision data and using on-policy distillation with adaptive loss balancing, achieving significant improvements on domain accuracy while recovering general capabilities.
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
This paper investigates the quantitative limits of parametric memory in LLMs using LoRA as a probe, establishing a power law relationship and introducing a threshold-guided optimization method called MemFT for improved memory performance.