Are Parameter-Efficient Fine-tuning Methods Really Different?

# Are Parameter-Efficient Fine-tuning Methods Really Different?
Source: [https://arxiv.org/html/2610.09122](https://arxiv.org/html/2610.09122)
Yikuan Li Pinyan Lu Fanghui Liu††thanks:Department of Statistics, The University of Warwick, UK\. Email:[Yikuan\.Li@warwick\.ac\.uk](mailto:[email protected])††thanks:Key Laboratory of Interdisciplinary Research of Computation and Economics, Shanghai University of Finance and Economics, China\. Email:[lu\.pinyan@mail\.shufe\.edu\.cn](mailto:[email protected])††thanks:School of Mathematical Sciences, Institute of Natural Sciences and MOE\-LSC, Shanghai Jiao Tong University, China\. Email:[fanghui\.liu@sjtu\.edu\.cn](mailto:[email protected])\(Corresponding author\)

###### Abstract

Parameter\-efficient fine\-tuning \(PEFT\) offers many parameterizations, yet their methodological and functional differences remain unclear\. We compare six methods in language and diffusion models to examine how their parameterizations relate to task performance, forgetting, and changes in pretrained weight geometry\. Motivated by the spectrum\-preserving design of orthogonal fine\-tuning \(OFT\), we first ask whether spectral preservation is itself important for adaptation and retention\. We find that the selected LoRA\-family methods also approximately preserve pretrained geometry, and that restoring their slightly drifted singular\-value spectra largely preserves task performance, questioning the necessity of explicit geometric preservation\. Beyond this, we observe that some methods exhibit distinct adaptation–retention trade\-offs that vary across settings: LoRA most consistently limits forgetting at competitive performance, DoRA achieves higher mean task scores than LoRA in most comparisons, while PiSSA often incurs greater retention costs\. Further intervention experiments suggest that while performance gains from different PEFT methods can be attributed to modifications in different groups of spectral components, we consistently find that restoring dominant rather than intermediate or trailing components produces the largest mean reduction in general\-text NLL or base\-image drift\. Together, these results motivate evaluating geometric constraints through their functional consequences rather than preservation alone\. Code is available at[https://github\.com/Kuaaannn/PEFT\_methods](https://github.com/Kuaaannn/PEFT_methods)\.

## 1Introduction

Parameter\-efficient fine\-tuning \(PEFT\)\([Houlsby et al\., 2019](https://arxiv.org/html/2610.09122#bib.bib46);[Li and Liang, 2021](https://arxiv.org/html/2610.09122#bib.bib47);[Hu et al\., 2022](https://arxiv.org/html/2610.09122#bib.bib2)\)has become a standard approach for adapting large pretrained models\. Over the past several years, a large number of PEFT algorithms have been proposed, often by modifying a small number of recurring parameterizations\.

One major paradigm is the LoRA family, which adapts pretrained weights through*additive*low\-rank updates\([Hu et al\., 2022](https://arxiv.org/html/2610.09122#bib.bib2)\)\. Subsequent variants modify the optimization, scaling, initialization, or magnitude–direction decomposition of these updates\([Hayou et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib6);[Kalajdzievski, 2023](https://arxiv.org/html/2610.09122#bib.bib7);[Wang et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib8);[Meng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib23);[Liu et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib4)\)\. A second paradigm is the OFT family, which adapts weights*multiplicatively*through structured orthogonal transformations and their variants\([Qiu et al\., 2023](https://arxiv.org/html/2610.09122#bib.bib3);[Qiu et al\., 2025b](https://arxiv.org/html/2610.09122#bib.bib5);[Qiu et al\., 2025a](https://arxiv.org/html/2610.09122#bib.bib22);[Yuan et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib25)\)\. These methods are motivated by seemingly different principles, and many are presented as principled improvements over vanilla LoRA\. Yet it remains unclear which of these differences survive careful tuning and, more importantly, which differences matter functionally\. This raises a basic question:

*When do typical PEFT methods really differ, and which differences matter?*

![Refer to caption](https://arxiv.org/html/2610.09122v1/peft_overview.png)Figure 1:Overview of the questions studied in this work\. \(a\) Six representative PEFT parameterizations\. \(b\) Changes in pretrained singular spectra and the singular\-value restoration intervention\. \(c\) Performance–retention comparisons across PEFT methods\. \(d\) Spectral interventions that selectively retain or restore dominant, intermediate, and trailing components\. Singular\-value decay curves and bar heights are schematic\.Recent work has begun to question whether apparent methodological improvements reflect genuinely different operating behavior\. For the LoRA family,[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.09122#bib.bib27)identify rescaling invariances linking initialization scale, update scaling, and factor learning rates, and show that vanilla LoRA can be highly competitive when properly configured\.[Lee et al\. \(2026\)](https://arxiv.org/html/2610.09122#bib.bib30)likewise find similar peak performance between LoRA and its variants after method\-specific learning\-rate tuning, yet 46 of 64 surveyed studies report a fixed learning rate\. Hugging Face PEFT’s mathematical\-reasoning and image\-generation benchmarks\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.09122#bib.bib48), v0\.21\.0\)also report a single training seed and, for most methods, only one learning\-rate setting\. Comparison settings also differ across studies\. On GSM8K and MATH, PiSSA’s main comparison adapts all linear layers across three 7B backbones for one epoch\([Meng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib23)\), whereas HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib25)\)adapts only LLaMA2\-7B’s query and value projections for two epochs\. These observations motivate common experimental settings, method\-specific tuning and multiple training seeds to separate configuration effects from methodological gains\.

Performance alone, however, may not reveal the full distinction between methods\. Two adapters can achieve similar downstream scores while differing substantially in how much pretrained behavior they disrupt\. This issue is particularly relevant to orthogonal adaptation\. OFT explicitly preserves hyperspherical geometry\([Qiu et al\., 2023](https://arxiv.org/html/2610.09122#bib.bib3)\), while HRA connects orthogonal transformations and low\-rank adaptation through Householder reflections\([Yuan et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib25)\)\. Such geometric constraints are motivated in part by improved generalization and preservation of pretrained knowledge\([Qiu et al\., 2023](https://arxiv.org/html/2610.09122#bib.bib3);[Yuan et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib25);[Huang et al\., 2026](https://arxiv.org/html/2610.09122#bib.bib1)\)\. But two questions remain unresolved: do unconstrained LoRA\-family methods actually depart substantially from these geometries in practice, and does tighter geometric preservation translate into better adaptation or retention?

### 1\.1Contributions and findings

We address these questions by combining controlled empirical comparisons with direct interventions on pretrained weight geometry\. We also review experimental controls in published PEFT comparisons to guide our study design \([Section2](https://arxiv.org/html/2610.09122#S2)\)\. We study six representative methods, including LoRA, DoRA, PiSSA, MiLoRA, OFT, and HRA, across language\-model adaptation and diffusion personalization\. Rather than comparing parameterizations only through final task scores, we examine three levels of behavior: their operating performance after method\-specific tuning, their retention of pretrained capabilities, and the functional consequences of the geometric changes they induce\. As summarized in[Figure1](https://arxiv.org/html/2610.09122#S1.F1), this analysis reveals a recurring mismatch:*parameterization labels can overstate geometric differences, whereas task scores can understate functional differences\.*Our study leads to three main findings\.

Explicit geometric preservation is not what primarily separates PEFT methods\.We first characterize adaptation relative to the two\-sided orthogonal orbit of the pretrained weights\. For small isotropically oriented additive updates, the fraction of weight displacement appearing as singular\-value displacement scales as1/max⁡\(m,n\)1/\\sqrt\{\\max\(m,n\)\}, showing that limited spectral drift can arise even without an explicit orthogonality constraint\. Consistent with this prediction, the LoRA\-family methods we study exhibit only modest spectral drift and very small changes in hyperspherical energy\. More importantly, restoring pretrained singular values while retaining the adapted singular bases largely preserves task performance\. Thus, explicit geometric preservation is neither unique to successful PEFT nor, in our tested settings, necessary for retaining its task gains\.

Similar task performance can conceal more persistent differences in retention\.After method\-specific learning\-rate tuning, the six methods often achieve competitive downstream performance, but their retention behavior differs more systematically\. Across the tested settings, LoRA most consistently limits forgetting while maintaining competitive performance\. DoRA achieves higher mean task scores than LoRA in most comparisons, typically with some additional forgetting, whereas PiSSA often incurs larger retention costs\. Crucially, tighter preservation of pretrained geometry does not reliably predict better retention\. The more persistent distinctions between methods therefore appear in their functional consequences, rather than simply in the geometric invariants imposed by their parameterizations\.

Adaptation is spectrally heterogeneous, whereas forgetting is spectrally structured\.We next intervene on dominant, intermediate, and trailing spectral components, ordered by pretrained singular value\. For task adaptation, there is no universal spectral location of useful change: PiSSA consistently concentrates task\-relevant changes in the dominant components, whereas for the other methods the sufficient components vary with the model, capacity, and task\. For forgetting, however, a markedly more consistent pattern emerges: restoring the dominant components produces the largest mean reduction in general\-text NLL or base\-image drift across all six methods and tested settings\. This ordering persists even in cases where the dominant\-component edit is smaller in norm than alternative edits\. These results suggest that downstream adaptation can be realized through multiple spectral routes, while forgetting is more consistently coupled to changes in dominant pretrained components\.

Taken together, our results suggest that PEFT methods are better distinguished by their operating regimes and the functional consequences of learned weight geometry than by parameterization labels or explicit geometric preservation alone\.

### 1\.2Related work

#### Adaptation, retention and parameterization\.

Within the LoRA family, several methods use singular directions of pretrained weights or gradients to initialize adaptation\([Meng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib23);[Wang et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib8);[Zhang et al\., 2025](https://arxiv.org/html/2610.09122#bib.bib33);[Ponkshe et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib34)\)\. Optimization and scaling studies explain how parameterization affects adaptation\([Hayou et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib6);[Kalajdzievski, 2023](https://arxiv.org/html/2610.09122#bib.bib7);[Wang et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib8)\);[Lee et al\. \(2026\)](https://arxiv.org/html/2610.09122#bib.bib30)link preferred learning rates to loss curvature\.[Biderman et al\. \(2024\)](https://arxiv.org/html/2610.09122#bib.bib9)compare LoRA’s adaptation and forgetting with full fine\-tuning\. PEFT\-Arena compares PEFT methods using a shared SFT learning rate and one run per configuration, leaving method\-specific tuning and seed uncertainty unresolved\([Huang et al\., 2026](https://arxiv.org/html/2610.09122#bib.bib1)\)\. ETHER questions the practical necessity of hyperspherical\-energy preservation\([Bini et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib32)\)\. We examine preservation alongside tuned retention comparisons and spectral interventions\.

#### Spectral structure and functional interventions\.

[Shuttleworth et al\. \(2025\)](https://arxiv.org/html/2610.09122#bib.bib19)identify intruder singular directions after LoRA adaptation\.[He and Cao \(2025\)](https://arxiv.org/html/2610.09122#bib.bib20)study spectrum and vector replacement after full fine\-tuning\.[Jin et al\. \(2026\)](https://arxiv.org/html/2610.09122#bib.bib31)connect forgetting and recovery during full\-parameter SFT/RL to singular\-vector changes\. CASA studies spectral preservation in diffusion\-LoRA transfer\([Wang et al\., 2026](https://arxiv.org/html/2610.09122#bib.bib21)\), while SVFT learns combinations of pretrained singular\-vector pairs\([Lingam et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib18)\)\. Our contribution is to connect tuning\-controlled comparisons with complementary keep/reset interventions, distinguishing recurring retention responses from setting\-dependent contributions to task performance\.

## 2Experimental comparisons in PEFT

[Lee et al\. \(2026\)](https://arxiv.org/html/2610.09122#bib.bib30)survey whether vanilla LoRA was evaluated across multiple hyperparameter settings\. We extend this assessment to uncertainty across training runs, validation\-based learning\-rate selection, disclosure of numerical search spaces and comparable experimental conditions, considering both the proposed method and its principal baseline\.

We also review method papers associated with Hugging Face PEFT v0\.21\.0\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.09122#bib.bib48)\), using sources available by September 2026\. The inventory comprises 71 supported entries, 70 identified papers and 64 papers with eligible comparisons between trained methods\. Appendix[F](https://arxiv.org/html/2610.09122#A6)details the criteria, review procedure and limitations\.

Most papers do not clearly establish key experimental controls \([Figure2](https://arxiv.org/html/2610.09122#S2.F2)\)\. Although shared checkpoints and data are documented in at least one main comparison in 63 and 62 of the 64 papers, respectively, only 25 establish all applicable comparison conditions together, including data exposure, adapted modules and trainable budgets\. Evidence for repeated training and learning\-rate selection is even scarcer: only nine document repeated training of both methods with uncertainty attributable to those runs, and seven establish separate validation\-based selection from at least three learning rates per method\. Most remaining cases are unclear in the reviewed sources\.

Figure 2:Experimental controls documented in PEFT method papers\. \(a\) Four controls in the 64 eligible papers\. \(b\) Components of comparable conditions, excluding inapplicable cases\. Each criterion counts a paper once\. Denominators appear in parentheses\. Bar lengths and labels show paper counts\. A control is documented when one eligible main comparison meets all applicable requirements\. An explicit departure requires evidence that every eligible comparison fails the definition\. Unclear covers missing or conflicting evidence and unresolved comparison coverage\. Component positives may come from different comparisons\. Definitions and coverage summaries appear in Appendix[F](https://arxiv.org/html/2610.09122#A6)\.These gaps matter because experimental conditions shape what a comparison can establish\. For example, gains with fewer trainable parameters can support an efficiency claim even when budgets differ\. Comparisons should therefore specify which conditions are comparable, select settings separately for each method and report uncertainty attributable to training runs\.

For modest adaptation datasets, independent supervised PEFT runs can be much cheaper than pretraining or rollout\-intensive post\-training such as reinforcement learning with verifiable rewards \(RLVR\)\. Method\-specific learning\-rate selection over an exponentially spaced grid and replication across training seeds should therefore be standard practice, with costs that are often manageable\. In RL, higher training costs often limit replication, but DataFlex\-RL illustrates why stronger controls matter: its primary comparison of 13 configurations over 12 matched seeds under a common training recipe finds no statistically reliable gains from the tested data policies over their respective baselines\([Liang et al\., 2026](https://arxiv.org/html/2610.09122#bib.bib49)\)\. These findings strengthen the case for controlled comparisons in PEFT, where replication is more affordable\. Accordingly, our study combines common experimental settings, method\-specific tuning and multiple training seeds to examine task performance and retention\.

## 3Methods and study design

We consider six representative PEFT algorithms: LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.09122#bib.bib2)\), DoRA\([Liu et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib4)\), PiSSA\([Meng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib23)\), MiLoRA\([Wang et al\., 2025](https://arxiv.org/html/2610.09122#bib.bib26)\), OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.09122#bib.bib3)\), and HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib25)\)\.

#### Parameterizations\.

The six methods span low\-rank addition, magnitude–direction decomposition, spectral initialization and orthogonal transformations\. For pretrained𝑾0∈ℝm×n\\bm\{W\}\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\},[Table1](https://arxiv.org/html/2610.09122#S3.T1)summarizes the six methods\. The low\-rank factors are𝑩∈ℝm×r\\bm\{B\}\\in\\mathbb\{R\}^\{m\\times r\}and𝑨∈ℝr×n\\bm\{A\}\\in\\mathbb\{R\}^\{r\\times n\}, with scales=α/rs=\\alpha/r\. PiSSA and MiLoRA freeze𝑾res=𝑾0−s​𝑩0​𝑨0\\bm\{W\}\_\{\\mathrm\{res\}\}=\\bm\{W\}\_\{0\}\-s\\bm\{B\}\_\{0\}\\bm\{A\}\_\{0\}, preserving the starting model\. For DoRA,𝒩\\mathcal\{N\}normalizes each row and𝒎\\bm\{m\}contains learned row magnitudes\. HRA uses𝒖i\\bm\{u\}\_\{i\}with‖𝒖i‖2=1\\\|\\bm\{u\}\_\{i\}\\\|\_\{2\}=1\. OFT uses Cayley blocks𝑹j=\(𝑰\+𝑸j\)​\(𝑰−𝑸j\)−1\\bm\{R\}\_\{j\}=\(\\bm\{I\}\+\\bm\{Q\}\_\{j\}\)\(\\bm\{I\}\-\\bm\{Q\}\_\{j\}\)^\{\-1\}with skew\-symmetric𝑸j\\bm\{Q\}\_\{j\}\. Our experiments use the OFTv2 implementation, which approximates the Cayley inverse with a truncated Neumann series\([Qiu et al\., 2025b](https://arxiv.org/html/2610.09122#bib.bib5)\)\. The resulting transform is only approximately orthogonal\. Appendix[C\.6](https://arxiv.org/html/2610.09122#A3.SS6)quantifies this deviation, and all functional evaluations use the native implementation\.

Table 1:Selected PEFT methods and their parameterizations\.
#### Experiment setup\.

We evaluate mathematics, coding and diffusion personalization, reporting means and sample SDs over three training seeds unless otherwise specified\. Qwen2\.5\-7B\([Qwen Team, 2024](https://arxiv.org/html/2610.09122#bib.bib15)\)and Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib40)\)each train on 20,000 examples per task from MetaMath\([Yu et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib12)\)for mathematics and CodeFeedback\([Zheng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib29)\)for coding\. Methods share data exposure and adapted modules, with approximately matched parameter budgets\. We select LRs by mean GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.09122#bib.bib41)\)development accuracy for mathematics or held\-out response NLL for coding, then report held\-out GSM8K accuracy and greedy HumanEval\+/MBPP\+ pass@1\([Liu et al\., 2023a](https://arxiv.org/html/2610.09122#bib.bib28)\)\. FineWiki\([Penedo, 2025](https://arxiv.org/html/2610.09122#bib.bib17)\)Δ​NLL=NLL∗−NLL0\\Delta\\mathrm\{NLL\}=\\mathrm\{NLL\}\_\{\*\}\-\\mathrm\{NLL\}\_\{0\}measures forgetting, with lower values indicating better retention\. For FLUX\.2\-klein\-base\-4B personalization\([Black Forest Labs, 2026](https://arxiv.org/html/2610.09122#bib.bib13)\), we compare all six methods on the benchmark cat at four approximately matched budgets and extend the LoRA/OFT comparison to 28 objects\([Ruiz et al\., 2023](https://arxiv.org/html/2610.09122#bib.bib10)\)\. Validation DINO\([Oquab et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib11)\)selects LRs, while held\-out test DINO measures fidelity\. Base\-image drift, CLIP alignment\([Radford et al\., 2021](https://arxiv.org/html/2610.09122#bib.bib14)\)and COCO flow loss\([Lin et al\., 2014](https://arxiv.org/html/2610.09122#bib.bib16)\)assess complementary retention outcomes\. Spectral interventions use the same development/validation selection criteria across all six methods on mathematics and diffusion and jointly edit all adapted matrices\. Each Pareto point represents one capacity/LR configuration\.

## 4Geometric preservation and functional consequences

We examine how PEFT methods change pretrained weight geometry and whether these changes are necessary for adaptation\. We first explain singular\-value and singular\-basis changes theoretically, then measure spectral and hyperspherical energy preservation in checkpoints\. Finally, we restore pretrained singular values to test whether adapted singular bases alone retain task gains\.

#### Theoretical perspective\.

For𝑾0,𝑾∗∈ℝm×n\\bm\{W\}\_\{0\},\\bm\{W\}\_\{\*\}\\in\\mathbb\{R\}^\{m\\times n\}, let𝑾j=𝑼j​diag⁡\(𝒔j\)​𝑽j⊤\\bm\{W\}\_\{j\}=\\bm\{U\}\_\{j\}\\operatorname\{diag\}\(\\bm\{s\}\_\{j\}\)\\bm\{V\}\_\{j\}^\{\\\!\\top\}denote thin SVDs with decreasing singular values, and letO⁡\(k\)O\(k\)denote the orthogonal group\. Relative weight and spectral displacement are defined bydW=‖𝑾∗−𝑾0‖F/‖𝑾0‖Fd\_\{W\}=\{\\\|\\bm\{W\}\_\{\*\}\-\\bm\{W\}\_\{0\}\\\|\_\{\\mathrm\{F\}\}\}/\{\\\|\\bm\{W\}\_\{0\}\\\|\_\{\\mathrm\{F\}\}\}anddS=‖𝒔∗−𝒔0‖2/‖𝒔0‖2d\_\{S\}=\{\\\|\\bm\{s\}\_\{\*\}\-\\bm\{s\}\_\{0\}\\\|\_\{2\}\}/\{\\\|\\bm\{s\}\_\{0\}\\\|\_\{2\}\}\.

###### Proposition 4\.1\(Distance to the orthogonal orbit\)\.

The restored matrix𝐖R=𝐔∗​diag⁡\(𝐬0\)​𝐕∗⊤\\bm\{W\}\_\{R\}=\\bm\{U\}\_\{\*\}\\operatorname\{diag\}\(\\bm\{s\}\_\{0\}\)\\bm\{V\}\_\{\*\}^\{\\\!\\top\}satisfies

min𝑸L∈O⁡\(m\),𝑸R∈O⁡\(n\)⁡‖𝑾∗−𝑸L​𝑾0​𝑸R⊤‖F=‖𝑾∗−𝑾R‖F=‖𝒔∗−𝒔0‖2,\\min\_\{\\bm\{Q\}\_\{L\}\\in O\(m\),\\,\\bm\{Q\}\_\{R\}\\in O\(n\)\}\\\|\\bm\{W\}\_\{\*\}\-\\bm\{Q\}\_\{L\}\\bm\{W\}\_\{0\}\\bm\{Q\}\_\{R\}^\{\\\!\\top\}\\\|\_\{\\mathrm\{F\}\}=\\\|\\bm\{W\}\_\{\*\}\-\\bm\{W\}\_\{R\}\\\|\_\{\\mathrm\{F\}\}=\\\|\\bm\{s\}\_\{\*\}\-\\bm\{s\}\_\{0\}\\\|\_\{2\},\(1\)and attains this minimum\([Mirsky, 1975](https://arxiv.org/html/2610.09122#bib.bib36)\)\. For nonzero base and update norms,dS/dWd\_\{S\}/d\_\{W\}is the distance to this orbit relative to the update norm\.

A small ratio means that changing the singular bases while keeping pretrained singular values closely approximates the adapted weights, with error measured relative to the update norm\.

###### Proposition 4\.2\(First\-order spectral and basis motion\)\.

Letq=min⁡\(m,n\)q=\\min\(m,n\)and suppose𝐖0\\bm\{W\}\_\{0\}hasqqdistinct positive singular values\. For𝐖∗​\(ϵ\)=𝐖0\+ϵ​𝐄\\bm\{W\}\_\{\*\}\(\\epsilon\)=\\bm\{W\}\_\{0\}\+\\epsilon\\bm\{E\}, set𝐬˙=d​𝐬​\(ϵ\)/d​ϵ\|ϵ=0\\dot\{\\bm\{s\}\}=\\left\.\\mathrm\{d\}\\bm\{s\}\(\\epsilon\)/\\mathrm\{d\}\\epsilon\\right\|\_\{\\epsilon=0\}\. Then

s˙i=𝒖0,i⊤​𝑬​𝒗0,i,𝑬N=𝑼0​diag⁡\(𝒔˙\)​𝑽0⊤,𝑬T=𝑬−𝑬N\\dot\{s\}\_\{i\}=\\bm\{u\}\_\{0,i\}^\{\\\!\\top\}\\bm\{E\}\\bm\{v\}\_\{0,i\},\\qquad\\bm\{E\}\_\{N\}=\\bm\{U\}\_\{0\}\\operatorname\{diag\}\(\\dot\{\\bm\{s\}\}\)\\bm\{V\}\_\{0\}^\{\\\!\\top\},\\qquad\\bm\{E\}\_\{T\}=\\bm\{E\}\-\\bm\{E\}\_\{N\}\(2\)are the spectral derivative\([Stewart, 1990](https://arxiv.org/html/2610.09122#bib.bib24)\), the normal component and the tangent component to the two\-sided orthogonal orbit, respectively\. In particular,𝐄T\\bm\{E\}\_\{T\}changes only singular bases to first order, and‖𝐄‖F2=‖𝐬˙‖22\+‖𝐄T‖F2\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\\|\\dot\{\\bm\{s\}\}\\\|\_\{2\}^\{2\}\+\\\|\\bm\{E\}\_\{T\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\. If‖𝐄‖F=1\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}=1and𝔼⁡\[vec⁡\(𝐄\)​vec⁡\(𝐄\)⊤\]=𝐈m​n/\(m​n\)\\mathbb\{E\}\[\\operatorname\{vec\}\(\\bm\{E\}\)\\operatorname\{vec\}\(\\bm\{E\}\)^\{\\\!\\top\}\]=\\bm\{I\}\_\{mn\}/\(mn\), then

𝔼​‖𝒔˙‖22=1max⁡\(m,n\),limϵ→0𝔼⁡\[\(dSdW\)2\]=1max⁡\(m,n\)\.\\mathbb\{E\}\\\|\\dot\{\\bm\{s\}\}\\\|\_\{2\}^\{2\}=\\frac\{1\}\{\\max\(m,n\)\},\\qquad\\lim\_\{\\epsilon\\to 0\}\\sqrt\{\\mathbb\{E\}\\left\[\\left\(\\frac\{d\_\{S\}\}\{d\_\{W\}\}\\right\)^\{2\}\\right\]\}=\\frac\{1\}\{\\sqrt\{\\max\(m,n\)\}\}\.\(3\)

Small isotropic updates therefore have RMSdS/dW≈1/max⁡\(m,n\)d\_\{S\}/d\_\{W\}\\approx 1/\\sqrt\{\\max\(m,n\)\}: spectral displacement can be a small share of weight displacement without an orthogonality constraint\. This conditional prediction also holds for uniformly oriented low\-rank updates \(proofs and pooling in Appendix[E](https://arxiv.org/html/2610.09122#A5)\)\.

#### Empirical evaluation\.

At the largest capacities, spectral displacement is a modest fraction of weight displacement even for unconstrained methods \([Figure1](https://arxiv.org/html/2610.09122#S1.F1)b\): about 11% for Llama MiLoRA versus 3–4% for LoRA, DoRA, and PiSSA, and 5\.8% and 8\.7% for FLUX PiSSA and MiLoRA, respectively\. We also measure hyperspherical energy \(HE\), whose preservation motivates OFT’s approach to adaptation\([Qiu et al\., 2023](https://arxiv.org/html/2610.09122#bib.bib3)\)\. For nonzero rows with distinct normalized directions,H⁡\(𝑾\)=∑i≠j‖𝒘^i−𝒘^j‖2−1H\(\\bm\{W\}\)=\\sum\_\{i\\neq j\}\\\|\\widehat\{\\bm\{w\}\}\_\{i\}\-\\widehat\{\\bm\{w\}\}\_\{j\}\\\|\_\{2\}^\{\-1\}, where𝒘^i=𝒘i/‖𝒘i‖2\\widehat\{\\bm\{w\}\}\_\{i\}=\\bm\{w\}\_\{i\}/\\\|\\bm\{w\}\_\{i\}\\\|\_\{2\}\. Right\-orthogonal transformations preserve hyperspherical energy exactly, whereas preserving singular values alone does not\. Nevertheless, all six methods approximately preserve HE: every checkpoint’s mean absolute relative change across adapted matrices is below0\.01%0\.01\\%at the largest capacities \(Appendix[C\.3](https://arxiv.org/html/2610.09122#A3.SS3)\)\. LoRA\-family methods therefore approximately preserve both quantities without enforcing orthogonality\. Their small spectral displacement further supports a two\-sided orthogonal approximation of adapted weights \(Proposition[4\.1](https://arxiv.org/html/2610.09122#S4.Thmproposition1)\)\. To assess the functional importance of singular\-basis changes relative to singular\-value scaling, we restore pretrained singular values while retaining adapted bases\([He and Cao, 2025](https://arxiv.org/html/2610.09122#bib.bib20);[Jin et al\., 2026](https://arxiv.org/html/2610.09122#bib.bib31)\)\. Across LLM and FLUX checkpoints, all points at the largest capacities remain within one percentage point of reconstruction \([Figure1](https://arxiv.org/html/2610.09122#S1.F1)b\)\. See Appendix[C](https://arxiv.org/html/2610.09122#A3)for additional results at other capacities\.

Figure 3:Task performance largely survives singular\-value restoration\. At the largest capacities and development/validation\-selected LRs, pale and saturated bars show trained and restored scores with sample SD error bars\. LLM panels report development accuracy, and FLUX reports same\-run test DINO for all six methods\.
#### Summary\.

In our settings, LoRA\-family methods approximately preserve the spectra and hyperspherical energy that OFT is designed to preserve exactly\. Singular\-value restoration largely retains task performance\. Tighter preservation must be judged by its task and retention outcomes\.

## 5Competitive performance, recurring retention differences

In this section, we study task performance and forgetting across PEFT methods and ask whether closer geometric preservation leads to better adaptation–retention trade\-offs\. We also examine method\-specific patterns across settings\.

### 5\.1Language models: competitive performance, different retention

We first examine whether comparable task performance in mathematics and coding is accompanied by comparable retention of general language modeling\. We consider both development\-selected test results and the mathematics performance–retention frontiers across learning rates and capacities\.

Table 2:Development\-selected LLM test scores \(%\) and FineWiki forgetting \(103​Δ​NLL10^\{3\}\\Delta\\mathrm\{NLL\}\)\. Budgets are approximately matched: S/M/L for GSM8K; one for CodeFeedback\. HE\+ denotes HumanEval\+\. Base is one unadapted evaluation\. Shading marks each column's best adapted mean, including ties\.\(a\) Mathematics: GSM8K test accuracy \(%,↑\\uparrow\)

\(b\) Coding: test pass@1 and general\-text forgetting

LoRA most consistently limits forgetting among all six methods\.Its mean NLL increase is lowest in 4 of 6 mathematics settings and both coding backbones \([Table2](https://arxiv.org/html/2610.09122#S5.T2)\), while task performance remains competitive across settings\. See Appendix[B\.6](https://arxiv.org/html/2610.09122#A2.SS6)for robustness checks\. OFT forgets least for Llama at the largest capacity, with LoRA second\. For Qwen at the smallest capacity, LoRA’s high\-LR selection forgets most and MiLoRA least\.

DoRA usually exceeds LoRA’s mean task scores\.It leads in 10/14 comparisons: all four coding comparisons, all four cat budgets, and two mathematics settings \(small Qwen and large Llama;[Table2](https://arxiv.org/html/2610.09122#S5.T2), Appendix[B\.6](https://arxiv.org/html/2610.09122#A2.SS6)\)\. DoRA’s forgetting ranks among the lowest three methods in every selected LLM setting\.

Figure 4:Development performance–retention envelopes across LRs and capacities\. Shapes denote increasing capacity: circle, square, triangle, then diamond \(FLUX only\)\. Faint points are dominated within their method\. Line styles identify method frontiers\. Black borders mark the shared frontier across methods\. Each icon counts one configuration in the full grid \(89 Qwen, 90 Llama, 192 FLUX\)\. Insets magnify boxed regions with uniform point styling\. \(a,b\) Dev accuracy versus FineWikiΔ\\DeltaNLL\. \(c\) Cat validation DINO versus general\-image drift\.PiSSA frequently incurs greater retention costs\.It has the largest mean NLL increase on both coding backbones and half the mathematics settings, yet leads Llama HumanEval\+\. At matched LRs and capacities, PiSSA forgets most in every complete LLM comparison and constituent seed \(further comparisons in Appendix[B\.6](https://arxiv.org/html/2610.09122#A2.SS6)\)\.

PiSSA also illustrates why closer geometric preservation need not improve retention\. At the largest Llama capacity, it has smaller HE and pairwise\-cosine changes than LoRA \(Appendix[C\.3](https://arxiv.org/html/2610.09122#A3.SS3)\) but a greater NLL increase in every seed \(means0\.06460\.0646versus0\.04870\.0487\)\.

On the development frontiers, LoRA and DoRA also appear repeatedly, with DoRA reaching the highest accuracy on both backbones \([Figure4](https://arxiv.org/html/2610.09122#S5.F4)a,b\)\. OFT contributes many Llama frontier points, whereas PiSSA contributes none\.

### 5\.2Diffusion: recurring drift patterns, distinct retention metrics

We next examine whether the retention patterns observed in language extend to diffusion personalization\. We compare all six methods on cat personalization across parameter budgets, then extend the LoRA–OFT comparison across objects using complementary measures of retention\.

Table 3:FLUX\-4B test/retention scores\. DINO and general drift/CLIP are×100\\times 100; COCO loss is unscaled\. Panel \(b\) averages objects per seed\. Shaded bold means mark the best method per budget/metric in \(a\), or capacity pair in \(b\)\. Budgets list rank/block sizes; HRA uses 16/32/62/124 vectors\.\(a\) Cat: six methods at four approximately matched budgets

\(b\) 28 objects: four capacities, three LRs

The retention pattern recurs across cat budgets\.LoRA has the lowest mean general drift at 3 of 4 validation\-selected budgets and the second lowest at the other \([Table3](https://arxiv.org/html/2610.09122#S5.T3)a\)\. DoRA has higher mean test DINO than LoRA at every budget, with additional drift, and leads fidelity at the smallest and largest budgets\. PiSSA exceeds LoRA’s drift in all 12 paired seeds; its medium\-budget fidelity lead again accompanies a retention cost\.

HRA has the highest mean test fidelity and lowest drift at the small budget and extends the pooled frontier, but its advantage does not persist at larger capacities\. LoRA and DoRA also contribute; PiSSA contributes no frontier point \([Figure4](https://arxiv.org/html/2610.09122#S5.F4)c\)\. MiLoRA has lower mean fidelity and higher drift than LoRA at every selected budget\.

Across 28 objects, LoRA and OFT have close test fidelity\. LoRA has lower image drift and higher CLIP alignment at all four selected capacity pairs, while OFT has lower COCO flow loss at three \([Table3](https://arxiv.org/html/2610.09122#S5.T3)b\)\. These metrics distinguish preservation of generated behavior from preservation of denoising predictions at fixed noisy states\.

#### Summary\.

Although task scores are often close, LoRA most consistently limits forgetting among all six methods at competitive performance\. DoRA usually exceeds LoRA’s mean task scores, while PiSSA often forgets more\. Other methods’ performance and forgetting depend on the setting\. The orthogonal methods OFT and HRA offer no consistent performance–retention advantage over the four LoRA\-family methods across the tested settings\.

## 6Spectrally Heterogeneous Adaptation, Structured Forgetting

We examine how dominant, intermediate, and trailing singular components contribute to adaptation and forgetting\. Using pretrained singular values throughout, we selectively restore or retain adapted singular vectors to measure each group’s contribution\.

### 6\.1Restoring groups of spectral components

For a component groupBB, we restore its pretrained singular vectors and retain adapted vectors elsewhere:

𝑾restore⁡\(B\)=𝑾keep⁡\(Bc\)=∑i∈Bs0,i​𝒖0,i​𝒗0,i⊤\+∑i∉Bs0,i​𝒖∗,i​𝒗∗,i⊤\.\\bm\{W\}\_\{\\mathrm\{restore\}\(B\)\}=\\bm\{W\}\_\{\\mathrm\{keep\}\(B^\{\\mathrm\{c\}\}\)\}=\\sum\_\{i\\in B\}s\_\{0,i\}\\bm\{u\}\_\{0,i\}\\bm\{v\}\_\{0,i\}^\{\\\!\\top\}\+\\sum\_\{i\\notin B\}s\_\{0,i\}\\bm\{u\}\_\{\*,i\}\\bm\{v\}\_\{\*,i\}^\{\\\!\\top\}\.\(4\)
Figure 5:Dominant\-component restoration consistently reduces forgetting, with varying effects on task performance\. Cells show mean±\\pmSD changes from trained models at the largest capacities and selected LRs\. Higher task scores and lower NLL/drift are better\. FLUX DINO/drift are scaled by 100\. Bold marks the largest absolute mean change across component groups per method\.#### Restoring dominant components consistently reduces forgetting\.

At the largest capacities, restoring dominant components gives the largest mean reduction in general\-text NLL or general\-image drift for every method \([Figure5](https://arxiv.org/html/2610.09122#S6.F5)\)\. This ordering holds across language capacities \(Appendix[C](https://arxiv.org/html/2610.09122#A3)\)\. MiLoRA shares this pattern despite greater spectral drift concentration in trailing components\.

#### The same restoration has different effects on task performance\.

At the largest language capacities, restoring dominant components largely preserves Qwen accuracy for LoRA, OFT, DoRA and MiLoRA, whereas PiSSA and HRA suffer large losses\. The same edit reduces LoRA and DoRA accuracy by about 27 percentage points on Llama\. PiSSA nearly loses all recorded accuracy on both models\. In FLUX, PiSSA, LoRA and DoRA all incur substantial losses\. OFT improves on average, with substantial seed variation\. HRA and MiLoRA incur smaller mean losses\. Thus, restoring dominant components consistently reduces forgetting, but its effect on task performance shows no consistent pattern across methods and models\.

### 6\.2Which components retain task performance on their own?

Retaining only adapted dominant components preserves PiSSA’s language accuracy within 1 percentage point of the full value\-restored model𝑾R\\bm\{W\}\_\{R\}across every tested setting, whereas retaining only the remaining components does not\. Other methods show no consistent sufficiency pattern across settings \(Appendix[C](https://arxiv.org/html/2610.09122#A3),[Figures13](https://arxiv.org/html/2610.09122#A3.F13)and[19](https://arxiv.org/html/2610.09122#A3.T19)\)\.

#### Summary\.

Task dependence varies across settings: PiSSA’s adapted dominant components suffice across the tested settings, whereas other methods may require intermediate and trailing components\. Restoring the dominant components consistently yields the largest mean reduction in NLL and drift, suggesting that forgetting is systematically associated with changes in dominant components\.

## 7Discussion

Across the tested settings, our comparison of selected PEFT methods finds no consistent performance–retention advantage from explicit geometric preservation\. With method\-specific learning\-rate tuning, LoRA most consistently limits forgetting while maintaining competitive task performance\. Restoring pretrained singular values largely preserves task performance, while restoring dominant components consistently yields the largest mean reduction in forgetting, although its effects on task performance vary\. These findings highlight the importance of evaluating adaptation and retention jointly\. In addition, shared learning rates and single\-seed results can bias comparisons, underscoring the need for method\-specific tuning and evaluation across multiple seeds\. We hope this study supports more reliable evaluations, informs method selection, and guides PEFT designs that better balance adaptation with the retention of pretrained capabilities\.

## Acknowledgments

We thank[Zulip](https://zulip.com/)for the project organization tool and[Sulis](https://warwick.ac.uk/research/rtp/sc/sulis/)for computational resources\.

## References

- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. SuttonProgram synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§B\.3](https://arxiv.org/html/2610.09122#A2.SS3.p3.1)\.
- Bidermanet al\.\(2024\)D\. Biderman, J\. Portes, J\. J\. G\. Ortiz, M\. Paul, P\. Greengard, C\. Jennings, D\. King, S\. Havens, V\. Chiley, J\. Frankle, C\. Blakeney, and J\. P\. CunninghamLoRA learns less and forgets less\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=aloEru2qCG)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1)\.
- Biniet al\.\(2024\)M\. Bini, K\. Roth, Z\. Akata, and A\. KhorevaETHER: efficient finetuning of large\-scale models with hyperplane reflections\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 4007–4026\.External Links:[Link](https://proceedings.mlr.press/v235/bini24a.html)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1)\.
- Black Forest Labs \(2026\)Black Forest LabsFLUX\.2\-klein\-base\-4B model card\.Note:Official model releaseExternal Links:[Link](https://huggingface.co/black-forest-labs/FLUX.2-klein-base-4B)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.Evaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§B\.3](https://arxiv.org/html/2610.09122#A2.SS3.p3.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The Llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuLoRA\+: efficient low rank adaptation of large models\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 17783–17806\.External Links:[Link](https://proceedings.mlr.press/v235/hayou24a.html)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1)\.
- He and Cao \(2025\)X\. He and X\. CaoUnderstanding post\-training structural changes in large language models\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2509.17866),[Link](https://arxiv.org/abs/2509.17866)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.09122#S4.SS0.SSS0.Px2.p1.1)\.
- Higham \(1986\)N\. J\. HighamComputing the polar decomposition—with applications\.SIAM Journal on Scientific and Statistical Computing7\(4\),pp\. 1160–1174\.External Links:ISSN 2168\-3417,[Link](http://dx.doi.org/10.1137/0907079),[Document](https://dx.doi.org/10.1137/0907079)Cited by:[§C\.6](https://arxiv.org/html/2610.09122#A3.SS6.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for NLP\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 2790–2799\.External Links:[Link](https://proceedings.mlr.press/v97/houlsby19a.html)Cited by:[§1](https://arxiv.org/html/2610.09122#S1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2610.09122#S1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1),[§3](https://arxiv.org/html/2610.09122#S3.p1.1)\.
- Huanget al\.\(2026\)Y\. Huang, R\. Peng, Z\. Qiu, J\. Kang, Y\. Wen, B\. Schölkopf, and W\. LiuPEFT\-Arena: understanding parameter\-efficient finetuning from a stability\-plasticity perspective\.arXiv preprint arXiv:2605\.28819\.Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p5.1)\.
- Jinet al\.\(2026\)H\. Jin, S\. Luan, T\. Ni, S\. Lyu, G\. Rabusseau, R\. Rabbany, D\. Precup, and M\. HamdaqaRL fine\-tuning heals OOD forgetting in SFT\.External Links:2509\.12235,[Link](https://arxiv.org/abs/2509.12235)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2610.09122#S4.SS0.SSS0.Px2.p1.1)\.
- Kalajdzievski \(2023\)D\. KalajdzievskiA rank stabilization scaling factor for fine\-tuning with LoRA\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2312.03732),[Link](https://arxiv.org/abs/2312.03732)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1)\.
- Leeet al\.\(2026\)Y\. Lee, C\. Ko, P\. Chen, and M\. YehLearning rate matters: vanilla LoRA may suffice for LLM fine\-tuning\.External Links:2602\.04998,[Link](https://arxiv.org/abs/2602.04998)Cited by:[Appendix F](https://arxiv.org/html/2610.09122#A6.SS0.SSS0.Px4.p1.1),[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p4.1),[§2](https://arxiv.org/html/2610.09122#S2.p1.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 4582–4597\.External Links:[Link](https://aclanthology.org/2021.acl-long.353/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.353)Cited by:[§1](https://arxiv.org/html/2610.09122#S1.p1.1)\.
- Lianget al\.\(2026\)H\. Liang, M\. Chen, H\. Feng, M\. Qiang, and W\. ZhangDataFlex\-RL: an evaluation platform for RLVR data policies\.External Links:2609\.06107,[Link](https://arxiv.org/abs/2609.06107)Cited by:[§2](https://arxiv.org/html/2610.09122#S2.p5.1)\.
- Likhosherstovet al\.\(2021\)V\. Likhosherstov, J\. Davis, K\. Choromanski, and A\. WellerCWY parametrization: a solution for parallelized optimization of orthogonal and Stiefel matrices\.InProceedings of The 24th International Conference on Artificial Intelligence and Statistics,A\. Banerjee and K\. Fukumizu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.130,pp\. 55–63\.External Links:[Link](https://proceedings.mlr.press/v130/likhosherstov21a.html)Cited by:[§B\.1](https://arxiv.org/html/2610.09122#A2.SS1.p1.2)\.
- Linet al\.\(2014\)T\. Lin, M\. Maire, S\. Belongie, J\. Hays, P\. Perona, D\. Ramanan, P\. Dollár, and C\. L\. ZitnickMicrosoft COCO: common objects in context\.InComputer Vision – ECCV 2014,D\. Fleet, T\. Pajdla, B\. Schiele, and T\. Tuytelaars \(Eds\.\),Cham,pp\. 740–755\.External Links:ISBN 978\-3\-319\-10602\-1Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Lingamet al\.\(2024\)V\. Lingam, A\. Tejaswi, A\. Vavre, A\. Shetty, G\. K\. Gudur, J\. Ghosh, A\. Dimakis, E\. Choi, A\. Bojchevski, and S\. SanghaviSVFT: parameter\-efficient fine\-tuning with singular vectors\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 41425–41446\.External Links:[Document](https://dx.doi.org/10.52202/079017-1311),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/48c368f105e8145b945227b73255635a-Paper-Conference.pdf)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px2.p1.1)\.
- Liuet al\.\(2023a\)J\. Liu, C\. S\. Xia, Y\. Wang, and L\. ZHANGIs your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 21558–21572\.External Links:[Document](https://dx.doi.org/10.52202/075280-0943),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/43e9d647ccd3e4b7b5baab53f0368686-Paper-Conference.pdf)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDoRA: weight\-decomposed low\-rank adaptation\.InProceedings of the 41st International Conference on Machine Learning,R\. Salakhutdinov, Z\. Kolter, K\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research, Vol\.235,pp\. 32100–32121\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bn.html)Cited by:[§B\.1](https://arxiv.org/html/2610.09122#A2.SS1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1),[§3](https://arxiv.org/html/2610.09122#S3.p1.1)\.
- Liuet al\.\(2018\)W\. Liu, R\. Lin, Z\. Liu, L\. Liu, Z\. Yu, B\. Dai, and L\. SongLearning towards minimum hyperspherical energy\.InAdvances in Neural Information Processing Systems,S\. Bengio, H\. Wallach, H\. Larochelle, K\. Grauman, N\. Cesa\-Bianchi, and R\. Garnett \(Eds\.\),Vol\.31,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2018/file/177540c7bcb8db31697b601642eac8d4-Paper.pdf)Cited by:[§C\.3](https://arxiv.org/html/2610.09122#A3.SS3.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023b\)X\. Liu, C\. Gong, and Q\. LiuFlow straight and fast: learning to generate and transfer data with rectified flow\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=XVjTT1nw5z)Cited by:[§A\.3](https://arxiv.org/html/2610.09122#A1.SS3.SSS0.Px2.p2.2)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§A\.1](https://arxiv.org/html/2610.09122#A1.SS1.p2.1)\.
- Mangrulkaret al\.\(2022\)S\. Mangrulkar, S\. Gugger, L\. Debut, Y\. Belkada, S\. Paul, B\. Bossan, and M\. TietzPEFT: state\-of\-the\-art parameter\-efficient fine\-tuning methods\.Note:[https://github\.com/huggingface/peft](https://github.com/huggingface/peft)Cited by:[Appendix F](https://arxiv.org/html/2610.09122#A6.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p4.1),[§2](https://arxiv.org/html/2610.09122#S2.p2.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 121038–121072\.External Links:[Document](https://dx.doi.org/10.52202/079017-3846),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/db36f4d603cc9e3a2a5e10b93e6428f2-Paper-Conference.pdf)Cited by:[§B\.1](https://arxiv.org/html/2610.09122#A2.SS1.p1.1),[§B\.3](https://arxiv.org/html/2610.09122#A2.SS3.p1.1),[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1),[§1](https://arxiv.org/html/2610.09122#S1.p4.1),[§3](https://arxiv.org/html/2610.09122#S3.p1.1)\.
- Mirsky \(1975\)L\. MirskyA trace inequality of John von Neumann\.Monatshefte für Mathematik79\(4\),pp\. 303–306\.External Links:ISSN 1436\-5081,[Link](http://dx.doi.org/10.1007/BF01647331),[Document](https://dx.doi.org/10.1007/bf01647331)Cited by:[§E\.1](https://arxiv.org/html/2610.09122#A5.SS1.p2.1.1.1),[Proposition 4\.1](https://arxiv.org/html/2610.09122#S4.Thmproposition1.p1.2.1)\.
- Oquabet al\.\(2024\)M\. Oquab, T\. Darcet, T\. Moutakanni, H\. V\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. HAZIZA, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jegou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. BojanowskiDINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research\.Note:Featured CertificationExternal Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=a68SUt6zFt)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Penedo \(2025\)G\. PenedoFineWiki\.Hugging Face Datasets\.Note:Source: Wikimedia Enterprise Snapshot API \(https://api\.enterprise\.wikimedia\.com/v2/snapshots\)\. Text licensed under CC BY\-SA 4\.0 with attribution to Wikipedia contributors\.External Links:[Link](https://huggingface.co/datasets/HuggingFaceFW/finewiki)Cited by:[§A\.1](https://arxiv.org/html/2610.09122#A1.SS1.p4.1),[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Ponksheet al\.\(2024\)K\. Ponkshe, R\. Singhal, E\. Gorbunov, A\. Tumanov, S\. Horvath, and P\. VepakommaInitialization using update approximation is a silver bullet for extremely efficient low\-rank fine\-tuning\.arXiv preprint arXiv:2411\.19557\.Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1)\.
- Qiuet al\.\(2025a\)Z\. Qiu, S\. Buchholz, T\. Xiao, M\. Dax, B\. Schölkopf, and W\. LiuReparameterized LLM training via orthogonal equivalence transformation\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference,pp\. 140775–140821\.External Links:[Document](https://dx.doi.org/10.52202/085713-4704),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/cde43c5d698076ae17f3838a80f5a1bc-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2610.09122#S1.p2.1)\.
- Qiuet al\.\(2023\)Z\. Qiu, W\. Liu, H\. Feng, Y\. Xue, Y\. Feng, Z\. Liu, D\. Zhang, A\. Weller, and B\. SchölkopfControlling text\-to\-image diffusion by orthogonal finetuning\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 79320–79362\.External Links:[Document](https://dx.doi.org/10.52202/075280-3472),[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/faacb7a4827b4d51e201666b93ab5fa7-Paper-Conference.pdf)Cited by:[§C\.3](https://arxiv.org/html/2610.09122#A3.SS3.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1),[§1](https://arxiv.org/html/2610.09122#S1.p5.1),[§3](https://arxiv.org/html/2610.09122#S3.p1.1),[§4](https://arxiv.org/html/2610.09122#S4.SS0.SSS0.Px2.p1.1)\.
- Qiuet al\.\(2025b\)Z\. Qiu, W\. Liu, A\. Weller, and B\. SchölkopfOrthogonal finetuning made scalable\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 31946–31963\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1627/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1627),ISBN 979\-8\-89176\-332\-6Cited by:[§A\.1](https://arxiv.org/html/2610.09122#A1.SS1.p1.1),[§A\.2](https://arxiv.org/html/2610.09122#A1.SS2.p1.1),[§C\.6](https://arxiv.org/html/2610.09122#A3.SS6.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1),[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2024\)Qwen TeamQwen2\.5: a party of foundation models\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5/)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. SutskeverLearning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning,M\. Meila and T\. Zhang \(Eds\.\),Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.External Links:[Link](https://proceedings.mlr.press/v139/radford21a.html)Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Ruizet al\.\(2023\)N\. Ruiz, Y\. Li, V\. Jampani, Y\. Pritch, M\. Rubinstein, and K\. AbermanDreamBooth: fine tuning text\-to\-image diffusion models for subject\-driven generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 22500–22510\.Cited by:[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Schreiber and Van Loan \(1989\)R\. Schreiber and C\. Van LoanA storage\-efficientW​YWYrepresentation for products of Householder transformations\.SIAM Journal on Scientific and Statistical Computing10\(1\),pp\. 53–57\.External Links:ISSN 2168\-3417,[Link](http://dx.doi.org/10.1137/0910005),[Document](https://dx.doi.org/10.1137/0910005)Cited by:[§B\.1](https://arxiv.org/html/2610.09122#A2.SS1.p1.2)\.
- Schulman and Thinking Machines Lab \(2025\)J\. Schulman and Thinking Machines LabLoRA without regret\.Thinking Machines Lab: Connectionism\.External Links:[Document](https://dx.doi.org/10.64434/tml.20250929),[Link](https://thinkingmachines.ai/blog/lora/)Cited by:[§1](https://arxiv.org/html/2610.09122#S1.p4.1)\.
- Shuttleworthet al\.\(2025\)R\. Shuttleworth, J\. Andreas, A\. Torralba, and P\. SharmaLoRA vs full fine\-tuning: an illusion of equivalence\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference,pp\. 174627–174662\.External Links:[Document](https://dx.doi.org/10.52202/085713-5808),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/ff541950d1e885af90f523571564a401-Paper-Conference.pdf)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px2.p1.1)\.
- Stewart \(1990\)G\. W\. StewartPerturbation theory for the singular value decomposition\.Technical reportTechnical ReportUMIACS\-TR\-90\-124, CS\-TR\-2539,University of Maryland, Institute for Advanced Computer Studies\.External Links:[Link](https://api.drum.lib.umd.edu/server/api/core/bitstreams/a38cc6db-9bb7-4f3b-ae43-55b5827bb9d4/content)Cited by:[§E\.2](https://arxiv.org/html/2610.09122#A5.SS2.p1.1.1.1),[Proposition 4\.2](https://arxiv.org/html/2610.09122#S4.Thmproposition2.p1.2.1)\.
- Wanget al\.\(2025\)H\. Wang, Y\. Li, S\. Wang, G\. Chen, and Y\. ChenMiLoRA: harnessing minor singular components for parameter\-efficient LLM finetuning\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4823–4836\.External Links:[Link](https://aclanthology.org/2025.naacl-long.248/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.248),ISBN 979\-8\-89176\-189\-6Cited by:[§B\.1](https://arxiv.org/html/2610.09122#A2.SS1.p1.1),[§3](https://arxiv.org/html/2610.09122#S3.p1.1)\.
- Wanget al\.\(2024\)S\. Wang, L\. Yu, and J\. LiLoRA\-GA: low\-rank adaptation with gradient approximation\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 54905–54931\.External Links:[Document](https://dx.doi.org/10.52202/079017-1741),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/62c4718cc334f6a0a62fb81c4a2095a1-Paper-Conference.pdf)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1)\.
- Wanget al\.\(2026\)Y\. Wang, W\. Zhong, L\. Bai, Z\. Zhou, S\. Shao, B\. Cheng, S\. Chen, S\. Yang, and Z\. XieExploring data\-free LoRA transferability for video diffusion models\.InInternational Conference on Machine Learning,External Links:[Link](https://icml.cc/virtual/2026/poster/64894)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px2.p1.1)\.
- Yuet al\.\(2024\)L\. Yu, W\. JIANG, H\. Shi, J\. YU, Z\. Liu, Y\. Zhang, J\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetaMath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 45040–45061\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/c400474e8a36d0812fdee52739288b12-Paper-Conference.pdf)Cited by:[§A\.1](https://arxiv.org/html/2610.09122#A1.SS1.p1.1),[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.
- Yuanet al\.\(2024\)S\. Yuan, H\. Liu, and H\. XuBridging the gap between low\-rank and orthogonal adaptation via Householder reflection adaptation\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 113484–113518\.External Links:[Document](https://dx.doi.org/10.52202/079017-3606),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/cdd0640218a27e9e2c0e52e324e25db0-Paper-Conference.pdf)Cited by:[§B\.1](https://arxiv.org/html/2610.09122#A2.SS1.p1.1),[§1](https://arxiv.org/html/2610.09122#S1.p2.1),[§1](https://arxiv.org/html/2610.09122#S1.p4.1),[§1](https://arxiv.org/html/2610.09122#S1.p5.1),[§3](https://arxiv.org/html/2610.09122#S3.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, F\. Liu, and Y\. ChenLoRA\-One: one\-step full gradient could suffice for fine\-tuning large language models, provably and efficiently\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 75513–75574\.External Links:[Link](https://proceedings.mlr.press/v267/zhang25ax.html)Cited by:[§1\.2](https://arxiv.org/html/2610.09122#S1.SS2.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12834–12859\.External Links:[Link](https://aclanthology.org/2024.findings-acl.762/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.762)Cited by:[§B\.3](https://arxiv.org/html/2610.09122#A2.SS3.p1.1),[§3](https://arxiv.org/html/2610.09122#S3.SS0.SSS0.Px2.p1.1)\.

###### Contents

1. [References](https://arxiv.org/html/2610.09122#bib)
2. [AExperimental settings and metrics](https://arxiv.org/html/2610.09122#A1)
3. [BComplete selected comparisons and retention checks](https://arxiv.org/html/2610.09122#A2)
4. [CGeometric measurements and language spectral interventions](https://arxiv.org/html/2610.09122#A3)
5. [DFLUX spectral interventions and numerical controls](https://arxiv.org/html/2610.09122#A4)
6. [EGeometry of additive updates](https://arxiv.org/html/2610.09122#A5)
7. [FReview of experimental settings in PEFT studies](https://arxiv.org/html/2610.09122#A6)

## Appendix AExperimental settings and metrics

Training and functional evaluations used a cluster of NVIDIA A100 GPUs\. Each task used a single GPU, with at most 30 GPUs in use simultaneously\. The post\-hoc hyperspherical\-energy measurements used NVIDIA L40 GPUs\.

Table 4:Study overview\. Each comparison uses a shared pretrained backbone\. Divergent runs are reported with the corresponding results\.### A\.1Language adaptation

Qwen2\.5\-7B and Llama\-3\.1\-8B use 20,000 MetaMath examples\([Yu et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib12)\), microbatch size two, gradient accumulation over 16 steps, sequence length 768, 625 optimizer updates, cosine decay, 3% warmup, gradient clipping at one and zero weight decay\. LoRA usesα=2​r\\alpha=2rand zero dropout\. OFT uses unshared coordinate blocks and OFTv2’s Cayley–Neumann parameterization with five terms\([Qiu et al\., 2025b](https://arxiv.org/html/2610.09122#bib.bib5)\)\. The seven projection types are query, key, value, output, up, gate and down, giving 196 adapted matrices in Qwen and 224 in Llama\. LRs are\{1,3,10,30,100\}×10−5\\\{1,3,10,30,100\\\}\\times 10^\{\-5\}and seeds are 13, 37 and 73\.

All six methods use AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.09122#bib.bib44)\)\. LoRA\-family ranks are 7/14/28 for Qwen and 7/15/30 for Llama\. OFT block sizes are 32/64/128, and HRA uses 16/32/64 Householder vectors\. For each model/method/capacity, mean accuracy on the fixed 1,000\-item GSM8K development panel selects one LR from fully completed configurations, with ties favoring the lower LR\. The resulting 108 checkpoints are evaluated on the official 1,319\-item test set\. The diverged Qwen PiSSA configuration is excluded from LR selection and complete\-configuration frontiers\. Appendix[B\.1](https://arxiv.org/html/2610.09122#A2.SS1)reports all selected outcomes and the later HRA LR extension separately\.

Coding uses the same two backbones, six methods and five\-rate grid at one capacity per method\. Its held\-out response\-NLL selection, completion\-based testing and full settings are specified in Appendix[B\.3](https://arxiv.org/html/2610.09122#A2.SS3)\.

Standard and intervention retention use FineWiki\-200\([Penedo, 2025](https://arxiv.org/html/2610.09122#bib.bib17)\), with contexts truncated to 768 tokens\. The intervention panel evaluates GSM8K development accuracy on 1,000 items\. General NLL and full\-vocabulary KL use all 200 FineWiki documents, with token\-pooled aggregation\. General NLL scores valid next\-token positions\. All intervention comparisons use the same evaluation inputs and checkpoint\-local references\.

### A\.2Diffusion personalization

FLUX\.2\-klein\-base\-4B uses AdamW, batch size two, weight decay10−410^\{\-4\}, gradient clipping at one, constant LR and 750 updates\. The benchmark\-cat set has 14 training, two validation and four test images\. Four approximately matched parameter pairs are r4/b32, r8/b64, r16/b128 and r32/b256\. LoRA usesα=r\\alpha=r\. OFT uses unshared coordinate blocks with OFTv2’s Cayley–Neumann parameterization\([Qiu et al\., 2025b](https://arxiv.org/html/2610.09122#bib.bib5)\)\. The LR grid is\{0\.5,1,3,5,10,30,50,100\}×10−5\\\{0\.5,1,3,5,10,30,50,100\\\}\\times 10^\{\-5\}, with seeds 0, 1 and 2\. Generation uses resolution 512, 20 inference steps and guidance 3\.5\. For each method/capacity, we select a shared LR by the mean validation DINO at step 700, the last validation evaluation, then report test and retention scores at final step 750\. The two validation and four test images are disjoint\.

The complete six\-method cat comparison uses LoRA/DoRA/PiSSA/MiLoRA ranks 4/8/16/32, OFT blocks 32/64/128/256, and HRA 16/32/62/124 Householder vectors\. All 576 runs across six methods, four capacities and eight LRs are complete\.

Target DINOv2 similarity averages the paired generated/reference test\-image cosines over four examples and three generation repeats\. General and class drift compare paired base/adapted generations using\(1−cosine\)/2\(1\-\\mathrm\{cosine\}\)/2, over five prompts in each bank\. General CLIP is normalized text/image cosine averaged over five general prompts\. Training and generation randomness contribute to the reported seed variation\.

The 28\-object study uses three training images, one validation image and one or two held\-out test images per object\. It crosses the same four capacity pairs with LRs\{3,5,10\}×10−5\\\{3,5,10\\\}\\times 10^\{\-5\}and seeds 0, 1 and 2, for 2,016 runs\. For each method/capacity, we select one LR by mean step\-700 validation DINO across all 28 objects and seeds, breaking ties toward the smaller LR\. That LR is shared across objects and seeds\. Test DINO, averaged over three generation repeats, and retention are measured at step 750\. The retention bank uses 200 COCO image/caption pairs and four scheduler\-index fractions,\{0\.125,0\.375,0\.625,0\.875\}\\\{0\.125,0\.375,0\.625,0\.875\\\}\. Image drift and CLIP use the same four general prompts for every object, depicting an angry Mona Lisa, the Eiffel Tower at night, an astronaut riding a horse, and an avocado\-shaped chair\.

### A\.3Metrics and uncertainty

#### Language likelihood\.

For documentjjand its valid next\-token positionsTjT\_\{j\}, general\-text NLL is

NLL\(𝚯\)=−1∑j\|Tj\|∑j∑t∈Tjlogp𝚯\(xj,t\|xj,<t\)\.\\mathrm\{NLL\}\(\\bm\{\\Theta\}\)=\-\\frac\{1\}\{\\sum\_\{j\}\|T\_\{j\}\|\}\\sum\_\{j\}\\sum\_\{t\\in T\_\{j\}\}\\log p\_\{\\bm\{\\Theta\}\}\\\!\\left\(x\_\{j,t\}\\middle\|x\_\{j,<t\}\\right\)\.All models use identical teacher\-forced contexts and masks, excluding padding\. The score weights tokens equally and measures fit to the general\-text bank\.

#### Diffusion outputs and predictive change\.

Writecos⁡\(𝒂,𝒃\)=𝒂⊤​𝒃‖𝒂‖2​‖𝒃‖2\\cos\(\\bm\{a\},\\bm\{b\}\)=\\frac\{\\bm\{a\}^\{\\\!\\top\}\\bm\{b\}\}\{\\left\\\|\\bm\{a\}\\right\\\|\_\{2\}\\left\\\|\\bm\{b\}\\right\\\|\_\{2\}\}, with normalized DINOv2 features𝐟⁡\(I\)\\bm\{\\mathrm\{f\}\}\(I\)and CLIP image/text features𝐠⁡\(I\),𝐡⁡\(c\)\\bm\{\\mathrm\{g\}\}\(I\),\\bm\{\\mathrm\{h\}\}\(c\)\. For generated imageIj𝚯I\_\{j\}^\{\\bm\{\\Theta\}\}, paired base imageIj0I\_\{j\}^\{0\}, target reference imageIjrefI\_\{j\}^\{\\rm ref\}and promptcjc\_\{j\}, the reported averages are

DINO\\displaystyle\\mathrm\{DINO\}=1J​∑jcos⁡\(𝐟⁡\(Ij𝚯\),𝐟⁡\(Ijref\)\),\\displaystyle=\\frac\{1\}\{J\}\\sum\_\{j\}\\cos\\\!\\left\(\\bm\{\\mathrm\{f\}\}\(I\_\{j\}^\{\\bm\{\\Theta\}\}\),\\bm\{\\mathrm\{f\}\}\(I\_\{j\}^\{\\rm ref\}\)\\right\),Dimage\\displaystyle D\_\{\\rm image\}=1J​∑j1−cos⁡\(𝐟⁡\(Ij𝚯\),𝐟⁡\(Ij0\)\)2,\\displaystyle=\\frac\{1\}\{J\}\\sum\_\{j\}\\frac\{1\-\\cos\\\!\\left\(\\bm\{\\mathrm\{f\}\}\(I\_\{j\}^\{\\bm\{\\Theta\}\}\),\\bm\{\\mathrm\{f\}\}\(I\_\{j\}^\{0\}\)\\right\)\}\{2\},CLIP\\displaystyle\\mathrm\{CLIP\}=1J​∑jcos⁡\(𝐠⁡\(Ij𝚯\),𝐡⁡\(cj\)\)\.\\displaystyle=\\frac\{1\}\{J\}\\sum\_\{j\}\\cos\\\!\\left\(\\bm\{\\mathrm\{g\}\}\(I\_\{j\}^\{\\bm\{\\Theta\}\}\),\\bm\{\\mathrm\{h\}\}\(c\_\{j\}\)\\right\)\.Each metric uses its own stated bank ofJJpairs\. General and class drift use different prompt banks\. Paired images share generation seeds\. Repeats are averaged before training\-seed summaries\. The 28\-object tables report absolute target DINO, COCO flow loss and CLIP alignment, together with base\-relative image drift\.

For the 28\-object COCO flow loss,𝒚j,k\\bm\{y\}\_\{j,k\}contains the fixed noisy latent, scheduler time and conditioning for image/caption pairjjin time bucketkk\. Withdjd\_\{j\}velocity coordinates,KKbuckets, flow target𝒖j,k\\bm\{u\}\_\{j,k\}and loss weightwj,kw\_\{j,k\},

ℒflow​\(𝚯\)=1N​K​∑j,kwj,kdj​‖𝒗𝚯​\(𝒚j,k\)−𝒖j,k‖22\.\\mathcal\{L\}\_\{\\rm flow\}\(\\bm\{\\Theta\}\)=\\frac\{1\}\{NK\}\\sum\_\{j,k\}\\frac\{w\_\{j,k\}\}\{d\_\{j\}\}\\left\\\|\\bm\{v\}\_\{\\bm\{\\Theta\}\}\(\\bm\{y\}\_\{j,k\}\)\-\\bm\{u\}\_\{j,k\}\\right\\\|\_\{2\}^\{2\}\.The rectified\-flow target is noise minus clean latent\([Liu et al\., 2023b](https://arxiv.org/html/2610.09122#bib.bib45)\)\. This fixed\-state prediction loss is distinct from generated\-image drift\.

#### Uncertainty\.

ForSStraining seeds with scoreszsz\_\{s\}, we reportz¯=S−1​∑szs\\bar\{z\}=S^\{\-1\}\\sum\_\{s\}z\_\{s\}and sampleSD=\(∑s\(zs−z¯\)2/\(S−1\)\)1/2\\mathrm\{SD\}=\\left\(\\sum\_\{s\}\(z\_\{s\}\-\\bar\{z\}\)^\{2\}/\(S\-1\)\\right\)^\{1/2\}\. Perturbation and generation repetitions are averaged within a checkpoint first\. For each 28\-object method/capacity/LR setting,zs=28−1​∑o=128zs,oz\_\{s\}=28^\{\-1\}\\sum\_\{o=1\}^\{28\}z\_\{s,o\}\. The SD describes variation across training seeds\.

## Appendix BComplete selected comparisons and retention checks

### B\.1Six\-method mathematics comparison and held\-out evaluation

We extend the same Qwen2\.5\-7B/Llama\-3\.1\-8B mathematics setup with four methods that vary adaptation structure\. DoRA separates weight\-vector magnitudes and directions, learning magnitudes alongside a low\-rank directional update\([Liu et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib4)\)\. PiSSA and MiLoRA initialize the trainable factors from leading and trailing singular components, respectively\([Meng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib23);[Wang et al\., 2025](https://arxiv.org/html/2610.09122#bib.bib26)\)\. In both cases, subtracting the initial low\-rank component from the frozen base preserves the starting model\. HRA learns a product of Householder reflections\([Yuan et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib25)\)\. In an input\-side convention,

𝑾=𝑾0​∏j=1rH𝑯j,𝑯j=𝑰−2​𝒗j​𝒗j⊤𝒗j⊤​𝒗j\.\\bm\{W\}=\\bm\{W\}\_\{0\}\\prod\_\{j=1\}^\{r\_\{H\}\}\\bm\{H\}\_\{j\},\\qquad\\bm\{H\}\_\{j\}=\\bm\{I\}\-2\\frac\{\\bm\{v\}\_\{j\}\\bm\{v\}\_\{j\}^\{\\\!\\top\}\}\{\\bm\{v\}\_\{j\}^\{\\\!\\top\}\\bm\{v\}\_\{j\}\}\.The implementation uses the compact WY representation\([Schreiber and Van Loan, 1989](https://arxiv.org/html/2610.09122#bib.bib35);[Likhosherstov et al\., 2021](https://arxiv.org/html/2610.09122#bib.bib39)\), paired identity initialization and no Gram–Schmidt orthogonalization\. HererHr\_\{H\}counts Householder vectors\. DoRA/PiSSA/MiLoRA use training ranks 7/14/28 for Qwen and 7/15/30 for Llama, withα=2​r\\alpha=2r\. HRA uses 16/32/64 vectors\. The capacity tiers approximately match trainable parameter counts\. PiSSA/MiLoRA serialization expresses their final differences against the original base at rank2​r2r\. The tables report the actual training rank and trainable parameter count\.

All methods use the same seven attention/MLP projection types, AdamW, batch 32, sequence length 768, 625 updates, cosine decay, warmup fraction 0\.03, zero weight decay and gradient clipping at one\. The original grid uses LRs\{1,3,10,30,100\}×10−5\\\{1,3,10,30,100\\\}\\times 10^\{\-5\}and seeds 13/37/73\. It contains 540 training outcomes, comprising 180 LoRA/OFT and 360 added\-method runs\. Qwen PiSSA r7 at LR10−310^\{\-3\}, seed 73, diverged at step 55, leaving 539 final development evaluations\. The group with two successful seeds is excluded from LR selection and complete\-configuration frontiers\.

#### Selection and test separation\.

One LR per model/method/capacity maximizes the exact total correct count across seeds on the fixed 1,000\-problem, group\-held\-out GSM8K development panel\. Ties choose the smaller LR\. The resulting 36 configurations and 108 checkpoints are fixed before test evaluation, with one LR shared across seeds\. Each checkpoint is evaluated on all 1,319 official GSM8K test problems\. Prior test\-set use for evaluation\-harness development was separate from this selection\. Generation uses the same native BF16 merged\-adapter path, prompt and answer parser, greedy decoding, a 300\-new\-token cap, an 800\-total\-token cap and a 1,024\-token engine context\. The six\-method intervention figures use development scores\.

Retention reuses each run’s recorded standard step\-625 FineWiki\-200 measurement, at length 768 and batch two\. We reportΔ​NLL=NLL625−NLL0\\Delta\\mathrm\{NLL\}=\\mathrm\{NLL\}\_\{625\}\-\\mathrm\{NLL\}\_\{0\}, using the corresponding pretrained reference\. Smaller values indicate better general\-text retention\. We report sample SD across training seeds\.

Table 5:Qwen2\.5\-7B results for all development\-selected configurations from the original five\-LR grid\. Parameters are in millions\. Ranks are training ranks \(HRA counts Householder vectors\)\. Accuracy is in percent, andΔ\\DeltaNLL is the recorded FineWiki\-200 NLL minus its matching pretrained value\. LRs were selected using development accuracy only\.Table 6:Llama\-3\.1\-8B results for all development\-selected configurations from the original five\-LR grid\. Parameters are in millions\. Ranks are training ranks \(HRA counts Householder vectors\)\. Accuracy is in percent, andΔ\\DeltaNLL is the recorded FineWiki\-200 NLL minus its matching pretrained value\. LRs were selected using development accuracy only\.Figure 6:Six\-method mathematics trade\-offs, focused on the frontier regions\. Each point is a complete capacity/LR configuration, with shapes denoting increasing capacity\. Dashed lines and solid points mark within\-method frontiers, and black borders mark the pooled frontier\. Dominated points are faint\. Frontiers use all 89 Qwen and 90 Llama configurations before axis clipping\. The later HRA extension and diverged PiSSA configuration are excluded\.
#### Performance and retention across settings\.

Qwen’s small test\-accuracy differences coexist with different retention costs\. Its high\-LR, small\-rank LoRA selection is an outlier with greater forgetting than the other methods\. LoRA’s larger\-rank selections have less forgetting\. HRA’s Qwen scores are competitive, and its original\-grid Llama scores improve with capacity alongside increasing retention cost\. These results support the main finding that competitive task scores can accompany different forgetting costs, with outcomes depending on the selected LR and capacity\. Appendix[B\.6](https://arxiv.org/html/2610.09122#A2.SS6)summarizes the trends across tasks\.

### B\.2Separate HRA learning\-rate extension

All three original Llama HRA winners occur at the largest LR,10−310^\{\-3\}\. A follow\-up therefore adds3×10−33\\times 10^\{\-3\}and10−210^\{\-2\}at each capacity and seed\. The extension was proposed after the original test report and is reported separately\. Of 18 training runs, 15 complete both development and official\-test evaluations\. All three r64 runs at10−210^\{\-2\}diverge\. Selection over the seven\-LR development grid chooses3×10−33\\times 10^\{\-3\}at r16/r32 and retains10−310^\{\-3\}at r64\. The main comparison retains the original five\-LR selections\.

Table 7:Llama\-3\.1\-8B HRA development\-selected outcomes after the separate seven\-LR follow\-up\. The r64 selection is unchanged from the original grid\.nnis the successful seed count\. LR selection uses development scores\.The development\-selected3×10−33\\times 10^\{\-3\}extension raises mean test accuracy at both r16 and r32, while increasing their retention costs\. At r64,3×10−33\\times 10^\{\-3\}reduces accuracy and increases NLL, so development selection retains the original LR\. HRA therefore illustrates the setting\-dependent trade\-offs in Section[5](https://arxiv.org/html/2610.09122#S5), with the useful LR range changing across capacities\.

### B\.3CodeFeedback selection and evaluation

We use the Python subset of CodeFeedback\([Zheng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib29)\), distributed for PiSSA\([Meng et al\., 2024](https://arxiv.org/html/2610.09122#bib.bib23)\)\. After exact prompt deduplication, a fixed split contains 20,000 training and 2,000 development examples\. Exact prompt matching against the test sets removes no additional examples\. All methods use one epoch, effective batch size 32, maximum training length 1,024, a cosine schedule with 3% warmup, and the same attention/MLP projections\. LoRA, DoRA, PiSSA and MiLoRA train at rank 7\. OFT uses block size 32 and HRA uses 16 Householder vectors\. PiSSA/MiLoRA’s converted rank\-14 difference adapters retain the same training capacity\.

For each method/backbone, the mean development response NLL across seeds 13, 37 and 73 selects one of\{10−5,3×10−5,10−4,3×10−4,10−3\}\\\{10^\{\-5\},3\\times 10^\{\-5\},10^\{\-4\},3\\times 10^\{\-4\},10^\{\-3\}\\\}, with ties favoring the lower LR\. NLL pools valid response\-token losses, with a 4,096\-token evaluation limit\. Thus the 180 development runs yield 36 selected test checkpoints, each using the best LR within the tested grid\.

Test evaluation uses EvalPlus base completion prompts, greedy decoding, and one completion per problem, capped at 1,024 new tokens\. We evaluate 164 HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.09122#bib.bib42)\)and 378 MBPP\([Austin et al\., 2021](https://arxiv.org/html/2610.09122#bib.bib43)\)problems\. Original and strengthened tests grade the same completion\.[Table8](https://arxiv.org/html/2610.09122#A2.T8)reports both versions\. The base model has one fixed evaluation, so its row has no training\-seed SD\.

General retention uses the fixed FineWiki\-200 corpus, a 768\-token limit, and valid\-token\-pooled NLL with float32 logits\. Each method’s forgetting is its NLL minus the corresponding unadapted model’s NLL\.

Table 8:CodeFeedback results for all original and strengthened\-test pass@1 scores \(%\)\. A single greedy completion is graded under both test suites\. LR minimizes mean held\-out response NLL\. Only selected checkpoints have test evaluations\. The Base row has no training\-seed SD\.#### Task gains and retention costs\.

All six methods improve Llama’s HumanEval\+ and MBPP\+ means over the base model, with rankings that differ across benchmarks and retention\. PiSSA leads on HumanEval\+, alongside higher retention costs\. Together, these results extend the distinction between task performance and forgetting to coding, with the size of the task gain depending on the backbone\.

### B\.4Six\-method cat comparison across learning rates and capacities

The complete cat study crosses six methods, four approximately matched parameter budgets, eight LRs and seeds 0, 1 and 2, giving 576 successful runs and 192 configuration means\. All methods share the cat split \(14 training, two validation and four test images\), adapted modules, training exposure and evaluation steps\. LoRA, DoRA, PiSSA and MiLoRA use ranks 4/8/16/32, OFT uses block sizes 32/64/128/256, and HRA uses 16/32/62/124 Householder vectors\. PiSSA/MiLoRA ranks refer to training, with doubled ranks used to serialize their differences from the base model\.

The LR grid is\{5×10−6,10−5,3×10−5,5×10−5,10−4,3×10−4,5×10−4,10−3\}\\\{5\\times 10^\{\-6\},10^\{\-5\},3\\times 10^\{\-5\},5\\times 10^\{\-5\},10^\{\-4\},3\\times 10^\{\-4\},5\\times 10^\{\-4\},10^\{\-3\}\\\}\. Within each method/capacity, mean validation DINO at step 700 selects one LR shared across seeds, with ties favoring smaller LR\. Test and retention use step 750\.[Table9](https://arxiv.org/html/2610.09122#A2.T9)gives all 24 selections and parameter counts, with sample seed SD\.

Table 9:Complete validation\-selected FLUX cat results\. Budgets are approximately matched, with HRA using 16/32/62/124 vectors and DoRA adding magnitude parameters\. Parameters are in millions\. Validation DINO at step 700 selects LR separately for each method/capacity\. Step 750 supplies test DINO, general drift and general CLIP\. Scores are×100\\times 100\.#### Low selected drift and capacity\-dependent fidelity\.

LoRA has the lowest selected mean general drift at 3 of 4 budgets and ranks among the lowest two at all 4\. PiSSA has greater drift than LoRA in all 12 paired seeds\. PiSSA has the highest selected drift at the small budget, OFT at the smallest and largest budgets, and HRA at the medium budget\. DoRA exceeds LoRA’s mean test DINO at every selected budget and leads at the smallest and largest\. MiLoRA has lower mean test DINO than LoRA at all four budgets, with a mean CLIP advantage only at the small budget\. These results support LoRA’s recurring retention advantage and DoRA’s higher mean task scores, alongside setting\-dependent outcomes for the other methods\.

Figure 7:Six\-method FLUX cat test trade\-offs, focused on the frontier regions\. Shapes denote increasing capacity\. Dashed lines and solid points mark within\-method frontiers, and black borders mark the pooled frontier\. Dominated points are faint\. Frontiers use all 192 configurations before axis clipping, with separate y ranges for each panel\. Test DINO describes final\-step outcomes, while validation DINO selects LR\.
#### Retention metrics\.

General drift and CLIP use five unrelated prompts, while class drift uses five cat prompts\. All six methods have these measurements\. Drift measures preservation of generated outputs, and CLIP measures prompt alignment\. Appendix[D\.1](https://arxiv.org/html/2610.09122#A4.SS1)examines the role of spectral components through six\-method interventions at the largest budget\.

### B\.528\-object comparison

[Table3](https://arxiv.org/html/2610.09122#S5.T3)b reports every validation\-selected capacity pair\. Each score first averages the 28 objects within a seed, then reports the mean and sample SD across seeds\. The shared\-LR selection and evaluation banks are specified in Appendix[A](https://arxiv.org/html/2610.09122#A1)\.

#### Retention across objects and metrics\.

At the validation\-selected settings, OFT has lower COCO flow loss at three of four capacities\. LoRA has lower general\-image drift and higher prompt alignment at every capacity\. The same distinction appears at a shared LR of5×10−55\\times 10^\{\-5\}, where OFT b64 has lower COCO loss than LoRA r8, but higher image drift and lower CLIP\. Flow loss evaluates velocity predictions on fixed noisy latents, whereas drift and CLIP evaluate the final generated images\. The object comparison supports LoRA’s retention advantage in generated outputs and shows why the metrics capture different aspects of preservation\.

### B\.6Retention across all six methods

We compare LoRA’s forgetting with*each of the other five methods*across twelve settings, comprising six mathematics model/capacity pairs, two coding backbones and four cat budgets\. Each method’s LR is chosen by task performance on the development set\. We compare forgetting within each setting, so language NLL and image drift remain on their own scales\.

#### LoRA most consistently limits forgetting\.

LoRA has the lowest mean forgetting in 9/12 settings\. OFT, MiLoRA and HRA each have the lowest forgetting in one setting, and DoRA and PiSSA in none \([Table10](https://arxiv.org/html/2610.09122#A2.T10)\)\. LoRA also has the lowest average rank, 1\.58, followed by DoRA at 2\.75\. The three exceptions are small\-capacity Qwen, where LoRA’s outlying high\-LR selection forgets most, large\-capacity Llama, where OFT has the lowest forgetting, and cat rank 8, where HRA has the lowest forgetting\.

Table 10:Forgetting across all six methods\. The lowest\-forgetting columns count settings in which a method has the lowest mean forgetting\. A rank of 1 means least forgetting, and lower average rank is better\. The right columns choose LR using the other two seeds, evaluate the remaining seed, then average those evaluations\. LoRA has the lowest forgetting in the most settings and the lowest average rank under both selections\.
#### Consistency across selection and weighting choices\.

First, we choose LR using two seeds and measure forgetting on the remaining seed, repeating for every seed\. The original grids, eligibility rules and smaller\-LR tie rule remain unchanged\. LoRA still has the lowest mean forgetting in 8/12 settings\. Each other method does so in at most one\. Its average rank remains lowest \(2\.08, versus 2\.50 for DoRA\)\. Second, we give each model/task group equal importance, preventing groups with more capacities from dominating the summary\. Third, we remove each backbone or individual setting in turn\. Finally, we compare results using one language seed and one image seed at a time, covering all nine combinations rather than matching unrelated seed numbers\.

Across all these checks, LoRA has the lowest forgetting in more settings than any other method \([Table11](https://arxiv.org/html/2610.09122#A2.T11)\)\. This supports the claim that LoRA most consistently limits forgetting across the tested methods\. After LR selection on other seeds, DoRA has a better average rank in three seed combinations and when FLUX is removed\.

Table 11:Robustness of the retention comparison\. Frequency columns count versions in which LoRA has the lowest forgetting in more settings than any other method\. Average\-rank columns count versions in which LoRA has the lowest average rank\. Equal weighting averages within the five model/task groups before averaging groups\. The nine seed combinations cross all language seeds with all image seeds\.
#### Task scores and PiSSA’s retention costs\.

DoRA has higher mean task scores than LoRA in 10/14 comparisons, covering two of six mathematics settings, all four coding comparisons and all four cat budgets\. This supports the reported trend in mean task scores\. Its coding and cat gains accompany additional forgetting\.

With backbone, capacity and LR fixed across all six methods, PiSSA forgets most in all 29/29 complete mathematics comparisons and 10/10 coding comparisons, including all 87 and 30 seed comparisons\. In cat, it ranks highest in 21/32 comparisons and among the highest two in all 32\. HRA ranks lowest in 25/32\. These matched\-LR comparisons reinforce the finding that PiSSA often incurs greater retention costs and show that HRA can retain well in specific settings\.

#### LoRA versus OFT across objects\.

After averaging seeds within each object, LoRA has lower drift than OFT for 23, 27, 28 and 28 of the 28 objects across increasing budgets\. This extends LoRA’s image\-drift advantage beyond cat personalization\.

## Appendix CGeometric measurements and language spectral interventions

### C\.1Checkpoints, controls and reference models

The intervention study covers 108 checkpoints selected by mean development accuracy within each model/method/capacity, with 22 interventions or controls per checkpoint\. Llama HRA uses the extended\-grid selections in Appendix[B\.2](https://arxiv.org/html/2610.09122#A2.SS2)\. All edits use the same GSM8K development and general\-text banks\. The tables below extend the largest\-capacity main figures to all capacities and selected LRs\.

Task and NLL changes use the trained model as reference\. Value\-restoration controls compare against reconstruction, while component sufficiency and factorial analyses use full value restoration\. Forgetting recovery is the fraction of the trained model’s NLL increase recovered by an edit, computed within each checkpoint before seed averaging\.

Edits jointly cover all adapted matrices\. SVD computations use FP32, followed by BF16 storage and evaluation\. The dominant, intermediate and trailing component groups divide the pretrained spectrum into approximate thirds, with boundaries adjusted to nearby spectral gaps\. Plots label these groups top, middle and tail\.

For the collectionℳ\\mathcal\{M\}of adapted matrices, the normalized displacements are

dW2=∑M∈ℳ‖𝑾∗,M−𝑾0,M‖F2∑M∈ℳ‖𝑾0,M‖F2,dS2=∑M∈ℳ‖𝒔∗,M−𝒔0,M‖22∑M∈ℳ‖𝒔0,M‖22\.d\_\{W\}^\{2\}=\\frac\{\\sum\_\{M\\in\\mathcal\{M\}\}\\left\\\|\\bm\{W\}\_\{\*,M\}\-\\bm\{W\}\_\{0,M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\}\{\\sum\_\{M\\in\\mathcal\{M\}\}\\left\\\|\\bm\{W\}\_\{0,M\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\},\\qquad d\_\{S\}^\{2\}=\\frac\{\\sum\_\{M\\in\\mathcal\{M\}\}\\left\\\|\\bm\{s\}\_\{\*,M\}\-\\bm\{s\}\_\{0,M\}\\right\\\|\_\{2\}^\{2\}\}\{\\sum\_\{M\\in\\mathcal\{M\}\}\\left\\\|\\bm\{s\}\_\{0,M\}\\right\\\|\_\{2\}^\{2\}\}\.
Each checkpoint pools all 196 Qwen or 224 Llama adapted matrices \(80 for the FLUX results in Appendix[D\.1](https://arxiv.org/html/2610.09122#A4.SS1)\), weighting their contributions through the squared norms\.

Table 12:Qwen2\.5\-7B spectral displacement at every selected capacity and LR\.Table 13:Qwen2\.5\-7B singular\-value restoration and reconstruction controls, with accuracy in percent\.Table 14:Qwen2\.5\-7B component\-restoration effects relative to the trained model, with accuracy changes in percentage points\.Table 15:Llama\-3\.1\-8B spectral displacement at every selected capacity and LR\.Table 16:Llama\-3\.1\-8B singular\-value restoration and reconstruction controls, with accuracy in percent\.Table 17:Llama\-3\.1\-8B component\-restoration effects relative to the trained model, with accuracy changes in percentage points\.
### C\.2Singular\-value displacement and restoration

At largest capacity, PiSSA concentrates over 99% of squared singular\-value displacement in dominant components on all three models\. MiLoRA concentrates about 99% on Qwen/FLUX and 95% on Llama in trailing components\. These profiles follow their initialization choices, while restoration measures how much the spectral changes contribute to task performance\.

Across all 108 checkpoints, singular\-value restoration changes general NLL by−0\.00535\-0\.00535to\+0\.01365\+0\.01365relative to reconstruction, alongside a maximum absolute task\-accuracy change of 1\.3 percentage points\. These small task effects extend the restoration finding across language capacities\.

#### A high\-LR LoRA outlier\.

Qwen LoRA r7 selects the highest tested LR,10−310^\{\-3\}\. Its spectral displacement is1\.67%1\.67\\%of the base norm and20\.53%20\.53\\%of the weight update, exceeding the ratios in the low\-drift settings\. Restoration still largely preserves task performance, with a mean accuracy change of\+0\.13\+0\.13pp relative to reconstruction \([Table13](https://arxiv.org/html/2610.09122#A3.T13)\)\. The restoration finding therefore also holds in this larger\-ratio setting\.

Figure 8:Singular\-value restoration across capacities and seeds\. Trained, reconstructed and restored models are shown separately, extending the main accuracy comparison with reconstruction controls and retention measurements\.Figure 9:Measured weight and singular\-value displacement across all six methods and both models at each selected capacity and LR\. Ratios express spectral displacement relative to weight displacement\.Figure 10:Singular\-value changes across spectral components at largest capacity\. PiSSA’s displacement concentrates near the leading end, whereas MiLoRA’s concentrates near the tail, consistent with their initialization choices\. The pretrained reference and method colors are retained\.

### C\.3Hyperspherical energy and pairwise geometry

We measure all adapted matrices in the largest\-capacity intervention checkpoints, with 196 per Qwen model, 224 per Llama model and 80 per FLUX cat model\. The 54 checkpoints and 9,000 matrix comparisons use the same selected LRs as the spectral and functional analyses\. This tests whether the approximate geometric preservation observed for LoRA\-family methods also extends to HE\.

#### Definitions and aggregation\.

For stored weights𝑾∈ℝn×d\\bm\{W\}\\in\\mathbb\{R\}^\{n\\times d\}, rows represent output neurons\. Let𝒘^i=𝒘i/‖𝒘i‖2\\widehat\{\\bm\{w\}\}\_\{i\}=\\bm\{w\}\_\{i\}/\\\|\\bm\{w\}\_\{i\}\\\|\_\{2\}\. Following[Liu et al\. \(2018\)](https://arxiv.org/html/2610.09122#bib.bib38)and[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.09122#bib.bib3), we compute

H⁡\(𝑾\)=∑i≠j1‖𝒘^i−𝒘^j‖2,δH=\|H⁡\(𝑾∗\)−H⁡\(𝑾0\)\|H⁡\(𝑾0\)\.H\(\\bm\{W\}\)=\\sum\_\{i\\neq j\}\\frac\{1\}\{\\\|\\widehat\{\\bm\{w\}\}\_\{i\}\-\\widehat\{\\bm\{w\}\}\_\{j\}\\\|\_\{2\}\},\\qquad\\delta\_\{H\}=\\frac\{\|H\(\\bm\{W\}\_\{\*\}\)\-H\(\\bm\{W\}\_\{0\}\)\|\}\{H\(\\bm\{W\}\_\{0\}\)\}\.The sum covers every ordered distinct row pair\. To measure pairwise changes without signed cancellation, we also compute

δC=\[1n⁡\(n−1\)​∑i≠j\(𝒘^∗,i⊤​𝒘^∗,j−𝒘^0,i⊤​𝒘^0,j\)2\]1/2\.\\delta\_\{C\}=\\left\[\\frac\{1\}\{n\(n\-1\)\}\\sum\_\{i\\neq j\}\\left\(\\widehat\{\\bm\{w\}\}\_\{\*,i\}^\{\\\!\\top\}\\widehat\{\\bm\{w\}\}\_\{\*,j\}\-\\widehat\{\\bm\{w\}\}\_\{0,i\}^\{\\\!\\top\}\\widehat\{\\bm\{w\}\}\_\{0,j\}\\right\)^\{2\}\\right\]^\{1/2\}\.We average each metric equally over matrices within a checkpoint, then summarize training seeds\. The absolute HE change is taken before matrix averaging\. A common right\-orthogonal transformation preserves row inner products and HE\.

Table 18:Geometric preservation at largest selected capacities\. Each checkpoint averages matrices equally, followed by the seed mean and sample SD\. HE changes are in units of10−4%10^\{\-4\}\\%, and cosine changes are in units of10−410^\{\-4\}\.
#### Results\.

All checkpoint means ofδH\\delta\_\{H\}are below0\.01%0\.01\\%, with maximum0\.00980%0\.00980\\%\([Table18](https://arxiv.org/html/2610.09122#A3.T18)\)\. This supports approximate HE preservation across the selected LoRA\-family methods as well as the orthogonal methods\. The pairwise measure shows how closely individual neuron relationships are preserved alongside aggregate HE\.

### C\.4Component effects, interactions and task performance

Restoring dominant components lowers general\-text NLL at every capacity and gives the largest reduction among the three groups in 106/108 checkpoints\. The exceptions to the largest reduction are Qwen DoRA r14/seed 73 and MiLoRA r28/seed 13\. Dominant restoration remains strongest in every configuration mean, supporting the consistent forgetting response across language capacities\.

![Refer to caption](https://arxiv.org/html/2610.09122v1/peft_app_02_all_configuration_top_resets.png)Figure 11:Dominant\-component restoration at every selected configuration and seed\. The retention response persists across the grid, while task\-accuracy responses vary\. Recovery is normalized by each checkpoint’s trained\-model NLL increase\.For a setAAof adapted component groups, define

𝑾⁡\(A\)=∑B∈A∑i∈Bs0,i​𝒖∗,i​𝒗∗,i⊤\+∑B∉A∑i∈Bs0,i​𝒖0,i​𝒗0,i⊤\.\\bm\{W\}\(A\)=\\sum\_\{B\\in A\}\\sum\_\{i\\in B\}s\_\{0,i\}\\bm\{u\}\_\{\*,i\}\\bm\{v\}\_\{\*,i\}^\{\\\!\\top\}\+\\sum\_\{B\\notin A\}\\sum\_\{i\\in B\}s\_\{0,i\}\\bm\{u\}\_\{0,i\}\\bm\{v\}\_\{0,i\}^\{\\\!\\top\}\.Its endpoints are the pretrained\-component reconstruction and the full value\-restored model\. For general NLLf⁡\(A\)f\(A\), the effect of adapting groupBBisf⁡\(A∪\{B\}\)−f⁡\(A\)f\(A\\cup\\\{B\\\}\)\-f\(A\)\. We evaluate it under all four combinations of pretrained or adapted components in the other groups\. Adapting dominant components increases NLL in every combination across all 108 checkpoints\. Intermediate and trailing effects change sign across combinations in 38 and 42 checkpoints, using a10−410^\{\-4\}tolerance\. Thus, the dominant\-component forgetting response persists as the other components change\.

All 36 configurations include evaluations of every component combination for all three seeds\.

Figure 12:Effects of adapting each component group under all combinations of the other groups\. Dominant\-component adaptation consistently increases general NLL, while intermediate and trailing effects can change sign\. Both NLL evaluations agree on the dominant\-component direction\.Figure 13:Task performance when retaining only some adapted components\. Cells show mean±\\pmSD changes from𝑾R\\bm\{W\}\_\{R\}\(all singular values restored\) at the largest capacities\. Columns retain the adapted dominant components alone or the intermediate and trailing components together\. Bold means mark losses exceeding 1 pp for LLMs or 1 DINO×100\\times 100unit for FLUX\.#### Task performance across selected capacities\.

[Table19](https://arxiv.org/html/2610.09122#A3.T19)extends the largest\-capacity comparison in[Figure13](https://arxiv.org/html/2610.09122#A3.F13)\. Qwen LoRA retains task performance with dominant components alone at small capacity, with neither component set alone at medium capacity, and with either set alone at large capacity\. Llama LoRA’s outcomes also vary across capacities and seeds\. PiSSA retains the dominant\-only pattern at every language capacity, supporting its consistent concentration of task gains in dominant components\. The other methods illustrate the setting\-dependent component contributions described in the main text, alongside the consistent forgetting response\.

Table 19:Task sufficiency across all selected language capacities\. Entries count seeds with accuracy loss≤1\\leq 1pp relative to full value restoration, out of three seeds per configuration\. Top and complement retain those adapted components on a pretrained background\. Tables[12](https://arxiv.org/html/2610.09122#A3.T12)and[15](https://arxiv.org/html/2610.09122#A3.T15)give capacities and selected LRs\.

### C\.5Consistency of the dominant\-component response

The comparison covers 42 selected configurations and 126 checkpoints across all six methods, all tested language capacities and the largest FLUX budget for each method\. Restoring dominant components gives the largest mean reduction in general\-text NLL or general\-image drift in all 42 configurations and a larger reduction than restoring either intermediate or trailing components in 123/126 checkpoints\. These counts support the recurring dominant\-component forgetting response across models and tasks\.

#### Consistency with smaller edits\.

We measure edit norm as the square root of the summed squared Frobenius changes from trained weights across all adapted matrices, using the stored weights\. We select cases where restoring intermediate or trailing components produces a larger edit norm than restoring dominant components\. Configuration comparisons use seed means of norms and forgetting reductions\. Dominant restoration still gives the largest mean forgetting reduction among all three restorations in 17/17 eligible configurations, comprising 8 Qwen, 6 Llama and 3 FLUX configurations\. At the checkpoint level, the count is 48/51, with 22/24 for Qwen, 18/18 for Llama and 8/9 for FLUX\. Each configuration or checkpoint is counted once, including when both alternatives have larger norms\. Requiring both alternatives to have larger norms gives 4/4 configurations and 10/12 checkpoints\. The eligible cases involve LoRA, DoRA and MiLoRA\. These counts show that the dominant\-component advantage can persist with smaller edits, so edit magnitude alone does not determine the ordering of forgetting reductions\.

### C\.6Native OFT orthogonality and polar projection

OFTv2 approximates the Cayley inverse with a truncated Neumann series\([Qiu et al\., 2025b](https://arxiv.org/html/2610.09122#bib.bib5)\)\. We measure‖𝑹​𝑹⊤−𝑰‖F\\\|\\bm\{R\}\\bm\{R\}^\{\\\!\\top\}\-\\bm\{I\}\\\|\_\{\\mathrm\{F\}\}, its maximum absolute entry, and‖𝑹−𝑹^‖F/‖𝑹‖F\\\|\\bm\{R\}\-\\widehat\{\\bm\{R\}\}\\\|\_\{\\mathrm\{F\}\}/\\\|\\bm\{R\}\\\|\_\{\\mathrm\{F\}\}, where𝑹^\\widehat\{\\bm\{R\}\}is the polar factor\([Higham, 1986](https://arxiv.org/html/2610.09122#bib.bib37)\)\. The relative distance expresses the deviation from orthogonality on a scale normalized by the transform norm\. These diagnostics cover 21 checkpoints each for Qwen and Llama, and 33 for FLUX\. The maximum native\-to\-polar distances are about 0\.154%, 0\.094% and 11\.0%, respectively \([Table20](https://arxiv.org/html/2610.09122#A3.T20)\)\. Task and retention evaluations use native transforms\. Polar projection measures their deviation from an exact orthogonal transform, clarifying the numerical precision of the geometric\-preservation comparison\.

Table 20:Native OFT\-transform diagnostics\. Maxima are taken over all matrices and checkpoints in each row\. Frobenius and entrywise maxima describe different norms\.

## Appendix DFLUX spectral interventions and numerical controls

### D\.1Complete six\-method FLUX interventions

The FLUX cat intervention study contains 18 checkpoints and 396 evaluations, covering all six methods at their largest tested budgets\. LR maximizes mean step\-700 validation DINO, with ties favoring lower LR\. Interventions use the final checkpoints and jointly replace all 80 adapted matrices\.

Table 21:FLUX intervention operating points selected by mean validation DINO\. Parameters are in millions\. Displacements pool all adapted matrices before normalization\. All edits act jointly on 80 matrices\.Table 22:FLUX value\-restoration controls\. DINO scores and paired differences are multiplied by 100\. The last column compares restoration with reconstruction through the same SVD/storage path\.#### Evaluation and numerical references\.

The BF16 evaluator generates512×512512\\times 512images with 20 sampling steps and guidance 3\.5\. Test DINO averages four held\-out image/prompt pairs with three generation repeats\. The drift bank has five class prompts and five general prompts, with general drift and CLIP using the latter\. Drift is\(1−DINO cosine\)/2\(1\-\\text\{DINO cosine\}\)/2relative to base\-generated images\.

Every change is paired within a checkpoint before computing the mean and sample SD\. Component effects use the same\-run merged trained model, restoration controls use reconstruction through the same SVD/storage path, and comparisons of retained task performance use full value restoration\. Same\-run references account for this difference\. All 396 evaluations succeed, with every restored matrix matched and within the recorded numerical tolerance\.

Figure 14:FLUX numerical controls\. The left panel compares value restoration with reconstruction\. The right panel compares the same\-run merged trained model with the earlier native\-adapter evaluation\. Diamonds and bars show means and sample SDs, with faint points for individual checkpoints\. Differences are in DINO scores×100\\times 100\.
#### Values and adapted bases\.

Value restoration nearly preserves fidelity for every method, whereas learned values on pretrained singular bases yield near\-base scores \([Table22](https://arxiv.org/html/2610.09122#A4.T22)\)\. This extends the main restoration finding to diffusion and points to the adapted bases as the principal source of task gains\. Spectral displacement concentrates in PiSSA’s dominant components and MiLoRA’s trailing components \([Figure15](https://arxiv.org/html/2610.09122#A4.F15)\)\.

Figure 15:FLUX spectral displacement\. The left panel shows the pooled spectral\-to\-weight displacement ratio with sample SDs\. The right panel shows mean shares of squared singular\-value displacement across component groups, with boundaries adjusted to pretrained spectral gaps\. Shares are computed from all measured spectral vectors\.

### D\.2Task performance and forgetting across spectral components

The dominant, intermediate and trailing component groups are approximate thirds adjusted to nearby pretrained spectral gaps\. Restore and keep operations replace complete rank\-one contributions, using pretrained singular values as in[Eq\.4](https://arxiv.org/html/2610.09122#S6.E4)\. The measured response includes these interactions between components\.

Restoring dominant components reduces general drift in every checkpoint and gives the largest mean reduction for each method\. It gives the largest individual reduction in 17 of 18 checkpoints\. MiLoRA seed zero has a slightly larger trailing\-component response\. Class\-prompt drift also decreases in every checkpoint, with dominant restoration giving the largest reduction\. Both drift measures therefore support the recurring dominant\-component response\. CLIP responses differ, with LoRA’s mean general\-prompt alignment decreasing\. OFT gains fidelity on average but loses fidelity in one checkpoint\.

Keeping only the adapted dominant components nearly preserves mean fidelity for LoRA, DoRA and PiSSA\. LoRA and DoRA reduce general drift in every checkpoint, whereas PiSSA retains essentially all its drift\. These responses show that similar retained task performance can accompany different retention outcomes\. Fidelity losses vary across seeds, including a larger DoRA loss\. We report DINO changes directly and use the 1 pp sufficiency rule only for language accuracy\. OFT/HRA have greater trained drift than PiSSA in these selected settings\.

Figure 16:FLUX component\-restoration responses\. The top row restores the named component group, and the bottom row keeps only that adapted group\. All changes use the same\-run trained model\. Lines show method means and faint points show individual checkpoints\.#### Summary\.

The FLUX results extend the singular\-value restoration and dominant\-component forgetting findings to image personalization\.

## Appendix EGeometry of additive updates

### E\.1Distance to a two\-sided orthogonal transformation

Writeq=min⁡\(m,n\)q=\\min\(m,n\)and letO⁡\(k\)O\(k\)denote the realk×kk\\times korthogonal group\. The orbit\{𝑸L𝑾0𝑸R⊤:𝑸L∈O\(m\),𝑸R∈O\(n\)\}\\\{\\bm\{Q\}\_\{L\}\\bm\{W\}\_\{0\}\\bm\{Q\}\_\{R\}^\{\\\!\\top\}:\\bm\{Q\}\_\{L\}\\in O\(m\),\\bm\{Q\}\_\{R\}\\in O\(n\)\\\}contains exactly the matrices with the same singular values as𝑾0\\bm\{W\}\_\{0\}\. It includes reflections as well as rotations\. A local smooth change of singular bases is generated by skew\-symmetric matrices\.

###### Proof of Proposition[4\.1](https://arxiv.org/html/2610.09122#S4.Thmproposition1)\.

Step 1\. Bound the distance from below\.Let𝑿=𝑸L​𝑾0​𝑸R⊤\\bm\{X\}=\\bm\{Q\}\_\{L\}\\bm\{W\}\_\{0\}\\bm\{Q\}\_\{R\}^\{\\\!\\top\}be any matrix in the orbit\. Orthogonal multiplication preserves singular values, so𝑿\\bm\{X\}has singular values𝒔0\\bm\{s\}\_\{0\}and‖𝑿‖F=‖𝒔0‖2\\\|\\bm\{X\}\\\|\_\{\\mathrm\{F\}\}=\\\|\\bm\{s\}\_\{0\}\\\|\_\{2\}\. Write⟨𝑨,𝑩⟩F=tr⁡\(𝑨⊤​𝑩\)\\langle\\bm\{A\},\\bm\{B\}\\rangle\_\{\\mathrm\{F\}\}=\\operatorname\{tr\}\(\\bm\{A\}^\{\\\!\\top\}\\bm\{B\}\)\. The singular\-value trace inequality\([Mirsky, 1975](https://arxiv.org/html/2610.09122#bib.bib36)\)bounds this inner product by the sum of the paired, decreasing singular values, giving

⟨𝑾∗,𝑿⟩F≤∑i=1qs∗,i​s0,i\.\\langle\\bm\{W\}\_\{\*\},\\bm\{X\}\\rangle\_\{\\mathrm\{F\}\}\\leq\\sum\_\{i=1\}^\{q\}s\_\{\*,i\}s\_\{0,i\}\.Expanding the squared distance therefore gives

‖𝑾∗−𝑿‖F2\\displaystyle\\\|\\bm\{W\}\_\{\*\}\-\\bm\{X\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=‖𝒔∗‖22\+‖𝒔0‖22−2​⟨𝑾∗,𝑿⟩F\\displaystyle=\\\|\\bm\{s\}\_\{\*\}\\\|\_\{2\}^\{2\}\+\\\|\\bm\{s\}\_\{0\}\\\|\_\{2\}^\{2\}\-2\\langle\\bm\{W\}\_\{\*\},\\bm\{X\}\\rangle\_\{\\mathrm\{F\}\}≥∑i=1q\(s∗,i−s0,i\)2=‖𝒔∗−𝒔0‖22\.\\displaystyle\\geq\\sum\_\{i=1\}^\{q\}\(s\_\{\*,i\}\-s\_\{0,i\}\)^\{2\}=\\\|\\bm\{s\}\_\{\*\}\-\\bm\{s\}\_\{0\}\\\|\_\{2\}^\{2\}\.Step 2\. Attain the bound by aligning the singular bases\.Take full SVDs𝑾j=𝑼j​𝚺j​𝑽j⊤\\bm\{W\}\_\{j\}=\\bm\{U\}\_\{j\}\\bm\{\\Sigma\}\_\{j\}\\bm\{V\}\_\{j\}^\{\\\!\\top\}forj∈\{0,∗\}j\\in\\\{0,\*\\\}\. Here𝑼j\\bm\{U\}\_\{j\}and𝑽j\\bm\{V\}\_\{j\}are square orthogonal completions of the thin bases, and𝚺j∈ℝm×n\\bm\{\\Sigma\}\_\{j\}\\in\\mathbb\{R\}^\{m\\times n\}is rectangular diagonal\. Choose𝑸L=𝑼∗​𝑼0⊤\\bm\{Q\}\_\{L\}=\\bm\{U\}\_\{\*\}\\bm\{U\}\_\{0\}^\{\\\!\\top\}and𝑸R=𝑽∗​𝑽0⊤\\bm\{Q\}\_\{R\}=\\bm\{V\}\_\{\*\}\\bm\{V\}\_\{0\}^\{\\\!\\top\}\. These matrices are orthogonal and give

𝑸L​𝑾0​𝑸R⊤=𝑼∗​𝚺0​𝑽∗⊤=𝑾R\.\\bm\{Q\}\_\{L\}\\bm\{W\}\_\{0\}\\bm\{Q\}\_\{R\}^\{\\\!\\top\}=\\bm\{U\}\_\{\*\}\\bm\{\\Sigma\}\_\{0\}\\bm\{V\}\_\{\*\}^\{\\\!\\top\}=\\bm\{W\}\_\{R\}\.Thus𝑾R\\bm\{W\}\_\{R\}keeps the base singular values while using the adapted singular bases\. Orthogonal invariance of the Frobenius norm yields

‖𝑾∗−𝑾R‖F=‖𝚺∗−𝚺0‖F=‖𝒔∗−𝒔0‖2\.\\\|\\bm\{W\}\_\{\*\}\-\\bm\{W\}\_\{R\}\\\|\_\{\\mathrm\{F\}\}=\\\|\\bm\{\\Sigma\}\_\{\*\}\-\\bm\{\\Sigma\}\_\{0\}\\\|\_\{\\mathrm\{F\}\}=\\\|\\bm\{s\}\_\{\*\}\-\\bm\{s\}\_\{0\}\\\|\_\{2\}\.This equals the lower bound from Step 1, proving the minimum\. No positivity or gap assumption was used, so the result also covers rank\-deficient matrices and repeated singular values\.

Step 3\. Relate the distance todS/dWd\_\{S\}/d\_\{W\}\.For nonzero base and update norms, the common normalization‖𝒔0‖2=‖𝑾0‖F\\\|\\bm\{s\}\_\{0\}\\\|\_\{2\}=\\\|\\bm\{W\}\_\{0\}\\\|\_\{\\mathrm\{F\}\}cancels, giving

dSdW=‖𝒔∗−𝒔0‖2‖𝑾∗−𝑾0‖F=‖𝑾∗−𝑾R‖F‖𝑾∗−𝑾0‖F\.\\frac\{d\_\{S\}\}\{d\_\{W\}\}=\\frac\{\\\|\\bm\{s\}\_\{\*\}\-\\bm\{s\}\_\{0\}\\\|\_\{2\}\}\{\\\|\\bm\{W\}\_\{\*\}\-\\bm\{W\}\_\{0\}\\\|\_\{\\mathrm\{F\}\}\}=\\frac\{\\\|\\bm\{W\}\_\{\*\}\-\\bm\{W\}\_\{R\}\\\|\_\{\\mathrm\{F\}\}\}\{\\\|\\bm\{W\}\_\{\*\}\-\\bm\{W\}\_\{0\}\\\|\_\{\\mathrm\{F\}\}\}\.For multiple matrices, summing the squared residuals before taking square roots gives the same interpretation for the pooled ratio\. ∎

### E\.2First\-order spectral and basis motion

###### Proof of Proposition[4\.2](https://arxiv.org/html/2610.09122#S4.Thmproposition2)\.

Step 1\. Express the derivative in singular coordinates\.The positive, distinct singular values allow a smooth local choice of singular vectors\([Stewart, 1990](https://arxiv.org/html/2610.09122#bib.bib24)\)\. Complete them to smooth square orthogonal bases and write𝑾0\+ϵ​𝑬=𝑼⁡\(ϵ\)​𝚺​\(ϵ\)​𝑽​\(ϵ\)⊤\\bm\{W\}\_\{0\}\+\\epsilon\\bm\{E\}=\\bm\{U\}\(\\epsilon\)\\bm\{\\Sigma\}\(\\epsilon\)\\bm\{V\}\(\\epsilon\)^\{\\\!\\top\}\. A dot denotes differentiation atϵ=0\\epsilon=0\. Define

𝑪=𝑼0⊤​𝑬​𝑽0,𝛀U=𝑼0⊤​𝑼˙,𝛀V=𝑽0⊤​𝑽˙\.\\bm\{C\}=\\bm\{U\}\_\{0\}^\{\\\!\\top\}\\bm\{E\}\\bm\{V\}\_\{0\},\\qquad\\bm\{\\Omega\}\_\{U\}=\\bm\{U\}\_\{0\}^\{\\\!\\top\}\\dot\{\\bm\{U\}\},\\qquad\\bm\{\\Omega\}\_\{V\}=\\bm\{V\}\_\{0\}^\{\\\!\\top\}\\dot\{\\bm\{V\}\}\.Differentiating𝑼⊤​𝑼=𝑰\\bm\{U\}^\{\\\!\\top\}\\bm\{U\}=\\bm\{I\}gives𝛀U⊤=−𝛀U\\bm\{\\Omega\}\_\{U\}^\{\\\!\\top\}=\-\\bm\{\\Omega\}\_\{U\}\. The same argument applies to𝛀V\\bm\{\\Omega\}\_\{V\}\. Now differentiate the SVD and multiply on the left by𝑼0⊤\\bm\{U\}\_\{0\}^\{\\\!\\top\}and on the right by𝑽0\\bm\{V\}\_\{0\}to obtain

𝑪=𝛀U​𝚺0\+𝚺˙−𝚺0​𝛀V,\\bm\{C\}=\\bm\{\\Omega\}\_\{U\}\\bm\{\\Sigma\}\_\{0\}\+\\dot\{\\bm\{\\Sigma\}\}\-\\bm\{\\Sigma\}\_\{0\}\\bm\{\\Omega\}\_\{V\},where𝚺˙\\dot\{\\bm\{\\Sigma\}\}is rectangular diagonal with entriess˙i\\dot\{s\}\_\{i\}\. Since skew\-symmetric matrices have zero diagonal, the two basis terms contribute nothing toCi​iC\_\{ii\}\. Hence

s˙i=Ci​i=𝒖0,i⊤​𝑬​𝒗0,i,1≤i≤q\.\\dot\{s\}\_\{i\}=C\_\{ii\}=\\bm\{u\}\_\{0,i\}^\{\\\!\\top\}\\bm\{E\}\\bm\{v\}\_\{0,i\},\\qquad 1\\leq i\\leq q\.
Step 2\. Show that the other coordinates describe basis motion\.Holding the singular values fixed leaves𝛀U​𝚺0−𝚺0​𝛀V\\bm\{\\Omega\}\_\{U\}\\bm\{\\Sigma\}\_\{0\}\-\\bm\{\\Sigma\}\_\{0\}\\bm\{\\Omega\}\_\{V\}, the tangent directions to the orbit in singular coordinates\. These directions have zero diagonal\. Conversely, fori≠j≤qi\\neq j\\leq q, the equations for any pair of off\-diagonal entries are

Ci​j=\(ΩU\)i​j​s0,j−s0,i​\(ΩV\)i​j,Cj​i=−\(ΩU\)i​j​s0,i\+s0,j​\(ΩV\)i​j\.C\_\{ij\}=\(\\Omega\_\{U\}\)\_\{ij\}s\_\{0,j\}\-s\_\{0,i\}\(\\Omega\_\{V\}\)\_\{ij\},\\qquad C\_\{ji\}=\-\(\\Omega\_\{U\}\)\_\{ij\}s\_\{0,i\}\+s\_\{0,j\}\(\\Omega\_\{V\}\)\_\{ij\}\.Solving this two\-variable system gives

\(ΩU\)i​j=s0,j​Ci​j\+s0,i​Cj​is0,j2−s0,i2,\(ΩV\)i​j=s0,i​Ci​j\+s0,j​Cj​is0,j2−s0,i2\.\(\\Omega\_\{U\}\)\_\{ij\}=\\frac\{s\_\{0,j\}C\_\{ij\}\+s\_\{0,i\}C\_\{ji\}\}\{s\_\{0,j\}^\{2\}\-s\_\{0,i\}^\{2\}\},\\qquad\(\\Omega\_\{V\}\)\_\{ij\}=\\frac\{s\_\{0,i\}C\_\{ij\}\+s\_\{0,j\}C\_\{ji\}\}\{s\_\{0,j\}^\{2\}\-s\_\{0,i\}^\{2\}\}\.The denominators are nonzero because the singular values are distinct\. Ifm\>nm\>n, the extra rows satisfyCa​j=\(ΩU\)a​j​s0,jC\_\{aj\}=\(\\Omega\_\{U\}\)\_\{aj\}s\_\{0,j\}fora\>qa\>q\. Ifn\>mn\>m, the extra columns satisfyCi​b=−s0,i​\(ΩV\)i​bC\_\{ib\}=\-s\_\{0,i\}\(\\Omega\_\{V\}\)\_\{ib\}forb\>qb\>q\. These equations are solvable because eachs0,i\>0s\_\{0,i\}\>0\. Thus every coordinate outside the firstqqdiagonal entries can be produced by a change of orthogonal bases\.

Step 3\. Separate the normal and tangent components\.The matrices𝒖0,i​𝒗0,i⊤\\bm\{u\}\_\{0,i\}\\bm\{v\}\_\{0,i\}^\{\\\!\\top\}are Frobenius\-orthonormal because their pairwise inner products are\(𝒖0,i⊤​𝒖0,j\)​\(𝒗0,i⊤​𝒗0,j\)=δi​j\(\\bm\{u\}\_\{0,i\}^\{\\\!\\top\}\\bm\{u\}\_\{0,j\}\)\(\\bm\{v\}\_\{0,i\}^\{\\\!\\top\}\\bm\{v\}\_\{0,j\}\)=\\delta\_\{ij\}\. By Step 2 they span the orthogonal complement of the tangent space\. The normal projection of𝑬\\bm\{E\}is therefore

𝑬N=∑i=1qs˙i​𝒖0,i​𝒗0,i⊤,𝑬T=𝑬−𝑬N\.\\bm\{E\}\_\{N\}=\\sum\_\{i=1\}^\{q\}\\dot\{s\}\_\{i\}\\bm\{u\}\_\{0,i\}\\bm\{v\}\_\{0,i\}^\{\\\!\\top\},\\qquad\\bm\{E\}\_\{T\}=\\bm\{E\}\-\\bm\{E\}\_\{N\}\.The remainder𝑬T\\bm\{E\}\_\{T\}is tangent and changes only the bases to first order\. Orthogonality gives

‖𝑬‖F2=‖𝑬N‖F2\+‖𝑬T‖F2=‖𝒔˙‖22\+‖𝑬T‖F2\.\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\\|\\bm\{E\}\_\{N\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\+\\\|\\bm\{E\}\_\{T\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\\|\\dot\{\\bm\{s\}\}\\\|\_\{2\}^\{2\}\+\\\|\\bm\{E\}\_\{T\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\.
Step 4\. Average the first\-order spectral energy\.Let𝒛i=vec⁡\(𝒖0,i​𝒗0,i⊤\)\\bm\{z\}\_\{i\}=\\operatorname\{vec\}\(\\bm\{u\}\_\{0,i\}\\bm\{v\}\_\{0,i\}^\{\\\!\\top\}\), so‖𝒛i‖2=1\\\|\\bm\{z\}\_\{i\}\\\|\_\{2\}=1ands˙i=𝒛i⊤​vec⁡\(𝑬\)\\dot\{s\}\_\{i\}=\\bm\{z\}\_\{i\}^\{\\\!\\top\}\\operatorname\{vec\}\(\\bm\{E\}\)\. The assumed second moment yields

𝔼⁡\[s˙i2\]=𝒛i⊤​𝔼​\[vec⁡\(𝑬\)​vec⁡\(𝑬\)⊤\]​𝒛i=‖𝒛i‖22m​n=1m​n\.\\mathbb\{E\}\[\\dot\{s\}\_\{i\}^\{2\}\]=\\bm\{z\}\_\{i\}^\{\\\!\\top\}\\mathbb\{E\}\[\\operatorname\{vec\}\(\\bm\{E\}\)\\operatorname\{vec\}\(\\bm\{E\}\)^\{\\\!\\top\}\]\\bm\{z\}\_\{i\}=\\frac\{\\\|\\bm\{z\}\_\{i\}\\\|\_\{2\}^\{2\}\}\{mn\}=\\frac\{1\}\{mn\}\.Summing theseqqequal contributions gives

𝔼​‖𝒔˙‖22=qm​n=1max⁡\(m,n\)\.\\mathbb\{E\}\\\|\\dot\{\\bm\{s\}\}\\\|\_\{2\}^\{2\}=\\frac\{q\}\{mn\}=\\frac\{1\}\{\\max\(m,n\)\}\.Since‖𝑬‖F=1\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}=1, Step 3 gives𝔼​‖𝑬T‖F2=1−1/max⁡\(m,n\)\\mathbb\{E\}\\\|\\bm\{E\}\_\{T\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=1\-1/\\max\(m,n\)for the complementary tangent energy\.

Step 5\. Pass from the derivative todS/dWd\_\{S\}/d\_\{W\}\.The base norms cancel as in Proposition[4\.1](https://arxiv.org/html/2610.09122#S4.Thmproposition1), and‖ϵ​𝑬‖F=\|ϵ\|\\\|\\epsilon\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}=\|\\epsilon\|\. Thus, forϵ≠0\\epsilon\\neq 0,

\(dSdW\)2=‖𝒔⁡\(𝑾0\+ϵ​𝑬\)−𝒔0‖22ϵ2\.\\left\(\\frac\{d\_\{S\}\}\{d\_\{W\}\}\\right\)^\{2\}=\\frac\{\\\|\\bm\{s\}\(\\bm\{W\}\_\{0\}\+\\epsilon\\bm\{E\}\)\-\\bm\{s\}\_\{0\}\\\|\_\{2\}^\{2\}\}\{\\epsilon^\{2\}\}\.With𝑾0\\bm\{W\}\_\{0\}fixed, positivity and the nonzero singular\-value gaps give the expansion

𝒔⁡\(𝑾0\+ϵ​𝑬\)−𝒔0=ϵ​𝒔˙\+O⁡\(ϵ2\),\\bm\{s\}\(\\bm\{W\}\_\{0\}\+\\epsilon\\bm\{E\}\)\-\\bm\{s\}\_\{0\}=\\epsilon\\dot\{\\bm\{s\}\}\+O\(\\epsilon^\{2\}\),whose remainder is bounded uniformly over‖𝑬‖F=1\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}=1\. Substitution into the ratio therefore gives\(dS/dW\)2=‖𝒔˙‖22\+O⁡\(\|ϵ\|\)\(d\_\{S\}/d\_\{W\}\)^\{2\}=\\\|\\dot\{\\bm\{s\}\}\\\|\_\{2\}^\{2\}\+O\(\|\\epsilon\|\)with a uniform error\. Taking expectations and applying Step 4 gives

limϵ→0𝔼⁡\[\(dSdW\)2\]=𝔼​‖𝒔˙‖22=1max⁡\(m,n\)\.\\lim\_\{\\epsilon\\to 0\}\\mathbb\{E\}\\left\[\\left\(\\frac\{d\_\{S\}\}\{d\_\{W\}\}\\right\)^\{2\}\\right\]=\\mathbb\{E\}\\\|\\dot\{\\bm\{s\}\}\\\|\_\{2\}^\{2\}=\\frac\{1\}\{\\max\(m,n\)\}\.Taking the square root proves the stated RMS limit\. ∎

###### Corollary E\.1\(Isotropic fixed\-rank perturbations\)\.

Let𝐖0\\bm\{W\}\_\{0\}satisfy Proposition[4\.2](https://arxiv.org/html/2610.09122#S4.Thmproposition2)and let𝐄=𝐏​𝚲​𝐐⊤/‖𝚲‖F\\bm\{E\}=\\bm\{P\}\\bm\{\\Lambda\}\\bm\{Q\}^\{\\\!\\top\}/\\\|\\bm\{\\Lambda\}\\\|\_\{\\mathrm\{F\}\}, where𝚲∈ℝr×r\\bm\{\\Lambda\}\\in\\mathbb\{R\}^\{r\\times r\}is fixed, nonzero diagonal and𝐏∈ℝm×r,𝐐∈ℝn×r\\bm\{P\}\\in\\mathbb\{R\}^\{m\\times r\},\\bm\{Q\}\\in\\mathbb\{R\}^\{n\\times r\}are independent uniform orthonormal frames, with1≤r≤q1\\leq r\\leq q\. Then[Equation3](https://arxiv.org/html/2610.09122#S4.E3)holds\.

###### Proof\.

Step 1\. Check the norm\.Since𝑷⊤​𝑷=𝑸⊤​𝑸=𝑰r\\bm\{P\}^\{\\\!\\top\}\\bm\{P\}=\\bm\{Q\}^\{\\\!\\top\}\\bm\{Q\}=\\bm\{I\}\_\{r\}, multiplication by these frames preserves the Frobenius norm of𝚲\\bm\{\\Lambda\}\. Hence‖𝑬‖F=1\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}=1\.

Step 2\. Compute the second moment\.Uniformity and independence of the frames make the distribution of𝑬\\bm\{E\}unchanged by any fixed left or right orthogonal multiplication\. In particular, a row sign flip shows𝔼⁡\[Ea​b​Ec​d\]=0\\mathbb\{E\}\[E\_\{ab\}E\_\{cd\}\]=0whena≠ca\\neq c\. Whena=ca=cbutb≠db\\neq d, a column sign flip gives the same conclusion\. Row and column permutations make all𝔼⁡\[Ea​b2\]\\mathbb\{E\}\[E\_\{ab\}^\{2\}\]equal\. Their sum is𝔼​‖𝑬‖F2=1\\mathbb\{E\}\\\|\\bm\{E\}\\\|\_\{\\mathrm\{F\}\}^\{2\}=1, so each equals1/\(m​n\)1/\(mn\)\. Consequently,

𝔼⁡\[vec⁡\(𝑬\)​vec⁡\(𝑬\)⊤\]=𝑰m​nm​n\.\\mathbb\{E\}\[\\operatorname\{vec\}\(\\bm\{E\}\)\\operatorname\{vec\}\(\\bm\{E\}\)^\{\\\!\\top\}\]=\\frac\{\\bm\{I\}\_\{mn\}\}\{mn\}\.Step 3\. Apply Proposition[4\.2](https://arxiv.org/html/2610.09122#S4.Thmproposition2)\.Steps 1 and 2 verify its two assumptions on𝑬\\bm\{E\}, so both conclusions in[Equation3](https://arxiv.org/html/2610.09122#S4.E3)follow\. ∎

###### Corollary E\.2\(Pooling matrices\)\.

For a finite collection of matrices satisfying the isotropic assumptions of Proposition[4\.2](https://arxiv.org/html/2610.09122#S4.Thmproposition2), let𝐖∗,j=𝐖0,j\+ϵ​aj​𝐄j\\bm\{W\}\_\{\*,j\}=\\bm\{W\}\_\{0,j\}\+\\epsilon a\_\{j\}\\bm\{E\}\_\{j\}, with fixedaj≥0a\_\{j\}\\geq 0and∑jaj2\>0\\sum\_\{j\}a\_\{j\}^\{2\}\>0\. The pooled displacements satisfy

limϵ→0𝔼⁡\[\(dSdW\)2\]=∑jaj2/max⁡\(mj,nj\)∑jaj2\.\\lim\_\{\\epsilon\\to 0\}\\mathbb\{E\}\\left\[\\left\(\\frac\{d\_\{S\}\}\{d\_\{W\}\}\\right\)^\{2\}\\right\]=\\frac\{\\sum\_\{j\}a\_\{j\}^\{2\}/\\max\(m\_\{j\},n\_\{j\}\)\}\{\\sum\_\{j\}a\_\{j\}^\{2\}\}\.

###### Proof\.

Step 1\. Write the pooled ratio\.Let𝒔∗,j\\bm\{s\}\_\{\*,j\}and𝒔0,j\\bm\{s\}\_\{0,j\}be the singular\-value vectors of the adapted and base matrices\. The common base\-norm normalization cancels, while‖𝑬j‖F=1\\\|\\bm\{E\}\_\{j\}\\\|\_\{\\mathrm\{F\}\}=1fixes each update norm\. Therefore

\(dSdW\)2=∑j‖𝒔∗,j−𝒔0,j‖22∑j‖ϵ​aj​𝑬j‖F2=∑j‖𝒔∗,j−𝒔0,j‖22ϵ2​∑jaj2\.\\left\(\\frac\{d\_\{S\}\}\{d\_\{W\}\}\\right\)^\{2\}=\\frac\{\\sum\_\{j\}\\\|\\bm\{s\}\_\{\*,j\}\-\\bm\{s\}\_\{0,j\}\\\|\_\{2\}^\{2\}\}\{\\sum\_\{j\}\\\|\\epsilon a\_\{j\}\\bm\{E\}\_\{j\}\\\|\_\{\\mathrm\{F\}\}^\{2\}\}=\\frac\{\\sum\_\{j\}\\\|\\bm\{s\}\_\{\*,j\}\-\\bm\{s\}\_\{0,j\}\\\|\_\{2\}^\{2\}\}\{\\epsilon^\{2\}\\sum\_\{j\}a\_\{j\}^\{2\}\}\.Step 2\. Expand each numerator\.Write𝒔˙j\\dot\{\\bm\{s\}\}\_\{j\}for the spectral derivative at𝑾0,j\\bm\{W\}\_\{0,j\}in direction𝑬j\\bm\{E\}\_\{j\}\. The uniform expansion from Proposition[4\.2](https://arxiv.org/html/2610.09122#S4.Thmproposition2)gives, for fixedaja\_\{j\},

𝔼​‖𝒔∗,j−𝒔0,j‖22ϵ2=aj2​𝔼​‖𝒔˙j‖22\+O⁡\(\|ϵ\|\)=aj2max⁡\(mj,nj\)\+O⁡\(\|ϵ\|\)\.\\frac\{\\mathbb\{E\}\\\|\\bm\{s\}\_\{\*,j\}\-\\bm\{s\}\_\{0,j\}\\\|\_\{2\}^\{2\}\}\{\\epsilon^\{2\}\}=a\_\{j\}^\{2\}\\mathbb\{E\}\\\|\\dot\{\\bm\{s\}\}\_\{j\}\\\|\_\{2\}^\{2\}\+O\(\|\\epsilon\|\)=\\frac\{a\_\{j\}^\{2\}\}\{\\max\(m\_\{j\},n\_\{j\}\)\}\+O\(\|\\epsilon\|\)\.Whenaj=0a\_\{j\}=0, the corresponding term is identically zero\.

Step 3\. Sum and take the limit\.There are finitely many matrices and∑jaj2\\sum\_\{j\}a\_\{j\}^\{2\}is fixed and positive\. Substituting Step 2 into Step 1 yields

limϵ→0𝔼⁡\[\(dSdW\)2\]=∑jaj2/max⁡\(mj,nj\)∑jaj2\.\\lim\_\{\\epsilon\\to 0\}\\mathbb\{E\}\\left\[\\left\(\\frac\{d\_\{S\}\}\{d\_\{W\}\}\\right\)^\{2\}\\right\]=\\frac\{\\sum\_\{j\}a\_\{j\}^\{2\}/\\max\(m\_\{j\},n\_\{j\}\)\}\{\\sum\_\{j\}a\_\{j\}^\{2\}\}\.Only linearity of expectation is used, so independence between matrices is unnecessary\. The result is an average of the individual squared\-ratio scales, weighted by their update energiesaj2a\_\{j\}^\{2\}\. ∎

Under the stated isotropic assumptions, the inverse\-dimension scale explains how an additive update can have small relative spectral displacement\. This motivates the empirical geometry measurements, while restoration tests the functional role of the spectral changes\.

## Appendix FReview of experimental settings in PEFT studies

#### Papers and comparisons\.

We review method papers associated with Hugging Face PEFT v0\.21\.0\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.09122#bib.bib48)\), using paper versions available by 26 September 2026\. Multiple supported entries from the same paper are counted once, and background citations are excluded\. The inventory contains 71 entries and 70 papers, with one entry lacking an identified method paper\. Sixty\-four papers have eligible main comparisons\. The other six concern training from scratch, contributions without adaptation training or results without a trained principal comparator outside auxiliary analyses\.

We read the papers, appendices, available supplements and relevant official experimental descriptions\. A comparison pairs the proposed method with its principal trained baseline for a stated pretrained model and training task\. Vanilla LoRA is preferred when present\. Several evaluations of the same adapted model remain one comparison, while separately trained tasks are distinguished\. Additional ablations do not establish controls for the main comparisons\. Capacity strata are retained within each comparison\.

#### Controls and evidence\.

[Table23](https://arxiv.org/html/2610.09122#A6.T23)defines the four controls\. Both methods must have the required evidence in the same comparison\. Comparable conditions are assessed separately for checkpoint, data, exposure, adapted modules and trainable budget\. Shared stopping rules can establish comparable exposure\. Parameter counts within 10% of the larger count, an explicit matched design or a controlled comparison across budgets establish budget comparability\. Equal rank alone is insufficient\. Intrinsic module differences and full fine\-tuning budgets are treated as inapplicable where direct matching is inappropriate\. A different budget can still support a useful efficiency comparison\.

Training runs may vary initialization, data order or sampled training examples\. Their uncertainty is distinguished from variation across generated samples, subjects, evaluation subsets or evaluation resamples\. Undefined error bars and dispersion pooled across training seeds and other units do not establish uncertainty attributable specifically to training\. A numerical search space does not establish equal tuning effort or separate validation selection\. An explicitly exhaustive design with one candidate can document disclosure\. Selection using the reported validation split is recorded separately from selection using an independent split\.

Two AI\-assisted source reviews were compared, and disagreements were revisited against the primary sources\. Differences in eligibility, reviewed versions, baseline attribution and experimental scope were reconciled using the definitions above\. Ambiguous or contradictory evidence remains unclear\. Relevant official descriptions provide supporting evidence, while generic defaults do not establish which procedures produced reported results\. AI\-assisted review can retain correlated errors\.

#### Aggregation and coverage\.

A paper documents a control if at least one eligible main comparison satisfies it\. An explicit departure requires evidence that every eligible comparison fails the definition\. Otherwise, the classification is unclear\. Inapplicable cases are excluded from the relevant denominator\. Missing information is never interpreted as evidence that a procedure was not performed\. For incomplete comparison coverage, known positives are retained, while universal claims are withheld unless a statement explicitly covers all main results\.

[Tables24](https://arxiv.org/html/2610.09122#A6.T24)and[25](https://arxiv.org/html/2610.09122#A6.T25)report coverage and domain summaries\. Sixteen papers have unresolved comparison coverage\. No reviewed comparison establishes all four controls together under these definitions\. Requiring three training runs retains the same nine replication positives, while requiring explicitly stated training randomness retains seven\. Excluding learning\-rate selection on the reported or an unnamed split leaves four selection positives\.

#### Scope of interpretation\.

Library membership defines a practical sample spanning several domains and research objectives\. It is not a representative sample of all PEFT research\. Domains overlap at paper level, and controls documented in different comparisons cannot be combined into a single qualifying experiment\. The cohort and validation\-selection definition differ from[Lee et al\. \(2026\)](https://arxiv.org/html/2610.09122#bib.bib30), so the percentages cannot be compared directly\. The review characterizes available evidence and does not estimate how undocumented procedures would change method rankings\.

Table 23:Definitions used in the review\. Evidence must cover both methods and the stated main comparison\. These are operational criteria for this review\.Table 24:Paper\-level coverage of experimental controls\. The first three columns partition the 64 eligible papers using the at least one comparison rule\. The final column requires established coverage of every applicable main comparison\. Unresolved coverage prevents a universal positive\.Table 25:Documented controls within each domain\. Entries give qualifying papers over applicable papers\. A paper can occur in several domains, so rows cannot be summed\. The decoder subset restricts comparisons to standard decoder adaptation against vanilla LoRA, with the exclusions described by Lee et al\. Its definitions and sampling frame still differ from theirs\.