参数高效微调的范畴论图谱：锥、轨道与规范

# A Categorical Atlas ofParameter-Efficient Fine-Tuning
Source: [https://arxiv.org/html/2610.12728](https://arxiv.org/html/2610.12728)
Xiaoyu LiUniversity of New South Walesxiaoyu\.li2@unsw\.edu\.auZhizhou ShaUniversity of Texas at Austinzhizhousha@utexas\.eduChiwun YangCity University of Hong Kongchristiannyang37@gmail\.comDai ShiUniversity of Cambridgeds2213@cam\.ac\.ukwith Claude Opus 5\.5 \(Anthropic\), in the spirit of Grothendieck’s rising sea

Technical report‡· October 2026 · Version 1

###### Abstract

Parameter\-efficient fine\-tuning has hundreds of methods, usually compared by parameter count and benchmark score\. We compare them as maps\. A pretrained checkpoint is a base point in weight space, and a weight\-space method is a smooth map from a small parameter space that starts at that point\. These maps form a slice category in which the frozen model is initial and full fine\-tuning terminal\. Three invariants of the map do most of the work\. Its image near the base point governs expressivity, through the first\-order image, the dimension, a min\-cut bound on rank and whether that point is smooth or an apex\. Its fibres, with the optimizer’s metric, govern training dynamics\. Base change moves the base point and covers splitting initialisations, quantisation, restarts and merging\. We prove, for example, that zero\-initialised LoRA, and Householder reflection adaptation at its default start under stated rank and parity conditions, begin at the apex of a cone; that no exact initialisation lets LoRA reach an update of rank above twice its own; that exactly orthogonal methods never change a weight’s singular values in exact arithmetic, however often they are merged; that under Adam with no epsilon, weight decay or clipping and one shared schedule, LoRA\+ from the standard zero initialisation follows exactly the trajectory of plain LoRA with its scale multiplied by the learning\-rate ratio; that Adam with shared hyperparameters respects only the signed permutations inside LoRA’s continuous symmetry group of full\-rank factors; and that the activation function decides when a scaling adapter folds into preceding weights\. We classify 382 methods from 138 papers by seven coordinates read off their formulas, with 403 simulation arrows and 485 obstructions\. Every numbered result is proved in full, and results due to others are credited where we re\-derive them\.

††footnotetext:The title pays homage to[Bronstein et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib44),*Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges*, whose programme of reading architectures through their symmetries this atlas extends to fine\-tuning\.‡‡footnotetext:This is a technical report rather than a journal or conference submission\. We share it, with its proofs, catalogue and interactive companion, as an open resource for everyone working on parameter\-efficient fine\-tuning, and we welcome corrections and suggestions for future versions\.###### Contents

1. [0Thesis](https://arxiv.org/html/2610.12728#S0)
2. [IThe pointed slice](https://arxiv.org/html/2610.12728#S1)
3. [IIThe image germ: expressivity](https://arxiv.org/html/2610.12728#S2)1. [II\.AInitialisations: pointings and strata](https://arxiv.org/html/2610.12728#S2.SS1) 2. [II\.BErlangen: invariants of group\-type methods](https://arxiv.org/html/2610.12728#S2.SS2) 3. [II\.CCut and volume](https://arxiv.org/html/2610.12728#S2.SS3)
4. [IIIFibres and metrics: dynamics](https://arxiv.org/html/2610.12728#S3)
5. [IVThe lens: tangent and cotangent methods](https://arxiv.org/html/2610.12728#S4)
6. [VBase change](https://arxiv.org/html/2610.12728#S5)
7. [VIExtensions and mergeability](https://arxiv.org/html/2610.12728#S6)
8. [VIIWorked examples](https://arxiv.org/html/2610.12728#S7)
9. [VIIIThe classification](https://arxiv.org/html/2610.12728#S8)1. [VIII\.ASeven axes, each decidable from the formula](https://arxiv.org/html/2610.12728#S8.SS1) 2. [VIII\.BThe periodic table](https://arxiv.org/html/2610.12728#S8.SS2) 3. [VIII\.CArrow rules: whenNNsimulatesMM](https://arxiv.org/html/2610.12728#S8.SS3)
10. [IXRosetta stone](https://arxiv.org/html/2610.12728#S9)
11. [XFalsifiable predictions](https://arxiv.org/html/2610.12728#S10)
12. [XIWhat is analogy, and what is open](https://arxiv.org/html/2610.12728#S11)
13. [XIIRelated work](https://arxiv.org/html/2610.12728#S12)
14. [XIIIConclusion](https://arxiv.org/html/2610.12728#S13)
15. [References](https://arxiv.org/html/2610.12728#bib)
16. [Technical appendix](https://arxiv.org/html/2610.12728#Ax1)
17. [AMathematical background](https://arxiv.org/html/2610.12728#A1)1. [Part A\. Categories without tears](https://arxiv.org/html/2610.12728#A1.SSx1) 2. [Part B\. Monoidal categories, string diagrams and lenses](https://arxiv.org/html/2610.12728#A1.SSx2) 3. [Part C\. Manifolds, germs and cones](https://arxiv.org/html/2610.12728#A1.SSx3) 4. [Part D\. Groups, symmetry and conservation](https://arxiv.org/html/2610.12728#A1.SSx4)
18. [BAuxiliary results](https://arxiv.org/html/2610.12728#A2)
19. [CDeferred proofs](https://arxiv.org/html/2610.12728#A3)1. [C\.1Remark I\.3](https://arxiv.org/html/2610.12728#A3.SS1) 2. [C\.2Observation I\.4](https://arxiv.org/html/2610.12728#A3.SS2) 3. [C\.3Definition I\.5](https://arxiv.org/html/2610.12728#A3.SS3) 4. [C\.4Observation I\.6](https://arxiv.org/html/2610.12728#A3.SS4) 5. [C\.5Observation I\.7](https://arxiv.org/html/2610.12728#A3.SS5) 6. [C\.6Proposition I\.8](https://arxiv.org/html/2610.12728#A3.SS6) 7. [C\.7Theorem II\.1](https://arxiv.org/html/2610.12728#A3.SS7) 8. [C\.8Theorem II\.2](https://arxiv.org/html/2610.12728#A3.SS8) 9. [C\.9Proposition II\.3](https://arxiv.org/html/2610.12728#A3.SS9) 10. [C\.10Lemma II\.4](https://arxiv.org/html/2610.12728#A3.SS10) 11. [C\.11Theorem II\.5](https://arxiv.org/html/2610.12728#A3.SS11) 12. [C\.12Corollary II\.6](https://arxiv.org/html/2610.12728#A3.SS12) 13. [C\.13Theorem II\.7](https://arxiv.org/html/2610.12728#A3.SS13) 14. [C\.14The LOFT example of Exposé II\.B](https://arxiv.org/html/2610.12728#A3.SS14) 15. [C\.15Proposition II\.8](https://arxiv.org/html/2610.12728#A3.SS15) 16. [C\.16Proposition II\.9](https://arxiv.org/html/2610.12728#A3.SS16) 17. [C\.17Theorem II\.10](https://arxiv.org/html/2610.12728#A3.SS17) 18. [C\.18Theorem II\.11](https://arxiv.org/html/2610.12728#A3.SS18) 19. [C\.19Proposition II\.12](https://arxiv.org/html/2610.12728#A3.SS19) 20. [C\.20Proposition II\.13](https://arxiv.org/html/2610.12728#A3.SS20) 21. [C\.21Definition II\.14](https://arxiv.org/html/2610.12728#A3.SS21) 22. [C\.22Observation II\.15](https://arxiv.org/html/2610.12728#A3.SS22) 23. [C\.23Proposition II\.16](https://arxiv.org/html/2610.12728#A3.SS23) 24. [C\.24Definition III\.1](https://arxiv.org/html/2610.12728#A3.SS24) 25. [C\.25Observation III\.2](https://arxiv.org/html/2610.12728#A3.SS25) 26. [C\.26Theorem III\.3](https://arxiv.org/html/2610.12728#A3.SS26) 27. [C\.27Proposition III\.4](https://arxiv.org/html/2610.12728#A3.SS27) 28. [C\.28Theorem III\.5](https://arxiv.org/html/2610.12728#A3.SS28) 29. [C\.29Corollary III\.6](https://arxiv.org/html/2610.12728#A3.SS29) 30. [C\.30Proposition III\.7](https://arxiv.org/html/2610.12728#A3.SS30) 31. [C\.31Proposition III\.8](https://arxiv.org/html/2610.12728#A3.SS31) 32. [C\.32Theorem III\.9](https://arxiv.org/html/2610.12728#A3.SS32) 33. [C\.33Theorem III\.10](https://arxiv.org/html/2610.12728#A3.SS33) 34. [C\.34Corollary III\.11](https://arxiv.org/html/2610.12728#A3.SS34) 35. [C\.35Theorem III\.12](https://arxiv.org/html/2610.12728#A3.SS35) 36. [C\.36Corollary III\.13](https://arxiv.org/html/2610.12728#A3.SS36) 37. [C\.37Definition IV\.1](https://arxiv.org/html/2610.12728#A3.SS37) 38. [C\.38Proposition IV\.2](https://arxiv.org/html/2610.12728#A3.SS38) 39. [C\.39Proposition IV\.3](https://arxiv.org/html/2610.12728#A3.SS39) 40. [C\.40Observation IV\.4](https://arxiv.org/html/2610.12728#A3.SS40) 41. [C\.41Definition V\.1](https://arxiv.org/html/2610.12728#A3.SS41) 42. [C\.42Proposition V\.2](https://arxiv.org/html/2610.12728#A3.SS42) 43. [C\.43Theorem V\.3](https://arxiv.org/html/2610.12728#A3.SS43) 44. [C\.44Proposition V\.4](https://arxiv.org/html/2610.12728#A3.SS44) 45. [C\.45Definition VI\.1](https://arxiv.org/html/2610.12728#A3.SS45) 46. [C\.46Lemma VI\.2](https://arxiv.org/html/2610.12728#A3.SS46) 47. [C\.47Theorem VI\.3](https://arxiv.org/html/2610.12728#A3.SS47) 48. [C\.48Proposition VI\.4](https://arxiv.org/html/2610.12728#A3.SS48) 49. [C\.49Theorem VI\.5](https://arxiv.org/html/2610.12728#A3.SS49)
20. [DThe catalogue](https://arxiv.org/html/2610.12728#A4)1. [D\.1Families and sources](https://arxiv.org/html/2610.12728#A4.SS1) 2. [D\.2Fact\-checking](https://arxiv.org/html/2610.12728#A4.SS2) 3. [D\.3The 2026 sweep and the authors’ addendum](https://arxiv.org/html/2610.12728#A4.SS3) 4. [D\.4Coordinates, arrows and obstructions](https://arxiv.org/html/2610.12728#A4.SS4) 5. [D\.5The complete tables](https://arxiv.org/html/2610.12728#A4.SS5)
21. [ENumerical checks and reproducibility](https://arxiv.org/html/2610.12728#A5)1. [E\.1The script](https://arxiv.org/html/2610.12728#A5.SS1) 2. [E\.2The checks, section by section](https://arxiv.org/html/2610.12728#A5.SS2) 3. [E\.3Numbers from other sources](https://arxiv.org/html/2610.12728#A5.SS3) 4. [E\.4Software](https://arxiv.org/html/2610.12728#A5.SS4) 5. [E\.5How the companion site and the figures compute what they show](https://arxiv.org/html/2610.12728#A5.SS5) 6. [E\.6The numbers behind the figures](https://arxiv.org/html/2610.12728#A5.SS6)
22. [FHow this paper and its website were made](https://arxiv.org/html/2610.12728#A6)1. [F\.1In brief](https://arxiv.org/html/2610.12728#A6.SS1) 2. [F\.2Who did what](https://arxiv.org/html/2610.12728#A6.SS2) 3. [F\.3Finding the material](https://arxiv.org/html/2610.12728#A6.SS3) 4. [F\.4From examples to a theory](https://arxiv.org/html/2610.12728#A6.SS4) 5. [F\.5How category theory unifies the field here](https://arxiv.org/html/2610.12728#A6.SS5) 6. [F\.6How the results were checked](https://arxiv.org/html/2610.12728#A6.SS6) 7. [F\.7Building the site and the paper](https://arxiv.org/html/2610.12728#A6.SS7) 8. [F\.8The film](https://arxiv.org/html/2610.12728#A6.SS8) 9. [F\.9Cost and interruptions](https://arxiv.org/html/2610.12728#A6.SS9) 10. [F\.10Reproducibility](https://arxiv.org/html/2610.12728#A6.SS10)

## EXPOSÉ 0Thesis

The right generality turns a hard theorem into a remark, and the remark into a design rule\. after Grothendieck

Parameter\-efficient fine\-tuning \(PEFT\) has hundreds of methods, usually compared by parameter count and benchmark score\. Those two numbers leave open what a method can reach, how it trains, and what happens to it when it is initialised differently, quantised, restarted or merged\. Each of them becomes a precise mathematical question once a method is treated as a map, and many then have exact answers\. This paper compares methods that way\.

Fine\-tuning moves a point\. A pretrained checkpointθ0\\theta\_\{0\}is a*base point*in weight spaceΘ\\Theta, and a weight\-space PEFT method is a*relative object*over it, namely a pointed smooth map

ρ:\(Q,q0\)⟶\(Θ,θ0\),ρ⁡\(q0\)=θ0,\\rho:\(Q,q\_\{0\}\)\\longrightarrow\(\\Theta,\\theta\_\{0\}\),\\qquad\\rho\(q\_\{0\}\)=\\theta\_\{0\},from a small spaceQQof trainable parameters into weight space\. These maps form the pointed slice category𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\)\. A morphismh:M→Nh:M\\to Nis a pointed smooth map withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}, read “NNsimulatesMM”\. The frozen model is initial and full fine\-tuning is terminal \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\), so every method pointed atθ0\\theta\_\{0\}sits in the interval between them\. Following Grothendieck’s relative point of view, the object of study is the morphismρ\\rhoitself\. Almost all the work is done by three invariants of the morphism, plus one further layer for architectures\. Figure[1](https://arxiv.org/html/2610.12728#S0.F1)draws the picture\.

The bicategory𝐏𝐚𝐫𝐚\\mathbf\{Para\}and the lens reading of backpropagation are due to[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\. The closest earlier unified frame for PEFT is that of[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165), who read adapters, prefix tuning and LoRA as modifications of specific hidden states\. Ours works on the weight side, where adapters and prefixes return as extensions of the network \([Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\)\.

Figure 1:Fine\-tuning moves a point\.Weight spaceΘ\\Thetais drawn as a chart around the pretrained pointθ0\\theta\_\{0\}; the grid is the lattice of a round\-to\-nearest quantiserκ\\kappathrough the zero weight00\(marked\+\+\)\. Solid curves are image germs of methods pointed atθ0\\theta\_\{0\}\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\), drawn darkest nearθ0\\theta\_\{0\}because only the germ there matters\. LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)is the coneθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}with its apex atθ0\\theta\_\{0\}\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\), shown as the rank\-≤1\\leq 1double conex2=y2\+z2x^\{2\}=y^\{2\}\+z^\{2\}of the symmetric2×22\\times 2slice in perspective\. OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)is the orbit ofθ0\\theta\_\{0\}under rotations, an arc of the circle about00throughθ0\\theta\_\{0\}, the rest of which is dotted\. \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)is a torus orbit, whose closure is the line through00andθ0\\theta\_\{0\}\([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\)\. BitFit\([Ben\-Zaken et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib27)\)moves along a coordinate line, parallel to the grid\. The dashed curves are not pointed atθ0\\theta\_\{0\}\. Prefix tuning\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\)extends the network and is unpointed \([Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\); drawn in weight space it is a curve that missesθ0\\theta\_\{0\}, at the distance marked by the dotted segment\. QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)is LoRA pointed at the nearest grid nodeκ​θ0\\kappa\\theta\_\{0\}, so its cone is LoRA’s cone translated by the defecte=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\(red arrow\), andθ0\\theta\_\{0\}is off its image unlessrank⁡e≤r\\rk e\\leq r\([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\.#### Three invariants and one further layer

The three invariants answer three practical questions: what a method can say, how it learns, and how it travels\.

##### The image germ atθ0\\theta\_\{0\}governs expressivity

This is what a method can say\. Nearθ0\\theta\_\{0\}, the weights a method reaches form a linear space, a cone, a group orbit, a torus saturation, or a meet of these\. Four data are read off this germ: the first\-order imageS1S\_\{1\}, the expressive dimensiond⁡\(M\)d\(M\), the min\-cut bound on rank \(the cut\-rank\), and the type of the base point, which is either a smooth point or an apex\. Zero\-initialised LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)starts at the apex of the coneθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, where first steps miss directions that the method reaches later; a PiSSA\-style split\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)starts at a smooth point\. Exposé[II](https://arxiv.org/html/2610.12728#S2)treats the germ in three parts: strata of initialisations \([II\.A](https://arxiv.org/html/2610.12728#S2.SS1)\), invariants of group\-type methods \([II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\), and cut and volume \([II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\)\.

##### The fibres, together with a metric, govern dynamics

This is how a method learns\. Many parameter values give the same weight, and the fibres ofρ\\rhoare gauge orbits; on LoRA’s full\-rank locus they are the orbits ofGLr\\mathrm\{GL\}\_\{r\}acting byg⋅\(B,A\)=\(B​g−1,g​A\)g\\cdot\(B,A\)=\(Bg^\{\-1\},gA\)\. Combined with the optimizer’s learning ratesΛ≻0\\Lambda\\succ 0, the parametrisation induces a cometric on weight space,

K=D​ρ​Λ​D​ρ⊤,K=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\},so isomorphic methods can train differently\. For LoRA with per\-block rates, the compact half of the gauge acts by isometries and the non\-compact half gives charges conserved under gradient flow\. The optimizer’s equivariance group decides what descends to the model \(Exposé[III](https://arxiv.org/html/2610.12728#S3)\)\. The lens structure of backpropagation sorts tangent\-side from cotangent\-side methods \(Exposé[IV](https://arxiv.org/html/2610.12728#S4)\)\. Through this lens, GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)with no weight decay and no full\-gradient clipping is one\-sided LoRA with a moving frozen factor and carried\-over optimizer state \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\); this identity is due to[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)\. This layer rests on work on factorised and linear networks: balancedness\([Du et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib100);[Arora et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib14)\), symmetries and conservation laws\([Kunin et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib230);[Zhao et al\., 2022b](https://arxiv.org/html/2610.12728#bib.bib580)\), and imbalance at initialisation\([Tarmoun et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib447);[Min et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib328)\)\.

##### Base change movesθ0\\theta\_\{0\}

This is how a method travels\. Initialisation by splitting, quantisation, restarts and merging all move the base point \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\)\. QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)is LoRA pointed at the quantised weightsκ​θ0\\kappa\\theta\_\{0\}, with defecte=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}, and its image missesθ0\\theta\_\{0\}wheneverrank⁡e\>r\\rk e\>r, which is the generic case \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\. Merge\-and\-restart with a fixed adapter shape reaches, in the limit, the orbit of the monoid generated by one cycle\. For ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)this means thatKKcycles reach exactly the updates of rank at mostmin⁡\(K​r,m,n\)\\min\(Kr,m,n\)\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.

##### Extensions and mergeability

The fourth layer is the one place where a method changes the networkffitself; the network otherwise enters only through its realisation map and the backpropagation lens \([Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\)\. Adapters, prefixes and ReFT first extendff\. Merging is then a lifting problem, certified by a local rewriting calculus of seven rules with side conditions, which gives sufficient conditions and box\-local obstructions \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\.

#### Contributions

Each result below is proved in Appendix[C](https://arxiv.org/html/2610.12728#A3), and the scope stated here is the scope of the proof\. Where a result, or a part of one, is due to others, the item says so\. We re\-derive such results in our language and add only their reading for fine\-tuning\.

1. 1\.A relative frame for PEFT\.Weight\-space methods are the objects of the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)\. Every such method lies in the interval \[frozen, full FT\], and “NNsimulatesMM” becomes an explicit pointed maphhwithρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\)\. Isomorphism invariants finer than the image, such asS1S\_\{1\}and the singular locus, separate initialisations that the image cannot \([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\. A categorical product produces a named method\. Freezing either frame of LoRA gives two one\-sided methods whose product is the sandwichR↦W0\+B0​R​A0R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}, which is LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\)in the SVD frames ofW0W\_\{0\}\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\)\. Image\-level meets, which are not products, organise others, such as HRA below\. The bicategory𝐏𝐚𝐫𝐚\\mathbf\{Para\}is that of[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85); the slice for PEFT and its invariants are new\.
2. 2\.No initialisation buys rank above2​r2r\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\. For an additive method, moving the starting point while subtracting a frozen residual keeps the method pointed atθ0\\theta\_\{0\}, and the union of all such re\-pointed images isθ0\\theta\_\{0\}plus the difference setC−CC\-Cof its update shape\. ForLoRAr\\mathrm\{LoRA\}\_\{r\}that difference set is the matrices of rank at mostmin⁡\(2​r,m,n\)\\min\(2r,m,n\)\. So no exact pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}, zero\-init or split \(PiSSA, OLoRA\([Büyükakyüz, 2024](https://arxiv.org/html/2610.12728#bib.bib47)\), CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\), LoRA\-GA\([Wang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib472)\)\), makes an update of rank above2​r2rreachable in any adapted matrix, and split initialisations attain2​r2rwhen2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\. The statement is elementary; the2​r2rbound underlies the PiSSA\-to\-LoRA conversion of[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324), implemented in Hugging Face \(HF\) PEFT\.
3. 3\.Three strata of initialisations, and two apexes\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3); Figure[3](https://arxiv.org/html/2610.12728#S2.F3)\)\. For LoRA of rankr<min⁡\(m,n\)r<\\min\(m,n\)on onem×nm\\times nweight there are three strata of exact full\-rank pointings:B0=0B\_\{0\}=0with full\-rankA0A\_\{0\}\(vanilla LoRA on linear layers, EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)\);A0=0A\_\{0\}=0with full\-rankB0B\_\{0\}\(Init\[B\]\([Hayou et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib158)\), HF’s default on embedding layers\); and a rank\-rrproductB0​A0B\_\{0\}A\_\{0\}subtracted from the frozen weight \(PiSSA, MiLoRA\([Wang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib465)\), OLoRA, CorDA, LoRA\-GA, Astra\([Liu et al\., 2026b](https://arxiv.org/html/2610.12728#bib.bib277)\)\)\. Two pointed LoRA methods in these strata are isomorphic over weight space exactly when they sharerow⁡A0\\operatorname\{row\}A\_\{0\},col⁡B0\\operatorname\{col\}B\_\{0\}orB0​A0B\_\{0\}A\_\{0\}respectively, so as a set the classes areGr⁡\(r,n\)⊔Gr⁡\(r,m\)⊔ℳr\\Gr\(r,n\)\\sqcup\\Gr\(r,m\)\\sqcup\\mathcal\{M\}\_\{r\}\. Zero\-factor pointings sit at the apex of the coneW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}and missr⁡\(n−r\)r\(n\-r\)orr⁡\(m−r\)r\(m\-r\)first\-order directions; split pointings sit at a smooth point, and neither image contains the other\. The classification is set\-level and in exact arithmetic\. Rank\-deficient pointings such as HF’s orthogonal init, frozen\-factor sub\-objects \(LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\), MiCA\([Rüdiger and Raschka, 2026](https://arxiv.org/html/2610.12728#bib.bib395)\)\) and quantised\-base defects \(QLoRA, LoftQ\([Li et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib255)\)\) lie outside the three strata\. LetW0W\_\{0\}have full column rank and letrrbe even with2≤r≤n−22\\leq r\\leq n\-2\. Then the image ofrr\-reflection HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)without Gram–Schmidt, HF PEFT’s default \(apply\_GS=False\), is exactly\{W0R:R∈SO\(n\),rank\(W0R−W0\)≤r\}\\\{W\_\{0\}R:R\\in\\mathrm\{SO\}\(n\),\\ \\rk\(W\_\{0\}R\-W\_\{0\}\)\\leq r\\\}\. This is the image\-level intersection ofLoRAr\\mathrm\{LoRA\}\_\{r\}with the rotation orbitW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), of dimensionr​n−r⁡\(r\+1\)/2rn\-r\(r\+1\)/2, and HRA is not the categorical product of the two\. At HF’s default paired initialisation \(u2​i−1=u2​iu\_\{2i\-1\}=u\_\{2i\}\),W0W\_\{0\}is a singular vertex of the image and the first\-order image has dimensionk​n−k⁡\(k\+1\)/2kn\-k\(k\+1\)/2withk=r/2k=r/2, so that initialisation is an apex like zero\-init LoRA\. Both theorems are new\.
4. 4\.Exactly orthogonal methods keep the spectrum\([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\. If every weight a method produces, including after merge\-and\-restart cycles, has the formW0​RW\_\{0\}RwithRRexactly orthogonal on the input side, then in exact arithmetic the neuron Gram matrixW​W⊤WW^\{\\top\}never changes\. Every neuron norm, every angle between neurons \(and with them the hyperspherical energy\) and every singular value ofW0W\_\{0\}stay fixed, however often the method is merged, and block OFT with a fixed partition also fixes each block Gram\. This covers exact\-Cayley OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)and block OFT, strict GOFT\([Ma et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib305)\)without its optional scaler, HRA, and ETHER\([Bini et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib36)\)as defined in its paper\. It does not cover variants with a per\-neuron or per\-input scale \(HF BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\), HF OFT 0\.14 to 0\.15, GOFT’s default code\), output\-side variants, or approximately orthogonal ones such as HF PEFT’s default Cayley–Neumann OFT, qGOFT and ETHER\+\. With small generators each Cayley–Neumann merge is a contraction, so repeated merging can only shrink neuron norms and singular values; low\-precision merges add rounding drift\. The invariance itself is elementary, and preserving hyperspherical energy is the motivation of OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\); new is the sorting of methods and implementations by whether they satisfy its hypothesis\.
5. 5\.Rank is a cut, dimension is a volume\([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1); Figure[4](https://arxiv.org/html/2610.12728#S2.F4)\)\. The rank of a map computed by a tensor network is at most the smallest product of wire dimensions across any cut separating inputs from outputs \(standard\)\. The expressive dimension is at most the budget minus the dimension of the generic gauge orbit, with equality when the gauge is locally transitive on generic fibres\. At equal budget, with2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\),r2≤min⁡\(m,n\)r^\{2\}\\leq\\min\(m,n\)and2​r2<m\+n−12r^\{2\}<m\+n\-1,LoHar,r\\mathrm\{LoHa\}\_\{r,r\}\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\)reaches at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}fewer dimensions thanLoRA2​r\\mathrm\{LoRA\}\_\{2r\}, aboutm\+nm\+nwhenr2≪\(m\+n\)/2r^\{2\}\\ll\(m\+n\)/2\. It buys rank up tor2r^\{2\}in exchange forr≥3r\\geq 3, and it is strictly dominated forr≤2r\\leq 2\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1, for exampler=64r=64on4096×40964096\\times 4096or LyCORIS\-styler=32r=32on 320\- to 640\-wide layers, the dimension bound no longer separates the two, and wherever it is attained, as the numerical checks find, LoHa matches or exceedsLoRA2​r\\mathrm\{LoRA\}\_\{2r\}in dimension as well as in rank\. GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\)\(k×kk\\times kblocks of rankr/kr/k, HF default hybrid\_r=0\) keeps LoRA’s budget and dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)and lifts the maximum rank tomin⁡\(k​r,m,n\)\\min\(kr,m,n\), but caps each block at rankr/kr/k; fork≥2k\\geq 2andk​r≤min⁡\(m,n\)kr\\leq\\min\(m,n\)neither image contains the other\. The census is new\.
6. 6\.Kronecker sums cut the other way\([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2); Figure[6](https://arxiv.org/html/2610.12728#S2.F6)\)\. Split the dimensions asm=m1​m2m=m\_\{1\}m\_\{2\}andn=n1​n2n=n\_\{1\}n\_\{2\}\. Akk\-term Kronecker sum∑iCi⊗Di\\sum\_\{i\}C\_\{i\}\\otimes D\_\{i\}is exactly a matrix of rank at mostkkafter the rearrangement of Van Loan and Pitsianis\([1993](https://arxiv.org/html/2610.12728#bib.bib459)\), so unconstrained Kronecker methods are LoRA across a different cut of the same four\-leg tensor\. With equal square splits of ad×dd\\times dweight, a generic rank\-one update has the largest Kronecker rank,dd, while the identity has Kronecker rank one\. Hence zero\-initLoRAr\\mathrm\{LoRA\}\_\{r\}and unconstrainedkk\-term Kronecker sums, forr,k<dr,k<d, have incomparable images\. LoKr\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\)updatesC⊗B​AC\\otimes BAwith inner rankr′≤min⁡\(m2,n2\)r^\{\\prime\}\\leq\\min\(m\_\{2\},n\_\{2\}\)have rank at mostmin⁡\(m1,n1\)​r′\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}, attained, so LoKr’s image lies inside that of zero\-initLoRAr\\mathrm\{LoRA\}\_\{r\}exactly whenr≥min⁡\(m1,n1\)​r′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}, and is incomparable with it otherwise whenmin⁡\(m1,m2\)​min⁡\(n1,n2\)\>1\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\)\>1\. The linear algebra is standard \(the rearrangement and the operator\-Schmidt decomposition\); the PEFT reading is new\.
7. 7\.A conserved charge\([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1),[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6); Figure[7](https://arxiv.org/html/2610.12728#S3.F7)\)\. With per\-block rates, LoRA’s charge Φ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}is conserved by every continuous\-time updateq˙=−Λ​D​ρq⊤​γ​\(t\)\\dot\{q\}=\-\\Lambda D\\rho\_\{q\}^\{\\top\}\\gamma\(t\)with constantΛ≻0\\Lambda\\succ 0, gradient or not, including minibatches, global\-norm clipping and DoRA’s\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\)detached norm\. The compact half of the gauge givesΛ\\Lambda\-isometries that leaveKKunchanged, so gauge\-related initialisations give identical weight trajectories whenever the signal depends onqqonly through\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\)\. With per\-rank\-channel rates, off\-diagonal charges and isometries exist exactly between channels with equal productsηB,j​ηA,j\\eta\_\{B,j\}\\eta\_\{A,j\}\. Discrete steps make the charge drift, with relative drift of second order in the step size, and Adam and momentum are not covered\. The law is a mild extension of known results:[Zhao et al\. \(2022b, Prop\. 5\.1\)](https://arxiv.org/html/2610.12728#bib.bib580)and[Marcotte et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib318), after the balancedness principle of[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100)and[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)and the symmetry analysis of[Kunin et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib230)\. New here is only the remark thatγ\\gammaneed not be a gradient\. Under an isotropic charge the weight dynamics have a closed form, with a crossover scale below which zero\-BBLoRA behaves like LoRA\-FA; this closed form is due to[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447)and[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328), and the PEFT reading is ours\.
8. 8\.LoRA\+ isα\\alphain disguise\([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7); Figure[8](https://arxiv.org/html/2610.12728#S3.F8)\)\. For zero\-init LoRA under Adam withε=0\\varepsilon=0, no weight decay or clipping, one learning\-rate schedule for both groups, and the same data order and dropout masks, LoRA\+\([Hayou et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib159)\)with ratioλ\\lambdafollows, in exact arithmetic, the same weight trajectory as LoRA withα\\alphamultiplied byλ\\lambda\. Per\-blockε\\varepsilonand decoupled weight decay keep the statement once they are rescaled too, and under SGD the factor isλ\\sqrt\{\\lambda\}\. At a fixed rank, rsLoRA\([Kalajdzievski, 2023](https://arxiv.org/html/2610.12728#bib.bib204)\)is LoRA with a rescaledα\\alphaorBBlearning rate; howα\\alphashould scale with rank is a separate question\. Both statements are instances of a hyperparameter torus: for a method whose update issstimes a multihomogeneous function of its parameter blocks, rescaling each block’s initial scale and learning rate together with the overall scaless\(and, under Adam, the per\-blockε\\varepsilon, decoupled decay and clipping thresholds\) leaves the weight trajectory exactly unchanged under Adam, given the same data order and dropout masks; under SGD the same holds with learning rates scaled by the square of the block factor\. The LoRA/Adam case is due to[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401); the general multihomogeneous torus and the SGD law are new\.
9. 9\.Adam sees a finite shadow of the gauge\([Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3); Figure[9](https://arxiv.org/html/2610.12728#S3.F9)\)\. Gradient descent is natural exactly where cometrics agree, which for LoRA’s linear gauge with block\-scalar learning rates, at points whereAAhas full row rank andr<nr<n, means exactly alongO⁡\(r\)\\mathrm\{O\}\(r\)\. On LoRA’s full\-rank locus with channel\-uniform hyperparameters, the largest subgroup ofGLr\\mathrm\{GL\}\_\{r\}under which the parameter update is equivariant isO⁡\(r\)\\mathrm\{O\}\(r\)for SGD, with or without momentum, and the signed permutationsBrB\_\{r\}, of order2r​r\!2^\{r\}r\!, for Adam and AdamW with anyβ\\beta’s, anyε≥0\\varepsilon\\geq 0and decoupled decay\. Positive diagonal rescalings become symmetries of Adam only jointly with per\-rank\-channel rescaling of the learning rates \(and ofε\\varepsilonand learning\-rate\-coupled decay when these are nonzero\), and only without global\-norm clipping\. Undamped scaled GD \(Riemannian preconditioning\) and LoRA\-RITE\([Yen et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib533)\)are equivariant under all ofGLr\\mathrm\{GL\}\_\{r\}; damping reduces scaled GD toO⁡\(r\)\\mathrm\{O\}\(r\), and wrapping it in Adam reduces it toBrB\_\{r\}\. Combination rules must be gauge\-natural as well\. A rule for combining LoRA factors is a well\-defined function of the adapters exactly when it is constant on the factorisation fibres, which task arithmetic on the products satisfies and post\-hoc product\-of\-sums rules such as LoraHub\([Huang et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib180)\)do not \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\. The hierarchy is new; scaled GD is due to[Tong et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib452)and, for LoRA,[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546)\.
10. 10\.GaLore is one\-sided LoRA with a moving frame\([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\. If an adapter is linear in its trainable tensor \(W=Wk\+ℒ​CW=W\_\{k\}\+\\mathcal\{L\}Con the current baseWkW\_\{k\}, withC0=0C\_\{0\}=0\) and the optimizer depends only on the gradients it receives \(SGD or momentum, Adam without weight decay, Adafactor without parameter scaling, no global\-norm clipping, the same implementation on both sides\), then trainingCCand merging gives exactly the same weights as running that optimizer on the compressed gradientℒ∗​∇W​L\\mathcal\{L\}^\{\*\}\\nabla\_\{W\}Land mapping back withℒ\\mathcal\{L\}\. In particular, GaLore with weight decay00is exactly one\-sided LoRA whose frozen factor is GaLore’s projector matrix, merged and re\-initialised every period with the trainable factor’s optimizer state carried over\. Decoupled weight decay, parameter\-scaled Adafactor, trust\-ratio optimizers and full\-gradient clipping break the identity, and Flora, which compresses only the momentum, is not covered\. Known:[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)for GaLore, and[Hao et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib154)for random projections\.
11. 11\.Base change has explicit defects and closures\([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3); Figure[10](https://arxiv.org/html/2610.12728#S5.F10)\)\. Quantising the frozen base moves the base point\. At initialisation QLoRA misses the pretrained weightWWby the generically full\-rank errore=κ​W−We=\\kappa W\-W, and it can reachWWonly ifrank⁡e≤r\\rk e\\leq r\. Splitting or error\-fitting initialisations change this defect in an order\-dependent way: split\-then\-quantise \(QPiSSA\) leavesκ⁡\(Wres\)−Wres\\kappa\(W\_\{\\mathrm\{res\}\}\)\-W\_\{\\mathrm\{res\}\}, and one\-step LoftQ leavese−s​SVDr​\(e\)e\-s\\,\\mathrm\{SVD\}\_\{r\}\(e\), which iseeminus its best rank\-rrapproximation only when the LoRA scales=α/rs=\\alpha/ris11, because HF’s LoftQ initialisation ignores the scale\. AfterKKmerge\-and\-restart cycles with a fixed shape, the reachable set isθ0\+C\+K\\theta\_\{0\}\+C^\{\+K\}for an additive shapeCC, orSK​θ0S^\{K\}\\theta\_\{0\}for a setSSof group elements containing the identity\. Methods whose shape is a fixed subspace, subgroup or submonoid gain nothing, and ReLoRA reaches exactlyW0\+\{rank≤min\(Kr,m,n\)\}W\_\{0\}\+\\\{\\rk\\leq\\min\(Kr,m,n\)\\\}\. Exact\-Cayley block OFT stays inW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\), up to closure, unless its block hypergraph is connected, which BOFT’s butterflies achieve only at full depth, and re\-HRA reaches\(W0\+ℳ≤K​r\)∩W0⋅SO⁡\(n\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq Kr\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\. New\.
12. 12\.The activation decides the merge\([Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4),[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6); Figure[12](https://arxiv.org/html/2610.12728#S6.F12)\)\. A few exact rewrite rules give sufficient conditions for folding a scaling or affine PEFT edit into existing weights with no change in output\. The feed\-forward scale of \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)always folds intoWdownW\_\{\\mathrm\{down\}\}\. It folds intoWupW\_\{\\mathrm\{up\}\}under SwiGLU or GeGLU for every scale and under ReLU for nonnegative ones, while under GELU only entries00and11fold\. Position\-selective edits \(ReFT\([Wu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib505)\), Activated LoRA\([Greenewald et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib136)\)\) and nonlinear adapters are not box\-locally mergeable, and exactly merging a generic QLoRA or LoftQ update leaves the quantised format, with QA\-LoRA\([Xu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib514)\)as the exception\. For attention, a prefix\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\)acts as an input\-gated parallel adapter\([He et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib165)\)that rescales the content attention weights within its layer and never reorders them\([Petrov et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib364)\), while LLaMA\-Adapter’s\([Zhang et al\., 2023h](https://arxiv.org/html/2610.12728#bib.bib562)\)separate softmax with a zero\-initialised gate leaves those weights unchanged\. The rules are elementary, and the merge intoWdownW\_\{\\mathrm\{down\}\}is noted by[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275); SmoothQuant\([Xiao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib508)\)and AWQ\([Lin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib268)\)already migrate scales through activations and meet the GELU obstruction\. New is only the packaging as a merge calculus for PEFT, with explicit side conditions\. The calculus gives sufficient conditions and box\-local obstructions; deciding mergeability of a whole network lies outside its scope\.
13. 13\.An atlas\(Exposés[VII](https://arxiv.org/html/2610.12728#S7)–[IX](https://arxiv.org/html/2610.12728#S9), Appendix[D](https://arxiv.org/html/2610.12728#A4); Figures[13](https://arxiv.org/html/2610.12728#S8.F13)and[14](https://arxiv.org/html/2610.12728#S8.F14)\)\. Fourteen methods are worked through the three invariants\. Seven coordinates, each read off a method’s formula \(kind, shape of the image germ, base point, gauge, covariance group, merge type and rebasing closure\), classify 382 methods from 138 papers, 82 of them in HF PEFT\. The atlas records 403 simulation arrows between methods, each exhibited by an explicit maphhwithρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}, and 485 obstructions, each naming the test that rules an arrow out\. The 142 core methods form the periodic table and the category drawn in Figure[14](https://arxiv.org/html/2610.12728#S8.F14)\. A Rosetta stone translates between the PEFT and the categorical vocabulary\.
14. 14\.Predictions and limits\(Exposés[X](https://arxiv.org/html/2610.12728#S10)and[XI](https://arxiv.org/html/2610.12728#S11)\)\. The theory makes falsifiable predictions, each with its optimizer scope, and states plainly which of its words are analogies and which problems are open\.

Some further results are classical or standard and carry their credit in their headings: the complete invariants of the basic actions \([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\), the Kempf–Ness slice of the LoRA gauge \([Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5);[Kempf and Ness, 1979](https://arxiv.org/html/2610.12728#bib.bib209)\) and the privileged basis fixed by an activation \([Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\)\.

##### Where the category theory earns its keep

There are seven places\.

1. 1\.The pointed slice gives the interval \[frozen, full FT\] for weight\-space methods and makes “simulation” precise \(Exposé[I](https://arxiv.org/html/2610.12728#S1)\)\.
2. 2\.Isomorphism invariants finer than the image \(S1S\_\{1\}, the singular locus\) classify initialisations \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\.
3. 3\.A categorical product produces a named method \(LoRA\-XS,[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\), and image\-level meets organise others \(HRA,[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\.
4. 4\.Base change separates function\-preserving reparametrisations, which act on the slice, from maps that move the base point \(quantisers\)\. The resulting pointing defects are elementary, but the slice makes their order dependence explicit \([Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6),[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\.
5. 5\.The lens 2\-functor𝐏𝐚𝐫𝐚⁡\(𝖱\)\\mathbf\{Para\}\(\\mathsf\{R\}\)separates tangent\-side from cotangent\-side methods\. The degree\-one collapse shows that every cotangent method whose optimizer reads only its gradients \(no weight decay, no parameter scaling, no full\-gradient clipping\) is, period by period, a forward linear method on a moving base \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.
6. 6\.Naturality: gradient descent is natural exactly where cometrics agree \(for LoRA’s linear gauge with block\-scalar rates, alongO⁡\(r\)\\mathrm\{O\}\(r\)whereAAhas full row rank andr<nr<n\), and combination rules must be gauge\-natural \([Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.
7. 7\.Symmetric monoidal structure turns rank into a min\-cut \([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\)\.

The rest is real algebraic geometry, invariant theory, Lie generation and gradient\-flow ODEs, and we say so where it happens\. True but idle constructions are omitted: coends for the rank filtration, a saturation monad, sheaf language for locality, the Grothendieck construction∫𝐌𝐞𝐭𝐡→𝐏𝐍𝐞𝐭\\int\\mathbf\{Meth\}\\to\\mathbf\{PNet\}, and coKleisli 2\-cells for mixtures\.

#### How the results were checked

Each of the 41 numbered results was checked by adversarial model referees \(Appendix[F](https://arxiv.org/html/2610.12728#A6)\) against its proof, against the literature, and against the methods as published and as coded, including the source of HF PEFT at commit 85be3f3 of 30 September 2026\. Of these, 14 stood as first written and 27 were repaired\. A second round, with one mathematical and one faithfulness referee for each result, re\-examined 26 of the repaired results and narrowed many of them\. Its objections mostly concerned a side clause: a missing hypothesis, a remark that went past its proof, or a description of a method that matched only one implementation\. For example, HF PEFT’s LoftQ initialiser turned out to ignore the LoRA scaleα/r\\alpha/r\([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\. Only the narrowed forms are stated in this paper\. Some results are new, such as the HRA apex; many are known and re\-derived with credit, as the contributions above record\.

#### How to read this paper

##### Plan

Exposé[I](https://arxiv.org/html/2610.12728#S1)sets up the pointed slice and its invariants\. Exposé[II](https://arxiv.org/html/2610.12728#S2)studies the image germ: initialisations and their strata \([II\.A](https://arxiv.org/html/2610.12728#S2.SS1)\), group\-type methods \([II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\), and cut and volume \([II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\)\. Exposé[III](https://arxiv.org/html/2610.12728#S3)treats fibres, metrics and training dynamics, Exposé[IV](https://arxiv.org/html/2610.12728#S4)the backpropagation lens, Exposé[V](https://arxiv.org/html/2610.12728#S5)base change, and Exposé[VI](https://arxiv.org/html/2610.12728#S6)extensions and mergeability\. Exposé[VII](https://arxiv.org/html/2610.12728#S7)works through fourteen methods, Exposé[VIII](https://arxiv.org/html/2610.12728#S8)gives the classification and Exposé[IX](https://arxiv.org/html/2610.12728#S9)the Rosetta stone\. Exposé[X](https://arxiv.org/html/2610.12728#S10)lists falsifiable predictions, Exposé[XI](https://arxiv.org/html/2610.12728#S11)separates analogy from theorem and collects open problems, Exposé[XII](https://arxiv.org/html/2610.12728#S12)discusses related work and Exposé[XIII](https://arxiv.org/html/2610.12728#S13)concludes\.

##### Labels

Table[1](https://arxiv.org/html/2610.12728#S0.T1)lists the labels\. Every numbered item keeps the number it has on the companion site\. Definitions, remarks and results share one counter within each exposé, so Definition I\.2 is followed by Remark I\.3 and Observation I\.4\.

Table 1:How to read the labels\. The coloured rule is the one that marks the environment in the text\.
##### Conventions

These carry real weight in the arguments\. Table[2](https://arxiv.org/html/2610.12728#S0.T2)collects the notation\.

- •*Linear box\.*A linear box computesy=W​xy=WxwithW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\},m=doutm=d\_\{\\mathrm\{out\}\}andn=dinn=d\_\{\\mathrm\{in\}\}\. RowiiofWWis output neuronii\.
- •*Two Gram matrices, two sides\.*Kout​\(W\)=W​W⊤K\_\{\\mathrm\{out\}\}\(W\)=WW^\{\\top\}is the neuron Gram matrix andKin​\(W\)=W⊤​WK\_\{\\mathrm\{in\}\}\(W\)=W^\{\\top\}Wthe input Gram matrix\. A right multiplicationW↦W​RW\\mapsto WRwithR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)\(the*input side*\) fixesKoutK\_\{\\mathrm\{out\}\}\. A left multiplicationW↦P​WW\\mapsto PWfixesKinK\_\{\\mathrm\{in\}\}\. Papers that storeW⊤W^\{\\top\}swap the words*input*and*output*; we translate every method into the convention above\.
- •*Ambient category\.*Maps and dynamics live in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\. Images are taken in𝐒𝐞𝐭\\mathbf\{Set\}\. Every image we meet is semialgebraic, or definable inℝexp\\mathbb\{R\}\_\{\\exp\}when softmax is involved, so dimensions, smooth points and tangent cones are well defined\.

Table 2:Notation\.
##### Proofs and appendices

The main text contains no proofs\. Complete proofs of all numbered results are in Appendix[C](https://arxiv.org/html/2610.12728#A3), in the same order as the statements, one subsection per result\. External theorems that the proofs use are quoted in Appendix[B](https://arxiv.org/html/2610.12728#A2)\. Appendix[A](https://arxiv.org/html/2610.12728#A1)collects the background the paper assumes, on categories, monoidal categories and lenses, manifolds, germs and cones, and groups, symmetry and conservation; its entries keep the numbers A\.1–D\.34 they carry on the companion site, and readers new to category theory or Lie groups can start there\. Appendix[D](https://arxiv.org/html/2610.12728#A4)describes the catalogue and Appendix[E](https://arxiv.org/html/2610.12728#A5)the numerical checks\.

##### The companion site

This paper is the long\-form version of the interactive site at[https://xiaoyulics\.com/peft\-atlas/](https://xiaoyulics.com/peft-atlas/)\. The two share every number, so “Theorem III\.10” names the same statement here, on the site and in the ancillary file propositions\.json\. The site states each result with its proof beneath it and has 20 interactive figures that recompute their numbers in the browser, among them an orbit explorer for LoRA\+, a Gram lab for orthogonal methods, a gauge lab, Noether hyperbolas and a merge game\. This paper gives the complete text, collects the proofs in Appendix[C](https://arxiv.org/html/2610.12728#A3), and replaces the interactive figures by static ones\. The catalogue, the arrows and obstructions, the numbered results with their review status, and the numerical checks accompany this paper as data files\. When citing a result, please give its number; for a result marked known, please also cite the authors named in its heading, to whom the result itself is due\.

## EXPOSÉ IThe pointed slice

We model a weight\-space PEFT method as a pointed map into the pretrained weights\. The frozen model and full fine\-tuning sit at the two ends of an interval, and sums and products of such maps already give methods in use\. This exposé sets up the category and its first invariants\. The theorems begin when we look closely at one image near one point, which is where Exposé[II](https://arxiv.org/html/2610.12728#S2)starts\.

#### Pretrained models and methods

A pretrained network is a function with a parameter slot and a chosen value in it\. Its natural home is the bicategory of parametrised maps \([Background B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

Definition I\.1\(Para; pretrained model\)\. For a cartesian category𝒞\\mathcal\{C\}, the bicategory𝐏𝐚𝐫𝐚⁡\(𝒞\)\\mathbf\{Para\}\(\\mathcal\{C\}\)is defined as follows\([Fong et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib114);[Cruttwell et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib85)\):•its 1\-cellsX→YX\\to Yare pairs\(P,f:P×X→Y\)\(P,\\,f:P\\times X\\to Y\);•its 2\-cells\(P,f\)⇒\(Q,g\)\(P,f\)\\Rightarrow\(Q,g\)are mapsr:Q→Pr:Q\\to Pwithg=f∘\(r×X\)g=f\\circ\(r\\times X\)\.Apretrained modelis a pointed 1\-cellF=\(Θ,θ0,f\)F=\(\\Theta,\\theta\_\{0\},f\)of𝐏𝐚𝐫𝐚⁡\(𝐃𝐢𝐟𝐟\)\\mathbf\{Para\}\(\\mathbf\{Diff\}\)\. ItsrealisationisΦf:Θ→C∞​\(X,Y\)\\Phi\_\{f\}:\\Theta\\to C^\{\\infty\}\(X,Y\),θ↦f⁡\(θ,−\)\\theta\\mapsto f\(\\theta,\-\)\. Fine\-tuning is a pathθ⁡\(t\)\\theta\(t\)withθ⁡\(0\)=θ0\\theta\(0\)=\\theta\_\{0\}\.

𝐏𝐚𝐫𝐚\\mathbf\{Para\}and the lens reading of backpropagation are due to[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85);[Gavranović et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib127)develop a broader programme on the same foundations\.

Definition I\.2\(methods and simulations\)\. A weight\-spacePEFT methodatθ0\\theta\_\{0\}is a pointed smooth mapM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\),ρ:Q→Θ\\rho:Q\\to\\Theta, withρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}\. UsuallydimQ≪dimΘ\\dim Q\\ll\\dim\\Theta, andθ0\\theta\_\{0\}\(or a splitting of it\) is frozen insideρ\\rho\. These form the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\):=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\.\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\):=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)\.•Amorphismh:M→Nh:M\\to Nis a pointed smooth map withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Read it as “NNsimulatesMM”\. ThenNNreproduces every weight thatMMreaches, together with its parameter path\. A*weak*morphism drops the conditionh⁡\(q0\)=q0′h\(q\_\{0\}\)=q\_\{0\}^\{\\prime\}\.•Thebudgetis\|M\|=dimQ\\lvert M\\rvert=\\dim Q\.•MMisadditiveifρ=θ0\+δ\\rho=\\theta\_\{0\}\+\\deltawithδ⁡\(q0\)=0\\delta\(q\_\{0\}\)=0andδ\\deltaindependent ofθ0\\theta\_\{0\}\.•MMis ofaction typeifρ⁡\(q\)=γ⁡\(q\)⋅θ0\\rho\(q\)=\\gamma\(q\)\\cdot\\theta\_\{0\}for a mapγ\\gammainto a group acting onΘ\\Theta\.

Example\(four objects\)\. On onem×nm\\times nbox, LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)isρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBAonQ=ℝm×r×ℝr×nQ=\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}, pointed at\(0,A0\)\(0,A\_\{0\}\)\. It is additive, with budgetr⁡\(m\+n\)r\(m\+n\)\. OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)with the exact Cayley map \([Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) isρ⁡\(Q\)=W0​C​\(Q\)\\rho\(Q\)=W\_\{0\}\\,C\(Q\)for skewQQ\(block\-diagonal in OFT\), pointed atQ=0Q=0\. It is of action type, withγ=C\\gamma=Clanding inSO⁡\(n\)\\mathrm\{SO\}\(n\)on the input side\. The frozen model is\(pt,∗,θ0\)\(\\mathrm\{pt\},\*,\\theta\_\{0\}\), the inclusion of the single pointθ0\\theta\_\{0\}, and full fine\-tuning is\(Θ,θ0,id\)\(\\Theta,\\theta\_\{0\},\\mathrm\{id\}\), the identity ofΘ\\Theta\.

Example\(one arrow\)\. Take LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\),B↦W0\+s​B​A0B\\mapsto W\_\{0\}\+sBA\_\{0\}withA0A\_\{0\}frozen, pointed atB=0B=0\. The pointed maph⁡\(B\)=\(B,A0\)h\(B\)=\(B,A\_\{0\}\)satisfiesρLoRA∘h=ρFA\\rho\_\{\\rm LoRA\}\\circ h=\\rho\_\{\\rm FA\}, so LoRA pointed at\(0,A0\)\(0,A\_\{0\}\)simulates LoRA\-FA \(Figure[2](https://arxiv.org/html/2610.12728#S1.F2)c\)\.

Random\-subspace training,ρ⁡\(z\)=θ0\+P​z\\rho\(z\)=\\theta\_\{0\}\+Pzwith a fixed random matrixPP, is how[Li et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib246)measured intrinsic dimension and how[Aghajanyan et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib5)fine\-tuned language models in small subspaces\. In the slice it is a linear object, withd⁡\(M\)=\|M\|d\(M\)=\\lvert M\\rvert\([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\) whenPPhas full column rank\.

\(pt,∗\)\{\\lx@inpgf@ignorespaces\(\\mathrm\{pt\},\*\)\}\(QM,q0\)\{\\lx@inpgf@ignorespaces\(Q\_\{M\},q\_\{0\}\)\}\(Θ,θ0\)\{\\lx@inpgf@ignorespaces\(\\Theta,\\theta\_\{0\}\)\}Frozen \(initial\)M\{\\color\[rgb\]\{0\.2734,0\.3359,0\.4141\}\\lx@inpgf@ignorespaces M\}Full FT \(terminal\)∗↦q0\\scriptstyle\{\\lx@inpgf@ignorespaces\*\\,\\mapsto\\,q\_\{0\}\}∗↦θ0\\scriptstyle\{\\color\[rgb\]\{0\.2734,0\.3359,0\.4141\}\\lx@inpgf@ignorespaces\*\\,\\mapsto\\,\\theta\_\{0\}\}ρM\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces\\rho\_\{M\}\}merge

\(a\) the interval \[frozen, full FT\]

\(QM,q0\)\{\\lx@inpgf@ignorespaces\(Q\_\{M\},q\_\{0\}\)\}\(QN,q0′\)\{\\lx@inpgf@ignorespaces\(Q\_\{N\},q\_\{0\}^\{\\prime\}\)\}\(Θ,θ0\)\{\\lx@inpgf@ignorespaces\(\\Theta,\\theta\_\{0\}\)\}h\\scriptstyle\{\\lx@inpgf@ignorespaces h\}ρM\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces\\rho\_\{M\}\}ρN\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces\\rho\_\{N\}\}

\(b\)NNsimulatesMM:ρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}

\(ℝm×r,0\)\{\\lx@inpgf@ignorespaces\(\\mathbb\{R\}^\{m\\times r\},0\)\}\(ℝm×r×ℝr×n,\(0,A0\)\)\{\\lx@inpgf@ignorespaces\(\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\},\(0,A\_\{0\}\)\)\}\(ℝm×n,W0\)\{\\lx@inpgf@ignorespaces\(\\mathbb\{R\}^\{m\\times n\},W\_\{0\}\)\}B↦\(B,A0\)\\scriptstyle\{\\lx@inpgf@ignorespaces B\\,\\mapsto\\,\(B,A\_\{0\}\)\}W0\+s​B​A0\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces W\_\{0\}\+sBA\_\{0\}\}W0\+s​B​A\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces W\_\{0\}\+sBA\}

\(c\) LoRA simulates LoRA\-FA

Figure 2:The pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\. \(a\) The interval of weight\-space methods\. For a methodMM, the first arrow picks the initialisationq0q\_\{0\}; it lies overΘ\\ThetabecauseMMstarts atθ0\\theta\_\{0\}\. The second arrow isρM\\rho\_\{M\}itself, the merge into the base weights, and the composite is the inclusion ofθ0\\theta\_\{0\}\. The frozen model is initial and full fine\-tuning terminal \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\)\. A pointing defect,ρM​\(q0\)≠θ0\\rho\_\{M\}\(q\_\{0\}\)\\neq\\theta\_\{0\}, leaves the triangle commuting only up to that defect \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\)\. Adapters and prefixes fall outside the interval because they extend the network \([Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\)\. \(b\) A simulation is a pointed smooth maphhthat makes the triangle over\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)commute; a weak simulation dropsh⁡\(q0\)=q0′h\(q\_\{0\}\)=q\_\{0\}^\{\\prime\}\. \(c\) An instance: LoRA pointed at\(0,A0\)\(0,A\_\{0\}\)simulates LoRA\-FA throughh⁡\(B\)=\(B,A0\)h\(B\)=\(B,A\_\{0\}\)\.
#### What Para adds

Remark I\.3\(orientation, and what Para adds\)\. Eachρ\\rhois a 2\-cellF⇒Fρ:=\(Q,f∘\(ρ×X\)\)F\\Rightarrow F^\{\\rho\}:=\(Q,f\\circ\(\\rho\\times X\)\)*out of*FF\. A simulationh:M→Nh:M\\to Nis a 2\-cellFρN⇒FρMF^\{\\rho\_\{N\}\}\\Rightarrow F^\{\\rho\_\{M\}\}, sinceρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Hence𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)is theopposite of the pointed coslice of reparametrisation 2\-cells underFF\([Background A\.7](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)\)\. The slice overFFis a different category, and its objects are extensions \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\.𝐌𝐞𝐭𝐡\\mathbf\{Meth\}depends only on\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)and not onff\. So for weight\-space methods,𝐏𝐚𝐫𝐚\\mathbf\{Para\}is bookkeeping\. The network enters in exactly three places, and we use𝐏𝐚𝐫𝐚\\mathbf\{Para\}only there:•extensions and mergeability \(Exposé[VI](https://arxiv.org/html/2610.12728#S6);[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2),[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\);•the realisationΦf\\Phi\_\{f\}, through functional images and descent \([Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\);•the backpropagation lens \(Exposé[IV](https://arxiv.org/html/2610.12728#S4);[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2),[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.

Calling𝐏𝐚𝐫𝐚\\mathbf\{Para\}bookkeeping concerns this slice only\. In the three places listed we use the framework of[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)directly, including their lens 2\-functor in Exposé[IV](https://arxiv.org/html/2610.12728#S4)\.

This is also why the next exposés work in weight space\. Reach inΘ\\Thetais a property of the image, and the cometricK=D​ρ​Λ​D​ρ⊤K=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}that shapes training is built fromρ\\rhoand the optimizer’s ratesΛ\\Lambda\. The network enters training through the gradient it hands to the method, and enters expressivity throughΦf\\Phi\_\{f\}once we ask about functions instead of weights\. So Exposés[II](https://arxiv.org/html/2610.12728#S2)and[III](https://arxiv.org/html/2610.12728#S3)work inΘ\\Thetaand flag the places whereffmatters\.

#### Frozen and full fine\-tuning

Observation I\.4\(universal properties\)\.In𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\):\(a\) the frozen model\(pt,∗,θ0\)\(\\mathrm\{pt\},\*,\\theta\_\{0\}\)is initial, and full fine\-tuning\(Θ,θ0,id\)\(\\Theta,\\theta\_\{0\},\\mathrm\{id\}\)is terminal\. The unique morphismM→M\\to\{\}Full FT isρM\\rho\_\{M\}itself\.\(b\) If the fibre productQM×ΘQNQ\_\{M\}\\times\_\{\\Theta\}Q\_\{N\}\([Background A\.11](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)\) exists in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, it is the productM×NM\\times N\. It exists, for instance, when the set\-level fibre product is an embedded submanifold ofQM×QNQ\_\{M\}\\times Q\_\{N\}: a clean intersection \([Background C\.7](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)\), or linear or affineρ\\rho’s as in[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)\. TransversalityρM⋔ρN\\rho\_\{M\}\\pitchfork\\rho\_\{N\}is never the reason when\|M\|\+\|N\|<dimΘ\\lvert M\\rvert\+\\lvert N\\rvert<\\dim\\Theta, since at\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\)it would force\|M\|\+\|N\|≥dimΘ\\lvert M\\rvert\+\\lvert N\\rvert\\geq\\dim\\Theta\. In𝐒𝐞𝐭\\mathbf\{Set\}the fibre product always exists andIm⁡\(M×N\)=Im⁡M∩Im⁡N\\Imop\(M\\times N\)=\\Imop M\\cap\\Imop N\. Whenever the product exists in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, its underlying set is this set\-level fibre product, so the image identity holds in every case\. The product can fail to exist\. OnΘ=ℝ\\Theta=\\mathbb\{R\}\(1×11\\times 1matrices\),LoRA1×Frozen\\mathrm\{LoRA\}\_\{1\}\\times\\text\{Frozen\}would have the cross\{BA=0\}⊂ℝ2\\\{BA=0\\\}\\subset\\mathbb\{R\}^\{2\}as underlying set\. No manifold structure on the cross has the universal property, because both axes would have to lift smoothly through the point over the origin, which forces rank22there\.\(c\)Im\\Imopis a functor \([Background A\.4](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)\) to the poset \([Background A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\) of subsets ofΘ\\Thetacontainingθ0\\theta\_\{0\}\. Conversely,Im⁡M⊆Im⁡N\\Imop M\\subseteq\\Imop Nyields a morphism of the set\-level slice𝐒𝐞𝐭∗/\(Θ,θ0\)\\mathbf\{Set\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\), though not necessarily a smooth one\. TakeρM​\(t\)=t\\rho\_\{M\}\(t\)=tandρN​\(u\)=u3\\rho\_\{N\}\(u\)=u^\{3\}onℝ\\mathbb\{R\}, pointed at00\. The images agree, yet no smoothh:ℝ→ℝh:\\mathbb\{R\}\\to\\mathbb\{R\}satisfiesρN∘h=id\\rho\_\{N\}\\circ h=\\mathrm\{id\}, that is,h​\(t\)3=th\(t\)^\{3\}=t\.

The proof is in Appendix[C\.2](https://arxiv.org/html/2610.12728#A3.SS2)\. The obstruction in \(b\) covers every pointing ofLoRA1\\mathrm\{LoRA\}\_\{1\}, including the standard one\(0,a0\)\(0,a\_\{0\}\)witha0≠0a\_\{0\}\\neq 0, whose base point is a smooth point of the cross; the obstruction sits at the origin\.

WHAT THIS EXPLAINSEvery weight\-space method pointed atθ0\\theta\_\{0\}lies in the interval \[frozen, full FT\]\. Adapters, prefixes and ReFT\([Wu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib505)\)are extensions outside𝐌𝐞𝐭𝐡\\mathbf\{Meth\}\(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\. “Merge into the base weights” is the canonical morphism to the terminal object\. In exact arithmetic on a full\-precision base this is definitional, since every object of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}is mergeable by construction\. Type\-preserving merges into quantised bases are lossy \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(d\)\), combining several adapters is a separate question \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\), and the remaining zero\-overhead questions live on extensions \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\. What bothMMandNNcan express,Im⁡M∩Im⁡N\\Imop M\\cap\\Imop N, always exists as a set\. The categorical product, when it exists, is the universal method simulated by both, and its image is exactly this set\.[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\) computes a genuine one \(LoRA\-XS\)\.[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(b\) and[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)compute meets, that is, intersections in the poset of subsets ofΘ\\Thetacontainingθ0\\theta\_\{0\}; they are not products in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}\.

##### Defaults that do not start at the checkpoint

Some defaults move the weights before training starts\. In Hugging Face PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)the default FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\)draws its spectrum from a standard normal, and the default vector bank and logits of VB\-LoRA\([Li et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib254)\)are random too\. LoRA with init\_lora\_weights=False, LoRA\-XS with its small randomRR, and every method on a quantised base also start away fromθ0\\theta\_\{0\}\. Each lives over its own starting weight or carries a pointing defect, which Exposé[V](https://arxiv.org/html/2610.12728#S5)measures\.

##### Lifting obstructions beyond first order

The gap between𝐒𝐞𝐭\\mathbf\{Set\}and𝐃𝐢𝐟𝐟\\mathbf\{Diff\}in \(c\) is a smooth lifting obstruction\.[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)below detects its first\-order instances\. In the cube\-root example,S1​\(M\)=ℝ⊈S1​\(N\)=0S\_\{1\}\(M\)=\\mathbb\{R\}\\not\\subseteq S\_\{1\}\(N\)=0\. Higher\-order obstructions need finer invariants\. The mapsρM​\(t\)=t3\\rho\_\{M\}\(t\)=t^\{3\}andρN​\(u\)=u5\\rho\_\{N\}\(u\)=u^\{5\}have equal images andS1=0S\_\{1\}=0on both sides, yet the only lift,t3/5t^\{3/5\}, is notC1C^\{1\}\.

#### Invariants of a method

The budget\|M\|\\lvert M\\rvertcounts parameters\. Two other numbers say more\. One is the dimension of what a method reaches; the other is the dimension of what it reaches to first order at the start, which is what the first gradient steps use\. For LoRA withr=16r=16on a4096×40964096\\times 4096matrix the budget is131,072131\{,\}072, the image has dimensionr⁡\(m\+n−r\)=130,816r\(m\+n\-r\)=130\{,\}816, and from zero initialisation the first steps use onlym​r=65,536mr=65\{,\}536directions \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\.

Definition I\.5\(invariants of an object\)\. ForM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\):•the imageIm⁡M=ρ⁡\(Q\)\\Imop M=\\rho\(Q\);•theexpressive dimensiond⁡\(M\)=dimIm⁡M=maxq⁡rank⁡d​ρqd\(M\)=\\dim\\Imop M=\\max\_\{q\}\\rk d\\rho\_\{q\};•thegauge wasteg⁡\(M\)=\|M\|−d⁡\(M\)g\(M\)=\\lvert M\\rvert\-d\(M\);•thefirst\-order imageS1​\(M\)=d​ρq0​\(Tq0​Q\)S\_\{1\}\(M\)=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\), and thefirst\-order deficit, which isd​\(M\)−dimS1​\(M\)d\(M\)\-\\dim S\_\{1\}\(M\);•thegauge group\{φ∈Diff⁡\(Q\):ρ∘φ=ρ\}\\\{\\varphi\\in\\mathrm\{Diff\}\(Q\):\\rho\\circ\\varphi=\\rho\\\}, which is unpointed, since a gauge transformation movesq0q\_\{0\}inside its fibre\. In practice we exhibit a Lie group \([Background D\.1](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) acting onQQthat preservesρ\\rhoand is locally transitive on generic fibres, and call it thegeneric gauge\.The base pointθ0\\theta\_\{0\}isregularforMMif the deficit is00, and anapexif the deficit is positive\. We sayvertexfor the subset\-level notion, the cone point of an image germ \([Background C\.10](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\), and keepapexfor the base\-point notion\. The two differ in general\. At a crossing of two smooth branches the image is singular, yet the deficit can be00\. In every case below with a positive deficit, the image germ is \(diffeomorphic to\) a cone with vertexθ0\\theta\_\{0\}; the exception is DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\), which inherits LoRA’s deficit through its torus factor\. A cone germ that is aC1C^\{1\}manifold at its vertex equals its tangent space \([Background C\.2](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\) there, so it is linear\. Hence the vertex of a non\-linear cone is a singular point\.

Remark\(the word “gauge”\)\. “Gauge” here means the symmetry group of a parametrisation, as in[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\. There is no connection, and away from the full\-rank stratum the fibres jump\. The word borrows, as an analogy, the physicist’s picture of redundant coordinates that leave every observation unchanged\. Only the group and its action enter the theorems\.

IfNNsimulatesMM, whateverMMcan do at its first step,NNcan do too\.

Observation I\.6\(S1S\_\{1\}is functorial\)\.IfM→NM\\to Nis a \(pointed\) morphism, thenS1​\(M\)⊆S1​\(N\)S\_\{1\}\(M\)\\subseteq S\_\{1\}\(N\)\. In particular, isomorphic methods have equalS1S\_\{1\}and equal first\-order deficit\. Monotonicity holds only along pointed morphisms\. Along a weak morphism, such as a gauge map movingq0q\_\{0\}inside its fibre,S1S\_\{1\}can change\.

The proof is in Appendix[C\.4](https://arxiv.org/html/2610.12728#A3.SS4)\.

WHAT THIS EXPLAINSS1S\_\{1\}is the set of weight\-space directions available to the linearised \(lazy, NTK\) dynamics at initialisation, since the lazy\-regime kernelJf​d​ρ​Λ​d​ρ⊤​Jf⊤J\_\{f\}\\,d\\rho\\,\\Lambda\\,d\\rho^\{\\top\}J\_\{f\}^\{\\top\}has rangeJf​\(S1\)J\_\{f\}\(S\_\{1\}\)\. That kernel also depends onΛ\\Lambdaand on the network JacobianJfJ\_\{f\}, not only onS1S\_\{1\}\. For random\-A0A\_\{0\}zero\-init LoRA it approximates the full fine\-tuning kernel\([Malladi et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib309)\), so a large first\-order deficit need not cost much on the data\.S1S\_\{1\}is an isomorphism invariant that separates initialisations the image cannot, namely zero\-initTA\\mathrm\{T\}\_\{A\}againstTB\\mathrm\{T\}\_\{B\}, and the choice ofrow⁡A0\\operatorname\{row\}A\_\{0\}\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\. The image already separates split initialisations, since it determinesPP\.

The kernel view of language\-model fine\-tuning, and the closeness of LoRA’s kernel to full fine\-tuning’s, are due to[Malladi et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib309)\.[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158)compared the zero initialisationsB0=0B\_\{0\}=0andA0=0A\_\{0\}=0as training procedures; in the slice they are already different objects\.

#### Sums

Weight space is a vector space, so running two methods side by side and adding their updates gives a method again\.

Observation I\.7\(additive monoidal structure\)\.Θ\\Thetais a vector space, so every object of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)can be writtenρ=θ0\+δ\\rho=\\theta\_\{0\}\+\\deltawithδ:=ρ−θ0\\delta:=\\rho\-\\theta\_\{0\}andδ⁡\(q0\)=0\\delta\(q\_\{0\}\)=0\. PutM⊕N:=\(QM×QN,\(q0,q0′\),θ0\+δM\+δN\)\.M\\oplus N:=\\big\(Q\_\{M\}\\times Q\_\{N\},\\ \(q\_\{0\},q\_\{0\}^\{\\prime\}\),\\ \\theta\_\{0\}\+\\delta\_\{M\}\+\\delta\_\{N\}\\big\)\.This is a symmetric monoidal structure \([Background B\.1](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\) with the frozen model as unit\. Its image is the Minkowski sumθ0\+\(Im⁡δM\+Im⁡δN\)\\theta\_\{0\}\+\(\\Imop\\delta\_\{M\}\+\\Imop\\delta\_\{N\}\)\([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\. It differs from the categorical product, whose image, when the product exists, is the intersectionIm⁡M∩Im⁡N\\Imop M\\cap\\Imop N\([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\)\. “Additive” in[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\(δ\\deltaindependent ofθ0\\theta\_\{0\}\) is a condition on*families*θ0↦M⁡\(θ0\)\\theta\_\{0\}\\mapsto M\(\\theta\_\{0\}\); it matters for base change \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\) and plays no role here\.WriteLoRAk\\mathrm\{LoRA\}\_\{k\}for LoRA with frozen residualθ0\\theta\_\{0\}; for split pointings the sum below has residualθ0−P−P′\\theta\_\{0\}\-P\-P^\{\\prime\}\. Concatenationh⁡\(B1,A1,B2,A2\)=\(\[B1​B2\],\[A1;A2\]\)h\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)=\(\[B\_\{1\}\\ B\_\{2\}\],\[A\_\{1\};A\_\{2\}\]\)is an isomorphismLoRAr⊕LoRAs≅LoRAr\+s\\mathrm\{LoRA\}\_\{r\}\\oplus\\mathrm\{LoRA\}\_\{s\}\\cong\\mathrm\{LoRA\}\_\{r\+s\}of unpointed objects\. It is a pointed isomorphism ontoLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}pointed at\(\[B0​B0′\],\[A0;A0′\]\)\(\[B\_\{0\}\\ B\_\{0\}^\{\\prime\}\],\[A\_\{0\};A\_\{0\}^\{\\prime\}\]\)\. ForTA\\mathrm\{T\}\_\{A\}pointings \(B0=0B\_\{0\}=0,rank⁡A0=r\\rk A\_\{0\}=r\), the target pointing hasrank⁡\[A0;A0′\]=r\+s\\rk\[A\_\{0\};A\_\{0\}^\{\\prime\}\]=r\+sif and only ifrow⁡A0∩row⁡A0′=0\\operatorname\{row\}A\_\{0\}\\cap\\operatorname\{row\}A\_\{0\}^\{\\prime\}=0; it lies in the stratumTA\\mathrm\{T\}\_\{A\}of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)when moreoverr\+s<min⁡\(m,n\)r\+s<\\min\(m,n\)\.Standard pointings \(TA\\mathrm\{T\}\_\{A\}, or symmetricallyTB\\mathrm\{T\}\_\{B\}\) exist only forr<min⁡\(m,n\)r<\\min\(m,n\)\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\), so the monoidal question is pointwise\. IfF⁡\(r\)F\(r\)andF⁡\(2​r\)F\(2r\)are standard \(2​r<min⁡\(m,n\)2r<\\min\(m,n\)\), thenF⁡\(r\)⊕F⁡\(r\)≇F⁡\(2​r\)F\(r\)\\oplus F\(r\)\\not\\cong F\(2r\)as pointed objects; for2​r≥min⁡\(m,n\)2r\\geq\\min\(m,n\)no standardF⁡\(2​r\)F\(2r\)exists\. Hence no assignmentr↦LoRArr\\mapsto\\mathrm\{LoRA\}\_\{r\}withF⁡\(1\)F\(1\)andF⁡\(2\)F\(2\)standard isstrong monoidal\([Background B\.15](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\) into pointed objects\. Forr≠sr\\neq sand standard pointings,F⁡\(r\)⊕F⁡\(s\)≅F⁡\(r\+s\)F\(r\)\\oplus F\(s\)\\cong F\(r\+s\)as pointed objects exactly when the row space of the target’sA0A\_\{0\}is the direct sum of the two source row spaces\. The assignment is strong monoidal on unpointed objects, up to weak morphisms, and on pointed objects with degenerate pointings, namely the origin\(B0,A0\)=\(0,0\)\(B\_\{0\},A\_\{0\}\)=\(0,0\), whereS1=0S\_\{1\}=0, or the stacked pointingsF⁡\(r\)=F​\(1\)⊕rF\(r\)=F\(1\)^\{\\oplus r\}\. Concatenation is strictly associative and unital, so discrete\(ℕ,\+\)\(\\mathbb\{N\},\+\)suffices for a plain strong monoidal functor\.*Symmetric*strong monoidality needs the permutation groupoid, since swapping and then concatenating permutes the blocks\.With per\-adapter scalessrs\_\{r\}\(α/r\\alpha/rfor LoRA,α/r\\alpha/\\sqrt\{r\}for rsLoRA\) and any target scales∗s\_\{\*\}, the maph=\(\[B1​B2\],\[c1​A1;c2​A2\]\)h=\(\[B\_\{1\}\\ B\_\{2\}\],\[c\_\{1\}A\_\{1\};c\_\{2\}A\_\{2\}\]\)withci=sri/s∗c\_\{i\}=s\_\{r\_\{i\}\}/s\_\{\*\}, wherer1=rr\_\{1\}=randr2=sr\_\{2\}=s, is an isomorphism overΘ\\Theta\. It is a linear bijection and preserves row spaces, so the pointing criterion is unchanged\.

The proof is in Appendix[C\.5](https://arxiv.org/html/2610.12728#A3.SS5)\. The negative claim rests ondimS1\\dim S\_\{1\}, which any pointed isomorphism preserves \([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. It ism​rmrforF⁡\(r\)⊕F⁡\(r\)F\(r\)\\oplus F\(r\)and2​m​r2mrfor a standardF⁡\(2​r\)F\(2r\)\. \[Num\] Atm=7m=7,dimS1=28\\dim S\_\{1\}=28for a standardLoRA4\\mathrm\{LoRA\}\_\{4\}, against1414forLoRA2⊕LoRA2\\mathrm\{LoRA\}\_\{2\}\\oplus\\mathrm\{LoRA\}\_\{2\}with the sameA0A\_\{0\}and2828with independentA0A\_\{0\}’s;F⁡\(1\)⊕F⁡\(2\)F\(1\)\\oplus F\(2\)and a standardF⁡\(3\)F\(3\)both give2121\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### The cat merge of Hugging Face PEFT

Hugging Face PEFT’s add\_weighted\_adapter\(…, combination\_type="cat"\)\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)evaluates the rescaled concatenationhhat trained points\. It takess∗=1s\_\{\*\}=1\. It sets the merged lora\_alpha to the new rankr\+sr\+s, multiplies eachAAblock bywi​sriw\_\{i\}s\_\{r\_\{i\}\}, so thatci=wi​sric\_\{i\}=w\_\{i\}s\_\{r\_\{i\}\}, and leaves theBBblocks alone\. It is the isomorphismhhwhen all weightswi=1w\_\{i\}=1and the merged adapter has use\_rslora=False\. In PEFT releases up to v0\.21\.1 the merged adapter inherits use\_rslora from the first source\. For rsLoRA sources\([Kalajdzievski, 2023](https://arxiv.org/html/2610.12728#bib.bib204)\)its scale is then\(r\+s\)/r\+s=r\+s\(r\+s\)/\\sqrt\{r\+s\}=\\sqrt\{r\+s\}, and the merged update isr\+s\\sqrt\{r\+s\}times too large\. PR \#3449 on the main branch fixes this by setting use\_rslora=False on the merged adapter\.

WHAT THIS EXPLAINSRoSA\([Nikdan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib350)\)\(LoRA⊕\\oplusa fixed\-mask sparse method\) and GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\)\(⊕\\oplusofk2k^\{2\}block\-embedded LoRAs\) are⊕\\oplus\-words, and their expressivity is a Minkowski sum\. ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)isLoRA⊕K\\mathrm\{LoRA\}^\{\\oplus K\}unrolled in time, apart from its full\-rank warm start\. A sum ofkkrank\-rrLoRAs, each with residualθ0\\theta\_\{0\}, is isomorphic in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}to one LoRA of rankk​rkr, pointed at the concatenated initialisation\. Underα/r\\alpha/rscaling the isomorphism rescales theAAblocks, so training at equalα\\alphaand learning rate follows a different trajectory\. The cat merge is the isomorphismhhevaluated at trained points; the merged adapter, if unfrozen, isLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}\.

##### What is and is not a sum

LoRA\-soup CAT\([Prabhakar et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib374)\)freezes the trained modules and learns layer\-wise weights, so it is a⊕\\oplusofkkfrozen line methodsαi↦αi​Bi​Ai\\alpha\_\{i\}\\mapsto\\alpha\_\{i\}B\_\{i\}A\_\{i\}\. Its per\-layer image is a span of dimension at mostkk\(generically exactlykk\), much smaller than the image ofLoRAk​r\\mathrm\{LoRA\}\_\{kr\}\. UniPELT\([Mao et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib315)\)and MAM\([He et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib165)\)arenot⊕\\oplus\-words in𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\. Their adapter and prefix parts live on extensions \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\), and UniPELT’s gates depend on the input\. The obstruction in[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)is the reuse of one pointing\. With a fixed pointing, stacking the sameA0A\_\{0\}twice gives a rank\-deficient frame, whichS1S\_\{1\}distinguishes from a standardLoRA2​r\\mathrm\{LoRA\}\_\{2r\}\.

#### Products

FreezeAAatA0A\_\{0\}and trainBB, or freezeBBatB0B\_\{0\}and trainAA\. The product of these two one\-sided methods is LoRA\-XS\. As a pullback square overΘ\\Thetait reads

ℝr×r\{\\lx@inpgf@ignorespaces\\mathbb\{R\}^\{r\\times r\}\}QFA=ℝm×r\{\\lx@inpgf@ignorespaces Q\_\{\\rm FA\}=\\mathbb\{R\}^\{m\\times r\}\}QFB=ℝr×n\{\\lx@inpgf@ignorespaces Q\_\{\\rm FB\}=\\mathbb\{R\}^\{r\\times n\}\}Θ=ℝm×n\{\\lx@inpgf@ignorespaces\\Theta=\\mathbb\{R\}^\{m\\times n\}\}R↦B0​R\\scriptstyle\{\\lx@inpgf@ignorespaces R\\,\\mapsto\\,B\_\{0\}R\}R↦R​A0\\scriptstyle\{\\lx@inpgf@ignorespaces R\\,\\mapsto\\,RA\_\{0\}\}⌟\{\\lx@inpgf@ignorespaces\\lrcorner\}B↦W0\+B​A0\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces B\\,\\mapsto\\,W\_\{0\}\+BA\_\{0\}\}A↦W0\+B0​A\\scriptstyle\{\\color\[rgb\]\{0\.0547,0\.4492,0\.5234\}\\lx@inpgf@ignorespaces A\\,\\mapsto\\,W\_\{0\}\+B\_\{0\}A\}and both composites sendRRtoW0\+B0​R​A0W\_\{0\}\+B\_\{0\}RA\_\{0\}\.

Proposition I\.8\(a product that produces a named method; an image\-level meet\)\.\(a\) LetA0∈ℝr×nA\_\{0\}\\in\\mathbb\{R\}^\{r\\times n\}have full row rank andB0∈ℝm×rB\_\{0\}\\in\\mathbb\{R\}^\{m\\times r\}full column rank\. Define the one\-sided methodsFA⁡\(A0\):B↦W0\+B​A0,FB⁡\(B0\):A↦W0\+B0​A,\\mathrm\{FA\}\(A\_\{0\}\):B\\mapsto W\_\{0\}\+BA\_\{0\},\\qquad\\mathrm\{FB\}\(B\_\{0\}\):A\\mapsto W\_\{0\}\+B\_\{0\}A,both pointed at00\. ThenFA\(A0\)×FB\(B0\)≅\(ℝr×r,0,R↦W0\+B0RA0\)\.\\mathrm\{FA\}\(A\_\{0\}\)\\times\\mathrm\{FB\}\(B\_\{0\}\)\\;\\cong\\;\\big\(\\mathbb\{R\}^\{r\\times r\},\\ 0,\\ R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}\\big\)\.The product supplies this sandwich for*any*frames\. Assumerank⁡W0≥r\\rk W\_\{0\}\\geq r\(σr\>0\\sigma\_\{r\}\>0\) and take the SVD framesB0=UrB\_\{0\}=U\_\{r\},A0=Σr​Vr⊤A\_\{0\}=\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\. The result isLoRA\-XS pointed atR=0R=0\.\(b\) LetW0W\_\{0\}have full row rank \(som≤nm\\leq n\), and letLoRAr\\mathrm\{LoRA\}\_\{r\}carry its standard pointing \(B0=0B\_\{0\}=0, imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\. The image\-level meet ofLoRAr\\mathrm\{LoRA\}\_\{r\}with the output\-scaling methodℓ↦diag⁡\(ℓ\)​W0\\ell\\mapsto\\diag\(\\ell\)W\_\{0\}is\{diag⁡\(ℓ\)​W0:\#⁡\{i:ℓi≠1\}≤r\}\\\{\\diag\(\\ell\)W\_\{0\}:\\\#\\\{i:\\ell\_\{i\}\\neq 1\\\}\\leq r\\\}, that is, “rescale at mostrroutput neurons”\.The LoRA\-XS paper\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\)writesx​W\+x​A​R​BxW\+xARBin row\-vector convention withA=Ur​ΣrA=U\_\{r\}\\Sigma\_\{r\}andB=Vr⊤B=V\_\{r\}^\{\\top\}\. Transposed, the update isB0​R⊤​A0B\_\{0\}R^\{\\top\}A\_\{0\}, so the isomorphism isR↦α​R⊤R\\mapsto\\alpha R^\{\\top\}\. LoRA\-XS initialisesR∼𝒩⁡\(0,σ2\)R\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)withσ=10−5\\sigma=10^\{\-5\}, a negligible pointing defect\. The output\-scaling method in \(b\) is the key/value part of \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)\. On the FFN, \(IA\)3scales the*input*ofWdownW\_\{\\rm down\},W↦W​diag⁡\(ℓ\)W\\mapsto W\\diag\(\\ell\); the mirror statement then needs full column rank, or holds generically forr<dr<d, whereddis the number of rows ofWdownW\_\{\\rm down\}\.

The proof is in Appendix[C\.6](https://arxiv.org/html/2610.12728#A3.SS6)\. In \(a\) the fibre product is ther2r^\{2\}\-dimensional linear subspace\{\(B0​R,R​A0\)\}\\\{\(B\_\{0\}R,RA\_\{0\}\)\\\}, and ifrank⁡W0<r\\rk W\_\{0\}<rthe frameΣr​Vr⊤\\Sigma\_\{r\}V\_\{r\}^\{\\top\}is rank\-deficient and the LoRA\-XS instance breaks\. \[Num\] The fibre product of \(a\) has dimension4=r24=r^\{2\}at\(m,n,r\)=\(6,8,2\)\(m,n,r\)=\(6,8,2\)\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSLoRA\-XS is the largest method that factors through*both*frozen frames\. In the frames of an initialisation it is the overlap that a split initialisation double\-counts \([Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)\); it is LoRA\-XS proper when those frames span the top\-rrsingular subspaces ofW0W\_\{0\}, as at a PiSSA initialisation\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)\. The meet in \(b\) is only set\-level\. Forn≥2n\\geq 2the product does not exist in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, because its fibre product is singular at the origin\(b,a,ℓ\)=\(0,0,𝟏\)\(b,a,\\ell\)=\(0,0,\\mathbf\{1\}\), by the rank argument of[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) \(Appendix[C\.6](https://arxiv.org/html/2610.12728#A3.SS6)\)\. Once a support is fixed, the meet is just \(IA\)3masked torrneurons; compare sparse activation scaling\([Stoehr et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib428)\)\.

##### The shape of the meet in \(b\)

The meet is the union of the\(mr\)\\binom\{m\}\{r\}coordinaterr\-planes “rescale at mostrroutput neurons”\. Forr=1r=1the fibre product is the union of the linear spaces\{b=0,ℓ=𝟏\}\\\{b=0,\\ \\ell=\\mathbf\{1\}\\\}and\{a=0,ℓ=𝟏\}\\\{a=0,\\ \\ell=\\mathbf\{1\}\\\}withmmtwo\-dimensional sheets\{b=c​ei,a=λ​wi,ℓi=1\+s​c​λ\}\\\{b=ce\_\{i\},\\ a=\\lambda w\_\{i\},\\ \\ell\_\{i\}=1\+sc\\lambda\\\}, wherewiw\_\{i\}is rowiiofW0W\_\{0\}\. Under the standard pointing \(b=0b=0,a0a\_\{0\}generic\) the base point is a smooth point of\{b=0,ℓ=𝟏\}\\\{b=0,\\ \\ell=\\mathbf\{1\}\\\}, so the singularity sits away from it, at the origin\.

#### The dictionary so far

Table[3](https://arxiv.org/html/2610.12728#S1.T3)collects the translations made in this exposé\. The full dictionary is the Rosetta stone of Exposé[IX](https://arxiv.org/html/2610.12728#S9)\.

Table 3:The dictionary for Exposé[I](https://arxiv.org/html/2610.12728#S1)\.

## EXPOSÉ IIThe image germ: expressivity

Expressivity is read off the germ of a method’s image at the pretrained point\. An initialisation decides whether that point sits at the apex of a cone, on a smooth sheet of one, or just off the image\.

Exposé[I](https://arxiv.org/html/2610.12728#S1)turned a PEFT method into a pointed mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\)\. Its imageIm⁡M=ρ⁡\(Q\)\\Imop M=\\rho\(Q\)is everything it can reach\. Fine\-tuning travels only a short way from the pretrained weights, so what matters is the image nearθ0\\theta\_\{0\}, its*germ*atθ0\\theta\_\{0\}\([Background C\.8](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\)\.

The germ takes few shapes \(Figure[1](https://arxiv.org/html/2610.12728#S0.F1)\)\. It is a linear space whenρ\\rhois affine, as for BitFit\([Ben\-Zaken et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib27)\), LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\), LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\), FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\)and \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)\. It is a cone \([Background C\.10](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\) whenρ\\rhois multihomogeneous and non\-linear and starts where the update vanishes, as LoRA’sB​ABA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)does at zero initialisation\. It is a piece of a group orbit for exactly orthogonal methods such as exact\-Cayley OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)\. DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\)adds an output gain to LoRA’s cone \([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\), and HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)meets that cone with a rotation orbit \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\), each under its stated hypotheses\. Adapters and prefixes change the network itself, and their germs live among functions \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\.

Four invariants of the germ carry most of the story, and isomorphic methods share them \([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\.

- •Thefirst\-order imageS1​\(M\)=d​ρq0​\(Tq0​Q\)S\_\{1\}\(M\)=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\), the directions a first step can take\.
- •Thetype of the base point:*regular*when the first\-order deficitd​\(M\)−dimS1​\(M\)d\(M\)\-\\dim S\_\{1\}\(M\)vanishes, an*apex*when it is positive \([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\.
- •Themaximum rankof an update, bounded by a minimum cut \([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\)\.
- •Theexpressive dimensiond⁡\(M\)=dimIm⁡Md\(M\)=\\dim\\Imop M, which falls short of the parameter count\|M\|\|M\|by the gauge wasteg⁡\(M\)=\|M\|−d⁡\(M\)g\(M\)=\|M\|\-d\(M\)\.

Subsection[II\.A](https://arxiv.org/html/2610.12728#S2.SS1)asks where a method starts, Subsection[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)what a group\-type method can never change \([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1),[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\), and Subsection[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)how rank differs from dimension \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. Expressivity here means reach, and finding a reachable point is left to Exposé[III](https://arxiv.org/html/2610.12728#S3)\.

Which functions a LoRA\-adapted network can represent is studied by[Zeng and Lee \(2023\)](https://arxiv.org/html/2610.12728#bib.bib542)\. The intrinsic dimension of[Li et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib246)and[Aghajanyan et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib5)depends on the data, whiled⁡\(M\)d\(M\)is a property of the method alone\.

### II\.A Initialisations: pointings and strata

Every LoRA recipe decides where training starts\. LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)setsB0=0B\_\{0\}=0and drawsA0A\_\{0\}at random\. PiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)takes both factors from the top of the SVD ofW0W\_\{0\}and subtracts their product from the frozen weight, LoRA\-GA\([Wang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib472)\)does the same with the first gradient, and CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\), OLoRA\([Büyükakyüz, 2024](https://arxiv.org/html/2610.12728#bib.bib47)\), MiLoRA\([Wang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib465)\)and EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)choose otherwise\. Each recipe is a*pointing*, a parameterq0q\_\{0\}with the frozen residual that makesρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}, and two pointings of one formula can be different objects of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\. Figure[3](https://arxiv.org/html/2610.12728#S2.F3)draws the typical cases\.

The answers occupy the rest of this subsection\. Forr<min⁡\(m,n\)r<\\min\(m,n\), full\-rank LoRA initialisations come in exactly three kinds up to reparametrisation, classified as sets byrow⁡A0\\operatorname\{row\}A\_\{0\}\(whenB0=0B\_\{0\}=0\), bycol⁡B0\\operatorname\{col\}B\_\{0\}\(whenA0=0A\_\{0\}=0\) or by the subtracted productB0​A0B\_\{0\}A\_\{0\}\(PiSSA\-type splits\), as[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)shows\. Zero\-factor starts sit at the apex of a cone withm​rmrorn​rnrfirst\-order directions, and split starts at a smooth point with allr⁡\(m\+n−r\)r\(m\+n\-r\)\. For a4096×40964096\\times 4096matrix at rank1616that is130,816130\{,\}816against65,53665\{,\}536\([Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)\)\. Neither kind reaches everything the other reaches\. No exact start of LoRArreaches an update of rank above2​r2r\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\), and a factor behind a closed gate, such asAAwhenB0=0B\_\{0\}=0, does not move the weights at first order \([Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)\)\.

Figure 3:The apex and the smooth point\.Panels \(a\) and \(b\) draw the symmetric sliceS=\(x\+yzzx−y\)S=\\left\(\\begin\{smallmatrix\}x\+y&z\\\\ z&x\-y\\end\{smallmatrix\}\\right\)of2×22\\times 2matrices, in which rank≤1\\leq 1is the grey double conex2=y2\+z2x^\{2\}=y^\{2\}\+z^\{2\}, and the modelρ⁡\(c,ϕ\)=Wres\+c​vϕ​vϕ⊤\\rho\(c,\\phi\)=W\_\{\\rm res\}\+c\\,v\_\{\\phi\}v\_\{\\phi\}^\{\\top\},vϕ=\(cos⁡ϕ,sin⁡ϕ\)v\_\{\\phi\}=\(\\cos\\phi,\\sin\\phi\), which plays LoRA \(ccforBB,vϕv\_\{\\phi\}forAA\); its image is the whole cone, sod=2d=2\. \(a\) Zero initialisation,c0=0c\_\{0\}=0andWres=θ0W\_\{\\rm res\}=\\theta\_\{0\}: the base pointθ0\\theta\_\{0\}is the vertex of the cone, and the first\-order imageS1S\_\{1\}\(teal\) is a single line, one ruling, so the deficit is11andθ0\\theta\_\{0\}is an apex \([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(c\)\)\. \(b\) A split \(PiSSA\-type\) initialisation,c0≠0c\_\{0\}\\neq 0andWres=θ0−PW\_\{\\rm res\}=\\theta\_\{0\}\-PwithP=c0​vϕ0​vϕ0⊤P=c\_\{0\}v\_\{\\phi\_\{0\}\}v\_\{\\phi\_\{0\}\}^\{\\top\}: the vertex moves toθ0−P\\theta\_\{0\}\-P\(open diamond\),θ0\\theta\_\{0\}lies on a smooth sheet of the cone, andS1S\_\{1\}is the tangent plane there, so the deficit is00\. Both values ofdimS1\\dim S\_\{1\}are Jacobian ranks of the model\. \(c\) Form=n=4096m=n=4096andr=16r=16, each pair of bars compares the expressive dimensiond⁡\(M\)d\(M\)\(grey\) withdimS1\\dim S\_\{1\}\(teal\); the dashed red box is the gap between them, the first\-order deficit\. Zero\-init LoRA, stratumTA\\mathrm\{T\}\_\{A\}, hasdimS1=m​r=65,536\\dim S\_\{1\}=mr=65\{,\}536againstd=r⁡\(m\+n−r\)=130,816d=r\(m\+n\-r\)=130\{,\}816; a split initialisation hasdimS1=d\\dim S\_\{1\}=d\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(b\)\); HRA’s paired initialisationu2​i−1=u2​iu\_\{2i\-1\}=u\_\{2i\},k=r/2k=r/2, hasdimS1=k​n−k⁡\(k\+1\)2=32,732\\dim S\_\{1\}=kn\-\\tfrac\{k\(k\+1\)\}\{2\}=32\{,\}732againstd=r​n−r⁡\(r\+1\)2=65,400d=rn\-\\tfrac\{r\(r\+1\)\}\{2\}=65\{,\}400, so it too starts at an apex \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\)\)\. The script checks each closed form against a Jacobian rank at\(m,n,r\)=\(11,9,4\)\(m,n,r\)=\(11,9,4\)\.#### Translates of one shape

An additive method adds updates from a fixed shapeCC, for LoRA the coneℳ≤r\\mathcal\{M\}\_\{\\leq r\}of matrices of rank at mostrr\([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\. Pointing it atqqslides the shape so thatδ⁡\(q\)\\delta\(q\)lands onθ0\\theta\_\{0\}, and reachable updates become differences of two points ofCC\([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

Theorem II\.1\(pointings of an additive method\)\.LetM=\(Q,q0,θ0\+δ\)M=\(Q,q\_\{0\},\\theta\_\{0\}\+\\delta\)be additive with shapeC=δ⁡\(Q\)C=\\delta\(Q\)\. Write−v\+C\-v\+Cfor a translate, andC−C:=\{c−c′:c,c′∈C\}C\-C:=\\\{c\-c^\{\\prime\}:c,c^\{\\prime\}\\in C\\\}for the Minkowski difference\.\(a\) For eachq∈Qq\\in Q, the re\-pointed methodMq:=\(Q,q,θ0−δ⁡\(q\)\+δ⁡\(⋅\)\)M\_\{q\}:=\(Q,q,\\theta\_\{0\}\-\\delta\(q\)\+\\delta\(\\cdot\)\), with frozen residualθ0−δ⁡\(q\)\\theta\_\{0\}\-\\delta\(q\), is pointed atθ0\\theta\_\{0\}, andIm⁡Mq=θ0\+\(−δ⁡\(q\)\+C\)\\Imop M\_\{q\}=\\theta\_\{0\}\+\(\-\\delta\(q\)\+C\)\.\(b\)⋃qIm⁡Mq=θ0\+\(C−C\)\\bigcup\_\{q\}\\Imop M\_\{q\}=\\theta\_\{0\}\+\(C\-C\)\.\(c\) For LoRAron a single linear box withs≠0s\\neq 0,C−C=ℳ≤min⁡\(2​r,m,n\)C\-C=\\mathcal\{M\}\_\{\\leq\\min\(2r,m,n\)\}; for several boxes,C−CC\-Cis the product of the per\-box varietiesℳ≤min⁡\(2​r,mi,ni\)\\mathcal\{M\}\_\{\\leq\\min\(2r,m\_\{i\},n\_\{i\}\)\}\. Henceno exact pointing of LoRAr\(frozen residualθ0−δ⁡\(q\)\\theta\_\{0\}\-\\delta\(q\)\), whether zero\-init or split \(PiSSA, OLoRA, CorDA, LoRA\-GA\), makes an updateθ−θ0\\theta\-\\theta\_\{0\}of rank\>2​r\>2rreachable in any adapted matrix\.More robustly, from*any*starting pointq0q\_\{0\}of LoRAr, pointed or not,ρ⁡\(q\)−ρ⁡\(q0\)=δ⁡\(q\)−δ⁡\(q0\)\\rho\(q\)\-\\rho\(q\_\{0\}\)=\\delta\(q\)\-\\delta\(q\_\{0\}\)has rank≤min⁡\(2​r,m,n\)\\leq\\min\(2r,m,n\)\. Quantised\-residual schemes \(QLoRA, LoftQ, QPiSSA\) are pointed only up to a defect that depends on the order of operations \(eefor QLoRA;[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\) for the others\)\. Measured fromθ0\\theta\_\{0\}rather than from their starting weight, their updates contain this defect, whose rank is generically at leastmin⁡\(m,n\)−r\\min\(m,n\)\-r\. The bound2​r2ris attained by split pointings when2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\.

The theorem is elementary, and its rank\-2​r2rconsequence is known in practice\.[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324)convert a trained PiSSA adapter into a rank\-2​r2rLoRA through

B​A−B0​A0=\[BB0\]​\[A−A0\],BA\-B\_\{0\}A\_\{0\}=\\begin\{bmatrix\}B&B\_\{0\}\\end\{bmatrix\}\\begin\{bmatrix\}A\\\\ \-A\_\{0\}\\end\{bmatrix\},\(1\)and HF PEFT ships the conversion\. The theorem adds that the conversion loses nothing and that rank2​r2ris needed in general\.

WHAT THIS EXPLAINSAn initialisation of an additive method chooses which*translate*of the shape passes throughθ0\\theta\_\{0\}\. Splitting initialisations \(PiSSA\-type\) buy “spectral replacement” updatesY−PY\-P, which can have rank2​r2r\. They pay by losing generic updates of every rank≥1\\geq 1\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\)\)\. Moving a weight matrix by more than rank2​r2rneeds rebasing, as in ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\), or another shape\.

#### Three strata

Isomorphisms of methods are pointed diffeomorphisms overΘ\\Theta\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\), such as the gauge\(B,A\)↦\(B​g−1,g​A\)\(B,A\)\\mapsto\(Bg^\{\-1\},gA\)withg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\. They preserve the image andS1S\_\{1\}\([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\) but may change training \(Exposé[III](https://arxiv.org/html/2610.12728#S3)\)\. Up to isomorphism, a full\-rank LoRA initialisation is determined by one subspace or one matrix\.

Theorem II\.2\(three strata of LoRA initialisations\)\.Letδ⁡\(B,A\)=B​A\\delta\(B,A\)=BAwith1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)\. Consider pointed LoRA objects in three distinguished strata of full\-rank pointings \(pointings whose nonzero factors have full rank\)\. OnlyS\\mathrm\{S\}is open and dense in the space of pointings;TA\\mathrm\{T\}\_\{A\}andTB\\mathrm\{T\}\_\{B\}are nowhere dense\.•\(TA\)\(\\mathrm\{T\}\_\{A\}\):B0=0B\_\{0\}=0andrank⁡A0=r\\rk A\_\{0\}=r;•\(TB\)\(\\mathrm\{T\}\_\{B\}\):A0=0A\_\{0\}=0andrank⁡B0=r\\rk B\_\{0\}=r;•\(S\)\(\\mathrm\{S\}\):rank⁡B0​A0=r\\rk B\_\{0\}A\_\{0\}=r, with frozen residualθ0−B0​A0\\theta\_\{0\}\-B\_\{0\}A\_\{0\}\.\(a\)Classification\.Within a stratum,Mq0≅Mq0′M\_\{q\_\{0\}\}\\cong M\_\{q\_\{0\}^\{\\prime\}\}if and only if, respectively,row⁡A0=row⁡A0′\\operatorname\{row\}A\_\{0\}=\\operatorname\{row\}A\_\{0\}^\{\\prime\};col⁡B0=col⁡B0′\\operatorname\{col\}B\_\{0\}=\\operatorname\{col\}B\_\{0\}^\{\\prime\}; orB0​A0=B0′​A0′B\_\{0\}A\_\{0\}=B\_\{0\}^\{\\prime\}A\_\{0\}^\{\\prime\}\. Objects in different strata are never isomorphic\. As a*set*, the isomorphism classes of the three strata therefore formGr⁡\(r,ℝn\)⊔Gr⁡\(r,ℝm\)⊔ℳr​\(m×n\)\.\\Gr\(r,\\mathbb\{R\}^\{n\}\)\\ \\sqcup\\ \\Gr\(r,\\mathbb\{R\}^\{m\}\)\\ \\sqcup\\ \\mathcal\{M\}\_\{r\}\(m\\times n\)\.\(2\)\(b\)First order\.dimS1\\dim S\_\{1\}ism​rmr,n​rnrandr⁡\(m\+n−r\)r\(m\+n\-r\)respectively, whiled=r⁡\(m\+n−r\)d=r\(m\+n\-r\)in every case\. The deficits arer⁡\(n−r\)r\(n\-r\),r⁡\(m−r\)r\(m\-r\)and00\.\(c\)Geometry\.InTA\\mathrm\{T\}\_\{A\}andTB\\mathrm\{T\}\_\{B\},Im=θ0\+ℳ≤r\\Imop=\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, whose vertexθ0\\theta\_\{0\}is anapex\. InS\\mathrm\{S\},Im=\(θ0−P\)\+ℳ≤r\\Imop=\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}withP=B0​A0P=B\_\{0\}A\_\{0\}, andθ0\\theta\_\{0\}is asmooth point\.\(d\)Incomparability\.The images ofS\\mathrm\{S\}\-type andT\\mathrm\{T\}\-type objects are incomparable\.\(e\)Scope\.The three strata do not exhaust the pointings of LoRAr\. Outside them lie:B0=0B\_\{0\}=0withrank⁡A0<r\\rk A\_\{0\}<r;A0=0A\_\{0\}=0withrank⁡B0<r\\rk B\_\{0\}<r;B0=A0=0B\_\{0\}=A\_\{0\}=0;0<rank⁡B0​A0<r0<\\rk B\_\{0\}A\_\{0\}<r; and pointings withB0,A0B\_\{0\},A\_\{0\}both nonzero andB0​A0=0B\_\{0\}A\_\{0\}=0\. HF PEFT ships one of the last kind: init\_lora\_weights="orthogonal" \(which requiresrreven\) lies in the stratum\(rk⁡B0,rk⁡A0,rk⁡B0​A0\)=\(r/2,r/2,0\)\(\\operatorname\{rk\}B\_\{0\},\\operatorname\{rk\}A\_\{0\},\\operatorname\{rk\}B\_\{0\}A\_\{0\}\)=\(r/2,r/2,0\), an apex withdimS1=r2​\(m\+n−r2\)\\dim S\_\{1\}=\\tfrac\{r\}\{2\}\(m\+n\-\\tfrac\{r\}\{2\}\)\. It is isomorphic to no object of the three strata\. Its imageθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}rules outS\\mathrm\{S\}, and itsS1S\_\{1\}determines\(col⁡B0,row⁡A0\)\(\\operatorname\{col\}B\_\{0\},\\operatorname\{row\}A\_\{0\}\), of dimensions\(r/2,r/2\)\(r/2,r/2\), against\(0,r\)\(0,r\)inTA\\mathrm\{T\}\_\{A\}and\(r,0\)\(r,0\)inTB\\mathrm\{T\}\_\{B\}\. The dimension ofS1S\_\{1\}alone does not separate it, since at\(m,n,r\)=\(4,5,2\)\(m,n,r\)=\(4,5,2\)it equalsm​rmr\. This placement holds in exact arithmetic; HF casts the two factors to the base dtype separately, and in bf16 the result is a generic pointing with a tiny defect\. For every pointing, the image determinesP=B0​A0P=B\_\{0\}A\_\{0\}\(its vertex isθ0−P\\theta\_\{0\}\-P\), andS1=\{X​A0\+B0​Y\}S\_\{1\}=\\\{XA\_\{0\}\+B\_\{0\}Y\\\}determines\(row⁡A0,col⁡B0\)\(\\operatorname\{row\}A\_\{0\},\\operatorname\{col\}B\_\{0\}\)whenever these are proper subspaces\. The classification is set\-level: in the space of pointingsTA\\mathrm\{T\}\_\{A\}lies in the closure ofS\\mathrm\{S\}, since\(ε​B′,A0\)→\(0,A0\)\(\\varepsilon B^\{\\prime\},A\_\{0\}\)\\to\(0,A\_\{0\}\), so the quotient is not a topological disjoint union\.

Numerically,dimS1=\(14,10,20\)\\dim S\_\{1\}=\(14,10,20\)for\(TA,TB,S\)\(\\mathrm\{T\}\_\{A\},\\mathrm\{T\}\_\{B\},\\mathrm\{S\}\)at\(m,n,r\)=\(7,5,2\)\(m,n,r\)=\(7,5,2\), anddimS1=11\\dim S\_\{1\}=11for HF’s orthogonal init \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\. HereGr\\Gris the Grassmannian \([Background C\.15](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\),ℳr\\mathcal\{M\}\_\{r\}the smooth stratum ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}\([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\), and smooth and singular points are those of[Background C\.9](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\.

Remark\(a moduli space of initialisations, as an analogy\)\. Calling the three spaces in \([2](https://arxiv.org/html/2610.12728#S2.E2)\) a moduli space \([Background C\.20](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\) borrows a word\. The bijection is of sets only\.TA\\mathrm\{T\}\_\{A\}lies in the closure ofS\\mathrm\{S\}, since\(ε​B′,A0\)→\(0,A0\)\(\\varepsilon B^\{\\prime\},A\_\{0\}\)\\to\(0,A\_\{0\}\), so the quotient is not a topological disjoint union, and no universal family is built\.

Most LoRA initialisation schemes in the literature are points of these three strata \(Table[4](https://arxiv.org/html/2610.12728#S2.T4)\)\. LoRA\-FA and MiCA freeze one factor and are sub\-objects \([Background A\.15](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\), and the last three rows of the table lie outside the strata\.

Table 4:LoRA initialisation schemes as points of the strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), with the sub\-objects and the exceptions\. Conventions:y=W​xy=Wx,W∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}; HF is Hugging Face PEFT\.WHAT THIS EXPLAINSZero\-init LoRA starts at a*singular point*\. At first order theAA\-direction is dead, so LoRA and LoRA\-FA \(which freezesAAand trainsBB;[Zhang et al\. \(2023d\)](https://arxiv.org/html/2610.12728#bib.bib556)use AdamW\), started from the sameA0A\_\{0\}, have the sameKq0K\_\{q\_\{0\}\}under plain gradient steps and the same first\-order trajectory under Adam \([Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)\)\. Splitting initialisations start at a smooth point with full first\-order reach\. We conjecture that this is one mechanism behind the faster early progress reported for PiSSA and LoRA\-GA\. LoRA\-GA attributes its speed\-up to gradient alignment, and MiLoRA also starts at a smooth point, so the smooth start alone does not account for the reported gains\. The strata classify*expressivity together withS1S\_\{1\}*\. What an optimizer additionally sees is the subject of[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\.

[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158)compare the two zero\-factor starts, “Init\[A\]” and “Init\[B\]”, through their training dynamics; hereS1S\_\{1\}already tells them apart\. The facts aboutℳ≤r\\mathcal\{M\}\_\{\\leq r\}used in the proof are classical \([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\.

Remark\(the first\-order deficit and the data\)\.S1S\_\{1\}counts directions in weight space\. The lazy\-regime kernelJf​d​ρ​Λ​d​ρ⊤​Jf⊤J\_\{f\}\\,d\\rho\\,\\Lambda\\,d\\rho^\{\\top\}J\_\{f\}^\{\\top\}also involves the network JacobianJfJ\_\{f\}and the optimizer’s metricΛ\\Lambda, and for random\-A0A\_\{0\}zero\-init LoRA it approximates the full fine\-tuning kernel\([Malladi et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib309)\)\. A large first\-order deficit need not cost much on the data\.

#### Counting first\-order directions

A first step of LoRA from\(B,A\)\(B,A\)addss⁡\(B˙​A\+B​A˙\)s\(\\dot\{B\}A\+B\\dot\{A\}\), a matrix with rows inrow⁡A\\operatorname\{row\}Aplus one with columns incol⁡B\\operatorname\{col\}B\. Counting both families and their overlap givesdimS1\\dim S\_\{1\}\.

Proposition II\.3\(first\-order rank formula: smoothness, alignment and scale\)\.Forκ⁡\(B,A\)=Wbase\+s​B​A\\kappa\(B,A\)=W\_\{\\rm base\}\+sBAwiths≠0s\\neq 0,rank⁡d​κ\(B,A\)=m​rk​A\+n​rk​B−rk⁡A​rk​Bfor all​\(B,A\),\\rk d\\kappa\_\{\(B,A\)\}=m\\operatorname\{rk\}A\+n\\operatorname\{rk\}B\-\\operatorname\{rk\}A\\operatorname\{rk\}B\\qquad\\text\{for all \}\(B,A\),\(3\)and\{X​A\}∩\{B​Y\}=\{B​R​A:R∈ℝr×r\}\\\{XA\\\}\\cap\\\{BY\\\}=\\\{BRA:R\\in\\mathbb\{R\}^\{r\\times r\}\\\}\. For full\-rank frames this overlap is the tangent space of the productFA⁡\(A\)×FB⁡\(B\)\\mathrm\{FA\}\(A\)\\times\\mathrm\{FB\}\(B\)of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\), re\-based atκ⁡\(B,A\)\\kappa\(B,A\), whose image is the meet of the two one\-sided images: a generalised LoRA\-XS\. It is LoRA\-XS proper when\(B,A\)\(B,A\)span the top\-rrsingular subspaces ofW0W\_\{0\}\(σr\>0\\sigma\_\{r\}\>0\), as at a PiSSA init; bottom frames \(MiLoRA\) give a generalised LoRA\-XS\.Under gradient flowB˙=−ηB​∇BL\\dot\{B\}=\-\\eta\_\{B\}\\nabla\_\{B\}L,A˙=−ηA​∇AL\\dot\{A\}=\-\\eta\_\{A\}\\nabla\_\{A\}L, withG=∇WLG=\\nabla\_\{W\}L,dd​t​L=−s2​\(ηB​‖G​A⊤‖F2\+ηA​‖B⊤​G‖F2\)\\tfrac\{d\}\{dt\}L=\-s^\{2\}\\big\(\\eta\_\{B\}\\\|GA^\{\\top\}\\\|\_\{F\}^\{2\}\+\\eta\_\{A\}\\\|B^\{\\top\}G\\\|\_\{F\}^\{2\}\\big\)\(4\)at every time, in particular at\(B0,A0\)\(B\_\{0\},A\_\{0\}\)\. To first order in the step sizes the same holds for full\-batch gradient descent, and in expectation for unbiased minibatch SGD\. Under Adam \(ε→0\\varepsilon\\to 0, with bias correction\) the first step changes the loss by−\|s\|​\(ηB​‖G​A0⊤‖1\+ηA​‖B0⊤​G‖1\)\+O⁡\(η2\),\-\\lvert s\\rvert\\big\(\\eta\_\{B\}\\\|GA\_\{0\}^\{\\top\}\\\|\_\{1\}\+\\eta\_\{A\}\\\|B\_\{0\}^\{\\top\}G\\\|\_\{1\}\\big\)\+O\(\\eta^\{2\}\),\(5\)with entrywiseℓ1\\ell\_\{1\}norms\. This first\-order term is linear in\|s\|\\lvert s\\rvert, against quadratic under gradient flow\.

Numerically, the rank formula \([3](https://arxiv.org/html/2610.12728#S2.E3)\) holds in400400random cases, including rank\-deficient factors, and the rate \([4](https://arxiv.org/html/2610.12728#S2.E4)\) matches finite differences \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

Example\(one attention projection of a 7B model\)\. At4096×40964096\\times 4096withr=16r=16, zero\-factor initialisations \(LoRA, QLoRA, EVA, LoRA\-FA\) havem​r=65,536mr=65\{,\}536first\-order directions, and split ones haver⁡\(m\+n−r\)=130,816r\(m\+n\-r\)=130\{,\}816, the dimension of LoRA’s image\.

WHAT THIS EXPLAINSZero\-factor initialisations \(LoRA, QLoRA, EVA, LoRA\-FA\) havem​rmrfirst\-order directions; split initialisations haver⁡\(m\+n−r\)r\(m\+n\-r\)\. “Better initialisation” in the literature mixes three distinct mechanisms, and the proposition separates them\.

##### Smoothness

PiSSA and LoRA\-GA start off the apex, so their first step sees allr⁡\(m\+n−r\)r\(m\+n\-r\)directions\.

##### Alignment

Under gradient flow, amongA0A\_\{0\}withA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cI, the termηB​‖G​A0⊤‖F2\\eta\_\{B\}\\\|GA\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\}of the initial rate \([4](https://arxiv.org/html/2610.12728#S2.E4)\), and with it the whole rate at a zero\-BBstart, is maximised by the top\-rrright singular space ofGG\. LoRA\-GA takes exactly this forA0A\_\{0\}, but takesB0B\_\{0\}from the left singular vectorsr\+1,…,2​rr\+1,\\dots,2r\(its Algorithm 1; HF’s default direction="ArB2r"\)\.GGthen has no component in the overlap\{B0​R​A0\}\\\{B\_\{0\}RA\_\{0\}\\\}, so the first gradient step approximates the rank\-2​r2rtruncation ofGG\. LoRA\-GA optimises gradient approximation, and gives up part of theBB\-term of the rate to do so\. EVA uses the top principal subspace of the input activations\. Its Theorem 3\.2 assumes the inputs independent of the upstream errors and bounds the expected per\-sample squared gradient norm with the uncentred𝔼⁡\[x​x⊤\]\\mathbb\{E\}\[xx^\{\\top\}\]\. That quantity is not the rate‖𝔼⁡\[G\]​A0⊤‖2\\\|\\mathbb\{E\}\[G\]A\_\{0\}^\{\\top\}\\\|^\{2\}, and HF’s EVA uses mean\-centred PCA, so EVA’s frame is a data\-only proxy\. Alignment is a gradient\-flow and SGD notion\. Under Adam the first\-step rate‖G​A0⊤‖1\\\|GA\_\{0\}^\{\\top\}\\\|\_\{1\}of \([5](https://arxiv.org/html/2610.12728#S2.E5)\) depends on the basis ofrow⁡A0\\operatorname\{row\}A\_\{0\}, not only on the subspace\.

##### Scale

‖G​A0⊤‖F2\\\|GA\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\}grows with‖A0‖2\\\|A\_\{0\}\\\|^\{2\}, and initial scale trades off against learning rate \([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\. PiSSA’sSr1/2S\_\{r\}^\{1/2\}\-weighted factors change it\. LoRA\-GA’s paper scales both factors bydout1/4/γd\_\{\\rm out\}^\{1/4\}/\\sqrt\{\\gamma\}; HF’s default scale="stable" scales onlyBB\(which also carriesSr\+1:2​rS\_\{r\+1:2r\}\), so there‖G​A0⊤‖F2\\\|GA\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\}is unchanged\. HF’s Kaiming default hasA0​A0⊤≈I/3A\_\{0\}A\_\{0\}^\{\\top\}\\approx I/3, againstA0​A0⊤=IA\_\{0\}A\_\{0\}^\{\\top\}=Ifor EVA with whiten=False\. Comparing rates across initialisations needsA0​A0⊤A\_\{0\}A\_\{0\}^\{\\top\}normalised\.

LoRA\-GA uses all three mechanisms, with alignment on theAAside only\. The gradient\-aligned frame is due to[Wang et al\. \(2024e\)](https://arxiv.org/html/2610.12728#bib.bib472), the activation frame to[Paischer et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib358), and the spectral split to[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324);[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)only sorts their effects into three terms\.

#### The zero gate

At a zero\-BBstart,B​ABAis linear inBBandB0=0B\_\{0\}=0, so movingAAalone changes nothing\. The same holds whenever a trainable gate starts closed\. Below,K=D​ρ​Λ​D​ρ⊤K=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}is the cometric on weights induced by an optimizer’s metricΛ\\Lambda\([Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3), Exposé[III](https://arxiv.org/html/2610.12728#S3)\)\.

Lemma II\.4\(zero gate\)\.LetQ=U×VQ=U\\times VwithUUa vector space, letρ=θ0\+β⁡\(u,v\)\\rho=\\theta\_\{0\}\+\\beta\(u,v\)withβ\\betasmooth, and supposeβ⁡\(u0,v\)=0\\beta\(u\_\{0\},v\)=0for allvv; for exampleβ⁡\(⋅,v\)\\beta\(\\cdot,v\)linear andu0=0u\_\{0\}=0\. ThenDv​ρq0=0D\_\{v\}\\rho\_\{q\_\{0\}\}=0\. Hence∇v\(L∘ρ\)​\(q0\)=0\\nabla\_\{v\}\(L\\circ\\rho\)\(q\_\{0\}\)=0,S1=Du​β​\(u0,v0\)​\(U\)S\_\{1\}=D\_\{u\}\\beta\(u\_\{0\},v\_\{0\}\)\(U\)\(=β⁡\(U,v0\)=\\beta\(U,v\_\{0\}\)in the linear case\), and the cometric atq0q\_\{0\}isKq0=Du​β​Λu​u​Du​β⊤K\_\{q\_\{0\}\}=D\_\{u\}\\beta\\,\\Lambda\_\{uu\}\\,D\_\{u\}\\beta^\{\\top\}, whereΛu​u\\Lambda\_\{uu\}is theu​uuu\-block ofΛ\\Lambda; no block\-diagonality is needed, since thevv\-columns ofD​ρD\\rhovanish\. The weight trajectoryθ⁡\(t\)\\theta\(t\)agrees with that of the method withvvfrozen atv0v\_\{0\}to first order intt\(gradient flow\), or in the step size \(one or more Adam steps\)\. The parametervvitself may still move: under terms not of the formL∘ρL\\circ\\rho\(regularisers, decoupled weight decay\), and under Adam, whose step\-2 move ofvvisO⁡\(η\)O\(\\eta\)although its effect onθ\\thetaisO⁡\(η2\)O\(\\eta^\{2\}\)\.*Function\-level variant\.*For an extension with realised functionF⁡\(u,v,x\)F\(u,v,x\), ifF⁡\(u0,v,x\)=F0​\(x\)F\(u\_\{0\},v,x\)=F\_\{0\}\(x\)for allvvandxx, thenDv​F=0D\_\{v\}F=0atq0q\_\{0\}, with the same conclusions at the level of functions\. No linearity is needed\.

Numerically, under Adam withη∈\{10−2,10−3,10−4\}\\eta\\in\\\{10^\{\-2\},10^\{\-3\},10^\{\-4\}\\\}, the step\-2 move ofAAin zero\-init LoRA scales asη\\eta, while the deviation ofΔ​W\\Delta Wfrom the frozen\-AAmethod scales asη2\\eta^\{2\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

Instances, assuming no weight decay onvv:

- •LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\), withu=Bu=B\.
- •VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\), withu=bu=b:θ\\thetafirst moves only alongbb\.
- •AdaLoRA\([Zhang et al\., 2023g](https://arxiv.org/html/2610.12728#bib.bib559)\), whose update isP​diag⁡\(λ\)​QP\\diag\(\\lambda\)Q, withu=λu=\\lambda, the diagonal of singular values, started at00\.θ\\thetainitially moves only along therrrank\-one directionspi​qi⊤p\_\{i\}q\_\{i\}^\{\\top\}, sodimS1=r\\dim S\_\{1\}=rfor genericP,QP,Q\.PPandQQdo move at step one, through AdaLoRA’s orthogonality penaltyR⁡\(P,Q\)R\(P,Q\)\(HF default orth\_reg\_weight=0\.5\), which has no first\-order effect onθ\\theta\.
- •LoHa\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\)with a zero factor \(HF zero\-initialises hada\_w2\_b\): only that factor moves\.
- •By the function\-level variant, the gate of LLaMA\-Adapter\([Zhang et al\., 2023h](https://arxiv.org/html/2610.12728#bib.bib562)\), which is the only parameter that moves at first order, and bottleneck adapters whose up\-projection starts at zero, where only that factor moves \([Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib175)start near that point\)\.

HRA’s paired initialisation has a different form and shows the same first\-order collapse \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\)\)\.

[Zhu et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib598)find over whole runs that a random, untrainedAAdoes nearly as well as a tuned one, and that trainingBBalone gives better generalisation bounds\.[Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)is only the first\-order statement\.

#### Off the point: quantised bases

QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)quantises the frozen weight, so it is LoRA pointed atκ​θ0\\kappa\\theta\_\{0\}, with*pointing defect*e=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\([Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1),[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(a\)\)\. Aseeis generically of full rank,θ0\\theta\_\{0\}lies outsideIm⁡LoRAr​\(κ​θ0\)\\Imop\\mathrm\{LoRA\}\_\{r\}\(\\kappa\\theta\_\{0\}\)unlessrank⁡e≤r\\rk e\\leq r, which is why[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)speaks of*exact*pointings\. When the initialisation also splits, the order of operations decides the defect \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\)\)\. In Table[5](https://arxiv.org/html/2610.12728#S2.T5),WWis the full\-precision weight,e=κ​W−We=\\kappa W\-Wandδ0=δ⁡\(q0\)\\delta\_\{0\}=\\delta\(q\_\{0\}\)\.

Table 5:Pointing defects of quantised\-base LoRA pipelines \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\)\)\.Remark\(LoftQ’s scale in HF PEFT\)\. HF PEFT’s LoftQ initialiser does not divide by the LoRA scaless\(α/r\\alpha/r, orα/r\\alpha/\\sqrt\{r\}under rsLoRA\), so fors≠1s\\neq 1the deployed defect ise−s​SVDr​\(e\)e\-s\\,\\mathrm\{SVD\}\_\{r\}\(e\)\. Ats=2s=2its Frobenius norm equals‖e‖\\\|e\\\|, and the fit buys nothing over QLoRA in that norm\.

Quantised bases for LoRA come from[Dettmers et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib92), fitting the adapter to the quantisation error from[Li et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib255), and the quantised PiSSA variant from[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324)\.

The strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)classify what a start can express\. The dynamics see a finer quotient\. For a zero\-BBstart, SGD seesA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}and Adam seesA0A\_\{0\}up to signed permutations of its rows \([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\. A dynamical classification of initialisations, including the degenerate pointings, is open \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

### II\.B Erlangen: invariants of group\-type methods

A geometry is what a group leaves unchanged\. Before asking how a method learns, ask what it cannot help keeping\. after Klein

A method that moves the pretrained weight by a group element can never change that group’s invariants\. Orbits also place HRA as a meet, DoRA as a saturation and the scaling methods on a ladder\.

##### In brief

- •A method whose weights are alwaysW0​RW\_\{0\}R, withRRexactly orthogonal, can never changeW​W⊤WW^\{\\top\}, so neuron norms, angles and singular values survive any number of merges in exact arithmetic\. HF PEFT’s default OFT, BOFT’s scale and GOFT’s default scaler break this\.
- •For injectiveW0W\_\{0\}and evenr≤n−2r\\leq n\-2, HRA reaches exactly the rank\-rrLoRA updates of the formW0​RW\_\{0\}R,R∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)\. Its HF PEFT default initialisation is an apex\.
- •DoRA reaches exactly the weights of LoRA plus a free gain per output neuron\. From the same start, any gap between the two comes from training\.
- •\(IA\)3, SSF’s scales and DoRA’s magnitude lie in the image of rank\-one HiRA,W0⊙\(J\+B​A\)W\_\{0\}\\odot\(J\+BA\), which forW0W\_\{0\}without zeros is LoRA seen throughX↦W0⊙XX\\mapsto W\_\{0\}\\odot X\.

Klein’s Erlangen programme of 1872\([Klein, 1872](https://arxiv.org/html/2610.12728#bib.bib217)\)\([Background D\.17](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\) defines a geometry by its group of motions and studies what the motions leave unchanged\. OFT, GOFT, HRA and ETHER multiply the pretrained matrix by an orthogonal matrix, \(IA\)3and SSF’s scale multiply it by a diagonal one, and DoRA’s magnitude does the same to a LoRA weight\. These are methods of*action type*\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\),ρ⁡\(q\)=γ⁡\(q\)⋅W0\\rho\(q\)=\\gamma\(q\)\\cdot W\_\{0\}withγ\\gammalanding in a group\. Whatever that group cannot change, the method cannot change, in training or after any number of merges\.

Klein’s programme is the model\.[Bronstein et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib44)made it the organising principle of geometric deep learning, for architectures\. Here it is applied to the fine\-tuning map\.

Analogy\(what “Erlangen” means here\)\. For a group\-type method the Erlangen reading is a theorem, since its image lies in one orbit\. LoRA’s coneW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}has a singular vertex atW0W\_\{0\}\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(c\)\), so it is no orbit; for cones the word names the shape of the image and its covariance group \([Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\)\.

One convention matters\. A linear box computesy=W​xy=WxwithW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}, and rowiiofWWis output neuronii\. The two Gram matrices, over neurons and over inputs, are

Kout​\(W\)=W​W⊤,Kin​\(W\)=W⊤​W\.K\_\{\\mathrm\{out\}\}\(W\)=WW^\{\\top\},\\qquad K\_\{\\mathrm\{in\}\}\(W\)=W^\{\\top\}W\.A right multiplicationW↦W​RW\\mapsto WRacts on the input side and fixesKoutK\_\{\\mathrm\{out\}\}; a left multiplicationW↦P​WW\\mapsto PWacts on the output side and fixesKinK\_\{\\mathrm\{in\}\}\. Papers that storeW⊤W^\{\\top\}swap these words, and we translate every method into this convention\.

#### Complete invariants

A complete invariant of a group action \([Background D\.15](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\) agrees on two matrices exactly when one can be carried to the other\. For the five actions in PEFT it is classical\. We writeTmT\_\{m\}for the invertible diagonal matrices,Tm\+=\{diag⁡\(d\):di\>0\}T\_\{m\}^\{\+\}=\\\{\\diag\(d\):d\_\{i\}\>0\\\}for its identity component \([Background D\.4](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\), and⊙\\odotfor the entrywise product\.

Theorem II\.5\(complete invariants of the basic actions; classical\)\.LetW,W′∈ℝm×nW,W^\{\\prime\}\\in\\mathbb\{R\}^\{m\\times n\}\.\(1\)*Right orthogonal \(input side\)\.*W′=W​RW^\{\\prime\}=WRfor someR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)iffW​W⊤=W′​W′⁣⊤WW^\{\\top\}=W^\{\\prime\}W^\{\\prime\\top\}\. ForSO⁡\(n\)\\mathrm\{SO\}\(n\), whenrank⁡W=n\\rk W=n, also require that the then uniqueRRhavedetR=1\\det R=1\.\(2\)*Left orthogonal\.*W′=P​WW^\{\\prime\}=PWfor someP∈O⁡\(m\)P\\in\\mathrm\{O\}\(m\)iffW⊤​W=W′⁣⊤​W′W^\{\\top\}W=W^\{\\prime\\top\}W^\{\\prime\}\.\(3\)*Two\-sided\.*W′=P​W​RW^\{\\prime\}=PWRfor someP∈O⁡\(m\)P\\in\\mathrm\{O\}\(m\)andR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)iffWWandW′W^\{\\prime\}have the same singular values, counted with multiplicity\.\(4\)*Left torus\.*IfWWhas no zero row,W′=D​WW^\{\\prime\}=DWfor someD∈Tm\+D\\in T\_\{m\}^\{\+\}iffwi′∈ℝ\>0​wiw\_\{i\}^\{\\prime\}\\in\\mathbb\{R\}\_\{\>0\}\\,w\_\{i\}for everyii\. For the full torusTmT\_\{m\}, whose signs are unconstrained, the complete invariant is the set of zero rows together with the row*lines*\.\(5\)*Full Hadamard torus\.*W′=E⊙WW^\{\\prime\}=E\\odot Wfor someE∈\(ℝ×\)m×nE\\in\(\\mathbb\{R\}^\{\\times\}\)^\{m\\times n\}iffWWandW′W^\{\\prime\}have the same zero pattern\.

Item \(1\) is the orbit\-separation consequence of the first fundamental theorem of invariant theory forO⁡\(n\)\\mathrm\{O\}\(n\)\([Background D\.16](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\); for a compact group, invariants separate orbits\. Equivalently, it is Witt\-type extension of isometries\. The two\-sided diagonal torus\{diag⁡\(a\)​X​diag⁡\(b\)\}≅\(Tm×Tn\)/ℝ×\\\{\\diag\(a\)\\,X\\,\\diag\(b\)\\\}\\cong\(T\_\{m\}\\times T\_\{n\}\)/\\mathbb\{R\}^\{\\times\}that appears as LoHa’s gauge \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\) is a different group from the full Hadamard torus in \(5\)\.

##### Caveat for methods

No PEFT method realises \(4\) or \(5\) as a group action on all of its parameters\. \(IA\)3entries and DoRA magnitudes are unconstrained reals initialised at11or at the row norms, so the relevant group in \(4\) isTmT\_\{m\}or its closure\. HiRA’s factorJ\+B​AJ\+BAcan vanish entrywise\. These methods can create new zeros, so for them only a one\-way statement transfers: zeros ofW0W\_\{0\}remain zero\. Table[6](https://arxiv.org/html/2610.12728#S2.T6)lists the five actions with the methods that live in each\.

Table 6:The five basic actions, their complete invariants \([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\) and the methods whose images lie in one orbit or orbit closure\.WHAT THIS EXPLAINSMoving a method from the input side to the output side tradesKoutK\_\{\\mathrm\{out\}\}forKinK\_\{\\mathrm\{in\}\}\. Letting a scale change sign trades rays for lines, and letting it reach zero leaves only the zeros ofW0W\_\{0\}\.

The interactive companion has a*Gram lab*for these invariants\. It fixes an invertible4×44\\times 4matrixW0W\_\{0\}with one zero entry and distinct singular values\. The reader picks one of ten methods and drags, types or randomises its parameters\. Each lamp is a float64 test, to10−1010^\{\-10\}, that an invariant ofW=ρ⁡\(q\)W=\\rho\(q\)\(the neuron Gram matrix, the cosines, the input Gram matrix, the singular values, the row lines, the zeros ofW0W\_\{0\}\) still equals its value atW0W\_\{0\}, and a further lamp testsrank⁡Δ​W≤2\\rk\\Delta W\\leq 2\. A lamp guaranteed by a theorem is marked as such, and a red lamp names the hypothesis that was broken\. Heatmaps, neuron arrows and strips showKoutK\_\{\\mathrm\{out\}\}, the cosines,KinK\_\{\\mathrm\{in\}\},WW,σ⁡\(W\)\\sigma\(W\), the rank ofΔ​W\\Delta WanddimS1\\dim S\_\{1\}againstdd\. OFT’s Cayley–Neumann switch \(HF’s default\), BOFT’ss1↦−s1s\_\{1\}\\mapsto\-s\_\{1\}, \(IA\)3’sℓ2↦0\\ell\_\{2\}\\mapsto 0and DoRA’sB↦0B\\mapsto 0each break or restore a hypothesis\.

#### What exactly orthogonal methods can never do

If every weight a method reaches isW0​RW\_\{0\}RwithRRorthogonal,[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(1\) and \(3\) freeze the neuron Gram matrix and the singular values, and merging cannot help, since products of orthogonal matrices are orthogonal\. The work lies in saying which implementations are exactly orthogonal, and on which side\.

Corollary II\.6\(what exactly orthogonal methods can never do\)\.Let a method have image inW0⋅O⁡\(n\)W\_\{0\}\\cdot\\mathrm\{O\}\(n\)\(input side\), and work in exact arithmetic\. This covers:•OFT and block OFT with an exactly orthogonal parametrisation: the original paper’s exact Cayley map, and HF PEFT from 0\.16 with use\_cayley\_neumann=False on Linear and Conv targets;•GOFT with strict Givens rotations and no scaler, as in the paper \(no\_scaling=True in the official peft\-givens code\);•HRA, and ETHER as defined in its paper \(not ETHER\+\);•any number of merge\-and\-restart cycles of these\.Then:•it preservesKoutK\_\{\\mathrm\{out\}\}, hence every neuron norm and the angle between any two nonzero neurons, and so the hyperspherical energy, which depends only on these angles;•it preserves the whole singular spectrum ofW0W\_\{0\};•block\-diagonal OFT with input blocksJJ, with the same partition in every cycle, preserves each partial Gram matrixW:,JW:,J⊤W\_\{:,J\}W\_\{:,J\}^\{\\top\}\.

Several methods that look orthogonal fail the hypothesis\. Each item below is a separate computation, and Table[7](https://arxiv.org/html/2610.12728#S2.T7)summarises them\.

Remark\(scope of Corollary II\.6; these statements do not follow from its hypothesis\)\.•*BOFT*as implemented in HF PEFT,W=diag⁡\(s\)​W0​R⊤W=\\diag\(s\)W\_\{0\}R^\{\\top\}withRRexactly orthogonal, hasKout​\(W\)=diag⁡\(s\)​Kout​\(W0\)​diag⁡\(s\)K\_\{\\mathrm\{out\}\}\(W\)=\\diag\(s\)K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\\diag\(s\)\. Socosi​j′=sign\(sisj\)cosi​j\\cos^\{\\prime\}\_\{ij\}=\\operatorname\{sign\}\(s\_\{i\}s\_\{j\}\)\\cos\_\{ij\}wheneversi​sj≠0s\_\{i\}s\_\{j\}\\neq 0\. It preserves\|cos\|\\lvert\\cos\\rvertwhen everysis\_\{i\}is nonzero, and preservescosi​j\\cos\_\{ij\}wheneversi​sj\>0s\_\{i\}s\_\{j\}\>0\. Hence it preserves every cosine when allsis\_\{i\}are nonzero and share a sign \(for instances∈Tm\+s\\in T\_\{m\}^\{\+\}, the constraint in the re\-scaled OFT of[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib381), Sec\. 3\.5\); when no cosine ofW0W\_\{0\}is zero, that condition is also necessary\. HF’s boft\_s is an unconstrained real initialised at11\. At the default boft\_n\_butterfly\_factor=1 the butterfly permutation is the identity, BOFT is block OFT times an output scale, and each input block satisfiesW:,JW:,J⊤=diag\(s\)\(W0\):,J\(W0\):,J⊤diag\(s\)W\_\{:,J\}W\_\{:,J\}^\{\\top\}=\\diag\(s\)\(W\_\{0\}\)\_\{:,J\}\(W\_\{0\}\)\_\{:,J\}^\{\\top\}\\diag\(s\)\. HF PEFT OFT 0\.14 to 0\.15 \(exact Cayley, input side\) carried a trainable output scale oft\_s and has the same form\.•*GOFT’s official code*by default \(no\_scaling=False\) adds a trainable input\-side diagonalgg, initialised at ones\. ThenW=W0​diag⁡\(g\)​GW=W\_\{0\}\\diag\(g\)GandKout​\(W\)=W0​diag⁡\(g\)2​W0⊤K\_\{\\mathrm\{out\}\}\(W\)=W\_\{0\}\\diag\(g\)^\{2\}W\_\{0\}^\{\\top\}, whatever the Givens partGGis\. This is the paper’s GOFT\* ablation, and it is not covered\.•*Output\-side implementations*\(LyCORIS diag\-OFT at its defaults; HF PEFT OFT up to 0\.13, which merged toR⊤​W0R^\{\\top\}W\_\{0\}; HF OFT on embedding layers; ETHER with flip\_side=True\) have image inO⁡\(m\)⋅W0\\mathrm\{O\}\(m\)\\cdot W\_\{0\}\. They preserveKinK\_\{\\mathrm\{in\}\}, the partial input GramsWI,:⊤WI,:W\_\{I,:\}^\{\\top\}W\_\{I,:\}and the spectrum, and they do not preserveKoutK\_\{\\mathrm\{out\}\}\. ETHER’s official code with ether\_nb\>1 applies a different reflection to each output\-row group, which keeps neuron norms and within\-group angles but neither the cross\-group cosines nor the spectrum\.•*HF PEFT’s default OFT*since 0\.18 \(use\_cayley\_neumann=True, 5 terms\) buildsR=I\+2​Q\+2​Q2\+2​Q3\+Q4=C⁡\(Q\)​\(I−Q4\)R=I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}=C\(Q\)\(I\-Q^\{4\}\), whereC⁡\(Q\)=\(I\+Q\)​\(I−Q\)−1C\(Q\)=\(I\+Q\)\(I\-Q\)^\{\-1\}is the exact Cayley map \(HF’s exact branch evaluated at−Q\-Q\)\. HenceR⊤​R=\(I−Q4\)2≠IR^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}\\neq I, and the singular values ofRRare\|1−θk4\|\\lvert 1\-\\theta\_\{k\}^\{4\}\\rvert, where±i​θk\\pm i\\theta\_\{k\}are the eigenvalues ofQQ: a systematic contraction for smallQQ, andRRis singular when someθk=1\\theta\_\{k\}=1\. Already the adapted weight hasKout=W0​\(I−Q4\)2​W0⊤K\_\{\\mathrm\{out\}\}=W\_\{0\}\(I\-Q^\{4\}\)^\{2\}W\_\{0\}^\{\\top\}, a drift of order∥Q∥4\\lVert Q\\rVert^\{4\}, and merge cycles compound it\. For∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\mathrm\{op\}\}\\leq 2^\{1/4\}every factor is a contraction, so the drift is a monotone shrinkage of neuron norms and singular values \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 4\)\. In 0\.16 and 0\.17 the last coefficient was22, soR⊤​R=I\+4​Q6\+4​Q8R^\{\\top\}R=I\+4Q^\{6\}\+4Q^\{8\}, still not orthogonal, with drift of order∥Q∥6\\lVert Q\\rVert^\{6\}\. \(HF’s docstring says the default is False; the dataclass default is True\.\)•*Quasi\-orthogonal variants*preserve neitherKoutK\_\{\\mathrm\{out\}\}nor its normalised form exactly: qGOFT, whose2×22\\times 2blocks are arbitrary matrices with only a soft orthogonality penalty; ETHER\+, which is two\-sided and whose factorI−u​u⊤\+v​v⊤I\-uu^\{\\top\}\+vv^\{\\top\}is singular whenu⟂vu\\perp v; and the default HF OFT above\. Oncen\>2n\>2, a single non\-orthogonal2×22\\times 2block, conformal or not, generically changes some cosine\.•*Finite precision\.*The invariance is exact only in exact arithmetic; low\-precision merges add a rounding drift that grows like a random walk over merge cycles\.

##### Numerical check

Withs=\(−0\.5,1,…,1\)s=\(\-0\.5,1,\\dots,1\), BOFT changes a cosine by1\.51\.5while\|cos\|\\lvert\\cos\\rvertis fixed to4×10−164\\times 10^\{\-16\}, and an all\-negativessfixes every cosine to10−1510^\{\-15\}\. GOFT with a scalerg=1±0\.05g=1\\pm 0\.05moves cosines and singular values\. A conformal block2​Rot​\(0\.3\)\\sqrt\{2\}\\,\\mathrm\{Rot\}\(0\.3\)on two of88input coordinates changes some cosine, by0\.170\.17\. HF’s default OFT satisfiesR⊤​R=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}to5×10−165\\times 10^\{\-16\}, with∥R⊤​R−I∥=1\.5×10−2\\lVert R^\{\\top\}R\-I\\rVert=1\.5\\times 10^\{\-2\}already for block size88and skew entries of size about0\.070\.07, and it is singular atθ=1\\theta=1\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

Table 7:Implementations that look orthogonal, read against[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\. Each row is a separate computation \(see the remarks after the corollary\)\.
##### Credit

[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib381)introduced orthogonal fine\-tuning, motivated by preserving hyperspherical energy\([Liu et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib283)\); their re\-scaled OFT \(Sec\. 3\.5\) adds the positive per\-neuron scale that HF BOFT carries\. BOFT is due to[Liu et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib285), building on butterfly factorisations\([Dao et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib88)\); GOFT and qGOFT are due to[Ma et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib305), with an official peft\-givens implementation; ETHER and ETHER\+ to[Bini et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib36); HRA to[Yuan et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib540)\. LyCORIS is described by[Yeh et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib532)\. The Cayley–Neumann default, BOFT’s scale and the output\-side merge are read from the Hugging Face PEFT source\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)\(the OFT configuration and the BOFT layer\)\.

WHAT THIS EXPLAINS“Orthogonal fine\-tuning preserves knowledge” corresponds to preserving a complete invariant: in exact arithmetic an exactly orthogonal method can*never*change spectra or neuron geometry, through any number of merges \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 7\), up to rounding\. So a variant that must change them adds a non\-orthogonal factor \(the per\-neuron scale of Qiu et al\.’s re\-scaled OFT, as in HF BOFT and HF OFT 0\.14 to 0\.15, or GOFT’s default input scaler\)\. HF’s default OFT is only close to orthogonal; with block size88and fresh skew generators with entries near0\.050\.05in every cycle, its merges shrinkKoutK\_\{\\mathrm\{out\}\}by about0\.2%0\.2\\%,1\.3%1\.3\\%,6%6\\%and22%22\\%after11,1010,5050and200200cycles \(relative Frobenius change, averaged over draws\)\. The statement concerns reach\. It does not show that preserving the invariant causes knowledge retention, and the control study of ETHER\([Bini et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib36)\)found a non\-orthogonal multiplicative variant comparably stable\. Earlier claims that BOFT or qGOFT preserve the full Gram matrix are corrected here\.

#### HRA is a meet of two images

A Householder reflectionHu=I−2​u​u⊤/∥u∥2H\_\{u\}=I\-2uu^\{\\top\}/\\lVert u\\rVert^\{2\}is orthogonal and differs fromIIby rank one\. SoW0Hu1⋯HurW\_\{0\}H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}lies in the orthogonal orbit ofW0W\_\{0\}and in LoRA’s coneW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, and for injectiveW0W\_\{0\}and evenrrthe Cartan–Dieudonné theorem \([Background D\.11](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) says that HRA fills the cone’s intersection with the rotation orbit\. HF PEFT repeats each reflection at initialisation, so to first order each pair can only turn its own vector against the rest of the space, and HRA starts at an apex, as zero\-init LoRA does \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\.

Theorem II\.7\(HRA is the meet of LoRA with the rotation orbit, and it starts at an apex\)\.LetW0∈ℝm×nW\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}be injective \(rank⁡W0=n\\rk W\_\{0\}=n\), and letrrbe even with2≤r≤n−22\\leq r\\leq n\-2\. DefineHRAr:\(u1,…,ur\)↦W0Hu1⋯Hur\\mathrm\{HRA\}\_\{r\}:\(u\_\{1\},\\dots,u\_\{r\}\)\\mapsto W\_\{0\}H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}withui≠0u\_\{i\}\\neq 0, whereHu=I−2​u​u⊤/∥u∥2H\_\{u\}=I\-2uu^\{\\top\}/\\lVert u\\rVert^\{2\}\. WriteOFTSO:=W0⋅SO⁡\(n\)\\mathrm\{OFT\}\_\{\\mathrm\{SO\}\}:=W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)for the rotation orbit\. \(TheO⁡\(n\)\\mathrm\{O\}\(n\)\-orbit would add the odd component:W0​He1W\_\{0\}H\_\{e\_\{1\}\}lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}but not inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\.\)\(a\)Meet\.Im⁡HRAr=\(W0\+ℳ≤r\)∩W0⋅SO⁡\(n\)\\Imop\\mathrm\{HRA\}\_\{r\}=\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\. Further,d⁡\(HRAr\)=r​n−r⁡\(r\+1\)2d\(\\mathrm\{HRA\}\_\{r\}\)=rn\-\\tfrac\{r\(r\+1\)\}\{2\}, so the generic fibre has dimensionr⁡\(r\+1\)2=r​n−d\\tfrac\{r\(r\+1\)\}\{2\}=rn\-d\. This gauge is non\-linear\. The evident continuous global symmetries are therrscalingsui↦ci​uiu\_\{i\}\\mapsto c\_\{i\}u\_\{i\}; the braid group also acts globally by Hurwitz moves\(ui,ui\+1\)↦\(Hui​ui\+1,ui\)\(u\_\{i\},u\_\{i\+1\}\)\\mapsto\(H\_\{u\_\{i\}\}u\_\{i\+1\},u\_\{i\}\), sinceHa​Hb=HHa​b​HaH\_\{a\}H\_\{b\}=H\_\{H\_\{a\}b\}H\_\{a\}\.\(b\)First\-order collapse\.At the paired initialisationu2​i−1=u2​i=wiu\_\{2i\-1\}=u\_\{2i\}=w\_\{i\}\(i≤k=r/2i\\leq k=r/2, with thewiw\_\{i\}independent andU=span⁡\{wi\}U=\\spanop\\\{w\_\{i\}\\\}\),S1=W0⋅\{Ω∈𝔰​𝔬​\(n\):PU⟂​Ω​PU⟂=0\},dimS1=k​n−k⁡\(k\+1\)2<d⁡\(HRAr\)\.S\_\{1\}=W\_\{0\}\\cdot\\\{\\Omega\\in\\mathfrak\{so\}\(n\):P\_\{U^\{\\perp\}\}\\Omega P\_\{U^\{\\perp\}\}=0\\\},\\qquad\\dim S\_\{1\}=kn\-\\tfrac\{k\(k\+1\)\}\{2\}<d\(\\mathrm\{HRA\}\_\{r\}\)\.\(c\)Vertex\.The germ ofIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}atW0W\_\{0\}is diffeomorphic to the germ at00ofΣr=\{Ω∈𝔰​𝔬​\(n\):rank⁡Ω≤r\}\\Sigma\_\{r\}=\\\{\\Omega\\in\\mathfrak\{so\}\(n\):\\rk\\Omega\\leq r\\\}, the skew\-determinantal cone \([Background C\.13](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\), which is not linear\. SoW0W\_\{0\}is a singular point \(the vertex\) of the image\. Together with \(b\), the paired initialisation is anapex\.\(d\)Scope\.•*Not a product\.*\(a\) is a meet in the poset of subsets ofΘ\\ThetacontainingW0W\_\{0\}\([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\)\)\. HRA is not the categorical product ofLoRAr\\mathrm\{LoRA\}\_\{r\}andOFTSO\\mathrm\{OFT\}\_\{\\mathrm\{SO\}\}: the set\-level fibre product\{\(B,A,R\):W0\+B​A=W0​R\}\\\{\(B,A,R\):W\_\{0\}\+BA=W\_\{0\}R\\\}has local dimensiond\+r2=r​n\+r⁡\(r−1\)2≠r​nd\+r^\{2\}=rn\+\\tfrac\{r\(r\-1\)\}\{2\}\\neq rnover a generic point of the meet\. \(It is not pure\-dimensional: overW0W\_\{0\}alone it contains\{\(B,0,I\)\}\\\{\(B,0,I\)\\\}, of dimensionm​rmr\.\)•*Exactly orthogonal OFT is smaller\.*The Cayley map never produces a rotation with eigenvalue−1\-1\. For exact full\-block Cayley OFT \(the original paper; HF with use\_cayley\_neumann=False\),\(W0\+ℳ≤r\)∩Im⁡OFT\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\Imop\\mathrm\{OFT\}is dense inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}but not equal to it:W0​He1​He2W\_\{0\}H\_\{e\_\{1\}\}H\_\{e\_\{2\}\}, withHe1​He2=diag⁡\(−1,−1,1,…,1\)H\_\{e\_\{1\}\}H\_\{e\_\{2\}\}=\\diag\(\-1,\-1,1,\\dots,1\), is missing\. For exact block Cayley OFT withb<nb<nthe intersection is a proper subset ofIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\.•*HF’s default OFT is incomparable\.*HF PEFT’s default OFT is not orthogonal \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2); use\_cayley\_neumann=True\)\. SinceR⊤​R=\(I−Q4\)2=IR^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}=IforcesQ=0Q=0nearQ=0Q=0, its image meetsIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}nearW0W\_\{0\}only atW0W\_\{0\}, while its intersection withW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}contains non\-orthogonal points such asW0​\(I\+2​Q\+2​Q2\+2​Q3\+Q4\)W\_\{0\}\(I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}\)withrank⁡Q=2\\rk Q=2\.•*Implementation\.*The theorem concerns HRA withλ<∞\\lambda<\\infty\(HF apply\_GS=False\)\. The strictly orthogonal Gram–Schmidt variant has the smaller imageW0​\(I−2​PU\)W\_\{0\}\(I\-2P\_\{U\}\), of dimensionr⁡\(n−r\)r\(n\-r\)\. The paired initialisation is HF’s default \(repeat\_interleave\) and is not stated in the HRA paper\. HF HRA acts on in\_features, so injectivity requires out\_features≥\{\}\\geq\{\}in\_features \(and full column rank\); the theorem does not cover down\_proj or GQA key/value projections\. The Gram–Schmidt variant never containsW0W\_\{0\}, sinceI−2​PU≠II\-2P\_\{U\}\\neq I, and at HF’s default paired initialisation it runs Gram–Schmidt on duplicated columns, which is numerically degenerate\.

##### Numerical check

Atn=9n=9, the paired\-initialisation Jacobian has rank88against generic rank1515forr=2r=2, and1515against2626forr=4r=4\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

The sharp Cartan–Dieudonné theorem, that a rotation moving akk\-dimensional space is a product of exactlykkreflections, is due to[Scherk \(1950\)](https://arxiv.org/html/2610.12728#bib.bib399); see[Artin \(1957\)](https://arxiv.org/html/2610.12728#bib.bib16)\. Householder parametrisations in deep learning go back to[Mhammedi et al\. \(2016\)](https://arxiv.org/html/2610.12728#bib.bib326), and Lie\-algebra charts such as the Cayley map to[Lezcano\-Casado and Martínez\-Rubio \(2019\)](https://arxiv.org/html/2610.12728#bib.bib244)\.[Yuan et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib540)proposed HRA as a bridge between low\-rank and orthogonal adaptation, which the theorem makes precise\. The paired initialisation is read from the HF PEFT HRA layer\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)\.

WHAT THIS EXPLAINS

- •HRA’s “bridge between low\-rank and orthogonal adaptation” is, at the level of images, the meet of LoRA with the rotation orbitW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\. It is a meet of images and no product of methods\. It strictly contains LoRA∩\{\}\\cap\{\}OFT for exactly orthogonal \(Cayley\) OFT, as in the original paper or HF with use\_cayley\_neumann=False; for HF’s default Cayley–Neumann OFT the two are incomparable\. Compare[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\), where LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\)is a genuine categorical productFA×FB\\mathrm\{FA\}\\times\\mathrm\{FB\}\.
- •HF recommends an evenrrbecause its paired initialisation needs pairs, and for oddrrit falls back to a random Kaiming initialisation\. The determinant identitydetR=\(−1\)r\\det R=\(\-1\)^\{r\}explains why no odd\-rrHRA can start atW0W\_\{0\}, so that fallback is a pointing defect\.
- •New here \(earlier analyses labelled this base point smooth\):HRA’s default initialisation is an apex\. At first order, each pair can only rotate its own vector against the rest of the space\. This is the multiplicative twin of LoRA’s deadAA\-direction \([Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)\), so HRA belongs in the apex column beside LoRA\. Whether the collapse slows early training is open \(§[XI](https://arxiv.org/html/2610.12728#S11)\)\.
- •Rebasing commutes with the meet, so afterKKcycles re\-HRA reaches\(W0\+ℳ≤K​r\)∩W0⋅SO⁡\(n\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq Kr\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 5\)\.

In the Gram lab, choosing HRA and resetting toq0q\_\{0\}givesdimS1=3\\dim S\_\{1\}=3againstd=5d=5, the values of \(a\) and \(b\) atn=4n=4,r=2r=2\. Movingu2u\_\{2\}keeps the Gram, spectrum and rank lamps lit together, and the presetu=\(e1,e2\)u=\(e\_\{1\},e\_\{2\}\)lands onW0​He1​He2W\_\{0\}H\_\{e\_\{1\}\}H\_\{e\_\{2\}\}, which exact Cayley OFT never reaches\.

#### LOFT fixes one leaf of the meet

The proof of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(§[C\.13](https://arxiv.org/html/2610.12728#A3.SS13)\) already shows how the meet is built\. It is the union of the orbitsW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\)over therr\-dimensional input subspacesVV, one subgroup orbit for each point of the GrassmannianGr⁡\(r,n\)\\Gr\(r,n\)\([Background C\.15](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\. Call each orbit a*leaf*\. Leaves overlap along lower\-rank strata, so the leaves cover the meet without partitioning it; a generic point lies on exactly one leaf, fixed by its moved spaceim⁡\(R−I\)\\operatorname\{im\}\(R\-I\)\. HRA’s reflection vectors move the leaf as it trains\. LOFT\([Zhao et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib582)\)freezes the leaf and makes the choice of leaf the design variable, which the paper calls the adaptation support\.

Remark\(2026: LoCO\)\. One component of LoCO\([Nguyen et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib347)\)is the Cayley rotation of a low\-rank skew generatorU​V⊤−V​U⊤UV^\{\\top\}\-VU^\{\\top\}, withU,V∈ℝn×rU,V\\in\\mathbb\{R\}^\{n\\times r\}\. For injectiveW0W\_\{0\}and2​r≤n−22r\\leq n\-2, its image isIm⁡HRA2​r\\Imop\\mathrm\{HRA\}\_\{2r\}minus the pointsW0​RW\_\{0\}Rat whichRRhas eigenvalue−1\-1, so like HRA it moves its leaf while it trains\. Writing the generator asZ​J​Z⊤ZJZ^\{\\top\}withZ=\[U​V\]Z=\[U\\ V\]shows that its generic fibre is an orbit of the linear groupSp⁡\(2​r,ℝ\)\\mathrm\{Sp\}\(2r,\\mathbb\{R\}\), acting byZ↦Z​MZ\\mapsto ZM\. The non\-linear gauge of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\) is therefore a property of the reflection parametrisation; LoCO’s component reaches an open dense part of the same set with the linear gaugeSp⁡\(2​r,ℝ\)\\mathrm\{Sp\}\(2r,\\mathbb\{R\}\)\. A sum of several LoCO components is in general only approximately orthogonal, and then leaves the meet\.

Example\(LOFT: a leaf, and how to choose it\)\. LetPr∈ℝr×nP\_\{r\}\\in\\mathbb\{R\}^\{r\\times n\}have orthonormal rows spanningVV, and letO⁡\(V\)=\{I\+Pr⊤​\(T−I\)​Pr:T∈O⁡\(r\)\}\\mathrm\{O\}\(V\)=\\\{I\+P\_\{r\}^\{\\top\}\(T\-I\)P\_\{r\}:T\\in\\mathrm\{O\}\(r\)\\\}be the orthogonal maps that fixV⟂V^\{\\perp\}pointwise\. LOFT trainsρ⁡\(E\)=W0​\(I\+Pr⊤​\(Q⁡\(E\)−I\)​Pr\)\\rho\(E\)=W\_\{0\}\\big\(I\+P\_\{r\}^\{\\top\}\(Q\(E\)\-I\)P\_\{r\}\\big\)withE∈𝔰​𝔬​\(r\)E\\in\\mathfrak\{so\}\(r\),E0=0E\_\{0\}=0, so\|M\|=r⁡\(r−1\)/2\\lvert M\\rvert=r\(r\-1\)/2\. We read it with an exactly orthogonalQQ\(the exponential, or the Cayley mapCC\); the released code defaults to a five\-term Cayley–Neumann series withR⊤​R=\(I−X6\)2R^\{\\top\}R=\(I\-X^\{6\}\)^\{2\}, which is only approximately orthogonal, so like HF’s default OFT \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\) that version leaves its leaf\.\(a\)*One leaf\.*IfW0​Pr⊤W\_\{0\}P\_\{r\}^\{\\top\}has rankrr, then\(W0\+W0​Pr⊤​ℝr×r​Pr\)∩W0⋅O⁡\(n\)=W0⋅O⁡\(V\)\\big\(W\_\{0\}\+W\_\{0\}P\_\{r\}^\{\\top\}\\,\\mathbb\{R\}^\{r\\times r\}P\_\{r\}\\big\)\\cap W\_\{0\}\\cdot\\mathrm\{O\}\(n\)=W\_\{0\}\\cdot\\mathrm\{O\}\(V\)\. For the principal supportPr=Vr⊤P\_\{r\}=V\_\{r\}^\{\\top\}the first set is the image of LoRA\-XS \(this needsσr​\(W0\)\>0\\sigma\_\{r\}\(W\_\{0\}\)\>0\), so the leaf is LoRA\-XS∩W0⋅O\(n\)\{\}\\cap W\_\{0\}\\cdot\\mathrm\{O\}\(n\)\. Principal\-support LOFT reaches an open dense part of its componentW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\); for injectiveW0W\_\{0\}and a CayleyQQ, its image is exactly LoRA\-XS∩\{\}\\cap\{\}the image of exact full\-block Cayley OFT\.\(b\)*Leaves over the Grassmannian\.*IfW0W\_\{0\}is injective,\(W0\+ℳ≤r\)∩W0⋅SO⁡\(n\)=⋃V∈Gr⁡\(r,n\)W0⋅SO⁡\(V\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)=\\bigcup\_\{V\\in\\Gr\(r,n\)\}W\_\{0\}\\cdot\\mathrm\{SO\}\(V\), which for evenrrisIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\)\)\. LOFT’s image is the whole leafW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\)with the exponential, and an open dense part of it with a Cayley map, which never reaches eigenvalue−1\-1\.\(c\)*A regular base point\.*IfW0​Pr⊤W\_\{0\}P\_\{r\}^\{\\top\}has rankrr, thenS1=W0​Pr⊤​𝔰​𝔬​\(r\)​PrS\_\{1\}=W\_\{0\}P\_\{r\}^\{\\top\}\\mathfrak\{so\}\(r\)P\_\{r\}has dimensionr⁡\(r−1\)/2=d⁡\(M\)r\(r\-1\)/2=d\(M\): LOFT starts at a smooth point of its leaf and has no gauge, while HRA’s paired initialisation is an apex \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\)\)\. Forr≥4r\\geq 4there is no smooth pointed arrow LOFT→\{\}\\to\{\}HRA at that initialisation, although the images are nested \(obstruction test O3 of §[VIII](https://arxiv.org/html/2610.12728#S8)\)\.\(d\)*Choosing the leaf\.*IfQ′​\(0\)=idQ^\{\\prime\}\(0\)=\\mathrm\{id\}\(the exponential, orE↦C⁡\(E/2\)E\\mapsto C\(E/2\)\), the gradient pulled back toEEatθ0\\theta\_\{0\}isPr​skew⁡\(W0⊤​G\)​Pr⊤P\_\{r\}\\skewop\(W\_\{0\}^\{\\top\}G\)P\_\{r\}^\{\\top\}withG=∇L​\(W0\)G=\\nabla L\(W\_\{0\}\); the Cayley mapCCitself doubles it, which does not change the best support\. Its squared norm is at most2​∑k≤r/2μk22\\sum\_\{k\\leq r/2\}\\mu\_\{k\}^\{2\}, where±i​μk\\pm i\\mu\_\{k\}are the eigenvalues ofskew⁡\(W0⊤​G\)\\skewop\(W\_\{0\}^\{\\top\}G\), with equality whenVVspans the invariant subspace of the top⌊r/2⌋\\lfloor r/2\\rfloorpairs\.

Appendix[C\.14](https://arxiv.org/html/2610.12728#A3.SS14)proves \(a\)–\(d\)\.

##### Numerical check

Forn=8n=8,m=10m=10andr=2,4,6r=2,4,6, the leaf identity, the union, the arrows into exact Cayley OFT and LoRA\-XS, anddimS1=r⁡\(r−1\)/2\\dim S\_\{1\}=r\(r\-1\)/2all hold to about10−1510^\{\-15\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

LOFT, the SkewGrad and GradSVD supports, part \(d\) and its bound \(their Proposition 2\), the reading of each support as selecting a Lie subgroup \(their Remark 1\) and the suggestion of dynamic supports \(their Remark 3\) are from[Zhao et al\. \(2026\)](https://arxiv.org/html/2610.12728#bib.bib582)\. Their Table 1 writes full and block OFT, GOFT, BOFT, HRA and PSOFT as choices of support, width, depth and transform in a multi\-factor version of the same primitive\. Parts \(a\)–\(c\) place the single\-factor method inside[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\. PSOFT is due to[Wu et al\. \(2025a\)](https://arxiv.org/html/2610.12728#bib.bib498); LoRA\-XS to[Bałazy et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib26)\.

WHAT THIS EXPLAINSOrthogonal methods differ in which leaves they use and how they combine them\. Block OFT, GOFT and BOFT multiply rotations of fixed coordinate subspaces; HRA moves its leaf while it trains; PSOFT takes the principal leaf and adds two tori; LOFT takes one leaf, chosen before training fromW0W\_\{0\}or from the gradient atθ0\\theta\_\{0\}\. By part \(d\), SkewGrad is the orthogonal counterpart of the alignment mechanism of[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3), and GradSVD transplants the gradient\-based frames of LoRA\-GA\([Wang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib472)\)\. With the leaf frozen, merge\-and\-restart gains nothing\. With finitely many supports revisited across restarts, the reach is the orbit of the connected group whose Lie algebra \([Background D\.8](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) the𝔰​𝔬​\(Vk\)\\mathfrak\{so\}\(V\_\{k\}\)generate \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\)\); numerically, generic supports withr≥2r\\geq 2generate all ofSO⁡\(n\)\\mathrm\{SO\}\(n\)once⌈n/r⌉\\lceil n/r\\rceilof them are used\. The principal support does not move when it is re\-selected after a merge, and data\-chosen supports have to be checked case by case\.

#### DoRA is a torus saturation of LoRA

DoRA normalises each row of LoRA’s matrixV=W0\+B​AV=W\_\{0\}\+BAand multiplies rowiiby a learned magnitudeμi\\mu\_\{i\}\. That is one row scaling, byμi/∥vi∥\\mu\_\{i\}/\\lVert v\_\{i\}\\rVert, so every DoRA weight is a diagonal matrix times a LoRA weight, and conversely up to a repair for vanishing rows\. We writeDiagm\\mathrm\{Diag\}\_\{m\}for the monoid of diagonal matrices \([Background D\.2](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\),Monm=Tm⋊Sm\\mathrm\{Mon\}\_\{m\}=T\_\{m\}\\rtimes S\_\{m\}for the monomial matrices \([Background D\.3](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\), andNNfor row normalisation\.

Proposition II\.8\(DoRA is LoRA followed by an output scale, at the level of images\)\.Assume1≤r≤min⁡\(m,n\)1\\leq r\\leq\\min\(m,n\)ands≠0s\\neq 0\. LetNNnormalise rows; DoRA’s magnitudeμ∈ℝm\\mu\\in\\mathbb\{R\}^\{m\}is per output neuron, as in HF PEFT and the official code \(the paper’s notation says “column\-wise”, in the transposed convention\)\. Images refer to the merged \(eval\-mode\) weight; HF’s train\-mode forward with lora\_dropout\>0\{\}\>0is not of the formW​xWx\. LetW0W\_\{0\}have no zero row, let𝒰\\mathcal\{U\}be the open set of matrices with no zero row, and letDiagm\\mathrm\{Diag\}\_\{m\}be the monoid of all diagonal matrices\. ThenImDoRAr=\{diag\(μ\)N\(V\):V∈\(W0\+ℳ≤r\)∩𝒰,μ∈ℝm\}=Diagm⋅\(W0\+ℳ≤r\),\\Imop\\mathrm\{DoRA\}\_\{r\}=\\\{\\diag\(\\mu\)\\,N\(V\):V\\in\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\mathcal\{U\},\\ \\mu\\in\\mathbb\{R\}^\{m\}\\\}=\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\),which is exactly the image of “LoRAr\\mathrm\{LoRA\}\_\{r\}followed by a trainable output scaleℓ∈ℝm\\ell\\in\\mathbb\{R\}^\{m\}”\. Withμ\\murestricted to\(ℝ×\)m\(\\mathbb\{R\}^\{\\times\}\)^\{m\}, or equivalently after intersecting with𝒰\\mathcal\{U\}:Im⁡DoRAr∩𝒰=Tm⋅\(W0\+ℳ≤r\)∩𝒰\\Imop\\mathrm\{DoRA\}\_\{r\}\\cap\\mathcal\{U\}=T\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\mathcal\{U\}\. Moreover:•d⁡\(DoRAr\)≤min⁡\(m​n,r⁡\(m\+n−r\)\+m\)d\(\\mathrm\{DoRA\}\_\{r\}\)\\leq\\min\\big\(mn,\\ r\(m\+n\-r\)\+m\\big\)\. At a parameter point withμ∈\(ℝ×\)m\\mu\\in\(\\mathbb\{R\}^\{\\times\}\)^\{m\}andX=s​B​A∈ℳrX=sBA\\in\\mathcal\{M\}\_\{r\}, withU=col⁡XU=\\operatorname\{col\}XandR=row⁡XR=\\operatorname\{row\}X, the Jacobian rank isr⁡\(m\+n−r\)\+m−dimker⁡\(δ↦PU⟂​diag⁡\(δ\)​W0​PR⟂\)r\(m\+n\-r\)\+m\-\\dim\\ker\\big\(\\delta\\mapsto P\_\{U^\{\\perp\}\}\\diag\(\\delta\)W\_\{0\}P\_\{R^\{\\perp\}\}\\big\); where someμi=0\\mu\_\{i\}=0it is smaller whenr≥2r\\geq 2, and forr=1r=1when some roww0,iw\_\{0,i\}withμi=0\\mu\_\{i\}=0lies outsideRR\. For genericW0W\_\{0\},d=min⁡\(m​n,r⁡\(m\+n−r\)\+m\)d=\\min\\big\(mn,\\ r\(m\+n\-r\)\+m\\big\); in particulard=r⁡\(m\+n−r\)\+md=r\(m\+n\-r\)\+miff\(m−r\)​\(n−r\)≥m\(m\-r\)\(n\-r\)\\geq m, which holds for all practical shapes\. For specialW0W\_\{0\}it is smaller: forW0=a​c⊤W\_\{0\}=ac^\{\\top\}with allai≠0a\_\{i\}\\neq 0andr<nr<n,d=r⁡\(m\+n−r\)\+m−rd=r\(m\+n\-r\)\+m\-r, strictly below the generic value iffr<mr<mandr≤n−2r\\leq n\-2\.•The image covariance group \([Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\) containsMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\.•AtB=0B=0, in HF’s magnitude coordinates,Δ​W=\(diag⁡\(μi/∥w0,i∥\)−I\)​W0\\Delta W=\\big\(\\diag\(\\mu\_\{i\}/\\lVert w\_\{0,i\}\\rVert\)\-I\\big\)W\_\{0\}, which can have rank up torank⁡W0\\rk W\_\{0\}, so DoRA escapes LoRA’s rank bound\.

##### Numerical check

The Jacobian rank is9=m​n9=mnat\(m,n,r\)=\(3,3,2\)\(m,n,r\)=\(3,3,2\), wherer⁡\(m\+n−r\)\+mr\(m\+n\-r\)\+mgives1111; it is3636at\(8,8,2\)\(8,8,2\), matching, and3434there when oneμi=0\\mu\_\{i\}=0; it is2424for a rank\-oneW0W\_\{0\}at\(6,6,2\)\(6,6,2\)\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

DoRA is due to[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib282)\. Its magnitude–direction split follows weight normalisation\([Salimans and Kingma, 2016](https://arxiv.org/html/2610.12728#bib.bib397)\)\. From a different direction,[Zhang et al\. \(2025g\)](https://arxiv.org/html/2610.12728#bib.bib576)argue that the magnitude of the update is the main driver of LoRA’s performance\.

WHAT THIS EXPLAINSAt the level of images, DoRA’s “magnitude–direction decoupling” is LoRA plusmmfree output gains\. What DoRA changes beyond that lives in its training dynamics\. Under gradient flow or SGD there are two different geometries\. The exact gradient of the normalised parametrisation projects the direction gradient row by row, which is a change of metric\. The detached backward of the DoRA paper \(Sec\. 4\.3\) and of HF PEFT \(weight\_norm\.detach\(\)\) removes that projection; the resulting update is a non\-gradient pseudo\-gradient that no metric describes, and it need not be a descent direction\. Both the paper and HF train with AdamW, under which neither variant follows a fixed metric, and the coordinates matter in their own right:μi\\mu\_\{i\}starts at∥w0,i∥\\lVert w\_\{0,i\}\\rVertandℓi\\ell\_\{i\}at11, so an equal step onμi\\mu\_\{i\}and on a gainℓi\\ell\_\{i\}changes the row gain by different amounts\.

##### Experiment D

With one optimizer \(AdamW, same learning rate, weight decay and data order\) and the sameW0W\_\{0\}, compare four arms: \(i\) LoRA \+ gainℓ\\ellwithℓ0=𝟏\\ell\_\{0\}=\\mathbf\{1\}; \(ii\) LoRA \+ gain in DoRA’s coordinates,W=diag⁡\(μi/∥w0,i∥\)​\(W0\+s​B​A\)W=\\diag\(\\mu\_\{i\}/\\lVert w\_\{0,i\}\\rVert\)\(W\_\{0\}\+sBA\)withμ0=\(∥w0,i∥\)i\\mu\_\{0\}=\(\\lVert w\_\{0,i\}\\rVert\)\_\{i\}, which isolates the optimizer\-coordinate effect without normalisation; \(iii\) DoRA with the exact backward; \(iv\) DoRA with HF’s detached backward\. Since the images are equal, any gap between arms comes from the parametrisation, the backward rule or the optimizer coordinates, and the four arms separate these\. The DoRA paper’s own ablation reports \(iii\) and \(iv\) within0\.20\.2points\.[Prediction X\.2](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3)uses the equal images to set up this controlled comparison\.

The covariance groupMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}decides where[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)applies\. One orthogonal rotation of the residual stream of a gain\-fused pre\-norm model, as in SliceGPT\([Ashkboos et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib18)\), acts on the input side ofqq,kk,vv, gate and up, where DoRA isGLn\\mathrm\{GL\}\_\{n\}\-covariant, so DoRA on those boxes reaches the same functions after the rotation as before; onooand down the sufficient condition fails\. Training outcomes are a separate question\.

#### The torus ladder

Write a weight asW0⊙\(J\+Z\)W\_\{0\}\\odot\(J\+Z\), withJJthe all\-ones matrix\. Output scalings useZ=\(ℓ−𝟏\)​𝟏⊤Z=\(\\ell\-\\mathbf\{1\}\)\\mathbf\{1\}^\{\\top\}, of rank one; two\-sided scalings usea​b⊤−Jab^\{\\top\}\-J, of rank at most two; HiRA allows anyZ=B​AZ=BAof rank at mostrr\. These choices form a ladder of images, and no rung can fill a zero ofW0W\_\{0\}\.

Proposition II\.9\(the torus ladder\)\.\(a\)HiRA\.ρ⁡\(B,A\)=W0⊙\(J\+B​A\)\\rho\(B,A\)=W\_\{0\}\\odot\(J\+BA\), whereJJis the all\-ones matrix; HF PEFT hasB0=0B\_\{0\}=0and no scale factor on Linear and Conv layers \(on embeddings it usesA0=0A\_\{0\}=0\)\. For everyW0W\_\{0\},Im⁡HiRAr=W0\+DW0​\(ℳ≤r\)\\Imop\\mathrm\{HiRA\}\_\{r\}=W\_\{0\}\+D\_\{W\_\{0\}\}\(\\mathcal\{M\}\_\{\\leq r\}\)withDW0:X↦W0⊙XD\_\{W\_\{0\}\}:X\\mapsto W\_\{0\}\\odot X\. IfW0W\_\{0\}has no zero entries,DW0∈GL⁡\(Θ\)D\_\{W\_\{0\}\}\\in\\mathrm\{GL\}\(\\Theta\), so inside the slice atθ0\\theta\_\{0\}HiRA is LoRA transported by a diagonal automorphism: it has the samed=r⁡\(m\+n−r\)d=r\(m\+n\-r\)\(forr≤min⁡\(m,n\)r\\leq\\min\(m,n\)\) and the same gauge\. In all casesrank⁡Δ​W≤r⋅rank⁡W0\\rk\\Delta W\\leq r\\cdot\\rk W\_\{0\}\. The transport depends onθ0\\theta\_\{0\}\([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.\(b\)Scalings\.Output\-side scalings,diag⁡\(ℓ\)​W0=W0⊙\(J\+\(ℓ−𝟏\)​𝟏⊤\)\\diag\(\\ell\)W\_\{0\}=W\_\{0\}\\odot\(J\+\(\\ell\-\\mathbf\{1\}\)\\mathbf\{1\}^\{\\top\}\), and input\-side scalings,W0​diag⁡\(ℓ\)=W0⊙\(J\+𝟏​\(ℓ−𝟏\)⊤\)W\_\{0\}\\diag\(\\ell\)=W\_\{0\}\\odot\(J\+\\mathbf\{1\}\(\\ell\-\\mathbf\{1\}\)^\{\\top\}\), have images inIm⁡HiRA1\\Imop\\mathrm\{HiRA\}\_\{1\}\. Output\-side examples: \(IA\)3on keys and values, the post\-linear scale of SSF, DoRA’s magnitude\. Input\-side examples: \(IA\)3’s feed\-forward vector, which scales the input ofWdownW\_\{\\mathrm\{down\}\}, and a LayerNorm gainγ\\gammafolded into the next linear map\. The fold is exact when the norm’s output is consumed only by linear maps: the input norms of pre\-norm blocks, and a final norm before an untied head \(in post\-norm BERT, the last block’s norm feeds only the head\)\. It does not apply to Gemma\-style post\-sublayer norms, which HF’s default LN\-tuning targets for Gemma 2 and 3, to QK\-norms, or to post\-norm residual paths\. Where it applies,γ\\gammais tied across all consumers of the norm \(Q, K and V, or gate and up\) rather than being a free point per box, and with tied embeddings a fold into the head breaks the tie\. LayerNorm biases and SSF’s shift fold into the next bias, not intoWW\. Two\-sided scalingsdiag⁡\(a\)​W0​diag⁡\(b\)=W0⊙\(a​b⊤\)\\diag\(a\)W\_\{0\}\\diag\(b\)=W\_\{0\}\\odot\(ab^\{\\top\}\)lie inIm⁡HiRA2\\Imop\\mathrm\{HiRA\}\_\{2\}\. As a morphism,h⁡\(ℓ\)=\(ℓ−𝟏,𝟏⊤\)h\(\\ell\)=\(\\ell\-\\mathbf\{1\},\\mathbf\{1\}^\{\\top\}\)lands inHiRA1\\mathrm\{HiRA\}\_\{1\}pointed at\(0,𝟏⊤\)\(0,\\mathbf\{1\}^\{\\top\}\), which is not HiRA’s random\-AAdefault pointing\.\(c\)Orbit dimension\.The two\-sided torus orbit\{diag⁡\(a\)​W0​diag⁡\(b\)\}\\\{\\diag\(a\)W\_\{0\}\\diag\(b\)\\\}has dimensionm\+n−c⁡\(W0\)m\+n\-c\(W\_\{0\}\), wherec⁡\(W0\)c\(W\_\{0\}\)is the number of connected components of the bipartite support graph ofW0W\_\{0\}, an isolated vertex counting as its own component\.\(d\)RoAd\.RoAd1\(the paper’s main variant, HF road\_1\) isW=R​W0W=RW\_\{0\}withRRblock\-diagonal in scaled rotationsαi​Rot​\(θi\)\\alpha\_\{i\}\\mathrm\{Rot\}\(\\theta\_\{i\}\)of output pairs\. It is the left action ofCO\+​\(2\)m/2≅\(ℂ×\)m/2\\mathrm\{CO\}^\{\+\}\(2\)^\{m/2\}\\cong\(\\mathbb\{C\}^\{\\times\}\)^\{m/2\}: it is\(IA\)3overℂ\\mathbb\{C\}\. It multiplies the input Gram matrix of each output pair byαi2\\alpha\_\{i\}^\{2\}\. Sinceαi\\alpha\_\{i\}may be00, its image is the orbit closure, a real\-linear space of dimension exactlymmwhenever no output\-pair block ofW0W\_\{0\}is zero\. RoAd2and RoAd4\(HF road\_2, road\_4\) let each block range over all ofM2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\)\. They realise the block\-diagonal action of the monoidM2​\(ℝ\)m/2M\_\{2\}\(\\mathbb\{R\}\)^\{m/2\}\(its group partGL2m/2\\mathrm\{GL\}\_\{2\}^\{m/2\}is non\-abelian\), with linear image\{blockdiag⁡\(Ri\)​W0\}\\\{\\mathrm\{blockdiag\}\(R\_\{i\}\)W\_\{0\}\\\}of dimension2​m2mwhen every pair block has rank22\. For them the Gram statement and “overℂ\\mathbb\{C\}” fail, and RoAd4carries a positive\-dimensional gauge \(two dimensions per row\)\. HF pairs outputjjwithj\+g/2j\+g/2inside each group of sizegg\(default6464\) rather than adjacent outputs as in the paper, and also rotates the bias; the statements above do not depend on the pairing\.\(e\)Zeros and covariance\.EveryWWreachable by HiRA, \(IA\)3or SSF vanishes whereverW0W\_\{0\}does,Z⁡\(W0\)⊆Z⁡\(W\)Z\(W\_\{0\}\)\\subseteq Z\(W\), and this persists under merging\. New zeros can appear \(ℓi=0\\ell\_\{i\}=0, or\(J\+B​A\)i​j=0\(J\+BA\)\_\{ij\}=0\), so the zero pattern itself is not invariant\. The image covariance group of output\-scaled boxes \(\(IA\)3onk,vk,v; post\-linear SSF\) isMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, and that of input\-scaled boxes \(\(IA\)3onWdownW\_\{\\mathrm\{down\}\}\) isGLm×Monn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}\. HiRA’s is exactlyMonm×Monn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}\(=GL1×GL1=\\mathrm\{GL\}\_\{1\}\\times\\mathrm\{GL\}\_\{1\}whenm=n=1m=n=1\)\.

##### Numerical check

The RoAd1orbit has dimension88and the RoAd2image dimension1616atm=8m=8\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

\(IA\)3is due to[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275), SSF to[Lian et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib259), HiRA to[Huang et al\. \(2025b\)](https://arxiv.org/html/2610.12728#bib.bib182), RoAd to[Liao and Monz \(2024\)](https://arxiv.org/html/2610.12728#bib.bib263), and LayerNorm tuning to[Zhao et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib578)\. Feature\-wise scaling as a building block goes back at least to FiLM\([Perez et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib363)\)\.

WHAT THIS EXPLAINSEach scaling method here is a rungW0⊙\(J\+Z\)W\_\{0\}\\odot\(J\+Z\): \(IA\)3, SSF’s scales and the gain part of LN\-tuning on norms that feed only linear maps lie inHiRA1\\mathrm\{HiRA\}\_\{1\}, on one side or the other, and HiRA is LoRA in the Hadamard frame ofW0W\_\{0\}\. DoRA’s image is theDiagm\\mathrm\{Diag\}\_\{m\}\-orbit of LoRA’s image \(image level only;[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\)\. DoRA and RoAd can fill zeros ofW0W\_\{0\}, so they become rungs only whenW0W\_\{0\}is zero\-free\. Then every image germ can be written asW0⊙\(J\+Z\)W\_\{0\}\\odot\(J\+Z\), so the ladder by itself constrains nothing there and serves as bookkeeping\. RoAd1is complex \(IA\)3and RoAd2,4are block\-M2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\)\(IA\)3; for genericW0W\_\{0\}neither lies on a low\-rank rung\. HiRA’s “high rank” is a property of a frame: its transportDW0D\_\{W\_\{0\}\}depends onθ0\\theta\_\{0\}and does not preserve rank\. Two re\-based cycles reachW0⊙\(J\+Z1\)⊙\(J\+Z2\)W\_\{0\}\\odot\(J\+Z\_\{1\}\)\\odot\(J\+Z\_\{2\}\), whose relative updateZ1\+Z2\+Z1⊙Z2Z\_\{1\}\+Z\_\{2\}\+Z\_\{1\}\\odot Z\_\{2\}has rank up tomin⁡\(2​r\+r2,m,n\)\\min\(2r\+r^\{2\},m,n\), against2​r2rfor LoRA \([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.

With HF defaults, \(IA\)3scales the output side of the attention projections it targets \(kkandvv, orqqandvv\) and the input side of down, so the residual rotation above, acting on the other side, leaves its reachable functions unchanged\. HiRA’s covariance group stops atMonm×Monn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}and fails the sufficient condition of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\. That alone does not prove that its reachable functions change; Conjecture A \(stated after[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\) says they do, and[Prediction X\.4](https://arxiv.org/html/2610.12728#S10.SS0.SSSx4.Px1)is the test\.

Corollary II\.6, Theorem II\.7 and Propositions II\.8 and II\.9 were repaired after two rounds of adversarial review; the referee record in §[XI](https://arxiv.org/html/2610.12728#S11)lists what changed\.

The groups of this exposé see orbits\. LoRA’s cone is governed instead by the rank of the update, read off a cut in a tensor network, while the size of an image is a dimension, which behaves like a volume\. Rank and volume can move in opposite directions, the subject of §[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\.

### II\.C Cut and volume

Rank lives on cuts and dimension lives in volumes\. Count the wrong one and you will compare the wrong things\. a working maxim

At a fixed budget, the narrowest cut of a method’s diagram bounds the rank it can reach\. The dimension of what it can reach is a separate count, and the two can move in opposite directions\.

##### In brief

- •A method’s update rank is bounded by the narrowest cut of its tensor\-network diagram\. The dimensiond⁡\(M\)d\(M\)of what it can reach is its parameter count minus the directions that change nothing\.
- •The two can move in opposite directions\. At the budget of LoRA8on a4096×40964096\\times 4096layer, LoHa4,4reaches rank1616and at least8,1598\{,\}159fewer dimensions; GraLoRA withk=2k=2reaches rank1616at LoRA’s dimension, with every block capped at rank44\.
- •Kronecker sums are LoRA across another cut of the same four\-leg tensor, and HiRA is LoRA afterX↦W0⊙XX\\mapsto W\_\{0\}\\odot X\(forW0W\_\{0\}without zero entries\)\. Such changes of frame keep dimension and move rank\.
- •LoRA respects every change of a layer’s coordinates and so reaches the same set of functions after a function\-preserving rotation of the network; PiSSA, MiLoRA and CorDA splits respect only rotations and uniform scalings\.

PEFT papers usually report a method’s expressiveness as a rank and its cost as a parameter count\. A third number, the dimension of the set of weights the method can reach, usually goes unreported\.

Draw a method as a network of boxes and wires\. The rank of its update is limited by the narrowest place where the network can be cut in two, so rank is a*cut*invariant\. The dimension counts the independent directions in which the method can move the weights; we call it a*volume*invariant, loosely, since no measure is computed\. In Figure[4](https://arxiv.org/html/2610.12728#S2.F4), several methods raise one while lowering the other\.

Figure 4:Rank is not volume\([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. One4096×40964096\\times 4096layer at the budget of LoRA8\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\),\|M\|=8​\(m\+n\)=65,536\\lvert M\\rvert=8\(m\+n\)=65\{,\}536\. Each method is placed by the largest rank ofΔ​W\\Delta Wit can reach \(horizontal, logarithmic scale\), which is bounded by the narrowest cut of its diagram \([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\), and by its expressive dimension as a fraction of the layer,d⁡\(M\)/\(m​n\)d\(M\)/\(mn\)\(vertical, in units of10−310^\{\-3\}, with the axis broken between3\.413\.41and3\.903\.90\), from the gauge census of[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\. The dashed line isd=\|M\|d=\\lvert M\\rvert, a method without gauge, and the dotted line isd=\|M\|−82=65,472d=\\lvert M\\rvert\-8^\{2\}=65\{,\}472, the dimension of LoRA8with itsGL8\\mathrm\{GL\}\_\{8\}gauge\. Along the dotted line the maximum rank runs from88for LoRA8, through GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\)withk=2k=2,44and88blocks \(the last two light\), to full rank for HiRA8\([Huang et al\., 2025b](https://arxiv.org/html/2610.12728#bib.bib182)\)\. MoRA\([Jiang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib196)\)and C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\)have no gauge and reach full rank; their points coincide and are drawn on either side of the full\-rank line\. FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\)loses one dimension for each sampled conjugate pair, and its maximum rank is known only through a bound \(open marker\)\. KronA\([Edalati et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib103)\)is shown at the factor shapes closest to the budget\. LoHa4,4\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\), below the break, sits at the proved upper bound of its dimension: it reaches rank1616with at least8,1598\{,\}159fewer dimensions than LoRA8\([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(c\)\)\. The parameters behind each point are in §[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\.
#### Rank is a cut

In LoRA every signal crosses the inner wire of dimensionrrbetweenAAandBB, soΔ​W=B​A\\Delta W=BAhas rank at mostrr\(Figure[5](https://arxiv.org/html/2610.12728#S2.F5)\)\. The argument works for any method drawn as a tensor network \([Background B\.7](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2)\), a string diagram \([Background B\.3](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\) in the symmetric monoidal category\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗\)\(\\mathbf\{FinVect\},\\otimes\)\. Split the boxes into an input side and an output side, and the network factors through the wires that cross\.

V=ℝnV=\\mathbb\{R\}^\{n\}U=ℝmU=\\mathbb\{R\}^\{m\}W0W\_\{0\}W0W\_\{0\}nnmm\+\+ssAABBnnrrmmnnmmrrW0\+s​B​AW\_\{0\}\+sBAFigure 5:LoRA as a plug\.\(a\) CuttingW0W\_\{0\}out of a linear layery=W0​xy=W\_\{0\}xleaves a holeV→UV\\to U, withV=ℝnV=\\mathbb\{R\}^\{n\}andU=ℝmU=\\mathbb\{R\}^\{m\}\. \(b\) LoRA fills the hole with the frozen boxW0W\_\{0\}\(grey\) plus the trainable chains​B​AsBA\(teal\), a formal sum, over which rank is subadditive\. The dashed cut putsAAon the input side andBBon the output side and crosses one wire of dimensionrr, sorank⁡\(s​B​A\)≤r\\rk\(sBA\)\\leq r\([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\)\. The dotted cuts put both boxes on one side and cross an open leg, with weightsnnandmm; the minimum over cuts ismin⁡\(n,r,m\)\\min\(n,r,m\)\.Theorem II\.10\(cut\-rank bound; standard\)\.LetΓ\\Gammabe a tensor network, evaluated in the symmetric monoidal category\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗\)\(\\mathbf\{FinVect\},\\otimes\), with input legsVVand output legsUU\. Every wire and open leg is assumed incident to a vertex: insert an identity box on any bare strand, or equivalently count each bare input–output wire as crossing every cut\. A crossing hyperedge \(copy spider\) contributes its dimension once\. Partition the vertices intoℐ⊔𝒪\\mathcal\{I\}\\sqcup\\mathcal\{O\}\. The weightwwof the cut is the product of the dimensions of the internal wires between the two sides and of the open legs attached on the wrong side \(input legs at𝒪\\mathcal\{O\}, output legs atℐ\\mathcal\{I\}\)\. Thenrank⁡⟦Γ⟧≤mincuts⁡w,\\rk\\llbracket\\Gamma\\rrbracket\\leq\\min\_\{\\mathrm\{cuts\}\}w,and rank is subadditive over formal sums\. The bound refers to a chosen presentation of the network\.

##### Numerical check

No violation in283283random networks \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

String diagrams go back to[Joyal and Street \(1991\)](https://arxiv.org/html/2610.12728#bib.bib202), and the copy spider \([Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\) to the Frobenius algebra of a basis\([Coecke et al\., 2013](https://arxiv.org/html/2610.12728#bib.bib81)\)\. Cut bounds on rank are standard tensor\-network reasoning \([Background B\.8](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2);[Kolda and Bader, 2009](https://arxiv.org/html/2610.12728#bib.bib222);[Cichocki et al\., 2016](https://arxiv.org/html/2610.12728#bib.bib78)\)\. The PEFT reading is ours\.

Without the convention on bare strands,Id⊗B​AI\_\{d\}\\otimes BAwould violate the bound\. The bound also depends on the drawing\. A naive expansion of HRA’s product of reflections has2r−12^\{r\}\-1terms, while the telescoping presentation

R−I=∑k=1r\(Hk−I\)Hk\+1⋯HrR\-I=\\sum\_\{k=1\}^\{r\}\(H\_\{k\}\-I\)\\,H\_\{k\+1\}\\cdots H\_\{r\}writesR−IR\-Iasrrterms of rank at most one and gives the sharp boundrr\.

##### Consequences

For one box,[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)gives the bounds in Table[8](https://arxiv.org/html/2610.12728#S2.T8)\. Relative toW0W\_\{0\}, splitting initialisations giveΔ​W=B​A−B0​A0\\Delta W=BA\-B\_\{0\}A\_\{0\}, of rank at most2​r2r\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\. The Givens bound uses the same telescoping presentation as HRA, each gate contributing a term of rank at most two, and HiRA’s bound factorsW0W\_\{0\}through a bond of dimensionrank⁡W0\\rk W\_\{0\}\.

Table 8:Cut bounds of[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)onrank⁡Δ​W\\rk\\Delta Wfor onem×nm\\times nbox, with the presentation each bound uses\.WHAT THIS EXPLAINSA large min\-cut is necessary for high rank and does not guarantee it\. The “high\-rank PEFT” literature consists of networks whose min\-cut avoids anrr\-dimensional bottleneck\. Whether the bound is attained is a separate question, and attaining it can cost volume\.

#### Kronecker adapters cut the other way

Ifm=m1​m2m=m\_\{1\}m\_\{2\}andn=n1​n2n=n\_\{1\}n\_\{2\}, anm×nm\\times nmatrix is a tensor with output legsU1,U2U\_\{1\},U\_\{2\}and input legsV1,V2V\_\{1\},V\_\{2\}\([Background B\.10](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\)\. LoRA’s rank is the rank across\{U1​U2\}\|\{V1​V2\}\\\{U\_\{1\}U\_\{2\}\\\}\\mid\\\{V\_\{1\}V\_\{2\}\\\}; a sum ofkkKronecker products has rank at mostkkacross\{U1​V1\}\|\{U2​V2\}\\\{U\_\{1\}V\_\{1\}\\\}\\mid\\\{U\_\{2\}V\_\{2\}\\\}\. The two cuts disagree sharply, sinceI⊗II\\otimes Iis one Kronecker term of full rank\.

Theorem II\.11\(Kronecker incomparability; the linear algebra is standard, the PEFT reading is new\)\.Writeℝm=ℝm1⊗ℝm2\\mathbb\{R\}^\{m\}=\\mathbb\{R\}^\{m\_\{1\}\}\\otimes\\mathbb\{R\}^\{m\_\{2\}\}andℝn=ℝn1⊗ℝn2\\mathbb\{R\}^\{n\}=\\mathbb\{R\}^\{n\_\{1\}\}\\otimes\\mathbb\{R\}^\{n\_\{2\}\}, and viewΔ∈ℝm×n\\Delta\\in\\mathbb\{R\}^\{m\\times n\}as a 4\-leg tensor\. Letrk\\operatorname\{rk\}be the rank across the cut\{U1​U2\}\|\{V1​V2\}\\\{U\_\{1\}U\_\{2\}\\\}\\mid\\\{V\_\{1\}V\_\{2\}\\\}andrk⊗\\operatorname\{rk\}\_\{\\otimes\}the rank across\{U1​V1\}\|\{U2​V2\}\\\{U\_\{1\}V\_\{1\}\\\}\\mid\\\{U\_\{2\}V\_\{2\}\\\}\. Putκ1:=min⁡\(m1,m2\)⋅min⁡\(n1,n2\),ν:=min⁡\(m1,n1\)⋅min⁡\(m2,n2\)\.\\kappa\_\{1\}:=\\min\(m\_\{1\},m\_\{2\}\)\\cdot\\min\(n\_\{1\},n\_\{2\}\),\\qquad\\nu:=\\min\(m\_\{1\},n\_\{1\}\)\\cdot\\min\(m\_\{2\},n\_\{2\}\)\.\(a\)rk⊗⁡Δ≤k\\operatorname\{rk\}\_\{\\otimes\}\\Delta\\leq kiffΔ=∑i≤kCi⊗Di\\Delta=\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}\. EveryΔ\\Deltahasrk⊗⁡Δ≤min⁡\(m1​n1,m2​n2\)\\operatorname\{rk\}\_\{\\otimes\}\\Delta\\leq\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\)\.\(b\)rk⁡\(C⊗D\)=rk⁡C⋅rk⁡D\\operatorname\{rk\}\(C\\otimes D\)=\\operatorname\{rk\}C\\cdot\\operatorname\{rk\}D, whilerk⊗⁡\(C⊗D\)=1\\operatorname\{rk\}\_\{\\otimes\}\(C\\otimes D\)=1forC,D≠0C,D\\neq 0\. Soν\\nuis the largest rank of a single Kronecker product\.\(c\) Ifu∈ℝmu\\in\\mathbb\{R\}^\{m\}andv∈ℝnv\\in\\mathbb\{R\}^\{n\}have Schmidt ranksρu,ρv\\rho\_\{u\},\\rho\_\{v\}\([Background B\.12](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)\), thenrk⊗⁡\(u​v⊤\)=ρu​ρv\\operatorname\{rk\}\_\{\\otimes\}\(uv^\{\\top\}\)=\\rho\_\{u\}\\rho\_\{v\}\. A generic rank\-one update therefore has Kronecker rankκ1\\kappa\_\{1\}\. For aligned splits,\(m1−m2\)​\(n1−n2\)≥0\(m\_\{1\}\-m\_\{2\}\)\(n\_\{1\}\-n\_\{2\}\)\\geq 0, which include equal square splits and every split HF PEFT’s factorisation produces \(it puts the smaller factor first on both sides\),κ1=min⁡\(m1​n1,m2​n2\)\\kappa\_\{1\}=\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\)is the maximal Kronecker rank\.\(d\)*Incomparability\.*For1≤r<ν1\\leq r<\\nuand1≤k<κ11\\leq k<\\kappa\_\{1\}, the setsℳ≤r\\mathcal\{M\}\_\{\\leq r\}and𝒦≤k:=\{Δ:rk⊗⁡Δ≤k\}\\mathcal\{K\}\_\{\\leq k\}:=\\\{\\Delta:\\operatorname\{rk\}\_\{\\otimes\}\\Delta\\leq k\\\}of unconstrainedkk\-term Kronecker sums are incomparable, for every split\. A generic rank\-one update lies in the first and not the second, andC⊗DC\\otimes Dwith full\-rank factors, of Kronecker rank11and rankν\\nu, lies in the second and not the first\. For equal square splits \(m=n=Nm=n=N, all factorsss\),κ1=ν=s2=N\\kappa\_\{1\}=\\nu=s^\{2\}=N, andINI\_\{N\}is such a product\.\(e\)*Constrained Kronecker methods\.*LetIm⁡M⊆W0\+𝒦≤k\\Imop M\\subseteq W\_\{0\}\+\\mathcal\{K\}\_\{\\leq k\}\. ThenIm⁡LoRAr⊈Im⁡M\\Imop\\mathrm\{LoRA\}\_\{r\}\\not\\subseteq\\Imop Mwheneverk<κ1k<\\kappa\_\{1\}, for every pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}\. GiveLoRAr\\mathrm\{LoRA\}\_\{r\}its zero\-init pointing, with imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. ThenIm⁡M⊈Im⁡LoRAr\\Imop M\\not\\subseteq\\Imop\\mathrm\{LoRA\}\_\{r\}iff someθ∈Im⁡M\\theta\\in\\Imop Mhasrank⁡\(θ−W0\)\>r\\rk\(\\theta\-W\_\{0\}\)\>r\. For LoKr,Δ​W=C⊗B​A\\Delta W=C\\otimes BAwithC∈ℝm1×n1C\\in\\mathbb\{R\}^\{m\_\{1\}\\times n\_\{1\}\},B∈ℝm2×r′B\\in\\mathbb\{R\}^\{m\_\{2\}\\times r^\{\\prime\}\}andA∈ℝr′×n2A\\in\\mathbb\{R\}^\{r^\{\\prime\}\\times n\_\{2\}\}, putreff′=min⁡\(r′,m2,n2\)r^\{\\prime\}\_\{\\mathrm\{eff\}\}=\\min\(r^\{\\prime\},m\_\{2\},n\_\{2\}\)\. The maximum rank ismin⁡\(m1,n1\)⋅reff′\\min\(m\_\{1\},n\_\{1\}\)\\cdot r^\{\\prime\}\_\{\\mathrm\{eff\}\}, attained\. SoIm⁡LoKrr′⊆Im⁡LoRAr\\Imop\\mathrm\{LoKr\}\_\{r^\{\\prime\}\}\\subseteq\\Imop\\mathrm\{LoRA\}\_\{r\}iffr≥min⁡\(m1,n1\)​reff′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}\_\{\\mathrm\{eff\}\}, and otherwise the two images are incomparable, providedκ1\>1\\kappa\_\{1\}\>1\. In HF PEFT the second LoKr factor is low\-rank only when its rank parameter satisfiesr′<max⁡\(m2,n2\)/2r^\{\\prime\}<\\max\(m\_\{2\},n\_\{2\}\)/2; otherwise it is full, and LoKr is unconstrained KronA, with maximum rankν=min⁡\(m1,n1\)​min⁡\(m2,n2\)\\nu=\\min\(m\_\{1\},n\_\{1\}\)\\min\(m\_\{2\},n\_\{2\}\)\. On a768768\-wide layer \(24⊗3224\\otimes 32\) the maximum rank therefore jumps from360360atr′=15r^\{\\prime\}=15to768768atr′=16r^\{\\prime\}=16\. With decompose\_both,CCis also capped at rankr′r^\{\\prime\}\.

##### Numerical check

Schmidt ranks\(1,3\)→3\(1,3\)\\to 3,\(2,3\)→6\(2,3\)\\to 6,\(3,3\)→9\(3,3\)\\to 9, each with ordinary rank11\. For the2×42\\times 4split ofℝ8\\mathbb\{R\}^\{8\},rk⊗⁡\(u​v⊤\)=4\\operatorname\{rk\}\_\{\\otimes\}\(uv^\{\\top\}\)=4, the maximal Kronecker rank there\. LoKr withs=4s=4,r′=1r^\{\\prime\}=1never exceeds rank44, and at\(m1,m2,n1,n2\)=\(2,8,8,2\)\(m\_\{1\},m\_\{2\},n\_\{1\},n\_\{2\}\)=\(2,8,8,2\)with inner dimension44, LoKr never exceeds rank4=min⁡\(m1,n1\)​reff′4=\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}\_\{\\mathrm\{eff\}\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

The rearrangementℛ⁡\(C⊗D\)=vec⁡C​vec⁡D⊤\\mathcal\{R\}\(C\\otimes D\)=\\vect C\\,\\vect D^\{\\top\}\([Background B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\) and the nearest\-Kronecker\-product problem are due to Van Loan and Pitsianis\([1993](https://arxiv.org/html/2610.12728#bib.bib459)\)\. Kronecker\-sum layers entered deep learning with PHM layers\([Zhang et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib543)\)and Compacter\([Mahabadi et al\., 2021a](https://arxiv.org/html/2610.12728#bib.bib307)\); KronA\([Edalati et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib103)\)and LoKr\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\)brought them to weight updates\.

WHAT THIS EXPLAINSUnconstrained Kronecker sums \(KronA with full factors, sums of Kronecker terms\) are LoRA across a different cut of the same 4\-leg tensor, so calling them higher\-rank LoRAs mixes two cuts\. With aligned splits, a generic rank\-one update needs the maximal number of Kronecker terms, and with fewer thanκ1\\kappa\_\{1\}terms a Kronecker sum misses it\. These methods also rest on a reshape that the network does not have\. Constrained Kronecker methods behave differently\. LoKr with inner rankr′r^\{\\prime\}lies inside a LoRA image only when the LoRA rank is at leastmin⁡\(m1,n1\)​r′\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}, aboutN​r′\\sqrt\{N\}\\,r^\{\\prime\}:192192on768768\-wide and512512on40964096\-wide layers at HF’s defaultr′=8r^\{\\prime\}=8\. At practical ranks, and at equal parameter budget, LoKr andLoRAr\\mathrm\{LoRA\}\_\{r\}are incomparable \(e\)\.

Compacter’s PHM weights are Kronecker\-structured weights inside a nonlinear bottleneck adapter and do not updateW0W\_\{0\}, and MPO adapters\([Liu et al\., 2021a](https://arxiv.org/html/2610.12728#bib.bib279)\)are a multi\-core reparametrisation that is lossless at full bond dimension\. The image comparison does not transfer to either without more work\.

##### Prediction

On targetsW0\+u​v⊤W\_\{0\}\+uv^\{\\top\}with genericu,vu,v, and a loss whose only minimiser is the target \(for instance regression on full\-rank inputs\), LoRA1reaches zero loss, while no Kronecker sum with fewer thanκ1\\kappa\_\{1\}terms does\. HF PEFT’s balanced factorisation is non\-square for many widths \(768→24×32768\\to 24\\times 32withκ1=576\\kappa\_\{1\}=576, the maximal Kronecker rank;2048→32×642048\\to 32\\times 64;5120→64×805120\\to 64\\times 80\) but square for10241024and40964096\(32×3232\\times 32,64×6464\\times 64\), whereκ1=N\\kappa\_\{1\}=N\. At an equalised budget neither family should dominate the other across tasks\.

Figure[6](https://arxiv.org/html/2610.12728#S2.F6)makes the three cuts of a16×1616\\times 16matrix concrete\. On the4⊗44\\otimes 4split, a genericu​v⊤uv^\{\\top\}has rank11and Kronecker rank1616\. LoRA1fits it with3232parameters, and a Kronecker sum needs1616terms and512512parameters\. ForI16=I4⊗I4I\_\{16\}=I\_\{4\}\\otimes I\_\{4\}the costs trade places\.

Figure 6:Cut the tensor\([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\)\. A16×1616\\times 16matrixΔ\\Deltais read as a four\-leg tensor with output legsU1,U2U\_\{1\},U\_\{2\}and input legsV1,V2V\_\{1\},V\_\{2\}, throughℝ16=ℝ4⊗ℝ4\\mathbb\{R\}^\{16\}=\\mathbb\{R\}^\{4\}\\otimes\\mathbb\{R\}^\{4\}on both sides\. The legs can be split into two pairs in three ways \(columns\): the ordinary cut\{U1​U2\}\|\{V1​V2\}\\\{U\_\{1\}U\_\{2\}\\\}\\mid\\\{V\_\{1\}V\_\{2\}\\\}, across which LoRA factors asΔ=B​A\\Delta=BA; the Kronecker cut\{U1​V1\}\|\{U2​V2\}\\\{U\_\{1\}V\_\{1\}\\\}\\mid\\\{U\_\{2\}V\_\{2\}\\\}, across which a sum of Kronecker products∑iCi⊗Di\\sum\_\{i\}C\_\{i\}\\otimes D\_\{i\}factors after the rearrangement of Van Loan and Pitsianis\([1993](https://arxiv.org/html/2610.12728#bib.bib459)\); and the third cut\{U1​V2\}\|\{U2​V1\}\\\{U\_\{1\}V\_\{2\}\\\}\\mid\\\{U\_\{2\}V\_\{1\}\\\}, the Kronecker cut with the two input legs swapped\. The glyph above each column draws that factorisation as two boxes joined by a bond across the cut\. Each panel shows the operator\-Schmidt spectrum across one cut, the singular valuesσi\\sigma\_\{i\}of the corresponding16×1616\\times 16reshape ofΔ\\Deltadivided by∥Δ∥F\\lVert\\Delta\\rVert\_\{F\}, on a logarithmic scale, for \(a\) a random rank\-one updateu​v⊤uv^\{\\top\}, \(b\) the identityI16=I4⊗I4I\_\{16\}=I\_\{4\}\\otimes I\_\{4\}, and \(c\)C⊗DC\\otimes Dwith random4×44\\times 4factors\. Filled markers lie above the rank threshold \(dashed line\), and open markers are numerically zero\. The ranks are\(1,16,16\)\(1,16,16\)foru​v⊤uv^\{\\top\}and\(16,1,16\)\(16,1,16\)forI16I\_\{16\}and forC⊗DC\\otimes D\. So a generic rank\-one update has the maximal Kronecker rankκ1=16\\kappa\_\{1\}=16, and a Kronecker product of full\-rank factors has the maximal rankν=16\\nu=16, which makes zero\-init LoRArandkk\-term Kronecker sums incomparable forr,k<16r,k<16\([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(b\)–\(d\)\)\. LoRA1fitsu​v⊤uv^\{\\top\}with3232parameters where a Kronecker sum needs1616terms and512512parameters, and forI16I\_\{16\}andC⊗DC\\otimes Dthe two costs trade places\.

#### Dimension is a volume

Movingq∈Qq\\in Qalong a fibre ofρ\\rhochanges nothing\. For LoRA,\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)gives the same weight as\(B,A\)\(B,A\)for everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}, sor2r^\{2\}of ther⁡\(m\+n\)r\(m\+n\)parameters are spent on a symmetry and the image has dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\. This is the expressive dimensiond⁡\(M\)d\(M\)of[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), and\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\)is the gauge waste\. “Gauge” means the symmetry group of a parametrisation \([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\. There is no connection, and away from the full\-rank stratum the fibres jump in dimension\.

##### Credit

The quotient\(ℝ∗m×r×ℝ∗r×n\)/GLr\(\\mathbb\{R\}^\{m\\times r\}\_\{\*\}\\times\\mathbb\{R\}^\{r\\times n\}\_\{\*\}\)/\\mathrm\{GL\}\_\{r\}is the classical geometry of fixed\-rank matrices\([Mishra et al\., 2012](https://arxiv.org/html/2610.12728#bib.bib330)\)\.[Putterman et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib375)build learners on LoRA weights that respect thisGLr\\mathrm\{GL\}\_\{r\}symmetry\.

The census below covers LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\), GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\), LoHa\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\), HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\), DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\), VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\), KronA\([Edalati et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib103)\), tensor\-train and Tucker adapters \([Background B\.9](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\), and the linear methods LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\), LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\), MiSS\([Kang and Yin, 2024](https://arxiv.org/html/2610.12728#bib.bib206)\), FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\), C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\)and masks\.

Proposition II\.12\(gauge census: rank is a cut, dimension is a volume\)\.\(a\)d⁡\(M\)≤\|M\|−dim\(generic gauge orbit\)d\(M\)\\leq\\lvert M\\rvert\-\\dim\(\\text\{generic gauge orbit\}\), with equality when the gauge acts locally transitively on generic fibres\.\(b\)The values in Table[9](https://arxiv.org/html/2610.12728#S2.T9)hold for onem×nm\\times nbox, withr≤min⁡\(m,n\)r\\leq\\min\(m,n\)throughout except for VeRA \(whose published RoBERTa\-base setting hasr=1024\>768r=1024\>768\)\. In the LoHa row,F=r1​\(m\+n−r1\)\+r2​\(m\+n−r2\)−\(m\+n−1\)F=r\_\{1\}\(m\+n\-r\_\{1\}\)\+r\_\{2\}\(m\+n\-r\_\{2\}\)\-\(m\+n\-1\)is a proved upper bound fordd\. The gauge column lists the exhibited Lie group\. It acts locally transitively on generic fibres away from them​nmncap, but not where the cap binds; for LoHa at\(m,n,r1,r2\)=\(4,4,2,2\)\(m,n,r\_\{1\},r\_\{2\}\)=\(4,4,2,2\)the orbit has dimension1515and the fibre1616\. BitFit does not touch them×nm\\times nbox; it is linear in the biases\. Hybrid GraLoRA \(hybrid\_r\>0\{\}\>0\) is not covered by the table\. FourierFT is injective only when its sampled frequencies contain no conjugate pair\(u,v\),\(−u,−v\)\(u,v\),\(\-u,\-v\); otherwised=\|M\|−\#​\{pairs\}d=\\lvert M\\rvert\-\\\#\\\{\\text\{pairs\}\\\}\. At LLM sizes this is close to\|M\|\\lvert M\\rvert, but at768×768768\\times 768with10001000frequencies a pair occurs in about half of the draws \(HF’s default seed777777gives exactly two pairs there\)\.\(c\)*Equal\-budget trades\.*Assume2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\),r2≤min⁡\(m,n\)r^\{2\}\\leq\\min\(m,n\)and2​r2<m\+n−12r^\{2\}<m\+n\-1\. LoHar,rand LoRA2rboth cost2​r​\(m\+n\)2r\(m\+n\), but LoHa reaches at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}*fewer*dimensions \(exactly that many whenFFis attained \[Num\]\)\. Forr≥3r\\geq 3it gains rank up tor2\>2​rr^\{2\}\>2rin exchange\. Forr≤2r\\leq 2,r2≤2​rr^\{2\}\\leq 2r, soIm⁡LoHar,r⊊Im⁡LoRA2​r\\Imop\\mathrm\{LoHa\}\_\{r,r\}\\subsetneq\\Imop\\mathrm\{LoRA\}\_\{2r\}and LoHa is strictly dominated\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1\(still with2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\) the bound no longer separates them, and whereverFFis attained LoHar,rmatches or exceeds LoRA2rin dimension as well as in rank, for instance at\(m,n,r\)=\(9,9,3\)\(m,n,r\)=\(9,9,3\), atr=64r=64on4096×40964096\\times 4096, and at LyCORIS\-styler=32r=32on320320\- to640640\-wide layers\. GraLoRAkcosts the same as LoRAr, reaches the same number of dimensions, and has maximum rankmin⁡\(k​r,m,n\)\\min\(kr,m,n\), but caps every block at rankr/kr/k\. A generic rank\-rrupdate has rankrron every block, so fork≥2k\\geq 2andk​r≤min⁡\(m,n\)kr\\leq\\min\(m,n\)the images of LoRArand GraLoRAkare incomparable\.

Table 9:The gauge census of[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(b\) for onem×nm\\times nbox: budget, exhibited Lie gauge with the dimension of its generic orbit, expressive dimension and maximum rank\. \[Num\] marks values checked as float64 Jacobian ranks \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\);FFis defined in[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(b\)\.
##### Numerical check

LoHa dimensions3838,5959,9393,101101at\(m,n,r\)=\(8,7,2\)\(m,n,r\)=\(8,7,2\),\(12,10,2\)\(12,10,2\),\(12,10,3\)\(12,10,3\),\(20,16,2\)\(20,16,2\), and16=m​n16=mnat\(4,4,2\)\(4,4,2\), where the formula gives1717andd⁡\(LoRA4\)=16d\(\\mathrm\{LoRA\}\_\{4\}\)=16too\. LoHa3,3against LoRA6:7373against7272at\(9,9,3\)\(9,9,3\)\. GraLoRA dimensions4848,6464,108108at\(m,n,r,k\)=\(8,8,4,2\)\(m,n,r,k\)=\(8,8,4,2\),\(12,8,4,2\)\(12,8,4,2\),\(12,12,6,3\)\(12,12,6,3\); GraLoRA generic rank8=k​r8=krat\(16,16,4,2\)\(16,16,4,2\)and12=min⁡\(m,n\)12=\\min\(m,n\)at\(12,12,6,3\)\(12,12,6,3\)\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

LoHa is the low\-rank Hadamard product of FedPara\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\); GraLoRA is due to[Jung et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib203)\. The Tucker gauge and dimension are due to[Koch and Lubich \(2010\)](https://arxiv.org/html/2610.12728#bib.bib220)and[Uschmajew and Vandereycken \(2013\)](https://arxiv.org/html/2610.12728#bib.bib454), and the TT case to[Holtz et al\. \(2012\)](https://arxiv.org/html/2610.12728#bib.bib172), building on[Oseledets \(2011\)](https://arxiv.org/html/2610.12728#bib.bib354)\. The intrinsic\-dimension experiments of[Li et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib246)and[Aghajanyan et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib5)measure how many random directions a task needs;d⁡\(M\)d\(M\)counts the directions a method’s image has, a complementary notion\.

LoHar,rmaps into LoRAr2\{\}\_\{r^\{2\}\}through the face\-splitting product \([Background B\.13](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)\), one of the arrow rules of Exposé[VIII](https://arxiv.org/html/2610.12728#S8); forr≤2r\\leq 2this places LoHar,rinside LoRA2r\.

WHAT THIS EXPLAINSRank is a*cut*invariant and dimension a*volume*invariant, and the literature conflates them\. Whenr≥3r\\geq 3and2​r22r^\{2\}is small next tom\+nm\+n, as on LLM projections at the usual ranks, LoHa reaches high rank on a thin, curved subvariety, because its Hadamard product spends a whole torus on gauge\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1, as in LyCORIS’s usual Stable Diffusion settings, it matches or exceeds LoRA2rin dimension too\. GraLoRA keeps LoRA’s dimension and multiplies the maximum rank bykk, but caps every block at rankr/kr/k\. It moves the cut, and the block caps pay for the higher maximum rank\.

#### Transport of structure

An invertible linear change of frame fixingθ0\\theta\_\{0\}carries dimension, gauge, first\-order image and degree along, since they are defined fromρ\\rhoand its differential \([Background A\.17](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\)\. It carries rank along only when it respects rows and columns, which the Kronecker rearrangement, Fourier and wavelet frames and the Hadamard rescaling byW0W\_\{0\}do not\.

Proposition II\.13\(transport of structure\)\.ForT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\)putτT​\(θ\)=θ0\+T⁡\(θ−θ0\)\\tau\_\{T\}\(\\theta\)=\\theta\_\{0\}\+T\(\\theta\-\\theta\_\{0\}\)andT∗​M:=\(Q,q0,τT∘ρ\)T\_\{\*\}M:=\(Q,q\_\{0\},\\tau\_\{T\}\\circ\\rho\)\.•T∗T\_\{\*\}is a strict automorphism of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\), with inverse\(T−1\)∗\(T^\{\-1\}\)\_\{\*\}\. It mapsS1S\_\{1\}toT​S1T\\,S\_\{1\}and preservesdd,gg, the gauge group, the first\-order deficit, polynomial degree, the regular or apex type and the germ type of the image\. It does not preserve rank, and, for the family extension below, it does not preserve the covariance group\.•KT∗​M=T​KM​T⊤K^\{T\_\{\*\}M\}=T\\,K^\{M\}\\,T^\{\\top\}for everyTT\. For any update rule that queries the loss only throughq↦L⁡\(ρ⁡\(q\)\)q\\mapsto L\(\\rho\(q\)\), with its own state onQQ,T∗​MT\_\{\*\}Mtrained onLLandMMtrained onL∘τTL\\circ\\tau\_\{T\}have identicalqq\-trajectories, so theirθ\\theta\-trajectories are conjugate byτT\\tau\_\{T\}\. Rules that read∇θL\\nabla\_\{\\theta\}Land solve a least\-squares problem onΘ\\Theta\(LoRA\-Pro\), andθ\\theta\-level weight decay or clipping, are covered only for orthogonalTT\. Orthogonality ofTTis exactly the condition forτT\\tau\_\{T\}to be a Frobenius isometry; for rules of the first kind it is needed only to keep the Frobenius spectrum ofKK, or an isometry\-invariant loss class such as∥θ−θ⋆∥2\\lVert\\theta\-\\theta^\{\\star\}\\rVert^\{2\}, unchanged\.•Coordinatewise optimizers \(SGD, Adam\) are also equivariant under coordinate permutations ofQQ\. The Kronecker identification below needs this, because it composesℛ∗−1\\mathcal\{R\}^\{\-1\}\_\{\*\}with the vec reshape ofQQ\.•*Families and rebasing\.*ForTTindependent of the base point, define the family extension\(T∗​M\)​\(θ\):=\(Q,q0,θ\+T⁡\(ρθ−θ\)\)\(T\_\{\*\}M\)\(\\theta\):=\(Q,q\_\{0\},\\theta\+T\(\\rho\_\{\\theta\}\-\\theta\)\)\. For an additive family with fixed shapeCC\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\),T∗​MT\_\{\*\}Mis additive with shapeT​CTC, so transport commutes with rebasing:RK​\(T∗​M\)=τT​\(RK​\(M\)\)R\_\{K\}\(T\_\{\*\}M\)=\\tau\_\{T\}\(R\_\{K\}\(M\)\)\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(a\)\)\. For action\-type families this can fail even forT=2​IT=2I, because the transported family acts byI\+T⁡\(g−I\)I\+T\(g\-I\)\. For OFT onΘ=ℝ1×2\\Theta=\\mathbb\{R\}^\{1\\times 2\}, two cycles ofθ↦θ⁡\(2​R−I\)\\theta\\mapsto\\theta\(2R\-I\)withR=rot⁡\(π\)R=\\operatorname\{rot\}\(\\pi\)reach9​θ09\\theta\_\{0\}, outside the transported circle\. Transports that depend onθ0\\theta\_\{0\}may fail \(HiRA below\) or may not \(LoRA transported by the SVD frame ofθ0\\theta\_\{0\}, sinceℳ≤r\\mathcal\{M\}\_\{\\leq r\}isO⁡\(m\)×O⁡\(n\)\\mathrm\{O\}\(m\)\\times\\mathrm\{O\}\(n\)\-invariant\)\. The covariance group \([Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\) is a property of a family and is compared through this extension\.Instances\.•*Kronecker family\.*An isomorphism of slices overΘ′=ℝm1​n1×m2​n2\\Theta^\{\\prime\}=\\mathbb\{R\}^\{m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}\}and overΘ\\Theta:∑i≤kCi⊗Di=ℛ∗−1​LoRAk\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}=\\mathcal\{R\}^\{\-1\}\_\{\*\}\\mathrm\{LoRA\}\_\{k\}, withQQreshaped by vec, dynamics included for coordinatewise optimizers\. KronA with unconstrained factors isℛ∗−1​LoRA1\\mathcal\{R\}^\{\-1\}\_\{\*\}\\mathrm\{LoRA\}\_\{1\}\. LoKr isℛ∗−1\\mathcal\{R\}^\{\-1\}\_\{\*\}ofLoRA1\\mathrm\{LoRA\}\_\{1\}with the second factor constrained tovec⁡ℳ≤r′\\vect\\mathcal\{M\}\_\{\\leq r^\{\\prime\}\}whenr′<max⁡\(m2,n2\)/2r^\{\\prime\}<\\max\(m\_\{2\},n\_\{2\}\)/2\(HF’s switch\); otherwise HF’s second factor is full and LoKr is KronA\. LoKr’s decompose\_both option also constrains the first factor\.•*Linear methods as coordinate subspaces in transported frames\.*–FourierFT: sparse in the real cosine half of the real Fourier frame\. With scalingcc\(default150150\), HF’s updateΔ​W=c​Re⁡\(ifft2⁡\(S\)\)\\Delta W=c\\operatorname\{Re\}\(\\operatorname\{ifft2\}\(S\)\)is the transport of a mask, withg=0g=0, only when the sampled index set has no conjugate pair\(u,v\),\(−u,−v\)\(u,v\),\(\-u,\-v\); otherwiseggis the number of such pairs\. Its image lies in the point\-symmetric subspaceΔ​Wj,l=Δ​W−j,−l\\Delta W\_\{j,l\}=\\Delta W\_\{\-j,\-l\}, and the frame is orthogonal only up to scale \(self\-conjugate frequencies have twice the squared norm\)\.–WaveFT: sparse in a wavelet frame, when HF’s pad–IDWT–crop map is invertible \(plausible for Haar on even shapes; unverified for other wavelet families\)\.–C3A: block\-circulant\. Overℝ\\mathbb\{R\}the frame is given by the powersPkP^\{k\}of the cyclic shift \(Gram matrixb​IbI\)\. Overℂ\\mathbb\{C\},circ⁡\(w\)=F−1​diag⁡\(F​w\)​F\\operatorname\{circ\}\(w\)=F^\{\-1\}\\diag\(Fw\)Ffor first columnww, and the real form has2×22\\times 2rotation–scaling blocks\. HF adds a factor1/in\_features1/\\texttt\{in\\\_features\}\.–LoRA\-XS, SVFT, TinyLoRA: masks or random subspaces in the SVD frame ofW0W\_\{0\}, aθ0\\theta\_\{0\}\-dependent transport\.–SHiRA, FISH, Trainable Tokens: masks in the standard frame\.–MiSS/Bone in its default \(balance\) mode:Δ​W=D⁡\(𝟏g⊤⊗Ir\)\\Delta W=D\(\\mathbf\{1\}\_\{g\}^\{\\top\}\\otimes I\_\{r\}\), which is LoRA\-FA with a frozen*summing*projection\. Its BAT mode,Δ​W=blocks⁡\(W0\)​M\+M\\Delta W=\\operatorname\{blocks\}\(W\_\{0\}\)\\,M\+Mblockwise, is linear in the trained blockMMbut depends onθ0\\theta\_\{0\}, so it is not additive\.–These frame readings of FourierFT and C3A are for init\_weights=True\. HF’s defaults start offθ0\\theta\_\{0\}: FourierFT draws its spectrum from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)with scaling150150\(at7682768^\{2\}with10001000frequencies,∥Δ​W0∥F≈4\.3\\lVert\\Delta W\_\{0\}\\rVert\_\{F\}\\approx 4\.3, about14%14\\%of a std\-0\.040\.04W0W\_\{0\}\), and C3A is Xavier\-uniform\. These are objects with a pointing defect\.•*HiRA*=\(DW0\)∗​LoRA=\(D\_\{W\_\{0\}\}\)\_\{\*\}\\mathrm\{LoRA\}inside the slice atθ0\\theta\_\{0\}, whenW0W\_\{0\}has no zero entries, withDW0:X↦W0⊙XD\_\{W\_\{0\}\}:X\\mapsto W\_\{0\}\\odot X\. BecauseDW0D\_\{W\_\{0\}\}depends onθ0\\theta\_\{0\}, re\-based HiRA is not transported ReLoRA\. Two cycles reachW0⊙\(J\+Z1\)⊙\(J\+Z2\)W\_\{0\}\\odot\(J\+Z\_\{1\}\)\\odot\(J\+Z\_\{2\}\), whose relative updateZ1\+Z2\+Z1⊙Z2Z\_\{1\}\+Z\_\{2\}\+Z\_\{1\}\\odot Z\_\{2\}has rank up tomin⁡\(2​r\+r2,m,n\)\\min\(2r\+r^\{2\},m,n\), attained, where transported ReLoRA stays at rank≤2​r\\leq 2r\.

##### Numerical check

∥KT∗​M−T​KM​T⊤∥=2×10−14\\lVert K^\{T\_\{\*\}M\}\-TK^\{M\}T^\{\\top\}\\rVert=2\\times 10^\{\-14\}for a GaussianTT\. Adam onT∗​MT\_\{\*\}MwithLLand onMMwithL∘τTL\\circ\\tau\_\{T\}agree to8×10−118\\times 10^\{\-11\}\. Two HiRA cycles give relative\-update rank33,88,1515atr=1,2,3r=1,2,3\(m=n=30m=n=30\) \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

The transported methods are FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\), WaveFT\([Bilican et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib35)\), C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\), LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\), SVFT\([Lingam et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib272)\), TinyLoRA\([Morris et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib336)\), SHiRA\([Bhardwaj et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib30)\), FISH\([Sung et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib438)\), Trainable Tokens\([team, 2025](https://arxiv.org/html/2610.12728#bib.bib449)\), MiSS\([Kang and Yin, 2024](https://arxiv.org/html/2610.12728#bib.bib206)\)and HiRA\([Huang et al\., 2025b](https://arxiv.org/html/2610.12728#bib.bib182)\); LoRA\-Pro is due to[Wang et al\. \(2024i\)](https://arxiv.org/html/2610.12728#bib.bib486)\.[Si et al\. \(2024c\)](https://arxiv.org/html/2610.12728#bib.bib417)organise PEFT around the SVD frame ofW0W\_\{0\}, a related unification\.

WHAT THIS EXPLAINS“High rank” claims \(MoRA, HiRA, LoKr, C3A, SHiRA, WaveFT\) are notGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-invariant, although rank isGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\-invariant\. The transports in play \(ℛ\\mathcal\{R\}, the real Fourier \(cosine\) frame,DW0D\_\{W\_\{0\}\}\) do not preserve the network function, and[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)shows that KronA andLoRAr\\mathrm\{LoRA\}\_\{r\}have incomparable images whenr<min⁡\(m1,n1\)​min⁡\(m2,n2\)r<\\min\(m\_\{1\},n\_\{1\}\)\\min\(m\_\{2\},n\_\{2\}\)andκ1\>1\\kappa\_\{1\}\>1\. Transport preserves every isomorphism invariant of the method \(dd,gg, the gauge group,dimS1\\dim S\_\{1\}, the deficit, the regular or apex type, the germ type and the degree\) and conjugates its dynamics\. Relative to the network’sGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}structure it changes, in particular, rank, covariance,S1S\_\{1\}as a subspace, and the image itself\. In Figure[4](https://arxiv.org/html/2610.12728#S2.F4)HiRA therefore sits at LoRA’s height and at full rank \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(a\)\)\.

##### Prediction

At matched learning\-rate sweeps, the performance gap between a transported method and its source \(Kronecker sums against LoRA; Fourier or wavelet methods against coordinate masks\) is explained by how well each image aligns with the task’s update, measured for instance by projecting a full fine\-tuningΔ​W\\Delta Wonto each image, more than by optimizer effects\.

#### The plug, and a reporting rule

CutW0W\_\{0\}out of a linear layer and a holeV→UV\\to Uremains\. A weight\-space method fills it with a small network of frozen and trainable boxes, a*plug*\(Figure[5](https://arxiv.org/html/2610.12728#S2.F5)\)\. Every invariant above is read off the plug: the budget \([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\), the min\-cut, the Jacobian rank \([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\), the gauges on its wires, and the merge type, set by whether an activation sits on the path \([Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\)\. The plug is a weight\-space cousin of the unified view of[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165), which puts adapters, prefix tuning and LoRA in one design space\.

Table[10](https://arxiv.org/html/2610.12728#S2.T10)spends one budget three ways on one4096×40964096\\times 4096box \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. The methods differ by at least8,0638\{,\}063dimensions and by a factor of four in rank\. The LoHa row uses the upper boundFF, whose equality is checked numerically at small sizes and open in general\.

Table 10:One4096×40964096\\times 4096box and one budget,\|M\|=131,072\\lvert M\\rvert=131\{,\}072, spent three ways \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\.##### Prediction

Benchmarks should reportd⁡\(M\)d\(M\)beside\|M\|\\lvert M\\rvert, with the maximum rank and the cut it refers to\. At budgets of a few thousand parameters, rankings at equal*effective*dimension can change\.

A small case shows every readout at once\. On an8×88\\times 8box, LoHa2,2has6464parameters, rank at most44, Jacobian rank4141and a gauge orbit of dimension23=4\+4\+1523=4\+4\+15that fills the fibre\. LoRA4costs the same and reaches77more dimensions, with LoHa’s image strictly inside it \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(c\), since2​r2=8<15=m\+n−12r^\{2\}=8<15=m\+n\-1\)\. The plug workbench of the companion site recomputes these invariants in float64 for fifteen presets and for plugs the reader edits: the budget, the degree, the min\-cut bound by brute force over vertex bipartitions beside the rank attained,d⁡\(M\)d\(M\)as a Jacobian rank, the gauge orbit with each generator checked to lie inker⁡d​ρ\\ker d\\rho, and the merge verdict\.

#### Covariance and descent

A method is usually one recipe applied to every pretrained weight\. Rename a layer’s input basis byhhand its output basis bygg, so thatW0W\_\{0\}becomesg​W0​h−1gW\_\{0\}h^\{\-1\}; the method is covariant under\(g,h\)\(g,h\)if its image moves the same way\.

Definition II\.14\(covariance group\)\. For a method defined for everyW0W\_\{0\}\(a natural family\), thecovariance group𝒢M⊆GLm×GLn\\mathcal\{G\}\_\{M\}\\subseteq\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}is the largest subgroup withIm⁡M⁡\(g​W0​h−1\)=g​Im⁡M⁡\(W0\)​h−1\\Imop M\(gW\_\{0\}h^\{\-1\}\)=g\\,\\Imop M\(W\_\{0\}\)\\,h^\{\-1\}\. When the method depends on random frames or on data, the identity is required in distribution, with the data transformed\.𝒢M\\mathcal\{G\}\_\{M\}is the*geometry the method lives in*, in Klein’s sense\.

Klein defined a geometry by its group of motions and studied what the group leaves unchanged\([Klein, 1872](https://arxiv.org/html/2610.12728#bib.bib217)\)\([Background D\.17](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\)\. For group\-type methods this reading is a theorem \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\)\. For cones such as LoRA’s image it is a naming device, and the content is the shape of the image and its covariance group\.

LoRA’s imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}is covariant under all ofGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, since rank ignores bases, while a coordinate mask has a much smaller group\. This matters because networks have symmetries of their own, such as permutations of hidden neurons, rotations of the residual stream and the query–key gauge of attention, which change the weights and keep the function\.

Observation II\.15\(descent through function\-preserving reparametrisations\)\.LetM=∏bMbM=\\prod\_\{b\}M\_\{b\}be a box\-wise method: eachMbM\_\{b\}is a member of a natural family with covariance group𝒢Mb\\mathcal\{G\}\_\{M\_\{b\}\}, and no trainable parameters or random frames are shared across boxes\. LetGarchG\_\{\\mathrm\{arch\}\}act onΘ=∏bΘb×Θfrozen\\Theta=\\prod\_\{b\}\\Theta\_\{b\}\\times\\Theta\_\{\\mathrm\{frozen\}\}, preserving this split and acting on each adapted box byWb↦gb​Wb​hb−1W\_\{b\}\\mapsto g\_\{b\}W\_\{b\}h\_\{b\}^\{\-1\}with\(gb,hb\)∈𝒢Mb\(g\_\{b\},h\_\{b\}\)\\in\\mathcal\{G\}\_\{M\_\{b\}\}\. Assumef⁡\(g​θ,−\)=f⁡\(θ,−\)f\(g\\theta,\-\)=f\(\\theta,\-\)on aGarchG\_\{\\mathrm\{arch\}\}\-stable set containingIm⁡M⁡\(θ0\)\\Imop M\(\\theta\_\{0\}\)\. ThenIm⁡M⁡\(g​θ0\)=g⋅Im⁡M⁡\(θ0\)\\Imop M\(g\\theta\_\{0\}\)=g\\cdot\\Imop M\(\\theta\_\{0\}\), and the set of*functions*reachable byMMfromg​θ0g\\theta\_\{0\}equals the set reachable fromθ0\\theta\_\{0\}\. For methods with random frames, the equality holds in distribution, provided the frames are drawn independently per box\. Boxes that share a frame need a pathwise condition: a law\-preserving mapT\(g,h\)T\_\{\(g,h\)\}on frame space withIm⁡Mb​\(gb​W​hb−1,T​F\)=gb​Im⁡Mb​\(W,F\)​hb−1\\Imop M\_\{b\}\(g\_\{b\}Wh\_\{b\}^\{\-1\};TF\)=g\_\{b\}\\,\\Imop M\_\{b\}\(W;F\)\\,h\_\{b\}^\{\-1\}for allWWandFF, with the sameTTon every box that sharesFF\(marginal covariance in law, as in[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), is not enough\)\. For data\-dependent methods, require covariance pathwise in the data, and that the box\-bbdata atg​θ0g\\theta\_\{0\}is the[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)transform under\(gb,hb\)\(g\_\{b\},h\_\{b\}\)of the data atθ0\\theta\_\{0\}\. This holds for intertwiner gauges such as hidden\-unit permutations, query–key and value–output head gauges and residual rotations\. The statement concerns reachable sets only; training trajectories and outcomes can still differ, for instance under Adam or non\-isotropic initialisations\.Examples ofGarchG\_\{\\mathrm\{arch\}\}:•hidden\-neuron permutations;•the query–key gauge\(WQ,WK\)↦\(g​WQ,g−⁣⊤​WK\)\(W\_\{Q\},W\_\{K\}\)\\mapsto\(gW\_\{Q\},g^\{\-\\top\}W\_\{K\}\)per head\. With no positional encoding or an absolute one,g∈GLdkg\\in\\mathrm\{GL\}\_\{d\_\{k\}\}\. Under RoPE,ggmust commute with every RoPE rotation, which leaves the per\-frequency2×22\\times 2rotation–scalings\(ℂ×\)dk/2\(\\mathbb\{C\}^\{\\times\}\)^\{d\_\{k\}/2\}, shared across a GQA group\. Under partial RoPE onrrof thedkd\_\{k\}dimensions the group is\(ℂ×\)r/2×GLdk−r\(\\mathbb\{C\}^\{\\times\}\)^\{r/2\}\\times\\mathrm\{GL\}\_\{d\_\{k\}\-r\}\. Under QK\-norm,ggmust in addition be orthogonal when the norm’sε\>0\\varepsilon\>0\(conformal,g∈CO⁡\(dk\)g\\in\\mathrm\{CO\}\(d\_\{k\}\), whenε=0\\varepsilon=0\) and satisfyg⊤​Γq​Rt​Γk​g=Γq​Rt​Γkg^\{\\top\}\\Gamma\_\{q\}R\_\{t\}\\Gamma\_\{k\}g=\\Gamma\_\{q\}R\_\{t\}\\Gamma\_\{k\}for every RoPE rotationRtR\_\{t\}, whereΓq,Γk\\Gamma\_\{q\},\\Gamma\_\{k\}are the gains\. OLMo 2 normalisesqqandkkacross all heads jointly;•a single global orthogonal rotationQQof the residual stream of a pre\-norm transformer whose norm gains can all be fused into the linear layers they feed, after LayerNorm is converted to RMSNorm \(SliceGPT’s computational invariance; QuaRot’s offline rotation\)\. SliceGPT’s per\-block rotationsQℓQ\_\{\\ell\}need extra residual matrices and change the architecture\. Sandwich and post\-sublayer norms \(Gemma 2 and 3, OLMo 2\) are excluded, because their gains cannot be fused anddiag⁡\(γ\)\\diag\(\\gamma\)does not commute withQQ\. The comparison is between the gain\-fused model and the fused\-and\-rotated model: gain fusion is a diagonal input rescaling outsideGarchG\_\{\\mathrm\{arch\}\}that already changes PiSSA’s split and block OFT’s image \([Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\), and converting LayerNorm to RMSNorm changes LoRA’s reachable set on the residual writers\.

##### Numerical check

A tied rank\-one update shared by two boxes violates the conclusion \(relative distance up to0\.300\.30\), while independent per\-box LoRA satisfies it\. A Haar frame shared by two boxes, each covariant in law, gives a joint statistic of0\.440\.44fromθ0\\theta\_\{0\}against0\.810\.81fromg​θ0g\\theta\_\{0\}\. Under RoPE a genericg∈GL8g\\in\\mathrm\{GL\}\_\{8\}changes attention scores by up to313313, a RoPE\-commutingggby5×10−145\\times 10^\{\-14\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

RoPE is due to[Su et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib432)\. Residual\-stream rotations are the device of SliceGPT\([Ashkboos et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib18)\)and of QuaRot\([Ashkboos et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib19)\)\. The architectures with sandwich or post\-sublayer norms are Gemma 2\([Gemma Team et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib129)\), Gemma 3\([Gemma Team et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib130)\)and OLMo 2\([Team OLMo et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib450)\)\.

WHAT THIS EXPLAINSDescent is decided by whether each induced pair\(gb,hb\)\(g\_\{b\},h\_\{b\}\)lies in𝒢Mb\\mathcal\{G\}\_\{M\_\{b\}\}\. For one\-sided methods \(\(IA\)3, DoRA, OFT, BOFT\) this is a question of side\. For coordinate\-structured methods \(LoHa withr≥2r\\geq 2, HiRA, masks, FourierFT, Kronecker methods\) it is a question of group\. They descend through hidden\-unit monomial gauges, sinceP⁡\(X⊙Y\)​Q=\(P​X​Q\)⊙\(P​Y​Q\)P\(X\\odot Y\)Q=\(PXQ\)\\odot\(PYQ\)for permutation matricesP,QP,Qand a diagonal scaling can be absorbed into one factor, and they do not descend through rotations\. A global residual rotation acts on the input side ofqq,kk,vv, gate and up, and on the output side ofooand down\.

Table[11](https://arxiv.org/html/2610.12728#S2.T11)sorts common methods by this criterion\. Failing a sufficient condition does not prove that the reachable function sets differ\.

Table 11:Methods against the sufficient condition of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\. For the residual rotation the comparison is between the gain\-fused model and the fused\-and\-rotated model\.Conjecture A\(reachable sets under a residual rotation\)\.Against a gain\-fused but unrotated control, fine\-tuning the fused\-and\-rotated, function\-identical model measurably changes the reachable function sets, and the results, of block OFT/BOFT on residual\-input projections, LoHa withr≥2r\\geq 2, HiRA and coordinate masks, because LLM residual streams have a privileged basis\.

QuaRot’s per\-head Hadamard onWvW\_\{v\}andWoutW\_\{\\mathrm\{out\}\}is a value–output gauge inGarchG\_\{\\mathrm\{arch\}\}\. Its online Hadamards \(across heads, before down\_proj, on keys\) preserve the realised function but insert new operations, so they change the architecture map and lie outsideGarchG\_\{\\mathrm\{arch\}\}\. The optimizer adds a further layer\. Adam is onlyBrB\_\{r\}\-equivariant \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\) and HF’s Kaiming\-uniformAAis not rotation\-invariant in law, so LoRA*trained with Adam*need not descend even though its image does\.

#### Covariant data\-aware splits

SVD\-based initialisations raise the same question\. PiSSA, MiLoRA and CorDA splitW0W\_\{0\}into a trainable part and a frozen residual by a truncated SVD ofW0W\_\{0\}or, for CorDA, ofW0W\_\{0\}weighted by the input second moment\. Truncated SVD commutes only with rotations and uniform scalings, so these splits depend on the input coordinates, and whitening removes that dependence\.

Proposition II\.16\(covariant data\-aware splits\)\.LetC=𝔼⁡\[x​x⊤\]≻0C=\\mathbb\{E\}\[xx^\{\\top\}\]\\succ 0be the input second moment, so that underx↦h​xx\\mapsto hxwe haveW↦W​h−1W\\mapsto Wh^\{\-1\}andC↦h​C​h⊤C\\mapsto hCh^\{\\top\}\. LetL​L⊤=CLL^\{\\top\}=Candp:=min⁡\(m,n\)p:=\\min\(m,n\)\. Write\[M\]r\[M\]\_\{r\}for the rank\-rrtruncated SVD,⟨M⟩r:=M−\[M\]p−r\\langle M\\rangle\_\{r\}:=M\-\[M\]\_\{p\-r\}for the bottomrrsingular components, andCO⁡\(n\)=ℝ\>0⋅O⁡\(n\)\\mathrm\{CO\}\(n\)=\\mathbb\{R\}\_\{\>0\}\\cdot\\mathrm\{O\}\(n\)\. Assume1≤r<p1\\leq r<pand the spectral gap each split needs:σr\>σr\+1\\sigma\_\{r\}\>\\sigma\_\{r\+1\}ofW0​LW\_\{0\}Lin \(a\); in \(b\),σr\>σr\+1\\sigma\_\{r\}\>\\sigma\_\{r\+1\}ofW0​CW\_\{0\}CorW0W\_\{0\}for top\-rrsplits, andσp−r\>σp−r\+1\\sigma\_\{p\-r\}\>\\sigma\_\{p\-r\+1\}for bottom\-rrsplits\. Covariance groups are taken over the generic set where these gaps hold\.\(a\) The whitened splitX∗=arg⁡minrank⁡X≤r⁡∥\(W0−X\)​L∥F=\[W0​L\]r​L−1=Πr​W0,X^\{\*\}=\\arg\\min\_\{\\rk X\\leq r\}\\lVert\(W\_\{0\}\-X\)L\\rVert\_\{F\}=\[W\_\{0\}L\]\_\{r\}L^\{\-1\}=\\Pi\_\{r\}W\_\{0\},whereΠr\\Pi\_\{r\}is the top\-rreigenprojector of the output second momentW0​C​W0⊤W\_\{0\}CW\_\{0\}^\{\\top\}, isGLn\\mathrm\{GL\}\_\{n\}\-covariant on the input side andCO⁡\(m\)\\mathrm\{CO\}\(m\)\-covariant on the output side; its joint group isCO⁡\(m\)×GLn\\mathrm\{CO\}\(m\)\\times\\mathrm\{GL\}\_\{n\}\. This is the classical reduced\-rank\-regression \(SVD\-LLM\) solution\.\(b\) The formula\-level CorDA splits, IPM \(X=\[W0​C\]r​C−1X=\[W\_\{0\}C\]\_\{r\}C^\{\-1\}, the minimiser of∥\(W0−X\)​C∥F\\lVert\(W\_\{0\}\-X\)C\\rVert\_\{F\}\) and KPM \(the bottom\-rrcomponents ofW0​CW\_\{0\}Cmapped back byC−1C^\{\-1\},X=⟨W0​C⟩r​C−1X=\\langle W\_\{0\}C\\rangle\_\{r\}C^\{\-1\}\), with or without CorDA’s trace\-relative dampingC\+κ⁡\(tr⁡C/n\)​IC\+\\kappa\(\\tr C/n\)I, and the PiSSA and MiLoRA splits \(\[W0\]r\[W\_\{0\}\]\_\{r\}and the bottom\-rrpart⟨W0⟩r\\langle W\_\{0\}\\rangle\_\{r\}\), have input\-side covariance group exactlyCO⁡\(n\)\\mathrm\{CO\}\(n\); jointly,CO⁡\(m\)×CO⁡\(n\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)\.\(c\)*Scope\.*An absolute dampingC\+λ​IC\+\\lambda Ireduces the input\-side group of \(a\) and of \(b\) toO⁡\(n\)\\mathrm\{O\}\(n\)\. The official and HF CorDA code divide each calibration sample by the absolute value of its largest entry before accumulatingCC, which keeps only positive scalings and coordinate permutations, so \(b\) describes the CorDA formulas and does not cover shipped CorDA under rotations\.

##### Numerical check

Underh=3​Qh=3QwithQQorthogonal, CorDA \(IPM and bottom\-rrKPM\), PiSSA and bottom\-rrMiLoRA move covariantly to4×10−154\\times 10^\{\-15\}\. Under a generichhonly the whitened split does \(2×10−152\\times 10^\{\-15\}, against0\.630\.63and1\.11\.1\)\. Atr=nr=nPiSSA isGLn\\mathrm\{GL\}\_\{n\}\-covariant\. An absolute damping0\.10\.1breaks the whitened split under a generichh\(0\.140\.14\), and underh=3​Qh=3Q\(about2×10−22\\times 10^\{\-2\}\), while trace\-relative damping keeps it \(3×10−153\\times 10^\{\-15\}\) \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

The splits are CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\), PiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)and MiLoRA\([Wang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib465)\)\. The whitened truncation is the one used by SVD\-LLM\([Wang et al\., 2025c](https://arxiv.org/html/2610.12728#bib.bib479)\)for compression\.

WHAT THIS EXPLAINSA design rule from the Erlangen programme\. A sufficient condition for a data\-aware Type II split to be independent of how a layer’s input coordinates are chosen is that it be a left multiple ofW0W\_\{0\}by a matrix\-valued function of the output second moment,X=Φ⁡\(W0​C​W0⊤\)​W0X=\\Phi\(W\_\{0\}CW\_\{0\}^\{\\top\}\)\\,W\_\{0\}\. The whitened \(SVD\-LLM\-style\) split and the eigenprojector split of Astra\([Liu et al\., 2026b](https://arxiv.org/html/2610.12728#bib.bib277)\)are instances; Astra’s paper form, a tail eigenprojector of the output covariance, has the same form withCCreplaced by the centred input covariance, which transforms in the same way\. Depending on the data only throughW0​C​W0⊤W\_\{0\}CW\_\{0\}^\{\\top\}is not enough, since PiSSA satisfies that vacuously and is notGLn\\mathrm\{GL\}\_\{n\}\-covariant\. Non\-conformal reparametrisations, such as the diagonal rescalings from norm\-gain folding or SmoothQuant\([Xiao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib508)\), change the CorDA and PiSSA splits\.

Three caveats apply\. An absolute dampingC\+λ​IC\+\\lambda Ireduces the covariance toO⁡\(n\)\\mathrm\{O\}\(n\), while CorDA’s trace\-relative damping keepsCO⁡\(n\)\\mathrm\{CO\}\(n\)\. The covariance is one\-sided\. No rank\-rrsplit that depends only on\(W0,C\)\(W\_\{0\},C\)can beGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\-covariant, since the stabiliser of a normal form contains an orthogonal group that fixes only full\-rank or zero splits, and whitening with the output second moment itself is degenerate \(every singular value of\(W0CW0⊤\)−1/2W0L\(W\_\{0\}CW\_\{0\}^\{\\top\}\)^\{\-1/2\}W\_\{0\}Lis11\)\. Descent through hidden\-layerGL\\mathrm\{GL\}gauges would need an output\-side statistic independent of\(W0,C\)\(W\_\{0\},C\), such as a backpropagated\-gradient second moment\. Training dynamics \(SGD, Adam\) remain non\-covariant\.

EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)is not a split\. It is Type I \(B0=0B\_\{0\}=0,W0W\_\{0\}not split\) and pointsrow⁡A0\\operatorname\{row\}A\_\{0\}at the top\-rrcentred principal subspace of the inputs\. At a fixed rank its imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}isGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\-covariant under[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), and only its pointingrow⁡A0\\operatorname\{row\}A\_\{0\}isCO⁡\(n\)\\mathrm\{CO\}\(n\)\-covariant\. With HF’s default rho=2\.0 the per\-layer rank is redistributed by explained\-variance ratios, which are onlyCO⁡\(n\)\\mathrm\{CO\}\(n\)\-invariant, so the image becomesGLm×CO⁡\(n\)\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{CO\}\(n\)\-covariant\.

Conjecture C\(whitened splits\)\.Whitened CorDA or PiSSA is more robust to function\-preserving input reparametrisations, and no worse otherwise\.

Rank is a cut, dimension a volume and covariance a group\. All three can be computed before training\. The next exposé turns from what a method can reach to how it moves there\.

## EXPOSÉ IIIFibres and metrics: dynamics

Expressivity lives on the image\. Training lives on the fibres, through a metric\. The image of a method fixes what it can reach, while its fibres and the optimizer’s metric fix how it gets there\. So two objects of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}can be isomorphic and still train differently, and methods that look different can train identically\.

Training happens upstairs, in the parameter spaceQQ, where each fibre ofρ\\rhocollects the parameters that give one weight\. The optimizer’s metric onQQ, pushed down to weight space, is what the model feels\. Every statement below names its optimizer\.

The exposé has four movements\. Training with ratesΛ\\Lambdamoves the weights byθ˙=−K​∇θ​L\\dot\{\\theta\}=\-K\\nabla\_\{\\theta\}L, with the cometricK=D​ρ​Λ​D​ρ⊤K=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}, which depends on where the parameters sit in their fibre \([Definition III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)to[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\. Gradient flow conserves LoRA’s chargeΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}, which sorts initialisations \([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)to[Proposition III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\); when the charge is isotropic, one scaleσ∗\\sigma^\{\*\}separates a LoRA\-FA\-like regime from balanced dynamics \([Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\)\. A 2\-torus of rescalings acts on LoRA’s knobs\(α,ηA,ηB,σA\)\(\\alpha,\\eta\_\{A\},\\eta\_\{B\},\\sigma\_\{A\}\)without changing training\. Under Adam withε=0\\varepsilon=0, no decay or clipping and one shared schedule, LoRA\+ with ratioλ\\lambdais plain LoRA withα\\alphamultiplied byλ\\lambda\([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), after[Schulman and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.12728#bib.bib401)\)\. Finally, SGD respects the subgroupO⁡\(r\)\\mathrm\{O\}\(r\)of the gauge and Adam only the signed permutations\. So from a zero\-BBstart the image depends only onrow⁡A0\\operatorname\{row\}A\_\{0\}, the SGD trajectory onA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}, and the Adam trajectory onA0A\_\{0\}up to signed row permutations \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.

#### One experiment first

Figure[8](https://arxiv.org/html/2610.12728#S3.F8)shows the cleanest consequence\. Train with Adam atε=0\\varepsilon=0fromB0=0B\_\{0\}=0, with no weight decay or clipping, one schedule for both parameter groups, and the same data order and dropout masks\. Then LoRA\+ with ratioλ\\lambda\(ηB=λ​ηA\\eta\_\{B\}=\\lambda\\eta\_\{A\}\) and plain LoRA withα\\alphamultiplied byλ\\lambda, at the sameηA\\eta\_\{A\}andA0A\_\{0\}, produce the same weights at every step \([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\. This LoRA/Adam statement is due to[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401), for the LoRA\+ of[Hayou et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib159);[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)says what is added here\.

#### The metric a method carries

A learning rate is a metric in disguise\. Gradient descent with ratesΛ\\Lambdamovesqqby−Λ​∇q\-\\Lambda\\nabla\_\{q\}, and pushed down byD​ρD\\rhothis is the weight gradient multiplied by a positive semidefinite operator\.

Definition III\.1\(trained method; cometric; three equivalences\)\. A*trained method*is a method together with optimizer data: a learning\-rate operatorΛ≻0\\Lambda\\succ 0\(per\-block rates\), an initial scale, and an update rule\. Gradient flowq˙=−Λ​∇q\(L∘ρ\)\\dot\{q\}=\-\\Lambda\\nabla\_\{q\}\(L\\circ\\rho\)moves the weights byθ˙=−Kq​∇θL,Kq=D​ρq​Λ​D​ρq⊤\.\\dot\{\\theta\}=\-K\_\{q\}\\nabla\_\{\\theta\}L,\\qquad K\_\{q\}=D\\rho\_\{q\}\\,\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\.We callKqK\_\{q\}theinduced cometric\. For a backward method with compressionctc\_\{t\}and anchorata\_\{t\}\([Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\),Kt=at​ctK\_\{t\}=a\_\{t\}c\_\{t\}\.Two methods are•*expressively equivalent*if their images agree;•*first\-order equivalent*if theirKq0K\_\{q\_\{0\}\}agree;•*dynamically equivalent*if they produce the sameθ\\theta\-trajectory for every loss\.

Cometrics compose the way derivatives do\.

Observation III\.2\(chain rule for cometrics\)\.Letσ:Q′→Q\\sigma:Q^\{\\prime\}\\to Qandρ:Q→X\\rho:Q\\to Xbe smooth, withX=ΘX=\\Thetaor a function or output space\. TrainQ′Q^\{\\prime\}by gradient flow with a constant rate operatorΛ′≻0\\Lambda^\{\\prime\}\\succ 0, and putKq′σ:=D​σq′​Λ′​D​σq′⊤K^\{\\sigma\}\_\{q^\{\\prime\}\}:=D\\sigma\_\{q^\{\\prime\}\}\\Lambda^\{\\prime\}D\\sigma\_\{q^\{\\prime\}\}^\{\\top\}\. ThenKq′ρ​σ=D​ρσ⁡\(q′\)​Kq′σ​D​ρσ⁡\(q′\)⊤,K^\{\\rho\\sigma\}\_\{q^\{\\prime\}\}=D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}\\,K^\{\\sigma\}\_\{q^\{\\prime\}\}\\,D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}^\{\\top\},and no metric onQQis used\.KKis a degenerate cometric \(positive semidefinite, of rank≤dimQ′\\leq\\dim Q^\{\\prime\}\); whenσ\\sigmais not injective,KσK^\{\\sigma\}is a field alongσ\\sigmaand fails to be a cometric onQQ\.For LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\),ρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBAwiths=α/rs=\\alpha/r\(α/r\\alpha/\\sqrt\{r\}for rsLoRA,[Kalajdzievski, 2023](https://arxiv.org/html/2610.12728#bib.bib204)\), block ratesΛ=diag⁡\(ηB​I,ηA​I\)\\Lambda=\\diag\(\\eta\_\{B\}I,\\eta\_\{A\}I\)and no LoRA dropout, under gradient flow \(equivalently, to first order in the step size for GD\),K\(B,A\)​\(G\)=s2​\(ηB​G​A⊤​A\+ηA​B​B⊤​G\)\.K\_\{\(B,A\)\}\(G\)=s^\{2\}\\big\(\\eta\_\{B\}\\,GA^\{\\top\}A\+\\eta\_\{A\}\\,BB^\{\\top\}G\\big\)\.Under Adam there is no state\-independent linear cometric;[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)and[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)treat Adam\.

NUMFinite\-difference Jacobians agree with the formula to6×10−96\\times 10^\{\-9\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSFor LoRA, the termηB​G​A⊤​A\\eta\_\{B\}\\,GA^\{\\top\}Alets the update reach all outputs while filtering its inputs throughrow⁡A\\operatorname\{row\}A, andηA​B​B⊤​G\\eta\_\{A\}\\,BB^\{\\top\}Gdoes the reverse\. AtB0=0B\_\{0\}=0only the first term survives, so the first\-step cometrics2​ηB​G​A0⊤​A0s^\{2\}\\eta\_\{B\}\\,GA\_\{0\}^\{\\top\}A\_\{0\}is fixed byA0A\_\{0\}\([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.

TakeXXto be the prefix space \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\. HF PrefixTuning with prefix\_projection=True \(the MLPσ\\sigmaof[Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\) and with prefix\_projection=False \(the P\-tuning v2 setting of[Liu et al\., 2021c](https://arxiv.org/html/2610.12728#bib.bib287), ignoring its classification head\) have the same prefix image whenever encoder\_hidden\_size≥\{\}\\geq\{\}num\_virtual\_tokens−1\{\}\-1\. Under gradient flow their cometrics on prefix space areD​σ​Λ′​D​σ⊤D\\sigma\\Lambda^\{\\prime\}D\\sigma^\{\\top\}andΛ\\Lambda, so the reparametrisation replaces the flat metric on prefix space by a point\-dependent one\. Prefix methods are extensions, so the target here is the prefix \(or function\) space rather thanΘ\\Theta\. This alone does not account for the instability that Li and Liang observed under AdamW\.

Backpropagation as a lens, the source of this chain rule, is due to[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\.

#### When a change of chart does not matter

A gauge transformation changes the parameters and keeps the weights\. It leaves training unchanged exactly when the cometrics agree, which for LoRA’s gaugeGLr\\mathrm\{GL\}\_\{r\}singles outO⁡\(r\)\\mathrm\{O\}\(r\)\.

Theorem III\.3\(gradient descent is natural exactly where cometrics agree; for LoRA’s linear gauge, exactly alongO⁡\(r\)\\mathrm\{O\}\(r\)\)\.\(a\) Letφ:Q→Q′\\varphi:Q\\to Q^\{\\prime\}be a diffeomorphism withρ′∘φ=ρ\\rho^\{\\prime\}\\circ\\varphi=\\rho, and letΛ,Λ′\\Lambda,\\Lambda^\{\\prime\}be the rate operators onQ,Q′Q,Q^\{\\prime\}\. Under gradient flow, the two methods induce the sameθ\\theta\-velocity atqqand atφ⁡\(q\)\\varphi\(q\)for every loss iffKqρ=Kφ⁡\(q\)ρ′K^\{\\rho\}\_\{q\}=K^\{\\rho^\{\\prime\}\}\_\{\\varphi\(q\)\}, that is,D​ρφ⁡\(q\)′​\(D​φq​Λ​D​φq⊤−Λ′\)​D​ρφ⁡\(q\)′⁣⊤=0\.D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}\\big\(D\\varphi\_\{q\}\\Lambda D\\varphi\_\{q\}^\{\\top\}\-\\Lambda^\{\\prime\}\\big\)D\\rho^\{\\prime\\top\}\_\{\\varphi\(q\)\}=0\.This holds wheneverD​φ​Λ​D​φ⊤=Λ′D\\varphi\\Lambda D\\varphi^\{\\top\}=\\Lambda^\{\\prime\}, which is an isometry whenΛ=Λ′=I\\Lambda=\\Lambda^\{\\prime\}=I\. That condition is necessary whenD​ρφ⁡\(q\)′D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}is injective, and fails to be necessary in general\. Every linear gauge of a linear parametrisation is natural; for instance, forρ⁡\(x,y\)=θ0\+x\+y\\rho\(x,y\)=\\theta\_\{0\}\+x\+ythe shear\(x,y\)↦\(2​x,y−x\)\(x,y\)\\mapsto\(2x,y\-x\)preservesρ\\rhoandKKwithout being orthogonal\. So is every pointwise\-orthogonal non\-linear LoRA gaugeφ​\(q\)=φg⁡\(q\)​\(q\)\\varphi\(q\)=\\varphi\_\{g\(q\)\}\(q\)withg⁡\(q\)∈O⁡\(r\)g\(q\)\\in\\mathrm\{O\}\(r\)\.\(b\) For LoRA’s linear gaugeφg​\(B,A\)=\(B​g−1,g​A\)\\varphi\_\{g\}\(B,A\)=\(Bg^\{\-1\},gA\)with block\-scalar rates, at points whereAAhas full row rank \(BBarbitrary, includingB=0B=0\) andr<nr<n, the criterion holds iffg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)\. The same holds symmetrically whereBBhas full column rank andr<mr<m\.

NUMAt a random point a non\-orthogonalgggives a cometric mismatch of about1818, an orthogonal one about10−1510^\{\-15\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSNo learner with a fixed Euclidean metric respects all invertible reparametrisations\. For LoRA’s linear gauge, plain gradient descent defines a family of training dynamics indexed by the cosetsO⁡\(r\)\\GLr\\mathrm\{O\}\(r\)\\backslash\\mathrm\{GL\}\_\{r\}, whichg↦g⊤​gg\\mapsto g^\{\\top\}gidentifies withSymr\+\\mathrm\{Sym\}^\{\+\}\_\{r\}\. Under gradient descent, LoRA\+’s rate ratioλ\\lambdacorresponds to the scalar gaugeg=λ1/4​Ig=\\lambda^\{1/4\}Itogether with an overall change of rate\. rsLoRA rescalesss, equivalently the learning rates and the initialisation; this is a move on the hyperparameter torus and involves no choice of gauge \([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\. Riemannian preconditioning and LoRA\-Pro remove the dependence on the representative \([Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\. Under AdamW, as used in practice, the relevant group is the groupBrB\_\{r\}of signed permutations \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.

TheGLr\\mathrm\{GL\}\_\{r\}symmetry ofW=B​AW=BAand the quotient geometry of fixed\-rank matrices are classical\([Mishra et al\., 2012](https://arxiv.org/html/2610.12728#bib.bib330)\)\.[Putterman et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib375)build learners on LoRA weights that respect it\.

Some published optimizers use instead a cometric that depends only on the weight, at least in an idealised form\.

Proposition III\.4\(cometrics that descend to the rank\-rrmanifold; known in part\)\.WriteU=col⁡BU=\\operatorname\{col\}BandV=row⁡AV=\\operatorname\{row\}A\. Statements \(a\), \(b\) and the first sentence of \(d\) are on the full\-rank locus; \(c\) needs onlyrank⁡A0=r\\rk A\_\{0\}=r, withBBarbitrary, includingB0=0B\_\{0\}=0\. All statements concern gradient flow, or \(S\)GD to first order, on the stated gradients, undamped and without Adam\. Cometrics include the learning rate\.\(a\) Scaled GD,δ​B=−η​∇B\(A​A⊤\)−1\\delta B=\-\\eta\\nabla\_\{B\}\(AA^\{\\top\}\)^\{\-1\}andδ​A=−η​\(B⊤​B\)−1​∇A\\delta A=\-\\eta\(B^\{\\top\}B\)^\{\-1\}\\nabla\_\{A\}, isGLr\\mathrm\{GL\}\_\{r\}\-equivariant with first\-order cometricK⁡\(G\)=η​s2​\(PU​G\+G​PV\)K\(G\)=\\eta s^\{2\}\(P\_\{U\}G\+GP\_\{V\}\)\.\(b\) LoRA\-Pro’s equivalent gradientg~=PU​G\+\(I−PU\)​G​PV\\tilde\{g\}=P\_\{U\}G\+\(I\-P\_\{U\}\)GP\_\{V\}is the Frobenius\-orthogonal projectionΠTW−W0​ℳr​\(G\)\\Pi\_\{T\_\{W\-W\_\{0\}\}\\mathcal\{M\}\_\{r\}\}\(G\)\. The flow of its SGD variant \(LoRA\-Pro Alg\. 1\) is the Riemannian gradient flow \([Background D\.24](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\) onW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\.\(c\) LoRA\-FA’s corrected gradients−2​∇B\(A0​A0⊤\)−1s^\{\-2\}\\nabla\_\{B\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}, fed to SGD withA0A\_\{0\}frozen, givesK⁡\(G\)=η​G​PVK\(G\)=\\eta\\,GP\_\{V\}withV=row⁡A0V=\\operatorname\{row\}A\_\{0\}, which depends onA0A\_\{0\}only throughV∈Gr⁡\(r,n\)V\\in\\Gr\(r,n\)\. Its original form, plain SGD onBB, hasK⁡\(G\)=s2​ηB​G​A0⊤​A0K\(G\)=s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}\.\(d\) The cometrics of \(a\) and \(b\) are functions ofWWon the full\-rank locus and define vector fields tangent toW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\. The field of \(c\) isG↦η​G​PVG\\mapsto\\eta GP\_\{V\}withV=row⁡A0V=\\operatorname\{row\}A\_\{0\}\. Its flow is the Frobenius gradient flow on the sliceW0\+\{X​A0\}=W0\+ℝm×n​PVW\_\{0\}\+\\\{XA\_\{0\}\\\}=W\_\{0\}\+\\mathbb\{R\}^\{m\\times n\}P\_\{V\}, of dimensionm​rmr, which containsW0W\_\{0\}and lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. The rank\-rrpart of the slice,NV=\{row\(W−W0\)=V\}N\_\{V\}=\\\{\\operatorname\{row\}\(W\-W\_\{0\}\)=V\\\}, is open and dense in it and is a submanifold ofW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\. OnNVN\_\{V\}the field is tangent toℳr\\mathcal\{M\}\_\{r\}and equalsη​G​Prow⁡\(W−W0\)\\eta GP\_\{\\operatorname\{row\}\(W\-W\_\{0\}\)\}, a function ofWW\. So \(c\) descends too, as a degenerate cometric of rankm​rmr\(less thanr⁡\(m\+n−r\)r\(m\+n\-r\)whenr<nr<n\) whose flow keepsrow⁡\(W−W0\)=row⁡A0\\operatorname\{row\}\(W\-W\_\{0\}\)=\\operatorname\{row\}A\_\{0\}fixed\. The cometric of plain GD depends on the representative; it is invariant only underO⁡\(r\)\\mathrm\{O\}\(r\)\.

The algorithms are known\. Scaled GD is due to[Tong et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib452)and[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546), and LoRA\-Pro to[Wang et al\. \(2024i\)](https://arxiv.org/html/2610.12728#bib.bib486)\. LoRA\-FA in its frozen\-AAform is[Zhang et al\. \(2023d\)](https://arxiv.org/html/2610.12728#bib.bib556); the corrected gradient of \(c\) comes from the 2026 revisions \(v2 and v3\) of that arXiv record and from the LoRA\-FA optimizer added to HF PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)in 2025\. TheGLr\\mathrm\{GL\}\_\{r\}\-invariance of the scaled\-GD metric and its descent to the quotientℳr\\mathcal\{M\}\_\{r\}are classical\([Mishra et al\., 2012](https://arxiv.org/html/2610.12728#bib.bib330);[Mishra and Sepulchre, 2016](https://arxiv.org/html/2610.12728#bib.bib331)\)\.

NUMGauge defects are about10−1510^\{\-15\}for \(a\), against5252for plain GD\. LoRA\-FA with AdamW fromA0A\_\{0\}and fromg​A0gA\_\{0\}differs byO⁡\(1\)O\(1\)after a few steps, against10−1510^\{\-15\}with SGD \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Practice

Each published method departs from these idealisations, and the departures change the symmetry \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.

- •The standard startB0=0B\_\{0\}=0lies off the full\-rank locus, where \(a\) and \(b\) are undefined without damping\.[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546)use\(B⊤​B\+δ​I\)−1\(B^\{\\top\}B\+\\delta I\)^\{\-1\}with an unspecifiedδ\>0\\delta\>0, which is onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant\. Their LLM experiments report both damped scaled GD and scaled AdamW, in which the preconditioner is applied before the Adam moments \(BrB\_\{r\}\)\. HF’s create\_riemannian\_optimizer damps as well\.
- •LoRA\-Pro’s experiments use its AdamW variant \(Alg\. 2, withm×nm\\times nmoments\),W˙=−ΠT​\(Adam⁡\(ΠT​G\)\)\\dot\{W\}=\-\\Pi\_\{T\}\(\\mathrm\{Adam\}\(\\Pi\_\{T\}G\)\)\. It descends toW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}without being a gradient flow, and its discrete update with the Sylvester\-optimal free matrix \([Background D\.33](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)\) isGLr\\mathrm\{GL\}\_\{r\}\-invariant only to first order\. The released code also damps both inverses \(pinv\(⋅\+10−8I\)\\operatorname\{pinv\}\(\\cdot\+10^\{\-8\}I\), soO⁡\(r\)\\mathrm\{O\}\(r\)\), replaces the step atB0=0B\_\{0\}=0by a LoRA\-FA\-type step, and runs with HF’s default global\-norm clipping\.
- •LoRA\-FA as published \(v3, Alg\. 1\) and HF’s create\_lorafa\_optimizer apply AdamW after the correction \(HF also damps with10−8​I10^\{\-8\}I\)\. The trajectory is then invariant only under signed row permutations ofA0A\_\{0\}\([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(b\)\), so it depends on the basis and scale ofA0A\_\{0\}as well as onrow⁡A0\\operatorname\{row\}A\_\{0\}\.

Remark\(2026: Riemannion\)\. Riemannion\([Bogachev et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib41)\)defines its optimizer onℳr\\mathcal\{M\}\_\{r\}directly\. Its variable is the pointX∈ℳrX\\in\\mathcal\{M\}\_\{r\}, stored as thin factors, and apart from weight decay each step uses onlyXX, the projected gradient and the momentum: the tangent projection of[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(b\), momentum carried along by the same projection, a Muon step\([Jordan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib201)\)that sets the nonzero singular values of the momentum to11and projects again, and a truncated SVD as retraction\. No Gram matrix is inverted, so nothing needs damping, and with weight decay off, theWW\-trajectory, momentum included, does not depend on howXXis factored\. The published Algorithm 4 writes its weight decay through the orthonormal factors, as a termγ​AL​BR⊤\\gamma A\_\{L\}B\_\{R\}^\{\\top\}, so this holds forγ=0\\gamma=0\. Its non\-linearity acts on a weight\-space matrix, while Adam’s acts coordinatewise on the factors, which is what cuts the gauge down toBrB\_\{r\}\([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.

WHAT THIS EXPLAINSUnder \(S\)GD each method is summarised by its induced weight\-space cometric \(Table[12](https://arxiv.org/html/2610.12728#S3.T12)\)\. LoRA, scaled GD and LoRA\-Pro share the imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}and differ only in cometric\. LoRA\-FA is the sub\-objectW0\+\{X​A0\}W\_\{0\}\+\\\{XA\_\{0\}\\\}\([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\. GaLore \(with a gradient\-SVD or a random projector\) and MeZO are full\-parameter methods, listed for comparison\. Flora proper compresses only its momentum, takes full\-space Adafactor steps, and has no cometric of this form \(Exposé[IV](https://arxiv.org/html/2610.12728#S4)\)\.

Table 12:Induced weight\-space cometrics under gradient flow or SGD \([Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\. The last three rows are full\-parameter methods, listed for comparison;PPandQQare GaLore’s left and right projectors,AtA\_\{t\}a random projector,zzthe random direction of MeZO\.“Transformation invariance” in the sense of Definition 1 of LoRA\-RITE\([Yen et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib533)\)means that theWW\-update is invariant under the gauge \(WW\-level descent\), which is weaker than parameter\-levelGLr\\mathrm\{GL\}\_\{r\}\-equivariance\. LoRA\-RITE’s algorithm satisfies the stronger property on the full\-rank locus \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.

#### Isometries and charges

Split LoRA’s gauge in two\. With per\-block scalar rates, the antisymmetric generators rotate the rank channels and preserve the rate metric, so the optimizer cannot feel them\. Along the symmetric ones, gradient flow conserves a quadratic charge \([Background D\.26](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4)\)\. For LoRA the charges form one matrixΦ\\Phi, whose vanishing is the balancedness of deep linear networks \([Background D\.28](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\)\.

Theorem III\.5\(Noether charges and isometries for cotangent signals; known,[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580)\)\.Let a Lie group act linearly onQ=ℝNQ=\\mathbb\{R\}^\{N\}withρ∘k=ρ\\rho\\circ k=\\rho, letΛ≻0\\Lambda\\succ 0be constant, and letγ⁡\(t\)∈Tρ⁡\(q⁡\(t\)\)∗​Θ\\gamma\(t\)\\in T^\{\*\}\_\{\\rho\(q\(t\)\)\}\\Thetabe any cotangent signal, gradient or not\. Consider the continuous\-time updatesq˙=−Λ​D​ρq⊤​γ​\(t\)\.\\dot\{q\}=\-\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\\gamma\(t\)\.\(i\) For every generatorξ\\xiwithΛ−1​ξ\\Lambda^\{\-1\}\\xisymmetric, the chargeJξ​\(q\)=12​⟨q,Λ−1​ξ​q⟩J\_\{\\xi\}\(q\)=\\tfrac\{1\}\{2\}\\langle q,\\Lambda^\{\-1\}\\xi q\\rangleis conserved\.\(ii\) For every generatorξ\\xiwithΛ−1​ξ\\Lambda^\{\-1\}\\xiantisymmetric,k=eτ​ξk=e^\{\\tau\\xi\}preserves⟨⋅,Λ−1⋅⟩\\langle\\cdot,\\Lambda^\{\-1\}\\cdot\\rangle, soKk​q=KqK\_\{kq\}=K\_\{q\}along the orbit; for the same cotangentγ\\gamma, theθ\\theta\-velocities atqqand atk​qkqagree\. Suppose moreover thatγ\\gammadepends onqqonly through\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\), asγ=∇θL\\gamma=\\nabla\_\{\\theta\}Lor an open\-loop signal does, and that solutions are unique \(γ\\gammameasurable inttand locally Lipschitz inθ\\theta,ρ\\rhoof classCloc1,1C^\{1,1\}\_\{\\mathrm\{loc\}\}\)\. Then theθ\\theta\-trajectories fromq0q\_\{0\}and fromk​q0kq\_\{0\}coincide\. Ifγ\\gammauses Euclidean norms onQQ, as global\-norm clipping does, the trajectory statement holds when, in addition,kkis a Euclidean isometry, and can fail otherwise\. So \(i\) and the velocity statement of \(ii\) hold for arbitraryγ\\gamma, while the trajectory statement needs a gradient\-like signal\.\(iii\) When𝔤\\mathfrak\{g\}is closed underξ↦Λ​ξ⊤​Λ−1\\xi\\mapsto\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}, every generator splits into a part of type \(i\) and a part of type \(ii\), a Cartan decomposition𝔤=𝔨⊕𝔭\\mathfrak\{g\}=\\mathfrak\{k\}\\oplus\\mathfrak\{p\}\([Background D\.27](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\)\. This holds for LoRA’s𝔤​𝔩r\\mathfrak\{gl\}\_\{r\}with per\-block scalar rates\. Without closure, there are generators of neither type\. With per\-rank\-channel ratesΛ⁡\(B,A\)=\(B​DB,DA​A\)\\Lambda\(B,A\)=\(BD\_\{B\},D\_\{A\}A\), whereDB=diag⁡\(ηB,j\)D\_\{B\}=\\diag\(\\eta\_\{B,j\}\)andDA=diag⁡\(ηA,j\)D\_\{A\}=\\diag\(\\eta\_\{A,j\}\), the generatorξX\\xi\_\{X\}is of type \(i\) iffX​DB−1XD\_\{B\}^\{\-1\}andDA−1​XD\_\{A\}^\{\-1\}Xare both symmetric, and of type \(ii\) iff both are antisymmetric\. An off\-diagonal pair\(i,j\)\(i,j\)contributes one charge and one isometry iffηB,i​ηA,i=ηB,j​ηA,j\\eta\_\{B,i\}\\eta\_\{A,i\}=\\eta\_\{B,j\}\\eta\_\{A,j\}, and𝔤\\mathfrak\{g\}is closed iff all the productsηB,j​ηA,j\\eta\_\{B,j\}\\eta\_\{A,j\}are equal\. In that case one gets aDD\-conjugate of𝔰​𝔬r⊕Symr\\mathfrak\{so\}\_\{r\}\\oplus\\mathrm\{Sym\}\_\{r\}whose isometries are not all Euclidean unless the rates are block\-scalar, so the clipping caveat of \(ii\) applies\. For pairwise distinct products, only the diagonalXXgive charges and no nonzeroXXgives an isometry\.

Part \(i\) is[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580), Prop\. 5\.1, including the table of symmetric and antisymmetric generators and symmetric learning rates\. Laws that hold for every loss and dataset are characterised by[Marcotte et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib318), and the principle goes back to[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100),[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)and[Kunin et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib230)\. New here are only the remark thatγ\\gammaneed not be a gradient, which is formally immediate \(along the realised trajectory,γ⁡\(t\)\\gamma\(t\)is the gradient of the time\-dependent linear loss⟨γ⁡\(t\),θ⟩\\langle\\gamma\(t\),\\theta\\rangle\), and the PEFT instances below\.

NUMWith a non\-gradient,qq\-dependentγ\\gammaon LoRA,Φ\\Phidrifts by1\.6×10−121\.6\\times 10^\{\-12\}\. With Euclidean clipping and a non\-Euclidean isometrykk, theθ\\theta\-trajectories fromq0q\_\{0\}andk​q0kq\_\{0\}differ by0\.060\.06\. With per\-channel ratesηB=\(1,1,1\)\\eta\_\{B\}=\(1,1,1\)andηA=\(1,1,2\)\\eta\_\{A\}=\(1,1,2\)atr=3r=3, closure fails, butX=E12\+E21X=E\_\{12\}\+E\_\{21\}still gives a conserved charge \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Instances

All instances assume gradient flow, or SGD without momentum, with per\-block rates\. AdamW and heavy\-ball momentum fall outside the theorem, since their buffers storeD​ρD\\rhoat past points\.

- •LoRA\.The generatorsξX​\(B,A\)=\(B​X,−X​A\)\\xi\_\{X\}\(B,A\)=\(BX,\-XA\)withΛ=diag⁡\(ηB,ηA\)\\Lambda=\\diag\(\\eta\_\{B\},\\eta\_\{A\}\)satisfy:Λ−1​ξX\\Lambda^\{\-1\}\\xi\_\{X\}is symmetric iffX=X⊤X=X^\{\\top\}\. So ther⁡\(r\+1\)/2r\(r\+1\)/2charges are the entries of Φ=B⊤​BηB−A​A⊤ηA,\\Phi=\\frac\{B^\{\\top\}B\}\{\\eta\_\{B\}\}\-\\frac\{AA^\{\\top\}\}\{\\eta\_\{A\}\},which is “balancedness”, the𝔭=Symr\\mathfrak\{p\}=\\mathrm\{Sym\}\_\{r\}half of𝔤​𝔩r=𝔰​𝔬r⊕Symr\\mathfrak\{gl\}\_\{r\}=\\mathfrak\{so\}\_\{r\}\\oplus\\mathrm\{Sym\}\_\{r\}\. The𝔰​𝔬r\\mathfrak\{so\}\_\{r\}generators act by Euclidean isometries, so \(ii\) holds for LoRA even under global\-norm clipping\.
- •DyLoRA\([Valipour et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib456)\)\. The gauge reduces to the torus, so onlydiag⁡Φ\\diag\\Phiis conserved\.
- •VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\)\. The gauge is\(b,d\)↦\(c​b,d/c\)\(b,d\)\\mapsto\(cb,d/c\), with charge‖b‖2/ηb−‖d‖2/ηd\\\|b\\\|^\{2\}/\\eta\_\{b\}\-\\\|d\\\|^\{2\}/\\eta\_\{d\}\. NOLA\([Koohpayegani et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib223)\)is analogous\.
- •AdaLoRA\([Zhang et al\., 2023g](https://arxiv.org/html/2610.12728#bib.bib559)\)\(P​Λ​QP\\Lambda Q\), without its orthogonality regulariser and between rank\-budget masking events\. Each torus edge gives a charge,‖pk‖2−λk2\\\|p\_\{k\}\\\|^\{2\}\-\\lambda\_\{k\}^\{2\}and‖qk‖2−λk2\\\|q\_\{k\}\\\|^\{2\}\-\\lambda\_\{k\}^\{2\}, at equal rates\. The regularised objective \(HF default orth\_reg\_weight=0\.5\) breaks these charges\.
- •HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)\. The rescalingui↦c​uiu\_\{i\}\\mapsto cu\_\{i\}gives a conserved‖ui‖2\\\|u\_\{i\}\\\|^\{2\}\.
- •ReLU bottleneck adapters\([Houlsby et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib175)\)\. Positive per\-unit rescalings give one charge per hidden unit\.
- •TT adapters\(for instance[Yang et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib529);[Anjum et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib11)\)\. One conserved matrix per bond\.
- •DoRA’s detached\-norm backward\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\)and global\-norm gradient clipping\.Both still have the formΛ​D​ρLoRA⊤​γ\\Lambda D\\rho\_\{\\mathrm\{LoRA\}\}^\{\\top\}\\gamma, withγ\\gammathe detached or clipped signal, soΦ\\Phiis still conserved\.

Under discrete GD with ratesηB,ηA\\eta\_\{B\},\\eta\_\{A\}, one step changesΦ\\Phiby exactly

ηB​A​G⊤​G​A⊤−ηA​B⊤​G​G⊤​B,\\eta\_\{B\}\\,AG^\{\\top\}GA^\{\\top\}\-\\eta\_\{A\}\\,B^\{\\top\}GG^\{\\top\}B,so the drift ofη​Φ\\eta\\Phi, or the drift relative toΦ\\Phi, isO⁡\(η2\)O\(\\eta^\{2\}\)per step\. Adam is not of the formΛ​D​ρ⊤​γ\\Lambda D\\rho^\{\\top\}\\gamma, so the drift ofΦ\\Phiunder Adam measures how far Adam departs from the gauge geometry; this is a prediction \(Exposé[X](https://arxiv.org/html/2610.12728#S10)\)\.

WHAT THIS EXPLAINSConservation needs only the shapeq˙=−Λ​D​ρ⊤​γ\\dot\{q\}=\-\\Lambda D\\rho^\{\\top\}\\gamma, so it holds for any loss and any data\. Withη\\etathe common scale of the rates, a GD step changesη​Φ\\eta\\PhibyO⁡\(η2\)O\(\\eta^\{2\}\)and an Adam step byO⁡\(η\)O\(\\eta\), as panel \(a\) of Figure[7](https://arxiv.org/html/2610.12728#S3.F7)shows\.

Analogy\(Noether; gauge\)\.*Noether*names the passage from a symmetry to a conserved quantity, here proved for gradient flow without a Lagrangian\.*Gauge*means the symmetry group of a parametrisation; no connection is involved\.

Figure 7:Noether hyperbolas and the crossover\([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1),[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\)\. \(a\) Scalar LoRA,w=w0\+s​b​aw=w\_\{0\}\+s\\,bawiths=1s=1, loss12​\(w−w⋆\)2\\tfrac\{1\}\{2\}\(w\-w^\{\\star\}\)^\{2\},w⋆−w0=1w^\{\\star\}\-w\_\{0\}=1and ratesηA=1\\eta\_\{A\}=1,ηB=2\\eta\_\{B\}=2\. Top: six gradient\-flow trajectories \(teal\) run from the filled dots along their own charge hyperbolasb2/ηB−a2/ηA=Φ0b^\{2\}/\\eta\_\{B\}\-a^\{2\}/\\eta\_\{A\}=\\Phi\_\{0\}\(dashed grey; the sliceΦ=0\\Phi=0dashed green\) to the fibres​a​b=w⋆−w0s\\,ab=w^\{\\star\}\-w\_\{0\}\(ochre\)\. From the zero\-BBstart\(1\.2,0\)\(1\.2,0\), Adam \(red\) leaves its hyperbola and lands at another point of the fibre\. Bottom: the driftmaxt≤10⁡\|Φ⁡\(t\)−Φ0\|\\max\_\{t\\leq 10\}\\lvert\\Phi\(t\)\-\\Phi\_\{0\}\\rvertfrom the same start against the step sizeη\\eta\(learning ratesη​ηA\\eta\\eta\_\{A\},η​ηB\\eta\\eta\_\{B\}\)\. A GD step changes the charge at orderη2\\eta^\{2\}, so over a fixed time the GD drift is proportional toη\\etaand vanishes in the flow limit\. An Adam step changes the charge at first order inη\\eta, and its drift stays of order one at every step size: Adam lies outside[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\([Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\)\. \(b\) Matrix LoRA withm=n=6m=n=6,r=2r=2, loss12​∥W−W⋆∥F2\\tfrac\{1\}\{2\}\\lVert W\-W^\{\\star\}\\rVert\_\{F\}^\{2\},B0=0B\_\{0\}=0andA0​A0⊤=IA\_\{0\}A\_\{0\}^\{\\top\}=Iexactly, so thatΦ=−κ​I\\Phi=\-\\kappa Iwithκ=1\\kappa=1and the crossover lies atσ∗=μ​\|κ\|/2=0\.5\\sigma^\{\*\}=\\mu\\lvert\\kappa\\rvert/2=0\.5\(dashed ochre\)\. Top: the closed\-form gainsgκ​\(σ2/μ2\)g\_\{\\kappa\}\(\\sigma^\{2\}/\\mu^\{2\}\)\(teal, theBB\-update\) andfκ​\(σ2/μ2\)f\_\{\\kappa\}\(\\sigma^\{2\}/\\mu^\{2\}\)\(purple, theAA\-update\) of[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), with the plateau\|κ\|\\lvert\\kappa\\rvertand the balanced asymptoteσ/μ\\sigma/\\mudotted\. The hollow circles, read off the factors along the integrated flow for both singular directions, lie on the curves\. Shading marks the one\-sided regime, the crossover, and the balanced regime \(grey\)\. Bottom: the singular valuesσ1​\(t\)\\sigma\_\{1\}\(t\)andσ2​\(t\)\\sigma\_\{2\}\(t\)ofΔ​W\\Delta W, with time running downward on a log scale\.σ1\\sigma\_\{1\}crossesσ∗\\sigma^\{\*\}and settles in the balanced regime;σ2\\sigma\_\{2\}stays belowσ∗\\sigma^\{\*\}, where theAA\-update supplies only7%7\\%of its growth rate, as in LoRA\-FA\. Integrators and tolerances are in §[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\.A conserved charge is fixed at time zero, so it labels initialisations\. Weight decay moves it on purpose, and its two conventions move it toward different targets\.

Corollary III\.6\(charge at initialisation; weight decay\)\.\(a\)Φ0=B0⊤​B0/ηB−A0​A0⊤/ηA\\Phi\_\{0\}=B\_\{0\}^\{\\top\}B\_\{0\}/\\eta\_\{B\}\-A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}is the value of the conserved charge, fixed by the initialisation and the rates\. SinceO⁡\(r\)\\mathrm\{O\}\(r\)acts onΦ\\Phiby conjugation and preservesθ\\theta\-trajectories \([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(ii\)\), only the spectrum ofΦ0\\Phi\_\{0\}labels aθ\\theta\-trajectory\.\(b\) LetGGbe any cotangent signal, with the scalessabsorbed into it\. Under gradient flow, if decay acts per unit time and identically on both factors,B˙=−ηB​G​A⊤−λ​B,A˙=−ηA​B⊤​G−λ​A,\\dot\{B\}=\-\\eta\_\{B\}GA^\{\\top\}\-\\lambda B,\\qquad\\dot\{A\}=\-\\eta\_\{A\}B^\{\\top\}G\-\\lambda A,thenΦ˙=−2​λ​Φ\\dot\{\\Phi\}=\-2\\lambda\\Phi\. With learning\-rate\-coupled decay \(PyTorch SGD weight\_decay; the decay term of PyTorch AdamW\),B˙=−ηB​\(G​A⊤\+λ​B\)\\dot\{B\}=\-\\eta\_\{B\}\(GA^\{\\top\}\+\\lambda B\)and likewise forAA, so thatΦ˙=−2​λ​\(B⊤​B−A​A⊤\)\\dot\{\\Phi\}=\-2\\lambda\(B^\{\\top\}B\-AA^\{\\top\}\)\. ThenΦ⁡\(t\)=e−2​η​λ​t​Φ​\(0\)\\Phi\(t\)=e^\{\-2\\eta\\lambda t\}\\Phi\(0\)holds whenηA=ηB=η\\eta\_\{A\}=\\eta\_\{B\}=\\eta\. ForηA≠ηB\\eta\_\{A\}\\neq\\eta\_\{B\}it fails in general, since−2​λ​\(B⊤​B−A​A⊤\)\-2\\lambda\(B^\{\\top\}B\-AA^\{\\top\}\)is not a function ofΦ\\Phi\. With an actual Adam\(W\) gradient term,Φ\\Phidrifts even without decay, at first order in the step size; Adam lies outside[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\. So neither law describes AdamW training\.

NUMTestingΦ⁡\(t\)=e−2​ηA​λ​t​Φ​\(0\)\\Phi\(t\)=e^\{\-2\\eta\_\{A\}\\lambda t\}\\Phi\(0\)atηB/ηA=4\\eta\_\{B\}/\\eta\_\{A\}=4with coupled decay gives a relative error of10\.710\.7; in the valid cases the error is that of the integrator, between10−810^\{\-8\}and2×10−42\\times 10^\{\-4\}\. One Adam sign step movesΦ\\Phiby an amount proportional to the learning rate, while an SGD step movesη​Φ\\eta\\Phiat second order \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINS

- •Zero\-product initialisations keep their charge under gradient flow, and that charge is maximally unbalanced \([Proposition III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\)\. The*relative*imbalance‖Φ‖/\(‖B‖2/ηB\+‖A‖2/ηA\)\\\|\\Phi\\\|/\(\\\|B\\\|^\{2\}/\\eta\_\{B\}\+\\\|A\\\|^\{2\}/\\eta\_\{A\}\)decays as the adapter grows \([Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\)\.
- •HF PiSSA always hasμ0=B0⊤​B0−A0​A0⊤=0\\mu\_\{0\}=B\_\{0\}^\{\\top\}B\_\{0\}\-A\_\{0\}A\_\{0\}^\{\\top\}=0\(unweighted\-balanced\), andΦ0=0\\Phi\_\{0\}=0only at equal rates\. So it starts on the slice that coupled decay targets, and off the per\-time\-decay slice whenηA≠ηB\\eta\_\{A\}\\neq\\eta\_\{B\}\.
- •Both decay conventions gauge\-fix at equilibrium, because each is aΛ\\Lambda\-gradient flow ofLLplus a fibre\-wise Kempf–Ness potential \([Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\)\. Per\-time decay is the flow ofL\+λ2​\(‖B‖2/ηB\+‖A‖2/ηA\)L\+\\tfrac\{\\lambda\}\{2\}\(\\\|B\\\|^\{2\}/\\eta\_\{B\}\+\\\|A\\\|^\{2\}/\\eta\_\{A\}\); it drivesΦ→0\\Phi\\to 0exponentially, onto the rate\-weighted sliceB⊤​B/ηB=A​A⊤/ηAB^\{\\top\}B/\\eta\_\{B\}=AA^\{\\top\}/\\eta\_\{A\}\. Coupled decay is the flow ofL\+λ2​\(‖B‖2\+‖A‖2\)L\+\\tfrac\{\\lambda\}\{2\}\(\\\|B\\\|^\{2\}\+\\\|A\\\|^\{2\}\); its equilibria lie on the unweighted balanced sliceμ=B⊤​B−A​A⊤=0\\mu=B^\{\\top\}B\-AA^\{\\top\}=0of[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(a\), with no closed\-form law forΦ\\PhiunlessηA=ηB\\eta\_\{A\}=\\eta\_\{B\}\. The two slices coincide iffηA=ηB\\eta\_\{A\}=\\eta\_\{B\}\. In particular, coupled decay also gauge\-fixes at LoRA\+ rates, ontoμ=0\\mu=0\.

NUMAtηB/ηA=4\\eta\_\{B\}/\\eta\_\{A\}=4, coupled decay converges toμ=0\\mu=0to within10−1210^\{\-12\}whileΦ\\Phistays of order11; per\-time decay converges toΦ=0\\Phi=0whileμ\\mustays of order11\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

The names Init\[A\] and Init\[B\], and the finding that these zero\-product starts train differently, are due to[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158)\. Exponential decay of such charges under weight decay is the rescale\-symmetry law of[Kunin et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib230)\.

#### The balanced slice

In Kempf–Ness theory, the least\-norm vectors on a closed orbit of a reductive group form one orbit of its compact part, and they are where the moment map vanishes \([Background D\.29](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1),[Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\)\. For LoRA the moment map isμ=B⊤​B−A​A⊤\\mu=B^\{\\top\}B\-AA^\{\\top\}, and the least\-norm representatives are the balanced ones\.

Proposition III\.7\(Kempf–Ness slice; Loewner floor; known,[Kempf and Ness \(1979\)](https://arxiv.org/html/2610.12728#bib.bib209);[Srebro et al\. \(2004\)](https://arxiv.org/html/2610.12728#bib.bib426)\)\.\(a\) On a full\-rankGLr\\mathrm\{GL\}\_\{r\}\-orbit, the critical points of‖B‖2\+‖A‖2\\\|B\\\|^\{2\}\+\\\|A\\\|^\{2\}are exactly the balanced representatives, those withμ=B⊤​B−A​A⊤=0\\mu=B^\{\\top\}B\-AA^\{\\top\}=0\. They form oneO⁡\(r\)\\mathrm\{O\}\(r\)\-orbit, they are the minima, and the minimum value is2​‖B​A‖∗2\\\|BA\\\|\_\{\*\}\.\(b\) Assume gradient flow with block\-scalar ratesηA,ηB\\eta\_\{A\},\\eta\_\{B\}, no weight decay, a loss that depends on\(B,A\)\(B,A\)only throughB​ABA, and any cotangent signal of the formΛ​D​ρ⊤​γ\\Lambda D\\rho^\{\\top\}\\gamma\(dropout, minibatches and clipping included\)\. FromB0=0B\_\{0\}=0,B⊤​B=ηBηA​\(A​A⊤−A0​A0⊤\)⪰0,B^\{\\top\}B=\\tfrac\{\\eta\_\{B\}\}\{\\eta\_\{A\}\}\\big\(AA^\{\\top\}\-A\_\{0\}A\_\{0\}^\{\\top\}\\big\)\\succeq 0,soA​A⊤⪰A0​A0⊤AA^\{\\top\}\\succeq A\_\{0\}A\_\{0\}^\{\\top\}for all time; theAA\-factor never shrinks below its initialisation in the Loewner order\.A​A⊤​\(t\)AA^\{\\top\}\(t\)need not be monotone intt; only this floor is guaranteed\. Discrete steps break the bound byO⁡\(η2\)O\(\\eta^\{2\}\)per step\. Under Adam it is not guaranteed and fails empirically\. With per\-time decayλ\\lambda, onlyA​A⊤⪰e−2​λ​t​A0​A0⊤AA^\{\\top\}\\succeq e^\{\-2\\lambda t\}A\_\{0\}A\_\{0\}^\{\\top\}holds\.

Part \(a\) is the theorem of[Kempf and Ness \(1979\)](https://arxiv.org/html/2610.12728#bib.bib209), in the real form of[Richardson and Slodowy \(1990\)](https://arxiv.org/html/2610.12728#bib.bib389); the nuclear\-norm identity is due to[Srebro et al\. \(2004\)](https://arxiv.org/html/2610.12728#bib.bib426)\. BaLoRA\([Castin et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib51)\)turns the slice into an algorithm, moving after every step to a product\-preserving representative withB⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\}diagonal\.

NUMOn55random orbits at\(m,n,r\)=\(7,5,3\)\(m,n,r\)=\(7,5,3\)the minimum equals2​‖B​A‖∗2\\\|BA\\\|\_\{\*\}to within10−1410^\{\-14\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSBalancedness is a precise notion: it is the Kempf–Ness slice of the gauge action\. An explicit penaltyλ⁡\(‖B‖2\+‖A‖2\)\\lambda\(\\\|B\\\|^\{2\}\+\\\|A\\\|^\{2\}\)is equivalent at its minimisers to a\(2​λ/s\)\(2\\lambda/s\)\-nuclear\-norm penalty onΔ​W=s​B​A\\Delta W=sBArestricted to rank≤r\\leq r\. SGD\-coupled decay with coefficientλ\\lambdais the flow ofL\+λ2​\(‖B‖2\+‖A‖2\)L\+\\tfrac\{\\lambda\}\{2\}\(\\\|B\\\|^\{2\}\+\\\|A\\\|^\{2\}\)and matches a\(λ/s\)\(\\lambda/s\)\-nuclear\-norm penalty \([Srebro et al\., 2004](https://arxiv.org/html/2610.12728#bib.bib426), whose argument also covers rank\-deficient factorisations\)\. In both cases the stationary points lie onμ=0\\mu=0\. With per\-time decay at unequal rates the relevant slice isΦ=0\\Phi=0\([Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\)\. Decoupled AdamW decay is not covered\.

The default start is as unbalanced as possible, and no start meets all three natural wishes\.

Proposition III\.8\(the balance–escape trilemma\)\.No LoRA initialisation is simultaneously\(a\)identity\-at\-init without a residual split \(B0​A0=0B\_\{0\}A\_\{0\}=0\),\(b\)balanced \(B0⊤​B0=c​A0​A0⊤B\_\{0\}^\{\\top\}B\_\{0\}=cA\_\{0\}A\_\{0\}^\{\\top\}for somec\>0c\>0: unweighted,c=1c=1, or rate\-weighted,Φ0=0\\Phi\_\{0\}=0withc=ηB/ηAc=\\eta\_\{B\}/\\eta\_\{A\}\), and\(c\)non\-stationary, meaning thatD​ρD\\rhoat\(B0,A0\)\(B\_\{0\},A\_\{0\}\)is not the zero map; equivalently,W˙​\(0\)≠0\\dot\{W\}\(0\)\\neq 0for some parameter velocity, under any first\-order optimizer\.Moreover, ifB0​A0=0B\_\{0\}A\_\{0\}=0then⟨B0⊤​B0,A0​A0⊤⟩F=0\\langle B\_\{0\}^\{\\top\}B\_\{0\},A\_\{0\}A\_\{0\}^\{\\top\}\\rangle\_\{F\}=0, so‖B0⊤​B0−c​A0​A0⊤‖F2=‖B0⊤​B0‖F2\+c2​‖A0​A0⊤‖F2,\\\|B\_\{0\}^\{\\top\}B\_\{0\}\-cA\_\{0\}A\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\}=\\\|B\_\{0\}^\{\\top\}B\_\{0\}\\\|\_\{F\}^\{2\}\+c^\{2\}\\\|A\_\{0\}A\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\},and zero\-product initialisations are maximally unbalanced\.

WHAT THIS EXPLAINSThe field splits into two families of initialisation\.

- •Type Ikeepsρ\\rhoand picksq0q\_\{0\}in the apex fibre\{BA=0\}\\\{BA=0\\\}: LoRA, EVA, Init\[B\]\. By the identity above these are maximally unbalanced\.
- •Type IIchangesρ\\rhoby an exact splitθ0=θres\+δ⁡\(q0\)\\theta\_\{0\}=\\theta\_\{\\mathrm\{res\}\}\+\\delta\(q\_\{0\}\): PiSSA, MiLoRA, OLoRA, LoRA\-GA\([Wang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib472)\), CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\)\. LoftQ\([Li et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib255)\)and QPiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)are approximate splits over a quantised residual, so they are not identity\-at\-init \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\.

The proposition forces every balanced, non\-stationary, identity\-at\-init LoRA initialisation to be Type II\. The converse fails: OLoRA \(B0⊤​B0=IB\_\{0\}^\{\\top\}B\_\{0\}=I,A0​A0⊤=Rr​Rr⊤A\_\{0\}A\_\{0\}^\{\\top\}=R\_\{r\}R\_\{r\}^\{\\top\}\) and CorDA are Type II and generally unbalanced\. A frozen residual is the price of any nonzeroB0​A0B\_\{0\}A\_\{0\}, and so in particular of balance, as in PiSSA and MiLoRA at equal rates\. By[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), Type II objects are different objects, incomparable with LoRA\.

#### Two regimes and a crossover

When the charge is a multiple of the identity, the LoRA flow has a closed form\. Below a scaleσ∗\\sigma^\{\*\}, one factor carries the motion, as in LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\); above it, the factors share the work, as in balanced deep linear networks\.

Theorem III\.9\(closed\-form isotropic\-charge LoRA dynamics; known,[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447)and[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328)\)\.Letρ⁡\(B,A\)=Wres\+s​B​A\\rho\(B,A\)=W\_\{\\mathrm\{res\}\}\+sBAbe trained by gradient flow with ratesηA,ηB\\eta\_\{A\},\\eta\_\{B\}, and suppose the charge is isotropic,Φ=−κ​Ir\\Phi=\-\\kappa I\_\{r\}\. Putμ=s​ηA​ηB\\mu=s\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}\(a rate, unrelated to the moment mapμ\\muof[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\), and on the locus whereZ=W−WresZ=W\-W\_\{\\mathrm\{res\}\}has rankrrwrite its compact SVDZ=U​Σ​V⊤Z=U\\Sigma V^\{\\top\}\. ThenW˙=−μ2​\[G​V​gκ​\(Σ2/μ2\)​V⊤\+U​fκ​\(Σ2/μ2\)​U⊤​G\],\\dot\{W\}=\-\\mu^\{2\}\\big\[G\\,Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}\+Uf\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)U^\{\\top\}G\\big\],wherefκ​\(x\)=x\+κ2/4−κ/2,gκ=fκ\+κ\.f\_\{\\kappa\}\(x\)=\\sqrt\{x\+\\kappa^\{2\}/4\}\-\\kappa/2,\\qquad g\_\{\\kappa\}=f\_\{\\kappa\}\+\\kappa\.The identity holds pointwise on\{Φ=−κ​I,rank⁡Z=r\}\\\{\\Phi=\-\\kappa I,\\ \\rk Z=r\\\}; gradient flow is used only to keepΦ\\Phifixed\. The two terms under the root are equal at the*crossover scale*σ∗=μ​\|κ\|/2\.\\sigma^\{\*\}=\\mu\\lvert\\kappa\\rvert/2\.•Forσ≪σ∗\\sigma\\ll\\sigma^\{\*\}withκ\>0\\kappa\>0,f→0f\\to 0andg→κg\\to\\kappa, soW˙≈−μ2​κ​G​PV\\dot\{W\}\\approx\-\\mu^\{2\}\\kappa\\,GP\_\{V\}\. These are LoRA\-FA dynamics\. The contribution ofAAtoW˙\\dot\{W\}isO⁡\(σ2/\(μ2​κ\)\)O\(\\sigma^\{2\}/\(\\mu^\{2\}\\kappa\)\), althoughAAitself still drifts slowly\. Forκ<0\\kappa<0the roles swap,W˙≈−μ2​\|κ\|​PU​G\\dot\{W\}\\approx\-\\mu^\{2\}\\lvert\\kappa\\rvert P\_\{U\}G, and onlyAAeffectively moves \(Init\[B\]\)\.•Forσ≫σ∗\\sigma\\gg\\sigma^\{\*\},f≈g≈σ/μf\\approx g\\approx\\sigma/\\mu, soW˙≈−μ⁡\[G​\(Z⊤​Z\)1/2\+\(Z​Z⊤\)1/2​G\]\.\\dot\{W\}\\approx\-\\mu\\big\[G\(Z^\{\\top\}Z\)^\{1/2\}\+\(ZZ^\{\\top\}\)^\{1/2\}G\\big\]\.These are the balanced dynamics of Arora, Cohen and Hazan\.•*Off the rank\-rrlocus\.*WriteW˙=−μ2​\(G​S\+T​G\)\\dot\{W\}=\-\\mu^\{2\}\(GS\+TG\)withS=A⊤​A/ηAS=A^\{\\top\}A/\\eta\_\{A\}andT=B​B⊤/ηBT=BB^\{\\top\}/\\eta\_\{B\}, so that on the rank\-rrlocusS=V​gκ​\(Σ2/μ2\)​V⊤S=Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}andT=U​fκ​\(Σ2/μ2\)​U⊤T=Uf\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)U^\{\\top\}\. Forκ≥0\\kappa\\geq 0,T=fκ​\(Z​Z⊤/μ2\)T=f\_\{\\kappa\}\(ZZ^\{\\top\}/\\mu^\{2\}\)still holds as a full matrix function, butSSgains the termκ⁡\(Prow⁡A−PV\)\\kappa\(P\_\{\\operatorname\{row\}A\}\-P\_\{V\}\); symmetrically, forκ<0\\kappa<0,TTgains\|κ\|​\(Pcol⁡B−PU\)\\lvert\\kappa\\rvert\(P\_\{\\operatorname\{col\}B\}\-P\_\{U\}\)\. At the default\-LoRA startZ=0Z=0\(B0=0B\_\{0\}=0\) this leavesS=κ​Prow⁡A0S=\\kappa P\_\{\\operatorname\{row\}A\_\{0\}\}andT=0T=0, soW˙=−μ2​κ​G​Prow⁡A0\\dot\{W\}=\-\\mu^\{2\}\\kappa\\,GP\_\{\\operatorname\{row\}A\_\{0\}\}, and theσ≪σ∗\\sigma\\ll\\sigma^\{\*\}regime extends continuously tot=0t=0\.

The closed form is due to[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447), who call this charge “λ\\lambda\-balanced”, and to[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328); see also[Dominé et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib97)and[Kunin et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib231)\. The balanced regime is that of[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)\. The PEFT reading is ours\. Panel \(b\) of Figure[7](https://arxiv.org/html/2610.12728#S3.F7)shows the closed\-form singular\-value dynamics and the crossover scaleσ∗\\sigma^\{\*\}\.

NUMThe formula matches direct integration to10−1210^\{\-12\}, and the pointwise identity holds to10−1510^\{\-15\}forκ∈\{−2,−0\.3,0,0\.3,2,10\}\\kappa\\in\\\{\-2,\-0\.3,0,0\.3,2,10\\\}and all shape cases \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSCall this the PEFT reading; it is scoped to gradient flow and SGD\.

- •*Default LoRA\.*WithB0=0B\_\{0\}=0andA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cI,κ=c/ηA\\kappa=c/\\eta\_\{A\}andσ∗=s​c2​ηB/ηA\\sigma^\{\*\}=\\tfrac\{sc\}\{2\}\\sqrt\{\\eta\_\{B\}/\\eta\_\{A\}\}; HF’s default givesc≈1/3c\\approx 1/3\.
- •*LoRA\+*\(λ=ηB/ηA\\lambda=\\eta\_\{B\}/\\eta\_\{A\}\) multipliesσ∗\\sigma^\{\*\}byλ\\sqrt\{\\lambda\}at fixedηA\\eta\_\{A\}, and its balanced\-regime rate is the geometric means​ηA​ηBs\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}\. LoRA\-FA is the endpointλ→∞\\lambda\\to\\inftyof the LoRA\+ family\.
- •*PiSSA at equal rates*hasκ=0\\kappa=0and is exactly balanced forever under gradient flow; SGD drifts byO⁡\(η2\)O\(\\eta^\{2\}\)per step, and Adam does not preserve it\.
- •*Exactness\.*The theorem is exact for EVA\- and LoRA\-GA\-type orthonormal frames\. For Kaiming initialisations it holds only approximately\. The eigenvalues ofA0​A0⊤A\_\{0\}A\_\{0\}^\{\\top\}lie nearc​\(1±r/n\)2c\(1\\pm\\sqrt\{r/n\}\)^\{2\}\([Marčenko and Pastur, 1967](https://arxiv.org/html/2610.12728#bib.bib317);[Auxiliary result B\.35](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\), so the crossover is a band of relative widthO⁡\(r/n\)O\(\\sqrt\{r/n\}\)rather than a single scale\. Atr=256r=256,n=4096n=4096, the eigenvalues of3​A0​A0⊤3A\_\{0\}A\_\{0\}^\{\\top\}span roughly0\.570\.57to1\.571\.57\.
- •*RandomAA\.*The one\-sided regime fits the finding of[Zhu et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib598)that a random, untrainedAAdoes nearly as well as a tuned one; the theorem says that this regime lasts whileΔ​W\\Delta Wstays well belowσ∗\\sigma^\{\*\}\.

PREDICTIONUnder SGD, singular values ofΔ​W\\Delta Wswitch from one\-sided to two\-sided growth as they crossσ∗\\sigma^\{\*\}\([Prediction X\.7](https://arxiv.org/html/2610.12728#S10.SS0.SSSx2.Px1)\)\.

#### The hyperparameter torus

Rescale blockiibycic\_\{i\}and dividessby the matching power of thecic\_\{i\}\. The method is the same object of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, and training is the same if every step rescales alike\. Adam ignores the size of a block’s gradient, so its rate must scale likecic\_\{i\}\. An SGD step scales with the gradient, which shrinks bycic\_\{i\}, so its rate must scale likeci2c\_\{i\}^\{2\}\.

Theorem III\.10\(the hyperparameter torus; the LoRA/Adam case is known,[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)\)\.Letρ⁡\(q\)=θb\+s​δ​\(q1,…,qk\)\\rho\(q\)=\\theta\_\{b\}\+s\\,\\delta\(q\_\{1\},\\dots,q\_\{k\}\)withδ\\deltamultihomogeneous of multidegree\(d1,…,dk\)\(d\_\{1\},\\dots,d\_\{k\}\)\. Train blockiifromqi​\(0\)=σi​uiq\_\{i\}\(0\)=\\sigma\_\{i\}u\_\{i\}with Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.12728#bib.bib216)\)\(β1∈\[0,1\)\\beta\_\{1\}\\in\[0,1\),β2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\), bias correction allowed\) at learning rateηi​\(t\)\\eta\_\{i\}\(t\), on the lossL∘ρL\\circ\\rhowith noqq\-dependent regulariser\. Assume no gradient clipping \(or per\-block clipping with thresholds rescaled as below\), common random numbers \(the same data order and dropout masks in both runs\), and exact arithmetic \(the result is bit\-exact in floating point when everycic\_\{i\}is a power of22\)\. Then the trajectoryt↦ρ⁡\(q⁡\(t\)\)t\\mapsto\\rho\(q\(t\)\)is invariant under the torus action ofc∈ℝ\>0kc\\in\\mathbb\{R\}\_\{\>0\}^\{k\}:\(s,ηi​\(t\),σi,εi,λi,Ci\)↦\(s​∏ici−di,ci​ηi​\(t\),ci​σi,εi/ci,λi/ci,Ci/ci\)\.\\big\(s,\\ \\eta\_\{i\}\(t\),\\ \\sigma\_\{i\},\\ \\varepsilon\_\{i\},\\ \\lambda\_\{i\},\\ C\_\{i\}\\big\)\\mapsto\\Big\(s\\prod\_\{i\}c\_\{i\}^\{\-d\_\{i\}\},\\ c\_\{i\}\\eta\_\{i\}\(t\),\\ c\_\{i\}\\sigma\_\{i\},\\ \\varepsilon\_\{i\}/c\_\{i\},\\ \\lambda\_\{i\}/c\_\{i\},\\ C\_\{i\}/c\_\{i\}\\Big\)\.Hereεi\\varepsilon\_\{i\}is a per\-block Adamε\\varepsilon\(with the convention0/0=00/0=0atε=0\\varepsilon=0\),λi\\lambda\_\{i\}a per\-block decoupled \(AdamW\) weight decay, andCiC\_\{i\}a per\-block clipping threshold\. Under SGD, with or without heavy\-ball or Nesterov momentum, the same holds withηi​\(t\)↦ci2​ηi​\(t\)\\eta\_\{i\}\(t\)\\mapsto c\_\{i\}^\{2\}\\eta\_\{i\}\(t\)and coupledL2L^\{2\}decayλi↦λi/ci2\\lambda\_\{i\}\\mapsto\\lambda\_\{i\}/c\_\{i\}^\{2\}\. In particular, any learning\-rate schedule shared by all blocks, warm\-up included, is preserved\.

The LoRA/Adam case, with theε\>0\\varepsilon\>0extension and the two effective hyperparameters, is due to[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)\(*LoRA Without Regret*\)\. New here are arbitrary multihomogeneousδ\\deltaand the SGD and heavy\-ballc2c^\{2\}law\.

NUMTrajectory differences are exactly0\.00\.0for power\-of\-twoccand about10−1510^\{\-15\}otherwise, including a warm\-up schedule and rescaledε\\varepsilonand decay\. Withε\\varepsilonor decay held fixed, or with global\-norm clipping, they are10−310^\{\-3\}toO⁡\(1\)O\(1\)\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSOnly the torus orbit of the hyperparameters reaches the weights\. LoRA is the casek=2k=2,d=\(1,1\)d=\(1,1\);δ\\deltamay be any multilinear update, such as that of VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\)in its two scaling vectors or that of LoHa\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\)in its four factors\.

Corollary III\.11\(LoRA\+ isα\\alphain disguise; rsLoRA is LoRA; \(a\) and the LoRA\+ reading are known,[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)\)\.For linear LoRA modules \(k=2k=2,d=\(1,1\)d=\(1,1\),σB=0\\sigma\_\{B\}=0\), under the hypotheses of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7):\(a\)*Effective hyperparameters\.*Under Adam the torus invariants of\(s,ηA,ηB,σA\)\(s,\\eta\_\{A\},\\eta\_\{B\},\\sigma\_\{A\}\)are\(s​ηA​ηB,σA/ηA\)\(s\\eta\_\{A\}\\eta\_\{B\},\\ \\sigma\_\{A\}/\\eta\_\{A\}\); the quantitiesα⋅initA⋅LRB\\alpha\\cdot\\mathrm\{init\}\_\{A\}\\cdot\\mathrm\{LR\}\_\{B\}andinitA/LRA\\mathrm\{init\}\_\{A\}/\\mathrm\{LR\}\_\{A\}of Schulman et al\. are equivalent generators\. Under SGD the invariants are\(s2​ηA​ηB,σA2/ηA\)\(s^\{2\}\\eta\_\{A\}\\eta\_\{B\},\\ \\sigma\_\{A\}^\{2\}/\\eta\_\{A\}\)\.\(b\)*LoRA\+\.*Under Adam withε=0\\varepsilon=0, no weight decay, no gradient clipping and a learning\-rate schedule shared by both groups, LoRA\+ with ratioλ\\lambdaat scalessproduces*exactly*the trajectory of plain LoRA at scaleλ​s\\lambda s\(sameηA\\eta\_\{A\}, sameA0A\_\{0\}\)\. Forε\>0\\varepsilon\>0the equivalence needs a per\-blockεB′=λ​ε\\varepsilon\_\{B\}^\{\\prime\}=\\lambda\\varepsilon, and with AdamW a per\-group decayλ⋅wd\\lambda\\cdot\\mathrm\{wd\}onBB\. Under SGD it holds at scaleλ​s\\sqrt\{\\lambda\}\\,s, or equivalently with plain LoRA at the single rateηA​ηB\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}andA0↦λ1/4​A0A\_\{0\}\\mapsto\\lambda^\{1/4\}A\_\{0\}\.\(c\)*rsLoRA\.*ρrs​\(B,A\)=ρLoRA​\(r​B,A\)\\rho\_\{\\mathrm\{rs\}\}\(B,A\)=\\rho\_\{\\mathrm\{LoRA\}\}\(\\sqrt\{r\}\\,B,A\)is an isomorphism of objects\. Up to the hyperparameter torus, rsLoRA can be represented as LoRA at the sameα\\alphawithηB\\eta\_\{B\}multiplied byr\\sqrt\{r\}under Adam \(byrrunder SGD\), or equally as LoRA withα\\alphamultiplied byr\\sqrt\{r\}at the same rates\. The reading “same object, different metric” is a choice of representative within the orbit\. Part \(c\) concerns a fixed rank; it does not address rsLoRA’s rank\-stability prescription, howα\\alphashould scale withrr\.

LoRA\+ is due to[Hayou et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib159), rsLoRA to[Kalajdzievski \(2023\)](https://arxiv.org/html/2610.12728#bib.bib204), and \(a\) with the LoRA\+ reading to[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)\. On the empirical side,[Lee et al\. \(2026\)](https://arxiv.org/html/2610.12728#bib.bib240)find that nine LoRA variants reach similar peak performance once the learning rate is tuned\.

NUMThe maximum trajectory difference is exactly0\.00\.0atε=0\\varepsilon=0for power\-of\-twoλ\\lambdaand about10−1410^\{\-14\}otherwise, against0\.930\.93for the unrescaled control\. It is5×10−95\\times 10^\{\-9\}at a sharedε=10−8\\varepsilon=10^\{\-8\}, and0\.60\.6with global\-norm clipping at0\.50\.5\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSLoRA’s four knobs are coordinates on a 4\-dimensional space acted on by a 2\-torus, and only the orbit matters\. LoRA\+, rsLoRA andα\\alpha\-tuning therefore collapse into one family\. The slogan depends on the chart\. HoldingηA\\eta\_\{A\}fixed, LoRA\+ is “α\\alphamultiplied byλ\\lambda”; holdingηB\\eta\_\{B\}fixed, it is plain LoRA withηA\\eta\_\{A\}divided byλ\\lambda, a slowerAA\-timescale\. In particular, Adam does not make the LoRA\+ ratio non\-absorbable\.

PREDICTIONAny gain of LoRA\+ over LoRA withα\\alphare\-tuned by the factorλ\\lambdamust come from the ingredients that break the extended torus: global\-norm clipping \(on by default in the HF Trainer, max\_grad\_norm=1\.0\),ε\\varepsilonor weight decay held equal across parameter groups \(as in HF PEFT’s LoRA\+ optimizer\), and learning\-rate schedules that differ between groups\. Shared schedules and warm\-up preserve the torus\. HF’s LoRA\+ optimizer treats embedding adapters and 1\-D parameters separately, so the statement covers linear modules only \([Prediction X\.1](https://arxiv.org/html/2610.12728#S10.SS0.SSSx2)\)\.

Figure 8:LoRA\+ isα\\alphain disguise\([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\. A seeded toy LoRA regression withB0=0B\_\{0\}=0is trained by Adam in float64, once as LoRA\+ with ratioλ=16\\lambda=16\(scaless,ηB=16​ηA\\eta\_\{B\}=16\\,\\eta\_\{A\}\) and once as plain LoRA withα\\alphamultiplied by1616\(scale16​s16s,ηB=ηA\\eta\_\{B\}=\\eta\_\{A\}\), at the sameηA\\eta\_\{A\}andA0A\_\{0\}\. \(a\) Atε=0\\varepsilon=0, without weight decay or clipping, the two loss curves coincide; the control, plain LoRA at the uncompensated scaless, ends far above them\. \(b\) The weight gap∥Wt−Wt′∥F\\lVert W\_\{t\}\-W^\{\\prime\}\_\{t\}\\rVert\_\{F\}between the two runs at every step, on a log scale above an exact\-zero floor\. Atε=0\\varepsilon=0the runs are bit\-identical at every step: the torus element relating them,c=\(1,1/16\)c=\(1,1/16\), is a power of two and rescales floating\-point numbers exactly\. Each red curve switches on one ingredient with a single value shared by both parameter groups: Adam’sε\\varepsilon, decoupled weight decay, or global\-norm clipping\. Each breaks the hyperparameter torus; the control is dotted\. Rescaling the same ingredients per block, as[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)prescribes, restores the exact zero \(green floor\)\. The values used are in §[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\.
#### What the optimizer can see

An optimizer is equivariant under a gauge elementggif starting from\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)gives the transported trajectory\. The Euclidean metric of SGD respects rotations of the rank channels\. Adam, acting coordinatewise, respects only the signed permutations\.

Remark\(2026: privacy noise\)\. Privacy noise fits the same pattern\. DP\-SGD\([Abadi et al\., 2016](https://arxiv.org/html/2610.12728#bib.bib1)\)on the factors adds isotropic Gaussian noiseξ\\xito the clipped factor gradients, andξ​g−1\\xi g^\{\-1\}has the law ofξ\\xionly for orthogonalgg, so it respectsO⁡\(r\)\\mathrm\{O\}\(r\)and no larger subgroup\. PRISM\([Wang and Zhang, 2026](https://arxiv.org/html/2610.12728#bib.bib475)\)clips and noises the projection of each gradient onto the tangent space ofℳr\\mathcal\{M\}\_\{r\}atB​ABA, which depends only oncol⁡B\\operatorname\{col\}Bandrow⁡A\\operatorname\{row\}A\. For full\-rank factors the privatised gradient then has the same law for every factorisation ofB​ABA\. Its Algorithm 1 then applies dampedr×rr\\times rpreconditioners \(its Eqs\. 24–26\), which the paper also calls gauge invariant\. They respect onlyO⁡\(r\)\\mathrm\{O\}\(r\)\. Under\(B,A\)↦\(c​B,A/c\)\(B,A\)\\mapsto\(cB,A/c\)the preconditioned factor steps do not change, soδ​B​A\\delta B\\,Ascales by1/c1/candB​δ​AB\\,\\delta Abycc\. The released code returns to balanced factors,B⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\}, after every step, where the remaining gauge isO⁡\(r\)\\mathrm\{O\}\(r\)\.

Theorem III\.12\(the optimizer gauge hierarchy\)\.On LoRA’s full\-rank locus, with hyperparameters shared by all rank channels, the largest subgroup of the gaugeGLr\\mathrm\{GL\}\_\{r\}under which an update rule is equivariant, at the level of parameters, is as follows\.\(a\) SGD and GD with any block rates, with or without heavy\-ball momentum:O⁡\(r\)\\mathrm\{O\}\(r\)\.\(b\) Adam and AdamW at fixed learning rates \(β1,β2∈\[0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\[0,1\), anyε≥0\\varepsilon\\geq 0, any weight decay\): the signed permutationsBrB\_\{r\}, of order2r​r\!2^\{r\}r\!\. A positive diagonalg=diag⁡\(d\)g=\\diag\(d\)is a symmetry only*jointly*with per\-rank\-channel hyperparameters, and only without global\-norm clipping\. Atε=0\\varepsilon=0and without weight decay it suffices to rescale the rates,ηB,j↦ηB,j/dj\\eta\_\{B,j\}\\mapsto\\eta\_\{B,j\}/d\_\{j\}andηA,j↦dj​ηA,j\\eta\_\{A,j\}\\mapsto d\_\{j\}\\eta\_\{A,j\}\. Withε\>0\\varepsilon\>0one must also give each channel its ownεB,j=dj​ε\\varepsilon\_\{B,j\}=d\_\{j\}\\varepsilonandεA,j=ε/dj\\varepsilon\_\{A,j\}=\\varepsilon/d\_\{j\}\. With decoupled decay multiplied by the learning rate \(PyTorch AdamW\) one must also setλB,j=dj​λ\\lambda\_\{B,j\}=d\_\{j\}\\lambdaandλA,j=λ/dj\\lambda\_\{A,j\}=\\lambda/d\_\{j\}; under the convention of[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2610.12728#bib.bib296), where decay is scaled only by the schedule multiplier,λ\\lambdaneeds no change\.\(c\) Undamped scaled GD \(δ=0\\delta=0; the Riemannian preconditioner of Tong, Ma and Chi and of Zhang and Pilanci, with or without heavy\-ball momentum\): all ofGLr\\mathrm\{GL\}\_\{r\}\. TheWW\-trajectory from gauge\-related initial states \(with zero or correspondingly transformed momentum\) is independent of the representative; a step may drop rank, so the iteration maps intoW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\.

Scaled GD is due to[Tong et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib452)and[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546)\. LoRA\-RITE\([Yen et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib533)\)makes gauge invariance a design requirement for LoRA optimizers and reports that it pays\.

##### Published variants

Damped scaled GD \(δ\>0\\delta\>0\) is onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant\. Wrapped in AdamW, as in Zhang and Pilanci’s scaled AdamW \(their practical variant\) and in HF’s create\_riemannian\_optimizer, it isBrB\_\{r\}\-equivariant by \(b\)\. The first\-orderWW\-level dynamics of LoRA\-Pro\([Wang et al\., 2024i](https://arxiv.org/html/2610.12728#bib.bib486)\)descend \([Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(b\)\), and so do those of its variant with free matrixX=0X=0; its published parameter update with the Sylvester\-optimalXXis onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant, and its discreteWW\-step depends on the representative atO⁡\(η2\)O\(\\eta^\{2\}\)\. LoRA\-RITE states onlyWW\-level invariance \(its Definition 1\)\. Its update \(unmagnified gradients, transported matrix moments, then multiplication by\(RB⊤\)\+\(R\_\{B\}^\{\\top\}\)^\{\+\}\) isGLr\\mathrm\{GL\}\_\{r\}\-equivariant at the parameter level on the full\-rank locus: under a gauge change its internal quantities change only by orthogonal matrices, and the final multiplication supplies the gauge element \(Appendix[C\.35](https://arxiv.org/html/2610.12728#A3.SS35), which follows its Algorithm 1 line by line; the resulting step is Eq\. 15 of[Yen et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib533)\)\. LoRA\-RITE is therefore an Adam\-type optimizer in class \(c\)\.

NUMFor Adam, the defects are10−1510^\{\-15\}for signed permutations,11\.311\.3for a positive diagonal,6\.66\.6for a rotation, and5×10−165\\times 10^\{\-16\}for the diagonal with jointly rescaled per\-channel rates atε=0\\varepsilon=0\. Withε=10−8\\varepsilon=10^\{\-8\}left unrescaled the joint torus fails by about2%2\\%\. LoRA\-RITE \(its Algorithm 1\) from\(B,A\)\(B,A\)and from\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)with a genericggagrees to3×10−143\\times 10^\{\-14\}over 30 steps \(referee replication\)\. LoRA\-Pro with the SylvesterXXhas a parameter\-level defect of1\.21\.2under a non\-orthogonalgg, against2×10−152\\times 10^\{\-15\}withX=0X=0or with an orthogonalgg\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSThere is a hierarchyBr⊂O⁡\(r\)⊂GLrB\_\{r\}\\subset\\mathrm\{O\}\(r\)\\subset\\mathrm\{GL\}\_\{r\}of optimizer geometries, strict forr≥2r\\geq 2\.GLr\\mathrm\{GL\}\_\{r\}\-equivariant rules, started from gauge\-related states, define an optimizer on the*model*rather than on a chart\. More generally any rule that maps gauge orbits to gauge orbits does \(for instance a step followed by a canonical refactorisation ofWW\), whileBrB\_\{r\}\- andO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant rules in general do not\. Plain Adam, the default, sees a finite shadow of the gauge\. LoRA\+ and rsLoRA act on the hyperparameter torus \([Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\) and leave the gauge of a fixed method alone; balanced or orthonormal initialisations choose a point in the fibre\. The positive torus is a symmetry of the*hyperparameters*; Adam at fixed rates does not have it\.

Figure 9:Update rules that see the gauge\([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\. \(a\) At a fixed seeded LoRA state\(B,A\)\(B,A\)with loss gradientGG, each cell is the relative gauge defect∥δ​W​\(g\)−δ​W​\(I\)∥F/∥δ​W​\(I\)∥F\\lVert\\delta W\(g\)\-\\delta W\(I\)\\rVert\_\{F\}/\\lVert\\delta W\(I\)\\rVert\_\{F\}of one update rule: the change in its first\-order weight updateδ​W=δ​B​A\+B​δ​A\\delta W=\\delta B\\,A\+B\\,\\delta Awhen the factors are replaced by the gauge\-equivalent pair\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)\. The columns are a rotation, a stretch, a signed permutation and a genericg∈GL2g\\in\\mathrm\{GL\}\_\{2\}; under each name is the smallest of the groupsB2=O⁡\(2\)∩Mon2B\_\{2\}=\\mathrm\{O\}\(2\)\\cap\\mathrm\{Mon\}\_\{2\},O⁡\(2\)\\mathrm\{O\}\(2\),Mon2\\mathrm\{Mon\}\_\{2\}\(monomial matrices\) andGL2\\mathrm\{GL\}\_\{2\}that contains it\. Green cells vanish up to float64 round\-off, and they occur exactly wheregglies in the rule’s gauge group \(right column\):O⁡\(2\)\\mathrm\{O\}\(2\)for SGD and for damped scaled GD,B2B\_\{2\}for Adam,Mon2\\mathrm\{Mon\}\_\{2\}for Adam whose per\-rank\-channel rates are rescaled jointly with a stretch, and all ofGL2\\mathrm\{GL\}\_\{2\}for undamped scaled GD and for LoRA\-Pro with the Sylvester\-optimal free matrix, whose weight updates depend only onWW\. Red cells are genuine defects\. \(b\) Three LoRA modules\(Bi,Ai\)\(B\_\{i\},A\_\{i\}\)with weightswiw\_\{i\}are merged after module 2 alone is re\-gauged to\(c​B2,A2/c\)\(cB\_\{2\},A\_\{2\}/c\)\. Task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\),∑iwi​Bi​Ai\\sum\_\{i\}w\_\{i\}B\_\{i\}A\_\{i\}, does not move\. The LoraHub merge\([Huang et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib180)\),\(∑iwi​Bi\)​\(∑jwj​Aj\)\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\)\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\), does, and the open circles, the closed form of[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4), match the recomputed change\. The settings are in §[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\.

#### What a zero\-BBstart chooses

Default LoRA starts atB0=0B\_\{0\}=0with a random full\-rankA0A\_\{0\}\. The image remembers onlyrow⁡A0\\operatorname\{row\}A\_\{0\}\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\. Training remembers what the optimizer’s gauge group cannot move\.

Corollary III\.13\(what a zero\-BBinitialisation really chooses\)\.Atq0=\(0,A0\)q\_\{0\}=\(0,A\_\{0\}\)withrank⁡A0=r\\rk A\_\{0\}=r, with hyperparameters shared by the rank channels and the same data order and dropout masks, two choices ofA0A\_\{0\}give the sameθ\\theta\-trajectory whenever the optimizer’s gauge group relates them\. The resulting invariant ofA0A\_\{0\}is:•row⁡A0∈Gr⁡\(r,n\)\\operatorname\{row\}A\_\{0\}\\in\\Gr\(r,n\)for an update rule that isGLr\\mathrm\{GL\}\_\{r\}\-equivariant at every point of its trajectory, includingB=0B=0and rank\-deficientBB, run without global\-norm clipping\. An example isδ​B=−η​∇B\(A​A⊤\)−1\\delta B=\-\\eta\\nabla\_\{B\}\(AA^\{\\top\}\)^\{\-1\},δ​A=−η⁡\(A​A⊤\)​∇A\\delta A=\-\\eta\(AA^\{\\top\}\)\\nabla\_\{A\}, which needs onlyrank⁡A=r\\rk A=r;•the PSD matrixA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}for SGD, which is exactly the initial preconditioner inW˙​\(0\)=−s2​ηB​G​A0⊤​A0;\\dot\{W\}\(0\)=\-s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\};•A0A\_\{0\}up to signed permutation of its rows for Adam\.The undamped scaled\-GD optimizers of[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(c\) are not defined atB=0B=0, since they need\(B⊤​B\)−1\(B^\{\\top\}B\)^\{\-1\}\. LoRA\-RITE uses pseudo\-inverses and so is defined there, but itsGLr\\mathrm\{GL\}\_\{r\}\-equivariance is established only on the full\-rank locus\. Their damped forms are onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant, so in practice their invariant is aδ\\delta\-dependent function ofA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}, and their Adam\-wrapped forms \(Riemannian\-preconditioned AdamW, HF’s LoRA\-FA optimizer\) have the invariantA0A\_\{0\}moduloBrB\_\{r\}\. Conversely, different invariants give different dynamics, for SGD and for Adam atε=0\\varepsilon=0\.Random\-AALoRA and EVA are different points of theB=0B=0stratum\{\(0,A\):rank⁡A=r\}\\\{\(0,A\):\\rk A=r\\\}of the apex fibreρ−1​\(θ0\)=\{\(B,A\):B​A=0\}\\rho^\{\-1\}\(\\theta\_\{0\}\)=\\\{\(B,A\):BA=0\\\}; Init\[B\] lies in another stratum of the same fibre\. LoRA\-FA is a different*object*, the sub\-objectFA⁡\(A0\)↪LoRA\(0,A0\)\\mathrm\{FA\}\(A\_\{0\}\)\\hookrightarrow\\mathrm\{LoRA\}\_\{\(0,A\_\{0\}\)\}\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\)\. More precisely, EVA is a point of the apex fibre ofLoRArl\\mathrm\{LoRA\}\_\{r\_\{l\}\}at its per\-layer redistributed rank and adjustedα\\alpha, with orthonormal rows only when whiten=False\.

Appendix[C\.36](https://arxiv.org/html/2610.12728#A3.SS36)proves the converse \(different invariants give different dynamics\) by comparing first steps\.[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158)showed that the two zero\-product starts, randomAAwithB=0B=0and randomBBwithA=0A=0, train differently; here they lie in different strata of one apex fibre, with charges of opposite sign\. EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)and random\-AALoRA share a stratum\.

NUMSGD fromA0A\_\{0\}and fromQ​A0QA\_\{0\}\(withQQorthogonal\) agree to10−1510^\{\-15\}, and fromA0A\_\{0\}andD​A0DA\_\{0\}\(withDDdiagonal\) differ by0\.810\.81\. Adam fromA0A\_\{0\}and fromP​A0PA\_\{0\}\(withPPa signed permutation\) agree to10−1610^\{\-16\}, and fromA0A\_\{0\}andQ​A0QA\_\{0\}differ by0\.980\.98\. The example rule fromA0A\_\{0\}and fromg​A0gA\_\{0\}agrees to3×10−153\\times 10^\{\-15\}\. LoRA\-RITE’s Algorithm 1, run as written with its pseudo\-inverses fromB0=0B\_\{0\}=0, gives the same weights fromA0A\_\{0\}and fromg​A0gA\_\{0\}for a genericggto10−1310^\{\-13\}, and weights that differ byO⁡\(1\)O\(1\)for a different row space \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSThe strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)are the expressive quotient\. The dynamics see a finer quotient, and how fine depends on the optimizer \(Table[13](https://arxiv.org/html/2610.12728#S3.T13)\)\. Undamped scaled GD is undefined atB=0B=0, so scaled\-GD optimizers enter in damped \(O⁡\(r\)\\mathrm\{O\}\(r\)\) or AdamW\-wrapped \(BrB\_\{r\}\) form\.

Table 13:What an update rule reads offA0A\_\{0\}from a zero\-BBstart \([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.Two problems stay open\. One is the apex fibre moduloBrB\_\{r\}for Adam, stratified byΦ0\\Phi\_\{0\}, degenerate pointings included\. The other is an Adam analogue of[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), a conserved or monotone quantity for sign\-like updates\.

Exposé[IV](https://arxiv.org/html/2610.12728#S4)reads the same cometric from the backward side of the lens, where methods that compress gradients have cometrics of their own\.

## EXPOSÉ IVThe lens: tangent and cotangent methods

Backpropagation pairs every map with its reverse derivative, so a method can shape what the model expresses on the forward side or compress the gradient on the backward side\. For linear methods and optimizers that read only their gradients \(no weight decay, no clipping of the full gradient\), the two sides give the same weights, and GaLore becomes one\-sided LoRA with a moving frame and carried optimizer state\.

#### Two sides of one port

A layer does two jobs in training\. Forward, it turns parameters and inputs into outputs\. Backward, it turns a cotangent at the output into cotangents at the input and at the parameters\. Backpropagation keeps the pair together, and a category theorist calls it a*lens*\([Background B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

Every method in Exposés[I](https://arxiv.org/html/2610.12728#S1)to[III](https://arxiv.org/html/2610.12728#S3)changes the forward job, replacing the parameter portΘ\\Thetaby a smaller spaceQQand a mapρ:Q→Θ\\rho:Q\\to\\Theta\. A second family edits only the backward job\. GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)compresses the weight gradientGGtoP⊤​GP^\{\\top\}G, withPPfrom an SVD ofGG, runs Adam on that small matrix and maps the result back withPP\. Every weight trains; only the optimizer state is small\. LISA\([Pan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib360)\)and MeZO\([Malladi et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib310)\)follow the same pattern\.

Whenρ\\rhois linear the two families meet\. The pullback of the forward method is then the compression of a backward method, and an optimizer that reads only its gradients cannot tell the two apart\. The equivalence between GaLore and one\-sided LoRA is due to[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453), who prove a duality between linear gradient transformations and adapters \(their Thm 1 and Cor\. 3, with weight decay in their App\. B\)\. The frozen\-AAcase under SGD, with random projections, appears earlier in[Hao et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib154)\. Our additions are the lens bookkeeping, the explicit hypotheses on the optimizer, and the table that sorts the cotangent family \(Table[15](https://arxiv.org/html/2610.12728#S4.T15)\)\.

#### Backpropagation as a lens

In the language of[Definition I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), a layer is a parametrised mapf:P×X→Yf:P\\times X\\to Y\. Its reverse derivative𝖱⁡\[f\]​\(x,y\)=Jf​\(x\)⊤​y\\mathsf\{R\}\[f\]\(x,y\)=J\_\{f\}\(x\)^\{\\top\}ytakes an output cotangent to an input cotangent, parameters included, and backpropagation is the 2\-functor

𝐏𝐚𝐫𝐚⁡\(𝖱\):𝐏𝐚𝐫𝐚⁡\(𝒞\)⟶𝐏𝐚𝐫𝐚⁡\(𝐋𝐞𝐧𝐬⁡\(𝒞\)\),f⟼\(f,𝖱⁡\[f\]\)\.\\mathbf\{Para\}\(\\mathsf\{R\}\):\\mathbf\{Para\}\(\\mathcal\{C\}\)\\longrightarrow\\mathbf\{Para\}\(\\mathbf\{Lens\}\(\\mathcal\{C\}\)\),\\qquad f\\longmapsto\(f,\\mathsf\{R\}\[f\]\)\.\(6\)The parametrised\-map construction and the reading of backpropagation as a functor into lenses are due to[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\([Background B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3),[Background B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

A reparametrisationρ\\rhois a 2\-cell, so the functor turns it into a lens\(ρ,𝖱⁡\[ρ\]\)\(\\rho,\\mathsf\{R\}\[\\rho\]\), and a reparametrisation therefore acts on both sides of the parameter port\. Its tangent side is the image, which is what the method can*express*\(Exposé[II](https://arxiv.org/html/2610.12728#S2)\)\. Its cotangent side𝖱⁡\[ρ\]​\(q,γ\)=D​ρq⊤​γ\\mathsf\{R\}\[\\rho\]\(q,\\gamma\)=D\\rho\_\{q\}^\{\\top\}\\gammapulls the weight gradient back toQQand, with the ratesΛ\\Lambda, induces the cometricK=D​ρ​Λ​D​ρ⊤K=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}\([Definition III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),[Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\), which is how the method*moves*\. A lens need not come from a map, though, and a backward half written down directly gives the second family\. The two kinds of lens on the parameter port are

Q\{\\lx@inpgf@ignorespaces Q\}Θ\{\\lx@inpgf@ignorespaces\\Theta\}Tq∗​Q\{\\lx@inpgf@ignorespaces T^\{\*\}\_\{q\}Q\}Tρ⁡\(q\)∗​Θ\{\\lx@inpgf@ignorespaces T^\{\*\}\_\{\\rho\(q\)\}\\Theta\}ρ\\scriptstyle\{\\lx@inpgf@ignorespaces\\rho\}D​ρq⊤\\scriptstyle\{\\lx@inpgf@ignorespaces D\\rho\_\{q\}^\{\\top\}\}Θ\{\\lx@inpgf@ignorespaces\\Theta\}Θ\{\\lx@inpgf@ignorespaces\\Theta\}Tθ​Θ\{\\lx@inpgf@ignorespaces T\_\{\\theta\}\\Theta\}S\{\\lx@inpgf@ignorespaces S\}Tθ∗​Θ\{\\lx@inpgf@ignorespaces T^\{\*\}\_\{\\theta\}\\Theta\}idΘ\\scriptstyle\{\\lx@inpgf@ignorespaces\\mathrm\{id\}\_\{\\Theta\}\}at\\scriptstyle\{\\lx@inpgf@ignorespaces a\_\{t\}\}ct\\scriptstyle\{\\lx@inpgf@ignorespaces c\_\{t\}\}on the left a forward method, whose backward half is forced by its forward half, and on the right a backward method, whose forward half is the identity and whose backward half is chosen freely\.

Definition IV\.1\(forward and backward methods\)\.•Aforwardmethod acts on the parameter port by a lens in the image of𝖱\\mathsf\{R\}, namely\(ρ,𝖱⁡\[ρ\]\)\(\\rho,\\mathsf\{R\}\[\\rho\]\)\.•Abackward, or cotangent\-side, method acts by a lens whose forward part isidΘ\\mathrm\{id\}\_\{\\Theta\}and whose backward part is a*compression*ct:Tθ∗​Θ→Sc\_\{t\}:T^\{\*\}\_\{\\theta\}\\Theta\\to Sinto a state spaceSS, paired with an*anchor*at:S→Tθ​Θa\_\{t\}:S\\to T\_\{\\theta\}\\Theta\. The update isθ←θ−η​at​\(opt⁡\(ct​\(∇L\)\)\)\.\\theta\\leftarrow\\theta\-\\eta\\,a\_\{t\}\\big\(\\mathrm\{opt\}\(c\_\{t\}\(\\nabla L\)\)\\big\)\.\(7\)Table[14](https://arxiv.org/html/2610.12728#S4.T14)lists examples\.

Table 14:Backward methods as compression–anchor pairs \([Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\)\. The last row is the only reading of Flora as published in the form of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2), the trivial lens with all compression insideopt\\mathrm\{opt\}\.A backward method has its own cometricKt=at​ctK\_\{t\}=a\_\{t\}c\_\{t\}\([Definition III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)\), for instanceK⁡\(G\)=P​P⊤​GK\(G\)=PP^\{\\top\}Gfor GaLore under SGD\. The update must lie in the image of the anchor\. Read with their projections as anchors, Flora as published\([Hao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib154)\), APOLLO\([Zhu et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib596)\)and Fira\([Chen et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib67)\)make updates that leaveim⁡a\\operatorname\{im\}a, so they are not backward methods in the sense of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)beyond the trivial reading with compression and anchor the identity, which the last row of Table[14](https://arxiv.org/html/2610.12728#S4.T14)records for Flora\. Flora compresses its momentum or gradient accumulator with a random projection and hands the decompressed tensor to Adafactor\([Shazeer and Stern, 2018](https://arxiv.org/html/2610.12728#bib.bib407)\), which acts in the full space, so its update need not lie in the image of the projection\. On resampling it transports its momentum to the new frame\. It fits[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)with a nontrivial anchor only in an idealised form with an SGD or heavy\-ball base\.

#### The degree\-one collapse

Ifρ⁡\(C\)=Wk\+ℒ​C\\rho\(C\)=W\_\{k\}\+\\mathcal\{L\}C, trainingCCand running the backward method withc=ℒ∗c=\\mathcal\{L\}^\{\*\},a=ℒa=\\mathcal\{L\}feed the optimizer the same gradientsℒ∗​G\\mathcal\{L\}^\{\*\}Gand so make the same weight updates\. A curvedρ\\rho, such as LoRA’sB​ABA, separates them after one step, and so does an optimizer that reads the parameter tensor, which isCCin one procedure andWWin the other\.

Proposition IV\.2\(the degree\-one collapse; known,[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)for GaLore \(their Thm 1 and Cor\. 3, App\. B for weight decay\) and[Hao et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib154)for random projections\)\.Letρk​\(C\)=Wk\+ℒ​C\\rho\_\{k\}\(C\)=W\_\{k\}\+\\mathcal\{L\}Cwithℒ\\mathcal\{L\}linear andC0=0C\_\{0\}=0\. Letopt\\mathrm\{opt\}be a first\-order optimizer, with no coupled or decoupled weight decay, whose state and output depend only on the sequence of gradients it is fed, and which does not read the parameter tensor: SGD, heavy\-ball or Nesterov momentum, Adam or AMSGrad, or Adafactor with scale\_parameter=False \(fixed or relative step, with its factorisation applied to the compressed gradient in the same tensor shape asCC\)\. Run both procedures below with the same implementation ofopt\\mathrm\{opt\}, on the same minibatches, and from the same optimizer state, fresh or carried over\. Then they produce the sameWW\-trajectory over a period:•trainingCCwithopt\\mathrm\{opt\}and merging at the end of the period;•the backward method withc=ℒ∗c=\\mathcal\{L\}^\{\*\}anda=ℒa=\\mathcal\{L\}, started atWkW\_\{k\}\.AdamW withλ\>0\\lambda\>0, LAMB and LARS trust ratios, parameter\-scaled Adafactor \(scale\_parameter=True, the default in HF and in galore\_torch’s GaLoreAdafactor\) and global\-norm clipping of the full gradient \(such as HF Trainer’s default max\_grad\_norm=1\.0, whenever it triggers\) all break the identity\. Injectivity ofℒ\\mathcal\{L\}is not needed\.In particular, on each matrix that GaLore projects \(its other parameters train in full in both procedures\), GaLore with a projectorPkP\_\{k\}fixed on each period equals one\-sided LoRAWk\+Pk​CW\_\{k\}\+P\_\{k\}C\(frozenB=PkB=P\_\{k\}, or frozenAAwhen projecting on the right\), merged and re\-based every period, withCC’s optimizer state \(moments and step counter\) carried over unchanged across re\-basings, with no reset\.PkP\_\{k\}must be exactly GaLore’s matrix, signs and column order included; its span alone does not suffice, because the state is reused without transport\. ReLoRA\-style resets give a different trajectory\. The identity is exact with weight decay00, no full\-gradient clipping, and an optimizer from the admissible class; galore\_torch’s GaLoreAdafactor defaults to scale\_parameter=True and is excluded\. GaLoreAdamW is the transformers\-style AdamW, which addsε\\varepsilontov\\sqrt\{v\}before the bias correction, so the LoRA side must use the same implementation\. GaLore’s scaleα\\alphamultiplies the optimizer output, so it acts as a learning\-rate factor; a LoRA\-style scale insideρ\\rhogives a different trajectory under Adam withε\>0\\varepsilon\>0\.

Remark\(scale and implementation\)\. Moving GaLore’sα\\alphainsideρ\\rhoas a LoRA scale multiplies the compressed gradient byα\\alpha\. Under Adam this gives GaLore’s trajectory only atε=0\\varepsilon=0, or with the LoRA side’sε\\varepsilonmultiplied byα\\alphaas well\. This is the exchange of scale for learning rate andε\\varepsilonon the hyperparameter torus of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\. Both sides must also run the same implementation: galore\_torch’s AdamW addsε\\varepsilontov\\sqrt\{v\}before the bias correction, as in transformers, and swapping in torch\.optim\.Adam on the LoRA side moves the trajectory measurably \(see the check below\)\.

NUMIn a toy check under Adam the two procedures agree to7×10−167\\times 10^\{\-16\}over one period, and to5×10−155\\times 10^\{\-15\}over four periods with the state carried on both sides\. Against the official galore\_torch optimizers \(least squares,W∈ℝ6×10W\\in\\mathbb\{R\}^\{6\\times 10\},r=2r=2,T=7T=7, four periods, total movement0\.0460\.046\), the two procedures agree to8×10−178\\times 10^\{\-17\}with the state carried over\. Resetting the LoRA side’s state separates them by0\.0290\.029, weight decay0\.10\.1by0\.0200\.020, HF\-default Adafactor by0\.0300\.030\(against6×10−176\\times 10^\{\-17\}with scale\_parameter=False\), and flipping the sign of one column ofPkP\_\{k\}, with the state carried, by0\.120\.12\. Using torch\.optim\.Adam on the LoRA side moves the trajectory by about4×10−54\\times 10^\{\-5\}, and placingα\\alphainsideρ\\rhoby0\.0190\.019atε=10−6\\varepsilon=10^\{\-6\}, with exact agreement atε=0\\varepsilon=0\. The companion site reruns a check of this kind live \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSBeyond one\-sided LoRA, GaLore adds two ingredients: a frame read off the gradient’s SVD, and the carry\-over of optimizer state across re\-basings\. The carry\-over involves no change of basis\. Moments written in the coordinates ofPk−1P\_\{k\-1\}are read as coordinates forPkP\_\{k\}\. Projection\-aware state updates, such as LDAdam\([Robert et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib392)\), and Flora transport the state instead\. The equivalence classifies the cotangent methods whose optimizer satisfies its hypotheses\. Table[15](https://arxiv.org/html/2610.12728#S4.T15)sorts them by where the projector comes from and how often it changes; its LoRA and ReLoRA row is a contrast row outside the equivalence\.

Table 15:The cotangent family by projector \(rows\) and schedule \(columns\)\.[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)covers the first three rows under its optimizer hypotheses, except EVA and LoRA\-GA, which initialise two\-sided LoRA and sit in the frozen column only for their first step\. The last row is a contrast row outside the equivalence\.aFor the original method under SGD\. HF PEFT’s LoRA\-FA optimizer preconditions by\(A0​A0⊤\+δ​I\)−1\(A\_\{0\}A\_\{0\}^\{\\top\}\+\\delta I\)^\{\-1\}, which in the undamped limit gives the row\-space projectorG​PVGP\_\{V\}of[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(c\); its Adam layer makes the update nonlinear inGG\.bFlora’s Alg\. 2 with an SGD or heavy\-ball base, with the momentum transported at resampling instead of carried over; Flora as published, with an Adafactor base, is outside[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)and outside[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\.

[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)covers the first three rows of Table[15](https://arxiv.org/html/2610.12728#S4.T15)under its optimizer hypotheses, except EVA and LoRA\-GA, which initialise two\-sided LoRA and sit in the frozen column only for their first step; it covers MeZO up to the finite\-difference error of its estimator\. LoRA’s map\(B,A\)↦W0\+s​B​A\(B,A\)\\mapsto W\_\{0\}\+sBAhas degree two, so its cometric moves with the state \([Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)\) and the collapse does not apply\. The other frozen entries gain nothing from merge\-and\-restart \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 2\); their re\-based neighbours move the frame and can reach all of weight space\. The LoRA\-FA cometric holds for the original method of[Zhang et al\. \(2023d\)](https://arxiv.org/html/2610.12728#bib.bib556)under SGD; the LoRA\-FA optimizer of HF PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)gives the row\-space projector of[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(c\), as noted under the table\.

Flora proper is not in the table\. Its Adafactor step acts on the full gradient, with parameter scaling by default, so its steps are not confined toim⁡Pt\\operatorname\{im\}P\_\{t\}and[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)does not apply; like LDAdam, it transports its optimizer state\. Only the idealised form with an SGD or heavy\-ball base sits in the random, re\-based cell, and there its momentum is transported rather than carried\.

#### When a backward method is a forward one

A backward method chooses at each weight a space of directionsDθ=im⁡a⁡\(θ\)D\_\{\\theta\}=\\operatorname\{im\}a\(\\theta\)\. It reads as a forward method only if these directions fit together intokk\-dimensional surfaces, the images of immersive parametrisations \([Background C\.4](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\)\. Frobenius’ theorem tests this \([Background D\.25](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3),[Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\)\. The bracket\[X,Y\]\[X,Y\], the gap by which a small loop alongXX,YY,−X\-X,−Y\-Yfails to close, must stay inDD\. Discrete steps and the optimizer’s metric ask for more\.

Proposition IV\.3\(integrability; after Frobenius’ theorem\)\.LetDDbe a smooth distribution of constant rankkkon an open setU⊆ΘU\\subseteq\\Theta, for instance the anchor distributionDθ=im⁡a⁡\(θ\)D\_\{\\theta\}=\\operatorname\{im\}a\(\\theta\)of a time\-independent backward method\. ThenDDis involutive onUUif and only if through everyθ∈U\\theta\\in Upasses an injective immersionρθ:\(ℝk,0\)→\(U,θ\)\\rho\_\{\\theta\}:\(\\mathbb\{R\}^\{k\},0\)\\to\(U,\\theta\)withim⁡D​ρθ=D\\operatorname\{im\}D\\rho\_\{\\theta\}=Dalong its image\. A single suchρ\\rhothroughθ0\\theta\_\{0\}gives involutivity only alongim⁡ρ\\operatorname\{im\}\\rho\. For example,D=span\{∂x,∂y\+xz∂z\}D=\\spanop\\\{\\partial\_\{x\},\\ \\partial\_\{y\}\+xz\\,\\partial\_\{z\}\\\}onℝ3\\mathbb\{R\}^\{3\}has\[∂x,∂y\+xz∂z\]=z∂z∉D\[\\partial\_\{x\},\\partial\_\{y\}\+xz\\,\\partial\_\{z\}\]=z\\,\\partial\_\{z\}\\notin Doffz=0z=0, yet the planez=0z=0is an integral manifold\.The dynamic content needs separate hypotheses\.•In continuous time, ifDDis involutive, a backward flow withθ˙∈Dθ\\dot\{\\theta\}\\in D\_\{\\theta\}stays on the leaf throughθ0\\theta\_\{0\}\.•In the discrete update \([7](https://arxiv.org/html/2610.12728#S4.E7)\) of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2), the iterates stay on a leaf for all step sizes and all optimizer outputsss\(all losses\) if and only if that leaf contains\(θ\+Dθ\)∩U\(\\theta\+D\_\{\\theta\}\)\\cap Uat every iterate, that is, if and only if the leaves are affine; an anchor constant over the period gives this \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.•Under gradient flow, dynamic equivalence with a forward lens\(ρ,𝖱⁡\[ρ\]\)\(\\rho,\\mathsf\{R\}\[\\rho\]\)forces matching frames,at​ct=D​ρ​Λ​D​ρ⊤a\_\{t\}c\_\{t\}=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}along the trajectory\. HenceK=a​cK=acis symmetric positive semidefinite and, for injectiveaa,c=M​a⊤c=Ma^\{\\top\}withMMsymmetric positive semidefinite onSS, possibly depending onθ\\theta\. If in additionρ\\rhois an immersion withdimQ=rank⁡D\\dim Q=\\rk D, thenρ\\rhois a local isometry from\(Q,Λ−1\)\(Q,\\Lambda^\{\-1\}\)onto the leaf with metric\(a​M​a⊤\)−1\|D\(aMa^\{\\top\}\)^\{\-1\}\|\_\{D\}, so that metric must be flat\. Forward methods with gauge fibres escape this requirement:ρ⁡\(B\)=B​B⊤\\rho\(B\)=BB^\{\\top\}on symmetric matrices induces the curved Bures–Wasserstein metric, and balanced full\-rank LoRA also induces a curved leaf metric\. For discrete steps, exact agreement for every step size further forcesρ\\rhoto be affine along each horizontal lineq\+t​Λ​D​ρq⊤​gq\+t\\,\\Lambda D\\rho\_\{q\}^\{\\top\}g\.*Instances\.*Every projector\-type backward method of Table[14](https://arxiv.org/html/2610.12728#S4.T14)\(GaLore with any periodTTand any projector, its random\-projector variant, LISA, MeZO\) has an anchor that does not depend onθ\\thetawithin its period\. Its distribution is constant, hence trivially involutive with affine leaves \(vacuously atT=1T=1, where a period is a single step\)\. Under the optimizer hypotheses of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\(no weight decay of either kind, no full\-gradient clipping; GaLore and MeZO need weight decay00, and LISA’s AdamW with positive decay violates them\), these methods are rebasing schemes of linear forward methods with periodTT, includingT=1T=1, whose reach[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)describes\. Each MeZO step is a step of the linear methodθt\+λ​zt\\theta\_\{t\}\+\\lambda z\_\{t\}, up to its SPSA estimator, and GaLore withT=1T=1is one\-sided LoRA re\-based every step\. What separates these methods from LoRA\-type methods is the resampling schedule together with the carry\-over of optimizer state; integrability plays no part in it\. Flora as published \(Adafactor base, with a full\-rank current\-gradient term in the code\), APOLLO \(a low\-rank\-estimated scaling applied to the full gradient\) and Fira \(a scaled full\-rank residual\) are not of the form of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)with their projections as anchors, because their updates leaveim⁡a\\operatorname\{im\}a\.Non\-involutivity proper needs aθ\\theta\-dependent anchor\. Full\-batch GaLore withT=1T=1, read as the distributionDW=Hom⁡\(ℝn,Ur​\(∇L​\(W\)\)\)D\_\{W\}=\\operatorname\{Hom\}\(\\mathbb\{R\}^\{n\},U\_\{r\}\(\\nabla L\(W\)\)\)\(orHom⁡\(Vr​\(∇L​\(W\)\),ℝm\)\\operatorname\{Hom\}\(V\_\{r\}\(\\nabla L\(W\)\),\\mathbb\{R\}^\{m\}\)whenm≥nm\\geq n, as GaLore projects\) wherever a singular\-value gap exists, is involutive if and only ifUr​\(∇L​\(W\)\)U\_\{r\}\(\\nabla L\(W\)\)\(respectivelyVr​\(∇L​\(W\)\)V\_\{r\}\(\\nabla L\(W\)\)\) is invariant alongDD\. That fails generically for everyr≥1r\\geq 1once both matrix dimensions are at least22\. For everyrrwith1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)it fails forL=12​∥W∥F2L=\\tfrac\{1\}\{2\}\\lVert W\\rVert\_\{F\}^\{2\}, and numerically it fails for least squares and cross\-entropy with several samples\. It holds, for instance, for single\-sample least squares, and automatically only when the distribution is a line field \(n​r=1nr=1\)\. Minibatch GaLore withT=1T=1and MeZO violate time\-independence, because their anchors depend on step\-wise randomness\.

NUMFor full\-batch GaLore withT=1T=1, the relative normal part of the bracket is0\.080\.08to0\.940\.94atr=1r=1and\(m,n\)∈\{\(4,3\),\(5,4\)\}\(m,n\)\\in\\\{\(4,3\),\(5,4\)\\\}, for least\-squares, tanh\-regression and cross\-entropy losses with several samples \(an independent replication\), and0\.210\.21to0\.950\.95at\(m,n,r\)=\(4,3,2\)\(m,n,r\)=\(4,3,2\)\. It vanishes only forn=1n=1or for single\-sample least squares\. AtW=diag⁡\(2,1\)W=\\diag\(2,1\)withL=12​∥W∥F2L=\\tfrac\{1\}\{2\}\\lVert W\\rVert\_\{F\}^\{2\}and the frameXi=u⁡\(W\)​ei⊤X\_\{i\}=u\(W\)e\_\{i\}^\{\\top\}, the bracket is\[X1,X2\]=−13​e2​e1⊤∉D\[X\_\{1\},X\_\{2\}\]=\-\\tfrac\{1\}\{3\}e\_\{2\}e\_\{1\}^\{\\top\}\\notin D, and the general diagonal formula behind it was checked by central differences for several\(m,n,r\)\(m,n,r\)\. The cometric ofB↦B​B⊤B\\mapsto BB^\{\\top\}takes the same value atBBand atB​RBR,R∈O⁡\(2\)R\\in\\mathrm\{O\}\(2\), to10−1210^\{\-12\}\. The companion site recomputes the bracket live atW∈ℝ3×4W\\in\\mathbb\{R\}^\{3\\times 4\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSThe lens 2\-functor \([6](https://arxiv.org/html/2610.12728#S4.E6)\) separates tangent\-side from cotangent\-side methods\. Frobenius integrability is the test for when aθ\\theta\-dependent anchor foliates weight space by images of forward methods\. For the PEFT zoo it does no discriminating work\. Every cotangent method whose update lies inim⁡a\\operatorname\{im\}a, whose optimizer does not read the parameter, and which has no decoupled decay onWWand no clipping of the full gradient \(GaLore with any projector and MeZO at weight decay00, LISA without decay\) is, period by period, a forward linear method on a moving base \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\)\. AtT=1T=1this reading is automatic\.

Analogy\(the lens duality\)\. “Tangent side” and “cotangent side” are exact words here\. A forward method acts on tangent vectors throughD​ρD\\rho, and a backward method acts on cotangent vectors throughcc\. Calling GaLore and one\-sided LoRA*dual*is exact only at degree one, and only for optimizers that do not read the parameter tensor, with no weight decay\. Beyond that range the word organises the zoo and proves nothing\.

#### What the backward pass must remember

Fine\-tuning memory consists of weights, gradients and optimizer state, and the activations stored for the backward pass\. ShrinkingQQshrinks the gradients and the optimizer state\. As for the activations, a frozen linear box’s reverse derivativeW⊤​yW^\{\\top\}yignores its input, and cotangents are needed only downstream of something trainable, so only a cone through the network costs activation memory\.

Observation IV\.4\(backward reach and the backprop cone; an upper bound\)\.Assume reverse mode in which each box keeps its own residual, with no checkpointing across boxes, and take boxes at the granularity where adapters are separate boxes \(for LoRA\-FA,A0A\_\{0\}andBBseparately\)\. A*VJP residual*is an input, an output, a sufficient statistic such as a ReLU mask, or a recipe to recompute one of these inside the box\. Then it suffices to keep a VJP residual for•each trainable box whose parameter\-VJP depends on its input: linear and multiplicative boxes such asWW,AA,BBand \(IA\)3scales; additive boxes such as biases and prompts need nothing;•each*nonlinear*box, bilinear boxes such as the attention products included, on a directed path from a trainable box to the loss\.Deterministic frozen linear boxes, and all boxes off this cone, need nothing; stochastic linear boxes on the cone \(dropout\) need their mask or random\-number state\. Alternatively, thekkcoordinates ofP⊤∇LP^\{\\top\}\\nabla Lcan be computed exactly bykkforward\-mode JVPs, with no reverse pass andO⁡\(1\)O\(1\)\-layer activation memory, each JVP costing a small constant multiple \(about1\.51\.5to33\) of a forward pass\. This always uses less activation memory than un\-checkpointed reverse mode, but it is time\-competitive with one VJP \(about33forward passes\) only forkkof about11or22\. It also needsPPbefore the passes, so it cannot produce GaLore’s SVD refresh, wherePPdepends on∇L\\nabla L, more cheaply than a backward pass \(∇L\\nabla Litself would take\|θ\|\\lvert\\theta\\rvertJVPs\)\.

WHAT THIS EXPLAINSThe bound reads off the activation memory of the zoo\.

- •LST\([Sung et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib439)\)\. The side network reads the backbone through trainable down\-projection ladders, so the backbone’s backward pass never runs\.
- •LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\)\. With lora\_dropout=0 \(the HF default\) it storesA0​xA\_\{0\}x\(rrnumbers per token\) instead ofxx\(nnnumbers\); with dropout on, thenn\-element mask is stored too\. The saving is realised only where no neighbouring nonlinear box already keepsxx\.
- •Top\-layer adaptationtruncates the cone, under the no\-checkpointing assumption\. With HF’s reentrant gradient checkpointing, PEFT calls enable\_input\_require\_grads\(\), which makes the embedding outputs require gradients, so cotangents run down to the embeddings and top\-layer adaptation no longer truncates the cone\.
- •LISA\([Pan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib360)\)trains the embedding and the LM head at every step, so its cone is the whole network and it gets no truncation\. On the cone, the unsampled layers’ frozen linear boxes drop their input residuals \(about20%20\\%of a block’s activations at LLaMA proportions without FlashAttention\), and relative to LoRA it also drops the adapter residuals, in line with the LISA paper’s report of less activation memory than LoRA\. Most of its saving against full fine\-tuning is in gradients and optimizer state\. A variant that froze the embedding would truncate the cone at its lowest sampled layer\.
- •Prompt tuning\([Lester et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib243)\)has aQQas small as that of VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\), FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\)or \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\), and the maximal cone, the whole network, which it shares with every method that trains something in the first layer\. Its prompt tokens also enlarge every stored activation\.
- •MeZO\([Malladi et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib310)\)is the casek=1k=1, with the JVP estimated by a central difference and no reverse pass\. Its estimator is unbiased for the gradient of the Gaussian\-smoothed loss and, for a loss with Lipschitz Hessian, has biasO⁡\(ε2\)O\(\\varepsilon^\{2\}\)against∇L\\nabla L; ReLU networks lack a Lipschitz Hessian\. Exact unbiasedness,𝔼\[\(∇L⋅z\)z\]=∇L\\mathbb\{E\}\[\(\\nabla L\\cdot z\)z\]=\\nabla L, holds for the forward gradient computed with an exact JVP\. GaLore, its random\-projector variant, Flora and LISA all take the reverse route\.

Ladder side\-tuning is due to[Sung et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib439), prompt tuning to[Lester et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib243), and the zeroth\-order estimator in this setting to[Malladi et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib310)\. That reverse derivatives of linear maps ignore their inputs is the observation behind many activation\-saving designs; the lens of[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)makes it visible box by box\.

So every cotangent method meeting the hypotheses of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)is, period by period, a linear forward method on a moving base\. What it reaches, and what a quantiser does to the base, are the subject of Exposé[V](https://arxiv.org/html/2610.12728#S5)\.

## EXPOSÉ VBase change

Study each object over a base, so that the base can be changed\. What survives the change is what the object really is\. after Grothendieck

Quantising and restarting move the point a method is attached to, and combining adapters builds one update from several\. Each move comes down to one computation, of the defect a quantiser leaves, the monoid one cycle generates, or the gauge a combination rule must ignore\.

##### In brief

- •QLoRA is LoRA anchored at the quantised weightκ​θ0\\kappa\\theta\_\{0\}, so it missesθ0\\theta\_\{0\}by the quantisation erroree, which is generically of full rank\. LoftQ and QPiSSA are built to shrink this defect, by quantising and splitting in opposite orders\. HF PEFT’s LoftQ initialiser ignores the LoRA scales=α/rs=\\alpha/r, so with its default single iteration it beats QLoRA in Frobenius norm only for0<s<20<s<2\.
- •With a fixed shape, merge\-and\-restart reaches exactly the monoid one cycle generates\. ReLoRA reaches full rank after⌈min⁡\(m,n\)/r⌉\\lceil\\min\(m,n\)/r\\rceilcycles, while fixed linear shapes such as BitFit, \(IA\)3or a fixed mask gain nothing\.
- •Exactly orthogonal block OFT on a fixed partition never leavesW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\), and re\-merged BOFT reaches all rotations only at full butterfly depth\. HF’s default Cayley–Neumann OFT is not orthogonal, and with small generators each merge can only shrink neuron norms and singular values\.
- •A rule combining LoRA factors is well defined only if it ignores each adapter’sGLr\\mathrm\{GL\}\_\{r\}gauge\. Task arithmetic on the products passes\. LoraHub and HF’s factor\-wise linear merge fail through cross terms, and become exact only when the adapters share a factor\.

#### The point moves

Every method so far has been an arrow into one fixed point\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)\. In practice the point moves\. Quantisation moves it once, ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)at every merge, and task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\), LoraHub\([Huang et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib180)\), TIES\([Yadav et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib517)\)and DARE\([Yu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib538)\)build a new update from several adapters\. A PEFT method is a formula that makes sense over any base point, so it is a familyθ0↦M⁡\(θ0\)\\theta\_\{0\}\\mapsto M\(\\theta\_\{0\}\), and this exposé relates its members\. Four kinds of move occur \(Table[16](https://arxiv.org/html/2610.12728#S5.T16)\)\.

Table 16:Four ways the base point moves, and what each one asks us to compute\.Definition V\.1\(base change and rebasing\)\.\(a\)A*function\-preserving*pointed reparametrisationϕ:\(Θ′,f′,θ0′\)→\(Θ,f,θ0\)\\phi:\(\\Theta^\{\\prime\},f^\{\\prime\},\\theta\_\{0\}^\{\\prime\}\)\\to\(\\Theta,f,\\theta\_\{0\}\), withf∘\(ϕ×idX\)=f′f\\circ\(\\phi\\times\\mathrm\{id\}\_\{X\}\)=f^\{\\prime\}, induces by postcompositionϕ\!:𝐌𝐞𝐭𝐡\(Θ′,θ0′\)→𝐌𝐞𝐭𝐡\(Θ,θ0\),\(Q,q0,ρ\)↦\(Q,q0,ϕ∘ρ\)\.\\phi\_\{\!\}:\\mathbf\{Meth\}\(\\Theta^\{\\prime\},\\theta\_\{0\}^\{\\prime\}\)\\to\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\),\\qquad\(Q,q\_\{0\},\\rho\)\\mapsto\(Q,q\_\{0\},\\phi\\circ\\rho\)\.\(b\)A mapκ:Θ→Θ\\kappa:\\Theta\\to\\Thetathat does*not*preserve the function, such as a quantiser, moves the base point\. Itspointing defectise=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\.\(c\)Arebasing scheme\(merge\-and\-restart\) picksθk\+1∈Im⁡M⁡\(θk\)\\theta\_\{k\+1\}\\in\\Imop M\(\\theta\_\{k\}\)\. Its reachable sets areRKR\_\{K\}, the pointsθK\\theta\_\{K\}it can reach inKKcycles, andR∞=⋃KRKR\_\{\\infty\}=\\bigcup\_\{K\}R\_\{K\}\.

Only part \(a\) is base change in the algebraic geometer’s sense \([Background A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\), and it costs nothing; whetherϕ\!\\phi\_\{\!\}turns LoRA into LoRA is the covariance question of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\. Parts \(b\) and \(c\) borrow the word, and the work lies there and in the combination rules\. Figure[10](https://arxiv.org/html/2610.12728#S5.F10)shows the restart computations of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\.

Remark\(what “base change” means here\)\. In algebraic geometry a family over a baseSSpulls back along any mapS′→SS^\{\\prime\}\\to S, and the statements worth having are the ones that survive the change\. Part \(a\) of[Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1)is the standard companion of pullback on slice categories, composition with a map of bases\. A quantiser changes the network’s function, so there is no commuting square to transport along, and we compute the defect it leaves instead\. A rebasing scheme is an iteration, and we compute its reachable set\. “Gauge” keeps the sense of Exposé[III](https://arxiv.org/html/2610.12728#S3), the symmetry group of a parametrisation \([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\); no connection is involved\.

#### Quantisation and splitting

For an additive method \([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\) whose starting point does not depend on the base, such as zero\-init LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\), quantising the base only translates the image, so QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)is LoRA with its vertex atκ​θ0\\kappa\\theta\_\{0\}\. A quantisation error is spread over all singular directions, soeeis generically of full rank and the image missesθ0\\theta\_\{0\}\. Methods whose starting point is computed from the base, such as PiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\), CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\), OLoRA\([Büyükakyüz, 2024](https://arxiv.org/html/2610.12728#bib.bib47)\), LoftQ\([Li et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib255)\)and DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\), meet two operations that do not commute, and the order fixes the defect \(tabulated in Table[5](https://arxiv.org/html/2610.12728#S2.T5)of Exposé[II\.A](https://arxiv.org/html/2610.12728#S2.SS1)\)\.

Remark\(2026: SRR\)\. SRR\([Cho et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib76)\)joins the first two orders of[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\) by splitting the rank budget\. It putsSVDk⁡\(W\)\\operatorname\{SVD\}\_\{k\}\(W\)into the adapter, quantisesWres=W−SVDk⁡\(W\)W\_\{\\rm res\}=W\-\\operatorname\{SVD\}\_\{k\}\(W\)and fits the quantisation error with the otherr−kr\-kranks\. Its defect is thereforeeke\_\{k\}minus its best rank\-\(r−k\)\(r\-k\)approximation, whereek=κ⁡\(Wres\)−Wrese\_\{k\}=\\kappa\(W\_\{\\rm res\}\)\-W\_\{\\rm res\}, andSVDk\\operatorname\{SVD\}\_\{k\}and best approximations are taken in its activation\-scaled norm\. With that scaling set to the identity,k=rk=rgives the split\-first defect of QPiSSA andk=0k=0that of one\-step LoftQ ats=1s=1\. SRR pickskkper matrix from a closed\-form proxy\.

Proposition V\.2\(quantisation and splitting\)\.WriteWWfor the full\-precision weight,κ\\kappafor the quantiser,e=κ​W−We=\\kappa W\-W,δ0=δ⁡\(q0\)\\delta\_\{0\}=\\delta\(q\_\{0\}\), andssfor the LoRA scaleα/r\\alpha/r\(α/r\\alpha/\\sqrt\{r\}under rsLoRA,[Kalajdzievski, 2023](https://arxiv.org/html/2610.12728#bib.bib204)\)\.\(a\) Additive families commute with translation,M⁡\(θ\+v\)=τv∘M⁡\(θ\)M\(\\theta\+v\)=\\tau\_\{v\}\\circ M\(\\theta\)\. Hence QLoRA is LoRA pointed atκ​θ0\\kappa\\theta\_\{0\}, with defectee, and it needs no new weight\-space structure\. Since−e\-eis generically of full rank,θ0∉Im⁡LoRAr​\(κ​θ0\)\\theta\_\{0\}\\notin\\Imop\\mathrm\{LoRA\}\_\{r\}\(\\kappa\\theta\_\{0\}\)unlessrank⁡e≤r\\rk e\\leq r\.\(b\) For splitting families derived from an SVD or from data, the order of operations matters, and three orders occur in practice\.•*Split, then quantise the residual*\(QPiSSA; HF’s recommended PiSSA and CorDA workflow; the quantisation half\-step of LoftQ fort≥2t\\geq 2, withWres=W−δ⁡\(qt−1\)W\_\{\\rm res\}=W\-\\delta\(q\_\{t\-1\}\)\)\. The defect isκ⁡\(Wres\)−Wres\\kappa\(W\_\{\\rm res\}\)\-W\_\{\\rm res\}\.•*Quantise, then fit the error*\(LoftQ with one iteration, HF’s default loftq\_iter=1; HF replace\_lora\_weights\_loftq\)\. HF sets the factors fromSVDr⁡\(W−κ​W\)\\operatorname\{SVD\}\_\{r\}\(W\-\\kappa W\)without dividing byss, unlike its PiSSA and OLoRA initialisations, soδ0=−s​SVDr⁡\(e\)\\delta\_\{0\}=\-s\\operatorname\{SVD\}\_\{r\}\(e\)and the defect ise−s​SVDr​\(e\)e\-s\\operatorname\{SVD\}\_\{r\}\(e\), with∥e−s​SVDr⁡\(e\)∥2=∥e−SVDr⁡\(e\)∥2\+\(1−s\)2​∥SVDr⁡\(e\)∥2\.\\lVert e\-s\\operatorname\{SVD\}\_\{r\}\(e\)\\rVert^\{2\}=\\lVert e\-\\operatorname\{SVD\}\_\{r\}\(e\)\\rVert^\{2\}\+\(1\-s\)^\{2\}\\lVert\\operatorname\{SVD\}\_\{r\}\(e\)\\rVert^\{2\}\.The defect iseeminus its best rank\-rrapproximation \(of rankrank⁡e−r\\rk e\-r\) iffs=1s=1, which is HF’s bare defaultr=α=8r=\\alpha=8\. Fors=2s=2its norm equals∥e∥\\lVert e\\rVert, no better than QLoRA\. With a callback, replace\_lora\_weights\_loftq can roll a module back to the zero initialisation, whose defect isee\.•*Quantise, split, re\-quantise*\(HF olora\_init on bitsandbytes weights\)\. The defect ise\+κ⁡\(r′\)−r′e\+\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}withr′=κ​W−δ0r^\{\\prime\}=\\kappa W\-\\delta\_\{0\}\.Quantise\-then\-split has defect exactlyeeiff the frozen part equalsκ​W−δ0\\kappa W\-\\delta\_\{0\}, for instanceκ​W\\kappa Wkept quantised minus a frozen copy of the rank\-rrδ0\\delta\_\{0\}, at a cost of aboutr⁡\(m\+n\)r\(m\+n\)extra numbers\. Re\-quantising the residual, as HF olora\_init does, generically addsκ⁡\(r′\)−r′\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}\.Action families whose pointing does not depend on the base \(OFT atR=IR=I, BOFT atR=IR=Iwith unit scale, HiRA atB=0B=0\), evaluated atκ​θ0\\kappa\\theta\_\{0\}, simply carry the defectee\. Methods whose pointing is computed from the base have an order ambiguity of their own\. DoRA’s magnitude initialised fromκ​θ0\\kappa\\theta\_\{0\}gives defectee, while initialised fromθ0\\theta\_\{0\}it givesdiag⁡\(∥w0,i∥/∥\(κ​θ0\)i∥\)​κ​θ0−θ0\.\\diag\\big\(\\lVert w\_\{0,i\}\\rVert/\\lVert\(\\kappa\\theta\_\{0\}\)\_\{i\}\\rVert\\big\)\\,\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\.For action families a base change also moves the absolute invariants, along with the base point\. NF4’s exact zero code enlarges the zero set that HiRA must preserve, and exactly orthogonal \(exact\-Cayley\) OFT preserves the Gram matrix ofκ​θ0\\kappa\\theta\_\{0\}rather than that ofθ0\\theta\_\{0\}; HF’s default Cayley–Neumann OFT preserves it only approximately\.\(c\) LoftQ alternates two updates on∥θ0−Q−L​R∥\\lVert\\theta\_\{0\}\-Q\-LR\\rVert, an exact rank\-rrSVD step and a round\-to\-nearest quantisation step with blockwise scales recomputed from its input\. The second is not an exact projection, so the objective need not decrease monotonically\. Withs=1s=1the output is pointed up to the attained residual\. It is then an*approximate section*, a choice\(Q,q0\)\(Q,q\_\{0\}\)with reduced∥ρ⁡\(q0\)−θ0∥\\lVert\\rho\(q\_\{0\}\)\-\\theta\_\{0\}\\rVert\. In general, withQTQ\_\{T\}and the low\-rank fitLT​RTL\_\{T\}R\_\{T\}afterTTiterations, HF’s deployed defect is\(QT\+LT​RT−W\)\+\(s−1\)​LT​RT,\(Q\_\{T\}\+L\_\{T\}R\_\{T\}\-W\)\+\(s\-1\)L\_\{T\}R\_\{T\},because HF’s internal residual also omitsss\. The official LoftQ checkpoints user=64r=64andα=16\\alpha=16, sos=1/4s=1/4, and then the deployed defect can grow withTTwhile the attained residual falls\.\(d\) For a generic high\-precision adapterΔ\\Delta, fine\-tune\-then\-quantise and quantise\-then\-fine\-tune do not commute\. The first lands on the quantisation grid\. In the second, the exactly merged weightκ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Deltaleaves the quantised type\. For a uniform fixed\-scale INT\-kkgrid,κ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Deltastays on the grid iffΔ\\Deltalies on the scale lattice \(within range\), so there the obstruction is the off\-grid update\. NF4 is non\-uniform and not closed under addition, and absmax grids whose scales are recomputed are not closed either, so for QLoRA’s NF4 the grid itself is part of the obstruction\. Type\-preserving merges, such as HF PEFT’s dequantise–add–requantise for bitsandbytes 4\- and 8\-bit layers, re\-quantise with recomputed block scales, so they stay in the quantised type but are lossy\. Under affine group quantisation QA\-LoRA is the structured exception\. Its group\-pooled adapter is constant on each quantisation group and merges into the zero\-points, so the merged model stays INT\-kk\.

##### Numerical check

On a256×256256\\times 256NF4 example whose weight has a dominant low\-rank part, withr=8r=8,s=1s=1and∥e∥=5\.65\\lVert e\\rVert=5\.65, QPiSSA’s defect is1\.451\.45, one\-step LoftQ’s is5\.225\.22, and the LoftQ objective rises in2727of6060half\-steps\. QPiSSA’s large advantage there relies on the dominant low\-rank part; on a Gaussian weight all three defects are close to∥e∥\\lVert e\\rVert\. Withs=2s=2the one\-step LoftQ defect equals∥e∥\\lVert e\\rVertexactly, and adding an NF4\-representable update to an NF4 weight leaves the NF4 grid in most entries \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### LoftQ as deployed

The official LoftQ checkpoints were made withr=64r=64,α=16\\alpha=16and five iterations, sos=1/4s=1/4\. In a referee’s simulation of that configuration on a Gaussian512×512512\\times 512NF4 weight, the attained residual was0\.67​∥e∥0\.67\\lVert e\\rVertand the deployed defect0\.96​∥e∥0\.96\\lVert e\\rVert\. ForT≥2T\\geq 2, HF’s internal residual also omitsss, so the objective in[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(c\) is not the defect HF starts from\. With a callback, replace\_lora\_weights\_loftq can roll a module back to the zero initialisation, and that module’s defect is thenee\.

##### Credit

Training LoRA over a 4\-bit NF4 base is due to QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\), building on the blockwise absmax quantisation of[Dettmers et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib91)\. Fitting the adapter to the quantisation error is LoftQ\([Li et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib255)\), and the split\-first variant is QPiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)\. The data\-aware and QR splits are CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\)and OLoRA\([Büyükakyüz, 2024](https://arxiv.org/html/2610.12728#bib.bib47)\)\. The group\-pooled adapter that merges into the zero\-points is QA\-LoRA\([Xu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib514)\)\. HiRA is due to[Huang et al\. \(2025b\)](https://arxiv.org/html/2610.12728#bib.bib182), and OFT to[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib381)\. The pipelines are as implemented in HF PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)\.

WHAT THIS EXPLAINSOver a quantised base, exactly orthogonal OFT keeps the neuron geometry ofκ​θ0\\kappa\\theta\_\{0\}\([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\) and HiRA keeps the zeros NF4 created \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(e\)\)\. QA\-LoRA makes merging commute with quantisation by living inside the quantiser’s own degrees of freedom\.

#### Restarting: what merge\-and\-restart can reach

Merge the adapter, restart at the merged point, repeat\. Each cycle adds one more copy of an additive shape or applies one more group element, so the long\-run reach is whatever one cycle*generates*\. A rank\-rrcone generates every matrix, and a subspace, subgroup or submonoid generates only itself\. Block\-orthogonal factors generate the rotations of each connected piece of their block hypergraph, because the bracket of infinitesimal rotations in the planes\(a,c\)\(a,c\)and\(c,b\)\(c,b\)is the infinitesimal rotation in the plane\(a,b\)\(a,b\)\.

Figure 10:Rebasing reach\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\. \(a\) The rank staircase form×n=24×16m\\times n=24\\times 16: the rank ofΔ​WK=∑k≤KBk​Ak\\Delta W\_\{K\}=\\sum\_\{k\\leq K\}B\_\{k\}A\_\{k\}afterKKmerge\-and\-restart cycles of ReLoRA, each adding a fresh random rank\-rrproduct, forr=3r=3\(circles\) andr=5r=5\(squares\)\. The measured ranks \(teal\) lie on the boundmin⁡\(K​r,m,n\)\\min\(Kr,m,n\)\(thick grey; dotted, the lineK​rKr\) at everyKK, so ReLoRA reaches full rank exactly atK∗=⌈16/r⌉K^\{\*\}=\\lceil 16/r\\rceilcycles, and not before \(corollary 1,[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(a\)\)\. LoRA\-XS with its frames frozen at the injective baseW0W\_\{0\}\(red,r=5r=5\) stays at rank55: an idempotent method gains nothing from rebasing \(corollary 2\)\. \(b\) Seven families of block\-orthogonal factors onn=16n=16coordinates: block OFT on a fixed partition with blocks of sizeb=4b=4andb=8b=8, BOFT withb=2b=2andm′=2,3,4m^\{\\prime\}=2,3,4butterfly factors, and Givens rotations in77planes and in1515planes forming a spanning tree\. Arcs are two\-element blocks and brackets larger blocks; dots are coloured by the connected component of the hypergraph the blocks generate, and hollow dots are isolated coordinates\. The light bars give the dimension∑C\(\|C\|2\)\\sum\_\{C\}\\binom\{\\lvert C\\rvert\}\{2\}of the generated Lie algebra \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(c\)\), and the dark bars the dimension reached in one cycle, againstdimSO⁡\(16\)=120\\dim\\mathrm\{SO\}\(16\)=120\(dashed\), with green bars for the families that reach it\. Only a connected hypergraph reaches120120; for BOFT that takes the full depthm′=4m^\{\\prime\}=4\(corollary 4\)\. The bar values are in §[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\.Theorem V\.3\(rebasing closure\)\.\(a\)*Additive methods with a fixed shapeC∋0C\\ni 0*, with frames, masks and compression operators frozen atθ0\\theta\_\{0\}across cycles\. ThenRK=θ0\+C\+KR\_\{K\}=\\theta\_\{0\}\+C^\{\+K\}, theKK\-fold Minkowski sum, andR∞=θ0\+Mon⁡\(C\)R\_\{\\infty\}=\\theta\_\{0\}\+\\operatorname\{Mon\}\(C\), the additive monoid generated byCC\([Background D\.10](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\)\. IfCCis closed under real scalings,R∞=θ0\+span⁡CR\_\{\\infty\}=\\theta\_\{0\}\+\\spanop C\.\(b\)*Action methods*, where one cycle applies an element of a setSSwith1∈S⊆G1\\in S\\subseteq G; for scaling methods,S⊆End⁡\(Θ\)S\\subseteq\\operatorname\{End\}\(\\Theta\)is a subset containing the identity\. ThenRK=SK​θ0R\_\{K\}=S^\{K\}\\theta\_\{0\}, andR∞R\_\{\\infty\}is the orbit of the monoid generated bySS\. SupposeSSlies in the subgroup generated by connected Lie subgroupsH1,…,HpH\_\{1\},\\dots,H\_\{p\}\([Background D\.8](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) and contains a neighbourhood of11in each\. Then the monoid generated bySSis the connected Lie subgroupHHwithLie⁡\(H\)=Lie⁡⟨𝔥1,…,𝔥p⟩\\operatorname\{Lie\}\(H\)=\\operatorname\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle\.\(c\)*Block\-orthogonal factors\.*Let the factors have coordinate blocksIj⊆\{1,…,n\}I\_\{j\}\\subseteq\\\{1,\\dots,n\\\}\(block OFT, BOFT factors, Givens planes\)\. ThenLie⁡⟨𝔰​𝔬​\(Ij\)⟩j=⨁C𝔰​𝔬​\(C\),\\operatorname\{Lie\}\\langle\\mathfrak\{so\}\(I\_\{j\}\)\\rangle\_\{j\}=\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\),summed over the connected componentsCCof the hypergraph\(\{1,…,n\},\{Ij\}\)\(\\\{1,\\dots,n\\\},\\\{I\_\{j\}\\\}\)\.*Corollaries\.*1\.ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)\.RK=W0\+ℳ≤min⁡\(K​r,m,n\)R\_\{K\}=W\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}\. It reaches full fine\-tuning iffK≥⌈min⁡\(m,n\)/r⌉K\\geq\\lceil\\min\(m,n\)/r\\rceil\.2\.Idempotent methodsgain nothing from rebasing\. These are the methods whose shape is exactly a subspace, subgroup or submonoid independent of the base, with frames, masks and compression operators frozen across cycles\. Examples are LoRA\-FA with a fixedAA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\), LoRA\-XS with frozen frames\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\), MoRA with fixed operators\([Jiang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib196)\), FourierFT with fixed indices\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\), WaveFT\([Bilican et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib35)\), C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\), MiSS\([Kang and Yin, 2024](https://arxiv.org/html/2610.12728#bib.bib206)\)\(default balance mode; bat mode is base\-dependent but still idempotent, since its blockwise mapsW↦\(W\+I\)​\(I\+M\)−IW\\mapsto\(W\+I\)\(I\+M\)\-Iform a monoid\), SHiRA\([Bhardwaj et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib30)\), BitFit\([Ben\-Zaken et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib27)\), FISH with a fixed mask\([Sung et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib438)\), LN\-tuning\([Zhao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib578)\), \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\), SSF\([Lian et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib259)\)and RoAd\([Liao and Monz, 2024](https://arxiv.org/html/2610.12728#bib.bib263)\)\. Block OFT with a fixed partition and an exact Cayley map gains only the measure\-zero set of block rotations with an eigenvalue−1\-1, soR∞=W0⋅∏SO⁡\(b\)=R1¯R\_\{\\infty\}=W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)=\\overline\{R\_\{1\}\}\. Constrained OFT \(COFT, anε\\varepsilon\-ball\) with an exact Cayley map gains up to∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\); HF’s coft=True keeps the Cayley–Neumann map by default and so falls under corollary 4\. Methods that recompute frames or masks from the merged weight \(an SVD\-recomputing LoRA\-XS, ReMoRA’s alternating operators, resampled indices\) do gain\.3\.VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\)\.VeRA∞=LoRA\-FA​\(A\)\\mathrm\{VeRA\}^\{\\infty\}=\\text\{LoRA\-FA\}\(A\)whenever the frozenBBhas no zero entries\.4\.OFT family\.Block OFT with an exact Cayley map is stuck inW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)\. HF’s default Cayley–Neumann factor isR=C⁡\(Q\)​\(I−Q4\)R=C\(Q\)\(I\-Q^\{4\}\), withCCthe exact Cayley map \([Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\), soR⊤​R=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}andRRis not orthogonal\. Its singular values are\|1−μk4\|\\lvert 1\-\\mu\_\{k\}^\{4\}\\rvert, where±i​μk\\pm i\\mu\_\{k\}are the eigenvalues ofQQ\. In the operating regime∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}\(everyμk≤21/4\\mu\_\{k\}\\leq 2^\{1/4\}\), the regime the truncated series is meant for, every factor is a contraction\. Re\-merging then decreasesKoutK\_\{\\rm out\}in the Loewner order and no singular value grows, so the drift is monotone shrinkage, and the reachable set lies inW0W\_\{0\}times the block contraction semigroup\. With unconstrainedQQthe factors include singular ones \(μk=1\\mu\_\{k\}=1\), and re\-merging generates∏\(GL\+​\(b\)∪ℳ≤b−2\)\\prod\\big\(\\mathrm\{GL\}^\{\+\}\(b\)\\cup\\mathcal\{M\}\_\{\\leq b\-2\}\\big\), or∏\(CO\+​\(2\)∪\{0\}\)\\prod\\big\(\\mathrm\{CO\}^\{\+\}\(2\)\\cup\\\{0\\\}\\big\)whenb=2b=2; this clause is established at the level of a proof sketch\. Iterated BOFT \(exact Cayley in HF\) withm′m^\{\\prime\}butterfly factors of block sizebbreachesdiag⁡\(ℝm\)⋅W0⋅P⁡\(∏SO⁡\(b​2m′−1\)\)​P⊤,\\diag\(\\mathbb\{R\}^\{m\}\)\\cdot W\_\{0\}\\cdot P\\Big\(\\prod\\mathrm\{SO\}\\big\(b\\,2^\{m^\{\\prime\}\-1\}\\big\)\\Big\)P^\{\\top\},because the factor hypergraph has components of sizeb​2m′−1b\\,2^\{m^\{\\prime\}\-1\}\. For injectiveW0W\_\{0\}this isdiag⁡\(ℝm\)⋅W0⋅SO⁡\(n\)\\diag\(\\mathbb\{R\}^\{m\}\)\\cdot W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)iffb​2m′−1=nb\\,2^\{m^\{\\prime\}\-1\}=n, the full\-depth butterflym′=1\+log2⁡\(n/b\)m^\{\\prime\}=1\+\\log\_\{2\}\(n/b\)\. HF’s default boft\_n\_butterfly\_factor=1 is block OFT timesdiag⁡\(s\)\\diag\(s\)\. A*single*full\-depth BOFT has dimension at mostm′​nb​\(b2\)m^\{\\prime\}\\tfrac\{n\}\{b\}\\binom\{b\}\{2\}, which is belowdimSO⁡\(n\)\\dim\\mathrm\{SO\}\(n\)whenm′​\(b−1\)<n−1m^\{\\prime\}\(b\-1\)<n\-1; its matrices are dense, while the set they form has positive codimension\.5\.Re\-HRA\(HF apply\_GS=False\), withW0W\_\{0\}injective andrreven\. ThenRK=\(W0\+ℳ≤K​r\)∩W0⋅SO⁡\(n\)R\_\{K\}=\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq Kr\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), so*rebasing commutes with the meet*of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\. This multiplicative twin of ReLoRA reachesW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)in at most⌈n/r⌉\\lceil n/r\\rceilcycles\.6\.GaLore \(with a gradient\-SVD or a random projector\), LISA and MeZO, under the optimizer hypotheses of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), are rebasing schemes of*linear*methods whose shape moves\. That is why, as a set over the sequences of shapes they can choose, they reach all ofΘ\\Theta, while a fixed linear shapeLLreaches onlyθ0\+L\\theta\_\{0\}\+L\(corollary 2\)\. Low rank is the per\-period obstruction only for GaLore; LISA and MeZO already make full\-rank per\-matrix updates and gain coverage\. Flora proper takes full\-space steps with a compressed momentum, so it has no per\-period low\-rank shape\.7\.Merging never rescues an exactly orthogonal method from its absolute invariants\.In exact arithmetic, every exactly orthogonal input\-side rebasing preservesKoutK\_\{\\rm out\}and the spectrum, and every output\-side one preservesKinK\_\{\\rm in\}and the spectrum \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\)\.

##### Numerical check

For the butterfly hypergraph atn=64n=64,b=4b=4, the components have sizes4,8,16,32,644,8,16,32,64form′=1,…,5m^\{\\prime\}=1,\\dots,5, so it is connected only atm′=5m^\{\\prime\}=5; an independent check with HF’s own permutation code gives the same sizes\. The Lie algebra generated by the Cayley–Neumann factors atb=4b=4has dimension16=dim𝔤​𝔩​\(4\)16=\\dim\\mathfrak\{gl\}\(4\)\. With skew entries of size about0\.050\.05,KoutK\_\{\\rm out\}moves by0\.2%0\.2\\%,1%1\\%,6%6\\%and22%22\\%\(relative\) after11,1010,5050and200200default\-OFT merges, decreasing in the Loewner order at every merge\. Random skewQQwith∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}never give∥R∥op\>1\\lVert R\\rVert\_\{\\rm op\}\>1, and atb=2b=2the generatorQ=\(0−110\)Q=\\left\(\\begin\{smallmatrix\}0&\-1\\\\ 1&0\\end\{smallmatrix\}\\right\)givesR=0R=0\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credit

Merge\-and\-restart training is ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)\. The orthogonal methods are OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\), with its constrained variant COFT, BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\), whose butterflies come from[Dao et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib88), and HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\), whose Householder products go back to[Mhammedi et al\. \(2016\)](https://arxiv.org/html/2610.12728#bib.bib326)\. Cayley parametrisations of orthogonal layers are studied in[Lezcano\-Casado and Martínez\-Rubio \(2019\)](https://arxiv.org/html/2610.12728#bib.bib244)\. The group theory is classical\. It uses Yamabe’s theorem on arcwise connected subgroups\([Yamabe, 1950](https://arxiv.org/html/2610.12728#bib.bib519)\)\([Background D\.9](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) and Scherk’s sharp form of the Cartan–Dieudonné theorem\([Scherk, 1950](https://arxiv.org/html/2610.12728#bib.bib399)\)\([Background D\.11](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\)\. That GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)is one\-sided LoRA re\-based every period is due to[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453), and the random\-projection case to Flora\([Hao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib154)\); see[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\. The other moving\-shape methods are LISA\([Pan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib360)\)and MeZO\([Malladi et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib310)\)\.

Table 17:What merge\-and\-restart reaches \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)and its corollaries\)\.WHAT THIS EXPLAINSReLoRA is[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)unrolled in time, since a sum ofKKrank\-rrLoRAs is one LoRA of rankK​rKr, so it walks the group its cone generates\. A fixed additive shape gains from restarts only when it is not closed under addition, as LoRA’s cone and VeRA’s shape are not\. Exact\-Cayley block OFT can never escape itsn/bn/bpartial\-Gram invariants however often it is re\-merged\. Butterfly permutations exist precisely to connect the block hypergraph, and they connect it fully only at full depth\.

Remark\(2026: POET\-X\)\. POET\-X\([Qiu et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib383)\), a pretraining method whoseW0W\_\{0\}is a random initialisation, changes the partition between cycles, on both sides of the weight\. Each round trains a block\-diagonal factor conjugated by a random permutation on each side, merges both into the weight and redraws the permutations\. With exact Cayley blocks every merged weight lies inSO⁡\(m\)​W0​SO​\(n\)\\mathrm\{SO\}\(m\)\\,W\_\{0\}\\,\\mathrm\{SO\}\(n\), so it keeps the singular values ofW0W\_\{0\}\([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\), and by[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\) and \(c\) blocks that together connect all coordinates of a side generate all ofSO\\mathrm\{SO\}on that side\. Its blocks are in fact\(I\+Q\)​\(I\+Q\+Q2\+Q3\)\(I\+Q\)\(I\+Q\+Q^\{2\}\+Q^\{3\}\), the Cayley–Neumann factor of corollary 4, so while everyμk≤21/4\\mu\_\{k\}\\leq 2^\{1/4\}no merge raises a singular value, and the spectrum can only drift down\.

Example\(one projection of a 7B model\)\. Take a4096×40964096\\times 4096projection\. ReLoRA withr=16r=16reaches full fine\-tuning after256256cycles and not before\. For injectiveW0W\_\{0\}, re\-HRA withr=8r=8reachesW0⋅SO⁡\(4096\)W\_\{0\}\\cdot\\mathrm\{SO\}\(4096\)within512512cycles\. BOFT withb=4b=4needsm′=11m^\{\\prime\}=11butterfly factors before re\-merging reaches that orbit up to output scaling; at HF’s default of one factor it stays inW0⋅∏SO⁡\(4\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(4\)up to the scale\. A single 11\-factor BOFT has dimension at most11⋅1024⋅6=67,58411\\cdot 1024\\cdot 6=67\{,\}584, againstdimSO⁡\(4096\)=8,386,560\\dim\\mathrm\{SO\}\(4096\)=8\{,\}386\{,\}560\.

##### Prediction

With an exact Cayley map and a fixed partition, re\-merging adds nothing to block OFT’s reach beyond closure, so re\-merged exact\-Cayley block OFT and COFT on a fixed partition never leaveW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)\. Enlarging the reach needs a connected block hypergraph \(partitions that change between cycles, or full\-depth butterflies\) or a COFT ball\. LoRA\-type cones gain from rebasing with a fixed shape, while linear additive shapes do not \(corollary 2\)\. These are statements about reach; whether extra reach improves a trained model is the experiment \([Prediction X\.6](https://arxiv.org/html/2610.12728#S10.SS0.SSSx5)\)\.

#### Combining: rules that must ignore the gauge

A LoRA module ships as factors\(B,A\)\(B,A\), the model sees onlyB​ABA, and\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)gives the same product for everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\. A combination rule should return the same model for every choice of factors, or two people with the same adapters can merge them into different models\. Task arithmetic adds the products and passes\. LoraHub adds theBB’s, adds theAA’s and multiplies, and fails, because its cross termsBi​AlB\_\{i\}A\_\{l\}see a rescaling\(c​B,A/c\)\(cB,A/c\)of one module\. The merge panel of Figure[9](https://arxiv.org/html/2610.12728#S3.F9)shows the LoraHub merge moving under that rescaling while task arithmetic stays fixed\.

Proposition V\.4\(combination rules: gauge and naturality\)\.\(a\) A ruleΦ⁡\(\(Bi,Ai\)i\)\\Phi\\big\(\(B\_\{i\},A\_\{i\}\)\_\{i\}\\big\)for combining LoRA factors descends to a function of the adapters\(Bi​Ai\)i\(B\_\{i\}A\_\{i\}\)\_\{i\}iff it is constant on the fibres of\(Bi,Ai\)↦\(Bi​Ai\)\(B\_\{i\},A\_\{i\}\)\\mapsto\(B\_\{i\}A\_\{i\}\)\. On the full\-rank locus this is equivalent to∏iGLr\\prod\_\{i\}\\mathrm\{GL\}\_\{r\}\-invariance\. It is also equivalent forΦ\\Phicontinuous on the whole factor space; this clause rests on the unique closed orbit in each fibre, which lies in the closure of every orbit of the fibre \([Background D\.31](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px2)\)\.•Task arithmetic∑iwi​si​Bi​Ai\\sum\_\{i\}w\_\{i\}s\_\{i\}B\_\{i\}A\_\{i\}, withsis\_\{i\}HF’s scaling \(αi/ri\\alpha\_\{i\}/r\_\{i\}, orαi/ri\\alpha\_\{i\}/\\sqrt\{r\_\{i\}\}under rsLoRA\), is invariant and lands inLoRA∑iri\\mathrm\{LoRA\}\_\{\\sum\_\{i\}r\_\{i\}\}\. It is HF’s cat with merged scale11\([Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\)\.•LoraHub’s factor\-wise rule\(∑iwi​Bi\)​\(∑jwj​Aj\)\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)\\big\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\\big\)is not invariant\. Replacing\(B2,A2\)\(B\_\{2\},A\_\{2\}\)by\(c​B2,A2/c\)\(cB\_\{2\},A\_\{2\}/c\)changes it by∑l≠2wl​w2​\[\(c−1\)​B2​Al\+\(c−1−1\)​Bl​A2\]\.\\sum\_\{l\\neq 2\}w\_\{l\}w\_\{2\}\\big\[\(c\-1\)B\_\{2\}A\_\{l\}\+\(c^\{\-1\}\-1\)B\_\{l\}A\_\{2\}\\big\]\.HF’s add\_weighted\_adapter\(…, combination\_type="linear"\) is a factor\-wise rule of the same type\. It weightsAiA\_\{i\}bysign⁡\(wi\)​\|wi​si\|\\operatorname\{sign\}\(w\_\{i\}\)\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}andBiB\_\{i\}by\|wi​si\|\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}and sums the factors separately, so it has cross terms\. On adapters that share a frozenAAit is still not exact up to a global factor, since the relative weights become\|wi​si\|\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}\. HF’s non\-SVD ties, dare\_\* and magnitude\_prune variants also act on factors; the \*\_svd variants act on full updates\.•Poly and MHR are different\. Their skill inventories are trained jointly through the product\-of\-sums map, which is invariant under the diagonalGLr\\mathrm\{GL\}\_\{r\}, one commonggfor every skill\. Numerically, for fixed generic routing with fewer skills than tasks and full\-rank factors, this is the whole continuous fibre\. With at least as many skills as tasks, or once the routing is counted as a parameter, there are further symmetries \(skill permutations, and re\-mixingsα↦α​M\\alpha\\mapsto\\alpha MwithM​𝟏=𝟏M\\mathbf\{1\}=\\mathbf\{1\}\)\. Their cross terms are a design choice of a jointly trained parametrisation, and raise no question of well\-definedness\.\(b\)*Well\-definedness versus exactness\.*A joint gauge fixing makes a factor\-wise rule well defined\. For rank\-preserving product\-of\-sums rules, exactness \(agreement with task arithmetic for all weightsww\) needs degree one inww, which for injective factors forces a shared factor; cat, which concatenates and raises the rank, is exact without one\. A frozen shared factor \(FFA\-LoRA; LoRA\-FA\) gives\(∑iwi​Bi\)​A=∑iwi​Bi​A\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)A=\\sum\_\{i\}w\_\{i\}B\_\{i\}Aexactly\. LoraHub with a sharedAAstill returns\(∑jwj\)​∑iwi​Bi​A\\big\(\\sum\_\{j\}w\_\{j\}\\big\)\\sum\_\{i\}w\_\{i\}B\_\{i\}A, exact only up to the global factor∑jwj\\sum\_\{j\}w\_\{j\}, since its weights are unnormalised\. A per\-module SVD canonical form leaves a\(ℤ/2\)r\(\\mathbb\{Z\}/2\)^\{r\}of sign flips per module, plus rotations within repeated singular values, and these change the cross terms\. What is needed is a*joint*frame that produces a shared factor, as KnOTS’s joint re\-factorisation of the concatenated updates through a sharedU​ΣU\\Sigmadoes\. Federated LoRA shows the difference\. FedAvg\-LoRA clients start each round from the same broadcast\(B,A\)\(B,A\), so their failure,12​\(B1\+B2\)⋅12​\(A1\+A2\)≠12​\(B1​A1\+B2​A2\),\\tfrac\{1\}\{2\}\(B\_\{1\}\+B\_\{2\}\)\\cdot\\tfrac\{1\}\{2\}\(A\_\{1\}\+A\_\{2\}\)\\neq\\tfrac\{1\}\{2\}\(B\_\{1\}A\_\{1\}\+B\_\{2\}A\_\{2\}\),is bilinearity, and it persists in a fully fixed gauge\.\(c\)*Naturality\.*On updates withNNcoordinates, writeTNT\_\{N\},BNB\_\{N\}andMonN\\mathrm\{Mon\}\_\{N\}for the invertible diagonal, signed permutation and monomial matrices\. Task arithmetic isGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-equivariant on updates\. DARE is coordinatewise and magnitude\-free,D⁡\(m⊙τ\)=m⊙\(D​τ\)D\(m\\odot\\tau\)=m\\odot\(D\\tau\)for every invertible diagonalDDand the same maskmm\. So DARE is exactly equivariant under the torusTNT\_\{N\}, sign flips included, for each mask, and under permutations in distribution; its distributional group is exactlyMonN\\mathrm\{Mon\}\_\{N\}\. TIES with global trimming \(top\-kkmagnitude, sign election by total mass as in the TIES paper and HF’s default "total", disjoint mean\) is equivariant underℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}, homotheties times signed permutations, generically \(off magnitude ties at the threshold\)\. With HF’s majority\_sign\_method="frequency", count ties are broken to\+1\+1and the group drops toℝ\>0⋅SN\\mathbb\{R\}\_\{\>0\}\\cdot S\_\{N\}\. Per\-tensor trimming changes the group to∏b\(ℝ×⋅BNb\)\\prod\_\{b\}\\big\(\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\_\{b\}\}\\big\), extended by permutations of equal\-size blocks\. DARE followed by full TIES \(with trimming\) isℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}\-equivariant in distribution; HF’s and MergeKit’s dare\_ties has no magnitude trim, so with "total" election it is exactlyTNT\_\{N\}\-equivariant per mask, and its group containsMonN\\mathrm\{Mon\}\_\{N\}\. None of the sparsifying rules \(DARE, TIES, DARE\-TIES\) isO⁡\(N\)\\mathrm\{O\}\(N\)\-equivariant\. Applied to factors and summed across adapters \(HF’s non\-SVD variants\), all of them are gauge\-dependent, and under independent per\-adapter gauges even DARE fails, through the cross terms of \(a\)\. DARE’s masking commutes withTrT\_\{r\}exactly for each mask and withMonr\\mathrm\{Mon\}\_\{r\}in distribution, so a single masked adapter, or a factor\-wise sum under one gauge shared by all adapters, isTrT\_\{r\}\-invariant per mask andMonr\\mathrm\{Mon\}\_\{r\}\-invariant in distribution\.

##### Numerical check

DARE with a random non\-unit diagonalDDis equivariant to2×10−152\\times 10^\{\-15\}per mask\. TIES is homogeneous to9×10−169\\times 10^\{\-16\}and fails by5\.65\.6under a generic positive diagonal\. The general LoraHub change formula matches with three modules to2×10−152\\times 10^\{\-15\}\. On the factors of two adapters, a diagonal gauge on one of them moves factor\-wise DARE by an amount of order one, while the same gauge shared by all adapters leaves it fixed to10−1510^\{\-15\}\. For Poly with33skills,r=2r=2and6×66\\times 6weights, the fibre has dimension3232with22tasks and4=dimGL24=\\dim\\mathrm\{GL\}\_\{2\}with44tasks \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### HF options

With HF’s majority\_sign\_method="frequency", sign counts are integers, so ties are common, and they are broken towards\+1\+1\. Sign flips and negative homotheties therefore fail for ties and dare\_ties alike\.

##### Credit

The rules studied here are task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\), LoraHub\([Huang et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib180)\), TIES\([Yadav et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib517)\), DARE\([Yu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib538)\)and KnOTS\([Stoica et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib429)\), whose joint SVD is the shared frame of \(b\); dare\_ties is as implemented in HF PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)and MergeKit\([Goddard et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib131)\)\. Jointly trained skill inventories are Poly\([Ponti et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib372)\)and MHR\([Caccia et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib48)\)\. Federated averaging of LoRA factors is used in FedIT\([Zhang et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib552)\), and the frozen shared factor that removes its bilinearity is FFA\-LoRA\([Sun et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib437)\)\. That functions of LoRA weights should respect theirGLr\\mathrm\{GL\}\_\{r\}symmetry is the starting point of theGL\\mathrm\{GL\}\-equivariant networks of[Putterman et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib375); for full models, hidden\-unit permutations play that role in Git Re\-Basin\([Ainsworth et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib6)\)\. The unique closed orbit in each fibre is classical invariant theory\.[Kempf and Ness \(1979\)](https://arxiv.org/html/2610.12728#bib.bib209)characterise closed orbits by norm\-minimising points \([Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\), here the balanced factorisations of[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), and[Richardson and Slodowy \(1990\)](https://arxiv.org/html/2610.12728#bib.bib389)treat real groups\. For optimizers, the same question leads to[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546)and[Yen et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib533); see[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\.

WHAT THIS EXPLAINSPost\-hoc factor merging of independently trained modules, as in LoraHub and HF’s linear, is not a function of the adapters\. It depends on how each module happened to be factorised, which nobody controls after training\. Federated LoRA fails through bilinearity instead, and FFA\-LoRA’s frozen shared factor repairs it\. KnOTS’s joint re\-factorisation produces a shared factor\.Design rule: combine LoRAs on products, or only after a joint re\-factorisation that produces a shared factor\.

##### Prediction

LoraHub behaves like update arithmetic, up to the factor∑jwj\\sum\_\{j\}w\_\{j\}, when its modules share theAA\-seed andAAstays close to its initialisation\. That regime is empirical and favoured by a large ratioηB/ηA\\eta\_\{B\}/\\eta\_\{A\}, as in LoRA\+\([Hayou et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib159)\)\. LoraHub degrades when the modules come from independent seeds \([Prediction X\.9](https://arxiv.org/html/2610.12728#S10.SS0.SSSx5.Px1)\)\.

#### What base change leaves for the network

Every move here stayed inside weight space, where merging into the base is the canonical arrow to the terminal object of[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)and always exists\. Adapters, prefixes and ReFT change the network itself, and for them merging is a lifting problem whose obstructions the activation function decides\. That is Exposé[VI](https://arxiv.org/html/2610.12728#S6)\.

## EXPOSÉ VIExtensions and mergeability

When a lift fails, look for the place where it fails, and give the obstruction a name\. after Grothendieck

Adapters, prefixes, prompts and ReFT enlarge the network before they train, so merging them back is a lifting problem\. Seven local rewrite rules give sufficient conditions for a merge and name the side condition that fails when they stop\.

Merging an extension back means finding a weight of the original network that computes the trained function\. The rules slide an edit along the wires of a block until a weight absorbs it\. They certify a merge when the edit acts alike at every position, the absorbing weights are unshared, shifts have bias slots and, under RoPE, scales are constant on rotary pairs\. A stuck rewrite shows only a box\-local obstruction\. Three examples show what the exposé finds\. The feed\-forward scale of \(IA\)3always folds intoWdownW\_\{\\mathrm\{down\}\}; it folds intoWupW\_\{\\mathrm\{up\}\}under SwiGLU for every scale and under ReLU for nonnegative ones, and under GELU only entries00and11fold\. A prefix is a gated parallel adapter whose gate depends on the input, so for generic frozen weights it cannot be folded into the head’s own weights\. Per hidden vector, LoReFT is LoRA on an inserted identity box, and since it acts at selected positions no merge is known\.

#### The weight that absorbs the edit

\(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)rescales the hidden units of a feed\-forward block by a learned vectorℓ\\ell, which one would like to fold into a frozen weight after training\. The weight that reads the hidden units,WdownW\_\{\\mathrm\{down\}\}, absorbsℓ\\ellunder every activation\. The weight that produces them,WupW\_\{\\mathrm\{up\}\}, absorbs it under SwiGLU for everyℓ\\ell, under ReLU when no entry is negative, and under GELU only for entries00or11\.

The activation belongs to the network, which a weight\-space method never sees \([Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\)\. Adapters, prefixes, prompts and ReFT add boxes the pretrained network lacked, and whether they can be removed depends on their surroundings\. Figure[11](https://arxiv.org/html/2610.12728#S6.F11)runs the rules of this exposé on one block\.

hhnnγ⊙\\gamma\\odotWQW\_\{Q\}WKW\_\{K\}WVW\_\{V\}ℓ\\ellℓv\\ell\_\{v\}RmR\_\{m\}RnR\_\{n\}⟨,⟩\\langle\\,,\\rangle∑jaj​vj\\sum\_\{j\}a\_\{j\}v\_\{j\}WOW\_\{O\}s,ts,t\+\+Φ\\PhiPPhhnnγ⊙\\gamma\\odotWgW\_\{g\}WupW\_\{\\mathrm\{up\}\}σ\\sigma⊙\\odotℓ\\ellWdownW\_\{\\mathrm\{down\}\}id\+Uσ\(D⋅\)\\mathrm\{id\}\+U\\sigma\(D\\,\\cdot\)\+\+σ=silu\\sigma=\\mathrm\{silu\}ℓ≥0\\ell\\geq 0ℓi∈\{0,1\}\\ell\_\{i\}\\in\\\{0,1\\\}Figure 11:The merge rules of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)on a pre\-LN transformer block, drawn as a string diagram with the residual stream along the bottom: \(a\) the attention sublayer under RoPE, \(b\) a SwiGLU feed\-forward sublayer\. Tide boxes are trainable boxes of the base network,∙\\bulletis a copy node,⊙\\odotthe product node of the gated block, and ochre diamonds are edits inserted by a method\. Moss arrows are rewrites licensed for every parameter value, ochre arrows rewrites licensed under a side condition, and seal marks what the rules do not merge\. A query scalingℓ\\ellfuses intoWQW\_\{Q\}by R1; R5 moves it ontoWKW\_\{K\}only ifℓ\\ellis constant on each rotary pair and across a grouped\-query sharing group\. A value scalingℓv\\ell\_\{v\}fuses intoWVW\_\{V\}\. A norm gain is copied to every reader \(R7\) and absorbed \(R6\)\. SSF’s scale and shift\(s,t\)\(s,t\)fuse intoWOW\_\{O\}only ifWOW\_\{O\}has a bias slot\. The \(IA\)3scale fuses intoWdownW\_\{\\mathrm\{down\}\}\(R1\), and through the product node intoWupW\_\{\\mathrm\{up\}\}for everyℓ\\ell\(R4, then R1\); it does not slide through the SiLU gate, and in an ungated block R3 carries it through a ReLU only forℓ≥0\\ell\\geq 0\. A prefix enters the softmax with an input\-dependent gate \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\), a nonlinear adapter is a non\-affine edit \([Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(c\)\), and LoReFT acts at selected positions of the residual stream \([Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)\(c\)\); R1–R7 merge none of the three\. The companion site’s merge game runs these rewrites interactively, with a float64 check of each licensed merge on seeded weights\.
#### Extensions, and merging as a lift

An adaptery↦y\+U​σ​\(D​y\)y\\mapsto y\+U\\sigma\(Dy\)\([Houlsby et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib175)\)adds weights\(U,D\)\(U,D\), and atU=0U=0the network computes the old function\. A prefix\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\)adds key–value pairs to every softmax, and ReFT\([Wu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib505)\)edits hidden states at chosen positions\. Each timeΘ\\Thetagrows toΘ×Θ′\\Theta\\times\\Theta^\{\\prime\}andffto a largerf~\\tilde\{f\}\. A prefix always takes softmax mass, and on a head withWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0no prefix leaves the output unchanged, so prefixes have no neutral setting there \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\)\.

Definition VI\.1\(extensions; mergeability\)\.•Anextensionis a function\-preserving pointed embeddingι:\(Θ,f,θ0\)→\(Θ×Θ′,f~,\(θ0,θ0′\)\)\\iota:\(\\Theta,f,\\theta\_\{0\}\)\\to\(\\Theta\\times\\Theta^\{\\prime\},\\tilde\{f\},\(\\theta\_\{0\},\\theta\_\{0\}^\{\\prime\}\)\),ι⁡\(θ\)=\(θ,θ0′\)\\iota\(\\theta\)=\(\\theta,\\theta\_\{0\}^\{\\prime\}\), withf~\(θ,θ0′,−\)=f\(θ,−\)\\tilde\{f\}\(\\theta,\\theta\_\{0\}^\{\\prime\},\-\)=f\(\\theta,\-\)\. The pointθ0′\\theta\_\{0\}^\{\\prime\}is theneutral point\(“identity at initialisation”\)\.•Anunpointed extensionhas no neutral point\.•A method on an extension ismergeableif there isμ:Q→Θ\\mu:Q\\to\\ThetawithΦf~∘ρ=Φf∘μ\\Phi\_\{\\tilde\{f\}\}\\circ\\rho=\\Phi\_\{f\}\\circ\\mu: a lift along the realisation\.•It isbox\-locally mergeableif the modified box function stays in the box’s function class: linear maps for a linear box, affine maps when the box has a bias slot\.[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)gives sufficient conditions, with explicit side conditions \(position\-uniformity, no weight sharing, bias slots\), under which this yields a merge into the base network\.

A merge is an arrowμ\\muthat makes the square of realisations commute:

Q\{\\lx@inpgf@ignorespaces Q\}Θ\{\\lx@inpgf@ignorespaces\\Theta\}Θ×Θ′\{\\lx@inpgf@ignorespaces\\Theta\\times\\Theta^\{\\prime\}\}C∞​\(X,Y\)\{\\lx@inpgf@ignorespaces C^\{\\infty\}\(X,Y\)\}μ\\scriptstyle\{\\lx@inpgf@ignorespaces\\mu\}ρ\\scriptstyle\{\\lx@inpgf@ignorespaces\\rho\}Φf\\scriptstyle\{\\lx@inpgf@ignorespaces\\Phi\_\{f\}\}Φf~\\scriptstyle\{\\lx@inpgf@ignorespaces\\Phi\_\{\\tilde\{f\}\}\}A weight\-space method merges byμ=ρ\\mu=\\rho, the canonical arrow to full fine\-tuning \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\)\), so the real questions live on extensions, in the slice of𝐏𝐚𝐫𝐚\\mathbf\{Para\}over the pretrained model \([Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\)\. Box\-local mergeability can be read off formulas, and the classification of Exposé[VIII](https://arxiv.org/html/2610.12728#S8)records it as a merge type:

- •𝖬𝟣\\mathsf\{M1\}: the edit merges into its own box;
- •𝖬𝟣→\\mathsf\{M1\}\{\\rightarrow\}: the rules move it into a neighbouring box, under side conditions;
- •𝖬𝗊\\mathsf\{Mq\}: an exact merge exists but leaves the quantised type;
- •𝖬​∞\\mathsf\{M\}\\infty: no merge; the rules are blocked box\-locally, or the edit lies outside them\.

WHAT THIS EXPLAINS“Zero inference latency” runs two questions together: whether the original architecture can compute the trained function at all, and whether a local edit finds the weight\. A zero up\-projection, or the zero gate of LLaMA\-Adapter, makes training start at the pretrained function, and by the function\-level form of[Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)onlyUU, or the gate, moves at first\.

Adapters are due to[Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib175), prefix tuning to[Li and Liang \(2021\)](https://arxiv.org/html/2610.12728#bib.bib252), prompt tuning to[Lester et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib243)and ReFT to[Wu et al\. \(2024d\)](https://arxiv.org/html/2610.12728#bib.bib505)\.[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)first placed adapters, prefixes and LoRA in one frame, as modifications of hidden states\. The construction𝐏𝐚𝐫𝐚\\mathbf\{Para\}is from[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\([Background B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

#### The activation fixes the privileged basis

An edit moves through a network by sliding along wires, and a linear map slides through a box when it commutes with the box\. Activations are the bottleneck\. Permutations commute with every elementwiseσ\\sigma, and for ReLU so do positive rescalings, sinceReLU⁡\(c​t\)=c​ReLU​\(t\)\\mathrm\{ReLU\}\(ct\)=c\\,\\mathrm\{ReLU\}\(t\)forc\>0c\>0\. The lemma says how little else does\.

Lemma VI\.2\(the activation fixes the privileged basis; classical,[Sussmann \(1992\)](https://arxiv.org/html/2610.12728#bib.bib440),[Chen et al\. \(1993\)](https://arxiv.org/html/2610.12728#bib.bib58),[Godfrey et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib132)\)\.Letσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}be an elementwise activation that is not affine, and letL∈GLdL\\in\\mathrm\{GL\}\_\{d\}commute with it as maps onℝd\\mathbb\{R\}^\{d\}:L​σ×d=σ×d​LL\\,\\sigma^\{\\times d\}=\\sigma^\{\\times d\}L\.\(a\) Ifσ∈C2\\sigma\\in C^\{2\}, thenL=P​DL=PDis monomial\.\(b\) Ifσ\\sigmais ReLU or leaky ReLU with slopeα≠1\\alpha\\neq 1, thenL=P​DL=PDwithDDa positive diagonal\.\(c\) If moreoverσ\\sigmais real\-analytic at00\([Background C\.22](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\) withσ⁡\(t\)=∑ak​tk\\sigma\(t\)=\\sum a\_\{k\}t^\{k\}, each diagonal entryccofDDsatisfiesc​ak=ck​akca\_\{k\}=c^\{k\}a\_\{k\}for allkk\. Soc=1c=1for GELU and SiLU, sinceσ′′​\(0\)≠0\\sigma^\{\\prime\\prime\}\(0\)\\neq 0, andc=±1c=\\pm 1for tanh\.

The lemma is classical\.[Sussmann \(1992\)](https://arxiv.org/html/2610.12728#bib.bib440)and[Chen et al\. \(1993\)](https://arxiv.org/html/2610.12728#bib.bib58)found the weight\-space symmetries of tanh networks, and[Godfrey et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib132)computed the linear maps that commute with each common activation\. Permutation symmetry is what[Ainsworth et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib6)use to align whole models before merging them\.

So rules R2 and R3 below exhaust the invertible linear maps that slide throughσ\\sigma: permutations for GELU and SiLU, and positive monomials for ReLU\. In gated blocks such as SwiGLU every diagonal slides through the linear branch \(rule R4\), so their hidden\-unit group is the full monomial groupMondff\\mathrm\{Mon\}\_\{d\_\{\\mathrm\{ff\}\}\}\.

WHAT THIS EXPLAINSWithin the lemma’s scope \(global linear intertwiners of an ungated elementwiseσ\\sigma\), the symmetries a method may ignore are*derived*fromσ\\sigma\. For ungated FFNs \(BERT, GPT\-2, ViT\), GELU and SiLU giveSdS\_\{d\}, tanh givesBdB\_\{d\}, and ReLU gives positive monomials\. Gated FFNs \(SwiGLU and GeGLU, as in LLaMA, Mistral, Qwen and Gemma\) are not elementwise, and their hidden\-unit group is the fullMondff\\mathrm\{Mon\}\_\{d\_\{\\mathrm\{ff\}\}\}, acting through the linear branch \(R4 of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\)\. Affine, bias\-absorbing symmetries can be larger still: sigmoid satisfiesσ⁡\(−z\)=1−σ⁡\(z\)\\sigma\(\-z\)=1\-\\sigma\(z\)\. The privileged basis of LayerNorm gains comes from the diagonal gain parametrisation, andσ\\sigmaplays no part in it\. A gain\-free RMSNorm isO⁡\(d\)\\mathrm\{O\}\(d\)\-equivariant, which SliceGPT\([Ashkboos et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib18)\)and QuaRot\([Ashkboos et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib19)\)use to rotate the residual stream \([Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\)\.

PREDICTIONOver random function\-preserving permutations of hidden units, LoRA’s outcome variance equals its seed variance, since the pretrained order is exchangeable\. FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\), LoKr\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\)and MoRA\([Jiang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib196)\)show larger variance, and so do BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\)and OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)on modules whose input side is the hidden layer \(down\_proj\)\. On up\_proj and gate\_proj a hidden\-unit permutation only reorders BOFT’s output rows, so BOFT is exactly equivariant there\.[Prediction X\.3](https://arxiv.org/html/2610.12728#S10.SS0.SSSx4)turns this into a permutation test\.

#### Seven local rules

Draw a transformer block as a string diagram \([Background B\.3](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\), with a copy node where a wire feeds several boxes and a product node in a gated block\. An edit is a small box on a wire\. To merge, slide it by equalities of maps until a trainable box absorbs it; where it gets stuck names the failing side condition\. A positional rotation \(RoPE,[Su et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib432)\), a shared weight, a missing bias slot or a second reader of a wire can each block a slide, and all four occur in production models\.

Analogy\(what “string diagram” and “spider” mean here\)\. The diagrams are string diagrams of a symmetric monoidal category, read literally\. Copy is the cartesian diagonalx↦\(x,x\)x\\mapsto\(x,x\)\([Background B\.4](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\), which commutes with every map, and the product node is⊙\\odot\. “Spider” is only a name for both; they fail the Frobenius law \([Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\), which is never used\.

String diagrams for monoidal categories go back to[Joyal and Street \(1991\)](https://arxiv.org/html/2610.12728#bib.bib202); spiders are from[Coecke et al\. \(2013\)](https://arxiv.org/html/2610.12728#bib.bib81)\.

Theorem VI\.3\(the merge calculus; known in part\)\.Work with activation\-side diagrams in a symmetric monoidal category of continuous, piecewise\-smooth maps with copy and discard nodes\. \(We call copy and the Hadamard product “spiders” by analogy only; in this cartesian setting they do not satisfy the Frobenius law\.\) A*trainable box*is a node whose weight ranges over a stated class: all linear maps, or all affine maps when the box has a bias slot\. R2 and R4–R7 are identities of maps; R1 is a closure property and R3 a characterisation\.Consequences:\(a\)Affine edits\.Let a method insert a position\-uniform affine editEqE\_\{q\}on a wirewwof the base networkff, for everyq∈Qq\\in Q\. Suppose every reader ofwwis a trainable, unshared box \(no weight tying\) whose class is closed under precomposition withEqE\_\{q\}; a shift needs a bias slot\. Then the method merges, byWc↦Wc​EqW\_\{c\}\\mapsto W\_\{c\}E\_\{q\}for all readersccat once\. Dually, it merges into an untied trainable box that writeswwwheneverEqE\_\{q\}composed with that box stays in its class\. Absorption into a box that the extension itself inserts does not count\. Edits that violate these hypotheses fall outside \(a\): on the reader side, R1–R7 do not merge position\-selective edits or edits on a wire that is also read by a skip connection, a norm or the network output\. Whole\-network non\-mergeability is open \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\. Position\-selective edits \(ReFT/LoReFT, and position\-gated LoRA variants such as Activated LoRA\) are not box\-locally mergeable\. Residual\-stream edits need case analysis: an edit merges into the box that produces the wire when that box can absorb it \(an untied embedding table; for a fixed\-position edit at layer00, a learned absolute positional table\), a shift folds into a writer’s bias when one exists, a diagonal edit after a post\-LN folds into that LN’s affine parameters, and an orthogonal edit in an RMSNorm model merges globally, SliceGPT/QuaRot\-style\. No merge is known for LoReFT’s non\-orthogonal low\-rank edit\.\(b\)Diagonal methods\.A diagonal method merges into a box reached by R1–R7 if all of the following hold: the rules move*every*copy ofdiag⁡\(ℓ\)\\diag\(\\ell\)into trainable boxes for allℓ\\ellin the method’s parameter domain \(withℓ≥0\\ell\\geq 0when R3 is used\); no position\-dependent operator lies on the path \(for RoPE,ℓ\\ellis constant on each rotary pair\); the target box is unshared, orℓ\\ellis constant across the sharing group; and every shift has a bias slot\. Conversely, the RoPE condition is necessary for sliding a query scaling ontoWKW\_\{K\}whenWQW\_\{Q\}andWKW\_\{K\}have full rank and at least two occurring relative offsets have rotation angles that are not multiples ofπ\\pi\. Without such genericity hypotheses the conditions are not necessary: a dead rotary pair, or zero entries ofℓ\\ellunder ReLU, can still merge\.\(c\)Non\-affine edits\.For a linear or affine box, an edit whose modified box function is non\-affine in the box input, for some parameter value, is not box\-locally mergeable\. For linear and affine boxes this is the definition; for non\-linear boxes such as attention, see[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\.\(d\)Type relativity\.A generic high\-precision update is off the quantisation grid, so exact merging of a QLoRA or LoftQ update leaves the quantised type, a type\-preserving merge is lossy, and QA\-LoRA’s group\-constant adapters are the exception \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(d\)\)\.

The rules are elementary, and the individual merges are known: \(IA\)3by[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275)\(Sec\. 3\.3\), SSF by re\-parameterisation in[Lian et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib259), and LoRA by linearity in[Hu et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib177)\. Moving scales through ReLU, SwiGLU’s linear branch and norm gains, and the GELU obstruction, are in SmoothQuant\([Xiao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib508)\)and AWQ\([Lin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib268)\)\. New here is only the packaging as one merge calculus for PEFT, with explicit side conditions\. QLoRA is[Dettmers et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib92), LoftQ[Li et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib255), QA\-LoRA[Xu et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib514)and Activated LoRA[Greenewald et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib136)\.[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4),[Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)and[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)were repaired after two rounds of adversarial review; the referee record \(§[XI](https://arxiv.org/html/2610.12728#S11.SS0.SSSx4)\) lists what changed\.

NUMUnder RoPE,⟨Rm​M​q,Rn​k⟩=⟨q,M⊤​Rn−m​k⟩\\langle R\_\{m\}Mq,R\_\{n\}k\\rangle=\\langle q,M^\{\\top\}R\_\{n\-m\}k\\rangleholds for a genericMMto10−1510^\{\-15\}while the slide onto the key side fails\. A query scaling slid ontoWKW\_\{K\}leaves a7\.6%7\.6\\%least\-squares residual unlessℓ\\ellis pair\-constant\. Folding post\-LN gains intoW1W\_\{1\}changes the output\. Legal rewrites agree to10−1510^\{\-15\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

Under SwiGLU, R4 movesℓ\\ellonto the linear branch and R1 fuses it intoWupW\_\{\\mathrm\{up\}\}, forℓ\\ellof either sign:

diag⁡\(ℓ\)​\(silu⁡\(Wg​x\)⊙Wu​x\)=silu⁡\(Wg​x\)⊙\(diag⁡\(ℓ\)​Wu​x\)=silu⁡\(Wg​x\)⊙\(\(diag⁡\(ℓ\)​Wu\)​x\)\.\\diag\(\\ell\)\\big\(\\mathrm\{silu\}\(W\_\{g\}x\)\\odot W\_\{u\}x\\big\)=\\mathrm\{silu\}\(W\_\{g\}x\)\\odot\\big\(\\diag\(\\ell\)\\,W\_\{u\}x\\big\)=\\mathrm\{silu\}\(W\_\{g\}x\)\\odot\\big\(\(\\diag\(\\ell\)\\,W\_\{u\}\)\\,x\\big\)\.\(8\)
##### Corollaries: scaling and reparametrisation methods

All weight\-space reparametrisation methods are mergeable, because their nonlinearities sit on the parameter side \(Cayley in OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\), the norm in DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\),u/∥u∥u/\\lVert u\\rVertin HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)\)\. In HF PEFT practice\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\), an adapter on one side of a tied embedding and LM head is not merged faithfully \(ensure\_weight\_tying=False by default\)\. The scaling methods merge as follows\.

- •\(IA\)3on keys and values is an output scaling ofWK,WVW\_\{K\},W\_\{V\}and fuses into them\. On the FFN, HF applies it to the input ofWdownW\_\{\\mathrm\{down\}\}, and it fuses there\. Both hold for everyσ\\sigma\.
- •Alternatively, the FFN scaling fuses intoWupW\_\{\\mathrm\{up\}\}under ReLU whenℓ≥0\\ell\\geq 0, a restriction of the domain \(IA\)3trains over, and*under SwiGLU/GeGLU for everyℓ\\ell*, by R4 applied tosilu⁡\(Wg​x\)⊙Wu​x\\mathrm\{silu\}\(W\_\{g\}x\)\\odot W\_\{u\}x, as in \([8](https://arxiv.org/html/2610.12728#S6.E8)\)\. Under GELU it does*not*fuse intoWupW\_\{\\mathrm\{up\}\}, except for entries withℓi∈\{0,1\}\\ell\_\{i\}\\in\\\{0,1\\\}, when the pre\-activations range overℝ\\mathbb\{R\}\(the argument is in Appendix[C\.47](https://arxiv.org/html/2610.12728#A3.SS47)\)\. TheWupW\_\{\\mathrm\{up\}\}statement is practically idle, sinceWdownW\_\{\\mathrm\{down\}\}always works, and AWQ’s code already migrates scales this way \(into up\_proj under SwiGLU, into fc1 under ReLU, and through an inserted scaled activation under GELU\)\.
- •A query scaling fuses intoWQW\_\{Q\}\(R1\)\. It moves ontoWKW\_\{K\}\(R5\) only without RoPE or withℓ\\ellconstant on rotary pairs, and only without grouped\-query \(GQA\) sharing or withℓ\\ellconstant across the group\.
- •SSF fuses into the preceding linear box when that box has a bias slot; in bias\-free LLaMA\-type layers its shift needs a new bias\. LN\-tuning edits existing LN parameters, so it is trivially mergeable\. Folding a LayerNorm’sγ,β\\gamma,\\betainto the following box works in pre\-LN blocks, where the norm feeds only the next boxes, when that box has a bias slot forW​βW\\beta; an RMSNorm gain has noβ\\betaand folds without one \(the SliceGPT/QuaRot fold\)\. Neither folds into a tied embedding or LM head without breaking the tie\.

Not box\-locally mergeable: nonlinear Houlsby, Pfeiffer and parallel adapters; routed mixtures \(MoLE, LoRAMoE, X\-LoRA\), unless the gate is constant; AdapterFusion, even with a constant gate, since it combines nonlinear adapters; LST; prefix\-type methods \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\)\.

The nonlinear adapters are[Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib175),[Pfeiffer et al\. \(2020a\)](https://arxiv.org/html/2610.12728#bib.bib366)\(with AdapterFusion\) and the parallel adapter of[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)\. The routed mixtures are MoLE\([Wu et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib501)\), LoRAMoE\([Dou et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib99)\)and X\-LoRA\([Buehler and Buehler, 2024](https://arxiv.org/html/2610.12728#bib.bib45)\)\. LST is[Sung et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib439)and LN\-tuning[Zhao et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib578)\. Masking \(IA\)3to a few neurons connects to the sparse activation scaling of[Stoehr et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib428)\.

WHAT THIS EXPLAINSExact merging into existing weights is certified by local rewriting, a sufficient test together with box\-local obstructions, under explicit side conditions \(position\-uniformity, no sharing, bias slots\), and the rewrite also tells you*which*weight absorbs the update\. It is not a decision procedure for whole\-network mergeability; a stuck rewrite certifies only a box\-local obstruction\. RoPE pins a query scaling toWQW\_\{Q\}unless it is pair\-constant\. Bias\-free layers leave a shift no slot, and a new bias, though nearly free at inference, changes the architecture\. Post\-LN feeds the normalised wire to the residual too, so R7 cannot carry its gains forward\. Zero inference latency is a weaker goal \(a new bias costs almost nothing\), and for quantised bases the type caveat \(d\) applies\.

Remark\(residual\-stream edits that do merge\)\. Some edits outside \(a\) merge anyway, all position\-uniform\. A diagonal edith↦s⊙h\+th\\mapsto s\\odot h\+tafter a post\-LN folds into that norm’s gain and shift,γ′=s⊙γ\\gamma^\{\\prime\}=s\\odot\\gammaandβ′=s⊙β\+t\\beta^\{\\prime\}=s\\odot\\beta\+t\. A shifth↦h\+th\\mapsto h\+tafter a pre\-LN block with biases folds into its FFN output bias\. An orthogonal edith↦Q​hh\\mapsto Qhafter blockllof an untied pre\-RMSNorm model merges globally, SliceGPT\-style: rotate the embedding table, fold the gains of blocks1,…,l1,\\dots,linto their input weights composed withQ⊤Q^\{\\top\}, compose their output weights withQQ, and leave later blocks unchanged\. No merge is known for LoReFT’s edit at interior positions\.

#### Linear insertions, and ReFT

A linear inserted module fuses at once by R1, and the question is which weight\-space method it becomes\. ReFT, read one hidden vector at a time, becomes LoRA on a box the network lacks, the identity, written in homogeneous coordinates\(h,1\)\(h,1\)as\[Id∣0\]\[I\_\{d\}\\mid 0\]so that its last column holds the bias\.

Proposition VI\.4\(linear insertions, and ReFT\)\.\(a\) A linear adapter in parallel with a single linear box,W0​x\+s​U​D​xW\_\{0\}x\+sUDx,*is*LoRA: it has the sameρ\\rho\([He et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib165), Table 1\)\. A linear adapter parallel to a whole sublayer \(He et al\.’s parallel adapter around attention or the FFN\) edits the weightless skip connection rather than a base box, and is not box\-locally mergeable when the sublayer is non\-affine\.\(b\) LetW0∈ℝm×nW\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}be bias\-free \(as LLaMA\-typeWoW\_\{o\}andWdownW\_\{\\mathrm\{down\}\}are\), andr≥1r\\geq 1\. A serial linear adapter afterW0W\_\{0\},\(I\+U​D\)​W0\(I\+UD\)W\_\{0\}withU∈ℝm×rU\\in\\mathbb\{R\}^\{m\\times r\}andD∈ℝr×mD\\in\\mathbb\{R\}^\{r\\times m\}\(scale absorbed\), has imageW0\+\{Y∈ℳ≤r:row⁡Y⊆row⁡W0\}\.W\_\{0\}\+\\\{Y\\in\\mathcal\{M\}\_\{\\leq r\}:\\operatorname\{row\}Y\\subseteq\\operatorname\{row\}W\_\{0\}\\\}\.It is expressively equivalent to LoRA iffrank⁡W0=n\\rk W\_\{0\}=n, and weakly \(unpointed\) isomorphic to LoRA iffW0W\_\{0\}is invertible\. The isomorphism\(U,D\)↦\(U,D​W0\)\(U,D\)\\mapsto\(U,DW\_\{0\}\)sends a base point\(U0,D0\)\(U\_\{0\},D\_\{0\}\)to\(U0,D0​W0\)\(U\_\{0\},D\_\{0\}W\_\{0\}\)\. For a fixedTA\\mathrm\{T\}\_\{A\}pointing\(0,A0\)\(0,A\_\{0\}\)withrank⁡A0=r\\rk A\_\{0\}=rand1≤r<m=n1\\leq r<m=n, a pointed isomorphism exists iffW0W\_\{0\}is invertible androw⁡A0=row⁡\(D0​W0\)\\operatorname\{row\}A\_\{0\}=\\operatorname\{row\}\(D\_\{0\}W\_\{0\}\), and thenU0=0U\_\{0\}=0automatically\. With a biasb0b\_\{0\}, the serial adapter also moves the bias byU​D​b0UDb\_\{0\}, so on the affine box its image differs from weight\-only LoRA’s\. Houlsby’s near\-identity initialisation \(weights from a zero\-mean Gaussian with standard deviation10−210^\{\-2\}, truncated at two standard deviations\) makes practical serial adapters slightly unpointed; the pointed statement applies to zero\-up initialisations\.\(c\) LoReFT\([Wu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib505)\)isΦ⁡\(h\)=h\+R⊤​\(W​h\+b−R​h\)\\Phi\(h\)=h\+R^\{\\top\}\(Wh\+b\-Rh\), withR∈ℝr×dR\\in\\mathbb\{R\}^\{r\\times d\}having orthonormal rows, applied at selected positions\. On one hidden vector it is the extension by anidentity boxfollowed by a rank\-≤r\\leq raffine edit,Φ⁡\(h\)=\(I\+R⊤​\(W−R\)\)​h\+R⊤​b,\\Phi\(h\)=\\big\(I\+R^\{\\top\}\(W\-R\)\\big\)h\+R^\{\\top\}b,which is LoRAron the*affine*identity box\[Id∣0\]\[I\_\{d\}\\mid 0\]in homogeneous coordinates, withB=R⊤B=R^\{\\top\}constrained to the Stiefel manifold andA=\[W−R∣b\]A=\[W\-R\\mid b\]\. Its image is\(I,0\)\+ℳ≤r​\(d×\(d\+1\)\)\(I,0\)\+\\mathcal\{M\}\_\{\\leq r\}\(d\\times\(d\+1\)\), of dimensionr⁡\(2​d\+1−r\)r\(2d\+1\-r\)\. The bias is confined tocol⁡B\\operatorname\{col\}B, so forr<dr<dthis is not LoRAr⊕\{\}\_\{r\}\{\}\\oplus\{\}BitFit, whose dimension is larger byd−rd\-r\. It is simulated, throughh⁡\(R,W,b\)=\(R⊤,\[W−R∣b\]\)h\(R,W,b\)=\(R^\{\\top\},\[W\-R\\mid b\]\), by LoRAron that affine box pointed at\(R0⊤,0\)\(R\_\{0\}^\{\\top\},0\), aTB\\mathrm\{T\}\_\{B\}pointing withcol⁡B0=row⁡R0\\operatorname\{col\}B\_\{0\}=\\operatorname\{row\}R\_\{0\}; this homogeneous\-coordinate LoRA has a bias column inAA, unlike HF LoRA\. It is not simulated byTA\\mathrm\{T\}\_\{A\}\-pointed LoRAr, since itsdimS1=r⁡\(d\+1\)\\dim S\_\{1\}=r\(d\+1\)exceedsr​drd\. It is not isomorphic to LoRAr: its parameter count is smaller byr⁡\(r\+1\)/2r\(r\+1\)/2, and its residual symmetry isO⁡\(r\)\\mathrm\{O\}\(r\), acting by\(R,W,b\)↦\(O​R,O​W,O​b\)\(R,W,b\)\\mapsto\(OR,OW,Ob\)\. It is pointed iffW=RW=Randb=0b=0\. The reference implementation \(pyreft\) initialisesWWandbbby the default nn\.Linear init, independently ofRR, so practical LoReFT has a pointing defectR⊤​\(\(W0−R0\)​h\+b0\)R^\{\\top\}\(\(W\_\{0\}\-R\_\{0\}\)h\+b\_\{0\}\): at initialisation it replaces theRR\-component ofhhby a random affine function, and the simulation above is then only a weak morphism\. pyreft also applies dropout to the whole output,hhincluded, so the training\-time map is notΦ\\Phi\.

NUMThe Jacobian rank is22=r⁡\(2​d\+1−r\)22=r\(2d\+1\-r\)atd=6d=6,r=2r=2, against2626for LoRAr⊕\{\}\_\{r\}\{\}\\oplus\{\}bias\. The serial adapter’s image has dimension77at\(m,n,r\)=\(5,3,1\)\(m,n,r\)=\(5,3,1\)and at\(4,8,1\)\(4,8,1\), against1111for LoRA at\(4,8,1\)\(4,8,1\)\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

WHAT THIS EXPLAINSPer hidden vector, LoReFT \(and DiReFT\) adds no new geometric type\. Its image is that of LoRAron an affine identity box the architecture did not have\.[Wu et al\. \(2024d\)](https://arxiv.org/html/2610.12728#bib.bib505)already describe DiReFT as LoRA applied to hidden representations at certain positions, and name position as what separates ReFT from PEFT; the new content here is the image dimension, the bias confinement, theO⁡\(r\)\\mathrm\{O\}\(r\)gauge and the initialisation defect\. ReFT as a framework also allows nonlinear interventions \(pyreft’s act\_fn\), to which this does not apply\. The main obstruction to merging is position\-selectivity: LoReFT acts only at selected prompt positions, outside[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(a\), so no position\-independent weight can absorb it\. Even applied at every position, at interior residual\-stream sites the stream is read by the skip connection and by every later block, and no linear box of the original architecture sits there \(at the embedding output an untied embedding table could absorb a position\-uniform edit,[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(a\)\)\. By[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)it is box\-locally mergeable into its own inserted box, which does not help\.

LoReFT, DiReFT and the reference implementation pyreft are due to[Wu et al\. \(2024d\)](https://arxiv.org/html/2610.12728#bib.bib505)\. Serial adapters are due to[Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib175)\. The parallel form, and the reading of LoRA as a linear adapter parallel to one box, are in[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)\. LoRA is[Hu et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib177)\.

#### Attention is a homomorphism followed by a perspective map

Fix a queryqq\. Attention over a contextKKof key–value pairs computes

Attnq​\(K\)=∑\(k,v\)∈Ke⟨q,k⟩/d​v∑\(k,v\)∈Ke⟨q,k⟩/d\.\\mathrm\{Attn\}\_\{q\}\(K\)=\\frac\{\\sum\_\{\(k,v\)\\in K\}e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\\,v\}\{\\sum\_\{\(k,v\)\\in K\}e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\}\.\(9\)The pair \(total weight, weighted sum of values\) is additive in the context, and only the division\(Z,S\)↦S/Z\(Z,S\)\\mapsto S/Z, the perspective map of convex analysis \([Background D\.34](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)\), is nonlinear\. With a prefixPPbefore the contentCC, the output averages whatPPalone andCCalone would give, weighted by their shares of the mass, so the prefix is a parallel branch gated by its share\.

Figure 12:A prefix is a gate\([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\)\. One queryqqattends to six content tokensCCand four prefix tokensPP, all seeded synthetic data\. \(a\) Prefix tuning\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\), one softmax overP⊎CP\\uplus C\. The prefix \(ochre\) takes the massλq=Zq​\(P\)/\(Zq​\(P\)\+Zq​\(C\)\)\\lambda\_\{q\}=Z\_\{q\}\(P\)/\(Z\_\{q\}\(P\)\+Z\_\{q\}\(C\)\), and each content weight \(teal\) is its value without the prefix \(dashed outline\) times1−λq1\-\\lambda\_\{q\}\. The strip below plots the ratio after/before for every content token; it is the same for all six, so the content weights keep their ratios and their order \(part \(c\), due to[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)\)\. \(b\) LLaMA\-Adapter\([Zhang et al\., 2023h](https://arxiv.org/html/2610.12728#bib.bib562)\)normalisesPPandCCseparately and adds the prefix branch with a gategg: the prefix weights sum togg, and the content weights equal the no\-prefix ones exactly \(part \(d\)\)\. \(c\) The value vectors projected on their two leading principal axes, with the hulls of the content values \(dashed\) and of the prefix values \(ochre\)\. The prefix\-tuning output \(black dot\) lies on the segment fromAttnq​\(C\)\\mathrm\{Attn\}\_\{q\}\(C\)toAttnq​\(P\)∈conv​VP\\mathrm\{Attn\}\_\{q\}\(P\)\\in\\mathrm\{conv\}\\,V\_\{P\}, the fractionλq\\lambda\_\{q\}of the way \(parts \(b\) and \(c\); the identity \(b\) is due to[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)\), and a largerλq\\lambda\_\{q\}moves it further towards the prefix hull\. The LLaMA\-Adapter outputAttnq​\(C\)\+g​Attnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(C\)\+g\\,\\mathrm\{Attn\}\_\{q\}\(P\)\(open grey\) is an additive step that leaves the segment\. The rigidity holds within one layer, relative to the same layer without the prefix: prefixes at lower layers change the queries, keys and values of higher ones\. The values used are in §[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\.Theorem VI\.5\(attention is a homomorphism followed by a perspective map; known in part\)\.Fix a queryqq\. Multisets of key–value pairs form a commutative monoid under⊎\\uplus\. DefineΣq​\(K\)=\(∑e⟨q,k⟩/d,∑e⟨q,k⟩/d​v\)\.\\Sigma\_\{q\}\(K\)=\\Big\(\\sum e^\{\\langle q,k\\rangle/\\sqrt\{d\}\},\\ \\sum e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\\,v\\Big\)\.\(a\)Σq\\Sigma\_\{q\}is a monoid homomorphism into\(ℝ≥0×ℝdv,\+\)\(\\mathbb\{R\}\_\{\\geq 0\}\\times\\mathbb\{R\}^\{d\_\{v\}\},\+\), landing in\{Z\>0\}∪\{\(0,0\)\}\\\{Z\>0\\\}\\cup\\\{\(0,0\)\\\}\. On nonempty multisets,Attnq=π∘Σq\\mathrm\{Attn\}\_\{q\}=\\pi\\circ\\Sigma\_\{q\}with the partial mapπ⁡\(Z,S\)=S/Z\\pi\(Z,S\)=S/Z, defined forZ\>0Z\>0\.\(b\)Prefix is a gated parallel adapter\.For nonemptyPPandCC,Attnq​\(P⊎C\)=\(1−λq\)​Attnq​\(C\)\+λq​Attnq​\(P\),λq=Zq​\(P\)Zq​\(P\)\+Zq​\(C\)\.\\mathrm\{Attn\}\_\{q\}\(P\\uplus C\)=\(1\-\\lambda\_\{q\}\)\\,\\mathrm\{Attn\}\_\{q\}\(C\)\+\\lambda\_\{q\}\\,\\mathrm\{Attn\}\_\{q\}\(P\),\\qquad\\lambda\_\{q\}=\\frac\{Z\_\{q\}\(P\)\}\{Z\_\{q\}\(P\)\+Z\_\{q\}\(C\)\}\.\(c\)Per\-layer pattern rigidity\.Within a single layer, for fixedqqand a fixed content multisetCC: the content attention weights are multiplied by1−λq1\-\\lambda\_\{q\}, their ratios are unchanged, and the output lies on the segment fromAttnq​\(C\)\\mathrm\{Attn\}\_\{q\}\(C\)toAttnq​\(P\)∈conv⁡VP\\mathrm\{Attn\}\_\{q\}\(P\)\\in\\operatorname\{conv\}V\_\{P\}, whereVPV\_\{P\}is the set of prefix values\. This rigidity is relative to the same layer’s computation without the prefix; it is relative to the pretrained pattern only at the lowest prefixed layer\. Lower\-layer prefixes change the queries and the content keys and values of higher layers, so attention patterns at the level of the network are*not*rigid \([Petrov et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib364), §6; the same authors show that prefix\-tuning a pretrained transformer can be a universal approximator,[Petrov et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib365)\)\.\(d\)LLaMA\-AdapternormalisesPPandCCseparately and adds a zero\-initialised gate\. It computesAttnq​\(C\)\+tanh⁡\(gl\)​Attnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(C\)\+\\tanh\(g\_\{l\}\)\\,\\mathrm\{Attn\}\_\{q\}\(P\), with one gate per head in the paper; HF PEFT uses one raw zero\-initialised scalar per layer, withouttanh\\tanh\. It uses two perspective maps, leaves the content weights exactly unchanged, and is pointed atg=0g=0\. Its prompt keys and values are the frozenWk,WvW\_\{k\},W\_\{v\}applied to one learned prompt per adapted layer, and it acts in the top layers only\.\(e\)Prompt→\\todeep prefix in causal decoders\.Assume that the soft prompt comes first, that attention is causal, and that content positions are offset by the prompt lengthmmin both methods \(as HF PEFT does via cache\_position\)\. Then at every layer the states at prompt positions depend only onpp, andh:p↦\(Kj​\(p\),Vj​\(p\)\)jh:p\\mapsto\(K\_\{j\}\(p\),V\_\{j\}\(p\)\)\_\{j\}satisfiesΦ∘ρprefix∘h=Φ∘ρprompt\\Phi\\circ\\rho\_\{\\mathrm\{prefix\}\}\\circ h=\\Phi\\circ\\rho\_\{\\mathrm\{prompt\}\}as functions of the content tokens, read at content positions\. The two methods live on different extensions, so this is an arrow of the unpointed slice over the realisationC∞​\(Xcontent,Ycontent\)C^\{\\infty\}\(X\_\{\\mathrm\{content\}\},Y\_\{\\mathrm\{content\}\}\), not of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}\. Under RoPE without the offset, a differenthhworks: rotate each prefix key by−m\-m\(positions−m,…,−1\-m,\\dots,\-1\)\. For additive relative biases \(ALiBi, T5\) the answer depends on how the implementation places the prefix keys\. The construction does not apply in bidirectional encoders of depth≥2\\geq 2\.\(f\)Obstructions\.Call a head*standard*if it is vanilla softmax attention without torch\-style add\_bias\_kv \(which would absorb a one\-token prefix\), add\_zero\_attn, learned attention sinks or softmax\-plus\-one\. On length\-one contexts, every weight setting of a standard head is affine inxx\. Some prefix makes the prefixed head non\-affine on length\-one contexts iffWo≠0W\_\{o\}\\neq 0and \(Wq≠0W\_\{q\}\\neq 0orWk⊤​bq≠0W\_\{k\}^\{\\top\}b\_\{q\}\\neq 0\); for bias\-free heads, iffWo≠0W\_\{o\}\\neq 0andWq≠0W\_\{q\}\\neq 0\. For such frozen weights and generic prefixes the prefixed head is not affine, because the gateλ⁡\(x\)\\lambda\(x\)is not constant\. So prefix tuning and P\-tuning v2 are not box\-locally mergeable into a standard head\. WhenWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0, no prefix is neutral, so prompt and prefix tuning are unpointed\.

The identity is standard\. Part \(b\) is due to[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165); see also[Chen et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib69)\. Part \(c\) and the soft\-prompt\-to\-prefix map in \(e\) are due to[Petrov et al\. \(2023, §2\.2 and §6\)](https://arxiv.org/html/2610.12728#bib.bib364), informally already[Li and Liang \(2021, §7\.2\)](https://arxiv.org/html/2610.12728#bib.bib252)\. LLaMA\-Adapter is[Zhang et al\. \(2023h\)](https://arxiv.org/html/2610.12728#bib.bib562), P\-tuning v2 is[Liu et al\. \(2021c\)](https://arxiv.org/html/2610.12728#bib.bib287), and prompt tuning is[Lester et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib243); on what prompts can express, see[Wang et al\. \(2023c\)](https://arxiv.org/html/2610.12728#bib.bib483)\. Ours are the monoid organisation, the explicit position hypotheses and the slice\-category typing of \(e\)\.

NUMOn a 3\-layer, 2\-head causal decoder, prompt\-to\-prefix agreement is9×10−169\\times 10^\{\-16\}with RoPE and the offset, against1\.31\.3without it and2×10−152\\times 10^\{\-15\}with keys rotated by−m\-m\. A head withWo=0W\_\{o\}=0stays affine althoughWq⊤​Wk≠0W\_\{q\}^\{\\top\}W\_\{k\}\\neq 0, and one withWq≠0W\_\{q\}\\neq 0,Wq⊤​Wk=0W\_\{q\}^\{\\top\}W\_\{k\}=0does not \(referee replication\)\. The identities \(a\)–\(d\) and the non\-affinity in \(f\) are checked to rounding error \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\); Figure[12](https://arxiv.org/html/2610.12728#S6.F12)draws them for one query\.

WHAT THIS EXPLAINSAt the lowest prefixed layer the content attention equals the pretrained one, rescaled by1−λq1\-\\lambda\_\{q\}\. At higher layers it changes only through the modified inputs, to first order additively in each position\([Petrov et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib364), §6\), so new query–key interactions cannot be imposed directly at a layer\. No network\-level impossibility follows\. LLaMA\-Adapter’s zero\-init attention is a structural change; categorically, its contribution is*pointedness*\. Prompt tuning maps into P\-tuning v2 in causal decoders\.[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)and[Li and Liang \(2021\)](https://arxiv.org/html/2610.12728#bib.bib252)assert that the inclusion is strict\. It is strict in a one\-layer, one\-token example, where a prompt fixes the prefix value as a function of its key, and we do not prove the generic statement\. Encoders remain open\.

#### Where each method lands

Table[18](https://arxiv.org/html/2610.12728#S6.T18)collects the verdicts of this exposé\.

Table 18:Extensions and their merge types\. The verdicts are box\-local \([Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\), and the attention results are per layer\.The verdicts are box\-local, and the attention results are per layer\. Two questions remain open \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\): whether prefix tuning is non\-mergeable for the whole network, and whether prompt tuning is strictly weaker than P\-tuning v2 in general causal decoders, together with how the two relate in encoders\.

## EXPOSÉ VIIWorked examples

This exposé reads fourteen methods through the three invariants and the merge layer\. For each method it says what the method reaches, how training moves it, what a moved base point does and how it merges, and it names the numbered result behind every claim\.

A method is a pointed mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\)\. Its image germ says what it can express, its fibres together with the optimizer’s metric say how it moves, base change says what a movedθ0\\theta\_\{0\}does, and extensions cover adapters and prompts\. The fourteen cards below apply that toolkit; Exposé[VIII](https://arxiv.org/html/2610.12728#S8)turns the answers into coordinates and arrows\.

Each card gives the object\(Q,q0,ρ\)\(Q,q\_\{0\},\\rho\)for one linear layery=W​xy=WxwithW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}, and then reads it through the invariants\. TheIMAGEline is the image germ atθ0\\theta\_\{0\}\(Exposé[II](https://arxiv.org/html/2610.12728#S2)\); theFIBREline is the fibre, the gauge and the induced metric \(Exposés[III](https://arxiv.org/html/2610.12728#S3)and[IV](https://arxiv.org/html/2610.12728#S4)\); theBASEline is base change, meaning pointings, quantised bases and merge\-and\-restart \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\); and theMERGEline is the extension and the merge type \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\. The prefix card adds anARROWSline\. Lines that an invariant leaves empty or trivial are omitted\. The merge types are those of Exposé[VIII](https://arxiv.org/html/2610.12728#S8), where M1 merges into its own box, M1→\\tointo a neighbouring box under side conditions, Mq leaves the quantised type when merged exactly, and M∞\\inftyis not box\-locally mergeable\. The header of each card names the source paper and the modification kind used in the catalogue, and its last line lists the results its claims rest on\. Their complete proofs are in Appendix[C](https://arxiv.org/html/2610.12728#A3); claims marked \[Num\] are numerical checks \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### Credits

Balancedness is due to[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100)and[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14), conserved charges to[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580), and the balanced slice to[Kempf and Ness \(1979\)](https://arxiv.org/html/2610.12728#bib.bib209)\. The crossover is the closed form of[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447)and[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328)\. The comparison of zero\-BBand zero\-AAstarts is due to[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158), and scaled GD to[Tong et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib452)and[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546)\. GaLore as one\-sided LoRA with carried state is due to[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)\. Prefixes as gated adapters are due to[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165), and the rigid content ratios and the prompt\-to\-prefix map to[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)\. The merge of \(IA\)3intoWdownW\_\{\\rm down\}is in[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275), and scale migration through SwiGLU is used by AWQ\([Lin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib268)\)\. LoKr’s transport is the rearrangement of[Van Loan and Pitsianis \(1993\)](https://arxiv.org/html/2610.12728#bib.bib459)\. The2​r2rbound underlies the PiSSA\-to\-LoRA conversion of[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324)\.

Example\(LoRA\)\.[Hu et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib177)·additiveρ⁡\(B,A\)=W0\+s​B​A,q0=\(0,A0\),B∈ℝm×r,A∈ℝr×n\.\\rho\(B,A\)=W\_\{0\}\+sBA,\\qquad q\_\{0\}=\(0,A\_\{0\}\),\\qquad B\\in\\mathbb\{R\}^\{m\\times r\},\\ A\\in\\mathbb\{R\}^\{r\\times n\}\.fnum@@desciitemImageThe coneW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)forr<min⁡\(m,n\)r<\\min\(m,n\)\. Withrank⁡A0=r\\rk A\_\{0\}=rthe first\-order image isS1=\{X​A0\}S\_\{1\}=\\\{XA\_\{0\}\\\}, of dimensionm​rmr, soθ0\\theta\_\{0\}is an apex with deficitr⁡\(n−r\)r\(n\-r\)\(Figure[3](https://arxiv.org/html/2610.12728#S2.F3)a\)\.fnum@@desciitemFibreGaugeGLr\\mathrm\{GL\}\_\{r\}, acting by\(B,A\)↦\(B​g−1,g​A\)\(B,A\)\\mapsto\(Bg^\{\-1\},gA\)\. Under gradient flow with block learning ratesηB,ηA\\eta\_\{B\},\\eta\_\{A\}and no dropout, the induced cometric isK⁡\(G\)=s2​\(ηB​G​A⊤​A\+ηA​B​B⊤​G\)K\(G\)=s^\{2\}\(\\eta\_\{B\}\\,GA^\{\\top\}A\+\\eta\_\{A\}\\,BB^\{\\top\}G\)\. The Lie algebra splits as𝔤​𝔩r=𝔰​𝔬r⊕Symr\\mathfrak\{gl\}\_\{r\}=\\mathfrak\{so\}\_\{r\}\\oplus\\mathrm\{Sym\}\_\{r\}\. The isometric half𝔰​𝔬r\\mathfrak\{so\}\_\{r\}leavesKKunchanged, and theSymr\\mathrm\{Sym\}\_\{r\}half carries the chargeΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}, conserved under gradient flow and equal to−A0A0⊤/ηA\-A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}from the zero\-BBstart\. For isotropicΦ=−κ​I\\Phi=\-\\kappa Ithe flow is solved in closed form\([Tarmoun et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib447);[Min et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib328)\)\. Withμ=s​ηA​ηB\\mu=s\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}and the crossover scaleσ∗=μ​\|κ\|/2\\sigma^\{\*\}=\\mu\\lvert\\kappa\\rvert/2, zero\-BBLoRA is one\-sided well belowσ∗\\sigma^\{\*\}, where effectively onlyBBmoves as in LoRA\-FA, and well above it follows the balanced dynamics of[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)\(Figure[7](https://arxiv.org/html/2610.12728#S3.F7)\)\. On the full\-rank locus, with channel\-uniform hyperparameters, SGD keepsO⁡\(r\)\\mathrm\{O\}\(r\)of the gauge, Adam and AdamW keep the signed permutationsBrB\_\{r\}, and undamped scaled GD keeps all ofGLr\\mathrm\{GL\}\_\{r\}\(Figure[9](https://arxiv.org/html/2610.12728#S3.F9)\)\. Under Adam withε=0\\varepsilon=0, no weight decay, no gradient clipping and a learning\-rate schedule shared by both blocks, LoRA\+ with learning\-rate ratioλ\\lambdafollows exactly the trajectory of LoRA withα\\alphamultiplied byλ\\lambda\([Schulman and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.12728#bib.bib401)\)\(Figure[8](https://arxiv.org/html/2610.12728#S3.F8)\)\.fnum@@desciitemBaseKKmerge\-and\-restart cycles reach exactlyW0\+ℳ≤min⁡\(K​r,m,n\)W\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}, so all ofΘ\\ThetaatK∗=⌈min⁡\(m,n\)/r⌉K^\{\*\}=\\lceil\\min\(m,n\)/r\\rceil\. Exact full\-rank pointings form three strata \(zeroBB, zeroAA, and a split that subtractsB0​A0B\_\{0\}A\_\{0\}from the frozen weight\)\. Their isomorphism classes are fixed byrow⁡A0\\operatorname\{row\}A\_\{0\},col⁡B0\\operatorname\{col\}B\_\{0\}andB0​A0B\_\{0\}A\_\{0\}respectively, so as a set they areGr⁡\(r,n\)⊔Gr⁡\(r,m\)⊔ℳr\\Gr\(r,n\)\\sqcup\\Gr\(r,m\)\\sqcup\\mathcal\{M\}\_\{r\}\.fnum@@desciitemMergeM1, since the trained updates​B​AsBAadds toW0W\_\{0\}\.Results\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)·[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)·[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)·[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)·[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)·[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)·[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)·[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(DoRA\)\.[Liu et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib282)·hybridρ⁡\(μ,B,A\)=diag⁡\(μ\)​N​\(W0\+s​B​A\),q0=\(μ0,0,A0\),μ0,i=∥w0,i∥\.\\rho\(\\mu,B,A\)=\\diag\(\\mu\)\\,N\(W\_\{0\}\+sBA\),\\qquad q\_\{0\}=\(\\mu\_\{0\},0,A\_\{0\}\),\\qquad\\mu\_\{0,i\}=\\lVert w\_\{0,i\}\\rVert\.HereNNnormalises each row to unit length andw0,iw\_\{0,i\}is theii\-th row ofW0W\_\{0\}\.fnum@@desciitemImageForW0W\_\{0\}without zero rows andr≤min⁡\(m,n\)r\\leq\\min\(m,n\)the image isDiagm⋅\(W0\+ℳ≤r\)=\{diag⁡\(ℓ\)​\(W0\+X\):rank⁡X≤r\}\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)=\\\{\\diag\(\\ell\)\(W\_\{0\}\+X\):\\rk X\\leq r\\\}, exactly LoRA followed by a free per\-output gain\. Its dimension ismin⁡\(m​n,r⁡\(m\+n−r\)\+m\)\\min\(mn,\\,r\(m\+n\-r\)\+m\)for genericW0W\_\{0\}\[Num\], and smaller for specialW0W\_\{0\}, for instancer⁡\(m\+n−r\)\+m−rr\(m\+n\-r\)\+m\-rwhenW0W\_\{0\}has rank one\. Its updates can exceed rankrr\. FromB0=0B\_\{0\}=0the base point is an apex, as for LoRA\.fnum@@desciitemFibreThe gauge containsGLr\\mathrm\{GL\}\_\{r\}, acting on\(B,A\)\(B,A\)as for LoRA\. For genericW0W\_\{0\}with\(m−r\)​\(n−r\)≥m\(m\-r\)\(n\-r\)\\geq mthe budgetr⁡\(m\+n\)\+mr\(m\+n\)\+mexceeds the dimension by exactlyr2r^\{2\}, soGLr\\mathrm\{GL\}\_\{r\}is the whole continuous gauge\. Gradient flow conserves the LoRA chargeΦ\\Phi, also through the detached norm\. The two backward rules give different geometries\. The exact gradient of the row normalisation projects the direction gradient row by row, which is a change of metric\. The detached backward of the DoRA paper \(its Sec\. 4\.3\) and of HF PEFT \(weight\_norm\.detach\(\)\) removes that projection and gives a non\-gradient update, which need not be a descent direction\. Under AdamW, which both the paper and HF use, neither variant follows a fixed metric, and the magnitude coordinates matter in their own right\. Sinceμi\\mu\_\{i\}has size about∥w0,i∥\\lVert w\_\{0,i\}\\rVertand an output gainℓi\\ell\_\{i\}has size11, an equal step on each changes the row gain by different amounts\.fnum@@desciitemMergeM1\. Since the images agree, any gap between DoRA and LoRA plus an output gain, started at the same weights, comes from training dynamics\. Experiment D \([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Prediction X\.2](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3)\) separates its three possible sources, the parametrisation, the backward rule and the optimizer coordinates\.Results\.[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)·[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)·[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)·[Prediction X\.2](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3)

Example\(OFT / BOFT\)\.[Qiu et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib381);[Liu et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib285)·multiplicativeW=W0​R⊤\(OFT\),W=diag⁡\(s\)​W0​R⊤\(BOFT\),R0=I,s0=𝟏\.W=W\_\{0\}R^\{\\top\}\\ \\ \\text\{\(OFT\)\},\\qquad W=\\diag\(s\)\\,W\_\{0\}R^\{\\top\}\\ \\ \\text\{\(BOFT\)\},\\qquad R\_\{0\}=I,\\ s\_\{0\}=\\mathbf\{1\}\.HereRRis block\-diagonal with Cayley blocksC⁡\(Q\)=\(I\+Q\)​\(I−Q\)−1C\(Q\)=\(I\+Q\)\(I\-Q\)^\{\-1\},QQskew\-symmetric of sizebb\(OFT\), or a product ofm′m^\{\\prime\}butterfly factors \(BOFT\)\.fnum@@desciitemImageWith exact Cayley maps the image is an orbit in the level set of the neuron Gram matrixKout=W​W⊤K\_\{\\rm out\}=WW^\{\\top\}, so every neuron norm, every angle between neurons and every singular value ofW0W\_\{0\}stays fixed, and block OFT also fixes then/bn/bpartial GramsW:,JW:,J⊤W\_\{:,J\}W\_\{:,J\}^\{\\top\}\. BOFT hasKout​\(W\)=diag⁡\(s\)​Kout​\(W0\)​diag⁡\(s\)K\_\{\\rm out\}\(W\)=\\diag\(s\)K\_\{\\rm out\}\(W\_\{0\}\)\\diag\(s\)\. It keeps\|cos\|\\lvert\\cos\\rvertbetween neurons when everysi≠0s\_\{i\}\\neq 0, and the cosines themselves when allsis\_\{i\}share a sign; with one butterfly factor it sends each block Gram to itsdiag⁡\(s\)\\diag\(s\)\-rescaling\. HF’s default Cayley–Neumann OFT,R=C⁡\(Q\)​\(I−Q4\)R=C\(Q\)\(I\-Q^\{4\}\), is only approximately orthogonal, sinceR⊤​R=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}\. The base point is regular\. For BOFT this holds when the factors’ plane sets are disjoint \(for instanceb=2b=2\), so that the factor Lie algebras form a direct sum\.fnum@@desciitemFibreFor exact\-Cayley block OFT the gauge is the stabiliser ofW0W\_\{0\}in the block group, which is trivial whenW0W\_\{0\}is injective\.fnum@@desciitemBaseExact\-Cayley block OFT on a fixed partition is idempotent up to closure, and stays inW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)however often it is merged\. Iterated BOFT withm′m^\{\\prime\}factors reachesdiag⋅W0⋅P⁡\(∏SO⁡\(b​2m′−1\)\)​P⊤\\diag\\cdot W\_\{0\}\\cdot P\\big\(\\prod\\mathrm\{SO\}\(b\\,2^\{m^\{\\prime\}\-1\}\)\\big\)P^\{\\top\}for a fixed permutationPP, and, for injectiveW0W\_\{0\},diag⋅W0⋅SO⁡\(n\)\\diag\\cdot W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)only at full butterfly depth\. Atn=64n=64andb=4b=4the block hypergraph has components of sizes4,8,16,32,644,8,16,32,64form′=1,…,5m^\{\\prime\}=1,\\dots,5, so it is connected only atm′=5m^\{\\prime\}=5\[Num\] \(Figure[10](https://arxiv.org/html/2610.12728#S5.F10)b\)\. With small generators \(∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}\) each merge of HF’s default is a contraction, so re\-merging can only shrink neuron norms and singular values\.fnum@@desciitemMergeM1\.Results\.[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)·[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(\(IA\)3\)\.[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275)·multiplicativeW=diag⁡\(ℓ\)​W0\(WK,WV\),W=W0​diag⁡\(ℓ\)\(Wdown\),ℓ0=𝟏\.W=\\diag\(\\ell\)\\,W\_\{0\}\\ \\ \(W\_\{K\},\\,W\_\{V\}\),\\qquad W=W\_\{0\}\\,\\diag\(\\ell\)\\ \\ \(W\_\{\\rm down\}\),\\qquad\\ell\_\{0\}=\\mathbf\{1\}\.fnum@@desciitemImageLinear, the closure of the orbit ofW0W\_\{0\}under a torus of one\-sided diagonal scalings\. Zeros ofW0W\_\{0\}stay zero, and row lines \(column lines onWdownW\_\{\\rm down\}\) stay fixed while allℓi≠0\\ell\_\{i\}\\neq 0\. The image lies inside that of rank\-1 HiRA,HiRA1\\mathrm\{HiRA\}\_\{1\}\. Its covariance group isMon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}on the output\-scaled boxes andGL×Mon\\mathrm\{GL\}\\times\\mathrm\{Mon\}onWdownW\_\{\\rm down\}\.fnum@@desciitemBaseIdempotent\.fnum@@desciitemMergeM1→\\to\. It fuses intoWKW\_\{K\},WVW\_\{V\}andWdownW\_\{\\rm down\}for every activation\. The FFN scale fuses alternatively intoWupW\_\{\\rm up\}under SwiGLU or GeGLU for everyℓ\\ell, through the linear branch, a scale migration that AWQ already uses; under ReLU forℓ≥0\\ell\\geq 0; and under GELU only for entriesℓi∈\{0,1\}\\ell\_\{i\}\\in\\\{0,1\\\}\.Results\.[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)·[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)·[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)

Example\(VeRA\)\.[Kopiczko et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib224)·additiveρ⁡\(b,d\)=W0\+Λb​B​Λd​A,q0=\(0,d0\)\.\\rho\(b,d\)=W\_\{0\}\+\\Lambda\_\{b\}B\\Lambda\_\{d\}A,\\qquad q\_\{0\}=\(0,d\_\{0\}\)\.HereΛb=diag⁡\(b\)∈ℝm×m\\Lambda\_\{b\}=\\diag\(b\)\\in\\mathbb\{R\}^\{m\\times m\}andΛd=diag⁡\(d\)∈ℝr×r\\Lambda\_\{d\}=\\diag\(d\)\\in\\mathbb\{R\}^\{r\\times r\}are trained, whileB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}andA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}are frozen, random and shared across layers\.fnum@@desciitemImageA toric cone of dimensionm\+r−1m\+r\-1\[Num\], with apex atθ0\\theta\_\{0\}\. Atb0=0b\_\{0\}=0onlybbreceives a gradient, so the first\-order image has dimension at mostmm, belowm\+r−1m\+r\-1whenr≥2r\\geq 2\.fnum@@desciitemFibreThe gauge isℝ×\\mathbb\{R\}^\{\\times\}, withccacting by\(b,d\)↦\(c​b,d/c\)\(b,d\)\\mapsto\(cb,d/c\), and gradient flow conserves the charge∥b∥2/ηb−∥d∥2/ηd\\lVert b\\rVert^\{2\}/\\eta\_\{b\}\-\\lVert d\\rVert^\{2\}/\\eta\_\{d\}\.fnum@@desciitemBaseIfBBhas no zero entry, rebasing reaches exactly LoRA\-FA\(A\)\(A\), the methodW0\+\{X​A\}W\_\{0\}\+\\\{XA\\\}, which is the target of the arrowh⁡\(b,d\)=Λb​B​Λdh\(b,d\)=\\Lambda\_\{b\}B\\Lambda\_\{d\}from VeRA to LoRA\-FA\(A\)\(A\)\.fnum@@desciitemMergeM1\.Results\.[Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)·[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)·[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(LoHa / LoKr\)\.[Hyeon\-Woo et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib186);[Yeh et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib532)·additiveW0\+\(B1​A1\)⊙\(B2​A2\)\(LoHa\),W0\+C⊗B​A\(LoKr\)\.W\_\{0\}\+\(B\_\{1\}A\_\{1\}\)\\odot\(B\_\{2\}A\_\{2\}\)\\ \\ \\text\{\(LoHa\)\},\\qquad W\_\{0\}\+C\\otimes BA\\ \\ \\text\{\(LoKr\)\}\.fnum@@desciitemImageLoHar,rhas updates of rank at mostmin⁡\(r2,m,n\)\\min\(r^\{2\},m,n\), by the cut\-rank bound\. Its dimension is at mostmin⁡\(m​n,2​r​\(m\+n\)−2​r2−\(m\+n−1\)\)\\min\\big\(mn,\\,2r\(m\+n\)\-2r^\{2\}\-\(m\+n\-1\)\\big\), with equality belowm​nmn\[Num\]\. The gauge isGLr×GLr\\mathrm\{GL\}\_\{r\}\\times\\mathrm\{GL\}\_\{r\}times the two\-sided diagonal torus\(B1,A1,B2,A2\)↦\(D1​B1,A1​D2,D1−1​B2,A2​D2−1\)\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)\\mapsto\(D\_\{1\}B\_\{1\},A\_\{1\}D\_\{2\},D\_\{1\}^\{\-1\}B\_\{2\},A\_\{2\}D\_\{2\}^\{\-1\}\), and their joint generic orbits have dimension2​r2\+m\+n−12r^\{2\}\+m\+n\-1\. HF zero\-initialises one factor, so the base point is an apex at which only that factor moves to first order\. At the common budget2​r​\(m\+n\)2r\(m\+n\)of LoHar,rand LoRA2rthree regimes occur\. Forr≤2r\\leq 2,r2≤2​rr^\{2\}\\leq 2r, so the image of LoHar,rlies inside that of LoRA2r, strictly when2​r2<m\+n−12r^\{2\}<m\+n\-1\. Forr≥3r\\geq 3and2​r2<m\+n−12r^\{2\}<m\+n\-1, LoHa trades at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}dimensions for rank up tor2r^\{2\}\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1, for instance atr=64r=64on4096×40964096\\times 4096or at LyCORIS\-styler=32r=32on320320\- to640640\-wide layers, it matches or exceeds LoRA2rin dimension as well as in rank\. At131,072131\{,\}072parameters on a4096×40964096\\times 4096matrix, LoRA16reaches130,816130\{,\}816dimensions and LoHa8,8at most122,753122\{,\}753\(Figure[4](https://arxiv.org/html/2610.12728#S2.F4)\)\.fnum@@desciitemImageAkk\-term Kronecker sum∑iCi⊗Di\\sum\_\{i\}C\_\{i\}\\otimes D\_\{i\}is a matrix of rank at mostkkafter the Van Loan–Pitsianis rearrangementℛ\\mathcal\{R\}, which sendsC⊗DC\\otimes Dtovec⁡\(C\)​vec⁡\(D\)⊤\\vect\(C\)\\vect\(D\)^\{\\top\}\. So LoKr is a constrained LoRA1transported byℛ−1\\mathcal\{R\}^\{\-1\}, which is LoRA across a different cut of the same four\-leg tensor\. Writem=m1​m2m=m\_\{1\}m\_\{2\}andn=n1​n2n=n\_\{1\}n\_\{2\}\. ForC∈ℝm1×n1C\\in\\mathbb\{R\}^\{m\_\{1\}\\times n\_\{1\}\}and inner rankr′≤min⁡\(m2,n2\)r^\{\\prime\}\\leq\\min\(m\_\{2\},n\_\{2\}\), LoKr’s maximum rank ismin⁡\(m1,n1\)​r′\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}, attained\. It lies inside zero\-init LoRAriffr≥min⁡\(m1,n1\)​r′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}, and is otherwise incomparable with it whenmin⁡\(m1,m2\)​min⁡\(n1,n2\)\>1\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\)\>1\(Figure[6](https://arxiv.org/html/2610.12728#S2.F6)\)\. In HF PEFT the second factor is low\-rank only whenr<max⁡\(m2,n2\)/2r<\\max\(m\_\{2\},n\_\{2\}\)/2; above that LoKr is an unconstrained Kronecker product\.fnum@@desciitemMergeM1\.Results\.[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)·[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)·[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)·[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)

Example\(FourierFT\)\.[Gao et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib124)·additiveW0\+α​Re⁡F−1​\(c\),c​supported onnffixed frequencies\.W\_\{0\}\+\\alpha\\,\\operatorname\{Re\}F^\{\-1\}\(c\),\\qquad c\\ \\text\{supported on $n\_\{f\}$ fixed frequencies\}\.HereFFis the two\-dimensional discrete Fourier transform andα\\alphaa fixed scaling \(HF’s default150150\)\.fnum@@desciitemImageA coordinate mask transported into the real cosine frame\. Its image lies in the point\-symmetric subspaceΔ​Wj,l=Δ​W−j,−l\\Delta W\_\{j,l\}=\\Delta W\_\{\-j,\-l\}\. Started atθ0\\theta\_\{0\}\(c0=0c\_\{0\}=0, HF’s init\_weights=True\) it is linear and regular\. Its dimension isnfn\_\{f\}minus the number of sampled conjugate pairs\(u,v\),\(−u,−v\)\(u,v\),\(\-u,\-v\), each of which adds one gauge dimension\. At LLM sizes this is close tonfn\_\{f\}, but at768×768768\\times 768with10001000frequencies a pair occurs in about half the draws, and HF’s default seed777777gives exactly two\. HF’s default random spectrum \(𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)\) starts elsewhere, with a defect about14%14\\%the size of a768×768768\\times 768W0W\_\{0\}of entry standard deviation0\.040\.04at10001000frequencies\. Its covariance groupΠcyc\\Pi\_\{\\rm cyc\}excludes most neuron permutations, so permuting hidden units can change what it reaches\.fnum@@desciitemBaseIdempotent with fixed indices\.fnum@@desciitemMergeM1\.Results\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)·[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)·[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(BitFit\)\.[Ben\-Zaken et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib27)·selectiveb↦b0\+βon every bias,W=W0\.b\\mapsto b\_\{0\}\+\\beta\\ \\ \\text\{on every bias\},\\qquad W=W\_\{0\}\.fnum@@desciitemImageA translation of the bias block, linear and regular\. It has no gauge, so its expressive dimension equals its parameter count\.fnum@@desciitemBaseIdempotent\.fnum@@desciitemMergeNothing to merge, since BitFit trains the model’s own biases\. Llama and Mistral have no biases, so it trains nothing there\.Results\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(Adapters\)\.[Houlsby et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib175)·architecturalx↦x\+U​σ​\(D​x\),D∈ℝr×d,U∈ℝd×r\.x\\mapsto x\+U\\sigma\(Dx\),\\qquad D\\in\\mathbb\{R\}^\{r\\times d\},\\ U\\in\\mathbb\{R\}^\{d\\times r\}\.fnum@@desciitemImageFunction level\. The adapter is an extension of the network, with neutral pointU=0U=0\(zero\-up initialisation\)\. Houlsby et al\.’s own near\-identity initialisation, with weights drawn from a zero\-mean Gaussian of standard deviation10−210^\{\-2\}truncated at two standard deviations, starts near the neutral point and leaves a small defect\.fnum@@desciitemFibreUnder Houlsby’s GELU the gauge is only the permutations of hidden units\. A ReLU, as in the Pfeiffer and parallel adapters, adds positive per\-unit rescalings, givingSr⋉ℝ\>0rS\_\{r\}\\ltimes\\mathbb\{R\}\_\{\>0\}^\{r\}\. AtU=0U=0onlyUUmoves to first order \(the zero gate, function\-level variant\)\.fnum@@desciitemMergeM∞\\infty, since it is not box\-locally mergeable for non\-linearσ\\sigma\. Withσ=id\\sigma=\\mathrm\{id\}, an adapter parallel to a single linear box is LoRA, with the sameρ\\rho; one parallel to a whole sublayer edits the weightless skip connection and stays M∞\\inftywhen the sublayer is non\-affine\. A serial linear adapter\(I\+U​D\)​W0\(I\+UD\)W\_\{0\}after a bias\-freeW0W\_\{0\}reaches exactly the rank\-≤r\\leq rupdates whose rows lie inrow⁡W0\\operatorname\{row\}W\_\{0\}, which is LoRA’s image iffrank⁡W0=n\\rk W\_\{0\}=n\.Results\.[Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)·[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)·[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)·[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)·[Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)

Example\(Prefix / prompt tuning\)\.[Li and Liang \(2021\)](https://arxiv.org/html/2610.12728#bib.bib252);[Lester et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib243)·augmentationAttnq​\(P⊎C\)=\(1−λq\)​Attnq​\(C\)\+λq​Attnq​\(P\),λq=Zq​\(P\)Zq​\(P\)\+Zq​\(C\)\.\\mathrm\{Attn\}\_\{q\}\(P\\uplus C\)=\(1\-\\lambda\_\{q\}\)\\,\\mathrm\{Attn\}\_\{q\}\(C\)\+\\lambda\_\{q\}\\,\\mathrm\{Attn\}\_\{q\}\(P\),\\qquad\\lambda\_\{q\}=\\frac\{Z\_\{q\}\(P\)\}\{Z\_\{q\}\(P\)\+Z\_\{q\}\(C\)\}\.HerePPandCCare the prefix and content multisets of key–value pairs, andZq​\(K\)=∑k∈Ke⟨q,k⟩/dZ\_\{q\}\(K\)=\\sum\_\{k\\in K\}e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\.fnum@@desciitemImagePrefix and prompt tuning are unpointed extensions on the sequence wire, in the sense of[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\): on a head withWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0no prefix is neutral\. A prefix acts as an input\-gated parallel adapter on attention\. Within its layer it multiplies the content attention weights by1−λq1\-\\lambda\_\{q\}and keeps their ratios, and the output moves on the segment towardsAttnq​\(P\)∈conv⁡VP\\mathrm\{Attn\}\_\{q\}\(P\)\\in\\operatorname\{conv\}V\_\{P\}\(Figure[12](https://arxiv.org/html/2610.12728#S6.F12)\)\. This rigidity is relative to the same layer without the prefix, and relative to the pretrained pattern only at the lowest prefixed layer\.fnum@@desciitemArrowsIn causal decoders whose content positions are offset by the prompt length, as in HF PEFT, every soft prompt is realised exactly by the deep prefix made of its own per\-layer keys and values\. This is an arrow of the unpointed slice, from prompt tuning into deep prefix tuning \(P\-tuning v2\)\.[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)and[Li and Liang \(2021\)](https://arxiv.org/html/2610.12728#bib.bib252)assert that the inclusion is strict\. It is strict in a one\-layer, one\-token example, where a prompt fixes the prefix value as a function of its key; the generic statement is not proved here, and encoders remain open\. LLaMA\-Adapter\([Zhang et al\., 2023h](https://arxiv.org/html/2610.12728#bib.bib562)\)normalises the prefix separately and adds a zero\-initialised gate, so it leaves the content attention weights unchanged and supplies the missing neutral point\.fnum@@desciitemMergeM∞\\infty\. For a standard softmax head withWo≠0W\_\{o\}\\neq 0andWq≠0W\_\{q\}\\neq 0, a generic prefix makes the head non\-affine even on single\-token inputs, so prefix tuning cannot be folded into the head’s own weights\.Results\.[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)·[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)

Example\(PiSSA / LoftQ / QLoRA\)\.[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324);[Li et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib255); [Dettmers et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib92)·additiveρ\(B,A\)=Wres\+sBA,sB0A0=\[W0\]r,Wres=W0−\[W0\]r\(PiSSA\),\\displaystyle\\rho\(B,A\)=W\_\{\\rm res\}\+sBA,\\qquad sB\_\{0\}A\_\{0\}=\[W\_\{0\}\]\_\{r\},\\qquad W\_\{\\rm res\}=W\_\{0\}\-\[W\_\{0\}\]\_\{r\}\\qquad\\text\{\(PiSSA\)\},ρ\(B,A\)=κθ0\+sBA,B0=0\(QLoRA\)\.\\displaystyle\\rho\(B,A\)=\\kappa\\theta\_\{0\}\+sBA,\\qquad B\_\{0\}=0\\qquad\\text\{\(QLoRA\)\}\.Here\[X\]r\[X\]\_\{r\}is the best rank\-rrapproximation ofXX,WresW\_\{\\rm res\}is frozen, andκ\\kappaquantises and dequantises\.[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324)takes=1s=1\.fnum@@desciitemImagePiSSA is a split \(Type II\) start\. Its image is the cone with vertexW0−\[W0\]rW\_\{0\}\-\[W\_\{0\}\]\_\{r\}, andθ0\\theta\_\{0\}is a smooth point of it \(Figure[3](https://arxiv.org/html/2610.12728#S2.F3)b\)\. Its image and zero\-init LoRA’s are incomparable\. No exact pointing of LoRAr, zero\-init or split, makes an update of rank above2​r2rreachable, and split pointings attain2​r2rwhen2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\.fnum@@desciitemFibreGaugeGLr\\mathrm\{GL\}\_\{r\}\. HF’s PiSSA factors giveΦ0=\(Sr/s\)​\(1/ηB−1/ηA\)\\Phi\_\{0\}=\(S\_\{r\}/s\)\(1/\\eta\_\{B\}\-1/\\eta\_\{A\}\), so the start is balanced,Φ0=0\\Phi\_\{0\}=0, at equal rates\. By the balance–escape trilemma a balanced, non\-stationary, identity\-at\-init start needs the frozen residual\.fnum@@desciitemBaseQLoRA is LoRA pointed atκ​θ0\\kappa\\theta\_\{0\}, with generically full\-rank defecte=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}, so its image missesθ0\\theta\_\{0\}unlessrank⁡e≤r\\rk e\\leq r\(Figure[1](https://arxiv.org/html/2610.12728#S0.F1)\)\. Splitting and error fitting change the defect, and the order of operations matters\. QPiSSA splits, then quantises the residual, and leavesκ⁡\(Wres\)−Wres\\kappa\(W\_\{\\rm res\}\)\-W\_\{\\rm res\}\. One\-step LoftQ quantises, then fits the error\. HF’s LoftQ initialisation ignores the LoRA scales=α/rs=\\alpha/r, so the defect ise−s​\[e\]re\-s\[e\]\_\{r\}\. It iseeminus its best rank\-rrapproximation only whens=1s=1, and fors=2s=2, as withα=2​r\\alpha=2r, its norm equals∥e∥\\lVert e\\rVert\.fnum@@desciitemMergePiSSA merges into its box \(M1\)\. On a quantised base the type is Mq\. Exact merges of generic high\-precision adapters leave the quantised type, and type\-preserving merges are lossy; QA\-LoRA’s group\-constant adapters under affine group quantisation are the exception\([Xu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib514)\)\.Results\.[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)·[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)·[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)·[Proposition III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)·[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)

Example\(ReLoRA\)\.[Lialin et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib258)·optimizerθk\+1=θk\+s​Bk​Ak,fresh​\(Bk,Ak\)​at each stage\.\\theta\_\{k\+1\}=\\theta\_\{k\}\+sB\_\{k\}A\_\{k\},\\qquad\\text\{fresh \}\(B\_\{k\},A\_\{k\}\)\\text\{ at each stage\}\.fnum@@desciitemBaseRebased LoRA\. AfterKKstages it reaches exactlyW0\+ℳ≤min⁡\(K​r,m,n\)W\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}, so all ofΘ\\ThetaatK∗=⌈min⁡\(m,n\)/r⌉K^\{\*\}=\\lceil\\min\(m,n\)/r\\rceil\(Figure[10](https://arxiv.org/html/2610.12728#S5.F10)a\)\. Over weight space it isLoRAr⊕K\\mathrm\{LoRA\}\_\{r\}^\{\\oplus K\}unrolled in time\.fnum@@desciitemFibreEach stage is an apex with gaugeGLr\\mathrm\{GL\}\_\{r\}\.fnum@@desciitemMergeEach stage merges \(M1\)\.Results\.[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(GaLore\)\.[Zhao et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib581)·optimizerWt\+1=Wt−η​Pt​opt​\(Pt⊤​Gt\),Pt∈ℝm×r\.W\_\{t\+1\}=W\_\{t\}\-\\eta\\,P\_\{t\}\\,\\mathrm\{opt\}\\big\(P\_\{t\}^\{\\top\}G\_\{t\}\\big\),\\qquad P\_\{t\}\\in\\mathbb\{R\}^\{m\\times r\}\.HereGtG\_\{t\}is the weight gradient, andPtP\_\{t\}is refreshed from its SVD everyTTsteps\.fnum@@desciitemFibreA backward lens with compressionPt⊤P\_\{t\}^\{\\top\}and anchorPtP\_\{t\}\. Under an optimizer that reads only its gradients, such as Adam with weight decay00and no full\-gradient clipping, each period is exactly one\-sided LoRAWk\+Pk​CW\_\{k\}\+P\_\{k\}CwithC0=0C\_\{0\}=0, merged and re\-initialised at the end of the period, with its optimizer state carried over rather than reset or transported\. Decoupled weight decay and full\-gradient clipping break this identity\.fnum@@desciitemBaseThe anchor is constant on each period, so it is integrable trivially, including atT=1T=1, and integrability does not separate GaLore from LoRA\. What separates them is the resampling schedule together with the carried state\. As a set over its possible sequences of projectors, the moving shape reaches all ofΘ\\Theta\.fnum@@desciitemMergeNot applicable, since GaLore writesWWitself\.Results\.[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)·[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)·[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)·[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)

Example\(Merging\)\.[Ilharco et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib187);[Huang et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib180)·compositionθ0\+∑iλi​τi\(task arithmetic\),\(∑iwi​Bi\)​\(∑jwj​Aj\)\(LoraHub\)\.\\theta\_\{0\}\+\\textstyle\\sum\_\{i\}\\lambda\_\{i\}\\tau\_\{i\}\\ \\ \\text\{\(task arithmetic\)\},\\qquad\\big\(\\textstyle\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)\\big\(\\textstyle\\sum\_\{j\}w\_\{j\}A\_\{j\}\\big\)\\ \\ \\text\{\(LoraHub\)\}\.fnum@@desciitemFibreTask arithmetic isGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-natural\. TIES\-Merging\([Yadav et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib517)\), with global trimming and sign election by total mass, is natural under homotheties times signed permutations\. DARE\([Yu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib538)\)is natural under every invertible coordinate rescaling for each mask and under permutations in law, so its distributional group is the monomial group\. None of these sparsifying rules is natural under rotations\.fnum@@desciitemFibrePost\-hoc factor\-wise rules \(LoraHub, and HF’s linear merge with its non\-SVD ties and dare variants\) depend on each adapter’s gauge, since replacing\(B2,A2\)\(B\_\{2\},A\_\{2\}\)by\(c​B2,A2/c\)\(cB\_\{2\},A\_\{2\}/c\)changes LoraHub’s result by∑l≠2wl​w2​\[\(c−1\)​B2​Al\+\(c−1−1\)​Bl​A2\]\\sum\_\{l\\neq 2\}w\_\{l\}w\_\{2\}\\big\[\(c\-1\)B\_\{2\}A\_\{l\}\+\(c^\{\-1\}\-1\)B\_\{l\}A\_\{2\}\\big\]\(Figure[9](https://arxiv.org/html/2610.12728#S3.F9)\)\. A shared factor, such as a common frozenAA, or a joint frame that produces one, such as the jointU​ΣU\\Sigmaof KnOTS\([Stoica et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib429)\), removes the gauge dependence\. With a sharedAA, LoraHub returns\(∑jwj\)​∑iwi​Bi​A\(\\sum\_\{j\}w\_\{j\}\)\\sum\_\{i\}w\_\{i\}B\_\{i\}A, exact up to the global factor∑jwj\\sum\_\{j\}w\_\{j\}\.fnum@@desciitemMergeMerging an adapter into a full\-precision base is the canonical arrow to the terminal object, full fine\-tuning\.Results\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)·[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)

Exposé[VIII](https://arxiv.org/html/2610.12728#S8)turns the answers on these cards into seven coordinates, each read off a method’s formula without training, and into the arrows between methods\.

## EXPOSÉ VIIIThe classification

A method is a pointed mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\)\. Its image germ says what it can express \(Exposé[II](https://arxiv.org/html/2610.12728#S2)\), its fibres and the optimizer’s metric how it moves \(Exposé[III](https://arxiv.org/html/2610.12728#S3)\), base change what a movedθ0\\theta\_\{0\}does \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\), and extensions cover adapters and prompts \(Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\. Exposé[VII](https://arxiv.org/html/2610.12728#S7)applied this toolkit to fourteen methods\. This exposé turns the answers into coordinates and arrows\.

Seven coordinates, each read off the formula, place all 382 catalogued methods in a periodic table \(Table[19](https://arxiv.org/html/2610.12728#S8.T19); §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1)and §[VIII\.B](https://arxiv.org/html/2610.12728#S8.SS2)\)\. Some combinations of coordinates are empty by theorem; for instance, no linear method starts at an apex\. A methodNNsimulates a methodMMwhen a pointed smooth maphhgivesρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Five rules produce every arrow, and the atlas records 403 of them, each with its maphh; three tests rule arrows out, and PiSSA and LoRA, for instance, have no arrow either way \(§[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3)\)\. Parameters are not reach\. At131,072131\{,\}072parameters on a4096×40964096\\times 4096matrix, LoRA16reaches130,816130\{,\}816directions and LoHa8,8at most122,753122\{,\}753\(§[VIII\.B](https://arxiv.org/html/2610.12728#S8.SS2.SSSx4)\)\.

Table 19:The periodic table of the 142 core methods\. Rows are kinds \(axis 1 of §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1)\), with reparametrisations split by how they couple toW0W\_\{0\}: additively, by a group action, by a Hadamard product or by selecting coordinates\. Columns are the shapes of the image germ \(axis 2\)\. A dot marks an empty cell\. All seven coordinates of the methods named here are in Table[22](https://arxiv.org/html/2610.12728#S8.T22), and those of every catalogued method in Table[36](https://arxiv.org/html/2610.12728#A4.T36)\.### VIII\.A Seven axes, each decidable from the formula

[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)gave adapters four design axes \(functional form, insertion form, modified representation and composition function\), and[Chen et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib60)searched a similar design space\. The seven axes below extend the idea to weight\-space, backward and post\-hoc methods, and each axis is tied to a result\.

Definition\(coordinates of a method\)\. The*coordinates*of a methodM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\)are its answers to the seven questions below, each read offρ\\rho,q0q\_\{0\}and the box thatρ\\rhoacts on, with no training\.

The axes and their values are as follows\. Table[21](https://arxiv.org/html/2610.12728#S8.T21)states the question each axis asks and defines every value, Table[20](https://arxiv.org/html/2610.12728#S8.T20)gives the coordinates of LoRA, \(IA\)3and prefix tuning, and the ancillary file catalogue\.json repeats the definitions with examples for every value\.

1. 1\.Kind\.R \(reparametrisation\)⋅\\cdotE \(pointed extension\)⋅\\cdotE∘\(unpointed extension\)⋅\\cdotB \(backward lens or schedule\)⋅\\cdotP \(post\-hoc operation\)⋅\\cdotFT\(family over tasks, inputs or tenants\)\.
2. 2\.Shapeof the image germ\. Lin \(ρ\\rhoaffine\)⋅\\cdotCone \(multihomogeneous, non\-linear\)⋅\\cdotCone∗\(translated cone, split pointing\)⋅\\cdotOrb \(group orbit\)⋅\\cdotSat \(group×\\timescone\)⋅\\cdotMeet \(orbit∩\\capcone\)⋅\\cdotFun \(function level\)\.
3. 3\.Base point, read fromS1S\_\{1\}\. regular⋅\\cdotapex \(deficit\>0\>0\)⋅\\cdotneutral⋅\\cdotdefect \(ρ⁡\(q0\)≠θ0\\rho\(q\_\{0\}\)\\neq\\theta\_\{0\}\)⋅\\cdotunpointed\.
4. 4\.Gauge, from wire insertions and spider slides, checked by Jacobian rank\. none⋅\\cdotscalar⋅\\cdottorus⋅\\cdotGL\\mathrm\{GL\}⋅\\cdotGL×\\mathrm\{GL\}\\times\{\}torus⋅\\cdotnon\-linear\.
5. 5\.Covariance group\(the Klein geometry of the image\)\.GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}⋅\\cdotCO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}⋅\\cdotGL×CO\\mathrm\{GL\}\\times\\mathrm\{CO\}⋅\\cdotMon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}⋅\\cdotMon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}⋅\\cdotΠX\\Pi\_\{X\}\(imposed structure\)⋅\\cdotframe \(random or data, in law\)\.
6. 6\.Merge type\.M1 \(into its box\)⋅\\cdotM1→\\to\(into a neighbour via R1–R7, under side conditions\)⋅\\cdotMq \(an exact merge leaves the quantised type\)⋅\\cdotM∞\\infty\(not box\-locally mergeable\)\.
7. 7\.Rebasing closure\.idempotent⋅\\cdot→\\tospan⋅\\cdot→\\togroup⋅\\cdotschedule\.

Table 20:The seven questions, answered for three methods\. Every answer is read off the formula with no training; the values of each axis are defined in Table[21](https://arxiv.org/html/2610.12728#S8.T21)\.#### Deciding each axis

Each axis has a procedure that uses onlyρ\\rho,q0q\_\{0\}and the box\.

- •*Kind\.*Check whether the trained model isf⁡\(ρ⁡\(q\),−\)f\(\\rho\(q\),\-\)with the original architecture \(R\), whether new boxes are inserted and some parameter value makes them the identity \(E\) or none does \(E∘\), whether only the gradient path or the schedule changes \(B\), whether the inputs are trained checkpoints \(P\), and whether the adapter depends on a task, an input or a tenant \(FT\)\. See[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)and[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\.
- •*Shape\.*Read the degree ofρ−θ0\\rho\-\\theta\_\{0\}inqq\. An affine map gives Lin and a multihomogeneous non\-linear one gives Cone\. A group acting onθ0\\theta\_\{0\}gives Orb, a torus acting on another method’s image gives Sat \([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\), and an intersection gives Meet \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\.
- •*Base point\.*Evaluateρ⁡\(q0\)\\rho\(q\_\{0\}\)and computeS1=d​ρq0​\(Tq0​Q\)S\_\{1\}=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\), then comparedimS1\\dim S\_\{1\}with the generic rankd⁡\(M\)d\(M\)\([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. For LoRA this is[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(b\)\.
- •*Gauge\.*Insertg​g−1gg^\{\-1\}on each internal wire between trainable boxes and slide diagonals through copy spiders, then confirm\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\)by a Jacobian rank at a random point \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(a\)\)\.
- •*Covariance\.*Test the identity of[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)on generators ofGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, in distribution when frames are random or come from data\.
- •*Merge\.*Run the rewrite rules R1–R7 of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)with their side conditions, adding the type condition of[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(d\) for a quantised base\. The rules are sufficient conditions\. They certify M1 and M1→\\to, and M∞\\inftyrecords a box\-local obstruction\. Whether a whole network admits some other merge is left open\.
- •*Rebasing\.*Apply[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(a\) to additive shapes and[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\) and \(c\) to actions\.

#### Conventions

- •*Null coordinates\.*“–” \(null in the classified data\) means that the axis does not apply, and no axis has a separate “not applicable” value\. The merge axis keeps a legacy key na, used for backward lenses and schedules, which write the full weight directly and so have nothing to merge; it means the same as “–”\.
- •*When an axis applies\.*Shape and gauge need a parameter space, so they are “–” for closed\-form post\-hoc operations with no tunable coefficient \(Fisher merging without mixing weights, RegMean, uniform model soups\), which are single points\. Base point is “–” only for methods with no base point of their own, the serving families S\-LoRA and Punica\. Rebasing needs a weight\-space merge, so it is “–” for every M∞\\inftymethod; function\-level closures belong in the notes\. Covariance applies to every method with a formula\.
- •*Function\-level covariance\.*For extensions and families \(shape Fun\), covariance is the function\-level analogue of[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6): the inserted or generated family on each adapted wire is invariant under conjugation by\(g,h\)\(g,h\)on that wire\. Routed mixtures are included, since router and experts transform together\.
- •*Post\-hoc operations\.*The parameter space of a P operation is its tunable coefficients \(λ\\lambdain task arithmetic, TIES, DARE, LoRA soups and CAT, LoraHub\), and its covariance is the equivariance of the combination rule \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.
- •*Nearest keys\.*Entries without an exact key are coded to the nearest one:CO×Borel→Π\\mathrm\{CO\}\\times\\mathrm\{Borel\}\\to\\Pi\(OLoRA\);O⁡\(r\)→GL\\mathrm\{O\}\(r\)\\to\\mathrm\{GL\}\(LoReFT, PoLAR\); stabiliser→\\tonone \(OFT\);CO⁡\(m\)×GL⁡\(n\)→CO×CO\\mathrm\{CO\}\(m\)\\times\\mathrm\{GL\}\(n\)\\to\\mathrm\{CO\}\\times\\mathrm\{CO\}\(Astra, whitened splits\);O⁡\(m\)×O⁡\(n\)→CO×CO\\mathrm\{O\}\(m\)\\times\\mathrm\{O\}\(n\)\\to\\mathrm\{CO\}\\times\\mathrm\{CO\}\(LoRA\-GA\);ℝ×​Bm×ℝ×​Bn→Mon×Mon\\mathbb\{R\}^\{\\times\}B\_\{m\}\\times\\mathbb\{R\}^\{\\times\}B\_\{n\}\\to\\mathrm\{Mon\}\\times\\mathrm\{Mon\}\(LoRA\-SB\);ℝ×​BN\\mathbb\{R\}^\{\\times\}B\_\{N\}and the torusTN→Mon×MonT\_\{N\}\\to\\mathrm\{Mon\}\\times\\mathrm\{Mon\}\(TIES, DARE\)\. One\-sided keys \(Mon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\},GL×CO\\mathrm\{GL\}\\times\\mathrm\{CO\}\) name the group up to exchanging the two sides; the tables of §[VIII\.B](https://arxiv.org/html/2610.12728#S8.SS2)write \(output side\)×\\times\(input side\)\.
- •*Instances\.*Arrows and obstructions relate instances of methods: a rank, a pointing, a hyperparameter regime\. The instance is recorded in the note that accompanies each arrow and obstruction in the ancillary files \(§[VIII\.B](https://arxiv.org/html/2610.12728#S8.SS2.SSSx2)\)\.

WHAT THIS EXPLAINSMethods with different shapes or base\-point types are never isomorphic, however alike their formulas \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\.

Table 21:The seven axes\. For each axis, the question it answers and the definition of every value, with the test that decides it where one is needed\. “–” \(a null coordinate\) means that the axis does not apply\.ValueDefinition1 Kind\(where the method acts\)\.*Does the method reparametrise the existing weights, extend the architecture \(with or without a neutral point\), act only on the backward pass or the schedule, operate post hoc on several trained points, or form a family over an auxiliary baseTT?*R
reparametrisationA pointed smooth mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\)into the existing weight space, an object of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)\. Test: the trained model isf⁡\(ρ⁡\(q\),−\)f\(\\rho\(q\),\-\)with the original architectureff\.E
pointed extensionA function\-preserving embeddingι⁡\(θ\)=\(θ,θ0′\)\\iota\(\\theta\)=\(\\theta,\\theta\_\{0\}^\{\\prime\}\)into a larger architecturef~\\tilde\{f\}with a neutral pointθ0′\\theta\_\{0\}^\{\\prime\}, so thatf~\(θ,θ0′,−\)=f\(θ,−\)\\tilde\{f\}\(\\theta,\\theta\_\{0\}^\{\\prime\},\-\)=f\(\\theta,\-\), followed by a reparametrisation ofΘ×Θ′\\Theta\\times\\Theta^\{\\prime\}\. Test: new boxes are inserted, and some parameter value makes them the identity\.E∘
unpointed extensionAn extension with no neutral point: no parameter value reproduces the frozen function \(for prompts and prefixes,[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\) proves this on a head withWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0\)\. Test: try to solvef~\(θ0,p,−\)=f\(θ0,−\)\\tilde\{f\}\(\\theta\_\{0\},p,\-\)=f\(\\theta\_\{0\},\-\)forpp\.B
backward lens or scheduleThe forward map isidΘ\\mathrm\{id\}\_\{\\Theta\}and the update isθ←θ−η​at​\(opt⁡\(ct​\(∇L\)\)\)\\theta\\leftarrow\\theta\-\\eta\\,a\_\{t\}\\big\(\\mathrm\{opt\}\(c\_\{t\}\(\\nabla L\)\)\\big\)with a compressionctc\_\{t\}and an anchorata\_\{t\}; or the method is a sequence of merge\-and\-restart stages \(a rebasing scheme\); or it is an optimizer that compresses only its own state \(Flora\)\. Test: the method changes how gradients or optimizer state are compressed, or when the base is re\-pointed, and leaves the parametrisation alone\.P
post\-hoc operationAn operationΘN→Θ\\Theta^\{N\}\\to\\Theta\(or on tuples of factors\) applied to several trained points\. Test: the inputs are already\-trained adapters or checkpoints\.FT
family over a baseTTATT\-pointa:T→Qa:T\\to Qof a method, givingρ∘a:T→Θ\\rho\\circ a:T\\to\\Theta, whereTTis a set of tasks \(hypernetworks\), the inputsXX\(routing\), a simplex \(mixing\) or a set of tenantsUU\(serving\)\. Test: the adapter depends on an auxiliary variable\.2 Shapeof the image germ \(expressivity\)\.*What is the germ atθ0\\theta\_\{0\}of the reachable setIm⁡ρ\\Imop\\rho? Read it from the formula: isρ\\rhoaffine, multihomogeneous, a group action, a group acting on a cone, an intersection, or defined only at function level?*Lin
linear \(flat\)ρ\\rhois affine inqq, soIm⁡ρ\\Imop\\rhois an affine subspace, andd⁡\(M\)d\(M\)is the rank of the linear map, which equals\|M\|\\lvert M\\rvertwhen the map is injective \(FourierFT loses one dimension per sampled conjugate frequency pair\)\. Every point is regular, and rebasing is idempotent when frames, masks and operators are frozen across cycles\.Cone
cone with vertex at the startρ−ρ⁡\(q0\)\\rho\-\\rho\(q\_\{0\}\)is multihomogeneous \(or a union of linear spaces\), and the image is a non\-linear cone with vertex at the startρ⁡\(q0\)\\rho\(q\_\{0\}\), which is then a singular point \(an apex\)\. The start isθ0\\theta\_\{0\}for pointed methods, and the quantised or truncated baseθ0\+e\\theta\_\{0\}\+efor QLoRA\-type defects \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\. Determinantal, Segre, Hadamard and tensor\-network varieties are included\.Cone∗
translated cone \(split\)ρ=θres\+δ⁡\(q\)\\rho=\\theta\_\{\\rm res\}\+\\delta\(q\)withδ\\deltamultihomogeneous andq0q\_\{0\}chosen so thatδ⁡\(q0\)=θ0−θres\\delta\(q\_\{0\}\)=\\theta\_\{0\}\-\\theta\_\{\\rm res\}has full rank\. The image is a cone whose apexθres\\theta\_\{\\rm res\}is notθ0\\theta\_\{0\}, andθ0\\theta\_\{0\}is a smooth point \(Type II objects\)\.Orb
group orbitρ⁡\(q\)=γ⁡\(q\)⋅θ0\\rho\(q\)=\\gamma\(q\)\\cdot\\theta\_\{0\}withγ⁡\(q\)\\gamma\(q\)in a group \(or monoid\) acting linearly onΘ\\Theta\. Exactly orthogonal actions preserve the Gram matrix and the spectrum\. Torus and Hadamard actions keep only the zeros ofW0W\_\{0\}\(and its row lines while the scales are nonzero\), since unconstrained scales can create new zeros\.Sat
torus saturationA group or monoid acting on a cone or an orbit:Im⁡M=T⋅Im⁡N\\Imop M=T\\cdot\\Imop Nfor a torus or diagonal monoidTTand a methodNN\.Meet
meet \(pullback\)The image is an intersection of a cone and an orbit \(an image\-level meet\), or the object is a fibre product of two methods \(a categorical product, which exists only when the set\-level fibre product is a manifold\)\.Fun
function levelThere is no weight\-space image: the method lives on an extension and is compared through the realisationΦf\\Phi\_\{f\}\.3 Base point\(the position ofθ0\\theta\_\{0\}\)\.*Isθ0=ρ⁡\(q0\)\\theta\_\{0\}=\\rho\(q\_\{0\}\), and if so, is the first\-order imageS1=d​ρq0​\(Tq0​Q\)S\_\{1\}=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\)as large as the expressive dimensiond⁡\(M\)d\(M\)?*regulardimS1=d⁡\(M\)\\dim S\_\{1\}=d\(M\), a first\-order deficit of00: linear methods, orbit methods with an injective orbit map, and split \(Type II\) initialisations\.apex
first\-order deficientdimS1<d⁡\(M\)\\dim S\_\{1\}<d\(M\)\. For cone\-shaped germsθ0\\theta\_\{0\}is the vertex\. LoRA’s deficit isr⁡\(n−r\)r\(n\-r\); HRA’s paired initialisation hasdimS1=k​n−k⁡\(k\+1\)/2\\dim S\_\{1\}=kn\-k\(k\+1\)/2againstd=r​n−r⁡\(r\+1\)/2d=rn\-r\(r\+1\)/2, withk=r/2k=r/2\. DoRA inherits LoRA’s deficit through its torus factor\.neutral
pointed extensionAn extension evaluated at its neutral point: zero up\-projection, zero gate, identity box\.defect
pointing defectρ⁡\(q0\)≠θ0\\rho\(q\_\{0\}\)\\neq\\theta\_\{0\}, by design or by base change; the defecte=ρ⁡\(q0\)−θ0e=\\rho\(q\_\{0\}\)\-\\theta\_\{0\}is part of the data\.unpointedNo parameter value reproduces the frozen function\.4 Gauge\(the generic fibre ofρ\\rho\)\.*Which transformations of the trainable parameters leaveρ\\rhounchanged, and what is the dimension\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\)of the generic fibre? Read it off by insertingg​g−1gg^\{\-1\}on internal wires between trainable boxes and sliding diagonals through spiders, and confirm it by the generic Jacobian rank\.*none
trivialρ\\rhois injective near generic points\. There are no charges, and factor\-wise operations are well defined\.scalar
one scalar slideA one\-parameter group\(b,d\)↦\(c​b,d/c\)\(b,d\)\\mapsto\(cb,d/c\)through frozen boxes, with one conserved charge under gradient flow\.torusA diagonal group \(per rank channel or per hidden unit\), whose charges are diagonal entries\.GL\\mathrm\{GL\}
general linear on internal wiresGLr\\mathrm\{GL\}\_\{r\}, or a product of general linear groups on bonds, acting by\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)\. The compact half𝔰​𝔬​\(r\)\\mathfrak\{so\}\(r\)gives isometries, and the non\-compact halfSymr\\mathrm\{Sym\}\_\{r\}givesr⁡\(r\+1\)/2r\(r\+1\)/2charges\.GL×\\mathrm\{GL\}\\times\{\}torus
with a Hadamard torusGL\\mathrm\{GL\}on each factor plus the torus\(D1​X​D2,D1−1​Y​D2−1\)\(D\_\{1\}XD\_\{2\},D\_\{1\}^\{\-1\}YD\_\{2\}^\{\-1\}\)sliding through the copy spider, of dimensionr12\+r22\+m\+n−1r\_\{1\}^\{2\}\+r\_\{2\}^\{2\}\+m\+n\-1\.non\-linearThe generic fibre is not an orbit of a linear group action\.5 Covariance group\(the Erlangen geometry of the image\)\.*For which\(g,h\)∈GLm×GLn\(g,h\)\\in\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}doesIm⁡M⁡\(g​W0​h−1\)=g​Im⁡M⁡\(W0\)​h−1\\Imop M\(gW\_\{0\}h^\{\-1\}\)=g\\,\\Imop M\(W\_\{0\}\)\\,h^\{\-1\}hold for everyW0W\_\{0\}\(in distribution if random frames or data are involved\)?*GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}
basis\-freeThe image transforms with every change of basis on both sides\. It satisfies the sufficient condition of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)for every box\-wise linear function\-preserving gauge\.CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}
spectral frameThe method uses an SVD or another orthogonally equivariant frame ofW0W\_\{0\}, of gradients or of data\. Truncated SVDs are also equivariant under positive scalings, so the group isCO⁡\(m\)×CO⁡\(n\)=\(ℝ\>0​O​\(m\)\)×\(ℝ\>0​O​\(n\)\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)=\(\\mathbb\{R\}\_\{\>0\}\\mathrm\{O\}\(m\)\)\\times\(\\mathbb\{R\}\_\{\>0\}\\mathrm\{O\}\(n\)\)\([Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\)\. Methods that use bare singular frames without singular values \(LoRA\-GA\) are scale\-invariant rather than covariant and get onlyO⁡\(m\)×O⁡\(n\)\\mathrm\{O\}\(m\)\\times\\mathrm\{O\}\(n\); they are coded with this nearest key\.GL×CO⁡\(n\)\\mathrm\{GL\}\\times\\mathrm\{CO\}\(n\)
input\-side orthogonalA right action ofO⁡\(n\)\\mathrm\{O\}\(n\)or a subgroup of it\. The covariance on the input side is the normaliser of the structure group:CO⁡\(n\)\\mathrm\{CO\}\(n\)for full OFT, the block normaliser for block OFT\.Mon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}
one\-sided torusA diagonal torus acts on one side, so only monomial changes of basis are allowed there\.Mon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}
coordinate geometryHadamard or support structure in the standard basis on both sides\. It satisfies the sufficient condition of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)for neuron permutations but not for rotations\.ΠX\\Pi\_\{X\}
imposed structureThe method imposes a reshape \(a tensor factorisation\), a block partition, a cyclic order or a pairing that the network does not have, and so breaks even neuron permutations \(the test of[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\)\.frame
random or data frameCovariance holds only in law, through the distribution of frozen random matrices or of data statistics\.6 Merge type\(lifting along the realisation\)\.*After training, does the modified box function stay in the box’s function class, and if not, can the rewrite rules R1–R7 of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(fusion, permutation slide, positive\-diagonal slide, spider slide, cup slide, norm absorption, copy slide\) move it into a neighbouring linear box?*M1
fuses into its own boxAll non\-linearity is on the parameter side, and the merge is the canonical mapρ\\rhoto the terminal object\.M1→\\to
fuses into a neighbourA diagonal method merges by R1–R7 into a neighbouring linear box under side conditions: a position\-independent path \(under RoPE, scales constant on rotary pairs\), an unshared target box \(no tying, and no GQA sharing unless the scale is constant across the group\), and a bias slot for shifts\. \(IA\)3always fuses intoWKW\_\{K\},WVW\_\{V\}andWdownW\_\{\\rm down\};WupW\_\{\\rm up\}is an alternative only under SwiGLU or GeGLU \(every scale\) or ReLU \(nonnegative scales\), and under GELU only for scales in\{0,1\}\\\{0,1\\\}\. The rules give sufficient conditions; they do not decide whole\-network mergeability\.Mq
leaves the quantised typeAn exact merge of a generic high\-precision update leaves the quantised type \(NF4 and recomputed\-scale absmax grids are not even closed under addition\)\. Type\-preserving merges \(dequantise, add and requantise, as HF PEFT does for bitsandbytes layers\) stay on the grid but are lossy\. QA\-LoRA’s group\-constant adapters are the exception\.M∞\\infty
obstructedThe activation side is non\-affine in the input for some parameter value \(non\-linear adapters, softmax\-gated prefixes, input\-dependent routing\), or the edit acts only at selected positions \(ReFT, position\-gated LoRA variants\)\. This is box\-local non\-mergeability under the rewrite calculus of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\. Whole\-network non\-mergeability is open, and some residual\-stream edits do merge: into an untied embedding table, a writer’s bias or a post\-LN affine, or globally for orthogonal edits in RMSNorm models\.7 Rebasing closure\(the reach of merge\-and\-restart\)\.*What does the reachable setRK=θ0\+C\+KR\_\{K\}=\\theta\_\{0\}\+C^\{\+K\}\(additive\) orRK=SK​θ0R\_\{K\}=S^\{K\}\\theta\_\{0\}\(action\) converge to as the numberKKof merge\-and\-restart cycles grows?*idempotentThe shape is exactly a subspace, subgroup or submonoid independent of the base, with frames, masks and operators frozen across cycles, so cycles gain nothing \(up to closure\)\. Recomputing frames or masks from the merged weight does gain\.→\\tospan
converges to the spanFor a cone closed under real scaling,R∞=θ0\+span⁡CR\_\{\\infty\}=\\theta\_\{0\}\+\\spanop C\. For LoRA,RK=W0\+ℳ≤min⁡\(K​r,m,n\)R\_\{K\}=W\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}, and full fine\-tuning is reached exactly atK=⌈min⁡\(m,n\)/r⌉K=\\lceil\\min\(m,n\)/r\\rceil\.→\\togroup
converges to a generated groupFor action methods,R∞R\_\{\\infty\}is the orbit of the monoid generated by one cycle’s image\. It is the connected Lie subgroup generated when that image contains a neighbourhood ofeein each generating subgroup \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\)\)\. For block\-orthogonal factors it is∏CSO⁡\(C\)\\prod\_\{C\}\\mathrm\{SO\}\(C\)over the connected componentsCCof the block hypergraph\. HF’s Cayley–Neumann OFT does not satisfy the hypothesis: for small generators it generates only a contraction semigroup\.schedule
a rebasing scheme itselfThe method is defined as a sequence of base changes with a moving shape, and its reach is the sum of the shapes visited\.

### VIII\.B The periodic table

Table[19](https://arxiv.org/html/2610.12728#S8.T19), at the head of this exposé, places the 142 core methods by kind and by the shape of the image germ, and splits reparametrisations by how they touchW0W\_\{0\}: additively, by a group action, by a Hadamard product or by selecting coordinates\. The split refines the additive, selective, reparametrisation\-based and hybrid classes of[Lialin et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib257); the survey of[Han et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib153)uses a similar split\.

Some combinations of coordinates are empty by theorem\. An affineρ\\rhohas the same differential everywhere, so no linear method is an apex, and in the catalogue none is\. Corollary 2 of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)ties idempotent rebasing to shapes that are fixed subspaces, subgroups or submonoids, and of the catalogue’s9292idempotent methods,8484have linear shapes and77are orbits\. The other, WASI\([Nguyen et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib348)\), trains on the rank\-KKvarietyℳ≤K\\mathcal\{M\}\_\{\\leq K\}itself, a set that does not move with the base\.

#### The table by method

Table[22](https://arxiv.org/html/2610.12728#S8.T22)gives all seven coordinates of the methods the theory works with, together with the expressive dimension and the maximum rank\. In it,ddis the expressive dimension,ρmax\\rho\_\{\\max\}the maximum rank, andkk\(in GraLoRA\) the number of blocks per side\. “Π\\Pi” means imposed structure and “fr” a random or data frame\. Covariance is written \(output side\)×\\times\(input side\)\. FXand FUare families over inputs and over tenants, coded FTin the catalogue, and LoRA\(B0,A0\)\{\}\_\{\(B\_\{0\},A\_\{0\}\)\}is LoRA pointed at\(B0,A0\)\(B\_\{0\},A\_\{0\}\)\.

Table 22:The periodic table by method: kind, shape \(withddandρmax\\rho\_\{\\max\}\), base point, gauge, covariance, merge type and rebasing closure\. The notation is explained before the table, and the marks †,≈\\approxand \(fn\.\) and the numbered notes after it\.MethodKindShape \(dd;ρmax\\rho\_\{\\max\}\)Base pt\.GaugeCovarianceMergeRebasingFrozen / Full FTR \(initial / terminal\)\{θ0\}\\\{\\theta\_\{0\}\\\}/Θ\\Theta– / regular–all /GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}–idLoRARConer⁡\(m\+n−r\)r\(m\{\+\}n\{\-\}r\);rrapexGLr\\mathrm\{GL\}\_\{r\}GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaatK=⌈min⁡\(m,n\)/r⌉K=\\lceil\\min\(m,n\)/r\\rceilrsLoRA, LoRA\+R1==LoRAapexGLr\\mathrm\{GL\}\_\{r\}GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaLoRA\-Pro, Riem\. precond\., LoRA\-RITER \+ metric2==LoRAapexGLr\\mathrm\{GL\}\_\{r\}2GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaDeLoRARCone⊆ℳ≤r\\subseteq\\mathcal\{M\}\_\{\\leq r\},∥Δ​W∥F≤λ⁡∥W0∥\\lVert\\Delta W\\rVert\_\{F\}\\leq\\lambda\\lVert W\_\{0\}\\rVertapex \(HF\) / split \(paper\)non\-linear, fibrer2\+1r^\{2\}\{\+\}13GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaGraLoRAR∏\\prodblock cones,r⁡\(m\+n−r\)r\(m\{\+\}n\{\-\}r\);min⁡\(k​r,m,n\)\\min\(kr,m,n\)apex∏GLr/k\\prod\\mathrm\{GL\}\_\{r/k\}Π\\Pi\(blocks\)M1→Θ\\to\\ThetaDyLoRARfiltration of conesapexTrT\_\{r\}GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaAdaLoRARbudgeted union of cones \(P​Λ​QP\\Lambda Q\)apex \(dimS1=r\\dim S\_\{1\}=r\)non\-linear, fibrer⁡\(r\+1\)r\(r\{\+\}1\)4GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaLoRA\-FA \(v1/v2\)RLin\{X​A0\}\\\{XA\_\{0\}\\\},m​rmrregularnoneGL×fr\\mathrm\{GL\}\\times\\mathrm\{fr\}M1idMiSS / BoneRLin==LoRA\-FA\(𝟏⊤⊗Ir\)\(\\mathbf\{1\}^\{\\top\}\\otimes I\_\{r\}\)\(balance mode\)regularnoneGL×Π\\mathrm\{GL\}\\times\\PiM1idMiCARLin\{Umin​Y\}\\\{U\_\{\\min\}Y\\\}regularnoneCO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}M1idLoRA\-XS / SVFT / TinyLoRARLin in SVD frame \(r2r^\{2\}/\|Ω\|\\lvert\\Omega\\rvert/uu\)regular \(LoRA\-XS, TinyLoRA≈\\approx\)noneCO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}M1id \(frames frozen\)LoRA\-SBRLinB​R​ABRAin gradient frame,r2r^\{2\}defect \(by design\)noneMon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}5M1idVeRARtoric cone,m\+r−1m\{\+\}r\{\-\}1;rrapexℝ×\\mathbb\{R\}^\{\\times\}Mon×fr\\mathrm\{Mon\}\\times\\mathrm\{fr\}M1→\\toLoRA\-FA\(A\)\(A\)NOLARbilinear cone,k\+l−1k\{\+\}l\{\-\}1apexℝ×\\mathbb\{R\}^\{\\times\}frM1→span⁡\{Bi​Aj\}\\to\\spanop\\\{B\_\{i\}A\_\{j\}\\\}RandLoRARsum of VeRA\-type cones; full rankapextorusfrM1→Θ\\to\\ThetaVB\-LoRA, Uni\-LoRARbank / random\-tied LoRA imagesdefect \(≈\\approx\)discreteΠ\\Pi/ fr6M1→Θ\\to\\ThetaDoRARSatDiagm⋅\\mathrm\{Diag\}\_\{m\}\\cdot\{\}Cone,min⁡\(m​n,r⁡\(m\+n−r\)\+m\)\\min\(mn,\\allowbreak r\(m\+n\-r\)\+m\); fullapexGLr\\mathrm\{GL\}\_\{r\}Mon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}M1→Θ\\to\\ThetaPiSSA / MiLoRAR \(Type II\)translated coneregular \(smooth\)GLr\\mathrm\{GL\}\_\{r\}CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}M1→Θ\\to\\ThetaOLoRAR \(Type II\)translated cone \(QR\)regular \(smooth\)GLr\\mathrm\{GL\}\_\{r\}CO×Borel\\mathrm\{CO\}\\times\\mathrm\{Borel\}7M1→Θ\\to\\ThetaLoRA\-GA / CorDAR \(Type II\)translated cone \(gradient / covariance\)regular \(smooth\)GLr\\mathrm\{GL\}\_\{r\}O×O\\mathrm\{O\}\\times\\mathrm\{O\}/CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}\(data\)8M1→Θ\\to\\ThetaAstraR \(Type II\)translated cone \(output\-activation second moment\)regular \(smooth\)GLr\\mathrm\{GL\}\_\{r\}CO×GL\\mathrm\{CO\}\\times\\mathrm\{GL\}\(data\)9M1→Θ\\to\\ThetaEVAR \(Type I\)==LoRA10apexGLr\\mathrm\{GL\}\_\{r\}GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}10M1→Θ\\to\\ThetaQLoRAR atκ​θ0\\kappa\\theta\_\{0\}cone atκ​θ0\\kappa\\theta\_\{0\}defecteeGLr\\mathrm\{GL\}\_\{r\}Mon \(grid\)Mq→Θ\\to\\ThetaLoftQR \(approx\. Type II\)translated cone over a quantised residualdefect11GLr\\mathrm\{GL\}\_\{r\}MonMq→Θ\\to\\ThetaLoHaRHadamard variety,≤min⁡\(m​n,2​r​\(m\+n\)−2​r2−\(m\+n−1\)\)\\leq\\min\(mn,\\allowbreak 2r\(m\+n\)\-2r^\{2\}\-\(m\+n\-1\)\);min⁡\(r2,m,n\)\\min\(r^\{2\},m,n\)apexGLr2×\\mathrm\{GL\}\_\{r\}^\{2\}\\times\{\}torusMon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}M1→Θ\\to\\ThetaLoKr / KronARℛ−1\\mathcal\{R\}^\{\-1\}\-transported cone;rank⁡C⋅r′\\rk C\\cdot r^\{\\prime\}apexℝ×\(×GLr′\)\\mathbb\{R\}^\{\\times\}\(\\times\\mathrm\{GL\}\_\{r^\{\\prime\}\}\)Π⊗\\Pi\_\{\\otimes\}M1→Θ\\to\\ThetaTT / Tucker \(FacT, LoTR; LoRETTA\)Rmeet of transported cones12apex / regular12∏GLre\\prod\\mathrm\{GL\}\_\{r\_\{e\}\}Π⊗\\Pi\_\{\\otimes\}M1→Θ\\to\\ThetaMoRARLin,r^2\\hat\{r\}^\{2\}; high rankregularnoneΠ\\Pi\(blocks\)M1id \(fixed operators\)HiRAR\(DW0\)∗\(D\_\{W\_\{0\}\}\)\_\{\*\}Cone;r​rank⁡W0r\\rk W\_\{0\}apexGLr\\mathrm\{GL\}\_\{r\}Mon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}M1→Θ\\to\\Theta\(not transported ReLoRA\)FourierFT / WaveFTRLin \(sparse in the real Fourier / wavelet frame\)regular13none13Πcyc\\Pi\_\{\\rm cyc\}/Π\\PiM1id \(fixed indices\)C3ARLin \(block\-circulant\),m​n/bmn/b; fullregularnoneΠcyc\\Pi\_\{\\rm cyc\}M1idSHiRA / FISHRLin \(fixed coordinate mask\)regularnoneMon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}M1idTrainable TokensRLin \(fixed row mask on the embedding and a tied head\)regularnoneGL×Mon\\mathrm\{GL\}\\times\\mathrm\{Mon\}14M1idDiff pruningR⋃\\bigcupkk\-sparse subspacesapexnoneMon×Mon\\mathrm\{Mon\}\\times\\mathrm\{Mon\}M1→Θ\\to\\ThetaBitFit / LN\-tuningRLin \(bias / LN parameters\)regularnoneGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}/ MonM1 / M1→\\toidOFT \(block\)ROrb of∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\)\(exact Cayley\); quasi\-orthogonal \(HF\)15regularstabiliser15GL×Nblk\\mathrm\{GL\}\\times N\_\{\\rm blk\}M1id up to closure \(exact\);KoutK\_\{\\rm out\}shrinks \(HF\)15BOFT \(HF\)Rdiag\(ℝm\)×\\diag\(\\mathbb\{R\}^\{m\}\)\\times\{\}butterfly productregular†inter\-factorMon×Π\\mathrm\{Mon\}\\times\\PiM1→diag⋅W0⋅\\to\\diag\\cdot W\_\{0\}\\cdot\{\}∏SO⁡\(b​2m′−1\)\\prod\\mathrm\{SO\}\(b\\,2^\{m^\{\\prime\}\-1\}\);SO⁡\(n\)\\mathrm\{SO\}\(n\)only at full depthGOFT \(strict\) / qGOFTRGivens product / free2×22\\times 2blocks16regularnone / torus16GL×Π\\mathrm\{GL\}\\times\\PiM1→W0​SO​\(n\)\\to W\_\{0\}\\mathrm\{SO\}\(n\)/W0​ℝn×nW\_\{0\}\\mathbb\{R\}^\{n\\times n\}16HRARMeet\(W0\+ℳ≤r\)∩W0​SO​\(n\)\(W\_\{0\}\{\+\}\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\mathrm\{SO\}\(n\)\(image level\),r​n−r⁡\(r\+1\)2rn\-\\tfrac\{r\(r\+1\)\}\{2\}apexnon\-linear, fibrer⁡\(r\+1\)2\\tfrac\{r\(r\+1\)\}\{2\}GL×CO\\mathrm\{GL\}\\times\\mathrm\{CO\}M1→W0​SO​\(n\)\\to W\_\{0\}\\mathrm\{SO\}\(n\)in⌈n/r⌉\\lceil n/r\\rceil\(IA\)3/ SSF / REDRLin==closure of a torus orbitregularnoneMon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}/GL×Mon\\mathrm\{GL\}\\times\\mathrm\{Mon\}17M1→\\toidRoAdRRoAd1: orbit closure of\(ℂ×\)m/2\(\\mathbb\{C\}^\{\\times\}\)^\{m/2\},mm; RoAd2,4: Lin,2​m2mregularnone \(RoAd4: per block\)Monpair×GL\\mathrm\{Mon\}\_\{\\rm pair\}\\times\\mathrm\{GL\}M1idHoulsby / PfeifferEFun \(serial, non\-linear\)neutral18SrS\_\{r\}/Sr⋉ℝ\>0rS\_\{r\}\\ltimes\\mathbb\{R\}\_\{\>0\}^\{r\}18GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty–Parallel adapterEFun19neutralSr⋉ℝ\>0rS\_\{r\}\\ltimes\\mathbb\{R\}^\{r\}\_\{\>0\}19GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty19–CompacterEFun with PHM weights20neutraltorus20Π⊗\\Pi\_\{\\otimes\}M∞\\infty–LSTE∘Fun \(side network\)21unpointed21GL\\mathrm\{GL\}\(side network\)GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty–LLaMA\-AdapterEFun \(separately normalised prefix\)neutral \(g=0g=0\)noneGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty–LoReFTE22LoRAronAff⁡\(Id\)\\mathrm\{Aff\}\(I\_\{d\}\),r⁡\(2​d\+1−r\)r\(2d\{\+\}1\{\-\}r\)defect22O⁡\(r\)\\mathrm\{O\}\(r\)22GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty–Prompt tuning / P\-tuning v1E∘Fun \(input state\)23unpointednone / non\-linear23GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty–Prefix / P\-tuning v2E∘Fun \(gated parallel adapter, per layer\)unpointednon\-linear / none24GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty–MPT / CPTE∘P∗⊙u​v⊤P^\{\*\}\\odot uv^\{\\top\}/ ball25unpointed / pointed25torus / none25GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}/CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}\(fn\.\)M∞\\infty–GaLoreBmoving Lin\{Pt​X\}\\\{P\_\{t\}X\\\}regularnoneCO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}\(gradient\)–→Θ\\to\\ThetaGaLore with a random projectorBmoving Lin\{X​At\}\\\{XA\_\{t\}\\\}regularnonefr–→Θ\\to\\ThetaFloraB26Lin=Θ=\\Theta26regularnonefr \(in law\)26––LISABmoving layer blocksregularnoneGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}27–→Θ\\to\\ThetaMeZOBmoving random lineregularnoneO⁡\(Θ\)\\mathrm\{O\}\(\\Theta\)in law–→Θ\\to\\ThetaReLoRABcone per stage→ℳ≤K​r\\to\\mathcal\{M\}\_\{\\leq Kr\}apex per stageGLr\\mathrm\{GL\}\_\{r\}per stageGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1scheduleTask arithmeticPLinθ0\+∑λi​τi\\theta\_\{0\}\+\\sum\\lambda\_\{i\}\\tau\_\{i\}regular \(λ=0\\lambda=0\)noneGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)M1idTIES / DAREPLin inλ\\lambda\(coordinatewise rule\)regularnoneℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}/MonN\\mathrm\{Mon\}\_\{N\}28M1idLoRA soups / CATP⊕\\oplusof frozen line methods29regularnoneGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1idLoraHub / Poly / MHRP / FTfactor\-wise product of sumsapex∏GLr\\prod\\mathrm\{GL\}\_\{r\}, broken by LoraHub30GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}/Π\\Pi30M1 per task–MoLE / LoRAMoE / X\-LoRAFXinput\-dependent mixtureneutral / unpointed31per expert / scalar / none31GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)31M∞\\infty31–S\-LoRA / PunicaFULoRA cone per tenant over a shared base–∏uGLru\\prod\_\{u\}\\mathrm\{GL\}\_\{r\_\{u\}\}GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1 per tenant32–Text\-to\-LoRA / HyperFormerFTaTT\-point of LoRA / of non\-linear adaptersapex / defect33GLr\\mathrm\{GL\}\_\{r\}/GL\\mathrm\{GL\}33GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}M1 / M∞\\infty33–UniPELT / MAME∘34Fun, not a⊕\\oplus\-word34unpointed34GL\\mathrm\{GL\}34GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}\(fn\.\)M∞\\infty34–##### Notes on the table

† BOFT’s identity is regular when the factors’ plane sets are disjoint \(for exampleb=2b=2\), because then the factor Lie algebras form a direct sum\. “≈\\approx” marks implementations whose default initialisation is near, but not exactly at,θ0\\theta\_\{0\}: the FourierFT and C3A defaults in HF PEFT, VB\-LoRA, Uni\-LoRA, TinyLoRA, and LoRA\-XS \(R0∼𝒩⁡\(0,10−10\)R\_\{0\}\\sim\\mathcal\{N\}\(0,10^\{\-10\}\)\)\. They are objects of the unpointed slice with a pointing defect, small for LoRA\-XS and TinyLoRA but not for HF’s FourierFT \(𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)spectrum with scaling150150, about14%14\\%of a std\-0\.040\.04W0W\_\{0\}at7682768^\{2\}with10001000frequencies\), and the table classifies their exactly pointed variants\. “CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}”: frames built from truncated SVDs are equivariant under positive scalings as well as orthogonal maps, so the spectral\-frame geometry isCO⁡\(m\)×CO⁡\(n\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)\([Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\); bare singular frames used without their singular values \(LoRA\-GA\) are scale\-invariant rather than covariant, which leavesO⁡\(m\)×O⁡\(n\)\\mathrm\{O\}\(m\)\\times\\mathrm\{O\}\(n\)\. “\(fn\.\)” marks the function\-level analogue of[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6): the inserted or generated family on each adapted wire is invariant under conjugation by\(g,h\)\(g,h\)on that wire\. “–” means that the axis does not apply \(null in the classified data\)\. Where the classified data refines an entry \(Kind, Gauge, Covariance and Merge of LST, UniPELT and MAM, MPT, Compacter, Houlsby, DeLoRA, AdaLoRA, LoRETTA, LoRA\-SB, Trainable Tokens, VB\-LoRA, LISA, HyperFormer, S\-LoRA and Punica\), the table follows the data\.

Dimensions hold under the hypotheses of the results that compute them\. HRA’s is forW0W\_\{0\}of full column rank and evenr≤n−2r\\leq n\-2\([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\); DoRA’s is for genericW0W\_\{0\}with no zero rows, and it drops for specialW0W\_\{0\}\([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\); GraLoRA’s is for HF’s default hybrid\_r=0 \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\); and LoHa’s entry is an upper bound, with equality checked numerically below them​nmncap \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\.

##### Notes by row

The superscript numbers in Table[22](https://arxiv.org/html/2610.12728#S8.T22)refer to these notes\.

1. 1rsLoRA and LoRA\+ are LoRA up to the hyperparameter torus\.
2. 2LoRA\-Pro, Riemannian preconditioning and LoRA\-RITE are reparametrisations with a descending metric \(their SGD forms\)\. TheGLr\\mathrm\{GL\}\_\{r\}gauge is respected only undamped; LoRA\-Pro’s parameter step keepsO⁡\(r\)\\mathrm\{O\}\(r\), and LoRA\-RITE keepsGLr\\mathrm\{GL\}\_\{r\}on the full\-rank locus \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.
3. 3DeLoRA: its per\-term scalings form a2​r2r\-torus inside the fibre; forr=1r=1the fibre is that torus\.
4. 4AdaLoRA: the fibre consists of two torus edges and the absorption ofΛ\\Lambda\.
5. 5LoRA\-SB: the covariance isℝ×​Bm×ℝ×​Bn\\mathbb\{R\}^\{\\times\}B\_\{m\}\\times\\mathbb\{R\}^\{\\times\}B\_\{n\}, because the frame is the SVD of a sign\-gradient\.
6. 6VB\-LoRA:Π\\Pi, the chunk partition tied across boxes by the shared bank\. Uni\-LoRA: a random frame\.
7. 7OLoRA: the QR frame makes the input side Borel\. The catalogue key isΠ\\Pi, since the order of the inputs matters\.
8. 8LoRA\-GA:O×O\\mathrm\{O\}\\times\\mathrm\{O\}, from data; its bare singular frames are scale\-invariant and depend on the sign convention\. CorDA:CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}, from data, at the level of the formula\.
9. 9Astra: the split isΦ⁡\(W0​C​W0⊤\)​W0\\Phi\(W\_\{0\}CW\_\{0\}^\{\\top\}\)W\_\{0\}\([Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\)\.
10. 10EVA is LoRA withrow​A0\\mathrm\{row\}\\,A\_\{0\}spanned by principal components of the activations\. Its covariance isGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}at a fixed rank \(frameCO\\mathrm\{CO\}\), andGL×CO\\mathrm\{GL\}\\times\\mathrm\{CO\}with HF’s default rho=2\.
11. 11LoftQ: the defect is the attained residual whens=1s=1; HF ignores the scales=α/rs=\\alpha/r\.
12. 12TT and Tucker methods sit at the apex for zero\-core initialisations\. LoRETTA’s compensated split initialisation is regular, and its shape is a meet of translated cones \(Cone∗\)\.
13. 13FourierFT and WaveFT are regular with init\_weights=True; HF’s FourierFT default is a defect \(≈\\approx\)\. The gauge gains one dimension for each sampled conjugate pair\.
14. 14Trainable Tokens:GL\\mathrm\{GL\}on the embedding dimensions, Mon on the vocabulary\.
15. 15OFT: with the exact Cayley map the shape is the orbit of∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\)in the level set ofKoutK\_\{\\rm out\}, and rebasing is the identity up to closure\. HF’s default Cayley–Neumann map is only quasi\-orthogonal, and its merges shrinkKoutK\_\{\\rm out\}monotonically when∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}\. The gauge is the stabiliser ofW0W\_\{0\}, trivial for injectiveW0W\_\{0\}; the catalogue key is “none”\.
16. 16GOFT and qGOFT: qGOFT replaces each Givens rotation by an arbitrary2×22\\times 2block, and its gauge is a torus of dimensiond−2d\-2\. GOFT’s official code adds an input scaler by default \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\), and the row is for strict GOFT\. When the planes connect all coordinates, as in GOFT’s layout, re\-merging reachesW0​SO​\(n\)W\_\{0\}\\mathrm\{SO\}\(n\)for GOFT\. For qGOFT it reachesW0​ℝn×nW\_\{0\}\\mathbb\{R\}^\{n\\times n\}, the matrices whose columns lie incol⁡W0\\operatorname\{col\}W\_\{0\}: its blocks include singular ones and ones of negative determinant, and on connected planes they generate the monoid of alln×nn\\times nmatrices \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\)\)\.
17. 17\(IA\)3, SSF and RED:Mon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}for output scalings,GL×Mon\\mathrm\{GL\}\\times\\mathrm\{Mon\}for scalingWdownW\_\{\\rm down\}\.
18. 18Houlsby and Pfeiffer adapters are neutral with a zero up\-projection; Houlsby’s own near\-identity initialisation leaves a small defect\. Houlsby’s gauge is onlySrS\_\{r\}\(GELU in the reference code, swish in AdapterHub\); Pfeiffer’s isSr⋉ℝ\>0rS\_\{r\}\\ltimes\\mathbb\{R\}\_\{\>0\}^\{r\}\(ReLU\)\.
19. 19A parallel adapter withσ=id\\sigma=\\mathrm\{id\}, parallel to one linear box, is LoRA and merges as M1\. The gaugeSr⋉ℝ\>0rS\_\{r\}\\ltimes\\mathbb\{R\}^\{r\}\_\{\>0\}is for ReLU, as in the code of He et al\.
20. 20Compacter: the PHM weights are Kronecker\-structured and shared\. The torus consists of per\-term slides; aGL\\mathrm\{GL\}mixing of the PHM index leaves the rank\-ρ\\rhoclass\.
21. 21LST: the side network is fed through trainable down\-projection ladders\. The up\-projection has rank at mostd/rd/r, so no parameter value reproducesff\.
22. 22LoReFT is an extension at an affine identity box over positions\. The pyreft initialisation leaves a defect; LoReFT is neutral iffW=RW=Randb=0b=0\. The catalogue key for its gauge isGL\\mathrm\{GL\}\.
23. 23Prompt tuning has no gauge\. P\-tuning v1 adds an LSTM or MLP 2\-cell, and its gauge is non\-linear, through that encoder\.
24. 24Prefix tuning has a non\-linear gauge through the MLP of Li and Liang; P\-tuning v2 trains the prefix directly and has none\.
25. 25MPT’s prompt isP∗⊙u​v⊤P^\{\*\}\\odot uv^\{\\top\}, and MPT is unpointed\. CPT’s prompt lies in a ball around the demonstrations and is pointed at the in\-context\-learning context\. MPT’s gauge is a torus of dimensionℓ\+d\\ell\+dwithP∗P^\{\*\}trainable \(ℝ×\\mathbb\{R\}^\{\\times\}withP∗P^\{\*\}frozen\); CPT has none\.
26. 26Flora compresses only its momentum and takes full\-space Adafactor steps \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\); its random frame is the momentum projection, in law\.
27. 27LISA: each stage frees whole boxes\.
28. 28TIES elects signs by total mass, and its covariance isℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}\. DARE’s isMonN\\mathrm\{Mon\}\_\{N\}: the torus exactly for each mask, the permutations in law\.
29. 29CAT is a⊕\\oplusof frozen line methods,span⁡\{Bi​Ai\}\\spanop\\\{B\_\{i\}A\_\{i\}\\\}per layer, inside LoRAkr\.
30. 30LoraHub, which combines adapters post hoc, is gauge\-dependent and breaks∏GLr\\prod\\mathrm\{GL\}\_\{r\}; Poly and MHR, trained jointly, keep a diagonalGLr\\mathrm\{GL\}\_\{r\}\. The covariance isGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}on LoraHub’s products andΠ\\Pi\(blocks\) for MHR\.
31. 31LoRAMoE is neutral, MoLE and X\-LoRA are unpointed\. The gauge isGLr\\mathrm\{GL\}\_\{r\}per expert for LoRAMoE, a scalar for MoLE \(only the gate trains\) and none for X\-LoRA \(only the head trains\)\. Router and experts transform together\. Merging is M1 when the gate is constant\.
32. 32S\-LoRA and Punica compute relative to a shared base, and each tenant’s fibre is a LoRA cone\. Each tenant is left unmerged by design, to share the base batch\.
33. 33Text\-to\-LoRA is at the apex; HyperFormer’s random hypernetworks leave a defect\. The gauge isGLr\\mathrm\{GL\}\_\{r\}on the generated rank wire for Text\-to\-LoRA andGL\\mathrm\{GL\}on the task\-embedding wire for HyperFormer, and factor targets are gauge\-sensitive\. Merging is M1 per task for Text\-to\-LoRA and M∞\\inftyfor HyperFormer\.
34. 34UniPELT’s gates also make it an FX\. Its adapter and prefix parts sit on extensions, so it is not a⊕\\oplus\-word\. It is unpointed: the prefix has no neutral value, and UniPELT’s gate scales the prefix keys rather than their softmax mass\. TheGL\\mathrm\{GL\}gauge comes from UniPELT’s LoRA summand and MAM’s prefix MLP\. Merging is M∞\\infty, also for UniPELT’s input\-gated LoRA part\.

#### Checks on the table

Several entries rest on numerical checks reported in Appendix[E](https://arxiv.org/html/2610.12728#A5): the GraLoRA and LoHa dimensions and the LoHa orbit, the HRA apex \(by the Jacobian rank at the paired initialisation\), the RoAd orbit dimension, the VeRA rebasing span, the DoRA dimension caps, the FourierFT conjugate pairs, the LoraHub and Poly fibres, and the Cayley–Neumann contraction \[Num\]\.

The classified data pass the following structural checks\. Every arrow and every obstruction joins two catalogued methods and names a valid rule or test\. A check at the level of method identifiers flags2626pairs that carry both an arrow and an obstruction, and88arrows from a method to itself\. Each flagged pair concerns two instances of the target: a different rank, pointing, hyperparameter regime or variant\. For example, PiSSArmaps into LoRA2rat the degenerate pointing with product00\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\), while the O3 obstruction concerns LoRArat its apex, whose image is incomparable with PiSSA’s \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\)\)\. Each self\-arrow is an inclusion between instances of one family, such as LoRAr↪\{\}\_\{r\}\\hookrightarrow\{\}LoRAr\+1, ReLoRAK↪\{\}^\{K\}\\hookrightarrow\{\}ReLoRAK\+1or BOFT withm′m^\{\\prime\}butterfly factors inside BOFT withm′\+1m^\{\\prime\}\+1\.

The data deviate from the conventions of §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1.SSSx2)in eight recorded places, which are kept as published in the ancillary files\.

1. \(1\)Sixteen function\-level families and post\-hoc compositions carry no covariance \(LoRAMoE, MixLoRA, HydraLoRA, ICEdit, MoV, Lily, MoLE, X\-LoRA, PHATGOOSE, Arrow, LoraRetriever, LoRA\-Flow, SMEAR, GOAT, AdapterSoup and LoRA Switch\)\. Under the conventions they carry their expert family’s function\-level value:GL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}for dense LoRA experts and routers, andMon×GL\\mathrm\{Mon\}\\times\\mathrm\{GL\}for MoV\.
2. \(2\)Initial\-state tuning and state\-offset tuning are M∞\\inftywith rebasing “idempotent”; under the conventions their rebasing is “–”\.
3. \(3\)Flora is coded as a per\-period linear method with rebasing “schedule”\. Flora takes full\-space steps and compresses only its momentum \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\), so its per\-step shape is all ofΘ\\Thetaand its rebasing is “–”, as in Table[22](https://arxiv.org/html/2610.12728#S8.T22)\.
4. \(4\)EVA is codedGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}, which holds at a fixed rank; HF’s default rho=2 givesGL×CO⁡\(n\)\\mathrm\{GL\}\\times\\mathrm\{CO\}\(n\)\([Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\)\.
5. \(5\)OFTv2’s rebasing note says that its Cayley–Neumann factors generate∏GL\+​\(b\)\\prod\\mathrm\{GL\}^\{\+\}\(b\)and thatKoutK\_\{\\rm out\}drifts\.[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)gives a contraction semigroup, with monotone shrinkage ofKoutK\_\{\\rm out\}, for small generators\.
6. \(6\)Fisher merging’s optional mixing weights would make it a coefficient family under the post\-hoc convention\.
7. \(7\)The entries MoV / MoLoRA and Child\-Tuning \(its D and F variants\) each cover two variants\.
8. \(8\)The “core” flag is not a coordinate\. It marks every method named in Table[22](https://arxiv.org/html/2610.12728#S8.T22)plus a curated set\.

#### The whole catalogue

Everything above rests on a catalogue of 382 methods and 138 papers, checked against arXiv and, for library claims, against Hugging Face PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)as of 30 September 2026; 82 of the methods ship in that library\. Each method entry gives the formula, budget, locus, kind, merge verdict, initialisation, relations, sources, coordinates and a categorical reading, most with arrows and obstructions\. Appendix[D](https://arxiv.org/html/2610.12728#A4)lists every method with its seven coordinates and its reference \(Table[36](https://arxiv.org/html/2610.12728#A4.T36)\) and every arrow with its maphh\(Table[37](https://arxiv.org/html/2610.12728#A4.T37)\); the ancillary files catalogue\.csv, arrows\.csv and obstructions\.csv carry the full entries\. Table[23](https://arxiv.org/html/2610.12728#S8.T23)counts the methods per coordinate value\. Cones and linear shapes account for240240of the 382 methods, and apex and regular base points are about equally common\.

Earlier maps made this one possible: the surveys of[Lialin et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib257),[Xu et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib512),[Han et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib153)and[Mao et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib316), the delta\-tuning study of[Ding et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib93), and the libraries AdapterHub\([Pfeiffer et al\., 2020b](https://arxiv.org/html/2610.12728#bib.bib367)\), Adapters\([Poth et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib373)\)and Hugging Face PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\), whose source settled many details\.

Table 23:The classification of all 382 catalogued methods: the number of methods with each coordinate value, over all methods / over the 142 core methods\. “–” counts null coordinates, including the legacy merge key na\. Per\-method values are in Table[36](https://arxiv.org/html/2610.12728#A4.T36)\.Figure[13](https://arxiv.org/html/2610.12728#S8.F13)dates each method by its first arXiv submission\. Landmarks come in rough order, with bottleneck adapters in 2019\([Houlsby et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib175)\), prefixes\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\)and LoRA’s cone\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)in 2021, the torus of \(IA\)3in 2022\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\), base change with QLoRA in 2023\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\), and GaLore’s moving frame in 2024\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\), the year of113113of the 382 methods\.

Figure 13:The catalogue in time\.Each of the 382 catalogued methods is one dot, placed at the month of its paper’s first arXiv submission, read from the YYMM prefix of its identifier; within a month the dots are spread evenly in the order of their arXiv numbers, and the six entries without an identifier sit at mid\-year\. The dots form a centred beeswarm, coloured by modification kind \(legend, with the number of entries of each kind\) and ringed when the method ships in Hugging Face PEFT \(82 entries\)\. The one entry before 2017, linear probing on DeCAF features\([Donahue et al\., 2013](https://arxiv.org/html/2610.12728#bib.bib98)\), sits left of the axis break\. The number under each year counts its entries, from1010in 2020 to113113in 2024; the busiest month is February 2024, with2020\. The 2026 count ends at the catalogue’s build date, 2 October 2026 \(dotted line\)\. Labels mark the bottleneck adapter\([Houlsby et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib175)\), prefix tuning\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\), LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\), \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\), QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\), DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\), GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)and the newest entry, FrameFT\([Adepu et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib3)\)\. Additive weight edits are the largest kind \(120120entries\), followed by composition and routing \(6565\) and optimizer\-side methods \(5151\)\.
#### What a budget buys

[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)separates the budget\|M\|\\lvert M\\rvert, what a method trains, from the expressive dimensiond⁡\(M\)d\(M\), the number of directions in which it can move the weights\. The gap is gauge waste \([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\): per matrix,r2r^\{2\}for LoRA, one for VeRA \(numerically\), zero for injective linear methods, and at least2​r2\+m\+n−12r^\{2\}\+m\+n\-1for LoHar,r\([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. At131,072131\{,\}072parameters on a4096×40964096\\times 4096matrix, LoRA16reaches130,816130\{,\}816directions at rank up to1616, and LoHa8,8at most122,753122\{,\}753at rank up to6464\. Rankings at matched budget and at matchedd⁡\(M\)d\(M\)can differ \([Prediction X\.10](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3.Px1)\)\. Figure[4](https://arxiv.org/html/2610.12728#S2.F4)compares maximum rank andd⁡\(M\)d\(M\)at equal budget for eight methods on such a layer, and the companion site’s budget calculator computes both counts for the adapted matrices of common base models\.

### VIII\.C Arrow rules: whenNNsimulatesMM

“Generalises” and “is a special case of” get one reading here\.NNsimulatesMMwhen a pointed smooth maph:QM→QNh:Q\_\{M\}\\to Q\_\{N\}satisfiesρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\), soNNreaches every weight thatMMreaches\. A morphismM→NM\\to Nis exhibited by an explicit pointedhh\. Five rules generate every arrow in the atlas, and three tests rule arrows out \(Table[24](https://arxiv.org/html/2610.12728#S8.T24)\)\.

Table 24:The arrow rules A1–A5, transport T and the obstruction tests O1–O3\. An arrowM→NM\\to Nis a pointed smooth maph:QM→QNh:Q\_\{M\}\\to Q\_\{N\}withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\.RuleStatementThe maphh, or the testExamplesA1
intervalFrozen→M→\{\}\\to M\\to\{\}Full FT for every weight\-spaceMM\.h=constq0h=\\mathrm\{const\}\_\{q\_\{0\}\};h=ρMh=\\rho\_\{M\}Merging is the arrow to Full FT\.A2
freezingFreezing coordinates at their base values gives a sub\-object\.h⁡\(q′\)=\(q′,q0′′\)h\(q^\{\\prime\}\)=\(q^\{\\prime\},q\_\{0\}^\{\\prime\\prime\}\)FA⁡\(A0\)↪\\mathrm\{FA\}\(A\_\{0\}\)\\hookrightarrow\{\}LoRA pointed at\(0,A0\)\(0,A\_\{0\}\); MiCA↪\{\}\\hookrightarrow\{\}LoRA pointed at\(B0,0\)\(B\_\{0\},0\)\.A3
factorisationρM=ρN∘h\\rho\_\{M\}=\\rho\_\{N\}\\circ hfor an explicit pointed maphh, usually polynomial\.VeRA→FA⁡\(A\)\{\}\\to\\mathrm\{FA\}\(A\):h⁡\(b,d\)=Λb​B​Λdh\(b,d\)=\\Lambda\_\{b\}B\\Lambda\_\{d\}\. LoHar1,r2→\{\}\_\{r\_\{1\},r\_\{2\}\}\\to\{\}LoRAr1​r2\{\}\_\{r\_\{1\}r\_\{2\}\}: face\-splitting products\. \(IA\)3→\{\}^\{3\}\\to\{\}HiRA1pointed at\(0,𝟏⊤\)\(0,\\mathbf\{1\}^\{\\top\}\):h⁡\(ℓ\)=\(ℓ−𝟏,𝟏⊤\)h\(\\ell\)=\(\\ell\-\\mathbf\{1\},\\mathbf\{1\}^\{\\top\}\)\.Isomorphisms: rsLoRA≅\{\}\\cong\{\}LoRA byh=\(r​B,A\)h=\(\\sqrt\{r\}\\,B,A\); MiSS \(balance mode\)≅FA⁡\(𝟏⊤⊗Ir\)\{\}\\cong\\mathrm\{FA\}\(\\mathbf\{1\}^\{\\top\}\\otimes I\_\{r\}\); a linear adapter parallel to one layer≅\{\}\\cong\{\}LoRA\.A4
productsA product comes with its two projections\.R↦B0​RR\\mapsto B\_\{0\}R,R↦R​A0R\\mapsto RA\_\{0\}LoRA\-XS→FA⁡\(A0\)\{\}\\to\\mathrm\{FA\}\(A\_\{0\}\),FB⁡\(B0\)\\mathrm\{FB\}\(B\_\{0\}\)\. HRA’s image lies in those of OFTSOand LoRAr, an image\-level inclusion only\.A5
sumsM→M⊕NM\\to M\\oplus N, andKKstages embed inK\+1K\+1\.q↦\(q,q0N\)q\\mapsto\(q,q\_\{0\}^\{N\}\)LoRAr⊕\{\}\_\{r\}\\oplus\{\}LoRAs≅\{\}\_\{s\}\\cong\{\}LoRAr\+s\(unpointed\); ReLoRAK→\{\}^\{K\}\\to\{\}ReLoRAK\+1\.T
transportT∗T\_\{\*\}forT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\)is an automorphism of the slice, giving one object in two frames\.no arrow; record the frameTTSums of Kronecker products areℛ∗−1\\mathcal\{R\}^\{\-1\}\_\{\*\}\{\}LoRA; HiRA is\(DW0\)∗\(D\_\{W\_\{0\}\}\)\_\{\*\}\{\}LoRA atθ0\\theta\_\{0\}whenW0W\_\{0\}has no zero entry\.O1
imageNo arrow ifIm⁡M⊈Im⁡N\\Imop M\\not\\subseteq\\Imop N, as whenNNkeeps an invariant thatMMbreaks\.compare imagesLoRA↛\{\}\\not\\to\{\}exact\-Cayley OFT, which keepsKoutK\_\{\\rm out\}\.O2
sizeNo arrow ifd⁡\(M\)\>d⁡\(N\)d\(M\)\>d\(N\)orρmax​\(M\)\>ρmax​\(N\)\\rho\_\{\\max\}\(M\)\>\\rho\_\{\\max\}\(N\)\.compare dimension and maximum rankLoRAr↛\{\}\_\{r\}\\not\\to\{\}LoRA\-FA, sincer⁡\(m\+n−r\)\>m​rr\(m\+n\-r\)\>mr\.O3
first orderNo arrow ifS1​\(M\)⊈S1​\(N\)S\_\{1\}\(M\)\\not\\subseteq S\_\{1\}\(N\), or ifMMandNNlie in different strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\.compareS1S\_\{1\}and strataTA\\mathrm\{T\}\_\{A\}\-↛TB\{\}\\not\\to\{\}\\mathrm\{T\}\_\{B\}\-pointed LoRA; PiSSA and LoRA, either way; LoRA↛\{\}\\not\\to\{\}AdaLoRA, though on one matrix their images agree\.#### The five rules

Write LoRA\(B0,A0\)\{\}\_\{\(B\_\{0\},A\_\{0\}\)\}for LoRA pointed at\(B0,A0\)\(B\_\{0\},A\_\{0\}\), and letFA⁡\(A0\):B↦W0\+B​A0\\mathrm\{FA\}\(A\_\{0\}\):B\\mapsto W\_\{0\}\+BA\_\{0\}andFB⁡\(B0\):A↦W0\+B0​A\\mathrm\{FB\}\(B\_\{0\}\):A\\mapsto W\_\{0\}\+B\_\{0\}Abe the two one\-sided LoRAs of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\.

- •A1, interval\.Frozen→M→\{\}\\to M\\to\{\}Full FT, withh=constq0h=\\mathrm\{const\}\_\{q\_\{0\}\}andh=ρMh=\\rho\_\{M\}respectively\.
- •A2, freezing gives sub\-objects\.Freezing a block of coordinatesq′′q^\{\\prime\\prime\}at its base valueq0′′q\_\{0\}^\{\\prime\\prime\}gives a monomorphismh⁡\(q′\)=\(q′,q0′′\)h\(q^\{\\prime\}\)=\(q^\{\\prime\},q\_\{0\}^\{\\prime\\prime\}\)\. Examples:FA⁡\(A0\)↪\\mathrm\{FA\}\(A\_\{0\}\)\\hookrightarrow\{\}LoRA\(0,A0\)\{\}\_\{\(0,A\_\{0\}\)\},FB⁡\(B0\)↪\\mathrm\{FB\}\(B\_\{0\}\)\\hookrightarrow\{\}LoRA, MiCA↪\{\}\\hookrightarrow\{\}LoRA\(B0,0\)\{\}\_\{\(B\_\{0\},0\)\}, and DyLoRAk↪\{\}\_\{k\}\\hookrightarrow\{\}DyLoRAk\+1by zero padding\.
- •A3, factorisation through a polynomial map\.IfρM=ρN∘h\\rho\_\{M\}=\\rho\_\{N\}\\circ hfor an explicit smooth pointedhh, thenM→NM\\to N; one checksh⁡\(q0M\)=q0Nh\(q\_\{0\}^\{M\}\)=q\_\{0\}^\{N\}\. Examples: - –VeRA→FA⁡\(A\)\{\}\\to\\mathrm\{FA\}\(A\),h=\(Λb​B​Λd\)h=\(\\Lambda\_\{b\}B\\Lambda\_\{d\}\); - –NOLA→\{\}\\to\{\}LoRAr,h=\(∑iαi​Bi,∑jβj​Aj\)h=\(\\sum\_\{i\}\\alpha\_\{i\}B\_\{i\},\\sum\_\{j\}\\beta\_\{j\}A\_\{j\}\); - –LoHar1,r2→\{\}\_\{r\_\{1\},r\_\{2\}\}\\to\{\}LoRAr1​r2\{\}\_\{r\_\{1\}r\_\{2\}\}, by the face\-splitting producth=\(B1∙B2,\(A1⊤∙A2⊤\)⊤\)h=\\big\(B\_\{1\}\\bullet B\_\{2\},\(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\}\\big\)\([Background B\.13](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)\); - –LoRAr↪\{\}\_\{r\}\\hookrightarrow\{\}LoRAr\+1,h=\(\[B​0\],\[A;a0\]\)h=\\big\(\[B\\ 0\],\[A;a\_\{0\}\]\\big\); - –DyLoRAk↪\{\}\_\{k\}\\hookrightarrow\{\}DyLoRAk\+1; - –LoRA\-XS→\{\}\\to\{\}PiSSA,h\(R\)=\(Ur\(Sr\+R\)Sr−1/2,Sr1/2Vr⊤\)h\(R\)=\\big\(U\_\{r\}\(S\_\{r\}\+R\)S\_\{r\}^\{\-1/2\},\\ S\_\{r\}^\{1/2\}V\_\{r\}^\{\\top\}\\big\); - –HRAr→\{\}\_\{r\}\\to\{\}LoRArpointed ath⁡\(q0\)h\(q\_\{0\}\), viaR−I=∑iH1⋯Hi−1\(Hi−I\)R\-I=\\sum\_\{i\}H\_\{1\}\\cdots H\_\{i\-1\}\(H\_\{i\}\-I\); - –\(IA\)3→\{\}^\{3\}\\to\{\}HiRA1pointed at\(0,𝟏⊤\)\(0,\\mathbf\{1\}^\{\\top\}\),h⁡\(ℓ\)=\(ℓ−𝟏,𝟏⊤\)h\(\\ell\)=\(\\ell\-\\mathbf\{1\},\\mathbf\{1\}^\{\\top\}\); - –isomorphisms: MiSS≅FA⁡\(𝟏⊤⊗Ir\)\{\}\\cong\\mathrm\{FA\}\(\\mathbf\{1\}^\{\\top\}\\otimes I\_\{r\}\); rsLoRA≅\{\}\\cong\{\}LoRA,h=\(r​B,A\)h=\(\\sqrt\{r\}\\,B,A\); a linear adapter parallel to one linear box≅\{\}\\cong\{\}LoRA \([Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)\)\.
- •A4, products and meets\.A product comes with two projections, and it is universal among the methods that both factors simulate\. Examples: LoRA\-XS→FA⁡\(A0\)\{\}\\to\\mathrm\{FA\}\(A\_\{0\}\)and LoRA\-XS→FB⁡\(B0\)\{\}\\to\\mathrm\{FB\}\(B\_\{0\}\), at the level of objects; HRA→\{\}\\to\{\}OFTSOand HRA→\{\}\\to\{\}LoRAr, at the level of images only\. An intersection of images gives only inclusions of images, not arrows of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}; for exampleIm\\Imop\{\}LoRAr∩Im\{\}\_\{r\}\\cap\\Imop\{\}\(IA\)3exists only as a set, and there is no product in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\.
- •A5, sums\.M→M⊕NM\\to M\\oplus N,q↦\(q,q0N\)q\\mapsto\(q,q\_\{0\}^\{N\}\), for additiveNNpointed atθ0\\theta\_\{0\}; for example LoRAr→\{\}\_\{r\}\\to\{\}LoRAr⊕\{\}\_\{r\}\\oplus\{\}sparse \(RoSA,[Nikdan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib350)\)\. Also LoRAr⊕\{\}\_\{r\}\\oplus\{\}LoRAs≅\{\}\_\{s\}\\cong\{\}LoRAr\+s, unpointed \([Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\), by concatenation with theAA\-blocks rescaled underα/r\\alpha/rscales; it is pointed exactly at the concatenated initialisation, which is a standard zero\-BBpointing iff the stackedA0A\_\{0\}has full row rank andr\+s<min⁡\(m,n\)r\+s<\\min\(m,n\)\. Rebasing stages giveM⊕K→M⊕\(K\+1\)M^\{\\oplus K\}\\to M^\{\\oplus\(K\+1\)\}\.

##### Transport is not an arrow

T∗T\_\{\*\}forT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\)is an automorphism of the slice \([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\. Methods related by transport, such as the Kronecker family and LoRA \(acrossΘ′≅Θ\\Theta^\{\\prime\}\\cong\\Theta\), are “the same object in another frame”\. Sums ofkkKronecker terms areℛ∗−1\\mathcal\{R\}^\{\-1\}\_\{\*\}\{\}LoRAkafter the Van Loan–Pitsianis rearrangement \([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\); HiRA is\(DW0\)∗\(D\_\{W\_\{0\}\}\)\_\{\*\}\{\}LoRA inside the slice atθ0\\theta\_\{0\}whenW0W\_\{0\}has no zero entries \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\); FourierFT is a mask in the real cosine Fourier frame, a transport only without sampled conjugate pairs and only for init\_weights=True\. Transports that depend onθ0\\theta\_\{0\}\(HiRA, SVD frames\) agree with their source only at the base point\. For additive families and a base\-independentTT, transport also commutes with merge\-and\-restart; for action\-type orθ0\\theta\_\{0\}\-dependent transports it need not\.

#### Three tests

There is no arrowM→NM\\to Nif any of the following holds\.

- •O1, image\.Im⁡M⊈Im⁡N\\Imop M\\not\\subseteq\\Imop N, for instance becauseNNpreserves an absolute invariant \(Gram matrix, spectrum, row lines, zero pattern\) thatMMbreaks\. Examples: LoRA↛\{\}\\not\\to\{\}OFT, LoRA↛\{\}\\not\\to\{\}\(IA\)3, DoRA↛\{\}\\not\\to\{\}LoRAr\.
- •O2, dimension and cut\-rank\.d⁡\(M\)\>d⁡\(N\)d\(M\)\>d\(N\), orρmax​\(M\)\>ρmax​\(N\)\\rho\_\{\\max\}\(M\)\>\\rho\_\{\\max\}\(N\)\. Examples: LoHar,rdoes not map into LoRAr′\{\}\_\{r^\{\\prime\}\}forr′<r2r^\{\\prime\}<r^\{2\}, since its generic rank isr2r^\{2\}; and LoRArdoes not map intokk\-term Kronecker sums fork<min⁡\(m1,m2\)​min⁡\(n1,n2\)k<\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\), the Kronecker rank of a generic rank\-one update \([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\)\.
- •O3, first order and base point\.S1​\(M\)⊈S1​\(N\)S\_\{1\}\(M\)\\not\\subseteq S\_\{1\}\(N\), orMMandNNlie in different strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(Type I apex against Type II smooth point\); theS1S\_\{1\}test applies even when the images agree\. Examples:TA\\mathrm\{T\}\_\{A\}\-stratum LoRA↛TB\{\}\\not\\to\{\}\\mathrm\{T\}\_\{B\}\-stratum LoRA; PiSSA↛\{\}\\not\\to\{\}LoRA and LoRA↛\{\}\\not\\to\{\}PiSSA\.

##### Why the rules give arrows and the tests rule them out

Each rule and test is an instance of a result stated earlier\. In each rulehhis smooth and pointed, andρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}is checked by multiplying out\. A1 is[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\), sinceρM∘constq0=θ0\\rho\_\{M\}\\circ\\mathrm\{const\}\_\{q\_\{0\}\}=\\theta\_\{0\}andid∘ρM=ρM\\mathrm\{id\}\\circ\\rho\_\{M\}=\\rho\_\{M\}\. A2 holds because freezingq′′q^\{\\prime\\prime\}atq0′′q\_\{0\}^\{\\prime\\prime\}leaves the rest of the formula unchanged\. For A3, take VeRA, whereW0\+Λb​B​Λd​A=W0\+\(Λb​B​Λd\)​AW\_\{0\}\+\\Lambda\_\{b\}B\\Lambda\_\{d\}A=W\_\{0\}\+\(\\Lambda\_\{b\}B\\Lambda\_\{d\}\)Aandhhsendsb0=0b\_\{0\}=0to the base point00of LoRA\-FA\. A4 is the universal property of the fibre product \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\),[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)\); for a meet such as HRA only the inclusion of images holds \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\. A5 is the identity

ρM⊕N​\(q,q0N\)=θ0\+δM​\(q\)\+δN​\(q0N\)=ρM​\(q\),\\rho\_\{M\\oplus N\}\(q,q\_\{0\}^\{N\}\)=\\theta\_\{0\}\+\\delta\_\{M\}\(q\)\+\\delta\_\{N\}\(q\_\{0\}^\{N\}\)=\\rho\_\{M\}\(q\),\(10\)from[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\.

An arrow givesIm⁡M=ρN​\(h⁡\(QM\)\)⊆Im⁡N\\Imop M=\\rho\_\{N\}\(h\(Q\_\{M\}\)\)\\subseteq\\Imop N, which is O1\. Images are semialgebraic, or definable when softmax enters \([Background C\.17](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\), so dimension is monotone under inclusion; and every update inIm⁡M−θ0\\Imop M\-\\theta\_\{0\}also lies inIm⁡N−θ0\\Imop N\-\\theta\_\{0\}, so its rank is bounded byρmax​\(N\)\\rho\_\{\\max\}\(N\)\. Together these give O2\. The chain rule atq0q\_\{0\}givesS1​\(M\)⊆S1​\(N\)S\_\{1\}\(M\)\\subseteq S\_\{1\}\(N\)\([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\), the first half of O3\. For the strata, the images ofT\\mathrm\{T\}\-type andS\\mathrm\{S\}\-type objects are incomparable \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\)\)\. Inside theT\\mathrm\{T\}strata,S1​\(TA\)=\{X​A0\}S\_\{1\}\(\\mathrm\{T\}\_\{A\}\)=\\\{XA\_\{0\}\\\}contains matrices whose column space leavescol​B0′\\mathrm\{col\}\\,B\_\{0\}^\{\\prime\}, so it does not lie inS1​\(TB\)=\{B0′​Y\}S\_\{1\}\(\\mathrm\{T\}\_\{B\}\)=\\\{B\_\{0\}^\{\\prime\}Y\\\}, and symmetrically\.

TransportT∗T\_\{\*\}is an automorphism of the slice with inverse\(T−1\)∗\(T^\{\-1\}\)\_\{\*\}\([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\. It identifiesMMandT∗​MT\_\{\*\}Mas one object in two frames and does not by itself give an arrow between them\.

##### Which published relations survive

Some become exact\. VeRA is LoRA\-FA\(A\)\(A\)with a constrained factor, and rsLoRA and a linear parallel adapter are LoRA\. Others are weaker than they sound\. HRA’s arrow to LoRArlands on a degenerate pointing \(B0​A0=0B\_\{0\}A\_\{0\}=0\) outside the strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), and LoRA is a special case of AdaLoRA only unpointed, since their first\-order images have dimensionsm​rmrandrr\([Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4)\)\.

#### The recorded arrows and obstructions

The atlas records 403 arrows, each with its maphh\(Table[37](https://arxiv.org/html/2610.12728#A4.T37)in Appendix[D](https://arxiv.org/html/2610.12728#A4)\):329329by factorisation \(A3\),6161by freezing \(A2\),1111by sums \(A5\) and22by products \(A4\)\. The interval arrows of A1 hold for every weight\-space method and are not listed\. The atlas also records 485 obstructions, each naming the test that rules an arrow out \(Table[25](https://arxiv.org/html/2610.12728#S8.T25)\)\. Of the recorded arrows,161161join two core methods, and Figure[14](https://arxiv.org/html/2610.12728#S8.F14)draws the category of core methods as a layered Hasse diagram \([Background A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\)\.

Table 25:The 485 recorded obstructionsM↛NM\\not\\to N, by the test that rules the arrow out\. The full list, with the failing invariant of each, is the ancillary file obstructions\.csv\.Figure 14:The category of core methods\.A layered Hasse diagram \([Background A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\), from Frozen \(initial, bottom\) to Full FT \(terminal, top\); the dashed outer frame stands for the A1 arrows Frozen→M→\{\}\\to M\\to\{\}Full FT of every weight\-space methodMM\([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\)\)\. Chips are the 142 core methods, coloured by modification kind and dashed for the5757that are not objects of the weight\-space slice\. An arrowM→NM\\to Npoints up and means thatNNsimulatesMMthrough a pointed smooth maphhwithρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\)\. All161161recorded arrows between two core methods are drawn, coloured by rule \(Table[24](https://arxiv.org/html/2610.12728#S8.T24)\):2929by freezing \(A2, hooked tail\),122122by factorisation \(A3\),22by products \(A4\) and88by sums or stages \(A5\)\. The77loops land on another member of a family, asLoRAr→LoRAr\+1\\mathrm\{LoRA\}\_\{r\}\\to\\mathrm\{LoRA\}\_\{r\+1\}does, and the55classes of mutually simulating families share a rung in a grey box\. Arrows into a method with four or more incoming arrows from outside its class end in one trunk \(5757into LoRA\), and the2828methods whose arrows all land in the LoRA class form the comb beneath it\. The2626lighter arrows are composites, also reached by a longer path of arrows\.The atlas is meant to grow\. A new method enters by answering the seven questions and exhibiting its arrows, each with its maphh; a disputed comparison is settled by an explicithhor by one of the three tests\.

## EXPOSÉ IXRosetta stone

A half\-seen analogy between two theories is the most fertile thing a mathematician can hold\. One day the hunch turns into certainty, and the twin theories show the source they share\. after Weil’s Rosetta\-stone letter of 1940

This exposé is a dictionary from fine\-tuning phrases to the objects of the relative theory, with the result behind each pairing that needs one\.

##### In brief

- •LoRA\+ isα\\alphain disguise\. Under Adam withε=0\\varepsilon=0, no weight decay or clipping and one schedule, LoRA\+ with ratioλ\\lambdatrains exactly like LoRA withα\\alphamultiplied byλ\\lambda, given the same data order and dropout masks\([Schulman and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.12728#bib.bib401)\)\.
- •Under Adam without weight decay or full\-gradient clipping, GaLore is one\-sided LoRA, restarted every period with its optimizer state kept\([Torroba\-Hennigen et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib453)\)\. QLoRA is LoRA attached at the quantised weight, offθ0\\theta\_\{0\}by a generically full\-rank error\.
- •Read from the mathematical side, some constructions are known methods\. In the SVD frames ofW0W\_\{0\}, whenrank⁡W0≥r\\rk W\_\{0\}\\geq r, the product of the two one\-sided LoRAs is LoRA\-XS, and for injectiveW0W\_\{0\}and evenr≤n−2r\\leq n\-2LoRA’s image meets the rotation orbit ofW0W\_\{0\}in the image of HRA \(without Gram–Schmidt, the HF default\)\.

A fine\-tuning paper says that LoRA\+ givesBBa larger learning rate thanAA\. The relative theory says that, under the hypotheses of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), LoRA\+ lies on the torus orbit of plain LoRA with a largerα\\alpha\. Passing between such sentences lets each side borrow the other’s theorems\. Table[28](https://arxiv.org/html/2610.12728#S9.T28), at the end of this exposé, is a dictionary for the passage\. It lists all 36 entries, each a PEFT phrase, its categorical or geometric counterpart, and a gloss citing the results behind it; the hypotheses live in the cited statements\. The sections below read six entries closely, one from each of Exposés[I](https://arxiv.org/html/2610.12728#S1)to[VI](https://arxiv.org/html/2610.12728#S6), and then read the dictionary from the right\.

#### How to read an entry

An entry claims that a phrase names a definite object, on which claims about the phrase can be checked\. There are three strengths \(Table[26](https://arxiv.org/html/2610.12728#S9.T26)\)\.

Table 26:The three strengths of a dictionary entry\.##### Credit

The language of parametrised maps and lenses on our right\-hand side comes from[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\. Inside PEFT,[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)wrote an earlier dictionary of this kind, reading adapters, prefix tuning and LoRA as modifications of one hidden state\.

The box below fixes each borrowed word’s sense\. Exposé[XI](https://arxiv.org/html/2610.12728#S11)says more \(Table[30](https://arxiv.org/html/2610.12728#S11.T30)\)\.

Analogy\(the borrowed words\)\.•*Noether*: conserved charges from symmetries, proved for continuous\-time flows with constant rates \([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\); Adam and momentum break them\.•*Gauge*: the symmetry group of a parametrisation, with no connection\.•*Erlangen*: a theorem for group\-type methods \([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\), and a name for the shape of cones such as LoRA’s image\.•*Moduli space*: LoRA initialisations in three full\-rank strata, classified only as a set \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\.•*Lens duality*: exact only at degree one, for optimizers that read neither the parameters nor the full gradient \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.

Most of the right\-hand column concerns images, which measure expressivity\. Learnability is a separate property, and entries with dynamical claims state their optimizer\.

#### Six entries, one from each exposé

##### I\. “MethodNNcan do whateverMMcan”

Example\(entries 3 and 4, Exposé[I](https://arxiv.org/html/2610.12728#S1)\)\.PEFT“methodNNcan do whateverMMcan”; the frozen model and full fine\-tuning\.CATEGORY THEORY · GEOMETRYa morphismh:M→Nh:M\\to Nof the pointed slice, a pointed smooth map withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}; the initial and terminal objects\.

The weak reading asks thatNNreach every weightMMreaches\. The useful one asks thatNNalso followMM’s parameter paths, through a maphhwithρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Methods atθ0\\theta\_\{0\}with such arrows form the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\)\. The frozen model is initial, full fine\-tuning is terminal, and the arrow fromMMto full fine\-tuning isρM\\rho\_\{M\}itself \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\)\)\. Image inclusion gives only a map of sets overΘ\\Theta, which need not be smooth \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\)\)\.

WHAT THIS EXPLAINSEvery weight\-space method pointed atθ0\\theta\_\{0\}lies between the frozen model and full fine\-tuning, and in exact arithmetic on a full\-precision base its merge is that canonical arrow\. Merging questions remain for extensions such as adapters, prefixes and ReFT \(reading VI below\), for quantised bases \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\) and for combinations of adapters \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.

##### A first\-order test, and defaults that start elsewhere

The first\-order imageS1S\_\{1\}detects part of the gap between sets and smooth maps\. TakeρM​\(t\)=t\\rho\_\{M\}\(t\)=tandρN​\(u\)=u3\\rho\_\{N\}\(u\)=u^\{3\}onℝ\\mathbb\{R\}, both pointed at00\. The images agree, butS1​\(M\)=ℝS\_\{1\}\(M\)=\\mathbb\{R\}is not insideS1​\(N\)=0S\_\{1\}\(N\)=0, so no smooth simulationM→NM\\to Nexists \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. Several initialisations start away fromθ0\\theta\_\{0\}and carry a pointing defect \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\): HF’s default FourierFT spectrum and VB\-LoRA bank, init\_lora\_weights=False, LoRA\-XS’s small randomRR, and quantised bases\.

##### II\. “The effective number of parameters”

Example\(entries 6 and 11, Exposé[II](https://arxiv.org/html/2610.12728#S2)\)\.PEFTexpressivity, the effective number of parameters; redundant parameters\.CATEGORY THEORY · GEOMETRYthe image germ ofρ\\rhoatθ0\\theta\_\{0\}and its dimensiond⁡\(M\)d\(M\); the gauge group \(automorphisms ofQQoverΘ\\Theta\) and the generic fibre\.

A parameter count measuresQQ, while expressivity lives in the image\. The difference is spent on the fibres ofρ\\rho, along which the gauge group moves\. LoRA spendsr2r^\{2\}parameters there, since\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)and\(B,A\)\(B,A\)give the same weights for everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\.

With\|M\|=dimQ\\lvert M\\rvert=\\dim Qandd⁡\(M\)=dimIm⁡Md\(M\)=\\dim\\Imop M\([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\),

d⁡\(M\)≤\|M\|−dim\(generic gauge orbit\),d\(M\)\\leq\\lvert M\\rvert\-\\dim\(\\text\{generic gauge orbit\}\),\(11\)with equality when the gauge is locally transitive on generic fibres \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(a\)\)\. ForLoRAr\\mathrm\{LoRA\}\_\{r\}on anm×nm\\times nbox withr≤min⁡\(m,n\)r\\leq\\min\(m,n\),d=r⁡\(m\+n−r\)d=r\(m\+n\-r\)\. CompareLoHar,r\\mathrm\{LoHa\}\_\{r,r\}withLoRA2​r\\mathrm\{LoRA\}\_\{2r\}at their shared budget2​r​\(m\+n\)2r\(m\+n\), with2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\. If2​r2<m\+n−12r^\{2\}<m\+n\-1, LoHa reaches at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}fewer dimensions; if moreoverr2≤min⁡\(m,n\)r^\{2\}\\leq\\min\(m,n\), it trades them for rank up tor2r^\{2\}whenr≥3r\\geq 3, and it lies strictly insideLoRA2​r\\mathrm\{LoRA\}\_\{2r\}whenr≤2r\\leq 2\([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(c\)\)\. If2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1, as forr=64r=64on4096×40964096\\times 4096or forr=32r=32on layers 320 to 640 wide, LoHa matches or beatsLoRA2​r\\mathrm\{LoRA\}\_\{2r\}in rank, and in dimension by its numerically checked dimension formula\. Rank, in turn, is bounded by a minimum cut of the method’s tensor network \([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\)\.

WHAT THIS EXPLAINSReportd⁡\(M\)d\(M\)beside\|M\|\\lvert M\\rvertin every comparison \([Prediction X\.10](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3.Px1)\)\.

##### Credit

LoHa is the Hadamard\-product parametrisation of FedPara\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\)\. Its dimension bound is proved, and equality is checked numerically \(Exposé[XI](https://arxiv.org/html/2610.12728#S11), Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

##### III\. “The LoRA\+ ratio,α\\alpha, the init scale, rsLoRA”

Example\(entry 23, Exposé[III](https://arxiv.org/html/2610.12728#S3)\)\.PEFTthe LoRA\+ ratio,α\\alpha, the initial scale, rsLoRA\.CATEGORY THEORY · GEOMETRYcoordinates on the orbit space of a hyperparameter torus\.

MultiplyingBBbyc\>0c\>0and dividing the scalessbyccleaves the weights unchanged, and Adam withε=0\\varepsilon=0ignores a positive rescaling of a block’s gradient history\. Rescaling the block’s learning rate too then reproduces the run step for step, and the per\-block factors form a torus acting on the hyperparameters\.

[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)makes this exact for any multihomogeneous method, assuming no parameter\-dependent regulariser, no global clipping, the same data order and dropout masks, and exact arithmetic\. For LoRA \([Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\(b\)\), under Adam withε=0\\varepsilon=0, no weight decay or clipping and one schedule for both groups, LoRA\+ with ratioλ\\lambdaat scalessproduces exactly the trajectory of plain LoRA at scaleλ​s\\lambda s, with the sameηA\\eta\_\{A\}andA0A\_\{0\}\. Forε\>0\\varepsilon\>0this needsεB′=λ​ε\\varepsilon\_\{B\}^\{\\prime\}=\\lambda\\varepsilon, and with AdamW a decayλ⋅wd\\lambda\\cdot\\mathrm\{wd\}onBB\. Under SGD the scale isλ​s\\sqrt\{\\lambda\}\\,s\.

WHAT THIS EXPLAINSOnly the torus orbit of\(s,ηA,ηB,σA\)\(s,\\eta\_\{A\},\\eta\_\{B\},\\sigma\_\{A\}\)matters\. Under Adam it is fixed bys​ηA​ηBs\\eta\_\{A\}\\eta\_\{B\}andσA/ηA\\sigma\_\{A\}/\\eta\_\{A\}\([Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\(a\)\), so at a fixed rank LoRA\+, rsLoRA andα\\alpha\-tuning are one family\. A gain of LoRA\+ over LoRA withα\\alphamultiplied byλ\\lambdaneeds something that breaks the torus, such as global\-norm clipping \(a Hugging Face Trainer default\), anε\\varepsilonor decay shared across groups, or different schedules \([Prediction X\.1](https://arxiv.org/html/2610.12728#S10.SS0.SSSx2)\)\. Figure[8](https://arxiv.org/html/2610.12728#S3.F8)runs both trainings\.

##### Credit

The LoRA and Adam case, with itsε\>0\\varepsilon\>0extension and the two effective hyperparameters, is due to[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401), for the method of[Hayou et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib159)\. The torus for any multihomogeneous method and thec2c^\{2\}law for SGD are added here\.

##### The same fibres under gradient flow

Entry 22 reads the same fibres under gradient flow\. With block\-scalar rates the gauge yields the conserved charge

Φ=B⊤​BηB−A​A⊤ηA\\Phi=\\frac\{B^\{\\top\}B\}\{\\eta\_\{B\}\}\-\\frac\{AA^\{\\top\}\}\{\\eta\_\{A\}\}\(12\)\([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\), after[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100),[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14),[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580)and[Marcotte et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib318)\. Balanced pointsB⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\}form the Kempf–Ness slice \([Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5);[Kempf and Ness, 1979](https://arxiv.org/html/2610.12728#bib.bib209);[Richardson and Slodowy, 1990](https://arxiv.org/html/2610.12728#bib.bib389)\)\.

##### IV\. “GaLore versus LoRA”

Example\(entry 25, Exposé[IV](https://arxiv.org/html/2610.12728#S4)\)\.PEFTGaLore versus LoRA\.CATEGORY THEORY · GEOMETRYcotangent\-side and tangent\-side lens 2\-cells, which coincide for linear charts \(the degree\-one collapse\)\.

Backpropagation makes each map a lens, running forward from parameters to outputs and backward from output gradients to parameter gradients\. LoRA restricts the forward side\. GaLore compresses the gradient before the optimizer and expands its output after\.

Letρk​\(C\)=Wk\+ℒ​C\\rho\_\{k\}\(C\)=W\_\{k\}\+\\mathcal\{L\}Cwithℒ\\mathcal\{L\}linear andC0=0C\_\{0\}=0\. Let the optimizer’s state and output depend only on the gradients fed to it, with one implementation on both sides \(SGD, heavy\-ball or Nesterov momentum, Adam without weight decay, Adafactor without parameter scaling\)\. Then trainingCCand merging at the end of a period gives the sameWW\-trajectory as the backward method with compressionℒ∗\\mathcal\{L\}^\{\*\}and anchorℒ\\mathcal\{L\}, started atWkW\_\{k\}from the same optimizer state \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\. Decoupled weight decay, LAMB and LARS trust ratios, parameter\-scaled Adafactor and full\-gradient clipping break the identity\.

WHAT THIS EXPLAINSUnder these hypotheses, GaLore with a projectorPkP\_\{k\}fixed on each period is one\-sided LoRAWk\+Pk​CW\_\{k\}\+P\_\{k\}C, merged and restarted every period withCC’s optimizer state carried over unchanged\. It adds two things to one\-sided LoRA, a frame from the gradient’s SVD and that carry\-over, which reads moments written forPk−1P\_\{k\-1\}as if written forPkP\_\{k\}\. LDAdam\([Robert et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib392)\)transports them instead\.

##### Credit

The identity for GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)is due to[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453), and its SGD case with a frozen random factor to[Hao et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib154)\.

##### The rest of the cotangent family

Under the same hypotheses the identity \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3),[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\) covers LoRA\-FA and GaLore with random projectors, MeZO up to its finite\-difference estimate, GaLore with gradient SVDs, and BitFit and LISA \(without weight decay\) with coordinate blocks\. Flora as published, APOLLO and Fira update outside their projected subspace and are not covered\.

##### V\. “QLoRA and LoftQ”

Example\(entry 28, Exposé[V](https://arxiv.org/html/2610.12728#S5)\)\.PEFTQLoRA, LoftQ\.CATEGORY THEORY · GEOMETRYbase change along a map that does not preserve the function; a pointing defect; an approximate section\.

A quantiserκ\\kappamoves the pretrained point toκ​θ0\\kappa\\theta\_\{0\}and changes the function\. Additive methods commute with translation,M⁡\(θ\+v\)=τv∘M⁡\(θ\)M\(\\theta\+v\)=\\tau\_\{v\}\\circ M\(\\theta\), so QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)is LoRA attached atκ​θ0\\kappa\\theta\_\{0\}, with pointing defecte=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\([Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1)\), andθ0\\theta\_\{0\}lies outside its image unlessrank⁡e≤r\\rk e\\leq r\([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(a\)\)\. Splitting initialisations come in three orders with three different defects \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\); Table[27](https://arxiv.org/html/2610.12728#S9.T27)\)\. WriteWWfor the full\-precision weight,δ0\\delta\_\{0\}for the initial adapter,Wres=W−δ0W\_\{\\mathrm\{res\}\}=W\-\\delta\_\{0\}ands=α/rs=\\alpha/r\.

Table 27:Three orders of quantising and splitting, and the pointing defectρ⁡\(q0\)−W\\rho\(q\_\{0\}\)\-Weach leaves \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\)\)\.WHAT THIS EXPLAINSMeasured from its own start, QLoRA obeys the rank bound of any LoRA \([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\)\); measured fromθ0\\theta\_\{0\}, its update also contains the generically full\-rank defect\. Sincee−SVDr​\(e\)e\-\\mathrm\{SVD\}\_\{r\}\(e\)is orthogonal toSVDr​\(e\)\\mathrm\{SVD\}\_\{r\}\(e\), one\-step LoftQ in HF beats QLoRA in Frobenius norm only for0<s<20<s<2\. Further LoftQ iterations need not decrease their objective, and ats=1s=1the output is an approximate section, pointed up to its attained residual \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(c\)\)\.

##### Credit

QLoRA is due to[Dettmers et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib92), LoftQ to[Li et al\. \(2023b\)](https://arxiv.org/html/2610.12728#bib.bib255), and PiSSA and QPiSSA to[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324)\.

##### VI\. “Merging an adapter into the weights”

Example\(entry 5, Exposé[VI](https://arxiv.org/html/2610.12728#S6)\)\.PEFTmerging an adapter into the weights; zero inference latency\.CATEGORY THEORY · GEOMETRYfor weight\-space methods, the canonical arrow to the terminal object; for extensions, a lift along the realisationΦf\\Phi\_\{f\}, certified box by box by rewriting\.

Reading I settled merging for weight\-space methods on a full\-precision base\. Adapters, prefixes and \(IA\)3scalings insert new boxes and live on an extension offf, so merging one is a lifting problem along the realisationθ↦f⁡\(θ,−\)\\theta\\mapsto f\(\\theta,\-\)\.

A methodρ\\rhoon an extensionf~\\tilde\{f\}is mergeable if someμ:Q→Θ\\mu:Q\\to\\ThetahasΦf~∘ρ=Φf∘μ\\Phi\_\{\\tilde\{f\}\}\\circ\\rho=\\Phi\_\{f\}\\circ\\mu\([Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\)\.[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)gives sufficient conditions box by box, through seven rewrite rules with side conditions such as position\-uniform edits, unshared absorbing weights and bias slots for shifts\. For instance, HF applies the feed\-forward vectorℓ\\ellof \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)to the input ofWdownW\_\{\\mathrm\{down\}\}, where it always fuses\. It fuses intoWupW\_\{\\mathrm\{up\}\}instead under SwiGLU or GeGLU for everyℓ\\ell, and under ReLU whenℓ≥0\\ell\\geq 0\. Under GELU, an entryℓi\\ell\_\{i\}on a nonzero row ofWupW\_\{\\mathrm\{up\}\}fuses there only if it is00or11\.

WHAT THIS EXPLAINS“Zero inference latency” is certified by local rewriting, a sufficient test that also names the absorbing weight; theWupW\_\{\\mathrm\{up\}\}route is idle in practice, asWdownW\_\{\\mathrm\{down\}\}always works\. Nonlinear adapters, input\-routed mixtures and prefixes do not merge box\-locally\. At its layer a prefix is a gated parallel adapter \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(b\)\), and it is not box\-locally mergeable for generic frozen weights and prefixes \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\); whole\-network merging of prefixes is open\.

##### Credit

The \(IA\)3merge is noted by[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275), and the gated\-adapter reading of prefix tuning is due to[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)\. Scale migration through SwiGLU, ReLU and norm layers, with the GELU obstruction, already appears in SmoothQuant\([Xiao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib508)\)and AWQ\([Lin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib268)\)\. The activation symmetries behind slides R2 and R3 \([Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\) are classical\([Sussmann, 1992](https://arxiv.org/html/2610.12728#bib.bib440);[Chen et al\., 1993](https://arxiv.org/html/2610.12728#bib.bib58);[Godfrey et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib132)\)\.

#### Reading from the right

Some mathematical constructions are methods already in use\.

- •The productFA⁡\(A0\)×FB⁡\(B0\)\\mathrm\{FA\}\(A\_\{0\}\)\\times\\mathrm\{FB\}\(B\_\{0\}\)of the two one\-sided LoRAs is the sandwichR↦W0\+B0​R​A0R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}\. In the SVD frames ofW0W\_\{0\}, withσr\>0\\sigma\_\{r\}\>0, it is LoRA\-XS pointed atR=0R=0\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\);[Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\)\.
- •For injectiveW0W\_\{0\}and evenrrwith2≤r≤n−22\\leq r\\leq n\-2, LoRA’s image meets the rotation orbitW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)exactly in the image of HRA without Gram–Schmidt, the HF default \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3);[Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)\. This is a meet of sets, and HRA is not the categorical product\.
- •Transport along a fixedT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\)is an automorphism of the slice\. It carries coordinate masks to FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\)\(with no conjugate frequency pair sampled\), WaveFT\([Bilican et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib35)\)\(when its padded inverse wavelet transform is invertible\) and C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\), each started at zero, and to LoRA\-XS and SVFT\([Lingam et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib272)\)in the SVD frame ofW0W\_\{0\}, which depends onθ0\\theta\_\{0\}\. The inverse of the rearrangement of[Van Loan and Pitsianis \(1993\)](https://arxiv.org/html/2610.12728#bib.bib459)carriesLoRAk\\mathrm\{LoRA\}\_\{k\}to unconstrained sums ofkkKronecker products \([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.
- •The monoidal sum⊕\\oplusadds updates, so its image is a Minkowski sum\. RoSA\([Nikdan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib350)\), GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\)and the stages of ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\), unrolled in time, are words in⊕\\oplus\([Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\)\.

WHAT THIS EXPLAINSProducts and meets can be tried on other pairs too\. Meets of images always exist as sets of weights, while products can fail to exist \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\)\. Which meets admit a polynomial chart, as LoRA\-XS and HRA’s image do, is an open problem \(§[XI](https://arxiv.org/html/2610.12728#S11.SS0.SSSx2), problem 4\)\.

#### The full dictionary

Table[28](https://arxiv.org/html/2610.12728#S9.T28)gives every entry\. Result numbers in the glosses point to the statements whose hypotheses the entry inherits; entries without a number are definitions or bookkeeping\.

Table 28:Rosetta stone: the full dictionary from PEFT phrases to their categorical or geometric counterparts\. The gloss names the results behind each pairing; their hypotheses are part of the entry\.\#PEFTCategorical / geometricGloss1pretrained checkpointbase pointθ0:1→Θ\\theta\_\{0\}:1\\to\\Theta; a pointed 1\-cell\(Θ,θ0,f\)\(\\Theta,\\theta\_\{0\},f\)of𝐏𝐚𝐫𝐚⁡\(𝐃𝐢𝐟𝐟\)\\mathbf\{Para\}\(\\mathbf\{Diff\}\)Fine\-tuning moves a point; everything is measured relative to it \([Def\. I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\)\.2PEFT methodobject of the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\), equivalently the opposite of the pointed coslice of reparametrisation 2\-cells underFFA method is a pointed mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\)\. The slice depends only on\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\);𝐏𝐚𝐫𝐚\\mathbf\{Para\}is bookkeeping for weight\-space methods \([Rem\. I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\)\.3“methodNNcan do whateverMMcan”morphism \(simulation\)h:M→Nh:M\\to NwithρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}Smooth simulations also carry parameter paths; at the level of sets they reduce to image inclusion \([Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\)\.4frozen model / full fine\-tuninginitial / terminal objectEvery weight\-space method lies in the interval \[frozen, full FT\]; adapters, prefixes and ReFT are extensions, not objects of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}\([Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\)\.5merging an adapter into the weightsthe canonical morphism to the terminal object \(it isρ\\rhoitself\); for extensions, a lift along the realisationΦf\\Phi\_\{f\}For weight\-space methods merging is definitional; for extensions zero inference overhead is a lifting problem, certified box\-locally by rewriting \(sufficient conditions and box\-local obstructions\) under side conditions such as position\-uniformity, no weight sharing and bias slots \([Thm VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\)\.6expressivity / effective number of parametersimage germ ofρ\\rhoatθ0\\theta\_\{0\}; its dimensiond⁡\(M\)d\(M\), the generic rank ofd​ρd\\rhoRank is a cut invariant; dimension is a volume invariant \([Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1),[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\.7lazy / NTK\-regime capabilityfirst\-order imageS1=d​ρq0​\(Tq0​Q\)S\_\{1\}=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\), a functorial isomorphism invariantThe lazy kernel has rangeJf​\(S1\)J\_\{f\}\(S\_\{1\}\);S1S\_\{1\}separates zero\-init pointings \(TA\\mathrm\{T\}\_\{A\}vsTB\\mathrm\{T\}\_\{B\}, the row space ofA0A\_\{0\}\) that the image cannot, while the image already separates split inits \([Obs\. I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\.8zero\-init vs PiSSA\-type initapex \(singular vertex\) vs smooth point of the image; Type I \(newq0q\_\{0\}in the apex fibre\) vs Type II \(newρ\\rhoby splitting\)Zero\-BBinit losesr⁡\(n−r\)r\(n\-r\)first\-order directions \(zero\-AAinitr⁡\(m−r\)r\(m\-r\)\); split inits are different objects, incomparable with LoRA \([Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Prop\. III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\)\.9the zoo of LoRA initialisationspoints of three distinguished strata of full\-rank pointings, classified as a set byGr⁡\(r,n\)⊔Gr⁡\(r,m\)⊔ℳr\\Gr\(r,n\)\\sqcup\\Gr\(r,m\)\\sqcup\\mathcal\{M\}\_\{r\}Vanilla LoRA on linear layers, EVA, Init\[B\], HF’s default LoRA on embedding layers, PiSSA, MiLoRA, OLoRA, CorDA, LoRA\-GA and Astra are points; LoRA\-FA and MiCA are frozen\-factor sub\-objects of the zero\-BBand zero\-AAstrata; HF’s orthogonal init lies outside the three strata, and LoftQ/QLoRA are defect pointings\. The optimizer sees a finer quotient \([Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Cor\. III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.10no init buys rank above2​r2runion over pointings,⋃qIm⁡Mq=θ0\+\(C−C\)=θ0\+ℳ≤min⁡\(2​r,m,n\)\\bigcup\_\{q\}\\Imop M\_\{q\}=\\theta\_\{0\}\+\(C\-C\)=\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(2r,m,n\)\}An exact initialisation picks a translate of the shape; from any starting point the update has rank at most2​r2r\. This is the known rank\-2​r2rPiSSA\-to\-LoRA conversion \([Thm II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\.11redundant parametersgauge group, the automorphisms ofQQoverΘ\\Theta; generic fibreLoRA wastesr2r^\{2\}parameters; LoHa wastes at leastr12\+r22\+m\+n−1r\_\{1\}^\{2\}\+r\_\{2\}^\{2\}\+m\+n\-1below them​nmncap \(exactly that many, numerically\), and more where the cap binds \([Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\.12high\-rank PEFT \(LoHa, LoKr, HiRA, MoRA, C3A\)tensor networks whose min\-cut avoids the bond; transport of structure alongGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)High rank is notGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-invariant \(though rank isGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\-invariant\) and need not mean large volume; a large min\-cut is necessary for high rank and is not sufficient \([Thms II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)–[II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Prop\. II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.13Kronecker adapters \(unconstrained Kronecker sums, KronA, LoKr\)LoRA across a different cut of the same 4\-leg tensor \(Van Loan–Pitsianis rearrangement, a coordinate permutation\)For aligned splits \(equal square splits, and every split HF PEFT’s factorisation produces\), a generic rank\-one update has maximal Kronecker rank, so unconstrained Kronecker sums with fewer terms and zero\-initLoRAr\\mathrm\{LoRA\}\_\{r\}below the rank of a full\-rank Kronecker product are incomparable\. LoKr with inner rankr′≤min⁡\(m2,n2\)r^\{\\prime\}\\leq\\min\(m\_\{2\},n\_\{2\}\)sits inside zero\-initLoRAr\\mathrm\{LoRA\}\_\{r\}exactly whenr≥min⁡\(m1,n1\)​r′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}; Compacter’s PHM weights live in a nonlinear adapter \([Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\)\.14“new method = old method”transportT∗T\_\{\*\}alongT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\): an automorphism of the category, not an arrow inside itFourierFT \(when no conjugate frequency pair is sampled\), WaveFT, C3A, LoRA\-XS, SVFT and SHiRA are coordinate masks in transported frames, and HiRA \(forW0W\_\{0\}without zero entries\) is LoRA in a transported frame; for LoRA\-XS, SVFT and HiRA the frame depends onθ0\\theta\_\{0\}\([Prop\. II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.15orthogonal fine\-tuning preserves knowledgethe image lies in a fibre of the complete invariantW↦W​W⊤W\\mapsto WW^\{\\top\}\(first fundamental theorem forO⁡\(n\)\\mathrm\{O\}\(n\)\)In exact arithmetic, exactly orthogonal input\-side methods \(OFT with an exact Cayley map, strict GOFT without its optional scaler, HRA\) can never change the spectrum or neuron angles, however often they are merged; HF’s default Cayley–Neumann OFT is only approximately orthogonal \([Cor\. II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\)\.16HRA bridges LoRA and OFTimage\-level meet with the rotation orbit,\(W0\+ℳ≤r\)∩W0​SO​\(n\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\,\\mathrm\{SO\}\(n\), for injectiveW0W\_\{0\}and evenr≤n−2r\\leq n\-2; not a product of methodsLoRA meets exact\-Cayley OFT in a strictly smaller set, and HF’s default Cayley–Neumann OFT, which is not orthogonal, is incomparable with HRA\. At HRA’s paired identity init the image germ is the skew\-determinantal cone, so that init is an apex \([Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\.17LoRA\-XScategorical productFA⁡\(A0\)×FB⁡\(B0\)\\mathrm\{FA\}\(A\_\{0\}\)\\times\\mathrm\{FB\}\(B\_\{0\}\)of the two one\-sided LoRAs, in the SVD frames ofW0W\_\{0\}\(withrank⁡W0≥r\\rk W\_\{0\}\\geq r\)The overlap that a split init double\-counts \([Prop\. I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6),[Prop\. II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)\)\.18DoRA’s magnitude/direction decouplingdiagonal saturationDiagm⋅\(W0\+ℳ≤r\)\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\), the image of LoRA followed by a trainable output scaleEqual images, forW0W\_\{0\}without zero rows\. Under gradient flow the exact gradient changes the metric and the detached backward is a non\-gradient update; under AdamW the magnitude coordinates matter too \([Prop\. II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\)\.19\(IA\)3, SSF, LN tuning, RoAd, HiRAthe scalings and HiRA are rungs of the Hadamard\-torus ladderW0⊙\(J\+Z\)W\_\{0\}\\odot\(J\+Z\);RoAd1\\mathrm\{RoAd\}\_\{1\}is the complex torus\(ℂ×\)m/2\(\\mathbb\{C\}^\{\\times\}\)^\{m/2\}, andRoAd2\\mathrm\{RoAd\}\_\{2\}andRoAd4\\mathrm\{RoAd\}\_\{4\}apply arbitrary2×22\\times 2blocks \(group part block\-GL2\\mathrm\{GL\}\_\{2\}\)On every rung zeros ofW0W\_\{0\}stay zero\. RoAd can fill them, so it is a rung only for zero\-freeW0W\_\{0\}, and for genericW0W\_\{0\}it lies on no low\-rank rung\. CovarianceMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}on output\-scaled boxes,GLm×Monn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}on down\_proj, exactlyMonm×Monn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}for HiRA, neverGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\([Prop\. II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\)\.20“the geometry a method lives in”covariance group𝒢M\\mathcal\{G\}\_\{M\}\(Klein\): the largest group of pairs\(g,h\)\(g,h\)withIm⁡M⁡\(g​W0​h−1\)=g​Im⁡M⁡\(W0\)​h−1\\Imop M\(gW\_\{0\}h^\{\-1\}\)=g\\,\\Imop M\(W\_\{0\}\)\\,h^\{\-1\}Descent depends on whether each induced pair\(g,h\)\(g,h\)lies in𝒢M\\mathcal\{G\}\_\{M\}, a question of side for one\-sided methods and of group for coordinate\-structured ones\. LoRA and \(IA\)3\(HF defaults\) provably keep their reachable function sets under a global residual rotation of a gain\-fused pre\-norm model; block OFT, LoHa \(r≥2r\\geq 2\), HiRA and masks fail the sufficient condition \([Obs\. II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6),[Conj\. A](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6.Px2)\)\.21how a method trainsinduced cometricK=D​ρ​Λ​D​ρ⊤K=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}\(pulled\-back gradient of the lens\)Two isomorphic objects can train differently; gradient descent is natural exactly where cometrics agree, which for LoRA’s linear gauge meansO⁡\(r\)\\mathrm\{O\}\(r\)\([Thm III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\.22balancednessB⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\}Noether charge of the non\-compact halfSymr\\mathrm\{Sym\}\_\{r\}of𝔤​𝔩r\\mathfrak\{gl\}\_\{r\}; the Kempf–Ness slice of the gauge actionConserved under gradient flow with constant rates for any cotangent signal of the formΛ​D​ρ⊤​γ\\Lambda\\,D\\rho^\{\\top\}\\gamma\(DoRA’s detached norm, clipping\), after[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580)\([Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\)\.23LoRA\+ ratio,α\\alpha, init scale, rsLoRAcoordinates on the orbit space of a hyperparameter torusUnder Adam withε=0\\varepsilon=0, no weight decay, no clipping and a shared schedule, LoRA\+ with ratioλ\\lambdais exactly LoRA withα\\alphatimesλ\\lambda\(the LoRA case is[Schulman and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.12728#bib.bib401)\); at a fixed rank, rsLoRA is isomorphic to LoRA \([Thm III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Cor\. III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\.24transformation\-invariant optimizer \(Riemannian preconditioning, ScaledGD, LoRA\-Pro, LoRA\-RITE\)update rule equivariant under the whole gaugeGLr\\mathrm\{GL\}\_\{r\}, so it descends to the quotientℳr\\mathcal\{M\}\_\{r\}On the full\-rank locus, undamped scaled GD and LoRA\-RITE areGLr\\mathrm\{GL\}\_\{r\}\-equivariant in the parameters; damped variants and LoRA\-Pro’s step are onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant, and Adam onlyBrB\_\{r\}\-equivariant \([Prop\. III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.25GaLore vs LoRAcotangent\-side vs tangent\-side lens 2\-cells, which coincide for linear charts \(the degree\-one collapse\)Without weight decay or full\-gradient clipping, GaLore is one\-sided LoRA with a moving frame, with optimizer state carried across re\-basings rather than transported \([Prop\. IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), credited to[Torroba\-Hennigen et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib453)\)\.26when a gradient\-projection method is really a reparametrisationFrobenius integrability of the anchor distributionGaLore, LISA and MeZO keep aθ\\theta\-independent anchor within each period, hence affine leaves, and under the hypotheses of[Prop\. IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)are rebasing schemes of linear forward methods; Frobenius integrability matters only forθ\\theta\-dependent anchors such as full\-batch GaLore withT=1T=1\([Prop\. IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\)\.27activation memorybackward reach of the composite lens \(the backprop cone\)An upper bound: LST avoids the backbone’s backward pass; LoRA\-FA storesA0​xA\_\{0\}x; LISA’s cone is the whole network because its embedding is always trained, though its frozen layers need not store their linear\-input residuals; MeZO has no reverse pass \([Obs\. IV\.4](https://arxiv.org/html/2610.12728#S4.SS0.SSSx5)\)\.28QLoRA / LoftQbase change along a map that does not preserve the function; pointing defect; approximate sectionSplit\-then\-quantise, quantise\-then\-fit \(LoftQ’s default, whose HF init ignores the LoRA scale\) and quantise\-split\-requantise leave three different defects \([Prop\. V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\.29ReLoRA, re\-OFT, iterated BOFT/HRAthe monoid \(or Lie subgroup\) generated by the reachable displacementsReLoRA reaches exactlyW0\+ℳ≤min⁡\(K​r,m,n\)W\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}; exact\-Cayley block OFT is stuck; iterated BOFT reaches the orbitW0​SO​\(n\)W\_\{0\}\\,\\mathrm\{SO\}\(n\), up to output scaling, only at full butterfly depth; re\-HRA reaches\(W0\+ℳ≤K​r\)∩W0​SO​\(n\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq Kr\}\)\\cap W\_\{0\}\\,\\mathrm\{SO\}\(n\)\([Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.30task arithmetic vs TIES/DAREGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-natural vs coordinatewise operations \(TIES with sign election by total mass: homotheties times signed permutations; DARE: the monomial group, its permutations in distribution\)Coordinate merges depend on the basis; factor\-space merges depend on the gauge \([Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.31LoraHub / Poly factor mixinga rule on factors that is not invariant under∏GLr\\prod\\mathrm\{GL\}\_\{r\}, hence not a function of the adaptersPost\-hoc LoraHub \(and HF’s linear combination\) needs a joint gauge fixing that produces a shared factor \(KnOTS\) or a shared frozen factor \(FFA\-LoRA\); Poly/MHR are trained jointly and, for fixed generic routing with fewer skills than tasks, have only the diagonalGLr\\mathrm\{GL\}\_\{r\}symmetry, which is checked numerically \([Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.32adapters \(Houlsby, parallel\), LLaMA\-Adapter, LoReFTpointed extensions: function\-preserving embeddings with a neutral pointLoReFT is LoRA on an inserted affine identity box \(bias confined tocol⁡B\\operatorname\{col\}B\), applied at selected positions; pyreft’s default init is not neutral \([Prop\. VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)\)\.33prompt / prefix tuningunpointed extensions on the sequence wirePrefix = gated parallel adapter \(attention = monoid homomorphism followed by a perspective map\); content\-attention ratios are rigid only within the layer where the prefix acts; prompt tuning maps into deep prefix tuning in causal decoders with matching position offsets \([Thm VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\)\.34privileged neuron basisautomorphism group of the elementwise activation \(theσ\\sigma\-spider\)For ungated FFNs: permutations for GELU/SiLU, signed permutations for tanh, positive monomials for ReLU; gated FFNs \(SwiGLU, GeGLU\) have the full monomial group \([Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\)\.35RoSA, GraLoRA, HF cat merges, ReLoRA stageswords in the convolution monoidal product⊕\\oplusof additive methods \(Minkowski sum of images\)⊕\\oplusis not the categorical product, whose image is the intersection when it exists\. LoRA\-soup CAT is a⊕\\oplusof frozen line methods \(a span\), and UniPELT/MAM are not⊕\\oplus\-words \([Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\)\.36target\_modules / applying a method to every layerassigning a 2\-cell to each generator box and composing horizontally;TT\-points for routing, tasks and tenantsNon\-local methods \(VB\-LoRA, Compacter, FacT, Uni\-LoRA\) share parameters across boxes and are not generator\-wise\.

## EXPOSÉ XFalsifiable predictions

Say what you proved, say what you only named, and say what would show you wrong\. a working maxim

This exposé states eleven experiments that could show the framework wrong, each for a named optimizer\.

##### In brief

- •A fair LoRA\+ baseline multipliesα\\alphaby the ratioλ\\lambda\. Under Adam, withε\\varepsilon, decay and per\-block clipping thresholds rescaled and one shared schedule, the two runs give identical weights \(after[Schulman and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.12728#bib.bib401)\)\.
- •LoRA’s chargeΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}is constant under gradient flow and, when a step sizeη\\etamultiplies fixed block rates, moves byO⁡\(η2\)O\(\\eta^\{2\}\)per SGD step, a cheap diagnostic of what the optimizer adds\.
- •Some claims need no training run\. Generic rank\-one targets separate LoRA from Kronecker adapters, exact block OFT re\-merged on a fixed partition never leavesW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\), and LoraHub’s merge depends on gauge choices unless the modules share a factor\.

#### Why the theory should risk something

A framework that only re\-describes known methods cannot be wrong, and so it earns little trust\. The claims below say what a training run will do\. Each prediction carries the optimizer scope of the result it rests on; a claim about images needs none, because no run can leave the image\. Table[29](https://arxiv.org/html/2610.12728#S10.T29)lists them\. Each keeps the number it carries on the companion site, and the subsections group them by subject \(optimizers, images, symmetry, base change and merging\), so the numbers do not run in order\.

Table 29:The eleven predictions, the optimizer scope each inherits, the results it rests on, and its kind\.Read the kind column as a warning label\.*Theorem*means the sharp half cannot fail inside its hypotheses\.*Theorem \+ data*pairs a proved mechanism with an empirical claim about its size\.*Conjecture*means we expect it and have not proved it\. The toys check mechanisms in float64 on small matrices, and the experiments on real models are still to be run\. Each prediction below lists its scope, the results that support it, a test, the outcome that would refute it, and the toy or figure that shows its mechanism\.

#### Predictions about optimizers

LoRA has four knobs: the scales=α/rs=\\alpha/r, the ratesηA,ηB\\eta\_\{A\},\\eta\_\{B\}and the initial sizeσA\\sigma\_\{A\}ofAA\. A two\-dimensional torus of rescalings moves all four without changing any weight\. LoRA\+ turns one knob, the ratioλ=ηB/ηA\\lambda=\\eta\_\{B\}/\\eta\_\{A\}, and under Adam that turn lands on the orbit of plain LoRA with a largerα\\alpha\.

Prediction X\.1\(LoRA\+ against a re\-tunedα\\alpha\)\. Any gain of LoRA\+ with ratioλ\\lambdaover plain LoRA withα\\alphamultiplied byλ\\lambda\(sameηA\\eta\_\{A\}, sameA0A\_\{0\}\) comes from the ingredients that break the extended torus: global\-norm clipping, anε\\varepsilonor weight decay shared by the two parameter groups, and schedules that differ between the groups\. With all of them off, or rescaled per block, the two runs produce the same weights at every step in exact arithmetic\.SCOPEAdam and AdamW \(β1∈\[0,1\)\\beta\_\{1\}\\in\[0,1\),β2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\), bias correction allowed\); linear LoRA modules fromB0=0B\_\{0\}=0; no regulariser on the factors; common data order and dropout masks\. A shared schedule, warm\-up included, preserves the torus\. HF PEFT’s LoRA\+ optimizer treats embeddings and 1\-D parameters separately, so only linear modules are covered\.SUPPORT[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)and[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\(b\), both verified\. Forε\>0\\varepsilon\>0the LoRA run needsεB=λ​ε\\varepsilon\_\{B\}=\\lambda\\varepsilononBB, and with AdamW a decayλ⋅wd\\lambda\\cdot\\mathrm\{wd\}onBB\. Under SGD the matching scale isλ​s\\sqrt\{\\lambda\}\\,s\.TESTTrain LoRA\+ withλ=16\\lambda=16against LoRA withα\\alphamultiplied by 16,ε\\varepsilononAAand16​ε16\\varepsilononBB, no decay and no clipping; the weights agree bit for bit, since 16 is a power of two\. Then switch on one torus breaker at a time \(sharedε\\varepsilon, shared AdamW decay, max\_grad\_norm=1\.0, group\-specific schedules\) and record the gap each opens\.WOULD FAIL IFa gap beyond rounding appeared with every hypothesis met\. A standard HF run clips by default and its LoRA\+ optimizer sharesε\\varepsilonand decay, so there the experiment measures how much of the gain each breaker carries\.TOYFigure[8](https://arxiv.org/html/2610.12728#S3.F8)runs LoRA\+ with ratioλ\\lambdaagainst LoRA withα⋅λ\\alpha\\cdot\\lambdaunder Adam withε=0\\varepsilon=0\. The loss curves coincide and the weight gap is exactly zero, and the gap jumps when a sharedε\>0\\varepsilon\>0, decoupled weight decay or global\-norm clipping is switched on\.

Remark\(why the test avoidsε=0\\varepsilon=0\)\. Literalε=0\\varepsilon=0needs the convention0/0=00/0=0\. AtB0=0B\_\{0\}=0the gradient ofAAvanishes, and a stock Adam divides zero by zero there\.

##### Credit

The LoRA and Adam case, its extension toε\>0\\varepsilon\>0and the two effective hyperparameters are due to[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401), for the method of[Hayou et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib159)\. The torus for any multihomogeneous method, and thec2c^\{2\}law for SGD, are added here\.

Under gradient flow LoRA conserves the chargeΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}, and whenΦ\\Phiis isotropic the dynamics have a closed form\. Small singular values of the update grow throughBBalone, as in LoRA\-FA; large ones grow through both factors in balance\.

Prediction X\.7\(the crossover scale\)\. Under SGD, the singular values ofΔ​W\\Delta Wswitch from one\-sided to two\-sided growth as they crossσ∗=μ​\|κ\|/2\\sigma^\{\*\}=\\mu\\lvert\\kappa\\rvert/2, whereμ=s​ηA​ηB\\mu=s\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}andΦ=−κ​Ir\\Phi=\-\\kappa I\_\{r\}\. For default LoRA withB0=0B\_\{0\}=0andA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cI,κ=c/ηA\\kappa=c/\\eta\_\{A\}andσ∗=s​c2​ηBηA\.\\sigma^\{\*\}=\\frac\{sc\}\{2\}\\sqrt\{\\frac\{\\eta\_\{B\}\}\{\\eta\_\{A\}\}\}\.\(13\)LoRA\+ with ratioλ\\lambdamultipliesσ∗\\sigma^\{\*\}byλ\\sqrt\{\\lambda\}\.SCOPEGradient flow exactly; SGD without momentum up to theO⁡\(η2\)O\(\\eta^\{2\}\)drift of the charge per step\. For Kaiming initialisationsA0​A0⊤A\_\{0\}A\_\{0\}^\{\\top\}is only close toc​IcI\(HF’s default hasc≈1/3c\\approx 1/3\), and the crossover becomes a band of relative widthO⁡\(r/n\)O\(\\sqrt\{r/n\}\)\. Adam is not covered\.SUPPORT[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), verified; known from the linear\-network literature, with the PEFT reading ours\.TESTTrain with SGD from orthonormal rows ofA0A\_\{0\}scaled byc\\sqrt\{c\}, so that the charge is exactly isotropic\. Take the SVDΔ​W=U​Σ​V⊤\\Delta W=U\\Sigma V^\{\\top\}at intervals and read offfi=ui⊤​\(B​B⊤/ηB\)​uif\_\{i\}=u\_\{i\}^\{\\top\}\(BB^\{\\top\}/\\eta\_\{B\}\)u\_\{i\}andgi=vi⊤​\(A⊤​A/ηA\)​vig\_\{i\}=v\_\{i\}^\{\\top\}\(A^\{\\top\}A/\\eta\_\{A\}\)v\_\{i\}\. Belowσ∗\\sigma^\{\*\},fi≈0f\_\{i\}\\approx 0andgi≈κg\_\{i\}\\approx\\kappa; above it,fi≈gi≈σi/μf\_\{i\}\\approx g\_\{i\}\\approx\\sigma\_\{i\}/\\mu\. Repeat at ratioλ\\lambda\.WOULD FAIL IFthe gains departed from the closed\-form valuesfi=fκ​\(σi2/μ2\)f\_\{i\}=f\_\{\\kappa\}\(\\sigma\_\{i\}^\{2\}/\\mu^\{2\}\)andgi=gκ​\(σi2/μ2\)g\_\{i\}=g\_\{\\kappa\}\(\\sigma\_\{i\}^\{2\}/\\mu^\{2\}\)of[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), withfκ​\(x\)=x\+κ2/4−κ/2f\_\{\\kappa\}\(x\)=\\sqrt\{x\+\\kappa^\{2\}/4\}\-\\kappa/2andgκ=fκ\+κg\_\{\\kappa\}=f\_\{\\kappa\}\+\\kappa, whileΦ\\Phistayed isotropic\. IfΦ\\Phihas drifted, the test says nothing\.TOYFigure[7](https://arxiv.org/html/2610.12728#S3.F7)\(b\) plots the closed\-form isotropic\-charge singular\-value dynamics and the crossoverσ∗\\sigma^\{\*\}\.

##### Credit

The closed form is due to[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447), who call the charge “λ\\lambda\-balanced”, and to[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328); see also[Dominé et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib97)and[Kunin et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib231)\. The balanced regime aboveσ∗\\sigma^\{\*\}is the dynamics of[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)\.

WHAT THIS EXPLAINSLoRA\+ movesσ∗\\sigma^\{\*\}up byλ\\sqrt\{\\lambda\}, so more of the update spends longer in the LoRA\-FA regime\. PiSSA at equal rates hasκ=0\\kappa=0, and under gradient flow it is balanced from the first step\.

Part of LoRA’s gauge is a symmetry ofρ\\rho, and gradient flow conserves its charge whatever signal drives it\. A discrete step conserves it up to second order in the step size, and nothing forces Adam’s sign\-like step to conserve it\.

Prediction X\.8\(drift of the charge\)\.Φ\\Phiis constant under gradient flow, moves byO⁡\(η2\)O\(\\eta^\{2\}\)per SGD step at fixed block ratesηA,ηB\\eta\_\{A\},\\eta\_\{B\}, and drifts under Adam\. We expect its drift under Adam to track other measures of how far Adam breaks LoRA’s gauge\.SCOPEGradient flow driven by any signalΛ​D​ρ⊤​γ\\Lambda D\\rho^\{\\top\}\\gamma, which covers minibatches, LoRA dropout, global\-norm clipping and DoRA’s detached norm; SGD without momentum or weight decay\. Momentum, Adam and AdamW fall outside[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\.SUPPORT[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(repaired; known after[Zhao et al\., 2022b](https://arxiv.org/html/2610.12728#bib.bib580)\); weight decay moves the charge on purpose \([Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\)\. With a step sizeη\\etamultiplying the block rates, a stepB←B−η​ηB​∇BB\\leftarrow B\-\\eta\\eta\_\{B\}\\nabla\_\{B\},A←A−η​ηA​∇AA\\leftarrow A\-\\eta\\eta\_\{A\}\\nabla\_\{A\}changes the charge by exactlyΔ​Φ=η2​\(ηB​∇B⊤∇B−ηA​∇A∇A⊤\)\.\\Delta\\Phi=\\eta^\{2\}\\big\(\\eta\_\{B\}\\,\\nabla\_\{B\}^\{\\top\}\\nabla\_\{B\}\-\\eta\_\{A\}\\,\\nabla\_\{A\}\\nabla\_\{A\}^\{\\top\}\\big\)\.\(14\)The identity uses only that both factor gradients come from one weight\-space gradientGG, so it covers minibatches, LoRA dropout on the input ofAA, and global clipping when∇A,∇B\\nabla\_\{A\},\\nabla\_\{B\}denote the gradients actually applied\.TESTLogΦ\\Phiduring training\. Under SGD at step sizesη\\eta,η/2\\eta/2,η/4\\eta/4, the per\-step change should match \([14](https://arxiv.org/html/2610.12728#S10.E14)\) to rounding and shrink about fourfold each time\. Under Adam, compare∥Φt−Φ0∥\\lVert\\Phi\_\{t\}\-\\Phi\_\{0\}\\rVert\(our suggestion\) with the gap between runs started atA0A\_\{0\}andg​A0gA\_\{0\},g∉Brg\\notin B\_\{r\}, which by[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)can differ\.WOULD FAIL IFthe SGD drift departed from \([14](https://arxiv.org/html/2610.12728#S10.E14)\), so that the update was not a plain gradient step, orΦ\\Phi\-drift under Adam showed no relation to any other measure of gauge breaking\.TOYFigure[7](https://arxiv.org/html/2610.12728#S3.F7)\(a\): scalar LoRA rides its hyperbolab2/ηB−a2/ηA=Φb^\{2\}/\\eta\_\{B\}\-a^\{2\}/\\eta\_\{A\}=\\Phiunder gradient flow, and the charge drifts under discrete GD and under Adam\.

##### Credit

The conservation law, with its symmetric and antisymmetric halves, is Prop\. 5\.1 of[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580); laws independent of loss and data are in[Marcotte et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib318)\. The principle goes back to[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100),[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)and[Kunin et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib230)\.

#### Predictions about images

DoRA splits each output neuron into a length and a direction\. At the level of images that adds exactly one free gain per output neuron to LoRA, so anything else that separates the two happens in training, where Adam notices the coordinates a parameter is written in\.

Prediction X\.2\(DoRA against LoRA \+ output gain\)\.Conjecture D\(DoRA against a gain in DoRA’s coordinates\)\. Started from the same weights and trained with AdamW at the same settings, LoRA followed by a trainable output gain matches DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\)once the gain is trained in DoRA’s coordinates \(μi\\mu\_\{i\}starting at∥w0,i∥\\lVert w\_\{0,i\}\\rVertin place ofℓi\\ell\_\{i\}starting at11\)\.SCOPEAdamW, as in the DoRA paper and HF PEFT, with one learning rate, decay and data order for every arm\. Images refer to the merged weight, withr≤min⁡\(m,n\)r\\leq\\min\(m,n\)andW0W\_\{0\}free of zero rows\. Every arm trainsr⁡\(m\+n\)\+mr\(m\+n\)\+mparameters\.SUPPORT[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\(repaired\):Im⁡DoRAr=Diagm⋅\(W0\+ℳ≤r\)\\Imop\\mathrm\{DoRA\}\_\{r\}=\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\), the image ofLoRAr\\mathrm\{LoRA\}\_\{r\}followed by an output gain\. The first form of Conjecture D, that any gap comes from the parametrisation, followed from this equality and could not fail; the form above can\.TESTStart four arms atW0W\_\{0\}with the sameA0A\_\{0\}, as in Experiment D of Exposé[II\.B](https://arxiv.org/html/2610.12728#S2.SS2): \(i\) LoRA with an output gain,ℓ0=𝟏\\ell\_\{0\}=\\mathbf\{1\}; \(ii\) the gain in DoRA’s coordinates,W=diag⁡\(μi/∥w0,i∥\)​\(W0\+s​B​A\)W=\\diag\(\\mu\_\{i\}/\\lVert w\_\{0,i\}\\rVert\)\(W\_\{0\}\+sBA\),μ0,i=∥w0,i∥\\mu\_\{0,i\}=\\lVert w\_\{0,i\}\\rVert; \(iii\) DoRA with the exact gradient of its normalisation; \(iv\) DoRA with the detached norm of the DoRA paper \(Sec\. 4\.3\) and HF PEFT \(weight\_norm\.detach\(\)\)\. Since the images are equal, any gap between arms comes from the parametrisation, the backward rule or the optimizer coordinates, and the four arms separate these\. This design is ours\.WOULD FAIL IFarm \(ii\) stayed clearly behind arm \(iv\)\. Comparing \(iii\) with \(iv\) then shows how much of the gap the detached backward carries\.TOYNone for training\. The companion site’s Gram lab shows the image\-level half: atB=0B=0DoRA acts as an output torus, and its update escapes rankrr\.

A rank\-one update writes one new feature into many neurons, and LoRA does this with a single rank\. A Kronecker sum measures rank across a different cut of the same four\-leg tensor, where, for the splits HF PEFT uses, a generic rank\-one update is as complex as any matrix can be\.

Prediction X\.5\(rank\-one targets\)\. Splitm=m1​m2m=m\_\{1\}m\_\{2\}andn=n1​n2n=n\_\{1\}n\_\{2\}\. On targetsW0\+u​v⊤W\_\{0\}\+uv^\{\\top\}with genericu,vu,v, and a loss whose only minimiser is the target,LoRA1\\mathrm\{LoRA\}\_\{1\}can reach zero loss while no sum of fewer thanκ1=min⁡\(m1,m2\)​min⁡\(n1,n2\)\\kappa\_\{1\}=\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\)\(15\)Kronecker products can\. For HF PEFT’s splitsκ1\\kappa\_\{1\}is the largest possible Kronecker rank:ddfor its square splits ofd=1024d=1024or40964096\(for instance the64×6464\\times 64split of40964096\), and576576for its24×3224\\times 32split of768768\. For LoRA ranks belowmin⁡\(m1,n1\)​min⁡\(m2,n2\)\\min\(m\_\{1\},n\_\{1\}\)\\min\(m\_\{2\},n\_\{2\}\)and fewer thanκ1\\kappa\_\{1\}terms, neither image contains the other\.SCOPEThe negative half concerns images and holds for every optimizer; the positive half asks the optimizer to find a point that exists\.SUPPORT[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(repaired\), for unconstrained Kronecker sums and for LoKr, sinceC⊗B​AC\\otimes BAis one Kronecker term\. With Schmidt decompositionsu=∑iσi​xi⊗yiu=\\sum\_\{i\}\\sigma\_\{i\}x\_\{i\}\\otimes y\_\{i\}andv=∑jτj​zj⊗wjv=\\sum\_\{j\}\\tau\_\{j\}z\_\{j\}\\otimes w\_\{j\}, the bestkk\-term sum leaves the squared error∑σi2​τj2\\sum\\sigma\_\{i\}^\{2\}\\tau\_\{j\}^\{2\}over all but thekklargest productsσi​τj\\sigma\_\{i\}\\tau\_\{j\}\.TESTOn one linear layer with whitened inputs and more samples than parameters \(or with the loss12​∥W−W0−u​v⊤∥F2\\tfrac\{1\}\{2\}\\lVert W\-W\_\{0\}\-uv^\{\\top\}\\rVert\_\{F\}^\{2\}\), trainLoRA1\\mathrm\{LoRA\}\_\{1\}and sums ofk=1,…,κ1k=1,\\dots,\\kappa\_\{1\}unconstrained Kronecker products, each with its own learning\-rate sweep, and plot the final loss againstkkbeside the floor\.WOULD FAIL IFa sum of fewer thanκ1\\kappa\_\{1\}terms went below the floor\.LoRA1\\mathrm\{LoRA\}\_\{1\}stalling above zero would be an optimization failure, against which the framework claims nothing\.TOYFigure[6](https://arxiv.org/html/2610.12728#S2.F6): for a random rank\-one update the operator\-Schmidt spectrum has one nonzero value across the ordinary cut andκ1\\kappa\_\{1\}across the Kronecker cut\.

##### Credit

The rearrangement and the nearest\-Kronecker\-product problem are due to[Van Loan and Pitsianis \(1993\)](https://arxiv.org/html/2610.12728#bib.bib459)\([Background B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\)\. The PEFT reading, that Kronecker sums are LoRA across another cut, is ours\. Kronecker adapters in practice are KronA\([Edalati et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib103)\)and LoKr\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\)\.

Parameter count measures what a method pays, expressive dimension what it can reach\. The difference is gauge, parameters that move without moving the weights\. It ranges from one parameter for VeRA throughr2r^\{2\}forLoRAr\\mathrm\{LoRA\}\_\{r\}to at least2​r2\+m\+n−12r^\{2\}\+m\+n\-1forLoHar,r\\mathrm\{LoHa\}\_\{r,r\}, which has the budget ofLoRA2​r\\mathrm\{LoRA\}\_\{2r\}\.

Prediction X\.10\(effective dimension\)\. Benchmarks should report the expressive dimensiond⁡\(M\)d\(M\)beside the budget\|M\|\\lvert M\\rvert\. At budgets of a few thousand parameters, rankings at equal effective dimension can differ from rankings at equal parameter count\.SCOPEOptimizer\-free bookkeeping; the change of ranking is the empirical claim\.SUPPORT[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(repaired\)\.LoRAr\\mathrm\{LoRA\}\_\{r\}wastesr2r^\{2\}parameters, VeRA\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\)one \(numerically, Appendix[E](https://arxiv.org/html/2610.12728#A5); forr≤nr\\leq nand generic frozen factors also by[Background C\.14](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\) and FourierFT one per sampled conjugate pair\. At the budget ofLoRA2​r\\mathrm\{LoRA\}\_\{2r\}, with2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\),r2≤min⁡\(m,n\)r^\{2\}\\leq\\min\(m,n\)and2​r2<m\+n−12r^\{2\}<m\+n\-1,LoHar,r\\mathrm\{LoHa\}\_\{r,r\}reaches at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}fewer dimensions, and is strictly dominated forr≤2r\\leq 2\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1, as forr=64r=64on4096×40964096\\times 4096orr=32r=32on 320\- to 640\-wide layers, LoHa matches or beatsLoRA2​r\\mathrm\{LoRA\}\_\{2r\}in dimension too, by its numerically checked dimension formula\.TESTOn a small\-budget benchmark, compare methods at matched\|M\|\\lvert M\\rvertand at matchedd⁡\(M\)d\(M\), computingddby a float64 Jacobian rank where no formula is proved\. Report the maximum rank too, and name its cut\.WOULD FAIL IFthe two rankings agreed at every budget tested, so that gauge is too small a share of the budget to matter there\.TOYFigure[4](https://arxiv.org/html/2610.12728#S2.F4)places LoRA, LoHa, GraLoRA, HiRA, KronA, MoRA, FourierFT and C3A at one budget on a4096×40964096\\times 4096layer by maximum rank and byd⁡\(M\)/\(m​n\)d\(M\)/\(mn\)\. The companion site’s budget calculator sets\|M\|\\lvert M\\rvertbesided⁡\(M\)d\(M\)on real base models\.

#### Predictions about symmetry

The hidden units of a feed\-forward block carry no order\. Permute them with the matching rows and columns, and the network computes the same function\. A method that never looks at unit order should not notice\. A method built on a frame, a reshape or a block partition of the hidden coordinates should\.

Prediction X\.3\(the permutation test\)\. Over random function\-preserving permutations of hidden units, LoRA’s outcome variance equals its seed variance\. FourierFT, LoKr and MoRA show larger variance, and so do BOFT and OFT on modules whose input side is the hidden layer \(down\_proj\)\. On up\_proj and gate\_proj the permutation only reorders BOFT’s output rows, so BOFT is exactly equivariant there\.SCOPEThe LoRA half holds for any coordinatewise optimizer \(SGD, Adam, AdamW\), which commutes with coordinate permutations ofQQ\(the third item of[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\), and an initialisation ofAAwith i\.i\.d\. entries\. The rest is empirical\.SUPPORT[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\(classical\): the linear maps that commute with GELU or SiLU are the permutations, and with ReLU the positive monomials\. Gated SwiGLU or GeGLU blocks admit the full monomial group through their linear branch \(rule R4 of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\)\.[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)turns covariance under those maps into equal reachable function sets\.TESTApplyKKrandom permutations to the rows of up\_proj and gate\_proj and the columns of down\_proj, and confirm that outputs agree to rounding\. Fine\-tune each method once per permuted copy with a fresh seed, andKKtimes on the original withKKseeds; compare the two variances, method by method\.WOULD FAIL IFLoRA’s permutation variance exceeded its seed variance, pointing to an implementation tied to unit order, or the listed methods showed no excess\.TOYNo variance test\. The companion site’s merge game runs the rule behind the lemma: a positive diagonal slides through ReLU and stops at GELU\.

##### Credit

Lemma VI\.2 is classical\([Sussmann, 1992](https://arxiv.org/html/2610.12728#bib.bib440);[Chen et al\., 1993](https://arxiv.org/html/2610.12728#bib.bib58);[Godfrey et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib132)\)\. The methods under test are FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\), LoKr\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\), MoRA\([Jiang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib196)\), BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\)and OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)\.

WHAT THIS EXPLAINSIf the prediction holds, seed variance understates the variance of methods tied to unit order across equally good checkpoints\.

SliceGPT rotates the residual stream of a model without changing what it computes\. A method whose image rotates with the weights reaches the same functions afterwards; a method built on the stream’s coordinates may lose or gain reach\.

Prediction X\.4\(the rotation test\)\. Take a pre\-norm transformer with its norm gains fused into the linear maps they feed and LayerNorm converted to RMSNorm, and rotate its residual stream by one global orthogonal matrix\. This provably leaves unchanged the reachable function sets of LoRA, of \(IA\)3with HF defaults, and of DoRA on the input\-rotated boxes \(qq,kk,vv, gate, up\)\.[Conjecture A](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6.Px2), restated: against the gain\-fused, unrotated model, the rotation measurably changes the fine\-tuning results of block OFT/BOFT on residual\-input projections, LoHa \(r≥2r\\geq 2\), HiRA and coordinate masks, because LLM residual streams have a privileged basis\.SCOPEThe proved half concerns reachable function sets, for every optimizer and for box\-wise methods with no shared parameters or random frames\. Training outcomes are exactly unchanged only for \(IA\)3, and in law for LoRA and DoRA under SGD with an isotropic initialisation\. HF’s Kaiming\-uniformAAis not rotation\-invariant in law, and Adam is onlyBrB\_\{r\}\-equivariant \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\), so both break this\. Sandwich and post\-norm models \(Gemma\-2/3, OLMo\-2\) and SliceGPT’s per\-block rotations are excluded\.SUPPORT[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\(repaired\), a sufficient condition\. The conjectured methods fail it, which by itself proves nothing\.TESTKeep the gain\-fused model as control, rotate a copy, and confirm that outputs agree to rounding\. Fine\-tune every method on both over several seeds\. LoRA’s reach provably does not change, so its change in results measures the share of the optimizer and the initialisation\.WOULD FAIL IFthe listed methods changed no more than LoRA does\.

##### Credit

The rotation is that of SliceGPT\([Ashkboos et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib18)\)\. In QuaRot\([Ashkboos et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib19)\), the per\-head Hadamard fused offline intoWvW\_\{v\}andWoutW\_\{\\mathrm\{out\}\}is a value–output head gauge, which[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)covers; the online Hadamards preserve the function but insert new operations, outside its hypothesis\.

PiSSA and CorDA choose their split from a truncated SVD, which follows the coordinates under rotations and uniform scalings and changes when they are stretched unevenly\. Whitening the input first removes that dependence from the split\.

Prediction X\.11\(whitened splits\)\.[Conjecture C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7.Px2), restated: whitened CorDA or PiSSA is more robust to function\-preserving input reparametrisations, and no worse otherwise\.SCOPEThe covariance of the split is optimizer\-free; the robustness of trained results is the empirical part\. Absolute dampingC\+λ​IC\+\\lambda Icuts the covariance toO⁡\(n\)\\mathrm\{O\}\(n\), while CorDA’s trace\-relative damping keepsCO⁡\(n\)\\mathrm\{CO\}\(n\)\. The released CorDA code divides each calibration sample by its largest entry, which breaks even rotations, so only the published formulas are covered\.SUPPORT[Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\(repaired\), forC=𝔼⁡\[x​x⊤\]≻0C=\\mathbb\{E\}\[xx^\{\\top\}\]\\succ 0,1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)and spectral gaps\. The whitened splitΠr​W0\\Pi\_\{r\}W\_\{0\}, withΠr\\Pi\_\{r\}the top\-rreigenprojector ofW0​C​W0⊤W\_\{0\}CW\_\{0\}^\{\\top\}, isGLn\\mathrm\{GL\}\_\{n\}\-covariant on the input side\. The PiSSA and MiLoRA splits and the paper\-form CorDA splits have input\-side group exactlyCO⁡\(n\)\\mathrm\{CO\}\(n\), so norm\-gain folding or SmoothQuant rescalings change them\.TESTPrepare function\-preserving versions of one checkpoint that differ by non\-conformal input maps \(folded RMSNorm gains, or a per\-channel rescaling with its inverse folded into the producer\)\. Initialise PiSSA, paper\-form CorDA and the whitened split on each, fine\-tune with one optimizer, and compare the spread across versions; compare all three on the unmodified checkpoint for the “no worse” half\.WOULD FAIL IFPiSSA and CorDA varied no more than the whitened split across versions, or the whitened split did worse on the unmodified checkpoint\.

##### Credit

The splits are PiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\), MiLoRA\([Wang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib465)\)and CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\); the whitened form is the truncation\-aware whitening of SVD\-LLM\([Wang et al\., 2025c](https://arxiv.org/html/2610.12728#bib.bib479)\)\. EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)starts atB0=0B\_\{0\}=0and splits nothing\.

#### Predictions about base change and merging

Merging and restarting lets a method compose its own moves\. For an additive cone such as ReLoRA’s, composition keeps buying rank\. For block OFT with a fixed partition, a product of block rotations is again a block rotation\.

Prediction X\.6\(re\-based OFT\)\. With an exact Cayley map and a fixed partition, merge\-and\-restart OFT reaches nothing beyond the closure of one cycle,W0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\), so fixed blocks gain nothing\. Partitions whose hypergraph is connected generateSO⁡\(n\)\\mathrm\{SO\}\(n\), so re\-based exact OFT gains reach only if its partition changes between cycles or its butterflies reach full depth\. LoRA\-type cones gain from rebasing with a fixed shape; fixed linear shapes \(masks, BitFit, \(IA\)3\) gain nothing\.SCOPEReachable sets, hence optimizer\-free, and silent on trained performance\. HF PEFT’s default Cayley–Neumann OFT is not orthogonal; with small generators \(∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\mathrm\{op\}\}\\leq 2^\{1/4\}\) each merge is a contraction, so repeated merging can only shrink neuron norms and singular values\. Iterated exact BOFT withm′m^\{\\prime\}butterfly factors reachesW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), up to its output scale, only at full depthb​2m′−1=nb\\,2^\{m^\{\\prime\}\-1\}=n\.SUPPORT[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(c\) \(repaired\) and its corollaries 2 and 4\.TESTRunKKcycles of merge\-and\-restart OFT with use\_cayley\_neumann=False and a fixed partition, logging the partial GramsW:,JW:,J⊤W\_\{:,J\}W\_\{:,J\}^\{\\top\}and the loss\. Repeat with a fresh random partition each cycle, and with ReLoRA at the same per\-cycle budget\. Only the fixed\-partition run keeps its partial Grams fixed to rounding\.WOULD FAIL IFa partial Gram moved beyond rounding under the fixed partition, so that the map was not exactly orthogonal\. Gains in loss from restarts alone are outside the claim\.TOYFigure[10](https://arxiv.org/html/2610.12728#S5.F10)\(b\) shows the components of the hypergraph generated by block\-OFT, butterfly and Givens factors on 16 coordinates and the dimension of the Lie algebra they generate againstdim𝔰​𝔬​\(16\)=120\\dim\\mathfrak\{so\}\(16\)=120; a fixed block partition never connects the hypergraph\.

##### Credit

OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)and BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\)introduced the block and butterfly parametrisations; ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)introduced merge\-and\-restart for LoRA\.

LoraHub mixes adapters factor by factor\. A factor belongs to no adapter in particular, since\(B,A\)\(B,A\)and\(c​B,A/c\)\(cB,A/c\)give the same update, so the mix depends on choices the adapters do not record unless the modules share a factor\.

Prediction X\.9\(LoraHub seeds\)\. LoraHub’s factor\-wise merge\(∑iwi​Bi\)​\(∑jwj​Aj\)\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\)\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\)behaves like update arithmetic, up to the factor∑jwj\\sum\_\{j\}w\_\{j\}, when its modules share theAA\-seed andAAstays close to its initialisation\. It degrades when the modules come from independent seeds\.SCOPEThe algebra is post hoc and optimizer\-free\. WhetherAAstays nearA0A\_\{0\}is empirical; a largeηB/ηA\\eta\_\{B\}/\\eta\_\{A\}, as in LoRA\+, favours it\.SUPPORT[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(repaired\)\. Replacing\(B2,A2\)\(B\_\{2\},A\_\{2\}\)by\(c​B2,A2/c\)\(cB\_\{2\},A\_\{2\}/c\)changes the merge by∑l≠2wl​w2​\[\(c−1\)​B2​Al\+\(c−1−1\)​Bl​A2\],\\sum\_\{l\\neq 2\}w\_\{l\}w\_\{2\}\\big\[\(c\-1\)B\_\{2\}A\_\{l\}\+\(c^\{\-1\}\-1\)B\_\{l\}A\_\{2\}\\big\],\(16\)so it is not a function of the adapters\. With one shared frozenAAit returns\(∑jwj\)​∑iwi​Bi​A\(\\sum\_\{j\}w\_\{j\}\)\\sum\_\{i\}w\_\{i\}B\_\{i\}Aexactly\. HF’s add\_weighted\_adapter with linear, and its non\-SVD ties and dare\_\* variants, are factor\-wise rules of the same kind\.TESTTrainNNmodules onNNtasks from a sharedAA\-seed and from independent seeds\. Merge each set by LoraHub’s rule at fixed weights and compare with\(∑jwj\)​∑iwi​Bi​Ai\(\\sum\_\{j\}w\_\{j\}\)\\sum\_\{i\}w\_\{i\}B\_\{i\}A\_\{i\}, as matrices and on the downstream metric, logging∥Ai−A0∥/∥A0∥\\lVert A\_\{i\}\-A\_\{0\}\\rVert/\\lVert A\_\{0\}\\rVert\.WOULD FAIL IFindependent seeds merged as well as shared ones, so that the cross termsBi​AjB\_\{i\}A\_\{j\},i≠ji\\neq j, cost nothing in practice\.TOYFigure[9](https://arxiv.org/html/2610.12728#S3.F9), merge panel: a gauge of one module moves the LoraHub merge and leaves task arithmetic unchanged\.

##### Credit

LoraHub is due to[Huang et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib180)\. FFA\-LoRA\([Sun et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib437)\)freezes a shared factor, and KnOTS\([Stoica et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib429)\)re\-factorises the modules jointly through a sharedU​ΣU\\Sigma\. Task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\)combines updates and has no gauge to break\.

WHAT THIS EXPLAINSA safe rule combines LoRAs on the productsBi​AiB\_\{i\}A\_\{i\}, or on factors only when the modules share one\.

#### One more expectation, and how to read a failure

One expectation is not on the list\. By[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4), unconstrained Kronecker sums are LoRA after a rearrangement of entries, and FourierFT \(with no sampled conjugate pair\), WaveFT \(when HF’s pad–IDWT–crop map is invertible\) and C3A are coordinate masks in a fixed frame\. An optimizer that sees the loss only through the trainable parameters runs each exactly as its source on a transported loss\. At matched learning\-rate sweeps we therefore expect the gap between a transported method and its source to follow how well each image aligns with the task’s update, measured by projecting a full fine\-tuningΔ​W\\Delta Wonto each image\.

A failure inside the hypotheses of[X\.1](https://arxiv.org/html/2610.12728#S10.SS0.SSSx2),[X\.5](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3)or[X\.6](https://arxiv.org/html/2610.12728#S10.SS0.SSSx5), or of a proved half, would mean an error in a proof, and we would mark that result refuted; a failure outside them shows which hypothesis matters\. A failure of[X\.2](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3),[X\.4](https://arxiv.org/html/2610.12728#S10.SS0.SSSx4.Px1)or[X\.11](https://arxiv.org/html/2610.12728#S10.SS0.SSSx4.Px2)would refute[Conjecture D](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3),[Conjecture A](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6.Px2)or[Conjecture C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7.Px2), respectively\.

## EXPOSÉ XIWhat is analogy, and what is open

This exposé gives a plain account of which borrowed words carry theorems, what is open, whose results are re\-derived here, and what the referees changed\. Referees repaired 27 of the 41 results, and a second review narrowed 19 of those again\.

#### What the borrowed words mean

Words borrowed from geometry and physics compress a great deal, and each also carries connotations no theorem here supports\. Table[30](https://arxiv.org/html/2610.12728#S11.T30)gives, for seven such words, the sense each has in this paper, the results behind it, and what it leaves out\.

Table 30:Analogy versus theorem: seven borrowed words, their meaning here, what is proved under each, and what is not claimed\.WordMeaning hereProvedNot claimedGrothendieckThe relative point of view: a method is a mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\), methods form the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\), and changingθ0\\theta\_\{0\}is base change\.Frozen model initial and full fine\-tuning terminal; products are fibre products in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}and can fail to exist \([Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\); a symmetric monoidal sum \([Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\); automorphisms by transport \([Prop\. II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\); pushforward along function\-preserving maps \([Def\. V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1)\(a\)\)\.No schemes, sheaves, descent theory or stacks\. Quantisation and rebasing borrow only the word “base change”, and idle constructions such as∫𝐌𝐞𝐭𝐡→𝐏𝐍𝐞𝐭\\int\\mathbf\{Meth\}\\to\\mathbf\{PNet\}are left out\. “Exposé” and the rising sea are homage\.ErlangenAfter[Klein \(1872\)](https://arxiv.org/html/2610.12728#bib.bib217), a method’s geometry is its covariance group𝒢M\\mathcal\{G\}\_\{M\}\([Def\. II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6),[Background D\.17](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\); what a group\-type method can never change is a complete invariant of the action\.Complete invariants of the orthogonal, torus and Hadamard actions \([Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\); exactly orthogonal input\-side methods keepKoutK\_\{\\mathrm\{out\}\}and the spectrum through any number of merges \([Cor\. II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\); covariance groups of named methods \([Prop\. II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(e\),[Prop\. II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\)\.For cones such as LoRA’s image the word only names the shape and its covariance\. Keeping an invariant concerns reach; that it causes knowledge retention is not shown\.NoetherThe pattern that turns a continuous symmetry ofρ\\rhointo a conserved quantity\.[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), for gradient flow driven by any signalΛ​D​ρ⊤​γ\\Lambda D\\rho^\{\\top\}\\gamma, a mild extension of[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580); for LoRA the charges are the entries ofΦ\\Phi\.No Lagrangian\. Discrete steps drift byO⁡\(η2\)O\(\\eta^\{2\}\)relative to the charge, and Adam and momentum break the law\.GaugeThe symmetry group of a parametrisation \([Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\); for LoRA,GLr\\mathrm\{GL\}\_\{r\}acting by\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)\.Generic gauges and fibre dimensions \([Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\); the part each optimizer respects,Br⊂O⁡\(r\)⊂GLrB\_\{r\}\\subset\\mathrm\{O\}\(r\)\\subset\\mathrm\{GL\}\_\{r\}\([Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\); the gauge dependence of factor\-wise merging \([Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.No connection or curvature\. Fibres are gauge orbits only on the full\-rank stratum; away from it the fibres jump\.ModuliA set of isomorphism classes \([Background C\.20](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\)\.Three strata of full\-rank LoRA pointings are classified, as a set, byGr⁡\(r,n\)⊔Gr⁡\(r,m\)⊔ℳr\\Gr\(r,n\)\\sqcup\\Gr\(r,m\)\\sqcup\\mathcal\{M\}\_\{r\}\([Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\)\.No topology: the natural quotient topology is not a disjoint union \(TA\\mathrm\{T\}\_\{A\}lies in the closure ofS\\mathrm\{S\}\)\. No stack is built, and the strata miss some pointings \(HF’s orthogonal init, defect pointings\)\.FibreThe preimageρ−1​\(θ\)\\rho^\{\-1\}\(\\theta\), the parameters that give the same weights, as in the title\.Fibre dimensions \([Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(a\)\); which points of the apex fibre\{BA=0\}\\\{BA=0\\\}an optimizer can tell apart \([Cor\. III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.ρ\\rhois no fibre bundle; the fibre dimension jumps between strata\.LensBackpropagation as the 2\-functor𝐏𝐚𝐫𝐚⁡\(𝖱\)\\mathbf\{Para\}\(\\mathsf\{R\}\), which sorts tangent\-side from cotangent\-side methods \([Background B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.The degree\-one collapse: GaLore with weight decay00is one\-sided LoRA with a moving frame and carried optimizer state \([Prop\. IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.The duality is exact only at degree one, for optimizers that do not read the parameter tensor, with no weight decay; decoupled decay, default Adafactor and full\-gradient clipping break it\.##### Credit

𝐏𝐚𝐫𝐚\\mathbf\{Para\}and the lens reading of backpropagation are due to[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\. The ETHER control study that tempers the knowledge\-retention reading of orthogonal methods is[Bini et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib36)\.

Three scopes travel with every claim\. Expressivity is a different property from learnability\. Only Exposé[III](https://arxiv.org/html/2610.12728#S3)is dynamical, and each of its results names its optimizer\. Mergeability is proved box by box under explicit side conditions \(position\-uniformity, no weight sharing, bias slots\); the rewrite rules give sufficient conditions only, and whole\-network statements are open\.

Most of the mathematics is real algebraic geometry, invariant theory, Lie generation and gradient\-flow ODEs\. The category theory does its own work in seven places: the interval \[frozen, full FT\] and simulation; invariants finer than the image; a product that yields LoRA\-XS, and meets that organise others; base change along maps that change the function; the lens; naturality of gradient descent and of combination rules; and rank as a min\-cut\.

Some statements rest on less than a full proof\. The equalities in the dimensions of LoHa \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\) and VeRA are checked numerically, and the bounds are proved; for VeRA withr≤nr\\leq nand generic frozen factors the equality also follows from the toric\-cone description of[Background C\.14](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3), and the generic equality for DoRA \([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\) is proved\. The case of unconstrained generators in corollary 4 of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)is recorded as a proof sketch, which Appendix[C\.43](https://arxiv.org/html/2610.12728#A3.SS43)completes\. The continuous case of[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(a\) and the converse in[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)are proved in Appendix[C](https://arxiv.org/html/2610.12728#A3), and[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)is argued around Frobenius’ theorem \([Background D\.25](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\. The ancillary script framework\_checks\.py re\-runs the main numbers in about a minute \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

#### Open problems

Each problem grew out of a proof that stopped short, and each is stated so that a solution would be recognisable\.

1. 1\.LoHa’s dimension\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)proves d⁡\(LoHar1,r2\)≤min⁡\(m​n,r1​\(m\+n−r1\)\+r2​\(m\+n−r2\)−\(m\+n−1\)\),d\(\\mathrm\{LoHa\}\_\{r\_\{1\},r\_\{2\}\}\)\\leq\\min\\big\(mn,\\ r\_\{1\}\(m\+n\-r\_\{1\}\)\+r\_\{2\}\(m\+n\-r\_\{2\}\)\-\(m\+n\-1\)\\big\),\(17\)with equality checked only numerically\. Prove it in general, and decide whether the Hadamard variety is closed\.
2. 2\.Initialisations, dynamically\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)classifies three strata by expressivity, and[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)shows that Adam seesA0A\_\{0\}only up to signed row permutations\. Classify the apex fibre moduloBrB\_\{r\}for Adam, stratified by the chargeΦ0\\Phi\_\{0\}\([Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\), including the degenerate pointings outside the three strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\.
3. 3\.HRA’s apex in practice\.HRA’s paired initialisation, HF PEFT’s default, starts at an apex \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\), for injectiveW0W\_\{0\}and evenr≤n−2r\\leq n\-2\), withdimS1=k​n−k⁡\(k\+1\)2\\dim S\_\{1\}=kn\-\\tfrac\{k\(k\+1\)\}\{2\},k=r/2k=r/2, againstd=r​n−r⁡\(r\+1\)2d=rn\-\\tfrac\{r\(r\+1\)\}\{2\}\. Decide whether this apex is a practical first\-order bottleneck that slows early training, by comparison with a regular initialisation at matched budget\.
4. 4\.Charts for meets\.LoRA\-XS \([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)\) and HRA’s image \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\)\) are meets with a polynomial chart; the meet of LoRA with output scaling \([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(b\)\) is a union of coordinate planes\. Characterise the pairs of named methods whose meet is parametrisable by a polynomial chart\.
5. 5\.The frontier\.Incomparable pairs are common: split against zero\-init LoRA \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\)\), Kronecker sums or GraLoRA against LoRA \([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(c\)\)\. Find the size\-kkfrontier of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, whose maximal elements are pairwise incomparable\.
6. 6\.An Adam analogue of the crossover\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)and[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)hold for gradient flow\. Find a conserved or monotone quantity for sign\-like updates, whose symmetry is onlyBrB\_\{r\}\([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\.
7. 7\.Prefixes and prompts\.For a vanilla softmax head withWo≠0W\_\{o\}\\neq 0andWq≠0W\_\{q\}\\neq 0, a generic prefix is not box\-locally mergeable \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\); prove or refute its non\-mergeability for the whole network\. In causal decoders every soft prompt is a deep prefix \([Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(e\)\)\.[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)and[Li and Liang \(2021\)](https://arxiv.org/html/2610.12728#bib.bib252)assert that prompt tuning is strictly weaker than deep prefix tuning \(P\-tuning v2,[Liu et al\., 2021c](https://arxiv.org/html/2610.12728#bib.bib287)\), and the inclusion is strict in a one\-layer, one\-token example\. Prove the generic strictness, and settle encoders\.

Conjectures[A](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6.Px2),[C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7.Px2)and[D](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3)are open as well\. They appear in Exposé[X](https://arxiv.org/html/2610.12728#S10)as Predictions[X\.4](https://arxiv.org/html/2610.12728#S10.SS0.SSSx4.Px1),[X\.11](https://arxiv.org/html/2610.12728#S10.SS0.SSSx4.Px2)and[X\.2](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3)\.

#### Debts: known results we re\-derive

Much of what this paper proves was known in another language or setting\. A re\-derivation earns its place only if the new language adds something, so Table[31](https://arxiv.org/html/2610.12728#S11.T31)states what we add next to each debt\.

Table 31:Known results we re\-derive, whom they are due to, and what is added here\.HereDue toAdded here[Def\. I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), Exposé[IV](https://arxiv.org/html/2610.12728#S4)[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114);[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)The pointed slice of methods; tangent and cotangent sides\.[Thm II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)The PiSSA\-to\-LoRA conversion of[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324), in HF PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)The rank\-2​r2rbound for every exact pointing, and from any start\.[Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)Classical: the first fundamental theorem forO⁡\(n\)\\mathrm\{O\}\(n\); Witt’s extension theorem \([Background D\.16](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\)Which invariant each group\-type method keeps\.[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)The standard cut\-rank bound for tensor networks \([Background B\.8](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2)\)A rank census of PEFT plugs\.[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)[Van Loan and Pitsianis \(1993\)](https://arxiv.org/html/2610.12728#bib.bib459); the operator\-Schmidt decomposition \([Background B\.12](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)\)Kronecker sums as LoRA across another cut; the LoKr inclusion criterion\.[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), TT/Tucker row[Koch and Lubich \(2010\)](https://arxiv.org/html/2610.12728#bib.bib220);[Uschmajew and Vandereycken \(2013\)](https://arxiv.org/html/2610.12728#bib.bib454);[Holtz et al\. \(2012\)](https://arxiv.org/html/2610.12728#bib.bib172)The same count, at minimal ranks, beside other PEFT families\.[Prop\. III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(c\)[Tong et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib452);[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546); LoRA\-Pro\([Wang et al\., 2024i](https://arxiv.org/html/2610.12728#bib.bib486)\); LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\); descent to the quotient:[Mishra et al\. \(2012\)](https://arxiv.org/html/2610.12728#bib.bib330),[Mishra and Sepulchre \(2016\)](https://arxiv.org/html/2610.12728#bib.bib331)One cometric table for \(S\)GD; the hierarchyBr⊂O⁡\(r\)⊂GLrB\_\{r\}\\subset\\mathrm\{O\}\(r\)\\subset\\mathrm\{GL\}\_\{r\}\.[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580);[Marcotte et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib318); the principle:[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100),[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14),[Kunin et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib230)The formally immediate remark that the signal need not be a gradient; PEFT instances \(DoRA’s detached norm, clipping, VeRA, AdaLoRA without its regulariser, HRA\)\.[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)[Kempf and Ness \(1979\)](https://arxiv.org/html/2610.12728#bib.bib209), in the real form of[Richardson and Slodowy \(1990\)](https://arxiv.org/html/2610.12728#bib.bib389);[Srebro et al\. \(2004\)](https://arxiv.org/html/2610.12728#bib.bib426)Balance as the Kempf–Ness slice of LoRA’s gauge; two decay conventions, two slices \([Cor\. III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\)\.[Thm III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447);[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328);[Dominé et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib97);[Kunin et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib231)The PEFT reading ofσ∗\\sigma^\{\*\}, its shift under LoRA\+, PiSSA’s balance, the Kaiming band\.[Thm III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Cor\. III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401), for LoRA\+\([Hayou et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib159)\)Any multihomogeneous method; the SGDc2c^\{2\}law; per\-blockε\\varepsilon, decay and clipping\.[Prop\. IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453); for random projections, Flora\([Hao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib154)\)The projector\-by\-schedule table of cotangent methods; optimizer state carried across re\-basings\.[Prop\. IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)Frobenius’ theorem \([Background D\.25](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)When aθ\\theta\-dependent anchor foliates weight space by images of forward methods; GaLore, LISA and MeZO keep a fixed anchor per period, so the test does not separate them\.[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)[Sussmann \(1992\)](https://arxiv.org/html/2610.12728#bib.bib440);[Chen et al\. \(1993\)](https://arxiv.org/html/2610.12728#bib.bib58);[Godfrey et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib132)Which symmetries a method may ignore; the monomial group of gated FFNs\.[Thm VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(IA\)3merging:[Liu et al\. \(2022a\)](https://arxiv.org/html/2610.12728#bib.bib275), Sec\. 3\.3; SSF’s re\-parameterisation\([Lian et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib259)\); LoRA’s linearity\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\); scale migration through SwiGLU: AWQ\([Lin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib268)\)One calculus of sufficient conditions, with explicit side conditions\.[Thm VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)The identity is standard; \(b\)[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165); \(c\) and the prompt\-to\-prefix map \(e\)[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364), Sec\. 2\.2, who also show that prefix\-tuning a pretrained transformer can be a universal approximator\([Petrov et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib365)\)The homomorphism\-then\-perspective organisation; explicit position hypotheses and the slice typing of \(e\)\.##### Credit

We also build on work without re\-deriving it\.[Malladi et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib309)found the lazy kernel of random\-A0A\_\{0\}LoRA close to that of full fine\-tuning, which[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)reads throughS1S\_\{1\}\. The Init\[B\] scheme in the table of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)is from[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158)\. With a fixed support, the meet of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(b\) is \(IA\)3masked torrneurons; compare the sparse activation scaling of[Stoehr et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib428)\.

#### The referee record

The framework was synthesised from four proposals and three referee reports\. Every numbered result then went to two independent referees, who checked it against its proof, the literature and the Hugging Face PEFT source, that is, for its proof and for its fit to the methods as published and as coded\. Of the 41 results, 14 stood as first written, with wording fixes, and 27 were repaired by an added hypothesis, a narrowed claim, a replaced proof or a withdrawn reading\. A second review re\-examined 26 of the repaired statements in 48 reports\. It raised a major objection to 18 of them, mostly to a side clause: a missing hypothesis, a remark that went past its proof, or a description of a method that matched only one implementation\. For example, HF PEFT’s LoftQ initialiser turned out to ignore the LoRA scaleα/r\\alpha/r\([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\. In all it found a false or overreaching clause in 19 statements, and those were narrowed again\. Only the final statements appear in this paper; the referee notes of both rounds are in the ancillary file propositions\.json\. Table[32](https://arxiv.org/html/2610.12728#S11.T32)records what changed, result by result\.

Table 32:The referee record: the status of each numbered result after both rounds of review, and what the referees changed\.ResultStatusWhat the referees changed[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3), universal propertiesrepairedProducts need a fibre product in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}and can fail to exist \(LoRA1×Frozen\\mathrm\{LoRA\}\_\{1\}\\times\\text\{Frozen\}on1×11\\times 1weights\)\.[Obs\. I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),S1S\_\{1\}is functorialverifiedMonotonicity stated only along pointed morphisms\.[Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), additive monoidal structurerepaired twicer↦LoRArr\\mapsto\\mathrm\{LoRA\}\_\{r\}is strong monoidal only unpointed or at degenerate pointings \(dimS1\\dim S\_\{1\}ism​rmragainst2​m​r2mr\); HF’s cat merge described exactly\.[Prop\. I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6), a product and a meetverifiedLoRA\-XS needsrank⁡W0≥r\\rk W\_\{0\}\\geq r; the set\-level meet in \(b\) is no new method\.[Thm II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1), pointings of an additive methodverifiedStated for exact pointings, with a robust form; credited to[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324)\.[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), three strata of LoRA initialisationsrepairedFull\-rank pointings only, and only as a set; HF’s orthogonal init and QLoRA/LoftQ defects lie outside\.[Prop\. II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3), first\-order rank formularepairedScale and gradient flow made explicit; Adam’s first step has anℓ1\\ell\_\{1\}rate; LoRA\-GA’sA0A\_\{0\}, not itsB0B\_\{0\}, maximises the rate\.[Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4), zero gateverifiedHypotheses stated; a function\-level variant for LLaMA\-Adapter; AdaLoRA instance corrected\.[Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), complete invariantsverifiedMethods that can create zeros \(\(IA\)3, HiRA\) keep only “zeros ofW0W\_\{0\}stay zero”\.[Cor\. II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2), what orthogonal methods never dorepaired twiceExactly orthogonal input\-side maps only; BOFT keeps cosines up to sign; HF’s Cayley–Neumann OFT and GOFT’s scaler excluded\.[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), HRA as a meet; its apexrepaired twiceAn image\-level meet withW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), no product; strictly above LoRA∩\\capOFT only for exactly orthogonal OFT\.[Prop\. II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5), DoRA’s imagerepaired twiceDiagm⋅\(W0\+ℳ≤r\)\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)forW0W\_\{0\}without zero rows; the rank formula needs nonzero magnitudes; Conjecture D made falsifiable\.[Prop\. II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6), the torus ladderrepairedRoAd1\\mathrm\{RoAd\}\_\{1\}is \(IA\)3overℂ\\mathbb\{C\}; only zeros ofW0W\_\{0\}persist; \(IA\)3’s FFN vector acts on the input ofWdownW\_\{\\mathrm\{down\}\}\.[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), cut\-rank boundverifiedA large min\-cut is necessary for high rank and is not sufficient\.[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2), Kronecker incomparabilityrepaired twiceκ1\\kappa\_\{1\}is the maximal Kronecker rank for aligned splits; LoKr withr′≤min⁡\(m2,n2\)r^\{\\prime\}\\leq\\min\(m\_\{2\},n\_\{2\}\)lies inLoRAr\\mathrm\{LoRA\}\_\{r\}iffr≥min⁡\(m1,n1\)​r′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}\.[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), gauge censusrepaired twiceGraLoRA’s maximum rank ismin⁡\(k​r,m,n\)\\min\(kr,m,n\); LoHa loses dimensions toLoRA2​r\\mathrm\{LoRA\}\_\{2r\}only when2​r2<m\+n−12r^\{2\}<m\+n\-1\.[Prop\. II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4), transport of structurerepaired twiceAnyT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\); commutes with rebasing for additive families and fixedTT; FourierFT needs no conjugate pair\.[Obs\. II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), descentrepaired twiceReachable sets only, for box\-wise methods without shared frames; residual rotations need a gain\-fused pre\-norm model\.[Prop\. II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7), covariant splitsrepaired twiceMiLoRA and CorDA\-KPM are bottom\-rrsplits; trace\-relative damping keepsCO⁡\(n\)\\mathrm\{CO\}\(n\); shipped CorDA code is not covered\.[Obs\. III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2), chain rule for cometricsverifiedScope stated: gradient flow, no LoRA dropout, constant rates\.[Thm III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3), naturality of gradient descentverifiedThe title “natural only along isometries” was false in general and was replaced\.[Prop\. III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3), cometrics that descendrepairedUndamped \(S\)GD only; the published methods use AdamW or damping\.[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), Noether chargesrepaired twiceIsometries need the signal to factor through\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\); relabelled known; the per\-channel\-rate remark needs distinct rate products\.[Cor\. III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1), charge at initialisation; decayrepairedThe decay commentary was reversed: coupled decay tends toμ=0\\mu=0, per\-time decay toΦ=0\\Phi=0\.[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), Kempf–Ness sliceverifiedUniqueness up toO⁡\(r\)\\mathrm\{O\}\(r\)proved; the Loewner floor renamed\.[Prop\. III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), balance–escape trilemmaverifiedBalance generalised toB0⊤​B0=c​A0​A0⊤B\_\{0\}^\{\\top\}B\_\{0\}=cA\_\{0\}A\_\{0\}^\{\\top\}\.[Thm III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), isotropic\-charge dynamicsverifiedPointwise on the rank\-rrlocus; the Kaiming crossover is a band\.[Thm III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), the hyperparameter torusverifiedHypotheses explicit; extended to schedules, per\-blockε\\varepsilon, decay and clipping\.[Cor\. III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), LoRA\+ and rsLoRAverifiedA shared warm\-up keeps the torus; linear modules only\.[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8), optimizer gauge hierarchyrepaired twiceAdam respectsBrB\_\{r\}, positive diagonals only with per\-channel rates,ε\\varepsilonand decay; damping cutsGLr\\mathrm\{GL\}\_\{r\}toO⁡\(r\)\\mathrm\{O\}\(r\)\.[Cor\. III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9), what zero\-BBchoosesrepairedThe apex fibre is\{BA=0\}\\\{BA=0\\\}; theGLr\\mathrm\{GL\}\_\{r\}case needs equivariance along the whole trajectory\.[Prop\. IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), degree\-one collapserepairedDecoupled decay, default Adafactor and full clipping break it; GaLore carries optimizer state\.[Prop\. IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4), integrabilityrepaired twiceThe “only if” was false; the top\-rrgradient field is non\-involutive for everyr≥1r\\geq 1\.[Obs\. IV\.4](https://arxiv.org/html/2610.12728#S4.SS0.SSSx5), the backprop conerepaired twiceAn upper bound; forward mode pays off only fork=O⁡\(1\)k=O\(1\)\.[Prop\. V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2), quantisation and splittingrepaired twiceThree orders, three defects; HF’s LoftQ init ignores the LoRA scale; QA\-LoRA merges on\-grid\.[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), rebasing closurerepaired twiceBOFT reachesSO⁡\(n\)\\mathrm\{SO\}\(n\)only at full depth; HF’s Cayley–Neumann merges contract\.[Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4), combination rulesrepaired twiceThe “iff” needs full rank or continuity; factor\-wise DARE is gauge\-dependent too\.[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3), activations fix the basisverifiedGated FFNs have the full monomial group\.[Thm VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4), the merge calculusrepaired twiceSufficient conditions only; RoPE needs scales constant on rotary pairs\.[Prop\. VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5), linear insertions, ReFTrepaired twiceLoReFT has dimensionr⁡\(2​d\+1−r\)r\(2d\+1\-r\)and a pyreft pointing defect; the serial\-adapter criterion needsrank⁡A0=r\\rk A\_\{0\}=r\.[Thm VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6), attention as a homomorphismrepaired twiceRigidity is per layer; the prompt\-to\-prefix map credited to[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)\.A repair is a record of an error that was caught\. We keep the record public so that a reader can weigh each result by its history, and so that the next error, when someone finds it, has an obvious place to go\.

## EXPOSÉ XIIRelated work

This paper draws on three bodies of work: the PEFT literature, with its surveys and unified views; the theory of factorised models, their symmetries and their optimisation; and applied category theory\. We take them in turn and say, for each, what we use and where our questions differ\. Results that we re\-derive from earlier work carry that credit in their own headings and in the contributions of Exposé[0](https://arxiv.org/html/2610.12728#S0); here we place the paper as a whole\. The catalogue of Appendix[D](https://arxiv.org/html/2610.12728#A4)cites all 382 methods and 138 papers it classifies, and this section names the representatives of each family\.

##### Surveys, unified views and design spaces

The closest earlier unified frame is that of[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)\. They read adapters\([Houlsby et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib175)\), prefix tuning\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252)\)and LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)as modifications of specific hidden states that differ along four axes: functional form, insertion form, modified representation and composition function\. Their observation that prefix tuning is a softmax\-gated parallel adapter is part \(b\) of[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6), and their MAM adapter, like UniPELT\([Mao et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib315)\), is a hybrid in our classification\. The delta\-tuning study of[Ding et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib93)sorts methods into addition\-, specification\- and reparametrisation\-based families and adds optimisation and optimal\-control perspectives\. The guide of[Lialin et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib257), the reviews of[Xu et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib512)and[Han et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib153), the LoRA survey of[Mao et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib316), the vision survey of[Xin et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib511), the prompting survey of[Liu et al\. \(2021b\)](https://arxiv.org/html/2610.12728#bib.bib280)and the modular view of[Pfeiffer et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib369)drew the family boundaries that our catalogue starts from\.

Other work searches over configurations\.[Chen et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib60)parametrise a design space by layer grouping, parameter allocation, tunable groups and strategy assignment, and AutoPEFT\([Zhou et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib592)\)and NOAH\([Zhang et al\., 2022b](https://arxiv.org/html/2610.12728#bib.bib568)\)search it automatically\.[Si et al\. \(2024c\)](https://arxiv.org/html/2610.12728#bib.bib417)organise PEFT as subspace tuning relative to the SVD frame ofW0W\_\{0\}, which is close in spirit to our transport of structure \([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\), and[Fu et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib117)read many methods as sparse fine\-tuning of an equivalent model\. Recent comparisons under careful learning\-rate tuning\([He et al\., 2026a](https://arxiv.org/html/2610.12728#bib.bib162);[Lee et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib240)\)find that vanilla LoRA matches most of its variants;[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)gives one exact mechanism behind such coincidences\. Libraries made the field comparable in practice: AdapterHub\([Pfeiffer et al\., 2020b](https://arxiv.org/html/2610.12728#bib.bib367)\), Adapters\([Poth et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib373)\), LLM\-Adapters\([Hu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib178)\)and Hugging Face PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\), whose source settled many of the conventions used here\.

These works classify methods by where a module is inserted and how it is composed, and compare them by benchmark\. We classify by invariants of the mapρ\\rhointo weight space\. Two methods with different code can then be shown to have the same image, as DoRA and LoRA followed by an output scale do whenW0W\_\{0\}has no zero row \([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\)\. Two methods can be shown incomparable, as LoRA and GraLoRA are at equal budget \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\) and zero\-init LoRA and Kronecker sums with few terms are \([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\)\. Each of the seven coordinates of Exposé[VIII](https://arxiv.org/html/2610.12728#S8)is read off a formula and tied to a result\.

##### Low\-rank adaptation and its variants

LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)trainsW0\+s​B​AW\_\{0\}\+sBAfromB0=0B\_\{0\}=0\. Its variants change the scale \(rsLoRA,[Kalajdzievski, 2023](https://arxiv.org/html/2610.12728#bib.bib204)\), the per\-factor learning rates \(LoRA\+,[Hayou et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib159)\), which factor is trained \(LoRA\-FA,[Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556); the asymmetry study of[Zhu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib598)\), or the split into magnitude and direction \(DoRA,[Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282), after weight normalisation,[Salimans and Kingma, 2016](https://arxiv.org/html/2610.12728#bib.bib397); BiDoRA,[Qin et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib378); DeLoRA,[Bini et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib37)\)\. Others change the shape of the factors: shared or random bases \(VeRA,[Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224); NOLA,[Koohpayegani et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib223); VB\-LoRA,[Li et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib254); Tied\-LoRA,[Renduchintala et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib388); PRoLoRA,[Wang et al\., 2024g](https://arxiv.org/html/2610.12728#bib.bib474); RandLoRA,[Albert et al\., 2025a](https://arxiv.org/html/2610.12728#bib.bib7); Uni\-LoRA,[Li et al\., 2025a](https://arxiv.org/html/2610.12728#bib.bib248)\), a small core between frozen or orthonormal frames \(LoRA\-XS,[Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26); SVFT,[Lingam et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib272); LoRA\-SB,[Ponkshe et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib371); PoLAR,[Lion et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib273); TinyLoRA,[Morris et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib336)\), a single factor and its transpose \(SingLoRA,[Bensaïd et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib28)\), or one matrix shared across input shards \(MiSS,[Kang and Yin, 2024](https://arxiv.org/html/2610.12728#bib.bib206)\)\.

A further group changes the training procedure\. ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\), COLA\([Xia et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib507)\), PeriodicLoRA\([Meng et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib325)\)and ScaLoRA\([Zhang et al\., 2025c](https://arxiv.org/html/2610.12728#bib.bib567)\)merge and restart; Delta\-LoRA\([Zi et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib600)\)also updatesW0W\_\{0\}; LoRA\-Pro\([Wang et al\., 2024i](https://arxiv.org/html/2610.12728#bib.bib486)\), AltLoRA\([Yu et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib539)\)and LoFT\([Tastan et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib448)\)match the full\-model gradient or optimizer state; Flat\-LoRA\([Li et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib250)\)seeks flat minima\.[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)map a low\-regret regime in which LoRA matches full fine\-tuning, and prove the LoRA/Adam case of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\. Our results place several of these variants exactly: LoRA\-XS is the categorical product of the two one\-sided methods obtained by freezing LoRA’s frames at the SVD frames ofW0W\_\{0\}\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\), LoRA\+ and rsLoRA are points of a hyperparameter torus under the stated optimizer conditions \([Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\), ReLoRA reaches exactly the updates of rank at mostmin⁡\(K​r,m,n\)\\min\(Kr,m,n\)afterKKcycles \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\), and on the full\-rank locus the SGD variant of LoRA\-Pro follows, under gradient flow, the Riemannian gradient flow onW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\([Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\.

##### Multiplicative and orthogonal methods

Orthogonal fine\-tuning \(OFT,[Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\) multipliesW0W\_\{0\}by a learned orthogonal matrix to preserve hyperspherical energy\([Liu et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib283)\), following orthogonal over\-parametrised training\([Liu et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib284)\)\. BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\)builds the rotation from butterfly factors\([Dao et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib88)\); OFTv2\([Qiu et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib382)\)rotates inputs and replaces the Cayley inverse by a Neumann series; GOFT and qGOFT\([Ma et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib305)\)use Givens rotations, ETHER\([Bini et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib36)\)hyperplane reflections and HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)products of Householder reflections, a parametrisation that goes back to[Mhammedi et al\. \(2016\)](https://arxiv.org/html/2610.12728#bib.bib326)\. PSOFT\([Wu et al\., 2025a](https://arxiv.org/html/2610.12728#bib.bib498)\)confines the rotation to a principal subspace, LoCO\([Nguyen et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib347)\)uses Cayley rotations of low\-rank skew generators, and LOFT\([Zhao et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib582)\)chooses the support of the rotation from the task\. Spectral methods rotate or rescale the singular frame ofW0W\_\{0\}\(SVDiff,[Han et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib152); Spectral Adapter,[Zhang and Pilanci, 2024b](https://arxiv.org/html/2610.12728#bib.bib547); SVF,[Sun et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib435)\), and POET\-X\([Qiu et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib383)\)trains weights as orthogonal equivalence transforms of a fixed matrix\. Lie\-algebra charts such as the exponential and Cayley maps are studied by[Lezcano\-Casado and Martínez\-Rubio \(2019\)](https://arxiv.org/html/2610.12728#bib.bib244), and LieRA\([Si et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib418)\)lifts multiplicative updates to tensors through the exponential map\.

Diagonal and affine scalings form the other half of the multiplicative family: \(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\), SSF\([Lian et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib259)\), RED\([Wu et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib499)\)and LayerNorm tuning\([Zhao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib578)\), preceded by FiLM\([Perez et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib363)\)\. RoAd\([Liao and Monz, 2024](https://arxiv.org/html/2610.12728#bib.bib263)\)applies2×22\\times 2rotations, and HiRA\([Huang et al\., 2025b](https://arxiv.org/html/2610.12728#bib.bib182)\)and ABBA\([Singhal et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib421)\)use Hadamard products withW0W\_\{0\}or with a second learned factor\. These papers motivate orthogonality and scaling by what they preserve\. We make the preserved quantities exact\. The complete invariants of the basic actions \([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\) say what an exactly orthogonal method can never change in exact arithmetic \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\), also after any number of merges \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\), and they separate exact methods from approximately orthogonal implementations such as the Cayley–Neumann default of HF PEFT\. HRA’s bridge between low\-rank and orthogonal adaptation becomes, forW0W\_\{0\}of full column rank and evenr≤n−2r\\leq n\-2, an image\-level meet with an apex at the default initialisation \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\), and the torus ladder \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\) places one\- and two\-sided rescalings inside the images of rank\-one and rank\-two HiRA\.

##### Structured shapes

A second line changes the shape of the update\. Kronecker products enter with PHM layers\([Zhang et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib543)\)and Compacter\([Mahabadi et al\., 2021a](https://arxiv.org/html/2610.12728#bib.bib307)\), and KronA\([Edalati et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib103)\), KAdaptation\([He et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib167)\), LoKr\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\)and KRAdapter\([Albert et al\., 2025b](https://arxiv.org/html/2610.12728#bib.bib8)\)apply them to weight updates\. Hadamard products of low\-rank factors come from FedPara\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\), known as LoHa in LyCORIS\. Tensor\-network updates include LoRETTA\([Yang et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib529)\), TT\-LoRA\([Anjum et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib11)\), LoTR\([Bershatsky et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib29)\), FacT\([Jie and Deng, 2022b](https://arxiv.org/html/2610.12728#bib.bib199)\), the Tucker\-core FLoRA\([Si et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib416)\), LoRTA\([Hounie et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib176)\), MetaTT\([Lopez\-Piqueres et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib295)\)and QuanTA\([Chen et al\., 2024f](https://arxiv.org/html/2610.12728#bib.bib72)\), resting on the decompositions surveyed by[Kolda and Bader \(2009\)](https://arxiv.org/html/2610.12728#bib.bib222), the tensor train of[Oseledets \(2011\)](https://arxiv.org/html/2610.12728#bib.bib354)and the tensor networks of[Cichocki et al\. \(2016\)](https://arxiv.org/html/2610.12728#bib.bib78)and[Wang et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib468)\. Square and block shapes appear in MoRA\([Jiang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib196)\), MELoRA\([Ren et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib387)\), GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\), MoRe\([Tan et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib442)\), built on Monarch matrices\([Dao et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib89);[Qiu et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib380)\), and DiaBlo\([Gurses et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib147)\)\. Fixed\-frame coordinate methods include FourierFT\([Gao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib124)\), WaveFT\([Bilican et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib35)\), LoCA\([Du et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib101)\), SSH\([Shen et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib408)\)and the circulant C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\)\. Sparse supports appear in SHiRA\([Bhardwaj et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib30)\), RoSA\([Nikdan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib350)\), SLTrain\([Han et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib151)\)and DSEE\([Chen et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib68)\), and in the selective methods BitFit\([Ben\-Zaken et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib27)\), diff pruning\([Guo et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib144)\)and FISH\([Sung et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib438)\); Sine\-LoRA\([Ji et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib193)\)raises rank with an elementwise nonlinearity\.

Many of these papers aim for high rank at a small budget, and we separate the two quantities in that aim\. Rank is bounded by a min\-cut of a tensor network \([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\), and dimension by the budget minus the generic gauge orbit \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\), so a method can gain rank at the same dimension, as GraLoRA does, or gain rank while it loses dimension, as LoHa does in the regime2​r2<m\+n−12r^\{2\}<m\+n\-1\. The rearrangement of[Van Loan and Pitsianis \(1993\)](https://arxiv.org/html/2610.12728#bib.bib459)makes unconstrained Kronecker sums LoRA across a different cut of the same tensor \([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\), and the gauges and dimensions of Tucker and tensor\-train shapes are those of[Koch and Lubich \(2010\)](https://arxiv.org/html/2610.12728#bib.bib220),[Uschmajew and Vandereycken \(2013\)](https://arxiv.org/html/2610.12728#bib.bib454)and[Holtz et al\. \(2012\)](https://arxiv.org/html/2610.12728#bib.bib172)\.

##### Initialisation and rank allocation

Initialisation methods choose the frames that zero\-initialised LoRA leaves free, or splitW0W\_\{0\}\. PiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\)and MiLoRA\([Wang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib465)\)move the principal or minor singular components ofW0W\_\{0\}into the adapter, OLoRA\([Büyükakyüz, 2024](https://arxiv.org/html/2610.12728#bib.bib47)\)uses a QR split, and CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\)and Astra\([Liu et al\., 2026b](https://arxiv.org/html/2610.12728#bib.bib277)\)use activation statistics\. LoRA\-GA\([Wang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib472)\), LoRA\-One\([Zhang et al\., 2025d](https://arxiv.org/html/2610.12728#bib.bib569)\)and GoRA\([He et al\., 2025a](https://arxiv.org/html/2610.12728#bib.bib161)\)align the adapter with the first gradient, EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)with activations, KaSA\([Wang et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib464)\)and MiCA\([Rüdiger and Raschka, 2026](https://arxiv.org/html/2610.12728#bib.bib395)\)with minor singular directions, LoRA\-Null\([Tang et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib445)\)with a null space of activations, and KeepLoRA\([Luo et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib299)\)with a residual subspace;[Zhang et al\. \(2025g\)](https://arxiv.org/html/2610.12728#bib.bib576)argue that the magnitude of the update matters most\.[Hayou et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib158)showed that the two zero\-product starts, Init\[A\] and Init\[B\], train differently, and[Li et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib245)show that a Nyström initialisation changes the convergence rate of scaled gradient descent\. Rank allocation is a separate axis, with AdaLoRA\([Zhang et al\., 2023g](https://arxiv.org/html/2610.12728#bib.bib559)\), DyLoRA\([Valipour et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib456)\), SoRA\([Ding et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib94)\), IncreLoRA\([Zhang et al\., 2023a](https://arxiv.org/html/2610.12728#bib.bib548)\), ALoRA\([Liu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib292)\), AutoLoRA\([Zhang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib564)\), ElaLoRA\([Chang et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib54)\), LoRA\-drop\([Zhou et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib593)\), the dynamic\-rank DoRA of[Mao et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib314), FlexLoRA\([Liu et al\., 2026c](https://arxiv.org/html/2610.12728#bib.bib278)\)and MatryoshkaLoRA\([Modoranu et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib334)\); PLoP\([Hayou et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib160)\)chooses where adapters go\.

These works compare initialisations by training outcomes\. We first compare them as objects\. The three strata of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)are told apart byS1S\_\{1\}before any step is taken, zero\-factor starts are apexes of a cone, and no exact pointing reaches rank above2​r2r\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\. The2​r2rbound underlies the PiSSA\-to\-LoRA conversion of[Meng et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib324), and split initialisations attain it when2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\. The optimizer then decides how much of the chosen frame training actually sees \([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.

##### Quantised bases and function\-preserving reparametrisations

QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)trains LoRA over a 4\-bit base, building on blockwise quantisation\([Dettmers et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib91)\)\. LoftQ\([Li et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib255)\)fits the adapter to the quantisation error, and LQ\-LoRA\([Guo et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib145)\), IR\-QLoRA\([Qin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib377)\), ApiQ\([Liao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib266)\), LoQT\([Loeschcke et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib294)\)and SRR\([Cho et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib76)\)refine how the error is split; QA\-LoRA\([Xu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib514)\), PEQA\([Kim et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib214)\)and L4Q\([Jeon et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib191)\)keep the merged model quantised\. Outside PEFT, SmoothQuant\([Xiao et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib508)\)and AWQ\([Lin et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib268)\)migrate scales through activations, SliceGPT\([Ashkboos et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib18)\)and QuaRot\([Ashkboos et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib19)\)rotate the residual stream without changing the network function, and SVD\-LLM\([Wang et al\., 2025c](https://arxiv.org/html/2610.12728#bib.bib479)\)uses the whitened truncation that reappears in[Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\. Base change \(Exposé[V](https://arxiv.org/html/2610.12728#S5)\) separates the two kinds of map\. Function\-preserving reparametrisations act on the slice and leave reachable function sets unchanged under the hypotheses of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), while quantisers move the base point and leave explicit, order\-dependent defects \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\)\.

##### Theory of LoRA and of fine\-tuning

*Intrinsic dimension\.*[Li et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib246)measured how many random directions a task needs,[Aghajanyan et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib5)found that fine\-tuning a language model needs few,[Qin et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib379)found a shared low\-dimensional subspace for soft prompts, and[Gur\-Ari et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib146)observed that gradients concentrate in a small subspace early in training\. These notions depend on the data, while the expressive dimensiond⁡\(M\)d\(M\)is a property of the method alone\.*Kernels\.*[Malladi et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib309)found that prompt\-based fine\-tuning often follows kernel dynamics, in which LoRA approximately preserves the kernel; the background is the neural tangent kernel\([Jacot et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib189)\)and lazy training\([Chizat et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib74)\)\.[Jang et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib190)show that LoRA in the NTK regime has no spurious local minima, and[Li et al\. \(2026b\)](https://arxiv.org/html/2610.12728#bib.bib256)extend the kernel view to zeroth\-order fine\-tuning\.*Expressivity\.*[Zeng and Lee \(2023\)](https://arxiv.org/html/2610.12728#bib.bib542)bound the rank LoRA needs to represent a smaller target network\.

*Dynamics\.*[Hayou et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib159)derive LoRA\+ from an infinite\-width analysis in the spirit of[Yang and Hu \(2021\)](https://arxiv.org/html/2610.12728#bib.bib523), and[Chen et al\. \(2026a\)](https://arxiv.org/html/2610.12728#bib.bib63)extend it to joint limits in width and rank\.[Xu et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib515)analyse gradient flow for LoRA on matrix factorisation,[Kang et al\. \(2026\)](https://arxiv.org/html/2610.12728#bib.bib205)spectral flows of Muon type,[Zhang et al\. \(2026g\)](https://arxiv.org/html/2610.12728#bib.bib577)the role of the scale factor and[Mu and Klabjan \(2025\)](https://arxiv.org/html/2610.12728#bib.bib338)convergence rates of LoRA gradient descent\.*Landscapes\.*[Kim et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib215)show that LoRA training converges either to a low\-rank, small\-magnitude global minimiser or to a qualitatively different solution, in the tradition of benign landscapes for factorised problems\([Burer and Monteiro, 2003](https://arxiv.org/html/2610.12728#bib.bib46);[Ge et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib128)\)and of implicit low\-rank bias in matrix factorisation\([Gunasekar et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib143);[Arora et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib15)\)\.

Empirically, LoRA and full fine\-tuning differ in what they learn and forget\([Biderman et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib34);[Shuttleworth et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib414)\), fine\-tuning has its own scaling laws\([Zhang et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib544)\), and reinforcement\-learning fine\-tuning changes small and structured parts of the network\([Mukherjee et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib339);[Zhu et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib597);[Shenfeld et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib409)\)\. Our dynamical results are exact statements under stated optimizer hypotheses \(Exposé[III](https://arxiv.org/html/2610.12728#S3)\)\. They complement asymptotic and statistical analyses, and they say nothing about generalisation\.

##### Weight\-space symmetry and conserved quantities

The weight\-space symmetries of tanh networks were found by[Sussmann \(1992\)](https://arxiv.org/html/2610.12728#bib.bib440)and[Chen et al\. \(1993\)](https://arxiv.org/html/2610.12728#bib.bib58), and[Godfrey et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib132)computed the linear maps that commute with each common activation;[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)restates these facts\. Rescaling symmetries affect sharpness\([Dinh et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib95)\)and motivate optimizers that respect them, such as Path\-SGD\([Neyshabur et al\., 2015](https://arxiv.org/html/2610.12728#bib.bib346)\)\. Under gradient flow, symmetries give conservation laws: balancedness of linear layers\([Du et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib100);[Arora et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib14)\), the Noether\-type laws of[Kunin et al\. \(2020\)](https://arxiv.org/html/2610.12728#bib.bib230)and[Tanaka and Kunin \(2021\)](https://arxiv.org/html/2610.12728#bib.bib444), the conserved quantities of[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580), whose Prop\. 5\.1 is the core of[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), the complete count of independent laws by[Marcotte et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib318), and its extension beyond Euclidean flows, momentum included, by[Marcotte et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib319)\. Symmetry teleportation\([Zhao et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib579)\)moves along symmetry orbits to speed up optimisation\. Imbalance at initialisation governs the closed\-form dynamics of[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447)and[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328), which[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)re\-derives for LoRA, after the exact solutions of[Saxe et al\. \(2014\)](https://arxiv.org/html/2610.12728#bib.bib398)for deep linear networks; the lazy and rich regimes are analysed by[Dominé et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib97)and[Kunin et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib231)\.

Kempf–Ness theory\([Kempf and Ness, 1979](https://arxiv.org/html/2610.12728#bib.bib209);[Richardson and Slodowy, 1990](https://arxiv.org/html/2610.12728#bib.bib389)\)characterises closed orbits by norm\-minimising points, which for LoRA are the balanced factorisations \([Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\), with the nuclear\-norm identity of[Srebro et al\. \(2004\)](https://arxiv.org/html/2610.12728#bib.bib426); Balanced LoRA\([Castin et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib51)\)turns this slice into an algorithm\. Permutation symmetry is used to align and merge whole models\([Ainsworth et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib6);[Entezari et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib105);[Singh and Jaggi, 2020](https://arxiv.org/html/2610.12728#bib.bib420)\)and to build networks that process weights\([Navon et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib343);[Zhou et al\., 2023a](https://arxiv.org/html/2610.12728#bib.bib591)\), and[Putterman et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib375)build learners on LoRA weights that respect theGLr\\mathrm\{GL\}\_\{r\}symmetry\. Our additions are the PEFT instances\. Per\-block rates split LoRA’s gauge into isometries and conserved charges \([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\), the optimizer decides which part of the gauge descends to the weights \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\), and combination rules must respect the gauge \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\. FedRot\-LoRA\([Zhang et al\., 2026c](https://arxiv.org/html/2610.12728#bib.bib550)\)and FLoRG\([Meng et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib323)\)meet the same gauge in federated averaging, and PRISM\([Wang and Zhang, 2026](https://arxiv.org/html/2610.12728#bib.bib475)\)in differentially private training\([Abadi et al\., 2016](https://arxiv.org/html/2610.12728#bib.bib1)\)\.

##### Riemannian and preconditioned optimisation on low\-rank manifolds

Optimisation over fixed\-rank matrices has a mature Riemannian theory\([Absil et al\., 2008](https://arxiv.org/html/2610.12728#bib.bib2);[Vandereycken, 2013](https://arxiv.org/html/2610.12728#bib.bib460)\), including the quotient geometry of factorisations moduloGLr\\mathrm\{GL\}\_\{r\}\([Mishra et al\., 2012](https://arxiv.org/html/2610.12728#bib.bib330);[Mishra and Sepulchre, 2016](https://arxiv.org/html/2610.12728#bib.bib331)\)and dynamical low\-rank approximation\([Koch and Lubich, 2010](https://arxiv.org/html/2610.12728#bib.bib220)\); natural gradients\([Amari, 1998](https://arxiv.org/html/2610.12728#bib.bib10)\)and Riemannian gradient flows of deep linear networks\([Bah et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib22)\)belong to the same circle of ideas\. Scaled gradient descent\([Tong et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib452)\)preconditions each factor by the Gram matrix of the other, and[Zhang and Pilanci \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib546)bring it to LoRA\. LoRA\-RITE\([Yen et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib533)\)makes invariance under reparametrisation of the factors a design requirement, LoRA\-S\([Zheng and Wu, 2026](https://arxiv.org/html/2610.12728#bib.bib587)\)lifts matrix\-space optimizers through the quotient, Riemannion\([Bogachev et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib41)\)takes a Muon step\([Jordan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib201)\)onℳr\\mathcal\{M\}\_\{r\}directly, and PoLAR\([Lion et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib273)\)trains orthonormal frames by Riemannian landing;[Aliev and Rakhuba \(2026\)](https://arxiv.org/html/2610.12728#bib.bib9)study the Riemannian structure of structured orthogonal classes\. We sort these update rules, together with SGD, Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.12728#bib.bib216)\)and AdamW\([Loshchilov and Hutter, 2019](https://arxiv.org/html/2610.12728#bib.bib296)\), by the largest subgroup of the gauge they respect \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\. On LoRA’s full\-rank locus with channel\-uniform hyperparameters, this isO⁡\(r\)\\mathrm\{O\}\(r\)for SGD, the signed permutations for Adam and AdamW, and all ofGLr\\mathrm\{GL\}\_\{r\}for undamped scaled GD and LoRA\-RITE\.[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)identifies which cometrics descend to the rank\-rrmanifold\.

##### Gradient projection and memory

GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)projects gradients onto a low\-rank subspace that is refreshed periodically\. Its relatives include GaLore 2\([Su et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib431)\), Q\-GaLore\([Zhang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib574)\), Fira\([Chen et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib67)\), APOLLO\([Zhu et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib596)\), Online Subspace Descent\([Liang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib260)\), LDAdam\([Robert et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib392)\)and LoRA\-Pre\([Wang et al\., 2026c](https://arxiv.org/html/2610.12728#bib.bib487)\), which compresses the momentum; Flora\([Hao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib154)\)uses random projections, and[He et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib168)analyse when such methods converge\. Memory is also saved by compressing optimizer state\([Shazeer and Stern, 2018](https://arxiv.org/html/2610.12728#bib.bib407);[Dettmers et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib91);[Zhang et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib570)\), by zeroth\-order training \(MeZO,[Malladi et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib310); see the benchmark of[Zhang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib566)\), by fused updates \(LOMO,[Lv et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib303); AdaLomo,[Lv et al\., 2023a](https://arxiv.org/html/2610.12728#bib.bib302)\), by sampling layers \(LISA,[Pan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib360); BAdam,[Luo et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib300)\), by side networks \(LST,[Sung et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib439)\) and by compressing activations \(VeLoRA,[Miles et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib327)\)\.[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)proved the duality between linear gradient transformations and adapters that[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)restates in lens form with explicit optimizer hypotheses\. The integrability condition \([Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\) and the backward cone \([Observation IV\.4](https://arxiv.org/html/2610.12728#S4.SS0.SSSx5)\) sort the rest of this family\. In continual learning, gradient projection memory\([Saha et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib396)\)and orthogonal gradient descent\([Farajtabar et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib109)\)project updates away from the subspaces that earlier tasks use, an idea that O\-LoRA\([Wang et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib477)\)and InfLoRA\([Liang and Li, 2024](https://arxiv.org/html/2610.12728#bib.bib261)\)carry over to adapters\.

##### Prompts and prefixes, and their limits

Soft prompts\([Lester et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib243);[Liu et al\., 2021d](https://arxiv.org/html/2610.12728#bib.bib288)\), deep prefixes\([Li and Liang, 2021](https://arxiv.org/html/2610.12728#bib.bib252);[Liu et al\., 2021c](https://arxiv.org/html/2610.12728#bib.bib287)\)and their variants \([Vu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib462);[Asai et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib17);[Wang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib485);[Shi and Lipani, 2023](https://arxiv.org/html/2610.12728#bib.bib411);[Razdaibiedina et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib384); in vision,[Jia et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib194);[Zhou et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib594);[Khattak et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib212)\) extend the network and leave its weights in place\.[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364)show that a prefix cannot change the relative attention pattern over the content and can only bias the output of an attention layer, while[Petrov et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib365)show that prefix\-tuning a pretrained transformer can still be a universal approximator\.[Wang et al\. \(2023c\)](https://arxiv.org/html/2610.12728#bib.bib483)prove universality and limitations of prompt tuning,[Oymak et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib357)analyse prompt\-attention,[Le et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib234)the statistical benefit of the prefix reparametrisation, and[Galim et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib119)show that for state\-space models prefix tuning is bounded by initial\-state tuning\.[Chen et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib69)connect prefixes to adapters through kernel estimators,[Khashabi et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib211)and[Su et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib433)study what soft prompts encode and how they transfer, and[Mondal and Shihab \(2026\)](https://arxiv.org/html/2610.12728#bib.bib335)give exact resource\-capped tests for when a prefix can reproduce a LoRA update at one head\.

[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)organises the attention side of this work as a monoid homomorphism followed by a perspective map\. It recovers the gated\-adapter reading of[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165)and the per\-layer rigidity of[Petrov et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib364), adds that LLaMA\-Adapter’s\([Zhang et al\., 2023h](https://arxiv.org/html/2610.12728#bib.bib562)\)separate softmax leaves the content attention weights unchanged, and shows that for a vanilla softmax head withWo≠0W\_\{o\}\\neq 0andWq≠0W\_\{q\}\\neq 0a generic prefix cannot be folded into the head’s own weights\. Representation edits \(ReFT,[Wu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib505); RED,[Wu et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib499); LoFiT,[Yin et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib535)\) and steering directions\([Li et al\., 2023a](https://arxiv.org/html/2610.12728#bib.bib249);[Zou et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib601)\)also act on activations, and[Adila et al\. \(2026\)](https://arxiv.org/html/2610.12728#bib.bib4)relate weight updates to activation shifts directly;[Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)treats linear insertions and ReFT\.

##### Model merging and composition

Weight averaging\([Wortsman et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib497);[Wortsman et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib496)\), Fisher\-weighted merging\([Matena and Raffel, 2021](https://arxiv.org/html/2610.12728#bib.bib320)\), RegMean\([Jin et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib200)\), task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\)and its tangent\-space analysis\([Ortiz\-Jimenez et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib352)\), TIES\([Yadav et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib517)\), DARE\([Yu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib538)\), AdaMerging\([Yang et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib521)\)and TSV\-Merge\([Gargiulo et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib125)\)combine full fine\-tuning updates, and MergeKit\([Goddard et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib131)\)packages many of these rules\. Linear mode connectivity\([Frankle et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib116)\)and permutation alignment\([Ainsworth et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib6);[Entezari et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib105)\)help explain when averaging works, and surveys and large\-scale studies map the field\([Yang et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib522);[Yadav et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib518);[Wang et al\., 2025e](https://arxiv.org/html/2610.12728#bib.bib484)\)\. For adapters, LoraHub\([Huang et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib180)\), LoRA Soups\([Prabhakar et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib374)\), AdapterSoup\([Chronopoulou et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib77)\), PEM arithmetic\([Zhang et al\., 2023c](https://arxiv.org/html/2610.12728#bib.bib554)\), ZipLoRA\([Shah et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib404)\), Mix\-of\-Show\([Gu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib139)\)and orthogonal adaptation\([Po et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib370)\)combine modules\. KnOTS\([Stoica et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib429)\), Core Space\([Panariello et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib361)\)and Compress\-then\-Merge\([He et al\., 2026c](https://arxiv.org/html/2610.12728#bib.bib169)\)first express the modules in a shared frame, and OrthoMerge\([Yang et al\., 2026a](https://arxiv.org/html/2610.12728#bib.bib524)\)averages OFT rotations in the Lie algebra\. Poly\([Ponti et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib372)\), MHR\([Caccia et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib48)\)and routed mixtures\([Wu et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib501);[Dou et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib99);[Buehler and Buehler, 2024](https://arxiv.org/html/2610.12728#bib.bib45)\)train combinations jointly, and serving systems such as S\-LoRA\([Sheng et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib410)\)and Punica\([Chen et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib61)\)keep many adapters separate at inference\.[Liu et al\. \(2026a\)](https://arxiv.org/html/2610.12728#bib.bib276)find that adaptive merging of recycled LoRAs gives limited benefit over training a new adapter\.

We contribute two exact constraints to this literature\. A rule for combining LoRA factors is a function of the adapters only when it is constant on factorisation fibres; task arithmetic on the products satisfies this, and post\-hoc product\-of\-sums rules such as LoraHub’s do not \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\. Folding a trained edit into existing weights is certified by a local rewrite calculus whose side conditions depend on the activation \([Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\); it gives sufficient conditions and box\-local obstructions, and SmoothQuant and AWQ already use the same scale migration\.

##### Category theory and geometry for machine learning

[Fong et al\. \(2017\)](https://arxiv.org/html/2610.12728#bib.bib114)defined learners as the morphisms of a symmetric monoidal category,[Fong and Johnson \(2019\)](https://arxiv.org/html/2610.12728#bib.bib113)related learners to lenses,[Cockett et al\. \(2019\)](https://arxiv.org/html/2610.12728#bib.bib79)axiomatised reverse derivative categories, and[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)gave a semantics of gradient\-based learning in𝐏𝐚𝐫𝐚⁡\(𝐋𝐞𝐧𝐬⁡\(𝒞\)\)\\mathbf\{Para\}\(\\mathbf\{Lens\}\(\\mathcal\{C\}\)\), which we use directly for the lens of Exposé[IV](https://arxiv.org/html/2610.12728#S4)\. Parametrised optics\([Capucci et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib50)\), learners’ languages\([Spivak, 2022](https://arxiv.org/html/2610.12728#bib.bib425)\), reverse derivative ascent\([Wilson and Zanasi, 2021](https://arxiv.org/html/2610.12728#bib.bib495)\)and the categorical account of automatic differentiation by[Elliott \(2018\)](https://arxiv.org/html/2610.12728#bib.bib104)belong to the same programme, which[Shiebler et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib412)survey\.[Gavranović et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib127)propose monads valued in a 2\-category of parametric maps as an algebraic theory of architectures, developed at length in[Gavranović \(2024\)](https://arxiv.org/html/2610.12728#bib.bib126)\. Geometric deep learning\([Bronstein et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib44)\), after group\-equivariant\([Cohen and Welling, 2016](https://arxiv.org/html/2610.12728#bib.bib82)\)and gauge\-equivariant\([Cohen et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib83)\)networks, organises architectures by their symmetries in the spirit of Klein’s programme\([Klein, 1872](https://arxiv.org/html/2610.12728#bib.bib217)\)\. The calculus of string diagrams we use was put on a rigorous footing by[Joyal and Street \(1991\)](https://arxiv.org/html/2610.12728#bib.bib202), and the spiders we use for copying come from[Coecke et al\. \(2013\)](https://arxiv.org/html/2610.12728#bib.bib81)\.

These works give categorical semantics to architectures and learning algorithms\. We study a narrower object, the fine\-tuning map relative to one fixed checkpoint, as an object of the slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\. The category theory is used in the seven places listed in Exposé[0](https://arxiv.org/html/2610.12728#S0), and the rest of the work is real algebraic geometry, invariant theory, Lie theory and gradient\-flow analysis\. Our Erlangen language applies Klein’s idea to the fine\-tuning map, where geometric deep learning applies it to architectures, and our gauge is the symmetry group of a parametrisation, with no connection, unlike the gauge of gauge\-equivariant networks \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

## EXPOSÉ XIIIConclusion

Fine\-tuning moves a point\. Treating a PEFT method as a pointed mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\)into weight space, and studying the map more than the module that implements it, turns many informal comparisons between methods into questions with exact answers\. Three invariants of the map do most of the work\. The image germ atθ0\\theta\_\{0\}governs what a method can reach, through its first\-order imageS1S\_\{1\}, its expressive dimensiond⁡\(M\)d\(M\), its cut\-rank and the type of its base point\. The fibres, together with the optimizer’s metric, govern how it trains\. Base change governs how it travels when it is initialised by splitting, quantised, restarted or merged\. Extensions such as adapters and prefixes add one further layer, where the network itself enters and merging becomes a lifting problem\.

Several results follow, each with the scope of its proof\. Zero\-initialised LoRA starts at the apex of a cone, and so does HRA at the paired initialisation of HF PEFT whenW0W\_\{0\}has full column rank andr≤n−2r\\leq n\-2is even \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\. No exact initialisation lets LoRA reach an update of rank above2​r2r\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\. Exactly orthogonal methods never change the singular values of a weight in exact arithmetic, however often they are merged \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\. Rank is a cut and dimension is a volume, so at equal budget a structured method can gain rank at the same dimension, as GraLoRA does, or trade dimension for rank, as LoHa does when2​r2<m\+n−12r^\{2\}<m\+n\-1andr≥3r\\geq 3\([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. Under Adam with noε\\varepsilon, weight decay or clipping and one shared schedule, LoRA\+ follows exactly the trajectory of LoRA withα\\alphamultiplied by the learning\-rate ratio \([Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7); the LoRA/Adam case is due to[Schulman and Thinking Machines Lab, 2025](https://arxiv.org/html/2610.12728#bib.bib401)\), and on LoRA’s full\-rank locus Adam with channel\-uniform hyperparameters respects only the signed permutations inside theGLr\\mathrm\{GL\}\_\{r\}gauge \([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\. Whether a scaling adapter folds into the weights before it depends on the activation, and a local rewrite calculus certifies when it does \([Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\)\. The atlas reads the same coordinates off 382 methods from 138 papers and records 403 simulation arrows and 485 obstructions, each backed by an explicit map or a named test\.

Several of these results are short once the setting is in place, which is what the rising sea on the title page refers to \(Appendix[F](https://arxiv.org/html/2610.12728#A6)\)\. In the slice, re\-pointing an additive method sweeps out the difference setC−CC\-Cof its update shape, and for LoRArthat set is the matrices of rank at most2​r2r\. The bound on every exact initialisation is then a few lines \([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\. Once LoRA’s factors are seen to carry a torus of rescalings that fixes the weight trajectory, LoRA\+ is a point on that torus \([Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\. In the lens, GaLore without weight decay becomes one\-sided LoRA on a moving base by a one\-line induction \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\. Read off a tensor diagram, the maximum rank of a structured adapter is a min\-cut \([Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\)\. Three of these four facts were known before this paper, from the PiSSA\-to\-LoRA conversion\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\),[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)and[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)\. What the setting adds is a short proof, an exact scope and the same proof for every method of the same shape\. Other results did not dissolve\. The classification of initialisations \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\) and rebasing closure \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\) required real algebraic geometry and Lie theory, and their proofs say where\.

For practitioners these statements become checks that can be made before training\. Compare methods at equal expressive dimension and cut\-rank as well as at equal budget\. Ask whether an initialisation is an apex, and which part of its frame the optimizer will see \([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\. Before treating a new hyperparameter as a new method, check whether it lies on a hyperparameter torus\. Before merging adapters factor by factor, check that the rule is constant on factorisation fibres \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\. Before relying on an orthogonal method to keep a spectrum, check that its implementation is exactly orthogonal\. Implementations matter throughout\. Several results distinguish a method as published from its HF PEFT implementation, such as OFT’s Cayley–Neumann default, HRA’s paired initialisation and LoftQ’s treatment of the scaleα/r\\alpha/r\([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\), and the catalogue records which form each statement covers\.

The limits are part of the statements\. Expressivity is not learnability, and only Exposé[III](https://arxiv.org/html/2610.12728#S3)is dynamical\. Its conservation laws hold under gradient flow, drift under discrete steps by an amount of second order in the step size relative to the charge, and do not cover Adam or momentum\. The classification of initialisations is set\-level and in exact arithmetic\. Mergeability is certified box by box under explicit side conditions, and the rewrite rules give sufficient conditions and local obstructions; they are not a decision procedure for whole networks\. Words such as Noether, Erlangen, gauge and moduli are analogies beyond what is proved, and Exposé[XI](https://arxiv.org/html/2610.12728#S11)says where each one stops\.

The falsifiable predictions of Exposé[X](https://arxiv.org/html/2610.12728#S10)say where the theory can be tested, each with its optimizer scope\. The open problems of Exposé[XI](https://arxiv.org/html/2610.12728#S11)say where it should grow next: a dynamical classification of initialisations modulo the optimizer’s shadow of the gauge, a conserved or monotone quantity for sign\-like updates such as Adam’s, the frontier of pairwise incomparable methods at a fixed budget, and whole\-network statements about prefix tuning\. We hope the frame serves as a habit of asking three questions of any new method: what its map can reach, how its fibres meet the optimizer, and what happens to it when the base point moves\.

## References

- Abadi et al\. \(2016\)Martín Abadi, Andy Chu, Ian Goodfellow, H\. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang\.Deep learning with differential privacy\.In*Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security \(CCS\)*, pages 308–318\. ACM, 2016\.doi:10\.1145/2976749\.2978318\.
- Absil et al\. \(2008\)P\.\-A\. Absil, R\. Mahony, and R\. Sepulchre\.*Optimization Algorithms on Matrix Manifolds*\.Princeton University Press, Princeton, NJ, 2008\.doi:10\.1515/9781400830244\.
- Adepu et al\. \(2026\)Harshavardhan Adepu, Li Zhang, Sanjiv Kumar, and Vikas Singh\.Fine\-Tuning of Transformer models with Frames\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2608\.26430](https://arxiv.org/abs/2608.26430)\.
- Adila et al\. \(2026\)Dyah Adila, John Cooper, Alexander Yun, Avi Trost, and Frederic Sala\.Weight Updates as Activation Shifts: A Principled Framework for Steering\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2603\.00425](https://arxiv.org/abs/2603.00425)\.
- Aghajanyan et al\. \(2020\)Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta\.Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine\-Tuning\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2020\.[https://arxiv\.org/abs/2012\.13255](https://arxiv.org/abs/2012.13255)\.
- Ainsworth et al\. \(2022\)Samuel K\. Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa\.Git Re\-Basin: Merging Models modulo Permutation Symmetries\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2209\.04836](https://arxiv.org/abs/2209.04836)\.
- Albert et al\. \(2025a\)Paul Albert, Frederic Z\. Zhang, Hemanth Saratchandran, Cristian Rodriguez\-Opazo, Anton van den Hengel, and Ehsan Abbasnejad\.RandLoRA: Full\-rank parameter\-efficient fine\-tuning of large models\.In*International Conference on Learning Representations \(ICLR\)*, 2025a\.[https://arxiv\.org/abs/2502\.00987](https://arxiv.org/abs/2502.00987)\.
- Albert et al\. \(2025b\)Paul Albert, Frederic Z\. Zhang, Hemanth Saratchandran, Anton van den Hengel, and Ehsan Abbasnejad\.Towards Higher Effective Rank in Parameter\-efficient Fine\-tuning using Khatri–Rao Product\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2025b\.[https://arxiv\.org/abs/2508\.00230](https://arxiv.org/abs/2508.00230)\.
- Aliev and Rakhuba \(2026\)Ali Aliev and Maxim Rakhuba\.Riemannian Structure and Optimization for a Class of Low\-Parametric Orthogonal Matrices\.*arXiv preprint arXiv:2609\.27982*, 2026\.[https://arxiv\.org/abs/2609\.27982](https://arxiv.org/abs/2609.27982)\.
- Amari \(1998\)Shun\-ichi Amari\.Natural gradient works efficiently in learning\.*Neural Computation*, 10\(2\):251–276, 1998\.doi:10\.1162/089976698300017746\.
- Anjum et al\. \(2024\)Afia Anjum, Maksim E\. Eren, Ismael Boureima, Boian Alexandrov, and Manish Bhattarai\.Tensor Train Low\-rank Approximation \(TT\-LoRA\): Democratizing AI with Accelerated LLMs\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2408\.01008](https://arxiv.org/abs/2408.01008)\.
- Ansell et al\. \(2021\)Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić\.Composable Sparse Fine\-Tuning for Cross\-Lingual Transfer\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://aclanthology\.org/2022\.acl\-long\.125/](https://aclanthology.org/2022.acl-long.125/)\.
- Ansell et al\. \(2024\)Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M\. Ponti\.Scaling Sparse Fine\-Tuning to Large Language Models\.*arXiv preprint arXiv:2401\.16405*, 2024\.[https://arxiv\.org/abs/2401\.16405](https://arxiv.org/abs/2401.16405)\.
- Arora et al\. \(2018\)Sanjeev Arora, Nadav Cohen, and Elad Hazan\.On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2018\.[https://arxiv\.org/abs/1802\.06509](https://arxiv.org/abs/1802.06509)\.
- Arora et al\. \(2019\)Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo\.Implicit Regularization in Deep Matrix Factorization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2019\.[https://arxiv\.org/abs/1905\.13655](https://arxiv.org/abs/1905.13655)\.
- Artin \(1957\)Emil Artin\.*Geometric Algebra*\.Number 3 in Interscience Tracts in Pure and Applied Mathematics\. Interscience Publishers, New York, 1957\.
- Asai et al\. \(2022\)Akari Asai, Mohammadreza Salehi, Matthew E\. Peters, and Hannaneh Hajishirzi\.ATTEMPT: Parameter\-Efficient Multi\-task Tuning via Attentional Mixtures of Soft Prompts\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2022\.[https://aclanthology\.org/2022\.emnlp\-main\.446/](https://aclanthology.org/2022.emnlp-main.446/)\.
- Ashkboos et al\. \(2024a\)Saleh Ashkboos, Maximilian L\. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman\.SliceGPT: Compress large language models by deleting rows and columns\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.[https://arxiv\.org/abs/2401\.15024](https://arxiv.org/abs/2401.15024)\.
- Ashkboos et al\. \(2024b\)Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L\. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman\.QuaRot: Outlier\-free 4\-bit inference in rotated LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024b\.[https://arxiv\.org/abs/2404\.00456](https://arxiv.org/abs/2404.00456)\.
- Azizi et al\. \(2024\)Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram\.LaMDA: Large Model Fine\-Tuning via Spectrally Decomposed Low\-Dimensional Adaptation\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, 2024\.[https://aclanthology\.org/2024\.findings\-emnlp\.563/](https://aclanthology.org/2024.findings-emnlp.563/)\.
- Bagherifard et al\. \(2025\)Mohammadtaha Bagherifard, Sahar Rajabi, Ali Edalat, and Yadollah Yaghoobzadeh\.GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2025\.[https://aclanthology\.org/2025\.acl\-short\.54/](https://aclanthology.org/2025.acl-short.54/)\.
- Bah et al\. \(2022\)Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenberg\.Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizers\.*Information and Inference: A Journal of the IMA*, 11\(1\):307–353, 2022\.doi:10\.1093/imaiai/iaaa039\.
- Bahng et al\. \(2022\)Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola\.Exploring Visual Prompts for Adapting Large\-Scale Models\.*arXiv preprint arXiv:2203\.17274*, 2022\.[https://arxiv\.org/abs/2203\.17274](https://arxiv.org/abs/2203.17274)\.
- Bai and Yin \(1993\)Z\. D\. Bai and Y\. Q\. Yin\.Limit of the smallest eigenvalue of a large dimensional sample covariance matrix\.*The Annals of Probability*, 21\(3\):1275–1294, 1993\.doi:10\.1214/aop/1176989118\.
- Bartels and Stewart \(1972\)R\. H\. Bartels and G\. W\. Stewart\.Algorithm 432: Solution of the matrix equationA​X\+X​B=CAX\+XB=C\.*Communications of the ACM*, 15\(9\):820–826, 1972\.doi:10\.1145/361573\.361582\.
- Bałazy et al\. \(2024\)Klaudia Bałazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor\.LoRA\-XS: Low\-Rank Adaptation with Extremely Small Number of Parameters\.In*ECAI*, 2024\.[https://arxiv\.org/abs/2405\.17604](https://arxiv.org/abs/2405.17604)\.
- Ben\-Zaken et al\. \(2021\)Elad Ben\-Zaken, Shauli Ravfogel, and Yoav Goldberg\.BitFit: Simple Parameter\-efficient Fine\-tuning for Transformer\-based Masked Language\-models\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://aclanthology\.org/2022\.acl\-short\.1/](https://aclanthology.org/2022.acl-short.1/)\.
- Bensaïd et al\. \(2025\)David Bensaïd, Noam Rotstein, Roy Velich, Daniel Bensaïd, and Ron Kimmel\.SingLoRA: Low Rank Adaptation Using a Single Matrix\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2507\.05566](https://arxiv.org/abs/2507.05566)\.
- Bershatsky et al\. \(2024\)Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, Aleksandr Mikhalev, and Ivan Oseledets\.LoTR: Low Tensor Rank Weight Adaptation\.*arXiv preprint arXiv:2402\.01376*, 2024\.[https://arxiv\.org/abs/2402\.01376](https://arxiv.org/abs/2402.01376)\.
- Bhardwaj et al\. \(2024\)Kartikeya Bhardwaj, Nilesh Prasad Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Shreya Kadambi, Rafael Esteves, Shubhankar Borse, Paul Whatmough, Risheek Garrepalli, Mart Van Baalen, Harris Teague, and Markus Nagel\.Sparse High Rank Adapters\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2406\.13175](https://arxiv.org/abs/2406.13175)\.
- Bhatia \(1997\)Rajendra Bhatia\.*Matrix Analysis*, volume 169 of*Graduate Texts in Mathematics*\.Springer, New York, 1997\.doi:10\.1007/978\-1\-4612\-0653\-8\.
- Bhatia and Rosenthal \(1997\)Rajendra Bhatia and Peter Rosenthal\.How and why to solve the operator equationA​X−X​B=YAX\-XB=Y\.*Bulletin of the London Mathematical Society*, 29\(1\):1–21, 1997\.doi:10\.1112/S0024609396001828\.
- Bhatia et al\. \(2019\)Rajendra Bhatia, Tanvi Jain, and Yongdo Lim\.On the Bures–Wasserstein distance between positive definite matrices\.*Expositiones Mathematicae*, 37\(2\):165–191, 2019\.doi:10\.1016/j\.exmath\.2018\.01\.002\.
- Biderman et al\. \(2024\)Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P\. Cunningham\.LoRA Learns Less and Forgets Less\.*TMLR*, 2024\.[https://arxiv\.org/abs/2405\.09673](https://arxiv.org/abs/2405.09673)\.
- Bilican et al\. \(2025\)Ahmet Bilican, M\. Akın Yılmaz, A\. Murat Tekalp, and R\. Gökberk Cinbiş\.Exploring Sparsity for Parameter Efficient Fine Tuning Using Wavelets for Vision\.*arXiv preprint arXiv:2505\.12532*, 2025\.[https://arxiv\.org/abs/2505\.12532](https://arxiv.org/abs/2505.12532)\.
- Bini et al\. \(2024\)Massimo Bini, Karsten Roth, Zeynep Akata, and Anna Khoreva\.ETHER: Efficient Finetuning of Large\-Scale Models with Hyperplane Reflections\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2405\.20271](https://arxiv.org/abs/2405.20271)\.
- Bini et al\. \(2025\)Massimo Bini, Leander Girrbach, and Zeynep Akata\.DeLoRA: Decoupling Angles and Strength in Low\-rank Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2503\.18225](https://arxiv.org/abs/2503.18225)\.
- Birkes \(1971\)David Birkes\.Orbits of linear algebraic groups\.*Annals of Mathematics*, 93\(3\):459–475, 1971\.doi:10\.2307/1970884\.
- Blau et al\. \(2024\)Tsachi Blau, Moshe Kimhi, Yonatan Belinkov, Alexander Bronstein, and Chaim Baskin\.Context\-aware Prompt Tuning: Advancing In\-Context Learning with Adversarial Methods\.*arXiv preprint arXiv:2410\.17222*, 2024\.[https://arxiv\.org/abs/2410\.17222](https://arxiv.org/abs/2410.17222)\.
- Bochnak et al\. \(1998\)Jacek Bochnak, Michel Coste, and Marie\-Françoise Roy\.*Real Algebraic Geometry*, volume 36 of*Ergebnisse der Mathematik und ihrer Grenzgebiete \(3\)*\.Springer, Berlin, Heidelberg, 1998\.doi:10\.1007/978\-3\-662\-03718\-8\.
- Bogachev et al\. \(2025\)Vladimir Bogachev, Vladimir Aletov, Alexander Molozhavenko, Denis Bobkov, Vera Soboleva, Aibek Alanov, and Maxim Rakhuba\.LoRA meets Riemannion: Muon Optimizer for Parametrization\-independent Low\-Rank Adapters\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2507\.12142](https://arxiv.org/abs/2507.12142)\.
- Bostock et al\. \(2011\)Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer\.D3: Data\-driven documents\.*IEEE Transactions on Visualization and Computer Graphics*, 17\(12\):2301–2309, 2011\.doi:10\.1109/TVCG\.2011\.185\.
- Boyd and Vandenberghe \(2004\)Stephen Boyd and Lieven Vandenberghe\.*Convex Optimization*\.Cambridge University Press, Cambridge, 2004\.doi:10\.1017/CBO9780511804441\.
- Bronstein et al\. \(2021\)Michael M\. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković\.Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges\.*arXiv preprint arXiv:2104\.13478*, 2021\.[https://arxiv\.org/abs/2104\.13478](https://arxiv.org/abs/2104.13478)\.
- Buehler and Buehler \(2024\)Eric L\. Buehler and Markus J\. Buehler\.X\-LoRA: Mixture of Low\-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design\.*APL Machine Learning*, 2024\.[https://arxiv\.org/abs/2402\.07148](https://arxiv.org/abs/2402.07148)\.
- Burer and Monteiro \(2003\)Samuel Burer and Renato D\. C\. Monteiro\.A nonlinear programming algorithm for solving semidefinite programs via low\-rank factorization\.*Mathematical Programming*, 95\(2\):329–357, 2003\.doi:10\.1007/s10107\-002\-0352\-8\.
- Büyükakyüz \(2024\)Kerim Büyükakyüz\.OLoRA: Orthonormal Low\-Rank Adaptation of Large Language Models\.*arXiv preprint arXiv:2406\.01775*, 2024\.[https://arxiv\.org/abs/2406\.01775](https://arxiv.org/abs/2406.01775)\.
- Caccia et al\. \(2022\)Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni\.Multi\-Head Adapter Routing for Cross\-Task Generalization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.[https://arxiv\.org/abs/2211\.03831](https://arxiv.org/abs/2211.03831)\.
- Cai et al\. \(2020\)Han Cai, Chuang Gan, Ligeng Zhu, and Song Han\.TinyTL: Reduce Memory, Not Parameters for Efficient On\-Device Learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.[https://arxiv\.org/abs/2007\.11622](https://arxiv.org/abs/2007.11622)\.
- Capucci et al\. \(2021\)Matteo Capucci, Bruno Gavranović, Jules Hedges, and Eigil Fjeldgren Rischel\.Towards Foundations of Categorical Cybernetics\.In*Applied Category Theory \(ACT\)*, 2021\.[https://arxiv\.org/abs/2105\.06332](https://arxiv.org/abs/2105.06332)\.
- Castin et al\. \(2026\)Valérie Castin, Kimia Nadjahi, Pierre Ablin, and Gabriel Peyré\.Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2605\.31484](https://arxiv.org/abs/2605.31484)\.
- Cayley \(1846\)Arthur Cayley\.Sur quelques propriétés des déterminants gauches\.*Journal für die reine und angewandte Mathematik*, 32:119–123, 1846\.doi:10\.1515/crll\.1846\.32\.119\.
- Cervone \(2012\)Davide Cervone\.MathJax: A platform for mathematics on the web\.*Notices of the American Mathematical Society*, 59\(2\):312–316, 2012\.doi:10\.1090/noti794\.
- Chang et al\. \(2025\)Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi, Andrew Sabot, Hong Jiang, and H\. T\. Kung\.ElaLoRA: Elastic & Learnable Low\-Rank Adaptation for Efficient Model Fine\-Tuning\.*arXiv preprint arXiv:2504\.00254*, 2025\.[https://arxiv\.org/abs/2504\.00254](https://arxiv.org/abs/2504.00254)\.
- Charakorn et al\. \(2025\)Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, and Robert Tjarko Lange\.Text\-to\-LoRA: Instant Transformer Adaption\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025\.[https://arxiv\.org/abs/2506\.06105](https://arxiv.org/abs/2506.06105)\.
- Charakorn et al\. \(2026\)Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, and Robert Tjarko Lange\.Doc\-to\-LoRA: Learning to Instantly Internalize Contexts\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2602\.15902](https://arxiv.org/abs/2602.15902)\.
- Chavan et al\. \(2023\)Arnav Chavan, Zhuang Liu, Deepak Gupta, Eric Xing, and Zhiqiang Shen\.One\-for\-All: Generalized LoRA for Parameter\-Efficient Fine\-tuning\.*arXiv preprint arXiv:2306\.07967*, 2023\.[https://arxiv\.org/abs/2306\.07967](https://arxiv.org/abs/2306.07967)\.
- Chen et al\. \(1993\)An Mei Chen, Haw\-minn Lu, and Robert Hecht\-Nielsen\.On the geometry of feedforward neural network error surfaces\.*Neural Computation*, 5\(6\):910–927, 1993\.doi:10\.1162/neco\.1993\.5\.6\.910\.
- Chen et al\. \(2024a\)Aochuan Chen, Jiashun Cheng, Zijing Liu, Ziqi Gao, Fugee Tsung, Yu Li, and Jia Li\.Parameter\-Efficient Fine\-Tuning via Circular Convolution\.In*Findings of the Association for Computational Linguistics: ACL 2025*, 2024a\.[https://aclanthology\.org/2025\.findings\-acl\.102/](https://aclanthology.org/2025.findings-acl.102/)\.
- Chen et al\. \(2023a\)Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang\.Parameter\-Efficient Fine\-Tuning Design Spaces\.In*International Conference on Learning Representations \(ICLR\)*, 2023a\.[https://arxiv\.org/abs/2301\.01821](https://arxiv.org/abs/2301.01821)\.
- Chen et al\. \(2023b\)Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy\.Punica: Multi\-Tenant LoRA Serving\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2023b\.[https://arxiv\.org/abs/2310\.18547](https://arxiv.org/abs/2310.18547)\.
- Chen et al\. \(2024b\)Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, and Ping Luo\.EfficientQAT: Efficient Quantization\-Aware Training for Large Language Models\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024b\.[https://arxiv\.org/abs/2407\.11062](https://arxiv.org/abs/2407.11062)\.
- Chen et al\. \(2026a\)Nan Chen, Soledad Villar, and Soufiane Hayou\.Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026a\.[https://arxiv\.org/abs/2602\.06204](https://arxiv.org/abs/2602.06204)\.
- Chen et al\. \(2024c\)Shangyu Chen, Zizheng Pan, Jianfei Cai, and Dinh Phung\.PaRa: Personalizing Text\-to\-Image Diffusion via Parameter Rank Reduction\.In*International Conference on Learning Representations \(ICLR\)*, 2024c\.[https://arxiv\.org/abs/2406\.05641](https://arxiv.org/abs/2406.05641)\.
- Chen et al\. \(2022a\)Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo\.AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022a\.[https://arxiv\.org/abs/2205\.13535](https://arxiv.org/abs/2205.13535)\.
- Chen et al\. \(2024d\)Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, and Hao Cheng\.Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass\.In*International Conference on Learning Representations \(ICLR\)*, 2024d\.[https://arxiv\.org/abs/2411\.05877](https://arxiv.org/abs/2411.05877)\.
- Chen et al\. \(2024e\)Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang\.Fira: Can We Achieve Full\-rank Training of LLMs Under Low\-rank Constraint?In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024e\.[https://arxiv\.org/abs/2410\.01623](https://arxiv.org/abs/2410.01623)\.
- Chen et al\. \(2021\)Xuxi Chen, Tianlong Chen, Weizhu Chen, Ahmed Hassan Awadallah, Zhangyang Wang, and Yu Cheng\.DSEE: Dually Sparsity\-embedded Efficient Tuning of Pre\-trained Language Models\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://arxiv\.org/abs/2111\.00160](https://arxiv.org/abs/2111.00160)\.
- Chen et al\. \(2022b\)Yifan Chen, Devamanyu Hazarika, Mahdi Namazifar, Yang Liu, Di Jin, and Dilek Hakkani\-Tur\.Inducer\-tuning: Connecting Prefix\-tuning and Adapter\-tuning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2022b\.[https://aclanthology\.org/2022\.emnlp\-main\.50/](https://aclanthology.org/2022.emnlp-main.50/)\.
- Chen et al\. \(2023c\)Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia\.LongLoRA: Efficient Fine\-tuning of Long\-Context Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2023c\.[https://openreview\.net/forum?id=6PmJoRfdaK](https://openreview.net/forum?id=6PmJoRfdaK)\.
- Chen et al\. \(2025\)Zeming Chen, Angelika Romanou, Gail Weiss, and Antoine Bosselut\.PERK: Long\-Context Reasoning as Parameter\-Efficient Test\-Time Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2507\.06415](https://arxiv.org/abs/2507.06415)\.
- Chen et al\. \(2024f\)Zhuo Chen, Rumen Dangovski, Charlotte Loh, Owen Dugan, Di Luo, and Marin Soljačić\.QuanTA: Efficient High\-Rank Fine\-Tuning of LLMs with Quantum\-Informed Tensor Adaptation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024f\.[https://arxiv\.org/abs/2406\.00132](https://arxiv.org/abs/2406.00132)\.
- Chen et al\. \(2026b\)Ziye Chen, Hongbin Lin, Chenyu Zhang, Xiangda Yan, Yongjie Yang, and Yao Shu\.AR1\-ZO: Topology\-Aware Rank\-1 Zeroth\-Order Queries for High\-Rank LoRA Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026b\.[https://arxiv\.org/abs/2605\.19767](https://arxiv.org/abs/2605.19767)\.
- Chizat et al\. \(2019\)Lénaïc Chizat, Edouard Oyallon, and Francis Bach\.On lazy training in differentiable programming\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.[https://arxiv\.org/abs/1812\.07956](https://arxiv.org/abs/1812.07956)\.
- Cho et al\. \(2024\)Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi\.Heterogeneous LoRA for Federated Fine\-tuning of On\-Device Foundation Models\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2024\.[https://aclanthology\.org/2024\.emnlp\-main\.717/](https://aclanthology.org/2024.emnlp-main.717/)\.
- Cho et al\. \(2026\)Yoonjun Cho, Dongjae Jeon, Soeun Kim, Moongyu Jeon, and Albert No\.Preserve\-Then\-Quantize: Balancing Rank Budgets for Quantization Error Reconstruction in LLMs\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2602\.02001](https://arxiv.org/abs/2602.02001)\.
- Chronopoulou et al\. \(2023\)Alexandra Chronopoulou, Matthew E\. Peters, Alexander Fraser, and Jesse Dodge\.AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models\.In*Findings of the Association for Computational Linguistics: EACL 2023*, 2023\.[https://aclanthology\.org/2023\.findings\-eacl\.153/](https://aclanthology.org/2023.findings-eacl.153/)\.
- Cichocki et al\. \(2016\)A\. Cichocki, N\. Lee, I\. V\. Oseledets, A\. H\. Phan, Q\. Zhao, and D\. Mandic\.Low\-Rank Tensor Networks for Dimensionality Reduction and Large\-Scale Optimization Problems: Perspectives and Challenges PART 1\.*Foundations and Trends in Machine Learning*, 9\(4–5\):249–429, 2016\.[https://arxiv\.org/abs/1609\.00893](https://arxiv.org/abs/1609.00893)\.
- Cockett et al\. \(2019\)Robin Cockett, Geoffrey Cruttwell, Jonathan Gallagher, Jean\-Simon Pacaud Lemay, Benjamin MacAdam, Gordon Plotkin, and Dorette Pronk\.Reverse Derivative Categories\.In*CSL*, 2019\.[https://arxiv\.org/abs/1910\.07065](https://arxiv.org/abs/1910.07065)\.
- Coecke and Kissinger \(2017\)Bob Coecke and Aleks Kissinger\.*Picturing Quantum Processes: A First Course in Quantum Theory and Diagrammatic Reasoning*\.Cambridge University Press, Cambridge, 2017\.ISBN 978\-1\-107\-10422\-8\.doi:10\.1017/9781316219317\.
- Coecke et al\. \(2013\)Bob Coecke, Dusko Pavlovic, and Jamie Vicary\.A new description of orthogonal bases\.*Mathematical Structures in Computer Science*, 23\(3\):555–567, 2013\.doi:10\.1017/S0960129512000047\.
- Cohen and Welling \(2016\)Taco S\. Cohen and Max Welling\.Group equivariant convolutional networks\.In*Proceedings of the 33rd International Conference on Machine Learning*, volume 48 of*Proceedings of Machine Learning Research*\. PMLR, 2016\.[https://arxiv\.org/abs/1602\.07576](https://arxiv.org/abs/1602.07576)\.
- Cohen et al\. \(2019\)Taco S\. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling\.Gauge equivariant convolutional networks and the icosahedral CNN\.In*Proceedings of the 36th International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*\. PMLR, 2019\.[https://arxiv\.org/abs/1902\.04615](https://arxiv.org/abs/1902.04615)\.
- Cox et al\. \(2011\)David A\. Cox, John B\. Little, and Henry K\. Schenck\.*Toric Varieties*, volume 124 of*Graduate Studies in Mathematics*\.American Mathematical Society, Providence, RI, 2011\.doi:10\.1090/gsm/124\.
- Cruttwell et al\. \(2021\)G\. S\. H\. Cruttwell, Bruno Gavranović, Neil Ghani, Paul Wilson, and Fabio Zanasi\.Categorical Foundations of Gradient\-Based Learning\.In*European Symposium on Programming \(ESOP\)*, 2021\.[https://arxiv\.org/abs/2103\.01931](https://arxiv.org/abs/2103.01931)\.
- Cui et al\. \(2016\)Shawn X\. Cui, Michael H\. Freedman, Or Sattath, Richard Stong, and Greg Minton\.Quantum max\-flow/min\-cut\.*Journal of Mathematical Physics*, 57\(6\):062206, 2016\.doi:10\.1063/1\.4954231\.
- Dang et al\. \(2025\)Sizhe Dang, Yangyang Guo, Yanjun Zhao, Haishan Ye, Xiaodong Zheng, Guang Dai, and Ivor Tsang\.FZOO: Fast Zeroth\-Order Optimizer for Fine\-Tuning Large Language Models towards Adam\-Scale Speed\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2506\.09034](https://arxiv.org/abs/2506.09034)\.
- Dao et al\. \(2019\)Tri Dao, Albert Gu, Matthew Eichhorn, Atri Rudra, and Christopher Ré\.Learning Fast Algorithms for Linear Transforms Using Butterfly Factorizations\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2019\.[https://arxiv\.org/abs/1903\.05895](https://arxiv.org/abs/1903.05895)\.
- Dao et al\. \(2022\)Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré\.Monarch: Expressive Structured Matrices for Efficient and Accurate Training\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2022\.[https://arxiv\.org/abs/2204\.00595](https://arxiv.org/abs/2204.00595)\.
- Davey and Priestley \(2002\)B\. A\. Davey and H\. A\. Priestley\.*Introduction to Lattices and Order*\.Cambridge University Press, Cambridge, second edition, 2002\.ISBN 978\-0\-521\-78451\-1\.doi:10\.1017/CBO9780511809088\.
- Dettmers et al\. \(2021\)Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer\.8\-bit Optimizers via Block\-wise Quantization\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.[https://arxiv\.org/abs/2110\.02861](https://arxiv.org/abs/2110.02861)\.
- Dettmers et al\. \(2023\)Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer\.QLoRA: Efficient Finetuning of Quantized LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2305\.14314](https://arxiv.org/abs/2305.14314)\.
- Ding et al\. \(2022\)Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi\-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai\-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun\.Parameter\-efficient fine\-tuning of large\-scale pre\-trained language models\.*Nature Machine Intelligence*, 2022\.[https://www\.nature\.com/articles/s42256\-023\-00626\-4](https://www.nature.com/articles/s42256-023-00626-4)\.
- Ding et al\. \(2023\)Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun\.Sparse Low\-rank Adaptation of Pre\-trained Language Models\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.[https://arxiv\.org/abs/2311\.11696](https://arxiv.org/abs/2311.11696)\.
- Dinh et al\. \(2017\)Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio\.Sharp minima can generalize for deep nets\.In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*\. PMLR, 2017\.[https://arxiv\.org/abs/1703\.04933](https://arxiv.org/abs/1703.04933)\.
- Dokme and Heck \(2026\)Atahan Dokme and Larry Heck\.Selective State\-Space Adaptation and Retrieval for Language Model Reasoning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2026\.[https://arxiv\.org/abs/2607\.19326](https://arxiv.org/abs/2607.19326)\.
- Dominé et al\. \(2025\)Clémentine C\. J\. Dominé, Nicolas Anguita, Alexandra M\. Proca, Lukas Braun, Daniel Kunin, Pedro A\. M\. Mediano, and Andrew M\. Saxe\.From lazy to rich: Exact learning dynamics in deep linear networks\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2409\.14623](https://arxiv.org/abs/2409.14623)\.
- Donahue et al\. \(2013\)Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell\.DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2013\.[https://arxiv\.org/abs/1310\.1531](https://arxiv.org/abs/1310.1531)\.
- Dou et al\. \(2023\)Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang\.LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE\-Style Plugin\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2023\.[https://aclanthology\.org/2024\.acl\-long\.106/](https://aclanthology.org/2024.acl-long.106/)\.
- Du et al\. \(2018\)Simon S\. Du, Wei Hu, and Jason D\. Lee\.Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2018\.[https://arxiv\.org/abs/1806\.00900](https://arxiv.org/abs/1806.00900)\.
- Du et al\. \(2025\)Zhekai Du, Yinjie Min, Jingjing Li, Ke Lu, Changliang Zou, Liuhua Peng, Tingjin Chu, and Mingming Gong\.LoCA: Location\-Aware Cosine Adaptation for Parameter\-Efficient Fine\-Tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2502\.06820](https://arxiv.org/abs/2502.06820)\.
- Eckart and Young \(1936\)Carl Eckart and Gale Young\.The approximation of one matrix by another of lower rank\.*Psychometrika*, 1\(3\):211–218, 1936\.doi:10\.1007/BF02288367\.
- Edalati et al\. \(2022\)Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J\. Clark, and Mehdi Rezagholizadeh\.KronA: Parameter Efficient Tuning with Kronecker Adapter\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.[https://arxiv\.org/abs/2212\.10650](https://arxiv.org/abs/2212.10650)\.
- Elliott \(2018\)Conal Elliott\.The simple essence of automatic differentiation\.*Proceedings of the ACM on Programming Languages*, 2\(ICFP\):1–29, 2018\.doi:10\.1145/3236765\.
- Entezari et al\. \(2022\)Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur\.The role of permutation invariance in linear mode connectivity of neural networks\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2110\.06296](https://arxiv.org/abs/2110.06296)\.
- Eyuboglu et al\. \(2025\)Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher Re\.Cartridges: Lightweight and general\-purpose long context representations via self\-study\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2506\.06266](https://arxiv.org/abs/2506.06266)\.
- Fan et al\. \(2025\)Chenghao Fan, Zhenyi Lu, Sichen Liu, Chengfeng Gu, Xiaoye Qu, Wei Wei, and Yu Cheng\.Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture\-of\-Experts Optimization Alignment\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025\.[https://arxiv\.org/abs/2502\.16894](https://arxiv.org/abs/2502.16894)\.
- Fan \(1949\)Ky Fan\.On a theorem of Weyl concerning eigenvalues of linear transformations I\.*Proceedings of the National Academy of Sciences*, 35\(11\):652–655, 1949\.doi:10\.1073/pnas\.35\.11\.652\.
- Farajtabar et al\. \(2019\)Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li\.Orthogonal Gradient Descent for Continual Learning\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, 2019\.[https://proceedings\.mlr\.press/v108/farajtabar20a\.html](https://proceedings.mlr.press/v108/farajtabar20a.html)\.
- Farhadzadeh et al\. \(2025\)Farzad Farhadzadeh, Debasmit Das, Shubhankar Borse, and Fatih Porikli\.LoRA\-X: Bridging Foundation Models with Training\-Free Cross\-Model Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/1add3bbdbc20c403a383482a665eb5a4\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/1add3bbdbc20c403a383482a665eb5a4-Abstract-Conference.html)\.
- Feng et al\. \(2025\)Guhao Feng, Shengjie Luo, Kai Hua, Ge Zhang, Di He, Wenhao Huang, and Tianle Cai\.In\-Place Test\-Time Training\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2604\.06169](https://arxiv.org/abs/2604.06169)\.
- Fillioux et al\. \(2025\)Leo Fillioux, Enzo Ferrante, Paul\-Henry Cournède, Maria Vakalopoulou, and Stergios Christodoulidis\.PVeRA: Probabilistic Vector\-Based Random Matrix Adaptation\.In*IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*, 2025\.[https://arxiv\.org/abs/2512\.07703](https://arxiv.org/abs/2512.07703)\.
- Fong and Johnson \(2019\)Brendan Fong and Michael Johnson\.Lenses and Learners\.In*Bx 2019 \(International Workshop on Bidirectional Transformations\)*, 2019\.[https://arxiv\.org/abs/1903\.03671](https://arxiv.org/abs/1903.03671)\.
- Fong et al\. \(2017\)Brendan Fong, David I\. Spivak, and Rémy Tuyéras\.Backprop as Functor: A compositional perspective on supervised learning\.In*Symposium on Logic in Computer Science \(LICS\)*, 2017\.[https://arxiv\.org/abs/1711\.10455](https://arxiv.org/abs/1711.10455)\.
- Fox \(1976\)Thomas Fox\.Coalgebras and cartesian categories\.*Communications in Algebra*, 4\(7\):665–667, 1976\.doi:10\.1080/00927877608822127\.
- Frankle et al\. \(2019\)Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M\. Roy, and Michael Carbin\.Linear Mode Connectivity and the Lottery Ticket Hypothesis\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2019\.[https://arxiv\.org/abs/1912\.05671](https://arxiv.org/abs/1912.05671)\.
- Fu et al\. \(2022\)Zihao Fu, Haoran Yang, Anthony Man\-Cho So, Wai Lam, Lidong Bing, and Nigel Collier\.On the Effectiveness of Parameter\-Efficient Fine\-Tuning\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2022\.[https://arxiv\.org/abs/2211\.15583](https://arxiv.org/abs/2211.15583)\.
- Gal et al\. \(2022\)Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H\. Bermano, Gal Chechik, and Daniel Cohen\-Or\.An Image is Worth One Word: Personalizing Text\-to\-Image Generation using Textual Inversion\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2208\.01618](https://arxiv.org/abs/2208.01618)\.
- Galim et al\. \(2024\)Kevin Galim, Wonjun Kang, Yuchen Zeng, Hyung Il Koo, and Kangwook Lee\.Parameter\-Efficient Fine\-Tuning of State Space Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2410\.09016](https://arxiv.org/abs/2410.09016)\.
- Gandikota et al\. \(2023\)Rohit Gandikota, Joanna Materzynska, Tingrui Zhou, Antonio Torralba, and David Bau\.Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models\.In*European Conference on Computer Vision \(ECCV\)*, 2023\.[https://arxiv\.org/abs/2311\.12092](https://arxiv.org/abs/2311.12092)\.
- Gangwar et al\. \(2026\)Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky, Hancao Li, Vivek Mittal, Lexing Ying, and Nickvash Kani\.GiVA: Gradient\-Informed Bases for Vector\-Based Adaptation\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, 2026\.[https://arxiv\.org/abs/2604\.21901](https://arxiv.org/abs/2604.21901)\.
- Gao et al\. \(2021\)Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao\.CLIP\-Adapter: Better Vision\-Language Models with Feature Adapters\.*International Journal of Computer Vision*, 2021\.[https://arxiv\.org/abs/2110\.04544](https://arxiv.org/abs/2110.04544)\.
- Gao et al\. \(2023\)Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao\.LLaMA\-Adapter V2: Parameter\-Efficient Visual Instruction Model\.*arXiv preprint arXiv:2304\.15010*, 2023\.[https://arxiv\.org/abs/2304\.15010](https://arxiv.org/abs/2304.15010)\.
- Gao et al\. \(2024\)Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li\.Parameter\-Efficient Fine\-Tuning with Discrete Fourier Transform\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2405\.03003](https://arxiv.org/abs/2405.03003)\.
- Gargiulo et al\. \(2024\)Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà\.Task Singular Vectors: Reducing Task Interference in Model Merging\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024\.[https://arxiv\.org/abs/2412\.00081](https://arxiv.org/abs/2412.00081)\.
- Gavranović \(2024\)Bruno Gavranović\.*Fundamental Components of Deep Learning: A category\-theoretic approach*\.PhD thesis, University of Strathclyde, 2024\.[https://arxiv\.org/abs/2403\.13001](https://arxiv.org/abs/2403.13001)\.
- Gavranović et al\. \(2024\)Bruno Gavranović, Paul Lessard, Andrew Dudzik, Tamara von Glehn, João G\. M\. Araújo, and Petar Veličković\.Position: Categorical Deep Learning is an Algebraic Theory of All Architectures\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2402\.15332](https://arxiv.org/abs/2402.15332)\.
- Ge et al\. \(2017\)Rong Ge, Chi Jin, and Yi Zheng\.No spurious local minima in nonconvex low rank problems: A unified geometric analysis\.In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*\. PMLR, 2017\.[https://arxiv\.org/abs/1704\.00708](https://arxiv.org/abs/1704.00708)\.
- Gemma Team et al\. \(2024\)Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, et al\.Gemma 2: Improving open language models at a practical size\.arXiv:2408\.00118, 2024\.[https://arxiv\.org/abs/2408\.00118](https://arxiv.org/abs/2408.00118)\.
- Gemma Team et al\. \(2025\)Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, et al\.Gemma 3 technical report\.arXiv:2503\.19786, 2025\.[https://arxiv\.org/abs/2503\.19786](https://arxiv.org/abs/2503.19786)\.
- Goddard et al\. \(2024\)Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz\.Arcee’s MergeKit: A toolkit for merging large language models\.In Franck Dernoncourt, Daniel Preoţiuc\-Pietro, and Anastasia Shimorina, editors,*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track*, pages 477–485, Miami, Florida, US, November 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.emnlp\-industry\.36\.
- Godfrey et al\. \(2022\)Charles Godfrey, Davis Brown, Tegan Emerson, and Henry Kvinge\.On the symmetries of deep learning models and their internal representations\.In*Advances in Neural Information Processing Systems 35 \(NeurIPS 2022\)*, pages 11893–11905, 2022\.doi:10\.52202/068431\-0864\.
- Goodman and Wallach \(2009\)Roe Goodman and Nolan R\. Wallach\.*Symmetry, Representations, and Invariants*, volume 255 of*Graduate Texts in Mathematics*\.Springer, New York, 2009\.doi:10\.1007/978\-0\-387\-79852\-3\.
- Goto \(1969\)Morikuni Goto\.On an arcwise connected subgroup of a Lie group\.*Proceedings of the American Mathematical Society*, 20\(1\):157–162, 1969\.doi:10\.1090/S0002\-9939\-1969\-0233923\-X\.
- Gray \(2006\)Robert M\. Gray\.Toeplitz and circulant matrices: A review\.*Foundations and Trends in Communications and Information Theory*, 2\(3\):155–239, 2006\.doi:10\.1561/0100000006\.
- Greenewald et al\. \(2025\)Kristjan Greenewald, Luis Lastras, Thomas Parnell, Vraj Shah, Lucian Popa, Giulio Zizzo, Chulaka Gunasekara, Ambrish Rawat, and David Cox\.Activated LoRA: Fine\-tuned LLMs for Intrinsics\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2504\.12397](https://arxiv.org/abs/2504.12397)\.
- Griewank and Walther \(2008\)Andreas Griewank and Andrea Walther\.*Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation*\.Society for Industrial and Applied Mathematics, Philadelphia, PA, second edition, 2008\.doi:10\.1137/1\.9780898717761\.
- Grothendieck \(2022\)Alexandre Grothendieck\.*Récoltes et semailles: Réflexions et témoignage sur un passé de mathématicien*\.Number 437–438 in Collection Tel\. Gallimard, Paris, 2022\.ISBN 978\-2\-07\-288975\-2\.2 vols\. Written 1983–1986; the images of the nut and the rising sea are in note no\. 122, “La mer qui monte…”\.
- Gu et al\. \(2023\)Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou\.Mix\-of\-Show: Decentralized Low\-Rank Adaptation for Multi\-Concept Customization of Diffusion Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2305\.18292](https://arxiv.org/abs/2305.18292)\.
- Gu et al\. \(2021\)Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang\.PPT: Pre\-trained Prompt Tuning for Few\-shot Learning\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://aclanthology\.org/2022\.acl\-long\.576/](https://aclanthology.org/2022.acl-long.576/)\.
- Gu et al\. \(2025\)Yuxuan Gu, Wuyang Zhou, Giorgos Iacovides, and Danilo Mandic\.TeRA: Vector\-based Random Tensor Network for High\-Rank Adaptation of Large Language Models\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2025\.[https://arxiv\.org/abs/2509\.03234](https://arxiv.org/abs/2509.03234)\.
- Guillemin and Pollack \(1974\)Victor Guillemin and Alan Pollack\.*Differential Topology*\.Prentice\-Hall, Englewood Cliffs, NJ, 1974\.
- Gunasekar et al\. \(2017\)Suriya Gunasekar, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro\.Implicit regularization in matrix factorization\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.[https://arxiv\.org/abs/1705\.09280](https://arxiv.org/abs/1705.09280)\.
- Guo et al\. \(2020\)Demi Guo, Alexander M\. Rush, and Yoon Kim\.Parameter\-Efficient Transfer Learning with Diff Pruning\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2020\.[https://aclanthology\.org/2021\.acl\-long\.378/](https://aclanthology.org/2021.acl-long.378/)\.
- Guo et al\. \(2023\)Han Guo, Philip Greengard, Eric P\. Xing, and Yoon Kim\.LQ\-LoRA: Low\-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2311\.12023](https://arxiv.org/abs/2311.12023)\.
- Gur\-Ari et al\. \(2018\)Guy Gur\-Ari, Daniel A\. Roberts, and Ethan Dyer\.Gradient Descent Happens in a Tiny Subspace\.*arXiv preprint arXiv:1812\.04754*, 2018\.[https://arxiv\.org/abs/1812\.04754](https://arxiv.org/abs/1812.04754)\.
- Gurses et al\. \(2025\)Selcuk Gurses, Aozhong Zhang, Yanxia Deng, Xun Dong, Xin Li, Naigang Wang, Penghang Yin, and Zi Yang\.DiaBlo: Diagonal Blocks Are Sufficient For Finetuning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2506\.03230](https://arxiv.org/abs/2506.03230)\.
- Hale \(1980\)Jack K\. Hale\.*Ordinary Differential Equations*\.Robert E\. Krieger Publishing Company, Huntington, NY, second edition, 1980\.ISBN 0\-89874\-011\-8\.
- Hall \(2015\)Brian C\. Hall\.*Lie Groups, Lie Algebras, and Representations: An Elementary Introduction*, volume 222 of*Graduate Texts in Mathematics*\.Springer, Cham, second edition, 2015\.doi:10\.1007/978\-3\-319\-13467\-3\.
- Hambardzumyan et al\. \(2021\)Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May\.WARP: Word\-level Adversarial ReProgramming\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://aclanthology\.org/2021\.acl\-long\.381/](https://aclanthology.org/2021.acl-long.381/)\.
- Han et al\. \(2024a\)Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, and Bamdev Mishra\.SLTrain: a sparse plus low\-rank approach for parameter and memory efficient pretraining\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024a\.[https://arxiv\.org/abs/2406\.02214](https://arxiv.org/abs/2406.02214)\.
- Han et al\. \(2023\)Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang\.SVDiff: Compact Parameter Space for Diffusion Fine\-Tuning\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023\.[https://arxiv\.org/abs/2303\.11305](https://arxiv.org/abs/2303.11305)\.
- Han et al\. \(2024b\)Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang\.Parameter\-Efficient Fine\-Tuning for Large Models: A Comprehensive Survey\.*TMLR*, 2024b\.[https://arxiv\.org/abs/2403\.14608](https://arxiv.org/abs/2403.14608)\.
- Hao et al\. \(2024\)Yongchang Hao, Yanshuai Cao, and Lili Mou\.Flora: Low\-Rank Adapters Are Secretly Gradient Compressors\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2402\.03293](https://arxiv.org/abs/2402.03293)\.
- Harris et al\. \(2020\)Charles R\. Harris, K\. Jarrod Millman, Stéfan J\. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J\. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H\. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard\-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E\. Oliphant\.Array programming with NumPy\.*Nature*, 585\(7825\):357–362, 2020\.doi:10\.1038/s41586\-020\-2649\-2\.
- Harris \(1992\)Joe Harris\.*Algebraic Geometry: A First Course*, volume 133 of*Graduate Texts in Mathematics*\.Springer, New York, 1992\.doi:10\.1007/978\-1\-4757\-2189\-8\.
- Hartford et al\. \(2024\)Eric Hartford, Lucas Atkins, Fernando Fernandes Neto, and David Golchinfar\.Spectrum: Targeted Training on Signal to Noise Ratio\.*arXiv preprint arXiv:2406\.06623*, 2024\.[https://arxiv\.org/abs/2406\.06623](https://arxiv.org/abs/2406.06623)\.
- Hayou et al\. \(2024a\)Soufiane Hayou, Nikhil Ghosh, and Bin Yu\.The Impact of Initialization on LoRA Finetuning Dynamics\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024a\.[https://arxiv\.org/abs/2406\.08447](https://arxiv.org/abs/2406.08447)\.
- Hayou et al\. \(2024b\)Soufiane Hayou, Nikhil Ghosh, and Bin Yu\.LoRA\+: Efficient Low Rank Adaptation of Large Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024b\.[https://arxiv\.org/abs/2402\.12354](https://arxiv.org/abs/2402.12354)\.
- Hayou et al\. \(2025\)Soufiane Hayou, Nikhil Ghosh, and Bin Yu\.PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2506\.20629](https://arxiv.org/abs/2506.20629)\.
- He et al\. \(2025a\)Haonan He, Peng Ye, Yuchen Ren, Yuan Yuan, Luyang Zhou, Shucun Ju, and Lei Chen\.GoRA: Gradient\-driven Adaptive Low Rank Adaptation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025a\.[https://arxiv\.org/abs/2502\.12171](https://arxiv.org/abs/2502.12171)\.
- He et al\. \(2026a\)Haonan He, Jingqi Ye, Minglei Li, Zhengbo Wang, Tao Chen, Lei Bai, and Peng Ye\.A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation\.*arXiv preprint arXiv:2601\.22708*, 2026a\.[https://arxiv\.org/abs/2601\.22708](https://arxiv.org/abs/2601.22708)\.
- He et al\. \(2023\)Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang\.Sensitivity\-Aware Visual Parameter\-Efficient Fine\-Tuning\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023\.[https://arxiv\.org/abs/2303\.08566](https://arxiv.org/abs/2303.08566)\.
- He et al\. \(2024a\)Haoze He, Juncheng Billy Li, Xuan Jiang, and Heather Miller\.SMT: Fine\-Tuning Large Language Models with Sparse Matrices\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.[https://openreview\.net/forum?id=GbgCRJedQ7](https://openreview.net/forum?id=GbgCRJedQ7)\.
- He et al\. \(2021\)Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg\-Kirkpatrick, and Graham Neubig\.Towards a Unified View of Parameter\-Efficient Transfer Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.[https://arxiv\.org/abs/2110\.04366](https://arxiv.org/abs/2110.04366)\.
- He et al\. \(2026b\)Lingfeng He, De Cheng, Huaijie Wang, Xi Yang, Nannan Wang, and Xinbo Gao\.Task\-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA\-based Continual Learning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026b\.[https://arxiv\.org/abs/2603\.00191](https://arxiv.org/abs/2603.00191)\.
- He et al\. \(2022\)Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang\.Parameter\-efficient Model Adaptation for Vision Transformers\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2022\.[https://arxiv\.org/abs/2203\.16329](https://arxiv.org/abs/2203.16329)\.
- He et al\. \(2024b\)Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan\.Subspace Optimization for Large Language Models with Convergence Guarantees\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024b\.[https://arxiv\.org/abs/2410\.11289](https://arxiv.org/abs/2410.11289)\.
- He et al\. \(2026c\)Zhengbao He, Ruiqi Ding, Zhehao Huang, Ruikai Yang, Tao Li, and Xiaolin Huang\.Compress then Merge: From Multiple LoRAs into One Low\-Rank Adapter\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026c\.[https://arxiv\.org/abs/2606\.03723](https://arxiv.org/abs/2606.03723)\.
- He et al\. \(2025b\)Zhiwei He, Zhaopeng Tu, Xing Wang, Xingyu Chen, Zhijie Wang, Jiahao Xu, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, and Rui Wang\.RaSA: Rank\-Sharing Low\-Rank Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2025b\.[https://arxiv\.org/abs/2503\.12576](https://arxiv.org/abs/2503.12576)\.
- Higham \(2002\)Nicholas J\. Higham\.*Accuracy and Stability of Numerical Algorithms*\.Society for Industrial and Applied Mathematics \(SIAM\), Philadelphia, PA, second edition, 2002\.ISBN 978\-0\-89871\-521\-7\.doi:10\.1137/1\.9780898718027\.
- Holtz et al\. \(2012\)Sebastian Holtz, Thorsten Rohwedder, and Reinhold Schneider\.On manifolds of tensors of fixed TT\-rank\.*Numerische Mathematik*, 120\(4\):701–731, 2012\.doi:10\.1007/s00211\-011\-0419\-7\.
- Horn and Johnson \(1991\)Roger A\. Horn and Charles R\. Johnson\.*Topics in Matrix Analysis*\.Cambridge University Press, Cambridge, 1991\.ISBN 978\-0\-521\-30587\-7\.doi:10\.1017/CBO9780511840371\.
- Horn and Johnson \(2013\)Roger A\. Horn and Charles R\. Johnson\.*Matrix Analysis*\.Cambridge University Press, Cambridge, second edition, 2013\.doi:10\.1017/CBO9781139020411\.
- Houlsby et al\. \(2019\)Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly\.Parameter\-Efficient Transfer Learning for NLP\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2019\.[https://arxiv\.org/abs/1902\.00751](https://arxiv.org/abs/1902.00751)\.
- Hounie et al\. \(2024\)Ignacio Hounie, Charilaos Kanatsoulis, Arnuv Tandon, and Alejandro Ribeiro\.LoRTA: Low Rank Tensor Adaptation of Large Language Models\.*arXiv preprint arXiv:2410\.04060*, 2024\.[https://arxiv\.org/abs/2410\.04060](https://arxiv.org/abs/2410.04060)\.
- Hu et al\. \(2021\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-Rank Adaptation of Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.[https://arxiv\.org/abs/2106\.09685](https://arxiv.org/abs/2106.09685)\.
- Hu et al\. \(2023\)Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee\-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka\-Wei Lee\.LLM\-Adapters: An Adapter Family for Parameter\-Efficient Fine\-Tuning of Large Language Models\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.[https://arxiv\.org/abs/2304\.01933](https://arxiv.org/abs/2304.01933)\.
- Huang et al\. \(2025a\)Baichuan Huang, Ananth Balashankar, and Amir Aminifar\.BEFT: Bias\-Efficient Fine\-Tuning of Language Models in Low\-Data Regimes\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2025a\.[https://aclanthology\.org/2026\.acl\-long\.1799/](https://aclanthology.org/2026.acl-long.1799/)\.
- Huang et al\. \(2023\)Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin\.LoraHub: Efficient Cross\-Task Generalization via Dynamic LoRA Composition\.In*Conference on Language Modeling \(COLM\)*, 2023\.[https://arxiv\.org/abs/2307\.13269](https://arxiv.org/abs/2307.13269)\.
- Huang et al\. \(2024\)Lianghua Huang, Wei Wang, Zhi\-Fan Wu, Yupeng Shi, Huanzhang Dou, Chen Liang, Yutong Feng, Yu Liu, and Jingren Zhou\.In\-Context LoRA for Diffusion Transformers\.*arXiv preprint arXiv:2410\.23775*, 2024\.[https://arxiv\.org/abs/2410\.23775](https://arxiv.org/abs/2410.23775)\.
- Huang et al\. \(2025b\)Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang\.HiRA: Parameter\-efficient Hadamard high\-rank adaptation for large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2025b\.[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/48c368f105e8145b945227b73255635a\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/48c368f105e8145b945227b73255635a-Abstract-Conference.html)\.
- Huang et al\. \(2025c\)Wei Huang, Yi Ge, Shuai Yang, Yicheng Xiao, Huizi Mao, Yujun Lin, Hanrong Ye, Sifei Liu, Ka Chun Cheung, Hongxu Yin, Yao Lu, Xiaojuan Qi, Song Han, and Yukang Chen\.QeRL: Beyond Efficiency – Quantization\-enhanced Reinforcement Learning for LLMs\.In*International Conference on Learning Representations \(ICLR\)*, 2025c\.[https://arxiv\.org/abs/2510\.11696](https://arxiv.org/abs/2510.11696)\.
- Huang et al\. \(2025d\)Zhehao Huang, Yuhang Liu, Baijiong Lin, Yixin Lou, Zhengbao He, Hanling Tian, Tao Li, and Xiaolin Huang\.RAIN\-Merging: A Gradient\-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format\.In*International Conference on Learning Representations \(ICLR\)*, 2025d\.[https://arxiv\.org/abs/2602\.22538](https://arxiv.org/abs/2602.22538)\.
- Hunter \(2007\)John D\. Hunter\.Matplotlib: A 2D graphics environment\.*Computing in Science & Engineering*, 9\(3\):90–95, 2007\.doi:10\.1109/MCSE\.2007\.55\.
- Hyeon\-Woo et al\. \(2021\)Nam Hyeon\-Woo, Moon Ye\-Bin, and Tae\-Hyun Oh\.FedPara: Low\-Rank Hadamard Product for Communication\-Efficient Federated Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.[https://arxiv\.org/abs/2108\.06098](https://arxiv.org/abs/2108.06098)\.
- Ilharco et al\. \(2022\)Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi\.Editing Models with Task Arithmetic\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2212\.04089](https://arxiv.org/abs/2212.04089)\.
- Ilin et al\. \(2026\)Ivan Ilin, Philip Zmushko, and Peter Richtárik\.Super\-Tuning: From Activation\-Aware Pruning to Sparse Fine\-Tuning\.*arXiv preprint arXiv:2607\.09287*, 2026\.[https://arxiv\.org/abs/2607\.09287](https://arxiv.org/abs/2607.09287)\.
- Jacot et al\. \(2018\)Arthur Jacot, Franck Gabriel, and Clément Hongler\.Neural tangent kernel: Convergence and generalization in neural networks\.In*Advances in Neural Information Processing Systems*, volume 31, 2018\.[https://arxiv\.org/abs/1806\.07572](https://arxiv.org/abs/1806.07572)\.
- Jang et al\. \(2024\)Uijeong Jang, Jason D\. Lee, and Ernest K\. Ryu\.LoRA Training in the NTK Regime has No Spurious Local Minima\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2402\.11867](https://arxiv.org/abs/2402.11867)\.
- Jeon et al\. \(2024\)Hyesung Jeon, Yulhwa Kim, and Jae joon Kim\.L4Q: Parameter Efficient Quantization\-Aware Fine\-Tuning on Large Language Models\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024\.[https://aclanthology\.org/2025\.acl\-long\.99/](https://aclanthology.org/2025.acl-long.99/)\.
- Jeon et al\. \(2026\)Hyesung Jeon, Hyeongju Ha, and Jae\-Joon Kim\.LRAgent: Efficient KV Cache Sharing for Multi\-LoRA LLM Agents\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2602\.01053](https://arxiv.org/abs/2602.01053)\.
- Ji et al\. \(2024\)Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, and Simon Lucey\.Efficient Learning With Sine\-Activated Low\-rank Matrices\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2403\.19243](https://arxiv.org/abs/2403.19243)\.
- Jia et al\. \(2022\)Menglin Jia, Luming Tang, Bor\-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser\-Nam Lim\.Visual Prompt Tuning\.In*European Conference on Computer Vision \(ECCV\)*, 2022\.[https://arxiv\.org/abs/2203\.12119](https://arxiv.org/abs/2203.12119)\.
- Jiang et al\. \(2026\)Juyong Jiang, Fan Wang, Hong Qi, Sunghun Kim, and Jing Tang\.FourierMoE: Fourier Mixture\-of\-Experts Adaptation of Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.[https://arxiv\.org/abs/2604\.01762](https://arxiv.org/abs/2604.01762)\.
- Jiang et al\. \(2024\)Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang\.MoRA: High\-Rank Updating for Parameter\-Efficient Fine\-Tuning\.*arXiv preprint arXiv:2405\.12130*, 2024\.[https://arxiv\.org/abs/2405\.12130](https://arxiv.org/abs/2405.12130)\.
- Jiang et al\. \(2023\)Zeyinzi Jiang, Chaojie Mao, Ziyuan Huang, Ao Ma, Yiliang Lv, Yujun Shen, Deli Zhao, and Jingren Zhou\.Res\-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2310\.19859](https://arxiv.org/abs/2310.19859)\.
- Jie and Deng \(2022a\)Shibo Jie and Zhi\-Hong Deng\.Convolutional Bypasses Are Better Vision Transformer Adapters\.In*ECAI*, 2022a\.[https://arxiv\.org/abs/2207\.07039](https://arxiv.org/abs/2207.07039)\.
- Jie and Deng \(2022b\)Shibo Jie and Zhi\-Hong Deng\.FacT: Factor\-Tuning for Lightweight Adaptation on Vision Transformer\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2022b\.[https://arxiv\.org/abs/2212\.03145](https://arxiv.org/abs/2212.03145)\.
- Jin et al\. \(2022\)Xisen Jin, Xiang Ren, Daniel Preotiuc\-Pietro, and Pengxiang Cheng\.Dataless Knowledge Fusion by Merging Weights of Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2212\.09849](https://arxiv.org/abs/2212.09849)\.
- Jordan et al\. \(2024\)Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein\.Muon: An optimizer for hidden layers in neural networks\.Blog post, 8 December 2024, 2024\.[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)\.
- Joyal and Street \(1991\)André Joyal and Ross Street\.The geometry of tensor calculus, I\.*Advances in Mathematics*, 88\(1\):55–112, 1991\.doi:10\.1016/0001\-8708\(91\)90003\-P\.
- Jung et al\. \(2025\)Yeonjoon Jung, Daehyun Ahn, Hyungjun Kim, Taesu Kim, and Eunhyeok Park\.GraLoRA: Granular Low\-Rank Adaptation for Parameter\-Efficient Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2505\.20355](https://arxiv.org/abs/2505.20355)\.
- Kalajdzievski \(2023\)Damjan Kalajdzievski\.A Rank Stabilization Scaling Factor for Fine\-Tuning with LoRA\.*arXiv preprint arXiv:2312\.03732*, 2023\.[https://arxiv\.org/abs/2312\.03732](https://arxiv.org/abs/2312.03732)\.
- Kang et al\. \(2026\)Changmin Kang, Jihun Yun, Baekrok Shin, Yeseul Cho, and Chulhee Yun\.Uniform Spectral Growth and Convergence of Muon in LoRA\-Style Matrix Factorization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.[https://arxiv\.org/abs/2602\.06385](https://arxiv.org/abs/2602.06385)\.
- Kang and Yin \(2024\)Jiale Kang and Qingyu Yin\.MiSS: Revisiting the Trade\-off in LoRA with an Efficient Shard\-Sharing Structure\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2409\.15371](https://arxiv.org/abs/2409.15371)\.
- Kang et al\. \(2025\)Wonjun Kang, Kevin Galim, Yuchen Zeng, Minjae Lee, Hyung Il Koo, and Nam Ik Cho\.State\-offset Tuning: State\-based Parameter\-Efficient Fine\-Tuning for State Space Models\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2025\.[https://arxiv\.org/abs/2503\.03499](https://arxiv.org/abs/2503.03499)\.
- Kato \(1995\)Tosio Kato\.*Perturbation Theory for Linear Operators*\.Classics in Mathematics\. Springer, Berlin, Heidelberg, second edition, 1995\.ISBN 978\-3\-540\-58661\-6\.doi:10\.1007/978\-3\-642\-66282\-9\.Reprint of the 1980 edition\.
- Kempf and Ness \(1979\)George Kempf and Linda Ness\.The length of vectors in representation spaces\.In Knud Lønsted, editor,*Algebraic Geometry \(Proceedings, Copenhagen 1978\)*, volume 732 of*Lecture Notes in Mathematics*, pages 233–243\. Springer, Berlin, Heidelberg, 1979\.doi:10\.1007/BFb0066647\.
- Khaki et al\. \(2025\)Samir Khaki, Xiuyu Li, Junxian Guo, Ligeng Zhu, Chenfeng Xu, Konstantinos N\. Plataniotis, Amir Yazdanbakhsh, Kurt Keutzer, Song Han, and Zhijian Liu\.SparseLoRA: Accelerating LLM Fine\-Tuning with Contextual Sparsity\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025\.[https://arxiv\.org/abs/2506\.16500](https://arxiv.org/abs/2506.16500)\.
- Khashabi et al\. \(2021\)Daniel Khashabi, Shane Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi\.Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2021\.[https://aclanthology\.org/2022\.naacl\-main\.266/](https://aclanthology.org/2022.naacl-main.266/)\.
- Khattak et al\. \(2022\)Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan\.MaPLe: Multi\-modal Prompt Learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2022\.[https://arxiv\.org/abs/2210\.03117](https://arxiv.org/abs/2210.03117)\.
- Kim et al\. \(2023a\)Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim\.SOLAR 10\.7B: Scaling Large Language Models with Simple yet Effective Depth Up\-Scaling\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2023a\.[https://arxiv\.org/abs/2312\.15166](https://arxiv.org/abs/2312.15166)\.
- Kim et al\. \(2023b\)Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee\.Memory\-Efficient Fine\-Tuning of Compressed Large Language Models via sub\-4\-bit Integer Quantization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023b\.[https://arxiv\.org/abs/2305\.14152](https://arxiv.org/abs/2305.14152)\.
- Kim et al\. \(2025\)Junsu Kim, Jaeyeon Kim, and Ernest K\. Ryu\.LoRA Training Provably Converges to a Low\-Rank Global Minimum or It Fails Loudly \(But it Probably Won’t Fail\)\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025\.[https://arxiv\.org/abs/2502\.09376](https://arxiv.org/abs/2502.09376)\.
- Kingma and Ba \(2015\)Diederik P\. Kingma and Jimmy Ba\.Adam: A method for stochastic optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2015\.[https://arxiv\.org/abs/1412\.6980](https://arxiv.org/abs/1412.6980)\.
- Klein \(1872\)Felix Klein\.*Vergleichende Betrachtungen über neuere geometrische Forschungen*\.Verlag von Andreas Deichert, Erlangen, 1872\.Reprinted in Mathematische Annalen 43 \(1893\), 63–100,[https://doi\.org/10\.1007/BF01446615](https://doi.org/10.1007/BF01446615)\.
- Knapp \(2002\)Anthony W\. Knapp\.*Lie Groups Beyond an Introduction*, volume 140 of*Progress in Mathematics*\.Birkhäuser, Boston, MA, second edition, 2002\.
- Kobayashi \(1958\)Shoshichi Kobayashi\.Fixed points of isometries\.*Nagoya Mathematical Journal*, 13:63–68, 1958\.doi:10\.1017/S0027763000023497\.
- Koch and Lubich \(2010\)Othmar Koch and Christian Lubich\.Dynamical tensor approximation\.*SIAM Journal on Matrix Analysis and Applications*, 31\(5\):2360–2375, 2010\.doi:10\.1137/09076578X\.
- Koike\-Akino et al\. \(2025\)Toshiaki Koike\-Akino, Francesco Tonin, Yongtao Wu, Frank Zhengqing Wu, Leyla Naz Candogan, and Volkan Cevher\.Quantum\-PEFT: Ultra parameter\-efficient fine\-tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2503\.05431](https://arxiv.org/abs/2503.05431)\.
- Kolda and Bader \(2009\)Tamara G\. Kolda and Brett W\. Bader\.Tensor decompositions and applications\.*SIAM Review*, 51\(3\):455–500, 2009\.doi:10\.1137/07070111X\.
- Koohpayegani et al\. \(2023\)Soroush Abbasi Koohpayegani, KL Navaneet, Parsa Nooralinejad, Soheil Kolouri, and Hamed Pirsiavash\.NOLA: Compressing LoRA using Linear Combination of Random Basis\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://openreview\.net/forum?id=TjfXcDgvzk](https://openreview.net/forum?id=TjfXcDgvzk)\.
- Kopiczko et al\. \(2023\)Dawid J\. Kopiczko, Tijmen Blankevoort, and Yuki M\. Asano\.VeRA: Vector\-based Random Matrix Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2310\.11454](https://arxiv.org/abs/2310.11454)\.
- Kowsher et al\. \(2026\)Md Kowsher, Haris Mansoor, Nusrat Jahan Prottasha, Ozlem Garibay, Victor Zhu, Zhengping Ji, and Chen Chen\.LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi\-task Learning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2604\.02338](https://arxiv.org/abs/2604.02338)\.
- Krantz and Parks \(2002\)Steven G\. Krantz and Harold R\. Parks\.*A Primer of Real Analytic Functions*\.Birkhäuser Advanced Texts\. Birkhäuser, Boston, MA, second edition, 2002\.doi:10\.1007/978\-0\-8176\-8134\-0\.
- Kumar et al\. \(2022\)Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang\.Fine\-Tuning can Distort Pretrained Features and Underperform Out\-of\-Distribution\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2202\.10054](https://arxiv.org/abs/2202.10054)\.
- Kumar et al\. \(2025\)Komal Kumar, Rao Muhammad Anwer, Fahad Shahbaz Khan, Salman Khan, Ivan Laptev, and Hisham Cholakkal\.DEFT: Decompositional Efficient Fine\-Tuning for Text\-to\-Image Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2509\.22793](https://arxiv.org/abs/2509.22793)\.
- Kumari et al\. \(2022\)Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun\-Yan Zhu\.Multi\-Concept Customization of Text\-to\-Image Diffusion\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2022\.[https://arxiv\.org/abs/2212\.04488](https://arxiv.org/abs/2212.04488)\.
- Kunin et al\. \(2020\)Daniel Kunin, Javier Sagastuy\-Brena, Surya Ganguli, Daniel L\. K\. Yamins, and Hidenori Tanaka\.Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics\.In*International Conference on Learning Representations \(ICLR\)*, 2020\.[https://arxiv\.org/abs/2012\.04728](https://arxiv.org/abs/2012.04728)\.
- Kunin et al\. \(2024\)Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli\.Get rich quick: Exact solutions reveal how unbalanced initializations promote rapid feature learning\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.[https://arxiv\.org/abs/2406\.06158](https://arxiv.org/abs/2406.06158)\.
- Lang et al\. \(2026\)Yicheng Lang, Changsheng Wang, Yihua Zhang, Mingyi Hong, Zheng Zhang, Wotao Yin, and Sijia Liu\.Powering Up Zeroth\-Order Training via Subspace Gradient Orthogonalization\.*arXiv preprint arXiv:2602\.17155*, 2026\.[https://arxiv\.org/abs/2602\.17155](https://arxiv.org/abs/2602.17155)\.
- Lanpouthakoun et al\. \(2026\)Andrew Lanpouthakoun, Aryaman Arora, Zhengxuan Wu, Dhruv Pai, Ben Keigwin, Dan Jurafsky, and Christopher Potts\.PreFT: Prefill\-only finetuning for efficient inference\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.[https://arxiv\.org/abs/2605\.14217](https://arxiv.org/abs/2605.14217)\.
- Le et al\. \(2024\)Minh Le, Chau Nguyen, Huy Nguyen, Quyen Tran, Trung Le, and Nhat Ho\.Revisiting Prefix\-tuning: Statistical Benefits of Reparameterization among Prompts\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2410\.02200](https://arxiv.org/abs/2410.02200)\.
- Le et al\. \(2025\)Minh Le, Anh Nguyen, Huy Nguyen, Chau Nguyen, Anh Tran, and Nhat Ho\.Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2501\.18936](https://arxiv.org/abs/2501.18936)\.
- Lee et al\. \(2025\)Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong\.AdaRank: Adaptive Rank Pruning for Enhanced Model Merging\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2503\.22178](https://arxiv.org/abs/2503.22178)\.
- Lee \(2013\)John M\. Lee\.*Introduction to Smooth Manifolds*, volume 218 of*Graduate Texts in Mathematics*\.Springer, New York, second edition, 2013\.doi:10\.1007/978\-1\-4419\-9982\-5\.
- Lee \(2018\)John M\. Lee\.*Introduction to Riemannian Manifolds*, volume 176 of*Graduate Texts in Mathematics*\.Springer, Cham, second edition, 2018\.doi:10\.1007/978\-3\-319\-91755\-9\.
- Lee et al\. \(2022\)Yoonho Lee, Annie S\. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Percy Liang, and Chelsea Finn\.Surgical Fine\-Tuning Improves Adaptation to Distribution Shifts\.In*International Conference on Learning Representations \(ICLR\)*, 2022\.[https://arxiv\.org/abs/2210\.11466](https://arxiv.org/abs/2210.11466)\.
- Lee et al\. \(2026\)Yu\-Ang Lee, Ching\-Yun Ko, Pin\-Yu Chen, and Mi\-Yen Yeh\.Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine\-tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.[https://arxiv\.org/abs/2602\.04998](https://arxiv.org/abs/2602.04998)\.
- Leinster \(1998\)Tom Leinster\.Basic bicategories, 1998\.[https://arxiv\.org/abs/math/9810017](https://arxiv.org/abs/math/9810017)\.
- Leinster \(2014\)Tom Leinster\.*Basic Category Theory*, volume 143 of*Cambridge Studies in Advanced Mathematics*\.Cambridge University Press, Cambridge, 2014\.ISBN 978\-1\-107\-04424\-1\.doi:10\.1017/CBO9781107360068\.
- Lester et al\. \(2021\)Brian Lester, Rami Al\-Rfou, and Noah Constant\.The Power of Scale for Parameter\-Efficient Prompt Tuning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2021\.[https://arxiv\.org/abs/2104\.08691](https://arxiv.org/abs/2104.08691)\.
- Lezcano\-Casado and Martínez\-Rubio \(2019\)Mario Lezcano\-Casado and David Martínez\-Rubio\.Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2019\.[https://arxiv\.org/abs/1901\.08428](https://arxiv.org/abs/1901.08428)\.
- Li et al\. \(2024a\)Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He\.On the Crucial Role of Initialization for Matrix Factorization\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.[https://arxiv\.org/abs/2410\.18965](https://arxiv.org/abs/2410.18965)\.
- Li et al\. \(2018\)Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski\.Measuring the Intrinsic Dimension of Objective Landscapes\.In*International Conference on Learning Representations \(ICLR\)*, 2018\.[https://arxiv\.org/abs/1804\.08838](https://arxiv.org/abs/1804.08838)\.
- Li et al\. \(2024b\)Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang\.MixLoRA: Enhancing Large Language Models Fine\-Tuning with LoRA\-based Mixture of Experts\.*arXiv preprint arXiv:2404\.15159*, 2024b\.[https://arxiv\.org/abs/2404\.15159](https://arxiv.org/abs/2404.15159)\.
- Li et al\. \(2025a\)Kaiyang Li, Shaobo Han, Qing Su, Wei Li, Zhipeng Cai, and Shihao Ji\.Uni\-LoRA: One Vector is All You Need\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025a\.[https://arxiv\.org/abs/2506\.00799](https://arxiv.org/abs/2506.00799)\.
- Li et al\. \(2023a\)Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg\.Inference\-Time Intervention: Eliciting Truthful Answers from a Language Model\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023a\.[https://arxiv\.org/abs/2306\.03341](https://arxiv.org/abs/2306.03341)\.
- Li et al\. \(2024c\)Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang\.Flat\-LoRA: Low\-Rank Adaptation over a Flat Loss Landscape\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024c\.[https://arxiv\.org/abs/2409\.14396](https://arxiv.org/abs/2409.14396)\.
- Li et al\. \(2025b\)Wenbing Li, Zikai Song, Hang Zhou, Yunyao Zhang, Junqing Yu, and Wei Yang\.LoRA\-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing\.In*International Conference on Learning Representations \(ICLR\)*, 2025b\.[https://arxiv\.org/abs/2507\.00029](https://arxiv.org/abs/2507.00029)\.
- Li and Liang \(2021\)Xiang Lisa Li and Percy Liang\.Prefix\-Tuning: Optimizing Continuous Prompts for Generation\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://aclanthology\.org/2021\.acl\-long\.353/](https://aclanthology.org/2021.acl-long.353/)\.
- Li et al\. \(2026a\)Xianming Li, Zongxi Li, Tsz fung Andrew Lee, Jing Li, Haoran Xie, and Qing Li\.ShadowPEFT: Shadow Network for Parameter\-Efficient Fine\-Tuning\.*arXiv preprint arXiv:2604\.19254*, 2026a\.[https://arxiv\.org/abs/2604\.19254](https://arxiv.org/abs/2604.19254)\.
- Li et al\. \(2024d\)Yang Li, Shaobo Han, and Shihao Ji\.VB\-LoRA: Extreme Parameter Efficient Fine\-Tuning with Vector Banks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024d\.[https://arxiv\.org/abs/2405\.15179](https://arxiv.org/abs/2405.15179)\.
- Li et al\. \(2023b\)Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao\.LoftQ: LoRA\-Fine\-Tuning\-Aware Quantization for Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2023b\.[https://arxiv\.org/abs/2310\.08659](https://arxiv.org/abs/2310.08659)\.
- Li et al\. \(2026b\)Zhe Li, Bicheng Ying, Zidong Liu, and Haibo Yang\.Learning Dynamics of Zeroth\-Order Optimization: A Kernel Perspective\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026b\.[https://arxiv\.org/abs/2605\.03373](https://arxiv.org/abs/2605.03373)\.
- Lialin et al\. \(2023a\)Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, and Anna Rumshisky\.Scaling Down to Scale Up: A Guide to Parameter\-Efficient Fine\-Tuning\.*arXiv preprint arXiv:2303\.15647*, 2023a\.[https://arxiv\.org/abs/2303\.15647](https://arxiv.org/abs/2303.15647)\.
- Lialin et al\. \(2023b\)Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky\.ReLoRA: High\-Rank Training Through Low\-Rank Updates\.In*International Conference on Learning Representations \(ICLR\)*, 2023b\.[https://arxiv\.org/abs/2307\.05695](https://arxiv.org/abs/2307.05695)\.
- Lian et al\. \(2022\)Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang\.Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.[https://arxiv\.org/abs/2210\.08823](https://arxiv.org/abs/2210.08823)\.
- Liang et al\. \(2024\)Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu\.Memory\-Efficient LLM Training with Online Subspace Descent\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2408\.12857](https://arxiv.org/abs/2408.12857)\.
- Liang and Li \(2024\)Yan\-Shuo Liang and Wu\-Jun Li\.InfLoRA: Interference\-Free Low\-Rank Adaptation for Continual Learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024\.[https://arxiv\.org/abs/2404\.00228](https://arxiv.org/abs/2404.00228)\.
- Liang et al\. \(2025\)Zhiyuan Liang, Dongwen Tang, Yuhao Zhou, Xuanlei Zhao, Mingjia Shi, Wangbo Zhao, Zekai Li, Peihao Wang, Konstantin Schürholt, Damian Borth, Michael M\. Bronstein, Yang You, Zhangyang Wang, and Kai Wang\.Drag\-and\-Drop LLMs: Zero\-Shot Prompt\-to\-Weights\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2506\.16406](https://arxiv.org/abs/2506.16406)\.
- Liao and Monz \(2024\)Baohao Liao and Christof Monz\.3\-in\-1: 2D Rotary Adaptation for Efficient Finetuning, Efficient Batching and Composability\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2409\.00119](https://arxiv.org/abs/2409.00119)\.
- Liao et al\. \(2023a\)Baohao Liao, Yan Meng, and Christof Monz\.Parameter\-Efficient Fine\-Tuning without Introducing New Latency\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2023a\.[https://aclanthology\.org/2023\.acl\-long\.233/](https://aclanthology.org/2023.acl-long.233/)\.
- Liao et al\. \(2023b\)Baohao Liao, Shaomu Tan, and Christof Monz\.Make Pre\-trained Model Reversible: From Parameter to Memory Efficient Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023b\.[https://arxiv\.org/abs/2306\.00477](https://arxiv.org/abs/2306.00477)\.
- Liao et al\. \(2024\)Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz\.ApiQ: Finetuning of 2\-Bit Quantized Large Language Model\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2024\.[https://aclanthology\.org/2024\.emnlp\-main\.1168/](https://aclanthology.org/2024.emnlp-main.1168/)\.
- Lin et al\. \(2025\)Jessy Lin, Luke Zettlemoyer, Gargi Ghosh, Wen\-Tau Yih, Aram Markosyan, Vincent\-Pierre Berges, and Barlas Oğuz\.Continual Learning via Sparse Memory Finetuning\.*arXiv preprint arXiv:2510\.15103*, 2025\.[https://arxiv\.org/abs/2510\.15103](https://arxiv.org/abs/2510.15103)\.
- Lin et al\. \(2024a\)Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei\-Ming Chen, Wei\-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han\.AWQ: Activation\-aware Weight Quantization for On\-Device LLM Compression and Acceleration\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, volume 6, pages 87–100, 2024a\.[https://proceedings\.mlsys\.org/paper\_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21\-Abstract\-Conference\.html](https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html)\.
- Lin et al\. \(2026a\)Moule Lin, Shuhao Guan, Andrea Patane, David Gregg, and Goetz Botterweck\.Bayesian\-LoRA: Probabilistic Low\-Rank Adaptation of Large Language Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026a\.[https://arxiv\.org/abs/2601\.21003](https://arxiv.org/abs/2601.21003)\.
- Lin et al\. \(2026b\)Wei Lin, Yining Jiang, Qingyu Song, Qiao Xiang, and Hong Xu\.AGZO: Activation\-Guided Zeroth\-Order Optimization for LLM Fine\-Tuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026b\.[https://arxiv\.org/abs/2601\.17261](https://arxiv.org/abs/2601.17261)\.
- Lin et al\. \(2024b\)Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, and Hong Mei\.LoRA Dropout as a Sparsity Regularizer for Overfitting Control\.*arXiv preprint arXiv:2404\.09610*, 2024b\.[https://arxiv\.org/abs/2404\.09610](https://arxiv.org/abs/2404.09610)\.
- Lingam et al\. \(2024\)Vijay Lingam, Atula Tejaswi, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Aleksandar Bojchevski, and Sujay Sanghavi\.SVFT: Parameter\-Efficient Fine\-Tuning with Singular Vectors\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2405\.19597](https://arxiv.org/abs/2405.19597)\.
- Lion et al\. \(2025\)Kai Lion, Liang Zhang, Bingcong Li, and Niao He\.PoLAR: Polar\-Decomposed Low\-Rank Adapter Representation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2506\.03133](https://arxiv.org/abs/2506.03133)\.
- Liu et al\. \(2024a\)Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A\. Smith\.Tuning Language Models by Proxy\.In*Conference on Language Modeling \(COLM\)*, 2024a\.[https://arxiv\.org/abs/2401\.08565](https://arxiv.org/abs/2401.08565)\.
- Liu et al\. \(2022a\)Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel\.Few\-Shot Parameter\-Efficient Fine\-Tuning is Better and Cheaper than In\-Context Learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022a\.[https://arxiv\.org/abs/2205\.05638](https://arxiv.org/abs/2205.05638)\.
- Liu et al\. \(2026a\)Haokun Liu, Gyung Hyun Je, Marco Ciccone, Zhenlin Xu, Prasanth YSS, and Colin Raffel\.The Appeal and Reality of Recycling LoRAs with Adaptive Merging\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026a\.[https://arxiv\.org/abs/2602\.12323](https://arxiv.org/abs/2602.12323)\.
- Liu et al\. \(2026b\)Kainan Liu, Yong Zhang, Ning Cheng, Yun Zhu, Yanmeng Wang, Shaojun Wang, and Jing Xiao\.Astra: Activation\-Space Tail\-Eigenvector Low\-Rank Adaptation of Large Language Models\.In*Findings of the Association for Computational Linguistics: ACL 2026*, 2026b\.[https://aclanthology\.org/2026\.findings\-acl\.1593/](https://aclanthology.org/2026.findings-acl.1593/)\.
- Liu et al\. \(2026c\)Muqing Liu, Chongjie Si, and Yuheng Jia\.FlexLoRA: Entropy\-Guided Flexible Low\-Rank Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2026c\.[https://arxiv\.org/abs/2601\.22905](https://arxiv.org/abs/2601.22905)\.
- Liu et al\. \(2021a\)Peiyu Liu, Ze\-Feng Gao, Wayne Xin Zhao, Zhi\-Yuan Xie, Zhong\-Yi Lu, and Ji\-Rong Wen\.Enabling lightweight fine\-tuning for pre\-trained language model compression based on matrix product operators\.In*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 5388–5398\. Association for Computational Linguistics, 2021a\.doi:10\.18653/v1/2021\.acl\-long\.418\.
- Liu et al\. \(2021b\)Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig\.Pre\-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing\.*ACM Computing Surveys*, 2021b\.[https://doi\.org/10\.1145/3560815](https://doi.org/10.1145/3560815)\.
- Liu et al\. \(2023a\)Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng\.When MOE Meets LLMs: Parameter Efficient Fine\-tuning for Multi\-task Medical Applications\.In*SIGIR*, 2023a\.[https://arxiv\.org/abs/2310\.18339](https://arxiv.org/abs/2310.18339)\.
- Liu et al\. \(2024b\)Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\.DoRA: Weight\-Decomposed Low\-Rank Adaptation\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024b\.[https://arxiv\.org/abs/2402\.09353](https://arxiv.org/abs/2402.09353)\.
- Liu et al\. \(2018\)Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song\.Learning towards Minimum Hyperspherical Energy\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2018\.[https://arxiv\.org/abs/1805\.09298](https://arxiv.org/abs/1805.09298)\.
- Liu et al\. \(2020\)Weiyang Liu, Rongmei Lin, Zhen Liu, James M\. Rehg, Liam Paull, Li Xiong, Le Song, and Adrian Weller\.Orthogonal Over\-Parameterized Training\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2020\.[https://arxiv\.org/abs/2004\.04690](https://arxiv.org/abs/2004.04690)\.
- Liu et al\. \(2023b\)Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J\. Black, Adrian Weller, and Bernhard Schölkopf\.Parameter\-Efficient Orthogonal Finetuning via Butterfly Factorization\.In*International Conference on Learning Representations \(ICLR\)*, 2023b\.[https://arxiv\.org/abs/2311\.06243](https://arxiv.org/abs/2311.06243)\.
- Liu et al\. \(2022b\)Xiangyang Liu, Tianxiang Sun, Xuanjing Huang, and Xipeng Qiu\.Late Prompt Tuning: A Late Prompt Could Be Better Than Many Prompts\.In*Findings of the Association for Computational Linguistics: EMNLP 2022*, 2022b\.[https://aclanthology\.org/2022\.findings\-emnlp\.95/](https://aclanthology.org/2022.findings-emnlp.95/)\.
- Liu et al\. \(2021c\)Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang\.P\-Tuning v2: Prompt Tuning Can Be Comparable to Fine\-tuning Universally Across Scales and Tasks\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021c\.[https://aclanthology\.org/2022\.acl\-short\.8/](https://aclanthology.org/2022.acl-short.8/)\.
- Liu et al\. \(2021d\)Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang\.GPT Understands, Too\.*AI Open*, 2021d\.[https://doi\.org/10\.1016/j\.aiopen\.2023\.08\.012](https://doi.org/10.1016/j.aiopen.2023.08.012)\.
- Liu et al\. \(2026d\)Yewei Liu, Xiyuan Wang, Yansheng Mao, Yoav Gelbery, Haggai Maron, and Muhan Zhang\.SHINE: A Scalable In\-Context Hypernetwork for Mapping Context to LoRA in a Single Pass\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026d\.[https://arxiv\.org/abs/2602\.06358](https://arxiv.org/abs/2602.06358)\.
- Liu et al\. \(2024c\)Yongkang Liu, Yiqun Zhang, Qian Li, Tong Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schütze\.HiFT: A Hierarchical Full Parameter Fine\-Tuning Strategy\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2024c\.[https://aclanthology\.org/2024\.emnlp\-main\.1015/](https://aclanthology.org/2024.emnlp-main.1015/)\.
- Liu et al\. \(2026e\)Yongkang Liu, Xing Li, Mengjie Zhao, Shanru Zhang, Zijing Wang, Qian Li, Shi Feng, Feiliang Ren, Daling Wang, and Hinrich Schütze\.SMoA: Spectrum Modulation Adapter for Parameter\-Efficient Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026e\.[https://arxiv\.org/abs/2605\.21147](https://arxiv.org/abs/2605.21147)\.
- Liu et al\. \(2024d\)Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham\.ALoRA: Allocating Low\-Rank Adaptation for Fine\-tuning Large Language Models\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2024d\.[https://arxiv\.org/abs/2403\.16187](https://arxiv.org/abs/2403.16187)\.
- Liu et al\. \(2025\)Zihang Liu, Tianyu Pang, Oleg Balabanov, Chaoqun Yang, Tianjin Huang, Lu Yin, Yaoqing Yang, and Shiwei Liu\.LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning\-Focused Supervised Fine\-Tuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025\.[https://arxiv\.org/abs/2506\.00772](https://arxiv.org/abs/2506.00772)\.
- Loeschcke et al\. \(2024\)Sebastian Loeschcke, Mads Toftrup, Michael J\. Kastoryano, Serge Belongie, and Vésteinn Snæbjarnarson\.LoQT: Low\-Rank Adapters for Quantized Pretraining\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2405\.16528](https://arxiv.org/abs/2405.16528)\.
- Lopez\-Piqueres et al\. \(2025\)Javier Lopez\-Piqueres, Pranav Deshpande, Archan Ray, Mattia J\. Villani, Marco Pistoia, and Niraj Kumar\.MetaTT: A Global Tensor\-Train Adapter for Parameter\-Efficient Fine\-Tuning\.*TMLR \(2026\)*, 2025\.[https://arxiv\.org/abs/2506\.09105](https://arxiv.org/abs/2506.09105)\.
- Loshchilov and Hutter \(2019\)Ilya Loshchilov and Frank Hutter\.Decoupled weight decay regularization\.In*International Conference on Learning Representations \(ICLR\)*, 2019\.[https://arxiv\.org/abs/1711\.05101](https://arxiv.org/abs/1711.05101)\.
- Ludwig \(2025\)Oswaldo Ludwig\.The Condition Number as a Scale\-Invariant Proxy for Information Encoding in Neural Units\.*arXiv preprint arXiv:2506\.16289*, 2025\.[https://arxiv\.org/abs/2506\.16289](https://arxiv.org/abs/2506.16289)\.
- Luo et al\. \(2023a\)Gen Luo, Minglang Huang, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Zhiyu Wang, and Rongrong Ji\.Towards Efficient Visual Adaption via Structural Re\-parameterization\.*arXiv preprint arXiv:2302\.08106*, 2023a\.[https://arxiv\.org/abs/2302\.08106](https://arxiv.org/abs/2302.08106)\.
- Luo et al\. \(2026\)Mao\-Lin Luo, Zi\-Hao Zhou, Yi\-Lin Zhang, Yuanyu Wan, Tong Wei, and Min\-Ling Zhang\.KeepLoRA: Continual Learning with Residual Gradient Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.[https://arxiv\.org/abs/2601\.19659](https://arxiv.org/abs/2601.19659)\.
- Luo et al\. \(2024\)Qijun Luo, Hengxu Yu, and Xiao Li\.BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2404\.02827](https://arxiv.org/abs/2404.02827)\.
- Luo et al\. \(2023b\)Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao\.LCM\-LoRA: A Universal Stable\-Diffusion Acceleration Module\.*arXiv preprint arXiv:2311\.05556*, 2023b\.[https://arxiv\.org/abs/2311\.05556](https://arxiv.org/abs/2311.05556)\.
- Lv et al\. \(2023a\)Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu\.AdaLomo: Low\-memory Optimization with Adaptive Learning Rate\.In*Findings of the Association for Computational Linguistics: ACL 2024*, 2023a\.[https://aclanthology\.org/2024\.findings\-acl\.742/](https://aclanthology.org/2024.findings-acl.742/)\.
- Lv et al\. \(2023b\)Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu\.Full Parameter Fine\-tuning for Large Language Models with Limited Resources\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2023b\.[https://arxiv\.org/abs/2306\.09782](https://arxiv.org/abs/2306.09782)\.
- Ma et al\. \(2022\)Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song\.XPrompt: Exploring the Extreme of Prompt Tuning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2022\.[https://aclanthology\.org/2022\.emnlp\-main\.758/](https://aclanthology.org/2022.emnlp-main.758/)\.
- Ma et al\. \(2024\)Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, and Junfeng Zhao\.Parameter Efficient Quasi\-Orthogonal Fine\-Tuning via Givens Rotation\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2404\.04316](https://arxiv.org/abs/2404.04316)\.
- Mac Lane \(1998\)Saunders Mac Lane\.*Categories for the Working Mathematician*, volume 5 of*Graduate Texts in Mathematics*\.Springer, New York, second edition, 1998\.ISBN 0\-387\-98403\-8\.doi:10\.1007/978\-1\-4757\-4721\-8\.
- Mahabadi et al\. \(2021a\)Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder\.Compacter: Efficient Low\-Rank Hypercomplex Adapter Layers\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2021a\.[https://arxiv\.org/abs/2106\.04647](https://arxiv.org/abs/2106.04647)\.
- Mahabadi et al\. \(2021b\)Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson\.Parameter\-efficient Multi\-task Fine\-tuning for Transformers via Shared Hypernetworks\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021b\.[https://aclanthology\.org/2021\.acl\-long\.47/](https://aclanthology.org/2021.acl-long.47/)\.
- Malladi et al\. \(2022\)Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora\.A Kernel\-Based View of Language Model Fine\-Tuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2022\.[https://arxiv\.org/abs/2210\.05643](https://arxiv.org/abs/2210.05643)\.
- Malladi et al\. \(2023\)Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D\. Lee, Danqi Chen, and Sanjeev Arora\.Fine\-Tuning Language Models with Just Forward Passes\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2305\.17333](https://arxiv.org/abs/2305.17333)\.
- Mallya et al\. \(2018\)Arun Mallya, Dillon Davis, and Svetlana Lazebnik\.Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights\.In*European Conference on Computer Vision \(ECCV\)*, 2018\.[https://arxiv\.org/abs/1801\.06519](https://arxiv.org/abs/1801.06519)\.
- Mangrulkar et al\. \(2022\)Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, Benjamin Bossan, and Marian Tietz\.PEFT: State\-of\-the\-art parameter\-efficient fine\-tuning methods\.[https://github\.com/huggingface/peft](https://github.com/huggingface/peft), 2022\.
- Manolache et al\. \(2026\)Alexandru\-Dragos Manolache, Yunqiang Li, and Jan van Gemert\.Low\-Rank Ternary Adaptation for Fine\-Tuning Transformers\.In*European Conference on Computer Vision \(ECCV\)*, 2026\.[https://arxiv\.org/abs/2608\.24469](https://arxiv.org/abs/2608.24469)\.
- Mao et al\. \(2024a\)Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu\.DoRA: Enhancing Parameter\-Efficient Fine\-Tuning with Dynamic Rank Distribution\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024a\.[https://aclanthology\.org/2024\.acl\-long\.626/](https://aclanthology.org/2024.acl-long.626/)\.
- Mao et al\. \(2021\)Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen tau Yih, and Madian Khabsa\.UniPELT: A Unified Framework for Parameter\-Efficient Language Model Tuning\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://arxiv\.org/abs/2110\.07577](https://arxiv.org/abs/2110.07577)\.
- Mao et al\. \(2024b\)Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao\.A Survey on LoRA of Large Language Models\.*Frontiers of Computer Science*, 2024b\.[https://arxiv\.org/abs/2407\.11046](https://arxiv.org/abs/2407.11046)\.
- Marčenko and Pastur \(1967\)V\. A\. Marčenko and L\. A\. Pastur\.Distribution of eigenvalues for some sets of random matrices\.*Mathematics of the USSR\-Sbornik*, 1\(4\):457–483, 1967\.doi:10\.1070/SM1967v001n04ABEH001994\.
- Marcotte et al\. \(2023\)Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré\.Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2307\.00144](https://arxiv.org/abs/2307.00144)\.
- Marcotte et al\. \(2024\)Sibylle Marcotte, Rémi Gribonval, and Gabriel Peyré\.Keep the momentum: Conservation laws beyond Euclidean gradient flows\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*\. PMLR, 2024\.[https://arxiv\.org/abs/2405\.12888](https://arxiv.org/abs/2405.12888)\.
- Matena and Raffel \(2021\)Michael Matena and Colin Raffel\.Merging Models with Fisher\-Weighted Averaging\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2021\.[https://arxiv\.org/abs/2111\.09832](https://arxiv.org/abs/2111.09832)\.
- Mattila \(1995\)Pertti Mattila\.*Geometry of Sets and Measures in Euclidean Spaces: Fractals and Rectifiability*, volume 44 of*Cambridge Studies in Advanced Mathematics*\.Cambridge University Press, Cambridge, 1995\.doi:10\.1017/CBO9780511623813\.
- McLarty \(2007\)Colin McLarty\.The rising sea: Grothendieck on simplicity and generality\.In Jeremy J\. Gray and Karen Hunger Parshall, editors,*Episodes in the History of Modern Algebra \(1800–1950\)*, volume 32 of*History of Mathematics*, pages 301–325\. American Mathematical Society, Providence, RI, 2007\.ISBN 978\-0\-8218\-4343\-7\.doi:10\.1090/hmath/032/14\.
- Meng et al\. \(2026\)Chuiyang Meng, Ming Tang, and Vincent W\. S\. Wong\.FLoRG: Federated Fine\-tuning with Low\-rank Gram Matrices and Procrustes Alignment\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.[https://arxiv\.org/abs/2602\.17095](https://arxiv.org/abs/2602.17095)\.
- Meng et al\. \(2024a\)Fanxu Meng, Zhaohui Wang, and Muhan Zhang\.PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024a\.[https://arxiv\.org/abs/2404\.02948](https://arxiv.org/abs/2404.02948)\.
- Meng et al\. \(2024b\)Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Peiyi Wang, Qingxiu Dong, Liang Chen, and Zhifang Sui\.PeriodicLoRA: Breaking the Low\-Rank Bottleneck in LoRA Optimization\.*arXiv preprint arXiv:2402\.16141*, 2024b\.[https://arxiv\.org/abs/2402\.16141](https://arxiv.org/abs/2402.16141)\.
- Mhammedi et al\. \(2016\)Zakaria Mhammedi, Andrew Hellicar, Ashfaqur Rahman, and James Bailey\.Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflections\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2016\.[https://arxiv\.org/abs/1612\.00188](https://arxiv.org/abs/1612.00188)\.
- Miles et al\. \(2024\)Roy Miles, Pradyumna Reddy, Ismail Elezi, and Jiankang Deng\.VeLoRA: Memory Efficient Training using Rank\-1 Sub\-Token Projections\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2405\.17991](https://arxiv.org/abs/2405.17991)\.
- Min et al\. \(2021\)Hancheng Min, Salma Tarmoun, René Vidal, and Enrique Mallada\.On the explicit role of initialization on the convergence and implicit bias of overparametrized linear networks\.In Marina Meila and Tong Zhang, editors,*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, pages 7760–7768\. PMLR, 2021\.[https://proceedings\.mlr\.press/v139/min21c\.html](https://proceedings.mlr.press/v139/min21c.html)\.
- Mirsky \(1960\)L\. Mirsky\.Symmetric gauge functions and unitarily invariant norms\.*The Quarterly Journal of Mathematics*, 11\(1\):50–59, 1960\.doi:10\.1093/qmath/11\.1\.50\.
- Mishra et al\. \(2012\)B\. Mishra, G\. Meyer, S\. Bonnabel, and R\. Sepulchre\.Fixed\-rank matrix factorizations and Riemannian low\-rank optimization\.*Computational Statistics*, 2012\.[https://arxiv\.org/abs/1209\.0430](https://arxiv.org/abs/1209.0430)\.
- Mishra and Sepulchre \(2016\)Bamdev Mishra and Rodolphe Sepulchre\.Riemannian Preconditioning\.*SIAM Journal on Optimization*, 26\(1\):635–660, 2016\.doi:10\.1137/140970860\.
- Mitchell et al\. \(2023\)Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D\. Manning\.An Emulator for Fine\-Tuning Large Language Models using Small Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2310\.12962](https://arxiv.org/abs/2310.12962)\.
- Mityagin \(2020\)Boris S\. Mityagin\.The zero set of a real analytic function\.*Mathematical Notes*, 107\(3–4\):529–530, 2020\.doi:10\.1134/S0001434620030189\.
- Modoranu et al\. \(2026\)Ionut\-Vlad Modoranu, Mher Safaryan, and Dan Alistarh\.MatryoshkaLoRA: Learning Accurate Hierarchical Low\-Rank Representations for LLM Fine\-Tuning\.*arXiv preprint arXiv:2605\.07850*, 2026\.[https://arxiv\.org/abs/2605\.07850](https://arxiv.org/abs/2605.07850)\.
- Mondal and Shihab \(2026\)Joyanta Jyoti Mondal and Ibne Farabi Shihab\.When Can Prefixes Compile LoRA? Exact Resource\-Capped Tests for Frozen Attention\.*arXiv preprint arXiv:2609\.36766*, 2026\.[https://arxiv\.org/abs/2609\.36766](https://arxiv.org/abs/2609.36766)\.
- Morris et al\. \(2026\)John X\. Morris, Niloofar Mireshghallah, Mark Ibrahim, and Saeed Mahloujifar\.Learning to Reason in 13 Parameters\.*arXiv preprint arXiv:2602\.04118*, 2026\.[https://arxiv\.org/abs/2602\.04118](https://arxiv.org/abs/2602.04118)\.
- Mou et al\. \(2023\)Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie\.T2I\-Adapter: Learning Adapters to Dig out More Controllable Ability for Text\-to\-Image Diffusion Models\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2023\.[https://arxiv\.org/abs/2302\.08453](https://arxiv.org/abs/2302.08453)\.
- Mu and Klabjan \(2025\)Siqiao Mu and Diego Klabjan\.On the Convergence Rate of LoRA Gradient Descent\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025\.[https://arxiv\.org/abs/2512\.18248](https://arxiv.org/abs/2512.18248)\.
- Mukherjee et al\. \(2025\)Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani\-Tur, and Hao Peng\.Reinforcement Learning Finetunes Small Subnetworks in Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2505\.11711](https://arxiv.org/abs/2505.11711)\.
- Mumford et al\. \(1994\)David Mumford, John Fogarty, and Frances Kirwan\.*Geometric Invariant Theory*, volume 34 of*Ergebnisse der Mathematik und ihrer Grenzgebiete \(2\)*\.Springer, Berlin, Heidelberg, third edition, 1994\.doi:10\.1007/978\-3\-642\-57916\-5\.
- Muqeeth et al\. \(2023\)Mohammed Muqeeth, Haokun Liu, and Colin Raffel\.Soft Merging of Experts with Adaptive Routing\.*TMLR*, 2023\.[https://openreview\.net/forum?id=7I199lc54z](https://openreview.net/forum?id=7I199lc54z)\.
- Muqeeth et al\. \(2024\)Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel\.Learning to Route Among Specialized Experts for Zero\-Shot Generalization\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2402\.05859](https://arxiv.org/abs/2402.05859)\.
- Navon et al\. \(2023\)Aviv Navon, Aviv Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik, and Haggai Maron\.Equivariant architectures for learning in deep weight spaces\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*\. PMLR, 2023\.[https://arxiv\.org/abs/2301\.12780](https://arxiv.org/abs/2301.12780)\.
- Nayak et al\. \(2025\)Nikhil Shivakumar Nayak, Krishnateja Killamsetty, Ligong Han, Abhishek Bhandwaldar, Prateek Chanda, Kai Xu, Hao Wang, Aldo Pareja, Oleg Silkin, Mustafa Eyceoz, and Akash Srivastava\.Sculpting Subspaces: Constrained Full Fine\-Tuning in LLMs for Continual Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2504\.07097](https://arxiv.org/abs/2504.07097)\.
- Nesterov and Spokoiny \(2017\)Yurii Nesterov and Vladimir Spokoiny\.Random gradient\-free minimization of convex functions\.*Foundations of Computational Mathematics*, 17\(2\):527–566, 2017\.doi:10\.1007/s10208\-015\-9296\-2\.
- Neyshabur et al\. \(2015\)Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro\.Path\-SGD: Path\-normalized optimization in deep neural networks\.In*Advances in Neural Information Processing Systems*, volume 28, 2015\.[https://arxiv\.org/abs/1506\.02617](https://arxiv.org/abs/1506.02617)\.
- Nguyen et al\. \(2026\)An Nguyen, Jaesik Choi, and Anh Tong\.LoCO: Low\-rank Compositional Rotation Fine\-tuning\.In*Proceedings of the International Joint Conference on Artificial Intelligence \(IJCAI\)*, 2026\.[https://arxiv\.org/abs/2605\.15916](https://arxiv.org/abs/2605.15916)\.
- Nguyen et al\. \(2025\)Le\-Trung Nguyen, Enzo Tartaglione, and Van\-Tam Nguyen\.Efficient Resource\-Constrained Training of Transformers via Subspace Optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2510\.09160](https://arxiv.org/abs/2510.09160)\.
- Nielsen and Chuang \(2000\)Michael A\. Nielsen and Isaac L\. Chuang\.*Quantum Computation and Quantum Information*\.Cambridge University Press, Cambridge, 2000\.ISBN 0\-521\-63503\-9\.
- Nikdan et al\. \(2024\)Mahdi Nikdan, Soroush Tabesh, Elvir Crnčević, and Dan Alistarh\.RoSA: Accurate Parameter\-Efficient Fine\-Tuning via Robust Adaptation\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2401\.04679](https://arxiv.org/abs/2401.04679)\.
- Noether \(1918\)Emmy Noether\.Invariante Variationsprobleme\.*Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathematisch\-Physikalische Klasse*, 1918:235–257, 1918\.[http://eudml\.org/doc/59024](http://eudml.org/doc/59024)\.
- Ortiz\-Jimenez et al\. \(2023\)Guillermo Ortiz\-Jimenez, Alessandro Favero, and Pascal Frossard\.Task Arithmetic in the Tangent Space: Improved Editing of Pre\-Trained Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2305\.12827](https://arxiv.org/abs/2305.12827)\.
- Orús \(2014\)Román Orús\.A practical introduction to tensor networks: Matrix product states and projected entangled pair states\.*Annals of Physics*, 349:117–158, 2014\.doi:10\.1016/j\.aop\.2014\.06\.013\.
- Oseledets \(2011\)Ivan V\. Oseledets\.Tensor\-train decomposition\.*SIAM Journal on Scientific Computing*, 33\(5\):2295–2317, 2011\.doi:10\.1137/090752286\.
- Ostapenko et al\. \(2024\)Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni\.Towards Modular LLMs by Building and Reusing a Library of LoRAs\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2405\.11157](https://arxiv.org/abs/2405.11157)\.
- Ouyang et al\. \(2025\)Ziheng Ouyang, Zhen Li, and Qibin Hou\.K\-LoRA: Unlocking Training\-Free Fusion of Any Subject and Style LoRAs\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2025\.[https://arxiv\.org/abs/2502\.18461](https://arxiv.org/abs/2502.18461)\.
- Oymak et al\. \(2023\)Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis\.On the Role of Attention in Prompt\-tuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2023\.[https://proceedings\.mlr\.press/v202/oymak23a\.html](https://proceedings.mlr.press/v202/oymak23a.html)\.
- Paischer et al\. \(2024\)Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin, Marc Peter Deisenroth, and Sepp Hochreiter\.Parameter Efficient Fine\-tuning via Explained Variance Adaptation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2410\.07170](https://arxiv.org/abs/2410.07170)\.
- Pan et al\. \(2022\)Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li\.ST\-Adapter: Parameter\-Efficient Image\-to\-Video Transfer Learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.[https://arxiv\.org/abs/2206\.13559](https://arxiv.org/abs/2206.13559)\.
- Pan et al\. \(2024\)Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang\.LISA: Layerwise Importance Sampling for Memory\-Efficient Large Language Model Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2403\.17919](https://arxiv.org/abs/2403.17919)\.
- Panariello et al\. \(2025\)Aniello Panariello, Daniel Marczak, Simone Magistri, Angelo Porrello, Bartłomiej Twardowski, Andrew D\. Bagdanov, Simone Calderara, and Joost van de Weijer\.Accurate and Efficient Low\-Rank Model Merging in Core Space\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2509\.17786](https://arxiv.org/abs/2509.17786)\.
- Paszke et al\. \(2019\)Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala\.PyTorch: An imperative style, high\-performance deep learning library\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.
- Perez et al\. \(2017\)Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville\.FiLM: Visual Reasoning with a General Conditioning Layer\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2017\.[https://arxiv\.org/abs/1709\.07871](https://arxiv.org/abs/1709.07871)\.
- Petrov et al\. \(2023\)Aleksandar Petrov, Philip H\. S\. Torr, and Adel Bibi\.When Do Prompting and Prefix\-Tuning Work? A Theory of Capabilities and Limitations\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2310\.19698](https://arxiv.org/abs/2310.19698)\.
- Petrov et al\. \(2024\)Aleksandar Petrov, Philip H\. S\. Torr, and Adel Bibi\.Prompting a Pretrained Transformer Can Be a Universal Approximator\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://proceedings\.mlr\.press/v235/petrov24a\.html](https://proceedings.mlr.press/v235/petrov24a.html)\.
- Pfeiffer et al\. \(2020a\)Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych\.AdapterFusion: Non\-Destructive Task Composition for Transfer Learning\.In*Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics \(EACL\)*, 2020a\.[https://arxiv\.org/abs/2005\.00247](https://arxiv.org/abs/2005.00247)\.
- Pfeiffer et al\. \(2020b\)Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulić, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych\.AdapterHub: A Framework for Adapting Transformers\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2020b\.[https://arxiv\.org/abs/2007\.07779](https://arxiv.org/abs/2007.07779)\.
- Pfeiffer et al\. \(2020c\)Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder\.MAD\-X: An Adapter\-Based Framework for Multi\-Task Cross\-Lingual Transfer\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2020c\.[https://arxiv\.org/abs/2005\.00052](https://arxiv.org/abs/2005.00052)\.
- Pfeiffer et al\. \(2023\)Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti\.Modular Deep Learning\.*TMLR*, 2023\.[https://arxiv\.org/abs/2302\.11529](https://arxiv.org/abs/2302.11529)\.
- Po et al\. \(2023\)Ryan Po, Guandao Yang, Kfir Aberman, and Gordon Wetzstein\.Orthogonal Adaptation for Modular Customization of Diffusion Models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023\.[https://arxiv\.org/abs/2312\.02432](https://arxiv.org/abs/2312.02432)\.
- Ponkshe et al\. \(2024\)Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma\.Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low\-Rank Fine\-Tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2411\.19557](https://arxiv.org/abs/2411.19557)\.
- Ponti et al\. \(2022\)Edoardo M\. Ponti, Alessandro Sordoni, Yoshua Bengio, and Siva Reddy\.Combining Parameter\-efficient Modules for Task\-level Generalisation\.In*Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics \(EACL\)*, 2022\.[https://aclanthology\.org/2023\.eacl\-main\.49/](https://aclanthology.org/2023.eacl-main.49/)\.
- Poth et al\. \(2023\)Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer\.Adapters: A Unified Library for Parameter\-Efficient and Modular Transfer Learning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2023\.[https://arxiv\.org/abs/2311\.11077](https://arxiv.org/abs/2311.11077)\.
- Prabhakar et al\. \(2024\)Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi\.LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks\.In*Proceedings of the International Conference on Computational Linguistics \(COLING\)*, 2024\.[https://arxiv\.org/abs/2410\.13025](https://arxiv.org/abs/2410.13025)\.
- Putterman et al\. \(2024\)Theo Putterman, Derek Lim, Yoav Gelberg, Stefanie Jegelka, and Haggai Maron\.Learning on LoRAs: GL\-Equivariant Processing of Low\-Rank Weight Spaces for Large Finetuned Models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2410\.04207](https://arxiv.org/abs/2410.04207)\.
- Qiao and Mahdavi \(2025\)Fuli Qiao and Mehrdad Mahdavi\.Merge before Forget: A Single LoRA Continual Learning via Continual Merging\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2512\.23017](https://arxiv.org/abs/2512.23017)\.
- Qin et al\. \(2024a\)Haotong Qin, Xudong Ma, Xingyu Zheng, Xiaoyang Li, Yang Zhang, Shouda Liu, Jie Luo, Xianglong Liu, and Michele Magno\.Accurate LoRA\-Finetuning Quantization of LLMs via Information Retention\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024a\.[https://arxiv\.org/abs/2402\.05445](https://arxiv.org/abs/2402.05445)\.
- Qin et al\. \(2024b\)Peijia Qin, Ruiyi Zhang, and Pengtao Xie\.BiDoRA: Bi\-level Optimization\-Based Weight\-Decomposed Low\-Rank Adaptation\.*TMLR*, 2024b\.[https://arxiv\.org/abs/2410\.09758](https://arxiv.org/abs/2410.09758)\.
- Qin et al\. \(2021\)Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou\.Exploring Universal Intrinsic Task Subspace via Prompt Tuning\.*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 2021\.[https://doi\.org/10\.1109/TASLP\.2024\.3430545](https://doi.org/10.1109/TASLP.2024.3430545)\.
- Qiu et al\. \(2024\)Shikai Qiu, Andres Potapczynski, Marc Finzi, Micah Goldblum, and Andrew Gordon Wilson\.Compute Better Spent: Replacing Dense Layers with Structured Matrices\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024\.[https://arxiv\.org/abs/2406\.06248](https://arxiv.org/abs/2406.06248)\.
- Qiu et al\. \(2023\)Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf\.Controlling Text\-to\-Image Diffusion by Orthogonal Finetuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2306\.07280](https://arxiv.org/abs/2306.07280)\.
- Qiu et al\. \(2025\)Zeju Qiu, Weiyang Liu, Adrian Weller, and Bernhard Schölkopf\.Orthogonal Finetuning Made Scalable\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2025\.[https://arxiv\.org/abs/2506\.19847](https://arxiv.org/abs/2506.19847)\.
- Qiu et al\. \(2026\)Zeju Qiu, Lixin Liu, Adrian Weller, Han Shi, and Weiyang Liu\.POET\-X: Memory\-efficient LLM Training by Scaling Orthogonal Transformation\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2603\.05500](https://arxiv.org/abs/2603.05500)\.
- Razdaibiedina et al\. \(2023\)Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi\.Residual Prompt Tuning: Improving Prompt Tuning with Residual Reparameterization\.In*Findings of the Association for Computational Linguistics: ACL 2023*, 2023\.[https://aclanthology\.org/2023\.findings\-acl\.421/](https://aclanthology.org/2023.findings-acl.421/)\.
- Rebuffi et al\. \(2017\)Sylvestre\-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi\.Learning multiple visual domains with residual adapters\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2017\.[https://arxiv\.org/abs/1705\.08045](https://arxiv.org/abs/1705.08045)\.
- Recht et al\. \(2010\)Benjamin Recht, Maryam Fazel, and Pablo A\. Parrilo\.Guaranteed minimum\-rank solutions of linear matrix equations via nuclear norm minimization\.*SIAM Review*, 52\(3\):471–501, 2010\.doi:10\.1137/070697835\.
- Ren et al\. \(2024\)Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei\.MELoRA: Mini\-Ensemble Low\-Rank Adapters for Parameter\-Efficient Fine\-Tuning\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024\.[https://arxiv\.org/abs/2402\.17263](https://arxiv.org/abs/2402.17263)\.
- Renduchintala et al\. \(2023\)Adithya Renduchintala, Tugrul Konuk, and Oleksii Kuchaiev\.Tied\-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2023\.[https://aclanthology\.org/2024\.naacl\-long\.481/](https://aclanthology.org/2024.naacl-long.481/)\.
- Richardson and Slodowy \(1990\)R\. W\. Richardson and P\. J\. Slodowy\.Minimum vectors for real reductive algebraic groups\.*Journal of the London Mathematical Society*, s2\-42\(3\):409–429, 1990\.doi:10\.1112/jlms/s2\-42\.3\.409\.
- Riehl \(2016\)Emily Riehl\.*Category Theory in Context*\.Aurora: Dover Modern Math Originals\. Dover Publications, Mineola, NY, 2016\.ISBN 978\-0\-486\-80903\-8\.
- Rinaldi et al\. \(2025\)Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Fengyuan Liu, Marco Ciccone, Angelo Porrello, and Simone Calderara\.Gradient\-Sign Masking for Task Vector Transport Across Pre\-Trained Models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2510\.09658](https://arxiv.org/abs/2510.09658)\.
- Robert et al\. \(2025\)Thomas Robert, Mher Safaryan, Ionut\-Vlad Modoranu, and Dan Alistarh\.LDAdam: Adaptive Optimization from Low\-Dimensional Gradient Statistics\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2410\.16103](https://arxiv.org/abs/2410.16103)\.
- Ruiz et al\. \(2023\)Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman\.HyperDreamBooth: HyperNetworks for Fast Personalization of Text\-to\-Image Models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023\.[https://arxiv\.org/abs/2307\.06949](https://arxiv.org/abs/2307.06949)\.
- Rücklé et al\. \(2020\)Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych\.AdapterDrop: On the Efficiency of Adapters in Transformers\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2020\.[https://arxiv\.org/abs/2010\.11918](https://arxiv.org/abs/2010.11918)\.
- Rüdiger and Raschka \(2026\)Sten Rüdiger and Sebastian Raschka\.MiCA Learns More Knowledge Than LoRA and Full Fine\-Tuning\.*arXiv preprint arXiv:2604\.01694*, 2026\.[https://arxiv\.org/abs/2604\.01694](https://arxiv.org/abs/2604.01694)\.
- Saha et al\. \(2021\)Gobinda Saha, Isha Garg, and Kaushik Roy\.Gradient Projection Memory for Continual Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.[https://openreview\.net/forum?id=3AOj0RCNC2](https://openreview.net/forum?id=3AOj0RCNC2)\.
- Salimans and Kingma \(2016\)Tim Salimans and Diederik P\. Kingma\.Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2016\.[https://arxiv\.org/abs/1602\.07868](https://arxiv.org/abs/1602.07868)\.
- Saxe et al\. \(2014\)Andrew M\. Saxe, James L\. McClelland, and Surya Ganguli\.Exact solutions to the nonlinear dynamics of learning in deep linear neural networks\.In*International Conference on Learning Representations \(ICLR\)*, 2014\.[https://arxiv\.org/abs/1312\.6120](https://arxiv.org/abs/1312.6120)\.
- Scherk \(1950\)Peter Scherk\.On the decomposition of orthogonalities into symmetries\.*Proceedings of the American Mathematical Society*, 1\(4\):481–491, 1950\.doi:10\.1090/S0002\-9939\-1950\-0036762\-3\.
- Schneider and Uschmajew \(2015\)Reinhold Schneider and André Uschmajew\.Convergence results for projected line\-search methods on varieties of low\-rank matrices via Łojasiewicz inequality\.*SIAM Journal on Optimization*, 25\(1\):622–646, 2015\.doi:10\.1137/140957822\.
- Schulman and Thinking Machines Lab \(2025\)John Schulman and Thinking Machines Lab\.LoRA without regret\.*Thinking Machines Lab: Connectionism*, 2025\.doi:10\.64434/tml\.20250929\.[https://thinkingmachines\.ai/blog/lora/](https://thinkingmachines.ai/blog/lora/)\.
- Selinger \(2011\)Peter Selinger\.A survey of graphical languages for monoidal categories\.In Bob Coecke, editor,*New Structures for Physics*, volume 813 of*Lecture Notes in Physics*, pages 289–355\. Springer, Berlin, Heidelberg, 2011\.doi:10\.1007/978\-3\-642\-12821\-9\_4\.
- Sengupta et al\. \(2024\)Ayan Sengupta, Vaibhav Seth, Arinjay Pathak, Aastha Verma, Natraj Raman, Sriram Gopalakrishnan, Niladri Chatterjee, and Tanmoy Chakraborty\.Robust and Efficient Fine\-tuning of LLMs with Bayesian Reparameterization of Low\-Rank Adaptation\.*TMLR*, 2024\.[https://arxiv\.org/abs/2411\.04358](https://arxiv.org/abs/2411.04358)\.
- Shah et al\. \(2023\)Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani\.ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs\.In*European Conference on Computer Vision \(ECCV\)*, 2023\.[https://arxiv\.org/abs/2311\.13600](https://arxiv.org/abs/2311.13600)\.
- Shang et al\. \(2025\)Sifeng Shang, Jiayi Zhou, Chenyu Lin, Minxian Li, and Kaiyang Zhou\.Fine\-tuning Quantized Neural Networks with Zeroth\-order Optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2505\.13430](https://arxiv.org/abs/2505.13430)\.
- Sharpe \(1997\)R\. W\. Sharpe\.*Differential Geometry: Cartan’s Generalization of Klein’s Erlangen Program*, volume 166 of*Graduate Texts in Mathematics*\.Springer, New York, 1997\.
- Shazeer and Stern \(2018\)Noam Shazeer and Mitchell Stern\.Adafactor: Adaptive Learning Rates with Sublinear Memory Cost\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2018\.[https://arxiv\.org/abs/1804\.04235](https://arxiv.org/abs/1804.04235)\.
- Shen et al\. \(2025\)Yixian Shen, Qi Bi, Jia\-Hong Huang, Hongyi Zhu, Andy D\. Pimentel, and Anuj Pathania\.SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2025\.[https://aclanthology\.org/2025\.naacl\-long\.522/](https://aclanthology.org/2025.naacl-long.522/)\.
- Shenfeld et al\. \(2025\)Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal\.RL’s Razor: Why Online Reinforcement Learning Forgets Less\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2509\.04259](https://arxiv.org/abs/2509.04259)\.
- Sheng et al\. \(2023\)Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E\. Gonzalez, and Ion Stoica\.S\-LoRA: Serving Thousands of Concurrent LoRA Adapters\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2023\.[https://arxiv\.org/abs/2311\.03285](https://arxiv.org/abs/2311.03285)\.
- Shi and Lipani \(2023\)Zhengxiang Shi and Aldo Lipani\.DePT: Decomposed Prompt Tuning for Parameter\-Efficient Fine\-tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2309\.05173](https://arxiv.org/abs/2309.05173)\.
- Shiebler et al\. \(2021\)Dan Shiebler, Bruno Gavranović, and Paul Wilson\.Category Theory in Machine Learning\.*arXiv preprint arXiv:2106\.07032*, 2021\.[https://arxiv\.org/abs/2106\.07032](https://arxiv.org/abs/2106.07032)\.
- Shin et al\. \(2020\)Taylor Shin, Yasaman Razeghi, Robert L\. Logan, Eric Wallace, and Sameer Singh\.AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2020\.[https://aclanthology\.org/2020\.emnlp\-main\.346/](https://aclanthology.org/2020.emnlp-main.346/)\.
- Shuttleworth et al\. \(2024\)Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma\.LoRA vs Full Fine\-tuning: An Illusion of Equivalence\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2410\.21228](https://arxiv.org/abs/2410.21228)\.
- Si et al\. \(2024a\)Chongjie Si, Zhiyi Shi, Shifan Zhang, Xiaokang Yang, Hanspeter Pfister, and Wei Shen\.Unleashing the Power of Task\-Specific Directions in Parameter Efficient Fine\-tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.[https://openreview\.net/forum?id=RYrJqz44p4](https://openreview.net/forum?id=RYrJqz44p4)\.
- Si et al\. \(2024b\)Chongjie Si, Xuehui Wang, Xue Yang, Zhengqin Xu, Qingyun Li, Jifeng Dai, Yu Qiao, Xiaokang Yang, and Wei Shen\.Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine\-tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2024b\.[https://arxiv\.org/abs/2405\.14739](https://arxiv.org/abs/2405.14739)\.
- Si et al\. \(2024c\)Chongjie Si, Xiaokang Yang, and Wei Shen\.See Further for Parameter Efficient Fine\-tuning by Standing on the Shoulders of Decomposition\.*arXiv preprint arXiv:2407\.05417*, 2024c\.[https://arxiv\.org/abs/2407\.05417](https://arxiv.org/abs/2407.05417)\.
- Si et al\. \(2025\)Chongjie Si, Zhiyi Shi, Xuehui Wang, Yichen Xiao, Xiaokang Yang, and Wei Shen\.Generalized Tensor\-based Parameter\-Efficient Fine\-Tuning via Lie Group Transformations\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2025\.[https://arxiv\.org/abs/2504\.00851](https://arxiv.org/abs/2504.00851)\.
- Singh et al\. \(2024\)Ayush Singh, Rajdeep Aher, and Shivank Garg\.LoRA\-Mini : Adaptation Matrices Decomposition and Selective Training\.*arXiv preprint arXiv:2411\.15804*, 2024\.[https://arxiv\.org/abs/2411\.15804](https://arxiv.org/abs/2411.15804)\.
- Singh and Jaggi \(2020\)Sidak Pal Singh and Martin Jaggi\.Model fusion via optimal transport\.In*Advances in Neural Information Processing Systems*, volume 33, 2020\.[https://arxiv\.org/abs/1910\.05653](https://arxiv.org/abs/1910.05653)\.
- Singhal et al\. \(2025\)Raghav Singhal, Kaustubh Ponkshe, Rohit Vartak, and Praneeth Vepakomma\.ABBA\-Adapters: Efficient and Expressive Fine\-Tuning of Foundation Models\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2505\.14238](https://arxiv.org/abs/2505.14238)\.
- Smith et al\. \(2022\)James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante\-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira\.CODA\-Prompt: COntinual Decomposed Attention\-based Prompting for Rehearsal\-Free Continual Learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2022\.[https://openaccess\.thecvf\.com/content/CVPR2023/html/Smith\_CODA\-Prompt\_COntinual\_Decomposed\_Attention\-Based\_Prompting\_for\_Rehearsal\-Free\_Continual\_Learning\_CVPR\_2023\_paper\.html](https://openaccess.thecvf.com/content/CVPR2023/html/Smith_CODA-Prompt_COntinual_Decomposed_Attention-Based_Prompting_for_Rehearsal-Free_Continual_Learning_CVPR_2023_paper.html)\.
- Song et al\. \(2023\)Weixi Song, Zuchao Li, Lefei Zhang, Hai Zhao, and Bo Du\.Sparse is Enough in Fine\-tuning Pre\-trained Large Language Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2023\.[https://arxiv\.org/abs/2312\.11875](https://arxiv.org/abs/2312.11875)\.
- Song et al\. \(2024\)Yurun Song, Junchen Zhao, Ian G\. Harris, and Sangeetha Abdu Jyothi\.ShareLoRA: Parameter Efficient and Robust Large Language Model Fine\-tuning via Shared Low\-Rank Adaptation\.*arXiv preprint arXiv:2406\.10785*, 2024\.[https://arxiv\.org/abs/2406\.10785](https://arxiv.org/abs/2406.10785)\.
- Spivak \(2022\)David I\. Spivak\.Learners’ languages\.In*Proceedings of the Fourth International Conference on Applied Category Theory \(ACT 2021\)*, volume 372 of*Electronic Proceedings in Theoretical Computer Science*, pages 14–28, 2022\.doi:10\.4204/EPTCS\.372\.2\.
- Srebro et al\. \(2004\)Nathan Srebro, Jason Rennie, and Tommi Jaakkola\.Maximum\-margin matrix factorization\.In*Advances in Neural Information Processing Systems 17 \(NIPS 2004\)*, pages 1329–1336, 2004\.[https://papers\.nips\.cc/paper\_files/paper/2004/hash/e0688d13958a19e087e123148555e4b4\-Abstract\.html](https://papers.nips.cc/paper_files/paper/2004/hash/e0688d13958a19e087e123148555e4b4-Abstract.html)\.
- Steitz and Roth \(2024\)Jan\-Martin O\. Steitz and Stefan Roth\.Adapters Strike Back\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024\.[https://arxiv\.org/abs/2406\.06820](https://arxiv.org/abs/2406.06820)\.
- Stoehr et al\. \(2024\)Niklas Stoehr, Kevin Du, Vésteinn Snæbjarnarson, Robert West, Ryan Cotterell, and Aaron Schein\.Activation scaling for steering and interpreting language models\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 8189–8200, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-emnlp\.479\.[https://aclanthology\.org/2024\.findings\-emnlp\.479/](https://aclanthology.org/2024.findings-emnlp.479/)\.
- Stoica et al\. \(2024\)George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman\.Model merging with SVD to tie the Knots\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2410\.19735](https://arxiv.org/abs/2410.19735)\.
- Su et al\. \(2026\)Barbara Su, Fangshuo Liao, and Anastasios Kyrillidis\.AdaPaD: Adaptive Parallel Deflation for PEFT with Self\-Correcting Rank Discovery\.*arXiv preprint arXiv:2605\.10741*, 2026\.[https://arxiv\.org/abs/2605\.10741](https://arxiv.org/abs/2605.10741)\.
- Su et al\. \(2025\)DiJia Su, Andrew Gu, Jane Xu, Yuandong Tian, and Jiawei Zhao\.GaLore 2: Large\-Scale LLM Pre\-Training by Gradient Low\-Rank Projection\.*arXiv preprint arXiv:2504\.20437*, 2025\.[https://arxiv\.org/abs/2504\.20437](https://arxiv.org/abs/2504.20437)\.
- Su et al\. \(2024\)Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu\.RoFormer: Enhanced transformer with rotary position embedding\.*Neurocomputing*, 568:127063, 2024\.doi:10\.1016/j\.neucom\.2023\.127063\.
- Su et al\. \(2021\)Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi\-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou\.On Transferability of Prompt Tuning for Natural Language Processing\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2021\.[https://aclanthology\.org/2022\.naacl\-main\.290/](https://aclanthology.org/2022.naacl-main.290/)\.
- Sun et al\. \(2026\)Mengyang Sun, Maochuan Dou, Tao Feng, Dan Zhang, Yihao Wang, Junpeng Liu, Yifan Zhu, and Jie Tang\.RotMoLE: Enhancing Mixture of Low\-Rank Experts through Rotational Gating Mechanism\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.[https://arxiv\.org/abs/2605\.25565](https://arxiv.org/abs/2605.25565)\.
- Sun et al\. \(2025\)Qi Sun, Edoardo Cetin, and Yujin Tang\.Transformer\-Squared: Self\-adaptive LLMs\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2501\.06252](https://arxiv.org/abs/2501.06252)\.
- Sun et al\. \(2022\)Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu\.Black\-Box Tuning for Language\-Model\-as\-a\-Service\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2022\.[https://proceedings\.mlr\.press/v162/sun22e\.html](https://proceedings.mlr.press/v162/sun22e.html)\.
- Sun et al\. \(2024\)Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding\.Improving LoRA in Privacy\-preserving Federated Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2403\.12313](https://arxiv.org/abs/2403.12313)\.
- Sung et al\. \(2021\)Yi\-Lin Sung, Varun Nair, and Colin Raffel\.Training Neural Networks with Fixed Sparse Masks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2021\.[https://arxiv\.org/abs/2111\.09839](https://arxiv.org/abs/2111.09839)\.
- Sung et al\. \(2022\)Yi\-Lin Sung, Jaemin Cho, and Mohit Bansal\.LST: Ladder Side\-Tuning for Parameter and Memory Efficient Transfer Learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.[https://arxiv\.org/abs/2206\.06522](https://arxiv.org/abs/2206.06522)\.
- Sussmann \(1992\)Héctor J\. Sussmann\.Uniqueness of the weights for minimal feedforward nets with a given input\-output map\.*Neural Networks*, 5\(4\):589–593, 1992\.doi:10\.1016/S0893\-6080\(05\)80037\-1\.
- Takatsu \(2011\)Asuka Takatsu\.Wasserstein geometry of Gaussian measures\.*Osaka Journal of Mathematics*, 48\(4\):1005–1026, 2011\.[https://projecteuclid\.org/journals/osaka\-journal\-of\-mathematics/volume\-48/issue\-4/Wasserstein\-geometry\-of\-Gaussian\-measures/ojm/1326291215\.full](https://projecteuclid.org/journals/osaka-journal-of-mathematics/volume-48/issue-4/Wasserstein-geometry-of-Gaussian-measures/ojm/1326291215.full)\.
- Tan et al\. \(2024a\)Wenxuan Tan, Nicholas Roberts, Tzu\-Heng Huang, Jitian Zhao, John Cooper, Samuel Guo, Chengyu Duan, and Frederic Sala\.MoRe Fine\-Tuning with 10x Fewer Parameters\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024a\.[https://arxiv\.org/abs/2408\.17383](https://arxiv.org/abs/2408.17383)\.
- Tan et al\. \(2024b\)Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang\.OminiControl: Minimal and Universal Control for Diffusion Transformer\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2024b\.[https://arxiv\.org/abs/2411\.15098](https://arxiv.org/abs/2411.15098)\.
- Tanaka and Kunin \(2021\)Hidenori Tanaka and Daniel Kunin\.Noether’s learning dynamics: Role of symmetry breaking in neural networks\.In*Advances in Neural Information Processing Systems*, volume 34, 2021\.[https://arxiv\.org/abs/2105\.02716](https://arxiv.org/abs/2105.02716)\.
- Tang et al\. \(2025\)Pengwei Tang, Xiaolin Hu, Yong Liu, Lizhong Ding, Dongjie Zhang, Xing Wu, and Debing Zhang\.Put the Space of LoRA Initialization to the Extreme to Preserve Pre\-trained Knowledge\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2025\.[https://ojs\.aaai\.org/index\.php/AAAI/article/view/40608](https://ojs.aaai.org/index.php/AAAI/article/view/40608)\.
- Tang and Yang \(2026\)Yixuan Tang and Yi Yang\.Crowded in B\-Space: Calibrating Shared Directions for LoRA Merging\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026\.[https://arxiv\.org/abs/2604\.16826](https://arxiv.org/abs/2604.16826)\.
- Tarmoun et al\. \(2021\)Salma Tarmoun, Guilherme França, Benjamin D\. Haeffele, and René Vidal\.Understanding the dynamics of gradient flow in overparameterized linear models\.In Marina Meila and Tong Zhang, editors,*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, pages 10153–10161\. PMLR, 2021\.[https://proceedings\.mlr\.press/v139/tarmoun21a\.html](https://proceedings.mlr.press/v139/tarmoun21a.html)\.
- Tastan et al\. \(2025\)Nurbek Tastan, Stefanos Laskaridis, Martin Takac, Karthik Nandakumar, and Samuel Horvath\.LoFT: Low\-Rank Adaptation That Behaves Like Full Fine\-Tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2505\.21289](https://arxiv.org/abs/2505.21289)\.
- team \(2025\)Hugging Face PEFT team\.Trainable Tokens\.Hugging Face PEFT \(software, no paper\), 2025\.[https://huggingface\.co/docs/peft/main/en/package\_reference/trainable\_tokens](https://huggingface.co/docs/peft/main/en/package_reference/trainable_tokens)\.
- Team OLMo et al\. \(2024\)Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, et al\.2 OLMo 2 furious\.arXiv:2501\.00656, 2024\.[https://arxiv\.org/abs/2501\.00656](https://arxiv.org/abs/2501.00656)\.
- Tian et al\. \(2024\)Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu\.HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine\-Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2404\.19245](https://arxiv.org/abs/2404.19245)\.
- Tong et al\. \(2021\)Tian Tong, Cong Ma, and Yuejie Chi\.Accelerating Ill\-Conditioned Low\-Rank Matrix Estimation via Scaled Gradient Descent\.*Journal of Machine Learning Research*, 22\(150\):1–63, 2021\.[https://jmlr\.org/papers/v22/20\-1067\.html](https://jmlr.org/papers/v22/20-1067.html)\.
- Torroba\-Hennigen et al\. \(2025\)Lucas Torroba\-Hennigen, Hunter Lang, Han Guo, and Yoon Kim\.On the Duality between Gradient Transformations and Adapters\.*arXiv preprint arXiv:2502\.13811*, 2025\.[https://arxiv\.org/abs/2502\.13811](https://arxiv.org/abs/2502.13811)\.
- Uschmajew and Vandereycken \(2013\)André Uschmajew and Bart Vandereycken\.The geometry of algorithms using hierarchical tensors\.*Linear Algebra and its Applications*, 439\(1\):133–166, 2013\.doi:10\.1016/j\.laa\.2013\.03\.016\.
- Vakil \(2025\)Ravi Vakil\.*The Rising Sea: Foundations of Algebraic Geometry*\.Princeton University Press, Princeton, NJ, 2025\.ISBN 978\-0\-691\-26866\-8\.
- Valipour et al\. \(2022\)Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi\.DyLoRA: Parameter Efficient Tuning of Pre\-trained Models using Dynamic Search\-Free Low\-Rank Adaptation\.In*Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics \(EACL\)*, 2022\.[https://arxiv\.org/abs/2210\.07558](https://arxiv.org/abs/2210.07558)\.
- van den Dries \(1998\)Lou van den Dries\.*Tame Topology and O\-minimal Structures*, volume 248 of*London Mathematical Society Lecture Note Series*\.Cambridge University Press, Cambridge, 1998\.doi:10\.1017/CBO9780511525919\.
- Van Loan \(2000\)Charles F\. Van Loan\.The ubiquitous Kronecker product\.*Journal of Computational and Applied Mathematics*, 123\(1–2\):85–100, 2000\.doi:10\.1016/S0377\-0427\(00\)00393\-9\.
- Van Loan and Pitsianis \(1993\)Charles F\. Van Loan and Nikos Pitsianis\.Approximation with Kronecker products\.In Marc S\. Moonen, Gene H\. Golub, and Bart L\. R\. De Moor, editors,*Linear Algebra for Large Scale and Real\-Time Applications*, volume 232 of*NATO ASI Series E: Applied Sciences*, pages 293–314\. Kluwer Academic Publishers, Dordrecht, 1993\.doi:10\.1007/978\-94\-015\-8196\-7\_17\.
- Vandereycken \(2013\)Bart Vandereycken\.Low\-rank matrix completion by Riemannian optimization\.*SIAM Journal on Optimization*, 23\(2\):1214–1236, 2013\.doi:10\.1137/110845768\.
- Virtanen et al\. \(2020\)Pauli Virtanen, Ralf Gommers, Travis E\. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J\. van der Walt, Matthew Brett, Joshua Wilson, K\. Jarrod Millman, Nikolay Mayorov, Andrew R\. J\. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W\. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E\. A\. Quintero, Charles R\. Harris, Anne M\. Archibald, Antônio H\. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1\.0 Contributors\.SciPy 1\.0: fundamental algorithms for scientific computing in Python\.*Nature Methods*, 17\(3\):261–272, 2020\.doi:10\.1038/s41592\-019\-0686\-2\.
- Vu et al\. \(2021\)Tu Vu, Brian Lester, Noah Constant, Rami Al\-Rfou, and Daniel Cer\.SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2021\.[https://aclanthology\.org/2022\.acl\-long\.346/](https://aclanthology.org/2022.acl-long.346/)\.
- Wan et al\. \(2025\)Guoan Wan, Tianyu Chen, Fangzheng Feng, Haoyi Zhou, and Runhua Xu\.FRoD: Full\-Rank Efficient Fine\-Tuning with Rotational Degrees for Fast Convergence\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2025\.[https://arxiv\.org/abs/2512\.23485](https://arxiv.org/abs/2512.23485)\.
- Wang et al\. \(2024a\)Fan Wang, Juyong Jiang, Chansung Park, Sunghun Kim, and Jing Tang\.KaSA: Knowledge\-Aware Singular\-Value Adaptation of Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.[https://arxiv\.org/abs/2412\.06071](https://arxiv.org/abs/2412.06071)\.
- Wang et al\. \(2024b\)Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen\.MiLoRA: Harnessing Minor Singular Components for Parameter\-Efficient LLM Finetuning\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2024b\.[https://aclanthology\.org/2025\.naacl\-long\.248/](https://aclanthology.org/2025.naacl-long.248/)\.
- Wang et al\. \(2024c\)Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun\.LoRA\-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024c\.[https://aclanthology\.org/2024\.acl\-long\.695/](https://aclanthology.org/2024.acl-long.695/)\.
- Wang et al\. \(2025a\)Haonan Wang, Brian Chen, Siquan Li, Xinhe Liang, Hwee Kuan Lee, Kenji Kawaguchi, and Tianyang Hu\.PrefixMemory\-Tuning: Modernizing Prefix\-Tuning by Decoupling the Prefix from Attention\.In*International Conference on Learning Representations \(ICLR\)*, 2025a\.[https://arxiv\.org/abs/2506\.13674](https://arxiv.org/abs/2506.13674)\.
- Wang et al\. \(2023a\)Maolin Wang, Yu Pan, Zenglin Xu, Guangxi Li, Xiangli Yang, Danilo Mandic, and Andrzej Cichocki\.Tensor Networks Meet Neural Networks: A Survey and Future Perspectives\.*arXiv preprint arXiv:2302\.09019*, 2023a\.[https://arxiv\.org/abs/2302\.09019](https://arxiv.org/abs/2302.09019)\.
- Wang et al\. \(2020\)Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang, and Ming Zhou\.K\-Adapter: Infusing Knowledge into Pre\-Trained Models with Adapters\.In*Findings of the Association for Computational Linguistics: ACL 2021*, 2020\.[https://arxiv\.org/abs/2002\.01808](https://arxiv.org/abs/2002.01808)\.
- Wang et al\. \(2024d\)Runqian Wang, Soumya Ghosh, David Cox, Diego Antognini, Aude Oliva, Rogerio Feris, and Leonid Karlinsky\.Trans\-LoRA: towards data\-free Transferable Parameter Efficient Finetuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024d\.[https://proceedings\.neurips\.cc/paper\_files/paper/2024/hash/708fdc7911f11585ee7161518e509ae6\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2024/hash/708fdc7911f11585ee7161518e509ae6-Abstract-Conference.html)\.
- Wang et al\. \(2025b\)Shangshang Wang, Julian Asilis, Ömer Faruk Akgül, Enes Burak Bilgin, Ollie Liu, and Willie Neiswanger\.Tina: Tiny Reasoning Models via LoRA\.In*International Conference on Learning Representations \(ICLR\)*, 2025b\.[https://arxiv\.org/abs/2504\.15777](https://arxiv.org/abs/2504.15777)\.
- Wang et al\. \(2024e\)Shaowen Wang, Linxi Yu, and Jian Li\.LoRA\-GA: Low\-Rank Adaptation with Gradient Approximation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024e\.[https://arxiv\.org/abs/2407\.05000](https://arxiv.org/abs/2407.05000)\.
- Wang et al\. \(2024f\)Sheng Wang, Liheng Chen, Pengan Chen, Jingwei Dong, Boyang Xue, Jiyue Jiang, Lingpeng Kong, and Chuan Wu\.MoS: Unleashing Parameter Efficiency of Low\-Rank Adaptation with Mixture of Shards\.In*International Conference on Learning Representations \(ICLR\)*, 2024f\.[https://arxiv\.org/abs/2410\.00938](https://arxiv.org/abs/2410.00938)\.
- Wang et al\. \(2024g\)Sheng Wang, Boyang Xue, Jiacheng Ye, Jiyue Jiang, Liheng Chen, Lingpeng Kong, and Chuan Wu\.PRoLoRA: Partial Rotation Empowers More Parameter\-Efficient LoRA\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024g\.[https://aclanthology\.org/2024\.acl\-long\.156/](https://aclanthology.org/2024.acl-long.156/)\.
- Wang and Zhang \(2026\)Shihao Wang and Xueru Zhang\.PRISM: Gauge\-Invariant Tangent\-Space Differentially Private LoRA\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2606\.00944](https://arxiv.org/abs/2606.00944)\.
- Wang et al\. \(2026a\)Shuaidi Wang, Zhan Zhuang, Ruping Huang, and Yu Zhang\.NaRA: Noise\-Aware LoRA for Parameter\-Efficient Fine\-Tuning of Diffusion LLMs\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026a\.[https://arxiv\.org/abs/2605\.29716](https://arxiv.org/abs/2605.29716)\.
- Wang et al\. \(2023b\)Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang\.Orthogonal Subspace Learning for Language Model Continual Learning\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, 2023b\.[https://aclanthology\.org/2023\.findings\-emnlp\.715/](https://aclanthology.org/2023.findings-emnlp.715/)\.
- Wang et al\. \(2026b\)Xiequn Wang, Zhan Zhuang, Shengda Luo, and Yu Zhang\.MoSA: Mosaic shared adaptation of large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2026b\.[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/dd37fdb24a4e1cfa3ed5c247217a7394\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/dd37fdb24a4e1cfa3ed5c247217a7394-Abstract-Conference.html)\.
- Wang et al\. \(2025c\)Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang\.SVD\-LLM: Truncation\-aware singular value decomposition for large language model compression\.In*International Conference on Learning Representations \(ICLR\)*, 2025c\.[https://arxiv\.org/abs/2403\.07378](https://arxiv.org/abs/2403.07378)\.
- Wang et al\. \(2025d\)Xinyu Wang, Jonas M\. Kübler, Kailash Budhathoki, Yida Wang, and Matthäus Kleindessner\.Block\-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025d\.[https://arxiv\.org/abs/2510\.23346](https://arxiv.org/abs/2510.23346)\.
- Wang et al\. \(2022a\)Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao\.AdaMix: Mixture\-of\-Adaptations for Parameter\-efficient Model Tuning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2022a\.[https://arxiv\.org/abs/2205\.12410](https://arxiv.org/abs/2205.12410)\.
- Wang et al\. \(2024h\)Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, and Hao Wang\.BLoB: Bayesian Low\-Rank Adaptation by Backpropagation for Large Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024h\.[https://arxiv\.org/abs/2406\.11675](https://arxiv.org/abs/2406.11675)\.
- Wang et al\. \(2023c\)Yihan Wang, Jatin Chauhan, Wei Wang, and Cho\-Jui Hsieh\.Universality and Limitations of Prompt Tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023c\.[https://arxiv\.org/abs/2305\.18787](https://arxiv.org/abs/2305.18787)\.
- Wang et al\. \(2025e\)Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, and Hongxia Yang\.Model Merging Scaling Laws in Large Language Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025e\.[https://arxiv\.org/abs/2509\.24244](https://arxiv.org/abs/2509.24244)\.
- Wang et al\. \(2023d\)Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim\.Multitask Prompt Tuning Enables Parameter\-Efficient Transfer Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2023d\.[https://arxiv\.org/abs/2303\.02861](https://arxiv.org/abs/2303.02861)\.
- Wang et al\. \(2024i\)Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan\.LoRA\-Pro: Are Low\-Rank Adapters Properly Optimized?In*International Conference on Learning Representations \(ICLR\)*, 2024i\.[https://arxiv\.org/abs/2407\.18242](https://arxiv.org/abs/2407.18242)\.
- Wang et al\. \(2026c\)Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan\.Taming Momentum: Rethinking Optimizer States Through Low\-Rank Approximation\.In*International Conference on Learning Representations \(ICLR\)*, 2026c\.[https://arxiv\.org/abs/2602\.24283](https://arxiv.org/abs/2602.24283)\.
- Wang et al\. \(2021\)Zifeng Wang, Zizhao Zhang, Chen\-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister\.Learning to Prompt for Continual Learning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2021\.[https://arxiv\.org/abs/2112\.08654](https://arxiv.org/abs/2112.08654)\.
- Wang et al\. \(2022b\)Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen\-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister\.DualPrompt: Complementary Prompting for Rehearsal\-free Continual Learning\.In*European Conference on Computer Vision \(ECCV\)*, 2022b\.[https://arxiv\.org/abs/2204\.04799](https://arxiv.org/abs/2204.04799)\.
- Wang et al\. \(2024j\)Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Y\. Wu\.Let the Expert Stick to His Last: Expert\-Specialized Fine\-Tuning for Sparse Architectural Large Language Models\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2024j\.[https://aclanthology\.org/2024\.emnlp\-main\.46/](https://aclanthology.org/2024.emnlp-main.46/)\.
- Wang et al\. \(2024k\)Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li\.FLoRA: Federated Fine\-Tuning Large Language Models with Heterogeneous Low\-Rank Adaptations\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024k\.[https://arxiv\.org/abs/2409\.05976](https://arxiv.org/abs/2409.05976)\.
- Wen and Chaudhuri \(2023\)Yeming Wen and Swarat Chaudhuri\.Batched Low\-Rank Adaptation of Foundation Models\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2312\.05677](https://arxiv.org/abs/2312.05677)\.
- Weyl \(1939\)Hermann Weyl\.*The Classical Groups: Their Invariants and Representations*, volume 1 of*Princeton Mathematical Series*\.Princeton University Press, Princeton, NJ, 1939\.
- Wilkie \(1996\)A\. J\. Wilkie\.Model completeness results for expansions of the ordered field of real numbers by restricted Pfaffian functions and the exponential function\.*Journal of the American Mathematical Society*, 9\(4\):1051–1094, 1996\.doi:10\.1090/S0894\-0347\-96\-00216\-0\.
- Wilson and Zanasi \(2021\)Paul Wilson and Fabio Zanasi\.Reverse derivative ascent: A categorical approach to learning Boolean circuits\.In*Proceedings of the 3rd Annual International Applied Category Theory Conference 2020 \(ACT 2020\)*, volume 333 of*Electronic Proceedings in Theoretical Computer Science*, pages 247–260, 2021\.doi:10\.4204/EPTCS\.333\.17\.
- Wortsman et al\. \(2021\)Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo\-Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt\.Robust fine\-tuning of zero\-shot models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2021\.[https://arxiv\.org/abs/2109\.01903](https://arxiv.org/abs/2109.01903)\.
- Wortsman et al\. \(2022\)Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo\-Lopes, Ari S\. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt\.Model soups: averaging weights of multiple fine\-tuned models improves accuracy without increasing inference time\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2022\.[https://arxiv\.org/abs/2203\.05482](https://arxiv.org/abs/2203.05482)\.
- Wu et al\. \(2025a\)Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang\.Efficient Orthogonal Fine\-Tuning with Principal Subspace Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2025a\.[https://arxiv\.org/abs/2505\.11235](https://arxiv.org/abs/2505.11235)\.
- Wu et al\. \(2024a\)Muling Wu, Wenhao Liu, Xiaohua Wang, Tianlong Li, Changze Lv, Zixuan Ling, Jianhao Zhu, Cenyuan Zhang, Xiaoqing Zheng, and Xuanjing Huang\.Advancing Parameter Efficiency in Fine\-tuning via Representation Editing\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2024a\.[https://aclanthology\.org/2024\.acl\-long\.726/](https://aclanthology.org/2024.acl-long.726/)\.
- Wu et al\. \(2024b\)Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong\.Mixture\-of\-Subspaces in Low\-Rank Adaptation\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2024b\.[https://aclanthology\.org/2024\.emnlp\-main\.450/](https://aclanthology.org/2024.emnlp-main.450/)\.
- Wu et al\. \(2024c\)Xun Wu, Shaohan Huang, and Furu Wei\.Mixture of LoRA Experts\.In*International Conference on Learning Representations \(ICLR\)*, 2024c\.[https://arxiv\.org/abs/2404\.13628](https://arxiv.org/abs/2404.13628)\.
- Wu et al\. \(2025b\)Yebo Wu, Jingguang Li, Zhijiang Guo, and Li Li\.Elastic Mixture of Rank\-Wise Experts for Knowledge Reuse in Federated Fine\-Tuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025b\.[https://arxiv\.org/abs/2512\.00902](https://arxiv.org/abs/2512.00902)\.
- Wu et al\. \(2025c\)Yichen Wu, Hongming Piao, Long\-Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei\.SD\-LoRA: Scalable Decoupled Low\-Rank Adaptation for Class Incremental Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2025c\.[https://arxiv\.org/abs/2501\.13198](https://arxiv.org/abs/2501.13198)\.
- Wu et al\. \(2025d\)Yize Wu, Ke Gao, Ling Li, and Yanjun Wu\.Stable\-LoRA: Stabilizing Feature Learning of Low\-Rank Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2025d\.[https://arxiv\.org/abs/2603\.05204](https://arxiv.org/abs/2603.05204)\.
- Wu et al\. \(2024d\)Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D\. Manning, and Christopher Potts\.ReFT: Representation Finetuning for Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024d\.[https://arxiv\.org/abs/2404\.03592](https://arxiv.org/abs/2404.03592)\.
- Wu et al\. \(2022\)Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yuxiao Dong, V\. G\. Vinod Vydiswaran, and Hao Ma\.IDPG: An Instance\-Dependent Prompt Generation Method\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2022\.[https://aclanthology\.org/2022\.naacl\-main\.403/](https://aclanthology.org/2022.naacl-main.403/)\.
- Xia et al\. \(2024\)Wenhan Xia, Chengwei Qin, and Elad Hazan\.Chain of LoRA: Efficient Fine\-tuning of Language Models via Residual Learning\.*arXiv preprint arXiv:2401\.04151*, 2024\.[https://arxiv\.org/abs/2401\.04151](https://arxiv.org/abs/2401.04151)\.
- Xiao et al\. \(2023\)Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han\.SmoothQuant: Accurate and efficient post\-training quantization for large language models\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 38087–38099\. PMLR, 2023\.[https://proceedings\.mlr\.press/v202/xiao23c\.html](https://proceedings.mlr.press/v202/xiao23c.html)\.
- Xiao et al\. \(2026\)Xi Xiao, Chenrui Ma, Yunbei Zhang, Chen Liu, Zhuxuanzi Wang, Yanshu Li, Lin Zhao, Guosheng Hu, Tianyang Wang, and Hao Xu\.Not All Directions Matter: Towards Structured and Task\-Aware Low\-Rank Model Adaptation\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2026\.[https://arxiv\.org/abs/2603\.14228](https://arxiv.org/abs/2603.14228)\.
- Xie et al\. \(2023\)Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li\.DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter\-Efficient Fine\-Tuning\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023\.[https://arxiv\.org/abs/2304\.06648](https://arxiv.org/abs/2304.06648)\.
- Xin et al\. \(2024\)Yi Xin, Jianjiang Yang, Siqi Luo, Yuntao Du, Qi Qin, Kangrui Cen, Yangfan He, Zhiwei Zhang, Bin Fu, Xiaokang Yang, Guangtao Zhai, Ming\-Hsuan Yang, and Xiaohong Liu\.Parameter\-Efficient Fine\-Tuning for Pre\-Trained Vision Models: A Survey and Benchmark\.*International Journal of Computer Vision*, 2024\.[https://arxiv\.org/abs/2402\.02242](https://arxiv.org/abs/2402.02242)\.
- Xu et al\. \(2023a\)Lingling Xu, Haoran Xie, Si\-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang\.Parameter\-Efficient Fine\-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment\.*IEEE TPAMI*, 2023a\.[https://arxiv\.org/abs/2312\.12148](https://arxiv.org/abs/2312.12148)\.
- Xu et al\. \(2021\)Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang\.Raise a Child in Large Language Model: Towards Effective and Generalizable Fine\-tuning\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2021\.[https://aclanthology\.org/2021\.emnlp\-main\.749/](https://aclanthology.org/2021.emnlp-main.749/)\.
- Xu et al\. \(2023b\)Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian\.QA\-LoRA: Quantization\-Aware Low\-Rank Adaptation of Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2023b\.[https://arxiv\.org/abs/2309\.14717](https://arxiv.org/abs/2309.14717)\.
- Xu et al\. \(2025\)Ziqing Xu, Hancheng Min, Lachlan Ewen MacDonald, Jinqi Luo, Salma Tarmoun, Enrique Mallada, and Rene Vidal\.Understanding the Learning Dynamics of LoRA: A Gradient Flow Perspective on Low\-Rank Adaptation in Matrix Factorization\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, 2025\.[https://arxiv\.org/abs/2503\.06982](https://arxiv.org/abs/2503.06982)\.
- Xue et al\. \(2026\)Bowen Xue, Zihan Min, Xingyang Li, Zhekai Zhang, Haocheng Xi, Lvmin Zhang, Maneesh Agrawala, Jun\-Yan Zhu, Song Han, Yujun Lin, and Muyang Li\.FourTune: Towards Fully 4\-Bit Efficient Post\-Training for Diffusion Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026\.[https://arxiv\.org/abs/2607\.05711](https://arxiv.org/abs/2607.05711)\.
- Yadav et al\. \(2023\)Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal\.TIES\-Merging: Resolving Interference When Merging Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.[https://arxiv\.org/abs/2306\.01708](https://arxiv.org/abs/2306.01708)\.
- Yadav et al\. \(2024\)Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai\.What Matters for Model Merging at Scale?*TMLR*, 2024\.[https://arxiv\.org/abs/2410\.03617](https://arxiv.org/abs/2410.03617)\.
- Yamabe \(1950\)Hidehiko Yamabe\.On an arcwise connected subgroup of a Lie group\.*Osaka Mathematical Journal*, 2\(1\):13–14, 1950\.[https://projecteuclid\.org/euclid\.ojm/1200685927](https://projecteuclid.org/euclid.ojm/1200685927)\.
- Yang et al\. \(2023a\)Adam X\. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison\.Bayesian Low\-rank Adaptation for Large Language Models\.In*International Conference on Learning Representations \(ICLR\)*, 2023a\.[https://proceedings\.iclr\.cc/paper\_files/paper/2024/hash/07c256a163a7559186ec1c71e95b9ec9\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2024/hash/07c256a163a7559186ec1c71e95b9ec9-Abstract-Conference.html)\.
- Yang et al\. \(2023b\)Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao\.AdaMerging: Adaptive Model Merging for Multi\-Task Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2023b\.[https://arxiv\.org/abs/2310\.02575](https://arxiv.org/abs/2310.02575)\.
- Yang et al\. \(2024a\)Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao\.Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities\.*ACM Computing Surveys*, 2024a\.[https://arxiv\.org/abs/2408\.07666](https://arxiv.org/abs/2408.07666)\.
- Yang and Hu \(2021\)Greg Yang and Edward J\. Hu\.Feature learning in infinite\-width neural networks\.In*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*\. PMLR, 2021\.[https://arxiv\.org/abs/2011\.14522](https://arxiv.org/abs/2011.14522)\.
- Yang et al\. \(2026a\)Sihan Yang, Kexuan Shi, and Weiyang Liu\.Orthogonal Model Merging\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026a\.[https://arxiv\.org/abs/2602\.05943](https://arxiv.org/abs/2602.05943)\.
- Yang et al\. \(2023c\)Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li\.AIM: Adapting Image Models for Efficient Video Action Recognition\.In*International Conference on Learning Representations \(ICLR\)*, 2023c\.[https://arxiv\.org/abs/2302\.03024](https://arxiv.org/abs/2302.03024)\.
- Yang et al\. \(2026b\)Xiaomeng Yang, Yanyu Li, Gordon Guocheng Qian, Ivan Skorokhodov, Viacheslav Ivanov, Avalon Vinella, Xuan Zhang, Yanzhi Wang, Sergey Tulyakov, and Anil Kag\.Prompt2Effect: Training\-Free Image\-to\-Video Model Specialization via LoRA Generation\.In*European Conference on Computer Vision \(ECCV\)*, 2026b\.[https://arxiv\.org/abs/2606\.13971](https://arxiv.org/abs/2606.13971)\.
- Yang et al\. \(2024b\)Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen\.S2FT: Efficient, Scalable and Generalizable LLM Fine\-tuning by Structured Sparsity\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024b\.[https://arxiv\.org/abs/2412\.06289](https://arxiv.org/abs/2412.06289)\.
- Yang et al\. \(2024c\)Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem\.CorDA: Context\-Oriented Decomposition Adaptation of Large Language Models for Task\-Aware Parameter\-Efficient Fine\-tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024c\.[https://arxiv\.org/abs/2406\.05223](https://arxiv.org/abs/2406.05223)\.
- Yang et al\. \(2024d\)Yifan Yang, Jiajun Zhou, Ngai Wong, and Zheng Zhang\.LoRETTA: Low\-Rank Economic Tensor\-Train Adaptation for Ultra\-Low\-Parameter Fine\-Tuning of Large Language Models\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2024d\.[https://arxiv\.org/abs/2402\.11417](https://arxiv.org/abs/2402.11417)\.
- Ye et al\. \(2023\)Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang\.IP\-Adapter: Text Compatible Image Prompt Adapter for Text\-to\-Image Diffusion Models\.*arXiv preprint arXiv:2308\.06721*, 2023\.[https://arxiv\.org/abs/2308\.06721](https://arxiv.org/abs/2308.06721)\.
- Ye et al\. \(2025\)Rongguang Ye, Ming Tang, and Edith C\. H\. Ngai\.On\-the\-Fly Adaptation to Quantization: Configuration\-Aware LoRA for Efficient Fine\-Tuning of Quantized LLMs\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2509\.25214](https://arxiv.org/abs/2509.25214)\.
- Yeh et al\. \(2023\)Shih\-Ying Yeh, Yu\-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, and Yanmin Gong\.Navigating Text\-To\-Image Customization: From LyCORIS Fine\-Tuning to Model Evaluation\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2309\.14859](https://arxiv.org/abs/2309.14859)\.
- Yen et al\. \(2024\)Jui\-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S\. Dhillon, Cho\-Jui Hsieh, and Sanjiv Kumar\.LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.[https://arxiv\.org/abs/2410\.20625](https://arxiv.org/abs/2410.20625)\.
- Yin et al\. \(2024a\)Dongshuo Yin, Leiyi Hu, Bin Li, Youqun Zhang, and Xue Yang\.5%\>100%: Breaking Performance Shackles of Full Fine\-Tuning on Visual Recognition Tasks\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2024a\.[https://arxiv\.org/abs/2408\.08345](https://arxiv.org/abs/2408.08345)\.
- Yin et al\. \(2024b\)Fangcong Yin, Xi Ye, and Greg Durrett\.LoFiT: Localized Fine\-tuning on LLM Representations\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024b\.[https://proceedings\.neurips\.cc/paper\_files/paper/2024/hash/122ea6470232ee5e79a2649243348005\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2024/hash/122ea6470232ee5e79a2649243348005-Abstract-Conference.html)\.
- Yin et al\. \(2023\)Junjie Yin, Jiahao Dong, Yingheng Wang, Christopher De Sa, and Volodymyr Kuleshov\.ModuLoRA: Finetuning 2\-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers\.*TMLR*, 2023\.[https://arxiv\.org/abs/2309\.16119](https://arxiv.org/abs/2309.16119)\.
- Yin et al\. \(1988\)Y\. Q\. Yin, Z\. D\. Bai, and P\. R\. Krishnaiah\.On the limit of the largest eigenvalue of the large dimensional sample covariance matrix\.*Probability Theory and Related Fields*, 78\(4\):509–521, 1988\.doi:10\.1007/BF00353874\.
- Yu et al\. \(2023\)Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li\.Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2023\.[https://arxiv\.org/abs/2311\.03099](https://arxiv.org/abs/2311.03099)\.
- Yu et al\. \(2025\)Xin Yu, Yujia Wang, Jinghui Chen, and Lingzhou Xue\.AltLoRA: Towards Better Gradient Approximation in Low\-Rank Adaptation with Alternating Projections\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2505\.12455](https://arxiv.org/abs/2505.12455)\.
- Yuan et al\. \(2024\)Shen Yuan, Haotian Liu, and Hongteng Xu\.Bridging The Gap between Low\-rank and Orthogonal Adaptation via Householder Reflection Adaptation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.[https://arxiv\.org/abs/2405\.17484](https://arxiv.org/abs/2405.17484)\.
- Zadouri et al\. \(2023\)Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker\.Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2309\.05444](https://arxiv.org/abs/2309.05444)\.
- Zeng and Lee \(2023\)Yuchen Zeng and Kangwook Lee\.The Expressive Power of Low\-Rank Adaptation\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2310\.17513](https://arxiv.org/abs/2310.17513)\.
- Zhang et al\. \(2021\)Aston Zhang, Yi Tay, Shuai Zhang, Alvin Chan, Anh Tuan Luu, Siu Cheung Hui, and Jie Fu\.Beyond Fully\-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.[https://arxiv\.org/abs/2102\.08597](https://arxiv.org/abs/2102.08597)\.
- Zhang et al\. \(2024a\)Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat\.When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method\.In*International Conference on Learning Representations \(ICLR\)*, 2024a\.[https://arxiv\.org/abs/2402\.17193](https://arxiv.org/abs/2402.17193)\.
- Zhang et al\. \(2026a\)Chengqian Zhang, Wei Zhu, and Kyumin Lee\.Hybrid\-LoRA: Bridging Full Fine\-Tuning and Low\-Rank Adaptation for Post\-Training\.*arXiv preprint arXiv:2605\.18822*, 2026a\.[https://arxiv\.org/abs/2605\.18822](https://arxiv.org/abs/2605.18822)\.
- Zhang and Pilanci \(2024a\)Fangzhao Zhang and Mert Pilanci\.Riemannian Preconditioned LoRA for Fine\-Tuning Foundation Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024a\.[https://arxiv\.org/abs/2402\.02347](https://arxiv.org/abs/2402.02347)\.
- Zhang and Pilanci \(2024b\)Fangzhao Zhang and Mert Pilanci\.Spectral Adapter: Fine\-Tuning in Spectral Space\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024b\.[https://arxiv\.org/abs/2405\.13952](https://arxiv.org/abs/2405.13952)\.
- Zhang et al\. \(2023a\)Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian\.IncreLoRA: Incremental Parameter Allocation Method for Parameter\-Efficient Fine\-tuning\.*arXiv preprint arXiv:2308\.12043*, 2023a\.[https://arxiv\.org/abs/2308\.12043](https://arxiv.org/abs/2308.12043)\.
- Zhang et al\. \(2026b\)Han\-Chen Zhang, Zi\-Hao Zhou, Mao\-Lin Luo, Shimin Di, Min\-Ling Zhang, and Tong Wei\.DC\-Merge: Improving Model Merging with Directional Consistency\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2026b\.[https://arxiv\.org/abs/2603\.06242](https://arxiv.org/abs/2603.06242)\.
- Zhang et al\. \(2026c\)Haoran Zhang, Dongjun Kim, Seohyeon Cha, and Haris Vikalo\.FedRot\-LoRA: Mitigating Rotational Misalignment in Federated LoRA\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026c\.[https://arxiv\.org/abs/2602\.23638](https://arxiv.org/abs/2602.23638)\.
- Zhang et al\. \(2019\)Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik\.Side\-Tuning: A Baseline for Network Adaptation via Additive Side Networks\.In*European Conference on Computer Vision \(ECCV\)*, 2019\.[https://arxiv\.org/abs/1912\.13503](https://arxiv.org/abs/1912.13503)\.
- Zhang et al\. \(2023b\)Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Yufan Zhou, Guoyin Wang, and Yiran Chen\.Towards Building the Federated GPT: Federated Instruction Tuning\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2023b\.[https://arxiv\.org/abs/2305\.05644](https://arxiv.org/abs/2305.05644)\.
- Zhang et al\. \(2026d\)Jiaying Zhang, Lei Shi, Jiguo Li, Jun Xu, Jiuchong Gao, Jinghua Hao, and Renqing He\.GeoRA: Geometry\-Aware Low\-Rank Adaptation for RLVR\.In*Proceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2026d\.[https://arxiv\.org/abs/2601\.09361](https://arxiv.org/abs/2601.09361)\.
- Zhang et al\. \(2023c\)Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He\.Composing Parameter\-Efficient Modules with Arithmetic Operations\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023c\.[https://arxiv\.org/abs/2306\.14870](https://arxiv.org/abs/2306.14870)\.
- Zhang et al\. \(2025a\)Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein\.LoRI: Reducing Cross\-Task Interference in Multi\-Task Low\-Rank Adaptation\.In*Conference on Language Modeling \(COLM\)*, 2025a\.[https://arxiv\.org/abs/2504\.07448](https://arxiv.org/abs/2504.07448)\.
- Zhang et al\. \(2023d\)Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li\.LoRA\-FA: Memory\-efficient Low\-rank Adaptation for Large Language Models Fine\-tuning\.*arXiv preprint arXiv:2308\.03303*, 2023d\.[https://arxiv\.org/abs/2308\.03303](https://arxiv.org/abs/2308.03303)\.
- Zhang et al\. \(2023e\)Lvmin Zhang, Anyi Rao, and Maneesh Agrawala\.Adding Conditional Control to Text\-to\-Image Diffusion Models\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2023e\.[https://arxiv\.org/abs/2302\.05543](https://arxiv.org/abs/2302.05543)\.
- Zhang et al\. \(2023f\)Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang\.LoRAPrune: Structured Pruning Meets Low\-Rank Parameter\-Efficient Fine\-Tuning\.In*Findings of the Association for Computational Linguistics: ACL 2024*, 2023f\.[https://arxiv\.org/abs/2305\.18403](https://arxiv.org/abs/2305.18403)\.
- Zhang et al\. \(2023g\)Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao\.AdaLoRA: Adaptive Budget Allocation for Parameter\-Efficient Fine\-Tuning\.In*International Conference on Learning Representations \(ICLR\)*, 2023g\.[https://arxiv\.org/abs/2303\.10512](https://arxiv.org/abs/2303.10512)\.
- Zhang et al\. \(2025b\)Qingyue Zhang, Chang Chu, Tianren Peng, Qi Li, Xiangyang Luo, Zhihao Jiang, and Shao\-Lun Huang\.LoRA\-DA: Data\-Aware Initialization for Low\-Rank Adaptation via Asymptotic Analysis\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025b\.[https://arxiv\.org/abs/2510\.24561](https://arxiv.org/abs/2510.24561)\.
- Zhang et al\. \(2022a\)Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li\.Tip\-Adapter: Training\-free Adaption of CLIP for Few\-shot Classification\.In*European Conference on Computer Vision \(ECCV\)*, 2022a\.[https://arxiv\.org/abs/2207\.09519](https://arxiv.org/abs/2207.09519)\.
- Zhang et al\. \(2023h\)Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao\.LLaMA\-Adapter: Efficient Fine\-tuning of Language Models with Zero\-init Attention\.In*International Conference on Learning Representations \(ICLR\)*, 2023h\.[https://arxiv\.org/abs/2303\.16199](https://arxiv.org/abs/2303.16199)\.
- Zhang et al\. \(2026e\)Ruijia Zhang, Jiacheng Zhu, Hanqing Zhu, and Laixi Shi\.Geometry\-Preserving Orthonormal Initialization for Low\-Rank Adaptation in RLVR\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2026e\.[https://arxiv\.org/abs/2606\.31813](https://arxiv.org/abs/2606.31813)\.
- Zhang et al\. \(2024b\)Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie\.AutoLoRA: Automatically Tuning Matrix Ranks in Low\-Rank Adaptation Based on Meta Learning\.In*Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\)*, 2024b\.[https://arxiv\.org/abs/2403\.09113](https://arxiv.org/abs/2403.09113)\.
- Zhang et al\. \(2023i\)Yihua Zhang, Hongkang Li, Yuguang Yao, Aochuan Chen, Shuai Zhang, Pin\-Yu Chen, Meng Wang, and Sijia Liu\.Visual prompting reimagined: The power of the Activation Prompts\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, 2023i\.[https://arxiv\.org/abs/2604\.06440](https://arxiv.org/abs/2604.06440)\.
- Zhang et al\. \(2024c\)Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin\-Yu Chen, Jason D\. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen\.Revisiting Zeroth\-Order Optimization for Memory\-Efficient LLM Fine\-Tuning: A Benchmark\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024c\.[https://arxiv\.org/abs/2402\.11592](https://arxiv.org/abs/2402.11592)\.
- Zhang et al\. \(2025c\)Yilang Zhang, Xiaodong Yang, Yiwei Cai, and Georgios B\. Giannakis\.ScaLoRA: Optimally Scaled Low\-Rank Adaptation for Efficient High\-Rank Fine\-Tuning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025c\.[https://arxiv\.org/abs/2510\.23818](https://arxiv.org/abs/2510.23818)\.
- Zhang et al\. \(2022b\)Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu\.Neural Prompt Search\.*IEEE TPAMI*, 2022b\.[https://doi\.org/10\.1109/TPAMI\.2024\.3435939](https://doi.org/10.1109/TPAMI.2024.3435939)\.
- Zhang et al\. \(2025d\)Yuanhe Zhang, Fanghui Liu, and Yudong Chen\.LoRA\-One: One\-Step Full Gradient Could Suffice for Fine\-Tuning Large Language Models, Provably and Efficiently\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2025d\.[https://arxiv\.org/abs/2502\.01235](https://arxiv.org/abs/2502.01235)\.
- Zhang et al\. \(2024d\)Yushun Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Diederik P\. Kingma, Yinyu Ye, Zhi\-Quan Luo, and Ruoyu Sun\.Adam\-mini: Use Fewer Learning Rates To Gain More\.In*International Conference on Learning Representations \(ICLR\)*, 2024d\.[https://arxiv\.org/abs/2406\.16793](https://arxiv.org/abs/2406.16793)\.
- Zhang et al\. \(2026f\)Yutong Zhang, Jiaxin Chen, Honglin Chen, Kaiqi Zheng, Shengcai Liao, Hanwen Zhong, Weixin Li, and Yunhong Wang\.Memory\-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2026f\.[https://arxiv\.org/abs/2604\.09088](https://arxiv.org/abs/2604.09088)\.
- Zhang et al\. \(2025e\)Yuxuan Zhang, Yirui Yuan, Yiren Song, Haofan Wang, and Jiaming Liu\.EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\)*, 2025e\.[https://arxiv\.org/abs/2503\.07027](https://arxiv.org/abs/2503.07027)\.
- Zhang et al\. \(2025f\)Zechuan Zhang, Ji Xie, Yu Lu, Zongxin Yang, and Yi Yang\.In\-Context Edit: Enabling Instructional Image Editing with In\-Context Generation in Large Scale Diffusion Transformer\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025f\.[https://arxiv\.org/abs/2504\.20690](https://arxiv.org/abs/2504.20690)\.
- Zhang et al\. \(2024e\)Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang\.Q\-GaLore: Quantized GaLore with INT4 Projection and Layer\-Adaptive Low\-Rank Gradients\.In*CPAL*, 2024e\.[https://proceedings\.mlr\.press/v280/zhang25a\.html](https://proceedings.mlr.press/v280/zhang25a.html)\.
- Zhang et al\. \(2023j\)Zhi Zhang, Qizhe Zhang, Zijun Gao, Renrui Zhang, Ekaterina Shutova, Shiji Zhou, and Shanghang Zhang\.Gradient\-based Parameter Selection for Efficient Fine\-Tuning\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2023j\.[https://arxiv\.org/abs/2312\.10136](https://arxiv.org/abs/2312.10136)\.
- Zhang et al\. \(2025g\)Zicheng Zhang, Haoran Li, Yifeng Zhang, Guoqiang Gong, Jiaxing Wang, Junxing Hu, Pengzhang Liu, and Qixia Jiang\.The Primacy of Magnitude in Low\-Rank Adaptation\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025g\.[https://arxiv\.org/abs/2507\.06558](https://arxiv.org/abs/2507.06558)\.
- Zhang et al\. \(2026g\)Zicheng Zhang, Haoran Li, Jiaxing Wang, Guoqiang Gong, Anqi Li, Yudong Hu, Ting Xiong, Yurong Gao, Junxing Hu, Zhida Jiang, Yifeng Zhang, Pengzhang Liu, and Qixia Jiang\.The Hidden Power of Scaling Factor in LoRA Optimization\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2026g\.[https://arxiv\.org/abs/2606\.12883](https://arxiv.org/abs/2606.12883)\.
- Zhao et al\. \(2023\)Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie\.Tuning LayerNorm in Attention: Towards Efficient Multi\-Modal LLM Finetuning\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.[https://arxiv\.org/abs/2312\.11420](https://arxiv.org/abs/2312.11420)\.
- Zhao et al\. \(2022a\)Bo Zhao, Nima Dehmamy, Robin Walters, and Rose Yu\.Symmetry teleportation for accelerated optimization\.In*Advances in Neural Information Processing Systems*, volume 35, 2022a\.[https://arxiv\.org/abs/2205\.10637](https://arxiv.org/abs/2205.10637)\.
- Zhao et al\. \(2022b\)Bo Zhao, Iordan Ganev, Robin Walters, Rose Yu, and Nima Dehmamy\.Symmetries, Flat Minima, and the Conserved Quantities of Gradient Flow\.In*International Conference on Learning Representations \(ICLR\)*, 2022b\.[https://arxiv\.org/abs/2210\.17216](https://arxiv.org/abs/2210.17216)\.
- Zhao et al\. \(2024a\)Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian\.GaLore: Memory\-Efficient LLM Training by Gradient Low\-Rank Projection\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024a\.[https://arxiv\.org/abs/2403\.03507](https://arxiv.org/abs/2403.03507)\.
- Zhao et al\. \(2026\)Lanxin Zhao, Bamdev Mishra, Pratik Jawanpuria, Lequan Lin, Dai Shi, Junbin Gao, and Andi Han\.LOFT: Low\-Rank Orthogonal Fine\-Tuning via Task\-Aware Support Selection\.*arXiv preprint arXiv:2605\.11872*, 2026\.[https://arxiv\.org/abs/2605\.11872](https://arxiv.org/abs/2605.11872)\.
- Zhao et al\. \(2020\)Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze\.Masking as an Efficient Alternative to Finetuning for Pretrained Language Models\.In*Proceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 2020\.[https://aclanthology\.org/2020\.emnlp\-main\.174/](https://aclanthology.org/2020.emnlp-main.174/)\.
- Zhao et al\. \(2024b\)Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu\.LoraRetriever: Input\-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild\.In*Findings of the Association for Computational Linguistics: ACL 2024*, 2024b\.[https://aclanthology\.org/2024\.findings\-acl\.263/](https://aclanthology.org/2024.findings-acl.263/)\.
- Zhao et al\. \(2024c\)Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu\.Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank\-Wise Clustering\.In*International Conference on Learning Representations \(ICLR\)*, 2024c\.[https://openreview\.net/forum?id=j6fsbpAllN](https://openreview.net/forum?id=j6fsbpAllN)\.
- Zheng et al\. \(2025\)Jingjing Zheng, Wanglong Lu, Yiming Dong, Chaojie Ji, Yankai Cao, and Zhouchen Lin\.AdaMSS: Adaptive multi\-subspace approach for parameter\-efficient fine\-tuning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://papers\.neurips\.cc/paper\_files/paper/2025/hash/1c85c302ece39939c1b334c78f7ee1b8\-Abstract\-Conference\.html](https://papers.neurips.cc/paper_files/paper/2025/hash/1c85c302ece39939c1b334c78f7ee1b8-Abstract-Conference.html)\.
- Zheng and Wu \(2026\)Jinyang Zheng and Tong Wu\.LoRA\-S: An efficient low rank adaptation scheme via Sylvester equation\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/a6a90bcc2aa470c3871b2d39a67d26e8\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/a6a90bcc2aa470c3871b2d39a67d26e8-Abstract-Conference.html)\.
- Zhong et al\. \(2024a\)Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen\.Multi\-LoRA Composition for Image Generation\.*TMLR*, 2024a\.[https://arxiv\.org/abs/2402\.16843](https://arxiv.org/abs/2402.16843)\.
- Zhong et al\. \(2024b\)Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang\.PEANuT: Parameter\-Efficient Adaptation with Weight\-aware Neural Tweakers\.In*Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\)*, 2024b\.[https://arxiv\.org/abs/2410\.01870](https://arxiv.org/abs/2410.01870)\.
- Zhong et al\. \(2024c\)Yibo Zhong, Jinman Zhao, and Yao Zhou\.Low\-Rank Interconnected Adaptation across Layers\.In*Findings of the Association for Computational Linguistics: ACL 2025*, 2024c\.[https://arxiv\.org/abs/2407\.09946](https://arxiv.org/abs/2407.09946)\.
- Zhou et al\. \(2023a\)Allan Zhou, Kaien Yang, Kaylee Burns, Adriano Cardace, Yiding Jiang, Samuel Sokota, J\. Zico Kolter, and Chelsea Finn\.Permutation equivariant neural functionals\.In*Advances in Neural Information Processing Systems*, volume 36, 2023a\.[https://arxiv\.org/abs/2302\.14040](https://arxiv.org/abs/2302.14040)\.
- Zhou et al\. \(2023b\)Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen\.AutoPEFT: Automatic Configuration Search for Parameter\-Efficient Fine\-Tuning\.*Transactions of the Association for Computational Linguistics*, 2023b\.[https://aclanthology\.org/2024\.tacl\-1\.29/](https://aclanthology.org/2024.tacl-1.29/)\.
- Zhou et al\. \(2024\)Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang\.LoRA\-drop: Efficient LoRA Parameter Pruning based on Output Evaluation\.In*Proceedings of the International Conference on Computational Linguistics \(COLING\)*, 2024\.[https://aclanthology\.org/2025\.coling\-main\.371/](https://aclanthology.org/2025.coling-main.371/)\.
- Zhou et al\. \(2021\)Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu\.Learning to Prompt for Vision\-Language Models\.*International Journal of Computer Vision*, 2021\.[https://link\.springer\.com/article/10\.1007/s11263\-022\-01653\-1](https://link.springer.com/article/10.1007/s11263-022-01653-1)\.
- Zhou et al\. \(2022\)Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu\.Conditional Prompt Learning for Vision\-Language Models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2022\.[https://arxiv\.org/abs/2203\.05557](https://arxiv.org/abs/2203.05557)\.
- Zhu et al\. \(2024a\)Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z\. Pan, Zhangyang Wang, and Jinwon Lee\.APOLLO: SGD\-like Memory, AdamW\-level Performance\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, 2024a\.[https://arxiv\.org/abs/2412\.05270](https://arxiv.org/abs/2412.05270)\.
- Zhu et al\. \(2025\)Hanqing Zhu, Zhenyu Zhang, Hanxian Huang, DiJia Su, Zechun Liu, Jiawei Zhao, Igor Fedorov, Hamed Pirsiavash, Zhizhou Sha, Jinwon Lee, David Z\. Pan, Zhangyang Wang, Yuandong Tian, and Kai Sheng Tai\.The Path Not Taken: RLVR Provably Learns Off the Principals\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2511\.08567](https://arxiv.org/abs/2511.08567)\.
- Zhu et al\. \(2024b\)Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon\.Asymmetry in Low\-Rank Adapters of Foundation Models\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, 2024b\.[https://arxiv\.org/abs/2402\.16842](https://arxiv.org/abs/2402.16842)\.
- Zhuang et al\. \(2025\)Yuan Zhuang, Yi Shen, Yuexin Bian, Qing Su, Shihao Ji, Yuanyuan Shi, and Fei Miao\.LD\-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.[https://arxiv\.org/abs/2509\.25684](https://arxiv.org/abs/2509.25684)\.
- Zi et al\. \(2023\)Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam\-Fai Wong, and Lei Zhang\.Delta\-LoRA: Fine\-Tuning High\-Rank Parameters with the Delta of Low\-Rank Matrices\.*arXiv preprint arXiv:2309\.02411*, 2023\.[https://arxiv\.org/abs/2309\.02411](https://arxiv.org/abs/2309.02411)\.
- Zou et al\. \(2023\)Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann\-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J\. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J\. Zico Kolter, and Dan Hendrycks\.Representation Engineering: A Top\-Down Approach to AI Transparency\.*arXiv preprint arXiv:2310\.01405*, 2023\.[https://arxiv\.org/abs/2310\.01405](https://arxiv.org/abs/2310.01405)\.
- Zweiger et al\. \(2025\)Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal\.Self\-Adapting Language Models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.[https://arxiv\.org/abs/2506\.10943](https://arxiv.org/abs/2506.10943)\.

## Technical appendix

The appendices fall into three groups\. Each can be read on its own, and every entry, proof and table points back to the place in the main text that uses it\.

##### Preliminaries

Appendix[A](https://arxiv.org/html/2610.12728#A1)collects the mathematical background the exposés assume: categories, slices and base change; monoidal categories, string diagrams and lenses; manifolds, germs and cones; and groups, symmetry and conservation laws\. Its 93 entries keep the numbers A\.1–D\.34 they carry on the companion site, and each states a notion with its hypotheses, shows it on a fine\-tuning example and lists the results that rely on it\. Appendix[B](https://arxiv.org/html/2610.12728#A2)quotes the external theorems that the proofs use, each in the exact form in which it is applied and with its source\.

##### Theory

Appendix[C](https://arxiv.org/html/2610.12728#A3)gives complete proofs of the numbered results, in the order of the main text and with one subsection per result\. Each subsection restates its result, in the form recorded after two rounds of review, before the proof\.

##### Data and reproducibility

Appendix[D](https://arxiv.org/html/2610.12728#A4)describes the catalogue of 382 methods from 138 papers\. It lists every method with its seven coordinates and its reference, and every one of the 403 recorded simulation arrows with its maphh\. Appendix[E](https://arxiv.org/html/2610.12728#A5)documents the numerical checks marked \[Num\] in the main text and explains how to rerun them from the data files that accompany this paper\.

##### Process

Appendix[F](https://arxiv.org/html/2610.12728#A6)records how this paper and its companion site were made: who did what, how the literature was searched and checked, how the theory was generalised and refereed, and what the work cost in agent runs and tokens\.

## Appendix AMathematical background

This appendix is for readers who know linear algebra, multivariable calculus, basic probability and the PEFT literature, and who have not studied category theory, Lie groups or real algebraic geometry\. It skips what ML readers already use, such as the SVD, rank, orthogonal matrices, eigenvalues, Jacobians, gradients, Adam, LoRA and the transformer block\. Each of the 93 entries gives a short definition or statement with its hypotheses, a small example from fine\-tuning, the places in the paper that use it, and usually a textbook for further reading\. Definitions carry an ochre rule and standard facts quoted from the literature a teal one; the one remark,[Background A\.20](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3), is set like a definition\.

Part A covers categories, slices and base change, Part B monoidal categories, string diagrams and lenses, Part C manifolds, cones and real algebraic geometry, and Part D groups, symmetry and conservation laws\. The parts can be read in any order, since each entry refers to the entries it depends on at their first use\. The exposés cite an entry at the first use of its term, as in[Background A\.8](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1), and each entry ends with the results that rely on it\.

A few symbols from the exposés recur in the examples\. A methodM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\)is a smooth mapρ:Q→Θ\\rho:Q\\to\\Thetafrom trainable parameters to weights withρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}\([Background A\.8](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)\), and\|M\|=dimQ\\lvert M\\rvert=\\dim Qis its parameter budget\. The expressive dimensiond⁡\(M\)d\(M\), the first\-order imageS1S\_\{1\}and the first\-order deficit are defined in[Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1); the base point is*regular*forMMwhen the deficit is00and an*apex*when it is positive\. LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)isρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBAwithB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}andA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}, and its full\-rank pointings fall into the strataTA\\mathrm\{T\}\_\{A\},TB\\mathrm\{T\}\_\{B\}andS\\mathrm\{S\}of[Background C\.19](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\.*Block\-scalar rates*give each factor one learning rate,ηB\\eta\_\{B\}forBBandηA\\eta\_\{A\}forAA\([Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\.

Table[33](https://arxiv.org/html/2610.12728#A1.T33)lists every entry with its area and the first places it is used\.

Table 33:Index of the background entries, with the area of each and the first places in the paper that use it\.No\.EntryAreaFirst places used[A\.1](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)categoryCategoriesTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)[A\.2](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)commutative diagramCategoriesFigure[2](https://arxiv.org/html/2610.12728#S1.F2),[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)[A\.3](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)isomorphismCategories[Obs\. I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)[A\.4](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)functor; functorial invariantCategories[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Obs\. I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[A\.5](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)natural transformation; naturalityCategories[Thm III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)[A\.6](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)slice categoryCategoriesTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)[A\.7](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)coslice; pointed objects; opposite categoryCategories[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Rem\. I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)[A\.8](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)CategoriesTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)[A\.9](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)initial and terminal objects; universal propertyLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\), Figure[2](https://arxiv.org/html/2610.12728#S1.F2)[A\.10](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)productLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\),[Prop\. I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)[A\.11](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)pullback; fibre productLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\),[Prop\. I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)[A\.12](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)limit; creation of limitsLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)[A\.13](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)adjunctionLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)[A\.14](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)two standard facts about limitsLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)[A\.15](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)monomorphism; sub\-objectLimits[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Cor\. III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)[A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)posets; meets; the expressivity orderLimits[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Prop\. I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(b\)[A\.17](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)automorphism of a category; transport of structureBase change[Prop\. II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)[A\.18](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)lift; lifting problem; obstructionBase change[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Obs\. I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)base change; pushforwardϕ\!\\phi\_\{\!\}; the relative point of viewBase changeExposés[0](https://arxiv.org/html/2610.12728#S0)–[I](https://arxiv.org/html/2610.12728#S1),[Def\. V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1)[A\.20](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)constructions the paper names and leaves unusedBase changeExposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\), Exposé[XI](https://arxiv.org/html/2610.12728#S11)\(seven borrowed words\)[B\.1](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)monoidal and symmetric monoidal categoriesMonoidal[Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5),[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)[B\.2](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)direct sum and tensor product in𝐅𝐢𝐧𝐕𝐞𝐜𝐭\\mathbf\{FinVect\}Monoidal[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)[B\.3](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)string diagramsMonoidalExposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3),[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)[B\.4](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)cartesian categories; copy and discardMonoidal[Def\. I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Thm VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)[B\.5](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)compact closed categories; cups, caps and wire\-bendingMonoidal[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Thm VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)[B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)Frobenius algebras and spidersMonoidalExposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3),[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)[B\.7](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2)tensor networks, contraction, bond dimensionTensor networks[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)[B\.8](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2)the min\-cut bound on rankTensor networks[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)[B\.9](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)tensor trains, Tucker and MPO; gauge on bondsTensor networks[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)[B\.10](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)Kronecker product, vec, tensor product of subspacesTensor networks[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)[B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)Van Loan–Pitsianis rearrangement; Kronecker rankTensor networks[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Prop\. II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)[B\.12](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)Schmidt and operator\-Schmidt decompositionsTensor networks[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2), Figure[6](https://arxiv.org/html/2610.12728#S2.F6)[B\.13](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)rank of a Hadamard product; face\-splitting productTensor networks[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Thm II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)[B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)sums of methods; Minkowski sums, differences and translatesPara, lenses[Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5),[Thm II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)[B\.15](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)strong monoidal functors;\(ℕ,\+\)\(\\mathbb\{N\},\+\)and the permutation groupoidPara, lenses[Obs\. I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), Exposé[XI](https://arxiv.org/html/2610.12728#S11)\(the referee record\)[B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)bicategories and the Para constructionPara, lenses[Def\. I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Rem\. I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)[B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)lenses, reverse derivatives and𝐏𝐚𝐫𝐚⁡\(𝖱\)\\mathbf\{Para\}\(\\mathsf\{R\}\)Para, lensesExposé[IV](https://arxiv.org/html/2610.12728#S4),[Def\. IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)[C\.1](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)smooth manifolds and smooth mapsManifolds[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[C\.2](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)tangent spaces, differentials and covectorsManifoldsTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)rank, expressive dimension and first\-order imageManifoldsTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[C\.4](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)immersions, submersions and embedded submanifoldsManifolds[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Prop\. IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)[C\.5](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)constant\-rank theoremManifolds[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)[C\.6](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)fibres and fibre dimensionManifoldsExposé[III](https://arxiv.org/html/2610.12728#S3),[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[C\.7](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)fibre products, transversality and clean intersectionManifolds[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Prop\. I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)[C\.8](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)germsSingularities[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), Exposé[II](https://arxiv.org/html/2610.12728#S2)\(the image germ\)[C\.9](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)smooth and singular pointsSingularities[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)[C\.10](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)cones and verticesSingularities[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Thm II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)[C\.11](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2.Px1)first\-order image versus imageSingularities[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Obs\. I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)the determinantal varietyVarietiesTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Thm II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)[C\.13](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)the skew\-determinantal coneVarieties[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)[C\.14](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)Segre and toric conesVarieties[Prop\. II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Prop\. II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)[C\.15](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)the GrassmannianVarieties[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)[C\.16](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)images of smooth maps need not be manifoldsTame geometry[Obs\. I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)[C\.17](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)semialgebraic and definable setsTame geometryExposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\), §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1)\(arrow rules\)[C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)Tarski–Seidenberg and definable dimensionTame geometry[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1)\(arrow rules\)[C\.19](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)stratification and local dimensionTame geometry[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)[C\.20](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)moduli sets and moduli spacesTame geometry[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), Exposé[IX](https://arxiv.org/html/2610.12728#S9)[C\.21](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)generic propertiesTame geometry[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Prop\. II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)[C\.22](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)real\-analytic functions and their zero setsTame geometry[Lem\. VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3),[Thm VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)[D\.1](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)Lie groups; the matrix groups of this paperLie theoryTable[2](https://arxiv.org/html/2610.12728#S0.T2),[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)[D\.2](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)tori, diagonal monoids and saturationLie theory[Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1),[Prop\. II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)[D\.3](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)permutations, monomial matrices, semidirect products, normalisersLie theory[Prop\. II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Prop\. II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)[D\.4](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)connectedness and the identity componentLie theory[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)[D\.5](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)Lie algebras; the commutator;𝔤​𝔩\\mathfrak\{gl\},𝔰​𝔬\\mathfrak\{so\},Sym\\mathrm\{Sym\},∧\\wedgeLie theory[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3),[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)[D\.6](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)the exponential map; one\-parameter subgroupsLie theory[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)[D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)the Cayley map and its Neumann truncationLie theory[Cor\. II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)[D\.8](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)Lie subgroups and subalgebras; the generated Lie algebraLie theory[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), Figure[10](https://arxiv.org/html/2610.12728#S5.F10)[D\.9](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)Yamabe’s theoremLie theory[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), §[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)[D\.10](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)monoids, units, monoid orbits, homomorphismsLie theory[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3),[Prop\. II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)[D\.11](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)Cartan–Dieudonné theorem; Scherk’s sharp formLie theory[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)[D\.12](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)group actionsInvariants[Def\. I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), Table[2](https://arxiv.org/html/2610.12728#S0.T2)[D\.13](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)orbits, stabilisers, free actionsInvariants[Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1),[Prop\. II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)[D\.14](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)Haar measure; invariant measures on homogeneous spacesInvariants[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), §[II\.A](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)[D\.15](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)invariants and complete invariantsInvariants[Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)[D\.16](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)first fundamental theorem forO⁡\(n\)\\mathrm\{O\}\(n\); Witt’s extension theoremInvariants[Thm II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), Exposé[XII](https://arxiv.org/html/2610.12728#S12)[D\.17](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)Klein’s Erlangen programmeInvariants[Def\. II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), Exposé[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)[D\.18](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)equivariance and invariance of maps, update rules and merge rulesInvariants[Def\. II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6),[Obs\. II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)[D\.19](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)multihomogeneous maps; multidegreeInvariants[Thm III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), Exposé[II](https://arxiv.org/html/2610.12728#S2)\(the image germ\)[D\.20](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)orbit space of a positive torusInvariants[Thm III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Cor\. III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)[D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)gauge symmetry of a parametrisationGauge, metrics[Def\. I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), Table[2](https://arxiv.org/html/2610.12728#S0.T2)[D\.22](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)quotients and descent; the quotient manifold theoremGauge, metrics[Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4),[Prop\. III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)[D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)metrics, cometrics and the pulled\-back cometricGauge, metrics[Def\. III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),[Obs\. III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)[D\.24](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)Riemannian gradient flow; tangent projection; retractionGauge, metrics[Prop\. III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)[D\.25](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)distributions, Lie brackets and Frobenius’ theoremGauge, metrics[Prop\. IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4), §[IV](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)[D\.26](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4)Noether’s theorem for gradient flowsConservation[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), Figure[7](https://arxiv.org/html/2610.12728#S3.F7)[D\.27](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)Cartan decomposition; polar decompositionConservation[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)[D\.28](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)balancedness; the conservedB⊤​B−A​A⊤B^\{\\top\}B\-AA^\{\\top\}Conservation[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Prop\. III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)[D\.29](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)the moment mapConservation[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Cor\. III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)[D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)Kempf–Ness theorem, real formConservation[Prop\. III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Cor\. III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)[D\.31](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px2)the closed orbit in each fibre of\(B,A\)↦B​A\(B,A\)\\mapsto BAConservation[Prop\. V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4), §[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)[D\.32](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)Ky Fan’s maximum principleMatrix tools[Prop\. II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3), §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)[D\.33](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)Sylvester equationsMatrix tools[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8), Figure[9](https://arxiv.org/html/2610.12728#S3.F9)[D\.34](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)the perspective map; convex combinationsMatrix tools[Thm VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6), Figure[12](https://arxiv.org/html/2610.12728#S6.F12)### Part A\. Categories without tears

Part A collects the category theory the exposés use, from the definition of a category up to base change, and illustrates every notion in one running example, the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)of weight\-space methods\. Entries[Background A\.1](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)to[Background A\.8](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)set up categories, functors and slices,[Background A\.9](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)to[Background A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)cover universal properties, limits and orders, and[Background A\.17](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)to[Background A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)cover transport, lifting and base change, with[Background A\.20](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)listing the constructions the paper names and leaves unused\.

#### Categories and slices

Background A\.1\(category\)\. Acategory𝒞\\mathcal\{C\}has objects and, for any two objectsX,YX,Y, a set𝒞⁡\(X,Y\)\\mathcal\{C\}\(X,Y\)ofmorphisms, or arrows,f:X→Yf:X\\to Y\. Arrowsf:X→Yf:X\\to Yandg:Y→Zg:Y\\to Zhave a compositeg∘f:X→Zg\\circ f:X\\to Z\. Composition is associative, and every object has an identity arrowidX\\mathrm\{id\}\_\{X\}withf∘idX=f=idY∘ff\\circ\\mathrm\{id\}\_\{X\}=f=\\mathrm\{id\}\_\{Y\}\\circ f\.

The paper uses𝐒𝐞𝐭\\mathbf\{Set\}\(sets and functions\),𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\(smooth manifolds and smooth maps\),𝐅𝐢𝐧𝐕𝐞𝐜𝐭\\mathbf\{FinVect\}\(finite\-dimensional real vector spaces and linear maps, where tensor networks are evaluated\) and the pointed categories𝐒𝐞𝐭∗\\mathbf\{Set\}\_\{\*\}and𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}of[Background A\.7](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)\. In𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)the objects are methodsM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\), and an arrowh:M→Nh:M\\to Nis a pointed smooth maph:QM→QNh:Q\_\{M\}\\to Q\_\{N\}withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}, read “NNsimulatesMM”\. Simulations compose, sinceρP∘k∘h=ρN∘h=ρM\\rho\_\{P\}\\circ k\\circ h=\\rho\_\{N\}\\circ h=\\rho\_\{M\}forh:M→Nh:M\\to Nandk:N→Pk:N\\to P\. For instance,h⁡\(B\)=\(B,A0\)h\(B\)=\(B,A\_\{0\}\)is an arrow from LoRA\-FA\(A0\)\(A\_\{0\}\)\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\)to LoRA pointed at\(0,A0\)\(0,A\_\{0\}\)\. The simulation arrows of the atlas are arrows of this category, each recorded with its maphh\.

Used in\.Table[2](https://arxiv.org/html/2610.12728#S0.T2),[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), Exposés[0](https://arxiv.org/html/2610.12728#S0)–[I](https://arxiv.org/html/2610.12728#S1), §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3), Figure[14](https://arxiv.org/html/2610.12728#S8.F14)\. Reading:[Leinster \(2014, Ch\. 1\)](https://arxiv.org/html/2610.12728#bib.bib242)\.

Background A\.2\(commutative diagram\)\. Adiagramis a family of objects joined by arrows, drawn as a directed graph\. Itcommutesif any two directed paths with the same start and the same end compose to the same arrow\. A triangle commutes when one side equals the composite of the other two, and a square commutes when its two routes between opposite corners agree\.

A simulation is a commuting triangle overΘ\\Theta, whose two paths fromQMQ\_\{M\}toΘ\\Thetaare

QM→ℎQN→ρNΘandQM→ρMΘ\.Q\_\{M\}\\xrightarrow\{\\;h\\;\}Q\_\{N\}\\xrightarrow\{\\;\\rho\_\{N\}\\;\}\\Theta\\qquad\\text\{and\}\\qquad Q\_\{M\}\\xrightarrow\{\\;\\rho\_\{M\}\\;\}\\Theta\.ForM=FA⁡\(A0\)M=\\mathrm\{FA\}\(A\_\{0\}\),NNLoRA pointed at\(0,A0\)\(0,A\_\{0\}\)andh⁡\(B\)=\(B,A0\)h\(B\)=\(B,A\_\{0\}\), both paths sendBBto the weightW0\+B​A0W\_\{0\}\+BA\_\{0\}\(scale absorbed intoBB\), so the triangle commutes\. Figure[2](https://arxiv.org/html/2610.12728#S1.F2)\(a\) draws, for a general method, the triangle of Figure[15](https://arxiv.org/html/2610.12728#A1.F15), which commutes only up to a pointing defect when the method starts away fromθ0\\theta\_\{0\}\([Background A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\)\. Commuting squares, used for merging on extensions, return in[Background A\.18](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\.

Used in\.Figure[2](https://arxiv.org/html/2610.12728#S1.F2),[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2),[Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1), Exposé[V](https://arxiv.org/html/2610.12728#S5)\. Reading:[Leinster \(2014, §1\.1\)](https://arxiv.org/html/2610.12728#bib.bib242)\.

Background A\.3\(isomorphism\)\. An arrowf:X→Yf:X\\to Yis anisomorphism, writtenX≅YX\\cong Y, if someg:Y→Xg:Y\\to Xsatisfiesg∘f=idXg\\circ f=\\mathrm\{id\}\_\{X\}andf∘g=idYf\\circ g=\\mathrm\{id\}\_\{Y\}\. The inverseggis then unique\. In𝐃𝐢𝐟𝐟\\mathbf\{Diff\}the isomorphisms are the diffeomorphisms, so a smooth bijection need not be one, ast↦t3t\\mapsto t^\{3\}shows\. In𝐌𝐞𝐭𝐡\\mathbf\{Meth\}an isomorphismM≅NM\\cong Nis a pointed diffeomorphismh:QM→QNh:Q\_\{M\}\\to Q\_\{N\}withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Its inverse lies overΘ\\Thetaautomatically, sinceρM∘h−1=ρN\\rho\_\{M\}\\circ h^\{\-1\}=\\rho\_\{N\}\.

The LoRA gauge \([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\(B,A\)↦\(B​g−1,g​A\)\(B,A\)\\mapsto\(Bg^\{\-1\},gA\),g∈GLrg\\in\\mathrm\{GL\}\_\{r\}, is an isomorphism from LoRA pointed at\(B0,A0\)\(B\_\{0\},A\_\{0\}\)to LoRA pointed at\(B0​g−1,g​A0\)\(B\_\{0\}g^\{\-1\},gA\_\{0\}\)\. Other examples are rsLoRA\([Kalajdzievski, 2023](https://arxiv.org/html/2610.12728#bib.bib204)\)≅\\congLoRA by\(B,A\)↦\(r​B,A\)\(B,A\)\\mapsto\(\\sqrt\{r\}\\,B,A\), andLoRAr⊕LoRAs≅LoRAr\+s\\mathrm\{LoRA\}\_\{r\}\\oplus\\mathrm\{LoRA\}\_\{s\}\\cong\\mathrm\{LoRA\}\_\{r\+s\}by concatenating factors \([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\. Isomorphic methods have the same image, the sameS1S\_\{1\}and the same first\-order deficit \([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. They can still train differently, because the optimizer acts on the coordinates ofQQ\. For LoRA’s gauge with block\-scalar rates \(one rate per factor\), gradient flow is unchanged exactly wheng∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\), at points whereAAhas full row rank andr<nr<n\([Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3);[Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\.

Used in\.[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), §[II\.A](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5), §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3)\.

Background A\.4\(functor; functorial invariant\)\. AfunctorF:𝒞→𝒟F:\\mathcal\{C\}\\to\\mathcal\{D\}sends each objectXXto an objectF​XFXand each arrowf:X→Yf:X\\to Yto an arrowF​f:F​X→F​YFf:FX\\to FY, withF⁡\(g∘f\)=F​g∘F​fF\(g\\circ f\)=Fg\\circ FfandF⁡\(idX\)=idF​XF\(\\mathrm\{id\}\_\{X\}\)=\\mathrm\{id\}\_\{FX\}\. A functor sends isomorphisms to isomorphisms, soF​XFXis an isomorphism invariant ofXX\. Aforgetfulfunctor discards structure, as𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}discards the base point\.

The image is a functorIm\\Imopfrom𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)to the subsets ofΘ\\Thetacontainingθ0\\theta\_\{0\}, ordered by inclusion \(a poset,[Background A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\), because an arrowh:M→Nh:M\\to NgivesIm⁡M=ρN​\(h⁡\(QM\)\)⊆Im⁡N\\Imop M=\\rho\_\{N\}\(h\(Q\_\{M\}\)\)\\subseteq\\Imop N\. In the same way the chain rule atq0q\_\{0\}makesS1S\_\{1\}a functor to the linear subspaces ofΘ\\Theta, withS1​\(M\)⊆S1​\(N\)S\_\{1\}\(M\)\\subseteq S\_\{1\}\(N\)along every pointed arrow\. Read backwards, functoriality gives the atlas test O1 and the first half of O3 \(§[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx2)\)\. IfIm⁡M⊈Im⁡N\\Imop M\\not\\subseteq\\Imop NorS1​\(M\)⊈S1​\(N\)S\_\{1\}\(M\)\\not\\subseteq S\_\{1\}\(N\), thenNNdoes not simulateMM\. Base changeϕ\!\\phi\_\{\!\}\([Background A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\) and transportT∗T\_\{\*\}\([Background A\.17](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\) are functors between slices\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4),[Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1), §[IV](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\. Reading:[Leinster \(2014, §1\.2\)](https://arxiv.org/html/2610.12728#bib.bib242)\.

Background A\.5\(natural transformation; naturality\)\. For functorsF,G:𝒞→𝒟F,G:\\mathcal\{C\}\\to\\mathcal\{D\}, anatural transformationα:F⇒G\\alpha:F\\Rightarrow Gis a family of arrowsαX:F​X→G​X\\alpha\_\{X\}:FX\\to GXwithG​f∘αX=αY∘F​fGf\\circ\\alpha\_\{X\}=\\alpha\_\{Y\}\\circ Fffor everyf:X→Yf:X\\to Y\. This naturality square says that applyingα\\alphaand then moving alongffagrees with moving first and applyingα\\alphaafter\. A groupKKcan be viewed as a category with one object whose arrows are the elements ofKK\. A functor from it to𝐒𝐞𝐭\\mathbf\{Set\}is then a set with aKK\-action, and a natural transformation between two such functors is exactly aKK\-equivariant map\.

The copy mapΔX​\(x\)=\(x,x\)\\Delta\_\{X\}\(x\)=\(x,x\)is natural,ΔY∘f=\(f×f\)∘ΔX\\Delta\_\{Y\}\\circ f=\(f\\times f\)\\circ\\Delta\_\{X\}, which is rule R7 of the merge calculus \([Background B\.4](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\)\. Elsewhere in the paper, natural for a group of changes of chart means equivariant in this sense \([Background D\.18](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\)\. Task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\)is natural for all ofGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\), and DARE\([Yu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib538)\)for invertible diagonal matrices at a fixed mask \([Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(c\)\)\. Gradient flow on LoRA commutes with the gaugeggexactly when the induced cometrics \([Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\) agree, which for block\-scalar rates singles outO⁡\(r\)\\mathrm\{O\}\(r\)\([Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\.

Used in\.Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\),[Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3), §[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4), §[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4),[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\. Reading:[Leinster \(2014, §1\.3\)](https://arxiv.org/html/2610.12728#bib.bib242)\.

Background A\.6\(slice category\)\. For an objectXXof𝒞\\mathcal\{C\}, theslice𝒞/X\\mathcal\{C\}/Xhas as objects the arrowsa:A→Xa:A\\to X, called objects overXX, and as morphisms\(A,a\)→\(B,b\)\(A,a\)\\to\(B,b\)the arrowsh:A→Bh:A\\to Bof𝒞\\mathcal\{C\}withb∘h=ab\\circ h=a\. Composition and identities are those of𝒞\\mathcal\{C\}\. The objectXXis thebase, andaais thestructure map\.

𝐃𝐢𝐟𝐟/Θ\\mathbf\{Diff\}/\\Thetaconsists of smooth mapsρ:Q→Θ\\rho:Q\\to\\Thetainto weight space without base points, and its morphisms are the weak morphisms of[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), which need not sendq0q\_\{0\}toq0′q\_\{0\}^\{\\prime\}\. A slice always has a terminal object, the identity\(X,idX\)\(X,\\mathrm\{id\}\_\{X\}\), which here is full fine\-tuning\. Working in a slice is the relative point of view of[Background A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3), in which an object is studied together with its map to a base\.

Used in\.Table[2](https://arxiv.org/html/2610.12728#S0.T2),[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\), Rosetta entries 3 and 4 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\), the seven borrowed words of Exposé[XI](https://arxiv.org/html/2610.12728#S11)\. Reading:[Mac Lane \(1998, §II\.6\)](https://arxiv.org/html/2610.12728#bib.bib306)\.

Background A\.7\(coslice; pointed objects; opposite category\)\. ThecosliceX/𝒞X/\\mathcal\{C\}has as objects the arrowsx:X→Ax:X\\to Aout ofXX, and as morphisms\(A,x\)→\(B,x′\)\(A,x\)\\to\(B,x^\{\\prime\}\)the arrowsh:A→Bh:A\\to Bwithh∘x=x′h\\circ x=x^\{\\prime\}\. For the one\-point spaceX=ptX=\\mathrm\{pt\}, an arrowpt→Q\\mathrm\{pt\}\\to Qis a pointq0q\_\{0\}, so𝐃𝐢𝐟𝐟∗:=pt/𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}:=\\mathrm\{pt\}/\\mathbf\{Diff\}is the category ofpointed manifolds\(Q,q0\)\(Q,q\_\{0\}\)andpointed maps, those withh⁡\(q0\)=q0′h\(q\_\{0\}\)=q\_\{0\}^\{\\prime\}\. Likewise𝐒𝐞𝐭∗:=pt/𝐒𝐞𝐭\\mathbf\{Set\}\_\{\*\}:=\\mathrm\{pt\}/\\mathbf\{Set\}\. Theopposite category𝒞op\\mathcal\{C\}^\{\\mathrm\{op\}\}has the same objects and every arrow reversed, and\(𝒞/X\)op≅X/𝒞op\(\\mathcal\{C\}/X\)^\{\\mathrm\{op\}\}\\cong X/\\mathcal\{C\}^\{\\mathrm\{op\}\}\.

A method sends its initialisation to the checkpoint,ρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}, so it is an arrow of𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}, and a weak morphism forgets the base points and lives in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\. Opposites enter in[Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\. A simulationh:M→Nh:M\\to Nruns fromQMQ\_\{M\}toQNQ\_\{N\}\. The network withQMQ\_\{M\}\-parameters is obtained from the one withQNQ\_\{N\}\-parameters by precomposing withhh,

f∘\(ρM×idX\)=f∘\(ρN×idX\)∘\(h×idX\),f\\circ\(\\rho\_\{M\}\\times\\mathrm\{id\}\_\{X\}\)=f\\circ\(\\rho\_\{N\}\\times\\mathrm\{id\}\_\{X\}\)\\circ\(h\\times\\mathrm\{id\}\_\{X\}\),withXXthe input space\. Read as a change of parameters for networks,hhtherefore points the other way, from theQNQ\_\{N\}\-network to theQMQ\_\{M\}\-network, and[Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)records this reversal with an opposite category \([Background B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

Used in\.[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2),[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\.

Background A\.8\(the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\)\. Combining[Background A\.6](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1)and[Background A\.7](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx1),𝐌𝐞𝐭𝐡⁡\(Θ,θ0\):=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\):=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)\. An object is a pointed smooth mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\), and a morphismh:M→Nh:M\\to Nis a pointed smooth map withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Equivalently, an object is a factorisationpt→q0Q→𝜌Θ\\mathrm\{pt\}\\xrightarrow\{\\,q\_\{0\}\\,\}Q\\xrightarrow\{\\,\\rho\\,\}\\Thetaof the pointθ0:pt→Θ\\theta\_\{0\}:\\mathrm\{pt\}\\to\\Theta, and a morphism is a map between the middle terms that is compatible with both ends\.

On one box, LoRA is\(ℝm×r×ℝr×n,\(0,A0\),\(B,A\)↦W0\+sBA\)\\big\(\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\},\\,\(0,A\_\{0\}\),\\,\(B,A\)\\mapsto W\_\{0\}\+sBA\\big\), and OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)isQ↦W0​C​\(Q\)Q\\mapsto W\_\{0\}\\,C\(Q\)on skew matrices \(block\-diagonal in OFT\), pointed at00, withCCthe Cayley map \([Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\)\. Full fine\-tuning is\(Θ,θ0,id\)\(\\Theta,\\theta\_\{0\},\\mathrm\{id\}\)and the frozen model is\(pt,∗,θ0\)\(\\mathrm\{pt\},\*,\\theta\_\{0\}\)\. The networkffappears nowhere in the definition, so the category depends only on\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)\([Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\)\. The factorisation picture already shows why the frozen model and full fine\-tuning sit at the two ends \([Background A\.9](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)\)\.

Used in\.Table[2](https://arxiv.org/html/2610.12728#S0.T2),[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2), Rosetta entries 3 and 4 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\)\.

#### Universal properties

Background A\.9\(initial and terminal objects; universal property\)\. An objectIIisinitialif every objectXXreceives exactly one arrowI→XI\\to X, andTTisterminalif everyXXhas exactly one arrowX→TX\\to T\. Each is unique up to a unique isomorphism when it exists\. Characterising an object by the arrows into or out of it in this way is called auniversal property\. In𝐒𝐞𝐭\\mathbf\{Set\}the empty set is initial and a one\-point set is terminal, and in𝐒𝐞𝐭∗\\mathbf\{Set\}\_\{\*\}and𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}the one\-point space is both\.

In𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)the frozen model\(pt,∗,θ0\)\(\\mathrm\{pt\},\*,\\theta\_\{0\}\)is initial\. The only pointed mappt→Q\\mathrm\{pt\}\\to Qis∗↦q0\*\\mapsto q\_\{0\}, and it lies overΘ\\Thetabecauseρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}\. Full fine\-tuning\(Θ,θ0,id\)\(\\Theta,\\theta\_\{0\},\\mathrm\{id\}\)is terminal, since an arrowhhfromMMmust satisfyid∘h=ρM\\mathrm\{id\}\\circ h=\\rho\_\{M\}\. So the unique arrow fromMMto Full FT isρM\\rho\_\{M\}itself, the merge, and every weight\-space method that starts atθ0\\theta\_\{0\}sits in the commuting triangle of Figure[15](https://arxiv.org/html/2610.12728#A1.F15)\.

MM\(Q,q0\)\(Q,q\_\{0\}\)\(pt,∗\)\(\\mathrm\{pt\},\*\)\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)q0q\_\{0\}ρ\\rhoθ0\\theta\_\{0\}Figure 15:The interval of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\. The left arrow is the only arrow out of the initial object, the right arrowρ\\rhois the only arrow into the terminal object, and the triangle commutes becauseρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}\. Figure[2](https://arxiv.org/html/2610.12728#S1.F2)\(a\) draws the same triangle in the main text, where a pointing defectρM​\(q0\)≠θ0\\rho\_\{M\}\(q\_\{0\}\)\\neq\\theta\_\{0\}leaves it commuting only up to that defect\.Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3), Figure[2](https://arxiv.org/html/2610.12728#S1.F2), Rosetta entries 3 and 4 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\), rule A1 in §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1), §[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx5)\.

Background A\.10\(product\)\. AproductofXXandYYis an objectX×YX\\times Ywith projectionspX:X×Y→Xp\_\{X\}:X\\times Y\\to XandpY:X×Y→Yp\_\{Y\}:X\\times Y\\to Ysuch that for every objectTTand arrowsu:T→Xu:T\\to X,v:T→Yv:T\\to Ythere is exactly one arrow⟨u,v⟩:T→X×Y\\langle u,v\\rangle:T\\to X\\times YwithpX∘⟨u,v⟩=up\_\{X\}\\circ\\langle u,v\\rangle=uandpY∘⟨u,v⟩=vp\_\{Y\}\\circ\\langle u,v\\rangle=v\. It is unique up to unique isomorphism, and it need not exist\. In a slice𝒞/Z\\mathcal\{C\}/Z, the product of\(X,a\)\(X,a\)and\(Y,b\)\(Y,b\)is the pullback ofaaandbbin𝒞\\mathcal\{C\}\([Background A\.11](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)\)\.

In𝐌𝐞𝐭𝐡\\mathbf\{Meth\}a productM×NM\\times Nhas arrows toMMand toNN, so both factors simulate it, and every methodTTwith arrowsu:T→Mu:T\\to Mandv:T→Nv:T\\to Nmaps to it by exactly one arrow compatible withuuandvv\. When it exists, its image isIm⁡M∩Im⁡N\\Imop M\\cap\\Imop N\. For the one\-sided LoRAs with frozen framesB0B\_\{0\}of full column rank andA0A\_\{0\}of full row rank,

FA\(A0\)×FB\(B0\)≅\(ℝr×r,0,R↦W0\+B0RA0\),\\mathrm\{FA\}\(A\_\{0\}\)\\times\\mathrm\{FB\}\(B\_\{0\}\)\\;\\cong\\;\\big\(\\mathbb\{R\}^\{r\\times r\},\\,0,\\,R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}\\big\),with projectionsR↦B0​RR\\mapsto B\_\{0\}RandR↦R​A0R\\mapsto RA\_\{0\}\. In the SVD frames ofW0W\_\{0\}, whenrank⁡W0≥r\\rk W\_\{0\}\\geq r, this is LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\)\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)\)\. The sum⊕\\oplusof[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)is a different operation, whose image is a Minkowski sum \([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\),[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6),[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3), rule A4 in §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1), §[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx3)\. Reading:[Leinster \(2014, §5\.1\)](https://arxiv.org/html/2610.12728#bib.bib242)\.

Background A\.11\(pullback; fibre product\)\. Givenf:X→Zf:X\\to Zandg:Y→Zg:Y\\to Z, apullbackis an objectPPwith arrowsp:P→Xp:P\\to Xandq:P→Yq:P\\to Ysuch thatf∘p=g∘qf\\circ p=g\\circ q, and such that wheneveru:T→Xu:T\\to Xandv:T→Yv:T\\to Ysatisfyf∘u=g∘vf\\circ u=g\\circ v, exactly onet:T→Pt:T\\to Phasp∘t=up\\circ t=uandq∘t=vq\\circ t=v\(Figure[16](https://arxiv.org/html/2610.12728#A1.F16)\)\. In𝐒𝐞𝐭\\mathbf\{Set\}the pullback is thefibre productX×ZY=\{\(x,y\):f⁡\(x\)=g⁡\(y\)\}X\\times\_\{Z\}Y=\\\{\(x,y\):f\(x\)=g\(y\)\\\}\. In𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, if this set is an embedded submanifold ofX×YX\\times Y, it is the pullback, and transversality \([Background C\.7](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)\) is the standard sufficient condition for that\. A pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}can fail to exist\.

Products in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}are fibre productsQM×ΘQN=\{\(q,q′\):ρM​\(q\)=ρN​\(q′\)\}Q\_\{M\}\\times\_\{\\Theta\}Q\_\{N\}=\\\{\(q,q^\{\\prime\}\):\\rho\_\{M\}\(q\)=\\rho\_\{N\}\(q^\{\\prime\}\)\\\}\. ForFA⁡\(A0\)\\mathrm\{FA\}\(A\_\{0\}\)andFB⁡\(B0\)\\mathrm\{FB\}\(B\_\{0\}\), with the ranks of[Background A\.10](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2), this set is the linear space\{\(B0​R,R​A0\)\}\\\{\(B\_\{0\}R,RA\_\{0\}\)\\\}, so the product exists\. ForLoRA1\\mathrm\{LoRA\}\_\{1\}and the frozen model on1×11\\times 1weights the set is the cross\{ba=0\}⊂ℝ2\\\{ba=0\\\}\\subset\\mathbb\{R\}^\{2\}, no manifold structure on it has the universal property, and the product does not exist \([Background A\.13](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)\)\. Pulling back along the pointθ0:pt→Θ\\theta\_\{0\}:\\mathrm\{pt\}\\to\\Thetagives the fibreρ−1​\(θ0\)\\rho^\{\-1\}\(\\theta\_\{0\}\), the underlying set ofM×FrozenM\\times\\text\{Frozen\}whenever that product exists\.

##### HRA is a meet of images, not a product

TakeW0W\_\{0\}injective andrreven with2≤r≤n−22\\leq r\\leq n\-2\. The fibre product ofLoRAr\\mathrm\{LoRA\}\_\{r\}with the rotation orbitW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)has local dimensiond⁡\(HRAr\)\+r2=r​n\+r⁡\(r−1\)/2d\(\\mathrm\{HRA\}\_\{r\}\)\+r^\{2\}=rn\+r\(r\-1\)/2over generic points of the meet \([Background C\.19](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\)\. This exceedsr​n=\|HRAr\|rn=\\lvert\\mathrm\{HRA\}\_\{r\}\\rvertbecauser≥2r\\geq 2, so HRA\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)is not the product of the two methods, and[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)keeps only the meet of their images \([Background A\.16](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\)\.

T\{\\lx@inpgf@ignorespaces T\}QM×ΘQN\{\\lx@inpgf@ignorespaces Q\_\{M\}\\times\_\{\\Theta\}Q\_\{N\}\}QN\{\\lx@inpgf@ignorespaces Q\_\{N\}\}QM\{\\lx@inpgf@ignorespaces Q\_\{M\}\}Θ\{\\lx@inpgf@ignorespaces\\Theta\}v\\scriptstyle\{\\lx@inpgf@ignorespaces v\}u\\scriptstyle\{\\lx@inpgf@ignorespaces u\}∃\!t\\scriptstyle\{\\lx@inpgf@ignorespaces\\exists\!\\,t\}pN\\scriptstyle\{\\lx@inpgf@ignorespaces p\_\{N\}\}pM\\scriptstyle\{\\lx@inpgf@ignorespaces p\_\{M\}\}⌟\{\\lx@inpgf@ignorespaces\\lrcorner\}ρN\\scriptstyle\{\\lx@inpgf@ignorespaces\\rho\_\{N\}\}ρM\\scriptstyle\{\\lx@inpgf@ignorespaces\\rho\_\{M\}\}Figure 16:A pullback square\. It commutes,ρM∘pM=ρN∘pN\\rho\_\{M\}\\circ p\_\{M\}=\\rho\_\{N\}\\circ p\_\{N\}, and any test objectTTwhose mapsu,vu,vagree overΘ\\Thetafactors through the corner by exactly onett\. The small corner mark is the usual sign of a pullback\.Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\),[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6),[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(d\),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\.

Background A\.12\(limit; creation of limits\)\. Aconeover a diagramDDin𝒞\\mathcal\{C\}is an objectLLwith an arrow to every object ofDD, such that each triangle formed with an arrow ofDDcommutes\. AlimitofDDis a cone through which every other cone factors by exactly one arrow\. This categorical cone is unrelated to the geometric cones of[Background C\.10](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\. Terminal objects, products and pullbacks are the limits of the empty diagram, of two objects, and of two arrows with a common target\. A functorU:𝒞→𝒟U:\\mathcal\{C\}\\to\\mathcal\{D\}creates limitsif, for every diagramDDin𝒞\\mathcal\{C\}and every limit cone ofU∘DU\\circ D, exactly one cone overDDis sent byUUto that limit cone, and this cone is a limit ofDD\.

The forgetful functor𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}creates limits \([Background A\.14](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)\)\. Concretely, when the underlying manifolds have a pullback, the pullback of pointed manifolds is that pullback, pointed at the pair of base points\. This is how[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) computes products of methods as fibre products in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\. Reading:[Mac Lane \(1998, Ch\. V\)](https://arxiv.org/html/2610.12728#bib.bib306);[Riehl \(2016, Ch\. 3\)](https://arxiv.org/html/2610.12728#bib.bib390)\.

Background A\.13\(adjunction\)\. A functorF:𝒞→𝒟F:\\mathcal\{C\}\\to\\mathcal\{D\}isleft adjointtoG:𝒟→𝒞G:\\mathcal\{D\}\\to\\mathcal\{C\}, andGGisright adjointtoFF, writtenF⊣GF\\dashv G, if there are bijections𝒟⁡\(F​X,Y\)≅𝒞⁡\(X,G​Y\)\\mathcal\{D\}\(FX,Y\)\\cong\\mathcal\{C\}\(X,GY\)natural inXXandYY\. Many forgetful functors are right adjoints, whose left adjoints add the forgotten structure freely\.

The forgetful functor𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}is right adjoint toX↦\(X⊔pt,pt\)X\\mapsto\(X\\sqcup\\mathrm\{pt\},\\mathrm\{pt\}\), which adds a disjoint base point, since a pointed map out ofX⊔ptX\\sqcup\\mathrm\{pt\}is the same as an arbitrary map out ofXX\(a manifold may have components of different dimensions\)\. Right adjoints preserve limits \([Background A\.14](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)\), so a product of methods, if it exists, is also a pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}and can be tested with unpointed maps\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) uses this to rule outLoRA1×Frozen\\mathrm\{LoRA\}\_\{1\}\\times\\text\{Frozen\}on1×11\\times 1weights\. The axis curvest↦\(t,0\)t\\mapsto\(t,0\)andt↦\(0,t\)t\\mapsto\(0,t\)in the cross would lift smoothly through the point over the origin, so the differential of the projection from the product to the parameter planeℝ2\\mathbb\{R\}^\{2\}ofLoRA1\\mathrm\{LoRA\}\_\{1\}would have rank22there\. That projection would then be a submersion near the point \([Background C\.4](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\), and its image would contain an open set, which the cross does not\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\. Reading:[Leinster \(2014, Ch\. 2 and Ch\. 6\)](https://arxiv.org/html/2610.12728#bib.bib242)\.

Fact A\.14\(two standard facts about limits\)\.Let𝒞\\mathcal\{C\}be a category\.\(a\)For every objectXXof𝒞\\mathcal\{C\}, the forgetful functorX/𝒞→𝒞X/\\mathcal\{C\}\\to\\mathcal\{C\}creates limits\. In particular𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}does\.\(b\)A right adjoint preserves every limit that exists in its domain\.

##### The idea behind \(a\) and \(b\)

\(a\) The structure arrowsX→DjX\\to D\_\{j\}of a diagram inX/𝒞X/\\mathcal\{C\}form a cone in𝒞\\mathcal\{C\}, which factors through the limit by exactly one arrow, and that arrow is the structure map of the limit\. \(b\) ForF⊣GF\\dashv G,

𝒞⁡\(X,G​limD\)≅𝒟⁡\(F​X,limD\)≅lim𝒟⁡\(F​X,Dj\)≅lim𝒞⁡\(X,G​Dj\),\\mathcal\{C\}\(X,G\\lim D\)\\cong\\mathcal\{D\}\(FX,\\lim D\)\\cong\\lim\\mathcal\{D\}\(FX,D\_\{j\}\)\\cong\\lim\\mathcal\{C\}\(X,GD\_\{j\}\),naturally inXX\.

[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) combines them\. Products in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}are pullbacks in𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}, which by \(a\) exist whenever the underlying pullback exists in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, and by \(b\) and[Background A\.13](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)only then\. The transversality hypothesis of[Background C\.7](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)fails at the base point whenever\|M\|\+\|N\|<dimΘ\\lvert M\\rvert\+\\lvert N\\rvert<\\dim\\Theta, with\|M\|=dimQM\\lvert M\\rvert=\\dim Q\_\{M\}\([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\), so the products that exist, such as LoRA\-XS, come from other structure \([Background A\.11](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)\)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\. Reading:[Mac Lane \(1998, Ch\. V\)](https://arxiv.org/html/2610.12728#bib.bib306);[Riehl \(2016, Ch\. 3 and Ch\. 4\)](https://arxiv.org/html/2610.12728#bib.bib390)\.

Background A\.15\(monomorphism; sub\-object\)\. An arrowm:A→Bm:A\\to Bis amonomorphismifm∘u=m∘vm\\circ u=m\\circ vimpliesu=vu=vfor all arrowsu,v:T→Au,v:T\\to A\. Asub\-objectofBBis a monomorphism intoBB, two of them being identified when they differ by an isomorphism overBB\. Testing with maps out of one\- and two\-point manifolds shows that the monomorphisms of𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, and those of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, are exactly the arrows whose underlying map is injective\. Such a map need not be an embedding, ast↦t3t\\mapsto t^\{3\}shows\.

Freezing coordinates at their base values gives an injective affine map, hence a sub\-object, written↪\\hookrightarrowin the atlas \(rule A2\)\. Examples areFA⁡\(A0\)↪LoRA\\mathrm\{FA\}\(A\_\{0\}\)\\hookrightarrow\\mathrm\{LoRA\}pointed at\(0,A0\)\(0,A\_\{0\}\), byB↦\(B,A0\)B\\mapsto\(B,A\_\{0\}\), and MiCA\([Rüdiger and Raschka, 2026](https://arxiv.org/html/2610.12728#bib.bib395)\)↪\{\}\\hookrightarrow\{\}LoRA pointed at\(B0,0\)\(B\_\{0\},0\), byA↦\(B0,A\)A\\mapsto\(B\_\{0\},A\)\. These maps are embeddings, a stronger property than injectivity\. The paper writes sub\-object and never monomorphism\. In[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9), LoRA\-FA is a sub\-object of LoRA and so a different object, while random\-AALoRA and EVA\([Paischer et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib358)\)are the same mapρ\\rhowith two different base points in its fibre overθ0\\theta\_\{0\}\.

Used in\.[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), rule A2 in §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\.

Background A\.16\(posets; meets; the expressivity order\)\. Apreorderis a category with at most one arrow between any two objects; writex≤yx\\leq ywhen there is one\. It is aposetifx≤y≤xx\\leq y\\leq xforcesx=yx=y\. In a poset the product ofxxandyyis theirmeetx∧yx\\wedge y, the greatest lower bound, and for subsets ordered by inclusion the meet is the intersection\. Two elements areincomparableif neitherx≤yx\\leq ynory≤xy\\leq x, andxxismaximalifx≤yx\\leq yimpliesx=yx=y\. Every category has aposet reflection, in whichX≤YX\\leq Ywhen some arrowX→YX\\to Yexists andX,YX,Yare identified when arrows run both ways\. AHasse diagramdraws a finite poset withyyabovexxwhenx<yx<y, joining them by an edge when nothing lies strictly between\.

The poset reflection of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, which we call the expressivity order, orders methods by simulation\. Figure[14](https://arxiv.org/html/2610.12728#S8.F14)draws it for the core methods as a layered Hasse diagram, from the frozen model at the bottom to full fine\-tuning at the top\. A meet of images always exists and need not be the image of a product\. For injectiveW0W\_\{0\}and evenr≤n−2r\\leq n\-2, the image of HRA without Gram–Schmidt is the meet of LoRA’s cone with the rotation orbitW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\([Background D\.11](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\), and HRA is not their product \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\. Split and zero\-init LoRA have incomparable images \([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\)\), so neither simulates the other\. The frontier problem asks for the maximal methods of budget at mostkk, which are automatically pairwise incomparable\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\),[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(b\),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), Figure[14](https://arxiv.org/html/2610.12728#S8.F14), the open problems of Exposé[XI](https://arxiv.org/html/2610.12728#S11)\. Reading:[Davey and Priestley \(2002, Ch\. 1 and Ch\. 2\)](https://arxiv.org/html/2610.12728#bib.bib90)\.

#### Base change and lifting

Background A\.17\(automorphism of a category; transport of structure\)\. Anautomorphismof a category𝒞\\mathcal\{C\}is a functorF:𝒞→𝒞F:\\mathcal\{C\}\\to\\mathcal\{C\}with an inverse functor, so that both composites are identity functors\. It carries initial objects, products, isomorphisms and every other notion defined from arrows alone to notions of the same kind\.Transport of structurebuilds such a functor from an invertible map of the underlying data\.

ForT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\)putτT​\(θ\)=θ0\+T⁡\(θ−θ0\)\\tau\_\{T\}\(\\theta\)=\\theta\_\{0\}\+T\(\\theta\-\\theta\_\{0\}\)andT∗​M:=\(Q,q0,τT∘ρ\)T\_\{\*\}M:=\(Q,q\_\{0\},\\tau\_\{T\}\\circ\\rho\)\. An arrowh:M→Nh:M\\to Nis also an arrowT∗​M→T∗​NT\_\{\*\}M\\to T\_\{\*\}N, sinceτT∘ρN∘h=τT∘ρM\\tau\_\{T\}\\circ\\rho\_\{N\}\\circ h=\\tau\_\{T\}\\circ\\rho\_\{M\}, soT∗T\_\{\*\}is a functor, with inverse\(T−1\)∗\(T^\{\-1\}\)\_\{\*\}\. It keepsd⁡\(M\)d\(M\)and the first\-order deficit \([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\) and the gauge group \([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\), and movesS1S\_\{1\}toT​S1TS\_\{1\}\. It does not keep rank, which depends on the matrix shape ofΘ\\ThetathatTTneed not respect\. Sums ofkkKronecker products are\(ℛ−1\)∗​LoRAk\(\\mathcal\{R\}^\{\-1\}\)\_\{\*\}\\mathrm\{LoRA\}\_\{k\}, LoRA on the matrix rearranged by Van Loan and Pitsianis and carried back \([Background B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\), and HiRA\([Huang et al\., 2025b](https://arxiv.org/html/2610.12728#bib.bib182)\)is\(DW0\)∗​LoRA\(D\_\{W\_\{0\}\}\)\_\{\*\}\\mathrm\{LoRA\}withDW0​\(X\)=W0⊙XD\_\{W\_\{0\}\}\(X\)=W\_\{0\}\\odot X, whenW0W\_\{0\}has no zero entries\. The atlas recordsMMandT∗​MT\_\{\*\}Mas one object in two frames, with no arrow between them\.

Used in\.[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3), rule T in §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1), §[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx3)\.

Background A\.18\(lift; lifting problem; obstruction\)\. Given arrowsp:E→Bp:E\\to Bandf:A→Bf:A\\to B, aliftofffalongppis an arrowg:A→Eg:A\\to Ewithp∘g=fp\\circ g=f\. Alifting problemasks whether a lift exists, and anobstructionis an invariant whose value shows that none does\. A morphismh:M→Nh:M\\to Nof𝐌𝐞𝐭𝐡\\mathbf\{Meth\}is exactly a pointed smooth lift ofρM\\rho\_\{M\}alongρN\\rho\_\{N\}\.

In𝐒𝐞𝐭\\mathbf\{Set\}a lift exists oncef⁡\(A\)⊆p⁡\(E\)f\(A\)\\subseteq p\(E\), by choosing preimages, so smoothness is the real constraint\. ForρM​\(t\)=t\\rho\_\{M\}\(t\)=tandρN​\(u\)=u3\\rho\_\{N\}\(u\)=u^\{3\}onℝ\\mathbb\{R\}the images agree, and the only lift is the cube root, which is not differentiable at00\. A pointed smooth lift forcesS1​\(M\)⊆S1​\(N\)S\_\{1\}\(M\)\\subseteq S\_\{1\}\(N\)by the chain rule, soS1S\_\{1\}is a first\-order obstruction, and hereS1​\(M\)=ℝ⊈S1​\(N\)=0S\_\{1\}\(M\)=\\mathbb\{R\}\\not\\subseteq S\_\{1\}\(N\)=0\. Fort3t^\{3\}againstu5u^\{5\}bothS1S\_\{1\}vanish, yet the only liftt3/5t^\{3/5\}is notC1C^\{1\}, so that obstruction is invisible toS1S\_\{1\}\. Merging a method on an extension is again a lifting problem, to findμ\\muwithΦf∘μ=Φf~∘ρ\\Phi\_\{f\}\\circ\\mu=\\Phi\_\{\\tilde\{f\}\}\\circ\\rho, a lift along the realisationΦf\\Phi\_\{f\}\([Background B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3),[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2), §[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2), Rosetta entry 5 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\)\.

Background A\.19\(base change; pushforwardϕ\!\\phi\_\{\!\}; the relative point of view\)\. A map of basesϕ:S′→S\\phi:S^\{\\prime\}\\to Sinduces thepushforwardϕ\!:𝒞/S′→𝒞/S\\phi\_\{\!\}:\\mathcal\{C\}/S^\{\\prime\}\\to\\mathcal\{C\}/S,\(X,x\)↦\(X,ϕ∘x\)\(X,x\)\\mapsto\(X,\\phi\\circ x\), by postcomposition\. When every object overSShas a pullback alongϕ\\phi, it also induces thepullbackϕ∗:𝒞/S→𝒞/S′\\phi^\{\*\}:\\mathcal\{C\}/S\\to\\mathcal\{C\}/S^\{\\prime\},\(Y,y\)↦\(Y×SS′,projection\)\(Y,y\)\\mapsto\(Y\\times\_\{S\}S^\{\\prime\},\\text\{projection\}\), and thenϕ\!⊣ϕ∗\\phi\_\{\!\}\\dashv\\phi^\{\*\}\. In algebraic geometry, base change usually meansϕ∗\\phi^\{\*\}\.

[Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1)\(a\) usesϕ\!\\phi\_\{\!\}on pointed slices\. A function\-preserving pointed reparametrisationϕ:\(Θ′,θ0′\)→\(Θ,θ0\)\\phi:\(\\Theta^\{\\prime\},\\theta\_\{0\}^\{\\prime\}\)\\to\(\\Theta,\\theta\_\{0\}\), withf∘\(ϕ×X\)=f′f\\circ\(\\phi\\times X\)=f^\{\\prime\}, givesϕ\!:𝐌𝐞𝐭𝐡\(Θ′,θ0′\)→𝐌𝐞𝐭𝐡\(Θ,θ0\)\\phi\_\{\!\}:\\mathbf\{Meth\}\(\\Theta^\{\\prime\},\\theta\_\{0\}^\{\\prime\}\)\\to\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\),ρ↦ϕ∘ρ\\rho\\mapsto\\phi\\circ\\rho, for instance for a permutation of hidden units\. Whetherϕ\!\\phi\_\{\!\}carries LoRA to LoRA is the covariance question of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\. A quantiser changes the function and induces no such functor, so QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)and ReLoRA\([Lialin et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib258)\)only borrow the name, and Exposé[V](https://arxiv.org/html/2610.12728#S5)computes pointing defects and reachable sets for them\.

Grothendieck’s relative point of view replaces a spaceXXby a morphismX→SX\\to S, an object over a base\. Properties then belong to the morphism, and the useful ones are those that survive pulling back along every change of baseS′→SS^\{\\prime\}\\to S\. The fibre over a pointssis itself the pullback alongs:pt→Ss:\\mathrm\{pt\}\\to S\. The paper takes this stance in a modest form\. A method is a morphism over the base pointθ0\\theta\_\{0\}, and quantisation, restarts and merging are read as moves of that base point\.

Used in\.Exposés[0](https://arxiv.org/html/2610.12728#S0)–[I](https://arxiv.org/html/2610.12728#S1), Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\),[Definition V\.1](https://arxiv.org/html/2610.12728#S5.SS0.SSSx1), Exposé[V](https://arxiv.org/html/2610.12728#S5), Rosetta entry 28 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\), the seven borrowed words of Exposé[XI](https://arxiv.org/html/2610.12728#S11)\. Reading:[Vakil \(2025\)](https://arxiv.org/html/2610.12728#bib.bib455), on fibred products and base change\.

Background A\.20\(constructions the paper names and leaves unused\)\. Coends, monads, sheaves, stacks, schemes, Grothendieck’s descent theory \(descent data, effective descent morphisms\), the Grothendieck construction∫𝐌𝐞𝐭𝐡→𝐏𝐍𝐞𝐭\\int\\mathbf\{Meth\}\\to\\mathbf\{PNet\}and coKleisli 2\-cells appear in the paper only in lists of constructions it leaves out \(Exposé[0](https://arxiv.org/html/2610.12728#S0), where the category theory earns its keep, and Exposé[XI](https://arxiv.org/html/2610.12728#S11), on what the borrowed words mean\)\. No result depends on them\. The elementary descent of[Background D\.22](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3), factoring a map that is constant on orbits through the orbit space, is a different and much simpler notion, and several results use it\.

For orientation only, the Grothendieck construction would assemble the slices of all pretrained models,𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)for each\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\), into one category of pairs \(network, method\), with a projection to the category𝐏𝐍𝐞𝐭\\mathbf\{PNet\}whose objects are pointed networks and whose arrows are function\-preserving reparametrisations\. A monad is an endofunctorTTwith natural transformationsid⇒T\\mathrm\{id\}\\Rightarrow TandT​T⇒TTT\\Rightarrow Tsatisfying unit and associativity laws, and a sheaf assigns data to open sets so that compatible local data glue to unique global data\.

Used in\.Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\), the seven borrowed words and what the borrowed words mean in Exposé[XI](https://arxiv.org/html/2610.12728#S11)\.

### Part B\. Monoidal categories, string diagrams and lenses

Part A compared methods through the arrows between them\. Part B covers the algebra behind drawing a method as boxes and wires, reading a rank bound off a cut of the drawing, and treating backpropagation as a pair of maps\. Entries[Background B\.1](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)to[Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)set up monoidal categories and their diagrams,[Background B\.7](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2)to[Background B\.13](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)turn them into rank and dimension counts for matrices built from small factors, and[Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)to[Background B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)cover sums of methods, parametrised maps and lenses\.

#### Monoidal categories

Background B\.1\(monoidal and symmetric monoidal categories\)\. Amonoidal categoryis a category𝒞\\mathcal\{C\}with a functor⊗:𝒞×𝒞→𝒞\\otimes:\\mathcal\{C\}\\times\\mathcal\{C\}\\to\\mathcal\{C\}, a unit objectII, and natural isomorphisms\(A⊗B\)⊗C≅A⊗\(B⊗C\)\(A\\otimes B\)\\otimes C\\cong A\\otimes\(B\\otimes C\)andI⊗A≅A≅A⊗II\\otimes A\\cong A\\cong A\\otimes I, thecoherence isomorphisms, which satisfy the pentagon and triangle equations\. Mac Lane’s coherence theorem then makes every diagram built from them commute, so brackets can be dropped\. The category issymmetricif it also has natural isomorphismsσA,B:A⊗B→B⊗A\\sigma\_\{A,B\}:A\\otimes B\\to B\\otimes AwithσB,A​σA,B=id\\sigma\_\{B,A\}\\sigma\_\{A,B\}=\\mathrm\{id\}that are compatible with the associator \(the hexagon equation\)\. Examples are\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗,ℝ\)\(\\mathbf\{FinVect\},\\otimes,\\mathbb\{R\}\),\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊕,0\)\(\\mathbf\{FinVect\},\\oplus,0\)and\(𝐃𝐢𝐟𝐟,×,pt\)\(\\mathbf\{Diff\},\\times,\\mathrm\{pt\}\)\.

On methods,M⊕NM\\oplus Nruns the two parameter spaces side by side and adds their updates \([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\. This makes𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)symmetric monoidal with the frozen model as unit\. Its coherence isomorphisms are those of×\\timeson pointed manifolds, and they lie overΘ\\Thetabecause addition is associative and commutative\.

Used in\.[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\),[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1),[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\. Reading:[Mac Lane \(1998, Ch\. VII and Ch\. XI\)](https://arxiv.org/html/2610.12728#bib.bib306)\.

Background B\.2\(direct sum and tensor product in𝐅𝐢𝐧𝐕𝐞𝐜𝐭\\mathbf\{FinVect\}\)\. The category𝐅𝐢𝐧𝐕𝐞𝐜𝐭\\mathbf\{FinVect\}has finite\-dimensional real vector spaces as objects and linear maps as morphisms, and two monoidal products\. Thedirect sumV⊕WV\\oplus Wis abiproduct, both the categorical product and the coproduct, withdim\(V⊕W\)=dimV\+dimW\\dim\(V\\oplus W\)=\\dim V\+\\dim W; on matricesf⊕gf\\oplus gis block\-diagonal\. Thetensor productV⊗WV\\otimes Whasdim\(V⊗W\)=dimV⋅dimW\\dim\(V\\otimes W\)=\\dim V\\cdot\\dim Wand is not a categorical product; on matricesf⊗gf\\otimes gis the Kronecker product \([Background B\.10](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\)\. Each bilinear mapV×W→UV\\times W\\to Ufactors through a unique linear mapV⊗W→UV\\otimes W\\to U\.

Two boxes drawn side by side therefore mean a block\-diagonal map in the activation diagrams of Exposé[VI](https://arxiv.org/html/2610.12728#S6), whose monoidal product is×=⊕\\times=\\oplus, and a Kronecker product in a tensor network\. Dimensions multiply along⊗\\otimes, which is why the weight of a cut in[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)is a product of wire dimensions\. LoRA’s residual branchW0\+B​AW\_\{0\}\+BAuses⊕\\oplus, since it sends one input to two branches and adds their outputs \(Figure[17](https://arxiv.org/html/2610.12728#A1.F17)\)\.

Used in\.[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3),[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4),[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\.

Background B\.3\(string diagrams\)\. Astring diagramdraws a morphismf:A→Bf:A\\to Bof a monoidal category as a box with an input wireAAand an output wireBB\. Compositiong∘fg\\circ fjoins boxes in series,f⊗gf\\otimes gplaces them side by side, an identity is a bare wire, the unitIIis drawn as no wire, and in a symmetric categoryσ\\sigmais a crossing\.[Joyal and Street \(1991\)](https://arxiv.org/html/2610.12728#bib.bib202)proved that the pictures are sound and complete\. In a monoidal category, diagrams related by a planar deformation denote the same morphism; in a symmetric one, diagrams with the same boxes joined in the same way do\. Laws such as\(g⊗g′\)∘\(f⊗f′\)=\(g∘f\)⊗\(g′∘f′\)\(g\\otimes g^\{\\prime\}\)\\circ\(f\\otimes f^\{\\prime\}\)=\(g\\circ f\)\\otimes\(g^\{\\prime\}\\circ f^\{\\prime\}\)can then be read off the picture\.

The plug of Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\(§[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5)\) is a string diagram\. LoRA is a boxA:ℝn→ℝrA:\\mathbb\{R\}^\{n\}\\to\\mathbb\{R\}^\{r\}followed byB:ℝr→ℝmB:\\mathbb\{R\}^\{r\}\\to\\mathbb\{R\}^\{m\}, drawn besideW0W\_\{0\}\(Figure[17](https://arxiv.org/html/2610.12728#A1.F17), read left to right\)\. The merge calculus moves an edit along wires by equalities of maps, so a successful slide proves that the merged network computes the same function\.

Used in\.Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3),[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), §[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5), §[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)and the analogy on string diagrams there,[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\. Reading:[Selinger \(2011\)](https://arxiv.org/html/2610.12728#bib.bib402)\.

Background B\.4\(cartesian categories; copy and discard\)\. A monoidal category iscartesianwhen⊗\\otimesis the categorical product×\\timesandIIis a terminal object11\. Every object then has acopymapΔX:X→X×X\\Delta\_\{X\}:X\\to X\\times X,x↦\(x,x\)x\\mapsto\(x,x\), and adiscardmap\!X:X→1\{\!\}\_\{X\}:X\\to 1, and copy is natural,\(f×f\)∘ΔX=ΔY∘f\(f\\times f\)\\circ\\Delta\_\{X\}=\\Delta\_\{Y\}\\circ ffor everyf:X→Yf:X\\to Y\. Conversely, by a theorem of[Fox \(1976\)](https://arxiv.org/html/2610.12728#bib.bib115), a symmetric monoidal category is cartesian exactly when every object carries a commutative comonoid, a copy and a discard map obeying coassociativity, counit and commutativity laws \([Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\), compatible with⊗\\otimesand natural in all morphisms\.𝐒𝐞𝐭\\mathbf\{Set\},𝐃𝐢𝐟𝐟\\mathbf\{Diff\}and\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊕\)\(\\mathbf\{FinVect\},\\oplus\)are cartesian\.\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗\)\(\\mathbf\{FinVect\},\\otimes\)is not, sincex↦x⊗xx\\mapsto x\\otimes xis not linear\.

Naturality of copy is rule R7 of the merge calculus\. A linear editEEon a wire read byWQW\_\{Q\},WKW\_\{K\}andWVW\_\{V\}equalsEEapplied on each of the three branches, so it is absorbed asWQ​EW\_\{Q\}E,WK​EW\_\{K\}E,WV​EW\_\{V\}E\. The paper builds𝐏𝐚𝐫𝐚⁡\(𝒞\)\\mathbf\{Para\}\(\\mathcal\{C\}\)\([Background B\.16](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\) over a cartesian𝒞\\mathcal\{C\}\. The linear copy of[Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)copies basis vectors only and is a different map\.

Used in\.[Definition I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), §[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)and the analogy on string diagrams there,[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\.

Background B\.5\(compact closed categories; cups, caps and wire\-bending\)\. A symmetric monoidal category iscompact closedif every objectAAhas a dualA∗A^\{\*\}with acupηA:I→A∗⊗A\\eta\_\{A\}:I\\to A^\{\*\}\\otimes Aand acapεA:A⊗A∗→I\\varepsilon\_\{A\}:A\\otimes A^\{\*\}\\to Isatisfying the snake equations\(εA⊗idA\)​\(idA⊗ηA\)=idA,\(idA∗⊗εA\)​\(ηA⊗idA∗\)=idA∗,\(\\varepsilon\_\{A\}\\otimes\\mathrm\{id\}\_\{A\}\)\(\\mathrm\{id\}\_\{A\}\\otimes\\eta\_\{A\}\)=\\mathrm\{id\}\_\{A\},\\qquad\(\\mathrm\{id\}\_\{A^\{\*\}\}\\otimes\\varepsilon\_\{A\}\)\(\\eta\_\{A\}\\otimes\\mathrm\{id\}\_\{A^\{\*\}\}\)=\\mathrm\{id\}\_\{A^\{\*\}\},with the unitorsI⊗A≅A≅A⊗II\\otimes A\\cong A\\cong A\\otimes Iof[Background B\.1](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)suppressed\. Drawn as bent wires, cups and caps turn an input wire into an output wire and back\. In𝐅𝐢𝐧𝐕𝐞𝐜𝐭\\mathbf\{FinVect\}the cap is evaluation, the cup is1↦∑iei∗⊗ei1\\mapsto\\sum\_\{i\}e\_\{i\}^\{\*\}\\otimes e\_\{i\}, and bending givesHom⁡\(V,U\)≅U⊗V∗\\mathrm\{Hom\}\(V,U\)\\cong U\\otimes V^\{\*\}\. With a fixed basis, a box slid around a cup or cap comes out as its transpose\.

Bending the input wire of anm×nm\\times nmatrix gives its vectorisation inℝm⊗ℝn\\mathbb\{R\}^\{m\}\\otimes\\mathbb\{R\}^\{n\}, and the Van Loan–Pitsianis rearrangement \([Background B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\) is a composite of bends and crossings\. Rule R5 of the merge calculus,⟨M​q,k⟩=⟨q,M⊤​k⟩\\langle Mq,k\\rangle=\\langle q,M^\{\\top\}k\\rangle, slidesMMaround the cap that computes an attention score\. The cut\-rank bound uses only⊗\\otimesand crossings\.

Used in\.[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\. Reading:[Coecke and Kissinger \(2017\)](https://arxiv.org/html/2610.12728#bib.bib80)\.

Background B\.6\(Frobenius algebras and spiders\)\. AFrobenius algebraon an objectVVis a monoidμ:V⊗V→V\\mu:V\\otimes V\\to V,η:I→V\\eta:I\\to Vand a comonoidδ:V→V⊗V\\delta:V\\to V\\otimes V,ε:V→I\\varepsilon:V\\to Iobeying theFrobenius law\(μ⊗id\)​\(id⊗δ\)=δ​μ=\(id⊗μ\)​\(δ⊗id\)\.\(\\mu\\otimes\\mathrm\{id\}\)\(\\mathrm\{id\}\\otimes\\delta\)=\\delta\\mu=\(\\mathrm\{id\}\\otimes\\mu\)\(\\delta\\otimes\\mathrm\{id\}\)\.A basise1,…,ene\_\{1\},\\dots,e\_\{n\}ofℝn\\mathbb\{R\}^\{n\}gives one, with*copy*δ⁡\(ei\)=ei⊗ei\\delta\(e\_\{i\}\)=e\_\{i\}\\otimes e\_\{i\},*merge*\(the multiplication; unrelated to merging an adapter into the weights\)μ⁡\(ei⊗ej\)=ei\\mu\(e\_\{i\}\\otimes e\_\{j\}\)=e\_\{i\}ifi=ji=jand00otherwise,η=∑iei\\eta=\\sum\_\{i\}e\_\{i\}andε⁡\(ei\)=1\\varepsilon\(e\_\{i\}\)=1\. It is commutative and special \(μ​δ=id\\mu\\delta=\\mathrm\{id\}\)\. For such algebras thespider theoremsays that every connected diagram ofμ,η,δ,ε\\mu,\\eta,\\delta,\\varepsilonwithk≥1k\\geq 1inputs andlloutputs equals onespider, which sendsei⊗ke\_\{i\}^\{\\otimes k\}toei⊗le\_\{i\}^\{\\otimes l\}and every other basis tensor to00\.

Merge multiplies coordinates,μ⁡\(x⊗y\)=x⊙y\\mu\(x\\otimes y\)=x\\odot y\. So for matricesX,Y:ℝn→ℝmX,Y:\\mathbb\{R\}^\{n\}\\to\\mathbb\{R\}^\{m\}the Hadamard product is the spider composite

X⊙Y=μm∘\(X⊗Y\)∘δn\.X\\odot Y=\\mu\_\{m\}\\circ\(X\\otimes Y\)\\circ\\delta\_\{n\}\.LoHa\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\)is a copy spider on the input feedingA1A\_\{1\}andA2A\_\{2\}, thenB1B\_\{1\}andB2B\_\{2\}, then a merge spider on the output, and in[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)a spider split by the cut counts its dimension once\. The copy and⊙\\odotnodes of Exposé[VI](https://arxiv.org/html/2610.12728#S6)live in a cartesian category \([Background B\.4](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\), where copy isx↦\(x,x\)x\\mapsto\(x,x\); together they fail the Frobenius law, so there “spider” is only a name\.

Used in\.Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3),[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), the analogy on string diagrams in §[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\. Reading:[Coecke et al\. \(2013\)](https://arxiv.org/html/2610.12728#bib.bib81)\.

#### Tensor networks and rank

Background B\.7\(tensor networks, contraction, bond dimension\)\. Atensor networkΓ\\Gammais a string diagram in\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗\)\(\\mathbf\{FinVect\},\\otimes\), possibly with spiders\. Each box is a tensor and each wire a finite\-dimensional space\. A wire joining two boxes meanscontraction, a sum over the index the wire carries\. The open wires \(legs\) are the inputsVVand outputsUU, and contracting every internal wire gives the value⟦Γ⟧:V→U\\llbracket\\Gamma\\rrbracket:V\\to U\. The dimension of an internal wire is itsbond dimension\. One value has many presentations, and quantities read off a network, such as its cuts, depend on the presentation\.

LoRA’s update is the two\-box networkB∘AB\\circ Awith one bond of dimensionrr, andΔ​W=B​A\\Delta W=BAcontracts it,\(B​A\)i​j=∑k≤rBi​k​Ak​j\(BA\)\_\{ij\}=\\sum\_\{k\\leq r\}B\_\{ik\}A\_\{kj\}\. HiRA factorsW0W\_\{0\}through a bond of dimensionrk⁡W0\\operatorname\{rk\}W\_\{0\}, while MoRA\([Jiang et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib196)\), C3A\([Chen et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib59)\)and MPO adapters have no narrow cut between inputs and outputs \(Table[8](https://arxiv.org/html/2610.12728#S2.T8), after[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\)\.

Used in\.[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3), §[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5), Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\), the known results credited in Exposé[XII](https://arxiv.org/html/2610.12728#S12)\. Reading:[Orús \(2014\)](https://arxiv.org/html/2610.12728#bib.bib353)\.

Fact B\.8\(the min\-cut bound on rank\)\.LetΓ\\Gammabe a tensor network with input legsVVand output legsUU, with every wire attached to a box \(put an identity box on any bare strand\)\. Split the boxes into an input sideℐ\\mathcal\{I\}and an output side𝒪\\mathcal\{O\}\. Theweightwwof the cut is the product of the dimensions of the wires joining the two sides and of the legs attached on the wrong side \(input legs at𝒪\\mathcal\{O\}, output legs atℐ\\mathcal\{I\}\); a spider with legs on both sides contributes its dimension once\. Thenrank⁡⟦Γ⟧≤mincuts⁡w\.\\rk\\llbracket\\Gamma\\rrbracket\\leq\\min\_\{\\text\{cuts\}\}w\.For sums of networks,rank⁡\(X\+Y\)≤rank⁡X\+rank⁡Y\\rk\(X\+Y\)\\leq\\rk X\+\\rk Y\.

##### The idea of the proof

Contract each side separately\. The value becomes a map into the tensor product of the crossing wires and wrong\-side legs followed by a map out of it, and that space has dimensionww\.

For LoRA the cut betweenAAandBBcrosses one wire, sorank⁡B​A≤r\\rk BA\\leq r\(Figure[17](https://arxiv.org/html/2610.12728#A1.F17)\)\. For LoHa the cut withA1,A2A\_\{1\},A\_\{2\}on the input side crosses two bonds and givesr1​r2r\_\{1\}r\_\{2\}\. The bound depends on the presentation\. HRA’s product ofrrreflections expands into2r−12^\{r\}\-1terms of rank at most one, while the telescoping sum

R−I=∑k=1r\(Hk−I\)Hk\+1⋯HrR\-I=\\sum\_\{k=1\}^\{r\}\(H\_\{k\}\-I\)H\_\{k\+1\}\\cdots H\_\{r\}givesrr\.

W0W\_\{0\}AABBnnmmrrrrFigure 17:LoRA as a string diagram, read left to right, with the scalessabsorbed intoBB\. The fork sends the input to both branches and the\+\+node adds their outputs\. These are the copy and sum maps of the biproduct⊕\\oplus\([Background B\.2](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\), so the diagram saysW=W0\+B​AW=W\_\{0\}\+BA\. The frozenW0W\_\{0\}is grey, and the trainableA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}andB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}are tinted\. The dashed cut separatesAAfromBBand crosses one wire of dimensionrr, sorank⁡\(W−W0\)=rank⁡B​A≤r\\rk\(W\-W\_\{0\}\)=\\rk BA\\leq r\. ForWWitself, subadditivity over the two branches gives onlyrank⁡W≤rank⁡W0\+r\\rk W\\leq\\rk W\_\{0\}\+r\.Used in\.[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3), §[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), the known results credited in Exposé[XII](https://arxiv.org/html/2610.12728#S12)\. Reading:[Kolda and Bader \(2009\)](https://arxiv.org/html/2610.12728#bib.bib222)\.

Background B\.9\(tensor trains, Tucker and MPO; gauge on bonds\)\. Atensor train\(TT\) writes a tensor with legs of dimensionsn1,…,ndn\_\{1\},\\dots,n\_\{d\}as a chain of coresGk∈ℝrk−1×nk×rkG\_\{k\}\\in\\mathbb\{R\}^\{r\_\{k\-1\}\\times n\_\{k\}\\times r\_\{k\}\},r0=rd=1r\_\{0\}=r\_\{d\}=1, joined by bonds of dimensionsr1,…,rd−1r\_\{1\},\\dots,r\_\{d\-1\}\. AnMPOis the same chain with one input and one output leg on each core\.Tuckerjoins a coreC∈ℝr1×⋯×rdC\\in\\mathbb\{R\}^\{r\_\{1\}\\times\\cdots\\times r\_\{d\}\}to factor matricesUk∈ℝnk×rkU\_\{k\}\\in\\mathbb\{R\}^\{n\_\{k\}\\times r\_\{k\}\}\. Insertingg−1​g=idg^\{\-1\}g=\\mathrm\{id\},g∈GLreg\\in\\mathrm\{GL\}\_\{r\_\{e\}\}, on a bondeeand absorbing each factor into its neighbouring box leaves the value unchanged, so the gauge \([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\) contains∏eGLre\\prod\_\{e\}\\mathrm\{GL\}\_\{r\_\{e\}\}\. When everyrer\_\{e\}is the rank of the matching unfolding \(minimal ranks\), the tensors of these ranks form a manifold of dimension∑\(core sizes\)−∑ere2\\sum\(\\text\{core sizes\}\)\-\\sum\_\{e\}r\_\{e\}^\{2\}\.

LoRA is the two\-core case,BBandAAjoined by one bond, with gauge\(B,A\)↦\(B​g−1,g​A\)\(B,A\)\\mapsto\(Bg^\{\-1\},gA\)and, forr≤min⁡\(m,n\)r\\leq\\min\(m,n\), image dimensionr⁡\(m\+n\)−r2r\(m\+n\)\-r^\{2\}\. TT and Tucker adapters haved⁡\(M\)=\|M\|−∑ere2d\(M\)=\\lvert M\\rvert\-\\sum\_\{e\}r\_\{e\}^\{2\}at minimal ranks \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\), and under gradient flow each bond carries a conserved matrix \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\)\.

Used in\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3), §[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\. Reading:[Oseledets \(2011\)](https://arxiv.org/html/2610.12728#bib.bib354);[Holtz et al\. \(2012\)](https://arxiv.org/html/2610.12728#bib.bib172);[Koch and Lubich \(2010\)](https://arxiv.org/html/2610.12728#bib.bib220)\.

Background B\.10\(Kronecker product, vec, tensor product of subspaces\)\. ForC∈ℝm1×n1C\\in\\mathbb\{R\}^\{m\_\{1\}\\times n\_\{1\}\}andD∈ℝm2×n2D\\in\\mathbb\{R\}^\{m\_\{2\}\\times n\_\{2\}\}, theKronecker productC⊗D∈ℝm1​m2×n1​n2C\\otimes D\\in\\mathbb\{R\}^\{m\_\{1\}m\_\{2\}\\times n\_\{1\}n\_\{2\}\}is the block matrix\[ci​j​D\]\[c\_\{ij\}D\], the matrix ofC⊗DC\\otimes Din the product basis\. It obeys the mixed\-product rule\(C⊗D\)​\(E⊗F\)=C​E⊗D​F\(C\\otimes D\)\(E\\otimes F\)=CE\\otimes DF, andrank⁡\(C⊗D\)=rank⁡C⋅rank⁡D\\rk\(C\\otimes D\)=\\rk C\\cdot\\rk D\. The mapu⊗v↦u​v⊤u\\otimes v\\mapsto uv^\{\\top\}identifiesℝm⊗ℝn\\mathbb\{R\}^\{m\}\\otimes\\mathbb\{R\}^\{n\}withℝm×n\\mathbb\{R\}^\{m\\times n\}\. For subspacesU⊆ℝmU\\subseteq\\mathbb\{R\}^\{m\}andV⊆ℝnV\\subseteq\\mathbb\{R\}^\{n\},U⊗VU\\otimes Vis then the space of matrices with column space inUUand row space inVV, of dimensiondimU⋅dimV\\dim U\\cdot\\dim V\.

LoKr’s\([Yeh et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib532)\)update isC⊗B​AC\\otimes BAwithCCranging over allm1×n1m\_\{1\}\\times n\_\{1\}matrices,B∈ℝm2×r′B\\in\\mathbb\{R\}^\{m\_\{2\}\\times r^\{\\prime\}\}andA∈ℝr′×n2A\\in\\mathbb\{R\}^\{r^\{\\prime\}\\times n\_\{2\}\}\. For inner rankr′≤min⁡\(m2,n2\)r^\{\\prime\}\\leq\\min\(m\_\{2\},n\_\{2\}\)its rankrk⁡C⋅rk⁡\(B​A\)\\operatorname\{rk\}C\\cdot\\operatorname\{rk\}\(BA\)is at mostmin⁡\(m1,n1\)​r′\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}, and this is attained\. SoIm⁡LoKr\\Imop\\mathrm\{LoKr\}lies inside the imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}of zero\-initLoRAr\\mathrm\{LoRA\}\_\{r\}exactly whenr≥min⁡\(m1,n1\)​r′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}\([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(e\)\)\. At a LoRA point\(B,A\)\(B,A\), the directions reached by moving one factor alone overlap in

\{X​A\}∩\{B​Y\}=col⁡B⊗row⁡A,\\\{XA\\\}\\cap\\\{BY\\\}=\\operatorname\{col\}B\\otimes\\operatorname\{row\}A,of dimensionrk⁡A⋅rk⁡B\\operatorname\{rk\}A\\cdot\\operatorname\{rk\}B\([Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)\)\.

Used in\.[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2), Exposé[II\.C](https://arxiv.org/html/2610.12728#S2.SS3),[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), the[LoHa / LoKr example](https://arxiv.org/html/2610.12728#S7.SS0.SSS0.Px1)of Exposé[VII](https://arxiv.org/html/2610.12728#S7)\. Reading:[Horn and Johnson \(1991, Ch\. 4\)](https://arxiv.org/html/2610.12728#bib.bib173)\.

Fact B\.11\(Van Loan–Pitsianis rearrangement; Kronecker rank\)\.Letm=m1​m2m=m\_\{1\}m\_\{2\}andn=n1​n2n=n\_\{1\}n\_\{2\}, and cutΔ∈ℝm×n\\Delta\\in\\mathbb\{R\}^\{m\\times n\}into anm1×n1m\_\{1\}\\times n\_\{1\}grid ofm2×n2m\_\{2\}\\times n\_\{2\}blocksΔi​j\\Delta\_\{ij\}\. Therearrangementℛ⁡\(Δ\)∈ℝm1​n1×m2​n2\\mathcal\{R\}\(\\Delta\)\\in\\mathbb\{R\}^\{m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}\}has the rowsvec⁡\(Δi​j\)⊤\\vect\(\\Delta\_\{ij\}\)^\{\\top\}, ordered withiirunning fastest\. Thenℛ⁡\(C⊗D\)=vec⁡\(C\)​vec⁡\(D\)⊤,\\mathcal\{R\}\(C\\otimes D\)=\\vect\(C\)\\,\\vect\(D\)^\{\\top\},andℛ\\mathcal\{R\}is a linear bijection that permutes entries, hence a Frobenius isometry\. So theKronecker rankrk⊗⁡Δ\\operatorname\{rk\}\_\{\\otimes\}\\Delta, the leastkkwithΔ=∑i≤kCi⊗Di\\Delta=\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}, equalsrank⁡ℛ⁡\(Δ\)\\rk\\mathcal\{R\}\(\\Delta\), and the nearest sum ofkkKronecker products in Frobenius norm comes from the rank\-kktruncated SVD ofℛ⁡\(Δ\)\\mathcal\{R\}\(\\Delta\)\(Van Loan and Pitsianis[1993](https://arxiv.org/html/2610.12728#bib.bib459)\)\.

##### The idea of the proof

Block\(i,j\)\(i,j\)ofC⊗DC\\otimes Disci​j​Dc\_\{ij\}D, so its row inℛ\\mathcal\{R\}isci​j​vec⁡\(D\)⊤c\_\{ij\}\\,\\vect\(D\)^\{\\top\}\. Linearity extends this to sums, and Eckart–Young applies toℛ⁡\(Δ\)\\mathcal\{R\}\(\\Delta\)becauseℛ\\mathcal\{R\}preserves the Frobenius norm\.

In string\-diagram termsℛ\\mathcal\{R\}bends and crosses wires, regrouping the four legs ofΔ\\DeltafromU1​U2\|V1​V2U\_\{1\}U\_\{2\}\\mid V\_\{1\}V\_\{2\}toU1​V1\|U2​V2U\_\{1\}V\_\{1\}\\mid U\_\{2\}V\_\{2\}\. A sum ofkkKronecker products is thereforeLoRAk\\mathrm\{LoRA\}\_\{k\}transported byℛ−1\\mathcal\{R\}^\{\-1\}\([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\), with rank taken across another cut\. ForN=s2N=s^\{2\},IN=Is⊗IsI\_\{N\}=I\_\{s\}\\otimes I\_\{s\}has Kronecker rank11and rankNN\. Figure[6](https://arxiv.org/html/2610.12728#S2.F6)shows the singular values across the ordinary cut, this Kronecker cut and a third one, for a random rank\-one update, the identity and a Kronecker productC⊗DC\\otimes D\. In this paper a*Kronecker sum*means a sum of Kronecker products; matrix analysis uses the same name forC⊗I\+I⊗DC\\otimes I\+I\\otimes D\.

Used in\.[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4), Figure[6](https://arxiv.org/html/2610.12728#S2.F6),[Prediction X\.5](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3), the known results and key references of Exposé[XII](https://arxiv.org/html/2610.12728#S12)\. Reading:[Van Loan \(2000\)](https://arxiv.org/html/2610.12728#bib.bib458)\.

Background B\.12\(Schmidt and operator\-Schmidt decompositions\)\. Everyu∈ℝm1⊗ℝm2u\\in\\mathbb\{R\}^\{m\_\{1\}\}\\otimes\\mathbb\{R\}^\{m\_\{2\}\}has aSchmidt decompositionu=∑i≤ρσi​xi⊗yiu=\\sum\_\{i\\leq\\rho\}\\sigma\_\{i\}x\_\{i\}\\otimes y\_\{i\}withσi\>0\\sigma\_\{i\}\>0and orthonormal families\(xi\)\(x\_\{i\}\),\(yi\)\(y\_\{i\}\)\. It is the SVD of them1×m2m\_\{1\}\\times m\_\{2\}reshape ofuu\(x⊗y↦x​y⊤x\\otimes y\\mapsto xy^\{\\top\}\), and theSchmidt rankρu\\rho\_\{u\}is the rank of that reshape, at mostmin⁡\(m1,m2\)\\min\(m\_\{1\},m\_\{2\}\)\. The SVD ofℛ⁡\(Δ\)\\mathcal\{R\}\(\\Delta\)likewise gives theoperator\-Schmidt decompositionΔ=∑k≤Ksk​Ck⊗Dk\\Delta=\\sum\_\{k\\leq K\}s\_\{k\}C\_\{k\}\\otimes D\_\{k\}, withsk\>0s\_\{k\}\>0, Frobenius\-orthonormal families\(Ck\)\(C\_\{k\}\)and\(Dk\)\(D\_\{k\}\), andK=rk⊗⁡ΔK=\\operatorname\{rk\}\_\{\\otimes\}\\Delta\.

Ifu=∑iσi​xi⊗yiu=\\sum\_\{i\}\\sigma\_\{i\}x\_\{i\}\\otimes y\_\{i\}andv=∑jτj​zj⊗wjv=\\sum\_\{j\}\\tau\_\{j\}z\_\{j\}\\otimes w\_\{j\}have Schmidt ranksρu\\rho\_\{u\}andρv\\rho\_\{v\}, the mixed\-product rule gives

u​v⊤=∑i,jσi​τj​\(xi​zj⊤\)⊗\(yi​wj⊤\),uv^\{\\top\}=\\sum\_\{i,j\}\\sigma\_\{i\}\\tau\_\{j\}\\,\(x\_\{i\}z\_\{j\}^\{\\top\}\)\\otimes\(y\_\{i\}w\_\{j\}^\{\\top\}\),an operator\-Schmidt decomposition withρu​ρv\\rho\_\{u\}\\rho\_\{v\}terms, sork⊗⁡\(u​v⊤\)=ρu​ρv\\operatorname\{rk\}\_\{\\otimes\}\(uv^\{\\top\}\)=\\rho\_\{u\}\\rho\_\{v\}\. A generic rank\-one update, whichLoRA1\\mathrm\{LoRA\}\_\{1\}reaches, thus needsκ1=min⁡\(m1,m2\)​min⁡\(n1,n2\)\\kappa\_\{1\}=\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\)Kronecker terms \([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(c\)\)\.

Used in\.[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2), Figure[6](https://arxiv.org/html/2610.12728#S2.F6),[Prediction X\.5](https://arxiv.org/html/2610.12728#S10.SS0.SSSx3), the known results credited in Exposé[XII](https://arxiv.org/html/2610.12728#S12)\. Reading:[Nielsen and Chuang \(2000, §2\.5\)](https://arxiv.org/html/2610.12728#bib.bib349)\.

Fact B\.13\(rank of a Hadamard product; face\-splitting product\)\.ForX,Y∈ℝm×nX,Y\\in\\mathbb\{R\}^\{m\\times n\},rank⁡\(X⊙Y\)≤rank⁡X⋅rank⁡Y\\rk\(X\\odot Y\)\\leq\\rk X\\cdot\\rk Y\. Explicitly, ifX=B1​A1X=B\_\{1\}A\_\{1\}andY=B2​A2Y=B\_\{2\}A\_\{2\}with inner dimensionsr1r\_\{1\}andr2r\_\{2\}, thenX⊙Y=\(B1∙B2\)​\(A1⊤∙A2⊤\)⊤,X\\odot Y=\(B\_\{1\}\\bullet B\_\{2\}\)\\,\(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\},where theface\-splitting productB1∙B2∈ℝm×r1​r2B\_\{1\}\\bullet B\_\{2\}\\in\\mathbb\{R\}^\{m\\times r\_\{1\}r\_\{2\}\}has as rowiithe Kronecker product of rowiiofB1B\_\{1\}and rowiiofB2B\_\{2\}\. In particularW0⊙\(B​A\)=∑k≤rdiag⁡\(bk\)​W0​diag⁡\(ak\),W\_\{0\}\\odot\(BA\)=\\sum\_\{k\\leq r\}\\diag\(b\_\{k\}\)\\,W\_\{0\}\\,\\diag\(a\_\{k\}\),summed over the columnsbkb\_\{k\}ofBBand the rowsaka\_\{k\}ofAA, has rank at mostr​rank⁡W0r\\rk W\_\{0\}\([Horn and Johnson, 1991](https://arxiv.org/html/2610.12728#bib.bib173), Ch\. 5\)\.

##### The idea of the proof

X⊙YX\\odot Yis the submatrix ofX⊗YX\\otimes Yon the rows\(i,i\)\(i,i\)and the columns\(j,j\)\(j,j\)\. In pictures it is the spider compositeμ∘\(X⊗Y\)∘δ\\mu\\circ\(X\\otimes Y\)\\circ\\deltaof[Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1), which factors through the two inner wires\.

LoHar1,r2\\mathrm\{LoHa\}\_\{r\_\{1\},r\_\{2\}\}therefore has rank at mostr1​r2r\_\{1\}r\_\{2\}, andh=\(B1∙B2,\(A1⊤∙A2⊤\)⊤\)h=\\big\(B\_\{1\}\\bullet B\_\{2\},\(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\}\\big\)is the atlas arrowLoHar1,r2→LoRAr1​r2\\mathrm\{LoHa\}\_\{r\_\{1\},r\_\{2\}\}\\to\\mathrm\{LoRA\}\_\{r\_\{1\}r\_\{2\}\}\. HiRA’s updateW0⊙B​AW\_\{0\}\\odot BAhas rank at mostr​rank⁡W0r\\rk W\_\{0\}, a bound that allows full rank whenW0W\_\{0\}has full rank \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(a\)\)\.

Used in\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6), §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3), Rosetta entries 6 and 11 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\)\.

#### Sums, Para and lenses

Background B\.14\(sums of methods; Minkowski sums, differences and translates\)\. For subsetsS,TS,Tof a vector space, theMinkowski sumisS\+T=\{s\+t:s∈S,t∈T\}S\+T=\\\{s\+t:s\\in S,\\ t\\in T\\\}, theMinkowski differenceisS−T=\{s−t:s∈S,t∈T\}S\-T=\\\{s\-t:s\\in S,\\ t\\in T\\\}, atranslateofSSisv\+S=\{v\+s:s∈S\}v\+S=\\\{v\+s:s\\in S\\\}, andC\+KC^\{\+K\}is the sum ofKKcopies ofCC\. A sum is the image ofS×TS\\times Tunder addition, so for semialgebraic setsdim\(S\+T\)≤dimS\+dimT\\dim\(S\+T\)\\leq\\dim S\+\\dim T\([Background C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\), and likewise for differences\. If0∈C0\\in C, the additive monoid \([Background D\.10](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) generated byCCisMon⁡\(C\)=⋃KC\+K\\operatorname\{Mon\}\(C\)=\\bigcup\_\{K\}C^\{\+K\}, and if moreoverℝ​C=C\\mathbb\{R\}C=Cit isspan⁡C\\spanop C\. WritingρM=θ0\+δM\\rho\_\{M\}=\\theta\_\{0\}\+\\delta\_\{M\}, the sum of methodsM⊕N=\(QM×QN,\(q0,q0′\),θ0\+δM\+δN\)M\\oplus N=\\big\(Q\_\{M\}\\times Q\_\{N\},\\,\(q\_\{0\},q\_\{0\}^\{\\prime\}\),\\,\\theta\_\{0\}\+\\delta\_\{M\}\+\\delta\_\{N\}\\big\)has imageθ0\+\(Im⁡δM\+Im⁡δN\)\\theta\_\{0\}\+\(\\Imop\\delta\_\{M\}\+\\Imop\\delta\_\{N\}\), and⊕\\oplusis a symmetric monoidal structure with the frozen model as unit \([Background B\.1](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\)\.

Concatenation\(\[B1​B2\],\[A1;A2\]\)\(\[B\_\{1\}\\ B\_\{2\}\],\[A\_\{1\};A\_\{2\}\]\)is an isomorphismLoRAr⊕LoRAs≅LoRAr\+s\\mathrm\{LoRA\}\_\{r\}\\oplus\\mathrm\{LoRA\}\_\{s\}\\cong\\mathrm\{LoRA\}\_\{r\+s\}of unpointed objects, with theAA\-blocks rescaled to the target scale when the scales differ, as underα/r\\alpha/rscaling \([Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\)\. It is also a pointed isomorphism ontoLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}pointed at the concatenated initialisation\. RoSA\([Nikdan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib350)\)and GraLoRA\([Jung et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib203)\)are⊕\\oplus\-words\. The categorical product of[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3), when it exists, has imageIm⁡M∩Im⁡N\\Imop M\\cap\\Imop N\([Background A\.10](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)\)\. Merging and restartingKKtimes with a fixed shapeCCreachesθ0\+C\+K\\theta\_\{0\}\+C^\{\+K\}\. Sinceℳ≤r\+K=ℳ≤min⁡\(K​r,m,n\)\\mathcal\{M\}\_\{\\leq r\}^\{\+K\}=\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}, ReLoRA reaches every matrix after⌈min⁡\(m,n\)/r⌉\\lceil\\min\(m,n\)/r\\rceilcycles \([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.

Re\-pointing an additive methodρ=θb\+δ\\rho=\\theta\_\{b\}\+\\deltawith shapeC=Im⁡δC=\\Imop\\deltaat a parameterqq, so thatθb=θ0−δ⁡\(q\)\\theta\_\{b\}=\\theta\_\{0\}\-\\delta\(q\), gives the translateθ0\+\(C−δ⁡\(q\)\)\\theta\_\{0\}\+\(C\-\\delta\(q\)\)as image, and all exact pointings together reachθ0\+\(C−C\)\\theta\_\{0\}\+\(C\-C\)\. For LoRA,ℳ≤r\\mathcal\{M\}\_\{\\leq r\}is closed under negation andℳ≤r\+ℳ≤r=ℳ≤min⁡\(2​r,m,n\)\\mathcal\{M\}\_\{\\leq r\}\+\\mathcal\{M\}\_\{\\leq r\}=\\mathcal\{M\}\_\{\\leq\\min\(2r,m,n\)\}by splitting an SVD \([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\), soC−C=ℳ≤min⁡\(2​r,m,n\)C\-C=\\mathcal\{M\}\_\{\\leq\\min\(2r,m,n\)\}, which is why no exact pointing reaches an update of rank above2​r2r\([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\.

Used in\.[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5),[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), §[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), §[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx3)\.

Background B\.15\(strong monoidal functors;\(ℕ,\+\)\(\\mathbb\{N\},\+\)and the permutation groupoid\)\. Astrong monoidal functorF:\(𝒞,⊗,I\)→\(𝒟,⊗,J\)F:\(\\mathcal\{C\},\\otimes,I\)\\to\(\\mathcal\{D\},\\otimes,J\)is a functor with isomorphismsϕA,B:F​A⊗F​B→F⁡\(A⊗B\)\\phi\_\{A,B\}:FA\\otimes FB\\to F\(A\\otimes B\), natural inAAandBB, andϕ0:J→F​I\\phi\_\{0\}:J\\to FI, compatible with associators and unitors\. It issymmetricif alsoF⁡\(σA,B\)∘ϕA,B=ϕB,A∘σF​A,F​BF\(\\sigma\_\{A,B\}\)\\circ\\phi\_\{A,B\}=\\phi\_\{B,A\}\\circ\\sigma\_\{FA,FB\}\. Thediscrete monoidal category\(ℕ,\+\)\(\\mathbb\{N\},\+\)has the natural numbers as objects, only identity morphisms,r⊗s=r\+sr\\otimes s=r\+sand identity symmetry\. Thepermutation groupoidℙ\\mathbb\{P\}has the same objects, the symmetric groupSrS\_\{r\}as the automorphisms ofrrand no other morphisms, with⊗\\otimesgiven by\+\+and block sums of permutations, and the block transposition inSr\+sS\_\{r\+s\}as symmetry\.

Concatenation \([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\) makesr↦LoRArr\\mapsto\\mathrm\{LoRA\}\_\{r\}strong monoidal from\(ℕ,\+\)\(\\mathbb\{N\},\+\)into unpointed methods\. It is not symmetric there, becauseϕs,r∘σ\\phi\_\{s,r\}\\circ\\sigmalists the blocks in the other order\. Onℙ\\mathbb\{P\}the block transposition acts onLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}by permuting inner channels,\(B,A\)↦\(B​P−1,P​A\)\(B,A\)\\mapsto\(BP^\{\-1\},PA\), and the symmetry square commutes\. Pointed objects behave differently\. IfF⁡\(r\)F\(r\)andF⁡\(2​r\)F\(2r\)are standardTA\\mathrm\{T\}\_\{A\}pointings \(B0=0B\_\{0\}=0andA0A\_\{0\}of full row rank,[Background C\.19](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\), which needs2​r<min⁡\(m,n\)2r<\\min\(m,n\), thenF⁡\(r\)⊕F⁡\(r\)≇F⁡\(2​r\)F\(r\)\\oplus F\(r\)\\not\\cong F\(2r\)as pointed objects, sincedimS1\\dim S\_\{1\}ism​rmragainst2​m​r2mr\([Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\)\.

Used in\.[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5), the referee record in Exposé[XI](https://arxiv.org/html/2610.12728#S11)\. Reading:[Mac Lane \(1998, Ch\. XI\)](https://arxiv.org/html/2610.12728#bib.bib306)\.

Background B\.16\(bicategories and the Para construction\)\. Abicategoryhas objects,1\-cellsbetween objects and2\-cellsbetween parallel 1\-cells\. 2\-cells compose vertically and horizontally, and composition of 1\-cells is associative and unital up to coherent invertible 2\-cells\. For a symmetric monoidal category𝒞\\mathcal\{C\}, here always cartesian, the bicategory𝐏𝐚𝐫𝐚⁡\(𝒞\)\\mathbf\{Para\}\(\\mathcal\{C\}\)has the objects of𝒞\\mathcal\{C\}\. A 1\-cellX→YX\\to Yis a pair\(P,f:P×X→Y\)\(P,\\,f:P\\times X\\to Y\), and\(P′,g\):Y→Z\(P^\{\\prime\},g\):Y\\to Zcomposes with it to\(P′×P,\(p′,p,x\)↦g\(p′,f\(p,x\)\)\)\\big\(P^\{\\prime\}\\times P,\\ \(p^\{\\prime\},p,x\)\\mapsto g\(p^\{\\prime\},f\(p,x\)\)\\big\)\. A 2\-cell\(P,f\)⇒\(Q,g\)\(P,f\)\\Rightarrow\(Q,g\)is areparametrisationr:Q→Pr:Q\\to Pwithg=f∘\(r×idX\)g=f\\circ\(r\\times\\mathrm\{id\}\_\{X\}\)\. A2\-functor\(pseudofunctor\) between bicategories sends objects, 1\-cells and 2\-cells to objects, 1\-cells and 2\-cells, and preserves both compositions up to coherent invertible 2\-cells\.

A linear layer is the 1\-cell\(ℝm×n,\(W,x\)↦Wx\):ℝn→ℝm\\big\(\\mathbb\{R\}^\{m\\times n\},\\,\(W,x\)\\mapsto Wx\\big\):\\mathbb\{R\}^\{n\}\\to\\mathbb\{R\}^\{m\}, and stacking a second layerℝm→ℝk\\mathbb\{R\}^\{m\}\\to\\mathbb\{R\}^\{k\}composes 1\-cells, with parameter spaceℝk×m×ℝm×n\\mathbb\{R\}^\{k\\times m\}\\times\\mathbb\{R\}^\{m\\times n\}\. A pretrained model is a pointed 1\-cellF=\(Θ,θ0,f\)F=\(\\Theta,\\theta\_\{0\},f\)of𝐏𝐚𝐫𝐚⁡\(𝐃𝐢𝐟𝐟\)\\mathbf\{Para\}\(\\mathbf\{Diff\}\), and its realisationΦf:Θ→C∞​\(X,Y\)\\Phi\_\{f\}:\\Theta\\to C^\{\\infty\}\(X,Y\),θ↦f⁡\(θ,−\)\\theta\\mapsto f\(\\theta,\-\), records the function at each weight\. LoRA is a 2\-cellF⇒\(Q,f∘\(ρ×idX\)\)F\\Rightarrow\(Q,f\\circ\(\\rho\\times\\mathrm\{id\}\_\{X\}\)\), the network with\(B,A\)\(B,A\)in its parameter slot, and a simulation is a 2\-cell between two such cells \([Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2)\)\. An adapter extension is a 2\-cell intoFF, and merging it is a lift throughΦf\\Phi\_\{f\}\([Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2)\)\. Composition is associative only up to\(P′′×P′\)×P≅P′′×\(P′×P\)\(P^\{\\prime\\prime\}\\times P^\{\\prime\}\)\\times P\\cong P^\{\\prime\\prime\}\\times\(P^\{\\prime\}\\times P\), hence a bicategory; over a strict monoidal category it is a 2\-category\.

Used in\.[Definition I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), §[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2),[Remark I\.3](https://arxiv.org/html/2610.12728#S1.SS0.SSSx2),[Definition VI\.1](https://arxiv.org/html/2610.12728#S6.SS0.SSSx2), §[IV](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2), the key references of Exposé[XII](https://arxiv.org/html/2610.12728#S12)\. Reading:[Leinster \(1998\)](https://arxiv.org/html/2610.12728#bib.bib241)\.

Background B\.17\(lenses, reverse derivatives and𝐏𝐚𝐫𝐚⁡\(𝖱\)\\mathbf\{Para\}\(\\mathsf\{R\}\)\)\. In a cartesian category, alens\(A,A′\)→\(B,B′\)\(A,A^\{\\prime\}\)\\to\(B,B^\{\\prime\}\)is a pair of maps, a forward partf:A→Bf:A\\to Band a backward partf♯:A×B′→A′f^\{\\sharp\}:A\\times B^\{\\prime\}\\to A^\{\\prime\}\. Lenses compose by\(g,g♯\)∘\(f,f♯\)=\(g∘f,\(a,c′\)↦f♯\(a,g♯\(f\(a\),c′\)\)\)\(g,g^\{\\sharp\}\)\\circ\(f,f^\{\\sharp\}\)=\\big\(g\\circ f,\\ \(a,c^\{\\prime\}\)\\mapsto f^\{\\sharp\}\(a,g^\{\\sharp\}\(f\(a\),c^\{\\prime\}\)\)\\big\)and form the category𝐋𝐞𝐧𝐬⁡\(𝒞\)\\mathbf\{Lens\}\(\\mathcal\{C\}\)\. For smooth maps between open subsets of Euclidean spaces, thereverse derivativeis𝖱⁡\[f\]​\(x,y\)=Jf​\(x\)⊤​y\.\\mathsf\{R\}\[f\]\(x,y\)=J\_\{f\}\(x\)^\{\\top\}y\.The chain ruleJg∘f​\(x\)⊤=Jf​\(x\)⊤​Jg​\(f⁡\(x\)\)⊤J\_\{g\\circ f\}\(x\)^\{\\top\}=J\_\{f\}\(x\)^\{\\top\}J\_\{g\}\(f\(x\)\)^\{\\top\}says thatf↦\(f,𝖱⁡\[f\]\)f\\mapsto\(f,\\mathsf\{R\}\[f\]\)is a functor into𝐋𝐞𝐧𝐬\\mathbf\{Lens\}\. Applied to parametrised maps it gives the 2\-functor𝐏𝐚𝐫𝐚⁡\(𝖱\):𝐏𝐚𝐫𝐚⁡\(𝒞\)→𝐏𝐚𝐫𝐚⁡\(𝐋𝐞𝐧𝐬⁡\(𝒞\)\)\\mathbf\{Para\}\(\\mathsf\{R\}\):\\mathbf\{Para\}\(\\mathcal\{C\}\)\\to\\mathbf\{Para\}\(\\mathbf\{Lens\}\(\\mathcal\{C\}\)\)of[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85), which is backpropagation\.

LoRA’s forward part isρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBA, and its backward part pulls the weight gradientGGback to

𝖱⁡\[ρ\]​\(\(B,A\),G\)=\(s​G​A⊤,s​B⊤​G\)\.\\mathsf\{R\}\[\\rho\]\\big\(\(B,A\),G\\big\)=\\big\(sGA^\{\\top\},\\,sB^\{\\top\}G\\big\)\.GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)keeps the forward partidΘ\\mathrm\{id\}\_\{\\Theta\}and uses the compressionG↦P⊤​GG\\mapsto P^\{\\top\}Gwith anchorPP, a lens whose backward part is not the reverse derivative of its forward part \([Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\)\. For linearρ⁡\(C\)=Wk\+ℒ​C\\rho\(C\)=W\_\{k\}\+\\mathcal\{L\}C,𝖱⁡\[ρ\]​\(C,G\)=ℒ∗​G\\mathsf\{R\}\[\\rho\]\(C,G\)=\\mathcal\{L\}^\{\*\}Gdoes not depend onCC, which is why the two readings agree at degree one \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.

Used in\.Exposé[IV](https://arxiv.org/html/2610.12728#S4), §[IV](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2),[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2),[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), Rosetta entry 25 \(§[IX](https://arxiv.org/html/2610.12728#S9.SS0.SSSx2)\), Exposé[0](https://arxiv.org/html/2610.12728#S0)\(where the category theory earns its keep\), the seven borrowed words of Exposé[XI](https://arxiv.org/html/2610.12728#S11)\. Reading:[Cruttwell et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib85)\.

### Part C\. Manifolds, germs and cones

Parts A and B used only the arrows of a method and its wiring\. Part C turns to its geometry\. A PEFT method in this paper is a smooth mapρ:\(Q,q0\)→\(Θ,θ0\)\\rho:\(Q,q\_\{0\}\)\\to\(\\Theta,\\theta\_\{0\}\), and its expressivity is read off the setIm⁡M=ρ⁡\(Q\)\\Imop M=\\rho\(Q\)nearθ0\\theta\_\{0\}\. This part collects the geometry behind that reading, from manifolds, differentials and rank for the map to germs, cones, singular points and semialgebraic dimension for the image, which is often not a manifold\.

#### Manifolds and smooth maps

Background C\.1\(smooth manifolds and smooth maps\)\. Asmooth manifoldof dimensionkkis a space covered bycharts, bijections from open pieces of it onto open subsets ofℝk\\mathbb\{R\}^\{k\}, whose changes of chart are smooth\. Every manifold in this paper is an open subset of a vector space or an embedded submanifold of one \([Background C\.4](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\), or is diffeomorphic to one, as quotients \([Background D\.22](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\) and the Grassmannian \([Background C\.15](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\) are\. Adiffeomorphismis a smooth bijection with smooth inverse, andDiff⁡\(Q\)\\mathrm\{Diff\}\(Q\)is the group of diffeomorphisms ofQQunder composition\. Alocal diffeomorphismmaps an open neighbourhood of each point diffeomorphically onto an open set; by the inverse function theorem,ρ\\rhodoes this nearqqwheneverd​ρqd\\rho\_\{q\}\([Background C\.2](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\) is invertible\.

LoRA’s\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)parameter spaceℝm×r×ℝr×n\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}is a vector space, and HRA’s\([Yuan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib540)\)\{\(u1,…,ur\):ui≠0\}\\\{\(u\_\{1\},\\dots,u\_\{r\}\):u\_\{i\}\\neq 0\\\}is open in\(ℝn\)r\(\\mathbb\{R\}^\{n\}\)^\{r\}\. The rotation groupSO⁡\(n\)\\mathrm\{SO\}\(n\)is a compact manifold of dimensionn⁡\(n−1\)/2n\(n\-1\)/2, and the Cayley map \([Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) is a diffeomorphism from𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)onto an open dense subset of it\. A method is a pointed smooth map, and its gauge group\{φ∈Diff⁡\(Q\):ρ∘φ=ρ\}\\\{\\varphi\\in\\mathrm\{Diff\}\(Q\):\\rho\\circ\\varphi=\\rho\\\}\([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\) contains LoRA’s\(B,A\)↦\(B​g−1,g​A\)\(B,A\)\\mapsto\(Bg^\{\-1\},gA\)for everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\.

Used in\.[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1),[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3),[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\. Reading:[Lee \(2013, Ch\. 1 and 2\)](https://arxiv.org/html/2610.12728#bib.bib237)\.

Background C\.2\(tangent spaces, differentials and covectors\)\. For a manifoldQ⊆ℝNQ\\subseteq\\mathbb\{R\}^\{N\}, thetangent spaceTq​QT\_\{q\}Qis the set of velocitiesc′​\(0\)c^\{\\prime\}\(0\)of smooth curvesccinQQwithc⁡\(0\)=qc\(0\)=q\. It is a linear space of dimensiondimQ\\dim Q, all ofℝN\\mathbb\{R\}^\{N\}whenQQis open\. Thedifferentialof a smoothρ\\rhoatqqis the linear mapd​ρq:Tq​Q→Tρ⁡\(q\)​Θd\\rho\_\{q\}:T\_\{q\}Q\\to T\_\{\\rho\(q\)\}\\Theta,c′​\(0\)↦\(ρ∘c\)′​\(0\)c^\{\\prime\}\(0\)\\mapsto\(\\rho\\circ c\)^\{\\prime\}\(0\)\. In coordinates it is the Jacobian, and the chain rule readsd​\(ρ∘h\)q=d​ρh⁡\(q\)∘d​hqd\(\\rho\\circ h\)\_\{q\}=d\\rho\_\{h\(q\)\}\\circ dh\_\{q\}\. Thecotangent spaceTθ∗​ΘT^\{\*\}\_\{\\theta\}\\Thetais the dual ofTθ​ΘT\_\{\\theta\}\\Theta\. The differentiald​LθdL\_\{\\theta\}of a loss is acovector, and it becomes a gradient vector∇L​\(θ\)\\nabla L\(\\theta\)only once an inner product is chosen\.

For LoRA,d​ρ\(B,A\)​\(B˙,A˙\)=s⁡\(B˙​A\+B​A˙\)d\\rho\_\{\(B,A\)\}\(\\dot\{B\},\\dot\{A\}\)=s\(\\dot\{B\}A\+B\\dot\{A\}\)\. For a rotation orbit,TI​SO​\(n\)=𝔰​𝔬​\(n\)T\_\{I\}\\mathrm\{SO\}\(n\)=\\mathfrak\{so\}\(n\)andTW0​R​\(W0⋅SO⁡\(n\)\)=W0​R​𝔰​𝔬​\(n\)T\_\{W\_\{0\}R\}\\big\(W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\\big\)=W\_\{0\}R\\,\\mathfrak\{so\}\(n\)\.[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)is the chain rule atq0q\_\{0\}\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)drives the parameters by any cotangent signalγ⁡\(t\)∈Tρ⁡\(q\)∗​Θ\\gamma\(t\)\\in T^\{\*\}\_\{\\rho\(q\)\}\\Theta, pulled back toQQasγ∘d​ρq\\gamma\\circ d\\rho\_\{q\}, which isD​ρ⊤​γD\\rho^\{\\top\}\\gammain coordinates\.

Used in\.the notation \(Table[2](https://arxiv.org/html/2610.12728#S0.T2)\),[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3),[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\. Reading:[Lee \(2013, Ch\. 3 and 11\)](https://arxiv.org/html/2610.12728#bib.bib237)\.

Background C\.3\(rank, expressive dimension and first\-order image\)\. Therankofρ\\rhoatqqisrank⁡d​ρq\\rk d\\rho\_\{q\}\. It is lower semicontinuous, since a nonzerok×kk\\times kminor of the Jacobian stays nonzero nearby, so\{q:rank⁡d​ρq≥k\}\\\{q:\\rk d\\rho\_\{q\}\\geq k\\\}is open\. Thegeneric rankismaxq⁡rank⁡d​ρq\\max\_\{q\}\\rk d\\rho\_\{q\}\. Ifρ\\rhois real\-analytic andQQis connected, the generic rank is attained on an open dense set whose complement has measure zero \([Background C\.22](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\)\. The paper writes\|M\|=dimQ\\lvert M\\rvert=\\dim Qfor the parameter budget,d⁡\(M\)d\(M\)for the generic rank,S1=d​ρq0​\(Tq0​Q\)S\_\{1\}=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\)for thefirst\-order imageandd⁡\(M\)−dimS1d\(M\)\-\\dim S\_\{1\}for thefirst\-order deficit\. For definableρ\\rhoon an open definableQQ, as for every method in this paper,[Background C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\(d\) shows thatd⁡\(M\)=dimIm⁡Md\(M\)=\\dim\\Imop M, theexpressive dimension\. The deficit vanishes exactly whenq0q\_\{0\}is a point of maximal rank; the base pointθ0\\theta\_\{0\}is then calledregularforMM, and anapexwhen the deficit is positive \([Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\.

[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)gives LoRA’s rank at every point,m​rk⁡A\+n​rk⁡B−rk⁡A​rk⁡Bm\\operatorname\{rk\}A\+n\\operatorname\{rk\}B\-\\operatorname\{rk\}A\\operatorname\{rk\}B\. At a random pair it isr⁡\(m\+n−r\)=dr\(m\+n\-r\)=d; at the zero\-init point\(0,A0\)\(0,A\_\{0\}\)it ism​rmr, a deficit ofr⁡\(n−r\)r\(n\-r\)\. Since the points of lower rank have measure zero, a float64 Jacobian rank at one random point computesd⁡\(M\)d\(M\)with probability one, up to the tolerance that decides numerical rank\. The plug workbench of the companion site \(§[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5)\) and the check script \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\) do exactly this\.

Used in\.the notation \(Table[2](https://arxiv.org/html/2610.12728#S0.T2)\),[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), the plug workbench \(§[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5)\), reproducing the numbers \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

Background C\.4\(immersions, submersions and embedded submanifolds\)\. A smoothρ:Q→Θ\\rho:Q\\to\\Thetais animmersionif everyd​ρqd\\rho\_\{q\}is injective \(rankdimQ\\dim Q\), and asubmersionif everyd​ρqd\\rho\_\{q\}is surjective \(rankdimΘ\\dim\\Theta\)\. A subsetX⊆ℝNX\\subseteq\\mathbb\{R\}^\{N\}is anembedded submanifoldof dimensionkkif near each of its points it is the zero set ofN−kN\-ksmooth functions with linearly independent gradients\. A submersion is an open map, so its image contains an open set\. Anembedding, an injective immersion that is a homeomorphism onto its image, has an embedded submanifold as image; an injective immersion such as the figure eight need not\.

The one\-sided methodFA⁡\(A0\):B↦W0\+B​A0\\mathrm\{FA\}\(A\_\{0\}\):B\\mapsto W\_\{0\}\+BA\_\{0\}is an injective immersion with affine imageW0\+\{B​A0\}W\_\{0\}\+\\\{BA\_\{0\}\\\}whenA0A\_\{0\}has full row rank\. On pairs of full rank,\(B,A\)↦B​A\(B,A\)\\mapsto BAis a submersion ontoℳr\\mathcal\{M\}\_\{r\}\([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) uses openness, in the argument recalled in[Background A\.13](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)\.[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)asks for injective immersions\(ℝk,0\)→\(U,θ\)\(\\mathbb\{R\}^\{k\},0\)\\to\(U,\\theta\)tangent to a distribution\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4), integrability \(§[IV](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\)\. Reading:[Lee \(2013, Ch\. 4 and 5\)](https://arxiv.org/html/2610.12728#bib.bib237)\.

Fact C\.5\(constant\-rank theorem\)\.Letρ:Q→Θ\\rho:Q\\to\\Thetabe smooth withrank⁡d​ρq=k\\rk d\\rho\_\{q\}=kfor allqqnearpp\. Then there are charts centred atppand atρ⁡\(p\)\\rho\(p\)in whichρ\\rhoreads\(x1,…,xdimQ\)↦\(x1,…,xk,0,…,0\)\(x\_\{1\},\\dots,x\_\{\\dim Q\}\)\\mapsto\(x\_\{1\},\\dots,x\_\{k\},0,\\dots,0\)\. Hence some neighbourhoodVVofpphasρ⁡\(V\)\\rho\(V\)an embeddedkk\-dimensional submanifold, andρ−1​\(ρ​\(p\)\)∩V\\rho^\{\-1\}\(\\rho\(p\)\)\\cap Van embedded submanifold of dimensiondimQ−k\\dim Q\-k\. By[Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)the hypothesis holds wherever the rank is maximal\.

##### The idea of the proof

Choose coordinates in which the top\-leftk×kk\\times kblock of the Jacobian is invertible\. The inverse function theorem straightens the firstkkoutput coordinates\. Constant rank then forces the other outputs to depend only on thosekk, and a second change of coordinates in the target removes them\.

The paper uses this theorem without naming it\. Near a point of maximal rank,ρ\\rhois locally a projection onto a sheet of dimensiond⁡\(M\)d\(M\), with fibres of dimension\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\), the gauge waste\. For LoRA at a full\-rank pair the sheet is a piece ofW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}and the fibre a piece of aGLr\\mathrm\{GL\}\_\{r\}\-orbit\.[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)makes the analogous constant\-rank assumption for distributions, where Frobenius’ theorem \([Background D\.25](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\) takes the place of this one\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Background C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\. Reading:[Lee \(2013, Ch\. 4, the rank theorem\)](https://arxiv.org/html/2610.12728#bib.bib237)\.

Background C\.6\(fibres and fibre dimension\)\. Thefibreofρ\\rhooverθ\\thetaisρ−1​\(θ\)\\rho^\{\-1\}\(\\theta\), the set of parameters that give the weightθ\\theta\. Ifrank⁡d​ρq=d⁡\(M\)\\rk d\\rho\_\{q\}=d\(M\),[Background C\.5](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)makes the fibre throughqqa submanifold of dimension\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\)nearqq\. For real\-analyticρ\\rhoon connectedQQthis holds for allqqoutside a closed set of measure zero, and these are thegeneric fibres\. Fibres through points of lower rank can have a different dimension\. The fibres of a fibre bundle all look alike locally, so a map whose fibre dimension jumps is no fibre bundle\.

For LoRA on the full\-rank stratum, the fibre through\(B,A\)\(B,A\)is the orbit\{\(B​g−1,g​A\):g∈GLr\}\\\{\(Bg^\{\-1\},gA\):g\\in\\mathrm\{GL\}\_\{r\}\\\}, of dimensionr2=r⁡\(m\+n\)−r⁡\(m\+n−r\)r^\{2\}=r\(m\+n\)\-r\(m\+n\-r\)\([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\. For zero\-init LoRA the fibre overθ0\\theta\_\{0\}is\{\(B,A\):B​A=0\}\\\{\(B,A\):BA=0\\\}\. It contains\{B=0\}\\\{B=0\\\}, of dimensionr​n\>r2rn\>r^\{2\}, and is itself a union of strata, such as\{\(0,A\):rank⁡A=r\}\\\{\(0,A\):\\rk A=r\\\}\([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\.

Used in\.Exposé[III](https://arxiv.org/html/2610.12728#S3),[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9), Rosetta entries 6 and 11 \(Exposé[IX](https://arxiv.org/html/2610.12728#S9)\), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

Background C\.7\(fibre products, transversality and clean intersection\)\. ForρM:QM→Θ\\rho\_\{M\}:Q\_\{M\}\\to\\ThetaandρN:QN→Θ\\rho\_\{N\}:Q\_\{N\}\\to\\Theta, the set\-levelfibre productisP=\{\(q,q′\):ρM​\(q\)=ρN​\(q′\)\}⊆QM×QNP=\\\{\(q,q^\{\\prime\}\):\\rho\_\{M\}\(q\)=\\rho\_\{N\}\(q^\{\\prime\}\)\\\}\\subseteq Q\_\{M\}\\times Q\_\{N\}\. The maps aretransverse,ρM⋔ρN\\rho\_\{M\}\\pitchfork\\rho\_\{N\}, ifd​ρM​\(Tq​QM\)\+d​ρN​\(Tq′​QN\)=Tθ​Θd\\rho\_\{M\}\(T\_\{q\}Q\_\{M\}\)\+d\\rho\_\{N\}\(T\_\{q^\{\\prime\}\}Q\_\{N\}\)=T\_\{\\theta\}\\Thetaat every\(q,q′\)∈P\(q,q^\{\\prime\}\)\\in P; thenPPis an embedded submanifold of dimension\|M\|\+\|N\|−dimΘ\\lvert M\\rvert\+\\lvert N\\rvert\-\\dim\\Theta, and it is the pullback ofρM\\rho\_\{M\}andρN\\rho\_\{N\}in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\([Background A\.11](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2)\)\. The intersection iscleanifPPis an embedded submanifold withT\(q,q′\)​P=\{\(v,w\):d​ρM​v=d​ρN​w\}T\_\{\(q,q^\{\\prime\}\)\}P=\\\{\(v,w\):d\\rho\_\{M\}v=d\\rho\_\{N\}w\\\}at every point\. Transverse maps meet cleanly, and so do affine maps\.

Transversality at\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\)needs\|M\|\+\|N\|≥dimΘ\\lvert M\\rvert\+\\lvert N\\rvert\\geq\\dim\\Theta, which fails for any two small methods, so the products the paper builds come from clean cases\. The affine methodsFA⁡\(A0\)\\mathrm\{FA\}\(A\_\{0\}\)andFB⁡\(B0\)\\mathrm\{FB\}\(B\_\{0\}\), withB0B\_\{0\}of full column rank andA0A\_\{0\}of full row rank, have the linear fibre product\{\(B0​R,R​A0\)\}\\\{\(B\_\{0\}R,RA\_\{0\}\)\\\}of dimensionr2r^\{2\}, which is LoRA\-XS\([Bałazy et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib26)\)\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\)\. OnΘ=ℝ\\Theta=\\mathbb\{R\},LoRA1\\mathrm\{LoRA\}\_\{1\}and the frozen model have the cross\{BA=0\}\\\{BA=0\\\}as fibre product\. It is no manifold at the origin, and no product exists \([Background A\.13](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)\)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(d\)\. Reading:[Guillemin and Pollack \(1974, Ch\. 1\)](https://arxiv.org/html/2610.12728#bib.bib142);[Lee \(2013, Ch\. 6\)](https://arxiv.org/html/2610.12728#bib.bib237)\.

#### Germs, cones and singular points

Background C\.8\(germs\)\. Two subsetsX,Y⊆ΘX,Y\\subseteq\\Thetahave the samegermatθ0\\theta\_\{0\}ifX∩U=Y∩UX\\cap U=Y\\cap Ufor some neighbourhoodUUofθ0\\theta\_\{0\}; two maps have the same germ atq0q\_\{0\}if they agree nearq0q\_\{0\}\. The germs\(X,θ0\)\(X,\\theta\_\{0\}\)and\(Y,0\)\(Y,0\)are diffeomorphic if a diffeomorphismφ\\varphibetween neighbourhoodsU∋θ0U\\ni\\theta\_\{0\}andV∋0V\\ni 0hasφ⁡\(θ0\)=0\\varphi\(\\theta\_\{0\}\)=0andφ⁡\(X∩U\)=Y∩V\\varphi\(X\\cap U\)=Y\\cap V\. Local properties, such as the dimension atθ0\\theta\_\{0\}, smoothness atθ0\\theta\_\{0\}and the tangent cone \([Background C\.11](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2.Px1)\), depend only on the germ\.

The image germ of a method is the germ ofIm⁡M=ρ⁡\(Q\)\\Imop M=\\rho\(Q\)atθ0\\theta\_\{0\}\. It uses all ofQQ, so it can be larger than the image of a small neighbourhood ofq0q\_\{0\}\. For zero\-init LoRA,s​B​AsBAwithBBsmall andAAarbitrary is a small update with any row space, so the germ is that of the whole coneθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}; parameters near\(0,A0\)\(0,A\_\{0\}\)only give updates with row space insiderow⁡A\\operatorname\{row\}Afor someAAnearA0A\_\{0\}\. For injectiveW0W\_\{0\}and evenrrwith2≤r≤n−22\\leq r\\leq n\-2, HRA’s image germ atW0W\_\{0\}is the image of the germ ofΣr\\Sigma\_\{r\}at00under the embeddingΩ↦W0​C​\(Ω\)\\Omega\\mapsto W\_\{0\}\\,C\(\\Omega\)of𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)intoℝm×n\\mathbb\{R\}^\{m\\times n\}\([Background C\.13](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3),[Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\)\.

Used in\.the image germ \(Exposé[II](https://arxiv.org/html/2610.12728#S2)\), the three invariants \(Exposé[0](https://arxiv.org/html/2610.12728#S0)\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), the seven axes \(§[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1)\)\.

Background C\.9\(smooth and singular points\)\. A pointxxof a setX⊆ℝNX\\subseteq\\mathbb\{R\}^\{N\}is asmooth pointifX∩UX\\cap Uis aC1C^\{1\}embedded submanifold for some neighbourhoodUUofxx, and asingular pointotherwise\. The singular points form thesingular locusSing⁡X\\operatorname\{Sing\}X\. Algebraic geometry definesSing\\operatorname\{Sing\}through the Jacobian of defining polynomials, and forℳ≤r\\mathcal\{M\}\_\{\\leq r\}the two notions agree\. Smoothness is a property of a set\. The paper’s regular base point and apex, defined by a zero or positive first\-order deficit \([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1),[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\), are properties of a method atq0q\_\{0\}\.

The cross\{BA=0\}⊆ℝ2\\\{BA=0\\\}\\subseteq\\mathbb\{R\}^\{2\}is singular at the origin and smooth elsewhere\. For1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\),Sing⁡ℳ≤r=ℳ≤r−1\\operatorname\{Sing\}\\mathcal\{M\}\_\{\\leq r\}=\\mathcal\{M\}\_\{\\leq r\-1\}\. A split initialisation has image\(θ0−P\)\+ℳ≤r\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}withrank⁡P=r\\rk P=r, soθ0\\theta\_\{0\}is a smooth point; zero\-init LoRA starts at the singular vertex\. A figure\-eight curve passing through its crossing with nonzero velocity has deficit00there, though its image is singular, so smooth and regular are different notions\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), Figure[3](https://arxiv.org/html/2610.12728#S2.F3),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), Erlangen \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\)\. Reading:[Harris \(1992, Lecture 14\)](https://arxiv.org/html/2610.12728#bib.bib156)\.

Background C\.10\(cones and vertices\)\. A setXXin a vector space is aconewithvertexvvifv∈Xv\\in Xandv\+t⁡\(x−v\)∈Xv\+t\(x\-v\)\\in Xfor allx∈Xx\\in Xandt≥0t\\geq 0, andlinearif it is an affine subspace\. Ifρ=θ0\+δ\\rho=\\theta\_\{0\}\+\\deltawithδ⁡\(t​u,w\)=t​δ​\(u,w\)\\delta\(tu,w\)=t\\,\\delta\(u,w\)fort≥0t\\geq 0on a blockuuof parameters, andQQis closed underu↦t​uu\\mapsto tu, thenIm⁡M\\Imop Mis a cone with vertexθ0\\theta\_\{0\}\. A cone that is aC1C^\{1\}embedded submanifold near its vertexvvequalsv\+Tv​Xv\+T\_\{v\}X, so the vertex of a non\-linear cone is a singular point\.

##### Why a smooth cone is linear

Each ray fromvvlies inXX, soX⊆v\+Tv​XX\\subseteq v\+T\_\{v\}X\. Nearvv,XXis a submanifold of full dimension in this affine space, so it contains a neighbourhood ofvvthere, and scaling spreads it over all ofv\+Tv​Xv\+T\_\{v\}X\.

ℳ≤r\\mathcal\{M\}\_\{\\leq r\}is a closed cone with vertex00, and it is not linear for1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\), since a rank\-rrand a rank\-one matrix can sum to rankr\+1r\+1\. Zero\-init LoRA hasδ=s​B​A\\delta=sBA, linear inBB, so its imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}has singular vertexW0W\_\{0\}\. Its deficitr⁡\(n−r\)r\(n\-r\)makesθ0\\theta\_\{0\}an apex too; the vertex is a property of the set, the apex one of the method\. Multihomogeneous updates such as LoHa’s\([Hyeon\-Woo et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib186)\)give cones in the same way \([Background D\.19](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\)\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), the image germ \(Exposé[II](https://arxiv.org/html/2610.12728#S2)\),[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), Figure[3](https://arxiv.org/html/2610.12728#S2.F3),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\.

Fact C\.11\(first\-order image versus image\)\.Thetangent coneof a setXXatxxis the set of limits oftk​\(xk−x\)t\_\{k\}\(x\_\{k\}\-x\)withxk∈Xx\_\{k\}\\in X,xk→xx\_\{k\}\\to xandtk\>0t\_\{k\}\>0; for a closed cone with vertexvvit isX−vX\-v\. For a methodMM: \(a\)S1S\_\{1\}lies in the tangent cone ofIm⁡M\\Imop Matθ0\\theta\_\{0\}; \(b\)dimS1≤d⁡\(M\)\\dim S\_\{1\}\\leq d\(M\); \(c\) ifIm⁡M\\Imop Mis aC1C^\{1\}manifold of dimensiond⁡\(M\)d\(M\)nearθ0\\theta\_\{0\}, thenS1⊆Tθ0​Im⁡MS\_\{1\}\\subseteq T\_\{\\theta\_\{0\}\}\\Imop M, with equality exactly when the deficit is00\.

For \(a\),d​ρq0​\(ξ\)d\\rho\_\{q\_\{0\}\}\(\\xi\)is the velocity of the curveρ⁡\(q0\+t​ξ\)\\rho\(q\_\{0\}\+t\\xi\)inside the image; for \(c\),ρ\\rhomaps a neighbourhood ofq0q\_\{0\}into the manifold\. ForTA\\mathrm\{T\}\_\{A\}\-pointed LoRA \(B0=0B\_\{0\}=0,[Background C\.19](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\) the tangent cone atθ0\\theta\_\{0\}isℳ≤r\\mathcal\{M\}\_\{\\leq r\}itself, and inside itS1=\{X​A0\}S\_\{1\}=\\\{XA\_\{0\}\\\}, the matrices with row space inrow⁡A0\\operatorname\{row\}A\_\{0\}, is a linear space of dimensionm​r<r⁡\(m\+n−r\)mr<r\(m\+n\-r\)\. Figure[3](https://arxiv.org/html/2610.12728#S2.F3)\(a\) drawsS1S\_\{1\}as one ruling of the cone, a line through the apex\. At a split pointing case \(c\) applies, andS1=TP​ℳrS\_\{1\}=T\_\{P\}\\mathcal\{M\}\_\{r\}, the tangent plane of Figure[3](https://arxiv.org/html/2610.12728#S2.F3)\(b\)\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), Figure[3](https://arxiv.org/html/2610.12728#S2.F3)\. Reading:[Harris \(1992, Lecture 20\)](https://arxiv.org/html/2610.12728#bib.bib156)\.

#### Matrix varieties

Background C\.12\(the determinantal variety\)\.ℳ≤r=\{X∈ℝm×n:rank⁡X≤r\}\\mathcal\{M\}\_\{\\leq r\}=\\\{X\\in\\mathbb\{R\}^\{m\\times n\}:\\rk X\\leq r\\\}, thedeterminantal variety, is the zero set of all\(r\+1\)×\(r\+1\)\(r\+1\)\\times\(r\+1\)minors, a closed cone with vertex00\. Let1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)\. The matrices of rank exactlyrrform an open dense subsetℳr\\mathcal\{M\}\_\{r\}ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}, an embedded submanifold of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\), andSing⁡ℳ≤r=ℳ≤r−1\\operatorname\{Sing\}\\mathcal\{M\}\_\{\\leq r\}=\\mathcal\{M\}\_\{\\leq r\-1\}\. IfX∈ℳrX\\in\\mathcal\{M\}\_\{r\}has column spaceUUand row spaceVV, andX=B​AX=BAwithB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}andA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}, thenTX​ℳr=\{B˙​A\+B​A˙\}=\{D:PU⟂​D​PV⟂=0\}\.T\_\{X\}\\mathcal\{M\}\_\{r\}=\\\{\\dot\{B\}A\+B\\dot\{A\}\\\}=\\\{D:P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}=0\\\}\.Alsoℳ≤r−ℳ≤r=ℳ≤min⁡\(2​r,m,n\)\\mathcal\{M\}\_\{\\leq r\}\-\\mathcal\{M\}\_\{\\leq r\}=\\mathcal\{M\}\_\{\\leq\\min\(2r,m,n\)\}\.

Zero\-init LoRA has imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, of dimension130,816130\{,\}816for a4096×40964096\\times 4096matrix atr=16r=16\. On full\-rank pairs,\(B,A\)↦B​A\(B,A\)\\mapsto BAis a submersion ontoℳr\\mathcal\{M\}\_\{r\}whose fibres areGLr\\mathrm\{GL\}\_\{r\}\-orbits, which identifiesℳr\\mathcal\{M\}\_\{r\}with the quotient\(ℝ∗m×r×ℝ∗r×n\)/GLr\(\\mathbb\{R\}^\{m\\times r\}\_\{\*\}\\times\\mathbb\{R\}^\{r\\times n\}\_\{\*\}\)/\\mathrm\{GL\}\_\{r\}\([Background D\.22](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\. The difference formula is why no pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}reaches an update of rank above2​r2r\([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3),[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\)\.

Used in\.the notation \(Table[2](https://arxiv.org/html/2610.12728#S0.T2)\), translates of one shape \(§[II\.A](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\),[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5), Exposé II\.C \(§[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\. Reading:[Harris \(1992, Lecture 9\)](https://arxiv.org/html/2610.12728#bib.bib156)\.

Background C\.13\(the skew\-determinantal cone\)\. Every skew\-symmetric matrix has even rank\. For evenrrwith2≤r≤n−22\\leq r\\leq n\-2, theskew\-determinantal coneΣr=\{Ω∈𝔰​𝔬​\(n\):rank⁡Ω≤r\}\\Sigma\_\{r\}=\\\{\\Omega\\in\\mathfrak\{so\}\(n\):\\rk\\Omega\\leq r\\\}is a closed cone with vertex00and dimensionr⁡\(n−r\)\+r⁡\(r−1\)/2=r​n−r⁡\(r\+1\)/2r\(n\-r\)\+r\(r\-1\)/2=rn\-r\(r\+1\)/2, counting a range inGr⁡\(r,n\)\\Gr\(r,n\)and a nondegenerate skew form on it\. Its smooth points are the matrices of rank exactlyrr, andSing⁡Σr=Σr−2\\operatorname\{Sing\}\\Sigma\_\{r\}=\\Sigma\_\{r\-2\}\. It is not linear, since skew matrices of rankrrand22can sum to rankr\+2≤nr\+2\\leq n, so its vertex is singular\.

For injectiveW0W\_\{0\}and evenrr, HRA’s image nearW0W\_\{0\}consists of theW0​RW\_\{0\}RwithR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)nearIIandrank⁡\(R−I\)≤r\\rk\(R\-I\)\\leq r\. The Cayley map satisfiesrank⁡\(C⁡\(Ω\)−I\)=rank⁡Ω\\rk\(C\(\\Omega\)\-I\)=\\rk\\Omega\([Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\), so the image germ atW0W\_\{0\}is the image of the germ ofΣr\\Sigma\_\{r\}at00under the embeddingΩ↦W0​C​\(Ω\)\\Omega\\mapsto W\_\{0\}\\,C\(\\Omega\),W0W\_\{0\}is a singular vertex, andd⁡\(HRAr\)=dimΣrd\(\\mathrm\{HRA\}\_\{r\}\)=\\dim\\Sigma\_\{r\}\. At the paired initialisationu2​i−1=u2​i=wiu\_\{2i\-1\}=u\_\{2i\}=w\_\{i\}\(i≤k=r/2i\\leq k=r/2, with thewiw\_\{i\}independent\), putU=span⁡\{w1,…,wk\}U=\\spanop\\\{w\_\{1\},\\dots,w\_\{k\}\\\}, sodimU=k\\dim U=k\. ThenS1=W0⋅𝒮S\_\{1\}=W\_\{0\}\\cdot\\mathcal\{S\}with

𝒮=\{Ω∈𝔰​𝔬​\(n\):PU⟂​Ω​PU⟂=0\}=U∧ℝn\\mathcal\{S\}=\\\{\\Omega\\in\\mathfrak\{so\}\(n\):P\_\{U^\{\\perp\}\}\\Omega P\_\{U^\{\\perp\}\}=0\\\}=U\\wedge\\mathbb\{R\}^\{n\}\([Background D\.5](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\), a linear space of skew matrices of rank at most2​k=r2k=rand of dimensionk​n−k⁡\(k\+1\)/2<d⁡\(HRAr\)kn\-k\(k\+1\)/2<d\(\\mathrm\{HRA\}\_\{r\}\)\. It lies insideΣr\\Sigma\_\{r\}, soS1S\_\{1\}lies inside the tangent coneW0⋅ΣrW\_\{0\}\\cdot\\Sigma\_\{r\}\([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\)\)\.

Used in\.[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), HRA is a meet \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\)\.

Background C\.14\(Segre and toric cones\)\. TheSegre coneinℝp⊗ℝq≅ℝp×q\\mathbb\{R\}^\{p\}\\otimes\\mathbb\{R\}^\{q\}\\cong\\mathbb\{R\}^\{p\\times q\}is the set\{a⊗b\}=ℳ≤1\\\{a\\otimes b\\\}=\\mathcal\{M\}\_\{\\leq 1\}of rank\-one tensors, a closed cone of dimensionp\+q−1p\+q\-1that spans the whole space and is non\-linear whenp,q≥2p,q\\geq 2\. It is the simplesttoric cone, the closure of the image of a torus\(ℝ×\)k\(\\mathbb\{R\}^\{\\times\}\)^\{k\}under a monomial map, here\(a,b\)↦\(ai​bj\)i​j\(a,b\)\\mapsto\(a\_\{i\}b\_\{j\}\)\_\{ij\}\. Since rank\-one matrices spanℝp×q\\mathbb\{R\}^\{p\\times q\},kkgeneric rank\-one matrices spanmin⁡\(k,p​q\)\\min\(k,pq\)dimensions\.

Under the Van Loan–Pitsianis rearrangement \([Background B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\), KronA’s\([Edalati et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib103)\)updateA⊗BA\\otimes Bbecomesvec⁡\(A\)​vec⁡\(B\)⊤\\vect\(A\)\\,\\vect\(B\)^\{\\top\}, so its image is a linear copy of a Segre cone \([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. VeRA’s\([Kopiczko et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib224)\)updateΛb​B​Λd​A\\Lambda\_\{b\}B\\Lambda\_\{d\}Ahas entries∑k\(bi​dk\)​Bi​k​Ak​j\\sum\_\{k\}\(b\_\{i\}d\_\{k\}\)B\_\{ik\}A\_\{kj\}, a fixed linear function of the rank\-one matrixb​d⊤bd^\{\\top\}\. This function is injective whenBBhas no zero entry andAAhas full row rank, so the image is a toric cone of dimensionm\+r−1m\+r\-1with vertexθ0\\theta\_\{0\}, as VeRA’s entry in Exposé[VII](https://arxiv.org/html/2610.12728#S7)records\.[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)uses the spanning fact for DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib282)\), withmmpoints of the Segre cone inℝm−r⊗ℝn−r\\mathbb\{R\}^\{m\-r\}\\otimes\\mathbb\{R\}^\{n\-r\}\.

Used in\.[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[VeRA](https://arxiv.org/html/2610.12728#S7.SS0.SSS0.Px1)\(Exposé[VII](https://arxiv.org/html/2610.12728#S7)\)\. Reading:[Harris \(1992, Lecture 2\)](https://arxiv.org/html/2610.12728#bib.bib156);[Cox et al\. \(2011, Ch\. 1\)](https://arxiv.org/html/2610.12728#bib.bib84)\.

Background C\.15\(the Grassmannian\)\. TheGrassmannianGr⁡\(r,n\)\\Gr\(r,n\)is the set ofrr\-dimensional linear subspaces ofℝn\\mathbb\{R\}^\{n\}, a compact manifold of dimensionr⁡\(n−r\)r\(n\-r\)\. One model is the full\-row\-rankr×nr\\times nmatrices up toA↦g​AA\\mapsto gA,g∈GLrg\\in\\mathrm\{GL\}\_\{r\}, throughA↦row⁡AA\\mapsto\\operatorname\{row\}A; another is the orthogonal projectorsP=P⊤=P2P=P^\{\\top\}=P^\{2\}withtr⁡P=r\\tr P=r\. NearVV, the graphs of linear mapsV→V⟂V\\to V^\{\\perp\}give a chart ontoHom⁡\(V,V⟂\)≅ℝr⁡\(n−r\)\\operatorname\{Hom\}\(V,V^\{\\perp\}\)\\cong\\mathbb\{R\}^\{r\(n\-r\)\}\.Gr⁡\(r,n\)\\Gr\(r,n\)carries a uniqueO⁡\(n\)\\mathrm\{O\}\(n\)\-invariant probability measure, the uniform or Haar measure, and the row space of a GaussianAAhas this law \([Background D\.14](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\)\.

TA\\mathrm\{T\}\_\{A\}\-pointed LoRA is classified up to isomorphism byrow⁡A0∈Gr⁡\(r,n\)\\operatorname\{row\}A\_\{0\}\\in\\Gr\(r,n\)\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\), and its deficitr⁡\(n−r\)r\(n\-r\)equalsdimGr⁡\(r,n\)\\dim\\Gr\(r,n\)\. First steps keep the row space of the update insiderow⁡A0\\operatorname\{row\}A\_\{0\}, and the missing directions are those that tilt it\. For injectiveW0W\_\{0\}and evenrrwith2≤r≤n−22\\leq r\\leq n\-2, HRA’s image is the union of the orbitsW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\)overV∈Gr⁡\(r,n\)V\\in\\Gr\(r,n\), and its dimensionr⁡\(n−r\)\+r⁡\(r−1\)/2r\(n\-r\)\+r\(r\-1\)/2counts a subspace and a rotation inside it\. LoRA\-FA\([Zhang et al\., 2023d](https://arxiv.org/html/2610.12728#bib.bib556)\), which trainsBBwithA0A\_\{0\}frozen, seesA0A\_\{0\}only throughV=row⁡A0V=\\operatorname\{row\}A\_\{0\}once its gradient is corrected as in[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(c\)\.

Used in\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), the checks \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), LOFT \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\) and the LOFT example there,[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\. Reading:[Lee \(2013, Ch\. 1\)](https://arxiv.org/html/2610.12728#bib.bib237);[Harris \(1992, Lecture 6\)](https://arxiv.org/html/2610.12728#bib.bib156)\.

#### Tame sets and genericity

Fact C\.16\(images of smooth maps need not be manifolds\)\.Even a polynomial map can have an image that is no manifold\. \(a\)*Cusps\.*t↦\(t2,t3\)t\\mapsto\(t^\{2\},t^\{3\}\)is injective with image\{y2=x3\}\\\{y^\{2\}=x^\{3\}\\\}, which is notC1C^\{1\}at00\. \(b\)*Crossings and cones\.*The cross\{BA=0\}⊆ℝ2\\\{BA=0\\\}\\subseteq\\mathbb\{R\}^\{2\}and the coneℳ≤r\\mathcal\{M\}\_\{\\leq r\}are singular at the origin\. \(c\)*Missing limits\.*\(x,y\)↦\(x,x​y\)\(x,y\)\\mapsto\(x,xy\)has image\{u≠0\}∪\{\(0,0\)\}\\\{u\\neq 0\\\}\\cup\\\{\(0,0\)\\\}in the\(u,v\)\(u,v\)\-plane, neither open nor closed\. \(d\)*Dense images\.*The line of irrational slope is an injective immersion into the torus with dense image\.

PEFT images show \(b\) routinely, as in the LoRA cone and HRA’s skew cone, so the paper measures them with tools that need no manifold structure, such as germs \([Background C\.8](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\), tangent cones \([Background C\.11](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2.Px1)\) and the dimension theory of semialgebraic and definable sets \([Background C\.17](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4),[Background C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\)\. Whether LoHa’s Hadamard image is closed, the question raised by \(c\), is open\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), the open problems \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

Background C\.17\(semialgebraic and definable sets\)\. Asemialgebraic setis a finite union of sets\{x∈ℝN:p\(x\)=0,q1\(x\)\>0,…,qk\(x\)\>0\}\\\{x\\in\\mathbb\{R\}^\{N\}:p\(x\)=0,\\ q\_\{1\}\(x\)\>0,\\dots,q\_\{k\}\(x\)\>0\\\}with polynomialsp,qip,q\_\{i\}, and a map is semialgebraic if its graph is\. A set isdefinableinℝexp\\mathbb\{R\}\_\{\\exp\}if a first\-order formula defines it from polynomials with real coefficients,exp\\exp, equality, order, Boolean operations and quantifiers overℝ\\mathbb\{R\}\.[Wilkie \(1996\)](https://arxiv.org/html/2610.12728#bib.bib494)proved thatℝexp\\mathbb\{R\}\_\{\\exp\}iso\-minimal, meaning every definable subset ofℝ\\mathbb\{R\}is a finite union of points and intervals\. Semialgebraic sets are the definable sets that need noexp\\exp\.

LoRA, DoRA \(whose row norms involve square roots\), HRA and exact\-Cayley OFT\([Qiu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib381)\)are built from polynomials, quotients and square roots, so their images are semialgebraic by[Background C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\(a\)\. A softmax brings inexp\\exp, and maps that contain one are definable inℝexp\\mathbb\{R\}\_\{\\exp\}\.

Used in\.where the category theory works \(Exposé[0](https://arxiv.org/html/2610.12728#S0)\), the arrow rules \(§[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3)\)\. Reading:[Bochnak et al\. \(1998, Ch\. 2\)](https://arxiv.org/html/2610.12728#bib.bib40);[Wilkie \(1996\)](https://arxiv.org/html/2610.12728#bib.bib494)\.

Fact C\.18\(Tarski–Seidenberg and definable dimension\)\.\(a\) The image of a semialgebraic set under a semialgebraic map is semialgebraic \(Tarski–Seidenberg\); images of definable sets under definable maps are definable\. \(b\) Every definable setXXis a finite disjoint union ofC1C^\{1\}embedded submanifolds, anddimX\\dim X, the largest of their dimensions, does not depend on the choice\. \(c\)X⊆YX\\subseteq YimpliesdimX≤dimY\\dim X\\leq\\dim Y,dimf⁡\(X\)≤dimX\\dim f\(X\)\\leq\\dim Xfor definableff, anddim\(X¯∖X\)<dimX\\dim\(\\overline\{X\}\\setminus X\)<\\dim Xfor nonemptyXX\. \(d\) For a definableC1C^\{1\}mapρ\\rhoon an open definableQ⊆ℝkQ\\subseteq\\mathbb\{R\}^\{k\},dimρ⁡\(Q\)=maxq⁡rank⁡d​ρq\\dim\\rho\(Q\)=\\max\_\{q\}\\rk d\\rho\_\{q\}\.

##### The idea of \(d\)

Near a point of maximal rank,[Background C\.5](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)gives a sheet of that dimension insideρ⁡\(Q\)\\rho\(Q\)\. Conversely, a definable section ofρ\\rho,C1C^\{1\}on a top\-dimensional piece ofρ⁡\(Q\)\\rho\(Q\), composes withρ\\rhoto the identity there, so the rank is at least that dimension\.

Part \(d\) is the identityd⁡\(M\)=dimIm⁡Md\(M\)=\\dim\\Imop Mof[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\. Part \(c\) is the monotonicity behind test O2 \(§[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx2)\), since an arrowM→NM\\to NgivesIm⁡M⊆Im⁡N\\Imop M\\subseteq\\Imop Nand henced⁡\(M\)≤d⁡\(N\)d\(M\)\\leq d\(N\); for instanceLoRAr↛LoRA\-FA\\mathrm\{LoRA\}\_\{r\}\\not\\to\\text\{LoRA\-FA\}becauser⁡\(m\+n−r\)\>m​rr\(m\+n\-r\)\>mr\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), the arrow rules \(§[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3)\), where the category theory works \(Exposé[0](https://arxiv.org/html/2610.12728#S0)\)\. Reading:[van den Dries \(1998, Ch\. 3 and 4\)](https://arxiv.org/html/2610.12728#bib.bib457);[Bochnak et al\. \(1998, Ch\. 2\)](https://arxiv.org/html/2610.12728#bib.bib40)\.

Background C\.19\(stratification and local dimension\)\. Astratificationof a setXXis a partition into finitely many embedded submanifolds, thestrata, such that the closure of each stratum is a union of strata\. Thelocal dimensiondimxX\\dim\_\{x\}Xis the dimension of the germ ofXXatxx, the largest dimension of a stratum whose closure containsxx\.XXispure\-dimensionalifdimxX\\dim\_\{x\}Xis the same at every point\. Definable sets admit stratifications, so these notions make sense for every image in this paper\.

The cross\{ba=0\}⊆ℝ2\\\{ba=0\\\}\\subseteq\\mathbb\{R\}^\{2\}is stratified as the origin and the four open half\-axes, and it is pure of dimension11\. Likewiseℳ≤r=ℳ0⊔ℳ1⊔⋯⊔ℳr\\mathcal\{M\}\_\{\\leq r\}=\\mathcal\{M\}\_\{0\}\\sqcup\\mathcal\{M\}\_\{1\}\\sqcup\\dots\\sqcup\\mathcal\{M\}\_\{r\}withℳk¯=ℳ≤k\\overline\{\\mathcal\{M\}\_\{k\}\}=\\mathcal\{M\}\_\{\\leq k\}, and it is pure of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\. For1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)singles out three strata of full\-rank LoRA pointings,TA\\mathrm\{T\}\_\{A\}\(B0=0B\_\{0\}=0,rank⁡A0=r\\rk A\_\{0\}=r\),TB\\mathrm\{T\}\_\{B\}\(A0=0A\_\{0\}=0,rank⁡B0=r\\rk B\_\{0\}=r\) and the open denseS\\mathrm\{S\}\(rank⁡B0​A0=r\\rk B\_\{0\}A\_\{0\}=r\), and the fibre\{BA=0\}\\\{BA=0\\\}overθ0\\theta\_\{0\}is itself stratified \([Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\)\. The HRA fibre product of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(d\) has local dimensiond⁡\(HRAr\)\+r2=r​n\+r⁡\(r−1\)/2d\(\\mathrm\{HRA\}\_\{r\}\)\+r^\{2\}=rn\+r\(r\-1\)/2over generic points of the meet and contains\{\(B,0,I\)\}\\\{\(B,0,I\)\\\}, of dimensionm​rmr, overW0W\_\{0\}, so it is not pure\-dimensional whenm​r\>r​n\+r⁡\(r−1\)/2mr\>rn\+r\(r\-1\)/2\.

Used in\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), the three strata \(§[II\.A](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9), the open problems \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\. Reading:[van den Dries \(1998, Ch\. 4\)](https://arxiv.org/html/2610.12728#bib.bib457);[Bochnak et al\. \(1998, Ch\. 9\)](https://arxiv.org/html/2610.12728#bib.bib40)\.

Background C\.20\(moduli sets and moduli spaces\)\. Classifying objects up to isomorphism gives a set of classes\. Amoduli spacealso carries a topology or geometry for which continuous or smooth families of objects correspond to continuous or smooth maps into it, ideally with a universal family over it from which every family is pulled back\. A disjoint union of spacesX⊔YX\\sqcup Yhas the topology in whichXXandYYare both open and closed\.

[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)classifies the three strata of LoRA pointings, as a set, byGr⁡\(r,n\)⊔Gr⁡\(r,m\)⊔ℳr\\Gr\(r,n\)\\sqcup\\Gr\(r,m\)\\sqcup\\mathcal\{M\}\_\{r\}\. The quotient topology does not split this way, since\(ε​B′,A0\)∈S\(\\varepsilon B^\{\\prime\},A\_\{0\}\)\\in\\mathrm\{S\}tends to\(0,A0\)∈TA\(0,A\_\{0\}\)\\in\\mathrm\{T\}\_\{A\}asε→0\\varepsilon\\to 0, soTA\\mathrm\{T\}\_\{A\}lies in the closure ofS\\mathrm\{S\}\. No universal family is built, and the paper uses “moduli” as an analogy\.

Used in\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), the Rosetta stone \(Exposé[IX](https://arxiv.org/html/2610.12728#S9)\), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\. Reading:[Harris \(1992, Lecture 4\)](https://arxiv.org/html/2610.12728#bib.bib156)\.

Background C\.21\(generic properties\)\. A property of points ofℝN\\mathbb\{R\}^\{N\}holdsgenericallyif the set where it fails lies in\{f=0\}\\\{f=0\\\}for some nonzero polynomialff\. The set\{f=0\}\\\{f=0\\\}is closed, nowhere dense and of Lebesgue measure zero, so a generic property holds with probability one at a random point whose law has a density, such as a Gaussian or uniform initialisation\. For points of a parametrised set, such as a Segre cone, “generic” refers to generic parameters\. To prove genericity it suffices to write failure as the vanishing of polynomials and exhibit one point where one of them is nonzero\. A weaker notion,almost everypoint, asks only that the failure set have Lebesgue measure zero, as a countable union of proper algebraic subsets does\.

[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\) uses a generic rank\-one update,[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)gives DoRA’s dimensionmin⁡\(m​n,r⁡\(m\+n−r\)\+m\)\\min\\big\(mn,\\,r\(m\+n\-r\)\+m\\big\)for genericW0W\_\{0\}, and[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)notes that QLoRA’s\([Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92)\)defecte=κ​W−We=\\kappa W\-Whas full rank for almost everyWW, in the weaker sense\. The quantiserκ\\kappais defined cell by cell rather than by one polynomial, so the argument runs cell by cell, and the failure set is a countable union of proper algebraic subsets of quantisation cells, which has measure zero\. A pretrainedW0W\_\{0\}is not random, and structured weights can lie in the exceptional set\. For a rank\-oneW0=a​c⊤W\_\{0\}=ac^\{\\top\}with allai≠0a\_\{i\}\\neq 0andr<nr<n,[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)finds DoRA’s dimensionr⁡\(m\+n−r\)\+m−rr\(m\+n\-r\)\+m\-r, strictly below the generic value exactly whenr<mr<mandr≤n−2r\\leq n\-2\.

Used in\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\.

Background C\.22\(real\-analytic functions and their zero sets\)\. A function isreal\-analyticon an open set if near every point it equals a convergent power series\. Polynomials,exp\\exp, softmax,tanh\\tanh, GELU and SiLU are real\-analytic, and so are sums, products, compositions and quotients with nonvanishing denominator\. On a connected open set, a real\-analytic function that vanishes on a nonempty open subset vanishes identically, and the zero set of one that is not identically zero is closed, nowhere dense and of measure zero\.

[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)compares Taylor coefficients at00\. A nonzero scaleccwithσ⁡\(c​t\)=c​σ​\(t\)\\sigma\(ct\)=c\\,\\sigma\(t\)must satisfyc​ak=ck​akca\_\{k\}=c^\{k\}a\_\{k\}for every coefficientaka\_\{k\}ofσ\\sigma, soc=1c=1for GELU and SiLU, whosea2≠0a\_\{2\}\\neq 0\.[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\) uses the zero\-set fact\. A prefixed head is analytic in the prefix, and one prefix with a nonzero second derivative makes the prefixes with affine output a nowhere\-dense set\. The same fact makes the maximal\-rank set of an analyticρ\\rhoopen and dense \([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\)\.

Used in\.[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3),[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6), Rosetta entry 5 \(Exposé[IX](https://arxiv.org/html/2610.12728#S9)\)\. Reading:[Krantz and Parks \(2002\)](https://arxiv.org/html/2610.12728#bib.bib226)\.

### Part D\. Groups, symmetry and conservation

Part D turns to symmetry\. Most PEFT parametrisations have symmetries, since different trainable parameters can give the same weights and some transformations of weight space carry a method’s image to itself\. This part collects the group theory the paper uses to describe those symmetries, the invariants they leave untouched, and the quantities they conserve under gradient flow\.

#### Groups and their Lie algebras

Background D\.1\(Lie groups; the matrix groups of this paper\)\. ALie groupis a group that is also a smooth manifold, with smooth multiplication and inversion\. Every closed subgroup ofGLn​\(ℝ\)\\mathrm\{GL\}\_\{n\}\(\\mathbb\{R\}\)is a Lie group \(Cartan’s closed\-subgroup theorem\), and all groups below are of this kind\. Its dimension is its dimension as a manifold\.

LoRA’s gauge isGLr\\mathrm\{GL\}\_\{r\}, acting on factors byg⋅\(B,A\)=\(B​g−1,g​A\)g\\cdot\(B,A\)=\(Bg^\{\-1\},gA\)\. Exactly orthogonal OFT keeps the weight inside the orbitW0⋅O⁡\(n\)W\_\{0\}\\cdot\\mathrm\{O\}\(n\), and HRA with evenrrkeeps it insideW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\.CO⁡\(n\)\\mathrm\{CO\}\(n\)is exactly the input\-side covariance group \([Background D\.17](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\) of the PiSSA\([Meng et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib324)\), MiLoRA\([Wang et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib465)\)and paper\-form CorDA\([Yang et al\., 2024c](https://arxiv.org/html/2610.12728#bib.bib528)\)splits\. LoCO’s\([Nguyen et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib347)\)skew generator satisfiesU​V⊤−V​U⊤=Z​J​Z⊤UV^\{\\top\}\-VU^\{\\top\}=ZJZ^\{\\top\}withZ=\[U​V\]Z=\[U\\ V\], so it is unchanged underZ↦Z​MZ\\mapsto ZMfor everyM∈Sp⁡\(2​r,ℝ\)M\\in\\mathrm\{Sp\}\(2r,\\mathbb\{R\}\)\.

Used in\.the notation \(Table[2](https://arxiv.org/html/2610.12728#S0.T2)\),[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1),[Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7), LOFT \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\),[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\. Reading:[Hall \(2015, Ch\. 1\)](https://arxiv.org/html/2610.12728#bib.bib149)\.

Background D\.2\(tori, diagonal monoids and saturation\)\. Thediagonal torusTm=\{diag⁡\(d\):di≠0\}≅\(ℝ×\)mT\_\{m\}=\\\{\\diag\(d\):d\_\{i\}\\neq 0\\\}\\cong\(\\mathbb\{R\}^\{\\times\}\)^\{m\}is a commutative Lie group with2m2^\{m\}connected components, andTm\+=\{diag⁡\(d\):di\>0\}≅ℝ\>0mT\_\{m\}^\{\+\}=\\\{\\diag\(d\):d\_\{i\}\>0\\\}\\cong\\mathbb\{R\}\_\{\>0\}^\{m\}is the component of the identity\.Diagm\\mathrm\{Diag\}\_\{m\}, the set of all diagonal matrices including singular ones, is a monoid under multiplication \([Background D\.10](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) whose invertible elements formTmT\_\{m\}\. Identifyingℝ2\\mathbb\{R\}^\{2\}withℂ\\mathbb\{C\}, the scaled rotationsa​Rot​\(θ\)a\\,\\mathrm\{Rot\}\(\\theta\)witha\>0a\>0formCO\+​\(2\)≅ℂ×\\mathrm\{CO\}^\{\+\}\(2\)\\cong\\mathbb\{C\}^\{\\times\}, througha​Rot​\(θ\)↦a​ei​θa\\,\\mathrm\{Rot\}\(\\theta\)\\mapsto ae^\{i\\theta\}\. If a group or monoidGGacts on a space containing a setXX, thesaturationG⋅X=\{g⋅x:g∈G,x∈X\}G\\cdot X=\\\{g\\cdot x:g\\in G,\\ x\\in X\\\}is the smallestGG\-stable set containingXX\.

\(IA\)3\([Liu et al\., 2022a](https://arxiv.org/html/2610.12728#bib.bib275)\)on keys and values mapsℓ↦diag⁡\(ℓ\)​W0\\ell\\mapsto\\diag\(\\ell\)W\_\{0\}, so its imageDiagm⋅W0\\mathrm\{Diag\}\_\{m\}\\cdot W\_\{0\}is the closure of the torus orbitTm⋅W0T\_\{m\}\\cdot W\_\{0\}\. DoRA’s image is the saturationDiagm⋅\(W0\+ℳ≤r\)\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)of LoRA’s cone\. RoAd1\([Liao and Monz, 2024](https://arxiv.org/html/2610.12728#bib.bib263)\)is the left action ofCO\+​\(2\)m/2≅\(ℂ×\)m/2\\mathrm\{CO\}^\{\+\}\(2\)^\{m/2\}\\cong\(\\mathbb\{C\}^\{\\times\}\)^\{m/2\}on pairs of output rows, which makes it \(IA\)3overℂ\\mathbb\{C\}\. LoHa’s gauge contains a two\-sided diagonal torus\.

Used in\.[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), DoRA \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), the hyperparameter torus \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\)\.

Background D\.3\(permutations, monomial matrices, semidirect products, normalisers\)\.SdS\_\{d\}is the group of permutations ofddobjects, realised by permutation matrices\. A matrix ismonomialif each row and each column has exactly one nonzero entry\. Every monomial matrix factors uniquely asD​PDPwithD∈TmD\\in T\_\{m\}andPPa permutation matrix, andTmT\_\{m\}is normal in the groupMonm\\mathrm\{Mon\}\_\{m\}of monomial matrices, soMonm=Tm⋊Sm\\mathrm\{Mon\}\_\{m\}=T\_\{m\}\\rtimes S\_\{m\}\. Here asemidirect productN⋊HN\\rtimes His a group with a normal subgroupNNand a subgroupHHsuch thatN∩H=\{e\}N\\cap H=\\\{e\\\}andN​HNHis the whole group\. Thesigned permutationsBr=\{±1\}r⋊Sr=Monr∩O⁡\(r\)B\_\{r\}=\\\{\\pm 1\\\}^\{r\}\\rtimes S\_\{r\}=\\mathrm\{Mon\}\_\{r\}\\cap\\mathrm\{O\}\(r\)are the monomial matrices with entries±1\\pm 1, and\|Br\|=2r​r\!\\lvert B\_\{r\}\\rvert=2^\{r\}r\!\. Thenormaliserof a subsetH⊆GH\\subseteq GisNG​\(H\)=\{g∈G:g​H​g−1=H\}N\_\{G\}\(H\)=\\\{g\\in G:gHg^\{\-1\}=H\\\}\.

A monomialggsatisfiesg​Diagm​g−1=Diagmg\\,\\mathrm\{Diag\}\_\{m\}\\,g^\{\-1\}=\\mathrm\{Diag\}\_\{m\}, and inGLm\\mathrm\{GL\}\_\{m\}the normaliser ofTmT\_\{m\}is exactlyMonm\\mathrm\{Mon\}\_\{m\}\. This putsMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}inside the covariance group \([Background D\.17](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\) of output scalings such as \(IA\)3and DoRA’s magnitude\. Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.12728#bib.bib216)\)acts coordinatewise, so on LoRA’s full\-rank locus with channel\-uniform hyperparameters \(one learning rate,ε\\varepsilonand decay per factor\) it is equivariant, of LoRA’s gauge, only underBr⊂O⁡\(r\)⊂GLrB\_\{r\}\\subset\\mathrm\{O\}\(r\)\\subset\\mathrm\{GL\}\_\{r\}\([Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\. By[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3), an invertible linear map that commutes with a non\-affineC2C^\{2\}activation is monomial\.

Used in\.DoRA \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6),[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4),[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3), the privileged basis \(§[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\)\.

Background D\.4\(connectedness and the identity component\)\. Theidentity componentG∘G^\{\\circ\}of a Lie groupGGis the connected component containingee\. It is an open normal subgroup\.GLn\\mathrm\{GL\}\_\{n\}has two components, told apart by the sign of the determinant, andGL\+\(n\)=\{det\>0\}\\mathrm\{GL\}^\{\+\}\(n\)=\\\{\\det\>0\\\}is the identity component; likewiseSO⁡\(n\)⊂O⁡\(n\)\\mathrm\{SO\}\(n\)\\subset\\mathrm\{O\}\(n\)andTm\+⊂TmT\_\{m\}^\{\+\}\\subset T\_\{m\}\. A subsetUUissymmetricifU−1=UU^\{\-1\}=U\. Every neighbourhood ofeecontains a symmetric one, and in a connected group a symmetric neighbourhoodUUofeegenerates the group as a monoid, so every element is a finite productu1⋯uku\_\{1\}\\cdots u\_\{k\}withui∈Uu\_\{i\}\\in U\.

In merge\-and\-restart of a method of action type \([Background D\.12](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\), each cycle applies one element of a setSS\. IfSScontains a neighbourhood of the identity in a connected groupHH, repeated merging reaches all ofHH\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\)\)\. HF’s\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)Cayley–Neumann OFT factors havedetR≥0\\det R\\geq 0, so the invertible ones lie inGL\+​\(b\)\\mathrm\{GL\}^\{\+\}\(b\); they generate∏GL\+​\(b\)\\prod\\mathrm\{GL\}^\{\+\}\(b\)as a group, or∏CO\+​\(2\)\\prod\\mathrm\{CO\}^\{\+\}\(2\)whenb=2b=2\.

Used in\.complete invariants \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), the corollaries of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.

Background D\.5\(Lie algebras; the commutator;𝔤​𝔩\\mathfrak\{gl\},𝔰​𝔬\\mathfrak\{so\},Sym\\mathrm\{Sym\},∧\\wedge\)\. TheLie algebraof a matrix groupG⊆GLnG\\subseteq\\mathrm\{GL\}\_\{n\}is𝔤=\{X:et​X∈G​for all​t∈ℝ\}\\mathfrak\{g\}=\\\{X:e^\{tX\}\\in G\\text\{ for all \}t\\in\\mathbb\{R\}\\\}, the tangent space ofGGatII\. It is a vector space closed under thecommutator\[X,Y\]=X​Y−Y​X\[X,Y\]=XY\-YX\. The paper uses𝔤​𝔩r\\mathfrak\{gl\}\_\{r\}\(allr×rr\\times rmatrices, forGLr\\mathrm\{GL\}\_\{r\}\),𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)\(skew matricesΩ⊤=−Ω\\Omega^\{\\top\}=\-\\Omega, forO⁡\(n\)\\mathrm\{O\}\(n\)andSO⁡\(n\)\\mathrm\{SO\}\(n\)\), the diagonal matrices \(forTmT\_\{m\}\),ℝ​I⊕𝔰​𝔬​\(n\)\\mathbb\{R\}I\\oplus\\mathfrak\{so\}\(n\)\(forCO⁡\(n\)\\mathrm\{CO\}\(n\)\) and𝔰​𝔭​\(2​r\)=\{X:X⊤​J\+J​X=0\}\\mathfrak\{sp\}\(2r\)=\\\{X:X^\{\\top\}J\+JX=0\\\}\. For a subspaceV⊆ℝnV\\subseteq\\mathbb\{R\}^\{n\},𝔰​𝔬​\(V\)\\mathfrak\{so\}\(V\)consists of the skew maps ofVVextended by00onV⟂V^\{\\perp\}\. The symmetric matricesSymr\\mathrm\{Sym\}\_\{r\}do not form a Lie algebra, since\[S,S′\]\[S,S^\{\\prime\}\]is skew\. Writev∧w=v​w⊤−w​v⊤∈𝔰​𝔬​\(n\)v\\wedge w=vw^\{\\top\}\-wv^\{\\top\}\\in\\mathfrak\{so\}\(n\),U∧VU\\wedge Vfor the span of theu∧vu\\wedge vwithu∈Uu\\in U,v∈Vv\\in V, andLa​b=ea∧eb=Ea​b−Eb​aL\_\{ab\}=e\_\{a\}\\wedge e\_\{b\}=E\_\{ab\}\-E\_\{ba\}\. Then\[La​c,Lc​b\]=La​b\[L\_\{ac\},L\_\{cb\}\]=L\_\{ab\}for distincta,b,ca,b,c\.

For injectiveW0W\_\{0\}, at HRA’s paired initialisationu2​i−1=u2​i=wiu\_\{2i\-1\}=u\_\{2i\}=w\_\{i\}\(i≤k=r/2i\\leq k=r/2, with independentwiw\_\{i\}andU=span⁡\{wi\}U=\\spanop\\\{w\_\{i\}\\\}of dimensionkk\), the first\-order image isS1=W0⋅\{Ω∈𝔰​𝔬​\(n\):PU⟂​Ω​PU⟂=0\}=W0⋅\(U∧ℝn\)S\_\{1\}=W\_\{0\}\\cdot\\\{\\Omega\\in\\mathfrak\{so\}\(n\):P\_\{U^\{\\perp\}\}\\Omega P\_\{U^\{\\perp\}\}=0\\\}=W\_\{0\}\\cdot\(U\\wedge\\mathbb\{R\}^\{n\}\), a subspace ofW0⋅𝔰​𝔬​\(n\)W\_\{0\}\\cdot\\mathfrak\{so\}\(n\)of dimensionk​n−k⁡\(k\+1\)/2kn\-k\(k\+1\)/2\([Background C\.13](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\. LOFT\([Zhao et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib582)\)trains a skewE∈𝔰​𝔬​\(r\)E\\in\\mathfrak\{so\}\(r\)inside a fixedrr\-dimensional input subspace\. The bracket identity lets rotations of overlapping coordinate blocks generate rotations of their union \([Background D\.8](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\)\.

Used in\.[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), the LOFT example and LOFT fixes one leaf \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\),[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\. Reading:[Hall \(2015, Ch\. 3\)](https://arxiv.org/html/2610.12728#bib.bib149)\.

Background D\.6\(the exponential map; one\-parameter subgroups\)\. ForX∈𝔤X\\in\\mathfrak\{g\}, the curvet↦et​Xt\\mapsto e^\{tX\}of matrix exponentials is aone\-parameter subgroup, a smooth homomorphism\(ℝ,\+\)→G\(\\mathbb\{R\},\+\)\\to G, and every continuous one\-parameter subgroup has this form for a uniqueXX, itsgenerator\. The exponential map𝔤→G\\mathfrak\{g\}\\to Ghas derivativeid\\mathrm\{id\}at00, so it maps a neighbourhood of00diffeomorphically onto a neighbourhood ofII\. It is onto forSO⁡\(n\)\\mathrm\{SO\}\(n\)and not onto forGL\+​\(n\)\\mathrm\{GL\}^\{\+\}\(n\), sincediag⁡\(−1,−2\)\\diag\(\-1,\-2\)iseXe^\{X\}for no realXX\.

The one\-parameter subgroupt↦\(B​e−t​X,et​X​A\)t\\mapsto\(Be^\{\-tX\},e^\{tX\}A\)of LoRA’s gauge moves along a fibre without changingB​ABA\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)builds its isometries ask=eτ​ξk=e^\{\\tau\\xi\}from generatorsξ\\xiwithΛ−1​ξ\\Lambda^\{\-1\}\\xiantisymmetric\. LOFT may use the exponential as its map𝔰​𝔬​\(r\)→SO⁡\(r\)\\mathfrak\{so\}\(r\)\\to\\mathrm\{SO\}\(r\), and the Kempf–Ness argument \([Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\) differentiates the norm alonget​ξe^\{t\\xi\}\.

Used in\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), LOFT \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\. Reading:[Hall \(2015, Ch\. 2\)](https://arxiv.org/html/2610.12728#bib.bib149)\.

Background D\.7\(the Cayley map and its Neumann truncation\)\. For skewΩ\\Omegathe eigenvalues are imaginary, soI−ΩI\-\\Omegais invertible, and theCayley mapisC⁡\(Ω\)=\(I\+Ω\)​\(I−Ω\)−1\.C\(\\Omega\)=\(I\+\\Omega\)\(I\-\\Omega\)^\{\-1\}\.It lands inSO⁡\(n\)\\mathrm\{SO\}\(n\), andC⁡\(Ω\)−I=2​Ω​\(I−Ω\)−1C\(\\Omega\)\-I=2\\Omega\(I\-\\Omega\)^\{\-1\}, sorank⁡\(C⁡\(Ω\)−I\)=rank⁡Ω\\rk\(C\(\\Omega\)\-I\)=\\rk\\Omega\. Its expansionC⁡\(Ω\)=I\+2​Ω\+2​Ω2\+⋯C\(\\Omega\)=I\+2\\Omega\+2\\Omega^\{2\}\+\\cdotsshows that its derivative at00is2​id2\\,\\mathrm\{id\}\. It is a diffeomorphism from𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)onto the open dense set of rotations with no eigenvalue−1\-1, with inverseR↦\(R−I\)​\(R\+I\)−1R\\mapsto\(R\-I\)\(R\+I\)^\{\-1\}, so it misses, for instance,diag⁡\(−1,−1,1,…,1\)\\diag\(\-1,\-1,1,\\dots,1\)\.

Exact\-Cayley OFT and block OFT are orthogonal, so they preserveKout=W​W⊤K\_\{\\mathrm\{out\}\}=WW^\{\\top\}\([Background D\.15](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\) and the spectrum\. HF PEFT’s default \(since version 0\.18\) truncates\(I−Q\)−1\(I\-Q\)^\{\-1\}toI\+Q\+Q2\+Q3I\+Q\+Q^\{2\}\+Q^\{3\}, which givesR=C⁡\(Q\)​\(I−Q4\)R=C\(Q\)\(I\-Q^\{4\}\)\. AsQ4Q^\{4\}is symmetric and commutes withC⁡\(Q\)C\(Q\),R⊤​R=\(I−Q4\)2≠IR^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}\\neq I, and for smallQQeach merge shrinksKoutK\_\{\\mathrm\{out\}\}\. The rank identity is how[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(c\) identifies HRA’s image germ with the cone of skew matrices of rank at mostrr\.

Used in\.the thesis \(Exposé[0](https://arxiv.org/html/2610.12728#S0)\),[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), the Gram lab of the companion site \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\), the corollaries of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.

Background D\.8\(Lie subgroups and subalgebras; the generated Lie algebra\)\. ALie subgroupofGGis a subgroup with a Lie group structure for which the inclusion is an injective immersion; it need not be closed\. Connected Lie subgroupsH⊆GH\\subseteq Gcorrespond one to one to Lie subalgebras𝔥⊆𝔤\\mathfrak\{h\}\\subseteq\\mathfrak\{g\}, withHHthe subgroup generated bye𝔥e^\{\\mathfrak\{h\}\}\. The Lie algebrageneratedby subalgebras𝔥1,…,𝔥p\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}, writtenLie⁡⟨𝔥1,…,𝔥p⟩\\operatorname\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle, is the smallest subalgebra containing them, the span of their elements and all iterated brackets of them\.

Block OFT, BOFT\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285)\)factors and Givens rotations\([Ma et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib305)\)use𝔰​𝔬​\(Ij\)\\mathfrak\{so\}\(I\_\{j\}\)for coordinate blocksIj⊆\{1,…,n\}I\_\{j\}\\subseteq\\\{1,\\dots,n\\\}\. Since\[La​c,Lc​b\]=La​b\[L\_\{ac\},L\_\{cb\}\]=L\_\{ab\}, two overlapping blocks generate𝔰​𝔬\\mathfrak\{so\}of their union, andLie⁡⟨𝔰​𝔬​\(Ij\)⟩j=⨁C𝔰​𝔬​\(C\)\\operatorname\{Lie\}\\langle\\mathfrak\{so\}\(I\_\{j\}\)\\rangle\_\{j\}=\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\), summed over the connected componentsCCof the hypergraph with vertices\{1,…,n\}\\\{1,\\dots,n\\\}and hyperedgesIjI\_\{j\}\. Two vertices share a component when a chain of hyperedges, each meeting the next, joins them\. Butterfly factors connect this hypergraph only at full depth\.

Used in\.[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), Figure[10](https://arxiv.org/html/2610.12728#S5.F10)\(b\), the corollaries of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\), LOFT fixes one leaf \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\)\. Reading:[Lee \(2013, Ch\. 19\)](https://arxiv.org/html/2610.12728#bib.bib237);[Hall \(2015, Ch\. 5\)](https://arxiv.org/html/2610.12728#bib.bib149)\.

Fact D\.9\(Yamabe’s theorem\)\.Every path\-connected subgroup of a Lie group is a Lie subgroup\. Consequently, ifH1,…,HpH\_\{1\},\\dots,H\_\{p\}are connected Lie subgroups ofGG, the subgroup they generate is the connected Lie subgroupHHwithLie⁡\(H\)=Lie⁡⟨𝔥1,…,𝔥p⟩\\operatorname\{Lie\}\(H\)=\\operatorname\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle\([Yamabe, 1950](https://arxiv.org/html/2610.12728#bib.bib519)\); a short proof is in[Goto \(1969\)](https://arxiv.org/html/2610.12728#bib.bib134)\.

The subgroup generated by theHiH\_\{i\}is path\-connected, because products of paths are paths, so the theorem makes it a Lie subgroup and the correspondence of[Background D\.8](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)names it\. With[Background D\.4](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1), this is why a re\-merged action method whose cycles contain a neighbourhood of the identity in eachHiH\_\{i\}reaches exactly the connected subgroup with Lie algebraLie⁡⟨𝔥1,…,𝔥p⟩\\operatorname\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle, and why exact\-Cayley block\-orthogonal factors reach∏CSO⁡\(C\)\\prod\_\{C\}\\mathrm\{SO\}\(C\)\.

Used in\.[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), the corollaries of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.

Background D\.10\(monoids, units, monoid orbits, homomorphisms\)\. Amonoidis a set with an associative product and an identity\. Its invertible elements form itsgroup of units\. The submonoidgeneratedby a subsetSSconsists of all finite productss1⋯sks\_\{1\}\\cdots s\_\{k\}of elements ofSS, the empty product being the identity\. A monoid acts on a set when1⋅x=x1\\cdot x=xands⋅\(t⋅x\)=\(s​t\)⋅xs\\cdot\(t\\cdot x\)=\(st\)\\cdot x, and themonoid orbitofxxis\{s⋅x\}\\\{s\\cdot x\\\}\. Ahomomorphismof monoids preserves products and the identity\. Examples areDiagm\\mathrm\{Diag\}\_\{m\},M2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\)andEnd⁡\(Θ\)\\operatorname\{End\}\(\\Theta\)under multiplication, with unitsTmT\_\{m\},GL2​\(ℝ\)\\mathrm\{GL\}\_\{2\}\(\\mathbb\{R\}\)andGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\), and finite multisets under disjoint union⊎\\uplus\. Additive monoidsMon⁡\(C\)\\mathrm\{Mon\}\(C\)are in[Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\.

Merge\-and\-restart reaches the orbit of the monoid that one cycle generates, which isθ0\+Mon⁡\(C\)\\theta\_\{0\}\+\\mathrm\{Mon\}\(C\)for an additive methodρ=θ0\+δ\\rho=\\theta\_\{0\}\+\\deltawith shapeC=Im⁡δC=\\Imop\\delta\([Background B\.14](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\) and⟨S⟩⋅θ0\\langle S\\rangle\\cdot\\theta\_\{0\}for a method of action type \([Background D\.12](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\) whose cycle applies elements ofSS\. RoAd2’s blocks range overM2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\)\. For a fixed query, attention’s pair \(total weight, weighted sum of values\) is a homomorphism from multisets of key–value pairs to\(ℝ≥0×ℝdv,\+\)\(\\mathbb\{R\}\_\{\\geq 0\}\\times\\mathbb\{R\}^\{d\_\{v\}\},\+\)\([Background D\.34](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)\)\.

Used in\.[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), restarting \(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\), DoRA \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6),[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6), attention \(§[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\)\.

Fact D\.11\(Cartan–Dieudonné theorem; Scherk’s sharp form\)\.For nonzerou∈ℝnu\\in\\mathbb\{R\}^\{n\}, thereflectionHu=I−2​u​u⊤/‖u‖2H\_\{u\}=I\-2uu^\{\\top\}/\\\|u\\\|^\{2\}fixes the hyperplaneu⟂u^\{\\perp\}and hasdetHu=−1\\det H\_\{u\}=\-1\. EveryR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)is a product of at mostnnreflections \(Cartan–Dieudonné\)\. In Scherk’s sharp form, ifk=rank⁡\(R−I\)k=\\rk\(R\-I\), thenRRis a product of exactlykkreflections and of no fewer\. HencedetR=\(−1\)k\\det R=\(\-1\)^\{k\}, and everyR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)is a product ofrank⁡\(R−I\)\\rk\(R\-I\)reflections, an even number\. See[Scherk \(1950\)](https://arxiv.org/html/2610.12728#bib.bib399)and[Artin \(1957, Ch\. III\)](https://arxiv.org/html/2610.12728#bib.bib16)\.

##### The idea of the proof

A product ofjjreflections fixes an intersection ofjjhyperplanes, sorank⁡\(R−I\)≤j\\rk\(R\-I\)\\leq j\. Conversely, pickxxwithR​x≠xRx\\neq xand putu=R​x−xu=Rx\-x\. ThenHuH\_\{u\}swapsR​xRxandxxand fixes every fixed vector ofRR, soHu​RH\_\{u\}Rfixes one more dimension, and induction onkkfinishes\.

HRA multipliesrrreflections, sorank⁡\(R−I\)≤r\\rk\(R\-I\)\\leq r, anddetR=1\\det R=1for evenrr\. Scherk’s form gives the converse\. For injectiveW0W\_\{0\}and evenrr, HRA reaches all of\(W0\+ℳ≤r\)∩W0⋅SO⁡\(n\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), padding with pairsHu​Hu=IH\_\{u\}H\_\{u\}=I, and re\-merged HRA reachesW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)within⌈n/r⌉\\lceil n/r\\rceilcycles\.

Used in\.[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), HRA is a meet \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\), the corollaries of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\)\.

#### Actions, orbits and invariants

Background D\.12\(group actions\)\. A \(left\)actionof a groupGGon a setXXis a map\(g,x\)↦g⋅x\(g,x\)\\mapsto g\\cdot xwithe⋅x=xe\\cdot x=xandg⋅\(h⋅x\)=\(g​h\)⋅xg\\cdot\(h\\cdot x\)=\(gh\)\\cdot x\. Arightactionx⋅gx\\cdot gsatisfiesx⋅\(g​h\)=\(x⋅g\)⋅hx\\cdot\(gh\)=\(x\\cdot g\)\\cdot h, andg⋅x:=x⋅g−1g\\cdot x:=x\\cdot g^\{\-1\}turns it into a left one\. The action issmoothifG×X→XG\\times X\\to Xis smooth, andlinearifXXis a vector space and each mapx↦g⋅xx\\mapsto g\\cdot xis linear\.

On weights,W↦W​RW\\mapsto WRis a right action ofO⁡\(n\)\\mathrm\{O\}\(n\)on the input side andW↦P​WW\\mapsto PWa left action ofO⁡\(m\)\\mathrm\{O\}\(m\)on the output side, and\(g,h\)⋅W=g​W​h−1\(g,h\)\\cdot W=gWh^\{\-1\}is a left action ofGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\. On LoRA’s factors,g⋅\(B,A\)=\(B​g−1,g​A\)g\\cdot\(B,A\)=\(Bg^\{\-1\},gA\)is a linear left action ofGLr\\mathrm\{GL\}\_\{r\}\. A method is ofaction typewhenρ⁡\(q\)=γ⁡\(q\)⋅θ0\\rho\(q\)=\\gamma\(q\)\\cdot\\theta\_\{0\}for a mapγ\\gammainto a group acting onΘ\\Theta, as for OFT and HRA\. \(Thisγ\\gammafollows the exposés and is unrelated to the cotangent signalγ⁡\(t\)\\gamma\(t\)of[Background C\.2](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)and[Background D\.26](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4)\.\) \(IA\)3has the same form with the monoidDiagm\\mathrm\{Diag\}\_\{m\}in place of a group, and DoRA’s magnitude applies that monoid to LoRA’s weight\.

Used in\.[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1), the notation \(Table[2](https://arxiv.org/html/2610.12728#S0.T2)\), Erlangen \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\),[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1),[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\.

Background D\.13\(orbits, stabilisers, free actions\)\. TheorbitofxxisG⋅x=\{g⋅x:g∈G\}G\\cdot x=\\\{g\\cdot x:g\\in G\\\}, its closure is theorbit closure, and itsstabiliseris the closed subgroupGx=\{g:g⋅x=x\}G\_\{x\}=\\\{g:g\\cdot x=x\\\}\. For a smooth action of a Lie group,g↦g⋅xg\\mapsto g\\cdot xinduces an injective immersionG/Gx→XG/G\_\{x\}\\to Xwith imageG⋅xG\\cdot x, so the orbit is an immersed submanifold of dimensiondimG−dimGx\\dim G\-\\dim G\_\{x\}\. Theinfinitesimal stabiliser\{ξ∈𝔤:ξ⋅x=0\}\\\{\\xi\\in\\mathfrak\{g\}:\\xi\\cdot x=0\\\}is the Lie algebra ofGxG\_\{x\}\. The action isfreeif every stabiliser is trivial, and thegeneric stabiliseris the stabiliser at a generic point, one in a suitable dense open set\. Each mapx↦g⋅xx\\mapsto g\\cdot xcarries an orbit onto itself, so all its points look alike; orbits arehomogeneous\.

GLr\\mathrm\{GL\}\_\{r\}acts freely on pairs withrank⁡A=r\\rk A=r, sinceg​A=AgA=Aforcesg=Ig=I, so LoRA’s generic fibre, a single orbit, has dimensionr2r^\{2\}\. LoHa’s joint gaugeGLr1×GLr2×Tm×Tn\\mathrm\{GL\}\_\{r\_\{1\}\}\\times\\mathrm\{GL\}\_\{r\_\{2\}\}\\times T\_\{m\}\\times T\_\{n\}has a one\-dimensional generic stabiliser, so its orbits have dimensionr12\+r22\+m\+n−1r\_\{1\}^\{2\}\+r\_\{2\}^\{2\}\+m\+n\-1\. By homogeneity, the two\-sided torus orbit\{diag⁡\(a\)​W0​diag⁡\(b\)\}\\\{\\diag\(a\)W\_\{0\}\\diag\(b\)\\\}has dimensionm\+n−c⁡\(W0\)m\+n\-c\(W\_\{0\}\)at each of its points, withc⁡\(W0\)c\(W\_\{0\}\)the number of connected components of the support graph, the bipartite graph on themmrows andnncolumns with an edge\{i,j\}\\\{i,j\\\}where\(W0\)i​j≠0\(W\_\{0\}\)\_\{ij\}\\neq 0\. The images of \(IA\)3and RoAd1are orbit closures\.

Used in\.Erlangen \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\),[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6), the image germ \(Exposé[II](https://arxiv.org/html/2610.12728#S2)\),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), Rosetta entries 6 and 11 \(Exposé[IX](https://arxiv.org/html/2610.12728#S9)\)\. Reading:[Lee \(2013, Ch\. 7 and 21\)](https://arxiv.org/html/2610.12728#bib.bib237)\.

Background D\.14\(Haar measure; invariant measures on homogeneous spaces\)\. A compact Lie groupKKcarries a unique invariant probability measure, itsHaar measure\. IfKKacts transitively onXX, so thatX≅K/KxX\\cong K/K\_\{x\}is ahomogeneous space, thenXXcarries a uniqueKK\-invariant probability measure, the image of Haar measure underk↦k⋅xk\\mapsto k\\cdot x\. The GrassmannianGr⁡\(r,n\)\\Gr\(r,n\)ofrr\-dimensional subspaces ofℝn\\mathbb\{R\}^\{n\}\([Background C\.15](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\) is a homogeneous space ofO⁡\(n\)\\mathrm\{O\}\(n\)\.

IfA0∈ℝr×nA\_\{0\}\\in\\mathbb\{R\}^\{r\\times n\}has i\.i\.d\. Gaussian entries, its law is unchanged byA0↦A0​RA\_\{0\}\\mapsto A\_\{0\}RforR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\), andA0A\_\{0\}has rankrralmost surely\. Sorow⁡A0\\operatorname\{row\}A\_\{0\}is anO⁡\(n\)\\mathrm\{O\}\(n\)\-invariant random point ofGr⁡\(r,n\)\\Gr\(r,n\), and by uniqueness it is Haar\-distributed\. Vanilla LoRA with a GaussianA0A\_\{0\}therefore picks a Haar\-random point ofGr⁡\(r,n\)\\Gr\(r,n\), the space that classifies its stratumTA\\mathrm\{T\}\_\{A\}\. Kaiming\-uniform entries are not rotation invariant, so under HF’s default the point is not exactly Haar\-distributed\.

Used in\.the three strata \(§[II\.A](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\. Reading:[Mattila \(1995, Ch\. 3\)](https://arxiv.org/html/2610.12728#bib.bib321)\.

Background D\.15\(invariants and complete invariants\)\. A mapf:X→Yf:X\\to Yisinvariantunder an action iff⁡\(g⋅x\)=f⁡\(x\)f\(g\\cdot x\)=f\(x\)for allggandxx, so it is constant on orbits\. It is acomplete invariantif moreoverf⁡\(x\)=f⁡\(x′\)f\(x\)=f\(x^\{\\prime\}\)impliesx′∈G⋅xx^\{\\prime\}\\in G\\cdot x; it then identifies the orbit spaceX/GX/Gwith the image offf\.

If a method’s image lies in one orbit, every invariant of that action stays fixed through training and merging\. This is why exactly orthogonal methods never changeKoutK\_\{\\mathrm\{out\}\}or the spectrum \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\)\. Isomorphism invariants of methods, such asS1S\_\{1\},d⁡\(M\)d\(M\)and the first\-order deficit, play the same role one level up, since methods with different values cannot be isomorphic\.

Used in\.[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), complete invariants and the Gram lab \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\),[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

Fact D\.16\(first fundamental theorem forO⁡\(n\)\\mathrm\{O\}\(n\); Witt’s extension theorem\)\.\(First fundamental theorem\.\) Every polynomial in the rowsw1,…,wm∈ℝnw\_\{1\},\\dots,w\_\{m\}\\in\\mathbb\{R\}^\{n\}ofWWthat is invariant underW↦W​RW\\mapsto WR,R∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\), is a polynomial in the inner products⟨wi,wj⟩\\langle w\_\{i\},w\_\{j\}\\rangle, the entries ofW​W⊤WW^\{\\top\}\. For a compact group acting linearly, invariant polynomials separate orbits, soW​W⊤WW^\{\\top\}is a complete invariant\. \(Witt’s extension theorem, Euclidean case\.\) A linear isometry between two subspaces ofℝn\\mathbb\{R\}^\{n\}extends to an element ofO⁡\(n\)\\mathrm\{O\}\(n\)\. See[Weyl \(1939, Ch\. II\)](https://arxiv.org/html/2610.12728#bib.bib493)and[Goodman and Wallach \(2009, Ch\. 5\)](https://arxiv.org/html/2610.12728#bib.bib133)\.

##### Witt gives the orbit separation directly

IfW​W⊤=W′​W′⊤WW^\{\\top\}=W^\{\\prime\}\{W^\{\\prime\}\}^\{\\top\}, the map∑ici​wi↦∑ici​wi′\\sum\_\{i\}c\_\{i\}w\_\{i\}\\mapsto\\sum\_\{i\}c\_\{i\}w\_\{i\}^\{\\prime\}is well defined and isometric on the row spaces, because‖∑ici​wi‖2=c⊤​W​W⊤​c\\\|\\sum\_\{i\}c\_\{i\}w\_\{i\}\\\|^\{2\}=c^\{\\top\}WW^\{\\top\}c\. Extend it by an isometry between the orthogonal complements toR⊤∈O⁡\(n\)R^\{\\top\}\\in\\mathrm\{O\}\(n\); thenW′=W​RW^\{\\prime\}=WR\.

This is item \(1\) of[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), the reason exactly orthogonal input\-side methods are pinned toKout​\(W0\)K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\. Applied toW⊤W^\{\\top\}, it givesKinK\_\{\\mathrm\{in\}\}for the output side\.

Used in\.[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1), the known results \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

Background D\.17\(Klein’s Erlangen programme\)\.[Klein \(1872\)](https://arxiv.org/html/2610.12728#bib.bib217)proposed to define a geometry by a space together with a group acting on it, and to study the properties the group preserves\. Euclidean geometry keeps lengths and angles, the invariants of rigid motions\. Affine geometry keeps parallelism and ratios along parallel lines but not lengths or angles, because its group is larger\. A larger group leaves fewer invariants\.

For a group\-type method the reading is a theorem, because the image lies in one orbit and so the method can change nothing the group preserves\. For a cone such as LoRA’s image the word only names itscovariance group, the largest𝒢M⊆GLm×GLn\\mathcal\{G\}\_\{M\}\\subseteq\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}withIm⁡M⁡\(g​W0​h−1\)=g​Im⁡M⁡\(W0\)​h−1\\Imop M\(gW\_\{0\}h^\{\-1\}\)=g\\,\\Imop M\(W\_\{0\}\)\\,h^\{\-1\}\([Background D\.18](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\)\. For LoRA this is all ofGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, for \(IA\)3on keys and values it isMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, and for the PiSSA split it isCO⁡\(m\)×CO⁡\(n\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)\.

Used in\.Erlangen and what “Erlangen” means here \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2)\),[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), the borrowed words \(Exposé[IX](https://arxiv.org/html/2610.12728#S9)\), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\. Reading:[Sharpe \(1997, Ch\. 4\)](https://arxiv.org/html/2610.12728#bib.bib406)\.

Background D\.18\(equivariance and invariance of maps, update rules and merge rules\)\. A mapF:X→YF:X\\to Ybetween spaces withGG\-actions isequivariantifF⁡\(g⋅x\)=g⋅F⁡\(x\)F\(g\\cdot x\)=g\\cdot F\(x\), andinvariantifF⁡\(g⋅x\)=F⁡\(x\)F\(g\\cdot x\)=F\(x\)\. An update rule on parameters is equivariant under a subgroupHHof the gauge if, started fromh⋅qh\\cdot qwith correspondingly transformed optimizer state, it produceshhapplied to the iterates started fromqq\. A rule that uses randomness isequivariant in distributionifF⁡\(g⋅x\)F\(g\\cdot x\)andg⋅F⁡\(x\)g\\cdot F\(x\)have the same law for everyxx\.

On LoRA’s full\-rank locus, with channel\-uniform hyperparameters, the largest such subgroup ofGLr\\mathrm\{GL\}\_\{r\}isO⁡\(r\)\\mathrm\{O\}\(r\)for SGD,BrB\_\{r\}for Adam and all ofGLr\\mathrm\{GL\}\_\{r\}for undamped scaled GD \([Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3),[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\)\. An equivariant rule started from gauge\-related points traces the same weights, sinceρ⁡\(h⋅q\)=ρ⁡\(q\)\\rho\(h\\cdot q\)=\\rho\(q\)\. Merge rules should be invariant under each adapter’s gauge\. Task arithmetic\([Ilharco et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib187)\)∑iwi​si​Bi​Ai\\sum\_\{i\}w\_\{i\}s\_\{i\}B\_\{i\}A\_\{i\}is, while LoraHub’s\([Huang et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib180)\)\(∑iwi​Bi\)​\(∑jwj​Aj\)\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\)\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\)is not\. On a weight spaceℝN\\mathbb\{R\}^\{N\}, DARE’s\([Yu et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib538)\)random drop\-and\-rescale is equivariant under the invertible diagonal matricesTNT\_\{N\}for each fixed mask, and under the monomial matricesMonN\\mathrm\{Mon\}\_\{N\}\([Background D\.3](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\) in distribution\.

Used in\.[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6),[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6),[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9), Figure[9](https://arxiv.org/html/2610.12728#S3.F9),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\.

Background D\.19\(multihomogeneous maps; multidegree\)\. Split the parameters into blocksq=\(q1,…,qk\)q=\(q\_\{1\},\\dots,q\_\{k\}\)\. A mapδ\\deltaismultihomogeneousofmultidegree\(d1,…,dk\)\(d\_\{1\},\\dots,d\_\{k\}\)ifδ\(c1q1,…,ckqk\)=c1d1⋯ckdkδ\(q\)\\delta\(c\_\{1\}q\_\{1\},\\dots,c\_\{k\}q\_\{k\}\)=c\_\{1\}^\{d\_\{1\}\}\\cdots c\_\{k\}^\{d\_\{k\}\}\\,\\delta\(q\)for all realcic\_\{i\}\. LoRA’sB​ABAhas multidegree\(1,1\)\(1,1\), and LoHa’s\(B1​A1\)⊙\(B2​A2\)\(B\_\{1\}A\_\{1\}\)\\odot\(B\_\{2\}A\_\{2\}\)has multidegree\(1,1,1,1\)\(1,1,1,1\)\.

If somedi≥1d\_\{i\}\\geq 1, the image ofδ\\deltacontains00and is closed under multiplication by every nonnegative scalar, so it is a cone with vertex00\([Background C\.10](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\)\. For a methodρ=θb\+s​δ\\rho=\\theta\_\{b\}\+s\\,\\deltastarted whereδ⁡\(q0\)=0\\delta\(q\_\{0\}\)=0, so thatθb=θ0\\theta\_\{b\}=\\theta\_\{0\}, the image is the coneθ0\+s​Im⁡δ\\theta\_\{0\}\+s\\,\\Imop\\deltawith vertexθ0\\theta\_\{0\}\. Forc∈ℝ\>0kc\\in\\mathbb\{R\}^\{k\}\_\{\>0\}, the changeqi↦ci​qiq\_\{i\}\\mapsto c\_\{i\}q\_\{i\}together withs↦s​∏ici−dis\\mapsto s\\prod\_\{i\}c\_\{i\}^\{\-d\_\{i\}\}leavesρ\\rhounchanged\. That rescaling is an isomorphism of methods and the source of the hyperparameter torus \([Background D\.20](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\)\.

Used in\.the image germ \(Exposé[II](https://arxiv.org/html/2610.12728#S2)\),[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), the seven axes \(§[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1)\)\.

Background D\.20\(orbit space of a positive torus\)\. Letℝ\>0k\\mathbb\{R\}^\{k\}\_\{\>0\}act on positive variablesx∈ℝ\>0Nx\\in\\mathbb\{R\}^\{N\}\_\{\>0\}byxj↦xj​∏iciaj​ix\_\{j\}\\mapsto x\_\{j\}\\prod\_\{i\}c\_\{i\}^\{a\_\{ji\}\}, with a real weight matrixa=\(aj​i\)a=\(a\_\{ji\}\)\. In logarithms this is translation by the column space ofaa, so the orbits are parallel affine subspaces and the orbit space is a vector space of dimensionN−rank⁡aN\-\\rk a\. The monomials∏jxjbj\\prod\_\{j\}x\_\{j\}^\{b\_\{j\}\}withb⊤​a=0b^\{\\top\}a=0are invariant, and a basis of such vectorsbbgives a complete set of invariants\.

Under the hypotheses of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), taken with Adam atε=0\\varepsilon=0, no weight decay or clipping and a shared schedule \(otherwise the per\-blockε\\varepsilon, decay and clipping thresholds rescale as well\), the torus acts on LoRA’s hyperparameters bys↦s/\(cA​cB\)s\\mapsto s/\(c\_\{A\}c\_\{B\}\),ηA↦cA​ηA\\eta\_\{A\}\\mapsto c\_\{A\}\\eta\_\{A\},ηB↦cB​ηB\\eta\_\{B\}\\mapsto c\_\{B\}\\eta\_\{B\},σA↦cA​σA\\sigma\_\{A\}\\mapsto c\_\{A\}\\sigma\_\{A\}, withσA\\sigma\_\{A\}the scale ofAA’s initialisation, without changing the weight trajectory\. The invariants ares​ηA​ηBs\\eta\_\{A\}\\eta\_\{B\}andσA/ηA\\sigma\_\{A\}/\\eta\_\{A\}, so only two of the four knobs matter\. LoRA\+\([Hayou et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib159)\)with ratioλ\\lambdareplacesηB\\eta\_\{B\}byλ​ηB\\lambda\\eta\_\{B\}, which changes the invariants exactly as replacingα\\alphabyλ​α\\lambda\\alphadoes\. Under SGD the rates scale byci2c\_\{i\}^\{2\}, and the invariants becomes2​ηA​ηBs^\{2\}\\eta\_\{A\}\\eta\_\{B\}andσA2/ηA\\sigma\_\{A\}^\{2\}/\\eta\_\{A\}\.

Used in\.[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), Figure[8](https://arxiv.org/html/2610.12728#S3.F8), Rosetta entry 23 \(Exposé[IX](https://arxiv.org/html/2610.12728#S9)\),[Prediction X\.1](https://arxiv.org/html/2610.12728#S10.SS0.SSSx2)\.

#### Gauge, quotients and metrics

Background D\.21\(gauge symmetry of a parametrisation\)\. Thegauge groupof a methodρ:Q→Θ\\rho:Q\\to\\Thetais the group of diffeomorphismsφ\\varphiofQQwithρ∘φ=ρ\\rho\\circ\\varphi=\\rho\. If a Lie groupGGacts onQQwithρ⁡\(g⋅q\)=ρ⁡\(q\)\\rho\(g\\cdot q\)=\\rho\(q\), each orbit lies inside a fibreρ−1​\(θ\)\\rho^\{\-1\}\(\\theta\)\([Background C\.6](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)\), sod⁡\(M\)≤\|M\|−dim\(generic orbit\)d\(M\)\\leq\\lvert M\\rvert\-\\dim\(\\text\{generic orbit\}\)\. The action islocally transitiveon a fibre when its orbits are open in the fibre, that is, when orbits and fibre have the same dimension there; on generic fibres this givesd⁡\(M\)=\|M\|−dim\(generic orbit\)d\(M\)=\\lvert M\\rvert\-\\dim\(\\text\{generic orbit\}\), and we then callGGthe generic gauge\. Here the word means only this symmetry group; no connection is involved\.

LoRA’sGLr\\mathrm\{GL\}\_\{r\}acts freely and locally transitively on full\-rank fibres, sod=r⁡\(m\+n\)−r2=r⁡\(m\+n−r\)d=r\(m\+n\)\-r^\{2\}=r\(m\+n\-r\), andr2r^\{2\}parameters are gauge waste\. HRA’s generic fibres have dimensionr⁡\(r\+1\)/2r\(r\+1\)/2, while itsrrscalingsui↦ci​uiu\_\{i\}\\mapsto c\_\{i\}u\_\{i\}sweep out onlyrrof those dimensions, so the scalings do not fill the fibres;[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\) describes the rest of the gauge as non\-linear\. Forr=2r=2the fibre through\(u1,u2\)\(u\_\{1\},u\_\{2\}\)has dimension33, made up of the two scalings and of rotatingu1u\_\{1\}andu2u\_\{2\}by a common angle inside their plane, which keeps the angle between them and henceHu1​Hu2H\_\{u\_\{1\}\}H\_\{u\_\{2\}\}, the rotation of that plane by twice the angle\. The plane depends on the point, so this move is not a linear action on parameters\. TheHurwitz moves\(ui,ui\+1\)↦\(Hui​ui\+1,ui\)\(u\_\{i\},u\_\{i\+1\}\)\\mapsto\(H\_\{u\_\{i\}\}u\_\{i\+1\},u\_\{i\}\)preserve the product of reflections, sinceHa​Hb=HHa​b​HaH\_\{a\}H\_\{b\}=H\_\{H\_\{a\}b\}H\_\{a\}, and they satisfy the braid relations, so the braid group onrrstrands, unrelated to the signed permutationsBrB\_\{r\}, acts on HRA’s parameters\. These moves are discrete and add no dimension to the fibre\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4), invariants of a method \(§[I](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\), the notation \(Table[2](https://arxiv.org/html/2610.12728#S0.T2)\), rank and dimension \(§[II\.C](https://arxiv.org/html/2610.12728#S2.SS3)\),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3), the analogy “Noether; gauge” \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

Fact D\.22\(quotients and descent; the quotient manifold theorem\)\.For an action ofGGonXX, theorbit spaceX/GX/Gis the set of orbits, with the mapπ:x↦G⋅x\\pi:x\\mapsto G\\cdot x\. A mapf:X→Yf:X\\to Ythat is constant on orbits factors uniquely asf=f¯∘πf=\\bar\{f\}\\circ\\pi; this isdescent\. \(Quotient manifold theorem\.\) If a Lie group acts smoothly, freely and properly on a manifoldXX, thenX/GX/Gis a manifold of dimensiondimX−dimG\\dim X\-\\dim G,π\\piis a submersion, andf¯\\bar\{f\}is smooth wheneverffis\. Properness means that\(g,x\)↦\(g⋅x,x\)\(g,x\)\\mapsto\(g\\cdot x,x\)is a proper map, one under which preimages of compact sets are compact\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Thm 21\.10\)\.

GLr\\mathrm\{GL\}\_\{r\}acts freely and properly on pairs\(B,A\)\(B,A\)of full rank, and\(B,A\)↦B​A\(B,A\)\\mapsto BAidentifies the quotient withℳr\\mathcal\{M\}\_\{r\}, of dimensionr⁡\(m\+n\)−r2r\(m\+n\)\-r^\{2\}\. A rule for combining LoRA factors is a function of the adapters exactly when it is constant on the fibres of\(Bi,Ai\)↦\(Bi​Ai\)\(B\_\{i\},A\_\{i\}\)\\mapsto\(B\_\{i\}A\_\{i\}\), and a cometric on factors that depends only onWWdescends toℳr\\mathcal\{M\}\_\{r\}\.

Used in\.[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), when a change of chart does not matter \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\.

Background D\.23\(metrics, cometrics and the pulled\-back cometric\)\. Ametricon a vector spaceVVis a positive\-definite inner product, equivalently a symmetric positive\-definite mapg:V→V∗g:V\\to V^\{\*\}, that is,⟨g​v,v⟩\>0\\langle gv,v\\rangle\>0forv≠0v\\neq 0\. Acometricis a symmetric positive semidefinite mapK:V∗→VK:V^\{\*\}\\to V\. It turns a covector, such as a gradient∇θL\\nabla\_\{\\theta\}L, into a velocity−K​∇θ​L\-K\\nabla\_\{\\theta\}L\. A metric gives the cometricK=g−1K=g^\{\-1\}, while a cometric may be degenerate\. Metrics pull back along immersions, and cometrics push forward along any smooth map\. For instance, gradient flowq˙=−Λ​∇q\(L∘ρ\)\\dot\{q\}=\-\\Lambda\\nabla\_\{q\}\(L\\circ\\rho\)moves the weights byθ˙=−Kq​∇θL\\dot\{\\theta\}=\-K\_\{q\}\\nabla\_\{\\theta\}L, withKq=D​ρq​Λ​D​ρq⊤\.K\_\{q\}=D\\rho\_\{q\}\\,\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\.

For LoRA withblock\-scalar rates, one rate per factor,Λ=diag⁡\(ηB​I,ηA​I\)\\Lambda=\\diag\(\\eta\_\{B\}I,\\eta\_\{A\}I\)on\(B,A\)\(B,A\)andK⁡\(G\)=s2​\(ηB​G​A⊤​A\+ηA​B​B⊤​G\)K\(G\)=s^\{2\}\(\\eta\_\{B\}GA^\{\\top\}A\+\\eta\_\{A\}BB^\{\\top\}G\)\. Two points of one fibre give the same weight velocity for every loss exactly when their cometrics agree\. At points whereAAhas full row rank andr<nr<n, this holds for LoRA’s linear gauge and plain gradient descent exactly alongO⁡\(r\)\\mathrm\{O\}\(r\)\. A rule descends to a cometric on weights when it gives the same weight velocity at all points of each fibre; with a point\-dependent preconditioner in place ofΛ\\Lambda, undamped scaled GD\([Tong et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib452);[Zhang and Pilanci, 2024a](https://arxiv.org/html/2610.12728#bib.bib546)\)and the SGD variant of LoRA\-Pro\([Wang et al\., 2024i](https://arxiv.org/html/2610.12728#bib.bib486)\)do, on the full\-rank locus\. Scaled GD multiplies theBB\-gradient by\(A​A⊤\)−1\(AA^\{\\top\}\)^\{\-1\}and theAA\-gradient by\(B⊤​B\)−1\(B^\{\\top\}B\)^\{\-1\}, and undamped means that no ridge termδ​I\\delta Iis added before inverting \([Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\.

Used in\.[Definition III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),[Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3), the metric a method carries \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)\), the three invariants \(Exposé[0](https://arxiv.org/html/2610.12728#S0)\)\.

Background D\.24\(Riemannian gradient flow; tangent projection; retraction\)\. On a submanifoldN⊆ΘN\\subseteq\\Thetawith the metric induced by the Frobenius inner product, the Riemannian gradient ofL\|NL\|\_\{N\}atθ\\thetais the orthogonal projectionΠTθ​N∇L\(θ\)\\Pi\_\{T\_\{\\theta\}N\}\\nabla L\(\\theta\)of the ambient gradient onto the tangent space, and theRiemannian gradient flowisθ˙=−ΠTθ​N∇L\(θ\)\\dot\{\\theta\}=\-\\Pi\_\{T\_\{\\theta\}N\}\\nabla L\(\\theta\)\. A discrete step along the tangent space leavesNN, so it is followed by aretraction, a smooth mapRθR\_\{\\theta\}from a neighbourhood of00inTθ​NT\_\{\\theta\}NtoNNwithRθ​\(0\)=θR\_\{\\theta\}\(0\)=\\thetaandD​Rθ​\(0\)=idDR\_\{\\theta\}\(0\)=\\mathrm\{id\}\.

AtW=W0\+XW=W\_\{0\}\+XwithX∈ℳrX\\in\\mathcal\{M\}\_\{r\},U=col⁡XU=\\operatorname\{col\}XandV=row⁡XV=\\operatorname\{row\}X, the tangent space is\{D:PU⟂​D​PV⟂=0\}\\\{D:P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}=0\\\}\([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\) and the projection isPU​G\+G​PV−PU​G​PVP\_\{U\}G\+GP\_\{V\}\-P\_\{U\}GP\_\{V\}\. This is LoRA\-Pro’s equivalent gradient, so the SGD variant of LoRA\-Pro follows the Riemannian gradient flow onW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\. Truncating the SVD ofX\+ξX\+\\xito rankrris a retraction ontoℳr\\mathcal\{M\}\_\{r\}, the one Riemannion\([Bogachev et al\., 2025](https://arxiv.org/html/2610.12728#bib.bib41)\)uses\.

Used in\.[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3), when a change of chart does not matter \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\),[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\. Reading:[Absil et al\. \(2008, Ch\. 3 and 4\)](https://arxiv.org/html/2610.12728#bib.bib2)\.

Fact D\.25\(distributions, Lie brackets and Frobenius’ theorem\)\.AdistributionDDof rankkkon an open setUUassigns to each pointθ\\thetaakk\-dimensional subspaceDθD\_\{\\theta\}of directions, smoothly inθ\\theta; the word is unrelated to probability distributions\. TheLie bracketof vector fields is\[X,Y\]=D​Y⋅X−D​X⋅Y\[X,Y\]=DY\\cdot X\-DX\\cdot Y; flowing alongXX,YY,−X\-X,−Y\-Yfor timettmisses the starting point byt2​\[X,Y\]\+O⁡\(t3\)t^\{2\}\[X,Y\]\+O\(t^\{3\}\)\.DDisinvolutiveif\[X,Y\]\[X,Y\]lies inDDwheneverXXandYYdo\. \(Frobenius\.\)DDis involutive iff every point lies on akk\-dimensionalintegral manifold, a submanifold whose tangent space at each of its pointsθ\\thetaisDθD\_\{\\theta\}\. A maximal connected integral manifold is aleaf\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Ch\. 19\)\. The Frobenius algebras of[Background B\.6](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)and the Frobenius norm share only the name\.

Onℝ3\\mathbb\{R\}^\{3\},D=span\{∂x,∂y\+x∂z\}D=\\spanop\\\{\\partial\_\{x\},\\ \\partial\_\{y\}\+x\\,\\partial\_\{z\}\\\}is not involutive, since\[∂x,∂y\+x∂z\]=∂z∉D\[\\partial\_\{x\},\\partial\_\{y\}\+x\\,\\partial\_\{z\}\]=\\partial\_\{z\}\\notin D, so no surface is tangent toDD\. A backward method with anchora⁡\(θ\)a\(\\theta\)choosesDθ=im⁡a⁡\(θ\)D\_\{\\theta\}=\\operatorname\{im\}a\(\\theta\), and it reads as a forward method only along integral manifolds\. GaLore\([Zhao et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib581)\)with a projector fixed on each period, LISA\([Pan et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib360)\)and MeZO\([Malladi et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib310)\)have constant distributions, which are involutive with affine leaves\. Full\-batch GaLore withT=1T=1, read asDW=Hom⁡\(ℝn,Ur​\(∇L​\(W\)\)\)D\_\{W\}=\\operatorname\{Hom\}\(\\mathbb\{R\}^\{n\},U\_\{r\}\(\\nabla L\(W\)\)\)wherever a singular\-value gap exists, already fails to be involutive forL=12​‖W‖F2L=\\tfrac\{1\}\{2\}\\\|W\\\|\_\{F\}^\{2\}\.

Used in\.when a backward method is a forward one \(§[IV](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\),[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\.

#### Conservation and balance

Fact D\.26\(Noether’s theorem for gradient flows\)\.Let a Lie group act linearly onQ=ℝNQ=\\mathbb\{R\}^\{N\}withρ⁡\(k⋅q\)=ρ⁡\(q\)\\rho\(k\\cdot q\)=\\rho\(q\), letΛ≻0\\Lambda\\succ 0be a constant rate matrix, and letγ⁡\(t\)\\gamma\(t\)be any time\-dependent covector onΘ\\Theta, a gradient or not\. Along every solution ofq˙=−Λ​D​ρq⊤​γ​\(t\)\\dot\{q\}=\-\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\\gamma\(t\), and for every generatorξ\\xiof the action withΛ−1​ξ\\Lambda^\{\-1\}\\xisymmetric, thechargeJξ​\(q\)=12​⟨q,Λ−1​ξ​q⟩J\_\{\\xi\}\(q\)=\\tfrac\{1\}\{2\}\\langle q,\\Lambda^\{\-1\}\\xi\\,q\\rangleis constant\([Kunin et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib230);[Zhao et al\., 2022b](https://arxiv.org/html/2610.12728#bib.bib580);[Marcotte et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib318)\)\.

##### The one\-line proof

Invariance ofρ\\rhogivesD​ρq​\(ξ​q\)=0D\\rho\_\{q\}\(\\xi q\)=0\. Hencedd​t​Jξ=⟨Λ−1​ξ​q,q˙⟩=−⟨ξ​q,D​ρq⊤​γ⟩=−⟨D​ρq​ξ​q,γ⟩=0\\tfrac\{d\}\{dt\}J\_\{\\xi\}=\\langle\\Lambda^\{\-1\}\\xi q,\\dot\{q\}\\rangle=\-\\langle\\xi q,D\\rho\_\{q\}^\{\\top\}\\gamma\\rangle=\-\\langle D\\rho\_\{q\}\\,\\xi q,\\gamma\\rangle=0\.

For scalar LoRAw=w0\+b​aw=w\_\{0\}\+bawith ratesηB,ηA\\eta\_\{B\},\\eta\_\{A\}, the chargeb2/ηB−a2/ηAb^\{2\}/\\eta\_\{B\}\-a^\{2\}/\\eta\_\{A\}is conserved, so every trajectory runs along a hyperbola, the Noether hyperbolas of Figure[7](https://arxiv.org/html/2610.12728#S3.F7)\(a\)\.[Noether \(1918\)](https://arxiv.org/html/2610.12728#bib.bib351)derived conservation laws from symmetries of a Lagrangian; this version needs no Lagrangian and holds for any loss and data\. For LoRA with ratesηB,ηA\\eta\_\{B\},\\eta\_\{A\}, the generatorsξX​\(B,A\)=\(B​X,−X​A\)\\xi\_\{X\}\(B,A\)=\(BX,\-XA\)pass the test exactly for symmetricXX, and ther⁡\(r\+1\)/2r\(r\+1\)/2charges are the entries ofΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}\. Minibatches, LoRA dropout, global\-norm clipping and DoRA’s detached norm keep the required form\. Momentum and Adam do not, and under discrete SGD the charge drifts, by an amount relative to the charge of orderη2\\eta^\{2\}per step\.

Used in\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), isometries and charges and the analogy “Noether; gauge” \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\), Figure[7](https://arxiv.org/html/2610.12728#S3.F7), the things one can check \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\),[Prediction X\.8](https://arxiv.org/html/2610.12728#S10.SS0.SSSx2.Px2), the seven borrowed words \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\)\.

Background D\.27\(Cartan decomposition; polar decomposition\)\. The mapϑ⁡\(ξ\)=−ξ⊤\\vartheta\(\\xi\)=\-\\xi^\{\\top\}is an automorphism of𝔤​𝔩r\\mathfrak\{gl\}\_\{r\}of order two, aCartan involution\. Its\+1\+1and−1\-1eigenspaces give theCartan decomposition𝔤​𝔩r=𝔰​𝔬​\(r\)⊕Symr\\mathfrak\{gl\}\_\{r\}=\\mathfrak\{so\}\(r\)\\oplus\\mathrm\{Sym\}\_\{r\},ξ=12​\(ξ−ξ⊤\)\+12​\(ξ\+ξ⊤\)\\xi=\\tfrac\{1\}\{2\}\(\\xi\-\\xi^\{\\top\}\)\+\\tfrac\{1\}\{2\}\(\\xi\+\\xi^\{\\top\}\), with\[𝔰​𝔬,𝔰​𝔬\]⊆𝔰​𝔬\[\\mathfrak\{so\},\\mathfrak\{so\}\]\\subseteq\\mathfrak\{so\},\[𝔰​𝔬,Sym\]⊆Sym\[\\mathfrak\{so\},\\mathrm\{Sym\}\]\\subseteq\\mathrm\{Sym\}and\[Sym,Sym\]⊆𝔰​𝔬\[\\mathrm\{Sym\},\\mathrm\{Sym\}\]\\subseteq\\mathfrak\{so\}\. Its group version is thepolar decomposition, which says that\(k,S\)↦k​eS\(k,S\)\\mapsto k\\,e^\{S\}is a diffeomorphismO⁡\(r\)×Symr→GLr\\mathrm\{O\}\(r\)\\times\\mathrm\{Sym\}\_\{r\}\\to\\mathrm\{GL\}\_\{r\}\. Henceg↦g⊤​gg\\mapsto g^\{\\top\}gidentifies the cosetsO⁡\(r\)​g\\mathrm\{O\}\(r\)gwith the positive\-definite matrices,O⁡\(r\)\\GLr≅Symr\+\\mathrm\{O\}\(r\)\\backslash\\mathrm\{GL\}\_\{r\}\\cong\\mathrm\{Sym\}^\{\+\}\_\{r\}\.

With block\-scalar rates \([Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\), the antisymmetric generatorsX∈𝔰​𝔬​\(r\)X\\in\\mathfrak\{so\}\(r\)of LoRA’s gauge act by rotations\(B​O⊤,O​A\)\(BO^\{\\top\},OA\)that preserve the rate metric, so the optimizer cannot feel them; they areisometries\. The symmetric generatorsX∈SymrX\\in\\mathrm\{Sym\}\_\{r\}give the conserved charges of[Background D\.26](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4)\. For general rates the split needs𝔤\\mathfrak\{g\}to be closed underξ↦Λ​ξ⊤​Λ−1\\xi\\mapsto\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}\. Gradient descent on LoRA is thus a family of dynamics indexed byO⁡\(r\)\\GLr\\mathrm\{O\}\(r\)\\backslash\\mathrm\{GL\}\_\{r\}, and the Kempf–Ness argument needs only curveset​ξe^\{t\\xi\}withξ\\xisymmetric\.

Used in\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), isometries and charges \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\), the three invariants and the thesis \(Exposé[0](https://arxiv.org/html/2610.12728#S0)\),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), when a change of chart does not matter \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\)\. Reading:[Knapp \(2002, §VI\.2\)](https://arxiv.org/html/2610.12728#bib.bib218);[Horn and Johnson \(2013, §7\.3\)](https://arxiv.org/html/2610.12728#bib.bib174)\.

Background D\.28\(balancedness; the conservedB⊤​B−A​A⊤B^\{\\top\}B\-AA^\{\\top\}\)\. A factorisationW=B​AW=BAisbalancedifB⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\}, and rate\-weighted balanced ifΦ=B⊤​B/ηB−A​A⊤/ηA=0\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}=0, that is,B⊤​B=c​A​A⊤B^\{\\top\}B=cAA^\{\\top\}withc=ηB/ηAc=\\eta\_\{B\}/\\eta\_\{A\}\. For a deep linear networkW=WL⋯W1W=W\_\{L\}\\cdots W\_\{1\}the same word meansWj\+1⊤​Wj\+1=Wj​Wj⊤W\_\{j\+1\}^\{\\top\}W\_\{j\+1\}=W\_\{j\}W\_\{j\}^\{\\top\}for everyjj\.

Under gradient flow with block\-scalar rates,Φ\\Phiis conserved \([Background D\.26](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4)\), so rate\-weighted balancedness, once it holds, holds forever; at equal rates the two notions agree\. Zero\-product startsB0​A0=0B\_\{0\}A\_\{0\}=0have⟨B0⊤​B0,A0​A0⊤⟩F=0\\langle B\_\{0\}^\{\\top\}B\_\{0\},A\_\{0\}A\_\{0\}^\{\\top\}\\rangle\_\{F\}=0, which makes them maximally unbalanced for given norms of the two Gram matrices, since the inner product of two positive semidefinite matrices is never negative\. FromB0=0B\_\{0\}=0, conservation givesB⊤​B=ηBηA​\(A​A⊤−A0​A0⊤\)B^\{\\top\}B=\\tfrac\{\\eta\_\{B\}\}\{\\eta\_\{A\}\}\(AA^\{\\top\}\-A\_\{0\}A\_\{0\}^\{\\top\}\), soA​A⊤AA^\{\\top\}never drops belowA0​A0⊤A\_\{0\}A\_\{0\}^\{\\top\}in the Loewner order\. HF’s PiSSA starts withB0⊤​B0=A0​A0⊤B\_\{0\}^\{\\top\}B\_\{0\}=A\_\{0\}A\_\{0\}^\{\\top\}\. WhenΦ\\Phiis isotropic, singular values ofΔ​W\\Delta Wwell above the crossover scaleσ∗\\sigma^\{\*\}of[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), set by the charge, the scale and the rates, follow the balanced dynamics of[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14)\.

Used in\.isometries and charges \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Proposition III\.8](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), the bibliography\. Reading:[Arora et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib14);[Du et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib100)\.

Background D\.29\(the moment map\)\. Let a groupG⊆GL⁡\(V\)G\\subseteq\\mathrm\{GL\}\(V\)that is closed under transpose act on a real inner\-product spaceVV, and split𝔤=𝔨⊕𝔭\\mathfrak\{g\}=\\mathfrak\{k\}\\oplus\\mathfrak\{p\}into its antisymmetric and symmetric elements\. The \(real\)moment mapμ:V→𝔭\\mu:V\\to\\mathfrak\{p\}is defined by⟨μ⁡\(v\),ξ⟩=12​dd​t\|t=0​‖et​ξ​v‖2=⟨ξ​v,v⟩\(ξ∈𝔭\)\.\\langle\\mu\(v\),\\xi\\rangle=\\tfrac\{1\}\{2\}\\tfrac\{d\}\{dt\}\\Big\|\_\{t=0\}\\\|e^\{t\\xi\}v\\\|^\{2\}=\\langle\\xi v,v\\rangle\\qquad\(\\xi\\in\\mathfrak\{p\}\)\.It records how the norm changes along the symmetric directions of the orbit; the antisymmetric directions preserve the norm\.

For LoRA’s gauge,dd​t\|0​\(‖B​e−t​X‖2\+‖et​X​A‖2\)=−2​tr⁡\(X​μ\)\\tfrac\{d\}\{dt\}\\big\|\_\{0\}\\big\(\\\|Be^\{\-tX\}\\\|^\{2\}\+\\\|e^\{tX\}A\\\|^\{2\}\\big\)=\-2\\tr\(X\\mu\)for symmetricXX, withμ=B⊤​B−A​A⊤\\mu=B^\{\\top\}B\-AA^\{\\top\}, so thisμ\\muis the moment map up to sign\. Its zero set is the balanced slice, and at equal ratesηA=ηB=η\\eta\_\{A\}=\\eta\_\{B\}=\\etait equalsη​Φ\\eta\\Phi, a multiple of the Noether charge\.

Used in\.the balanced slice \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1), Figure[7](https://arxiv.org/html/2610.12728#S3.F7)\.

Fact D\.30\(Kempf–Ness theorem, real form\)\.LetG⊆GL⁡\(V\)G\\subseteq\\mathrm\{GL\}\(V\)be a real algebraic group, cut out by polynomial equations, that is closed under transpose \(such a group isreductive\), for instanceGLr\\mathrm\{GL\}\_\{r\}acting on LoRA factors, and letK=G∩O⁡\(V\)K=G\\cap\\mathrm\{O\}\(V\)\. Forv∈Vv\\in V, \(i\)g=eg=eis a critical point ofg↦‖g⋅v‖2g\\mapsto\\\|g\\cdot v\\\|^\{2\}iffμ⁡\(v\)=0\\mu\(v\)=0, and thenvvhas least norm onG⋅vG\\cdot v; \(ii\)G⋅vG\\cdot vis closed iff it contains a point withμ=0\\mu=0; \(iii\) on a closed orbit, the points withμ=0\\mu=0form a singleKK\-orbit; \(iv\) the closure of every orbit contains exactly one closed orbit\([Kempf and Ness, 1979](https://arxiv.org/html/2610.12728#bib.bib209);[Birkes, 1971](https://arxiv.org/html/2610.12728#bib.bib38);[Richardson and Slodowy, 1990](https://arxiv.org/html/2610.12728#bib.bib389)\)\.

##### The idea of the proof

For symmetricξ\\xi,‖et​ξ​v‖2=∑ie2​t​λi​‖vi‖2\\\|e^\{t\\xi\}v\\\|^\{2\}=\\sum\_\{i\}e^\{2t\\lambda\_\{i\}\}\\\|v\_\{i\}\\\|^\{2\}in an eigenbasis ofξ\\xi, which is convex intt\. By the polar decomposition every orbit point isk​eξ​vk\\,e^\{\\xi\}vwithk∈Kk\\in Knorm\-preserving, so critical points are minima and two minima differ by an element ofKK\.

Form=n=r=1m=n=r=1,t∈ℝ×t\\in\\mathbb\{R\}^\{\\times\}acts byt⋅\(b,a\)=\(b/t,t​a\)t\\cdot\(b,a\)=\(b/t,ta\)andμ=b2−a2\\mu=b^\{2\}\-a^\{2\}\. The orbits are the hyperbolasb​a=c≠0ba=c\\neq 0, which are closed and meetμ=0\\mu=0in the two points±\(b,a\)\\pm\(b,a\)with\|b\|=\|a\|\\lvert b\\rvert=\\lvert a\\rvert, oneO⁡\(1\)\\mathrm\{O\}\(1\)\-orbit; the two punctured axes, which are not closed and avoidμ=0\\mu=0; and the origin\. For LoRA in general,K=O⁡\(r\)K=\\mathrm\{O\}\(r\), and a full\-rank orbit is closed, being a whole fibre of\(B,A\)↦B​A\(B,A\)\\mapsto BA\. The least\-norm factorisations of a fixedB​ABAare the balanced ones, they form oneO⁡\(r\)\\mathrm\{O\}\(r\)\-orbit, and there‖B‖F2\+‖A‖F2=2​‖B​A‖∗\\\|B\\\|\_\{F\}^\{2\}\+\\\|A\\\|\_\{F\}^\{2\}=2\\\|BA\\\|\_\{\*\}\. Weight decay adds a norm to the loss \([Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\)\. With learning\-rate\-coupled decay, which multiplies the decay of each factor by its rate,Φ˙=−2​λ​μ\\dot\{\\Phi\}=\-2\\lambda\\mu, so the equilibria lie onμ=0\\mu=0, andμ\\mudecays exponentially at equal rates\. With per\-time decay, the sameλ\\lambdafor both factors,Φ˙=−2​λ​Φ\\dot\{\\Phi\}=\-2\\lambda\\Phi, soΦ→0\\Phi\\to 0\.

Used in\.the balanced slice \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5),[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1), combining \(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\), the known results \(Exposé[XI](https://arxiv.org/html/2610.12728#S11)\), the classification \(Exposé[VIII](https://arxiv.org/html/2610.12728#S8)\)\.

Fact D\.31\(the closed orbit in each fibre of\(B,A\)↦B​A\(B,A\)\\mapsto BA\)\.ForGLr\\mathrm\{GL\}\_\{r\}acting on factor pairs, fixWWof rankk≤rk\\leq r\. Among the pairs withB​A=WBA=W, those withrank⁡B=rank⁡A=k\\rk B=\\rk A=kform a single orbit\. It is closed, and it lies in the closure of every orbit in the fibre\{BA=W\}\\\{BA=W\\\}\. Consequently a continuousGLr\\mathrm\{GL\}\_\{r\}\-invariant function is constant on each fibre of\(B,A\)↦B​A\(B,A\)\\mapsto BA\. For a complex reductive group acting on a vector space, every fibre of the quotient map likewise contains exactly one closed orbit\([Mumford et al\., 1994](https://arxiv.org/html/2610.12728#bib.bib340), Ch\. 1\)\.

Form=n=r=1m=n=r=1\([Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\), the fibre\{ba=0\}\\\{ba=0\\\}consists of the origin and the two punctured axes, and the origin, its only closed orbit, lies in the closure of both axes\. This is the step behind[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(a\)\. A continuous rule for combining LoRA factors that is invariant under each adapter’sGLr\\mathrm\{GL\}\_\{r\}is automatically a function of the adaptersBi​AiB\_\{i\}A\_\{i\}, also off the full\-rank locus\. Continuity matters\. The projector ontorow⁡A\\operatorname\{row\}A, defined whererank⁡A=r\\rk A=r, is invariant, yet it differs at\(0,A\)\(0,A\)and\(0,A′\)\(0,A^\{\\prime\}\), which have the same product00\.

Used in\.[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4), combining \(§[V](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\)\.

#### Matrix and convexity tools

Fact D\.32\(Ky Fan’s maximum principle\)\.LetS∈ℝn×nS\\in\\mathbb\{R\}^\{n\\times n\}be symmetric with eigenvaluesλ1≥⋯≥λn\\lambda\_\{1\}\\geq\\dots\\geq\\lambda\_\{n\}, and let1≤k≤n1\\leq k\\leq n\. Thenmax\{tr\(PSP⊤\):P∈ℝk×n,PP⊤=Ik\}=λ1\+⋯\+λk,\\max\\\{\\tr\(PSP^\{\\top\}\):P\\in\\mathbb\{R\}^\{k\\times n\},\\ PP^\{\\top\}=I\_\{k\}\\\}=\\lambda\_\{1\}\+\\dots\+\\lambda\_\{k\},attained when the rows ofPPspan a top\-kkeigenspace\. For any matrixXXwith singular valuesσ1≥σ2≥⋯\\sigma\_\{1\}\\geq\\sigma\_\{2\}\\geq\\cdots, applying this toS=X⊤​XS=X^\{\\top\}XgivesmaxP⁡‖X​P⊤‖F2=σ12\+⋯\+σk2\\max\_\{P\}\\\|XP^\{\\top\}\\\|\_\{F\}^\{2\}=\\sigma\_\{1\}^\{2\}\+\\dots\+\\sigma\_\{k\}^\{2\}, and for squareXXalso‖P​X​P⊤‖F2≤‖X​P⊤‖F2≤σ12\+⋯\+σk2\\\|PXP^\{\\top\}\\\|\_\{F\}^\{2\}\\leq\\\|XP^\{\\top\}\\\|\_\{F\}^\{2\}\\leq\\sigma\_\{1\}^\{2\}\+\\dots\+\\sigma\_\{k\}^\{2\}\. See[Fan \(1949\)](https://arxiv.org/html/2610.12728#bib.bib108)and[Bhatia \(1997, Ch\. III\)](https://arxiv.org/html/2610.12728#bib.bib31)\.

AmongA0A\_\{0\}withA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cI, LoRA’s initial rate‖G​A0⊤‖F2\\\|GA\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\}under gradient flow is largest whenrow⁡A0\\operatorname\{row\}A\_\{0\}is the top\-rrright singular space ofGG, which is LoRA\-GA’s\([Wang et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib472)\)choice forA0A\_\{0\}\. For LOFT, the squared norm of the pulled\-back gradientPr​skew⁡\(W0⊤​G\)​Pr⊤P\_\{r\}\\skewop\(W\_\{0\}^\{\\top\}G\)P\_\{r\}^\{\\top\}is at most2​∑k≤⌊r/2⌋μk22\\sum\_\{k\\leq\\lfloor r/2\\rfloor\}\\mu\_\{k\}^\{2\}, where±i​μk\\pm i\\mu\_\{k\}are the eigenvalues of the skew matrixskew⁡\(W0⊤​G\)\\skewop\(W\_\{0\}^\{\\top\}G\), whose singular values areμ1,μ1,μ2,μ2,…\\mu\_\{1\},\\mu\_\{1\},\\mu\_\{2\},\\mu\_\{2\},\\dots\. The floor appears because a skew matrix compressed to an odd\-dimensional subspace has a zero eigenvalue, and Cauchy interlacing bounds the rest\.

Used in\.LOFT \(§[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\),[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)\.

Fact D\.33\(Sylvester equations\)\.ForS∈ℝp×pS\\in\\mathbb\{R\}^\{p\\times p\},T∈ℝq×qT\\in\\mathbb\{R\}^\{q\\times q\}andC∈ℝp×qC\\in\\mathbb\{R\}^\{p\\times q\}, theSylvester equationS​X\+X​T=CSX\+XT=Chas a unique solutionXXfor everyCCiffSSand−T\-Thave no eigenvalue in common, since the eigenvalues ofX↦S​X\+X​TX\\mapsto SX\+XTare the sumsλi​\(S\)\+λj​\(T\)\\lambda\_\{i\}\(S\)\+\\lambda\_\{j\}\(T\)\. IfSSandTTare symmetric positive definite, this holds andX=∫0∞e−t​S​C​e−t​T​𝑑tX=\\int\_\{0\}^\{\\infty\}e^\{\-tS\}Ce^\{\-tT\}\\,dt\. The Bartels–Stewart algorithm\([Bartels and Stewart, 1972](https://arxiv.org/html/2610.12728#bib.bib25)\)solves the equation through Schur forms ofSSandTT\([Horn and Johnson, 1991](https://arxiv.org/html/2610.12728#bib.bib173), Ch\. 4\)\.

LoRA\-Pro’s equivalent gradient leaves a freer×rr\\times rmatrixXX\. The published method picks it by solvingB⊤​B​X\+X​A​A⊤=CB^\{\\top\}B\\,X\+X\\,AA^\{\\top\}=C, withCCbuilt from the gradient\. On the full\-rank locus both Gram matrices are positive definite, soXXexists and is unique\. The first\-order weight velocity does not depend onXX, while the resulting parameter update is onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant\.

Used in\.[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8), when a change of chart does not matter \(§[III](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\), Figure[9](https://arxiv.org/html/2610.12728#S3.F9)\.

Background D\.34\(the perspective map; convex combinations\)\. Theperspective mapisπ⁡\(Z,S\)=S/Z\\pi\(Z,S\)=S/Zon\{\(Z,S\)∈ℝ×ℝd:Z\>0\}\\\{\(Z,S\)\\in\\mathbb\{R\}\\times\\mathbb\{R\}^\{d\}:Z\>0\\\}\. It is constant on rays,π⁡\(t​Z,t​S\)=π⁡\(Z,S\)\\pi\(tZ,tS\)=\\pi\(Z,S\)fort\>0t\>0, and it turns sums into convex combinations,π⁡\(Z1\+Z2,S1\+S2\)=λ​π​\(Z1,S1\)\+\(1−λ\)​π​\(Z2,S2\),λ=Z1Z1\+Z2\.\\pi\(Z\_\{1\}\+Z\_\{2\},S\_\{1\}\+S\_\{2\}\)=\\lambda\\,\\pi\(Z\_\{1\},S\_\{1\}\)\+\(1\-\\lambda\)\\,\\pi\(Z\_\{2\},S\_\{2\}\),\\qquad\\lambda=\\frac\{Z\_\{1\}\}\{Z\_\{1\}\+Z\_\{2\}\}\.

For a fixed queryqq, attention isAttnq​\(K\)=π⁡\(Σq​\(K\)\)\\mathrm\{Attn\}\_\{q\}\(K\)=\\pi\(\\Sigma\_\{q\}\(K\)\), where

Σq​\(K\)=\(∑e⟨q,k⟩/d,∑e⟨q,k⟩/d​v\)\\Sigma\_\{q\}\(K\)=\\Big\(\\sum e^\{\\langle q,k\\rangle/\\sqrt\{d\}\},\\ \\sum e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\\,v\\Big\)is a monoid homomorphism on multisets of key–value pairs \([Background D\.10](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\)\. With a prefixPPbefore the contentCC, the identity above gives

Attnq​\(P⊎C\)=\(1−λq\)​Attnq​\(C\)\+λq​Attnq​\(P\),\\mathrm\{Attn\}\_\{q\}\(P\\uplus C\)=\(1\-\\lambda\_\{q\}\)\\,\\mathrm\{Attn\}\_\{q\}\(C\)\+\\lambda\_\{q\}\\,\\mathrm\{Attn\}\_\{q\}\(P\),withAttnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(P\)in the convex hullconv⁡VP\\operatorname\{conv\}V\_\{P\}of the prefix values, so a prefix acts as a parallel branch gated by its share of the mass\.

Used in\.[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6), attention \(§[VI](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\), Figure[12](https://arxiv.org/html/2610.12728#S6.F12)\. Reading:[Boyd and Vandenberghe \(2004, §2\.3\.3 and §3\.2\.6\)](https://arxiv.org/html/2610.12728#bib.bib43)\.

## Appendix BAuxiliary results

This appendix quotes the results from outside the paper that the proofs in Appendix[C](https://arxiv.org/html/2610.12728#A3)use\. Each is stated in the form in which it is applied, with a precise source, and is followed by the places that use it\. Several are textbook facts that a specialist applies without comment\. They are listed so that every step of Appendix[C](https://arxiv.org/html/2610.12728#A3)can be traced to a statement with hypotheses\. For a few elementary facts we give the one\-line reason instead of a reference\. Background definitions \(manifolds, germs, group actions, lenses\) are in Appendix[A](https://arxiv.org/html/2610.12728#A1), and where an entry has a companion there we name it\.

Three results of the main text are external results restated in the paper’s conventions and proved there:[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\(the min\-cut bound, here[Auxiliary result B\.14](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\),[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(Noether charges,[Auxiliary result B\.29](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\) and[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(a\) \(the Kempf–Ness slice,[Auxiliary result B\.26](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\)\. The entries below give their source forms\.

The entries are grouped by subject: smooth maps and limits \(B\.1–B\.4\), real algebraic and tame geometry \(B\.5–B\.10\), matrix analysis \(B\.11–B\.21\), Lie groups and invariant theory \(B\.22–B\.27\), dynamics, optimisation and differentiation \(B\.28–B\.35\), three elementary facts \(B\.36\), and spectral projectors, Riemannian submanifolds and floating\-point scaling \(B\.37–B\.39\)\. Notation follows the exposés:ℳ≤r\\mathcal\{M\}\_\{\\leq r\}is the set ofm×nm\\times nmatrices of rank at mostrr,ℳr\\mathcal\{M\}\_\{r\}those of rank exactlyrr,PUP\_\{U\}the orthogonal projector onto a subspaceUU, and∥⋅∥\\lVert\\cdot\\rVertthe Frobenius norm unless marked otherwise\. Entries of Appendix[A](https://arxiv.org/html/2610.12728#A1)are cited as “Background A\.1” to “Background D\.34”; the numbers of its Part B are unrelated to the numbers B\.1–B\.39 used here\.

#### Smooth maps, fibre products and limits

Auxiliary result B\.1\(rank, the rank theorem and submersions\)\.\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Prop\. 4\.1, Thm 4\.12, Prop\. 4\.28, Thm 5\.12\)LetF:Q→ΘF:Q\\to\\Thetabe a smooth map between manifolds of dimensionsNNandDD\.\(a\)For everykk, the set\{q:rank⁡d​Fq≥k\}\\\{q:\\rk dF\_\{q\}\\geq k\\\}is open\. In particular the set whererank⁡d​Fq\\rk dF\_\{q\}takes its maximal value is open and nonempty, and ifrank⁡d​Fp=D\\rk dF\_\{p\}=DthenFFis a submersion on a neighbourhood ofpp\.\(b\)\(Rank theorem\.\) Ifrank⁡d​Fq=k\\rk dF\_\{q\}=kfor allqqin a neighbourhood ofpp, there are charts centred atppand atF⁡\(p\)F\(p\)in whichFFreads\(x1,…,xN\)↦\(x1,…,xk,0,…,0\)\(x\_\{1\},\\dots,x\_\{N\}\)\\mapsto\(x\_\{1\},\\dots,x\_\{k\},0,\\dots,0\)\. Hencepphas a neighbourhoodVVsuch thatF⁡\(V\)F\(V\)is an embedded submanifold of dimensionkkandF−1​\(F​\(p\)\)∩VF^\{\-1\}\(F\(p\)\)\\cap Vis an embedded submanifold of dimensionN−kN\-k\.\(c\)A submersion is an open map\. In particular, ifrank⁡d​F=D\\rk dF=Don a neighbourhoodUUofpp, thenF⁡\(U\)F\(U\)contains an open neighbourhood ofF⁡\(p\)F\(p\)\.

*Reason for \(a\)\.*In coordinates,rank⁡d​Fq≥k\\rk dF\_\{q\}\\geq ksays that somek×kk\\times kminor of the Jacobian is nonzero atqq, an open condition; Lee’s Prop\. 4\.1 is the casek=Dk=D\. Companion:[Background C\.5](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\), where the lifts of both axes forcerank⁡d​ιp∗=2\\rk d\\iota\_\{p^\{\*\}\}=2, the rank stays22nearp∗p^\{\*\}, and the image of a neighbourhood ofp∗p^\{\*\}contains an open subset ofℝ2\\mathbb\{R\}^\{2\};[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\(with[Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\);[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(a\) \(fibre dimension\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\)near points of maximal rank\);[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(d\) \(rankrris an open condition on the slice\)\. The proofs of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)and[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cite it\.

Auxiliary result B\.2\(transversality and fibre products\)\.\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Cor\. 5\.30 and Thm 6\.30\);\([Guillemin and Pollack, 1974](https://arxiv.org/html/2610.12728#bib.bib142), §1\.5\)LetρM:QM→Θ\\rho\_\{M\}:Q\_\{M\}\\to\\ThetaandρN:QN→Θ\\rho\_\{N\}:Q\_\{N\}\\to\\Thetabe smooth, withdimQM=\|M\|\\dim Q\_\{M\}=\\lvert M\\rvertanddimQN=\|N\|\\dim Q\_\{N\}=\\lvert N\\rvert, and letP=\{\(q,q′\):ρM​\(q\)=ρN​\(q′\)\}⊆QM×QNP=\\\{\(q,q^\{\\prime\}\):\\rho\_\{M\}\(q\)=\\rho\_\{N\}\(q^\{\\prime\}\)\\\}\\subseteq Q\_\{M\}\\times Q\_\{N\}\.\(a\)Ifd​ρM​\(Tq​QM\)\+d​ρN​\(Tq′​QN\)=Tθ​Θd\\rho\_\{M\}\(T\_\{q\}Q\_\{M\}\)\+d\\rho\_\{N\}\(T\_\{q^\{\\prime\}\}Q\_\{N\}\)=T\_\{\\theta\}\\Thetaat every\(q,q′\)∈P\(q,q^\{\\prime\}\)\\in P, thenPPis an embedded submanifold ofQM×QNQ\_\{M\}\\times Q\_\{N\}of dimension\|M\|\+\|N\|−dimΘ\\lvert M\\rvert\+\\lvert N\\rvert\-\\dim\\Theta\.\(b\)Transversality at a single point ofPPforces\|M\|\+\|N\|≥dimΘ\\lvert M\\rvert\+\\lvert N\\rvert\\geq\\dim\\Theta\.\(c\)IfS⊆XS\\subseteq Xis an embedded submanifold, every smooth mapT→XT\\to Xwith image inSSis smooth as a map intoSS\. Consequently, wheneverPPis an embedded submanifold ofQM×QNQ\_\{M\}\\times Q\_\{N\}\(under \(a\), or whenρM\\rho\_\{M\}andρN\\rho\_\{N\}are affine, so thatPPis an affine subspace\),PPwith its two projections is the pullback ofρM\\rho\_\{M\}andρN\\rho\_\{N\}in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\.

*Reason\.*P=\(ρM×ρN\)−1​\(ΔΘ\)P=\(\\rho\_\{M\}\\times\\rho\_\{N\}\)^\{\-1\}\(\\Delta\_\{\\Theta\}\), andρM×ρN\\rho\_\{M\}\\times\\rho\_\{N\}is transverse to the diagonalΔΘ\\Delta\_\{\\Theta\}, of codimensiondimΘ\\dim\\Theta, exactly when the condition in \(a\) holds; Lee’s Thm 6\.30\(a\) gives \(a\)\. Part \(b\) is a dimension count, since the left side of the condition has dimension at most\|M\|\+\|N\|\\lvert M\\rvert\+\\lvert N\\rvert\. Companion:[Background C\.7](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\),[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(d\)\. The proof of[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.3\(limits in slices and coslices; right adjoints\)\.\([Mac Lane, 1998](https://arxiv.org/html/2610.12728#bib.bib306), §III\.4, §V\.1 Exercise 1, §V\.5 Thm 1\)Let𝒞\\mathcal\{C\}be a category\.\(a\)In the slice𝒞/X\\mathcal\{C\}/X, a product ofa:A→Xa:A\\to Xandb:B→Xb:B\\to Xis a pullbackA×XBA\\times\_\{X\}Bofaaandbbin𝒞\\mathcal\{C\}, with its map toXX; one exists exactly when the other does\.\(b\)For an objectXX, the projection from the cosliceX/𝒞X/\\mathcal\{C\}\(the comma category\(X↓𝒞\)\(X\\downarrow\\mathcal\{C\}\)\) to𝒞\\mathcal\{C\}creates limits: every limit cone in𝒞\\mathcal\{C\}of the underlying diagram lifts to exactly one cone inX/𝒞X/\\mathcal\{C\}over the given diagram, and that cone is a limit\.\(c\)A functor that has a left adjoint preserves all limits that exist in its domain\. The forgetful functors𝐒𝐞𝐭∗→𝐒𝐞𝐭\\mathbf\{Set\}\_\{\*\}\\to\\mathbf\{Set\}and𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}have the left adjointX↦X⊔ptX\\mapsto X\\sqcup\\mathrm\{pt\}\.

*Form used\.*A product in the pointed slice𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)is a pullback over\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)in𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}by \(a\)\. By \(b\) a pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}of the underlying maps lifts to𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}with base point\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\), and by \(c\) any pullback in𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}is a pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, so it may be tested with unpointed curves\. Companions:[Background A\.11](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2),[Background A\.12](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1),[Background A\.13](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px1)\.

Used in\.[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) \(products in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}as pullbacks, tested with unpointed curves\)\. The proof of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.4\(smooth descent along submersions; quotient manifold theorem\)\.\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Thms 4\.29, 4\.30, 21\.10\)\(a\)Letπ:X→Y\\pi:X\\to Ybe a surjective smooth submersion\. A mapf¯:Y→Z\\bar\{f\}:Y\\to Zis smooth ifff¯∘π\\bar\{f\}\\circ\\piis smooth, and every smoothf:X→Zf:X\\to Zthat is constant on the fibres ofπ\\pifactors uniquely asf=f¯∘πf=\\bar\{f\}\\circ\\piwithf¯\\bar\{f\}smooth\.\(b\)If a Lie groupGGacts smoothly, freely and properly on a manifoldXX, thenX/GX/Ghas a unique smooth structure for whichπ:X→X/G\\pi:X\\to X/Gis a submersion, anddimX/G=dimX−dimG\\dim X/G=\\dim X\-\\dim G\.

*Instance\.*On pairs\(B,A\)\(B,A\)of full rank,\(B,A\)↦B​A\(B,A\)\\mapsto BAis a surjective submersion ontoℳr\\mathcal\{M\}\_\{r\}whose fibres are theGLr\\mathrm\{GL\}\_\{r\}\-orbits\{\(B​g−1,g​A\)\}\\\{\(Bg^\{\-1\},gA\)\\\}\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\), so a function of the factors that is constant on these orbits is a smooth function ofWWon the full\-rank locus\. Companion:[Background D\.22](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\.

Used in\.[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(d\) \(cometrics that are functions ofWW\),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(a\) \(descent of combination rules on the full\-rank locus\)\.

#### Real algebraic and tame geometry

Auxiliary result B\.5\(determinantal varieties\)\.\([Harris, 1992](https://arxiv.org/html/2610.12728#bib.bib156), Lectures 9, 12 and 14\);\([Schneider and Uschmajew, 2015](https://arxiv.org/html/2610.12728#bib.bib400), Thm 3\.2\)Let1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)\.\(a\)ℳ≤r⊆ℝm×n\\mathcal\{M\}\_\{\\leq r\}\\subseteq\\mathbb\{R\}^\{m\\times n\}is the common zero set of the\(r\+1\)×\(r\+1\)\(r\+1\)\\times\(r\+1\)minors\. It is closed and closed under all real scalings, so it is a closed cone with vertex00, of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\.\(b\)ℳr\\mathcal\{M\}\_\{r\}is open and dense inℳ≤r\\mathcal\{M\}\_\{\\leq r\}and is an embedded submanifold of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\. IfX∈ℳrX\\in\\mathcal\{M\}\_\{r\}has column spaceUUand row spaceVV, andX=B​AX=BAwithB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\},A∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}, thenTX​ℳr=\{B˙​A\+B​A˙\}=\{D:PU⟂​D​PV⟂=0\},T\_\{X\}\\mathcal\{M\}\_\{r\}=\\\{\\dot\{B\}A\+B\\dot\{A\}\\\}=\\\{D:P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}=0\\\},and\(B,A\)↦B​A\(B,A\)\\mapsto BAis a submersion from the pairs of full rank ontoℳr\\mathcal\{M\}\_\{r\}whose fibres are the orbits\{\(B​g−1,g​A\):g∈GLr\}\\\{\(Bg^\{\-1\},gA\):g\\in\\mathrm\{GL\}\_\{r\}\\\}\.\(c\)At a pointXXof ranks<rs<r, with column spaceUUand row spaceVV, the tangent cone ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}is\{D:rank⁡\(PU⟂​D​PV⟂\)≤r−s\}\\\{D:\\rk\(P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}\)\\leq r\-s\\\}, which is not a linear space\. Henceℳ≤r\\mathcal\{M\}\_\{\\leq r\}is not aC1C^\{1\}submanifold nearXX, andSing​ℳ≤r=ℳ≤r−1\\mathrm\{Sing\}\\,\\mathcal\{M\}\_\{\\leq r\}=\\mathcal\{M\}\_\{\\leq r\-1\}, algebraically and in theC1C^\{1\}sense\.

*Reason for the last claim of \(c\)\.*PU⟂​D​PV⟂P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}ranges over all\(m−s\)×\(n−s\)\(m\-s\)\\times\(n\-s\)matrices, and sincer−s<min⁡\(m−s,n−s\)r\-s<\\min\(m\-s,n\-s\)two of rankr−sr\-scan sum to a matrix of larger rank\. The tangent cone of aC1C^\{1\}submanifold is its tangent space, a linear space\. Companion:[Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\.

Used in\.[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(a\), \(b\), \(c\), \(e\) \(the image determines its vertex\),[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\(tangent space at\(I,X\)\(I,X\)\),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\(the translation stabiliser ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}is\{0\}\\\{0\\\}\),[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(b\), \(d\),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(a\)\. The proofs of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)and[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cite it\.

Auxiliary result B\.6\(cones that areC1C^\{1\}at their vertex\)\.elementaryLetC⊆ℝNC\\subseteq\\mathbb\{R\}^\{N\}be a closed set with0∈C0\\in Candt​C⊆CtC\\subseteq Cfor everyt\>0t\>0\. IfCCis aC1C^\{1\}embedded submanifold near00, thenCCequals the linear subspaceT0​CT\_\{0\}C\. Consequently, the vertex of a closed cone that is not a linear subspace is a singular point\.

*Reason\.*The tangent cone ofCCat00, the set of limits oftk​xkt\_\{k\}x\_\{k\}withxk∈Cx\_\{k\}\\in C,xk→0x\_\{k\}\\to 0andtk\>0t\_\{k\}\>0, equalsCCbecauseCCis a closed cone, and the tangent cone of aC1C^\{1\}submanifold at a point is its tangent space\. Companions:[Background C\.9](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2),[Background C\.10](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx2)\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4),[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(a\), \(c\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(c\),[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(e\) \(the closed cone𝒦≤k\\mathcal\{K\}\_\{\\leq k\}\)\.

Auxiliary result B\.7\(real normal forms; the skew\-determinantal cone\)\.\([Horn and Johnson, 2013](https://arxiv.org/html/2610.12728#bib.bib174), Ch\. 2\)\(a\)EveryΩ∈𝔰​𝔬​\(n\)\\Omega\\in\\mathfrak\{so\}\(n\)is orthogonally similar to a block\-diagonal matrix with2×22\\times 2blocksμk​\(0−110\)\\mu\_\{k\}\\left\(\\begin\{smallmatrix\}0&\-1\\\\ 1&0\\end\{smallmatrix\}\\right\),μ1≥μ2≥⋯\>0\\mu\_\{1\}\\geq\\mu\_\{2\}\\geq\\dots\>0, and zeros\. Hencerank⁡Ω\\rk\\Omegais even, the nonzero eigenvalues are±i​μk\\pm i\\mu\_\{k\}, the singular values areμ1,μ1,μ2,μ2,…\\mu\_\{1\},\\mu\_\{1\},\\mu\_\{2\},\\mu\_\{2\},\\dots, andΩ4\\Omega^\{4\}is symmetric positive semidefinite with eigenvaluesμk4\\mu\_\{k\}^\{4\}\. A skew matrix of odd size has00as an eigenvalue\.\(b\)EveryR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)is orthogonally similar to a block\-diagonal matrix with2×22\\times 2rotation blocks of anglesφk∈\(0,π\]\\varphi\_\{k\}\\in\(0,\\pi\]and ones\. Thenrank⁡\(R−I\)\\rk\(R\-I\)is twice the number of blocks,im⁡\(R−I\)\\mathrm\{im\}\(R\-I\)is the sum of the rotation planes, andR=eXR=e^\{X\}for a skewXXwhose eigenvalues are±i​φk\\pm i\\varphi\_\{k\}and00\.\(c\)For evenrrwith2≤r≤n−22\\leq r\\leq n\-2, the setΣr=\{Ω∈𝔰​𝔬​\(n\):rank⁡Ω≤r\}\\Sigma\_\{r\}=\\\{\\Omega\\in\\mathfrak\{so\}\(n\):\\rk\\Omega\\leq r\\\}is a closed cone of dimensionr⁡\(n−r\)\+r⁡\(r−1\)/2=r​n−r⁡\(r\+1\)/2r\(n\-r\)\+r\(r\-1\)/2=rn\-r\(r\+1\)/2\. Its points of rank exactlyrrform an open dense submanifold of that dimension, andΣr\\Sigma\_\{r\}is not a linear space\.

*Reason for \(c\)\.*A skew matrix of rankrris a rangeV∈Gr⁡\(r,n\)V\\in\\Gr\(r,n\)together with a nondegenerate skew form onVV, an open subset of𝔰​𝔬​\(r\)\\mathfrak\{so\}\(r\); and skew matrices of ranksrrand22can sum to rankr\+2≤nr\+2\\leq n\. Companion:[Background C\.13](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\.

Used in\.[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\(the Cayley–Neumann factor\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\), \(c\), \(d\) \(the dimension count and the density argument\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 4, the LOFT example of Exposé II\.B\.

Auxiliary result B\.8\(Segre cones\)\.\([Harris, 1992](https://arxiv.org/html/2610.12728#bib.bib156), Lecture 2\)The Segre coneℳ≤1\(p×q\)=\{ab⊤:a∈ℝp,b∈ℝq\}\\mathcal\{M\}\_\{\\leq 1\}\(p\\times q\)=\\\{ab^\{\\top\}:a\\in\\mathbb\{R\}^\{p\},b\\in\\mathbb\{R\}^\{q\}\\\}is a closed cone of dimensionp\+q−1p\+q\-1, it is not a linear space whenp,q≥2p,q\\geq 2, and it spansℝp×q\\mathbb\{R\}^\{p\\times q\}\. Hencekkgeneric rank\-one matrices span a space of dimensionmin⁡\(k,p​q\)\\min\(k,pq\): the parameters\(a1,b1,…,ak,bk\)\(a\_\{1\},b\_\{1\},\\dots,a\_\{k\},b\_\{k\}\)for whicha1​b1⊤,…,ak​bk⊤a\_\{1\}b\_\{1\}^\{\\top\},\\dots,a\_\{k\}b\_\{k\}^\{\\top\}span less form a proper algebraic subset of\(ℝp×ℝq\)k\(\\mathbb\{R\}^\{p\}\\times\\mathbb\{R\}^\{q\}\)^\{k\}\.

*Reason for the last claim\.*Spanning dimension at leastddis the nonvanishing of somed×dd\\times dminor of the coefficient matrix, a polynomial in the parameters, and distinct matrix unitsEi​jE\_\{ij\}show that it is not identically zero ford=min⁡\(k,p​q\)d=\\min\(k,pq\)\. Companion:[Background C\.14](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\.

Used in\.[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\(themmpoints\(E⊤​ei\)⊗\(F⊤​w0,i\)\(E^\{\\top\}e\_\{i\}\)\\otimes\(F^\{\\top\}w\_\{0,i\}\)give the generic value ofd⁡\(DoRAr\)d\(\\mathrm\{DoRA\}\_\{r\}\)\),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(the KronA image\), the VeRA card of Exposé VII\.

Auxiliary result B\.9\(semialgebraic and definable sets; dimension\)\.\([Bochnak et al\., 1998](https://arxiv.org/html/2610.12728#bib.bib40), Ch\. 2 and 9\);\([van den Dries, 1998](https://arxiv.org/html/2610.12728#bib.bib457), Ch\. 4, 6 and 7\);\([Wilkie, 1996](https://arxiv.org/html/2610.12728#bib.bib494)\)\(a\)\(Tarski–Seidenberg\.\) The image of a semialgebraic set under a semialgebraic map is semialgebraic\. \(Wilkie\.\) The expansionℝexp\\mathbb\{R\}\_\{\\exp\}of the real field by the exponential function is o\-minimal, so sets and maps defined by first\-order formulas from polynomials andexp\\exp\(softmax included\) are definable, and images of definable sets under definable maps are definable\.\(b\)Every definable setXXis a finite disjoint union ofC1C^\{1\}embedded submanifolds \(cells\), anddimX\\dim X, the largest of their dimensions, does not depend on the decomposition\. IfX⊆YX\\subseteq YthendimX≤dimY\\dim X\\leq\\dim Y, anddimf⁡\(X\)≤dimX\\dim f\(X\)\\leq\\dim Xfor every definable mapff\.\(c\)\(Definable choice and generic smoothness\.\) A definable surjection has a definable section, and a definable map isC1C^\{1\}outside a definable subset of its domain of smaller dimension\.\(d\)Consequently, for a definableC1C^\{1\}mapρ\\rhoon an open definable setQ⊆ℝkQ\\subseteq\\mathbb\{R\}^\{k\},dimρ⁡\(Q\)=maxq∈Q⁡rank⁡d​ρq\.\\dim\\rho\(Q\)=\\max\_\{q\\in Q\}\\rk d\\rho\_\{q\}\.

*Reason for \(d\)\.*At a point of maximal rank, the rank theorem \([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\) puts a sheet of that dimension insideρ⁡\(Q\)\\rho\(Q\)\. Conversely, by \(c\) a definable sectionssofρ\\rhoisC1C^\{1\}on an open subsetSSof a top\-dimensional cell ofρ⁡\(Q\)\\rho\(Q\), andρ∘s=idS\\rho\\circ s=\\mathrm\{id\}\_\{S\}givesrank⁡d​ρs⁡\(y\)≥dimS=dimρ⁡\(Q\)\\rk d\\rho\_\{s\(y\)\}\\geq\\dim S=\\dim\\rho\(Q\)\. Companions:[Background C\.17](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4),[Background C\.18](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\.

Used in\.[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\(d⁡\(M\)=dimIm⁡M=maxq⁡rank⁡d​ρqd\(M\)=\\dim\\Imop M=\\max\_\{q\}\\rk d\\rho\_\{q\}\), the obstructiond⁡\(M\)\>d⁡\(N\)d\(M\)\>d\(N\)of Exposé VIII\.C,[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(a\)\. The proofs of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)and[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cite it\.

Auxiliary result B\.10\(genericity; zero sets; the identity theorem\)\.\([Mityagin, 2020](https://arxiv.org/html/2610.12728#bib.bib333)\);\([Krantz and Parks, 2002](https://arxiv.org/html/2610.12728#bib.bib226), Ch\. 1 and 2\)\(a\)LetU⊆ℝNU\\subseteq\\mathbb\{R\}^\{N\}be open and connected andf:U→ℝf:U\\to\\mathbb\{R\}real\-analytic, for instance a polynomial\. Ifffvanishes on a nonempty open subset ofUU, thenf≡0f\\equiv 0\. Iff≢0f\\not\\equiv 0, its zero set is closed, nowhere dense and of Lebesgue measure zero\.\(b\)Hence a property that fails only on the common zero set of finitely many real\-analytic functions on a connected domain, not all identically zero, holds on an open dense set of full measure, and with probability one under every law with a density\. To prove such genericity it suffices to write failure as the vanishing of these functions and to exhibit one point where one of them does not vanish\. A countable union of such exceptional sets still has measure zero\.\(c\)A function equal to a convergent power series∑kak​tk\\sum\_\{k\}a\_\{k\}t^\{k\}near00determines its coefficients\. In particular, ifσ⁡\(c​t\)=c​σ​\(t\)\\sigma\(ct\)=c\\,\\sigma\(t\)near00for somec≠0c\\neq 0, thenc​ak=ck​akca\_\{k\}=c^\{k\}a\_\{k\}for everykk\.

Companions:[Background C\.21](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1),[Background C\.22](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\.

Used in\.[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(d\) \(a generic rank\-one update\),[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5),[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(c\) \(generic Schmidt ranks\),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(the generic rank of GraLoRA\),[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(a\) \(eehas full rank for almost everyWW, cell by cell of the quantiser\),[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\(c\),[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(the GELU obstruction\),[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\) \(prefixes with affine output form a nowhere\-dense analytic subset\)\. The proofs of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6),[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1),[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4),[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)and[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cite it\.

#### Matrix analysis

Auxiliary result B\.11\(singular value decomposition; Eckart–Young–Mirsky\)\.\([Eckart and Young, 1936](https://arxiv.org/html/2610.12728#bib.bib102);[Mirsky, 1960](https://arxiv.org/html/2610.12728#bib.bib329)\);\([Horn and Johnson, 2013](https://arxiv.org/html/2610.12728#bib.bib174), Ch\. 2 and 7\)\(a\)EveryX∈ℝm×nX\\in\\mathbb\{R\}^\{m\\times n\}of rankkkhas a singular value decompositionX=∑i≤kσi​ui​vi⊤X=\\sum\_\{i\\leq k\}\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{\\top\}withσ1≥⋯≥σk\>0\\sigma\_\{1\}\\geq\\dots\\geq\\sigma\_\{k\}\>0and orthonormal families\(ui\)\(u\_\{i\}\),\(vi\)\(v\_\{i\}\); two matrices lie in one orbit of\(P,R\)↦P​X​R\(P,R\)\\mapsto PXR,P∈O⁡\(m\)P\\in\\mathrm\{O\}\(m\),R∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\), iff they have the same singular values\. Applied to them1×m2m\_\{1\}\\times m\_\{2\}reshape ofu∈ℝm1⊗ℝm2u\\in\\mathbb\{R\}^\{m\_\{1\}\}\\otimes\\mathbb\{R\}^\{m\_\{2\}\}\(x⊗y↦x​y⊤x\\otimes y\\mapsto xy^\{\\top\}\), the decomposition is the Schmidt decompositionu=∑iσi​xi⊗yiu=\\sum\_\{i\}\\sigma\_\{i\}x\_\{i\}\\otimes y\_\{i\}, whose number of terms, the Schmidt rank, is the rank of the reshape\. Every matrix of rank at mostrrfactors asQ​VQVwithQ∈ℝm×rQ\\in\\mathbb\{R\}^\{m\\times r\}having orthonormal columns\.\(b\)\(Eckart–Young–Mirsky\.\) For0≤s<k0\\leq s<kthe truncation\[X\]s=∑i≤sσi​ui​vi⊤\[X\]\_\{s\}=\\sum\_\{i\\leq s\}\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{\\top\}minimises∥X−Y∥\\lVert X\-Y\\rVertoverrank⁡Y≤s\\rk Y\\leq s, with minimum\(∑i\>sσi2\)1/2\\big\(\\sum\_\{i\>s\}\\sigma\_\{i\}^\{2\}\\big\)^\{1/2\}, and the minimiser is unique iffσs\>σs\+1\\sigma\_\{s\}\>\\sigma\_\{s\+1\}\. The matrices\[X\]s\[X\]\_\{s\}andX−\[X\]sX\-\[X\]\_\{s\}are Frobenius\-orthogonal, so for every realcc∥X−c​\[X\]s∥2=∥X−\[X\]s∥2\+\(1−c\)2​∥\[X\]s∥2\.\\lVert X\-c\[X\]\_\{s\}\\rVert^\{2\}=\\lVert X\-\[X\]\_\{s\}\\rVert^\{2\}\+\(1\-c\)^\{2\}\\lVert\[X\]\_\{s\}\\rVert^\{2\}\.\(c\)\(Reduced\-rank regression\.\) ForL∈GLnL\\in\\mathrm\{GL\}\_\{n\},arg⁡minrank⁡Y≤s​∥\(X−Y\)​L∥=\[X​L\]s​L−1\\arg\\min\_\{\\rk Y\\leq s\}\\lVert\(X\-Y\)L\\rVert=\[XL\]\_\{s\}L^\{\-1\}, unique iffσs​\(X​L\)\>σs\+1​\(X​L\)\\sigma\_\{s\}\(XL\)\>\\sigma\_\{s\+1\}\(XL\); and\[X​L\]s=Πs​X​L\[XL\]\_\{s\}=\\Pi\_\{s\}XL, whereΠs\\Pi\_\{s\}is the orthogonal projector onto the top\-sseigenspace ofX​L​L⊤​X⊤XLL^\{\\top\}X^\{\\top\}\.

*Reason for \(c\)\.*SubstituteZ=Y​LZ=YL, which has the same rank asYY, and apply \(b\) toX​LXL; the left singular vectors ofX​LXLare the eigenvectors ofX​L​L⊤​X⊤XLL^\{\\top\}X^\{\\top\}\.

Used in\.[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\) \(the SVD frames of LoRA\-XS\),[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\) \(splitting an SVD\),[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(3\),[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(c\) \(Schmidt decompositions\),[Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\(a\) and the lemma in \(b\),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(a\) \(the balanced representative\),[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\(compact SVD\),[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\), \(c\) \(LoftQ’s rank\-rrstep and its defect\),[Proposition VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)\(c\) \(the factorisationR⊤​VR^\{\\top\}V\), Prediction X\.5 \(the distance tokk\-term Kronecker sums\)\. The proofs of[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)and[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cite it\.

Auxiliary result B\.12\(rank inequalities; the Hadamard rank bound\)\.\([Horn and Johnson, 1991](https://arxiv.org/html/2610.12728#bib.bib173), Ch\. 4 and 5\)For matrices of compatible sizes:\(a\)rank⁡\(X\+Y\)≤rank⁡X\+rank⁡Y\\rk\(X\+Y\)\\leq\\rk X\+\\rk Yandrank⁡\(X​Y\)≤min⁡\(rank⁡X,rank⁡Y\)\\rk\(XY\)\\leq\\min\(\\rk X,\\rk Y\);\(b\)rank⁡\(C⊗D\)=rank⁡C⋅rank⁡D\\rk\(C\\otimes D\)=\\rk C\\cdot\\rk D;\(c\)\(Hadamard rank bound\)rank⁡\(X⊙Y\)≤rank⁡X⋅rank⁡Y\\rk\(X\\odot Y\)\\leq\\rk X\\cdot\\rk Y\. Explicitly, ifX=B1​A1X=B\_\{1\}A\_\{1\}andY=B2​A2Y=B\_\{2\}A\_\{2\}with inner dimensionsr1r\_\{1\}andr2r\_\{2\}, thenX⊙Y=\(B1∙B2\)​\(A1⊤∙A2⊤\)⊤,X\\odot Y=\(B\_\{1\}\\bullet B\_\{2\}\)\\,\(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\},where rowiiof the face\-splitting productB1∙B2∈ℝm×r1​r2B\_\{1\}\\bullet B\_\{2\}\\in\\mathbb\{R\}^\{m\\times r\_\{1\}r\_\{2\}\}is the Kronecker product of the rowsiiofB1B\_\{1\}andB2B\_\{2\}\. In particularW0⊙\(B​A\)=∑k≤rdiag⁡\(bk\)​W0​diag⁡\(ak\)W\_\{0\}\\odot\(BA\)=\\sum\_\{k\\leq r\}\\diag\(b\_\{k\}\)\\,W\_\{0\}\\,\\diag\(a\_\{k\}\), summed over the columnsbkb\_\{k\}ofBBand the rowsaka\_\{k\}ofAA, has rank at mostr​rank⁡W0r\\rk W\_\{0\}\.

*Reason for \(c\)\.*X⊙YX\\odot Yis the submatrix ofX⊗YX\\otimes Yon the rows\(i,i\)\(i,i\)and the columns\(j,j\)\(j,j\), so \(b\) applies\. Companion:[Background B\.13](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)\.

Used in\.[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\),[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(a\) \(HiRA\),[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)\(subadditivity\),[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(b\),[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(LoHa, GraLoRA\),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\(two HiRA cycles\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollaries 1 and 5, the face\-splitting arrow of rule A3 \(Exposé VIII\.C\)\.

Auxiliary result B\.13\(the Van Loan–Pitsianis rearrangement; operator\-Schmidt decomposition\)\.\([Van Loan and Pitsianis, 1993](https://arxiv.org/html/2610.12728#bib.bib459)\);\([Van Loan, 2000](https://arxiv.org/html/2610.12728#bib.bib458)\)Letm=m1​m2m=m\_\{1\}m\_\{2\}andn=n1​n2n=n\_\{1\}n\_\{2\}, cutΔ∈ℝm×n\\Delta\\in\\mathbb\{R\}^\{m\\times n\}into anm1×n1m\_\{1\}\\times n\_\{1\}grid ofm2×n2m\_\{2\}\\times n\_\{2\}blocksΔi​j\\Delta\_\{ij\}, and letℛ⁡\(Δ\)∈ℝm1​n1×m2​n2\\mathcal\{R\}\(\\Delta\)\\in\\mathbb\{R\}^\{m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}\}have the rowsvec⁡\(Δi​j\)⊤\\vect\(\\Delta\_\{ij\}\)^\{\\top\}, ordered withiirunning fastest\.\(a\)ℛ\\mathcal\{R\}is a linear bijection that permutes entries, hence a Frobenius isometry, andℛ⁡\(C⊗D\)=vec⁡C​vec⁡D⊤\\mathcal\{R\}\(C\\otimes D\)=\\vect C\\,\\vect D^\{\\top\}forC∈ℝm1×n1C\\in\\mathbb\{R\}^\{m\_\{1\}\\times n\_\{1\}\}andD∈ℝm2×n2D\\in\\mathbb\{R\}^\{m\_\{2\}\\times n\_\{2\}\}\.\(b\)HenceΔ=∑i≤kCi⊗Di\\Delta=\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}for someCi,DiC\_\{i\},D\_\{i\}iffrank⁡ℛ⁡\(Δ\)≤k\\rk\\mathcal\{R\}\(\\Delta\)\\leq k\. The Kronecker rankrank⊗⁡Δ\\rk\_\{\\otimes\}\\Deltaequalsrank⁡ℛ⁡\(Δ\)≤min⁡\(m1​n1,m2​n2\)\\rk\\mathcal\{R\}\(\\Delta\)\\leq\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\), and the nearest sum ofkkKronecker products in Frobenius norm isℛ−1​\(\[ℛ⁡\(Δ\)\]k\)\\mathcal\{R\}^\{\-1\}\\big\(\[\\mathcal\{R\}\(\\Delta\)\]\_\{k\}\\big\)\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\)\)\.\(c\)The SVD ofℛ⁡\(Δ\)\\mathcal\{R\}\(\\Delta\)gives the operator\-Schmidt decompositionΔ=∑k≤Ksk​Ck⊗Dk\\Delta=\\sum\_\{k\\leq K\}s\_\{k\}C\_\{k\}\\otimes D\_\{k\}withsk\>0s\_\{k\}\>0, Frobenius\-orthonormal families\(Ck\)\(C\_\{k\}\),\(Dk\)\(D\_\{k\}\)andK=rank⊗⁡ΔK=\\rk\_\{\\otimes\}\\Delta\. Conversely, any expansion of this form with positive coefficients and Frobenius\-orthonormal families is one, so its number of terms isrank⊗⁡Δ\\rk\_\{\\otimes\}\\Delta\.

Companions:[Background B\.11](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1),[Background B\.12](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px2)\.

Used in\.[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(a\)–\(e\),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\(the Kronecker family asℛ∗−1​LoRAk\\mathcal\{R\}^\{\-1\}\_\{\*\}\\mathrm\{LoRA\}\_\{k\}\), Prediction X\.5\. The proof of[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.14\(the min\-cut bound for tensor networks\)\.\([Cui et al\., 2016](https://arxiv.org/html/2610.12728#bib.bib86)\)LetΓ\\Gammabe a tensor network evaluated in\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗\)\(\\mathbf\{FinVect\},\\otimes\), every wire attached to a box, and letℐ⊔𝒪\\mathcal\{I\}\\sqcup\\mathcal\{O\}be a partition of its boxes\. Contracting each side separately factors the value⟦Γ⟧\\llbracket\\Gamma\\rrbracketthrough the tensor product of the wires that cross the partition and of the open legs attached on the wrong side, sorank⁡⟦Γ⟧\\rk\\llbracket\\Gamma\\rrbracketis at most the productwwof their dimensions, and at mostmin⁡w\\min wover all partitions\. For fixed bond dimensions the largest rank over all choices of tensors can be strictly smaller than the minimal cut weight; equality holds, for instance, when every bond dimension is a power of one integer\.

*Form used\.*[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)states the bound with the paper’s conventions \(bare strands, copy spiders counted once, formal sums\) and proves it; only the upper bound is used\. Companion:[Background B\.8](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2)\.

Used in\.[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)and its consequences,[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(maximum ranks\), the “high\-rank PEFT” remark of Exposé II\.C\.

Auxiliary result B\.15\(manifolds of fixed Tucker and tensor\-train rank\)\.\([Koch and Lubich, 2010](https://arxiv.org/html/2610.12728#bib.bib220)\)\(Tucker\);\([Holtz et al\., 2012](https://arxiv.org/html/2610.12728#bib.bib172)\)\(TT\);\([Uschmajew and Vandereycken, 2013](https://arxiv.org/html/2610.12728#bib.bib454)\)\(hierarchical Tucker\)\(a\)\(Tucker\.\) For an admissible multilinear rank\(r1,…,rd\)\(r\_\{1\},\\dots,r\_\{d\}\)of tensors inℝn1×⋯×nd\\mathbb\{R\}^\{n\_\{1\}\\times\\dots\\times n\_\{d\}\}, the tensors of exactly that rank form an embedded submanifold of dimension∏krk\+∑krk​\(nk−rk\)\\prod\_\{k\}r\_\{k\}\+\\sum\_\{k\}r\_\{k\}\(n\_\{k\}\-r\_\{k\}\)\. In the parametrisation by a core inℝr1×⋯×rd\\mathbb\{R\}^\{r\_\{1\}\\times\\dots\\times r\_\{d\}\}and factor matrices of full column rank, with∏krk\+∑knk​rk\\prod\_\{k\}r\_\{k\}\+\\sum\_\{k\}n\_\{k\}r\_\{k\}parameters, the fibres are the orbits of∏kGLrk\\prod\_\{k\}\\mathrm\{GL\}\_\{r\_\{k\}\}, so the dimension is the parameter count minus∑krk2\\sum\_\{k\}r\_\{k\}^\{2\}\.\(b\)\(Tensor train\.\) For an admissible TT rank\(r1,…,rd−1\)\(r\_\{1\},\\dots,r\_\{d\-1\}\), withr0=rd=1r\_\{0\}=r\_\{d\}=1, the tensors of exactly that rank form an embedded submanifold of dimension∑k=1drk−1​nk​rk−∑k=1d−1rk2\\sum\_\{k=1\}^\{d\}r\_\{k\-1\}n\_\{k\}r\_\{k\}\-\\sum\_\{k=1\}^\{d\-1\}r\_\{k\}^\{2\}, the number of core entries minus the dimension of the bond gauge∏k=1d−1GLrk\\prod\_\{k=1\}^\{d\-1\}\\mathrm\{GL\}\_\{r\_\{k\}\}\.

Here admissible means realised by some tensor; at minimal ranks the bond gauge acts freely\. Companion:[Background B\.9](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx2.Px1)\.

Used in\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(b\), row “TT / Tucker”\.

Auxiliary result B\.16\(circulant matrices and the discrete Fourier transform\)\.\([Gray, 2006](https://arxiv.org/html/2610.12728#bib.bib135), Thm 3\.1\)\(a\)A circulant matrix with first columnw∈ℝbw\\in\\mathbb\{R\}^\{b\}satisfiescirc⁡\(w\)=F−1​diag⁡\(F​w\)​F\\mathrm\{circ\}\(w\)=F^\{\-1\}\\diag\(Fw\)\\,F, withFFthe discrete Fourier matrix\. The powersP0,…,Pb−1P^\{0\},\\dots,P^\{b\-1\}of the cyclic shift are a basis of the circulants, with Gram matrixb​IbIin the Frobenius inner product\.\(b\)Overℝ\\mathbb\{R\}, since the eigenvalues of a real circulant at conjugate frequencies are conjugate,circ⁡\(w\)\\mathrm\{circ\}\(w\)is orthogonally similar to a block\-diagonal matrix of2×22\\times 2rotation–scaling blocks and1×11\\times 1blocks at the self\-conjugate frequencies\.\(c\)For a real coefficientccand a frequency\(u,v\)\(u,v\), the real part of the inverse two\-dimensional transform ofc​e\(u,v\)c\\,e\_\{\(u,v\)\}is the cosine patternRe​ifft2​\(c​e\(u,v\)\)j,l=cm​n​cos⁡\(2​π​\(u​j/m\+v​l/n\)\)\.\\mathrm\{Re\}\\,\\mathrm\{ifft2\}\(c\\,e\_\{\(u,v\)\}\)\_\{j,l\}=\\tfrac\{c\}\{mn\}\\cos\\big\(2\\pi\(uj/m\+vl/n\)\\big\)\.It is the same for\(u,v\)\(u,v\)and\(−u,−v\)\(\-u,\-v\), and it is even under\(j,l\)↦\(−j,−l\)\(j,l\)\\mapsto\(\-j,\-l\), indices taken modulommandnn\.

Used in\.[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(the FourierFT gauge from conjugate pairs\),[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\(FourierFT and C3A as masks in transported frames\)\. The proof of[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.17\(Ky Fan’s maximum principle; interlacing\)\.\([Fan, 1949](https://arxiv.org/html/2610.12728#bib.bib108)\);\([Bhatia, 1997](https://arxiv.org/html/2610.12728#bib.bib31), Ch\. III\);\([Horn and Johnson, 1991](https://arxiv.org/html/2610.12728#bib.bib173), Ch\. 3\)\(a\)\(Ky Fan\.\) LetS∈ℝn×nS\\in\\mathbb\{R\}^\{n\\times n\}be symmetric with eigenvaluesλ1≥⋯≥λn\\lambda\_\{1\}\\geq\\dots\\geq\\lambda\_\{n\}and let1≤k≤n1\\leq k\\leq n\. Thenmax\{tr\(PSP⊤\):P∈ℝk×n,PP⊤=Ik\}=λ1\+⋯\+λk,\\max\\\{\\tr\(PSP^\{\\top\}\):P\\in\\mathbb\{R\}^\{k\\times n\},\\ PP^\{\\top\}=I\_\{k\}\\\}=\\lambda\_\{1\}\+\\dots\+\\lambda\_\{k\},attained when the rows ofPPspan a top\-kkeigenspace\. ForS=X⊤​XS=X^\{\\top\}Xthis readsmaxP⁡∥X​P⊤∥2=σ1​\(X\)2\+⋯\+σk​\(X\)2\.\\max\_\{P\}\\lVert XP^\{\\top\}\\rVert^\{2\}=\\sigma\_\{1\}\(X\)^\{2\}\+\\dots\+\\sigma\_\{k\}\(X\)^\{2\}\.\(b\)\(Interlacing for compressions\.\) IfP∈ℝk×nP\\in\\mathbb\{R\}^\{k\\times n\}andQ∈ℝl×mQ\\in\\mathbb\{R\}^\{l\\times m\}have orthonormal rows, thenσj​\(Q​X​P⊤\)≤σj​\(X\)\\sigma\_\{j\}\(QXP^\{\\top\}\)\\leq\\sigma\_\{j\}\(X\)for everyjjand everyX∈ℝm×nX\\in\\mathbb\{R\}^\{m\\times n\}; for symmetricSS,λj​\(P​S​P⊤\)≤λj​\(S\)\\lambda\_\{j\}\(PSP^\{\\top\}\)\\leq\\lambda\_\{j\}\(S\)\.\(c\)Consequently, ifS∈𝔰​𝔬​\(n\)S\\in\\mathfrak\{so\}\(n\)has eigenvalues±i​μ1,±i​μ2,…\\pm i\\mu\_\{1\},\\pm i\\mu\_\{2\},\\dotswithμ1≥μ2≥…\\mu\_\{1\}\\geq\\mu\_\{2\}\\geq\\dots, then for everyP∈ℝk×nP\\in\\mathbb\{R\}^\{k\\times n\}with orthonormal rows∥P​S​P⊤∥2≤2​∑j≤⌊k/2⌋μj2,\\lVert PSP^\{\\top\}\\rVert^\{2\}\\leq 2\\sum\_\{j\\leq\\lfloor k/2\\rfloor\}\\mu\_\{j\}^\{2\},with equality when the row space ofPPcontains the invariant subspace of the top⌊k/2⌋\\lfloor k/2\\rfloorrotation pairs ofSS\.

*Reason for \(c\)\.*P​S​P⊤PSP^\{\\top\}is skew of sizekk, so by[Auxiliary result B\.7](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(a\) its singular values areν1,ν1,ν2,ν2,…\\nu\_\{1\},\\nu\_\{1\},\\nu\_\{2\},\\nu\_\{2\},\\dots, with an extra00whenkkis odd\. By \(b\),νj=σ2​j−1​\(P​S​P⊤\)≤σ2​j−1​\(S\)=μj\\nu\_\{j\}=\\sigma\_\{2j\-1\}\(PSP^\{\\top\}\)\\leq\\sigma\_\{2j\-1\}\(S\)=\\mu\_\{j\}, and∥P​S​P⊤∥2=2​∑j≤⌊k/2⌋νj2\\lVert PSP^\{\\top\}\\rVert^\{2\}=2\\sum\_\{j\\leq\\lfloor k/2\\rfloor\}\\nu\_\{j\}^\{2\}\. If the row space contains an invariant subspaceWWof the top pairs, thenW⟂W^\{\\perp\}is invariant too, and the compression isS\|WS\|\_\{W\}plus zeros\. Companion:[Background D\.32](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)\.

Used in\.[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)\(alignment: amongA0A\_\{0\}withA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cIthe rate∥G​A0⊤∥2\\lVert GA\_\{0\}^\{\\top\}\\rVert^\{2\}is largest on the top\-rrright singular space ofGG\), the LOFT example of Exposé II\.B, part \(d\)\.

Auxiliary result B\.18\(singular values of products; the Loewner order\)\.\([Horn and Johnson, 1991](https://arxiv.org/html/2610.12728#bib.bib173), Ch\. 3\);\([Bhatia, 1997](https://arxiv.org/html/2610.12728#bib.bib31), Ch\. III\);\([Horn and Johnson, 2013](https://arxiv.org/html/2610.12728#bib.bib174), Ch\. 7\)\(a\)\(Horn\.\)σi\+j−1​\(X​Y\)≤σi​\(X\)​σj​\(Y\)\\sigma\_\{i\+j\-1\}\(XY\)\\leq\\sigma\_\{i\}\(X\)\\,\\sigma\_\{j\}\(Y\)\. In particularσi​\(X​Y\)≤σi​\(X\)​∥Y∥op,σi​\(X​Y\)≤∥X∥op​σi​\(Y\),\\sigma\_\{i\}\(XY\)\\leq\\sigma\_\{i\}\(X\)\\,\\lVert Y\\rVert\_\{\\rm op\},\\qquad\\sigma\_\{i\}\(XY\)\\leq\\lVert X\\rVert\_\{\\rm op\}\\,\\sigma\_\{i\}\(Y\),and a product of contractions \(matrices with∥⋅∥op≤1\\lVert\\cdot\\rVert\_\{\\rm op\}\\leq 1\) is a contraction\.\(b\)If∥Y∥op≤1\\lVert Y\\rVert\_\{\\rm op\}\\leq 1, thenY⊤​Y⪯IY^\{\\top\}Y\\preceq I, andW​Y⊤​Y​W⊤⪯W​W⊤WY^\{\\top\}YW^\{\\top\}\\preceq WW^\{\\top\}for everyWW\.\(c\)\(Weyl monotonicity\.\) IfS⪯TS\\preceq Tare symmetric, thenλi​\(S\)≤λi​\(T\)\\lambda\_\{i\}\(S\)\\leq\\lambda\_\{i\}\(T\)for everyii\. With \(b\),σi​\(W​Y⊤\)≤σi​\(W\)\\sigma\_\{i\}\(WY^\{\\top\}\)\\leq\\sigma\_\{i\}\(W\)whenever∥Y∥op≤1\\lVert Y\\rVert\_\{\\rm op\}\\leq 1\.

Used in\.[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\(the Cayley–Neumann factor contracts for smallQQ\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 4 \(each default\-OFT merge decreasesKoutK\_\{\\rm out\}in the Loewner order and lowers every singular value\)\.

Auxiliary result B\.19\(polar, Cartan andK​A​KKAKdecompositions\)\.\([Horn and Johnson, 2013](https://arxiv.org/html/2610.12728#bib.bib174), Ch\. 2 and 7\)\(a\)\(Polar\.\) Everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}is uniquelyg=k​eSg=k\\,e^\{S\}withk∈O⁡\(r\)k\\in\\mathrm\{O\}\(r\)andS∈SymrS\\in\\mathrm\{Sym\}\_\{r\}, and\(k,S\)↦k​eS\(k,S\)\\mapsto k\\,e^\{S\}is a diffeomorphismO⁡\(r\)×Symr→GLr\\mathrm\{O\}\(r\)\\times\\mathrm\{Sym\}\_\{r\}\\to\\mathrm\{GL\}\_\{r\}\. Henceg↦g⊤​g=e2​Sg\\mapsto g^\{\\top\}g=e^\{2S\}identifies the cosetsO⁡\(r\)​g\\mathrm\{O\}\(r\)gwith the positive definite matrices,O⁡\(r\)\\GLr≅Symr\+\\mathrm\{O\}\(r\)\\backslash\\mathrm\{GL\}\_\{r\}\\cong\\mathrm\{Sym\}^\{\+\}\_\{r\}\.\(b\)\(Cartan\.\)𝔤​𝔩r=𝔰​𝔬​\(r\)⊕Symr\\mathfrak\{gl\}\_\{r\}=\\mathfrak\{so\}\(r\)\\oplus\\mathrm\{Sym\}\_\{r\}throughξ=12​\(ξ−ξ⊤\)\+12​\(ξ\+ξ⊤\)\\xi=\\tfrac\{1\}\{2\}\(\\xi\-\\xi^\{\\top\}\)\+\\tfrac\{1\}\{2\}\(\\xi\+\\xi^\{\\top\}\), with\[𝔰​𝔬,𝔰​𝔬\]⊆𝔰​𝔬\[\\mathfrak\{so\},\\mathfrak\{so\}\]\\subseteq\\mathfrak\{so\},\[𝔰​𝔬,Sym\]⊆Sym\[\\mathfrak\{so\},\\mathrm\{Sym\}\]\\subseteq\\mathrm\{Sym\}and\[Sym,Sym\]⊆𝔰​𝔬\[\\mathrm\{Sym\},\\mathrm\{Sym\}\]\\subseteq\\mathfrak\{so\}\.\(c\)\(K​A​KKAK\.\) Everyg∈GL\+​\(b\)g\\in\\mathrm\{GL\}^\{\+\}\(b\)isk1​diag⁡\(d\)​k2k\_\{1\}\\diag\(d\)\\,k\_\{2\}withk1,k2∈SO⁡\(b\)k\_\{1\},k\_\{2\}\\in\\mathrm\{SO\}\(b\)andd∈ℝ\>0bd\\in\\mathbb\{R\}^\{b\}\_\{\>0\}\.

*Reason\.*\(a\) combines the polar formg=k​Pg=kP,PPpositive definite and unique, with the fact thatexp:Symr→Symr\+\\exp:\\mathrm\{Sym\}\_\{r\}\\to\\mathrm\{Sym\}\_\{r\}^\{\+\}is a diffeomorphism \(spectral theorem\)\. \(c\) is the SVD with signs absorbed: if both singular frames have determinant−1\-1, flip the first column of each\. Companion:[Background D\.27](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\.

Used in\.[Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(gradient descent on LoRA as a family indexed byO⁡\(r\)\\GLr\\mathrm\{O\}\(r\)\\backslash\\mathrm\{GL\}\_\{r\}\),[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(iii\),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(a\) \(reduction toh⁡\(t\)=f⁡\(et​ξ⋅x\)h\(t\)=f\(e^\{t\\xi\}\\cdot x\)withξ\\xisymmetric\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 4 \(the monoid generated by Cayley–Neumann factors\)\.

Auxiliary result B\.20\(Sylvester equations\)\.\([Bhatia and Rosenthal, 1997](https://arxiv.org/html/2610.12728#bib.bib32)\);\([Horn and Johnson, 1991](https://arxiv.org/html/2610.12728#bib.bib173), Ch\. 4\)ForS∈ℝp×pS\\in\\mathbb\{R\}^\{p\\times p\}andT∈ℝq×qT\\in\\mathbb\{R\}^\{q\\times q\}, the mapX↦S​X\+X​TX\\mapsto SX\+XTonℝp×q\\mathbb\{R\}^\{p\\times q\}has the eigenvaluesλi​\(S\)\+λj​\(T\)\\lambda\_\{i\}\(S\)\+\\lambda\_\{j\}\(T\)\. SoS​X\+X​T=CSX\+XT=Chas a unique solution for everyCCiffSSand−T\-Thave no common eigenvalue\. IfSSandTTare symmetric positive definite, the solution isX=∫0∞e−t​S​C​e−t​T​𝑑tX=\\int\_\{0\}^\{\\infty\}e^\{\-tS\}Ce^\{\-tT\}\\,dt\.

Companion:[Background D\.33](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)\.

Used in\.[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(practice: LoRA\-Pro’s Sylvester\-optimal free matrix exists and is unique on the full\-rank locus\),[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(published variants\)\. The proof of[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.21\(the nuclear norm as a factorisation norm\)\.\([Srebro et al\., 2004](https://arxiv.org/html/2610.12728#bib.bib426), Lemma 1\); see also[Recht et al\. \(2010\)](https://arxiv.org/html/2610.12728#bib.bib386)ForX∈ℝm×nX\\in\\mathbb\{R\}^\{m\\times n\}and everyr≥rank⁡Xr\\geq\\rk X,∥X∥∗=min\{12\(∥B∥2\+∥A∥2\):B∈ℝm×r,A∈ℝr×n,BA=X\}=minB​A=X∥B∥∥A∥,\\lVert X\\rVert\_\{\*\}=\\min\\Big\\\{\\tfrac\{1\}\{2\}\\big\(\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}\\big\):B\\in\\mathbb\{R\}^\{m\\times r\},\\ A\\in\\mathbb\{R\}^\{r\\times n\},\\ BA=X\\Big\\\}=\\min\_\{BA=X\}\\lVert B\\rVert\\,\\lVert A\\rVert,attained atB=U​Σ1/2​OB=U\\Sigma^\{1/2\}O,A=O⊤​Σ1/2​V⊤A=O^\{\\top\}\\Sigma^\{1/2\}V^\{\\top\}for an SVDX=U​Σ​V⊤X=U\\Sigma V^\{\\top\}padded to inner sizerrand anyO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\)\.

Used in\.[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(a\) \(the minimum value2​∥B​A∥∗2\\lVert BA\\rVert\_\{\*\}on a full\-rank orbit\) and its commentary \(weight decay on the factors as a nuclear\-norm penalty\)\.

#### Lie groups and invariant theory

Auxiliary result B\.22\(the Cartan–Dieudonné theorem in Scherk’s sharp form\)\.\([Scherk, 1950](https://arxiv.org/html/2610.12728#bib.bib399)\);\([Artin, 1957](https://arxiv.org/html/2610.12728#bib.bib16), Ch\. III\)Foru≠0u\\neq 0letHu=I−2​u​u⊤/∥u∥2H\_\{u\}=I\-2uu^\{\\top\}/\\lVert u\\rVert^\{2\}, the reflection that fixesu⟂u^\{\\perp\};detHu=−1\\det H\_\{u\}=\-1\. A product ofjjreflections fixes an intersection ofjjhyperplanes, so it satisfiesrank⁡\(R−I\)≤j\\rk\(R\-I\)\\leq j\. EveryR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)is a product of at mostnnreflections \(Cartan–Dieudonné\)\. For the Euclidean form the bound is sharp \(Scherk\): ifk=rank⁡\(R−I\)k=\\rk\(R\-I\), thenRRis a product of exactlykkreflections and of no fewer\. HencedetR=\(−1\)rank⁡\(R−I\)\\det R=\(\-1\)^\{\\rk\(R\-I\)\}, and everyR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)is a product ofrank⁡\(R−I\)\\rk\(R\-I\)reflections, an even number\.

*Form used\.*Scherk’s theorem for a nondegenerate quadratic space adds two reflections when the moved spaceim⁡\(R−I\)\\mathrm\{im\}\(R\-I\)is totally isotropic; a definite form has no nonzero isotropic vectors, so this case does not occur\. Companion:[Background D\.11](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\.

Used in\.[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\) \(⊇\\supseteq, and the parity ofrr\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 5 \(re\-HRA\)\.

Auxiliary result B\.23\(the Cayley map\)\.\([Cayley, 1846](https://arxiv.org/html/2610.12728#bib.bib52)\);\([Lezcano\-Casado and Martínez\-Rubio, 2019](https://arxiv.org/html/2610.12728#bib.bib244)\)LetΩ∈𝔰​𝔬​\(n\)\\Omega\\in\\mathfrak\{so\}\(n\)\.\(a\)I−ΩI\-\\Omegais invertible, since the eigenvalues ofΩ\\Omegaare imaginary, andC⁡\(Ω\)=\(I\+Ω\)​\(I−Ω\)−1C\(\\Omega\)=\(I\+\\Omega\)\(I\-\\Omega\)^\{\-1\}lies inSO⁡\(n\)\\mathrm\{SO\}\(n\)and commutes with every polynomial inΩ\\Omega\.\(b\)C⁡\(Ω\)−I=2​Ω​\(I−Ω\)−1C\(\\Omega\)\-I=2\\Omega\(I\-\\Omega\)^\{\-1\}, sorank⁡\(C⁡\(Ω\)−I\)=rank⁡Ω\\rk\(C\(\\Omega\)\-I\)=\\rk\\Omega; andC⁡\(Ω\)=I\+2​Ω\+2​Ω2\+…C\(\\Omega\)=I\+2\\Omega\+2\\Omega^\{2\}\+\\dots, sod​C0=2​iddC\_\{0\}=2\\,\\mathrm\{id\}\.\(c\)CCis a diffeomorphism from𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)onto the open dense subset\{R∈SO⁡\(n\):−1∉spec​R\}\\\{R\\in\\mathrm\{SO\}\(n\):\-1\\notin\\mathrm\{spec\}\\,R\\\}, with inverseR↦\(R−I\)​\(R\+I\)−1R\\mapsto\(R\-I\)\(R\+I\)^\{\-1\}\. In particular it missesdiag⁡\(−1,−1,1,…,1\)\\diag\(\-1,\-1,1,\\dots,1\)\.\(d\)Every element ofSO⁡\(n\)\\mathrm\{SO\}\(n\)is a product of two elements of the image ofCC\.\(e\)\(Neumann truncation\.\)\(I−Q\)​\(I\+Q\+Q2\+Q3\)=I−Q4\(I\-Q\)\(I\+Q\+Q^\{2\}\+Q^\{3\}\)=I\-Q^\{4\}, so for skewQQR:=I\+2​Q\+2​Q2\+2​Q3\+Q4=C⁡\(Q\)​\(I−Q4\),R⊤​R=\(I−Q4\)2,R:=I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}=C\(Q\)\\,\(I\-Q^\{4\}\),\\qquad R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\},becauseQ4Q^\{4\}is symmetric and commutes with the orthogonal matrixC⁡\(Q\)C\(Q\); the singular values ofRRare\|1−μk4\|\\lvert 1\-\\mu\_\{k\}^\{4\}\\rvert, where±i​μk\\pm i\\mu\_\{k\}are the eigenvalues ofQQ\.

*Reason for \(d\)\.*By[Auxiliary result B\.7](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\),R=eXR=e^\{X\}withXXskew and eigenvalues±i​φ\\pm i\\varphi,φ∈\[0,π\]\\varphi\\in\[0,\\pi\]\. TheneX/2e^\{X/2\}has eigenvaluese±iφ/2≠−1e^\{\\pm i\\varphi/2\}\\neq\-1, andR=\(eX/2\)2R=\(e^\{X/2\}\)^\{2\}\. Companion:[Background D\.7](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\.

Used in\.[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\(HF PEFT’s default OFT\),[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(c\) \(the germ of the image atW0W\_\{0\}\) and \(d\) \(exact Cayley OFT never produces eigenvalue−1\-1\),[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollaries 2 and 4, the LOFT example of Exposé II\.B\. The proof of[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.24\(Lie subgroups, generation, and Yamabe’s theorem\)\.\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Prop\. 7\.14, Thms 19\.25, 19\.26\);\([Yamabe, 1950](https://arxiv.org/html/2610.12728#bib.bib519)\);\([Goto, 1969](https://arxiv.org/html/2610.12728#bib.bib134)\)\(a\)In a connected Lie group every neighbourhood of the identity generates the group\. A symmetric neighbourhoodU=U−1U=U^\{\-1\}therefore generates it as a monoid\.\(b\)For every Lie subalgebra𝔥\\mathfrak\{h\}of the Lie algebra ofGGthere is a unique connected Lie subgroup ofGGwith Lie algebra𝔥\\mathfrak\{h\}, namely the subgroup generated byexp⁡𝔥\\exp\\mathfrak\{h\}\. Lie subgroups are weakly embedded, so an inclusionH⊆H′H\\subseteq H^\{\\prime\}of Lie subgroups is smooth and givesLie⁡\(H\)⊆Lie⁡\(H′\)\\mathrm\{Lie\}\(H\)\\subseteq\\mathrm\{Lie\}\(H^\{\\prime\}\)\.\(c\)\(Yamabe\.\) Every arcwise connected subgroup of a Lie group is a Lie subgroup\.\(d\)Consequently, the subgroup generated by connected Lie subgroupsH1,…,HpH\_\{1\},\\dots,H\_\{p\}ofGGis the connected Lie subgroupHHwithLie⁡\(H\)=Lie⁡⟨𝔥1,…,𝔥p⟩\\mathrm\{Lie\}\(H\)=\\mathrm\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle, the smallest Lie subalgebra containing every𝔥i\\mathfrak\{h\}\_\{i\}\.

*Reason for \(d\)\.*The generated subgroup is arcwise connected, so it is a connected Lie subgroupHHby \(c\), andLie⁡\(H\)⊇𝔥i\\mathrm\{Lie\}\(H\)\\supseteq\\mathfrak\{h\}\_\{i\}for everyiiby \(b\)\. The connected Lie subgroupH′H^\{\\prime\}with Lie algebraLie​⟨𝔥i⟩\\mathrm\{Lie\}\\langle\\mathfrak\{h\}\_\{i\}\\ranglecontains everyHiH\_\{i\}, soH⊆H′H\\subseteq H^\{\\prime\}, henceLie⁡\(H\)=Lie⁡\(H′\)\\mathrm\{Lie\}\(H\)=\\mathrm\{Lie\}\(H^\{\\prime\}\)andH=H′H=H^\{\\prime\}by uniqueness\. Companions:[Background D\.4](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1),[Background D\.8](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1),[Background D\.9](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\.

Used in\.[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(b\), \(c\) and corollary 4; the LOFT example of Exposé II\.B \(supports revisited across restarts\)\.

Auxiliary result B\.25\(first fundamental theorem forO⁡\(n\)\\mathrm\{O\}\(n\); Witt’s extension theorem\)\.\([Weyl, 1939](https://arxiv.org/html/2610.12728#bib.bib493), Ch\. II\);\([Goodman and Wallach, 2009](https://arxiv.org/html/2610.12728#bib.bib133), Ch\. 5\);\([Artin, 1957](https://arxiv.org/html/2610.12728#bib.bib16), Ch\. III\)\(a\)Every polynomial in the rowsw1,…,wm∈ℝnw\_\{1\},\\dots,w\_\{m\}\\in\\mathbb\{R\}^\{n\}ofWWthat is invariant underW↦W​RW\\mapsto WR,R∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\), is a polynomial in the inner products⟨wi,wj⟩\\langle w\_\{i\},w\_\{j\}\\rangle\. Since the invariant polynomials of a compact group acting linearly separate its orbits,W​W⊤WW^\{\\top\}is a complete invariant of the rightO⁡\(n\)\\mathrm\{O\}\(n\)\-action\.\(b\)\(Witt, Euclidean case\.\) A linear isometry between two subspaces ofℝn\\mathbb\{R\}^\{n\}extends to an element ofO⁡\(n\)\\mathrm\{O\}\(n\)\.

*Form used\.*Part \(b\) gives the orbit statement of \(a\) directly: ifW​W⊤=W′​W′⁣⊤WW^\{\\top\}=W^\{\\prime\}W^\{\\prime\\top\}, the map∑ici​wi↦∑ici​wi′\\sum\_\{i\}c\_\{i\}w\_\{i\}\\mapsto\\sum\_\{i\}c\_\{i\}w\_\{i\}^\{\\prime\}is a well\-defined isometry of the row spaces, and its extension isR⊤R^\{\\top\}withW′=W​RW^\{\\prime\}=WR\. Companion:[Background D\.16](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2)\.

Used in\.[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(1\), \(2\),[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)\(leftO⁡\(r\)\\mathrm\{O\}\(r\)\-orbits of full\-rankA0A\_\{0\}are classified byA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}\)\.

Auxiliary result B\.26\(the Kempf–Ness theorem in the real form used\)\.\([Kempf and Ness, 1979](https://arxiv.org/html/2610.12728#bib.bib209)\);\([Richardson and Slodowy, 1990](https://arxiv.org/html/2610.12728#bib.bib389)\);\([Birkes, 1971](https://arxiv.org/html/2610.12728#bib.bib38)\)LetVVbe a real inner\-product space,G⊆GL⁡\(V\)G\\subseteq\\mathrm\{GL\}\(V\)a real algebraic group that is closed under transpose,K=G∩O⁡\(V\)K=G\\cap\\mathrm\{O\}\(V\), and𝔭\\mathfrak\{p\}the symmetric elements of its Lie algebra\. Define the moment mapμ:V→𝔭\\mu:V\\to\\mathfrak\{p\}by⟨μ⁡\(v\),ξ⟩=12​dd​t\|t=0​∥et​ξ​v∥2=⟨ξ​v,v⟩\\langle\\mu\(v\),\\xi\\rangle=\\tfrac\{1\}\{2\}\\tfrac\{d\}\{dt\}\\big\|\_\{t=0\}\\lVert e^\{t\\xi\}v\\rVert^\{2\}=\\langle\\xi v,v\\rangle\. Then:\(i\)g=eg=eis a critical point ofg↦∥g⋅v∥2g\\mapsto\\lVert g\\cdot v\\rVert^\{2\}iffμ⁡\(v\)=0\\mu\(v\)=0, and then∥v∥≤∥g⋅v∥\\lVert v\\rVert\\leq\\lVert g\\cdot v\\rVertfor everyg∈Gg\\in G;\(ii\)G⋅vG\\cdot vis closed iff it meetsμ−1​\(0\)\\mu^\{\-1\}\(0\);\(iii\)ifG⋅vG\\cdot vis closed,G⋅v∩μ−1​\(0\)G\\cdot v\\cap\\mu^\{\-1\}\(0\)is a singleKK\-orbit;\(iv\)the closure of every orbit contains exactly one closed orbit\.

*Form used\.*ForGLr\\mathrm\{GL\}\_\{r\}acting onV=ℝm×r×ℝr×nV=\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}by\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)with the Frobenius inner product, the image is closed under transpose \(the adjoint of the action ofggis the action ofg⊤g^\{\\top\}\),K=O⁡\(r\)K=\\mathrm\{O\}\(r\), and12​dd​t\|0​\(∥B​e−t​ξ∥2\+∥et​ξ​A∥2\)=⟨ξ,A​A⊤−B⊤​B⟩\\tfrac\{1\}\{2\}\\tfrac\{d\}\{dt\}\\big\|\_\{0\}\(\\lVert Be^\{\-t\\xi\}\\rVert^\{2\}\+\\lVert e^\{t\\xi\}A\\rVert^\{2\}\)=\\langle\\xi,AA^\{\\top\}\-B^\{\\top\}B\\rangle\. So the moment map is−μ\-\\muwithμ=B⊤​B−A​A⊤\\mu=B^\{\\top\}B\-AA^\{\\top\}as in[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), its zero set is the balanced slice, a full\-rank orbit is closed \(it is a whole fibre of\(B,A\)↦B​A\(B,A\)\\mapsto BA\), and by[Auxiliary result B\.21](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)the least value of∥B∥2\+∥A∥2\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}on it is2​∥B​A∥∗2\\lVert BA\\rVert\_\{\*\}\. Companions:[Background D\.29](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1),[Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\.

Used in\.[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(a\),[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\(decay as a Kempf–Ness potential\),[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(a\) \(part \(iv\)\)\.

Auxiliary result B\.27\(closed orbits in the fibres of\(B,A\)↦B​A\(B,A\)\\mapsto BA\)\.\([Goodman and Wallach, 2009](https://arxiv.org/html/2610.12728#bib.bib133), Ch\. 5\);\([Mumford et al\., 1994](https://arxiv.org/html/2610.12728#bib.bib340), Ch\. 1\); part \(c\) elementary\(a\)\(First fundamental theorem forGLr\\mathrm\{GL\}\_\{r\}\.\) The polynomial invariants ofGLr​\(ℂ\)\\mathrm\{GL\}\_\{r\}\(\\mathbb\{C\}\)acting onℂm×r×ℂr×n\\mathbb\{C\}^\{m\\times r\}\\times\\mathbb\{C\}^\{r\\times n\}by\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)are generated by the entries ofB​ABA, the contractions of the rows ofBBwith the columns ofAA\.\(b\)\(Reductive quotients\.\) For a complex reductive group acting on an affine variety, every fibre of the quotient map contains exactly one closed orbit, and it lies in the closure of every orbit in that fibre\.\(c\)\(Real form used\.\) FixW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}of rankk≤rk\\leq r\. Among the real pairs withB​A=WBA=W, those withrank⁡B=rank⁡A=k\\rk B=\\rk A=kform a singleGLr\\mathrm\{GL\}\_\{r\}\-orbitOWO\_\{W\}\. It is closed and lies in the closure of every orbit in the fibre\{BA=W\}\\\{BA=W\\\}\. Hence a continuousGLr\\mathrm\{GL\}\_\{r\}\-invariant function of\(B,A\)\(B,A\)is constant on each fibre of\(B,A\)↦B​A\(B,A\)\\mapsto BA\.

*Reason for \(c\)\.*IfB​A=WBA=Wwithrank⁡B=rank⁡A=k\\rk B=\\rk A=k, thencol​B=col​W\\mathrm\{col\}\\,B=\\mathrm\{col\}\\,Wandrow​A=row​W\\mathrm\{row\}\\,A=\\mathrm\{row\}\\,W; a gauge bringsBBto\[U0​0\]\[U\_\{0\}\\ 0\]for a fixed basisU0U\_\{0\}ofcol​W\\mathrm\{col\}\\,W, which fixes the firstkkrows ofAA, and the stabiliser of\[U0​0\]\[U\_\{0\}\\ 0\]clears the remaining rows, which lie inrow​W\\mathrm\{row\}\\,W\. The setOW=\{B​A=W,rank⁡B≤k,rank⁡A≤k\}O\_\{W\}=\\\{BA=W,\\ \\rk B\\leq k,\\ \\rk A\\leq k\\\}is closed\. For a general pair in the fibre, splitℝr=C1⊕C2⊕C3⊕C4\\mathbb\{R\}^\{r\}=C\_\{1\}\\oplus C\_\{2\}\\oplus C\_\{3\}\\oplus C\_\{4\}withim​A=C1⊕C2\\mathrm\{im\}\\,A=C\_\{1\}\\oplus C\_\{2\},ker⁡B=C2⊕C3\\ker B=C\_\{2\}\\oplus C\_\{3\}anddimC1=k\\dim C\_\{1\}=k, and letgtg\_\{t\}act by11,tt,11,t−1t^\{\-1\}on the four summands\. Ast→0t\\to 0,\(B​gt−1,gt​A\)\(Bg\_\{t\}^\{\-1\},g\_\{t\}A\)converges to a pair in the fibre whose factors have rank at mostkk, so to a point ofOWO\_\{W\}\. Companion:[Background D\.31](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px2)\.

Used in\.[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)\(a\) \(a continuous combination rule that is invariant under each adapter’sGLr\\mathrm\{GL\}\_\{r\}is a function of the adapters, also off the full\-rank locus\)\.

#### Dynamics, optimisation and differentiation

Auxiliary result B\.28\(existence and uniqueness under Carathéodory conditions\)\.\([Hale, 1980](https://arxiv.org/html/2610.12728#bib.bib148), Ch\. I, Thms 5\.1 and 5\.3\)LetD⊆ℝN\+1D\\subseteq\\mathbb\{R\}^\{N\+1\}be open and letF:D→ℝNF:D\\to\\mathbb\{R\}^\{N\}be measurable inttfor each fixedxxand continuous inxxfor each fixedtt, with\|F⁡\(t,x\)\|≤mU​\(t\)\\lvert F\(t,x\)\\rvert\\leq m\_\{U\}\(t\)on each compactU⊆DU\\subseteq Dfor an integrablemUm\_\{U\}\. Then through every\(t0,x0\)∈D\(t\_\{0\},x\_\{0\}\)\\in Dthere is an absolutely continuous solution ofx˙=F⁡\(t,x\)\\dot\{x\}=F\(t,x\), holding for almost everytt\. If moreover\|F⁡\(t,x\)−F⁡\(t,y\)\|≤kU​\(t\)​\|x−y\|\\lvert F\(t,x\)\-F\(t,y\)\\rvert\\leq k\_\{U\}\(t\)\\lvert x\-y\\rverton each compactUUfor an integrablekUk\_\{U\}, the solution through each point is unique\.

*Form used\.*Ifγ\\gammais measurable inttand locally Lipschitz inθ\\theta, with locally integrable bounds and Lipschitz constants, andρ\\rhois of classCloc1,1C^\{1,1\}\_\{\\rm loc\}, thenF⁡\(t,q\)=−Λ​D​ρq⊤​γ​\(t,ρ⁡\(q\)\)F\(t,q\)=\-\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\\gamma\(t,\\rho\(q\)\)satisfies these conditions, so the trajectory fromk​q0kq\_\{0\}is unique\.

Used in\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(ii\) \(the trajectory statement\)\.

Auxiliary result B\.29\(Noether’s theorem for gradient flows\)\.\([Kunin et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib230)\);\([Zhao et al\., 2022b](https://arxiv.org/html/2610.12728#bib.bib580), Prop\. 5\.1\);\([Marcotte et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib318)\);\([Du et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib100)\)Let a Lie group act linearly onQ=ℝNQ=\\mathbb\{R\}^\{N\}withρ⁡\(k⋅q\)=ρ⁡\(q\)\\rho\(k\\cdot q\)=\\rho\(q\), and letΛ≻0\\Lambda\\succ 0be constant\. Along every solution ofq˙=−Λ​D​ρq⊤​γ​\(t\)\\dot\{q\}=\-\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\\gamma\(t\), whereγ⁡\(t\)\\gamma\(t\)is any cotangent signal, gradient or not, and for every generatorξ\\xiof the action withΛ−1​ξ\\Lambda^\{\-1\}\\xisymmetric, the chargeJξ​\(q\)=12​⟨q,Λ−1​ξ​q⟩J\_\{\\xi\}\(q\)=\\tfrac\{1\}\{2\}\\langle q,\\Lambda^\{\-1\}\\xi q\\rangleis constant\. For LoRA with block ratesηB,ηA\\eta\_\{B\},\\eta\_\{A\}and generatorsξX​\(B,A\)=\(B​X,−X​A\)\\xi\_\{X\}\(B,A\)=\(BX,\-XA\),XXsymmetric, the charges are the entries ofΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}\.

*Form used\.*The cited sources treat gradient flows,[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580)withΛ=I\\Lambda=I; their proof uses onlyD​ρq​\(ξ​q\)=0D\\rho\_\{q\}\(\\xi q\)=0, so it covers arbitrary signalsγ\\gammaand constantΛ\\Lambda, which is how[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(i\) states and proves it\. Companions:[Background D\.26](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4),[Background D\.28](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1)\.

Used in\.[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(i\),[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1),[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\(b\),[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\(the charge stays at−κ​I\-\\kappa I\)\.

Auxiliary result B\.30\(Riemannian gradients on embedded submanifolds\)\.\([Absil et al\., 2008](https://arxiv.org/html/2610.12728#bib.bib2), §3\.6\)LetN⊆ℝDN\\subseteq\\mathbb\{R\}^\{D\}be an embedded submanifold with the metric induced by the Euclidean \(Frobenius\) inner product, andffsmooth nearNN\. The Riemannian gradient off\|Nf\|\_\{N\}atxxisPTx​N∇f\(x\)P\_\{T\_\{x\}N\}\\nabla f\(x\), the orthogonal projection of the Euclidean gradient onto the tangent space, and the Riemannian gradient flow isx˙=−PTx​N∇f\(x\)\\dot\{x\}=\-P\_\{T\_\{x\}N\}\\nabla f\(x\)\.

*Form used\.*ForN=W0\+ℳrN=W\_\{0\}\+\\mathcal\{M\}\_\{r\}atW=W0\+XW=W\_\{0\}\+X, withU=col​XU=\\mathrm\{col\}\\,XandV=row​XV=\\mathrm\{row\}\\,X,[Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\) gives the projectionG↦PU​G\+G​PV−PU​G​PVG\\mapsto P\_\{U\}G\+GP\_\{V\}\-P\_\{U\}GP\_\{V\}\. Companion:[Background D\.24](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\.

Used in\.[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(b\) \(LoRA\-Pro’s equivalent gradient\)\.

Auxiliary result B\.31\(Frobenius’ theorem\)\.\([Lee, 2013](https://arxiv.org/html/2610.12728#bib.bib237), Prop\. 19\.3, Thms 19\.12 and 19\.21\)LetDDbe a smooth distribution of constant rankkkon an open setUU\.\(a\)IfNNis an integral manifold ofDD\(Tθ​N=DθT\_\{\\theta\}N=D\_\{\\theta\}forθ∈N\\theta\\in N\) andX,YX,Yare sections ofDD, then\[X,Y\]θ∈Dθ\[X,Y\]\_\{\\theta\}\\in D\_\{\\theta\}for everyθ∈N\\theta\\in N\. Hence an integrable distribution is involutive\.\(b\)\(Frobenius\.\) Every involutive distribution is completely integrable: each point has a chart in whichDDis spanned by the firstkkcoordinate fields\. In such a chart a curve withθ˙​\(t\)∈Dθ⁡\(t\)\\dot\{\\theta\}\(t\)\\in D\_\{\\theta\(t\)\}keeps its last coordinates constant, so it stays in one integral manifold\.\(c\)\(Global Frobenius\.\) The maximal connected integral manifolds of an involutive distribution form a foliation ofUU\.

Companion:[Background D\.25](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\.

Used in\.[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\.

Auxiliary result B\.32\(flatness and the Bures–Wasserstein metric\)\.\([Lee, 2018](https://arxiv.org/html/2610.12728#bib.bib238), Ch\. 7\);\([Takatsu, 2011](https://arxiv.org/html/2610.12728#bib.bib441)\);\([Bhatia et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib33)\)\(a\)Riemannian curvature is preserved by local isometries, and a Riemannian manifold is flat iff it is locally isometric to an open subset of a Euclidean space\.\(b\)On the positive definiten×nn\\times nmatrices, the Bures–Wasserstein metric is the quotient of the Euclidean \(Frobenius\) metric underB↦B​B⊤B\\mapsto BB^\{\\top\}, whose fibres are the orbitsB↦B​RB\\mapsto BR,R∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\); up to a constant factor fixed by normalisation,B↦B​B⊤B\\mapsto BB^\{\\top\}is a Riemannian submersion onto it\. Its sectional curvature is nonnegative, and forn≥2n\\geq 2it is positive on some planes, so the metric is not flat\.

*Form used\.*By \(a\), an immersionρ\\rhowithdimQ=rank⁡D\\dim Q=\\rk Dthat is a local isometry from the flat\(Q,Λ−1\)\(Q,\\Lambda^\{\-1\}\)onto a leaf forces the leaf metric to be flat\. The mapρ⁡\(B\)=B​B⊤\\rho\(B\)=BB^\{\\top\}has positive\-dimensional fibres, so it is no immersion; its cometricD​ρ​D​ρ⊤​\(G\)=2​\(G​θ\+θ​G\)D\\rho\\,D\\rho^\{\\top\}\(G\)=2\(G\\theta\+\\theta G\)atθ=B​B⊤\\theta=BB^\{\\top\}does not depend on the point of the fibre, and by \(b\) it is the cometric of the curved Bures–Wasserstein metric\.

Used in\.[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\(dynamic equivalence with a forward lens forces a flat leaf metric whendimQ=rank⁡D\\dim Q=\\rk D\)\.

Auxiliary result B\.33\(the cost of forward and reverse automatic differentiation\)\.\([Griewank and Walther, 2008](https://arxiv.org/html/2610.12728#bib.bib137), Ch\. 3 and 4\)LetL:ℝN→ℝL:\\mathbb\{R\}^\{N\}\\to\\mathbb\{R\}be evaluated by a straight\-line program\.\(a\)Forward mode evaluatesL⁡\(θ\)L\(\\theta\)together with one directional derivative,D​L​\(θ\)​\[p\]DL\(\\theta\)\[p\], which equals⟨∇L​\(θ\),p⟩\\langle\\nabla L\(\\theta\),p\\rangle, at a cost bounded by a small constant multiple of the cost of evaluatingLL\. It carries each tangent alongside its value and stores no trace\.\(b\)Reverse mode evaluates∇L​\(θ\)\\nabla L\(\\theta\)at a cost bounded by a small constant multiple of the cost of evaluatingLL, independent ofNN, but needs the intermediate values of the evaluation, stored or recomputed, during the reverse sweep\.

*Form used\.*Thekkcoordinates⟨pi,∇L​\(θ\)⟩\\langle p\_\{i\},\\nabla L\(\\theta\)\\rangleofP⊤∇LP^\{\\top\}\\nabla Lcostkkforward passes and no reverse pass;∇L\\nabla Litself by forward mode would costNNof them\.

Used in\.[Observation IV\.4](https://arxiv.org/html/2610.12728#S4.SS0.SSSx5)\.

Auxiliary result B\.34\(central differences and Gaussian smoothing\)\.\([Nesterov and Spokoiny, 2017](https://arxiv.org/html/2610.12728#bib.bib345)\); part \(a\) by Taylor’s theoremLetL:ℝN→ℝL:\\mathbb\{R\}^\{N\}\\to\\mathbb\{R\}be twice differentiable with anℓ2\\ell\_\{2\}\-Lipschitz Hessian \(constantL2L\_\{2\}\),ε\>0\\varepsilon\>0andz∈ℝNz\\in\\mathbb\{R\}^\{N\}\.\(a\)\|L⁡\(θ\+ε​z\)−L⁡\(θ−ε​z\)2​ε−⟨∇L​\(θ\),z⟩\|≤L26​ε2​∥z∥3\\Big\\lvert\\dfrac\{L\(\\theta\+\\varepsilon z\)\-L\(\\theta\-\\varepsilon z\)\}\{2\\varepsilon\}\-\\langle\\nabla L\(\\theta\),z\\rangle\\Big\\rvert\\leq\\dfrac\{L\_\{2\}\}\{6\}\\,\\varepsilon^\{2\}\\lVert z\\rVert^\{3\}\.\(b\)Forz∼𝒩⁡\(0,IN\)z\\sim\\mathcal\{N\}\(0,I\_\{N\}\),𝔼⁡\[⟨∇L​\(θ\),z⟩​z\]=∇L​\(θ\)\\mathbb\{E\}\\big\[\\langle\\nabla L\(\\theta\),z\\rangle z\\big\]=\\nabla L\(\\theta\), and for LipschitzLL,𝔼⁡\[12​ε​\(L⁡\(θ\+ε​z\)−L⁡\(θ−ε​z\)\)​z\]=∇Lε​\(θ\)\\mathbb\{E\}\\big\[\\tfrac\{1\}\{2\\varepsilon\}\(L\(\\theta\+\\varepsilon z\)\-L\(\\theta\-\\varepsilon z\)\)\\,z\\big\]=\\nabla L\_\{\\varepsilon\}\(\\theta\), the gradient of the Gaussian smoothingLε​\(θ\)=𝔼u​L​\(θ\+ε​u\)L\_\{\\varepsilon\}\(\\theta\)=\\mathbb\{E\}\_\{u\}L\(\\theta\+\\varepsilon u\)\.\(c\)Hence the central two\-point estimator is unbiased for∇Lε\\nabla L\_\{\\varepsilon\}, and by \(a\) its bias relative to∇L\\nabla Lis at mostL26​ε2​𝔼​∥z∥4=L26​ε2​N​\(N\+2\)\\tfrac\{L\_\{2\}\}\{6\}\\varepsilon^\{2\}\\,\\mathbb\{E\}\\lVert z\\rVert^\{4\}=\\tfrac\{L\_\{2\}\}\{6\}\\varepsilon^\{2\}N\(N\+2\)\.

Used in\.[Observation IV\.4](https://arxiv.org/html/2610.12728#S4.SS0.SSSx5)\(MeZO’s estimator\),[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\(instances: each MeZO step is a step of a linear method up to its estimator\)\. The proof of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

Auxiliary result B\.35\(extreme eigenvalues of Wishart\-type matrices\)\.\([Marčenko and Pastur, 1967](https://arxiv.org/html/2610.12728#bib.bib317)\);\([Yin et al\., 1988](https://arxiv.org/html/2610.12728#bib.bib537)\);\([Bai and Yin, 1993](https://arxiv.org/html/2610.12728#bib.bib24)\)LetA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}have independent, identically distributed entries with mean00, variancec/nc/nand finite fourth moment\. Asr,n→∞r,n\\to\\inftywithr/n→y∈\(0,1\]r/n\\to y\\in\(0,1\], the empirical eigenvalue distribution ofA​A⊤AA^\{\\top\}converges to the Marčenko–Pastur law supported on\[c​\(1−y\)2,c​\(1\+y\)2\]\[c\(1\-\\sqrt\{y\}\)^\{2\},c\(1\+\\sqrt\{y\}\)^\{2\}\], and the largest and smallest eigenvalues converge almost surely toc​\(1\+y\)2c\(1\+\\sqrt\{y\}\)^\{2\}andc​\(1−y\)2c\(1\-\\sqrt\{y\}\)^\{2\}\.

*Form used\.*HF PEFT’s Kaiming\-uniformA0A\_\{0\}has entriesU\(−n−1/2,n−1/2\)U\(\-n^\{\-1/2\},n^\{\-1/2\}\), soc=1/3c=1/3, and the eigenvalues ofA0​A0⊤A\_\{0\}A\_\{0\}^\{\\top\}fill a band of relative widthO⁡\(r/n\)O\(\\sqrt\{r/n\}\)aroundcc\. No proof depends on this entry\.

Used in\.the commentary on[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\(exactness for Kaiming initialisations\), Prediction X\.7\.

#### Elementary facts

Auxiliary result B\.36\(three elementary facts\)\.elementary; part \(c\) in[Horn and Johnson, 2013](https://arxiv.org/html/2610.12728#bib.bib174), Ch\. 1\(a\)Ifσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}is differentiable at00andσ⁡\(ℓ​t\)=ℓ​σ​\(t\)\\sigma\(\\ell t\)=\\ell\\,\\sigma\(t\)for allℓ≥0\\ell\\geq 0andt∈ℝt\\in\\mathbb\{R\}, thenσ\\sigmais linear\.\(b\)A real vector space is not a finite union of proper subspaces\. In particular, a hyperplane ofℝd\\mathbb\{R\}^\{d\}that is contained in a finite union of coordinate hyperplanes is one of them\.\(c\)ForX∈ℝp×qX\\in\\mathbb\{R\}^\{p\\times q\}andY∈ℝq×pY\\in\\mathbb\{R\}^\{q\\times p\},X​YXYandY​XYXhave the same nonzero eigenvalues, with multiplicities\.

*Reasons\.*\(a\) Takingℓ=0\\ell=0givesσ⁡\(0\)=0\\sigma\(0\)=0, andσ⁡\(t\)=t​σ​\(1\)\\sigma\(t\)=t\\,\\sigma\(1\)fort\>0t\>0andσ⁡\(t\)=−t​σ​\(−1\)\\sigma\(t\)=\-t\\,\\sigma\(\-1\)fort<0t<0; the one\-sided derivatives at00areσ⁡\(1\)\\sigma\(1\)and−σ⁡\(−1\)\-\\sigma\(\-1\), so differentiability forcesσ⁡\(t\)=t​σ​\(1\)\\sigma\(t\)=t\\,\\sigma\(1\)\. ReLU and leaky ReLU satisfy the identity without being linear\. \(b\) IfV=V1∪⋯∪VkV=V\_\{1\}\\cup\\dots\\cup V\_\{k\}withkkminimal, pickx∈V1x\\in V\_\{1\}outside the otherViV\_\{i\}andy∉V1y\\notin V\_\{1\}\. No pointy\+t​xy\+txlies inV1V\_\{1\}, so two of them,t≠t′t\\neq t^\{\\prime\}, lie in oneViV\_\{i\}withi≥2i\\geq 2, and thenx∈Vix\\in V\_\{i\}, a contradiction\. The second claim applies the first toH=⋃j\(H∩\{xj=0\}\)H=\\bigcup\_\{j\}\(H\\cap\\\{x\_\{j\}=0\\\}\)\.

Used in\.[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(rule R3: positively homogeneous activations\),[Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\(b\) \(the kink argument for ReLU\),[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)\(the nonzero eigenvalues ofT=β​β⊤T=\\beta\\beta^\{\\top\}are those ofβ⊤​β\\beta^\{\\top\}\\beta\)\. The proof of[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)in Appendix[C](https://arxiv.org/html/2610.12728#A3)also cites it\.

#### Spectral projectors, Riemannian submanifolds and floating point

Auxiliary result B\.37\(smooth spectral projectors\)\.\([Kato, 1995](https://arxiv.org/html/2610.12728#bib.bib208), Ch\. I, §5 and Ch\. II, §1\);\([Bhatia, 1997](https://arxiv.org/html/2610.12728#bib.bib31), Ch\. III\)LetS:V→Sym⁡\(m\)S:V\\to\\mathrm\{Sym\}\(m\)be smooth on an open setVVof a real vector space, with eigenvaluesλ1​\(W\)≥⋯≥λm​\(W\)\\lambda\_\{1\}\(W\)\\geq\\dots\\geq\\lambda\_\{m\}\(W\), and letλr​\(W∗\)\>λr\+1​\(W∗\)\\lambda\_\{r\}\(W\_\{\*\}\)\>\\lambda\_\{r\+1\}\(W\_\{\*\}\)at someW∗∈VW\_\{\*\}\\in V\. Then the gapλr\>λr\+1\\lambda\_\{r\}\>\\lambda\_\{r\+1\}persists on a neighbourhoodV∗V\_\{\*\}ofW∗W\_\{\*\}, and onV∗V\_\{\*\}the orthogonal projectorΠ⁡\(W\)\\Pi\(W\)onto the sum of the eigenspaces ofλ1​\(W\),…,λr​\(W\)\\lambda\_\{1\}\(W\),\\dots,\\lambda\_\{r\}\(W\)is the Riesz projectorΠ⁡\(W\)=12​π​i​∮Γ\(z−S⁡\(W\)\)−1​𝑑z\\Pi\(W\)=\\frac\{1\}\{2\\pi i\}\\oint\_\{\\Gamma\}\\big\(z\-S\(W\)\\big\)^\{\-1\}\\,dzfor a fixed contourΓ\\Gammainℂ\\mathbb\{C\}that enclosesλ1​\(W∗\),…,λr​\(W∗\)\\lambda\_\{1\}\(W\_\{\*\}\),\\dots,\\lambda\_\{r\}\(W\_\{\*\}\)and no other eigenvalue ofS⁡\(W∗\)S\(W\_\{\*\}\)\. In particularΠ\\Piis smooth onV∗V\_\{\*\}\.

*Reason\.*By Weyl’s perturbation inequality the eigenvalues are11\-Lipschitz in the operator norm, so nearW∗W\_\{\*\}the contour still separates the toprreigenvalues andz−S⁡\(W\)z\-S\(W\)is invertible onΓ\\Gamma\. Matrix inversion is smooth, so the integral is smooth inWW, and at eachWWit is the orthogonal projector onto the enclosed eigenspaces\.

Used in\.[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\(step 9: the GaLore fieldΠ⁡\(W\)\\Pi\(W\)is smooth where a singular\-value gap exists\)\.

Auxiliary result B\.38\(isometries, totally geodesic submanifolds and the Gauss equation\)\.\([Lee, 2018](https://arxiv.org/html/2610.12728#bib.bib238), Ch\. 4, 5 and 8\);\([Kobayashi, 1958](https://arxiv.org/html/2610.12728#bib.bib219)\)Let\(M,g\)\(M,g\)be a Riemannian manifold\.\(a\)A geodesic is determined by its initial point and initial velocity, and an isometry of\(M,g\)\(M,g\)maps geodesics to geodesics\.\(b\)LetF⊆MF\\subseteq Mbe an embedded submanifold with the induced metric\. If every geodesic ofMMwith initial velocity tangent toFFstays inFFfor small times \(FFis*totally geodesic*\), the second fundamental form ofFFvanishes, and by the Gauss equation the sectional curvature ofFFon every22\-planeΠ⊆Tx​F\\Pi\\subseteq T\_\{x\}Fequals the sectional curvature ofMMonΠ\\Pi\.

*Form used\.*Letτ\\taube an isometry andFFan embedded submanifold, open in its fixed\-point set, withd​τ=idd\\tau=\\mathrm\{id\}onT​FTF\. By \(a\) a geodesicγ\\gammatangent toFFandτ∘γ\\tau\\circ\\gammahave the same initial data, soτ∘γ=γ\\tau\\circ\\gamma=\\gammaandγ\\gammastays inFFfor small times\. SoFFis totally geodesic, and \(b\) applies\.

Used in\.[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\(step 6: balanced full\-rank LoRA induces a curved leaf metric\)\.

Auxiliary result B\.39\(exact scaling by powers of two in floating point\)\.\([Higham, 2002](https://arxiv.org/html/2610.12728#bib.bib171), Ch\. 2\)In binary floating\-point arithmetic with correctly rounded operations \(IEEE 754, rounding to nearest\), letc=2kc=2^\{k\}withk∈ℤk\\in\\mathbb\{Z\}\. If all operands, exact results and rounded results below stay in the normal range \(no overflow and no underflow\), thenfl⁡\(c​x\)=c​x\\mathrm\{fl\}\(cx\)=cx, and for floating\-point numbersx,yx,y,fl⁡\(c​x±c​y\)=c​fl​\(x±y\),fl⁡\(c​x⋅y\)=c​fl​\(x​y\),fl⁡\(c​x/y\)=c​fl​\(x/y\),fl⁡\(c2​x\)=c​fl​\(x\)\.\\mathrm\{fl\}\(cx\\pm cy\)=c\\,\\mathrm\{fl\}\(x\\pm y\),\\quad\\mathrm\{fl\}\(cx\\cdot y\)=c\\,\\mathrm\{fl\}\(xy\),\\quad\\mathrm\{fl\}\(cx/y\)=c\\,\\mathrm\{fl\}\(x/y\),\\quad\\mathrm\{fl\}\\big\(\\sqrt\{c^\{2\}x\}\\big\)=c\\,\\mathrm\{fl\}\\big\(\\sqrt\{x\}\\big\)\.

*Reason\.*In the normal range, multiplication by2k2^\{k\}only shifts the exponent, so it maps the set of floating\-point numbers onto itself and scales all distances between them bycc\. Rounding to the nearest floating\-point number, with its tie\-breaking rule, therefore commutes with it, and each identity follows fromfl⁡\(c​z\)=c​fl​\(z\)\\mathrm\{fl\}\(cz\)=c\\,\\mathrm\{fl\}\(z\)applied to the exact resultzz\.

Used in\.[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)\(bit\-exact trajectories when every block factor is a power of two\)\.

## Appendix CDeferred proofs

This appendix proves the numbered results of the main text in the order in which they appear there, one subsection per result\. Each subsection first restates the result in its final form, the form recorded after the two rounds of review, and then proves it\. External theorems enter through Appendix[B](https://arxiv.org/html/2610.12728#A2), cited as Auxiliary results B\.1–B\.39, and background notions through Appendix[A](https://arxiv.org/html/2610.12728#A1)\. Exposé[0](https://arxiv.org/html/2610.12728#S0)states its headline results without numbers; each of them is proved below or in the later parts of this appendix under its number\. Definitions I\.1 and I\.2 need no proof, since𝐏𝐚𝐫𝐚⁡\(𝒞\)\\mathbf\{Para\}\(\\mathcal\{C\}\)is a bicategory\([Fong et al\., 2017](https://arxiv.org/html/2610.12728#bib.bib114);[Cruttwell et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib85)\)and𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)is a slice of a coslice; Remark I\.3 and the two definitions that contain a claim, Definitions I\.5 and II\.14, receive short proofs\. Unless we say otherwise, parameter spaces are open definable subsets of a vector space and the mapsρ\\rhoare semialgebraic or definable inℝexp\\mathbb\{R\}\_\{\\exp\}\([Background C\.17](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\), dimensions are those of[Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2), “generic” has the meaning of[Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2), and LoRA on onem×nm\\times nbox isρ⁡\(B,A\)=Wres\+s​B​A\\rho\(B,A\)=W\_\{\\mathrm\{res\}\}\+sBAwiths≠0s\\neq 0,B∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}andA∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}\. For subspacesV⊆ℝnV\\subseteq\\mathbb\{R\}^\{n\}andU⊆ℝmU\\subseteq\\mathbb\{R\}^\{m\}we writeℛ⁡\(V\)=\{Z∈ℝm×n:row⁡Z⊆V\}\\mathcal\{R\}\(V\)=\\\{Z\\in\\mathbb\{R\}^\{m\\times n\}:\\operatorname\{row\}Z\\subseteq V\\\}and𝒞⁡\(U\)=\{Z:col⁡Z⊆U\}\\mathcal\{C\}\(U\)=\\\{Z:\\operatorname\{col\}Z\\subseteq U\\\}, andx∧y=x​y⊤−y​x⊤x\\wedge y=xy^\{\\top\}\-yx^\{\\top\}\. Values marked \[Num\] in a statement are checked numerically and not proved; Appendix[E](https://arxiv.org/html/2610.12728#A5)documents them\.

### C\.1 Remark I\.3

Remark I\.3\(orientation, and what Para adds\)\.Eachρ\\rhois a 2\-cellF⇒Fρ:=\(Q,f∘\(ρ×X\)\)F\\Rightarrow F^\{\\rho\}:=\(Q,f\\circ\(\\rho\\times X\)\)*out of*FF\. A simulationh:M→Nh:M\\to Nis a 2\-cellFρN⇒FρMF^\{\\rho\_\{N\}\}\\Rightarrow F^\{\\rho\_\{M\}\}, sinceρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\. Hence𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)is the opposite of the pointed coslice of reparametrisation 2\-cells underFF\. In particular𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)depends only on\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)and not onff\.

Let𝒫\\mathcal\{P\}be the category whose objects are the 1\-cellsX→YX\\to Yof𝐏𝐚𝐫𝐚⁡\(𝐃𝐢𝐟𝐟\)\\mathbf\{Para\}\(\\mathbf\{Diff\}\)and whose arrows are the 2\-cells, composed vertically\. By[Definition I\.1](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)a 2\-cell\(P,f\)⇒\(Q,g\)\(P,f\)\\Rightarrow\(Q,g\)is a smooth mapr:Q→Pr:Q\\to Pwithg=f∘\(r×X\)g=f\\circ\(r\\times X\)\. The vertical composite ofr:\(P,f\)⇒\(Q,g\)r:\(P,f\)\\Rightarrow\(Q,g\)andr′:\(Q,g\)⇒\(Q′,g′\)r^\{\\prime\}:\(Q,g\)\\Rightarrow\(Q^\{\\prime\},g^\{\\prime\}\)isr∘r′:Q′→Pr\\circ r^\{\\prime\}:Q^\{\\prime\}\\to P, a 2\-cell becauseg′=g∘\(r′×X\)=f∘\(\(r∘r′\)×X\)g^\{\\prime\}=g\\circ\(r^\{\\prime\}\\times X\)=f\\circ\(\(r\\circ r^\{\\prime\}\)\\times X\)\.

An object of the cosliceF/𝒫F/\\mathcal\{P\}is a 2\-cell out ofF=\(Θ,f\)F=\(\\Theta,f\)\. A 2\-cell fromFFto\(Q,g\)\(Q,g\)is a smoothρ:Q→Θ\\rho:Q\\to\\Thetawithg=f∘\(ρ×X\)g=f\\circ\(\\rho\\times X\), so its target is forced to beFρF^\{\\rho\}and the object is determined by the pair\(Q,ρ\)\(Q,\\rho\)\. An arrow fromρM:F⇒FρM\\rho\_\{M\}:F\\Rightarrow F^\{\\rho\_\{M\}\}toρN:F⇒FρN\\rho\_\{N\}:F\\Rightarrow F^\{\\rho\_\{N\}\}is a 2\-cellk:FρM⇒FρNk:F^\{\\rho\_\{M\}\}\\Rightarrow F^\{\\rho\_\{N\}\}whose vertical composite withρM\\rho\_\{M\}isρN\\rho\_\{N\}\. Concretely, it is a smooth mapk:QN→QMk:Q\_\{N\}\\to Q\_\{M\}with

f∘\(ρN×X\)=f∘\(ρM×X\)∘\(k×X\)andρM∘k=ρN\.f\\circ\(\\rho\_\{N\}\\times X\)=f\\circ\(\\rho\_\{M\}\\times X\)\\circ\(k\\times X\)\\qquad\\text\{and\}\\qquad\\rho\_\{M\}\\circ k=\\rho\_\{N\}\.The second condition implies the first\. So the arrows\(QM,ρM\)→\(QN,ρN\)\(Q\_\{M\},\\rho\_\{M\}\)\\to\(Q\_\{N\},\\rho\_\{N\}\)ofF/𝒫F/\\mathcal\{P\}are exactly the smooth mapsk:QN→QMk:Q\_\{N\}\\to Q\_\{M\}overΘ\\Theta, which are the arrowsN→MN\\to Mof𝐃𝐢𝐟𝐟/Θ\\mathbf\{Diff\}/\\Theta, and composition is reversed\. Restricting to pointed parameter spaces withρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}and to pointedkkgives the pointed coslice, which is therefore𝐌𝐞𝐭𝐡​\(Θ,θ0\)op\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)^\{\\mathrm\{op\}\}\. The networkffentered only through the first condition, which is implied by the second for everyff, so𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)depends only on\(Θ,θ0\)\(\\Theta,\\theta\_\{0\}\)\. ∎

### C\.2 Observation I\.4

Observation I\.4\(universal properties\)\.In𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\):\(a\) the frozen model\(pt,∗,θ0\)\(\\mathrm\{pt\},\*,\\theta\_\{0\}\)is initial, and full fine\-tuning\(Θ,θ0,id\)\(\\Theta,\\theta\_\{0\},\\mathrm\{id\}\)is terminal\. The unique morphismM→M\\to\{\}Full FT isρM\\rho\_\{M\}itself\.\(b\) If the fibre productQM×ΘQNQ\_\{M\}\\times\_\{\\Theta\}Q\_\{N\}exists in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, it is the productM×NM\\times N\. It exists, for instance, when the set\-level fibre product is an embedded submanifold ofQM×QNQ\_\{M\}\\times Q\_\{N\}: a clean intersection, or linear or affineρ\\rho’s as in[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)\. TransversalityρM⋔ρN\\rho\_\{M\}\\pitchfork\\rho\_\{N\}is never the reason when\|M\|\+\|N\|<dimΘ\\lvert M\\rvert\+\\lvert N\\rvert<\\dim\\Theta, since at\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\)it would force\|M\|\+\|N\|≥dimΘ\\lvert M\\rvert\+\\lvert N\\rvert\\geq\\dim\\Theta\. In𝐒𝐞𝐭\\mathbf\{Set\}the fibre product always exists andIm⁡\(M×N\)=Im⁡M∩Im⁡N\\Imop\(M\\times N\)=\\Imop M\\cap\\Imop N\. Whenever the product exists in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, its underlying set is this set\-level fibre product, so the image identity holds in every case\. The product can fail to exist\. OnΘ=ℝ\\Theta=\\mathbb\{R\}\(1×11\\times 1matrices\),LoRA1×Frozen\\mathrm\{LoRA\}\_\{1\}\\times\\text\{Frozen\}would have the cross\{BA=0\}⊂ℝ2\\\{BA=0\\\}\\subset\\mathbb\{R\}^\{2\}as underlying set\. No manifold structure on the cross has the universal property, because both axes would have to lift smoothly through the point over the origin, which forces rank22there\.\(c\)Im\\Imopis a functor to the poset of subsets ofΘ\\Thetacontainingθ0\\theta\_\{0\}\. Conversely,Im⁡M⊆Im⁡N\\Imop M\\subseteq\\Imop Nyields a morphism of the set\-level slice𝐒𝐞𝐭∗/\(Θ,θ0\)\\mathbf\{Set\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\), though not necessarily a smooth one\. TakeρM​\(t\)=t\\rho\_\{M\}\(t\)=tandρN​\(u\)=u3\\rho\_\{N\}\(u\)=u^\{3\}onℝ\\mathbb\{R\}, pointed at00\. The images agree, yet no smoothh:ℝ→ℝh:\\mathbb\{R\}\\to\\mathbb\{R\}satisfiesρN∘h=id\\rho\_\{N\}\\circ h=\\mathrm\{id\}, that is,h​\(t\)3=th\(t\)^\{3\}=t\.

Recall that𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)=𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)=\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\)\.

*\(a\)*LetM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\)\. An arrow from the frozen model toMMis a pointed smooth maph:pt→Qh:\\mathrm\{pt\}\\to Qwithρ∘h=θ0\\rho\\circ h=\\theta\_\{0\}\. Pointedness forcesh⁡\(∗\)=q0h\(\*\)=q\_\{0\}, and the condition overΘ\\Thetathen holds becauseρ⁡\(q0\)=θ0\\rho\(q\_\{0\}\)=\\theta\_\{0\}\. So there is exactly one such arrow\. An arrowh:M→\(Θ,θ0,id\)h:M\\to\(\\Theta,\\theta\_\{0\},\\mathrm\{id\}\)is a pointed smooth maph:Q→Θh:Q\\to\\Thetawithid∘h=ρ\\mathrm\{id\}\\circ h=\\rho\. The only candidate ish=ρh=\\rho, which is smooth and pointed\.

*\(b\)*By[Auxiliary result B\.3](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(a\), a product ofMMandNNin the slice𝐃𝐢𝐟𝐟∗/\(Θ,θ0\)\\mathbf\{Diff\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)is the same as a pullback ofρM\\rho\_\{M\}andρN\\rho\_\{N\}in𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}\. Suppose the pullbackPPof the underlying maps exists in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, with projectionsπM,πN\\pi\_\{M\},\\pi\_\{N\}\. The constant mapspt→QM\\mathrm\{pt\}\\to Q\_\{M\}andpt→QN\\mathrm\{pt\}\\to Q\_\{N\}atq0q\_\{0\}andq0′q\_\{0\}^\{\\prime\}agree overΘ\\Theta, both composing toθ0\\theta\_\{0\}, so they factor through a unique pointp0∈Pp\_\{0\}\\in P\. With base pointp0p\_\{0\},PPis the pullback in𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}\. For a pointed test objectTTand pointed mapshM,hNh\_\{M\},h\_\{N\}that agree overΘ\\Theta, the induced mapT→PT\\to Psends the base point to a point whose projections areq0q\_\{0\}andq0′q\_\{0\}^\{\\prime\}, and that point isp0p\_\{0\}by uniqueness\. This is the creation of limits by𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}\([Auxiliary result B\.3](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\)\.

If the set\-level fibre productS=\{\(q,q′\):ρM​\(q\)=ρN​\(q′\)\}S=\\\{\(q,q^\{\\prime\}\):\\rho\_\{M\}\(q\)=\\rho\_\{N\}\(q^\{\\prime\}\)\\\}is an embedded submanifold ofQM×QNQ\_\{M\}\\times Q\_\{N\}, thenSSwith its two projections is the pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\([Auxiliary result B\.2](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(c\)\)\. Indeed, two smooth mapsT→QMT\\to Q\_\{M\}andT→QNT\\to Q\_\{N\}that agree overΘ\\Thetagive a smooth mapT→QM×QNT\\to Q\_\{M\}\\times Q\_\{N\}with image inSS, which is smooth intoSS, and the factorisation is unique becauseS→QM×QNS\\to Q\_\{M\}\\times Q\_\{N\}is injective\. A clean intersection has, by definition, an embedded submanifoldSS\([Background C\.7](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1.Px1)\), and for affineρM\\rho\_\{M\}andρN\\rho\_\{N\}on vector spacesSSis an affine subspace\.

Transversality at\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\)asksd​ρM​\(Tq0​QM\)\+d​ρN​\(Tq0′​QN\)=Tθ0​Θd\\rho\_\{M\}\(T\_\{q\_\{0\}\}Q\_\{M\}\)\+d\\rho\_\{N\}\(T\_\{q\_\{0\}^\{\\prime\}\}Q\_\{N\}\)=T\_\{\\theta\_\{0\}\}\\Theta\. The left side has dimension at most\|M\|\+\|N\|\\lvert M\\rvert\+\\lvert N\\rvert, so transversality fails at the base point whenever\|M\|\+\|N\|<dimΘ\\lvert M\\rvert\+\\lvert N\\rvert<\\dim\\Theta\([Auxiliary result B\.2](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\)\.

In𝐒𝐞𝐭\\mathbf\{Set\}the fibre product isSS, and its image inΘ\\ThetaisIm⁡M∩Im⁡N\\Imop M\\cap\\Imop N: a pair\(q,q′\)∈S\(q,q^\{\\prime\}\)\\in Slies overρM​\(q\)=ρN​\(q′\)\\rho\_\{M\}\(q\)=\\rho\_\{N\}\(q^\{\\prime\}\), which belongs to both images, and a pointθ\\thetaof both images has preimagesqqandq′q^\{\\prime\}with\(q,q′\)∈S\(q,q^\{\\prime\}\)\\in S\. Now suppose that a productPPexists in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}, and test it with the pointed00\-manifoldT=\{∗,t\}T=\\\{\*,t\\\}, pointed at∗\*\. A pointed mapT→PT\\to Pis the choice of the image oftt, that is, a point ofPP\. A pair of pointed mapsT→QMT\\to Q\_\{M\}andT→QNT\\to Q\_\{N\}that agree overΘ\\Thetais a point\(q,q′\)\(q,q^\{\\prime\}\)ofSS\. The universal property makes these correspond bijectively throughp↦\(πM​\(p\),πN​\(p\)\)p\\mapsto\(\\pi\_\{M\}\(p\),\\pi\_\{N\}\(p\)\)\. So the canonical mapP→QM×QNP\\to Q\_\{M\}\\times Q\_\{N\}is a bijection ontoSS, andIm⁡\(M×N\)=ρM​\(πM​\(P\)\)=Im⁡M∩Im⁡N\\Imop\(M\\times N\)=\\rho\_\{M\}\(\\pi\_\{M\}\(P\)\)=\\Imop M\\cap\\Imop N\.

TakeΘ=ℝ\\Theta=\\mathbb\{R\}andLoRA1:\(B,A\)↦θ0\+s​B​A\\mathrm\{LoRA\}\_\{1\}:\(B,A\)\\mapsto\\theta\_\{0\}\+sBAonℝ2\\mathbb\{R\}^\{2\}, withs≠0s\\neq 0and frozen residualθ0\\theta\_\{0\}, so that its pointings are the\(B0,A0\)\(B\_\{0\},A\_\{0\}\)withB0​A0=0B\_\{0\}A\_\{0\}=0\. Its set\-level fibre product with the frozen model is the cross𝒳=\{\(B,A\):B​A=0\}\\mathcal\{X\}=\\\{\(B,A\):BA=0\\\}\. Suppose a productPPexists, and letι:P→ℝ2\\iota:P\\to\\mathbb\{R\}^\{2\}be its projection to the parameters ofLoRA1\\mathrm\{LoRA\}\_\{1\}\. The forgetful functor𝐃𝐢𝐟𝐟∗→𝐃𝐢𝐟𝐟\\mathbf\{Diff\}\_\{\*\}\\to\\mathbf\{Diff\}has a left adjoint, so it preserves this pullback \([Auxiliary result B\.3](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(c\)\), andPPis also the pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}ofρLoRA1\\rho\_\{\\mathrm\{LoRA\}\_\{1\}\}and the inclusion ofθ0\\theta\_\{0\}\. Hence every smooth curvec:ℝ→ℝ2c:\\mathbb\{R\}\\to\\mathbb\{R\}^\{2\}withρ∘c≡θ0\\rho\\circ c\\equiv\\theta\_\{0\}has a unique smooth liftc~:ℝ→P\\tilde\{c\}:\\mathbb\{R\}\\to Pwithι∘c~=c\\iota\\circ\\tilde\{c\}=c, and by the previous paragraphι\\iotais a bijection onto𝒳\\mathcal\{X\}\. Letp∗p^\{\*\}be the point ofPPover\(0,0\)\(0,0\)\. The axis curvesc1​\(t\)=\(t,0\)c\_\{1\}\(t\)=\(t,0\)andc2​\(t\)=\(0,t\)c\_\{2\}\(t\)=\(0,t\)lie in𝒳\\mathcal\{X\}, so they lift to smooth curves throughp∗p^\{\*\}, andd​ιp∗d\\iota\_\{p^\{\*\}\}maps their velocities toc1′​\(0\)=\(1,0\)c\_\{1\}^\{\\prime\}\(0\)=\(1,0\)andc2′​\(0\)=\(0,1\)c\_\{2\}^\{\\prime\}\(0\)=\(0,1\)\. Hencerank⁡d​ιp∗=2\\rk d\\iota\_\{p^\{\*\}\}=2\. The rank cannot exceed22and cannot drop nearp∗p^\{\*\}\([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(a\)\), soι\\iotais a submersion on a neighbourhoodVVofp∗p^\{\*\}, andι⁡\(V\)\\iota\(V\)contains a nonempty open subset ofℝ2\\mathbb\{R\}^\{2\}\([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(c\)\)\. Butι⁡\(V\)⊆𝒳\\iota\(V\)\\subseteq\\mathcal\{X\}, which contains no nonempty open set\. So no product exists\. The argument used only the point over the origin, so it covers every pointing, including the standard one\(0,a0\)\(0,a\_\{0\}\)witha0≠0a\_\{0\}\\neq 0, whose base point is a smooth point of𝒳\\mathcal\{X\}\. \(A split pointing, withB0​A0≠0B\_\{0\}A\_\{0\}\\neq 0and residualθ0−s​B0​A0\\theta\_\{0\}\-sB\_\{0\}A\_\{0\}, has a hyperbola overθ0\\theta\_\{0\}, an embedded submanifold, and there the product exists\.\)

*\(c\)*Ifh:M→Nh:M\\to Nis an arrow, thenρM=ρN∘h\\rho\_\{M\}=\\rho\_\{N\}\\circ hgivesIm⁡M=ρN​\(h⁡\(QM\)\)⊆Im⁡N\\Imop M=\\rho\_\{N\}\(h\(Q\_\{M\}\)\)\\subseteq\\Imop N, and both images containθ0\\theta\_\{0\}; functoriality is automatic because the target is a poset\. Conversely, letIm⁡M⊆Im⁡N\\Imop M\\subseteq\\Imop N\. Choose, by the axiom of choice, a preimages⁡\(θ\)∈ρN−1​\(θ\)s\(\\theta\)\\in\\rho\_\{N\}^\{\-1\}\(\\theta\)for eachθ∈Im⁡M\\theta\\in\\Imop M, withs⁡\(θ0\)=q0′s\(\\theta\_\{0\}\)=q\_\{0\}^\{\\prime\}, and puth=s∘ρMh=s\\circ\\rho\_\{M\}\. ThenρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}andh⁡\(q0\)=q0′h\(q\_\{0\}\)=q\_\{0\}^\{\\prime\}, sohhis an arrow of𝐒𝐞𝐭∗/\(Θ,θ0\)\\mathbf\{Set\}\_\{\*\}/\(\\Theta,\\theta\_\{0\}\)\. ForρM​\(t\)=t\\rho\_\{M\}\(t\)=tandρN​\(u\)=u3\\rho\_\{N\}\(u\)=u^\{3\}both images areℝ\\mathbb\{R\}\. A map withρN∘h=id\\rho\_\{N\}\\circ h=\\mathrm\{id\}satisfiesh​\(t\)3=th\(t\)^\{3\}=t, soh⁡\(t\)=t1/3h\(t\)=t^\{1/3\}, the unique real cube root, whose difference quotientt−2/3t^\{\-2/3\}at00is unbounded\. So noC1C^\{1\}arrow, and in particular no smooth one, exists\. ∎

### C\.3 Definition I\.5

Definition I\.5\(invariants of an object\)\.ForM=\(Q,q0,ρ\)M=\(Q,q\_\{0\},\\rho\):•the imageIm⁡M=ρ⁡\(Q\)\\Imop M=\\rho\(Q\);•theexpressive dimensiond⁡\(M\)=dimIm⁡M=maxq⁡rank⁡d​ρqd\(M\)=\\dim\\Imop M=\\max\_\{q\}\\rk d\\rho\_\{q\};•thegauge wasteg⁡\(M\)=\|M\|−d⁡\(M\)g\(M\)=\\lvert M\\rvert\-d\(M\);•thefirst\-order imageS1​\(M\)=d​ρq0​\(Tq0​Q\)S\_\{1\}\(M\)=d\\rho\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\), and thefirst\-order deficitd​\(M\)−dimS1​\(M\)d\(M\)\-\\dim S\_\{1\}\(M\);•thegauge group\{φ∈Diff⁡\(Q\):ρ∘φ=ρ\}\\\{\\varphi\\in\\mathrm\{Diff\}\(Q\):\\rho\\circ\\varphi=\\rho\\\}, which is unpointed, since a gauge transformation movesq0q\_\{0\}inside its fibre\. In practice we exhibit a Lie group acting onQQthat preservesρ\\rhoand is locally transitive on generic fibres, and call it thegeneric gauge\.The base pointθ0\\theta\_\{0\}isregularforMMif the deficit is00, and anapexif the deficit is positive\. We sayvertexfor the subset\-level notion, the cone point of an image germ, and keepapexfor the base\-point notion\. The two differ in general\. At a crossing of two smooth branches the image is singular, yet the deficit can be00\. In every case below with a positive deficit, the image germ is \(diffeomorphic to\) a cone with vertexθ0\\theta\_\{0\}; the exception is DoRA, which inherits LoRA’s deficit through its torus factor\. A cone germ that is aC1C^\{1\}manifold at its vertex equals its tangent space there, so it is linear\. Hence the vertex of a non\-linear cone is a singular point\.

The definition contains three claims: the two descriptions ofd⁡\(M\)d\(M\)agree, the deficit is never negative \(so that regular and apex exhaust the cases\), and a cone germ that isC1C^\{1\}at its vertex is linear\.

*d⁡\(M\)d\(M\)is well defined\.*LetQQbe an open definable subset ofℝN\\mathbb\{R\}^\{N\}andρ\\rhodefinable and smooth, and letk=maxq⁡rank⁡d​ρqk=\\max\_\{q\}\\rk d\\rho\_\{q\}\. The set where the rank equalskkis open and nonempty, because a nonzerok×kk\\times kminor of the Jacobian stays nonzero nearby \([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(a\)\)\. At a pointppof it the rank is constant nearby, so by the rank theorem \([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\) a small open ballV∋pV\\ni p, a definable set, has imageρ⁡\(V\)\\rho\(V\)an embeddedkk\-dimensional submanifold\. It is definable, of dimensionkk, and contained inIm⁡M\\Imop M, sodimIm⁡M≥k\\dim\\Imop M\\geq k\([Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\)\. Conversely, lete=dimIm⁡Me=\\dim\\Imop M\. By[Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\),Im⁡M\\Imop Mcontains a definableC1C^\{1\}cellYYof dimensionee\. By definable choiceρ\\rhohas a definable sectionσ:Y→Q\\sigma:Y\\to Q, and by generic smoothnessσ\\sigmaisC1C^\{1\}on a nonempty open subsetY′Y^\{\\prime\}ofYY\([Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(c\)\)\. Differentiatingρ∘σ=idY′\\rho\\circ\\sigma=\\mathrm\{id\}\_\{Y^\{\\prime\}\}aty∈Y′y\\in Y^\{\\prime\}givesd​ρσ⁡\(y\)∘d​σy=idTy​Yd\\rho\_\{\\sigma\(y\)\}\\circ d\\sigma\_\{y\}=\\mathrm\{id\}\_\{T\_\{y\}Y\}, sorank⁡d​ρσ⁡\(y\)≥e\\rk d\\rho\_\{\\sigma\(y\)\}\\geq e, andk≥ek\\geq e\. HencedimIm⁡M=maxq⁡rank⁡d​ρq\\dim\\Imop M=\\max\_\{q\}\\rk d\\rho\_\{q\}, which is[Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(d\)\. The parameter spaces of the paper are open definable sets, for instance\{\(u1,…,ur\):ui≠0\}\\\{\(u\_\{1\},\\dots,u\_\{r\}\):u\_\{i\}\\neq 0\\\}for HRA, and the maps are built from polynomials, quotients, square roots and, for softmax, the exponential \([Background C\.17](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4)\)\.

*The deficit is nonnegative\.*dimS1​\(M\)=rank⁡d​ρq0≤maxq⁡rank⁡d​ρq=d⁡\(M\)\\dim S\_\{1\}\(M\)=\\rk d\\rho\_\{q\_\{0\}\}\\leq\\max\_\{q\}\\rk d\\rho\_\{q\}=d\(M\)\. Likewised⁡\(M\)≤dimQ=\|M\|d\(M\)\\leq\\dim Q=\\lvert M\\rvert, sog⁡\(M\)≥0g\(M\)\\geq 0\.

*Cones\.*LetX⊆ℝDX\\subseteq\\mathbb\{R\}^\{D\}agree nearvvwith a coneKKwith vertexvv, meaningv∈Kv\\in Kandv\+t⁡\(x−v\)∈Kv\+t\(x\-v\)\\in Kfor allx∈Kx\\in Kandt≥0t\\geq 0, and supposeXXis aC1C^\{1\}embedded submanifold nearvv, with tangent spaceTTatvv\. Translate so thatv=0v=0\. Near00,XXis the graph of aC1C^\{1\}mapφ\\varphifrom a neighbourhood of00inTTtoT⟂T^\{\\perp\}withφ⁡\(0\)=0\\varphi\(0\)=0andd​φ0=0d\\varphi\_\{0\}=0\. Forx∈Kx\\in Kand smallt\>0t\>0the pointt​xtxlies inKKnear00, hence inXX, sot​PT⟂​x=φ⁡\(t​PT​x\)=o⁡\(t\)tP\_\{T^\{\\perp\}\}x=\\varphi\(tP\_\{T\}x\)=o\(t\), and thereforePT⟂​x=0P\_\{T^\{\\perp\}\}x=0\. ThusK⊆TK\\subseteq T\. Near00,XXis then a graph overTTthat lies inTT, soφ=0\\varphi=0andXXcontains a neighbourhoodWWof00inTT\. Fory∈Ty\\in Tand smallε\>0\\varepsilon\>0we haveε​y∈W⊆K\\varepsilon y\\in W\\subseteq K, soy=ε−1​\(ε​y\)∈Ky=\\varepsilon^\{\-1\}\(\\varepsilon y\)\\in K\. HenceK=TK=T, and the germ ofXXatvvis the germ of the affine spacev\+Tv\+T\. IfKKis not an affine subspace,XXcannot be aC1C^\{1\}manifold atvv, sovvis a singular point \(compare[Auxiliary result B\.6](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. Finally, a figure\-eight curve through its crossing with nonzero velocity has deficit00there and a singular image, so vertex and apex are different notions\. ∎

### C\.4 Observation I\.6

Observation I\.6\(S1S\_\{1\}is functorial\)\.IfM→NM\\to Nis a \(pointed\) morphism, thenS1​\(M\)⊆S1​\(N\)S\_\{1\}\(M\)\\subseteq S\_\{1\}\(N\)\. In particular, isomorphic methods have equalS1S\_\{1\}and equal first\-order deficit\. Monotonicity holds only along pointed morphisms\. Along a weak morphism, such as a gauge map movingq0q\_\{0\}inside its fibre,S1S\_\{1\}can change\.

Leth:M→Nh:M\\to Nbe a pointed arrow, soρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}andh⁡\(q0\)=q0′h\(q\_\{0\}\)=q\_\{0\}^\{\\prime\}\. The chain rule atq0q\_\{0\}givesd​\(ρN\)q0′∘d​hq0=d​\(ρM\)q0d\(\\rho\_\{N\}\)\_\{q\_\{0\}^\{\\prime\}\}\\circ dh\_\{q\_\{0\}\}=d\(\\rho\_\{M\}\)\_\{q\_\{0\}\}, soS1​\(M\)=d​\(ρN\)q0′​\(d​hq0​\(Tq0​QM\)\)⊆d​\(ρN\)q0′​\(Tq0′​QN\)=S1​\(N\)S\_\{1\}\(M\)=d\(\\rho\_\{N\}\)\_\{q\_\{0\}^\{\\prime\}\}\\big\(dh\_\{q\_\{0\}\}\(T\_\{q\_\{0\}\}Q\_\{M\}\)\\big\)\\subseteq d\(\\rho\_\{N\}\)\_\{q\_\{0\}^\{\\prime\}\}\(T\_\{q\_\{0\}^\{\\prime\}\}Q\_\{N\}\)=S\_\{1\}\(N\)\. Ifhhis an isomorphism,h−1h^\{\-1\}is a pointed arrowN→MN\\to M, soS1​\(N\)⊆S1​\(M\)S\_\{1\}\(N\)\\subseteq S\_\{1\}\(M\)as well\. MoreoverρM=ρN∘h\\rho\_\{M\}=\\rho\_\{N\}\\circ hwithhhbijective givesIm⁡M=Im⁡N\\Imop M=\\Imop N, sod⁡\(M\)=d⁡\(N\)d\(M\)=d\(N\)\(equivalently,rank⁡d​\(ρM\)q=rank⁡d​\(ρN\)h⁡\(q\)\\rk d\(\\rho\_\{M\}\)\_\{q\}=\\rk d\(\\rho\_\{N\}\)\_\{h\(q\)\}sinced​hqdh\_\{q\}is invertible\), and the deficits agree\.

Withouth⁡\(q0\)=q0′h\(q\_\{0\}\)=q\_\{0\}^\{\\prime\}the chain rule comparesS1​\(M\)S\_\{1\}\(M\)with the first\-order image ofNNath⁡\(q0\)h\(q\_\{0\}\), which need not be the base point ofNN\. For instance, take LoRA with frozen residualθ0\\theta\_\{0\}and1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\), and letMMandNNbe its pointings at\(0,A0\)\(0,A\_\{0\}\)and at\(B0,0\)\(B\_\{0\},0\), withA0A\_\{0\}andB0B\_\{0\}of full rank; both points lie in the fibre overθ0\\theta\_\{0\}\. The identity ofQQis a weak morphismM→NM\\to N, yetS1​\(M\)=\{X​A0\}=ℛ⁡\(row⁡A0\)S\_\{1\}\(M\)=\\\{XA\_\{0\}\\\}=\\mathcal\{R\}\(\\operatorname\{row\}A\_\{0\}\)andS1​\(N\)=\{B0​Y\}=𝒞⁡\(col⁡B0\)S\_\{1\}\(N\)=\\\{B\_\{0\}Y\\\}=\\mathcal\{C\}\(\\operatorname\{col\}B\_\{0\}\)are not nested:ℛ⁡\(row⁡A0\)\\mathcal\{R\}\(\\operatorname\{row\}A\_\{0\}\)containsei​v⊤e\_\{i\}v^\{\\top\}for everyiiand everyv∈row⁡A0v\\in\\operatorname\{row\}A\_\{0\}, which is not in𝒞⁡\(col⁡B0\)\\mathcal\{C\}\(\\operatorname\{col\}B\_\{0\}\)whenei∉col⁡B0e\_\{i\}\\notin\\operatorname\{col\}B\_\{0\}, and symmetrically\. ∎

### C\.5 Observation I\.7

Observation I\.7\(additive monoidal structure\)\.Θ\\Thetais a vector space, so every object of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)can be writtenρ=θ0\+δ\\rho=\\theta\_\{0\}\+\\deltawithδ:=ρ−θ0\\delta:=\\rho\-\\theta\_\{0\}andδ⁡\(q0\)=0\\delta\(q\_\{0\}\)=0\. PutM⊕N:=\(QM×QN,\(q0,q0′\),θ0\+δM\+δN\)\.M\\oplus N:=\\big\(Q\_\{M\}\\times Q\_\{N\},\\ \(q\_\{0\},q\_\{0\}^\{\\prime\}\),\\ \\theta\_\{0\}\+\\delta\_\{M\}\+\\delta\_\{N\}\\big\)\.This is a symmetric monoidal structure with the frozen model as unit\. Its image is the Minkowski sumθ0\+\(Im⁡δM\+Im⁡δN\)\\theta\_\{0\}\+\(\\Imop\\delta\_\{M\}\+\\Imop\\delta\_\{N\}\)\. It differs from the categorical product, whose image, when the product exists, is the intersectionIm⁡M∩Im⁡N\\Imop M\\cap\\Imop N\. “Additive” in[Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\(δ\\deltaindependent ofθ0\\theta\_\{0\}\) is a condition on*families*θ0↦M⁡\(θ0\)\\theta\_\{0\}\\mapsto M\(\\theta\_\{0\}\); it matters for base change and plays no role here\.WriteLoRAk\\mathrm\{LoRA\}\_\{k\}for LoRA with frozen residualθ0\\theta\_\{0\}; for split pointings the sum below has residualθ0−P−P′\\theta\_\{0\}\-P\-P^\{\\prime\}\. Concatenationh⁡\(B1,A1,B2,A2\)=\(\[B1​B2\],\[A1;A2\]\)h\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)=\(\[B\_\{1\}\\ B\_\{2\}\],\[A\_\{1\};A\_\{2\}\]\)is an isomorphismLoRAr⊕LoRAs≅LoRAr\+s\\mathrm\{LoRA\}\_\{r\}\\oplus\\mathrm\{LoRA\}\_\{s\}\\cong\\mathrm\{LoRA\}\_\{r\+s\}of unpointed objects\. It is a pointed isomorphism ontoLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}pointed at\(\[B0​B0′\],\[A0;A0′\]\)\(\[B\_\{0\}\\ B\_\{0\}^\{\\prime\}\],\[A\_\{0\};A\_\{0\}^\{\\prime\}\]\)\. ForTA\\mathrm\{T\}\_\{A\}pointings \(B0=0B\_\{0\}=0,rank⁡A0=r\\rk A\_\{0\}=r\), the target pointing hasrank⁡\[A0;A0′\]=r\+s\\rk\[A\_\{0\};A\_\{0\}^\{\\prime\}\]=r\+sif and only ifrow⁡A0∩row⁡A0′=0\\operatorname\{row\}A\_\{0\}\\cap\\operatorname\{row\}A\_\{0\}^\{\\prime\}=0; it lies in the stratumTA\\mathrm\{T\}\_\{A\}of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)when moreoverr\+s<min⁡\(m,n\)r\+s<\\min\(m,n\)\.Standard pointings \(TA\\mathrm\{T\}\_\{A\}, or symmetricallyTB\\mathrm\{T\}\_\{B\}\) exist only forr<min⁡\(m,n\)r<\\min\(m,n\), so the monoidal question is pointwise\. IfF⁡\(r\)F\(r\)andF⁡\(2​r\)F\(2r\)are standard \(2​r<min⁡\(m,n\)2r<\\min\(m,n\)\), thenF⁡\(r\)⊕F⁡\(r\)≇F⁡\(2​r\)F\(r\)\\oplus F\(r\)\\not\\cong F\(2r\)as pointed objects; for2​r≥min⁡\(m,n\)2r\\geq\\min\(m,n\)no standardF⁡\(2​r\)F\(2r\)exists\. Hence no assignmentr↦LoRArr\\mapsto\\mathrm\{LoRA\}\_\{r\}withF⁡\(1\)F\(1\)andF⁡\(2\)F\(2\)standard isstrong monoidalinto pointed objects\. Forr≠sr\\neq sand standard pointings,F⁡\(r\)⊕F⁡\(s\)≅F⁡\(r\+s\)F\(r\)\\oplus F\(s\)\\cong F\(r\+s\)as pointed objects exactly when the row space of the target’sA0A\_\{0\}is the direct sum of the two source row spaces\. The assignment is strong monoidal on unpointed objects, up to weak morphisms, and on pointed objects with degenerate pointings, namely the origin\(B0,A0\)=\(0,0\)\(B\_\{0\},A\_\{0\}\)=\(0,0\), whereS1=0S\_\{1\}=0, or the stacked pointingsF⁡\(r\)=F​\(1\)⊕rF\(r\)=F\(1\)^\{\\oplus r\}\. Concatenation is strictly associative and unital, so discrete\(ℕ,\+\)\(\\mathbb\{N\},\+\)suffices for a plain strong monoidal functor\.*Symmetric*strong monoidality needs the permutation groupoid, since swapping and then concatenating permutes the blocks\.With per\-adapter scalessrs\_\{r\}\(α/r\\alpha/rfor LoRA,α/r\\alpha/\\sqrt\{r\}for rsLoRA\) and any target scales∗s\_\{\*\}, the maph=\(\[B1​B2\],\[c1​A1;c2​A2\]\)h=\(\[B\_\{1\}\\ B\_\{2\}\],\[c\_\{1\}A\_\{1\};c\_\{2\}A\_\{2\}\]\)withci=sri/s∗c\_\{i\}=s\_\{r\_\{i\}\}/s\_\{\*\}, wherer1=rr\_\{1\}=randr2=sr\_\{2\}=s, is an isomorphism overΘ\\Theta\. It is a linear bijection and preserves row spaces, so the pointing criterion is unchanged\.

*The monoidal structure\.*M⊕NM\\oplus Nis an object:QM×QNQ\_\{M\}\\times Q\_\{N\}is a manifold,θ0\+δM\+δN\\theta\_\{0\}\+\\delta\_\{M\}\+\\delta\_\{N\}is smooth and takes the valueθ0\\theta\_\{0\}at\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\)\. For arrowsh:M→M′h:M\\to M^\{\\prime\}andk:N→N′k:N\\to N^\{\\prime\}, the pointed maph×kh\\times klies overΘ\\ThetabecauseδM′∘h=ρM′∘h−θ0=δM\\delta\_\{M^\{\\prime\}\}\\circ h=\\rho\_\{M^\{\\prime\}\}\\circ h\-\\theta\_\{0\}=\\delta\_\{M\}and likewise forkk; this makes⊕\\oplusa functor\. The associator\(Q1×Q2\)×Q3≅Q1×\(Q2×Q3\)\(Q\_\{1\}\\times Q\_\{2\}\)\\times Q\_\{3\}\\cong Q\_\{1\}\\times\(Q\_\{2\}\\times Q\_\{3\}\), the unitorspt×Q≅Q≅Q×pt\\mathrm\{pt\}\\times Q\\cong Q\\cong Q\\times\\mathrm\{pt\}and the symmetryQM×QN≅QN×QMQ\_\{M\}\\times Q\_\{N\}\\cong Q\_\{N\}\\times Q\_\{M\}are the structure maps of the cartesian monoidal structure of𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}\. They lie overΘ\\Thetabecause addition is associative and commutative and the frozen model hasδ=0\\delta=0\. Their coherence \(pentagon, triangle, hexagon and the symmetry being an involution\) holds in𝐃𝐢𝐟𝐟∗\\mathbf\{Diff\}\_\{\*\}, and the forgetful functor𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)→𝐃𝐢𝐟𝐟∗\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\\to\\mathbf\{Diff\}\_\{\*\}is faithful, so it holds in𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\.

*Image\.*\{θ0\+δM​\(q\)\+δN​\(q′\)\}=θ0\+\(Im⁡δM\+Im⁡δN\)\\\{\\theta\_\{0\}\+\\delta\_\{M\}\(q\)\+\\delta\_\{N\}\(q^\{\\prime\}\)\\\}=\\theta\_\{0\}\+\(\\Imop\\delta\_\{M\}\+\\Imop\\delta\_\{N\}\)\. The categorical product has imageIm⁡M∩Im⁡N\\Imop M\\cap\\Imop Nwhen it exists \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\), and the two differ in general: for zero\-initLoRA1\\mathrm\{LoRA\}\_\{1\}on a2×22\\times 2box,M⊕MM\\oplus Mhas imageθ0\+ℳ≤2\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq 2\}, while any product ofMMwith itself has imageθ0\+ℳ≤1\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq 1\}\.

*Concatenation\.*The maphhonly regroups coordinates, so it is a linear bijection between the parameter spaces, andθ0\+s⁡\[B1​B2\]​\[A1;A2\]=θ0\+s​B1​A1\+s​B2​A2\\theta\_\{0\}\+s\[B\_\{1\}\\ B\_\{2\}\]\[A\_\{1\};A\_\{2\}\]=\\theta\_\{0\}\+sB\_\{1\}A\_\{1\}\+sB\_\{2\}A\_\{2\}, so it lies overΘ\\Theta\. It is a pointed isomorphism ontoLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}pointed at the image of\(q0,q0′\)\(q\_\{0\},q\_\{0\}^\{\\prime\}\), the concatenated point\. For split pointings withP=s​B0​A0P=sB\_\{0\}A\_\{0\}andP′=s​B0′​A0′P^\{\\prime\}=sB\_\{0\}^\{\\prime\}A\_\{0\}^\{\\prime\}, the sum is\(θ0−P−P′\)\+s⁡\[B1​B2\]​\[A1;A2\]\(\\theta\_\{0\}\-P\-P^\{\\prime\}\)\+s\[B\_\{1\}\\ B\_\{2\}\]\[A\_\{1\};A\_\{2\}\], which isLoRAr\+s\\mathrm\{LoRA\}\_\{r\+s\}with residualθ0−P−P′\\theta\_\{0\}\-P\-P^\{\\prime\}\. ForTA\\mathrm\{T\}\_\{A\}pointings, the concatenated pointing hasB=0B=0andrank⁡\[A0;A0′\]=dim\(row⁡A0\+row⁡A0′\)=r\+s−dim\(row⁡A0∩row⁡A0′\)\\rk\[A\_\{0\};A\_\{0\}^\{\\prime\}\]=\\dim\(\\operatorname\{row\}A\_\{0\}\+\\operatorname\{row\}A\_\{0\}^\{\\prime\}\)=r\+s\-\\dim\(\\operatorname\{row\}A\_\{0\}\\cap\\operatorname\{row\}A\_\{0\}^\{\\prime\}\), which isr\+sr\+sexactly when the intersection is00\. The stratumTA\\mathrm\{T\}\_\{A\}of rankr\+sr\+sis defined forr\+s<min⁡\(m,n\)r\+s<\\min\(m,n\), which gives the last condition\.

*Row and column types\.*We use two elementary facts about the spacesℛ⁡\(V\)\\mathcal\{R\}\(V\)and𝒞⁡\(U\)\\mathcal\{C\}\(U\)of the introduction, which return in[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\. First,ℛ⁡\(V\)=\{X​A:X∈ℝm×k\}\\mathcal\{R\}\(V\)=\\\{XA:X\\in\\mathbb\{R\}^\{m\\times k\}\\\}for anyAAof full row rank withrow⁡A=V\\operatorname\{row\}A=V, sinceZ=Z​A\+​AZ=ZA^\{\+\}Awhenrow⁡Z⊆V\\operatorname\{row\}Z\\subseteq V; sodimℛ⁡\(V\)=m​dimV\\dim\\mathcal\{R\}\(V\)=m\\dim V, andℛ⁡\(V\)\\mathcal\{R\}\(V\)determinesVVas the sum of the row spaces of its elements\. Likewise𝒞⁡\(U\)=\{B​Y\}\\mathcal\{C\}\(U\)=\\\{BY\\\}forcol⁡B=U\\operatorname\{col\}B=U, of dimensionn​dimUn\\dim U, determinesUU\. Second, ifV≠0V\\neq 0andℛ⁡\(V\)=𝒞⁡\(U\)\\mathcal\{R\}\(V\)=\\mathcal\{C\}\(U\), thenU=ℝmU=\\mathbb\{R\}^\{m\}, becauseℛ⁡\(V\)\\mathcal\{R\}\(V\)containsei​v⊤e\_\{i\}v^\{\\top\}for everyiiand every nonzerov∈Vv\\in V, andei​v⊤∈𝒞⁡\(U\)e\_\{i\}v^\{\\top\}\\in\\mathcal\{C\}\(U\)forcesei∈Ue\_\{i\}\\in U\.

*The negative claim\.*Standard pointings are the strataTA\\mathrm\{T\}\_\{A\}andTB\\mathrm\{T\}\_\{B\}of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2), which are defined only for ranks belowmin⁡\(m,n\)\\min\(m,n\)\. LetF⁡\(r\)F\(r\)andF⁡\(2​r\)F\(2r\)be standard, so2​r<min⁡\(m,n\)2r<\\min\(m,n\)\. IfF⁡\(r\)F\(r\)is pointed at\(0,A0\)\(0,A\_\{0\}\), thenF⁡\(r\)⊕F⁡\(r\)F\(r\)\\oplus F\(r\)is pointed at\(\(0,A0\),\(0,A0\)\)\(\(0,A\_\{0\}\),\(0,A\_\{0\}\)\)and its first\-order image is\{X1​A0\+X2​A0\}=ℛ⁡\(row⁡A0\)\\\{X\_\{1\}A\_\{0\}\+X\_\{2\}A\_\{0\}\\\}=\\mathcal\{R\}\(\\operatorname\{row\}A\_\{0\}\), of dimensionm​rmr\. IfF⁡\(r\)F\(r\)is inTB\\mathrm\{T\}\_\{B\}, it is𝒞⁡\(col⁡B0\)\\mathcal\{C\}\(\\operatorname\{col\}B\_\{0\}\), of dimensionn​rnr\. A standardF⁡\(2​r\)F\(2r\)has first\-order imageℛ⁡\(V′′\)\\mathcal\{R\}\(V^\{\\prime\\prime\}\)withdimV′′=2​r\\dim V^\{\\prime\\prime\}=2ror𝒞⁡\(U′′\)\\mathcal\{C\}\(U^\{\\prime\\prime\}\)withdimU′′=2​r\\dim U^\{\\prime\\prime\}=2r\. A pointed isomorphism would make the two first\-order images equal \([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. Two row\-type spaces determine their subspaces, of dimensionsr≠2​rr\\neq 2r; two column\-type spaces likewise; and a row\-type space withV≠0V\\neq 0equals a column\-type space only if the column subspace is all ofℝm\\mathbb\{R\}^\{m\}, impossible forU′′U^\{\\prime\\prime\}of dimension2​r<m2r<m\(and symmetrically\)\. SoF⁡\(r\)⊕F⁡\(r\)≇F⁡\(2​r\)F\(r\)\\oplus F\(r\)\\not\\cong F\(2r\)\. A strong monoidal functorFFwould supply a pointed isomorphismF⁡\(1\)⊕F⁡\(1\)≅F⁡\(2\)F\(1\)\\oplus F\(1\)\\cong F\(2\), so none exists whenF⁡\(1\)F\(1\)andF⁡\(2\)F\(2\)are standard\.

*The caser≠sr\\neq s\.*LetF⁡\(r\)F\(r\),F⁡\(s\)F\(s\)andF⁡\(r\+s\)F\(r\+s\)be standard of typeTA\\mathrm\{T\}\_\{A\}, pointed at\(0,A0\)\(0,A\_\{0\}\),\(0,A0′\)\(0,A\_\{0\}^\{\\prime\}\)and\(0,A0′′\)\(0,A\_\{0\}^\{\\prime\\prime\}\); the typeTB\\mathrm\{T\}\_\{B\}is the transpose\. ThenS1​\(F⁡\(r\)⊕F⁡\(s\)\)=\{X​A0\+X′​A0′\}=ℛ⁡\(row⁡A0\+row⁡A0′\)S\_\{1\}\(F\(r\)\\oplus F\(s\)\)=\\\{XA\_\{0\}\+X^\{\\prime\}A\_\{0\}^\{\\prime\}\\\}=\\mathcal\{R\}\(\\operatorname\{row\}A\_\{0\}\+\\operatorname\{row\}A\_\{0\}^\{\\prime\}\)andS1​\(F⁡\(r\+s\)\)=ℛ⁡\(row⁡A0′′\)S\_\{1\}\(F\(r\+s\)\)=\\mathcal\{R\}\(\\operatorname\{row\}A\_\{0\}^\{\\prime\\prime\}\), withdimrow⁡A0′′=r\+s\\dim\\operatorname\{row\}A\_\{0\}^\{\\prime\\prime\}=r\+s\. A pointed isomorphism forcesrow⁡A0′′=row⁡A0\+row⁡A0′\\operatorname\{row\}A\_\{0\}^\{\\prime\\prime\}=\\operatorname\{row\}A\_\{0\}\+\\operatorname\{row\}A\_\{0\}^\{\\prime\}, and the dimension count makes the sum direct\. Conversely, ifrow⁡A0′′=row⁡A0⊕row⁡A0′\\operatorname\{row\}A\_\{0\}^\{\\prime\\prime\}=\\operatorname\{row\}A\_\{0\}\\oplus\\operatorname\{row\}A\_\{0\}^\{\\prime\}, then\[A0;A0′\]\[A\_\{0\};A\_\{0\}^\{\\prime\}\]has full row rank and the row space ofA0′′A\_\{0\}^\{\\prime\\prime\}, soA0′′=g⁡\[A0;A0′\]A\_\{0\}^\{\\prime\\prime\}=g\[A\_\{0\};A\_\{0\}^\{\\prime\}\]for a uniqueg∈GLr\+sg\\in\\mathrm\{GL\}\_\{r\+s\}\. Concatenation followed by the gauge\(B,A\)↦\(B​g−1,g​A\)\(B,A\)\\mapsto\(Bg^\{\-1\},gA\), which preservesB​ABA, is a pointed isomorphismF⁡\(r\)⊕F⁡\(s\)→F⁡\(r\+s\)F\(r\)\\oplus F\(s\)\\to F\(r\+s\)\.

*Strong monoidality\.*The concatenation mapsϕr,s=h\\phi\_\{r,s\}=hsatisfyϕr\+s,t∘\(ϕr,s×id\)=ϕr,s\+t∘\(id×ϕs,t\)∘a\\phi\_\{r\+s,t\}\\circ\(\\phi\_\{r,s\}\\times\\mathrm\{id\}\)=\\phi\_\{r,s\+t\}\\circ\(\\mathrm\{id\}\\times\\phi\_\{s,t\}\)\\circ a, both sides sending\(B1,A1,B2,A2,B3,A3\)\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\},B\_\{3\},A\_\{3\}\)to\(\[B1​B2​B3\],\[A1;A2;A3\]\)\(\[B\_\{1\}\\ B\_\{2\}\\ B\_\{3\}\],\[A\_\{1\};A\_\{2\};A\_\{3\}\]\), andϕ0,r\\phi\_\{0,r\},ϕr,0\\phi\_\{r,0\}are the unitors, withLoRA0\\mathrm\{LoRA\}\_\{0\}the frozen model\. Discrete\(ℕ,\+\)\(\\mathbb\{N\},\+\)has only identity arrows, so naturality is vacuous, andr↦LoRArr\\mapsto\\mathrm\{LoRA\}\_\{r\}is strong monoidal on unpointed objects\. As maps between chosen pointings, theϕr,s\\phi\_\{r,s\}are weak morphisms\. They are pointed isomorphisms when everyF⁡\(r\)F\(r\)is pointed at the origin, sincehhmaps origin to origin, and whenF⁡\(r\)F\(r\)is pointed at the concatenation ofrrcopies of the base point ofF⁡\(1\)F\(1\), sincehhmaps concatenations to concatenations\. The symmetry of discrete\(ℕ,\+\)\(\\mathbb\{N\},\+\)is the identity, so symmetry would requireϕr,s=ϕs,r∘σ\\phi\_\{r,s\}=\\phi\_\{s,r\}\\circ\\sigma, whereσ\\sigmaswaps the factors\. The left side sends\(B1,A1,B2,A2\)\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)to\(\[B1​B2\],\[A1;A2\]\)\(\[B\_\{1\}\\ B\_\{2\}\],\[A\_\{1\};A\_\{2\}\]\)and the right side to\(\[B2​B1\],\[A2;A1\]\)\(\[B\_\{2\}\\ B\_\{1\}\],\[A\_\{2\};A\_\{1\}\]\), which differ forr,s≥1r,s\\geq 1\. On the permutation groupoid, a permutationπ∈Sr\\pi\\in S\_\{r\}acts onLoRAr\\mathrm\{LoRA\}\_\{r\}by the gauge\(B,A\)↦\(B​Pπ−1,Pπ​A\)\(B,A\)\\mapsto\(BP\_\{\\pi\}^\{\-1\},P\_\{\\pi\}A\), which is functorial and lies overΘ\\Theta, the mapsϕr,s\\phi\_\{r,s\}are natural for block sums of permutations, and with the block transpositionτr,s∈Sr\+s\\tau\_\{r,s\}\\in S\_\{r\+s\}as symmetry,F⁡\(τr,s\)∘ϕr,s=ϕs,r∘σF\(\\tau\_\{r,s\}\)\\circ\\phi\_\{r,s\}=\\phi\_\{s,r\}\\circ\\sigma, because permuting the channel blocks of\(\[B1​B2\],\[A1;A2\]\)\(\[B\_\{1\}\\ B\_\{2\}\],\[A\_\{1\};A\_\{2\}\]\)gives\(\[B2​B1\],\[A2;A1\]\)\(\[B\_\{2\}\\ B\_\{1\}\],\[A\_\{2\};A\_\{1\}\]\)\. The origin and the stacked pointings are fixed by channel permutations, so the same holds for them\.

*Rescaled concatenation\.*s∗​\[B1​B2\]​\[c1​A1;c2​A2\]=s∗​c1​B1​A1\+s∗​c2​B2​A2=sr​B1​A1\+ss​B2​A2s\_\{\*\}\[B\_\{1\}\\ B\_\{2\}\]\[c\_\{1\}A\_\{1\};c\_\{2\}A\_\{2\}\]=s\_\{\*\}c\_\{1\}B\_\{1\}A\_\{1\}\+s\_\{\*\}c\_\{2\}B\_\{2\}A\_\{2\}=s\_\{r\}B\_\{1\}A\_\{1\}\+s\_\{s\}B\_\{2\}A\_\{2\}\. The scales are nonzero, sohhis a linear bijection, androw⁡\(ci​Ai\)=row⁡Ai\\operatorname\{row\}\(c\_\{i\}A\_\{i\}\)=\\operatorname\{row\}A\_\{i\}, so the criterion for the concatenated pointing is the same as before\. ∎

### C\.6 Proposition I\.8

Proposition I\.8\(a product that produces a named method; an image\-level meet\)\.\(a\) LetA0∈ℝr×nA\_\{0\}\\in\\mathbb\{R\}^\{r\\times n\}have full row rank andB0∈ℝm×rB\_\{0\}\\in\\mathbb\{R\}^\{m\\times r\}full column rank\. Define the one\-sided methodsFA⁡\(A0\):B↦W0\+B​A0,FB⁡\(B0\):A↦W0\+B0​A,\\mathrm\{FA\}\(A\_\{0\}\):B\\mapsto W\_\{0\}\+BA\_\{0\},\\qquad\\mathrm\{FB\}\(B\_\{0\}\):A\\mapsto W\_\{0\}\+B\_\{0\}A,both pointed at00\. ThenFA\(A0\)×FB\(B0\)≅\(ℝr×r,0,R↦W0\+B0RA0\)\.\\mathrm\{FA\}\(A\_\{0\}\)\\times\\mathrm\{FB\}\(B\_\{0\}\)\\;\\cong\\;\\big\(\\mathbb\{R\}^\{r\\times r\},\\ 0,\\ R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}\\big\)\.The product supplies this sandwich for*any*frames\. Assumerank⁡W0≥r\\rk W\_\{0\}\\geq r\(σr\>0\\sigma\_\{r\}\>0\) and take the SVD framesB0=UrB\_\{0\}=U\_\{r\},A0=Σr​Vr⊤A\_\{0\}=\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\. The result isLoRA\-XS pointed atR=0R=0\.\(b\) LetW0W\_\{0\}have full row rank \(som≤nm\\leq n\), and letLoRAr\\mathrm\{LoRA\}\_\{r\}carry its standard pointing \(B0=0B\_\{0\}=0, imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\. The image\-level meet ofLoRAr\\mathrm\{LoRA\}\_\{r\}with the output\-scaling methodℓ↦diag⁡\(ℓ\)​W0\\ell\\mapsto\\diag\(\\ell\)W\_\{0\}is\{diag⁡\(ℓ\)​W0:\#⁡\{i:ℓi≠1\}≤r\}\\\{\\diag\(\\ell\)W\_\{0\}:\\\#\\\{i:\\ell\_\{i\}\\neq 1\\\}\\leq r\\\}, that is, “rescale at mostrroutput neurons”\.The LoRA\-XS paper writesx​W\+x​A​R​BxW\+xARBin row\-vector convention withA=Ur​ΣrA=U\_\{r\}\\Sigma\_\{r\}andB=Vr⊤B=V\_\{r\}^\{\\top\}\. Transposed, the update isB0​R⊤​A0B\_\{0\}R^\{\\top\}A\_\{0\}, so the isomorphism isR↦α​R⊤R\\mapsto\\alpha R^\{\\top\}\. LoRA\-XS initialisesR∼𝒩⁡\(0,σ2\)R\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}\)withσ=10−5\\sigma=10^\{\-5\}, a negligible pointing defect\. The output\-scaling method in \(b\) is the key/value part of \(IA\)3\. On the FFN, \(IA\)3scales the*input*ofWdownW\_\{\\rm down\},W↦W​diag⁡\(ℓ\)W\\mapsto W\\diag\(\\ell\); the mirror statement then needs full column rank, or holds generically forr<dr<d, whereddis the number of rows ofWdownW\_\{\\rm down\}\.

*\(a\)*WriteB0\+=\(B0⊤​B0\)−1​B0⊤B\_\{0\}^\{\+\}=\(B\_\{0\}^\{\\top\}B\_\{0\}\)^\{\-1\}B\_\{0\}^\{\\top\}andA0\+=A0⊤​\(A0​A0⊤\)−1A\_\{0\}^\{\+\}=A\_\{0\}^\{\\top\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}, so thatB0\+​B0=Ir=A0​A0\+B\_\{0\}^\{\+\}B\_\{0\}=I\_\{r\}=A\_\{0\}A\_\{0\}^\{\+\}\. A point of the set\-level fibre product is a pair\(B,A\)\(B,A\)withB​A0=B0​A=:DBA\_\{0\}=B\_\{0\}A=:D\. Thenrow⁡D⊆row⁡A0\\operatorname\{row\}D\\subseteq\\operatorname\{row\}A\_\{0\}andcol⁡D⊆col⁡B0\\operatorname\{col\}D\\subseteq\\operatorname\{col\}B\_\{0\}, soD=B0​B0\+​D​A0\+​A0=B0​R​A0D=B\_\{0\}B\_\{0\}^\{\+\}DA\_\{0\}^\{\+\}A\_\{0\}=B\_\{0\}RA\_\{0\}withR=B0\+​D​A0\+R=B\_\{0\}^\{\+\}DA\_\{0\}^\{\+\}, andRRis unique becauseB0B\_\{0\}is injective andA0A\_\{0\}surjective\. FromB​A0=B0​R​A0BA\_\{0\}=B\_\{0\}RA\_\{0\}andA0​A0\+=IA\_\{0\}A\_\{0\}^\{\+\}=Iwe getB=B0​RB=B\_\{0\}R, and fromB0​A=B0​R​A0B\_\{0\}A=B\_\{0\}RA\_\{0\}we getA=R​A0A=RA\_\{0\}\. So the fibre product is ther2r^\{2\}\-dimensional linear subspace\{\(B0​R,R​A0\)\}\\\{\(B\_\{0\}R,RA\_\{0\}\)\\\}ofℝm×r×ℝr×n\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}, andj:R↦\(B0​R,R​A0\)j:R\\mapsto\(B\_\{0\}R,RA\_\{0\}\)is a linear isomorphism onto it, with inverse\(B,A\)↦B0\+​B\(B,A\)\\mapsto B\_\{0\}^\{\+\}B\. Both compositesR↦W0\+B0​R​A0R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}agree, andj⁡\(0\)=\(0,0\)j\(0\)=\(0,0\)is the pair of base points\.

The universal property: letTTbe a pointed manifold andh1:T→ℝm×rh\_\{1\}:T\\to\\mathbb\{R\}^\{m\\times r\},h2:T→ℝr×nh\_\{2\}:T\\to\\mathbb\{R\}^\{r\\times n\}pointed smooth maps withW0\+h1​\(t\)​A0=W0\+B0​h2​\(t\)W\_\{0\}\+h\_\{1\}\(t\)A\_\{0\}=W\_\{0\}\+B\_\{0\}h\_\{2\}\(t\)\. By the previous paragraph\(h1​\(t\),h2​\(t\)\)=j⁡\(R⁡\(t\)\)\(h\_\{1\}\(t\),h\_\{2\}\(t\)\)=j\(R\(t\)\)withR⁡\(t\)=B0\+​h1​\(t\)R\(t\)=B\_\{0\}^\{\+\}h\_\{1\}\(t\), a pointed smooth map, and it is the only map with this property becausejjis injective\. So\(ℝr×r,0,R↦W0\+B0RA0\)\(\\mathbb\{R\}^\{r\\times r\},0,R\\mapsto W\_\{0\}\+B\_\{0\}RA\_\{0\}\)with the projectionsR↦B0​RR\\mapsto B\_\{0\}RandR↦R​A0R\\mapsto RA\_\{0\}is the product in𝐌𝐞𝐭𝐡⁡\(Θ,W0\)\\mathbf\{Meth\}\(\\Theta,W\_\{0\}\)\([Auxiliary result B\.2](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(c\),[Auxiliary result B\.3](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\)\. This holds for any frames of full rank\. WithB0=UrB\_\{0\}=U\_\{r\}andA0=Σr​Vr⊤A\_\{0\}=\\Sigma\_\{r\}V\_\{r\}^\{\\top\}from an SVD ofW0W\_\{0\}\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\), the frames have full rank exactly whenσr\>0\\sigma\_\{r\}\>0; ifrank⁡W0<r\\rk W\_\{0\}<r,Σr​Vr⊤\\Sigma\_\{r\}V\_\{r\}^\{\\top\}is rank\-deficient and the hypothesis fails\.

The row\-vector convention: LoRA\-XS acts on row vectors byx↦x⁡\(Wrow\+α​A​R​B\)x\\mapsto x\(W\_\{\\rm row\}\+\\alpha ARB\), whereWrow=W0⊤W\_\{\\rm row\}=W\_\{0\}^\{\\top\}has SVDU′​Σ​V′⁣⊤U^\{\\prime\}\\Sigma V^\{\\prime\\top\},A=Ur′​ΣrA=U^\{\\prime\}\_\{r\}\\Sigma\_\{r\}andB=Vr′⁣⊤B=V\_\{r\}^\{\\prime\\top\}\. In our convention the update is the transpose,α​Vr′​R⊤​Σr​Ur′⁣⊤\\alpha V^\{\\prime\}\_\{r\}R^\{\\top\}\\Sigma\_\{r\}U\_\{r\}^\{\\prime\\top\}\. SinceW0=V′​Σ​U′⁣⊤W\_\{0\}=V^\{\\prime\}\\Sigma U^\{\\prime\\top\}, the left and right singular frames ofW0W\_\{0\}areUr=Vr′U\_\{r\}=V^\{\\prime\}\_\{r\}andVr=Ur′V\_\{r\}=U^\{\\prime\}\_\{r\}, so the update isUr​\(α​R⊤\)​Σr​Vr⊤=B0​\(α​R⊤\)​A0U\_\{r\}\(\\alpha R^\{\\top\}\)\\Sigma\_\{r\}V\_\{r\}^\{\\top\}=B\_\{0\}\(\\alpha R^\{\\top\}\)A\_\{0\}\. Hence LoRA\-XS is the product composed with the linear isomorphismR↦α​R⊤R\\mapsto\\alpha R^\{\\top\}, which fixes00\. A random initialRRmakesρ⁡\(R0\)−W0=α​Ur​R0⊤​Σr​Vr⊤\\rho\(R\_\{0\}\)\-W\_\{0\}=\\alpha U\_\{r\}R\_\{0\}^\{\\top\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\}nonzero, of the size ofσ\\sigma, a pointing defect\.

*\(b\)*The image ofLoRAr\\mathrm\{LoRA\}\_\{r\}with its standard pointing isW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\([Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(c\)\), and the output\-scaling method has image\{diag⁡\(ℓ\)​W0\}\\\{\\diag\(\\ell\)W\_\{0\}\\\}\. A pointdiag⁡\(ℓ\)​W0\\diag\(\\ell\)W\_\{0\}lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}iffrank⁡\(diag⁡\(ℓ−𝟏\)​W0\)≤r\\rk\\big\(\\diag\(\\ell\-\\mathbf\{1\}\)W\_\{0\}\\big\)\\leq r\. The rows ofdiag⁡\(ℓ−𝟏\)​W0\\diag\(\\ell\-\\mathbf\{1\}\)W\_\{0\}are\(ℓi−1\)​wi\(\\ell\_\{i\}\-1\)w\_\{i\}, and the nonzero ones are linearly independent because the rows ofW0W\_\{0\}are, so the rank is\#​\{i:ℓi≠1\}\\\#\\\{i:\\ell\_\{i\}\\neq 1\\\}\. The meet in the poset of subsets is the intersection, which is the stated set\.

For the input side,W0​diag⁡\(ℓ\)−W0=W0​diag⁡\(ℓ−𝟏\)W\_\{0\}\\diag\(\\ell\)\-W\_\{0\}=W\_\{0\}\\diag\(\\ell\-\\mathbf\{1\}\)consists of the columns\(ℓj−1\)​cj\(\\ell\_\{j\}\-1\)c\_\{j\}ofW0W\_\{0\}, so its rank is the rank of the columns ofW0W\_\{0\}indexed byJ=\{j:ℓj≠1\}J=\\\{j:\\ell\_\{j\}\\neq 1\\\}\. IfW0W\_\{0\}has full column rank this is\|J\|\\lvert J\\rvert, and the meet is\{W0​diag⁡\(ℓ\):\|J\|≤r\}\\\{W\_\{0\}\\diag\(\\ell\):\\lvert J\\rvert\\leq r\\\}\. ForWdown∈ℝd×dffW\_\{\\rm down\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\rm ff\}\}withd<dffd<d\_\{\\rm ff\}, full column rank is impossible\. For genericW0W\_\{0\}every set of at mostddcolumns is independent \(the failure is the vanishing of all maximal minors of the chosen columns\), so the rank ismin⁡\(\|J\|,d\)\\min\(\\lvert J\\rvert,d\), and whenr<dr<dthe conditionmin⁡\(\|J\|,d\)≤r\\min\(\\lvert J\\rvert,d\)\\leq ris again\|J\|≤r\\lvert J\\rvert\\leq r\.

*No product in \(b\)\.*The commentary after the proposition says that, forn≥2n\\geq 2,LoRAr\\mathrm\{LoRA\}\_\{r\}and the output\-scaling method have no product; the argument is that of[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\. Their set\-level fibre product isℱ=\{\(B,A,ℓ\):W0\+s​B​A=diag⁡\(ℓ\)​W0\}\\mathcal\{F\}=\\\{\(B,A,\\ell\):W\_\{0\}\+sBA=\\diag\(\\ell\)W\_\{0\}\\\}, whatever the pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}\(with residualW0W\_\{0\}\)\. Suppose a productPPexists\. By the proof of[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\) its canonical mapι:P→ℝm×r×ℝr×n×ℝm\\iota:P\\to\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}\\times\\mathbb\{R\}^\{m\}is a bijection ontoℱ\\mathcal\{F\},PPis also the pullback in𝐃𝐢𝐟𝐟\\mathbf\{Diff\}, and smooth curves inℱ\\mathcal\{F\}lift\. Letp∗p^\{\*\}be the point over\(0,0,𝟏\)\(0,0,\\mathbf\{1\}\)\. For everyB′B^\{\\prime\}andA′A^\{\\prime\}the curvest↦\(t​B′,0,𝟏\)t\\mapsto\(tB^\{\\prime\},0,\\mathbf\{1\}\)andt↦\(0,t​A′,𝟏\)t\\mapsto\(0,tA^\{\\prime\},\\mathbf\{1\}\)lie inℱ\\mathcal\{F\}and lift throughp∗p^\{\*\}, so withpr\\mathrm\{pr\}the projection to the LoRA parameters,d​\(pr∘ι\)p∗d\(\\mathrm\{pr\}\\circ\\iota\)\_\{p^\{\*\}\}is ontoℝm×r×ℝr×n\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}\. By[Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(a\),\(c\),pr∘ι\\mathrm\{pr\}\\circ\\iotais a submersion nearp∗p^\{\*\}and its image contains a nonempty open set of pairs\(B,A\)\(B,A\)\. But every pair inpr⁡\(ℱ\)\\mathrm\{pr\}\(\\mathcal\{F\}\)hasB​A=diag⁡\(ℓ−𝟏\)​W0/sBA=\\diag\(\\ell\-\\mathbf\{1\}\)W\_\{0\}/s, so the first row ofB​ABAis parallel to the first roww0,1w\_\{0,1\}ofW0W\_\{0\}: the2×22\\times 2minors of the2×n2\\times nmatrix with rowse1⊤​B​Ae\_\{1\}^\{\\top\}BAandw0,1⊤w\_\{0,1\}^\{\\top\}vanish\. Forn≥2n\\geq 2one of these minors is a polynomial in\(B,A\)\(B,A\)that is not identically zero, so its zero set has empty interior \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(a\)\), a contradiction\. \(Form=n=1m=n=1the fibre product is the graph ofℓ=1\+s​B​A/W0\\ell=1\+sBA/W\_\{0\}, and the product exists\.\) ∎

### C\.7 Theorem II\.1

Theorem II\.1\(pointings of an additive method\)\.LetM=\(Q,q0,θ0\+δ\)M=\(Q,q\_\{0\},\\theta\_\{0\}\+\\delta\)be additive with shapeC=δ⁡\(Q\)C=\\delta\(Q\)\. Write−v\+C\-v\+Cfor a translate, andC−C:=\{c−c′:c,c′∈C\}C\-C:=\\\{c\-c^\{\\prime\}:c,c^\{\\prime\}\\in C\\\}for the Minkowski difference\.\(a\) For eachq∈Qq\\in Q, the re\-pointed methodMq:=\(Q,q,θ0−δ⁡\(q\)\+δ⁡\(⋅\)\)M\_\{q\}:=\(Q,q,\\theta\_\{0\}\-\\delta\(q\)\+\\delta\(\\cdot\)\), with frozen residualθ0−δ⁡\(q\)\\theta\_\{0\}\-\\delta\(q\), is pointed atθ0\\theta\_\{0\}, andIm⁡Mq=θ0\+\(−δ⁡\(q\)\+C\)\\Imop M\_\{q\}=\\theta\_\{0\}\+\(\-\\delta\(q\)\+C\)\.\(b\)⋃qIm⁡Mq=θ0\+\(C−C\)\\bigcup\_\{q\}\\Imop M\_\{q\}=\\theta\_\{0\}\+\(C\-C\)\.\(c\) For LoRAron a single linear box withs≠0s\\neq 0,C−C=ℳ≤min⁡\(2​r,m,n\)C\-C=\\mathcal\{M\}\_\{\\leq\\min\(2r,m,n\)\}; for several boxes,C−CC\-Cis the product of the per\-box varietiesℳ≤min⁡\(2​r,mi,ni\)\\mathcal\{M\}\_\{\\leq\\min\(2r,m\_\{i\},n\_\{i\}\)\}\. Henceno exact pointing of LoRAr\(frozen residualθ0−δ⁡\(q\)\\theta\_\{0\}\-\\delta\(q\)\), whether zero\-init or split \(PiSSA, OLoRA, CorDA, LoRA\-GA\), makes an updateθ−θ0\\theta\-\\theta\_\{0\}of rank\>2​r\>2rreachable in any adapted matrix\.More robustly, from*any*starting pointq0q\_\{0\}of LoRAr, pointed or not,ρ⁡\(q\)−ρ⁡\(q0\)=δ⁡\(q\)−δ⁡\(q0\)\\rho\(q\)\-\\rho\(q\_\{0\}\)=\\delta\(q\)\-\\delta\(q\_\{0\}\)has rank≤min⁡\(2​r,m,n\)\\leq\\min\(2r,m,n\)\. Quantised\-residual schemes \(QLoRA, LoftQ, QPiSSA\) are pointed only up to a defect that depends on the order of operations \(eefor QLoRA;[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(b\) for the others\)\. Measured fromθ0\\theta\_\{0\}rather than from their starting weight, their updates contain this defect, whose rank is generically at leastmin⁡\(m,n\)−r\\min\(m,n\)\-r\. The bound2​r2ris attained by split pointings when2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\.

*\(a\)*MqM\_\{q\}has parameter spaceQQand the smooth mapρq=θ0−δ⁡\(q\)\+δ⁡\(⋅\)\\rho\_\{q\}=\\theta\_\{0\}\-\\delta\(q\)\+\\delta\(\\cdot\), withρq​\(q\)=θ0\\rho\_\{q\}\(q\)=\\theta\_\{0\}, so it is pointed atθ0\\theta\_\{0\}, andIm⁡Mq=θ0−δ⁡\(q\)\+δ⁡\(Q\)=θ0\+\(−δ⁡\(q\)\+C\)\\Imop M\_\{q\}=\\theta\_\{0\}\-\\delta\(q\)\+\\delta\(Q\)=\\theta\_\{0\}\+\(\-\\delta\(q\)\+C\)\.

*\(b\)*Asqqranges overQQ,δ⁡\(q\)\\delta\(q\)ranges overCC, so⋃qIm⁡Mq=θ0\+⋃c′∈C\(C−c′\)=θ0\+\(C−C\)\\bigcup\_\{q\}\\Imop M\_\{q\}=\\theta\_\{0\}\+\\bigcup\_\{c^\{\\prime\}\\in C\}\(C\-c^\{\\prime\}\)=\\theta\_\{0\}\+\(C\-C\)\.

*\(c\)*On one boxC=\{s​B​A\}C=\\\{sBA\\\}\. Sinces≠0s\\neq 0and every matrix of rank at mostmin⁡\(r,m,n\)\\min\(r,m,n\)factors asB​ABAwith inner dimensionrr\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\),C=ℳ≤min⁡\(r,m,n\)C=\\mathcal\{M\}\_\{\\leq\\min\(r,m,n\)\}, which we writeℳ≤r\\mathcal\{M\}\_\{\\leq r\}\. ForX,Y∈CX,Y\\in C,rank⁡\(X−Y\)≤rank⁡X\+rank⁡Y≤2​r\\rk\(X\-Y\)\\leq\\rk X\+\\rk Y\\leq 2randrank⁡\(X−Y\)≤min⁡\(m,n\)\\rk\(X\-Y\)\\leq\\min\(m,n\)\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. Conversely, letZZhave rankk≤min⁡\(2​r,m,n\)k\\leq\\min\(2r,m,n\)and SVDZ=∑i≤kσi​ui​vi⊤Z=\\sum\_\{i\\leq k\}\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{\\top\}\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. PutZ1=∑i≤min⁡\(k,r\)σi​ui​vi⊤Z\_\{1\}=\\sum\_\{i\\leq\\min\(k,r\)\}\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{\\top\}andZ2=Z−Z1Z\_\{2\}=Z\-Z\_\{1\}, both of rank at mostrr\. The coneCCis closed under negation, since−B​A=\(−B\)​A\-BA=\(\-B\)A, soZ=Z1−\(−Z2\)∈C−CZ=Z\_\{1\}\-\(\-Z\_\{2\}\)\\in C\-C\. On several boxes, each with its own factors,δ\\deltais the product of the per\-box maps, soCCis the product of the per\-box shapes andC−CC\-Cthe product of the per\-box differences\.

An exact pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}is a re\-pointed methodMqM\_\{q\}of \(a\), so its image lies inθ0\+\(C−C\)\\theta\_\{0\}\+\(C\-C\), and in every box the updateθ−θ0\\theta\-\\theta\_\{0\}has rank at mostmin⁡\(2​r,mb,nb\)\\min\(2r,m\_\{b\},n\_\{b\}\)\. For an arbitrary starting pointq0q\_\{0\}and an arbitrary residualWresW\_\{\\rm res\},ρ⁡\(q\)−ρ⁡\(q0\)=δ⁡\(q\)−δ⁡\(q0\)∈C−C\\rho\(q\)\-\\rho\(q\_\{0\}\)=\\delta\(q\)\-\\delta\(q\_\{0\}\)\\in C\-Cwith no condition at all\.

For a pipeline with pointing defectept=ρ⁡\(q0\)−θ0e\_\{\\rm pt\}=\\rho\(q\_\{0\}\)\-\\theta\_\{0\}, the update measured fromθ0\\theta\_\{0\}is

θ−θ0=\(ρ⁡\(q\)−ρ⁡\(q0\)\)\+ept,\\theta\-\\theta\_\{0\}=\\big\(\\rho\(q\)\-\\rho\(q\_\{0\}\)\\big\)\+e\_\{\\rm pt\},a matrix of rank at most2​r2rplus the defect\. The defects are those of[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2):e=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}for QLoRA,e−s​SVDr​\(e\)e\-s\\,\\mathrm\{SVD\}\_\{r\}\(e\)for one\-step LoftQ, andκ⁡\(Wres\)−Wres\\kappa\(W\_\{\\rm res\}\)\-W\_\{\\rm res\}for QPiSSA\. Each is a quantisation error, or a quantisation error minus a matrix of rank at mostrr, so it has rank at leastmin⁡\(m,n\)−r\\min\(m,n\)\-rwhenever the quantisation error involved has full rank \([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. Full rank is the generic case recorded in[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(a\)\. For instance, a round\-to\-nearest quantiser with fixed scales is constant on each open cell of constant codes, so theree⁡\(W\)=c−We\(W\)=c\-W, which is rank\-deficient only on a translate ofℳ≤min⁡\(m,n\)−1\\mathcal\{M\}\_\{\\leq\\min\(m,n\)\-1\}, a proper algebraic subset of the cell; the cells cover all weights outside a null set \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\)\.

Attainment: let2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\), and take a split pointing withP=s​B0​A0P=sB\_\{0\}A\_\{0\}of rankrr\. ChooseY=s​B​AY=sBAof rankrrwithcol⁡Y∩col⁡P=0\\operatorname\{col\}Y\\cap\\operatorname\{col\}P=0androw⁡Y∩row⁡P=0\\operatorname\{row\}Y\\cap\\operatorname\{row\}P=0, possible since2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\. WritingY=UY​VY⊤Y=U\_\{Y\}V\_\{Y\}^\{\\top\}andP=UP​VP⊤P=U\_\{P\}V\_\{P\}^\{\\top\}with factors of full column rankrr, we haveY−P=\[UY​UP\]​\[VY−VP\]⊤Y\-P=\[U\_\{Y\}\\ U\_\{P\}\]\\,\[V\_\{Y\}\\ \{\-V\_\{P\}\}\]^\{\\top\}, and bothm×2​rm\\times 2randn×2​rn\\times 2rfactors have independent columns, sorank⁡\(Y−P\)=2​r\\rk\(Y\-P\)=2r\. The weightθ0−P\+Y\\theta\_\{0\}\-P\+Ylies in the image of the split pointing, with updateY−PY\-Pof rank2​r2r\. ∎

### C\.8 Theorem II\.2

Theorem II\.2\(three strata of LoRA initialisations\)\.Letδ⁡\(B,A\)=B​A\\delta\(B,A\)=BAwith1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)\. Consider pointed LoRA objects in three distinguished strata of full\-rank pointings \(pointings whose nonzero factors have full rank\)\. OnlyS\\mathrm\{S\}is open and dense in the space of pointings;TA\\mathrm\{T\}\_\{A\}andTB\\mathrm\{T\}\_\{B\}are nowhere dense\.•\(TA\)\(\\mathrm\{T\}\_\{A\}\):B0=0B\_\{0\}=0andrank⁡A0=r\\rk A\_\{0\}=r;•\(TB\)\(\\mathrm\{T\}\_\{B\}\):A0=0A\_\{0\}=0andrank⁡B0=r\\rk B\_\{0\}=r;•\(S\)\(\\mathrm\{S\}\):rank⁡B0​A0=r\\rk B\_\{0\}A\_\{0\}=r, with frozen residualθ0−B0​A0\\theta\_\{0\}\-B\_\{0\}A\_\{0\}\.\(a\)Classification\.Within a stratum,Mq0≅Mq0′M\_\{q\_\{0\}\}\\cong M\_\{q\_\{0\}^\{\\prime\}\}if and only if, respectively,row⁡A0=row⁡A0′\\operatorname\{row\}A\_\{0\}=\\operatorname\{row\}A\_\{0\}^\{\\prime\};col⁡B0=col⁡B0′\\operatorname\{col\}B\_\{0\}=\\operatorname\{col\}B\_\{0\}^\{\\prime\}; orB0​A0=B0′​A0′B\_\{0\}A\_\{0\}=B\_\{0\}^\{\\prime\}A\_\{0\}^\{\\prime\}\. Objects in different strata are never isomorphic\. As a*set*, the isomorphism classes of the three strata therefore formGr⁡\(r,ℝn\)⊔Gr⁡\(r,ℝm\)⊔ℳr​\(m×n\)\\Gr\(r,\\mathbb\{R\}^\{n\}\)\\sqcup\\Gr\(r,\\mathbb\{R\}^\{m\}\)\\sqcup\\mathcal\{M\}\_\{r\}\(m\\times n\)\.\(b\)First order\.dimS1\\dim S\_\{1\}ism​rmr,n​rnrandr⁡\(m\+n−r\)r\(m\+n\-r\)respectively, whiled=r⁡\(m\+n−r\)d=r\(m\+n\-r\)in every case\. The deficits arer⁡\(n−r\)r\(n\-r\),r⁡\(m−r\)r\(m\-r\)and00\.\(c\)Geometry\.InTA\\mathrm\{T\}\_\{A\}andTB\\mathrm\{T\}\_\{B\},Im=θ0\+ℳ≤r\\Imop=\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, whose vertexθ0\\theta\_\{0\}is anapex\. InS\\mathrm\{S\},Im=\(θ0−P\)\+ℳ≤r\\Imop=\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}withP=B0​A0P=B\_\{0\}A\_\{0\}, andθ0\\theta\_\{0\}is asmooth point\.\(d\)Incomparability\.The images ofS\\mathrm\{S\}\-type andT\\mathrm\{T\}\-type objects are incomparable\.\(e\)Scope\.The three strata do not exhaust the pointings of LoRAr\. Outside them lie:B0=0B\_\{0\}=0withrank⁡A0<r\\rk A\_\{0\}<r;A0=0A\_\{0\}=0withrank⁡B0<r\\rk B\_\{0\}<r;B0=A0=0B\_\{0\}=A\_\{0\}=0;0<rank⁡B0​A0<r0<\\rk B\_\{0\}A\_\{0\}<r; and pointings withB0,A0B\_\{0\},A\_\{0\}both nonzero andB0​A0=0B\_\{0\}A\_\{0\}=0\. HF PEFT ships one of the last kind: init\_lora\_weights="orthogonal" \(which requiresrreven\) lies in the stratum\(rk⁡B0,rk⁡A0,rk⁡B0​A0\)=\(r/2,r/2,0\)\(\\operatorname\{rk\}B\_\{0\},\\operatorname\{rk\}A\_\{0\},\\operatorname\{rk\}B\_\{0\}A\_\{0\}\)=\(r/2,r/2,0\), an apex withdimS1=r2​\(m\+n−r2\)\\dim S\_\{1\}=\\tfrac\{r\}\{2\}\(m\+n\-\\tfrac\{r\}\{2\}\)\. It is isomorphic to no object of the three strata\. Its imageθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}rules outS\\mathrm\{S\}, and itsS1S\_\{1\}determines\(col⁡B0,row⁡A0\)\(\\operatorname\{col\}B\_\{0\},\\operatorname\{row\}A\_\{0\}\), of dimensions\(r/2,r/2\)\(r/2,r/2\), against\(0,r\)\(0,r\)inTA\\mathrm\{T\}\_\{A\}and\(r,0\)\(r,0\)inTB\\mathrm\{T\}\_\{B\}\. The dimension ofS1S\_\{1\}alone does not separate it, since at\(m,n,r\)=\(4,5,2\)\(m,n,r\)=\(4,5,2\)it equalsm​rmr\. This placement holds in exact arithmetic; HF casts the two factors to the base dtype separately, and in bf16 the result is a generic pointing with a tiny defect\. For every pointing, the image determinesP=B0​A0P=B\_\{0\}A\_\{0\}\(its vertex isθ0−P\\theta\_\{0\}\-P\), andS1=\{X​A0\+B0​Y\}S\_\{1\}=\\\{XA\_\{0\}\+B\_\{0\}Y\\\}determines\(row⁡A0,col⁡B0\)\(\\operatorname\{row\}A\_\{0\},\\operatorname\{col\}B\_\{0\}\)whenever these are proper subspaces\. The classification is set\-level: in the space of pointingsTA\\mathrm\{T\}\_\{A\}lies in the closure ofS\\mathrm\{S\}, since\(ε​B′,A0\)→\(0,A0\)\(\\varepsilon B^\{\\prime\},A\_\{0\}\)\\to\(0,A\_\{0\}\), so the quotient is not a topological disjoint union\.

Absorb the scalessintoBB; this changes neither the strata nor the images\. The space of pointings isℝm×r×ℝr×n\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}, and the residual is determined by the pointing\.

*Openness\.*rank⁡B0​A0≤min⁡\(rank⁡B0,rank⁡A0\)\\rk B\_\{0\}A\_\{0\}\\leq\\min\(\\rk B\_\{0\},\\rk A\_\{0\}\), and if both factors have rankrrthenB0B\_\{0\}is injective andA0A\_\{0\}surjective, sorank⁡B0​A0=r\\rk B\_\{0\}A\_\{0\}=r\. HenceS=\{rankB0=rankA0=r\}\\mathrm\{S\}=\\\{\\rk B\_\{0\}=\\rk A\_\{0\}=r\\\}is the set where somer×rr\\times rminor ofB0​A0B\_\{0\}A\_\{0\}is nonzero: it is open, and dense because its complement is a proper algebraic subset \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(a\)\)\. The stratumTA\\mathrm\{T\}\_\{A\}lies in the proper linear subspace\{B0=0\}\\\{B\_\{0\}=0\\\}, which is closed with empty interior, soTA\\mathrm\{T\}\_\{A\}is nowhere dense, and likewiseTB\\mathrm\{T\}\_\{B\}\.

*A translation argument\.*We use the following fact here and in Theorem II\.11 and Proposition II\.16\. Let1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)andD≠0D\\neq 0, of rankkk\. Ifk≥r\+1k\\geq r\+1putX=0X=0\. Otherwise chooseXXof rankr\+1−k≤rr\+1\-k\\leq rwithcol⁡X∩col⁡D=0\\operatorname\{col\}X\\cap\\operatorname\{col\}D=0androw⁡X∩row⁡D=0\\operatorname\{row\}X\\cap\\operatorname\{row\}D=0, possible sincer\+1≤min⁡\(m,n\)r\+1\\leq\\min\(m,n\)\. FactoringD=UD​VD⊤D=U\_\{D\}V\_\{D\}^\{\\top\}andX=UX​VX⊤X=U\_\{X\}V\_\{X\}^\{\\top\}with factors of full column rank,D\+X=\[UD​UX\]​\[VD​VX\]⊤D\+X=\[U\_\{D\}\\ U\_\{X\}\]\[V\_\{D\}\\ V\_\{X\}\]^\{\\top\}with both factors of full column rank, sorank⁡\(D\+X\)=r\+1\\rk\(D\+X\)=r\+1\. In either caseX∈ℳ≤rX\\in\\mathcal\{M\}\_\{\\leq r\}andD\+X∉ℳ≤rD\+X\\notin\\mathcal\{M\}\_\{\\leq r\}\. Consequently,

c\+ℳ≤r=c′\+ℳ≤r⟹c=c′,c\+\\mathcal\{M\}\_\{\\leq r\}=c^\{\\prime\}\+\\mathcal\{M\}\_\{\\leq r\}\\quad\\Longrightarrow\\quad c=c^\{\\prime\},\(18\)since otherwise, withD=c−c′D=c\-c^\{\\prime\}, the pointc\+X=c′\+\(D\+X\)c\+X=c^\{\\prime\}\+\(D\+X\)lies in the left side and not in the right side\. The translation stabiliser ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}is\{0\}\\\{0\\\}\.

*\(a\), “if”\.*InTA\\mathrm\{T\}\_\{A\}, letrow⁡A0=row⁡A0′\\operatorname\{row\}A\_\{0\}=\\operatorname\{row\}A\_\{0\}^\{\\prime\}with both of rankrr\. ThenA0′=A0′​A0\+​A0=g​A0A\_\{0\}^\{\\prime\}=A\_\{0\}^\{\\prime\}A\_\{0\}^\{\+\}A\_\{0\}=gA\_\{0\}withg=A0′​A0\+∈ℝr×rg=A\_\{0\}^\{\\prime\}A\_\{0\}^\{\+\}\\in\\mathbb\{R\}^\{r\\times r\}, invertible becauserank⁡A0′=r\\rk A\_\{0\}^\{\\prime\}=r\. The mapφg​\(B,A\)=\(B​g−1,g​A\)\\varphi\_\{g\}\(B,A\)=\(Bg^\{\-1\},gA\)is a diffeomorphism of the parameter space withρ∘φg=ρ\\rho\\circ\\varphi\_\{g\}=\\rho\(both residuals areθ0\\theta\_\{0\}\) andφg​\(0,A0\)=\(0,A0′\)\\varphi\_\{g\}\(0,A\_\{0\}\)=\(0,A\_\{0\}^\{\\prime\}\), a pointed isomorphism overΘ\\Theta\. InTB\\mathrm\{T\}\_\{B\},col⁡B0=col⁡B0′\\operatorname\{col\}B\_\{0\}=\\operatorname\{col\}B\_\{0\}^\{\\prime\}givesB0′=B0​kB\_\{0\}^\{\\prime\}=B\_\{0\}kwithk∈GLrk\\in\\mathrm\{GL\}\_\{r\}, andφk−1​\(B0,0\)=\(B0′,0\)\\varphi\_\{k^\{\-1\}\}\(B\_\{0\},0\)=\(B\_\{0\}^\{\\prime\},0\)\. InS\\mathrm\{S\}, letB0​A0=B0′​A0′=PB\_\{0\}A\_\{0\}=B\_\{0\}^\{\\prime\}A\_\{0\}^\{\\prime\}=Pof rankrr\. Thencol⁡B0=col⁡P=col⁡B0′\\operatorname\{col\}B\_\{0\}=\\operatorname\{col\}P=\\operatorname\{col\}B\_\{0\}^\{\\prime\}, soB0′=B0​kB\_\{0\}^\{\\prime\}=B\_\{0\}kfor a uniquek∈GLrk\\in\\mathrm\{GL\}\_\{r\}, andB0​k​A0′=B0​A0B\_\{0\}kA\_\{0\}^\{\\prime\}=B\_\{0\}A\_\{0\}givesk​A0′=A0kA\_\{0\}^\{\\prime\}=A\_\{0\}becauseB0B\_\{0\}is injective\. So\(B0′,A0′\)=φk−1​\(B0,A0\)\(B\_\{0\}^\{\\prime\},A\_\{0\}^\{\\prime\}\)=\\varphi\_\{k^\{\-1\}\}\(B\_\{0\},A\_\{0\}\), and the residualsθ0−P\\theta\_\{0\}\-Pagree\.

*\(a\), “only if”\.*InTA\\mathrm\{T\}\_\{A\},d​ρ\(0,A0\)​\(B˙,A˙\)=B˙​A0d\\rho\_\{\(0,A\_\{0\}\)\}\(\\dot\{B\},\\dot\{A\}\)=\\dot\{B\}A\_\{0\}, soS1=\{X​A0\}=ℛ⁡\(row⁡A0\)S\_\{1\}=\\\{XA\_\{0\}\\\}=\\mathcal\{R\}\(\\operatorname\{row\}A\_\{0\}\), which determinesrow⁡A0\\operatorname\{row\}A\_\{0\}\(proof of[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\); pointed isomorphisms preserveS1S\_\{1\}\([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. InTB\\mathrm\{T\}\_\{B\},S1=𝒞⁡\(col⁡B0\)S\_\{1\}=\\mathcal\{C\}\(\\operatorname\{col\}B\_\{0\}\)determinescol⁡B0\\operatorname\{col\}B\_\{0\}\. InS\\mathrm\{S\}the image is\(θ0−P\)\+ℳ≤r\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}by \(c\), it is an isomorphism invariant, and by \([18](https://arxiv.org/html/2610.12728#A3.E18)\) it determinesPP\.

*Different strata\.*S1​\(TA\)=ℛ⁡\(V\)S\_\{1\}\(\\mathrm\{T\}\_\{A\}\)=\\mathcal\{R\}\(V\)withdimV=r≥1\\dim V=r\\geq 1andS1​\(TB\)=𝒞⁡\(U\)S\_\{1\}\(\\mathrm\{T\}\_\{B\}\)=\\mathcal\{C\}\(U\)withdimU=r<m\\dim U=r<m\. They differ by the second fact in the proof of[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\. \(Whenm≠nm\\neq ntheir dimensionsm​rmrandn​rnralready differ\.\)S\\mathrm\{S\}differs from both becausedimS1​\(S\)=r⁡\(m\+n−r\)\\dim S\_\{1\}\(\\mathrm\{S\}\)=r\(m\+n\-r\)by \(b\), which exceedsm​rmrbyr⁡\(n−r\)\>0r\(n\-r\)\>0andn​rnrbyr⁡\(m−r\)\>0r\(m\-r\)\>0\. Isomorphism classes inTA\\mathrm\{T\}\_\{A\}correspond torow⁡A0\\operatorname\{row\}A\_\{0\}, and everyrr\-dimensional subspace occurs; inTB\\mathrm\{T\}\_\{B\}tocol⁡B0\\operatorname\{col\}B\_\{0\}; inS\\mathrm\{S\}toPP, and every rank\-rrmatrix has a full\-rank factorisation\. This gives the stated disjoint union of sets\.

*\(b\)*At\(B0,A0\)\(B\_\{0\},A\_\{0\}\),S1=\{X​A0\+B0​Y\}S\_\{1\}=\\\{XA\_\{0\}\+B\_\{0\}Y\\\}, of dimensionm​rank​A0\+n​rank​B0−rank⁡A0​rank​B0m\\rk A\_\{0\}\+n\\rk B\_\{0\}\-\\rk A\_\{0\}\\rk B\_\{0\}by[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3):m​rmrinTA\\mathrm\{T\}\_\{A\},n​rnrinTB\\mathrm\{T\}\_\{B\}andr⁡\(m\+n−r\)r\(m\+n\-r\)inS\\mathrm\{S\}\. In every case the image is a translate ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}by \(c\), of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(a\)\), sod=r⁡\(m\+n−r\)d=r\(m\+n\-r\), and the deficits arer⁡\(n−r\)r\(n\-r\),r⁡\(m−r\)r\(m\-r\)and00\.

*\(c\)*InTA\\mathrm\{T\}\_\{A\}andTB\\mathrm\{T\}\_\{B\}the residual isθ0\\theta\_\{0\}and the image isθ0\+\{B​A\}=θ0\+ℳ≤r\\theta\_\{0\}\+\\\{BA\\\}=\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. This is a cone with vertexθ0\\theta\_\{0\}, and it is not an affine subspace, because∑i≤rEi​i\\sum\_\{i\\leq r\}E\_\{ii\}andEr\+1,r\+1E\_\{r\+1,r\+1\}lie inℳ≤r\\mathcal\{M\}\_\{\\leq r\}while their sum has rankr\+1≤min⁡\(m,n\)r\+1\\leq\\min\(m,n\)\. By the cone part of[Definition I\.5](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\([Auxiliary result B\.6](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\),θ0\\theta\_\{0\}is a singular point, and by \(b\) the deficit is positive, soθ0\\theta\_\{0\}is an apex\. InS\\mathrm\{S\}the image is\(θ0−P\)\+ℳ≤r\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}, andθ0=\(θ0−P\)\+P\\theta\_\{0\}=\(\\theta\_\{0\}\-P\)\+PwithP∈ℳrP\\in\\mathcal\{M\}\_\{r\}\. The stratumℳr\\mathcal\{M\}\_\{r\}is open inℳ≤r\\mathcal\{M\}\_\{\\leq r\}and an embedded submanifold of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\), so the image is aC1C^\{1\}submanifold nearθ0\\theta\_\{0\}, which is a smooth point; its deficit is00\.

*\(d\)*TheT\\mathrm\{T\}\-image isθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}and theS\\mathrm\{S\}\-image\(θ0−P\)\+ℳ≤r\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}withP≠0P\\neq 0\. The translation argument withD=PD=PgivesX∈ℳ≤rX\\in\\mathcal\{M\}\_\{\\leq r\}withθ0\+X\\theta\_\{0\}\+Xoutside theS\\mathrm\{S\}\-image, and withD=−PD=\-Pit givesX′∈ℳ≤rX^\{\\prime\}\\in\\mathcal\{M\}\_\{\\leq r\}withθ0−P\+X′\\theta\_\{0\}\-P\+X^\{\\prime\}outside theT\\mathrm\{T\}\-image\. \(For a generic rank\-oneXX,rank⁡\(P\+X\)=rank⁡\(X−P\)=r\+1\\rk\(P\+X\)=\\rk\(X\-P\)=r\+1already\.\)

*\(e\)*Every pointing has a rank triple\(rank⁡B0,rank⁡A0,rank⁡B0​A0\)\(\\rk B\_\{0\},\\rk A\_\{0\},\\rk B\_\{0\}A\_\{0\}\)\. The strata are\(0,r,0\)\(0,r,0\),\(r,0,0\)\(r,0,0\)and\(r,r,r\)\(r,r,r\)\. IfB0=0B\_\{0\}=0andA0≠0A\_\{0\}\\neq 0is not of rankrr, orA0=0A\_\{0\}=0andB0≠0B\_\{0\}\\neq 0is not of rankrr, orB0=A0=0B\_\{0\}=A\_\{0\}=0, the pointing is in the first three listed families\. Otherwise both factors are nonzero, and eitherrank⁡B0​A0=r\\rk B\_\{0\}A\_\{0\}=r, which forces both ranks to berrand givesS\\mathrm\{S\}, or0<rank⁡B0​A0<r0<\\rk B\_\{0\}A\_\{0\}<r, orB0​A0=0B\_\{0\}A\_\{0\}=0\. So the list is exhaustive\.

For every pointing,S1=\{X​A0\+B0​Y\}=\{Z:Z​\(row⁡A0\)⟂⊆col⁡B0\}S\_\{1\}=\\\{XA\_\{0\}\+B\_\{0\}Y\\\}=\\\{Z:Z\(\\operatorname\{row\}A\_\{0\}\)^\{\\perp\}\\subseteq\\operatorname\{col\}B\_\{0\}\\\}\. Indeed,\(X​A0\+B0​Y\)​w=B0​Y​w\(XA\_\{0\}\+B\_\{0\}Y\)w=B\_\{0\}Ywforw∈ker⁡A0=\(row⁡A0\)⟂w\\in\\ker A\_\{0\}=\(\\operatorname\{row\}A\_\{0\}\)^\{\\perp\}; conversely, ifZ​ker⁡A0⊆col⁡B0Z\\ker A\_\{0\}\\subseteq\\operatorname\{col\}B\_\{0\}, thenZ=Z​A0\+​A0\+B0​B0\+​Z​\(I−A0\+​A0\)Z=ZA\_\{0\}^\{\+\}A\_\{0\}\+B\_\{0\}B\_\{0\}^\{\+\}Z\(I\-A\_\{0\}^\{\+\}A\_\{0\}\)\. Its annihilator in the Frobenius inner product is\{W:WA0⊤=0,B0⊤W=0\}\\\{W:WA\_\{0\}^\{\\top\}=0,\\ B\_\{0\}^\{\\top\}W=0\\\}, the matrices with column space in\(col⁡B0\)⟂\(\\operatorname\{col\}B\_\{0\}\)^\{\\perp\}and row space in\(row⁡A0\)⟂\(\\operatorname\{row\}A\_\{0\}\)^\{\\perp\}, that is\(col⁡B0\)⟂⊗\(row⁡A0\)⟂\(\\operatorname\{col\}B\_\{0\}\)^\{\\perp\}\\otimes\(\\operatorname\{row\}A\_\{0\}\)^\{\\perp\}\. When both subspaces are proper, both complements are nonzero, and the column spaces of the elements of the annihilator span\(col⁡B0\)⟂\(\\operatorname\{col\}B\_\{0\}\)^\{\\perp\}while their row spaces span\(row⁡A0\)⟂\(\\operatorname\{row\}A\_\{0\}\)^\{\\perp\}\. SoS1S\_\{1\}determines\(col⁡B0,row⁡A0\)\(\\operatorname\{col\}B\_\{0\},\\operatorname\{row\}A\_\{0\}\)and their dimensions, and so does every pointed isomorphism class \([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. The image is\(θ0−P\)\+ℳ≤r\(\\theta\_\{0\}\-P\)\+\\mathcal\{M\}\_\{\\leq r\}withP=B0​A0P=B\_\{0\}A\_\{0\}\(residualθ0−P\\theta\_\{0\}\-P\), and it determinesPPby \([18](https://arxiv.org/html/2610.12728#A3.E18)\)\.

HF’s orthogonal initialisation draws oneQ∈O⁡\(r\)Q\\in\\mathrm\{O\}\(r\)and setsA0=\(GAQ\[0::2\]\)⊤/10A\_\{0\}=\(G\_\{A\}Q\_\{\[0::2\]\}\)^\{\\top\}/10andB0=GBQ\[1::2\]/10B\_\{0\}=G\_\{B\}Q\_\{\[1::2\]\}/10, withGA∈ℝn×r/2G\_\{A\}\\in\\mathbb\{R\}^\{n\\times r/2\}andGB∈ℝm×r/2G\_\{B\}\\in\\mathbb\{R\}^\{m\\times r/2\}random andQ\[0::2\]Q\_\{\[0::2\]\},Q\[1::2\]Q\_\{\[1::2\]\}the even\- and odd\-indexed rows ofQQ\. Distinct rows ofQQare orthogonal, soB0A0=GBQ\[1::2\]Q\[0::2\]⊤GA⊤/100=0B\_\{0\}A\_\{0\}=G\_\{B\}Q\_\{\[1::2\]\}Q\_\{\[0::2\]\}^\{\\top\}G\_\{A\}^\{\\top\}/100=0, and both factors have rankr/2r/2for genericGA,GBG\_\{A\},G\_\{B\}\. The residual isθ0\\theta\_\{0\}and the image isθ0\+ℳ≤r\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, whose vertex is singular\. By[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3),dimS1=m​r2\+n​r2−r24=r2​\(m\+n−r2\)\\dim S\_\{1\}=m\\tfrac\{r\}\{2\}\+n\\tfrac\{r\}\{2\}\-\\tfrac\{r^\{2\}\}\{4\}=\\tfrac\{r\}\{2\}\(m\+n\-\\tfrac\{r\}\{2\}\), and the deficitr⁡\(m\+n−r\)−r2​\(m\+n−r2\)=r2​\(m\+n−3​r2\)r\(m\+n\-r\)\-\\tfrac\{r\}\{2\}\(m\+n\-\\tfrac\{r\}\{2\}\)=\\tfrac\{r\}\{2\}\(m\+n\-\\tfrac\{3r\}\{2\}\)is positive, so the base point is an apex\. Its image differs from everyS\\mathrm\{S\}\-image by \([18](https://arxiv.org/html/2610.12728#A3.E18)\), and its pair of subspace dimensions\(r/2,r/2\)\(r/2,r/2\), determined byS1S\_\{1\}since both subspaces are proper, differs from\(0,r\)\(0,r\)and\(r,0\)\(r,0\); so it is isomorphic to no object of the three strata\. At\(m,n,r\)=\(4,5,2\)\(m,n,r\)=\(4,5,2\),r2​\(m\+n−r2\)=8=m​r\\tfrac\{r\}\{2\}\(m\+n\-\\tfrac\{r\}\{2\}\)=8=mr\.

Finally, forB′B^\{\\prime\}of rankrrandε≠0\\varepsilon\\neq 0,\(ε​B′,A0\)∈S\(\\varepsilon B^\{\\prime\},A\_\{0\}\)\\in\\mathrm\{S\}, and\(ε​B′,A0\)→\(0,A0\)∈TA\(\\varepsilon B^\{\\prime\},A\_\{0\}\)\\to\(0,A\_\{0\}\)\\in\\mathrm\{T\}\_\{A\}asε→0\\varepsilon\\to 0, soTA\\mathrm\{T\}\_\{A\}lies in the closure ofS\\mathrm\{S\}\. ∎

### C\.9 Proposition II\.3

Proposition II\.3\(first\-order rank formula: smoothness, alignment and scale\)\.Forκ⁡\(B,A\)=Wbase\+s​B​A\\kappa\(B,A\)=W\_\{\\rm base\}\+sBAwiths≠0s\\neq 0,rank⁡d​κ\(B,A\)=m​rk​A\+n​rk​B−rk⁡A​rk​Bfor all​\(B,A\),\\rk d\\kappa\_\{\(B,A\)\}=m\\operatorname\{rk\}A\+n\\operatorname\{rk\}B\-\\operatorname\{rk\}A\\operatorname\{rk\}B\\qquad\\text\{for all \}\(B,A\),and\{X​A\}∩\{B​Y\}=\{B​R​A:R∈ℝr×r\}\\\{XA\\\}\\cap\\\{BY\\\}=\\\{BRA:R\\in\\mathbb\{R\}^\{r\\times r\}\\\}\. For full\-rank frames this overlap is the tangent space of the productFA⁡\(A\)×FB⁡\(B\)\\mathrm\{FA\}\(A\)\\times\\mathrm\{FB\}\(B\)of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\), re\-based atκ⁡\(B,A\)\\kappa\(B,A\), whose image is the meet of the two one\-sided images: a generalised LoRA\-XS\. It is LoRA\-XS proper when\(B,A\)\(B,A\)span the top\-rrsingular subspaces ofW0W\_\{0\}\(σr\>0\\sigma\_\{r\}\>0\), as at a PiSSA init; bottom frames \(MiLoRA\) give a generalised LoRA\-XS\.Under gradient flowB˙=−ηB​∇BL\\dot\{B\}=\-\\eta\_\{B\}\\nabla\_\{B\}L,A˙=−ηA​∇AL\\dot\{A\}=\-\\eta\_\{A\}\\nabla\_\{A\}L, withG=∇WLG=\\nabla\_\{W\}L,dd​t​L=−s2​\(ηB​‖G​A⊤‖F2\+ηA​‖B⊤​G‖F2\)\\tfrac\{d\}\{dt\}L=\-s^\{2\}\\big\(\\eta\_\{B\}\\\|GA^\{\\top\}\\\|\_\{F\}^\{2\}\+\\eta\_\{A\}\\\|B^\{\\top\}G\\\|\_\{F\}^\{2\}\\big\)at every time, in particular at\(B0,A0\)\(B\_\{0\},A\_\{0\}\)\. To first order in the step sizes the same holds for full\-batch gradient descent, and in expectation for unbiased minibatch SGD\. Under Adam \(ε→0\\varepsilon\\to 0, with bias correction\) the first step changes the loss by−\|s\|​\(ηB​‖G​A0⊤‖1\+ηA​‖B0⊤​G‖1\)\+O⁡\(η2\),\-\\lvert s\\rvert\\big\(\\eta\_\{B\}\\\|GA\_\{0\}^\{\\top\}\\\|\_\{1\}\+\\eta\_\{A\}\\\|B\_\{0\}^\{\\top\}G\\\|\_\{1\}\\big\)\+O\(\\eta^\{2\}\),with entrywiseℓ1\\ell\_\{1\}norms\. This first\-order term is linear in\|s\|\\lvert s\\rvert, against quadratic under gradient flow\.

*Rank\.*d​κ\(B,A\)​\(B˙,A˙\)=s⁡\(B˙​A\+B​A˙\)d\\kappa\_\{\(B,A\)\}\(\\dot\{B\},\\dot\{A\}\)=s\(\\dot\{B\}A\+B\\dot\{A\}\), ands≠0s\\neq 0, so the image of the differential is\{X​A\}\+\{B​Y\}=ℛ⁡\(row⁡A\)\+𝒞⁡\(col⁡B\)\\\{XA\\\}\+\\\{BY\\\}=\\mathcal\{R\}\(\\operatorname\{row\}A\)\+\\mathcal\{C\}\(\\operatorname\{col\}B\)\. Witha=rank⁡Aa=\\rk Aandb=rank⁡Bb=\\rk B, the two summands have dimensionsm​amaandn​bnb\(proof of[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\)\. A matrixDDlies in both iffrow⁡D⊆row⁡A\\operatorname\{row\}D\\subseteq\\operatorname\{row\}Aandcol⁡D⊆col⁡B\\operatorname\{col\}D\\subseteq\\operatorname\{col\}B\. The space of suchDDiscol⁡B⊗row⁡A\\operatorname\{col\}B\\otimes\\operatorname\{row\}A, of dimensiona​bab, and every suchDDsatisfiesD=B​B\+​D​A\+​A=B⁡\(B\+​D​A\+\)​AD=BB^\{\+\}DA^\{\+\}A=B\(B^\{\+\}DA^\{\+\}\)A, sinceB​B\+BB^\{\+\}andA\+​AA^\{\+\}Aare the projectors ontocol⁡B\\operatorname\{col\}Bandrow⁡A\\operatorname\{row\}A\. ConverselyB​R​ABRAlies in both summands\. So\{X​A\}∩\{B​Y\}=\{B​R​A\}\\\{XA\\\}\\cap\\\{BY\\\}=\\\{BRA\\\}andrank⁡d​κ=m​a\+n​b−a​b\\rk d\\kappa=ma\+nb\-ab, for every\(B,A\)\(B,A\)\.

*The overlap\.*LetBBhave full column rank andAAfull row rank\. Re\-based atW=κ⁡\(B,A\)W=\\kappa\(B,A\), the one\-sided methodsX↦W\+X​AX\\mapsto W\+XAandY↦W\+s​B​YY\\mapsto W\+sBYare those of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\) with framesAAands​BsB, so their product isR↦W\+s​B​R​AR\\mapsto W\+sBRA, a linear method whose imageW\+\{B​R​A\}W\+\\\{BRA\\\}is its own tangent space atWW\. That image is the meet\(W\+\{X​A\}\)∩\(W\+\{B​Y\}\)\(W\+\\\{XA\\\}\)\\cap\(W\+\\\{BY\\\}\)by the identity just proved\. Ifcol⁡B=col⁡Ur\\operatorname\{col\}B=\\operatorname\{col\}U\_\{r\}androw⁡A=row⁡Vr⊤\\operatorname\{row\}A=\\operatorname\{row\}V\_\{r\}^\{\\top\}for the top\-rrsingular frames ofW0W\_\{0\}withσr\>0\\sigma\_\{r\}\>0, thenB=Ur​k1B=U\_\{r\}k\_\{1\}andA=k2​Vr⊤A=k\_\{2\}V\_\{r\}^\{\\top\}withk1,k2∈GLrk\_\{1\},k\_\{2\}\\in\\mathrm\{GL\}\_\{r\}, and\{B​R​A\}=\{Ur​R′​Σr​Vr⊤\}\\\{BRA\\\}=\\\{U\_\{r\}R^\{\\prime\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\\\}withR′=k1​R​k2​Σr−1R^\{\\prime\}=k\_\{1\}Rk\_\{2\}\\Sigma\_\{r\}^\{\-1\}ranging overℝr×r\\mathbb\{R\}^\{r\\times r\}, which is LoRA\-XS’s update space \([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)\)\. For other frames, such as the bottom singular frames of MiLoRA, the overlap is the same kind of sandwich in those frames\.

*Gradient flow\.*WithW=κ⁡\(B,A\)W=\\kappa\(B,A\), the chain rule gives∇B\(L∘κ\)=s​G​A⊤\\nabla\_\{B\}\(L\\circ\\kappa\)=sGA^\{\\top\}and∇A\(L∘κ\)=s​B⊤​G\\nabla\_\{A\}\(L\\circ\\kappa\)=sB^\{\\top\}G\. Along the flow,

dd​t​L=⟨G,s⁡\(B˙​A\+B​A˙\)⟩=s⁡⟨G​A⊤,B˙⟩\+s⁡⟨B⊤​G,A˙⟩=−s2​\(ηB​‖G​A⊤‖F2\+ηA​‖B⊤​G‖F2\)\.\\tfrac\{d\}\{dt\}L=\\langle G,s\(\\dot\{B\}A\+B\\dot\{A\}\)\\rangle=s\\langle GA^\{\\top\},\\dot\{B\}\\rangle\+s\\langle B^\{\\top\}G,\\dot\{A\}\\rangle=\-s^\{2\}\\big\(\\eta\_\{B\}\\\|GA^\{\\top\}\\\|\_\{F\}^\{2\}\+\\eta\_\{A\}\\\|B^\{\\top\}G\\\|\_\{F\}^\{2\}\\big\)\.One step of full\-batch gradient descent movesWWbys⁡\(Δ​B​A\+B​Δ​A\)\+s​Δ​B​Δ​As\(\\Delta B\\,A\+B\\,\\Delta A\)\+s\\,\\Delta B\\,\\Delta AwithΔ​B=−ηB​s​G​A⊤\\Delta B=\-\\eta\_\{B\}sGA^\{\\top\}andΔ​A=−ηA​s​B⊤​G\\Delta A=\-\\eta\_\{A\}sB^\{\\top\}G\. ForLLof classC2C^\{2\}nearWW, the change of loss is⟨G,s⁡\(Δ​B​A\+B​Δ​A\)⟩\\langle G,s\(\\Delta B\\,A\+B\\,\\Delta A\)\\rangleplus terms of second order in the step sizes, and the first term is the expression above withη\\etain place ofη​d​t\\eta\\,dt\. With an unbiased minibatch gradientG^\\hat\{G\},𝔼​G^=G\\mathbb\{E\}\\hat\{G\}=G, the first\-order change is−s2​\(ηB​⟨G​A⊤,G^​A⊤⟩\+ηA​⟨B⊤​G,B⊤​G^⟩\)\-s^\{2\}\(\\eta\_\{B\}\\langle GA^\{\\top\},\\hat\{G\}A^\{\\top\}\\rangle\+\\eta\_\{A\}\\langle B^\{\\top\}G,B^\{\\top\}\\hat\{G\}\\rangle\), whose expectation is the same expression\.

*Adam\.*With bias correction, Adam’s first moment and second moment after one step arem^1=g\\hat\{m\}\_\{1\}=gandv^1=g⊙g\\hat\{v\}\_\{1\}=g\\odot g, whereggis the first gradient, so the first update is−ηg/\(\|g\|\+ε\)\-\\eta\\,g/\(\\lvert g\\rvert\+\\varepsilon\)entrywise, which tends to−η​sign⁡\(g\)\-\\eta\\operatorname\{sign\}\(g\)asε→0\\varepsilon\\to 0\(entries withg=0g=0do not move\)\. ForBB,g=s​G​A0⊤g=sGA\_\{0\}^\{\\top\}, soδ​B=−ηB​sign⁡\(s\)​sign⁡\(G​A0⊤\)\\delta B=\-\\eta\_\{B\}\\operatorname\{sign\}\(s\)\\operatorname\{sign\}\(GA\_\{0\}^\{\\top\}\), and its first\-order effect on the loss is⟨G,s​δ​B​A0⟩=s⁡⟨G​A0⊤,δ​B⟩=−ηB​\|s\|​‖G​A0⊤‖1\\langle G,s\\,\\delta B\\,A\_\{0\}\\rangle=s\\langle GA\_\{0\}^\{\\top\},\\delta B\\rangle=\-\\eta\_\{B\}\\lvert s\\rvert\\\|GA\_\{0\}^\{\\top\}\\\|\_\{1\}\. TheAA\-term is the same withB0⊤​GB\_\{0\}^\{\\top\}G, and the cross terms​δ​B​δ​As\\,\\delta B\\,\\delta Aand the Taylor remainder areO⁡\(η2\)O\(\\eta^\{2\}\)\.

*The alignment remark after the proposition\.*AmongA0A\_\{0\}withA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cI, writeA0=c​PA\_\{0\}=\\sqrt\{c\}\\,PwithPPof orthonormal rows\. TheBB\-term of the gradient\-flow rate iss2​ηB​‖G​A0⊤‖F2=s2​ηB​c​‖G​P⊤‖F2s^\{2\}\\eta\_\{B\}\\\|GA\_\{0\}^\{\\top\}\\\|\_\{F\}^\{2\}=s^\{2\}\\eta\_\{B\}c\\,\\\|GP^\{\\top\}\\\|\_\{F\}^\{2\}, and by Ky Fan’s maximum principle \([Auxiliary result B\.17](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\) its largest value overPPiss2​ηB​c​∑i≤rσi​\(G\)2s^\{2\}\\eta\_\{B\}c\\sum\_\{i\\leq r\}\\sigma\_\{i\}\(G\)^\{2\}, attained whenrow⁡A0\\operatorname\{row\}A\_\{0\}is a top\-rrright singular subspace ofGG\. Under Adam the first\-step term‖G​A0⊤‖1\\\|GA\_\{0\}^\{\\top\}\\\|\_\{1\}is not invariant underA0↦O​A0A\_\{0\}\\mapsto OA\_\{0\}withO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\), so it depends on the basis ofrow⁡A0\\operatorname\{row\}A\_\{0\}and not only on the subspace\. ∎

### C\.10 Lemma II\.4

Lemma II\.4\(zero gate\)\.LetQ=U×VQ=U\\times VwithUUa vector space, letρ=θ0\+β⁡\(u,v\)\\rho=\\theta\_\{0\}\+\\beta\(u,v\)withβ\\betasmooth, and supposeβ⁡\(u0,v\)=0\\beta\(u\_\{0\},v\)=0for allvv; for exampleβ⁡\(⋅,v\)\\beta\(\\cdot,v\)linear andu0=0u\_\{0\}=0\. ThenDv​ρq0=0D\_\{v\}\\rho\_\{q\_\{0\}\}=0\. Hence∇v\(L∘ρ\)​\(q0\)=0\\nabla\_\{v\}\(L\\circ\\rho\)\(q\_\{0\}\)=0,S1=Du​β​\(u0,v0\)​\(U\)S\_\{1\}=D\_\{u\}\\beta\(u\_\{0\},v\_\{0\}\)\(U\)\(=β⁡\(U,v0\)=\\beta\(U,v\_\{0\}\)in the linear case\), and the cometric atq0q\_\{0\}isKq0=Du​β​Λu​u​Du​β⊤K\_\{q\_\{0\}\}=D\_\{u\}\\beta\\,\\Lambda\_\{uu\}\\,D\_\{u\}\\beta^\{\\top\}, whereΛu​u\\Lambda\_\{uu\}is theu​uuu\-block ofΛ\\Lambda; no block\-diagonality is needed, since thevv\-columns ofD​ρD\\rhovanish\. The weight trajectoryθ⁡\(t\)\\theta\(t\)agrees with that of the method withvvfrozen atv0v\_\{0\}to first order intt\(gradient flow\), or in the step size \(one or more Adam steps\)\. The parametervvitself may still move: under terms not of the formL∘ρL\\circ\\rho\(regularisers, decoupled weight decay\), and under Adam, whose step\-2 move ofvvisO⁡\(η\)O\(\\eta\)although its effect onθ\\thetaisO⁡\(η2\)O\(\\eta^\{2\}\)\.*Function\-level variant\.*For an extension with realised functionF⁡\(u,v,x\)F\(u,v,x\), ifF⁡\(u0,v,x\)=F0​\(x\)F\(u\_\{0\},v,x\)=F\_\{0\}\(x\)for allvvandxx, thenDv​F=0D\_\{v\}F=0atq0q\_\{0\}, with the same conclusions at the level of functions\. No linearity is needed\.

Sinceβ⁡\(u0,v\)=0\\beta\(u\_\{0\},v\)=0for everyvv, differentiating invvgivesDv​β​\(u0,v\)=0D\_\{v\}\\beta\(u\_\{0\},v\)=0for everyvv, in particular atv0v\_\{0\}\. SoD​ρq0=\[Du​β​\(u0,v0\)​0\]D\\rho\_\{q\_\{0\}\}=\[\\,D\_\{u\}\\beta\(u\_\{0\},v\_\{0\}\)\\ \\ 0\\,\]\. Then∇v\(L∘ρ\)\(q0\)=Dvρq0⊤∇L\(θ0\)=0\\nabla\_\{v\}\(L\\circ\\rho\)\(q\_\{0\}\)=D\_\{v\}\\rho\_\{q\_\{0\}\}^\{\\top\}\\nabla L\(\\theta\_\{0\}\)=0,S1=Du​β​\(u0,v0\)​\(U\)S\_\{1\}=D\_\{u\}\\beta\(u\_\{0\},v\_\{0\}\)\(U\), and in the linear caseDu​β​\(0,v0\)​u=β⁡\(u,v0\)D\_\{u\}\\beta\(0,v\_\{0\}\)u=\\beta\(u,v\_\{0\}\)\. For any symmetricΛ\\Lambdawritten in blocks,

Kq0=\[Du​β0\]​\[Λu​uΛu​vΛv​uΛv​v\]​\[Du​β⊤0\]=Du​β​Λu​u​Du​β⊤\.K\_\{q\_\{0\}\}=\\begin\{bmatrix\}D\_\{u\}\\beta&0\\end\{bmatrix\}\\begin\{bmatrix\}\\Lambda\_\{uu\}&\\Lambda\_\{uv\}\\\\ \\Lambda\_\{vu\}&\\Lambda\_\{vv\}\\end\{bmatrix\}\\begin\{bmatrix\}D\_\{u\}\\beta^\{\\top\}\\\\ 0\\end\{bmatrix\}=D\_\{u\}\\beta\\,\\Lambda\_\{uu\}\\,D\_\{u\}\\beta^\{\\top\}\.
*Gradient flow\.*Forq˙=−Λ∇\(L∘ρ\)\(q\)\\dot\{q\}=\-\\Lambda\\nabla\(L\\circ\\rho\)\(q\), the weightsθ⁡\(t\)=ρ⁡\(q⁡\(t\)\)\\theta\(t\)=\\rho\(q\(t\)\)satisfyθ˙\(0\)=−Kq0∇L\(θ0\)\\dot\{\\theta\}\(0\)=\-K\_\{q\_\{0\}\}\\nabla L\(\\theta\_\{0\}\)\. The method withvvfrozen,u↦ρ⁡\(u,v0\)u\\mapsto\\rho\(u,v\_\{0\}\)with the metricΛu​u\\Lambda\_\{uu\}, has the same velocity att=0t=0, so the two weight trajectories agree to first order intt\.

*Adam\.*Letη\\etabe the step size andε\>0\\varepsilon\>0fixed\. At step one thevv\-gradient vanishes, so Adam’s first and second moments invvvanish andvvdoes not move; theuu\-update is computed from the same gradient as for the frozen method\. So after one step the two methods have the same parameters and the same weights\. At step two, the gradient invvisDvβ\(u1,v0\)⊤∇LD\_\{v\}\\beta\(u\_\{1\},v\_\{0\}\)^\{\\top\}\\nabla L\. Sinceβ\\betais smooth andDv​β​\(u0,⋅\)≡0D\_\{v\}\\beta\(u\_\{0\},\\cdot\)\\equiv 0, we have‖Dv​β​\(u,v\)‖=O⁡\(‖u−u0‖\)\\\|D\_\{v\}\\beta\(u,v\)\\\|=O\(\\\|u\-u\_\{0\}\\\|\)uniformly forvvnearv0v\_\{0\}, and‖u1−u0‖=O⁡\(η\)\\\|u\_\{1\}\-u\_\{0\}\\\|=O\(\\eta\), so this gradient isO⁡\(η\)O\(\\eta\), and the move ofvvis at most of orderη\\eta; it is of exact orderη\\etawhenε\\varepsilonis negligible against the gradient, since Adam normalises it\. Its effect onθ\\thetais

ρ⁡\(u,v\)−ρ⁡\(u,v0\)=∫01Dv​β​\(u,v0\+τ⁡\(v−v0\)\)​\(v−v0\)​𝑑τ=O⁡\(‖u−u0‖​‖v−v0‖\)=O⁡\(η2\)\.\\rho\(u,v\)\-\\rho\(u,v\_\{0\}\)=\\int\_\{0\}^\{1\}D\_\{v\}\\beta\\big\(u,v\_\{0\}\+\\tau\(v\-v\_\{0\}\)\\big\)\(v\-v\_\{0\}\)\\,d\\tau=O\(\\\|u\-u\_\{0\}\\\|\\,\\\|v\-v\_\{0\}\\\|\)=O\(\\eta^\{2\}\)\.For any fixed numberkkof steps the same estimate propagates: the Adam update is a smooth function of the gradient history whenε\>0\\varepsilon\>0, and by induction on the step,vt−v0=O⁡\(η\)v\_\{t\}\-v\_\{0\}=O\(\\eta\), theuu\-iterates of the two methods differ byO⁡\(η2\)O\(\\eta^\{2\}\)\(their gradients differ byO⁡\(η\)O\(\\eta\)and are multiplied byη\\eta\), and the weights differ byO⁡\(η2\)O\(\\eta^\{2\}\)by the display\. Forε→0\\varepsilon\\to 0the same argument holds as long as no entry of the frozen method’s gradient vanishes in the firstkksteps, so that the normalisation stays smooth\. Terms that are not of the formL∘ρL\\circ\\rho, such as a regulariserR⁡\(v\)R\(v\)or decoupled weight decayv↦\(1−η​λ\)​vv\\mapsto\(1\-\\eta\\lambda\)v, movevvalready at step one, which is why the statement concerns the weight trajectory\.

*Function\-level variant\.*IfF⁡\(u0,v,x\)=F0​\(x\)F\(u\_\{0\},v,x\)=F\_\{0\}\(x\)for allvvandxx, thenDv​F​\(u0,v0,x\)=0D\_\{v\}F\(u\_\{0\},v\_\{0\},x\)=0for everyxx, and the argument above applies verbatim with the realised function in place ofθ\\theta\. ∎

### C\.11 Theorem II\.5

Theorem II\.5\(complete invariants of the basic actions; classical\)\.LetW,W′∈ℝm×nW,W^\{\\prime\}\\in\\mathbb\{R\}^\{m\\times n\}\.\(1\)*Right orthogonal \(input side\)\.*W′=W​RW^\{\\prime\}=WRfor someR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)iffW​W⊤=W′​W′⁣⊤WW^\{\\top\}=W^\{\\prime\}W^\{\\prime\\top\}\. ForSO⁡\(n\)\\mathrm\{SO\}\(n\), whenrank⁡W=n\\rk W=n, also require that the then uniqueRRhavedetR=1\\det R=1\.\(2\)*Left orthogonal\.*W′=P​WW^\{\\prime\}=PWfor someP∈O⁡\(m\)P\\in\\mathrm\{O\}\(m\)iffW⊤​W=W′⁣⊤​W′W^\{\\top\}W=W^\{\\prime\\top\}W^\{\\prime\}\.\(3\)*Two\-sided\.*W′=P​W​RW^\{\\prime\}=PWRfor someP∈O⁡\(m\)P\\in\\mathrm\{O\}\(m\)andR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)iffWWandW′W^\{\\prime\}have the same singular values, counted with multiplicity\.\(4\)*Left torus\.*IfWWhas no zero row,W′=D​WW^\{\\prime\}=DWfor someD∈Tm\+D\\in T\_\{m\}^\{\+\}iffwi′∈ℝ\>0​wiw\_\{i\}^\{\\prime\}\\in\\mathbb\{R\}\_\{\>0\}\\,w\_\{i\}for everyii\. For the full torusTmT\_\{m\}, whose signs are unconstrained, the complete invariant is the set of zero rows together with the row*lines*\.\(5\)*Full Hadamard torus\.*W′=E⊙WW^\{\\prime\}=E\\odot Wfor someE∈\(ℝ×\)m×nE\\in\(\\mathbb\{R\}^\{\\times\}\)^\{m\\times n\}iffWWandW′W^\{\\prime\}have the same zero pattern\.

Writewi,wi′∈ℝnw\_\{i\},w\_\{i\}^\{\\prime\}\\in\\mathbb\{R\}^\{n\}for the rows ofWWandW′W^\{\\prime\}\.

*\(1\)*IfW′=W​RW^\{\\prime\}=WRwithRRorthogonal, thenW′​W′⁣⊤=W​R​R⊤​W⊤=W​W⊤W^\{\\prime\}W^\{\\prime\\top\}=WRR^\{\\top\}W^\{\\top\}=WW^\{\\top\}\. Conversely, letW​W⊤=W′​W′⁣⊤WW^\{\\top\}=W^\{\\prime\}W^\{\\prime\\top\}\. Forc∈ℝmc\\in\\mathbb\{R\}^\{m\},

∥∑ici​wi∥2=c⊤​W​W⊤​c=c⊤​W′​W′⁣⊤​c=∥∑ici​wi′∥2\.\\Big\\lVert\\sum\_\{i\}c\_\{i\}w\_\{i\}\\Big\\rVert^\{2\}=c^\{\\top\}WW^\{\\top\}c=c^\{\\top\}W^\{\\prime\}W^\{\\prime\\top\}c=\\Big\\lVert\\sum\_\{i\}c\_\{i\}w\_\{i\}^\{\\prime\}\\Big\\rVert^\{2\}\.Henceφ:∑ici​wi↦∑ici​wi′\\varphi:\\sum\_\{i\}c\_\{i\}w\_\{i\}\\mapsto\\sum\_\{i\}c\_\{i\}w\_\{i\}^\{\\prime\}is well defined onrow⁡W\\operatorname\{row\}W\(if∑ici​wi=∑ici′​wi\\sum\_\{i\}c\_\{i\}w\_\{i\}=\\sum\_\{i\}c\_\{i\}^\{\\prime\}w\_\{i\}, the vector∑i\(ci−ci′\)​wi′\\sum\_\{i\}\(c\_\{i\}\-c\_\{i\}^\{\\prime\}\)w\_\{i\}^\{\\prime\}has norm00\), linear, isometric and ontorow⁡W′\\operatorname\{row\}W^\{\\prime\}\. In particular both row spaces have dimensionrank⁡W​W⊤\\rk WW^\{\\top\}\. By Witt’s extension theorem \([Auxiliary result B\.25](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(b\)\),φ\\varphiextends to someS∈O⁡\(n\)S\\in\\mathrm\{O\}\(n\), for instance by an isometry between the orthogonal complements, which have equal dimension\. ThenS​wi=wi′Sw\_\{i\}=w\_\{i\}^\{\\prime\}for everyii, that isW′=W​S⊤W^\{\\prime\}=WS^\{\\top\}, andR=S⊤∈O⁡\(n\)R=S^\{\\top\}\\in\\mathrm\{O\}\(n\)\.

ForSO⁡\(n\)\\mathrm\{SO\}\(n\): ifrank⁡W<n\\rk W<n, the complement\(row⁡W\)⟂\(\\operatorname\{row\}W\)^\{\\perp\}is nonzero, and composing the extension with a reflection of that complement changesdetS\\det Swithout changingS​wiSw\_\{i\}; so someR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)exists as soon as someR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\)does\. Ifrank⁡W=n\\rk W=n, thenWWhas full column rank andW​R=W′WR=W^\{\\prime\}determinesR=W\+​W′R=W^\{\+\}W^\{\\prime\}uniquely, soW′∈W⋅SO⁡\(n\)W^\{\\prime\}\\in W\\cdot\\mathrm\{SO\}\(n\)iff thisRRhas determinant11\.

*\(2\)*W′=P​WW^\{\\prime\}=PWiffW′⁣⊤=W⊤​P⊤W^\{\\prime\\top\}=W^\{\\top\}P^\{\\top\}withP⊤∈O⁡\(m\)P^\{\\top\}\\in\\mathrm\{O\}\(m\), and by \(1\) applied toW⊤W^\{\\top\}andW′⁣⊤W^\{\\prime\\top\}this happens iffW⊤​W=W′⁣⊤​W′W^\{\\top\}W=W^\{\\prime\\top\}W^\{\\prime\}\.

*\(3\)*IfW′=P​W​RW^\{\\prime\}=PWR, thenW′⁣⊤​W′=R⊤​\(W⊤​W\)​RW^\{\\prime\\top\}W^\{\\prime\}=R^\{\\top\}\(W^\{\\top\}W\)Rhas the eigenvalues ofW⊤​WW^\{\\top\}W, so the singular values agree with multiplicity\. Conversely, ifW=U​Σ​V⊤W=U\\Sigma V^\{\\top\}andW′=U′​Σ​V′⁣⊤W^\{\\prime\}=U^\{\\prime\}\\Sigma V^\{\\prime\\top\}are singular value decompositions with the sameΣ\\Sigma\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\), thenW′=\(U′​U⊤\)​W​\(V​V′⁣⊤\)W^\{\\prime\}=\(U^\{\\prime\}U^\{\\top\}\)\\,W\\,\(VV^\{\\prime\\top\}\)with both factors orthogonal\.

*\(4\)*ForD=diag⁡\(d\)D=\\diag\(d\)the rows ofD​WDWaredi​wid\_\{i\}w\_\{i\}\. Ifd∈ℝ\>0md\\in\\mathbb\{R\}^\{m\}\_\{\>0\}, thenwi′=di​wi∈ℝ\>0​wiw\_\{i\}^\{\\prime\}=d\_\{i\}w\_\{i\}\\in\\mathbb\{R\}\_\{\>0\}w\_\{i\}\. Conversely, ifwi′=di​wiw\_\{i\}^\{\\prime\}=d\_\{i\}w\_\{i\}withdi\>0d\_\{i\}\>0for everyii\(did\_\{i\}is unique becausewi≠0w\_\{i\}\\neq 0\), thenW′=diag⁡\(d\)​WW^\{\\prime\}=\\diag\(d\)Wwithdiag⁡\(d\)∈Tm\+\\diag\(d\)\\in T\_\{m\}^\{\+\}\. ForTmT\_\{m\}, the rowsdi​wid\_\{i\}w\_\{i\}withdi≠0d\_\{i\}\\neq 0vanish exactly wherewiw\_\{i\}does and otherwise span the lineℝ​wi\\mathbb\{R\}w\_\{i\}\. Conversely, ifWWandW′W^\{\\prime\}have the same zero rows andℝ​wi′=ℝ​wi\\mathbb\{R\}w\_\{i\}^\{\\prime\}=\\mathbb\{R\}w\_\{i\}for every nonzero row, thenwi′=di​wiw\_\{i\}^\{\\prime\}=d\_\{i\}w\_\{i\}withdi≠0d\_\{i\}\\neq 0, and withdi=1d\_\{i\}=1on the zero rowsW′=diag⁡\(d\)​WW^\{\\prime\}=\\diag\(d\)W,diag⁡\(d\)∈Tm\\diag\(d\)\\in T\_\{m\}\.

*\(5\)*\(E⊙W\)i​j=Ei​j​Wi​j\(E\\odot W\)\_\{ij\}=E\_\{ij\}W\_\{ij\}vanishes iffWi​jW\_\{ij\}does, becauseEi​j≠0E\_\{ij\}\\neq 0\. Conversely, ifWWandW′W^\{\\prime\}have the same zero pattern, putEi​j=Wi​j′/Wi​jE\_\{ij\}=W^\{\\prime\}\_\{ij\}/W\_\{ij\}on the support andEi​j=1E\_\{ij\}=1off it; thenEEhas no zero entry andW′=E⊙WW^\{\\prime\}=E\\odot W\. ∎

Item \(1\) is also the orbit\-separation statement of the first fundamental theorem forO⁡\(n\)\\mathrm\{O\}\(n\)\([Auxiliary result B\.25](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(a\)\); the proof above uses only the extension of isometries\.

### C\.12 Corollary II\.6

Corollary II\.6\(what exactly orthogonal methods can never do\)\.Let a method have image inW0⋅O⁡\(n\)W\_\{0\}\\cdot\\mathrm\{O\}\(n\)\(input side\), and work in exact arithmetic\. This covers:•OFT and block OFT with an exactly orthogonal parametrisation: the original paper’s exact Cayley map, and HF PEFT from 0\.16 with use\_cayley\_neumann=False on Linear and Conv targets;•GOFT with strict Givens rotations and no scaler, as in the paper \(no\_scaling=True in the official peft\-givens code\);•HRA, and ETHER as defined in its paper \(not ETHER\+\);•any number of merge\-and\-restart cycles of these\.Then:•it preservesKoutK\_\{\\mathrm\{out\}\}, hence every neuron norm and the angle between any two nonzero neurons, and so the hyperspherical energy, which depends only on these angles;•it preserves the whole singular spectrum ofW0W\_\{0\};•block\-diagonal OFT with input blocksJJ, with the same partition in every cycle, preserves each partial Gram matrixW:,JW:,J⊤W\_\{:,J\}W\_\{:,J\}^\{\\top\}\.

*Coverage\.*Each listed parametrisation producesW=W0​RW=W\_\{0\}RwithR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\), read in the conventiony=W​xy=Wx, where an input\-side rotation acts on the columns ofW0W\_\{0\}\. The exact Cayley mapC⁡\(Ω\)=\(I\+Ω\)​\(I−Ω\)−1C\(\\Omega\)=\(I\+\\Omega\)\(I\-\\Omega\)^\{\-1\}of a skew matrix is orthogonal \([Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(a\)\); HF PEFT’s input\-side OFT merges toW0​R⊤W\_\{0\}R^\{\\top\}, andR⊤R^\{\\top\}is orthogonal too\. A block\-diagonal matrix with orthogonal blocks is orthogonal\. A Givens rotation is orthogonal, and so is a product of them\. HRA multiplies Householder reflectionsHu=I−2​u​u⊤/∥u∥2H\_\{u\}=I\-2uu^\{\\top\}/\\lVert u\\rVert^\{2\}, which are orthogonal, and ETHER as defined in its paper multipliesW0W\_\{0\}on the input side by one reflectionI−2​u^​u^⊤I\-2\\hat\{u\}\\hat\{u\}^\{\\top\}with∥u^∥=1\\lVert\\hat\{u\}\\rVert=1\. For merge and restart, if every weight reachable in cyclekkisWk−1​RkW\_\{k\-1\}R\_\{k\}withRk∈O⁡\(n\)R\_\{k\}\\in\\mathrm\{O\}\(n\), then by inductionWk=W0R1⋯RkW\_\{k\}=W\_\{0\}R\_\{1\}\\cdots R\_\{k\}, andR1⋯Rk∈O\(n\)R\_\{1\}\\cdots R\_\{k\}\\in\\mathrm\{O\}\(n\)\.

*Gram matrix and spectrum\.*ForW=W0​RW=W\_\{0\}RwithR∈O⁡\(n\)R\\in\\mathrm\{O\}\(n\),Kout​\(W\)=W0​R​R⊤​W0⊤=Kout​\(W0\)K\_\{\\mathrm\{out\}\}\(W\)=W\_\{0\}RR^\{\\top\}W\_\{0\}^\{\\top\}=K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(1\)\)\. The neuron norms are∥wi∥2=Kout​\(W\)i​i\\lVert w\_\{i\}\\rVert^\{2\}=K\_\{\\mathrm\{out\}\}\(W\)\_\{ii\}, and the cosine between two nonzero neurons isKout​\(W\)i​j/\(Kout​\(W\)i​i​Kout​\(W\)j​j\)1/2K\_\{\\mathrm\{out\}\}\(W\)\_\{ij\}/\(K\_\{\\mathrm\{out\}\}\(W\)\_\{ii\}K\_\{\\mathrm\{out\}\}\(W\)\_\{jj\}\)^\{1/2\}, so both are preserved\. The hyperspherical energy of[Liu et al\. \(2018\)](https://arxiv.org/html/2610.12728#bib.bib283)is a function of the distances between the normalised neurons,∥w^i−w^j∥2=2−2cosi​j\\lVert\\hat\{w\}\_\{i\}\-\\hat\{w\}\_\{j\}\\rVert^\{2\}=2\-2\\cos\_\{ij\}, so it is preserved as well\. The singular values ofWWare the square roots of the eigenvalues ofKout​\(W\)=Kout​\(W0\)K\_\{\\mathrm\{out\}\}\(W\)=K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\(equivalently,[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(3\) withP=IP=I\)\.

*Partial Grams\.*LetR=blockdiag⁡\(RJ\)JR=\\operatorname\{blockdiag\}\(R\_\{J\}\)\_\{J\}be block\-diagonal for a partition of the input coordinates into blocksJJ, with orthogonal blocks\. The columns ofW0​RW\_\{0\}Rindexed byJJare\(W0R\):,J=\(W0\):,JRJ\(W\_\{0\}R\)\_\{:,J\}=\(W\_\{0\}\)\_\{:,J\}R\_\{J\}, soW:,JW:,J⊤=\(W0\):,JRJRJ⊤\(W0\):,J⊤=\(W0\):,J\(W0\):,J⊤W\_\{:,J\}W\_\{:,J\}^\{\\top\}=\(W\_\{0\}\)\_\{:,J\}R\_\{J\}R\_\{J\}^\{\\top\}\(W\_\{0\}\)\_\{:,J\}^\{\\top\}=\(W\_\{0\}\)\_\{:,J\}\(W\_\{0\}\)\_\{:,J\}^\{\\top\}\. With the same partition in every cycle,R1⋯RkR\_\{1\}\\cdots R\_\{k\}is again block\-diagonal with orthogonal blocks, so the identity survives merging\. ∎

The remarks after the corollary in §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)make several claims that do not follow from its hypothesis\. We prove them in the same order\. Throughout,Kout​\(W\)=W​W⊤K\_\{\\mathrm\{out\}\}\(W\)=WW^\{\\top\}, and for an input\-side factorGGwe haveKout​\(W0​G\)=W0​G​G⊤​W0⊤K\_\{\\mathrm\{out\}\}\(W\_\{0\}G\)=W\_\{0\}\\,GG^\{\\top\}\\,W\_\{0\}^\{\\top\}\.

*BOFT and HF OFT 0\.14–0\.15\.*ForW=diag⁡\(s\)​W0​R⊤W=\\diag\(s\)W\_\{0\}R^\{\\top\}withRRorthogonal,

Kout​\(W\)=diag⁡\(s\)​W0​R⊤​R​W0⊤​diag⁡\(s\)=diag⁡\(s\)​Kout​\(W0\)​diag⁡\(s\),K\_\{\\mathrm\{out\}\}\(W\)=\\diag\(s\)\\,W\_\{0\}R^\{\\top\}RW\_\{0\}^\{\\top\}\\diag\(s\)=\\diag\(s\)\\,K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\\,\\diag\(s\),with entriessi​sj​Kout​\(W0\)i​js\_\{i\}s\_\{j\}K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\_\{ij\}\. Ifsi​sj≠0s\_\{i\}s\_\{j\}\\neq 0and the neuronsi,ji,jofW0W\_\{0\}are nonzero,

cosi​j′=si​sj​Kout​\(W0\)i​j\|si\|​\|sj\|​\(Kout​\(W0\)i​i​Kout​\(W0\)j​j\)1/2=sign\(sisj\)cosi​j\.\\cos^\{\\prime\}\_\{ij\}=\\frac\{s\_\{i\}s\_\{j\}K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\_\{ij\}\}\{\\lvert s\_\{i\}\\rvert\\lvert s\_\{j\}\\rvert\\,\(K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\_\{ii\}K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)\_\{jj\}\)^\{1/2\}\}=\\operatorname\{sign\}\(s\_\{i\}s\_\{j\}\)\\cos\_\{ij\}\.So\|cosi​j\|\\lvert\\cos\_\{ij\}\\rvertis preserved wheneversi​sj≠0s\_\{i\}s\_\{j\}\\neq 0, andcosi​j\\cos\_\{ij\}itself wheneversi​sj\>0s\_\{i\}s\_\{j\}\>0\. If everysis\_\{i\}is nonzero and all share a sign, every cosine is preserved\. Conversely, if every cosine ofW0W\_\{0\}is nonzero and every cosine is preserved, then everysis\_\{i\}is nonzero \(otherwise neuroniiofWWvanishes and its cosines are undefined\) andsign⁡\(si​sj\)=1\\operatorname\{sign\}\(s\_\{i\}s\_\{j\}\)=1for alli≠ji\\neq j, so allsis\_\{i\}share a sign\. With the butterfly permutation equal to the identity,RRis block\-diagonal with orthogonal blocksRJR\_\{J\}over a partition of the input coordinates, andW:,J=diag\(s\)\(W0\):,JRJ⊤W\_\{:,J\}=\\diag\(s\)\(W\_\{0\}\)\_\{:,J\}R\_\{J\}^\{\\top\}, soW:,JW:,J⊤=diag\(s\)\(W0\):,J\(W0\):,J⊤diag\(s\)W\_\{:,J\}W\_\{:,J\}^\{\\top\}=\\diag\(s\)\(W\_\{0\}\)\_\{:,J\}\(W\_\{0\}\)\_\{:,J\}^\{\\top\}\\diag\(s\)\. HF PEFT OFT 0\.14–0\.15 has the same form with its output scale oft\_s in place ofss, which gives the same conclusions\.

*GOFT with its scaler\.*ForW=W0​diag⁡\(g\)​GW=W\_\{0\}\\diag\(g\)GwithGGorthogonal, whateverGGis,

Kout​\(W\)=W0​diag⁡\(g\)​G​G⊤​diag⁡\(g\)​W0⊤=W0​diag​\(g\)2​W0⊤\.K\_\{\\mathrm\{out\}\}\(W\)=W\_\{0\}\\diag\(g\)\\,GG^\{\\top\}\\diag\(g\)\\,W\_\{0\}^\{\\top\}=W\_\{0\}\\diag\(g\)^\{2\}W\_\{0\}^\{\\top\}\.This differs fromKout​\(W0\)K\_\{\\mathrm\{out\}\}\(W\_\{0\}\)in general \(forW0=InW\_\{0\}=I\_\{n\}it isdiag⁡\(g\)2\\diag\(g\)^\{2\}\), so the image is not contained inW0⋅O⁡\(n\)W\_\{0\}\\cdot\\mathrm\{O\}\(n\)and the corollary does not apply\.

*Output\-side implementations\.*IfW=P​W0W=PW\_\{0\}withP∈O⁡\(m\)P\\in\\mathrm\{O\}\(m\), thenKin​\(W\)=W0⊤​P⊤​P​W0=Kin​\(W0\)K\_\{\\mathrm\{in\}\}\(W\)=W\_\{0\}^\{\\top\}P^\{\\top\}PW\_\{0\}=K\_\{\\mathrm\{in\}\}\(W\_\{0\}\), and the singular values are preserved by[Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(3\)\. IfPPis block\-diagonal with orthogonal blocksPIP\_\{I\}over a partition of the output coordinates, the rows indexed byIIsatisfyWI,:=PI\(W0\)I,:W\_\{I,:\}=P\_\{I\}\(W\_\{0\}\)\_\{I,:\}, soWI,:⊤WI,:=\(W0\)I,:⊤\(W0\)I,:W\_\{I,:\}^\{\\top\}W\_\{I,:\}=\(W\_\{0\}\)\_\{I,:\}^\{\\top\}\(W\_\{0\}\)\_\{I,:\}\. In generalKout​\(P​W0\)=P​Kout​\(W0\)​P⊤K\_\{\\mathrm\{out\}\}\(PW\_\{0\}\)=PK\_\{\\mathrm\{out\}\}\(W\_\{0\}\)P^\{\\top\}differs fromKout​\(W0\)K\_\{\\mathrm\{out\}\}\(W\_\{0\}\), for instance forW0=E11W\_\{0\}=E\_\{11\}andPPa rotation movinge1e\_\{1\}\. ETHER’s code with ether\_nb\>1\{\}\>1multiplies the rows of output groupggon the right by a reflectionHgH\_\{g\},WIg,:=\(W0\)Ig,:HgW\_\{I\_\{g\},:\}=\(W\_\{0\}\)\_\{I\_\{g\},:\}H\_\{g\}\. Within a group,WIg,:WIg,:⊤=\(W0\)Ig,:\(W0\)Ig,:⊤W\_\{I\_\{g\},:\}W\_\{I\_\{g\},:\}^\{\\top\}=\(W\_\{0\}\)\_\{I\_\{g\},:\}\(W\_\{0\}\)\_\{I\_\{g\},:\}^\{\\top\}, which keeps neuron norms and within\-group angles\. Across groups the inner products becomew0,i⊤​Hg​Hg′​w0,jw\_\{0,i\}^\{\\top\}H\_\{g\}H\_\{g^\{\\prime\}\}w\_\{0,j\}\. For two groups of one row each, withw0,1=w0,2=e1w\_\{0,1\}=w\_\{0,2\}=e\_\{1\}inℝ2\\mathbb\{R\}^\{2\},H1=IH\_\{1\}=IandH2H\_\{2\}the reflection that swapse1e\_\{1\}ande2e\_\{2\}, the two neurons becomee1e\_\{1\}ande2e\_\{2\}: the cosine drops from11to00, and the singular values change from\(2,0\)\(\\sqrt\{2\},0\)to\(1,1\)\(1,1\)\.

*HF PEFT’s default OFT\.*LetQQbe skew\. Then\(I−Q\)​\(I\+Q\+Q2\+Q3\)=I−Q4\(I\-Q\)\(I\+Q\+Q^\{2\}\+Q^\{3\}\)=I\-Q^\{4\}andI−QI\-Qis invertible \([Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(a\),\(e\)\), so

R:=I\+2​Q\+2​Q2\+2​Q3\+Q4=\(I\+Q\)​\(I\+Q\+Q2\+Q3\)=\(I\+Q\)​\(I−Q\)−1​\(I−Q4\)=C⁡\(Q\)​\(I−Q4\),R:=I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}=\(I\+Q\)\(I\+Q\+Q^\{2\}\+Q^\{3\}\)=\(I\+Q\)\(I\-Q\)^\{\-1\}\(I\-Q^\{4\}\)=C\(Q\)\\,\(I\-Q^\{4\}\),where all factors are polynomials inQQor their inverses and commute\. The matrixQ4Q^\{4\}is symmetric, since\(Q4\)⊤=\(−Q\)4\(Q^\{4\}\)^\{\\top\}=\(\-Q\)^\{4\}, andC⁡\(Q\)C\(Q\)is orthogonal, soR⊤​R=\(I−Q4\)​C​\(Q\)⊤​C​\(Q\)​\(I−Q4\)=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)C\(Q\)^\{\\top\}C\(Q\)\(I\-Q^\{4\}\)=\(I\-Q^\{4\}\)^\{2\}\. If±i​θk\\pm i\\theta\_\{k\}are the eigenvalues ofQQ, thenQ4Q^\{4\}has eigenvaluesθk4\\theta\_\{k\}^\{4\}\(and00\), so the singular values ofRRare\|1−θk4\|\\lvert 1\-\\theta\_\{k\}^\{4\}\\rvert\(and11\)\. ThusR⊤​R=IR^\{\\top\}R=Ionly if everyθk\\theta\_\{k\}is00or21/42^\{1/4\}; in particularR⊤​R≠IR^\{\\top\}R\\neq Ifor everyQ≠0Q\\neq 0with∥Q∥op<21/4\\lVert Q\\rVert\_\{\\mathrm\{op\}\}<2^\{1/4\}, andRRis singular when someθk=1\\theta\_\{k\}=1\. The adapted weight isW=W0​R⊤W=W\_\{0\}R^\{\\top\}, withKout​\(W\)=W0​R⊤​R​W0⊤=W0​\(I−Q4\)2​W0⊤K\_\{\\mathrm\{out\}\}\(W\)=W\_\{0\}R^\{\\top\}RW\_\{0\}^\{\\top\}=W\_\{0\}\(I\-Q^\{4\}\)^\{2\}W\_\{0\}^\{\\top\}, and\(I−Q4\)2−I=Q8−2​Q4\(I\-Q^\{4\}\)^\{2\}\-I=Q^\{8\}\-2Q^\{4\}is of order∥Q∥4\\lVert Q\\rVert^\{4\}\. When∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\mathrm\{op\}\}\\leq 2^\{1/4\}, every\|1−θk4\|≤1\\lvert 1\-\\theta\_\{k\}^\{4\}\\rvert\\leq 1, soRRis a contraction, and each mergeW↦W​R⊤W\\mapsto WR^\{\\top\}givesW​R⊤​R​W⊤⪯W​W⊤WR^\{\\top\}RW^\{\\top\}\\preceq WW^\{\\top\}and lowers every singular value \([Auxiliary result B\.18](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\),\(c\)\);[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), corollary 4, records the iteration\. In PEFT 0\.16 and 0\.17 the last coefficient was22,R=p⁡\(Q\)R=p\(Q\)withp⁡\(x\)=1\+2​x\+2​x2\+2​x3\+2​x4p\(x\)=1\+2x\+2x^\{2\}\+2x^\{3\}\+2x^\{4\}\. ThenR⊤=p⁡\(−Q\)R^\{\\top\}=p\(\-Q\), and writingp=E\+Op=E\+Owith even partE=1\+2​x2\+2​x4E=1\+2x^\{2\}\+2x^\{4\}and odd partO=2​x\+2​x3O=2x\+2x^\{3\},

p⁡\(x\)​p​\(−x\)=E2−O2=\(1\+4​x2\+8​x4\+8​x6\+4​x8\)−\(4​x2\+8​x4\+4​x6\)=1\+4​x6\+4​x8,p\(x\)p\(\-x\)=E^\{2\}\-O^\{2\}=\(1\+4x^\{2\}\+8x^\{4\}\+8x^\{6\}\+4x^\{8\}\)\-\(4x^\{2\}\+8x^\{4\}\+4x^\{6\}\)=1\+4x^\{6\}\+4x^\{8\},soR⊤​R=I\+4​Q6\+4​Q8R^\{\\top\}R=I\+4Q^\{6\}\+4Q^\{8\}\. Its eigenvalues are1\+4​θk6​\(θk2−1\)1\+4\\theta\_\{k\}^\{6\}\(\\theta\_\{k\}^\{2\}\-1\), so it differs fromIIunless everyθk\\theta\_\{k\}is00or11, and the deviation is of order∥Q∥6\\lVert Q\\rVert^\{6\}\.

*One non\-orthogonal plane changes a cosine\.*We prove the following, which covers qGOFT, ETHER\+ and HF’s default OFT\. Letn\>2n\>2and letG∈ℝn×nG\\in\\mathbb\{R\}^\{n\\times n\}be an input\-side factor such thatS=G​G⊤S=GG^\{\\top\}acts as the identity on the orthogonal complement of a planeΠ\\Pi, mapsΠ\\Piinto itself, andS\|Π≠IΠS\|\_\{\\Pi\}\\neq I\_\{\\Pi\}\(that is,GGis not orthogonal onΠ\\Pi\)\. Then forW0W\_\{0\}outside a nowhere\-dense null set some cosine ofW0​GW\_\{0\}Gdiffers from the corresponding cosine ofW0W\_\{0\}\. Indeed, for two rowsw1,w2w\_\{1\},w\_\{2\}ofW0W\_\{0\}put

ϕ⁡\(W0\)=\(w1⊤​S​w2\)2​∥w1∥2​∥w2∥2−\(w1⊤​w2\)2​\(w1⊤​S​w1\)​\(w2⊤​S​w2\),\\phi\(W\_\{0\}\)=\(w\_\{1\}^\{\\top\}Sw\_\{2\}\)^\{2\}\\,\\lVert w\_\{1\}\\rVert^\{2\}\\lVert w\_\{2\}\\rVert^\{2\}\-\(w\_\{1\}^\{\\top\}w\_\{2\}\)^\{2\}\\,\(w\_\{1\}^\{\\top\}Sw\_\{1\}\)\(w\_\{2\}^\{\\top\}Sw\_\{2\}\),a polynomial inW0W\_\{0\}\. Where both cosines are defined,cos′12=cos12\\cos^\{\\prime\}\_\{12\}=\\cos\_\{12\}forcesϕ⁡\(W0\)=0\\phi\(W\_\{0\}\)=0\. Choose a unit vectorc⟂Πc\\perp\\Pi\(possible sincen\>2n\>2\) andx∈Πx\\in\\Piwithx⊤​S​x≠∥x∥2x^\{\\top\}Sx\\neq\\lVert x\\rVert^\{2\}\(possible sinceS\|Π−IΠS\|\_\{\\Pi\}\-I\_\{\\Pi\}is a nonzero symmetric map\), and takew1=cw\_\{1\}=c,w2=c\+xw\_\{2\}=c\+x\. Thenw1⊤​S​w2=1w\_\{1\}^\{\\top\}Sw\_\{2\}=1,w1⊤​S​w1=1w\_\{1\}^\{\\top\}Sw\_\{1\}=1,w2⊤​S​w2=1\+x⊤​S​xw\_\{2\}^\{\\top\}Sw\_\{2\}=1\+x^\{\\top\}Sx,∥w2∥2=1\+∥x∥2\\lVert w\_\{2\}\\rVert^\{2\}=1\+\\lVert x\\rVert^\{2\}andw1⊤​w2=1w\_\{1\}^\{\\top\}w\_\{2\}=1, soϕ=∥x∥2−x⊤​S​x≠0\\phi=\\lVert x\\rVert^\{2\}\-x^\{\\top\}Sx\\neq 0\. By[Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\),ϕ≠0\\phi\\neq 0on an open dense set of full measure, which proves the claim\. A conformal blockc​Rot\\sqrt\{c\}\\,\\mathrm\{Rot\}withc≠1c\\neq 1on two input coordinates hasS\|Π=c​IS\|\_\{\\Pi\}=cI, so it is covered\. For qGOFT, take one block non\-orthogonal and the others equal toII\. For ETHER\+, take the output factor equal toII\(its two vectors equal\) and the input factorI−u​u⊤\+v​v⊤I\-uu^\{\\top\}\+vv^\{\\top\}with unitu≠±vu\\neq\\pm v: it is symmetric, equal toIIon\{u,v\}⟂\\\{u,v\\\}^\{\\perp\}, preservesΠ=span⁡\{u,v\}\\Pi=\\spanop\\\{u,v\\\}, and onΠ\\Pihas eigenvalues1±\(1−\(u⊤​v\)2\)1/21\\pm\(1\-\(u^\{\\top\}v\)^\{2\}\)^\{1/2\}, so it is not orthogonal there; foru⟂vu\\perp vit killsuuand is singular\. For HF’s default OFT, takeQ=θ⁡\(e1​e2⊤−e2​e1⊤\)Q=\\theta\(e\_\{1\}e\_\{2\}^\{\\top\}\-e\_\{2\}e\_\{1\}^\{\\top\}\)withθ∉\{0,21/4\}\\theta\\notin\\\{0,2^\{1/4\}\\\}; thenR⊤​R=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}is\(1−θ4\)2​I\(1\-\\theta^\{4\}\)^\{2\}IonΠ=span⁡\{e1,e2\}\\Pi=\\spanop\\\{e\_\{1\},e\_\{2\}\\\}andIIelsewhere\.

*Finite precision\.*The last remark concerns floating\-point arithmetic, and Appendix[E](https://arxiv.org/html/2610.12728#A5)reports the measured drift; in exact arithmetic the corollary holds as proved\.

### C\.13 Theorem II\.7

Theorem II\.7\(HRA is the meet of LoRA with the rotation orbit, and it starts at an apex\)\.LetW0∈ℝm×nW\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}be injective \(rank⁡W0=n\\rk W\_\{0\}=n\), and letrrbe even with2≤r≤n−22\\leq r\\leq n\-2\. DefineHRAr:\(u1,…,ur\)↦W0Hu1⋯Hur\\mathrm\{HRA\}\_\{r\}:\(u\_\{1\},\\dots,u\_\{r\}\)\\mapsto W\_\{0\}H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}withui≠0u\_\{i\}\\neq 0, whereHu=I−2​u​u⊤/∥u∥2H\_\{u\}=I\-2uu^\{\\top\}/\\lVert u\\rVert^\{2\}\. WriteOFTSO:=W0⋅SO⁡\(n\)\\mathrm\{OFT\}\_\{\\mathrm\{SO\}\}:=W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)for the rotation orbit\. \(TheO⁡\(n\)\\mathrm\{O\}\(n\)\-orbit would add the odd component:W0​He1W\_\{0\}H\_\{e\_\{1\}\}lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}but not inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\.\)\(a\)Meet\.Im⁡HRAr=\(W0\+ℳ≤r\)∩W0⋅SO⁡\(n\)\\Imop\\mathrm\{HRA\}\_\{r\}=\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\. Further,d⁡\(HRAr\)=r​n−r⁡\(r\+1\)2d\(\\mathrm\{HRA\}\_\{r\}\)=rn\-\\tfrac\{r\(r\+1\)\}\{2\}, so the generic fibre has dimensionr⁡\(r\+1\)2=r​n−d\\tfrac\{r\(r\+1\)\}\{2\}=rn\-d\. This gauge is non\-linear\. The evident continuous global symmetries are therrscalingsui↦ci​uiu\_\{i\}\\mapsto c\_\{i\}u\_\{i\}; the braid group also acts globally by Hurwitz moves\(ui,ui\+1\)↦\(Hui​ui\+1,ui\)\(u\_\{i\},u\_\{i\+1\}\)\\mapsto\(H\_\{u\_\{i\}\}u\_\{i\+1\},u\_\{i\}\), sinceHa​Hb=HHa​b​HaH\_\{a\}H\_\{b\}=H\_\{H\_\{a\}b\}H\_\{a\}\.\(b\)First\-order collapse\.At the paired initialisationu2​i−1=u2​i=wiu\_\{2i\-1\}=u\_\{2i\}=w\_\{i\}\(i≤k=r/2i\\leq k=r/2, with thewiw\_\{i\}independent andU=span⁡\{wi\}U=\\spanop\\\{w\_\{i\}\\\}\),S1=W0⋅\{Ω∈𝔰​𝔬​\(n\):PU⟂​Ω​PU⟂=0\},dimS1=k​n−k⁡\(k\+1\)2<d⁡\(HRAr\)\.S\_\{1\}=W\_\{0\}\\cdot\\\{\\Omega\\in\\mathfrak\{so\}\(n\):P\_\{U^\{\\perp\}\}\\Omega P\_\{U^\{\\perp\}\}=0\\\},\\qquad\\dim S\_\{1\}=kn\-\\tfrac\{k\(k\+1\)\}\{2\}<d\(\\mathrm\{HRA\}\_\{r\}\)\.\(c\)Vertex\.The germ ofIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}atW0W\_\{0\}is diffeomorphic to the germ at00of the coneΣr=\{Ω∈𝔰​𝔬​\(n\):rank⁡Ω≤r\}\\Sigma\_\{r\}=\\\{\\Omega\\in\\mathfrak\{so\}\(n\):\\rk\\Omega\\leq r\\\}\(the skew\-determinantal cone\), which is not linear\. SoW0W\_\{0\}is a singular point \(the vertex\) of the image\. Together with \(b\), the paired initialisation is anapex\.\(d\)Scope\.•*Not a product\.*\(a\) is a meet in the poset of subsets ofΘ\\ThetacontainingW0W\_\{0\}\([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\)\)\. HRA is not the categorical product ofLoRAr\\mathrm\{LoRA\}\_\{r\}andOFTSO\\mathrm\{OFT\}\_\{\\mathrm\{SO\}\}: the set\-level fibre product\{\(B,A,R\):W0\+B​A=W0​R\}\\\{\(B,A,R\):W\_\{0\}\+BA=W\_\{0\}R\\\}has local dimensiond\+r2=r​n\+r⁡\(r−1\)2≠r​nd\+r^\{2\}=rn\+\\tfrac\{r\(r\-1\)\}\{2\}\\neq rnover a generic point of the meet\. \(It is not pure\-dimensional: overW0W\_\{0\}alone it contains\{\(B,0,I\)\}\\\{\(B,0,I\)\\\}, of dimensionm​rmr\.\)•*Exactly orthogonal OFT is smaller\.*The Cayley map never produces a rotation with eigenvalue−1\-1\. For exact full\-block Cayley OFT \(the original paper; HF with use\_cayley\_neumann=False\),\(W0\+ℳ≤r\)∩Im⁡OFT\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\Imop\\mathrm\{OFT\}is dense inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}but not equal to it:W0​He1​He2W\_\{0\}H\_\{e\_\{1\}\}H\_\{e\_\{2\}\}, withHe1​He2=diag⁡\(−1,−1,1,…,1\)H\_\{e\_\{1\}\}H\_\{e\_\{2\}\}=\\diag\(\-1,\-1,1,\\dots,1\), is missing\. For exact block Cayley OFT withb<nb<nthe intersection is a proper subset ofIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\.•*HF’s default OFT is incomparable\.*HF PEFT’s default OFT is not orthogonal \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\); its flag is use\_cayley\_neumann=True\. SinceR⊤​R=\(I−Q4\)2=IR^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}=IforcesQ=0Q=0nearQ=0Q=0, its image meetsIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}nearW0W\_\{0\}only atW0W\_\{0\}, while its intersection withW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}contains non\-orthogonal points such asW0​\(I\+2​Q\+2​Q2\+2​Q3\+Q4\)W\_\{0\}\(I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}\)withrank⁡Q=2\\rk Q=2\.•*Implementation\.*The theorem concerns HRA withλ<∞\\lambda<\\infty\(HF apply\_GS=False\)\. The strictly orthogonal Gram–Schmidt variant has the smaller imageW0​\(I−2​PU\)W\_\{0\}\(I\-2P\_\{U\}\), of dimensionr⁡\(n−r\)r\(n\-r\)\. The paired initialisation is HF’s default \(repeat\_interleave\) and is not stated in the HRA paper\. HF HRA acts on in\_features, so injectivity requires out\_features≥\{\}\\geq\{\}in\_features \(and full column rank\); the theorem does not cover down\_proj or GQA key/value projections\. The Gram–Schmidt variant never containsW0W\_\{0\}, sinceI−2​PU≠II\-2P\_\{U\}\\neq I, and at HF’s default paired initialisation it runs Gram–Schmidt on duplicated columns, which is numerically degenerate\.

Foru≠0u\\neq 0writeu^=u/∥u∥\\hat\{u\}=u/\\lVert u\\rVert\. The reflectionHu=I−2​u^​u^⊤H\_\{u\}=I\-2\\hat\{u\}\\hat\{u\}^\{\\top\}is symmetric and orthogonal, has determinant−1\-1, fixesu⟂u^\{\\perp\}pointwise, andHu−IH\_\{u\}\-Ihas rank one\. Forg∈O⁡\(n\)g\\in\\mathrm\{O\}\(n\),g​Hu​g−1=Hg​ugH\_\{u\}g^\{\-1\}=H\_\{gu\}\. Forx,y∈ℝnx,y\\in\\mathbb\{R\}^\{n\}putx∧y=x​y⊤−y​x⊤∈𝔰​𝔬​\(n\)x\\wedge y=xy^\{\\top\}\-yx^\{\\top\}\\in\\mathfrak\{so\}\(n\), and for a subspaceA⊆ℝnA\\subseteq\\mathbb\{R\}^\{n\}letℝn∧A=span\{x∧a:x∈ℝn,a∈A\}\\mathbb\{R\}^\{n\}\\wedge A=\\spanop\\\{x\\wedge a:x\\in\\mathbb\{R\}^\{n\},\\ a\\in A\\\}\. BecauseW0W\_\{0\}is injective,X↦W0​XX\\mapsto W\_\{0\}Xis an injective linear mapℝn×n→ℝm×n\\mathbb\{R\}^\{n\\times n\}\\to\\mathbb\{R\}^\{m\\times n\}; in particularW0​R=W0​R′W\_\{0\}R=W\_\{0\}R^\{\\prime\}impliesR=R′R=R^\{\\prime\}\.

*Two lemmas\.*\(i\) For every subspaceAAof dimensionaa,

ℝn∧A=\{Ω∈𝔰​𝔬​\(n\):PA⟂​Ω​PA⟂=0\},dim\(ℝn∧A\)=a​n−a⁡\(a\+1\)2\.\\mathbb\{R\}^\{n\}\\wedge A=\\\{\\Omega\\in\\mathfrak\{so\}\(n\):P\_\{A^\{\\perp\}\}\\Omega P\_\{A^\{\\perp\}\}=0\\\},\\qquad\\dim\(\\mathbb\{R\}^\{n\}\\wedge A\)=an\-\\tfrac\{a\(a\+1\)\}\{2\}\.Forx,y∈A⟂x,y\\in A^\{\\perp\}andz∈ℝnz\\in\\mathbb\{R\}^\{n\},a′∈Aa^\{\\prime\}\\in A,y⊤​\(z∧a′\)​x=\(y⊤​z\)​\(a′⁣⊤​x\)−\(y⊤​a′\)​\(z⊤​x\)=0y^\{\\top\}\(z\\wedge a^\{\\prime\}\)x=\(y^\{\\top\}z\)\(a^\{\\prime\\top\}x\)\-\(y^\{\\top\}a^\{\\prime\}\)\(z^\{\\top\}x\)=0, which gives “⊆\\subseteq”\. Take an orthonormal basise1,…,eae\_\{1\},\\dots,e\_\{a\}ofAAandea\+1,…,ene\_\{a\+1\},\\dots,e\_\{n\}ofA⟂A^\{\\perp\}\. The matricesep∧eqe\_\{p\}\\wedge e\_\{q\}\(p<qp<q\) form a basis of𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\), andPA⟂​Ω​PA⟂=0P\_\{A^\{\\perp\}\}\\Omega P\_\{A^\{\\perp\}\}=0says exactly that the coefficients withp,q\>ap,q\>avanish, so the right side is spanned by theep∧eqe\_\{p\}\\wedge e\_\{q\}withp≤ap\\leq a, which lie inℝn∧A\\mathbb\{R\}^\{n\}\\wedge A\. Its dimension isdim𝔰​𝔬​\(n\)−dim𝔰​𝔬​\(n−a\)=a​n−a⁡\(a\+1\)/2\\dim\\mathfrak\{so\}\(n\)\-\\dim\\mathfrak\{so\}\(n\-a\)=an\-a\(a\+1\)/2\.

\(ii\) LetR\(u\)=Hu1⋯HurR\(u\)=H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}, putgk=Hu1⋯Huk−1g\_\{k\}=H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{k\-1\}\}\(sog1=Ig\_\{1\}=I\) andak=gk​u^ka\_\{k\}=g\_\{k\}\\hat\{u\}\_\{k\}, and letA⁡\(u\)=span⁡\{a1,…,ar\}A\(u\)=\\spanop\\\{a\_\{1\},\\dots,a\_\{r\}\\\}\. Then the right\-translated differential satisfies

\{d​Ru​\(u˙\)​R​\(u\)−1:u˙∈\(ℝn\)r\}=ℝn∧A⁡\(u\)\.\\\{dR\_\{u\}\(\\dot\{u\}\)\\,R\(u\)^\{\-1\}:\\dot\{u\}\\in\(\\mathbb\{R\}^\{n\}\)^\{r\}\\\}=\\mathbb\{R\}^\{n\}\\wedge A\(u\)\.Indeed, varying onlyuku\_\{k\}in the directionu˙k\\dot\{u\}\_\{k\}movesu^k\\hat\{u\}\_\{k\}byδ=\(I−u^k​u^k⊤\)​u˙k/∥uk∥\\delta=\(I\-\\hat\{u\}\_\{k\}\\hat\{u\}\_\{k\}^\{\\top\}\)\\dot\{u\}\_\{k\}/\\lVert u\_\{k\}\\rVert, which ranges overu^k⟂\\hat\{u\}\_\{k\}^\{\\perp\}, andHukH\_\{u\_\{k\}\}byH˙=−2​\(δ​u^k⊤\+u^k​δ⊤\)\\dot\{H\}=\-2\(\\delta\\hat\{u\}\_\{k\}^\{\\top\}\+\\hat\{u\}\_\{k\}\\delta^\{\\top\}\)\. SinceHuk​u^k=−u^kH\_\{u\_\{k\}\}\\hat\{u\}\_\{k\}=\-\\hat\{u\}\_\{k\}andHuk​δ=δH\_\{u\_\{k\}\}\\delta=\\delta, we getH˙​Huk=2​\(δ​u^k⊤−u^k​δ⊤\)=2​δ∧u^k\\dot\{H\}H\_\{u\_\{k\}\}=2\(\\delta\\hat\{u\}\_\{k\}^\{\\top\}\-\\hat\{u\}\_\{k\}\\delta^\{\\top\}\)=2\\,\\delta\\wedge\\hat\{u\}\_\{k\}\. Hence

R˙R−1=gkH˙Huk\+1⋯HurR−1=gk\(H˙Huk\)gk−1=2\(gkδ\)∧ak,\\dot\{R\}\\,R^\{\-1\}=g\_\{k\}\\dot\{H\}\\,H\_\{u\_\{k\+1\}\}\\cdots H\_\{u\_\{r\}\}\\,R^\{\-1\}=g\_\{k\}\(\\dot\{H\}H\_\{u\_\{k\}\}\)g\_\{k\}^\{\-1\}=2\\,\(g\_\{k\}\\delta\)\\wedge a\_\{k\},usingHuk\+1⋯HurR−1=\(gkHuk\)−1H\_\{u\_\{k\+1\}\}\\cdots H\_\{u\_\{r\}\}R^\{\-1\}=\(g\_\{k\}H\_\{u\_\{k\}\}\)^\{\-1\}andg⁡\(x∧y\)​g⊤=g​x∧g​yg\(x\\wedge y\)g^\{\\top\}=gx\\wedge gyfor orthogonalgg\. Asδ\\deltaranges overu^k⟂\\hat\{u\}\_\{k\}^\{\\perp\},gk​δg\_\{k\}\\deltaranges overak⟂a\_\{k\}^\{\\perp\}, andak⟂∧ak=ℝn∧aka\_\{k\}^\{\\perp\}\\wedge a\_\{k\}=\\mathbb\{R\}^\{n\}\\wedge a\_\{k\}becauseak∧ak=0a\_\{k\}\\wedge a\_\{k\}=0\. Summing overkkgives the claim\.

*\(a\), “⊆\\subseteq”\.*R=Hu1⋯HurR=H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}is orthogonal withdetR=\(−1\)r=1\\det R=\(\-1\)^\{r\}=1, and it fixes⋂iui⟂\\bigcap\_\{i\}u\_\{i\}^\{\\perp\}, of dimension at leastn−rn\-r, pointwise\. Sorank⁡\(R−I\)≤r\\rk\(R\-I\)\\leq randW0​R−W0=W0​\(R−I\)∈ℳ≤rW\_\{0\}R\-W\_\{0\}=W\_\{0\}\(R\-I\)\\in\\mathcal\{M\}\_\{\\leq r\}\.

*\(a\), “⊇\\supseteq”\.*LetW0​R=W0\+XW\_\{0\}R=W\_\{0\}\+XwithR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)andrank⁡X≤r\\rk X\\leq r\. SinceW0W\_\{0\}is injective,k′:=rank⁡\(R−I\)=rank⁡X≤rk^\{\\prime\}:=\\rk\(R\-I\)=\\rk X\\leq r\. By the Cartan–Dieudonné theorem in Scherk’s sharp form \([Auxiliary result B\.22](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\),RRis a product of exactlyk′k^\{\\prime\}reflections anddetR=\(−1\)k′\\det R=\(\-1\)^\{k^\{\\prime\}\}, sok′k^\{\\prime\}is even\. Append\(r−k′\)/2\(r\-k^\{\\prime\}\)/2pairsHv​Hv=IH\_\{v\}H\_\{v\}=I; thenRRis a product of exactlyrrreflections andW0​R∈Im⁡HRArW\_\{0\}R\\in\\Imop\\mathrm\{HRA\}\_\{r\}\.

*Leaves\.*ForV∈Gr⁡\(r,n\)V\\in\\Gr\(r,n\)letSO⁡\(V\)=\{S∈SO⁡\(n\):S​x=x​for​x∈V⟂\}\\mathrm\{SO\}\(V\)=\\\{S\\in\\mathrm\{SO\}\(n\):Sx=x\\text\{ for \}x\\in V^\{\\perp\}\\\}\. Then

\(W0\+ℳ≤r\)∩W0⋅SO⁡\(n\)=⋃V∈Gr⁡\(r,n\)W0⋅SO⁡\(V\)\.\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)=\\bigcup\_\{V\\in\\Gr\(r,n\)\}W\_\{0\}\\cdot\\mathrm\{SO\}\(V\)\.IfS∈SO⁡\(V\)S\\in\\mathrm\{SO\}\(V\), thenS−IS\-Ivanishes onV⟂V^\{\\perp\}, sorank⁡\(S−I\)≤r\\rk\(S\-I\)\\leq r\. Conversely, ifrank⁡\(R−I\)≤r\\rk\(R\-I\)\\leq r, the fixed spaceF=ker⁡\(R−I\)F=\\ker\(R\-I\)has dimension at leastn−rn\-r, and anyrr\-dimensionalV⊇F⟂V\\supseteq F^\{\\perp\}hasV⟂⊆FV^\{\\perp\}\\subseteq F, soR∈SO⁡\(V\)R\\in\\mathrm\{SO\}\(V\)\. A point withrank⁡\(R−I\)=r\\rk\(R\-I\)=rlies on exactly one leaf,V=im⁡\(R−I\)V=\\operatorname\{im\}\(R\-I\)\.

*TheO⁡\(n\)\\mathrm\{O\}\(n\)\-orbit\.*W0​He1−W0=W0​\(He1−I\)W\_\{0\}H\_\{e\_\{1\}\}\-W\_\{0\}=W\_\{0\}\(H\_\{e\_\{1\}\}\-I\)has rank one, soW0​He1∈W0\+ℳ≤rW\_\{0\}H\_\{e\_\{1\}\}\\in W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. IfW0​He1=W0​RW\_\{0\}H\_\{e\_\{1\}\}=W\_\{0\}RwithR=Hu1⋯HurR=H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}, injectivity givesR=He1R=H\_\{e\_\{1\}\}, whose determinant is−1≠\(−1\)r\-1\\neq\(\-1\)^\{r\}\.

*Dimension and fibre\.*ρ=HRAr\\rho=\\mathrm\{HRA\}\_\{r\}is a rational map on the connected open setQ=\(ℝn∖\{0\}\)rQ=\(\\mathbb\{R\}^\{n\}\\setminus\\\{0\\\}\)^\{r\}, andd​ρu​\(u˙\)=W0​d​Ru​\(u˙\)d\\rho\_\{u\}\(\\dot\{u\}\)=W\_\{0\}\\,dR\_\{u\}\(\\dot\{u\}\)has the rank ofd​RudR\_\{u\}, which by lemma \(ii\) equalsdim\(ℝn∧A⁡\(u\)\)\\dim\(\\mathbb\{R\}^\{n\}\\wedge A\(u\)\)\. By lemma \(i\) this isa​n−a⁡\(a\+1\)/2an\-a\(a\+1\)/2witha=dimA⁡\(u\)≤ra=\\dim A\(u\)\\leq r, a non\-decreasing function ofaafora≤na\\leq n\. Foruk=eku\_\{k\}=e\_\{k\}we getgk=He1⋯Hek−1g\_\{k\}=H\_\{e\_\{1\}\}\\cdots H\_\{e\_\{k\-1\}\}andak=eka\_\{k\}=e\_\{k\}, soa=ra=ris attained\. By[Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(d\),

d⁡\(HRAr\)=maxu⁡rank⁡d​ρu=r​n−r⁡\(r\+1\)2\.d\(\\mathrm\{HRA\}\_\{r\}\)=\\max\_\{u\}\\rk d\\rho\_\{u\}=rn\-\\tfrac\{r\(r\+1\)\}\{2\}\.The computation uses neither the parity ofrrnorr≤n−2r\\leq n\-2, so the same value holds for every1≤r≤n1\\leq r\\leq n; forr∈\{n−1,n\}r\\in\\\{n\-1,n\\\}it equalsdimSO⁡\(n\)\\dim\\mathrm\{SO\}\(n\)\. At a point of maximal rank the fibre is locally a submanifold of dimensionr​n−d=r⁡\(r\+1\)/2rn\-d=r\(r\+1\)/2\([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\), and the points of maximal rank, those witha1,…,ara\_\{1\},\\dots,a\_\{r\}independent, are open and dense inQQ\([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. As a check, a generic point of the meet is fixed by its moved spaceim⁡\(R−I\)∈Gr⁡\(r,n\)\\operatorname\{im\}\(R\-I\)\\in\\Gr\(r,n\), of dimensionr⁡\(n−r\)r\(n\-r\), and by a fixed\-point\-free element ofSO⁡\(r\)\\mathrm\{SO\}\(r\)on it, an open subset of dimensionr⁡\(r−1\)/2r\(r\-1\)/2; the sum isdd\.

*Symmetries\.*Hc​u=HuH\_\{cu\}=H\_\{u\}forc≠0c\\neq 0, so the scalings\(ℝ×\)r\(\\mathbb\{R\}^\{\\times\}\)^\{r\}preserveρ\\rho\. Fromg​Hb​g−1=Hg​bgH\_\{b\}g^\{\-1\}=H\_\{gb\}withg=Hag=H\_\{a\}we getHa​Hb=HHa​b​HaH\_\{a\}H\_\{b\}=H\_\{H\_\{a\}b\}H\_\{a\}, so the Hurwitz moveσi:\(ui,ui\+1\)↦\(Hui​ui\+1,ui\)\\sigma\_\{i\}:\(u\_\{i\},u\_\{i\+1\}\)\\mapsto\(H\_\{u\_\{i\}\}u\_\{i\+1\},u\_\{i\}\)preserves the product, andHui​ui\+1≠0H\_\{u\_\{i\}\}u\_\{i\+1\}\\neq 0\. It is invertible, with inverse\(ui,ui\+1\)↦\(ui\+1,Hui\+1​ui\)\(u\_\{i\},u\_\{i\+1\}\)\\mapsto\(u\_\{i\+1\},H\_\{u\_\{i\+1\}\}u\_\{i\}\), the movesσi,σj\\sigma\_\{i\},\\sigma\_\{j\}with\|i−j\|≥2\\lvert i\-j\\rvert\\geq 2commute, andσ1​σ2​σ1=σ2​σ1​σ2\\sigma\_\{1\}\\sigma\_\{2\}\\sigma\_\{1\}=\\sigma\_\{2\}\\sigma\_\{1\}\\sigma\_\{2\}: on a triple both composites give\(a,b,c\)↦\(Ha​Hb​c,Ha​b,a\)\(a,b,c\)\\mapsto\(H\_\{a\}H\_\{b\}c,\\,H\_\{a\}b,\\,a\), usingHHa​b​Ha​c=Ha​Hb​cH\_\{H\_\{a\}b\}H\_\{a\}c=H\_\{a\}H\_\{b\}c\. So the braid group acts\. The scalings haverr\-dimensional orbits, while the generic fibre has dimensionr⁡\(r\+1\)/2\>rr\(r\+1\)/2\>rforr≥2r\\geq 2, and the braid moves are discrete\. So the exhibited symmetries are not locally transitive on generic fibres; the remaining fibre directions come from refactorisingRRinside its moved space, and this is what we mean by a non\-linear gauge\.

*\(b\)*At the paired initialisation,g2​i−1=\(Hw1Hw1\)⋯\(Hwi−1Hwi−1\)=Ig\_\{2i\-1\}=\(H\_\{w\_\{1\}\}H\_\{w\_\{1\}\}\)\\cdots\(H\_\{w\_\{i\-1\}\}H\_\{w\_\{i\-1\}\}\)=I, soa2​i−1=w^ia\_\{2i\-1\}=\\hat\{w\}\_\{i\}anda2​i=Hwi​w^i=−w^ia\_\{2i\}=H\_\{w\_\{i\}\}\\hat\{w\}\_\{i\}=\-\\hat\{w\}\_\{i\}\. HenceA⁡\(u\)=UA\(u\)=U,R⁡\(u\)=IR\(u\)=I, and by lemmas \(ii\) and \(i\)

S1=W0⋅\(ℝn∧U\)=W0⋅\{Ω∈𝔰​𝔬​\(n\):PU⟂​Ω​PU⟂=0\},dimS1=k​n−k⁡\(k\+1\)2,S\_\{1\}=W\_\{0\}\\cdot\(\\mathbb\{R\}^\{n\}\\wedge U\)=W\_\{0\}\\cdot\\\{\\Omega\\in\\mathfrak\{so\}\(n\):P\_\{U^\{\\perp\}\}\\Omega P\_\{U^\{\\perp\}\}=0\\\},\\qquad\\dim S\_\{1\}=kn\-\\tfrac\{k\(k\+1\)\}\{2\},sinceW0W\_\{0\}is injective\. Equivalently, a direct computation for one pair gives

dd​ε​\(Hw\+ε​a​Hw\+ε​b\)\|ε=0=2∥w∥​\(v​w^⊤−w^​v⊤\),v=\(I−w^​w^⊤\)​\(a−b\),\\tfrac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\varepsilon\}\\big\(H\_\{w\+\\varepsilon a\}H\_\{w\+\\varepsilon b\}\\big\)\\big\|\_\{\\varepsilon=0\}=\\tfrac\{2\}\{\\lVert w\\rVert\}\\big\(v\\hat\{w\}^\{\\top\}\-\\hat\{w\}v^\{\\top\}\\big\),\\qquad v=\(I\-\\hat\{w\}\\hat\{w\}^\{\\top\}\)\(a\-b\),and because every pair equalsIIat the base point the differential of the product is the sum of the pair differentials\. Finally, withr=2​kr=2k,

d−dimS1=2​k​n−k⁡\(2​k\+1\)−k​n\+k⁡\(k\+1\)2=k⁡\(2​n−3​k−1\)2,d\-\\dim S\_\{1\}=2kn\-k\(2k\+1\)\-kn\+\\tfrac\{k\(k\+1\)\}\{2\}=\\tfrac\{k\(2n\-3k\-1\)\}\{2\},which is positive becausen≥2​k\+2n\\geq 2k\+2gives2​n−3​k−1≥k\+32n\-3k\-1\\geq k\+3\.

*\(c\)*By[Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(b\),\(c\), the Cayley mapC⁡\(Ω\)=\(I\+Ω\)​\(I−Ω\)−1C\(\\Omega\)=\(I\+\\Omega\)\(I\-\\Omega\)^\{\-1\}is a diffeomorphism from𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)onto the open set𝒪=\{R∈SO⁡\(n\):−1∉spec⁡R\}\\mathcal\{O\}=\\\{R\\in\\mathrm\{SO\}\(n\):\-1\\notin\\operatorname\{spec\}R\\\}, which containsII, andrank⁡\(C⁡\(Ω\)−I\)=rank⁡Ω\\rk\(C\(\\Omega\)\-I\)=\\rk\\Omega\. By \(a\),Im⁡HRAr∩W0⋅𝒪=W0⋅C⁡\(Σr\)\\Imop\\mathrm\{HRA\}\_\{r\}\\cap W\_\{0\}\\cdot\\mathcal\{O\}=W\_\{0\}\\cdot C\(\\Sigma\_\{r\}\)\. The mapΩ↦W0​C​\(Ω\)\\Omega\\mapsto W\_\{0\}C\(\\Omega\)is a diffeomorphism of𝔰​𝔬​\(n\)\\mathfrak\{so\}\(n\)onto an open subset of the embedded submanifoldW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\(a linear injection ofSO⁡\(n\)\\mathrm\{SO\}\(n\)\), and it carriesΣr\\Sigma\_\{r\}ontoIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}nearW0W\_\{0\}\. So the germs correspond\.Σr\\Sigma\_\{r\}is a closed cone and is not linear \([Auxiliary result B\.7](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(c\)\): withr=2​kr=2k, the matrices∑j≤ke2​j−1∧e2​j\\sum\_\{j\\leq k\}e\_\{2j\-1\}\\wedge e\_\{2j\}ander\+1∧er\+2e\_\{r\+1\}\\wedge e\_\{r\+2\}lie inΣr\\Sigma\_\{r\}\(herer\+2≤nr\+2\\leq n\), and their sum has rankr\+2r\+2\. By[Auxiliary result B\.6](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2),Σr\\Sigma\_\{r\}is not aC1C^\{1\}submanifold near00\. IfIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}were aC1C^\{1\}submanifold nearW0W\_\{0\}, it would be one ofW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), and its preimageΣr\\Sigma\_\{r\}under the diffeomorphism would beC1C^\{1\}near00\. SoW0W\_\{0\}is a singular point, the vertex of the germ\. A singular image alone does not give a positive deficit; the apex statement is \(b\)\.

*\(d\), not a product\.*Letℱ=\{\(B,A,R\)∈ℝm×r×ℝr×n×SO⁡\(n\):W0\+B​A=W0​R\}\\mathcal\{F\}=\\\{\(B,A,R\)\\in\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}\\times\\mathrm\{SO\}\(n\):W\_\{0\}\+BA=W\_\{0\}R\\\}be the set\-level fibre product\. The points of the meet of the formW0​C​\(Ω\)W\_\{0\}C\(\\Omega\)withrank⁡Ω=r\\rk\\Omega=rform add\-dimensional submanifold𝒢\\mathcal\{G\}, the image under the diffeomorphism of \(c\) of the rank\-rrpart ofΣr\\Sigma\_\{r\}, which is open and dense inΣr\\Sigma\_\{r\}and of dimensiondd\([Auxiliary result B\.7](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(c\)\)\.𝒢\\mathcal\{G\}is open in the meet, since thererank⁡\(R−I\)≥r\\rk\(R\-I\)\\geq rand−1∉spec⁡R\-1\\notin\\operatorname\{spec\}Rare open conditions, and it is dense: anRRin the meet with eigenvalue−1\-1is a limit of rotations with the same moved planes and the anglesπ\\pireplaced byπ−ε\\pi\-\\varepsilon\([Auxiliary result B\.7](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\)\. Over a pointW0​R∈𝒢W\_\{0\}R\\in\\mathcal\{G\}, the updateW0​\(R−I\)W\_\{0\}\(R\-I\)has rank exactlyrr, so near such a pointℱ\\mathcal\{F\}consists of the pairs\(B,A\)\(B,A\)of full rank withB​A∈W0​\(R−I\)BA\\in W\_\{0\}\(R\-I\)forW0​R∈𝒢W\_\{0\}R\\in\\mathcal\{G\}, withRRrecovered fromB​ABAby injectivity\. The map\(B,A\)↦B​A\(B,A\)\\mapsto BAis a submersion from pairs of full rank ontoℳr\\mathcal\{M\}\_\{r\}with fibres theGLr\\mathrm\{GL\}\_\{r\}\-orbits, of dimensionr2r^\{2\}\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\)\. A submersion is transverse to every submanifold, so by[Auxiliary result B\.2](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(a\), applied with targetℳr\\mathcal\{M\}\_\{r\},ℱ\\mathcal\{F\}is there a submanifold of dimensiond\+r2=r​n\+r⁡\(r−1\)/2d\+r^\{2\}=rn\+r\(r\-1\)/2\. If HRA, or any method with anr​nrn\-dimensional parameter space, were the product, its underlying set would be in bijection withℱ\\mathcal\{F\}through its two smooth projections \([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(b\)\), soℱ\\mathcal\{F\}would be the image of anr​nrn\-manifoldPPunder a smooth mapι\\iotainto the ambient spaceE=ℝm×r×ℝr×n×ℝn×nE=\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}\\times\\mathbb\{R\}^\{n\\times n\}\. This is impossible, becauseℱ\\mathcal\{F\}contains an embedded submanifold𝒩\\mathcal\{N\}of dimensionD=r​n\+r⁡\(r−1\)/2\>r​nD=rn\+r\(r\-1\)/2\>rn\(the part over𝒢\\mathcal\{G\}\)\. Fixp∈𝒩p\\in\\mathcal\{N\}and letπ:E→Tp​𝒩≅ℝD\\pi:E\\to T\_\{p\}\\mathcal\{N\}\\cong\\mathbb\{R\}^\{D\}be the orthogonal projection, which is11\-Lipschitz\. Its restriction to𝒩\\mathcal\{N\}has invertible differential atpp, so by the inverse function theoremπ⁡\(N′\)\\pi\(N^\{\\prime\}\)contains a closed ballB¯\\bar\{B\}of positive volume for a compact neighbourhoodN′N^\{\\prime\}ofppin𝒩\\mathcal\{N\}\. CoverPPby countably many compact setsKj=φj​\(\[0,1\]r​n\)K\_\{j\}=\\varphi\_\{j\}\(\[0,1\]^\{rn\}\), images of cubes under charts, on whichι∘φj\\iota\\circ\\varphi\_\{j\}is Lipschitz with a constantLjL\_\{j\}\. Cutting the cube intokr​nk^\{rn\}subcubes of side1/k1/kcoversπ⁡\(ι⁡\(Kj\)\)\\pi\(\\iota\(K\_\{j\}\)\)bykr​nk^\{rn\}sets of diameter at mostLj​r​n/kL\_\{j\}\\sqrt\{rn\}/k, so its outer Lebesgue measure is at mostkr​n​ωD​\(Lj​r​n/k\)Dk^\{rn\}\\,\\omega\_\{D\}\\,\(L\_\{j\}\\sqrt\{rn\}/k\)^\{D\}, which tends to00ask→∞k\\to\\inftybecauseD\>r​nD\>rn\. The countably many null setsπ⁡\(ι⁡\(Kj\)\)\\pi\(\\iota\(K\_\{j\}\)\)would then coverπ⁡\(N′\)⊇B¯\\pi\(N^\{\\prime\}\)\\supseteq\\bar\{B\}, which is impossible\. For the parenthetical remark: overW0W\_\{0\}itselfℱ\\mathcal\{F\}contains\{\(B,0,I\)\}\\\{\(B,0,I\)\\\}, of dimensionm​rmr, while over points of𝒢\\mathcal\{G\}its fibre has dimensionr2r^\{2\}\. So the fibres ofℱ\\mathcal\{F\}over the meet jump, and wheneverm​r\>d\+r2mr\>d\+r^\{2\}\(for instance whenm≥n\+rm\\geq n\+r, as at\(m,n,r\)=\(12,9,4\)\(m,n,r\)=\(12,9,4\), wherem​r=48\>42mr=48\>42\) the local dimension ofℱ\\mathcal\{F\}at these points exceedsd\+r2d\+r^\{2\}andℱ\\mathcal\{F\}is not pure\-dimensional\.

*\(d\), exactly orthogonal OFT\.*IfC⁡\(Ω\)​x=−xC\(\\Omega\)x=\-x, theny=\(I−Ω\)−1​xy=\(I\-\\Omega\)^\{\-1\}xsatisfies\(I\+Ω\)​y=−\(I−Ω\)​y\(I\+\\Omega\)y=\-\(I\-\\Omega\)y, so2​y=02y=0andx=0x=0; this is the statement thatCCmisses eigenvalue−1\-1\. Exact full\-block Cayley OFT has imageW0⋅C⁡\(𝔰​𝔬​\(n\)\)=W0⋅𝒪W\_\{0\}\\cdot C\(\\mathfrak\{so\}\(n\)\)=W\_\{0\}\\cdot\\mathcal\{O\}\(input\-side formsW0​C​\(Ω\)⊤=W0​C​\(−Ω\)W\_\{0\}C\(\\Omega\)^\{\\top\}=W\_\{0\}C\(\-\\Omega\)give the same set\)\. Its intersection withW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}isW0⋅C⁡\(Σr\)⊆Im⁡HRArW\_\{0\}\\cdot C\(\\Sigma\_\{r\}\)\\subseteq\\Imop\\mathrm\{HRA\}\_\{r\}, which contains𝒢\\mathcal\{G\}and is therefore dense inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\. The pointW0​He1​He2W\_\{0\}H\_\{e\_\{1\}\}H\_\{e\_\{2\}\}lies inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\(pad with pairs\), and by injectivity it equalsW0​RW\_\{0\}Ronly forR=diag⁡\(−1,−1,1,…,1\)R=\\diag\(\-1,\-1,1,\\dots,1\), which has eigenvalue−1\-1; so it is missing\. For block Cayley OFT with block sizeb<nb<nthe image consists ofW0​RW\_\{0\}RwithRRblock\-diagonal for a partition with at least two blocks\. Choose coordinatesp,qp,qin different blocks andv=\(ep\+eq\)/2v=\(e\_\{p\}\+e\_\{q\}\)/\\sqrt\{2\}\. ThenR=Hep​Hv∈SO⁡\(n\)R=H\_\{e\_\{p\}\}H\_\{v\}\\in\\mathrm\{SO\}\(n\)hasrank⁡\(R−I\)≤2≤r\\rk\(R\-I\)\\leq 2\\leq randR​ep=Hep​eq=eqRe\_\{p\}=H\_\{e\_\{p\}\}e\_\{q\}=e\_\{q\}, soRRis not block\-diagonal\. HenceW0​R∈Im⁡HRArW\_\{0\}R\\in\\Imop\\mathrm\{HRA\}\_\{r\}lies outside the image of block OFT, by injectivity\.

*\(d\), HF’s default OFT\.*Its image consists of theW0​R​\(Q\)W\_\{0\}R\(Q\),R⁡\(Q\)=I\+2​Q\+2​Q2\+2​Q3\+Q4R\(Q\)=I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}, withQQskew and block\-diagonal \(the input\-side form usesR​\(Q\)⊤=R⁡\(−Q\)R\(Q\)^\{\\top\}=R\(\-Q\), of the same kind\)\. By the proof of[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),R​\(Q\)⊤​R​\(Q\)=\(I−Q4\)2R\(Q\)^\{\\top\}R\(Q\)=\(I\-Q^\{4\}\)^\{2\}has eigenvalues\(1−θk4\)2\(1\-\\theta\_\{k\}^\{4\}\)^\{2\}, soR⁡\(Q\)R\(Q\)is orthogonal only if everyθk∈\{0,21/4\}\\theta\_\{k\}\\in\\\{0,2^\{1/4\}\\\}\. If∥Q∥op<21/4\\lVert Q\\rVert\_\{\\mathrm\{op\}\}<2^\{1/4\}this forces everyθk=0\\theta\_\{k\}=0, henceQ=0Q=0since a skew matrix is normal\. AsIm⁡HRAr⊆W0⋅O⁡\(n\)\\Imop\\mathrm\{HRA\}\_\{r\}\\subseteq W\_\{0\}\\cdot\\mathrm\{O\}\(n\)andW0W\_\{0\}is injective, the parameters nearQ=0Q=0reachIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}only atW0W\_\{0\}\. ForQ=θ⁡\(e1​e2⊤−e2​e1⊤\)Q=\\theta\(e\_\{1\}e\_\{2\}^\{\\top\}\-e\_\{2\}e\_\{1\}^\{\\top\}\)withθ∉\{0,21/4\}\\theta\\notin\\\{0,2^\{1/4\}\\\}\(one plane, inside one block\),R⁡\(Q\)−I=Q⁡\(2​I\+2​Q\+2​Q2\+Q3\)R\(Q\)\-I=Q\(2I\+2Q\+2Q^\{2\}\+Q^\{3\}\)has rank at most2≤r2\\leq rwhileR⁡\(Q\)R\(Q\)is not orthogonal, soW0​R​\(Q\)∈W0\+ℳ≤rW\_\{0\}R\(Q\)\\in W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}lies outsideW0⋅O⁡\(n\)⊇Im⁡HRArW\_\{0\}\\cdot\\mathrm\{O\}\(n\)\\supseteq\\Imop\\mathrm\{HRA\}\_\{r\}\. Conversely, an orthogonalR⁡\(Q\)R\(Q\)acts on each plane ofQQwithθk=21/4\\theta\_\{k\}=2^\{1/4\}as−C⁡\(21/4​J\)\-C\(2^\{1/4\}J\), a rotation byπ\+2​arctan⁡21/4\\pi\+2\\arctan 2^\{1/4\}, and as the identity onker⁡Q\\ker Q; so a rotation by a small angleφ≠0\\varphi\\neq 0in one plane, which lies inIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}by \(a\), is not of the formR⁡\(Q\)R\(Q\)\. The two images are therefore incomparable\.

*\(d\), implementation\.*For orthonormalu^1,…,u^r\\hat\{u\}\_\{1\},\\dots,\\hat\{u\}\_\{r\},\(I−2​u^1​u^1⊤\)​\(I−2​u^2​u^2⊤\)=I−2​u^1​u^1⊤−2​u^2​u^2⊤\(I\-2\\hat\{u\}\_\{1\}\\hat\{u\}\_\{1\}^\{\\top\}\)\(I\-2\\hat\{u\}\_\{2\}\\hat\{u\}\_\{2\}^\{\\top\}\)=I\-2\\hat\{u\}\_\{1\}\\hat\{u\}\_\{1\}^\{\\top\}\-2\\hat\{u\}\_\{2\}\\hat\{u\}\_\{2\}^\{\\top\}becauseu^1⊤​u^2=0\\hat\{u\}\_\{1\}^\{\\top\}\\hat\{u\}\_\{2\}=0, and by induction∏i\(I−2​u^i​u^i⊤\)=I−2​PU\\prod\_\{i\}\(I\-2\\hat\{u\}\_\{i\}\\hat\{u\}\_\{i\}^\{\\top\}\)=I\-2P\_\{U\}withU=span⁡\{u^i\}U=\\spanop\\\{\\hat\{u\}\_\{i\}\\\}\. So the Gram–Schmidt variant has image\{W0​\(I−2​PU\):U∈Gr⁡\(r,n\)\}\\\{W\_\{0\}\(I\-2P\_\{U\}\):U\\in\\Gr\(r,n\)\\\}\. The mapU↦W0​\(I−2​PU\)U\\mapsto W\_\{0\}\(I\-2P\_\{U\}\)is injective \(PUP\_\{U\}determinesUU, andW0W\_\{0\}is injective\) and is an immersion of the compact manifoldGr⁡\(r,n\)\\Gr\(r,n\), so the image has dimensionr⁡\(n−r\)r\(n\-r\)\. It never containsW0W\_\{0\}, sinceI−2​PU≠II\-2P\_\{U\}\\neq IforU≠0U\\neq 0\. With duplicated columns, Gram–Schmidt meets a column that lies in the span of the previous ones, whose residual vanishes, and normalising it divides by zero up to rounding\. Injectivity ofW0∈ℝout×inW\_\{0\}\\in\\mathbb\{R\}^\{\\texttt\{out\}\\times\\texttt\{in\}\}needsout≥in\\texttt\{out\}\\geq\\texttt\{in\}and full column rank, which fails for down\_proj and for GQA key and value projections\. ∎

### C\.14 The LOFT example of Exposé II\.B

Example\(LOFT: a leaf, and how to choose it; §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)\)\.LetPr∈ℝr×nP\_\{r\}\\in\\mathbb\{R\}^\{r\\times n\}have orthonormal rows spanningVV, and letO⁡\(V\)=\{I\+Pr⊤​\(T−I\)​Pr:T∈O⁡\(r\)\}\\mathrm\{O\}\(V\)=\\\{I\+P\_\{r\}^\{\\top\}\(T\-I\)P\_\{r\}:T\\in\\mathrm\{O\}\(r\)\\\}be the orthogonal maps that fixV⟂V^\{\\perp\}pointwise\. LOFT trainsρ⁡\(E\)=W0​\(I\+Pr⊤​\(Q⁡\(E\)−I\)​Pr\)\\rho\(E\)=W\_\{0\}\\big\(I\+P\_\{r\}^\{\\top\}\(Q\(E\)\-I\)P\_\{r\}\\big\)withE∈𝔰​𝔬​\(r\)E\\in\\mathfrak\{so\}\(r\),E0=0E\_\{0\}=0, so\|M\|=r⁡\(r−1\)/2\\lvert M\\rvert=r\(r\-1\)/2\. We read it with an exactly orthogonalQQ\(the exponential, or the Cayley mapCC\); the released code defaults to a five\-term Cayley–Neumann series withR⊤​R=\(I−X6\)2R^\{\\top\}R=\(I\-X^\{6\}\)^\{2\}, which is only approximately orthogonal, so like HF’s default OFT \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\) that version leaves its leaf\.\(a\)*One leaf\.*IfW0​Pr⊤W\_\{0\}P\_\{r\}^\{\\top\}has rankrr, then\(W0\+W0​Pr⊤​ℝr×r​Pr\)∩W0⋅O⁡\(n\)=W0⋅O⁡\(V\)\\big\(W\_\{0\}\+W\_\{0\}P\_\{r\}^\{\\top\}\\,\\mathbb\{R\}^\{r\\times r\}P\_\{r\}\\big\)\\cap W\_\{0\}\\cdot\\mathrm\{O\}\(n\)=W\_\{0\}\\cdot\\mathrm\{O\}\(V\)\. For the principal supportPr=Vr⊤P\_\{r\}=V\_\{r\}^\{\\top\}the first set is the image of LoRA\-XS \(this needsσr​\(W0\)\>0\\sigma\_\{r\}\(W\_\{0\}\)\>0\), so the leaf is LoRA\-XS∩W0⋅O\(n\)\{\}\\cap W\_\{0\}\\cdot\\mathrm\{O\}\(n\)\. Principal\-support LOFT reaches an open dense part of its componentW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\); for injectiveW0W\_\{0\}and a CayleyQQ, its image is exactly LoRA\-XS∩\{\}\\cap\{\}the image of exact full\-block Cayley OFT\.\(b\)*Leaves over the Grassmannian\.*IfW0W\_\{0\}is injective,\(W0\+ℳ≤r\)∩W0⋅SO⁡\(n\)=⋃V∈Gr⁡\(r,n\)W0⋅SO⁡\(V\)\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)=\\bigcup\_\{V\\in\\Gr\(r,n\)\}W\_\{0\}\\cdot\\mathrm\{SO\}\(V\), which for evenrrisIm⁡HRAr\\Imop\\mathrm\{HRA\}\_\{r\}\([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\)\)\. LOFT’s image is the whole leafW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\)with the exponential, and an open dense part of it with a Cayley map, which never reaches eigenvalue−1\-1\.\(c\)*A regular base point\.*IfW0​Pr⊤W\_\{0\}P\_\{r\}^\{\\top\}has rankrr, thenS1=W0​Pr⊤​𝔰​𝔬​\(r\)​PrS\_\{1\}=W\_\{0\}P\_\{r\}^\{\\top\}\\mathfrak\{so\}\(r\)P\_\{r\}has dimensionr⁡\(r−1\)/2=d⁡\(M\)r\(r\-1\)/2=d\(M\): LOFT starts at a smooth point of its leaf and has no gauge, while HRA’s paired initialisation is an apex \([Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\)\)\. Forr≥4r\\geq 4there is no smooth pointed arrow LOFT→\{\}\\to\{\}HRA at that initialisation, although the images are nested \(obstruction test O3 of §[VIII](https://arxiv.org/html/2610.12728#S8)\)\.\(d\)*Choosing the leaf\.*IfQ′​\(0\)=idQ^\{\\prime\}\(0\)=\\mathrm\{id\}\(the exponential, orE↦C⁡\(E/2\)E\\mapsto C\(E/2\)\), the gradient pulled back toEEatθ0\\theta\_\{0\}isPr​skew⁡\(W0⊤​G\)​Pr⊤P\_\{r\}\\skewop\(W\_\{0\}^\{\\top\}G\)P\_\{r\}^\{\\top\}withG=∇L​\(W0\)G=\\nabla L\(W\_\{0\}\); the Cayley mapCCitself doubles it, which does not change the best support\. Its squared norm is at most2​∑k≤r/2μk22\\sum\_\{k\\leq r/2\}\\mu\_\{k\}^\{2\}, where±i​μk\\pm i\\mu\_\{k\}are the eigenvalues ofskew⁡\(W0⊤​G\)\\skewop\(W\_\{0\}^\{\\top\}G\), with equality whenVVspans the invariant subspace of the top⌊r/2⌋\\lfloor r/2\\rfloorpairs\.

\(a\) A rotationT∈O⁡\(r\)T\\in\\mathrm\{O\}\(r\)givesW0​\(I\+Pr⊤​\(T−I\)​Pr\)=W0\+W0​Pr⊤​\(T−I\)​PrW\_\{0\}\(I\+P\_\{r\}^\{\\top\}\(T\-I\)P\_\{r\}\)=W\_\{0\}\+W\_\{0\}P\_\{r\}^\{\\top\}\(T\-I\)P\_\{r\}, so “⊇\\supseteq” holds\. Conversely letW=W0\+B​X​PrW=W\_\{0\}\+BXP\_\{r\}withB=W0​Pr⊤B=W\_\{0\}P\_\{r\}^\{\\top\}, and supposeW=W0​SW=W\_\{0\}SwithS∈O⁡\(n\)S\\in\\mathrm\{O\}\(n\)\. ThenW​W⊤=W0​W0⊤WW^\{\\top\}=W\_\{0\}W\_\{0\}^\{\\top\}, and expanding withPr​Pr⊤=IrP\_\{r\}P\_\{r\}^\{\\top\}=I\_\{r\}givesB⁡\(X\+X⊤\+X​X⊤\)​B⊤=0B\\,\(X\+X^\{\\top\}\+XX^\{\\top\}\)\\,B^\{\\top\}=0\. SinceBBhas full column rank,X\+X⊤\+X​X⊤=0X\+X^\{\\top\}\+XX^\{\\top\}=0, that is\(I\+X\)​\(I\+X\)⊤=I\(I\+X\)\(I\+X\)^\{\\top\}=I, soT=I\+X∈O⁡\(r\)T=I\+X\\in\\mathrm\{O\}\(r\)andW=W0​\(I\+Pr⊤​\(T−I\)​Pr\)∈W0⋅O⁡\(V\)W=W\_\{0\}\(I\+P\_\{r\}^\{\\top\}\(T\-I\)P\_\{r\}\)\\in W\_\{0\}\\cdot\\mathrm\{O\}\(V\)\. ForPr=Vr⊤P\_\{r\}=V\_\{r\}^\{\\top\}we haveB=Ur​ΣrB=U\_\{r\}\\Sigma\_\{r\}, and\{Ur​Σr​X​Vr⊤\}=\{Ur​R​Σr​Vr⊤\}\\\{U\_\{r\}\\Sigma\_\{r\}XV\_\{r\}^\{\\top\}\\\}=\\\{U\_\{r\}R\\,\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\\\}throughR=Σr​X​Σr−1R=\\Sigma\_\{r\}X\\Sigma\_\{r\}^\{\-1\}, which is LoRA\-XS’s image\. With the exponential,TTranges overSO⁡\(r\)\\mathrm\{SO\}\(r\)and LOFT reaches the whole componentW0⋅SO⁡\(V\)W\_\{0\}\\cdot\\mathrm\{SO\}\(V\); with a Cayley map it reaches theS⁡\(T\)=I\+Pr⊤​\(T−I\)​PrS\(T\)=I\+P\_\{r\}^\{\\top\}\(T\-I\)P\_\{r\}withT∈SO⁡\(r\)T\\in\\mathrm\{SO\}\(r\)and no eigenvalue−1\-1, an open dense part of it\. For injectiveW0W\_\{0\},W0​SW\_\{0\}SdeterminesSS, so meeting with exact full\-block Cayley OFT keeps exactly theS⁡\(T\)S\(T\)withT∈SO⁡\(r\)T\\in\\mathrm\{SO\}\(r\)and no eigenvalue−1\-1\(the eigenvalues ofS⁡\(T\)S\(T\)are those ofTTtogether with11\), which is the Cayley image\.

\(b\) EveryS∈SO⁡\(V\)S\\in\\mathrm\{SO\}\(V\)hasrank⁡\(S−I\)≤r\\rk\(S\-I\)\\leq r, which gives “⊇\\supseteq”\. Conversely, ifW0​S−W0W\_\{0\}S\-W\_\{0\}has rank at mostrrandW0W\_\{0\}is injective, thenrank⁡\(S−I\)≤r\\rk\(S\-I\)\\leq r\. The fixed spaceF=ker⁡\(S−I\)F=\\ker\(S\-I\)has dimension at leastn−rn\-r, andF⟂F^\{\\perp\}is invariant\. Anyrr\-dimensionalV⊇F⟂V\\supseteq F^\{\\perp\}works:SSis the identity onV⟂⊆FV^\{\\perp\}\\subseteq Fand has determinant11onVV\. The statements about LOFT’s image are those of \(a\)\.

\(c\) The differential at00isD​ρ0​\(E\)=W0​Pr⊤​Q′​\(0\)​\(E\)​PrD\\rho\_\{0\}\(E\)=W\_\{0\}P\_\{r\}^\{\\top\}Q^\{\\prime\}\(0\)\(E\)P\_\{r\}, withQ′​\(0\)Q^\{\\prime\}\(0\)invertible\. If this vanishes, thenW0​Pr⊤​Q′​\(0\)​\(E\)=0W\_\{0\}P\_\{r\}^\{\\top\}Q^\{\\prime\}\(0\)\(E\)=0becausePrP\_\{r\}has full row rank, andE=0E=0becauseW0​Pr⊤W\_\{0\}P\_\{r\}^\{\\top\}has full column rank\. SodimS1=\|M\|≥d⁡\(M\)≥dimS1\\dim S\_\{1\}=\\lvert M\\rvert\\geq d\(M\)\\geq\\dim S\_\{1\}\. A smooth pointed arrow would giveS1​\(LOFT\)⊆S1​\(HRA\)S\_\{1\}\(\\mathrm\{LOFT\}\)\\subseteq S\_\{1\}\(\\mathrm\{HRA\}\)\([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\), and by[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\) and injectivity this asks everyΩ∈𝔰​𝔬​\(V\)\\Omega\\in\\mathfrak\{so\}\(V\)to satisfyPU⟂​Ω​PU⟂=0P\_\{U^\{\\perp\}\}\\Omega P\_\{U^\{\\perp\}\}=0\. Sincedim\(V∩U⟂\)≥r−r/2≥2\\dim\(V\\cap U^\{\\perp\}\)\\geq r\-r/2\\geq 2, any two orthonormala,b∈V∩U⟂a,b\\in V\\cap U^\{\\perp\}givea∧b=a​b⊤−b​a⊤a\\wedge b=ab^\{\\top\}\-ba^\{\\top\}violating it\.

\(d\) WithQ′​\(0\)=idQ^\{\\prime\}\(0\)=\\mathrm\{id\},

dd​t​L​\(ρ⁡\(t​E\)\)\|t=0=⟨G,W0​Pr⊤​E​Pr⟩=⟨Pr​W0⊤​G​Pr⊤,E⟩=⟨Pr​skew⁡\(W0⊤​G\)​Pr⊤,E⟩,\\tfrac\{\\mathrm\{d\}\}\{\\mathrm\{d\}t\}L\\big\(\\rho\(tE\)\\big\)\\big\|\_\{t=0\}=\\langle G,W\_\{0\}P\_\{r\}^\{\\top\}EP\_\{r\}\\rangle=\\langle P\_\{r\}W\_\{0\}^\{\\top\}GP\_\{r\}^\{\\top\},E\\rangle=\\langle P\_\{r\}\\skewop\(W\_\{0\}^\{\\top\}G\)P\_\{r\}^\{\\top\},E\\rangle,sinceEEis skew\. The squared norm of this gradient is∥Pr​skew⁡\(W0⊤​G\)​Pr⊤∥2\\lVert P\_\{r\}\\skewop\(W\_\{0\}^\{\\top\}G\)P\_\{r\}^\{\\top\}\\rVert^\{2\}, the compression of the skew matrixskew⁡\(W0⊤​G\)\\skewop\(W\_\{0\}^\{\\top\}G\)toVV, so the bound and its equality case are[Auxiliary result B\.17](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\) withk=rk=r\. ∎

### C\.15 Proposition II\.8

Proposition II\.8\(DoRA is LoRA followed by an output scale, at the level of images\)\.Assume1≤r≤min⁡\(m,n\)1\\leq r\\leq\\min\(m,n\)ands≠0s\\neq 0\. LetNNnormalise rows; DoRA’s magnitudeμ∈ℝm\\mu\\in\\mathbb\{R\}^\{m\}is per output neuron, as in HF PEFT and the official code \(the paper’s notation says “column\-wise”, in the transposed convention\)\. Images refer to the merged \(eval\-mode\) weight; HF’s train\-mode forward with lora\_dropout\>0\{\}\>0is not of the formW​xWx\. LetW0W\_\{0\}have no zero row, let𝒰\\mathcal\{U\}be the open set of matrices with no zero row, and letDiagm\\mathrm\{Diag\}\_\{m\}be the monoid of all diagonal matrices\. ThenImDoRAr=\{diag\(μ\)N\(V\):V∈\(W0\+ℳ≤r\)∩𝒰,μ∈ℝm\}=Diagm⋅\(W0\+ℳ≤r\),\\Imop\\mathrm\{DoRA\}\_\{r\}=\\\{\\diag\(\\mu\)\\,N\(V\):V\\in\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\mathcal\{U\},\\ \\mu\\in\\mathbb\{R\}^\{m\}\\\}=\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\),which is exactly the image of “LoRAr\\mathrm\{LoRA\}\_\{r\}followed by a trainable output scaleℓ∈ℝm\\ell\\in\\mathbb\{R\}^\{m\}”\. Withμ\\murestricted to\(ℝ×\)m\(\\mathbb\{R\}^\{\\times\}\)^\{m\}, or equivalently after intersecting with𝒰\\mathcal\{U\}:Im⁡DoRAr∩𝒰=Tm⋅\(W0\+ℳ≤r\)∩𝒰\\Imop\\mathrm\{DoRA\}\_\{r\}\\cap\\mathcal\{U\}=T\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\mathcal\{U\}\. Moreover:•d⁡\(DoRAr\)≤min⁡\(m​n,r⁡\(m\+n−r\)\+m\)d\(\\mathrm\{DoRA\}\_\{r\}\)\\leq\\min\\big\(mn,\\ r\(m\+n\-r\)\+m\\big\)\. At a parameter point withμ∈\(ℝ×\)m\\mu\\in\(\\mathbb\{R\}^\{\\times\}\)^\{m\}andX=s​B​A∈ℳrX=sBA\\in\\mathcal\{M\}\_\{r\}, withU=col⁡XU=\\operatorname\{col\}XandR=row⁡XR=\\operatorname\{row\}X, the Jacobian rank isr⁡\(m\+n−r\)\+m−dimker⁡\(δ↦PU⟂​diag⁡\(δ\)​W0​PR⟂\)r\(m\+n\-r\)\+m\-\\dim\\ker\\big\(\\delta\\mapsto P\_\{U^\{\\perp\}\}\\diag\(\\delta\)W\_\{0\}P\_\{R^\{\\perp\}\}\\big\); where someμi=0\\mu\_\{i\}=0it is smaller whenr≥2r\\geq 2, and forr=1r=1when some roww0,iw\_\{0,i\}withμi=0\\mu\_\{i\}=0lies outsideRR\. For genericW0W\_\{0\},d=min⁡\(m​n,r⁡\(m\+n−r\)\+m\)d=\\min\\big\(mn,\\ r\(m\+n\-r\)\+m\\big\); in particulard=r⁡\(m\+n−r\)\+md=r\(m\+n\-r\)\+miff\(m−r\)​\(n−r\)≥m\(m\-r\)\(n\-r\)\\geq m, which holds for all practical shapes\. For specialW0W\_\{0\}it is smaller: forW0=a​c⊤W\_\{0\}=ac^\{\\top\}with allai≠0a\_\{i\}\\neq 0andr<nr<n,d=r⁡\(m\+n−r\)\+m−rd=r\(m\+n\-r\)\+m\-r, strictly below the generic value iffr<mr<mandr≤n−2r\\leq n\-2\.•The image covariance group \([Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\) containsMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\.•AtB=0B=0, in HF’s magnitude coordinates,Δ​W=\(diag⁡\(μi/∥w0,i∥\)−I\)​W0\\Delta W=\\big\(\\diag\(\\mu\_\{i\}/\\lVert w\_\{0,i\}\\rVert\)\-I\\big\)W\_\{0\}, which can have rank up torank⁡W0\\rk W\_\{0\}, so DoRA escapes LoRA’s rank bound\.

DoRA is the mapρ⁡\(μ,B,A\)=diag⁡\(μ\)​N​\(V\)\\rho\(\\mu,B,A\)=\\diag\(\\mu\)N\(V\)withV=W0\+s​B​AV=W\_\{0\}\+sBA, defined on the open set𝒟=\{\(μ,B,A\):V∈𝒰\}\\mathcal\{D\}=\\\{\(\\mu,B,A\):V\\in\\mathcal\{U\}\\\}, whereN⁡\(V\)N\(V\)divides rowiiby∥vi∥\\lVert v\_\{i\}\\rVert\. Sinces≠0s\\neq 0and every matrix of rank at mostrrfactors with inner dimensionrr\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\),\{s​B​A\}=ℳ≤r\\\{sBA\\\}=\\mathcal\{M\}\_\{\\leq r\}\.

*The image\.*diag⁡\(μ\)​N​\(V\)=diag⁡\(μi/∥vi∥\)​V∈Diagm⋅\(W0\+ℳ≤r\)\\diag\(\\mu\)N\(V\)=\\diag\(\\mu\_\{i\}/\\lVert v\_\{i\}\\rVert\)V\\in\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\), which gives the first inclusion; the first equality is the definition ofρ\\rho\. Conversely, takediag⁡\(ℓ\)​V\\diag\(\\ell\)VwithV=W0\+XV=W\_\{0\}\+X,rank⁡X≤r\\rk X\\leq r, and letZ=\{i:vi=0\}Z=\\\{i:v\_\{i\}=0\\\}\. For0<ε≤10<\\varepsilon\\leq 1put

V′=W0\+\(I−ε​∑i∈ZEi​i\)​X\.V^\{\\prime\}=W\_\{0\}\+\\Big\(I\-\\varepsilon\\sum\_\{i\\in Z\}E\_\{ii\}\\Big\)X\.Its update still has rank at mostrr\. Its rows outsideZZare those ofVV, and fori∈Zi\\in Zwe havexi=−w0,ix\_\{i\}=\-w\_\{0,i\}, so rowiiofV′V^\{\\prime\}isw0,i−\(1−ε\)​w0,i=ε​w0,i≠0w\_\{0,i\}\-\(1\-\\varepsilon\)w\_\{0,i\}=\\varepsilon w\_\{0,i\}\\neq 0\. HenceV′∈𝒰V^\{\\prime\}\\in\\mathcal\{U\}\. Withℓi′=0\\ell\_\{i\}^\{\\prime\}=0onZZandℓi′=ℓi\\ell^\{\\prime\}\_\{i\}=\\ell\_\{i\}elsewhere,diag⁡\(ℓ′\)​V′=diag⁡\(ℓ\)​V\\diag\(\\ell^\{\\prime\}\)V^\{\\prime\}=\\diag\(\\ell\)V, since both vanish on the rows inZZ\. Takingμi=ℓi′​∥vi′∥\\mu\_\{i\}=\\ell\_\{i\}^\{\\prime\}\\lVert v\_\{i\}^\{\\prime\}\\rVertgivesdiag⁡\(μ\)​N​\(V′\)=diag⁡\(ℓ\)​V\\diag\(\\mu\)N\(V^\{\\prime\}\)=\\diag\(\\ell\)V\. The setDiagm⋅\(W0\+ℳ≤r\)\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)is by definition the image of\(ℓ,B,A\)↦diag⁡\(ℓ\)​\(W0\+s​B​A\)\(\\ell,B,A\)\\mapsto\\diag\(\\ell\)\(W\_\{0\}\+sBA\)\.

*The torus form\.*Ifμ∈\(ℝ×\)m\\mu\\in\(\\mathbb\{R\}^\{\\times\}\)^\{m\}, thendiag⁡\(μi/∥vi∥\)∈Tm\\diag\(\\mu\_\{i\}/\\lVert v\_\{i\}\\rVert\)\\in T\_\{m\}and the result has no zero row\. Conversely, ifD​V∈𝒰DV\\in\\mathcal\{U\}withD∈TmD\\in T\_\{m\}andV∈W0\+ℳ≤rV\\in W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, thenV∈𝒰V\\in\\mathcal\{U\}andμi=Di​i​∥vi∥≠0\\mu\_\{i\}=D\_\{ii\}\\lVert v\_\{i\}\\rVert\\neq 0realisesD​VDV\. Finally,diag⁡\(ℓ\)​V∈𝒰\\diag\(\\ell\)V\\in\\mathcal\{U\}forces everyℓi≠0\\ell\_\{i\}\\neq 0andV∈𝒰V\\in\\mathcal\{U\}, soIm⁡DoRAr∩𝒰=Tm⋅\(W0\+ℳ≤r\)∩𝒰\\Imop\\mathrm\{DoRA\}\_\{r\}\\cap\\mathcal\{U\}=T\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\cap\\mathcal\{U\}, and these are exactly the values atμ∈\(ℝ×\)m\\mu\\in\(\\mathbb\{R\}^\{\\times\}\)^\{m\}\.

*A change of coordinates\.*Letρ′​\(ℓ,B,A\)=diag⁡\(ℓ\)​\(W0\+s​B​A\)\\rho^\{\\prime\}\(\\ell,B,A\)=\\diag\(\\ell\)\(W\_\{0\}\+sBA\)be LoRA followed by an output scale\. The mapφ⁡\(μ,B,A\)=\(μi/∥vi∥,B,A\)\\varphi\(\\mu,B,A\)=\(\\mu\_\{i\}/\\lVert v\_\{i\}\\rVert,B,A\)is a diffeomorphism of𝒟\\mathcal\{D\}onto\{\(ℓ,B,A\):W0\+s​B​A∈𝒰\}\\\{\(\\ell,B,A\):W\_\{0\}\+sBA\\in\\mathcal\{U\}\\\}, with inverse\(ℓ,B,A\)↦\(ℓi​∥vi∥,B,A\)\(\\ell,B,A\)\\mapsto\(\\ell\_\{i\}\\lVert v\_\{i\}\\rVert,B,A\), andρ=ρ′∘φ\\rho=\\rho^\{\\prime\}\\circ\\varphi\. So the Jacobian rank ofρ\\rhoatppequals that ofρ′\\rho^\{\\prime\}atφ⁡\(p\)\\varphi\(p\), andμi=0\\mu\_\{i\}=0iffℓi=0\\ell\_\{i\}=0\.

*Upper bound\.*ρ′\\rho^\{\\prime\}factors through\(ℓ,B,A\)↦\(ℓ,W0\+s​B​A\)∈ℝm×\(W0\+ℳ≤r\)\(\\ell,B,A\)\\mapsto\(\\ell,W\_\{0\}\+sBA\)\\in\\mathbb\{R\}^\{m\}\\times\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\), whose image has dimensionm\+r⁡\(m\+n−r\)m\+r\(m\+n\-r\)\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(a\); forr=min⁡\(m,n\)r=\\min\(m,n\),ℳ≤r=ℝm×n\\mathcal\{M\}\_\{\\leq r\}=\\mathbb\{R\}^\{m\\times n\}has dimensionm​n=r⁡\(m\+n−r\)mn=r\(m\+n\-r\)\)\. A polynomial map does not raise dimension \([Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\), and triviallyd≤m​nd\\leq mn\.

*The Jacobian rank\.*The differential ofρ′\\rho^\{\\prime\}at\(ℓ,B,A\)\(\\ell,B,A\)is\(δ,B˙,A˙\)↦diag⁡\(δ\)​V\+diag⁡\(ℓ\)​s​\(B˙​A\+B​A˙\)\(\\delta,\\dot\{B\},\\dot\{A\}\)\\mapsto\\diag\(\\delta\)V\+\\diag\(\\ell\)\\,s\(\\dot\{B\}A\+B\\dot\{A\}\)\. Letℓ\\ellbe invertible andX=s​B​A∈ℳrX=sBA\\in\\mathcal\{M\}\_\{r\}, so thatBBandAAhave full rank and\{s⁡\(B˙​A\+B​A˙\)\}=TX​ℳr=\{D:PU⟂​D​PR⟂=0\}\\\{s\(\\dot\{B\}A\+B\\dot\{A\}\)\\\}=T\_\{X\}\\mathcal\{M\}\_\{r\}=\\\{D:P\_\{U^\{\\perp\}\}DP\_\{R^\{\\perp\}\}=0\\\}, of dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\)\. Substitutingδ=diag⁡\(ℓ\)​δ′\\delta=\\diag\(\\ell\)\\delta^\{\\prime\}, the image of the differential isdiag⁡\(ℓ\)​\[\{diag⁡\(δ′\)​V\}\+TX​ℳr\]\\diag\(\\ell\)\\big\[\\\{\\diag\(\\delta^\{\\prime\}\)V\\\}\+T\_\{X\}\\mathcal\{M\}\_\{r\}\\big\], whose dimension is that of the bracket\. The first summand has dimensionmm, becauseVVhas no zero row\. A matrixdiag⁡\(δ′\)​V\\diag\(\\delta^\{\\prime\}\)Vlies inTX​ℳrT\_\{X\}\\mathcal\{M\}\_\{r\}iffPU⟂​diag⁡\(δ′\)​V​PR⟂=0P\_\{U^\{\\perp\}\}\\diag\(\\delta^\{\\prime\}\)VP\_\{R^\{\\perp\}\}=0, andV​PR⟂=W0​PR⟂VP\_\{R^\{\\perp\}\}=W\_\{0\}P\_\{R^\{\\perp\}\}becauseX​PR⟂=0XP\_\{R^\{\\perp\}\}=0\. So the intersection has the dimension of the kernelKKofδ↦PU⟂​diag⁡\(δ\)​W0​PR⟂\\delta\\mapsto P\_\{U^\{\\perp\}\}\\diag\(\\delta\)W\_\{0\}P\_\{R^\{\\perp\}\}, and the rank isr⁡\(m\+n−r\)\+m−dimKr\(m\+n\-r\)\+m\-\\dim K\.

*Zero magnitudes\.*LetZ0=\{i:ℓi=0\}≠∅Z\_\{0\}=\\\{i:\\ell\_\{i\}=0\\\}\\neq\\emptyset, keepX∈ℳrX\\in\\mathcal\{M\}\_\{r\}, and write𝒮=\{diag⁡\(δ\)​V\}\\mathcal\{S\}=\\\{\\diag\(\\delta\)V\\\}andT=TX​ℳrT=T\_\{X\}\\mathcal\{M\}\_\{r\}\. LetΠ0\\Pi\_\{0\}keep the rows inZ0Z\_\{0\}andΠ1\\Pi\_\{1\}the others\. In the image of the differential, the rows inZ0Z\_\{0\}areδi​vi\\delta\_\{i\}v\_\{i\}and involve onlyδZ0\\delta\_\{Z\_\{0\}\}, while the other rows do not involveδZ0\\delta\_\{Z\_\{0\}\}; so the image has dimension\|Z0\|\+dimΠ1​\(𝒮\+T\)\\lvert Z\_\{0\}\\rvert\+\\dim\\Pi\_\{1\}\(\\mathcal\{S\}\+T\), after the same substitution as above on the rows outsideZ0Z\_\{0\}\. On the other hand, the valueF=dim\(𝒮\+T\)F=\\dim\(\\mathcal\{S\}\+T\)of the formula satisfiesF=dimΠ1​\(𝒮\+T\)\+dim\(\(𝒮\+T\)∩ker⁡Π1\)F=\\dim\\Pi\_\{1\}\(\\mathcal\{S\}\+T\)\+\\dim\\big\(\(\\mathcal\{S\}\+T\)\\cap\\ker\\Pi\_\{1\}\\big\), and\(𝒮\+T\)∩ker⁡Π1\(\\mathcal\{S\}\+T\)\\cap\\ker\\Pi\_\{1\}contains every∑i∈Z0ei​\(ci​vi\+yi\)⊤\\sum\_\{i\\in Z\_\{0\}\}e\_\{i\}\(c\_\{i\}v\_\{i\}\+y\_\{i\}\)^\{\\top\}withyi∈Ry\_\{i\}\\in R, sinceei​vi⊤∈𝒮e\_\{i\}v\_\{i\}^\{\\top\}\\in\\mathcal\{S\}andei​y⊤∈Te\_\{i\}y^\{\\top\}\\in Tfory∈Ry\\in R\. Asvi−w0,i∈Rv\_\{i\}\-w\_\{0,i\}\\in R, the rank at the point is at most

F−∑i∈Z0\(dimspan⁡\(w0,i,R\)−1\)≤F−\|Z0\|​\(r−1\),F\-\\sum\_\{i\\in Z\_\{0\}\}\\big\(\\dim\\spanop\(w\_\{0,i\},R\)\-1\\big\)\\leq F\-\\lvert Z\_\{0\}\\rvert\(r\-1\),which is smaller thanFFwheneverr≥2r\\geq 2or somew0,iw\_\{0,i\}withi∈Z0i\\in Z\_\{0\}lies outsideRR, in particular at generic points with someμi=0\\mu\_\{i\}=0\.

*The generic value\.*Ifr=min⁡\(m,n\)r=\\min\(m,n\), the image isDiagm⋅ℝm×n=ℝm×n\\mathrm\{Diag\}\_\{m\}\\cdot\\mathbb\{R\}^\{m\\times n\}=\\mathbb\{R\}^\{m\\times n\}andd=m​nd=mn, which is the claimed value\. Letr<min⁡\(m,n\)r<\\min\(m,n\), and letE∈ℝm×\(m−r\)E\\in\\mathbb\{R\}^\{m\\times\(m\-r\)\}andF∈ℝn×\(n−r\)F\\in\\mathbb\{R\}^\{n\\times\(n\-r\)\}have orthonormal columns spanningU⟂U^\{\\perp\}andR⟂R^\{\\perp\}\. ThenPU⟂​D​PR⟂=0P\_\{U^\{\\perp\}\}DP\_\{R^\{\\perp\}\}=0iffE⊤​D​F=0E^\{\\top\}DF=0, and

E⊤​diag⁡\(δ\)​W0​F=∑iδi​\(E⊤​ei\)​\(F⊤​w0,i\)⊤,E^\{\\top\}\\diag\(\\delta\)W\_\{0\}F=\\sum\_\{i\}\\delta\_\{i\}\\,\(E^\{\\top\}e\_\{i\}\)\(F^\{\\top\}w\_\{0,i\}\)^\{\\top\},som−dimKm\-\\dim Kis the dimensionkkof the span of themmrank\-one matricesti=\(E⊤​ei\)​\(F⊤​w0,i\)⊤t\_\{i\}=\(E^\{\\top\}e\_\{i\}\)\(F^\{\\top\}w\_\{0,i\}\)^\{\\top\}inℝ\(m−r\)×\(n−r\)\\mathbb\{R\}^\{\(m\-r\)\\times\(n\-r\)\}, and the rank isr⁡\(m\+n−r\)\+kr\(m\+n\-r\)\+kwithk≤min⁡\(m,\(m−r\)​\(n−r\)\)k\\leq\\min\\big\(m,\(m\-r\)\(n\-r\)\\big\)\. Sincer⁡\(m\+n−r\)\+\(m−r\)​\(n−r\)=m​nr\(m\+n\-r\)\+\(m\-r\)\(n\-r\)=mn, it suffices to reachk=min⁡\(m,\(m−r\)​\(n−r\)\)k=\\min\(m,\(m\-r\)\(n\-r\)\)\. A witness: chooseEEwhose rowspi=E⊤​eip\_\{i\}=E^\{\\top\}e\_\{i\}are in general position \(everym−rm\-rof them independent; orthonormalising the columns of a generic matrix keeps this\), letU=\(col⁡E\)⟂U=\(\\operatorname\{col\}E\)^\{\\perp\},R=span⁡\{e1,…,er\}R=\\spanop\\\{e\_\{1\},\\dots,e\_\{r\}\\\}andF=\[er\+1⋯en\]F=\[e\_\{r\+1\}\\cdots e\_\{n\}\]\. Assign to eachiia column indexc⁡\(i\)∈\{1,…,n−r\}c\(i\)\\in\\\{1,\\dots,n\-r\\\}: ifm≤\(m−r\)​\(n−r\)m\\leq\(m\-r\)\(n\-r\), use each index at mostm−rm\-rtimes; otherwise give each index to exactlym−rm\-rof the first\(m−r\)​\(n−r\)\(m\-r\)\(n\-r\)values ofii, and arbitrary indices to the rest\. Putw0,i=er\+c⁡\(i\)w\_\{0,i\}=e\_\{r\+c\(i\)\}, soF⊤​w0,i=ec⁡\(i\)F^\{\\top\}w\_\{0,i\}=e\_\{c\(i\)\}andti=pi​ec⁡\(i\)⊤t\_\{i\}=p\_\{i\}e\_\{c\(i\)\}^\{\\top\}haspip\_\{i\}in columnc⁡\(i\)c\(i\)\. Matrices supported in different columns are independent, and in one column at mostm−rm\-rvectors in general position are independent, while exactlym−rm\-rof them form a basis; so thetit\_\{i\}spanmin⁡\(m,\(m−r\)​\(n−r\)\)\\min\(m,\(m\-r\)\(n\-r\)\)dimensions\. ThisW0W\_\{0\}has no zero row, and withB,AB,Aof full rank spanningUUandRRandμi=∥vi∥\\mu\_\{i\}=\\lVert v\_\{i\}\\rVertthe point lies in𝒟\\mathcal\{D\}, sincevi=w0,i\+xiv\_\{i\}=w\_\{0,i\}\+x\_\{i\}withxi∈R⟂w0,ix\_\{i\}\\in R\\perp w\_\{0,i\}\. Now writePU⟂=I−B​\(B⊤​B\)−1​B⊤P\_\{U^\{\\perp\}\}=I\-B\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}andPR⟂=I−A⊤​\(A​A⊤\)−1​AP\_\{R^\{\\perp\}\}=I\-A^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}A\. After clearing the denominatorsdet\(B⊤​B\)\\det\(B^\{\\top\}B\)anddet\(A​A⊤\)\\det\(AA^\{\\top\}\), thek×kk\\times kminors of the matrix ofδ↦PU⟂​diag⁡\(δ\)​W0​PR⟂\\delta\\mapsto P\_\{U^\{\\perp\}\}\\diag\(\\delta\)W\_\{0\}P\_\{R^\{\\perp\}\}are polynomials in\(W0,B,A\)\(W\_\{0\},B,A\), and one of them is nonzero at the witness\. Viewed as a polynomial in\(B,A\)\(B,A\), its coefficients are polynomials inW0W\_\{0\}, not all identically zero; forW0W\_\{0\}outside their common zero set it is nonzero on an open dense set of\(B,A\)\(B,A\)\([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\), which meets the open dense set whereB,AB,Ahave full rank andW0\+s​B​A∈𝒰W\_\{0\}\+sBA\\in\\mathcal\{U\}\. At such a point withℓ=𝟏\\ell=\\mathbf\{1\}the rank isr⁡\(m\+n−r\)\+min⁡\(m,\(m−r\)​\(n−r\)\)r\(m\+n\-r\)\+\\min\(m,\(m\-r\)\(n\-r\)\)\. With the upper bound,d=min⁡\(m​n,r⁡\(m\+n−r\)\+m\)d=\\min\(mn,r\(m\+n\-r\)\+m\)forW0W\_\{0\}outside a proper algebraic subset, andd=r⁡\(m\+n−r\)\+md=r\(m\+n\-r\)\+miff\(m−r\)​\(n−r\)≥m\(m\-r\)\(n\-r\)\\geq m\.

*Rank\-oneW0W\_\{0\}\.*LetW0=a​c⊤W\_\{0\}=ac^\{\\top\}with allai≠0a\_\{i\}\\neq 0andr<nr<n\. At a point withℓ\\ellinvertible andX∈ℳrX\\in\\mathcal\{M\}\_\{r\},PU⟂​diag⁡\(δ\)​a​c⊤​PR⟂=\(PU⟂​\(δ⊙a\)\)​\(PR⟂​c\)⊤P\_\{U^\{\\perp\}\}\\diag\(\\delta\)ac^\{\\top\}P\_\{R^\{\\perp\}\}=\(P\_\{U^\{\\perp\}\}\(\\delta\\odot a\)\)\(P\_\{R^\{\\perp\}\}c\)^\{\\top\}\. Ifc∉Rc\\notin R, this vanishes iffδ⊙a∈U\\delta\\odot a\\in U, soK=diag⁡\(a\)−1​UK=\\diag\(a\)^\{\-1\}Uhas dimensionrrand the rank isr⁡\(m\+n−r\)\+m−rr\(m\+n\-r\)\+m\-r\. Ifc∈Rc\\in R, thenK=ℝmK=\\mathbb\{R\}^\{m\}and the rank isr⁡\(m\+n−r\)r\(m\+n\-r\)\. The points withℓ\\ellinvertible,X∈ℳrX\\in\\mathcal\{M\}\_\{r\}andc∉Rc\\notin Rform an open dense subset of the domain \(sincer<nr<n\), and the points of maximal rank form an open nonempty subset, so the two meet andd=r⁡\(m\+n−r\)\+m−rd=r\(m\+n\-r\)\+m\-r\. This is at mostm​nmn, since\(m−r\)​\(n−r−1\)≥0\(m\-r\)\(n\-r\-1\)\\geq 0\. It is strictly below the generic valuer⁡\(m\+n−r\)\+min⁡\(m,\(m−r\)​\(n−r\)\)r\(m\+n\-r\)\+\\min\(m,\(m\-r\)\(n\-r\)\)iffm−r<\(m−r\)​\(n−r\)m\-r<\(m\-r\)\(n\-r\), that is iffr<mr<mandr≤n−2r\\leq n\-2\.

*Covariance\.*Letg=Pπ​Dg=P\_\{\\pi\}Dbe monomial, withPπP\_\{\\pi\}a permutation matrix andD∈TmD\\in T\_\{m\}, and leth∈GLnh\\in\\mathrm\{GL\}\_\{n\}\. Theng​W0​h−1gW\_\{0\}h^\{\-1\}has no zero row, andg​diag⁡\(x\)​g−1=diag⁡\(Pπ​x\)g\\,\\diag\(x\)\\,g^\{\-1\}=\\diag\(P\_\{\\pi\}x\), sog​Diagm​g−1=Diagmg\\,\\mathrm\{Diag\}\_\{m\}\\,g^\{\-1\}=\\mathrm\{Diag\}\_\{m\}; alsog​ℳ≤r​h−1=ℳ≤rg\\mathcal\{M\}\_\{\\leq r\}h^\{\-1\}=\\mathcal\{M\}\_\{\\leq r\}\. Hence

g⁡\(Diagm⋅\(W0\+ℳ≤r\)\)​h−1=Diagm⋅\(g​W0​h−1\+ℳ≤r\)=Im⁡DoRAr​\(g​W0​h−1\)\.g\\,\\big\(\\mathrm\{Diag\}\_\{m\}\\cdot\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\)\\big\)\\,h^\{\-1\}=\\mathrm\{Diag\}\_\{m\}\\cdot\(gW\_\{0\}h^\{\-1\}\+\\mathcal\{M\}\_\{\\leq r\}\)=\\Imop\\mathrm\{DoRA\}\_\{r\}\(gW\_\{0\}h^\{\-1\}\)\.Monm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}is a subgroup with this property, so it lies in the covariance group \(proof of[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\)\.

*The update atB=0B=0\.*HF initialisesμi=∥w0,i∥\\mu\_\{i\}=\\lVert w\_\{0,i\}\\rVertand then trainsμ\\mu; atB=0B=0the weight isdiag⁡\(μi/∥w0,i∥\)​W0\\diag\(\\mu\_\{i\}/\\lVert w\_\{0,i\}\\rVert\)W\_\{0\}, soΔ​W=\(diag⁡\(μi/∥w0,i∥\)−I\)​W0\\Delta W=\(\\diag\(\\mu\_\{i\}/\\lVert w\_\{0,i\}\\rVert\)\-I\)W\_\{0\}\. Withμi=2​∥w0,i∥\\mu\_\{i\}=2\\lVert w\_\{0,i\}\\rVertfor everyii,Δ​W=W0\\Delta W=W\_\{0\}, of rankrank⁡W0\\rk W\_\{0\}, which exceeds2​r2rwheneverrank⁡W0\>2​r\\rk W\_\{0\}\>2r, beyond the bound of[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\)\. ∎

### C\.16 Proposition II\.9

Proposition II\.9\(the torus ladder\)\.\(a\)HiRA\.ρ⁡\(B,A\)=W0⊙\(J\+B​A\)\\rho\(B,A\)=W\_\{0\}\\odot\(J\+BA\), whereJJis the all\-ones matrix; HF PEFT hasB0=0B\_\{0\}=0and no scale factor on Linear and Conv layers \(on embeddings it usesA0=0A\_\{0\}=0\)\. For everyW0W\_\{0\},Im⁡HiRAr=W0\+DW0​\(ℳ≤r\)\\Imop\\mathrm\{HiRA\}\_\{r\}=W\_\{0\}\+D\_\{W\_\{0\}\}\(\\mathcal\{M\}\_\{\\leq r\}\)withDW0:X↦W0⊙XD\_\{W\_\{0\}\}:X\\mapsto W\_\{0\}\\odot X\. IfW0W\_\{0\}has no zero entries,DW0∈GL⁡\(Θ\)D\_\{W\_\{0\}\}\\in\\mathrm\{GL\}\(\\Theta\), so inside the slice atθ0\\theta\_\{0\}HiRA is LoRA transported by a diagonal automorphism: it has the samed=r⁡\(m\+n−r\)d=r\(m\+n\-r\)\(forr≤min⁡\(m,n\)r\\leq\\min\(m,n\)\) and the same gauge\. In all casesrank⁡Δ​W≤r⋅rank⁡W0\\rk\\Delta W\\leq r\\cdot\\rk W\_\{0\}\. The transport depends onθ0\\theta\_\{0\}\([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.\(b\)Scalings\.Output\-side scalings,diag⁡\(ℓ\)​W0=W0⊙\(J\+\(ℓ−𝟏\)​𝟏⊤\)\\diag\(\\ell\)W\_\{0\}=W\_\{0\}\\odot\(J\+\(\\ell\-\\mathbf\{1\}\)\\mathbf\{1\}^\{\\top\}\), and input\-side scalings,W0​diag⁡\(ℓ\)=W0⊙\(J\+𝟏​\(ℓ−𝟏\)⊤\)W\_\{0\}\\diag\(\\ell\)=W\_\{0\}\\odot\(J\+\\mathbf\{1\}\(\\ell\-\\mathbf\{1\}\)^\{\\top\}\), have images inIm⁡HiRA1\\Imop\\mathrm\{HiRA\}\_\{1\}\. Output\-side examples: \(IA\)3on keys and values, the post\-linear scale of SSF, DoRA’s magnitude\. Input\-side examples: \(IA\)3’s feed\-forward vector, which scales the input ofWdownW\_\{\\mathrm\{down\}\}, and a LayerNorm gainγ\\gammafolded into the next linear map\. The fold is exact when the norm’s output is consumed only by linear maps: the input norms of pre\-norm blocks, and a final norm before an untied head \(in post\-norm BERT, the last block’s norm feeds only the head\)\. It does not apply to Gemma\-style post\-sublayer norms, which HF’s default LN\-tuning targets for Gemma 2 and 3, to QK\-norms, or to post\-norm residual paths\. Where it applies,γ\\gammais tied across all consumers of the norm \(Q, K and V, or gate and up\) rather than being a free point per box, and with tied embeddings a fold into the head breaks the tie\. LayerNorm biases and SSF’s shift fold into the next bias, not intoWW\. Two\-sided scalingsdiag⁡\(a\)​W0​diag⁡\(b\)=W0⊙\(a​b⊤\)\\diag\(a\)W\_\{0\}\\diag\(b\)=W\_\{0\}\\odot\(ab^\{\\top\}\)lie inIm⁡HiRA2\\Imop\\mathrm\{HiRA\}\_\{2\}\. As a morphism,h⁡\(ℓ\)=\(ℓ−𝟏,𝟏⊤\)h\(\\ell\)=\(\\ell\-\\mathbf\{1\},\\mathbf\{1\}^\{\\top\}\)lands inHiRA1\\mathrm\{HiRA\}\_\{1\}pointed at\(0,𝟏⊤\)\(0,\\mathbf\{1\}^\{\\top\}\), which is not HiRA’s random\-AAdefault pointing\.\(c\)Orbit dimension\.The two\-sided torus orbit\{diag⁡\(a\)​W0​diag⁡\(b\)\}\\\{\\diag\(a\)W\_\{0\}\\diag\(b\)\\\}has dimensionm\+n−c⁡\(W0\)m\+n\-c\(W\_\{0\}\), wherec⁡\(W0\)c\(W\_\{0\}\)is the number of connected components of the bipartite support graph ofW0W\_\{0\}, an isolated vertex counting as its own component\.\(d\)RoAd\.RoAd1\(the paper’s main variant, HF road\_1\) isW=R​W0W=RW\_\{0\}withRRblock\-diagonal in scaled rotationsαi​Rot​\(θi\)\\alpha\_\{i\}\\mathrm\{Rot\}\(\\theta\_\{i\}\)of output pairs\. It is the left action ofCO\+​\(2\)m/2≅\(ℂ×\)m/2\\mathrm\{CO\}^\{\+\}\(2\)^\{m/2\}\\cong\(\\mathbb\{C\}^\{\\times\}\)^\{m/2\}: it is\(IA\)3overℂ\\mathbb\{C\}\. It multiplies the input Gram matrix of each output pair byαi2\\alpha\_\{i\}^\{2\}\. Sinceαi\\alpha\_\{i\}may be00, its image is the orbit closure, a real\-linear space of dimension exactlymmwhenever no output\-pair block ofW0W\_\{0\}is zero\. RoAd2and RoAd4\(HF road\_2, road\_4\) let each block range over all ofM2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\)\. They realise the block\-diagonal action of the monoidM2​\(ℝ\)m/2M\_\{2\}\(\\mathbb\{R\}\)^\{m/2\}\(its group partGL2m/2\\mathrm\{GL\}\_\{2\}^\{m/2\}is non\-abelian\), with linear image\{blockdiag⁡\(Ri\)​W0\}\\\{\\mathrm\{blockdiag\}\(R\_\{i\}\)W\_\{0\}\\\}of dimension2​m2mwhen every pair block has rank22\. For them the Gram statement and “overℂ\\mathbb\{C\}” fail, and RoAd4carries a positive\-dimensional gauge \(two dimensions per row\)\. HF pairs outputjjwithj\+g/2j\+g/2inside each group of sizegg\(default6464\) rather than adjacent outputs as in the paper, and also rotates the bias; the statements above do not depend on the pairing\.\(e\)Zeros and covariance\.EveryWWreachable by HiRA, \(IA\)3or SSF vanishes whereverW0W\_\{0\}does,Z⁡\(W0\)⊆Z⁡\(W\)Z\(W\_\{0\}\)\\subseteq Z\(W\), and this persists under merging\. New zeros can appear \(ℓi=0\\ell\_\{i\}=0, or\(J\+B​A\)i​j=0\(J\+BA\)\_\{ij\}=0\), so the zero pattern itself is not invariant\. The image covariance group of output\-scaled boxes \(\(IA\)3onk,vk,v; post\-linear SSF\) isMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, and that of input\-scaled boxes \(\(IA\)3onWdownW\_\{\\mathrm\{down\}\}\) isGLm×Monn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}\. HiRA’s is exactlyMonm×Monn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}\(=GL1×GL1=\\mathrm\{GL\}\_\{1\}\\times\\mathrm\{GL\}\_\{1\}whenm=n=1m=n=1\)\.

*\(a\)*W0⊙\(J\+B​A\)=W0\+W0⊙\(B​A\)W\_\{0\}\\odot\(J\+BA\)=W\_\{0\}\+W\_\{0\}\\odot\(BA\), and\{B​A\}=ℳ≤r\\\{BA\\\}=\\mathcal\{M\}\_\{\\leq r\}, soIm⁡HiRAr=W0\+DW0​\(ℳ≤r\)\\Imop\\mathrm\{HiRA\}\_\{r\}=W\_\{0\}\+D\_\{W\_\{0\}\}\(\\mathcal\{M\}\_\{\\leq r\}\)\. In the standard basisEi​jE\_\{ij\}ofΘ=ℝm×n\\Theta=\\mathbb\{R\}^\{m\\times n\},DW0D\_\{W\_\{0\}\}is diagonal with entries\(W0\)i​j\(W\_\{0\}\)\_\{ij\}, so it is invertible iff no entry ofW0W\_\{0\}vanishes\. Then, withτ⁡\(θ\)=W0\+DW0​\(θ−W0\)\\tau\(\\theta\)=W\_\{0\}\+D\_\{W\_\{0\}\}\(\\theta\-W\_\{0\}\), HiRA isτ∘ρLoRA\\tau\\circ\\rho\_\{\\mathrm\{LoRA\}\}for LoRA with residualW0W\_\{0\}and scale11\. Sinceτ\\tauis an affine bijection,D⁡\(τ∘ρ\)=DW0∘D​ρD\(\\tau\\circ\\rho\)=D\_\{W\_\{0\}\}\\circ D\\rhohas the same rank asD​ρD\\rhoat every point, soddis unchanged \([Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(d\)\), andτ∘ρ∘φ=τ∘ρ\\tau\\circ\\rho\\circ\\varphi=\\tau\\circ\\rhoiffρ∘φ=ρ\\rho\\circ\\varphi=\\rho, so the gauge group is unchanged; this is the transport\(DW0\)∗\(D\_\{W\_\{0\}\}\)\_\{\*\}of[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\. Forr≤min⁡\(m,n\)r\\leq\\min\(m,n\),d⁡\(LoRAr\)=r⁡\(m\+n−r\)d\(\\mathrm\{LoRA\}\_\{r\}\)=r\(m\+n\-r\)\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. For the rank, with columnsbkb\_\{k\}ofBBand rowsaka\_\{k\}ofAA,W0⊙\(B​A\)=∑k≤rdiag⁡\(bk\)​W0​diag⁡\(ak\)W\_\{0\}\\odot\(BA\)=\\sum\_\{k\\leq r\}\\diag\(b\_\{k\}\)\\,W\_\{0\}\\,\\diag\(a\_\{k\}\)\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\), a sum ofrrmatrices of rank at mostrank⁡W0\\rk W\_\{0\}\. The mapDW0D\_\{W\_\{0\}\}is built fromW0=θ0W\_\{0\}=\\theta\_\{0\}, which is the dependence on the base point\.

*\(b\)*Entrywise,\(diag⁡\(ℓ\)​W0\)i​j=ℓi​\(W0\)i​j=\(W0\)i​j​\(1\+\(ℓi−1\)\)\(\\diag\(\\ell\)W\_\{0\}\)\_\{ij\}=\\ell\_\{i\}\(W\_\{0\}\)\_\{ij\}=\(W\_\{0\}\)\_\{ij\}\\big\(1\+\(\\ell\_\{i\}\-1\)\\big\), sodiag⁡\(ℓ\)​W0=W0⊙\(J\+\(ℓ−𝟏\)​𝟏⊤\)\\diag\(\\ell\)W\_\{0\}=W\_\{0\}\\odot\(J\+\(\\ell\-\\mathbf\{1\}\)\\mathbf\{1\}^\{\\top\}\), and\(ℓ−𝟏\)​𝟏⊤=B​A\(\\ell\-\\mathbf\{1\}\)\\mathbf\{1\}^\{\\top\}=BAwithB=ℓ−𝟏B=\\ell\-\\mathbf\{1\}andA=𝟏⊤A=\\mathbf\{1\}^\{\\top\}, a point ofHiRA1\\mathrm\{HiRA\}\_\{1\}\. The input side is the transpose computation\. For two\-sided scalings,diag⁡\(a\)​W0​diag⁡\(b\)=W0⊙\(a​b⊤\)\\diag\(a\)W\_\{0\}\\diag\(b\)=W\_\{0\}\\odot\(ab^\{\\top\}\)anda​b⊤−J=\[a,−𝟏\]​\[b,1\]⊤ab^\{\\top\}\-J=\[a,\\,\-\\mathbf\{1\}\]\\,\[b,\\,\\mathbf\{1\}\]^\{\\top\}has rank at most22\. The maph⁡\(ℓ\)=\(ℓ−𝟏,𝟏⊤\)h\(\\ell\)=\(\\ell\-\\mathbf\{1\},\\mathbf\{1\}^\{\\top\}\)is smooth, sends𝟏\\mathbf\{1\}to\(0,𝟏⊤\)\(0,\\mathbf\{1\}^\{\\top\}\)and satisfiesρHiRA1∘h⁡\(ℓ\)=diag⁡\(ℓ\)​W0\\rho\_\{\\mathrm\{HiRA\}\_\{1\}\}\\circ h\(\\ell\)=\\diag\(\\ell\)W\_\{0\}, so it is a pointed morphism intoHiRA1\\mathrm\{HiRA\}\_\{1\}pointed there\. For a norm whose outputγ⊙x^\\gamma\\odot\\hat\{x\}\(withx^\\hat\{x\}the normalised input\) feeds only linear mapsWW,W⁡\(γ⊙x^\)=\(W​diag⁡\(γ\)\)​x^W\(\\gamma\\odot\\hat\{x\}\)=\(W\\diag\(\\gamma\)\)\\hat\{x\}, so the gain is an input\-side scaling of each consumer, the sameγ\\gammafor all of them; a biasβ\\betagivesW​βW\\beta, a change of the next bias\. When the norm’s output is added to the residual stream \(post\-sublayer norms, post\-norm residual paths\) or is used inside attention scores \(QK\-norms\), no linear map of the adapted box absorbsγ\\gamma\. With tied embeddings the head isE⊤E^\{\\top\}, and replacing it byE⊤​diag⁡\(γ\)E^\{\\top\}\\diag\(\\gamma\)unties it\.

*\(c\)*The orbit mapΨ⁡\(a,b\)=diag⁡\(a\)​W0​diag⁡\(b\)\\Psi\(a,b\)=\\diag\(a\)W\_\{0\}\\diag\(b\)ofTm×TnT\_\{m\}\\times T\_\{n\}has differential at\(𝟏,𝟏\)\(\\mathbf\{1\},\\mathbf\{1\}\)

\(α,β\)↦diag⁡\(α\)​W0\+W0​diag⁡\(β\)=\(\(αi\+βj\)​\(W0\)i​j\)i​j\.\(\\alpha,\\beta\)\\mapsto\\diag\(\\alpha\)W\_\{0\}\+W\_\{0\}\\diag\(\\beta\)=\\big\(\(\\alpha\_\{i\}\+\\beta\_\{j\}\)\(W\_\{0\}\)\_\{ij\}\\big\)\_\{ij\}\.Its kernel is cut out byαi\+βj=0\\alpha\_\{i\}\+\\beta\_\{j\}=0along every edge\(i,j\)\(i,j\)of the bipartite support graph\. On a connected component the value at one vertex determines all the others \(withβ=−α\\beta=\-\\alphaacross each edge\), and an isolated vertex carries one free coordinate; so the kernel has dimensionc⁡\(W0\)c\(W\_\{0\}\)and the rank ism\+n−c⁡\(W0\)m\+n\-c\(W\_\{0\}\)\. BecauseΨ⁡\(g​g′\)=g⋅Ψ⁡\(g′\)\\Psi\(gg^\{\\prime\}\)=g\\cdot\\Psi\(g^\{\\prime\}\)for the commutative groupTm×TnT\_\{m\}\\times T\_\{n\}, the differential has this rank at every point of the group, so the orbit, an immersed submanifold of dimensiondim\(Tm×Tn\)\\dim\(T\_\{m\}\\times T\_\{n\}\)minus that of the stabiliser, has dimensionm\+n−c⁡\(W0\)m\+n\-c\(W\_\{0\}\)\.

*\(d\)*Identify an output pair, a2×n2\\times nblockXXwith rowsx1,x2x\_\{1\},x\_\{2\}, with the complex rowz=x1\+i​x2∈ℂnz=x\_\{1\}\+\\mathrm\{i\}x\_\{2\}\\in\\mathbb\{C\}^\{n\}\. ForRot⁡\(θ\)=\(cos⁡θ−sin⁡θsin⁡θcos⁡θ\)\\mathrm\{Rot\}\(\\theta\)=\\left\(\\begin\{smallmatrix\}\\cos\\theta&\-\\sin\\theta\\\\ \\sin\\theta&\\cos\\theta\\end\{smallmatrix\}\\right\)the rows ofα​Rot​\(θ\)​X\\alpha\\mathrm\{Rot\}\(\\theta\)Xareα⁡\(cos⁡θ​x1−sin⁡θ​x2\)\\alpha\(\\cos\\theta\\,x\_\{1\}\-\\sin\\theta\\,x\_\{2\}\)andα⁡\(sin⁡θ​x1\+cos⁡θ​x2\)\\alpha\(\\sin\\theta\\,x\_\{1\}\+\\cos\\theta\\,x\_\{2\}\), which correspond toα​ei​θ​z\\alpha e^\{\\mathrm\{i\}\\theta\}z\. So RoAd1multiplies each pair by a complex scalar, the left action ofCO\+​\(2\)m/2≅\(ℂ×\)m/2\\mathrm\{CO\}^\{\+\}\(2\)^\{m/2\}\\cong\(\\mathbb\{C\}^\{\\times\}\)^\{m/2\}, the complex analogue of \(IA\)3’s real output scaling\. The input Gram matrix of the pair becomes\(α​Rot​X\)⊤​\(α​Rot​X\)=α2​X⊤​X\(\\alpha\\mathrm\{Rot\}X\)^\{\\top\}\(\\alpha\\mathrm\{Rot\}X\)=\\alpha^\{2\}X^\{\\top\}X\. As\(α,θ\)\(\\alpha,\\theta\)ranges overℝ×ℝ\\mathbb\{R\}\\times\\mathbb\{R\},α​ei​θ\\alpha e^\{\\mathrm\{i\}\\theta\}ranges over all ofℂ\\mathbb\{C\}, so the image is∏iℂ⋅zi\\prod\_\{i\}\\mathbb\{C\}\\cdot z\_\{i\}, the closure of the orbit, a real\-linear space of dimension2​\#​\{i:zi≠0\}2\\,\\\#\\\{i:z\_\{i\}\\neq 0\\\}, equal tommwhen no pair block vanishes\. In RoAd2each row of a block has its own polar pair, so each row of the2×22\\times 2block is an arbitrary vector ofℝ2\\mathbb\{R\}^\{2\}and the block is an arbitrary element ofM2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\); RoAd4also reaches every element ofM2​\(ℝ\)M\_\{2\}\(\\mathbb\{R\}\), with eight parameters for four entries, so its fibres have dimension44per block, two per row\. The image\{Ri​Xi\}\\\{R\_\{i\}X\_\{i\}\\\}of one block is the set of2×n2\\times nmatrices whose rows lie inrow⁡Xi\\operatorname\{row\}X\_\{i\}, of dimension2​rank⁡Xi2\\rk X\_\{i\}, so the total is2​m2mwhen every block has rank22\. ForR=diag⁡\(1,2\)R=\\diag\(1,2\)andX=I2X=I\_\{2\},\(R​X\)⊤​\(R​X\)=diag⁡\(1,4\)\(RX\)^\{\\top\}\(RX\)=\\diag\(1,4\)is not a multiple ofX⊤​XX^\{\\top\}X, andRRdoes not commute withRot⁡\(π/2\)\\mathrm\{Rot\}\(\\pi/2\), so it is not multiplication by a complex number\. The pairing of outputs and the rotated bias do not enter these computations\.

*\(e\) Zeros\.*Wi​j=\(W0\)i​j​\(1\+\(B​A\)i​j\)W\_\{ij\}=\(W\_\{0\}\)\_\{ij\}\(1\+\(BA\)\_\{ij\}\)for HiRA,ℓi​\(W0\)i​j\\ell\_\{i\}\(W\_\{0\}\)\_\{ij\}or\(W0\)i​j​ℓj\(W\_\{0\}\)\_\{ij\}\\ell\_\{j\}for \(IA\)3, andγi​\(W0\)i​j\\gamma\_\{i\}\(W\_\{0\}\)\_\{ij\}for SSF’s post\-linear scale \(its shift changes the bias\), so\(W0\)i​j=0\(W\_\{0\}\)\_\{ij\}=0impliesWi​j=0W\_\{ij\}=0\. After a merge the new baseW1W\_\{1\}hasZ⁡\(W0\)⊆Z⁡\(W1\)Z\(W\_\{0\}\)\\subseteq Z\(W\_\{1\}\), and induction gives the claim for any number of cycles\. New zeros:ℓi=0\\ell\_\{i\}=0kills rowii, andB=−eiB=\-e\_\{i\},A=ej⊤A=e\_\{j\}^\{\\top\}give\(J\+B​A\)i​j=0\(J\+BA\)\_\{ij\}=0\.

*\(e\) Output and input scalings\.*HereIm⁡M⁡\(W0\)=Diagm⋅W0\\Imop M\(W\_\{0\}\)=\\mathrm\{Diag\}\_\{m\}\\cdot W\_\{0\}\. Forggmonomial andh∈GLnh\\in\\mathrm\{GL\}\_\{n\},g​diag⁡\(ℓ\)​W0​h−1=diag⁡\(ℓ′\)​g​W0​h−1g\\,\\diag\(\\ell\)\\,W\_\{0\}h^\{\-1\}=\\diag\(\\ell^\{\\prime\}\)\\,gW\_\{0\}h^\{\-1\}withdiag⁡\(ℓ′\)=g​diag⁡\(ℓ\)​g−1\\diag\(\\ell^\{\\prime\}\)=g\\diag\(\\ell\)g^\{\-1\}, andℓ′\\ell^\{\\prime\}ranges overℝm\\mathbb\{R\}^\{m\}withℓ\\ell, soMonm×GLn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}is contained in the covariance group\. Conversely, let\(g,h\)\(g,h\)be in it and takeW0=ej​c⊤W\_\{0\}=e\_\{j\}c^\{\\top\}withc≠0c\\neq 0\. ThenIm⁡M⁡\(ej​c⊤\)=ℝ​ej​c⊤\\Imop M\(e\_\{j\}c^\{\\top\}\)=\\mathbb\{R\}\\,e\_\{j\}c^\{\\top\}, andg​Im⁡M⁡\(W0\)​h−1=ℝ⁡\(g​ej\)​\(h−⁣⊤​c\)⊤g\\,\\Imop M\(W\_\{0\}\)h^\{\-1\}=\\mathbb\{R\}\\,\(ge\_\{j\}\)\(h^\{\-\\top\}c\)^\{\\top\}is a line, whileIm⁡M⁡\(g​W0​h−1\)=\{y​\(h−⁣⊤​c\)⊤:supp⁡y⊆supp⁡\(g​ej\)\}\\Imop M\(gW\_\{0\}h^\{\-1\}\)=\\\{y\(h^\{\-\\top\}c\)^\{\\top\}:\\operatorname\{supp\}y\\subseteq\\operatorname\{supp\}\(ge\_\{j\}\)\\\}has dimension\|supp⁡\(g​ej\)\|\\lvert\\operatorname\{supp\}\(ge\_\{j\}\)\\rvert\. Equality for everyjjforces every column ofggto have one nonzero entry, soggis monomial\. The input side is the transpose\.

*\(e\) HiRA\.*For permutation matricesP,P′P,P^\{\\prime\}and diagonalD,D′D,D^\{\\prime\}we haveP⁡\(X⊙Y\)​P′=\(P​X​P′\)⊙\(P​Y​P′\)P\(X\\odot Y\)P^\{\\prime\}=\(PXP^\{\\prime\}\)\\odot\(PYP^\{\\prime\}\)andD⁡\(X⊙Y\)​D′=\(D​X​D′\)⊙YD\(X\\odot Y\)D^\{\\prime\}=\(DXD^\{\\prime\}\)\\odot Y\. Forg=P​Dg=PDandh−1=D′​P′h^\{\-1\}=D^\{\\prime\}P^\{\\prime\}this gives

g⁡\(W0⊙\(J\+Z\)\)​h−1=\(g​W0​h−1\)⊙\(J\+P​Z​P′\),g\\big\(W\_\{0\}\\odot\(J\+Z\)\\big\)h^\{\-1\}=\(gW\_\{0\}h^\{\-1\}\)\\odot\(J\+PZP^\{\\prime\}\),andP​ℳ≤r​P′=ℳ≤rP\\mathcal\{M\}\_\{\\leq r\}P^\{\\prime\}=\\mathcal\{M\}\_\{\\leq r\}, soMonm×Monn\\mathrm\{Mon\}\_\{m\}\\times\\mathrm\{Mon\}\_\{n\}is contained in the covariance group\. Conversely, let\(g,h\)\(g,h\)be in it and takeW0=Ei​jW\_\{0\}=E\_\{ij\}, whose imageEi​j⊙\(J\+ℳ≤r\)=ℝ​Ei​jE\_\{ij\}\\odot\(J\+\\mathcal\{M\}\_\{\\leq r\}\)=\\mathbb\{R\}E\_\{ij\}is a line\. Withu=g​eiu=ge\_\{i\}andv=h−⁣⊤​ejv=h^\{\-\\top\}e\_\{j\}, covariance requiresIm⁡M⁡\(u​v⊤\)=ℝ​u​v⊤\\Imop M\(uv^\{\\top\}\)=\\mathbb\{R\}\\,uv^\{\\top\}\. Supposeu​v⊤uv^\{\\top\}has two nonzero entries\(k,l\)≠\(k′,l′\)\(k,l\)\\neq\(k^\{\\prime\},l^\{\\prime\}\)\. TakingZ=Ek​l∈ℳ≤1Z=E\_\{kl\}\\in\\mathcal\{M\}\_\{\\leq 1\}givesY=u​v⊤\+uk​vl​Ek​l∈Im⁡M⁡\(u​v⊤\)Y=uv^\{\\top\}\+u\_\{k\}v\_\{l\}E\_\{kl\}\\in\\Imop M\(uv^\{\\top\}\)\. IfY=λ​u​v⊤Y=\\lambda uv^\{\\top\}, the entry\(k′,l′\)\(k^\{\\prime\},l^\{\\prime\}\)forcesλ=1\\lambda=1and the entry\(k,l\)\(k,l\)forcesuk​vl=0u\_\{k\}v\_\{l\}=0, a contradiction\. So\|supp⁡u\|=\|supp⁡v\|=1\\lvert\\operatorname\{supp\}u\\rvert=\\lvert\\operatorname\{supp\}v\\rvert=1for alli,ji,j, which makesggandh−⁣⊤h^\{\-\\top\}, hencehh, monomial\. Form=n=1m=n=1,Mon1=GL1\\mathrm\{Mon\}\_\{1\}=\\mathrm\{GL\}\_\{1\}\. ∎

### C\.17 Theorem II\.10

Theorem II\.10\(cut\-rank bound; standard\)\.LetΓ\\Gammabe a tensor network, evaluated in the symmetric monoidal category\(𝐅𝐢𝐧𝐕𝐞𝐜𝐭,⊗\)\(\\mathbf\{FinVect\},\\otimes\), with input legsVVand output legsUU\. Every wire and open leg is assumed incident to a vertex: insert an identity box on any bare strand, or equivalently count each bare input–output wire as crossing every cut\. A crossing hyperedge \(copy spider\) contributes its dimension once\. Partition the vertices intoℐ⊔𝒪\\mathcal\{I\}\\sqcup\\mathcal\{O\}\. The weightwwof the cut is the product of the dimensions of the internal wires between the two sides and of the open legs attached on the wrong side \(input legs at𝒪\\mathcal\{O\}, output legs atℐ\\mathcal\{I\}\)\. Thenrank⁡⟦Γ⟧≤mincuts⁡w,\\rk\\llbracket\\Gamma\\rrbracket\\leq\\min\_\{\\mathrm\{cuts\}\}w,and rank is subadditive over formal sums\. The bound refers to a chosen presentation of the network\.

Fix a cutℐ⊔𝒪\\mathcal\{I\}\\sqcup\\mathcal\{O\}\. WriteVℐ,V𝒪V\_\{\\mathcal\{I\}\},V\_\{\\mathcal\{O\}\}for the tensor products of the input legs attached toℐ\\mathcal\{I\}and to𝒪\\mathcal\{O\},Uℐ,U𝒪U\_\{\\mathcal\{I\}\},U\_\{\\mathcal\{O\}\}likewise for the output legs, andCCfor the tensor product of the spaces carried by the crossing wires, each crossing hyperedge counted once\. Contracting every wire internal toℐ\\mathcal\{I\}produces a tensor whose free indices are those ofVℐV\_\{\\mathcal\{I\}\},UℐU\_\{\\mathcal\{I\}\}andCC; read it as a linear mapTℐ:Vℐ→Uℐ⊗CT\_\{\\mathcal\{I\}\}:V\_\{\\mathcal\{I\}\}\\to U\_\{\\mathcal\{I\}\}\\otimes C\. Likewise the𝒪\\mathcal\{O\}side givesT𝒪:C⊗V𝒪→U𝒪T\_\{\\mathcal\{O\}\}:C\\otimes V\_\{\\mathcal\{O\}\}\\to U\_\{\\mathcal\{O\}\}\. A crossing hyperedge is a single summation index shared by tensors on both sides, so it is summed once, over a basis of its space, when the two sides are joined\. Contracting the crossing indices joins the two halves, and up to the symmetry isomorphisms that reorder tensor factors,

⟦Γ⟧=\(idUℐ⊗T𝒪\)∘\(Tℐ⊗idV𝒪\):Vℐ⊗V𝒪→Uℐ⊗C⊗V𝒪→Uℐ⊗U𝒪\.\\llbracket\\Gamma\\rrbracket=\(\\mathrm\{id\}\_\{U\_\{\\mathcal\{I\}\}\}\\otimes T\_\{\\mathcal\{O\}\}\)\\circ\(T\_\{\\mathcal\{I\}\}\\otimes\\mathrm\{id\}\_\{V\_\{\\mathcal\{O\}\}\}\):V\_\{\\mathcal\{I\}\}\\otimes V\_\{\\mathcal\{O\}\}\\to U\_\{\\mathcal\{I\}\}\\otimes C\\otimes V\_\{\\mathcal\{O\}\}\\to U\_\{\\mathcal\{I\}\}\\otimes U\_\{\\mathcal\{O\}\}\.A composite factors through its middle space, so its rank is at mostdimUℐ⋅dimC⋅dimV𝒪=w\\dim U\_\{\\mathcal\{I\}\}\\cdot\\dim C\\cdot\\dim V\_\{\\mathcal\{O\}\}=w\. Only the symmetric monoidal structure was used: no cups or caps\. A bare strand from an input to an output, carried by an identity box, has one of its two legs on the wrong side of every cut, which is the convention of the statement; without it,Id⊗B​AI\_\{d\}\\otimes BAwould violate the bound\. Minimising over cuts gives the bound, and subadditivity over formal sums isrank⁡\(X\+Y\)≤rank⁡X\+rank⁡Y\\rk\(X\+Y\)\\leq\\rk X\+\\rk Y\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. The weight depends on the drawing, which is why the bound refers to a presentation; compare[Auxiliary result B\.14](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\. ∎

*The bounds of Table[8](https://arxiv.org/html/2610.12728#S2.T8)\.*Each follows from a cut in the stated presentation or from subadditivity\. The LoRA familys​B​AsBAfactors through the bond of dimensionrr\. A split initialisation givesΔ​W=s​B​A−s​B0​A0\\Delta W=sBA\-sB\_\{0\}A\_\{0\}, a sum of two such terms \([Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\)\)\. For HRA, the telescoping identity

R−I=∑k=1r\(Hk⋯Hr−Hk\+1⋯Hr\)=∑k=1r\(Hk−I\)Hk\+1⋯HrR\-I=\\sum\_\{k=1\}^\{r\}\\big\(H\_\{k\}\\cdots H\_\{r\}\-H\_\{k\+1\}\\cdots H\_\{r\}\\big\)=\\sum\_\{k=1\}^\{r\}\(H\_\{k\}\-I\)H\_\{k\+1\}\\cdots H\_\{r\}writesR−IR\-Iasrrterms of rank at mostrank⁡\(Hk−I\)=1\\rk\(H\_\{k\}\-I\)=1, sorank⁡\(W0​R−W0\)≤r\\rk\(W\_\{0\}R\-W\_\{0\}\)\\leq r; a Givens rotation differs fromIIin a2×22\\times 2block, so the same identity gives2​g2gforgggates\. ReLoRA afterKKphases is a sum ofKKrank\-rrterms, and GraLoRA factors throughℝk​r\\mathbb\{R\}^\{kr\}\(proof of[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\)\. LoHa’s Hadamard product, drawn with copy spiders on the two legs, factors through the face\-splitting bond of dimensionr1​r2r\_\{1\}r\_\{2\}\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\)\. HiRA’s updateW0⊙B​A=∑k≤rdiag⁡\(bk\)​W0​diag⁡\(ak\)W\_\{0\}\\odot BA=\\sum\_\{k\\leq r\}\\diag\(b\_\{k\}\)W\_\{0\}\\diag\(a\_\{k\}\)factors throughrrcopies of a bond of dimensionrank⁡W0\\rk W\_\{0\}\. LoKr givesrank⁡\(C⊗B​A\)=rank⁡C⋅rank⁡\(B​A\)≤rank⁡C⋅r′\\rk\(C\\otimes BA\)=\\rk C\\cdot\\rk\(BA\)\\leq\\rk C\\cdot r^\{\\prime\}\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\)\)\. FourierFT’s update is a sum ofncn\_\{c\}real cosine patternscos⁡\(2​π​\(u​j/m\+v​l/n\)\)=Re⁡\(xu​yv⊤\)\\cos\\big\(2\\pi\(uj/m\+vl/n\)\\big\)=\\operatorname\{Re\}\(x\_\{u\}y\_\{v\}^\{\\top\}\)with\(xu\)j=e2​π​i​u​j/m\(x\_\{u\}\)\_\{j\}=e^\{2\\pi\\mathrm\{i\}uj/m\}and\(yv\)l=e2​π​i​v​l/n\(y\_\{v\}\)\_\{l\}=e^\{2\\pi\\mathrm\{i\}vl/n\}\([Auxiliary result B\.16](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\), andRe⁡\(x​y⊤\)=Re⁡x​Re​y⊤−Im⁡x​Im​y⊤\\operatorname\{Re\}\(xy^\{\\top\}\)=\\operatorname\{Re\}x\\operatorname\{Re\}y^\{\\top\}\-\\operatorname\{Im\}x\\operatorname\{Im\}y^\{\\top\}has rank at most22\. MoRA’s reshape, C3A’s circulant blocks and MPO adapters have no narrow cut in their presentation, so the theorem bounds them only bymin⁡\(m,n\)\\min\(m,n\)\.

### C\.18 Theorem II\.11

Theorem II\.11\(Kronecker incomparability; the linear algebra is standard, the PEFT reading is new\)\.Writeℝm=ℝm1⊗ℝm2\\mathbb\{R\}^\{m\}=\\mathbb\{R\}^\{m\_\{1\}\}\\otimes\\mathbb\{R\}^\{m\_\{2\}\}andℝn=ℝn1⊗ℝn2\\mathbb\{R\}^\{n\}=\\mathbb\{R\}^\{n\_\{1\}\}\\otimes\\mathbb\{R\}^\{n\_\{2\}\}, and viewΔ∈ℝm×n\\Delta\\in\\mathbb\{R\}^\{m\\times n\}as a 4\-leg tensor\. Letrk\\operatorname\{rk\}be the rank across the cut\{U1​U2\}\|\{V1​V2\}\\\{U\_\{1\}U\_\{2\}\\\}\\mid\\\{V\_\{1\}V\_\{2\}\\\}andrk⊗\\operatorname\{rk\}\_\{\\otimes\}the rank across\{U1​V1\}\|\{U2​V2\}\\\{U\_\{1\}V\_\{1\}\\\}\\mid\\\{U\_\{2\}V\_\{2\}\\\}\. Putκ1:=min⁡\(m1,m2\)⋅min⁡\(n1,n2\),ν:=min⁡\(m1,n1\)⋅min⁡\(m2,n2\)\.\\kappa\_\{1\}:=\\min\(m\_\{1\},m\_\{2\}\)\\cdot\\min\(n\_\{1\},n\_\{2\}\),\\qquad\\nu:=\\min\(m\_\{1\},n\_\{1\}\)\\cdot\\min\(m\_\{2\},n\_\{2\}\)\.\(a\)rk⊗⁡Δ≤k\\operatorname\{rk\}\_\{\\otimes\}\\Delta\\leq kiffΔ=∑i≤kCi⊗Di\\Delta=\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}\. EveryΔ\\Deltahasrk⊗⁡Δ≤min⁡\(m1​n1,m2​n2\)\\operatorname\{rk\}\_\{\\otimes\}\\Delta\\leq\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\)\.\(b\)rk⁡\(C⊗D\)=rk⁡C⋅rk⁡D\\operatorname\{rk\}\(C\\otimes D\)=\\operatorname\{rk\}C\\cdot\\operatorname\{rk\}D, whilerk⊗⁡\(C⊗D\)=1\\operatorname\{rk\}\_\{\\otimes\}\(C\\otimes D\)=1forC,D≠0C,D\\neq 0\. Soν\\nuis the largest rank of a single Kronecker product\.\(c\) Ifu∈ℝmu\\in\\mathbb\{R\}^\{m\}andv∈ℝnv\\in\\mathbb\{R\}^\{n\}have Schmidt ranksρu,ρv\\rho\_\{u\},\\rho\_\{v\}, thenrk⊗⁡\(u​v⊤\)=ρu​ρv\\operatorname\{rk\}\_\{\\otimes\}\(uv^\{\\top\}\)=\\rho\_\{u\}\\rho\_\{v\}\. A generic rank\-one update therefore has Kronecker rankκ1\\kappa\_\{1\}\. For aligned splits,\(m1−m2\)​\(n1−n2\)≥0\(m\_\{1\}\-m\_\{2\}\)\(n\_\{1\}\-n\_\{2\}\)\\geq 0, which include equal square splits and every split HF PEFT’s factorisation produces \(it puts the smaller factor first on both sides\),κ1=min⁡\(m1​n1,m2​n2\)\\kappa\_\{1\}=\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\)is the maximal Kronecker rank\.\(d\)*Incomparability\.*For1≤r<ν1\\leq r<\\nuand1≤k<κ11\\leq k<\\kappa\_\{1\}, the setsℳ≤r\\mathcal\{M\}\_\{\\leq r\}and𝒦≤k:=\{Δ:rk⊗⁡Δ≤k\}\\mathcal\{K\}\_\{\\leq k\}:=\\\{\\Delta:\\operatorname\{rk\}\_\{\\otimes\}\\Delta\\leq k\\\}of unconstrainedkk\-term Kronecker sums are incomparable, for every split\. A generic rank\-one update lies in the first and not the second, andC⊗DC\\otimes Dwith full\-rank factors, of Kronecker rank11and rankν\\nu, lies in the second and not the first\. For equal square splits \(m=n=Nm=n=N, all factorsss\),κ1=ν=s2=N\\kappa\_\{1\}=\\nu=s^\{2\}=N, andINI\_\{N\}is such a product\.\(e\)*Constrained Kronecker methods\.*LetIm⁡M⊆W0\+𝒦≤k\\Imop M\\subseteq W\_\{0\}\+\\mathcal\{K\}\_\{\\leq k\}\. ThenIm⁡LoRAr⊈Im⁡M\\Imop\\mathrm\{LoRA\}\_\{r\}\\not\\subseteq\\Imop Mwheneverk<κ1k<\\kappa\_\{1\}, for every pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}\. GiveLoRAr\\mathrm\{LoRA\}\_\{r\}its zero\-init pointing, with imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. ThenIm⁡M⊈Im⁡LoRAr\\Imop M\\not\\subseteq\\Imop\\mathrm\{LoRA\}\_\{r\}iff someθ∈Im⁡M\\theta\\in\\Imop Mhasrank⁡\(θ−W0\)\>r\\rk\(\\theta\-W\_\{0\}\)\>r\. For LoKr,Δ​W=C⊗B​A\\Delta W=C\\otimes BAwithC∈ℝm1×n1C\\in\\mathbb\{R\}^\{m\_\{1\}\\times n\_\{1\}\},B∈ℝm2×r′B\\in\\mathbb\{R\}^\{m\_\{2\}\\times r^\{\\prime\}\}andA∈ℝr′×n2A\\in\\mathbb\{R\}^\{r^\{\\prime\}\\times n\_\{2\}\}, putreff′=min⁡\(r′,m2,n2\)r^\{\\prime\}\_\{\\mathrm\{eff\}\}=\\min\(r^\{\\prime\},m\_\{2\},n\_\{2\}\)\. The maximum rank ismin⁡\(m1,n1\)⋅reff′\\min\(m\_\{1\},n\_\{1\}\)\\cdot r^\{\\prime\}\_\{\\mathrm\{eff\}\}, attained\. SoIm⁡LoKrr′⊆Im⁡LoRAr\\Imop\\mathrm\{LoKr\}\_\{r^\{\\prime\}\}\\subseteq\\Imop\\mathrm\{LoRA\}\_\{r\}iffr≥min⁡\(m1,n1\)​reff′r\\geq\\min\(m\_\{1\},n\_\{1\}\)\\,r^\{\\prime\}\_\{\\mathrm\{eff\}\}, and otherwise the two images are incomparable, providedκ1\>1\\kappa\_\{1\}\>1\. In HF PEFT the second LoKr factor is low\-rank only when its rank parameter satisfiesr′<max⁡\(m2,n2\)/2r^\{\\prime\}<\\max\(m\_\{2\},n\_\{2\}\)/2; otherwise it is full, and LoKr is unconstrained KronA, with maximum rankν=min⁡\(m1,n1\)​min⁡\(m2,n2\)\\nu=\\min\(m\_\{1\},n\_\{1\}\)\\min\(m\_\{2\},n\_\{2\}\)\. On a768768\-wide layer \(24⊗3224\\otimes 32\) the maximum rank therefore jumps from360360atr′=15r^\{\\prime\}=15to768768atr′=16r^\{\\prime\}=16\. With decompose\_both,CCis also capped at rankr′r^\{\\prime\}\.

Indexℝm\\mathbb\{R\}^\{m\}by pairs\(i1,i2\)\(i\_\{1\},i\_\{2\}\)andℝn\\mathbb\{R\}^\{n\}by\(j1,j2\)\(j\_\{1\},j\_\{2\}\)\. The rank across\{U1​U2\}\|\{V1​V2\}\\\{U\_\{1\}U\_\{2\}\\\}\\mid\\\{V\_\{1\}V\_\{2\}\\\}is the ordinary rank ofΔ\\Delta\. The rank across\{U1​V1\}\|\{U2​V2\}\\\{U\_\{1\}V\_\{1\}\\\}\\mid\\\{U\_\{2\}V\_\{2\}\\\}is the rank of the rearrangementℛ⁡\(Δ\)∈ℝm1​n1×m2​n2\\mathcal\{R\}\(\\Delta\)\\in\\mathbb\{R\}^\{m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}\}with entriesℛ​\(Δ\)\(i1​j1\),\(i2​j2\)=Δ\(i1​i2\),\(j1​j2\)\\mathcal\{R\}\(\\Delta\)\_\{\(i\_\{1\}j\_\{1\}\),\(i\_\{2\}j\_\{2\}\)\}=\\Delta\_\{\(i\_\{1\}i\_\{2\}\),\(j\_\{1\}j\_\{2\}\)\}, a linear bijection that permutes entries, withℛ⁡\(C⊗D\)=vec⁡C​vec⁡D⊤\\mathcal\{R\}\(C\\otimes D\)=\\vect C\\,\\vect D^\{\\top\}\([Auxiliary result B\.13](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. \(In this proof and the next,ℛ\\mathcal\{R\}denotes this rearrangement\.\)

*\(a\)*rank⁡ℛ⁡\(Δ\)≤k\\rk\\mathcal\{R\}\(\\Delta\)\\leq kiffℛ⁡\(Δ\)=∑i≤kxi​yi⊤\\mathcal\{R\}\(\\Delta\)=\\sum\_\{i\\leq k\}x\_\{i\}y\_\{i\}^\{\\top\}, iffΔ=∑i≤kℛ−1​\(xi​yi⊤\)=∑i≤kCi⊗Di\\Delta=\\sum\_\{i\\leq k\}\\mathcal\{R\}^\{\-1\}\(x\_\{i\}y\_\{i\}^\{\\top\}\)=\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}withvec⁡Ci=xi\\vect C\_\{i\}=x\_\{i\}andvec⁡Di=yi\\vect D\_\{i\}=y\_\{i\}\. The matrixℛ⁡\(Δ\)\\mathcal\{R\}\(\\Delta\)has sizem1​n1×m2​n2m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}, which gives the second claim \([Auxiliary result B\.13](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\)\)\.

*\(b\)*rank⁡\(C⊗D\)=rank⁡C⋅rank⁡D\\rk\(C\\otimes D\)=\\rk C\\cdot\\rk D\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\)\), andℛ⁡\(C⊗D\)=vec⁡C​vec⁡D⊤\\mathcal\{R\}\(C\\otimes D\)=\\vect C\\vect D^\{\\top\}has rank11forC,D≠0C,D\\neq 0\. The largest value ofrank⁡C⋅rank⁡D\\rk C\\cdot\\rk Dismin⁡\(m1,n1\)​min⁡\(m2,n2\)=ν\\min\(m\_\{1\},n\_\{1\}\)\\min\(m\_\{2\},n\_\{2\}\)=\\nu, attained by factors of full rank\.

*\(c\)*Take Schmidt decompositionsu=∑i≤ρuσi​xi⊗yiu=\\sum\_\{i\\leq\\rho\_\{u\}\}\\sigma\_\{i\}x\_\{i\}\\otimes y\_\{i\}andv=∑j≤ρvτj​zj⊗wjv=\\sum\_\{j\\leq\\rho\_\{v\}\}\\tau\_\{j\}z\_\{j\}\\otimes w\_\{j\}with positive coefficients and orthonormal families \([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. By the mixed\-product rule,

u​v⊤=∑i,jσi​τj​\(xi​zj⊤\)⊗\(yi​wj⊤\)\.uv^\{\\top\}=\\sum\_\{i,j\}\\sigma\_\{i\}\\tau\_\{j\}\\,\(x\_\{i\}z\_\{j\}^\{\\top\}\)\\otimes\(y\_\{i\}w\_\{j\}^\{\\top\}\)\.The two families\(xi​zj⊤\)i​j\(x\_\{i\}z\_\{j\}^\{\\top\}\)\_\{ij\}and\(yi​wj⊤\)i​j\(y\_\{i\}w\_\{j\}^\{\\top\}\)\_\{ij\}are Frobenius\-orthonormal, because⟨xi​zj⊤,xi′​zj′⊤⟩=\(xi⊤​xi′\)​\(zj⊤​zj′\)\\langle x\_\{i\}z\_\{j\}^\{\\top\},\\,x\_\{i^\{\\prime\}\}z\_\{j^\{\\prime\}\}^\{\\top\}\\rangle=\(x\_\{i\}^\{\\top\}x\_\{i^\{\\prime\}\}\)\\,\(z\_\{j\}^\{\\top\}z\_\{j^\{\\prime\}\}\)and similarly for the second family, and theρu​ρv\\rho\_\{u\}\\rho\_\{v\}coefficients are positive\. By[Auxiliary result B\.13](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\) this is an operator\-Schmidt decomposition, sorank⊗⁡\(u​v⊤\)=ρu​ρv\\rk\_\{\\otimes\}\(uv^\{\\top\}\)=\\rho\_\{u\}\\rho\_\{v\}\. The Schmidt rank ofuuis the rank of itsm1×m2m\_\{1\}\\times m\_\{2\}reshape, which equalsmin⁡\(m1,m2\)\\min\(m\_\{1\},m\_\{2\}\)outside the zero set of the maximal minors, a proper algebraic subset \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\); likewise forvv\. So for genericu,vu,v,rank⊗⁡\(u​v⊤\)=min⁡\(m1,m2\)​min⁡\(n1,n2\)=κ1\\rk\_\{\\otimes\}\(uv^\{\\top\}\)=\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\)=\\kappa\_\{1\}\. If\(m1−m2\)​\(n1−n2\)≥0\(m\_\{1\}\-m\_\{2\}\)\(n\_\{1\}\-n\_\{2\}\)\\geq 0, then eitherm1≤m2m\_\{1\}\\leq m\_\{2\}andn1≤n2n\_\{1\}\\leq n\_\{2\}, soκ1=m1​n1=min⁡\(m1​n1,m2​n2\)\\kappa\_\{1\}=m\_\{1\}n\_\{1\}=\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\), orm1≥m2m\_\{1\}\\geq m\_\{2\}andn1≥n2n\_\{1\}\\geq n\_\{2\}, soκ1=m2​n2=min⁡\(m1​n1,m2​n2\)\\kappa\_\{1\}=m\_\{2\}n\_\{2\}=\\min\(m\_\{1\}n\_\{1\},m\_\{2\}n\_\{2\}\); in both casesκ1\\kappa\_\{1\}is the bound of \(a\)\. Equal square splits are aligned, and so is any split with the smaller factor first on both sides\.

*\(d\)*A generic rank\-oneu​v⊤uv^\{\\top\}lies inℳ≤1⊆ℳ≤r\\mathcal\{M\}\_\{\\leq 1\}\\subseteq\\mathcal\{M\}\_\{\\leq r\}and hasrank⊗=κ1\>k\\rk\_\{\\otimes\}=\\kappa\_\{1\}\>kby \(c\), so it is not in𝒦≤k\\mathcal\{K\}\_\{\\leq k\}\. A productC⊗DC\\otimes Dwith factors of full rank hasrank⊗=1≤k\\rk\_\{\\otimes\}=1\\leq kand rankν\>r\\nu\>rby \(b\)\. Neither argument uses alignment\. For equal square splitsκ1=ν=s⋅s=N\\kappa\_\{1\}=\\nu=s\\cdot s=NandIN=Is⊗IsI\_\{N\}=I\_\{s\}\\otimes I\_\{s\}\.

*\(e\)*Any pointing ofLoRAr\\mathrm\{LoRA\}\_\{r\}has imageWres\+ℳ≤rW\_\{\\mathrm\{res\}\}\+\\mathcal\{M\}\_\{\\leq r\}for its frozen residualWresW\_\{\\mathrm\{res\}\}\. SupposeWres\+ℳ≤r⊆W0\+𝒦≤kW\_\{\\mathrm\{res\}\}\+\\mathcal\{M\}\_\{\\leq r\}\\subseteq W\_\{0\}\+\\mathcal\{K\}\_\{\\leq k\}, and putc=Wres−W0c=W\_\{\\mathrm\{res\}\}\-W\_\{0\}, soc\+ℳ≤r⊆𝒦≤kc\+\\mathcal\{M\}\_\{\\leq r\}\\subseteq\\mathcal\{K\}\_\{\\leq k\}\. The set𝒦≤k\\mathcal\{K\}\_\{\\leq k\}is closed \(the zero set of the\(k\+1\)×\(k\+1\)\(k\+1\)\\times\(k\+1\)minors ofℛ⁡\(⋅\)\\mathcal\{R\}\(\\cdot\)\) and closed under scaling\. ForX∈ℳ≤rX\\in\\mathcal\{M\}\_\{\\leq r\}andt\>0t\>0,c\+t​X∈𝒦≤kc\+tX\\in\\mathcal\{K\}\_\{\\leq k\}, hencec/t\+X∈𝒦≤kc/t\+X\\in\\mathcal\{K\}\_\{\\leq k\}, and lettingt→∞t\\to\\inftygivesX∈𝒦≤kX\\in\\mathcal\{K\}\_\{\\leq k\}\. Soℳ≤r⊆𝒦≤k\\mathcal\{M\}\_\{\\leq r\}\\subseteq\\mathcal\{K\}\_\{\\leq k\}, which contradicts \(d\) whenk<κ1k<\\kappa\_\{1\}\. The second clause restatesIm⁡LoRAr=W0\+ℳ≤r\\Imop\\mathrm\{LoRA\}\_\{r\}=W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}for the zero\-init pointing\.

For LoKr,rank⁡\(C⊗B​A\)=rank⁡C⋅rank⁡\(B​A\)≤min⁡\(m1,n1\)⋅min⁡\(r′,m2,n2\)\\rk\(C\\otimes BA\)=\\rk C\\cdot\\rk\(BA\)\\leq\\min\(m\_\{1\},n\_\{1\}\)\\cdot\\min\(r^\{\\prime\},m\_\{2\},n\_\{2\}\)\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\),\(b\)\), with equality forCCof full rank andB​ABAof rankreff′r^\{\\prime\}\_\{\\mathrm\{eff\}\}, which exists by[Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\. Ifr≥min⁡\(m1,n1\)​reff′r\\geq\\min\(m\_\{1\},n\_\{1\}\)r^\{\\prime\}\_\{\\mathrm\{eff\}\}, every update has rank at mostrrandIm⁡LoKrr′⊆W0\+ℳ≤r\\Imop\\mathrm\{LoKr\}\_\{r^\{\\prime\}\}\\subseteq W\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. Otherwise the maximising update has rank aboverr, soIm⁡LoKrr′⊈Im⁡LoRAr\\Imop\\mathrm\{LoKr\}\_\{r^\{\\prime\}\}\\not\\subseteq\\Imop\\mathrm\{LoRA\}\_\{r\}; and sinceIm⁡LoKrr′⊆W0\+𝒦≤1\\Imop\\mathrm\{LoKr\}\_\{r^\{\\prime\}\}\\subseteq W\_\{0\}\+\\mathcal\{K\}\_\{\\leq 1\}, the first clause withk=1<κ1k=1<\\kappa\_\{1\}givesIm⁡LoRAr⊈Im⁡LoKrr′\\Imop\\mathrm\{LoRA\}\_\{r\}\\not\\subseteq\\Imop\\mathrm\{LoKr\}\_\{r^\{\\prime\}\}\. When HF’s second factor is full, LoKr isC⊗DC\\otimes Dwith both factors free, whose maximum rank isν\\nuby \(b\)\. On a768768\-wide layer HF factors768=24⋅32768=24\\cdot 32on both sides, som1=n1=24m\_\{1\}=n\_\{1\}=24andm2=n2=32m\_\{2\}=n\_\{2\}=32: forr′=15<16r^\{\\prime\}=15<16the maximum rank is24⋅15=36024\\cdot 15=360, and forr′=16r^\{\\prime\}=16the second factor is full and the maximum is24⋅32=76824\\cdot 32=768\. The decompose\_both option factorsCCthrough rankr′r^\{\\prime\}, which capsrank⁡C\\rk Catr′r^\{\\prime\}\. The thresholds and factorisations are read from the HF PEFT source\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)\. ∎

### C\.19 Proposition II\.12

Proposition II\.12\(gauge census: rank is a cut, dimension is a volume\)\.\(a\)d⁡\(M\)≤\|M\|−dim\(generic gauge orbit\)d\(M\)\\leq\\lvert M\\rvert\-\\dim\(\\text\{generic gauge orbit\}\), with equality when the gauge acts locally transitively on generic fibres\.\(b\)The values in Table[9](https://arxiv.org/html/2610.12728#S2.T9)hold for onem×nm\\times nbox, withr≤min⁡\(m,n\)r\\leq\\min\(m,n\)throughout except for VeRA \(whose published RoBERTa\-base setting hasr=1024\>768r=1024\>768\)\. In the LoHa row,F=r1​\(m\+n−r1\)\+r2​\(m\+n−r2\)−\(m\+n−1\)F=r\_\{1\}\(m\+n\-r\_\{1\}\)\+r\_\{2\}\(m\+n\-r\_\{2\}\)\-\(m\+n\-1\)is a proved upper bound fordd\. The gauge column lists the exhibited Lie group\. It acts locally transitively on generic fibres away from them​nmncap, but not where the cap binds; for LoHa at\(m,n,r1,r2\)=\(4,4,2,2\)\(m,n,r\_\{1\},r\_\{2\}\)=\(4,4,2,2\)the orbit has dimension1515and the fibre1616\. BitFit does not touch them×nm\\times nbox; it is linear in the biases\. Hybrid GraLoRA \(hybrid\_r\>0\{\}\>0\) is not covered by the table\. FourierFT is injective only when its sampled frequencies contain no conjugate pair\(u,v\),\(−u,−v\)\(u,v\),\(\-u,\-v\); otherwised=\|M\|−\#​\{pairs\}d=\\lvert M\\rvert\-\\\#\\\{\\text\{pairs\}\\\}\. At LLM sizes this is close to\|M\|\\lvert M\\rvert, but at768×768768\\times 768with10001000frequencies a pair occurs in about half of the draws \(HF’s default seed777777gives exactly two pairs there\)\.\(c\)*Equal\-budget trades\.*Assume2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\),r2≤min⁡\(m,n\)r^\{2\}\\leq\\min\(m,n\)and2​r2<m\+n−12r^\{2\}<m\+n\-1\. LoHar,rand LoRA2rboth cost2​r​\(m\+n\)2r\(m\+n\), but LoHa reaches at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}*fewer*dimensions \(exactly that many whenFFis attained \[Num\]\)\. Forr≥3r\\geq 3it gains rank up tor2\>2​rr^\{2\}\>2rin exchange\. Forr≤2r\\leq 2,r2≤2​rr^\{2\}\\leq 2r, soIm⁡LoHar,r⊊Im⁡LoRA2​r\\Imop\\mathrm\{LoHa\}\_\{r,r\}\\subsetneq\\Imop\\mathrm\{LoRA\}\_\{2r\}and LoHa is strictly dominated\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1\(still with2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\)\) the bound no longer separates them, and whereverFFis attained LoHar,rmatches or exceeds LoRA2rin dimension as well as in rank, for instance at\(m,n,r\)=\(9,9,3\)\(m,n,r\)=\(9,9,3\), atr=64r=64on4096×40964096\\times 4096, and at LyCORIS\-styler=32r=32on320320\- to640640\-wide layers\. GraLoRAkcosts the same as LoRAr, reaches the same number of dimensions, and has maximum rankmin⁡\(k​r,m,n\)\\min\(kr,m,n\), but caps every block at rankr/kr/k\. A generic rank\-rrupdate has rankrron every block, so fork≥2k\\geq 2andk​r≤min⁡\(m,n\)kr\\leq\\min\(m,n\)the images of LoRArand GraLoRAkare incomparable\.

*\(a\)*By[Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(d\),d⁡\(M\)d\(M\)is the maximal rank ofd​ρd\\rho, attained on an open set, which is dense whenρ\\rhois real\-analytic on a connected domain \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. At a pointqqof maximal rank the fibre throughqqis, nearqq, a submanifold of dimension\|M\|−d⁡\(M\)\\lvert M\\rvert\-d\(M\)\([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\), and it contains the gauge orbit ofqq, an immersed submanifold\. At genericqqthis givesdim\(orbit\)≤\|M\|−d⁡\(M\)\\dim\(\\text\{orbit\}\)\\leq\\lvert M\\rvert\-d\(M\)\. If the gauge acts locally transitively on generic fibres, the orbit is open in the fibre and the two dimensions agree\.

*\(b\), LoRA\.*GLr\\mathrm\{GL\}\_\{r\}acts on pairs of full rank by\(B​g−1,g​A\)\(Bg^\{\-1\},gA\), freely, and its orbits are the fibres of\(B,A\)↦B​A\(B,A\)\\mapsto BAoverℳr\\mathcal\{M\}\_\{r\}\([Auxiliary result B\.5](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\), sod=r⁡\(m\+n\)−r2=r⁡\(m\+n−r\)d=r\(m\+n\)\-r^\{2\}=r\(m\+n\-r\)\. The maximum rank isrr\.

*GraLoRA\.*Withk\|m,n,rk\\mid m,n,r, the update is ak×kk\\times kgrid of blocks of sizep×q=mk×nkp\\times q=\\tfrac\{m\}\{k\}\\times\\tfrac\{n\}\{k\}, each an independent LoRA of rankt=r/kt=r/k, so the image isW0W\_\{0\}plus the product ofk2k^\{2\}copies ofℳ≤t​\(p×q\)\\mathcal\{M\}\_\{\\leq t\}\(p\\times q\), of dimensionk2​t​\(p\+q−t\)=r⁡\(m\+n−r\)k^\{2\}\\,t\(p\+q\-t\)=r\(m\+n\-r\)sincet≤min⁡\(p,q\)t\\leq\\min\(p,q\)\. The gauge∏GLt\\prod\\mathrm\{GL\}\_\{t\}acts freely on generic parameters with orbits of total dimensionk2​t2=r2k^\{2\}t^\{2\}=r^\{2\}, equal to the fibre dimension\. For the rank, write block\(i,j\)\(i,j\)asUi​j​Vi​j⊤U\_\{ij\}V\_\{ij\}^\{\\top\}; thenΔ​W=blockdiag⁡\(L1,…,Lk\)​S\\Delta W=\\operatorname\{blockdiag\}\(L\_\{1\},\\dots,L\_\{k\}\)\\,SwithLi=\[Ui​1⋯Ui​k\]∈ℝp×rL\_\{i\}=\[U\_\{i1\}\\cdots U\_\{ik\}\]\\in\\mathbb\{R\}^\{p\\times r\}andS∈ℝk​r×nS\\in\\mathbb\{R\}^\{kr\\times n\}stacking theVi​j⊤V\_\{ij\}^\{\\top\}placed in column blockjj, sorank⁡Δ​W≤min⁡\(k​r,m,n\)\\rk\\Delta W\\leq\\min\(kr,m,n\)\. \(Placing rank\-ttblocks only on the diagonal would give rankrr, notk​rkr\.\) To see thatmin⁡\(k​r,m,n\)\\min\(kr,m,n\)is attained, assumep≤qp\\leq q\(otherwise transpose\) and puta=min⁡\(k​t,p\)a=\\min\(kt,p\), so thatmin⁡\(k​r,m,n\)=k​a\\min\(kr,m,n\)=ka\. Writea=k​α\+βa=k\\alpha\+\\betawith0≤β<k0\\leq\\beta<k, and letNi​j=α\+\[\(j−i\)modk<β\]N\_\{ij\}=\\alpha\+\[\\,\(j\-i\)\\bmod k<\\beta\\,\]\. Then every row and column sum ofNNisa≤p≤qa\\leq p\\leq q, andNi​j≤tN\_\{ij\}\\leq t, becauseα≤t\\alpha\\leq twithα=t\\alpha=tonly whenβ=0\\beta=0\. In block rowiipickNi​jN\_\{ij\}distinct rows for eachjj\(in alla≤pa\\leq prows\), in block columnjjpickNi​jN\_\{ij\}distinct columns for eachii\(in alla≤qa\\leq qcolumns\), and put a11at the matched pairs\. The resulting0/10/1matrix has at most one11in each row and column, so its rank is∑i​jNi​j=k​a\\sum\_\{ij\}N\_\{ij\}=ka, and its block\(i,j\)\(i,j\)has rankNi​j≤tN\_\{ij\}\\leq t, so it lies in the image\. Rank at leastk​akais the nonvanishing of a minor, a polynomial in the block factors, so it holds for generic parameters \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\.

*LoHa\.*The update is\(B1​A1\)⊙\(B2​A2\)\(B\_\{1\}A\_\{1\}\)\\odot\(B\_\{2\}A\_\{2\}\)with budget\(r1\+r2\)​\(m\+n\)\(r\_\{1\}\+r\_\{2\}\)\(m\+n\)\. The torus acts by\(B1,A1,B2,A2\)↦\(D1​B1,A1​D2,D1−1​B2,A2​D2−1\)\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)\\mapsto\(D\_\{1\}B\_\{1\},A\_\{1\}D\_\{2\},D\_\{1\}^\{\-1\}B\_\{2\},A\_\{2\}D\_\{2\}^\{\-1\}\)and preserves the update, because\(D1​X​D2\)⊙\(D1−1​Y​D2−1\)=X⊙Y\(D\_\{1\}XD\_\{2\}\)\\odot\(D\_\{1\}^\{\-1\}YD\_\{2\}^\{\-1\}\)=X\\odot Yfor diagonalD1,D2D\_\{1\},D\_\{2\}; together withGLr1×GLr2\\mathrm\{GL\}\_\{r\_\{1\}\}\\times\\mathrm\{GL\}\_\{r\_\{2\}\}acting on the two LoRA factors this gives an action ofG=GLr1×GLr2×Tm×TnG=\\mathrm\{GL\}\_\{r\_\{1\}\}\\times\\mathrm\{GL\}\_\{r\_\{2\}\}\\times T\_\{m\}\\times T\_\{n\}\. We compute the stabiliser at generic parameters\. Ifm=n=r1m=n=r\_\{1\}, thenr1​\(m\+n−r1\)=m​nr\_\{1\}\(m\+n\-r\_\{1\}\)=mnandF≥m​nF\\geq mn\(sincer2​\(m\+n−r2\)−\(m\+n−1\)=\(r2−1\)​\(m\+n−r2−1\)≥0r\_\{2\}\(m\+n\-r\_\{2\}\)\-\(m\+n\-1\)=\(r\_\{2\}\-1\)\(m\+n\-r\_\{2\}\-1\)\\geq 0\), so the boundd≤min⁡\(m​n,F\)d\\leq\\min\(mn,F\)is trivial; otherwise, by symmetry under transposition, assumem≥r1\+1m\\geq r\_\{1\}\+1\. Let\(g1,g2,D1,D2\)\(g\_\{1\},g\_\{2\},D\_\{1\},D\_\{2\}\)fix the point\. FromD1​B1​g1−1=B1D\_\{1\}B\_\{1\}g\_\{1\}^\{\-1\}=B\_\{1\}, every row ofB1B\_\{1\}is a left eigenvector ofg1g\_\{1\}; for genericB1B\_\{1\}anyr1r\_\{1\}of its rows form a basis and the remaining rows have all coordinates nonzero in that basis, which forces a single eigenvalue, sog1=c​Ig\_\{1\}=cIandD1=c​ID\_\{1\}=cI\. Fromg1​A1​D2=A1g\_\{1\}A\_\{1\}D\_\{2\}=A\_\{1\}and nonzero columns ofA1A\_\{1\},D2=c−1​ID\_\{2\}=c^\{\-1\}I\. FromD1−1​B2​g2−1=B2D\_\{1\}^\{\-1\}B\_\{2\}g\_\{2\}^\{\-1\}=B\_\{2\}andB2B\_\{2\}of full column rank \(m≥r2m\\geq r\_\{2\}\),g2=c−1​Ig\_\{2\}=c^\{\-1\}I, and then the last equation holds\. So the stabiliser is\{\(c​I,c−1​I,c​I,c−1​I\)\}\\\{\(cI,c^\{\-1\}I,cI,c^\{\-1\}I\)\\\}, of dimension11, and the generic orbit has dimensionr12\+r22\+m\+n−1r\_\{1\}^\{2\}\+r\_\{2\}^\{2\}\+m\+n\-1\. By \(a\),d≤\(r1\+r2\)​\(m\+n\)−r12−r22−\(m\+n−1\)=Fd\\leq\(r\_\{1\}\+r\_\{2\}\)\(m\+n\)\-r\_\{1\}^\{2\}\-r\_\{2\}^\{2\}\-\(m\+n\-1\)=F, and triviallyd≤m​nd\\leq mn\. Whered=Fd=Fthe orbit has the dimension of the fibre and the action is locally transitive; at\(4,4,2,2\)\(4,4,2,2\)the orbit has dimension4\+4\+7=154\+4\+7=15while the \[Num\] valued=16d=16gives a fibre of dimension32−16=1632\-16=16\. For the rank,rank⁡\(X⊙Y\)≤r1​r2\\rk\(X\\odot Y\)\\leq r\_\{1\}r\_\{2\}\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\) and≤min⁡\(m,n\)\\leq\\min\(m,n\)\. To attaink=min⁡\(r1​r2,m,n\)k=\\min\(r\_\{1\}r\_\{2\},m,n\), choose distinct pairs\(ai,bi\)∈\[r1\]×\[r2\]\(a\_\{i\},b\_\{i\}\)\\in\[r\_\{1\}\]\\times\[r\_\{2\}\]fori≤ki\\leq k, let the firstkkrows ofB1B\_\{1\}beeai⊤e\_\{a\_\{i\}\}^\{\\top\}and the firstkkcolumns ofA1A\_\{1\}beeaie\_\{a\_\{i\}\}, likewiseB2,A2B\_\{2\},A\_\{2\}with thebib\_\{i\}, and set the rest to zero\. Then\(B1A1\)i​j=\[ai=aj\]\(B\_\{1\}A\_\{1\}\)\_\{ij\}=\[a\_\{i\}=a\_\{j\}\]and\(B2A2\)i​j=\[bi=bj\]\(B\_\{2\}A\_\{2\}\)\_\{ij\}=\[b\_\{i\}=b\_\{j\}\]fori,j≤ki,j\\leq k, so the Hadamard product isIkI\_\{k\}in the top\-left corner and zero elsewhere, of rankkk; by[Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)this rank is generic\.

*HRA\.*The proof of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(a\) computesd=r​n−r⁡\(r\+1\)/2d=rn\-r\(r\+1\)/2and the fibre dimensionr⁡\(r\+1\)/2r\(r\+1\)/2for every1≤r≤n1\\leq r\\leq n, with no use of parity or ofr≤n−2r\\leq n\-2; forr∈\{n−1,n\}r\\in\\\{n\-1,n\\\}the value isdimSO⁡\(n\)\\dim\\mathrm\{SO\}\(n\)\. The scalings\(ℝ×\)r\(\\mathbb\{R\}^\{\\times\}\)^\{r\}are the exhibited Lie gauge, andW0Hu1⋯Hur−W0W\_\{0\}H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}\-W\_\{0\}has rank at mostrr\. Evenness ofrris used only by HF’s paired initialisation, which needs pairs to start atW0W\_\{0\}\.

*DoRA\.*The value ofddis[Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)\.GLr\\mathrm\{GL\}\_\{r\}acts on\(B,A\)\(B,A\)by\(B​g−1,g​A\)\(Bg^\{\-1\},gA\), fixingμ\\mu, freely on pairs of full rank and preservingρ\\rho\. For genericW0W\_\{0\}with\(m−r\)​\(n−r\)≥m\(m\-r\)\(n\-r\)\\geq mthe fibre has dimensionr⁡\(m\+n\)\+m−r⁡\(m\+n−r\)−m=r2r\(m\+n\)\+m\-r\(m\+n\-r\)\-m=r^\{2\}, soGLr\\mathrm\{GL\}\_\{r\}is locally transitive on generic fibres\. Otherwised=m​nd=mnand the fibre has dimensionr⁡\(m\+n\)\+m−m​n=r2\+m−\(m−r\)​\(n−r\)\>r2r\(m\+n\)\+m\-mn=r^\{2\}\+m\-\(m\-r\)\(n\-r\)\>r^\{2\}, which is66at\(3,3,2\)\(3,3,2\); for rank\-oneW0W\_\{0\},d=r⁡\(m\+n−r\)\+m−rd=r\(m\+n\-r\)\+m\-rand the fibre has dimensionr2\+rr^\{2\}\+r\. The update can have rank up tomin⁡\(m,n\)\\min\(m,n\)\([Proposition II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5), last item\)\.

*VeRA\.*The update isdiag⁡\(b\)​B​diag⁡\(d\)​A\\diag\(b\)\\,B\\diag\(d\)\\,Awith frozen randomB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\},A∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}and trainableb∈ℝmb\\in\\mathbb\{R\}^\{m\},d∈ℝrd\\in\\mathbb\{R\}^\{r\}, so\|M\|=m\+r\\lvert M\\rvert=m\+r, and\(b,d\)↦\(c​b,c−1​d\)\(b,d\)\\mapsto\(cb,c^\{\-1\}d\)preserves it\. The valued⁡\(M\)=m\+r−1d\(M\)=m\+r\-1is checked numerically \[Num\]\. The rank is at mostrank⁡\(B​diag⁡\(d\)​A\)≤min⁡\(r,m,n\)\\rk\(B\\diag\(d\)A\)\\leq\\min\(r,m,n\), and for invertiblediag⁡\(b\)\\diag\(b\),diag⁡\(d\)\\diag\(d\)it equalsrank⁡\(B​A′\)\\rk\(BA^\{\\prime\}\)withA′=diag⁡\(d\)​AA^\{\\prime\}=\\diag\(d\)A, which ismin⁡\(r,m,n\)\\min\(r,m,n\)for generic frozen matrices \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. None of this usesr≤min⁡\(m,n\)r\\leq\\min\(m,n\)\.

*KronA\.*The updateA⊗BA\\otimes Bsatisfiesℛ⁡\(A⊗B\)=vec⁡A​vec⁡B⊤\\mathcal\{R\}\(A\\otimes B\)=\\vect A\\vect B^\{\\top\}, so the image isW0W\_\{0\}plus the preimage under the linear bijectionℛ\\mathcal\{R\}\([Auxiliary result B\.13](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\) of the Segre coneℳ≤1​\(m1​n1×m2​n2\)\\mathcal\{M\}\_\{\\leq 1\}\(m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}\), of dimensionm1​n1\+m2​n2−1m\_\{1\}n\_\{1\}\+m\_\{2\}n\_\{2\}\-1\([Auxiliary result B\.8](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. IfA⊗B=A′⊗B′≠0A\\otimes B=A^\{\\prime\}\\otimes B^\{\\prime\}\\neq 0, thenvec⁡A​vec​B⊤=vec⁡A′​vec​B′⁣⊤\\vect A\\vect B^\{\\top\}=\\vect A^\{\\prime\}\\vect B^\{\\prime\\top\}, soA′=c​AA^\{\\prime\}=cAandB′=c−1​BB^\{\\prime\}=c^\{\-1\}B; the fibre over a nonzero update is the orbit ofℝ×\\mathbb\{R\}^\{\\times\}\. The maximum rank isν\\nuby[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(b\)\.

*Tensor\-train and Tucker adapters\.*The update is a fixed reshape, a linear bijection, of a tensor in TT or Tucker format, soddis the dimension of the set of tensors of the given admissible minimal ranks, which is the parameter count minus∑ere2\\sum\_\{e\}r\_\{e\}^\{2\}, with the bond gauge∏eGLre\\prod\_\{e\}\\mathrm\{GL\}\_\{r\_\{e\}\}acting on the cores \([Auxiliary result B\.15](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\)\.

*Linear methods\.*Forρ⁡\(q\)=θ0\+L​q\\rho\(q\)=\\theta\_\{0\}\+LqwithLLlinear,d=rank⁡Ld=\\rk L, the fibres are the cosetsq\+ker⁡Lq\+\\ker L, and translation byker⁡L\\ker Lacts transitively on each\. LoRA\-FA, LoRA\-XS, FourierFT, C3A and masks are of this form; so is MiSS in each HF mode, including the BAT mode, whose updateblocks⁡\(W0\)​M\+M\\operatorname\{blocks\}\(W\_\{0\}\)M\+Mis linear in the trained blockMM\([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\)\.

*FourierFT\.*HF’s update isc​Re⁡\(ifft2⁡\(S\)\)c\\operatorname\{Re\}\(\\operatorname\{ifft2\}\(S\)\)withSSsupported on a sampled setΩ\\Omegaof\|M\|\\lvert M\\rvertfrequencies\. By[Auxiliary result B\.16](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\), frequencyp=\(u,v\)p=\(u,v\)contributessp​κps\_\{p\}\\kappa\_\{p\}withκp​\(j,l\)=1m​n​cos⁡\(2​π​\(u​j/m\+v​l/n\)\)\\kappa\_\{p\}\(j,l\)=\\tfrac\{1\}\{mn\}\\cos\\big\(2\\pi\(uj/m\+vl/n\)\\big\), andκp=κ−p\\kappa\_\{p\}=\\kappa\_\{\-p\}\. Writeχp​\(j,l\)=e2​π​i​\(u​j/m\+v​l/n\)\\chi\_\{p\}\(j,l\)=e^\{2\\pi\\mathrm\{i\}\(uj/m\+vl/n\)\}, so that⟨χp,χp′⟩=mn\[p=p′\]\\langle\\chi\_\{p\},\\chi\_\{p^\{\\prime\}\}\\rangle=mn\\,\[p=p^\{\\prime\}\]andcos⁡\(⋯\)=12​\(χp\+χ−p\)\\cos\(\\cdots\)=\\tfrac\{1\}\{2\}\(\\chi\_\{p\}\+\\chi\_\{\-p\}\)\. Hence⟨κp,κp′⟩=0\\langle\\kappa\_\{p\},\\kappa\_\{p^\{\\prime\}\}\\rangle=0unlessp′=±pp^\{\\prime\}=\\pm p, and everyκp≠0\\kappa\_\{p\}\\neq 0\. So the patterns of distinct classes\{p,−p\}\\\{p,\-p\\\}are linearly independent, and the rank ofs↦∑psp​κps\\mapsto\\sum\_\{p\}s\_\{p\}\\kappa\_\{p\}is the number of classes met byΩ\\Omega, which is\|Ω\|\\lvert\\Omega\\rvertminus the number of conjugate pairs insideΩ\\Omega\. Each pair\{p,−p\}⊆Ω\\\{p,\-p\\\}\\subseteq\\Omegacontributes the kernel directionsp=1s\_\{p\}=1,s−p=−1s\_\{\-p\}=\-1\. The frequencies of pairs in random draws and for HF’s default seed are numerical observations\.

*\(c\)*Assume2​r≤min⁡\(m,n\)2r\\leq\\min\(m,n\),r2≤min⁡\(m,n\)r^\{2\}\\leq\\min\(m,n\)and2​r2<m\+n−12r^\{2\}<m\+n\-1\. Both methods cost2​r​\(m\+n\)2r\(m\+n\), andd⁡\(LoRA2​r\)=2​r​\(m\+n−2​r\)d\(\\mathrm\{LoRA\}\_\{2r\}\)=2r\(m\+n\-2r\)\. By \(b\),d⁡\(LoHar,r\)≤F=2​r​\(m\+n−r\)−\(m\+n−1\)d\(\\mathrm\{LoHa\}\_\{r,r\}\)\\leq F=2r\(m\+n\-r\)\-\(m\+n\-1\), and

d⁡\(LoRA2​r\)−F=\(m\+n−1\)−2​r2\>0\.d\(\\mathrm\{LoRA\}\_\{2r\}\)\-F=\(m\+n\-1\)\-2r^\{2\}\>0\.So LoHa reaches at least\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}fewer dimensions; sinceF<d⁡\(LoRA2​r\)≤m​nF<d\(\\mathrm\{LoRA\}\_\{2r\}\)\\leq mn, the cap does not bind, and the gap is exactly\(m\+n−1\)−2​r2\(m\+n\-1\)\-2r^\{2\}whend⁡\(LoHar,r\)=Fd\(\\mathrm\{LoHa\}\_\{r,r\}\)=F\. Its maximum rank ismin⁡\(r2,m,n\)=r2\\min\(r^\{2\},m,n\)=r^\{2\}by \(b\), against2​r2rforLoRA2​r\\mathrm\{LoRA\}\_\{2r\}, andr2\>2​rr^\{2\}\>2riffr≥3r\\geq 3\. Forr≤2r\\leq 2, every LoHa update has rank at mostr2≤2​rr^\{2\}\\leq 2r, soIm⁡LoHar,r⊆W0\+ℳ≤2​r=Im⁡LoRA2​r\\Imop\\mathrm\{LoHa\}\_\{r,r\}\\subseteq W\_\{0\}\+\\mathcal\{M\}\_\{\\leq 2r\}=\\Imop\\mathrm\{LoRA\}\_\{2r\}\(both pointed atW0W\_\{0\}, as HF zero\-initialises one LoHa factor\), and the inclusion is strict because the dimensions differ\. When2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1,F≥d⁡\(LoRA2​r\)F\\geq d\(\\mathrm\{LoRA\}\_\{2r\}\)and the bound does not separate them; whereverd⁡\(LoHar,r\)=Fd\(\\mathrm\{LoHa\}\_\{r,r\}\)=F, LoHa reaches at least as many dimensions, and its maximum rankmin⁡\(r2,m,n\)\\min\(r^\{2\},m,n\)is at leastmin⁡\(2​r,m,n\)=2​r\\min\(2r,m,n\)=2r, becauser≥2r\\geq 2here \(forr=1r=1,2≥m\+n−12\\geq m\+n\-1contradictsm,n≥2m,n\\geq 2\)\. The examples satisfy2​r2≥m\+n−12r^\{2\}\\geq m\+n\-1:18≥1718\\geq 17at\(9,9,3\)\(9,9,3\),8192≥81918192\\geq 8191atr=64r=64on4096×40964096\\times 4096, and2048≥m\+n−12048\\geq m\+n\-1forr=32r=32andm,n≤640m,n\\leq 640\.

GraLoRAkhas budgetk2⋅t⁡\(p\+q\)=r⁡\(m\+n\)k^\{2\}\\cdot t\(p\+q\)=r\(m\+n\), dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)and maximum rankmin⁡\(k​r,m,n\)\\min\(kr,m,n\)by \(b\)\. Letk≥2k\\geq 2andk​r≤min⁡\(m,n\)kr\\leq\\min\(m,n\)\. A rank\-rrupdateB​ABAhas blocksBi​AjB\_\{i\}A\_\{j\}, withBiB\_\{i\}the rows ofBBin block rowiiandAjA\_\{j\}the columns ofAAin block columnjj; for genericB,AB,AeveryBiB\_\{i\}\(p×rp\\times r,r≤pr\\leq p\) andAjA\_\{j\}\(r×qr\\times q,r≤qr\\leq q\) has rankrr, so every block has rankr\>r/kr\>r/k, and the update is not in GraLoRA’s image\. Conversely GraLoRA reaches rankk​r\>rkr\>r, outsideW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. Both images containW0W\_\{0\}, so they are incomparable\. ∎

### C\.20 Proposition II\.13

Proposition II\.13\(transport of structure\)\.ForT∈GL⁡\(Θ\)T\\in\\mathrm\{GL\}\(\\Theta\)putτT​\(θ\)=θ0\+T⁡\(θ−θ0\)\\tau\_\{T\}\(\\theta\)=\\theta\_\{0\}\+T\(\\theta\-\\theta\_\{0\}\)andT∗​M:=\(Q,q0,τT∘ρ\)T\_\{\*\}M:=\(Q,q\_\{0\},\\tau\_\{T\}\\circ\\rho\)\.•T∗T\_\{\*\}is a strict automorphism of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\), with inverse\(T−1\)∗\(T^\{\-1\}\)\_\{\*\}\. It mapsS1S\_\{1\}toT​S1T\\,S\_\{1\}and preservesdd,gg, the gauge group, the first\-order deficit, polynomial degree, the regular or apex type and the germ type of the image\. It does not preserve rank, and, for the family extension below, it does not preserve the covariance group\.•KT∗​M=T​KM​T⊤K^\{T\_\{\*\}M\}=T\\,K^\{M\}\\,T^\{\\top\}for everyTT\. For any update rule that queries the loss only throughq↦L⁡\(ρ⁡\(q\)\)q\\mapsto L\(\\rho\(q\)\), with its own state onQQ,T∗​MT\_\{\*\}Mtrained onLLandMMtrained onL∘τTL\\circ\\tau\_\{T\}have identicalqq\-trajectories, so theirθ\\theta\-trajectories are conjugate byτT\\tau\_\{T\}\. Rules that read∇θL\\nabla\_\{\\theta\}Land solve a least\-squares problem onΘ\\Theta\(LoRA\-Pro\), andθ\\theta\-level weight decay or clipping, are covered only for orthogonalTT\. Orthogonality ofTTis exactly the condition forτT\\tau\_\{T\}to be a Frobenius isometry; for rules of the first kind it is needed only to keep the Frobenius spectrum ofKK, or an isometry\-invariant loss class such as∥θ−θ⋆∥2\\lVert\\theta\-\\theta^\{\\star\}\\rVert^\{2\}, unchanged\.•Coordinatewise optimizers \(SGD, Adam\) are also equivariant under coordinate permutations ofQQ\. The Kronecker identification below needs this, because it composesℛ∗−1\\mathcal\{R\}^\{\-1\}\_\{\*\}with the vec reshape ofQQ\.•*Families and rebasing\.*ForTTindependent of the base point, define the family extension\(T∗​M\)​\(θ\):=\(Q,q0,θ\+T⁡\(ρθ−θ\)\)\(T\_\{\*\}M\)\(\\theta\):=\(Q,q\_\{0\},\\theta\+T\(\\rho\_\{\\theta\}\-\\theta\)\)\. For an additive family with fixed shapeCC\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\),T∗​MT\_\{\*\}Mis additive with shapeT​CTC, so transport commutes with rebasing:RK​\(T∗​M\)=τT​\(RK​\(M\)\)R\_\{K\}\(T\_\{\*\}M\)=\\tau\_\{T\}\(R\_\{K\}\(M\)\)\([Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(a\)\)\. For action\-type families this can fail even forT=2​IT=2I, because the transported family acts byI\+T⁡\(g−I\)I\+T\(g\-I\)\. For OFT onΘ=ℝ1×2\\Theta=\\mathbb\{R\}^\{1\\times 2\}, two cycles ofθ↦θ⁡\(2​R−I\)\\theta\\mapsto\\theta\(2R\-I\)withR=rot⁡\(π\)R=\\operatorname\{rot\}\(\\pi\)reach9​θ09\\theta\_\{0\}, outside the transported circle\. Transports that depend onθ0\\theta\_\{0\}may fail \(HiRA below\) or may not \(LoRA transported by the SVD frame ofθ0\\theta\_\{0\}, sinceℳ≤r\\mathcal\{M\}\_\{\\leq r\}isO⁡\(m\)×O⁡\(n\)\\mathrm\{O\}\(m\)\\times\\mathrm\{O\}\(n\)\-invariant\)\. The covariance group \([Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)\) is a property of a family and is compared through this extension\.Instances\.•*Kronecker family\.*An isomorphism of slices overΘ′=ℝm1​n1×m2​n2\\Theta^\{\\prime\}=\\mathbb\{R\}^\{m\_\{1\}n\_\{1\}\\times m\_\{2\}n\_\{2\}\}and overΘ\\Theta:∑i≤kCi⊗Di=ℛ∗−1​LoRAk\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}=\\mathcal\{R\}^\{\-1\}\_\{\*\}\\mathrm\{LoRA\}\_\{k\}, withQQreshaped by vec, dynamics included for coordinatewise optimizers\. KronA with unconstrained factors isℛ∗−1​LoRA1\\mathcal\{R\}^\{\-1\}\_\{\*\}\\mathrm\{LoRA\}\_\{1\}\. LoKr isℛ∗−1\\mathcal\{R\}^\{\-1\}\_\{\*\}ofLoRA1\\mathrm\{LoRA\}\_\{1\}with the second factor constrained tovec⁡ℳ≤r′\\vect\\mathcal\{M\}\_\{\\leq r^\{\\prime\}\}whenr′<max⁡\(m2,n2\)/2r^\{\\prime\}<\\max\(m\_\{2\},n\_\{2\}\)/2\(HF’s switch\); otherwise HF’s second factor is full and LoKr is KronA\. LoKr’s decompose\_both option also constrains the first factor\.•*Linear methods as coordinate subspaces in transported frames\.*FourierFT: sparse in the real cosine half of the real Fourier frame\. With scalingcc\(default150150\), HF’s updateΔ​W=c​Re⁡\(ifft2⁡\(S\)\)\\Delta W=c\\operatorname\{Re\}\(\\operatorname\{ifft2\}\(S\)\)is the transport of a mask, withg=0g=0, only when the sampled index set has no conjugate pair\(u,v\),\(−u,−v\)\(u,v\),\(\-u,\-v\); otherwiseggis the number of such pairs\. Its image lies in the point\-symmetric subspaceΔ​Wj,l=Δ​W−j,−l\\Delta W\_\{j,l\}=\\Delta W\_\{\-j,\-l\}, and the frame is orthogonal only up to scale \(self\-conjugate frequencies have twice the squared norm\)\. WaveFT: sparse in a wavelet frame, when HF’s pad–IDWT–crop map is invertible \(plausible for Haar on even shapes; unverified for other wavelet families\)\. C3A: block\-circulant; overℝ\\mathbb\{R\}the frame is given by the powersPkP^\{k\}of the cyclic shift \(Gram matrixb​IbI\); overℂ\\mathbb\{C\},circ⁡\(w\)=F−1​diag⁡\(F​w\)​F\\operatorname\{circ\}\(w\)=F^\{\-1\}\\diag\(Fw\)Ffor first columnww, and the real form has2×22\\times 2rotation–scaling blocks; HF adds a factor1/in\_features1/\\texttt\{in\\\_features\}\. LoRA\-XS, SVFT, TinyLoRA: masks or random subspaces in the SVD frame ofW0W\_\{0\}, aθ0\\theta\_\{0\}\-dependent transport\. SHiRA, FISH, Trainable Tokens: masks in the standard frame\. MiSS/Bone in its default \(balance\) mode:Δ​W=D⁡\(𝟏g⊤⊗Ir\)\\Delta W=D\(\\mathbf\{1\}\_\{g\}^\{\\top\}\\otimes I\_\{r\}\), which is LoRA\-FA with a frozen*summing*projection; its BAT mode,Δ​W=blocks⁡\(W0\)​M\+M\\Delta W=\\operatorname\{blocks\}\(W\_\{0\}\)\\,M\+Mblockwise, is linear in the trained blockMMbut depends onθ0\\theta\_\{0\}, so it is not additive\. These frame readings of FourierFT and C3A are for init\_weights=True; HF’s defaults start offθ0\\theta\_\{0\}\(FourierFT draws its spectrum from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)with scaling150150, so at7682768^\{2\}with10001000frequencies∥Δ​W0∥F≈4\.3\\lVert\\Delta W\_\{0\}\\rVert\_\{F\}\\approx 4\.3, about14%14\\%of a std\-0\.040\.04W0W\_\{0\}; C3A is Xavier\-uniform\), and these are objects with a pointing defect\.•*HiRA*=\(DW0\)∗​LoRA=\(D\_\{W\_\{0\}\}\)\_\{\*\}\\mathrm\{LoRA\}inside the slice atθ0\\theta\_\{0\}, whenW0W\_\{0\}has no zero entries, withDW0:X↦W0⊙XD\_\{W\_\{0\}\}:X\\mapsto W\_\{0\}\\odot X\. BecauseDW0D\_\{W\_\{0\}\}depends onθ0\\theta\_\{0\}, re\-based HiRA is not transported ReLoRA\. Two cycles reachW0⊙\(J\+Z1\)⊙\(J\+Z2\)W\_\{0\}\\odot\(J\+Z\_\{1\}\)\\odot\(J\+Z\_\{2\}\), whose relative updateZ1\+Z2\+Z1⊙Z2Z\_\{1\}\+Z\_\{2\}\+Z\_\{1\}\\odot Z\_\{2\}has rank up tomin⁡\(2​r\+r2,m,n\)\\min\(2r\+r^\{2\},m,n\), attained, where transported ReLoRA stays at rank≤2​r\\leq 2r\.

*The automorphism\.*τT\\tau\_\{T\}is an affine bijection fixingθ0\\theta\_\{0\}with linear partTT, soT∗​MT\_\{\*\}Mis pointed\. Ifh:M→Nh:M\\to NsatisfiesρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}, thenτT∘ρN∘h=τT∘ρM\\tau\_\{T\}\\circ\\rho\_\{N\}\\circ h=\\tau\_\{T\}\\circ\\rho\_\{M\}, sohhis also an arrowT∗​M→T∗​NT\_\{\*\}M\\to T\_\{\*\}N; letT∗T\_\{\*\}act as the identity on arrows\. This is a functor, and\(T−1\)∗​T∗=id\(T^\{\-1\}\)\_\{\*\}T\_\{\*\}=\\mathrm\{id\}on the nose becauseτT−1∘τT=id\\tau\_\{T^\{\-1\}\}\\circ\\tau\_\{T\}=\\mathrm\{id\}\. SoT∗T\_\{\*\}is a strict automorphism\.

*Invariants\.*D​\(τT∘ρ\)q=T​D​ρqD\(\\tau\_\{T\}\\circ\\rho\)\_\{q\}=T\\,D\\rho\_\{q\}for everyqq\. SoS1​\(T∗​M\)=T​S1​\(M\)S\_\{1\}\(T\_\{\*\}M\)=T\\,S\_\{1\}\(M\), and the rank of the differential is unchanged at every point; hencedd\([Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(d\)\),dimS1\\dim S\_\{1\}, the deficit,g=\|M\|−dg=\\lvert M\\rvert\-d, and the regular or apex type are unchanged\. SinceτT\\tau\_\{T\}is injective,τT∘ρ∘φ=τT∘ρ\\tau\_\{T\}\\circ\\rho\\circ\\varphi=\\tau\_\{T\}\\circ\\rhoiffρ∘φ=ρ\\rho\\circ\\varphi=\\rho, so the gauge groups coincide\. Composition with an affine map does not raise polynomial degree, andτT−1\\tau\_\{T^\{\-1\}\}undoes it, so the degree is unchanged\. FinallyτT\\tau\_\{T\}is a diffeomorphism ofΘ\\Thetafixingθ0\\theta\_\{0\}that carriesIm⁡M\\Imop MontoIm⁡T∗​M\\Imop T\_\{\*\}M, so the germs atθ0\\theta\_\{0\}are diffeomorphic\. Rank is not preserved: anyTTwithT⁡\(E11\)=InT\(E\_\{11\}\)=I\_\{n\}\(n≥2n\\geq 2,m=nm=n\) sends a rank\-one update to a full\-rank one\.

*Covariance is not preserved\.*Take the family extension of LoRA1byT=DV:X↦V⊙XT=D\_\{V\}:X\\mapsto V\\odot X, withVVfixed, without zero entries and of rank at least22; its image atWWisW\+V⊙ℳ≤1W\+V\\odot\\mathcal\{M\}\_\{\\leq 1\}\. If\(g,h\)\(g,h\)were in its covariance group for every pair, comparing the images atg​W​h−1gWh^\{\-1\}andWWwould giveV⊙ℳ≤1=g⁡\(V⊙ℳ≤1\)​h−1V\\odot\\mathcal\{M\}\_\{\\leq 1\}=g\\,\(V\\odot\\mathcal\{M\}\_\{\\leq 1\}\)\\,h^\{\-1\}, soV⊙ℳ≤1V\\odot\\mathcal\{M\}\_\{\\leq 1\}would be a union ofGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}\-orbits, that is of rank strataℳk\\mathcal\{M\}\_\{k\}\. It has dimensionm\+n−1m\+n\-1, sinceDVD\_\{V\}is invertible, whiledimℳk=k⁡\(m\+n−k\)\>m\+n−1\\dim\\mathcal\{M\}\_\{k\}=k\(m\+n\-k\)\>m\+n\-1for2≤k≤min⁡\(m,n\)2\\leq k\\leq\\min\(m,n\)\. So it would lie inℳ≤1\\mathcal\{M\}\_\{\\leq 1\}, and in particularV⊙J=VV\\odot J=Vwould have rank at most one, a contradiction\. Hence the covariance group of the transported family is a proper subgroup ofGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, the covariance group of LoRA\.

*Cometric and dynamics\.*KT∗​M=D⁡\(τT​ρ\)​Λ​D​\(τT​ρ\)⊤=T​D​ρ​Λ​D​ρ⊤​T⊤=T​KM​T⊤K^\{T\_\{\*\}M\}=D\(\\tau\_\{T\}\\rho\)\\Lambda D\(\\tau\_\{T\}\\rho\)^\{\\top\}=T\\,D\\rho\\Lambda D\\rho^\{\\top\}\\,T^\{\\top\}=TK^\{M\}T^\{\\top\}\. The methodT∗​MT\_\{\*\}Mtrained onLLminimisesL∘τT∘ρL\\circ\\tau\_\{T\}\\circ\\rhoonQQ, andMMtrained onL∘τTL\\circ\\tau\_\{T\}minimises\(L∘τT\)∘ρ\(L\\circ\\tau\_\{T\}\)\\circ\\rho, the same function\. A rule that sees the loss only through this function, from the same initial state \(and, for stochastic rules, the same random stream\), therefore produces the sameqq\-trajectory, and the weights areτT​\(ρ​\(q​\(t\)\)\)\\tau\_\{T\}\(\\rho\(q\(t\)\)\)againstρ⁡\(q⁡\(t\)\)\\rho\(q\(t\)\)\. A rule that reads∇θL\\nabla\_\{\\theta\}Land takes the Frobenius least\-squares step onΘ\\Thetamovesqqby−\(Dρ⊤T⊤TDρ\)\+Dρ⊤T⊤∇L\-\(D\\rho^\{\\top\}T^\{\\top\}TD\\rho\)^\{\+\}D\\rho^\{\\top\}T^\{\\top\}\\nabla LunderT∗​MT\_\{\*\}M, and by−\(Dρ⊤Dρ\)\+Dρ⊤T⊤∇L\-\(D\\rho^\{\\top\}D\\rho\)^\{\+\}D\\rho^\{\\top\}T^\{\\top\}\\nabla LunderMMwith lossL∘τTL\\circ\\tau\_\{T\}, since∇θ\(L∘τT\)=T⊤∇L\\nabla\_\{\\theta\}\(L\\circ\\tau\_\{T\}\)=T^\{\\top\}\\nabla L\. They agree whenT⊤​T=IT^\{\\top\}T=Iand differ in general: forΘ=ℝ2\\Theta=\\mathbb\{R\}^\{2\},ρ⁡\(q\)=q​e1\\rho\(q\)=qe\_\{1\}andT=diag⁡\(2,1\)T=\\diag\(2,1\)the steps are−12∂1L\-\\tfrac\{1\}\{2\}\\partial\_\{1\}Land−2∂1L\-2\\partial\_\{1\}L\. Weight decay and clipping onθ\\thetause the Frobenius norm, whichτT\\tau\_\{T\}preserves iffTTis orthogonal\. For orthogonalTT,T​K​T⊤TKT^\{\\top\}is similar toKK, so the Frobenius spectrum of the cometric is kept, and∥τT​\(θ\)−τT​\(θ⋆\)∥=∥θ−θ⋆∥\\lVert\\tau\_\{T\}\(\\theta\)\-\\tau\_\{T\}\(\\theta^\{\\star\}\)\\rVert=\\lVert\\theta\-\\theta^\{\\star\}\\rVert\.

*Permutations ofQQ\.*A coordinatewise rule updates each coordinate from that coordinate’s own gradient history with shared hyperparameters\. Ifq⁡\(t\)q\(t\)is its trajectory for an objectivefffromq0q\_\{0\}, thenπ​q​\(t\)\\pi q\(t\)is its trajectory forf∘π−1f\\circ\\pi^\{\-1\}fromπ​q0\\pi q\_\{0\}, for any coordinate permutationπ\\pi\.

*Families and rebasing\.*For an additive family,ρθ=θ\+δ\\rho\_\{\\theta\}=\\theta\+\\deltawithδ\\deltaindependent ofθ\\thetaand shapeC=δ⁡\(Q\)C=\\delta\(Q\), the extension isθ\+T​δ\\theta\+T\\delta, additive with shapeT​CTC\. AfterKKmerge\-and\-restart cycles an additive family reachesRK=θ0\+C\+⋯\+CR\_\{K\}=\\theta\_\{0\}\+C\+\\dots\+C\(KKsummands\), and sinceTTis linear,θ0\+\(T​C\)\+⋯\+\(T​C\)=θ0\+T⁡\(C\+⋯\+C\)=τT​\(RK\)\\theta\_\{0\}\+\(TC\)\+\\dots\+\(TC\)=\\theta\_\{0\}\+T\(C\+\\dots\+C\)=\\tau\_\{T\}\(R\_\{K\}\)\. For OFT onℝ1×2\\mathbb\{R\}^\{1\\times 2\}the family is\{θ​R:R∈SO⁡\(2\)\}\\\{\\theta R:R\\in\\mathrm\{SO\}\(2\)\\\}and its extension byT=2​IT=2Iisθ↦θ\+2​\(θ​R−θ\)=θ⁡\(2​R−I\)\\theta\\mapsto\\theta\+2\(\\theta R\-\\theta\)=\\theta\(2R\-I\)\. WithR=rot⁡\(π\)=−IR=\\operatorname\{rot\}\(\\pi\)=\-I, two cycles giveθ0​\(−3​I\)​\(−3​I\)=9​θ0\\theta\_\{0\}\(\-3I\)\(\-3I\)=9\\theta\_\{0\}\. Every rebasing of OFT stays inθ0⋅SO⁡\(2\)\\theta\_\{0\}\\cdot\\mathrm\{SO\}\(2\), whose transport\{θ0​\(2​S−I\):S∈SO⁡\(2\)\}\\\{\\theta\_\{0\}\(2S\-I\):S\\in\\mathrm\{SO\}\(2\)\\\}consists of vectors of norm at most3​∥θ0∥3\\lVert\\theta\_\{0\}\\rVert, so9​θ09\\theta\_\{0\}lies outside it whenθ0≠0\\theta\_\{0\}\\neq 0\. For LoRA transported byX↦U⁡\(θ\)​X​V​\(θ\)⊤X\\mapsto U\(\\theta\)XV\(\\theta\)^\{\\top\}with orthogonalU⁡\(θ\),V⁡\(θ\)U\(\\theta\),V\(\\theta\), the transported shape isU​ℳ≤r​V⊤=ℳ≤rU\\mathcal\{M\}\_\{\\leq r\}V^\{\\top\}=\\mathcal\{M\}\_\{\\leq r\}, so the transported family equals LoRA’s, and both sides of the rebasing identity areθ0\+ℳ≤min⁡\(K​r,m,n\)\\theta\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}\(by the splitting argument of[Theorem II\.1](https://arxiv.org/html/2610.12728#S2.SS1.SSSx1)\(c\)\)\.

*Kronecker family\.*By[Auxiliary result B\.13](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\),ℛ:Θ→Θ′\\mathcal\{R\}:\\Theta\\to\\Theta^\{\\prime\}is a linear bijection with

ℛ⁡\(∑i≤kCi⊗Di\)=∑ivec⁡Ci​vec​Di⊤=B​A,\\mathcal\{R\}\\Big\(\\sum\_\{i\\leq k\}C\_\{i\}\\otimes D\_\{i\}\\Big\)=\\sum\_\{i\}\\vect C\_\{i\}\\,\\vect D\_\{i\}^\{\\top\}=BA,whereBBhas columnsvec⁡C1,…,vec⁡Ck\\vect C\_\{1\},\\dots,\\vect C\_\{k\}andAAhas rowsvec⁡D1⊤,…,vec⁡Dk⊤\\vect D\_\{1\}^\{\\top\},\\dots,\\vect D\_\{k\}^\{\\top\}\. So thekk\-term Kronecker method isℛ−1∘ρLoRAk\\mathcal\{R\}^\{\-1\}\\circ\\rho\_\{\\mathrm\{LoRA\}\_\{k\}\}up to the vec relabelling ofQQ, and the argument forT∗T\_\{\*\}applies verbatim to an isomorphism between two weight spaces\. Its dynamics agree with LoRAk’s on the pulled\-back loss for rules that query the loss throughρ\\rho, and the vec relabelling is a coordinate permutation, harmless for coordinatewise rules\. KronA isk=1k=1\. For LoKr,ℛ⁡\(C⊗B​A\)=vec⁡C​vec⁡\(B​A\)⊤\\mathcal\{R\}\(C\\otimes BA\)=\\vect C\\vect\(BA\)^\{\\top\}, LoRA1with second factor invec⁡ℳ≤r′\\vect\\mathcal\{M\}\_\{\\leq r^\{\\prime\}\}, when HF factors the second factor; otherwise it is KronA\. With decompose\_both the first factor is factored too\.

*FourierFT\.*By the computation in the proof of[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), the patternsκp\\kappa\_\{p\}of distinct classes\{p,−p\}\\\{p,\-p\\\}are orthogonal; with the sine patternssin⁡\(2​π​\(u​j/m\+v​l/n\)\)\\sin\(2\\pi\(uj/m\+vl/n\)\)of the classes withp≠−pp\\neq\-pthey form an orthogonal basis ofℝm×n\\mathbb\{R\}^\{m\\times n\}\. LetTTsend the standard basis to this basis \(cosines on the positions of the coefficients, sines elsewhere\)\. IfΩ\\Omegahas no conjugate pair, FourierFT isT∗T\_\{\*\}of the mask onΩ\\Omega, withg=0g=0; otherwise the two coefficients of a pair hit the same pattern andggis the number of pairs\. Eachκp\\kappa\_\{p\}is even under\(j,l\)↦\(−j,−l\)\(j,l\)\\mapsto\(\-j,\-l\)\(indices modulom,nm,n\), so the image lies in the point\-symmetric subspace\. A self\-conjugate frequency hasκp=1m​n​χp\\kappa\_\{p\}=\\tfrac\{1\}\{mn\}\\chi\_\{p\}withχp\\chi\_\{p\}real, of squared norm1m​n\\tfrac\{1\}\{mn\}, while forp≠−pp\\neq\-pthe squared norm is12​m​n\\tfrac\{1\}\{2mn\}\. HF’s default draws10001000coefficients from𝒩⁡\(0,1\)\\mathcal\{N\}\(0,1\)and multiplies by150150; for generic frequencies𝔼​∥Δ​W0∥F2=1502⋅1000/\(2⋅7682\)≈19\.1\\mathbb\{E\}\\lVert\\Delta W\_\{0\}\\rVert\_\{F\}^\{2\}=150^\{2\}\\cdot 1000/\(2\\cdot 768^\{2\}\)\\approx 19\.1, a root\-mean\-square norm of4\.374\.37\. A single draw deviates from it by about2%2\\%, since the sum of10001000squared standard normals has relative standard deviation2/1000\\sqrt\{2/1000\}, which gives the value∥Δ​W0∥F≈4\.3\\lVert\\Delta W\_\{0\}\\rVert\_\{F\}\\approx 4\.3quoted\. Against∥W0∥F≈0\.04⋅768≈30\.7\\lVert W\_\{0\}\\rVert\_\{F\}\\approx 0\.04\\cdot 768\\approx 30\.7this is about14%14\\%\.

*Other frames\.*If HF’s pad–IDWT–crop mapPPis invertible, WaveFT isP∗P\_\{\*\}of a mask by definition\. For C3A, the powersP0,…,Pb−1P^\{0\},\\dots,P^\{b\-1\}of the cyclic shift span theb×bb\\times bcirculants and have Gram matrixb​IbI, andcirc⁡\(w\)=F−1​diag⁡\(F​w\)​F\\operatorname\{circ\}\(w\)=F^\{\-1\}\\diag\(Fw\)F; the real form follows from conjugate eigenvalues \([Auxiliary result B\.16](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\),\(b\)\)\. So a block\-circulant update is a mask in the frame\{Ei​j\(blk\)⊗Pk\}\\\{E^\{\(\\mathrm\{blk\}\)\}\_\{ij\}\\otimes P^\{k\}\\\}\. LoRA\-XS’s updateUr​R​Σr​Vr⊤=∑i,j≤rRi​j​σj​ui​vj⊤U\_\{r\}R\\Sigma\_\{r\}V\_\{r\}^\{\\top\}=\\sum\_\{i,j\\leq r\}R\_\{ij\}\\sigma\_\{j\}u\_\{i\}v\_\{j\}^\{\\top\}is a scaled mask in the orthonormal frame\(ui​vj⊤\)\(u\_\{i\}v\_\{j\}^\{\\top\}\)built from the SVD ofW0W\_\{0\}; SVFT’s updateU​M​V⊤UMV^\{\\top\}with sparseMMis a mask there too, and TinyLoRA’s is a random subspace of it\. SHiRA, FISH and Trainable Tokens update a fixed set of entries\. In MiSS’s balance mode HF computes the update on an inputxxasDDtimes the sum of theggchunks ofxxof sizerr, that isD⁡\(𝟏g⊤⊗Ir\)​xD\(\\mathbf\{1\}\_\{g\}^\{\\top\}\\otimes I\_\{r\}\)x; this is LoRA\-FA with the frozen projection𝟏g⊤⊗Ir\\mathbf\{1\}\_\{g\}^\{\\top\}\\otimes I\_\{r\}\. Its BAT mode is linear inMMwith coefficients fromW0W\_\{0\}, so it is linear but not additive\.

*HiRA\.*By[Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(a\), HiRA is\(DW0\)∗​LoRA\(D\_\{W\_\{0\}\}\)\_\{\*\}\\mathrm\{LoRA\}whenW0W\_\{0\}has no zero entries\. A merge givesW1=W0⊙\(J\+Z1\)W\_\{1\}=W\_\{0\}\\odot\(J\+Z\_\{1\}\), and the next cycleW1⊙\(J\+Z2\)=W0⊙\(J\+Z1\)⊙\(J\+Z2\)=W0⊙\(J\+Z1\+Z2\+Z1⊙Z2\)W\_\{1\}\\odot\(J\+Z\_\{2\}\)=W\_\{0\}\\odot\(J\+Z\_\{1\}\)\\odot\(J\+Z\_\{2\}\)=W\_\{0\}\\odot\(J\+Z\_\{1\}\+Z\_\{2\}\+Z\_\{1\}\\odot Z\_\{2\}\)\. WithZk=Bk​AkZ\_\{k\}=B\_\{k\}A\_\{k\},

Z1\+Z2\+Z1⊙Z2=\[B1,B2,B1∙B2\]​\[A1;A2;\(A1⊤∙A2⊤\)⊤\]Z\_\{1\}\+Z\_\{2\}\+Z\_\{1\}\\odot Z\_\{2\}=\\big\[B\_\{1\},\\ B\_\{2\},\\ B\_\{1\}\\bullet B\_\{2\}\\big\]\\,\\big\[A\_\{1\};\\ A\_\{2\};\\ \(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\}\\big\]\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\), which factors throughℝp\\mathbb\{R\}^\{p\}withp=2​r\+r2p=2r\+r^\{2\}\. Call the two factorsL∈ℝm×pL\\in\\mathbb\{R\}^\{m\\times p\}andL′∈ℝp×nL^\{\\prime\}\\in\\mathbb\{R\}^\{p\\times n\}\. The rows ofLLand the columns ofL′L^\{\\prime\}are points\(x,y,x⊗y\)\(x,y,x\\otimes y\)of a variety𝒱\\mathcal\{V\}that spansℝr⊕ℝr⊕ℝr2\\mathbb\{R\}^\{r\}\\oplus\\mathbb\{R\}^\{r\}\\oplus\\mathbb\{R\}^\{r^\{2\}\}: a linear form vanishing on𝒱\\mathcal\{V\}vanishes on\(x,0,0\)\(x,0,0\), on\(0,y,0\)\(0,y,0\)and then on every\(0,0,x⊗y\)\(0,0,x\\otimes y\)\. Choosing points of𝒱\\mathcal\{V\}one at a time outside the span of the previous ones,mmrows can be chosen to spanmin⁡\(m,p\)\\min\(m,p\)dimensions, and likewisenncolumns\. Ifp≤min⁡\(m,n\)p\\leq\\min\(m,n\), thenLLhas full column rank andL′L^\{\\prime\}full row rank, sorank⁡L​L′=p\\rk LL^\{\\prime\}=p\. Ifp\>min⁡\(m,n\)p\>\\min\(m,n\), saym≤nm\\leq n, chooseLLof rankmm, that isL:ℝp→ℝmL:\\mathbb\{R\}^\{p\}\\to\\mathbb\{R\}^\{m\}onto; thenL⁡\(𝒱\)L\(\\mathcal\{V\}\)spansℝm\\mathbb\{R\}^\{m\}, son≥mn\\geq mcolumns ofL′L^\{\\prime\}can be chosen with images spanningℝm\\mathbb\{R\}^\{m\}, andrank⁡L​L′=m\\rk LL^\{\\prime\}=m; the casen<mn<mis the transpose\. In every case rankmin⁡\(2​r\+r2,m,n\)\\min\(2r\+r^\{2\},m,n\)is attained, hence generic \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. Transported ReLoRA instead reachesDW0​\(Z1\+Z2\)D\_\{W\_\{0\}\}\(Z\_\{1\}\+Z\_\{2\}\), whose preimageZ1\+Z2Z\_\{1\}\+Z\_\{2\}has rank at most2​r2r\. ∎

### C\.21 Definition II\.14

Definition II\.14\(covariance group\)\.For a method defined for everyW0W\_\{0\}\(a natural family\), thecovariance group𝒢M⊆GLm×GLn\\mathcal\{G\}\_\{M\}\\subseteq\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}is the largest subgroup withIm⁡M⁡\(g​W0​h−1\)=g​Im⁡M⁡\(W0\)​h−1\\Imop M\(gW\_\{0\}h^\{\-1\}\)=g\\,\\Imop M\(W\_\{0\}\)\\,h^\{\-1\}\. When the method depends on random frames or on data, the identity is required in distribution, with the data transformed\.𝒢M\\mathcal\{G\}\_\{M\}is the*geometry the method lives in*, in Klein’s sense\.

The definition contains one claim: a largest such subgroup exists\.

Let𝒮\\mathcal\{S\}be the set of\(g,h\)∈GLm×GLn\(g,h\)\\in\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}withIm⁡M⁡\(g​W​h−1\)=g​Im⁡M⁡\(W\)​h−1\\Imop M\(gWh^\{\-1\}\)=g\\,\\Imop M\(W\)\\,h^\{\-1\}for everyWWin the domain of the family \(which we assume stable underGLm×GLn\\mathrm\{GL\}\_\{m\}\\times\\mathrm\{GL\}\_\{n\}, or at least under the pairs considered\)\. The pair\(I,I\)\(I,I\)lies in𝒮\\mathcal\{S\}\. If\(g1,h1\),\(g2,h2\)∈𝒮\(g\_\{1\},h\_\{1\}\),\(g\_\{2\},h\_\{2\}\)\\in\\mathcal\{S\}, then for everyWW

Im⁡M⁡\(g1​g2​W​h2−1​h1−1\)=g1​Im⁡M⁡\(g2​W​h2−1\)​h1−1=g1​g2​Im⁡M⁡\(W\)​h2−1​h1−1\.\\Imop M\(g\_\{1\}g\_\{2\}Wh\_\{2\}^\{\-1\}h\_\{1\}^\{\-1\}\)=g\_\{1\}\\,\\Imop M\(g\_\{2\}Wh\_\{2\}^\{\-1\}\)\\,h\_\{1\}^\{\-1\}=g\_\{1\}g\_\{2\}\\,\\Imop M\(W\)\\,h\_\{2\}^\{\-1\}h\_\{1\}^\{\-1\}\.If\(g,h\)∈𝒮\(g,h\)\\in\\mathcal\{S\}, apply its identity atg−1​W​hg^\{\-1\}Wh:Im⁡M⁡\(W\)=g​Im⁡M⁡\(g−1​W​h\)​h−1\\Imop M\(W\)=g\\,\\Imop M\(g^\{\-1\}Wh\)\\,h^\{\-1\}, soIm⁡M⁡\(g−1​W​h\)=g−1​Im⁡M⁡\(W\)​h\\Imop M\(g^\{\-1\}Wh\)=g^\{\-1\}\\Imop M\(W\)h\. So𝒮\\mathcal\{S\}is a subgroup, every subgroup with the property is contained in it, and𝒢M=𝒮\\mathcal\{G\}\_\{M\}=\\mathcal\{S\}\. For random frames or data, read each identity as equality of the laws of the random sets, the pushforwards of the frame and data laws underF↦Im⁡M⁡\(W,F\)F\\mapsto\\Imop M\(W;F\), with the data transformed by the action of\(g,h\)\(g,h\)\(for instancex↦h​xx\\mapsto hx, soC↦h​C​h⊤C\\mapsto hCh^\{\\top\}\)\. The same computation applies, with each equality an equality of laws and withX↦g​X​h−1X\\mapsto gXh^\{\-1\}acting on laws by pushforward; it uses only that the data transformations compose as a group action\. ∎

### C\.22 Observation II\.15

Observation II\.15\(descent through function\-preserving reparametrisations\)\.LetM=∏bMbM=\\prod\_\{b\}M\_\{b\}be a box\-wise method: eachMbM\_\{b\}is a member of a natural family with covariance group𝒢Mb\\mathcal\{G\}\_\{M\_\{b\}\}, and no trainable parameters or random frames are shared across boxes\. LetGarchG\_\{\\mathrm\{arch\}\}act onΘ=∏bΘb×Θfrozen\\Theta=\\prod\_\{b\}\\Theta\_\{b\}\\times\\Theta\_\{\\mathrm\{frozen\}\}, preserving this split and acting on each adapted box byWb↦gb​Wb​hb−1W\_\{b\}\\mapsto g\_\{b\}W\_\{b\}h\_\{b\}^\{\-1\}with\(gb,hb\)∈𝒢Mb\(g\_\{b\},h\_\{b\}\)\\in\\mathcal\{G\}\_\{M\_\{b\}\}\. Assumef⁡\(g​θ,−\)=f⁡\(θ,−\)f\(g\\theta,\-\)=f\(\\theta,\-\)on aGarchG\_\{\\mathrm\{arch\}\}\-stable set containingIm⁡M⁡\(θ0\)\\Imop M\(\\theta\_\{0\}\)\. ThenIm⁡M⁡\(g​θ0\)=g⋅Im⁡M⁡\(θ0\)\\Imop M\(g\\theta\_\{0\}\)=g\\cdot\\Imop M\(\\theta\_\{0\}\), and the set of*functions*reachable byMMfromg​θ0g\\theta\_\{0\}equals the set reachable fromθ0\\theta\_\{0\}\. For methods with random frames, the equality holds in distribution, provided the frames are drawn independently per box\. Boxes that share a frame need a pathwise condition: a law\-preserving mapT\(g,h\)T\_\{\(g,h\)\}on frame space withIm⁡Mb​\(gb​W​hb−1,T​F\)=gb​Im⁡Mb​\(W,F\)​hb−1\\Imop M\_\{b\}\(g\_\{b\}Wh\_\{b\}^\{\-1\};TF\)=g\_\{b\}\\,\\Imop M\_\{b\}\(W;F\)\\,h\_\{b\}^\{\-1\}for allWWandFF, with the sameTTon every box that sharesFF\(marginal covariance in law, as in[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), is not enough\)\. For data\-dependent methods, require covariance pathwise in the data, and that the box\-bbdata atg​θ0g\\theta\_\{0\}is the[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)transform under\(gb,hb\)\(g\_\{b\},h\_\{b\}\)of the data atθ0\\theta\_\{0\}\. This holds for intertwiner gauges such as hidden\-unit permutations, query–key and value–output head gauges and residual rotations\. The statement concerns reachable sets only; training trajectories and outcomes can still differ, for instance under Adam or non\-isotropic initialisations\.Examples ofGarchG\_\{\\mathrm\{arch\}\}:•hidden\-neuron permutations;•the query–key gauge\(WQ,WK\)↦\(g​WQ,g−⁣⊤​WK\)\(W\_\{Q\},W\_\{K\}\)\\mapsto\(gW\_\{Q\},g^\{\-\\top\}W\_\{K\}\)per head\. With no positional encoding or an absolute one,g∈GLdkg\\in\\mathrm\{GL\}\_\{d\_\{k\}\}\. Under RoPE,ggmust commute with every RoPE rotation, which leaves the per\-frequency2×22\\times 2rotation–scalings\(ℂ×\)dk/2\(\\mathbb\{C\}^\{\\times\}\)^\{d\_\{k\}/2\}, shared across a GQA group\. Under partial RoPE onrrof thedkd\_\{k\}dimensions the group is\(ℂ×\)r/2×GLdk−r\(\\mathbb\{C\}^\{\\times\}\)^\{r/2\}\\times\\mathrm\{GL\}\_\{d\_\{k\}\-r\}\. Under QK\-norm,ggmust in addition be orthogonal when the norm’sε\>0\\varepsilon\>0\(conformal,g∈CO⁡\(dk\)g\\in\\mathrm\{CO\}\(d\_\{k\}\), whenε=0\\varepsilon=0\) and satisfyg⊤​Γq​Rt​Γk​g=Γq​Rt​Γkg^\{\\top\}\\Gamma\_\{q\}R\_\{t\}\\Gamma\_\{k\}g=\\Gamma\_\{q\}R\_\{t\}\\Gamma\_\{k\}for every RoPE rotationRtR\_\{t\}, whereΓq,Γk\\Gamma\_\{q\},\\Gamma\_\{k\}are the gains\. OLMo 2 normalisesqqandkkacross all heads jointly;•a single global orthogonal rotationQQof the residual stream of a pre\-norm transformer whose norm gains can all be fused into the linear layers they feed, after LayerNorm is converted to RMSNorm \(SliceGPT’s computational invariance; QuaRot’s offline rotation\)\. SliceGPT’s per\-block rotationsQℓQ\_\{\\ell\}need extra residual matrices and change the architecture\. Sandwich and post\-sublayer norms \(Gemma 2 and 3, OLMo 2\) are excluded, because their gains cannot be fused anddiag⁡\(γ\)\\diag\(\\gamma\)does not commute withQQ\. The comparison is between the gain\-fused model and the fused\-and\-rotated model: gain fusion is a diagonal input rescaling outsideGarchG\_\{\\mathrm\{arch\}\}that already changes PiSSA’s split and block OFT’s image \([Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)\), and converting LayerNorm to RMSNorm changes LoRA’s reachable set on the residual writers\.

*The deterministic statement\.*No parameters are shared, soIm⁡M⁡\(θ\)=∏bIm⁡Mb​\(Wb\)×\{θfrozen\}\\Imop M\(\\theta\)=\\prod\_\{b\}\\Imop M\_\{b\}\(W\_\{b\}\)\\times\\\{\\theta\_\{\\mathrm\{frozen\}\}\\\}\. Forg∈Garchg\\in G\_\{\\mathrm\{arch\}\}, which preserves the split,

Im⁡M⁡\(g​θ0\)\\displaystyle\\Imop M\(g\\theta\_\{0\}\)=∏bIm⁡Mb​\(gb​W0,b​hb−1\)×\{g​θ0,frozen\}\\displaystyle=\\prod\_\{b\}\\Imop M\_\{b\}\(g\_\{b\}W\_\{0,b\}h\_\{b\}^\{\-1\}\)\\times\\\{g\\theta\_\{0,\\mathrm\{frozen\}\}\\\}=∏bgb​Im⁡Mb​\(W0,b\)​hb−1×\{g​θ0,frozen\}=g⋅Im⁡M⁡\(θ0\),\\displaystyle=\\prod\_\{b\}g\_\{b\}\\,\\Imop M\_\{b\}\(W\_\{0,b\}\)\\,h\_\{b\}^\{\-1\}\\times\\\{g\\theta\_\{0,\\mathrm\{frozen\}\}\\\}=g\\cdot\\Imop M\(\\theta\_\{0\}\),using\(gb,hb\)∈𝒢Mb\(g\_\{b\},h\_\{b\}\)\\in\\mathcal\{G\}\_\{M\_\{b\}\}in each factor\. The reachable functions are then

\{Φf​\(θ\):θ∈Im⁡M⁡\(g​θ0\)\}=\{Φf​\(g​θ\):θ∈Im⁡M⁡\(θ0\)\}=\{Φf​\(θ\):θ∈Im⁡M⁡\(θ0\)\},\\\{\\Phi\_\{f\}\(\\theta\):\\theta\\in\\Imop M\(g\\theta\_\{0\}\)\\\}=\\\{\\Phi\_\{f\}\(g\\theta\):\\theta\\in\\Imop M\(\\theta\_\{0\}\)\\\}=\\\{\\Phi\_\{f\}\(\\theta\):\\theta\\in\\Imop M\(\\theta\_\{0\}\)\\\},sincef⁡\(g​θ,−\)=f⁡\(θ,−\)f\(g\\theta,\-\)=f\(\\theta,\-\)forθ∈Im⁡M⁡\(θ0\)\\theta\\in\\Imop M\(\\theta\_\{0\}\)\. \(Stability of the set underGarchG\_\{\\mathrm\{arch\}\}is not needed\.\)

*Independent frames\.*With framesFbF\_\{b\}drawn independently per box, the law of the tuple of setsIm⁡Mb​\(gb​W0,b​hb−1,Fb\)\\Imop M\_\{b\}\(g\_\{b\}W\_\{0,b\}h\_\{b\}^\{\-1\};F\_\{b\}\)is the product of the marginal laws, and by covariance in law each marginal equals the law ofgb​Im⁡Mb​\(W0,b,Fb\)​hb−1g\_\{b\}\\Imop M\_\{b\}\(W\_\{0,b\};F\_\{b\}\)h\_\{b\}^\{\-1\}\. So the tuple has the law of the tuple of setsgb​Im⁡Mb​\(W0,b,Fb\)​hb−1g\_\{b\}\\Imop M\_\{b\}\(W\_\{0,b\};F\_\{b\}\)h\_\{b\}^\{\-1\}\. The reachable function set is a fixed measurable function of the image, and the function\-level identity above holds pathwise, so the laws of the reachable function sets agree\.

*Shared frames\.*Under the pathwise condition,Im⁡M⁡\(g​θ0,T​F\)=g⋅Im⁡M⁡\(θ0,F\)\\Imop M\(g\\theta\_\{0\};TF\)=g\\cdot\\Imop M\(\\theta\_\{0\};F\)for everyFF, with the sameTTin every box that usesFF, andT​FTFhas the law ofFF\. HenceIm⁡M⁡\(g​θ0,F\)\\Imop M\(g\\theta\_\{0\};F\)andg⋅Im⁡M⁡\(θ0,F\)g\\cdot\\Imop M\(\\theta\_\{0\};F\)have the same law\. Marginal covariance does not suffice\. Take two boxes inℝ2×1\\mathbb\{R\}^\{2\\times 1\}sharing a Haar frameF∈O⁡\(2\)F\\in\\mathrm\{O\}\(2\), with the natural familyIm⁡Mb​\(W,F\)=W\+ℝ​F​a​\(W\)\\Imop M\_\{b\}\(W;F\)=W\+\\mathbb\{R\}\\,Fa\(W\), wherea⁡\(W\)=\(w12,w22\)/∥\(w12,w22\)∥a\(W\)=\(w\_\{1\}^\{2\},w\_\{2\}^\{2\}\)/\\lVert\(w\_\{1\}^\{2\},w\_\{2\}^\{2\}\)\\rVertforW≠0W\\neq 0\. Forg∈O⁡\(2\)g\\in\\mathrm\{O\}\(2\), bothF​a​\(g​W\)Fa\(gW\)andg​F​a​\(W\)gFa\(W\)are uniform on the circle, so each box is covariant in law under\(g,1\)\(g,1\)\. Atθ0=\(e1,e2\)\\theta\_\{0\}=\(e\_\{1\},e\_\{2\}\)the two lines have directionsF​e1Fe\_\{1\}andF​e2Fe\_\{2\}, which are orthogonal, and so are those ofg⋅Im⁡M⁡\(θ0,F\)g\\cdot\\Imop M\(\\theta\_\{0\};F\)\. Forggthe rotation byπ/4\\pi/4,g​θ0=\(\(e1\+e2\)/2,\(e2−e1\)/2\)g\\theta\_\{0\}=\\big\(\(e\_\{1\}\+e\_\{2\}\)/\\sqrt\{2\},\(e\_\{2\}\-e\_\{1\}\)/\\sqrt\{2\}\\big\)hasa⁡\(g​e1\)=a⁡\(g​e2\)=\(e1\+e2\)/2a\(ge\_\{1\}\)=a\(ge\_\{2\}\)=\(e\_\{1\}\+e\_\{2\}\)/\\sqrt\{2\}, so the two lines ofIm⁡M⁡\(g​θ0,F\)\\Imop M\(g\\theta\_\{0\};F\)are parallel\. The joint laws differ, and the conclusion fails; the numerical check reports a function\-level instance\.

*Data\.*Suppose eachMbM\_\{b\}is covariant pathwise in its data,

Im⁡Mb​\(gb​W​hb−1,\(gb,hb\)⋅D\)=gb​Im⁡Mb​\(W,D\)​hb−1,\\Imop M\_\{b\}\\big\(g\_\{b\}Wh\_\{b\}^\{\-1\};\(g\_\{b\},h\_\{b\}\)\\cdot D\\big\)=g\_\{b\}\\,\\Imop M\_\{b\}\(W;D\)\\,h\_\{b\}^\{\-1\},and the box\-bbdata atg​θ0g\\theta\_\{0\}is\(gb,hb\)⋅Db\(g\_\{b\},h\_\{b\}\)\\cdot D\_\{b\}\. Then the deterministic argument applies with the data fixed\. For an intertwiner gauge the input of boxbbin the network with weightsg​θg\\thetaishbh\_\{b\}times its input with weightsθ\\theta, for every network input, so calibration inputs transform byx↦hb​xx\\mapsto h\_\{b\}xand second moments byC↦hb​C​hb⊤C\\mapsto h\_\{b\}Ch\_\{b\}^\{\\top\}, as[Definition II\.14](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6)requires\. The examples below are of this kind: a hidden\-unit permutation permutes the inputs of the next box; the query–key and value–output gauges leave the inputs ofWQ,WK,WVW\_\{Q\},W\_\{K\},W\_\{V\}unchanged \(h=Ih=I\) and transform the input ofWOW\_\{O\}; a residual rotationQQrotates every normalised residual input byQQ\. Training is not covered:[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)shows that Adam is not equivariant under general rotations of LoRA’s gauge, and a Kaiming\-uniformA0A\_\{0\}is not rotation\-invariant in law\.

*Hidden\-neuron permutations\.*For a permutation matrixPP,σ⁡\(P​z\)=P​σ​\(z\)\\sigma\(Pz\)=P\\sigma\(z\)for an entrywise activation andP⁡\(u⊙v\)=P​u⊙P​vP\(u\\odot v\)=Pu\\odot Pv, so\(Wup,Wgate,Wdown\)↦\(P​Wup,P​Wgate,Wdown​P⊤\)\(W\_\{\\mathrm\{up\}\},W\_\{\\mathrm\{gate\}\},W\_\{\\mathrm\{down\}\}\)\\mapsto\(PW\_\{\\mathrm\{up\}\},PW\_\{\\mathrm\{gate\}\},W\_\{\\mathrm\{down\}\}P^\{\\top\}\)preserves the block\.

*The query–key gauge\.*Without positional rotation the score isx⊤​WQ⊤​WK​yx^\{\\top\}W\_\{Q\}^\{\\top\}W\_\{K\}y, and since\(g​WQ\)⊤​g−⁣⊤​WK=WQ⊤​WK\(gW\_\{Q\}\)^\{\\top\}g^\{\-\\top\}W\_\{K\}=W\_\{Q\}^\{\\top\}W\_\{K\}, everyg∈GLdkg\\in\\mathrm\{GL\}\_\{d\_\{k\}\}works\. Under RoPE the score between positionsssandttisx⊤​WQ⊤​Rt−s​WK​yx^\{\\top\}W\_\{Q\}^\{\\top\}R\_\{t\-s\}W\_\{K\}ywith commuting rotationsRτR\_\{\\tau\}acting by angleτ​ωj\\tau\\omega\_\{j\}on thejj\-th coordinate plane\. After the gauge it isx⊤​WQ⊤​g⊤​Rt−s​g−⁣⊤​WK​yx^\{\\top\}W\_\{Q\}^\{\\top\}g^\{\\top\}R\_\{t\-s\}g^\{\-\\top\}W\_\{K\}y\. ForWQ,WKW\_\{Q\},W\_\{K\}of full row rank, invariance for all inputs and positions is equivalent tog⊤​Rτ​g−⁣⊤=Rτg^\{\\top\}R\_\{\\tau\}g^\{\-\\top\}=R\_\{\\tau\}for everyτ\\tau, that is,g⊤g^\{\\top\}commutes with everyRτR\_\{\\tau\}\. RoPE’s frequenciesωj=base−2j/dk\\omega\_\{j\}=\\mathrm\{base\}^\{\-2j/d\_\{k\}\}are distinct and lie in\(0,1\]⊂\(0,π\)\(0,1\]\\subset\(0,\\pi\), soR1R\_\{1\}has the distinct eigenvaluese±i​ωje^\{\\pm\\mathrm\{i\}\\omega\_\{j\}\}; a matrix commuting withR1R\_\{1\}preserves each coordinate plane and commutes with a nontrivial rotation on it, so it is a rotation–scalinga​I2\+b​JaI\_\{2\}\+bJthere\. The commutant is\(ℂ×\)dk/2\(\\mathbb\{C\}^\{\\times\}\)^\{d\_\{k\}/2\}in the invertible case, and it is closed under transposition, so the condition ong⊤g^\{\\top\}is the same condition ongg\. In a GQA group oneWKW\_\{K\}is paired with severalWQW\_\{Q\}, so all of them use the samegg\. Under partial RoPE,R1R\_\{1\}is the identity on thedk−rd\_\{k\}\-runrotated coordinates and has no eigenvalue11on the others, so commuting matrices preserve this splitting, and the group is\(ℂ×\)r/2×GLdk−r\(\\mathbb\{C\}^\{\\times\}\)^\{r/2\}\\times\\mathrm\{GL\}\_\{d\_\{k\}\-r\}\.

*QK\-norm\.*The score becomes\(Γq​N​\(q\)\)⊤​Rt​\(Γk​N​\(k\)\)\(\\Gamma\_\{q\}N\(q\)\)^\{\\top\}R\_\{t\}\(\\Gamma\_\{k\}N\(k\)\)withq=WQ​xq=W\_\{Q\}x,k=WK​yk=W\_\{K\}yandN\(v\)=v\(∥v∥2/dk\+ε\)−1/2N\(v\)=v\\,\(\\lVert v\\rVert^\{2\}/d\_\{k\}\+\\varepsilon\)^\{\-1/2\}\. Ifε\>0\\varepsilon\>0andggis orthogonal, theng−⁣⊤=gg^\{\-\\top\}=gandN⁡\(g​v\)=g​N​\(v\)N\(gv\)=gN\(v\), so the score becomesN​\(q\)⊤​g⊤​Γq​Rt​Γk​g​N​\(k\)N\(q\)^\{\\top\}g^\{\\top\}\\Gamma\_\{q\}R\_\{t\}\\Gamma\_\{k\}g\\,N\(k\), which is invariant under the stated relation\. Ifε=0\\varepsilon=0, thenNNremoves scale, andg=c​Og=cOwithc\>0c\>0,OOorthogonal, givesN⁡\(g​q\)=O​N​\(q\)N\(gq\)=ON\(q\)andN⁡\(g−⁣⊤​k\)=O​N​\(k\)N\(g^\{\-\\top\}k\)=ON\(k\); the relation is then required ofO=g/cO=g/c\. Conversely, letWQ,WKW\_\{Q\},W\_\{K\}have full row rank, so thatqqandkkrange overℝdk\\mathbb\{R\}^\{d\_\{k\}\}\. Invariance att=0t=0gives⟨Γq​N​\(g​q\),Γk​N​\(g−⁣⊤​k\)⟩=⟨Γq​N​\(q\),Γk​N​\(k\)⟩\\langle\\Gamma\_\{q\}N\(gq\),\\Gamma\_\{k\}N\(g^\{\-\\top\}k\)\\rangle=\\langle\\Gamma\_\{q\}N\(q\),\\Gamma\_\{k\}N\(k\)\\ranglefor allq,kq,k\. The values ofNNspanℝdk\\mathbb\{R\}^\{d\_\{k\}\}\(they fill a ball whenε\>0\\varepsilon\>0and a sphere whenε=0\\varepsilon=0\), and the gains are invertible, so we may choosek1,…,kdkk\_\{1\},\\dots,k\_\{d\_\{k\}\}for which the vectorsΓk​N​\(g−⁣⊤​ki\)\\Gamma\_\{k\}N\(g^\{\-\\top\}k\_\{i\}\)form a basis\. Solving the resulting linear equations givesΓq​N​\(g​q\)=A′​Γq​N​\(q\)\\Gamma\_\{q\}N\(gq\)=A^\{\\prime\}\\,\\Gamma\_\{q\}N\(q\)for a fixed matrixA′A^\{\\prime\}, soN⁡\(g​q\)=A​N​\(q\)N\(gq\)=A\\,N\(q\)withA=Γq−1​A′​ΓqA=\\Gamma\_\{q\}^\{\-1\}A^\{\\prime\}\\Gamma\_\{q\}, andAAis invertible because the valuesN⁡\(g​q\)N\(gq\)spanℝdk\\mathbb\{R\}^\{d\_\{k\}\}\. Ifε\>0\\varepsilon\>0, lettingq→0q\\to 0\(whereN⁡\(v\)=v/ε\+O⁡\(∥v∥3\)N\(v\)=v/\\sqrt\{\\varepsilon\}\+O\(\\lVert v\\rVert^\{3\}\)\) givesA=gA=g, and then∥g​q∥=∥q∥\\lVert gq\\rVert=\\lVert q\\rVertfor allqq, soggis orthogonal\. Ifε=0\\varepsilon=0,∥N⁡\(v\)∥=dk\\lVert N\(v\)\\rVert=\\sqrt\{d\_\{k\}\}for allv≠0v\\neq 0, soAAis orthogonal, andg​qgqis a positive multiple ofA​qAqfor everyqq, soA−1​gA^\{\-1\}ghas every vector as an eigenvector andg=c​Ag=cAwithc\>0c\>0\. Withggorthogonal \(orO=g/cO=g/c\), invariance for allttand allN⁡\(q\),N⁡\(k\)N\(q\),N\(k\)forces the stated relation\.

*Residual rotations\.*In a pre\-norm transformer with gains fused and RMSNormN\(h\)=h\(∥h∥2/d\+ε\)−1/2N\(h\)=h\\,\(\\lVert h\\rVert^\{2\}/d\+\\varepsilon\)^\{\-1/2\}, replace the residual streamhhbyQ​hQh: embeddingsE↦Q​EE\\mapsto QE, the writersWO,Wdown↦Q​WO,Q​WdownW\_\{O\},W\_\{\\mathrm\{down\}\}\\mapsto QW\_\{O\},QW\_\{\\mathrm\{down\}\}, and every reader \(theq,k,vq,k,v, gate and up projections and the head\)W↦W​Q⊤W\\mapsto WQ^\{\\top\}\. Since∥Q​h∥=∥h∥\\lVert Qh\\rVert=\\lVert h\\rVert,N⁡\(Q​h\)=Q​N​\(h\)N\(Qh\)=QN\(h\), so each reader seesW​Q⊤​Q​N​\(h\)=W​N​\(h\)WQ^\{\\top\}QN\(h\)=WN\(h\), and the function is unchanged\. A gain that is not fused would sit betweenNNand the reader asdiag⁡\(γ\)\\diag\(\\gamma\), anddiag⁡\(γ\)​Q≠Q​diag⁡\(γ\)\\diag\(\\gamma\)Q\\neq Q\\diag\(\\gamma\)unlessγ\\gammais constant\. A post\-sublayer norm writesh\+diag⁡\(γ\)​N​\(WO​z\)h\+\\diag\(\\gamma\)N\(W\_\{O\}z\), anddiag⁡\(γ\)​N​\(Q​WO​z\)=diag⁡\(γ\)​Q​N​\(WO​z\)≠Q​diag⁡\(γ\)​N​\(WO​z\)\\diag\(\\gamma\)N\(QW\_\{O\}z\)=\\diag\(\\gamma\)QN\(W\_\{O\}z\)\\neq Q\\diag\(\\gamma\)N\(W\_\{O\}z\), so the rotation does not preserve the function\. LayerNorm subtracts the mean, the projectionI−𝟏𝟏⊤/dI\-\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}/d, which commutes withQQonly ifQ​𝟏=±𝟏Q\\mathbf\{1\}=\\pm\\mathbf\{1\}; SliceGPT therefore folds the projection into the writers and uses RMSNorm\. In the original model a LayerNorm ignores the component of the residual stream along𝟏\\mathbf\{1\}, so LoRA’s updates of a writer reach functions only through\(I−𝟏𝟏⊤/d\)​Δ​W\(I\-\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}/d\)\\Delta W; in the converted model the writer is\(I−𝟏𝟏⊤/d\)​WO\(I\-\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}/d\)W\_\{O\}and an updateΔ​W\\Delta Wwith a component along𝟏\\mathbf\{1\}is seen by every later RMSNorm, so the reachable function sets differ\. Gain fusion replaces a readerWWbyW​diag⁡\(γ\)W\\diag\(\\gamma\), a non\-conformal input rescaling whenγ\\gammais not constant; by[Proposition II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)it changes PiSSA’s split, and it changes block OFT’s image\{W​diag⁡\(γ\)​R\}\\\{W\\diag\(\\gamma\)R\\\}against\{W​R​diag⁡\(γ\)\}\\\{WR\\diag\(\\gamma\)\\\}unlessdiag⁡\(γ\)\\diag\(\\gamma\)commutes with the block\-orthogonal group, that is unlessγ\\gammais constant on each block\. Consecutive SliceGPT rotationsQℓ−1≠QℓQ\_\{\\ell\-1\}\\neq Q\_\{\\ell\}require the matrixQℓ−1⊤​QℓQ\_\{\\ell\-1\}^\{\\top\}Q\_\{\\ell\}on the residual path, a new operation\. ∎

### C\.23 Proposition II\.16

Proposition II\.16\(covariant data\-aware splits\)\.LetC=𝔼⁡\[x​x⊤\]≻0C=\\mathbb\{E\}\[xx^\{\\top\}\]\\succ 0be the input second moment, so that underx↦h​xx\\mapsto hxwe haveW↦W​h−1W\\mapsto Wh^\{\-1\}andC↦h​C​h⊤C\\mapsto hCh^\{\\top\}\. LetL​L⊤=CLL^\{\\top\}=Candp:=min⁡\(m,n\)p:=\\min\(m,n\)\. Write\[M\]r\[M\]\_\{r\}for the rank\-rrtruncated SVD,⟨M⟩r:=M−\[M\]p−r\\langle M\\rangle\_\{r\}:=M\-\[M\]\_\{p\-r\}for the bottomrrsingular components, andCO⁡\(n\)=ℝ\>0⋅O⁡\(n\)\\mathrm\{CO\}\(n\)=\\mathbb\{R\}\_\{\>0\}\\cdot\\mathrm\{O\}\(n\)\. Assume1≤r<p1\\leq r<pand the spectral gap each split needs:σr\>σr\+1\\sigma\_\{r\}\>\\sigma\_\{r\+1\}ofW0​LW\_\{0\}Lin \(a\); in \(b\),σr\>σr\+1\\sigma\_\{r\}\>\\sigma\_\{r\+1\}ofW0​CW\_\{0\}CorW0W\_\{0\}for top\-rrsplits, andσp−r\>σp−r\+1\\sigma\_\{p\-r\}\>\\sigma\_\{p\-r\+1\}for bottom\-rrsplits\. Covariance groups are taken over the generic set where these gaps hold\.\(a\) The whitened splitX∗=arg⁡minrank⁡X≤r⁡∥\(W0−X\)​L∥F=\[W0​L\]r​L−1=Πr​W0,X^\{\*\}=\\arg\\min\_\{\\rk X\\leq r\}\\lVert\(W\_\{0\}\-X\)L\\rVert\_\{F\}=\[W\_\{0\}L\]\_\{r\}L^\{\-1\}=\\Pi\_\{r\}W\_\{0\},whereΠr\\Pi\_\{r\}is the top\-rreigenprojector of the output second momentW0​C​W0⊤W\_\{0\}CW\_\{0\}^\{\\top\}, isGLn\\mathrm\{GL\}\_\{n\}\-covariant on the input side andCO⁡\(m\)\\mathrm\{CO\}\(m\)\-covariant on the output side; its joint group isCO⁡\(m\)×GLn\\mathrm\{CO\}\(m\)\\times\\mathrm\{GL\}\_\{n\}\. This is the classical reduced\-rank\-regression \(SVD\-LLM\) solution\.\(b\) The formula\-level CorDA splits, IPM \(X=\[W0​C\]r​C−1X=\[W\_\{0\}C\]\_\{r\}C^\{\-1\}, the minimiser of∥\(W0−X\)​C∥F\\lVert\(W\_\{0\}\-X\)C\\rVert\_\{F\}\) and KPM \(the bottom\-rrcomponents ofW0​CW\_\{0\}Cmapped back byC−1C^\{\-1\},X=⟨W0​C⟩r​C−1X=\\langle W\_\{0\}C\\rangle\_\{r\}C^\{\-1\}\), with or without CorDA’s trace\-relative dampingC\+κ⁡\(tr⁡C/n\)​IC\+\\kappa\(\\tr C/n\)I, and the PiSSA and MiLoRA splits \(\[W0\]r\[W\_\{0\}\]\_\{r\}and the bottom\-rrpart⟨W0⟩r\\langle W\_\{0\}\\rangle\_\{r\}\), have input\-side covariance group exactlyCO⁡\(n\)\\mathrm\{CO\}\(n\); jointly,CO⁡\(m\)×CO⁡\(n\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)\.\(c\)*Scope\.*An absolute dampingC\+λ​IC\+\\lambda Ireduces the input\-side group of \(a\) and of \(b\) toO⁡\(n\)\\mathrm\{O\}\(n\)\. The official and HF CorDA code divide each calibration sample by the absolute value of its largest entry before accumulatingCC, which keeps only positive scalings and coordinate permutations, so \(b\) describes the CorDA formulas and does not cover shipped CorDA under rotations\.

A splitX=X⁡\(W0,C\)X=X\(W\_\{0\},C\)of rank at mostrryields the method with frozen residualW0−XW\_\{0\}\-Xand image\(W0−X\)\+ℳ≤r\(W\_\{0\}\-X\)\+\\mathcal\{M\}\_\{\\leq r\}\. Under\(g,h\)\(g,h\), withC′=h​C​h⊤C^\{\\prime\}=hCh^\{\\top\}, covariance of images means\(g​W0​h−1−X⁡\(g​W0​h−1,C′\)\)\+ℳ≤r=g⁡\(W0−X⁡\(W0,C\)\)​h−1\+ℳ≤r\(gW\_\{0\}h^\{\-1\}\-X\(gW\_\{0\}h^\{\-1\},C^\{\\prime\}\)\)\+\\mathcal\{M\}\_\{\\leq r\}=g\(W\_\{0\}\-X\(W\_\{0\},C\)\)h^\{\-1\}\+\\mathcal\{M\}\_\{\\leq r\}, usingg​ℳ≤r​h−1=ℳ≤rg\\mathcal\{M\}\_\{\\leq r\}h^\{\-1\}=\\mathcal\{M\}\_\{\\leq r\}\. Sincer<pr<p, the translation stabiliser ofℳ≤r\\mathcal\{M\}\_\{\\leq r\}is\{0\}\\\{0\\\}\(equation \([18](https://arxiv.org/html/2610.12728#A3.E18)\)\), so this is equivalent to

X⁡\(g​W0​h−1,h​C​h⊤\)=g​X​\(W0,C\)​h−1\.X\(gW\_\{0\}h^\{\-1\},hCh^\{\\top\}\)=g\\,X\(W\_\{0\},C\)\\,h^\{\-1\}\.
*Lemma\.*Let1≤s<p1\\leq s<p,g∈GLmg\\in\\mathrm\{GL\}\_\{m\}andk∈GLnk\\in\\mathrm\{GL\}\_\{n\}\. Then\[g​M​k\]s=g​\[M\]s​k\[gMk\]\_\{s\}=g\[M\]\_\{s\}kfor everyM∈ℝm×nM\\in\\mathbb\{R\}^\{m\\times n\}for which both truncations are unique iffg∈CO⁡\(m\)g\\in\\mathrm\{CO\}\(m\)andk∈CO⁡\(n\)k\\in\\mathrm\{CO\}\(n\)\. “If”: forg=a​Pg=aPandk=b​Qk=bQwitha,b\>0a,b\>0andP,QP,Qorthogonal, an SVDM=∑iσi​ui​vi⊤M=\\sum\_\{i\}\\sigma\_\{i\}u\_\{i\}v\_\{i\}^\{\\top\}gives the SVDg​M​k=∑i\(a​b​σi\)​\(P​ui\)​\(Q⊤​vi\)⊤gMk=\\sum\_\{i\}\(ab\\sigma\_\{i\}\)\(Pu\_\{i\}\)\(Q^\{\\top\}v\_\{i\}\)^\{\\top\}with the same ordering, so\[g​M​k\]s=g​\[M\]s​k\[gMk\]\_\{s\}=g\[M\]\_\{s\}k\. “Only if”: letUs⊆ℝmU\_\{s\}\\subseteq\\mathbb\{R\}^\{m\}be anyss\-dimensional subspace andy⟂Usy\\perp U\_\{s\}a unit vector, and likewiseVs⊆ℝnV\_\{s\}\\subseteq\\mathbb\{R\}^\{n\}andx⟂Vsx\\perp V\_\{s\}\(possible becauses<ps<p\)\. LetMsM\_\{s\}have ranksswith column spaceUsU\_\{s\}and row spaceVsV\_\{s\}, and for smallt\>0t\>0putM=Ms\+t​y​x⊤M=M\_\{s\}\+t\\,yx^\{\\top\}\. Then\[M\]s=Ms\[M\]\_\{s\}=M\_\{s\}, andg​M​k=g​Ms​k\+t⁡\(g​y\)​\(k⊤​x\)⊤gMk=gM\_\{s\}k\+t\\,\(gy\)\(k^\{\\top\}x\)^\{\\top\}, whose truncation is unique for smallttbecauseg​Ms​kgM\_\{s\}khas rankss\. The identity gives\[g​M​k\]s=g​Ms​k\[gMk\]\_\{s\}=gM\_\{s\}k, so the residualt⁡\(g​y\)​\(k⊤​x\)⊤t\(gy\)\(k^\{\\top\}x\)^\{\\top\}of the truncation must have its column space orthogonal to that of\[g​M​k\]s\[gMk\]\_\{s\}and its row space orthogonal to that of\[g​M​k\]s\[gMk\]\_\{s\}\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\):\[X\]s\[X\]\_\{s\}andX−\[X\]sX\-\[X\]\_\{s\}are built from disjoint sets of singular vectors\)\. Sog​y⟂g​Usgy\\perp gU\_\{s\}andk⊤​x⟂k⊤​Vsk^\{\\top\}x\\perp k^\{\\top\}V\_\{s\}\. Since every pair of orthogonal unit vectorsu,yu,yoccurs withu∈Usu\\in U\_\{s\}andy⟂Usy\\perp U\_\{s\},ggmaps orthogonal vectors to orthogonal vectors\. Then for orthonormal eigenvectorse,fe,fofg⊤​gg^\{\\top\}gwith eigenvaluesα,β\\alpha,\\beta,0=⟨g⁡\(e\+f\),g⁡\(e−f\)⟩=α−β0=\\langle g\(e\+f\),g\(e\-f\)\\rangle=\\alpha\-\\beta, sog⊤​gg^\{\\top\}gis scalar andg∈CO⁡\(m\)g\\in\\mathrm\{CO\}\(m\); likewisek⊤∈CO⁡\(n\)k^\{\\top\}\\in\\mathrm\{CO\}\(n\)\.

*\(a\)*By reduced\-rank regression \([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\),X∗=\[W0​L\]r​L−1X^\{\*\}=\[W\_\{0\}L\]\_\{r\}L^\{\-1\}, unique under the gap, and\[W0​L\]r=Πr​W0​L\[W\_\{0\}L\]\_\{r\}=\\Pi\_\{r\}W\_\{0\}LwithΠr\\Pi\_\{r\}the top\-rreigenprojector ofW0​L​L⊤​W0⊤=W0​C​W0⊤W\_\{0\}LL^\{\\top\}W\_\{0\}^\{\\top\}=W\_\{0\}CW\_\{0\}^\{\\top\}; soX∗=Πr​W0X^\{\*\}=\\Pi\_\{r\}W\_\{0\}, which does not depend on the choice ofLL\. Under\(g,h\)\(g,h\)the output second moment becomesg​W0​h−1​h​C​h⊤​h−⁣⊤​W0⊤​g⊤=g​W0​C​W0⊤​g⊤gW\_\{0\}h^\{\-1\}\\,hCh^\{\\top\}\\,h^\{\-\\top\}W\_\{0\}^\{\\top\}g^\{\\top\}=gW\_\{0\}CW\_\{0\}^\{\\top\}g^\{\\top\}, and with the square rooth​LhLofh​C​h⊤hCh^\{\\top\},

X∗​\(g​W0​h−1,h​C​h⊤\)=\[g​W0​L\]r​L−1​h−1\.X^\{\*\}\(gW\_\{0\}h^\{\-1\},hCh^\{\\top\}\)=\[gW\_\{0\}L\]\_\{r\}\\,L^\{\-1\}h^\{\-1\}\.This equalsg​X∗​h−1=g​\[W0​L\]r​L−1​h−1gX^\{\*\}h^\{\-1\}=g\[W\_\{0\}L\]\_\{r\}L^\{\-1\}h^\{\-1\}for allW0W\_\{0\}\(with gaps\) iff\[g​M\]r=g​\[M\]r\[gM\]\_\{r\}=g\[M\]\_\{r\}for allM=W0​LM=W\_\{0\}L, which range over all matrices with a gap\. By the lemma withk=Ik=Ithis holds iffg∈CO⁡\(m\)g\\in\\mathrm\{CO\}\(m\), with no condition onhh\. So the joint group isCO⁡\(m\)×GLn\\mathrm\{CO\}\(m\)\\times\\mathrm\{GL\}\_\{n\}, and its input and output parts areGLn\\mathrm\{GL\}\_\{n\}andCO⁡\(m\)\\mathrm\{CO\}\(m\)\.

*\(b\), undamped\.*For IPM,X⁡\(W,C\)=\[W​C\]r​C−1X\(W,C\)=\[WC\]\_\{r\}C^\{\-1\}minimises∥\(W−X\)​C∥F\\lVert\(W\-X\)C\\rVert\_\{F\}\([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\) withL=CL=C\)\. Under\(g,h\)\(g,h\), withM=W0​CM=W\_\{0\}C,

X⁡\(g​W0​h−1,h​C​h⊤\)=\[g​W0​C​h⊤\]r​h−⁣⊤​C−1​h−1=\[g​M​h⊤\]r​h−⁣⊤​C−1​h−1,X\(gW\_\{0\}h^\{\-1\},hCh^\{\\top\}\)=\[gW\_\{0\}Ch^\{\\top\}\]\_\{r\}h^\{\-\\top\}C^\{\-1\}h^\{\-1\}=\[gMh^\{\\top\}\]\_\{r\}\\,h^\{\-\\top\}C^\{\-1\}h^\{\-1\},and covariance asks this to equalg​\[M\]r​C−1​h−1g\[M\]\_\{r\}C^\{\-1\}h^\{\-1\}, that is\[g​M​h⊤\]r=g​\[M\]r​h⊤\[gMh^\{\\top\}\]\_\{r\}=g\[M\]\_\{r\}h^\{\\top\}\. AsW0W\_\{0\}varies,M=W0​CM=W\_\{0\}Cranges over all matrices, so by the lemma covariance holds iffg∈CO⁡\(m\)g\\in\\mathrm\{CO\}\(m\)andh⊤∈CO⁡\(n\)h^\{\\top\}\\in\\mathrm\{CO\}\(n\), that ish∈CO⁡\(n\)h\\in\\mathrm\{CO\}\(n\)\. KPM:X=⟨W0​C⟩r​C−1=W0−\[W0​C\]p−r​C−1X=\\langle W\_\{0\}C\\rangle\_\{r\}C^\{\-1\}=W\_\{0\}\-\[W\_\{0\}C\]\_\{p\-r\}C^\{\-1\}, and the same computation withs=p−r∈\[1,p−1\]s=p\-r\\in\[1,p\-1\]gives the condition\[g​M​h⊤\]p−r=g​\[M\]p−r​h⊤\[gMh^\{\\top\}\]\_\{p\-r\}=g\[M\]\_\{p\-r\}h^\{\\top\}, with the gapσp−r\>σp−r\+1\\sigma\_\{p\-r\}\>\\sigma\_\{p\-r\+1\}\. PiSSA:\[g​W0​h−1\]r=g​\[W0\]r​h−1\[gW\_\{0\}h^\{\-1\}\]\_\{r\}=g\[W\_\{0\}\]\_\{r\}h^\{\-1\}for allW0W\_\{0\}iffg∈CO⁡\(m\)g\\in\\mathrm\{CO\}\(m\)andh−1∈CO⁡\(n\)h^\{\-1\}\\in\\mathrm\{CO\}\(n\); MiLoRA:⟨W⟩r=W−\[W\]p−r\\langle W\\rangle\_\{r\}=W\-\[W\]\_\{p\-r\}reduces to the lemma ats=p−rs=p\-r\. In each case the input\-side group \(withg=Ig=I\) isCO⁡\(n\)\\mathrm\{CO\}\(n\)and the joint group isCO⁡\(m\)×CO⁡\(n\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)\.

*\(b\), trace\-relative damping\.*WriteCκ=C\+κ⁡\(tr⁡C/n\)​IC\_\{\\kappa\}=C\+\\kappa\(\\tr C/n\)Iwithκ\>0\\kappa\>0and considerXκ​\(W,C\)=X⁡\(W,Cκ\)X\_\{\\kappa\}\(W,C\)=X\(W,C\_\{\\kappa\}\)for IPM \(KPM is identical withs=p−rs=p\-r\)\. Forh=c​Q∈CO⁡\(n\)h=cQ\\in\\mathrm\{CO\}\(n\),h​Cκ​h⊤=c2​Q​C​Q⊤\+κ⁡\(tr⁡C/n\)​c2​I=\(h​C​h⊤\)κhC\_\{\\kappa\}h^\{\\top\}=c^\{2\}QCQ^\{\\top\}\+\\kappa\(\\tr C/n\)c^\{2\}I=\(hCh^\{\\top\}\)\_\{\\kappa\}, sincetr⁡\(c2​Q​C​Q⊤\)=c2​tr⁡C\\tr\(c^\{2\}QCQ^\{\\top\}\)=c^\{2\}\\tr C; so damping commutes withCO⁡\(n\)\\mathrm\{CO\}\(n\)and the damped splits inherit the covariance of the undamped ones\. Conversely, let\(g,h\)\(g,h\)be in the joint group, putS1=\(h​C​h⊤\)κS\_\{1\}=\(hCh^\{\\top\}\)\_\{\\kappa\},S2=CκS\_\{2\}=C\_\{\\kappa\}andN=W0​h−1N=W\_\{0\}h^\{\-1\}\. Covariance reads\[g​N​S1\]r​S1−1=g​\[N​h​S2\]r​S2−1​h−1\[gNS\_\{1\}\]\_\{r\}S\_\{1\}^\{\-1\}=g\[NhS\_\{2\}\]\_\{r\}S\_\{2\}^\{\-1\}h^\{\-1\}\. WithM=N​S1M=NS\_\{1\}, which ranges over all matrices, andG=S1−1​h​S2G=S\_\{1\}^\{\-1\}hS\_\{2\}, this becomes\[g​M\]r​G=g​\[M​G\]r\[gM\]\_\{r\}G=g\[MG\]\_\{r\}, or\[g​M′​G−1\]r=g​\[M′\]r​G−1\[gM^\{\\prime\}G^\{\-1\}\]\_\{r\}=g\[M^\{\\prime\}\]\_\{r\}G^\{\-1\}withM′=M​GM^\{\\prime\}=MG\. By the lemma,g∈CO⁡\(m\)g\\in\\mathrm\{CO\}\(m\)andG∈CO⁡\(n\)G\\in\\mathrm\{CO\}\(n\)for everyC≻0C\\succ 0, that is,

h​S22​h⊤=c​\(C\)2​S12for every​C≻0\.h\\,S\_\{2\}^\{2\}\\,h^\{\\top\}=c\(C\)^\{2\}\\,S\_\{1\}^\{2\}\\qquad\\text\{for every \}C\\succ 0\.Both sides are continuous inCC, andc​\(C\)2=tr⁡\(h​S22​h⊤\)/tr⁡\(S12\)c\(C\)^\{2\}=\\tr\(hS\_\{2\}^\{2\}h^\{\\top\}\)/\\tr\(S\_\{1\}^\{2\}\)is too, so the identity extends toC=v​v⊤C=vv^\{\\top\}withv≠0v\\neq 0\. There, withw=h​vw=hv,a=κ​∥v∥2/n\>0a=\\kappa\\lVert v\\rVert^\{2\}/n\>0andb=κ​∥w∥2/nb=\\kappa\\lVert w\\rVert^\{2\}/n,S22=\(∥v∥2\+2​a\)​v​v⊤\+a2​IS\_\{2\}^\{2\}=\(\\lVert v\\rVert^\{2\}\+2a\)vv^\{\\top\}\+a^\{2\}IandS12=\(∥w∥2\+2​b\)​w​w⊤\+b2​IS\_\{1\}^\{2\}=\(\\lVert w\\rVert^\{2\}\+2b\)ww^\{\\top\}\+b^\{2\}I, so

h​h⊤=αw​I\+βw​w​w⊤\(αw=c2​b2/a2\),hh^\{\\top\}=\\alpha\_\{w\}I\+\\beta\_\{w\}\\,ww^\{\\top\}\\qquad\(\\alpha\_\{w\}=c^\{2\}b^\{2\}/a^\{2\}\),for everyw∈ℝnw\\in\\mathbb\{R\}^\{n\}, sincew=h​vw=hvranges over all vectors\. Soh​h⊤−αw​Ihh^\{\\top\}\-\\alpha\_\{w\}Ihas rank at most one for everyww\. Ifh​h⊤hh^\{\\top\}were not scalar, it would have an eigenvalueα\\alphaof multiplicityn−1n\-1withh​h⊤=α​I\+γ​e​e⊤hh^\{\\top\}=\\alpha I\+\\gamma ee^\{\\top\},γ≠0\\gamma\\neq 0; takingwwneither parallel nor orthogonal toee, the matrix\(α−αw\)​I\+γ​e​e⊤\(\\alpha\-\\alpha\_\{w\}\)I\+\\gamma ee^\{\\top\}would have rank at most one with range alongww, which is impossible \(forn≥3n\\geq 3it has rank at leastn−1≥2n\-1\\geq 2unlessαw=α\\alpha\_\{w\}=\\alpha, and then its range isℝ​e≠ℝ​w\\mathbb\{R\}e\\neq\\mathbb\{R\}w; forn=2n=2its range isℝ​e\\mathbb\{R\}eore⟂e^\{\\perp\}\)\. Soh​h⊤hh^\{\\top\}is scalar andh∈CO⁡\(n\)h\\in\\mathrm\{CO\}\(n\)\. The input\-side group of the damped splits is therefore exactlyCO⁡\(n\)\\mathrm\{CO\}\(n\), and the joint groupCO⁡\(m\)×CO⁡\(n\)\\mathrm\{CO\}\(m\)\\times\\mathrm\{CO\}\(n\)\.

*\(c\), absolute damping\.*Q⁡\(C\+λ​I\)​Q⊤=Q​C​Q⊤\+λ​IQ\(C\+\\lambda I\)Q^\{\\top\}=QCQ^\{\\top\}\+\\lambda IforQ∈O⁡\(n\)Q\\in\\mathrm\{O\}\(n\), so the damped splits of \(a\) and \(b\) areO⁡\(n\)\\mathrm\{O\}\(n\)\-covariant\. We show that no otherhhis in their input\-side groups\.

For \(b\): the undamped splits are invariant underC↦t​CC\\mapsto tC, so the damped split at\(W,t​C\)\(W,tC\)equals the undamped split at\(W,C\+\(λ/t\)​I\)\(W,C\+\(\\lambda/t\)I\), which tends to the undamped split at\(W,C\)\(W,C\)ast→∞t\\to\\inftywherever the gap holds, because a truncation is continuous where it is unique \(it is the unique minimiser of a continuous function over the closed setℳ≤s\\mathcal\{M\}\_\{\\leq s\}\)\. The same holds at the transformed data\(W​h−1,t​h​C​h⊤\)\(Wh^\{\-1\},thCh^\{\\top\}\)\. So everyhhin the damped group lies in the undamped groupCO⁡\(n\)\\mathrm\{CO\}\(n\)\.

For \(a\): withS=\(h⊤​h\)−1S=\(h^\{\\top\}h\)^\{\-1\}, the transformed damped moment isW0​h−1​\(h​C​h⊤\+λ​I\)​h−⁣⊤​W0⊤=W0​\(C\+λ​S\)​W0⊤W\_\{0\}h^\{\-1\}\(hCh^\{\\top\}\+\\lambda I\)h^\{\-\\top\}W\_\{0\}^\{\\top\}=W\_\{0\}\(C\+\\lambda S\)W\_\{0\}^\{\\top\}, so covariance readsΠr​\(W0​\(C\+λ​S\)​W0⊤\)​W0=Πr​\(W0​\(C\+λ​I\)​W0⊤\)​W0\\Pi\_\{r\}\\big\(W\_\{0\}\(C\+\\lambda S\)W\_\{0\}^\{\\top\}\\big\)W\_\{0\}=\\Pi\_\{r\}\\big\(W\_\{0\}\(C\+\\lambda I\)W\_\{0\}^\{\\top\}\\big\)W\_\{0\}\. Both projectors have range incol⁡W0\\operatorname\{col\}W\_\{0\}, onto whichW0W\_\{0\}maps, so this is equality of the projectors\. ReplacingCCbyt​CtCand lettingt→0t\\to 0givesΠr​\(W0​S​W0⊤\)=Πr​\(W0​W0⊤\)\\Pi\_\{r\}\(W\_\{0\}SW\_\{0\}^\{\\top\}\)=\\Pi\_\{r\}\(W\_\{0\}W\_\{0\}^\{\\top\}\)for everyW0W\_\{0\}for which both have a gap atrr\. SupposeSSis not scalar, with orthonormal eigenvectorse,fe,ffor eigenvaluesα\>β\\alpha\>\\beta, and complete them to an orthonormal eigenbasise,f,g3,…,gne,f,g\_\{3\},\\dots,g\_\{n\}ofSS\. Choose orthonormalz1,…,zpz\_\{1\},\\dots,z\_\{p\}withzr=cos⁡ϕ​e\+sin⁡ϕ​fz\_\{r\}=\\cos\\phi\\,e\+\\sin\\phi\\,f,zr\+1=−sin⁡ϕ​e\+cos⁡ϕ​fz\_\{r\+1\}=\-\\sin\\phi\\,e\+\\cos\\phi\\,ffor someϕ∉π2​ℤ\\phi\\notin\\tfrac\{\\pi\}\{2\}\\mathbb\{Z\}, and the otherziz\_\{i\}among thegjg\_\{j\}; putW0=∑i≤pdi​ei​zi⊤W\_\{0\}=\\sum\_\{i\\leq p\}d\_\{i\}e\_\{i\}z\_\{i\}^\{\\top\}withd1\>⋯\>dp\>0d\_\{1\}\>\\dots\>d\_\{p\}\>0\. ThenW0​W0⊤=diag⁡\(d12,…,dp2\)W\_\{0\}W\_\{0\}^\{\\top\}=\\diag\(d\_\{1\}^\{2\},\\dots,d\_\{p\}^\{2\}\)\(padded\), with top\-rreigenspacespan⁡\{e1,…,er\}\\spanop\\\{e\_\{1\},\\dots,e\_\{r\}\\\}, whileW0​S​W0⊤W\_\{0\}SW\_\{0\}^\{\\top\}is diagonal except for the entrydr​dr\+1​\(β−α\)​sin⁡ϕ​cos⁡ϕ≠0d\_\{r\}d\_\{r\+1\}\(\\beta\-\\alpha\)\\sin\\phi\\cos\\phi\\neq 0in positions\(r,r\+1\)\(r,r\+1\)and\(r\+1,r\)\(r\+1,r\)\. Takingdid\_\{i\}fori<ri<rmuch larger anddid\_\{i\}fori\>r\+1i\>r\+1much smaller thandr,dr\+1d\_\{r\},d\_\{r\+1\}, the top\-rreigenspace ofW0​S​W0⊤W\_\{0\}SW\_\{0\}^\{\\top\}isspan⁡\{e1,…,er−1\}\\spanop\\\{e\_\{1\},\\dots,e\_\{r\-1\}\\\}plus the top eigenvector of the2×22\\times 2block, which is notere\_\{r\}; both matrices have a gap atrr, and the projectors differ\. HenceSSis scalar andh∈CO⁡\(n\)h\\in\\mathrm\{CO\}\(n\)\.

It remains to excludeh=c​Qh=cQwithc≠1c\\neq 1; sinceQQis in the group, it suffices to excludeh=c​Ih=cI\. For \(a\),h=c​Ih=cIturns the condition intoΠr​\(W0​\(C\+λ​c−2​I\)​W0⊤\)=Πr​\(W0​\(C\+λ​I\)​W0⊤\)\\Pi\_\{r\}\(W\_\{0\}\(C\+\\lambda c^\{\-2\}I\)W\_\{0\}^\{\\top\}\)=\\Pi\_\{r\}\(W\_\{0\}\(C\+\\lambda I\)W\_\{0\}^\{\\top\}\)\. ReplacingCCbyt​CtC, the functionP⁡\(μ\)=Πr​\(W0​C​W0⊤\+μ​W0​W0⊤\)P\(\\mu\)=\\Pi\_\{r\}\(W\_\{0\}CW\_\{0\}^\{\\top\}\+\\mu W\_\{0\}W\_\{0\}^\{\\top\}\)would satisfyP⁡\(μ\)=P⁡\(c−2​μ\)P\(\\mu\)=P\(c^\{\-2\}\\mu\)for allμ\>0\\mu\>0, henceP⁡\(μ\)=P⁡\(c−2​j​μ\)P\(\\mu\)=P\(c^\{\-2j\}\\mu\)for allj∈ℤj\\in\\mathbb\{Z\}, and lettingj→±∞j\\to\\pm\\inftygivesP⁡\(0\)=P⁡\(∞\)P\(0\)=P\(\\infty\), the top\-rreigenprojectors ofW0​C​W0⊤W\_\{0\}CW\_\{0\}^\{\\top\}and ofW0​W0⊤W\_\{0\}W\_\{0\}^\{\\top\}\(both limits exist under gaps\)\. TakeW0W\_\{0\}rectangular\-diagonal with entriesd1\>⋯\>dp\>0d\_\{1\}\>\\dots\>d\_\{p\}\>0andC=diag⁡\(ci\)C=\\diag\(c\_\{i\}\)withci=i/di2c\_\{i\}=i/d\_\{i\}^\{2\}fori≤pi\\leq pandci=1c\_\{i\}=1beyond\. ThenW0​W0⊤W\_\{0\}W\_\{0\}^\{\\top\}has top\-rreigenspacespan⁡\{e1,…,er\}\\spanop\\\{e\_\{1\},\\dots,e\_\{r\}\\\}andW0​C​W0⊤=diag⁡\(1,2,…,p\)W\_\{0\}CW\_\{0\}^\{\\top\}=\\diag\(1,2,\\dots,p\)\(padded\) has top\-rreigenspacespan⁡\{ep−r\+1,…,ep\}\\spanop\\\{e\_\{p\-r\+1\},\\dots,e\_\{p\}\\\}, which differ sincer<pr<p\. For \(b\),h=c​Ih=cIgives, by scale invariance and linearity inWW,X⁡\(W0,C\+λ​c−2​I\)=X⁡\(W0,C\+λ​I\)X\(W\_\{0\},C\+\\lambda c^\{\-2\}I\)=X\(W\_\{0\},C\+\\lambda I\), and the same iteration applied toG⁡\(μ\)=X⁡\(W0,C\+μ​I\)G\(\\mu\)=X\(W\_\{0\},C\+\\mu I\)givesG⁡\(0\)=G⁡\(∞\)G\(0\)=G\(\\infty\)withG⁡\(0\)=X⁡\(W0,C\)G\(0\)=X\(W\_\{0\},C\)andG⁡\(∞\)=X⁡\(W0,I\)G\(\\infty\)=X\(W\_\{0\},I\)\. With the sameW0W\_\{0\}andCC,W0​CW\_\{0\}Cis rectangular\-diagonal with the increasing entriesi/dii/d\_\{i\}, so for IPMG⁡\(0\)=∑i\>p−rdi​Ei​iG\(0\)=\\sum\_\{i\>p\-r\}d\_\{i\}E\_\{ii\}whileG⁡\(∞\)=\[W0\]r=∑i≤rdi​Ei​iG\(\\infty\)=\[W\_\{0\}\]\_\{r\}=\\sum\_\{i\\leq r\}d\_\{i\}E\_\{ii\}; for KPM the roles of top and bottom are exchanged, with the same conclusion\. Soc​IcIis not in the group, and the damped input\-side groups are exactlyO⁡\(n\)\\mathrm\{O\}\(n\)\.

*\(c\), per\-sample normalisation\.*The shipped code divides a samplexxbyν⁡\(x\)=\|maxj⁡xj\|\\nu\(x\)=\\lvert\\max\_\{j\}x\_\{j\}\\rvert\. Ifh=c​Ph=cPwithc\>0c\>0andPPa permutation matrix,ν⁡\(h​x\)=c​ν​\(x\)\\nu\(hx\)=c\\,\\nu\(x\), so the accumulatedCCtransforms ash​C​h⊤hCh^\{\\top\}up to the factorc−2c^\{\-2\}, to which the splits are insensitive, andh∈CO⁡\(n\)h\\in\\mathrm\{CO\}\(n\)\. Conversely, supposeν⁡\(h​x\)=c​ν​\(x\)\\nu\(hx\)=c\\,\\nu\(x\)for allxxand somec\>0c\>0\(n≥2n\\geq 2\)\. Forx=−ejx=\-e\_\{j\},ν⁡\(x\)=0\\nu\(x\)=0, somaxi⁡\(−hi​j\)=0\\max\_\{i\}\(\-h\_\{ij\}\)=0andh≥0h\\geq 0entrywise; forx=ejx=e\_\{j\},maxi⁡hi​j=c\\max\_\{i\}h\_\{ij\}=c; forx=𝟏x=\\mathbf\{1\}, every row sum ofhhis at mostcc\. So a rowiji\_\{j\}withhij​j=ch\_\{i\_\{j\}j\}=chas no other nonzero entry, the rowsiji\_\{j\}are distinct, andh=c​Ph=cP\. Under a rotation the normalisation does not commute withhh, and the shipped split is not covariant\. For instance, withn=m=2n=m=2,r=1r=1, samplesx1=e1x\_\{1\}=e\_\{1\}andx2=e1\+e2x\_\{2\}=e\_\{1\}\+e\_\{2\}\(soν=1\\nu=1for both andC=x1​x1⊤\+x2​x2⊤=\(2111\)C=x\_\{1\}x\_\{1\}^\{\\top\}\+x\_\{2\}x\_\{2\}^\{\\top\}=\\left\(\\begin\{smallmatrix\}2&1\\\\ 1&1\\end\{smallmatrix\}\\right\), up to a factor\), andh=Qh=Qthe rotation byπ/4\\pi/4: the rotated samples are\(1,1\)/2\(1,1\)/\\sqrt\{2\}and\(0,2\)\(0,\\sqrt\{2\}\), normalised to\(1,1\)\(1,1\)and\(0,1\)\(0,1\), so the shipped moment isC′=\(1112\)C^\{\\prime\}=\\left\(\\begin\{smallmatrix\}1&1\\\\ 1&2\\end\{smallmatrix\}\\right\), whileQ​C​Q⊤=12​\(1115\)QCQ^\{\\top\}=\\tfrac\{1\}\{2\}\\left\(\\begin\{smallmatrix\}1&1\\\\ 1&5\\end\{smallmatrix\}\\right\)\. WithW0=QW\_\{0\}=Q, the shipped IPM split at the rotated data isX⁡\(I,C′\)=\[C′\]1​C′−1X\(I,C^\{\\prime\}\)=\[C^\{\\prime\}\]\_\{1\}C^\{\\prime\-1\}, the top eigenprojector ofC′C^\{\\prime\}, while covariance would requireX⁡\(Q,C\)​Q⊤=Q​\[C\]1​C−1​Q⊤X\(Q,C\)Q^\{\\top\}=Q\[C\]\_\{1\}C^\{\-1\}Q^\{\\top\}, the top eigenprojector ofQ​C​Q⊤QCQ^\{\\top\}\. The two symmetric matrices do not commute, so their top eigenvectors differ, and covariance fails\. ∎

### C\.24 Definition III\.1

Definition III\.1\(trained method; cometric; three equivalences\)\.A*trained method*is a method together with optimizer data: a learning\-rate operatorΛ≻0\\Lambda\\succ 0\(per\-block rates\), an initial scale, and an update rule\. Gradient flowq˙=−Λ​∇q\(L∘ρ\)\\dot\{q\}=\-\\Lambda\\nabla\_\{q\}\(L\\circ\\rho\)moves the weights byθ˙=−Kq​∇θL,Kq=D​ρq​Λ​D​ρq⊤\.\\dot\{\\theta\}=\-K\_\{q\}\\nabla\_\{\\theta\}L,\\qquad K\_\{q\}=D\\rho\_\{q\}\\,\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\.We callKqK\_\{q\}theinduced cometric\. For a backward method with compressionctc\_\{t\}and anchorata\_\{t\}\([Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\),Kt=at​ctK\_\{t\}=a\_\{t\}c\_\{t\}\. Two methods are*expressively equivalent*if their images agree,*first\-order equivalent*if theirKq0K\_\{q\_\{0\}\}agree, and*dynamically equivalent*if they produce the sameθ\\theta\-trajectory for every loss\.

A definition needs no proof\. This subsection fixes the conventions used in all proofs of Exposé III and checks the two identities that the definition records\.

##### Conventions

Weight spaceΘ=ℝm×n\\Theta=\\mathbb\{R\}^\{m\\times n\}carries the Frobenius inner product⟨X,Y⟩=tr⁡\(X⊤​Y\)\\langle X,Y\\rangle=\\tr\(X^\{\\top\}Y\), and every parameter space carries its Euclidean \(Frobenius\) inner product\. Gradients are identified with covectors through these inner products, and the transpose of a linear map is its adjoint\. A covector onΘ\\Thetais written as a matrix, usuallyGG\. WhenG=∇θLG=\\nabla\_\{\\theta\}Lis a weight gradient we say so; otherwiseGGis an arbitrary cotangent signal\.

LoRA isρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBAwithB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\},A∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\}ands\>0s\>0\. Its differential and the adjoint of the differential are

D​ρ\(B,A\)​\(B˙,A˙\)=s⁡\(B˙​A\+B​A˙\),D​ρ\(B,A\)⊤​\(G\)=\(s​G​A⊤,s​B⊤​G\),D\\rho\_\{\(B,A\)\}\(\\dot\{B\},\\dot\{A\}\)=s\(\\dot\{B\}A\+B\\dot\{A\}\),\\qquad D\\rho\_\{\(B,A\)\}^\{\\top\}\(G\)=\(sGA^\{\\top\},\\ sB^\{\\top\}G\),\(19\)since⟨G,s⁡\(B˙​A\+B​A˙\)⟩=⟨s​G​A⊤,B˙⟩\+⟨s​B⊤​G,A˙⟩\\langle G,s\(\\dot\{B\}A\+B\\dot\{A\}\)\\rangle=\\langle sGA^\{\\top\},\\dot\{B\}\\rangle\+\\langle sB^\{\\top\}G,\\dot\{A\}\\rangle\. WithG=∇θL​\(ρ⁡\(B,A\)\)G=\\nabla\_\{\\theta\}L\(\\rho\(B,A\)\)this gives∇B\(L∘ρ\)=s​G​A⊤\\nabla\_\{B\}\(L\\circ\\rho\)=sGA^\{\\top\}and∇A\(L∘ρ\)=s​B⊤​G\\nabla\_\{A\}\(L\\circ\\rho\)=sB^\{\\top\}G, written∇B\\nabla\_\{B\}and∇A\\nabla\_\{A\}\.*Block\-scalar rates*areΛ⁡\(B,A\)=\(ηB​B,ηA​A\)\\Lambda\(B,A\)=\(\\eta\_\{B\}B,\\eta\_\{A\}A\)withηA,ηB\>0\\eta\_\{A\},\\eta\_\{B\}\>0\.

The gauge groupGLr\\mathrm\{GL\}\_\{r\}acts byg⋅\(B,A\)=\(B​g−1,g​A\)g\\cdot\(B,A\)=\(Bg^\{\-1\},gA\), andρ⁡\(g⋅q\)=ρ⁡\(q\)\\rho\(g\\cdot q\)=\\rho\(q\)\([Background D\.21](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3)\)\. Atq′=g⋅qq^\{\\prime\}=g\\cdot qthe weight is unchanged, hence so is the weight gradientGG, and \([19](https://arxiv.org/html/2610.12728#A3.E19)\) gives

∇B′=s​G​A⊤​g⊤=∇Bg⊤,∇A′=s​g−⁣⊤​B⊤​G=g−⁣⊤​∇A\.\\nabla\_\{B^\{\\prime\}\}=sGA^\{\\top\}g^\{\\top\}=\\nabla\_\{B\}g^\{\\top\},\\qquad\\nabla\_\{A^\{\\prime\}\}=sg^\{\-\\top\}B^\{\\top\}G=g^\{\-\\top\}\\nabla\_\{A\}\.\(20\)The*full\-rank locus*is\{rankB=rankA=r\}\\\{\\rk B=\\rk A=r\\\}\. ThereB​ABAhas rankrr,U:=col⁡B=col⁡\(B​A\)U:=\\operatorname\{col\}B=\\operatorname\{col\}\(BA\)andV:=row⁡A=row⁡\(B​A\)V:=\\operatorname\{row\}A=\\operatorname\{row\}\(BA\), and the orthogonal projectors onto them arePU=B​\(B⊤​B\)−1​B⊤P\_\{U\}=B\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}andPV=A⊤​\(A​A⊤\)−1​AP\_\{V\}=A^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}A\.

An update rule is*equivariant*undergg\([Background D\.18](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\) if, for every sequence of losses \(the same data order and dropout masks in both runs\), the run started fromg⋅q0g\\cdot q\_\{0\}with correspondingly transformed optimizer state producesg⋅qtg\\cdot q\_\{t\}at every steptt\. Equivariance gives identical weight trajectories, sinceρ⁡\(g⋅qt\)=ρ⁡\(qt\)\\rho\(g\\cdot q\_\{t\}\)=\\rho\(q\_\{t\}\)\. It is checked one step at a time\. Ifqt′=g⋅qtq\_\{t\}^\{\\prime\}=g\\cdot q\_\{t\}, the two runs see the same weight and the same loss, hence the sameGG, and their gradients are related by \([20](https://arxiv.org/html/2610.12728#A3.E20)\)\. It then suffices that the steps are transported,

δ​B′=δ​B​g−1,δ​A′=g​δ​A,\\delta B^\{\\prime\}=\\delta B\\,g^\{\-1\},\\qquad\\delta A^\{\\prime\}=g\\,\\delta A,\(21\)together with the optimizer state, and induction overttconcludes\.

By the chain rule∇q\(L∘ρ\)=D​ρq⊤​∇θL\\nabla\_\{q\}\(L\\circ\\rho\)=D\\rho\_\{q\}^\{\\top\}\\nabla\_\{\\theta\}L, so gradient flow moves the weights byθ˙=D​ρq​q˙=−D​ρq​Λ​D​ρq⊤​∇θL=−Kq​∇θL\\dot\{\\theta\}=D\\rho\_\{q\}\\dot\{q\}=\-D\\rho\_\{q\}\\Lambda D\\rho\_\{q\}^\{\\top\}\\nabla\_\{\\theta\}L=\-K\_\{q\}\\nabla\_\{\\theta\}L\. The operatorKqK\_\{q\}is symmetric, and⟨γ,Kq​γ⟩=⟨D​ρq⊤​γ,Λ​D​ρq⊤​γ⟩≥0\\langle\\gamma,K\_\{q\}\\gamma\\rangle=\\langle D\\rho\_\{q\}^\{\\top\}\\gamma,\\Lambda D\\rho\_\{q\}^\{\\top\}\\gamma\\rangle\\geq 0\. SinceΛ≻0\\Lambda\\succ 0,ker⁡Kq=ker⁡D​ρq⊤\\ker K\_\{q\}=\\ker D\\rho\_\{q\}^\{\\top\}, sorank⁡Kq=rank⁡D​ρq≤dimQ\\rk K\_\{q\}=\\rk D\\rho\_\{q\}\\leq\\dim Q\. ThusKqK\_\{q\}is a cometric in the sense of[Background D\.23](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3), possibly degenerate\.

A backward method updatesθ←θ−η​at​\(opt⁡\(ct​\(∇L\)\)\)\\theta\\leftarrow\\theta\-\\eta\\,a\_\{t\}\(\\mathrm\{opt\}\(c\_\{t\}\(\\nabla L\)\)\)\([Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)\)\. Whenopt\\mathrm\{opt\}is the identity, that is, plain \(S\)GD on the compressed gradient, andη\\etais absorbed intoata\_\{t\}, the continuous\-time update isθ˙=−at​ct​∇θL\\dot\{\\theta\}=\-a\_\{t\}c\_\{t\}\\nabla\_\{\\theta\}L, which is the statementKt=at​ctK\_\{t\}=a\_\{t\}c\_\{t\}\. This operator is symmetric positive semidefinite whenat=ct⊤a\_\{t\}=c\_\{t\}^\{\\top\}, as for GaLore \(Pt​Pt⊤P\_\{t\}P\_\{t\}^\{\\top\}\), LISA \(a coordinate projection\) and MeZO \(zt​zt⊤z\_\{t\}z\_\{t\}^\{\\top\}\)\.

Finally, for two methods trained by gradient flow, dynamical equivalence implies first\-order equivalence\. For a linear lossL⁡\(θ\)=⟨γ,θ⟩L\(\\theta\)=\\langle\\gamma,\\theta\\ranglethe initial weight velocities are−Kq0​γ\-K\_\{q\_\{0\}\}\\gammaand−Kq0′′​γ\-K^\{\\prime\}\_\{q\_\{0\}^\{\\prime\}\}\\gamma, andγ\\gammais arbitrary\. ∎

### C\.25 Observation III\.2

Observation III\.2\(chain rule for cometrics\)\.Letσ:Q′→Q\\sigma:Q^\{\\prime\}\\to Qandρ:Q→X\\rho:Q\\to Xbe smooth, withX=ΘX=\\Thetaor a function or output space\. TrainQ′Q^\{\\prime\}by gradient flow with a constant rate operatorΛ′≻0\\Lambda^\{\\prime\}\\succ 0, and putKq′σ:=D​σq′​Λ′​D​σq′⊤K^\{\\sigma\}\_\{q^\{\\prime\}\}:=D\\sigma\_\{q^\{\\prime\}\}\\Lambda^\{\\prime\}D\\sigma\_\{q^\{\\prime\}\}^\{\\top\}\. ThenKq′ρ​σ=D​ρσ⁡\(q′\)​Kq′σ​D​ρσ⁡\(q′\)⊤,K^\{\\rho\\sigma\}\_\{q^\{\\prime\}\}=D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}\\,K^\{\\sigma\}\_\{q^\{\\prime\}\}\\,D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}^\{\\top\},and no metric onQQis used\.KKis a degenerate cometric \(positive semidefinite, of rank≤dimQ′\\leq\\dim Q^\{\\prime\}\); whenσ\\sigmais not injective,KσK^\{\\sigma\}is a field alongσ\\sigmaand fails to be a cometric onQQ\.For LoRA,ρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBAwiths=α/rs=\\alpha/r\(orα/r\\alpha/\\sqrt\{r\}for rsLoRA\), with block ratesΛ=diag⁡\(ηB​I,ηA​I\)\\Lambda=\\diag\(\\eta\_\{B\}I,\\eta\_\{A\}I\)and no LoRA dropout, under gradient flow \(equivalently, to first order in the step size for GD\),K\(B,A\)​\(G\)=s2​\(ηB​G​A⊤​A\+ηA​B​B⊤​G\)\.K\_\{\(B,A\)\}\(G\)=s^\{2\}\\big\(\\eta\_\{B\}\\,GA^\{\\top\}A\+\\eta\_\{A\}\\,BB^\{\\top\}G\\big\)\.Under Adam there is no state\-independent linear cometric;[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)and[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)treat Adam\.

*Chain rule\.*By definitionKq′ρ​σ=D​\(ρ​σ\)q′​Λ′​D​\(ρ​σ\)q′⊤K^\{\\rho\\sigma\}\_\{q^\{\\prime\}\}=D\(\\rho\\sigma\)\_\{q^\{\\prime\}\}\\Lambda^\{\\prime\}D\(\\rho\\sigma\)\_\{q^\{\\prime\}\}^\{\\top\}, andD​\(ρ​σ\)q′=D​ρσ⁡\(q′\)​D​σq′D\(\\rho\\sigma\)\_\{q^\{\\prime\}\}=D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}D\\sigma\_\{q^\{\\prime\}\}\. Hence

Kq′ρ​σ=D​ρσ⁡\(q′\)​\(D​σq′​Λ′​D​σq′⊤\)​D​ρσ⁡\(q′\)⊤=D​ρσ⁡\(q′\)​Kq′σ​D​ρσ⁡\(q′\)⊤\.K^\{\\rho\\sigma\}\_\{q^\{\\prime\}\}=D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}\\big\(D\\sigma\_\{q^\{\\prime\}\}\\Lambda^\{\\prime\}D\\sigma\_\{q^\{\\prime\}\}^\{\\top\}\\big\)D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}^\{\\top\}=D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}K^\{\\sigma\}\_\{q^\{\\prime\}\}D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}^\{\\top\}\.Equivalently, the flowq˙′=−Λ′​∇q′\(L∘ρ∘σ\)=−Λ′​D​σ⊤​D​ρ⊤​d​L\\dot\{q\}^\{\\prime\}=\-\\Lambda^\{\\prime\}\\nabla\_\{q^\{\\prime\}\}\(L\\circ\\rho\\circ\\sigma\)=\-\\Lambda^\{\\prime\}D\\sigma^\{\\top\}D\\rho^\{\\top\}dLmovesx=ρ⁡\(σ⁡\(q′\)\)x=\\rho\(\\sigma\(q^\{\\prime\}\)\)byx˙=−D​ρ​Kσ​D​ρ⊤​d​L\\dot\{x\}=\-D\\rho\\,K^\{\\sigma\}D\\rho^\{\\top\}dL\. In this formulaD​ρσ⁡\(q′\)⊤:Tx∗​X→Tσ⁡\(q′\)∗​QD\\rho\_\{\\sigma\(q^\{\\prime\}\)\}^\{\\top\}:T^\{\*\}\_\{x\}X\\to T^\{\*\}\_\{\\sigma\(q^\{\\prime\}\)\}QandD​σq′⊤:Tσ⁡\(q′\)∗​Q→Tq′∗​Q′D\\sigma\_\{q^\{\\prime\}\}^\{\\top\}:T^\{\*\}\_\{\\sigma\(q^\{\\prime\}\)\}Q\\to T^\{\*\}\_\{q^\{\\prime\}\}Q^\{\\prime\}are dual maps, andKq′σ:Tσ⁡\(q′\)∗​Q→Tσ⁡\(q′\)​QK^\{\\sigma\}\_\{q^\{\\prime\}\}:T^\{\*\}\_\{\\sigma\(q^\{\\prime\}\)\}Q\\to T\_\{\\sigma\(q^\{\\prime\}\)\}Qis built fromΛ′\\Lambda^\{\\prime\}alone\. No inner product onQQenters\.

*Degeneracy\.*As in §[C\.24](https://arxiv.org/html/2610.12728#A3.SS24),Kq′ρ​σK^\{\\rho\\sigma\}\_\{q^\{\\prime\}\}is symmetric and positive semidefinite, and its rank is at mostrank⁡D​σq′≤dimQ′\\rk D\\sigma\_\{q^\{\\prime\}\}\\leq\\dim Q^\{\\prime\}\. The operatorKq′σK^\{\\sigma\}\_\{q^\{\\prime\}\}depends onq′q^\{\\prime\}and not only onσ⁡\(q′\)\\sigma\(q^\{\\prime\}\)\. For example, withσ⁡\(a,b\)=a​b\\sigma\(a,b\)=abfromℝ2\\mathbb\{R\}^\{2\}toℝ\\mathbb\{R\}andΛ′=I\\Lambda^\{\\prime\}=I,K\(a,b\)σ=a2\+b2K^\{\\sigma\}\_\{\(a,b\)\}=a^\{2\}\+b^\{2\}takes the value22at\(1,1\)\(1,1\)and17/417/4at\(2,1/2\)\(2,1/2\), two points over the same valueσ=1\\sigma=1\. So for non\-injectiveσ\\sigma,KσK^\{\\sigma\}is a field alongσ\\sigma\(a section ofσ∗​\(T​Q⊗T​Q\)\\sigma^\{\*\}\(TQ\\otimes TQ\)\) and need not descend toQQ\.

*LoRA\.*By \([19](https://arxiv.org/html/2610.12728#A3.E19)\),Λ​D​ρ⊤​\(G\)=\(ηB​s​G​A⊤,ηA​s​B⊤​G\)\\Lambda D\\rho^\{\\top\}\(G\)=\(\\eta\_\{B\}sGA^\{\\top\},\\eta\_\{A\}sB^\{\\top\}G\), and applyingD​ρD\\rhogives

K\(B,A\)​\(G\)=s⁡\(\(ηB​s​G​A⊤\)​A\+B⁡\(ηA​s​B⊤​G\)\)=s2​\(ηB​G​A⊤​A\+ηA​B​B⊤​G\)\.K\_\{\(B,A\)\}\(G\)=s\\big\(\(\\eta\_\{B\}sGA^\{\\top\}\)A\+B\(\\eta\_\{A\}sB^\{\\top\}G\)\\big\)=s^\{2\}\\big\(\\eta\_\{B\}GA^\{\\top\}A\+\\eta\_\{A\}BB^\{\\top\}G\\big\)\.A discrete GD stepδ​B=−ηB​s​G​A⊤\\delta B=\-\\eta\_\{B\}sGA^\{\\top\},δ​A=−ηA​s​B⊤​G\\delta A=\-\\eta\_\{A\}sB^\{\\top\}Gchanges the weight by

ρ⁡\(B\+δ​B,A\+δ​A\)−ρ⁡\(B,A\)=s⁡\(δ​B​A\+B​δ​A\)\+s​δ​B​δ​A=−K\(B,A\)​\(G\)\+s3​ηA​ηB​G​A⊤​B⊤​G,\\rho\(B\+\\delta B,A\+\\delta A\)\-\\rho\(B,A\)=s\(\\delta BA\+B\\delta A\)\+s\\,\\delta B\\,\\delta A=\-K\_\{\(B,A\)\}\(G\)\+s^\{3\}\\eta\_\{A\}\\eta\_\{B\}\\,GA^\{\\top\}B^\{\\top\}G,and the last term is of orderη2\\eta^\{2\}\. With LoRA dropout the factors are trained on masked inputs\. The same computation then givesK\(B,A\)K\_\{\(B,A\)\}applied to the masked signalG~\\tilde\{G\}\(for one example with output gradientδ\\delta, inputxxand mask𝗆\\mathsf\{m\},G~=δ​\(𝗆⊙x\)⊤\\tilde\{G\}=\\delta\(\\mathsf\{m\}\\odot x\)^\{\\top\}\), which differs from∇θL\\nabla\_\{\\theta\}L\. This is why the formula withG=∇θLG=\\nabla\_\{\\theta\}Lassumes no dropout\.

*Adam\.*Adam’s step depends on its moment buffers, so it is not a function ofqqandGGalone, and even the first step from zero state has no linear cometric\. TakeB=0B=0andAAof full row rank\. Then∇A=0\\nabla\_\{A\}=0, soAAdoes not move, and the bias\-corrected first step onBBisδ​B=−ηB​ϕ​\(s​G​A⊤\)\\delta B=\-\\eta\_\{B\}\\phi\(sGA^\{\\top\}\), withϕ⁡\(x\)=x/\(\|x\|\+ε\)\\phi\(x\)=x/\(\\lvert x\\rvert\+\\varepsilon\)entrywise and0/0=00/0=0\. \(Without bias correction the step is a positive multiple of the same expression with a rescaledε\\varepsilon\.\) The weight moves byδ​W=s​δ​B​A=−s​ηB​ϕ​\(s​G​A⊤\)​A\\delta W=s\\,\\delta B\\,A=\-s\\eta\_\{B\}\\phi\(sGA^\{\\top\}\)A, which is nonzero wheneverG​A⊤≠0GA^\{\\top\}\\neq 0, becauseAAhas full row rank\. Supposeδ​W=−K⁡\(G\)\\delta W=\-K\(G\)for a linear mapKK\. Atε=0\\varepsilon=0,ϕ⁡\(c​x\)=ϕ⁡\(x\)\\phi\(cx\)=\\phi\(x\)forc\>0c\>0givesK⁡\(G\)=K⁡\(c​G\)=c​K​\(G\)K\(G\)=K\(cG\)=cK\(G\), soK=0K=0\. Forε\>0\\varepsilon\>0,\|ϕ\|<1\\lvert\\phi\\rvert<1makesG↦δ​WG\\mapsto\\delta Wbounded, and a bounded linear map is00\. Both conclusions contradictδ​W≠0\\delta W\\neq 0\. ∎

### C\.26 Theorem III\.3

Theorem III\.3\(gradient descent is natural exactly where cometrics agree; for LoRA’s linear gauge, exactly alongO⁡\(r\)\\mathrm\{O\}\(r\)\)\.\(a\) Letφ:Q→Q′\\varphi:Q\\to Q^\{\\prime\}be a diffeomorphism withρ′∘φ=ρ\\rho^\{\\prime\}\\circ\\varphi=\\rho, and letΛ,Λ′\\Lambda,\\Lambda^\{\\prime\}be the rate operators onQ,Q′Q,Q^\{\\prime\}\. Under gradient flow, the two methods induce the sameθ\\theta\-velocity atqqand atφ⁡\(q\)\\varphi\(q\)for every loss iffKqρ=Kφ⁡\(q\)ρ′K^\{\\rho\}\_\{q\}=K^\{\\rho^\{\\prime\}\}\_\{\\varphi\(q\)\}, that is,D​ρφ⁡\(q\)′​\(D​φq​Λ​D​φq⊤−Λ′\)​D​ρφ⁡\(q\)′⁣⊤=0\.D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}\\big\(D\\varphi\_\{q\}\\Lambda D\\varphi\_\{q\}^\{\\top\}\-\\Lambda^\{\\prime\}\\big\)D\\rho^\{\\prime\\top\}\_\{\\varphi\(q\)\}=0\.This holds wheneverD​φ​Λ​D​φ⊤=Λ′D\\varphi\\Lambda D\\varphi^\{\\top\}=\\Lambda^\{\\prime\}, which is an isometry whenΛ=Λ′=I\\Lambda=\\Lambda^\{\\prime\}=I\. That condition is necessary whenD​ρφ⁡\(q\)′D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}is injective, and fails to be necessary in general\. Every linear gauge of a linear parametrisation is natural; for instance, forρ⁡\(x,y\)=θ0\+x\+y\\rho\(x,y\)=\\theta\_\{0\}\+x\+ythe shear\(x,y\)↦\(2​x,y−x\)\(x,y\)\\mapsto\(2x,y\-x\)preservesρ\\rhoandKKwithout being orthogonal\. So is every pointwise\-orthogonal non\-linear LoRA gaugeφ​\(q\)=φg⁡\(q\)​\(q\)\\varphi\(q\)=\\varphi\_\{g\(q\)\}\(q\)withg⁡\(q\)∈O⁡\(r\)g\(q\)\\in\\mathrm\{O\}\(r\)\.\(b\) For LoRA’s linear gaugeφg​\(B,A\)=\(B​g−1,g​A\)\\varphi\_\{g\}\(B,A\)=\(Bg^\{\-1\},gA\)with block\-scalar rates, at points whereAAhas full row rank \(BBarbitrary, includingB=0B=0\) andr<nr<n, the criterion holds iffg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)\. The same holds symmetrically whereBBhas full column rank andr<mr<m\.

\(a\) Fixq∈Qq\\in Qand putθ=ρ⁡\(q\)=ρ′​\(φ⁡\(q\)\)\\theta=\\rho\(q\)=\\rho^\{\\prime\}\(\\varphi\(q\)\)\. Gradient flow with ratesΛ\\LambdaonQQmoves the weight atqqby−Kqρ​∇θL\-K^\{\\rho\}\_\{q\}\\nabla\_\{\\theta\}L\(§[C\.24](https://arxiv.org/html/2610.12728#A3.SS24)\); gradient flow with ratesΛ′\\Lambda^\{\\prime\}onQ′Q^\{\\prime\}moves it atφ⁡\(q\)\\varphi\(q\)by−Kφ⁡\(q\)ρ′​∇θL\-K^\{\\rho^\{\\prime\}\}\_\{\\varphi\(q\)\}\\nabla\_\{\\theta\}L\. AsLLranges over the linear losses⟨γ,⋅⟩\\langle\\gamma,\\cdot\\rangle, the covector∇θL​\(θ\)=γ\\nabla\_\{\\theta\}L\(\\theta\)=\\gammaranges over all ofTθ∗​ΘT^\{\*\}\_\{\\theta\}\\Theta, so the two velocities agree for every loss iff the linear mapsKqρK^\{\\rho\}\_\{q\}andKφ⁡\(q\)ρ′K^\{\\rho^\{\\prime\}\}\_\{\\varphi\(q\)\}agree\. Differentiatingρ=ρ′∘φ\\rho=\\rho^\{\\prime\}\\circ\\varphigivesD​ρq=D​ρφ⁡\(q\)′​D​φqD\\rho\_\{q\}=D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}D\\varphi\_\{q\}, so

Kqρ−Kφ⁡\(q\)ρ′=D​ρφ⁡\(q\)′​\(D​φq​Λ​D​φq⊤−Λ′\)​D​ρφ⁡\(q\)′⁣⊤,K^\{\\rho\}\_\{q\}\-K^\{\\rho^\{\\prime\}\}\_\{\\varphi\(q\)\}=D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}\\big\(D\\varphi\_\{q\}\\Lambda D\\varphi\_\{q\}^\{\\top\}\-\\Lambda^\{\\prime\}\\big\)D\\rho^\{\\prime\\top\}\_\{\\varphi\(q\)\},which is the displayed criterion\.

IfD​φq​Λ​D​φq⊤=Λ′D\\varphi\_\{q\}\\Lambda D\\varphi\_\{q\}^\{\\top\}=\\Lambda^\{\\prime\}, the bracket vanishes\. ForΛ=Λ′=I\\Lambda=\\Lambda^\{\\prime\}=Ithe condition says thatD​φqD\\varphi\_\{q\}is orthogonal; in general it says thatφ\\varphiis an isometry from\(Q,⟨⋅,Λ−1⋅⟩\)\(Q,\\langle\\cdot,\\Lambda^\{\-1\}\\cdot\\rangle\)to\(Q′,⟨⋅,Λ′−1⋅⟩\)\(Q^\{\\prime\},\\langle\\cdot,\\Lambda^\{\\prime\-1\}\\cdot\\rangle\)\. IfD​ρφ⁡\(q\)′D\\rho^\{\\prime\}\_\{\\varphi\(q\)\}is injective, it has a left inverseEE, and multiplyingD​ρ′​M​D​ρ′⁣⊤=0D\\rho^\{\\prime\}MD\\rho^\{\\prime\\top\}=0byEEon the left and byE⊤E^\{\\top\}on the right givesM=0M=0; so the condition is then necessary\.

It is not necessary in general\. Letρ⁡\(x\)=θ0\+ℒ​x\\rho\(x\)=\\theta\_\{0\}\+\\mathcal\{L\}xbe affine with linear partℒ\\mathcal\{L\}, and letPPbe a linear gauge,ℒ​P=ℒ\\mathcal\{L\}P=\\mathcal\{L\}, withΛ′=Λ\\Lambda^\{\\prime\}=\\Lambda\. ThenD​ρ′=ℒD\\rho^\{\\prime\}=\\mathcal\{L\}andD​φ=PD\\varphi=P, andℒ​P​Λ​P⊤​ℒ⊤=ℒ​Λ​ℒ⊤\\mathcal\{L\}P\\Lambda P^\{\\top\}\\mathcal\{L\}^\{\\top\}=\\mathcal\{L\}\\Lambda\\mathcal\{L\}^\{\\top\}, so the criterion holds for everyΛ\\Lambda\. Forρ⁡\(x,y\)=θ0\+x\+y\\rho\(x,y\)=\\theta\_\{0\}\+x\+yandΛ=I\\Lambda=I, the shearP⁡\(x,y\)=\(2​x,y−x\)P\(x,y\)=\(2x,y\-x\)satisfiesℒ​P=ℒ\\mathcal\{L\}P=\\mathcal\{L\}, while in block formP​P⊤=\(4−2−22\)≠IPP^\{\\top\}=\\begin\{pmatrix\}4&\-2\\\\ \-2&2\\end\{pmatrix\}\\neq I\.

For a pointwise gaugeφ​\(q\)=φg⁡\(q\)​\(q\)\\varphi\(q\)=\\varphi\_\{g\(q\)\}\(q\)of LoRA, withg:Q→GLrg:Q\\to\\mathrm\{GL\}\_\{r\}smooth andφ\\varphia diffeomorphism, the product rule givesD​φq=φg⁡\(q\)\+VqD\\varphi\_\{q\}=\\varphi\_\{g\(q\)\}\+V\_\{q\}\. Hereφg⁡\(q\)\\varphi\_\{g\(q\)\}is the linear map\(B˙,A˙\)↦\(B˙​g​\(q\)−1,g⁡\(q\)​A˙\)\(\\dot\{B\},\\dot\{A\}\)\\mapsto\(\\dot\{B\}g\(q\)^\{\-1\},g\(q\)\\dot\{A\}\), andVq​\(v\)=dd​ϵ\|ϵ=0​φg⁡\(q\+ϵ​v\)​\(q\)V\_\{q\}\(v\)=\\frac\{d\}\{d\\epsilon\}\\big\|\_\{\\epsilon=0\}\\varphi\_\{g\(q\+\\epsilon v\)\}\(q\)is tangent to theGLr\\mathrm\{GL\}\_\{r\}\-orbit ofφ⁡\(q\)\\varphi\(q\)\. That orbit lies in a fibre ofρ\\rho, soD​ρφ⁡\(q\)​Vq=0D\\rho\_\{\\varphi\(q\)\}V\_\{q\}=0andD​ρφ⁡\(q\)​D​φq=D​ρφ⁡\(q\)​φg⁡\(q\)D\\rho\_\{\\varphi\(q\)\}D\\varphi\_\{q\}=D\\rho\_\{\\varphi\(q\)\}\\varphi\_\{g\(q\)\}\. The criterion depends onD​φqD\\varphi\_\{q\}only throughD​ρφ⁡\(q\)​D​φqD\\rho\_\{\\varphi\(q\)\}D\\varphi\_\{q\}, so it reduces to the criterion for the linear gaugeφg⁡\(q\)\\varphi\_\{g\(q\)\}atqq, which holds forg⁡\(q\)∈O⁡\(r\)g\(q\)\\in\\mathrm\{O\}\(r\)by \(b\)\.

\(b\) HereQ′=QQ^\{\\prime\}=Q,ρ′=ρ\\rho^\{\\prime\}=\\rho,Λ′=Λ\\Lambda^\{\\prime\}=\\Lambdais block\-scalar andφg​\(q\)=g⋅q\\varphi\_\{g\}\(q\)=g\\cdot q\. By[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),

Kg⋅q​\(G\)=s2​\(ηB​G​A⊤​g⊤​g​A\+ηA​B​g−1​g−⁣⊤​B⊤​G\),K\_\{g\\cdot q\}\(G\)=s^\{2\}\\big\(\\eta\_\{B\}GA^\{\\top\}g^\{\\top\}gA\+\\eta\_\{A\}Bg^\{\-1\}g^\{\-\\top\}B^\{\\top\}G\\big\),soKq=Kg⋅qK\_\{q\}=K\_\{g\\cdot q\}iff, for everyG∈ℝm×nG\\in\\mathbb\{R\}^\{m\\times n\},

G​X=Y​G,X=ηB​A⊤​\(g⊤​g−I\)​A∈ℝn×n,Y=ηA​B​\(I−g−1​g−⁣⊤\)​B⊤∈ℝm×m\.GX=YG,\\qquad X=\\eta\_\{B\}A^\{\\top\}\(g^\{\\top\}g\-I\)A\\in\\mathbb\{R\}^\{n\\times n\},\\quad Y=\\eta\_\{A\}B\(I\-g^\{\-1\}g^\{\-\\top\}\)B^\{\\top\}\\in\\mathbb\{R\}^\{m\\times m\}\.
*Matrix\-unit lemma: ifG​X=Y​GGX=YGfor everyG∈ℝm×nG\\in\\mathbb\{R\}^\{m\\times n\}, thenX=c​InX=cI\_\{n\}andY=c​ImY=cI\_\{m\}for somec∈ℝc\\in\\mathbb\{R\}\.*TakeG=Ei​j=ei​ej⊤G=E\_\{ij\}=e\_\{i\}e\_\{j\}^\{\\top\}\. Comparing the\(k,l\)\(k,l\)entries ofEi​j​XE\_\{ij\}XandY​Ei​jYE\_\{ij\}givesδi​k​Xj​l=Yk​i​δj​l\\delta\_\{ik\}X\_\{jl\}=Y\_\{ki\}\\delta\_\{jl\}\. Withk=ik=iandl≠jl\\neq jthis givesXj​l=0X\_\{jl\}=0; withl=jl=jandk≠ik\\neq iit givesYk​i=0Y\_\{ki\}=0; withk=ik=iandl=jl=jit givesXj​j=Yi​iX\_\{jj\}=Y\_\{ii\}for alli,ji,j\.

Nowrank⁡X≤r<n\\rk X\\leq r<n, soX=c​InX=cI\_\{n\}forcesc=0c=0, that is,A⊤​\(g⊤​g−I\)​A=0A^\{\\top\}\(g^\{\\top\}g\-I\)A=0\. SinceAAhas full row rank,A​A⊤AA^\{\\top\}is invertible, and multiplying by\(A​A⊤\)−1​A\(AA^\{\\top\}\)^\{\-1\}Aon the left and byA⊤​\(A​A⊤\)−1A^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}on the right givesg⊤​g=Ig^\{\\top\}g=I\. Conversely,g∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)makesXXandYYvanish\. Only the full row rank ofAAandr<nr<nwere used, andBBis arbitrary, includingB=0B=0\. If insteadBBhas full column rank andr<mr<m, thenrank⁡Y≤r<m\\rk Y\\leq r<mforcesc=0c=0, soB⁡\(I−g−1​g−⁣⊤\)​B⊤=0B\(I\-g^\{\-1\}g^\{\-\\top\}\)B^\{\\top\}=0; multiplying by\(B⊤​B\)−1​B⊤\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}on the left and byB​\(B⊤​B\)−1B\(B^\{\\top\}B\)^\{\-1\}on the right givesg−1​g−⁣⊤=Ig^\{\-1\}g^\{\-\\top\}=I, and againg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)\. ∎

Remark\(the family of plain\-GD dynamics; LoRA\+ under GD\)\. The formula forKg⋅qK\_\{g\\cdot q\}depends onggonly throughg⊤​gg^\{\\top\}g, sinceg−1​g−⁣⊤=\(g⊤​g\)−1g^\{\-1\}g^\{\-\\top\}=\(g^\{\\top\}g\)^\{\-1\}, andg⊤​g=h⊤​hg^\{\\top\}g=h^\{\\top\}hiffh​g−1∈O⁡\(r\)hg^\{\-1\}\\in\\mathrm\{O\}\(r\)\. So along a gauge orbit the cometrics of plain gradient descent are indexed byO⁡\(r\)\\GLr\\mathrm\{O\}\(r\)\\backslash\\mathrm\{GL\}\_\{r\}, whichg↦g⊤​gg\\mapsto g^\{\\top\}gidentifies withSymr\+\\mathrm\{Sym\}\_\{r\}^\{\+\}\([Background D\.27](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1),[Auxiliary result B\.19](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. Forg=c​Ig=cIwithc\>0c\>0, gradient flow with a single rateη\\etastarted fromg⋅qg\\cdot q, pulled back byg−1g^\{\-1\}, satisfiesB˙=c​B˙′=−η​c2​∇B\\dot\{B\}=c\\dot\{B\}^\{\\prime\}=\-\\eta c^\{2\}\\nabla\_\{B\}andA˙=c−1​A˙′=−η​c−2​∇A\\dot\{A\}=c^\{\-1\}\\dot\{A\}^\{\\prime\}=\-\\eta c^\{\-2\}\\nabla\_\{A\}by \([20](https://arxiv.org/html/2610.12728#A3.E20)\)\. This is LoRA with rates\(ηA,ηB\)=\(η​c−2,η​c2\)\(\\eta\_\{A\},\\eta\_\{B\}\)=\(\\eta c^\{\-2\},\\eta c^\{2\}\), and the same holds step by step for GD, whose steps are linear in the gradients\. Choosingc=λ1/4c=\\lambda^\{1/4\}andη=λ​ηA\\eta=\\sqrt\{\\lambda\}\\,\\eta\_\{A\}gives LoRA\+ with ratioλ=ηB/ηA\\lambda=\\eta\_\{B\}/\\eta\_\{A\}\. Under gradient descent, LoRA\+ is therefore the scalar gaugeλ1/4​I\\lambda^\{1/4\}Itogether with an overall change of rate\.

### C\.27 Proposition III\.4

Proposition III\.4\(cometrics that descend to the rank\-rrmanifold; known in part\)\.WriteU=col⁡BU=\\operatorname\{col\}BandV=row⁡AV=\\operatorname\{row\}A\. Statements \(a\), \(b\) and the first sentence of \(d\) are on the full\-rank locus; \(c\) needs onlyrank⁡A0=r\\rk A\_\{0\}=r, withBBarbitrary, includingB0=0B\_\{0\}=0\. All statements concern gradient flow, or \(S\)GD to first order, on the stated gradients, undamped and without Adam\. Cometrics include the learning rate\.\(a\) Scaled GD,δ​B=−η​∇B\(A​A⊤\)−1\\delta B=\-\\eta\\nabla\_\{B\}\(AA^\{\\top\}\)^\{\-1\}andδ​A=−η​\(B⊤​B\)−1​∇A\\delta A=\-\\eta\(B^\{\\top\}B\)^\{\-1\}\\nabla\_\{A\}, isGLr\\mathrm\{GL\}\_\{r\}\-equivariant with first\-order cometricK⁡\(G\)=η​s2​\(PU​G\+G​PV\)K\(G\)=\\eta s^\{2\}\(P\_\{U\}G\+GP\_\{V\}\)\.\(b\) LoRA\-Pro’s equivalent gradientg~=PU​G\+\(I−PU\)​G​PV\\tilde\{g\}=P\_\{U\}G\+\(I\-P\_\{U\}\)GP\_\{V\}is the Frobenius\-orthogonal projectionΠTW−W0​ℳr​\(G\)\\Pi\_\{T\_\{W\-W\_\{0\}\}\\mathcal\{M\}\_\{r\}\}\(G\)\. The flow of its SGD variant \(LoRA\-Pro Alg\. 1\) is the Riemannian gradient flow onW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\.\(c\) LoRA\-FA’s corrected gradients−2​∇B\(A0​A0⊤\)−1s^\{\-2\}\\nabla\_\{B\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}, fed to SGD withA0A\_\{0\}frozen, givesK⁡\(G\)=η​G​PVK\(G\)=\\eta\\,GP\_\{V\}withV=row⁡A0V=\\operatorname\{row\}A\_\{0\}, which depends onA0A\_\{0\}only throughV∈Gr⁡\(r,n\)V\\in\\Gr\(r,n\)\. Its original form, plain SGD onBB, hasK⁡\(G\)=s2​ηB​G​A0⊤​A0K\(G\)=s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}\.\(d\) The cometrics of \(a\) and \(b\) are functions ofWWon the full\-rank locus and define vector fields tangent toW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\. The field of \(c\) isG↦η​G​PVG\\mapsto\\eta GP\_\{V\}withV=row⁡A0V=\\operatorname\{row\}A\_\{0\}\. Its flow is the Frobenius gradient flow on the sliceW0\+\{X​A0\}=W0\+ℝm×n​PVW\_\{0\}\+\\\{XA\_\{0\}\\\}=W\_\{0\}\+\\mathbb\{R\}^\{m\\times n\}P\_\{V\}, of dimensionm​rmr, which containsW0W\_\{0\}and lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. The rank\-rrpart of the slice,NV=\{row\(W−W0\)=V\}N\_\{V\}=\\\{\\operatorname\{row\}\(W\-W\_\{0\}\)=V\\\}, is open and dense in it and is a submanifold ofW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\. OnNVN\_\{V\}the field is tangent toℳr\\mathcal\{M\}\_\{r\}and equalsη​G​Prow⁡\(W−W0\)\\eta GP\_\{\\operatorname\{row\}\(W\-W\_\{0\}\)\}, a function ofWW\. So \(c\) descends too, as a degenerate cometric of rankm​rmr\(less thanr⁡\(m\+n−r\)r\(m\+n\-r\)whenr<nr<n\) whose flow keepsrow⁡\(W−W0\)=row⁡A0\\operatorname\{row\}\(W\-W\_\{0\}\)=\\operatorname\{row\}A\_\{0\}fixed\. The cometric of plain GD depends on the representative; it is invariant only underO⁡\(r\)\\mathrm\{O\}\(r\)\.

In \(a\), \(b\) and the first sentence of \(d\),\(B,A\)\(B,A\)lies on the full\-rank locus and the conventions of §[C\.24](https://arxiv.org/html/2610.12728#A3.SS24)apply\. Write𝒯:=TW−W0​ℳr\\mathcal\{T\}:=T\_\{W\-W\_\{0\}\}\\mathcal\{M\}\_\{r\}, whereW−W0=s​B​AW\-W\_\{0\}=sBAhas rankrr, column spaceUUand row spaceVV\.

\(a\) The steps areδ​B=−η​∇B\(A​A⊤\)−1=−η​s​G​A⊤​\(A​A⊤\)−1\\delta B=\-\\eta\\nabla\_\{B\}\(AA^\{\\top\}\)^\{\-1\}=\-\\eta s\\,GA^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}andδ​A=−η​\(B⊤​B\)−1​∇A=−η​s​\(B⊤​B\)−1​B⊤​G\\delta A=\-\\eta\(B^\{\\top\}B\)^\{\-1\}\\nabla\_\{A\}=\-\\eta s\\,\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}G\. Hence

δ​W\\displaystyle\\delta W=s⁡\(δ​B​A\+B​δ​A\)\+s​δ​B​δ​A=−η​s2​\(G​A⊤​\(A​A⊤\)−1​A\+B​\(B⊤​B\)−1​B⊤​G\)\+O⁡\(η2\)\\displaystyle=s\(\\delta BA\+B\\delta A\)\+s\\,\\delta B\\,\\delta A=\-\\eta s^\{2\}\\big\(GA^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}A\+B\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}G\\big\)\+O\(\\eta^\{2\}\)=−η​s2​\(G​PV\+PU​G\)\+O⁡\(η2\),\\displaystyle=\-\\eta s^\{2\}\(GP\_\{V\}\+P\_\{U\}G\)\+O\(\\eta^\{2\}\),and the flow version givesW˙=−η​s2​\(PU​G\+G​PV\)\\dot\{W\}=\-\\eta s^\{2\}\(P\_\{U\}G\+GP\_\{V\}\)exactly\. Left multiplication byPUP\_\{U\}and right multiplication byPVP\_\{V\}are orthogonal projections of\(ℝm×n,⟨⋅,⋅⟩\)\(\\mathbb\{R\}^\{m\\times n\},\\langle\\cdot,\\cdot\\rangle\), soK⁡\(G\)=η​s2​\(PU​G\+G​PV\)K\(G\)=\\eta s^\{2\}\(P\_\{U\}G\+GP\_\{V\}\)is symmetric positive semidefinite\. For equivariance, letq′=g⋅qq^\{\\prime\}=g\\cdot q\. By \([20](https://arxiv.org/html/2610.12728#A3.E20)\) and\(A′​A′⁣⊤\)−1=\(g​A​A⊤​g⊤\)−1=g−⁣⊤​\(A​A⊤\)−1​g−1\(A^\{\\prime\}A^\{\\prime\\top\}\)^\{\-1\}=\(gAA^\{\\top\}g^\{\\top\}\)^\{\-1\}=g^\{\-\\top\}\(AA^\{\\top\}\)^\{\-1\}g^\{\-1\},

δ​B′=−η​∇Bg⊤​g−⁣⊤​\(A​A⊤\)−1​g−1=δ​B​g−1,\\delta B^\{\\prime\}=\-\\eta\\nabla\_\{B\}g^\{\\top\}g^\{\-\\top\}\(AA^\{\\top\}\)^\{\-1\}g^\{\-1\}=\\delta B\\,g^\{\-1\},and\(B′⁣⊤​B′\)−1=\(g−⁣⊤​B⊤​B​g−1\)−1=g​\(B⊤​B\)−1​g⊤\(B^\{\\prime\\top\}B^\{\\prime\}\)^\{\-1\}=\(g^\{\-\\top\}B^\{\\top\}Bg^\{\-1\}\)^\{\-1\}=g\(B^\{\\top\}B\)^\{\-1\}g^\{\\top\}givesδ​A′=−η​g​\(B⊤​B\)−1​g⊤​g−⁣⊤​∇A=g​δ​A\\delta A^\{\\prime\}=\-\\eta g\(B^\{\\top\}B\)^\{\-1\}g^\{\\top\}g^\{\-\\top\}\\nabla\_\{A\}=g\\,\\delta A\. So the discrete step satisfies \([21](https://arxiv.org/html/2610.12728#A3.E21)\) for everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\.

\(b\)*The tangent space\.*We show that

\{XA\+BY:X∈ℝm×r,Y∈ℝr×n\}=\{D:PU⟂DPV⟂=0\}=𝒯\.\\\{XA\+BY:\\ X\\in\\mathbb\{R\}^\{m\\times r\},\\ Y\\in\\mathbb\{R\}^\{r\\times n\}\\\}=\\\{D:\\ P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}=0\\\}=\\mathcal\{T\}\.The first inclusion⊆\\subseteqholds becauseA​PV⟂=0AP\_\{V^\{\\perp\}\}=0andPU⟂​B=0P\_\{U^\{\\perp\}\}B=0\. Conversely, ifPU⟂​D​PV⟂=0P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\}=0, thenD=PU​D\+PU⟂​D​PV=B⁡\[\(B⊤​B\)−1​B⊤​D\]\+\[PU⟂​D​A⊤​\(A​A⊤\)−1\]​AD=P\_\{U\}D\+P\_\{U^\{\\perp\}\}DP\_\{V\}=B\\big\[\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}D\\big\]\+\\big\[P\_\{U^\{\\perp\}\}DA^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}\\big\]A\. The common space has dimensionm​n−\(m−r\)​\(n−r\)=r⁡\(m\+n−r\)=dimℳrmn\-\(m\-r\)\(n\-r\)=r\(m\+n\-r\)=\\dim\\mathcal\{M\}\_\{r\}\([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\. The curvest↦s⁡\(B\+t​B˙\)​\(A\+t​A˙\)t\\mapsto s\(B\+t\\dot\{B\}\)\(A\+t\\dot\{A\}\)lie inℳr\\mathcal\{M\}\_\{r\}for smalltt, so\{B˙​A\+B​A˙\}⊆𝒯\\\{\\dot\{B\}A\+B\\dot\{A\}\\\}\\subseteq\\mathcal\{T\}, and equality of dimensions gives equality\.

*The projection\.*Let𝒩=\{PU⟂​E​PV⟂:E∈ℝm×n\}\\mathcal\{N\}=\\\{P\_\{U^\{\\perp\}\}EP\_\{V^\{\\perp\}\}:E\\in\\mathbb\{R\}^\{m\\times n\}\\\}\. ForD∈𝒯D\\in\\mathcal\{T\}we have⟨D,PU⟂​E​PV⟂⟩=⟨PU⟂​D​PV⟂,E⟩=0\\langle D,P\_\{U^\{\\perp\}\}EP\_\{V^\{\\perp\}\}\\rangle=\\langle P\_\{U^\{\\perp\}\}DP\_\{V^\{\\perp\}\},E\\rangle=0, so𝒩⟂𝒯\\mathcal\{N\}\\perp\\mathcal\{T\}\. MoreoverG=\(G−PU⟂​G​PV⟂\)\+PU⟂​G​PV⟂G=\(G\-P\_\{U^\{\\perp\}\}GP\_\{V^\{\\perp\}\}\)\+P\_\{U^\{\\perp\}\}GP\_\{V^\{\\perp\}\}, with the first term in𝒯\\mathcal\{T\}and the second in𝒩\\mathcal\{N\}\. Hence

Π𝒯​\(G\)=G−\(I−PU\)​G​\(I−PV\)=PU​G\+G​PV−PU​G​PV=PU​G\+\(I−PU\)​G​PV=g~\.\\Pi\_\{\\mathcal\{T\}\}\(G\)=G\-\(I\-P\_\{U\}\)G\(I\-P\_\{V\}\)=P\_\{U\}G\+GP\_\{V\}\-P\_\{U\}GP\_\{V\}=P\_\{U\}G\+\(I\-P\_\{U\}\)GP\_\{V\}=\\tilde\{g\}\.
*LoRA\-Pro’s SGD variant\.*Algorithm 1 of[Wang et al\. \(2024i\)](https://arxiv.org/html/2610.12728#bib.bib486)takes the stepsδ​B=−η​gB\\delta B=\-\\eta g^\{B\}andδ​A=−η​gA\\delta A=\-\\eta g^\{A\}along a pair that minimises∥s⁡\(gB​A\+B​gA\)−G∥F\\lVert s\(g^\{B\}A\+Bg^\{A\}\)\-G\\rVert\_\{F\}\(their Theorem 2\.1\)\. As\(gB,gA\)\(g^\{B\},g^\{A\}\)ranges over all pairs,s⁡\(gB​A\+B​gA\)s\(g^\{B\}A\+Bg^\{A\}\)ranges over𝒯\\mathcal\{T\}, and the closest point of𝒯\\mathcal\{T\}toGGisΠ𝒯​G\\Pi\_\{\\mathcal\{T\}\}G\. So every minimiser satisfiess⁡\(gB​A\+B​gA\)=Π𝒯​Gs\(g^\{B\}A\+Bg^\{A\}\)=\\Pi\_\{\\mathcal\{T\}\}G; explicitly, the minimisers are

gA=s−1​\(B⊤​B\)−1​B⊤​G\+X​A,gB=s−1​\(I−PU\)​G​A⊤​\(A​A⊤\)−1−B​X,X∈ℝr×r,g^\{A\}=s^\{\-1\}\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}G\+XA,\\qquad g^\{B\}=s^\{\-1\}\(I\-P\_\{U\}\)GA^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}\-BX,\\qquad X\\in\\mathbb\{R\}^\{r\\times r\},for whichs​B​gA=PU​G\+s​B​X​AsBg^\{A\}=P\_\{U\}G\+sBXAands​gB​A=\(I−PU\)​G​PV−s​B​X​Asg^\{B\}A=\(I\-P\_\{U\}\)GP\_\{V\}\-sBXA\. For every choice of the free matrixXX,

δ​W=s⁡\(δ​B​A\+B​δ​A\)\+s​δ​B​δ​A=−η​Π𝒯​G\+O⁡\(η2\)\.\\delta W=s\(\\delta BA\+B\\delta A\)\+s\\,\\delta B\\,\\delta A=\-\\eta\\,\\Pi\_\{\\mathcal\{T\}\}G\+O\(\\eta^\{2\}\)\.The flow version isW˙=−η​ΠTW−W0​ℳr​∇θL​\(W\)\\dot\{W\}=\-\\eta\\,\\Pi\_\{T\_\{W\-W\_\{0\}\}\\mathcal\{M\}\_\{r\}\}\\nabla\_\{\\theta\}L\(W\)\. SinceΠ𝒯​∇θ​L\\Pi\_\{\\mathcal\{T\}\}\\nabla\_\{\\theta\}Lis the Riemannian gradient ofLLrestricted toW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}for the metric induced by the Frobenius inner product \([Background D\.24](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx3),[Auxiliary result B\.30](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5);[Absil et al\., 2008](https://arxiv.org/html/2610.12728#bib.bib2), Ch\. 3\), this is the Riemannian gradient flow onW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}, run at rateη\\eta\.

\(c\) HereA=A0A=A\_\{0\}is frozen withrank⁡A0=r\\rk A\_\{0\}=r, andBBis arbitrary\. The corrected step isδ​B=−η​s−2​∇B\(A0​A0⊤\)−1=−η​s−1​G​A0⊤​\(A0​A0⊤\)−1\\delta B=\-\\eta s^\{\-2\}\\nabla\_\{B\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}=\-\\eta s^\{\-1\}GA\_\{0\}^\{\\top\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}, so

Δ​W=s​δ​B​A0=−η​G​A0⊤​\(A0​A0⊤\)−1​A0=−η​G​PV,\\Delta W=s\\,\\delta B\\,A\_\{0\}=\-\\eta\\,GA\_\{0\}^\{\\top\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}A\_\{0\}=\-\\eta\\,GP\_\{V\},exactly, sinceA0A\_\{0\}does not move\.PVP\_\{V\}depends only onV=row⁡A0V=\\operatorname\{row\}A\_\{0\}\. The original LoRA\-FA stepδ​B=−ηB​∇B=−ηB​s​G​A0⊤\\delta B=\-\\eta\_\{B\}\\nabla\_\{B\}=\-\\eta\_\{B\}sGA\_\{0\}^\{\\top\}givesΔ​W=s​δ​B​A0=−s2​ηB​G​A0⊤​A0\\Delta W=s\\,\\delta B\\,A\_\{0\}=\-s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}\.

\(d\) On the full\-rank locusPUP\_\{U\}andPVP\_\{V\}are the projectors ontocol⁡\(W−W0\)\\operatorname\{col\}\(W\-W\_\{0\}\)androw⁡\(W−W0\)\\operatorname\{row\}\(W\-W\_\{0\}\), so the cometrics of \(a\) and \(b\) are functions ofWW\. Their values lie in𝒯\\mathcal\{T\}: this is clear forΠ𝒯\\Pi\_\{\\mathcal\{T\}\}, andPU⟂​\(PU​G\+G​PV\)​PV⟂=0P\_\{U^\{\\perp\}\}\(P\_\{U\}G\+GP\_\{V\}\)P\_\{V^\{\\perp\}\}=0\.

For \(c\), the trajectory satisfiesW−W0=s​B​A0∈\{X​A0:X∈ℝm×r\}W\-W\_\{0\}=sBA\_\{0\}\\in\\\{XA\_\{0\}:X\\in\\mathbb\{R\}^\{m\\times r\}\\\}, and\{X​A0\}=\{Z:row⁡Z⊆V\}=ℝm×n​PV\\\{XA\_\{0\}\\\}=\\\{Z:\\operatorname\{row\}Z\\subseteq V\\\}=\\mathbb\{R\}^\{m\\times n\}P\_\{V\}, a linear space of dimensionm​rmr\. Right multiplication byPVP\_\{V\}is the Frobenius\-orthogonal projection onto it, soW˙=−η​∇θL​\(W\)​PV\\dot\{W\}=\-\\eta\\nabla\_\{\\theta\}L\(W\)P\_\{V\}is the gradient flow ofLLrestricted to the affine sliceW0\+ℝm×n​PVW\_\{0\}\+\\mathbb\{R\}^\{m\\times n\}P\_\{V\}with its flat Frobenius metric\. The slice containsW0W\_\{0\}\(X=0X=0\) and lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, sincerank⁡X​A0≤r\\rk XA\_\{0\}\\leq r\. BecauseX↦X​A0X\\mapsto XA\_\{0\}is injective,rank⁡X​A0=rank⁡X\\rk XA\_\{0\}=\\rk X, androw⁡\(X​A0\)=V\\operatorname\{row\}\(XA\_\{0\}\)=Viffrank⁡X=r\\rk X=r\. SoNV=W0\+\{X​A0:rank⁡X=r\}N\_\{V\}=W\_\{0\}\+\\\{XA\_\{0\}:\\rk X=r\\\}, which is open in the slice because rankrris an open condition \([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(a\)\), and dense because everyX∈ℝm×rX\\in\\mathbb\{R\}^\{m\\times r\}\(m≥rm\\geq r\) is a limit of rank\-rrmatricesX\+t​EX\+tEast→0t\\to 0: ifEEhasIrI\_\{r\}in its firstrrrows and zeros elsewhere, the topr×rr\\times rminor ofX\+t​EX\+tEis a monic polynomial of degreerrintt, so it vanishes for only finitely manytt\.NVN\_\{V\}is an open subset of an affine subspace, hence an embedded submanifold ofℝm×n\\mathbb\{R\}^\{m\\times n\}, and it lies in the embedded submanifoldW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\. The inclusionNV↪W0\+ℳrN\_\{V\}\\hookrightarrow W\_\{0\}\+\\mathcal\{M\}\_\{r\}is smooth by[Auxiliary result B\.2](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(c\), an immersion because its composite with the inclusionW0\+ℳr↪ℝm×nW\_\{0\}\+\\mathcal\{M\}\_\{r\}\\hookrightarrow\\mathbb\{R\}^\{m\\times n\}is one, and a homeomorphism onto its image because both sets carry the subspace topology ofℝm×n\\mathbb\{R\}^\{m\\times n\}\. So it is an embedding, andNVN\_\{V\}is an embedded submanifold ofW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\([Background C\.4](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\)\. AtW∈NVW\\in N\_\{V\},row⁡\(W−W0\)=V\\operatorname\{row\}\(W\-W\_\{0\}\)=V, soG​PV=G​Prow⁡\(W−W0\)GP\_\{V\}=GP\_\{\\operatorname\{row\}\(W\-W\_\{0\}\)\}is a function ofWW, and, withUW=col⁡\(W−W0\)U\_\{W\}=\\operatorname\{col\}\(W\-W\_\{0\}\),PUW⟂​\(G​PV\)​PV⟂=0P\_\{U\_\{W\}^\{\\perp\}\}\(GP\_\{V\}\)P\_\{V^\{\\perp\}\}=0shows that it lies inTW−W0​ℳrT\_\{W\-W\_\{0\}\}\\mathcal\{M\}\_\{r\}\(it lies inℝm×n​PV=T​NV\\mathbb\{R\}^\{m\\times n\}P\_\{V\}=TN\_\{V\}as well\)\. The operatorG↦η​G​PVG\\mapsto\\eta GP\_\{V\}has rankdimℝm×n​PV=m​r\\dim\\mathbb\{R\}^\{m\\times n\}P\_\{V\}=mr, andm​r<r⁡\(m\+n−r\)mr<r\(m\+n\-r\)whenr<nr<n\. Along the flowW⁡\(t\)W\(t\)stays in the slice, sorow⁡\(W⁡\(t\)−W0\)⊆V\\operatorname\{row\}\(W\(t\)\-W\_\{0\}\)\\subseteq Vfor alltt, with equality wheneverW⁡\(t\)∈NVW\(t\)\\in N\_\{V\}\.

The last sentence of \(d\) is[Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(b\)\. ∎

Remark\(the departures listed under Practice\)\. The symmetry statements in the Practice paragraph after[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)are proved with[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8): damping\(B⊤​B\+δ​I\)−1\(B^\{\\top\}B\+\\delta I\)^\{\-1\},\(A​A⊤\+δ​I\)−1\(AA^\{\\top\}\+\\delta I\)^\{\-1\}withδ\>0\\delta\>0leaves exactlyO⁡\(r\)\\mathrm\{O\}\(r\); an AdamW step applied after the preconditioner, as in scaled AdamW and in HF’s LoRA\-FA optimizer, leaves exactly the signed permutationsBrB\_\{r\}; and LoRA\-Pro’s Sylvester\-optimal free matrix leaves exactlyO⁡\(r\)\\mathrm\{O\}\(r\)at the parameter level\. For LoRA\-FA with the undamped correction and AdamW atε=0\\varepsilon=0, the first step fromB0=0B\_\{0\}=0isΔ​W=−s​ηB​sign⁡\(G​A0⊤​\(A0​A0⊤\)−1\)​A0\\Delta W=\-s\\eta\_\{B\}\\operatorname\{sign\}\\big\(GA\_\{0\}^\{\\top\}\(A\_\{0\}A\_\{0\}^\{\\top\}\)^\{\-1\}\\big\)A\_\{0\}\. ReplacingA0A\_\{0\}by2​A02A\_\{0\}, which has the same row space, doubles this step, so the trajectory depends onA0A\_\{0\}beyond its row space \(see also §[C\.36](https://arxiv.org/html/2610.12728#A3.SS36)\)\.

### C\.28 Theorem III\.5

Theorem III\.5\(Noether charges and isometries for cotangent signals; known,[Zhao et al\. \(2022b\)](https://arxiv.org/html/2610.12728#bib.bib580)\)\.Let a Lie group act linearly onQ=ℝNQ=\\mathbb\{R\}^\{N\}withρ∘k=ρ\\rho\\circ k=\\rho, letΛ≻0\\Lambda\\succ 0be constant, and letγ⁡\(t\)∈Tρ⁡\(q⁡\(t\)\)∗​Θ\\gamma\(t\)\\in T^\{\*\}\_\{\\rho\(q\(t\)\)\}\\Thetabe any cotangent signal, gradient or not\. Consider the continuous\-time updatesq˙=−Λ​D​ρq⊤​γ​\(t\)\.\\dot\{q\}=\-\\Lambda\\,D\\rho\_\{q\}^\{\\top\}\\gamma\(t\)\.\(i\) For every generatorξ\\xiwithΛ−1​ξ\\Lambda^\{\-1\}\\xisymmetric, the chargeJξ​\(q\)=12​⟨q,Λ−1​ξ​q⟩J\_\{\\xi\}\(q\)=\\tfrac\{1\}\{2\}\\langle q,\\Lambda^\{\-1\}\\xi q\\rangleis conserved\.\(ii\) For every generatorξ\\xiwithΛ−1​ξ\\Lambda^\{\-1\}\\xiantisymmetric,k=eτ​ξk=e^\{\\tau\\xi\}preserves⟨⋅,Λ−1⋅⟩\\langle\\cdot,\\Lambda^\{\-1\}\\cdot\\rangle, soKk​q=KqK\_\{kq\}=K\_\{q\}along the orbit; for the same cotangentγ\\gamma, theθ\\theta\-velocities atqqand atk​qkqagree\. Suppose moreover thatγ\\gammadepends onqqonly through\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\), asγ=∇θL\\gamma=\\nabla\_\{\\theta\}Lor an open\-loop signal does, and that solutions are unique \(γ\\gammameasurable inttand locally Lipschitz inθ\\theta,ρ\\rhoof classCloc1,1C^\{1,1\}\_\{\\mathrm\{loc\}\}\)\. Then theθ\\theta\-trajectories fromq0q\_\{0\}and fromk​q0kq\_\{0\}coincide\. Ifγ\\gammauses Euclidean norms onQQ, as global\-norm clipping does, the trajectory statement holds when, in addition,kkis a Euclidean isometry, and can fail otherwise\. So \(i\) and the velocity statement of \(ii\) hold for arbitraryγ\\gamma, while the trajectory statement needs a gradient\-like signal\.\(iii\) When𝔤\\mathfrak\{g\}is closed underξ↦Λ​ξ⊤​Λ−1\\xi\\mapsto\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}, every generator splits into a part of type \(i\) and a part of type \(ii\), a Cartan decomposition𝔤=𝔨⊕𝔭\\mathfrak\{g\}=\\mathfrak\{k\}\\oplus\\mathfrak\{p\}\. This holds for LoRA’s𝔤​𝔩r\\mathfrak\{gl\}\_\{r\}with per\-block scalar rates\. Without closure, there are generators of neither type\. With per\-rank\-channel ratesΛ⁡\(B,A\)=\(B​DB,DA​A\)\\Lambda\(B,A\)=\(BD\_\{B\},D\_\{A\}A\), whereDB=diag⁡\(ηB,j\)D\_\{B\}=\\diag\(\\eta\_\{B,j\}\)andDA=diag⁡\(ηA,j\)D\_\{A\}=\\diag\(\\eta\_\{A,j\}\), the generatorξX\\xi\_\{X\}is of type \(i\) iffX​DB−1XD\_\{B\}^\{\-1\}andDA−1​XD\_\{A\}^\{\-1\}Xare both symmetric, and of type \(ii\) iff both are antisymmetric\. An off\-diagonal pair\(i,j\)\(i,j\)contributes one charge and one isometry iffηB,i​ηA,i=ηB,j​ηA,j\\eta\_\{B,i\}\\eta\_\{A,i\}=\\eta\_\{B,j\}\\eta\_\{A,j\}, and𝔤\\mathfrak\{g\}is closed iff all the productsηB,j​ηA,j\\eta\_\{B,j\}\\eta\_\{A,j\}are equal\. In that case one gets aDD\-conjugate of𝔰​𝔬r⊕Symr\\mathfrak\{so\}\_\{r\}\\oplus\\mathrm\{Sym\}\_\{r\}whose isometries are not all Euclidean unless the rates are block\-scalar, so the clipping caveat of \(ii\) applies\. For pairwise distinct products, only the diagonalXXgive charges and no nonzeroXXgives an isometry\.

Part \(i\) is Noether’s theorem in the form of[Auxiliary result B\.29](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5), which allows an arbitrary cotangent signal; we give the short argument in full\. Solutions are absolutely continuous and the identities below hold for almost everytt\. Forξ\\xiin the Lie algebra𝔤⊆𝔤​𝔩​\(N\)\\mathfrak\{g\}\\subseteq\\mathfrak\{gl\}\(N\)of the group,ρ⁡\(eτ​ξ​q\)=ρ⁡\(q\)\\rho\(e^\{\\tau\\xi\}q\)=\\rho\(q\)for allτ\\tau; differentiating atτ=0\\tau=0gives the infinitesimal invariance

D​ρq​\(ξ​q\)=0\(q∈Q,ξ∈𝔤\)\.D\\rho\_\{q\}\(\\xi q\)=0\\qquad\(q\\in Q,\\ \\xi\\in\\mathfrak\{g\}\)\.\(22\)
\(i\) IfΛ−1​ξ\\Lambda^\{\-1\}\\xiis symmetric, the gradient ofJξJ\_\{\\xi\}isΛ−1​ξ​q\\Lambda^\{\-1\}\\xi q\. Along a solution, using thatΛ\\Lambdais symmetric and \([22](https://arxiv.org/html/2610.12728#A3.E22)\),

dd​t​Jξ​\(q\)=⟨Λ−1​ξ​q,q˙⟩=−⟨Λ−1​ξ​q,Λ​D​ρq⊤​γ⟩=−⟨ξ​q,D​ρq⊤​γ⟩=−⟨D​ρq​ξ​q,γ⟩=0\.\\tfrac\{d\}\{dt\}J\_\{\\xi\}\(q\)=\\langle\\Lambda^\{\-1\}\\xi q,\\dot\{q\}\\rangle=\-\\langle\\Lambda^\{\-1\}\\xi q,\\Lambda D\\rho\_\{q\}^\{\\top\}\\gamma\\rangle=\-\\langle\\xi q,D\\rho\_\{q\}^\{\\top\}\\gamma\\rangle=\-\\langle D\\rho\_\{q\}\\xi q,\\gamma\\rangle=0\.Nothing aboutγ\\gammawas used, so it may depend onqqin any way\.

\(ii\)*Isometry and cometric\.*IfΛ−1​ξ\\Lambda^\{\-1\}\\xiis antisymmetric, thenξ⊤​Λ−1=−Λ−1​ξ\\xi^\{\\top\}\\Lambda^\{\-1\}=\-\\Lambda^\{\-1\}\\xi, so

dd​τ​\(eτ​ξ⊤​Λ−1​eτ​ξ\)=eτ​ξ⊤​\(ξ⊤​Λ−1\+Λ−1​ξ\)​eτ​ξ=0,\\tfrac\{d\}\{d\\tau\}\\big\(e^\{\\tau\\xi^\{\\top\}\}\\Lambda^\{\-1\}e^\{\\tau\\xi\}\\big\)=e^\{\\tau\\xi^\{\\top\}\}\(\\xi^\{\\top\}\\Lambda^\{\-1\}\+\\Lambda^\{\-1\}\\xi\)e^\{\\tau\\xi\}=0,andk=eτ​ξk=e^\{\\tau\\xi\}satisfiesk⊤​Λ−1​k=Λ−1k^\{\\top\}\\Lambda^\{\-1\}k=\\Lambda^\{\-1\}, equivalentlyk​Λ​k⊤=Λk\\Lambda k^\{\\top\}=\\Lambda\. Fromρ∘k=ρ\\rho\\circ k=\\rhowe getD​ρq=D​ρk​q​kD\\rho\_\{q\}=D\\rho\_\{kq\}k, so

Kq=D​ρk​q​k​Λ​k⊤​D​ρk​q⊤=D​ρk​q​Λ​D​ρk​q⊤=Kk​q\.K\_\{q\}=D\\rho\_\{kq\}\\,k\\Lambda k^\{\\top\}D\\rho\_\{kq\}^\{\\top\}=D\\rho\_\{kq\}\\Lambda D\\rho\_\{kq\}^\{\\top\}=K\_\{kq\}\.Sinceρ⁡\(k​q\)=ρ⁡\(q\)\\rho\(kq\)=\\rho\(q\), the same covectorγ\\gammaacts at both points, and the velocities−Kq​γ\-K\_\{q\}\\gammaand−Kk​q​γ\-K\_\{kq\}\\gammaagree\. The rest of the argument uses only the two propertiesk​Λ​k⊤=Λk\\Lambda k^\{\\top\}=\\Lambdaandρ∘k=ρ\\rho\\circ k=\\rho, so it applies to every suchkk, not only to those of the formeτ​ξe^\{\\tau\\xi\}\.

*Trajectories\.*Letγ=γ⁡\(t,ρ⁡\(q\)\)\\gamma=\\gamma\(t,\\rho\(q\)\)and letq⁡\(t\)q\(t\)solveq˙=F⁡\(t,q\):=−Λ​D​ρq⊤​γ​\(t,ρ⁡\(q\)\)\\dot\{q\}=F\(t,q\):=\-\\Lambda D\\rho\_\{q\}^\{\\top\}\\gamma\(t,\\rho\(q\)\)fromq0q\_\{0\}\. Putp⁡\(t\)=k​q​\(t\)p\(t\)=kq\(t\)\. Then, usingD​ρq⊤=k⊤​D​ρk​q⊤D\\rho\_\{q\}^\{\\top\}=k^\{\\top\}D\\rho\_\{kq\}^\{\\top\}andρ⁡\(p\)=ρ⁡\(q\)\\rho\(p\)=\\rho\(q\),

p˙=k​q˙=−k​Λ​k⊤​D​ρp⊤​γ​\(t,ρ⁡\(q\)\)=−Λ​D​ρp⊤​γ​\(t,ρ⁡\(p\)\)=F⁡\(t,p\),\\dot\{p\}=k\\dot\{q\}=\-k\\Lambda k^\{\\top\}D\\rho\_\{p\}^\{\\top\}\\gamma\(t,\\rho\(q\)\)=\-\\Lambda D\\rho\_\{p\}^\{\\top\}\\gamma\(t,\\rho\(p\)\)=F\(t,p\),soppsolves the same equation fromk​q0kq\_\{0\}\. Under the stated regularity, read locally uniformly intt,FFis measurable inttand locally bounded and locally Lipschitz inpp, becauseD​ρD\\rhoandρ\\rhoare locally Lipschitz andγ\\gammais locally Lipschitz inθ\\theta\. By Gronwall’s inequality two solutions with the same initial value coincide \([Auxiliary result B\.28](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\), soppis the solution fromk​q0kq\_\{0\}, andρ⁡\(p⁡\(t\)\)=ρ⁡\(q⁡\(t\)\)\\rho\(p\(t\)\)=\\rho\(q\(t\)\)for alltt\.

*Clipping\.*Supposeγ\\gammadepends onqqthrough\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\)and through Euclidean norms∥D​ρq⊤​v∥\\lVert D\\rho\_\{q\}^\{\\top\}v\\rVertof pulled\-back covectorsvvthat are themselves functions of\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\)\. Global\-norm clipping is of this kind:

γ=min⁡\(1,C/∥D​ρq⊤​∇θL∥\)​∇θL\.\\gamma=\\min\\big\(1,C/\\lVert D\\rho\_\{q\}^\{\\top\}\\nabla\_\{\\theta\}L\\rVert\\big\)\\,\\nabla\_\{\\theta\}L\.Ifkkis also a Euclidean isometry, then∥D​ρk​q⊤​v∥=∥k−⁣⊤​D​ρq⊤​v∥=∥D​ρq⊤​v∥\\lVert D\\rho\_\{kq\}^\{\\top\}v\\rVert=\\lVert k^\{\-\\top\}D\\rho\_\{q\}^\{\\top\}v\\rVert=\\lVert D\\rho\_\{q\}^\{\\top\}v\\rVert, soγ\\gammatakes the same value atqqand atk​qkq, and the argument above goes through\. Otherwise it can fail\. TakeQ=ℝ2Q=\\mathbb\{R\}^\{2\},Θ=ℝ\\Theta=\\mathbb\{R\},Λ=diag⁡\(1,4\)\\Lambda=\\diag\(1,4\),ρ⁡\(q\)=12​q⊤​Λ−1​q\\rho\(q\)=\\tfrac\{1\}\{2\}q^\{\\top\}\\Lambda^\{\-1\}qandL⁡\(θ\)=θ2/2L\(\\theta\)=\\theta^\{2\}/2\. The group\{k:k⊤​Λ−1​k=Λ−1\}\\\{k:k^\{\\top\}\\Lambda^\{\-1\}k=\\Lambda^\{\-1\}\\\}acts linearly and preservesρ\\rho; its generatorξ=Λ1/2JΛ−1/2\\xi=\\Lambda^\{1/2\}J\\Lambda^\{\-1/2\}, withJ=\(0−110\)J=\\begin\{pmatrix\}0&\-1\\\\ 1&0\\end\{pmatrix\}, hasΛ−1ξ=Λ−1/2JΛ−1/2\\Lambda^\{\-1\}\\xi=\\Lambda^\{\-1/2\}J\\Lambda^\{\-1/2\}antisymmetric, andk=e\(π/2\)​ξ=Λ1/2e\(π/2\)​JΛ−1/2k=e^\{\(\\pi/2\)\\xi\}=\\Lambda^\{1/2\}e^\{\(\\pi/2\)J\}\\Lambda^\{\-1/2\}mapsq0=\(1,0\)q\_\{0\}=\(1,0\)tok​q0=\(0,2\)kq\_\{0\}=\(0,2\), both withθ=1/2\\theta=1/2\. HereD​ρq⊤​γ=γ​Λ−1​qD\\rho\_\{q\}^\{\\top\}\\gamma=\\gamma\\Lambda^\{\-1\}qandθ˙=−γ​q⊤​Λ−1​q=−2​θ​γ\\dot\{\\theta\}=\-\\gamma\\,q^\{\\top\}\\Lambda^\{\-1\}q=\-2\\theta\\gamma\. The parameter gradientsL′​\(θ\)​Λ−1​qL^\{\\prime\}\(\\theta\)\\Lambda^\{\-1\}qhave norms1/21/2atq0q\_\{0\}and1/41/4atk​q0kq\_\{0\}, so withC=1/10C=1/10the clipping factors are1/51/5and2/52/5, and the initial velocities areθ˙=−1/10\\dot\{\\theta\}=\-1/10andθ˙=−1/5\\dot\{\\theta\}=\-1/5\. Theθ\\theta\-trajectories differ, while without clipping they coincide by the argument above\.

\(iii\)*Splitting\.*Putϑ⁡\(ξ\)=−Λ​ξ⊤​Λ−1\\vartheta\(\\xi\)=\-\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}\. It is an involution of𝔤​𝔩​\(N\)\\mathfrak\{gl\}\(N\)\(ϑ2​\(ξ\)=Λ​Λ−1​ξ​Λ​Λ−1=ξ\\vartheta^\{2\}\(\\xi\)=\\Lambda\\Lambda^\{\-1\}\\xi\\Lambda\\Lambda^\{\-1\}=\\xi\) and a Lie algebra automorphism:ϑ⁡\(\[ξ,ζ\]\)=−Λ⁡\(ζ⊤​ξ⊤−ξ⊤​ζ⊤\)​Λ−1=\[ϑ​ξ,ϑ​ζ\]\\vartheta\(\[\\xi,\\zeta\]\)=\-\\Lambda\(\\zeta^\{\\top\}\\xi^\{\\top\}\-\\xi^\{\\top\}\\zeta^\{\\top\}\)\\Lambda^\{\-1\}=\[\\vartheta\\xi,\\vartheta\\zeta\]\. A generator is of type \(ii\) iffΛ−1​ξ\\Lambda^\{\-1\}\\xiis antisymmetric, iffΛ​ξ⊤​Λ−1=−ξ\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}=\-\\xi, iffϑ​ξ=ξ\\vartheta\\xi=\\xi; it is of type \(i\) iffϑ​ξ=−ξ\\vartheta\\xi=\-\\xi\. If𝔤\\mathfrak\{g\}is closed underξ↦Λ​ξ⊤​Λ−1\\xi\\mapsto\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}, it isϑ\\vartheta\-stable, and

ξ=12​\(ξ\+Λ​ξ⊤​Λ−1\)\+12​\(ξ−Λ​ξ⊤​Λ−1\)\\xi=\\tfrac\{1\}\{2\}\(\\xi\+\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}\)\+\\tfrac\{1\}\{2\}\(\\xi\-\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}\)writes everyξ∈𝔤\\xi\\in\\mathfrak\{g\}as a type \(i\) part \(in the−1\-1eigenspace𝔭\\mathfrak\{p\}ofϑ\\vartheta\) plus a type \(ii\) part \(in the\+1\+1eigenspace𝔨\\mathfrak\{k\}\)\. Sinceϑ\\varthetais an automorphism,\[𝔨,𝔨\]⊆𝔨\[\\mathfrak\{k\},\\mathfrak\{k\}\]\\subseteq\\mathfrak\{k\},\[𝔨,𝔭\]⊆𝔭\[\\mathfrak\{k\},\\mathfrak\{p\}\]\\subseteq\\mathfrak\{p\}and\[𝔭,𝔭\]⊆𝔨\[\\mathfrak\{p\},\\mathfrak\{p\}\]\\subseteq\\mathfrak\{k\}, which is the Cartan decomposition \([Background D\.27](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1),[Auxiliary result B\.19](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\)\)\. If someξ∈𝔤\\xi\\in\\mathfrak\{g\}hasΛ​ξ⊤​Λ−1∉𝔤\\Lambda\\xi^\{\\top\}\\Lambda^\{\-1\}\\notin\\mathfrak\{g\}, thenϑ​ξ≠±ξ\\vartheta\\xi\\neq\\pm\\xi, soξ\\xiis of neither type\.

*LoRA\.*The generators of the gauge areξX​\(B,A\)=\(B​X,−X​A\)\\xi\_\{X\}\(B,A\)=\(BX,\-XA\)forX∈𝔤​𝔩rX\\in\\mathfrak\{gl\}\_\{r\}\(the derivative ofτ↦e−τ​X⋅\(B,A\)\\tau\\mapsto e^\{\-\\tau X\}\\cdot\(B,A\)\), andX↦ξXX\\mapsto\\xi\_\{X\}is injective\. For per\-rank\-channel ratesΛ⁡\(B,A\)=\(B​DB,DA​A\)\\Lambda\(B,A\)=\(BD\_\{B\},D\_\{A\}A\),

Λ−1​ξX​\(B,A\)=\(B​X​DB−1,−DA−1​X​A\),\(Λ−1​ξX\)⊤​\(B,A\)=\(B​\(X​DB−1\)⊤,−\(DA−1​X\)⊤​A\),\\Lambda^\{\-1\}\\xi\_\{X\}\(B,A\)=\(BXD\_\{B\}^\{\-1\},\\ \-D\_\{A\}^\{\-1\}XA\),\\qquad\(\\Lambda^\{\-1\}\\xi\_\{X\}\)^\{\\top\}\(B,A\)=\\big\(B\(XD\_\{B\}^\{\-1\}\)^\{\\top\},\\ \-\(D\_\{A\}^\{\-1\}X\)^\{\\top\}A\\big\),because⟨B′,B​M⟩=⟨B′​M⊤,B⟩\\langle B^\{\\prime\},BM\\rangle=\\langle B^\{\\prime\}M^\{\\top\},B\\rangleand⟨A′,N​A⟩=⟨N⊤​A′,A⟩\\langle A^\{\\prime\},NA\\rangle=\\langle N^\{\\top\}A^\{\\prime\},A\\rangle\. HenceΛ−1​ξX\\Lambda^\{\-1\}\\xi\_\{X\}is symmetric iffX​DB−1XD\_\{B\}^\{\-1\}andDA−1​XD\_\{A\}^\{\-1\}Xare symmetric, and antisymmetric iff both are antisymmetric\. Entrywise,\(X​DB−1\)i​j=Xi​j/ηB,j\(XD\_\{B\}^\{\-1\}\)\_\{ij\}=X\_\{ij\}/\\eta\_\{B,j\}and\(DA−1​X\)i​j=Xi​j/ηA,i\(D\_\{A\}^\{\-1\}X\)\_\{ij\}=X\_\{ij\}/\\eta\_\{A,i\}\. Fori≠ji\\neq j, the two symmetry conditions readXj​i=Xi​j​ηB,i/ηB,jX\_\{ji\}=X\_\{ij\}\\eta\_\{B,i\}/\\eta\_\{B,j\}andXj​i=Xi​j​ηA,j/ηA,iX\_\{ji\}=X\_\{ij\}\\eta\_\{A,j\}/\\eta\_\{A,i\}; the antisymmetry conditions are the same with a minus sign\. Both admitXi​j≠0X\_\{ij\}\\neq 0iffηB,i/ηB,j=ηA,j/ηA,i\\eta\_\{B,i\}/\\eta\_\{B,j\}=\\eta\_\{A,j\}/\\eta\_\{A,i\}, that is, iffηB,i​ηA,i=ηB,j​ηA,j\\eta\_\{B,i\}\\eta\_\{A,i\}=\\eta\_\{B,j\}\\eta\_\{A,j\}, and then the pair\(i,j\)\(i,j\)carries a one\-dimensional space of each type\. On the diagonal the symmetry conditions are empty and the antisymmetry conditions forceXi​i=0X\_\{ii\}=0\. So for pairwise distinct products the type \(i\) generators are exactly the diagonalXX, and the only type \(ii\) generator isX=0X=0\.

For closure,ξX⊤​\(B,A\)=\(B​X⊤,−X⊤​A\)\\xi\_\{X\}^\{\\top\}\(B,A\)=\(BX^\{\\top\},\-X^\{\\top\}A\), soΛ​ξX⊤​Λ−1​\(B,A\)=\(B​DB−1​X⊤​DB,−DA​X⊤​DA−1​A\)\\Lambda\\xi\_\{X\}^\{\\top\}\\Lambda^\{\-1\}\(B,A\)=\(BD\_\{B\}^\{\-1\}X^\{\\top\}D\_\{B\},\\ \-D\_\{A\}X^\{\\top\}D\_\{A\}^\{\-1\}A\)\. An operator\(B,A\)↦\(B​M,−N​A\)\(B,A\)\\mapsto\(BM,\-NA\)lies in𝔤=\{ξY\}\\mathfrak\{g\}=\\\{\\xi\_\{Y\}\\\}iffM=NM=N, so closure holds iffDB−1​X⊤​DB=DA​X⊤​DA−1D\_\{B\}^\{\-1\}X^\{\\top\}D\_\{B\}=D\_\{A\}X^\{\\top\}D\_\{A\}^\{\-1\}for allXX\. The\(i,j\)\(i,j\)entries areXj​i​ηB,j/ηB,iX\_\{ji\}\\eta\_\{B,j\}/\\eta\_\{B,i\}andXj​i​ηA,i/ηA,jX\_\{ji\}\\eta\_\{A,i\}/\\eta\_\{A,j\}, which agree for allXXiff all productsηB,j​ηA,j\\eta\_\{B,j\}\\eta\_\{A,j\}are equal\. With block\-scalar rates \(DB=ηB​ID\_\{B\}=\\eta\_\{B\}I,DA=ηA​ID\_\{A\}=\\eta\_\{A\}I\) this holds, the type \(i\) generators are the symmetricXXand the type \(ii\) ones the antisymmetricXX, and𝔤​𝔩r=𝔰​𝔬r⊕Symr\\mathfrak\{gl\}\_\{r\}=\\mathfrak\{so\}\_\{r\}\\oplus\\mathrm\{Sym\}\_\{r\}\.

When all products equalpp,DB=p​DA−1D\_\{B\}=pD\_\{A\}^\{\-1\}, and the type \(i\) generators areX=S​DBX=SD\_\{B\}withSSsymmetric \(thenDA−1​X=p​DA−1​S​DA−1D\_\{A\}^\{\-1\}X=pD\_\{A\}^\{\-1\}SD\_\{A\}^\{\-1\}is symmetric too\); the type \(ii\) generators areX=K​DBX=KD\_\{B\}withKKantisymmetric\. WithD=DB1/2D=D\_\{B\}^\{1/2\}, conjugationX↦D​X​D−1X\\mapsto DXD^\{\-1\}mapsS​DBSD\_\{B\}toD​S​DDSDandK​DBKD\_\{B\}toD​K​DDKD, so𝔨⊕𝔭=D−1​\(𝔰​𝔬r⊕Symr\)​D\\mathfrak\{k\}\\oplus\\mathfrak\{p\}=D^\{\-1\}\(\\mathfrak\{so\}\_\{r\}\\oplus\\mathrm\{Sym\}\_\{r\}\)D\. The isometries are the group elementsD−1​O​DD^\{\-1\}ODwithO∈SO⁡\(r\)O\\in\\mathrm\{SO\}\(r\), andD−1​O​DD^\{\-1\}ODis orthogonal iffOOcommutes withD2=DBD^\{2\}=D\_\{B\}\. UnlessDBD\_\{B\}is a multiple of the identity \(the block\-scalar case\), some of these isometries are not orthogonal, so they act onQQby non\-Euclidean maps and the clipping caveat of \(ii\) applies\. ∎

Remark\(instances and the discrete step\)\. Each instance listed after[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)applies \(i\) to the generator named there; we record the checks\. All assume gradient flow, or SGD without momentum, with per\-block rates\.•*LoRA\.*For symmetricXX,JξX​\(B,A\)=12​\(⟨B,B​X⟩/ηB−⟨A,X​A⟩/ηA\)=12​tr⁡\(X​Φ\)J\_\{\\xi\_\{X\}\}\(B,A\)=\\tfrac\{1\}\{2\}\\big\(\\langle B,BX\\rangle/\\eta\_\{B\}\-\\langle A,XA\\rangle/\\eta\_\{A\}\\big\)=\\tfrac\{1\}\{2\}\\tr\(X\\Phi\)withΦ=B⊤​B/ηB−A​A⊤/ηA\\Phi=B^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}, so ther⁡\(r\+1\)/2r\(r\+1\)/2charges are the entries ofΦ\\Phi\. The𝔰​𝔬r\\mathfrak\{so\}\_\{r\}generators act by\(B​O⊤,O​A\)\(BO^\{\\top\},OA\)withO∈SO⁡\(r\)O\\in\\mathrm\{SO\}\(r\), which are Euclidean isometries commuting withΛ\\Lambda, so \(ii\) holds under global\-norm clipping\.•*DyLoRA\.*At each step the loss usesρb​\(B,A\)=W0\+sb​B​Eb​A\\rho\_\{b\}\(B,A\)=W\_\{0\}\+s\_\{b\}BE\_\{b\}Afor a sampledbb, withEb=diag⁡\(1,…,1,0,…,0\)E\_\{b\}=\\diag\(1,\\dots,1,0,\\dots,0\)of rankbb\. The update is−Λ​D​ρb⁡\(t\)⊤​γ\-\\Lambda D\\rho\_\{b\(t\)\}^\{\\top\}\\gamma, and the proof of \(i\) needs onlyD​ρb⁡\(t\)​\(ξ​q\)=0D\\rho\_\{b\(t\)\}\(\\xi q\)=0\. Nowρb​\(g⋅q\)=ρb​\(q\)\\rho\_\{b\}\(g\\cdot q\)=\\rho\_\{b\}\(q\)for all\(B,A\)\(B,A\)iffg−1​Eb​g=Ebg^\{\-1\}E\_\{b\}g=E\_\{b\}, that is, iffggpreservesspan⁡\(e1,…,eb\)\\spanop\(e\_\{1\},\\dots,e\_\{b\}\)andspan⁡\(eb\+1,…,er\)\\spanop\(e\_\{b\+1\},\\dots,e\_\{r\}\)\. When everyb∈\{1,…,r\}b\\in\\\{1,\\dots,r\\\}can be sampled,ggthen preserves each linespan⁡\(eb\)=span⁡\(e1,…,eb\)∩span⁡\(eb,…,er\)\\spanop\(e\_\{b\}\)=\\spanop\(e\_\{1\},\\dots,e\_\{b\}\)\\cap\\spanop\(e\_\{b\},\\dots,e\_\{r\}\), so it is diagonal\. DiagonalXXare symmetric, which gives the chargesdiag⁡Φ\\diag\\Phi\.•*VeRA\.*ρ⁡\(b,d\)=W0\+diag⁡\(b\)​Bf​diag⁡\(d\)​Af\\rho\(b,d\)=W\_\{0\}\+\\diag\(b\)B\_\{\\mathrm\{f\}\}\\diag\(d\)A\_\{\\mathrm\{f\}\}with frozenBf,AfB\_\{\\mathrm\{f\}\},A\_\{\\mathrm\{f\}\}is invariant under\(b,d\)↦\(c​b,d/c\)\(b,d\)\\mapsto\(cb,d/c\), with generator\(b,−d\)\(b,\-d\)and charge12​\(∥b∥2/ηb−∥d∥2/ηd\)\\tfrac\{1\}\{2\}\(\\lVert b\\rVert^\{2\}/\\eta\_\{b\}\-\\lVert d\\rVert^\{2\}/\\eta\_\{d\}\)\. NOLA’s two coefficient vectors behave in the same way\.•*AdaLoRA\.*ρ⁡\(P,λ,Q\)=W0\+P​diag⁡\(λ\)​Q\\rho\(P,\\lambda,Q\)=W\_\{0\}\+P\\diag\(\\lambda\)Qis invariant under\(pk,λk\)↦\(c​pk,λk/c\)\(p\_\{k\},\\lambda\_\{k\}\)\\mapsto\(cp\_\{k\},\\lambda\_\{k\}/c\)and under\(λk,qk\)↦\(λk/c,c​qk\)\(\\lambda\_\{k\},q\_\{k\}\)\\mapsto\(\\lambda\_\{k\}/c,cq\_\{k\}\); at equal rates the charges are proportional to∥pk∥2−λk2\\lVert p\_\{k\}\\rVert^\{2\}\-\\lambda\_\{k\}^\{2\}and∥qk∥2−λk2\\lVert q\_\{k\}\\rVert^\{2\}\-\\lambda\_\{k\}^\{2\}\. The orthogonality regulariser∥P⊤​P−I∥2\+∥Q​Q⊤−I∥2\\lVert P^\{\\top\}P\-I\\rVert^\{2\}\+\\lVert QQ^\{\\top\}\-I\\rVert^\{2\}is not invariant under these scalings, so the regularised loss does not factor throughρ\\rho, and rank\-budget masking changesρ\\rho\.•*HRA\.*ρ⁡\(u1,…,ur\)=W0​∏i\(I−2​ui​ui⊤/∥ui∥2\)\\rho\(u\_\{1\},\\dots,u\_\{r\}\)=W\_\{0\}\\prod\_\{i\}\(I\-2u\_\{i\}u\_\{i\}^\{\\top\}/\\lVert u\_\{i\}\\rVert^\{2\}\)is invariant underui↦c​uiu\_\{i\}\\mapsto cu\_\{i\}, with charge12​∥ui∥2/η\\tfrac\{1\}\{2\}\\lVert u\_\{i\}\\rVert^\{2\}/\\eta\.•*ReLU bottleneck adapters*h↦h\+U​relu⁡\(D​h\+β\)h\\mapsto h\+U\\operatorname\{relu\}\(Dh\+\\beta\)\. Multiplying rowjjofDDandβj\\beta\_\{j\}byc\>0c\>0and columnjjofUUby1/c1/cpreserves the function, sincerelu⁡\(c​z\)=c​relu⁡\(z\)\\operatorname\{relu\}\(cz\)=c\\operatorname\{relu\}\(z\)\. The target is a function space, which[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)allows, and the derivative ofρ\\rhoalong the scaling direction vanishes by homogeneity even at the kinks\. Each hidden unit gives the charge12\(\(∥Dj,:∥2\+βj2\)/ηD−∥U:,j∥2/ηU\)\\tfrac\{1\}\{2\}\\big\(\(\\lVert D\_\{j,:\}\\rVert^\{2\}\+\\beta\_\{j\}^\{2\}\)/\\eta\_\{D\}\-\\lVert U\_\{:,j\}\\rVert^\{2\}/\\eta\_\{U\}\\big\), with a common rate forDDandβ\\beta\.•*TT adapters\.*Neighbouring cores share a bond of dimensionrkr\_\{k\}, on whichGL⁡\(rk\)\\mathrm\{GL\}\(r\_\{k\}\)acts byh−1h^\{\-1\}on one side andhhon the other\. The symmetric generators give one conserved symmetricrk×rkr\_\{k\}\\times r\_\{k\}matrix per bond, the Gram matrix of the left unfolding of one core divided by its rate minus that of the right unfolding of the next core divided by its rate\.•*DoRA’s detached\-norm backward, global\-norm clipping, LoRA dropout\.*DoRA writesW=V​diag⁡\(𝗀/∥V∥c\)W=V\\diag\(\\mathsf\{g\}/\\lVert V\\rVert\_\{c\}\)withV=W0\+s​B​AV=W\_\{0\}\+sBA, column norms∥V∥c\\lVert V\\rVert\_\{c\}and magnitudes𝗀\\mathsf\{g\}\. Treating∥V∥c\\lVert V\\rVert\_\{c\}as a constant in the backward pass gives∇B=s​γ​A⊤\\nabla\_\{B\}=s\\gamma A^\{\\top\}and∇A=s​B⊤​γ\\nabla\_\{A\}=sB^\{\\top\}\\gammawithγ=∇WL​diag⁡\(𝗀/∥V∥c\)\\gamma=\\nabla\_\{W\}L\\,\\diag\(\\mathsf\{g\}/\\lVert V\\rVert\_\{c\}\)\. Clipping multiplies the factor gradients by one scalar, and dropout replacesGGby the masked signal of §[C\.25](https://arxiv.org/html/2610.12728#A3.SS25)\. In each case the update of\(B,A\)\(B,A\)is−Λ​D​ρLoRA⊤​γ\-\\Lambda D\\rho\_\{\\mathrm\{LoRA\}\}^\{\\top\}\\gammafor someγ\\gamma, soΦ\\Phiis conserved\.•*Discrete GD\.*Withssabsorbed intoGG, one stepB\+=B−ηB​G​A⊤B^\{\+\}=B\-\\eta\_\{B\}GA^\{\\top\},A\+=A−ηA​B⊤​GA^\{\+\}=A\-\\eta\_\{A\}B^\{\\top\}Ggives1ηB​B\+⁣⊤​B\+=1ηB​B⊤​B−\(A​G⊤​B\+B⊤​G​A⊤\)\+ηB​A​G⊤​G​A⊤,\\displaystyle\\tfrac\{1\}\{\\eta\_\{B\}\}B^\{\+\\top\}B^\{\+\}=\\tfrac\{1\}\{\\eta\_\{B\}\}B^\{\\top\}B\-\(AG^\{\\top\}B\+B^\{\\top\}GA^\{\\top\}\)\+\\eta\_\{B\}AG^\{\\top\}GA^\{\\top\},1ηA​A\+​A\+⁣⊤=1ηA​A​A⊤−\(B⊤​G​A⊤\+A​G⊤​B\)\+ηA​B⊤​G​G⊤​B,\\displaystyle\\tfrac\{1\}\{\\eta\_\{A\}\}A^\{\+\}A^\{\+\\top\}=\\tfrac\{1\}\{\\eta\_\{A\}\}AA^\{\\top\}\-\(B^\{\\top\}GA^\{\\top\}\+AG^\{\\top\}B\)\+\\eta\_\{A\}B^\{\\top\}GG^\{\\top\}B,soΦ\+−Φ=ηB​A​G⊤​G​A⊤−ηA​B⊤​G​G⊤​B\\Phi^\{\+\}\-\\Phi=\\eta\_\{B\}AG^\{\\top\}GA^\{\\top\}\-\\eta\_\{A\}B^\{\\top\}GG^\{\\top\}Bexactly\. This is of orderη\\etawhileΦ\\Phiis of order1/η1/\\eta, soη​Φ\\eta\\Phi, orΦ\\Phirelative to its size, drifts byO⁡\(η2\)O\(\\eta^\{2\}\)per step\.•*Momentum and Adam\.*A heavy\-ball step adds a buffer of gradients taken at past points,D​ρqs⊤​GsD\\rho\_\{q\_\{s\}\}^\{\\top\}G\_\{s\}withs<ts<t, and Adam rescales gradients coordinatewise\. Neither step has the form−Λ​D​ρq⊤​γ\-\\Lambda D\\rho\_\{q\}^\{\\top\}\\gammaat the currentqqwith a constantΛ\\Lambda, so \(i\) does not apply\.

### C\.29 Corollary III\.6

Corollary III\.6\(charge at initialisation; weight decay\)\.\(a\)Φ0=B0⊤​B0/ηB−A0​A0⊤/ηA\\Phi\_\{0\}=B\_\{0\}^\{\\top\}B\_\{0\}/\\eta\_\{B\}\-A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}is the value of the conserved charge, fixed by the initialisation and the rates\. SinceO⁡\(r\)\\mathrm\{O\}\(r\)acts onΦ\\Phiby conjugation and preservesθ\\theta\-trajectories \([Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(ii\)\), only the spectrum ofΦ0\\Phi\_\{0\}labels aθ\\theta\-trajectory\.\(b\) LetGGbe any cotangent signal, with the scalessabsorbed into it\. Under gradient flow, if decay acts per unit time and identically on both factors,B˙=−ηB​G​A⊤−λ​B,A˙=−ηA​B⊤​G−λ​A,\\dot\{B\}=\-\\eta\_\{B\}GA^\{\\top\}\-\\lambda B,\\qquad\\dot\{A\}=\-\\eta\_\{A\}B^\{\\top\}G\-\\lambda A,thenΦ˙=−2​λ​Φ\\dot\{\\Phi\}=\-2\\lambda\\Phi\. With learning\-rate\-coupled decay \(PyTorch SGD weight\_decay; the decay term of PyTorch AdamW\),B˙=−ηB​\(G​A⊤\+λ​B\)\\dot\{B\}=\-\\eta\_\{B\}\(GA^\{\\top\}\+\\lambda B\)and likewise forAA, so thatΦ˙=−2​λ​\(B⊤​B−A​A⊤\)\\dot\{\\Phi\}=\-2\\lambda\(B^\{\\top\}B\-AA^\{\\top\}\)\. ThenΦ⁡\(t\)=e−2​η​λ​t​Φ​\(0\)\\Phi\(t\)=e^\{\-2\\eta\\lambda t\}\\Phi\(0\)holds whenηA=ηB=η\\eta\_\{A\}=\\eta\_\{B\}=\\eta\. ForηA≠ηB\\eta\_\{A\}\\neq\\eta\_\{B\}it fails in general, since−2​λ​\(B⊤​B−A​A⊤\)\-2\\lambda\(B^\{\\top\}B\-AA^\{\\top\}\)is not a function ofΦ\\Phi\. With an actual Adam\(W\) gradient term,Φ\\Phidrifts even without decay, at first order in the step size; Adam lies outside[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\. So neither law describes AdamW training\.

\(a\)*Conservation and conjugation\.*With block\-scalar rates the entries ofΦ\\Phiare the charges of[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(i\) \(§[C\.28](https://arxiv.org/html/2610.12728#A3.SS28), remark on instances\), so under gradient flowΦ⁡\(t\)=Φ0\\Phi\(t\)=\\Phi\_\{0\}, the value fixed by\(B0,A0\)\(B\_\{0\},A\_\{0\}\)and\(ηA,ηB\)\(\\eta\_\{A\},\\eta\_\{B\}\)\. ForO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\), the mapk:\(B,A\)↦\(B​O,O⊤​A\)k:\(B,A\)\\mapsto\(BO,O^\{\\top\}A\)is the gauge elementg=O⊤g=O^\{\\top\}, and

Φ⁡\(B​O,O⊤​A\)=O⊤​B⊤​BηB​O−O⊤​A​A⊤ηA​O=O⊤​Φ​\(B,A\)​O\.\\Phi\(BO,O^\{\\top\}A\)=O^\{\\top\}\\tfrac\{B^\{\\top\}B\}\{\\eta\_\{B\}\}O\-O^\{\\top\}\\tfrac\{AA^\{\\top\}\}\{\\eta\_\{A\}\}O=O^\{\\top\}\\Phi\(B,A\)\\,O\.The linear mapkkis orthogonal onQQand commutes with block\-scalarΛ\\Lambda, sok​Λ​k⊤=Λk\\Lambda k^\{\\top\}=\\Lambda, andρ∘k=ρ\\rho\\circ k=\\rho\. The trajectory argument of[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(ii\) uses only these two properties \(§[C\.28](https://arxiv.org/html/2610.12728#A3.SS28)\), so theθ\\theta\-trajectories fromq0q\_\{0\}and fromk​q0kq\_\{0\}coincide for every signal that depends onqqonly through\(t,ρ⁡\(q\)\)\(t,\\rho\(q\)\), and also under global\-norm clipping, sincekkis a Euclidean isometry\. This covers all ofO⁡\(r\)\\mathrm\{O\}\(r\), reflections included\. So initialisations whose charges are conjugate underO⁡\(r\)\\mathrm\{O\}\(r\)produce the sameθ\\theta\-trajectory, and a function ofΦ0\\Phi\_\{0\}that labelsθ\\theta\-trajectories must be invariant underΦ0↦O⊤​Φ0​O\\Phi\_\{0\}\\mapsto O^\{\\top\}\\Phi\_\{0\}O\. By the spectral theorem theO⁡\(r\)\\mathrm\{O\}\(r\)\-conjugacy class of the symmetric matrixΦ0\\Phi\_\{0\}is determined by its eigenvalues, so such a function is a function of the spectrum ofΦ0\\Phi\_\{0\}\.

*The table\.*Each entry follows by substituting the initialisation intoΦ0\\Phi\_\{0\}\.

- •*HF default, Linear and Conv\.*B0=0B\_\{0\}=0givesΦ0=−A0A0⊤/ηA\\Phi\_\{0\}=\-A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}\. HF initialisesA0A\_\{0\}with kaiming\_uniform\_ ata=5a=\\sqrt\{5\}, whose bound is2/\(1\+a2\)3/n=n−1/2\\sqrt\{2/\(1\+a^\{2\}\)\}\\sqrt\{3/n\}=n^\{\-1/2\}, withnnthe fan\-in \(in\_channels times the product of the kernel sizes for Conv\)\. An entryU\(−n−1/2,n−1/2\)U\(\-n^\{\-1/2\},n^\{\-1/2\}\)has variance1/\(3​n\)1/\(3n\)and fourth moment1/\(5​n2\)1/\(5n^\{2\}\)\. A diagonal entry ofA0​A0⊤A\_\{0\}A\_\{0\}^\{\\top\}is a sum ofnnindependent squares, with mean1/31/3and variancen⁡\(1/\(5​n2\)−1/\(9​n2\)\)=4/\(45​n\)n\\big\(1/\(5n^\{2\}\)\-1/\(9n^\{2\}\)\\big\)=4/\(45n\); an off\-diagonal entry is a sum ofnnindependent products, with mean00and variancen⋅1/\(9​n2\)=1/\(9​n\)n\\cdot 1/\(9n^\{2\}\)=1/\(9n\)\. So𝔼⁡\[A0​A0⊤\]=13​Ir\\mathbb\{E\}\[A\_\{0\}A\_\{0\}^\{\\top\}\]=\\tfrac\{1\}\{3\}I\_\{r\}, with fluctuations of standard deviation2/\(3​5​n\)2/\(3\\sqrt\{5n\}\)on the diagonal and1/\(3​n\)1/\(3\\sqrt\{n\}\)off it\.
- •*Embeddings, Init\[B\] and mica\.*A0=0A\_\{0\}=0givesΦ0=B0⊤​B0/ηB\\Phi\_\{0\}=B\_\{0\}^\{\\top\}B\_\{0\}/\\eta\_\{B\}\. For mica,BBis frozen, so its rate is00and[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)does not apply to it; the entry records the formula and its sign type, and the training dynamics are those of the one\-sided methodA↦W0\+s​B0​AA\\mapsto W\_\{0\}\+sB\_\{0\}A, along whichB0⊤​B0B\_\{0\}^\{\\top\}B\_\{0\}is constant andA​A⊤AA^\{\\top\}moves\.
- •*EVA\.*B0=0B\_\{0\}=0and orthonormal rows,A0​A0⊤=IA\_\{0\}A\_\{0\}^\{\\top\}=I, giveΦ0=−I/ηA\\Phi\_\{0\}=\-I/\\eta\_\{A\}\.
- •*PiSSA in HF PEFT\.*pissa\_init takes the top\-rrsingular triples\(Ur,Sr,Vr\)\(U\_\{r\},S\_\{r\},V\_\{r\}\)ofW0W\_\{0\}, dividesSrS\_\{r\}by the scaless, and setsA0=\(Sr/s\)1/2​Vr⊤A\_\{0\}=\(S\_\{r\}/s\)^\{1/2\}V\_\{r\}^\{\\top\}andB0=Ur​\(Sr/s\)1/2B\_\{0\}=U\_\{r\}\(S\_\{r\}/s\)^\{1/2\}\. WithUr⊤​Ur=Vr⊤​Vr=IU\_\{r\}^\{\\top\}U\_\{r\}=V\_\{r\}^\{\\top\}V\_\{r\}=I,B0⊤​B0=A0​A0⊤=Sr/sB\_\{0\}^\{\\top\}B\_\{0\}=A\_\{0\}A\_\{0\}^\{\\top\}=S\_\{r\}/s, soΦ0=\(Sr/s\)​\(1/ηB−1/ηA\)\\Phi\_\{0\}=\(S\_\{r\}/s\)\(1/\\eta\_\{B\}\-1/\\eta\_\{A\}\), which vanishes iffηA=ηB\\eta\_\{A\}=\\eta\_\{B\}whenSr≠0S\_\{r\}\\neq 0\. The same computation with the bottom\-rrtriples gives the MiLoRA entry\.
- •*OLoRA\.*B0=QrB\_\{0\}=Q\_\{r\}has orthonormal columns, soΦ0=I/ηB−Rr​Rr⊤/ηA\\Phi\_\{0\}=I/\\eta\_\{B\}\-R\_\{r\}R\_\{r\}^\{\\top\}/\\eta\_\{A\}\.

\(b\) Write both decay conventions asB˙=−ηB​G​A⊤−cB​B\\dot\{B\}=\-\\eta\_\{B\}GA^\{\\top\}\-c\_\{B\}BandA˙=−ηA​B⊤​G−cA​A\\dot\{A\}=\-\\eta\_\{A\}B^\{\\top\}G\-c\_\{A\}A, withcB=cA=λc\_\{B\}=c\_\{A\}=\\lambdaper unit time andcB=ηB​λc\_\{B\}=\\eta\_\{B\}\\lambda,cA=ηA​λc\_\{A\}=\\eta\_\{A\}\\lambdawhen coupled\. Then

dd​t​B⊤​BηB=−\(A​G⊤​B\+B⊤​G​A⊤\)−2​cB​B⊤​BηB,dd​t​A​A⊤ηA=−\(B⊤​G​A⊤\+A​G⊤​B\)−2​cA​A​A⊤ηA,\\tfrac\{d\}\{dt\}\\tfrac\{B^\{\\top\}B\}\{\\eta\_\{B\}\}=\-\(AG^\{\\top\}B\+B^\{\\top\}GA^\{\\top\}\)\-2c\_\{B\}\\tfrac\{B^\{\\top\}B\}\{\\eta\_\{B\}\},\\qquad\\tfrac\{d\}\{dt\}\\tfrac\{AA^\{\\top\}\}\{\\eta\_\{A\}\}=\-\(B^\{\\top\}GA^\{\\top\}\+AG^\{\\top\}B\)\-2c\_\{A\}\\tfrac\{AA^\{\\top\}\}\{\\eta\_\{A\}\},and subtracting, theGG\-terms cancel:

Φ˙=−2​cB​B⊤​BηB\+2​cA​A​A⊤ηA\.\\dot\{\\Phi\}=\-2c\_\{B\}\\tfrac\{B^\{\\top\}B\}\{\\eta\_\{B\}\}\+2c\_\{A\}\\tfrac\{AA^\{\\top\}\}\{\\eta\_\{A\}\}\.Per unit time this isΦ˙=−2​λ​Φ\\dot\{\\Phi\}=\-2\\lambda\\Phi, soΦ⁡\(t\)=e−2​λ​t​Φ​\(0\)\\Phi\(t\)=e^\{\-2\\lambda t\}\\Phi\(0\)\. Coupled, it isΦ˙=−2​λ​\(B⊤​B−A​A⊤\)\\dot\{\\Phi\}=\-2\\lambda\(B^\{\\top\}B\-AA^\{\\top\}\); whenηA=ηB=η\\eta\_\{A\}=\\eta\_\{B\}=\\etathe right\-hand side is−2​λ​η​Φ\-2\\lambda\\eta\\Phi, soΦ⁡\(t\)=e−2​η​λ​t​Φ​\(0\)\\Phi\(t\)=e^\{\-2\\eta\\lambda t\}\\Phi\(0\)\. ForηA≠ηB\\eta\_\{A\}\\neq\\eta\_\{B\}there are states with the sameΦ\\Phiand differentB⊤​B−A​A⊤B^\{\\top\}B\-AA^\{\\top\}: replacing\(B⊤​B,A​A⊤\)\(B^\{\\top\}B,AA^\{\\top\}\)by\(B⊤​B\+ηB​C,A​A⊤\+ηA​C\)\(B^\{\\top\}B\+\\eta\_\{B\}C,AA^\{\\top\}\+\\eta\_\{A\}C\)withC⪰0C\\succeq 0\(realised by suitable factors, sincem,n≥rm,n\\geq r\) leavesΦ\\Phiunchanged and changesB⊤​B−A​A⊤B^\{\\top\}B\-AA^\{\\top\}by\(ηB−ηA\)​C\(\\eta\_\{B\}\-\\eta\_\{A\}\)C\. SinceΦ˙\\dot\{\\Phi\}does not involveGG, two such initial states have equalΦ⁡\(0\)\\Phi\(0\)and differentΦ˙​\(0\)\\dot\{\\Phi\}\(0\), so no lawΦ˙=F⁡\(Φ\)\\dot\{\\Phi\}=F\(\\Phi\), and in particular no exponential law, holds for all trajectories\. Some trajectories do satisfy one: withB=0B=0andG=0G=0,Φ⁡\(t\)=e−2​ηA​λ​t​Φ​\(0\)\\Phi\(t\)=e^\{\-2\\eta\_\{A\}\\lambda t\}\\Phi\(0\); so the law fails in general and not always\.

*Adam\.*One Adam step can moveη​Φ\\eta\\Phiat first order\. Takem=n=r=1m=n=r=1,ssabsorbed intoGG, ratesηB=ηA=η\\eta\_\{B\}=\\eta\_\{A\}=\\eta,\(b,a\)=\(2,1\)\(b,a\)=\(2,1\),G=1G=1and zero optimizer state\. The bias\-corrected first step atε=0\\varepsilon=0isδ​b=−η​sign⁡\(G​a\)=−η\\delta b=\-\\eta\\operatorname\{sign\}\(Ga\)=\-\\etaandδ​a=−η​sign⁡\(b​G\)=−η\\delta a=\-\\eta\\operatorname\{sign\}\(bG\)=\-\\eta, so

η​Φ\+−η​Φ=\(2−η\)2−\(1−η\)2−\(4−1\)=−2​η,\\eta\\Phi^\{\+\}\-\\eta\\Phi=\(2\-\\eta\)^\{2\}\-\(1\-\\eta\)^\{2\}\-\(4\-1\)=\-2\\eta,of first order inη\\eta; the GD step of the previous remark movesη​Φ\\eta\\PhibyO⁡\(η2\)O\(\\eta^\{2\}\)\. Adam’s step is not of the form−Λ​D​ρ⊤​γ\-\\Lambda D\\rho^\{\\top\}\\gamma, so it lies outside[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), and neither decay law describes AdamW training\. ∎

Remark\(the two decay conventions as gauge fixings\)\. Per\-time decay is theΛ\\Lambda\-gradient flow ofL\+λ2​\(∥B∥2/ηB\+∥A∥2/ηA\)L\+\\tfrac\{\\lambda\}\{2\}\\big\(\\lVert B\\rVert^\{2\}/\\eta\_\{B\}\+\\lVert A\\rVert^\{2\}/\\eta\_\{A\}\\big\), since−ηB​∇B\(λ2​∥B∥2/ηB\)=−λ​B\-\\eta\_\{B\}\\nabla\_\{B\}\\big\(\\tfrac\{\\lambda\}\{2\}\\lVert B\\rVert^\{2\}/\\eta\_\{B\}\\big\)=\-\\lambda B\. Coupled decay is theΛ\\Lambda\-gradient flow ofL\+λ2​\(∥B∥2\+∥A∥2\)L\+\\tfrac\{\\lambda\}\{2\}\(\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}\)\. At an equilibrium of the coupled flow,∇B=−λ​B\\nabla\_\{B\}=\-\\lambda Band∇A=−λ​A\\nabla\_\{A\}=\-\\lambda A; sinceB⊤​∇B=s​B⊤​G​A⊤=∇AA⊤B^\{\\top\}\\nabla\_\{B\}=sB^\{\\top\}GA^\{\\top\}=\\nabla\_\{A\}A^\{\\top\}, this givesλ⁡\(B⊤​B−A​A⊤\)=0\\lambda\(B^\{\\top\}B\-AA^\{\\top\}\)=0, so equilibria lie on the unweighted balanced sliceμ=B⊤​B−A​A⊤=0\\mu=B^\{\\top\}B\-AA^\{\\top\}=0\. At an equilibrium of the per\-time flow,ηB​∇B=−λ​B\\eta\_\{B\}\\nabla\_\{B\}=\-\\lambda BandηA​∇A=−λ​A\\eta\_\{A\}\\nabla\_\{A\}=\-\\lambda A, and the same identity givesΦ=0\\Phi=0; per\-time decay also drivesΦ→0\\Phi\\to 0exponentially by \(b\)\. The two slices\{B⊤B=AA⊤\}\\\{B^\{\\top\}B=AA^\{\\top\}\\\}and\{B⊤B=\(ηB/ηA\)AA⊤\}\\\{B^\{\\top\}B=\(\\eta\_\{B\}/\\eta\_\{A\}\)AA^\{\\top\}\\\}coincide iffηA=ηB\\eta\_\{A\}=\\eta\_\{B\}\. HF’s PiSSA hasB0⊤​B0=A0​A0⊤B\_\{0\}^\{\\top\}B\_\{0\}=A\_\{0\}A\_\{0\}^\{\\top\}at every rate, so it starts on the coupled\-decay sliceμ=0\\mu=0, and off the per\-time sliceΦ=0\\Phi=0whenηA≠ηB\\eta\_\{A\}\\neq\\eta\_\{B\}\.

### C\.30 Proposition III\.7

Proposition III\.7\(Kempf–Ness slice; Loewner floor; known,[Kempf and Ness \(1979\)](https://arxiv.org/html/2610.12728#bib.bib209);[Srebro et al\. \(2004\)](https://arxiv.org/html/2610.12728#bib.bib426)\)\.\(a\) On a full\-rankGLr\\mathrm\{GL\}\_\{r\}\-orbit, the critical points of∥B∥2\+∥A∥2\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}are exactly the balanced representatives, those withμ=B⊤​B−A​A⊤=0\\mu=B^\{\\top\}B\-AA^\{\\top\}=0\. They form oneO⁡\(r\)\\mathrm\{O\}\(r\)\-orbit, they are the minima, and the minimum value is2​∥B​A∥∗2\\lVert BA\\rVert\_\{\*\}\.\(b\) Assume gradient flow with block\-scalar ratesηA,ηB\\eta\_\{A\},\\eta\_\{B\}, no weight decay, a loss that depends on\(B,A\)\(B,A\)only throughB​ABA, and any cotangent signal of the formΛ​D​ρ⊤​γ\\Lambda D\\rho^\{\\top\}\\gamma\(dropout, minibatches and clipping included\)\. FromB0=0B\_\{0\}=0,B⊤​B=ηBηA​\(A​A⊤−A0​A0⊤\)⪰0,B^\{\\top\}B=\\tfrac\{\\eta\_\{B\}\}\{\\eta\_\{A\}\}\\big\(AA^\{\\top\}\-A\_\{0\}A\_\{0\}^\{\\top\}\\big\)\\succeq 0,soA​A⊤⪰A0​A0⊤AA^\{\\top\}\\succeq A\_\{0\}A\_\{0\}^\{\\top\}for all time; theAA\-factor never shrinks below its initialisation in the Loewner order\.A​A⊤​\(t\)AA^\{\\top\}\(t\)need not be monotone intt; only this floor is guaranteed\. Discrete steps break the bound byO⁡\(η2\)O\(\\eta^\{2\}\)per step\. Under Adam it is not guaranteed and fails empirically\. With per\-time decayλ\\lambda, onlyA​A⊤⪰e−2​λ​t​A0​A0⊤AA^\{\\top\}\\succeq e^\{\-2\\lambda t\}A\_\{0\}A\_\{0\}^\{\\top\}holds\.

\(a\) Letx=\(B,A\)x=\(B,A\)haverank⁡B=rank⁡A=r\\rk B=\\rk A=r, putf⁡\(B,A\)=∥B∥2\+∥A∥2f\(B,A\)=\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}, and let𝒪=GLr⋅x\\mathcal\{O\}=\\mathrm\{GL\}\_\{r\}\\cdot x\. Every factorisationB′​A′B^\{\\prime\}A^\{\\prime\}ofB​ABAwith inner dimensionrrlies in𝒪\\mathcal\{O\}: fromcol⁡B′=col⁡\(B​A\)=col⁡B\\operatorname\{col\}B^\{\\prime\}=\\operatorname\{col\}\(BA\)=\\operatorname\{col\}Bwe getB′=B​g−1B^\{\\prime\}=Bg^\{\-1\}for a uniqueg∈GLrg\\in\\mathrm\{GL\}\_\{r\}, and thenB​g−1​A′=B​ABg^\{\-1\}A^\{\\prime\}=BAgivesA′=g​AA^\{\\prime\}=gAbecauseBBis injective\. As in Kempf–Ness theory \([Background D\.30](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1),[Auxiliary result B\.26](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\),xxis a critical point offfon𝒪\\mathcal\{O\}whene∈GLre\\in\\mathrm\{GL\}\_\{r\}is a critical point ofg↦f⁡\(g⋅x\)g\\mapsto f\(g\\cdot x\)\. Withet​ξ⋅x=\(B​e−t​ξ,et​ξ​A\)e^\{t\\xi\}\\cdot x=\(Be^\{\-t\\xi\},e^\{t\\xi\}A\),

dd​t\|t=0​f​\(et​ξ⋅x\)=−2​⟨B,B​ξ⟩\+2​⟨A,ξ​A⟩=−2​tr⁡\(ξ⁡\(B⊤​B−A​A⊤\)\)=−2​tr⁡\(ξs​μ\),\\tfrac\{d\}\{dt\}\\Big\|\_\{t=0\}f\(e^\{t\\xi\}\\cdot x\)=\-2\\langle B,B\\xi\\rangle\+2\\langle A,\\xi A\\rangle=\-2\\tr\\big\(\\xi\(B^\{\\top\}B\-AA^\{\\top\}\)\\big\)=\-2\\tr\(\\xi\_\{s\}\\mu\),whereξs\\xi\_\{s\}is the symmetric part ofξ\\xiand the antisymmetric part drops out becauseμ\\muis symmetric\. This vanishes for allξ\\xiiffμ=0\\mu=0\(takeξ=μ\\xi=\\mu\)\. So the critical points are exactly the balanced points of𝒪\\mathcal\{O\}\.

*Convexity along symmetric directions\.*The functionffisO⁡\(r\)\\mathrm\{O\}\(r\)\-invariant,f⁡\(B​O⊤,O​A\)=f⁡\(B,A\)f\(BO^\{\\top\},OA\)=f\(B,A\), andμ\\muisO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant,μ⁡\(O⋅x\)=O​μ​\(x\)​O⊤\\mu\(O\\cdot x\)=O\\mu\(x\)O^\{\\top\}\. By the polar decomposition everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}isg=O​eξg=Oe^\{\\xi\}withO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\)andξ\\xisymmetric \([Background D\.27](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx4.Px1),[Auxiliary result B\.19](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\);[Horn and Johnson, 2013](https://arxiv.org/html/2610.12728#bib.bib174), §7\.3\), so every point of𝒪\\mathcal\{O\}isO⋅\(eξ⋅x\)O\\cdot\(e^\{\\xi\}\\cdot x\)andf⁡\(g⋅x\)=f⁡\(eξ⋅x\)f\(g\\cdot x\)=f\(e^\{\\xi\}\\cdot x\)\. For symmetricξ=R​diag⁡\(d\)​R⊤\\xi=R\\diag\(d\)R^\{\\top\}withRRorthogonal, puth⁡\(t\)=f⁡\(et​ξ⋅x\)h\(t\)=f\(e^\{t\\xi\}\\cdot x\)\. Orthogonal invariance of the Frobenius norm gives

h\(t\)=∥BRe−t​diag⁡\(d\)∥2\+∥et​diag⁡\(d\)R⊤A∥2=∑i\(∥\(BR\):,i∥2e−2​t​di\+∥\(R⊤A\)i,:∥2e2​t​di\),h\(t\)=\\lVert BRe^\{\-t\\diag\(d\)\}\\rVert^\{2\}\+\\lVert e^\{t\\diag\(d\)\}R^\{\\top\}A\\rVert^\{2\}=\\sum\_\{i\}\\Big\(\\lVert\(BR\)\_\{:,i\}\\rVert^\{2\}e^\{\-2td\_\{i\}\}\+\\lVert\(R^\{\\top\}A\)\_\{i,:\}\\rVert^\{2\}e^\{2td\_\{i\}\}\\Big\),h′′\(t\)=4∑idi2\(∥\(BR\):,i∥2e−2​t​di\+∥\(R⊤A\)i,:∥2e2​t​di\)≥0\.h^\{\\prime\\prime\}\(t\)=4\\sum\_\{i\}d\_\{i\}^\{2\}\\Big\(\\lVert\(BR\)\_\{:,i\}\\rVert^\{2\}e^\{\-2td\_\{i\}\}\+\\lVert\(R^\{\\top\}A\)\_\{i,:\}\\rVert^\{2\}e^\{2td\_\{i\}\}\\Big\)\\geq 0\.Sohhis convex\.

*Balanced points are the minima\.*Letxxbe balanced andy=O⋅\(eξ⋅x\)∈𝒪y=O\\cdot\(e^\{\\xi\}\\cdot x\)\\in\\mathcal\{O\}\. Thenh′​\(0\)=−2​tr⁡\(ξ​μ​\(x\)\)=0h^\{\\prime\}\(0\)=\-2\\tr\(\\xi\\mu\(x\)\)=0, and convexity givesf⁡\(y\)=h⁡\(1\)≥h⁡\(0\)=f⁡\(x\)f\(y\)=h\(1\)\\geq h\(0\)=f\(x\)\. So every balanced point is a global minimum on𝒪\\mathcal\{O\}; conversely every minimum is a critical point, hence balanced\.

*OneO⁡\(r\)\\mathrm\{O\}\(r\)\-orbit\.*Letxxandy=O⋅\(eξ⋅x\)y=O\\cdot\(e^\{\\xi\}\\cdot x\)both be balanced\. Theneξ⋅x=O−1⋅ye^\{\\xi\}\\cdot x=O^\{\-1\}\\cdot yis balanced by equivariance ofμ\\mu, soh′​\(0\)=0h^\{\\prime\}\(0\)=0andh′​\(1\)=−2​tr⁡\(ξ​μ​\(eξ⋅x\)\)=0h^\{\\prime\}\(1\)=\-2\\tr\\big\(\\xi\\mu\(e^\{\\xi\}\\cdot x\)\\big\)=0\. Sinceh′h^\{\\prime\}is non\-decreasing,h′≡0h^\{\\prime\}\\equiv 0on\[0,1\]\[0,1\], soh′′​\(0\)=0h^\{\\prime\\prime\}\(0\)=0\. Every column\(BR\):,i\(BR\)\_\{:,i\}is nonzero becauseB​RBRhas full column rank, so alldi=0d\_\{i\}=0,ξ=0\\xi=0andy=O⋅xy=O\\cdot x\. Conversely theO⁡\(r\)\\mathrm\{O\}\(r\)\-orbit of a balanced point consists of balanced points\.

*Existence and value\.*LetB​A=U​Σ​V⊤BA=U\\Sigma V^\{\\top\}be a compact SVD \([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\) withΣ∈ℝr×r\\Sigma\\in\\mathbb\{R\}^\{r\\times r\}positive diagonal\. The factorisation\(U​Σ1/2,Σ1/2​V⊤\)\(U\\Sigma^\{1/2\},\\Sigma^\{1/2\}V^\{\\top\}\)lies in𝒪\\mathcal\{O\}by the first paragraph, is balanced since both Gram matrices equalΣ\\Sigma, and hasf=2​tr⁡Σ=2​∥B​A∥∗f=2\\tr\\Sigma=2\\lVert BA\\rVert\_\{\*\}\. This is the minimum value\.

\(b\) Under these hypotheses the update has the form of[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), soΦ\\Phiis conserved\. WithB0=0B\_\{0\}=0,Φ0=−A0A0⊤/ηA\\Phi\_\{0\}=\-A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}, andB⊤B/ηB−AA⊤/ηA=−A0A0⊤/ηAB^\{\\top\}B/\\eta\_\{B\}\-AA^\{\\top\}/\\eta\_\{A\}=\-A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}rearranges toB⊤​B=ηBηA​\(A​A⊤−A0​A0⊤\)B^\{\\top\}B=\\tfrac\{\\eta\_\{B\}\}\{\\eta\_\{A\}\}\(AA^\{\\top\}\-A\_\{0\}A\_\{0\}^\{\\top\}\)\. SinceB⊤​B⪰0B^\{\\top\}B\\succeq 0,A​A⊤⪰A0​A0⊤AA^\{\\top\}\\succeq A\_\{0\}A\_\{0\}^\{\\top\}\.

*No monotonicity\.*Takem=n=r=1m=n=r=1,a⁡\(0\)=a0\>0a\(0\)=a\_\{0\}\>0,b⁡\(0\)=0b\(0\)=0and the signalγ=−1\\gamma=\-1on\[0,1\)\[0,1\)andγ=\+1\\gamma=\+1on\[1,2\]\[1,2\], the gradients of the minibatch losses−w\-wandww\. The flow isb˙=−ηB​s​γ​a\\dot\{b\}=\-\\eta\_\{B\}s\\gamma a,a˙=−ηA​s​γ​b\\dot\{a\}=\-\\eta\_\{A\}s\\gamma b\. On\[0,1\]\[0,1\],a⁡\(t\)=a0​cosh⁡\(s​ηA​ηB​t\)a\(t\)=a\_\{0\}\\cosh\(s\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}\\,t\)increases\. On\[1,2\]\[1,2\]the system is the time reversal of the first one, so it retraces its path and returns to\(a0,0\)\(a\_\{0\},0\)att=2t=2, andaadecreases\. The floora2≥a02a^\{2\}\\geq a\_\{0\}^\{2\}holds throughout, buta2a^\{2\}is not monotone\.

*Discrete steps\.*By the discrete formula of §[C\.28](https://arxiv.org/html/2610.12728#A3.SS28), afterkkGD steps \(withssabsorbed intoGjG\_\{j\}\)

Ak​Ak⊤\\displaystyle A\_\{k\}A\_\{k\}^\{\\top\}=ηA​\(1ηB​Bk⊤​Bk−Φk\)⪰−ηA​Φk=A0​A0⊤−∑j<k\(ηA​ηB​Aj​Gj⊤​Gj​Aj⊤−ηA2​Bj⊤​Gj​Gj⊤​Bj\)\\displaystyle=\\eta\_\{A\}\\Big\(\\tfrac\{1\}\{\\eta\_\{B\}\}B\_\{k\}^\{\\top\}B\_\{k\}\-\\Phi\_\{k\}\\Big\)\\succeq\-\\eta\_\{A\}\\Phi\_\{k\}=A\_\{0\}A\_\{0\}^\{\\top\}\-\\sum\_\{j<k\}\\big\(\\eta\_\{A\}\\eta\_\{B\}A\_\{j\}G\_\{j\}^\{\\top\}G\_\{j\}A\_\{j\}^\{\\top\}\-\\eta\_\{A\}^\{2\}B\_\{j\}^\{\\top\}G\_\{j\}G\_\{j\}^\{\\top\}B\_\{j\}\\big\)⪰A0​A0⊤−ηA​ηB​∑j<kAj​Gj⊤​Gj​Aj⊤,\\displaystyle\\succeq A\_\{0\}A\_\{0\}^\{\\top\}\-\\eta\_\{A\}\\eta\_\{B\}\\sum\_\{j<k\}A\_\{j\}G\_\{j\}^\{\\top\}G\_\{j\}A\_\{j\}^\{\\top\},so the floor holds up to an error of orderη2\\eta^\{2\}per step\. The identity itself breaks at that order already at the first step, whereB1=−ηB​G0​A0⊤B\_\{1\}=\-\\eta\_\{B\}G\_\{0\}A\_\{0\}^\{\\top\}andA1=A0A\_\{1\}=A\_\{0\}giveB1⊤​B1=ηB2​A0​G0⊤​G0​A0⊤B\_\{1\}^\{\\top\}B\_\{1\}=\\eta\_\{B\}^\{2\}A\_\{0\}G\_\{0\}^\{\\top\}G\_\{0\}A\_\{0\}^\{\\top\}againstηBηA​\(A1​A1⊤−A0​A0⊤\)=0\\tfrac\{\\eta\_\{B\}\}\{\\eta\_\{A\}\}\(A\_\{1\}A\_\{1\}^\{\\top\}\-A\_\{0\}A\_\{0\}^\{\\top\}\)=0\.

*Adam\.*Takem=n=r=1m=n=r=1,s=1s=1,\(b0,a0\)=\(0,1\)\(b\_\{0\},a\_\{0\}\)=\(0,1\), minibatch gradientsG1=1G\_\{1\}=1andG2=−1G\_\{2\}=\-1, anyε≥0\\varepsilon\\geq 0and bias correction\. At step 1,∇a=b0​G1=0\\nabla\_\{a\}=b\_\{0\}G\_\{1\}=0, soa1=1a\_\{1\}=1, andb1=−ηB​ϕ​\(G1​a0\)b\_\{1\}=\-\\eta\_\{B\}\\phi\(G\_\{1\}a\_\{0\}\)has sign−1\-1\. At step 2,∇a=b1​G2\>0\\nabla\_\{a\}=b\_\{1\}G\_\{2\}\>0is the first nonzero gradient ofaa, so Adam’s step onaais−ηAm^/\(v^\+ε\)\-\\eta\_\{A\}\\hat\{m\}/\(\\sqrt\{\\hat\{v\}\}\+\\varepsilon\)withm^=∇a/\(1\+β1\)\>0\\hat\{m\}=\\nabla\_\{a\}/\(1\+\\beta\_\{1\}\)\>0; atε=0\\varepsilon=0it equals−ηA1\+β2/\(1\+β1\)\-\\eta\_\{A\}\\sqrt\{1\+\\beta\_\{2\}\}/\(1\+\\beta\_\{1\}\)\. Soa2<1=a0a\_\{2\}<1=a\_\{0\}for everyε≥0\\varepsilon\\geq 0, and atε=0\\varepsilon=0the violation is of first order inηA\\eta\_\{A\}\.

*Per\-time decay\.*By[Corollary III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\(b\),Φ\(t\)=e−2​λ​tΦ\(0\)=−e−2​λ​tA0A0⊤/ηA\\Phi\(t\)=e^\{\-2\\lambda t\}\\Phi\(0\)=\-e^\{\-2\\lambda t\}A\_\{0\}A\_\{0\}^\{\\top\}/\\eta\_\{A\}, soA​A⊤=e−2​λ​t​A0​A0⊤\+ηAηB​B⊤​B⪰e−2​λ​t​A0​A0⊤AA^\{\\top\}=e^\{\-2\\lambda t\}A\_\{0\}A\_\{0\}^\{\\top\}\+\\tfrac\{\\eta\_\{A\}\}\{\\eta\_\{B\}\}B^\{\\top\}B\\succeq e^\{\-2\\lambda t\}A\_\{0\}A\_\{0\}^\{\\top\}\. ∎

Remark\(nuclear\-norm penalties\)\. By \(a\),min⁡\{λ⁡\(∥B∥2\+∥A∥2\):s​B​A=Δ​W\}=2​λ​∥Δ​W∥∗/s\\min\\\{\\lambda\(\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}\):sBA=\\Delta W\\\}=2\\lambda\\lVert\\Delta W\\rVert\_\{\*\}/sforΔ​W\\Delta Wof rankrr;[Srebro et al\. \(2004\)](https://arxiv.org/html/2610.12728#bib.bib426)show \([Auxiliary result B\.21](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\) thatmin⁡\{12​\(∥B∥2\+∥A∥2\):B​A=Z\}=∥Z∥∗\\min\\\{\\tfrac\{1\}\{2\}\(\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}\):BA=Z\\\}=\\lVert Z\\rVert\_\{\*\}for every inner dimension at leastrank⁡Z\\rk Z, which covers rank\-deficientΔ​W\\Delta W\. So the explicit penaltyλ⁡\(∥B∥2\+∥A∥2\)\\lambda\(\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}\)matches a\(2​λ/s\)\(2\\lambda/s\)\-nuclear\-norm penalty at its minimisers, and coupled decay with coefficientλ\\lambda, the flow ofL\+λ2​\(∥B∥2\+∥A∥2\)L\+\\tfrac\{\\lambda\}\{2\}\(\\lVert B\\rVert^\{2\}\+\\lVert A\\rVert^\{2\}\), matches a\(λ/s\)\(\\lambda/s\)\-nuclear\-norm penalty\.

### C\.31 Proposition III\.8

Proposition III\.8\(the balance–escape trilemma\)\.No LoRA initialisation is simultaneously\(a\)identity\-at\-init without a residual split \(B0​A0=0B\_\{0\}A\_\{0\}=0\),\(b\)balanced \(B0⊤​B0=c​A0​A0⊤B\_\{0\}^\{\\top\}B\_\{0\}=cA\_\{0\}A\_\{0\}^\{\\top\}for somec\>0c\>0: unweighted,c=1c=1, or rate\-weighted,Φ0=0\\Phi\_\{0\}=0withc=ηB/ηAc=\\eta\_\{B\}/\\eta\_\{A\}\), and\(c\)non\-stationary, meaning thatD​ρD\\rhoat\(B0,A0\)\(B\_\{0\},A\_\{0\}\)is not the zero map; equivalently,W˙​\(0\)≠0\\dot\{W\}\(0\)\\neq 0for some parameter velocity, under any first\-order optimizer\.Moreover, ifB0​A0=0B\_\{0\}A\_\{0\}=0then⟨B0⊤​B0,A0​A0⊤⟩F=0\\langle B\_\{0\}^\{\\top\}B\_\{0\},A\_\{0\}A\_\{0\}^\{\\top\}\\rangle\_\{F\}=0, so∥B0⊤​B0−c​A0​A0⊤∥F2=∥B0⊤​B0∥F2\+c2​∥A0​A0⊤∥F2,\\lVert B\_\{0\}^\{\\top\}B\_\{0\}\-cA\_\{0\}A\_\{0\}^\{\\top\}\\rVert\_\{F\}^\{2\}=\\lVert B\_\{0\}^\{\\top\}B\_\{0\}\\rVert\_\{F\}^\{2\}\+c^\{2\}\\lVert A\_\{0\}A\_\{0\}^\{\\top\}\\rVert\_\{F\}^\{2\},and zero\-product initialisations are maximally unbalanced\.

Assume \(a\) and \(b\)\. Then0=B0⊤​\(B0​A0\)=\(B0⊤​B0\)​A0=c​A0​A0⊤​A00=B\_\{0\}^\{\\top\}\(B\_\{0\}A\_\{0\}\)=\(B\_\{0\}^\{\\top\}B\_\{0\}\)A\_\{0\}=cA\_\{0\}A\_\{0\}^\{\\top\}A\_\{0\}, and multiplying byA0⊤A\_\{0\}^\{\\top\}on the left givesc​\(A0⊤​A0\)2=0c\(A\_\{0\}^\{\\top\}A\_\{0\}\)^\{2\}=0\. The matrixA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}is symmetric positive semidefinite with vanishing square, so its eigenvalues vanish andA0⊤​A0=0A\_\{0\}^\{\\top\}A\_\{0\}=0, that is,A0=0A\_\{0\}=0\. ThenB0⊤​B0=c​A0​A0⊤=0B\_\{0\}^\{\\top\}B\_\{0\}=cA\_\{0\}A\_\{0\}^\{\\top\}=0, soB0=0B\_\{0\}=0\. At\(0,0\)\(0,0\),D​ρ​\(B˙,A˙\)=s⁡\(B˙⋅0\+0⋅A˙\)=0D\\rho\(\\dot\{B\},\\dot\{A\}\)=s\(\\dot\{B\}\\cdot 0\+0\\cdot\\dot\{A\}\)=0, which contradicts \(c\)\. The equivalence in \(c\) is the definition of the zero map: some parameter velocity hasW˙=D​ρ​\(B˙,A˙\)≠0\\dot\{W\}=D\\rho\(\\dot\{B\},\\dot\{A\}\)\\neq 0iffD​ρ≠0D\\rho\\neq 0\. At\(0,0\)\(0,0\)also∇B=s​G​A⊤=0\\nabla\_\{B\}=sGA^\{\\top\}=0and∇A=s​B⊤​G=0\\nabla\_\{A\}=sB^\{\\top\}G=0, so every update rule built from these gradients and the current parameters, including Adam, momentum and weight decay started from zero state, stays at\(0,0\)\(0,0\)\.

For the identity,⟨B0⊤​B0,A0​A0⊤⟩F=tr⁡\(B0⊤​B0​A0​A0⊤\)=tr⁡\(B0⊤​\(B0​A0\)​A0⊤\)=0\\langle B\_\{0\}^\{\\top\}B\_\{0\},A\_\{0\}A\_\{0\}^\{\\top\}\\rangle\_\{F\}=\\tr\(B\_\{0\}^\{\\top\}B\_\{0\}A\_\{0\}A\_\{0\}^\{\\top\}\)=\\tr\\big\(B\_\{0\}^\{\\top\}\(B\_\{0\}A\_\{0\}\)A\_\{0\}^\{\\top\}\\big\)=0, and expanding the square gives the displayed formula\. For positive semidefinitePPandNN,⟨P,N⟩F=tr⁡\(P1/2​N​P1/2\)≥0\\langle P,N\\rangle\_\{F\}=\\tr\(P^\{1/2\}NP^\{1/2\}\)\\geq 0, so∥P−c​N∥F2≤∥P∥F2\+c2​∥N∥F2\\lVert P\-cN\\rVert\_\{F\}^\{2\}\\leq\\lVert P\\rVert\_\{F\}^\{2\}\+c^\{2\}\\lVert N\\rVert\_\{F\}^\{2\}, with equality iff⟨P,N⟩F=0\\langle P,N\\rangle\_\{F\}=0\. Zero\-product initialisations attain this maximum for the given norms of the two Gram matrices\. ∎

### C\.32 Theorem III\.9

Theorem III\.9\(closed\-form isotropic\-charge LoRA dynamics; known,[Tarmoun et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib447)and[Min et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib328)\)\.Letρ⁡\(B,A\)=Wres\+s​B​A\\rho\(B,A\)=W\_\{\\mathrm\{res\}\}\+sBAbe trained by gradient flow with ratesηA,ηB\\eta\_\{A\},\\eta\_\{B\}, and suppose the charge is isotropic,Φ=−κ​Ir\\Phi=\-\\kappa I\_\{r\}\. Putμ=s​ηA​ηB\\mu=s\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}\(a rate, unrelated to the moment mapμ\\muof[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5)\), and on the locus whereZ=W−WresZ=W\-W\_\{\\mathrm\{res\}\}has rankrrwrite its compact SVDZ=U​Σ​V⊤Z=U\\Sigma V^\{\\top\}\. ThenW˙=−μ2​\[G​V​gκ​\(Σ2/μ2\)​V⊤\+U​fκ​\(Σ2/μ2\)​U⊤​G\],fκ​\(x\)=x\+κ2/4−κ/2,gκ=fκ\+κ\.\\dot\{W\}=\-\\mu^\{2\}\\big\[G\\,Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}\+Uf\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)U^\{\\top\}G\\big\],\\qquad f\_\{\\kappa\}\(x\)=\\sqrt\{x\+\\kappa^\{2\}/4\}\-\\kappa/2,\\quad g\_\{\\kappa\}=f\_\{\\kappa\}\+\\kappa\.The identity holds pointwise on\{Φ=−κ​I,rank⁡Z=r\}\\\{\\Phi=\-\\kappa I,\\ \\rk Z=r\\\}; gradient flow is used only to keepΦ\\Phifixed\. The two terms under the root are equal at the*crossover scale*σ∗=μ​\|κ\|/2\\sigma^\{\*\}=\\mu\\lvert\\kappa\\rvert/2\.•Forσ≪σ∗\\sigma\\ll\\sigma^\{\*\}withκ\>0\\kappa\>0,f→0f\\to 0andg→κg\\to\\kappa, soW˙≈−μ2​κ​G​PV\\dot\{W\}\\approx\-\\mu^\{2\}\\kappa\\,GP\_\{V\}\. These are LoRA\-FA dynamics\. The contribution ofAAtoW˙\\dot\{W\}isO⁡\(σ2/\(μ2​κ\)\)O\(\\sigma^\{2\}/\(\\mu^\{2\}\\kappa\)\), althoughAAitself still drifts slowly\. Forκ<0\\kappa<0the roles swap,W˙≈−μ2​\|κ\|​PU​G\\dot\{W\}\\approx\-\\mu^\{2\}\\lvert\\kappa\\rvert P\_\{U\}G, and onlyAAeffectively moves \(Init\[B\]\)\.•Forσ≫σ∗\\sigma\\gg\\sigma^\{\*\},f≈g≈σ/μf\\approx g\\approx\\sigma/\\mu, soW˙≈−μ⁡\[G​\(Z⊤​Z\)1/2\+\(Z​Z⊤\)1/2​G\]\\dot\{W\}\\approx\-\\mu\\big\[G\(Z^\{\\top\}Z\)^\{1/2\}\+\(ZZ^\{\\top\}\)^\{1/2\}G\\big\]\. These are the balanced dynamics of Arora, Cohen and Hazan\.•*Off the rank\-rrlocus\.*WriteW˙=−μ2​\(G​S\+T​G\)\\dot\{W\}=\-\\mu^\{2\}\(GS\+TG\)withS=A⊤​A/ηAS=A^\{\\top\}A/\\eta\_\{A\}andT=B​B⊤/ηBT=BB^\{\\top\}/\\eta\_\{B\}, so that on the rank\-rrlocusS=V​gκ​\(Σ2/μ2\)​V⊤S=Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}andT=U​fκ​\(Σ2/μ2\)​U⊤T=Uf\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)U^\{\\top\}\. Forκ≥0\\kappa\\geq 0,T=fκ​\(Z​Z⊤/μ2\)T=f\_\{\\kappa\}\(ZZ^\{\\top\}/\\mu^\{2\}\)still holds as a full matrix function, butSSgains the termκ⁡\(Prow⁡A−PV\)\\kappa\(P\_\{\\operatorname\{row\}A\}\-P\_\{V\}\); symmetrically, forκ<0\\kappa<0,TTgains\|κ\|​\(Pcol⁡B−PU\)\\lvert\\kappa\\rvert\(P\_\{\\operatorname\{col\}B\}\-P\_\{U\}\)\. At the default\-LoRA startZ=0Z=0\(B0=0B\_\{0\}=0\) this leavesS=κ​Prow⁡A0S=\\kappa P\_\{\\operatorname\{row\}A\_\{0\}\}andT=0T=0, soW˙=−μ2​κ​G​Prow⁡A0\\dot\{W\}=\-\\mu^\{2\}\\kappa\\,GP\_\{\\operatorname\{row\}A\_\{0\}\}, and theσ≪σ∗\\sigma\\ll\\sigma^\{\*\}regime extends continuously tot=0t=0\.

LetGGbe the signal driving the flow,B˙=−ηB​s​G​A⊤\\dot\{B\}=\-\\eta\_\{B\}sGA^\{\\top\}andA˙=−ηA​s​B⊤​G\\dot\{A\}=\-\\eta\_\{A\}sB^\{\\top\}G; under gradient flowG=∇θLG=\\nabla\_\{\\theta\}L, and[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\(i\) keepsΦ=−κ​I\\Phi=\-\\kappa Ifor all time if it holds att=0t=0\. Everything below uses only the value ofΦ\\Phiat the point considered\.

*Normalisation\.*Putβ=B/ηB\\beta=B/\\sqrt\{\\eta\_\{B\}\}andα=A/ηA\\alpha=A/\\sqrt\{\\eta\_\{A\}\}\. Thenβ⊤​β−α​α⊤=Φ=−κ​I\\beta^\{\\top\}\\beta\-\\alpha\\alpha^\{\\top\}=\\Phi=\-\\kappa Iandβ​α=B​A/ηA​ηB=Z/μ\\beta\\alpha=BA/\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}=Z/\\mu\. LetT=β​β⊤=B​B⊤/ηBT=\\beta\\beta^\{\\top\}=BB^\{\\top\}/\\eta\_\{B\}andS=α⊤​α=A⊤​A/ηAS=\\alpha^\{\\top\}\\alpha=A^\{\\top\}A/\\eta\_\{A\}\. Then

W˙=s⁡\(B˙​A\+B​A˙\)=−s2​\(ηB​G​A⊤​A\+ηA​B​B⊤​G\)=−s2​ηA​ηB​\(G​S\+T​G\)=−μ2​\(G​S\+T​G\)\.\\dot\{W\}=s\(\\dot\{B\}A\+B\\dot\{A\}\)=\-s^\{2\}\\big\(\\eta\_\{B\}GA^\{\\top\}A\+\\eta\_\{A\}BB^\{\\top\}G\\big\)=\-s^\{2\}\\eta\_\{A\}\\eta\_\{B\}\(GS\+TG\)=\-\\mu^\{2\}\(GS\+TG\)\.
*Two quadratic identities\.*Usingα​α⊤=β⊤​β\+κ​I\\alpha\\alpha^\{\\top\}=\\beta^\{\\top\}\\beta\+\\kappa Iandβ⊤​β=α​α⊤−κ​I\\beta^\{\\top\}\\beta=\\alpha\\alpha^\{\\top\}\-\\kappa I,

Z​Z⊤μ2=β​α​α⊤​β⊤=β⁡\(β⊤​β\+κ​I\)​β⊤=T2\+κ​T,Z⊤​Zμ2=α⊤​β⊤​β​α=α⊤​\(α​α⊤−κ​I\)​α=S2−κ​S\.\\tfrac\{ZZ^\{\\top\}\}\{\\mu^\{2\}\}=\\beta\\alpha\\alpha^\{\\top\}\\beta^\{\\top\}=\\beta\(\\beta^\{\\top\}\\beta\+\\kappa I\)\\beta^\{\\top\}=T^\{2\}\+\\kappa T,\\qquad\\tfrac\{Z^\{\\top\}Z\}\{\\mu^\{2\}\}=\\alpha^\{\\top\}\\beta^\{\\top\}\\beta\\alpha=\\alpha^\{\\top\}\(\\alpha\\alpha^\{\\top\}\-\\kappa I\)\\alpha=S^\{2\}\-\\kappa S\.These hold at every point withΦ=−κ​I\\Phi=\-\\kappa I\.

*Spectra on the rank\-rrlocus\.*Ifrank⁡Z=r\\rk Z=r, thenβ\\betaandα⊤\\alpha^\{\\top\}have full column rank, soβ⊤​β≻0\\beta^\{\\top\}\\beta\\succ 0andα​α⊤≻0\\alpha\\alpha^\{\\top\}\\succ 0\. The nonzero eigenvalues ofT=β​β⊤T=\\beta\\beta^\{\\top\}are the eigenvalues ofβ⊤​β=α​α⊤−κ​I\\beta^\{\\top\}\\beta=\\alpha\\alpha^\{\\top\}\-\\kappa I\([Auxiliary result B\.36](https://arxiv.org/html/2610.12728#A2.SS0.SSSx6)\(c\)\), which are positive and exceed−κ\-\\kappa; so they exceedmax⁡\(0,−κ\)\\max\(0,\-\\kappa\)\. Likewise the nonzero eigenvalues ofSSare those ofα​α⊤=β⊤​β\+κ​I\\alpha\\alpha^\{\\top\}=\\beta^\{\\top\}\\beta\+\\kappa I, which exceedmax⁡\(0,κ\)\\max\(0,\\kappa\)\. The range ofTTiscol⁡β=col⁡Z=U\\operatorname\{col\}\\beta=\\operatorname\{col\}Z=U, and that ofSSisrow⁡α=row⁡Z=V\\operatorname\{row\}\\alpha=\\operatorname\{row\}Z=V\.

*Inversion\.*The polynomialp⁡\(t\)=t2\+κ​tp\(t\)=t^\{2\}\+\\kappa tis strictly increasing on\[−κ/2,∞\)\[\-\\kappa/2,\\infty\), which contains the nonzero eigenvalues ofTTbecausemax\(0,−κ\)≥−κ/2\\max\(0,\-\\kappa\)\\geq\-\\kappa/2, andp⁡\(t\)\>0p\(t\)\>0there whilep⁡\(0\)=0p\(0\)=0\. Soppis injective on the spectrum ofTT, the eigenspaces ofTTare those ofp⁡\(T\)=Z​Z⊤/μ2=U⁡\(Σ2/μ2\)​U⊤p\(T\)=ZZ^\{\\top\}/\\mu^\{2\}=U\(\\Sigma^\{2\}/\\mu^\{2\}\)U^\{\\top\}\(by the compact SVD ofZZ,[Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\), and on the eigenvectoruiu\_\{i\}withZ​Z⊤​ui=σi2​uiZZ^\{\\top\}u\_\{i\}=\\sigma\_\{i\}^\{2\}u\_\{i\}the eigenvalue ofTTis the unique roott\>max⁡\(0,−κ\)t\>\\max\(0,\-\\kappa\)oft2\+κ​t=σi2/μ2t^\{2\}\+\\kappa t=\\sigma\_\{i\}^\{2\}/\\mu^\{2\}, namelyfκ​\(σi2/μ2\)f\_\{\\kappa\}\(\\sigma\_\{i\}^\{2\}/\\mu^\{2\}\)\. OnU⟂U^\{\\perp\}bothTTandZ​Z⊤ZZ^\{\\top\}vanish\. HenceT=U​fκ​\(Σ2/μ2\)​U⊤T=Uf\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)U^\{\\top\}\. In the same wayq⁡\(s\)=s2−κ​sq\(s\)=s^\{2\}\-\\kappa sis strictly increasing on\[κ/2,∞\)\[\\kappa/2,\\infty\), the root abovemax⁡\(0,κ\)\\max\(0,\\kappa\)ofs2−κ​s=xs^\{2\}\-\\kappa s=xisκ/2\+x\+κ2/4=gκ​\(x\)\\kappa/2\+\\sqrt\{x\+\\kappa^\{2\}/4\}=g\_\{\\kappa\}\(x\), andS=V​gκ​\(Σ2/μ2\)​V⊤S=Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}\. The compact SVD matters:fκ​\(0\)=\|κ\|f\_\{\\kappa\}\(0\)=\\lvert\\kappa\\rvertforκ<0\\kappa<0andgκ​\(0\)=κg\_\{\\kappa\}\(0\)=\\kappaforκ\>0\\kappa\>0, so the full matrix functionsfκ​\(Z​Z⊤/μ2\)f\_\{\\kappa\}\(ZZ^\{\\top\}/\\mu^\{2\}\)andgκ​\(Z⊤​Z/μ2\)g\_\{\\kappa\}\(Z^\{\\top\}Z/\\mu^\{2\}\)would put nonzero values onker⁡Z⊤\\ker Z^\{\\top\}andker⁡Z\\ker Z, whereTTandSSvanish\. Substituting intoW˙=−μ2​\(G​S\+T​G\)\\dot\{W\}=\-\\mu^\{2\}\(GS\+TG\)gives the formula\. The crossover:x=σ2/μ2x=\\sigma^\{2\}/\\mu^\{2\}equalsκ2/4\\kappa^\{2\}/4iffσ=μ​\|κ\|/2\\sigma=\\mu\\lvert\\kappa\\rvert/2\.

*The regimes\.*Forκ\>0\\kappa\>0,fκ​\(x\)=x/\(x\+κ2/4\+κ/2\)∈\[0,x/κ\]f\_\{\\kappa\}\(x\)=x/\\big\(\\sqrt\{x\+\\kappa^\{2\}/4\}\+\\kappa/2\\big\)\\in\[0,x/\\kappa\]andgκ​\(x\)−κ=fκ​\(x\)g\_\{\\kappa\}\(x\)\-\\kappa=f\_\{\\kappa\}\(x\)\. SoW˙=−μ2​κ​G​PV\+E\\dot\{W\}=\-\\mu^\{2\}\\kappa\\,GP\_\{V\}\+EwithE=−μ2​\(G​V​fκ​\(Σ2/μ2\)​V⊤\+T​G\)E=\-\\mu^\{2\}\\big\(GVf\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}\+TG\\big\)and∥E∥≤2​\(σ12/κ\)​∥G∥\\lVert E\\rVert\\leq 2\(\\sigma\_\{1\}^\{2\}/\\kappa\)\\lVert G\\rVert, whereσ1\\sigma\_\{1\}is the largest singular value ofZZ\. The contribution−μ2​T​G=s​B​A˙\-\\mu^\{2\}TG=s\\,B\\dot\{A\}ofAAhas gain at mostfκ​\(σ12/μ2\)≤σ12/\(μ2​κ\)f\_\{\\kappa\}\(\\sigma\_\{1\}^\{2\}/\\mu^\{2\}\)\\leq\\sigma\_\{1\}^\{2\}/\(\\mu^\{2\}\\kappa\), against the gaingκ≥κg\_\{\\kappa\}\\geq\\kappaof theBB\-term\. The factorAAstill moves, at speed∥A˙∥≤ηA​s​∥B∥2​∥G∥\\lVert\\dot\{A\}\\rVert\\leq\\eta\_\{A\}s\\lVert B\\rVert\_\{2\}\\lVert G\\rVertwith∥B∥22=ηB​∥T∥2≤ηB​σ12/\(μ2​κ\)\\lVert B\\rVert\_\{2\}^\{2\}=\\eta\_\{B\}\\lVert T\\rVert\_\{2\}\\leq\\eta\_\{B\}\\sigma\_\{1\}^\{2\}/\(\\mu^\{2\}\\kappa\)\. Forκ<0\\kappa<0the same bounds hold with the roles exchanged:gκ​\(x\)∈\[0,x/\|κ\|\]g\_\{\\kappa\}\(x\)\\in\[0,x/\\lvert\\kappa\\rvert\]andfκ=gκ\+\|κ\|f\_\{\\kappa\}=g\_\{\\kappa\}\+\\lvert\\kappa\\rvert, soW˙=−μ2​\|κ\|​PU​G\+O⁡\(σ12/\|κ\|\)​∥G∥\\dot\{W\}=\-\\mu^\{2\}\\lvert\\kappa\\rvert P\_\{U\}G\+O\(\\sigma\_\{1\}^\{2\}/\\lvert\\kappa\\rvert\)\\lVert G\\rVert\. For largexx,0≤x\+κ2/4−x≤\|κ\|/20\\leq\\sqrt\{x\+\\kappa^\{2\}/4\}\-\\sqrt\{x\}\\leq\\lvert\\kappa\\rvert/2gives\|fκ​\(x\)−x\|≤\|κ\|\\lvert f\_\{\\kappa\}\(x\)\-\\sqrt\{x\}\\rvert\\leq\\lvert\\kappa\\rvertand\|gκ​\(x\)−x\|≤\|κ\|\\lvert g\_\{\\kappa\}\(x\)\-\\sqrt\{x\}\\rvert\\leq\\lvert\\kappa\\rvert\. Atx=σi2/μ2x=\\sigma\_\{i\}^\{2\}/\\mu^\{2\}the relative error is at most\|κ\|​μ/σi=2​σ∗/σi\\lvert\\kappa\\rvert\\mu/\\sigma\_\{i\}=2\\sigma^\{\*\}/\\sigma\_\{i\}, so forσi≫σ∗\\sigma\_\{i\}\\gg\\sigma^\{\*\},f≈g≈σi/μf\\approx g\\approx\\sigma\_\{i\}/\\muand

W˙≈−μ2​\[G​V​\(Σ/μ\)​V⊤\+U⁡\(Σ/μ\)​U⊤​G\]=−μ⁡\[G​\(Z⊤​Z\)1/2\+\(Z​Z⊤\)1/2​G\]\.\\dot\{W\}\\approx\-\\mu^\{2\}\\big\[GV\(\\Sigma/\\mu\)V^\{\\top\}\+U\(\\Sigma/\\mu\)U^\{\\top\}G\\big\]=\-\\mu\\big\[G\(Z^\{\\top\}Z\)^\{1/2\}\+\(ZZ^\{\\top\}\)^\{1/2\}G\\big\]\.This is the end\-to\-end dynamics of a balanced depth\-2 linear network with rateμ\\mu\([Arora et al\., 2018](https://arxiv.org/html/2610.12728#bib.bib14), Thm\. 1 withN=2N=2\); atκ=0\\kappa=0it is exact, sincef0​\(x\)=g0​\(x\)=xf\_\{0\}\(x\)=g\_\{0\}\(x\)=\\sqrt\{x\}\.

*Off the rank\-rrlocus\.*Letκ\>0\\kappa\>0andrank⁡Z<r\\rk Z<r\. The nonzero eigenvalues ofTTare those ofβ⊤​β⪰0\\beta^\{\\top\}\\beta\\succeq 0, and on themp⁡\(t\)=t2\+κ​t\>0p\(t\)=t^\{2\}\+\\kappa t\>0; ifZ​Z⊤​v=0ZZ^\{\\top\}v=0thenT⁡\(T\+κ​I\)​v=0T\(T\+\\kappa I\)v=0withT\+κ​I≻0T\+\\kappa I\\succ 0, soT​v=0Tv=0\. Henceker⁡T=ker⁡Z​Z⊤\\ker T=\\ker ZZ^\{\\top\}and the inversion above givesT=fκ​\(Z​Z⊤/μ2\)T=f\_\{\\kappa\}\(ZZ^\{\\top\}/\\mu^\{2\}\)as a full matrix function, usingfκ​\(0\)=0f\_\{\\kappa\}\(0\)=0forκ≥0\\kappa\\geq 0\. ForSS, decomposerow⁡A=row⁡Z⊕\(row⁡A∩ker⁡Z\)\\operatorname\{row\}A=\\operatorname\{row\}Z\\oplus\(\\operatorname\{row\}A\\cap\\ker Z\)orthogonally, which is possible sincerow⁡Z⊆row⁡A\\operatorname\{row\}Z\\subseteq\\operatorname\{row\}Aand\(row⁡Z\)⟂=ker⁡Z\(\\operatorname\{row\}Z\)^\{\\perp\}=\\ker Z\. On\(row⁡A\)⟂=ker⁡α\(\\operatorname\{row\}A\)^\{\\perp\}=\\ker\\alpha,S=0S=0\. Ifv=α⊤​w∈row⁡Av=\\alpha^\{\\top\}w\\in\\operatorname\{row\}AhasZ​v=0Zv=0, then0=β​α​α⊤​w=β⁡\(β⊤​β\+κ​I\)​w0=\\beta\\alpha\\alpha^\{\\top\}w=\\beta\(\\beta^\{\\top\}\\beta\+\\kappa I\)w, soβ⊤​β​\(β⊤​β\+κ​I\)​w=0\\beta^\{\\top\}\\beta\(\\beta^\{\\top\}\\beta\+\\kappa I\)w=0; asβ⊤​β\+κ​I≻0\\beta^\{\\top\}\\beta\+\\kappa I\\succ 0commutes withβ⊤​β⪰0\\beta^\{\\top\}\\beta\\succeq 0, this forcesβ⊤​β​w=0\\beta^\{\\top\}\\beta w=0, henceβ​w=0\\beta w=0, andS​v=α⊤​\(α​α⊤\)​w=α⊤​\(β⊤​β\+κ​I\)​w=κ​vSv=\\alpha^\{\\top\}\(\\alpha\\alpha^\{\\top\}\)w=\\alpha^\{\\top\}\(\\beta^\{\\top\}\\beta\+\\kappa I\)w=\\kappa v\. Onrow⁡Z\\operatorname\{row\}Zthe eigenvalues ofSSexceedκ\\kappaand the inversion givesV​gκ​\(Σ2/μ2\)​V⊤Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}\. Together,S=V​gκ​\(Σ2/μ2\)​V⊤\+κ⁡\(Prow⁡A−PV\)S=Vg\_\{\\kappa\}\(\\Sigma^\{2\}/\\mu^\{2\}\)V^\{\\top\}\+\\kappa\(P\_\{\\operatorname\{row\}A\}\-P\_\{V\}\)\. The caseκ<0\\kappa<0is symmetric under\(B,A,Z,ηB,ηA,κ\)↦\(A⊤,B⊤,Z⊤,ηA,ηB,−κ\)\(B,A,Z,\\eta\_\{B\},\\eta\_\{A\},\\kappa\)\\mapsto\(A^\{\\top\},B^\{\\top\},Z^\{\\top\},\\eta\_\{A\},\\eta\_\{B\},\-\\kappa\), which exchangesSSandTT\. Atκ=0\\kappa=0the correction terms vanish\. AtZ=0Z=0withB0=0B\_\{0\}=0andA0​A0⊤=c​IA\_\{0\}A\_\{0\}^\{\\top\}=cI,κ=c/ηA\>0\\kappa=c/\\eta\_\{A\}\>0,T=0T=0andS=κ​Prow⁡A0S=\\kappa P\_\{\\operatorname\{row\}A\_\{0\}\}, soW˙=−μ2​κ​G​Prow⁡A0=−s2​ηB​c​G​Prow⁡A0\\dot\{W\}=\-\\mu^\{2\}\\kappa\\,GP\_\{\\operatorname\{row\}A\_\{0\}\}=\-s^\{2\}\\eta\_\{B\}cGP\_\{\\operatorname\{row\}A\_\{0\}\}, which agrees with the direct value−s2​ηB​G​A0⊤​A0\-s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}becauseA0⊤​A0=c​Prow⁡A0A\_\{0\}^\{\\top\}A\_\{0\}=cP\_\{\\operatorname\{row\}A\_\{0\}\}\. AsZ→0Z\\to 0along the flow withV→row⁡A0V\\to\\operatorname\{row\}A\_\{0\}, the rank\-rrformula tends to the same value, sincegκ​\(0\)=κg\_\{\\kappa\}\(0\)=\\kappaandfκ​\(0\)=0f\_\{\\kappa\}\(0\)=0\. ∎

### C\.33 Theorem III\.10

Theorem III\.10\(the hyperparameter torus; the LoRA/Adam case is known,[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)\)\.Letρ⁡\(q\)=θb\+s​δ​\(q1,…,qk\)\\rho\(q\)=\\theta\_\{b\}\+s\\,\\delta\(q\_\{1\},\\dots,q\_\{k\}\)withδ\\deltamultihomogeneous of multidegree\(d1,…,dk\)\(d\_\{1\},\\dots,d\_\{k\}\)\. Train blockiifromqi​\(0\)=σi​uiq\_\{i\}\(0\)=\\sigma\_\{i\}u\_\{i\}with Adam \(β1∈\[0,1\)\\beta\_\{1\}\\in\[0,1\),β2∈\(0,1\)\\beta\_\{2\}\\in\(0,1\), bias correction allowed\) at learning rateηi​\(t\)\\eta\_\{i\}\(t\), on the lossL∘ρL\\circ\\rhowith noqq\-dependent regulariser\. Assume no gradient clipping \(or per\-block clipping with thresholds rescaled as below\), common random numbers \(the same data order and dropout masks in both runs\), and exact arithmetic \(the result is bit\-exact in floating point when everycic\_\{i\}is a power of22\)\. Then the trajectoryt↦ρ⁡\(q⁡\(t\)\)t\\mapsto\\rho\(q\(t\)\)is invariant under the torus action ofc∈ℝ\>0kc\\in\\mathbb\{R\}\_\{\>0\}^\{k\}:\(s,ηi​\(t\),σi,εi,λi,Ci\)↦\(s​∏ici−di,ci​ηi​\(t\),ci​σi,εi/ci,λi/ci,Ci/ci\)\.\\big\(s,\\ \\eta\_\{i\}\(t\),\\ \\sigma\_\{i\},\\ \\varepsilon\_\{i\},\\ \\lambda\_\{i\},\\ C\_\{i\}\\big\)\\mapsto\\Big\(s\\prod\_\{i\}c\_\{i\}^\{\-d\_\{i\}\},\\ c\_\{i\}\\eta\_\{i\}\(t\),\\ c\_\{i\}\\sigma\_\{i\},\\ \\varepsilon\_\{i\}/c\_\{i\},\\ \\lambda\_\{i\}/c\_\{i\},\\ C\_\{i\}/c\_\{i\}\\Big\)\.Hereεi\\varepsilon\_\{i\}is a per\-block Adamε\\varepsilon\(with the convention0/0=00/0=0atε=0\\varepsilon=0\),λi\\lambda\_\{i\}a per\-block decoupled \(AdamW\) weight decay, andCiC\_\{i\}a per\-block clipping threshold\. Under SGD, with or without heavy\-ball or Nesterov momentum, the same holds withηi​\(t\)↦ci2​ηi​\(t\)\\eta\_\{i\}\(t\)\\mapsto c\_\{i\}^\{2\}\\eta\_\{i\}\(t\)and coupledL2L^\{2\}decayλi↦λi/ci2\\lambda\_\{i\}\\mapsto\\lambda\_\{i\}/c\_\{i\}^\{2\}\. In particular, any learning\-rate schedule shared by all blocks, warm\-up included, is preserved\.

*The rescaling is an isomorphism\.*Writec⊙q=\(c1​q1,…,ck​qk\)c\\odot q=\(c\_\{1\}q\_\{1\},\\dots,c\_\{k\}q\_\{k\}\)andρ′=θb\+s′​δ\\rho^\{\\prime\}=\\theta\_\{b\}\+s^\{\\prime\}\\deltawiths′=s​∏ici−dis^\{\\prime\}=s\\prod\_\{i\}c\_\{i\}^\{\-d\_\{i\}\}\. Multihomogeneity \([Background D\.19](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\) givesρ′​\(c⊙q\)=θb\+s​∏ici−di​∏icidi​δ​\(q\)=ρ⁡\(q\)\\rho^\{\\prime\}\(c\\odot q\)=\\theta\_\{b\}\+s\\prod\_\{i\}c\_\{i\}^\{\-d\_\{i\}\}\\prod\_\{i\}c\_\{i\}^\{d\_\{i\}\}\\delta\(q\)=\\rho\(q\)for allqq, soq↦c⊙qq\\mapsto c\\odot qis a diffeomorphism overΘ\\Thetathat maps the base pointq⁡\(0\)q\(0\)toq′​\(0\)=c⊙q⁡\(0\)q^\{\\prime\}\(0\)=c\\odot q\(0\); it is an isomorphism in𝐌𝐞𝐭𝐡\\mathbf\{Meth\}\. Letℓt=Lt∘ρ\\ell\_\{t\}=L\_\{t\}\\circ\\rhoandℓt′=Lt∘ρ′\\ell^\{\\prime\}\_\{t\}=L\_\{t\}\\circ\\rho^\{\\prime\}be the step\-ttlosses, with the same minibatch and dropout masks in both runs and noqq\-dependent regulariser\. Differentiatingℓt′​\(c⊙q\)=ℓt​\(q\)\\ell^\{\\prime\}\_\{t\}\(c\\odot q\)=\\ell\_\{t\}\(q\)gives

∇qiℓt′​\(c⊙q\)=ci−1​∇qiℓt​\(q\)\.\\nabla\_\{q\_\{i\}\}\\ell^\{\\prime\}\_\{t\}\(c\\odot q\)=c\_\{i\}^\{\-1\}\\nabla\_\{q\_\{i\}\}\\ell\_\{t\}\(q\)\.\(23\)
*The optimizer\.*For blockiiat stept≥1t\\geq 1, Adam\(W\) computes the gradientgtg\_\{t\}ofℓt\\ell\_\{t\}atqt−1q\_\{t\-1\}, clips it per block,gt←gt​min⁡\(1,Ci/∥gt∥\)g\_\{t\}\\leftarrow g\_\{t\}\\min\(1,C\_\{i\}/\\lVert g\_\{t\}\\rVert\)\(with factor11whengt=0g\_\{t\}=0, and no clipping step when there is no clipping\), and sets

mt=β1​mt−1\+\(1−β1\)​gt,vt=β2​vt−1\+\(1−β2\)​gt⊙gt,\\displaystyle m\_\{t\}=\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)g\_\{t\},\\qquad v\_\{t\}=\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)\\,g\_\{t\}\\odot g\_\{t\},qt=qt−1−ηi​\(t\)​\(mt/b1,tvt/b2,t\+εi\+λi​qt−1\),\\displaystyle q\_\{t\}=q\_\{t\-1\}\-\\eta\_\{i\}\(t\)\\Big\(\\frac\{m\_\{t\}/b\_\{1,t\}\}\{\\sqrt\{v\_\{t\}/b\_\{2,t\}\}\+\\varepsilon\_\{i\}\}\+\\lambda\_\{i\}q\_\{t\-1\}\\Big\),withm0=v0=0m\_\{0\}=v\_\{0\}=0, entrywise square root and division,bj,t=1−βjtb\_\{j,t\}=1\-\\beta\_\{j\}^\{t\}with bias correction andbj,t=1b\_\{j,t\}=1without, and0/0=00/0=0\. PyTorch’s AdamW first multipliesqt−1q\_\{t\-1\}by1−ηi​\(t\)​λi1\-\\eta\_\{i\}\(t\)\\lambda\_\{i\}and then takes the Adam step, which is the same update\.

*Induction\.*Run the original hyperparameters fromq⁡\(0\)q\(0\)and the transformed ones fromq′​\(0\)=c⊙q⁡\(0\)q^\{\\prime\}\(0\)=c\\odot q\(0\), which is whatσi↦ci​σi\\sigma\_\{i\}\\mapsto c\_\{i\}\\sigma\_\{i\}means\. Suppose that after stept−1t\-1, in every block,

qt−1′=ci​qt−1,mt−1′=ci−1​mt−1,vt−1′=ci−2​vt−1,q^\{\\prime\}\_\{t\-1\}=c\_\{i\}q\_\{t\-1\},\\qquad m^\{\\prime\}\_\{t\-1\}=c\_\{i\}^\{\-1\}m\_\{t\-1\},\\qquad v^\{\\prime\}\_\{t\-1\}=c\_\{i\}^\{\-2\}v\_\{t\-1\},which holds att=1t=1\. By \([23](https://arxiv.org/html/2610.12728#A3.E23)\) the raw gradients satisfygt′=ci−1​gtg^\{\\prime\}\_\{t\}=c\_\{i\}^\{\-1\}g\_\{t\}\. WithCi′=Ci/ciC^\{\\prime\}\_\{i\}=C\_\{i\}/c\_\{i\}the clipping factors agree,min⁡\(1,Ci′/∥gt′∥\)=min⁡\(1,Ci/∥gt∥\)\\min\(1,C^\{\\prime\}\_\{i\}/\\lVert g^\{\\prime\}\_\{t\}\\rVert\)=\\min\(1,C\_\{i\}/\\lVert g\_\{t\}\\rVert\), so the clipped gradients also satisfygt′=ci−1​gtg^\{\\prime\}\_\{t\}=c\_\{i\}^\{\-1\}g\_\{t\}\. The moment recursions are linear ingtg\_\{t\}andgt⊙gtg\_\{t\}\\odot g\_\{t\}, somt′=ci−1​mtm^\{\\prime\}\_\{t\}=c\_\{i\}^\{\-1\}m\_\{t\}andvt′=ci−2​vtv^\{\\prime\}\_\{t\}=c\_\{i\}^\{\-2\}v\_\{t\}\. Withεi′=εi/ci\\varepsilon^\{\\prime\}\_\{i\}=\\varepsilon\_\{i\}/c\_\{i\}the normalised step is unchanged,

mt′/b1,tvt′/b2,t\+εi′=ci−1​mt/b1,tci−1​\(vt/b2,t\+εi\)=mt/b1,tvt/b2,t\+εi,\\frac\{m^\{\\prime\}\_\{t\}/b\_\{1,t\}\}\{\\sqrt\{v^\{\\prime\}\_\{t\}/b\_\{2,t\}\}\+\\varepsilon^\{\\prime\}\_\{i\}\}=\\frac\{c\_\{i\}^\{\-1\}m\_\{t\}/b\_\{1,t\}\}\{c\_\{i\}^\{\-1\}\\big\(\\sqrt\{v\_\{t\}/b\_\{2,t\}\}\+\\varepsilon\_\{i\}\\big\)\}=\\frac\{m\_\{t\}/b\_\{1,t\}\}\{\\sqrt\{v\_\{t\}/b\_\{2,t\}\}\+\\varepsilon\_\{i\}\},including the entries wherevt=0v\_\{t\}=0, at whichmt=0m\_\{t\}=0as well and both sides are00\. The decay term transforms asλi′​qt−1′=\(λi/ci\)​ci​qt−1=λi​qt−1\\lambda^\{\\prime\}\_\{i\}q^\{\\prime\}\_\{t\-1\}=\(\\lambda\_\{i\}/c\_\{i\}\)c\_\{i\}q\_\{t\-1\}=\\lambda\_\{i\}q\_\{t\-1\}\. Withηi′​\(t\)=ci​ηi​\(t\)\\eta^\{\\prime\}\_\{i\}\(t\)=c\_\{i\}\\eta\_\{i\}\(t\)the whole step is multiplied bycic\_\{i\}, soqt′=ci​qtq^\{\\prime\}\_\{t\}=c\_\{i\}q\_\{t\}, which closes the induction\. Thenρ′​\(qt′\)=ρ⁡\(qt\)\\rho^\{\\prime\}\(q^\{\\prime\}\_\{t\}\)=\\rho\(q\_\{t\}\)for alltt\. The value ofηi​\(t\)\\eta\_\{i\}\(t\)at each step was arbitrary, so any schedule works; a schedule shared by all blocks,ηi​\(t\)=ηi​φ​\(t\)\\eta\_\{i\}\(t\)=\\eta\_\{i\}\\varphi\(t\), becomesci​ηi​φ​\(t\)c\_\{i\}\\eta\_\{i\}\\varphi\(t\), the same schedule\.

*SGD\.*With coupledL2L^\{2\}decay the step\-ttdirection isgt\+λi​qt−1g\_\{t\}\+\\lambda\_\{i\}q\_\{t\-1\}, the heavy\-ball buffer isbt=μm​bt−1\+\(1−τ\)​\(gt\+λi​qt−1\)b\_\{t\}=\\mu\_\{m\}b\_\{t\-1\}\+\(1\-\\tau\)\(g\_\{t\}\+\\lambda\_\{i\}q\_\{t\-1\}\)\(with momentumμm\\mu\_\{m\}and dampeningτ\\tau\), the direction isbtb\_\{t\}, orgt\+λi​qt−1\+μm​btg\_\{t\}\+\\lambda\_\{i\}q\_\{t\-1\}\+\\mu\_\{m\}b\_\{t\}for Nesterov, andqt=qt−1−ηi​\(t\)⋅\(direction\)q\_\{t\}=q\_\{t\-1\}\-\\eta\_\{i\}\(t\)\\cdot\(\\text\{direction\}\)\. Supposeqt−1′=ci​qt−1q^\{\\prime\}\_\{t\-1\}=c\_\{i\}q\_\{t\-1\}andbt−1′=ci−1​bt−1b^\{\\prime\}\_\{t\-1\}=c\_\{i\}^\{\-1\}b\_\{t\-1\}\. Thengt′=ci−1​gtg^\{\\prime\}\_\{t\}=c\_\{i\}^\{\-1\}g\_\{t\}and, withλi′=λi/ci2\\lambda^\{\\prime\}\_\{i\}=\\lambda\_\{i\}/c\_\{i\}^\{2\},λi′​qt−1′=ci−1​λi​qt−1\\lambda^\{\\prime\}\_\{i\}q^\{\\prime\}\_\{t\-1\}=c\_\{i\}^\{\-1\}\\lambda\_\{i\}q\_\{t\-1\}; every quantity entering the direction is multiplied byci−1c\_\{i\}^\{\-1\}, sobt′=ci−1​btb^\{\\prime\}\_\{t\}=c\_\{i\}^\{\-1\}b\_\{t\}and the direction is multiplied byci−1c\_\{i\}^\{\-1\}\. Withηi′​\(t\)=ci2​ηi​\(t\)\\eta^\{\\prime\}\_\{i\}\(t\)=c\_\{i\}^\{2\}\\eta\_\{i\}\(t\)the step is multiplied bycic\_\{i\}, andqt′=ci​qtq^\{\\prime\}\_\{t\}=c\_\{i\}q\_\{t\}\. Per\-block clipping withCi/ciC\_\{i\}/c\_\{i\}is handled as for Adam\.

*Floating point\.*If everycic\_\{i\}is a power of22, multiplication byci±1c\_\{i\}^\{\\pm 1\}and by∏ici∓di\\prod\_\{i\}c\_\{i\}^\{\\mp d\_\{i\}\}is exact, and it commutes with every correctly rounded addition, multiplication, division and square root, barring overflow and underflow \([Auxiliary result B\.39](https://arxiv.org/html/2610.12728#A2.SS0.SSSx7)\)\. Then the forward pass computes bit\-identical weights and outputs in both runs, the backward pass computes gradients that are exactlyci−1c\_\{i\}^\{\-1\}times the original ones, and every quantity in the induction above holds exactly in floating point, so the two trajectories agree bit for bit\. ∎

### C\.34 Corollary III\.11

Corollary III\.11\(LoRA\+ isα\\alphain disguise; rsLoRA is LoRA; \(a\) and the LoRA\+ reading are known,[Schulman and Thinking Machines Lab \(2025\)](https://arxiv.org/html/2610.12728#bib.bib401)\)\.For linear LoRA modules \(k=2k=2,d=\(1,1\)d=\(1,1\),σB=0\\sigma\_\{B\}=0\), under the hypotheses of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7):\(a\)*Effective hyperparameters\.*Under Adam the torus invariants of\(s,ηA,ηB,σA\)\(s,\\eta\_\{A\},\\eta\_\{B\},\\sigma\_\{A\}\)are\(s​ηA​ηB,σA/ηA\)\(s\\eta\_\{A\}\\eta\_\{B\},\\ \\sigma\_\{A\}/\\eta\_\{A\}\); the quantitiesα⋅initA⋅LRB\\alpha\\cdot\\mathrm\{init\}\_\{A\}\\cdot\\mathrm\{LR\}\_\{B\}andinitA/LRA\\mathrm\{init\}\_\{A\}/\\mathrm\{LR\}\_\{A\}of Schulman et al\. are equivalent generators\. Under SGD the invariants are\(s2​ηA​ηB,σA2/ηA\)\(s^\{2\}\\eta\_\{A\}\\eta\_\{B\},\\ \\sigma\_\{A\}^\{2\}/\\eta\_\{A\}\)\.\(b\)*LoRA\+\.*Under Adam withε=0\\varepsilon=0, no weight decay, no gradient clipping and a learning\-rate schedule shared by both groups, LoRA\+ with ratioλ\\lambdaat scalessproduces*exactly*the trajectory of plain LoRA at scaleλ​s\\lambda s\(sameηA\\eta\_\{A\}, sameA0A\_\{0\}\)\. Forε\>0\\varepsilon\>0the equivalence needs a per\-blockεB′=λ​ε\\varepsilon\_\{B\}^\{\\prime\}=\\lambda\\varepsilon, and with AdamW a per\-group decayλ⋅wd\\lambda\\cdot\\mathrm\{wd\}onBB\. Under SGD it holds at scaleλ​s\\sqrt\{\\lambda\}\\,s, or equivalently with plain LoRA at the single rateηA​ηB\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}andA0↦λ1/4​A0A\_\{0\}\\mapsto\\lambda^\{1/4\}A\_\{0\}\.\(c\)*rsLoRA\.*ρrs​\(B,A\)=ρLoRA​\(r​B,A\)\\rho\_\{\\mathrm\{rs\}\}\(B,A\)=\\rho\_\{\\mathrm\{LoRA\}\}\(\\sqrt\{r\}\\,B,A\)is an isomorphism of objects\. Up to the hyperparameter torus, rsLoRA can be represented as LoRA at the sameα\\alphawithηB\\eta\_\{B\}multiplied byr\\sqrt\{r\}under Adam \(byrrunder SGD\), or equally as LoRA withα\\alphamultiplied byr\\sqrt\{r\}at the same rates\. The reading “same object, different metric” is a choice of representative within the orbit\. Part \(c\) concerns a fixed rank; it does not address rsLoRA’s rank\-stability prescription, howα\\alphashould scale withrr\.

Hereδ⁡\(B,A\)=B​A\\delta\(B,A\)=BAhas multidegree\(1,1\)\(1,1\), andσB=0\\sigma\_\{B\}=0meansB0=0B\_\{0\}=0, which everycBc\_\{B\}leaves unchanged\.

\(a\) Under Adam,\(cA,cB\)\(c\_\{A\},c\_\{B\}\)acts bys↦s/\(cA​cB\)s\\mapsto s/\(c\_\{A\}c\_\{B\}\),ηA↦cA​ηA\\eta\_\{A\}\\mapsto c\_\{A\}\\eta\_\{A\},ηB↦cB​ηB\\eta\_\{B\}\\mapsto c\_\{B\}\\eta\_\{B\}andσA↦cA​σA\\sigma\_\{A\}\\mapsto c\_\{A\}\\sigma\_\{A\}\. In logarithms this is translation by the column space of

a=\(−1−1100110\)\(rows​log⁡s,log⁡ηA,log⁡ηB,log⁡σA\),a=\\begin\{pmatrix\}\-1&\-1\\\\ 1&0\\\\ 0&1\\\\ 1&0\\end\{pmatrix\}\\qquad\(\\text\{rows \}\\log s,\\log\\eta\_\{A\},\\log\\eta\_\{B\},\\log\\sigma\_\{A\}\),which has rank22, so the orbit space has dimension22and a basis of the vectorsbbwithb⊤​a=0b^\{\\top\}a=0gives a complete set of invariant monomials \([Background D\.20](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx2.Px1)\)\. The vectors\(1,1,1,0\)\(1,1,1,0\)and\(0,−1,0,1\)\(0,\-1,0,1\)satisfyb⊤​a=0b^\{\\top\}a=0and are independent; they gives​ηA​ηBs\\eta\_\{A\}\\eta\_\{B\}andσA/ηA\\sigma\_\{A\}/\\eta\_\{A\}\. Sinceα∝s\\alpha\\propto sat fixedrr, Schulman et al\.’s quantities ares​σA​ηB=\(s​ηA​ηB\)​\(σA/ηA\)s\\sigma\_\{A\}\\eta\_\{B\}=\(s\\eta\_\{A\}\\eta\_\{B\}\)\(\\sigma\_\{A\}/\\eta\_\{A\}\)andσA/ηA\\sigma\_\{A\}/\\eta\_\{A\}, with exponent vectors\(1,0,1,1\)\(1,0,1,1\)and\(0,−1,0,1\)\(0,\-1,0,1\), another basis of the same lattice of invariant monomials\. Under SGD the rates scale bycA2c\_\{A\}^\{2\}andcB2c\_\{B\}^\{2\}; the weight matrix has rows\(−1,−1\)\(\-1,\-1\),\(2,0\)\(2,0\),\(0,2\)\(0,2\),\(1,0\)\(1,0\), again of rank22, and\(2,1,1,0\)\(2,1,1,0\)and\(0,−1,0,2\)\(0,\-1,0,2\)give the invariantss2​ηA​ηBs^\{2\}\\eta\_\{A\}\\eta\_\{B\}andσA2/ηA\\sigma\_\{A\}^\{2\}/\\eta\_\{A\}\.

\(b\) LoRA\+ hasηB=λ​ηA\\eta\_\{B\}=\\lambda\\eta\_\{A\}\. Under Adam apply the torus elementcA=1c\_\{A\}=1,cB=1/λc\_\{B\}=1/\\lambda: it maps\(s,ηA,λ​ηA,σA\)\(s,\\eta\_\{A\},\\lambda\\eta\_\{A\},\\sigma\_\{A\}\)to\(λ​s,ηA,ηA,σA\)\(\\lambda s,\\eta\_\{A\},\\eta\_\{A\},\\sigma\_\{A\}\), plain LoRA at scaleλ​s\\lambda swith the sameηA\\eta\_\{A\}and the sameA0A\_\{0\}; a shared scheduleφ⁡\(t\)\\varphi\(t\)stays shared\.[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)givesεB′=ε/cB=λ​ε\\varepsilon\_\{B\}^\{\\prime\}=\\varepsilon/c\_\{B\}=\\lambda\\varepsilonandwdB′=wd/cB=λ⋅wd\\mathrm\{wd\}\_\{B\}^\{\\prime\}=\\mathrm\{wd\}/c\_\{B\}=\\lambda\\cdot\\mathrm\{wd\}, which are the stated per\-block values, and both vanish whenε=wd=0\\varepsilon=\\mathrm\{wd\}=0\. Under SGD,cA=1c\_\{A\}=1andcB=λ−1/2c\_\{B\}=\\lambda^\{\-1/2\}giveηB′=cB2​λ​ηA=ηA\\eta\_\{B\}^\{\\prime\}=c\_\{B\}^\{2\}\\lambda\\eta\_\{A\}=\\eta\_\{A\}ands′=λ​ss^\{\\prime\}=\\sqrt\{\\lambda\}\\,s\. The single\-rate form usescA=λ1/4c\_\{A\}=\\lambda^\{1/4\}andcB=λ−1/4c\_\{B\}=\\lambda^\{\-1/4\}: thenηA′=λ​ηA\\eta\_\{A\}^\{\\prime\}=\\sqrt\{\\lambda\}\\,\\eta\_\{A\},ηB′=λ−1/2ληA=ληA\\eta\_\{B\}^\{\\prime\}=\\lambda^\{\-1/2\}\\lambda\\eta\_\{A\}=\\sqrt\{\\lambda\}\\,\\eta\_\{A\}, both equal toηA​ηB\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\},s′=ss^\{\\prime\}=s, andσA′=λ1/4​σA\\sigma\_\{A\}^\{\\prime\}=\\lambda^\{1/4\}\\sigma\_\{A\}, that is,A0↦λ1/4​A0A\_\{0\}\\mapsto\\lambda^\{1/4\}A\_\{0\}\.

\(c\)ρrs​\(B,A\)=W0\+\(α/r\)​B​A=W0\+\(α/r\)​\(r​B\)​A=ρLoRA​\(r​B,A\)\\rho\_\{\\mathrm\{rs\}\}\(B,A\)=W\_\{0\}\+\(\\alpha/\\sqrt\{r\}\)BA=W\_\{0\}\+\(\\alpha/r\)\(\\sqrt\{r\}B\)A=\\rho\_\{\\mathrm\{LoRA\}\}\(\\sqrt\{r\}B,A\), and\(B,A\)↦\(r​B,A\)\(B,A\)\\mapsto\(\\sqrt\{r\}B,A\)is a pointed diffeomorphism \(0↦00\\mapsto 0in theBB\-factor\), hence an isomorphism of objects\. The rsLoRA scalesrs=α/rs\_\{\\mathrm\{rs\}\}=\\alpha/\\sqrt\{r\}is the LoRA scale forα′=r​α\\alpha^\{\\prime\}=\\sqrt\{r\}\\,\\alpha, which is the second representation\. Applying the torus elementcA=1c\_\{A\}=1,cB=rc\_\{B\}=\\sqrt\{r\}to rsLoRA givess′=srs/r=α/rs^\{\\prime\}=s\_\{\\mathrm\{rs\}\}/\\sqrt\{r\}=\\alpha/r, the LoRA scale at the sameα\\alpha, withηB′=r​ηB\\eta\_\{B\}^\{\\prime\}=\\sqrt\{r\}\\,\\eta\_\{B\}under Adam andηB′=r​ηB\\eta\_\{B\}^\{\\prime\}=r\\eta\_\{B\}under SGD, and withεB′=ε/r\\varepsilon\_\{B\}^\{\\prime\}=\\varepsilon/\\sqrt\{r\}andwdB′=wd/r\\mathrm\{wd\}\_\{B\}^\{\\prime\}=\\mathrm\{wd\}/\\sqrt\{r\}when these are nonzero\. Both representations lie on the torus orbit of rsLoRA’s hyperparameters, which is what “up to the hyperparameter torus” means\. The rankrris fixed throughout\. ∎

### C\.35 Theorem III\.12

Theorem III\.12\(the optimizer gauge hierarchy\)\.On LoRA’s full\-rank locus, with hyperparameters shared by all rank channels, the largest subgroup of the gaugeGLr\\mathrm\{GL\}\_\{r\}under which an update rule is equivariant, at the level of parameters, is as follows\.\(a\) SGD and GD with any block rates, with or without heavy\-ball momentum:O⁡\(r\)\\mathrm\{O\}\(r\)\.\(b\) Adam and AdamW at fixed learning rates \(β1,β2∈\[0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\[0,1\), anyε≥0\\varepsilon\\geq 0, any weight decay\): the signed permutationsBrB\_\{r\}, of order2r​r\!2^\{r\}r\!\. A positive diagonalg=diag⁡\(d\)g=\\diag\(d\)is a symmetry only*jointly*with per\-rank\-channel hyperparameters, and only without global\-norm clipping\. Atε=0\\varepsilon=0and without weight decay it suffices to rescale the rates,ηB,j↦ηB,j/dj\\eta\_\{B,j\}\\mapsto\\eta\_\{B,j\}/d\_\{j\}andηA,j↦dj​ηA,j\\eta\_\{A,j\}\\mapsto d\_\{j\}\\eta\_\{A,j\}\. Withε\>0\\varepsilon\>0one must also give each channel its ownεB,j=dj​ε\\varepsilon\_\{B,j\}=d\_\{j\}\\varepsilonandεA,j=ε/dj\\varepsilon\_\{A,j\}=\\varepsilon/d\_\{j\}\. With decoupled decay multiplied by the learning rate \(PyTorch AdamW\) one must also setλB,j=dj​λ\\lambda\_\{B,j\}=d\_\{j\}\\lambdaandλA,j=λ/dj\\lambda\_\{A,j\}=\\lambda/d\_\{j\}; under the convention of[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2610.12728#bib.bib296), where decay is scaled only by the schedule multiplier,λ\\lambdaneeds no change\.\(c\) Undamped scaled GD \(δ=0\\delta=0; the Riemannian preconditioner of Tong, Ma and Chi and of Zhang and Pilanci, with or without heavy\-ball momentum\): all ofGLr\\mathrm\{GL\}\_\{r\}\. TheWW\-trajectory from gauge\-related initial states \(with zero or correspondingly transformed momentum\) is independent of the representative; a step may drop rank, so the iteration maps intoW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\.

We use the conventions of §[C\.24](https://arxiv.org/html/2610.12728#A3.SS24): atg⋅qg\\cdot qthe gradients are \([20](https://arxiv.org/html/2610.12728#A3.E20)\), and equivariance underggmeans \([21](https://arxiv.org/html/2610.12728#A3.E21)\) for all gradients, together with transported optimizer state\. The set of suchggis a subgroup\. On the full\-rank locus∇B=s​G​A⊤\\nabla\_\{B\}=sGA^\{\\top\}ranges over all ofℝm×r\\mathbb\{R\}^\{m\\times r\}asGGranges overℝm×n\\mathbb\{R\}^\{m\\times n\}, becauseAAhas full row rank; this is the only use of the locus in the inclusions⊆\\subseteqbelow\.

\(a\) An SGD step isδ​B=−ηB​∇B\\delta B=\-\\eta\_\{B\}\\nabla\_\{B\},δ​A=−ηA​∇A\\delta A=\-\\eta\_\{A\}\\nabla\_\{A\}\. Atg⋅qg\\cdot q,δ​B′=−ηB​∇Bg⊤\\delta B^\{\\prime\}=\-\\eta\_\{B\}\\nabla\_\{B\}g^\{\\top\}, and \([21](https://arxiv.org/html/2610.12728#A3.E21)\) requires∇B\(g⊤−g−1\)=0\\nabla\_\{B\}\(g^\{\\top\}\-g^\{\-1\}\)=0for all∇B∈ℝm×r\\nabla\_\{B\}\\in\\mathbb\{R\}^\{m\\times r\}, that is,g⊤=g−1g^\{\\top\}=g^\{\-1\}\. Conversely, forg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\),δ​B′=δ​B​g⊤=δ​B​g−1\\delta B^\{\\prime\}=\\delta Bg^\{\\top\}=\\delta Bg^\{\-1\}andδ​A′=−ηA​g−⁣⊤​∇A=g​δ​A\\delta A^\{\\prime\}=\-\\eta\_\{A\}g^\{\-\\top\}\\nabla\_\{A\}=g\\,\\delta A\. A heavy\-ball or Nesterov buffer is a linear combination of past gradients, all taken at points related by the samegg, so for orthogonalggit transforms like the gradients,bB′=bB​g−1b^\{\\prime\}\_\{B\}=b\_\{B\}g^\{\-1\}andbA′=g​bAb^\{\\prime\}\_\{A\}=gb\_\{A\}, and the step is transported\. The first step from a zero buffer is a positive multiple of an SGD step \(the multiple is11for heavy ball and1\+μm1\+\\mu\_\{m\}for Nesterov with momentum coefficientμm\\mu\_\{m\}\), so the inclusion⊆O⁡\(r\)\\subseteq\\mathrm\{O\}\(r\)also holds with momentum\.

\(b\)*The inclusion⊇\\supseteq\.*Letggbe a signed permutation, sog−1=g⊤g^\{\-1\}=g^\{\\top\}and∇B′=∇Bg−1\\nabla\_\{B^\{\\prime\}\}=\\nabla\_\{B\}g^\{\-1\}permutes the columns of∇B\\nabla\_\{B\}and flips some signs; likewise∇A′=g​∇A\\nabla\_\{A^\{\\prime\}\}=g\\nabla\_\{A\}permutes and flips rows\. Adam is coordinatewise: the first momentmmis permuted with the same sign flips, the second momentvvis permuted without sign changes, so the normalised step is transported, and so is a decoupled or coupled decay term, which is linear in the parameters\. Hence the step satisfies \([21](https://arxiv.org/html/2610.12728#A3.E21)\)\. Sinceggis orthogonal it preserves∥∇B∥\\lVert\\nabla\_\{B\}\\rVertand∥∇A∥\\lVert\\nabla\_\{A\}\\rVert, so global\-norm clipping does not affect this inclusion\.

*The inclusion⊆\\subseteq\.*Consider the first step from zero state at a full\-rank point\. With bias correction it isδ​B=−η⁡\(ϕ⁡\(∇B\)\+λ​B\)\\delta B=\-\\eta\\big\(\\phi\(\\nabla\_\{B\}\)\+\\lambda B\\big\)withϕ⁡\(x\)=x/\(\|x\|\+ε\)\\phi\(x\)=x/\(\\lvert x\\rvert\+\\varepsilon\)entrywise and0/0=00/0=0; without bias correctionϕ\\phiis replaced by a positive multiple ofx/\(\|x\|\+ε′\)x/\(\\lvert x\\rvert\+\\varepsilon^\{\\prime\}\)with a rescaledε′\\varepsilon^\{\\prime\}, which changes nothing below\. Atg⋅qg\\cdot qthe decay termλ​B​g−1\\lambda Bg^\{\-1\}is already transported, so equivariance requiresϕ⁡\(∇Bg⊤\)=ϕ⁡\(∇B\)​g−1\\phi\(\\nabla\_\{B\}g^\{\\top\}\)=\\phi\(\\nabla\_\{B\}\)g^\{\-1\}, row by row:

ϕ⁡\(x​g⊤\)=ϕ⁡\(x\)​hfor every row vector​x∈ℝ1×r,h:=g−1\.\\phi\(xg^\{\\top\}\)=\\phi\(x\)h\\qquad\\text\{for every row vector \}x\\in\\mathbb\{R\}^\{1\\times r\},\\qquad h:=g^\{\-1\}\.\(24\)\(With coupled decay the argument ofϕ\\phiis∇B\+λ​B\\nabla\_\{B\}\+\\lambda B; letting∇B\\nabla\_\{B\}grow along a fixed direction leads to the limit \([25](https://arxiv.org/html/2610.12728#A3.E25)\) below in the same way\.\)

*Caseε=0\\varepsilon=0\.*Thenϕ=sign\\phi=\\operatorname\{sign\}and \([24](https://arxiv.org/html/2610.12728#A3.E24)\) reads

sign⁡\(x​g⊤\)=sign⁡\(x\)​hfor every​x\.\\operatorname\{sign\}\(xg^\{\\top\}\)=\\operatorname\{sign\}\(x\)h\\qquad\\text\{for every \}x\.\(25\)Fix𝗌∈\{±1\}r\\mathsf\{s\}\\in\\\{\\pm 1\\\}^\{r\}\. The open orthant\{x:sign⁡\(x\)=𝗌\}\\\{x:\\operatorname\{sign\}\(x\)=\\mathsf\{s\}\\\}is not contained in the finite union of the hyperplanes\{x:\(x​g⊤\)j=0\}\\\{x:\(xg^\{\\top\}\)\_\{j\}=0\\\}, which are proper becauseggis invertible \(a finite union of hyperplanes has empty interior\), so it contains anxxwithx​g⊤xg^\{\\top\}free of zeros\. For thatxx, \([25](https://arxiv.org/html/2610.12728#A3.E25)\) gives𝗌​h∈\{±1\}r\\mathsf\{s\}h\\in\\\{\\pm 1\\\}^\{r\}\. Thus𝗌​h∈\{±1\}r\\mathsf\{s\}h\\in\\\{\\pm 1\\\}^\{r\}for every𝗌\\mathsf\{s\}\. Flipping𝗌i\\mathsf\{s\}\_\{i\}changes\(𝗌​h\)j\(\\mathsf\{s\}h\)\_\{j\}by2​𝗌i​hi​j2\\mathsf\{s\}\_\{i\}h\_\{ij\}, a difference of two elements of\{±1\}\\\{\\pm 1\\\}, sohi​j∈\{0,±1\}h\_\{ij\}\\in\\\{0,\\pm 1\\\}\. If columnjjofhhhad no nonzero entry,\(𝗌​h\)j=0\(\\mathsf\{s\}h\)\_\{j\}=0; if it hadt≥2t\\geq 2nonzero entries, the choice𝗌i=hi​j\\mathsf\{s\}\_\{i\}=h\_\{ij\}on its support would give\(𝗌​h\)j=t≥2\(\\mathsf\{s\}h\)\_\{j\}=t\\geq 2\. So every column ofhhhas exactly one entry, equal to±1\\pm 1, and these entries lie in distinct rows becausehhis invertible\. Hencehh, and with itg=h−1g=h^\{\-1\}, is a signed permutation matrix\.

*Caseε\>0\\varepsilon\>0\.*For a fixedxxandτ→∞\\tau\\to\\infty,ϕ⁡\(τ​y\)→sign⁡\(y\)\\phi\(\\tau y\)\\to\\operatorname\{sign\}\(y\)entrywise for everyyy, so applying \([24](https://arxiv.org/html/2610.12728#A3.E24)\) toτ​x\\tau xand lettingτ→∞\\tau\\to\\inftygives \([25](https://arxiv.org/html/2610.12728#A3.E25)\), and the caseε=0\\varepsilon=0showsg∈Brg\\in B\_\{r\}\. \(Forτ→0\\tau\\to 0,ϕ⁡\(τ​y\)=τ​y/ε\+O⁡\(τ2\)\\phi\(\\tau y\)=\\tau y/\\varepsilon\+O\(\\tau^\{2\}\), and the same comparison givesg⊤=hg^\{\\top\}=h, that is,g∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\), which is consistent\.\) The order ofBrB\_\{r\}is2r​r\!2^\{r\}r\!:r\!r\!permutations times2r2^\{r\}sign patterns\.

*The joint torus\.*Letg=diag⁡\(d\)g=\\diag\(d\)withdj\>0d\_\{j\}\>0, and give columnjjofBBthe hyperparameters\(ηB,j′,εB,j′,λB,j′\)\(\\eta^\{\\prime\}\_\{B,j\},\\varepsilon^\{\\prime\}\_\{B,j\},\\lambda^\{\\prime\}\_\{B,j\}\)and rowjjofAAthe hyperparameters\(ηA,j′,εA,j′,λA,j′\)\(\\eta^\{\\prime\}\_\{A,j\},\\varepsilon^\{\\prime\}\_\{A,j\},\\lambda^\{\\prime\}\_\{A,j\}\)\. By \([20](https://arxiv.org/html/2610.12728#A3.E20)\), columnjjof∇B′\\nabla\_\{B^\{\\prime\}\}isdjd\_\{j\}times columnjjof∇B\\nabla\_\{B\}, while \([21](https://arxiv.org/html/2610.12728#A3.E21)\) asks for columnjjofδ​B′\\delta B^\{\\prime\}to be1/dj1/d\_\{j\}times that ofδ​B\\delta B\. Suppose that before a stepB′:,j=B:,j/djB^\{\\prime\}\_\{:,j\}=B\_\{:,j\}/d\_\{j\},m′:,j=djm:,jm^\{\\prime\}\_\{:,j\}=d\_\{j\}m\_\{:,j\}andv′:,j=dj2v:,jv^\{\\prime\}\_\{:,j\}=d\_\{j\}^\{2\}v\_\{:,j\}\(true for zero initial state\)\. After the moment updates these relations persist, and the normalised step of columnjjis

dj​m^dj​v^\+εB,j′=m^v^\+εB,j′/dj,\\frac\{d\_\{j\}\\hat\{m\}\}\{d\_\{j\}\\sqrt\{\\hat\{v\}\}\+\\varepsilon^\{\\prime\}\_\{B,j\}\}=\\frac\{\\hat\{m\}\}\{\\sqrt\{\\hat\{v\}\}\+\\varepsilon^\{\\prime\}\_\{B,j\}/d\_\{j\}\},which equals the original normalised step iffεB,j′=dj​ε\\varepsilon^\{\\prime\}\_\{B,j\}=d\_\{j\}\\varepsilon\(atε=0\\varepsilon=0both are00\)\. The update of columnjjis then−ηB,j′\(step\+λB,j′B:,j/dj\)\-\\eta^\{\\prime\}\_\{B,j\}\\big\(\\text\{step\}\+\\lambda^\{\\prime\}\_\{B,j\}B\_\{:,j\}/d\_\{j\}\\big\), which equals−1djη\(step\+λB:,j\)\-\\tfrac\{1\}\{d\_\{j\}\}\\eta\\big\(\\text\{step\}\+\\lambda B\_\{:,j\}\\big\)iffηB,j′=η/dj\\eta^\{\\prime\}\_\{B,j\}=\\eta/d\_\{j\}andηB,j′​λB,j′/dj=η​λ/dj\\eta^\{\\prime\}\_\{B,j\}\\lambda^\{\\prime\}\_\{B,j\}/d\_\{j\}=\\eta\\lambda/d\_\{j\}, that is,λB,j′=dj​λ\\lambda^\{\\prime\}\_\{B,j\}=d\_\{j\}\\lambda\. Under the convention of[Loshchilov and Hutter \(2019\)](https://arxiv.org/html/2610.12728#bib.bib296)the decay term is−ζtλ′B,jB′:,j\-\\zeta\_\{t\}\\lambda^\{\\prime\}\_\{B,j\}B^\{\\prime\}\_\{:,j\}with the schedule multiplierζt\\zeta\_\{t\}alone, and it is transported iffλB,j′=λ\\lambda^\{\\prime\}\_\{B,j\}=\\lambda\. Rowjjof∇A′\\nabla\_\{A^\{\\prime\}\}is1/dj1/d\_\{j\}times that of∇A\\nabla\_\{A\}and the step must be multiplied bydjd\_\{j\}, which givesεA,j′=ε/dj\\varepsilon^\{\\prime\}\_\{A,j\}=\\varepsilon/d\_\{j\},ηA,j′=dj​η\\eta^\{\\prime\}\_\{A,j\}=d\_\{j\}\\etaandλA,j′=λ/dj\\lambda^\{\\prime\}\_\{A,j\}=\\lambda/d\_\{j\}\(no change under Loshchilov–Hutter\)\. The relations persist by induction\. With channel\-uniform hyperparameters a positive diagonalg≠Ig\\neq Iis not inBrB\_\{r\}, so by the inclusion⊆\\subseteqit is a symmetry only jointly with these per\-channel values\. The induction compares the two runs at equal clipping factors\. Global\-norm clipping multiplies all gradients bymin⁡\(1,C/∥\(∇B,∇A\)∥\)\\min\(1,C/\\lVert\(\\nabla\_\{B\},\\nabla\_\{A\}\)\\rVert\), and∥\(∇B′,∇A′\)∥2=∑jdj2∥∇B,:,j∥2\+∑jdj−2∥∇A,j,:∥2\\lVert\(\\nabla\_\{B^\{\\prime\}\},\\nabla\_\{A^\{\\prime\}\}\)\\rVert^\{2\}=\\sum\_\{j\}d\_\{j\}^\{2\}\\lVert\\nabla\_\{B,:,j\}\\rVert^\{2\}\+\\sum\_\{j\}d\_\{j\}^\{\-2\}\\lVert\\nabla\_\{A,j,:\}\\rVert^\{2\}differs from the original norm ford≠𝟏d\\neq\\mathbf\{1\}in general; the two runs then weight their gradient histories differently, and the relations above fail\. This is why the joint symmetry is stated without global\-norm clipping\.

\(c\) By[Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(a\) the undamped scaled\-GD step satisfies \([21](https://arxiv.org/html/2610.12728#A3.E21)\) for everyg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\. With heavy\-ball momentum on the preconditioned gradients,pB=∇B\(A​A⊤\)−1p\_\{B\}=\\nabla\_\{B\}\(AA^\{\\top\}\)^\{\-1\}transforms aspB′=pB​g−1p\_\{B^\{\\prime\}\}=p\_\{B\}g^\{\-1\}, so a buffer of such terms transforms in the same way and so does the step\. With momentum on the raw gradients followed by preconditioning, the buffer transforms asbB′=bB​g⊤b\_\{B^\{\\prime\}\}=b\_\{B\}g^\{\\top\}because all past points are related by the samegg, andbB′​\(A′​A′⁣⊤\)−1=bB​g⊤​g−⁣⊤​\(A​A⊤\)−1​g−1b\_\{B^\{\\prime\}\}\(A^\{\\prime\}A^\{\\prime\\top\}\)^\{\-1\}=b\_\{B\}g^\{\\top\}g^\{\-\\top\}\(AA^\{\\top\}\)^\{\-1\}g^\{\-1\}is again transported\. TheAA\-factor is symmetric\. So from gauge\-related initial states, with zero or correspondingly transformed momentum,qt′=g⋅qtq^\{\\prime\}\_\{t\}=g\\cdot q\_\{t\}for allttand theWW\-trajectories coincide\. Each iterate lies inW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}, and the iteration is defined as long as both factors keep full rank; a step can lower the rank ofB\+δ​BB\+\\delta BorA\+δ​AA\+\\delta A, so the map goes intoW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}rather thanW0\+ℳrW\_\{0\}\+\\mathcal\{M\}\_\{r\}\.

The hierarchyBr⊂O⁡\(r\)⊂GLrB\_\{r\}\\subset\\mathrm\{O\}\(r\)\\subset\\mathrm\{GL\}\_\{r\}is strict forr≥2r\\geq 2, sinceBrB\_\{r\}is finite,O⁡\(r\)\\mathrm\{O\}\(r\)is infinite and compact, andGLr\\mathrm\{GL\}\_\{r\}is not compact\. ∎

*Damped scaled GD\.*With a damping constantδ\>0\\delta\>0the two inverses become\(A​A⊤\+δ​I\)−1\(AA^\{\\top\}\+\\delta I\)^\{\-1\}and\(B⊤​B\+δ​I\)−1\(B^\{\\top\}B\+\\delta I\)^\{\-1\}\. Equivariance of theBB\-step requires, for all∇B\\nabla\_\{B\},

∇Bg⊤​\(g​A​A⊤​g⊤\+δ​I\)−1=∇B\(A​A⊤\+δ​I\)−1​g−1,that is,g⊤​\(g​A​A⊤​g⊤\+δ​I\)−1=\(A​A⊤\+δ​I\)−1​g−1\.\\nabla\_\{B\}g^\{\\top\}\(gAA^\{\\top\}g^\{\\top\}\+\\delta I\)^\{\-1\}=\\nabla\_\{B\}\(AA^\{\\top\}\+\\delta I\)^\{\-1\}g^\{\-1\},\\quad\\text\{that is,\}\\quad g^\{\\top\}\(gAA^\{\\top\}g^\{\\top\}\+\\delta I\)^\{\-1\}=\(AA^\{\\top\}\+\\delta I\)^\{\-1\}g^\{\-1\}\.Inverting both sides gives\(g​A​A⊤​g⊤\+δ​I\)​g−⁣⊤=g⁡\(A​A⊤\+δ​I\)\(gAA^\{\\top\}g^\{\\top\}\+\\delta I\)g^\{\-\\top\}=g\(AA^\{\\top\}\+\\delta I\), that is,g​A​A⊤\+δ​g−⁣⊤=g​A​A⊤\+δ​ggAA^\{\\top\}\+\\delta g^\{\-\\top\}=gAA^\{\\top\}\+\\delta g, sog−⁣⊤=gg^\{\-\\top\}=gandg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)\. Conversely, for orthogonalggwe have\(g​A​A⊤​g⊤\+δ​I\)−1=g​\(A​A⊤\+δ​I\)−1​g⊤\(gAA^\{\\top\}g^\{\\top\}\+\\delta I\)^\{\-1\}=g\(AA^\{\\top\}\+\\delta I\)^\{\-1\}g^\{\\top\}, and both steps are transported\. The computation does not use full rank, so it holds at every point, includingB=0B=0\.

*Scaled AdamW*applies Adam to the preconditioned gradientpB=∇B\(A​A⊤\+δ​I\)−1p\_\{B\}=\\nabla\_\{B\}\(AA^\{\\top\}\+\\delta I\)^\{\-1\}\(and similarly forAA\), withδ≥0\\delta\\geq 0\. For a signed permutationgg,pB′=pB​g−1p\_\{B^\{\\prime\}\}=p\_\{B\}g^\{\-1\}by the previous paragraph, and Adam commutes with signed permutations, so the rule isBrB\_\{r\}\-equivariant\. Conversely, the first step requiresϕ⁡\(∇BM′\)=ϕ⁡\(∇BM\)​g−1\\phi\(\\nabla\_\{B\}M^\{\\prime\}\)=\\phi\(\\nabla\_\{B\}M\)g^\{\-1\}for all∇B\\nabla\_\{B\}, withM=\(A​A⊤\+δ​I\)−1M=\(AA^\{\\top\}\+\\delta I\)^\{\-1\}andM′=g⊤​\(g​A​A⊤​g⊤\+δ​I\)−1M^\{\\prime\}=g^\{\\top\}\(gAA^\{\\top\}g^\{\\top\}\+\\delta I\)^\{\-1\}invertible\. Writingy=x​My=xMand lettingyygrow, as in the caseε\>0\\varepsilon\>0above, givessign⁡\(y​M−1​M′\)=sign⁡\(y\)​g−1\\operatorname\{sign\}\(yM^\{\-1\}M^\{\\prime\}\)=\\operatorname\{sign\}\(y\)g^\{\-1\}for allyy, and the orthant argument, which used only invertibility of the matrix on the left, givesg−1∈Brg^\{\-1\}\\in B\_\{r\}\.

*LoRA\-Pro with the Sylvester\-optimalXX\.*Write𝒫=ℝm×r×ℝr×n\\mathcal\{P\}=\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}with its Frobenius inner product,u=\(uB,uA\)∈𝒫u=\(u\_\{B\},u\_\{A\}\)\\in\\mathcal\{P\}for a parameter velocity, and𝒦q=ker⁡D​ρq\\mathcal\{K\}\_\{q\}=\\ker D\\rho\_\{q\}\. On the full\-rank locus𝒦q=\{\(−B​X,X​A\):X∈ℝr×r\}\\mathcal\{K\}\_\{q\}=\\\{\(\-BX,XA\):X\\in\\mathbb\{R\}^\{r\\times r\}\\\}: ifB˙​A=−B​A˙\\dot\{B\}A=\-B\\dot\{A\}, thenB˙=−B​X\\dot\{B\}=\-BXwithX=A˙​A⊤​\(A​A⊤\)−1X=\\dot\{A\}A^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}, andB​X​A=B​A˙BXA=B\\dot\{A\}givesA˙=X​A\\dot\{A\}=XA\. By §[C\.27](https://arxiv.org/html/2610.12728#A3.SS27)\(b\), LoRA\-Pro’s directions\(gB,gA\)\(g^\{B\},g^\{A\}\)form the affine spaceℳq​\(G\)=\{u:D​ρq​u=Π𝒯​G\}\\mathcal\{M\}\_\{q\}\(G\)=\\\{u:D\\rho\_\{q\}u=\\Pi\_\{\\mathcal\{T\}\}G\\\}, and LoRA\-Pro choosesXXby minimising the Frobenius distance from\(gB,gA\)\(g^\{B\},g^\{A\}\)to LoRA’s own gradientsD​ρq⊤​G=\(s​G​A⊤,s​B⊤​G\)D\\rho\_\{q\}^\{\\top\}G=\(sGA^\{\\top\},sB^\{\\top\}G\); the minimiser solves a Sylvester equation \([Background D\.33](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5),[Auxiliary result B\.20](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\)\. SinceD​ρq⊤​G⟂𝒦qD\\rho\_\{q\}^\{\\top\}G\\perp\\mathcal\{K\}\_\{q\}, foru∈ℳq​\(G\)u\\in\\mathcal\{M\}\_\{q\}\(G\)the inner product⟨u,D​ρq⊤​G⟩\\langle u,D\\rho\_\{q\}^\{\\top\}G\\rangleis constant, and minimising∥u−D​ρq⊤​G∥2\\lVert u\-D\\rho\_\{q\}^\{\\top\}G\\rVert^\{2\}overℳq​\(G\)\\mathcal\{M\}\_\{q\}\(G\)is the same as minimising∥u∥2\\lVert u\\rVert^\{2\}\. So LoRA\-Pro’s direction is the minimum\-norm solutionuq​\(G\)u\_\{q\}\(G\)ofD​ρq​u=Π𝒯​GD\\rho\_\{q\}u=\\Pi\_\{\\mathcal\{T\}\}G, that is,uq​\(G\)=D​ρq\+​Gu\_\{q\}\(G\)=D\\rho\_\{q\}^\{\+\}G\.

We show thatug⋅q​\(G\)=φg​\(uq​\(G\)\)u\_\{g\\cdot q\}\(G\)=\\varphi\_\{g\}\(u\_\{q\}\(G\)\)for allGGiffg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\), whereφg​\(u\)=\(uB​g−1,g​uA\)\\varphi\_\{g\}\(u\)=\(u\_\{B\}g^\{\-1\},gu\_\{A\}\)is the linear action\. SinceD​ρg⋅q​φg=D​ρqD\\rho\_\{g\\cdot q\}\\varphi\_\{g\}=D\\rho\_\{q\}andΠ𝒯\\Pi\_\{\\mathcal\{T\}\}depends only onWW, substitutingu=φg​\(w\)u=\\varphi\_\{g\}\(w\)shows thatug⋅q​\(G\)=φg​\(w∗\)u\_\{g\\cdot q\}\(G\)=\\varphi\_\{g\}\(w^\{\*\}\), wherew∗w^\{\*\}minimises⟨w,𝒩​w⟩\\langle w,\\mathcal\{N\}w\\rangleoverℳq​\(G\)\\mathcal\{M\}\_\{q\}\(G\)with𝒩=φg⊤​φg\\mathcal\{N\}=\\varphi\_\{g\}^\{\\top\}\\varphi\_\{g\}\. The adjoint ofφg\\varphi\_\{g\}isφg⊤​\(vB,vA\)=\(vB​g−⁣⊤,g⊤​vA\)\\varphi\_\{g\}^\{\\top\}\(v\_\{B\},v\_\{A\}\)=\(v\_\{B\}g^\{\-\\top\},g^\{\\top\}v\_\{A\}\), since⟨wB​g−1,vB⟩=⟨wB,vB​g−⁣⊤⟩\\langle w\_\{B\}g^\{\-1\},v\_\{B\}\\rangle=\\langle w\_\{B\},v\_\{B\}g^\{\-\\top\}\\rangleand⟨g​wA,vA⟩=⟨wA,g⊤​vA⟩\\langle gw\_\{A\},v\_\{A\}\\rangle=\\langle w\_\{A\},g^\{\\top\}v\_\{A\}\\rangle, so𝒩⁡\(wB,wA\)=\(wB​g−1​g−⁣⊤,g⊤​g​wA\)=\(wB​\(g⊤​g\)−1,\(g⊤​g\)​wA\)\\mathcal\{N\}\(w\_\{B\},w\_\{A\}\)=\(w\_\{B\}g^\{\-1\}g^\{\-\\top\},g^\{\\top\}gw\_\{A\}\)=\(w\_\{B\}\(g^\{\\top\}g\)^\{\-1\},\(g^\{\\top\}g\)w\_\{A\}\)\. So equivariance holds iff, on every coset of𝒦q\\mathcal\{K\}\_\{q\}\(asGGvaries,ℳq​\(G\)\\mathcal\{M\}\_\{q\}\(G\)runs through all cosets, sinceD​ρqD\\rho\_\{q\}maps onto𝒯\\mathcal\{T\}\), the𝒩\\mathcal\{N\}\-minimum\-norm point equals the Euclidean one\. The two minimisers are the intersections of the coset with𝒦q⟂𝒩=𝒩−1\(𝒦q⟂\)\\mathcal\{K\}\_\{q\}^\{\\perp\_\{\\mathcal\{N\}\}\}=\\mathcal\{N\}^\{\-1\}\(\\mathcal\{K\}\_\{q\}^\{\\perp\}\)and with𝒦q⟂\\mathcal\{K\}\_\{q\}^\{\\perp\}; they agree on every coset iff these two complements are equal \(each point of either complement is the minimiser on its own coset\), iff𝒩​𝒦q=𝒦q\\mathcal\{N\}\\mathcal\{K\}\_\{q\}=\\mathcal\{K\}\_\{q\}\(𝒩\\mathcal\{N\}is symmetric and invertible\)\. PutH=g⊤​g≻0H=g^\{\\top\}g\\succ 0\. Then𝒩⁡\(−B​X,X​A\)=\(−B​X​H−1,H​X​A\)\\mathcal\{N\}\(\-BX,XA\)=\(\-BXH^\{\-1\},HXA\)lies in𝒦q\\mathcal\{K\}\_\{q\}iff someYYhasB​Y=B​X​H−1BY=BXH^\{\-1\}andY​A=H​X​AYA=HXA, that is \(asBBis injective andAAhas full row rank\),X​H−1=H​XXH^\{\-1\}=HX\. Requiring this for everyX∈ℝr×rX\\in\\mathbb\{R\}^\{r\\times r\}, the matrix\-unit lemma of §[C\.26](https://arxiv.org/html/2610.12728#A3.SS26)givesH−1=c​I=HH^\{\-1\}=cI=H, soc2=1c^\{2\}=1, andc=1c=1becauseHHis positive definite\. Henceg⊤​g=Ig^\{\\top\}g=Iandg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)\. Conversely, orthogonalgggive𝒩=I\\mathcal\{N\}=I\. So the parameter update with the Sylvester\-optimalXXis equivariant exactly underO⁡\(r\)\\mathrm\{O\}\(r\)\.

The variant withX=0X=0isgA=s−1​\(B⊤​B\)−1​B⊤​Gg^\{A\}=s^\{\-1\}\(B^\{\\top\}B\)^\{\-1\}B^\{\\top\}G,gB=s−1​\(I−PU\)​G​A⊤​\(A​A⊤\)−1g^\{B\}=s^\{\-1\}\(I\-P\_\{U\}\)GA^\{\\top\}\(AA^\{\\top\}\)^\{\-1\}\. Atg⋅qg\\cdot q,PUP\_\{U\}is unchanged and

gA′=s−1​g​\(B⊤​B\)−1​g⊤​g−⁣⊤​B⊤​G=g​gA,gB′=s−1​\(I−PU\)​G​A⊤​g⊤​g−⁣⊤​\(A​A⊤\)−1​g−1=gB​g−1,g^\{A^\{\\prime\}\}=s^\{\-1\}g\(B^\{\\top\}B\)^\{\-1\}g^\{\\top\}g^\{\-\\top\}B^\{\\top\}G=g\\,g^\{A\},\\qquad g^\{B^\{\\prime\}\}=s^\{\-1\}\(I\-P\_\{U\}\)GA^\{\\top\}g^\{\\top\}g^\{\-\\top\}\(AA^\{\\top\}\)^\{\-1\}g^\{\-1\}=g^\{B\}g^\{\-1\},so it isGLr\\mathrm\{GL\}\_\{r\}\-equivariant and its discreteWW\-step descends exactly\. The first\-orderWW\-step of either variant is−η​Π𝒯​G\-\\eta\\Pi\_\{\\mathcal\{T\}\}G\([Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\(b\)\)\. For the Sylvester variant,φg−1​ug⋅q​\(G\)−uq​\(G\)∈𝒦q\\varphi\_\{g\}^\{\-1\}u\_\{g\\cdot q\}\(G\)\-u\_\{q\}\(G\)\\in\\mathcal\{K\}\_\{q\}equals\(−B​Y,Y​A\)\(\-BY,YA\)for someYY, and the second\-order terms​δ​B​δ​A=η2​s​gB​gAs\\,\\delta B\\,\\delta A=\\eta^\{2\}s\\,g^\{B\}g^\{A\}becomesη2​s​\(gB−B​Y\)​\(gA\+Y​A\)\\eta^\{2\}s\(g^\{B\}\-BY\)\(g^\{A\}\+YA\)atg⋅qg\\cdot q\. The differenceη2​s​\(gB​Y​A−B​Y​gA−B​Y2​A\)\\eta^\{2\}s\(g^\{B\}YA\-BYg^\{A\}\-BY^\{2\}A\)is exactlyη2\\eta^\{2\}times a matrix that does not depend onη\\eta, and it is nonzero in general\. Numerically, at random full\-rank points with\(m,n,r\)=\(7,6,3\)\(m,n,r\)=\(7,6,3\),s=0\.8s=0\.8andgga Gaussian matrix plus2​I2I, the largest entry of the difference of the twoWW\-steps atη=0\.1\\eta=0\.1ranges from8×10−48\\times 10^\{\-4\}to1\.2×10−21\.2\\times 10^\{\-2\}over seven random draws, and it falls by a factor of exactly100100each timeη\\etafalls by1010\. It is of order10−1510^\{\-15\}\(rounding\) for an orthogonalggand for the variant withX=0X=0\.

*LoRA\-RITE\.*In the notation of[Yen et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib533)the update isA​B⊤AB^\{\\top\}withA∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\}andB∈ℝn×rB\\in\\mathbb\{R\}^\{n\\times r\}, and the gauge isA2=A1​RA\_\{2\}=A\_\{1\}R,B2=B1​R−⁣⊤B\_\{2\}=B\_\{1\}R^\{\-\\top\}withR∈GLrR\\in\\mathrm\{GL\}\_\{r\}\. We follow their Algorithm 1 line by line on the full\-rank locus, where its pseudo\-inverses are inverses\. The rule is symmetric inAAandBB, so we treat the step ofAA\. Algorithm 1 writesB=UB​RBB=U\_\{B\}R\_\{B\}by polar decomposition, withUBU\_\{B\}having orthonormal columns, and forms, at steptt:

- •the unmagnified gradient∇¯A,t=∇A,tRBt−1\\bar\{\\nabla\}\_\{A,t\}=\\nabla\_\{A,t\}R\_\{B\_\{t\}\}^\{\-1\}and the basis changePA,t=UBt⊤​UBt−1P\_\{A,t\}=U\_\{B\_\{t\}\}^\{\\top\}U\_\{B\_\{t\-1\}\};
- •a second momentV¯A,t\\bar\{V\}\_\{A,t\}, a combination with fixed weights ofPA,t​V¯A,t−1​PA,t⊤P\_\{A,t\}\\bar\{V\}\_\{A,t\-1\}P\_\{A,t\}^\{\\top\}and∇¯A,t⊤​∇¯A,t\\bar\{\\nabla\}\_\{A,t\}^\{\\top\}\\bar\{\\nabla\}\_\{A,t\};
- •the escaped mass ρA,t=ρA,t−1\+dλ​\(V¯A,t−1,PA,t​V¯A,t−1​PA,t⊤\),dλ​\(E1,E2\)=maxi⁡\|λi​\(E1\)−λi​\(E2\)\|,\\rho\_\{A,t\}=\\rho\_\{A,t\-1\}\+d\_\{\\lambda\}\\big\(\\bar\{V\}\_\{A,t\-1\},P\_\{A,t\}\\bar\{V\}\_\{A,t\-1\}P\_\{A,t\}^\{\\top\}\\big\),\\qquad d\_\{\\lambda\}\(E\_\{1\},E\_\{2\}\)=\\max\_\{i\}\\lvert\\lambda\_\{i\}\(E\_\{1\}\)\-\\lambda\_\{i\}\(E\_\{2\}\)\\rvert,which depends only on the spectra of its two arguments;
- •the preconditioned step, the first moment and the step S¯A,t=∇¯A,t\(V¯A,t\+ρA,tI\)−1/2,M¯A,t=β1M¯A,t−1PA,t⊤\+\(1−β1\)S¯A,t,δAt=−ηM¯A,tRBt−⁣⊤\.\\bar\{S\}\_\{A,t\}=\\bar\{\\nabla\}\_\{A,t\}\(\\bar\{V\}\_\{A,t\}\+\\rho\_\{A,t\}I\)^\{\-1/2\},\\quad\\bar\{M\}\_\{A,t\}=\\beta\_\{1\}\\bar\{M\}\_\{A,t\-1\}P\_\{A,t\}^\{\\top\}\+\(1\-\\beta\_\{1\}\)\\bar\{S\}\_\{A,t\},\\quad\\delta A\_\{t\}=\-\\eta\\,\\bar\{M\}\_\{A,t\}R\_\{B\_\{t\}\}^\{\-\\top\}\.

Run the algorithm from\(A1,B1\)\(A\_\{1\},B\_\{1\}\)and from\(A2,B2\)=\(A1​R,B1​R−⁣⊤\)\(A\_\{2\},B\_\{2\}\)=\(A\_\{1\}R,B\_\{1\}R^\{\-\\top\}\), with zero initial moments and escaped mass\. Suppose that at stepttthe parameters of the two runs are related byRR\. ThenB2,tB\_\{2,t\}andB1,tB\_\{1,t\}have the same column space, soUB2,t=UB1,t​OtU\_\{B\_\{2,t\}\}=U\_\{B\_\{1,t\}\}O\_\{t\}for someOt∈O⁡\(r\)O\_\{t\}\\in\\mathrm\{O\}\(r\), andRB2,t=UB2,t⊤​B2,t=Ot⊤​RB1,t​R−⁣⊤R\_\{B\_\{2,t\}\}=U\_\{B\_\{2,t\}\}^\{\\top\}B\_\{2,t\}=O\_\{t\}^\{\\top\}R\_\{B\_\{1,t\}\}R^\{\-\\top\}\. WithGGthe gradient in the weight,∇A2=G​B2=∇A1R−⁣⊤\\nabla\_\{A\_\{2\}\}=GB\_\{2\}=\\nabla\_\{A\_\{1\}\}R^\{\-\\top\}, so

∇¯A2,t=∇A1,tR−⁣⊤​R⊤​RB1,t−1​Ot=∇¯A1,t​Ot\.\\bar\{\\nabla\}\_\{A\_\{2\},t\}=\\nabla\_\{A\_\{1\},t\}R^\{\-\\top\}R^\{\\top\}R\_\{B\_\{1,t\}\}^\{\-1\}O\_\{t\}=\\bar\{\\nabla\}\_\{A\_\{1\},t\}O\_\{t\}\.Suppose also that the internal states are related byV¯t−1\(2\)=Ot−1⊤​V¯t−1\(1\)​Ot−1\\bar\{V\}^\{\(2\)\}\_\{t\-1\}=O\_\{t\-1\}^\{\\top\}\\bar\{V\}^\{\(1\)\}\_\{t\-1\}O\_\{t\-1\},M¯t−1\(2\)=M¯t−1\(1\)​Ot−1\\bar\{M\}^\{\(2\)\}\_\{t\-1\}=\\bar\{M\}^\{\(1\)\}\_\{t\-1\}O\_\{t\-1\}andρt−1\(2\)=ρt−1\(1\)\\rho^\{\(2\)\}\_\{t\-1\}=\\rho^\{\(1\)\}\_\{t\-1\}; att=1t=1this holds because all three start at zero\. ThenPA,t\(2\)=Ot⊤​PA,t\(1\)​Ot−1P^\{\(2\)\}\_\{A,t\}=O\_\{t\}^\{\\top\}P^\{\(1\)\}\_\{A,t\}O\_\{t\-1\}, soPA,t\(2\)​V¯t−1\(2\)​PA,t\(2\)⊤=Ot⊤​PA,t\(1\)​V¯t−1\(1\)​PA,t\(1\)⊤​OtP^\{\(2\)\}\_\{A,t\}\\bar\{V\}^\{\(2\)\}\_\{t\-1\}P^\{\(2\)\\top\}\_\{A,t\}=O\_\{t\}^\{\\top\}P^\{\(1\)\}\_\{A,t\}\\bar\{V\}^\{\(1\)\}\_\{t\-1\}P^\{\(1\)\\top\}\_\{A,t\}O\_\{t\}andV¯t\(2\)=Ot⊤​V¯t\(1\)​Ot\\bar\{V\}^\{\(2\)\}\_\{t\}=O\_\{t\}^\{\\top\}\\bar\{V\}^\{\(1\)\}\_\{t\}O\_\{t\}\. Conjugation by an orthogonal matrix keeps eigenvalues, soρt\(2\)=ρt\(1\)\\rho^\{\(2\)\}\_\{t\}=\\rho^\{\(1\)\}\_\{t\}, and it commutes with\(⋅\+ρI\)−1/2\(\\,\\cdot\+\\rho I\)^\{\-1/2\}, soS¯t\(2\)=S¯t\(1\)​Ot\\bar\{S\}^\{\(2\)\}\_\{t\}=\\bar\{S\}^\{\(1\)\}\_\{t\}O\_\{t\}\. SinceM¯t−1\(2\)​PA,t\(2\)⊤=M¯t−1\(1\)​PA,t\(1\)⊤​Ot\\bar\{M\}^\{\(2\)\}\_\{t\-1\}P^\{\(2\)\\top\}\_\{A,t\}=\\bar\{M\}^\{\(1\)\}\_\{t\-1\}P^\{\(1\)\\top\}\_\{A,t\}O\_\{t\}, alsoM¯t\(2\)=M¯t\(1\)​Ot\\bar\{M\}^\{\(2\)\}\_\{t\}=\\bar\{M\}^\{\(1\)\}\_\{t\}O\_\{t\}\. FinallyRB2,t−⁣⊤=Ot⊤​RB1,t−⁣⊤​RR\_\{B\_\{2,t\}\}^\{\-\\top\}=O\_\{t\}^\{\\top\}R\_\{B\_\{1,t\}\}^\{\-\\top\}R, so

δ​A2,t=−η​M¯t\(1\)​Ot​Ot⊤​RB1,t−⁣⊤​R=δ​A1,t​R,\\delta A\_\{2,t\}=\-\\eta\\,\\bar\{M\}^\{\(1\)\}\_\{t\}O\_\{t\}O\_\{t\}^\{\\top\}R\_\{B\_\{1,t\}\}^\{\-\\top\}R=\\delta A\_\{1,t\}R,the transported step\. The step ofBBis the same computation withAAandBBexchanged andRRreplaced byR−⁣⊤R^\{\-\\top\}, soδ​B2,t=δ​B1,t​R−⁣⊤\\delta B\_\{2,t\}=\\delta B\_\{1,t\}R^\{\-\\top\}\. The new parameters are again related byRR, and the new internal states by the orthogonal matricesOtO\_\{t\}, which completes the induction\. The resulting step agrees with Eq\. 15 of[Yen et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib533),δ​A2=δ​A1​B1⊤​\(B2⊤\)\+\\delta A\_\{2\}=\\delta A\_\{1\}B\_\{1\}^\{\\top\}\(B\_\{2\}^\{\\top\}\)^\{\+\}, since\(B2⊤\)\+=B1​\(B1⊤​B1\)−1​R\(B\_\{2\}^\{\\top\}\)^\{\+\}=B\_\{1\}\(B\_\{1\}^\{\\top\}B\_\{1\}\)^\{\-1\}RwhenB1B\_\{1\}has full column rank, and with the numerical replication reported after[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8), which agrees to3×10−143\\times 10^\{\-14\}\. LoRA\-RITE is therefore an Adam\-type optimizer in class \(c\)\. ∎

### C\.36 Corollary III\.13

Corollary III\.13\(what a zero\-BBinitialisation really chooses\)\.Atq0=\(0,A0\)q\_\{0\}=\(0,A\_\{0\}\)withrank⁡A0=r\\rk A\_\{0\}=r, with hyperparameters shared by the rank channels and the same data order and dropout masks, two choices ofA0A\_\{0\}give the sameθ\\theta\-trajectory whenever the optimizer’s gauge group relates them\. The resulting invariant ofA0A\_\{0\}is:•row⁡A0∈Gr⁡\(r,n\)\\operatorname\{row\}A\_\{0\}\\in\\Gr\(r,n\)for an update rule that isGLr\\mathrm\{GL\}\_\{r\}\-equivariant at every point of its trajectory, includingB=0B=0and rank\-deficientBB, run without global\-norm clipping\. An example isδ​B=−η​∇B\(A​A⊤\)−1\\delta B=\-\\eta\\nabla\_\{B\}\(AA^\{\\top\}\)^\{\-1\},δ​A=−η⁡\(A​A⊤\)​∇A\\delta A=\-\\eta\(AA^\{\\top\}\)\\nabla\_\{A\}, which needs onlyrank⁡A=r\\rk A=r;•the PSD matrixA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}for SGD, which is exactly the initial preconditioner inW˙​\(0\)=−s2​ηB​G​A0⊤​A0\\dot\{W\}\(0\)=\-s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\};•A0A\_\{0\}up to signed permutation of its rows for Adam\.The undamped scaled\-GD optimizers of[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(c\) are not defined atB=0B=0, since they need\(B⊤​B\)−1\(B^\{\\top\}B\)^\{\-1\}\. LoRA\-RITE uses pseudo\-inverses and so is defined there, but itsGLr\\mathrm\{GL\}\_\{r\}\-equivariance is established only on the full\-rank locus\. Their damped forms are onlyO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant, so in practice their invariant is aδ\\delta\-dependent function ofA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}, and their Adam\-wrapped forms \(Riemannian\-preconditioned AdamW, HF’s LoRA\-FA optimizer\) have the invariantA0A\_\{0\}moduloBrB\_\{r\}\. Conversely, different invariants give different dynamics, for SGD and for Adam atε=0\\varepsilon=0\.Random\-AALoRA and EVA are different points of theB=0B=0stratum\{\(0,A\):rank⁡A=r\}\\\{\(0,A\):\\rk A=r\\\}of the apex fibreρ−1​\(θ0\)=\{\(B,A\):B​A=0\}\\rho^\{\-1\}\(\\theta\_\{0\}\)=\\\{\(B,A\):BA=0\\\}; Init\[B\] lies in another stratum of the same fibre\. LoRA\-FA is a different*object*, the sub\-objectFA⁡\(A0\)↪LoRA\(0,A0\)\\mathrm\{FA\}\(A\_\{0\}\)\\hookrightarrow\\mathrm\{LoRA\}\_\{\(0,A\_\{0\}\)\}\([Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\)\. More precisely, EVA is a point of the apex fibre ofLoRArl\\mathrm\{LoRA\}\_\{r\_\{l\}\}at its per\-layer redistributed rank and adjustedα\\alpha, with orthonormal rows only when whiten=False\.

*Equivariance atB=0B=0\.*Only the inclusions⊇\\supseteqof[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)are used\. For SGD andg∈O⁡\(r\)g\\in\\mathrm\{O\}\(r\)the identitiesδ​B′=−ηB​∇Bg⊤=δ​B​g−1\\delta B^\{\\prime\}=\-\\eta\_\{B\}\\nabla\_\{B\}g^\{\\top\}=\\delta Bg^\{\-1\}andδ​A′=g​δ​A\\delta A^\{\\prime\}=g\\,\\delta Ahold at every point, with no rank condition, and similarly for momentum buffers\. For Adam andg∈Brg\\in B\_\{r\}the coordinatewise argument also holds at every point, and the initial state\(m,v\)=\(0,0\)\(m,v\)=\(0,0\)is fixed byBrB\_\{r\}\. Orthogonalggpreserve∥∇B∥\\lVert\\nabla\_\{B\}\\rVertand∥∇A∥\\lVert\\nabla\_\{A\}\\rVertby \([20](https://arxiv.org/html/2610.12728#A3.E20)\), so global\-norm clipping factors agree in the two runs\. The start transforms asg⋅\(0,A0\)=\(0,g​A0\)g\\cdot\(0,A\_\{0\}\)=\(0,gA\_\{0\}\), since0⋅g−1=00\\cdot g^\{\-1\}=0\. By the induction of §[C\.24](https://arxiv.org/html/2610.12728#A3.SS24),A0A\_\{0\}andg​A0gA\_\{0\}give the sameθ\\theta\-trajectory whenevergglies in the optimizer’s group\. In the first bullet the hypothesis is equivariance underGLr\\mathrm\{GL\}\_\{r\}at every point of the trajectory, and the same induction applies\.

*The example rule\.*Atg⋅qg\\cdot q, by \([20](https://arxiv.org/html/2610.12728#A3.E20)\) and\(A′​A′⁣⊤\)−1=g−⁣⊤​\(A​A⊤\)−1​g−1\(A^\{\\prime\}A^\{\\prime\\top\}\)^\{\-1\}=g^\{\-\\top\}\(AA^\{\\top\}\)^\{\-1\}g^\{\-1\},

δ​B′=−η​∇Bg⊤​g−⁣⊤​\(A​A⊤\)−1​g−1=δ​B​g−1,δ​A′=−η​g​A​A⊤​g⊤​g−⁣⊤​∇A=g​δ​A\.\\delta B^\{\\prime\}=\-\\eta\\nabla\_\{B\}g^\{\\top\}g^\{\-\\top\}\(AA^\{\\top\}\)^\{\-1\}g^\{\-1\}=\\delta B\\,g^\{\-1\},\\qquad\\delta A^\{\\prime\}=\-\\eta\\,gAA^\{\\top\}g^\{\\top\}g^\{\-\\top\}\\nabla\_\{A\}=g\\,\\delta A\.Onlyrank⁡A=r\\rk A=rwas used, soBBmay be00or rank deficient\. Sinceδ​A=−η​A​A⊤​s​B⊤​G=A​N\\delta A=\-\\eta AA^\{\\top\}sB^\{\\top\}G=A\\,NwithN=−η​s​A⊤​B⊤​GN=\-\\eta sA^\{\\top\}B^\{\\top\}G, the step mapsAAtoA⁡\(I\+N\)A\(I\+N\), which keeps rankrrwheneverI\+NI\+Nis invertible, in particular for smallη\\eta; in continuous timeA˙=A​N​\(t\)\\dot\{A\}=AN\(t\)givesA⁡\(t\)=A0​Ψ​\(t\)A\(t\)=A\_\{0\}\\Psi\(t\)withΨ\\Psian invertible fundamental matrix, sorank⁡A⁡\(t\)=r\\rk A\(t\)=rfor alltt\.

*The invariants\.*The invariant ofA0A\_\{0\}is its orbit under the optimizer’s group acting byA0↦g​A0A\_\{0\}\\mapsto gA\_\{0\}\. ForO⁡\(r\)\\mathrm\{O\}\(r\),A0′=O​A0A\_\{0\}^\{\\prime\}=OA\_\{0\}for someO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\)iffA0′⁣⊤​A0′=A0⊤​A0A\_\{0\}^\{\\prime\\top\}A\_\{0\}^\{\\prime\}=A\_\{0\}^\{\\top\}A\_\{0\}\([Theorem II\.5](https://arxiv.org/html/2610.12728#S2.SS2.SSSx1)\(2\)\)\. ForBrB\_\{r\}the orbit is the set of signed row permutations ofA0A\_\{0\}\. ForGLr\\mathrm\{GL\}\_\{r\}, two full\-row\-rank matrices satisfyA0′=g​A0A\_\{0\}^\{\\prime\}=gA\_\{0\}for some \(unique\)ggiff they have the same row space \(proof of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\), so the invariant isrow⁡A0∈Gr⁡\(r,n\)\\operatorname\{row\}A\_\{0\}\\in\\Gr\(r,n\)\([Background C\.15](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\. AtB0=0B\_\{0\}=0the LoRA cometric of[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)isK\(0,A0\)​\(G\)=s2​ηB​G​A0⊤​A0K\_\{\(0,A\_\{0\}\)\}\(G\)=s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}, soW˙​\(0\)=−s2​ηB​G​A0⊤​A0\\dot\{W\}\(0\)=\-s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}\.

*Practical forms\.*Undamped scaled GD needs\(B⊤​B\)−1\(B^\{\\top\}B\)^\{\-1\}, which does not exist atB=0B=0\. The damped forms areO⁡\(r\)\\mathrm\{O\}\(r\)\-equivariant at every point \(§[C\.35](https://arxiv.org/html/2610.12728#A3.SS35), published variants\), so their trajectories from\(0,A0\)\(0,A\_\{0\}\)depend onA0A\_\{0\}only throughA0⊤​A0A\_\{0\}^\{\\top\}A\_\{0\}, in a way that depends onδ\\delta\. A preconditioned step wrapped in Adam\(W\) isBrB\_\{r\}\-equivariant at every point \(same place\), so the invariant isA0A\_\{0\}moduloBrB\_\{r\}\.

*Converse for SGD\.*The first SGD step from\(0,A0\)\(0,A\_\{0\}\)isδ​B=−ηB​s​G​A0⊤\\delta B=\-\\eta\_\{B\}sGA\_\{0\}^\{\\top\},δ​A=−ηA​s​B0⊤​G=0\\delta A=\-\\eta\_\{A\}sB\_\{0\}^\{\\top\}G=0, soδ​W1=s​δ​B​A0=−s2​ηB​G​A0⊤​A0\\delta W\_\{1\}=s\\,\\delta B\\,A\_\{0\}=\-s^\{2\}\\eta\_\{B\}GA\_\{0\}^\{\\top\}A\_\{0\}exactly\. IfA0A\_\{0\}andA0′A\_\{0\}^\{\\prime\}give the same trajectory for every loss, the linear losses⟨G,⋅⟩\\langle G,\\cdot\\ranglegiveG​A0⊤​A0=G​A0′⁣⊤​A0′GA\_\{0\}^\{\\top\}A\_\{0\}=GA\_\{0\}^\{\\prime\\top\}A\_\{0\}^\{\\prime\}for everyGG, soA0⊤​A0=A0′⁣⊤​A0′A\_\{0\}^\{\\top\}A\_\{0\}=A\_\{0\}^\{\\prime\\top\}A\_\{0\}^\{\\prime\}\. Hence different invariants give different first steps for some loss\.

*Converse for Adam atε=0\\varepsilon=0\.*The first step from\(0,A0\)\(0,A\_\{0\}\)with zero state has∇A=0\\nabla\_\{A\}=0, soδ​A=0\\delta A=0, andδ​B=−ηB​sign⁡\(s​G​A0⊤\)\\delta B=\-\\eta\_\{B\}\\operatorname\{sign\}\(sGA\_\{0\}^\{\\top\}\), soδ​W1=−s​ηB​sign⁡\(G​A0⊤\)​A0\\delta W\_\{1\}=\-s\\eta\_\{B\}\\operatorname\{sign\}\(GA\_\{0\}^\{\\top\}\)A\_\{0\}\(without bias correction, a positive multiple of this\)\. IfA0A\_\{0\}andA0′A\_\{0\}^\{\\prime\}give the same trajectory for every loss, then row by row

F⁡\(x\):=sign⁡\(x​A0⊤\)​A0=∑j=1rsign⁡\(x⋅aj\)​ajF\(x\):=\\operatorname\{sign\}\(xA\_\{0\}^\{\\top\}\)A\_\{0\}=\\sum\_\{j=1\}^\{r\}\\operatorname\{sign\}\(x\\cdot a\_\{j\}\)\\,a\_\{j\}equalsF′​\(x\)=∑jsign⁡\(x⋅aj′\)​aj′F^\{\\prime\}\(x\)=\\sum\_\{j\}\\operatorname\{sign\}\(x\\cdot a^\{\\prime\}\_\{j\}\)\\,a^\{\\prime\}\_\{j\}for everyx∈ℝ1×nx\\in\\mathbb\{R\}^\{1\\times n\}, whereaja\_\{j\}andaj′a^\{\\prime\}\_\{j\}are the rows ofA0A\_\{0\}andA0′A\_\{0\}^\{\\prime\}\. The rowsaja\_\{j\}are linearly independent, so the hyperplanesHj=aj⟂H\_\{j\}=a\_\{j\}^\{\\perp\}are pairwise distinct, and eachHjH\_\{j\}contains pointsxxlying on no otherHkH\_\{k\}\(a hyperplane is not covered by finitely many proper subspaces of it,[Auxiliary result B\.36](https://arxiv.org/html/2610.12728#A2.SS0.SSSx6)\(b\)\)\. At such a point, crossingHjH\_\{j\}in the direction of increasingx⋅ajx\\cdot a\_\{j\}changesFFby2​aj≠02a\_\{j\}\\neq 0, whileFFis constant on the connected components of the complement of⋃jHj\\bigcup\_\{j\}H\_\{j\}\. So the closure of the set of discontinuities ofFFis⋃jHj\\bigcup\_\{j\}H\_\{j\}, and likewise forF′F^\{\\prime\}\. FromF=F′F=F^\{\\prime\}we get⋃jHj=⋃kHk′\\bigcup\_\{j\}H\_\{j\}=\\bigcup\_\{k\}H^\{\\prime\}\_\{k\}, and since a hyperplane contained in a finite union of hyperplanes equals one of them \([Auxiliary result B\.36](https://arxiv.org/html/2610.12728#A2.SS0.SSSx6)\(b\) again\), there is a permutationπ\\piwithHj=Hπ⁡\(j\)′H\_\{j\}=H^\{\\prime\}\_\{\\pi\(j\)\}, that is,aπ⁡\(j\)′=cj​aja^\{\\prime\}\_\{\\pi\(j\)\}=c\_\{j\}a\_\{j\}withcj≠0c\_\{j\}\\neq 0\. CrossingHjH\_\{j\}as above changesF′F^\{\\prime\}by2​sign⁡\(cj\)​aπ⁡\(j\)′=2​\|cj\|​aj2\\operatorname\{sign\}\(c\_\{j\}\)a^\{\\prime\}\_\{\\pi\(j\)\}=2\\lvert c\_\{j\}\\rvert a\_\{j\}, and comparing with2​aj2a\_\{j\}gives\|cj\|=1\\lvert c\_\{j\}\\rvert=1\. SoA0′A\_\{0\}^\{\\prime\}is a signed row permutation ofA0A\_\{0\}\. For the exampleGLr\\mathrm\{GL\}\_\{r\}\-rule the same argument is immediate: its first step isδ​W1=−η​s2​G​Prow⁡A0\\delta W\_\{1\}=\-\\eta s^\{2\}GP\_\{\\operatorname\{row\}A\_\{0\}\}, which determinesrow⁡A0\\operatorname\{row\}A\_\{0\}\.

*Strata and the sub\-object\.*Sinces\>0s\>0,ρ−1\(θ0\)=\{\(B,A\):W0\+sBA=W0\}=\{BA=0\}\\rho^\{\-1\}\(\\theta\_\{0\}\)=\\\{\(B,A\):W\_\{0\}\+sBA=W\_\{0\}\\\}=\\\{BA=0\\\}\. Random\-AALoRA and EVA start at\(0,A0\)\(0,A\_\{0\}\)withrank⁡A0=r\\rk A\_\{0\}=r, in the stratum\{\(0,A\):rank⁡A=r\}\\\{\(0,A\):\\rk A=r\\\}, at points that differ for a generic random draw; Init\[B\] starts at\(B0,0\)\(B\_\{0\},0\)withrank⁡B0=r\\rk B\_\{0\}=r, in the disjoint stratum\{\(B,0\):rank⁡B=r\}\\\{\(B,0\):\\rk B=r\\\}\. The mapB↦\(B,A0\)B\\mapsto\(B,A\_\{0\}\)is injective, pointed \(0↦\(0,A0\)0\\mapsto\(0,A\_\{0\}\)\) and commutes with the maps toΘ\\Theta, sinceρ⁡\(B,A0\)=W0\+s​B​A0\\rho\(B,A\_\{0\}\)=W\_\{0\}\+sBA\_\{0\}; after the rescalingB↦s​BB\\mapsto sBits source isFA⁡\(A0\)\\mathrm\{FA\}\(A\_\{0\}\)of[Proposition I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\. An injective morphism is a monomorphism, soFA⁡\(A0\)\\mathrm\{FA\}\(A\_\{0\}\)is a sub\-object ofLoRA\(0,A0\)\\mathrm\{LoRA\}\_\{\(0,A\_\{0\}\)\}\([Background A\.15](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx2.Px2)\)\. Its imageW0\+\{X​A0\}W\_\{0\}\+\\\{XA\_\{0\}\\\}has dimensionm​rmr, smaller than the dimensionr⁡\(m\+n−r\)r\(m\+n\-r\)of the imageW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}ofLoRA\(0,A0\)\\mathrm\{LoRA\}\_\{\(0,A\_\{0\}\)\}whenr<nr<n, and images are isomorphism invariants, so the two objects are not isomorphic\. The last sentence describes EVA’s construction and needs no proof\. ∎

### C\.37 Definition IV\.1

Definition IV\.1\(forward and backward methods\)\.•Aforwardmethod acts on the parameter port by a lens in the image of𝖱\\mathsf\{R\}, namely\(ρ,𝖱⁡\[ρ\]\)\(\\rho,\\mathsf\{R\}\[\\rho\]\)\.•Abackward, or cotangent\-side, method acts by a lens whose forward part isidΘ\\mathrm\{id\}\_\{\\Theta\}and whose backward part is a*compression*ct:Tθ∗​Θ→Sc\_\{t\}:T^\{\*\}\_\{\\theta\}\\Theta\\to Sinto a state spaceSS, paired with an*anchor*at:S→Tθ​Θa\_\{t\}:S\\to T\_\{\\theta\}\\Theta\. The update isθ←θ−η​at​\(opt⁡\(ct​\(∇L\)\)\)\.\\theta\\leftarrow\\theta\-\\eta\\,a\_\{t\}\\big\(\\mathrm\{opt\}\(c\_\{t\}\(\\nabla L\)\)\\big\)\.Examples: GaLore, withct=Pt⊤c\_\{t\}=P\_\{t\}^\{\\top\}andat=Pta\_\{t\}=P\_\{t\}forPtP\_\{t\}from a gradient SVD; GaLore with a random projector, the same withPtP\_\{t\}random; LISA, a layer\-block coordinate projection paired with the same projection; MeZO, withct​\(g\)=⟨zt,g⟩c\_\{t\}\(g\)=\\langle z\_\{t\},g\\rangleestimated by finite differences without a backward pass, andat​\(λ\)=λ​zta\_\{t\}\(\\lambda\)=\\lambda z\_\{t\}\. Flora as published fits only as the trivial lens, with compression and anchor the identity and the random compression of its momentum insideopt\\mathrm\{opt\}\.

A definition needs no proof\. This subsection fixes the conventions used in the proofs of Exposé IV and checks the identities that the definition and its table record\.

##### Conventions

As in §[C\.24](https://arxiv.org/html/2610.12728#A3.SS24), weight spaceΘ=ℝm×n\\Theta=\\mathbb\{R\}^\{m\\times n\}, or a product of such boxes, carries the Frobenius inner product, covectors are written as matrices, and the transposeℒ∗\\mathcal\{L\}^\{\*\}of a linear mapℒ\\mathcal\{L\}is its adjoint\. We model a first\-order optimizer as a machine with a state\. At stepttit holds a stateςt\\varsigma\_\{t\}, receives a tensorgtg\_\{t\}\(the gradient it is fed\) and the tensorxtx\_\{t\}it updates, returns an outputUt=Ft​\(ςt,gt,xt\)U\_\{t\}=F\_\{t\}\(\\varsigma\_\{t\},g\_\{t\},x\_\{t\}\), moves to the stateςt\+1=Ht​\(ςt,gt,xt\)\\varsigma\_\{t\+1\}=H\_\{t\}\(\\varsigma\_\{t\},g\_\{t\},x\_\{t\}\), and the update isxt\+1=xt−ηt​Utx\_\{t\+1\}=x\_\{t\}\-\\eta\_\{t\}U\_\{t\}\. The maps may depend on the step countertt, for instance through Adam’s bias corrections or Adafactor’s relative step size\. The optimizer*does not read the parameter tensor*whenFtF\_\{t\}andHtH\_\{t\}do not depend onxtx\_\{t\}\. SGD, heavy\-ball and Nesterov momentum, Adam and AMSGrad, and Adafactor with scale\_parameter=False are of this kind; Adafactor’s factored second moments and its clipping of the update byRMS⁡\(Ut\)\\mathrm\{RMS\}\(U\_\{t\}\)read onlygtg\_\{t\}and the state\. The following ingredients readxtx\_\{t\}, or a gradient other thangtg\_\{t\}:

- •coupled weight decay, which feedsgt\+λ​xtg\_\{t\}\+\\lambda x\_\{t\}in place ofgtg\_\{t\};
- •decoupled weight decay \(AdamW withλ\>0\\lambda\>0\), which adds−ηt​λ​xt\-\\eta\_\{t\}\\lambda x\_\{t\}to the update;
- •LAMB and LARS, which multiply the update by a trust ratioϕ⁡\(∥xt∥\)/∥Ut∥\\phi\(\\lVert x\_\{t\}\\rVert\)/\\lVert U\_\{t\}\\rVert;
- •Adafactor with parameter scaling, which multiplies the step bymax⁡\(ϵ2,RMS⁡\(xt\)\)\\max\(\\epsilon\_\{2\},\\mathrm\{RMS\}\(x\_\{t\}\)\)\(the option scale\_parameter=True\);
- •clipping by the global norm of the full gradient, which rescalesgtg\_\{t\}bymin⁡\(1,τ/∥Gt∥\)\\min\(1,\\tau/\\lVert G\_\{t\}\\rVert\), whereGtG\_\{t\}is the gradient of every trainable tensor of the procedure\.

*The lens of a reparametrisation\.*A smoothρ:Q→Θ\\rho:Q\\to\\Thetahas reverse derivative𝖱⁡\[ρ\]​\(q,γ\)=Jρ​\(q\)⊤​γ=D​ρq⊤​γ\\mathsf\{R\}\[\\rho\]\(q,\\gamma\)=J\_\{\\rho\}\(q\)^\{\\top\}\\gamma=D\\rho\_\{q\}^\{\\top\}\\gamma\([Background B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\. For a compositeρ∘σ\\rho\\circ\\sigmathe chain rule gives

𝖱⁡\[ρ∘σ\]​\(q′,γ\)=D​σq′⊤​D​ρσ⁡\(q′\)⊤​γ=𝖱⁡\[σ\]​\(q′,𝖱⁡\[ρ\]​\(σ⁡\(q′\),γ\)\),\\mathsf\{R\}\[\\rho\\circ\\sigma\]\(q^\{\\prime\},\\gamma\)=D\\sigma\_\{q^\{\\prime\}\}^\{\\top\}D\\rho\_\{\\sigma\(q^\{\\prime\}\)\}^\{\\top\}\\gamma=\\mathsf\{R\}\[\\sigma\]\\big\(q^\{\\prime\},\\mathsf\{R\}\[\\rho\]\(\\sigma\(q^\{\\prime\}\),\\gamma\)\\big\),which is composition of lenses, soρ↦\(ρ,𝖱⁡\[ρ\]\)\\rho\\mapsto\(\\rho,\\mathsf\{R\}\[\\rho\]\)respects composition of reparametrisations\. The gradient that a forward method feeds its optimizer is the pullback∇q\(L∘ρ\)=D​ρq⊤​∇θL=𝖱⁡\[ρ\]​\(q,∇θL\)\\nabla\_\{q\}\(L\\circ\\rho\)=D\\rho\_\{q\}^\{\\top\}\\nabla\_\{\\theta\}L=\\mathsf\{R\}\[\\rho\]\(q,\\nabla\_\{\\theta\}L\)\.

*A forward step to first order\.*One step of a forward method setsq←q−η​Uq\\leftarrow q\-\\eta UwithU=opt\(Dρq⊤∇L\)U=\\mathrm\{opt\}\(D\\rho\_\{q\}^\{\\top\}\\nabla L\), so the weight moves to

ρ⁡\(q−η​U\)=θ−η​D​ρq​U\+O⁡\(η2\)\.\\rho\(q\-\\eta U\)=\\theta\-\\eta\\,D\\rho\_\{q\}\\,U\+O\(\\eta^\{2\}\)\.To first order inη\\etathis is the backward update withc=D​ρq⊤c=D\\rho\_\{q\}^\{\\top\}anda=D​ρqa=D\\rho\_\{q\}\. The two agree exactly whenρ\\rhois affine along the step, which is the situation of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\.

*The examples\.*For GaLore with left projection,Pt∈ℝm×rP\_\{t\}\\in\\mathbb\{R\}^\{m\\times r\}has orthonormal columns,S=ℝr×nS=\\mathbb\{R\}^\{r\\times n\},ct​\(G\)=Pt⊤​Gc\_\{t\}\(G\)=P\_\{t\}^\{\\top\}Gandat​\(U\)=Pt​Ua\_\{t\}\(U\)=P\_\{t\}U, so the updateW←W−η​α​Pt​opt​\(Pt⊤​G\)W\\leftarrow W\-\\eta\\alpha P\_\{t\}\\,\\mathrm\{opt\}\(P\_\{t\}^\{\\top\}G\)is the displayed one withα\\alphaabsorbed intoη\\eta; with right projection,ct​\(G\)=G​Ptc\_\{t\}\(G\)=GP\_\{t\}andat​\(U\)=U​Pt⊤a\_\{t\}\(U\)=UP\_\{t\}^\{\\top\}\. In each caseat=ct∗a\_\{t\}=c\_\{t\}^\{\*\}, since⟨Pt​U,G⟩=⟨U,Pt⊤​G⟩\\langle P\_\{t\}U,G\\rangle=\\langle U,P\_\{t\}^\{\\top\}G\\rangle\. LISA’sSSis the space of the sampled layers \(with the embedding and the head\),ctc\_\{t\}restricts a gradient to it andata\_\{t\}includes it back, so againat=ct∗a\_\{t\}=c\_\{t\}^\{\*\}\. For MeZO,S=ℝS=\\mathbb\{R\},at​\(λ\)=λ​zta\_\{t\}\(\\lambda\)=\\lambda z\_\{t\}andct​\(g\)=⟨zt,g⟩=at∗​\(g\)c\_\{t\}\(g\)=\\langle z\_\{t\},g\\rangle=a\_\{t\}^\{\*\}\(g\)\. MeZO does not computect​\(∇L\)c\_\{t\}\(\\nabla L\); it estimates it by the central difference\(L⁡\(θ\+ε​zt\)−L⁡\(θ−ε​zt\)\)/\(2​ε\)\\big\(L\(\\theta\+\\varepsilon z\_\{t\}\)\-L\(\\theta\-\\varepsilon z\_\{t\}\)\\big\)/\(2\\varepsilon\), whose error isO⁡\(ε2\)O\(\\varepsilon^\{2\}\)for a loss with Lipschitz Hessian \([Auxiliary result B\.34](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(a\)\), so it is the displayed update up to its estimator\. Flora as published \(its reference implementation\) returns\(1−β\)​g^t\+β​upt​\(Mt\)\(1\-\\beta\)\\hat\{g\}\_\{t\}\+\\beta\\,\\mathrm\{up\}\_\{t\}\(M\_\{t\}\), whereg^t\\hat\{g\}\_\{t\}is the Adafactor\-normalised full gradient andupt​\(Mt\)\\mathrm\{up\}\_\{t\}\(M\_\{t\}\)the decompressed momentum\([Hao et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib154)\)\. The first term is generically of full rank, so the update does not lie inim⁡Pt\\operatorname\{im\}P\_\{t\}, and the only reading in the form of the definition isct=at=idc\_\{t\}=a\_\{t\}=\\mathrm\{id\}with the random compression insideopt\\mathrm\{opt\}\.

*The cometric of a backward method\.*With plain \(S\)GD on the compressed gradient,opt=id\\mathrm\{opt\}=\\mathrm\{id\}, the update isθ←θ−ηatct∇L\\theta\\leftarrow\\theta\-\\eta\\,a\_\{t\}c\_\{t\}\\nabla L, so the induced cometric isKt=at​ctK\_\{t\}=a\_\{t\}c\_\{t\}\([Definition III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)\)\. For the first three examplesat=ct∗a\_\{t\}=c\_\{t\}^\{\*\}, soKt=at​at∗K\_\{t\}=a\_\{t\}a\_\{t\}^\{\*\}is symmetric positive semidefinite; for GaLore with left projection,Kt​\(G\)=Pt​Pt⊤​GK\_\{t\}\(G\)=P\_\{t\}P\_\{t\}^\{\\top\}G\. ∎

### C\.38 Proposition IV\.2

Proposition IV\.2\(the degree\-one collapse; known,[Torroba\-Hennigen et al\. \(2025\)](https://arxiv.org/html/2610.12728#bib.bib453)for GaLore \(their Thm 1 and Cor\. 3, App\. B for weight decay\) and[Hao et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib154)for random projections\)\.Letρk​\(C\)=Wk\+ℒ​C\\rho\_\{k\}\(C\)=W\_\{k\}\+\\mathcal\{L\}Cwithℒ\\mathcal\{L\}linear andC0=0C\_\{0\}=0\. Letopt\\mathrm\{opt\}be a first\-order optimizer, with no coupled or decoupled weight decay, whose state and output depend only on the sequence of gradients it is fed, and which does not read the parameter tensor: SGD, heavy\-ball or Nesterov momentum, Adam or AMSGrad, or Adafactor with scale\_parameter=False \(fixed or relative step, with its factorisation applied to the compressed gradient in the same tensor shape asCC\)\. Run both procedures below with the same implementation ofopt\\mathrm\{opt\}, on the same minibatches, and from the same optimizer state, fresh or carried over\. Then they produce the sameWW\-trajectory over a period:•trainingCCwithopt\\mathrm\{opt\}and merging at the end of the period;•the backward method withc=ℒ∗c=\\mathcal\{L\}^\{\*\}anda=ℒa=\\mathcal\{L\}, started atWkW\_\{k\}\.AdamW withλ\>0\\lambda\>0, LAMB and LARS trust ratios, parameter\-scaled Adafactor \(scale\_parameter=True, the default in HF and in galore\_torch’s GaLoreAdafactor\) and global\-norm clipping of the full gradient \(such as HF Trainer’s default max\_grad\_norm=1\.0, whenever it triggers\) all break the identity\. Injectivity ofℒ\\mathcal\{L\}is not needed\.In particular, on each matrix that GaLore projects \(its other parameters train in full in both procedures\), GaLore with a projectorPkP\_\{k\}fixed on each period equals one\-sided LoRAWk\+Pk​CW\_\{k\}\+P\_\{k\}C\(frozenB=PkB=P\_\{k\}, or frozenAAwhen projecting on the right\), merged and re\-based every period, withCC’s optimizer state \(moments and step counter\) carried over unchanged across re\-basings, with no reset\.PkP\_\{k\}must be exactly GaLore’s matrix, signs and column order included; its span alone does not suffice, because the state is reused without transport\. ReLoRA\-style resets give a different trajectory\. The identity is exact with weight decay00, no full\-gradient clipping, and an optimizer from the admissible class; galore\_torch’s GaLoreAdafactor defaults to scale\_parameter=True and is excluded\. GaLoreAdamW is the transformers\-style AdamW, which addsε\\varepsilontov\\sqrt\{v\}before the bias correction, so the LoRA side must use the same implementation\. GaLore’s scaleα\\alphamultiplies the optimizer output, so it acts as a learning\-rate factor; a LoRA\-style scale insideρ\\rhogives a different trajectory under Adam withε\>0\\varepsilon\>0\.

We use the optimizer model of §[C\.37](https://arxiv.org/html/2610.12728#A3.SS37)\. Fix a period ofTTsteps, writeLtL\_\{t\}for the loss of the minibatch used at steptt\(the same in both procedures\) andηt\\eta\_\{t\}for the step size\.

*The two procedures\.*In the first, the trainable tensor isCCwithC0=0C\_\{0\}=0and the weight isWt\(1\):=ρk​\(Ct\)=Wk\+ℒ​CtW^\{\(1\)\}\_\{t\}:=\\rho\_\{k\}\(C\_\{t\}\)=W\_\{k\}\+\\mathcal\{L\}C\_\{t\}\. By the chain rule the gradient fed toopt\\mathrm\{opt\}is

gt\(1\)=∇C\(Lt∘ρk\)\(Ct\)=ℒ∗∇Lt\(Wt\(1\)\),g^\{\(1\)\}\_\{t\}=\\nabla\_\{C\}\(L\_\{t\}\\circ\\rho\_\{k\}\)\(C\_\{t\}\)=\\mathcal\{L\}^\{\*\}\\nabla L\_\{t\}\\big\(W^\{\(1\)\}\_\{t\}\\big\),opt\\mathrm\{opt\}returnsUt\(1\)U^\{\(1\)\}\_\{t\}, andCt\+1=Ct−ηt​Ut\(1\)C\_\{t\+1\}=C\_\{t\}\-\\eta\_\{t\}U^\{\(1\)\}\_\{t\}\. In the second,W0\(2\)=WkW^\{\(2\)\}\_\{0\}=W\_\{k\}, the gradient fed toopt\\mathrm\{opt\}isgt\(2\)=c\(∇Lt\(Wt\(2\)\)\)=ℒ∗∇Lt\(Wt\(2\)\)g^\{\(2\)\}\_\{t\}=c\\big\(\\nabla L\_\{t\}\(W^\{\(2\)\}\_\{t\}\)\\big\)=\\mathcal\{L\}^\{\*\}\\nabla L\_\{t\}\\big\(W^\{\(2\)\}\_\{t\}\\big\),opt\\mathrm\{opt\}returnsUt\(2\)U^\{\(2\)\}\_\{t\}, andWt\+1\(2\)=Wt\(2\)−ηt​ℒ​Ut\(2\)W^\{\(2\)\}\_\{t\+1\}=W^\{\(2\)\}\_\{t\}\-\\eta\_\{t\}\\mathcal\{L\}U^\{\(2\)\}\_\{t\}\.

*Induction\.*We show that for0≤t≤T0\\leq t\\leq Tthe weightsWt\(1\)W^\{\(1\)\}\_\{t\}andWt\(2\)W^\{\(2\)\}\_\{t\}agree and the two optimizer states agree\. Att=0t=0,W0\(1\)=Wk\+ℒ​0=Wk=W0\(2\)W^\{\(1\)\}\_\{0\}=W\_\{k\}\+\\mathcal\{L\}0=W\_\{k\}=W^\{\(2\)\}\_\{0\}, and the states agree by hypothesis\. Suppose both claims hold attt\. The two procedures evaluate the same minibatch loss at the same weight, so they compute the same weight gradient and feedopt\\mathrm\{opt\}the same tensorgt:=gt\(1\)=gt\(2\)g\_\{t\}:=g^\{\(1\)\}\_\{t\}=g^\{\(2\)\}\_\{t\}\. Sinceopt\\mathrm\{opt\}is the same implementation and does not read the parameter tensor, its output and its next state are the same function of the state, ofgtg\_\{t\}and ofttin both procedures\. HenceUt\(1\)=Ut\(2\)=:UtU^\{\(1\)\}\_\{t\}=U^\{\(2\)\}\_\{t\}=:U\_\{t\}and the next states agree\. Then

Wt\+1\(1\)=Wk\+ℒ⁡\(Ct−ηt​Ut\)=Wt\(1\)−ηt​ℒ​Ut=Wt\+1\(2\)\.W^\{\(1\)\}\_\{t\+1\}=W\_\{k\}\+\\mathcal\{L\}\(C\_\{t\}\-\\eta\_\{t\}U\_\{t\}\)=W^\{\(1\)\}\_\{t\}\-\\eta\_\{t\}\\mathcal\{L\}U\_\{t\}=W^\{\(2\)\}\_\{t\+1\}\.At the end of the period merging sets the weight toρk​\(CT\)=WT\(1\)=WT\(2\)\\rho\_\{k\}\(C\_\{T\}\)=W^\{\(1\)\}\_\{T\}=W^\{\(2\)\}\_\{T\}\. The argument never invertsℒ\\mathcal\{L\}, so injectivity is not needed\. Every operation thatopt\\mathrm\{opt\}applies togtg\_\{t\}and its state, such as Adafactor’s factorisation in the shape ofCC\(which is the shape ofgtg\_\{t\}in both procedures\) and its update clipping, is covered, since it reads nothing else\.

*The excluded ingredients\.*Each of them reads a tensor that differs between the procedures, and in each case the first step already separates the trajectories for suitable data; it suffices to exhibit this\. Takeℒ\\mathcal\{L\}injective,Wk≠0W\_\{k\}\\neq 0and a minibatch gradientG0G\_\{0\}withℒ∗​G0≠0\\mathcal\{L\}^\{\*\}G\_\{0\}\\neq 0\.

- •*Decoupled decay\.*The first procedure setsC1=C0−η0​U0−η0​λ​C0=−η0​U0C\_\{1\}=C\_\{0\}\-\\eta\_\{0\}U\_\{0\}\-\\eta\_\{0\}\\lambda C\_\{0\}=\-\\eta\_\{0\}U\_\{0\}, soW1\(1\)=Wk−η0​ℒ​U0W^\{\(1\)\}\_\{1\}=W\_\{k\}\-\\eta\_\{0\}\\mathcal\{L\}U\_\{0\}; the second setsW1\(2\)=Wk−η0​ℒ​U0−η0​λ​WkW^\{\(2\)\}\_\{1\}=W\_\{k\}\-\\eta\_\{0\}\\mathcal\{L\}U\_\{0\}\-\\eta\_\{0\}\\lambda W\_\{k\}\. They differ byη0​λ​Wk≠0\\eta\_\{0\}\\lambda W\_\{k\}\\neq 0\. In general decay pullsWWtowardsWkW\_\{k\}in the first procedure and towards00in the second\.
- •*Coupled decay\.*The first procedure feedsℒ∗​G0\+λ​C0=ℒ∗​G0\\mathcal\{L\}^\{\*\}G\_\{0\}\+\\lambda C\_\{0\}=\\mathcal\{L\}^\{\*\}G\_\{0\}, the secondℒ∗​\(G0\+λ​Wk\)\\mathcal\{L\}^\{\*\}\(G\_\{0\}\+\\lambda W\_\{k\}\)\. These differ wheneverℒ∗​Wk≠0\\mathcal\{L\}^\{\*\}W\_\{k\}\\neq 0, and for SGD the first steps then differ, sinceℒ\\mathcal\{L\}is injective\.
- •*Trust ratios and parameter scaling\.*The first procedure reads∥C0∥=RMS⁡\(C0\)=0\\lVert C\_\{0\}\\rVert=\\mathrm\{RMS\}\(C\_\{0\}\)=0, the second∥Wk∥\>0\\lVert W\_\{k\}\\rVert\>0andRMS⁡\(Wk\)\>0\\mathrm\{RMS\}\(W\_\{k\}\)\>0\. The resulting step sizes differ for genericWkW\_\{k\}, because the first does not depend onWkW\_\{k\}and the second does\.
- •*Global\-norm clipping of the full gradient\.*The first procedure clips by the norm of its trainable gradientℒ∗​G0\\mathcal\{L\}^\{\*\}G\_\{0\}, the second \(GaLore in HF Trainer\) by the norm of the full weight gradientG0G\_\{0\}\. Under SGD with∥ℒ∗​G0∥≤τ<∥G0∥\\lVert\\mathcal\{L\}^\{\*\}G\_\{0\}\\rVert\\leq\\tau<\\lVert G\_\{0\}\\rVertthe first step is−η0​ℒ​ℒ∗​G0\-\\eta\_\{0\}\\mathcal\{L\}\\mathcal\{L\}^\{\*\}G\_\{0\}in the first procedure and−η0​\(τ/∥G0∥\)​ℒ​ℒ∗​G0\-\\eta\_\{0\}\(\\tau/\\lVert G\_\{0\}\\rVert\)\\mathcal\{L\}\\mathcal\{L\}^\{\*\}G\_\{0\}in the second; they differ becauseℒ​ℒ∗​G0≠0\\mathcal\{L\}\\mathcal\{L\}^\{\*\}G\_\{0\}\\neq 0\.

*GaLore\.*On a projected matrix with left projection, GaLore’s update in periodkkisW←W−η​α​Pk​opt​\(Pk⊤​G\)W\\leftarrow W\-\\eta\\alpha P\_\{k\}\\,\\mathrm\{opt\}\(P\_\{k\}^\{\\top\}G\)\. This is the backward method withc=Pk⊤c=P\_\{k\}^\{\\top\}anda=Pka=P\_\{k\}, and forℒ​C=Pk​C\\mathcal\{L\}C=P\_\{k\}Cwe haveℒ∗=Pk⊤\\mathcal\{L\}^\{\*\}=P\_\{k\}^\{\\top\}; the factorα\\alphamultiplies the output ofopt\\mathrm\{opt\}, so it is part of the step size\. With right projection,ℒ​C=C​Pk⊤\\mathcal\{L\}C=CP\_\{k\}^\{\\top\}andℒ∗​G=G​Pk\\mathcal\{L\}^\{\*\}G=GP\_\{k\}\. Now induct over periods\. Suppose that at the start of periodkkthe whole model, every projected and unprojected tensor, is the same in GaLore and in the re\-based one\-sided LoRA, and that so are all optimizer states\. The unprojected parameters train in full with the same optimizer in both, so they follow the same trajectory once the projected matrices do\. GaLore computesPkP\_\{k\}from the gradient at the start of the period, and the one\-sided LoRA uses that same matrix as its frozen factor\. The first part of the proposition, applied withℒ​C=Pk​C\\mathcal\{L\}C=P\_\{k\}C, gives the same trajectory through the period, and mergingPk​CTP\_\{k\}C\_\{T\}intoWkW\_\{k\}gives the sameWk\+1W\_\{k\+1\}\. GaLore keeps the state ofopt\\mathrm\{opt\}, which lives on tensors of the shape ofCC, and the one\-sided LoRA carriesCC’s state over while resettingCCto00; so the states at the start of periodk\+1k\+1agree too\. This proves the corollary\.

*The exact matrix\.*ReplacingPkP\_\{k\}byPk​OP\_\{k\}OwithO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\)keeps the span and changes the trajectory when state is carried\. Take heavy\-ball momentum with coefficientβ\>0\\beta\>0and SGD steps,Ut=mt\+1=β​mt\+gtU\_\{t\}=m\_\{t\+1\}=\\beta m\_\{t\}\+g\_\{t\}, and a carried momentummmwhose first row is nonzero, and letO=diag⁡\(−1,1,…,1\)O=\\diag\(\-1,1,\\dots,1\)\. WithPkP\_\{k\}the first step of the period movesWWby−η​Pk​\(β​m\+Pk⊤​G\)\-\\eta P\_\{k\}\(\\beta m\+P\_\{k\}^\{\\top\}G\)\. WithPk​OP\_\{k\}Othe compressed gradient isO​Pk⊤​GOP\_\{k\}^\{\\top\}G, the momentum is reused without transport, andWWmoves by−η​Pk​O​\(β​m\+O​Pk⊤​G\)=−η​Pk​\(β​O​m\+Pk⊤​G\)\-\\eta P\_\{k\}O\(\\beta m\+OP\_\{k\}^\{\\top\}G\)=\-\\eta P\_\{k\}\(\\beta Om\+P\_\{k\}^\{\\top\}G\)\. The difference is−η​β​Pk​\(O​m−m\)≠0\-\\eta\\beta P\_\{k\}\(Om\-m\)\\neq 0, sincePkP\_\{k\}is injective andO​m−mOm\-mis minus twice the first row ofmm\. The same computation applies to Adam’s first moment\.

*Resets\.*ReLoRA resets the optimizer state at every re\-basing\. For heavy\-ball momentum the first step of periodk\+1k\+1is then−η​Pk\+1​g\-\\eta P\_\{k\+1\}ginstead of−η​Pk\+1​\(β​m\+g\)\-\\eta P\_\{k\+1\}\(\\beta m\+g\), which differs whenever the carried momentummmis nonzero\.

*Implementation and scale\.*With bias correctionsb1=1−β1tb\_\{1\}=1\-\\beta\_\{1\}^\{t\}andb2=1−β2tb\_\{2\}=1\-\\beta\_\{2\}^\{t\}, transformers\-style AdamW and torch\.optim\.Adam take the steps

η​b2b1​mtvt\+εandη​mt/b1vt/b2\+ε=η​b2b1​mtvt\+ε​b2\.\\eta\\,\\frac\{\\sqrt\{b\_\{2\}\}\}\{b\_\{1\}\}\\,\\frac\{m\_\{t\}\}\{\\sqrt\{v\_\{t\}\}\+\\varepsilon\}\\qquad\\text\{and\}\\qquad\\eta\\,\\frac\{m\_\{t\}/b\_\{1\}\}\{\\sqrt\{v\_\{t\}/b\_\{2\}\}\+\\varepsilon\}=\\eta\\,\\frac\{\\sqrt\{b\_\{2\}\}\}\{b\_\{1\}\}\\,\\frac\{m\_\{t\}\}\{\\sqrt\{v\_\{t\}\}\+\\varepsilon\\sqrt\{b\_\{2\}\}\}\.Forε\>0\\varepsilon\>0the two differ, because they use different effectiveε\\varepsilon; hence the hypothesis of one implementation\. Finally, put GaLore’sα\\alphainsideρ\\rho,ρ⁡\(C\)=Wk\+α​Pk​C\\rho\(C\)=W\_\{k\}\+\\alpha P\_\{k\}C\. The compressed gradient becomesα​Pk⊤​G\\alpha P\_\{k\}^\{\\top\}G, Adam’s moments becomeα​mt\\alpha m\_\{t\}andα2​vt\\alpha^\{2\}v\_\{t\}, and the weight moves by

−η​α​Pk​α​m^tα​v^t\+ε=−η​α​Pk​m^tv^t\+ε/α,\-\\eta\\,\\alpha P\_\{k\}\\frac\{\\alpha\\hat\{m\}\_\{t\}\}\{\\alpha\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\varepsilon\}=\-\\eta\\,\\alpha P\_\{k\}\\frac\{\\hat\{m\}\_\{t\}\}\{\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\varepsilon/\\alpha\},against−ηαPkm^t/\(v^t\+ε\)\-\\eta\\alpha P\_\{k\}\\,\\hat\{m\}\_\{t\}/\(\\sqrt\{\\hat\{v\}\_\{t\}\}\+\\varepsilon\)for GaLore\. These agree forε=0\\varepsilon=0, or when the LoRA side’sε\\varepsilonis multiplied byα\\alpha, and differ forε\>0\\varepsilon\>0andα≠1\\alpha\\neq 1wheneverm^t≠0\\hat\{m\}\_\{t\}\\neq 0\. ∎

Remark\(numerical checks\)\. The values reported after[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\(7×10−167\\times 10^\{\-16\}over one period and5×10−155\\times 10^\{\-15\}over four under Adam with the state carried;8×10−178\\times 10^\{\-17\}against the official galore\_torch optimizers; the separations0\.0290\.029,0\.0200\.020,0\.0300\.030and0\.120\.12for a reset, weight decay0\.10\.1, HF\-default Adafactor and a flipped column\) instantiate the identity and the four breaking mechanisms above \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

### C\.39 Proposition IV\.3

Proposition IV\.3\(integrability; after Frobenius’ theorem\)\.LetDDbe a smooth distribution of constant rankkkon an open setU⊆ΘU\\subseteq\\Theta, for instance the anchor distributionDθ=im⁡a⁡\(θ\)D\_\{\\theta\}=\\operatorname\{im\}a\(\\theta\)of a time\-independent backward method\. ThenDDis involutive onUUif and only if through everyθ∈U\\theta\\in Upasses an injective immersionρθ:\(ℝk,0\)→\(U,θ\)\\rho\_\{\\theta\}:\(\\mathbb\{R\}^\{k\},0\)\\to\(U,\\theta\)withim⁡D​ρθ=D\\operatorname\{im\}D\\rho\_\{\\theta\}=Dalong its image\. A single suchρ\\rhothroughθ0\\theta\_\{0\}gives involutivity only alongim⁡ρ\\operatorname\{im\}\\rho\. For example,D=span\{∂x,∂y\+xz∂z\}D=\\spanop\\\{\\partial\_\{x\},\\ \\partial\_\{y\}\+xz\\,\\partial\_\{z\}\\\}onℝ3\\mathbb\{R\}^\{3\}has\[∂x,∂y\+xz∂z\]=z∂z∉D\[\\partial\_\{x\},\\partial\_\{y\}\+xz\\,\\partial\_\{z\}\]=z\\,\\partial\_\{z\}\\notin Doffz=0z=0, yet the planez=0z=0is an integral manifold\.The dynamic content needs separate hypotheses\.•In continuous time, ifDDis involutive, a backward flow withθ˙∈Dθ\\dot\{\\theta\}\\in D\_\{\\theta\}stays on the leaf throughθ0\\theta\_\{0\}\.•In the discrete update of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2), the iterates stay on a leaf for all step sizes and all optimizer outputsss\(all losses\) if and only if that leaf contains\(θ\+Dθ\)∩U\(\\theta\+D\_\{\\theta\}\)\\cap Uat every iterate, that is, if and only if the leaves are affine; an anchor constant over the period gives this \([Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\)\.•Under gradient flow, dynamic equivalence with a forward lens\(ρ,𝖱⁡\[ρ\]\)\(\\rho,\\mathsf\{R\}\[\\rho\]\)forces matching frames,at​ct=D​ρ​Λ​D​ρ⊤a\_\{t\}c\_\{t\}=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}along the trajectory\. HenceK=a​cK=acis symmetric positive semidefinite and, for injectiveaa,c=M​a⊤c=Ma^\{\\top\}withMMsymmetric positive semidefinite onSS, possibly depending onθ\\theta\. If in additionρ\\rhois an immersion withdimQ=rank⁡D\\dim Q=\\rk D, thenρ\\rhois a local isometry from\(Q,Λ−1\)\(Q,\\Lambda^\{\-1\}\)onto the leaf with metric\(a​M​a⊤\)−1\|D\(aMa^\{\\top\}\)^\{\-1\}\|\_\{D\}, so that metric must be flat\. Forward methods with gauge fibres escape this requirement:ρ⁡\(B\)=B​B⊤\\rho\(B\)=BB^\{\\top\}on symmetric matrices induces the curved Bures–Wasserstein metric, and balanced full\-rank LoRA also induces a curved leaf metric\. For discrete steps, exact agreement for every step size further forcesρ\\rhoto be affine along each horizontal lineq\+t​Λ​D​ρq⊤​gq\+t\\,\\Lambda D\\rho\_\{q\}^\{\\top\}g\.*Instances\.*Every projector\-type backward method \(GaLore with any periodTTand any projector, its random\-projector variant, LISA, MeZO\) has an anchor that does not depend onθ\\thetawithin its period\. Its distribution is constant, hence trivially involutive with affine leaves \(vacuously atT=1T=1, where a period is a single step\)\. Under the optimizer hypotheses of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)\(no weight decay of either kind, no full\-gradient clipping; GaLore and MeZO need weight decay00, and LISA’s AdamW with positive decay violates them\), these methods are rebasing schemes of linear forward methods with periodTT, includingT=1T=1, whose reach[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)describes\. Each MeZO step is a step of the linear methodθt\+λ​zt\\theta\_\{t\}\+\\lambda z\_\{t\}, up to its SPSA estimator, and GaLore withT=1T=1is one\-sided LoRA re\-based every step\. What separates these methods from LoRA\-type methods is the resampling schedule together with the carry\-over of optimizer state; integrability plays no part in it\. Flora as published \(Adafactor base, with a full\-rank current\-gradient term in the code\), APOLLO \(a low\-rank\-estimated scaling applied to the full gradient\) and Fira \(a scaled full\-rank residual\) are not of the form of[Definition IV\.1](https://arxiv.org/html/2610.12728#S4.SS0.SSSx2)with their projections as anchors, because their updates leaveim⁡a\\operatorname\{im\}a\.Non\-involutivity proper needs aθ\\theta\-dependent anchor\. Full\-batch GaLore withT=1T=1, read as the distributionDW=Hom⁡\(ℝn,Ur​\(∇L​\(W\)\)\)D\_\{W\}=\\operatorname\{Hom\}\(\\mathbb\{R\}^\{n\},U\_\{r\}\(\\nabla L\(W\)\)\)\(orHom⁡\(Vr​\(∇L​\(W\)\),ℝm\)\\operatorname\{Hom\}\(V\_\{r\}\(\\nabla L\(W\)\),\\mathbb\{R\}^\{m\}\)whenm≥nm\\geq n, as GaLore projects\) wherever a singular\-value gap exists, is involutive if and only ifUr​\(∇L​\(W\)\)U\_\{r\}\(\\nabla L\(W\)\)\(respectivelyVr​\(∇L​\(W\)\)V\_\{r\}\(\\nabla L\(W\)\)\) is invariant alongDD\. That fails generically for everyr≥1r\\geq 1once both matrix dimensions are at least22\. For everyrrwith1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)it fails forL=12​∥W∥F2L=\\tfrac\{1\}\{2\}\\lVert W\\rVert\_\{F\}^\{2\}, and numerically it fails for least squares and cross\-entropy with several samples\. It holds, for instance, for single\-sample least squares, and automatically only when the distribution is a line field \(n​r=1nr=1\)\. Minibatch GaLore withT=1T=1and MeZO violate time\-independence, because their anchors depend on step\-wise randomness\.

Throughout, a*leaf*of an involutiveDDis a maximal connected integral manifold, and the leaves form a foliation ofUU\([Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(c\)\)\.

*\(1\) The Frobenius equivalence\.*SupposeDDis involutive and letθ∈U\\theta\\in U\. By[Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(b\) there is a chartφ:V→\(−δ,δ\)dimΘ\\varphi:V\\to\(\-\\delta,\\delta\)^\{\\dim\\Theta\}centred atθ\\thetain whichDDis spanned by the firstkkcoordinate fields\. Letψ:ℝk→\(−δ,δ\)k\\psi:\\mathbb\{R\}^\{k\}\\to\(\-\\delta,\\delta\)^\{k\}be the diffeomorphismψi​\(y\)=2​δπ​arctan⁡yi\\psi\_\{i\}\(y\)=\\tfrac\{2\\delta\}\{\\pi\}\\arctan y\_\{i\}and putρθ​\(y\)=φ−1​\(ψ⁡\(y\),0\)\\rho\_\{\\theta\}\(y\)=\\varphi^\{\-1\}\(\\psi\(y\),0\)\. Thenρθ​\(0\)=θ\\rho\_\{\\theta\}\(0\)=\\theta,ρθ\\rho\_\{\\theta\}is an injective immersion intoUU, andim⁡D​ρθ​\(y\)\\operatorname\{im\}D\\rho\_\{\\theta\}\(y\)is spanned by the firstkkcoordinate fields atρθ​\(y\)\\rho\_\{\\theta\}\(y\), which isDρθ​\(y\)D\_\{\\rho\_\{\\theta\}\(y\)\}\. Conversely, suppose such immersions exist through every point, and letX,YX,Ybe sections ofDDnearθ\\theta\. An immersion is locally an embedding \([Auxiliary result B\.1](https://arxiv.org/html/2610.12728#A2.SS0.SSSx1)\(b\)\), soρθ\\rho\_\{\\theta\}maps a neighbourhood of00onto an embedded integral manifold throughθ\\theta, and[Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(a\) gives\[X,Y\]θ∈Dθ\[X,Y\]\_\{\\theta\}\\in D\_\{\\theta\}\. Sinceθ\\thetawas arbitrary,DDis involutive\.

The same application of[Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(a\) to the local pieces ofim⁡ρ\\operatorname\{im\}\\rhoshows that a single integral immersionρ\\rhoforces\[X,Y\]∈D\[X,Y\]\\in Dat the points ofim⁡ρ\\operatorname\{im\}\\rho; offim⁡ρ\\operatorname\{im\}\\rhoit says nothing, as the example shows\. In the example,X1=∂xX\_\{1\}=\\partial\_\{x\}andX2=∂y\+xz∂zX\_\{2\}=\\partial\_\{y\}\+xz\\,\\partial\_\{z\}are linearly independent everywhere, soDDhas constant rank22, and\[X1,X2\]=∂x\(xz\)∂z=z∂z\[X\_\{1\},X\_\{2\}\]=\\partial\_\{x\}\(xz\)\\,\\partial\_\{z\}=z\\,\\partial\_\{z\}\. A vector ofDDat\(x,y,z\)\(x,y,z\)has the form\(a,b,b​x​z\)\(a,b,bxz\), which equals\(0,0,z\)\(0,0,z\)only ifa=b=0a=b=0andz=0z=0; so\[X1,X2\]∉D\[X\_\{1\},X\_\{2\}\]\\notin Dwhereverz≠0z\\neq 0, andDDis involutive on no open set meeting\{z≠0\}\\\{z\\neq 0\\\}\. On the planez=0z=0,D=span\{∂x,∂y\}D=\\spanop\\\{\\partial\_\{x\},\\partial\_\{y\}\\\}is its tangent plane, soρ⁡\(u,v\)=\(u,v,0\)\\rho\(u,v\)=\(u,v,0\)is an injective immersion withim⁡D​ρ=D\\operatorname\{im\}D\\rho=Dalong its image, and there\[X1,X2\]=0∈D\[X\_\{1\},X\_\{2\}\]=0\\in D\.

*\(2\) Continuous time\.*LetDDbe involutive, letθ⁡\(t\)\\theta\(t\),t∈Jt\\in Jan interval containing00, be an absolutely continuous curve inUUwithθ⁡\(0\)=θ0\\theta\(0\)=\\theta\_\{0\}andθ˙​\(t\)∈Dθ⁡\(t\)\\dot\{\\theta\}\(t\)\\in D\_\{\\theta\(t\)\}for almost everytt, and letNNbe the leaf throughθ0\\theta\_\{0\}\. PutJN=\{t∈J:θ⁡\(t\)∈N\}J\_\{N\}=\\\{t\\in J:\\theta\(t\)\\in N\\\}, which contains00\. Fixt∗∈Jt\_\{\*\}\\in Jand a flat chart aroundθ⁡\(t∗\)\\theta\(t\_\{\*\}\)as in[Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(b\)\. Forttneart∗t\_\{\*\}the curve stays in the chart and keeps its lastdimΘ−k\\dim\\Theta\-kcoordinates constant, so it stays in the sliceΣ\\Sigmathroughθ⁡\(t∗\)\\theta\(t\_\{\*\}\)\. A slice is a connected integral manifold, and a connected integral manifold that meets a leaf lies in it \([Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(c\)\), so ifΣ\\SigmameetsNNit lies inNN\. Ift∗∈JNt\_\{\*\}\\in J\_\{N\}, the sliceΣ\\SigmameetsNNatθ⁡\(t∗\)\\theta\(t\_\{\*\}\), so a neighbourhood oft∗t\_\{\*\}lies inJNJ\_\{N\}:JNJ\_\{N\}is open\. Iftn→t∗t\_\{n\}\\to t\_\{\*\}withtn∈JNt\_\{n\}\\in J\_\{N\}, then for largennthe pointθ⁡\(tn\)\\theta\(t\_\{n\}\)lies inΣ∩N\\Sigma\\cap N, soΣ⊆N\\Sigma\\subseteq Nandt∗∈JNt\_\{\*\}\\in J\_\{N\}:JNJ\_\{N\}is closed\. HenceJN=JJ\_\{N\}=J\.

*\(3\) Discrete steps\.*The update isθ′=θ−η​a​\(θ\)​s\\theta^\{\\prime\}=\\theta\-\\eta\\,a\(\\theta\)swithη\>0\\eta\>0ands∈Ss\\in Sarbitrary\. Asη\\etaandssrange,−η​a​\(θ\)​s\-\\eta\\,a\(\\theta\)sranges over the linear spaceim⁡a⁡\(θ\)=Dθ\\operatorname\{im\}a\(\\theta\)=D\_\{\\theta\}, so the possible next iterates inUUform\(θ\+Dθ\)∩U\(\\theta\+D\_\{\\theta\}\)\\cap U\. They all lie on the leafNNthroughθ\\thetaiff\(θ\+Dθ\)∩U⊆N\(\\theta\+D\_\{\\theta\}\)\\cap U\\subseteq N; this is the first equivalence\. Call the leaves*affine*when every leafNNequals\(θ\+Dθ\)∩U\(\\theta\+D\_\{\\theta\}\)\\cap Ufor eachθ∈N\\theta\\in N\. If the containment holds at every point of every leaf, then for eachθ∈N\\theta\\in Na smallkk\-disc of the planeθ\+Dθ\\theta\+D\_\{\\theta\}aroundθ\\thetalies inNN\. Take a flat chart aroundθ\\theta\([Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(b\)\) and shrink the disc into it\. The leaves form a foliation \([Auxiliary result B\.31](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(c\)\), soNNmeets the chart in countably many slices, on each of which the lastdimΘ−k\\dim\\Theta\-kcoordinates are constant\. On the connected disc these coordinates are continuous and take countably many values, so they are constant, and the disc lies in the slice throughθ\\theta, akk\-dimensional integral manifold, in which it is open by invariance of domain\. Hence its tangent planes, all equal toDθD\_\{\\theta\}, are the planesDxD\_\{x\}at its points, soDDis locally constant alongNN, so constant on the connectedNN; every path inNNthen has velocity inDθD\_\{\\theta\}, soN⊆θ\+DθN\\subseteq\\theta\+D\_\{\\theta\}, and with the containmentN=\(θ\+Dθ\)∩UN=\(\\theta\+D\_\{\\theta\}\)\\cap U\. The converse is clear\. For an anchor that is constant over the period,Dθ=VD\_\{\\theta\}=Vis defined onU=ΘU=\\Theta, sections ofDDareVV\-valued functions whose derivatives and brackets are againVV\-valued, and the leaves are the planesθ\+V\\theta\+V; the condition holds, as[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)uses\.

*\(4\) Frames\.*A backward method under gradient flow \(opt=id\\mathrm\{opt\}=\\mathrm\{id\}\) moves the weight byθ˙=−atct∇L\\dot\{\\theta\}=\-a\_\{t\}c\_\{t\}\\nabla L; the forward method moves it byθ˙=−DρqΛDρq⊤∇L\\dot\{\\theta\}=\-D\\rho\_\{q\}\\Lambda D\\rho\_\{q\}^\{\\top\}\\nabla L\([Definition III\.1](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)\)\. We read “for every loss” as including time\-dependent losses, as minibatch training produces\. Let both methods follow the same trajectory up to timet1t\_\{1\}for a lossLL, and switch att1t\_\{1\}toL\+⟨γ,⋅⟩L\+\\langle\\gamma,\\cdot\\rangle\. The trajectories still agree up tot1t\_\{1\}, and their right derivatives att1t\_\{1\}agree, soat1​ct1​\(∇L\+γ\)=Kq⁡\(t1\)​\(∇L\+γ\)a\_\{t\_\{1\}\}c\_\{t\_\{1\}\}\(\\nabla L\+\\gamma\)=K\_\{q\(t\_\{1\}\)\}\(\\nabla L\+\\gamma\)for every covectorγ\\gamma, that is,at1​ct1=D​ρ​Λ​D​ρ⊤a\_\{t\_\{1\}\}c\_\{t\_\{1\}\}=D\\rho\\,\\Lambda\\,D\\rho^\{\\top\}atq⁡\(t1\)q\(t\_\{1\}\)\. This operator is symmetric positive semidefinite, with range inim⁡a=D\\operatorname\{im\}a=D\. Ifaais injective, leta\+=\(a⊤​a\)−1​a⊤a^\{\+\}=\(a^\{\\top\}a\)^\{\-1\}a^\{\\top\}, soP=a​a\+P=aa^\{\+\}is the orthogonal projector ontoim⁡a\\operatorname\{im\}a, and putM=a\+​\(a​c\)​\(a\+\)⊤M=a^\{\+\}\(ac\)\(a^\{\+\}\)^\{\\top\}, which is symmetric positive semidefinite\. Sincea​cacis symmetric with range inim⁡a\\operatorname\{im\}a,a​c=P⁡\(a​c\)=\(a​c\)​Pac=P\(ac\)=\(ac\)P, soa​c=P⁡\(a​c\)​P=a​M​a⊤ac=P\(ac\)P=aMa^\{\\top\}, anda⁡\(c−M​a⊤\)=0a\(c\-Ma^\{\\top\}\)=0givesc=M​a⊤c=Ma^\{\\top\}\.

Now letρ\\rhobe an immersion withdimQ=rank⁡D=k\\dim Q=\\rk D=k\. ThenD​ρq​Λ​D​ρq⊤D\\rho\_\{q\}\\Lambda D\\rho\_\{q\}^\{\\top\}has rankkkand range inDD, soim⁡D​ρq=Dρ⁡\(q\)\\operatorname\{im\}D\\rho\_\{q\}=D\_\{\\rho\(q\)\}, andρ\\rhomaps small open sets ofQQdiffeomorphically onto open pieces of leaves\. OnDDthe cometricK=a​M​a⊤K=aMa^\{\\top\}is invertible as a map fromD∗D^\{\*\}toDD, and its inverse is a metricgDg\_\{D\}onDD:gD​\(v,v\)=⟨ξ,v⟩g\_\{D\}\(v,v\)=\\langle\\xi,v\\ranglewheneverK​ξ=vK\\xi=v\. Foru∈Tq​Qu\\in T\_\{q\}Qputv=D​ρq​uv=D\\rho\_\{q\}u\. SinceD​ρqD\\rho\_\{q\}is injective,D​ρq⊤D\\rho\_\{q\}^\{\\top\}is onto, so there isξ\\xiwithD​ρq⊤​ξ=Λ−1​uD\\rho\_\{q\}^\{\\top\}\\xi=\\Lambda^\{\-1\}u, and thenK​ξ=D​ρq​Λ​Λ−1​u=vK\\xi=D\\rho\_\{q\}\\Lambda\\Lambda^\{\-1\}u=v\. Hence

gD​\(D​ρq​u,D​ρq​u\)=⟨ξ,D​ρq​u⟩=⟨D​ρq⊤​ξ,u⟩=⟨Λ−1​u,u⟩\.g\_\{D\}\(D\\rho\_\{q\}u,D\\rho\_\{q\}u\)=\\langle\\xi,D\\rho\_\{q\}u\\rangle=\\langle D\\rho\_\{q\}^\{\\top\}\\xi,u\\rangle=\\langle\\Lambda^\{\-1\}u,u\\rangle\.Soρ\\rhois a local isometry fromQQwith the constant metricΛ−1\\Lambda^\{\-1\}, which is flat, onto the leaf with metricgD=\(a​M​a⊤\)−1\|Dg\_\{D\}=\(aMa^\{\\top\}\)^\{\-1\}\|\_\{D\}, and by[Auxiliary result B\.32](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(a\) the leaf metric is flat\.

*\(5\) Gauge fibres:B​B⊤BB^\{\\top\}\.*Letρ⁡\(B\)=B​B⊤\\rho\(B\)=BB^\{\\top\}onQ=ℝn×nQ=\\mathbb\{R\}^\{n\\times n\}withΛ=I\\Lambda=I, landing inSym⁡\(n\)\\mathrm\{Sym\}\(n\), and pair covectors withSym⁡\(n\)\\mathrm\{Sym\}\(n\)by the trace\. ThenD​ρB​\(B˙\)=B˙​B⊤\+B​B˙⊤D\\rho\_\{B\}\(\\dot\{B\}\)=\\dot\{B\}B^\{\\top\}\+B\\dot\{B\}^\{\\top\}, and for symmetricGG,⟨G,B˙​B⊤\+B​B˙⊤⟩=⟨2​G​B,B˙⟩\\langle G,\\dot\{B\}B^\{\\top\}\+B\\dot\{B\}^\{\\top\}\\rangle=\\langle 2GB,\\dot\{B\}\\rangle, soD​ρB⊤​G=2​G​BD\\rho\_\{B\}^\{\\top\}G=2GBand

K⁡\(G\)=D​ρB​\(2​G​B\)=2​\(G​θ\+θ​G\),θ=B​B⊤\.K\(G\)=D\\rho\_\{B\}\(2GB\)=2\(G\\theta\+\\theta G\),\\qquad\\theta=BB^\{\\top\}\.This depends only onθ\\theta, not on the point of the fibre\{B​R:R∈O⁡\(n\)\}\\\{BR:R\\in\\mathrm\{O\}\(n\)\\\}\. On the positive definite matrices it is invertible \([Auxiliary result B\.20](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\), soD=Sym⁡\(n\)D=\\mathrm\{Sym\}\(n\)there\. Puta=K1/2a=K^\{1/2\}andc=a⊤c=a^\{\\top\}\. The backward flowθ˙=−K\(θ\)∇L\\dot\{\\theta\}=\-K\(\\theta\)\\nabla Land the forward flow pushed toθ\\thetasolve the same ordinary differential equation with smooth right\-hand side, so they agree for every loss\. The leaf metricK−1K^\{\-1\}is a constant multiple of the Bures–Wasserstein metric \([Auxiliary result B\.32](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(b\)\), which is not flat forn≥2n\\geq 2\. HeredimQ=n2\>dimSym⁡\(n\)\\dim Q=n^\{2\}\>\\dim\\mathrm\{Sym\}\(n\), soρ\\rhois no immersion, in line with \(4\)\.

*\(6\) Gauge fibres: balanced full\-rank LoRA\.*Takem=n=r≥2m=n=r\\geq 2,s=1s=1and equal block ratesηA=ηB=1\\eta\_\{A\}=\\eta\_\{B\}=1, and restrict to balanced factors,B⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\}, withW=B​AW=BAinvertible; gradient flow preserves balancedness by the Noether charge of[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)\. FromB⊤​B=A​A⊤B^\{\\top\}B=AA^\{\\top\},W⊤​W=A⊤​\(B⊤​B\)​A=\(A⊤​A\)2W^\{\\top\}W=A^\{\\top\}\(B^\{\\top\}B\)A=\(A^\{\\top\}A\)^\{2\}andW​W⊤=B⁡\(A​A⊤\)​B⊤=\(B​B⊤\)2WW^\{\\top\}=B\(AA^\{\\top\}\)B^\{\\top\}=\(BB^\{\\top\}\)^\{2\}, soA⊤​A=\|W\|:=\(W⊤​W\)1/2A^\{\\top\}A=\\lvert W\\rvert:=\(W^\{\\top\}W\)^\{1/2\}andB​B⊤=\|W⊤\|:=\(W​W⊤\)1/2BB^\{\\top\}=\\lvert W^\{\\top\}\\rvert:=\(WW^\{\\top\}\)^\{1/2\}, the unique positive definite square roots\. By[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2),

KW​\(G\)=G​\|W\|\+\|W⊤\|​G,K\_\{W\}\(G\)=G\\,\\lvert W\\rvert\+\\lvert W^\{\\top\}\\rvert\\,G,a function ofWWalone, invertible by[Auxiliary result B\.20](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3); as in \(5\) a backward method witha=K1/2a=K^\{1/2\},c=a⊤c=a^\{\\top\}reproduces the flow, and the leaf is an open subset ofGLn\\mathrm\{GL\}\_\{n\}with metricgW​\(X,X\)=⟨KW−1​X,X⟩g\_\{W\}\(X,X\)=\\langle K\_\{W\}^\{\-1\}X,X\\rangle\. The transposeτ⁡\(W\)=W⊤\\tau\(W\)=W^\{\\top\}is an isometry ofgg:KW⊤​\(G⊤\)=\(KW​G\)⊤K\_\{W^\{\\top\}\}\(G^\{\\top\}\)=\(K\_\{W\}G\)^\{\\top\}, soKW⊤−1​\(X⊤\)=\(KW−1​X\)⊤K\_\{W^\{\\top\}\}^\{\-1\}\(X^\{\\top\}\)=\(K\_\{W\}^\{\-1\}X\)^\{\\top\}andgW⊤​\(X⊤,X⊤\)=gW​\(X,X\)g\_\{W^\{\\top\}\}\(X^\{\\top\},X^\{\\top\}\)=g\_\{W\}\(X,X\)\. Its fixed points inGLn\\mathrm\{GL\}\_\{n\}are the invertible symmetric matrices, and the positive definite ones form an open subsetFFof them, an embedded submanifold on whichd​τ=idd\\tau=\\mathrm\{id\}\. Agg\-geodesicγ\\gammawithγ⁡\(0\)∈F\\gamma\(0\)\\in Fandγ˙​\(0\)\\dot\{\\gamma\}\(0\)tangent toFFis carried byτ\\tauto a geodesic with the same initial data \([Auxiliary result B\.38](https://arxiv.org/html/2610.12728#A2.SS0.SSSx7)\(a\)\), soτ∘γ=γ\\tau\\circ\\gamma=\\gammaandγ\\gammastays inFFfor small times\. Every geodesic tangent toFFstays inFF, soFFis totally geodesic, and by the Gauss equation its intrinsic sectional curvatures equal those ofggon its tangent planes \([Auxiliary result B\.38](https://arxiv.org/html/2610.12728#A2.SS0.SSSx7)\(b\)\)\. OnFF,\|W\|=\|W⊤\|=W\\lvert W\\rvert=\\lvert W^\{\\top\}\\rvert=W, and the induced metric is⟨LW−1​X,X⟩\\langle L\_\{W\}^\{\-1\}X,X\\ranglewithLW​\(G\)=G​W\+W​GL\_\{W\}\(G\)=GW\+WGonSym⁡\(n\)\\mathrm\{Sym\}\(n\), half the cometric of \(5\); this is a constant multiple of the Bures–Wasserstein metric, which has a plane of positive sectional curvature forn≥2n\\geq 2\([Auxiliary result B\.32](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(b\)\)\. Sogghas a plane of nonzero sectional curvature and is not flat\.

*\(7\) Discrete steps and horizontal lines\.*A plain GD step of the forward method fromqq, withg=∇L​\(ρ​\(q\)\)g=\\nabla L\(\\rho\(q\)\), lands atρ⁡\(q−η​v\)\\rho\(q\-\\eta v\)withv=Λ​D​ρq⊤​gv=\\Lambda D\\rho\_\{q\}^\{\\top\}g; the backward step lands atρ⁡\(q\)−η​a​c​g\\rho\(q\)\-\\eta\\,ac\\,g\. Agreement for smallη\\etaforcesa​c=D​ρq​Λ​D​ρq⊤ac=D\\rho\_\{q\}\\Lambda D\\rho\_\{q\}^\{\\top\}as in \(4\), so the backward step isρ⁡\(q\)−η​D​ρq​v\\rho\(q\)\-\\eta\\,D\\rho\_\{q\}v\. Agreement for everyη\>0\\eta\>0, and for−g\-gas well asgg\(another loss\), saysρ⁡\(q\+t​v\)=ρ⁡\(q\)\+t​D​ρq​v\\rho\(q\+tv\)=\\rho\(q\)\+t\\,D\\rho\_\{q\}vfor all realttin the domain:ρ\\rhois affine along the lineq\+t​Λ​D​ρq⊤​gq\+t\\Lambda D\\rho\_\{q\}^\{\\top\}g\.

*\(8\) Instances\.*Within a period, GaLore’s anchor is a fixed matrixPkP\_\{k\}, LISA’s a fixed coordinate inclusion, and MeZO’s the fixed vectorztz\_\{t\}for its single step; none depends onθ\\theta, so \(3\) applies onU=ΘU=\\Theta\(vacuously when a period is one step\)\. Each pair hasc=a∗c=a^\{\*\}\(§[C\.37](https://arxiv.org/html/2610.12728#A3.SS37)\), so withℒ=a\\mathcal\{L\}=ait is the backward method of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), and under that proposition’s optimizer hypotheses each period is a period of the linear forward methodθk\+ℒ​C\\theta\_\{k\}\+\\mathcal\{L\}C, merged at its end\. For MeZO,ℒ⁡\(λ\)=λ​zt\\mathcal\{L\}\(\\lambda\)=\\lambda z\_\{t\}, and the identification holds up to the error of the central difference \([Auxiliary result B\.34](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\)\. Integrability holds for all of them, so it cannot tell them apart, and by[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)what distinguishes them from LoRA is how the projector changes and which state is carried\. Flora’s update contains the full\-rank term\(1−β\)​g^t\(1\-\\beta\)\\hat\{g\}\_\{t\}, APOLLO’s isG​diag⁡\(s\)G\\,\\diag\(s\)withGGthe full gradient\([Zhu et al\., 2024a](https://arxiv.org/html/2610.12728#bib.bib596)\), and Fira’s adds a scaled multiple of\(I−Pt​Pt⊤\)​G\(I\-P\_\{t\}P\_\{t\}^\{\\top\}\)G\([Chen et al\., 2024e](https://arxiv.org/html/2610.12728#bib.bib67)\); for gradients with\(I−Pt​Pt⊤\)​G≠0\(I\-P\_\{t\}P\_\{t\}^\{\\top\}\)G\\neq 0each update leavesim⁡Pt\\operatorname\{im\}P\_\{t\}, so none is of the displayed form witha=Pta=P\_\{t\}\.

*\(9\) The GaLore field\.*LetLLbe smooth\. On the open set whereσr​\(∇L​\(W\)\)\>σr\+1​\(∇L​\(W\)\)\\sigma\_\{r\}\(\\nabla L\(W\)\)\>\\sigma\_\{r\+1\}\(\\nabla L\(W\)\), letΠ⁡\(W\)\\Pi\(W\)be the orthogonal projector ontoUr​\(∇L​\(W\)\)U\_\{r\}\(\\nabla L\(W\)\), the top\-rreigenspace ofS\(W\)=∇L\(W\)∇L\(W\)⊤S\(W\)=\\nabla L\(W\)\\nabla L\(W\)^\{\\top\}\. It is a smooth function ofWWthere, being a Riesz projector ofS⁡\(W\)S\(W\)whose spectral gap persists \([Auxiliary result B\.37](https://arxiv.org/html/2610.12728#A2.SS0.SSSx7)\)\. The fieldsXE​\(W\)=Π​\(W\)​EX\_\{E\}\(W\)=\\Pi\(W\)E,E∈ℝm×nE\\in\\mathbb\{R\}^\{m\\times n\}, spanDW=\{Z:col⁡Z⊆Ur\}D\_\{W\}=\\\{Z:\\operatorname\{col\}Z\\subseteq U\_\{r\}\\\}\. WriteΠ′​\(Z\)=D​Π​\(W\)​\[Z\]\\Pi^\{\\prime\}\(Z\)=D\\Pi\(W\)\[Z\]\. DifferentiatingΠ2=Π\\Pi^\{2\}=\\PigivesΠ′​Π\+Π​Π′=Π′\\Pi^\{\\prime\}\\Pi\+\\Pi\\Pi^\{\\prime\}=\\Pi^\{\\prime\}, henceΠ​Π′​Π=0\\Pi\\Pi^\{\\prime\}\\Pi=0and\(I−Π\)​Π′​\(I−Π\)=0\(I\-\\Pi\)\\Pi^\{\\prime\}\(I\-\\Pi\)=0\. WithN⁡\(Z\):=\(I−Π\)​Π′​\(Z\)​ΠN\(Z\):=\(I\-\\Pi\)\\Pi^\{\\prime\}\(Z\)\\Piand the symmetry ofΠ′​\(Z\)\\Pi^\{\\prime\}\(Z\),

Π′​\(Z\)=N⁡\(Z\)\+N​\(Z\)⊤,\\Pi^\{\\prime\}\(Z\)=N\(Z\)\+N\(Z\)^\{\\top\},\(26\)soΠ′​\(Z\)=0\\Pi^\{\\prime\}\(Z\)=0iffN⁡\(Z\)=0N\(Z\)=0\. The bracket is\[XE,XF\]=Π′​\(Π​E\)​F−Π′​\(Π​F\)​E\[X\_\{E\},X\_\{F\}\]=\\Pi^\{\\prime\}\(\\Pi E\)F\-\\Pi^\{\\prime\}\(\\Pi F\)E, and its component normal toDDis

\(I−Π\)​\[XE,XF\]=N⁡\(Π​E\)​Π​F−N⁡\(Π​F\)​Π​E\.\(I\-\\Pi\)\[X\_\{E\},X\_\{F\}\]=N\(\\Pi E\)\\,\\Pi F\-N\(\\Pi F\)\\,\\Pi E\.*Criterion\.*IfUrU\_\{r\}is invariant alongDD, that isΠ′​\(Z\)=0\\Pi^\{\\prime\}\(Z\)=0forZ∈DZ\\in D, the normal component vanishes andDDis involutive\. Conversely, letDDbe involutive andn≥2n\\geq 2\. Foru,u′∈Uru,u^\{\\prime\}\\in U\_\{r\}andx,y∈ℝnx,y\\in\\mathbb\{R\}^\{n\}linearly independent, takeΠ​E=u​x⊤\\Pi E=ux^\{\\top\}andΠ​F=u′​y⊤\\Pi F=u^\{\\prime\}y^\{\\top\}: involutivity givesN⁡\(u​x⊤\)​u′​y⊤=N⁡\(u′​y⊤\)​u​x⊤N\(ux^\{\\top\}\)u^\{\\prime\}\\,y^\{\\top\}=N\(u^\{\\prime\}y^\{\\top\}\)u\\,x^\{\\top\}\. The left side has its rows inspan⁡\{y\}\\spanop\\\{y\\\}and the right side inspan⁡\{x\}\\spanop\\\{x\\\}, so both vanish, andN⁡\(u​x⊤\)​u′=0N\(ux^\{\\top\}\)u^\{\\prime\}=0\. Asu′u^\{\\prime\}ranges overUrU\_\{r\}andN⁡\(Z\)N\(Z\)vanishes onUr⟂U\_\{r\}^\{\\perp\},N⁡\(u​x⊤\)=0N\(ux^\{\\top\}\)=0for allu∈Uru\\in U\_\{r\}andx∈ℝnx\\in\\mathbb\{R\}^\{n\}\(givenxx, pickyyindependent of it\), soN=0N=0onDDby linearity, andUrU\_\{r\}is invariant alongDD\. For right projection apply this to the transpose\. The failure proved next covers every1≤r<min⁡\(m,n\)1\\leq r<\\min\(m,n\)withm,n≥2m,n\\geq 2; the remaining case with a gap isn=r=1n=r=1, whereDW=U1D\_\{W\}=U\_\{1\}is a line field, and a line field is always involutive, since\[f​X,g​X\]=\(f​X​g−g​X​f\)​X\[fX,gX\]=\(f\\,Xg\-g\\,Xf\)X\.

*Failure forL=12​∥W∥F2L=\\tfrac\{1\}\{2\}\\lVert W\\rVert\_\{F\}^\{2\}\.*Here∇L​\(W\)=W\\nabla L\(W\)=W\. Letp=min⁡\(m,n\)≥2p=\\min\(m,n\)\\geq 2,1≤r<p1\\leq r<p, andW=∑i≤pσi​ei​ei⊤W=\\sum\_\{i\\leq p\}\\sigma\_\{i\}e\_\{i\}e\_\{i\}^\{\\top\}withσ1\>⋯\>σp\>0\\sigma\_\{1\}\>\\dots\>\\sigma\_\{p\}\>0, soUr=span⁡\{e1,…,er\}U\_\{r\}=\\spanop\\\{e\_\{1\},\\dots,e\_\{r\}\\\}andS=W​W⊤S=WW^\{\\top\}is diagonal with entriesλi=σi2\\lambda\_\{i\}=\\sigma\_\{i\}^\{2\}\(i≤pi\\leq p\) and00\. For a smooth symmetric familyS⁡\(t\)S\(t\)withS⁡\(0\)=SS\(0\)=S, differentiating\[S⁡\(t\),Π⁡\(t\)\]=0\[S\(t\),\\Pi\(t\)\]=0gives\[S˙,Π\]\+\[S,Π˙\]=0\[\\dot\{S\},\\Pi\]\+\[S,\\dot\{\\Pi\}\]=0; the\(i,j\)\(i,j\)entry withi≤r<ji\\leq r<jreads−S˙i​j\+\(λi−λj\)​Π˙i​j=0\-\\dot\{S\}\_\{ij\}\+\(\\lambda\_\{i\}\-\\lambda\_\{j\}\)\\dot\{\\Pi\}\_\{ij\}=0, andΠ˙\\dot\{\\Pi\}vanishes on the diagonal blocks by the identities above\. SoΠ˙i​j=S˙i​j/\(λi−λj\)\\dot\{\\Pi\}\_\{ij\}=\\dot\{S\}\_\{ij\}/\(\\lambda\_\{i\}\-\\lambda\_\{j\}\)fori≤r<ji\\leq r<j\. TakeZ=e1​x⊤∈DZ=e\_\{1\}x^\{\\top\}\\in D, soS˙=Z​W⊤\+W​Z⊤\\dot\{S\}=ZW^\{\\top\}\+WZ^\{\\top\}\. ThenS˙1​j=x⊤​W⊤​ej=σj​xj\\dot\{S\}\_\{1j\}=x^\{\\top\}W^\{\\top\}e\_\{j\}=\\sigma\_\{j\}x\_\{j\}forr<j≤pr<j\\leq p,S˙1​j=0\\dot\{S\}\_\{1j\}=0forj\>pj\>p, andS˙i​j=0\\dot\{S\}\_\{ij\}=0for2≤i≤r<j2\\leq i\\leq r<j\. Hence

N⁡\(e1​x⊤\)=∑j=r\+1pσj​xjσ12−σj2​ej​e1⊤\.N\(e\_\{1\}x^\{\\top\}\)=\\sum\_\{j=r\+1\}^\{p\}\\frac\{\\sigma\_\{j\}x\_\{j\}\}\{\\sigma\_\{1\}^\{2\}\-\\sigma\_\{j\}^\{2\}\}\\,e\_\{j\}e\_\{1\}^\{\\top\}\.WithE=e1​er\+1⊤E=e\_\{1\}e\_\{r\+1\}^\{\\top\}andF=e1​e1⊤F=e\_\{1\}e\_\{1\}^\{\\top\}we getN⁡\(Π​F\)=0N\(\\Pi F\)=0and

\(I−Π\)​\[XE,XF\]=N⁡\(e1​er\+1⊤\)​e1​e1⊤=σr\+1σ12−σr\+12​er\+1​e1⊤≠0,\(I\-\\Pi\)\[X\_\{E\},X\_\{F\}\]=N\(e\_\{1\}e\_\{r\+1\}^\{\\top\}\)\\,e\_\{1\}e\_\{1\}^\{\\top\}=\\frac\{\\sigma\_\{r\+1\}\}\{\\sigma\_\{1\}^\{2\}\-\\sigma\_\{r\+1\}^\{2\}\}\\,e\_\{r\+1\}e\_\{1\}^\{\\top\}\\neq 0,soDDis not involutive atWW\. Atm=n=2m=n=2,r=1r=1andW=diag⁡\(2,1\)W=\\diag\(2,1\), the frameXi=Xe1​ei⊤X\_\{i\}=X\_\{e\_\{1\}e\_\{i\}^\{\\top\}\}gives the full bracket\[X1,X2\]=Π′​\(e1​e1⊤\)​e1​e2⊤−Π′​\(e1​e2⊤\)​e1​e1⊤=0−13​\(e1​e2⊤\+e2​e1⊤\)​e1​e1⊤=−13​e2​e1⊤\[X\_\{1\},X\_\{2\}\]=\\Pi^\{\\prime\}\(e\_\{1\}e\_\{1\}^\{\\top\}\)e\_\{1\}e\_\{2\}^\{\\top\}\-\\Pi^\{\\prime\}\(e\_\{1\}e\_\{2\}^\{\\top\}\)e\_\{1\}e\_\{1\}^\{\\top\}=0\-\\tfrac\{1\}\{3\}\(e\_\{1\}e\_\{2\}^\{\\top\}\+e\_\{2\}e\_\{1\}^\{\\top\}\)e\_\{1\}e\_\{1\}^\{\\top\}=\-\\tfrac\{1\}\{3\}e\_\{2\}e\_\{1\}^\{\\top\}; the frameXi=u⁡\(W\)​ei⊤X\_\{i\}=u\(W\)e\_\{i\}^\{\\top\}, withuuthe top left singular vector, gives the same value, sinceD​u​\[e1​e1⊤\]=0Du\[e\_\{1\}e\_\{1\}^\{\\top\}\]=0andD​u​\[e1​e2⊤\]=13​e2Du\[e\_\{1\}e\_\{2\}^\{\\top\}\]=\\tfrac\{1\}\{3\}e\_\{2\}at thisWW\.

*Genericity\.*For least squaresL⁡\(W\)=12​∥W​X−Y∥F2L\(W\)=\\tfrac\{1\}\{2\}\\lVert WX\-Y\\rVert\_\{F\}^\{2\}the normal component of the bracket of two fixed frame fields is a real\-analytic function of\(W,X,Y\)\(W,X,Y\)on the open set where the gap exists\. It does not vanish identically on the connected component containing\(W,In,0\)\(W,I\_\{n\},0\)withWWas above, where it is the value just computed, so it is nonzero on an open dense subset of that component \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. The values for cross\-entropy and for several samples are the numerical checks reported with the proposition\.

*Single\-sample least squares\.*With one sample,∇L​\(W\)=\(W​x−y\)​x⊤\\nabla L\(W\)=\(Wx\-y\)x^\{\\top\}has rank one and, whereW​x≠yWx\\neq y,U1=span⁡\{W​x−y\}U\_\{1\}=\\spanop\\\{Wx\-y\\\}\. MovingWWalongZ=u​v⊤Z=uv^\{\\top\}withu∈U1u\\in U\_\{1\}changesW​x−yWx\-ybyu⁡\(v⊤​x\)∈U1u\(v^\{\\top\}x\)\\in U\_\{1\}, soU1U\_\{1\}is constant alongDD, andDDis involutive by the criterion\.

*Time\-dependence\.*Minibatch GaLore withT=1T=1draws a new minibatch, and MeZO a newztz\_\{t\}, at every step, so their anchors are functions of\(t,θ\)\(t,\\theta\)and not ofθ\\thetaalone\. ∎

### C\.40 Observation IV\.4

Observation IV\.4\(backward reach and the backprop cone; an upper bound\)\.Assume reverse mode in which each box keeps its own residual, with no checkpointing across boxes, and take boxes at the granularity where adapters are separate boxes \(for LoRA\-FA,A0A\_\{0\}andBBseparately\)\. A*VJP residual*is an input, an output, a sufficient statistic such as a ReLU mask, or a recipe to recompute one of these inside the box\. Then it suffices to keep a VJP residual for•each trainable box whose parameter\-VJP depends on its input: linear and multiplicative boxes such asWW,AA,BBand \(IA\)3scales; additive boxes such as biases and prompts need nothing;•each*nonlinear*box, bilinear boxes such as the attention products included, on a directed path from a trainable box to the loss\.Deterministic frozen linear boxes, and all boxes off this cone, need nothing; stochastic linear boxes on the cone \(dropout\) need their mask or random\-number state\. Alternatively, thekkcoordinates ofP⊤∇LP^\{\\top\}\\nabla Lcan be computed exactly bykkforward\-mode JVPs, with no reverse pass andO⁡\(1\)O\(1\)\-layer activation memory, each JVP costing a small constant multiple \(about1\.51\.5to33\) of a forward pass\. This always uses less activation memory than un\-checkpointed reverse mode, but it is time\-competitive with one VJP \(about33forward passes\) only forkkof about11or22\. It also needsPPbefore the passes, so it cannot produce GaLore’s SVD refresh, wherePPdepends on∇L\\nabla L, more cheaply than a backward pass \(∇L\\nabla Litself would take\|θ\|\\lvert\\theta\\rvertJVPs\)\.

*The computation\.*The network is a directed acyclic graph of boxes\. A box receives inputsxx\(activations\), may carry parameterspp, and produces an outputy=f⁡\(x,p\)y=f\(x,p\); the loss is the output of the last box\. Reverse mode propagates cotangents backwards\. For a box with output cotangenty¯\\bar\{y\}it computes the input cotangentsx¯=Jx​f​\(x,p\)⊤​y¯\\bar\{x\}=J\_\{x\}f\(x,p\)^\{\\top\}\\bar\{y\}and, for a trainable box, the parameter cotangentp¯=Jp​f​\(x,p\)⊤​y¯\\bar\{p\}=J\_\{p\}f\(x,p\)^\{\\top\}\\bar\{y\}\([Background B\.17](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx3)\)\. Call the*cone*the set of boxes on a directed path from a trainable box to the loss\.

*Pruning\.*The parameter gradients are sums over paths from the parameters to the loss of products of Jacobians, so only cotangents of wires on such paths contribute\. A wire whose value does not depend on any trainable parameter carries a cotangent that is never multiplied into a parameter gradient, so its cotangent need not be formed\. Hence it suffices to computey¯\\bar\{y\}for the outputs of boxes on the cone,x¯\\bar\{x\}for those inputs of cone boxes that are produced on the cone, andp¯\\bar\{p\}for the trainable boxes\. Boxes off the cone take no part in the backward sweep and need no residual\.

*What each box on the cone needs\.*We list, for each kind of box, what the backward sweep reads besidesy¯\\bar\{y\}and the weights, which are stored anyway\.

- •A deterministic frozen linear or affine box,y=W​x\+by=Wx\+b:x¯=W⊤​y¯\\bar\{x\}=W^\{\\top\}\\bar\{y\}does not involvexx, so no residual\.
- •A dropout box,y=𝗆⊙x/\(1−p\)y=\\mathsf\{m\}\\odot x/\(1\-p\):x¯=𝗆⊙y¯/\(1−p\)\\bar\{x\}=\\mathsf\{m\}\\odot\\bar\{y\}/\(1\-p\), so it needs its mask, or the random state that regenerates it\.
- •A trainable linear box,y=W​xy=Wx:W¯=y¯​x⊤\\bar\{W\}=\\bar\{y\}\\,x^\{\\top\}, summed over tokens, so it needs its inputxx;x¯=W⊤​y¯\\bar\{x\}=W^\{\\top\}\\bar\{y\}needs nothing more\. A trainable multiplicative box,y=ℓ⊙xy=\\ell\\odot x:ℓ¯=∑y¯⊙x\\bar\{\\ell\}=\\sum\\bar\{y\}\\odot x, so it needsxx, andx¯=ℓ⊙y¯\\bar\{x\}=\\ell\\odot\\bar\{y\}\.
- •A trainable additive box,y=x\+by=x\+b, or a prompt,y=\[P;x\]y=\[P;x\]:b¯=∑y¯\\bar\{b\}=\\sum\\bar\{y\}andP¯\\bar\{P\}is the slice ofy¯\\bar\{y\}at the prompt positions, andx¯\\bar\{x\}isy¯\\bar\{y\}or its slice; no residual\.
- •A nonlinear box,y=f⁡\(x\)y=f\(x\), possibly with frozen parameters:x¯=Jf​\(x\)⊤​y¯\\bar\{x\}=J\_\{f\}\(x\)^\{\\top\}\\bar\{y\}depends onxxthroughJf​\(x\)J\_\{f\}\(x\), so it needs something from whichJf​\(x\)⊤​y¯J\_\{f\}\(x\)^\{\\top\}\\bar\{y\}can be formed: the input, the output \(for tanh, sigmoid or softmax\), a mask \(for ReLU\), or a recipe to recompute one of them\. A bilinear box is nonlinear in its joint input; fory=a⊙by=a\\odot b,a¯=y¯⊙b\\bar\{a\}=\\bar\{y\}\\odot bandb¯=y¯⊙a\\bar\{b\}=\\bar\{y\}\\odot a, and for the attention productsQ​K⊤QK^\{\\top\}andP​VPVthe same holds with matrix products\.

A trainable nonlinear box lies on the cone, being on a path from itself to the loss, so it is covered by the second bullet, and also by the first if its parameter\-VJP reads its input\. Processing the cone in reverse topological order, each box hasy¯\\bar\{y\}available \(it is a sum of input cotangents of its readers on the cone\) together with its residual, so the sweep computes every parameter gradient exactly\. This proves that the residuals listed suffice\. For LoRA\-FA,A0A\_\{0\}is a frozen linear box and needs nothing, whileBBis a trainable linear box whose input isA0​xA\_\{0\}x, soBBkeepsA0​xA\_\{0\}x\.

*The forward\-mode route\.*Theii\-th coordinate ofP⊤∇LP^\{\\top\}\\nabla Lis⟨pi,∇L​\(θ\)⟩=D​L​\(θ\)​\[pi\]\\langle p\_\{i\},\\nabla L\(\\theta\)\\rangle=DL\(\\theta\)\[p\_\{i\}\], the directional derivative along theii\-th column ofPP\. By[Auxiliary result B\.33](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(a\), forward mode computes it alongside one evaluation ofLLat a small constant multiple of that cost, carrying each tangent with its value and discarding both once consumed\. In a layered network this keeps the values and tangents ofO⁡\(1\)O\(1\)layers alive at a time, whereas un\-checkpointed reverse mode keeps the residuals of every box on the cone until the backward sweep reaches it\. Thekkcoordinates costkksuch passes\. One reverse pass costs about three forward passes, so with a per\-JVP cost of1\.51\.5to33forward passes the forward route is no slower only when1\.5​k1\.5kto3​k3kis at most about33, that is, forkkof about11or22\. The directionspip\_\{i\}must be fixed before the passes\. GaLore’s refresh computesPPfrom the SVD of the full gradient∇L\\nabla Lof each projected matrix, and∇L\\nabla Lby forward mode costs one pass per coordinate,\|θ\|\\lvert\\theta\\rvertpasses, against one reverse pass \([Auxiliary result B\.33](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\(b\)\)\.

*MeZO\.*The central difference estimatesD​L​\(θ\)​\[z\]DL\(\\theta\)\[z\]with errorO⁡\(ε2\)O\(\\varepsilon^\{2\}\)whenLLhas a Lipschitz Hessian, and is unbiased for the gradient of the Gaussian smoothing ofLL\([Auxiliary result B\.34](https://arxiv.org/html/2610.12728#A2.SS0.SSSx5)\)\. ∎

### C\.41 Definition V\.1

Definition V\.1\(base change and rebasing\)\.\(a\)A*function\-preserving*pointed reparametrisationϕ:\(Θ′,f′,θ0′\)→\(Θ,f,θ0\)\\phi:\(\\Theta^\{\\prime\},f^\{\\prime\},\\theta\_\{0\}^\{\\prime\}\)\\to\(\\Theta,f,\\theta\_\{0\}\), withf∘\(ϕ×idX\)=f′f\\circ\(\\phi\\times\\mathrm\{id\}\_\{X\}\)=f^\{\\prime\}, induces by postcompositionϕ\!:𝐌𝐞𝐭𝐡\(Θ′,θ0′\)→𝐌𝐞𝐭𝐡\(Θ,θ0\),\(Q,q0,ρ\)↦\(Q,q0,ϕ∘ρ\)\.\\phi\_\{\!\}:\\mathbf\{Meth\}\(\\Theta^\{\\prime\},\\theta\_\{0\}^\{\\prime\}\)\\to\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\),\\qquad\(Q,q\_\{0\},\\rho\)\\mapsto\(Q,q\_\{0\},\\phi\\circ\\rho\)\.\(b\)A mapκ:Θ→Θ\\kappa:\\Theta\\to\\Thetathat does*not*preserve the function, such as a quantiser, moves the base point\. Itspointing defectise=κ​θ0−θ0e=\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\.\(c\)Arebasing scheme\(merge\-and\-restart\) picksθk\+1∈Im⁡M⁡\(θk\)\\theta\_\{k\+1\}\\in\\Imop M\(\\theta\_\{k\}\)\. Its reachable sets areRKR\_\{K\}, the pointsθK\\theta\_\{K\}it can reach inKKcycles, andR∞=⋃KRKR\_\{\\infty\}=\\bigcup\_\{K\}R\_\{K\}\.

A definition needs no proof\. We check thatϕ\!\\phi\_\{\!\}is a functor that preserves the realised functions, and record the recursion for the reachable sets that the proofs of[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)use\.

\(a\) Sinceϕ⁡\(θ0′\)=θ0\\phi\(\\theta\_\{0\}^\{\\prime\}\)=\\theta\_\{0\}, the mapϕ∘ρ\\phi\\circ\\rhosendsq0q\_\{0\}toθ0\\theta\_\{0\}, soϕ\!M\\phi\_\{\!\}Mis an object of𝐌𝐞𝐭𝐡⁡\(Θ,θ0\)\\mathbf\{Meth\}\(\\Theta,\\theta\_\{0\}\)\. A morphismh:M→Nh:M\\to Nis a pointed smooth map withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\); then\(ϕ∘ρN\)∘h=ϕ∘ρM\(\\phi\\circ\\rho\_\{N\}\)\\circ h=\\phi\\circ\\rho\_\{M\}, so the samehhis a morphismϕ\!M→ϕ\!N\\phi\_\{\!\}M\\to\\phi\_\{\!\}N\. Settingϕ\!h=h\\phi\_\{\!\}h=hpreserves identities and composition, soϕ\!\\phi\_\{\!\}is a functor, the postcomposition functor of slice categories \([Background A\.19](https://arxiv.org/html/2610.12728#A1.SSx1.SSSx3)\)\. It changes neither the parameter space nor the realised functions: forq∈Qq\\in Q,

f⁡\(ϕ⁡\(ρ⁡\(q\)\),x\)=f′​\(ρ⁡\(q\),x\)for all​x∈X,f\\big\(\\phi\(\\rho\(q\)\),x\\big\)=f^\{\\prime\}\\big\(\\rho\(q\),x\\big\)\\qquad\\text\{for all \}x\\in X,soϕ\!M\\phi\_\{\!\}Mcomputes exactly the functionsMMcomputes, andImϕ\!M=ϕ\(ImM\)\\Imop\\phi\_\{\!\}M=\\phi\(\\Imop M\)\.

\(b\) is a definition\. \(c\) PutR0=\{θ0\}R\_\{0\}=\\\{\\theta\_\{0\}\\\}\. Each cycle trains the methodM⁡\(θk\)M\(\\theta\_\{k\}\)over the current base and merges, so the points reachable inK\+1K\+1cycles are

RK\+1=⋃θ∈RKIm⁡M⁡\(θ\)\.R\_\{K\+1\}=\\bigcup\_\{\\theta\\in R\_\{K\}\}\\Imop M\(\\theta\)\.When everyM⁡\(θ\)M\(\\theta\)is pointed atθ\\theta, that isρθ​\(q0\)=θ\\rho\_\{\\theta\}\(q\_\{0\}\)=\\theta, we haveθ∈Im⁡M⁡\(θ\)\\theta\\in\\Imop M\(\\theta\), soRK⊆RK\+1R\_\{K\}\\subseteq R\_\{K\+1\}andR∞R\_\{\\infty\}is an increasing union\. ∎

### C\.42 Proposition V\.2

Proposition V\.2\(quantisation and splitting\)\.WriteWWfor the full\-precision weight,κ\\kappafor the quantiser,e=κ​W−We=\\kappa W\-W,δ0=δ⁡\(q0\)\\delta\_\{0\}=\\delta\(q\_\{0\}\), andssfor the LoRA scaleα/r\\alpha/r\(α/r\\alpha/\\sqrt\{r\}under rsLoRA\)\.\(a\) Additive families commute with translation,M⁡\(θ\+v\)=τv∘M⁡\(θ\)M\(\\theta\+v\)=\\tau\_\{v\}\\circ M\(\\theta\)\. Hence QLoRA is LoRA pointed atκ​θ0\\kappa\\theta\_\{0\}, with defectee, and it needs no new weight\-space structure\. Since−e\-eis generically of full rank,θ0∉Im⁡LoRAr​\(κ​θ0\)\\theta\_\{0\}\\notin\\Imop\\mathrm\{LoRA\}\_\{r\}\(\\kappa\\theta\_\{0\}\)unlessrank⁡e≤r\\rk e\\leq r\.\(b\) For splitting families derived from an SVD or from data, the order of operations matters, and three orders occur in practice\.•*Split, then quantise the residual*\(QPiSSA; HF’s recommended PiSSA and CorDA workflow; the quantisation half\-step of LoftQ fort≥2t\\geq 2, withWres=W−δ⁡\(qt−1\)W\_\{\\rm res\}=W\-\\delta\(q\_\{t\-1\}\)\)\. The defect isκ⁡\(Wres\)−Wres\\kappa\(W\_\{\\rm res\}\)\-W\_\{\\rm res\}\.•*Quantise, then fit the error*\(LoftQ with one iteration, HF’s default loftq\_iter=1; HF replace\_lora\_weights\_loftq\)\. HF sets the factors fromSVDr⁡\(W−κ​W\)\\operatorname\{SVD\}\_\{r\}\(W\-\\kappa W\)without dividing byss, unlike its PiSSA and OLoRA initialisations, soδ0=−s​SVDr⁡\(e\)\\delta\_\{0\}=\-s\\operatorname\{SVD\}\_\{r\}\(e\)and the defect ise−s​SVDr​\(e\)e\-s\\operatorname\{SVD\}\_\{r\}\(e\), with∥e−s​SVDr⁡\(e\)∥2=∥e−SVDr⁡\(e\)∥2\+\(1−s\)2​∥SVDr⁡\(e\)∥2\.\\lVert e\-s\\operatorname\{SVD\}\_\{r\}\(e\)\\rVert^\{2\}=\\lVert e\-\\operatorname\{SVD\}\_\{r\}\(e\)\\rVert^\{2\}\+\(1\-s\)^\{2\}\\lVert\\operatorname\{SVD\}\_\{r\}\(e\)\\rVert^\{2\}\.The defect iseeminus its best rank\-rrapproximation \(of rankrank⁡e−r\\rk e\-r\) iffs=1s=1, which is HF’s bare defaultr=α=8r=\\alpha=8\. Fors=2s=2its norm equals∥e∥\\lVert e\\rVert, no better than QLoRA\. With a callback, replace\_lora\_weights\_loftq can roll a module back to the zero initialisation, whose defect isee\.•*Quantise, split, re\-quantise*\(HF olora\_init on bitsandbytes weights\)\. The defect ise\+κ⁡\(r′\)−r′e\+\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}withr′=κ​W−δ0r^\{\\prime\}=\\kappa W\-\\delta\_\{0\}\.Quantise\-then\-split has defect exactlyeeiff the frozen part equalsκ​W−δ0\\kappa W\-\\delta\_\{0\}, for instanceκ​W\\kappa Wkept quantised minus a frozen copy of the rank\-rrδ0\\delta\_\{0\}, at a cost of aboutr⁡\(m\+n\)r\(m\+n\)extra numbers\. Re\-quantising the residual, as HF olora\_init does, generically addsκ⁡\(r′\)−r′\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}\.Action families whose pointing does not depend on the base \(OFT atR=IR=I, BOFT atR=IR=Iwith unit scale, HiRA atB=0B=0\), evaluated atκ​θ0\\kappa\\theta\_\{0\}, simply carry the defectee\. Methods whose pointing is computed from the base have an order ambiguity of their own\. DoRA’s magnitude initialised fromκ​θ0\\kappa\\theta\_\{0\}gives defectee, while initialised fromθ0\\theta\_\{0\}it givesdiag⁡\(∥w0,i∥/∥\(κ​θ0\)i∥\)​κ​θ0−θ0\\diag\\big\(\\lVert w\_\{0,i\}\\rVert/\\lVert\(\\kappa\\theta\_\{0\}\)\_\{i\}\\rVert\\big\)\\,\\kappa\\theta\_\{0\}\-\\theta\_\{0\}\. For action families a base change also moves the absolute invariants, along with the base point\. NF4’s exact zero code enlarges the zero set that HiRA must preserve, and exactly orthogonal \(exact\-Cayley\) OFT preserves the Gram matrix ofκ​θ0\\kappa\\theta\_\{0\}rather than that ofθ0\\theta\_\{0\}; HF’s default Cayley–Neumann OFT preserves it only approximately\.\(c\) LoftQ alternates two updates on∥θ0−Q−L​R∥\\lVert\\theta\_\{0\}\-Q\-LR\\rVert, an exact rank\-rrSVD step and a round\-to\-nearest quantisation step with blockwise scales recomputed from its input\. The second is not an exact projection, so the objective need not decrease monotonically\. Withs=1s=1the output is pointed up to the attained residual\. It is then an*approximate section*, a choice\(Q,q0\)\(Q,q\_\{0\}\)with reduced∥ρ⁡\(q0\)−θ0∥\\lVert\\rho\(q\_\{0\}\)\-\\theta\_\{0\}\\rVert\. In general, withQTQ\_\{T\}and the low\-rank fitLT​RTL\_\{T\}R\_\{T\}afterTTiterations, HF’s deployed defect is\(QT\+LT​RT−W\)\+\(s−1\)​LT​RT\(Q\_\{T\}\+L\_\{T\}R\_\{T\}\-W\)\+\(s\-1\)L\_\{T\}R\_\{T\}, because HF’s internal residual also omitsss\. The official LoftQ checkpoints user=64r=64andα=16\\alpha=16, sos=1/4s=1/4, and then the deployed defect can grow withTTwhile the attained residual falls\.\(d\) For a generic high\-precision adapterΔ\\Delta, fine\-tune\-then\-quantise and quantise\-then\-fine\-tune do not commute\. The first lands on the quantisation grid\. In the second, the exactly merged weightκ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Deltaleaves the quantised type\. For a uniform fixed\-scale INT\-kkgrid,κ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Deltastays on the grid iffΔ\\Deltalies on the scale lattice \(within range\), so there the obstruction is the off\-grid update\. NF4 is non\-uniform and not closed under addition, and absmax grids whose scales are recomputed are not closed either, so for QLoRA’s NF4 the grid itself is part of the obstruction\. Type\-preserving merges, such as HF PEFT’s dequantise–add–requantise for bitsandbytes 4\- and 8\-bit layers, re\-quantise with recomputed block scales, so they stay in the quantised type but are lossy\. Under affine group quantisation QA\-LoRA is the structured exception\. Its group\-pooled adapter is constant on each quantisation group and merges into the zero\-points, so the merged model stays INT\-kk\.

*Quantisers\.*We use two properties of the blockwise quantisers in practice \(NF4 and INT\-kk, with a fixed scale or the absmax scale of each block,[Dettmers et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib92);[Dettmers et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib91)\)\. First,κ\\kapparounds each entry, divided by its block’s scale, to the nearest point of a finite codebook and multiplies back, so its values, the*representable*matrices, form a Lebesgue\-null set: a finite set for fixed scales, and for absmax scales a finite union of sets with one free scale per block ofβ≥2\\beta\\geq 2entries\. Second,κ\\kappais idempotent,κ⁡\(κ​X\)=κ​X\\kappa\(\\kappa X\)=\\kappa X, and it fixes exactly the representable matrices\.

\(a\) An additive family hasρθ​\(q\)=θ\+δ⁡\(q\)\\rho\_\{\\theta\}\(q\)=\\theta\+\\delta\(q\)withδ\\deltaindependent of the base andδ⁡\(q0\)=0\\delta\(q\_\{0\}\)=0\([Definition I\.2](https://arxiv.org/html/2610.12728#S1.SS0.SSSx1)\)\. Thenρθ\+v​\(q\)=v\+ρθ​\(q\)=τv​\(ρθ​\(q\)\)\\rho\_\{\\theta\+v\}\(q\)=v\+\\rho\_\{\\theta\}\(q\)=\\tau\_\{v\}\(\\rho\_\{\\theta\}\(q\)\), which is the first claim\. QLoRA trains zero\-initialised LoRA over the frozen baseκ​θ0\\kappa\\theta\_\{0\}, so it isLoRAr​\(κ​θ0\)\\mathrm\{LoRA\}\_\{r\}\(\\kappa\\theta\_\{0\}\), withρ⁡\(q0\)=κ​θ0\\rho\(q\_\{0\}\)=\\kappa\\theta\_\{0\}and defectρ⁡\(q0\)−θ0=e\\rho\(q\_\{0\}\)\-\\theta\_\{0\}=e\. Since every matrix of rank at mostrrhas the forms​B​AsBA\(s≠0s\\neq 0\),θ0=κ​θ0\+s​B​A\\theta\_\{0\}=\\kappa\\theta\_\{0\}\+sBAfor some\(B,A\)\(B,A\)iffrank⁡\(−e\)≤r\\rk\(\-e\)\\leq r\.

*The word “generically”\.*For a fixed\-scale quantiser,κ\\kappais constant on each open cell of a decomposition ofΘ\\Thetainto countably many cells and a null set of boundaries, so on a celle=K−We=K\-WwithKKconstant\. The rank\-deficient matrices are the common zeros of the maximal minors, a proper algebraic subset, andW↦K−WW\\mapsto K\-Wis a bijection, soeehas full rank for almost everyWWin each cell, hence for almost everyWW\([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. For an absmax quantiser with blocks ofβ≥2\\beta\\geq 2consecutive entries of a row, take the cells on which every code and the position of every block maximum are constant\. With the scale stored exactly, the block maximum is represented exactly, soeevanishes at those positions, and each other entry isei​j=a​ci​j−Wi​je\_\{ij\}=a\\,c\_\{ij\}\-W\_\{ij\}withaathe scale of its block andci​jc\_\{ij\}its code; varyingWi​jW\_\{ij\}alone movesei​je\_\{ij\}alone\. So on each celleefills an open subset of the spaceZZof matrices vanishing at the block maxima, and its rank is, for almost everyWWin the cell, the largest rank inZZ\. A row of lengthnnhas at most⌈n/β⌉\+1\\lceil n/\\beta\\rceil\+1forced zeros, soZZcontains a matrix of rankr\+1r\+1wheneverr\+1≤n−⌈n/β⌉−1r\+1\\leq n\-\\lceil n/\\beta\\rceil\-1andr<mr<m\(placer\+1r\+1nonzero entries in distinct rows and columns, avoiding the forced zeros, greedily\)\. Hencerank⁡e\>r\\rk e\>r, andθ0∉Im⁡LoRAr​\(κ​θ0\)\\theta\_\{0\}\\notin\\Imop\\mathrm\{LoRA\}\_\{r\}\(\\kappa\\theta\_\{0\}\), for almost everyWWin this range\. Full rank holds for almost everyWWin each cell whose zero pattern admits a matrix of full rank; the other cells, in which the block maxima of many rows fall in a few common columns, have positive but tiny measure \(for instance, a column of zeros needs the maximum of its block in every row, probabilityβ−m\\beta^\{\-m\}per column for weights with independent continuous entries\)\.

\(b\) In each pipeline the initial weight isρ⁡\(q0\)=F\+δ0\\rho\(q\_\{0\}\)=F\+\\delta\_\{0\}, whereFFis the frozen part andδ0\\delta\_\{0\}the initial adapter, and the defect isF\+δ0−WF\+\\delta\_\{0\}\-W\.

- •*Split first\.*F=κ⁡\(Wres\)F=\\kappa\(W\_\{\\rm res\}\)withWres=W−δ0W\_\{\\rm res\}=W\-\\delta\_\{0\}, so the defect isκ⁡\(Wres\)\+δ0−W=κ⁡\(Wres\)−Wres\\kappa\(W\_\{\\rm res\}\)\+\\delta\_\{0\}\-W=\\kappa\(W\_\{\\rm res\}\)\-W\_\{\\rm res\}\. In LoftQ’s quantisation half\-step fort≥2t\\geq 2the same computation applies withδ0\\delta\_\{0\}replaced by the current fitδ⁡\(qt−1\)=Lt−1​Rt−1\\delta\(q\_\{t\-1\}\)=L\_\{t\-1\}R\_\{t\-1\}\.
- •*Quantise, then fit\.*F=κ​WF=\\kappa Wand HF setsL​R=SVDr⁡\(W−κ​W\)=−SVDr⁡\(e\)LR=\\operatorname\{SVD\}\_\{r\}\(W\-\\kappa W\)=\-\\operatorname\{SVD\}\_\{r\}\(e\)\(the best rank\-rrapproximation is odd,SVDr⁡\(−X\)=−SVDr⁡\(X\)\\operatorname\{SVD\}\_\{r\}\(\-X\)=\-\\operatorname\{SVD\}\_\{r\}\(X\)\), and the forward pass multiplies byss, soδ0=−s​SVDr⁡\(e\)\\delta\_\{0\}=\-s\\operatorname\{SVD\}\_\{r\}\(e\)and the defect ise−s​SVDr​\(e\)e\-s\\operatorname\{SVD\}\_\{r\}\(e\)\. WriteP=SVDr⁡\(e\)P=\\operatorname\{SVD\}\_\{r\}\(e\)\. By[Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\),PPande−Pe\-Pare Frobenius\-orthogonal, so∥e−s​P∥2=∥\(e−P\)\+\(1−s\)​P∥2=∥e−P∥2\+\(1−s\)2​∥P∥2\\lVert e\-sP\\rVert^\{2\}=\\lVert\(e\-P\)\+\(1\-s\)P\\rVert^\{2\}=\\lVert e\-P\\rVert^\{2\}\+\(1\-s\)^\{2\}\\lVert P\\rVert^\{2\}\. Fore≠0e\\neq 0,P≠0P\\neq 0, so the defect equalse−Pe\-Piffs=1s=1; then it has rankrank⁡e−r\\rk e\-rwhenrank⁡e≥r\\rk e\\geq r, being the sum of the singular terms beyond therr\-th\. HF’s defaultsr=α=8r=\\alpha=8gives=α/r=1s=\\alpha/r=1\. Fors=2s=2,∥e−2​P∥2=∥e−P∥2\+∥P∥2=∥e∥2\\lVert e\-2P\\rVert^\{2\}=\\lVert e\-P\\rVert^\{2\}\+\\lVert P\\rVert^\{2\}=\\lVert e\\rVert^\{2\}, again by orthogonality; in general∥e−s​P∥<∥e∥\\lVert e\-sP\\rVert<\\lVert e\\rVertiff\(1−s\)2<1\(1\-s\)^\{2\}<1, that is0<s<20<s<2\. A module rolled back to the zero initialisation hasδ0=0\\delta\_\{0\}=0and defectκ​W−W=e\\kappa W\-W=e\.
- •*Quantise, split, re\-quantise\.*The split is computed fromκ​W\\kappa W, the residualr′=κ​W−δ0r^\{\\prime\}=\\kappa W\-\\delta\_\{0\}is re\-quantised, andF=κ⁡\(r′\)F=\\kappa\(r^\{\\prime\}\)\. The defect isκ⁡\(r′\)\+δ0−W=\(κ⁡\(r′\)−r′\)\+\(κ​W−W\)=e\+κ⁡\(r′\)−r′\\kappa\(r^\{\\prime\}\)\+\\delta\_\{0\}\-W=\(\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}\)\+\(\\kappa W\-W\)=e\+\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}\.

For quantise\-then\-split with frozen partFF, the defectF\+δ0−WF\+\\delta\_\{0\}\-Wequalse=κ​W−We=\\kappa W\-WiffF=κ​W−δ0F=\\kappa W\-\\delta\_\{0\}\. Storingκ​W\\kappa Wquantised together with the frozen factors ofδ0=s​L0​R0\\delta\_\{0\}=sL\_\{0\}R\_\{0\}, which holdr⁡\(m\+n\)r\(m\+n\)numbers, realises thisFFexactly\. Re\-quantising instead addsκ⁡\(r′\)−r′\\kappa\(r^\{\\prime\}\)\-r^\{\\prime\}, which vanishes iffr′r^\{\\prime\}is representable, that is, iffr′r^\{\\prime\}lies in the null set of representable matrices\.

*Action families\.*OFT atR=IR=Igivesρ⁡\(q0\)=κ​θ0​I=κ​θ0\\rho\(q\_\{0\}\)=\\kappa\\theta\_\{0\}\\,I=\\kappa\\theta\_\{0\}, BOFT atR=IR=Iwith unit scale givesdiag⁡\(𝟏\)​κ​θ0​I=κ​θ0\\diag\(\\mathbf\{1\}\)\\,\\kappa\\theta\_\{0\}\\,I=\\kappa\\theta\_\{0\}, and HiRA atB=0B=0givesκ​θ0\+κ​θ0⊙\(0⋅A\)=κ​θ0\\kappa\\theta\_\{0\}\+\\kappa\\theta\_\{0\}\\odot\(0\\cdot A\)=\\kappa\\theta\_\{0\}; each has defectee\. DoRA atB=0B=0computesρ⁡\(q0\)=diag⁡\(μ0,i/∥\(κ​θ0\)i∥\)​κ​θ0\\rho\(q\_\{0\}\)=\\diag\\big\(\\mu\_\{0,i\}/\\lVert\(\\kappa\\theta\_\{0\}\)\_\{i\}\\rVert\\big\)\\kappa\\theta\_\{0\}, normalising the rows of its frozen baseκ​θ0\\kappa\\theta\_\{0\}\(assumed nonzero\) and rescaling them by the magnitude vectorμ0\\mu\_\{0\}\. Ifμ0\\mu\_\{0\}holds the row norms ofκ​θ0\\kappa\\theta\_\{0\}, thenρ⁡\(q0\)=κ​θ0\\rho\(q\_\{0\}\)=\\kappa\\theta\_\{0\}and the defect isee; if it holds those ofθ0\\theta\_\{0\}, the defect is the displayed matrix\.

*Moved invariants\.*HiRA’s updateW↦W⊙\(𝟏\+B​A\)W\\mapsto W\\odot\(\\mathbf\{1\}\+BA\)preserves every zero entry of its base \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\(e\)\)\. NF4 contains the code00, so every entry ofθ0\\theta\_\{0\}whose ratio to its block’s absmax rounds to00becomes an exact zero ofκ​θ0\\kappa\\theta\_\{0\}, while zero entries ofθ0\\theta\_\{0\}stay zero; the zero set ofκ​θ0\\kappa\\theta\_\{0\}therefore contains that ofθ0\\theta\_\{0\}, and generically more\. Exact\-Cayley OFT hasρ⁡\(Q\)​ρ​\(Q\)⊤=κ​θ0​C​\(Q\)​C​\(Q\)⊤​κ​θ0⊤=κ​θ0​κ​θ0⊤\\rho\(Q\)\\rho\(Q\)^\{\\top\}=\\kappa\\theta\_\{0\}\\,C\(Q\)C\(Q\)^\{\\top\}\\kappa\\theta\_\{0\}^\{\\top\}=\\kappa\\theta\_\{0\}\\kappa\\theta\_\{0\}^\{\\top\}\([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\)\. HF’s default factor isR=C⁡\(Q\)​\(I−Q4\)R=C\(Q\)\(I\-Q^\{4\}\)\([Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(e\)\), soρ​ρ⊤=κ​θ0​\(I−Q4\)2​κ​θ0⊤=κ​θ0​κ​θ0⊤\+O⁡\(∥Q∥4\)\\rho\\rho^\{\\top\}=\\kappa\\theta\_\{0\}\(I\-Q^\{4\}\)^\{2\}\\kappa\\theta\_\{0\}^\{\\top\}=\\kappa\\theta\_\{0\}\\kappa\\theta\_\{0\}^\{\\top\}\+O\(\\lVert Q\\rVert^\{4\}\)\.

\(c\) LoftQ starts fromL0​R0=0L\_\{0\}R\_\{0\}=0and, fort≥1t\\geq 1, setsQt=κ⁡\(θ0−Lt−1​Rt−1\)Q\_\{t\}=\\kappa\(\\theta\_\{0\}\-L\_\{t\-1\}R\_\{t\-1\}\)andLt​Rt=SVDr⁡\(θ0−Qt\)L\_\{t\}R\_\{t\}=\\operatorname\{SVD\}\_\{r\}\(\\theta\_\{0\}\-Q\_\{t\}\)\. By[Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\) the SVD step minimises∥θ0−Qt−L​R∥\\lVert\\theta\_\{0\}\-Q\_\{t\}\-LR\\rVertover rank\-rrproducts, so it never increases the objective\. The quantisation step returnsκ⁡\(y\)\\kappa\(y\)fory=θ0−Lt−1​Rt−1y=\\theta\_\{0\}\-L\_\{t\-1\}R\_\{t\-1\}, which need not be the representable matrix nearest toyy, and the objective can increase\. An example with blocks equal to the rows of a2×32\\times 3matrix, absmax scales, the codebook\{0,±12,±1\}\\\{0,\\pm\\tfrac\{1\}\{2\},\\pm 1\\\}andr=1r=1:

θ0=\(11−54541−1\),Q1=κ⁡\(θ0\)=54​\(11−111−1\),θ0−Q1=14​\(−1−100−11\)\.\\theta\_\{0\}=\\begin\{pmatrix\}1&1&\-\\tfrac\{5\}\{4\}\\\\ \\tfrac\{5\}\{4\}&1&\-1\\end\{pmatrix\},\\quad Q\_\{1\}=\\kappa\(\\theta\_\{0\}\)=\\tfrac\{5\}\{4\}\\begin\{pmatrix\}1&1&\-1\\\\ 1&1&\-1\\end\{pmatrix\},\\quad\\theta\_\{0\}\-Q\_\{1\}=\\tfrac\{1\}\{4\}\\begin\{pmatrix\}\-1&\-1&0\\\\ 0&\-1&1\\end\{pmatrix\}\.Here\(θ0−Q1\)​\(θ0−Q1\)⊤=116​\(2112\)\(\\theta\_\{0\}\-Q\_\{1\}\)\(\\theta\_\{0\}\-Q\_\{1\}\)^\{\\top\}=\\tfrac\{1\}\{16\}\\left\(\\begin\{smallmatrix\}2&1\\\\ 1&2\\end\{smallmatrix\}\\right\)has eigenvalues316\\tfrac\{3\}\{16\}and116\\tfrac\{1\}\{16\}, with top eigenvector\(1,1\)/2\(1,1\)/\\sqrt\{2\}, soL1​R1=12​\(1111\)​\(θ0−Q1\)=18​\(−1−21−1−21\)L\_\{1\}R\_\{1\}=\\tfrac\{1\}\{2\}\\left\(\\begin\{smallmatrix\}1&1\\\\ 1&1\\end\{smallmatrix\}\\right\)\(\\theta\_\{0\}\-Q\_\{1\}\)=\\tfrac\{1\}\{8\}\\left\(\\begin\{smallmatrix\}\-1&\-2&1\\\\ \-1&\-2&1\\end\{smallmatrix\}\\right\)and the objective after the SVD step is∥θ0−Q1−L1​R1∥2=116\\lVert\\theta\_\{0\}\-Q\_\{1\}\-L\_\{1\}R\_\{1\}\\rVert^\{2\}=\\tfrac\{1\}\{16\}\. The next quantisation step acts ony=θ0−L1​R1=18​\(910−111110−9\)y=\\theta\_\{0\}\-L\_\{1\}R\_\{1\}=\\tfrac\{1\}\{8\}\\left\(\\begin\{smallmatrix\}9&10&\-11\\\\ 11&10&\-9\\end\{smallmatrix\}\\right\)\. Each row has absmax118\\tfrac\{11\}\{8\}, and the ratios911\\tfrac\{9\}\{11\}and1011\\tfrac\{10\}\{11\}round to11, soQ2=118​\(11−111−1\)Q\_\{2\}=\\tfrac\{11\}\{8\}\\left\(\\begin\{smallmatrix\}1&1&\-1\\\\ 1&1&\-1\\end\{smallmatrix\}\\right\)and

∥θ0−Q2−L1​R1∥2=∥y−Q2∥2=2​\(116\+164\)=532\>116\.\\lVert\\theta\_\{0\}\-Q\_\{2\}\-L\_\{1\}R\_\{1\}\\rVert^\{2\}=\\lVert y\-Q\_\{2\}\\rVert^\{2\}=2\\Big\(\\tfrac\{1\}\{16\}\+\\tfrac\{1\}\{64\}\\Big\)=\\tfrac\{5\}\{32\}\>\\tfrac\{1\}\{16\}\.The objective rose, andQ1Q\_\{1\}, which is representable, is closer toyythanκ⁡\(y\)\\kappa\(y\)\. The deployed weight afterTTiterations isQT\+s​LT​RTQ\_\{T\}\+sL\_\{T\}R\_\{T\}, because HF stores the fitted factors and the forward pass multiplies byss; its defect is

QT\+s​LT​RT−W=\(QT\+LT​RT−W\)\+\(s−1\)​LT​RT\.Q\_\{T\}\+sL\_\{T\}R\_\{T\}\-W=\(Q\_\{T\}\+L\_\{T\}R\_\{T\}\-W\)\+\(s\-1\)L\_\{T\}R\_\{T\}\.Fors=1s=1this is minus the attained residual, so the output is pointed up to that residual\. Fors≠1s\\neq 1the second term\(s−1\)​LT​RT\(s\-1\)L\_\{T\}R\_\{T\}is proportional to the fit itself and does not shrink with the residual; in the referee simulation of the official configuration \(r=64r=64,α=16\\alpha=16,s=1/4s=1/4, five iterations, a Gaussian512×512512\\times 512NF4 weight\) the attained residual falls to0\.67​∥e∥0\.67\\lVert e\\rVertwhile the deployed defect rises from0\.91​∥e∥0\.91\\lVert e\\rVertto0\.96​∥e∥0\.96\\lVert e\\rVert\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

\(d\) Fine\-tuning and then quantising givesκ⁡\(θ0\+Δ\)\\kappa\(\\theta\_\{0\}\+\\Delta\), a value ofκ\\kappa\. Exact merging after quantising givesκ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Delta, which is representable iffΔ\\Deltalies in the translate of the representable set by−κ​θ0\-\\kappa\\theta\_\{0\}, a null set; so for genericΔ\\Deltait leaves the quantised type and the two orders differ\. For a uniform grid with a fixed scaleaaper block, the representable matrices areaatimes integer codes in range, soκ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Deltais representable iffΔ=a⁡\(c′−c0\)\\Delta=a\(c^\{\\prime\}\-c\_\{0\}\)withc′c^\{\\prime\}in range, that is, iffΔ\\Deltalies on the scale lattice\. NF4 is not closed under addition\. Its smallest positive code isq1≈0\.0796q\_\{1\}\\approx 0\.0796, and2​q1≈0\.15922q\_\{1\}\\approx 0\.1592is not a code \(the next codes are0\.07960\.0796and0\.16090\.1609\)\. The blocks\(1,q1,0,…\)\(1,q\_\{1\},0,\\dots\)and\(0,q1,0,…\)\(0,q\_\{1\},0,\\dots\)are representable, with absmax scales11andq1q\_\{1\}, but their sum\(1,2​q1,0,…\)\(1,2q\_\{1\},0,\\dots\)has absmax11and needs the code2​q12q\_\{1\}\. Absmax grids with uniform codes of stephhare not closed either:\(1,h,0,…\)\(1,h,0,\\dots\)and\(h,0,0,…\)\(h,0,0,\\dots\)are representable, while their sum\(1\+h,h,0,…\)\(1\+h,h,0,\\dots\)would need the codeh/\(1\+h\)h/\(1\+h\), which is not a multiple ofhhfor0<h<10<h<1\. A type\-preserving merge returnsκ⁡\(κ​θ0\+Δ\)\\kappa\(\\kappa\\theta\_\{0\}\+\\Delta\), which is representable and differs fromκ​θ0\+Δ\\kappa\\theta\_\{0\}\+\\Deltawhenever the latter is not, so it is lossy for genericΔ\\Delta\. Finally, affine group quantisation stores each groupggasag​cg\+βg​𝟏a\_\{g\}c\_\{g\}\+\\beta\_\{g\}\\mathbf\{1\}, with integer codescgc\_\{g\}and a high\-precision scaleaga\_\{g\}and zero\-pointβg\\beta\_\{g\}\. QA\-LoRA pools its input over the quantisation groups, so its update is constant on each group,Δg=δg​𝟏\\Delta\_\{g\}=\\delta\_\{g\}\\mathbf\{1\}\([Xu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib514)\), andag​cg\+βg​𝟏\+δg​𝟏=ag​cg\+\(βg\+δg\)​𝟏a\_\{g\}c\_\{g\}\+\\beta\_\{g\}\\mathbf\{1\}\+\\delta\_\{g\}\\mathbf\{1\}=a\_\{g\}c\_\{g\}\+\(\\beta\_\{g\}\+\\delta\_\{g\}\)\\mathbf\{1\}keeps the codes and moves only the zero\-point\. ∎

### C\.43 Theorem V\.3

Theorem V\.3\(rebasing closure\)\.\(a\)*Additive methods with a fixed shapeC∋0C\\ni 0*, with frames, masks and compression operators frozen atθ0\\theta\_\{0\}across cycles\. ThenRK=θ0\+C\+KR\_\{K\}=\\theta\_\{0\}\+C^\{\+K\}, theKK\-fold Minkowski sum, andR∞=θ0\+Mon⁡\(C\)R\_\{\\infty\}=\\theta\_\{0\}\+\\operatorname\{Mon\}\(C\), the additive monoid generated byCC\. IfCCis closed under real scalings,R∞=θ0\+span⁡CR\_\{\\infty\}=\\theta\_\{0\}\+\\spanop C\.\(b\)*Action methods*, where one cycle applies an element of a setSSwith1∈S⊆G1\\in S\\subseteq G; for scaling methods,S⊆End⁡\(Θ\)S\\subseteq\\operatorname\{End\}\(\\Theta\)is a subset containing the identity\. ThenRK=SK​θ0R\_\{K\}=S^\{K\}\\theta\_\{0\}, andR∞R\_\{\\infty\}is the orbit of the monoid generated bySS\. SupposeSSlies in the subgroup generated by connected Lie subgroupsH1,…,HpH\_\{1\},\\dots,H\_\{p\}and contains a neighbourhood of11in each\. Then the monoid generated bySSis the connected Lie subgroupHHwithLie⁡\(H\)=Lie⁡⟨𝔥1,…,𝔥p⟩\\operatorname\{Lie\}\(H\)=\\operatorname\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle\.\(c\)*Block\-orthogonal factors\.*Let the factors have coordinate blocksIj⊆\{1,…,n\}I\_\{j\}\\subseteq\\\{1,\\dots,n\\\}\(block OFT, BOFT factors, Givens planes\)\. ThenLie⁡⟨𝔰​𝔬​\(Ij\)⟩j=⨁C𝔰​𝔬​\(C\)\\operatorname\{Lie\}\\langle\\mathfrak\{so\}\(I\_\{j\}\)\\rangle\_\{j\}=\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\), summed over the connected componentsCCof the hypergraph\(\{1,…,n\},\{Ij\}\)\(\\\{1,\\dots,n\\\},\\\{I\_\{j\}\\\}\)\.*Corollaries\.*1\.ReLoRA\.RK=W0\+ℳ≤min⁡\(K​r,m,n\)R\_\{K\}=W\_\{0\}\+\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}\. It reaches full fine\-tuning iffK≥⌈min⁡\(m,n\)/r⌉K\\geq\\lceil\\min\(m,n\)/r\\rceil\.2\.Idempotent methodsgain nothing from rebasing\. These are the methods whose shape is exactly a subspace, subgroup or submonoid independent of the base, with frames, masks and compression operators frozen across cycles: LoRA\-FA with a fixedAA, LoRA\-XS with frozen frames, MoRA with fixed operators, FourierFT with fixed indices, WaveFT, C3A, MiSS \(default balance mode; bat mode is base\-dependent but still idempotent, since its blockwise mapsW↦\(W\+I\)​\(I\+M\)−IW\\mapsto\(W\+I\)\(I\+M\)\-Iform a monoid\), SHiRA, BitFit, FISH with a fixed mask, LN\-tuning, \(IA\)3, SSF and RoAd\. Block OFT with a fixed partition and an exact Cayley map gains only the measure\-zero set of block rotations with an eigenvalue−1\-1, soR∞=W0⋅∏SO⁡\(b\)=R1¯R\_\{\\infty\}=W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)=\\overline\{R\_\{1\}\}\. Constrained OFT \(COFT, anε\\varepsilon\-ball\) with an exact Cayley map gains up to∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\); HF’s coft=True keeps the Cayley–Neumann map by default and so falls under corollary 4\. Methods that recompute frames or masks from the merged weight \(an SVD\-recomputing LoRA\-XS, ReMoRA’s alternating operators, resampled indices\) do gain\.3\.VeRA\.VeRA∞=LoRA\-FA​\(A\)\\mathrm\{VeRA\}^\{\\infty\}=\\text\{LoRA\-FA\}\(A\)whenever the frozenBBhas no zero entries\.4\.OFT family\.Block OFT with an exact Cayley map is stuck inW0⋅∏SO⁡\(b\)W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)\. HF’s default Cayley–Neumann factor isR=C⁡\(Q\)​\(I−Q4\)R=C\(Q\)\(I\-Q^\{4\}\), withCCthe exact Cayley map, soR⊤​R=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}andRRis not orthogonal\. Its singular values are\|1−μk4\|\\lvert 1\-\\mu\_\{k\}^\{4\}\\rvert, where±i​μk\\pm i\\mu\_\{k\}are the eigenvalues ofQQ\. In the operating regime∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}, every factor is a contraction\. Re\-merging then decreasesKoutK\_\{\\rm out\}in the Loewner order and no singular value grows, so the drift is monotone shrinkage, and the reachable set lies inW0W\_\{0\}times the block contraction semigroup\. With unconstrainedQQthe factors include singular ones \(μk=1\\mu\_\{k\}=1\), and re\-merging generates∏\(GL\+​\(b\)∪ℳ≤b−2\)\\prod\\big\(\\mathrm\{GL\}^\{\+\}\(b\)\\cup\\mathcal\{M\}\_\{\\leq b\-2\}\\big\), or∏\(CO\+​\(2\)∪\{0\}\)\\prod\\big\(\\mathrm\{CO\}^\{\+\}\(2\)\\cup\\\{0\\\}\\big\)whenb=2b=2; this clause is established at the level of a proof sketch\. Iterated BOFT \(exact Cayley in HF\) withm′m^\{\\prime\}butterfly factors of block sizebbreachesdiag⁡\(ℝm\)⋅W0⋅P⁡\(∏SO⁡\(b​2m′−1\)\)​P⊤\\diag\(\\mathbb\{R\}^\{m\}\)\\cdot W\_\{0\}\\cdot P\\big\(\\prod\\mathrm\{SO\}\(b\\,2^\{m^\{\\prime\}\-1\}\)\\big\)P^\{\\top\}, because the factor hypergraph has components of sizeb​2m′−1b\\,2^\{m^\{\\prime\}\-1\}\. For injectiveW0W\_\{0\}this isdiag⁡\(ℝm\)⋅W0⋅SO⁡\(n\)\\diag\(\\mathbb\{R\}^\{m\}\)\\cdot W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)iffb​2m′−1=nb\\,2^\{m^\{\\prime\}\-1\}=n, the full\-depth butterflym′=1\+log2⁡\(n/b\)m^\{\\prime\}=1\+\\log\_\{2\}\(n/b\)\. HF’s default boft\_n\_butterfly\_factor=1 is block OFT timesdiag⁡\(s\)\\diag\(s\)\. A*single*full\-depth BOFT has dimension at mostm′​nb​\(b2\)m^\{\\prime\}\\tfrac\{n\}\{b\}\\binom\{b\}\{2\}, which is belowdimSO⁡\(n\)\\dim\\mathrm\{SO\}\(n\)whenm′​\(b−1\)<n−1m^\{\\prime\}\(b\-1\)<n\-1; its matrices are dense, while the set they form has positive codimension\.5\.Re\-HRA\(HF apply\_GS=False\), withW0W\_\{0\}injective andrreven\. ThenRK=\(W0\+ℳ≤K​r\)∩W0⋅SO⁡\(n\)R\_\{K\}=\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq Kr\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\), so rebasing commutes with the meet of[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\. This multiplicative twin of ReLoRA reachesW0⋅SO⁡\(n\)W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)in at most⌈n/r⌉\\lceil n/r\\rceilcycles\.6\.GaLore \(with a gradient\-SVD or a random projector\), LISA and MeZO, under the optimizer hypotheses of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), are rebasing schemes of*linear*methods whose shape moves\. That is why, as a set over the sequences of shapes they can choose, they reach all ofΘ\\Theta, while a fixed linear shapeLLreaches onlyθ0\+L\\theta\_\{0\}\+L\(corollary 2\)\. Low rank is the per\-period obstruction only for GaLore; LISA and MeZO already make full\-rank per\-matrix updates and gain coverage\. Flora proper takes full\-space steps with a compressed momentum, so it has no per\-period low\-rank shape\.7\.Merging never rescues an exactly orthogonal method from its absolute invariants\.In exact arithmetic, every exactly orthogonal input\-side rebasing preservesKoutK\_\{\\rm out\}and the spectrum, and every output\-side one preservesKinK\_\{\\rm in\}and the spectrum\.

\(a\) With frames, masks and compression operators frozen, every cycle trains the same additive shape over the current base:Im⁡M⁡\(θ\)=θ\+C\\Imop M\(\\theta\)=\\theta\+Cfor everyθ\\theta\. By the recursion of §[C\.41](https://arxiv.org/html/2610.12728#A3.SS41),R1=θ0\+CR\_\{1\}=\\theta\_\{0\}\+CandRK\+1=⋃θ∈RK\(θ\+C\)=RK\+CR\_\{K\+1\}=\\bigcup\_\{\\theta\\in R\_\{K\}\}\(\\theta\+C\)=R\_\{K\}\+C, soRK=θ0\+C\+KR\_\{K\}=\\theta\_\{0\}\+C^\{\+K\}by induction\. Since0∈C0\\in C, the setsC\+KC^\{\+K\}increase, and their union is the additive monoidMon⁡\(C\)\\operatorname\{Mon\}\(C\)generated byCC\([Background D\.10](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx1)\); henceR∞=θ0\+Mon⁡\(C\)R\_\{\\infty\}=\\theta\_\{0\}\+\\operatorname\{Mon\}\(C\)\. Ifλ​C⊆C\\lambda C\\subseteq Cfor every realλ\\lambda, then−C=C\-C=C, soMon⁡\(C\)\\operatorname\{Mon\}\(C\)is closed under negation and is an additive subgroup, andλ⁡\(x1\+⋯\+xK\)=λ​x1\+⋯\+λ​xK\\lambda\(x\_\{1\}\+\\dots\+x\_\{K\}\)=\\lambda x\_\{1\}\+\\dots\+\\lambda x\_\{K\}shows that it is closed under scaling\. So it is a linear subspace containingCCand contained inspan⁡C\\spanop C, and equalsspan⁡C\\spanop C\.

\(b\) Each cycle applies one element ofSSto the current point, soIm⁡M⁡\(θ\)=S​θ\\Imop M\(\\theta\)=S\\theta, and the recursion givesRK=SK​θ0R\_\{K\}=S^\{K\}\\theta\_\{0\}\. Since1∈S1\\in S, the setsSKS^\{K\}increase and their union is the monoidMon⁡\(S\)\\operatorname\{Mon\}\(S\)generated bySS, soR∞=Mon⁡\(S\)​θ0R\_\{\\infty\}=\\operatorname\{Mon\}\(S\)\\,\\theta\_\{0\}, the orbit ofθ0\\theta\_\{0\}under that monoid\. For the second statement, letUi⊆SU\_\{i\}\\subseteq Sbe a neighbourhood of11inHiH\_\{i\}\. ThenUi∩Ui−1U\_\{i\}\\cap U\_\{i\}^\{\-1\}is a symmetric neighbourhood of11in the connected Lie groupHiH\_\{i\}, so it generatesHiH\_\{i\}as a monoid \([Auxiliary result B\.24](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(a\)\), andHi⊆Mon⁡\(S\)H\_\{i\}\\subseteq\\operatorname\{Mon\}\(S\)\. EachHiH\_\{i\}is a group, so the monoid generated byH1∪⋯∪HpH\_\{1\}\\cup\\dots\\cup H\_\{p\}is closed under inverses and equals the subgroup they generate, which is the connected Lie subgroupHHwithLie⁡\(H\)=Lie⁡⟨𝔥1,…,𝔥p⟩\\operatorname\{Lie\}\(H\)=\\operatorname\{Lie\}\\langle\\mathfrak\{h\}\_\{1\},\\dots,\\mathfrak\{h\}\_\{p\}\\rangle\([Auxiliary result B\.24](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(d\)\)\. SoH⊆Mon⁡\(S\)H\\subseteq\\operatorname\{Mon\}\(S\)\. ConverselyS⊆HS\\subseteq Hby hypothesis, andHHis closed under products, soMon⁡\(S\)⊆H\\operatorname\{Mon\}\(S\)\\subseteq H\.

\(c\) WriteLa​b=Ea​b−Eb​aL\_\{ab\}=E\_\{ab\}\-E\_\{ba\}, so𝔰𝔬\(I\)=span\{La​b:a,b∈I\}\\mathfrak\{so\}\(I\)=\\spanop\\\{L\_\{ab\}:a,b\\in I\\\}\. For distincta,b,ca,b,c,

La​cLc​b=Ea​b,Lc​bLa​c=Eb​a,so\[La​c,Lc​b\]=La​b,L\_\{ac\}L\_\{cb\}=E\_\{ab\},\\qquad L\_\{cb\}L\_\{ac\}=E\_\{ba\},\\qquad\\text\{so\}\\qquad\[L\_\{ac\},L\_\{cb\}\]=L\_\{ab\},whileLa​b​Lc​d=Lc​d​La​b=0L\_\{ab\}L\_\{cd\}=L\_\{cd\}L\_\{ab\}=0when\{a,b\}∩\{c,d\}=∅\\\{a,b\\\}\\cap\\\{c,d\\\}=\\emptyset\. Let𝔤=Lie⁡⟨𝔰​𝔬​\(Ij\)⟩j\\mathfrak\{g\}=\\operatorname\{Lie\}\\langle\\mathfrak\{so\}\(I\_\{j\}\)\\rangle\_\{j\}\. Ifa≠ba\\neq blie in one componentCC, join them by a shortest chaina=c0,c1,…,ct=ba=c\_\{0\},c\_\{1\},\\dots,c\_\{t\}=bof distinct coordinates with consecutive ones in a common block\. By induction ontt,La​b∈𝔤L\_\{ab\}\\in\\mathfrak\{g\}: fort=1t=1,La​b∈𝔰​𝔬​\(Ij\)L\_\{ab\}\\in\\mathfrak\{so\}\(I\_\{j\}\)for a block containing both; fort\>1t\>1,La​ct−1∈𝔤L\_\{ac\_\{t\-1\}\}\\in\\mathfrak\{g\}by induction andLct−1​b∈𝔤L\_\{c\_\{t\-1\}b\}\\in\\mathfrak\{g\}, and their bracket isLa​bL\_\{ab\}\. So⨁C𝔰​𝔬​\(C\)⊆𝔤\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\)\\subseteq\\mathfrak\{g\}\. Conversely,⨁C𝔰​𝔬​\(C\)\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\)is a Lie subalgebra, since each𝔰​𝔬​\(C\)\\mathfrak\{so\}\(C\)is one and elements of different components have disjoint supports and commute, and it contains every𝔰​𝔬​\(Ij\)\\mathfrak\{so\}\(I\_\{j\}\), because each block lies in one component\. Hence𝔤=⨁C𝔰​𝔬​\(C\)\\mathfrak\{g\}=\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\), of dimension∑C\(\|C\|2\)\\sum\_\{C\}\\binom\{\\lvert C\\rvert\}\{2\}\.

*Corollary 1\.*LoRA’s shape\{s​B​A\}\\\{sBA\\\}isℳ≤r\\mathcal\{M\}\_\{\\leq r\}, which contains00and is closed under real scalings, so \(a\) applies\. By[Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\), a sum ofKKmatrices of rank at mostrrhas rank at mostmin⁡\(K​r,m,n\)\\min\(Kr,m,n\)\. Conversely, a matrix of rankk≤min⁡\(K​r,m,n\)k\\leq\\min\(Kr,m,n\)is a sum ofkkrank\-one terms of its SVD \([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\), which split intoKKgroups of at mostrrterms, each inℳ≤r\\mathcal\{M\}\_\{\\leq r\}\. SoC\+K=ℳ≤min⁡\(K​r,m,n\)C^\{\+K\}=\\mathcal\{M\}\_\{\\leq\\min\(Kr,m,n\)\}\. This is all ofℝm×n\\mathbb\{R\}^\{m\\times n\}iffK​r≥min⁡\(m,n\)Kr\\geq\\min\(m,n\), that is, iffK≥⌈min⁡\(m,n\)/r⌉K\\geq\\lceil\\min\(m,n\)/r\\rceil\.

*Corollary 2\.*If the shape is an additive submonoidCCindependent of the base \(in particular a linear subspace\), thenC\+K=CC^\{\+K\}=CandRK=R1R\_\{K\}=R\_\{1\}for allKKby \(a\); if it is a submonoidSSof a group or ofEnd⁡\(Θ\)\\operatorname\{End\}\(\\Theta\), thenSK=SS^\{K\}=SandRK=R1R\_\{K\}=R\_\{1\}by \(b\)\. The listed methods are of this kind once their frames, masks and operators are frozen: the updates of LoRA\-FA \(\{X​A\}\\\{XA\\\}\), LoRA\-XS \(\{U​M​V⊤\}\\\{UMV^\{\\top\}\\\}\), MoRA \(linear in its square matrix\), FourierFT and WaveFT \(fixed coefficient positions\), C3A \(block\-circulant matrices\), MiSS in balance mode \(linear in its shared block\), SHiRA, BitFit, FISH and LN\-tuning \(fixed coordinates\) form linear subspaces; the scalings of \(IA\)3,W↦diag⁡\(ℓ\)​WW\\mapsto\\diag\(\\ell\)W, form a monoid inEnd⁡\(Θ\)\\operatorname\{End\}\(\\Theta\); SSF’s mapsy↦s⊙y\+ty\\mapsto s\\odot y\+tform a monoid under composition,s′⊙\(s⊙y\+t\)\+t′=\(s′​s\)⊙y\+\(s′​t\+t′\)s^\{\\prime\}\\odot\(s\\odot y\+t\)\+t^\{\\prime\}=\(s^\{\\prime\}s\)\\odot y\+\(s^\{\\prime\}t\+t^\{\\prime\}\); RoAd’s2×22\\times 2blocks range over the scaled rotations or over all2×22\\times 2matrices, both monoids; and MiSS’s bat\-mode maps satisfy\(W\+I\)​\(I\+M\)​\(I\+M′\)−I\(W\+I\)\(I\+M\)\(I\+M^\{\\prime\}\)\-Iwith\(I\+M\)​\(I\+M′\)=I\+M′′\(I\+M\)\(I\+M^\{\\prime\}\)=I\+M^\{\\prime\\prime\},M′′=M\+M′\+M​M′M^\{\\prime\\prime\}=M\+M^\{\\prime\}\+MM^\{\\prime\}again in the class, so they form a monoid\.

For block OFT with an exact Cayley map and a fixed partition,S=∏Cay⁡\(b\)S=\\prod\\mathrm\{Cay\}\(b\)withCay⁡\(b\)=C⁡\(𝔰​𝔬​\(b\)\)=\{R∈SO⁡\(b\):−1∉spec⁡R\}\\mathrm\{Cay\}\(b\)=C\(\\mathfrak\{so\}\(b\)\)=\\\{R\\in\\mathrm\{SO\}\(b\):\-1\\notin\\operatorname\{spec\}R\\\}, open and dense inSO⁡\(b\)\\mathrm\{SO\}\(b\)\([Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(c\)\)\. Every element ofSO⁡\(b\)\\mathrm\{SO\}\(b\)is a product of two elements ofCay⁡\(b\)\\mathrm\{Cay\}\(b\)\([Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(d\)\), soS2=∏SO⁡\(b\)S^\{2\}=\\prod\\mathrm\{SO\}\(b\), andRK=W0⋅∏SO⁡\(b\)R\_\{K\}=W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\)forK≥2K\\geq 2\. By continuity ofg↦W0​gg\\mapsto W\_\{0\}gand compactness of∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\),R1¯=W0⋅∏Cay⁡\(b\)¯=W0⋅∏SO⁡\(b\)\\overline\{R\_\{1\}\}=W\_\{0\}\\cdot\\overline\{\\prod\\mathrm\{Cay\}\(b\)\}=W\_\{0\}\\cdot\\prod\\mathrm\{SO\}\(b\), andR∞∖R1R\_\{\\infty\}\\setminus R\_\{1\}lies in the image of the Haar\-null set\{∏jdet\(I\+Rj\)=0\}\\\{\\prod\_\{j\}\\det\(I\+R\_\{j\}\)=0\\\}\. For COFT with an exact Cayley map,SSis the image underCCof a ball about00, which contains a neighbourhood of11in∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\)becauseCCis a local diffeomorphism at00; by \(b\) with the single connected group∏SO⁡\(b\)\\prod\\mathrm\{SO\}\(b\),Mon⁡\(S\)=∏SO⁡\(b\)\\operatorname\{Mon\}\(S\)=\\prod\\mathrm\{SO\}\(b\)\.

Methods whose shape changes with the base escape \(a\)\. For linear shapesL1≠L2L\_\{1\}\\neq L\_\{2\}used in two cycles \(resampled indices, alternating operators\),R2⊇θ0\+L1\+L2⊋θ0\+L1=R1R\_\{2\}\\supseteq\\theta\_\{0\}\+L\_\{1\}\+L\_\{2\}\\supsetneq\\theta\_\{0\}\+L\_\{1\}=R\_\{1\}wheneverL2⊈L1L\_\{2\}\\not\\subseteq L\_\{1\}\. For LoRA\-XS with frames recomputed from the merged weight, taker=1r=1andW0=diag⁡\(2,1\)W\_\{0\}=\\diag\(2,1\):R1=\{diag⁡\(2\+t,1\)\}R\_\{1\}=\\\{\\diag\(2\+t,1\)\\\}; the pointdiag⁡\(−12,1\)∈R1\\diag\(\-\\tfrac\{1\}\{2\},1\)\\in R\_\{1\}has top singular directione2e\_\{2\}, so the second cycle adds\{t​e2​e2⊤\}\\\{te\_\{2\}e\_\{2\}^\{\\top\}\\\}and reachesdiag⁡\(−12,1\+t\)∉R1\\diag\(\-\\tfrac\{1\}\{2\},1\+t\)\\notin R\_\{1\}\.

*Corollary 3\.*VeRA’s shape isC=\{Λb​B​Λd​A\}C=\\\{\\Lambda\_\{b\}B\\Lambda\_\{d\}A\\\}withBB,AAfrozen andb∈ℝmb\\in\\mathbb\{R\}^\{m\},d∈ℝrd\\in\\mathbb\{R\}^\{r\}trainable\. It contains00, is closed under real scalings \(b↦λ​bb\\mapsto\\lambda b\), and lies in\{X​A:X∈ℝm×r\}\\\{XA:X\\in\\mathbb\{R\}^\{m\\times r\}\\\}\. The entries ofΛb​B​Λd\\Lambda\_\{b\}B\\Lambda\_\{d\}arebi​Bi​j​djb\_\{i\}B\_\{ij\}d\_\{j\}, sob=eib=e\_\{i\},d=ejd=e\_\{j\}givesBi​j​Ei​jB\_\{ij\}E\_\{ij\}; if everyBi​j≠0B\_\{ij\}\\neq 0, the span of the matricesΛb​B​Λd\\Lambda\_\{b\}B\\Lambda\_\{d\}isℝm×r\\mathbb\{R\}^\{m\\times r\}, andspan⁡C=\{X​A\}\\spanop C=\\\{XA\\\}\. By \(a\),R∞=W0\+\{X​A\}R\_\{\\infty\}=W\_\{0\}\+\\\{XA\\\}, the image of LoRA\-FA with the sameAA\.

*Corollary 4\.**Exact Cayley\.*This is corollary 2\.

*The Cayley–Neumann factor\.*By[Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(e\),R=C⁡\(Q\)​\(I−Q4\)R=C\(Q\)\(I\-Q^\{4\}\),R⊤​R=\(I−Q4\)2R^\{\\top\}R=\(I\-Q^\{4\}\)^\{2\}, and the singular values ofRRare\|1−μk4\|\\lvert 1\-\\mu\_\{k\}^\{4\}\\rvert, each twice, and11for a zero eigenvalue ofQQ\. SoRRis orthogonal only when everyμk4∈\{0,2\}\\mu\_\{k\}^\{4\}\\in\\\{0,2\\\}\. If∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}, then0≤μk4≤20\\leq\\mu\_\{k\}^\{4\}\\leq 2and\|1−μk4\|≤1\\lvert 1\-\\mu\_\{k\}^\{4\}\\rvert\\leq 1, soRRis a contraction\. For an input\-side mergeW↦W​RW\\mapsto WRwith block\-diagonalRR,

Kout​\(W​R\)=W​R​R⊤​W⊤=W​\(I−Q4\)2​W⊤⪯W​W⊤=Kout​\(W\),K\_\{\\rm out\}\(WR\)=WRR^\{\\top\}W^\{\\top\}=W\(I\-Q^\{4\}\)^\{2\}W^\{\\top\}\\preceq WW^\{\\top\}=K\_\{\\rm out\}\(W\),becauseC⁡\(Q\)C\(Q\)commutes withQ4Q^\{4\}and\(I−Q4\)2⪯I\(I\-Q^\{4\}\)^\{2\}\\preceq I\([Auxiliary result B\.18](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(b\)\); by[Auxiliary result B\.18](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\) no singular value ofWWgrows\. Products of block\-diagonal contractions are block\-diagonal contractions \([Auxiliary result B\.18](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\), so the reachable set lies inW0W\_\{0\}times that semigroup\.

*UnconstrainedQQ\.*Fix one block of sizebband letSb=\{R⁡\(Q\):Q∈𝔰​𝔬​\(b\)\}S\_\{b\}=\\\{R\(Q\):Q\\in\\mathfrak\{so\}\(b\)\\\}; since the blocks are independent andR⁡\(0\)=IR\(0\)=I, the monoid generated by the block\-diagonal factors is∏Mon⁡\(Sb\)\\prod\\operatorname\{Mon\}\(S\_\{b\}\)\. WriteQ=O⁡\(⨁kμk​J⊕0\)​O⊤Q=O\(\\bigoplus\_\{k\}\\mu\_\{k\}J\\oplus 0\)O^\{\\top\}withO∈O⁡\(b\)O\\in\\mathrm\{O\}\(b\),J=\(0−110\)J=\\left\(\\begin\{smallmatrix\}0&\-1\\\\ 1&0\\end\{smallmatrix\}\\right\)andμk≥0\\mu\_\{k\}\\geq 0; thenI−Q4=O⁡\(⨁k\(1−μk4\)​I2⊕I\)​O⊤I\-Q^\{4\}=O\(\\bigoplus\_\{k\}\(1\-\\mu\_\{k\}^\{4\}\)I\_\{2\}\\oplus I\)O^\{\\top\}\.

*⊆\\subseteq\.*detR=detC⁡\(Q\)​det\(I−Q4\)=∏k\(1−μk4\)2≥0\\det R=\\det C\(Q\)\\det\(I\-Q^\{4\}\)=\\prod\_\{k\}\(1\-\\mu\_\{k\}^\{4\}\)^\{2\}\\geq 0, andrank⁡R=rank⁡\(I−Q4\)=b−2​\#​\{k:μk=1\}\\rk R=\\rk\(I\-Q^\{4\}\)=b\-2\\\#\\\{k:\\mu\_\{k\}=1\\\}\. So an invertible factor lies inGL\+​\(b\)\\mathrm\{GL\}^\{\+\}\(b\)and a singular one has rank at mostb−2b\-2\. Products of elements ofGL\+​\(b\)\\mathrm\{GL\}^\{\+\}\(b\)stay in it, and any product with a factor of rank at mostb−2b\-2has rank at mostb−2b\-2\. HenceMon⁡\(Sb\)⊆GL\+​\(b\)∪ℳ≤b−2\\operatorname\{Mon\}\(S\_\{b\}\)\\subseteq\\mathrm\{GL\}^\{\+\}\(b\)\\cup\\mathcal\{M\}\_\{\\leq b\-2\}\.

*Positive scalings\.*C⁡\(−Q\)=C​\(Q\)−1C\(\-Q\)=C\(Q\)^\{\-1\}commutes withQ4Q^\{4\}, soR⁡\(Q\)​R​\(−Q\)=\(I−Q4\)2=O⁡\(⨁k\(1−μk4\)2​I2⊕I\)​O⊤R\(Q\)R\(\-Q\)=\(I\-Q^\{4\}\)^\{2\}=O\(\\bigoplus\_\{k\}\(1\-\\mu\_\{k\}^\{4\}\)^\{2\}I\_\{2\}\\oplus I\)O^\{\\top\}\. Asμk\\mu\_\{k\}ranges over\[0,∞\)\[0,\\infty\),\(1−μk4\)2\(1\-\\mu\_\{k\}^\{4\}\)^\{2\}takes every positive value \(μk4=1\+λ\\mu\_\{k\}^\{4\}=1\+\\sqrt\{\\lambda\}givesλ\\lambda\)\. TakingOOa permutation matrix,Mon⁡\(Sb\)\\operatorname\{Mon\}\(S\_\{b\}\)contains every positive diagonal matrix that is constant on the pairs of a matching of the coordinates and11off it\. Forb≥3b\\geq 3these generate all positive diagonal matrices: their logarithms include every multiple ofei\+eje\_\{i\}\+e\_\{j\},i≠ji\\neq j, andei=12​\(\(ei\+ej\)\+\(ei\+ek\)−\(ej\+ek\)\)e\_\{i\}=\\tfrac\{1\}\{2\}\\big\(\(e\_\{i\}\+e\_\{j\}\)\+\(e\_\{i\}\+e\_\{k\}\)\-\(e\_\{j\}\+e\_\{k\}\)\\big\), so products of commuting such matrices realise everydiag⁡\(ex\)\\diag\(e^\{x\}\)\. Forb=2b=2they are the positive scalars\.

*Rotations\.*For∥Q∥op<1\\lVert Q\\rVert\_\{\\rm op\}<1,I−Q4I\-Q^\{4\}is positive definite with eigenvalues1−μk41\-\\mu\_\{k\}^\{4\}on the planes ofQQ\. ChooseQ′=O⁡\(⨁kμk′​J⊕0\)​O⊤Q^\{\\prime\}=O\(\\bigoplus\_\{k\}\\mu\_\{k\}^\{\\prime\}J\\oplus 0\)O^\{\\top\}with the sameOOand\(1−μk′4\)2=\(1−μk4\)−1\(1\-\\mu\_\{k\}^\{\\prime 4\}\)^\{2\}=\(1\-\\mu\_\{k\}^\{4\}\)^\{\-1\}; thenR⁡\(Q′\)​R​\(−Q′\)=\(I−Q4\)−1R\(Q^\{\\prime\}\)R\(\-Q^\{\\prime\}\)=\(I\-Q^\{4\}\)^\{\-1\}, andC⁡\(Q\)=R⁡\(Q\)​R​\(Q′\)​R​\(−Q′\)∈Mon⁡\(Sb\)C\(Q\)=R\(Q\)R\(Q^\{\\prime\}\)R\(\-Q^\{\\prime\}\)\\in\\operatorname\{Mon\}\(S\_\{b\}\)\. The set\{C⁡\(Q\):∥Q∥op<1\}\\\{C\(Q\):\\lVert Q\\rVert\_\{\\rm op\}<1\\\}is a neighbourhood ofIIinSO⁡\(b\)\\mathrm\{SO\}\(b\)\([Auxiliary result B\.23](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(c\)\), symmetric becauseC⁡\(−Q\)=C​\(Q\)−1C\(\-Q\)=C\(Q\)^\{\-1\}, so it generatesSO⁡\(b\)\\mathrm\{SO\}\(b\)as a monoid \([Auxiliary result B\.24](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(a\)\)\. With the positive diagonals and the decompositionGL\+​\(b\)=SO⁡\(b\)​D\+​SO​\(b\)\\mathrm\{GL\}^\{\+\}\(b\)=\\mathrm\{SO\}\(b\)\\,D^\{\+\}\\mathrm\{SO\}\(b\)\([Auxiliary result B\.19](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(c\)\),GL\+​\(b\)⊆Mon⁡\(Sb\)\\mathrm\{GL\}^\{\+\}\(b\)\\subseteq\\operatorname\{Mon\}\(S\_\{b\}\)forb≥3b\\geq 3; forb=2b=2,SO⁡\(2\)⋅ℝ\>0=CO\+​\(2\)\\mathrm\{SO\}\(2\)\\cdot\\mathbb\{R\}\_\{\>0\}=\\mathrm\{CO\}^\{\+\}\(2\)\.

*Singular blocks\.*LetV⊆ℝbV\\subseteq\\mathbb\{R\}^\{b\}have dimensionb−2b\-2and chooseOOmapping the last coordinate plane ontoV⟂V^\{\\perp\}, andQV=O⁡\(0⊕J\)​O⊤Q\_\{V\}=O\(0\\oplus J\)O^\{\\top\}\. ThenQV4Q\_\{V\}^\{4\}is the projector ontoV⟂V^\{\\perp\},C⁡\(QV\)C\(Q\_\{V\}\)acts asJJonV⟂V^\{\\perp\}\(since\(I\+J\)​\(I−J\)−1=J\(I\+J\)\(I\-J\)^\{\-1\}=J\) and as the identity onVV, andR⁡\(QV\)=C⁡\(QV\)​\(I−QV4\)=PVR\(Q\_\{V\}\)=C\(Q\_\{V\}\)\(I\-Q\_\{V\}^\{4\}\)=P\_\{V\}, the orthogonal projector ontoVV\. IfPUP\_\{U\}withdimU=k\\dim U=k,1≤k≤b−21\\leq k\\leq b\-2, is in the monoid, chooseU′⊆UU^\{\\prime\}\\subseteq Uof dimensionk−1k\-1andW′⊆U⟂W^\{\\prime\}\\subseteq U^\{\\perp\}of dimensionb−1−k≤dimU⟂b\-1\-k\\leq\\dim U^\{\\perp\}, and putV=U′⊕W′V=U^\{\\prime\}\\oplus W^\{\\prime\}, of dimensionb−2b\-2\. SinceW′⟂UW^\{\\prime\}\\perp U,PU​PV=PU​\(PU′\+PW′\)=PU′P\_\{U\}P\_\{V\}=P\_\{U\}\(P\_\{U^\{\\prime\}\}\+P\_\{W^\{\\prime\}\}\)=P\_\{U^\{\\prime\}\}\. By downward induction every orthogonal projector of rank at mostb−2b\-2lies in the monoid\. A matrixXXof rankk≤b−2k\\leq b\-2isX=U​diag⁡\(σ1,…,σk,1,…,1\)​Pk​V⊤X=U\\diag\(\\sigma\_\{1\},\\dots,\\sigma\_\{k\},1,\\dots,1\)\\,P\_\{k\}\\,V^\{\\top\}by its SVD, withPkP\_\{k\}the projector onto the firstkkcoordinates\. Changing the sign of the last column ofUU, or of the last row ofV⊤V^\{\\top\}, leaves this product unchanged, becausePkP\_\{k\}kills the last coordinate; so both outer factors may be taken inGL\+​\(b\)\\mathrm\{GL\}^\{\+\}\(b\), andX∈Mon⁡\(Sb\)X\\in\\operatorname\{Mon\}\(S\_\{b\}\)\. Forb=2b=2,R⁡\(μ​J\)=\(1−μ4\)​C​\(μ​J\)R\(\\mu J\)=\(1\-\\mu^\{4\}\)\\,C\(\\mu J\)is a real multiple of a rotation, soMon⁡\(S2\)⊆CO\+​\(2\)∪\{0\}\\operatorname\{Mon\}\(S\_\{2\}\)\\subseteq\\mathrm\{CO\}^\{\+\}\(2\)\\cup\\\{0\\\}, andR⁡\(J\)=0R\(J\)=0\. AltogetherMon⁡\(Sb\)=GL\+​\(b\)∪ℳ≤b−2\\operatorname\{Mon\}\(S\_\{b\}\)=\\mathrm\{GL\}^\{\+\}\(b\)\\cup\\mathcal\{M\}\_\{\\leq b\-2\}forb≥3b\\geq 3andCO\+​\(2\)∪\{0\}\\mathrm\{CO\}^\{\+\}\(2\)\\cup\\\{0\\\}forb=2b=2, which is the clause; the steps above complete the sketch recorded in the main text\.

*BOFT\.*Group the coordinates into runs ofb/2b/2consecutive coordinates, indexed byp=0,1,…,2​n/b−1p=0,1,\\dots,2n/b\-1\. Up to one fixed relabelling of coordinates, thell\-th butterfly factor of BOFT \(l=1,…,m′l=1,\\dots,m^\{\\prime\}\) is block\-diagonal with one orthogonalb×bb\\times bblock, an exact Cayley image, on each pair of runs whose indices differ exactly in binary digitl−1l\-1\([Liu et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib285);[Dao et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib88)\); forl=1l=1these aren/bn/bconsecutive blocks of sizebb, block OFT\. Two runs lie in one component of the factor hypergraph iff their indices agree in all digits fromm′m^\{\\prime\}on, so the components consist of2m′2^\{m^\{\\prime\}\}runs,b​2m′−1b\\,2^\{m^\{\\prime\}\-1\}coordinates\. HF’s BOFT multiplies the output by a trainable scale,W↦diag⁡\(s\)​W​RW\\mapsto\\diag\(s\)\\,W\\,R, and left and right multiplications commute, so afterKKcycles the weight isdiag\(sK⋯s1\)W0R1⋯RK\\diag\(s\_\{K\}\\cdots s\_\{1\}\)\\,W\_\{0\}\\,R\_\{1\}\\cdots R\_\{K\}; the products of scale vectors already fillℝm\\mathbb\{R\}^\{m\}atK=1K=1\. The orthogonal part is generated by the setSSof products of them′m^\{\\prime\}butterfly factors\. LetHlH\_\{l\}be the group of all block rotations of levelll, connected;SSlies in the group they generate, and setting all but one factor to the identity shows thatSScontains a neighbourhood of11in eachHlH\_\{l\}, the exact Cayley image being open\. By \(b\) and \(c\),Mon⁡\(S\)\\operatorname\{Mon\}\(S\)is the connected Lie subgroup with Lie algebra⨁C𝔰​𝔬​\(C\)\\bigoplus\_\{C\}\\mathfrak\{so\}\(C\), which is the group of block rotations on the components,P⁡\(∏SO⁡\(b​2m′−1\)\)​P⊤P\\big\(\\prod\\mathrm\{SO\}\(b\\,2^\{m^\{\\prime\}\-1\}\)\\big\)P^\{\\top\}for the permutationPPthat makes the components consecutive\. This proves the reach\. Ifb​2m′−1=nb\\,2^\{m^\{\\prime\}\-1\}=nthere is one component and the reach isdiag⁡\(ℝm\)⋅W0⋅SO⁡\(n\)\\diag\(\\mathbb\{R\}^\{m\}\)\\cdot W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\. Ifc:=b​2m′−1<nc:=b\\,2^\{m^\{\\prime\}\-1\}<n, the groupG=P⁡\(∏SO⁡\(c\)\)​P⊤G=P\\big\(\\prod\\mathrm\{SO\}\(c\)\\big\)P^\{\\top\}has dimensionn⁡\(c−1\)/2<n⁡\(n−1\)/2n\(c\-1\)/2<n\(n\-1\)/2; and forW0W\_\{0\}of full column rank the reachable sets differ \(forW0=0W\_\{0\}=0both are\{0\}\\\{0\\\}\)\. Indeed, ifdiag⁡\(ℝm\)​W0​G=diag⁡\(ℝm\)​W0​SO​\(n\)\\diag\(\\mathbb\{R\}^\{m\}\)W\_\{0\}G=\\diag\(\\mathbb\{R\}^\{m\}\)W\_\{0\}\\mathrm\{SO\}\(n\), then eachR∈SO⁡\(n\)R\\in\\mathrm\{SO\}\(n\)satisfiesW0​R=diag⁡\(s\)​W0​gW\_\{0\}R=\\diag\(s\)W\_\{0\}gfor somessandg∈Gg\\in G, soM=R​g−1M=Rg^\{\-1\}satisfiesW0​M=diag⁡\(s\)​W0W\_\{0\}M=\\diag\(s\)W\_\{0\}: every rowwiw\_\{i\}ofW0W\_\{0\}is an eigenvector ofM⊤M^\{\\top\}\. These rows spanℝn\\mathbb\{R\}^\{n\}, soM⊤M^\{\\top\}is diagonalisable with real eigenvalues; being orthogonal,MMis an orthogonal involutionI−2​PEI\-2P\_\{E\}whose−1\-1\-eigenspaceEEis spanned by the rows ofW0W\_\{0\}it contains\. SoMMranges over a finite setF0F\_\{0\}, andSO⁡\(n\)⊆F0​G\\mathrm\{SO\}\(n\)\\subseteq F\_\{0\}G, a finite union of submanifolds of lower dimension, which is impossible\. Withm′=1m^\{\\prime\}=1the single level is block OFT, so HF’s default boft\_n\_butterfly\_factor=1 is block OFT timesdiag⁡\(s\)\\diag\(s\)\.

*A single BOFT\.*Its parameters numberm′​nb​\(b2\)m^\{\\prime\}\\frac\{n\}\{b\}\\binom\{b\}\{2\}, so its image has at most this dimension \([Auxiliary result B\.9](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(b\)\), andm′​nb​b⁡\(b−1\)2<n⁡\(n−1\)2m^\{\\prime\}\\frac\{n\}\{b\}\\frac\{b\(b\-1\)\}\{2\}<\\frac\{n\(n\-1\)\}\{2\}iffm′​\(b−1\)<n−1m^\{\\prime\}\(b\-1\)<n\-1; the image then has positive codimension inSO⁡\(n\)\\mathrm\{SO\}\(n\)and is Haar\-null\. At full depth,2m′=2​n/b2^\{m^\{\\prime\}\}=2n/bruns form one component, and for every pair of coordinates\(i,j\)\(i,j\)there is a chainj=c0,c1,…,cm′=ij=c\_\{0\},c\_\{1\},\\dots,c\_\{m^\{\\prime\}\}=iin whichcl−1c\_\{l\-1\}andclc\_\{l\}lie in a common block of thell\-th factor applied: each factor lets the run index flip one binary digit, a different one for each factor, and lets the position inside the run be chosen freely\. So the entry\(i,j\)\(i,j\)of the product, a polynomial in the block entries, is nonzero at a product of signed permutation matrices of determinant11\(one per block\) that routesjjtoiialong the chain\. It is therefore not identically zero on the connected group∏lHl\\prod\_\{l\}H\_\{l\}, and since the Cayley parametrisation is a diffeomorphism onto an open subset of that group, the entry is nonzero for parameters outside a nowhere\-dense analytic subset \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. Alln2n^\{2\}entries are nonzero for generic parameters, so the matrices are dense\.

*Corollary 5\.*With apply\_GS=False, one HRA cycle appliesHu1⋯HurH\_\{u\_\{1\}\}\\cdots H\_\{u\_\{r\}\}on the input side\. LetSk=\{R∈SO⁡\(n\):rank⁡\(R−I\)≤k\}S\_\{k\}=\\\{R\\in\\mathrm\{SO\}\(n\):\\rk\(R\-I\)\\leq k\\\}for evenkk\. A product ofrrreflections lies inSO⁡\(n\)\\mathrm\{SO\}\(n\)\(rreven\) and hasrank⁡\(R−I\)≤r\\rk\(R\-I\)\\leq r\([Auxiliary result B\.22](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\); conversely, ifR∈SrR\\in S\_\{r\}thenj=rank⁡\(R−I\)j=\\rk\(R\-I\)is even andRRis a product of exactlyjjreflections \([Auxiliary result B\.22](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\), which pairsHu​Hu=IH\_\{u\}H\_\{u\}=Ipad torr\. So one cycle applies exactly the elements ofSrS\_\{r\}, and by \(b\)RK=W0⋅SrKR\_\{K\}=W\_\{0\}\\cdot S\_\{r\}^\{K\}\. Next,SrK=SK​rS\_\{r\}^\{K\}=S\_\{Kr\}\. For⊆\\subseteq,R1​R2−I=R1​\(R2−I\)\+\(R1−I\)R\_\{1\}R\_\{2\}\-I=R\_\{1\}\(R\_\{2\}\-I\)\+\(R\_\{1\}\-I\)givesrank⁡\(R1​R2−I\)≤rank⁡\(R2−I\)\+rank⁡\(R1−I\)\\rk\(R\_\{1\}R\_\{2\}\-I\)\\leq\\rk\(R\_\{2\}\-I\)\+\\rk\(R\_\{1\}\-I\)\([Auxiliary result B\.12](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\)\. For⊇\\supseteq, writeR∈SK​rR\\in S\_\{Kr\}as a product ofj=rank⁡\(R−I\)≤K​rj=\\rk\(R\-I\)\\leq Krreflections,jjeven, split them intoKKconsecutive groups of even size at mostrr, and pad each group with pairs to sizerr\. Finally, sinceW0W\_\{0\}is injective,rank⁡\(W0​R−W0\)=rank⁡\(R−I\)\\rk\(W\_\{0\}R\-W\_\{0\}\)=\\rk\(R\-I\), soW0⋅SK​r=\(W0\+ℳ≤K​r\)∩W0⋅SO⁡\(n\)W\_\{0\}\\cdot S\_\{Kr\}=\(W\_\{0\}\+\\mathcal\{M\}\_\{\\leq Kr\}\)\\cap W\_\{0\}\\cdot\\mathrm\{SO\}\(n\)\. WhenK​r≥nKr\\geq n,SK​r=SO⁡\(n\)S\_\{Kr\}=\\mathrm\{SO\}\(n\), which gives the last sentence\.

*Corollary 6\.*By[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)and §[C\.39](https://arxiv.org/html/2610.12728#A3.SS39)\(8\), each period of these methods is a period of the linear methodθk\+ℒk​C\\theta\_\{k\}\+\\mathcal\{L\}\_\{k\}Cwith shapeLk=im⁡ℒkL\_\{k\}=\\operatorname\{im\}\\mathcal\{L\}\_\{k\}, soRK=θ0\+L1\+⋯\+LKR\_\{K\}=\\theta\_\{0\}\+L\_\{1\}\+\\dots\+L\_\{K\}for the chosen shapes\. For GaLore with left projection,Lk=\{Z:col⁡Z⊆col⁡Pk\}L\_\{k\}=\\\{Z:\\operatorname\{col\}Z\\subseteq\\operatorname\{col\}P\_\{k\}\\\}, and column spaces that together spanℝm\\mathbb\{R\}^\{m\}give all ofℝm×n\\mathbb\{R\}^\{m\\times n\}after⌈m/r⌉\\lceil m/r\\rceilperiods; for LISA, layer blocks covering every layer do the same; for MeZO,Lk=ℝ​zkL\_\{k\}=\\mathbb\{R\}z\_\{k\}, andzkz\_\{k\}running through a basis giveΘ\\Theta\. A fixed linear shapeLLis its own additive monoid, so it reaches onlyθ0\+L\\theta\_\{0\}\+L\(corollary 2\)\. A GaLore period moves a projected matrix by an update of rank at mostrr; a LISA period moves each sampled matrix by an unconstrained update, and a MeZO step by a multiple of the Gaussianzkz\_\{k\}, whose restriction to each matrix has full rank almost surely\. Flora’s steps contain a full\-rank term \(§[C\.37](https://arxiv.org/html/2610.12728#A3.SS37)\)\.

*Corollary 7\.*For orthogonalRR,\(W​R\)​\(W​R\)⊤=W​W⊤\(WR\)\(WR\)^\{\\top\}=WW^\{\\top\}and\(R​W\)⊤​\(R​W\)=W⊤​W\(RW\)^\{\\top\}\(RW\)=W^\{\\top\}W;W​RWRandR​WRWhave the singular values ofWW\. By induction over cycles, every input\-side rebasing preservesKoutK\_\{\\rm out\}and every output\-side oneKinK\_\{\\rm in\}, and both preserve the spectrum \([Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)\)\. ∎

Remark\(numerical checks\)\. The component sizes4,8,16,32,644,8,16,32,64forn=64n=64,b=4b=4andm′=1,…,5m^\{\\prime\}=1,\\dots,5reported after[Theorem V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)are theb​2m′−1b\\,2^\{m^\{\\prime\}\-1\}of the BOFT paragraph; the Loewner\-monotone decrease ofKoutK\_\{\\rm out\}over200200default\-OFT merges, the contraction bound for random skewQQwith∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\}, andR=0R=0atb=2b=2,Q=JQ=J, instantiate the Cayley–Neumann paragraph \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

### C\.44 Proposition V\.4

Proposition V\.4\(combination rules: gauge and naturality\)\.\(a\) A ruleΦ⁡\(\(Bi,Ai\)i\)\\Phi\\big\(\(B\_\{i\},A\_\{i\}\)\_\{i\}\\big\)for combining LoRA factors descends to a function of the adapters\(Bi​Ai\)i\(B\_\{i\}A\_\{i\}\)\_\{i\}iff it is constant on the fibres of\(Bi,Ai\)↦\(Bi​Ai\)\(B\_\{i\},A\_\{i\}\)\\mapsto\(B\_\{i\}A\_\{i\}\)\. On the full\-rank locus this is equivalent to∏iGLr\\prod\_\{i\}\\mathrm\{GL\}\_\{r\}\-invariance\. It is also equivalent forΦ\\Phicontinuous on the whole factor space; this clause rests on the unique closed orbit in each fibre, which lies in the closure of every orbit of the fibre\.•Task arithmetic∑iwi​si​Bi​Ai\\sum\_\{i\}w\_\{i\}s\_\{i\}B\_\{i\}A\_\{i\}, withsis\_\{i\}HF’s scaling \(αi/ri\\alpha\_\{i\}/r\_\{i\}, orαi/ri\\alpha\_\{i\}/\\sqrt\{r\_\{i\}\}under rsLoRA\), is invariant and lands inLoRA∑iri\\mathrm\{LoRA\}\_\{\\sum\_\{i\}r\_\{i\}\}\. It is HF’s cat with merged scale11\.•LoraHub’s factor\-wise rule\(∑iwi​Bi\)​\(∑jwj​Aj\)\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)\\big\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\\big\)is not invariant\. Replacing\(B2,A2\)\(B\_\{2\},A\_\{2\}\)by\(c​B2,A2/c\)\(cB\_\{2\},A\_\{2\}/c\)changes it by∑l≠2wl​w2​\[\(c−1\)​B2​Al\+\(c−1−1\)​Bl​A2\]\.\\sum\_\{l\\neq 2\}w\_\{l\}w\_\{2\}\\big\[\(c\-1\)B\_\{2\}A\_\{l\}\+\(c^\{\-1\}\-1\)B\_\{l\}A\_\{2\}\\big\]\.HF’s add\_weighted\_adapter\(…, combination\_type="linear"\) is a factor\-wise rule of the same type\. It weightsAiA\_\{i\}bysign⁡\(wi\)​\|wi​si\|\\operatorname\{sign\}\(w\_\{i\}\)\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}andBiB\_\{i\}by\|wi​si\|\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}and sums the factors separately, so it has cross terms\. On adapters that share a frozenAAit is still not exact up to a global factor, since the relative weights become\|wi​si\|\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}\. HF’s non\-SVD ties, dare\_\* and magnitude\_prune variants also act on factors; the \*\_svd variants act on full updates\.•Poly and MHR are different\. Their skill inventories are trained jointly through the product\-of\-sums map, which is invariant under the diagonalGLr\\mathrm\{GL\}\_\{r\}, one commonggfor every skill\. Numerically, for fixed generic routing with fewer skills than tasks and full\-rank factors, this is the whole continuous fibre\. With at least as many skills as tasks, or once the routing is counted as a parameter, there are further symmetries \(skill permutations, and re\-mixingsα↦α​M\\alpha\\mapsto\\alpha MwithM​𝟏=𝟏M\\mathbf\{1\}=\\mathbf\{1\}\)\. Their cross terms are a design choice of a jointly trained parametrisation, and raise no question of well\-definedness\.\(b\)*Well\-definedness versus exactness\.*A joint gauge fixing makes a factor\-wise rule well defined\. For rank\-preserving product\-of\-sums rules, exactness \(agreement with task arithmetic for all weightsww\) needs degree one inww, which for injective factors forces a shared factor; cat, which concatenates and raises the rank, is exact without one\. A frozen shared factor \(FFA\-LoRA; LoRA\-FA\) gives\(∑iwi​Bi\)​A=∑iwi​Bi​A\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)A=\\sum\_\{i\}w\_\{i\}B\_\{i\}Aexactly\. LoraHub with a sharedAAstill returns\(∑jwj\)​∑iwi​Bi​A\\big\(\\sum\_\{j\}w\_\{j\}\\big\)\\sum\_\{i\}w\_\{i\}B\_\{i\}A, exact only up to the global factor∑jwj\\sum\_\{j\}w\_\{j\}, since its weights are unnormalised\. A per\-module SVD canonical form leaves a\(ℤ/2\)r\(\\mathbb\{Z\}/2\)^\{r\}of sign flips per module, plus rotations within repeated singular values, and these change the cross terms\. What is needed is a*joint*frame that produces a shared factor, as KnOTS’s joint re\-factorisation of the concatenated updates through a sharedU​ΣU\\Sigmadoes\. Federated LoRA shows the difference\. FedAvg\-LoRA clients start each round from the same broadcast\(B,A\)\(B,A\), so their failure,12​\(B1\+B2\)⋅12​\(A1\+A2\)≠12​\(B1​A1\+B2​A2\)\\tfrac\{1\}\{2\}\(B\_\{1\}\+B\_\{2\}\)\\cdot\\tfrac\{1\}\{2\}\(A\_\{1\}\+A\_\{2\}\)\\neq\\tfrac\{1\}\{2\}\(B\_\{1\}A\_\{1\}\+B\_\{2\}A\_\{2\}\), is bilinearity, and it persists in a fully fixed gauge\.\(c\)*Naturality\.*On updates withNNcoordinates, writeTNT\_\{N\},BNB\_\{N\}andMonN\\mathrm\{Mon\}\_\{N\}for the invertible diagonal, signed permutation and monomial matrices\. Task arithmetic isGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-equivariant on updates\. DARE is coordinatewise and magnitude\-free,D⁡\(m⊙τ\)=m⊙\(D​τ\)D\(m\\odot\\tau\)=m\\odot\(D\\tau\)for every invertible diagonalDDand the same maskmm\. So DARE is exactly equivariant under the torusTNT\_\{N\}, sign flips included, for each mask, and under permutations in distribution; its distributional group is exactlyMonN\\mathrm\{Mon\}\_\{N\}\. TIES with global trimming \(top\-kkmagnitude, sign election by total mass as in the TIES paper and HF’s default "total", disjoint mean\) is equivariant underℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}, homotheties times signed permutations, generically \(off magnitude ties at the threshold\)\. With HF’s majority\_sign\_method="frequency", count ties are broken to\+1\+1and the group drops toℝ\>0⋅SN\\mathbb\{R\}\_\{\>0\}\\cdot S\_\{N\}\. Per\-tensor trimming changes the group to∏b\(ℝ×⋅BNb\)\\prod\_\{b\}\\big\(\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\_\{b\}\}\\big\), extended by permutations of equal\-size blocks\. DARE followed by full TIES \(with trimming\) isℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}\-equivariant in distribution; HF’s and MergeKit’s dare\_ties has no magnitude trim, so with "total" election it is exactlyTNT\_\{N\}\-equivariant per mask, and its group containsMonN\\mathrm\{Mon\}\_\{N\}\. None of the sparsifying rules \(DARE, TIES, DARE\-TIES\) isO⁡\(N\)\\mathrm\{O\}\(N\)\-equivariant\. Applied to factors and summed across adapters \(HF’s non\-SVD variants\), all of them are gauge\-dependent, and under independent per\-adapter gauges even DARE fails, through the cross terms of \(a\)\. DARE’s masking commutes withTrT\_\{r\}exactly for each mask and withMonr\\mathrm\{Mon\}\_\{r\}in distribution, so a single masked adapter, or a factor\-wise sum under one gauge shared by all adapters, isTrT\_\{r\}\-invariant per mask andMonr\\mathrm\{Mon\}\_\{r\}\-invariant in distribution\.

Writeπ⁡\(\(Bi,Ai\)i\)=\(Bi​Ai\)i\\pi\\big\(\(B\_\{i\},A\_\{i\}\)\_\{i\}\\big\)=\(B\_\{i\}A\_\{i\}\)\_\{i\}, and let adapteriihave rankrir\_\{i\}\(the statement’srrwhen all ranks agree\); its gauge groupGLri\\mathrm\{GL\}\_\{r\_\{i\}\}acts by\(Bi​gi−1,gi​Ai\)\(B\_\{i\}g\_\{i\}^\{\-1\},g\_\{i\}A\_\{i\}\), andπ\\piis invariant under∏iGLri\\prod\_\{i\}\\mathrm\{GL\}\_\{r\_\{i\}\}\.

\(a\)*Descent\.*IfΦ=φ∘π\\Phi=\\varphi\\circ\\pi, thenΦ\\Phiis constant on the fibres ofπ\\pi\. Conversely, ifΦ\\Phiis constant on fibres,φ⁡\(π⁡\(x\)\):=Φ⁡\(x\)\\varphi\(\\pi\(x\)\):=\\Phi\(x\)is a well\-defined function on the image ofπ\\pi\.

*Full\-rank locus\.*Letrank⁡Bi=rank⁡Ai=ri\\rk B\_\{i\}=\\rk A\_\{i\}=r\_\{i\}andBi′​Ai′=Bi​AiB\_\{i\}^\{\\prime\}A\_\{i\}^\{\\prime\}=B\_\{i\}A\_\{i\}\. The product has rankrir\_\{i\}, sorank⁡Bi′=rank⁡Ai′=ri\\rk B\_\{i\}^\{\\prime\}=\\rk A\_\{i\}^\{\\prime\}=r\_\{i\}, and the full\-rank locus is a union of whole fibres\. Moreovercol⁡Bi′=col⁡\(Bi​Ai\)=col⁡Bi\\operatorname\{col\}B\_\{i\}^\{\\prime\}=\\operatorname\{col\}\(B\_\{i\}A\_\{i\}\)=\\operatorname\{col\}B\_\{i\}, soBi′=Bi​gi−1B\_\{i\}^\{\\prime\}=B\_\{i\}g\_\{i\}^\{\-1\}for a uniquegi∈GLrig\_\{i\}\\in\\mathrm\{GL\}\_\{r\_\{i\}\}; thenBi​Ai=Bi​gi−1​Ai′B\_\{i\}A\_\{i\}=B\_\{i\}g\_\{i\}^\{\-1\}A\_\{i\}^\{\\prime\}, and injectivity ofBiB\_\{i\}givesAi′=gi​AiA\_\{i\}^\{\\prime\}=g\_\{i\}A\_\{i\}\. So each fibre in the full\-rank locus is a single∏iGLri\\prod\_\{i\}\\mathrm\{GL\}\_\{r\_\{i\}\}\-orbit \(as in[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\), and there constancy on fibres is the same as invariance\.

*Continuous rules\.*LetΦ\\Phibe continuous on the whole factor space and invariant, and letx,x′x,x^\{\\prime\}lie in the fibre over\(Wi\)i\(W\_\{i\}\)\_\{i\},rank⁡Wi=ki≤ri\\rk W\_\{i\}=k\_\{i\}\\leq r\_\{i\}\. By[Auxiliary result B\.27](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(c\), in each factor the pairs with productWiW\_\{i\}and both factors of rankkik\_\{i\}form oneGLri\\mathrm\{GL\}\_\{r\_\{i\}\}\-orbitOWiO\_\{W\_\{i\}\}, which lies in the closure of every orbit of the fibre\. Applying this factor by factor, there are gaugesg⁡\(t\),g′​\(t\)∈∏iGLrig\(t\),g^\{\\prime\}\(t\)\\in\\prod\_\{i\}\\mathrm\{GL\}\_\{r\_\{i\}\}withg⁡\(t\)⋅x→og\(t\)\\cdot x\\to oandg′​\(t\)⋅x′→o′g^\{\\prime\}\(t\)\\cdot x^\{\\prime\}\\to o^\{\\prime\}ast→0t\\to 0, whereo,o′∈∏iOWio,o^\{\\prime\}\\in\\prod\_\{i\}O\_\{W\_\{i\}\}, ando′=h⋅oo^\{\\prime\}=h\\cdot ofor some gaugehh\. By invariance and continuity,

Φ⁡\(x\)=limt→0Φ⁡\(g⁡\(t\)⋅x\)=Φ⁡\(o\)=Φ⁡\(h⋅o\)=Φ⁡\(o′\)=limt→0Φ⁡\(g′​\(t\)⋅x′\)=Φ⁡\(x′\)\.\\Phi\(x\)=\\lim\_\{t\\to 0\}\\Phi\(g\(t\)\\cdot x\)=\\Phi\(o\)=\\Phi\(h\\cdot o\)=\\Phi\(o^\{\\prime\}\)=\\lim\_\{t\\to 0\}\\Phi\(g^\{\\prime\}\(t\)\\cdot x^\{\\prime\}\)=\\Phi\(x^\{\\prime\}\)\.SoΦ\\Phiis constant on fibres; the converse holds because orbits lie in fibres\. The review record graded this clause as a sketch; with the degeneration of[Auxiliary result B\.27](https://arxiv.org/html/2610.12728#A2.SS0.SSSx4)\(c\), whose reason is given in Appendix[B](https://arxiv.org/html/2610.12728#A2), the steps above complete it\. Invariance alone does not suffice off the full\-rank locus: on the invariant open set\{rankA=r\}\\\{\\rk A=r\\\}the projectorProw⁡AP\_\{\\operatorname\{row\}A\}isGLr\\mathrm\{GL\}\_\{r\}\-invariant, sincerow⁡\(g​A\)=row⁡A\\operatorname\{row\}\(gA\)=\\operatorname\{row\}A, yet\(0,A\)\(0,A\)and\(0,A′\)\(0,A^\{\\prime\}\)withrow⁡A≠row⁡A′\\operatorname\{row\}A\\neq\\operatorname\{row\}A^\{\\prime\}lie in the fibre over00and receive different values\. ThisΦ\\Phihas no continuous extension to the whole factor space\.

*Task arithmetic\.*∑iwi​si​Bi​Ai\\sum\_\{i\}w\_\{i\}s\_\{i\}B\_\{i\}A\_\{i\}depends only on the products, so it is invariant, and

∑iwi​si​Bi​Ai=\[w1​s1​B1​\|⋯\|​wN​sN​BN\]​\[A1AN\]\\sum\_\{i\}w\_\{i\}s\_\{i\}B\_\{i\}A\_\{i\}=\\big\[\\,w\_\{1\}s\_\{1\}B\_\{1\}\\ \\big\|\\ \\cdots\\ \\big\|\\ w\_\{N\}s\_\{N\}B\_\{N\}\\,\\big\]\\begin\{bmatrix\}A\_\{1\}\\\\ \\vdots\\\\ A\_\{N\}\\end\{bmatrix\}is a LoRA update of rank at most∑iri\\sum\_\{i\}r\_\{i\}with scale11, the concatenation cat of[Observation I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)\.

*LoraHub\.*Expanding,\(∑iwi​Bi\)​\(∑jwj​Aj\)=∑i,jwi​wj​Bi​Aj\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)\\big\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\\big\)=\\sum\_\{i,j\}w\_\{i\}w\_\{j\}B\_\{i\}A\_\{j\}\. Under the gauge\(B2,A2\)↦\(c​B2,A2/c\)\(B\_\{2\},A\_\{2\}\)\\mapsto\(cB\_\{2\},A\_\{2\}/c\)the terms withi,j≠2i,j\\neq 2and the termw22​B2​A2w\_\{2\}^\{2\}B\_\{2\}A\_\{2\}are unchanged, eachw2​wl​B2​Alw\_\{2\}w\_\{l\}B\_\{2\}A\_\{l\}\(l≠2l\\neq 2\) gains the factorcc, and eachwl​w2​Bl​A2w\_\{l\}w\_\{2\}B\_\{l\}A\_\{2\}gainsc−1c^\{\-1\}, which gives the displayed change\. Forc≠1c\\neq 1, nonzero weights and factors for which the matricesB2​AlB\_\{2\}A\_\{l\},Bl​A2B\_\{l\}A\_\{2\}\(l≠2l\\neq 2\) are linearly independent \(true for generic factors\), the change is nonzero\. HF’s linear combination is\(∑iβi​Bi\)​\(∑jαj​Aj\)\\big\(\\sum\_\{i\}\\beta\_\{i\}B\_\{i\}\\big\)\\big\(\\sum\_\{j\}\\alpha\_\{j\}A\_\{j\}\\big\)withβi=\|wi​si\|\\beta\_\{i\}=\\sqrt\{\\lvert w\_\{i\}s\_\{i\}\\rvert\}andαj=sign⁡\(wj\)​\|wj​sj\|\\alpha\_\{j\}=\\operatorname\{sign\}\(w\_\{j\}\)\\sqrt\{\\lvert w\_\{j\}s\_\{j\}\\rvert\}\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\), a product of sums with the same cross terms and the same failure\. If all adapters shareAA, it equals\(∑jαj\)​∑iβi​Bi​A\\big\(\\sum\_\{j\}\\alpha\_\{j\}\\big\)\\sum\_\{i\}\\beta\_\{i\}B\_\{i\}A, which is a fixed multiple of∑iwi​si​Bi​A\\sum\_\{i\}w\_\{i\}s\_\{i\}B\_\{i\}Afor allBiB\_\{i\}only if theβi\\beta\_\{i\}are proportional to thewi​siw\_\{i\}s\_\{i\}, that is, only if all\|wi​si\|\\lvert w\_\{i\}s\_\{i\}\\rvertwithwi​si≠0w\_\{i\}s\_\{i\}\\neq 0are equal \(and the signs agree\); for instancew1​s1=1w\_\{1\}s\_\{1\}=1andw2​s2=4w\_\{2\}s\_\{2\}=4give relative weights1:21:2instead of1:41:4\.

*Poly and MHR\.*For taskttwith routing weightsαt​j\\alpha\_\{tj\}over skillsjjthe update isWt=\(∑jαt​j​Bj\)​\(∑jαt​j​Aj\)W\_\{t\}=\\big\(\\sum\_\{j\}\\alpha\_\{tj\}B\_\{j\}\\big\)\\big\(\\sum\_\{j\}\\alpha\_\{tj\}A\_\{j\}\\big\)\. A commongggives\(∑jαt​j​Bj​g−1\)​\(∑jαt​j​g​Aj\)=Wt\\big\(\\sum\_\{j\}\\alpha\_\{tj\}B\_\{j\}g^\{\-1\}\\big\)\\big\(\\sum\_\{j\}\\alpha\_\{tj\}gA\_\{j\}\\big\)=W\_\{t\}, so the diagonalGLr\\mathrm\{GL\}\_\{r\}preserves the map\. Per\-skill gauges\(gj\)\(g\_\{j\}\)that preserve it for all routings in an open set are diagonal on any pair of mixed skills with complementary column spaces:Wt=∑i,jαt​i​αt​j​Bi​AjW\_\{t\}=\\sum\_\{i,j\}\\alpha\_\{ti\}\\alpha\_\{tj\}B\_\{i\}A\_\{j\}is a polynomial inα\\alpha, and comparing the coefficients ofαt​i​αt​j\\alpha\_\{ti\}\\alpha\_\{tj\},i≠ji\\neq j, gives

Bi​\(gi−1​gj−I\)​Aj\+Bj​\(gj−1​gi−I\)​Ai=0\.B\_\{i\}\(g\_\{i\}^\{\-1\}g\_\{j\}\-I\)A\_\{j\}\+B\_\{j\}\(g\_\{j\}^\{\-1\}g\_\{i\}\-I\)A\_\{i\}=0\.Ifcol⁡Bi∩col⁡Bj=0\\operatorname\{col\}B\_\{i\}\\cap\\operatorname\{col\}B\_\{j\}=0, the two terms vanish separately, and for injectiveBiB\_\{i\}and surjectiveAjA\_\{j\}this forcesgi=gjg\_\{i\}=g\_\{j\}\. That for one fixed generic routing with fewer skills than tasks the continuous fibre is exactly the diagonal orbit is the numerical check reported below\. With at least as many skills as tasks the fibre is larger\. If there are more skills than tasks, a vectorv≠0v\\neq 0with∑jvj​αt​j=0\\sum\_\{j\}v\_\{j\}\\alpha\_\{tj\}=0for everyttexists, andBj↦Bj\+vj​XB\_\{j\}\\mapsto B\_\{j\}\+v\_\{j\}Xleaves every∑jαt​j​Bj\\sum\_\{j\}\\alpha\_\{tj\}B\_\{j\}unchanged for everyXX\. If there are equally many and the routing matrix is invertible, the task\-wise sums\(Ct,Dt\)=\(∑jαt​j​Bj,∑jαt​j​Aj\)\(C\_\{t\},D\_\{t\}\)=\\big\(\\sum\_\{j\}\\alpha\_\{tj\}B\_\{j\},\\sum\_\{j\}\\alpha\_\{tj\}A\_\{j\}\\big\)are free coordinates, and each task has its ownGLr\\mathrm\{GL\}\_\{r\}gauge\. Counting the routing as a parameter, relabelling the skills together with the columns ofα\\alphapreserves the map, and so does\(α,B\)↦\(α​M,M−1​B\)\(\\alpha,B\)\\mapsto\(\\alpha M,M^\{\-1\}B\)for invertibleMMwithM​𝟏=𝟏M\\mathbf\{1\}=\\mathbf\{1\}, whereBBstacks the skills \(and likewise forAA\), since\(α​M\)​\(M−1​B\)=α​B\(\\alpha M\)\(M^\{\-1\}B\)=\\alpha BandM​𝟏=𝟏M\\mathbf\{1\}=\\mathbf\{1\}keeps the rows ofα\\alphasumming to11\. Poly and MHR train skills and routing jointly through this map, so it is the method’s ownρ\\rho; the question of \(a\) concerns combining modules that were factorised independently\.

\(b\)*Gauge fixing\.*If a sectionσ\\sigmaofπ\\pichooses the factors from the products, thenΦ∘σ\\Phi\\circ\\sigmais a function of the products\.*Exactness\.*Task arithmeticT⁡\(w\)=∑iwi​Bi​AiT\(w\)=\\sum\_\{i\}w\_\{i\}B\_\{i\}A\_\{i\}\(scales absorbed\) is linear inww\. Consider rank\-preserving products of sums in which each adapter’s factor carries its own weightwiw\_\{i\}, in one sum or in both\. If both sums are weighted, as in LoraHub, the ruleP⁡\(w\)P\(w\)is homogeneous of degree two inww, andP⁡\(w\)=T⁡\(w\)P\(w\)=T\(w\)for allwwwould give, replacingwwbyλ​w\\lambda w,λ2​P​\(w\)=λ​T​\(w\)\\lambda^\{2\}P\(w\)=\\lambda T\(w\)for allλ\\lambda, henceT≡0T\\equiv 0\. So an exact rule of this kind has degree one inww: one of its factors does not depend onww, say\(∑iwi​Bi\)​X\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)X\. Then exactness for allwwmeansBi​X=Bi​AiB\_\{i\}X=B\_\{i\}A\_\{i\}for everyii, and for injectiveBiB\_\{i\}this givesAi=XA\_\{i\}=X: theAA\-factor is shared\. The symmetric caseX⁡\(∑jwj​Aj\)X\\big\(\\sum\_\{j\}w\_\{j\}A\_\{j\}\\big\)with surjectiveAjA\_\{j\}forces a sharedBB\. cat realisesT⁡\(w\)T\(w\)exactly, as shown in \(a\), with inner dimension∑iri\\sum\_\{i\}r\_\{i\}\. With a frozen sharedAA,\(∑iwi​Bi\)​A=∑iwi​Bi​A\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)A=\\sum\_\{i\}w\_\{i\}B\_\{i\}A, and LoraHub’s rule gives\(∑iwi​Bi\)​\(∑jwj\)​A\\big\(\\sum\_\{i\}w\_\{i\}B\_\{i\}\\big\)\\big\(\\sum\_\{j\}w\_\{j\}\\big\)A\.

*SVD canonical forms\.*IfBi​Ai=Ui​Σi​Vi⊤B\_\{i\}A\_\{i\}=U\_\{i\}\\Sigma\_\{i\}V\_\{i\}^\{\\top\}and the canonical factors areBi=Ui​Σi1/2B\_\{i\}=U\_\{i\}\\Sigma\_\{i\}^\{1/2\},Ai=Σi1/2​Vi⊤A\_\{i\}=\\Sigma\_\{i\}^\{1/2\}V\_\{i\}^\{\\top\}\(or any convention built fromUi,Σi,ViU\_\{i\},\\Sigma\_\{i\},V\_\{i\}\), then changing the signs of thejj\-th column ofUiU\_\{i\}and ofViV\_\{i\}gives another SVD and another canonical pair, multiplying columnjjofBiB\_\{i\}and rowjjofAiA\_\{i\}by−1\-1\. The productBi​AiB\_\{i\}A\_\{i\}is unchanged, while a cross termBi​AlB\_\{i\}A\_\{l\}\(l≠il\\neq i\) changes by−2​bi​j​al​j⊤\-2\\,b\_\{ij\}a\_\{lj\}^\{\\top\}, wherebi​jb\_\{ij\}is columnjjofBiB\_\{i\}andal​j⊤a\_\{lj\}^\{\\top\}rowjjofAlA\_\{l\}, which is nonzero generically\. Whenσj=σj\+1\\sigma\_\{j\}=\\sigma\_\{j\+1\}, rotating the corresponding columns ofUiU\_\{i\}andViV\_\{i\}by the sameO∈O⁡\(2\)O\\in\\mathrm\{O\}\(2\)also preserves the product and changes the cross terms\.*KnOTS\.*Write the horizontal concatenation\[Δ1​∣⋯∣​ΔN\]=U​Σ​\[V1⊤​∣⋯∣​VN⊤\]\[\\Delta\_\{1\}\\mid\\cdots\\mid\\Delta\_\{N\}\]=U\\Sigma\[V\_\{1\}^\{\\top\}\\mid\\cdots\\mid V\_\{N\}^\{\\top\}\]by an SVD\([Stoica et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib429)\); thenΔi=\(U​Σ\)​Vi⊤\\Delta\_\{i\}=\(U\\Sigma\)V\_\{i\}^\{\\top\}share the left factorU​ΣU\\Sigma, and\(U​Σ\)​\(∑iwi​Vi⊤\)=∑iwi​Δi\(U\\Sigma\)\\big\(\\sum\_\{i\}w\_\{i\}V\_\{i\}^\{\\top\}\\big\)=\\sum\_\{i\}w\_\{i\}\\Delta\_\{i\}exactly\.*FedAvg\-LoRA\.*Expanding,

12​\(B1\+B2\)⋅12​\(A1\+A2\)−12​\(B1​A1\+B2​A2\)=−14​\(B1−B2\)​\(A1−A2\),\\tfrac\{1\}\{2\}\(B\_\{1\}\+B\_\{2\}\)\\cdot\\tfrac\{1\}\{2\}\(A\_\{1\}\+A\_\{2\}\)\-\\tfrac\{1\}\{2\}\(B\_\{1\}A\_\{1\}\+B\_\{2\}A\_\{2\}\)=\-\\tfrac\{1\}\{4\}\(B\_\{1\}\-B\_\{2\}\)\(A\_\{1\}\-A\_\{2\}\),which is nonzero whenever\(B1−B2\)​\(A1−A2\)≠0\(B\_\{1\}\-B\_\{2\}\)\(A\_\{1\}\-A\_\{2\}\)\\neq 0, whatever gauge the broadcast pair is in\.

\(c\) Write a merge rule asΨ⁡\(τ1,…,τK\)\\Psi\(\\tau\_\{1\},\\dots,\\tau\_\{K\}\)for task vectorsτk∈ℝN\\tau\_\{k\}\\in\\mathbb\{R\}^\{N\}\(or the matrices withNNentries, flattened\)\. It isgg\-equivariant ifΨ⁡\(g​τ1,…,g​τK\)=g​Ψ​\(τ1,…,τK\)\\Psi\(g\\tau\_\{1\},\\dots,g\\tau\_\{K\}\)=g\\Psi\(\\tau\_\{1\},\\dots,\\tau\_\{K\}\), read for random rules either for each realisation of the randomness \(per mask\) or as equality of laws \(in distribution\)\.

*Task arithmetic*is linear, henceGL⁡\(Θ\)\\mathrm\{GL\}\(\\Theta\)\-equivariant\.

*DARE\.*DARE⁡\(τ\)=m⊙τ/\(1−p\)\\mathrm\{DARE\}\(\\tau\)=m\\odot\\tau/\(1\-p\), with a maskmmof independent Bernoulli\(1−p\)\(1\-p\)entries,0<p<10<p<1, drawn independently for each task, followed by a weighted sum\. For invertible diagonalDD,D⁡\(m⊙τ\)=m⊙\(D​τ\)D\(m\\odot\\tau\)=m\\odot\(D\\tau\), soDARE⁡\(D​τ\)=D​DARE​\(τ\)\\mathrm\{DARE\}\(D\\tau\)=D\\,\\mathrm\{DARE\}\(\\tau\)for each mask, and the weighted sum preserves this\. For a permutation matrixPP,P⁡\(m⊙τ\)=\(P​m\)⊙\(P​τ\)P\(m\\odot\\tau\)=\(Pm\)\\odot\(P\\tau\)andP​mPmhas the law ofmm, soDARE⁡\(P​τ\)\\mathrm\{DARE\}\(P\\tau\)has the law ofP​DARE​\(τ\)P\\,\\mathrm\{DARE\}\(\\tau\), jointly over the tasks\. Together,MonN=SN⋉TN\\mathrm\{Mon\}\_\{N\}=S\_\{N\}\\ltimes T\_\{N\}acts equivariantly in distribution\. Conversely, supposeDARE⁡\(g​τ\)\\mathrm\{DARE\}\(g\\tau\)andg​DARE​\(τ\)g\\,\\mathrm\{DARE\}\(\\tau\)have the same law for everyτ\\tau, and takeτ=ek\\tau=e\_\{k\}\. The law ofg​DARE​\(ek\)g\\,\\mathrm\{DARE\}\(e\_\{k\}\)is supported on the two points00andg​ek/\(1−p\)ge\_\{k\}/\(1\-p\), whileDARE⁡\(g​ek\)\\mathrm\{DARE\}\(ge\_\{k\}\)takes the2q2^\{q\}distinct valuesm⊙g​ek/\(1−p\)m\\odot ge\_\{k\}/\(1\-p\), each with positive probability, whereqqis the number of nonzero entries ofg​ekge\_\{k\}\. Soq=1q=1: every column ofgghas a single nonzero entry, andg∈MonNg\\in\\mathrm\{Mon\}\_\{N\}\.

*TIES with global trimming*\([Yadav et al\., 2023](https://arxiv.org/html/2610.12728#bib.bib517)\)\. Eachτk\\tau\_\{k\}is trimmed to itsk0k\_\{0\}entries of largest magnitude, givingτ^k\\hat\{\\tau\}\_\{k\}; the signγi=sign⁡\(∑kτ^k,i\)\\gamma\_\{i\}=\\operatorname\{sign\}\\big\(\\sum\_\{k\}\\hat\{\\tau\}\_\{k,i\}\\big\)is elected by total mass; and the output isλ\\lambdatimes the disjoint mean,1\|Ai\|​∑k∈Aiτ^k,i\\frac\{1\}\{\\lvert A\_\{i\}\\rvert\}\\sum\_\{k\\in A\_\{i\}\}\\hat\{\\tau\}\_\{k,i\}overAi=\{k:sign⁡τ^k,i=γi\}A\_\{i\}=\\\{k:\\operatorname\{sign\}\\hat\{\\tau\}\_\{k,i\}=\\gamma\_\{i\}\\\}\. Letg=c​P​Sg=cPSwithc≠0c\\neq 0,PPa permutation matrix andS=diag⁡\(s1,…,sN\)S=\\diag\(s\_\{1\},\\dots,s\_\{N\}\)a diagonal sign matrix, and letjjbe the coordinate thatPPmoves toii\. The magnitudes ofg​τkg\\tau\_\{k\}are\|c\|\\lvert c\\rverttimes the permuted magnitudes ofτk\\tau\_\{k\}, so, off ties at the threshold, trimming keeps the corresponding entries andg​τk^=g​τ^k\\widehat\{g\\tau\_\{k\}\}=g\\hat\{\\tau\}\_\{k\}\. The elected sign atiibecomessign⁡\(c\)​sj​γj\\operatorname\{sign\}\(c\)s\_\{j\}\\gamma\_\{j\}, the setAiA\_\{i\}for the transformed input isAjA\_\{j\}, and the mean atiiisc​sjcs\_\{j\}times the mean atjj\. So the output isggtimes the output, and TIES isℝ×⋅BN\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}\-equivariant off ties\. A non\-scalar positive diagonal fails: withN=2N=2, one task,k0=1k\_\{0\}=1,τ=\(2,1\)\\tau=\(2,1\)andD=diag⁡\(1,3\)D=\\diag\(1,3\), TIES returnsλ⁡\(2,0\)\\lambda\(2,0\)onτ\\tauandλ⁡\(0,3\)≠D​λ​\(2,0\)\\lambda\(0,3\)\\neq D\\lambda\(2,0\)onD​τD\\tau\.

*Frequency election\.*Hereγi=sign⁡\(∑ksign⁡τ^k,i\)\\gamma\_\{i\}=\\operatorname\{sign\}\\big\(\\sum\_\{k\}\\operatorname\{sign\}\\hat\{\\tau\}\_\{k,i\}\\big\), with00replaced by\+1\+1\. Positive homotheties and permutations leave the counts unchanged or permute them, so the argument above givesℝ\>0⋅SN\\mathbb\{R\}\_\{\>0\}\\cdot S\_\{N\}\-equivariance\. Sign changes fail on ties, which have positive probability because the counts are integers\. With two tasks andτ^1,i=1\\hat\{\\tau\}\_\{1,i\}=1,τ^2,i=−1\\hat\{\\tau\}\_\{2,i\}=\-1, the tie elects\+1\+1and the output atiiisλ\\lambda; after flipping coordinateiithe entries are−1,\+1\-1,\+1, the tie again elects\+1\+1, and the output atiiisλ\\lambda, not−λ\-\\lambda\. The same example defeats−I\-I\.

*Per\-tensor trimming\.*If coordinates are grouped into tensorsbbof sizesNbN\_\{b\}and each tensor is trimmed to its own top entries, an independent homothetycbc\_\{b\}and signed permutation inside each tensor preserve the trimmed sets, and permuting tensors of equal size \(and equalk0k\_\{0\}\) preserves them too; the election and the mean are coordinatewise, so the argument above applies blockwise\.

*DARE followed by TIES\.*Forg∈ℝ×⋅BN⊆MonNg\\in\\mathbb\{R\}^\{\\times\}\\cdot B\_\{N\}\\subseteq\\mathrm\{Mon\}\_\{N\}, the masked inputs\(DARE⁡\(g​τk\)\)k\(\\mathrm\{DARE\}\(g\\tau\_\{k\}\)\)\_\{k\}have the law of\(g​DARE​\(τk\)\)k\(g\\,\\mathrm\{DARE\}\(\\tau\_\{k\}\)\)\_\{k\}, and TIES commutes withggoff magnitude ties among nonzero entries \(ties among entries zeroed by the mask do not matter, since a kept zero is zero\), so the composite is equivariant in distribution\. HF’s and MergeKit’s dare\_ties\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312);[Goddard et al\., 2024](https://arxiv.org/html/2610.12728#bib.bib131)\)omit the trim: for invertible diagonalDDand fixed masks,DARE⁡\(D​τk\)=D​DARE​\(τk\)\\mathrm\{DARE\}\(D\\tau\_\{k\}\)=D\\,\\mathrm\{DARE\}\(\\tau\_\{k\}\), the elected sign atiiis multiplied bysign⁡di\\operatorname\{sign\}d\_\{i\}, the setsAiA\_\{i\}are unchanged and the mean is multiplied bydid\_\{i\}; so it is exactlyTNT\_\{N\}\-equivariant per mask, and with permutations in distribution its group containsMonN\\mathrm\{Mon\}\_\{N\}\.

*No rotations\.*LetRRrotate the plane ofe1,e2e\_\{1\},e\_\{2\}by30∘30^\{\\circ\}, soR​e1=\(cos⁡30∘,sin⁡30∘,0,…\)Re\_\{1\}=\(\\cos 30^\{\\circ\},\\sin 30^\{\\circ\},0,\\dots\)\. For DARE,R​DARE​\(e1\)R\\,\\mathrm\{DARE\}\(e\_\{1\}\)takes the values00andR​e1/\(1−p\)Re\_\{1\}/\(1\-p\), each with two nonzero entries or none, whileDARE⁡\(R​e1\)\\mathrm\{DARE\}\(Re\_\{1\}\)has exactly one nonzero entry with probability2​p​\(1−p\)\>02p\(1\-p\)\>0; the laws differ\. For TIES with one task andk0=1k\_\{0\}=1,TIES⁡\(e1\)=λ​e1\\mathrm\{TIES\}\(e\_\{1\}\)=\\lambda e\_\{1\}, soR​TIES​\(e1\)R\\,\\mathrm\{TIES\}\(e\_\{1\}\)has two nonzero entries, whileTIES\(Re1\)=λcos30∘e1\\mathrm\{TIES\}\(Re\_\{1\}\)=\\lambda\\cos 30^\{\\circ\}e\_\{1\}has one\. With one task, dare\_ties reduces to DARE, and DARE followed by trimmed TIES returns, with positive probability, the trimmed vectorλcos30∘e1/\(1−p\)\\lambda\\cos 30^\{\\circ\}e\_\{1\}/\(1\-p\)onR​e1Re\_\{1\}, which no value ofR​TIES​\(DARE⁡\(e1\)\)∈\{0,λ​R​e1/\(1−p\)\}R\\,\\mathrm\{TIES\}\(\\mathrm\{DARE\}\(e\_\{1\}\)\)\\in\\\{0,\\lambda Re\_\{1\}/\(1\-p\)\\\}matches\.

*Factor\-level rules\.*HF’s non\-SVD variants apply the sparsifier to eachBiB\_\{i\}and eachAjA\_\{j\}with independent masks and then form\(∑ivi​DARE​\(Bi\)\)​\(∑jvj​DARE​\(Aj\)\)\\big\(\\sum\_\{i\}v\_\{i\}\\,\\mathrm\{DARE\}\(B\_\{i\}\)\\big\)\\big\(\\sum\_\{j\}v\_\{j\}\\,\\mathrm\{DARE\}\(A\_\{j\}\)\\big\)\(and the analogues for TIES\)\. TheBB\-masks and theAA\-masks are independent and DARE is unbiased,𝔼​DARE​\(X\)=X\\mathbb\{E\}\\,\\mathrm\{DARE\}\(X\)=X, so the expected output is the LoraHub\-type rule\(∑ivi​Bi\)​\(∑jvj​Aj\)\\big\(\\sum\_\{i\}v\_\{i\}B\_\{i\}\\big\)\\big\(\\sum\_\{j\}v\_\{j\}A\_\{j\}\\big\)\. A gauge\(c​B2,A2/c\)\(cB\_\{2\},A\_\{2\}/c\)on one adapter changes this expectation by the formula of \(a\), which is nonzero for generic factors, so the output’s law changes: even DARE is not invariant under independent per\-adapter gauges, not even in distribution\. On the other hand, letg∈Trg\\in T\_\{r\}act on all adapters at once,Bi↦Bi​g−1B\_\{i\}\\mapsto B\_\{i\}g^\{\-1\}andAj↦g​AjA\_\{j\}\\mapsto gA\_\{j\}\. Right multiplication by a diagonal matrix scales columns, which commutes with an entrywise mask, soM⊙\(Bi​g−1\)=\(M⊙Bi\)​g−1M\\odot\(B\_\{i\}g^\{\-1\}\)=\(M\\odot B\_\{i\}\)g^\{\-1\}andM′⊙\(g​Aj\)=g⁡\(M′⊙Aj\)M^\{\\prime\}\\odot\(gA\_\{j\}\)=g\(M^\{\\prime\}\\odot A\_\{j\}\), and for every choice of masks

\(∑ivi​Mi⊙Bi​g−1\)​\(∑jvj​Mj′⊙g​Aj\)=\(∑ivi​Mi⊙Bi\)​g−1​g​\(∑jvj​Mj′⊙Aj\)\.\\Big\(\\sum\_\{i\}v\_\{i\}\\,M\_\{i\}\\odot B\_\{i\}g^\{\-1\}\\Big\)\\Big\(\\sum\_\{j\}v\_\{j\}\\,M\_\{j\}^\{\\prime\}\\odot gA\_\{j\}\\Big\)=\\Big\(\\sum\_\{i\}v\_\{i\}\\,M\_\{i\}\\odot B\_\{i\}\\Big\)g^\{\-1\}g\\Big\(\\sum\_\{j\}v\_\{j\}\\,M\_\{j\}^\{\\prime\}\\odot A\_\{j\}\\Big\)\.For a permutationg=Pg=P,M⊙\(Bi​P⊤\)=\(\(M​P\)⊙Bi\)​P⊤M\\odot\(B\_\{i\}P^\{\\top\}\)=\\big\(\(MP\)\\odot B\_\{i\}\\big\)P^\{\\top\}, andM​PMPhas the law ofMM; with all masks permuted by the samePPtheir joint law is unchanged, so the output’s law is invariant\. A single masked adapter is the case of one summand\. This givesTrT\_\{r\}\-invariance per mask andMonr\\mathrm\{Mon\}\_\{r\}\-invariance in distribution\. ∎

Remark\(numerical checks\)\. The checks reported after[Proposition V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4), namely DARE’s per\-mask equivariance under a random diagonal to2×10−152\\times 10^\{\-15\}, TIES’s homogeneity and its failure by5\.65\.6under a positive diagonal, the LoraHub formula with three modules, the order\-one change of factor\-wise DARE under a one\-adapter gauge against10−1510^\{\-15\}under a shared one, and the Poly fibre dimensions3232\(two tasks\) and4=dimGL24=\\dim\\mathrm\{GL\}\_\{2\}\(four tasks\) for three skills,r=2r=2and6×66\\times 6weights, instantiate the statements proved above; the last one is the numerical part of the Poly bullet \(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

### C\.45 Definition VI\.1

Definition VI\.1\(extensions; mergeability\)\.•Anextensionis a function\-preserving pointed embeddingι:\(Θ,f,θ0\)→\(Θ×Θ′,f~,\(θ0,θ0′\)\),ι⁡\(θ\)=\(θ,θ0′\),\\iota:\(\\Theta,f,\\theta\_\{0\}\)\\to\(\\Theta\\times\\Theta^\{\\prime\},\\tilde\{f\},\(\\theta\_\{0\},\\theta\_\{0\}^\{\\prime\}\)\),\\qquad\\iota\(\\theta\)=\(\\theta,\\theta\_\{0\}^\{\\prime\}\),withf~\(θ,θ0′,−\)=f\(θ,−\)\\tilde\{f\}\(\\theta,\\theta\_\{0\}^\{\\prime\},\-\)=f\(\\theta,\-\)\. The pointθ0′\\theta\_\{0\}^\{\\prime\}is theneutral point\(“identity at initialisation”\)\.•Anunpointed extensionhas no neutral point\.•A method on an extension ismergeableif there isμ:Q→Θ\\mu:Q\\to\\ThetawithΦf~∘ρ=Φf∘μ\\Phi\_\{\\tilde\{f\}\}\\circ\\rho=\\Phi\_\{f\}\\circ\\mu: a lift along the realisation\.•It isbox\-locally mergeableif the modified box function stays in the box’s function class: linear maps for a linear box, affine maps when the box has a bias slot\.[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)gives sufficient conditions, with explicit side conditions \(position\-uniformity, no weight sharing, bias slots\), under which this yields a merge into the base network\.

A definition needs no proof\. We check the facts that the definition and the square after it use\. HereΦf:Θ→C∞​\(X,Y\)\\Phi\_\{f\}:\\Theta\\to C^\{\\infty\}\(X,Y\),θ↦f⁡\(θ,−\)\\theta\\mapsto f\(\\theta,\-\), is the realisation, and a box’s function class consists of the maps it can compute with some weight, applied identically at every position\.

*Extensions\.*The mapι⁡\(θ\)=\(θ,θ0′\)\\iota\(\\theta\)=\(\\theta,\\theta\_\{0\}^\{\\prime\}\)is smooth, injective and immersive, and a homeomorphism onto the closed setΘ×\{θ0′\}\\Theta\\times\\\{\\theta\_\{0\}^\{\\prime\}\\\}, so it is an embedding\. It is pointed,ι⁡\(θ0\)=\(θ0,θ0′\)\\iota\(\\theta\_\{0\}\)=\(\\theta\_\{0\},\\theta\_\{0\}^\{\\prime\}\), and function\-preserving,Φf~∘ι=Φf\\Phi\_\{\\tilde\{f\}\}\\circ\\iota=\\Phi\_\{f\}, which is the displayed condition\.

*Weight\-space methods merge\.*A method\(Q,q0,ρ\)\(Q,q\_\{0\},\\rho\)onΘ\\Thetaitself, read on the trivial extension \(Θ′\\Theta^\{\\prime\}a point\), satisfies the merge condition withμ=ρ\\mu=\\rho, the arrow to full fine\-tuning of[Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(a\)\. More generally, a method on an extension whoseρ\\rhofactors asι∘μ\\iota\\circ\\mumerges byμ\\mu\.

*Merges are lifts, and need not be unique\.*The realisationΦf\\Phi\_\{f\}need not be injective\. Every function\-preserving reparametrisationϕ\\phiof the base gives another liftϕ∘μ\\phi\\circ\\mu; permutations of hidden units are examples \([Lemma VI\.2](https://arxiv.org/html/2610.12728#S6.SS0.SSSx3)\)\. Let a pointed method merge, withρ⁡\(q0\)\\rho\(q\_\{0\}\)the neutral point\(θ0,θ0′\)\(\\theta\_\{0\},\\theta\_\{0\}^\{\\prime\}\)\. Then

Φf​\(μ⁡\(q0\)\)=Φf~​\(θ0,θ0′\)=Φf​\(θ0\),\\Phi\_\{f\}\(\\mu\(q\_\{0\}\)\)=\\Phi\_\{\\tilde\{f\}\}\(\\theta\_\{0\},\\theta\_\{0\}^\{\\prime\}\)=\\Phi\_\{f\}\(\\theta\_\{0\}\),soμ⁡\(q0\)\\mu\(q\_\{0\}\)realises the pretrained function, without having to equalθ0\\theta\_\{0\}\.

*Box\-local mergeability\.*If the edit of a method modifies a single linear \(affine\) box of the base network, and for everyqqthe modified box function is linear \(affine\), position\-independent and computed by an unshared weightW⁡\(q\)W\(q\)of that box, then replacing the box’s weight byW⁡\(q\)W\(q\)and keeping the others gives aμ\\muwithΦf∘μ=Φf~∘ρ\\Phi\_\{f\}\\circ\\mu=\\Phi\_\{\\tilde\{f\}\}\\circ\\rho\. The side conditions are exactly what makes this replacement legitimate, and[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(a\) spells them out\. ∎

### C\.46 Lemma VI\.2

Lemma VI\.2\(the activation fixes the privileged basis; classical,[Sussmann \(1992\)](https://arxiv.org/html/2610.12728#bib.bib440),[Chen et al\. \(1993\)](https://arxiv.org/html/2610.12728#bib.bib58),[Godfrey et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib132)\)\.Letσ:ℝ→ℝ\\sigma:\\mathbb\{R\}\\to\\mathbb\{R\}be an elementwise activation that is not affine, and letL∈GLdL\\in\\mathrm\{GL\}\_\{d\}commute with it as maps onℝd\\mathbb\{R\}^\{d\}:L​σ×d=σ×d​LL\\,\\sigma^\{\\times d\}=\\sigma^\{\\times d\}L\.\(a\) Ifσ∈C2\\sigma\\in C^\{2\}, thenL=P​DL=PDis monomial\.\(b\) Ifσ\\sigmais ReLU or leaky ReLU with slopeα≠1\\alpha\\neq 1, thenL=P​DL=PDwithDDa positive diagonal\.\(c\) If moreoverσ\\sigmais real\-analytic at00withσ⁡\(t\)=∑ak​tk\\sigma\(t\)=\\sum a\_\{k\}t^\{k\}, each diagonal entryccofDDsatisfiesc​ak=ck​akca\_\{k\}=c^\{k\}a\_\{k\}for allkk\. Soc=1c=1for GELU and SiLU, sinceσ′′​\(0\)≠0\\sigma^\{\\prime\\prime\}\(0\)\\neq 0, andc=±1c=\\pm 1for tanh\.

Writeℓi\\ell\_\{i\}for rowiiofLL; it is nonzero becauseLLis invertible\. Rowiiof the identityL​σ×d​\(x\)=σ×d​\(L​x\)L\\,\\sigma^\{\\times d\}\(x\)=\\sigma^\{\\times d\}\(Lx\)reads

∑jLi​j​σ​\(xj\)=σ⁡\(ℓi⋅x\)for all​x∈ℝd\.\\sum\_\{j\}L\_\{ij\}\\,\\sigma\(x\_\{j\}\)=\\sigma\(\\ell\_\{i\}\\cdot x\)\\qquad\\text\{for all \}x\\in\\mathbb\{R\}^\{d\}\.\(27\)Once each row has a single nonzero entry, the positions of these entries are distinct, since otherwise a column ofLLwould vanish; soL=P​DL=PDwithPPa permutation matrix andDDan invertible diagonal matrix\.

\(a\) Forj≠kj\\neq k, the mixed partial∂j∂k\\partial\_\{j\}\\partial\_\{k\}of the left side of \([27](https://arxiv.org/html/2610.12728#A3.E27)\) vanishes, since each term depends on one coordinate, and the mixed partial of the right side isLi​j​Li​k​σ′′​\(ℓi⋅x\)L\_\{ij\}L\_\{ik\}\\,\\sigma^\{\\prime\\prime\}\(\\ell\_\{i\}\\cdot x\)\. Sinceσ\\sigmaisC2C^\{2\}and not affine,σ′′​\(t0\)≠0\\sigma^\{\\prime\\prime\}\(t\_\{0\}\)\\neq 0for somet0t\_\{0\}, and sinceℓi≠0\\ell\_\{i\}\\neq 0there isxxwithℓi⋅x=t0\\ell\_\{i\}\\cdot x=t\_\{0\}\. HenceLi​j​Li​k=0L\_\{ij\}L\_\{ik\}=0for allj≠kj\\neq k: rowiihas a single nonzero entry\.

\(b\) Letσ⁡\(t\)=t\\sigma\(t\)=tfort≥0t\\geq 0andσ⁡\(t\)=α​t\\sigma\(t\)=\\alpha tfort<0t<0, withα≠1\\alpha\\neq 1\(α=0\\alpha=0is ReLU\)\. Thenσ\\sigmais smooth off00, and not differentiable at00, where its one\-sided derivatives11andα\\alphadiffer\. The left side of \([27](https://arxiv.org/html/2610.12728#A3.E27)\) is a sum of functions of separate coordinates, so it is differentiable atxxiff each term withLi​j≠0L\_\{ij\}\\neq 0is differentiable inxjx\_\{j\}; its non\-differentiability set is⋃j:Li​j≠0\{xj=0\}\\bigcup\_\{j:L\_\{ij\}\\neq 0\}\\\{x\_\{j\}=0\\\}\. The right side is smooth whereℓi⋅x≠0\\ell\_\{i\}\\cdot x\\neq 0, and at a point withℓi⋅x=0\\ell\_\{i\}\\cdot x=0its one\-sided directional derivatives along±ℓi\\pm\\ell\_\{i\}are∥ℓi∥2\\lVert\\ell\_\{i\}\\rVert^\{2\}and−α​∥ℓi∥2\-\\alpha\\lVert\\ell\_\{i\}\\rVert^\{2\}, which are not negatives of each other, so it is not differentiable there; its non\-differentiability set is the hyperplane\{ℓi⋅x=0\}\\\{\\ell\_\{i\}\\cdot x=0\\\}\. The two sets are equal\. By[Auxiliary result B\.36](https://arxiv.org/html/2610.12728#A2.SS0.SSSx6)\(b\) the hyperplane is one of the coordinate hyperplanes\{xj0=0\}\\\{x\_\{j\_\{0\}\}=0\\\}, and each\{xj=0\}\\\{x\_\{j\}=0\\\}withLi​j≠0L\_\{ij\}\\neq 0is contained in it and so equal to it; henceLi​j≠0L\_\{ij\}\\neq 0only forj=j0j=j\_\{0\}\. Then \([27](https://arxiv.org/html/2610.12728#A3.E27)\) readsLi​j0​σ​\(t\)=σ⁡\(Li​j0​t\)L\_\{ij\_\{0\}\}\\sigma\(t\)=\\sigma\(L\_\{ij\_\{0\}\}t\)for alltt\. Att=1t=1this isLi​j0=σ⁡\(Li​j0\)L\_\{ij\_\{0\}\}=\\sigma\(L\_\{ij\_\{0\}\}\); ifLi​j0<0L\_\{ij\_\{0\}\}<0the right side isα​Li​j0\\alpha L\_\{ij\_\{0\}\}, which would forceα=1\\alpha=1\. SoLi​j0\>0L\_\{ij\_\{0\}\}\>0, andDDis positive\.

\(c\) A permutation matrix commutes with every elementwise map \(rule R2 of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\), soD​σ×d=P−1​L​σ×d=P−1​σ×d​L=σ×d​P−1​L=σ×d​DD\\sigma^\{\\times d\}=P^\{\-1\}L\\sigma^\{\\times d\}=P^\{\-1\}\\sigma^\{\\times d\}L=\\sigma^\{\\times d\}P^\{\-1\}L=\\sigma^\{\\times d\}D, that is,c​σ​\(t\)=σ⁡\(c​t\)c\\,\\sigma\(t\)=\\sigma\(ct\)for each diagonal entryc≠0c\\neq 0and alltt\. Near00both sides are power series,∑kc​ak​tk=∑kak​ck​tk\\sum\_\{k\}ca\_\{k\}t^\{k\}=\\sum\_\{k\}a\_\{k\}c^\{k\}t^\{k\}, soc​ak=ck​akca\_\{k\}=c^\{k\}a\_\{k\}for everykk\([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(c\)\)\. For GELU,σ⁡\(t\)=t​Φ𝒩​\(t\)\\sigma\(t\)=t\\,\\Phi\_\{\\mathcal\{N\}\}\(t\)with the standard normal distribution functionΦ𝒩​\(t\)=12\+t/2​π\+O⁡\(t3\)\\Phi\_\{\\mathcal\{N\}\}\(t\)=\\tfrac\{1\}\{2\}\+t/\\sqrt\{2\\pi\}\+O\(t^\{3\}\), soa2=1/2​π≠0a\_\{2\}=1/\\sqrt\{2\\pi\}\\neq 0; for SiLU,σ⁡\(t\)=t/\(1\+e−t\)=t2\+t24\+O⁡\(t4\)\\sigma\(t\)=t/\(1\+e^\{\-t\}\)=\\tfrac\{t\}\{2\}\+\\tfrac\{t^\{2\}\}\{4\}\+O\(t^\{4\}\), soa2=14≠0a\_\{2\}=\\tfrac\{1\}\{4\}\\neq 0\. In both casesc=c2c=c^\{2\}, andc≠0c\\neq 0givesc=1c=1\. For tanh,a3=−13≠0a\_\{3\}=\-\\tfrac\{1\}\{3\}\\neq 0, soc=c3c=c^\{3\}andc=±1c=\\pm 1; both occur, tanh being odd\. ∎

### C\.47 Theorem VI\.3

Theorem VI\.3\(the merge calculus; known in part\)\.Work with activation\-side diagrams in a symmetric monoidal category of continuous, piecewise\-smooth maps with copy and discard nodes\. \(We call copy and the Hadamard product “spiders” by analogy only; in this cartesian setting they do not satisfy the Frobenius law\.\) A*trainable box*is a node whose weight ranges over a stated class: all linear maps, or all affine maps when the box has a bias slot\. R2 and R4–R7 are identities of maps; R1 is a closure property and R3 a characterisation\.Consequences:\(a\)Affine edits\.Let a method insert a position\-uniform affine editEqE\_\{q\}on a wirewwof the base networkff, for everyq∈Qq\\in Q\. Suppose every reader ofwwis a trainable, unshared box \(no weight tying\) whose class is closed under precomposition withEqE\_\{q\}; a shift needs a bias slot\. Then the method merges, byWc↦Wc​EqW\_\{c\}\\mapsto W\_\{c\}E\_\{q\}for all readersccat once\. Dually, it merges into an untied trainable box that writeswwwheneverEqE\_\{q\}composed with that box stays in its class\. Absorption into a box that the extension itself inserts does not count\. Edits that violate these hypotheses fall outside \(a\): on the reader side, R1–R7 do not merge position\-selective edits or edits on a wire that is also read by a skip connection, a norm or the network output\. Whole\-network non\-mergeability is open\. Position\-selective edits \(ReFT/LoReFT, and position\-gated LoRA variants such as Activated LoRA\) are not box\-locally mergeable\. Residual\-stream edits need case analysis: an edit merges into the box that produces the wire when that box can absorb it \(an untied embedding table; for a fixed\-position edit at layer00, a learned absolute positional table\), a shift folds into a writer’s bias when one exists, a diagonal edit after a post\-LN folds into that LN’s affine parameters, and an orthogonal edit in an RMSNorm model merges globally, SliceGPT/QuaRot\-style\. No merge is known for LoReFT’s non\-orthogonal low\-rank edit\.\(b\)Diagonal methods\.A diagonal method merges into a box reached by R1–R7 if all of the following hold: the rules move*every*copy ofdiag⁡\(ℓ\)\\diag\(\\ell\)into trainable boxes for allℓ\\ellin the method’s parameter domain \(withℓ≥0\\ell\\geq 0when R3 is used\); no position\-dependent operator lies on the path \(for RoPE,ℓ\\ellis constant on each rotary pair\); the target box is unshared, orℓ\\ellis constant across the sharing group; and every shift has a bias slot\. Conversely, the RoPE condition is necessary for sliding a query scaling ontoWKW\_\{K\}whenWQW\_\{Q\}andWKW\_\{K\}have full rank and at least two occurring relative offsets have rotation angles that are not multiples ofπ\\pi\. Without such genericity hypotheses the conditions are not necessary: a dead rotary pair, or zero entries ofℓ\\ellunder ReLU, can still merge\.\(c\)Non\-affine edits\.For a linear or affine box, an edit whose modified box function is non\-affine in the box input, for some parameter value, is not box\-locally mergeable\. For linear and affine boxes this is the definition; for non\-linear boxes such as attention, see[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\.\(d\)Type relativity\.A generic high\-precision update is off the quantisation grid, so exact merging of a QLoRA or LoftQ update leaves the quantised type, a type\-preserving merge is lossy, and QA\-LoRA’s group\-constant adapters are the exception \([Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(d\)\)\.

A diagram denotes a map on activations, and two diagrams related by an identity of maps compute the same network function; a rewrite that ends with every edit inside trainable boxes of the base network therefore exhibits a mergeμ\\mu, the new weights of those boxes\. “Position\-independent” means that the same map acts at every token position\.

*R1\.*Composites and sums of linear \(affine\) maps are linear \(affine\)\. If each acts identically at every position, so does the result, which is then computed by one weight \(and bias\) at all positions\.

*R2\.*For a permutationπ\\piwith matrixPP,\(P​σ×d​\(x\)\)i=σ⁡\(xπ−1​\(i\)\)=σ⁡\(\(P​x\)i\)\(P\\sigma^\{\\times d\}\(x\)\)\_\{i\}=\\sigma\(x\_\{\\pi^\{\-1\}\(i\)\}\)=\\sigma\(\(Px\)\_\{i\}\)\.

*R3\.*Coordinatewise the identity saysℓi​σ​\(t\)=σ⁡\(ℓi​t\)\\ell\_\{i\}\\sigma\(t\)=\\sigma\(\\ell\_\{i\}t\)for allℓi≥0\\ell\_\{i\}\\geq 0andtt; withd=1d=1this is positive homogeneity of degree one, and conversely positive homogeneity gives the identity in every coordinate\. A positively homogeneousσ\\sigmasatisfiesσ⁡\(t\)=t​σ​\(1\)\\sigma\(t\)=t\\,\\sigma\(1\)fort≥0t\\geq 0andσ⁡\(t\)=\|t\|​σ​\(−1\)\\sigma\(t\)=\\lvert t\\rvert\\,\\sigma\(\-1\)fort<0t<0, so it is linear on each half\-line\. ReLU and leaky ReLU are of this form\. GELU, SiLU and tanh are differentiable at00and not linear, so by[Auxiliary result B\.36](https://arxiv.org/html/2610.12728#A2.SS0.SSSx6)\(a\) they are not positively homogeneous\.

*R4*holds entrywise,ℓi​ai​bi=\(ℓi​ai\)​bi\\ell\_\{i\}a\_\{i\}b\_\{i\}=\(\\ell\_\{i\}a\_\{i\}\)b\_\{i\}\.

*R5\.*⟨M​q,k⟩=q⊤​M⊤​k=⟨q,M⊤​k⟩\\langle Mq,k\\rangle=q^\{\\top\}M^\{\\top\}k=\\langle q,M^\{\\top\}k\\rangle\. Under RoPE,RmR\_\{m\}is block\-diagonal with the planar rotationsρ⁡\(m​ωj\)\\rho\(m\\omega\_\{j\}\)on the rotary pairs, in either pairing convention, soRm⊤=R−mR\_\{m\}^\{\\top\}=R\_\{\-m\}andRm⊤​Rn=Rn−mR\_\{m\}^\{\\top\}R\_\{n\}=R\_\{n\-m\}\. Hence⟨Rm​M​q,Rn​k⟩=q⊤​M⊤​Rn−m​k=⟨q,M⊤​Rn−m​k⟩\\langle R\_\{m\}Mq,R\_\{n\}k\\rangle=q^\{\\top\}M^\{\\top\}R\_\{n\-m\}k=\\langle q,M^\{\\top\}R\_\{n\-m\}k\\ranglefor everyMM, while⟨Rm​q,Rn​M⊤​k⟩=⟨q,Rn−m​M⊤​k⟩\\langle R\_\{m\}q,R\_\{n\}M^\{\\top\}k\\rangle=\\langle q,R\_\{n\-m\}M^\{\\top\}k\\rangle\. These agree for allq,kq,kiffM⊤​Rn−m=Rn−m​M⊤M^\{\\top\}R\_\{n\-m\}=R\_\{n\-m\}M^\{\\top\}, and for all occurring\(m,n\)\(m,n\)iffM⊤M^\{\\top\}commutes with every occurringRΔR\_\{\\Delta\}\. ForM=diag⁡\(ℓ\)M=\\diag\(\\ell\)the commutation holds pair by pair\. On a pair with entries\(ℓa,ℓb\)\(\\ell\_\{a\},\\ell\_\{b\}\)and rotation angleφ\\varphi,

\(ℓa00ℓb\)​\(cos⁡φ−sin⁡φsin⁡φcos⁡φ\)−\(cos⁡φ−sin⁡φsin⁡φcos⁡φ\)​\(ℓa00ℓb\)=\(ℓb−ℓa\)​sin⁡φ​\(0110\),\\begin\{pmatrix\}\\ell\_\{a\}&0\\\\ 0&\\ell\_\{b\}\\end\{pmatrix\}\\begin\{pmatrix\}\\cos\\varphi&\-\\sin\\varphi\\\\ \\sin\\varphi&\\cos\\varphi\\end\{pmatrix\}\-\\begin\{pmatrix\}\\cos\\varphi&\-\\sin\\varphi\\\\ \\sin\\varphi&\\cos\\varphi\\end\{pmatrix\}\\begin\{pmatrix\}\\ell\_\{a\}&0\\\\ 0&\\ell\_\{b\}\\end\{pmatrix\}=\(\\ell\_\{b\}\-\\ell\_\{a\}\)\\sin\\varphi\\begin\{pmatrix\}0&1\\\\ 1&0\\end\{pmatrix\},which vanishes iffℓa=ℓb\\ell\_\{a\}=\\ell\_\{b\}orφ∈π​ℤ\\varphi\\in\\pi\\mathbb\{Z\}\. So the slide holds iffℓ\\ellis constant on every pair whose angleΔ​ωj\\Delta\\omega\_\{j\}is not a multiple ofπ\\pifor some occurringΔ\\Delta\. For RoPE,0<ωj≤10<\\omega\_\{j\}\\leq 1, so every pair qualifies as soon as the offsetΔ=1\\Delta=1occurs; distinct frequencies are not needed for diagonalMM\.

*R6\.*By linearity,W⁡\(γ⊙n⁡\(x\)\+β\)=W​diag⁡\(γ\)​n​\(x\)\+W​βW\(\\gamma\\odot n\(x\)\+\\beta\)=W\\diag\(\\gamma\)\\,n\(x\)\+W\\beta\. The first term is a linear box applied to the normalised wire, and the constantW​βW\\betais absorbed by a bias slot,b↦b\+W​βb\\mapsto b\+W\\beta; an RMSNorm gain hasβ=0\\beta=0\.

*R7\.*WithΔ⁡\(x\)=\(x,x\)\\Delta\(x\)=\(x,x\)the copy map,Δ∘D=\(D×D\)∘Δ\\Delta\\circ D=\(D\\times D\)\\circ\\Deltafor every mapDD: the naturality of the cartesian diagonal \([Background B\.4](https://arxiv.org/html/2610.12728#A1.SSx2.SSSx1)\)\.

\(a\) LetEq​\(x\)=Aq​x\+cqE\_\{q\}\(x\)=A\_\{q\}x\+c\_\{q\}act at every position onww\. By R7 applyingEqE\_\{q\}before the copy node that feeds the readers equals applying it on each reader’s input\. For a readerccwith weightWcW\_\{c\}and biasbcb\_\{c\},Wc​\(Aq​x\+cq\)\+bc=\(Wc​Aq\)​x\+\(Wc​cq\+bc\)W\_\{c\}\(A\_\{q\}x\+c\_\{q\}\)\+b\_\{c\}=\(W\_\{c\}A\_\{q\}\)x\+\(W\_\{c\}c\_\{q\}\+b\_\{c\}\), which lies in the class ofccby the closure hypothesis \(for a linear class this requirescq=0c\_\{q\}=0, and for a shift a bias slot\)\. The readers are unshared, so their weights can be changed independently; andEqE\_\{q\}is the same at every position, so one weight serves all positions \(R1\)\. Settingμ⁡\(q\)\\mu\(q\)to the base weights withWc↦Wc​AqW\_\{c\}\\mapsto W\_\{c\}A\_\{q\}andbc↦bc\+Wc​cqb\_\{c\}\\mapsto b\_\{c\}\+W\_\{c\}c\_\{q\}for every reader givesΦf∘μ=Φf~∘ρ\\Phi\_\{f\}\\circ\\mu=\\Phi\_\{\\tilde\{f\}\}\\circ\\rho\. Dually, ifwwis the output of an untied boxp⁡\(x\)=Wp​x\+bpp\(x\)=W\_\{p\}x\+b\_\{p\}andEq∘pE\_\{q\}\\circ plies in its class, replaceppbyEq∘pE\_\{q\}\\circ p, that is,Wp↦Aq​WpW\_\{p\}\\mapsto A\_\{q\}W\_\{p\}andbp↦Aq​bp\+cqb\_\{p\}\\mapsto A\_\{q\}b\_\{p\}\+c\_\{q\}\(R1\)\. A merge must land in the base networkΘ\\Theta, so absorption into a box that only the extension contains is not a merge\. On the reader side the rules have nothing to absorb an edit read by a skip connection, a norm or the network output, since those readers carry no trainable weight, and R1 does not fuse a position\-selective edit into a weight that acts at every position\. A position\-selective edit modifies the box function at some positions only; for parameter values at which it is not the identity, the modified computation is not one map applied identically at every position, so it is not in the box’s class, and the edit is not box\-locally mergeable\.

*Residual\-stream cases\.*\(i\) Ifwwis the output of an untied embedding tableℰ\\mathcal\{E\}, the dual case applies: replacing each rowe⁡\(v\)e\(v\)byAq​e​\(v\)\+cqA\_\{q\}e\(v\)\+c\_\{q\}absorbs a position\-uniform affine edit\. For an additive edith↦h\+ch\\mapsto h\+cat one fixed positionp0p\_\{0\}of layer00in a model with a learned absolute positional table, replacing rowp0p\_\{0\}of that table by itself plusccabsorbs it\. \(ii\) A shifth↦h\+th\\mapsto h\+tof the stream after a block whose last box has a biasbbis absorbed byb↦b\+tb\\mapsto b\+t, since the block adds its output to the stream\. \(iii\) In a post\-LN model the normalised wire is the stream,h=γ⊙n⁡\(x\)\+βh=\\gamma\\odot n\(x\)\+\\beta, and a diagonal edit givess⊙h\+t=\(s⊙γ\)⊙n⁡\(x\)\+\(s⊙β\+t\)s\\odot h\+t=\(s\\odot\\gamma\)\\odot n\(x\)\+\(s\\odot\\beta\+t\), absorbed byγ↦s⊙γ\\gamma\\mapsto s\\odot\\gamma,β↦s⊙β\+t\\beta\\mapsto s\\odot\\beta\+t\. \(iv\) In an untied pre\-RMSNorm model, let blockjjmap the stream byh↦h\+Fj​\(h\)h\\mapsto h\+F\_\{j\}\(h\)withFj​\(h\)=Wout\(j\)​Gj​\(Win\(j\)​diag⁡\(γj\)​n​\(h\)\)F\_\{j\}\(h\)=W^\{\(j\)\}\_\{\\rm out\}\\,G\_\{j\}\\big\(W^\{\(j\)\}\_\{\\rm in\}\\diag\(\\gamma\_\{j\}\)\\,n\(h\)\\big\), wheren⁡\(h\)=h/rms⁡\(h\)n\(h\)=h/\\mathrm\{rms\}\(h\),GjG\_\{j\}is the internal computation \(attention with its positional encoding applied after the input projections, or the gated MLP\), andWin\(j\)W^\{\(j\)\}\_\{\\rm in\}collects the input projections\. LetQQbe orthogonal and the edit beh↦Q​hh\\mapsto Qhafter blockll\. Replace the embedding rowse⁡\(v\)e\(v\)byQ​e​\(v\)Qe\(v\), and forj≤lj\\leq lreplaceWin\(j\)W^\{\(j\)\}\_\{\\rm in\}byWin\(j\)​diag⁡\(γj\)​Q⊤W^\{\(j\)\}\_\{\\rm in\}\\diag\(\\gamma\_\{j\}\)Q^\{\\top\},γj\\gamma\_\{j\}by𝟏\\mathbf\{1\}andWout\(j\)W^\{\(j\)\}\_\{\\rm out\}byQ​Wout\(j\)QW^\{\(j\)\}\_\{\\rm out\}\(and any biasbbof an output projection byQ​bQb\)\. Sincerms⁡\(Q​h\)=rms⁡\(h\)\\mathrm\{rms\}\(Qh\)=\\mathrm\{rms\}\(h\),n⁡\(Q​h\)=Q​n​\(h\)n\(Qh\)=Qn\(h\), and by induction the new streams areQ​hjQh\_\{j\}forj≤lj\\leq l:Q​hj−1\+Q​Wout\(j\)​Gj​\(Win\(j\)​diag⁡\(γj\)​Q⊤​Q​n​\(hj−1\)\)=Q⁡\(hj−1\+Fj​\(hj−1\)\)Qh\_\{j\-1\}\+QW^\{\(j\)\}\_\{\\rm out\}G\_\{j\}\\big\(W^\{\(j\)\}\_\{\\rm in\}\\diag\(\\gamma\_\{j\}\)Q^\{\\top\}Qn\(h\_\{j\-1\}\)\\big\)=Q\\big\(h\_\{j\-1\}\+F\_\{j\}\(h\_\{j\-1\}\)\\big\)\. After blockllthe new network carriesQ​hlQh\_\{l\}, as the edited one does, and the later blocks are unchanged, so the outputs agree\. This is the construction of[Ashkboos et al\. \(2024a\)](https://arxiv.org/html/2610.12728#bib.bib18);[Ashkboos et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib19)\.

\(b\) The rules used along the path are identities of maps under their side conditions: R3 needsℓ≥0\\ell\\geq 0, the slide of R5 under RoPE needsℓ\\ellconstant on the rotary pairs, and R6 needs a bias slot for the shift\. Applying them moves every copy ofdiag⁡\(ℓ\)\\diag\(\\ell\)next to a trainable box, and R1 fuses it; this is \(a\) applied at the box reached\. An unshared target can absorb its copy independently, and a target shared across a group absorbs a single matrix, which is possible for every member exactly whenℓ\\ellis constant across the group\. For the converse under RoPE, suppose the query scalingD=diag⁡\(ℓ\)D=\\diag\(\\ell\)merges into someWK′W\_\{K\}^\{\\prime\}withWQW\_\{Q\}unchanged\. Equality of all logits for all inputs givesx⊤​WQ⊤​D​RΔ​WK​x′=x⊤​WQ⊤​RΔ​WK′​x′x^\{\\top\}W\_\{Q\}^\{\\top\}DR\_\{\\Delta\}W\_\{K\}x^\{\\prime\}=x^\{\\top\}W\_\{Q\}^\{\\top\}R\_\{\\Delta\}W\_\{K\}^\{\\prime\}x^\{\\prime\}for every occurring offsetΔ=n−m\\Delta=n\-m, soWQ⊤​\(RΔ​WK′−D​RΔ​WK\)=0W\_\{Q\}^\{\\top\}\(R\_\{\\Delta\}W\_\{K\}^\{\\prime\}\-DR\_\{\\Delta\}W\_\{K\}\)=0\. IfWQW\_\{Q\}has full row rank,WQ⊤W\_\{Q\}^\{\\top\}is injective andWK′=R−Δ​D​RΔ​WKW\_\{K\}^\{\\prime\}=R\_\{\-\\Delta\}DR\_\{\\Delta\}W\_\{K\}for every occurringΔ\\Delta; ifWKW\_\{K\}has full row rank too, it has a right inverse, andR−Δ​D​RΔR\_\{\-\\Delta\}DR\_\{\\Delta\}is one matrix for all occurringΔ\\Delta\. On a rotary pairjj, letΔ\\DeltaandΔ′\\Delta^\{\\prime\}be occurring offsets with\(Δ−Δ′\)​ωj∉π​ℤ\(\\Delta\-\\Delta^\{\\prime\}\)\\omega\_\{j\}\\notin\\pi\\mathbb\{Z\}; thenDDcommutes withRΔ−Δ′R\_\{\\Delta\-\\Delta^\{\\prime\}\}on that pair, and by the computation in R5,ℓ\\ellis constant on it\. In causal attentionΔ′=0\\Delta^\{\\prime\}=0occurs \(a query attends to its own position\), so it suffices that one occurring offset has an angle on pairjjthat is not a multiple ofπ\\pi, which the hypothesis provides pair by pair\. The genericity hypotheses are needed\. IfWQW\_\{Q\}has a dead pair \(both rows zero\) andℓ\\ellis constant on every other pair, thenWQ⊤​D​RΔ=WQ⊤​RΔ​DW\_\{Q\}^\{\\top\}DR\_\{\\Delta\}=W\_\{Q\}^\{\\top\}R\_\{\\Delta\}D, since both sides vanish on the dead pair andDDcommutes withRΔR\_\{\\Delta\}on the others, soWK′=D​WKW\_\{K\}^\{\\prime\}=DW\_\{K\}works withℓ\\ellarbitrary on the dead pair\. Under ReLU,diag⁡\(ℓ\)​ReLU​\(W​x\)=ReLU⁡\(diag⁡\(ℓ\)​W​x\)\\diag\(\\ell\)\\mathrm\{ReLU\}\(Wx\)=\\mathrm\{ReLU\}\(\\diag\(\\ell\)Wx\)holds for everyℓ≥0\\ell\\geq 0, zero entries included\.

\(c\) For a linear or affine box the class is the linear or affine maps, so a non\-affine modified box function is outside it, which is the definition of failing box\-local mergeability\.

\(d\) is[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(d\)\. ∎

##### The corollaries after[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)

*Weight\-space methods*merge byμ=ρ\\mu=\\rho\(§[C\.45](https://arxiv.org/html/2610.12728#A3.SS45)\); their nonlinearities act on the parameters\. If an embedding and an LM head share one tensor, a merge that changes one side must change the other or break the tie, which is why an adapter on one side is not merged faithfully when the tie is not maintained\.*\(IA\)3on keys and values*isdiag⁡\(ℓ\)​WK​x=\(diag⁡\(ℓ\)​WK\)​x\\diag\(\\ell\)W\_\{K\}x=\(\\diag\(\\ell\)W\_\{K\}\)xand likewise forWVW\_\{V\}\(R1\)\.*On the FFN*, applied to the input ofWdownW\_\{\\rm down\}, it isWdown​diag⁡\(ℓ\)​h=\(Wdown​diag⁡\(ℓ\)\)​hW\_\{\\rm down\}\\diag\(\\ell\)h=\(W\_\{\\rm down\}\\diag\(\\ell\)\)h\(R1\); neither step involvesσ\\sigma\.*IntoWupW\_\{\\rm up\}*: under ReLU, R3 givesdiag⁡\(ℓ\)​ReLU​\(Wup​x\)=ReLU⁡\(diag⁡\(ℓ\)​Wup​x\)\\diag\(\\ell\)\\mathrm\{ReLU\}\(W\_\{\\rm up\}x\)=\\mathrm\{ReLU\}\(\\diag\(\\ell\)W\_\{\\rm up\}x\)forℓ≥0\\ell\\geq 0, while an entryℓi<0\\ell\_\{i\}<0cannot fuse, sinceℓi​ReLU​\(wi⋅x\)≤0\\ell\_\{i\}\\mathrm\{ReLU\}\(w\_\{i\}\\cdot x\)\\leq 0and is negative somewhere, whereasReLU⁡\(wi′⋅x\)≥0\\mathrm\{ReLU\}\(w\_\{i\}^\{\\prime\}\\cdot x\)\\geq 0\. Under SwiGLU or GeGLU, R4 and R1 give \([8](https://arxiv.org/html/2610.12728#S6.E8)\) for everyℓ\\ell\.*Under GELU*, supposeℓi​GELU​\(wi⋅x\)=GELU⁡\(wi′⋅x\)\\ell\_\{i\}\\,\\mathrm\{GELU\}\(w\_\{i\}\\cdot x\)=\\mathrm\{GELU\}\(w\_\{i\}^\{\\prime\}\\cdot x\)for allxx, withwi≠0w\_\{i\}\\neq 0so that the pre\-activation ranges overℝ\\mathbb\{R\}\. Ifℓi=0\\ell\_\{i\}=0,wi′=0w\_\{i\}^\{\\prime\}=0works\. Ifℓi≠0\\ell\_\{i\}\\neq 0, the right side vanishes on the hyperplanewi⋅x=0w\_\{i\}\\cdot x=0, andGELU⁡\(t\)=t​Φ𝒩​\(t\)\\mathrm\{GELU\}\(t\)=t\\,\\Phi\_\{\\mathcal\{N\}\}\(t\)vanishes only att=0t=0, sowi′⋅x=0w\_\{i\}^\{\\prime\}\\cdot x=0there andwi′=c​wiw\_\{i\}^\{\\prime\}=cw\_\{i\}\. Thenℓi​GELU​\(t\)=GELU⁡\(c​t\)\\ell\_\{i\}\\,\\mathrm\{GELU\}\(t\)=\\mathrm\{GELU\}\(ct\)for alltt, and comparing the coefficientsa1=12a\_\{1\}=\\tfrac\{1\}\{2\}anda2=1/2​πa\_\{2\}=1/\\sqrt\{2\\pi\}\([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\(c\)\) givesc=ℓic=\\ell\_\{i\}andℓi=ℓi2\\ell\_\{i\}=\\ell\_\{i\}^\{2\}, soℓi=1\\ell\_\{i\}=1\. Hence a fusedWup′W\_\{\\rm up\}^\{\\prime\}exists iff everyℓi∈\{0,1\}\\ell\_\{i\}\\in\\\{0,1\\\}\.*A query scaling*fuses intoWQW\_\{Q\}by R1; ontoWKW\_\{K\}it needs R5’s slide, henceℓ\\ellconstant on the rotary pairs under RoPE \(necessary under the genericity hypotheses of \(b\)\), and with grouped\-query attention oneWKW\_\{K\}serves several query headshh, so the same argument with full\-rankWQ\(h\)W\_\{Q\}^\{\(h\)\}forcesWK′=Dh​WKW\_\{K\}^\{\\prime\}=D\_\{h\}W\_\{K\}for everyhhin the group, henceDhD\_\{h\}constant across the group whenWKW\_\{K\}has full row rank\.*SSF*after a boxW​x\+bWx\+bgivess⊙\(W​x\+b\)\+t=\(diag⁡\(s\)​W\)​x\+\(s⊙b\+t\)s\\odot\(Wx\+b\)\+t=\(\\diag\(s\)W\)x\+\(s\\odot b\+t\), which needs a bias slot whent≠0t\\neq 0\.*LN\-tuning*changes existing parameters, soμ\\muis the new parameter vector\.*Folding a LayerNorm*into the following boxes in a pre\-LN block, where the norm feeds only those boxes, is R7 followed by R6, which needs a bias slot forW​βW\\beta; an RMSNorm gain has noβ\\beta\. Folding a gain into a tied embedding or head changes the other side of the tie\.*Not box\-locally mergeable*: forσ\\sigmanon\-affine and nonzerouu,dd, the mapy↦y\+u​σ​\(d⊤​y\)y\\mapsto y\+u\\,\\sigma\(d^\{\\top\}y\)is non\-affine, so a serial Houlsby or Pfeiffer adapter after a boxWWwithd⊤​W≠0d^\{\\top\}W\\neq 0, and a parallel adapterx↦W​x\+u​σ​\(d⊤​x\)x\\mapsto Wx\+u\\,\\sigma\(d^\{\\top\}x\), modify the box function non\-affinely for some parameter value; a routed mixturex↦W​x\+∑ege​\(x\)​Be​Ae​xx\\mapsto Wx\+\\sum\_\{e\}g\_\{e\}\(x\)B\_\{e\}A\_\{e\}xwith a softmax gate is real\-analytic in its input and parameters and non\-affine at a witness \(two experts withB1​A1=IB\_\{1\}A\_\{1\}=I,B2​A2=0B\_\{2\}A\_\{2\}=0and gate logits differing byγ⊤​x\\gamma^\{\\top\}x,γ≠0\\gamma\\neq 0, giveW​x\+g⁡\(γ⊤​x\)​xWx\+g\(\\gamma^\{\\top\}x\)\\,xwithggthe logistic function, non\-affine along any line on whichγ⊤​x\\gamma^\{\\top\}xvaries\), hence non\-affine for generic parameters \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\), and linear, so mergeable, when the gate is constant; AdapterFusion with a constant gate is a fixed combination of nonlinear adapters, still non\-affine; LST adds the output of a nonlinear side network; and prefix\-type methods are treated in[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(f\)\. In each case \(c\) applies\. For LLaMA\-Adapter with a two\-token promptp1,p2p\_\{1\},p\_\{2\}and gateg≠0g\\neq 0, the prompt pairs areki=Wk​pi\+bkk\_\{i\}=W\_\{k\}p\_\{i\}\+b\_\{k\}andvi=Wv​pi\+bvv\_\{i\}=W\_\{v\}p\_\{i\}\+b\_\{v\}, and a head on a length\-one context computesWo​\(Wv​x\+bv\)\+g​Wo​Attnq⁡\(x\)​\(P\)\+boW\_\{o\}\(W\_\{v\}x\+b\_\{v\}\)\+g\\,W\_\{o\}\\mathrm\{Attn\}\_\{q\(x\)\}\(P\)\+b\_\{o\}withAttnq​\(P\)=v2\+logit−1⁡\(⟨q,Wk​δ⟩/d\)​Wv​δ\\mathrm\{Attn\}\_\{q\}\(P\)=v\_\{2\}\+\\operatorname\{logit\}^\{\-1\}\\big\(\\langle q,W\_\{k\}\\delta\\rangle/\\sqrt\{d\}\\big\)\\,W\_\{v\}\\delta,δ=p1−p2\\delta=p\_\{1\}\-p\_\{2\}\. IfWq⊤​Wk≠0W\_\{q\}^\{\\top\}W\_\{k\}\\neq 0andWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0, chooseuuandδ\\deltawith⟨Wq​u,Wk​δ⟩≠0\\langle W\_\{q\}u,W\_\{k\}\\delta\\rangle\\neq 0andWo​Wv​δ≠0W\_\{o\}W\_\{v\}\\delta\\neq 0\(each condition fails only on a proper subspace or a nowhere\-dense set\)\. Alongx=t​ux=tuthe logistic factor is a bounded, non\-constant real\-analytic function oftt, hence not affine, so the head is not affine\. By[Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)the same holds for generic prompts with two or more tokens, which is the entry for LLaMA\-Adapter in the summary table of Exposé[VI](https://arxiv.org/html/2610.12728#S6); with one prompt token,Attnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(P\)is constant and the head stays affine\.

### C\.48 Proposition VI\.4

Proposition VI\.4\(linear insertions, and ReFT\)\.\(a\) A linear adapter in parallel with a single linear box,W0​x\+s​U​D​xW\_\{0\}x\+sUDx,*is*LoRA: it has the sameρ\\rho\. A linear adapter parallel to a whole sublayer \(He et al\.’s parallel adapter around attention or the FFN\) edits the weightless skip connection rather than a base box, and is not box\-locally mergeable when the sublayer is non\-affine\.\(b\) LetW0∈ℝm×nW\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}be bias\-free \(as LLaMA\-typeWoW\_\{o\}andWdownW\_\{\\mathrm\{down\}\}are\), andr≥1r\\geq 1\. A serial linear adapter afterW0W\_\{0\},\(I\+U​D\)​W0\(I\+UD\)W\_\{0\}withU∈ℝm×rU\\in\\mathbb\{R\}^\{m\\times r\}andD∈ℝr×mD\\in\\mathbb\{R\}^\{r\\times m\}\(scale absorbed\), has imageW0\+\{Y∈ℳ≤r:row⁡Y⊆row⁡W0\}W\_\{0\}\+\\\{Y\\in\\mathcal\{M\}\_\{\\leq r\}:\\operatorname\{row\}Y\\subseteq\\operatorname\{row\}W\_\{0\}\\\}\. It is expressively equivalent to LoRA iffrank⁡W0=n\\rk W\_\{0\}=n, and weakly \(unpointed\) isomorphic to LoRA iffW0W\_\{0\}is invertible\. The isomorphism\(U,D\)↦\(U,D​W0\)\(U,D\)\\mapsto\(U,DW\_\{0\}\)sends a base point\(U0,D0\)\(U\_\{0\},D\_\{0\}\)to\(U0,D0​W0\)\(U\_\{0\},D\_\{0\}W\_\{0\}\)\. For a fixedTA\\mathrm\{T\}\_\{A\}pointing\(0,A0\)\(0,A\_\{0\}\)withrank⁡A0=r\\rk A\_\{0\}=rand1≤r<m=n1\\leq r<m=n, a pointed isomorphism exists iffW0W\_\{0\}is invertible androw⁡A0=row⁡\(D0​W0\)\\operatorname\{row\}A\_\{0\}=\\operatorname\{row\}\(D\_\{0\}W\_\{0\}\), and thenU0=0U\_\{0\}=0automatically\. With a biasb0b\_\{0\}, the serial adapter also moves the bias byU​D​b0UDb\_\{0\}, so on the affine box its image differs from weight\-only LoRA’s\. Houlsby’s near\-identity initialisation \(weights from a zero\-mean Gaussian with standard deviation10−210^\{\-2\}, truncated at two standard deviations\) makes practical serial adapters slightly unpointed; the pointed statement applies to zero\-up initialisations\.\(c\) LoReFT isΦ⁡\(h\)=h\+R⊤​\(W​h\+b−R​h\)\\Phi\(h\)=h\+R^\{\\top\}\(Wh\+b\-Rh\), withR∈ℝr×dR\\in\\mathbb\{R\}^\{r\\times d\}having orthonormal rows, applied at selected positions\. On one hidden vector it is the extension by anidentity boxfollowed by a rank\-≤r\\leq raffine edit,Φ⁡\(h\)=\(I\+R⊤​\(W−R\)\)​h\+R⊤​b,\\Phi\(h\)=\\big\(I\+R^\{\\top\}\(W\-R\)\\big\)h\+R^\{\\top\}b,which is LoRAron the*affine*identity box\[Id∣0\]\[I\_\{d\}\\mid 0\]in homogeneous coordinates, withB=R⊤B=R^\{\\top\}constrained to the Stiefel manifold andA=\[W−R∣b\]A=\[W\-R\\mid b\]\. Its image is\(I,0\)\+ℳ≤r​\(d×\(d\+1\)\)\(I,0\)\+\\mathcal\{M\}\_\{\\leq r\}\(d\\times\(d\+1\)\), of dimensionr⁡\(2​d\+1−r\)r\(2d\+1\-r\)\. The bias is confined tocol⁡B\\operatorname\{col\}B, so forr<dr<dthis is not LoRAr⊕\{\}\_\{r\}\{\}\\oplus\{\}BitFit, whose dimension is larger byd−rd\-r\. It is simulated, throughh⁡\(R,W,b\)=\(R⊤,\[W−R∣b\]\)h\(R,W,b\)=\(R^\{\\top\},\[W\-R\\mid b\]\), by LoRAron that affine box pointed at\(R0⊤,0\)\(R\_\{0\}^\{\\top\},0\), aTB\\mathrm\{T\}\_\{B\}pointing withcol⁡B0=row⁡R0\\operatorname\{col\}B\_\{0\}=\\operatorname\{row\}R\_\{0\}; this homogeneous\-coordinate LoRA has a bias column inAA, unlike HF LoRA\. It is not simulated byTA\\mathrm\{T\}\_\{A\}\-pointed LoRAr, since itsdimS1=r⁡\(d\+1\)\\dim S\_\{1\}=r\(d\+1\)exceedsr​drd\. It is not isomorphic to LoRAr: its parameter count is smaller byr⁡\(r\+1\)/2r\(r\+1\)/2, and its residual symmetry isO⁡\(r\)\\mathrm\{O\}\(r\), acting by\(R,W,b\)↦\(O​R,O​W,O​b\)\(R,W,b\)\\mapsto\(OR,OW,Ob\)\. It is pointed iffW=RW=Randb=0b=0\. The reference implementation \(pyreft\) initialisesWWandbbby the default nn\.Linear init, independently ofRR, so practical LoReFT has a pointing defectR⊤​\(\(W0−R0\)​h\+b0\)R^\{\\top\}\(\(W\_\{0\}\-R\_\{0\}\)h\+b\_\{0\}\): at initialisation it replaces theRR\-component ofhhby a random affine function, and the simulation above is then only a weak morphism\. pyreft also applies dropout to the whole output,hhincluded, so the training\-time map is notΦ\\Phi\.

\(a\) The adapted box computesx↦\(W0\+s​U​D\)​xx\\mapsto\(W\_\{0\}\+sUD\)x, soρ⁡\(U,D\)=W0\+s​U​D\\rho\(U,D\)=W\_\{0\}\+sUDonℝm×r×ℝr×n\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}, which is LoRA’sρ⁡\(B,A\)=W0\+s​B​A\\rho\(B,A\)=W\_\{0\}\+sBAunder\(U,D\)=\(B,A\)\(U,D\)=\(B,A\), with the zero\-up pointingU=0U=0\([He et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib165), Table 1\)\. A linear adapter around a whole sublayerFFgivesx↦x\+F⁡\(x\)\+s​U​D​xx\\mapsto x\+F\(x\)\+sUDx; the terms​U​D​xsUDxis added to the weightless skip connection, which no box of the base network computes, so there is no base box whose function the edit modifies\. For a non\-affine sublayer such asF⁡\(x\)=W2​σ​\(W1​x\)F\(x\)=W\_\{2\}\\sigma\(W\_\{1\}x\)withσ\\sigmanon\-affine, no base box both readsxxand writes linearly into the block’s output:W1W\_\{1\}’s output passes throughσ\\sigma, andW2W\_\{2\}readsσ⁡\(W1​x\)\\sigma\(W\_\{1\}x\)rather thanxx\. So no box ofFFcan absorbs​U​D​xsUDxwithin its class\. IfF⁡\(x\)=M​x\+cF\(x\)=Mx\+cwere affine,x\+\(M\+s​U​D\)​x\+cx\+\(M\+sUD\)x\+cwould fuse intoMM\.

\(b\)*Image\.*ρ⁡\(U,D\)=\(I\+U​D\)​W0=W0\+U​D​W0\\rho\(U,D\)=\(I\+UD\)W\_\{0\}=W\_\{0\}\+UDW\_\{0\}\. EachU​D​W0UDW\_\{0\}has rank at mostrrand its rows are combinations of the rows ofW0W\_\{0\}\. Conversely, ifrank⁡Y≤r\\rk Y\\leq randrow⁡Y⊆row⁡W0\\operatorname\{row\}Y\\subseteq\\operatorname\{row\}W\_\{0\}, thenY=Y​W0\+​W0Y=YW\_\{0\}^\{\+\}W\_\{0\}, sinceW0\+​W0W\_\{0\}^\{\+\}W\_\{0\}is the orthogonal projector ontorow⁡W0\\operatorname\{row\}W\_\{0\};X=Y​W0\+X=YW\_\{0\}^\{\+\}has rank at mostrr, soX=U​DX=UDwithU∈ℝm×rU\\in\\mathbb\{R\}^\{m\\times r\},D∈ℝr×mD\\in\\mathbb\{R\}^\{r\\times m\}, andY=U​D​W0Y=UDW\_\{0\}\.*Expressive equivalence\.*LoRA’s image isW0\+ℳ≤rW\_\{0\}\+\\mathcal\{M\}\_\{\\leq r\}\. Ifrow⁡W0=ℝn\\operatorname\{row\}W\_\{0\}=\\mathbb\{R\}^\{n\}the two images agree\. If not, pickv∉row⁡W0v\\notin\\operatorname\{row\}W\_\{0\}; thene1​v⊤∈ℳ≤1⊆ℳ≤re\_\{1\}v^\{\\top\}\\in\\mathcal\{M\}\_\{\\leq 1\}\\subseteq\\mathcal\{M\}\_\{\\leq r\}\(herer≥1r\\geq 1\) has a row outsiderow⁡W0\\operatorname\{row\}W\_\{0\}, so it lies in LoRA’s image minusW0W\_\{0\}but not in the adapter’s\.

*Weak isomorphism\.*IfW0W\_\{0\}is invertible, thenm=nm=nandh⁡\(U,D\)=\(U,D​W0\)h\(U,D\)=\(U,DW\_\{0\}\)is a linear bijectionℝm×r×ℝr×m→ℝm×r×ℝr×n\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times m\}\\to\\mathbb\{R\}^\{m\\times r\}\\times\\mathbb\{R\}^\{r\\times n\}withW0\+U⁡\(D​W0\)=ρ⁡\(U,D\)W\_\{0\}\+U\(DW\_\{0\}\)=\\rho\(U,D\)\(LoRA withs=1s=1, orUUrescaled\), so it is an isomorphism overΘ\\Theta; it sends\(U0,D0\)\(U\_\{0\},D\_\{0\}\)to\(U0,D0​W0\)\(U\_\{0\},D\_\{0\}W\_\{0\}\)\. Conversely, a weak isomorphism is a diffeomorphismhhwithρLoRA∘h=ρ\\rho\_\{\\mathrm\{LoRA\}\}\\circ h=\\rho, so the images agree, which forcesrank⁡W0=n\\rk W\_\{0\}=n\([Observation I\.4](https://arxiv.org/html/2610.12728#S1.SS0.SSSx3)\(c\)\), and the dimensions agree,2​m​r=r⁡\(m\+n\)2mr=r\(m\+n\), which forcesm=nm=n; soW0W\_\{0\}is square of full rank\.

*Pointed isomorphism\.*Let1≤r<m=n1\\leq r<m=n,rank⁡A0=r\\rk A\_\{0\}=r, and let the adapter be pointed at\(U0,D0\)\(U\_\{0\},D\_\{0\}\), that is,U0​D0​W0=0U\_\{0\}D\_\{0\}W\_\{0\}=0\. “If”:W0W\_\{0\}invertible givesU0​D0=0U\_\{0\}D\_\{0\}=0\. Sincerow⁡\(D0​W0\)=row⁡A0\\operatorname\{row\}\(D\_\{0\}W\_\{0\}\)=\\operatorname\{row\}A\_\{0\}has dimensionrr,D0​W0D\_\{0\}W\_\{0\}and henceD0D\_\{0\}have rankrr, soD0D\_\{0\}has a right inverse andU0=0U\_\{0\}=0\. Twor×nr\\times nmatrices of rankrrwith the same row space differ by an invertible left factor, soA0=g​D0​W0A\_\{0\}=gD\_\{0\}W\_\{0\}for someg∈GLrg\\in\\mathrm\{GL\}\_\{r\}\. The map\(U,D\)↦\(U​g−1,g​D​W0\)\(U,D\)\\mapsto\(Ug^\{\-1\},gDW\_\{0\}\)is a linear bijection overΘ\\Thetathat sends\(U0,D0\)=\(0,D0\)\(U\_\{0\},D\_\{0\}\)=\(0,D\_\{0\}\)to\(0,A0\)\(0,A\_\{0\}\)\. “Only if”: a pointed isomorphism is a weak one, soW0W\_\{0\}is invertible, and it preservesS1S\_\{1\}\([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\)\. Differentiating,

S1​\(adapter at​\(U0,D0\)\)\\displaystyle S\_\{1\}\(\\text\{adapter at \}\(U\_\{0\},D\_\{0\}\)\)=\{\(X​D0\+U0​Y\)​W0\}=\{X​D0​W0\+U0​Y′\},\\displaystyle=\\\{\(XD\_\{0\}\+U\_\{0\}Y\)W\_\{0\}\\\}=\\\{XD\_\{0\}W\_\{0\}\+U\_\{0\}Y^\{\\prime\}\\\},S1​\(LoRA​at​\(0,A0\)\)\\displaystyle S\_\{1\}\(\\mathrm\{LoRA\}\\text\{ at \}\(0,A\_\{0\}\)\)=\{X​A0\},\\displaystyle=\\\{XA\_\{0\}\\\},withX∈ℝm×rX\\in\\mathbb\{R\}^\{m\\times r\}andY′=Y​W0Y^\{\\prime\}=YW\_\{0\}ranging overℝr×n\\mathbb\{R\}^\{r\\times n\}becauseW0W\_\{0\}is invertible; and\{X​A0\}=\{Z:row⁡Z⊆row⁡A0\}\\\{XA\_\{0\}\\\}=\\\{Z:\\operatorname\{row\}Z\\subseteq\\operatorname\{row\}A\_\{0\}\\\}\. If\(U0\)i​j≠0\(U\_\{0\}\)\_\{ij\}\\neq 0, choosev∉row⁡A0v\\notin\\operatorname\{row\}A\_\{0\}, possible sincer<nr<n, andY′=ej​v⊤Y^\{\\prime\}=e\_\{j\}v^\{\\top\}: rowiiofU0​Y′U\_\{0\}Y^\{\\prime\}is\(U0\)i​j​v⊤\(U\_\{0\}\)\_\{ij\}v^\{\\top\}, outsiderow⁡A0\\operatorname\{row\}A\_\{0\}, a contradiction\. SoU0=0U\_\{0\}=0, and\{X​D0​W0\}=\{X​A0\}\\\{XD\_\{0\}W\_\{0\}\\\}=\\\{XA\_\{0\}\\\}givesrow⁡\(D0​W0\)=row⁡A0\\operatorname\{row\}\(D\_\{0\}W\_\{0\}\)=\\operatorname\{row\}A\_\{0\}\.

*Bias\.*\(I\+U​D\)​\(W0​x\+b0\)=\(W0\+U​D​W0\)​x\+\(b0\+U​D​b0\)\(I\+UD\)\(W\_\{0\}x\+b\_\{0\}\)=\(W\_\{0\}\+UDW\_\{0\}\)x\+\(b\_\{0\}\+UDb\_\{0\}\)\. On the affine box the adapter’s image contains maps with biasb0\+U​D​b0≠b0b\_\{0\}\+UDb\_\{0\}\\neq b\_\{0\}wheneverb0≠0b\_\{0\}\\neq 0\(chooseU​DUDwithU​D​b0≠0UDb\_\{0\}\\neq 0\), while weight\-only LoRA keeps the biasb0b\_\{0\}\.*Houlsby initialisation\.*With both factors drawn at random,U0​D0​W0≠0U\_\{0\}D\_\{0\}W\_\{0\}\\neq 0almost surely, so the initial weight missesW0W\_\{0\}\.

\(c\)*Algebra\.*Expanding,Φ⁡\(h\)=h\+R⊤​\(W−R\)​h\+R⊤​b=\(I\+R⊤​\(W−R\)\)​h\+R⊤​b\\Phi\(h\)=h\+R^\{\\top\}\(W\-R\)h\+R^\{\\top\}b=\\big\(I\+R^\{\\top\}\(W\-R\)\\big\)h\+R^\{\\top\}b\. In homogeneous coordinates\(h,1\)\(h,1\)an affine maph↦M​h\+ch\\mapsto Mh\+cis the matrix\[M∣c\]∈ℝd×\(d\+1\)\[M\\mid c\]\\in\\mathbb\{R\}^\{d\\times\(d\+1\)\}, the identity box is\[Id∣0\]\[I\_\{d\}\\mid 0\], and

\[I\+R⊤​\(W−R\)\|R⊤​b\]=\[Id∣0\]\+R⊤​\[W−R∣b\]=\[Id∣0\]\+B​A,B=R⊤,A=\[W−R∣b\]\.\\big\[\\,I\+R^\{\\top\}\(W\-R\)\\ \\big\|\\ R^\{\\top\}b\\,\\big\]=\[I\_\{d\}\\mid 0\]\+R^\{\\top\}\\,\[W\-R\\mid b\]=\[I\_\{d\}\\mid 0\]\+BA,\\qquad B=R^\{\\top\},\\ A=\[W\-R\\mid b\]\.*Image\.*As\(W,b\)\(W,b\)ranges freely,V=\[W−R∣b\]V=\[W\-R\\mid b\]ranges over all ofℝr×\(d\+1\)\\mathbb\{R\}^\{r\\times\(d\+1\)\}, so the image is\[Id∣0\]\+\{R⊤​V\}\[I\_\{d\}\\mid 0\]\+\\\{R^\{\\top\}V\\\}over StiefelR⊤R^\{\\top\}\. Every matrix of rank at mostrrfactors asR⊤​VR^\{\\top\}VwithR⊤R^\{\\top\}having orthonormal columns \([Auxiliary result B\.11](https://arxiv.org/html/2610.12728#A2.SS0.SSSx3)\(a\)\), so the image is\[Id∣0\]\+ℳ≤r​\(d×\(d\+1\)\)\[I\_\{d\}\\mid 0\]\+\\mathcal\{M\}\_\{\\leq r\}\(d\\times\(d\+1\)\), of dimensionr⁡\(d\+\(d\+1\)−r\)=r⁡\(2​d\+1−r\)r\(d\+\(d\+1\)\-r\)=r\(2d\+1\-r\)\([Background C\.12](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx3)\)\. The biasR⊤​bR^\{\\top\}blies incol⁡R⊤=col⁡B\\operatorname\{col\}R^\{\\top\}=\\operatorname\{col\}B\. LoRAr⊕\{\}\_\{r\}\{\}\\oplus\{\}BitFit has image\{\[Id\+Y∣c\]:rankY≤r,c∈ℝd\}\\\{\[I\_\{d\}\+Y\\mid c\]:\\rk Y\\leq r,\\ c\\in\\mathbb\{R\}^\{d\}\\\}, of dimensionr⁡\(2​d−r\)\+dr\(2d\-r\)\+d, which exceedsr⁡\(2​d\+1−r\)r\(2d\+1\-r\)byd−rd\-r; forr<dr<dthe images differ\.*Simulation\.*The maph⁡\(R,W,b\)=\(R⊤,\[W−R∣b\]\)h\(R,W,b\)=\(R^\{\\top\},\[W\-R\\mid b\]\)is smooth, and LoRAron the affine box evaluated ath⁡\(R,W,b\)h\(R,W,b\)is\[Id∣0\]\+R⊤​\[W−R∣b\]=Φ\[I\_\{d\}\\mid 0\]\+R^\{\\top\}\[W\-R\\mid b\]=\\Phi, sohhis a morphism to that LoRAr\. It sends the neutral point\(R0,R0,0\)\(R\_\{0\},R\_\{0\},0\)to\(R0⊤,0\)\(R\_\{0\}^\{\\top\},0\), a pointing in the stratumTB\\mathrm\{T\}\_\{B\}\(A0=0A\_\{0\}=0,B0B\_\{0\}of rankrr\) withcol⁡B0=row⁡R0\\operatorname\{col\}B\_\{0\}=\\operatorname\{row\}R\_\{0\}\.*S1S\_\{1\}\.*At the neutral point\[W−R∣b\]=0\[W\-R\\mid b\]=0, so the derivative ofR⊤​\[W−R∣b\]R^\{\\top\}\[W\-R\\mid b\]isR0⊤​\[W˙−R˙∣b˙\]R\_\{0\}^\{\\top\}\[\\dot\{W\}\-\\dot\{R\}\\mid\\dot\{b\}\], and as\(W˙,b˙\)\(\\dot\{W\},\\dot\{b\}\)range freelyS1=\{R0⊤​V\}S\_\{1\}=\\\{R\_\{0\}^\{\\top\}V\\\}, of dimensionr⁡\(d\+1\)r\(d\+1\)becauseR0⊤R\_\{0\}^\{\\top\}is injective\. LoRArpointed at\(0,A0\)\(0,A\_\{0\}\)withrank⁡A0=r\\rk A\_\{0\}=rhasS1=\{X​A0\}S\_\{1\}=\\\{XA\_\{0\}\\\}, of dimensiond​rdr\. A pointed morphism would give an inclusion ofS1S\_\{1\}’s \([Observation I\.6](https://arxiv.org/html/2610.12728#S1.SS0.SSSx4)\), impossible sincer⁡\(d\+1\)\>r​dr\(d\+1\)\>rd\.

*Not isomorphic\.*The parameter space is the Stiefel manifold of orthonormalrr\-frames inℝd\\mathbb\{R\}^\{d\}, of dimensionr​d−r⁡\(r\+1\)/2rd\-r\(r\+1\)/2, timesℝr×d×ℝr\\mathbb\{R\}^\{r\\times d\}\\times\\mathbb\{R\}^\{r\}, of total dimension2​r​d\+r−r⁡\(r\+1\)/22rd\+r\-r\(r\+1\)/2; LoRAron the affine box has dimensionr⁡\(2​d\+1\)r\(2d\+1\)\. The differencer⁡\(r\+1\)/2r\(r\+1\)/2is positive, so no diffeomorphism, hence no isomorphism, exists\.*Symmetry\.*ForO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\),O​RORhas orthonormal rows and\(O​R\)⊤​\(O​W−O​R\)=R⊤​\(W−R\)\(OR\)^\{\\top\}\(OW\-OR\)=R^\{\\top\}\(W\-R\),\(O​R\)⊤​O​b=R⊤​b\(OR\)^\{\\top\}Ob=R^\{\\top\}b, soΦ\\Phiis unchanged\. Conversely, ifR⊤​V=R′⁣⊤​V′R^\{\\top\}V=R^\{\\prime\\top\}V^\{\\prime\}withrank⁡V=r\\rk V=r, then the column space of the product iscol⁡R⊤=col⁡R′⁣⊤\\operatorname\{col\}R^\{\\top\}=\\operatorname\{col\}R^\{\\prime\\top\}, soR′=O​RR^\{\\prime\}=ORfor someO∈O⁡\(r\)O\\in\\mathrm\{O\}\(r\), and injectivity ofR⊤R^\{\\top\}givesV′=O​VV^\{\\prime\}=OV, that is,W′=O​WW^\{\\prime\}=OWandb′=O​bb^\{\\prime\}=Ob: the generic fibres are theO⁡\(r\)\\mathrm\{O\}\(r\)\-orbits, of dimensionr⁡\(r−1\)/2r\(r\-1\)/2, againstr2r^\{2\}for LoRA\.*Pointedness\.*Φ=id\\Phi=\\mathrm\{id\}iffR⊤​\[W−R∣b\]=0R^\{\\top\}\[W\-R\\mid b\]=0iffW=RW=Randb=0b=0, becauseR⊤R^\{\\top\}is injective\.

*The pyreft initialisation\.*At\(R0,W0,b0\)\(R\_\{0\},W\_\{0\},b\_\{0\}\)withW0W\_\{0\},b0b\_\{0\}drawn independently ofR0R\_\{0\},Φ0​\(h\)−h=R0⊤​\(\(W0−R0\)​h\+b0\)\\Phi\_\{0\}\(h\)\-h=R\_\{0\}^\{\\top\}\\big\(\(W\_\{0\}\-R\_\{0\}\)h\+b\_\{0\}\\big\), which is nonzero almost surely\. SinceR0​R0⊤=IrR\_\{0\}R\_\{0\}^\{\\top\}=I\_\{r\},R0​Φ0​\(h\)=R0​h\+\(W0​h\+b0−R0​h\)=W0​h\+b0R\_\{0\}\\Phi\_\{0\}\(h\)=R\_\{0\}h\+\(W\_\{0\}h\+b\_\{0\}\-R\_\{0\}h\)=W\_\{0\}h\+b\_\{0\}, while\(I−R0⊤​R0\)​Φ0​\(h\)=\(I−R0⊤​R0\)​h\(I\-R\_\{0\}^\{\\top\}R\_\{0\}\)\\Phi\_\{0\}\(h\)=\(I\-R\_\{0\}^\{\\top\}R\_\{0\}\)h: theR0R\_\{0\}\-component ofhhis replaced by the random affine functionW0​h\+b0W\_\{0\}h\+b\_\{0\}and the rest is kept\. The maphhsends this initial point to\(R0⊤,\[W0−R0∣b0\]\)\(R\_\{0\}^\{\\top\},\[W\_\{0\}\-R\_\{0\}\\mid b\_\{0\}\]\), not to the base point\(R0⊤,0\)\(R\_\{0\}^\{\\top\},0\), so it is a weak morphism only\. The remark on dropout describes the implementation\([Wu et al\., 2024d](https://arxiv.org/html/2610.12728#bib.bib505)\)\. ∎

### C\.49 Theorem VI\.5

Theorem VI\.5\(attention is a homomorphism followed by a perspective map; known in part\)\.Fix a queryqq\. Multisets of key–value pairs form a commutative monoid under⊎\\uplus\. DefineΣq​\(K\)=\(∑e⟨q,k⟩/d,∑e⟨q,k⟩/d​v\)\.\\Sigma\_\{q\}\(K\)=\\Big\(\\sum e^\{\\langle q,k\\rangle/\\sqrt\{d\}\},\\ \\sum e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\\,v\\Big\)\.\(a\)Σq\\Sigma\_\{q\}is a monoid homomorphism into\(ℝ≥0×ℝdv,\+\)\(\\mathbb\{R\}\_\{\\geq 0\}\\times\\mathbb\{R\}^\{d\_\{v\}\},\+\), landing in\{Z\>0\}∪\{\(0,0\)\}\\\{Z\>0\\\}\\cup\\\{\(0,0\)\\\}\. On nonempty multisets,Attnq=π∘Σq\\mathrm\{Attn\}\_\{q\}=\\pi\\circ\\Sigma\_\{q\}with the partial mapπ⁡\(Z,S\)=S/Z\\pi\(Z,S\)=S/Z, defined forZ\>0Z\>0\.\(b\)Prefix is a gated parallel adapter\.For nonemptyPPandCC,Attnq​\(P⊎C\)=\(1−λq\)​Attnq​\(C\)\+λq​Attnq​\(P\),λq=Zq​\(P\)Zq​\(P\)\+Zq​\(C\)\.\\mathrm\{Attn\}\_\{q\}\(P\\uplus C\)=\(1\-\\lambda\_\{q\}\)\\,\\mathrm\{Attn\}\_\{q\}\(C\)\+\\lambda\_\{q\}\\,\\mathrm\{Attn\}\_\{q\}\(P\),\\qquad\\lambda\_\{q\}=\\frac\{Z\_\{q\}\(P\)\}\{Z\_\{q\}\(P\)\+Z\_\{q\}\(C\)\}\.\(c\)Per\-layer pattern rigidity\.Within a single layer, for fixedqqand a fixed content multisetCC: the content attention weights are multiplied by1−λq1\-\\lambda\_\{q\}, their ratios are unchanged, and the output lies on the segment fromAttnq​\(C\)\\mathrm\{Attn\}\_\{q\}\(C\)toAttnq​\(P\)∈conv⁡VP\\mathrm\{Attn\}\_\{q\}\(P\)\\in\\operatorname\{conv\}V\_\{P\}, whereVPV\_\{P\}is the set of prefix values\. This rigidity is relative to the same layer’s computation without the prefix; it is relative to the pretrained pattern only at the lowest prefixed layer\. Lower\-layer prefixes change the queries and the content keys and values of higher layers, so attention patterns at the level of the network are*not*rigid\.\(d\)LLaMA\-AdapternormalisesPPandCCseparately and adds a zero\-initialised gate\. It computesAttnq​\(C\)\+tanh⁡\(gl\)​Attnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(C\)\+\\tanh\(g\_\{l\}\)\\,\\mathrm\{Attn\}\_\{q\}\(P\), with one gate per head in the paper; HF PEFT uses one raw zero\-initialised scalar per layer, withouttanh\\tanh\. It uses two perspective maps, leaves the content weights exactly unchanged, and is pointed atg=0g=0\. Its prompt keys and values are the frozenWk,WvW\_\{k\},W\_\{v\}applied to one learned prompt per adapted layer, and it acts in the top layers only\.\(e\)Prompt→\\todeep prefix in causal decoders\.Assume that the soft prompt comes first, that attention is causal, and that content positions are offset by the prompt lengthmmin both methods \(as HF PEFT does via cache\_position\)\. Then at every layer the states at prompt positions depend only onpp, andh:p↦\(Kj​\(p\),Vj​\(p\)\)jh:p\\mapsto\(K\_\{j\}\(p\),V\_\{j\}\(p\)\)\_\{j\}satisfiesΦ∘ρprefix∘h=Φ∘ρprompt\\Phi\\circ\\rho\_\{\\mathrm\{prefix\}\}\\circ h=\\Phi\\circ\\rho\_\{\\mathrm\{prompt\}\}as functions of the content tokens, read at content positions\. The two methods live on different extensions, so this is an arrow of the unpointed slice over the realisationC∞​\(Xcontent,Ycontent\)C^\{\\infty\}\(X\_\{\\mathrm\{content\}\},Y\_\{\\mathrm\{content\}\}\), not of𝐌𝐞𝐭𝐡\\mathbf\{Meth\}\. Under RoPE without the offset, a differenthhworks: rotate each prefix key by−m\-m\(positions−m,…,−1\-m,\\dots,\-1\)\. For additive relative biases \(ALiBi, T5\) the answer depends on how the implementation places the prefix keys\. The construction does not apply in bidirectional encoders of depth≥2\\geq 2\.\(f\)Obstructions\.Call a head*standard*if it is vanilla softmax attention without torch\-style add\_bias\_kv \(which would absorb a one\-token prefix\), add\_zero\_attn, learned attention sinks or softmax\-plus\-one\. On length\-one contexts, every weight setting of a standard head is affine inxx\. Some prefix makes the prefixed head non\-affine on length\-one contexts iffWo≠0W\_\{o\}\\neq 0and \(Wq≠0W\_\{q\}\\neq 0orWk⊤​bq≠0W\_\{k\}^\{\\top\}b\_\{q\}\\neq 0\); for bias\-free heads, iffWo≠0W\_\{o\}\\neq 0andWq≠0W\_\{q\}\\neq 0\. For such frozen weights and generic prefixes the prefixed head is not affine, because the gateλ⁡\(x\)\\lambda\(x\)is not constant\. So prefix tuning and P\-tuning v2 are not box\-locally mergeable into a standard head\. WhenWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0, no prefix is neutral, so prompt and prefix tuning are unpointed\.

Writeϵq​\(k\)=e⟨q,k⟩/d\>0\\epsilon\_\{q\}\(k\)=e^\{\\langle q,k\\rangle/\\sqrt\{d\}\}\>0, and for a multisetKKputZq​\(K\)=∑\(k,v\)∈Kϵq​\(k\)Z\_\{q\}\(K\)=\\sum\_\{\(k,v\)\\in K\}\\epsilon\_\{q\}\(k\)andSq​\(K\)=∑\(k,v\)∈Kϵq​\(k\)​vS\_\{q\}\(K\)=\\sum\_\{\(k,v\)\\in K\}\\epsilon\_\{q\}\(k\)v, soΣq​\(K\)=\(Zq​\(K\),Sq​\(K\)\)\\Sigma\_\{q\}\(K\)=\(Z\_\{q\}\(K\),S\_\{q\}\(K\)\)\.

\(a\) Sums over a multiset union split,Σq​\(K⊎K′\)=Σq​\(K\)\+Σq​\(K′\)\\Sigma\_\{q\}\(K\\uplus K^\{\\prime\}\)=\\Sigma\_\{q\}\(K\)\+\\Sigma\_\{q\}\(K^\{\\prime\}\), andΣq​\(∅\)=\(0,0\)\\Sigma\_\{q\}\(\\emptyset\)=\(0,0\), soΣq\\Sigma\_\{q\}is a monoid homomorphism\. Each term ofZqZ\_\{q\}is positive, soZq​\(K\)\>0Z\_\{q\}\(K\)\>0for nonemptyKKandZq​\(∅\)=0Z\_\{q\}\(\\emptyset\)=0\. By definition,Attnq​\(K\)=Sq​\(K\)/Zq​\(K\)=π⁡\(Σq​\(K\)\)\\mathrm\{Attn\}\_\{q\}\(K\)=S\_\{q\}\(K\)/Z\_\{q\}\(K\)=\\pi\(\\Sigma\_\{q\}\(K\)\)for nonemptyKK; thisπ\\piis the perspective map of[Background D\.34](https://arxiv.org/html/2610.12728#A1.SSx4.SSSx5)\.

\(b\) Withλ=ZP/\(ZP\+ZC\)\\lambda=Z\_\{P\}/\(Z\_\{P\}\+Z\_\{C\}\)andZP,ZC\>0Z\_\{P\},Z\_\{C\}\>0,

π⁡\(ZP\+ZC,SP\+SC\)=SP\+SCZP\+ZC=ZPZP\+ZC​SPZP\+ZCZP\+ZC​SCZC=λ​Attnq​\(P\)\+\(1−λ\)​Attnq​\(C\)\.\\pi\(Z\_\{P\}\+Z\_\{C\},S\_\{P\}\+S\_\{C\}\)=\\frac\{S\_\{P\}\+S\_\{C\}\}\{Z\_\{P\}\+Z\_\{C\}\}=\\frac\{Z\_\{P\}\}\{Z\_\{P\}\+Z\_\{C\}\}\\frac\{S\_\{P\}\}\{Z\_\{P\}\}\+\\frac\{Z\_\{C\}\}\{Z\_\{P\}\+Z\_\{C\}\}\\frac\{S\_\{C\}\}\{Z\_\{C\}\}=\\lambda\\,\\mathrm\{Attn\}\_\{q\}\(P\)\+\(1\-\\lambda\)\\,\\mathrm\{Attn\}\_\{q\}\(C\)\.
\(c\) InP⊎CP\\uplus Ca content pair\(k,v\)∈C\(k,v\)\\in Chas weightϵq​\(k\)/\(ZP\+ZC\)=\(1−λ\)​ϵq​\(k\)/ZC\\epsilon\_\{q\}\(k\)/\(Z\_\{P\}\+Z\_\{C\}\)=\(1\-\\lambda\)\\,\\epsilon\_\{q\}\(k\)/Z\_\{C\}, which is1−λ1\-\\lambdatimes its weight inCCalone; so all content weights scale by one factor and their ratios are unchanged\. By \(b\) the output lies on the segment fromAttnq​\(C\)\\mathrm\{Attn\}\_\{q\}\(C\)toAttnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(P\), withλ∈\(0,1\)\\lambda\\in\(0,1\), andAttnq​\(P\)=∑\(k,v\)∈P\(ϵq​\(k\)/ZP\)​v\\mathrm\{Attn\}\_\{q\}\(P\)=\\sum\_\{\(k,v\)\\in P\}\(\\epsilon\_\{q\}\(k\)/Z\_\{P\}\)\\,vis a convex combination of the prefix values\. At the lowest prefixed layer the query and the content pairs are computed from inputs that no prefix has touched, so they are those of the pretrained model, and the content weights are the pretrained ones times1−λq1\-\\lambda\_\{q\}\. At a higher layer the inputs come from lower layers whose outputs \(b\) has changed, soqqandCCthemselves differ from the pretrained ones, and the statement compares only with the same layer’s computation on those inputs\. The failure of network\-level rigidity is shown in[Petrov et al\. \(2023, §6\)](https://arxiv.org/html/2610.12728#bib.bib364), and universality of prefix\-tuning in[Petrov et al\. \(2024\)](https://arxiv.org/html/2610.12728#bib.bib365)\.

\(d\) LLaMA\-Adapter\([Zhang et al\., 2023h](https://arxiv.org/html/2610.12728#bib.bib562)\)computes the two softmaxes separately, so its output isπ⁡\(Σq​\(C\)\)\+tanh⁡\(gl\)​π​\(Σq​\(P\)\)\\pi\(\\Sigma\_\{q\}\(C\)\)\+\\tanh\(g\_\{l\}\)\\,\\pi\(\\Sigma\_\{q\}\(P\)\), with two perspective maps\. A content weight isϵq​\(k\)/Zq​\(C\)\\epsilon\_\{q\}\(k\)/Z\_\{q\}\(C\), the same as without the adapter\. Atg=0g=0,tanh⁡\(0\)=0\\tanh\(0\)=0\(and HF’s raw gate is00\), so the output isAttnq​\(C\)\\mathrm\{Attn\}\_\{q\}\(C\), the pretrained one, and the method is pointed there\. The remaining clauses describe how the method is built\.

\(e\) Number the positions of the prompt\-tuned model0,…,m−10,\\dots,m\-1for the soft promptp=\(p0,…,pm−1\)p=\(p\_\{0\},\\dots,p\_\{m\-1\}\)andm,…,m\+n−1m,\\dots,m\+n\-1for the content, and give the content the same positionsm,…,m\+n−1m,\\dots,m\+n\-1in the prefix\-tuned model\. LetKj​\(p\),Vj​\(p\)K\_\{j\}\(p\),V\_\{j\}\(p\)be the keys and values, after the positional encoding, that the prompt\-tuned model forms at the prompt positions in layerjj, as they are cached\. We show by induction over layersjjthat \(i\) the hidden states of the prompt\-tuned model at prompt positions depend only onpp, and \(ii\) the hidden states at content positions agree in the prompt\-tuned model and in the prefix\-tuned model with prefixh⁡\(p\)=\(Kj​\(p\),Vj​\(p\)\)jh\(p\)=\(K\_\{j\}\(p\),V\_\{j\}\(p\)\)\_\{j\}\. At the input, \(i\) holds trivially and \(ii\) holds because both models embed the same content tokens at the same positions\. Suppose \(i\) and \(ii\) hold at the input of layerjj\. Under causal masking a prompt position attends only to prompt positions, so its output depends only on prompt states, which gives \(i\) at the next layer\. A content positionttattends to the prompt positions and to the content positionsu≤tu\\leq t\. Its query, and the content keys and values, are computed from equal content states with equal positional encodings, by \(ii\)\. Its prompt keys and values areKj​\(p\),Vj​\(p\)K\_\{j\}\(p\),V\_\{j\}\(p\)in the prompt\-tuned model by definition, and they are supplied verbatim by the prefix in the other\. With rotary encodings, a score is⟨Rt​q,Rs​ks⟩\\langle R\_\{t\}q,R\_\{s\}k\_\{s\}\\ranglein both models, the cached keyRs​ksR\_\{s\}k\_\{s\}being the same object\. So the query and the multiset of key–value pairs are the same, and the attention outputs agree; the position\-wise sublayers then give \(ii\) at the next layer\. Reading the content positions at the top givesΦ∘ρprefix∘h=Φ∘ρprompt\\Phi\\circ\\rho\_\{\\rm prefix\}\\circ h=\\Phi\\circ\\rho\_\{\\rm prompt\}\. The prompt\-tuned model and the prefix\-tuned model are methods on different extensions of the base network \(input tokens in one, key–value slots in the other\), sohhis a map over the common realisation as functions of the content, an arrow of the slice overC∞​\(Xcontent,Ycontent\)C^\{\\infty\}\(X\_\{\\rm content\},Y\_\{\\rm content\}\)\.

Without the offset, the prefix\-tuned model places the content at positions0,…,n−10,\\dots,n\-1\. For RoPE puth′​\(p\)=\(R−m​Kj​\(p\),Vj​\(p\)\)jh^\{\\prime\}\(p\)=\\big\(R\_\{\-m\}K\_\{j\}\(p\),V\_\{j\}\(p\)\\big\)\_\{j\}, rotating the cached key from prompt positionsstos−ms\-m\. A content query at positiont−mt\-mthen scores⟨Rt−m​q,Rs−m​ks⟩=⟨q,Rs−t​ks⟩=⟨Rt​q,Rs​ks⟩\\langle R\_\{t\-m\}q,R\_\{s\-m\}k\_\{s\}\\rangle=\\langle q,R\_\{s\-t\}k\_\{s\}\\rangle=\\langle R\_\{t\}q,R\_\{s\}k\_\{s\}\\rangle, content–content scores depend only on differences of positions and are unchanged, and values carry no rotation, so the induction goes through withh′h^\{\\prime\}\. For additive relative biases the score of a content query against a prefix key depends on the distance the implementation assigns to that key, and the induction goes through exactly when these distances equalt−st\-s\. In a bidirectional encoder of depth at least22, prompt positions attend to content positions, so already the second layer’s prompt states depend on the content, and step \(i\) fails\.

\(f\)*Affine without prefix\.*On a context of one tokenxxthe softmax has one term, equal to11, so a standard head computesWo​\(Wv​x\+bv\)\+boW\_\{o\}\(W\_\{v\}x\+b\_\{v\}\)\+b\_\{o\}, which is affine inxxfor every weight setting\. \(With add\_zero\_attn, a sink or softmax\-plus\-one there is a second term, and this fails\.\)

*With a prefix\.*Let the prefix beP=\{\(ki,vi\)\}P=\\\{\(k\_\{i\},v\_\{i\}\)\\\}withNP≥1N\_\{P\}\\geq 1fixed pairs, and writeq⁡\(x\)=Wq​x\+bqq\(x\)=W\_\{q\}x\+b\_\{q\},k⁡\(x\)=Wk​x\+bkk\(x\)=W\_\{k\}x\+b\_\{k\},v⁡\(x\)=Wv​x\+bvv\(x\)=W\_\{v\}x\+b\_\{v\}, the content scoreς⁡\(x\)=⟨q⁡\(x\),k⁡\(x\)⟩/d\\varsigma\(x\)=\\langle q\(x\),k\(x\)\\rangle/\\sqrt\{d\}andZP​\(x\)=∑iϵq⁡\(x\)​\(ki\)Z\_\{P\}\(x\)=\\sum\_\{i\}\\epsilon\_\{q\(x\)\}\(k\_\{i\}\)\. By \(b\),

headP​\(x\)=Wo​\[\(1−λ⁡\(x\)\)​v​\(x\)\+λ⁡\(x\)​Attnq⁡\(x\)​\(P\)\]\+bo,λ⁡\(x\)=ZP​\(x\)ZP​\(x\)\+eς⁡\(x\)∈\(0,1\)\.\\mathrm\{head\}\_\{P\}\(x\)=W\_\{o\}\\big\[\(1\-\\lambda\(x\)\)\\,v\(x\)\+\\lambda\(x\)\\,\\mathrm\{Attn\}\_\{q\(x\)\}\(P\)\\big\]\+b\_\{o\},\\qquad\\lambda\(x\)=\\frac\{Z\_\{P\}\(x\)\}\{Z\_\{P\}\(x\)\+e^\{\\varsigma\(x\)\}\}\\in\(0,1\)\.\(28\)*Only if\.*IfWo=0W\_\{o\}=0the output is the constantbob\_\{o\}\. IfWq=0W\_\{q\}=0andWk⊤​bq=0W\_\{k\}^\{\\top\}b\_\{q\}=0, thenq=bqq=b\_\{q\}is constant, every prefix score⟨bq,ki⟩/d\\langle b\_\{q\},k\_\{i\}\\rangle/\\sqrt\{d\}is constant, and the content score⟨bq,Wk​x\+bk⟩/d=\(\(Wk⊤​bq\)⊤​x\+⟨bq,bk⟩\)/d\\langle b\_\{q\},W\_\{k\}x\+b\_\{k\}\\rangle/\\sqrt\{d\}=\\big\(\(W\_\{k\}^\{\\top\}b\_\{q\}\)^\{\\top\}x\+\\langle b\_\{q\},b\_\{k\}\\rangle\\big\)/\\sqrt\{d\}is constant; soλ\\lambdaandAttnq​\(P\)\\mathrm\{Attn\}\_\{q\}\(P\)are constant and the output is affine\.

*If\.*SupposeWo≠0W\_\{o\}\\neq 0and \(Wq≠0W\_\{q\}\\neq 0orWk⊤​bq≠0W\_\{k\}^\{\\top\}b\_\{q\}\\neq 0\)\. Take allNPN\_\{P\}prefix pairs equal to\(k1,v1\)\(k\_\{1\},v\_\{1\}\); thenAttnq⁡\(x\)​\(P\)=v1\\mathrm\{Attn\}\_\{q\(x\)\}\(P\)=v\_\{1\}andλ⁡\(x\)=logit−1⁡\(δ⁡\(x\)\)\\lambda\(x\)=\\operatorname\{logit\}^\{\-1\}\\big\(\\delta\(x\)\\big\)withδ⁡\(x\)=⟨q⁡\(x\),k1⟩/d\+log⁡NP−ς⁡\(x\)\\delta\(x\)=\\langle q\(x\),k\_\{1\}\\rangle/\\sqrt\{d\}\+\\log N\_\{P\}\-\\varsigma\(x\)\. Restrict to a linex=t​ux=tu\. IfWq≠0W\_\{q\}\\neq 0, chooseuuwithWq​u≠0W\_\{q\}u\\neq 0;ς⁡\(t​u\)\\varsigma\(tu\)is a fixed polynomial of degree at most two intt, and⟨q⁡\(t​u\),k1⟩\\langle q\(tu\),k\_\{1\}\\ranglehas the slope⟨Wq​u,k1⟩\\langle W\_\{q\}u,k\_\{1\}\\rangle, whichk1k\_\{1\}can make any real number, sok1k\_\{1\}can be chosen withδ⁡\(t​u\)\\delta\(tu\)non\-constant\. IfWq=0W\_\{q\}=0andWk⊤​bq≠0W\_\{k\}^\{\\top\}b\_\{q\}\\neq 0, chooseuuwith⟨Wk⊤​bq,u⟩≠0\\langle W\_\{k\}^\{\\top\}b\_\{q\},u\\rangle\\neq 0; thenς⁡\(t​u\)\\varsigma\(tu\)has nonzero slope while the prefix score is constant, soδ⁡\(t​u\)\\delta\(tu\)is non\-constant for everyk1k\_\{1\}\. In both casesλ⁡\(t\):=λ⁡\(t​u\)\\lambda\(t\):=\\lambda\(tu\)is real\-analytic, takes values in\(0,1\)\(0,1\)and is non\-constant, soλ′′​\(t0\)≠0\\lambda^\{\\prime\\prime\}\(t\_\{0\}\)\\neq 0for somet0t\_\{0\}; otherwiseλ\\lambdawould be affine and bounded, hence constant\. Choosev∗v\_\{\*\}withWo​v∗≠0W\_\{o\}v\_\{\*\}\\neq 0and putv1=M​v∗v\_\{1\}=Mv\_\{\*\}\. Sincev⁡\(t​u\)v\(tu\)is affine intt,

d2d​t2​headP​\(t​u\)=λ′′​\(t\)​\(M​Wo​v∗−Wo​v​\(t​u\)\)−2​λ′​\(t\)​Wo​Wv​u,\\frac\{d^\{2\}\}\{dt^\{2\}\}\\mathrm\{head\}\_\{P\}\(tu\)=\\lambda^\{\\prime\\prime\}\(t\)\\big\(MW\_\{o\}v\_\{\*\}\-W\_\{o\}v\(tu\)\\big\)\-2\\lambda^\{\\prime\}\(t\)\\,W\_\{o\}W\_\{v\}u,and att0t\_\{0\}the termM​λ′′​\(t0\)​Wo​v∗M\\lambda^\{\\prime\\prime\}\(t\_\{0\}\)W\_\{o\}v\_\{\*\}dominates for largeMM, so the second derivative is nonzero and the head is not affine\. For bias\-free headsWk⊤​bq=0W\_\{k\}^\{\\top\}b\_\{q\}=0, which gives the second form of the criterion\.

*Generic prefixes\.*The prefixed output is real\-analytic in\(x,P\)\(x,P\)on the connected space of inputs and prefixes withNPN\_\{P\}pairs\. It is affine inxxfor a givenPPiff all second partial derivatives inxxvanish at everyxx; this is the common zero set, overxxand pairs of coordinates, of real\-analytic functions ofPP\. The witness above is a prefix at which one of these functions is nonzero, so that function is not identically zero, and the prefixes with affine output lie in its zero set, which is closed, nowhere dense and null \([Auxiliary result B\.10](https://arxiv.org/html/2610.12728#A2.SS0.SSSx2)\)\. The same argument applied toλ\\lambdashows that the gate is non\-constant for generic prefixes\. Box\-locally, the class of a standard head is the set of functions it computes with some weights; on length\-one contexts these are all affine, while for generic prefixes the prefixed head is not, so prefix tuning and P\-tuning v2 are not box\-locally mergeable into the head\.

*No neutral prefix\.*A neutral prefix would leave the head unchanged on every input, in particular on length\-one contexts, where by \([28](https://arxiv.org/html/2610.12728#A3.E28)\) this meansλ⁡\(x\)​\(Wo​Attnq⁡\(x\)​\(P\)−Wo​v​\(x\)\)=0\\lambda\(x\)\\big\(W\_\{o\}\\mathrm\{Attn\}\_\{q\(x\)\}\(P\)\-W\_\{o\}v\(x\)\\big\)=0, and sinceλ⁡\(x\)\>0\\lambda\(x\)\>0,Wo​Attnq⁡\(x\)​\(P\)=Wo​\(Wv​x\+bv\)W\_\{o\}\\mathrm\{Attn\}\_\{q\(x\)\}\(P\)=W\_\{o\}\(W\_\{v\}x\+b\_\{v\}\)for allxx\. The left side lies in the compact setWo​conv⁡VPW\_\{o\}\\operatorname\{conv\}V\_\{P\}; ifWo​Wv≠0W\_\{o\}W\_\{v\}\\neq 0, choosex0x\_\{0\}withWo​Wv​x0≠0W\_\{o\}W\_\{v\}x\_\{0\}\\neq 0, and alongx=t​x0x=tx\_\{0\}the right side is unbounded\. So no prefix is neutral\. For a base network consisting of one such head, the extension by prefixes therefore has no neutral point, and by \(e\), at the first layer of a causal decoder a soft prompt acts through its key–value pairs exactly as a one\-layer prefix, so the same base network admits no neutral prompt: prompt and prefix tuning are unpointed\. ∎

Remark\(numerical checks\)\. The values reported after[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\(prompt\-to\-prefix agreement9×10−169\\times 10^\{\-16\}with RoPE and the offset,1\.31\.3without it and2×10−152\\times 10^\{\-15\}with keys rotated by−m\-m; an affine head withWo=0W\_\{o\}=0althoughWq⊤​Wk≠0W\_\{q\}^\{\\top\}W\_\{k\}\\neq 0, and a non\-affine one withWq≠0W\_\{q\}\\neq 0andWq⊤​Wk=0W\_\{q\}^\{\\top\}W\_\{k\}=0\) instantiate parts \(e\) and \(f\) as proved above: the criterion of \(f\) involvesWoW\_\{o\},WqW\_\{q\}andWk⊤​bqW\_\{k\}^\{\\top\}b\_\{q\}, notWq⊤​WkW\_\{q\}^\{\\top\}W\_\{k\}\(Appendix[E](https://arxiv.org/html/2610.12728#A5)\)\.

## Appendix DThe catalogue

The atlas of Exposé[VIII](https://arxiv.org/html/2610.12728#S8)rests on a catalogue of 382 methods and 138 papers\. This appendix says how the catalogue was assembled and fact\-checked, how each method received its seven coordinates, its arrows and its obstructions, and how these were checked in turn\. It ends with the complete tables: every method with its coordinates and its reference \(Table[36](https://arxiv.org/html/2610.12728#A4.T36)\), every recorded arrow with its maphh\(Table[37](https://arxiv.org/html/2610.12728#A4.T37)\), and the arrows and obstructions counted by rule and by test \(Table[35](https://arxiv.org/html/2610.12728#A4.T35)\)\.

##### Data files

The data accompany this paper in machine\-readable form, generated from the same source as the tables below and as the companion site\.

catalogue\.jsonEvery method \(382\) and every paper \(138\), each with its verified metadata \(title, full author list, year of the first arXiv version, venue, links to the paper and the code\) and its BibTeX record\. A method entry adds the formula, the trainable\-parameter count, the locus, the modification kind, the merge verdict with its reason, the initialisation, the relations to other methods, the sources consulted, the seven coordinates with a justification for each, the categorical reading, and the method’s arrows and obstructions\. The file also holds the definitions of the axis values, with the question that decides each axis, and the arrow rules\.

catalogue\.csvOne row per method, with the main fields of catalogue\.json and one column per coordinate\.

arrows\.csvEvery recorded arrowM→NM\\to Nwith its rule, its maphhand a note that names the instance ofNNit lands in \(rank, pointing or regime\)\.

obstructions\.csvEvery recorded obstructionM↛NM\\not\\to Nwith the test that rules the arrow out and the invariant that fails\.

propositions\.jsonEvery numbered result \(41 of them\) with its statement, proof, review status, second\-round verdict, public statement and referee notes\.

framework\_checks\.pyThe numerical checks of Appendix[E](https://arxiv.org/html/2610.12728#A5)\.

### D\.1 Families and sources

Every entry was written once and then attacked once, by a separate reader looking for what the first got wrong\. The literature was split into the thirteen families of Table[34](https://arxiv.org/html/2610.12728#A4.T34), then completed by four gap\-filling batches, which added the methods a completeness review found missing, and by two addenda on 2 October 2026: methods the authors reported missing, and a sweep of the 2026 literature run in two passes \(§[D\.3](https://arxiv.org/html/2610.12728#A4.SS3)\)\. The family boundaries owe much to earlier maps of the field: the unified view of[He et al\. \(2021\)](https://arxiv.org/html/2610.12728#bib.bib165), the delta\-tuning study of[Ding et al\. \(2022\)](https://arxiv.org/html/2610.12728#bib.bib93), the guide of[Lialin et al\. \(2023a\)](https://arxiv.org/html/2610.12728#bib.bib257), the survey of[Han et al\. \(2024b\)](https://arxiv.org/html/2610.12728#bib.bib153)and the modular view of[Pfeiffer et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib369)\.

Table 34:How the catalogue was assembled\. Each family was written by one reader and attacked by another; the last column \(Corr\.\) counts the logged corrections\. The families overlap, so the records were merged into the catalogue on a canonical identifier\.FamilyWhat it coversMethodsPapersCorr\.LoRA coreLoRA and its variants201221Initialisation and rankBase\-point splits; rank allocation25646ShapeKronecker, Hadamard, tensor and sparse shapes29949MultiplicativeGroup and monoid actions onW0W\_\{0\}32954SharingShared or random bases28446SelectiveSubsets of existing weights26737AdaptersInserted modules and side networks311030PromptsSoft prompts, prefixes and learned states271339Quantisation and memoryQuantised bases, gradient memory24544CompositionMerging, routing, mixtures42861HF PEFTThe library’s tuners and utilities741053Frontier2025–2026: RL, reasoning, diffusion21548TheoryTheory and foundations05023Gap\-filling \(4 batches\)Methods a completeness review found missing361862Addendum \(2 Oct 2026\)Methods the authors reported missing102Addendum \(2026 sweep, pass 1\)arXiv, January to September 2026; ICLR and ICML 2026298126Addendum \(2026 sweep, pass 2\)Further candidates, mostly ICLR, ICML and NeurIPS 2026326164All recordsBefore merging477180905CatalogueAfter merging duplicates382138##### Merging

The families overlap, so the477477method records and180180paper records were merged into 382 methods and 138 papers\. Records were matched on a canonical identifier and a table of aliases, because matching by name would fold LoRA\+\([Hayou et al\., 2024b](https://arxiv.org/html/2610.12728#bib.bib159)\)into LoRA\([Hu et al\., 2021](https://arxiv.org/html/2610.12728#bib.bib177)\)and O\-LoRA\([Wang et al\., 2023b](https://arxiv.org/html/2610.12728#bib.bib477)\)into OLoRA\([Büyükakyüz, 2024](https://arxiv.org/html/2610.12728#bib.bib47)\)\. Of the 382 entries,358358are methods proper; the others are1717training recipes,44software systems,22analyses and11baseline \(linear probing\)\. The first arXiv versions of the methods run from 2013 to 2026, and4747methods were first posted in 2026\.

##### The bibliography

The BibTeX record of each entry comes from the arXiv metadata of its identifier, with the full author list, for493493identifiers, and was written by hand for the nine entries without one\. These502502records form the catalogue part of this paper’s bibliography, and the citation keys of Table[36](https://arxiv.org/html/2610.12728#A4.T36)point into it\.

### D\.2 Fact\-checking

Each field was checked against a primary source\.

- •Titles, authors and first\-version years were checked against the arXiv metadata\.
- •Venues were checked against the official records: OpenReview, the ACL Anthology, PMLR and CVF open access, with Crossref, Semantic Scholar and DBLP for the remaining cases\.
- •Formulas and parameter counts were checked against the papers, and against the official code where an initialisation was in doubt\.
- •Every claim about Hugging Face PEFT\([Mangrulkar et al\., 2022](https://arxiv.org/html/2610.12728#bib.bib312)\)was checked against its source at commit 85be3f3 \(85be3f3a1a58c4d153ffc4631c57196430af87a4\) of 30 September 2026\. The library’s code settled many questions of convention, such as the side ofWWon which a rotation acts, the default initialisation of each factor, and whether a scale such asα/r\\alpha/ris applied\. The column “HF” of Table[36](https://arxiv.org/html/2610.12728#A4.T36)marks the 82 methods available at that commit\.

Every change was logged with its reason,905905in all \(Table[34](https://arxiv.org/html/2610.12728#A4.T34)\)\. The merge verdict of each method is recorded beside its reason:250250methods fold back into the frozen weights always,4141under stated conditions and9191not at all\. This verdict concerns the deployed method; the merge coordinate of Exposé[VIII](https://arxiv.org/html/2610.12728#S8)refines it by the rewrite rules of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\.

### D\.3 The 2026 sweep and the authors’ addendum

On 2 October 2026 the authors pointed out methods missing from the catalogue\. LOFT\([Zhao et al\., 2026](https://arxiv.org/html/2610.12728#bib.bib582)\)was added at once as an addendum of its own, checked against arXiv and the paper \(two corrections logged\), and classified like every other method\. Its coordinates are R, Orb, regular, none,CO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}, M1 and idempotent; it has arrows into OFT, LoRA\-XS, PSOFT and LoRA, and forr≥4r\\geq 4its first\-order image rules out a smooth pointed arrow into HRA at HF’s paired initialisation \(test O3\)\. Two independent referees checked these coordinates, arrows and the obstruction, numerically and against the paper and the code\. §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4)reads LOFT as a fixed leaf of HRA’s meet\.

The other gaps were sought by a sweep of the 2026 literature, run in two passes on the same day\. A pool of265265candidates was drawn from six slices:

- •three quarterly arXiv slices, for first versions posted from January to March, April to June and July to September 2026, with6060candidates each;
- •the accepted papers of ICLR 2026,5,4685\{,\}468of them screened through their abstracts, with4848candidates;
- •the accepted papers of ICML 2026,6,7976\{,\}797screened, with a second pass over ACL, EACL and CVPR 2026, together giving3131candidates;
- •the 2026 papers citing anchor methods \(LoRA, DoRA, PiSSA, QLoRA, OFT, VeRA, HRA, GaLore, PSOFT and LoRA\-GA\), ranked by citations, recency and acceptance, with the 2026 release notes and pull requests of Hugging Face PEFT, giving66candidates\.

Acceptance at ICLR and ICML 2026 was confirmed by an exact title match against the official lists of accepted papers or by the authors’ own arXiv comment\. Candidates were matched against the catalogue by arXiv identifier and by name, so that near\-homonyms stay apart: the entropy\-guided FlexLoRA of 2026\([Liu et al\., 2026c](https://arxiv.org/html/2610.12728#bib.bib278)\)is not the federated FlexLoRA of 2024 \(arXiv:2402\.11505\), and the per\-token MoLoRA of 2026 \(arXiv:2603\.15965\) is not the MoLoRA of[Zadouri et al\. \(2023\)](https://arxiv.org/html/2610.12728#bib.bib541)\. The first pass admitted2929methods and88papers\. The second re\-screened the rest of the pool and admitted3232methods and66papers, mostly from ICLR, ICML and NeurIPS 2026\. Each admitted entry went through the same fact\-checking as the families, with126126and164164logged corrections, and each of the6161new methods received an independent check of its coordinates and arrows on the same day\. The arXiv queries searched abstracts for LoRA, PEFT, adapter, parameter\-efficient and prompt tuning, so a 2026 method paper whose abstract uses none of these terms lies outside the sweep’s reach\.

### D\.4 Coordinates, arrows and obstructions

##### Assignment

Each method was classified from its formula, in thirteen batches: ten of3232methods for the catalogue as it stood on 1 October 2026, one for the authors’ addendum, and one for each pass of the 2026 sweep \(2929and3232methods\)\. A first reader answered the seven questions of §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1), which fix the kind, the shape of the image germ, the base point, the gauge, the covariance group, the merge type and the rebasing closure, following the decision procedures of §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1.SSSx1)and the conventions of §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1.SSSx2)\. Each answer carries a short justification, kept in catalogue\.json\. A null coordinate, printed “–”, means that the axis does not apply, for instance the shape of a method with no germ atθ0\\theta\_\{0\}\. Where an answer turns on the dimension of an image or a fibre, or on the first\-order image at the initialisation, it was confirmed by a float64 Jacobian rank at a random point and at the base point \([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\)\.

Each arrowM→NM\\to Ncomes with an explicit pointed smooth maphhsatisfyingρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}, obtained by one of the rules A2–A5 of §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx1), and with the instance ofNNit lands in: a rank, a pointing or a hyperparameter regime\. Each obstructionM↛NM\\not\\to Nnames the test O1–O3 of §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3.SSSx2)that rules the arrow out and the invariant that fails\. The interval arrowsFrozen→M→Full FT\\text\{Frozen\}\\to M\\to\\text\{Full FT\}of rule A1 hold for every weight\-space method and are not recorded\. The 142 core methods are those named in the periodic table \(Table[19](https://arxiv.org/html/2610.12728#S8.T19)\) together with a curated set of others; the flag is not a coordinate\.

##### Independent check

A second reader re\-derived every coordinate from the formula and re\-checked every arrow, by recomputingρN∘h\\rho\_\{N\}\\circ hand the pointingh⁡\(q0M\)h\(q\_\{0\}^\{M\}\), and every obstruction, by recomputing the failing invariant\. Each method carries a check note with the outcome\. More than half of the notes record no change; the others record each change with its reason, and4747record a numerical Jacobian computation \(a rank, a fibre dimension or a deficiency\)\.

A consolidation pass on 1 October 2026, over the320320methods classified by then, unified the data and touched135135methods, each change logged in its check note\.

1. \(1\)137137coordinates used a value “n/a” that the axis definitions do not contain\.136136became null, matching the “–” of the tables, and the base point of GenKnowSub became “unpointed”, as for the other parameterless post\-hoc operations\. The merge axis keeps its legacy key na, read as null\.
2. \(2\)The row labels of the periodic table were unified across batches\.
3. \(3\)The covariance of LISA and BAdam changed fromΠ\\PitoGL×GL\\mathrm\{GL\}\\times\\mathrm\{GL\}, matching HiFT, surgical fine\-tuning and BitFit: each stage frees whole boxes, and the layer partition breaks no neuron permutation\.
4. \(4\)Four obstruction reasons now say which instance of the target they concern \(DSEE, GaLore, Spectral AdapterAand MoV, each against LoRA\)\.

Where a formula disagreed with an earlier draft of the periodic table, the value the formula supports was kept and the table of Exposé[VIII](https://arxiv.org/html/2610.12728#S8)was revised to agree, in thirteen places\. For example, LST, UniPELT and MAM are unpointed extensions, MPT and Compacter have torus gauges, the Houlsby adapter’s GELU or swish leaves only the permutations of hidden units, and the gauges of DeLoRA and AdaLoRA are non\-linear\. The6262later entries, LOFT and the6161methods of the 2026 sweep, were classified and checked in the same way\.

##### Structural checks on the final data

For this appendix the structural checks were re\-run on the ancillary files\. Every arrow and every obstruction joins two catalogued methods and names a valid rule or test; every arrow has a maphhand every obstruction a reason; every coordinate is a value defined for its axis; and no arrow or obstruction is recorded twice\. A check at the level of method identifiers flags four kinds of pattern, and each was resolved by the instance it concerns\.

- •*Pairs with both an arrow and an obstruction*\(2626\)\. In every case the two concern different instances of the target: a different rank, pointing, regime or variant \(§[VIII\.B](https://arxiv.org/html/2610.12728#S8.SS2.SSSx2)\)\. For example, ABBA is isomorphic to LoHa pointed at the factors of\[W0\]r1\[W\_\{0\}\]\_\{r\_\{1\}\}, while the O3 obstruction concerns LoHa at HF’s random default pointing\.
- •*Arrows from a method to itself*\(88\)\. Each is an inclusion between instances of one family, such as LoRAr↪\{\}\_\{r\}\\hookrightarrow\{\}LoRAr\+1or ReLoRAK↪\{\}^\{K\}\\hookrightarrow\{\}ReLoRAK\+1\.
- •*Cycles*\(1313groups of mutually reachable methods\)\. The cycles in these groups are of four kinds\. Eight are isomorphisms: DoRA and BiDoRA, ReLoRA and COLA, AsyLoRA and LoRA\-FA, AdaptFormer and the parallel adapter, S\-LoRA and Punica, EFT and proxy tuning, LoRAMoE and MoLoRA, and AdaMerging and LoRA soups \(CAT\), which differ only in pointing\. Three join methods with equal images that are not isomorphic: prompt tuning with MPT and with P\-tuning, and QLoRA with IR\-QLoRA\. Five go up in rank or in the number of stages and are not isomorphisms: LoRA with ReLoRA, LoRA with PEANuT, LoRA\-FA with Flora, surgical fine\-tuning with HiFT, and KronA with LoKr\. Initial\-state tuning and prefix tuning simulate each other on S4 state\-space layers only\.
- •*ObstructionsM↛NM\\not\\to Nwith no direct arrow but a path of arrows fromMMtoNN*\(2020\)\. In each, the obstruction concerns a fixed rank or pointing of the target and the path ends in another instance\. FourierFT withncn\_\{c\}frequencies, for example, maps into LoRA\-FA2​nc\{\}\_\{2n\_\{c\}\}and so into LoRA2​nc\{\}\_\{2n\_\{c\}\}, while the O2 obstruction concerns LoRArat a matched budget\.

No arrow contradicts an obstruction once the stated rank or pointing is taken into account\. The eight places where the data depart from the conventions of §[VIII\.A](https://arxiv.org/html/2610.12728#S8.SS1.SSSx2)are listed in §[VIII\.B](https://arxiv.org/html/2610.12728#S8.SS2.SSSx2), and Table[23](https://arxiv.org/html/2610.12728#S8.T23)counts the methods per coordinate value\.

Table 35:The 403 recorded arrows by rule \(top\) and the 485 recorded obstructions by test \(bottom\)\. The rules and tests are those of §[VIII\.C](https://arxiv.org/html/2610.12728#S8.SS3);161161of the arrows join two core methods\. Every arrow is listed in Table[37](https://arxiv.org/html/2610.12728#A4.T37), and every obstruction, with its failing invariant, in the ancillary file obstructions\.csv\.

### D\.5 The complete tables

Table[36](https://arxiv.org/html/2610.12728#A4.T36)lists every method with its reference, the year of its first arXiv version and its seven coordinates\. A method without an arXiv version carries the year it first became public\. The year in a citation is that of the version cited, so the two can differ: LoRA\-S, for instance, first appeared in 2025 and is cited as[Zheng and Wu \(2026\)](https://arxiv.org/html/2610.12728#bib.bib587), its ICLR 2026 version\. The rows are grouped by kind, with the reparametrisations split as in the periodic table \(additive, action, Hadamard and selective\), and sorted by name within each group\. The abbreviations are those of Table[23](https://arxiv.org/html/2610.12728#S8.T23), with “unpt\.” for unpointed, “nonlin\.” for a non\-linear gauge, GL×\\timesT forGL\\mathrm\{GL\}times a Hadamard torus, O×\\timesO for the spectral\-frame valueCO×CO\\mathrm\{CO\}\\times\\mathrm\{CO\}, “idemp\.” for idempotent and “sched\.” for a scheme that is itself a rebasing schedule\. The merge value na is the legacy key for backward methods, read as “–”\.

Table[37](https://arxiv.org/html/2610.12728#A4.T37)lists every recorded arrow, sorted by source, with its rule and its maphh\. Eachhhis stated in the notation of the target’s formula in catalogue\.json, and the instance of the target \(its rank, its pointing or its regime\) is in the note column of arrows\.csv\.

Table 36:The complete catalogue: every method with its seven coordinates \(§[VIII](https://arxiv.org/html/2610.12728#S8)\)\. Core methods are marked∙\\bullet; HF marks methods available in Hugging Face PEFT at the pinned commit\. “–” means the axis does not apply\.MethodReferenceYearKindShapeBase pt\.GaugeCovar\.MergeRebaseHFR, additiveABBA[Singhal et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib421)2025RConeapexGL×\\timesTMon×\\timesMonM1span∙\\bulletAdaLoRA[Zhang et al\.,2023g](https://arxiv.org/html/2610.12728#bib.bib559)2023RConeapexnonlin\.GL×\\timesGLM1span✓∙\\bulletAdaMSS[Zheng et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib586)2025RConeapexGLO×\\timesOM1span✓AdaPaD[Su et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib430)2026RConedefectGLGL×\\timesGLM1spanALoRA[Liu et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib292)2024RConeapexGLGL×\\timesGLM1spanAltLoRA[Yu et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib539)2025RConeapexGLGL×\\timesGLM1spanApiQ[Liao et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib266)2024RCone∗defectGLMon×\\timesMonMqspanAR1\-ZO[Chen et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib73)2026RConeapexGLGL×\\timesGLM1span∙\\bulletAstra[Liu et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib277)2026RCone∗regularGLO×\\timesOM1span✓AsyLoRA[Zhu et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib598)2024RLinregularnoneframeM1idemp\.AutoLoRA[Zhang et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib564)2024RConeapexGLGL×\\timesGLM1span∙\\bulletBD\-LoRA[Wang et al\.,2025d](https://arxiv.org/html/2610.12728#bib.bib480)2025RConeapexGLΠ\\PiM1span✓∙\\bulletC3A[Chen et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib59)2024RLinregularnoneΠ\\PiM1idemp\.✓Concept Sliders[Gandikota et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib120)2023RConeapexGLGL×\\timesGLM1span∙\\bulletCorDA[Yang et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib528)2024RCone∗regularGLO×\\timesOM1span✓∙\\bulletDEFT[Kumar et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib228)2025RConeapexGLGL×\\timesGLM1span✓∙\\bulletDeLoRA[Bini et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib37)2025RConeapexnonlin\.GL×\\timesGLM1span✓DoRA \(dynamic rank\)[Mao et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib314)2024RConeapexnonlin\.GL×\\timesGLM1spanDSEE[Chen et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib68)2021RConeapexGLMon×\\timesMonM1span∙\\bulletDyLoRA[Valipour et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib456)2022RConeapextorusGL×\\timesGLM1spanElaLoRA[Chang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib54)2025RConeapexnonlin\.GL×\\timesGLM1span∙\\bulletEVA[Paischer et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib358)2024RConeapexGLGL×\\timesGLM1span✓∙\\bulletFacT[Jie and Deng,2022b](https://arxiv.org/html/2610.12728#bib.bib199)2022RConeapexGLΠ\\PiM1spanFedRot\-LoRA[Zhang et al\.,2026c](https://arxiv.org/html/2610.12728#bib.bib550)2026RConeapexGLGL×\\timesGLM1span∙\\bulletFFA\-LoRA[Sun et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib437)2024RLinregularnoneframeM1idemp\.Flat\-LoRA[Li et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib250)2024RConeapexGLGL×\\timesGLM1spanFlexLoRA[Liu et al\.,2026c](https://arxiv.org/html/2610.12728#bib.bib278)2026RConeapexnonlin\.GL×\\timesGLM1spanFLoRA \(Tucker core\)[Si et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib416)2024RConeapexGLGL×\\timesGLM1spanFLoRG[Meng et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib323)2026RConedefectGLframeM1span∙\\bulletFourierFT[Gao et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib124)2024RLinregularnoneΠ\\PiM1idemp\.✓FourTune[Xue et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib516)2026RConedefectGLMon×\\timesMonMqspanFrameFT[Adepu et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib3)2026RLinregularnoneΠ\\PiM1idemp\.∙\\bulletFRoD[Wan et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib463)2025RLinregularnoneO×\\timesOM1idemp\.✓GeoRA[Zhang et al\.,2026d](https://arxiv.org/html/2610.12728#bib.bib553)2026RCone∗regularGLMon×\\timesMonM1spanGiVA[Gangwar et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib121)2026RConeapexscalarO×\\timesOM1spanGoRA[He et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib161)2025RConedefectGLGL×\\timesGLM1span∙\\bulletGraLoRA[Jung et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib203)2025RConeapexGLΠ\\PiM1span✓HetLoRA[Cho et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib75)2024RConeapexGLGL×\\timesGLM1spanHybrid\-LoRA[Zhang et al\.,2026a](https://arxiv.org/html/2610.12728#bib.bib545)2026RConeapexGLGL×\\timesGLM1spanIC\-LoRA[Huang et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib181)2024RConeapexGLGL×\\timesGLM1spanIncreLoRA[Zhang et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib548)2023RConeapexnonlin\.GL×\\timesGLM1span∙\\bulletInfLoRA[Liang and Li,2024](https://arxiv.org/html/2610.12728#bib.bib261)2024RLinregularnoneGL×\\timesCOM1sched\.Intrinsic SAID[Aghajanyan et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib5)2020RConeapexscalarΠ\\PiM1spanIR\-QLoRA[Qin et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib377)2024RConedefectGLMon×\\timesMonMqspanKAdaptation[He et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib167)2022RConeapexGLΠ\\PiM1span∙\\bulletKaSA[Wang et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib464)2024RConedefectnonlin\.O×\\timesOM1span✓KeepLoRA[Luo et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib299)2026RLinregularnoneO×\\timesOM1sched\.KRAdapter[Albert et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib8)2025RConeapextorusΠ\\PiM1span∙\\bulletKronA[Edalati et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib103)2022RConeapexscalarΠ\\PiM1spanLaMDA[Azizi et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib20)2024RLinregularGLO×\\timesOM1idemp\.LoCA[Du et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib101)2025RLinregularnoneΠ\\PiM1idemp\.LoDA[He et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib166)2026RLinregularnoneGL×\\timesCOM1sched\.LoFT[Tastan et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib448)2025RConeapexGLGL×\\timesGLM1span∙\\bulletLoftQ[Li et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib255)2023RCone∗defectGLMon×\\timesMonMqspan✓∙\\bulletLoHa[Hyeon\-Woo et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib186)2021RConeapexGL×\\timesTMon×\\timesMonM1span✓∙\\bulletLoKr[Yeh et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib532)2023RConeapexGLΠ\\PiM1span✓∙\\bulletLongLoRA[Chen et al\.,2023c](https://arxiv.org/html/2610.12728#bib.bib70)2023RConeapexGLMon×\\timesGLM1span∙\\bulletLoRA[Hu et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib177)2021RConeapexGLGL×\\timesGLM1span✓LoRA Dropout[Lin et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib271)2024RConeapexGLGL×\\timesGLM1spanLoRA Without Regret[Schulman and Thinking Machines Lab,2025](https://arxiv.org/html/2610.12728#bib.bib401)2025RConeapexGLGL×\\timesGLM1span✓∙\\bulletLoRA\+[Hayou et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib159)2024RConeapexGLGL×\\timesGLM1span✓LoRA\-DA[Zhang et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib560)2025RConedefectGLGL×\\timesGLM1spanLoRA\-Dash[Si et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib415)2024RConeapexGLO×\\timesOM1spanLoRA\-drop[Zhou et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib593)2024RConeapexGLGL×\\timesGLM1span∙\\bulletLoRA\-FA[Zhang et al\.,2023d](https://arxiv.org/html/2610.12728#bib.bib556)2023RLinregularnoneframeM1idemp\.✓∙\\bulletLoRA\-GA[Wang et al\.,2024e](https://arxiv.org/html/2610.12728#bib.bib472)2024RCone∗regularGLO×\\timesOM1span✓LoRA\-Mini[Singh et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib419)2024RConedefectGLframeM1spanLoRA\-Null[Tang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib445)2025RCone∗regularGLGL×\\timesCOM1spanLoRA\-One[Zhang et al\.,2025d](https://arxiv.org/html/2610.12728#bib.bib569)2025RConedefectGLGL×\\timesGLM1span∙\\bulletLoRA\-Pro[Wang et al\.,2024i](https://arxiv.org/html/2610.12728#bib.bib486)2024RConeapexGLGL×\\timesGLM1span∙\\bulletLoRA\-RITE[Yen et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib533)2024RConeapexGLGL×\\timesGLM1spanLoRA\-RLPO / LoRA\-RLMO[Zhang et al\.,2026e](https://arxiv.org/html/2610.12728#bib.bib563)2026RConeapexGLGL×\\timesGLM1spanLoRA\-S \(AdamS, LRACS\)[Zheng and Wu,2026](https://arxiv.org/html/2610.12728#bib.bib587)2025RConeapexGLGL×\\timesGLM1span∙\\bulletLoRA\-SB[Ponkshe et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib371)2024RLindefectnoneMon×\\timesMonM1idemp\.∙\\bulletLoRA\-XS[Bałazy et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib26)2024RLinregularnoneO×\\timesOM1idemp\.∙\\bulletLoRETTA[Yang et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib529)2024RCone∗regularGLΠ\\PiM1spanLoRI[Zhang et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib555)2025RLinregularnoneframeM1idemp\.LoRTA[Hounie et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib176)2024RConeapextorusΠ\\PiM1spanLoTR[Bershatsky et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib29)2024RConeapexGLGL×\\timesGLM1spanLQ\-LoRA[Guo et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib145)2023RCone∗defectGLMon×\\timesMonMqspanMatryoshkaLoRA[Modoranu et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib334)2026RConeapexGLGL×\\timesGLM1spanMELoRA[Ren et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib387)2024RConeapexGLΠ\\PiM1spanMetaTT[Lopez\-Piqueres et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib295)2025RConeapexGLGL×\\timesGLM1span∙\\bulletMiCA[Rüdiger and Raschka,2026](https://arxiv.org/html/2610.12728#bib.bib395)2026RLinregularnoneO×\\timesOM1idemp\.✓∙\\bulletMiLoRA[Wang et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib465)2024RCone∗regularGLO×\\timesOM1span∙\\bulletMiSS \(formerly Bone\)[Kang and Yin,2024](https://arxiv.org/html/2610.12728#bib.bib206)2024RLinregularnoneΠ\\PiM1idemp\.✓ModuLoRA[Yin et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib536)2023RConedefectGLMon×\\timesMonMqspanMonteCLoRA[Sengupta et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib403)2024RConeapexGLGL×\\timesGLM1span✓∙\\bulletMoRA[Jiang et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib196)2024RLinregularnoneΠ\\PiM1idemp\.MoRe[Tan et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib442)2024RConeapextorusΠ\\PiM1spanMoS[Wang et al\.,2024f](https://arxiv.org/html/2610.12728#bib.bib473)2024RConeapexscalarΠ\\PiM1spanMoSA[Wang et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib478)2026RLinregularnoneframeM1idemp\.MoSLoRA[Wu et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib500)2024RConeapexGLGL×\\timesGLM1span∙\\bulletNOLA[Koohpayegani et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib223)2023RConeapexscalarframeM1span∙\\bulletO\-LoRA[Wang et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib477)2023RConeapexGLGL×\\timesGLM1sched\.∙\\bulletOLoRA[Büyükakyüz,2024](https://arxiv.org/html/2610.12728#bib.bib47)2024RCone∗regularGLΠ\\PiM1span✓Orthogonal Adaptation[Po et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib370)2023RLinregularnoneframeM1idemp\.∙\\bulletOSF[Nayak et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib344)2025RLinregularnonlin\.O×\\timesOM1idemp\.✓∙\\bulletPEANuT[Zhong et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib589)2024RConeapexnonlin\.Mon×\\timesGLM1span✓∙\\bulletPiSSA[Meng et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib324)2024RCone∗regularGLO×\\timesOM1span✓PLoP[Hayou et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib160)2025RConeapexGLGL×\\timesGLM1spanPoLAR[Lion et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib273)2025RConeapexGLGL×\\timesGLM1span∙\\bulletPRISM[Wang and Zhang,2026](https://arxiv.org/html/2610.12728#bib.bib475)2026RCone∗regularGLO×\\timesOM1spanPRoLoRA[Wang et al\.,2024g](https://arxiv.org/html/2610.12728#bib.bib474)2024RConeapexGLΠ\\PiM1span∙\\bulletPVeRA[Fillioux et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib112)2025RConeapexscalarframeM1span✓∙\\bulletQA\-LoRA[Xu et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib514)2023RConedefectGLΠ\\PiM1span✓QeRL[Huang et al\.,2025c](https://arxiv.org/html/2610.12728#bib.bib183)2025RConedefectGLMon×\\timesMonMqspan∙\\bulletQLoRA[Dettmers et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib92)2023RConedefectGLMon×\\timesMonMqspan✓QuanTA[Chen et al\.,2024f](https://arxiv.org/html/2610.12728#bib.bib72)2024RCone∗regularGLΠ\\PiM1spanQuantum\-PEFT[Koike\-Akino et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib221)2025RConeapexnonlin\.Π\\PiM1span∙\\bulletRandLoRA[Albert et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib7)2025RConeapextorusframeM1span✓RaSA[He et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib170)2025RConeapexGLGL×\\timesGLM1span∙\\bulletRiemannian LoRA[Zhang and Pilanci,2024a](https://arxiv.org/html/2610.12728#bib.bib546)2024RConeapexGLGL×\\timesGLM1span✓Riemannion[Bogachev et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib41)2025RCone∗regularnoneO×\\timesOM1span∙\\bulletRoSA[Nikdan et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib350)2024RConeapexGLMon×\\timesMonM1span∙\\bulletrsLoRA[Kalajdzievski,2023](https://arxiv.org/html/2610.12728#bib.bib204)2023RConeapexGLGL×\\timesGLM1span✓SD\-LoRA[Wu et al\.,2025c](https://arxiv.org/html/2610.12728#bib.bib503)2025RConedefectGLGL×\\timesGLM1sched\.SEAL[Zweiger et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib602)2025RConeapexGLGL×\\timesGLM1spanShareLoRA[Song et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib424)2024RConeapexGLGL×\\timesGLM1spanSine\-LoRA[Ji et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib193)2024RConeapexGLMon×\\timesMonM1spanSingLoRA[Bensaïd et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib28)2025RConedefectGLΠ\\PiM1spanSLAO \(Merge before Forget\)[Qiao and Mahdavi,2025](https://arxiv.org/html/2610.12728#bib.bib376)2025RConedefectGLGL×\\timesGLM1spanSLTrain[Han et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib151)2024RConedefectGLMon×\\timesMonM1spanSoRA[Ding et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib94)2023RConeapexnonlin\.GL×\\timesGLM1spanSparseLoRA[Khaki et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib210)2025RConeapexGLGL×\\timesGLM1spanSpectral AdapterA[Zhang and Pilanci,2024b](https://arxiv.org/html/2610.12728#bib.bib547)2024RCone∗regularGLO×\\timesOM1spanSRR \(Structured Residual Reconstruction\)[Cho et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib76)2026RCone∗defectGLMon×\\timesMonMqspanSSH[Shen et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib408)2025RLinregularnoneΠ\\PiM1idemp\.Stable\-LoRA[Wu et al\.,2025d](https://arxiv.org/html/2610.12728#bib.bib504)2025RConeapexGLGL×\\timesGLM1spanStructLoRA[Xiao et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib509)2026RLindefectnonlin\.GL×\\timesGLM1idemp\.∙\\bulletSVFT[Lingam et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib272)2024RLinregularnoneO×\\timesOM1idemp\.TeRA[Gu et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib141)2025RConeapextorusΠ\\PiM1spanTied\-LoRA[Renduchintala et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib388)2023RConeapextorusMon×\\timesGLM1spanTina[Wang et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib471)2025RConeapexGLGL×\\timesGLM1span✓∙\\bulletTinyLoRA[Morris et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib336)2026RLinregularnoneO×\\timesOM1idemp\.✓Trans\-LoRA[Wang et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib470)2024RConeapexGLGL×\\timesGLM1spanTT\-LoRA[Anjum et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib11)2024RConedefectGLΠ\\PiM1span∙\\bulletUni\-LoRA[Li et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib248)2025RConedefectnoneframeM1span✓∙\\bulletVB\-LoRA[Li et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib254)2024RConedefectnoneΠ\\PiM1span✓VeLoRA[Miles et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib327)2024RConeapexGLGL×\\timesGLM1span✓∙\\bulletVeRA[Kopiczko et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib224)2023RConeapexscalarframeM1span✓WASI[Nguyen et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib348)2025RCone∗defectGLO×\\timesOM1idemp\.∙\\bulletWaveFT[Bilican et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib35)2025RLinregularnoneΠ\\PiM1idemp\.✓R, action∙\\bullet\(IA\)3[Liu et al\.,2022a](https://arxiv.org/html/2610.12728#bib.bib275)2022RLinregularnoneMon×\\timesGLM1→\\toidemp\.✓BiDoRA[Qin et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib378)2024RSatapexGLMon×\\timesGLM1span∙\\bulletBOFT[Liu et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib285)2023RSatregularnonlin\.Π\\PiM1group✓∙\\bulletDoRA[Liu et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib282)2024RSatapexGLMon×\\timesGLM1span✓EfficientQAT[Chen et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib62)2024RLindefectnoneΠ\\PiM1idemp\.ETHER[Bini et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib36)2024ROrbunpt\.torusGL×\\timesCOM1groupETHER\+[Bini et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib36)2024RConeapextorusO×\\timesOM1group∙\\bulletGOFT[Ma et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib305)2024ROrbregularnoneΠ\\PiM1group∙\\bulletHRA[Yuan et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib540)2024RMeetapexnonlin\.GL×\\timesCOM1group✓L4Q[Jeon et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib191)2024RLindefectnonlin\.Π\\PiM1span∙\\bulletLoCO[Nguyen et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib347)2026RMeetapexGLGL×\\timesCOM1group∙\\bulletLOFT[Zhao et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib582)2026ROrbregularnoneO×\\timesOM1idemp\.LoRA\-X[Farhadzadeh et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib110)2025RLinregularnoneO×\\timesOM1idemp\.LoRAPrune[Zhang et al\.,2023f](https://arxiv.org/html/2610.12728#bib.bib558)2023RSatapexGLMon×\\timesGLM1spanMasking[Zhao et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib583)2020ROrbdefecttorusMon×\\timesMonM1idemp\.∙\\bulletOFT[Qiu et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib381)2023ROrbregularnoneGL×\\timesCOM1idemp\.✓∙\\bulletOFTv2[Qiu et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib382)2025ROrbregularnoneGL×\\timesCOM1group✓PaRa[Chen et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib64)2024ROrbunpt\.GLGL×\\timesCOM1group✓∙\\bulletPEQA[Kim et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib214)2023RLindefectnoneMon×\\timesMonM1idemp\.∙\\bulletPiggyback[Mallya et al\.,2018](https://arxiv.org/html/2610.12728#bib.bib311)2018ROrbregulartorusMon×\\timesMonM1idemp\.∙\\bulletPSOFT[Wu et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib498)2025RSatapexscalarO×\\timesOM1group✓∙\\bulletqGOFT[Ma et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib305)2024ROrbregulartorusΠ\\PiM1groupQZO[Shang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib405)2025RLindefectnoneΠ\\PiM1idemp\.∙\\bulletRED[Wu et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib499)2024RLinregularnoneMon×\\timesGLM1→\\toidemp\.RepAdapter[Luo et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib298)2023RConeapexGLΠ\\PiM1→\\togroupResidual Adapters[Rebuffi et al\.,2017](https://arxiv.org/html/2610.12728#bib.bib385)2017RLinregulartorusGL×\\timesGLM1→\\toidemp\.∙\\bulletRoAd[Liao and Monz,2024](https://arxiv.org/html/2610.12728#bib.bib263)2024ROrbregularnoneMon×\\timesGLM1idemp\.✓Spectral AdapterR[Zhang and Pilanci,2024b](https://arxiv.org/html/2610.12728#bib.bib547)2024ROrbregularnoneO×\\timesOM1idemp\.∙\\bulletSSF[Lian et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib259)2022RLinregularnoneMon×\\timesGLM1→\\toidemp\.SVDiff[Han et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib152)2023RLinregularnoneO×\\timesOM1idemp\.SVF \(Transformer2\)[Sun et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib435)2025RLinregularnoneO×\\timesOM1idemp\.R, HadamardFLoRA[Wen and Chaudhuri,2023](https://arxiv.org/html/2610.12728#bib.bib492)2023RConeapexGLMon×\\timesMonM1group∙\\bulletGLoRA[Chavan et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib57)2023RConeapexGLMon×\\timesMonM1span✓∙\\bulletHiRA[Huang et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib182)2025RConeapexGLMon×\\timesMonM1span✓LieRA[Si et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib418)2025RConeapexGLMon×\\timesMonM1spanLow\-Rank Ternary Adaptation[Manolache et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib313)2026ROrbregularnonlin\.Π\\PiM1idemp\.SMoA[Liu et al\.,2026e](https://arxiv.org/html/2610.12728#bib.bib291)2026RConeapexGLMon×\\timesMonM1spanR, selective∙\\bulletBEFT[Huang et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib179)2025RLinregularnoneGL×\\timesGLM1idemp\.✓∙\\bulletBitFit[Ben\-Zaken et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib27)2021RLinregularnoneGL×\\timesGLM1idemp\.∙\\bulletChild\-Tuning[Xu et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib513)2021RLinregularnoneMon×\\timesMonM1idemp\.DiaBlo[Gurses et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib147)2025RLinregularnoneΠ\\PiM1idemp\.∙\\bulletDiff Pruning[Guo et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib144)2020RConeapexnoneMon×\\timesMonM1spanDiffFit[Xie et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib510)2023RLinregularnoneMon×\\timesGLM1→\\toidemp\.ESFT[Wang et al\.,2024j](https://arxiv.org/html/2610.12728#bib.bib490)2024RLinregularnoneGL×\\timesGLM1idemp\.∙\\bulletFISH Mask[Sung et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib438)2021RLinregularnoneMon×\\timesMonM1idemp\.GPS[Zhang et al\.,2023j](https://arxiv.org/html/2610.12728#bib.bib575)2023RLinregularnoneMon×\\timesMonM1idemp\.KappaTune[Ludwig,2025](https://arxiv.org/html/2610.12728#bib.bib297)2025RLinregularnoneO×\\timesOM1idemp\.✓∙\\bulletLinear probing[Donahue et al\.,2013](https://arxiv.org/html/2610.12728#bib.bib98)2013RLinregularnoneGL×\\timesGLM1idemp\.∙\\bulletLN Tuning[Zhao et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib578)2023RLinregularnoneMon×\\timesGLM1→\\toidemp\.✓LoFiT[Yin et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib535)2024RLinregularnoneGL×\\timesGLM1→\\toidemp\.∙\\bulletLT\-SFT[Ansell et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib12)2021RLinregularnoneMon×\\timesMonM1idemp\.MDPD[Zhang et al\.,2026f](https://arxiv.org/html/2610.12728#bib.bib571)2026RLinregularnoneMon×\\timesGLM1→\\toidemp\.PaFi[Liao et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib264)2023RLinregularnoneMon×\\timesMonM1idemp\.SAM \(sparse\)[Fu et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib117)2022RLinregularnoneMon×\\timesMonM1idemp\.∙\\bulletSHiRA[Bhardwaj et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib30)2024RLinregularnoneMon×\\timesMonM1idemp\.✓SIFT[Song et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib423)2023RLinregularnoneMon×\\timesMonM1idemp\.SMT[He et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib164)2024RLinregularnoneΠ\\PiM1idemp\.Spectrum[Hartford et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib157)2024RLinregularnoneO×\\timesOM1idemp\.SpIEL[Ansell et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib13)2024RConeapexnoneMon×\\timesMonM1spanSPT[He et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib163)2023RConeapexGLMon×\\timesMonM1span∙\\bulletSuper\-Tuning \(Super / Supra\)[Ilin et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib188)2026RLinregularnoneMon×\\timesMonM1idemp\.✓∙\\bulletSurgical FT[Lee et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib239)2022RLinregularnoneGL×\\timesGLM1idemp\.S2FT[Yang et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib527)2024RLinregularnoneMon×\\timesGLM1idemp\.TinyTL[Cai et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib49)2020RLindefectnoneMon×\\timesMonM1idemp\.∙\\bulletTrainable Tokens[team,2025](https://arxiv.org/html/2610.12728#bib.bib449)2025RLinregularnoneMon×\\timesGLM1idemp\.✓E \(pointed ext\.\)AdaMix[Wang et al\.,2022a](https://arxiv.org/html/2610.12728#bib.bib481)2022EFunneutralnoneGL×\\timesGLM∞\\infty–∙\\bulletAdapter \(Houlsby\)[Houlsby et al\.,2019](https://arxiv.org/html/2610.12728#bib.bib175)2019EFunneutralnoneGL×\\timesGLM∞\\infty–Adapter\+[Steitz and Roth,2024](https://arxiv.org/html/2610.12728#bib.bib427)2024EFunneutraltorusGL×\\timesGLM∞\\infty–AdapterDrop[Rücklé et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib394)2020EFunneutraltorusGL×\\timesGLM∞\\infty–AdaptFormer[Chen et al\.,2022a](https://arxiv.org/html/2610.12728#bib.bib65)2022EFunneutraltorusGL×\\timesGLM∞\\infty–AIM[Yang et al\.,2023c](https://arxiv.org/html/2610.12728#bib.bib525)2023EFunneutralnoneGL×\\timesGLM∞\\infty–AP \(Activation Prompt\)[Zhang et al\.,2023i](https://arxiv.org/html/2610.12728#bib.bib565)2023ELinneutralnoneGL×\\timesGLM∞\\infty–CLIP\-Adapter[Gao et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib122)2021EFundefecttorusGL×\\timesGLM∞\\infty–∙\\bulletCompacter[Mahabadi et al\.,2021a](https://arxiv.org/html/2610.12728#bib.bib307)2021EFunneutraltorusΠ\\PiM∞\\infty–∙\\bulletControlNet[Zhang et al\.,2023e](https://arxiv.org/html/2610.12728#bib.bib557)2023EFunneutralGLGL×\\timesGLM∞\\infty–Convpass[Jie and Deng,2022a](https://arxiv.org/html/2610.12728#bib.bib198)2022EFunneutralnoneGL×\\timesGLM∞\\infty–∙\\bulletCustom Diffusion[Kumari et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib229)2022ELinneutralnoneGL×\\timesGLM1idemp\.Initial State Tuning[Galim et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib119)2024ELinneutralnoneGL×\\timesGLM∞\\inftyidemp\.IP\-Adapter[Ye et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib530)2023EFundefecttorusGL×\\timesGLM∞\\infty–K\-Adapter[Wang et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib469)2020EFunneutralGLGL×\\timesGLM∞\\infty–∙\\bulletLLaMA\-Adapter[Zhang et al\.,2023h](https://arxiv.org/html/2610.12728#bib.bib562)2023EFunneutralnoneGL×\\timesGLM∞\\infty–✓LLaMA\-Adapter V2[Gao et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib123)2023EFunneutraltorusMon×\\timesGLM∞\\infty–∙\\bulletLoReFT[Wu et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib505)2024EFundefectGLGL×\\timesGLM∞\\infty–MAD\-X[Pfeiffer et al\.,2020c](https://arxiv.org/html/2610.12728#bib.bib368)2020EFunneutraltorusΠ\\PiM∞\\infty–Mona[Yin et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib534)2024EFundefecttorusMon×\\timesMonM∞\\infty–∙\\bulletParallel Adapter[He et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib165)2021EFunneutraltorusGL×\\timesGLM∞\\infty–∙\\bulletPfeiffer Adapter[Pfeiffer et al\.,2020a](https://arxiv.org/html/2610.12728#bib.bib366)2020EFunneutraltorusGL×\\timesGLM∞\\infty–PHM\-Adapter[Mahabadi et al\.,2021a](https://arxiv.org/html/2610.12728#bib.bib307)2021EFunneutralGLΠ\\PiM∞\\infty–PrefixMemory\-Tuning[Wang et al\.,2025a](https://arxiv.org/html/2610.12728#bib.bib467)2025EFunneutralnoneGL×\\timesGLM∞\\infty–Res\-Tuning[Jiang et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib197)2023EFunneutralnoneGL×\\timesGLM∞\\infty–∙\\bulletShadowPEFT[Li et al\.,2026a](https://arxiv.org/html/2610.12728#bib.bib253)2026EFunneutralGLMon×\\timesMonM∞\\infty–✓Side\-Tuning[Zhang et al\.,2019](https://arxiv.org/html/2610.12728#bib.bib551)2019EFunneutraltorusGL×\\timesGLM∞\\infty–ST\-Adapter[Pan et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib359)2022EFunneutraltorusGL×\\timesGLM∞\\infty–State\-offset Tuning[Kang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib207)2025ELinneutralnoneGL×\\timesGLM∞\\inftyidemp\.T2I\-Adapter[Mou et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib337)2023EFundefecttorusGL×\\timesGLM∞\\infty–∙\\bulletTextual Inversion[Gal et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib118)2022ELinneutralnoneMon×\\timesGLM1idemp\.Tip\-Adapter[Zhang et al\.,2022a](https://arxiv.org/html/2610.12728#bib.bib561)2022EFundefectnoneO×\\timesOM∞\\infty–Visual Prompting[Bahng et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib23)2022ELindefectnoneΠ\\PiM1→\\toidemp\.E0\(unpointed\)AutoPEFT[Zhou et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib592)2023E0Fununpt\.torusGL×\\timesGLM∞\\infty–∙\\bulletAutoPrompt[Shin et al\.,2020](https://arxiv.org/html/2610.12728#bib.bib413)2020E0Fununpt\.noneGL×\\timesGLM∞\\infty–BBT[Sun et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib436)2022E0Fununpt\.noneframeM∞\\infty–∙\\bulletCartridges[Eyuboglu et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib106)2025E0Fununpt\.noneGL×\\timesGLM∞\\infty–✓∙\\bulletCoOp[Zhou et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib594)2021E0Fununpt\.noneGL×\\timesGLM1→\\to–∙\\bulletCPT[Blau et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib39)2024E0Fununpt\.noneO×\\timesOM∞\\infty–✓DePT[Shi and Lipani,2023](https://arxiv.org/html/2610.12728#bib.bib411)2023E0Fununpt\.GLGL×\\timesGLM∞\\infty–EasyControl[Zhang et al\.,2025e](https://arxiv.org/html/2610.12728#bib.bib572)2025E0Fununpt\.GLGL×\\timesGLM∞\\infty–LoRA layer replication \(DUS\)[Kim et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib213)2023E0Fununpt\.GLGL×\\timesGLM1span✓∙\\bulletLST[Sung et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib439)2022E0Fununpt\.GLGL×\\timesGLM∞\\infty–∙\\bulletMAM Adapter[He et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib165)2021E0Fununpt\.GLGL×\\timesGLM∞\\infty–MaPLe[Khattak et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib212)2022E0Fununpt\.nonlin\.GL×\\timesGLM∞\\infty–MEFT[Liao et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib265)2023E0Fununpt\.noneGL×\\timesGLM∞\\infty–∙\\bulletMPT[Wang et al\.,2023d](https://arxiv.org/html/2610.12728#bib.bib485)2023E0Fununpt\.torusGL×\\timesGLM∞\\infty–✓NOAH[Zhang et al\.,2022b](https://arxiv.org/html/2610.12728#bib.bib568)2022E0Fununpt\.GLGL×\\timesGLM∞\\infty–OminiControl[Tan et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib443)2024E0Fununpt\.GLGL×\\timesGLM∞\\infty–∙\\bulletP\-Tuning[Liu et al\.,2021d](https://arxiv.org/html/2610.12728#bib.bib288)2021E0Fununpt\.nonlin\.GL×\\timesGLM∞\\infty–✓∙\\bulletP\-Tuning v2[Liu et al\.,2021c](https://arxiv.org/html/2610.12728#bib.bib287)2021E0Fununpt\.noneGL×\\timesGLM∞\\infty–✓PPT[Gu et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib140)2021E0Fununpt\.noneGL×\\timesGLM∞\\infty–∙\\bulletPrefix\-Tuning[Li and Liang,2021](https://arxiv.org/html/2610.12728#bib.bib252)2021E0Fununpt\.nonlin\.GL×\\timesGLM∞\\infty–✓∙\\bulletPrompt Tuning[Lester et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib243)2021E0Fununpt\.noneGL×\\timesGLM∞\\infty–✓Residual Prompt Tuning[Razdaibiedina et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib384)2023E0Fununpt\.nonlin\.GL×\\timesGLM∞\\infty–SPoT[Vu et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib462)2021E0Fununpt\.noneGL×\\timesGLM∞\\infty–∙\\bulletUniPELT[Mao et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib315)2021E0Fununpt\.GLGL×\\timesGLM∞\\infty–∙\\bulletVPT[Jia et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib194)2022E0Fununpt\.torusGL×\\timesGLM∞\\infty–WARP[Hambardzumyan et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib150)2021E0Fununpt\.noneGL×\\timesGLM∞\\infty–XPrompt[Ma et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib304)2022E0Fununpt\.noneΠ\\PiM∞\\infty–B \(backward\)AdaLomo[Lv et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib302)2023BLinregularnoneGL×\\timesGLnaidemp\.Adam\-mini[Zhang et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib570)2024BLinregularnoneGL×\\timesGLnaidemp\.AGZO[Lin et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib270)2026BLinregularnoneframe–sched\.APOLLO[Zhu et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib596)2024BLinregularnoneGL×\\timesGLnaidemp\.BAdam[Luo et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib300)2024BLinregularnoneGL×\\timesGLnasched\.COLA[Xia et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib507)2024BConeapexGLGL×\\timesGLM1sched\.Delta\-LoRA[Zi et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib600)2023BConeapexGLGL×\\timesGLM1sched\.Fira[Chen et al\.,2024e](https://arxiv.org/html/2610.12728#bib.bib67)2024BLinregularnoneGL×\\timesGLnaidemp\.∙\\bulletFlora[Hao et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib154)2024BLinregularnoneframenasched\.FLoRA \(federated\)[Wang et al\.,2024k](https://arxiv.org/html/2610.12728#bib.bib491)2024BConeapexGLGL×\\timesGLM1sched\.FZOO[Dang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib87)2025BLinregularnoneMon×\\timesMon–sched\.∙\\bulletGaLore[Zhao et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib581)2024BLinregularnoneO×\\timesOnasched\.GaLore 2[Su et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib431)2025BLinregularnoneO×\\timesOnasched\.HiFT[Liu et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib290)2024BLinregularnoneGL×\\timesGLnasched\.LIFT[Liu et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib293)2025BLinregularnoneMon×\\timesMonnasched\.∙\\bulletLISA[Pan et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib360)2024BLinregularnoneGL×\\timesGLnasched\.LOMO[Lv et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib303)2023BLinregularnoneGL×\\timesGLnaidemp\.LoQT[Loeschcke et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib294)2024BLindefectnoneMon×\\timesMonMqsched\.LoRA\-Pre[Wang et al\.,2026c](https://arxiv.org/html/2610.12728#bib.bib487)2026BLinregularnoneGL×\\timesGL––∙\\bulletLP\-FT[Kumar et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib227)2022BLinregularnoneGL×\\timesGLnasched\.∙\\bulletMeZO[Malladi et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib310)2023BLinregularnoneframenasched\.Online Subspace Descent[Liang et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib260)2024BLinregularnoneO×\\timesOnasched\.PeriodicLoRA[Meng et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib325)2024BConeapexGLGL×\\timesGLM1sched\.POET\-X[Qiu et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib383)2026BOrbregularnoneMon×\\timesMonM1sched\.Q\-GaLore[Zhang et al\.,2024e](https://arxiv.org/html/2610.12728#bib.bib574)2024BLindefectnoneMon×\\timesMonnasched\.∙\\bulletReLoRA[Lialin et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib258)2023BConeapexGLGL×\\timesGLM1sched\.ScaLoRA[Zhang et al\.,2025c](https://arxiv.org/html/2610.12728#bib.bib567)2025BCone∗regularGLGL×\\timesGLM1sched\.Sparse Memory Finetuning[Lin et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib267)2025BLinregularnoneMon×\\timesGLnasched\.ZO\-Muon[Lang et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib232)2026BLinregularnoneframe–sched\.P \(post hoc\)AdaMerging[Yang et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib521)2023PLindefectnoneGL×\\timesGLM1idemp\.AdapterSoup[Chronopoulou et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib77)2023PFununpt\.––M∞\\infty–AdaRank[Lee et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib236)2025PConedefecttorusO×\\timesOM1spanCompress\-then\-Merge \(CtM\)[He et al\.,2026c](https://arxiv.org/html/2610.12728#bib.bib169)2026PLinregularnoneO×\\timesOM1idemp\.Core Space[Panariello et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib361)2025PLinregularnoneO×\\timesOM1idemp\.∙\\bulletDARE[Yu et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib538)2023PLinregularnoneMon×\\timesMonM1idemp\.✓DC\-Merge[Zhang et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib549)2026PLinregularnoneO×\\timesOM1idemp\.EFT[Mitchell et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib332)2023PFunneutral–GL×\\timesGLM∞\\infty–Fisher Merging[Matena and Raffel,2021](https://arxiv.org/html/2610.12728#bib.bib320)2021P–unpt\.–Mon×\\timesMonM1–GenKnowSub[Bagherifard et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib21)2025PFununpt\.–O×\\timesOM∞\\infty–✓GradFix[Rinaldi et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib391)2025PLinregularnoneMon×\\timesMonM1idemp\.∙\\bulletIntruder\-dimension reduction[Shuttleworth et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib414)2024PLindefectnoneO×\\timesOM1idemp\.✓∙\\bulletKnOTS[Stoica et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib429)2024PLinregularnoneO×\\timesOM1idemp\.∙\\bulletLCM\-LoRA[Luo et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib301)2023PLinregularnoneGL×\\timesGLM1idemp\.✓∙\\bulletLoRA Soups \(CAT\)[Prabhakar et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib374)2024PLinregularnoneGL×\\timesGLM1idemp\.LoRA Switch / LoRA Composite[Zhong et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib588)2024PFununpt\.––M∞\\infty–LoRA\-LEGO[Zhao et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib585)2024P–unpt\.–O×\\timesOM1–∙\\bulletLoraHub[Huang et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib180)2023PConeapexnoneGL×\\timesGLM1–Mix\-of\-Show[Gu et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib139)2023P–unpt\.–GL×\\timesGLM1–∙\\bulletModel Soups[Wortsman et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib497)2022P–unpt\.–GL×\\timesGLM1–OrthoMerge[Yang et al\.,2026a](https://arxiv.org/html/2610.12728#bib.bib524)2026P–unpt\.–GL×\\timesCOM1–PEM Arithmetic[Zhang et al\.,2023c](https://arxiv.org/html/2610.12728#bib.bib554)2023P–unpt\.–GL×\\timesGLM1–✓Pico[Tang and Yang,2026](https://arxiv.org/html/2610.12728#bib.bib446)2026P–unpt\.–O×\\timesOM1–∙\\bulletProxy\-Tuning[Liu et al\.,2024a](https://arxiv.org/html/2610.12728#bib.bib274)2024PFunneutral–GL×\\timesGLM∞\\infty–RAIN\-Merging[Huang et al\.,2025d](https://arxiv.org/html/2610.12728#bib.bib184)2025PLinregularnoneGL×\\timesCOM1idemp\.RegMean[Jin et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib200)2022P–unpt\.–Mon×\\timesGLM1–∙\\bulletTask Arithmetic[Ilharco et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib187)2022PLinregularnoneGL×\\timesGLM1idemp\.✓∙\\bulletTIES\-Merging[Yadav et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib517)2023PLinregularnoneMon×\\timesMonM1idemp\.✓TSV\-Merge[Gargiulo et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib125)2024PLinregularnoneO×\\timesOM1idemp\.∙\\bulletWiSE\-FT[Wortsman et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib496)2021PLinregularnoneGL×\\timesGLM1idemp\.✓ZipLoRA[Shah et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib404)2023PLindefectnoneMon×\\timesGLM1idemp\.FT\(family\)∙\\bulletAdapterFusion[Pfeiffer et al\.,2020a](https://arxiv.org/html/2610.12728#bib.bib366)2020FTFundefectGLGL×\\timesGLM∞\\infty–∙\\bulletaLoRA[Greenewald et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib136)2025FTFunapexGLGL×\\timesGLM∞\\infty–✓∙\\bulletArrow[Ostapenko et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib355)2024FTFununpt\.––M∞\\infty–✓ATTEMPT[Asai et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib17)2022FTFununpt\.nonlin\.Mon×\\timesMonM∞\\infty–Bayesian\-LoRA[Lin et al\.,2026a](https://arxiv.org/html/2610.12728#bib.bib269)2026FTConedefectnonlin\.GL×\\timesGLM∞\\infty–BLoB[Wang et al\.,2024h](https://arxiv.org/html/2610.12728#bib.bib482)2024FTConeapextorusGL×\\timesGLM∞\\infty–CoA\-LoRA[Ye et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib531)2025FTCone∗defectGLMon×\\timesMonMq–CoCoOp[Zhou et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib595)2022FTFununpt\.torusGL×\\timesGLM∞\\infty–CODA\-Prompt[Smith et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib422)2022FTFununpt\.torusMon×\\timesGLM∞\\infty–DnD[Liang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib262)2025FTConedefectGLGL×\\timesGLM1–Doc\-to\-LoRA \(D2L\)[Charakorn et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib56)2026FTConedefectGLGL×\\timesGLM1–DualPrompt[Wang et al\.,2022b](https://arxiv.org/html/2610.12728#bib.bib489)2022FTFununpt\.torusO×\\timesOM∞\\infty–FourierMoE[Jiang et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib195)2026FTFunneutralnoneΠ\\PiM∞\\infty–GenerativeAdapter[Chen et al\.,2024d](https://arxiv.org/html/2610.12728#bib.bib66)2024FTConedefectGLGL×\\timesGLM1–GOAT[Fan et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib107)2025FTFundefectGL–M∞\\infty–∙\\bulletHydraLoRA[Tian et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib451)2024FTFunneutralGL–M∞\\infty–∙\\bulletHyperDreamBooth[Ruiz et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib393)2023FTConedefectGLframeM1sched\.∙\\bulletHyperFormer[Mahabadi et al\.,2021b](https://arxiv.org/html/2610.12728#bib.bib308)2021FTFundefectGLGL×\\timesGLM∞\\infty–ICEdit[Zhang et al\.,2025f](https://arxiv.org/html/2610.12728#bib.bib573)2025FTFunneutralGL–M∞\\infty–IDPG[Wu et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib506)2022FTFununpt\.GLΠ\\PiM∞\\infty–In\-Place TTT[Feng et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib111)2025FTFunneutraltorusGL×\\timesCOM∞\\infty–K\-LoRA[Ouyang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib356)2025FT–unpt\.–Mon×\\timesMonM1–L2P[Wang et al\.,2021](https://arxiv.org/html/2610.12728#bib.bib488)2021FTFununpt\.torusO×\\timesOM∞\\infty–∙\\bulletLaplace\-LoRA[Yang et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib520)2023FTConeapexGLGL×\\timesGLM∞\\infty–LD\-MoLE[Zhuang et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib599)2025FTFunneutralGLGL×\\timesGLM∞\\infty–∙\\bulletLily[Zhong et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib590)2024FTFunneutralGL–M∞\\infty–✓LiME[Kowsher et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib225)2026FTFunneutralGL×\\timesTΠ\\PiM∞\\infty–LoRA\-Flow[Wang et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib466)2024FTFununpt\.none–M∞\\infty–LoRA\-Mixer[Li et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib251)2025FTFununpt\.GLGL×\\timesGLM∞\\infty–∙\\bulletLoRAMoE[Dou et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib99)2023FTFunneutralGL–M∞\\infty–LoraRetriever[Zhao et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib584)2024FTFununpt\.––M∞\\infty–LPT[Liu et al\.,2022b](https://arxiv.org/html/2610.12728#bib.bib286)2022FTFununpt\.torusGL×\\timesGLM∞\\infty–LRAgent[Jeon et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib192)2026FTConeapexGLGL×\\timesGLM1–MaLoRA / MaRA[Dokme and Heck,2026](https://arxiv.org/html/2610.12728#bib.bib96)2026FTFunneutralGLGL×\\timesGLM∞\\infty–∙\\bulletMHR[Caccia et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib48)2022FTConeapexGLΠ\\PiM1span✓MixLoRA[Li et al\.,2024b](https://arxiv.org/html/2610.12728#bib.bib247)2024FTFunneutralGL–M∞\\infty–MOELoRA[Liu et al\.,2023a](https://arxiv.org/html/2610.12728#bib.bib281)2023FTConeapexGLGL×\\timesGLM1span∙\\bulletMoLE[Wu et al\.,2024c](https://arxiv.org/html/2610.12728#bib.bib501)2024FTFununpt\.scalar–M∞\\infty–MoV / MoLoRA[Zadouri et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib541)2023FTFunneutralnone–M∞\\infty–NaRA[Wang et al\.,2026a](https://arxiv.org/html/2610.12728#bib.bib476)2026FTConeapexGLGL×\\timesGLM1–PERK[Chen et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib71)2025FTConedefectGLGL×\\timesGLM1–∙\\bulletPHATGOOSE[Muqeeth et al\.,2024](https://arxiv.org/html/2610.12728#bib.bib342)2024FTFununpt\.nonlin\.–M∞\\infty–∙\\bulletPolytropon \(Poly\)[Ponti et al\.,2022](https://arxiv.org/html/2610.12728#bib.bib372)2022FTConeapexGLGL×\\timesGLM1span✓PreFT[Lanpouthakoun et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib233)2026FTFunapexGLGL×\\timesGLM∞\\infty–Prompt2Effect[Yang et al\.,2026b](https://arxiv.org/html/2610.12728#bib.bib526)2026FTConedefectGLGL×\\timesGLM1–∙\\bulletPunica[Chen et al\.,2023b](https://arxiv.org/html/2610.12728#bib.bib61)2023FTCone–GLGL×\\timesGLM1–RotMoLE[Sun et al\.,2026](https://arxiv.org/html/2610.12728#bib.bib434)2026FTFunneutralGLGL×\\timesGLM∞\\infty–∙\\bulletS\-LoRA[Sheng et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib410)2023FTCone–GLGL×\\timesGLM1–SHINE[Liu et al\.,2026d](https://arxiv.org/html/2610.12728#bib.bib289)2026FTConedefectGLGL×\\timesGLM1–SmartFed \(MoRE \+ EEQA\)[Wu et al\.,2025b](https://arxiv.org/html/2610.12728#bib.bib502)2025FTFununpt\.noneGL×\\timesGLM∞\\infty–SMEAR[Muqeeth et al\.,2023](https://arxiv.org/html/2610.12728#bib.bib341)2023FTFundefecttorus–M∞\\infty–∙\\bulletText\-to\-LoRA \(T2L\)[Charakorn et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib55)2025FTConeapexGLGL×\\timesGLM1–VAPT[Le et al\.,2025](https://arxiv.org/html/2610.12728#bib.bib235)2025FTFununpt\.torusGL×\\timesGLM∞\\infty–∙\\bulletX\-LoRA[Buehler and Buehler,2024](https://arxiv.org/html/2610.12728#bib.bib45)2024FTFununpt\.none–M∞\\infty–✓Table 37:Every recorded simulation arrowM→NM\\to N\(rules A2–A5 of §[VIII](https://arxiv.org/html/2610.12728#S8)\), with its explicit map\. Notes on the instances \(rank, pointing\) are in the ancillary file arrows\.csv\.SourceMMTargetNNRulePointed maphhwithρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\(IA\)3DoRAA2ℓ↦\(B=0,A=A0,μ=ℓ⊙\(‖w0,i‖\)i\)\\ell\\mapsto\\big\(B=0,\\ A=A\_\{0\},\\ \\mu=\\ell\\odot\(\\\|w\_\{0,i\}\\\|\)\_\{i\}\\big\)\(IA\)3HiRAA3ℓ↦\(ℓ−𝟏,1⊤\)​\(k,v\);ℓ↦\(𝟏,\(ℓ−𝟏\)⊤\)​\(Wdown\)\\ell\\mapsto\(\\ell\-\\mathbf\{1\},\\ \\mathbf\{1\}^\{\\top\}\)\\ \\ \(k,v\);\\hskip 16\.38895pt\\ell\\mapsto\(\\mathbf\{1\},\\ \(\\ell\-\\mathbf\{1\}\)^\{\\top\}\)\\ \\ \(W\_\{\\rm down\}\)ABBALoHaA3h⁡\(B1,A1,B2,A2\)=\(B1,A1,c​B2,A2\),c=α/r1​r2γLoHah\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)=\(B\_\{1\},A\_\{1\},cB\_\{2\},A\_\{2\}\),\\ \\ c=\\frac\{\\alpha/\\sqrt\{r\_\{1\}r\_\{2\}\}\}\{\\gamma\_\{\\rm LoHa\}\}ABBALoRAA3h=\(αr1​r2​B1∙B2,\(A1⊤∙A2⊤\)⊤\)h=\\big\(\\tfrac\{\\alpha\}\{\\sqrt\{r\_\{1\}r\_\{2\}\}\}\\,B\_\{1\}\\bullet B\_\{2\},\\;\(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\}\\big\)AdaLomoLOMOA3h=idΘh=\\mathrm\{id\}\_\{\\Theta\}AdaLoRALoRAA3h⁡\(P,Λ,Q\)=\(P​Λ,Q\)h\(P,\\Lambda,Q\)=\(P\\Lambda,\\ Q\)AdaLoRAPoLARA3h\(P,Λ,Q\)=\(P\(P⊤P\)−1/2,\(P⊤P\)1/2Λ\(QQ⊤\)1/2,Q⊤\(QQ⊤\)−1/2\)h\(P,\\Lambda,Q\)=\\big\(P\(P^\{\\top\}P\)^\{\-1/2\},\\ \(P^\{\\top\}P\)^\{1/2\}\\Lambda\(QQ^\{\\top\}\)^\{1/2\},\\ Q^\{\\top\}\(QQ^\{\\top\}\)^\{\-1/2\}\\big\)Adam\-miniLOMOA3h=idΘh=\\mathrm\{id\}\_\{\\Theta\}AdaMergingLoRA Soups \(CAT\)A3h=id:λil↦αilh=\\mathrm\{id\}:\\ \\lambda\_\{i\}^\{l\}\\mapsto\\alpha\_\{i\}^\{l\}AdaMixAdapter \(Houlsby\)A3h⁡\(\(Wkd​o​w​n\)k≤M,Wu​p\)=\(1M​∑kWkd​o​w​n,Wu​p\)h\\big\(\(W^\{down\}\_\{k\}\)\_\{k\\leq M\},\\,W^\{up\}\\big\)=\\big\(\\tfrac\{1\}\{M\}\\textstyle\\sum\_\{k\}W^\{down\}\_\{k\},\\ W^\{up\}\\big\)AdaMSSLoRAA3h\(\(Ck,Bk\)k\)=\(\[E𝒮1C1⋯E𝒮KCK\],\[B1A~1;…;BKA~K\]\)h\(\(C\_\{k\},B\_\{k\}\)\_\{k\}\)=\\big\(\[E\_\{\\mathcal\{S\}\_\{1\}\}C\_\{1\}\\cdots E\_\{\\mathcal\{S\}\_\{K\}\}C\_\{K\}\],\\ \[B\_\{1\}\\tilde\{A\}\_\{1\};\\dots;B\_\{K\}\\tilde\{A\}\_\{K\}\]\\big\)AdaPaDLoRAA3h=\(\[bi,1⋯bi,ri\],\[ai,1⊤;…;ai,ri⊤\]\)h=\\big\(\[\\,b\_\{i,1\}\\ \\cdots\\ b\_\{i,r\_\{i\}\}\\,\],\\ \[\\,a\_\{i,1\}^\{\\top\};\\dots;a\_\{i,r\_\{i\}\}^\{\\top\}\\,\]\\big\)AdapterDropAdapter \(Houlsby\)A2h⁡\(q\>n\)=\(\(Ul=0,Dl=Dl0\)l≤n,q\>n\)h\(q\_\{\>n\}\)=\\big\(\(U\_\{l\}\{=\}0,\\,D\_\{l\}\{=\}D\_\{l\}^\{0\}\)\_\{l\\leq n\},\\ q\_\{\>n\}\\big\)AdapterDropPfeiffer AdapterA2h⁡\(q\>n\)=\(\(Ul=0,Dl=Dl0\)l≤n,q\>n\)h\(q\_\{\>n\}\)=\\big\(\(U\_\{l\}\{=\}0,\\,D\_\{l\}\{=\}D\_\{l\}^\{0\}\)\_\{l\\leq n\},\\ q\_\{\>n\}\\big\)AdaptFormerParallel AdapterA3h⁡\(Wd​o​w​n,bd​o​w​n,Wu​p,bu​p\)=\(Wd​o​w​n,bd​o​w​n,0\.1s​Wu​p,0\.1s​bu​p\)h\(W\_\{down\},b\_\{down\},W\_\{up\},b\_\{up\}\)=\\big\(W\_\{down\},b\_\{down\},\\tfrac\{0\.1\}\{s\}W\_\{up\},\\tfrac\{0\.1\}\{s\}b\_\{up\}\\big\)AGZOLoRA\-FAA3λ↦λ​Rt\\lambda\\mapsto\\lambda R\_\{t\}ALoRALoRAA3h⁡\(B,A\)=\(B​diag​\(α\),A\)h\(B,A\)=\(B\\,\\mathrm\{diag\}\(\\alpha\),\\ A\)AltLoRALoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}AltLoRARiemannian LoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}AP \(Activation Prompt\)BitFitA3δ=𝟏⊗v↦Δ​bdown\(l−1\)=v\\delta=\\mathbf\{1\}\\otimes v\\ \\mapsto\\ \\Delta b\_\{\\rm down\}^\{\(l\-1\)\}=vAR1\-ZOLoRAA3h⁡\(B,A\)=\(αs​B,A\)h\(B,A\)=\\big\(\\tfrac\{\\alpha\}\{s\}B,\\ A\\big\)ArrowLoRAA3hx​\(∗\)=\(∑ipi​\(x\)​Bi,∑ipi​\(x\)​Ai\)h\_\{x\}\(\\ast\)=\\big\(\\textstyle\\sum\_\{i\}p\_\{i\}\(x\)B\_\{i\},\\ \\sum\_\{i\}p\_\{i\}\(x\)A\_\{i\}\\big\)ArrowLoRA Soups \(CAT\)A3hx​\(∗\)=\(αi=pi​\(x\)​si\)ih\_\{x\}\(\\ast\)=\\big\(\\alpha\_\{i\}=p\_\{i\}\(x\)\\,s\_\{i\}\\big\)\_\{i\}AstraLoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)AsyLoRALoRAA2h⁡\(B\)=\(B,Q\)h\(B\)=\(B,\\ Q\)AsyLoRALoRA\-FAA3h⁡\(B\)=B​Q​A0\+h\(B\)=B\\,QA\_\{0\}^\{\+\}AutoLoRALoRAA3h⁡\(b,a,β\)=\(\[softmax​\(β\)j​bj\]j,\[aj⊤\]j\)h\(b,a,\\beta\)=\\big\(\[\\mathrm\{softmax\}\(\\beta\)\_\{j\}b\_\{j\}\]\_\{j\},\\ \[a\_\{j\}^\{\\top\}\]\_\{j\}\\big\)AutoPromptPrompt TuningA3h⁡\(w1,…,wℓ\)=\(Ew1,…,Ewℓ\)h\(w\_\{1\},\\dots,w\_\{\\ell\}\)=\(E\_\{w\_\{1\}\},\\dots,E\_\{w\_\{\\ell\}\}\)Bayesian\-LoRALoRAA3h=\(B¯,A¯\),B¯=TrB​𝔼q​\[UB\]​TcB,A¯=TrA​𝔼q​\[UA\]​TcAh=\\big\(\\bar\{B\},\\ \\bar\{A\}\\big\),\\ \\ \\bar\{B\}=T\_\{r\}^\{B\}\\,\\mathbb\{E\}\_\{q\}\[U^\{B\}\]\\,T\_\{c\}^\{B\},\\ \\ \\bar\{A\}=T\_\{r\}^\{A\}\\,\\mathbb\{E\}\_\{q\}\[U^\{A\}\]\\,T\_\{c\}^\{A\}BBTPrompt TuningA3h⁡\(z\)=A​z\+p0h\(z\)=Az\+p\_\{0\}BD\-LoRAGraLoRAA3h\(\{Bi\},A\)=\(Bi​j=Bi,Ai​j⊤=A:,Jj\(i\)\)i,j≤Nh\(\\\{B\_\{i\}\\\},A\)=\\big\(B\_\{ij\}=B\_\{i\},\\;A\_\{ij\}^\{\\top\}=A^\{\(i\)\}\_\{:,J\_\{j\}\}\\big\)\_\{i,j\\leq N\}BD\-LoRALoRAA3h⁡\(B1,…,BN,A\)=\(diag⁡\(B1,…,BN\),A\)h\(B\_\{1\},\\dots,B\_\{N\},A\)=\\big\(\\mathrm\{diag\}\(B\_\{1\},\\dots,B\_\{N\}\),\\;A\\big\)BEFTBitFitA2h⁡\(Δ​bv\)=\(Δ​bv,0other biases\)h\(\\Delta b\_\{v\}\)=\\big\(\\Delta b\_\{v\},\\,0\_\{\\text\{other biases\}\}\\big\)BiDoRADoRAA3id\\mathrm\{id\}BitFitSSFA2h⁡\(Δ​b\)=\(γ=𝟏,β=Δ​b\)h\(\\Delta b\)=\\big\(\\gamma=\\mathbf\{1\},\\ \\beta=\\Delta b\\big\)BLoBLoRAA3h⁡\(M,G,B\)=\(B,M\)h\(M,G,B\)=\(B,\\ M\)BOFTBOFTA2\(Q1,…,Qm′,s\)↦\(Q1,…,Qm′,0,s\)\(Q\_\{1\},\\dots,Q\_\{m^\{\\prime\}\},s\)\\mapsto\(Q\_\{1\},\\dots,Q\_\{m^\{\\prime\}\},0,s\)CartridgesPrefix\-TuningA2h⁡\(Z\)=\(\[zsink\(l\);Zk\(l\)\],\[zv,sink\(l\);Zv\(l\)\]\)l≤Lh\(Z\)=\\big\(\[z^\{\(l\)\}\_\{\\mathrm\{sink\}\};Z\_\{k\}^\{\(l\)\}\],\\,\[z^\{\(l\)\}\_\{v,\\mathrm\{sink\}\};Z\_\{v\}^\{\(l\)\}\]\\big\)\_\{l\\leq L\}Child\-TuningDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)Child\-TuningFISH MaskA3idℝM\\mathrm\{id\}\_\{\\mathbb\{R\}^\{M\}\}CoA\-LoRALQ\-LoRAA3h𝐂​\(θ,L1,L2\)=\(L1​\(Ir\+Uθ​\(Q𝐂\)\),L2\)h\_\{\\mathbf\{C\}\}\(\\theta,L\_\{1\},L\_\{2\}\)=\\big\(L\_\{1\}\\,\(I\_\{r\}\+U\_\{\\theta\}\(Q\_\{\\mathbf\{C\}\}\)\),\\ L\_\{2\}\\big\)COLALoRAA3h\(\(Bi,Ai\)i\)=\(\[B1⋯BM\],\[A1;…;AM\]\)h\(\(B\_\{i\},A\_\{i\}\)\_\{i\}\)=\(\[B\_\{1\}\\cdots B\_\{M\}\],\\,\[A\_\{1\};\\dots;A\_\{M\}\]\)COLAReLoRAA3h=id​\(stagewise\)h=\\mathrm\{id\}\\ \\text\{\(stagewise\)\}CompacterAdapter \(Houlsby\)A3h=\(Wd​o​w​n=∑iAi⊗siD​tiD⊤,Wu​p=∑iAi⊗siU​tiU⊤\)h=\\big\(W\_\{down\}=\\textstyle\\sum\_\{i\}A\_\{i\}\\otimes s\_\{i\}^\{D\}t\_\{i\}^\{D\\top\},\\;W\_\{up\}=\\sum\_\{i\}A\_\{i\}\\otimes s\_\{i\}^\{U\}t\_\{i\}^\{U\\top\}\\big\)CompacterPHM\-AdapterA3h\(\{Ai\},\{si∙,ℓ,ti∙,ℓ\}\)=\(Ai∙,ℓ=Ai,Bi∙,ℓ=si∙,ℓti∙,ℓ⊤\)ℓ,∙∈\{D,U\}h\\big\(\\\{A\_\{i\}\\\},\\\{s\_\{i\}^\{\\bullet,\\ell\},t\_\{i\}^\{\\bullet,\\ell\}\\\}\\big\)=\\big\(A\_\{i\}^\{\\bullet,\\ell\}=A\_\{i\},\\;B\_\{i\}^\{\\bullet,\\ell\}=s\_\{i\}^\{\\bullet,\\ell\}t\_\{i\}^\{\\bullet,\\ell\\top\}\\big\)\_\{\\ell,\\ \\bullet\\in\\\{D,U\\\}\}Compress\-then\-Merge \(CtM\)LoRAA3λ↦\(U​∑tλt​O\(t\),V⊤\)\\lambda\\mapsto\\big\(U\\textstyle\\sum\_\{t\}\\lambda\_\{t\}O^\{\(t\)\},\\ V^\{\\top\}\\big\)Concept SlidersLoRAA3h=idh=\\mathrm\{id\}Concept SlidersTask ArithmeticA3h⁡\(α1,…,αK\)=\(λk=αk\),τk=Bk​Akh\(\\alpha\_\{1\},\\dots,\\alpha\_\{K\}\)=\(\\lambda\_\{k\}=\\alpha\_\{k\}\),\\hskip 8\.19447pt\\tau\_\{k\}=B\_\{k\}A\_\{k\}CoOpCoCoOpA2h⁡\(V\)=\(V,θ​with​W2=0,b2=0\)h\(V\)=\\big\(V,\\ \\theta\\ \\text\{with\}\\ W\_\{2\}=0,\\ b\_\{2\}=0\\big\)CoOpPrompt TuningA3h=idℝℓ×dh=\\mathrm\{id\}\_\{\\mathbb\{R\}^\{\\ell\\times d\}\}CorDALoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)Core SpaceTask ArithmeticA3h=idλh=\\mathrm\{id\}\_\{\\lambda\}CPTPrompt TuningA2h⁡\(Ectx\)=\(Ectx,Elabel0\)​\(‖Ectx−Ectx0‖≤ε\)h\(E\_\{\\mathrm\{ctx\}\}\)=\\big\(E\_\{\\mathrm\{ctx\}\},\\,E^\{0\}\_\{\\mathrm\{label\}\}\\big\)\\hskip 8\.19447pt\(\\\|E\_\{\\mathrm\{ctx\}\}\-E^\{0\}\_\{\\mathrm\{ctx\}\}\\\|\\leq\\varepsilon\)DC\-MergeLoRAA3h⁡\(α\)=\(α​U~​\(M~⊙ℳ\),V~⊤\)h\(\\alpha\)=\\big\(\\alpha\\,\\widetilde\{U\}\(\\widetilde\{M\}\\odot\\mathcal\{M\}\),\\ \\widetilde\{V\}^\{\\top\}\\big\)DEFTLoRAA3\(P,R\)↦\(P,R−relu\(P\)⊤W0\)\(QR:\(Q\(P\),R−Q\(P\)⊤W0\)\)\(P,R\)\\mapsto\\big\(P,\\ R\-\\mathrm\{relu\}\(P\)^\{\\top\}W\_\{0\}\\big\)\\hskip 8\.19447pt\(\\text\{QR: \}\(Q\(P\),\\ R\-Q\(P\)^\{\\top\}W\_\{0\}\)\)DeLoRADoRAA3\(B,A,λ\)↦\(B′,A′,μ=\(∥\(W0\+B′A′\)i,:∥\)i\),\(B′,A′\)as in the LoRA arrow\(B,A,\\lambda\)\\mapsto\\big\(B^\{\\prime\},A^\{\\prime\},\\ \\mu=\(\\\|\(W\_\{0\}\+B^\{\\prime\}A^\{\\prime\}\)\_\{i,:\}\\\|\)\_\{i\}\\big\),\\ \(B^\{\\prime\},A^\{\\prime\}\)\\ \\text\{as in the LoRA arrow\}DeLoRALoRAA3\(B,A,λ\)↦\(λr​B​diag​\(1/max⁡\(‖bi‖,ε\)\),diag⁡\(1/‖ai‖\)​A​Dw\)\(B,A,\\lambda\)\\mapsto\\Big\(\\tfrac\{\\lambda\}\{r\}\\,B\\,\\mathrm\{diag\}\\big\(1/\\max\(\\\|b\_\{i\}\\\|,\\varepsilon\)\\big\),\\ \\mathrm\{diag\}\\big\(1/\\\|a\_\{i\}\\\|\\big\)\\,A\\,D\_\{w\}\\Big\)Delta\-LoRALoRAA3h⁡\(B,A\)=\(\[\(1\+λ\)​B,−λ​BK\],\[A;AK\]\)h\(B,A\)=\\big\(\[\(1\+\\lambda\)B,\\ \-\\lambda B\_\{K\}\],\\ \[A;\\ A\_\{K\}\]\\big\)DiaBloDiff PruningA2D↦\(α0,ιblk​D\)D\\mapsto\\big\(\\alpha\_\{0\},\\ \\iota\_\{\\rm blk\}D\\big\)DiaBloFISH MaskA3h=id,M=supp​blkdiag​\(𝟏do​u​t/N×di​n/N,…\)h=\\mathrm\{id\},\\hskip 16\.38895ptM=\\mathrm\{supp\}\\,\\mathrm\{blkdiag\}\(\\mathbf\{1\}\_\{d\_\{out\}/N\\times d\_\{in\}/N\},\\dots\)DnDLoRAA3hc​\(ϕ\)=detok⁡\(Gϕ​\(c\)\)h\_\{c\}\(\\phi\)=\\mathrm\{detok\}\\big\(G\_\{\\phi\}\(c\)\\big\)Doc\-to\-LoRA \(D2L\)LoRAA3hc\(ϕ\)=\(αl\[Bl\(1\)⋯Bl\(K\)\],\[Al\(1\);…;Al\(K\)\]\)h\_\{c\}\(\\phi\)=\\big\(\\alpha\_\{l\}\[B^\{\(1\)\}\_\{l\}\\ \\cdots\\ B^\{\(K\)\}\_\{l\}\],\\ \[A^\{\(1\)\}\_\{l\};\\dots;A^\{\(K\)\}\_\{l\}\]\\big\)DoRABiDoRAA3id\\mathrm\{id\}DoRAGLoRAA3\(B,A,μ\)↦\(BA=c−𝟏,AA=𝟏⊤;BB=diag\(c\)sB,AB=A\),c=μ⊘\(∥\(W0\+sBA\)i,:∥\)i\(B,A,\\mu\)\\mapsto\\big\(B\_\{A\}=c\-\\mathbf\{1\},\\ A\_\{A\}=\\mathbf\{1\}^\{\\top\};\\ B\_\{B\}=\\mathrm\{diag\}\(c\)\\,sB,\\ A\_\{B\}=A\\big\),\\hskip 8\.19447ptc=\\mu\\oslash\\big\(\\\|\(W\_\{0\}\+sBA\)\_\{i,:\}\\\|\\big\)\_\{i\}DoRA \(dynamic rank\)AdaLoRAA3h=idh=\\mathrm\{id\}DoRA \(dynamic rank\)LoRAA3h⁡\(B,c,A\)=\(B​diag​\(c\),A\)h\(B,c,A\)=\(B\\,\\mathrm\{diag\}\(c\),\\ A\)DoRA \(dynamic rank\)SoRAA3h⁡\(B,c,A\)=\(B​diag​\(c\)​diag​\(g0\)−1,g0,A\)h\(B,c,A\)=\\big\(B\\,\\mathrm\{diag\}\(c\)\\,\\mathrm\{diag\}\(g\_\{0\}\)^\{\-1\},\\ g\_\{0\},\\ A\\big\)DSEELoRAA3\(U,V,S\)↦\(\[U∣EIdiag\(SΩ\)\],\[V;EJ⊤\]\),EI=\[ei1⋯eic\],EJ=\[ej1⋯ejc\]\(U,V,S\)\\mapsto\\big\(\[\\,U\\mid E\_\{I\}\\,\\mathrm\{diag\}\(S\_\{\\Omega\}\)\\,\],\\ \[\\,V;\\ E\_\{J\}^\{\\top\}\\,\]\\big\),\\hskip 8\.19447ptE\_\{I\}=\[e\_\{i\_\{1\}\}\\cdots e\_\{i\_\{c\}\}\],\\ E\_\{J\}=\[e\_\{j\_\{1\}\}\\cdots e\_\{j\_\{c\}\}\]DyLoRADyLoRAA2h\(Bb,Ab\)=\(\[rmaxbBb,0\],\[Ab;A0,\(b\+1\):\]\)h\(B\_\{b\},A\_\{b\}\)=\\big\(\[\\tfrac\{r\_\{\\max\}\}\{b\}B\_\{b\},\\ 0\],\\ \[A\_\{b\};\\ A\_\{0,\(b\+1\):\}\]\\big\)DyLoRALoRAA3h=idh=\\mathrm\{id\}EasyControlPrefix\-TuningA3hc​\(B,A\)=\(Kl​\(c,B,A\),Vl​\(c,B,A\)\)l≤Lh\_\{c\}\(B,A\)=\\big\(K\_\{l\}\(c;B,A\),\\ V\_\{l\}\(c;B,A\)\\big\)\_\{l\\leq L\}EFTProxy\-TuningA3h=idh=\\mathrm\{id\}ElaLoRAAdaLoRAA3h=idh=\\mathrm\{id\}ElaLoRALoRAA3h⁡\(P,Λ,Q\)=\(P​Λ,Q\)h\(P,\\Lambda,Q\)=\(P\\Lambda,\\ Q\)ETHER\+LoRAA3q↦\(\[\[v~^j,−u~^j\]j,W0​\[v^j,−u^j\]j\],\[\[v~^j,u~^j\]j⊤​W0​H\+;\[v^j,u^j\]j⊤\]\)q\\mapsto\\Big\(\\big\[\\,\[\\hat\{\\tilde\{v\}\}\_\{j\},\-\\hat\{\\tilde\{u\}\}\_\{j\}\]\_\{j\},\\ W\_\{0\}\[\\hat\{v\}\_\{j\},\-\\hat\{u\}\_\{j\}\]\_\{j\}\\,\\big\],\\ \\big\[\\,\[\\hat\{\\tilde\{v\}\}\_\{j\},\\hat\{\\tilde\{u\}\}\_\{j\}\]\_\{j\}^\{\\top\}W\_\{0\}H^\{\+\};\\ \[\\hat\{v\}\_\{j\},\\hat\{u\}\_\{j\}\]\_\{j\}^\{\\top\}\\,\\big\]\\Big\)EVALoRAA3h=id:\(B,A\)↦\(B,A\)h=\\mathrm\{id\}:\(B,A\)\\mapsto\(B,A\)FacTFacTA3h⁡\(U,P,C,V\)=\(U,\(∑tPi,t​Ct\)i,V\)h\(U,P,C,V\)=\\big\(U,\\ \(\\textstyle\\sum\_\{t\}P\_\{i,t\}C\_\{t\}\)\_\{i\},\\ V\\big\)FacTLoRAA3h⁡\(U,Σ,V\)=\(s​U​Σi,V⊤\)ih\(U,\\Sigma,V\)=\\big\(sU\\Sigma\_\{i\},\\ V^\{\\top\}\\big\)\_\{i\}FedRot\-LoRALoRAA3h=idh=\\mathrm\{id\}FFA\-LoRALoRAA2h⁡\(B\)=\(B,A0\)h\(B\)=\(B,\\ A\_\{0\}\)FFA\-LoRALoRA\-FAA3h=idh=\\mathrm\{id\}FISH MaskDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)FISH MaskRoSAA5x↦\(\(0,A0\),x\)x\\mapsto\\big\(\(0,A\_\{0\}\),\\ x\\big\)Flat\-LoRALoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}FlexLoRAAdaLoRAA3h=idh=\\mathrm\{id\}FlexLoRAElaLoRAA3h=idh=\\mathrm\{id\}FlexLoRALoRAA3h⁡\(P,Λ,Q\)=\(P​Λ,Q\)h\(P,\\Lambda,Q\)=\(P\\Lambda,\\ Q\)FLoRAHiRAA3h⁡\(B,A\)=\(\[1r​B−𝟏m\],\[A;𝟏n⊤\]\)h\(B,A\)=\\big\(\[\\tfrac\{1\}\{r\}B\\;\\,\{\-\\mathbf\{1\}\_\{m\}\}\],\\,\[A;\\mathbf\{1\}\_\{n\}^\{\\top\}\]\\big\)FloraLoRA\-FAA3C↦s−1​C​\(A0:=At\)C\\mapsto s^\{\-1\}C\\hskip 8\.19447pt\(A\_\{0\}:=A\_\{t\}\)FloraReLoRAA2C↦\(s−1​C,At\)C\\mapsto\(s^\{\-1\}C,\\ A\_\{t\}\)FLoRA \(federated\)ReLoRAA3h\(\(Bk,Ak\)k\)=\(\[p1B1⋯pKBK\],\[A1;…;AK\]\)per roundh\\big\(\(B\_\{k\},A\_\{k\}\)\_\{k\}\\big\)=\\big\(\[p\_\{1\}B\_\{1\}\\cdots p\_\{K\}B\_\{K\}\],\\ \[A\_\{1\};\\dots;A\_\{K\}\]\\big\)\\ \\ \\text\{per round\}FLoRA \(Tucker core\)LoRAA3h⁡\(A,G,B\)=\(s​A​G,B⊤\)h\(A,G,B\)=\\big\(sAG,\\;B^\{\\top\}\\big\)FLoRGLoRAA3A↦\(s​L​A⊤,A​R\)A\\mapsto\\big\(s\\,LA^\{\\top\},\\ AR\\big\)FLoRGSingLoRAA3A↦s​r/α​A⊤A\\mapsto\\sqrt\{sr/\\alpha\}\\,A^\{\\top\}FourierFTFourierFTA2h⁡\(cΩ\)=\(cΩ,0Ω′∖Ω\)h\(c\_\{\\Omega\}\)=\(c\_\{\\Omega\},\\ 0\_\{\\Omega^\{\\prime\}\\setminus\\Omega\}\)FourierFTLoRA\-FAA3h\(c\)=αm​nBFdiag\(c,−c\),BF=\[cosθksinθk\]k,AF=\[cosφk;sinφk\]kh\(c\)=\\tfrac\{\\alpha\}\{mn\}\\,B\_\{F\}\\,\\mathrm\{diag\}\(c,\-c\),\\ \\ B\_\{F\}=\[\\cos\\theta\_\{k\}\\ \\sin\\theta\_\{k\}\]\_\{k\},\\ A\_\{F\}=\[\\cos\\varphi\_\{k\};\\ \\sin\\varphi\_\{k\}\]\_\{k\}FourierMoELoRA\-FAA3hx\(\(ci\)i,Φ\)=η∑i∈𝒮Φ​\(x\)gΦ,i\(x\)B\(ci\),B\(c\)=2do​u​t​di​n\[akcosθk−bksinθk,−\(aksinθk\+bkcosθk\)\]kh\_\{x\}\\big\(\(c\_\{i\}\)\_\{i\},\\Phi\\big\)=\\eta\\textstyle\\sum\_\{i\\in\\mathcal\{S\}\_\{\\Phi\}\(x\)\}g\_\{\\Phi,i\}\(x\)\\,B\(c\_\{i\}\),\\ \\ B\(c\)=\\tfrac\{2\}\{d\_\{out\}d\_\{in\}\}\\big\[\\,a\_\{k\}\\cos\\theta\_\{k\}\-b\_\{k\}\\sin\\theta\_\{k\},\\ \-\(a\_\{k\}\\sin\\theta\_\{k\}\+b\_\{k\}\\cos\\theta\_\{k\}\)\\,\\big\]\_\{k\}FrameFTLoRA\-FAA3h\(CΩ\)=αdo​u​t​di​nPo​u​tC:,J,A0=\(Pi​n\):,J⊤h\(C\_\{\\Omega\}\)=\\tfrac\{\\alpha\}\{\\sqrt\{d\_\{out\}d\_\{in\}\}\}\\,P\_\{out\}\\,C\_\{:,J\},\\hskip 16\.38895ptA\_\{0\}=\(P\_\{in\}\)\_\{:,J\}^\{\\top\}FRoDFRoDA2h⁡\(Σ\)=\(Σ,S=0\)h\(\\Sigma\)=\(\\Sigma,\\ S=0\)GaLoreLoRAA2C↦\(Pk,s−1​C\)C\\mapsto\(P\_\{k\},\\ s^\{\-1\}C\)GaLoreLoRA\-FAA3C↦s−1​C​\(A0:=Pk⊤\)C\\mapsto s^\{\-1\}C\\hskip 8\.19447pt\(A\_\{0\}:=P\_\{k\}^\{\\top\}\)GaLore 2GaLoreA3id\\mathrm\{id\}GenerativeAdapterLoRAA3hc​\(A1,A2,B1,B2\)=\(A1​Ur​\(c\),Vr​\(c\)⊤​B2\),Ur​Σr​Vr⊤=SVDr​\(A2​Hc⊤​Hc​B1\)h\_\{c\}\(A\_\{1\},A\_\{2\},B\_\{1\},B\_\{2\}\)=\\big\(A\_\{1\}U\_\{r\}\(c\),\\ V\_\{r\}\(c\)^\{\\top\}B\_\{2\}\\big\),\\ \\ U\_\{r\}\\Sigma\_\{r\}V\_\{r\}^\{\\top\}=\\mathrm\{SVD\}\_\{r\}\(A\_\{2\}H\_\{c\}^\{\\top\}H\_\{c\}B\_\{1\}\)GeoRALoRAA3\(B,A\)↦\(\[B,−B0\],\[A;A0\]\)\(B,A\)\\mapsto\\big\(\[B,\\ \-B\_\{0\}\],\\ \[A;\\ A\_\{0\}\]\\big\)GiVALoRAA3\(γ,λ\)↦\(Γ​B​Λ,Vr⊤\)\(\\gamma,\\lambda\)\\mapsto\(\\Gamma B\\Lambda,\\ V\_\{r\}^\{\\top\}\)GiVALoRA\-FAA3\(γ,λ\)↦Γ​B​Λ\(\\gamma,\\lambda\)\\mapsto\\Gamma B\\LambdaGOATLoRAA3hx\(B,A,Wz\)=\(\[sw1\(x\)B1⋯swE\(x\)BE−sEB01⋯−sEB0E\],\[A1;…;AE;A01;…;A0E\]\)h\_\{x\}\(B,A,W\_\{z\}\)=\\big\(\[s\\,w^\{1\}\(x\)B^\{1\}\\ \\cdots\\ s\\,w^\{E\}\(x\)B^\{E\}\\ \\ \-\\tfrac\{s\}\{E\}B\_\{0\}^\{1\}\\ \\cdots\\ \-\\tfrac\{s\}\{E\}B\_\{0\}^\{E\}\],\\ \[A^\{1\};\\dots;A^\{E\};A\_\{0\}^\{1\};\\dots;A\_\{0\}^\{E\}\]\\big\)GOFTBOFTA3θ↦\(t=tan⁡\(θ/2\)​on the kept​2×2​blocks,0​on the others,s=𝟏\)\\theta\\mapsto\\big\(t=\\tan\(\\theta/2\)\\ \\text\{on the kept \}2\\times 2\\text\{ blocks\},\\ 0\\ \\text\{on the others\},\\ s=\\mathbf\{1\}\\big\)GOFTqGOFTA3θ↦\(G~t=\(cos⁡θt−sin⁡θtsin⁡θtcos⁡θt\)\)t\\theta\\mapsto\\Big\(\\tilde\{G\}\_\{t\}=\\begin\{pmatrix\}\\cos\\theta\_\{t\}&\-\\sin\\theta\_\{t\}\\\\ \\sin\\theta\_\{t\}&\\cos\\theta\_\{t\}\\end\{pmatrix\}\\Big\)\_\{t\}GoRALoRAA3h⁡\(B,A\)=\(c​B,A\),c=α/rℓsh\(B,A\)=\(cB,A\),\\ c=\\tfrac\{\\alpha/\\sqrt\{r\_\{\\ell\}\}\}\{s\}GPSDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)GradFixFISH MaskA3h⁡\(α\)=α⁡\(m⊙τA\)h\(\\alpha\)=\\alpha\\,\(m\\odot\\tau\_\{A\}\)GraLoRALoRAA3h\(\{Ui​j,Vi​j\}\)=\(blockdiag\(L1,…,Lk\),S\),Li=\[Ui​1⋯Ui​k\],S=stack of theVi​j⊤on column blockjh\(\\\{U\_\{ij\},V\_\{ij\}\\\}\)=\\big\(\\mathrm\{blockdiag\}\(L\_\{1\},\\dots,L\_\{k\}\),\\;S\\big\),\\ L\_\{i\}=\[U\_\{i1\}\\cdots U\_\{ik\}\],\\ S=\\text\{stack of the \}V\_\{ij\}^\{\\top\}\\text\{ on column block \}jHetLoRADyLoRAA3h=idh=\\mathrm\{id\}HetLoRALoRAA3h=idh=\\mathrm\{id\}HiFTSurgical FTA3idΘGj\\mathrm\{id\}\_\{\\Theta\_\{G\_\{j\}\}\}HiRAGLoRAA2h⁡\(B,A\)=\(𝒜​\-path=B​A,all other GLoRA paths frozen at​0\)h\(B,A\)=\\big\(\\mathcal\{A\}\\text\{\-path\}=BA,\\ \\text\{all other GLoRA paths frozen at \}0\\big\)HiRALoHaA2h⁡\(B,A\)=\(U0,V0,B,A\),U0​V0=W0/γ​a fixed rank\-​rk​W0​factorisationh\(B,A\)=\(U\_\{0\},V\_\{0\},B,A\),\\ \\ U\_\{0\}V\_\{0\}=W\_\{0\}/\\gamma\\ \\text\{a fixed rank\-\}\\mathrm\{rk\}W\_\{0\}\\text\{ factorisation\}HRALoRAA3u↦\(W0B\(u\),A\(u\)\),B\(u\)=\[−2Hu1⋯Hui−1u^i\]i=1r,A\(u\)=\[u^1⊤;…;u^r⊤\]u\\mapsto\\big\(W\_\{0\}B\(u\),\\ A\(u\)\\big\),\\hskip 8\.19447ptB\(u\)=\\big\[\-2H\_\{u\_\{1\}\}\\cdots H\_\{u\_\{i\-1\}\}\\hat\{u\}\_\{i\}\\big\]\_\{i=1\}^\{r\},\\ A\(u\)=\\big\[\\hat\{u\}\_\{1\}^\{\\top\};\\dots;\\hat\{u\}\_\{r\}^\{\\top\}\\big\]HydraLoRALoRAA3hx​\(A,B1,…,BN,Wg\)=\(∑iωi​\(x\)​Bi,A\)h\_\{x\}\(A,B\_\{1\},\\dots,B\_\{N\},W\_\{g\}\)=\\big\(\\textstyle\\sum\_\{i\}\\omega\_\{i\}\(x\)B\_\{i\},\\ A\\big\)HydraLoRALoRAMoEA3h⁡\(A,\(Bi\)i,Wg\)=\(\(rα​Bi\)i,\(A,…,A\),Wg\)h\(A,\(B\_\{i\}\)\_\{i\},W\_\{g\}\)=\\big\(\(\\tfrac\{r\}\{\\alpha\}B\_\{i\}\)\_\{i\},\\ \(A,\\dots,A\),\\ W\_\{g\}\\big\)HyperDreamBoothLoRAA3h⁡\(Bt​r,At​r\)=\(Po​u​t​Bt​r,At​r​Pi​n\)h\(B\_\{tr\},A\_\{tr\}\)=\\big\(P\_\{out\}B\_\{tr\},\\ A\_\{tr\}P\_\{in\}\\big\)HyperDreamBoothNOLAA3h⁡\(Bt​r,At​r\)=\(α=vec​At​r,β=vec​Bt​r\),NOLA bases​\{pk​ej⊤\},\{ej​ql⊤\}h\(B\_\{tr\},A\_\{tr\}\)=\\big\(\\alpha=\\mathrm\{vec\}\\,A\_\{tr\},\\ \\beta=\\mathrm\{vec\}\\,B\_\{tr\}\\big\),\\ \\text\{NOLA bases \}\\\{p\_\{k\}e\_\{j\}^\{\\top\}\\\},\\\{e\_\{j\}q\_\{l\}^\{\\top\}\\\}IC\-LoRALoRAA3h=idh=\\mathrm\{id\}ICEditLoRAA3hx\(B,A,g\)=\(αr\[G1\(x\)B1⋯G4\(x\)B4\],\[A1;…;A4\]\)h\_\{x\}\(B,A,g\)=\\big\(\\tfrac\{\\alpha\}\{r\}\[G\_\{1\}\(x\)B\_\{1\}\\ \\cdots\\ G\_\{4\}\(x\)B\_\{4\}\],\\ \[A\_\{1\};\\dots;A\_\{4\}\]\\big\)In\-Place TTTLoRAA3hc,i\(Wt​a​r​g​e​t,k\)=\(η\[V^\[1\]⊤⋯V^\[i−1\]⊤\],\[Z\[1\];…;Z\[i−1\]\]\)h\_\{c,i\}\(W\_\{target\},k\)=\\big\(\\eta\\,\[\\hat\{V\}\_\{\[1\]\}^\{\\top\}\\ \\cdots\\ \\hat\{V\}\_\{\[i\-1\]\}^\{\\top\}\],\\ \[Z\_\{\[1\]\};\\dots;Z\_\{\[i\-1\]\}\]\\big\)IncreLoRAAdaLoRAA3h=id:\(B,Λ,A\)↦\(P,Λ,Q\)h=\\mathrm\{id\}:\(B,\\Lambda,A\)\\mapsto\(P,\\Lambda,Q\)IncreLoRAIncreLoRAA5h⁡\(q\)=\(q,\(bnew,0,anew\)\)h\(q\)=\\big\(q,\\ \(b\_\{\\rm new\},0,a\_\{\\rm new\}\)\\big\)IncreLoRALoRAA3h⁡\(B,Λ,A\)=\(B​Λ,A\)h\(B,\\Lambda,A\)=\(B\\Lambda,\\ A\)InfLoRALoRAA2h⁡\(B\)=\(B,At\)h\(B\)=\(B,\\ A\_\{t\}\)InfLoRALoRA\-FAA3h=idh=\\mathrm\{id\}Initial State TuningPrefix\-TuningA3h⁡\(h′\)=\(𝒞ℓ,l\+​hl′\)l,𝒞ℓ,l=\[B¯l,A¯l​B¯l,…,A¯lℓ−1​B¯l\]h\(h^\{\\prime\}\)=\\big\(\\mathcal\{C\}\_\{\\ell,l\}^\{\+\}\\,h^\{\\prime\}\_\{l\}\\big\)\_\{l\},\\hskip 8\.19447pt\\mathcal\{C\}\_\{\\ell,l\}=\[\\bar\{B\}\_\{l\},\\bar\{A\}\_\{l\}\\bar\{B\}\_\{l\},\\dots,\\bar\{A\}\_\{l\}^\{\\ell\-1\}\\bar\{B\}\_\{l\}\]Intruder\-dimension reductionSVDiffA3h⁡\(λ\)=δ,δj=\(λ−1\)​σj​\(j∈ℐ\),δj=0​otherwiseh\(\\lambda\)=\\delta,\\hskip 8\.19447pt\\delta\_\{j\}=\(\\lambda\-1\)\\sigma\_\{j\}\\ \(j\\in\\mathcal\{I\}\),\\ \\ \\delta\_\{j\}=0\\ \\text\{otherwise\}IR\-QLoRAQLoRAA3\(B,A,β1,β2\)↦\(αs​\(B\+β2​E\),A\+β1​P\)\(B,A,\\beta\_\{1\},\\beta\_\{2\}\)\\mapsto\(\\tfrac\{\\alpha\}\{s\}\(B\+\\beta\_\{2\}E\),\\ A\+\\beta\_\{1\}P\)K\-LoRALoRA Soups \(CAT\)A3ht​\(∗\)=\(αcl,αsl\)=\{\(1,0\)layer​l​selects content at​t\(0,1\)otherwiseh\_\{t\}\(\\ast\)=\\big\(\\alpha\_\{c\}^\{l\},\\alpha\_\{s\}^\{l\}\\big\)=\\begin\{cases\}\(1,0\)&\\text\{layer \}l\\text\{ selects content at \}t\\\\ \(0,1\)&\\text\{otherwise\}\\end\{cases\}K\-LoRAZipLoRAA3ht​\(∗\)=\(mcl,msl\)∈\{\(𝟏,0\),\(0,𝟏\)\}h\_\{t\}\(\\ast\)=\\big\(m\_\{c\}^\{l\},m\_\{s\}^\{l\}\\big\)\\in\\\{\(\\mathbf\{1\},0\),\(0,\\mathbf\{1\}\)\\\}KAdaptationLoRAA3h=\(\[A1⊗u1,…,An⊗un\],\[In⊗v1⊤;…;In⊗vn⊤\]\)h=\\big\(\[A\_\{1\}\\otimes u\_\{1\},\\dots,A\_\{n\}\\otimes u\_\{n\}\],\\;\[I\_\{n\}\\otimes v\_\{1\}^\{\\top\};\\dots;I\_\{n\}\\otimes v\_\{n\}^\{\\top\}\]\\big\)KaSALoRAA3h⁡\(B,Δ​σ,A\)=\(αr​B​diag​\(Δ​σ\),A\)h\(B,\\Delta\\sigma,A\)=\\big\(\\tfrac\{\\alpha\}\{r\}B\\,\\mathrm\{diag\}\(\\Delta\\sigma\),\\ A\\big\)KeepLoRALoRAA3B↦\(B−Bt\(0\),At\)B\\mapsto\\big\(B\-B\_\{t\}^\{\(0\)\},\\ A\_\{t\}\\big\)KeepLoRALoRA\-FAA3B↦B−Bt\(0\)B\\mapsto B\-B\_\{t\}^\{\(0\)\}KnOTSTask ArithmeticA3h=idλh=\\mathrm\{id\}\_\{\\lambda\}KRAdapterLoHaA2h⁡\(U,V\)=\(B1=S1,A1=U,B2=S2,A2=V\),S1,S2​the​\{0,1\}​row selectors of the Kronecker indexh\(U,V\)=\(B\_\{1\}=S\_\{1\},\\;A\_\{1\}=U,\\;B\_\{2\}=S\_\{2\},\\;A\_\{2\}=V\),\\ \\ S\_\{1\},S\_\{2\}\\ \\text\{the \}\\\{0,1\\\}\\text\{ row selectors of the Kronecker index\}KronALoKrA3h⁡\(Ak,Bk\)=\(C=Ak,right factor=Bk\)​\(LoKr with neither factor decomposed\)h\(A\_\{k\},B\_\{k\}\)=\\big\(C=A\_\{k\},\\;\\text\{right factor\}=B\_\{k\}\\big\)\\ \\ \\text\{\(LoKr with neither factor decomposed\)\}LaMDALoRAA3h⁡\(U,S\)=\(α​U​S−Ur,Σr​Vr⊤\)h\(U,S\)=\\big\(\\alpha US\-U\_\{r\},\\;\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\\big\)LaMDALoRA\-FAA3h⁡\(U,S\)=α​U​S−Ur​into​FA​\(Σr​Vr⊤\)h\(U,S\)=\\alpha US\-U\_\{r\}\\ \\text\{ into \}\\ \\mathrm\{FA\}\(\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\)LaMDAPiSSAA3h⁡\(U,S\)=\(α​U​S​Σr1/2,Σr1/2​Vr⊤\)h\(U,S\)=\\big\(\\alpha US\\,\\Sigma\_\{r\}^\{1/2\},\\;\\Sigma\_\{r\}^\{1/2\}V\_\{r\}^\{\\top\}\\big\)Laplace\-LoRALoRAA3h=id​\(the MAP object\)h=\\mathrm\{id\}\\ \\ \\text\{\(the MAP object\)\}LCM\-LoRALoRAA3h⁡\(λ1,λ2\)=\(\[λ1​B′​λ2​BLCM\],\[A′;ALCM\]\)h\(\\lambda\_\{1\},\\lambda\_\{2\}\)=\\big\(\[\\lambda\_\{1\}B^\{\\prime\}\\ \\ \\lambda\_\{2\}B\_\{\\rm LCM\}\],\\ \[A^\{\\prime\};A\_\{\\rm LCM\}\]\\big\)LCM\-LoRATask ArithmeticA3h⁡\(λ1,λ2\)=\(λ1,λ2\),τ1=B′​A′,τ2=BLCM​ALCMh\(\\lambda\_\{1\},\\lambda\_\{2\}\)=\(\\lambda\_\{1\},\\lambda\_\{2\}\),\\hskip 8\.19447pt\\tau\_\{1\}=B^\{\\prime\}A^\{\\prime\},\\ \\tau\_\{2\}=B\_\{\\rm LCM\}A\_\{\\rm LCM\}LD\-MoLELoRAA3hx\(\(Bi,Ai\)i,Wg​a​t​e,f\)=\(αr​s\[px,1B1⋯px,EBE\],\[A1;…;AE\]\)h\_\{x\}\\big\(\(B\_\{i\},A\_\{i\}\)\_\{i\},W\_\{gate\},f\\big\)=\\big\(\\tfrac\{\\alpha\}\{rs\}\\,\[\\,p\_\{x,1\}B\_\{1\}\\ \\cdots\\ p\_\{x,E\}B\_\{E\}\\,\],\\ \[A\_\{1\};\\dots;A\_\{E\}\]\\big\)LieRAFLoRAA3h⁡\(B,A\)=\(\[𝟏m​B\],\[𝟏n⊤;A\]\)h\(B,A\)=\\big\(\[\\mathbf\{1\}\_\{m\}\\;B\],\\,\[\\mathbf\{1\}\_\{n\}^\{\\top\};A\]\\big\)LieRAHiRAA3h⁡\(B,A\)=\(B,A\)h\(B,A\)=\(B,A\)LieRALoHaA3h⁡\(B,A\)=\(B,A,U0,V0⊤\),U0​V0⊤=W0h\(B,A\)=\(B,A,U\_\{0\},V\_\{0\}^\{\\top\}\),\\hskip 8\.19447ptU\_\{0\}V\_\{0\}^\{\\top\}=W\_\{0\}LIFTDiff PruningA2x↦\(α0,ιM0​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\_\{0\}\}x\)LiMELoRAA3hx​\(B,A,\(pi\)i,ps,γ\)=\(diag⁡\(𝒫⁡\(x\)\+γ​ps\)​B,A\)h\_\{x\}\(B,A,\(p\_\{i\}\)\_\{i\},p\_\{s\},\\gamma\)=\\big\(\\mathrm\{diag\}\(\\mathcal\{P\}\(x\)\+\\gamma p\_\{s\}\)\\,B,\\ A\\big\)Linear probingBitFitA2h⁡\(V\)=\(Δ​b=0,V\)h\(V\)=\(\\Delta b=0,\\ V\)Linear probingLP\-FTA5V↦\(V,q0\(2\)\)V\\mapsto\(V,\\ q\_\{0\}^\{\(2\)\}\)Linear probingSurgical FTA2h⁡\(V\)=V​\(block = last layer\)h\(V\)=V\\ \\ \\text\{\(block = last layer\)\}LLaMA\-AdapterLLaMA\-Adapter V2A2h⁡\(P,g\)=\(P,g,s=𝟏,b=0,γ=γ0,visual branch at its zero gate\)h\(P,g\)=\\big\(P,\\,g,\\ s=\\mathbf\{1\},\\ b=0,\\ \\gamma=\\gamma\_\{0\},\\ \\text\{visual branch at its zero gate\}\\big\)LN TuningSSFA3\(γ,β\)↦\(γ′=γ⊘γ0,β′=β−\(γ⊘γ0\)⊙β0\)​on SSF’s post\-norm wrappers; other wrappers at​\(𝟏,0\)\(\\gamma,\\beta\)\\mapsto\\big\(\\gamma^\{\\prime\}=\\gamma\\oslash\\gamma\_\{0\},\\ \\beta^\{\\prime\}=\\beta\-\(\\gamma\\oslash\\gamma\_\{0\}\)\\odot\\beta\_\{0\}\\big\)\\ \\text\{on SSF's post\-norm wrappers; other wrappers at \}\(\\mathbf\{1\},0\)LoCALoRA\-FAA3h\(a\)=\[a1Cl11,:⊤⋯aℬCl1ℬ,:⊤\]intoFA\(A0\),A0=\[Dl21,:;…;Dl2ℬ,:\]h\(a\)=\\big\[a\_\{1\}C\_\{l^\{1\}\_\{1\},:\}^\{\\top\}\\ \\cdots\\ a\_\{\\mathcal\{B\}\}C\_\{l^\{1\}\_\{\\mathcal\{B\}\},:\}^\{\\top\}\\big\]\\ \\text\{ into \}\\ \\mathrm\{FA\}\(A\_\{0\}\),\\ A\_\{0\}=\\big\[D\_\{l^\{2\}\_\{1\},:\};\\dots;D\_\{l^\{2\}\_\{\\mathcal\{B\}\},:\}\\big\]LoCOLoRAA3\(U,V\)↦\(2​W0​X​\(I2​r−Y⊤​X\)−1,Y⊤\),X=\[U\|−V\],Y=\[V\|U\]\(U,V\)\\mapsto\\big\(2W\_\{0\}X\(I\_\{2r\}\-Y^\{\\top\}X\)^\{\-1\},\\ Y^\{\\top\}\\big\),\\ \\ X=\[U\\,\|\\,\-V\],\\ Y=\[V\\,\|\\,U\]LoCOOFTA3\(U,V\)↦Q=−\(U​V⊤−V​U⊤\)\(U,V\)\\mapsto Q=\-\(UV^\{\\top\}\-VU^\{\\top\}\)LoDALoRAA3\(BG,BI\)↦\(\[wG​BG,BI\],\[AG;AI\]\)\(B\_\{G\},B\_\{I\}\)\\mapsto\\big\(\[\\,w\_\{G\}B\_\{G\},\\ B\_\{I\}\\,\],\\ \[A\_\{G\};A\_\{I\}\]\\big\)LoDALoRA\-FAA3\(BG,BI\)↦\[wG​BG,BI\]\(B\_\{G\},B\_\{I\}\)\\mapsto\[\\,w\_\{G\}B\_\{G\},\\ B\_\{I\}\\,\]LoFiTBitFitA3h⁡\(v\)=\(Δ​bO\(l\)=∑i∈TlW\(i\)O​vli,other biases​0\)h\(v\)=\\big\(\\Delta b\_\{O\}^\{\(l\)\}=\\textstyle\\sum\_\{i\\in T\_\{l\}\}W^\{O\}\_\{\(i\)\}v\_\{l\}^\{i\},\\ \\text\{other biases \}0\\big\)LoFTLoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LOFTLoRAA3E↦\(B=W0​Pr⊤​\(C⁡\(E\)−Ir\),A=Pr\)E\\mapsto\\big\(B=W\_\{0\}P\_\{r\}^\{\\top\}\(C\(E\)\-I\_\{r\}\),\\ A=P\_\{r\}\\big\)LOFTLoRA\-XSA3E↦R=Σr​\(C⁡\(E\)−I\)​Σr−1E\\mapsto R=\\Sigma\_\{r\}\\big\(C\(E\)\-I\\big\)\\Sigma\_\{r\}^\{\-1\}LOFTOFTA3E↦Q=−Pr⊤​E​Pr∈𝔰​𝔬​\(n\)E\\mapsto Q=\-P\_\{r\}^\{\\top\}EP\_\{r\}\\in\\mathfrak\{so\}\(n\)LOFTPSOFTA3E↦\(R=C\(E\)⊤,α=β=𝟏\)E\\mapsto\\big\(R=C\(E\)^\{\\top\},\\ \\alpha=\\beta=\\mathbf\{1\}\\big\)LoftQLoRAA3h⁡\(B,A\)=\(\[B,−BT\],\[A;AT\]\)h\(B,A\)=\(\[B,\\,\-B\_\{T\}\],\\,\[A;\\,A\_\{T\}\]\)LoHaLoRAA3h⁡\(B1,A1,B2,A2\)=\(γ​B1∙B2,\(A1⊤∙A2⊤\)⊤\)h\(B\_\{1\},A\_\{1\},B\_\{2\},A\_\{2\}\)=\\big\(\\gamma\\,B\_\{1\}\\bullet B\_\{2\},\\;\(A\_\{1\}^\{\\top\}\\bullet A\_\{2\}^\{\\top\}\)^\{\\top\}\\big\)LoKrKronAA3h⁡\(C,B,A\)=\(C,B​A\)h\(C,B,A\)=\(C,\\;BA\)LoKrLoRAA3h⁡\(C,B,A\)=\(C⊗B,Iuq⊗A\)h\(C,B,A\)=\\big\(C\\otimes B,\\;I\_\{u\_\{q\}\}\\otimes A\\big\)LoQTGaLoreA3B↦s​BB\\mapsto sBLoQTLoRAA2B↦\(Pk,ss′​B\)B\\mapsto\(P\_\{k\},\\ \\tfrac\{s\}\{s^\{\\prime\}\}B\)LoRADoRAA3h\(B,A\)=\(\(∥\(W0\+sBA\)i,:∥\)i,B,A\)h\(B,A\)=\\big\(\(\\lVert\(W\_\{0\}\+sBA\)\_\{i,:\}\\rVert\)\_\{i\},\\ B,\\ A\\big\)LoRALoRAA3h⁡\(B,A\)=\(\[B​0\],\[A;a0\]\)h\(B,A\)=\(\[B\\ 0\],\\,\[A;\\,a\_\{0\}\]\)LoRAPEANuTA3h⁡\(B,A\)=\(\[M,−M\],\[B⊤;−B⊤\]\),M=\(W0\+\)⊤​A⊤h\(B,A\)=\\big\(\[M,\\,\-M\],\\ \[B^\{\\top\};\\,\-B^\{\\top\}\]\\big\),\\hskip 8\.19447ptM=\(W\_\{0\}^\{\+\}\)^\{\\top\}A^\{\\top\}LoRAReLoRAA5q↦\(q,q0\(2\),…,q0\(K\)\)q\\mapsto\(q,\\,q\_\{0\}^\{\(2\)\},\\dots,q\_\{0\}^\{\(K\)\}\)LoRA DropoutLoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LoRA Soups \(CAT\)AdaMergingA3h=id:αil↦λilh=\\mathrm\{id\}:\\ \\alpha\_\{i\}^\{l\}\\mapsto\\lambda\_\{i\}^\{l\}LoRA Soups \(CAT\)LoRAA3h\(α\)=\(\[α1B1⋯αNBN\],\[A1;…;AN\]\)h\(\\alpha\)=\\big\(\[\\alpha\_\{1\}B\_\{1\}\\ \\cdots\\ \\alpha\_\{N\}B\_\{N\}\],\\ \[A\_\{1\};\\dots;A\_\{N\}\]\\big\)LoRA Switch / LoRA CompositeLoRA Soups \(CAT\)A3ht​\(∗\)=wi⁡\(t\)​ei⁡\(t\)h\_\{t\}\(\\ast\)=w\_\{i\(t\)\}\\,e\_\{i\(t\)\}LoRA Without RegretLoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LoRA\+LoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LoRA\-DashLoRAA3h⁡\(B,A,Δ​σ\)=\(\[B​U¯​diag​\(Δ​σ\)\],\[A;V¯⊤\]\)h\(B,A,\\Delta\\sigma\)=\\big\(\[B\\ \\ \\bar\{U\}\\mathrm\{diag\}\(\\Delta\\sigma\)\],\\ \[A;\\ \\bar\{V\}^\{\\top\}\]\\big\)LoRA\-dropLoRAA3h⁡\(\(Bi,Ai\)keep,\(Bs,As\)\)=\(\(Bi,Ai\)keep,\(Bs,As\)j∈share\)h\\big\(\(B\_\{i\},A\_\{i\}\)\_\{\\rm keep\},\(B\_\{s\},A\_\{s\}\)\\big\)=\\big\(\(B\_\{i\},A\_\{i\}\)\_\{\\rm keep\},\(B\_\{s\},A\_\{s\}\)\_\{j\\in\\rm share\}\\big\)LoRA\-FAAsyLoRAA3h⁡\(B\)=B​A0​Q⊤h\(B\)=B\\,A\_\{0\}Q^\{\\top\}LoRA\-FAFloraA5B↦\(B,0,…,0\)B\\mapsto\(B,\\,0,\\dots,0\)LoRA\-FALoRAA2h⁡\(B\)=\(B,A0\)h\(B\)=\(B,\\ A\_\{0\}\)LoRA\-FlowLoRA Soups \(CAT\)A3hx​\(Wgate,b\)=\(αil=wil​\(x\)\)i,lh\_\{x\}\(W\_\{\\rm gate\},b\)=\\big\(\\alpha\_\{i\}^\{l\}=w\_\{i\}^\{l\}\(x\)\\big\)\_\{i,l\}LoRA\-GALoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)LoRA\-LEGOLoRA Soups \(CAT\)A3h\(∗\)=N−1/2𝟏h\(\\ast\)=N^\{\-1/2\}\\mathbf\{1\}LoRA\-MiniLoRAA3h⁡\(C,D\)=\(P​C,D​Q\)h\(C,D\)=\(PC,\\;DQ\)LoRA\-MixerLoRAA3hx​\(\(Be,Ae\)e,G\)=\(\[1e∈TopK⁡\(x\)​pe​\(x\)​Be\]e=1E,\[A1;…;AE\]\)h\_\{x\}\\big\(\(B\_\{e\},A\_\{e\}\)\_\{e\},G\\big\)=\\big\(\\,\[\\,\\mathbb\{1\}\_\{e\\in\\mathrm\{TopK\}\(x\)\}\\,p\_\{e\}\(x\)\\,B\_\{e\}\\,\]\_\{e=1\}^\{E\},\\ \[A\_\{1\};\\dots;A\_\{E\}\]\\,\\big\)LoRA\-NullLoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)LoRA\-PreAdam\-miniA3h=idΘh=\\mathrm\{id\}\_\{\\Theta\}LoRA\-ProLoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LoRA\-RITELoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LoRA\-RLPO / LoRA\-RLMOLoRAA3h=id:\(B,A\)↦\(B,A\)h=\\mathrm\{id\}:\(B,A\)\\mapsto\(B,A\)LoRA\-S \(AdamS, LRACS\)LoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}LoRA\-SBLoRA\-FAA4h⁡\(R\)=s​Ur​Rh\(R\)=sU\_\{r\}RLoRA\-XLoRAA3h⁡\(Δ​Σ\)=\(Ur,Δ​Σ​Vr⊤\)h\(\\Delta\\Sigma\)=\(U\_\{r\},\\ \\Delta\\Sigma V\_\{r\}^\{\\top\}\)LoRA\-XLoRA\-XSA3h⁡\(Δ​Σ\)=Δ​Σ​Σr−1h\(\\Delta\\Sigma\)=\\Delta\\Sigma\\,\\Sigma\_\{r\}^\{\-1\}LoRA\-XSVDiffA2h⁡\(δ​σ\)=\(δ​σ1,…,δ​σr,0,…,0\)h\(\\delta\\sigma\)=\(\\delta\\sigma\_\{1\},\\dots,\\delta\\sigma\_\{r\},0,\\dots,0\)LoRA\-XSVFTA2h\(δσ\)=M,Mi​i=δσi\(i≤r\),0otherwiseh\(\\delta\\sigma\)=M,\\ \\ M\_\{ii\}=\\delta\\sigma\_\{i\}\\ \(i\\leq r\),\\ \\ 0\\ \\text\{otherwise\}LoRA\-XSLoRAA3h⁡\(R\)=\(Ur​R,Σr​Vr⊤\)h\(R\)=\\big\(U\_\{r\}R,\\;\\Sigma\_\{r\}V\_\{r\}^\{\\top\}\\big\)LoRA\-XSLoRA\-FAA4h⁡\(R\)=Ur​R​into​FA​\(A0\),A0=Σr​Vr⊤h\(R\)=U\_\{r\}R\\ \\text\{ into \}\\ \\mathrm\{FA\}\(A\_\{0\}\),\\ A\_\{0\}=\\Sigma\_\{r\}V\_\{r\}^\{\\top\}LoRA\-XSPiSSAA3h⁡\(R\)=\(Ur​Σr1/2\+Ur​R​Σr1/2,Σr1/2​Vr⊤\)h\(R\)=\\big\(U\_\{r\}\\Sigma\_\{r\}^\{1/2\}\+U\_\{r\}R\\,\\Sigma\_\{r\}^\{1/2\},\\;\\Sigma\_\{r\}^\{1/2\}V\_\{r\}^\{\\top\}\\big\)LoRA\-XSSVFTA3h⁡\(R\)=\[R​Σr000\]⁡\(Ω=top\-left​r×r​block\)h\(R\)=\\begin\{bmatrix\}R\\Sigma\_\{r\}&0\\\\ 0&0\\end\{bmatrix\}\\ \\ \(\\Omega=\\text\{top\-left \}r\\times r\\text\{ block\}\)LoraHubLoRAA3h⁡\(w\)=\(∑iwi​Bi,∑iwi​Ai\)h\(w\)=\\big\(\\textstyle\\sum\_\{i\}w\_\{i\}B\_\{i\},\\ \\sum\_\{i\}w\_\{i\}A\_\{i\}\\big\)LoRAMoELoRAA3hx\(B,A,Wg\)=\(αr\[ω1\(x\)B1⋯ωN\(x\)BN\],\[A1;…;AN\]\)h\_\{x\}\(B,A,W\_\{g\}\)=\\big\(\\tfrac\{\\alpha\}\{r\}\[\\omega\_\{1\}\(x\)B\_\{1\}\\ \\cdots\\ \\omega\_\{N\}\(x\)B\_\{N\}\],\\ \[A\_\{1\};\\dots;A\_\{N\}\]\\big\)LoRAMoEMoV / MoLoRAA3h⁡\(Bi,Ai,Wg\)=\(αr​Bi,Ai,Wg⊤\)h\(B\_\{i\},A\_\{i\},W\_\{g\}\)=\(\\tfrac\{\\alpha\}\{r\}B\_\{i\},\\,A\_\{i\},\\,W\_\{g\}^\{\\top\}\)LoRAPruneDoRAA3h⁡\(B,A,m\)=\(μ=m⊙∥W0\+s​B​A∥row,B,A\)h\(B,A,m\)=\\big\(\\mu=m\\odot\\lVert W\_\{0\}\+sBA\\rVert\_\{\\rm row\},\\ B,\\ A\\big\)LoraRetrieverLoRA Soups \(CAT\)A3hx\(∗\)=\(αϕ=𝟏\[ϕ∈Φx\]/k\)ϕh\_\{x\}\(\\ast\)=\\big\(\\alpha\_\{\\phi\}=\\mathbf\{1\}\[\\phi\\in\\Phi\_\{x\}\]/k\\big\)\_\{\\phi\}LoraRetrieverLoraHubA3hx​\(∗\)=w=𝟏Φx/kh\_\{x\}\(\\ast\)=w=\\mathbf\{1\}\_\{\\Phi\_\{x\}\}/kLoReFTLoRAA3h⁡\(R,W,b\)=\(R⊤,\[W−R∣b\]\)h\(R,W,b\)=\\big\(R^\{\\top\},\\ \[W\-R\\mid b\]\\big\)LoRETTALoRAA3h⁡\(\{Gu​pi\},\{Gd​o​w​ni\}\)=\(TT⁡\(\{Gu​pi\}\),TT⁡\(\{Gd​o​w​ni\}\)\)h\(\\\{G^\{i\}\_\{up\}\\\},\\\{G^\{i\}\_\{down\}\\\}\)=\\big\(\\mathrm\{TT\}\(\\\{G^\{i\}\_\{up\}\\\}\),\\;\\mathrm\{TT\}\(\\\{G^\{i\}\_\{down\}\\\}\)\\big\)LoRILoRAA2h⁡\(X\)=\(ιM​X,A0\)h\(X\)=\(\\iota\_\{M\}X,\\ A\_\{0\}\)LoRILoRA\-FAA2h⁡\(X\)=ιM​Xh\(X\)=\\iota\_\{M\}XLoRTAFacTA3h=\(U=ℬ,V=A,Σ\(l,i\)=Diag\(CL\[l,:\]⊙CM\[i,:\]\)\)h=\\big\(U=\\mathcal\{B\},\\;V=A,\\;\\Sigma\_\{\(l,i\)\}=\\mathrm\{Diag\}\(C\_\{L\}\[l,:\]\\odot C\_\{M\}\[i,:\]\)\\big\)LoRTALoRAA3h=\(ℬDiag\(CL\[l,:\]⊙CM\[i,:\]\),A⊤\)\(l,i\),ℬ=\[BDiag\(CH\[1,:\]\);…;BDiag\(CH\[H,:\]\)\]h=\\big\(\\mathcal\{B\}\\,\\mathrm\{Diag\}\(C\_\{L\}\[l,:\]\\odot C\_\{M\}\[i,:\]\),\\;A^\{\\top\}\\big\)\_\{\(l,i\)\},\\ \\ \\mathcal\{B\}=\\big\[B\\,\\mathrm\{Diag\}\(C\_\{H\}\[1,:\]\);\\dots;B\\,\\mathrm\{Diag\}\(C\_\{H\}\[H,:\]\)\\big\]LoRTALoTRA3h=\(ALoTR=A,BLoTR=ℬ,G\(l,i\)⊤=Diag\(CL\[l,:\]⊙CM\[i,:\]\)\)h=\\big\(A\_\{\\rm LoTR\}=A,\\;B\_\{\\rm LoTR\}=\\mathcal\{B\},\\;G\_\{\(l,i\)\}^\{\\top\}=\\mathrm\{Diag\}\(C\_\{L\}\[l,:\]\\odot C\_\{M\}\[i,:\]\)\\big\)LoRTAMetaTTA3h=\(G1=A,G2\[l\]=Diag\(CL\[l,:\]\),G3\[i\]=Diag\(CM\[i,:\]\),GH\[k\]=Diag\(CH\[k,:\]\),G4=B⊤\)h=\\big\(G\_\{1\}=A,\\;G\_\{2\}\[l\]=\\mathrm\{Diag\}\(C\_\{L\}\[l,:\]\),\\;G\_\{3\}\[i\]=\\mathrm\{Diag\}\(C\_\{M\}\[i,:\]\),\\;G\_\{H\}\[k\]=\\mathrm\{Diag\}\(C\_\{H\}\[k,:\]\),\\;G\_\{4\}=B^\{\\top\}\\big\)LoTRFacTA3h⁡\(A,B,\{Gs\}\)=\(V=A,U=B,Σs=Gs⊤\)h\(A,B,\\\{G\_\{s\}\\\}\)=\\big\(V=A,\\;U=B,\\;\\Sigma\_\{s\}=G\_\{s\}^\{\\top\}\\big\)LoTRFLoRA \(Tucker core\)A3h⁡\(A,B,\{Gs\}\)=\(AF=B,GF=Gs⊤,BF=A\)sh\(A,B,\\\{G\_\{s\}\\\}\)=\\big\(A\_\{F\}=B,\\;G\_\{F\}=G\_\{s\}^\{\\top\},\\;B\_\{F\}=A\\big\)\_\{s\}LoTRLoRAA3h=\(B​Gs⊤,A⊤\)sh=\\big\(BG\_\{s\}^\{\\top\},\\;A^\{\\top\}\\big\)\_\{s\}LoTRShareLoRAA3h=\(\{Bs=BGs⊤\}s,A⊤\)h=\\big\(\\\{B\_\{s\}=BG\_\{s\}^\{\\top\}\\\}\_\{s\},\\;A^\{\\top\}\\big\)LRAgentLoRAA3hi​\(A,B1,…,BN\)=\(Bi,A\)h\_\{i\}\(A,B\_\{1\},\\dots,B\_\{N\}\)=\(B\_\{i\},\\ A\)LRAgentS\-LoRAA3\(A,\(Bi\)i\)↦\(\(Bi,A\)\)i\(A,\(B\_\{i\}\)\_\{i\}\)\\mapsto\\big\(\(B\_\{i\},A\)\\big\)\_\{i\}LT\-SFTDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)MaLoRA / MaRALoRAA3hx1:t\(A,B,P,Mamba,w,b\)=\(αr​sλ\(x1:t\)B,A\)h\_\{x\_\{1:t\}\}\(A,B,P,\\mathrm\{Mamba\},w,b\)=\\big\(\\tfrac\{\\alpha\}\{rs\}\\,\\lambda\(x\_\{1:t\}\)\\,B,\\ A\\big\)MatryoshkaLoRADyLoRAA3h=idh=\\mathrm\{id\}MatryoshkaLoRALoRAA3\(B,A\)↦\(skB:,1:k,A1:k,:\)\(B,A\)\\mapsto\\big\(s\_\{k\}B\_\{:,1:k\},\\ A\_\{1:k,:\}\\big\)MatryoshkaLoRASoRAA2\(B,A\)↦\(B,p,A\)\(B,A\)\\mapsto\(B,\\ p,\\ A\)MDPDLN TuningA3h=idh=\\mathrm\{id\}MELoRAGraLoRAA2\(Bi,Ai\)i↦\(Bi​i=Bi,Ai​i=Ai⊤,\(Bi​j,Ai​j\)=\(0,Ai​j,0\)​\(i≠j\)\)\(B\_\{i\},A\_\{i\}\)\_\{i\}\\mapsto\\big\(B\_\{ii\}=B\_\{i\},\\ A\_\{ii\}=A\_\{i\}^\{\\top\};\\ \(B\_\{ij\},A\_\{ij\}\)=\(0,A\_\{ij,0\}\)\\ \(i\\neq j\)\\big\)MELoRALoRAA2\(Bi,Ai\)i↦\(blockdiag⁡\(B1,…,Bn\),blockdiag⁡\(A1,…,An\)\)\(B\_\{i\},A\_\{i\}\)\_\{i\}\\mapsto\\big\(\\mathrm\{blockdiag\}\(B\_\{1\},\\dots,B\_\{n\}\),\\ \\mathrm\{blockdiag\}\(A\_\{1\},\\dots,A\_\{n\}\)\\big\)MetaTTLoRAA3h=\(G4⊤​G3​\[m\]⊤​G2​\[l\]⊤,G1⊤\)\(l,m\)h=\\big\(G\_\{4\}^\{\\top\}G\_\{3\}\[m\]^\{\\top\}G\_\{2\}\[l\]^\{\\top\},\\;G\_\{1\}^\{\\top\}\\big\)\_\{\(l,m\)\}MetaTTLoTRA3h⁡\(G1,\{G2​\[l\]\},\{G3​\[m\]\},G4\)=\(A=G1,B=G4⊤,G\(l,m\)=G2​\[l\]​G3​\[m\]\)​per type​mh\(G\_\{1\},\\\{G\_\{2\}\[l\]\\\},\\\{G\_\{3\}\[m\]\\\},G\_\{4\}\)=\\big\(A=G\_\{1\},\\;B=G\_\{4\}^\{\\top\},\\;G\_\{\(l,m\)\}=G\_\{2\}\[l\]G\_\{3\}\[m\]\\big\)\\ \\text\{per type \}mMHRLoRAA3hτ​\(A,B,Z\)=\(Bτ,Aτ\)h\_\{\\tau\}\(A,B,Z\)=\(B^\{\\tau\},\\,A^\{\\tau\}\)MiCALoRAA2h⁡\(A\)=\(U−r,αr​A\)h\(A\)=\\big\(U\_\{\-r\},\\ \\tfrac\{\\alpha\}\{r\}A\\big\)MiCAMiLoRAA3h\(A\)=\(UmΣm1/2,Σm1/2Vm⊤\+αrΣm−1/2A\)h\(A\)=\\big\(U\_\{m\}\\Sigma\_\{m\}^\{1/2\},\\ \\Sigma\_\{m\}^\{1/2\}V\_\{m\}^\{\\top\}\+\\tfrac\{\\alpha\}\{r\}\\Sigma\_\{m\}^\{\-1/2\}A\\big\)MiLoRALoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)MiSS \(formerly Bone\)LoRAA2h⁡\(D\)=\(D,1din/r⊤⊗Ir\)h\(D\)=\(D,\\ \\mathbf\{1\}^\{\\top\}\_\{d\_\{\\rm in\}/r\}\\otimes I\_\{r\}\)MiSS \(formerly Bone\)LoRA\-FAA3h⁡\(D\)=D,A0=𝟏din/r⊤⊗Irh\(D\)=D,\\ \\ A\_\{0\}=\\mathbf\{1\}^\{\\top\}\_\{d\_\{\\rm in\}/r\}\\otimes I\_\{r\}Mix\-of\-ShowRegMeanA3h⁡\(∗\)=\(α=1\)h\(\\ast\)=\(\\alpha=1\)Model SoupsTask ArithmeticA3h⁡\(∗\)=𝟏S/\|S\|h\(\\ast\)=\\mathbf\{1\}\_\{S\}/\|S\|ModuLoRAQLoRAA3h=idh=\\mathrm\{id\}MOELoRALoRAA3hτ\(B,A,e,WT\)=\(\[ωτ​1B1⋯ωτ​NBN\],\[A1;…;AN\]\)h\_\{\\tau\}\(B,A,e,W\_\{T\}\)=\\big\(\[\\omega\_\{\\tau 1\}B\_\{1\}\\ \\cdots\\ \\omega\_\{\\tau N\}B\_\{N\}\],\\ \[A\_\{1\};\\dots;A\_\{N\}\]\\big\)MonteCLoRALoRAA3h⁡\(B,μ,ξ\)=\(B,μ\)h\(B,\\mu,\\xi\)=\(B,\\ \\mu\)MoRAKronAA2M↦\(Ak,Bk\)=\(Id/r^,M/s\)M\\mapsto\(A\_\{k\},B\_\{k\}\)=\(I\_\{d/\\hat\{r\}\},\\ M/s\)MoReLoRAA2\(L,R\)↦\(P1​L​P2,R\)\(L,R\)\\mapsto\(P\_\{1\}LP\_\{2\},\\ R\)MoSLoRAA3h⁡\(poolB,poolA\)=\(Routec​\(poolB,Iℓb\),Router​\(poolA,Iℓa\)\)ℓh\(\\mathrm\{pool\}\_\{B\},\\mathrm\{pool\}\_\{A\}\)=\\big\(\\mathrm\{Route\}^\{c\}\(\\mathrm\{pool\}\_\{B\},I^\{b\}\_\{\\ell\}\),\\;\\mathrm\{Route\}^\{r\}\(\\mathrm\{pool\}\_\{A\},I^\{a\}\_\{\\ell\}\)\\big\)\_\{\\ell\}MoSLoRAFLoRA \(Tucker core\)A3h=idh=\\mathrm\{id\}MoSLoRALoRAA3h⁡\(B,M,A\)=\(B​M,A\)h\(B,M,A\)=\(BM,\\ A\)MoV / MoLoRA\(IA\)3A3hx​\(ℓ1,…,ℓN,Wg\)=∑isi​\(x\)​ℓih\_\{x\}\(\\ell\_\{1\},\\dots,\\ell\_\{N\},W\_\{g\}\)=\\textstyle\\sum\_\{i\}s\_\{i\}\(x\)\\ell\_\{i\}MoV / MoLoRALoRAMoEA3h⁡\(Bi,Ai,Wg\)=\(rα​Bi,Ai,Wg⊤\)​\(MoLoRA variant\)h\(B\_\{i\},A\_\{i\},W\_\{g\}\)=\(\\tfrac\{r\}\{\\alpha\}B\_\{i\},\\,A\_\{i\},\\,W\_\{g\}^\{\\top\}\)\\ \\text\{\(MoLoRA variant\)\}MPTPrompt TuningA3h⁡\(P∗,u,v\)=P∗⊙u​v⊤h\(P^\{\*\},u,v\)=P^\{\*\}\\odot uv^\{\\top\}NaRALoRAA3hλ​\(B,A,ϕ\)=\(B​Cϕ​\(λ\),A\)h\_\{\\lambda\}\(B,A,\\phi\)=\\big\(B\\,C\_\{\\phi\}\(\\lambda\),\\ A\\big\)NaRAMoSLoRAA3hλ​\(B,A,ϕ\)=\(B,Cϕ​\(λ\)/s,A\)h\_\{\\lambda\}\(B,A,\\phi\)=\\big\(B,\\ C\_\{\\phi\}\(\\lambda\)/s,\\ A\\big\)NOLALoRAA3h⁡\(α,β\)=\(∑jβj​Bj,∑iαi​Ai\)h\(\\alpha,\\beta\)=\\Big\(\\sum\_\{j\}\\beta\_\{j\}B\_\{j\},\\ \\sum\_\{i\}\\alpha\_\{i\}A\_\{i\}\\Big\)O\-LoRALoRAA5h=\(\[B1⋯BT\],\[A1;…;AT\]\)h=\\big\(\[B\_\{1\}\\ \\cdots\\ B\_\{T\}\],\\ \[A\_\{1\};\\dots;A\_\{T\}\]\\big\)O\-LoRAReLoRAA3h=id​on each stage’s​\(Bt,At\)h=\\mathrm\{id\}\\ \\text\{on each stage's \}\(B\_\{t\},A\_\{t\}\)OFTBOFTA2Q↦\(Q,s=𝟏\)​in BOFT​\(1,b\)Q\\mapsto\(Q,\\ s=\\mathbf\{1\}\)\\ \\text\{in BOFT\}\(1,b\)OFTGOFTA3t↦\(θk\(1\)=2​arctan⁡tk,θ\(ℓ\)=0​\(ℓ≥2\)\)t\\mapsto\\big\(\\theta^\{\(1\)\}\_\{k\}=2\\arctan t\_\{k\},\\ \\theta^\{\(\\ell\)\}=0\\ \(\\ell\\geq 2\)\\big\)OFTv2qGOFTA3t↦\(G~k\(1\)=\(1−tk2\)​\(1−tk2−2​tk2​tk1−tk2\),G~\(ℓ\)=I⁡\(ℓ≥2\)\)t\\mapsto\\Big\(\\tilde\{G\}^\{\(1\)\}\_\{k\}=\(1\-t\_\{k\}^\{2\}\)\\begin\{pmatrix\}1\-t\_\{k\}^\{2\}&\-2t\_\{k\}\\\\ 2t\_\{k\}&1\-t\_\{k\}^\{2\}\\end\{pmatrix\},\\ \\tilde\{G\}^\{\(\\ell\)\}=I\\ \(\\ell\\geq 2\)\\Big\)OLoRALoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)Online Subspace DescentLoRAA2h⁡\(X\)=\(Pt,X\)h\(X\)=\(P\_\{t\},\\,X\)Orthogonal AdaptationLoRAA2h⁡\(B\)=\(B,Ai\)h\(B\)=\(B,\\,A\_\{i\}\)Orthogonal AdaptationLoRA\-FAA3h=idh=\\mathrm\{id\}OrthoMergeOFTA3∗↦Qm=cN​∑iQi\\ast\\mapsto Q\_\{m\}=\\tfrac\{c\}\{N\}\\textstyle\\sum\_\{i\}Q\_\{i\}OSFMiLoRAA3\(U,Σ,V\)↦\(UΣSl​o−1/2,Sl​o1/2V⊤\)\(U,\\Sigma,V\)\\mapsto\\big\(U\\,\\Sigma\\,S\_\{lo\}^\{\-1/2\},\\ S\_\{lo\}^\{1/2\}V^\{\\top\}\\big\)P\-TuningPrompt TuningA3h⁡\(P,ψ\)=fψ​\(P\)h\(P,\\psi\)=f\_\{\\psi\}\(P\)PaFiDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)Parallel AdapterAdaptFormerA3h⁡\(Wd​o​w​n,bd​o​w​n,Wu​p,bu​p\)=\(Wd​o​w​n,bd​o​w​n,s0\.1​Wu​p,s0\.1​bu​p\)h\(W\_\{down\},b\_\{down\},W\_\{up\},b\_\{up\}\)=\\big\(W\_\{down\},b\_\{down\},\\tfrac\{s\}\{0\.1\}W\_\{up\},\\tfrac\{s\}\{0\.1\}b\_\{up\}\\big\)PEANuTLoRAA3h⁡\(Θ1,Θ2\)=\(Θ2⊤,σ⁡\(Θ1⊤​W0\)\)h\(\\Theta\_\{1\},\\Theta\_\{2\}\)=\(\\Theta\_\{2\}^\{\\top\},\\ \\sigma\(\\Theta\_\{1\}^\{\\top\}W\_\{0\}\)\)PEM ArithmeticTask ArithmeticA3h⁡\(λ\)=\(λ,1−λ\)h\(\\lambda\)=\(\\lambda,\\,1\-\\lambda\)PEQAEfficientQATA3Δ​s↦Δ​s⊗𝟏n/g⊤\\Delta s\\mapsto\\Delta s\\otimes\\mathbf\{1\}\_\{n/g\}^\{\\top\}PeriodicLoRAReLoRAA3h=id​\(stagewise\)h=\\mathrm\{id\}\\ \\text\{\(stagewise\)\}PERKLoRAA3hc​\(θa​d​a​p​t​e​r,α,ψ\)=ϕ\(N\)​\(θa​d​a​p​t​e​r,α,ψ,c\)h\_\{c\}\(\\theta\_\{adapter\},\\alpha,\\psi\)=\\phi^\{\(N\)\}\(\\theta\_\{adapter\},\\alpha,\\psi;c\)PHATGOOSELoRAA3hx\(v\)=\(\[wx,1\(v\)B1⋯wx,N\(v\)BN\],\[A1;…;AN\]\),wx,z=0off the top\-kh\_\{x\}\(v\)=\\big\(\[w\_\{x,1\}\(v\)B\_\{1\}\\ \\cdots\\ w\_\{x,N\}\(v\)B\_\{N\}\],\\ \[A\_\{1\};\\dots;A\_\{N\}\]\\big\),\\ \\ w\_\{x,z\}=0\\ \\text\{off the top\-\}kPHM\-AdapterAdapter \(Houlsby\)A3h⁡\(\{AiD,BiD,AiU,BiU\}\)=\(Wd​o​w​n=∑iAiD⊗BiD,Wu​p=∑iAiU⊗BiU\)h\(\\\{A\_\{i\}^\{D\},B\_\{i\}^\{D\},A\_\{i\}^\{U\},B\_\{i\}^\{U\}\\\}\)=\\big\(W\_\{down\}=\\textstyle\\sum\_\{i\}A\_\{i\}^\{D\}\\otimes B\_\{i\}^\{D\},\\;W\_\{up\}=\\sum\_\{i\}A\_\{i\}^\{U\}\\otimes B\_\{i\}^\{U\}\\big\)PicoLoRAA3h\(∗\)=\(\[γSB1⋯γSBT\],\[A1;…;AT\]\)h\(\\ast\)=\\big\(\[\\,\\gamma SB\_\{1\}\\ \\cdots\\ \\gamma SB\_\{T\}\\,\],\\ \[A\_\{1\};\\dots;A\_\{T\}\]\\big\)PiggybackDiff PruningA3M↦\(α0,−\(𝟏−M\)⊙W0\)M\\mapsto\\big\(\\alpha\_\{0\},\\ \-\(\\mathbf\{1\}\-M\)\\odot W\_\{0\}\\big\)PiggybackHiRAA3M↦\(M−𝟏,In\)M\\mapsto\(M\-\\mathbf\{1\},\\ I\_\{n\}\)PiSSALoRAA3h⁡\(B,A\)=\(\[B,−B0\],\[A;A0\]\)h\(B,A\)=\(\[B,\\,\-B\_\{0\}\],\\,\[A;\\,A\_\{0\}\]\)PLoPLoRAA2h⁡\(qT\)=\(qT,q0\|Tc\)h\(q\_\{T\}\)=\\big\(q\_\{T\},\\ q\_\{0\}\|\_\{T^\{c\}\}\\big\)PoLARLoRAA3h⁡\(X,Θ,Y\)=\(X​Θ,Y⊤\)h\(X,\\Theta,Y\)=\(X\\Theta,\\ Y^\{\\top\}\)Polytropon \(Poly\)LoRAA3hτ​\(A,B,Z\)=\(∑jατ​j​Bj,∑jατ​j​Aj\)h\_\{\\tau\}\(A,B,Z\)=\\big\(\\textstyle\\sum\_\{j\}\\alpha\_\{\\tau j\}B\_\{j\},\\ \\sum\_\{j\}\\alpha\_\{\\tau j\}A\_\{j\}\\big\)Polytropon \(Poly\)MHRA3h⁡\(A,B,Z\)=\(A,B,Z⊗𝟏h\)h\(A,B,Z\)=\(A,\\,B,\\,Z\\otimes\\mathbf\{1\}\_\{h\}\)PPTPrompt TuningA3h=idℝℓ×dh=\\mathrm\{id\}\_\{\\mathbb\{R\}^\{\\ell\\times d\}\}Prefix\-TuningInitial State TuningA3h⁡\(x−ℓ\.\.−1\(l\)\)=\(∑i=1ℓA¯li−1​B¯l​x−i\(l\)\)lh\\big\(x^\{\(l\)\}\_\{\-\\ell\.\.\-1\}\\big\)=\\Big\(\\textstyle\\sum\_\{i=1\}^\{\\ell\}\\bar\{A\}\_\{l\}^\{\\,i\-1\}\\bar\{B\}\_\{l\}\\,x^\{\(l\)\}\_\{\-i\}\\Big\)\_\{l\}Prefix\-TuningMAM AdapterA5h⁡\(P′,θ\)=\(P′,θ,Wup=0,Wdown0\)h\(P^\{\\prime\},\\theta\)=\\big\(P^\{\\prime\},\\,\\theta,\\,W\_\{\\mathrm\{up\}\}=0,\\,W^\{0\}\_\{\\mathrm\{down\}\}\\big\)Prefix\-TuningP\-Tuning v2A3h⁡\(P′,θ\)=\(MLPθ​\(P′\),Wcls0\)h\(P^\{\\prime\},\\theta\)=\\big\(\\mathrm\{MLP\}\_\{\\theta\}\(P^\{\\prime\}\),\\,W^\{0\}\_\{\\mathrm\{cls\}\}\\big\)PreFTLoReFTA3\(W1,W2,b\)↦\(R=\(W2W2⊤\)−1/2W2,W=R\+sr\(W2W2⊤\)1/2W1,b′=sr\(W2W2⊤\)1/2b\)\(W\_\{1\},W\_\{2\},b\)\\mapsto\\big\(R=\(W\_\{2\}W\_\{2\}^\{\\top\}\)^\{\-1/2\}W\_\{2\},\\ W=R\+s\_\{r\}\(W\_\{2\}W\_\{2\}^\{\\top\}\)^\{1/2\}W\_\{1\},\\ b^\{\\prime\}=s\_\{r\}\(W\_\{2\}W\_\{2\}^\{\\top\}\)^\{1/2\}b\\big\)PRISMLoRAA3\(B,A\)↦\(\[B,B0\],\[A;−A0\]\)\(B,A\)\\mapsto\\big\(\[B,\\ B\_\{0\}\],\\ \[A;\\ \-A\_\{0\}\]\\big\)PRISMPiSSAA3\(B,A\)↦\(s​B,s​A\)\(B,A\)\\mapsto\(\\sqrt\{s\}\\,B,\\ \\sqrt\{s\}\\,A\)PRoLoRALoRAA3h⁡\(Bu,B0,Au,A0\)=\(ℬ⁡\(Bu,B0\),𝒜⁡\(Au,A0\)\)h\(B\_\{u\},B\_\{0\},A\_\{u\},A\_\{0\}\)=\\big\(\\mathcal\{B\}\(B\_\{u\},B\_\{0\}\),\\;\\mathcal\{A\}\(A\_\{u\},A\_\{0\}\)\\big\)Prompt TuningATTEMPTA3h⁡\(P\)=\(Ptgt=\(t\+1\)​P−∑j≤tPjt\+2,γ=0,β=0,Wdown0,Wup0\)h\(P\)=\\Big\(P\_\{\\mathrm\{tgt\}\}=\\tfrac\{\(t\+1\)P\-\\sum\_\{j\\leq t\}P\_\{j\}\}\{t\+2\},\\ \\gamma=0,\\ \\beta=0,\\ W^\{0\}\_\{\\mathrm\{down\}\},\\ W^\{0\}\_\{\\mathrm\{up\}\}\\Big\)Prompt TuningMPTA3h⁡\(P\)=\(P⊘u0​v0⊤,u0,v0\)h\(P\)=\\big\(P\\oslash u\_\{0\}v\_\{0\}^\{\\top\},\\,u\_\{0\},\\,v\_\{0\}\\big\)Prompt TuningP\-TuningA3h\(P\)=\(P0′,ψ0withW2↦W20\+H0\+\(P−P0\)\)h\(P\)=\\big\(P^\{\\prime\}\_\{0\},\\;\\psi\_\{0\}\\ \\text\{with\}\\ W\_\{2\}\\mapsto W\_\{2\}^\{0\}\+H\_\{0\}^\{\+\}\(P\-P\_\{0\}\)\\big\)Prompt TuningP\-Tuning v2A3h⁡\(p\)=\(\(Kj​\(p\),Vj​\(p\)\)j≤L,Wcls0\)h\(p\)=\\big\(\(K\_\{j\}\(p\),V\_\{j\}\(p\)\)\_\{j\\leq L\},\\,W^\{0\}\_\{\\mathrm\{cls\}\}\\big\)Prompt2EffectLoRAA3hc​\(ϕ\)=\(B^ϕ​\(c,W0\),A^ϕ​\(c,W0\)\)h\_\{c\}\(\\phi\)=\\big\(\\hat\{B\}\_\{\\phi\}\(c,W\_\{0\}\),\\ \\hat\{A\}\_\{\\phi\}\(c,W\_\{0\}\)\\big\)Proxy\-TuningEFTA3h=idh=\\mathrm\{id\}PSOFTLoRA\-XSA3\(Q,α,β\)↦Σr​\(Dβ​C​\(Q\)⊤​Dα−I\)​Σr−1\(Q,\\alpha,\\beta\)\\mapsto\\Sigma\_\{r\}\\big\(D\_\{\\beta\}C\(Q\)^\{\\top\}D\_\{\\alpha\}\-I\\big\)\\Sigma\_\{r\}^\{\-1\}PSOFTPiSSAA3\(Q,α,β\)↦\(UrΣrDβC\(Q\)⊤DαΣr−1/2,Σr1/2Vr⊤\)\(Q,\\alpha,\\beta\)\\mapsto\\big\(U\_\{r\}\\Sigma\_\{r\}D\_\{\\beta\}C\(Q\)^\{\\top\}D\_\{\\alpha\}\\Sigma\_\{r\}^\{\-1/2\},\\ \\Sigma\_\{r\}^\{1/2\}V\_\{r\}^\{\\top\}\\big\)PunicaLoRAA3hu​\(∗\)=\(Bu,Au\)h\_\{u\}\(\\ast\)=\(B\_\{u\},\\ A\_\{u\}\)PunicaS\-LoRAA3h=id∏uQLoRAruh=\\mathrm\{id\}\_\{\\prod\_\{u\}Q\_\{\\mathrm\{LoRA\}\_\{r\_\{u\}\}\}\}PVeRALoRA\-FAA3h⁡\(b,dμ,dσ\)=α​Λb​B​Λdμh\(b,d\_\{\\mu\},d\_\{\\sigma\}\)=\\alpha\\Lambda\_\{b\}B\\Lambda\_\{d\_\{\\mu\}\}PVeRAVeRAA3h⁡\(b,dμ,dσ\)=\(b,dμ\)h\(b,d\_\{\\mu\},d\_\{\\sigma\}\)=\(b,\\ d\_\{\\mu\}\)QA\-LoRAQLoRAA3\(B,A\)↦\(ss′​B,A​P\)\(B,A\)\\mapsto\(\\tfrac\{s\}\{s^\{\\prime\}\}B,\\ AP\)QLoRAIR\-QLoRAA2\(B,A\)↦\(sα​B,A,0,0\)\(B,A\)\\mapsto\(\\tfrac\{s\}\{\\alpha\}B,\\ A,\\ 0,\\ 0\)Quantum\-PEFTAdaLoRAA3h⁡\(θU,θV,λ\)=\(P=QP​\(θU\)​Ido​u​t×K,Λ=diag⁡\(λ\),Q=\(QP​\(θV\)​Idi​n×K\)⊤\)h\(\\theta\_\{U\},\\theta\_\{V\},\\lambda\)=\\big\(P=Q\_\{P\}\(\\theta\_\{U\}\)I\_\{d\_\{out\}\\times K\},\\;\\Lambda=\\mathrm\{diag\}\(\\lambda\),\\;Q=\(Q\_\{P\}\(\\theta\_\{V\}\)I\_\{d\_\{in\}\\times K\}\)^\{\\top\}\\big\)Quantum\-PEFTLoRAA3h=\(U⁡\(θU\)​diag​\(λ\),V​\(θV\)⊤\)h=\\big\(U\(\\theta\_\{U\}\)\\,\\mathrm\{diag\}\(\\lambda\),\\;V\(\\theta\_\{V\}\)^\{\\top\}\\big\)QZOEfficientQATA3h=id:S↦Sh=\\mathrm\{id\}:\\ S\\mapsto SRandLoRALoRAA3h⁡\(Λ,Γ\)=\(\[B1​Λ1,…,Bn​Λn\],\[A​Γ1;…;A​Γn\]\)h\(\\Lambda,\\Gamma\)=\\big\(\[B\_\{1\}\\Lambda\_\{1\},\\dots,B\_\{n\}\\Lambda\_\{n\}\],\\ \[A\\Gamma\_\{1\};\\dots;A\\Gamma\_\{n\}\]\\big\)RaSALoRAA3h=\(\[B~i​BS\]​Di,\[A~i;AS\]\)ih=\\big\(\[\\tilde\{B\}\_\{i\}\\ \\ B\_\{S\}\]\\,D\_\{i\},\\;\[\\tilde\{A\}\_\{i\};\\,A\_\{S\}\]\\big\)\_\{i\}REDSSFA2\(ℓs,ℓb\)↦\(\(γ,β\)FFN​out=\(ℓs,ℓb\),all other SSF wrappers at​\(𝟏,0\)\)\(\\ell\_\{s\},\\ell\_\{b\}\)\\mapsto\\big\(\(\\gamma,\\beta\)\_\{\\rm FFN\\ out\}=\(\\ell\_\{s\},\\ell\_\{b\}\);\\ \\text\{all other SSF wrappers at \}\(\\mathbf\{1\},0\)\\big\)ReLoRACOLAA3h=id​\(stagewise\)h=\\mathrm\{id\}\\ \\text\{\(stagewise\)\}ReLoRALoRAA3h\(\(Bk,Ak\)k\)=\(\[B1⋯BK\],\[A1;…;AK\]\)h\(\(B\_\{k\},A\_\{k\}\)\_\{k\}\)=\(\[B\_\{1\}\\cdots B\_\{K\}\],\\,\[A\_\{1\};\\dots;A\_\{K\}\]\)ReLoRAReLoRAA5\(q1,…,qK\)↦\(q1,…,qK,q0\(K\+1\)\)\(q\_\{1\},\\dots,q\_\{K\}\)\\mapsto\(q\_\{1\},\\dots,q\_\{K\},\\,q\_\{0\}^\{\(K\+1\)\}\)RepAdapterLoRAA3h⁡\(Wu,Wd\)=\(\(s​W0\(x\)​Wu,Wd\)\)x∈\{q,k,v\}h\(W\_\{u\},W\_\{d\}\)=\\big\(\(sW\_\{0\}^\{\(x\)\}W\_\{u\},\\ W\_\{d\}\)\\big\)\_\{x\\in\\\{q,k,v\\\}\}Residual AdaptersLoRA\-FAA3h⁡\(α,D1,D2\)=D2​\(I\+α​D1\)−I​as the trainable​B,with​A0=W0,r=Ch\(\\alpha,D\_\{1\},D\_\{2\}\)=D\_\{2\}\(I\+\\alpha D\_\{1\}\)\-I\\ \\ \\text\{as the trainable \}B,\\ \\text\{with \}A\_\{0\}=W\_\{0\},\\ r=CResidual Prompt TuningPrompt TuningA3h⁡\(P,Wdown,Wup,LN\)=LN⁡\(ReLU⁡\(P​Wdown\)​Wup\)\+Ph\(P,W\_\{\\rm down\},W\_\{\\rm up\},\\mathrm\{LN\}\)=\\mathrm\{LN\}\\big\(\\mathrm\{ReLU\}\(PW\_\{\\rm down\}\)W\_\{\\rm up\}\\big\)\+PRiemannian LoRALoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}RiemannionLoRAA3X↦\(\[X​\(B0\+​X\)\+,−B0\],\[B0\+​X;A0\]\),X0=B0​A0X\\mapsto\\big\(\[\\,X\(B\_\{0\}^\{\+\}X\)^\{\+\},\\ \-B\_\{0\}\\,\],\\ \[\\,B\_\{0\}^\{\+\}X;\\ A\_\{0\}\\,\]\\big\),\\hskip 16\.38895ptX\_\{0\}=B\_\{0\}A\_\{0\}RotMoLELoRAA3hx​\(\(Bi,Ai,qi\)i,Wg,Wθ\)=\(\[1i∈TopK⁡\(x\)​gi​\(x\)​Bi​Ri​\(x\)\]i=1n,\[A1;…;An\]\)h\_\{x\}\\big\(\(B\_\{i\},A\_\{i\},q\_\{i\}\)\_\{i\},W\_\{g\},W\_\{\\theta\}\\big\)=\\big\(\\,\[\\,\\mathbb\{1\}\_\{i\\in\\mathrm\{TopK\}\(x\)\}\\,g\_\{i\}\(x\)\\,B\_\{i\}R\_\{i\}\(x\)\\,\]\_\{i=1\}^\{n\},\\ \[A\_\{1\};\\dots;A\_\{n\}\]\\big\)rsLoRALoRAA3h⁡\(B,A\)=\(r​B,A\)h\(B,A\)=\(\\sqrt\{r\}\\,B,\\ A\)S\-LoRALoRA Soups \(CAT\)A3hu​\(∗\)=euh\_\{u\}\(\\ast\)=e\_\{u\}S\-LoRAPunicaA3h=id∏uQLoRAruh=\\mathrm\{id\}\_\{\\prod\_\{u\}Q\_\{\\mathrm\{LoRA\}\_\{r\_\{u\}\}\}\}S2FTDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)S2FTLoRAA2V↦\(s−1​V,US⊤\)V\\mapsto\(s^\{\-1\}V,\\ U\_\{S\}^\{\\top\}\)S2FTLoRA\-FAA3V↦s−1​V​\(A0:=US⊤\)V\\mapsto s^\{\-1\}V\\hskip 8\.19447pt\(A\_\{0\}:=U\_\{S\}^\{\\top\}\)SAM \(sparse\)Diff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)ScaLoRALoRAA3h⁡\(B,A\)=\(\[B,−B~t\],\[A;A~t\]\)h\(B,A\)=\\big\(\[\\,B,\\ \-\\widetilde\{B\}\_\{t\}\\,\],\\ \[\\,A;\\ \\widetilde\{A\}\_\{t\}\\,\]\\big\)SD\-LoRALoRAA3h=\(\[α1B^1⋯αt−1B^t−1αtBt/∥BtAt∥F\],\[A^1;…;A^t−1;At\]\),Dk=B^kA^kh=\\big\(\[\\alpha\_\{1\}\\hat\{B\}\_\{1\}\\ \\cdots\\ \\alpha\_\{t\-1\}\\hat\{B\}\_\{t\-1\}\\ \\ \\alpha\_\{t\}B\_\{t\}/\\lVert B\_\{t\}A\_\{t\}\\rVert\_\{F\}\],\\ \[\\hat\{A\}\_\{1\};\\dots;\\hat\{A\}\_\{t\-1\};A\_\{t\}\]\\big\),\\ \\ D\_\{k\}=\\hat\{B\}\_\{k\}\\hat\{A\}\_\{k\}SEALLoRAA3h=idh=\\mathrm\{id\}ShareLoRALoRAA3h⁡\(\(Bℓ\)ℓ,A\)=\(Bℓ,A\)ℓh\\big\(\(B\_\{\\ell\}\)\_\{\\ell\},A\\big\)=\(B\_\{\\ell\},A\)\_\{\\ell\}SHINELoRAA3hc​\(Δmeta,m,ψ\)=M2Pψ​\(ℳ⁡\(c\)\)lh\_\{c\}\(\\Delta\_\{\\rm meta\},m,\\psi\)=\\mathrm\{M2P\}\_\{\\psi\}\\big\(\\mathcal\{M\}\(c\)\\big\)\_\{l\}SHiRARoSAA5h⁡\(Δ\)=\(B=0,A=A0,S=M⊙Δ\)h\(\\Delta\)=\\big\(B=0,\\;A=A\_\{0\},\\;S=M\\odot\\Delta\\big\)SIFTDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)SingLoRALoRAA3h\(A\)=\(αruA\[1:do​u​t\],A\[1:di​n\]⊤\)h\(A\)=\\big\(\\tfrac\{\\alpha\}\{r\}\\,u\\,A\_\{\[1:d\_\{out\}\]\},\\ A\_\{\[1:d\_\{in\}\]\}^\{\\top\}\\big\)SLAO \(Merge before Forget\)LoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}SLTrainRoSAA3h=idh=\\mathrm\{id\}SmartFed \(MoRE \+ EEQA\)LoRAA3hx​\(Wr​o​u​t​e​r\)=\(B^​diag​\(g~x​\(Wr​o​u​t​e​r\)\),A^\)h\_\{x\}\(W\_\{router\}\)=\\big\(\\widehat\{B\}\\,\\mathrm\{diag\}\(\\tilde\{g\}\_\{x\}\(W\_\{router\}\)\),\\ \\widehat\{A\}\\big\)SMEARAdapter \(Houlsby\)A3hv​\(θ1,…,θN,Wr\)=∑iR​\(v\)i​θih\_\{v\}\(\\theta\_\{1\},\\dots,\\theta\_\{N\},W\_\{r\}\)=\\textstyle\\sum\_\{i\}R\(v\)\_\{i\}\\theta\_\{i\}SMoAHiRAA3\(Bk,Ak\)k↦\(Po​u​t⊤​blkdiag​\(B1,…,BK\),blkdiag⁡\(A1,…,AK\)​Pi​n\)\(B\_\{k\},A\_\{k\}\)\_\{k\}\\mapsto\\big\(P\_\{out\}^\{\\top\}\\,\\mathrm\{blkdiag\}\(B\_\{1\},\\dots,B\_\{K\}\),\\ \\mathrm\{blkdiag\}\(A\_\{1\},\\dots,A\_\{K\}\)\\,P\_\{in\}\\big\)SMTDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)SoRALoRAA3h⁡\(B,g,A\)=\(B​diag​\(g\),A\)h\(B,g,A\)=\(B\\,\\mathrm\{diag\}\(g\),\\ A\)Sparse Memory FinetuningFISH MaskA3h=id,M=rows⁡\(𝒯t\)h=\\mathrm\{id\},\\ \\ M=\\mathrm\{rows\}\(\\mathcal\{T\}\_\{t\}\)SparseLoRALoRAA3h=idh=\\mathrm\{id\}Spectral AdapterAPiSSAA3\(AU,AV\)↦\(\(U1\+AU\)​S11/2,S11/2​\(V1\+AV\)⊤\)\(A\_\{U\},A\_\{V\}\)\\mapsto\\big\(\(U\_\{1\}\+A\_\{U\}\)S\_\{1\}^\{1/2\},\\ S\_\{1\}^\{1/2\}\(V\_\{1\}\+A\_\{V\}\)^\{\\top\}\\big\)Spectral AdapterRLoRA\-XSA3\(QU,QV\)↦C⁡\(QU\)​S1​C​\(QV\)⊤​S1−1−I\(Q\_\{U\},Q\_\{V\}\)\\mapsto C\(Q\_\{U\}\)\\,S\_\{1\}\\,C\(Q\_\{V\}\)^\{\\top\}S\_\{1\}^\{\-1\}\-ISpectral AdapterRPiSSAA3\(QU,QV\)↦\(U1C\(QU\)S1C\(QV\)⊤S1−1/2,S11/2V1⊤\)\(Q\_\{U\},Q\_\{V\}\)\\mapsto\\big\(U\_\{1\}C\(Q\_\{U\}\)S\_\{1\}C\(Q\_\{V\}\)^\{\\top\}S\_\{1\}^\{\-1/2\},\\ S\_\{1\}^\{1/2\}V\_\{1\}^\{\\top\}\\big\)SpIELDiff PruningA2\(η,ϕ\)↦\(α0,scatter⁡\(η,ϕ\)\)\(\\eta,\\phi\)\\mapsto\\big\(\\alpha\_\{0\},\\ \\mathrm\{scatter\}\(\\eta,\\phi\)\\big\)SPoTPrompt TuningA3h=idℝℓ×dh=\\mathrm\{id\}\_\{\\mathbb\{R\}^\{\\ell\\times d\}\}SPTRoSAA2\(\(Bs,As\)s∈S,\(xu\)u∈U\)↦\(\(Bs,As\)s∈S,\(0,A0,u\)u∈U,ι⁡\(x\)\)\\big\(\(B\_\{s\},A\_\{s\}\)\_\{s\\in S\},\\,\(x\_\{u\}\)\_\{u\\in U\}\\big\)\\mapsto\\big\(\(B\_\{s\},A\_\{s\}\)\_\{s\\in S\},\\,\(0,A\_\{0,u\}\)\_\{u\\in U\};\\ \\iota\(x\)\\big\)SRR \(Structured Residual Reconstruction\)LoRAA3\(B,A\)↦\(\[B,−B0\],\[A;A0\]\)\(B,A\)\\mapsto\\big\(\[B,\\ \-B\_\{0\}\],\\ \[A;\\ A\_\{0\}\]\\big\)SSFGLoRAA3\(γ,β\)↦\(BA=γ−𝟏,AA=𝟏⊤,D=γ−𝟏,E=β\)​\(other paths at zero\)\(\\gamma,\\beta\)\\mapsto\\big\(B\_\{A\}=\\gamma\-\\mathbf\{1\},\\ A\_\{A\}=\\mathbf\{1\}^\{\\top\};\\ D=\\gamma\-\\mathbf\{1\};\\ E=\\beta\\big\)\\ \\text\{\(other paths at zero\)\}Stable\-LoRALoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}StructLoRALoRAA3\(A,B,m\)↦\(αr​B​diag​\(m\),A\)\(A,B,m\)\\mapsto\\big\(\\tfrac\{\\alpha\}\{r\}B\\,\\mathrm\{diag\}\(m\),\\ A\\big\)StructLoRASoRAA3\(A,B,m\)↦\(B,αr​m,A\)\(A,B,m\)\\mapsto\\big\(B,\\ \\tfrac\{\\alpha\}\{r\}m,\\ A\\big\)Super\-Tuning \(Super / Supra\)PaFiA5h⁡\(U\)=\(U,γ0,β0\)h\(U\)=\\big\(U,\\,\\gamma\_\{0\},\\,\\beta\_\{0\}\\big\)Super\-Tuning \(Super / Supra\)SHiRAA3h=idh=\\mathrm\{id\}Surgical FTDiff PruningA2x↦\(α0,ιM​x\)x\\mapsto\(\\alpha\_\{0\},\\ \\iota\_\{M\}x\)Surgical FTHiFTA5q↦\(stagej:q;other stages:0\)q\\mapsto\(\\text\{stage \}j:q;\\ \\text\{other stages\}:0\)SVDiffSVF \(Transformer2\)A3δ↦ReLU⁡\(σ\+δ\)⊘σ\\delta\\mapsto\\mathrm\{ReLU\}\(\\sigma\+\\delta\)\\oslash\\sigmaSVDiffSVFTA3δ↦ReLU⁡\(σ\+δ\)−σ\\delta\\mapsto\\mathrm\{ReLU\}\(\\sigma\+\\delta\)\-\\sigmaSVF \(Transformer2\)SVFTA3z↦σ⊙\(z−𝟏\)z\\mapsto\\sigma\\odot\(z\-\\mathbf\{1\}\)SVFTLoRA\-FAA3h\(M\)=U:,RMR,C,A0=V:,C⊤h\(M\)=U\_\{:,R\}\\,M\_\{R,C\},\\ \\ A\_\{0\}=V\_\{:,C\}^\{\\top\}Task ArithmeticAdaMergingA3h⁡\(λ\)=\(λil=λi\)i,lh\(\\lambda\)=\(\\lambda\_\{i\}^\{l\}=\\lambda\_\{i\}\)\_\{i,l\}Task ArithmeticLoRA Soups \(CAT\)A3h⁡\(λ\)=\(αil=λi\)i,lh\(\\lambda\)=\(\\alpha\_\{i\}^\{l\}=\\lambda\_\{i\}\)\_\{i,l\}Task ArithmeticZipLoRAA3h⁡\(λc,λs\)=\(λc​𝟏,λs​𝟏\)h\(\\lambda\_\{c\},\\lambda\_\{s\}\)=\(\\lambda\_\{c\}\\mathbf\{1\},\\ \\lambda\_\{s\}\\mathbf\{1\}\)TeRAVeRAA3h⁡\(d\(1\),d\(2\)\)=\(b=d\(1\),d=d\(2\)\)h\(d^\{\(1\)\},d^\{\(2\)\}\)=\\big\(b=d^\{\(1\)\},\\ d=d^\{\(2\)\}\\big\)Text\-to\-LoRA \(T2L\)LoRAA3hz​\(θ\)=hθ​\(concat⁡\[f⁡\(z\),E⁡\[m\],E⁡\[l\]\]\)h\_\{z\}\(\\theta\)=h\_\{\\theta\}\\big\(\\mathrm\{concat\}\[f\(z\),E\[m\],E\[l\]\]\\big\)Textual InversionCustom DiffusionA2h⁡\(v\)=\(Δ​Wk=0,Δ​Wv=0,V∗=v\)h\(v\)=\(\\Delta W^\{k\}=0,\\ \\Delta W^\{v\}=0,\\ V^\{\*\}=v\)Textual InversionTrainable TokensA3h=id:v↦V\{S∗\}h=\\mathrm\{id\}:\\ v\\mapsto V\_\{\\\{S\_\{\*\}\\\}\}Tied\-LoRALoRAA3h=\(Λvℓ​B​Λuℓ,A\)ℓh=\\big\(\\Lambda\_\{v\_\{\\ell\}\}B\\Lambda\_\{u\_\{\\ell\}\},\\;A\\big\)\_\{\\ell\}Tied\-LoRAShareLoRAA3h⁡\(\{vℓ,uℓ\}ℓ,B,A\)=\(\{Λvℓ​B​Λuℓ\}ℓ,A\)h\(\\\{v\_\{\\ell\},u\_\{\\ell\}\\\}\_\{\\ell\},B,A\)=\\big\(\\\{\\Lambda\_\{v\_\{\\ell\}\}B\\Lambda\_\{u\_\{\\ell\}\}\\\}\_\{\\ell\},\\;A\\big\)TinaLoRAA3h=idh=\\mathrm\{id\}TinyLoRALoRA\-XSA3h\(v\)=Σr1/2\(∑iviPi\)Σr−1/2h\(v\)=\\Sigma\_\{r\}^\{1/2\}\\Big\(\\sum\_\{i\}v\_\{i\}P\_\{i\}\\Big\)\\Sigma\_\{r\}^\{\-1/2\}TinyLoRAPiSSAA3h⁡\(v\)=\(Ur​Σr1/2​\(I\+∑ivi​Pi\),Σr1/2​Vr⊤\)h\(v\)=\\Big\(U\_\{r\}\\Sigma\_\{r\}^\{1/2\}\\big\(I\+\\textstyle\\sum\_\{i\}v\_\{i\}P\_\{i\}\\big\),\\ \\Sigma\_\{r\}^\{1/2\}V\_\{r\}^\{\\top\}\\Big\)Trainable TokensLoRAA2h⁡\(X\)=\(ST,X\),X=VT−E0,Th\(X\)=\(S\_\{T\},\\,X\),\\hskip 8\.19447ptX=V\_\{T\}\-E\_\{0,T\}Trainable TokensSHiRAA3h=idh=\\mathrm\{id\}Trans\-LoRALoRAA3h=idh=\\mathrm\{id\}TT\-LoRALoRAA3h\(𝒞1,…,𝒞D\)=\(𝒞1×1⋯×1𝒞c,𝒞c\+1×1⋯×1𝒞D\)reshaped todo​u​t×rc,rc×di​nh\(\\mathcal\{C\}\_\{1\},\\dots,\\mathcal\{C\}\_\{D\}\)=\\big\(\\mathcal\{C\}\_\{1\}\\times^\{1\}\\cdots\\times^\{1\}\\mathcal\{C\}\_\{c\},\\;\\mathcal\{C\}\_\{c\+1\}\\times^\{1\}\\cdots\\times^\{1\}\\mathcal\{C\}\_\{D\}\\big\)\\ \\text\{reshaped to \}d\_\{out\}\\times r\_\{c\},\\ r\_\{c\}\\times d\_\{in\}Uni\-LoRALoRAA3h⁡\(θd\)=P​θd=\(Bℓ​\(P​θd\),Aℓ​\(P​θd\)\)ℓh\(\\theta\_\{d\}\)=P\\theta\_\{d\}=\\big\(B^\{\\ell\}\(P\\theta\_\{d\}\),A^\{\\ell\}\(P\\theta\_\{d\}\)\\big\)\_\{\\ell\}VAPTVPTA3hx​\(q\)=\(\(P\(l\)​\(X~\(l\)​\(x,q\)\)\)l≤L,Head\)h\_\{x\}\(q\)=\\big\(\(P^\{\(l\)\}\(\\tilde\{X\}^\{\(l\)\}\(x;q\)\)\)\_\{l\\leq L\},\\ \\mathrm\{Head\}\\big\)VB\-LoRALoRAA3h⁡\(α,σ\)=\(B⁡\(α,σ\),A⁡\(α,σ\)\),u=∑ssoftmax​\(TopK⁡\(σ,k\)\)s​αsh\(\\alpha,\\sigma\)=\\big\(B\(\\alpha,\\sigma\),\\ A\(\\alpha,\\sigma\)\\big\),\\ \\ u=\\textstyle\\sum\_\{s\}\\mathrm\{softmax\}\(\\mathrm\{TopK\}\(\\sigma,k\)\)\_\{s\}\\,\\alpha\_\{s\}VeLoRALoRAA3h=idQh=\\mathrm\{id\}\_\{Q\}VeRALoRAA3h⁡\(b,d\)=\(Λb​B​Λd,A\)h\(b,d\)=\(\\Lambda\_\{b\}B\\Lambda\_\{d\},\\ A\)VeRALoRA\-FAA3h⁡\(b,d\)=Λb​B​Λdh\(b,d\)=\\Lambda\_\{b\}B\\Lambda\_\{d\}VPTP\-Tuning v2A3h⁡\(P,Head\)=\(\(LN⁡\(Pi−1\)​Wk\(i\)\+𝟏​bk\(i\)⊤,LN⁡\(Pi−1\)​Wv\(i\)\+𝟏​bv\(i\)⊤\)i≤L,Head\)h\(P,\\mathrm\{Head\}\)=\\Big\(\\big\(\\mathrm\{LN\}\(P\_\{i\-1\}\)W\_\{k\}^\{\(i\)\}\+\\mathbf\{1\}b\_\{k\}^\{\(i\)\\top\},\\,\\mathrm\{LN\}\(P\_\{i\-1\}\)W\_\{v\}^\{\(i\)\}\+\\mathbf\{1\}b\_\{v\}^\{\(i\)\\top\}\\big\)\_\{i\\leq L\},\\,\\mathrm\{Head\}\\Big\)WaveFTLoRA\-FAA3h⁡\(c\)=λ​BW​CΩ​\(c\),AW=\[ψj⊤\]j∈cols⁡\(Ω\)h\(c\)=\\lambda\\,B\_\{W\}\\,C\_\{\\Omega\}\(c\),\\ \\ A\_\{W\}=\[\\psi\_\{j\}^\{\\top\}\]\_\{j\\in\\mathrm\{cols\}\(\\Omega\)\}WiSE\-FTConcept SlidersA3h⁡\(α\)=\(α​B1,A1\)h\(\\alpha\)=\(\\alpha B\_\{1\},\\ A\_\{1\}\)WiSE\-FTTask ArithmeticA3h⁡\(α\)=\(λ=α\)h\(\\alpha\)=\(\\lambda=\\alpha\)X\-LoRALoRA Soups \(CAT\)A3hx​\(ϕ\)=\(αil=λil​\(x,ϕ\)​αi/ri\)h\_\{x\}\(\\phi\)=\\big\(\\alpha\_\{i\}^\{l\}=\\lambda\_\{i\}^\{l\}\(x;\\phi\)\\,\\alpha\_\{i\}/r\_\{i\}\\big\)XPromptPrompt TuningA2h⁡\(PS\)=\(PS,0Sc\)h\(P\_\{S\}\)=\\big\(P\_\{S\},\\,0\_\{S^\{c\}\}\\big\)ZipLoRALoRAA3h⁡\(mc,ms\)=\(\[Bc​Bs\],\[Ac​diag​\(mc\);As​diag​\(ms\)\]\)h\(m\_\{c\},m\_\{s\}\)=\\big\(\[B\_\{c\}\\ B\_\{s\}\],\\ \[A\_\{c\}\\,\\mathrm\{diag\}\(m\_\{c\}\);\\ A\_\{s\}\\,\\mathrm\{diag\}\(m\_\{s\}\)\]\\big\)ZO\-MuonLoRAA2X↦\(Pk,X\)X\\mapsto\(P\_\{k\},\\ X\)

## Appendix ENumerical checks and reproducibility

Claims marked \[Num\] in the main text are numerical checks and are not counted as proved\. Most of them are re\-run by one script, framework\_checks\.py, one of the data files that accompany this paper\. This appendix says what the script computes and how to read its output \(§[E\.1](https://arxiv.org/html/2610.12728#A5.SS1)\), lists every check with the result it supports and the value it prints \(§[E\.2](https://arxiv.org/html/2610.12728#A5.SS2), Table[38](https://arxiv.org/html/2610.12728#A5.T38)\), names the few numbers of the main text that come from elsewhere \(§[E\.3](https://arxiv.org/html/2610.12728#A5.SS3)\), records the software \(§[E\.4](https://arxiv.org/html/2610.12728#A5.SS4)\), describes how the companion site and the figures of this paper compute what they show \(§[E\.5](https://arxiv.org/html/2610.12728#A5.SS5)\), and gives the settings and values behind each figure \(§[E\.6](https://arxiv.org/html/2610.12728#A5.SS6)\)\.

### E\.1 The script

The script runs with python3 framework\_checks\.py and needs PyTorch\([Paszke et al\., 2019](https://arxiv.org/html/2610.12728#bib.bib362)\), NumPy\([Harris et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib155)\)and SciPy\([Virtanen et al\., 2020](https://arxiv.org/html/2610.12728#bib.bib461)\)\. It works in float64 throughout \(torch\.set\_default\_dtype\(torch\.float64\)\) and on the CPU\. It seeds PyTorch’s generator at the start and again before several checks, so that a run prints the same numbers every time on a given machine\. On a laptop \(Apple M5\) it finishes in about5050seconds\.

It computes three kinds of quantity\.

- •*Dimensions\.*The expressive dimensiond⁡\(M\)d\(M\)of a method is the generic rank of its Jacobian \([Background C\.3](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx1)\)\. The script forms the Jacobian ofρ\\rhoby automatic differentiation \(torch\.autograd\.functional\.jacobian\) at a random point, or at the base point when it measures a first\-order imageS1S\_\{1\}, and counts the singular values above10−910^\{\-9\}times the largest\. The points of lower rank have measure zero \([Background C\.21](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1),[Background C\.22](https://arxiv.org/html/2610.12728#A1.SSx3.SSSx4.Px1)\), so this computes the generic rank with probability one, up to the tolerance\.
- •*Optimizer runs\.*Adam, decoupled weight decay and gradient descent are written out step by step, with the bias correction and the convention0/0=00/0=0atε=0\\varepsilon=0, so that nothing is hidden in a library default\. Gradient flows are integrated by explicit Euler steps or by the classical fourth\-order Runge–Kutta method\.
- •*Identities and defects\.*An identity or an equivariance is tested by computing both sides and printing the Frobenius norm of their difference\.

Each line prints a value beside its prediction\. Some lines are controls, cases in which the theory predicts that an identity fails; they show that the test would detect a failure\.

##### How sharp the rank decisions are

The script takes4545Jacobian ranks\. In every one of them the smallest singular value it keeps is at least3\.9×10−33\.9\\times 10^\{\-3\}times the largest, and the largest singular value it discards is at most4\.3×10−164\.3\\times 10^\{\-16\}times the largest, which is rounding error\. The threshold10−910^\{\-9\}lies more than six orders of magnitude from both, so no rank in Table[38](https://arxiv.org/html/2610.12728#A5.T38)depends on its choice\. We measured these margins with a copy of the script whose rank function also records the two ratios; the copy prints the same output\.

##### What changes between machines

The ranks and dimensions are integer decisions taken with the margins above, and the exact zero of check 3 comes from rescaling by a power of two, which is exact in IEEE arithmetic\. These outputs are expected to be the same on any machine\. Values of order10−1510^\{\-15\}are rounding errors, and their leading digits change with the processor and the linear\-algebra library\. Values of order one depend on the random draws, which follow from the seeds\.

### E\.2 The checks, section by section

The script has thirteen sections\. Sections 1 to 11 are the numerical claims of the framework, section 12 re\-runs the main numbers of the first referee round, and section 13 those of the second\. Table[38](https://arxiv.org/html/2610.12728#A5.T38)lists every check in the script’s order, with the result it supports, what it computes and what it prints in the run of 2 October 2026\. Every printed value agrees with its prediction, and every control fails, by amounts between1\.6×10−21\.6\\times 10^\{\-2\}and5252\. Matrix shapes are written\(m,n,r\)\(m,n,r\)for anm×nm\\times nweight and rankrr; a defect is the Frobenius norm of the difference between the two sides of the identity tested\.

Table 38:The checks of framework\_checks\.py, in the script’s order, with the result each one supports, what it computes, and what it prints\. Controls are cases in which the theory predicts a failure\.CheckSupportsWhat the script computesWhat it printsSections 1–11: the numerical claims of the framework1[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\(b\)HRArwithout Gram–Schmidt,u↦W0H\(u1\)⋯H\(ur\)u\\mapsto W\_\{0\}H\(u\_\{1\}\)\\cdots H\(u\_\{r\}\), with an injectiveW0∈ℝ\(n\+2\)×nW\_\{0\}\\in\\mathbb\{R\}^\{\(n\+2\)\\times n\}\. Jacobian rank at a random point, and at HF’s paired initialisation \(u2​i−1=u2​iu\_\{2i\-1\}=u\_\{2i\}, sok=r/2k=r/2distinct vectors\)\.At\(n,r\)=\(9,2\)\(n,r\)=\(9,2\),\(9,4\)\(9,4\),\(10,2\)\(10,2\),\(10,4\)\(10,4\),\(12,6\)\(12,6\): generic1515,2626,1717,3030,5151, equal tor​n−r⁡\(r\+1\)/2rn\-r\(r\+1\)/2; paired88,1515,99,1717,3030, equal tok​n−k⁡\(k\+1\)/2kn\-k\(k\+1\)/2\.2[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(b\)The first Adam step from zero state withε=0\\varepsilon=0is sign descent\. Its first\-order weight updated​B​A\+B​d​AdB\\,A\+B\\,dAat\(B,A\)\(B,A\)and at\(B​g,g−1​A\)\(Bg,g^\{\-1\}A\),\(m,n,r\)=\(6,5,3\)\(m,n,r\)=\(6,5,3\)\.Defect1\.4×10−151\.4\\times 10^\{\-15\}for a signed permutationgg\. Controls:11\.311\.3forg=diag⁡\(2,0\.5,3\)g=\\diag\(2,0\.5,3\)and6\.66\.6for a random rotation\. With per\-channel rates\(ηB​dj,ηA/dj\)\(\\eta\_\{B\}d\_\{j\},\\eta\_\{A\}/d\_\{j\}\)the diagonalg=diag⁡\(d\)g=\\diag\(d\)gives5\.0×10−165\.0\\times 10^\{\-16\}\.3[Cor III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7),[Thm III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)200200steps of bias\-corrected Adam withε=0\\varepsilon=0on a least\-squares toy \(W0∈ℝ7×6W\_\{0\}\\in\\mathbb\{R\}^\{7\\times 6\},r=2r=2,B0=0B\_\{0\}=0\): LoRA\+ with ratioλ=8\\lambda=8and scaless, against LoRA with scaleλ​s\\lambda sand equal rates\.Largest weight difference over the whole trajectory0\.00\.0, exactly, sinceλ\\lambdais a power of two\. Control without the rescaling:0\.930\.93\.4[Prop II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)GraLoRA withk×kk\\times kblocks of rankr/kr/k: Jacobian rank, and the rank ofΔ​W\\Delta Wat a random point\.At\(m,n,r,k\)=\(8,8,4,2\)\(m,n,r,k\)=\(8,8,4,2\),\(12,8,4,2\)\(12,8,4,2\),\(12,12,6,3\)\(12,12,6,3\): dimension4848,6464,108108, equal tor⁡\(m\+n−r\)r\(m\+n\-r\); rank ofΔ​W\\Delta W88,88,1212, against LoRA’s cap44,44,66\.5[Prop II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)LoHar,r,\(B1​A1\)⊙\(B2​A2\)\(B\_\{1\}A\_\{1\}\)\\odot\(B\_\{2\}A\_\{2\}\): Jacobian rank\.At\(m,n,r\)=\(8,7,2\)\(m,n,r\)=\(8,7,2\),\(12,10,2\)\(12,10,2\),\(12,10,3\)\(12,10,3\):3838,5959,9393, equal to the bound2​r​\(m\+n−r\)−\(m\+n−1\)2r\(m\+n\-r\)\-\(m\+n\-1\)\.6[Cor III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)\(b\)Euler integration \(h=10−4h=10^\{\-4\},T=2T=2\) of LoRA’s gradient flow with decayλ=0\.5\\lambda=0\.5at\(m,n,r\)=\(5,4,2\)\(m,n,r\)=\(5,4,2\); the final chargeΦ\\Phiagainst its closed form\.Coupled decay atηB/ηA=4\\eta\_\{B\}/\\eta\_\{A\}=4againste−2​ηA​λ​t​Φ​\(0\)e^\{\-2\\eta\_\{A\}\\lambda t\}\\Phi\(0\): relative error10\.710\.7\(control\)\. Coupled decay atηA=ηB\\eta\_\{A\}=\\eta\_\{B\}:2\.5×10−42\.5\\times 10^\{\-4\}\. Per\-time decay atηB/ηA=4\\eta\_\{B\}/\\eta\_\{A\}=4againste−2​λ​t​Φ​\(0\)e^\{\-2\\lambda t\}\\Phi\(0\):2\.3×10−42\.3\\times 10^\{\-4\}\. Both are integrator errors\.7[Thm III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3),[Prop III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)At a random point,\(m,n,r\)=\(7,6,2\)\(m,n,r\)=\(7,6,2\), under a genericg∈GL2g\\in\\mathrm\{GL\}\_\{2\}: the plain\-GD cometricG​A⊤​A\+B​B⊤​GGA^\{\\top\}A\+BB^\{\\top\}Gand the undamped scaled\-GD cometric; and LoRA\-Pro’s updateΠ=PU​G\+G​PV−PU​G​PV\\Pi=P\_\{U\}G\+GP\_\{V\}\-P\_\{U\}GP\_\{V\}paired with a random tangent vectorT=X​A\+B​YT=XA\+BY\.Gauge defect52\.352\.3for plain GD \(control\) and1\.1×10−141\.1\\times 10^\{\-14\}for scaled GD, which equalsPU​G\+G​PVP\_\{U\}G\+GP\_\{V\}to8\.7×10−168\.7\\times 10^\{\-16\}\.⟨G−Π,T⟩=0\.0\\langle G\-\\Pi,T\\rangle=0\.0, soΠ\\Piis the orthogonal projection onto the tangent space\.8[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)The span of200200random VeRA updatesdiag⁡\(b\)​B​diag⁡\(d\)​A\\diag\(b\)\\,B\\,\\diag\(d\)\\,Awith frozenBBandAA,\(m,n,r\)=\(6,5,2\)\(m,n,r\)=\(6,5,2\)\.Dimension12=m​r12=mr: the span is\{X​A\}\\\{XA\\\}, the reach of re\-merged VeRA\.9[Prop II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)RoAd1atm=8m=8, one angle and one scale per output pair acting onW0W\_\{0\}by2×22\\times 2blocks: Jacobian rank\.8=m8=m, the number of parameters: an injective map with anmm\-dimensional image\.10[Prop II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)The rank of\(d​B,d​A\)↦d​B​A\+B​d​A\(dB,dA\)\\mapsto dB\\,A\+B\\,dAwithrank⁡A=rA\\rk A=r\_\{A\}andrank⁡B=rB\\rk B=r\_\{B\}, at\(m,n,r\)=\(7,6,3\)\(m,n,r\)=\(7,6,3\)\.\(rA,rB\)=\(3,0\)\(r\_\{A\},r\_\{B\}\)=\(3,0\),\(3,3\)\(3,3\),\(2,1\)\(2,1\):2121,3030,1818, equal tom​rA\+n​rB−rA​rBmr\_\{A\}\+nr\_\{B\}\-r\_\{A\}r\_\{B\}\.11[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)The product of two rotations, each a product of two reflections inℝ8\\mathbb\{R\}^\{8\}, as in two cycles of HRA2\.rank⁡\(R1​R2−I\)=4≤2\+2\\rk\(R\_\{1\}R\_\{2\}\-I\)=4\\leq 2\+2: subadditivity, attained\.Section 12: the first referee round12a[Prop II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)DoRA as in HF PEFT,diag⁡\(μi/∥Vi∥\)​V\\diag\(\\mu\_\{i\}/\\lVert V\_\{i\}\\rVert\)\\,VwithV=W0\+B​AV=W\_\{0\}\+BA: Jacobian rank\.9=m​n9=mnat\(3,3,2\)\(3,3,2\), wherer⁡\(m\+n−r\)\+m=11r\(m\+n\-r\)\+m=11;3636at\(8,8,2\)\(8,8,2\), matching the formula;2424for a rank\-oneW0W\_\{0\}at\(6,6,2\)\(6,6,2\), equal tor⁡\(m\+n−r\)\+m−rr\(m\+n\-r\)\+m\-r\.12b[Prop II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)The rank of GraLoRA’sΔ​W\\Delta Wat a random point\.8=min⁡\(k​r,m,n\)8=\\min\(kr,m,n\)at\(16,16,4,2\)\(16,16,4,2\);12=min⁡\(m,n\)12=\\min\(m,n\)at\(12,12,6,3\)\(12,12,6,3\)\.12c[Prop II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)LoHa at\(4,4,2\)\(4,4,2\): Jacobian rank\.16=m​n16=mn, where the bound gives1717\.12d[Prop VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)LoReFT ond=6d=6withr=2r=2,\(R,W,b\)↦\(I\+R⊤​\(W−R\),R⊤​b\)\(R,W,b\)\\mapsto\\big\(I\+R^\{\\top\}\(W\-R\),\\,R^\{\\top\}b\\big\)with the rows ofRRorthonormalised: Jacobian rank\.22=r⁡\(2​d\+1−r\)22=r\(2d\+1\-r\), against2626for LoRArwith a free bias\.12e[Prop VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)\(b\)The serial adapter\(I\+U​D\)​W0\(I\+UD\)W\_\{0\}at\(m,n,r\)=\(5,3,1\)\(m,n,r\)=\(5,3,1\): Jacobian rank\.77, equal to LoRA’sr⁡\(m\+n−r\)r\(m\+n\-r\), since thisW0W\_\{0\}is injective\.12f[Prop II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1),[Prop II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)FourierFT atoms: the real parts of inverse two\-dimensional DFTs of single frequencies on8×88\\times 8\.The frequencies\(1,2\)\(1,2\)and\(7,6\)\(7,6\)give the same atom; three frequencies containing one conjugate pair span22dimensions\.12g[Cor II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2),[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)HF’s default Cayley–Neumann factorR=I\+2​Q\+2​Q2\+2​Q3\+Q4R=I\+2Q\+2Q^\{2\}\+2Q^\{3\}\+Q^\{4\}for a skewQQof block size88with entries of size about0\.070\.07; then200200successive merges with entries of size about0\.050\.05\.∥R⊤​R−\(I−Q4\)2∥=4\.7×10−16\\lVert R^\{\\top\}R\-\(I\-Q^\{4\}\)^\{2\}\\rVert=4\.7\\times 10^\{\-16\}and∥R⊤​R−I∥=1\.5×10−2\\lVert R^\{\\top\}R\-I\\rVert=1\.5\\times 10^\{\-2\}\. Relative drift ofKoutK\_\{\\rm out\}:0\.0020\.002,0\.0100\.010,0\.0640\.064,0\.2240\.224after11,1010,5050,200200merges\.12h[Cor II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)BOFT’sdiag⁡\(s\)​W0​R⊤\\diag\(s\)\\,W\_\{0\}R^\{\\top\}with an exact CayleyRRands=\(−0\.5,1,…,1\)s=\(\-0\.5,1,\\dots,1\)\.Largest change of a cosine between neurons1\.501\.50; of an absolute cosine5\.6×10−165\.6\\times 10^\{\-16\}\.12i[Prop V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)DARE with a fixed mask under an invertible diagonalDDwith signs; TIES \(top20%20\\%kept, sign election by total mass\) under scalings\.DARE⁡\(D​τ\)−D​DARE​\(τ\)\\mathrm\{DARE\}\(D\\tau\)\-D\\,\\mathrm\{DARE\}\(\\tau\):1\.8×10−151\.8\\times 10^\{\-15\}\. TIES:8\.9×10−168\.9\\times 10^\{\-16\}for the factor3\.73\.7,0\.00\.0for−2\-2, and5\.65\.6for a generic positive diagonal \(control\)\.12j[Prop II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)The whitened split\[W0​L\]r​L−1\[W\_\{0\}L\]\_\{r\}L^\{\-1\}, the paper\-form CorDA split\[W0​C\]r​C−1\[W\_\{0\}C\]\_\{r\}C^\{\-1\}and PiSSA’s\[W0\]r\[W\_\{0\}\]\_\{r\}at\(m,n,r\)=\(6,5,2\)\(m,n,r\)=\(6,5,2\): the defect off⁡\(W0​h−1,h​C​h⊤\)f\(W\_\{0\}h^\{\-1\},hCh^\{\\top\}\)againstf⁡\(W0,C\)​h−1f\(W\_\{0\},C\)h^\{\-1\}for an input changehh\.Underh=3​Oh=3O,O∈O⁡\(n\)O\\in\\mathrm\{O\}\(n\):2\.2×10−152\.2\\times 10^\{\-15\},1\.8×10−151\.8\\times 10^\{\-15\},8\.1×10−168\.1\\times 10^\{\-16\}\. Under a generichh:1\.3×10−151\.3\\times 10^\{\-15\},0\.630\.63,1\.11\.1\. Whitened split with dampingC\+0\.1​IC\+0\.1I, generichh:0\.140\.14\. The whitened split equalsΠr​W0\\Pi\_\{r\}W\_\{0\}to5\.3×10−155\.3\\times 10^\{\-15\}\.12k[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)\(b\)The first Adam step withε=10−8\\varepsilon=10^\{\-8\}and gradients of size10−610^\{\-6\}, underg=diag⁡\(0\.3,1,4\)g=\\diag\(0\.3,1,4\)with per\-channel rates\.Relative defect1\.6×10−21\.6\\times 10^\{\-2\}withε\\varepsilonshared \(control\),4\.9×10−174\.9\\times 10^\{\-17\}withε\\varepsilonrescaled per channel\.12l[Thm III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)LoRA\-Pro’s parameter step at\(m,n,r\)=\(8,7,3\)\(m,n,r\)=\(8,7,3\),s=1\.5s=1\.5, with the Sylvester\-optimal free matrixXXand withX=0X=0\.Non\-orthogonalgg:1\.21\.2with the SylvesterXX,1\.9×10−151\.9\\times 10^\{\-15\}withX=0X=0\. Orthogonalgg:1\.2×10−151\.2\\times 10^\{\-15\}and1\.4×10−151\.4\\times 10^\{\-15\}\.12m[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2),[Prop II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)dimS1\\dim S\_\{1\}at\(m,n,r\)=\(7,5,2\)\(m,n,r\)=\(7,5,2\)for the three strata and for HF’s orthogonal initialisation\.TA\\mathrm\{T\}\_\{A\}:14=m​r14=mr;TB\\mathrm\{T\}\_\{B\}:10=n​r10=nr;S\\mathrm\{S\}:20=r⁡\(m\+n−r\)20=r\(m\+n\-r\)\. Orthogonal init:\|B0​A0\|≤3\.5×10−18\\lvert B\_\{0\}A\_\{0\}\\rvert\\leq 3\.5\\times 10^\{\-18\}anddimS1=11=m\+n−1\\dim S\_\{1\}=11=m\+n\-1, the rank formula atrank⁡A0=rank⁡B0=1\\rk A\_\{0\}=\\rk B\_\{0\}=1\.12n[Obs I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5),[Prop I\.8](https://arxiv.org/html/2610.12728#S1.SS0.SSSx6)\(a\)dimS1\\dim S\_\{1\}of a standard LoRA4and of LoRA2⊕\{\}\_\{2\}\\oplus\{\}LoRA2with oneA0A\_\{0\}used twice \(m=7m=7,n=6n=6\); the dimension of the fibre product ofB↦W0\+B​A0B\\mapsto W\_\{0\}\+BA\_\{0\}andA↦W0\+B0​AA\\mapsto W\_\{0\}\+B\_\{0\}Aat\(6,8,2\)\(6,8,2\)\.2828against1414; fibre product4=r24=r^\{2\}\.12o[Prop II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)Two HiRA cycles atr=1r=1on a6×56\\times 5weight with no zero entry: the rank of the relative update\(W2−W0\)⊘W0\(W\_\{2\}\-W\_\{0\}\)\\oslash W\_\{0\}\.33, above2​r=22r=2and equal tomin⁡\(2​r\+r2,m,n\)\\min\(2r\+r^\{2\},m,n\)\.12p[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)The Kronecker rank of a genericu​v⊤uv^\{\\top\}for the2×42\\times 4split ofℝ8\\mathbb\{R\}^\{8\}; the largest rank of a LoKr updateC⊗DC\\otimes DwithC∈ℝ4×4C\\in\\mathbb\{R\}^\{4\\times 4\}and a rank\-oneD∈ℝ4×4D\\in\\mathbb\{R\}^\{4\\times 4\}over200200draws\.4=min⁡\(m1,m2\)​min⁡\(n1,n2\)4=\\min\(m\_\{1\},m\_\{2\}\)\\min\(n\_\{1\},n\_\{2\}\), the largest Kronecker rank for this split;44\.12q[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3)\(c\), cor\. 4Connected components of the hypergraph of BOFT blocks of sizeb=4b=4onn=64n=64coordinates afterm′m^\{\\prime\}butterfly factors\.Component sizes44,88,1616,3232,6464form′=1,…,5m^\{\\prime\}=1,\\dots,5: connected only at full depth\.12r[Cor III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)A long gradient flow \(T=200T=200\) fromB0=0B\_\{0\}=0atηB/ηA=4\\eta\_\{B\}/\\eta\_\{A\}=4, decayλ=0\.1\\lambda=0\.1,\(m,n,r\)=\(12,10,3\)\(m,n,r\)=\(12,10,3\): the final sizes ofμ=B⊤​B−A​A⊤\\mu=B^\{\\top\}B\-AA^\{\\top\}and ofΦ\\Phi, relative toB⊤​BB^\{\\top\}Band toB⊤​B/ηBB^\{\\top\}B/\\eta\_\{B\}\.Coupled decay:μ\\mu1\.0×10−121\.0\\times 10^\{\-12\},Φ\\Phi3\.03\.0\. Per\-time decay:μ\\mu0\.750\.75,Φ\\Phi1\.4×10−121\.4\\times 10^\{\-12\}\.12s[Cor III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9)AGLr\\mathrm\{GL\}\_\{r\}\-equivariant example rule,B←B−η​GB​\(A​A⊤\)−1B\\leftarrow B\-\\eta\\,G\_\{B\}\(AA^\{\\top\}\)^\{\-1\}andA←A−η​A​A⊤​GAA\\leftarrow A\-\\eta\\,AA^\{\\top\}G\_\{A\}, for5050steps fromB0=0B\_\{0\}=0, started atA0A\_\{0\}and atg​A0gA\_\{0\}\.Difference of the final weights1\.8×10−151\.8\\times 10^\{\-15\}\.12t[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), cor\. 4The Lie algebra generated by the logarithms of1212random HF Cayley–Neumann factors withb=4b=4\.Dimension16=dim𝔤​𝔩​\(4\)16=\\dim\\mathfrak\{gl\}\(4\), againstdim𝔰​𝔬​\(4\)=6\\dim\\mathfrak\{so\}\(4\)=6\. This identifies the generated group only; re\-merging reaches a monoid \(check 13n\)\.Section 13: the second referee round13a[Obs I\.7](https://arxiv.org/html/2610.12728#S1.SS0.SSSx5)dimS1\\dim S\_\{1\}atm=7m=7,n=6n=6for LoRA2⊕\{\}\_\{2\}\\oplus\{\}LoRA2with independentA0A\_\{0\}’s, forF⁡\(1\)⊕F⁡\(2\)F\(1\)\\oplus F\(2\), and for a standardF⁡\(3\)F\(3\)\.2828;2121and2121\.13b[Thm II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)\(e\)HF’s orthogonal initialisation against aTA\\mathrm\{T\}\_\{A\}point at\(m,n,r\)=\(4,5,2\)\(m,n,r\)=\(4,5,2\)\.dimS1=8=m​r\\dim S\_\{1\}=8=mrfor both, but the two first\-order images together span1414dimensions, so they differ\.13c[Cor II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2)BOFT with everysi<0s\_\{i\}<0; GOFT with its scalerg=1±0\.05g=1\\pm 0\.05; a conformal block2​Rot​\(0\.3\)\\sqrt\{2\}\\,\\mathrm\{Rot\}\(0\.3\)on two of eight inputs; the Cayley–Neumann factor atθ=1\\theta=1withb=2b=2\.Cosine change4\.4×10−164\.4\\times 10^\{\-16\}\. GOFT: cosine change0\.0440\.044, relative singular\-value change0\.0330\.033, andKout=W0​diag⁡\(g\)2​W0⊤K\_\{\\rm out\}=W\_\{0\}\\diag\(g\)^\{2\}W\_\{0\}^\{\\top\}to6\.1×10−156\.1\\times 10^\{\-15\}\. Conformal block: cosine change0\.190\.19\.∥R∥=0\\lVert R\\rVert=0, soR=0R=0\.13d[Thm II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)The Hurwitz moveHa​Hb=HHa​b​HaH\_\{a\}H\_\{b\}=H\_\{H\_\{a\}b\}H\_\{a\}for reflections inℝ6\\mathbb\{R\}^\{6\}\.Defect4\.4×10−164\.4\\times 10^\{\-16\}\.13e[Prop II\.8](https://arxiv.org/html/2610.12728#S2.SS2.SSSx5)DoRA at\(8,8,2\)\(8,8,2\)with one magnitudeμi=0\\mu\_\{i\}=0: Jacobian rank\.3434, against3636when everyμi≠0\\mu\_\{i\}\\neq 0\.13f[Thm II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(e\)The largest rank of a LoKr updateC⊗B​AC\\otimes BAover5050draws\.\(m1,m2,n1,n2\)=\(2,8,8,2\)\(m\_\{1\},m\_\{2\},n\_\{1\},n\_\{2\}\)=\(2,8,8,2\)with inner dimension44:44\.\(4,2,2,8\)\(4,2,2,8\)with inner dimension55:44\. Predictedmin⁡\(m1,n1\)​min⁡\(r′,m2,n2\)=4\\min\(m\_\{1\},n\_\{1\}\)\\min\(r^\{\\prime\},m\_\{2\},n\_\{2\}\)=4and44\.13g[Prop II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)LoHar,ragainst LoRA2r; the orbit of LoHa’s exhibited gauge; HRA atr≥n−1r\\geq n\-1\.\(9,9,3\)\(9,9,3\): LoHa7373, LoRA67272\.\(4,4,2\)\(4,4,2\):1616and1616\. LoHa at\(4,4,2\)\(4,4,2\): orbit dimension1515, fibre dimension1616\. HRA atn=5n=5:1010forr=4r=4and forr=5r=5, equal todimSO⁡\(5\)\\dim\\mathrm\{SO\}\(5\)\.13h[Prop II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)OFT transported byT=2​IT=2I, through two merge\-and\-restart cycles of the rotation−I\-Ifromθ0=\(1,0\)\\theta\_\{0\}=\(1,0\); HiRA after two cycles atr=1,2,3r=1,2,3on a30×3030\\times 30weight\.θ2=\(9,0\)\\theta\_\{2\}=\(9,0\), of norm99, off the rotation orbit ofθ0\\theta\_\{0\}\. HiRA’s relative\-update ranks33,88,1515, equal tomin⁡\(2​r\+r2,30\)\\min\(2r\+r^\{2\},30\)\.13i[Prop II\.16](https://arxiv.org/html/2610.12728#S2.SS3.SSSx7)Bottom\-rrsplits \(CorDA’s knowledge\-preserving form onW0​CW\_\{0\}C; MiLoRA onW0W\_\{0\}\) and whitened splits with absolute damping0\.10\.1and with trace\-relative damping0\.1​tr⁡C/n0\.1\\tr C/n, at\(m,n,r\)=\(7,6,2\)\(m,n,r\)=\(7,6,2\)\.Underh=3​Oh=3O:3\.1×10−153\.1\\times 10^\{\-15\},4\.0×10−154\.0\\times 10^\{\-15\},2\.3×10−22\.3\\times 10^\{\-2\},3\.0×10−153\.0\\times 10^\{\-15\}\. Under a generichh:4\.94\.9,0\.340\.34,4\.64\.6,5\.95\.9\. Atr=nr=n, PiSSA under a generichh:3\.1×10−143\.1\\times 10^\{\-14\}\.13j[Thm III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4)Per\-channel ratesηB=\(1,1,1\)\\eta\_\{B\}=\(1,1,1\)andηA=\(1,1,2\)\\eta\_\{A\}=\(1,1,2\)at\(m,n,r\)=\(5,4,3\)\(m,n,r\)=\(5,4,3\)on the non\-quadratic loss∑i​jlog⁡cosh⁡Wi​j−⟨Y,W⟩\\sum\_\{ij\}\\log\\cosh W\_\{ij\}\-\\langle Y,W\\rangle, by Runge–Kutta withh=10−3h=10^\{\-3\}for10001000steps; one discrete GD step\.Drift of the charge of channels1,21,2\(equal productsηB,j​ηA,j\\eta\_\{B,j\}\\eta\_\{A,j\}\)4\.2×10−124\.2\\times 10^\{\-12\}; of the form of channels1,31,3,3\.13\.1\. One GD step changesΦ\\PhibyηB​A​G⊤​G​A⊤−ηA​B⊤​G​G⊤​B\\eta\_\{B\}AG^\{\\top\}GA^\{\\top\}\-\\eta\_\{A\}B^\{\\top\}GG^\{\\top\}Bto2\.4×10−152\.4\\times 10^\{\-15\}\.13k[Cor III\.6](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4.Px1)One Adam sign step:∥Δ​Φ∥\\lVert\\Delta\\Phi\\rVertdivided by the learning rate\.20\.920\.9,21\.121\.1,21\.121\.1at learning rates10−210^\{\-2\},10−310^\{\-3\},10−410^\{\-4\}: first order in the step size\.13l[Prop IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)GaLore’s field forT=1T=1atW=diag⁡\(2,1\)W=\\diag\(2,1\)withL=∥W∥2/2L=\\lVert W\\rVert^\{2\}/2: the Lie bracket ofXi​\(W\)=u1​\(W\)​ei⊤X\_\{i\}\(W\)=u\_\{1\}\(W\)e\_\{i\}^\{\\top\}by automatic differentiation; the cometric ofB↦B​B⊤B\\mapsto BB^\{\\top\}atBBand atB​RBR,R∈O⁡\(2\)R\\in\\mathrm\{O\}\(2\)\.\[X1,X2\]=−13​e2​e1⊤\[X\_\{1\},X\_\{2\}\]=\-\\tfrac\{1\}\{3\}e\_\{2\}e\_\{1\}^\{\\top\}to six digits, as predicted\. Cometric difference9\.5×10−169\.5\\times 10^\{\-16\}\.13m[Prop V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)Blockwise absmax NF4 \(blocks of6464\) on256×256256\\times 256weights withr=8r=8: the defects of QLoRA, QPiSSA and one\-step LoftQ; NF4 under addition\.GaussianWW:∥e∥=0\.945\\lVert e\\rVert=0\.945,∥e−SVD8​\(e\)∥=0\.889\\lVert e\-\\mathrm\{SVD\}\_\{8\}\(e\)\\rVert=0\.889, LoftQ withs=2s=2:0\.945=∥e∥0\.945=\\lVert e\\rVert, QPiSSA0\.8870\.887\. Low rank plus noise:∥e∥=0\.592\\lVert e\\rVert=0\.592, QPiSSA0\.2270\.227, LoftQ withs=1s=1:0\.5490\.549\.96\.7%96\.7\\%of the entries ofκ⁡\(W\)\+κ⁡\(Δ\)\\kappa\(W\)\+\\kappa\(\\Delta\)lie off the NF4 grid\.13n[Thm V\.3](https://arxiv.org/html/2610.12728#S5.SS0.SSSx3), cor\. 4The operator norm of Cayley–Neumann factors for20002000random skewQ∈ℝ4×4Q\\in\\mathbb\{R\}^\{4\\times 4\}with∥Q∥op≤21/4\\lVert Q\\rVert\_\{\\rm op\}\\leq 2^\{1/4\};200200successive merges onW∈ℝ6×8W\\in\\mathbb\{R\}^\{6\\times 8\}\.Largest∥R∥op\\lVert R\\rVert\_\{\\rm op\}:11to rounding\. Largest eigenvalue ofKout​\(t\+1\)−Kout​\(t\)K\_\{\\rm out\}\(t\+1\)\-K\_\{\\rm out\}\(t\):−5\.5×10−9\-5\.5\\times 10^\{\-9\}, soKoutK\_\{\\rm out\}decreases in the Loewner order at every merge\.13o[Prop V\.4](https://arxiv.org/html/2610.12728#S5.SS0.SSSx4)Factor\-wise DARE on two adapters under a diagonal gauge of one adapter or of both; the LoraHub change formula with three modules; the fibre of Poly withK=3K=3skills,r=2r=2and6×66\\times 6weights\.Gauge on one adapter:39\.539\.5\(control\); on both:5\.6×10−155\.6\\times 10^\{\-15\}\. LoraHub formula:4\.0×10−154\.0\\times 10^\{\-15\}\. Poly fibre:3232with22tasks,4=dimGL24=\\dim\\mathrm\{GL\}\_\{2\}with44tasks\.13p[Thm VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)\(R5\)Transposition under RoPE withdk=8d\_\{k\}=8:⟨Rm​M​q,Rn​k⟩\\langle R\_\{m\}Mq,R\_\{n\}k\\rangleagainst⟨q,M⊤​Rn−m​k⟩\\langle q,M^\{\\top\}R\_\{n\-m\}k\\rangle, and against the slide onto the key side,⟨Rm​q,Rn​M⊤​k⟩\\langle R\_\{m\}q,R\_\{n\}M^\{\\top\}k\\rangle\.2\.2×10−162\.2\\times 10^\{\-16\}; the key\-side slide differs by2\.22\.2\(control\)\.13q[Prop VI\.4](https://arxiv.org/html/2610.12728#S6.SS0.SSSx5)\(b\)The serial adapter at\(m,n,r\)=\(4,8,1\)\(m,n,r\)=\(4,8,1\): Jacobian rank\.77, againstd⁡\(LoRA1\)=11d\(\\mathrm\{LoRA\}\_\{1\}\)=11: withm<nm<nthe update rows stay inrow⁡W0\\operatorname\{row\}W\_\{0\}\.
### E\.3 Numbers from other sources

A few \[Num\] values in the main text are not printed by the script\. They come from the replications that the referees ran in the two review rounds, recorded with the review notes of each result, and from the check scripts that accompanied the four proposals from which the framework was synthesised\. They are the rank formula of[Proposition II\.3](https://arxiv.org/html/2610.12728#S2.SS1.SSSx3)in400400random cases and its loss rate, the Adam steps after[Lemma II\.4](https://arxiv.org/html/2610.12728#S2.SS1.SSSx4), the LOFT identities of §[II\.B](https://arxiv.org/html/2610.12728#S2.SS2.SSSx4), the RoAd2dimension \([Proposition II\.9](https://arxiv.org/html/2610.12728#S2.SS2.SSSx6)\), the283283random networks of[Theorem II\.10](https://arxiv.org/html/2610.12728#S2.SS3.SSSx1), VeRA’s dimensionm\+r−1m\+r\-1and the LoHa dimension at\(20,16,2\)\(20,16,2\)in the census of[Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1), the operator\-Schmidt ranks after[Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2), the transported cometric and Adam runs of[Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4), the shared\-frame and RoPE tests of[Observation II\.15](https://arxiv.org/html/2610.12728#S2.SS3.SSSx6), the finite\-difference cometrics of[Observation III\.2](https://arxiv.org/html/2610.12728#S3.SS0.SSSx2)and[Theorem III\.3](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3), LoRA\-FA under AdamW \([Proposition III\.4](https://arxiv.org/html/2610.12728#S3.SS0.SSSx3)\), the non\-gradient signals and the clipping of[Theorem III\.5](https://arxiv.org/html/2610.12728#S3.SS0.SSSx4), the Kempf–Ness minimum of[Proposition III\.7](https://arxiv.org/html/2610.12728#S3.SS0.SSSx5), the closed form of[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6), the warm\-up,ε\\varepsilon, decay and clipping runs of[Theorem III\.10](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7)and[Corollary III\.11](https://arxiv.org/html/2610.12728#S3.SS0.SSSx7), LoRA\-RITE, LoRA\-Pro’s second\-orderWW\-step defect and the zero\-BBruns of[Theorem III\.12](https://arxiv.org/html/2610.12728#S3.SS0.SSSx8)and[Corollary III\.13](https://arxiv.org/html/2610.12728#S3.SS0.SSSx9), the GaLore runs of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3), the general bracket formula of[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4), the NF4 example with a dominant low\-rank part in[Proposition V\.2](https://arxiv.org/html/2610.12728#S5.SS0.SSSx2)\(check 13m runs an analogous one at another scale\), the query\-scaling and post\-LN tests of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4), and the decoder tests of[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)\. Where a referee’s replication and the script use different random draws, the main text may quote the referee’s value; the conformal block of[Corollary II\.6](https://arxiv.org/html/2610.12728#S2.SS2.SSSx2), for instance, changes some cosine by0\.170\.17in the referee’s draw and by0\.190\.19in the script’s \(check 13c\)\.

### E\.4 Software

Table[39](https://arxiv.org/html/2610.12728#A5.T39)lists the software behind the run of Table[38](https://arxiv.org/html/2610.12728#A5.T38), the figures and this document\. Only the first four rows are needed to re\-run the checks\.

Table 39:Software and hardware for the checks, the figures and the companion site\.
### E\.5 How the companion site and the figures compute what they show

The companion site is one static page\. A build script assembles it from one HTML file per exposé, and a second script compiles its data file from the verified catalogue, the arXiv metadata, the classification batches, the axis definitions, the Rosetta stone and the numbered results\. The data file is loaded as a script, so the page runs from a web host, from disk or in a sandboxed viewer without fetching its data; it loads D3, MathJax and its fonts from public CDNs\. The tables of Appendix[D](https://arxiv.org/html/2610.12728#A4)and the ancillary files are generated from the same data\.

Each figure of the site is a small module in plain JavaScript that computes every number it shows in the reader’s browser, when it first comes into view and again at every interaction; nothing is typed in by hand\. JavaScript numbers are IEEE doubles, so the arithmetic is float64, as in the script\. A shared kernel provides dense matrix products, Kronecker and Hadamard products, a cyclic Jacobi eigensolver for symmetric matrices, singular values, numerical rank, random orthogonal matrices and a seeded generator \(mulberry32, with Box–Muller normals\), so every figure opens on the same default state\. Figures that must see exact zeros, such as the operator\-Schmidt spectra behind Figure[6](https://arxiv.org/html/2610.12728#S2.F6)and the invariant lamps of the Gram lab, use a one\-sided Jacobi SVD, which resolves singular values down to about10−1610^\{\-16\}of the norm, where an eigensolver applied to the Gram matrix stops near10−810^\{\-8\}\.

- •The orbit explorer trains a seeded toy regression twice, by a hand\-written bias\-corrected Adam or by gradient descent\. Its sliders move on a factor\-of\-two grid, so every torus element it can reach rescales doubles exactly, and atε=0\\varepsilon=0without decay or clipping its metermaxt⁡∥Wt−Wt′∥F\\max\_\{t\}\\lVert W\_\{t\}\-W^\{\\prime\}\_\{t\}\\rVert\_\{F\}reads exactly00, as check 3 does\.
- •The gauge lab computes the update of SGD, Adam, Adam with per\-channel rates, damped and undamped scaled GD and LoRA\-Pro at\(B,A\)\(B,A\)and at\(B​g−1,g​A\)\(Bg^\{\-1\},gA\)and prints the relative defect, as checks 2, 7, 12k and 12l do\. It recomputes task arithmetic and LoraHub from the factors at every gauge, as check 13o does\.
- •The Noether figure integrates scalar LoRA under gradient flow by fourth\-order Runge–Kutta withh=10−3h=10^\{\-3\}, under gradient descent by Euler steps and under Adam step by step, so that the charge is computed and never assumed\. Its matrix panel uses the implicit midpoint rule, which keeps quadratic invariants to rounding, and compares the closed\-form velocity of[Theorem III\.9](https://arxiv.org/html/2610.12728#S3.SS0.SSSx6)with the integrated one\.
- •The apex figure checks its counters by a Jacobian rank on mount, reproducing the \[Num\] lines of[Theorem II\.2](https://arxiv.org/html/2610.12728#S2.SS1.SSSx2)and[Theorem II\.7](https://arxiv.org/html/2610.12728#S2.SS2.SSSx3)\. The rank\-versus\-volume figure rebuilds each parametrisation, forms its Jacobian by central differences, which are exact here because each map is affine in every single coordinate, and takes its numerical rank by Householder QR with column pivoting; for a large layer it runs on a small copy with the same rank parameters\. The plug workbench \(§[II\.C](https://arxiv.org/html/2610.12728#S2.SS3.SSSx5)\) measures dimensions the same way\.
- •The merge game checks every rewrite of[Theorem VI\.3](https://arxiv.org/html/2610.12728#S6.SS0.SSSx4)on seeded random weights of a small pre\-LN transformer block and prints the relative output difference, about10−1610^\{\-16\}for a legal rewrite, or the residual of a refused one\. The prefix figure computes its attention weights from seeded synthetic tokens\. The rebasing figure measures the rank of accumulated ReLoRA updates by Jacobi SVD againstmin⁡\(K​r,m,n\)\\min\(Kr,m,n\)\. The lens figure runs two float64 checks on mount, of[Proposition IV\.2](https://arxiv.org/html/2610.12728#S4.SS0.SSSx3)and of the bracket of[Proposition IV\.3](https://arxiv.org/html/2610.12728#S4.SS0.SSSx4)\.
- •The budget calculator evaluates parameter counts from each base model’s config\.json and reproduces on screen the parameter count of the official checkpoint\. The catalogue, the periodic table, the timeline, the category graph and the citation panel compute every count from the data file at load, and the graph’s layered layout is a deterministic Sugiyama procedure\.

The static figures of this paper are drawn by Python scripts kept with the authors’ sources, most of them ports of the site’s modules, with NumPy and Matplotlib\. They recompute every plotted number in float64 from the site’s toys and seeds\. Figure[8](https://arxiv.org/html/2610.12728#S3.F8), for example, ports the orbit explorer and check 3, and Figure[9](https://arxiv.org/html/2610.12728#S3.F9)ports the gauge lab and checks 2, 7, 12k, 12l and 13o\.

### E\.6 The numbers behind the figures

The captions of the figures describe what each panel shows\. This subsection records the settings and the values behind them, as the figure scripts print them\. Every value is recomputed in float64 when a script runs\.

##### Figure[4](https://arxiv.org/html/2610.12728#S2.F4)

GraLoRA withk=2k=2,44and88blocks reaches ranks1616,3232and6464atd=65,472d=65\{,\}472\. HiRA8, the transportX↦W0⊙XX\\mapsto W\_\{0\}\\odot Xof LoRA8\([Proposition II\.13](https://arxiv.org/html/2610.12728#S2.SS3.SSSx4)\), reaches full rank for genericW0W\_\{0\}\. MoRA in its reshape formΔ​W=I16⊗M\\Delta W=I\_\{16\}\\otimes MwithM∈ℝ256×256M\\in\\mathbb\{R\}^\{256\\times 256\}, and C3A with256×256256\\times 256circulant blocks, are injective, sod=\|M\|d=\\lvert M\\rvert, and both reach full rank\. FourierFT withnc=65,536n\_\{c\}=65\{,\}536frequencies from HF PEFT’s default draw \(seed777777\) has120120conjugate pairs, sod=65,416d=65\{,\}416, and its maximum rank is known only through the boundmin⁡\(2​nc,m,n\)\\min\(2n\_\{c\},m,n\)\. No KronA factor shapes have budget exactly65,53665\{,\}536\. The closest,256×256⊗16×16256\\times 256\\otimes 16\\times 16, cost65,79265\{,\}792and gived=65,791d=65\{,\}791and rankmin⁡\(m1,n1\)​min⁡\(m2,n2\)=4,096\\min\(m\_\{1\},n\_\{1\}\)\\min\(m\_\{2\},n\_\{2\}\)=4\{,\}096\([Theorem II\.11](https://arxiv.org/html/2610.12728#S2.SS3.SSSx2)\(b\)\)\. LoHa4,4is placed at the proved upper boundF=57,313F=57\{,\}313; it reaches rank1616, as GraLoRA withk=2k=2does, with at least\(m\+n−1\)−2​r2=8,159\(m\+n\-1\)\-2r^\{2\}=8\{,\}159fewer dimensions than LoRA8\([Proposition II\.12](https://arxiv.org/html/2610.12728#S2.SS3.SSSx3.Px1)\(c\)\)\. Every dimension and maximum rank shown is confirmed by float64 Jacobian ranks and random\-point ranks on a64×6464\\times 64copy\.

##### Figure[6](https://arxiv.org/html/2610.12728#S2.F6)

The vectorsu,vu,vand the factorsC,DC,Dare seeded Gaussians, and the rank threshold is10−1010^\{\-10\}\. The numerically zero values are at most1\.3×10−161\.3\\times 10^\{\-16\}; values below10−1710^\{\-17\}, exact zeros included, are drawn at10−1710^\{\-17\}, and every nonzero value is at least9\.5×10−49\.5\\times 10^\{\-4\}\. Each reshape permutes the entries, so the squared normalised singular values sum to11on every cut\. A bond of dimensionkkcostsk⁡\(4⋅4\+4⋅4\)=32​kk\(4\\cdot 4\+4\\cdot 4\)=32kparameters across any of the three cuts\.

##### Figure[7](https://arxiv.org/html/2610.12728#S3.F7)

Panel \(a\) integrates gradient flow by fourth\-order Runge–Kutta with step10−310^\{\-3\}\. The charge is integrated, not imposed, and stays atΦ0\\Phi\_\{0\}to within6×10−116\\times 10^\{\-11\}on every run\. From the zero\-BBstart\(1\.2,0\)\(1\.2,0\),Φ0=−1\.44\\Phi\_\{0\}=\-1\.44, and Adam withη=0\.1\\eta=0\.1ends atΦ=−0\.53\\Phi=\-0\.53\. A GD step multiplies the charge by exactly1−η2​ηA​ηB​s2​\(wk−w⋆\)21\-\\eta^\{2\}\\eta\_\{A\}\\eta\_\{B\}\\,s^\{2\}\(w\_\{k\}\-w^\{\\star\}\)^\{2\}; the GD drift is1\.2×10−31\.2\\times 10^\{\-3\}atη=0\.0025\\eta=0\.0025and0\.140\.14atη=0\.2\\eta=0\.2, and5\.8×10−135\.8\\times 10^\{\-13\}for the flow\. The Adam drift stays between0\.550\.55and1\.51\.5\. In panel \(b\) the seeded target update has singular values2\.02\.0and0\.30\.3, withs=1s=1andηA=ηB=1\\eta\_\{A\}=\\eta\_\{B\}=1, soμ=s​ηA​ηB=1\\mu=s\\sqrt\{\\eta\_\{A\}\\eta\_\{B\}\}=1\. The flow is integrated by the implicit midpoint rule\. The hollow circles areui⊤​\(B​B⊤/ηB\)​uiu\_\{i\}^\{\\top\}\(BB^\{\\top\}/\\eta\_\{B\}\)u\_\{i\}andvi⊤​\(A⊤​A/ηA\)​viv\_\{i\}^\{\\top\}\(A^\{\\top\}A/\\eta\_\{A\}\)v\_\{i\}; the charge stays at−κ​I\-\\kappa Ito2×10−142\\times 10^\{\-14\}, and the closed\-formW˙\\dot\{W\}matchess⁡\(B˙​A\+B​A˙\)s\(\\dot\{B\}A\+B\\dot\{A\}\)to a relative1\.3×10−131\.3\\times 10^\{\-13\}\. The one\-sided regime is where one gain exceeds ten times the other,σ<0\.70​σ∗\\sigma<0\.70\\,\\sigma^\{\*\}, and the balanced regime where the two gains are within a factor two,σ\>2\.83​σ∗\\sigma\>2\.83\\,\\sigma^\{\*\}\.σ1\\sigma\_\{1\}crossesσ∗\\sigma^\{\*\}att≈0\.49t\\approx 0\.49and settles at2\.02\.0, where theAA\-update supplies38%38\\%of its growth rate;σ2\\sigma\_\{2\}settles at0\.30\.3, where that share is7%7\\%\.

##### Figure[8](https://arxiv.org/html/2610.12728#S3.F8)

The toy hasW0∈ℝ8×6W\_\{0\}\\in\\mathbb\{R\}^\{8\\times 6\}, a rank\-two target update andr=2r=2\. Both runs take 300 steps of a hand\-written, bias\-corrected Adam in float64 and reach a loss of2\.7×10−72\.7\\times 10^\{\-7\}; the control ends at5\.7×10−25\.7\\times 10^\{\-2\}\. Atε=0\\varepsilon=0the runs are bit\-identical at all 301 steps \(check 3 of the script\)\. The red curves use Adam’sε=10−8\\varepsilon=10^\{\-8\}or10−610^\{\-6\}\(the gap is proportional toε\\varepsilon\), PyTorch\-style decoupled weight decay0\.010\.01, and global\-norm clipping at11, which clips 8 steps of the LoRA\+ run and 37 of the LoRA run\. Their maximal gaps are4×10−84\\times 10^\{\-8\},4×10−64\\times 10^\{\-6\},7×10−37\\times 10^\{\-3\}and0\.430\.43, against2\.02\.0for the control\. The per\-block rescaling setsεB=16​ε\\varepsilon\_\{B\}=16\\,\\varepsilon, decay16×0\.0116\\times 0\.01onBB, andBB’s clipping threshold multiplied by1616\.

##### Figure[9](https://arxiv.org/html/2610.12728#S3.F9)

The state isB∈ℝ5×2B\\in\\mathbb\{R\}^\{5\\times 2\},A∈ℝ2×4A\\in\\mathbb\{R\}^\{2\\times 4\}\. The rotation is by30∘30^\{\\circ\}, the stretch isdiag⁡\(e0\.5,e−0\.4\)\\diag\(e^\{0\.5\},e^\{\-0\.4\}\), and the signed permutation swaps the two rank channels and flips one\. Adam is its first step from zero state, which atε=0\\varepsilon=0is sign descent; the per\-channel rescaling forg=diag⁡\(d\)g=\\diag\(d\)isηB,j↦ηB,j/dj\\eta\_\{B,j\}\\mapsto\\eta\_\{B,j\}/d\_\{j\},ηA,j↦dj​ηA,j\\eta\_\{A,j\}\\mapsto d\_\{j\}\\,\\eta\_\{A,j\}, and the damped scaled GD hasδ=0\.1\\delta=0\.1\. The first\-order weight update isPU​G\+G​PVP\_\{U\}G\+GP\_\{V\}for undamped scaled GD andPU​G\+G​PV−PU​G​PVP\_\{U\}G\+GP\_\{V\}\-P\_\{U\}GP\_\{V\}for LoRA\-Pro\. In panel \(b\) the weights arew=\(0\.6,0\.5,0\.4\)w=\(0\.6,0\.5,0\.4\)\. Task arithmetic changes by less than10−1610^\{\-16\}relative to its norm, the LoraHub merge by up to3\.43\.4times its norm atc=1/8c=1/8, and the closed form∑l≠2wl​w2​\[\(c−1\)​B2​Al\+\(c−1−1\)​Bl​A2\]\\sum\_\{l\\neq 2\}w\_\{l\}w\_\{2\}\\big\[\(c\-1\)B\_\{2\}A\_\{l\}\+\(c^\{\-1\}\-1\)B\_\{l\}A\_\{2\}\\big\]matches the recomputed change to10−1510^\{\-15\}\.

##### Figure[10](https://arxiv.org/html/2610.12728#S5.F10)

In panel \(a\), ReLoRA reaches the full rank16=min⁡\(m,n\)16=\\min\(m,n\)after66cycles forr=3r=3and after44forr=5r=5\. LoRA\-XS adds updatesUr​Rk​Vr⊤U\_\{r\}R\_\{k\}V\_\{r\}^\{\\top\}inside oner2r^\{2\}\-dimensional subspace\. In panel \(b\), BOFT’s levelℓ\\ellpairs coordinateccwithc⊕2ℓc\\oplus 2^\{\\ell\}\. Each light bar equals both the dimension of the Lie closure of the blocks’ plane rotations under brackets and the measured dimension of the reachable set after1616merge cycles of exact\-Cayley factors \(the rank of the right\-trivialised Jacobian of the product map\)\. Block OFT reaches2424and5656in one cycle and never more\. BOFT goes from1616to2424and from2424to5656at depthsm′=2,3m^\{\\prime\}=2,3, and only the full depthm′=1\+log2⁡\(16/b\)=4m^\{\\prime\}=1\+\\log\_\{2\}\(16/b\)=4connects the hypergraph: a single full\-depth BOFT has dimension3232, and merging reaches120120after44cycles\. The77Givens planes leave components of sizes4,3,2,24,3,2,2and five isolated coordinates, so they generate only1111dimensions; the1515\-plane tree reaches120120after99cycles\.

##### Figure[12](https://arxiv.org/html/2610.12728#S6.F12)

The query hasq∈ℝ4q\\in\\mathbb\{R\}^\{4\}and∥q∥=2\.4\\lVert q\\rVert=2\.4; the logits are⟨q,k⟩/2\\langle q,k\\rangle/2, and the prefix keys are scaled by1\.21\.2\. In panel \(a\),λq=0\.44\\lambda\_\{q\}=0\.44, so every content weight is multiplied by0\.560\.56, and the six ratios agree to within10−1510^\{\-15\}\. In panel \(b\),g=0\.5g=0\.5and the ratios are exactly11\. In panel \(c\), the two principal axes keep88%88\\%of the variance of the values, and the LLaMA\-Adapter output lies at distance0\.650\.65from the segment inℝ4\\mathbb\{R\}^\{4\}\. The script evaluates the joint softmax directly and recovers the identity of part \(b\) of[Theorem VI\.5](https://arxiv.org/html/2610.12728#S6.SS0.SSSx6)to4×10−164\\times 10^\{\-16\}\.

## Appendix FHow this paper and its website were made

This appendix records how the paper and its companion site were produced: who did what, how the material was found, how the results were generalised, unified and checked, and what it cost\.

### F\.1 In brief

- •1 October 2026, 17:06 AEST, to 4 October 2026, about 01:00 AEST, when the first full version of this appendix was written: about 56 hours of wall\-clock time\. At least one agent was running for about 23 of those hours\. The rest was a 43\-minute wait for an account usage reset, two interruptions, and long pauses between requests\.
- •The authors directed one orchestrating session of Claude \(Anthropic’s Claude Opus 5\.5\) running in Claude Code, in short messages and a few answers to multiple\-choice questions\. That session wrote and launched 20 multi\-agent workflows, which did most of the research, writing and checking, and two small ones that checked and redrafted this appendix\. It did the planning, integration and publishing itself\.
- •Those workflows started 422 agent runs \(420 distinct agent calls, two of which were started twice\)\. 358 completed, 29 failed when the account reached a spend limit, and 35 were stopped or cut off\.
- •The agents processed about 117 million fresh tokens: prompts written to the cache plus generated text, of which 8\.6 million were generated\. They also re\-read 4\.1 billion tokens from the prompt cache\. The orchestrating session added about 10 million fresh tokens and 0\.2 billion cache reads\.
- •The outputs are a website \(15 sections including a version of this appendix, 20 interactive figures plus an animated hero, and a checked catalogue of 382 methods and 138 papers\) and a paper of about 400 pages with the same numbering\.
- •On 5 October a later session made a narrated film of the atlas, 16 minutes 44 seconds long, and a two\-minute teaser\. Its five agents processed about 5\.0 million fresh tokens, and the session itself about 1\.4 million\.

### F\.2 Who did what

The authors think that when the literature on parameter\-efficient fine\-tuning \(PEFT\) already holds several hundred methods, the field gains more from the approach Grothendieck pictured as a rising sea than from one more method\. This atlas catalogues 382 of them\.

In his memoir*Récoltes et Semailles*, written 1983–1986, Grothendieck compared a hard problem to a nut\. One can strike the shell with hammer and chisel, at point after point, until it cracks\. Or one can soak the nut in a softening liquid and let time pass, until the pressure of a hand is enough\. On a larger scale the second way is the rising sea\. It advances so slowly and quietly that nothing seems to change, until it surrounds the problem and the problem is dissolved by a theory that reaches well beyond it\([Grothendieck, 2022](https://arxiv.org/html/2610.12728#bib.bib138);[McLarty, 2007](https://arxiv.org/html/2610.12728#bib.bib322)\)\. In the authors’ analogy, a 383rd method would be one more strike of the chisel, and the sea is a theory in which the existing methods appear as instances of one notion\.

The authors have packaged this way of working as a Claude Code skill\. It asks for the natural home of a result, for objects defined by their relationships and for the relative point of view, so that hard results dissolve\. Its binding rule is that every reframe must pay, by a shorter proof, a dropped assumption or a real unity, or be rejected\.

This view shaped the question the authors chose, to re\-found PEFT “as Grothendieck would” and to make the result an interactive site\. They also made the main decisions of scope and presentation\. Their instructions to the session were short messages: pointing out some missing methods in the literature, asking for a sweep of 2026 work, a mathematical appendix, more readable typefaces, a LaTeX paper and this appendix, supplying the author list and the title, and asking the session to resume after interruptions\.

The orchestrating session did the following:

- •planned the work and wrote the seed sketch of the theory;
- •designed the visual identity, the data pipeline, the build and test tools and the LaTeX template;
- •launched and resumed the workflows, read their reports and decided what to run next;
- •applied some changes itself, such as the new typography rules, and rebuilt, tested and published the site, and built the paper\.

The sub\-agents researched the literature, proposed, judged and refereed the theory, classified the methods, wrote the sections, drew and tested the figures, and reviewed each other’s work\. They also made the remaining decisions, including which 2026 candidates entered the catalogue and what was cut from the site in its final review\.

A workflow is a script for Claude Code’s Workflow tool\. It starts sub\-agents in parallel or in sequence, hands each a prompt and files, and keeps a journal of their results\. Every sub\-agent ran the same model, claude\-opus\-5\-5, whose training data ends in June 2026\. All had web search and fetch and a shell on one Apple M5 laptop, with Python 3\.9 \(NumPy, PyTorch, SciPy, Matplotlib\), Node\.js, headless Chrome and TeX Live 2026\. Below, an independent check means one made by a separate agent with its own context and instructions\.

### F\.3 Finding the material

The literature work was split into thirteen families:

- •the Hugging Face PEFT library;
- •LoRA and its training\-dynamics variants;
- •initialisation and rank allocation;
- •sharing and extreme efficiency;
- •structured shapes;
- •multiplicative and orthogonal methods;
- •adapters;
- •prompts and states;
- •selective fine\-tuning;
- •quantisation and memory;
- •composition and merging;
- •theory and foundations;
- •the 2025–2026 frontier\.

One agent swept each family\. It worked from the arXiv API, Semantic Scholar, OpenReview, the ACL Anthology, the official conference listings and the Hugging Face PEFT source code\. Each sweeper was told to read a paper’s method section before writing down its formula\. As each family finished, an independent fact\-checker re\-verified every entry:

- •title, first author and first\-version year against the arXiv API;
- •the venue against the proceedings;
- •the formula against the method section;
- •the parameter count, re\-derived independently;
- •mergeability and initialisation;
- •Hugging Face PEFT support, against the repository at commit 85be3f3 \(30 September 2026\)\.

A completeness critic then compared the whole catalogue with the library and with its own knowledge of the field\. It listed 34 missing methods, and 32 of them were researched and checked in four more batches\. The checkers, including those of the four gap\-filling batches, made 613 corrections before the first release\. Merging duplicates on a canonical identifier and an alias table gave 320 methods and 124 papers\. Matching by name would have folded LoRA\+ into LoRA\.

After the first build, the authors pointed out methods missing from the catalogue\. They were added and refereed, and a dedicated sweep of 2026 work followed, in which six scouts covered:

- •arXiv submissions from January to September 2026, one scout per quarter, by keyword queries over titles and abstracts;
- •the accepted papers of ICLR 2026 and ICML 2026, with ACL and CVPR 2026;
- •the citation graph of ten anchor PEFT papers and the 2026 Hugging Face PEFT releases\.

A triage agent then chose which candidates entered the atlas\. It admitted new 2026 methods with at least one signal \(a 2026 venue, inclusion in Hugging Face PEFT, citations, a widely used codebase, or a structure the seven coordinates would treat differently\), at most 36 per pass\. The first triage saw only the first 105 of the 265 candidates not already in the atlas, the last of them cut off mid\-entry, because the orchestrator had truncated the list it passed in\. A second pass triaged the remaining 161 candidates and the scouts’ overflow lists\. Together the passes added 61 methods and 14 theory papers\. They were researched and fact\-checked in the same way, giving 382 methods, 138 papers and 905 corrections in total\.

The BibTeX of the 502 catalogue entries is built from arXiv metadata for the 493 that have an arXiv identifier, and the other nine are built from the catalogue’s own fields\. Eight of those nine were corrected by hand for the paper\. The paper’s 130 other references, mostly mathematics and related work, were written by the agents who cited them, who were told to check every field against the publisher, DOI or arXiv page\.

### F\.4 From examples to a theory

The orchestrator wrote a seed sketch of eleven numbered points, about 1,000 words, which the architects could adopt, correct or reject\. Its points were:

- •methods as reparametrisations in Para and as objects of a pointed slice;
- •frozen and full fine\-tuning as initial and terminal;
- •mergeability as a lifting problem;
- •Klein’s Erlangen programme for group\-type methods;
- •theGLr\\mathrm\{GL\}\_\{r\}gauge of LoRA and its conserved charge;
- •string diagrams for structured updates;
- •a monoidal sum;
- •base change;
- •the lens duality between forward and backward methods;
- •prompts and prefixes as learned input states;
- •naturality, so that a method defined once per layer type extends to every network\.

Four architect agents each developed a complete framework from this sketch, with definitions, propositions and proofs, from a different point of view:

- •parametric lenses;
- •Grothendieck’s relative point of view;
- •the Erlangen programme with gauge symmetry;
- •string diagrams\.

Three judge agents scored all four proposals, each from an assigned point of view:

- •a category theorist’s, for rigour;
- •a PEFT researcher’s, for faithfulness to the methods;
- •a science communicator’s, for clarity and visual potential\.

All three recommended the relative point of view as the backbone\. One synthesis agent then built the framework on it\. As the judges advised, it took the lens and cometric from the parametric\-lens proposal, the covariance and optimizer gauge from the Erlangen proposal and the diagrams from the string\-diagram proposal, and it cut the constructions they judged decorative\. The standing rule was that a categorical construction must explain, classify or predict something about PEFT, or be cut\. The result was 41 numbered results, a seven\-axis classification and a dictionary between the two vocabularies\.

Generalisation usually followed one pattern:

1. 1\.Take an observation about a single method\. Zero\-initialised LoRA cannot move in some directions at the first step\.
2. 2\.Find the invariant it depends on\. Here it is the first\-order image compared with the dimension of the image germ\.
3. 3\.State the result for every method with that coordinate value\. Any method whose base point is the vertex of a cone has the same deficit\.
4. 4\.Look for consequences the source papers do not state\. Householder reflection adaptation at its default start is also an apex\.
5. 5\.Test the claim numerically and send it to referees\.

The same pattern turned “LoRA\+ helps” into an exact statement about Adam’s invariance to per\-block rescaling\. It turned “orthogonal methods preserve structure” into complete invariants of each group action\. It turned “this adapter can be merged” into a local rewriting test\. The test certifies exact merging under stated side conditions, names the weight that absorbs the update, and depends on the activation, the normalisation and the bias slots\.

The seven coordinates are decidable from a method’s formula\. That made it possible to classify the first 320 methods in ten batches of 32, and the 62 later additions in three more\. In each batch a second agent re\-derived every coordinate, checked every arrow’s factorisation and base point by hand, and spot\-checked a few gauge dimensions and first\-order deficits numerically\. A consolidator then reconciled the first ten batches with each other and with the theory, and the final site review made the prose agree with the data\. Every arrow between methods records its explicit map h withρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}\(403 of them\)\. Every recorded non\-arrow names the test that rules it out \(485\)\.

### F\.5 How category theory unifies the field here

The unifying move is to study each method through its map into weight space\. A weight\-space method is a pointed map from its small parameter space into weight space that sends its starting point to the pretrained weights\. Comparing methods then means comparing maps:

- •a method N simulates a method M when M’s map factors through N’s, that isρN∘h=ρM\\rho\_\{N\}\\circ h=\\rho\_\{M\}for a pointed map h;
- •doing nothing and full fine\-tuning are the initial and terminal objects, so they bracket every weight\-space method;
- •three invariants of the map carry most of the content: its image germ, its fibres with the optimizer’s metric, and its behaviour under base change\.

Extensions such as adapters and prefixes, which change the network itself, fit through a realisation map\. Mergeability becomes the question of whether a lift exists\.

Exposé XI lists what is analogy and what is theorem\. The words Noether, Erlangen, gauge and Grothendieck are used in a precise, limited sense, and the paper says which\.

### F\.6 How the results were checked

Verification ran in layers:

- •Sources\.Every catalogue entry was checked by an agent other than the one that wrote it, against arXiv metadata, the paper and the library source\.
- •Adversarial model referees, three rounds\.Each of the 41 results went to two referee agents with different lenses: mathematical correctness, and faithfulness to the papers and code\. They were told to default to “refuted” when in doubt\. - –Round 1 produced 82 verdicts, 45 of them major objections \(41 refutations\)\. 27 results were repaired and none was false outright\. - –Round 2 re\-examined 26 of the 27 repaired results in 48 reports\. Six of its 54 runs were stopped by the spend limit, so Corollary III\.13 received no second\-round report and four results received one; in the final site review the section editors took the missing referees’ place\. The reports raised 28 major objections against 18 results, and 19 results were narrowed again\. - –Each round\-2 report ended with the strongest statement its referee would sign\. These became one public statement per result in propositions\.json\. The writers and referees of the site and the paper were told to state nothing stronger, and checked for it\. - –A third round, four mathematical referees during the writing of the paper, made 63 further fixes\. - –Those fixes were then carried back into the site, the data and the theory files, 108 edits in all\.
- •Numerical checks\.framework\_checks\.py, one of the paper’s data files, re\-runs most claims marked \[Num\] in about a minute\. The paper’s Appendix E lists the few that rest only on the referees’ own replications\. Each interactive figure computes its numbers in the browser\. The engineers of the theory\-driven figures checked their key numbers against framework\_checks\.py or a separate NumPy computation, and in the final site review a second agent spot\-checked the key numbers of every figure\.
- •Engineering checks\.Every figure was rendered headlessly in light and dark themes at desktop and phone widths, and its builder looked at the images\. A whole\-site test drives Chrome to check console errors, unmounted figures, overflow, MathJax errors and broken in\-page links\. The paper is compiled from a clean copy of its sources with pdfLaTeX alone and checked for errors and undefined references\.
- •This appendix\.Two agents checked every number and date in it against the workflow journals, the agents’ transcripts and the project files, and its text was corrected where they disagreed\. Its opening was then redrafted by three agents writing independently, merged by a fourth and checked by two more, one for facts and one for style\.

These checks do not amount to human peer review\. Every proposer, writer and referee was the same model, so their errors can be correlated, and the referees of the paper also edited the files they reviewed\. A numerical check is not a proof, and every check runs on small float64 examples\. No fine\-tuning run on a real model was made, so the eleven predictions of Exposé X are untested\. As Exposé XI records, the dimension equalities for LoHa and, in general, for VeRA are checked numerically; one clause of corollary 4 of Theorem V\.3 is labelled a proof sketch, although Appendix C gives an argument; and Conjectures A, C and D are open\. The catalogue is a selection and can still miss methods\. Its 2026 additions came from keyword queries on arXiv, four conference lists and one citation graph, filtered by a triage agent, and the completeness critic’s own knowledge of the field ends with the model’s training data\. In the catalogue, 51 method entries and 7 paper entries carry a medium\-confidence flag\. Some methods were missing until the authors noticed them\. The authors are responsible for what the site and the paper say\.

### F\.7 Building the site and the paper

The site is a static page\. Its prose lives in fifteen HTML sections, counting a version of this appendix, and its data is generated from the verified catalogue and the theory files\. Each of its 21 figure scripts was written by an engineer agent \(four were finished by a second agent after the spend limit cut off the first\), self\-tested and later reviewed by another\. The page runs from any static host, straight from disk and inside a sandboxed artifact\. Its data is a script, so nothing is fetched at run time apart from D3, MathJax and the fonts, which come from public CDNs\.

The typography was changed once, at the authors’ request\. The session set one passage of the site in four typeface schemes, and the authors chose the one it recommended, with Source Serif 4 for text, IBM Plex Sans for labels and IBM Plex Mono for numbers\. Seven engineer agents then raised the text inside all 21 figure modules to at least 12 pixels where a reader must read it, and 11 pixels for tick labels\. Later the authors found the display face of the title unattractive, and the title was reset in Source Serif 4 as well\.

The paper uses a template designed for it: SGA\-style exposés with Roman numerals, so that every result has the same number on the site and in the paper\. Twenty\-one writer agents converted the full 35,000\-word theory into LaTeX, with complete proofs in Appendix C\. Six figure engineers redrew the key figures as vector PDFs in the same typefaces\. An assembler, four mathematical referees, a typesetting referee and a finalizer took it to a clean build of 393 pages, about 400 with this appendix\.

### F\.8 The film

On the evening of 5 October, at the authors’ request, a later session of Claude Code made a narrated film of the atlas, 16 minutes 44 seconds long, and then a two\-minute teaser for X\. The authors chose the narrating voice, Sulafat from Gemini 3\.8 Flash TTS, and a quiet visual style from a short list of options\.

The session wrote the script and checked each claim against the propositions and the paper, and a separate agent then fact\-checked it adversarially\. Every number on screen comes from a short computation in video/sim/\. The opening, for example, trains LoRA\+ and plain LoRA with a scaled alpha side by side, and the largest difference between their weights stays exactly zero\. The session wrote a shared Manim library, a brief and one chapter, and four agents built the other seven chapters from the brief\. The session then synthesised the narration beat by beat, timed each animation cue to the pauses detected in the audio, reviewed every chapter on contact sheets of its frames, and assembled the film with captions and chapter marks\. The film plays on the companion site, and the title page of the paper links to it\.

### F\.9 Cost and interruptions

Table[40](https://arxiv.org/html/2610.12728#A6.T40)breaks the agent runs and tokens down by phase\.

Table 40:Agent runs and tokens by phase\. Fresh tokens are prompts written to the cache plus generated text; re\-reads from the prompt cache are not counted\.The table counts every started agent run, one transcript each: the 422 runs of the workflows and the film’s five agents, a script fact\-checker and four chapter builders\. Cache re\-reads are not counted \(4\.1 billion tokens for the workflows, 0\.16 billion for the film’s agents\), and neither are the sessions that directed the work: the orchestrating session of the workflows, and the film’s session, which added 1\.4 million fresh tokens, 0\.55 million of them generated\. Also not counted are the two later workflows that checked and redrafted this appendix \(nine agents\), and the film’s narration: about 15 minutes of speech from Gemini 3\.8 Flash TTS, plus voice tests and a few retakes, which Google bills separately\.

The work was interrupted three times:

- •at 20:52 on 1 October an account spend limit made 29 agent runs fail, 12 of them mid\-run and 17 within seconds of starting;
- •at about 20:40 on 2 October the host machine crashed while the paper was being written;
- •at about 23:10 on 2 October the session was cut off again, and it resumed at 02:04 the next morning\.

Each time, the orchestrator inspected the workflow journals and launched fresh workflows containing only the unfinished items, telling each agent to read and complete its predecessor’s partial file\. The six stopped second\-round referee runs were not repeated; the final site review covered them instead\. A first attempt to resume in place re\-ran agents that had already finished\. The workflow tool replays only the longest unchanged prefix of agent calls, so that attempt was stopped within two minutes\. The other process lessons were:

- •pass large inputs to agents through files, never as truncated prompt text;
- •refereeing a repaired result again is worth it;
- •corrections found while writing the paper must flow back to the site\.

### F\.10 Reproducibility

The catalogue and the checks accompany the paper as data files:

- •catalogue\.csv and catalogue\.json, with arrows\.csv and obstructions\.csv;
- •propositions\.json, which holds each numbered result with its statement, proof, public statement and the notes of both review rounds;
- •framework\_checks\.py, the numerical checks\.

The LaTeX source carries the text and the generated tables, and the site’s figure code can be read in the page\. The public repository,[https://github\.com/xiaoyulics/peft\-atlas](https://github.com/xiaoyulics/peft-atlas), holds the rest of the record:

- •the raw and verified literature files and the classification batches, in research/;
- •the full theory text and the round\-2 reports, in theory/;
- •the build and test scripts, in site/build, site/test and paper/tools, and the scripts of the paper’s figures, in paper/figures/src;
- •the film’s script, scenes and computations, in video/\.

The 20 workflow scripts, which contain every agent prompt, and the agents’ transcripts stay with the authors\.