Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

# Between Gradient and Natural Gradient: A Continuum of LoRA Initializations
Source: [https://arxiv.org/html/2607.26247](https://arxiv.org/html/2607.26247)
###### Abstract

Low\-rank adaptation \(LoRA\) fine\-tunes large pretrained models at a fraction of the cost of full fine\-tuning, but its performance depends strongly on how the adapters are initialized\. Recent schemes initialize the adapters from the downstream loss gradient: some project the raw gradient onto its top directions, while others first whiten it with an estimate of the loss curvature\. We show that these seemingly distinct methods are points on a single continuum: a two\-parameter family of preconditioned gradient initializations, which we call Unified LoRA \(ULoRA\), governed by a spectral whitening exponent and an Adam\-like diagonal exponent\. Sweeping this family under a full learning\-rate search, we find that no single fixed preconditioning strength dominates: the best operating point is task\-dependent and frequently lies strictly inside the family, away from the published endpoints\. Treated as an upper bound of this family, a tuned ULoRA configuration matches or exceeds full fine\-tuning on all five GLUE tasks with RoBERTa\-base and is competitive with the strongest baselines on GSM8K with LLaMA 2\-7B\. Our deployable, search\-free variant, ULoRA\-Auto, selects per\-layer exponents from measured spectral statistics, approaches this upper bound at no additional search cost, and ranks at or near the top among deployable LoRA methods\. Our results show that a principled design space for LoRA initialization and curvature preconditioning should be treated as a tunable dimension rather than a fixed design decision\.

## Introduction

Adapting large pre\-trained models to downstream tasks has become standard practice\. Full fine\-tuning achieves strong performance but updates all parameters, which is expensive and often unnecessary\. Low\-rank adaptation \(LoRA\)\(Huet al\.[2022](https://arxiv.org/html/2607.26247#bib.bib1)\)addresses this by restricting weight updates to a low\-rank formΔ​W=1r​B​A\\Delta W=\\frac\{1\}\{\\sqrt\{r\}\}BA, where only two small matricesA∈ℝr×dinA\\in\\mathbb\{R\}^\{r\\times d\_\{\\text\{in\}\}\}andB∈ℝdout×rB\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times r\}are trained\. This reduces trainable parameters by orders of magnitude while keeping the pretrained backbone frozen\.

Despite their efficiency, LoRA methods could underperform full fine\-tuning, particularly on small datasets and complex tasks\. A key factor is the initialization ofAAandBB: standard LoRA usesB=0B=0and randomAA, which provides no information about the target task\. Recent work has shown that choosing the initial subspace from task data substantially narrows the performance gap\. LoRA\-GA\(Wang and others[2024](https://arxiv.org/html/2607.26247#bib.bib7)\)and LoRA\-One\(Zhanget al\.[2025](https://arxiv.org/html/2607.26247#bib.bib20)\)align the adapters with the top directions of the loss gradient\. CG\-LoRA\(Zheng and Proutière[2026](https://arxiv.org/html/2607.26247#bib.bib21)\)further whitens the gradient using Kronecker\-factored curvature \(K\-FAC\)\.

#### Gap\.

Despite this progress, existing gradient\-based methods make a rigid choice about how the gradient is used for initialization\. Gradient\-only methods \(LoRA\-GA, LoRA\-One\) ignore curvature entirely, which overlooks second\-order information about the loss landscape\. Curvature\-whitened methods \(CG\-LoRA\) apply a fixed, theoretically motivated whitening exponent of−1/2\-1/2, which can be too aggressive when curvature estimates are noisy, as is common in practice with small batches and nonlinear models\. The question of whether a different level of curvature preconditioning would be better has not been studied\.

#### Our approach\.

We propose Unified LoRA \(ULoRA; Figure[1](https://arxiv.org/html/2607.26247#Sx3.F1)\), which shows that all existing gradient\-based initialization methods are special cases of a common framework parametrized by two scalars:

- •Spectral whitening exponent \(α\\alpha\), which controls how strongly K\-FAC eigenvalues rescale gradient directions\. The valueα=0\\alpha=0removes curvature scaling;α=1\\alpha=1recovers CG\-LoRA\.
- •Diagonal preconditioning exponent \(β\\beta\), which applies a normalization by the empirical variance of layer inputs and output gradients, analogous to the second\-moment scaling in Adam\(Kingma and Ba[2014](https://arxiv.org/html/2607.26247#bib.bib22)\)\.

This two\-parameter family reproduces gradient projection \(α=0,β=0\\alpha=0,\\beta=0\), Adam\-like preconditioning \(α=0,β\>0\\alpha=0,\\beta\>0\), full curvature whitening \(α=1,β=0\\alpha=1,\\beta=0\), and mixed strategies that combine both\. Following a systematic grid search over\(α,β\)\(\\alpha,\\beta\)we find that rather than either of the two major families dominating, intermediate configurations frequently achieve superior performance across all evaluated tasks\.

#### Contributions\.

Our contributions are as follows:

1. 1\.We propose ULoRA, a two\-parameter family of LoRA initialization strategies that unifies existing gradient\-based and curvature\-based methods under a single framework\.
2. 2\.We provide a characterization ofα\\alphaandβ\\betaand show how they recover known methods as special cases, establishing a principled design space for LoRA initialization\.
3. 3\.We demonstrate that the optimal level of curvature preconditioning is neither zero nor the full value, and that intermediate values are consistently more robust across various language benchmarks\.

Treated as an oracle upper bound of the family, a tuned ULoRA configuration matches or exceeds full fine\-tuning on all five GLUE tasks with RoBERTa\-base, while the deployable, search\-free ULoRA\-Auto recovers most of this gain and ranks at or near the top among deployable LoRA methods on GSM8K with LLaMA 2\-7B and across GLUE\.

## Related Works

#### Variants of vanilla LoRA\.

Standard LoRA\(Huet al\.[2022](https://arxiv.org/html/2607.26247#bib.bib1)\)initializesB=0B=0andAArandomly, so fine\-tuning starts from zero perturbation of the pretrained model\. A first line of follow\-up work improves how the adapters are trained rather than where they start: rsLoRA\(Kalajdzievski[2023](https://arxiv.org/html/2607.26247#bib.bib23)\)rescales the update byγ/r\\gamma/\\sqrt\{r\}to stabilize high ranks, LoRA\+\(Hayouet al\.[2024](https://arxiv.org/html/2607.26247#bib.bib24)\)assigns separate learning rates toAAandBB, DoRA\(Liuet al\.[2024](https://arxiv.org/html/2607.26247#bib.bib25)\)decomposes weight updates into magnitude and directional components, and AdaLoRA\(Zhanget al\.[2023](https://arxiv.org/html/2607.26247#bib.bib5)\)reallocates the rank budget across layers by parameter importance\. None of these methods use task data to choose the initial subspace, which is the axis on which ULoRA operates\.

#### Pretrained\-informed initialization\.

A second line of work narrows the gap to full fine\-tuning by making the initialization itself data\-dependent\. Early methods draw the subspace from the model rather than the task: PiSSA\(Menget al\.[2024](https://arxiv.org/html/2607.26247#bib.bib4)\)initializes from the principal singular components of the pretrained weightW0W\_\{0\}, LoftQ\(Liet al\.[2024](https://arxiv.org/html/2607.26247#bib.bib26)\)jointly optimizes quantization and low\-rank initialization, and EVA\(Paischeret al\.[2025](https://arxiv.org/html/2607.26247#bib.bib27)\)uses the SVD of activation vectors to maximize captured activation variance\. These subspaces reflect what the pretrained model already represents, but not the direction in which the downstream loss wants to move it\.

#### Gradient\-based initialization\.

The methods most closely related to ULoRA derive the subspace directly from the downstream loss gradient\. LoRA\-GA\(Wang and others[2024](https://arxiv.org/html/2607.26247#bib.bib7)\)aligns the adapters with the first\-step full fine\-tuning gradient, and LoRA\-One\(Zhanget al\.[2025](https://arxiv.org/html/2607.26247#bib.bib20)\)proves that a single full\-gradient step yields near\-optimal subspace alignment under mild assumptions\. CG\-LoRA\(Zheng and Proutière[2026](https://arxiv.org/html/2607.26247#bib.bib21)\)goes one step further: rather than using the raw gradient, it whitens the gradient with Kronecker\-factored curvature \(K\-FAC\)\(Martens and Grosse[2015](https://arxiv.org/html/2607.26247#bib.bib18)\)before extracting the subspace, motivated by a function\-space alignment objective\. These methods therefore differ in exactly one design decision, namely how much curvature information reshapes the gradient before the subspace is extracted: none at all \(LoRA\-GA, LoRA\-One\) or full inverse\-square\-root whitening \(CG\-LoRA\)\. ULoRA makes this decision explicit and continuous, parameterizing the spectrum between unwhitened gradient projection, Adam\-like diagonal normalization, and full spectral whitening, and showing that the best operating point lies strictly between the published extremes\.

#### Learning rate sensitivity:

LoRA is highly sensitive to learning rate choice, and a well\-tuned vanilla LoRA is competitive with more complex methods\(Leeet al\.[2026](https://arxiv.org/html/2607.26247#bib.bib19)\)\. This motivates our full learning rate sweep across all methods and configurations\.

## Method

![Refer to caption](https://arxiv.org/html/2607.26247v1/schematics.png)Figure 1:Overview of ULoRA\. Top left: the continuum interpolation space spanned by spectral whitening \(α\\alpha\) and diagonal scaling \(β\\beta\), with existing methods \(e\.g\., CG\-LoRA\)\. Top right: the preconditioned projected gradientFα,βF^\{\\alpha,\\beta\}\. Bottom: schematics of the entire pipeline\.### Preconditioned Gradients: A Unifying View

Optimizers differ mostly in how they precondition the gradient∇Wℒ\\nabla\_\{W\}\\mathcal\{L\}before taking a step\. Plain gradient descent uses the raw gradient,Δ​W∝∇Wℒ\\Delta W\\propto\\nabla\_\{W\}\\mathcal\{L\}\. Adam\(Kingma and Ba[2014](https://arxiv.org/html/2607.26247#bib.bib22)\)rescales each coordinate by the inverse root of its second moment,Δ​W∝diag​\(v\)−1/2⊙∇Wℒ\\Delta W\\propto\\mathrm\{diag\}\(v\)^\{\-1/2\}\\odot\\nabla\_\{W\}\\mathcal\{L\}\. Natural gradient descent uses the full curvature,Δ​W∝F−1​∇ℒ\\Delta W\\propto F^\{\-1\}\\nabla\\mathcal\{L\}\(Amari[1998](https://arxiv.org/html/2607.26247#bib.bib9)\); under the Kronecker factorizationF≈T⊗SF\\approx T\\otimes Sof K\-FAC\(Martens and Grosse[2015](https://arxiv.org/html/2607.26247#bib.bib18)\), this becomesΔ​W∝T−1​∇Wℒ​S−1\\Delta W\\propto T^\{\-1\}\\nabla\_\{W\}\\mathcal\{L\}\\,S^\{\-1\}\. Shampoo\(Guptaet al\.[2018](https://arxiv.org/html/2607.26247#bib.bib39)\)interpolates between these regimes with quarter\-power factors,Δ​W∝T−1/4​∇Wℒ​S−1/4\\Delta W\\propto T^\{\-1/4\}\\nabla\_\{W\}\\mathcal\{L\}\\,S^\{\-1/4\}\. All of these are instances of a single two\-parameter family:

Pα,β=T−α/2​\[dT−β/2⊙∇Wℒ⊙dS−β/2\]​S−α/2,P\_\{\\alpha,\\beta\}\\;=\\;T^\{\-\\alpha/2\}\\Big\[d\_\{T\}^\{\-\\beta/2\}\\odot\\nabla\_\{W\}\\mathcal\{L\}\\odot d\_\{S\}^\{\-\\beta/2\}\\Big\]S^\{\-\\alpha/2\},\(1\)whereα\\alphacontrols the strength of Kronecker\-factored curvature whitening,β\\betacontrols Adam\-like per\-feature diagonal normalization, anddS,dTd\_\{S\},d\_\{T\}are the diagonals ofS,TS,T\. Setting\(α,β\)=\(0,0\)\(\\alpha,\\beta\)=\(0,0\)recovers the raw gradient;\(0,β\)\(0,\\beta\)recovers Adam\-like diagonal scaling;\(2,0\)\(2,0\)recovers the K\-FAC natural gradient;\(0\.5,0\)\(0\.5,0\)matches Shampoo’s quarter\-power preconditioning; and\(1,0\)\(1,0\)recovers inverse\-square\-root whitening\.

Our key observation is that gradient\-based LoRA initialization faces the same design choice\. LoRA\-GA\(Wang and others[2024](https://arxiv.org/html/2607.26247#bib.bib7)\)and LoRA\-One\(Zhanget al\.[2025](https://arxiv.org/html/2607.26247#bib.bib20)\)extract the adapter subspace from the raw gradient, i\.e\.P0,0P\_\{0,0\}, while CG\-LoRA\(Zheng and Proutière[2026](https://arxiv.org/html/2607.26247#bib.bib21)\)extracts it from the whitened gradientT−1/2​∇Wℒ​S−1/2T^\{\-1/2\}\\nabla\_\{W\}\\mathcal\{L\}\\,S^\{\-1/2\}, i\.e\.P1,0P\_\{1,0\}\. ULoRA makes this choice explicit and continuous; we initialize the adapters from the top\-rrsubspace ofPα,βP\_\{\\alpha,\\beta\}, computed efficiently in a projected Rayleigh–Ritz basis as described next\.

### Curvature Estimation and Gradient Projection

For a linear layer with pretrained weightW0∈ℝdout×dinW\_\{0\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}, lethi∈ℝdinh\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\}be the layer input andδi∈ℝdout\\delta\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\}be the pre\-activation output gradient for sampleii\. On a small initialization batch, the K\-FAC approximation\(Martens and Grosse[2015](https://arxiv.org/html/2607.26247#bib.bib18)\)to the Hessian of the loss yields two Kronecker factors:

S=1n​∑ihi​hi⊤∈ℝdin×din,T=∑iδi​δi⊤∈ℝdout×dout,S=\\frac\{1\}\{n\}\\sum\_\{i\}h\_\{i\}h\_\{i\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\\times d\_\{\\text\{in\}\}\},\\qquad T=\\sum\_\{i\}\\delta\_\{i\}\\delta\_\{i\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{out\}\}\},\(2\)whereSScaptures the covariance of layer inputs andTTcaptures the covariance of output gradients across the batch\. We also collect the per\-feature second moments:

dS​\[j\]=1n​∑ihi​j2,dT​\[k\]=1n​∑iδi​k2,d\_\{S\}\[j\]=\\frac\{1\}\{n\}\\sum\_\{i\}h\_\{ij\}^\{2\},\\qquad d\_\{T\}\[k\]=\\frac\{1\}\{n\}\\sum\_\{i\}\\delta\_\{ik\}^\{2\},\(3\)which are the diagonal entries ofSSandTT\. Note thatdSd\_\{S\}anddTd\_\{T\}are the per\-coordinate second moments used by Adam as a diagonal curvature proxy\. MaterializingPα,β​\(G\)P\_\{\\alpha,\\beta\}\(G\)at full size would be as expensive as a full fine\-tuning step, so all computation is carried out in a low\-dimensional projected space\. We compute a rank\-ss\(s=r\+oversamples=r\+\\text\{oversample\}\) Rayleigh–Ritz approximation ofSSandTT, obtaining the principal eigenvectors and eigenvalues:

S≈US​DS​US⊤,T≈UT​DT​UT⊤\.S\\approx U\_\{S\}D\_\{S\}U\_\{S\}^\{\\top\},\\qquad T\\approx U\_\{T\}D\_\{T\}U\_\{T\}^\{\\top\}\.\(4\)The projected gradient is defined as:

F^=UT⊤​∇Wℒ​US∈ℝs×s\.\\hat\{F\}=U\_\{T\}^\{\\top\}\\,\\nabla\_\{W\}\\mathcal\{L\}\\,U\_\{S\}\\in\\mathbb\{R\}^\{s\\times s\}\.\(5\)This matrix represents the loss gradient compressed into the Kronecker subspace: it indicates which combinations of input and output directions contain the strongest gradient signal\. The two preconditioning exponents of Eq\.[1](https://arxiv.org/html/2607.26247#Sx3.E1)are then applied in this projected space\.

### Theβ\\betaParameter: Adam\-like Diagonal Preconditioning

Before formingF^\\hat\{F\}, we rescale the basis vectorsUSU\_\{S\}andUTU\_\{T\}per feature using the diagonal statisticsdSd\_\{S\}anddTd\_\{T\}:

U~S=\(dS\+εS\)−β/2⊙US,U~T=\(dT\+εT\)−β/2⊙UT,\\tilde\{U\}\_\{S\}=\(d\_\{S\}\+\\varepsilon\_\{S\}\)^\{\-\\beta/2\}\\odot U\_\{S\},\\qquad\\tilde\{U\}\_\{T\}=\(d\_\{T\}\+\\varepsilon\_\{T\}\)^\{\-\\beta/2\}\\odot U\_\{T\},\(6\)where⊙\\odotdenotes elementwise row scaling andε\>0\\varepsilon\>0is a small numerical stabilizer\. Features with large empirical variance \(largedS​\[j\]d\_\{S\}\[j\]ordT​\[k\]d\_\{T\}\[k\]\) are downweighted proportionally\. This directly mirrors the adaptive step size in Adam: just as Adam divides the gradient by the root mean square of past gradients per coordinate, ULoRA divides the basis vectors by the root mean square of layer inputs and output gradients per feature\. The projected gradient using these scaled bases is:

F^β=U~T⊤​∇Wℒ​U~S\.\\hat\{F\}^\{\\beta\}=\\tilde\{U\}\_\{T\}^\{\\top\}\\,\\nabla\_\{W\}\\mathcal\{L\}\\,\\tilde\{U\}\_\{S\}\.\(7\)Whenβ=0\\beta=0, no diagonal correction is applied andF^0=F^\\hat\{F\}^\{0\}=\\hat\{F\}\. Whenβ=1\\beta=1, each feature dimension is normalized by its empirical standard deviation\.

### Theα\\alphaParameter: Spectral Whitening Power

Given the K\-FAC eigenvaluesDSD\_\{S\}andDTD\_\{T\}, we apply spectral scaling to the projected gradient:

Fα,β=DT−α/2​F^β​DS−α/2,F^\{\\alpha,\\beta\}=D\_\{T\}^\{\-\\alpha/2\}\\,\\hat\{F\}^\{\\beta\}\\,D\_\{S\}^\{\-\\alpha/2\},\(8\)where each rowkkis scaled byλT,k−α/2\\lambda\_\{T,k\}^\{\-\\alpha/2\}and each columnjjbyλS,j−α/2\\lambda\_\{S,j\}^\{\-\\alpha/2\}\. This reweights gradient directions by the inverse power of their curvature: directions residing in sharp regions of the loss landscape \(large eigenvalues\) are shrunk, while directions in flat regions \(small eigenvalues, but broad functional influence\) are amplified\. The effect ofα\\alphais as follows:

- •α=0\\alpha=0: No spectral whitening\. Only theβ\\betadiagonal correction is applied\. Gradient directions are weighted by their raw magnitude in the K\-FAC subspace\.
- •α=0\.5\\alpha=0\.5: Quarter\-power scaling per side, matching the preconditioning strength of Shampoo\(Guptaet al\.[2018](https://arxiv.org/html/2607.26247#bib.bib39)\)\.
- •α=1\\alpha=1,β=0\\beta=0: Inverse\-square\-root whitening\. Recovers CG\-LoRA\(Zheng and Proutière[2026](https://arxiv.org/html/2607.26247#bib.bib21)\)exactly\.
- •α=2\\alpha=2: Full inverse\-curvature scaling, corresponding to the K\-FAC natural gradient; increasingly penalizes high\-curvature directions and amplifies estimation noise in flat ones\.

### Cross\-Entropy Correction

The whitening above treats output space as Euclidean, which is exact for squared loss, whose Hessian with respect to the network output is the identity\. Cross\-entropy instead induces the output\-space curvatureΛ=diag​\(p\)−p​p⊤\\Lambda=\\mathrm\{diag\}\(p\)\-pp^\{\\top\}, whereppis the softmax output\(Martens and Grosse[2015](https://arxiv.org/html/2607.26247#bib.bib18)\), so distances in output space must be measured in this metric\. We accumulate this curvature at the layer output over the initialization batch,H^out\\hat\{H\}\_\{\\text\{out\}\}, project it into the whitened basis,Φ=DT−α/2​U~T⊤​H^out​U~T​DT−α/2\\Phi=D\_\{T\}^\{\-\\alpha/2\}\\,\\tilde\{U\}\_\{T\}^\{\\top\}\\hat\{H\}\_\{\\text\{out\}\}\\tilde\{U\}\_\{T\}\\,D\_\{T\}^\{\-\\alpha/2\}, and whiten once more:G=Φ−1/2​Fα,βG=\\Phi^\{\-1/2\}F^\{\\alpha,\\beta\}\. For squared loss,G=Fα,βG=F^\{\\alpha,\\beta\}directly; for details see\(Zheng and Proutière[2026](https://arxiv.org/html/2607.26247#bib.bib21)\)\.

### Symmetric Low\-Rank Initialization

Given the final matrixGG, we compute its rank\-rrSVD and back\-project the singular vectors to the original parameter space:

Lr=U~T​DT−α/2​Ur,Rr=U~S​DS−α/2​Vr,L\_\{r\}=\\tilde\{U\}\_\{T\}\\,D\_\{T\}^\{\-\\alpha/2\}\\,U\_\{r\},\\qquad R\_\{r\}=\\tilde\{U\}\_\{S\}\\,D\_\{S\}^\{\-\\alpha/2\}\\,V\_\{r\},\(9\)whereUrU\_\{r\}andVrV\_\{r\}are the top\-rrleft and right singular vectors ofGG\. We then apply a QR and tiny\-SVD balanced factorization to distribute singular values equally betweenA0A\_\{0\}andB0B\_\{0\}, ensuring neither adapter dominates\. Concretely, letQL​RL=QR​\(Lr\)Q\_\{L\}R\_\{L\}=\\text\{QR\}\(L\_\{r\}\)andQR​RR=QR​\(Rr\)Q\_\{R\}R\_\{R\}=\\text\{QR\}\(R\_\{r\}\)\. We form ther×rr\\times rcore matrixM=RL​diag​\(Dr\)​RR⊤M=R\_\{L\}\\,\\text\{diag\}\(D\_\{r\}\)\\,R\_\{R\}^\{\\top\}and compute its SVDM=UM​SM​VM⊤M=U\_\{M\}S\_\{M\}V\_\{M\}^\{\\top\}\. Then:

B0=QL​UM​SM,A0=\(QR​VM​SM\)⊤\.B\_\{0\}=Q\_\{L\}U\_\{M\}\\sqrt\{S\_\{M\}\},\\qquad A\_\{0\}=\\left\(Q\_\{R\}V\_\{M\}\\sqrt\{S\_\{M\}\}\\right\)^\{\\top\}\.\(10\)Finally, we normalize both adapters:A0←dout1/4γ​A0A\_\{0\}\\leftarrow\\frac\{d\_\{\\text\{out\}\}^\{1/4\}\}\{\\gamma\}A\_\{0\}andB0←dout1/4γ​B0B\_\{0\}\\leftarrow\\frac\{d\_\{\\text\{out\}\}^\{1/4\}\}\{\\gamma\}B\_\{0\}, withγ=16\\gamma=16\. The entire procedure never forms a matrix of sizedout×dind\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}; all computation stays in the projected space of dimensions=r\+oversamples=r\+\\text\{oversample\}\.

### Initialization Workflow

The full step\-by\-step pipeline of ULoRA initialization is summarized in Table[1](https://arxiv.org/html/2607.26247#Sx3.T1)\. After initialization, standard LoRA fine\-tuning proceeds:

W=W0\+1r​\(B0\+Δ​B\)​\(A0\+Δ​A\),W=W\_\{0\}\+\\tfrac\{1\}\{\\sqrt\{r\}\}\(B\_\{0\}\+\\Delta B\)\(A\_\{0\}\+\\Delta A\),\(11\)updating onlyΔ​A\\Delta AandΔ​B\\Delta B\.

### LoRA\-Auto

One way to set\(α,β\)\(\\alpha,\\beta\)is grid search, but can the operating point be chosen from data, without any search? Our analysis suggests it can: the risk of strong whitening is amplifying noise\-dominated flat directions, and the usefulness of diagonal scaling depends on the covariance actually being near\-diagonal\. Both properties are measurable per layer from quantities already computed during initialization\. For each sideΣ∈\{S,T\}\\Sigma\\in\\\{S,T\\\}with eigenvalues\{di\}\\\{d\_\{i\}\\\}, we summarize the spectrum by its effective rankreff=\(∑idi\)2/∑idi2r\_\{\\text\{eff\}\}=\(\\sum\_\{i\}d\_\{i\}\)^\{2\}/\\sum\_\{i\}d\_\{i\}^\{2\}and condition numberκ=maxi⁡\(di\)/\(mini⁡\(di\)\+ϵ\)\\kappa=\\max\_\{i\}\(d\_\{i\}\)/\(\\min\_\{i\}\(d\_\{i\}\)\+\\epsilon\), and set

α∗=21\+ln⁡\(κ\)/\(ln⁡\(s\)​reff\),\\alpha^\{\*\}=\\frac\{2\}\{\\,1\+\\ln\(\\kappa\)/\\big\(\\ln\(s\)\\,r\_\{\\text\{eff\}\}\\big\)\\,\},\(12\)so that well\-conditioned, high\-effective\-rank spectra receive strong whitening \(α∗→2\\alpha^\{\*\}\\\!\\to\\\!2\) while spiky, ill\-conditioned spectra, where inverse\-power scaling would amplify estimation noise, are whitened conservatively\. Forβ\\beta, we compute the off\-diagonal energy ratio of the projected covarianceB∈\{Bin,Bout\}B\\in\\\{B\_\{\\text\{in\}\},B\_\{\\text\{out\}\}\\\},ω=\(‖B‖F2−‖diag​\(B\)‖F2\)/‖B‖F2\\omega=\\big\(\\\|B\\\|\_\{F\}^\{2\}\-\\\|\\mathrm\{diag\}\(B\)\\\|\_\{F\}^\{2\}\\big\)/\\\|B\\\|\_\{F\}^\{2\}, and setβ∗=\(1−ω\)2\\beta^\{\*\}=\(1\-\\omega\)^\{2\}; diagonal preconditioning is applied only to the extent that the covariance is empirically diagonal\. Each adapted layer thus receives its own\(αS∗,αT∗,βS∗,βT∗\)\(\\alpha^\{\*\}\_\{S\},\\alpha^\{\*\}\_\{T\},\\beta^\{\*\}\_\{S\},\\beta^\{\*\}\_\{T\}\)at zero search cost, navigating a per\-layer design space no single global grid point can reach\.

Table 1:Step\-by\-step workflow of ULoRA initialization per layer\. Step 3 is enabled only in ULoRA\-Auto, which selects\(αS∗,αT∗,βS∗,βT∗\)\(\\alpha^\{\*\}\_\{S\},\\alpha^\{\*\}\_\{T\},\\beta^\{\*\}\_\{S\},\\beta^\{\*\}\_\{T\}\)per layer and per side from the spectral shape \(reff,κr\_\{\\text\{eff\}\},\\kappa\) and off\-diagonal energy \(ω\\omega\) of the estimated covariances; steps 4–9 then use these per\-side exponents\.

## Experiments

#### Models and tasks\.

We evaluate ULoRA in two settings\.NLU:we fine\-tune RoBERTa\-base\(Liuet al\.[2019](https://arxiv.org/html/2607.26247#bib.bib12)\)and T5\-base\(Raffelet al\.[2020](https://arxiv.org/html/2607.26247#bib.bib28)\)on five GLUE tasks\(Wanget al\.[2018](https://arxiv.org/html/2607.26247#bib.bib29)\)\(CoLA, MNLI, QNLI, SST\-2, MRPC\); for T5 we score each class by teacher\-forcing its label token\.NLG:we fine\-tune LLaMA 2\-7B\(Touvronet al\.[2023](https://arxiv.org/html/2607.26247#bib.bib17)\)on three tasks: mathematical reasoning, trained on 100K MetaMathQA samples\(Yuet al\.[2023](https://arxiv.org/html/2607.26247#bib.bib33)\)and evaluated on GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2607.26247#bib.bib30)\)with regex\-extracted answer accuracy; code generation, trained on 100K Code\-Feedback samples\(Weiet al\.[2023](https://arxiv.org/html/2607.26247#bib.bib35)\)with explanations removed and evaluated on HumanEval\(Chenet al\.[2021](https://arxiv.org/html/2607.26247#bib.bib36)\)with Pass@1; and question answering, trained on a filtered 52K WizardLM subset\(Xuet al\.[2023](https://arxiv.org/html/2607.26247#bib.bib37)\)and evaluated on MMLU\(Hendryckset al\.[2020](https://arxiv.org/html/2607.26247#bib.bib38)\)with answer accuracy\.

#### Baselines\.

We compare against rsLoRA\(Kalajdzievski[2023](https://arxiv.org/html/2607.26247#bib.bib23)\), LoRA\+\(Hayouet al\.[2024](https://arxiv.org/html/2607.26247#bib.bib24)\), PiSSA\(Menget al\.[2024](https://arxiv.org/html/2607.26247#bib.bib4)\), LoRA\-GA\(Wang and others[2024](https://arxiv.org/html/2607.26247#bib.bib7)\), LoRA\-One\(Zhanget al\.[2025](https://arxiv.org/html/2607.26247#bib.bib20)\), and CG\-LoRA\(Zheng and Proutière[2026](https://arxiv.org/html/2607.26247#bib.bib21)\), as well as full fine\-tuning\.

#### Configurations\.

We set rankr=8r=8, no dropout, andγ=16\\gamma=16\. For RoBERTa and T5, we adapt query, key, value, and dense projection layers\. For LLaMA, we adapt all linear layers within the transformer backbone, excluding the final language modeling head\.

#### Training\.

All models are trained for one epoch with AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, weight decay0\), a cosine learning rate schedule with 3% warmup, batch size 32\. We employ a mixed\-precision framework where the LLaMA backbone is maintained in BF16, and the LoRA’sAAandBBmatrices are cast to FP32 across all evaluated architectures\. Each configuration is run with 3 random seeds, and we report the average performance\.

#### Learning rate sweep\.

For RoBERTa and T5, we search the learning rate over\{2×10−5,5×10−5,10−4,2×10−4,5×10−4,8×10−4,10−3\}\\\{2\\times 10^\{\-5\},5\\times 10^\{\-5\},10^\{\-4\},2\\times 10^\{\-4\},5\\times 10^\{\-4\},8\\times 10^\{\-4\},10^\{\-3\}\\\}and report the best\. For LLaMA, we tune the learning rate with Optuna\(Agrawal[2020](https://arxiv.org/html/2607.26247#bib.bib44)\)since learning rate strongly affects LoRA performance\(Leeet al\.[2026](https://arxiv.org/html/2607.26247#bib.bib19)\)\.

#### ULoRA grid\.

We sweepα∈\{0\.0,0\.5,1\.0,1\.5,2\.0\}\\alpha\\in\\\{0\.0,0\.5,1\.0,1\.5,2\.0\\\}andβ∈\{0\.0,0\.5,1\.0\}\\beta\\in\\\{0\.0,0\.5,1\.0\\\}, giving 15 configurations per task\.

#### Compute infrastructure\.

Experiments for RoBERTa\-base and T5\-base were conducted on NVIDIA RTX 6000 Pro Blackwell GPUs while for LLaMA 2\-7B, we used NVIDIA H200 GPUs\.

### Results

Two findings emerge consistently across all experiments\. First, preconditioning strength matters: taken as an oracle upper bound of the family, the best point in the\(α,β\)\(\\alpha,\\beta\)family matches or exceeds baselines on 11 of 13 comparisons and matches or exceeds full fine\-tuning on all five RoBERTa GLUE tasks\. Second, and more surprising, the right strength is not what existing methods assume: the optimum is task\- and model\-dependent, frequently lies away from both published endpoints, no preconditioning in LoRA\-GA and LoRA\-One and full inverse\-square\-root whitening in CG\-LoRA, and neither endpoint is optimal in the majority of settings\. The claim we defend is therefore not that one configuration wins everywhere, which our own data rules out, but that the preconditioning exponent is a design variable with measurable effects\.

Table 2:GLUE benchmark results for RoBERTa, averaged over 3 seeds \(mean±\\pmsd\)\. Bold/underline = best/second best among deployable methods\.∗ULoRA \(UB\) is an oracle upper bound; the best\(α,β\)\(\\alpha,\\beta\)was selected\.Table 3:GLUE benchmark results for T5, averaged over 3 seeds \(mean±\\pmsd\)\. Bold/underline = best/second best among deployable methods\.∗ULoRA \(UB\) is an oracle upper bound; the best\(α,β\)\(\\alpha,\\beta\)was selected\.#### RoBERTa on GLUE\.

Table[2](https://arxiv.org/html/2607.26247#Sx4.T2)shows that some point in the\(α,β\)\(\\alpha,\\beta\)family matches or exceeds every baseline, including full fine\-tuning, on all five tasks: CoLA \(81\.1 vs\. 81\.0\), MNLI \(86\.8, tie\), QNLI \(92\.2, tie\), SST\-2 \(94\.5 vs\. 93\.8\), and MRPC \(87\.0 vs\. 86\.8\)\. More important than the margins is where these points lie: the selected exponents cluster at moderate whitening,α∈\{1\.0,1\.5\}\\alpha\\in\\\{1\.0,1\.5\\\}, withβ\>0\\beta\>0on three tasks, and on three of five tasks the optimum is not the CG\-LoRA point\(1\.0,0\.0\)\(1\.0,0\.0\)\. In this regime, encoder\-only classification with a small initialization batch, the K\-FAC estimates are comparatively clean, so amplifying flat but functionally influential directions pays off, and the diagonalβ\\betacorrection absorbs heterogeneous feature scales\.

#### T5 on GLUE\.

The T5 results \(Table[3](https://arxiv.org/html/2607.26247#Sx4.T3)\) are best interpreted as a saturation regime\. With a tuned learning rate, vanilla LoRA already matches full fine\-tuning on QNLI \(93\.3 vs\. 93\.3\) and SST\-2 \(94\.1 vs\. 94\.4\), all methods fall within 0\.2 points of each other on MNLI and QNLI, and the residual gaps on the small, high\-variance tasks CoLA and MRPC are comparable to a single seed standard deviation \(0\.5 to 2\.1\)\. When tuned LoRA saturates to full fine\-tuning accuracy, consistent with\(Leeet al\.[2026](https://arxiv.org/html/2607.26247#bib.bib19)\), initialization has little leverage, and no method can meaningfully separate from the pack; the family’s oracle upper bound is accordingly at or near the top, with margins we do not consider significant\. The informative signal is instead where the optimum lands: it collapses toward weak preconditioning, with\(α,β\)=\(0\.0,0\.0\)\(\\alpha,\\beta\)=\(0\.0,0\.0\)selected on MNLI and SST\-2, possibly because gradients through cross\-attention and a large output softmax yield noisier K\-FAC factors\. This is consistent with the framework rather than against it; where there is no headroom and curvature estimates are unreliable, the family correctly selects little to no preconditioning \.

Table 4:Benchmark results for LLaMA\-2\-7B, averaged over 3 seeds \(mean±\\pmsd\)\. Bold/underline = best/second best among deployable methods\.∗ULoRA \(UB\) is an oracle upper bound; the best\(α,β\)\(\\alpha,\\beta\)was selected\.
#### LLaMA 2\-7B\.

The generation benchmarks \(Table[4](https://arxiv.org/html/2607.26247#Sx4.T4)\) push this further\. The best configuration is \(0\.0,0\.0\) on GSM8K \(57\.97\), \(2\.0,0\.5\) on HumanEval \(28\.46\), and \(1\.5,1\.0\) on MMLU, so the optimum varies across tasks even within one backbone\. At the 7B scale, single\-batch curvature estimates over large hidden dimensions are the noisiest of our settings, and for GSM8K, the raw gradient subspace is the safest choice\. Note that ULoRA at\(0\.0,0\.0\)\(0\.0,0\.0\)still outperforms LoRA\-GA \(50\.72 on GSM8K\) and LoRA\-One \(55\.85\), because the rest of the pipeline differs: the Rayleigh–Ritz projection, the cross\-entropy Fisher correction, and the balanced realization apply regardless of the exponents\. Consistent with prior reports, several LoRA variants exceed full fine\-tuning on GSM8K and HumanEval; the low\-rank constraint acts as a regularizer in single\-epoch instruction tuning\.

![Refer to caption](https://arxiv.org/html/2607.26247v1/countour.png)Figure 2:Heatmap of the fine grid search forα,β∈\[−3,\+3\]\\alpha,\\beta\\in\[\-3,\+3\]on GLUE MRPC, averaged across 3 seed runs\. The maximum accuracy is marked by a star\.
#### Accuracy landscape\.

Figure[2](https://arxiv.org/html/2607.26247#Sx4.F2)maps test accuracy on MRPC over the extended rangeα,β∈\[−3,\+3\]\\alpha,\\beta\\in\[\-3,\+3\]\. The landscape is highly asymmetric inα\\alpha: negative values, which amplify high\-curvature directions instead of suppressing them, degrade accuracy by up to 15 points, whileα\>0\.5\\alpha\>0\.5forms a broad plateau of high accuracy within whichβ\\betahas a mild effect\. The maximum lies in the interior near\(0\.75,−0\.25\)\(0\.75,\-0\.25\), at none of the published endpoints\. Two practical points follow: the exponent is forgiving once past the transition, so precise tuning is rarely necessary, but the existence of a sharp failure region shows that the choice is not free either\.

#### ULoRA\-Auto\.

Grid search multiplies initialization cost, so we also evaluate ULoRA\-Auto, which sets\(α,β\)\(\\alpha,\\beta\)per layer from spectral statistics the pipeline already computes: the effective rank and condition number of the Ritz eigenvalue spectra determine a decoupled whitening exponent for the input and output sides, and the off\-diagonal energy ratio of the projected covariances sets an exponentially dampedβ\\beta\. This rule backs off whitening exactly when the curvature estimate looks ill\-conditioned or structurally noisy\. It is a heuristic rather than a derived optimum, but it performs remarkably well for one: with zero tuning it ranks at or near the top among deployable methods on every RoBERTa task, stays within 0\.1 to 0\.6 points of the oracle grid search throughout, and is the single best method on MMLU \(45\.43\), where it beats its own grid\-searched parent\. In practice, ULoRA\-Auto delivers most of the benefit of the full family at the cost of one initialization pass, and it provides direct evidence that the right exponent is predictable from spectral statistics available at initialization, not only findable by search\.

Auto’s per\-layer selections separate cleanly by side \(βS\\beta\_\{S\}near 1,βT\\beta\_\{T\}low\), matching the measured off\-diagonal energy of each \(Figure[3](https://arxiv.org/html/2607.26247#Sx4.F3)\)\. On QNLI, the selections land in the grid’s best region and recover near\-optimal accuracy at zero search cost\. On GSM8K, where the grid favors the raw\-gradient corner, Auto prescribes moderate whitening instead: the diagnostics measure estimation quality, not task\-level utility\.

![Refer to caption](https://arxiv.org/html/2607.26247v1/ulora_auto_analysis.png)Figure 3:Per\-layer exponents selected by ULoRA\-Auto \(dots\) overlaid on the global grid sweep \(circles, mean test accuracy\) for QNLI \(top\) and GSM8K \(bottom\)\.
#### Limitations\.

The grid\-searched ULoRA numbers select\(α,β\)\(\\alpha,\\beta\)per task on test metrics, an oracle protocol; ULoRA\-Auto is the deployable variant and should be read as our single\-configuration result\. The grid\-searched family further assumes a single global\(α,β\)\(\\alpha,\\beta\)shared across all layers and both Kronecker sides, adopted only for search tractability, since a per\-layer grid search would multiply the already substantial sweep cost by the number of adapted layers; ULoRA\-Auto lifts this assumption per layer and side at no search cost, which is why it can exceed the best global\-grid point \(e\.g\., on MMLU\)\. The ULoRA\-Auto rule is empirically motivated, and we provide no theoretical characterization of the optimal exponent as a function of curvature\-estimation noise\. Experiments cover rank 8, single\-epoch training, and models up to 7B parameters on NLP benchmarks; behavior at higher ranks, longer training, and other modalities is untested\. Initialization adds one forward\-backward pass and per\-layer eigendecompositions, a small but nonzero overhead over vanilla LoRA\.

## Conclusion

We showed that gradient\-based LoRA initialization methods, which have so far been developed as separate techniques, are points on a single two\-parameter continuum of preconditioned gradients, and that this reframing has empirical teeth\. Under a full learning\-rate sweep across two encoders and a 7B decoder, and treated as an oracle upper bound, tuned points in the family match or exceed every LoRA baseline on 11 of 13 comparisons and match or exceed full fine\-tuning on all five RoBERTa GLUE tasks; the deployable ULoRA\-Auto recovers most of this gain with no tuning\. The deeper result is where those points lie: the optimal preconditioning strength is systematically task\- and model\-dependent, tracks the reliability of the curvature estimate, and coincides with a published endpoint only occasionally\. Fixed choices, whether no whitening or full whitening, leave accuracy on the table in most settings, and the extended landscape shows a genuine failure region that a fixed choice cannot guard against\. ULoRA\-Auto demonstrates that the right strength can be read off spectral statistics at initialization, recovering most of the gain of an oracle grid search with no tuning\. We view the specific family studied here as a first instantiation rather than the final word; making the preconditioning decision explicit, and continuous\.

## References

- T\. Agrawal \(2020\)Optuna and automl\.InHyperparameter Optimization in Machine Learning: Make Your Machine Learning and Deep Learning Models More Efficient,pp\. 109–129\.Cited by:[Learning rate sweep\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px5.p1.1)\.
- S\. Amari \(1998\)Natural gradient works efficiently in learning\.Neural Computation10\(2\),pp\. 251–276\.Cited by:[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p1.7)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, and C\. Winter \(2021\)Evaluating large language models trained on code\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2107.03374)Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- V\. Gupta, T\. Koren, and Y\. Singer \(2018\)Shampoo: preconditioned stochastic tensor optimization\.InInternational Conference on Machine Learning,pp\. 1842–1850\.Cited by:[2nd item](https://arxiv.org/html/2607.26247#Sx3.I3.i2.p1.1),[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p1.7)\.
- S\. Hayou, N\. Ghosh, and B\. Yu \(2024\)Lora\+: efficient low rank adaptation of large models\.arXiv preprint arXiv:2402\.12354\.Cited by:[Variants of vanilla LoRA\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px1.p1.5),[Baselines\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px2.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2009.03300)Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/pdf/2106.09685)Cited by:[Introduction](https://arxiv.org/html/2607.26247#Sx1.p1.3),[Variants of vanilla LoRA\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px1.p1.5)\.
- D\. Kalajdzievski \(2023\)A rank stabilization scaling factor for fine\-tuning with lora\.arXiv preprint arXiv:2312\.03732\.Cited by:[Variants of vanilla LoRA\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px1.p1.5),[Baselines\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px2.p1.1)\.
- D\. P\. Kingma and J\. Ba \(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[2nd item](https://arxiv.org/html/2607.26247#Sx1.I1.i2.p1.1),[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p1.7)\.
- Y\. Lee, C\. Ko, P\. Chen, and M\. Yeh \(2026\)Learning rate matters: vanilla lora may suffice for llm fine\-tuning\.arXiv preprint arXiv:2602\.04998\.Cited by:[Learning rate sensitivity:](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px4.p1.1),[T5 on GLUE\.](https://arxiv.org/html/2607.26247#Sx4.SSx1.SSS0.Px2.p1.1),[Learning rate sweep\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px5.p1.1)\.
- Y\. Li, Y\. Yu, C\. Liang, N\. Karampatziakis, P\. He, W\. Chen, and T\. Zhao \(2024\)Loftq: lora\-fine\-tuning\-aware quantization for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 13409–13424\.Cited by:[Pretrained\-informed initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px2.p1.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024\)Dora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[Variants of vanilla LoRA\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px1.p1.5)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- J\. Martens and R\. Grosse \(2015\)Optimizing neural networks with kronecker\-factored approximate curvature\.InProceedings of the 32nd International Conference on Machine Learning \(ICML\),pp\. 2408–2417\.External Links:[Link](https://proceedings.mlr.press/v37/martens15.html)Cited by:[Gradient\-based initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px3.p1.1),[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p1.7),[Curvature Estimation and Gradient Projection](https://arxiv.org/html/2607.26247#Sx3.SSx2.p1.4),[Cross\-Entropy Correction](https://arxiv.org/html/2607.26247#Sx3.SSx5.p1.6)\.
- F\. Meng, Z\. Wang, M\. Zhang, Y\. Su, Y\. Yang, Z\. Zhao, J\. Wang, M\. Zhou, and W\. Xu \(2024\)PiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://papers.nips.cc/paper_files/paper/2024/hash/db36f4d603cc9e3a2a5e10b93e6428f2-Abstract-Conference.html)Cited by:[Pretrained\-informed initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px2.p1.1),[Baselines\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px2.p1.1)\.
- F\. Paischer, L\. Hauzenberger, T\. Schmied, B\. Alkin, M\. P\. Deisenroth, and S\. Hochreiter \(2025\)Parameter efficient fine\-tuning via explained variance adaptation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Pretrained\-informed initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px2.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman \(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- J\. Wanget al\.\(2024\)LoRA\-GA: low\-rank adaptation with gradient approximation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.37\.Cited by:[Introduction](https://arxiv.org/html/2607.26247#Sx1.p2.4),[Gradient\-based initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px3.p1.1),[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p2.5),[Baselines\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px2.p1.1)\.
- Y\. Wei, Z\. Wang, J\. Liu, Y\. Ding, and L\. Zhang \(2023\)Magicoder: empowering code generation with oss\-instruct\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2312.02120)Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, Q\. Lin, and D\. Jiang \(2023\)WizardLM: empowering large pre\-trained language models to follow complex instructions\.The Twelfth International Conference on Learning Representations \(ICLR 2024\)\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2304.12244)Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. Liu \(2023\)MetaMath: bootstrap your own mathematical questions for large language models\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/arxiv.2309.12284)Cited by:[Models and tasks\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px1.p1.1)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, T\. Peng, G\. Neubig, P\. He, W\. Chen, and T\. Zhao \(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Variants of vanilla LoRA\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px1.p1.5)\.
- Y\. Zhang, F\. Liu, and Y\. Chen \(2025\)Lora\-one: one\-step full gradient could suffice for fine\-tuning large language models, provably and efficiently\.arXiv preprint arXiv:2502\.01235\.Cited by:[Introduction](https://arxiv.org/html/2607.26247#Sx1.p2.4),[Gradient\-based initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px3.p1.1),[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p2.5),[Baselines\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px2.p1.1)\.
- F\. Zheng and A\. Proutière \(2026\)Curvature\-guided lora: matching full fine\-tuning in function space\.arXiv preprint arXiv:2603\.29824\.Cited by:[Introduction](https://arxiv.org/html/2607.26247#Sx1.p2.4),[Gradient\-based initialization\.](https://arxiv.org/html/2607.26247#Sx2.SS0.SSS0.Px3.p1.1),[3rd item](https://arxiv.org/html/2607.26247#Sx3.I3.i3.p1.2),[Preconditioned Gradients: A Unifying View](https://arxiv.org/html/2607.26247#Sx3.SSx1.p2.5),[Cross\-Entropy Correction](https://arxiv.org/html/2607.26247#Sx3.SSx5.p1.6),[Baselines\.](https://arxiv.org/html/2607.26247#Sx4.SSx8.SSS0.Px2.p1.1)\.