One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

# Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
Source: [https://arxiv.org/html/2609.05885](https://arxiv.org/html/2609.05885)
## One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine\-Tuning

###### Abstract

Low\-rank adaptation \(LoRA\) has become the standard for parameter\-efficient fine\-tuning of large language models\. Most LoRA variants follow a*uniform\-LR convention*, applying a single global learning rate across every rank\-one component of every adapter\. We show that this convention overlooks substantial within\-module heterogeneity, where the rank\-one components of a LoRA adapter update at highly uneven rates and low\-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget\. To address this, we propose an adaptive anisotropic learning\-rate model that assigns each rank\-one component its own effective learning rate, computed online from training\-time signals and mean\-normalized per module to preserve the global LR budget\. AnLR\-LoRA instantiates this model with two signals available during AdamW optimization, namely function\-space velocity and Adam SNR, as a lightweight scheme with no extra trainable parameters\. Across commonsense reasoning, natural language generation and visual instruction\-tuning benchmarks, AnLR\-LoRA consistently improves over LoRA while encouraging broader use of rank capacity, with gains that remain robust across a wide range of global learning rates and transfer cleanly to other LoRA variants\.

## 1Introduction

Large language models \(LLMs\) have become the foundation of modern natural language processing, but adapting them to downstream tasks via full

Figure 1:Per\-rank\-one update velocity of LoRA modules\. Within a single LoRA, therrrank\-one components do not update at the same rate\.fine\-tuning is prohibitively expensive\. Parameter\-efficient fine\-tuning \(PEFT\) methods address this cost by training only a small subset of parameters\([Houlsby et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib18);[Li and Liang, 2021](https://arxiv.org/html/2609.05885#bib.bib19);[Jia et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib22)\)\. Low\-Rank Adaptation \(LoRA\)\([Hu et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib4)\)has emerged as a standard choice for LLM adaptation, with its low\-rank update parameterization offering a favorable balance between adaptation quality and computational cost\. Yet recent work identifies the learning rate \(LR\) as the dominant factor in LoRA tuning\([Lee et al\., 2026](https://arxiv.org/html/2609.05885#bib.bib31)\), which makes the*form*of the learning\-rate convention, not just its value, a worthwhile object of study\.

Standard LoRA training uses AdamW with a single global learning rate applied uniformly across every rank\-one component of every adapter\. We refer to this design as the*uniform\-LR convention*, which is inherited by most LoRA variants\. Our analysis \(Sec\.[3](https://arxiv.org/html/2609.05885#S3)\), however, shows that under this convention therrrank\-one components within a single adapter exhibit anisotropic per\-step velocities \(Fig\.[1](https://arxiv.org/html/2609.05885#S1.F1)\)\. This anisotropy emerges early in training and persists throughout\. This imbalance compounds over training, as modules whose rank\-one components evolve slowly converge to a concentrated spectrum of the LoRA updateB​ABA, leaving much of the nominal rank budget underutilized\. The anisotropy is*within\-module*and arises only during training, and it cannot be corrected by tuning a single global LR\. This suggests that both the granularity and adaptivity of LoRA’s LR allocation deserve closer examination\.

This anisotropy in LoRA training motivates assigning learning rates at the level of individual rank\-one components and adjusting them as training proceeds\. However, existing approaches still allocate learning rates at coarser granularities\. LoRA\+\([Hayou et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib26)\)partially relaxes the uniform\-LR convention by assigning separate learning rates to matricesAAandBB, but the adjustment remains at the matrix level and is fixed analytically before training\. Beyond LoRA\+, per\-matrix and per\-adapter schemes\([Chen et al\., 2026](https://arxiv.org/html/2609.05885#bib.bib36);[Huang and Balestriero, 2024](https://arxiv.org/html/2609.05885#bib.bib48)\)adjust the learning rate above the rank\-one level, while methods that operate at the rank\-one level intervene through gating\([Ding et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib20)\)or singular\-triplet pruning\([Zhang et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib5)\)rather than learning\-rate scheduling\. No existing method assigns a distinct, online adaptive learning rate to each rank\-one component within a LoRA adapter\.

We propose an*adaptive anisotropic learning\-rate*model for LoRA fine\-tuning, with a principled instantiation, Anisotropic Learning\-Rate LoRA \(AnLR\-LoRA\)\. AnLR\-LoRA assigns per\-rank\-one multipliers that redistribute a fixed global LR across therrrank\-one components of each adapter, driven online by two training\-time signals:*velocity*, the rate of change of each rank\-one contribution to the adapter update, which captures function\-space activity; and*Adam SNR*, the signal\-to\-noise ratio of each component’s parameters, which captures parameter\-space confidence\. The two signals are fused log\-additively and mean\-normalized per module, then applied as a post\-step scaling of the AdamW update, equivalent to running AdamW with an effective per\-component LR \(Proposition[2](https://arxiv.org/html/2609.05885#Thmproposition2)\)\. These signals adaptively encourage learning on low\-velocity rank\-one components that follow a reliable update direction\. AnLR\-LoRA thus promotes effective learning with no architectural changes and negligible computational overhead\.

Our contributions are summarized as follows:

- •We identify*anisotropic*learning rates, with distinct rates per rank\-one component, as the natural next granularity in the adaptive\-LR lineage and show per\-rank\-one dynamics are anisotropic within modules, persistent through training and tied to a measurable spectral concentration ofB​ABAbelow its nominal rank\.
- •We propose an adaptive anisotropic LR model that redistributes a fixed LR budget across rank\-one components using training\-time signals, and instantiate it as AnLR\-LoRA with signals of velocity and Adam SNR ratio\. AnLR\-LoRA enables low\-velocity rank\-one components with reliable update directions to actively engage in the training process\.
- •Extensive experiments show that adaptive anisotropic learning rate encourages utilization of rank budget and improves the adaptation of LoRA across various settings\. This benefit remains robust under varying learning rate and transfers to LoRA variants\.

## 2Related Work

### 2\.1Parameter\-Efficient Fine\-Tuning

Parameter\-efficient fine\-tuning \(PEFT\) reduces fine\-tuning cost by training only a small subset of model parameters\. Representative approaches include adapter modules inserted between transformer layers\([Houlsby et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib18);[Chen et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib21)\), prompt and prefix tuning that prepend learnable tokens to the input or to the attention state\([Li and Liang, 2021](https://arxiv.org/html/2609.05885#bib.bib19);[Jia et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib22)\), and methods that perform full\-parameter training inside a low\-rank gradient subspace\([Zhao et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib24);[Chen et al\., 2025](https://arxiv.org/html/2609.05885#bib.bib33)\)\. PEFT modules can also be composed into expandable mixtures of experts for continual learning\([Wang et al\., 2025b](https://arxiv.org/html/2609.05885#bib.bib55);[Zhao et al\., 2026](https://arxiv.org/html/2609.05885#bib.bib58)\)\. We focus on low\-rank adaptation, the standard PEFT approach for LLMs, reviewing its variants and optimization choices below\.

### 2\.2Architectural Variants of LoRA

LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib4)\)introduces a low\-rank update to a frozen pretrained model, parameterized by two factor matrices initialized so that the update is zero at the start of training\. DoRA modify this parameterization by separating magnitude and direction\([Liu et al\., 2024b](https://arxiv.org/html/2609.05885#bib.bib1)\)\. Initialization\-based variants place the factors on informative starting directions, from the principal or minor singular components\([Meng et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib2);[Wang et al\., 2025a](https://arxiv.org/html/2609.05885#bib.bib3)\)to the first\-step gradient\([Wang et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib32);[Zhang et al\., 2025b](https://arxiv.org/html/2609.05885#bib.bib37);[Si et al\., 2025](https://arxiv.org/html/2609.05885#bib.bib6)\)\. Some methods adjust the effective rank during training, such as SoRA\([Ding et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib20)\)with learnable gates, AdaLoRA\([Zhang et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib5)\)with singular\-triplet pruning, and DyLoRA\([Valipour et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib23)\)with sub\-rank sampling\. Low\-rank structure is similarly exploited in continual learning, where rank\-one components serve as incremental experts\([Lu et al\., 2026](https://arxiv.org/html/2609.05885#bib.bib56)\)and rank minimization acts as an implicit regularizer against forgetting\([Lu et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib57)\)\. Across these designs, all retain LoRA’s single global learning rate, applied uniformly across all parameters\.

\(a\)Visualization of per\-rank\-one velocity across training steps\.\(b\)Rank\-one velocity vs magnitude accumulated over training\.\(c\)Module velocity vs singular spectrum concentration at convergence\.
Figure 2:Per\-rank\-one anisotropy in LoRA training on math reasoning\. \(a\) The within\-module velocity ordering is established early and persists in training\. \(b\) High\-velocity components accumulate larger contribution magnitude throughout training\. \(c\) Lower mean per\-rank\-one velocity is associated with more concentrated singular spectra ofB​ABAat convergence, under\-utilizing the available rank budget\.
### 2\.3Optimization and Hyperparameter Choices for LoRA

A parallel line studies how LoRA is trained rather than parameterized\. The learning rate is the load\-bearing hyperparameter across LoRA variants once well\-tuned\([Lee et al\., 2026](https://arxiv.org/html/2609.05885#bib.bib31)\)\.[Hayou et al\. \(2024\)](https://arxiv.org/html/2609.05885#bib.bib26)observe that the two factor matrices play asymmetric roles during fine\-tuning and assign each its own learning rate\.[Chen et al\. \(2026\)](https://arxiv.org/html/2609.05885#bib.bib36)characterize how the global rate should scale with adapter rank, and rsLoRA\([Kalajdzievski, 2023](https://arxiv.org/html/2609.05885#bib.bib17)\)introduces a rank\-stabilized output scaling\. Other methods modify the optimizer update itself: LoRA\-Pro\([Wang et al\., 2025c](https://arxiv.org/html/2609.05885#bib.bib25)\), AltLoRA\([Yu et al\., 2025](https://arxiv.org/html/2609.05885#bib.bib34)\)and RefLoRA\([Zhang et al\., 2025a](https://arxiv.org/html/2609.05885#bib.bib35)\)adjust the per\-step gradient or factorization to better approximate a full fine\-tuning step\. These works operate at module or matrix granularity and none assigns a separate learning rate per rank component or derives one from training\-time signals\. LoRA\+\([Hayou et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib26)\)is closest to ours in showing that per\-component learning rates outperform the single\-LR convention, but its two matrix\-level learning rates are fixed at training start\. As we show in Sec\.[3](https://arxiv.org/html/2609.05885#S3), rank\-one components within a matrix remain heterogeneous throughout training, so per\-matrix allocation discards information preserved by per\-rank learning rates\. We derive these rates from two training\-time signals: the rate of change of each rank’s contribution and the consistency of its Adam gradient\.

## 3Analysis of Rank\-One Training Dynamics in LoRA

This analysis provides the empirical basis for the intervention proposed in Sec\.[4](https://arxiv.org/html/2609.05885#S4)\. We establish three observations from a LoRA fine\-tuning run of LLaMA2\-7B on MetaMathQA\. The analysis covers224224LoRA modules in total\. We refer to the Frobenius\-norm rate of change of each rank\-one contributionB:,kAk,:B\_\{:,k\}A\_\{k,:\}as its*per\-rank\-one velocity*, defined formally in Sec\.[4\.3](https://arxiv.org/html/2609.05885#S4.SS3.SSS0.Px2)\. We further define its*per\-rank\-one magnitude*as the cumulative Frobenius norm ofB:,kAk,:B\_\{:,k\}A\_\{k,:\}at each training step\. These measures allow us to characterize how adaptation is distributed across rank\-one components, both dynamically during optimization and cumulatively over training\.

### 3\.1Per\-Rank\-One Velocity is Anisotropic within Each Module

Within a single LoRA adapter, therrrank\-one components do not update at the same rate\. Fig\.[1](https://arxiv.org/html/2609.05885#S1.F1)shows the normalized update velocity of individual rank\-one components in LoRA modules, with the dashed line indicating the module\-wise mean\. The update activity is clearly uneven\. Some ranks move much faster than the module average, whereas others remain substantially slower\. Across 224 LoRA modules in our analysis, the median max/min velocity ratio is2×2\\times, the 90th percentile reaches4×4\\times, and the most imbalanced 1% exceed8×8\\times\. This heterogeneity appears across different projection layers, suggesting that LoRA rank\-one components are not optimized uniformly under a shared learning rate\. This anisotropy is*within\-module*: every rank inside a LoRA adapter shares the same global learning rate by construction, so the imbalance cannot be addressed by conventional uniform learning\-rate scheduling\. A more direct mechanism is therefore to redistribute the learning rate among therrrank\-one components within each module, rather than increasing the overall learning rate\.

### 3\.2The Anisotropy is Established Early and Persists

The velocity heterogeneity is not a transient effect\. Within each module, the rank\-one components that are the most and the least active early in training remain so through to convergence\. Fig\.[2\(a\)](https://arxiv.org/html/2609.05885#S2.F2.sf1)shows per\-rank\-one velocity trajectories in a LoRA module, with each rank colored by its index\. The ordering is established early in training, within approximately the first 1k steps and preserved thereafter\. The persistent velocity ordering translates directly to a contribution\-magnitude ordering\. Fig\.[2\(b\)](https://arxiv.org/html/2609.05885#S2.F2.sf2)shows the mean contribution magnitude of the top\-velocity quartile pulls ahead in early training and the gap persists throughout the training process\. Under uniform LR there is no mechanism to redistribute capacity, so the imbalance compounds\. The ordering is*not predictable fromA\(0\),B\(0\)A^\{\(0\)\},B^\{\(0\)\}alone*, but is stabilized within the early training stage\. The right per\-rank\-one allocation should therefore be discovered during training and applied throughout, not preset analytically\.

### 3\.3Low\-Velocity Modules Under\-Utilize their Nominal Rank Budget

The within\-module magnitude imbalance has a further consequence: modules whose rank\-one components evolve most slowly under uniform LR end up with the most concentrated singular spectra in the learned composite updateB​ABA, packing their squared\-Frobenius energy into only a few singular directions and leaving the rest of the nominal rank budget effectively unused\. As shown in Fig\.[2\(c\)](https://arxiv.org/html/2609.05885#S2.F2.sf3), low\-velocity modules exhibit a much sharper cumulative energy curve than high\-velocity modules, indicating that their learned update is dominated by a small number of directions\. In other words, despite having the same nominal rank budget, slow\-moving modules exploit only a fraction of the available capacity under uniform learning rate\.

The concentration is induced by the optimization process rather than being an inherent property of the downstream update\. Full fine\-tuning provides a reference for what an update at this rank can look like\. In a full fine\-tuning experiment on Commonsense170K with LLaMA2\-7B, the best rank\-32 approximation to the full fine\-tuning update has an effective rank of 27\.2 out of 32, whereas LoRA at the same rank learns an update with an effective rank of only 12\.7\. Concentration under uniform LR thus leaves rank capacity under\-used and limits how much of the update energy a LoRA adapter can capture, as discussed in Appendix[D\.1](https://arxiv.org/html/2609.05885#A4.SS1)\. The spectral concentration is the*handle*our intervention pulls on: by redistributing learning rate toward slower rank\-one components, AnLR\-LoRA aims to encourage broader use of the available rank capacity\. We verify in Sec\.[5](https://arxiv.org/html/2609.05885#S5)whether this translates into downstream task gains\.

## 4Methodology

Sec\.[3](https://arxiv.org/html/2609.05885#S3)shows that LoRA rank\-one components exhibit anisotropic training dynamics within each module, and that this anisotropy emerges early and persists throughout fine\-tuning\. We now formalize a learning\-rate scheme that responds to this anisotropy\. The proposed*adaptive anisotropic LR model*attaches a per\-rank\-one multipliersk\(t\)s\_\{k\}^\{\(t\)\}to the global learning rate within each adapter, redistributing the module\-level LR across therrrank\-one components\. AnLR\-LoRA instantiates this model by computingsk\(t\)s\_\{k\}^\{\(t\)\}online from the velocity and Adam SNR signals available during Adam optimization\.

### 4\.1The Uniform\-LR Convention

Standard LoRA fine\-tuning trainsθ=\{A,B\}\\theta=\\\{A,B\\\}for each adapter under AdamW with a single global learning rate scheduleηt\\eta\_\{t\}and decoupled weight\-decay coefficientλ\\lambda\. At steptt, AdamW maintains exponential moving averages of the gradient and its element\-wise square,

m\(t\)=β1​m\(t−1\)\+\(1−β1\)​g\(t\),m^\{\(t\)\}=\\beta\_\{1\}m^\{\(t\-1\)\}\+\(1\-\\beta\_\{1\}\)g^\{\(t\)\},v\(t\)=β2​v\(t−1\)\+\(1−β2\)​g\(t\)⊙g\(t\),v^\{\(t\)\}=\\beta\_\{2\}v^\{\(t\-1\)\}\+\(1\-\\beta\_\{2\}\)g^\{\(t\)\}\\odot g^\{\(t\)\},with bias\-corrected estimatesm^\(t\)=m\(t\)/\(1−β1t\)\\hat\{m\}^\{\(t\)\}=m^\{\(t\)\}/\(1\-\\beta\_\{1\}^\{t\}\)andv^\(t\)=v\(t\)/\(1−β2t\)\\hat\{v\}^\{\(t\)\}=v^\{\(t\)\}/\(1\-\\beta\_\{2\}^\{t\}\)\. The AdamW update direction is

u\(t\)=m^\(t\)v^\(t\)\+ε\+λ​θ\(t\),u^\{\(t\)\}\\;=\\;\\frac\{\\hat\{m\}^\{\(t\)\}\}\{\\sqrt\{\\hat\{v\}^\{\(t\)\}\}\+\\varepsilon\}\\;\+\\;\\lambda\\,\\theta^\{\(t\)\},\(1\)the sum of a preconditioned momentum term and a decoupled weight\-decay offset\. The parameter update is

θ\(t\+1\)=θ\(t\)−ηt​u\(t\)\.\\theta^\{\(t\+1\)\}\\;=\\;\\theta^\{\(t\)\}\-\\eta\_\{t\}\\,u^\{\(t\)\}\.\(2\)
In this formulation, the*per\-parameter effective learning rate*isηt,θi=ηt\\eta\_\{t,\\theta\_\{i\}\}=\\eta\_\{t\}for every parameterθi\\theta\_\{i\}in every rank\-one component of every adapter, because the only placeηt\\eta\_\{t\}enters is as the outer scalar multiplyingu\(t\)u^\{\(t\)\}\. We call this the*uniform\-LR convention*\. It assumes both*uniform granularity*, where allrrrank\-one components of an adapter share a single LR, and*uniform schedule shape*, where the same global LR scheduleηt\\eta\_\{t\}is applied to every parameter\.

### 4\.2Adaptive Anisotropic Learning\-rate

The analysis in Sec\.[3](https://arxiv.org/html/2609.05885#S3)motivates two changes to the uniform\-LR convention: adaptivity and granularity\.*Adaptivity*means that the learning rate should depend on the training\-time state, rather than being fixed by a global schedule alone\. This follows the broader principle of preconditioned optimization\([Kingma and Ba, 2014](https://arxiv.org/html/2609.05885#bib.bib49);[Duchi et al\., 2011](https://arxiv.org/html/2609.05885#bib.bib50);[You et al\., 2017](https://arxiv.org/html/2609.05885#bib.bib51);[You et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib52)\), but has not been explored at the level of individual LoRA rank\-one components\.*Granularity*refers to the unit of learning\-rate control, which should match the structure of the LoRA update\. Since the LoRA update decomposes intoΔW=BA=∑k=1rB:,kAk,:\\Delta W=BA=\\sum\_\{k=1\}^\{r\}B\_\{:,k\}A\_\{k,:\}, the rank\-one component is the finest structured unit of the LoRA parameterization and is where the empirical anisotropy emerges\. Per\-parameter\([Kingma and Ba, 2014](https://arxiv.org/html/2609.05885#bib.bib49);[Huang and Balestriero, 2024](https://arxiv.org/html/2609.05885#bib.bib48)\)and per\-matrix\([Hayou et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib26)\)granularities leave it unaddressed\.

We attach a positive, time\-varying multipliersk\(t\)s\_\{k\}^\{\(t\)\}to each rank\-one componentθk=\(Ak,:,B:,k\)\\theta\_\{k\}=\(A\_\{k,:\},\\,B\_\{:,k\}\)\. Letuk\(t\)u\_\{k\}^\{\(t\)\}denote the restriction of the AdamW direction in Eq\.[1](https://arxiv.org/html/2609.05885#S4.E1)to the parameters of componentkk\. The adaptive\-LR model uses a per\-component*effective learning rate*ηt,k=sk\(t\)​ηt\\eta\_\{t,k\}=s\_\{k\}^\{\(t\)\}\\eta\_\{t\}with the update

θk\(t\+1\)=θk\(t\)−sk\(t\)ηtuk\(t\),k=1,…,r,\\theta\_\{k\}^\{\(t\+1\)\}\\;=\\;\\theta\_\{k\}^\{\(t\)\}\-s\_\{k\}^\{\(t\)\}\\,\\eta\_\{t\}\\,u\_\{k\}^\{\(t\)\},\\quad k=1,\\dots,r,\(3\)which multipliesηt\\eta\_\{t\}bysk\(t\)s\_\{k\}^\{\(t\)\}in Eq\.[2](https://arxiv.org/html/2609.05885#S4.E2)and leaves AdamW’s state unchanged\. Guided by these observations, we impose two requirements on the multipliers\. First, because the anisotropy is within\-module, the multipliers should*redistribute*the module\-level learning rate rather than change its average scale\. We therefore enforce𝔼k​\[sk\(t\)\]=1\\mathbb\{E\}\_\{k\}\\\!\\big\[s\_\{k\}^\{\(t\)\}\\big\]\\;=\\;1within each module\. Second, because the rank ordering emerges during training, the multipliers should be computed online from training\-time state,sk\(t\)=ϕ⁡\(𝒮k\(t\)\)s\_\{k\}^\{\(t\)\}=\\phi\(\\mathcal\{S\}\_\{k\}^\{\(t\)\}\), without introducing extra trainable parameters or additional forward/backward passes\. Vanilla LoRA is recovered as the special casesk\(t\)=1s\_\{k\}^\{\(t\)\}=1, while coarser schemes such as LoRA\+ can be viewed as fixed, matrix\-level alternatives\. AnLR\-LoRA instantiates this model by specifying the state𝒮k\(t\)\\mathcal\{S\}\_\{k\}^\{\(t\)\}and the mapping functionϕ\\phiin the following section\.

### 4\.3AnLR\-LoRA: Instantiating the Model with Two Training\-Time Signals

AnLR\-LoRA instantiates the adaptive anisotropic LR model by defining the state𝒮k\(t\)\\mathcal\{S\}\_\{k\}^\{\(t\)\}with two complementary training\-time signals and specifying the mappingϕ\\phias a log\-additive and mean\-normalized function\.

#### What information should the multiplier use?

To decide whether to boost or dampen rank\-one componentkk,sk\(t\)s\_\{k\}^\{\(t\)\}should capture two aspects of its training dynamics: how actively the component is currently changing, and whether its change direction is reliable\. These two aspects are complementary\. A slow\-moving component may follow a consistent but under\-exploited direction, in which case increasing its LR is beneficial\. Alternatively, it may receive noisy gradients with no stable direction, in which case boosting it too much would amplify noise\. We therefore use velocity to measure component activity and Adam SNR to measure update reliability\.

#### Velocity: function\-space activity\.

Thekk\-th rank\-one component contributes the rank\-one matrixB:,kAk,:B\_\{:,k\}A\_\{k,:\}to the LoRA update\. We define its*velocity*as the Frobenius norm of the instantaneous change in this contribution,

νk=‖dd​t\(B:,kAk,:\)‖F,\\nu\_\{k\}\\;=\\;\\left\\\|\\frac\{d\}\{dt\}\\big\(B\_\{:,k\}A\_\{k,:\}\\big\)\\right\\\|\_\{F\},\(4\)approximated from the current LoRA gradients∇AL,∇BL\\nabla\_\{A\}L,\\nabla\_\{B\}Lvia the product rule:

νk\(t\)=‖\(∇BL\):,k\(t\)Ak,:\(t\)\+B:,k\(t\)\(∇AL\)k,:\(t\)‖F\.\\nu\_\{k\}^\{\(t\)\}\\;=\\;\\left\\\|\(\\nabla\_\{B\}L\)\_\{:,k\}^\{\(t\)\}\\,A\_\{k,:\}^\{\(t\)\}\\;\+\\;B\_\{:,k\}^\{\(t\)\}\\,\(\\nabla\_\{A\}L\)\_\{k,:\}^\{\(t\)\}\\right\\\|\_\{F\}\.\(5\)
To reduce step\-level noise, we maintain an EMA\-smoothed velocityν¯k\(t\)=ρ​ν¯k\(t−1\)\+\(1−ρ\)​νk\(t\)\\bar\{\\nu\}\_\{k\}^\{\(t\)\}=\\rho\\bar\{\\nu\}\_\{k\}^\{\(t\-1\)\}\+\(1\-\\rho\)\\nu\_\{k\}^\{\(t\)\}\. Velocity is defined on the*output*of the rank\-one component rather than on its parameters, so it captures the multiplicative coupling betweenAk,:A\_\{k,:\}andB:,kB\_\{:,k\}, which is not reflected by per\-parameter statistics alone\. It is computable from quantities the optimizer step already produces, without additional forward/backward passes\.

#### Adam SNR: parameter\-space confidence\.

Letm^\\hat\{m\}andv^\\hat\{v\}denote Adam’s bias\-corrected first and second moment estimates\. For each parameter, the element\-wise SNR is defined assnr⁡\(θi\)=\|m^i\|v^i\+ε\\mathrm\{snr\}\(\\theta\_\{i\}\)=\\frac\{\|\\hat\{m\}\_\{i\}\|\}\{\\sqrt\{\\hat\{v\}\_\{i\}\}\+\\varepsilon\}, whereε\\varepsilonis a small constant for numerical stability\. For rank componentkk, we aggregate the SNR values over the corresponding row ofAAand column ofBBand obtain

snrk=\(‖\|m^Ak,:\|v^Ak,:\+ε‖22\+‖\|m^B:,k\|v^B:,k\+ε‖22\)1/2\.\\mathrm\{snr\}\_\{k\}=\\left\(\\left\\\|\\frac\{\|\\hat\{m\}\_\{A\_\{k,:\}\}\|\}\{\\sqrt\{\\hat\{v\}\_\{A\_\{k,:\}\}\}\+\\varepsilon\}\\right\\\|\_\{2\}^\{2\}\+\\left\\\|\\frac\{\|\\hat\{m\}\_\{B\_\{:,k\}\}\|\}\{\\sqrt\{\\hat\{v\}\_\{B\_\{:,k\}\}\}\+\\varepsilon\}\\right\\\|\_\{2\}^\{2\}\\right\)^\{1/2\}\.\(6\)
Highsnrk\\mathrm\{snr\}\_\{k\}indicates that the gradient direction has been consistent across recent batches, identifying a stable update direction, while lowsnrk\\mathrm\{snr\}\_\{k\}indicates noisy and less reliable updates\. The SNR can be obtained directly from AdamW’s optimizer state and AnLR\-LoRA merely*promotes*this quantity from an implicit step modulator to an explicit per\-component meta\-signal\.

Table 1:Commonsense reasoning accuracy \(%\) with LLaMA2\-7B, LLaMA3\-8B and Qwen2\.5\-7B\.
#### Log\-additive fusion\.

The two signals are combined multiplicatively in the multiplier space\. Velocity suppresses the multiplier for fast\-moving components, while SNR increases it for components with reliable update directions\. This suggests an additive formulation in log space\. After per\-module mean\-normalization onν¯k\\bar\{\\nu\}\_\{k\}andsnrk\\mathrm\{snr\}\_\{k\}, we obtainν~k\\tilde\{\\nu\}\_\{k\}andsnr~k\\widetilde\{\\mathrm\{snr\}\}\_\{k\}, and the AnLR\-LoRA multiplier is defined as

log⁡sk\(t\)=α​log⁡1ν~k\(t\)\+β​log⁡snr~k\(t\)\.\\log s\_\{k\}^\{\(t\)\}\\;=\\;\\alpha\\,\\log\\tfrac\{1\}\{\\tilde\{\\nu\}\_\{k\}^\{\(t\)\}\}\\;\+\\;\\beta\\,\\log\\widetilde\{\\mathrm\{snr\}\}\_\{k\}^\{\(t\)\}\.\(7\)The velocity term boosts slow\-moving components and dampens fast\-moving ones, while the SNR term favors components with more reliable update directions\. For numerical stability, we clamp the log\-multiplier to\[−log⁡κ,log⁡κ\]\[\-\\log\\kappa,\\log\\kappa\]with a clamp boundκ\>1\\kappa\>1\. The resulting multipliers are then normalized within each module to satisfy𝔼k​\[ηt⋅sk\]=ηt\\mathbb\{E\}\_\{k\}\[\\eta\_\{t\}\\cdot s\_\{k\}\]=\\eta\_\{t\}\. In practice, we activate anisotropic scheduling after a short warmup period, during which all multipliers are set tosk\(t\)=1s\_\{k\}^\{\(t\)\}=1\. This avoids using unstable early\-step velocity and SNR estimates before the optimizer statistics become reliable\. Thus, AnLR\-LoRA redistributes the learning rate across rank components rather than amplifying the overall learning rate\.

### 4\.4Application of Per\-Rank\-One LR via Post\-Step Delta Scaling

We apply the effective per\-component learning ratesk\(t\)​ηts\_\{k\}^\{\(t\)\}\\eta\_\{t\}through post\-step delta scaling\. At each step, we first perform a standard AdamW update

θ^k\(t\+1\)=θk\(t\)−ηt​uk\(t\)\.\\hat\{\\theta\}\_\{k\}^\{\(t\+1\)\}=\\theta\_\{k\}^\{\(t\)\}\-\\eta\_\{t\}u\_\{k\}^\{\(t\)\}\.\(8\)We then rescale the resulting per\-component parameter delta:

θk\(t\+1\)=θk\(t\)\+sk\(t\)​\(θ^k\(t\+1\)−θk\(t\)\)=θk\(t\)−sk\(t\)​ηt​uk\(t\)\.\\theta\_\{k\}^\{\(t\+1\)\}=\\theta\_\{k\}^\{\(t\)\}\+s\_\{k\}^\{\(t\)\}\\bigl\(\\hat\{\\theta\}\_\{k\}^\{\(t\+1\)\}\-\\theta\_\{k\}^\{\(t\)\}\\bigr\)=\\theta\_\{k\}^\{\(t\)\}\-s\_\{k\}^\{\(t\)\}\\eta\_\{t\}u\_\{k\}^\{\(t\)\}\.\(9\)This update directly realizes the adaptive\-LR model in Eq\.[3](https://arxiv.org/html/2609.05885#S4.E3)\. For fixed AdamW directionuk\(t\)u\_\{k\}^\{\(t\)\}, it is equivalent to using the effective per\-component learning ratesk\(t\)​ηts\_\{k\}^\{\(t\)\}\\eta\_\{t\}\.

We use post\-step scaling for two reasons\. First,sk\(t\)s\_\{k\}^\{\(t\)\}is computed from signals of the current optimizer step and cannot be preset in advance\. Second, scaling the parameter delta avoids the cancellation that would occur if the gradient were scaled before AdamW normalization\. The implementation is lightweight\. It requires one parameter snapshot and one element\-wise rescaling per step, while leaving AdamW’s first\- and second\-moment states unchanged\.

Table 2:Evaluation results for dialogue, math and coding with LLaMA2\-7B\.Table 3:Results on visual instruction tuning with LLaVA\-1\.5\-7B\. FT results cited from[Liu et al\. \(2024b\)](https://arxiv.org/html/2609.05885#bib.bib1)\.![Refer to caption](https://arxiv.org/html/2609.05885v1/fig_efflr_heatmap_q_proj_L0.png)\(a\)Per\-rank\-one LR multiplierlog⁡sk\\log s\_\{k\}over training\.
\(b\)Singular spectrum concentration of the updateB​ABAat convergence\.\(c\)Per\-index spectral energy shift from LoRA to AnLR\-LoRA\.
Figure 3:Effect of AnLR\-LoRA’s anisotropic learning rate\.Figure 4:Visualization of per\-rank\-one velocity of AnLR\-LoRA\.

## 5Experiments

We evaluate AnLR\-LoRA across commonsense reasoning, math reasoning, code generation, dialogue and visual instruction tuning, spanning1818datasets across four backbones, including LLaMA2\-7B\([Touvron et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib7)\), LLaMA3\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib8)\), Qwen2\.5\-7B\([Qwen et al\., 2025](https://arxiv.org/html/2609.05885#bib.bib38)\)and LLaVA\-1\.5\-7B\([Liu et al\., 2024a](https://arxiv.org/html/2609.05885#bib.bib39)\)\. We compare AnLR\-LoRA against LoRA\([Hu et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib4)\)as well as the optimization\-related variants rsLoRA\([Kalajdzievski, 2023](https://arxiv.org/html/2609.05885#bib.bib17)\)and LoRA\+\([Hayou et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib26)\)\. All experiments run on NVIDIA H100 and H200 GPUs\. Full implementation details are in Appendix[B\.2](https://arxiv.org/html/2609.05885#A2.SS2)\.

### 5\.1Main Results

#### Commonsense Reasoning\.

We finetune three language backbones on Commonsense170K\([Hu et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib9)\)and evaluate on eight tasks: BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib10)\), PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2609.05885#bib.bib12)\), SIQA\([Sap et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib13)\), HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib11)\), WinoGrande\([Sakaguchi et al\., 2021](https://arxiv.org/html/2609.05885#bib.bib14)\), ARC\-e, ARC\-c\([Clark et al\., 2018](https://arxiv.org/html/2609.05885#bib.bib15)\)and OBQA\([Mihaylov et al\., 2018](https://arxiv.org/html/2609.05885#bib.bib16)\)\. As shown in Tab\.[1](https://arxiv.org/html/2609.05885#S4.T1), AnLR\-LoRA achieves the best average accuracy on all three backbones, outperforming standard LoRA by up to4\.3%4\.3\\%under the same parameter and learning\-rate budget\. It also consistently surpasses the optimization\-related baselines rsLoRA and LoRA\+ across models of different families and sizes, suggesting that the improvement comes from redistributing a fixed learning\-rate budget across rank\-one components rather than tuning a single global learning rate\.

#### Natural Language Generation\.

We further finetune LLaMA2\-7B on the 100K subset of WizardLM\([Xu et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib53)\), MetaMathQA\([Yu et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib27)\)and Code\-Feedback\([Zheng et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib28)\), and evaluate AnLR\-LoRA on dialogue with MT\-Bench\([Zheng et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib40)\), math reasoning with GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2609.05885#bib.bib29)\)and code generation with HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2609.05885#bib.bib30)\), respectively\. As shown in Tab\.[2](https://arxiv.org/html/2609.05885#S4.T2), AnLR\-LoRA improves over both LoRA and LoRA\+ on all three tasks, with the largest gain on HumanEval \(\+3\.66%\+3\.66\\%over LoRA\)\. The gains suggest that adaptive anisotropic learning\-rate also benefits open\-ended generation\.

#### Visual Instruction Tuning\.

We extend AnLR\-LoRA to the multimodal setting by finetuning LLaVA\-1\.5\-7B on the instruction tuning dataset\([Liu et al\., 2024a](https://arxiv.org/html/2609.05885#bib.bib39)\)and evaluating on seven vision\-language benchmarks\. As shown in Tab\.[3](https://arxiv.org/html/2609.05885#S4.T3), AnLR\-LoRA achieves the best average score, matching or surpassing LoRA on most benchmarks and obtaining the largest gain on VizWiz \(\+3\.07%\+3\.07\\%over LoRA\)\. The improvement remains consistent at a much larger rank \(r=128r=128\) and in the multimodal setting, showing that adaptive LR remains effective beyond standard language tasks\.

### 5\.2Ablation Studies and Analyses

AnLR\-LoRA encourages broader use of rank budget\.We analyze how AnLR\-LoRA changes the training dynamics of LoRA fine\-tuning\. Fig\.[3\(a\)](https://arxiv.org/html/2609.05885#S4.F3.sf1)shows that AnLR\-LoRA produces per\-rank\-one multipliers that vary across training steps, rather than assigning a fixed importance ordering\. A rank\-one component can be boosted or dampened at different stages of training, indicating that the method adapts to the current optimization state instead of imposing a static rank prior\. This dynamic redistribution encourages different rank\-one components to participate throughout training, rather than allowing a small subset of rank\-one components to consistently dominate the update \(Fig\.[4](https://arxiv.org/html/2609.05885#S4.F4)\)\.

Figure 5:Average commonsense reasoning accuracy on LLaMA2\-7B across varying LR\. AnLR\-LoRA exceeds LoRA at every LR, with the gap widening as LR moves away from the optimal range\.This anisotropic LR schedule also changes the structure of the learned update\. Compared to LoRA, which concentrates update energy in the leading singular directions, AnLR\-LoRA produces a flatter cumulative energy curve \(Fig\.[3\(b\)](https://arxiv.org/html/2609.05885#S4.F3.sf2)\) and redistributes spectral energy toward lower singular directions \(Fig\.[3\(c\)](https://arxiv.org/html/2609.05885#S4.F3.sf3)\)\. This indicates that AnLR\-LoRA reduces spectral concentration in the learned update and encourages broader use of the available rank capacity\.

Analysis on varying LR\.Fig\.[5](https://arxiv.org/html/2609.05885#S5.F5)examines whether the gains of AnLR\-LoRA can be explained by simply tuning the global learning rate\. We compare LoRA and AnLR\-LoRA on LLaMA2\-7B across a wide range of learning rates\. LoRA is highly sensitive to the choice of LR and its average accuracy varies by 16\.5 points and peaks at 77\.6\. Increasing or decreasing the global learning rate can improve some tasks but degrade others, and performance deteriorates when the learning rate moves away from a narrow favorable range \(Tab\.[10](https://arxiv.org/html/2609.05885#A4.T10)\)\. This indicates that simply changing the overall learning rate is not sufficient to resolve the uneven rank\-one learning dynamics observed in Sec\.[3](https://arxiv.org/html/2609.05885#S3)\.

AnLR\-LoRA consistently outperforms LoRA across varying learning rates\. The gain over LoRA widens as the learning rate moves away from the optimal range, which shows AnLR\-LoRA degrades much more gracefully under poorly tuned LR\. The robustness across LR settings highlights the advantage of adaptive anisotropic LR as it provides a finer\-grained optimization mechanism with gains that no single global LR can achieve\.

Table 4:Results of applying anisotropic learning rate to LoRA variants\.Table 5:Ablation of AnLR\-LoRA’s per\-rank\-one scaling signal\.Application to other LoRA variants\.We extend our evaluation of anisotropic LR to other LoRA variants in Tab\.[4](https://arxiv.org/html/2609.05885#S5.T4)\. We apply the same schedule to PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2609.05885#bib.bib2), MiLoRA[Wang et al\. \(2025a\)](https://arxiv.org/html/2609.05885#bib.bib3)and LoRA\-Dash[Si et al\. \(2025\)](https://arxiv.org/html/2609.05885#bib.bib6), covering variants that modify LoRA initialization or update behavior\. Anisotropic LR consistently improves all three methods on math reasoning and code generation tasks, suggesting that our approach is largely orthogonal to existing LoRA\-based methods and can provide complementary gains\.

#### Ablation on training\-time state signals\.

Tab\.[5](https://arxiv.org/html/2609.05885#S5.T5)ablates the signals used for adaptive LR scaling\. Velocity\-only scaling consistently improves over the uniform baseline, confirming that rank\-one update activity is informative for learning\-rate redistribution\. SNR scaling allocates more learning rate to components with reliable update directions\. Combining the two yields the best results on both GSM8K and HumanEval, suggesting that the two signals offer complementary guidance\. Appendix[D\.2](https://arxiv.org/html/2609.05885#A4.SS2)further verifies this mechanism, showing that the velocity term reduces the within\-module velocity imbalance and the SNR term prevents components with noisy update directions from being over\-amplified\. The full AnLR\-LoRA schedule therefore benefits from combining activity\-aware redistribution with confidence\-aware refinement\.

## 6Conclusion

This paper revisits the uniform\-LR convention in LoRA fine\-tuning and shows that it overlooks substantial heterogeneity in per\-rank\-one training dynamics within a single LoRA adapter\. We propose an adaptive anisotropic LR model that assigns each rank\-one component its own LR based on training\-time signals, and instantiate it as AnLR\-LoRA using function\-space velocity and Adam SNR to encourage learning on low\-velocity components with reliable update directions\. Experiments across language and multimodal settings demonstrate that AnLR\-LoRA improves LoRA adaptation and encourages broader use of rank capacity, highlighting anisotropic LR redistribution as a simple and effective optimization dimension\.

## Limitations

AnLR\-LoRA is one instantiation within a broader adaptive anisotropic LR framework, leaving alternative signal choices, fusion strategies and more sophisticated uncertainty measures for update direction as directions for future work\. Our rank\-utilization analysis verifies that the full fine\-tuning update spreads its energy across the rank budget, and characterizing how broadly this holds across tasks would further delineate when anisotropic LR is most beneficial\. The extension to broader scenarios, such as multilingual, long\-context and larger\-scale training, also remains to be explored\.

## Acknowledgements

This work was partially supported by the ARC DECRA Fellowship \(DE230101591\) and the ARC Discovery Project Grant \(DP260103379\) awarded to D\. Gong, as well as PhD scholarship support from UNSW and CSIRO Data61\.

## References

- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[item 3](https://arxiv.org/html/2609.05885#A2.I1.i3.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.CoRRabs/2107\.03374\.External Links:[Link](https://arxiv.org/abs/2107.03374),2107\.03374Cited by:[item 4](https://arxiv.org/html/2609.05885#A2.I2.i4.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px2.p1.1)\.
- Chenet al\.\(2026\)N\. Chen, S\. Villar, and S\. HayouLearning rate scaling across lora ranks and transfer to full finetuning\.arXiv preprint arXiv:2602\.06204\.Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p4.1),[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1)\.
- Chenet al\.\(2022\)S\. Chen, C\. Ge, Z\. Tong, J\. Wang, Y\. Song, J\. Wang, and P\. LuoAdaptformer: adapting vision transformers for scalable visual recognition\.Advances in Neural Information Processing Systems35,pp\. 16664–16678\.Cited by:[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Chenet al\.\(2025\)X\. Chen, K\. Feng, C\. Li, X\. Lai, X\. Yue, Y\. Yuan, and G\. WangFira: can we achieve full\-rank training of llms under low\-rank constraint?\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/aeae2c860cbe283ef73344c4ecd52567-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolq: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[item 2](https://arxiv.org/html/2609.05885#A2.I1.i2.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[item 7](https://arxiv.org/html/2609.05885#A2.I1.i7.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.CoRRabs/2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168),2110\.14168Cited by:[item 2](https://arxiv.org/html/2609.05885#A2.I2.i2.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px2.p1.1)\.
- Dinget al\.\(2023\)N\. Ding, X\. Lv, Q\. Wang, Y\. Chen, B\. Zhou, Z\. Liu, and M\. SunSparse low\-rank adaptation of pre\-trained language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6\-10, 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),pp\. 4133–4145\.External Links:[Link](https://doi.org/10.18653/v1/2023.emnlp-main.252),[Document](https://dx.doi.org/10.18653/V1/2023.EMNLP-MAIN.252)Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p4.1),[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Duchiet al\.\(2011\)J\. Duchi, E\. Hazan, and Y\. SingerAdaptive subgradient methods for online learning and stochastic optimization\.\.Journal of machine learning research12\(7\)\.Cited by:[§4\.2](https://arxiv.org/html/2609.05885#S4.SS2.p1.1)\.
- Goyalet al\.\(2017\)Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. ParikhMaking the v in vqa matter: elevating the role of image understanding in visual question answering\.External Links:1612\.00837,[Link](https://arxiv.org/abs/1612.00837)Cited by:[item 2](https://arxiv.org/html/2609.05885#A2.I3.i2.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang, A\. Mitra, A\. Sravankumar, A\. Korenev, A\. Hinsvark, A\. Rao, A\. Zhang, A\. Rodriguez, A\. Gregerson, A\. Spataru, B\. Roziere, B\. Biron, B\. Tang, B\. Chern, C\. Caucheteux, C\. Nayak, C\. Bi, C\. Marra, C\. McConnell, C\. Keller, C\. Touret, C\. Wu, C\. Wong, C\. C\. Ferrer, C\. Nikolaidis, D\. Allonsius, D\. Song, D\. Pintz, D\. Livshits, D\. Wyatt, D\. Esiobu, D\. Choudhary, D\. Mahajan, D\. Garcia\-Olano, D\. Perino, D\. Hupkes, E\. Lakomkin, E\. AlBadawy, E\. Lobanova, E\. Dinan, E\. M\. Smith, F\. Radenovic, F\. Guzmán, F\. Zhang, G\. Synnaeve, G\. Lee, G\. L\. Anderson, G\. Thattai, G\. Nail, G\. Mialon, G\. Pang, G\. Cucurell, H\. Nguyen, H\. Korevaar, H\. Xu, H\. Touvron, I\. Zarov, I\. A\. Ibarra, I\. Kloumann, I\. Misra, I\. Evtimov, J\. Zhang, J\. Copet, J\. Lee, J\. Geffert, J\. Vranes, J\. Park, J\. Mahadeokar, J\. Shah, J\. van der Linde, J\. Billock, J\. Hong, J\. Lee, J\. Fu, J\. Chi, J\. Huang, J\. Liu, J\. Wang, J\. Yu, J\. Bitton, J\. Spisak, J\. Park, J\. Rocca, J\. Johnstun, J\. Saxe, J\. Jia, K\. V\. Alwala, K\. Prasad, K\. Upasani, K\. Plawiak, K\. Li, K\. Heafield, K\. Stone, K\. El\-Arini, K\. Iyer, K\. Malik, K\. Chiu, K\. Bhalla, K\. Lakhotia, L\. Rantala\-Yeary, L\. van der Maaten, L\. Chen, L\. Tan, L\. Jenkins, L\. Martin, L\. Madaan, L\. Malo, L\. Blecher, L\. Landzaat, L\. de Oliveira, M\. Muzzi, M\. Pasupuleti, M\. Singh, M\. Paluri, M\. Kardas, M\. Tsimpoukelli, M\. Oldham, M\. Rita, M\. Pavlova, M\. Kambadur, M\. Lewis, M\. Si, M\. K\. Singh, M\. Hassan, N\. Goyal, N\. Torabi, N\. Bashlykov, N\. Bogoychev, N\. Chatterji, N\. Zhang, O\. Duchenne, O\. Çelebi, P\. Alrassy, P\. Zhang, P\. Li, P\. Vasic, P\. Weng, P\. Bhargava, P\. Dubal, P\. Krishnan, P\. S\. Koura, P\. Xu, Q\. He, Q\. Dong, R\. Srinivasan, R\. Ganapathy, R\. Calderer, R\. S\. Cabral, R\. Stojnic, R\. Raileanu, R\. Maheswari, R\. Girdhar, R\. Patel, R\. Sauvestre, R\. Polidoro, R\. Sumbaly, R\. Taylor, R\. Silva, R\. Hou, R\. Wang, S\. Hosseini, S\. Chennabasappa, S\. Singh, S\. Bell, S\. S\. Kim, S\. Edunov, S\. Nie, S\. Narang, S\. Raparthy, S\. Shen, S\. Wan, S\. Bhosale, S\. Zhang, S\. Vandenhende, S\. Batra, S\. Whitman, S\. Sootla, S\. Collot, S\. Gururangan, S\. Borodinsky, T\. Herman, T\. Fowler, T\. Sheasha, T\. Georgiou, T\. Scialom, T\. Speckbacher, T\. Mihaylov, T\. Xiao, U\. Karn, V\. Goswami, V\. Gupta, V\. Ramanathan, V\. Kerkez, V\. Gonguet, V\. Do, V\. Vogeti, V\. Albiero, V\. Petrovic, W\. Chu, W\. Xiong, W\. Fu, W\. Meers, X\. Martinet, X\. Wang, X\. Wang, X\. E\. Tan, X\. Xia, X\. Xie, X\. Jia, X\. Wang, Y\. Goldschlag, Y\. Gaur, Y\. Babaei, Y\. Wen, Y\. Song, Y\. Zhang, Y\. Li, Y\. Mao, Z\. D\. Coudert, Z\. Yan, Z\. Chen, Z\. Papakipos, A\. Singh, A\. Srivastava, A\. Jain, A\. Kelsey, A\. Shajnfeld, A\. Gangidi, A\. Victoria, A\. Goldstand, A\. Menon, A\. Sharma, A\. Boesenberg, A\. Baevski, A\. Feinstein, A\. Kallet, A\. Sangani, A\. Teo, A\. Yunus, A\. Lupu, A\. Alvarado, A\. Caples, A\. Gu, A\. Ho, A\. Poulton, A\. Ryan, A\. Ramchandani, A\. Dong, A\. Franco, A\. Goyal, A\. Saraf, A\. Chowdhury, A\. Gabriel, A\. Bharambe, A\. Eisenman, A\. Yazdan, B\. James, B\. Maurer, B\. Leonhardi, B\. Huang, B\. Loyd, B\. D\. Paola, B\. Paranjape, B\. Liu, B\. Wu, B\. Ni, B\. Hancock, B\. Wasti, B\. Spence, B\. Stojkovic, B\. Gamido, B\. Montalvo, C\. Parker, C\. Burton, C\. Mejia, C\. Liu, C\. Wang, C\. Kim, C\. Zhou, C\. Hu, C\. Chu, C\. Cai, C\. Tindal, C\. Feichtenhofer, C\. Gao, D\. Civin, D\. Beaty, D\. Kreymer, D\. Li, D\. Adkins, D\. Xu, D\. Testuggine, D\. David, D\. Parikh, D\. Liskovich, D\. Foss, D\. Wang, D\. Le, D\. Holland, E\. Dowling, E\. Jamil, E\. Montgomery, E\. Presani, E\. Hahn, E\. Wood, E\. Le, E\. Brinkman, E\. Arcaute, E\. Dunbar, E\. Smothers, F\. Sun, F\. Kreuk, F\. Tian, F\. Kokkinos, F\. Ozgenel, F\. Caggioni, F\. Kanayet, F\. Seide, G\. M\. Florez, G\. Schwarz, G\. Badeer, G\. Swee, G\. Halpern, G\. Herman, G\. Sizov, Guangyi, Zhang, G\. Lakshminarayanan, H\. Inan, H\. Shojanazeri, H\. Zou, H\. Wang, H\. Zha, H\. Habeeb, H\. Rudolph, H\. Suk, H\. Aspegren, H\. Goldman, H\. Zhan, I\. Damlaj, I\. Molybog, I\. Tufanov, I\. Leontiadis, I\. Veliche, I\. Gat, J\. Weissman, J\. Geboski, J\. Kohli, J\. Lam, J\. Asher, J\. Gaya, J\. Marcus, J\. Tang, J\. Chan, J\. Zhen, J\. Reizenstein, J\. Teboul, J\. Zhong, J\. Jin, J\. Yang, J\. Cummings, J\. Carvill, J\. Shepard, J\. McPhie, J\. Torres, J\. Ginsburg, J\. Wang, K\. Wu, K\. H\. U, K\. Saxena, K\. Khandelwal, K\. Zand, K\. Matosich, K\. Veeraraghavan, K\. Michelena, K\. Li, K\. Jagadeesh, K\. Huang, K\. Chawla, K\. Huang, L\. Chen, L\. Garg, L\. A, L\. Silva, L\. Bell, L\. Zhang, L\. Guo, L\. Yu, L\. Moshkovich, L\. Wehrstedt, M\. Khabsa, M\. Avalani, M\. Bhatt, M\. Mankus, M\. Hasson, M\. Lennie, M\. Reso, M\. Groshev, M\. Naumov, M\. Lathi, M\. Keneally, M\. Liu, M\. L\. Seltzer, M\. Valko, M\. Restrepo, M\. Patel, M\. Vyatskov, M\. Samvelyan, M\. Clark, M\. Macey, M\. Wang, M\. J\. Hermoso, M\. Metanat, M\. Rastegari, M\. Bansal, N\. Santhanam, N\. Parks, N\. White, N\. Bawa, N\. Singhal, N\. Egebo, N\. Usunier, N\. Mehta, N\. P\. Laptev, N\. Dong, N\. Cheng, O\. Chernoguz, O\. Hart, O\. Salpekar, O\. Kalinli, P\. Kent, P\. Parekh, P\. Saab, P\. Balaji, P\. Rittner, P\. Bontrager, P\. Roux, P\. Dollar, P\. Zvyagina, P\. Ratanchandani, P\. Yuvraj, Q\. Liang, R\. Alao, R\. Rodriguez, R\. Ayub, R\. Murthy, R\. Nayani, R\. Mitra, R\. Parthasarathy, R\. Li, R\. Hogan, R\. Battey, R\. Wang, R\. Howes, R\. Rinott, S\. Mehta, S\. Siby, S\. J\. Bondu, S\. Datta, S\. Chugh, S\. Hunt, S\. Dhillon, S\. Sidorov, S\. Pan, S\. Mahajan, S\. Verma, S\. Yamamoto, S\. Ramaswamy, S\. Lindsay, S\. Lindsay, S\. Feng, S\. Lin, S\. C\. Zha, S\. Patil, S\. Shankar, S\. Zhang, S\. Zhang, S\. Wang, S\. Agarwal, S\. Sajuyigbe, S\. Chintala, S\. Max, S\. Chen, S\. Kehoe, S\. Satterfield, S\. Govindaprasad, S\. Gupta, S\. Deng, S\. Cho, S\. Virk, S\. Subramanian, S\. Choudhury, S\. Goldman, T\. Remez, T\. Glaser, T\. Best, T\. Koehler, T\. Robinson, T\. Li, T\. Zhang, T\. Matthews, T\. Chou, T\. Shaked, V\. Vontimitta, V\. Ajayi, V\. Montanez, V\. Mohan, V\. S\. Kumar, V\. Mangla, V\. Ionescu, V\. Poenaru, V\. T\. Mihailescu, V\. Ivanov, W\. Li, W\. Wang, W\. Jiang, W\. Bouaziz, W\. Constable, X\. Tang, X\. Wu, X\. Wang, X\. Wu, X\. Gao, Y\. Kleinman, Y\. Chen, Y\. Hu, Y\. Jia, Y\. Qi, Y\. Li, Y\. Zhang, Y\. Zhang, Y\. Adi, Y\. Nam, Yu, Wang, Y\. Zhao, Y\. Hao, Y\. Qian, Y\. Li, Y\. He, Z\. Rait, Z\. DeVito, Z\. Rosnbrick, Z\. Wen, Z\. Yang, Z\. Zhao, and Z\. MaThe llama 3 herd of models\.External Links:2407\.21783,[Link](https://arxiv.org/abs/2407.21783)Cited by:[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Gurariet al\.\(2018\)D\. Gurari, Q\. Li, A\. J\. Stangl, A\. Guo, C\. Lin, K\. Grauman, J\. Luo, and J\. P\. BighamVizWiz grand challenge: answering visual questions from blind people\.External Links:1802\.08218,[Link](https://arxiv.org/abs/1802.08218)Cited by:[item 4](https://arxiv.org/html/2609.05885#A2.I3.i4.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuLoRA\+: efficient low rank adaptation of large models\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:[Link](https://openreview.net/forum?id=NEv8YqBROO)Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p4.1),[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1),[§4\.2](https://arxiv.org/html/2609.05885#S4.SS2.p1.1),[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1),[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Huet al\.\(2023\)Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. LeeLlm\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 5254–5276\.Cited by:[item 1](https://arxiv.org/html/2609.05885#A2.I1.i1.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Huang and Balestriero \(2024\)H\. Huang and R\. BalestrieroAllora: adaptive learning rate mitigates lora fatal flaws\.arXiv preprint arXiv:2410\.09692\.Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p4.1),[§4\.2](https://arxiv.org/html/2609.05885#S4.SS2.p1.1)\.
- Hudson and Manning \(2019\)D\. A\. Hudson and C\. D\. ManningGQA: a new dataset for real\-world visual reasoning and compositional question answering\.External Links:1902\.09506,[Link](https://arxiv.org/abs/1902.09506)Cited by:[item 3](https://arxiv.org/html/2609.05885#A2.I3.i3.p1.1)\.
- Jiaet al\.\(2022\)M\. Jia, L\. Tang, B\. Chen, C\. Cardie, S\. Belongie, B\. Hariharan, and S\. LimVisual prompt tuning\.InEuropean conference on computer vision,pp\. 709–727\.Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Kalajdzievski \(2023\)D\. KalajdzievskiA rank stabilization scaling factor for fine\-tuning with lora\.arXiv preprint arXiv:2312\.03732\.Cited by:[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1),[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Kingma and Ba \(2014\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§4\.2](https://arxiv.org/html/2609.05885#S4.SS2.p1.1)\.
- Leeet al\.\(2026\)Y\. Lee, C\. Ko, P\. Chen, and M\. YehLearning rate matters: vanilla lora may suffice for llm fine\-tuning\.arXiv preprint arXiv:2602\.04998\.Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.arXiv preprint arXiv:2101\.00190\.Cited by:[§1](https://arxiv.org/html/2609.05885#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Liet al\.\(2023\)Y\. Li, Y\. Du, K\. Zhou, J\. Wang, X\. Zhao, and J\. WenEvaluating object hallucination in large vision\-language models\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=xozJw0kZXF)Cited by:[item 7](https://arxiv.org/html/2609.05885#A2.I3.i7.p1.1)\.
- Liuet al\.\(2024a\)H\. Liu, C\. Li, Y\. Li, and Y\. J\. LeeImproved baselines with visual instruction tuning\.External Links:2310\.03744,[Link](https://arxiv.org/abs/2310.03744)Cited by:[item 1](https://arxiv.org/html/2609.05885#A2.I3.i1.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Liuet al\.\(2024b\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§D\.9](https://arxiv.org/html/2609.05885#A4.SS9.p1.1),[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1),[Table 3](https://arxiv.org/html/2609.05885#S4.T3)\.
- Liuet al\.\(2024c\)Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu,et al\.Mmbench: is your multi\-modal model an all\-around player?\.InEuropean conference on computer vision,pp\. 216–233\.Cited by:[item 8](https://arxiv.org/html/2609.05885#A2.I3.i8.p1.1)\.
- Luet al\.\(2024\)H\. Lu, C\. Zhao, J\. Xue, L\. Yao, K\. Moore, and D\. GongTake only what you need: rank minimization as an implicit forgetting regularizer in continual learning\.arXiv preprint arXiv:2412\.01004\.Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Luet al\.\(2026\)H\. Lu, C\. Zhao, J\. Xue, L\. Yao, K\. Moore, and D\. GongLittle by little: continual learning via incremental mixture of rank\-1 associative memory experts\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=P247k4ELcn)Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Luet al\.\(2022\)P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. KalyanLearn to explain: multimodal reasoning via thought chains for science question answering\.Advances in neural information processing systems35,pp\. 2507–2521\.Cited by:[item 5](https://arxiv.org/html/2609.05885#A2.I3.i5.p1.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPissa: principal singular values and singular vectors adaptation of large language models\.Advances in Neural Information Processing Systems37,pp\. 121038–121072\.Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1),[§5\.2](https://arxiv.org/html/2609.05885#S5.SS2.p5.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.arXiv preprint arXiv:1809\.02789\.Cited by:[item 8](https://arxiv.org/html/2609.05885#A2.I1.i8.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Qwenet al\.\(2025\)Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Roy and Vetterli \(2007\)O\. Roy and M\. VetterliThe effective rank: a measure of effective dimensionality\.In2007 15th European signal processing conference,pp\. 606–610\.Cited by:[§D\.1](https://arxiv.org/html/2609.05885#A4.SS1.SSS0.Px1.p1.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[item 6](https://arxiv.org/html/2609.05885#A2.I1.i6.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. LeBras, and Y\. ChoiSocialiqa: commonsense reasoning about social interactions\.arXiv preprint arXiv:1904\.09728\.Cited by:[item 4](https://arxiv.org/html/2609.05885#A2.I1.i4.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Siet al\.\(2025\)C\. Si, Z\. Shi, S\. Zhang, X\. Yang, H\. Pfister, and W\. ShenUnleashing the power of task\-specific directions in parameter efficient fine\-tuning\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=RYrJqz44p4)Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1),[§5\.2](https://arxiv.org/html/2609.05885#S5.SS2.p5.1)\.
- Singhet al\.\(2019\)A\. Singh, V\. Natarajan, M\. Shah, Y\. Jiang, X\. Chen, D\. Batra, D\. Parikh, and M\. RohrbachTowards vqa models that can read\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[item 6](https://arxiv.org/html/2609.05885#A2.I3.i6.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§5](https://arxiv.org/html/2609.05885#S5.p1.1)\.
- Valipouret al\.\(2023\)M\. Valipour, M\. Rezagholizadeh, I\. Kobyzev, and A\. GhodsiDyLoRA: parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2\-6, 2023,A\. Vlachos and I\. Augenstein \(Eds\.\),pp\. 3266–3279\.External Links:[Link](https://doi.org/10.18653/v1/2023.eacl-main.239),[Document](https://dx.doi.org/10.18653/V1/2023.EACL-MAIN.239)Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Wanget al\.\(2025a\)H\. Wang, Y\. Li, S\. Wang, G\. Chen, and Y\. ChenMiLoRA: harnessing minor singular components for parameter\-efficient LLM finetuning\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4823–4836\.External Links:[Link](https://aclanthology.org/2025.naacl-long.248/),ISBN 979\-8\-89176\-189\-6Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1),[§5\.2](https://arxiv.org/html/2609.05885#S5.SS2.p5.1)\.
- Wanget al\.\(2025b\)H\. Wang, H\. Lu, L\. Yao, and D\. GongSelf\-expansion of pre\-trained models with mixture of adapters for continual learning\.In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 10087–10098\.Cited by:[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Wanget al\.\(2024\)S\. Wang, L\. Yu, and J\. LiLora\-ga: low\-rank adaptation with gradient approximation\.Advances in Neural Information Processing Systems37,pp\. 54905–54931\.Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Wanget al\.\(2025c\)Z\. Wang, J\. Liang, R\. He, Z\. Wang, and T\. TanLoRA\-pro: are low\-rank adapters properly optimized?\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=gTwRMU3lJ5)Cited by:[§D\.9](https://arxiv.org/html/2609.05885#A4.SS9.p1.1),[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1)\.
- Xuet al\.\(2024\)C\. Xu, Q\. Sun, K\. Zheng, X\. Geng, P\. Zhao, J\. Feng, C\. Tao, Q\. Lin, and D\. JiangWizardLM: empowering large pre\-trained language models to follow complex instructions\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=CfXh93NDgH)Cited by:[item 5](https://arxiv.org/html/2609.05885#A2.I2.i5.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px2.p1.1)\.
- Youet al\.\(2017\)Y\. You, I\. Gitman, and B\. GinsburgLarge batch training of convolutional networks\.External Links:1708\.03888,[Link](https://arxiv.org/abs/1708.03888)Cited by:[§4\.2](https://arxiv.org/html/2609.05885#S4.SS2.p1.1)\.
- Youet al\.\(2019\)Y\. You, J\. Li, S\. Reddi, J\. Hseu, S\. Kumar, S\. Bhojanapalli, X\. Song, J\. Demmel, K\. Keutzer, and C\. HsiehLarge batch optimization for deep learning: training bert in 76 minutes\.arXiv preprint arXiv:1904\.00962\.Cited by:[§4\.2](https://arxiv.org/html/2609.05885#S4.SS2.p1.1)\.
- Yuet al\.\(2024\)L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetaMath: bootstrap your own mathematical questions for large language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=N8N0hgNDRt)Cited by:[item 1](https://arxiv.org/html/2609.05885#A2.I2.i1.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px2.p1.1)\.
- Yuet al\.\(2025\)X\. Yu, Y\. Wang, J\. Chen, and L\. XueAltLoRA: towards better gradient approximation in low\-rank adaptation with alternating projections\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/35c3db8db1b962538b0958af56ba329b-Abstract-Conference.html)Cited by:[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaswag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[item 5](https://arxiv.org/html/2609.05885#A2.I1.i5.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2023\)Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdaptive budget allocation for parameter\-efficient fine\-tuning\.InThe Eleventh International Conference on Learning Representations,Cited by:[§D\.4](https://arxiv.org/html/2609.05885#A4.SS4.p1.1),[§D\.9](https://arxiv.org/html/2609.05885#A4.SS9.p1.1),[§1](https://arxiv.org/html/2609.05885#S1.p4.1),[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Zhanget al\.\(2025a\)Y\. Zhang, B\. Li, and G\. B\. GiannakisRefLoRA: refactored low\-rank adaptation for efficient fine\-tuning of large models\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/85c1ee0b3cdf49b84f14b163b01f2711-Abstract-Conference.html)Cited by:[§2\.3](https://arxiv.org/html/2609.05885#S2.SS3.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, H\. Li, Y\. Zhang, G\. Gong, J\. Wang, J\. Hu, P\. Liu, and Q\. JiangThe primacy of magnitude in low\-rank adaptation\.InAdvances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2\-7, 2025 / Mexico City, Mexico, November 30 \- December 5, 2025,D\. Belgrave, C\. Zhang, L\. N\. Montoya, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, N\. Chen, I\. V\. M\. Ruíz, and A\. Loaiza\-Bonilla \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2025/hash/0010665e949927b74faf6e3ada6d7f72-Abstract-Conference.html)Cited by:[§2\.2](https://arxiv.org/html/2609.05885#S2.SS2.p1.1)\.
- Zhaoet al\.\(2026\)C\. Zhao, M\. Li, H\. Lu, and D\. GongOn token’s dilemma: dynamic moe with drift\-aware token assignment for continual learning of large vision language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 3941–3952\.Cited by:[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Zhaoet al\.\(2024\)J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. TianGaLore: memory\-efficient llm training by gradient low\-rank projection\.External Links:2403\.03507Cited by:[§2\.1](https://arxiv.org/html/2609.05885#S2.SS1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging llm\-as\-a\-judge with mt\-bench and chatbot arena\.External Links:2306\.05685,[Link](https://arxiv.org/abs/2306.05685)Cited by:[item 6](https://arxiv.org/html/2609.05885#A2.I2.i6.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpenCodeInterpreter: integrating code generation with execution and refinement\.InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11\-16, 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Findings of ACL, Vol\.ACL 2024,pp\. 12834–12859\.External Links:[Link](https://doi.org/10.18653/v1/2024.findings-acl.762),[Document](https://dx.doi.org/10.18653/V1/2024.FINDINGS-ACL.762)Cited by:[item 3](https://arxiv.org/html/2609.05885#A2.I2.i3.p1.1),[§5\.1](https://arxiv.org/html/2609.05885#S5.SS1.SSS0.Px2.p1.1)\.

## Appendix AMore Details about AnLR\-LoRA

### A\.1Pre\- and Post\-Step Scaling under AdamW

This section states and proves Propositions[1](https://arxiv.org/html/2609.05885#Thmproposition1)and[2](https://arxiv.org/html/2609.05885#Thmproposition2), which together delimit the design space for per\-rank\-one effective learning rates under AdamW\. Proposition[1](https://arxiv.org/html/2609.05885#Thmproposition1)shows that pre\-step gradient scaling is ineffective for this purpose\. Proposition[2](https://arxiv.org/html/2609.05885#Thmproposition2)shows that post\-step delta scaling is equivalent to direct outer\-rate substitution\.

#### Setup\.

AdamW’s update direction restricted to thekk\-th rank\-one component is

uk\(t\)=m^k\(t\)v^k\(t\)\+ε\+λ​θk\(t\),u\_\{k\}^\{\(t\)\}=\\frac\{\\hat\{m\}\_\{k\}^\{\(t\)\}\}\{\\sqrt\{\\hat\{v\}\_\{k\}^\{\(t\)\}\}\+\\varepsilon\}\+\\lambda\\theta\_\{k\}^\{\(t\)\},\(10\)and the post\-step delta\-scaled update is

θk\(t\+1\)=θk\(t\)\+sk\(t\)​\(θ^k\(t\+1\)−θk\(t\)\),\\theta\_\{k\}^\{\(t\+1\)\}=\\theta\_\{k\}^\{\(t\)\}\+s\_\{k\}^\{\(t\)\}\\bigl\(\\hat\{\\theta\}\_\{k\}^\{\(t\+1\)\}\-\\theta\_\{k\}^\{\(t\)\}\\bigr\),\(11\)withθ^k\(t\+1\)=θk\(t\)−ηt​uk\(t\)\\hat\{\\theta\}\_\{k\}^\{\(t\+1\)\}=\\theta\_\{k\}^\{\(t\)\}\-\\eta\_\{t\}u\_\{k\}^\{\(t\)\}\.

###### Proposition 1\(Pre\-step gradient scaling\)\.

Let\{θ\(t\)\}\\\{\\theta^\{\(t\)\}\\\}be the AdamW trajectory under gradient sequence\{g\(t\)\}\\\{g^\{\(t\)\}\\\}with learning rateηt\\eta\_\{t\}, moment decaysβ1,β2\\beta\_\{1\},\\beta\_\{2\}, decoupled weight decayλ\\lambdaand stability constantε\\varepsilon\. Replacing the gradient byc⋅g\(t\)c\\cdot g^\{\(t\)\}at every step, for any fixedc\>0c\>0, produces a trajectory equal to the unscaled one whenε=0\\varepsilon=0, and differs by anO⁡\(ε\)O\(\\varepsilon\)perturbation otherwise\.

###### Proof\.

Consider two AdamW trajectories initialized identically atθ\(0\)\\theta^\{\(0\)\}, withm\(0\)=v\(0\)=0m^\{\(0\)\}=v^\{\(0\)\}=0and shared\(ηt,λ,β1,β2,ε\)\(\\eta\_\{t\},\\lambda,\\beta\_\{1\},\\beta\_\{2\},\\varepsilon\):

- •Run A driven by\{g\(t\)\}t≥1\\\{g^\{\(t\)\}\\\}\_\{t\\geq 1\}\.
- •Run B driven by\{c​g\(t\)\}t≥1\\\{c\\,g^\{\(t\)\}\\\}\_\{t\\geq 1\}, fixedc\>0c\>0\.

#### Step 1: moment scaling\.

We show by induction that

mB\(t\)=c​mA\(t\),vB\(t\)=c2​vA\(t\)\.m\_\{B\}^\{\(t\)\}=c\\,m\_\{A\}^\{\(t\)\},\\quad v\_\{B\}^\{\(t\)\}=c^\{2\}v\_\{A\}^\{\(t\)\}\.\(12\)The base case \(t=0t=0\) is trivial\. For the inductive step,

mB\(t\)\\displaystyle m\_\{B\}^\{\(t\)\}=β1​mB\(t−1\)\+\(1−β1\)​gB\(t\)\\displaystyle=\\beta\_\{1\}m\_\{B\}^\{\(t\-1\)\}\+\(1\-\\beta\_\{1\}\)g\_\{B\}^\{\(t\)\}=c⁡\[β1​mA\(t−1\)\+\(1−β1\)​g\(t\)\]\\displaystyle=c\\bigl\[\\beta\_\{1\}m\_\{A\}^\{\(t\-1\)\}\+\(1\-\\beta\_\{1\}\)g^\{\(t\)\}\\bigr\]=c​mA\(t\),\\displaystyle=c\\,m\_\{A\}^\{\(t\)\},and similarlyvB\(t\)=c2​vA\(t\)v\_\{B\}^\{\(t\)\}=c^\{2\}v\_\{A\}^\{\(t\)\}\. Bias correction applies symmetrically, som^B\(t\)=c​m^A\(t\)\\hat\{m\}\_\{B\}^\{\(t\)\}=c\\,\\hat\{m\}\_\{A\}^\{\(t\)\}andv^B\(t\)=c2​v^A\(t\)\\hat\{v\}\_\{B\}^\{\(t\)\}=c^\{2\}\\hat\{v\}\_\{A\}^\{\(t\)\}\.

#### Step 2: update direction\.

Letσt:=v^A\(t\)\\sigma\_\{t\}:=\\sqrt\{\\hat\{v\}\_\{A\}^\{\(t\)\}\}for brevity\. The AdamW direction in Run B is

uB\(t\)\\displaystyle u\_\{B\}^\{\(t\)\}=m^B\(t\)v^B\(t\)\+ε\+λ​θB\(t\)\\displaystyle=\\frac\{\\hat\{m\}\_\{B\}^\{\(t\)\}\}\{\\sqrt\{\\hat\{v\}\_\{B\}^\{\(t\)\}\}\+\\varepsilon\}\+\\lambda\\theta\_\{B\}^\{\(t\)\}=c​m^A\(t\)c​σt\+ε\+λ​θB\(t\)\\displaystyle=\\frac\{c\\,\\hat\{m\}\_\{A\}^\{\(t\)\}\}\{c\\,\\sigma\_\{t\}\+\\varepsilon\}\+\\lambda\\theta\_\{B\}^\{\(t\)\}=m^A\(t\)σt\+ε/c\+λ​θB\(t\)\.\\displaystyle=\\frac\{\\hat\{m\}\_\{A\}^\{\(t\)\}\}\{\\sigma\_\{t\}\+\\varepsilon/c\}\+\\lambda\\theta\_\{B\}^\{\(t\)\}\.
Caseε=0\\varepsilon=0\.The factorcccancels, givinguB\(t\)=uA\(t\)u\_\{B\}^\{\(t\)\}=u\_\{A\}^\{\(t\)\}wheneverθB\(t\)=θA\(t\)\\theta\_\{B\}^\{\(t\)\}=\\theta\_\{A\}^\{\(t\)\}\. The parameter update preserves the equality, so induction yieldsθB\(t\)=θA\(t\)\\theta\_\{B\}^\{\(t\)\}=\\theta\_\{A\}^\{\(t\)\}for allt≥0t\\geq 0\.

Caseε\>0\\varepsilon\>0\.uB\(t\)u\_\{B\}^\{\(t\)\}differs fromuA\(t\)u\_\{A\}^\{\(t\)\}only by the substitutionε↦ε/c\\varepsilon\\mapsto\\varepsilon/c:

uB\(t\)−uA\(t\)=m^A\(t\)​ε​\(1−1/c\)\(σt\+ε/c\)​\(σt\+ε\),u\_\{B\}^\{\(t\)\}\-u\_\{A\}^\{\(t\)\}=\\frac\{\\hat\{m\}\_\{A\}^\{\(t\)\}\\,\\varepsilon\\,\(1\-1/c\)\}\{\(\\sigma\_\{t\}\+\\varepsilon/c\)\(\\sigma\_\{t\}\+\\varepsilon\)\},which isO⁡\(ε\)O\(\\varepsilon\)and negligible wheneverσt≫ε\\sigma\_\{t\}\\gg\\varepsilon, the operating regime of AdamW\.

#### Conclusion\.

Pre\-step gradient scaling byccyields a trajectory equal to the unscaled one up toO⁡\(ε\)O\(\\varepsilon\)\. A constant factorsks\_\{k\}applied togk\(t\)g\_\{k\}^\{\(t\)\}is therefore cancelled by Adam’s preconditioner\. A time\-varyingsk\(t\)s\_\{k\}^\{\(t\)\}is not exactly cancelled, but it acts by perturbing the moment estimates rather than by rescaling the step, so it does not install a per\-component effective learning rate either\. ∎

###### Proposition 2\(Post\-step delta scaling\)\.

Letθk\(t\+1\)\\theta^\{\(t\+1\)\}\_\{k\}be defined by Eq\.[11](https://arxiv.org/html/2609.05885#A1.E11), whereθ^k\(t\+1\)\\hat\{\\theta\}^\{\(t\+1\)\}\_\{k\}is one AdamW step onθk\(t\)\\theta^\{\(t\)\}\_\{k\}with learning rateηt\\eta\_\{t\}and decoupled weight decayλ\\lambda\. The resulting trajectory\{θk\(t\)\}\\\{\\theta^\{\(t\)\}\_\{k\}\\\}is identical to the trajectory obtained by running AdamW on the same gradient sequence with outer learning ratesk\(t\)​ηts\_\{k\}^\{\(t\)\}\\eta\_\{t\}and unchanged decoupled weight decayλ\\lambda\.

###### Proof\.

Substituting the AdamW step into Eq\.[11](https://arxiv.org/html/2609.05885#A1.E11):

θk\(t\+1\)\\displaystyle\\theta\_\{k\}^\{\(t\+1\)\}=θk\(t\)\+sk\(t\)​\(θ^k\(t\+1\)−θk\(t\)\)\\displaystyle=\\theta\_\{k\}^\{\(t\)\}\+s\_\{k\}^\{\(t\)\}\\bigl\(\\hat\{\\theta\}\_\{k\}^\{\(t\+1\)\}\-\\theta\_\{k\}^\{\(t\)\}\\bigr\)=θk\(t\)\+sk\(t\)​\(−ηt​uk\(t\)\)\\displaystyle=\\theta\_\{k\}^\{\(t\)\}\+s\_\{k\}^\{\(t\)\}\\bigl\(\-\\eta\_\{t\}u\_\{k\}^\{\(t\)\}\\bigr\)=θk\(t\)−\(sk\(t\)​ηt\)​uk\(t\)\.\\displaystyle=\\theta\_\{k\}^\{\(t\)\}\-\\bigl\(s\_\{k\}^\{\(t\)\}\\eta\_\{t\}\\bigr\)\\,u\_\{k\}^\{\(t\)\}\.This is the AdamW update with outer learning rateηt\\eta\_\{t\}replaced bysk\(t\)​ηts\_\{k\}^\{\(t\)\}\\eta\_\{t\}, leavingm^k\(t\)\\hat\{m\}\_\{k\}^\{\(t\)\},v^k\(t\)\\hat\{v\}\_\{k\}^\{\(t\)\}andλ\\lambdaunchanged\. Expanding via Eq\.[10](https://arxiv.org/html/2609.05885#A1.E10):

θk\(t\+1\)\\displaystyle\\theta\_\{k\}^\{\(t\+1\)\}=θk\(t\)−\(sk\(t\)​ηt\)​m^k\(t\)v^k\(t\)\+ε\\displaystyle=\\theta\_\{k\}^\{\(t\)\}\-\\bigl\(s\_\{k\}^\{\(t\)\}\\eta\_\{t\}\\bigr\)\\frac\{\\hat\{m\}\_\{k\}^\{\(t\)\}\}\{\\sqrt\{\\hat\{v\}\_\{k\}^\{\(t\)\}\}\+\\varepsilon\}−\(sk\(t\)​ηt​λ\)​θk\(t\),\\displaystyle\\quad\-\\bigl\(s\_\{k\}^\{\(t\)\}\\eta\_\{t\}\\lambda\\bigr\)\\,\\theta\_\{k\}^\{\(t\)\},which matches a standard AdamW update with outer rateηt′=sk\(t\)​ηt\\eta^\{\\prime\}\_\{t\}=s\_\{k\}^\{\(t\)\}\\eta\_\{t\}and shrinkage coefficientηt′​λ\\eta^\{\\prime\}\_\{t\}\\lambdaonθk\(t\)\\theta\_\{k\}^\{\(t\)\}\. The moment recurrences formkm\_\{k\}andvkv\_\{k\}depend only on the gradient sequence and\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\), not on the outer rate, so they are unchanged by the substitution\. Therefore, post\-step delta scaling at every step produces the same trajectory as running AdamW on the same gradient sequence with outer learning ratesk\(t\)​ηts\_\{k\}^\{\(t\)\}\\eta\_\{t\}and unchanged decoupled weight decayλ\\lambda\. ∎

#### Remarks\.

Together, the two propositions delimit the admissible implementations of per\-rank\-one effective learning rates under AdamW: pre\-step gradient scaling is ineffective for this purpose \(Proposition[1](https://arxiv.org/html/2609.05885#Thmproposition1)\), while post\-step delta scaling and direct per\-component outer\-rate substitution are equivalent \(Proposition[2](https://arxiv.org/html/2609.05885#Thmproposition2)\)\. AnLR\-LoRA uses the post\-step delta form because it is a drop\-in modification of the AdamW step output that requires no change to the optimizer’s internal state or update rule\.

### A\.2Training Procedure of AnLR\-LoRA

Alg\.[1](https://arxiv.org/html/2609.05885#alg1)summarizes the training procedure of AnLR\-LoRA\. At each step, we compute per\-rank\-one velocity from the loss gradients onAAandBBand per\-rank\-one Adam SNR from Adam’s bias\-corrected moment estimates\. After warmup, the two normalized signals are fused into per\-rank\-one multipliers\. A standard AdamW step then produces provisional LoRA parameters, and the multipliers are applied through post\-step delta scaling on the resulting updates\. This implementation leaves the AdamW optimizer state unchanged and introduces no additional forward or backward passes\.

Algorithm 1AnLR\-LoRA1:LoRA parameters

\{A,B\}\\\{A,B\\\}, AdamW optimizer, LR schedule

ηt\\eta\_\{t\}, warmup

TwarmT\_\{\\mathrm\{warm\}\}, EMA coefficient

ρ\\rho, fusion weights

α,β\\alpha,\\beta, clamp factor

κ\\kappa
2:Initialize velocity EMA

ν¯k←0\\bar\{\\nu\}\_\{k\}\\leftarrow 0for each rank\-one component

3:foreach training step

ttdo

4:Save a snapshot of current LoRA parameters

θ\(t\)=\{A\(t\),B\(t\)\}\\theta^\{\(t\)\}=\\\{A^\{\(t\)\},B^\{\(t\)\}\\\}
5:Take one standard AdamW step to obtain provisional parameters

θ^\(t\+1\)\\hat\{\\theta\}^\{\(t\+1\)\}
6:foreach LoRA moduledo

7:Compute per\-rank velocity

νk\(t\)\\nu\_\{k\}^\{\(t\)\}using Eq\.[5](https://arxiv.org/html/2609.05885#S4.E5)

8:Update EMA velocity:

ν¯k\(t\)←ρ​ν¯k\(t−1\)\+\(1−ρ\)​νk\(t\)\\bar\{\\nu\}\_\{k\}^\{\(t\)\}\\leftarrow\\rho\\bar\{\\nu\}\_\{k\}^\{\(t\-1\)\}\+\(1\-\\rho\)\\nu\_\{k\}^\{\(t\)\}
9:Compute per\-rank Adam SNR

snrk\(t\)\\mathrm\{snr\}\_\{k\}^\{\(t\)\}using Eq\.[6](https://arxiv.org/html/2609.05885#S4.E6)

10:Normalize

ν¯k\(t\)\\bar\{\\nu\}\_\{k\}^\{\(t\)\}and

snrk\(t\)\\mathrm\{snr\}\_\{k\}^\{\(t\)\}within the module

11:if

t\>Twarmt\>T\_\{\\mathrm\{warm\}\}then

12:Compute

log⁡sk\(t\)←α​log⁡\(1/ν~k\(t\)\)\+β​log⁡snr~k\(t\)\\log s\_\{k\}^\{\(t\)\}\\leftarrow\\alpha\\log\(1/\\tilde\{\\nu\}\_\{k\}^\{\(t\)\}\)\+\\beta\\log\\widetilde\{\\mathrm\{snr\}\}\_\{k\}^\{\(t\)\}
13:Clamp

log⁡sk\(t\)\\log s\_\{k\}^\{\(t\)\}to

\[−log⁡κ,log⁡κ\]\[\-\\log\\kappa,\\log\\kappa\]
14:Set

sk\(t\)←exp⁡\(log⁡sk\(t\)\)s\_\{k\}^\{\(t\)\}\\leftarrow\\exp\(\\log s\_\{k\}^\{\(t\)\}\)and mean\-normalize across ranks

15:Apply post\-step delta scaling:

θk\(t\+1\)←θk\(t\)\+sk\(t\)​\(θ^k\(t\+1\)−θk\(t\)\)\\theta\_\{k\}^\{\(t\+1\)\}\\leftarrow\\theta\_\{k\}^\{\(t\)\}\+s\_\{k\}^\{\(t\)\}\(\\hat\{\\theta\}\_\{k\}^\{\(t\+1\)\}\-\\theta\_\{k\}^\{\(t\)\}\)

## Appendix BExperimental Setups

### B\.1Datasets

We describe the datasets used in our experiments on commonsense reasoning, generative tasks and visual instruction tuning\.

Commonsense Reasoning\.

1. 1\.Commonsense170K\([Hu et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib9)\)is a large\-scale instruction tuning dataset containing approximately 170K commonsense reasoning examples collected from multiple benchmarks\.
2. 2\.BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib10)\)\(3\.2K test set\) is a yes/no question answering benchmark built from naturally occurring Google search queries paired with Wikipedia passages\.
3. 3\.PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2609.05885#bib.bib12)\)\(3K test set\) evaluates physical commonsense reasoning by asking models to select the more plausible solution to an everyday task\.
4. 4\.SIQA\([Sap et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib13)\)\(1\.95K test set\) measures commonsense reasoning about social situations and human interactions\.
5. 5\.HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib11)\)\(10K test set\) is a commonsense completion benchmark requiring models to select the most plausible continuation from multiple candidates\.
6. 6\.WinoGrande\([Sakaguchi et al\., 2021](https://arxiv.org/html/2609.05885#bib.bib14)\)\(1\.7K test set\) evaluates commonsense reasoning through pronoun resolution problems with ambiguous references\.
7. 7\.ARC\-e\(2\.3K test set\) andARC\-c\(1\.1K test set\)\([Clark et al\., 2018](https://arxiv.org/html/2609.05885#bib.bib15)\)are science question answering benchmarks containing elementary\-level and challenge\-level multiple\-choice questions, respectively\.
8. 8\.OBQA\([Mihaylov et al\., 2018](https://arxiv.org/html/2609.05885#bib.bib16)\)\(500 test set\) is an open\-book question answering benchmark requiring external commonsense and science knowledge\.

Natural Language Generation\.

1. 1\.MetaMathQA\([Yu et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib27)\)is a large\-scale mathematical instruction tuning dataset\. Following prior work, we use its 100K subset for training in math reasoning experiments\.
2. 2\.GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2609.05885#bib.bib29)\)\(1\.3K test set\) is a benchmark of grade\-school mathematical reasoning problems\.
3. 3\.Code\-Feedback\([Zheng et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib28)\)contains code solutions paired with natural language feedback\. We use its 100K subset for code generation training\.
4. 4\.HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2609.05885#bib.bib30)\)\(164 test set\) evaluates code generation through function synthesis problems with unit\-test\-based execution evaluation\.
5. 5\.WizardLM\([Xu et al\., 2024](https://arxiv.org/html/2609.05885#bib.bib53)\)is an instruction tuning dataset generated with Evol\-Instruct, which progressively rewrites seed instructions into more complex and diverse training examples\.
6. 6\.MT\-Bench\([Zheng et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib40)\)\(80 test set\) is a multi\-turn dialogue benchmark evaluated using LLM\-based judgment\.

Vision\-Language Benchmarks\.

1. 1\.LLaVA\-1\.5 Instruction Tuning Dataset\([Liu et al\., 2024a](https://arxiv.org/html/2609.05885#bib.bib39)\)contains approximately 665K multimodal instruction\-following examples used for visual instruction tuning\.
2. 2\.VQAv2\{\}^\{\\text\{v2\}\}\([Goyal et al\., 2017](https://arxiv.org/html/2609.05885#bib.bib41)\)\(10\.7K test set\) is a visual question answering benchmark containing open\-ended questions about natural images\.
3. 3\.GQA\([Hudson and Manning, 2019](https://arxiv.org/html/2609.05885#bib.bib42)\)\(12\.5K test set\) evaluates compositional reasoning and relational understanding in visual question answering\.
4. 4\.VizWiz\([Gurari et al\., 2018](https://arxiv.org/html/2609.05885#bib.bib43)\)\(8K test set\) is a visual question answering benchmark built from images captured by blind users\.
5. 5\.SQA\([Lu et al\., 2022](https://arxiv.org/html/2609.05885#bib.bib45)\)\(4\.2K test set\) is a science\-oriented visual reasoning benchmark based on diagram understanding\.
6. 6\.VQAT\{\}^\{\\text\{T\}\}\([Singh et al\., 2019](https://arxiv.org/html/2609.05885#bib.bib44)\)\(5k test set\) evaluates visual question answering with text\-rich visual inputs\.
7. 7\.POPE\([Li et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib46)\)\(9K test set\) evaluates object hallucination in multimodal large language models\.
8. 8\.MMBench\([Liu et al\., 2024c](https://arxiv.org/html/2609.05885#bib.bib47)\)\(8\.2K test set\) is a comprehensive multimodal benchmark covering perception, reasoning and instruction following\.

### B\.2Implementation Details

All experiments were conducted on NVIDIA H100 and H200 GPUs, with each finetuning run requiring at most several GPU\-hours\. We implement all methods using the PEFT library111https://github\.com/huggingface/peft\. We report the hyperparameter settings for commonsense reasoning in Tab\.[6](https://arxiv.org/html/2609.05885#A2.T6), covering finetuning on LLaMA2\-7B222https://huggingface\.co/meta\-llama/Llama\-2\-7b\-hf, LLaMA3\-8B333https://huggingface\.co/meta\-llama/Meta\-Llama\-3\-8Band Qwen2\.5\-7B444https://huggingface\.co/Qwen/Qwen2\.5\-7B\-Instruct\-1M\. The shared hyperparameter configuration for dialogue, math reasoning and code generation experiments with LLaMA2\-7B is reported in Tab\.[7](https://arxiv.org/html/2609.05885#A2.T7)\. For visual instruction tuning with LLaVA\-1\.5\-7B, we report the corresponding hyperparameter configuration in Tab\.[8](https://arxiv.org/html/2609.05885#A2.T8)\. For all settings, we insert LoRA modules into the query, key and value projections of attention layers and the up\- and down\-projection layers of FFN modules\. The log\-additive fusion hyperparametersα\\alpha,β\\betaandκ\\kappaare set to2\.02\.0,1\.01\.0and3\.03\.0, respectively, with robustness analyses provided in the Appendix\. We perform multiple runs with independent seeds and report the average results\.

#### Baseline tuning protocol\.

All baselines share the training configuration of our method as reported in Tab\.[6](https://arxiv.org/html/2609.05885#A2.T6), Tab\.[7](https://arxiv.org/html/2609.05885#A2.T7)and Tab\.[8](https://arxiv.org/html/2609.05885#A2.T8), with method\-specific hyperparameters set to their officially recommended values\. A further learning\-rate sweep on LLaMA2\-7B commonsense reasoning \(Tab\.[9](https://arxiv.org/html/2609.05885#A2.T9)\) confirms that rsLoRA and LoRA\+ attain their best accuracy at the learning rate used in our setup\. AnLR\-LoRA outperforms all baselines at every learning rate in the sweep, including each baseline’s own best setting, and the reported gains therefore do not stem from favorable learning\-rate choices\.

Table 6:Hyperparameter configuration on commonsense reasoning\.Table 7:Hyperparameter configuration on dialogue, math reasoning and code generation\.Table 8:Hyperparameter configuration on visual instruction tuning\.Table 9:Average commonsense reasoning accuracy \(%\) across varying learning rates on LLaMA2\-7B\.

## Appendix CLicenses and Usage

All datasets and pretrained models used in this work are publicly available and used in accordance with their respective licenses and terms of use, including LLaMA2, LLaMA3, Qwen2\.5 and LLaVA\-1\.5\-7B\. Our use of these resources is limited to academic research and evaluation purposes\.

## Appendix DMore Experiments and Analyses

### D\.1Analysis of Spectral Concentration

#### The full fine\-tuning update has a nearly flat spectrum within the rank budget\.

We take full fine\-tuning of LLaMA2\-7B on Commonsense170K as a reference for the update that a rank\-3232adapter approximates\. We measure the effective rank of an update as the exponential of the Shannon entropy of its singular\-energy distribution, which equalskkwhen the energy is spread uniformly overkkdirections\([Roy and Vetterli, 2007](https://arxiv.org/html/2609.05885#bib.bib54)\)\. The best rank\-3232approximation of the full fine\-tuning update has an effective rank of27\.227\.2out of3232, whereas uniform\-LR LoRA at rank3232learns an update with effective rank12\.712\.7\. This gap bounds how much of the full fine\-tuning update a LoRA update can capture, since by the Eckart–Young theorem an update that spreads its energy overkkdirections captures at most the energy in the top\-kksingular values\. At the learned effective rank of 12\.7, this ceiling is59\.4%59\.4\\%of what the rank\-3232budget could capture, compared with90\.5%90\.5\\%at the effective rank of the full fine\-tuning update\. Moreover, the trailing singular directions \(indices 25–32\) carry28%28\\%as much energy of the full fine\-tuning update per direction as the leading ones, yet receive only2%2\\%as much energy from LoRA\. These directions therefore hold adaptation that uniform\-LR training leaves unused\.

#### Flattening the spectrum directly improves accuracy\.

To isolate the effect of a flatter spectrum on adaptation, we train standard LoRA with an additional loss penalty on the participation ratio of the singular values ofB​ABAas a diagnostic intervention\. The penalty is minimized when the spectrum is flat and involves no learning\-rate intervention\. For the LLaMA2\-7B experiment on math reasoning, with a penalty coefficient of10−310^\{\-3\}, the performance improves from 59\.59 to 60\.96 and the effective rank of the learned update reaches 7\.22 out of 8\. Increasing the coefficient to10−210^\{\-2\}flattens the spectrum further to an effective rank of 7\.89 and raises accuracy to 61\.11\. Broader use of the rank budget is therefore itself beneficial\.

### D\.2Mechanism Analysis of the Two Signals

This section examines the training dynamics underlying the two signals of AnLR\-LoRA, measuring what velocity and Adam SNR capture in LoRA training and how each term of the multiplier shapes the learned update\.

#### Low\-velocity components carry under\-exploited learning signal\.

We analyze whether slow rank\-one components carry meaningful learning signal that a larger learning rate could exploit, using a vanilla LoRA run on commonsense reasoning with LLaMA2\-7B\. We track the velocity, SNR and update\-direction consistency of each rank\-one component throughout training\. Slow and fast rank\-one components show comparable direction consistency, with mean cosine similarities of 0\.22 and 0\.24, indicating that low velocity does not correspond to an absence of a stable update direction\. The difference in update velocity arises from the multiplicative structure ofB​ABA, under which the rate of change of a rank\-one contribution scales with the norms of its factors\. Within the slow half of the components, higher SNR is associated with higher direction consistency, and the SNR term distinguishes slow components with stable directions from those with unstable ones\. The combined velocity–SNR score predicts the subsequent growth of each component’s contribution with a Spearman correlation of 0\.69\. The components that the multiplier boosts are therefore those that continue to develop under uniform LR but at a slower rate, and redistributing learning rate toward them recovers the under\-exploited learning signal\.

#### Effect of each term\.

For the uniform, velocity\-only, SNR\-only and velocity\+SNR configurations of Tab\.[5](https://arxiv.org/html/2609.05885#S5.T5)on LLaMA2\-7B, we log per\-rank velocity, per\-rank SNR and the applied multipliers across all target modules\. The velocity term reduces the within\-module velocity imbalance, halving the coefficient of variation of per\-rank velocity from 0\.12 under uniform LR to 0\.06\. Without the velocity term, the imbalance remains at the level of uniform LR\. The SNR term concentrates the redistributed learning rate on reliable components, lowering the share of the total boost received by the bottom\-SNR quartile from 37\.4% under velocity\-only scaling to 18\.3%, below the proportional share of 25%\. The two terms jointly reduce the fraction of spectrally concentrated modules from 28\.1% to 16\.3%, and neither term alone reaches the accuracy of their combination in Tab\.[5](https://arxiv.org/html/2609.05885#S5.T5)\.

### D\.3Robustness to LR Choice

Tab\.[10](https://arxiv.org/html/2609.05885#A4.T10)provides the full per\-task results for the varying LR experiment discussed in Sec\.[5](https://arxiv.org/html/2609.05885#S5)\. We evaluate six learning rates fromη=5​e\-​5\\eta=5\\text\{e\-\}5to5​e\-​45\\text\{e\-\}4on LLaMA\-2\-7B\. LoRA is highly sensitive to the choice of learning rate\. Its average accuracy peaks at77\.6%77\.6\\%nearη=2​e\-​4\\eta=2\\text\{e\-\}4and drops substantially when the learning rate is smaller or larger, reaching66\.2%66\.2\\%atη=5​e\-​5\\eta=5\\text\{e\-\}5and61\.1%61\.1\\%atη=5​e\-​4\\eta=5\\text\{e\-\}4\. Across the evaluated range, LoRA varies by16\.5%16\.5\\%, and no setting of the global learning rate lifts LoRA above the ceiling of77\.6%77\.6\\%\. AnLR\-LoRA improves over LoRA at every learning rate, with gains ranging from3\.3%3\.3\\%at LoRA’s best setting to13\.7%13\.7\\%at the lower end of the sweep\. AnLR\-LoRA also outperforms LoRA at LoRA’s optimal learning rate, suggesting a genuine performance gain that cannot be explained by LR tuning alone\. The total variation of AnLR\-LoRA across evaluated LR range is only7\.7%7\.7\\%\(73\.3%73\.3\\%to81\.0%81\.0\\%\), which is much more robust compared to LoRA\. These results demonstrate adaptive anisotropic learning rate at the rank\-one level improves both accuracy and robustness across global learning\-rate choices\.

Table 10:Comparison of LoRA and AnLR\-LoRA with varying learning rates on LLaMA2\-7B\.Table 11:Ablation study on LoRA rank with LLaMA2\-7B on math reasoning task\.Table 12:Ablation on per\-rank\-one LR scaling strategy\.
### D\.4Per\-rank\-one LR Scaling Signal

Tab\.[12](https://arxiv.org/html/2609.05885#A4.T12)compares three choices of per\-rank\-one LR multipliers on commonsense reasoning\. Random multipliers collapse the average accuracy to54\.7%54\.7\\%,26\.3%26\.3\\%below AnLR\-LoRA, confirming that the multipliers must derive from a meaningful training\-time signal rather than from random perturbation\. The parameter\-sensitivity signal∥∇W⊙W∥\\\|\\nabla W\\odot W\\\|\([Zhang et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib5)\)recovers most of the performance \(79\.3%79\.3\\%\) and outperforms standard LoRA with uniform\-LR \(77\.5%77\.5\\%\), indicating that other principled signals can also populate the adaptive anisotropic LR framework\. AnLR\-LoRA’s velocity and Adam SNR fusion achieves the highest average accuracy \(81\.0%81\.0\\%\), which is1\.7%1\.7\\%above the parameter\-sensitivity alternative\.

### D\.5Robustness to Rank Choice

Tab\.[11](https://arxiv.org/html/2609.05885#A4.T11)reports the effect of the LoRA rank on math reasoning\. AnLR\-LoRA improves over LoRA at every rank tested \(r∈\{4,8,32\}r\\in\\\{4,8,32\\\}\) by2\.3%2\.3\\%to3\.4%3\.4\\%, with no additional parameter cost\. AnLR\-LoRA atr=4r=4\(60\.96%60\.96\\%\) already exceeds LoRA atr=32r=32\(58\.23%58\.23\\%\), consistent with our finding that AnLR\-LoRA makes broader use of the nominal rank budget\.

Figure 6:Accuracy across the velocity weightα\\alphaon math reasoning and code generation tasks\.Figure 7:Accuracy across the SNR weightβ\\betaon math reasoning and code generation tasks\.
### D\.6Robustness to Hyperparameter Choice

Fig\.[6](https://arxiv.org/html/2609.05885#A4.F6)and Fig\.[7](https://arxiv.org/html/2609.05885#A4.F7)report AnLR\-LoRA’s sensitivity to the fusion weightsα\\alphaandβ\\betaon math reasoning and code generation tasks\. Across both tasks, performance is relatively stable across the evaluated range ofα∈\{0\.5,1\.0,1\.5,2\.0,2\.5\}\\alpha\\in\\\{0\.5,1\.0,1\.5,2\.0,2\.5\\\}andβ∈\{0\.5,1\.0,1\.5,2\.0\}\\beta\\in\\\{0\.5,1\.0,1\.5,2\.0\\\}\. Fig\.[8](https://arxiv.org/html/2609.05885#A4.F8), Fig\.[9](https://arxiv.org/html/2609.05885#A4.F9)and Fig\.[10](https://arxiv.org/html/2609.05885#A4.F10)further sweep the velocity EMA coefficientρ\\rho, the clamp boundκ\\kappaand the warmup lengthTwarmT\_\{\\mathrm\{warm\}\}on commonsense reasoning with LLaMA2\-7B and LLaMA3\-8B, showing similarly stable results across the evaluated values\. AnLR\-LoRA is therefore robust to the choice of hyperparameters and does not require careful tuning to achieve strong performance\.

Figure 8:Average commonsense reasoning accuracy across the velocity EMA coefficientρ\\rho\.Figure 9:Average commonsense reasoning accuracy across the clamp boundκ\\kappa\.Figure 10:Average commonsense reasoning accuracy across the warmup lengthTwarmT\_\{\\mathrm\{warm\}\}\.Table 13:Accuracies with standard deviations on commonsense reasoning tasks\.Table 14:Accuracies with standard deviations on natural language generation and visual instruction tuning tasks\.
### D\.7Standard Deviation Across Seeds

We conduct all main experiments over three independent runs with different random seeds and report the mean and standard deviation in Tabs\.[13](https://arxiv.org/html/2609.05885#A4.T13)and[14](https://arxiv.org/html/2609.05885#A4.T14)\. The low variance across seeds indicates that the improvements of AnLR\-LoRA over LoRA are robust to the choice of random seed\.

Table 15:Evaluation results with Qwen2\.5\-14B\.
### D\.8Experiments on Qwen2\.5\-14B

We further extend the math reasoning evaluation to the larger Qwen2\.5\-14B backbone \(Tab\.[15](https://arxiv.org/html/2609.05885#A4.T15)\)\. AnLR\-LoRA attains the best accuracy, outperforming LoRA, LoRA\+ and rsLoRA by 0\.83, 1\.32 and 4\.32 points, respectively\. The consistent gain shows that adaptive anisotropic LR remains effective at larger model scales\.

Table 16:Comparison with additional LoRA variants on commonsense reasoning tasks with average accuracy reported\.
### D\.9Comparison with Additional LoRA Variants

Tab\.[16](https://arxiv.org/html/2609.05885#A4.T16)extends the commonsense reasoning comparison to more LoRA variants, including AdaLoRA\([Zhang et al\., 2023](https://arxiv.org/html/2609.05885#bib.bib5)\), DoRA\([Liu et al\., 2024b](https://arxiv.org/html/2609.05885#bib.bib1)\)and LoRA\-Pro\([Wang et al\., 2025c](https://arxiv.org/html/2609.05885#bib.bib25)\), trained under the protocol of Tab\.[1](https://arxiv.org/html/2609.05885#S4.T1)with their official configurations\. AdaLoRA adaptively allocates the rank budget across modules by importance\-based pruning of singular triplets\. DoRA decomposes the adapted weight into a learnable magnitude and a LoRA\-updated direction\. LoRA\-Pro adjusts the gradients to better approximate the full fine\-tuning update, requiring per\-module matrix inversions and the solution of a Sylvester equation at each optimization step\. AnLR\-LoRA achieves the best performance on every backbone without introducing additional trainable parameters or expensive matrix operations, relying only on lightweight rescaling of the optimizer updates\.

### D\.10Analysis on Memory Overhead of the Parameter Snapshot

Post\-step delta scaling maintains a transient snapshot of the LoRA factorsAAandBBwithin each optimizer step\. The snapshot introduces no trainable parameters, and its size is determined by the adapter rather than by the base model\. On LLaMA2\-7B, it occupies 56 MB, corresponding to 0\.42% of the base\-weight memory\. This overhead decreases further for larger backbones, with an estimated 0\.20% at the 70B scale and 0\.10% at the 405B scale\. The snapshot can also be eliminated entirely by fusing the rescaling into the optimizer step\.

## Appendix ELLM Usage

AI assistants were used only for language refinement during manuscript preparation\. They were not involved in proposing the core method, designing the experiments, or drawing scientific conclusions\. Since this work studies parameter\-efficient finetuning for large language models, pretrained models including LLaMA2\-7B, LLaMA3\-8B, Qwen2\.5\-7B, Qwen2\.5\-14B and LLaVA\-1\.5\-7B were used as part of the experimental setup and evaluation\.