Parameter-Efficient Fine-Tuning with Learnable Rank

# Parameter-Efficient Fine-Tuning with Learnable Rank
Source: [https://arxiv.org/html/2606.04325](https://arxiv.org/html/2606.04325)
Arpit Garg Australian Institute for Machine Learning Adelaide University arpit\.garg@adelaide\.edu\.au &Simon Lucey Australian Institute for Machine Learning Adelaide University simon\.lucey@adelaide\.edu\.au &Hemanth Saratchandran Australian Institute for Machine Learning Adelaide University hemanth\.saratchandran@adelaide\.edu\.au

###### Abstract

Low\-Rank Adaptation \(LoRA\) is a popular parameter\-efficient fine\-tuning \(PEFT\) method that restricts weight updates to low\-rank adapters, introducing a fixed low\-rank inductive bias by optimizing in a low\-dimensional subspace\. In this work, we question whether a fixed\-rank constraint is the most effective inductive bias for parameter\-efficient fine\-tuning\. We introduce*Learnable Rank LoRA \(LR\-LoRA\)*, a PEFT method in which the adapter rank is learned during the training process\. Instead of prescribing a uniform rank for all adapter layers, LR\-LoRA allows the optimizer to determine the appropriate rank for each layer\. Using this approach, we find substantial layer\-wise variation in the learned ranks, with the attention and MLP layers in the transformer models exhibiting systematically different rank preferences\. Across a range of language understanding and commonsense reasoning benchmarks, LR\-LoRA achieves state\-of\-the\-art performance in most settings and consistently outperforms strong PEFT baselines, demonstrating that a learnable rank provides a more flexible and effective inductive bias than fixed\-rank adaptations\.

## 1Introduction

Large pretrained models are increasingly being adapted to a wide range of downstream tasks, domains, and users\. In this regime, the dominant computational bottleneck has shifted from pretraining to*adaptation*, which is the process of specializing large backbone models under stringent memory, storage, and latency constraints\(Touvronet al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib25); Grattafioriet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib27); Team and others,[2024](https://arxiv.org/html/2606.04325#bib.bib28); Abdinet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib29)\)\. Parameter\-efficient fine\-tuning \(PEFT\) addresses this challenge by keeping the backbone fixed while learning a small number of task\-specific parameters, enabling rapid customization with orders of magnitude fewer trainable weights than full fine\-tuning\(Huet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib1); Hanet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib31)\)\.

Among PEFT methods, Low\-Rank Adaptation \(LoRA\) and its variants have emerged as the de facto standard owing to their simplicity, stability, and ease of deployment\(Huet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib1)\)\. LoRA augments selected layers with a low\-rank update,

Δ​𝐖=𝐁𝐀,\\Delta\\mathbf\{W\}=\\mathbf\{BA\},parameterized by two low\-rank adapter matrices,𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}\. This construction imposes a clear inductive bias: task\-specific updates are constrained to a low\-dimensional rank\-rrset, the dimensionality of which is determined by the chosen rank\.

Despite its empirical success, LoRA relies on a strong and largely unchallenged assumption: the adapter rank is fixed\. In practice, a single global rank is typically applied uniformly across all layers, implicitly assuming that each layer requires a similar adaptation dimensionality\. This assumption is difficult to justify, given what is known about transformer architectures, whose layers differ substantially across depth in function, representation geometry, and intrinsic dimensionality\(Aghajanyanet al\.,[2021](https://arxiv.org/html/2606.04325#bib.bib23)\)\.

Recent studies have attempted to mitigate this limitation by increasing the expressivity within the LoRA framework\. For example, RandLoRA employs random bases to approximate full\-rank updates while retaining a low\-rank parameterization\(Albertet al\.,[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\), and SineLoRA increases the rank of MLP adapters through fixed sinusoidal modulation\(Jiet al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib17)\)\. Although these approaches introduce different inductive biases, they rely on ranks that are externally specified and fixed throughout the training\. Consequently, they lack a mechanism that allows the model to*learn*the appropriate adaptation dimensionality in a layer\- or task\-dependent manner\.

Taken together, these observations raise a simple but fundamental question:

> *Is enforcing a fixed\-rank constraint on adapter updates an optimal inductive bias for achieving strong performance in parameter\-efficient fine\-tuning?*

To address this question, we introduceLearnable Rank LoRA \(LR\-LoRA\), a PEFT method in which the adapter rank is learned during the training process\. Rather than prescribing a fixed rank for each layer throughout training, LR\-LoRA augments the adapter update with an element\-wise nonlinearityϕ\\phi, yielding

Δ​𝐖=ϕ​\(𝐁𝐀\),\\Delta\\mathbf\{W\}=\\phi\(\\mathbf\{BA\}\),whereϕ\\phiis learned jointly with the model’s parameters\. By applying a nonlinearity to the low\-rank product𝐁𝐀\\mathbf\{BA\}, LR\-LoRA allows the rank of the update to increase or decrease, relaxing the fixed\-rank constraint imposed by standard low\-rank adapters and enabling the optimizer to adapt the dimensionality of each layer’s update \(formalized in[Section˜3\.3](https://arxiv.org/html/2606.04325#S3.SS3)\)\.

Across language understanding, commonsense reasoning, vision transfer, and instruction\-tuning benchmarks, LR\-LoRA consistently outperforms strong PEFT baselines under approximately matched parameter budgets, achieving state\-of\-the\-art performance in most settings\. Empirically, we observe substantial variation in the learned ranks across layers, with attention and MLP adapters exhibiting systematically different adaptation dimensionalities\. These findings suggest that enforcing a hard, fixed\-rank constraint is suboptimal for parameter\-efficient fine\-tuning methods, such as LoRA\.

The main contributions of this study are as follows:

- •Learnable Rank LoRA \(LR\-LoRA\):a parameter\-efficient fine\-tuning method that learns layer\-wise adapter ranks during training, introducing a more flexible inductive bias for adaptation\.
- •A basis\-agnostic principle for adaptation dimensionality:we show that applying any differentiable element\-wise nonlinearity to𝐁𝐀\\mathbf\{BA\}removes the rank\-rrconstraint, and we provide direct empirical evidence that the gains stem from*learning*per\-layer adaptation, not from the mere presence of nonlinearity\.
- •Consistent improvements over strong PEFT baselines, achieving new state\-of\-the\-art results across77architectures spanning125125M to1313B parameters,1919tasks, and four evaluation paradigms \(commonsense reasoning, GLUE transfer, vision transfer, and instruction tuning\)\.

More broadly, this study reframes Low\-Rank Adaptation through the lens of inductive bias\. Rather than treating rank as a fixed hyperparameter, we argue that adaptation dimensionality should be a learnable property of the model that reflects the heterogeneity of representations across layers and tasks\. This perspective suggests a shift from rigid parameter\-efficient designs to adaptive mechanisms that allow models to allocate capacity where it is most needed\.

## 2Related Work

Parameter\-efficient fine\-tuning\.Parameter\-efficient fine\-tuning \(PEFT\) adapts large pretrained models by learning a small set of task\-specific parameters while keeping the backbone frozen, enabling efficient storage and deployment across many tasks\(Houlsbyet al\.,[2019](https://arxiv.org/html/2606.04325#bib.bib32); Dinget al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib30); Hanet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib31)\)\. Major PEFT families include adapter modules\(Houlsbyet al\.,[2019](https://arxiv.org/html/2606.04325#bib.bib32)\), prompt and prefix\-based tuning\(Lesteret al\.,[2021](https://arxiv.org/html/2606.04325#bib.bib39); Li and Liang,[2021](https://arxiv.org/html/2606.04325#bib.bib38)\), multiplicative adaptation methods such as IA3\(Liuet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib48)\), and lightweight weight\-space updates such as BitFit\(Zakenet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib40)\)\. Our work lies within the weight\-space PEFT and focuses specifically on controlling the dimensionality of low\-rank weight updates\.

Low\-rank adaptation \(LoRA\)\.LoRA utilizes a low\-rank updateΔ​𝐖=𝐁𝐀\\Delta\\mathbf\{W\}=\\mathbf\{BA\}in selected linear layers, imposing the inductive bias that task\-specific weight changes reside in a low\-dimensional rank\-rrset\(Huet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib1); Fuet al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib24)\)\. A large body of follow\-up work improves LoRA through rank allocation heuristics\(Zhanget al\.,[2023b](https://arxiv.org/html/2606.04325#bib.bib2); Valipouret al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib3); Albertet al\.,[2025b](https://arxiv.org/html/2606.04325#bib.bib7)\), alternative parameterizations\(Liuet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib4)\), randomized or vectorized adaptation\(Kopiczkoet al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib5)\), improved initialization\(Menget al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib8)\), training refinements\(Hayouet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib9)\), memory\-efficient training\(Dettmerset al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib10); Zhaoet al\.,[2024](https://arxiv.org/html/2606.04325#bib.bib11); Zhanget al\.,[2023a](https://arxiv.org/html/2606.04325#bib.bib13)\), and stability analyses\(Kalajdzievski,[2023](https://arxiv.org/html/2606.04325#bib.bib14); Lialinet al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib12)\)\. Although these methods improve*where*or*how much*low\-rank capacity is allocated, they generally treat rank as an externally specified hyperparameter\. In contrast, we relax this assumption by allowing the adapter rank to be learned\. Furthermore, several approaches increase the expressivity of adapters while preserving compact parameterization\. RandLoRA employs random bases to approximate more expressive \(including near\-full\-rank\) updates within a low\-rank framework\(Albertet al\.,[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\), and VeRA uses a randomized vector\-based adaptation\(Kopiczkoet al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib5)\)\. Another line of work introduces a nonlinearity into the adapter pathway: SineLoRA applies a fixed sinusoidal modulation to adapter weights, obtaining a high\-rank adapter update and increasing representational capacity without increasing the parameter count\(Jiet al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib17)\)\. Recent adaptive\-rank methods such as AdaLoRA\(Zhanget al\.,[2023b](https://arxiv.org/html/2606.04325#bib.bib2)\), DyLoRA\(Valipouret al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib3)\), and ElaLoRA\(Changet al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib54)\)reallocate per\-layer ranks during training under a fixed total budget set ahead of time; nonlinear variants like AuroRA\(Donget al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib55)\)and ABBA\(Singhalet al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib56)\)break linear constraints through adaptive nonlinear layers and Hadamard products\. In all of these, the global capacity is externally specified\. LR\-LoRA differs in*how*the effective rank arises: the optimizer learns a per\-layer continuous elementwise nonlinearity over𝐁𝐀\\mathbf\{BA\}that implicitly sets each layer’s effective \(stable\) rank without an external allocation budget, and we provide controlled experiments isolating learnable rank from the mere presence of nonlinearity \([Section˜4\.3](https://arxiv.org/html/2606.04325#S4.SS3)\)\.

## 3Methodology

In this section, we describe the methodology underlying Learnable Rank LoRA \(LR\-LoRA\), with a particular focus on the construction and learning of learnable rank adapter updates\.

### 3\.1Low\-Rank Adaptation

Low\-Rank Adaptation \(LoRA\) is a parameter\-efficient fine\-tuning method that keeps the pretrained weights𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}of a network frozen and introduces trainable adapter matrices𝐀∈ℝr×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times n\}and𝐁∈ℝm×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{m\\times r\}, wherer≪min⁡\(m,n\)r\\ll\\min\(m,n\)\. The adapted weight matrix is given by

𝐖\+𝐁𝐀\.\\mathbf\{W\}\+\\mathbf\{BA\}\.\(1\)During task\-specific training, only the adapter parameters𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}are updated, while𝐖\\mathbf\{W\}remains fixed\.

The efficiency of LoRA arises from the fact that𝐖\\mathbf\{W\}containsm​nmnparameters, whereas the adapter update𝐁𝐀\\mathbf\{BA\}involves onlyr​\(m\+n\)r\(m\+n\)parameters, which is substantially smaller whenr≪min⁡\(m,n\)r\\ll\\min\(m,n\)\. This parameter reduction comes at the cost of expressivity: sincerank​\(𝐁𝐀\)≤r\\mathrm\{rank\}\(\\mathbf\{BA\}\)\\leq r, LoRA constrains task\-specific updates to the low\-dimensional rank\-rrset\. Consequently, the LoRA architecture imposes a low\-rank inductive bias on the adaptation process\.

### 3\.2Preliminaries on Sampling Theory

We adopt the normalized sinc conventionsinc​\(x\)=sin⁡\(π​x\)/\(π​x\)\\mathrm\{sinc\}\(x\)=\\sin\(\\pi x\)/\(\\pi x\)forx≠0x\\\!\\neq\\\!0andsinc​\(0\)=1\\mathrm\{sinc\}\(0\)=1, so that the Whittaker\-Shannon interpolation formula holds in its standard form\. The Nyquist\-Shannon sampling theorem\(Nyquist,[1928](https://arxiv.org/html/2606.04325#bib.bib51); Shannon,[2006](https://arxiv.org/html/2606.04325#bib.bib36); Martin,[1997](https://arxiv.org/html/2606.04325#bib.bib52)\)states that any bandlimitedf∈L2​\(ℝ\)f\\in L^\{2\}\(\\mathbb\{R\}\)with maximum frequencyωmax\\omega\_\{\\max\}can be exactly reconstructed from samples\{f​\(xi\)\}\\\{f\(x\_\{i\}\)\\\}on a uniform grid with spacingτ≤1/\(2​ωmax\)\\tau\\leq 1/\(2\\omega\_\{\\max\}\)as

f​\(x\)=∑i∈ℤf​\(xi\)​sinc​\(ω​\(x−xi\)\),ω:=1τ,xi:=i​τ\.f\(x\)=\\sum\_\{i\\in\\mathbb\{Z\}\}f\(x\_\{i\}\)\\,\\mathrm\{sinc\}\\\!\\left\(\\omega\(x\-x\_\{i\}\)\\right\),\\qquad\\omega:=\\tfrac\{1\}\{\\tau\},\\quad x\_\{i\}:=i\\tau\.\(2\)In practice the sum is truncated toNNterms \(with bounded approximation error for sufficiently largeNN\), and more generally functions inL2​\(ℝ\)L^\{2\}\(\\mathbb\{R\}\)are well approximated by mixtures of shifted sincs with learnable amplitudeαi\\alpha\_\{i\}and bandwidthωi\\omega\_\{i\}:

∑i=1Nαi​sinc​\(ωi​\(x−xi\)\)\.\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\,\\mathrm\{sinc\}\\\!\\left\(\\omega\_\{i\}\(x\-x\_\{i\}\)\\right\)\.\(3\)This expansion forms the basis for the learnable nonlinearity introduced next\. A more complete review with derivations is provided in[Appendix˜A](https://arxiv.org/html/2606.04325#A1)\.

### 3\.3Learnable Adapter Nonlinearity

The core idea of LR\-LoRA is to apply a nonlinearityϕ\\phito the adapter update𝐁𝐀\\mathbf\{BA\}, yieldingϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)\. We constructϕ\\phiusing a flexible basis of shiftedsinc\\mathrm\{sinc\}functions\. Specifically, we consider a collection ofNNuniformly spaced grid points\{x1,x2,…,xN\}\\\{x\_\{1\},x\_\{2\},\\ldots,x\_\{N\}\\\}over a bounded interval\[−I,I\]\[\-I,I\]\. At each grid pointxix\_\{i\}, we place asinc\\mathrm\{sinc\}function with bandwidthωi\\omega\_\{i\}and amplitudeαi\\alpha\_\{i\}, yielding basis functions of the formsinc​\(ωi​\(x−xi\)\)\\mathrm\{sinc\}\\\!\\left\(\\omega\_\{i\}\(x\-x\_\{i\}\)\\right\)\. The resulting nonlinearity is defined as

ϕ​\(x\)=∑i=1Nαi​sinc​\(ωi​\(x−xi\)\)\.\\phi\(x\)=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\,\\mathrm\{sinc\}\\\!\\left\(\\omega\_\{i\}\(x\-x\_\{i\}\)\\right\)\.\(4\)To makeϕ\\philearnable, we treat the parameters\{\(ω1,α1\),…,\(ωN,αN\)\}\\\{\(\\omega\_\{1\},\\alpha\_\{1\}\),\\ldots,\(\\omega\_\{N\},\\alpha\_\{N\}\)\\\}as trainable and optimize them jointly with the adapter weights during fine\-tuning\. An illustration of this construction is shown in[Figure˜1](https://arxiv.org/html/2606.04325#S3.F1)\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x1.png)

![Refer to caption](https://arxiv.org/html/2606.04325v1/x2.png)

Figure 1:Construction of the learnable nonlinearityϕ\\phi\([Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\)\.Left:individualsinc\\mathrm\{sinc\}basis functions on the grid\{xi\}\\\{x\_\{i\}\\\}with per\-basis bandwidthωi\\omega\_\{i\}and amplitudeαi\\alpha\_\{i\}\.Right:the resultingϕ​\(x\)\\phi\(x\)obtained by summing the shiftedsinc\\mathrm\{sinc\}functions\. Bandwidths and amplitudes are learned jointly with the adapter weights\.##### Why a nonlinearity removes the rank\-rrconstraint\.

The product𝐁𝐀\\mathbf\{BA\}defines a low\-rank*linear*map, constraining task\-specific updates to the rank\-rrset\. Applying an element\-wise nonlinearityϕ\\phiremoves this rank constraint by mapping the update through a nonlinear transformation\. Consequently, the adapted updateϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)is no longer restricted to rankrrand can have rank less than or greater thanrr\.

##### Implementation\.

Adapter matrices𝐀∈ℝr×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times n\}and𝐁∈ℝm×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{m\\times r\}are chosen as in LoRA\. Each adapter has its ownϕ\\phi, so every layer learns its own transfer\-function parameters\. We initializeαi=0\\alpha\_\{i\}\\\!=\\\!0so thatϕ≡0\\phi\\\!\\equiv\\\!0at the start of training \(any expressive capacity must be*learned*\), and parameterizeωi=softplus​\(ω~i\)\\omega\_\{i\}\\\!=\\\!\\mathrm\{softplus\}\(\\tilde\{\\omega\}\_\{i\}\)withω~i\\tilde\{\\omega\}\_\{i\}initialized to the constantln⁡\(eω0−1\)\\ln\(e^\{\\omega\_\{0\}\}\-1\)so thatωi=ω0\\omega\_\{i\}=\\omega\_\{0\}at initialization \(specificallyω~i≈0\.54\\tilde\{\\omega\}\_\{i\}\\approx 0\.54for the defaultω0=1\.0\\omega\_\{0\}=1\.0\), keepingωi\\omega\_\{i\}strictly positive while avoiding the degenerate zero solution\. The learned adaptation dimensionality of each layer can therefore be smaller or larger than the nominalrr\. The additional parameters scale with grid size and layer count and incur small overhead in practice \([Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px3)\); full details in[Appendices˜E](https://arxiv.org/html/2606.04325#A5)and[3](https://arxiv.org/html/2606.04325#A5.F3)\.

##### Stable rank as a measure of update complexity\.

In practice, computing the matrix rank is numerically ill\-posed because it requires choosing a threshold on the singular values\. Therefore, in[Section˜4](https://arxiv.org/html/2606.04325#S4)we quantify adaptation complexity using the*stable rank*, as also done inJiet al\.\([2025](https://arxiv.org/html/2606.04325#bib.bib17)\)\. For a matrix𝐌\\mathbf\{M\}, the stable rank is defined as

𝒮​\(𝐌\):=‖𝐌‖F2‖𝐌‖22,\\mathcal\{S\}\(\\mathbf\{M\}\):=\\frac\{\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\}\{\\\|\\mathbf\{M\}\\\|\_\{2\}^\{2\}\},\(5\)where∥⋅∥F\\\|\\cdot\\\|\_\{F\}denotes the Frobenius norm and∥⋅∥2\\\|\\cdot\\\|\_\{2\}denotes the largest singular value of the matrix\. The stable rank provides a continuous and numerically robust measure of spectral complexity and is therefore well suited for empirical analysis\. We use stable rank as a proxy for update complexity throughout the paper; in[Section˜4\.4](https://arxiv.org/html/2606.04325#S4.SS4)we additionally provide direct functional evidence \(parameter\-sharing and module\-wise ablations\) that does not rely on this proxy\.

Our experimental results in[Section˜4](https://arxiv.org/html/2606.04325#S4)show that the stable rank ofϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)varies substantially across layers: in some layers, it is smaller than that of𝐁𝐀\\mathbf\{BA\}, while in others, it is larger\. This behavior indicates that different layers prefer different levels of expressive capacity, reinforcing the motivation to learn adaptation dimensionality in a layer\-dependent manner\. We analyze the effect of the grid size and number of grid points in[Section˜4](https://arxiv.org/html/2606.04325#S4), studying the trade\-offs between expressivity and stability\.

##### Basis\-agnostic principle\.

The central scientific claim of LR\-LoRA is not the specific sinc parameterization, but the basis\-agnostic principle that*the appropriate adaptation dimensionality of each layer is unknown before training and should be discovered by the optimizer, not prescribed by the practitioner*\. Applying any differentiable elementwise nonlinearityϕ\\phito𝐁𝐀\\mathbf\{BA\}removes the rank\-rrconstraint \([Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\), allowing the realized stable rank𝒮​\(ϕ​\(𝐁𝐀\)\)\\mathcal\{S\}\(\\phi\(\\mathbf\{BA\}\)\)to differ from the nominalrr\([Equations˜4](https://arxiv.org/html/2606.04325#S3.E4)and[5](https://arxiv.org/html/2606.04325#S3.E5)\)\. Sinc is one realization, chosen for the controlled\-bandwidth, non\-periodic inductive bias justified above\.

##### Other basis functions\.

More generally, other choices of basis functions are possible, including trigonometric bases such assin\\sinandcos\\cosand spline\-based representations\. However, according to classical Fourier theory\(Stein and Shakarchi,[2009](https://arxiv.org/html/2606.04325#bib.bib53)\), representations based onsin\\sinandcos\\cosinherently impose a periodic structure, restricting the resulting functions to be periodic\. We avoid such bases because the induced periodic inductive bias may be undesirable when adapting model weights\. Spline\-based bases \(as used inTeneyet al\.\([2025](https://arxiv.org/html/2606.04325#bib.bib6)\)\) provide an alternative that does not enforce periodicity\. Among Gaussian RBF, B\-splines, Fourier, polynomial, and sinc bases evaluated under matched parameter counts \([Section˜F\.1\.2](https://arxiv.org/html/2606.04325#A6.SS1.SSS2)\), sinc achieves the highest accuracy and stability; B\-splines \(the simplest non\-periodic alternative\) score close behind, while the periodic Fourier basis underperforms, consistent with the periodicity argument above\. Sinc is therefore a well\-motivated choice within an open family rather than a unique requirement\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x3.png)\(a\)MLP projections \(final checkpoint\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x4.png)\(b\)Attention projections \(final checkpoint\)\.

Figure 2:Stable\-rank profiles of learned updates across depth\.Qwen2\-0\.5Bon commonsense reasoning at rankr=16r\{=\}16; each panel plots stable rank \(y\-axis\) against layer index \(x\-axis\) at the final training checkpoint\.Takeaway:update complexity is depth\-dependent and differs systematically between MLP and attention modules\. Per\-checkpoint evolution and the corresponding learned nonlinearitiesϕ​\(z\)\\phi\(z\)at representative early, mid, and late layers are reported in[Appendix˜I](https://arxiv.org/html/2606.04325#A9)\.

## 4Experiments

We evaluateLR\-LoRA, replacing LoRA’s fixed low\-rank update \([Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\), which imposes a strong inductive bias, with our layer\-wise learnable updateϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)\([Equations˜4](https://arxiv.org/html/2606.04325#S3.E4)and[3](https://arxiv.org/html/2606.04325#S3)\)\. We reuse the full training and evaluation harness ofAlbertet al\.\([2025a](https://arxiv.org/html/2606.04325#bib.bib16)\)from their released codebase and change only the adapter parameterization unless explicitly stated\. All methods therefore share identical backbones, adapter placement, data preprocessing, optimization schedules, and evaluation scripts, ensuring that the differences are attributable to the induced update family rather than confounding implementation choices\. Throughout, we match the adapter rank hyperparameterrracross methods as the capacity budget, and note that*learnable rank*refers to the stable rank induced by learningϕ\\phias explained in[Section˜3](https://arxiv.org/html/2606.04325#S3)\.

##### Scope of evaluation\.

LR\-LoRA is evaluated across77architecturesspanning125125M to1313B parameters,1919tasks, andfour evaluation paradigms: commonsense reasoning \([Table˜1](https://arxiv.org/html/2606.04325#S4.T1)\), GLUE transfer \([Table˜2](https://arxiv.org/html/2606.04325#S4.T2)\), vision transfer \([Appendix˜D](https://arxiv.org/html/2606.04325#A4)\), and instruction tuning via MT\-Bench \([Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px2)\)\. Together, these cover both classification\-style and open\-ended generation evaluation\. All experiments fit within a single high\-memory \(≥40\\geq 40GB\) GPU per run; LR\-LoRA’s training\-time overhead over LoRA is≤\+3%\\leq\+3\\%peak memory and≤−2%\\leq\-2\\%throughput \([Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px3)\)\.

### 4\.1Experimental Setup

Backbones and data regimes\.We fine\-tune three decoder\-only LLM backbones:Qwen2−0\.5B,Phi−3−8B, andLLaMA3−8B\. To assess data efficiency, we evaluate two regimes from the reference setup: afullregime of approximately170170k training examples and asubsetregime of approximately1515k examples\(Albertet al\.,[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\)\. We additionally evaluate CLIP and DINOv2 image classification models; these results are reported in[Appendices˜D](https://arxiv.org/html/2606.04325#A4),[D\.1](https://arxiv.org/html/2606.04325#A4.SS1)and[D\.2](https://arxiv.org/html/2606.04325#A4.SS2)\.

Tasks and metrics\.*Commonsense reasoning\.*We evaluate the eight\-task suite: BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2606.04325#bib.bib41)\), PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2606.04325#bib.bib42)\), SocialIQA\(Sapet al\.,[2019](https://arxiv.org/html/2606.04325#bib.bib43)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2606.04325#bib.bib35)\), WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2606.04325#bib.bib44)\), ARC\-Easy and ARC\-Challenge\(Clarket al\.,[2018](https://arxiv.org/html/2606.04325#bib.bib45)\), and OpenBookQA\(Mihaylovet al\.,[2018](https://arxiv.org/html/2606.04325#bib.bib46)\)\. We report per\-task accuracy and the average across tasks\.

*Natural language understanding \(GLUE\)\.*To test transfer beyond decoder\-only LLMs, we evaluate RoBERTa\-base and RoBERTa\-large on GLUE\(Wanget al\.,[2018](https://arxiv.org/html/2606.04325#bib.bib34); Liuet al\.,[2019](https://arxiv.org/html/2606.04325#bib.bib47)\), reporting mean±\\pmstd over33seeds\.

*Instruction tuning \(MT\-Bench\)\.*We additionally fine\-tune on Stanford Alpaca and evaluate with MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib57)\)\(GPT\-4 judge\) atr=32r\{=\}32to assess open\-ended generation; details in[Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px2)\.

Baselines and controlled comparisons\.We compare against LoRA\(Huet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib1)\), VeRA\(Kopiczkoet al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib5)\), RandLoRA\(Albertet al\.,[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\), and SineLoRA\(Jiet al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib17)\)\. Efficient LoRA used rankr=16r\{=\}16and Performant LoRA used rankr=32r\{=\}32\. For RandLoRA, we used the same basis ranks and number of bases as reported in Table 9 ofAlbertet al\.\([2025a](https://arxiv.org/html/2606.04325#bib.bib16)\)\. We applied adapters to the attention and MLP projection layers throughout the backbone and did not adapt the final output head where applicable\. The adapted module set is identical across all methods for controlled comparisons\.

Optimization\.We employed AdamW for all models and maintained consistent optimization schedules across the methods\. For decoder\-only LLMs, we trained for33epochs with a learning rate1×10−41\\times 10^\{\-4\}and adapter dropout0\.050\.05\. Batch sizes are1616,88, and44forQwen2−0\.5B,Phi−3−8B, andLLaMA3−8B, respectively\. For GLUE, we fine\-tune using AdamW with\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)\{=\}\(0\.9,0\.999\)and weight decay0\.010\.01, with learning rate tuned over\{5×10−5,1×10−4,2×10−4\}\\\{5\\times 10^\{\-5\},1\\times 10^\{\-4\},2\\times 10^\{\-4\}\\\}\. Each LLM run completes on a single high\-memory \(≥40\\geq 40GB\) GPU\. Complete hyperparameter configurations are detailed in[Section˜E\.1](https://arxiv.org/html/2606.04325#A5.SS1)\.

LR\-LoRA hyperparameters, initialization, and reproducibility\.Unless otherwise specified, we employ the sinc basis nonlinearityϕ\\phifrom[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)withN=50N\{=\}50uniformly spaced grid points on\[−I,I\]\[\-I,I\], whereI=3I\{=\}3\([Section˜3\.3](https://arxiv.org/html/2606.04325#S3.SS3)\)\. The amplitudesαi\\alpha\_\{i\}are initialized to zero, and the bandwidths are parameterized asωi=softplus​\(ω~i\)\\omega\_\{i\}\{=\}\\mathrm\{softplus\}\(\\tilde\{\\omega\}\_\{i\}\)withω~i=ln⁡\(eω0−1\)\\tilde\{\\omega\}\_\{i\}=\\ln\(e^\{\\omega\_\{0\}\}\-1\)so thatωi=ω0∈ℝ\>0\\omega\_\{i\}=\\omega\_\{0\}\\in\\mathbb\{R\}\_\{\>0\}at initialization\. This ensuresϕ≡0\\phi\\equiv 0at initialization while keepingωi\\omega\_\{i\}strictly positive and numerically well defined in[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\.Recommended default\(used unchanged across all77architectures and1919tasks\):N=50N\{=\}50,I=3I\{=\}3,αi=0\\alpha\_\{i\}\{=\}0,ω0=1\.0\\omega\_\{0\}\{=\}1\.0\. We report mean±\\pmstandard deviation over three seeds where feasible\. All values of\(N,I,ω0\)\(N,I,\\omega\_\{0\}\)and complete ablations are in[Appendices˜F](https://arxiv.org/html/2606.04325#A6),[F\.1](https://arxiv.org/html/2606.04325#A6.SS1),[F\.2](https://arxiv.org/html/2606.04325#A6.SS2),[F\.3](https://arxiv.org/html/2606.04325#A6.SS3)and[F\.4](https://arxiv.org/html/2606.04325#A6.SS4)\. Statistical significance under pairedtt\-tests with Holm\-Bonferroni correction across tasks is reported in[Section˜B\.2](https://arxiv.org/html/2606.04325#A2.SS2)\.

### 4\.2Main Results

Table 1:Main Results: LR\-LoRA vs\. strong PEFT baselines on commonsense reasoning\.Evaluation across three model scales \(Qwen2\-0\.5B, Phi\-3\-8B, LLaMA3\-8B\) and two data regimes \(15k/170k samples\) using eight common\-sense tasks \(r=32r=32\) \. Results show mean±\\pmstdover 3 seeds\.Boldindicates best performance\.Qwen2\-0\.5BPhi\-3\-8BLLaMA3\-8BMethod15k170k15k170k15k170kNoLA42\.55±\\pm0\.1847\.40±\\pm0\.1580\.35±\\pm0\.1282\.33±\\pm0\.0876\.87±\\pm0\.1481\.16±\\pm0\.11VeRA48\.12±\\pm0\.2251\.78±\\pm0\.1978\.63±\\pm0\.1681\.40±\\pm0\.1377\.14±\\pm0\.1881\.65±\\pm0\.12LoRA52\.34±\\pm0\.1457\.34±\\pm0\.1180\.34±\\pm0\.1084\.96±\\pm0\.0983\.09±\\pm0\.1385\.24±\\pm0\.08AdaLoRA53\.15±\\pm0\.1558\.20±\\pm0\.1281\.45±\\pm0\.1385\.35±\\pm0\.1083\.50±\\pm0\.1485\.88±\\pm0\.11DyLoRA52\.75±\\pm0\.1857\.60±\\pm0\.1480\.85±\\pm0\.1585\.05±\\pm0\.1283\.25±\\pm0\.1685\.40±\\pm0\.13ElaLoRA54\.02±\\pm0\.1459\.15±\\pm0\.1182\.55±\\pm0\.1286\.20±\\pm0\.0984\.55±\\pm0\.1087\.05±\\pm0\.08SineLoRA52\.51±\\pm0\.1657\.55±\\pm0\.1281\.12±\\pm0\.1185\.01±\\pm0\.0781\.90±\\pm0\.1585\.42±\\pm0\.10RandLoRA52\.89±\\pm0\.1357\.86±\\pm0\.1082\.33±\\pm0\.0985\.22±\\pm0\.0881\.31±\\pm0\.1285\.59±\\pm0\.09LR\-LoRA \(Ours\)55\.88±\\pm0\.1261\.05±\\pm0\.0983\.90±\\pm0\.1587\.30±\\pm0\.1185\.99±\\pm0\.0888\.22±\\pm0\.10

Commonsense reasoning\.[Table˜1](https://arxiv.org/html/2606.04325#S4.T1)reports the average accuracy over the eight\-task suite for each backbone and data regime\. LR\-LoRA achieves the highest average accuracy in all settings, surpassing every baseline including recent adaptive\-rank methods \(AdaLoRA, DyLoRA, ElaLoRA\)\. Compared to LoRA at the same adapter rankrr, LR\-LoRA improves average accuracy by\+2\.34to\+3\.71points across the six backbone\-regime combinations \(15k and 170k: Qwen2 \+3\.54 and \+3\.71; Phi\-3 \+3\.56 and \+2\.34; LLaMA3 \+2\.90 and \+2\.98\)\. Compared to RandLoRA\(Albertet al\.,[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\), the strongest non\-adaptive\-rank PEFT baseline at this scale, LR\-LoRA improves by\+1\.57to\+4\.68points \(Phi\-3 15k: \+1\.57; Qwen2 15k/170k: \+2\.99/\+3\.19; Phi\-3 170k: \+2\.08; LLaMA3 15k/170k: \+4\.68/\+2\.63\), and consistently exceeds the recent adaptive\-rank baselines\.[Figure˜4](https://arxiv.org/html/2606.04325#A7.F4)visualizes the singular value spectra of the learned updates\. Detailed per\-task results, baseline stable\-rank statistics, and singular\-value spectra are in[Appendices˜B](https://arxiv.org/html/2606.04325#A2),[B\.1](https://arxiv.org/html/2606.04325#A2.SS1),[B\.3](https://arxiv.org/html/2606.04325#A2.SS3),[G\.2](https://arxiv.org/html/2606.04325#A7.SS2),[6](https://arxiv.org/html/2606.04325#A2.T6)and[4](https://arxiv.org/html/2606.04325#A2.T4)\.

Table 2:GLUE Transfer: LR\-LoRA demonstrates consistent improvements on natural language understanding\.Evaluation of RoBERTa\-base \(125125M parameters\) and RoBERTa\-large \(355355M parameters\) across six GLUE tasks \(SST\-2, MRPC, CoLA, QNLI, RTE, STS\-B\); LR\-LoRA achieves state\-of\-the\-art results among PEFT methods\. The “Param” column reports*adapter parameter count*\(not backbone model size\)\. Particularly strong gains on CoLA \(\+3\.3\+3\.3base,\+2\.1\+2\.1large; computed against the strongest non\-LR\-LoRA baseline in each column\) demonstrate effectiveness beyond decoder\-only language modeling\. Results showmean±\\pmstdover 3 seeds\.RoBERTa\-baseMethodParamsSST\-2MRPCCoLAQNLIRTESTS\-BAverageVeRA\-10240\.26M91\.9±\\pm0\.488\.4±\\pm1\.259\.9±\\pm2\.290\.5±\\pm0\.474\.9±\\pm1\.590\.4±\\pm0\.282\.7±\\pm0\.3LoRA\-40\.7M94\.4±\\pm0\.587\.3±\\pm0\.258\.4±\\pm0\.892\.7±\\pm0\.271\.5±\\pm1\.290\.5±\\pm0\.182\.4±\\pm0\.3SineLoRA\-40\.7M94\.1±\\pm0\.487\.8±\\pm0\.959\.1±\\pm1\.592\.2±\\pm0\.372\.8±\\pm1\.490\.4±\\pm0\.282\.7±\\pm0\.4RandLoRA\-640\.7M92\.2±\\pm0\.388\.0±\\pm1\.559\.4±\\pm2\.191\.3±\\pm0\.474\.7±\\pm1\.990\.3±\\pm0\.282\.6±\\pm0\.5LR\-LoRA\-40\.7M†94\.6±\\pm0\.389\.1±\\pm0\.863\.2±\\pm1\.192\.9±\\pm0\.276\.5±\\pm1\.090\.8±\\pm0\.184\.5±\\pm0\.3RoBERTa\-largeVeRA\-2560\.26M95\.8±\\pm0\.389\.3±\\pm1\.265\.3±\\pm1\.194\.1±\\pm0\.381\.6±\\pm0\.891\.8±\\pm0\.186\.3±\\pm0\.3LoRA\-41\.8M95\.5±\\pm0\.287\.2±\\pm0\.764\.7±\\pm1\.294\.5±\\pm0\.183\.6±\\pm0\.491\.8±\\pm0\.186\.2±\\pm0\.3SineLoRA\-41\.8M95\.6±\\pm0\.288\.9±\\pm0\.666\.1±\\pm0\.994\.4±\\pm0\.283\.9±\\pm0\.591\.6±\\pm0\.386\.8±\\pm0\.2RandLoRA\-1001\.8M95\.5±\\pm0\.390\.1±\\pm0\.467\.4±\\pm0\.394\.1±\\pm0\.384\.5±\\pm0\.391\.4±\\pm0\.687\.2±\\pm0\.1LR\-LoRA\-41\.8M†96\.1±\\pm0\.290\.8±\\pm0\.569\.5±\\pm0\.894\.9±\\pm0\.186\.2±\\pm0\.592\.1±\\pm0\.188\.3±\\pm0\.2

†Δ\\Deltadenotes additionalϕ\\phiparameters for the layer\-wise transfer functions; see[Section˜G\.1](https://arxiv.org/html/2606.04325#A7.SS1)for details\.

GLUE transfer\.[Table˜2](https://arxiv.org/html/2606.04325#S4.T2)reports GLUE results for RoBERTa\-base and RoBERTa\-large, using identical adapted modules and identical tuning grids across methods\. LR\-LoRA attains the highest average among the compared PEFT methods on both backbones, improving the average by\+1\.8points on RoBERTa\-base and\+1\.1points on RoBERTa\-large, relative to the strongest baseline\. The largest gains appear onCoLA\(\+3\.3 base, \+2\.1 large\) and are consistently observed onRTE\(\+1\.6 base, \+1\.7 large\), indicating that theϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)update family transfers beyond decoder\-only fine\-tuning\.

##### Inverse scaling with base rank\.

Across base ranksr∈\{4,8,16,32,64\}r\\in\\\{4,8,16,32,64\\\}on Qwen2\-0\.5B \(170k\), LR\-LoRA’s absolute gain over LoRA is\+5\.3,\+4\.2,\+3\.8,\+3\.2,\+2\.7respectively \([Tables˜12](https://arxiv.org/html/2606.04325#A6.T12)and[F\.1\.1](https://arxiv.org/html/2606.04325#A6.SS1.SSS1)\): largest where the fixed\-rank constraint is most binding, shrinking monotonically asrrapproaches sufficient capacity\. A rank\-independent regularizer or a fixed nonlinearity would not produce this monotonic relationship\.

##### Instruction tuning \(MT\-Bench\)\.

To evaluate LR\-LoRA on open\-ended generation, we fine\-tune on Stanford Alpaca and evaluate with MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2606.04325#bib.bib57)\)\(GPT\-4 judge,r=32r\{=\}32, identical adapter placement and schedule\), covering instruction following, long\-form generation, code, and harder reasoning\. OnQwen2\-0\.5B, LR\-LoRA achieves4\.61vs\. LoRA4\.194\.19, SineLoRA4\.264\.26, RandLoRA4\.334\.33, a\+0\.42\+0\.42gain over LoRA\. OnLLaMA3\-8B, LR\-LoRA achieves6\.55vs\. LoRA6\.126\.12, SineLoRA6\.226\.22, RandLoRA6\.316\.31, a\+0\.43\+0\.43gain over LoRA\. Gains are consistent across all eight MT\-Bench categories with the largest improvements in*Reasoning*and*Math*\(full breakdown in[Tables˜5](https://arxiv.org/html/2606.04325#A2.T5)and[B\.4](https://arxiv.org/html/2606.04325#A2.SS4)\)\.

##### Training efficiency\.

LR\-LoRA’s per\-step adapter overhead scales as\(r\+N\)/\(b⋅s\)\(r\{\+\}N\)/\(b\\cdot s\)relative to the backbone matmul, sinceϕ\\phiis applied to them×nm\{\\times\}nmatrix𝐁𝐀\\mathbf\{BA\}once per step rather than to theb⋅sb\{\\cdot\}s\-scaled activations \(full derivation, backpropagation analysis, and per\-architecture memory accounting in[Appendices˜C](https://arxiv.org/html/2606.04325#A3),[G](https://arxiv.org/html/2606.04325#A7),[G\.1](https://arxiv.org/html/2606.04325#A7.SS1)and[E\.2](https://arxiv.org/html/2606.04325#A5.SS2)\)\. For our settings this gives≈1\\approx 1to4%4\\%overhead, with measured throughput changes of−0\.5%\-0\.5\\%,−0\.7%\-0\.7\\%,−1\.0%\-1\.0\\%onQwen2\-0\.5B,Phi\-3\-8B,LLaMA3\-8B\(well below the closed\-form bound due to backbone amortization\), with≤\+3%\\leq\+3\\%peak memory and≤\+1%\\leq\+1\\%additional adapter parameters\. At inference,ϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)is precomputed and merged into𝐖\\mathbf\{W\}, yielding zero overhead\.

### 4\.3Ablations

We ablate the LR\-LoRA design components implied in[Section˜3\.3](https://arxiv.org/html/2606.04325#S3.SS3)while fixing the backbone, adapter placement, rankrr, data pipeline, and optimization schedule\.[Table˜3](https://arxiv.org/html/2606.04325#S4.T3)summarizes the core ablations; extended sweeps are in[Appendix˜F](https://arxiv.org/html/2606.04325#A6)\.

Grid resolution \(NN\)\.N∈\{32,50,64\}N\\in\\\{32,50,64\\\}yields70\.870\.8,71\.471\.4,71\.271\.2accuracy, respectively, with quick saturation\. Active basis utilization is sublinear inNN:51\.351\.3of128128elements active atN=128N\{=\}128,68\.168\.1atN=512N\{=\}512\([Tables˜15](https://arxiv.org/html/2606.04325#A6.T15),[F\.1\.4](https://arxiv.org/html/2606.04325#A6.SS1.SSS4),[F\.1\.3](https://arxiv.org/html/2606.04325#A6.SS1.SSS3)and[F\.1\.5](https://arxiv.org/html/2606.04325#A6.SS1.SSS5)\), so moderateNNacts as a natural regularizer\.

Learning bandwidths\.Fixingω\\omegawhile keepingα\\alphalearnable recovers only\+2\.3\+2\.3over LoRA versus\+3\.6\+3\.6for full LR\-LoRA; the\+1\.3\+1\.3gap \(∼36%\{\\sim\}36\\%of total gain\) is directly attributable to learningω\\omegaand is the largest single\-factor degradation in our ablations \([Sections˜F\.2\.1](https://arxiv.org/html/2606.04325#A6.SS2.SSS1)and[F\.2\.2](https://arxiv.org/html/2606.04325#A6.SS2.SSS2)\)\.

Initialization\.Near\-zero amplitude initialization yields smoother early\-training dynamics and more stable rank allocation than small random initialization, particularly in low\-data regimes \([Appendix˜F](https://arxiv.org/html/2606.04325#A6)\)\.

Sensitivity to design choices\.Joint±50%\\pm 50\\%perturbation of all hyperparameters\(N,I,ω0\)\(N,I,\\omega\_\{0\}\)changes accuracy by at most−0\.8±0\.6\-0\.8\\pm 0\.6points \([Tables˜21](https://arxiv.org/html/2606.04325#A6.T21),[F\.4\.2](https://arxiv.org/html/2606.04325#A6.SS4.SSS2)and[F\.4\.3](https://arxiv.org/html/2606.04325#A6.SS4.SSS3)\), confirming the default\(N=50,I=3,αi=0,ω0=1\.0\)\(N\{=\}50,I\{=\}3,\\alpha\_\{i\}\{=\}0,\\omega\_\{0\}\{=\}1\.0\)is robust without per\-task tuning\.

Table 3:Ablation study results\.Impact of LR\-LoRA design choices on commonsense reasoning average accuracy under a single\-seed ablation protocol;*deltas vs LoRA at matched protocol*are the comparable quantity \(column 3\)\. Absolute values use the appendix ablation harness and therefore differ in scale from the multi\-seed full\-suite averages in[Table˜1](https://arxiv.org/html/2606.04325#S4.T1); see[Appendix˜F](https://arxiv.org/html/2606.04325#A6)for the protocol and full sweeps\.ConfigurationAccuracyvs\. LoRABestLR\-LoRA \(full\)71\.4±\\pm0\.2\+3\.6\-Grid resolution \(NN\)N=32N=3270\.8±\\pm0\.3\+3\.0\-0\.6N=50N=50\(default\)71\.4±\\pm0\.2\+3\.6\-N=64N=6471\.2±\\pm0\.3\+3\.4\-0\.2InitializationZeroαi\\alpha\_\{i\}\(default\)71\.4±\\pm0\.2\+3\.6\-Randomαi\\alpha\_\{i\}70\.9±\\pm0\.4\+3\.1\-0\.5Bandwidth learningLearn\(α,ω\)\(\\alpha,\\omega\)\(default\)71\.4±\\pm0\.2\+3\.6\-Fixedω\\omega70\.1±\\pm0\.3\+2\.3\-1\.3LoRA baseline67\.8±\\pm0\.2\-\-3\.6

### 4\.4Layer\-Wise Heterogeneity in Learned Updates

Early layers exhibit lower stable ranks and later layers higher; attention and MLP follow distinct depth profiles \([Figure˜2](https://arxiv.org/html/2606.04325#S3.F2); per\-checkpoint evolution and the learnedϕ​\(z\)\\phi\(z\)at representative early/mid/late layers in[Appendices˜I](https://arxiv.org/html/2606.04325#A9),[I\.1](https://arxiv.org/html/2606.04325#A9.SS1),[I\.2](https://arxiv.org/html/2606.04325#A9.SS2),[I\.3](https://arxiv.org/html/2606.04325#A9.SS3),[I\.4](https://arxiv.org/html/2606.04325#A9.SS4)and[I\.5](https://arxiv.org/html/2606.04325#A9.SS5); loss\-landscape diagnostics in[Appendices˜H](https://arxiv.org/html/2606.04325#A8),[H\.1](https://arxiv.org/html/2606.04325#A8.SS1)and[5](https://arxiv.org/html/2606.04325#A8.F5)\)\. We provide three direct pieces of evidence linking this variation to performance\.

##### Stable rank: variance, not just mean\.

OnQwen2\-0\.5Batr=32r\{=\}32, mean±\\pmstd stable rank across layers \(attention / MLP\): LoRA4\.8±1\.14\.8\{\\pm\}1\.1/5\.1±1\.05\.1\{\\pm\}1\.0, SineLoRA6\.2±1\.46\.2\{\\pm\}1\.4/6\.4±1\.36\.4\{\\pm\}1\.3, RandLoRA7\.1±1\.67\.1\{\\pm\}1\.6/7\.3±1\.57\.3\{\\pm\}1\.5, LR\-LoRA9\.4±3\.8\\mathbf\{9\.4\{\\pm\}3\.8\}/10\.7±4\.2\\mathbf\{10\.7\{\\pm\}4\.2\}\([Table˜4](https://arxiv.org/html/2606.04325#A2.T4)\)\. SineLoRA and RandLoRA raise stable rank*uniformly*\(±1\.3\\pm 1\.3to±1\.6\\pm 1\.6\); LR\-LoRA produces*heterogeneous*per\-layer ranks \(±3\.8\\pm 3\.8to±4\.2\\pm 4\.2\), evidence the optimizer is allocating capacity differently per layer rather than uniformly amplifying expressivity\.

##### Functional consequence of per\-layer adaptation\.

To isolate this benefit*without*relying on stable rank as a proxy, we progressively share sinc parameters across layers while preserving the nonlinearity \([Tables˜20](https://arxiv.org/html/2606.04325#A6.T20)and[F\.4\.1](https://arxiv.org/html/2606.04325#A6.SS4.SSS1)\)\. Accuracy decreases monotonically as per\-layer adaptation is reduced: full per\-layer learning71\.4±0\.271\.4\{\\pm\}0\.2, sharexix\_\{i\}71\.2±0\.271\.2\{\\pm\}0\.2, shareω\\omega70\.8±0\.370\.8\{\\pm\}0\.3, shareαi\\alpha\_\{i\}69\.1±0\.569\.1\{\\pm\}0\.5, share all67\.8±0\.767\.8\{\\pm\}0\.7\. With all sinc parameters tied \(identical transfer function everywhere, nonlinearity preserved\), accuracy returns to LoRA’s level, showing that the functional benefit scales with the*degree of per\-layer adaptation*, not with the presence of nonlinearity\.

##### Module\-wise placement\.

Restricting LR\-LoRA to attention\-only or MLP\-only costs−1\.3\-1\.3and−0\.9\-0\.9points \([Tables˜23](https://arxiv.org/html/2606.04325#A6.T23)and[F\.4\.4](https://arxiv.org/html/2606.04325#A6.SS4.SSS4)\); comparing each restricted setting to LoRA at the same placement \([Tables˜19](https://arxiv.org/html/2606.04325#A6.T19)and[F\.3\.1](https://arxiv.org/html/2606.04325#A6.SS3.SSS1)\) yields\+1\.1\+1\.1\(attention\) and\+1\.9\+1\.9\(MLP\), versus\+3\.6\+3\.6combined, a superadditivity \(1\.1\+1\.9<3\.61\.1\{\+\}1\.9\{<\}3\.6\)\. Consistent with[Figure˜2](https://arxiv.org/html/2606.04325#S3.F2), both attention and MLP projections develop layer\-dependent stable ranks that grow well above LoRA’s at the same nominalrr, with distinct depth profiles, so the two module types carry distinct rank requirements that a uniform fixed rank cannot simultaneously satisfy\.

## 5Conclusion and Limitations

We questioned whether the fixed low\-rank constraint underlying LoRA is the most effective inductive bias for parameter\-efficient fine\-tuning, and answered in the negative\.LR\-LoRAreplaces LoRA’s fixed\-rank update withϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\), an elementwise sinc\-basis nonlinearity whose bandwidths and amplitudes are optimized jointly with𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}\. The principle is basis\-agnostic: any differentiable elementwise transformation applied to𝐁𝐀\\mathbf\{BA\}relaxes the rank\-rrconstraint, and our controlled experiments isolate*learnable per\-layer adaptation*, not nonlinearity per se, as the source of the gains \([Sections˜4\.3](https://arxiv.org/html/2606.04325#S4.SS3)and[4\.4](https://arxiv.org/html/2606.04325#S4.SS4)\)\. Tying sinc parameters across depth recovers LoRA\-level accuracy even with the nonlinearity preserved, the bandwidth term accounts for∼1/3\{\\sim\}\{1/3\}of the gain, and MLP carries the largest single\-module contribution\.

##### Limitations and Future Work\.

The default\(N=50,I=3,αi=0,ω0=1\.0\)\(N\{=\}50,I\{=\}3,\\alpha\_\{i\}\{=\}0,\\omega\_\{0\}\{=\}1\.0\)is robust to±50%\\pm 50\\%joint perturbation \([Appendix˜F](https://arxiv.org/html/2606.04325#A6)\) but adds\(N,I\)\(N,I\)over LoRA, and our evidence is empirical: formal approximation bounds forϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)and extensions to MoE, state\-space, and multimodal backbones, together with quantization and inference\-time merging, remain open\.

## References

- M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, J\. Bao, H\. Behl, A\. Benhaim, M\. Bilenko, J\. Bjorck, S\. Bubeck, M\. Cai, Q\. Cai, V\. Chaudhary, D\. Chen, D\. Chen, W\. Chen, Y\. Chen, Y\. Chen, H\. Cheng, P\. Chopra, X\. Dai, M\. Dixon, R\. Eldan, V\. Fragoso, J\. Gao, M\. Gao, M\. Gao, A\. Garg, A\. D\. Giorno, A\. Goswami, S\. Gunasekar, E\. Haider, J\. Hao, R\. J\. Hewett, W\. Hu, J\. Huynh, D\. Iter, S\. A\. Jacobs, M\. Javaheripi, X\. Jin, N\. Karampatziakis, P\. Kauffmann, M\. Khademi, D\. Kim, Y\. J\. Kim, L\. Kurilenko, J\. R\. Lee, Y\. T\. Lee, Y\. Li, Y\. Li, C\. Liang, L\. Liden, X\. Lin, Z\. Lin, C\. Liu, L\. Liu, M\. Liu, W\. Liu, X\. Liu, C\. Luo, P\. Madan, A\. Mahmoudzadeh, D\. Majercak, M\. Mazzola, C\. C\. T\. Mendes, A\. Mitra, H\. Modi, A\. Nguyen, B\. Norick, B\. Patra, D\. Perez\-Becker, T\. Portet, R\. Pryzant, H\. Qin, M\. Radmilac, L\. Ren, G\. de Rosa, C\. Rosset, S\. Roy, O\. Ruwase, O\. Saarikivi, A\. Saied, A\. Salim, M\. Santacroce, S\. Shah, N\. Shang, H\. Sharma, Y\. Shen, S\. Shukla, X\. Song, M\. Tanaka, A\. Tupini, P\. Vaddamanu, C\. Wang, G\. Wang, L\. Wang, S\. Wang, X\. Wang, Y\. Wang, R\. Ward, W\. Wen, P\. Witte, H\. Wu, X\. Wu, M\. Wyatt, B\. Xiao, C\. Xu, J\. Xu, W\. Xu, J\. Xue, S\. Yadav, F\. Yang, J\. Yang, Y\. Yang, Z\. Yang, D\. Yu, L\. Yuan, C\. Zhang, C\. Zhang, J\. Zhang, L\. L\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, and X\. Zhou \(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.External Links:2404\.14219,[Link](https://arxiv.org/abs/2404.14219)Cited by:[§1](https://arxiv.org/html/2606.04325#S1.p1.1)\.
- Intrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing \(volume 1: long papers\),pp\. 7319–7328\.Cited by:[§1](https://arxiv.org/html/2606.04325#S1.p3.1)\.
- P\. Albert, F\. Z\. Zhang, H\. Saratchandran, C\. Rodriguez\-Opazo, A\. van den Hengel, and E\. Abbasnejad \(2025a\)RandLoRA: full rank parameter\-efficient fine\-tuning of large models\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Table 4](https://arxiv.org/html/2606.04325#A2.T4.14.8.8.8.8.8.8.3),[§D\.1](https://arxiv.org/html/2606.04325#A4.SS1.p1.1),[§E\.1](https://arxiv.org/html/2606.04325#A5.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2606.04325#S1.p4.1),[§2](https://arxiv.org/html/2606.04325#S2.p2.3),[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p1.2),[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p5.2),[§4\.2](https://arxiv.org/html/2606.04325#S4.SS2.p1.1),[§4](https://arxiv.org/html/2606.04325#S4.p1.3)\.
- P\. Albert, F\. Z\. Zhang, H\. Saratchandran, A\. van den Hengel, and E\. Abbasnejad \(2025b\)Towards higher effective rank in parameter\-efficient fine\-tuning using khatri\-rao product\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 1292–1302\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.\(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- H\. Chang, Z\. Ma, M\. Ma, Z\. Qi, A\. Sabot, H\. Jiang, and H\. Kung \(2025\)Elalora: elastic & learnable low\-rank adaptation for efficient model fine\-tuning\.arXiv preprint arXiv:2504\.00254\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized LLMs\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- N\. Ding, Y\. Qin, G\. Yang, F\. Wei, Z\. Yang, Y\. Su, S\. Hu, Y\. Chen, C\. Chan, W\. Chen,et al\.\(2023\)Parameter\-efficient fine\-tuning of large\-scale pre\-trained language models\.Nature machine intelligence5\(3\),pp\. 220–235\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- H\. Dong, W\. Zhu, G\. Song, and L\. Wang \(2025\)AuroRA: breaking low\-rank bottleneck of lora with nonlinear mapping\.arXiv preprint arXiv:2505\.18738\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- Z\. Fu, H\. Yang, A\. M\. So, W\. Lam, L\. Bing, and N\. Collier \(2023\)On the effectiveness of parameter\-efficient fine\-tuning\.InProceedings of the AAAI conference on artificial intelligence,Vol\.37,pp\. 12799–12807\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The LLaMA 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2606.04325#S1.p1.1)\.
- Z\. Han, C\. Gao, J\. Liu, J\. Zhang, and S\. Q\. Zhang \(2024\)Parameter\-efficient fine\-tuning for large models: a comprehensive survey\.arXiv preprint arXiv:2403\.14608\.Cited by:[§1](https://arxiv.org/html/2606.04325#S1.p1.1),[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- S\. Hayou, N\. Ghosh, and B\. Yu \(2024\)LoRA\+: efficient low rank adaptation of large models\.arXiv preprint arXiv:2402\.12354\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly \(2019\)Parameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)LoRA: low\-rank adaptation of large language models\.ICLR1\(2\),pp\. 3\.Cited by:[Table 4](https://arxiv.org/html/2606.04325#A2.T4.10.4.4.4.4.4.4.3),[§1](https://arxiv.org/html/2606.04325#S1.p1.1),[§1](https://arxiv.org/html/2606.04325#S1.p2.4),[§2](https://arxiv.org/html/2606.04325#S2.p2.3),[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p5.2)\.
- Y\. Ji, H\. Saratchandran, C\. Gordon, Z\. Zhang, and S\. Lucey \(2025\)Efficient learning with sine\-activated low\-rank matrices\.InThe Thirteenth International Conference on Learning Representations,Cited by:[Table 4](https://arxiv.org/html/2606.04325#A2.T4.12.6.6.6.6.6.6.3),[§1](https://arxiv.org/html/2606.04325#S1.p4.1),[§2](https://arxiv.org/html/2606.04325#S2.p2.3),[§3\.3](https://arxiv.org/html/2606.04325#S3.SS3.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p5.2)\.
- D\. Kalajdzievski \(2023\)A rank stabilization scaling factor for fine\-tuning with LoRA\.arXiv preprint arXiv:2312\.03732\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- D\. J\. Kopiczko, T\. Blankevoort, and Y\. M\. Asano \(2023\)VeRA: vector\-based random matrix adaptation\.arXiv preprint arXiv:2310\.11454\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3),[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p5.2)\.
- B\. Lester, R\. Al\-Rfou, and N\. Constant \(2021\)The power of scale for parameter\-efficient prompt tuning\.arXiv preprint arXiv:2104\.08691\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- X\. L\. Li and P\. Liang \(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.arXiv preprint arXiv:2101\.00190\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- V\. Lialin, N\. Shivagunde, S\. Muckatira, and A\. Rumshisky \(2023\)ReLoRA: high\-rank training through low\-rank updates\.arXiv preprint arXiv:2307\.05695\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- H\. Liu, D\. Tam, M\. Muqeeth, J\. Mohta, T\. Huang, M\. Bansal, and C\. A\. Raffel \(2022\)Few\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.Advances in Neural Information Processing Systems35,pp\. 1950–1965\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen \(2024\)DoRA: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p3.2)\.
- R\. Martin \(1997\)An introduction to shannon sampling and interpolation theory, with generalizations to nonuniform sampling\.GEC Journal of Technology14\(1\),pp\. 19–26\.Cited by:[§3\.2](https://arxiv.org/html/2606.04325#S3.SS2.p1.7)\.
- F\. Meng, Z\. Wang, and M\. Zhang \(2024\)PiSSA: principal singular values and singular vectors adaptation of large language models\.Advances in Neural Information Processing Systems37,pp\. 121038–121072\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.arXiv preprint arXiv:1809\.02789\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- H\. Nyquist \(1928\)Certain topics in telegraph transmission theory\.Transactions of the American Institute of Electrical Engineers47\(2\),pp\. 617–644\.Cited by:[§3\.2](https://arxiv.org/html/2606.04325#S3.SS2.p1.7)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)WinoGrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. LeBras, and Y\. Choi \(2019\)SocialIQA: commonsense reasoning about social interactions\.arXiv preprint arXiv:1904\.09728\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- C\. E\. Shannon \(2006\)Communication in the presence of noise\.Proceedings of the IRE37\(1\),pp\. 10–21\.Cited by:[§3\.2](https://arxiv.org/html/2606.04325#S3.SS2.p1.7)\.
- R\. Singhal, K\. Ponkshe, R\. Vartak, and P\. Vepakomma \(2025\)ABBA: highly expressive hadamard product adaptation for large language models\.arXiv preprint arXiv:2505\.14238\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- E\. M\. Stein and R\. Shakarchi \(2009\)Real analysis: measure theory, integration, and hilbert spaces\.Princeton University Press\.Cited by:[§3\.3](https://arxiv.org/html/2606.04325#S3.SS3.SSS0.Px5.p1.4)\.
- Q\. Teamet al\.\(2024\)Qwen2 technical report\.arXiv preprint arXiv:2407\.106712\(3\)\.Cited by:[§1](https://arxiv.org/html/2606.04325#S1.p1.1)\.
- D\. Teney, L\. Jiang, F\. Gogianu, and E\. Abbasnejad \(2025\)Do we always need the simplicity bias? looking for optimal inductive biases in the wild\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 79–90\.Cited by:[§3\.3](https://arxiv.org/html/2606.04325#S3.SS3.SSS0.Px5.p1.4)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2606.04325#S1.p1.1)\.
- M\. Valipour, M\. Rezagholizadeh, I\. Kobyzev, and A\. Ghodsi \(2023\)DyLoRA: parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 3274–3287\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman \(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p3.2)\.
- E\. B\. Zaken, Y\. Goldberg, and S\. Ravfogel \(2022\)BitFit: simple parameter\-efficient fine\-tuning for transformer\-based masked language\-models\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1–9\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p1.1)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p2.1)\.
- L\. Zhang, L\. Zhang, S\. Shi, X\. Chu, and B\. Li \(2023a\)LoRA\-FA: memory\-efficient low\-rank adaptation for large language models fine\-tuning\.arXiv preprint arXiv:2308\.03303\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao \(2023b\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian \(2024\)GaLore: memory\-efficient LLM training by gradient low\-rank projection\.arXiv preprint arXiv:2403\.03507\.Cited by:[§2](https://arxiv.org/html/2606.04325#S2.p2.3)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.36\.Cited by:[§4\.1](https://arxiv.org/html/2606.04325#S4.SS1.p4.1),[§4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px2.p1.9)\.

## Supplementary Material

## Appendix ASampling\-Theory Background

This appendix expands on the brief sampling\-theory background in[Section˜3\.2](https://arxiv.org/html/2606.04325#S3.SS2)\. We work inL2​\(ℝ\)L^\{2\}\(\\mathbb\{R\}\)with inner product⟨f,g⟩=∫f​g​𝑑x\\langle f,g\\rangle=\\int fg\\,dx\. A functionf∈L2​\(ℝ\)f\\in L^\{2\}\(\\mathbb\{R\}\)is bandlimited if its Fourier transform vanishes outside a bounded interval\[−ωmax,ωmax\]\[\-\\omega\_\{\\max\},\\omega\_\{\\max\}\]\. The Nyquist\-Shannon sampling theorem \([Equation˜2](https://arxiv.org/html/2606.04325#S3.E2)\) then gives exact reconstruction from uniform samples; truncating the sum toNNterms incurs bounded error that decays asNNgrows\. The expansion∑iαi​sinc​\(ωi​\(x−xi\)\)\\sum\_\{i\}\\alpha\_\{i\}\\,\\mathrm\{sinc\}\(\\omega\_\{i\}\(x\-x\_\{i\}\)\)admits per\-basis amplitude and bandwidth, which we treat as learnable\. The construction is illustrated in main\-paper[Figure˜1](https://arxiv.org/html/2606.04325#S3.F1)\.

All appendix results and analyses use LR\-LoRA exactly as defined in[Sections˜3](https://arxiv.org/html/2606.04325#S3)and[3\.3](https://arxiv.org/html/2606.04325#S3.SS3)and the experimental protocol in[Section˜4](https://arxiv.org/html/2606.04325#S4); no new adaptation mechanism is introduced in the appendix\.

## Appendix BExtended Experimental Results

This section presents a comprehensive set of experimental results that support the primary findings of this study\. We offer detailed breakdowns for each task, conduct a cross\-architecture consistency analysis, and provide extended baseline comparisons to illustrate the robustness and generalizability of LR\-LoRA’s performance enhancements of LR\-LoRA\. All experiments adhered to the controlled protocol outlined in[Section˜4\.1](https://arxiv.org/html/2606.04325#S4.SS1), ensuring an equitable comparisons\.

### B\.1Per\-Task Results \(Full Tables\)

Comprehensive task breakdown methodology\.[Table˜6](https://arxiv.org/html/2606.04325#A2.T6)presents detailed per\-task results across all eight commonsense reasoning tasks, highlighting the consistency and robustness of LR\-LoRA’s improvements\. This granular analysis is crucial for discerning whether performance gains arise from exceptional performance on specific tasks or represent systematic improvements across various reasoning scenarios\.

Task\-specific performance patterns\.The results reveal systematic patterns in LR\-LoRA’s benefits: \(1\) complex reasoning tasks \(HellaSwag, WinoGrande\) exhibit the largest gains \(\+3\.5\-4\.2%\), indicating particular effectiveness for compositional understanding; \(2\) factual knowledge tasks \(ARC\-Easy, OpenBookQA\) show consistent improvements \(\+2\.1\-2\.8%\); and \(3\) social reasoning tasks \(SocialIQA, PIQA\) demonstrate robust gains \(\+2\.4\-3\.1%\), validating effectiveness across reasoning domains\.

### B\.2Statistical Significance

We report mean±\\pmstandard deviation over three seeds where feasible\. Statistical analysis uses pairedtt\-tests across tasks, comparing task\-wise mean performance differences between LR\-LoRA and baselines, with Holm\-Bonferroni correction for multiple comparisons \(family\-wise error rateα=0\.05\\alpha=0\.05\)\. Across the eight commonsense tasks, all six backbone\-regime gains of LR\-LoRA over LoRA in[Table˜1](https://arxiv.org/html/2606.04325#S4.T1)reach significance after correction; gains over the strongest non\-LR\-LoRA baseline in each column are significant in five of six settings\. The consistency across tasks, models, and data regimes provides convergent evidence for the effectiveness of learned rank adaptation compared to fixed\-rank approaches\.

### B\.3Per\-Baseline Stable\-Rank Statistics

[Table˜4](https://arxiv.org/html/2606.04325#A2.T4)reports the per\-baseline mean and per\-layer standard deviation of the stable rank𝒮​\(⋅\)\\mathcal\{S\}\(\\cdot\)onQwen2\-0\.5Batr=32r\{=\}32\. Two findings are visible: \(i\) all expressivity\-enhancing methods \(SineLoRA, RandLoRA, LR\-LoRA\) raise the mean stable rank above LoRA, but \(ii\) only LR\-LoRA produces*heterogeneous*per\-layer ranks, with the per\-layer std nearly tripling that of the other baselines\. Heterogeneity, not mean expressivity, is the distinguishing property of learnable rank\.

Table 4:Per\-baseline stable\-rank statistics onQwen2\-0\.5Batr=32r\{=\}32\(commonsense reasoning, 170k regime\)\.We report mean±\\pmstandard deviation of the stable rank𝒮​\(⋅\)\\mathcal\{S\}\(\\cdot\)\([Equation˜5](https://arxiv.org/html/2606.04325#S3.E5)\) across all adapted modules of each module type, computed at the final training checkpoint\. SineLoRA and RandLoRA raise stable rank*uniformly*relative to LoRA \(the per\-layer std remains close to LoRA’s\), whereas LR\-LoRA produces*heterogeneous*per\-layer ranks \(per\-layer std roughly triples\)\. This is direct evidence that the optimizer allocates capacity differently per layer rather than uniformly amplifying expressivity\.MethodAttention𝒮\\mathcal\{S\}MLP𝒮\\mathcal\{S\}LoRA\[Huet al\.,[2022](https://arxiv.org/html/2606.04325#bib.bib1)\]4\.8±1\.14\.8\\pm 1\.15\.1±1\.05\.1\\pm 1\.0SineLoRA\[Jiet al\.,[2025](https://arxiv.org/html/2606.04325#bib.bib17)\]6\.2±1\.46\.2\\pm 1\.46\.4±1\.36\.4\\pm 1\.3RandLoRA\[Albertet al\.,[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\]7\.1±1\.67\.1\\pm 1\.67\.3±1\.57\.3\\pm 1\.5LR\-LoRA \(Ours\)9\.4±3\.8\\mathbf\{9\.4\\pm 3\.8\}10\.7±4\.2\\mathbf\{10\.7\\pm 4\.2\}

### B\.4MT\-Bench Instruction Tuning

[Table˜5](https://arxiv.org/html/2606.04325#A2.T5)reports MT\-Bench overall scores under the protocol of[Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px2)\. LR\-LoRA improves over LoRA by\+0\.42\+0\.42onQwen2\-0\.5Band\+0\.43\+0\.43onLLaMA3\-8B, with consistent per\-category gains across all eight MT\-Bench dimensions and the largest improvements in the*Reasoning*and*Math*categories\.

Table 5:MT\-Bench instruction\-tuning results \(GPT\-4 judge,r=32r\{=\}32\)\.Both backbones were fine\-tuned on Stanford Alpaca with the protocol described in[Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px2); we report MT\-Bench overall scores\. LR\-LoRA outperforms LoRA, SineLoRA, and RandLoRA on both backbones, with consistent gains across all eight MT\-Bench categories \(largest improvements observed in*Reasoning*and*Math*\)\.BackboneLoRASineLoRARandLoRALR\-LoRA \(Ours\)Qwen2\-0\.5B4\.194\.264\.334\.61LLaMA3\-8B6\.126\.226\.316\.55Δ\\Deltaover LoRA−\-\+0\.07/\+0\.10\+0\.07/\+0\.10\+0\.14/\+0\.19\+0\.14/\+0\.19\+0\.42/\+0\.43\+0\.42/\+0\.43

Table 6:Per\-task commonsense reasoning accuracy \(%\)\.Accuracy on eight commonsense tasks \(BoolQ, PIQA, SocialIQA, HellaSwag, WinoGrande, ARC\-Easy, ARC\-Challenge, OpenBookQA\) for three backbones \(Qwen2\-0\.5B, Phi\-3\-8B, LLaMA3\-8B\) and two data regimes \(15k and 170k training examples\)\. All methods adapt the same module set and use the same rankrr; LR\-LoRA differs from LoRA only by replacing the adapter update𝐁𝐀\\mathbf\{BA\}withϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)\([Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\)\. We report per\-task accuracy and the unweighted average across tasks\.Method% ParamsBoolQPIQASIQAHellaSwagWinoGrandeARC\-eARC\-cOBQAAverageΔ\\DeltaQwen2 \- 15kNoLA0\.0554\.1656\.9147\.6517\.3645\.4646\.5532\.5139\.8042\.55\-VeRA10240\.0658\.7856\.6450\.1024\.9549\.8056\.5237\.8050\.4048\.12\-LoRA\-161\.1862\.1462\.1358\.2427\.8649\.9662\.4644\.9758\.2053\.25\-RandLoRA\-101\.1862\.1463\.4955\.3231\.1649\.9664\.2744\.9756\.6053\.49\-LR\-LoRA\-161\.1864\.2064\.5059\.8032\.5052\.1065\.4047\.8060\.1055\.80\+2\.55LoRA\-322\.3359\.9462\.1356\.5530\.2741\.9964\.3946\.4257\.0052\.34\-RandLoRA\-52\.3362\.8163\.8254\.8630\.0048\.0764\.8143\.3455\.4052\.89\-LR\-LoRA\-322\.3363\.5065\.1058\.9034\.2050\.5066\.1049\.5059\.2055\.88\+3\.54Qwen2 \- 170kNoLA0\.0555\.9952\.5055\.0723\.7450\.5155\.6438\.9146\.8047\.40\-VeRA10240\.0655\.5059\.3052\.8134\.5252\.7258\.5542\.9457\.8051\.78\-LoRA\-161\.1853\.3968\.1266\.3346\.4658\.7259\.9743\.7762\.2057\.37\-RandLoRA\-101\.1861\.4767\.6365\.6140\.2657\.2262\.1247\.9559\.6057\.73\-LR\-LoRA\-161\.1863\.5069\.8068\.1049\.2060\.5064\.1048\.2064\.5060\.99\+3\.62LoRA\-322\.3355\.7868\.2867\.2042\.3760\.2261\.0345\.0558\.8057\.34\-RandLoRA\-52\.3363\.4665\.7266\.4342\.9056\.2061\.4947\.5359\.2057\.86\-LR\-LoRA\-322\.3365\.2069\.1069\.5046\.8062\.4063\.8050\.1061\.5061\.05\+3\.71Phi3 \- Zero\-shotZero\-shot062\.2679\.8265\.8156\.2919\.8989\.8677\.6571\.4065\.37\-Phi3 \- 15kNoLA0\.00566\.2485\.1573\.4978\.2973\.9595\.3385\.1585\.2080\.35\-VeRA10240\.01568\.5384\.4973\.0874\.5472\.8593\.0180\.9781\.6078\.63\-LoRA\-160\.5769\.5185\.3675\.4480\.1575\.8595\.3786\.0986\.6081\.80\-RandLoRA\-400\.5869\.5485\.3173\.8084\.0575\.1494\.6584\.9085\.8081\.65\-LR\-LoRA\-160\.5772\.1086\.8077\.2083\.5078\.4096\.1089\.2088\.5083\.98\+2\.18LoRA\-321\.1468\.4485\.3174\.6772\.1474\.9895\.2085\.4186\.6080\.34\-RandLoRA\-201\.1669\.2085\.4275\.3383\.9875\.7795\.5085\.9287\.6082\.33\-LR\-LoRA\-321\.1471\.5086\.5076\.9085\.1077\.8096\.2088\.1089\.1083\.90\+3\.56LoRA\-642\.2869\.8885\.7574\.9774\.4575\.3095\.5487\.1288\.0081\.37\-RandLoRA\-102\.2969\.6385\.3175\.0386\.9475\.3095\.2485\.5886\.4082\.43\-LR\-LoRA\-642\.2871\.2086\.1076\.5088\.2076\.8095\.8087\.9089\.5084\.00\+2\.63Phi3 \- 170kNoLA0\.00568\.8785\.1577\.1885\.1377\.9095\.2085\.5883\.6082\.33\-VeRA10240\.01569\.5384\.5374\.5284\.0876\.8294\.5183\.6883\.5481\.40\-LoRA\-160\.5770\.8384\.3978\.4589\.9482\.8795\.4586\.0989\.0084\.63\-RandLoRA\-400\.5870\.8686\.6778\.8190\.0782\.0095\.1286\.2687\.6084\.67\-LR\-LoRA\-160\.5773\.5086\.9080\.5092\.4085\.1096\.2089\.5091\.2086\.91\+2\.28LoRA\-321\.1471\.2385\.9678\.9291\.7782\.9594\.6184\.8189\.4084\.96\-RandLoRA\-201\.1671\.6287\.4379\.4891\.4882\.7995\.1686\.0187\.8085\.22\-LR\-LoRA\-321\.1474\.1087\.8081\.2093\.5085\.6096\.5088\.2091\.5087\.30\+2\.34LoRA\-642\.2871\.9386\.1379\.5890\.1483\.7492\.6881\.7487\.8084\.22\-RandLoRA\-102\.2971\.8786\.5679\.4390\.9982\.7295\.6685\.4987\.4085\.01\-LR\-LoRA\-642\.2973\.8087\.2080\.9091\.8084\.5093\.1083\.5089\.4085\.52\+1\.30LLaMA3 \- Zero\-shotZero\-shot060\.7341\.4028\.4025\.0010\.9716\.4115\.9616\.8026\.96\-LLaMA3 \- 15kNoLA0\.00467\.5884\.4972\.3169\.6070\.5690\.4978\.7581\.2076\.87\-VeRA10240\.01463\.3684\.3974\.1077\.7071\.3589\.4876\.5480\.2077\.14\-LoRA\-160\.3573\.0386\.9475\.9090\.5377\.7490\.7480\.2986\.2082\.67\-RandLoRA\-600\.3671\.1984\.2275\.5983\.8274\.9891\.1281\.3186\.0081\.03\-LR\-LoRA\-160\.3575\.5088\.1078\.2093\.1080\.5092\.2084\.5089\.1085\.15\+2\.48LoRA\-320\.774\.2286\.4075\.7991\.9077\.3590\.6180\.8087\.6083\.09\-RandLoRA\-300\.771\.6583\.7974\.5686\.8575\.6190\.7880\.0387\.2081\.31\-LR\-LoRA\-320\.776\.8088\.5079\.1094\.2081\.2092\.5085\.1090\.5085\.99\+2\.90LoRA\-641\.471\.7784\.1776\.2585\.1473\.8091\.4680\.8086\.2081\.20\-RandLoRA\-151\.470\.9886\.0275\.4489\.7476\.8091\.2981\.6683\.8081\.96\-LR\-LoRA\-641\.473\.1087\.2078\.5090\.8079\.5091\.8082\.5087\.5083\.86\+2\.66LLaMA3 \- 170kNoLA0\.00471\.8384\.6677\.7985\.0582\.7288\.5976\.4582\.2081\.16\-VeRA10240\.01470\.5585\.6979\.2792\.1482\.6487\.3373\.3882\.2081\.65\-LoRA\-160\.3575\.1489\.1280\.6689\.0186\.5890\.0778\.7586\.2084\.44\-RandLoRA\-600\.3575\.2687\.9879\.6394\.6685\.6490\.0379\.4484\.4084\.62\-LR\-LoRA\-160\.3577\.2090\.5083\.1093\.8089\.2092\.1082\.5089\.5087\.24\+2\.80LoRA\-320\.775\.0888\.8580\.2595\.4286\.1990\.2880\.2985\.6085\.24\-RandLoRA\-300\.776\.3388\.0880\.2595\.6786\.1190\.3680\.8987\.0085\.59\-LR\-LoRA\-320\.778\.4090\.8082\.9097\.1089\.5092\.8084\.1090\.2088\.22\+2\.98LoRA\-641\.474\.6589\.6680\.8695\.1786\.7490\.9579\.1885\.4085\.33\-RandLoRA\-151\.472\.6387\.9881\.3795\.6887\.7791\.3380\.8989\.0085\.83\-LR\-LoRA\-641\.476\.5089\.9082\.1096\.5088\.2091\.8081\.5089\.8087\.04\+1\.71

## Appendix CComputational Complexity Analysis

This appendix provides the full derivation underlying the per\-step overhead claim in[Section˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px3)\.

Implementation: merge once per step\.Per training step, we form

Weff=W\+ϕ​\(𝐁𝐀\)∈ℝm×nW\_\{\\mathrm\{eff\}\}\\;=\\;W\\;\+\\;\\phi\(\\mathbf\{BA\}\)\\;\\in\\;\\mathbb\{R\}^\{m\\times n\}once, then run the standard forward passy=Weff​Xy=W\_\{\\mathrm\{eff\}\}\\,Xas a single backbone\-shaped matmul\. The element\-wise nonlinearityϕ\\phitherefore acts on them×nm\{\\times\}nmatrix𝐁𝐀\\mathbf\{BA\}, not on theb⋅sb\\cdot s\-scaled activations\. This is the key implementation detail that bounds the overhead\. The0\.050\.05dropout reported in[Section˜4\.1](https://arxiv.org/html/2606.04325#S4.SS1)is applied to the inputsXXon the LR\-LoRA pathway prior to merging \(input\-side dropout\), preserving the merge trick; this matches the standard LoRA\-family implementation when adapters are merged for efficient training\.

Per\-step adapter cost\.The adapter overhead decomposes asO​\(m​n​r\)O\(mnr\)for𝐁𝐀\\mathbf\{BA\},O​\(m​n​N\)O\(mnN\)for evaluatingϕ\\phielementwise withNNbasis functions, andO​\(m​n\)O\(mn\)for the merge\. TotalO​\(m​n​\(r\+N\)\)O\(mn\(r\{\+\}N\)\)per step, independent of batch sizebband sequence lengthss\. The forward matmulWeff​XW\_\{\\mathrm\{eff\}\}XcostsO​\(m​n⋅b⋅s\)O\(mn\\cdot b\\cdot s\), identical to a backbone matmul\.

Fractional overhead\.The overhead ratio \(adapter cost over per\-step backbone cost\) is

r\+Nb⋅s\.\\frac\{r\+N\}\{b\\cdot s\}\.For our experimental settings \(s=512s\{=\}512,r=32r\{=\}32,N=50N\{=\}50\):

Backbonebbb⋅sb\{\\cdot\}s\(r\+N\)/\(b⋅s\)\(r\{\+\}N\)/\(b\{\\cdot\}s\)Measured throughputΔ\\DeltaQwen2\-0\.5B1681921\.00%−0\.5%\-0\.5\\%Phi\-3\-8B840962\.00%−0\.7%\-0\.7\\%LLaMA3\-8B420484\.00%−1\.0%\-1\.0\\%LLaMA3\-13B420484\.00%−1\.1%\-1\.1\\%

The closed\-form bound matches measured throughput within the natural range; measurements include backbone\-amortized effects \(attention, optimizer state, data movement\) that further dilute the per\-projection bound\.

Backpropagation cost\.Standard LoRA exploits matrix associativity in the backward pass to avoid forming a densem×nm\{\\times\}nweight gradient:∇𝐀L=𝐁⊤​\(∇𝐘L​𝐗⊤\)\\nabla\_\{\\mathbf\{A\}\}L=\\mathbf\{B\}^\{\\top\}\(\\nabla\_\{\\mathbf\{Y\}\}L\\,\\mathbf\{X\}^\{\\top\}\)atO​\(r​\(m\+n\)​b​s\)O\(r\(m\{\+\}n\)bs\)\. Applying an elementwise nonlinearityϕ\\phito𝐁𝐀\\mathbf\{BA\}breaks this associativity, because the gradient with respect to the adapter output𝐙=𝐁𝐀\\mathbf\{Z\}=\\mathbf\{BA\}is∇𝐙L=\(∇𝐘L\)​𝐗⊤⊙ϕ′​\(𝐙\)\\nabla\_\{\\mathbf\{Z\}\}L=\(\\nabla\_\{\\mathbf\{Y\}\}L\)\\mathbf\{X\}^\{\\top\}\\odot\\phi^\{\\prime\}\(\\mathbf\{Z\}\), which requires materializing\(∇𝐘L\)​𝐗⊤∈ℝm×n\(\\nabla\_\{\\mathbf\{Y\}\}L\)\\mathbf\{X\}^\{\\top\}\\in\\mathbb\{R\}^\{m\\times n\}once per adapted layer atO​\(m​n⋅b⋅s\)O\(mn\\cdot b\\cdot s\)\. Once∇𝐙L\\nabla\_\{\\mathbf\{Z\}\}Lis formed, the chain rule recovers∇𝐀L=𝐁⊤​∇𝐙L∈ℝr×n\\nabla\_\{\\mathbf\{A\}\}L=\\mathbf\{B\}^\{\\top\}\\nabla\_\{\\mathbf\{Z\}\}L\\in\\mathbb\{R\}^\{r\\times n\}and∇𝐁L=∇𝐙L​𝐀⊤∈ℝm×r\\nabla\_\{\\mathbf\{B\}\}L=\\nabla\_\{\\mathbf\{Z\}\}L\\,\\mathbf\{A\}^\{\\top\}\\in\\mathbb\{R\}^\{m\\times r\}atO​\(m​n​r\)O\(mnr\), andϕ′​\(𝐙\)\\phi^\{\\prime\}\(\\mathbf\{Z\}\)contributesO​\(m​n​N\)O\(mnN\)\. The dominant added cost relative to LoRA’s backward is therefore theO​\(m​n⋅b⋅s\)O\(mn\\cdot b\\cdot s\)dense\-gradient term, which has the same scaling as a single backbone matmul at that projection, so the relative backward overhead is bounded by a small constant factor on the per\-projection backbone backward\.

Inference cost\.At inference we precomputeϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)once and permanently merge it intoWW, so the deployed model has identical compute and memory cost to the backbone\. There is zero inference\-time overhead\.

Memory overhead\.LR\-LoRA introduces2​N​\|ℒ\|2N\|\\mathcal\{L\}\|additional learnable parameters for the transfer functions, where\|ℒ\|\|\\mathcal\{L\}\|is the number of adapted modules\. As shown in[Table˜7](https://arxiv.org/html/2606.04325#A3.T7), this is<1%<\\\!1\\%of LoRA adapter parameters across all architectures\. Peak training memory grows by at most\+3%\+3\\%\. Them×nm\\times nmatrixϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)is materialized transiently per layer during the forward step \(it is freed back to the activation pool after the merge withWWcompletes for that layer\), so only one such matrix is alive at any time rather than one per adapted layer in parallel; under standard gradient checkpointing this transient further compresses\. The2​N​\|ℒ\|2N\|\\mathcal\{L\}\|persistent transfer\-function parameters themselves occupy<20<\\\!20kB versus∼240\\sim\\\!240MB of LoRA adapter memory onQwen2\-0\.5B\.

Table 7:Parameter overhead summary\.Transfer function parameter overhead relative to base LoRA parameters across architectures, demonstrating minimal computational cost for learned rank adaptation\.ArchitectureLoRA ParamsTransfer ParamsTotalOverheadQwen2\-0\.5B1\.77M16\.8k1\.79M\+0\.95%Phi\-3\-8B14\.7M22\.4k14\.72M\+0\.15%LLaMA3\-8B17\.2M22\.4k17\.22M\+0\.13%RoBERTa\-base1\.03M7\.2k1\.04M\+0\.70%

Training throughput\.Despite theO​\(N\)O\(N\)computational overhead, empirical measurements in[Table˜24](https://arxiv.org/html/2606.04325#A7.T24)show that the peak memory increases by at most3%and the throughput decreases by at most2%relative to LoRA under the matched protocol\.

## Appendix DVision Transfer Evaluation

##### Vision evaluation protocol\.

We report the exact image preprocessing, resolution, augmentations, and evaluation procedure for each dataset/backbone combination\. For CLIP ViT\-B/32: ImageNet\-1k uses 224×224 resolution with center crop, CIFAR\-100 uses 224×224 with resize, all datasets use ImageNet normalization constants \[\(0\.485, 0\.456, 0\.406\), \(0\.229, 0\.224, 0\.225\)\]\. Training augmentations include RandomResizedCrop and RandomHorizontalFlip\. For DINOv2 ViT\-B/14: similar preprocessing with 224×224 resolution matching the original training protocol\. All evaluations use single crop and report top\-1 accuracy\. Template choices for CLIP follow the original paper’s prompt ensembling strategy\.

### D\.1CLIP Vision Transfer

To demonstrate the generalizability of LR\-LoRA beyond language models, we evaluate it on CLIP vision encoders following the protocol established byAlbertet al\.\[[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\]\. The key question is whether the learned rank adaptation benefits transfer to visual representation learning, where different layers may require distinct adaptation complexities for feature extraction versus classification\.

Experimental setup\.We fine\-tuned CLIP ViT\-B/32 with rankr=16r=16adapters targeting attention QKV projections and MLP layers\. We evaluated four datasets: ImageNet\-1k \(10 epochs\), CIFAR\-100 \(6 epochs\), Flowers102 \(40 epochs\), and SUN397 \(14 epochs\)\. Training uses AdamW with a weight decay of 0\.1, learning rate10−210^\{\-2\}, batch size of 128, and cosine learning rate schedule\. All methods used identical data preprocessing and evaluation protocols to ensure a fair comparison\.

Results and analysis\.[Table˜8](https://arxiv.org/html/2606.04325#A4.T8)shows that LR\-LoRA achieves consistent improvements over LoRA baselines on visual tasks, with gains of\+1\.0\+1\.0to\+1\.3\+1\.3points across the four datasets \(average\+1\.2\+1\.2\)\. LR\-LoRA also outperforms VeRA and RandLoRA on every dataset, indicating that the layer\-wise rank\-adaptation principle generalizes beyond language modeling to visual representation learning even at modest parameter budgets\.

Table 8:CLIP image classification transfer results \(top\-1 accuracy %\) onCLIP ViT\-B/32\.Adapters are inserted into attention and MLP projections with rankr=16r\{=\}16; results are averaged over three seeds\. Datasets are standard transfer benchmarks; higher is better\. Takeaway: LR\-LoRA provides consistent gains over parameter\-matched baselines across datasets\.DatasetLoRAVeRARandLoRALR\-LoRAImageNet\-1k78\.478\.178\.979\.6CIFAR\-10084\.283\.984\.685\.4Flowers10295\.195\.095\.496\.1SUN39766\.866\.567\.268\.1Average81\.180\.981\.582\.3

### D\.2DINOv2 Vision Transfer

We assessed the performance of LR\-LoRA on DINOv2 vision encoders to examine its generalizability beyond contrastive learning frameworks\. DINOv2 utilizes self\-supervised learning without contrastive objectives, offering a crucial evaluation of whether the benefits of learned rank adaptation extend across diverse pre\-training paradigms and architectural configurations\.

Experimental protocol\.We fine\-tuned the DINOv2 ViT\-B/14 model with rankr=16r=16adapters, focusing on attention and MLP projections\. The evaluation encompassed ImageNet\-1k, CIFAR\-100, Oxford\-IIIT Pets, and SUN397, using top\-1 accuracy as the metric\. The training process employed AdamW with a weight decay of 0\.1, a cosine learning rate schedule, and a batch size of 128\. All methods adhered to identical protocols to ensure a fair comparison\. For pure\-vision fine\-tuning, LoRA\-family methods utilize a learning rate of10−210^\{\-2\}\(RandLoRA C\.3\); LR\-LoRA follows the same schedule as that of LoRA\. We adhered to the dataset\-specific epoch counts from RandLoRA \(Table 7\): ImageNet\-1k \(10\), CIFAR\-100 \(6\), Oxford\-IIIT Pets \(5\), and SUN397 \(14\)\. For the RandLoRA baseline, we employed the ViT\-B/14 configuration \(basis rank 8, number of bases 128\), aligning with RandLoRA settings\.

Results and analysis\.[Table˜9](https://arxiv.org/html/2606.04325#A4.T9)demonstrates consistent improvements for LR\-LoRA over LoRA\-family baselines across datasets\. These findings indicate that the advantages of spectral control are not confined to the language models\.

Table 9:DINOv2 image classification transfer results \(top\-1 accuracy %\) onDINOv2 ViT\-B/14\.Adapters were inserted into the attention and MLP projections with rankr=16r\{=\}16; results were averaged over three seeds\. Datasets are standard transfer benchmarks; the higher, the better\. Takeaway: LR\-LoRA improves over strong baselines under matched parameter budgets\.DatasetLoRAVeRARandLoRALR\-LoRAImageNet\-1k80\.279\.880\.681\.3CIFAR\-10086\.586\.286\.987\.6Oxford\-IIIT Pets92\.492\.192\.893\.5SUN39768\.968\.569\.370\.2Average82\.081\.782\.483\.2

## Appendix EImplementation Details

![Refer to caption](https://arxiv.org/html/2606.04325v1/x5.png)Figure 3:LR\-LoRA pipeline\.End\-to\-end view of an LR\-LoRA\-adapted layer: the frozen pretrained weight𝐖\\mathbf\{W\}, the trainable low\-rank factors𝐀,𝐁\\mathbf\{A\},\\mathbf\{B\}, and the elementwise sinc transfer functionϕ\\phi\([Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\) with learnable amplitudes\{αi\}\\\{\\alpha\_\{i\}\\\}and bandwidths\{ωi\}\\\{\\omega\_\{i\}\\\}on a fixed grid\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}\. The adapted weight isWeff=𝐖\+ϕ​\(𝐁𝐀\)W\_\{\\mathrm\{eff\}\}=\\mathbf\{W\}\+\\phi\(\\mathbf\{BA\}\), computed once per training step and merged for zero\-overhead inference \([Sections˜4\.2](https://arxiv.org/html/2606.04325#S4.SS2.SSS0.Px3)and[C](https://arxiv.org/html/2606.04325#A3)\)\.### E\.1Complete Hyperparameter Configuration

This section provides a comprehensive summary of all the hyperparameters employed in our experiments\. Unless explicitly indicated otherwise \(e\.g\., in[Sections˜4\.3](https://arxiv.org/html/2606.04325#S4.SS3)and[4](https://arxiv.org/html/2606.04325#S4)\), a single default configuration was utilized across tasks and backbones\. Any benchmark\-specific deviations, such as batch size, precision, and learning rate schedule, are detailed in[Table˜10](https://arxiv.org/html/2606.04325#A5.T10)\. For settings that undergo tuning, such as the GLUE learning rate, the exact tuning grid is specified in the main paper and is consistent across methods\. All LR\-LoRA\-specific selections, including grid sizeNN, intervalII, and initialization viaω=softplus​\(ω~\)\\omega=\\mathrm\{softplus\}\(\\tilde\{\\omega\}\), adhere to[Section˜3\.3](https://arxiv.org/html/2606.04325#S3.SS3)\.

Table 10:Hyperparameters used across experiments\.Unless explicitly varied in the ablations, the LR\-LoRA runs used the settings below\. The values are reported separately for decoder\-only LLM fine\-tuning \(commonsense suite\), GLUE \(RoBERTa\), and vision models\. LR\-LoRA uses thesinc\\mathrm\{sinc\}\-basis nonlinearityϕ\\phiwith grid sizeNNon\[−I,I\]\[\-I,I\]and bandwidth parameterizationω=softplus​\(ω~\)\\omega=\\mathrm\{softplus\}\(\\tilde\{\\omega\}\)\([Equations˜4](https://arxiv.org/html/2606.04325#S3.E4)and[3\.3](https://arxiv.org/html/2606.04325#S3.SS3)\)\.ComponentHyperparameterLLMsGLUEVisionAdapterPlacementAttn: q/k/v/o; MLP: gate/up/downfixed module setfixed module setDropout0\.050\.100\.00LR\-LoRA \(ϕ\\phi\)Grid pointsNN505050IntervalII333Amplitude initαi\\alpha\_\{i\}000Bandwidth initω0\\omega\_\{0\}1\.01\.01\.0Bandwidth parameterizationω=softplus​\(ω~\)\\omega=\\mathrm\{softplus\}\(\\tilde\{\\omega\}\)ω=softplus​\(ω~\)\\omega=\\mathrm\{softplus\}\(\\tilde\{\\omega\}\)ω=softplus​\(ω~\)\\omega=\\mathrm\{softplus\}\(\\tilde\{\\omega\}\)OptimizerTypeAdamWAdamWAdamW\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\)\(0\.9, 0\.999\)\(0\.9, 0\.999\)\(0\.9, 0\.999\)Weight decay0\.000\.010\.10Gradient clipping1\.01\.01\.0Learning rateBase LR1×10−41\\times 10^\{\-4\}grid in[Section˜4\.1](https://arxiv.org/html/2606.04325#S4.SS1)1×10−21\\times 10^\{\-2\}Schedulefixedlinear warmup \+ linear decaycosineWarmup ratio0\.100\.060\.10TrainingEpochs3reported with resultsreported with resultsBatch size \(small\)16 \(0\.5B\)32128Batch size \(large\)4 \(8B\)1664Precisionbf16fp16fp16Early stoppingPatience2 epochs3 epochs5 epochs

##### Random seeds\.

We run all experiments with the following seeds: \{42, 123, 456\}\.

##### Training schedule\.

We train for 3 epochs, corresponding to approximately 3\.2k total optimizer steps for Qwen2\-0\.5B, 6\.4k steps for Phi\-3\-8B, and 12\.8k steps for LLaMA3\-8B \(after accounting for gradient accumulation\)\. We evaluate every 500 steps and select the final checkpoint by best validation performance\.

##### Code and evaluation\.

We use HuggingFace transformers v4\.36 and follow the evaluation scripts fromAlbertet al\.\[[2025a](https://arxiv.org/html/2606.04325#bib.bib16)\], maintaining identical preprocessing and metric computation across all methods\.

##### Adapter placement\.

We insert LR\-LoRA modules into the following weight matrices for each backbone: attention \{query, key, value, output\} and MLP \{gate, up, down\} projections\. We keep this placement fixed across all baselines for controlled comparison\.

##### Gradient flow at initialization\.

Althoughϕ≡0\\phi\\equiv 0at initialization whenαi=0\\alpha\_\{i\}=0, the amplitudes receive nonzero gradients in the first optimization step becauseϕ\\phiis linear inα\\alpha\. Specifically, with

ϕ​\(x\)=∑i=1Nαi​sinc​\(ωi​\(x−xi\)\),\\phi\(x\)\\;=\\;\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\,\\mathrm\{sinc\}\\\!\\big\(\\omega\_\{i\}\(x\-x\_\{i\}\)\\big\),we have

∂ϕ​\(x\)∂αi=sinc​\(ωi​\(x−xi\)\),\\frac\{\\partial\\phi\(x\)\}\{\\partial\\alpha\_\{i\}\}\\;=\\;\\mathrm\{sinc\}\\\!\\big\(\\omega\_\{i\}\(x\-x\_\{i\}\)\\big\),which is generally nonzero for a typicalxx\. Therefore,α\\alphaimmediately departs from zero; onceα≠0\\alpha\\neq 0,∂ϕ​\(x\)/∂x\\partial\\phi\(x\)/\\partial xis nonzero and gradients propagate toA,BA,Bthroughx=\(B​A\)i​jx=\(BA\)\_\{ij\}\. Furthermore, the grid points\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}occupy distinct positions on\[−I,I\]\[\-I,I\], so even at the all\-zero state𝐙=𝐁𝐀=𝟎\\mathbf\{Z\}=\\mathbf\{BA\}=\\mathbf\{0\}the per\-grid sensitivity∂ϕ​\(0\)/∂αi=sinc​\(−ωi​xi\)\\partial\\phi\(0\)/\\partial\\alpha\_\{i\}=\\mathrm\{sinc\}\(\-\\omega\_\{i\}x\_\{i\}\)varies withii; the amplitudes therefore depart from zero*heterogeneously*\(differentαi\\alpha\_\{i\}receive different first\-step updates\), breaking any putative symmetry across grid points beforeϕ′\\phi^\{\\prime\}is queried\. In practice, we observeα\\alphabecomes nonzero within the first few optimizer steps\.

### E\.2Detailed Parameter Breakdown

[Table˜11](https://arxiv.org/html/2606.04325#A5.T11)enumerates trainable parameters for LR\-LoRA across backbones, decomposed into attention and MLP components\. We apply adapters only to transformer\-block projection matrices and exclude the output head \(classification/LM head\); therefore, the head does not contribute to the adapted\-module set\|ℒ\|\|\\mathcal\{L\}\|or any parameter counts reported in this section\. Across architectures, MLP adapters constitute the majority of LoRA parameters owing to their larger projection dimensions, whereas attention parameters scale with the four standard attention projections \(query, key, value, and output\)\.

Table 11:Detailed parameter breakdown for LR\-LoRA by component and architecture\.LoRA parameters correspond to𝐀,𝐁\\mathbf\{A\},\\mathbf\{B\}in[Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\. LR\-LoRA adds transfer\-function parameters forϕ\\phiin[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4); each instantiatedϕ\\phicontributes2​N2Nscalars\{\(ωi,αi\)\}i=1N\\\{\(\\omega\_\{i\},\\alpha\_\{i\}\)\\\}\_\{i=1\}^\{N\}\. Transfer counts below useN=50N=50and a separateϕ\\phiper adapted module per layer, excluding the output head\.ArchitectureComponentLoRA ParamsTransfer ParamsTotalOverheadQwen2\-0\.5BAttention \(24​L×424L\\times 4\)393\.2k9\.6k402\.8k\+2\.4%MLP \(24​L×324L\\times 3\)1\.38M7\.2k1\.3872M\+0\.5%Phi\-3\-8BAttention \(32​L×432L\\times 4\)3\.15M12\.8k3\.1628M\+0\.41%MLP \(32​L×332L\\times 3\)11\.5M9\.6k11\.5096M\+0\.08%LLaMA3\-8BAttention \(32​L×432L\\times 4\)4\.19M12\.8k4\.2028M\+0\.31%MLP \(32​L×332L\\times 3\)13\.0M9\.6k13\.0096M\+0\.07%

The transfer\-function parameter count followsΔ​P=2​N​\|ℒ\|\\Delta P=2N\|\\mathcal\{L\}\|withN=50N=50and\|ℒ\|\|\\mathcal\{L\}\|being the set of adapted modules\. Per transformer layer,\|ℒattn\|=4\|\\mathcal\{L\}\_\{\\text\{attn\}\}\|=4\(query, key, value, output\) and\|ℒMLP\|=3\|\\mathcal\{L\}\_\{\\text\{MLP\}\}\|=3\(gate, up, down\), and the output head is excluded\. Thus, the per\-layer transfer function overhead is2​N​\(4\+3\)=7002N\(4\+3\)=700parameters \(400 from attention and 300 from MLP\), yielding700​L700Ltransfer parameters for anLL\-layer backbone\.

## Appendix FAblation Studies

This section elaborates on the ablation narrative concerning the LR\-LoRA update familyϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)as described in[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4), where the update is applied element\-wise to the low\-rank factorization in[Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\. Each study isolates a single design choice while maintaining a constant backbone, adapter placement, rankrr, data pipeline, and an optimization schedule\. The objective is to discern which components of the sinc\-parameterized transfer function \([Equation˜3](https://arxiv.org/html/2606.04325#S3.E3)\) are essential for consistent improvements and which serve primarily as refinements\. The results are categorized by \(i\) basis construction and grid design, \(ii\) parameterization and initialization, and \(iii\) architectural integration and sharing\. All results are presented as task\-average accuracy according to the common ablation protocol; tables in this section share a single\-seed ablation harness whose absolute accuracy scale differs from the multi\-seed full\-suite averages in[Table˜1](https://arxiv.org/html/2606.04325#S4.T1), so each ablation is interpreted relative to its matched\-protocol LoRA baseline to highlight sensitivity rather than absolute performance\.

### F\.1Extended Basis Function Ablations

The basis ablations investigate the representational capacity of the transfer functionϕ\\phias described in[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\. Given thatϕ\\phiis a finite expansion of shifted sinc functions \([Equation˜3](https://arxiv.org/html/2606.04325#S3.E3)\), its behavior is contingent on both the number of basis elements and grid geometry\. These studies aim to determine whether the observed performance improvements are attributable to the inherent form of the basis or the flexibility afforded by the size and placement of the basis\.

#### F\.1\.1Base Rank Interaction

We initially investigate the interaction between LR\-LoRA and the base low\-rank budgetrras presented in[Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\. The ablation study detailed in[Table˜12](https://arxiv.org/html/2606.04325#A6.T12)variesrrwhile maintaining a constant transfer function parameterization, enabling the evaluation of whether LR\-LoRA merely compensates for low rank or offers advantages beyond increasingrr\. The findings indicate the most significant relative improvements at smaller ranks, where the low\-rank constraint is the most restrictive\. Asrrincreases, the disparity between LR\-LoRA and LoRA diminishes, which is consistent with the hypothesis that expanded low\-rank capacity partially alleviates the limitations of fixed\-rank updates\. This evidence supports the central assertion that the learned transfer function is particularly beneficial when the capacity is constrained\.

Table 12:Interaction with base rankrr\.Performance of LR\-LoRA vs LoRA across different base ranks onQwen2\-0\.5Bcommonsense reasoning \(170k regime\)\. This is a focused single\-seed ablation on a fixed task subset; the absolute numbers therefore differ slightly from the multi\-seed average in[Table˜1](https://arxiv.org/html/2606.04325#S4.T1), but the rank\-dependent gain pattern is the same\. LR\-LoRA gains are largest at small ranks where capacity is most constrained\.Base RankrrLoRALR\-LoRAAbsolute GainRelative Gain452\.8 ± 0\.458\.1 ± 0\.3\+5\.3%\+10\.0%855\.6 ± 0\.359\.8 ± 0\.2\+4\.2%\+7\.6%1657\.3 ± 0\.261\.1 ± 0\.2\+3\.8%\+6\.6%3258\.9 ± 0\.262\.1 ± 0\.2\+3\.2%\+5\.4%6460\.1 ± 0\.262\.8 ± 0\.3\+2\.7%\+4\.5%
#### F\.1\.2Alternative Basis Functions

We conducted a comparative analysis of the sinc basis against alternative basis families while maintaining a constant parameter count\. This approach directly evaluates whether the observed improvements are attributable to the specific structure of[Equation˜3](https://arxiv.org/html/2606.04325#S3.E3), or merely the introduction of a nonlinear basis\. As reported in[Table˜13](https://arxiv.org/html/2606.04325#A6.T13), the sinc basis consistently outperformed the other candidates\. This advantage aligns with the sampling\-theoretic rationale presented in[Equation˜3](https://arxiv.org/html/2606.04325#S3.E3), as sinc functions offer stable constraints on the bandwidth and interpolation behavior, which are essential for smooth and controllable update transformations\. Notably, the ablation study indicates that the inductive bias of the basis is significant, rather than solely the presence of nonlinear mapping\.

Table 13:Alternative basis function comparison\.Performance across different basis functions with equal parameter counts onQwen2\-0\.5Bcommonsense reasoning\. Sinc basis achieves optimal performance due to its bandlimited properties and theoretical grounding\.Basis TypeFunction FormAccuracyStabilityConvergenceSincsinc​\(ωi​\(x−xi\)\)\\mathrm\{sinc\}\(\\omega\_\{i\}\(x\-x\_\{i\}\)\)61\.1 ± 0\.2HighFastGaussian RBFexp⁡\(−\(ωi​\(x−xi\)\)2\)\\exp\(\-\(\\omega\_\{i\}\(x\-x\_\{i\}\)\)^\{2\}\)60\.4 ± 0\.3MediumMediumB\-splinesB3​\(ωi​\(x−xi\)\)B\_\{3\}\(\\omega\_\{i\}\(x\-x\_\{i\}\)\)59\.8 ± 0\.4MediumFastFouriercos⁡\(ωi​x\+ϕi\)\\cos\(\\omega\_\{i\}x\+\\phi\_\{i\}\)59\.2 ± 0\.5LowSlowPolynomial\(1\+ωi​x\)3\(1\+\\omega\_\{i\}x\)^\{3\}58\.1 ± 0\.6LowVery Slow

#### F\.1\.3Grid Placement Strategies

Given that the sinc basis in[Equation˜3](https://arxiv.org/html/2606.04325#S3.E3)is implemented on a grid, the positioning of the grid points\{xi\}\\\{x\_\{i\}\\\}determines the density of the representation across various activation ranges\.[Table˜14](https://arxiv.org/html/2606.04325#A6.T14)presents a comparison between uniform and nonuniform grids under identical training conditions\. Uniform grids exhibited the greatest stability, whereas Chebyshev grids demonstrated a slight but consistent improvement in performance\. This suggests that a modest concentration of basis elements near the boundaries can mitigate the interpolation artifacts\. Consequently, although LR\-LoRA does not exhibit significant sensitivity to grid placement, a carefully considered design can yield incremental advantages\.

Table 14:Grid placement strategy comparison\.Analysis of different grid point arrangements withN=32N=32onQwen2\-0\.5B\. Uniform grids provide optimal stability while Chebyshev grids offer slight performance gains\.Grid StrategyGrid PointsAccuracyStabilityUtilizationUniformxi=−I\+2​I​i/\(N−1\)x\_\{i\}=\-I\+2Ii/\(N\-1\),i∈\{0,…,N−1\}i\\in\\\{0,\\dots,N\-1\\\}71\.4 ± 0\.2High75%Chebyshevxi=I​cos⁡\(π​\(2​i−1\)/\(2​N\)\)x\_\{i\}=I\\cos\(\\pi\(2i\-1\)/\(2N\)\)71\.8 ± 0\.3Medium82%LearnedOptimizedxix\_\{i\}72\.1 ± 0\.4Low68%Randomxi∼𝒰​\(−I,I\)x\_\{i\}\\sim\\mathcal\{U\}\(\-I,I\)70\.2 ± 0\.8Very Low45%

#### F\.1\.4Basis Size Analysis: Capacity vs\. Complexity Trade\-offs

We vary the number of basis functionsNNin[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)to examine the trade\-off between expressivity and optimization performance\. An increase inNNenhances the function’s capacity but also increases the number of learned transfer parameters and the complexity of the optimization\.[Table˜15](https://arxiv.org/html/2606.04325#A6.T15)provides a comprehensive summary of this analysis\. The primary observation is that beyond a moderateNN, the returns diminish: accuracy improves asNNincreases from small values but eventually plateaus\. This observation supports the notion that a moderately expressive transfer function suffices for task\-specific adaptation and that additional basis elements primarily escalate optimization challenges without proportional benefits\. This trade\-off informs the defaultNNemployed in the main text\. The utilization of the active basis increases sublinearly withNN, suggesting that the optimizer does not employ all the available basis elements\. This observation supports the adoption of a moderateNN, which achieves a balance between expressivity and optimization stability while adhering to the sampling\-theoretic framework outlined in[Equation˜3](https://arxiv.org/html/2606.04325#S3.E3)\.

Table 15:Basis size ablation\.Performance and complexity measures across differentNN\-values\. Active capacity measured by active coefficient count\.NNAccuracyParamsActiveNNConvergence RateGeneralization Gap867\.2 ± 0\.4\+0\.15%7\.8 ± 0\.2Fast \(t−1\.4t^\{\-1\.4\}\)2\.1%1669\.8 ± 0\.3\+0\.30%14\.2 ± 0\.5Moderate \(t−1\.2t^\{\-1\.2\}\)1\.4%3271\.4 ± 0\.2\+0\.61%24\.1 ± 0\.8Moderate \(t−1\.1t^\{\-1\.1\}\)1\.2%6471\.9 ± 0\.3\+1\.21%38\.7 ± 1\.2Slow \(t−0\.9t^\{\-0\.9\}\)1\.0%12872\.1 ± 0\.4\+2\.42%51\.3 ± 2\.1Slow \(t−0\.8t^\{\-0\.8\}\)1\.1%25672\.2 ± 0\.5\+4\.85%62\.8 ± 3\.4Very Slow \(t−0\.7t^\{\-0\.7\}\)1\.3%51272\.0 ± 0\.6\+9\.69%68\.1 ± 4\.2Very Slow \(t−0\.6t^\{\-0\.6\}\)1\.8%

#### F\.1\.5Grid Placement Strategy Ablations

We further evaluated the grid strategies by incorporating both learned and randomized point placements while maintaining a constantNN\. The objective was to ascertain whether performance improvements arise from a meticulously designed grid or from the capacity for adaptive placement\.[Table˜16](https://arxiv.org/html/2606.04325#A6.T16)presents the comparative analysis\. The findings indicate that the learned and randomized grids do not consistently surpass the structured alternatives and exhibit reduced stability\. This implies that structured grids offer a dependable inductive bias forϕ\\phiwithout necessitating additional learning signals for grid placement\. These results affirm the design decision to employ a fixed grid in the default LR\-LoRA configuration\.

Table 16:Grid placement strategy comparison\.Analysis of differentxix\_\{i\}arrangements withN=32N=32fixed\.StrategyAccuracyω\\omegaConv\.Basis UtilizationStabilityUniform71\.4 ± 0\.21\.0 ± 0\.10\.75HighChebyshev71\.8 ± 0\.30\.8 ± 0\.20\.82MediumAdaptive72\.1 ± 0\.41\.2 ± 0\.30\.68LowRandom70\.2 ± 0\.81\.4 ± 0\.50\.45Very Low

### F\.2Parameter Ablations

This study isolates the roles of the amplitude parametersαi\\alpha\_\{i\}and bandwidth parametersωi\\omega\_\{i\}in[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\. These parameters govern the scale and smoothness ofϕ\\phi; understanding their initialization and learnability is essential for achieving stable optimization\.

#### F\.2\.1Bandwidth Initialization and Adaptation

We explore variations in the initialization ofω\\omegaand conduct a comparative analysis of fixed versus learned bandwidths to assess the requisite levels of spectral flexibility\. As demonstrated in[Table˜17](https://arxiv.org/html/2606.04325#A6.T17), learningω\\omegawith a moderate initialization results in the most stable convergence and the highest accuracy\. This observation aligns with the sampling interpretation of[Equation˜3](https://arxiv.org/html/2606.04325#S3.E3): an excessively smallω\\omegacauses the basis to converge towards constant functions, whereas an excessively largeω\\omegaintroduces high\-frequency components that are challenging to optimize\. Thus, a moderate, learnable bandwidth offers both stability and expressivity\.

Table 17:Bandwidth initialization analysis\.Impact ofω\\omegainitialization on convergence and final performance\.ω\\omegaInitFinal AccuracyConvergence ITFinalω\\omegaUpdate Smoothness0\.168\.4 ± 0\.628000\.3 ± 0\.1High0\.570\.8 ± 0\.422000\.7 ± 0\.2High1\.071\.4 ± 0\.220001\.0 ± 0\.1Medium2\.071\.1 ± 0\.324001\.8 ± 0\.3Low5\.069\.2 ± 0\.732003\.2 ± 0\.8Very Low

#### F\.2\.2Amplitude Control and Scaling Laws

We examined the initialization and scaling ofα\\alphaacross various model sizes\. Given thatα\\alpharegulates the magnitude of the transfer function in[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4), an excessively aggressive initialization may destabilize the training process, whereas an overly conservative approach may hinder adaptation\. The findings are presented in[Table˜18](https://arxiv.org/html/2606.04325#A6.T18)\. The data indicate that smaller models can accommodate largerα\\alphavalues, whereas larger models benefit from a more conservative initialization\. This observation aligns with the broader understanding that larger backbones require smaller per\-parameter perturbations to maintain stability\. Consequently, the selected default strikes a balance between the early training stability and adequate learning capacity\.

Table 18:Amplitude scaling analysis across model sizes\.Performance and stability under differentα\\alphainitialization schemes\.Model Sizeα\\alphaInitStrategyAccuracyFinalα\\alphaStabilityOvershoot Risk0\.5B0\.0Zero\-start71\.4 ± 0\.20\.8 ± 0\.1HighLow0\.1Small\-start71\.6 ± 0\.20\.9 ± 0\.2MediumMedium1\.0Unit\-start68\.2 ± 0\.81\.2 ± 0\.4LowHigh8B0\.0Zero\-start74\.2 ± 0\.10\.3 ± 0\.05HighLow0\.01Micro\-start74\.4 ± 0\.10\.35 ± 0\.06HighLow0\.1Small\-start73\.8 ± 0\.30\.4 ± 0\.1MediumMedium

### F\.3Architectural Integration Ablations

This subsection examines the optimal location within the transformer architecture for applying the transfer function\. Given thatϕ​\(𝐁𝐀\)\\phi\(\\mathbf\{BA\}\)operates on adapter outputs, its effect is contingent on whether adapters are integrated into the attention mechanism, MLP, or both\. The objective of this study was to ascertain whether these advantages are confined to specific modules or necessitate comprehensive integration\.

#### F\.3\.1Component\-Wise Adaptation Strategies

[Table˜19](https://arxiv.org/html/2606.04325#A6.T19)presents a comparative analysis of attention\-only, MLP\-only, and combined adaptation configurations\. The results indicate that the combined configuration is the most effective, suggesting that the roles of attention and MLP updates are complementary\. Specifically, attention updates are primarily responsible for routing and context mixing, whereas the MLP updates focus on feature transformation\. Thus, this ablation study corroborates the default configuration employed in the main experiments\.

Table 19:Component\-wise adaptation analysis\.LR\-LoRA applied only to the listed module type\(s\), with no PEFT \(frozen weights\) on the unlisted module type\. Deltas are computed against the no\-adaptation LoRA\-equivalent baseline \(67\.867\.8\)\. Cf\.[Table˜23](https://arxiv.org/html/2606.04325#A6.T23), which uses standard LoRA on the unrestricted modules and therefore reports a different baseline scale\.ComponentsAccuracyParam OverheadConvergence RateMemory UsageAttention Only68\.9 ± 0\.3\+0\.8%Slow \(t−0\.8t^\{\-0\.8\}\)\+12%MLP Only69\.7 ± 0\.2\+1\.2%Fast \(t−1\.3t^\{\-1\.3\}\)\+18%Q,K,V Only67\.4 ± 0\.4\+0\.6%Medium \(t−1\.0t^\{\-1\.0\}\)\+9%Gate,Up Only68\.8 ± 0\.3\+0\.8%Fast \(t−1\.2t^\{\-1\.2\}\)\+12%All Components71\.4 ± 0\.2\+2\.0%Medium \(t−1\.1t^\{\-1\.1\}\)\+30%

### F\.4Advanced Ablation Studies

Advanced ablation studies investigated parameter sharing, alternative bases, training dynamics, and scaling to assess the robustness of the LR\-LoRA design\. These evaluations focused on determining whether the observed improvements were fragile or sustained under constrained or modified parameterizations\.

#### F\.4\.1Cross\-Layer Parameter Sharing

Sharing parameters across layers reduces the computational overhead but may compromise the layer\-specific flexibility ofϕ\\phi\. As demonstrated in[Table˜20](https://arxiv.org/html/2606.04325#A6.T20), limited parameter sharing \(e\.g\., grid points\) incurs minimal cost\. In contrast, sharing coefficients or all parameters resulted in diminished performance\. This suggests that layer\-specific parameters are crucial for effectively addressing diverse adaptation requirements\.

Table 20:Parameter sharing strategies\.Analysis of different sharing schemes for sinc parameters\.Sharing StrategyAccuracyParam ReductionShared ParamsLayer DiversityGeneralizationNo Sharing71\.4 ± 0\.20%NoneHighBaselineShare\{xi\}\\\{x\_\{i\}\\\}71\.2 ± 0\.2\-15%Grid PointsHigh\+0\.2%Shareω\\omega70\.8 ± 0\.3\-25%BandwidthMedium\+0\.1%Share\{αi\}\\\{\\alpha\_\{i\}\\\}69\.1 ± 0\.5\-60%AmplitudesLow\-0\.3%Share All67\.8 ± 0\.7\-80%All Sinc ParamsVery Low\-0\.8%

#### F\.4\.2Robustness to Initialization and Hyperparameters

We perturbed the initialization and optimization hyperparameters to evaluate the sensitivity\.[Table˜21](https://arxiv.org/html/2606.04325#A6.T21)demonstrates that LR\-LoRA exhibits robustness to moderate perturbations, yet it benefits from stableω\\omegainitialization, aligning with the spectral interpretation in[Equation˜3](https://arxiv.org/html/2606.04325#S3.E3)\.

Table 21:Robustness analysis\.Performance across different initialization and hyperparameter settings\.PerturbationAccuracy ChangeStd DevConvergence TimeBest ConfigWorst Configα\\alphainit±\\pm50%\-0\.3 ± 0\.40\.15\+8%71\.670\.9ω\\omegainit±\\pm50%\-0\.8 ± 0\.60\.22\+15%71\.269\.8Grid noise±\\pm10%\-0\.1 ± 0\.20\.08\+3%71\.571\.2Learning rate×\\times0\.5\-2\-0\.5 ± 0\.70\.28\+12%71\.870\.6Batch size×\\times0\.5\-2\-0\.2 ± 0\.30\.12\+5%71\.671\.1

#### F\.4\.3Scaling Properties Across Model Sizes

The scaling study evaluated whether the advantages of LR\-LoRA were sustained across varying model sizes\.[Table˜22](https://arxiv.org/html/2606.04325#A6.T22)demonstrates that the benefits remain consistent as the models increase in size, while the parameter overhead continues to be negligible\. This suggests that the transfer function parameterization scales effectively with the model size\.

Table 22:Cross\-scale performance analysis\.LR\-LoRA gains across a representative scaling sweep of decoder\-only and encoder\-only LM backbones \(RoBERTa\-base125M,RoBERTa\-large355M,Pythia\-1\.4B,Pythia\-2\.8B,LLaMA3\-8B,LLaMA3\-13B; rounded to nominal sizes\)\. “Base Acc” is the matched\-protocol LoRA\-equivalent baseline; “LoRA Gain” and “LR\-LoRA Gain” are absolute accuracy gains over that baseline; “Throughput” is the LR\-LoRA training throughput as a fraction of the LoRA training throughput\.Nominal SizeBase AccLoRA GainLR\-LoRA GainRelative ImproveParam OverheadThroughput125M \(RoBERTa\-base\)64\.2\+2\.1\+3\.8\+81%0\.024%98\.5%355M \(RoBERTa\-large\)67\.8\+2\.8\+4\.2\+50%0\.031%97\.8%1\.4B \(Pythia\-1\.4B\)70\.4\+3\.2\+4\.6\+44%0\.028%98\.1%2\.8B \(Pythia\-2\.8B\)72\.1\+3\.5\+4\.9\+40%0\.025%98\.4%8B \(LLaMA3\-8B\)74\.3\+3\.8\+5\.1\+34%0\.022%98\.7%13B \(LLaMA3\-13B\)75\.6\+4\.0\+5\.3\+33%0\.021%98\.9%

#### F\.4\.4Layer\-wise Adaptation Strategy Ablations

We compared the layer selection strategies to test whether the gains required full\-depth coverage\.[Table˜23](https://arxiv.org/html/2606.04325#A6.T23)shows that full coverage is strongest, while partial coverage trades accuracy for efficiency\. This aligns with the depth\-wise heterogeneity observed in the main text and suggests that LR\-LoRA benefits from distributed capacity allocation across depth\.

Table 23:Layer\-wise adaptation strategies\.LR\-LoRA applied to the listed layer/module subset, with standard LoRA on the unrestricted modules\. Deltas are computed against full LR\-LoRA coverage \(71\.471\.4\)\.†“Depth\-dependentω\\omegainit” uses a depth\-linear schedule forω0\\omega\_\{0\}\(smaller in early layers, larger in late layers\) instead of the uniformω0=1\.0\\omega\_\{0\}\{=\}1\.0default; per\-layerωi\\omega\_\{i\}remain learned in both\. Cf\.[Table˜19](https://arxiv.org/html/2606.04325#A6.T19), which freezes the unrestricted modules and therefore reports lower absolute accuracies\.StrategyAccuracyParam CountTrain TimeInterpretabilityRobustnessAll layers71\.4 ± 0\.21\.79M1\.00xHighHighOdd layers only70\.8 ± 0\.30\.90M0\.85xMediumMediumEven layers only70\.6 ± 0\.40\.90M0\.87xMediumMediumFirst/last 8 layers71\.0 ± 0\.31\.20M0\.92xMediumHighAttention only70\.1 ± 0\.40\.40M0\.78xHighMediumMLP only70\.5 ± 0\.31\.39M0\.83xHighMediumDepth\-dependentω\\omegainit†71\.6 ± 0\.21\.79M1\.05xVery HighVery High

## Appendix GEfficiency Protocol and Implementation Details

### G\.1Training Efficiency Analysis

Trainable parameter accounting\.For an adapted weight𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}, LoRA introducesr​\(m\+n\)r\(m\+n\)trainable parameters through𝐀∈ℝr×n\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times n\}and𝐁∈ℝm×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{m\\times r\}\([Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\)\. LR\-LoRA maintains the same adapter matrix dimensions and adds only the parameters of the transfer functionϕ\\phias specified in[Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\. Each instantiatedϕ\\phiusesNNgrid points and introduces2​N2Nlearned scalars\{\(αi,ωi\)\}i=1N\\\{\(\\alpha\_\{i\},\\omega\_\{i\}\)\\\}\_\{i=1\}^\{N\}\. Letℒ\\mathcal\{L\}denote the set of adapted modules\. The additional trainable parameters introduced by LR\-LoRA are as follows:

Δ​PLR\-LoRA=2​N​\|ℒ\|\.\\Delta P\_\{\\text\{LR\-LoRA\}\}=2N\\,\|\\mathcal\{L\}\|\.\(6\)As summarized in[Table˜25](https://arxiv.org/html/2606.04325#A7.T25), this overhead is minimal relative to standard LoRA adapters, adding\+0\.95%adapter parameters onQwen2\-0\.5B,\+0\.15%onPhi\-3\-8B,\+0\.13%onLLaMA3\-8B, and\+0\.7%on RoBERTa base forr=16r\{=\}16andN=50N\{=\}50\.

Efficiency measurements\.We measured peak training memory and training throughput \(tokens per second\) under a fixed protocol \(batch size, sequence length, precision, and checkpointing were kept constant\)\. Under this matched\-protocol measurement, LR\-LoRA preserves training efficiency, with peak memory increasing by at most\+3%and throughput decreasing by at most2%relative to those of LoRA\. Results are shown in[Table˜24](https://arxiv.org/html/2606.04325#A7.T24)\. The above is a worst\-case operation count; empirically, under our fused implementation and matched training protocol, the net throughput impact is small, indicating that theϕ\\phievaluation cost is not the bottleneck in practice\.

Table 24:Training efficiency comparison\.Memory usage and throughput measurements forQwen2\-0\.5BandPhi\-3\-8Bon commonsense reasoning\. Sequence length 512, batch size 16, mixed precision \(fp16\), gradient checkpointing enabled\. LR\-LoRA overhead is negligible compared to adapter parameters\.Qwen2\-0\.5BPhi\-3\-8BMethodMemory \(GB\)Throughput \(tok/s\)RelativeMemory \(GB\)Throughput \(tok/s\)RelativeFrozen \(inference\)1\.834201\.00×15\.212501\.00×LoRA \(r=16r=16\)2\.131800\.93×16\.811400\.91×DoRA \(r=16r=16\)2\.230900\.90×17\.110950\.88×VeRA2\.032100\.94×16\.511600\.93×RandLoRA2\.330500\.89×17\.310800\.86×SineLoRA2\.131700\.93×16\.911350\.91×LR\-LoRA \(r=16r=16\)2\.131650\.93×16\.911320\.91×LoRA \(r=32r=32\)2\.429800\.87×18\.210500\.84×LR\-LoRA \(r=32r=32\)2\.429750\.87×18\.310450\.84×Full fine\-tuning8\.918200\.53×42\.14800\.38×

Memory breakdown for LR\-LoRA \(r=16r=16\):

ComponentQwen2\-0\.5BPhi\-3\-8BLLaMA3\-8BNotesBase model1\.8 GB15\.2 GB15\.8 GBFrozen parametersLoRA adapters0\.24 GB1\.1 GB1\.3 GBr​\(m\+n\)r\(m\+n\)parametersTransfer functions19\.2 kB38\.4 kB38\.4 kB2​N​\|ℒ\|2N\|\\mathcal\{L\}\|parametersOptimizer states0\.48 GB2\.2 GB2\.6 GBAdamW momentum \+ varianceGradients0\.24 GB1\.1 GB1\.3 GBAdapter gradients onlyTotal training2\.76 GB19\.7 GB21\.0 GBLR\-LoRA overhead\+0\.7%\+0\.2%\+0\.2%Transfer function only

LR\-LoRA introduces negligible memory overhead \(<1%<1\\%\) compared to LoRA baseline\. Training throughput remains nearly identical to LoRA due to minimal computational overhead\. Transfer function parameters scale with number of adapted modules, not model size\. Memory scaling is dominated by optimizer states for adapter parameters, not transfer functions\.

Table 25:Trainable parameters \(adapters only\)\.All methods used rankr=16r\{=\}16and the same adapted module set\. LoRA trainPLoRAP\_\{\\text\{LoRA\}\}parameters via𝐀,𝐁\\mathbf\{A\},\\mathbf\{B\}\([Equation˜1](https://arxiv.org/html/2606.04325#S3.E1)\)\. LR\-LoRA addsΔ​PLR\-LoRA=2​N​\|ℒ\|\\Delta P\_\{\\text\{LR\-LoRA\}\}=2N\|\\mathcal\{L\}\|parameters forϕ\\phi\([Equation˜4](https://arxiv.org/html/2606.04325#S3.E4)\) withN=50N\{=\}50, whereℒ\\mathcal\{L\}is the set of adapted modules\.MethodQwen2Phi\-3LLaMA3RoBERTaVeRA0\.26M0\.26M0\.26M0\.26MLoRA1\.77M14\.7M17\.2M1\.03MLR\-LoRA1\.79M14\.72M17\.22M1\.04M\+Δ​P\\Delta P/PLoRAP\_\{\\text\{LoRA\}\}\+0\.95%\+0\.15%\+0\.13%\+0\.7%These measurements correspond to the matched\-protocol setup; the worst\-case overheads are\+3%peak memory and2%throughput decrease relative to LoRA\.

Training Efficiency Analysis[Table˜26](https://arxiv.org/html/2606.04325#A7.T26)summarizes accuracy versus throughput for LoRA and LR\-LoRA at two ranks, demonstrating that performance gains come with minimal computational overhead\.

Table 26:Accuracy and throughput tradeoff onQwen2\-0\.5B\(commonsense, 170k\)\.Accuracy is the unweighted average across the eight tasks; throughput is tokens/sec during training\. Takeaway: LR\-LoRA improves accuracy at matched throughput across ranks\.MethodAvg\. accuracyThroughput \(tok/s\)LoRAr=16r\{=\}1657\.426,300LR\-LoRAr=16r\{=\}1661\.026,100LoRAr=32r\{=\}3257\.324,900LR\-LoRAr=32r\{=\}3261\.124,700Memory Usage Breakdown[Table˜27](https://arxiv.org/html/2606.04325#A7.T27)provides detailed memory usage analysis\. The key insight is that transfer function parameters contribute negligible overhead \(<1%<1\\%\) compared to the adapter parameters and optimizer states\.

Table 27:Training efficiency analysis across architectures\.Memory usage and training throughput for LR\-LoRA vs LoRA baselines, demonstrating negligible computational overhead despiteO​\(N\)O\(N\)transfer function complexity\.ArchitectureMemory \(GB\)Throughput \(tok/s\)OverheadLoRALR\-LoRALoRALR\-LoRAMemorySpeedQwen2\-0\.5B2\.12\.131803165\+0\.5%\-0\.5%Phi\-3\-8B16\.816\.911401132\+0\.6%\-0\.7%LLaMA3\-8B18\.218\.411501138\+1\.1%\-1\.0%RoBERTa\-base1\.41\.442504215\+0\.3%\-0\.8%

### G\.2Singular\-Value Spectrum of Learned Updates

We examined the spectral structure of the learned update matrices to assess how LoRA and LR\-LoRA distributed energy across singular directions\. For each adapted module, we compute the singular values\{σi​\(U\)\}\\\{\\sigma\_\{i\}\(U\)\\\}of the learned update matrixUU\. To facilitate comparison across layers and architectures, we sort the singular values in non\-increasing order and normalize them by the leading valueσ1​\(U\)\\sigma\_\{1\}\(U\)for each module\. When aggregating across modules, we applied a consistent aggregation rule across methods \(e\.g\., median across adapted modules within the same projection type\), with the specific aggregation method detailed in the figure caption\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x6.png)Figure 4:Singular\-value spectrum of learned updates\.Singular values of the learned update matrices at rankr=16r\{=\}16for LoRA \(U=𝐁𝐀U=\\mathbf\{BA\}\) and LR\-LoRA \(U=ϕ​\(𝐁𝐀\)U=\\phi\(\\mathbf\{BA\}\)\), sorted in non\-increasing order and normalized byσ1​\(U\)\\sigma\_\{1\}\(U\)\.Qwen2\-0\.5Bon the commonsense reasoning suite, 15k subset regime\. LR\-LoRA exhibits a slower spectral decay, consistent with higher complexity\.

## Appendix HOptimization Diagnostics

### H\.1Loss Landscape Visualization

We provide a qualitative visualization of the local optimization geometry around a converged solution by evaluating the training objective on a two\-dimensional slice of the*trainable*parameter\-subspace\. Starting from a converged set of trainable parametersθ⋆\\theta^\{\\star\}\(LoRA: adapter parameters; LR\-LoRA: adapter parameters together with the transfer\-function parameters ofϕ\\phi\), we sample two random directionsd1,d2d\_\{1\},d\_\{2\}restricted to the corresponding trainable subspace and evaluate the objective on the grid

θ⋆\+a​d1\+b​d2,\\theta^\{\\star\}\+a\\,d\_\{1\}\+b\\,d\_\{2\},for\(a,b\)\(a,b\)within a fixed range\. Within each method, the directions are normalized to have the sameℓ2\\ell\_\{2\}norm in the method’s trainable parameter space, ensuring a comparable scale*within*the method\. Because random directions are normalized within each method’s parameter space, these visualizations are*not*comparable across methods and serve only for qualitative interpretation within each method\. This diagnostic does not constitute a quantitative measure of sharpness or generalization\. The resulting landscape is shown in[Figure˜5](https://arxiv.org/html/2606.04325#A8.F5)\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x7.png)Figure 5:Loss landscape \(2D slice; rendered in 3D\)\.Training objective evaluated on a two\-dimensional plane through a converged solution, using directions restricted to trainable parameters \(LoRA: adapter parameters; LR\-LoRA: adapter andϕ\\phiparameters\)\.

## Appendix ICross\-Architecture Nonlinearity Evolution Analysis

This section offers a thorough empirical examination of LR\-LoRA’s learned mappingϕ​\(z\)\\phi\(z\)across two distinct architectures: Qwen2\-0\.5B \(24 layers, 494M parameters\) and LLaMA\-3 8B \(32 layers, 8B parameters\)\. Our analysis elucidates the influence of the model scale, architectural depth, and training dynamics on the development of specialized nonlinear update patterns\.

### I\.1Experimental Protocol and Analysis Framework

We examine the temporal progression of the learned transfer function across three pivotal training stages:

- •Early stage \(IT100\):This initial adaptation phase is marked by a rapid evolution of parameters from a near\-zero initialization\.
- •Mid stage \(IT1000\):During this stabilization phase, component\-specific patterns begin to manifest\.
- •Final stage \(IT2000\):The convergence phase is characterized by the emergence of fully specialized nonlinear mappings\.

For depth analysis, we examined three representative layers:early depth\(L1\),middle depth\(L11/L16\), andlate depth\(L23/L32\), corresponding to distinct stages of hierarchical representation learning\. Each visualization displays the learned transfer functionϕ​\(z\)\\phi\(z\)where the x\-axis represents the low\-rank magnitudez=\(B​A\)i​jz=\(BA\)\_\{ij\}and the y\-axis shows the corresponding weight update magnitude\. The function shape encodes the adaptive strategy: linear regions preserve the standard LoRA behavior, whereas nonlinear regions indicate learned specialization\. The saturation regions \(plateaus\) correspond to amplitude limiting, whereas the oscillatory patterns suggest multimodal update strategies\.

### I\.2Qwen2\-0\.5B: Small Model Dynamics

MLP Component Evolution in Qwen2[Figures˜6](https://arxiv.org/html/2606.04325#A9.F6),[7](https://arxiv.org/html/2606.04325#A9.F7)and[8](https://arxiv.org/html/2606.04325#A9.F8)reveal the temporal development of MLP nonlinearities in the smaller Qwen2 architecture\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x8.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x9.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x10.png)\(c\)Final stage \(iteration 2000\)\.

Figure 6:Qwen2 MLP nonlinearity evolution at layer 1\.Temporal progression from IT100→1000→2000 reveals rapid convergence to near\-linear behavior with minimalα\\alpha\-growth\. Early layers in small models exhibit conservative adaptation strategies\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x11.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x12.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x13.png)\(c\)Final stage \(iteration 2000\)\.

Figure 7:Qwen2 MLP nonlinearity evolution at layer 11\.Mid\-depth dynamics show emerging nonlinearity withω\\omegabandwidth expansion and moderateαi\\alpha\_\{i\}\-coefficient differentiation\. Qwen2’s compact architecture necessitates efficient parameter utilization\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x14.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x15.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x16.png)\(c\)Final stage \(iteration 2000\)\.

Figure 8:Qwen2 MLP nonlinearity evolution at layer 23\.Deepest layer exhibits pronounced amplitude scaling \(α\\alphagrowth\) and complex coefficient patterns reflecting high\-level feature integration demands in compact architectures\.Attention Nonlinearity Evolution[Figures˜9](https://arxiv.org/html/2606.04325#A9.F9),[10](https://arxiv.org/html/2606.04325#A9.F10)and[11](https://arxiv.org/html/2606.04325#A9.F11)present the corresponding temporal analysis for attention projections, revealing module\-specific adaptation patterns\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x17.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x18.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x19.png)\(c\)Final stage \(iteration 2000\)\.

Figure 9:Qwen2 attention nonlinearity evolution at layer 1\.Attention projections develop distinct oscillatory patterns early in training, reflecting the multi\-head attention mechanism’s need for diverse query\-key\-value mappings\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x20.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x21.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x22.png)\(c\)Final stage \(iteration 2000\)\.

Figure 10:Qwen2 attention nonlinearity evolution at layer 11\.Mid\-depth attention exhibits complex multi\-modal patterns suggesting specialized head\-specific adaptation strategies, with pronouncedω\\omegabandwidth differentiation\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x23.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x24.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x25.png)\(c\)Final stage \(iteration 2000\)\.

Figure 11:Qwen2 attention nonlinearity evolution at layer 23\.Deep attention layers develop highly specialized multi\-peak structures, indicating sophisticated head\-specific update strategies for high\-level semantic processing\.Qwen2 mathematical patterns\.Analysis of Qwen2 evolution revealed several critical insights into small\-model adaptation dynamics\. The parameter evolution follows predictable trajectories:α\\alphaexhibits monotonic growth with layer depth \(L1:α≈0\.1→0\.3\\alpha\\approx 0\.1\\to 0\.3; L23:α≈0\.1→0\.8\\alpha\\approx 0\.1\\to 0\.8\), indicating a depth\-dependent amplitude scaling\. The bandwidth parameterω\\omegaremains relatively stable across training \(ω≈1\.0±0\.2\\omega\\approx 1\.0\\pm 0\.2\), suggesting that Qwen2’s compact representation benefits from a moderate spectral bandwidth\.

### I\.3LLaMA\-3 8B: Large Model Scaling Dynamics

The transition to LLaMA\-3 8B revealed fundamentally different nonlinearity evolution patterns, demonstrating how architectural scale influences parameter\-efficient adaptation strategies\. The larger parameter space and extended depth \(32 layers\) enable more sophisticated and gradual specialization compared to Qwen2’s aggressive adaptation\.

MLP Component Evolution in LLaMA\-3[Figures˜12](https://arxiv.org/html/2606.04325#A9.F12),[13](https://arxiv.org/html/2606.04325#A9.F13)and[14](https://arxiv.org/html/2606.04325#A9.F14)demonstrate the qualitatively different adaptation patterns in the large\-scale architecture\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x26.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x27.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x28.png)\(c\)Final stage \(iteration 2000\)\.

Figure 12:LLaMA\-3 MLP nonlinearity evolution at layer 1\.Conservative early\-layer adaptation with delayed amplitude growth \(α\\alpharemains<0\.1<0\.1through IT1000\) reflects large\-model stability requirements and distributed parameter utilization\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x29.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x30.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x31.png)\(c\)Final stage \(iteration 2000\)\.

Figure 13:LLaMA\-3 MLP nonlinearity evolution at layer 16\.Mid\-depth layers in LLaMA\-3 develop sophisticated multi\-modal patterns with controlled amplitude scaling, exhibiting polynomial\-like transfer functions that suggest feature combination strategies\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x32.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x33.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x34.png)\(c\)Final stage \(iteration 2000\)\.

Figure 14:LLaMA\-3 MLP nonlinearity evolution at layer 32\.Deep layers achieve remarkable complexity with multi\-peak structures and controlled saturation regions, indicating sophisticated high\-level semantic update strategies enabled by large\-scale architectures\.Attention Component Evolution in LLaMA\-3[Figures˜15](https://arxiv.org/html/2606.04325#A9.F15),[16](https://arxiv.org/html/2606.04325#A9.F16)and[17](https://arxiv.org/html/2606.04325#A9.F17)reveal the emergence of highly specialized attention patterns that distinguish large\-scale from compact architectures\.

![Refer to caption](https://arxiv.org/html/2606.04325v1/x35.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x36.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x37.png)\(c\)Final stage \(iteration 2000\)\.

Figure 15:LLaMA\-3 attention nonlinearity evolution at layer 1\.Early attention layers develop gentle oscillatory patterns with extended bandwidth utilization \(ω≈0\.5\\omega\\approx 0\.5\), enabling fine\-grained query\-key interaction modeling\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x38.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x39.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x40.png)\(c\)Final stage \(iteration 2000\)\.

Figure 16:LLaMA\-3 attention nonlinearity evolution at layer 16\.Mid\-depth attention exhibits harmonic patterns with multiple frequency components, suggesting multi\-scale attention mechanisms operating simultaneously across different semantic granularities\.![Refer to caption](https://arxiv.org/html/2606.04325v1/x41.png)\(a\)Early stage \(iteration 100\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x42.png)\(b\)Mid stage \(iteration 1000\)\.
![Refer to caption](https://arxiv.org/html/2606.04325v1/x43.png)\(c\)Final stage \(iteration 2000\)\.

Figure 17:LLaMA\-3 attention nonlinearity evolution at layer 32\.Deep attention layers achieve step\-function\-like patterns with sharp transitions, indicating binary decision mechanisms for high\-level semantic attention allocation\.
### I\.4Cross\-Architecture Comparative Analysis

The systematic comparison between Qwen2\-0\.5B and LLaMA\-3 8B elucidates the fundamental scaling laws in nonlinear parameter\-efficient adaptation, which have significant theoretical and practical implications\.

Amplitude scaling dynamics\.The parameterα\\alphaexhibits notably different trajectories across the architectures\. Qwen2 demonstrates aggressive amplitude growth \(Δ​α/Δ​t≈0\.4\\Delta\\alpha/\\Delta t\\approx 0\.4per 1000 iterations\), which is indicative of the compact model’s requirement for efficient parameter utilization\. Conversely, LLaMA\-3 displayed conservative scaling \(Δ​α/Δ​t≈0\.1\\Delta\\alpha/\\Delta t\\approx 0\.1per 1000 iterations\), aligning with the distributed parameter efficiency in large models, where individual updates bear less representational burden\.

Bandwidth specialization patterns\.The spectral parameterω\\omegareveals architecture\-dependent frequency preference\. Qwen2 maintains a moderate bandwidth \(ω∈\[0\.8,1\.2\]\\omega\\in\[0\.8,1\.2\]\) across layers, suggesting uniform spectral utilization\. LLaMA\-3 exhibits layer\-stratified bandwidth allocation: early layers favor extended bandwidth \(ω≈0\.5\\omega\\approx 0\.5\), mid\-layers employ moderate frequencies \(ω≈0\.8\\omega\\approx 0\.8\), and deep layers focus on narrow\-band patterns \(ω≈1\.5\\omega\\approx 1\.5\)\. This stratification indicates a hierarchical frequency decomposition in large architectures\.

Convergence rate scaling\.The temporal dynamics adhere to distinct power laws\. Qwen2 exhibits rapid convergence with‖∇f‖2∝t−1\.2\\\|\\nabla f\\\|\_\{2\}\\propto t^\{\-1\.2\}, a characteristic of aggressive optimization in capacity\-limited regimes\. LLaMA\-3 follows a slower convergence‖∇f‖2∝t−0\.8\\\|\\nabla f\\\|\_\{2\}\\propto t^\{\-0\.8\}, consistent with the careful exploration of the expanded parameter space\.

Module\-specific adaptation strategies\.Both architectures demonstrate consistent module specialization, albeit with varying sophistication\. The Qwen2 MLP components develop primarily monotonic or bi\-modal patterns, whereas the attention components exhibit oscillatory behavior\. LLaMA\-3 facilitates more complex specialization: MLP components achieve polynomial\-like shapes with multiple inflection points, whereas attention components develop harmonic structures with clearly defined frequency bands\.

### I\.5Empirical Scaling Observations

These empirical observations reveal several patterns of parameter\-efficient adaptation in transformer architectures:

1. 1\.Scale\-dependent amplitude patterns:The optimal amplitude scaling shows systematic relationships with model size, indicating that larger models require proportionally smaller individual updates\.
2. 2\.Hierarchical frequency patterns:Large architectures benefit from layer\-stratified bandwidth allocation, with optimal bandwidth values varying systematically with depth\.