TaRA: Training-Aware Low-Rank Adaptation Initialization

# TaRA: Training-Aware Low-Rank Adaptation Initialization
Source: [https://arxiv.org/html/2609.02639](https://arxiv.org/html/2609.02639)
Eunhyeok ParkEmail:[eh\.park@postech\.ac\.kr](mailto:[email protected])Affiliation:Department of Computer Science and EngineeringAffiliation:Graduate School of Artificial IntelligencePohang University of Science and Technology \(POSTECH\)

###### Abstract

Low\-Rank Adaptation \(LoRA\) has become a de facto standard for parameter\-efficient fine\-tuning \(PEFT\), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low\-rank decomposition\. Existing approaches attempt to construct high\-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients\. However, these methods do not directly account for the training dynamics of the full\-rank model\. In this paper, we propose Training\-aware Low\-Rank Adaptation Initialization \(TaRA\), a method that initializes LoRA such that the gradients induced by the low\-rank factors closely approximate the gradient of the corresponding full\-rank weight matrix\. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead\. Across diverse and challenging fine\-tuning tasks, TaRA consistently outperforms prior state\-of\-the\-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization\.

## 1Introduction

Parameter\-efficient fine\-tuning \(PEFT\) has significantly reduced the cost of adapting large language models \(LLMs\), enabling their widespread deployment in practical applications[Houlsby et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib1);[Li and Liang \(2021\)](https://arxiv.org/html/2609.02639#bib.bib3)\. Among PEFT methods, Low\-Rank Adaptation \(LoRA\)[Hu et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib2)has emerged as the most widely adopted approach due to its simplicity, substantial reduction in fine\-tuning resource requirements, and absence of inference\-time overhead, as the learned updates can be merged into the base weights after training\. These advantages have established LoRA as a de facto standard for efficient LLM fine\-tuning\.

Figure 1:Gradient similarity, measured as the cosine similarity between the one\-step gradient of the combined LoRA adapter and that of the full weight, correlates with downstream accuracy\.Despite these benefits, LoRA introduces an inherent trade\-off: the structured bottleneck imposed by low\-rank decomposition alters the optimization trajectory relative to full\-rank training, which can lead to suboptimal convergence even when model capacity is sufficient\. Consequently, improving the effectiveness of LoRA without sacrificing its efficiency has become an important research direction[Meng et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib4);[Buyukakyuz \(2024\)](https://arxiv.org/html/2609.02639#bib.bib62);[Hayou et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib63)\.

One promising direction is to design more informed initialization strategies for the low\-rank adapters\. While the original LoRA initializes its adapters with random or zero values, subsequent studies have shown that low\-rank initializations guided by pretrained weight distributions \(e\.g\., PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib4)\), joint weight\-data statistics \(e\.g\., CorDA[Yang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib5)\), or gradients from a single forward\-backward pass \(e\.g\., LoRA\-GA, LoRA\-One[Wang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib7);[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib6)\) consistently yield improved fine\-tuning performance\. These methods preserve the practical advantages of LoRA while alleviating degradation caused by low\-rank constraints\.

To further address this limitation, we propose Training\-aware Low\-Rank Adaptation \(TaRA\), a new LoRA initialization method\. Unlike prior approaches that approximate tensor statistics, TaRA is designed to preserve directions that are important to the local gradient behavior of full\-weight training, allowing the low\-rank parameterization to better reflect training\-relevant information at initialization\. Grounded in a mathematical formulation, TaRA derives a low\-rank initialization that preserves training\-relevant directions in the local gradient field by jointly leveraging activation covariance, gradient covariance, and pretrained weights from a single forward\-backward pass\. As illustrated in Figure[1](https://arxiv.org/html/2609.02639#S1.F1), this initialization achieves substantially higher gradient alignment than prior methods at the same rank, leading to consistent downstream improvements\. Extensive experiments on challenging reasoning tasks, including mathematical problem solving, code generation, and commonsense reasoning, demonstrate that TaRA outperforms existing approaches\.

## 2Prior Work on LoRA Initialization

Figure 2:Illustration of our single\-layer procedure\. We collect activations𝐗\\mathbf\{X\}and gradients𝐆\\mathbf\{G\}to form covariances𝚺X\\mathbf\{\\Sigma\}\_\{X\}and𝚺G\\mathbf\{\\Sigma\}\_\{G\}\. PiSSA applies SVD to𝐖\\mathbf\{W\}; CorDA applies SVD to𝐖​𝚺X\\mathbf\{W\}\\mathbf\{\\Sigma\}\_\{X\}and maps back with𝚺X−1\\mathbf\{\\Sigma\}\_\{X\}^\{\-1\}; LoRA\-One applies SVD to−𝐆\-\\mathbf\{G\}\. In contrast, TaRA applies SVD to𝚺G​𝐖​𝚺X\\mathbf\{\\Sigma\}\_\{G\}\\mathbf\{W\}\\mathbf\{\\Sigma\}\_\{X\}and maps back with𝚺G−1\\mathbf\{\\Sigma\}\_\{G\}^\{\-1\}and𝚺X−1\\mathbf\{\\Sigma\}\_\{X\}^\{\-1\}\.In the original LoRA, the input\-side factor matrix𝐀\\mathbf\{A\}is randomly initialized, while the output\-side factor matrix𝐁\\mathbf\{B\}is set to zero\. Although this design ensures that fine\-tuning starts from the pretrained representations, the resulting initialization contains no meaningful task\-related information, which can lead to slow early\-stage optimization and suboptimal convergence[Wang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib7)\.

Motivated by these limitations, numerous studies have explored improved initialization strategies for LoRA \(Figure[2](https://arxiv.org/html/2609.02639#S2.F2)\)\. Although the decomposed path produces a non\-zero update, fine\-tuning can still start from the pretrained weight𝐖0\\mathbf\{W\}\_\{0\}by absorbing this modification into a frozen residual weight, defined as𝐖r​e​s=𝐖0−𝐁𝐀\\mathbf\{W\}\_\{res\}=\\mathbf\{W\}\_\{0\}\-\\mathbf\{B\}\\mathbf\{A\}\. With an appropriate choice of𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, a properly initialized model can accelerate training and typically converges to better solutions than random initialization\.

A key early work in this line of research is PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib4), which proposes a data\-agnostic initialization that leverages the pretrained weight matrix𝐖0∈ℝd​o​u​t×di​n\\mathbf\{W\}\_\{0\}\\in\\mathbb\{R\}^\{d\{out\}\\times d\_\{in\}\}\. Specifically, PiSSA performs singular value decomposition \(SVD\),𝐖0=𝐔𝐒𝐕⊤\\mathbf\{W\}\_\{0\}=\\mathbf\{U\}\\mathbf\{S\}\\mathbf\{V\}^\{\\top\}, and initializes LoRA using the top\-rrsingular components:

𝐁\\displaystyle\\mathbf\{B\}=𝐔\[:,:r\]𝐒1/2\[:r,:r\]∈ℝd​o​u​t×r,\\displaystyle=\\mathbf\{U\}\{\[:,:r\]\}\\mathbf\{S\}^\{1/2\}\{\[:r,:r\]\}\\in\\mathbb\{R\}^\{d\{out\}\\times r\},\(1\)𝐀\\displaystyle\\mathbf\{A\}=𝐒1/2\[:r,:r\]𝐕⊤\[:,:r\]∈ℝr×di​n\.\\displaystyle=\\mathbf\{S\}^\{1/2\}\{\[:r,:r\]\}\\mathbf\{V\}^\{\\top\}\{\[:,:r\]\}\\in\\mathbb\{R\}^\{r\\times d\_\{in\}\}\.\(2\)
More recently, CorDA[Yang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib5)extends this idea by incorporating input statistics in addition to pretrained weights\. CorDA first estimates the input activation covariance𝚺X=𝐗𝐗⊤\\mathbf\{\\Sigma\}\_\{X\}=\\mathbf\{X\}\\mathbf\{X\}^\{\\top\}from input activations𝐗\\mathbf\{X\}, and then performs the following decomposition:

SVD​\(𝐖0​𝚺X\)​𝚺X−1=𝐔¯​𝐒¯​𝐕¯⊤\.\\text\{SVD\}\(\\mathbf\{W\}\_\{0\}\\mathbf\{\\Sigma\}\_\{X\}\)\\mathbf\{\\Sigma\}\_\{X\}^\{\-1\}=\\mathbf\{\\bar\{U\}\}\\mathbf\{\\bar\{S\}\}\\mathbf\{\\bar\{V\}\}^\{\\top\}\.\(3\)The resulting components are used to initialize𝐁\\mathbf\{B\},𝐀\\mathbf\{A\}, and𝐖r​e​s\\mathbf\{W\}\_\{res\}in the same manner as PiSSA\. Both PiSSA and CorDA align the LoRA subspace with principal components derived from pretrained weights or input\-dependent statistics, resulting in faster convergence and improved accuracy\.

Orthogonal to these approaches, LoRA\-GA[Wang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib7)and its successor, LoRA\-One[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib6), leverage gradient information\. These methods are motivated by the observation that the fine\-tuning quality of LoRA depends on the alignment of its update direction with the dominant singular subspace of the full fine\-tuning gradient\. Based on this insight, LoRA\-One performs SVD of the one\-step full gradient of weight𝐆\\mathbf\{G\},

SVD​\(−𝐆\)=𝐔^​𝐒^​𝐕^⊤,\\text\{SVD\}\(\-\\mathbf\{G\}\)=\\mathbf\{\\hat\{U\}\}\\mathbf\{\\hat\{S\}\}\\mathbf\{\\hat\{V\}\}^\{\\top\},\(4\)and use the top\-rrsingular components to initialize the LoRA\. By leveraging gradient information, these methods initialize the LoRA modules along directions that are partially informed by training signals, often leading to improved fine\-tuning performance in practice\.

However, despite utilizing internal tensor statistics of the pretrained model, prior approaches primarily focus on enhancing the representational capacity of the low\-rank decomposition, rather than explicitly aligning it with the learning dynamics of full fine\-tuning\. Consequently, a performance gap remains, which cannot be effectively closed by merely increasing the rank or tuning hyperparameters\.

## 3Proposed Idea: TaRA

In this section, we introduce TaRA, a novel LoRA initialization method designed to enhance fine\-tuning quality under a strict low\-rank constraint\. We begin by formalizing the objective, derive a low\-rank solution, and then describe its practical implementation\.

### 3\.1Motivation

The key motivation behind TaRA is simple yet fundamental: even under the low\-rank bottleneck imposed by LoRA, we aim to construct a low\-rank parameterization whose induced local gradient behavior closely approximates that of the corresponding full\-rank weight\. In other words, TaRA embeds training\-relevant directions of the local gradient field into the low\-rank structure\. This perspective naturally leads to the following question:*Under a rank\-rrconstraint, how can we construct a low\-rank parameterization that preserves the directions most relevant to the local gradient behavior of full fine\-tuning?*

To formalize this intuition, we consider the following objective:

θ~=arg⁡minθ⁡‖∇ℒ​\(θ\)−∇ℒ​\(θ0\)‖F2,s\.t\.​rank​\(θ\)≤r,\\begin\{split\}\\tilde\{\\theta\}=\\arg\\min\_\{\\theta\}\\left\\lVert\\nabla\\mathcal\{L\}\(\\theta\)\-\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)\\right\\rVert^\{2\}\_\{F\},\\\\ \\quad\\text\{s\.t\. \}\\mathrm\{rank\}\(\\theta\)\\leq r,\\end\{split\}\(5\)
whereℒ\\mathcal\{L\}denotes the task loss\. Intuitively, this objective seeks a rank\-constrained parameterization that preserves the local gradient behavior of the corresponding full\-rank parameter atθ0\\theta\_\{0\}\.

### 3\.2Training\-Relevant Decomposition

To obtain a tractable formulation, we adopt a standard second\-order view of the loss landscape around the initialization point\. In particular, we focus on the one\-step gradient at initialization[Wang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib7), which captures the dominant training signal in the early stage of optimization\. In practice, this approximation is sufficient to guide effective low\-rank initialization, as demonstrated by the convergence results in Section[4\.2](https://arxiv.org/html/2609.02639#S4.SS2)\.

Using a second\-order Taylor expansion of the loss aroundθ0\\theta\_\{0\}, we obtain

ℒ⁡\(θ\)≈ℒ\(θ0\)\+∇ℒ\(θ0\)⊤\(θ−θ0\)\+12​\(θ−θ0\)⊤​𝐇​\(θ−θ0\),\\begin\{split\}\\mathcal\{L\}\(\\theta\)\\approx\{\}&\\mathcal\{L\}\(\\theta\_\{0\}\)\+\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)^\{\\top\}\(\\theta\-\\theta\_\{0\}\)\\\\ &\+\\frac\{1\}\{2\}\(\\theta\-\\theta\_\{0\}\)^\{\\top\}\\mathbf\{H\}\(\\theta\-\\theta\_\{0\}\),\\end\{split\}\(6\)
where𝐇\\mathbf\{H\}denotes the Hessian evaluated atθ0\\theta\_\{0\}\.

In large models, directly computing the Hessian is infeasible\. Following common practice, we instead use the Fisher information matrixℱ\\mathcal\{F\}as a tractable surrogate for local curvature[Chekalina et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib57);[Martens \(2020\)](https://arxiv.org/html/2609.02639#bib.bib58)\. Using the K\-FAC factorization[Martens and Grosse \(2015\)](https://arxiv.org/html/2609.02639#bib.bib10), the Fisher matrix can be written as

ℱ≈𝚺X⊗𝚺G,\\mathcal\{F\}\\approx\\mathbf\{\\Sigma\}\_\{X\}\\otimes\\mathbf\{\\Sigma\}\_\{G\},\(7\)
where𝚺G=𝐆𝐆⊤\\mathbf\{\\Sigma\}\_\{G\}=\\mathbf\{G\}\\mathbf\{G\}^\{\\top\}denotes the gradient covariance\. Under this formulation, differentiating Eq\. \([6](https://arxiv.org/html/2609.02639#S3.E6)\) with respect toθ\\thetayields \(derivation is in Appendix\):

∇ℒ​\(θ\)−∇ℒ​\(θ0\)≈𝚺G​\(θ−θ0\)​𝚺X\.\\begin\{split\}\\nabla\\mathcal\{L\}\(\\theta\)\-\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)\\approx\\mathbf\{\\Sigma\}\_\{G\}\(\\theta\-\\theta\_\{0\}\)\\mathbf\{\\Sigma\}\_\{X\}\.\\end\{split\}\(8\)
This relation reveals a key insight: the change in gradient induced by a parameter update is modulated by the curvature structure of the loss\. Consequently, directions associated with large Fisher or Hessian values have a disproportionately large influence on the optimization trajectory\. Preserving these directions is therefore crucial for faithfully approximating full fine\-tuning under a low\-rank space\.

Empirically, the Hessian spectrum of modern neural networks is known to exhibit a highly structured form, consisting of a small set of prominent outlier eigenvectors and a near\-zero bulk that is largely orthogonal to them[Gur\-Ari et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib8);[Sagun et al\. \(2017\)](https://arxiv.org/html/2609.02639#bib.bib9)\. Moreover, prior work shows that gradient descent effectively evolves within a small subspace spanned by the top Hessian eigenvectors[Gur\-Ari et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib8)\. These observations suggest that dominant curvature directions provide a natural criterion for identifying effective low\-rank training updates\.

Motivated by this insight, we seek a rank\-rrapproximationθ~\\tilde\{\\theta\}that minimizes the curvature\-weighted gradient deviation in Eq\. \([8](https://arxiv.org/html/2609.02639#S3.E8)\):

θ~≈arg⁡minθ⁡∥𝚺G​\(θ−θ0\)​𝚺X∥F2,s\.t\.rank​\(θ\)≤r\.\\begin\{split\}\\tilde\{\\theta\}\\approx\\arg\\min\_\{\\theta\}\\lVert\\mathbf\{\\Sigma\}\_\{G\}\(\\theta\-\\theta\_\{0\}\)\\mathbf\{\\Sigma\}\_\{X\}\\rVert^\{2\}\_\{F\},\\\\ \\quad\\text\{s\.t\. \}\\text\{rank\}\(\\theta\)\\leq r\.\\end\{split\}\(9\)
This corresponds to a classical low\-rank matrix approximation problem\. By the Eckart–Young theorem[Eckart and Young \(1936\)](https://arxiv.org/html/2609.02639#bib.bib11);[Golub and Van Loan \(2013\)](https://arxiv.org/html/2609.02639#bib.bib12), the optimal solution is obtained by retaining the top\-rrsingular components of decomposition:

θ~≈𝚺G−1​SVDr⁡\(𝚺G​θ0​𝚺X\)​𝚺X−1,\\begin\{split\}\\tilde\{\\theta\}\\approx\\mathbf\{\\Sigma\}\_\{G\}^\{\-1\}\\operatorname\{SVD\}\_\{r\}\(\\mathbf\{\\Sigma\}\_\{G\}\\theta\_\{0\}\\mathbf\{\\Sigma\}\_\{X\}\)\\mathbf\{\\Sigma\}\_\{X\}^\{\-1\},\\end\{split\}\(10\)
whereSVDr⁡\(⋅\)\\operatorname\{SVD\}\_\{r\}\(\\cdot\)denotes the truncated SVD\.111This is often confused with Fisher\-weighted model compression; key differences are described in Appendix[H](https://arxiv.org/html/2609.02639#A8)\.

This result shows that the optimal low\-rank approximation corresponds to preserving the dominant curvature directions of the pretrained parameterθ0\\theta\_\{0\}\.

### 3\.3Implementation of TaRA

Algorithm 1TaRA, init for single layerInput:Pre\-trained weight of single linear layer

𝐖0∈ℝdo​u​t×di​n\\mathbf\{W\}\_\{0\}\\in\\mathbb\{R\}^\{d\_\{out\}\\times d\_\{in\}\}, calibration dataset

DD, LoRA rank

r∈ℕr\\in\\mathbb\{N\}
Collect:1: Collect input activations in forward pass

𝐗∈ℝdi​n×\|B\|​L\\mathbf\{X\}\\in\\mathbb\{R\}^\{d\_\{in\}\\times\|B\|L\}2: Collect weight gradients in backward pass

𝐆∈ℝdo​u​t×\|B\|​L\\mathbf\{G\}\\in\\mathbb\{R\}^\{d\_\{out\}\\times\|B\|L\}3: Compute activation covariance

𝚺X←𝐗𝐗⊤∈ℝdi​n×di​n\\mathbf\{\\Sigma\}\_\{X\}\\leftarrow\\mathbf\{X\}\\mathbf\{X\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{in\}\\times d\_\{in\}\}4: Compute gradient covariance

𝚺G←𝐆𝐆⊤∈ℝdo​u​t×do​u​t\\mathbf\{\\Sigma\}\_\{G\}\\leftarrow\\mathbf\{G\}\\mathbf\{G\}^\{\\top\}\\in\\mathbb\{R\}^\{d\_\{out\}\\times d\_\{out\}\}
Init:1:

𝐔~,𝐒~,𝐕~⊤←SVD​\(𝚺G​𝐖0​𝚺X\)\\tilde\{\\mathbf\{U\}\},\\tilde\{\\mathbf\{S\}\},\\tilde\{\\mathbf\{V\}\}^\{\\top\}\\leftarrow\\text\{SVD\}\(\\mathbf\{\\Sigma\}\_\{G\}\\mathbf\{W\}\_\{0\}\\mathbf\{\\Sigma\}\_\{X\}\)2:

𝐁←𝚺G−1𝐔~\[:,:r\]𝐒~1/2\[:r,:r\]\\mathbf\{B\}\\leftarrow\\mathbf\{\\Sigma\}\_\{G\}^\{\-1\}\\tilde\{\\mathbf\{U\}\}\_\{\[:,:r\]\}\\tilde\{\\mathbf\{S\}\}^\{1/2\}\_\{\[:r,:r\]\}∈ℝdo​u​t×r\\in\\mathbb\{R\}^\{d\_\{out\}\\times r\}3:

𝐀←𝐒~1/2\[:r,:r\]𝐕~⊤\[:,:r\]𝚺X−1\\mathbf\{A\}\\leftarrow\\tilde\{\\mathbf\{S\}\}^\{1/2\}\_\{\[:r,:r\]\}\\tilde\{\\mathbf\{V\}\}^\{\\top\}\_\{\[:,:r\]\}\\mathbf\{\\Sigma\}\_\{X\}^\{\-1\}∈ℝr×di​n\\in\\mathbb\{R\}^\{r\\times d\_\{in\}\}4:

𝐖r​e​s←𝐖0−𝐁𝐀\\mathbf\{W\}\_\{res\}\\leftarrow\\mathbf\{W\}\_\{0\}\-\\mathbf\{B\}\\mathbf\{A\}∈ℝdo​u​t×di​n\\in\\mathbb\{R\}^\{d\_\{out\}\\times d\_\{in\}\}

Based on the above formulation, TaRA can be implemented as an efficient layer\-wise LoRA initialization procedure for linear layers, summarized in Algorithm[1](https://arxiv.org/html/2609.02639#alg1)\. In theCollectstage, we gather task\-dependent statistics using a small calibration datasetDD\. During the forward pass, we record input activations and concatenate them to form𝐗∈ℝdin×\|B\|​L\\mathbf\{X\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\\times\|B\|L\}\. During the backward pass, we collect the corresponding weight gradients to obtain𝐆∈ℝdout×\|B\|​L\\mathbf\{G\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times\|B\|L\}\. These statistics are used to estimate the covariance matrices𝚺X\\mathbf\{\\Sigma\}\_\{X\}and𝚺G\\mathbf\{\\Sigma\}\_\{G\}\. In theInitstage, we compute the SVD of𝚺G​𝐖0​𝚺X\\mathbf\{\\Sigma\}\_\{G\}\\mathbf\{W\}\_\{0\}\\mathbf\{\\Sigma\}\_\{X\}and project the resulting components back to the original space using𝚺G−1\\mathbf\{\\Sigma\}\_\{G\}^\{\-1\}and𝚺X−1\\mathbf\{\\Sigma\}\_\{X\}^\{\-1\}\. This produces singular components that capture the training\-relevant directions under the Fisher\-weighted metric\.

In practice, the covariance matrices may be rank\-deficient[Yankun et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib56);[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib51), which makes direct inversion unstable\. To ensure numerical stability, we apply diagonal damping,𝚺G←𝚺G\+c​β​𝐈\\mathbf\{\\Sigma\}\_\{G\}\\leftarrow\\mathbf\{\\Sigma\}\_\{G\}\+c\\beta\\mathbf\{I\}and𝚺X←𝚺X\+c​β​𝐈\\mathbf\{\\Sigma\}\_\{X\}\\leftarrow\\mathbf\{\\Sigma\}\_\{X\}\+c\\beta\\mathbf\{I\}, a well\-known technique in second\-order and Fisher\-based optimization methods[Martens and Grosse \(2015\)](https://arxiv.org/html/2609.02639#bib.bib10);[Ledoit and Wolf \(2012\)](https://arxiv.org/html/2609.02639#bib.bib59)\. Following[Yang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib5), we setβ\\betato the mean singular value of each covariance matrix and usec=10−2c=10^\{\-2\}\(see Appendix[D](https://arxiv.org/html/2609.02639#A4)for further analysis\)\.

After computing the decomposition, we truncate it to rankrrand construct the LoRA factors\. The trainable matrices𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}are initialized from the top\-rrsingular components, scaled by the inverse covariance matrices to align them with the curvature\-weighted subspace\. We then update the frozen residual weight so that the model initially preserves the pretrained function\. Applying this procedure independently to each linear layer yields a training\-friendly LoRA initialization\.

Table 1:Comparison of LoRA, PiSSA, CorDA, LoRA\-One, and TaRA on natural language generation tasks\. Each result is the mean over three seeds, and the standard deviation is shown in small gray text\.Redcolor indicates the best PEFT result within each model block, excluding Full FT\.

## 4Experiments

We evaluate the proposed method through extensive experiments across multiple LLMs and a diverse set of downstream tasks\. We compare TaRA against full fine\-tuning and several LoRA initialization baselines, including LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib2), PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib4), CorDA[Yang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib5), and LoRA\-One[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib6)\. We also include the state\-of\-the\-art LoRA variants, MiSS[Kang and Yin \(\)](https://arxiv.org/html/2609.02639#bib.bib64)and LoRAM[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.02639#bib.bib65)for comparison\. Our evaluation covers two task categories: natural language generation \(NLG\) and natural language understanding \(NLU\), and systematically examines performance across different model and LoRA rank budgets\.

### 4\.1Experimental Setup

#### Natural Language Generation Tasks\.

For NLG, we consider math problem solving and code generation\. We fine\-tune LLaMA\-2\-7B[Touvron et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib13)on 100K samples from MetaMathQA[Yu et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib14)for math and 100K samples from CodeFeedback\-Filtered\-Instruction[Zheng et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib18)for code\. Models are trained with LoRA ranks of 128, 64 and 32 to assess robustness across rank budgets \(LoRAα\\alphais set equal to the rank\)\.

For evaluation, math performance is measured on GSM8K\-D[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib15)and MATH[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib17)using direct prompting, as well as GSM8K\-COT[Wei et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib16)under an 8\-shot chain\-of\-thought setting with the Language Model Evaluation Harness[Biderman et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib21)\. Code generation is evaluated on HumanEval[Chen \(2021\)](https://arxiv.org/html/2609.02639#bib.bib19)and MBPP[Austin et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib20)using the BigCode Evaluation Harness[Ben Allal et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib22)\. All evaluation harnesses are used with their default configurations\.

\(a\)r=128r=128\(b\)r=32r=32
Figure 3:Training loss over steps on LLaMA\-2\-7B fine\-tuned with MetaMathQA\. We compare LoRA, PiSSA, CorDA, LoRA\-One, and TaRA at \(a\)r=128r=128and \(b\)r=32r=32; insets show the first 30 steps\.Figure 4:GSM8K\-D accuracy over training steps when fine\-tuning LLaMA\-2\-7B on the MetaMathQA dataset\.
#### Natural Language Understanding Tasks\.

For NLU, we focus on commonsense reasoning\. Models are fine\-tuned on the Commonsense\-170K dataset[Hu et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib23)\. To evaluate scalability and generality, we conduct experiments on three base models—DeepSeek\-R1\-Distill\-Qwen\-1\.5B[Guo et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib31), LLaMA\-2\-7b[Touvron et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib13), LLaMA\-3\.1\-8B[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib32), and Qwen\-3\-8B[Yang et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib33)—with a fixed LoRA rank of 128 \(LoRAα\\alphais set equal to the rank\)\.

We evaluate performance on eight benchmarks: BoolQ[Clark et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib24), PIQA[Bisk et al\. \(2020\)](https://arxiv.org/html/2609.02639#bib.bib25), SIQA[Sap et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib26), HellaSwag[Zellers et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib27), WinoGrande[Sakaguchi et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib28), ARC\-Challenge, ARC\-Easy[Clark et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib29), and OBQA[Mihaylov et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib30)\.

For full fine\-tuning, LoRA, PiSSA, CorDA and TaRA, we follow the hyperparameter settings of CorDA[Yang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib5)and select the best learning rate\. For LoRA\-One, which is sensitive to learning\-rate choice, we use the optimal value reported in its original paper\. For all methods that require calibration \(CorDA, LoRA\-One, and TaRA\), we set the calibration set size to 256 and conducted all experiments under this setting\. All results are obtained from the final training checkpoint and reported in terms of task accuracy\. Additional details are provided in the Appendix\. All experiments are conducted on A100 GPUs \(80GB\)\.

### 4\.2Natural Language Generation Task

Table[1](https://arxiv.org/html/2609.02639#S3.T1)compares performance on natural language generation benchmarks across different LoRA ranks \(r∈128,64,32r\\in\{128,64,32\}\)\. Overall, TaRA achieves the highest average performance at all ranks, consistently outperforming prior initialization methods such as PiSSA, CorDA, and LoRA\-One\. Atr=128r=128, TaRA attains the best results on GSM8K\-D/GSM8K\-COT, MATH, and MBPP, and also yields the highest average score\. Atr=64r=64, TaRA remains the top\-performing method on GSM8K\-D/GSM8K\-COT, MATH, and MBPP, leading to the best average performance as well\. Even atr=32r=32, TaRA exhibits a relatively small performance drop compared to other baselines, achieving the best performance across all tasks and demonstrating the most stable behavior across ranks\. These results highlight the strength of the proposed initialization in efficiently capturing training\-relevant directions under a constrained parameter budget\.

Notably, LoRA\-One also underperforms PiSSA and CorDA atr=128r=128, suggesting that directly applying SVD to gradients is ineffective at higher ranks\. Overall, these results indicate that neither simple SVD\-based nor random initialization is sufficient to capture training\-relevant directions under tight constraints, whereas TaRA does so effectively\.

Figures[3\(a\)](https://arxiv.org/html/2609.02639#S4.F3.sf1)and[3\(b\)](https://arxiv.org/html/2609.02639#S4.F3.sf2)show the training loss curves on MetaMathQA forr=128r=128andr=32r=32\. In both settings, TaRA converges more smoothly and reaches a lower loss than prior methods\. This suggests that the proposed initialization provides a more effective update subspace at the early stage of training, enabling more efficient optimization, and that its benefit persists into later phases of training[Gur\-Ari et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib8)\. This advantage persists throughout fine\-tuning, as also reflected in GSM8K\-D accuracy over training steps \(Figure[4](https://arxiv.org/html/2609.02639#S4.F4)\)\. While PiSSA and CorDA exhibit limited improvement and LoRA\-One improves only in later stages, TaRA achieves rapid early gains and consistently attains the best performance at both intermediate and final stages\.

Table 2:Comparison with LoRA variants on GSM8K\-D and GSM8K\-CoT\. Baseline results report the mean and variance over three seeds\.Redcolor indicates the best result in each task\.Table 3:Comparison of Full FT, LoRA, PiSSA, CorDA, LoRA\-One, and TaRA on commonsense reasoning tasks\. Results are averaged over three seeds, with standard deviations shown in gray text\.RedandOrangecolor denote the best and second\-best PEFT results in each model, excluding Full FT\. Abbr\. HS = HellaSwag, WG = WinoGrande\.#### Comparison with Non\-Initialization PEFT variants

To compare TaRA with recent PEFT variants beyond LoRA initialization methods, we evaluated MiSS and LoRAM on mathematical reasoning tasks\. For both MiSS and LoRAM, we used the same learning rate as TaRA, 4e\-5\. To approximately match the number of trainable parameters, we set the rank to 256 for MiSS and 128 for LoRAM\. Table[2](https://arxiv.org/html/2609.02639#S4.T2)presents the results\. Overall, TaRA demonstrates that, even when using the standard LoRA parameterization, a well\-designed initialization can outperform recent PEFT variants on mathematical reasoning tasks\.

### 4\.3Natural Language Understanding Task

Table[3](https://arxiv.org/html/2609.02639#S4.T3)compares performance on standard commonsense reasoning benchmarks across three models: DeepSeek\-R1\-Distill\-Qwen\-1\.5B, LLaMA\-2\-7B LLaMA\-3\.1\-8B, and Qwen\-3\-8B\. Overall, TaRA achieves the best average performance on DeepSeek\-R1\-Distill\-Qwen\-1\.5B, LLaMA\-2\-7B and LLaMA\-3\.1\-8B, outperforming PiSSA, CorDA, and LoRA\-One\. On Qwen\-3\-8B, TaRA slightly underperforms LoRA\-One in terms of average score but remains competitive, consistently outperforming LoRA, PiSSA, and CorDA\. While fine\-tuning performance is often distribution\-dependent and no method consistently dominates across all settings, a well\-known phenomenon also observed in prior work, we evaluate methods based on their generality across diverse scenarios\. Under this criterion, TaRA consistently performs well, achieving top\-1 performance in 15 out of 32 cases and second\-best in 11 more\. In contrast, LoRA\-One and CorDA achieve top\-1 in only 8 and 5 cases, respectively, highlighting the robustness of TaRA beyond specific models or tasks\.

### 4\.4Gradient Alignment Analysis

Figure 5:Gradient similarity, measured as the cosine similarity between the n\-step \(5, 10, 50, 100, 200 and 300\) gradient of the combined LoRA adapter and that of the full weight, correlates with downstream accuracy\.To verify whether the proposed method can effectively extract principal components that are*important from a training perspective*, we use the one\-step gradient computed from the full\-rank weight𝐖0\\mathbf\{W\}\_\{0\}as a reference\. We apply PiSSA, CorDA, and TaRA to obtain rank\-rrapproximations, restore them to the original parameter shape, and compute the corresponding one\-step gradients\. We then measure the cosine similarity between the gradient induced by each low\-rank parameterization and the gradient of the corresponding full\-rank weight to quantify their local gradient alignment\.

We conduct this experiment on RoBERTa\-base[Liu et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib34)trained for one step on the CoLA task[Warstadt et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib36)in the GLUE benchmark[Wang et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib35)\. Cosine similarities are computed for all linear layers over 100 random samples and averaged across layers\.

As shown in Figure[1](https://arxiv.org/html/2609.02639#S1.F1), TaRA achieves substantially higher alignment with the full\-rank gradient than PiSSA and CorDA across all ranks\. PiSSA shows little improvement as the rank increases, while CorDA exhibits only modest gains due to its task\-aware initialization\. In contrast, TaRA demonstrates a sharp increase in cosine similarity with rank and consistently attains the highest alignment, resulting in better fine\-tuning quality\. These results indicate that TaRA more effectively preserves training\-relevant principal components that are critical for optimization\.

Additionally, we further analyzed the effect of initialization beyond the very early stage of training\. Figure[5](https://arxiv.org/html/2609.02639#S4.F5)presents the gradient alignment measured after n training steps \(5, 10, 50, 100, 200, and 300\) under the same setup as the one\-step analysis above\. We observe that TaRA maintains substantially higher alignment than the other methods even after multiple training steps\. We also found an interesting trend in this experiment: although the gradient similarity decreases from its initial level as training progresses, after a certain point it no longer drops and instead maintains a consistent level of similarity to Full FT222We provide an intuitive explanation of this phenomenon in Appendix[F](https://arxiv.org/html/2609.02639#A6)\.\. These results imply that, if the initialization induces gradients similar to those of Full FT at the beginning of training, its effect may persist throughout the training process\.

Table 4:Initialization and training time comparison\.\(a\)init precision ablation\(b\)calib size ablation
Figure 6:\(a\) Ablation of the numeric precision used to collect covariance statistics for TaRA initialization \(FP8/FP16/FP32\), reporting GSM8K\-D accuracy across ranksr∈16,32,64,128r\\in\{16,32,64,128\}\. \(b\) Ablation of the calibration set size \(32, 64, 128, 256\), reporting math reasoning tasks accuracy\.
### 4\.5Initialization Overhead Analysis

Table[4](https://arxiv.org/html/2609.02639#S4.T4)reports the initialization time, including covariance collection and SVD computation\. For LLaMA\-2\-7B on MetaMathQA and CodeFeedback, initialization accounts for only 4\-5% of the total fine\-tuning time, demonstrating that its computational overhead is affordable in practice\.

Since TaRA requires collecting both activation and gradient covariances, its initialization can incur nontrivial memory overhead\. To reduce this cost, we explore lower\-precision accumulation of covariance statistics\. Figure[6\(a\)](https://arxiv.org/html/2609.02639#S4.F6.sf1)shows the results in which covariances are collected in FP8, FP16, or FP32, followed by initialization and evaluation on LLaMA\-2\-7B for GSM8K\-D across LoRA ranks\. Lowering the precision results in only minor accuracy variations, with slight gains or losses depending on the rank\. These results indicate that low\-precision covariance collection is an effective alternative to reduce memory usage during initialization\.

Additionally, we analyzed the performance variation with respect to the calibration set size\. Figure[6\(b\)](https://arxiv.org/html/2609.02639#S4.F6.sf2)shows the performance trends across different calibration set sizes \(32,64,128, and 256\)\. The results show that TaRA achieves stable performance even with a small calibration set, with only minor variation in performance\. This observation suggests that TaRA remains effective even when only a small calibration set is available\.

## 5Related Work

A wide range of PEFT methods have been proposed to reduce the cost of adapting large language models[Ding et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib37);[Xu et al\. \(2023a\)](https://arxiv.org/html/2609.02639#bib.bib38)\. These methods span several paradigms, including adapter\-based approaches[Houlsby et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib1);[Lei et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib39);[He et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib40);[Rücklé et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib41);[Zhao et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib42);[Pfeiffer et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib43), prompt\-based methods[Li and Liang \(2021\)](https://arxiv.org/html/2609.02639#bib.bib3);[Hambardzumyan et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib44);[Lester et al\. \(2021\)](https://arxiv.org/html/2609.02639#bib.bib45);[Vu et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib46);[Asai et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib47), and low\-rank adaptation techniques[Hu et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib2);[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib48);[Valipour et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib49)\. Adapter\- and prompt\-based methods introduce additional trainable components and optimize only these parameters, but they modify the model architecture and often incur inference overhead\.

LoRA[Hu et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib2)addresses this limitation by freezing pretrained weights and learning a low\-rank update parameterized by matrices𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}, which project activations into and out of a bottleneck space\. LoRA is simple, computationally efficient, and allows the learned update to be merged into the base weights after training, enabling inference without additional overhead\.

Building on LoRA, many extensions have been developed to further improve efficiency and performance\. These efforts primarily focus on allocating update capacity across layers and enabling training under resource constraints, particularly memory limitations\. Representative approaches include adaptive rank allocation[Zhang et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib48);[Valipour et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib49), redesigning low\-rank update structures[Liu et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib50);[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib51);[Qiu et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib52), and integrating LoRA with pruning[Zhang et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib53)or quantization[Dettmers et al\. \(2023\)](https://arxiv.org/html/2609.02639#bib.bib54);[Xu et al\. \(2023b\)](https://arxiv.org/html/2609.02639#bib.bib55)\. Together, these methods mitigate the accuracy degradation caused by rank constraints while preserving LoRA’s core benefit\.

Notably, these advances are largely orthogonal to the choice of LoRA initialization\. As a result, they are complementary to the method proposed in this work, and combining them may yield further performance gains\.

## 6Conclusion

In this paper, we propose training\-aware low\-rank adaptation initialization \(TaRA\), a new PEFT approach that explicitly prioritizes training\-relevant directions\. TaRA performs a covariance\-aware SVD that jointly incorporates activation and gradient statistics to identify directions that are most influential during optimization\. Using the resulting low\-rank approximation, we show that the induced one\-step gradient closely matches that of full fine\-tuning, indicating effective preservation of training\-relevant information\. When applied to LoRA initialization, TaRA consistently outperforms existing methods and achieves state\-of\-the\-art performance across a wide range of benchmarks\.

## 7Limitations

First, TaRA is tailored to the calibration and training distribution, so its gains can be less stable under distribution shift\. HumanEval and MBPP are out\-of\-distribution relative to the CodeFeedback training data, and their performance varies non\-monotonically with rank across the compared methods \(Appendix[E](https://arxiv.org/html/2609.02639#A5)\)\. Although TaRA remains effective on in\-distribution tasks, improving its robustness to unseen distributions is an important direction for future work\.

Second, compared with calibration\-free methods such as LoRA and PiSSA, TaRA requires a task\-specific calibration stage to collect activation and gradient covariance statistics\. This introduces additional data access and one\-time computation and memory costs before fine\-tuning\. Our overhead and calibration size analyses show that these costs are manageable in practice, but reducing this calibration dependency would further improve the applicability of TaRA\.

## Acknowledgements

This work was supported by IITP and NRF grant funded by the Korea government\(MSIT\) \(RS\-2024\-00415602, RS\-2023\-00228970, RS\-2019\-II191906\)\.

## References

- Asaiet al\.\(2022\)A\. Asai, M\. Salehi, M\. E\. Peters, and H\. HajishirziAttempt: parameter\-efficient multi\-task tuning via attentional mixtures of soft prompts\.arXiv preprint arXiv:2205\.11961\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Austinet al\.\(2021\)J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Ben Allalet al\.\(2022\)L\. Ben Allal, N\. Muennighoff, L\. Kumar Umapathi, B\. Lipkin, and L\. von WerraA framework for the evaluation of code generation models\.GitHub\.Note:[https://github\.com/bigcode\-project/bigcode\-evaluation\-harness](https://github.com/bigcode-project/bigcode-evaluation-harness)Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Bidermanet al\.\(2024\)S\. Biderman, H\. Schoelkopf, L\. Sutawika,et al\.Lessons from the trenches on reproducible evaluation of language models\.External Links:2405\.14782Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Buyukakyuz \(2024\)BuyukakyuzOlora: orthonormal low\-rank adaptation of large language models\. arxiv 2024\.arXiv preprint arXiv:2406\.01775\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p2.1)\.
- Chekalinaet al\.\(2025\)V\. Chekalina, D\. Moskovskiy, D\. Cherniuk, M\. Kurkin, A\. Kuznetsov, and E\. FrolovGeneralized fisher\-weighted svd: scalable kronecker\-factored fisher approximation for compressing large language models\.arXiv preprint arXiv:2505\.17974\.Cited by:[Appendix H](https://arxiv.org/html/2609.02639#A8.p1.1.1),[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p4.1)\.
- Chen \(2021\)M\. ChenEvaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolq: exploring the surprising difficulty of natural yes/no questions\.arXiv preprint arXiv:1905\.10044\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Dinget al\.\(2023\)N\. Ding, Y\. Qin, G\. Yang, F\. Wei, Z\. Yang, Y\. Su, S\. Hu, Y\. Chen, C\. Chan, W\. Chen,et al\.Parameter\-efficient fine\-tuning of large\-scale pre\-trained language models\.Nature machine intelligence5\(3\),pp\. 220–235\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Eckart and Young \(1936\)C\. Eckart and G\. YoungThe approximation of one matrix by another of lower rank\.Psychometrika1\(3\),pp\. 211–218\.Cited by:[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p12.1)\.
- Golub and Van Loan \(2013\)G\. H\. Golub and C\. F\. Van LoanMatrix computations\.JHU press\.Cited by:[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p12.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p1.1)\.
- Gur\-Ariet al\.\(2018\)G\. Gur\-Ari, D\. A\. Roberts, and E\. DyerGradient descent happens in a tiny subspace\.arXiv preprint arXiv:1812\.04754\.Cited by:[Appendix F](https://arxiv.org/html/2609.02639#A6.p12.1),[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p9.1),[§4\.2](https://arxiv.org/html/2609.02639#S4.SS2.p3.1)\.
- Hambardzumyanet al\.\(2021\)K\. Hambardzumyan, H\. Khachatrian, and J\. MayWarp: word\-level adversarial reprogramming\.arXiv preprint arXiv:2101\.00121\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuThe impact of initialization on lora finetuning dynamics\.Advances in Neural Information Processing Systems37,pp\. 117015–117040\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p2.1)\.
- Heet al\.\(2021\)J\. He, C\. Zhou, X\. Ma, T\. Berg\-Kirkpatrick, and G\. NeubigTowards a unified view of parameter\-efficient transfer learning\.arXiv preprint arXiv:2110\.04366\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p1.1),[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- \[24\]Y\. Hsu, T\. Hua, S\. Chang, Q\. Lou, Y\. Shen, and H\. JinLanguage model compression with weighted low\-rank factorization, 2022\.URL https://arxiv\. org/abs/2207\.001124\.Cited by:[Appendix H](https://arxiv.org/html/2609.02639#A8.p1.1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p1.1),[§4](https://arxiv.org/html/2609.02639#S4.p1.1),[§5](https://arxiv.org/html/2609.02639#S5.p1.1),[§5](https://arxiv.org/html/2609.02639#S5.p2.1)\.
- Huet al\.\(2023\)Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. LeeLlm\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 5254–5276\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p1.1)\.
- Huaet al\.\(2022\)T\. Hua, Y\. Hsu, F\. Wang, Q\. Lou, Y\. Shen, and H\. JinNumerical optimizations for weighted low\-rank estimation on language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 1404–1416\.Cited by:[Appendix H](https://arxiv.org/html/2609.02639#A8.p1.1.1)\.
- \[28\]J\. Kang and Q\. YinMiss: revisiting the trade\-off in lora with an efficient shard\-sharing structure, 2025\.URL https://arxiv\. org/abs/2409\.15371\.Cited by:[§4](https://arxiv.org/html/2609.02639#S4.p1.1)\.
- Ledoit and Wolf \(2003\)O\. Ledoit and M\. WolfHoney, i shrunk the sample covariance matrix\.Cited by:[Appendix E](https://arxiv.org/html/2609.02639#A5.p2.1)\.
- Ledoit and Wolf \(2012\)O\. Ledoit and M\. WolfNonlinear shrinkage estimation of large\-dimensional covariance matrices\.Cited by:[§3\.3](https://arxiv.org/html/2609.02639#S3.SS3.p2.1)\.
- Leiet al\.\(2023\)T\. Lei, J\. Bai, S\. Brahma, J\. Ainslie, K\. Lee, Y\. Zhou, N\. Du, V\. Zhao, Y\. Wu, B\. Li,et al\.Conditional adapters: parameter\-efficient transfer learning with fast inference\.Advances in Neural Information Processing Systems36,pp\. 8152–8172\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.arXiv preprint arXiv:2104\.08691\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Li and Liang \(2021\)X\. L\. Li and P\. LiangPrefix\-tuning: optimizing continuous prompts for generation\.arXiv preprint arXiv:2101\.00190\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p1.1),[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Liuet al\.\(2019\)Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. StoyanovRoberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[Appendix B](https://arxiv.org/html/2609.02639#A2.p2.1),[§4\.4](https://arxiv.org/html/2609.02639#S4.SS4.p2.1)\.
- Martens and Grosse \(2015\)J\. Martens and R\. GrosseOptimizing neural networks with kronecker\-factored approximate curvature\.InInternational conference on machine learning,pp\. 2408–2417\.Cited by:[Appendix A](https://arxiv.org/html/2609.02639#A1.SS0.SSS0.Px3.p1.1),[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p4.1),[§3\.3](https://arxiv.org/html/2609.02639#S3.SS3.p2.1)\.
- Martens \(2020\)J\. MartensNew insights and perspectives on the natural gradient method\.Journal of Machine Learning Research21\(146\),pp\. 1–76\.Cited by:[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p4.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPissa: principal singular values and singular vectors adaptation of large language models\.Advances in Neural Information Processing Systems37,pp\. 121038–121072\.Cited by:[Appendix C](https://arxiv.org/html/2609.02639#A3.p2.1),[§1](https://arxiv.org/html/2609.02639#S1.p2.1),[§1](https://arxiv.org/html/2609.02639#S1.p3.1),[§2](https://arxiv.org/html/2609.02639#S2.p3.1),[§4](https://arxiv.org/html/2609.02639#S4.p1.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.arXiv preprint arXiv:1809\.02789\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Pfeifferet al\.\(2021\)J\. Pfeiffer, A\. Kamath, A\. Rücklé, K\. Cho, and I\. GurevychAdapterfusion: non\-destructive task composition for transfer learning\.InProceedings of the 16th conference of the European chapter of the association for computational linguistics: main volume,pp\. 487–503\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Qiuet al\.\(2023\)Z\. Qiu, W\. Liu, H\. Feng, Y\. Xue, Y\. Feng, Z\. Liu, D\. Zhang, A\. Weller, and B\. SchölkopfControlling text\-to\-image diffusion by orthogonal finetuning\.Advances in Neural Information Processing Systems36,pp\. 79320–79362\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Rückléet al\.\(2021\)A\. Rücklé, G\. Geigle, M\. Glockner, T\. Beck, J\. Pfeiffer, N\. Reimers, and I\. GurevychAdapterdrop: on the efficiency of adapters in transformers\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 7930–7946\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Sagunet al\.\(2017\)L\. Sagun, U\. Evci, V\. U\. Guney, Y\. Dauphin, and L\. BottouEmpirical analysis of the hessian of over\-parametrized neural networks\.arXiv preprint arXiv:1706\.04454\.Cited by:[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p9.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Sapet al\.\(2019\)M\. Sap, H\. Rashkin, D\. Chen, R\. LeBras, and Y\. ChoiSocialiqa: commonsense reasoning about social interactions\.arXiv preprint arXiv:1904\.09728\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p1.1)\.
- Valipouret al\.\(2023\)M\. Valipour, M\. Rezagholizadeh, I\. Kobyzev, and A\. GhodsiDyLoRA: parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 3274–3287\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1),[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Vuet al\.\(2022\)T\. Vu, B\. Lester, N\. Constant, R\. Al\-Rfou, and D\. CerSpot: better frozen model adaptation through soft prompt transfer\.InProceedings of the 60th annual meeting of the association for computational linguistics,pp\. 5039–5059\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Wanget al\.\(2018\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[Appendix B](https://arxiv.org/html/2609.02639#A2.p2.1),[§4\.4](https://arxiv.org/html/2609.02639#S4.SS4.p2.1)\.
- Wanget al\.\(2024\)S\. Wang, L\. Yu, and J\. LiLora\-ga: low\-rank adaptation with gradient approximation\.Advances in Neural Information Processing Systems37,pp\. 54905–54931\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p3.1),[§2](https://arxiv.org/html/2609.02639#S2.p1.1),[§2](https://arxiv.org/html/2609.02639#S2.p5.1),[§3\.2](https://arxiv.org/html/2609.02639#S3.SS2.p1.1)\.
- Warstadtet al\.\(2019\)A\. Warstadt, A\. Singh, and S\. R\. BowmanNeural network acceptability judgments\.Transactions of the Association for Computational Linguistics7,pp\. 625–641\.Cited by:[§4\.4](https://arxiv.org/html/2609.02639#S4.SS4.p2.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p2.1)\.
- Xuet al\.\(2023a\)L\. Xu, H\. Xie, S\. J\. Qin, X\. Tao, and F\. L\. WangParameter\-efficient fine\-tuning methods for pretrained language models: a critical review and assessment\.arXiv preprint arXiv:2312\.12148\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Xuet al\.\(2023b\)Y\. Xu, L\. Xie, X\. Gu, X\. Chen, H\. Chang, H\. Zhang, Z\. Chen, X\. Zhang, and Q\. TianQa\-lora: quantization\-aware low\-rank adaptation of large language models\.arXiv preprint arXiv:2309\.14717\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p1.1)\.
- Yanget al\.\(2024\)Y\. Yang, X\. Li, Z\. Zhou, S\. Song, J\. Wu, L\. Nie, and B\. GhanemCorda: context\-oriented decomposition adaptation of large language models for task\-aware parameter\-efficient fine\-tuning\.Advances in Neural Information Processing Systems37,pp\. 71768–71791\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p3.1),[§2](https://arxiv.org/html/2609.02639#S2.p4.1),[§3\.3](https://arxiv.org/html/2609.02639#S3.SS3.p2.1),[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p3.1),[§4](https://arxiv.org/html/2609.02639#S4.p1.1)\.
- Yankunet al\.\(2025\)H\. Yankun, L\. Xing, Z\. Hui\-Ling, Y\. Xianzhi, L\. Wulong, and Y\. MingxuanSVDq: 1\.25\-bit and 410x key cache compression for llm attention\.arXiv preprint arXiv:2502\.15304\.Cited by:[§3\.3](https://arxiv.org/html/2609.02639#S3.SS3.p2.1)\.
- Yuet al\.\(2023\)L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetamath: bootstrap your own mathematical questions for large language models\.arXiv preprint arXiv:2309\.12284\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaswag: can a machine really finish your sentence?\.arXiv preprint arXiv:1905\.07830\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px2.p2.1)\.
- Zhanget al\.\(2024\)M\. Zhang, H\. Chen, C\. Shen, Z\. Yang, L\. Ou, X\. Yu, and B\. ZhuangLoRAPrune: structured pruning meets low\-rank parameter\-efficient fine\-tuning\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 3013–3026\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Zhanget al\.\(2023\)Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdalora: adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1),[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, F\. Liu, and Y\. ChenLoRA\-one: one\-step full gradient could suffice for fine\-tuning large language models, provably and efficiently\.arXiv preprint arXiv:2502\.01235\.Cited by:[§1](https://arxiv.org/html/2609.02639#S1.p3.1),[§2](https://arxiv.org/html/2609.02639#S2.p5.1),[§4](https://arxiv.org/html/2609.02639#S4.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, H\. Li, Y\. Zhang, G\. Gong, J\. Wang, J\. Hu, P\. Liu, and Q\. JiangThe primacy of magnitude in low\-rank adaptation\.Advances in Neural Information Processing Systems38,pp\. 39–69\.Cited by:[§4](https://arxiv.org/html/2609.02639#S4.p1.1)\.
- Zhaoet al\.\(2022\)H\. Zhao, H\. Tan, and H\. MeiTiny\-attention adapter: contexts are more important than the number of parameters\.arXiv preprint arXiv:2211\.01979\.Cited by:[§5](https://arxiv.org/html/2609.02639#S5.p1.1)\.
- Zhaoet al\.\(2024\)J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. TianGalore: memory\-efficient llm training by gradient low\-rank projection\.arXiv preprint arXiv:2403\.03507\.Cited by:[§3\.3](https://arxiv.org/html/2609.02639#S3.SS3.p2.1),[§5](https://arxiv.org/html/2609.02639#S5.p3.1)\.
- Zhenget al\.\(2024\)T\. Zheng, G\. Zhang, T\. Shen, X\. Liu, B\. Y\. Lin, J\. Fu, W\. Chen, and X\. YueOpencodeinterpreter: integrating code generation with execution and refinement\.arXiv preprint arXiv:2402\.14658\.Cited by:[§4\.1](https://arxiv.org/html/2609.02639#S4.SS1.SSS0.Px1.p1.1)\.

## Appendix

## Appendix AProof of Equation

#### Second\-order Taylor expansion\.

Letℒ\\mathcal\{L\}be the loss andθ\\thetathe parameter matrix of a model, with reference pointθ0\\theta\_\{0\}\. A second\-order Taylor expansion ofℒ⁡\(θ\)\\mathcal\{L\}\(\\theta\)aroundθ0\\theta\_\{0\}gives

ℒ\(θ\)≈ℒ\(θ0\)\+∇ℒ\(θ0\)⊤\(θ−θ0\)\+12\(θ−θ0\)⊤H\(θ−θ0\),H:=∇2ℒ\(θ0\)\.\\mathcal\{L\}\(\\theta\)\\approx\\mathcal\{L\}\(\\theta\_\{0\}\)\+\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)^\{\\top\}\(\\theta\-\\theta\_\{0\}\)\+\\frac\{1\}\{2\}\(\\theta\-\\theta\_\{0\}\)^\{\\top\}H\(\\theta\-\\theta\_\{0\}\),\\qquad H:=\\nabla^\{2\}\\mathcal\{L\}\(\\theta\_\{0\}\)\.

#### Hessian to Fisher curvature surrogate\.

Instead of explicitly forming the HessianHH, we use the Fisher information matrixℱ\\mathcal\{F\}as a curvature surrogate, and replace the quadratic curvature term byℱ\\mathcal\{F\}:

ℒ\(θ\)≈ℒ\(θ0\)\+∇ℒ\(θ0\)⊤\(θ−θ0\)\+12\(θ−θ0\)⊤ℱ\(θ−θ0\)\.\\mathcal\{L\}\(\\theta\)\\approx\\mathcal\{L\}\(\\theta\_\{0\}\)\+\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)^\{\\top\}\(\\theta\-\\theta\_\{0\}\)\+\\frac\{1\}\{2\}\(\\theta\-\\theta\_\{0\}\)^\{\\top\}\\mathcal\{F\}\(\\theta\-\\theta\_\{0\}\)\.

#### Fisher to K\-FAC approximation\.

Following the K\-FAC approximation\([Martens and Grosse, 2015](https://arxiv.org/html/2609.02639#bib.bib10)\), we factorize the Fisher matrix as

ℱ≈ΣX⊗ΣG,\\mathcal\{F\}\\approx\\Sigma\_\{X\}\\otimes\\Sigma\_\{G\},whereΣX\\Sigma\_\{X\}andΣG\\Sigma\_\{G\}are the activation and gradient covariance matrices, respectively\. Using standard Kronecker/vectorization identities, the quadratic form induced byℱ\\mathcal\{F\}can be written as

\(θ−θ0\)⊤​\(ΣX⊗ΣG\)​\(θ−θ0\)=tr⁡\(\(θ−θ0\)⊤​ΣG​\(θ−θ0\)​ΣX\)\.\(\\theta\-\\theta\_\{0\}\)^\{\\top\}\(\\Sigma\_\{X\}\\otimes\\Sigma\_\{G\}\)\(\\theta\-\\theta\_\{0\}\)=\\mathrm\{tr\}\\\!\\left\(\(\\theta\-\\theta\_\{0\}\)^\{\\top\}\\Sigma\_\{G\}\(\\theta\-\\theta\_\{0\}\)\\Sigma\_\{X\}\\right\)\.

#### Quadratic term\.

We define the corresponding curvature\-induced quadratic term as

P⁡\(θ\):=12​tr​\(ΣG​\(θ−θ0\)​ΣX​\(θ−θ0\)⊤\)\.P\(\\theta\):=\\frac\{1\}\{2\}\\,\\mathrm\{tr\}\\\!\\left\(\\Sigma\_\{G\}\(\\theta\-\\theta\_\{0\}\)\\Sigma\_\{X\}\(\\theta\-\\theta\_\{0\}\)^\{\\top\}\\right\)\.Then the Taylor model can be rewritten as

ℒ\(θ\)≈ℒ\(θ0\)\+∇ℒ\(θ0\)⊤\(θ−θ0\)\+P\(θ\)\.\\mathcal\{L\}\(\\theta\)\\approx\\mathcal\{L\}\(\\theta\_\{0\}\)\+\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)^\{\\top\}\(\\theta\-\\theta\_\{0\}\)\+P\(\\theta\)\.

#### Differential/trace computation of∇P​\(θ\)\\nabla P\(\\theta\)\.

LetE:=θ−θ0E:=\\theta\-\\theta\_\{0\}\. Then

P⁡\(θ\)=12​tr​\(ΣG​E​ΣX​E⊤\)\.P\(\\theta\)=\\frac\{1\}\{2\}\\,\\mathrm\{tr\}\\\!\\left\(\\Sigma\_\{G\}E\\Sigma\_\{X\}E^\{\\top\}\\right\)\.Taking the differential yields

d​P​\(θ\)=12​tr​\(ΣG​d​E​ΣX​E⊤\)\+12​tr​\(ΣG​E​ΣX​d​E⊤\)\.dP\(\\theta\)=\\frac\{1\}\{2\}\\mathrm\{tr\}\\\!\\left\(\\Sigma\_\{G\}\\,dE\\,\\Sigma\_\{X\}\\,E^\{\\top\}\\right\)\+\\frac\{1\}\{2\}\\mathrm\{tr\}\\\!\\left\(\\Sigma\_\{G\}\\,E\\,\\Sigma\_\{X\}\\,dE^\{\\top\}\\right\)\.Using cyclicity of trace andtr⁡\(A​d​E⊤\)=tr⁡\(A⊤​d​E\)\\mathrm\{tr\}\(A\\,dE^\{\\top\}\)=\\mathrm\{tr\}\(A^\{\\top\}dE\), we obtain

d​P​\(θ\)=12​tr​\(ΣX​E⊤​ΣG​d​E\)\+12​tr​\(\(ΣG​E​ΣX\)⊤​d​E\)\.dP\(\\theta\)=\\frac\{1\}\{2\}\\mathrm\{tr\}\\\!\\left\(\\Sigma\_\{X\}E^\{\\top\}\\Sigma\_\{G\}\\,dE\\right\)\+\\frac\{1\}\{2\}\\mathrm\{tr\}\\\!\\left\(\(\\Sigma\_\{G\}E\\Sigma\_\{X\}\)^\{\\top\}dE\\right\)\.SinceΣX\\Sigma\_\{X\}andΣG\\Sigma\_\{G\}are covariance matrices, we use symmetric estimates so thatΣX⊤=ΣX\\Sigma\_\{X\}^\{\\top\}=\\Sigma\_\{X\}andΣG⊤=ΣG\\Sigma\_\{G\}^\{\\top\}=\\Sigma\_\{G\}, which makes the two terms equal and yields

d​P​\(θ\)=tr⁡\(\(ΣG​E​ΣX\)⊤​d​E\)\.dP\(\\theta\)=\\mathrm\{tr\}\\\!\\left\(\(\\Sigma\_\{G\}E\\Sigma\_\{X\}\)^\{\\top\}dE\\right\)\.

#### Identifying the gradient and the final relation\.

By the defining relation for a scalar function,

d​P​\(θ\)=tr⁡\(\(∇EP​\(θ\)\)⊤​d​E\),dP\(\\theta\)=\\mathrm\{tr\}\\\!\\left\(\(\\nabla\_\{E\}P\(\\theta\)\)^\{\\top\}dE\\right\),we identify

∇EP​\(θ\)=ΣG​E​ΣX\.\\nabla\_\{E\}P\(\\theta\)=\\Sigma\_\{G\}E\\Sigma\_\{X\}\.Finally, sinceE=θ−θ0E=\\theta\-\\theta\_\{0\}impliesd​E=d​θdE=d\\theta, we conclude

∇P​\(θ\)=ΣG​\(θ−θ0\)​ΣX\.\\nabla P\(\\theta\)=\\Sigma\_\{G\}\(\\theta\-\\theta\_\{0\}\)\\Sigma\_\{X\}\.Taking the gradient of the Taylor modelℒ\(θ\)≈ℒ\(θ0\)\+∇ℒ\(θ0\)⊤\(θ−θ0\)\+P\(θ\)\\mathcal\{L\}\(\\theta\)\\approx\\mathcal\{L\}\(\\theta\_\{0\}\)\+\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)^\{\\top\}\(\\theta\-\\theta\_\{0\}\)\+P\(\\theta\)gives

∇ℒ​\(θ\)≈∇ℒ​\(θ0\)\+∇P​\(θ\),\\nabla\\mathcal\{L\}\(\\theta\)\\approx\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)\+\\nabla P\(\\theta\),and therefore

∇ℒ​\(θ\)−∇ℒ​\(θ0\)≈ΣG​\(θ−θ0\)​ΣX\.\\nabla\\mathcal\{L\}\(\\theta\)\-\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)\\approx\\Sigma\_\{G\}\(\\theta\-\\theta\_\{0\}\)\\Sigma\_\{X\}\.
Figure 7:Loss landscape

## Appendix BGLUE Benchmark Evaluation

This section evaluates TaRA on encoder\-only language models using the GLUE benchmark\.

Table 5:Performance on the GLUE benchmark, averaged over five seeds\.Redcolor indicates the best PEFT result in each column, excluding Full FT\.We fine\-tune RoBERTa\-base[Liu et al\. \(2019\)](https://arxiv.org/html/2609.02639#bib.bib34)on five GLUE tasks[Wang et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib35)—CoLA, SST\-2, MRPC, STS\-B, and QNLI—and compare Full Fine\-Tuning, LoRA, PiSSA, CorDA, and TaRA\. Table[5](https://arxiv.org/html/2609.02639#A2.T5)reports the mean performance over five random seeds\. TaRA achieves the best results on CoLA, SST\-2, STS\-B, and QNLI, and also yields the strongest average performance across the five tasks\. These results indicate that TaRA remains effective for encoder\-only language models, not only decoder\-centric LLMs\.

For all PEFT methods, we use LoRA withr=α=128r=\\alpha=128, and train for 3 epochs with batch size 32\.

## Appendix CLoss\-Landscape Analysis

Figure[7](https://arxiv.org/html/2609.02639#A1.F7)visualizes the gradient trajectories of Full Fine\-Tuning, PiSSA, CorDA, and TaRA in a toy setting\. Full Fine\-Tuning and PiSSA follow a noticeably curved trajectory, taking a relatively indirect route before converging toward a local minimum\. CorDA exhibits a less circuitous early trajectory, suggesting improved initial alignment\. In contrast, TaRA is initialized with a direction that heads more directly toward the local minimum and ultimately converges to a lower objective value than PiSSA and CorDA\. This behavior is consistent with our hypothesis: by better preserving training\-relevant components, TaRA facilitates faster and more effective optimization\.

This toy experiment follows the setup of PiSSA[Meng et al\. \(2024\)](https://arxiv.org/html/2609.02639#bib.bib4)\. We pre\-train a three\-linear\-layer network on 10,000 MNIST samples from the odd\-number classes, and then fine\-tune on 2,000 samples from the even\-number classes\. We use LoRA with rank and scaling set tor=α=16r=\\alpha=16, and adopt a learning rate of5×10−45\\times 10^\{\-4\}\.

Figure 8:Damping coefficient ablation
## Appendix DDiagonal Damping Analysis

Figure[8](https://arxiv.org/html/2609.02639#A3.F8)shows the results of our CoLA\-task analysis on the effect of the diagonal damping coefficientcc, including performance stabilization due to damping \(left y\-axis\) and the change in the resulting SVDUUandVVspaces compared with the no\-damping case \(right y\-axis\)\. The results show that whenccis smaller than10−210^\{\-2\}, includingc=0c=0\(= no damping\), task accuracy collapses to 0\.0, indicating a failure to stabilize training and complete training breakdown\. In contrast, whenc≥10−2c\\geq 10^\{\-2\}, the performance becomes stable and training proceeds normally\. Since10−210^\{\-2\}stabilizes the second\-order information while preserving the original space in the no\-damping case as much as possible, we choose10−210^\{\-2\}as an appropriate damping coefficient\.

## Appendix EMitigating Accuracy Fluctuations on Out\-of\-Distribution Tasks

Table 6:Effect of Ledoit–Wolf \(LW\) shrinkage on code generation \(OOD tasks\) performance\. LW results useλ=0\.3\\lambda=0\.3\. Each result is the mean over three seeds, and the standard deviation is shown in small gray text\.Redcolor indicates the best result within each rank block\.As shown in Table[1](https://arxiv.org/html/2609.02639#S3.T1), the accuracies on HumanEval and MBPP do not exhibit a consistent trend as the LoRA rank increases\. Unlike the mathematical reasoning tasks, these code generation benchmarks represent an out\-of\-distribution \(OOD\) setting, where the distributions of the training and evaluation datasets differ\. Since TaRA estimates activation and gradient statistics from the training data, its initialization is explicitly tailored to the training distribution\. While this can facilitate effective adaptation on the training data, the resulting initialization may not necessarily be optimal under a distribution shift at evaluation time\.

Ledoit–Wolf \(LW\) shrinkage[Ledoit and Wolf \(2003\)](https://arxiv.org/html/2609.02639#bib.bib66)has been widely used to stabilize covariance estimation when only a limited number of samples are available\. In our setting, we apply shrinkage in the following form:

Σ~=\(1−λ\)​Σ^calib\+λ​I,\\widetilde\{\\Sigma\}=\(1\-\\lambda\)\\widehat\{\\Sigma\}\_\{\\mathrm\{calib\}\}\+\\lambda I,whereIIdenotes the identity matrix\. The same shrinkage can be directly incorporated into the covariance estimation procedure of TaRA\. The purpose of this experiment is to reduce the extent to which TaRA’s training\-aware initialization is biased toward the calibration distribution\. Asλ\\lambdaincreases, the estimated covariance becomes progressively less dependent on the calibration data\. In the limit asλ\\lambdaapproaches 1, the covariance becomes data\-agnostic, and the resulting initialization coincides with PiSSA under our formulation\.

Table[6](https://arxiv.org/html/2609.02639#A5.T6)reports the code generation results obtained by estimating the covariance matrices with LW shrinkage, initializing the LoRA adapters using TaRA, and subsequently training them under the same setting\. Compared with the results without shrinkage, incorporating LW shrinkage yields substantially more consistent and nearly monotonic performance across different ranks\. These results suggest that regularizing the training\-derived covariance statistics can improve the robustness of TaRA under distribution shift, particularly for OOD code generation tasks\.

## Appendix FWhy One\-Step Gradient Alignment Persists Beyond Initialization

Although TaRA is derived from a one\-step gradient matching objective, Figure[5](https://arxiv.org/html/2609.02639#S4.F5)shows that its gradient alignment with Full FT remains substantially higher than those of the baselines even after multiple optimization steps\. As a possible intuition, we consider an idealized setting in which Full FT and TaRA follow the same local training dynamics\. We use the local quadratic approximation adopted in Section[3\.2](https://arxiv.org/html/2609.02639#S3.SS2)\.

Let

gt:=∇L​\(θt\),g\_\{t\}:=\\nabla L\(\\theta\_\{t\}\),
and consider gradient descent with learning rateη\\eta:

θt\+1=θt−η​gt\.\\theta\_\{t\+1\}=\\theta\_\{t\}\-\\eta g\_\{t\}\.
Under the local quadratic approximation aroundθ0\\theta\_\{0\}in Eq\.[6](https://arxiv.org/html/2609.02639#S3.E6), the gradient can be approximated as

∇L​\(θ\)≈∇L​\(θ0\)\+H⁡\(θ−θ0\),\\nabla L\(\\theta\)\\approx\\nabla L\(\\theta\_\{0\}\)\+H\(\\theta\-\\theta\_\{0\}\),
whereH:=∇2L​\(θ0\)H:=\\nabla^\{2\}L\(\\theta\_\{0\}\)denotes the local Hessian\. Therefore,

gt\+1\\displaystyle g\_\{t\+1\}=∇L​\(θt\+1\)\\displaystyle=\\nabla L\(\\theta\_\{t\+1\}\)≈∇L​\(θ0\)\+H⁡\(θt\+1−θ0\)\\displaystyle\\approx\\nabla L\(\\theta\_\{0\}\)\+H\(\\theta\_\{t\+1\}\-\\theta\_\{0\}\)=∇L​\(θ0\)\+H⁡\(θt−θ0\)−η​H​gt\\displaystyle=\\nabla L\(\\theta\_\{0\}\)\+H\(\\theta\_\{t\}\-\\theta\_\{0\}\)\-\\eta Hg\_\{t\}≈gt−η​H​gt\\displaystyle\\approx g\_\{t\}\-\\eta Hg\_\{t\}=\(I−η​H\)​gt,\\displaystyle=\(I\-\\eta H\)g\_\{t\},
whereIIdenotes the identity matrix\. Repeatedly applying this relation gives

gt≈\(I−η​H\)t​g0\.g\_\{t\}\\approx\(I\-\\eta H\)^\{t\}g\_\{0\}\.
This relation provides a simple interpretation of the empirical behavior in Figure[5](https://arxiv.org/html/2609.02639#S4.F5)\. Under a locally stable curvature, later gradients are not generated along arbitrary, unrelated directions; instead, they are obtained by repeatedly applying a polynomial in the local Hessian to the initial gradient\. This is consistent with prior observations that training gradients tend to concentrate in a small subspace associated with dominant Hessian directions, and that this subspace remains relatively stable during training[Gur\-Ari et al\. \(2018\)](https://arxiv.org/html/2609.02639#bib.bib8)\.

Consequently, if an initialization captures training\-relevant curvature directions at the beginning of optimization, these directions can remain relevant over subsequent steps\. Since TaRA explicitly constructs its initialization using curvature\-aware activation and gradient statistics, its high one\-step gradient alignment can therefore provide a useful initialization bias beyond the first optimization step\. This local analysis is intended as an intuitive explanation rather than a formal guarantee for the entire nonlinear training trajectory\.

## Appendix GCalibration Set Size and LoRA alpha Ablation

Table 7:Ablation on calibration set size\. Each result is the mean over three seeds, and the standard deviation is shown in small gray text\.Redcolor indicates the best result in each column\.Table[7](https://arxiv.org/html/2609.02639#A7.T7)shows the performance variation across different calibration set sizes at rank 128\. The results indicate that TaRA remains stable even with small calibration sets, with only minor performance changes\. This suggests that TaRA can still produce meaningful results even when the calibration set must be kept small due to limited available data\.

Table[8](https://arxiv.org/html/2609.02639#A7.T8)shows the performance variation across different LoRAα\\alphavalues at rank 128\. The results show that TaRA achieves the best performance when LoRAα\\alphais set equal to the LoRA rank\.

Table 8:Ablation on LoRAα\\alpha\. Each result is the mean over three seeds, and the standard deviation is shown in small gray text\.Redcolor indicates the best result in each column\.
## Appendix HDifference between Fisher\-based Model Compression and TaRA

Table 9:Ablation on the square\-root scaling term\. Each result is the mean over three seeds, and the standard deviation is shown in small gray text\.Redcolor indicates the best result in each column\.Recent LLM compression methods perform compression using Fisher information[Hsu et al\. \(\)](https://arxiv.org/html/2609.02639#bib.bib60);[Hua et al\. \(2022\)](https://arxiv.org/html/2609.02639#bib.bib61);[Chekalina et al\. \(2025\)](https://arxiv.org/html/2609.02639#bib.bib57)\. From the compression perspective, the following formulation is used:

minθ⁡ℒ⁡\(θ\)−ℒ⁡\(θ0\)≈minθ⁡‖ℱ1/2​\(θ−θ0\)‖F2\.\\min\_\{\\theta\}\\,\\mathcal\{L\}\(\\theta\)\-\\mathcal\{L\}\(\\theta\_\{0\}\)\\approx\\min\_\{\\theta\}\\,\\left\\\|\\mathcal\{F\}^\{1/2\}\(\\theta\-\\theta\_\{0\}\)\\right\\\|\_\{F\}^\{2\}\.
It uses a formulation that findsθ\\thetaby minimizing the change in the loss functionℒ\\mathcal\{L\}\. Specifically, it seeksθ\\thetathat remains as close as possible to the original modelθ0\\theta\_\{0\}, weighted by the square\-root Fisher information matrixℱ1/2\\mathcal\{F\}^\{1/2\}\.

In contrast, TaRA uses a formulation that findsθ\\thetaby minimizing the change in the gradient:

minθ⁡‖∇ℒ​\(θ\)−∇ℒ​\(θ0\)‖≈minθ⁡‖ℱ⁡\(θ−θ0\)‖\.\\min\_\{\\theta\}\\,\\left\\\|\\nabla\\mathcal\{L\}\(\\theta\)\-\\nabla\\mathcal\{L\}\(\\theta\_\{0\}\)\\right\\\|\\approx\\min\_\{\\theta\}\\,\\left\\\|\\mathcal\{F\}\(\\theta\-\\theta\_\{0\}\)\\right\\\|\.
It uses a formulation that findsθ\\thetaby keeping it as close as possible to the original modelθ0\\theta\_\{0\}, weighted by the Fisher information matrixℱ\\mathcal\{F\}without taking its square root\.

Table[9](https://arxiv.org/html/2609.02639#A8.T9)compares these two formulations for LoRA initialization on the MATH task at rank 128\. The results show that applying LoRA initialization using the square\-root formulation from the compression perspective leads to suboptimal performance\. This supports the significance of the theoretical analysis of TaRA from the training perspective\.

## Appendix IHyperparameters

SectionHyperparameterValueLoRARankrr\(= Scalingα\\alpha\)32, 64, 128Dropout0\.0TrainingEpochs1Learning rate \(*except*LoRA\-One\)4×10−54\\times 10^\{\-5\}Learning rate \(for LoRA\-One\)2×10−42\\times 10^\{\-4\}Batch size8Gradient accumulation steps16Seed0, 1, 2ModelModel namemeta\-llama/Llama\-2\-7b\-hfMax sequence length512DataDataset \(for math\)meta\-math/MetaMathQADataset \(for code\)m\-a\-p/CodeFeedback\-Filtered\-InstructionDataset splittrain\[:100000\]OptimizationOptimizeradamw\_torchWeight decay0\.0Warmup ratio0\.03LR schedulercosineTable 10:Hyperparameters used to fine\-tune models on the natural language generation tasks\.SectionHyperparameterValueLoRARankrr\(= Scalingα\\alpha\)128Dropout0\.0TrainingEpochs1Learning rate \(*except*LoRA\-One\)2×10−52\\times 10^\{\-5\}Learning rate \(for LoRA\-One\)2×10−42\\times 10^\{\-4\}Batch size8Gradient accumulation steps16Seed0, 1, 2ModelModel name 1deepseek\-ai/DeepSeek\-R1\-Distill\-Qwen\-1\.5BModel name 2meta\-llama/Llama\-2\-7b\-hfModel name 3meta\-llama/Llama\-3\.1\-8BModel name 4Qwen/Qwen3\-8BMax sequence length256DataDatasetzwhe99/commonsense\_170kOptimizationOptimizeradamw\_torchWeight decay0\.0Warmup ratio0\.03LR schedulercosineTable 11:Hyperparameters used to fine\-tune models on the natural language understanding tasks\.This section documents the hyperparameters used in Section[4](https://arxiv.org/html/2609.02639#S4)\. Table[10](https://arxiv.org/html/2609.02639#A9.T10)lists the hyperparameters for the natural language generation experiments, while Table[11](https://arxiv.org/html/2609.02639#A9.T11)reports those used for the natural language understanding experiments\.