When Does Subspace Direction Matter for LoRA? Regime Analysis of the Magnitude Principle in Few-Shot Adaptation
Summary
该论文研究 LoRA 适配器初始输入子空间方向的重要性为何依赖于数据 regime,提出基于激活 PCA 的 PivotLoRA 初始化方法,并在少样本 BERT/Llama 分类及 GSM8K、StrategyQA 任务上验证其优于原版 LoRA。
View Cached Full Text
Cached at: 11/19/26, 09:00 AM
# When Does Subspace Direction Matter for LoRA?Regime Analysis of the Magnitude Principle in Few-Shot Adaptation
Source: [https://arxiv.org/html/2611.23428](https://arxiv.org/html/2611.23428)
Cencheng ShenAffiliation:Microsoft ResearchEmail:[c\.shen@microsoft\.com](mailto:)Peng ZhaoAffiliation:Univ\. of DelawareEmail:[pzhao@udel\.edu](mailto:)
###### Abstract
We show that the importance of a LoRA adapter’s initial input\-subspace direction is regime\-dependent, and we characterize this dependence in theory and experiment\. This refines the recent view that spectral initializers help through update magnitude alone\. Direction matters most when labeled data are scarce relative to the representation dimension and the activation subspace used for initialization captures supervised task structure\. In this regime, the adapter’s row span remains sensitive to its starting point, making task\-aligned initialization especially useful\. A spiked\-covariance perturbation analysis characterizes the finite\-sample stability of the estimated subspace\. A first\-step analysis of LoRA training explains why the subspace’s alignment with the task matters and motivates an alignment coefficientρ\\rhothat closely tracks gains across classification tasks\. We test this account with PivotLoRA, which initializes adapters from activation PCA at prediction\-relevant positions and applies one output\-preserving orthogonal pivot early in training\. In few\-shot BERT\-base classification, PivotLoRA achieves the best average rank among 13 methods and improves over vanilla LoRA in 29 of 30 settings, even though it starts from a smaller input\-factor initialization norm\. Additional classification experiments extend the evaluation to 7B and 8B Llama decoders\. Classification gains diminish with more labeled data, consistent with the proposed regime dependence\. A targeted extension to completion\-only reasoning also shows improvements on GSM8K and StrategyQA\. These findings identify conditions under which initialization direction matters, with subspace–task alignment offering a low\-cost diagnostic before training for classification\.
## 1Introduction
Low\-Rank Adaptation\([Hu et al\., 2022](https://arxiv.org/html/2611.23428#bib.bib1)\)efficiently adapts pretrained models by parameterizing weight updates with low\-rank factors\. The input factor determines the feature subspace through which adaptation initially proceeds\. Existing initialization methods derive directions for this subspace from singular components of pretrained weights\([Meng et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib2);[Wang et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib3)\), task gradients\([Wang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib4);[Zhang et al\., 2025a](https://arxiv.org/html/2611.23428#bib.bib5)\), or activation statistics\([Paischer et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib7);[Yang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib8)\)\. Recent work attributes gains from pretrained\-weight spectral initialization primarily to amplified update magnitudes instead of the selected singular directions themselves\([Zhang et al\., 2025b](https://arxiv.org/html/2611.23428#bib.bib9)\)\. This leaves open when initialization direction materially affects adaptation\. We study this question primarily in classification, with a targeted extension to generation, by applying PCA to frozen\-model activations from the adaptation set at positions associated with the supervised prediction\. These include the\[CLS\]position in encoder classification, the mean over input positions in decoder classification, where the label describes the whole sequence, and hidden states that predict supervised response tokens in completion\-only generation\.
Our central claim is that the value of initialization direction is regime\-dependent\. Direction matters most when labeled data are scarce relative to the representation dimension and the estimated subspace captures supervised task structure\. A spiked\-covariance Davis–Kahan analysis shows what limits such an estimate, since its error depends on the effective dimension\-to\-sample ratio and the spectral gap\. The same qualitative pattern appears in synthetic experiments and transformer activations, where the ambient ratiod/Nd/Nprovides a simple proxy for the dimensional regime\. For BERT\-base,d=768d=768andN=16N=16givesd/N=48d/N=48; for 7B and 8B decoders withd=4096d=4096, the high\-dimensional regime persists atN=64N=64\. A larged/Nd/Nis not sufficient, however, because the selected subspace must also contain structure relevant to the supervised objective\.
The role of alignment also appears in the first useful LoRA update\. Standard LoRA initializes the output factor withB=0B=0, so initiallyΔW=0\\Delta W=0and the gradient with respect to the input factorAAis zero\. The first nonzero adapter signal enters through the projected featuresAh\(x\)Ah\(x\)in the update toBB\. Under a class\-conditional model, the squared norm of the initial population gradient depends on the supervised structure captured by the row span ofAA\. We quantify this structure with an alignment coefficientρ\\rho, defined as the fraction of between\-class scatter captured by a candidate subspace\. For BERT\-base with rankr=8r=8, a random subspace captures approximatelyr/d≈1%r/d\\approx 1\\%of this signal in expectation, whereas prediction\-position PCA captures up to84%84\\%on high\-alignment tasks\. Across seven classification tasks,ρ\\rhotracks PivotLoRA’s shared\-protocol gain with Pearsonr=0\.93r=0\.93and exact permutationp=0\.002p=0\.002\. The coefficientρ\\rhothus offers a low\-cost measure of subspace–task alignment for classification\.
Motivated by this analysis, we proposePivotLoRA\. The method initializes the LoRA input factor from prediction\-relevant activation PCA and allows early task gradients to refine its span\. As the factors evolve, however, the input basis can acquire unequal scales and correlations that affect subsequent optimization\. We therefore separate learning the span from normalizing its coordinates\. AfterKKepochs, PivotLoRA performs one output\-preserving orthogonal pivot\. Under a full\-row\-rank condition, the pivot replaces the learned input basis with an orthonormal basis for the same row span and re\-encodes the output factor to preserve the adapter update\. This retains the task\-adapted directions while resettingAA⊤AA^\{\\top\}to the identity, allowing training to continue in normalized coordinates without changing the model function at the pivot\.
Our primary evaluation focuses on few\-shot classification, covering 30 BERT\-base settings and 13 methods, with additional classification experiments on Llama\-2\-7B and Llama\-3\.1\-8B\. PivotLoRA ranks first overall and improves over vanilla LoRA in 29 of 30 primary BERT settings, with the largest shared\-protocol gains concentrated in low\-sample, high\-alignment tasks\. PivotLoRA achieves these gains despite starting from a smaller initialization norm than vanilla LoRA\. Separately, restoring LoRAM’s intended initialization magnitude does not eliminate its performance deficit relative to vanilla LoRA\. In full\-data classification, the gains from initialization direction diminish, consistent with the proposed dependence on labeled sample size\. Beyond classification, we conduct a targeted evaluation of the activation construction for generation\. GPT\-2 summarization shows that a larged/Nd/Nalone is not enough, as PivotLoRA is competitive with LoRA there but not decisively better\. We also evaluate response\-predictor activation initialization in completion\-only adaptation with Llama\-3\.1\-8B\. Under a matched protocol, this extension improves generated\-answer accuracy on GSM8K and StrategyQA by 4\.5 and 3\.6 percentage points, respectively\.
Contributions\.First, we identify a regime in which LoRA initialization direction matters, combining a high dimension\-to\-sample ratio with strong subspace–task alignment\. Second, we support this view through a spiked\-covariance Davis–Kahan analysis, a local first\-step LoRA analysis, and validation on synthetic and transformer activations\. Third, we proposePivotLoRA, which combines prediction\-relevant activation\-subspace initialization with one output\-preserving orthogonal reparameterization\. Fourth, we introduceρ\\rhoas a frozen\-forward\-pass diagnostic for classification and evaluate PivotLoRA primarily through few\-shot classification across encoder and decoder models, complemented by full\-data and summarization controls and a targeted extension to generated\-answer reasoning\.
## 2Related Work
#### LoRA initialization\.
Initializers for LoRA\([Hu et al\., 2022](https://arxiv.org/html/2611.23428#bib.bib1)\)draw the adapter subspace from three sources\. Weight\-based methods use singular components of the pretrained matrix, either the principal ones\([Meng et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib2)\)or the minor ones\([Wang et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib3)\)\. Gradient\-based methods take the SVD of a task gradient\([Wang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib4);[Zhang et al\., 2025a](https://arxiv.org/html/2611.23428#bib.bib5)\)or of a Fisher\-weighted variant\([Zhang et al\., 2026](https://arxiv.org/html/2611.23428#bib.bib6)\)\. Activation\-based methods estimate directions from downstream inputs, either from all\-token covariances\([Paischer et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib7)\)or by modulating the weight decomposition with activation statistics\([Yang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib8)\)\. PivotLoRA belongs to the third family but estimates its subspace at prediction\-relevant positions, and we test when direction matters beyond scale\.
#### Magnitude and optimization accounts\.
[Zhang et al\. \(2025b\)](https://arxiv.org/html/2611.23428#bib.bib9)argue that the gains of weight\-spectral initializers largely reflect amplified update magnitude, and[Lee et al\. \(2026\)](https://arxiv.org/html/2611.23428#bib.bib15)find that vanilla LoRA matches many of its variants once each method’s learning rate is tuned\. Differential factor learning rates\([Hayou et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib10)\)change the optimization without changing the adapter subspace, whereas periodic re\-initialization or resampling\([Yang et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib13);[Hao et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib14)\)replaces directions during training\. The PivotLoRA pivot instead adds no new direction and preserves the adapter update exactly, changing only the coordinates in which optimization continues\. Our analysis complements these accounts by characterizing a regime in which direction matters, with controls that separate direction from learning\-rate allocation, together with initialization\-norm comparisons \(extended review in Appendix[A](https://arxiv.org/html/2611.23428#A1)\)\.
## 3When Does Initialization Direction Matter?
We argue that LoRA initialization direction matters most in a regime that combines a high dimension\-to\-sample ratio with strong alignment between the estimated activation subspace and supervised task structure\. A spiked\-covariance perturbation analysis characterizes the finite\-sample stability of activation PCA, while an alignment coefficientρ\\rhoquantifies the supervised structure captured by the resulting subspace\. The analysis is most direct for classification settings in which predictions are formed from a designated representation, such as the\[CLS\]representation in BERT\.
#### Notation\.
LetNNdenote the total number of labeled examples in the few\-shot training set, sampled approximately class\-balanced acrossCCclasses, so each class has roughlync≈N/Cn\_\{c\}\\approx N/Cexamples\. PivotLoRA estimates its PCA subspace from the global target\-token covariance over allNNexamples\. The between\-class signal used by the alignment diagnostic depends on class means estimated from approximatelyncn\_\{c\}examples each\. SinceCCis fixed within a task, replacingNNbyncn\_\{c\}changes the dimension\-to\-sample ratio only by a task\-dependent constant\. We therefore express the theoretical regime in terms ofdeff/Nd\_\{\\mathrm\{eff\}\}/Nand used/Nd/Nempirically as a simple architecture\-level proxy, not as the literal quantity in the perturbation bound\.
#### Subspace estimation\.
We model target\-token activations at an adapted layer as draws from a spiked covariance distribution
Σ=∑i=1r∗\(λi−σ2\)vivi⊤\+σ2I,\\Sigma=\\sum\_\{i=1\}^\{r^\{\\ast\}\}\(\\lambda\_\{i\}\-\\sigma^\{2\}\)v\_\{i\}v\_\{i\}^\{\\top\}\+\\sigma^\{2\}I,\(1\)whereλi\>σ2\\lambda\_\{i\}\>\\sigma^\{2\}, the vectorsviv\_\{i\}are orthonormal signal directions, andσ2I\\sigma^\{2\}Iis an isotropic noise floor\. This model is idealized, but it approximates the variance concentration observed in transformer activations\. Appendix[B\.1](https://arxiv.org/html/2611.23428#A2.SS1)shows that the top eight components of BERT\-base\[CLS\]activations capture 38–46% of the total variance\.
Under a sub\-Gaussian sampling assumption, combining Davis–Kahan perturbation theory\([Davis and Kahan, 1970](https://arxiv.org/html/2611.23428#bib.bib31);[Yu et al\., 2015](https://arxiv.org/html/2611.23428#bib.bib32)\)with effective\-rank covariance concentration gives the regime\-level scaling
𝔼\[‖sinΘ\(V^,V∗\)‖F\]≲min\{r∗,r∗‖Σ‖opδ\(deffN\+deffN\)\},\\mathbb\{E\}\\left\[\\\|\\sin\\Theta\(\\hat\{V\},V^\{\\ast\}\)\\\|\_\{F\}\\right\]\\lesssim\\min\\left\\\{\\sqrt\{r^\{\\ast\}\},\\sqrt\{r^\{\\ast\}\}\\,\\frac\{\\\|\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\delta\}\\left\(\\sqrt\{\\frac\{d\_\{\\mathrm\{eff\}\}\}\{N\}\}\+\\frac\{d\_\{\\mathrm\{eff\}\}\}\{N\}\\right\)\\right\\\},\(2\)whereδ=λr∗−σ2\\delta=\\lambda\_\{r^\{\\ast\}\}\-\\sigma^\{2\}is the eigengap and
deff=Tr\(Σ\)‖Σ‖opd\_\{\\mathrm\{eff\}\}=\\frac\{\\operatorname\{Tr\}\(\\Sigma\)\}\{\\\|\\Sigma\\\|\_\{\\mathrm\{op\}\}\}is the effective rank\. The quantitydeffd\_\{\\mathrm\{eff\}\}reflects activation anisotropy more faithfully than the ambient dimensiondd\. Appendix[B\.2](https://arxiv.org/html/2611.23428#A2.SS2)gives the derivation\. The bound provides stronger control whenNNis large relative todeffd\_\{\\mathrm\{eff\}\}and the eigengap is large\. When either weakens, the estimated subspace becomes more sensitive to finite\-sample variation\.
#### Empirical validation\.
Figure[1](https://arxiv.org/html/2611.23428#S3.F1)evaluates this prediction using PCA estimation error from both a synthetic spiked model and real BERT\-base activations\. For the real\-activation experiment, we compute PCA on few\-shot\[CLS\]embeddings from AG News and SST\-2 and measure principal angles to a reference subspace estimated from the full training set\. Real activations follow the same qualitative pattern as the synthetic model, with stable estimation at lowd/Nd/Nand rapid degradation neard/N≈10d/N\\approx 10in the evaluated configurations\. The onset of degradation depends on the covariance spectrum and sampling conditions and is not determined byd/Nd/Nalone\.
Figure 1:PCA estimation error vs\.d/Nd/N\.\(a\)Synthetic spiked covariance\.\(b\)BERT\-base\[CLS\]activations on AG News and SST\-2\.\(c\)Overlay\. The shaded region marks the empirical high\-d/Nd/Nregime associated with the largest direction\-aware gains in our experiments\. Error bars show±1\\pm 1std\.
#### Subspace\-task alignment\.
A reliably estimated activation subspace is useful only if it also captures supervised task structure\. LetΣtask\\Sigma\_\{\\mathrm\{task\}\}denote the between\-class scatter of target\-token representations,
Σtask=1N∑c=1Cnc\(μc−μ¯\)\(μc−μ¯\)⊤,\\Sigma\_\{\\mathrm\{task\}\}=\\frac\{1\}\{N\}\\sum\_\{c=1\}^\{C\}n\_\{c\}\(\\mu\_\{c\}\-\\bar\{\\mu\}\)\(\\mu\_\{c\}\-\\bar\{\\mu\}\)^\{\\top\},wherencn\_\{c\}is the number of examples in classcc,μc\\mu\_\{c\}is the class mean, andμ¯\\bar\{\\mu\}is the global mean\.
###### Definition 1\(Subspace\-task alignment\)\.
For a rank\-rrsubspace with orthonormal basisU∈ℝd×rU\\in\\mathbb\{R\}^\{d\\times r\}, define
ρ\(U,Σtask\)=Tr\(U⊤ΣtaskU\)Tr\(Σtask\)∈\[0,1\]\.\\rho\(U,\\Sigma\_\{\\mathrm\{task\}\}\)=\\frac\{\\mathrm\{Tr\}\(U^\{\\top\}\\Sigma\_\{\\mathrm\{task\}\}U\)\}\{\\mathrm\{Tr\}\(\\Sigma\_\{\\mathrm\{task\}\}\)\}\\in\[0,1\]\.\(3\)The valueρ\\rhomeasures the fraction of between\-class signal captured by the subspace\. It equals one when the column span ofUUcontains the range ofΣtask\\Sigma\_\{\\mathrm\{task\}\}\. The metric requires discrete class labels and is not defined for open\-ended generation\.
#### Why direction affects the first LoRA update\.
Consider a LoRA contribution
Δz=sBAh\(x\),s=α/r,\\Delta z=sBAh\(x\),\\qquad s=\\alpha/r,whereh\(x\)∈ℝdh\(x\)\\in\\mathbb\{R\}^\{d\}is the target\-token representation\. LetU∈ℝd×rU\\in\\mathbb\{R\}^\{d\\times r\}have orthonormal columns and initializeA=U⊤A=U^\{\\top\}, so that the adapter observes the projected featuresU⊤h\(x\)U^\{\\top\}h\(x\)\. At standard zero\-BBinitialization,Δz=0\\Delta z=0and∇Aℒ=0\\nabla\_\{A\}\\mathcal\{L\}=0exactly, while∇Bℒ\\nabla\_\{B\}\\mathcal\{L\}can be nonzero\. The first adapter update therefore acts onBB, and its dependence on the initial subspace enters through the projected features\.
Let
GU=∇Bℒ\(0,U\)G\_\{U\}=\\nabla\_\{B\}\\mathcal\{L\}\(0;U\)denote the initial population gradient\. Under the class\-conditional mean, cross\-covariance, centering, and proportionality assumptions stated in Appendix[B\.3](https://arxiv.org/html/2611.23428#A2.SS3), its squared Frobenius norm satisfies
‖GU‖F2=s2γTr\(U⊤ΣtaskU\)\\\|G\_\{U\}\\\|\_\{F\}^\{2\}=s^\{2\}\\gamma\\,\\operatorname\{Tr\}\\left\(U^\{\\top\}\\Sigma\_\{\\mathrm\{task\}\}U\\right\)\(4\)for a task\-dependent constantγ\>0\\gamma\>0\. Thus, the first\-order loss decrease from an update toBBis controlled by the supervised structure captured by the row span ofAA\. Appendix[B\.3](https://arxiv.org/html/2611.23428#A2.SS3)gives the derivation and states the assumptions explicitly\.
IfUUis sampled uniformly from rank\-rrsubspaces independently of the fixed task matrix, then its expected captured fraction of between\-class signal isr/dr/d\. For BERT\-base withr=8r=8, the random\-subspace expectation isr/d≈0\.010r/d\\approx 0\.010, while the most aligned task reachesρ≈0\.84\\rho\\approx 0\.84in our measurements, as reported in Appendix[E\.7](https://arxiv.org/html/2611.23428#A5.SS7)\. Direction\-aware initialization is therefore expected to be useful whenρ\\rhosubstantially exceedsr/dr/dand the relevant activation structure can be estimated from the few\-shot sample\.
The initial span also changes slowly\. The gradient with respect toAAis proportional toBB, which starts at zero, soAAmoves slowly; in our classification runs its norm stays within 5% of its initial value until the pivot \(Appendix[F\.6](https://arxiv.org/html/2611.23428#A6.SS6)\)\.
## 4Method
PivotLoRA operationalizes the two conditions of Section[3](https://arxiv.org/html/2611.23428#S3)by initializing the LoRA input factor from activations at prediction\-relevant positions and performing one output\-preserving orthogonal pivot after early training drift \(Figure[2](https://arxiv.org/html/2611.23428#S4.F2)\)\.
Prediction positions depend on the objective\. For BERT classification, we use the\[CLS\]representation\. Decoder\-scale classification, whose label describes the whole sequence, uses the mean of adapter inputs over input positions\. Completion\-only reasoning uses hidden states that predict supervised response tokens\. For the GPT\-2 summarization control we compare the sequence mean with response positions \(Appendix[E\.4](https://arxiv.org/html/2611.23428#A5.SS4)\)\.
All methods use the standard LoRA parameterization
ΔW=αrBA,\\Delta W=\\frac\{\\alpha\}\{r\}BA,\(5\)whereA∈ℝr×dA\\in\\mathbb\{R\}^\{r\\times d\}stores input directions as rows andB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}maps low\-rank coordinates to the output space\. Unless stated otherwise, we use rankr=8r=8and scalingα=16\\alpha=16\.
Figure 2:Overview of PivotLoRA\. Activations at supervised prediction positions initialize the row span ofAℓA\_\{\\ell\}, withBℓ=0B\_\{\\ell\}=0so thatΔWℓ=0\\Delta W\_\{\\ell\}=0\. For classification, the activation source is the\[CLS\]representation \(BERT\) or the sequence mean \(decoders\); for completion\-only reasoning, it comprises hidden states that predict supervised response tokens\. AfterKKepochs, PivotLoRA replaces the learned input basis with an orthonormal basis for the same row span and re\-encodesBℓB\_\{\\ell\}so thatBℓAℓB\_\{\\ell\}A\_\{\\ell\}is preserved\.### 4\.1Supervised\-Position Subspace Initialization
Activation initializers differ in the positions used to estimate their subspaces\. All\-token pooling provides more activation vectors and can estimate shared structure more reliably, whereas prediction\-position pooling targets the representation directly connected to the supervised objective\. Activation\-source selection is therefore an alignment–estimation trade\-off, and no single source need be optimal\. Our primary controlled classification construction uses prediction\-position PCA\. A matched ablation shows that all\-token and per\-projection sources can score higher \(Appendix[E\.12](https://arxiv.org/html/2611.23428#A5.SS12)\)\.
Let𝒟\\mathcal\{D\}containNNtraining examples\. A frozen forward pass produces one activation vectorxℓ,n∈ℝdx\_\{\\ell,n\}\\in\\mathbb\{R\}^\{d\}for each examplennand adapted layerℓ\\ell\. For classification,xℓ,nx\_\{\\ell,n\}is the adapter input at the\[CLS\]position \(BERT\) or its mean over input positions \(decoders\)\. For completion\-only reasoning, it is the pooled response\-predictor representation defined in Section[4\.2](https://arxiv.org/html/2611.23428#S4.SS2)\. We form
Xℓ=\[xℓ,1⊤xℓ,N⊤\]∈ℝN×d,μℓ=1NXℓ⊤𝟏,X¯ℓ=Xℓ−𝟏μℓ⊤\.X\_\{\\ell\}=\\begin\{bmatrix\}x\_\{\\ell,1\}^\{\\top\}\\\\ \\vdots\\\\ x\_\{\\ell,N\}^\{\\top\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{N\\times d\},\\qquad\\mu\_\{\\ell\}=\\frac\{1\}\{N\}X\_\{\\ell\}^\{\\top\}\\mathbf\{1\},\\qquad\\bar\{X\}\_\{\\ell\}=X\_\{\\ell\}\-\\mathbf\{1\}\\mu\_\{\\ell\}^\{\\top\}\.The empirical covariance is
Σ^ℓ=1NX¯ℓ⊤X¯ℓ\.\\widehat\{\\Sigma\}\_\{\\ell\}=\\frac\{1\}\{N\}\\bar\{X\}\_\{\\ell\}^\{\\top\}\\bar\{X\}\_\{\\ell\}\.\(6\)LetQℓ∈ℝd×rQ\_\{\\ell\}\\in\\mathbb\{R\}^\{d\\times r\}contain the leadingrreigenvectors ofΣ^ℓ\\widehat\{\\Sigma\}\_\{\\ell\}, ordered by decreasing eigenvalue:
Σ^ℓQℓ=QℓΛℓ,Qℓ⊤Qℓ=Ir\.\\widehat\{\\Sigma\}\_\{\\ell\}Q\_\{\\ell\}=Q\_\{\\ell\}\\Lambda\_\{\\ell\},\\qquad Q\_\{\\ell\}^\{\\top\}Q\_\{\\ell\}=I\_\{r\}\.We initialize
Aℓ\(0\)=cRℓ⊤Qℓ⊤,Bℓ\(0\)=𝟎,c=5/d,A\_\{\\ell\}^\{\(0\)\}=c\\,R\_\{\\ell\}^\{\\top\}Q\_\{\\ell\}^\{\\top\},\\qquad B\_\{\\ell\}^\{\(0\)\}=\\mathbf\{0\},\\qquad c=\\sqrt\{5/d\},\(7\)whereRℓ∈ℝr×rR\_\{\\ell\}\\in\\mathbb\{R\}^\{r\\times r\}is a fixed random orthogonal matrix drawn once per adapter\. BecauseRℓR\_\{\\ell\}is orthogonal, the row span ofAℓ\(0\)A\_\{\\ell\}^\{\(0\)\}is exactly the leading PCA subspace;RℓR\_\{\\ell\}only mixes coordinates within it\. The scale gives‖Aℓ\(0\)‖F=5r/d\\\|A\_\{\\ell\}^\{\(0\)\}\\\|\_\{F\}=\\sqrt\{5r/d\}, smaller than the approximatelyr/3\\sqrt\{r/3\}of the Kaiming\-uniform initialization used by vanilla LoRA \(about7×7\\timessmaller ford=768d=768and17×17\\timesford=4096d=4096\)\. These ratios hold for the reasoning code\. In our classification code, vanilla LoRA’sAAis stored as ad×rd\\times rtensor, so Kaiming’s fan\-in isrrand‖A‖F≈d/3\\\|A\\\|\_\{F\}\\approx\\sqrt\{d/3\}, about1616for BERT and3737for Llama; there PivotLoRA starts about70×70\\timesand370×370\\timessmaller\. On BERT, vanilla LoRA with the standard initialization is the weaker baseline, exceeding this version in only 6 of 30 settings, and PivotLoRA exceeds it in 28 of 30 \(Appendix[C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px10)\)\. The initialization is therefore not norm\-matched to vanilla LoRA, and because it is smaller, PivotLoRA’s gains do not come from a larger initialization norm\. SinceBℓ\(0\)=𝟎B\_\{\\ell\}^\{\(0\)\}=\\mathbf\{0\}, initialization leaves the pretrained model function unchanged\. We call this initialization without the pivot TT\-PCA \(target\-token PCA\)\. We prefer PCA to LDA for three reasons\. PCA needs no labels, so the same construction applies to completion\-only reasoning, where there are no classes\. It keepsρ\\rhoan independent measure of how much supervised structure an unsupervised subspace captures\. LDA also yields at mostC−1≤3C\-1\\leq 3directions here, fewer thanr=8r=8\. The TT\-LDA ablation \(Appendix[C\.2](https://arxiv.org/html/2611.23428#A3.SS2)\) has a better average rank than TT\-PCA but uses a different learning\-rate structure \(Appendix[F\.9](https://arxiv.org/html/2611.23428#A6.SS9)\)\.
In the main classification protocol, the PCA eigenvalues, in decreasing order, determine an allocation of the learning rate across therrcolumns ofBB\. Because ofRℓR\_\{\\ell\}, the input direction paired with each column mixes several principal directions\. The averageBB\-to\-AAlearning\-rate ratio isλratio=16\\lambda\_\{\\mathrm\{ratio\}\}=16, matching the LoRA\+ scale of[Hayou et al\. \(2024\)](https://arxiv.org/html/2611.23428#bib.bib10)\. This allocation is an optimization heuristic and is not part of the subspace definition\. Classification\-specific learning\-rate equations appear in Appendix[C\.1](https://arxiv.org/html/2611.23428#A3.SS1)\.
### 4\.2Response\-Position Initialization for Generative Reasoning
For completion\-only fine\-tuning, each sequence contains a prompt followed by a supervised response, with prompt tokens excluded from the language\-modeling loss\. Because hidden statettpredicts tokent\+1t\+1, letMn,t\+1∈\{0,1\}M\_\{n,t\+1\}\\in\\\{0,1\\\}indicate whether tokent\+1t\+1is supervised\. The corresponding predictor positions are
𝒫n=\{t:Mn,t\+1=1\}\.\\mathcal\{P\}\_\{n\}=\\left\\\{t:M\_\{n,t\+1\}=1\\right\\\}\.For adapted layerℓ\\ell, lethℓ,n,t∈ℝdh\_\{\\ell,n,t\}\\in\\mathbb\{R\}^\{d\}denote the adapter input at predictor positiontt\. PivotLoRA forms one activation vector per training example:
xℓ,n=1\|𝒫n\|∑t∈𝒫nhℓ,n,t\.x\_\{\\ell,n\}=\\frac\{1\}\{\|\\mathcal\{P\}\_\{n\}\|\}\\sum\_\{t\\in\\mathcal\{P\}\_\{n\}\}h\_\{\\ell,n,t\}\.\(8\)We then center the resulting example\-level activation matrix and compute its leadingrrprincipal directions using Eq\. \([6](https://arxiv.org/html/2611.23428#S4.E6)\)\. For the reasoning experiments, these directions initializeAℓ\(0\)=5/dQℓ⊤A\_\{\\ell\}^\{\(0\)\}=\\sqrt\{5/d\}\\,Q\_\{\\ell\}^\{\\top\}, withBℓ\(0\)=𝟎B\_\{\\ell\}^\{\(0\)\}=\\mathbf\{0\}, the same scale as Eq\. \([7](https://arxiv.org/html/2611.23428#S4.E7)\) without the rotation\. Appendix[D](https://arxiv.org/html/2611.23428#A4)gives the prompt templates, masking rules, generation settings, and remaining training configuration\.
### 4\.3Output\-Preserving Orthogonal Pivot
BecauseAAremains trainable, its initially orthogonal rows can become anisotropic during early optimization\. PivotLoRA performs a single reparameterization afterKKepochs while preserving both the learned adapter update and the model function\.
LetA\(K\)∈ℝr×dA^\{\(K\)\}\\in\\mathbb\{R\}^\{r\\times d\}andB\(K\)∈ℝm×rB^\{\(K\)\}\\in\\mathbb\{R\}^\{m\\times r\}denote the factors at the pivot\. Assuming thatA\(K\)A^\{\(K\)\}has full row rank, we compute the reduced QR decomposition of its transpose \(in double precision for reasoning and in float32 for classification\):
\(A\(K\)\)⊤=QR,Q⊤Q=Ir,\\bigl\(A^\{\(K\)\}\\bigr\)^\{\\top\}=QR,\\qquad Q^\{\\top\}Q=I\_\{r\},\(9\)and define
Anew=Q⊤\.A^\{\\mathrm\{new\}\}=Q^\{\\top\}\.The rows ofAnewA^\{\\mathrm\{new\}\}are orthonormal and have the same row span asA\(K\)A^\{\(K\)\}\. We re\-encode the output factor as
Bnew=B\(K\)A\(K\)\(Anew\)⊤\.B^\{\\mathrm\{new\}\}=B^\{\(K\)\}A^\{\(K\)\}\\bigl\(A^\{\\mathrm\{new\}\}\\bigr\)^\{\\top\}\.\(10\)BecauseAnewA^\{\\mathrm\{new\}\}is an orthonormal basis for the row span ofA\(K\)A^\{\(K\)\}, we haveA\(K\)=A\(K\)\(Anew\)⊤AnewA^\{\(K\)\}=A^\{\(K\)\}\(A^\{\\mathrm\{new\}\}\)^\{\\top\}A^\{\\mathrm\{new\}\}, and therefore
BnewAnew=B\(K\)A\(K\)\.B^\{\\mathrm\{new\}\}A^\{\\mathrm\{new\}\}=B^\{\(K\)\}A^\{\(K\)\}\.\(11\)The fixed scaleα/r\\alpha/rin Eq\. \([5](https://arxiv.org/html/2611.23428#S4.E5)\) is unchanged\. The pivot therefore preserves the LoRA update, and hence the model function, up to floating\-point error\.
Although the represented update is unchanged, subsequent factorized optimization need not be invariant to the reparameterization\. In particular, ifG=∇ΔWℒG=\\nabla\_\{\\Delta W\}\\mathcal\{L\}, then
∇Bℒ=αrGA⊤\.\\nabla\_\{B\}\\mathcal\{L\}=\\frac\{\\alpha\}\{r\}GA^\{\\top\}\.The geometry of theBB\-factor update therefore depends on the Gram matrixAA⊤AA^\{\\top\}\. The pivot restores
Anew\(Anew\)⊤=Ir,A^\{\\mathrm\{new\}\}\\bigl\(A^\{\\mathrm\{new\}\}\\bigr\)^\{\\top\}=I\_\{r\},removing coordinate anisotropy from this Gram matrix without replacing the learned row span\. Concretely, a plain gradient step onBBchanges the update by−η\(α/r\)2GA⊤A\-\\eta\(\\alpha/r\)^\{2\}GA^\{\\top\}A, so before the pivot each direction in the row span is scaled by the corresponding squared singular value ofAA\. After the pivot,A⊤AA^\{\\top\}Ais the orthogonal projector onto that span, and every learned direction receives the same step\. The pivot also returns the rows ofAAto unit norm, which enlarges the effective step onBB\. In classification, the norm ofAAchanges little before the pivot, so the pivot raises it about twelvefold\. In reasoning,AAgrows about sixfold and its Gram matrix has an effective rank of about two or fewer out of eight, and the pivot restores all eight \(Appendix[F\.6](https://arxiv.org/html/2611.23428#A6.SS6)\)\.
Optimizer\-state handling is distinct from the algebraic preservation guarantee\. In the classification implementation, we reset the Adam moments forAAand retain those forBB\. In the reasoning implementation, we reset the optimizer states for both transformed factors while leaving the global learning\-rate schedule unchanged\. In the classification ablation \(Appendix[F\.7](https://arxiv.org/html/2611.23428#A6.SS7)\), the QR pivot itself accounts for most of PivotLoRA’s improvement over TT\-PCA, while the optimizer\-state reset produces smaller, task\-dependent differences; we therefore regard the reset as an optional stabilizer\. The pivot itself takes under a second, and most of PivotLoRA’s 5\.8% wall\-time overhead on BERT\-base comes from the per\-rank optimizer groups \(Appendix[G\.3](https://arxiv.org/html/2611.23428#A7.SS3)\)\.
For each pivoted adapter we record the relative update\-preservation error and the row\-orthogonality error ofAnewA^\{\\mathrm\{new\}\}; both stay below10−610^\{\-6\}in every run \(Appendix[F\.6](https://arxiv.org/html/2611.23428#A6.SS6)\)\. Unless stated otherwise, PivotLoRA pivots atK=3K=3\.
## 5Experiments
We evaluate PivotLoRA on classification and generation tasks with four pretrained models\. BERT\-base provides the main controlled regime analysis, decoder\-scale classification tests transfer to larger causal models, GPT\-2 summarization tests whether a larged/Nd/Nsuffices, and GSM8K and StrategyQA test the response\-position extension for generated\-answer reasoning\.
### 5\.1Setup
#### Models, tasks, and protocol\.
We evaluate BERT\-base\([Devlin et al\., 2019](https://arxiv.org/html/2611.23428#bib.bib16)\)on five GLUE tasks \(SST\-2, CoLA, MNLI, QNLI, and RTE\)\([Wang et al\., 2019](https://arxiv.org/html/2611.23428#bib.bib20)\)and AG News\([Zhang et al\., 2015](https://arxiv.org/html/2611.23428#bib.bib27)\)atN∈\{16,32,64,128,256\}N\\in\\\{16,32,64,128,256\\\}total labeled examples, sampled approximately class\-balanced, with BoolQ as an auxiliary seventh task for the alignment analysis\. We evaluate Llama\-2\-7B\([Touvron et al\., 2023](https://arxiv.org/html/2611.23428#bib.bib18)\)and Llama\-3\.1\-8B\([Grattafiori et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib19)\)on SST\-2 and AG News atN∈\{64,256\}N\\in\\\{64,256\\\}, and additionally evaluate Llama\-3\.1\-8B on BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2611.23428#bib.bib26)\)\. We evaluate GPT\-2\([Radford et al\., 2019](https://arxiv.org/html/2611.23428#bib.bib17)\)on XSum\([Narayan et al\., 2018](https://arxiv.org/html/2611.23428#bib.bib28)\)and CNN/DailyMail\([Hermann et al\., 2015](https://arxiv.org/html/2611.23428#bib.bib29);[Nallapati et al\., 2016](https://arxiv.org/html/2611.23428#bib.bib30)\)withN∈\{64,256\}N\\in\\\{64,256\\\}training sequences\. Finally, we evaluate completion\-only reasoning with Llama\-3\.1\-8B on GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2611.23428#bib.bib35)\)and StrategyQA\([Geva et al\., 2021](https://arxiv.org/html/2611.23428#bib.bib36)\)usingN=128N=128training sequences, three paired seeds, and the complete test split of each dataset\.
The primary BERT benchmark contains 30 task–NNsettings over three seeds\. All methods use rankr=8r=8, query and value adapters, AdamW, 15 epochs, and batch size 16\. The shared BERT base learning\-rate anchor isηA=2×10−4\\eta\_\{A\}=2\\times 10^\{\-4\}\. Unless a baseline specifies otherwise,αLoRA=16\\alpha\_\{\\mathrm\{LoRA\}\}=16\. We compare LoRA, DoRA, AdaLoRA, PiSSA, LoRA\-GA, LoRA\-One, LoRA\+, EVA, LoRAM, and SRLoRA, plus the TT\-PCA and TT\-LDA ablations\. Method\-specific learning\-rate allocations, the shared initialization scaling, other implementation settings, sampling, hardware, and evaluation details appear in Appendix[C](https://arxiv.org/html/2611.23428#A3)\.
For generated\-answer reasoning, paired LoRA and PivotLoRA runs share the sampled examples, base AdamW configuration, learning\-rate schedule, tokenization, completion mask, training budget, decoding procedure, and answer scoring\. Response\-position PivotLoRA additionally applies PCA initialization, one output\-preserving pivot after epochK=3K=3, and its prescribed post\-pivot optimizer\-state reset\. Unlike the classification protocol, neither reasoning method uses differential or eigenvalue\-scaled learning rates\.
### 5\.2Few\-Shot Classification on BERT\-Base
Table[3](https://arxiv.org/html/2611.23428#S5.T3)summarizes all six primary tasks, reporting each method’s mean score across tasks at eachNN\(accuracy, or Matthews correlation for CoLA\), its average rank over the 30 task–NNsettings, and the number of settings in which it exceeds vanilla LoRA\. Per\-task results with standard deviations appear in Appendix[E\.1](https://arxiv.org/html/2611.23428#A5.SS1)\. Under the shared protocol, no prior initialization method consistently outperforms vanilla LoRA over all 30 settings\. TT\-PCA surpasses the published baselines on average, supporting activation\-based initialization in this regime\.
PivotLoRA achieves the best average rank and improves over vanilla LoRA in 29 of 30 primary settings \(Appendix[E\.2](https://arxiv.org/html/2611.23428#A5.SS2)\)\. It is strongest atN≤64N\\leq 64, with an average rank of 1\.39 and the best or tied\-best result in 13 of 18 settings\. The margins narrow atN∈\{128,256\}N\\in\\\{128,256\\\}asd/Nd/Ndecreases\. On AG News atN=64N=64, for example, PivotLoRA reaches about 89% of the full\-data accuracy, compared with 68% for LoRA \(Appendix[E\.11](https://arxiv.org/html/2611.23428#A5.SS11)\)\.
Table 1:BERT\-base, mean over six tasks and three seeds at eachNN; rank and\>\{\>\}LoRA over 30 settings\.Bold/underline: best/second\.Labeled examplesNNMethod163264128256Rank↓\\downarrow\>\{\>\}LoRALoRA\.402\.428\.488\.566\.6185\.23–DoRA\.404\.423\.486\.561\.6145\.8310/30AdaLoRA\.379\.381\.407\.433\.45811\.302/30PiSSA\.383\.395\.426\.499\.5819\.872/30LoRA\-GA\.383\.391\.427\.505\.6019\.305/30LoRA\-One\.379\.388\.431\.523\.6068\.476/30LoRA\+\.459\.481\.438\.425\.4158\.439/30EVA\.381\.391\.429\.520\.6068\.475/30LoRAM\.379\.387\.434\.517\.6088\.436/30SRLoRA\.401\.423\.472\.554\.6176\.934/30TT\-PCA\.415\.445\.546\.606\.6313\.7724/30TT\-LDA\.415\.455\.555\.607\.6343\.1325/30PivotLoRA\.451\.500\.575\.606\.6411\.8329/30
Table 2:Decoder\-scale classification atN=64N=64, mean over three seeds\.Llama\-2\-7BLlama\-3\.1\-8BMethodSST\-2AG NewsSST\-2AG NewsLoRA\.708\.748\.704\.759PiSSA\.635\.700\.622\.715LoRAM\.763\.742\.662\.748EVA\.713\.745\.643\.743PivotLoRA\.769\.784\.709\.799
Table 3:Llama\-3\.1\-8B reasoning atN=128N\{=\}128: accuracy \(%\) over three seeds,±\\pmone sample SD\. McNemar tests are exact and pooled over seeds\.MethodGSM8KStrategyQABase \(no adapter\)25\.654\.4LoRA28\.7±\\pm0\.264\.1±\\pm2\.3PivotLoRA33\.2±\\pm2\.567\.7±\\pm1\.4Paired gain\+4\.5±2\.3\+4\.5\\pm 2\.3\+3\.6±3\.1\+3\.6\\pm 3\.1Seeds improved3/32/3 \(1 tie\)McNemarpp<10−6<10^\{\-6\}0\.004
#### Sensitivity and full fine\-tuning\.
AtN=64N=64, a representative learning\-rate sweep narrows some shared\-protocol margins, but PivotLoRA remains strongest on average over AG News, SST\-2, and CoLA among the swept methods, with 0\.667 versus 0\.655 for TT\-PCA, 0\.636 for LoRA\+, and 0\.622 for LoRA \(Appendices[F\.9](https://arxiv.org/html/2611.23428#A6.SS9)and[F\.4](https://arxiv.org/html/2611.23428#A6.SS4)\)\. At the sameNN, PivotLoRA exceeds the mean full\-fine\-tuning score on all six primary tasks while training approximately 0\.30M adapter parameters \(Appendix[E\.6](https://arxiv.org/html/2611.23428#A5.SS6)\)\.
#### Direction versus learning\-rate allocation\.
AtN=64N=64, a random input basis drawn as in vanilla LoRA and given the same eigenvalue\-scaled learning rates reaches 0\.444, 0\.571, and 0\.028 on AG News, SST\-2, and CoLA, versus 0\.751, 0\.769, and 0\.268 for TT\-PCA, while TT\-PCA with uniform or permuted rates is essentially unchanged \(Appendix[F\.10](https://arxiv.org/html/2611.23428#A6.SS10)\)\. The PCA directions, not the rate allocation, carry the gain\.
#### Contribution of the pivot\.
Applied to the same TT\-PCA initialization without the optimizer\-state reset, the pivot alone improves the score in 12 of 15 task–NNsettings, by0\.0370\.037on average and by up to0\.1280\.128on AG News atN=32N=32\(Appendix[F\.7](https://arxiv.org/html/2611.23428#A6.SS7)\)\. Each tested pivot epoch,K∈\{1,2,3,5,7,10\}K\\in\\\{1,2,3,5,7,10\\\}, outperforms LoRA at everyN≤64N\\leq 64on SST\-2, CoLA, and AG News, so the defaultK=3K=3is not a finely tuned choice \(Appendix[F\.5](https://arxiv.org/html/2611.23428#A6.SS5)\)\. The gain also does not depend on how theBBlearning rate is handled after the pivot, since leaving it unchanged gives\+0\.036\+0\.036over TT\-PCA \(Appendix[F\.8](https://arxiv.org/html/2611.23428#A6.SS8)\)\.
#### Robustness\.
A two\-sided sign test on the 29\-of\-30 record givesp=5\.8×10−8p=5\.8\\times 10^\{\-8\}, and five\-seed reruns of seven representative settings yield bootstrap 95% intervals for the paired gain over LoRA that exclude zero in every case, with mean gains from\+0\.032\+0\.032on QNLI to\+0\.231\+0\.231on AG News \(Appendices[F\.1](https://arxiv.org/html/2611.23428#A6.SS1)and[F\.2](https://arxiv.org/html/2611.23428#A6.SS2)\)\. Restoring LoRAM’s intended initialization magnitude does not close its gap to LoRA \(Appendix[C](https://arxiv.org/html/2611.23428#A3)\), so initialization magnitude alone does not explain the gains\.
### 5\.3Decoder\-Scale Classification
The activation\-subspace pattern also appears in decoder\-scale classification \(Table[3](https://arxiv.org/html/2611.23428#S5.T3)\)\. AtN=64N=64, PivotLoRA ranks first on both backbones, while EVA does not close the aggregate gap under the same per\-layer rank budget\. AtN=256N=256, PivotLoRA trails LoRA in three of the four SST\-2 and AG News settings, whereas on BoolQ with Llama\-3\.1\-8B PivotLoRA leads LoRA at bothN=64N=64andN=256N=256\(\.602 versus \.554 and \.618 versus \.576\)\. Complete results appear in Appendix[E\.3](https://arxiv.org/html/2611.23428#A5.SS3)\. Because the decoder activation source is a sequence mean, these gains suggest that a coarse source suffices when the label describes the whole sequence\. AtN=256N=256, the all\-token EVA initializer becomes competitive, consistent with the alignment–estimation trade\-off, since a larger sample favors the source with more activation vectors\.
### 5\.4Generated\-Answer Reasoning
Figure 3:Alignmentρ\\rhoversus PivotLoRA’s gain over LoRA across seven BERT tasks atN=64N=64\(CoLA in Matthews correlation points\)\.
Settingd/Nd/NSourceGainWinsBERT,N≤64N\\leq 6412–48\[CLS\]\+6\.9\+6\.918/18BERT,N≥128N\\geq 1283–6\[CLS\]\+3\.2\+3\.211/12BERT, full data<0\.4\{<\}0\.4\[CLS\]\+0\.4\+0\.43/6Llama\-2/3\.1,N=64N=6464seq\. mean\+3\.8\+3\.85/5Llama\-2/3\.1,N=256N=25616seq\. mean\+0\.4\+0\.42/5Llama\-3\.1, reasoning32response\+4\.1\+4\.12/2GPT\-2, summarization3–12response\+0\.3\+0\.33/4Table 4:Regime map\. Mean gain of PivotLoRA over LoRA in points of each task’s primary metric, and settings won \(details in Table[29](https://arxiv.org/html/2611.23428#A5.T29)\)\.d/Nd/Nis comparable within a backbone\.
Table[3](https://arxiv.org/html/2611.23428#S5.T3)reports completion\-only fine\-tuning with Llama\-3\.1\-8B on GSM8K \(mathematical word problems with worked solutions\) and StrategyQA \(yes/no questions supervised with supporting facts\)\. Within each dataset and seed, LoRA and PivotLoRA share the sameN=128N=128training examples and are evaluated on the complete test split \(1,319 and 687 questions\)\. PivotLoRA improves on LoRA on both datasets, at every seed on GSM8K, and an exact item\-level McNemar test pooled over seeds rejects equal accuracy on both\. Pooled over the three seeds, PivotLoRA alone answers 655 GSM8K test items correctly against 475 for LoRA alone, and 350 against 276 on StrategyQA\. Both methods follow the trained answer format in at least 95% of generations, so formatting does not explain the gains\. These comparisons evaluate the full procedure \(Appendix[D](https://arxiv.org/html/2611.23428#A4)\)\. In these runs, early training concentrates each input factor on about two or fewer of its eight directions before the pivot, and the pivot restores an isotropic basis for the same span \(Appendix[F\.6](https://arxiv.org/html/2611.23428#A6.SS6)\)\.
### 5\.5Regime Summary and Alignment Diagnostic
Table[4](https://arxiv.org/html/2611.23428#S5.T4)organizes every evaluated setting byd/Nd/Nand activation source, following Remark[1](https://arxiv.org/html/2611.23428#Thmremark1)\. Within each backbone the gain shrinks asd/Nd/Nfalls, and with full training sets PivotLoRA, LoRA, and PiSSA perform comparably \(Appendix[E\.5](https://arxiv.org/html/2611.23428#A5.SS5)\)\. On GPT\-2 summarization, PivotLoRA is competitive with LoRA but not decisively better under the single learning rate used for reasoning \(\+0\.2\+0\.2to\+0\.3\+0\.3ROUGE\-L points\), and it trails under the classification learning\-rate recipe whichever positions it pools \(Appendix[E\.4](https://arxiv.org/html/2611.23428#A5.SS4)\)\. Because the same response\-position source helps reasoning, the difference may reflect the backbone, the adapted modules, or how much loss\-relevant structure the leading directions capture in summarization\. A larged/Nd/Nis therefore not sufficient by itself\.
Across the six primary BERT tasks plus auxiliary BoolQ, the classification alignment coefficientρ\\rhoatN=64N=64is strongly associated with PivotLoRA’s shared\-protocol gain over vanilla LoRA, with Pearsonr=0\.93r=0\.93and exact permutationp=0\.002p=0\.002as shown in Figure[3](https://arxiv.org/html/2611.23428#S5.F3)\. This is consistent with Equation \([4](https://arxiv.org/html/2611.23428#S3.E4)\), in whichρ\\rhois the share of between\-class signal available to the first update ofBB\. Spearman correlation is 0\.86, leave\-one\-task\-out Pearson correlations stay within\[0\.87,0\.96\]\[0\.87,0\.96\], and a cross\-split estimate preserves the task ordering \(Appendices[E\.7](https://arxiv.org/html/2611.23428#A5.SS7)and[E\.8](https://arxiv.org/html/2611.23428#A5.SS8)\)\. Bothd/Nd/Nandρ\\rhoare available before training\. Whend/Nd/Nis large andρ\\rhostays high across resampled subsets, direction\-aware initialization is likely to help, and when either is small, a well\-tuned vanilla LoRA is a strong default\.
## 6Conclusion
Initialization direction is not uniformly consequential for LoRA\. It matters when few labeled examples leave the adapter’s row span sensitive to its starting point and the estimated activation subspace carries supervised structure, and it fades once data are abundant or the activation source is poorly aligned with the objective\. An alignment measure that does not need class labels, such as one built from loss gradients at initialization, would extend the diagnostic to generative tasks, and calibrating it across backbones would turn the regime account into a decision rule\. Because the pivot applies to any full\-row\-rank starting basis, it could also test whether these regime boundaries hold for weight\- or gradient\-based initializers\.
## Acknowledgments and Disclosure of Funding
#### Funding and support\.
We thank the University of Delaware for providing computational resources\. Peng Zhao’s work was partially supported by a research gift from Microsoft Research\.
#### Competing interests\.
The authors declare no competing interests\.
## References
- Baiket al\.\(2005\)J\. Baik, G\. Ben Arous, and S\. PéchéPhase transition of the largest eigenvalue for nonnull complex sample covariance matrices\.The Annals of Probability33\(5\),pp\. 1643–1697\.External Links:[Document](https://dx.doi.org/10.1214/009117905000000233)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px4.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1300),[Link](https://aclanthology.org/N19-1300/)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:2110\.14168,[Document](https://dx.doi.org/10.48550/arXiv.2110.14168),[Link](https://arxiv.org/abs/2110.14168)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Daganet al\.\(2006\)I\. Dagan, O\. Glickman, and B\. MagniniThe PASCAL recognising textual entailment challenge\.InMachine Learning Challenges: Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment,J\. Quiñonero\-Candela, I\. Dagan, B\. Magnini, and F\. d’Alché\-Buc \(Eds\.\),Lecture Notes in Computer Science, Vol\.3944,pp\. 177–190\.External Links:[Document](https://dx.doi.org/10.1007/11736790%5F9),[Link](https://link.springer.com/chapter/10.1007/11736790_9)Cited by:[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px3.p1.1)\.
- Davis and Kahan \(1970\)C\. Davis and W\. M\. KahanThe rotation of eigenvectors by a perturbation\. III\.SIAM Journal on Numerical Analysis7\(1\),pp\. 1–46\.External Links:[Document](https://dx.doi.org/10.1137/0707001)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2611.23428#S3.SS0.SSS0.Px2.p2.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423),[Link](https://aclanthology.org/N19-1423/)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Gevaet al\.\(2021\)M\. Geva, D\. Khashabi, E\. Segal, T\. Khot, D\. Roth, and J\. BerantDid Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies\.Transactions of the Association for Computational Linguistics9,pp\. 346–361\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00370),[Link](https://aclanthology.org/2021.tacl-1.21/)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Grattafioriet al\.\(2024\)A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan, A\. Yang, A\. Fan, A\. Goyal, A\. Hartshorn, A\. Yang,et al\.The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:2407\.21783,[Document](https://dx.doi.org/10.48550/arXiv.2407.21783),[Link](https://arxiv.org/abs/2407.21783)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Haoet al\.\(2024\)Y\. Hao, Y\. Cao, and L\. MouFlora: low\-rank adapters are secretly gradient compressors\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 17554–17571\.External Links:[Link](https://proceedings.mlr.press/v235/hao24a.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px2.p1.1)\.
- Hayouet al\.\(2024\)S\. Hayou, N\. Ghosh, and B\. YuLoRA\+: efficient low rank adaptation of large models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 17783–17806\.External Links:[Link](https://proceedings.mlr.press/v235/hayou24a.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2611.23428#S4.SS1.p5.1)\.
- Hermannet al\.\(2015\)K\. M\. Hermann, T\. Kočiský, E\. Grefenstette, L\. Espeholt, W\. Kay, M\. Suleyman, and P\. BlunsomTeaching machines to read and comprehend\.InAdvances in Neural Information Processing Systems,Vol\.28\.External Links:[Link](https://proceedings.neurips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Johnstone \(2001\)I\. M\. JohnstoneOn the distribution of the largest eigenvalue in principal components analysis\.The Annals of Statistics29\(2\),pp\. 295–327\.External Links:[Document](https://dx.doi.org/10.1214/aos/1009210544)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px4.p1.1)\.
- Leeet al\.\(2026\)Y\. Lee, C\. Ko, P\. Chen, and M\. YehLearning rate matters: vanilla LoRA may suffice for LLM fine\-tuning\.InAdvances in Neural Information Processing Systems,External Links:2602\.04998,[Link](https://arxiv.org/abs/2602.04998)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px3.p2.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024\)S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. ChenDoRA: weight\-decomposed low\-rank adaptation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32100–32121\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bn.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1)\.
- Menget al\.\(2024\)F\. Meng, Z\. Wang, and M\. ZhangPiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-3846),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/db36f4d603cc9e3a2a5e10b93e6428f2-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p2.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§E\.5](https://arxiv.org/html/2611.23428#A5.SS5.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Nallapatiet al\.\(2016\)R\. Nallapati, B\. Zhou, C\. dos Santos, Ç\. Gülçehre, and B\. XiangAbstractive text summarization using sequence\-to\-sequence RNNs and beyond\.InProceedings of the 20th SIGNLL Conference on Computational Natural Language Learning,Berlin, Germany,pp\. 280–290\.External Links:[Document](https://dx.doi.org/10.18653/v1/K16-1028),[Link](https://aclanthology.org/K16-1028/)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Narayanet al\.\(2018\)S\. Narayan, S\. B\. Cohen, and M\. LapataDon’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 1797–1807\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1206),[Link](https://aclanthology.org/D18-1206/)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Paischeret al\.\(2025\)F\. Paischer, L\. Hauzenberger, T\. Schmied, B\. Alkin, M\. Deisenroth, and S\. HochreiterParameter efficient fine\-tuning via explained variance adaptation\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 46215–46267\.External Links:[Document](https://dx.doi.org/10.52202/085713-1539),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/41d33bd41fd44bd9dba0e092047cf213-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p4.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,Austin, Texas,pp\. 2383–2392\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1264),[Link](https://aclanthology.org/D16-1264/)Cited by:[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px3.p1.1)\.
- Socheret al\.\(2013\)R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Ng, and C\. PottsRecursive deep models for semantic compositionality over a sentiment treebank\.InProceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,Seattle, Washington, USA,pp\. 1631–1642\.External Links:[Link](https://aclanthology.org/D13-1170/)Cited by:[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px3.p1.1)\.
- Touvronet al\.\(2023\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale, D\. Bikel, L\. Blecher, C\. Canton Ferrer, M\. Chen, G\. Cucurull, D\. Esiobu, J\. Fernandes, J\. Fu, W\. Fu, B\. Fuller, C\. Gao, V\. Goswami, N\. Goyal, A\. Hartshorn, S\. Hosseini, R\. Hou, H\. Inan, M\. Kardas, V\. Kerkez, M\. Khabsa, I\. Kloumann, A\. Korenev, P\. S\. Koura, M\. Lachaux, T\. Lavril, J\. Lee, D\. Liskovich, Y\. Lu, Y\. Mao, X\. Martinet, T\. Mihaylov, P\. Mishra, I\. Molybog, Y\. Nie, A\. Poulton, J\. Reizenstein, R\. Rungta, K\. Saladi, A\. Schelten, R\. Silva, E\. M\. Smith, R\. Subramanian, X\. E\. Tan, B\. Tang, R\. Taylor, A\. Williams, J\. X\. Kuan, P\. Xu, Z\. Yan, I\. Zarov, Y\. Zhang, A\. Fan, M\. Kambadur, S\. Narang, A\. Rodriguez, R\. Stojnic, S\. Edunov, and T\. ScialomLlama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.External Links:2307\.09288,[Document](https://dx.doi.org/10.48550/arXiv.2307.09288),[Link](https://arxiv.org/abs/2307.09288)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2019\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJ4km2R5t7)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)H\. Wang, Y\. Li, S\. Wang, G\. Chen, and Y\. ChenMiLoRA: harnessing minor singular components for parameter\-efficient LLM finetuning\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4823–4836\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.248),[Link](https://aclanthology.org/2025.naacl-long.248/)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p2.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)S\. Wang, L\. Yu, and J\. LiLoRA\-GA: low\-rank adaptation with gradient approximation\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-1741),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/62c4718cc334f6a0a62fb81c4a2095a1-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p3.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Warstadtet al\.\(2019\)A\. Warstadt, A\. Singh, and S\. R\. BowmanNeural network acceptability judgments\.Transactions of the Association for Computational Linguistics7,pp\. 625–641\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00290),[Link](https://aclanthology.org/Q19-1040/)Cited by:[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px3.p1.1)\.
- Williamset al\.\(2018\)A\. Williams, N\. Nangia, and S\. R\. BowmanA broad\-coverage challenge corpus for sentence understanding through inference\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),New Orleans, Louisiana,pp\. 1112–1122\.External Links:[Document](https://dx.doi.org/10.18653/v1/N18-1101),[Link](https://aclanthology.org/N18-1101/)Cited by:[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px3.p1.1)\.
- Yanget al\.\(2025\)H\. Yang, L\. Wang, and M\. Z\. HossainSRLoRA: subspace recomposition in low\-rank adaptation via importance\-based fusion and reinitialization\.arXiv preprint arXiv:2505\.12433\.External Links:2505\.12433,[Document](https://dx.doi.org/10.48550/arXiv.2505.12433),[Link](https://arxiv.org/abs/2505.12433)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px2.p1.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2024\)Y\. Yang, X\. Li, Z\. Zhou, S\. L\. Song, J\. Wu, L\. Nie, and B\. GhanemCorDA: context\-oriented decomposition adaptation of large language models for task\-aware parameter\-efficient fine\-tuning\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-2292),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/83f95bb0ac5046338ea2afe3390e9f4b-Abstract-Conference.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p4.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2015\)Y\. Yu, T\. Wang, and R\. J\. SamworthA useful variant of the Davis–Kahan theorem for statisticians\.Biometrika102\(2\),pp\. 315–323\.External Links:[Document](https://dx.doi.org/10.1093/biomet/asv008)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2611.23428#S3.SS0.SSS0.Px2.p2.1)\.
- Zhanget al\.\(2023\)Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. ZhaoAdaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=lq62uWRJjiY)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px3.p1.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1)\.
- Zhanget al\.\(2026\)Q\. Zhang, C\. Chu, T\. Peng, Q\. Li, X\. Luo, Z\. Jiang, and S\. HuangLoRA\-DA: data\-aware initialization for low\-rank adaptation via asymptotic analysis\.InProceedings of the 43rd International Conference on Machine Learning,External Links:2510\.24561,[Link](https://arxiv.org/abs/2510.24561)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p3.1),[§F\.3](https://arxiv.org/html/2611.23428#A6.SS3.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2015\)X\. Zhang, J\. Zhao, and Y\. LeCunCharacter\-level convolutional networks for text classification\.InAdvances in Neural Information Processing Systems,Vol\.28\.External Links:[Link](https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html)Cited by:[§5\.1](https://arxiv.org/html/2611.23428#S5.SS1.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025a\)Y\. Zhang, F\. Liu, and Y\. ChenLoRA\-One: one\-step full gradient could suffice for fine\-tuning large language models, provably and efficiently\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 75513–75574\.External Links:[Link](https://proceedings.mlr.press/v267/zhang25ax.html)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p3.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025b\)Z\. Zhang, H\. Li, Y\. Zhang, G\. Gong, J\. Wang, J\. Hu, P\. Liu, and Q\. JiangThe primacy of magnitude in low\-rank adaptation\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:2507\.06558,[Link](https://arxiv.org/abs/2507.06558)Cited by:[Appendix A](https://arxiv.org/html/2611.23428#A1.SS0.SSS0.Px1.p6.1),[Appendix C](https://arxiv.org/html/2611.23428#A3.SS0.SSS0.Px8.p1.1),[§1](https://arxiv.org/html/2611.23428#S1.p1.1),[§2](https://arxiv.org/html/2611.23428#S2.SS0.SSS0.Px2.p1.1),[Remark 1](https://arxiv.org/html/2611.23428#Thmremark1.p1.1.1)\.
## Appendix AExtended Related Work
#### LoRA initialization\.
Vanilla LoRA\[[Hu et al\., 2022](https://arxiv.org/html/2611.23428#bib.bib1)\]initializesAAwith Kaiming\-uniform noise andB=0B=0, so thatΔW=0\\Delta W=0at the start of training\. Subsequent work has explored several sources of initialization signal\.
*Weight\-based methods*derive the adapter subspace from the pretrained weight matrixW0W\_\{0\}\. PiSSA\[[Meng et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib2)\]initializesAAandBBfrom the top\-rrsingular components ofW0W\_\{0\}and freezes the residualWres=W0−W0\(r\)W^\{\\mathrm\{res\}\}=W\_\{0\}\-W\_\{0\}^\{\(r\)\}\. MiLoRA\[[Wang et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib3)\]takes the complementary approach and initializes from minor singular components to preserve the dominant directions ofW0W\_\{0\}\. Both methods are task\-agnostic and derive their subspaces from pretrained weight geometry alone\. Because our benchmark already includes PiSSA as a representative weight\-spectral baseline, we do not evaluate MiLoRA separately\.
*Gradient\-based methods*use the downstream task loss\. LoRA\-GA\[[Wang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib4)\]computes a forward–backward pass and initializes from the SVD of∇Wℒ\\nabla\_\{W\}\\mathcal\{L\}\. LoRA\-One\[[Zhang et al\., 2025a](https://arxiv.org/html/2611.23428#bib.bib5)\]uses balanced singular\-value scaling betweenAAandBB\. LoRA\-DA\[[Zhang et al\., 2026](https://arxiv.org/html/2611.23428#bib.bib6)\]augments the gradient with a diagonal Fisher estimate and solves a quadratic program for initialization\. Appendix[F\.3](https://arxiv.org/html/2611.23428#A6.SS3)reports our LoRA\-DA implementation under the few\-shot protocol used in this work\.
*Activation\-based methods*estimate a subspace from the input distribution\. EVA\[[Paischer et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib7)\]collects activations from all token positions in a calibration minibatch, computes an SVD of their covariance, and initializesAAfrom the resulting right singular vectors\. The original method also permits adaptive redistribution of rank across layers\. In our controlled comparison, we disable redistribution and use uniform rankr=8r=8at every adapted layer\. CorDA\[[Yang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib8)\]modulates the weight SVD by an activation covariance, thereby orienting the decomposition toward downstream input statistics\. CorDA requires covariance inversion, which was numerically unstable in our smallest\-NNsetting, so it is not included in the primary comparison\.
Activation\-based initializers also differ in which token positions supply the activation statistics\. All\-token pooling, as used by EVA, provides more activation observations, whereas PivotLoRA’s prediction\-position pooling targets representations associated with the supervised objective\. This distinction introduces an alignment–estimation trade\-off, with no universal advantage for either source\.
LoRAM\[[Zhang et al\., 2025b](https://arxiv.org/html/2611.23428#bib.bib9)\]provides a magnitude\-based perspective\. It argues that gains from pretrained weight\-spectral initializers can often be explained by update\-magnitude amplification, not by the semantic content of the selected singular directions\. LoRAM approximates these gains using magnitude\-scaled orthogonal bases without storing spectral components\. Our setting is complementary\. PivotLoRA studies directions estimated from downstream activation covariance and includes controls that separate prediction\-position direction from eigenvalue\-scaled learning rates\.
#### Dynamic adapter methods\.
SRLoRA\[[Yang et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib13)\]periodically fuses low\-importance rank\-one pairs into the frozen weights and reinitializes from unused SVD directions\. Flora\[[Hao et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib14)\]resamples random projections at fixed intervals\. PivotLoRA instead performs one output\-preserving orthogonal reparameterization\. It replaces the learned input basis with an orthonormal basis for the same row span and transformsBBso that the represented update is preserved up to numerical precision\. The pivot is therefore a coordinate change, unlike periodic rank replacement or random resampling\.
#### Structural and optimization variants\.
DoRA\[[Liu et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib11)\]decomposes the adapted weight into magnitude and direction components\. AdaLoRA\[[Zhang et al\., 2023](https://arxiv.org/html/2611.23428#bib.bib12)\]parameterizes adapters as SVD triplets and prunes singular values during training\. LoRA\+\[[Hayou et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib10)\]retains the standard factorization but uses different learning rates for the two factors, typically withηB/ηA≈16\\eta\_\{B\}/\\eta\_\{A\}\\approx 16\. These methods alter the adapter parameterization, rank allocation, or factor\-wise optimization rule\. PivotLoRA retains the standard low\-rank update and instead changes how its input\-side coordinates are initialized and, once during training, orthogonally reparameterized\.
Learning\-rate selection can materially affect comparisons among LoRA methods\.[Lee et al\. \[2026\]](https://arxiv.org/html/2611.23428#bib.bib15)systematically re\-evaluate vanilla LoRA and nine variants over learning rate, batch size, rank, and training duration\. They find that different methods favor different learning\-rate ranges and that, after broad tuning, their peak performance is often similar\. This result cautions against interpreting improvements obtained under one shared configuration as evidence of an intrinsic methodological advantage\. It also motivates our method\-specific learning\-rate sensitivity analysis \(Appendix[F\.9](https://arxiv.org/html/2611.23428#A6.SS9)\), which tests whether the principal comparisons persist over a prespecified range under our few\-shot protocol\.
#### Subspace estimation in high dimensions\.
The Davis–KahansinΘ\\sin\\Thetatheorem\[[Davis and Kahan, 1970](https://arxiv.org/html/2611.23428#bib.bib31),[Yu et al\., 2015](https://arxiv.org/html/2611.23428#bib.bib32)\]bounds the angle between estimated and population eigenspaces under spectral perturbation\. Under spiked covariance models\[[Johnstone, 2001](https://arxiv.org/html/2611.23428#bib.bib34)\], PCA recovery depends on sample size, effective dimension, and spectral separation\. In proportional asymptotic regimes, eigenvectors can also undergo a phase transition near the BBP threshold\[[Baik et al\., 2005](https://arxiv.org/html/2611.23428#bib.bib33)\]\. We use these results as regime\-level motivation for dependence ondeff/Nd\_\{\\mathrm\{eff\}\}/N, not as a complete probabilistic model of transformer fine\-tuning\.
## Appendix BAdditional Theory
### B\.1Empirical Variance Concentration in\[CLS\]Activations
Section[3](https://arxiv.org/html/2611.23428#S3)models prediction\-position activations through the spiked covariance form
Σ=∑i=1r∗\(λi−σ2\)vivi⊤\+σ2I,\\Sigma=\\sum\_\{i=1\}^\{r^\{\\ast\}\}\(\\lambda\_\{i\}\-\\sigma^\{2\}\)v\_\{i\}v\_\{i\}^\{\\top\}\+\\sigma^\{2\}I,whereλi\>σ2\\lambda\_\{i\}\>\\sigma^\{2\}, the vectorsv1,…,vr∗v\_\{1\},\\ldots,v\_\{r^\{\\ast\}\}are orthonormal signal directions, andσ2I\\sigma^\{2\}Iis an isotropic noise floor\. Under this parameterization, the population eigenvalue alongviv\_\{i\}isλi\\lambda\_\{i\}, and each remaining population eigenvalue isσ2\\sigma^\{2\}\.
The model is idealized but captures the possibility that substantial activation variance is concentrated in a low\-dimensional subspace\. We examine this premise using frozen BERT\-base\[CLS\]representations\.
We run the frozen encoder onN=64N=64total labeled examples from AG News and SST\-2, sampled approximately class\-balanced\. We extract the final\-layer\[CLS\]representation, construct
X¯=X−𝟏μ⊤∈ℝN×768,μ=N−1X⊤𝟏,\\bar\{X\}=X\-\\mathbf\{1\}\\mu^\{\\top\}\\in\\mathbb\{R\}^\{N\\times 768\},\\qquad\\mu=N^\{\-1\}X^\{\\top\}\\mathbf\{1\},and compute
Σ^=N−1X¯⊤X¯\.\\widehat\{\\Sigma\}=N^\{\-1\}\\bar\{X\}^\{\\top\}\\bar\{X\}\.Figure[4](https://arxiv.org/html/2611.23428#A2.F4)reports the leading eigenvalues and cumulative explained variance\.
Figure 4:Eigenvalue spectrum and cumulative variance of frozen BERT\-base\[CLS\]activations atN=64N=64\. The topr=8r=8components capture 38% of the total variance for AG News and 46% for SST\-2\. The spectra decay continuously, with no sharp finite\-rank cutoff\.The top eight components use approximately one percent of the 768\-dimensional representation space while capturing 38–46% of total variance\. BecauseN=64N=64, the sample covariance has rank at most 63, so this share is measured over at most 63 nonzero sample eigenvalues\. This indicates substantial low\-dimensional concentration without an exact finite\-rank cutoff, so we use the spiked model and the perturbation calculation below as regime\-level guidance\.
### B\.2Derivation of Equation \([2](https://arxiv.org/html/2611.23428#S3.E2)\)
LetX1,…,XN∈ℝdX\_\{1\},\\ldots,X\_\{N\}\\in\\mathbb\{R\}^\{d\}be independent, centered, sub\-Gaussian activation vectors with covarianceΣ\\Sigma\. LetΣ^\\widehat\{\\Sigma\}be their sample covariance, letV∗V^\{\\ast\}contain the leadingr∗r^\{\\ast\}population eigenvectors, and letV^\\widehat\{V\}contain the corresponding sample eigenvectors\. We take the adapter rank equal to the number of spikes,r=r∗r=r^\{\\ast\}\. Forr≠r∗r\\neq r^\{\\ast\}the gap isλr−λr\+1\\lambda\_\{r\}\-\\lambda\_\{r\+1\}\. Under the spiked model, the eigengap is
δ=λr∗−σ2\>0\.\\delta=\\lambda\_\{r^\{\\ast\}\}\-\\sigma^\{2\}\>0\.
An operator\-norm Davis–Kahan bound gives
‖sinΘ\(V^,V∗\)‖op≲‖Σ^−Σ‖opδ\.\\left\\\|\\sin\\Theta\(\\widehat\{V\},V^\{\\ast\}\)\\right\\\|\_\{\\mathrm\{op\}\}\\lesssim\\frac\{\\\|\\widehat\{\\Sigma\}\-\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\delta\}\.Effective\-rank covariance concentration gives
𝔼\[‖Σ^−Σ‖op\]≲‖Σ‖op\(deffN\+deffN\),\\mathbb\{E\}\\left\[\\\|\\widehat\{\\Sigma\}\-\\Sigma\\\|\_\{\\mathrm\{op\}\}\\right\]\\lesssim\\\|\\Sigma\\\|\_\{\\mathrm\{op\}\}\\left\(\\sqrt\{\\frac\{d\_\{\\mathrm\{eff\}\}\}\{N\}\}\+\\frac\{d\_\{\\mathrm\{eff\}\}\}\{N\}\\right\),where
deff=Tr\(Σ\)‖Σ‖op\.d\_\{\\mathrm\{eff\}\}=\\frac\{\\operatorname\{Tr\}\(\\Sigma\)\}\{\\\|\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\.
For a principal\-angle matrix of rank at mostr∗r^\{\\ast\},
‖sinΘ‖F≤r∗‖sinΘ‖op,‖sinΘ‖F≤r∗\.\\\|\\sin\\Theta\\\|\_\{F\}\\leq\\sqrt\{r^\{\\ast\}\}\\,\\\|\\sin\\Theta\\\|\_\{\\mathrm\{op\}\},\\qquad\\\|\\sin\\Theta\\\|\_\{F\}\\leq\\sqrt\{r^\{\\ast\}\}\.Combining these inequalities yields
𝔼\[‖sinΘ\(V^,V∗\)‖F\]≲min\{r∗,r∗‖Σ‖opδ\(deffN\+deffN\)\},\\mathbb\{E\}\\left\[\\\|\\sin\\Theta\(\\widehat\{V\},V^\{\\ast\}\)\\\|\_\{F\}\\right\]\\lesssim\\min\\left\\\{\\sqrt\{r^\{\\ast\}\},\\sqrt\{r^\{\\ast\}\}\\,\\frac\{\\\|\\Sigma\\\|\_\{\\mathrm\{op\}\}\}\{\\delta\}\\left\(\\sqrt\{\\frac\{d\_\{\\mathrm\{eff\}\}\}\{N\}\}\+\\frac\{d\_\{\\mathrm\{eff\}\}\}\{N\}\\right\)\\right\\\},which is Equation \([2](https://arxiv.org/html/2611.23428#S3.E2)\)\.
The bound provides stronger control whenNNis large relative todeffd\_\{\\mathrm\{eff\}\}and when the eigengapδ\\deltais large\. If either condition deteriorates, the bound becomes weaker, which is consistent with greater finite\-sample sensitivity of the estimated subspace\. Transformer activations need not be independent or exactly sub\-Gaussian, and their covariance need not follow a literal spiked model, so the calculation is not a finite\-sample theorem for transformer fine\-tuning\.
The PCA covariance uses allNNexamples\. By contrast, the alignment diagnostic depends on class means estimated from approximatelyN/CN/Cexamples per class\. We therefore express PCA estimation throughdeff/Nd\_\{\\mathrm\{eff\}\}/Nwhile recognizing the additional uncertainty in class\-conditional quantities\.
### B\.3First\-Step LoRA Signal and Task Alignment
This section derives the local first\-step relationship used in Section[3](https://arxiv.org/html/2611.23428#S3)\. Consider
Δz=sBU⊤h\(x\),s=α/r,\\Delta z=s\\,BU^\{\\top\}h\(x\),\\qquad s=\\alpha/r,whereU∈ℝd×rU\\in\\mathbb\{R\}^\{d\\times r\}has orthonormal columns andB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}\. This corresponds toA=U⊤A=U^\{\\top\}\.
Define
q\(x,y\)=∇Δzℓ\(z0\(x\)\+Δz,y\)\|Δz=0q\(x,y\)=\\left\.\\nabla\_\{\\Delta z\}\\ell\\\!\\left\(z\_\{0\}\(x\)\+\\Delta z,y\\right\)\\right\|\_\{\\Delta z=0\}and
ℒ\(B,U\)=𝔼\(x,y\)\[ℓ\(z0\(x\)\+sBU⊤h\(x\),y\)\]\.\\mathcal\{L\}\(B;U\)=\\mathbb\{E\}\_\{\(x,y\)\}\\left\[\\ell\\\!\\left\(z\_\{0\}\(x\)\+sBU^\{\\top\}h\(x\),y\\right\)\\right\]\.
###### Proposition 1\(First\-step signal in a fixed LoRA subspace\)\.
Assume thatℒ\(B,U\)\\mathcal\{L\}\(B;U\)is twice differentiable nearB=0B=0with a locally bounded Hessian\. Let
GU=∇Bℒ\(0,U\)\.G\_\{U\}=\\nabla\_\{B\}\\mathcal\{L\}\(0;U\)\.A gradient stepB1=−ηGUB\_\{1\}=\-\\eta G\_\{U\}fromB0=0B\_\{0\}=0satisfies
ℒ\(B1,U\)−ℒ\(0,U\)=−η‖GU‖F2\+O\(η2‖GU‖F2\)\.\\mathcal\{L\}\(B\_\{1\};U\)\-\\mathcal\{L\}\(0;U\)=\-\\eta\\\|G\_\{U\}\\\|\_\{F\}^\{2\}\+O\\\!\\left\(\\eta^\{2\}\\\|G\_\{U\}\\\|\_\{F\}^\{2\}\\right\)\.
Suppose further that, conditional on classy=cy=c,
h\(x\)=μc\+ε,q\(x,y\)=gc\+ξ,h\(x\)=\\mu\_\{c\}\+\\varepsilon,\\qquad q\(x,y\)=g\_\{c\}\+\\xi,where
𝔼\[ε∣y=c\]=0,𝔼\[ξ∣y=c\]=0,\\mathbb\{E\}\[\\varepsilon\\mid y=c\]=0,\\qquad\\mathbb\{E\}\[\\xi\\mid y=c\]=0,and the class\-conditional cross\-covariance vanishes:
𝔼\[ξε⊤∣y=c\]=0\.\\mathbb\{E\}\\left\[\\xi\\varepsilon^\{\\top\}\\mid y=c\\right\]=0\.Let the class priors beπc\\pi\_\{c\}, define
μ¯=∑c=1Cπcμc,\\bar\{\\mu\}=\\sum\_\{c=1\}^\{C\}\\pi\_\{c\}\\mu\_\{c\},and assume the class\-gradient codes are centered:
∑c=1Cπcgc=0\.\\sum\_\{c=1\}^\{C\}\\pi\_\{c\}g\_\{c\}=0\.Then
‖GU‖F2=s2Tr\(U⊤HgradU\),\\\|G\_\{U\}\\\|\_\{F\}^\{2\}=s^\{2\}\\operatorname\{Tr\}\\left\(U^\{\\top\}H\_\{\\mathrm\{grad\}\}U\\right\),where
Hgrad=∑c,c′=1Cπcπc′⟨gc,gc′⟩\(μc−μ¯\)\(μc′−μ¯\)⊤\.H\_\{\\mathrm\{grad\}\}=\\sum\_\{c,c^\{\\prime\}=1\}^\{C\}\\pi\_\{c\}\\pi\_\{c^\{\\prime\}\}\\langle g\_\{c\},g\_\{c^\{\\prime\}\}\\rangle\(\\mu\_\{c\}\-\\bar\{\\mu\}\)\(\\mu\_\{c^\{\\prime\}\}\-\\bar\{\\mu\}\)^\{\\top\}\.If the gradient\-weighted class geometry satisfies
Hgrad=γΣtaskH\_\{\\mathrm\{grad\}\}=\\gamma\\Sigma\_\{\\mathrm\{task\}\}for some task\-dependentγ\>0\\gamma\>0, then
‖GU‖F2=s2γTr\(U⊤ΣtaskU\)\.\\\|G\_\{U\}\\\|\_\{F\}^\{2\}=s^\{2\}\\gamma\\operatorname\{Tr\}\\left\(U^\{\\top\}\\Sigma\_\{\\mathrm\{task\}\}U\\right\)\.
###### Proof\.
Differentiating through the adapter atB=0B=0gives
GU=s𝔼\[q\(x,y\)\(U⊤h\(x\)\)⊤\]\.G\_\{U\}=s\\,\\mathbb\{E\}\\left\[q\(x,y\)\(U^\{\\top\}h\(x\)\)^\{\\top\}\\right\]\.Conditioning ony=cy=cand expandingq=gc\+ξq=g\_\{c\}\+\\xiandh=μc\+εh=\\mu\_\{c\}\+\\varepsilongives
𝔼\[qh⊤∣y=c\]=gcμc⊤,\\mathbb\{E\}\[qh^\{\\top\}\\mid y=c\]=g\_\{c\}\\mu\_\{c\}^\{\\top\},because the conditional residual means and cross\-covariance vanish\. Hence
GU=s∑c=1Cπcgc\(U⊤μc\)⊤\.G\_\{U\}=s\\sum\_\{c=1\}^\{C\}\\pi\_\{c\}g\_\{c\}\(U^\{\\top\}\\mu\_\{c\}\)^\{\\top\}\.The centering condition on the class\-gradient codes permits subtraction of the global representation mean:
GU=s∑c=1Cπcgc\(U⊤\(μc−μ¯\)\)⊤\.G\_\{U\}=s\\sum\_\{c=1\}^\{C\}\\pi\_\{c\}g\_\{c\}\\left\(U^\{\\top\}\(\\mu\_\{c\}\-\\bar\{\\mu\}\)\\right\)^\{\\top\}\.Expanding the squared Frobenius norm and rewriting it as a trace yields
‖GU‖F2=s2Tr\(U⊤HgradU\)\.\\\|G\_\{U\}\\\|\_\{F\}^\{2\}=s^\{2\}\\operatorname\{Tr\}\\left\(U^\{\\top\}H\_\{\\mathrm\{grad\}\}U\\right\)\.The special caseHgrad=γΣtaskH\_\{\\mathrm\{grad\}\}=\\gamma\\Sigma\_\{\\mathrm\{task\}\}gives the final equality\.
For the loss\-decrease statement, a Taylor expansion atB=0B=0gives
ℒ\(−ηGU,U\)=ℒ\(0,U\)−η‖GU‖F2\+O\(η2‖GU‖F2\)\.\\mathcal\{L\}\(\-\\eta G\_\{U\};U\)=\\mathcal\{L\}\(0;U\)\-\\eta\\\|G\_\{U\}\\\|\_\{F\}^\{2\}\+O\\\!\\left\(\\eta^\{2\}\\\|G\_\{U\}\\\|\_\{F\}^\{2\}\\right\)\.∎
#### Interpretation\.
The most general quantity in the proposition isHgradH\_\{\\mathrm\{grad\}\}, not the unweighted between\-class scatter\. The reduction toΣtask\\Sigma\_\{\\mathrm\{task\}\}requires the stated proportionality assumption\. The proposition is therefore a local explanatory calculation and not a convergence theorem\. It takesA=U⊤A=U^\{\\top\}\. For the implemented initializationA=cR⊤Q⊤A=cR^\{\\top\}Q^\{\\top\}of Eq\. \([7](https://arxiv.org/html/2611.23428#S4.E7)\), the rotationRRcancels and‖G‖F2\\\|G\\\|\_\{F\}^\{2\}gains a factorc2c^\{2\}, so its dependence onTr\(Q⊤ΣtaskQ\)\\operatorname\{Tr\}\(Q^\{\\top\}\\Sigma\_\{\\mathrm\{task\}\}Q\)is unchanged\. The proposition uses population priorsπc\\pi\_\{c\}, and the empiricalΣtask\\Sigma\_\{\\mathrm\{task\}\}of Section[3](https://arxiv.org/html/2611.23428#S3)replaces them withnc/Nn\_\{c\}/N\.
IfUUis sampled uniformly from the Grassmannian of rank\-rrsubspaces and is independent of a fixed positive\-semidefinite task matrix, then
𝔼U\[Tr\(U⊤ΣtaskU\)Tr\(Σtask\)\]=rd\.\\mathbb\{E\}\_\{U\}\\left\[\\frac\{\\operatorname\{Tr\}\(U^\{\\top\}\\Sigma\_\{\\mathrm\{task\}\}U\)\}\{\\operatorname\{Tr\}\(\\Sigma\_\{\\mathrm\{task\}\}\)\}\\right\]=\\frac\{r\}\{d\}\.For BERT\-base withr=8r=8andd=768d=768, this expectation is approximately0\.0100\.010\. The largest measured target\-token alignment is approximatelyρ=0\.84\\rho=0\.84, as reported in Appendix[E\.7](https://arxiv.org/html/2611.23428#A5.SS7)\.
### B\.4Output Preservation Under the Orthogonal Pivot
###### Proposition 2\(Exact preservation under orthogonal reparameterization\)\.
LetA∈ℝr×dA\\in\\mathbb\{R\}^\{r\\times d\}have full row rank and letB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\}\. LetAnew∈ℝr×dA^\{\\text\{new\}\}\\in\\mathbb\{R\}^\{r\\times d\}be an orthonormal row basis forrow\(A\)\\text\{row\}\(A\), so that
Anew\(Anew\)⊤=IrA^\{\\text\{new\}\}\(A^\{\\text\{new\}\}\)^\{\\top\}=I\_\{r\}and
row\(Anew\)=row\(A\)\.\\text\{row\}\(A^\{\\text\{new\}\}\)=\\text\{row\}\(A\)\.Define
Bnew=BA\(Anew\)⊤\.B^\{\\text\{new\}\}=BA\(A^\{\\text\{new\}\}\)^\{\\top\}\.Then
BnewAnew=BA\.B^\{\\text\{new\}\}A^\{\\text\{new\}\}=BA\.Consequently, the LoRA updateΔW=\(α/r\)BA\\Delta W=\(\\alpha/r\)BAis preserved exactly in exact arithmetic\.
###### Proof\.
The matrix
P=\(Anew\)⊤AnewP=\(A^\{\\mathrm\{new\}\}\)^\{\\top\}A^\{\\mathrm\{new\}\}is the orthogonal projector ontorow\(A\)\\operatorname\{row\}\(A\)\. Every row ofAAlies in that subspace, soAP=AAP=A\. Therefore
BnewAnew=BA\(Anew\)⊤Anew=BAP=BA\.B^\{\\mathrm\{new\}\}A^\{\\mathrm\{new\}\}=BA\(A^\{\\mathrm\{new\}\}\)^\{\\top\}A^\{\\mathrm\{new\}\}=BAP=BA\.∎
Because the construction depends only on the current full\-row\-rank adapter factors, the pivot can follow any initializer, whether PCA\-, weight\-, activation\-, or gradient\-based\. In finite precision, the equality is affected by QR and matrix\-multiplication error\. Appendix[F\.6](https://arxiv.org/html/2611.23428#A6.SS6)reports these errors for the classification and reasoning runs\.
ForG=∇ΔWℒG=\\nabla\_\{\\Delta W\}\\mathcal\{L\}and, up to the factorα/r\\alpha/r,ΔW=BA\\Delta W=BA,
∇Bℒ=GA⊤,∇Aℒ=B⊤G\.\\nabla\_\{B\}\\mathcal\{L\}=GA^\{\\top\},\\qquad\\nabla\_\{A\}\\mathcal\{L\}=B^\{\\top\}G\.TheBB\-gradient geometry therefore depends onAA⊤AA^\{\\top\}\. The pivot replaces the current row coordinates with an orthonormal basis, makingAA⊤=IrAA^\{\\top\}=I\_\{r\}while preservingBABA\. Adam and finite\-step SGD are not generally invariant to this factor reparameterization, so subsequent optimization can change even though the adapter output is unchanged at the pivot\.
## Appendix CExperimental Details
Code, configurations, sampled\-example manifests, result summaries, selected run artifacts, and available prediction artifacts are released at[https://github\.com/Nischal1011/pivotlora](https://github.com/Nischal1011/pivotlora)\.
#### Software and hardware\.
The BERT\-base and GPT\-2 experiments were run on a single NVIDIA T4 with 16 GB of memory, while decoder\-scale classification used a single NVIDIA L4 with 24 GB\. Generated\-answer reasoning used one NVIDIA GeForce RTX 5090 with 31\.8 GB of memory, PyTorch2\.9\.0\+cu128, and CUDA 12\.8, with the base model in bf16 and the adapter factors in float32\.
#### Reproducibility\.
The classification implementation enables deterministic PyTorch and cuDNN settings\. Main classification results use seeds 42, 43, and 44 with approximately class\-balanced sampling unless otherwise noted\. Deterministic configuration reduces avoidable variation but does not guarantee bitwise identity across different hardware, driver, and library versions\.
#### Asset licenses\.
BERT\-base is released under Apache\-2\.0, GPT\-2 under a modified MIT license, Llama\-2\-7B under the Llama 2 Community License, and Llama\-3\.1\-8B under the Llama 3\.1 Community License; we accessed the Llama checkpoints after accepting these terms\. GSM8K and StrategyQA are released under MIT, BoolQ under CC BY\-SA 3\.0, QNLI \(derived from SQuAD\[[Rajpurkar et al\., 2016](https://arxiv.org/html/2611.23428#bib.bib24)\]\) under CC BY\-SA 4\.0, MNLI\[[Williams et al\., 2018](https://arxiv.org/html/2611.23428#bib.bib23)\]under the OANC terms with CC BY\-SA portions, and CNN/DailyMail under Apache\-2\.0 as distributed on Hugging Face\. SST\-2\[[Socher et al\., 2013](https://arxiv.org/html/2611.23428#bib.bib21)\], CoLA\[[Warstadt et al\., 2019](https://arxiv.org/html/2611.23428#bib.bib22)\], RTE\[[Dagan et al\., 2006](https://arxiv.org/html/2611.23428#bib.bib25)\], AG News, and XSum are distributed for research use without an explicit license\. All assets are used only for research evaluation, and no model weights or dataset contents are redistributed with our code\.
#### Adapter targets\.
For BERT\-base, we adaptattention\.self\.queryandattention\.self\.value, giving 24 adapters\. For Llama\-2\-7B and Llama\-3\.1\-8B classification, we adaptself\_attn\.q\_projandself\_attn\.v\_proj, giving 64 adapters\. For GPT\-2, we adaptattn\.c\_projandmlp\.c\_fc\.
#### BERT classification details\.
The task\-specific linear classification head is trained jointly with the adapter parameters\. Its initialization is matched across methods and seeds\. The reported adapter parameter count excludes the classification head\. Few\-shot examples are sampled from the training split\. Evaluation uses the validation split when available, the matched validation split for MNLI, and the test split for AG News in the dataset distribution used by our implementation\.
#### Decoder\-scale classification\.
Decoder\-scale experiments use sequence\-classification models with a task\-specific head trained jointly with the adapters\. Sentence\-pair fields are concatenated before tokenization\. The activation source is the mean of each adapter’s inputs over all positions of the padded input sequence, computed in one frozen forward pass\. EVA provides an activation\-based comparison, although it differs from PivotLoRA in components beyond activation pooling\.
#### Few\-shot sampling convention\.
The BERT columns labeled 16, 32, 64, 128, and 256 denote the total number of labeled examplesNN, sampled approximately class\-balanced\. Thus, aCC\-class task contains approximatelyN/CN/Cexamples per class\. The PCA covariance is estimated from allNNexamples\.
#### Baseline configurations\.
PiSSA\[[Meng et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib2)\]uses the top\-rrSVD ofW0W\_\{0\}with a frozen residual andαLoRA=r\\alpha\_\{\\mathrm\{LoRA\}\}=r\. LoRAM\[[Zhang et al\., 2025b](https://arxiv.org/html/2611.23428#bib.bib9)\]estimatesσmax\\sigma\_\{\\max\}with 10 power iterations, constructs a fixed\-seed random orthogonal basis, and scales it bylog\(1\+σmax\)\\log\(1\+\\sigma\_\{\\max\}\)\. LoRA\-GA\[[Wang et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib4)\]uses an SVD of a one\-step gradient with residual subtraction\. LoRA\-One\[[Zhang et al\., 2025a](https://arxiv.org/html/2611.23428#bib.bib5)\]uses balanced SVD scaling without its optional Scaled\-Adam preconditioner\. EVA\[[Paischer et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib7)\]uses all\-token activation SVD\. We disable adaptive rank redistribution and setρEVA=1\.0\\rho\_\{\\mathrm\{EVA\}\}=1\.0, so each adapted layer uses rankr=8r=8\. LoRA\+\[[Hayou et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib10)\]uses Kaiming initialization andηB=16ηA\\eta\_\{B\}=16\\eta\_\{A\}\. DoRA\[[Liu et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib11)\]uses its magnitude\-direction parameterization\. AdaLoRA\[[Zhang et al\., 2023](https://arxiv.org/html/2611.23428#bib.bib12)\]uses SVD triplets with rank pruning\. SRLoRA\[[Yang et al\., 2025](https://arxiv.org/html/2611.23428#bib.bib13)\]fuses rank\-one pairs everymax\(5,⌊T/10⌋\)\\max\(5,\\lfloor T/10\\rfloor\)optimizer steps, whereTTis the total number of training steps\. The LoRA\-DA implementation uses diagonal Fisher estimation with residual subtraction\.
#### Baseline fidelity\.
In our classification implementation, every supplied initialization ofAA, including those of PiSSA, LoRA\-GA, LoRA\-One, LoRAM, and EVA, passes through the same LoRA layer, which multiplies it by5/d\\sqrt\{5/d\}\(Eq\. \([7](https://arxiv.org/html/2611.23428#S4.E7)\)\)\. Because this rescales the baselines’ intended initialization magnitudes, we audited the two baselines for which magnitude is most central atN=64N=64with three seeds \(Table[5](https://arxiv.org/html/2611.23428#A3.T5)\)\. The reference PEFT implementation of PiSSA matches our implementation to within 0\.012 on all three tasks\. Restoring LoRAM’s intendedlog\(1\+σmax\)\\log\(1\+\\sigma\_\{\\max\}\)magnitude changes its score by\+0\.019\+0\.019,−0\.017\-0\.017, and−0\.035\-0\.035, and it remains below vanilla LoRA on every task\. The rescaling therefore does not change the baseline conclusions in Table[3](https://arxiv.org/html/2611.23428#S5.T3), and the as\-released runs below leave PivotLoRA with the highest mean at everyNN\.
Table 5:Baseline\-fidelity audit atN=64N=64, mean over three seeds\. Each block is a separate matched run; compare within blocks\. CoLA uses Matthews correlation\.MethodAG NewsSST\-2CoLALoRA\.634\.633\.183PiSSA \(our implementation\)\.477\.552\.099PiSSA \(reference PEFT\)\.472\.548\.087LoRA\.636\.639\.201LoRAM \(our implementation\)\.476\.597\.141LoRAM \(intended magnitude\)\.495\.580\.106
#### Baselines as released\.
To check that the shared rescaling does not favor PivotLoRA, we also ran LoRAM, LoRA\-GA, LoRA\-One, and SRLoRA as released, following each method’s official initialization and scaling, and SRLoRA’s published schedule, without the shared5/d\\sqrt\{5/d\}factor, together with vanilla LoRA under the standard PEFT initialization \(A∈ℝr×dA\\in\\mathbb\{R\}^\{r\\times d\},‖A‖F≈r/3\\\|A\\\|\_\{F\}\\approx\\sqrt\{r/3\}\)\. All runs use the matched BERT protocol below with three seeds \(Table[6](https://arxiv.org/html/2611.23428#A3.T6)\)\. LoRA\-GA as released is stronger than our version, but PivotLoRA keeps the highest mean at everyNNagainst every baseline in this set\.
Table 6:BERT\-base baselines run as released, mean over six tasks and three seeds\. The last column counts the 30 task–NNsettings in which PivotLoRA has the higher mean\.Method163264128256PivotLoRA aheadLoRA \(as in Table[3](https://arxiv.org/html/2611.23428#S5.T3)\)\.402\.428\.488\.566\.61829/30LoRA \(standard PEFT initialization\)\.385\.391\.438\.501\.59928/30LoRAM \(as released\)\.390\.400\.433\.487\.57329/30LoRA\-GA \(as released\)\.427\.452\.523\.583\.63624/30LoRA\-One \(as released\)\.409\.417\.462\.575\.62128/30SRLoRA \(as released\)\.389\.394\.427\.478\.55530/30PivotLoRA\.451\.500\.575\.606\.641–
#### Matched BERT protocol\.
The main BERT comparison uses rankr=8r=8, AdamW, 15 epochs, batch size 16, query and value adapters, and a shared base learning\-rate anchor ofηA=2×10−4\\eta\_\{A\}=2\\times 10^\{\-4\}\. Unless a baseline’s standard parameterization requires otherwise,αLoRA=16\\alpha\_\{\\mathrm\{LoRA\}\}=16\. The learning\-rate sweeps in Appendix[F\.9](https://arxiv.org/html/2611.23428#A6.SS9)test sensitivity to this anchor\. Every method is evaluated after each epoch, and we report each run’s best evaluation score over the 15 epochs, using the same rule for all methods; CoLA’s Matthews correlation is floored at 0\.
### C\.1Classification Learning\-Rate Allocation
Here,ηA\\eta\_\{A\}denotes the base learning\-rate anchor\. Methods with shared factor\-wise rates apply this rate directly to both LoRA factors, whereas methods with differential rates derive theirBB\-factor rates from it\. At initialization, TT\-PCA and PivotLoRA assign the rank\-iicoordinate ofBBa learning rate proportional to the corresponding PCA eigenvalue:
ηB,i=λiλ¯λratioηA,λ¯=1r∑j=1rλj,λratio=16\.\\eta\_\{B,i\}=\\frac\{\\lambda\_\{i\}\}\{\\bar\{\\lambda\}\}\\,\\lambda\_\{\\mathrm\{ratio\}\}\\eta\_\{A\},\\qquad\\bar\{\\lambda\}=\\frac\{1\}\{r\}\\sum\_\{j=1\}^\{r\}\\lambda\_\{j\},\\qquad\\lambda\_\{\\mathrm\{ratio\}\}=16\.\(12\)Normalizing byλ¯\\bar\{\\lambda\}makes the averageBB\-to\-AAlearning\-rate ratio equal to 16\. This allocation is an optimizer heuristic and is not part of the activation\-subspace definition\. The controls in Appendix[F\.10](https://arxiv.org/html/2611.23428#A6.SS10)separate the selected directions from the eigenvalue\-to\-coordinate rate assignment\.
TT\-PCA uses these rates throughout training\. PivotLoRA uses them except for the 10 optimizer steps after the pivot, during which a commonBBrate rises fromηA\\eta\_\{A\}to16ηA16\\eta\_\{A\}as a precaution, because the pivot resets the Adam state ofAAwhile the state ofBBstill refers to the previous coordinates\. Appendix[F\.8](https://arxiv.org/html/2611.23428#A6.SS8)shows that this precaution is not needed\. Post\-pivot learning\-rate policies differ by at most 0\.011 in average score, and the pivot with no learning\-rate change performs as well\. The reasoning experiments instead use a single learning rate of2×10−42\\times 10^\{\-4\}for both factors, with no differential or eigenvalue\-scaled rates and no post\-pivot change\.
### C\.2TT\-LDA Ablation
TT\-LDA uses the same prediction\-position representations as TT\-PCA but constructs supervised linear\-discriminant directions\. Because aCC\-class problem supplies at mostC−1C\-1discriminant directions, residual PCA directions fill the unused coordinates whenC−1<rC\-1<r\. The combined basis is orthogonalized to rankr=8r=8\. TT\-LDA uses the uniform rateηB=16ηA\\eta\_\{B\}=16\\eta\_\{A\}, with no eigenvalue scaling across ranks\. We use TT\-LDA as an ablation of the activation statistic, not as the default method\. Comparative results appear in Appendix[F\.9](https://arxiv.org/html/2611.23428#A6.SS9)\.
### C\.3Classification Pivot Configuration
The primary classification configuration uses pivot epochK=3K=3\. At the pivot, reduced QR is applied toA⊤A^\{\\top\}in float32, andBBis re\-encoded according to Eq\. \([10](https://arxiv.org/html/2611.23428#S4.E10)\)\. Adam first\- and second\-moment states forAAare reset\. The state forBBis retained, and its learning rate is handled as described in Appendix[C\.1](https://arxiv.org/html/2611.23428#A3.SS1)\. The reset policy is an implementation choice and is separate from the algebraic output\-preservation guarantee\.
Algorithm 1Classification PivotLoRA1:Frozen model
ℳ\\mathcal\{M\}, labeled data
𝒟\\mathcal\{D\}, rank
rr, pivot epoch
KK
2:Collect prediction\-position activations
XℓX\_\{\\ell\}at each adapted layer
3:foreach adapted layer
ℓ\\elldo
4:Center
XℓX\_\{\\ell\}and compute its top\-
rrPCA basis
5:Initialize
row\(Aℓ\)\\operatorname\{row\}\(A\_\{\\ell\}\)from the PCA basis
6:Set
Bℓ←0B\_\{\\ell\}\\leftarrow 0
7:Assign the
BℓB\_\{\\ell\}coordinate rates using Eq\. \([12](https://arxiv.org/html/2611.23428#A3.E12)\)
8:endfor
9:Train through epoch
KK
10:foreach adapted layer
ℓ\\elldo
11:
Aold←AℓA\_\{\\mathrm\{old\}\}\\leftarrow A\_\{\\ell\}and
Bold←BℓB\_\{\\mathrm\{old\}\}\\leftarrow B\_\{\\ell\}
12:
Q,R←QR\(Aold⊤\)Q,R\\leftarrow\\operatorname\{QR\}\(A\_\{\\mathrm\{old\}\}^\{\\top\}\)
13:
Anew←Q⊤A\_\{\\mathrm\{new\}\}\\leftarrow Q^\{\\top\}
14:
Bnew←BoldAoldAnew⊤B\_\{\\mathrm\{new\}\}\\leftarrow B\_\{\\mathrm\{old\}\}A\_\{\\mathrm\{old\}\}A\_\{\\mathrm\{new\}\}^\{\\top\}
15:Replace
\(Aℓ,Bℓ\)\(A\_\{\\ell\},B\_\{\\ell\}\)by
\(Anew,Bnew\)\(A\_\{\\mathrm\{new\}\},B\_\{\\mathrm\{new\}\}\)
16:Reset the Adam state for
AℓA\_\{\\ell\}
17:endfor
18:Ramp the
BBrate from
ηA\\eta\_\{A\}to
16ηA16\\eta\_\{A\}over 10 steps, then resume Eq\. \([12](https://arxiv.org/html/2611.23428#A3.E12)\)
19:Train for the remaining epochs
#### GPT\-2 summarization configuration\.
The loss is applied only to the supervised summary, and the tied language\-modeling head remains frozen\. We cross two activation sources with two learning\-rate recipes\. The sources are the mean over all positions of the padded sequence and the mean over response positions \(those predicting supervised summary tokens\)\. The recipes are the classification recipe of Eq\. \([12](https://arxiv.org/html/2611.23428#A3.E12)\) withλratio=16\\lambda\_\{\\mathrm\{ratio\}\}=16and a single rate of2×10−42\\times 10^\{\-4\}for both factors, as in the reasoning experiments \(Appendix[E\.4](https://arxiv.org/html/2611.23428#A5.SS4)\)\.
## Appendix DGenerated\-Answer Reasoning Details
We evaluate response\-position adaptation on GSM8K and StrategyQA\. LoRA and PivotLoRA use matched data, training, and evaluation protocols, and PivotLoRA additionally uses activation\-based initialization and one output\-preserving pivot\. Exact commands and lower\-level implementation details are provided with the released code\.
StrategyQA was selected with a screening step fixed in advance that ran only the base model and LoRA\. A candidate dataset was kept if LoRA was clearly above chance, followed the trained answer format in at least 90% of generations, and rarely hit the generation limit\. No PivotLoRA run was made on a candidate before it passed\. We also considered MATH\-500 but stopped at this stage\. Across three pipeline checks, each method solved between 1 and 10 of 50 problems in our 128\-example fine\-tuning setting, and fewer than 80% of trained\-model generations followed the boxed\-answer format, so a comparison would have been dominated by noise and format failures\.
### D\.1Data and Response\-Position Activations
GSM8K contains 7,473 training examples and 1,319 test examples, and StrategyQA \(theChilleD/StrategyQAsplit\) contains 1,603 training examples and 687 test examples\. For StrategyQA, the one training question that also appears in the test split is removed from the training pool\. For each dataset, we sampleN=128N=128training examples without replacement using seeds\{42,43,44\}\\\{42,43,44\\\}\. LoRA and PivotLoRA use the same subset at each seed\.
The GSM8K prompt is
```
Question: {question}
Answer:
```
with a single space followed by the dataset’s original answer field as the completion\. StrategyQA uses the GSM8K prompt with completion
```
{facts}
#### {yes|no}
```
wherefactsis the dataset’s supporting\-facts field\.
Each prompt ends atAnswer:and each completion begins with a space, so the prompt used at inference is an exact token\-level prefix of every training sequence, which the implementation asserts for every training example\. Generation therefore starts from the same token boundary seen in training\.
Prompt tokens are excluded from the language\-modeling loss, whereas completion tokens and the appended EOS token are supervised\. Sequences are jointly tokenized and truncated to a maximum length of 512 tokens\. Because hidden statettpredicts tokent\+1t\+1, the response\-predictor positions for examplennare
𝒫n=\{t:Mn,t\+1=1\},\\mathcal\{P\}\_\{n\}=\\\{t:M\_\{n,t\+1\}=1\\\},whereMMdenotes the completion mask\. At adapted layerℓ\\ell, the activation used for PCA is
xℓ,n=1\|𝒫n\|∑t∈𝒫nhℓ,n,t\.x\_\{\\ell,n\}=\\frac\{1\}\{\|\\mathcal\{P\}\_\{n\}\|\}\\sum\_\{t\\in\\mathcal\{P\}\_\{n\}\}h\_\{\\ell,n,t\}\.The resulting example\-level activation matrix is centered before PCA\. Averaging within each example prevents response length from determining its weight in the covariance estimate\.
### D\.2Training and Pivot Configuration
Table 7:Configuration for the reasoning experiments, shared by both datasets\.FieldValueCheckpointmeta\-llama/Llama\-3\.1\-8BTarget modulesq\_proj,v\_projAdapted modules64Rankr=8r=8LoRA scalingαLoRA=16\\alpha\_\{\\mathrm\{LoRA\}\}=16OptimizerAdamWLearning rate2×10−42\\times 10^\{\-4\}Weight decay0\.01ScheduleLinear decay, no warm\-upEpochs15Per\-device batch size4Gradient accumulation2Effective batch size8Optimizer steps240Maximum sequence length512Base\-model precisionbf16Adapter precisionfloat32Pivot epoch3Pivot step48The PCA basis is orthogonalized and scaled by5/d\\sqrt\{5/d\}, withd=4096d=4096, before assignment to the LoRA input factorAA, with no random rotation\. PCA is computed in float32\. PivotLoRA performs one pivot at epoch 3, corresponding to global step 48, using float64 QR, and the transformed factors are stored in float32\. All 64 query and value adapters are transformed\.
Immediately after the pivot, the AdamW states for bothAAandBBare reset, while the learning\-rate schedule continues from the current global step\. Resetting both states differs from the classification policy \(Appendix[C\.3](https://arxiv.org/html/2611.23428#A3.SS3)\)\.
The5/d\\sqrt\{5/d\}basis scale givesAAabout17×17\\timessmaller Frobenius norm than vanilla Kaiming\-uniform initialization, so it is not norm\-matched\. These experiments therefore evaluate the complete initialization\-and\-pivot procedure, not direction in isolation\.
### D\.3Generation and Evaluation
Evaluation uses greedy decoding with batch size 1 and at most 512 new tokens, terminating earlier at EOS\. The model uses EOS for padding andsdpaattention\.
The evaluator first extracts a numeric value following the final\#\#\#\#delimiter\. If none is present, it uses the final numeric value in the generated response\. Predictions and references are normalized as numeric strings and compared by exact equality\. Signed integers, comma\-grouped numbers, and decimals are supported; fractions and percentages are not converted\. Outputs without a valid numeric value are counted as incorrect\. For StrategyQA, the evaluator instead extracts the first standaloneyesornoafter the final\#\#\#\#, falling back to the last standaloneyesornoin the response, and compares it with the label\.
We additionally report format compliance, the fraction of generations that follow the trained answer format, which ends in\#\#\#\# <number\>for GSM8K and in\#\#\#\# yesor\#\#\#\# nofor StrategyQA\. As a reference, the frozen base model without adapters is evaluated once with the same prompt, decoding, and scorer\.
### D\.4Per\-Seed Results
Table 8:GSM8K results\. Accuracy is numeric exact match on the 1,319\-example test set\. Format is the percentage of generations following the trained answer format\. The base model has no adapter and is evaluated once\.MethodSeedCorrectAccuracy \(%\)Parse rate \(%\)Format \(%\)Base model–33825\.63100\.000\.0LoRA4237628\.51100\.0098\.0LoRA4338128\.89100\.0096\.0LoRA4437828\.66100\.0095\.8PivotLoRA4242131\.92100\.0098\.2PivotLoRA4347636\.09100\.0097\.6PivotLoRA4441831\.69100\.0097\.4PivotLoRA improves on LoRA at every seed, by\+3\.41\+3\.41,\+7\.20\+7\.20, and\+3\.03\+3\.03points \(mean\+4\.55±2\.31\+4\.55\\pm 2\.31, sample SD over seeds\), and even the smallest gain exceeds LoRA’s own seed\-to\-seed spread \(±0\.19\\pm 0\.19\)\. Pooled over the three seeds, PivotLoRA alone answers 655 test items correctly against 475 for LoRA alone \(exact McNemarp=9\.5×10−8p=9\.5\\times 10^\{\-8\}\), and the test is significant at each seed \(p≤0\.036p\\leq 0\.036\)\.
Table 9:StrategyQA results\. Accuracy is yes/no exact match on the 687\-example test split\. Columns as in Table[8](https://arxiv.org/html/2611.23428#A4.T8)\.MethodSeedCorrectAccuracy \(%\)Parse rate \(%\)Format \(%\)Base model–37454\.4485\.150\.0LoRA4243963\.9098\.1195\.2LoRA4345766\.5299\.8599\.4LoRA4442662\.0199\.4296\.7PivotLoRA4247669\.2999\.8599\.6PivotLoRA4345766\.5299\.7199\.4PivotLoRA4446367\.3999\.8598\.0PivotLoRA improves on LoRA by\+5\.39\+5\.39points at two seeds and ties at the third \(mean\+3\.59±3\.11\+3\.59\\pm 3\.11\)\. The tie does not reflect identical outputs, since each method alone answers 98 questions correctly\. Pooled over the three seeds, PivotLoRA alone answers 350 test items correctly against 276 for LoRA alone \(exact McNemarp=0\.004p=0\.004\)\.
## Appendix EExtended Experimental Results
### E\.1Complete BERT\-Base Results
Tables[10](https://arxiv.org/html/2611.23428#A5.T10)–[16](https://arxiv.org/html/2611.23428#A5.T16)report mean and sample standard deviation over three seeds\. The six primary tasks contain 30 task–NNsettings\. BoolQ is an auxiliary classification task used in the alignment analysis\. All methods follow the shared benchmark protocol together with the method\-specific configurations documented in Appendix[C](https://arxiv.org/html/2611.23428#A3)\.
Table 10:SST\-2 accuracy, reported as mean±\\pmstandard deviation over three seeds\. Columns denote the total number of labeled examplesNN\.Method163264128256LoRA\.524±\\pm\.011\.546±\\pm\.037\.652±\\pm\.014\.772±\\pm\.006\.844±\\pm\.012DoRA\.528±\\pm\.014\.544±\\pm\.036\.647±\\pm\.018\.761±\\pm\.010\.841±\\pm\.008AdaLoRA\.502±\\pm\.019\.500±\\pm\.008\.548±\\pm\.016\.580±\\pm\.026\.633±\\pm\.019PiSSA\.503±\\pm\.023\.500±\\pm\.008\.548±\\pm\.027\.665±\\pm\.058\.823±\\pm\.002LoRA\-GA\.503±\\pm\.018\.500±\\pm\.007\.565±\\pm\.033\.664±\\pm\.077\.849±\\pm\.004LoRA\-One\.497±\\pm\.015\.501±\\pm\.007\.565±\\pm\.032\.711±\\pm\.059\.854±\\pm\.010LoRA\+\.551±\\pm\.033\.569±\\pm\.042\.538±\\pm\.038\.509±\\pm\.000\.509±\\pm\.000EVA\.503±\\pm\.017\.500±\\pm\.007\.561±\\pm\.031\.683±\\pm\.032\.855±\\pm\.010LoRAM\.498±\\pm\.015\.500±\\pm\.007\.568±\\pm\.030\.702±\\pm\.060\.849±\\pm\.005SRLoRA\.524±\\pm\.011\.536±\\pm\.036\.632±\\pm\.014\.750±\\pm\.014\.843±\\pm\.011TT\-PCA\.520±\\pm\.024\.530±\\pm\.011\.755±\\pm\.007\.817±\\pm\.003\.862±\\pm\.007TT\-LDA\.523±\\pm\.020\.542±\\pm\.014\.783±\\pm\.007\.828±\\pm\.014\.859±\\pm\.011PivotLoRA\.543±\\pm\.038\.633±\\pm\.018\.795±\\pm\.016\.823±\\pm\.017\.850±\\pm\.010Table 11:CoLA Matthews correlation, reported as mean±\\pmstandard deviation over three seeds\.Method163264128256LoRA\.081±\\pm\.056\.131±\\pm\.047\.163±\\pm\.046\.264±\\pm\.072\.407±\\pm\.010DoRA\.085±\\pm\.060\.128±\\pm\.041\.171±\\pm\.042\.257±\\pm\.070\.393±\\pm\.007AdaLoRA\.096±\\pm\.055\.081±\\pm\.040\.100±\\pm\.029\.152±\\pm\.033\.183±\\pm\.023PiSSA\.090±\\pm\.064\.111±\\pm\.040\.111±\\pm\.083\.206±\\pm\.059\.338±\\pm\.022LoRA\-GA\.086±\\pm\.061\.097±\\pm\.045\.103±\\pm\.078\.214±\\pm\.059\.382±\\pm\.003LoRA\-One\.087±\\pm\.060\.089±\\pm\.039\.111±\\pm\.062\.240±\\pm\.067\.387±\\pm\.007LoRA\+\.071±\\pm\.023\.105±\\pm\.041\.038±\\pm\.012\.000±\\pm\.000\.000±\\pm\.000EVA\.089±\\pm\.061\.095±\\pm\.036\.101±\\pm\.070\.243±\\pm\.058\.387±\\pm\.017LoRAM\.089±\\pm\.061\.085±\\pm\.036\.112±\\pm\.070\.239±\\pm\.058\.390±\\pm\.017SRLoRA\.081±\\pm\.058\.129±\\pm\.030\.161±\\pm\.051\.240±\\pm\.065\.405±\\pm\.008TT\-PCA\.103±\\pm\.044\.137±\\pm\.046\.274±\\pm\.094\.381±\\pm\.049\.397±\\pm\.019TT\-LDA\.106±\\pm\.036\.150±\\pm\.041\.281±\\pm\.100\.377±\\pm\.071\.412±\\pm\.016PivotLoRA\.124±\\pm\.015\.171±\\pm\.049\.288±\\pm\.098\.365±\\pm\.071\.411±\\pm\.025Table 12:MNLI matched\-validation accuracy, reported as mean±\\pmstandard deviation over three seeds\.Method163264128256LoRA\.350±\\pm\.008\.356±\\pm\.006\.366±\\pm\.003\.356±\\pm\.008\.392±\\pm\.011DoRA\.349±\\pm\.008\.355±\\pm\.006\.366±\\pm\.002\.359±\\pm\.008\.392±\\pm\.012AdaLoRA\.350±\\pm\.004\.354±\\pm\.004\.347±\\pm\.013\.352±\\pm\.009\.356±\\pm\.013PiSSA\.352±\\pm\.003\.354±\\pm\.002\.349±\\pm\.014\.354±\\pm\.009\.383±\\pm\.023LoRA\-GA\.350±\\pm\.003\.356±\\pm\.003\.348±\\pm\.013\.351±\\pm\.009\.397±\\pm\.014LoRA\-One\.350±\\pm\.004\.355±\\pm\.004\.348±\\pm\.013\.357±\\pm\.010\.401±\\pm\.012LoRA\+\.353±\\pm\.012\.355±\\pm\.001\.352±\\pm\.019\.355±\\pm\.000\.355±\\pm\.000EVA\.350±\\pm\.004\.356±\\pm\.003\.347±\\pm\.014\.352±\\pm\.008\.397±\\pm\.015LoRAM\.350±\\pm\.004\.356±\\pm\.003\.349±\\pm\.014\.356±\\pm\.008\.404±\\pm\.015SRLoRA\.349±\\pm\.007\.356±\\pm\.005\.367±\\pm\.003\.357±\\pm\.007\.390±\\pm\.009TT\-PCA\.349±\\pm\.005\.359±\\pm\.007\.367±\\pm\.003\.378±\\pm\.011\.424±\\pm\.025TT\-LDA\.352±\\pm\.007\.357±\\pm\.005\.366±\\pm\.003\.379±\\pm\.007\.425±\\pm\.024PivotLoRA\.356±\\pm\.010\.362±\\pm\.008\.367±\\pm\.005\.389±\\pm\.009\.454±\\pm\.041Table 13:QNLI accuracy, reported as mean±\\pmstandard deviation over three seeds\.Method163264128256LoRA\.519±\\pm\.037\.521±\\pm\.040\.555±\\pm\.042\.626±\\pm\.061\.662±\\pm\.040DoRA\.517±\\pm\.039\.519±\\pm\.040\.556±\\pm\.042\.622±\\pm\.055\.663±\\pm\.035AdaLoRA\.513±\\pm\.029\.514±\\pm\.029\.527±\\pm\.023\.537±\\pm\.029\.546±\\pm\.024PiSSA\.510±\\pm\.027\.512±\\pm\.041\.528±\\pm\.021\.574±\\pm\.035\.616±\\pm\.038LoRA\-GA\.512±\\pm\.027\.514±\\pm\.035\.528±\\pm\.024\.567±\\pm\.045\.641±\\pm\.043LoRA\-One\.511±\\pm\.028\.516±\\pm\.031\.529±\\pm\.023\.574±\\pm\.049\.644±\\pm\.045LoRA\+\.539±\\pm\.033\.576±\\pm\.052\.554±\\pm\.040\.515±\\pm\.012\.505±\\pm\.000EVA\.511±\\pm\.028\.516±\\pm\.031\.532±\\pm\.024\.575±\\pm\.052\.645±\\pm\.050LoRAM\.512±\\pm\.028\.516±\\pm\.031\.530±\\pm\.024\.567±\\pm\.052\.644±\\pm\.050SRLoRA\.512±\\pm\.038\.519±\\pm\.039\.551±\\pm\.040\.618±\\pm\.059\.662±\\pm\.034TT\-PCA\.517±\\pm\.034\.525±\\pm\.031\.574±\\pm\.059\.640±\\pm\.057\.677±\\pm\.024TT\-LDA\.517±\\pm\.035\.528±\\pm\.033\.572±\\pm\.048\.638±\\pm\.055\.681±\\pm\.029PivotLoRA\.533±\\pm\.026\.547±\\pm\.036\.594±\\pm\.049\.645±\\pm\.050\.691±\\pm\.013Table 14:RTE accuracy, reported as mean±\\pmstandard deviation over three seeds\.Method163264128256LoRA\.508±\\pm\.040\.549±\\pm\.036\.551±\\pm\.022\.569±\\pm\.015\.549±\\pm\.003DoRA\.511±\\pm\.040\.546±\\pm\.024\.555±\\pm\.015\.568±\\pm\.018\.540±\\pm\.007AdaLoRA\.510±\\pm\.035\.526±\\pm\.011\.551±\\pm\.022\.540±\\pm\.012\.524±\\pm\.014PiSSA\.505±\\pm\.033\.528±\\pm\.032\.539±\\pm\.019\.555±\\pm\.025\.528±\\pm\.015LoRA\-GA\.516±\\pm\.036\.525±\\pm\.018\.552±\\pm\.016\.554±\\pm\.015\.527±\\pm\.014LoRA\-One\.515±\\pm\.037\.528±\\pm\.017\.561±\\pm\.021\.561±\\pm\.007\.540±\\pm\.013LoRA\+\.545±\\pm\.028\.552±\\pm\.011\.527±\\pm\.000\.527±\\pm\.000\.527±\\pm\.000EVA\.513±\\pm\.037\.534±\\pm\.024\.555±\\pm\.017\.563±\\pm\.013\.539±\\pm\.024LoRAM\.513±\\pm\.037\.530±\\pm\.024\.563±\\pm\.017\.566±\\pm\.013\.551±\\pm\.024SRLoRA\.513±\\pm\.046\.552±\\pm\.031\.539±\\pm\.021\.567±\\pm\.018\.549±\\pm\.009TT\-PCA\.522±\\pm\.051\.563±\\pm\.033\.555±\\pm\.025\.561±\\pm\.022\.552±\\pm\.005TT\-LDA\.516±\\pm\.055\.567±\\pm\.023\.558±\\pm\.020\.561±\\pm\.028\.555±\\pm\.007PivotLoRA\.551±\\pm\.019\.584±\\pm\.017\.569±\\pm\.025\.549±\\pm\.010\.563±\\pm\.012Table 15:AG News accuracy, reported as mean±\\pmstandard deviation over three seeds\.Method163264128256LoRA\.431±\\pm\.016\.466±\\pm\.050\.640±\\pm\.033\.807±\\pm\.006\.855±\\pm\.006DoRA\.432±\\pm\.015\.446±\\pm\.041\.619±\\pm\.016\.801±\\pm\.010\.857±\\pm\.010AdaLoRA\.304±\\pm\.027\.308±\\pm\.016\.368±\\pm\.038\.438±\\pm\.031\.504±\\pm\.028PiSSA\.338±\\pm\.031\.364±\\pm\.007\.479±\\pm\.024\.643±\\pm\.036\.795±\\pm\.024LoRA\-GA\.330±\\pm\.022\.354±\\pm\.012\.466±\\pm\.038\.679±\\pm\.042\.812±\\pm\.030LoRA\-One\.314±\\pm\.025\.338±\\pm\.012\.475±\\pm\.033\.696±\\pm\.055\.808±\\pm\.032LoRA\+\.695±\\pm\.011\.730±\\pm\.044\.619±\\pm\.136\.646±\\pm\.281\.594±\\pm\.123EVA\.317±\\pm\.024\.345±\\pm\.012\.481±\\pm\.040\.702±\\pm\.070\.815±\\pm\.029LoRAM\.311±\\pm\.024\.333±\\pm\.012\.482±\\pm\.040\.672±\\pm\.070\.809±\\pm\.029SRLoRA\.426±\\pm\.021\.446±\\pm\.045\.584±\\pm\.015\.795±\\pm\.013\.851±\\pm\.004TT\-PCA\.478±\\pm\.028\.558±\\pm\.012\.753±\\pm\.041\.858±\\pm\.005\.874±\\pm\.006TT\-LDA\.475±\\pm\.026\.587±\\pm\.015\.769±\\pm\.027\.861±\\pm\.009\.871±\\pm\.008PivotLoRA\.600±\\pm\.049\.705±\\pm\.047\.834±\\pm\.018\.866±\\pm\.004\.878±\\pm\.001Table 16:BoolQ accuracy, reported as mean±\\pmstandard deviation over three seeds\.Method163264128256LoRA\.602±\\pm\.005\.603±\\pm\.010\.551±\\pm\.048\.567±\\pm\.039\.601±\\pm\.019DoRA\.601±\\pm\.005\.606±\\pm\.007\.553±\\pm\.043\.573±\\pm\.032\.596±\\pm\.022AdaLoRA\.593±\\pm\.018\.603±\\pm\.012\.549±\\pm\.030\.542±\\pm\.024\.576±\\pm\.017PiSSA\.597±\\pm\.015\.603±\\pm\.012\.549±\\pm\.029\.563±\\pm\.030\.591±\\pm\.035LoRA\-GA\.596±\\pm\.015\.602±\\pm\.011\.565±\\pm\.006\.562±\\pm\.030\.584±\\pm\.017LoRA\-One\.593±\\pm\.018\.602±\\pm\.011\.538±\\pm\.032\.549±\\pm\.010\.580±\\pm\.017LoRA\+\.591±\\pm\.042\.622±\\pm\.000\.621±\\pm\.001\.622±\\pm\.000\.622±\\pm\.000EVA\.594±\\pm\.017\.603±\\pm\.012\.537±\\pm\.033\.538±\\pm\.014\.579±\\pm\.020LoRAM\.593±\\pm\.018\.603±\\pm\.012\.545±\\pm\.029\.549±\\pm\.009\.576±\\pm\.017SRLoRA\.602±\\pm\.005\.603±\\pm\.010\.549±\\pm\.050\.566±\\pm\.039\.599±\\pm\.021TT\-PCA\.596±\\pm\.016\.601±\\pm\.014\.568±\\pm\.014\.574±\\pm\.006\.612±\\pm\.010TT\-LDA\.596±\\pm\.016\.602±\\pm\.013\.569±\\pm\.016\.567±\\pm\.009\.612±\\pm\.005PivotLoRA\.596±\\pm\.016\.601±\\pm\.014\.567±\\pm\.029\.589±\\pm\.011\.611±\\pm\.010
### E\.2Full Aggregate Rankings
Table[17](https://arxiv.org/html/2611.23428#A5.T17)aggregates the 30 primary BERT task–NNsettings\. PivotLoRA has average rank 1\.83 across all 13 methods and has the strictly highest mean in 19 of 30 settings, counting TT\-PCA and TT\-LDA, and ties for highest on MNLI atN=64N=64\. It exceeds vanilla LoRA in 29 of the 30 settings\.
Table 17:Aggregate statistics across the 30 primary BERT\-base few\-shot settings\. “Wins vs\. PivotLoRA” counts settings in which the listed method has a strictly higher mean than PivotLoRA\.MethodAverage rank \(↓\\downarrow\)Wins vs\. PivotLoRAPivotLoRA \(Ours\)1\.83N/ATT\-LDA3\.135/30TT\-PCA3\.773/30LoRA5\.231/30DoRA5\.831/30SRLoRA6\.931/30LoRA\+8\.435/30LoRAM8\.431/30LoRA\-One8\.472/30EVA8\.472/30LoRA\-GA9\.301/30PiSSA9\.871/30AdaLoRA11\.300/30
### E\.3Decoder\-Scale Classification Results
The decoder\-scale experiments test whether the activation\-subspace pattern extends beyond BERT\-base\. Tables[18](https://arxiv.org/html/2611.23428#A5.T18)and[19](https://arxiv.org/html/2611.23428#A5.T19)report SST\-2 and AG News, while Table[20](https://arxiv.org/html/2611.23428#A5.T20)reports the BoolQ extension\. PivotLoRA ranks first atN=64N=64on SST\-2 and AG News for both backbones\. AtN=256N=256, the differences are smaller and PivotLoRA is not uniformly best\.
EVA is included as an activation\-based baseline under the same per\-layer rank budget\. It treats every token activation as a separate sample, whereas PivotLoRA uses one sequence\-mean vector per example\.
Table 18:Llama\-2\-7B few\-shot classification, reported as mean±\\pmstandard deviation over three seeds\. Bold marks the best mean among methods evaluated in the corresponding setting\.MethodSST\-2 64SST\-2 256AG News 64AG News 256LoRA\.708±\\pm\.051\.924±\\pm\.007\.748±\\pm\.027\.879±\\pm\.004EVA\.713±\\pm\.015\.928±\\pm\.016\.745±\\pm\.027\.879±\\pm\.004PiSSA\.635±\\pm\.058\.922±\\pm\.010\.700±\\pm\.014\.865±\\pm\.010LoRAM\.763±\\pm\.025\.929±\\pm\.014\.742±\\pm\.027\.876±\\pm\.003PivotLoRA\.769±\\pm\.030\.912±\\pm\.020\.784±\\pm\.029\.861±\\pm\.020Table 19:Llama\-3\.1\-8B few\-shot classification, reported as mean±\\pmsample standard deviation over three seeds\. EVA entries atN=64N=64report population standard deviations because per\-seed values were not retained\. Bold marks the best mean among methods evaluated in the corresponding setting\.MethodSST\-2 64SST\-2 256AG News 64AG News 256LoRA\.704±\\pm\.037\.883±\\pm\.030\.759±\\pm\.023\.880±\\pm\.009EVA\.643±\\pm\.050\.909±\\pm\.017\.743±\\pm\.034\.876±\\pm\.009PiSSA\.622±\\pm\.043\.876±\\pm\.030\.715±\\pm\.027\.864±\\pm\.010LoRAM\.662±\\pm\.050\.901±\\pm\.012\.748±\\pm\.047\.872±\\pm\.010PivotLoRA\.709±\\pm\.085\.899±\\pm\.012\.799±\\pm\.001\.870±\\pm\.030Table 20:Llama\-3\.1\-8B classification on BoolQ, reported as mean±\\pmpopulation standard deviation over three seeds \(per\-seed values were not retained\)\.MethodN=64N=64N=256N=256LoRA\.554±\\pm\.025\.576±\\pm\.020PiSSA\.527±\\pm\.023\.536±\\pm\.021LoRAM\.534±\\pm\.032\.573±\\pm\.014PivotLoRA\.602±\\pm\.009\.618±\\pm\.004
### E\.4Summarization Control
GPT\-2 summarization tests whether a larged/Nd/Nalone produces large gains \(Table[21](https://arxiv.org/html/2611.23428#A5.T21); XSum and CNN/DailyMail, ROUGE\-L on 500 validation examples, three seeds, one matched sweep\)\. Under the single learning rate used for reasoning, PivotLoRA exceeds LoRA by only\+0\.2\+0\.2points with the sequence mean and\+0\.3\+0\.3with response positions, winning 8 and 9 of the 12 seed\-level comparisons, respectively\. These gains are much smaller than the classification and reasoning gains at comparabled/Nd/N\. Under the classification recipe, PivotLoRA trails LoRA by 1\.5 to 5\.1 points with either source, so the learning\-rate recipe matters more than the activation source\. Response positions score above the sequence mean in all eight comparisons, by\+0\.1\+0\.1to\+0\.2\+0\.2points under the single rate and by\+0\.2\+0\.2to\+1\.2\+1\.2under the classification recipe\.
Table 21:GPT\-2 few\-shot summarization, ROUGE\-L means over three seeds\. “classification” uses eigenvalue\-scaled rates withλratio=16\\lambda\_\{\\mathrm\{ratio\}\}=16; “single rate” uses2×10−42\\times 10^\{\-4\}for both factors, as in the reasoning experiments\.XSumCNN/DailyMailMethodSource / recipe6425664256LoRA—\.185\.184\.212\.227PivotLoRAsequence mean / classification\.135\.164\.193\.208PivotLoRAresponse / classification\.147\.168\.198\.210PivotLoRAsequence mean / single rate\.188\.191\.209\.228PivotLoRAresponse / single rate\.191\.192\.211\.229
### E\.5Full\-Data Reference
Table[22](https://arxiv.org/html/2611.23428#A5.T22)compares PivotLoRA and LoRA when the full BERT training split is used\. The differences range from−0\.009\-0\.009to\+0\.022\+0\.022in the task’s primary score\.
Table 22:BERT\-base on the full training split using seed 42\. CoLA uses Matthews correlation, and the remaining tasks use accuracy\.MethodSST\-2CoLAMNLIQNLIRTEAG NewsLoRA\.923\.533\.806\.893\.610\.940PivotLoRA\.914\.555\.817\.889\.610\.944Δ\\Delta−\.009\-\.009\+\.022\+\.011−\.004\-\.004\.000\+\.004#### PiSSA under full data\.
Because weight\-spectral initializers such as PiSSA are often motivated by full\-data fine\-tuning, we ran a separate matched single\-seed audit on the full training splits using the reference PEFT implementation of PiSSA\[[Meng et al\., 2024](https://arxiv.org/html/2611.23428#bib.bib2)\]\. LoRA, PiSSA, and PivotLoRA share the data, seed, optimizer, and training budget within this audit \(Table[23](https://arxiv.org/html/2611.23428#A5.T23)\)\. Because it is a separate execution, its LoRA and PivotLoRA values differ slightly from Table[22](https://arxiv.org/html/2611.23428#A5.T22)\. PiSSA is within\+0\.018\+0\.018of LoRA on five tasks and0\.0210\.021below it on CoLA, and PivotLoRA is at or above PiSSA on all six tasks\. With one seed, the three methods are comparable in the full\-data regime\. The audit does not measure convergence speed, where PiSSA reports its advantages\.
Table 23:Single\-seed full\-data audit with the reference PEFT implementation of PiSSA \(seed 42\)\. CoLA uses Matthews correlation; the other tasks use accuracy\.MethodSST\-2CoLAMNLIQNLIRTEAG NewsLoRA\.911\.515\.809\.876\.545\.941PiSSA\.915\.494\.812\.879\.563\.941PivotLoRA\.916\.565\.815\.902\.606\.943
### E\.6Full Fine\-Tuning Reference
Table[24](https://arxiv.org/html/2611.23428#A5.T24)compares PivotLoRA with full fine\-tuning atN=64N=64\. Full fine\-tuning is a strong reference but does not match the adapter parameter budget\. PivotLoRA exceeds the mean full fine\-tuning score on all six tasks while training about370×370\\timesfewer parameters \(approximately 0\.30M\)\.
Table 24:Full fine\-tuning, LoRA, and PivotLoRA on BERT\-base atN=64N=64\. Full fine\-tuning is reported as mean±\\pmstandard deviation over three seeds\. LoRA and PivotLoRA entries are the corresponding three\-seed means from the primary tables\.TaskFull FT \(110M\)LoRA \(0\.30M\)PivotLoRA \(0\.30M\)SST\-2\.700±\\pm\.053\.652\.795CoLA\.286±\\pm\.043\.163\.288AG News\.791±\\pm\.042\.640\.834MNLI\.345±\\pm\.007\.366\.367QNLI\.555±\\pm\.004\.555\.594RTE\.520±\\pm\.046\.551\.569
### E\.7Alignment Estimation Procedure and Task\-Level Gains
For each BERT classification task, we run the frozen encoder on allNNlabeled examples and extract final\-layer\[CLS\]representations\. The few\-shot set is sampled approximately class\-balanced\. Letncn\_\{c\}be the number of examples in classcc, letμc\\mu\_\{c\}be its mean representation, and letμ¯\\bar\{\\mu\}be the global mean\. We compute
Σtask=1N∑c=1Cnc\(μc−μ¯\)\(μc−μ¯\)⊤\.\\Sigma\_\{\\mathrm\{task\}\}=\\frac\{1\}\{N\}\\sum\_\{c=1\}^\{C\}n\_\{c\}\(\\mu\_\{c\}\-\\bar\{\\mu\}\)\(\\mu\_\{c\}\-\\bar\{\\mu\}\)^\{\\top\}\.IfUUcontains the top\-rrPCA directions computed from the same prediction\-position representations, the alignment coefficient is
ρ=Tr\(U⊤ΣtaskU\)Tr\(Σtask\)\.\\rho=\\frac\{\\operatorname\{Tr\}\\left\(U^\{\\top\}\\Sigma\_\{\\mathrm\{task\}\}U\\right\)\}\{\\operatorname\{Tr\}\\left\(\\Sigma\_\{\\mathrm\{task\}\}\\right\)\}\.This diagnostic requires no gradients and is computed from one frozen forward pass followed by covariance and trace\-ratio calculations\.
Table[25](https://arxiv.org/html/2611.23428#A5.T25)contains the seven values used for the main correlation analysis\. CoLA gains are measured in Matthews correlation points, and all other gains are accuracy points\. Across the seven tasks, the Pearson correlation isr=0\.93r=0\.93\. An exact two\-sided permutation test evaluates all7\!=5,0407\!=5\{,\}040assignments of the gain values to the fixed alignment values and givesp=0\.002p=0\.002\.
Table 25:Alignmentρ\\rhoatN=64N=64and PivotLoRA’s shared\-protocol gain over vanilla LoRA\.Taskρ\\rhoFisher ratioPivotLoRA gain \(points\)AG News0\.8430\.14\+19\.4SST\-20\.6670\.02\+14\.3CoLA0\.5300\.02\+12\.5QNLI0\.4580\.02\+3\.9MNLI0\.4190\.03\+0\.1BoolQ0\.4020\.01\+1\.6RTE0\.2930\.01\+1\.8The class\-based definition ofρ\\rhorequires discrete labels\. It is not directly defined for open\-ended generated responses and is not used to predict the reasoning gains\.
### E\.8Robustness of the Alignment Diagnostic
The seven\-task Spearman correlation is 0\.86\. Leave\-one\-task\-out Pearson correlations range from 0\.87 to 0\.96\. Removing AG News still givesr=0\.87r=0\.87, so the association is not determined only by the largest\-gain task\.
As a same\-sample\-bias check, we compute a cross\-split estimate on AG News, SST\-2, and CoLA\. The PCA subspace is estimated using one half of theN=64N=64set, whileΣtask\\Sigma\_\{\\mathrm\{task\}\}is estimated using the other half\. We then exchange the halves and average the two estimates\. The absolute values decrease because each estimate uses fewer examples, but the ordering AG News\>\>SST\-2\>\>CoLA is preserved\.
Table 26:Cross\-split robustness check atN=64N=64\. CoLA gains are in Matthews correlation points, and the other gains are in accuracy points\.TaskOriginalρ\\rhoCross\-splitρ\\rhoPivotLoRA gainAG News0\.8430\.390\+19\.4SST\-20\.6670\.285\+14\.3CoLA0\.5300\.259\+12\.5#### Subset variability\.
To estimate the sampling uncertainty ofρ\\rho, we recomputed it on ten independently drawn class\-balancedN=64N=64subsets per task \(Table[27](https://arxiv.org/html/2611.23428#A5.T27)\)\. AG News is consistently high, and SST\-2 is high in eight of ten subsets, whereas QNLI, MNLI, RTE, and BoolQ exceed0\.60\.6in only one subset each\. In this BERT\-base, rank\-8,N=64N=64configuration, values near or above0\.60\.6therefore separate the two consistently high\-alignment tasks from the low\-gain group\. CoLA, atρ=0\.53\\rho=0\.53, lies between the two groups on both alignment and gain\.
Table 27:ρ\\rhoover ten class\-balancedN=64N=64subsets per task, with the shared\-protocol PivotLoRA gain from Table[25](https://arxiv.org/html/2611.23428#A5.T25)\. CoLA was not included in this resampling\.TaskMean±\\pmSDRangeSubsets withρ≥0\.6\\rho\\geq 0\.6PivotLoRA gain \(points\)AG News0\.843±\\pm0\.0170\.823–0\.87410/10\+19\.4SST\-20\.673±\\pm0\.1850\.191–0\.8248/10\+14\.3QNLI0\.440±\\pm0\.0790\.360–0\.6231/10\+3\.9BoolQ0\.457±\\pm0\.1370\.306–0\.7491/10\+1\.6MNLI0\.444±\\pm0\.1200\.289–0\.6471/10\+0\.1RTE0\.391±\\pm0\.1090\.255–0\.6061/10\+1\.8
#### Class imbalance\.
On AG News we drew 30N=64N=64subsets under each of three class\-count profiles\. Balanced sampling \(16/16/16/16\) givesρ=0\.869±0\.027\\rho=0\.869\\pm 0\.027, moderate imbalance \(28/16/12/8\) gives0\.866±0\.0240\.866\\pm 0\.024, and severe imbalance \(40/12/8/4\) gives0\.818±0\.0280\.818\\pm 0\.028, with a minimum of0\.7550\.755\. Severe imbalance lowers the estimate but does not change the high\-alignment conclusion in this experiment\. We did not test label noise\. Becauseρ\\rhois built from class means, noisy labels can distort it directly\.
For practical use, we recommend estimating uncertainty by recomputingρ\\rhoover class\-stratified resamples of the cached activations\. A low or unstable estimate should be treated as weak evidence for direction\-aware initialization, and method selection should still be validated under matched tuning when feasible\. We do not recommend a universal threshold\. Instead,ρ\\rhoshould be treated as a comparative screening diagnostic under a fixed model, rank, representation, and sampling protocol\.
### E\.9Encoder\-Specific Class Structure
Table[28](https://arxiv.org/html/2611.23428#A5.T28)reports BERT\-specific measurements from pretrained\[CLS\]representations\. Fisher’s ratio summarizes overall class separation, whileρ\\rhomeasures the fraction of between\-class scatter captured by the top\-rrPCA subspace\. These quantities need not produce identical task orderings because one measures the amount of between\-class separation and the other measures its orientation relative to the leading covariance directions\.
Table 28:Class structure in frozen BERT\[CLS\]representations, averaged over three seeds\. Columns denote the total number of labeled examples\.N=16N=16N=32N=32N=64N=64N=128N=128N=256N=256*Fisher’s discriminant ratio*tr\(SB\)/tr\(SW\)\\operatorname\{tr\}\(S\_\{B\}\)/\\operatorname\{tr\}\(S\_\{W\}\)SST\-20\.070\.040\.020\.010\.01CoLA0\.090\.050\.020\.010\.01AG News0\.370\.180\.140\.110\.10MNLI0\.160\.070\.030\.020\.01RTE0\.070\.030\.010\.010\.00QNLI0\.070\.040\.020\.010\.01*PCA\-class alignmentρ\\rho*SST\-20\.8030\.6530\.6730\.4950\.582CoLA0\.8240\.7210\.5300\.4030\.566AG News0\.9110\.8210\.8430\.8760\.901MNLI0\.7600\.5360\.4230\.3100\.351RTE0\.7050\.4780\.2990\.3450\.280QNLI0\.7560\.5600\.4530\.4240\.437Table[28](https://arxiv.org/html/2611.23428#A5.T28)comes from a separate run of the class\-structure analysis that drew its few\-shot subsets independently, so itsN=64N=64column differs from Table[25](https://arxiv.org/html/2611.23428#A5.T25)by at most 0\.006 \(0\.673 versus 0\.667 on SST\-2\)\. The correlation analysis uses Table[25](https://arxiv.org/html/2611.23428#A5.T25)\.
### E\.10Per\-Setting Gains Behind the Regime Map
Table[29](https://arxiv.org/html/2611.23428#A5.T29)gives the range of per\-setting gains behind Table[4](https://arxiv.org/html/2611.23428#S5.T4)\. BERT rows cover the six primary tasks\. Llama classification rows cover SST\-2 and AG News on Llama\-2\-7B and Llama\-3\.1\-8B and BoolQ on Llama\-3\.1\-8B\. The reasoning row covers GSM8K and StrategyQA on Llama\-3\.1\-8B atN=128N=128and evaluates the complete response\-position procedure \(Appendix[D](https://arxiv.org/html/2611.23428#A4)\)\. The GPT\-2 row covers XSum and CNN/DailyMail atN∈\{64,256\}N\\in\\\{64,256\\\}with response positions and the single learning rate \(Appendix[E\.4](https://arxiv.org/html/2611.23428#A5.SS4)\)\.
Table 29:PivotLoRA gain over vanilla LoRA by regime, in points of each task’s primary metric \(ROUGE\-L for GPT\-2\)\.Settingddd/Nd/NMean gainRangeBERT\-base,N≤64N\\leq 6476812–48\+6\.9\+6\.9\+0\.1\+0\.1to\+23\.9\+23\.9BERT\-base,N≥128N\\geq 1287683–6\+3\.2\+3\.2−2\.0\-2\.0to\+10\.1\+10\.1BERT\-base, full data7680\.002–0\.31\+0\.4\+0\.4−0\.9\-0\.9to\+2\.2\+2\.2Llama classification,N=64N=64409664\+3\.8\+3\.8\+0\.5\+0\.5to\+6\.1\+6\.1Llama classification,N=256N=256409616\+0\.4\+0\.4−1\.8\-1\.8to\+4\.2\+4\.2Llama\-3\.1\-8B reasoning409632\+4\.1\+4\.1\+3\.6\+3\.6to\+4\.5\+4\.5GPT\-2 summarization7683–12\+0\.3\+0\.3−0\.1\-0\.1to\+0\.8\+0\.8
### E\.11Data\-Efficiency Illustration
Figure[5](https://arxiv.org/html/2611.23428#A5.F5)provides a task\-specific illustration on AG News\. AtN=64N=64, PivotLoRA reaches approximately 89% of the displayed full\-data accuracy ceiling, compared with approximately 68% for vanilla LoRA\.
Figure 5:Data efficiency on AG News with BERT\-base\. The shaded region marks the high\-d/Nd/Nfew\-shot regime\. The dashed line denotes the displayed full\-data accuracy reference\.
### E\.12Activation\-Source Ablation
We compare activation sources for estimating the PivotLoRA initialization subspace\. Target\-position initialization uses one\[CLS\]activation per example, whereas all\-token initialization uses all non\-padding token activations\. The first two configurations use the same source for query and value adapters\. The last two use the target position for query adapters and either all tokens or attention\-weighted pooling for value adapters\. All other components, including the sampled examples, rank, scale, learning\-rate allocation, pivot timing, optimizer\-state policy, and training budget, are held fixed within each paired comparison\.
Table 30:Activation\-source ablation for BERT\-base PivotLoRA atN=64N=64\. Values are mean±\\pmsample standard deviation over three paired seeds\. Paired gains are computed as all\-token minus target\-position accuracy\. The last two rows use different sources for query and value adapters\.Activation sourceAG NewsSST\-2Target position \(\[CLS\]\)\.814±\\pm\.015\.778±\\pm\.063All non\-padding tokens\.845±\\pm\.008\.802±\\pm\.048Target for query, all tokens for value\.835±\\pm\.007\.784±\\pm\.024Target for query, attention\-weighted for value\.813±\\pm\.017\.666±\\pm\.045Paired gain of all\-token\+\.032\+\.023All\-token seed\-level wins3/32/3This ablation is a separate matched sweep, so its absolute values should be compared within Table[30](https://arxiv.org/html/2611.23428#A5.T30)\.
Under this matched comparison, all\-token initialization has a higher mean by 3\.15 points on AG News and 2\.33 points on SST\-2, with seed\-level wins in three of three and two of three comparisons, respectively\. The SST\-2 difference is small relative to seed variation\.
Assigning the target position to query adapters and all tokens to value adapters also improves on the uniform target\-only configuration on AG News \(\+2\.1 points, 3/3 seeds\) and slightly on SST\-2 \(\+0\.6 points, 1/3 seeds\), showing that sources can be selected per projection type\. The tested attention\-weighted value source is neutral on AG News \(−0\.1\-0\.1points\) and lower on SST\-2 \(−11\.2\-11\.2points\), so distributed\-token pooling is not automatically beneficial\.
Target\-position PCA uses exactly 64 activation vectors per layer, whereas all\-token PCA uses about 3,400 vectors on AG News and 830 on SST\-2\. The corresponding top\-eight explained\-variance ratios, averaged over the inputs of the twelve adapted layers \(Appendix[B\.1](https://arxiv.org/html/2611.23428#A2.SS1)reports the final layer\), are 55\.88% and 67\.75% for target\-position PCA, compared with 15\.14% and 21\.16% for all\-token PCA\. More broadly, the comparison illustrates the alignment–estimation trade\-off\. Prediction\-position pooling targets the representation used for supervision, whereas all\-token pooling estimates its subspace from a substantially larger activation sample\.
## Appendix FRobustness and Ablation Studies
Several analyses below were run as separate sweeps under the primary protocol\. Their absolute scores differ slightly from the primary tables, so each is compared within its own table\.
### F\.1Aggregate Paired Robustness Analysis
Each individual task–NNsetting contains only three seeds, so we do not emphasize per\-setting significance tests\. Instead, we summarize the direction of the mean differences across the 30 primary BERT settings\.
PivotLoRA exceeds vanilla LoRA in 29 of the 30 settings, with the only negative difference at RTEN=128N=128\. A two\-sided exact sign test on these 30 setting\-level comparisons gives
p=5\.8×10−8p=5\.8\\times 10^\{\-8\}under a null model in which positive and negative differences are equally likely\. Because the settings share a backbone and tasks, they are not independent, and the test summarizes the benchmark without a strict inferential guarantee\. The five\-seed experiments below provide complementary seed\-level evidence for selected settings\.
### F\.2Five\-Seed Confirmation
All seven bootstrap intervals for the paired gain exclude zero, with mean gains from\+0\.032\+0\.032on QNLI to\+0\.231\+0\.231on AG News atN=32N=32\. We reran seven representative BERT settings with seeds 42–46\. Table[31](https://arxiv.org/html/2611.23428#A6.T31)reports the mean, standard deviation, and bootstrap 95% confidence interval for the mean paired difference\. The intervals were computed from seed\-aligned LoRA and PivotLoRA runs\.
Table 31:Five\-seed confirmation on representative BERT\-base settings using seeds 42–46\. The confidence interval is a bootstrap interval for the mean paired difference, PivotLoRA minus LoRA\.TaskNNLoRAPivotLoRAΔ\\Delta95% CIAG News16\.397±\\pm\.044\.577±\\pm\.035\+\.180\[\+\.145, \+\.217\]AG News32\.490±\\pm\.016\.721±\\pm\.047\+\.231\[\+\.191, \+\.265\]AG News64\.650±\\pm\.032\.816±\\pm\.022\+\.166\[\+\.128, \+\.205\]SST\-232\.525±\\pm\.026\.615±\\pm\.064\+\.090\[\+\.056, \+\.125\]SST\-264\.655±\\pm\.032\.790±\\pm\.022\+\.135\[\+\.112, \+\.156\]CoLA64\.148±\\pm\.061\.265±\\pm\.095\+\.117\[\+\.070, \+\.169\]QNLI64\.564±\\pm\.035\.596±\\pm\.037\+\.032\[\+\.010, \+\.047\]
### F\.3Fisher\-Gradient Baseline
We implement a Fisher\-gradient variant inspired by LoRA\-DA\[[Zhang et al\., 2026](https://arxiv.org/html/2611.23428#bib.bib6)\]\. For per\-example gradientsgig\_\{i\}, it estimates
Fdiag=1N∑i=1Ngi⊙giF\_\{\\mathrm\{diag\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}g\_\{i\}\\odot g\_\{i\}and performs an SVD of the Fisher\-weighted mean gradient
G¯⊙Fdiag\.\\bar\{G\}\\odot\\sqrt\{F\_\{\\mathrm\{diag\}\}\}\.Tables[32](https://arxiv.org/html/2611.23428#A6.T32)and[33](https://arxiv.org/html/2611.23428#A6.T33)report the resulting scores, with vanilla LoRA rerun in the same sweep\. LoRA\-DA is below vanilla LoRA in all 10 settings, by 0\.006 to 0\.131\.
Table 32:LoRA\-DA on AG News\. Values are accuracy means and standard deviations over three seeds\. LoRA is rerun in the same sweep\.Method163264128256LoRA\.410±\\pm\.029\.477±\\pm\.078\.624±\\pm\.054\.774±\\pm\.033\.858±\\pm\.010LoRA\-DA\.317±\\pm\.034\.352±\\pm\.015\.493±\\pm\.056\.679±\\pm\.066\.814±\\pm\.030Table 33:LoRA\-DA on SST\-2\. Values are accuracy means and standard deviations over three seeds\. LoRA is rerun in the same sweep\.Method163264128256LoRA\.541±\\pm\.017\.527±\\pm\.026\.639±\\pm\.063\.777±\\pm\.010\.847±\\pm\.007LoRA\-DA\.528±\\pm\.016\.510±\\pm\.020\.598±\\pm\.028\.702±\\pm\.056\.841±\\pm\.001
### F\.4LoRA\+ Behavior at Low Labeled\-Example Counts
LoRA\+ has the best mean in 5 of the 18 primary settings withN≤64N\\leq 64, all atN≤32N\\leq 32, where activation\-based estimates are noisiest\. It retains a random LoRA input basis and changes only the relative learning rates of the two factors\.
Under the shared learning\-rate protocol, LoRA\+ collapses to a constant prediction on most tasks atN≥128N\\geq 128\(zero standard deviation across seeds\), and its AG News scores vary widely\. Appendix[F\.9](https://arxiv.org/html/2611.23428#A6.SS9)includes a separate learning\-rate and differential\-ratio sensitivity analysis\.
### F\.5Pivot Timing
We vary the pivot epoch over
K∈\{1,2,3,5,7,10\}K\\in\\\{1,2,3,5,7,10\\\}on SST\-2, CoLA, and AG News, with vanilla LoRA rerun in the same sweep\. Every tested pivot epoch outperforms LoRA at everyN≤64N\\leq 64on all three tasks, and the defaultK=3K=3is competitive but not uniformly optimal\. An early pivot allows more training in the orthonormal coordinates, while a later pivot permits more task\-gradient drift before the coordinate change\.
Table 34:Pivot timing on SST\-2\. Values are accuracy means over three seeds\.KK1632641282561\.570\.601\.762\.818\.8542\.568\.630\.779\.821\.8543 \(default\)\.567\.608\.786\.829\.8445\.565\.619\.794\.825\.8517\.569\.659\.797\.820\.85110\.561\.612\.783\.813\.850LoRA\.543\.526\.641\.777\.846Table 35:Pivot timing on CoLA\. Values are Matthews correlation means over three seeds\.KK1632641282561\.127\.183\.285\.403\.4122\.126\.191\.296\.378\.4003 \(default\)\.119\.182\.289\.366\.3935\.132\.172\.276\.355\.3887\.128\.183\.279\.365\.40310\.125\.175\.270\.369\.401LoRA\.100\.113\.200\.339\.400Table 36:Pivot timing on AG News\. Values are accuracy means over three seeds\.KK1632641282561\.589\.708\.799\.857\.8772\.593\.717\.802\.861\.8783 \(default\)\.604\.707\.816\.866\.8785\.595\.724\.817\.865\.8797\.583\.719\.821\.865\.87710\.516\.701\.813\.865\.879LoRA\.411\.477\.631\.774\.857
### F\.6Pivot Preservation and Orthogonality Diagnostics
For each classification adapter, we measure the relative preservation error
ϵpivot=‖B\(K\)A\(K\)−BnewAnew‖F‖B\(K\)A\(K\)‖F\+10−12\.\\epsilon\_\{\\mathrm\{pivot\}\}=\\frac\{\\left\\\|B^\{\(K\)\}A^\{\(K\)\}\-B^\{\\mathrm\{new\}\}A^\{\\mathrm\{new\}\}\\right\\\|\_\{F\}\}\{\\left\\\|B^\{\(K\)\}A^\{\(K\)\}\\right\\\|\_\{F\}\+10^\{\-12\}\}\.The factorα/r\\alpha/rcancels from this ratio\. We also measure the residual between the old and new row spans, the row\-orthogonality error
ϵorth\(A\)=‖AA⊤−Ir‖F,\\epsilon\_\{\\mathrm\{orth\}\}\(A\)=\\\|AA^\{\\top\}\-I\_\{r\}\\\|\_\{F\},and the change in the layerwise alignment score\. The classification values in Table[37](https://arxiv.org/html/2611.23428#A6.T37)are aggregated over three seeds and 72 adapter instances per task, comprising 24 adapted modules for each seed\.
Table 37:Classification output\-preservation and orthogonality diagnostics\. The alignment change excludes layer 0 because its embedding\-level\[CLS\]scatter is near zero\.TaskMeanϵpivot\\epsilon\_\{\\mathrm\{pivot\}\}Max\.ϵpivot\\epsilon\_\{\\mathrm\{pivot\}\}Mean span residualϵorth\(A\(K\)\)\\epsilon\_\{\\mathrm\{orth\}\}\(A^\{\(K\)\}\)ϵorth\(Anew\)\\epsilon\_\{\\mathrm\{orth\}\}\(A^\{\\mathrm\{new\}\}\)MeanΔρ\\Delta\\rhoAG News3\.14×10−73\.14\\times 10^\{\-7\}8\.82×10−78\.82\\times 10^\{\-7\}2\.94×10−72\.94\\times 10^\{\-7\}2\.8094\.84×10−74\.84\\times 10^\{\-7\}2\.77×10−92\.77\\times 10^\{\-9\}CoLA3\.00×10−73\.00\\times 10^\{\-7\}5\.07×10−75\.07\\times 10^\{\-7\}2\.87×10−72\.87\\times 10^\{\-7\}2\.8094\.69×10−74\.69\\times 10^\{\-7\}−2\.31×10−9\-2\.31\\times 10^\{\-9\}SST\-22\.79×10−72\.79\\times 10^\{\-7\}7\.91×10−77\.91\\times 10^\{\-7\}2\.89×10−72\.89\\times 10^\{\-7\}2\.8094\.76×10−74\.76\\times 10^\{\-7\}2\.56×10−102\.56\\times 10^\{\-10\}
The output and span residuals remain below10−610^\{\-6\}\. The post\-pivot orthogonality errors are also below10−610^\{\-6\}, and the alignment change is near numerical zero\. These measurements confirm that the classification pivot preserves the represented update and learned row span while changing their coordinate parameterization\. In the reasoning runs, the maximum relative preservation error is1\.44×10−71\.44\\times 10^\{\-7\}and the maximum post\-pivot orthogonality error is3\.51×10−83\.51\\times 10^\{\-8\}\.
#### What the pivot changes\.
Writingσ1,…,σr\\sigma\_\{1\},\\dots,\\sigma\_\{r\}for the singular values ofA\(K\)A^\{\(K\)\},ϵorth\(A\(K\)\)2=r−2∑iσi2\+∑iσi4\\epsilon\_\{\\mathrm\{orth\}\}\(A^\{\(K\)\}\)^\{2\}=r\-2\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\+\\sum\_\{i\}\\sigma\_\{i\}^\{4\}\. When‖A\(K\)‖F2=∑iσi2\\\|A^\{\(K\)\}\\\|\_\{F\}^\{2\}=\\sum\_\{i\}\\sigma\_\{i\}^\{2\}is also logged, as in the reasoning runs, these two quantities determine the effective rankreff=\(∑iσi2\)2/∑iσi4r\_\{\\mathrm\{eff\}\}=\(\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\)^\{2\}/\\sum\_\{i\}\\sigma\_\{i\}^\{4\}of the pre\-pivot Gram matrix\. In classification, the value2\.8092\.809is what a scaled orthonormal matrix5/dU⊤\\sqrt\{5/d\}\\,U^\{\\top\}would give \(2\.8102\.810ford=768d=768\), which places‖A\(K\)‖F\\\|A^\{\(K\)\}\\\|\_\{F\}within about 5% of its initial value\. The pivot therefore raises the row norms ofAAabout twelvefold, although the three\-decimal value cannot resolve whetherAAalso became anisotropic\. In the reasoning runs, the median adapter’s‖A\(K\)‖F\\\|A^\{\(K\)\}\\\|\_\{F\}grows to5\.7×5\.7\\timesits initial value, and the medianreffr\_\{\\mathrm\{eff\}\}is1\.91\.9–2\.12\.1on GSM8K and1\.41\.4–1\.51\.5on StrategyQA across seeds, out ofr=8r=8\. Early training therefore concentrates the learned input basis on a few directions, and the pivot restores an isotropic basis for the same span\.
### F\.7Orthogonal Pivot and Optimizer\-State Ablation
We separate the output\-preserving QR pivot from the classification optimizer\-state policy\.QR\-Pivot\-NoResetapplies the pivot but keeps the Adam state ofAA, whereasPivotLoRAresets it\. Both use the same post\-pivot learning\-rate handling \(Appendix[C\.1](https://arxiv.org/html/2611.23428#A3.SS1)\)\.
Table 38:Classification optimizer\-state ablation on SST\-2\. Values are accuracy means over three seeds\.MethodN=16N=16N=32N=32N=64N=64N=128N=128N=256N=256TT\-PCA\.520\.530\.755\.817\.858QR\-Pivot\-NoReset\.552\.623\.795\.829\.858PivotLoRA\.541\.628\.784\.828\.847Table 39:Classification optimizer\-state ablation on CoLA\. Values are Matthews correlation means over three seeds\.MethodN=16N=16N=32N=32N=64N=64N=128N=128N=256N=256TT\-PCA\.104\.137\.270\.383\.402QR\-Pivot\-NoReset\.117\.172\.301\.381\.398PivotLoRA\.117\.174\.304\.381\.393Table 40:Classification optimizer\-state ablation on AG News\. Values are accuracy means over three seeds\.MethodN=16N=16N=32N=32N=64N=64N=128N=128N=256N=256TT\-PCA\.478\.561\.756\.857\.874QR\-Pivot\-NoReset\.589\.689\.811\.859\.878PivotLoRA\.597\.703\.811\.857\.878The QR pivot accounts for most of the improvement over TT\-PCA\. Without the reset, it improves on TT\-PCA in 12 of 15 settings, by0\.0370\.037on average and by up to0\.1280\.128on AG News atN=32N=32\. The additional reset produces smaller, task\-dependent differences and is not uniformly superior toQR\-Pivot\-NoReset\. We therefore treat the classification reset as a stabilization policy and not as the main source of the gain\.
This ablation does not apply directly to the reasoning experiments, which reset the optimizer states of bothAAandBB, and no reasoning reset\-only or no\-reset ablation was conducted\.
### F\.8Post\-Pivot Learning\-Rate Policy
The pivot’s gain does not depend on how theBBlearning rate is handled afterward\. All pivot variants share the TT\-PCA initialization, the eigenvalue\-scaled rates before the pivot, the QR pivot atK=3K=3, and the Adam reset forAA, and differ only in theBBrate after the pivot\. The implemented policy raises a common rate fromηA\\eta\_\{A\}to16ηA16\\eta\_\{A\}over 10 steps and then resumes the eigenvalue\-scaled rates\. The alternatives keep the eigenvalue\-scaled rates unchanged, switch to a uniform16ηA16\\eta\_\{A\}for the rest of training, or ramp to that uniform rate and keep it\. The sweep covers SST\-2, CoLA, and AG News atN∈\{16,32,64,128,256\}N\\in\\\{16,32,64,128,256\\\}with three seeds\.
Table 41:Post\-pivot learning\-rate policy\. Task columns average over the five values ofNN\(CoLA in Matthews correlation\); “Wins” counts the 15 task–NNsettings in which a variant exceeds TT\-PCA\.VariantSST\-2CoLAAG NewsAverageWinsTT\-PCA \(no pivot\)\.697\.257\.717\.557–Pivot, no rate change\.734\.275\.770\.59311/15Pivot, implemented policy\.728\.276\.772\.59212/15Pivot, uniform rate\.744\.266\.777\.59613/15Pivot, ramp then uniform rate\.723\.265\.766\.58512/15Every policy improves on TT\-PCA by0\.0280\.028–0\.0390\.039on average \(0\.0360\.036with no rate change\), and the policies differ by at most0\.0110\.011\.
### F\.9Learning\-Rate Structure in the Ablation Ladder
TT\-PCA and TT\-LDA differ in both initialization direction and learning\-rate structure\. TT\-PCA uses the eigenvalue\-scaled rates in Eq\. \([12](https://arxiv.org/html/2611.23428#A3.E12)\), whereas TT\-LDA uses the uniform differential rateηB=16ηA\\eta\_\{B\}=16\\eta\_\{A\}\. Their direct comparison is consequently an ablation ladder and does not isolate PCA from LDA\.
Across the full BERT benchmark, TT\-LDA, TT\-PCA, and PivotLoRA have average ranks 3\.13, 3\.77, and 1\.83, respectively\. LoRA\+ retains random input directions and applies a16×16\\timesdifferential rate but has average rank 8\.43 under the shared protocol\. This aggregate comparison argues against an explanation based only on the averageBB\-to\-AAlearning\-rate ratio\. The direct controls in Appendix[F\.10](https://arxiv.org/html/2611.23428#A6.SS10)isolate the PCA directions from the eigenvalue\-to\-coordinate rate assignment\.
#### Learning\-rate sensitivity\.
We sweep learning rates on AG News, SST\-2, and CoLA atN=64N=64\.
For each method, we sweep the base learning\-rate anchor over
ηA∈\{5×10−5,2×10−4,8×10−4\}\.\\eta\_\{A\}\\in\\left\\\{5\\times 10^\{\-5\},2\\times 10^\{\-4\},8\\times 10^\{\-4\}\\right\\\}\.The primary shared\-protocol anchor isηA=2×10−4\\eta\_\{A\}=2\\times 10^\{\-4\}\. Differential\-rate methods derive theirBB\-factor rates from this anchor\. For LoRA\+, model selection additionally ranges over
ηB/ηA∈\{4,8,16,32\}\.\\eta\_\{B\}/\\eta\_\{A\}\\in\\\{4,8,16,32\\\}\.
Table[42](https://arxiv.org/html/2611.23428#A6.T42)reports the average score at the primary anchor and the average of each method’s best swept score\. Tuning reduces several margins, but PivotLoRA remains strongest on average among the swept methods\.
Table 42:Learning\-rate sensitivity atN=64N=64\. Scores are unweighted averages of each task’s primary metric over AG News, SST\-2, and CoLA\. “Avg\. anchor” uses the primary base learning\-rate anchorηA=2×10−4\\eta\_\{A\}=2\\times 10^\{\-4\}, together with each method’s prescribed factor\-wise allocation\. “Avg\. best” averages each method’s best score over the base\-rate sweep\. For LoRA\+, selection additionally includesηB/ηA∈\{4,8,16,32\}\\eta\_\{B\}/\\eta\_\{A\}\\in\\\{4,8,16,32\\\}\. Gaps are measured relative to PivotLoRA under the corresponding comparison\.MethodAvg\. anchorAvg\. bestPivot anchor gapPivot best gapLoRA\.522\.622\+\.139\+\.045LoRA\+\.326\.636\+\.335\+\.031PiSSA\.420\.606\+\.241\+\.061LoRAM\.428\.617\+\.233\+\.050EVA\.424\.618\+\.237\+\.048TT\-PCA\.643\.655\+\.017\+\.012PivotLoRA\.661\.667\.000\.000Table 43:Best swept scores by task atN=64N=64\. Each score is selected over the base learning\-rate anchors\{5×10−5,2×10−4,8×10−4\}\\\{5\\times 10^\{\-5\},2\\times 10^\{\-4\},8\\times 10^\{\-4\}\\\}\. For LoRA\+, selection additionally ranges over the differential ratiosηB/ηA∈\{4,8,16,32\}\\eta\_\{B\}/\\eta\_\{A\}\\in\\\{4,8,16,32\\\}\.MethodAG NewsSST\-2CoLALoRA\.781\.784\.301LoRA\+\.833\.757\.318PiSSA\.754\.733\.330LoRAM\.732\.789\.330EVA\.742\.786\.327TT\-PCA\.817\.821\.328PivotLoRA\.843\.822\.335
### F\.10Direction versus Learning\-Rate Scaling Controls
We evaluate three controls atN=64N=64to separate target\-token PCA directions from the associated learning\-rate allocation\.
- •Random\+EigenLRuses a random input basis drawn as in vanilla LoRA \(Kaiming\-uniform, not orthonormalized\) with the same eigenvalue\-scaled per\-rankBBlearning rates\.
- •TT\-PCA\+UniformLRretains the target\-token PCA basis and uses a uniform differential learning rate\.
- •TT\-PCA\+ShuffledLRretains the target\-token PCA basis and randomly permutes the eigenvalue\-to\-coordinate rate assignment\.
The TT\-PCA row in Table[44](https://arxiv.org/html/2611.23428#A6.T44)comes from the same control sweep and is the reference for the three variants\.
Table 44:Direction versus learning\-rate scaling controls atN=64N=64using BERT\-base\. Values are mean±\\pmstandard deviation over three seeds from the separate diagnostic sweep\.MethodAG NewsSST\-2CoLALoRA\.640±\\pm\.033\.647±\\pm\.040\.188±\\pm\.042TT\-PCA\.751±\\pm\.025\.769±\\pm\.038\.268±\\pm\.092Random\+EigenLR\.444±\\pm\.268\.571±\\pm\.031\.028±\\pm\.035TT\-PCA\+UniformLR\.752±\\pm\.019\.769±\\pm\.035\.276±\\pm\.100TT\-PCA\+ShuffledLR\.754±\\pm\.028\.765±\\pm\.028\.281±\\pm\.088A random basis does not become competitive merely by receiving the eigenvalue\-scaled rates\. Conversely, the target\-token PCA basis retains its performance when the rates are made uniform or their assignments are permuted\. Within this control sweep, the selected directions therefore explain more of the advantage than the eigenvalue\-to\-coordinate rate assignment\. Because the classification implementation applies a fixed random rotation within the PCA span \(Eq\. \([7](https://arxiv.org/html/2611.23428#S4.E7)\)\), each rate\-scaled row ofBBalready corresponds to a mixture of principal directions, so the shuffled control therefore tests the rate allocation itself, not a pairing between individual directions and rates\. This experiment separates direction from learning\-rate allocation but not from initialization norm\.
## Appendix GPer\-Seed and Cost Details
### G\.1Per\-Seed Decoder\-Scale Results
Tables[45](https://arxiv.org/html/2611.23428#A7.T45)and[46](https://arxiv.org/html/2611.23428#A7.T46)provide the per\-seed decoder\-scale results\. PivotLoRA has the highestN=64N=64aggregate mean on SST\-2 and AG News for both backbones, although individual seed\-level exceptions occur\.
Table 45:Llama\-2\-7B per\-seed classification results in bf16 on an NVIDIA L4\.SST\-2,N=64N=64SST\-2,N=256N=256AG News,N=64N=64AG News,N=256N=256s42s43s44s42s43s44s42s43s44s42s43s44LoRA\.711\.757\.655\.931\.917\.922\.724\.743\.777\.877\.884\.876PiSSA\.683\.571\.650\.911\.927\.930\.706\.709\.684\.867\.874\.854LoRAM\.765\.787\.737\.927\.944\.915\.719\.771\.737\.878\.877\.872PivotLoRA\.802\.742\.764\.927\.921\.889\.809\.791\.753\.876\.869\.839Table 46:Llama\-3\.1\-8B per\-seed classification results in bf16 on an NVIDIA L4\.SST\-2,N=64N=64SST\-2,N=256N=256AG News,N=64N=64AG News,N=256N=256s42s43s44s42s43s44s42s43s44s42s43s44LoRA\.728\.723\.661\.912\.885\.851\.758\.783\.737\.870\.885\.886PiSSA\.626\.578\.663\.911\.865\.853\.710\.744\.691\.865\.874\.854LoRAM\.720\.640\.627\.907\.909\.888\.735\.800\.708\.877\.879\.861PivotLoRA\.735\.778\.614\.888\.912\.899\.800\.799\.798\.889\.884\.836
### G\.2Initialization Overhead
Table[47](https://arxiv.org/html/2611.23428#A7.T47)reports initialization time and additional persistent storage for representative methods\. PivotLoRA performs one activation pass and PCA initialization before training and one QR\-based coordinate change at the pivot\. The timing is small relative to the approximately 69\-second classification training run used in the cost comparison below\.
Table 47:Initialization overhead for BERT\-base with 12 layers and rankr=8r=8on an NVIDIA T4\.MethodInitialization timeAdditional persistent storageLoRA<0\.1<0\.1sNonePivotLoRA<1<1s plus<1<1s at pivotNonePiSSA∼2\{\\sim\}2s, layerwise SVDResidualWresW^\{\\mathrm\{res\}\}LoRA\-GA∼3\{\\sim\}3s, gradient plus SVDNoneLoRA\-One∼3\{\\sim\}3s, gradient plus SVDNoneEVA∼1\{\\sim\}1s, activation SVDNoneLoRAM<0\.5<0\.5s, power iterationNone
### G\.3Computational Cost
Tables[48](https://arxiv.org/html/2611.23428#A7.T48)and[49](https://arxiv.org/html/2611.23428#A7.T49)report end\-to\-end training time and peak memory\. On the BERT measurement, PivotLoRA adds 4\.0 seconds relative to LoRA, corresponding to 5\.8% wall\-time overhead\. Most of this difference comes from the per\-rank optimizer groups for the eigenvalue\-scaled learning rates, and the QR pivot itself takes under a second\. Peak memory is nearly unchanged\.
Table 48:BERT\-base training time and peak memory atN=64N=64on SST\-2, averaged over two seeds on one NVIDIA T4\.MethodWall time \(s\)Peak VRAM \(MB\)Overhead vs\. LoRALoRA69\.41,266N/ADoRA69\.51,399\+0\.1 s \(\+0\.2%\)AdaLoRA102\.21,264\+32\.9 s \(\+47\.4%\)PiSSA68\.61,267−0\.8\-0\.8s \(−1\.2%\-1\.2\\%\)LoRA\-GA68\.01,272−1\.4\-1\.4s \(−1\.9%\-1\.9\\%\)LoRA\-One69\.01,272−0\.4\-0\.4s \(−0\.6%\-0\.6\\%\)LoRA\+68\.91,267−0\.5\-0\.5s \(−0\.7%\-0\.7\\%\)EVA69\.81,267\+0\.4 s \(\+0\.6%\)LoRAM68\.91,267−0\.5\-0\.5s \(−0\.7%\-0\.7\\%\)SRLoRA89\.41,267\+20\.0 s \(\+28\.8%\)TT\-PCA73\.61,267\+4\.2 s \(\+6\.0%\)TT\-LDA69\.31,267−0\.1\-0\.1s \(−0\.1%\-0\.1\\%\)PivotLoRA73\.41,268\+4\.0 s \(\+5\.8%\)
TT\-PCA and PivotLoRA use per\-rank optimizer groups for their eigenvalue\-scaled rates\. TT\-LDA uses the uniform differential rateηB=16ηA\\eta\_\{B\}=16\\eta\_\{A\}\. The 0\.2\-second difference between TT\-PCA and PivotLoRA is within run\-to\-run timing variation\.
Table 49:Llama\-2\-7B classification training cost on SST\-2 atN=64N=64, using bf16 on one NVIDIA L4\.MethodWall time \(s\)Peak VRAM \(GB\)LoRA∼74\{\\sim\}7414\.6PiSSA∼73\{\\sim\}7314\.6PivotLoRA∼73\{\\sim\}7314\.6
## Appendix HLimitations
#### Activation\-source and task scope\.
Activation\-based initialization works best with a source matched to the supervision, such as the\[CLS\]representation for BERT classification, the sequence mean for decoder classification, and hidden states that predict supervised response tokens for completion\-only reasoning\. Target\-position pooling emphasizes representations tied to prediction, whereas all\-token pooling provides more observations but includes variation less specific to the target, and neither should be expected to dominate across tasks\. Query and value adapters share one source rule in our main configuration\. Sources can be chosen per projection type \(Appendix[E\.12](https://arxiv.org/html/2611.23428#A5.SS12)\), but we do not study this systematically\.
#### Theoretical and diagnostic scope\.
The coefficientρ\\rhois defined through between\-class covariance, so it applies most directly to labeled classification, and its association with gains is measured across seven BERT tasks under one protocol\. Generation and continuous targets would require a different definition of supervised structure\. The perturbation result assumes independent sub\-Gaussian observations, an approximately spiked covariance, and a positive eigengap, and the first\-step analysis adds a class\-conditional mean model and an assumption relating gradient\-weighted geometry to between\-class scatter\. Both give a regime\-level account and are not convergence guarantees for transformer fine\-tuning\.
#### Model and task coverage\.
The decoder classification experiments cover fewer tasks and baselines than the BERT study and check whether the main pattern persists at larger scale, not full rankings of methods for 7B and 8B models\. The reasoning experiments cover mathematical and yes/no reasoning with one 8B model and do not support claims about instruction tuning, multilingual adaptation, or unrestricted generation\.
#### Reasoning attribution and statistical scope\.
The reasoning procedure combines response\-position PCA, a fixed5/d\\sqrt\{5/d\}basis scale, one output\-preserving pivot, and a reset of the AdamW state for both factors\. Without norm\-matched, reset\-only, or direction\-versus\-rate controls in this setting, the results evaluate the complete procedure and do not isolate subspace direction as the cause\. The McNemar tests in Appendix[D\.4](https://arxiv.org/html/2611.23428#A4.SS4)treat test items as the unit of variation and do not capture variation across training runs\.
#### Pivot scope\.
The pivot orthonormalizes coordinates within the row span learned before epochKKand adds no new directions\. Replacing low\-utility rank directions during training while preserving the accumulated update is a natural extension\.
#### Broader impacts\.
Parameter\-efficient adaptation lowers the cost of few\-shot specialization, which may also lower the cost of adapting pretrained models for harmful or deceptive uses\. PivotLoRA should be used under the same safety, data\-governance, and misuse\-mitigation practices that apply to the underlying pretrained models\.Similar Articles
Beyond LoRA: Is Sparsity-Induced Adaptation Better?
This paper proposes sparsity-induced adaptations to LoRA, including Cheap LoRA (cLA) and a chained circulant variant (c³LA), and provides theoretical generalization bounds along with empirical evaluations showing up to 10% training time reduction and 15% peak GPU memory savings while maintaining competitive performance.
LoRAWeaver: Directional Adapter Routing in LoRA Representation Space
LoRAWeaver 提出一种免训练的 LoRA 适配器路由方法,通过在 LoRA 输出空间中将查询表示与任务原型进行余弦相似度匹配来选择适配器,在 FLAN-v2 48 任务基准上优于其他免训练基线并接近训练式检索编码器。
@jbhuang0604: LoRA, low-rank adaptation, is arguably the most popular parameter-efficient fine-tuning method for LLMs. But how does i…
LoRA (low-rank adaptation) is the most popular parameter-efficient fine-tuning method for LLMs. This video introduces how LoRA and its variants (LoRA+, QLoRA, VeRA, DoRA) work.
Rethinking LoRA Initialization for Robust Asymmetric Learning Rates
A paper analyzing LoRA's sensitivity to learning-rate choices and proposing Init[A²], a new initialization scheme with variance scaling that suppresses initialization-induced noise, stabilizing asymmetric LoRA+ training and widening the stable learning-rate region for LLM fine-tuning.
RDP LoRA: Geometry-Driven Identification for Parameter-Efficient Adaptation in Large Language Models
RDP-LoRA uses geometric trajectory analysis and the Ramer-Douglas-Peucker algorithm to automatically select the most impactful layers for parameter-efficient fine-tuning, outperforming full-layer and random LoRA baselines.