FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning

# FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
Source: [https://arxiv.org/html/2605.22869](https://arxiv.org/html/2605.22869)
Yequan Zhao1Ruijie Zhang1Liyan Tan1Niall Moran2Tong Qin2Zheng Zhang1 1University of California at Santa Barbara2Amazon Lab126 \{yequan\_zhao, ruijiezhang, liyan\_tan\}@ucsb\.edu \{nialmora, tqinmath\}@amazon\.com zhengzhang@ece\.ucsb\.edu

###### Abstract

Both full fine\-tuning \(Full FT\) and parameter\-efficient methods like LoRA add weight updates without regard to the spectral structure that pretraining has established\. This allows noisy gradients from a small fine\-tuning distribution to freely perturb the robust features learned through pretraining\. We first identify*spectral preconditioning*as the key missing ingredient: reparameterizing each weight𝐖\\mathbf\{W\}through its full\-rank SVD and freezing one singular basis confines every update to the pretrained column space, yielding a preconditioned optimizer that outperforms unconstrained Full FT at the same parameter count\. To make this insight practical, we proposeFuRA\(Full\-RankAdaptation\), which factorizes𝐖\\mathbf\{W\}via a block tensor\-train decomposition𝐖=𝐋​𝐒​𝐑\\mathbf\{W\}=\\mathbf\{L\}\\,\\mathbf\{S\}\\,\\mathbf\{R\}: the large core𝐋\\mathbf\{L\}is frozen at the pretrained block\-wise SVD basis while only the small core𝐑\\mathbf\{R\}and per\-block singular values𝐒\\mathbf\{S\}are trained\. This single design choice simultaneously delivers full\-rank spectral preconditioning, full\-rank update capacity, and parameter, step time, memory efficiency on par with LoRA\.FuRAoutperforms Full FT on LLM fine\-tuning \(\+1\.37\+1\.37on LLaMA\-3\-8B commonsense reasoning\), LLM math reinforcement learning, and VLM visual instruction tuning\. The 4\-bit quantized version QFuRA also outperforms QLoRA\. Code is available at[https://github\.com/olokevin/FuRA\-NIPS](https://github.com/olokevin/FuRA-NIPS)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.22869v1/x1.png)Figure 1:FuRAdelivers higher accuracy than Full FT with LoRA\-level runtime and the lowest GPU memory\.Large language modelsGrattafioriet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib17)\); Qwen Team \([2025](https://arxiv.org/html/2605.22869#bib.bib16)\)\(LLM\) acquire rich, transferable representations during pretraining on web\-scale corpora\. Fine\-tuning these models on a small, task\-specific dataset, whether through supervised finetuning \(SFT\) or reinforcement learning with verifiable rewards \(RLVR\)Shaoet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib8)\); Yuet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib9)\), can unlock strong downstream performance\. Full fine\-tuning \(Full FT\) updates every parameter and remains the performance ceiling on most benchmarks, but incurs prohibitive memory and compute costs\. Parameter\-efficient methods such as LoRAHuet al\.\([2022](https://arxiv.org/html/2605.22869#bib.bib1)\)circumvent this cost by constraining the updateΔ​𝐖\\Delta\\mathbf\{W\}to a low\-rank subspace, yet a persistent accuracy gap relative to Full FT\(Schulman and Thinking Machines Lab,[2025](https://arxiv.org/html/2605.22869#bib.bib21)\)motivates continued research into more expressive alternatives includingLiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\); Menget al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib2)\); Zhanget al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib5)\); Liuet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib7)\)\.

We argue that both Full FT and LoRA share a deeper limitation:neither respects the spectral structure of the pretrained weight\.The update is unconstrained in direction and can push the model along axes orthogonal to the feature manifold that pretraining established\. Specifically, the updateΔ​𝐖\\Delta\\mathbf\{W\}, whether computed directly from gradients \(Full FT\) or accumulated in a low\-rank adapter𝐁𝐀\\mathbf\{B\}\\mathbf\{A\}\(LoRA\), is added to𝐖\\mathbf\{W\}without regard to its learned spectral geometry\. Because the fine\-tuning data is orders of magnitude smaller and narrower in distribution, its gradients are noisier estimates of the true task\-relevant directions\. Allowing these noisy updates to freely perturb robust pretrained features risks degrading generalization, a failure mode closely related to catastrophic forgettingKirkpatricket al\.\([2017](https://arxiv.org/html/2605.22869#bib.bib66)\)\. This observation suggests a different design principle: rather than adding an unconstrained perturbation to𝐖\\mathbf\{W\}, fine\-tuning shouldre\-align the pretrained featuresin a way that is guided by, and confined to, the subspace the model has already learned\. If the pretrained weight encodes a coordinate system for useful representations, adaptation should re\-orient within that coordinate system rather than demolish and rebuild it\.

Through a controlled study of SVD FT \(§[3\.2](https://arxiv.org/html/2605.22869#S3.SS2)\), we further validate that confining weight updates to the pretrained column space can produce superior performance in LLM fine tuning\. However, directly doing so can introduce a large number of training variables like full FT\. This motivates us to developFuRA\(Full\-RankAdaptation, Figure[2](https://arxiv.org/html/2605.22869#S1.F2)\), a parameter\-efficient method that simultaneously delivers 1\) full\-rank update capacity, 2\) high accuracy through full\-rank spectral preconditioned update, and 3\) LoRA\-level parameter efficiency\.

FuRAoutperforms other PEFT methods and Full FT without sacrificing inference efficiency:\+2\.87\+2\.87over DoRALiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\)and\+1\.37\+1\.37over Full FT on LLaMA\-3\-8B Commonsense SFT, outperforms Full FT on Qwen\-1\.7B/7B math RLVR,\+1\.1\+1\.1over Full FT on VLM visual instruction tuning, all with similar memory and wall\-clock step time of LoRA\. The 4\-bit quantized version QFuRA also surpasses QLoRADettmerset al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib10)\)by\+2\.51\+2\.51on LLaMA\-3\-70B math fine\-tuning\.

Contributions\.Our research contributions are summarized below\.

- •We study the spectral properties of LLM fine\-tuning and identify spectral preconditioning as a key missing ingredient\. We show that both full fine\-tuning and LoRA ignore the pretrained spectral structure, while controlled SVD\-based fine\-tuning demonstrates that aligning updates with this structure yields better performance than full fine\-tuning\.
- •Based on these observations, we proposeFuRA, a block tensor\-train factorization that makes spectral preconditioning a practical PEFT method\. A single architectural choice simultaneously achieves full\-rank spectral preconditioning, full\-rank update capacity, and LoRA\-level parameter and computing efficiency\.
- •We prove two key properties of FuRA: although it trains only a small fraction of parameters, it can realize full\-rank updates; and its effective update acts as a column\-space projection combined with singular\-value preconditioning\.
- •We demonstrate thatFuRAoutperforms Full FT on both SFT and RLVR tasks with only<2%<2\\%trainable parameters and no task\-specific rank tuning, establishing a new state of the art for parameter\-efficient LLM adaptation\.

![Refer to caption](https://arxiv.org/html/2605.22869v1/x2.png)Figure 2:FuRAreplaces each pretrained linear layer𝐖\\mathbf\{W\}with a lossless block tensor\-train factorization𝐖=𝐋​𝐒​𝐑\\mathbf\{W\}=\\mathbf\{L\}\\,\\mathbf\{S\}\\,\\mathbf\{R\}\. Tensors are flattened to matrix for better demonstration\. Each slice is initialized by the full\-rank SVD of weight block𝐖k\\mathbf\{W\}\_\{k\}, and performs spectral preconditioned update\.
## 2Background and Related Work

PEFT methods for LLMs\.LoRAHuet al\.\([2022](https://arxiv.org/html/2605.22869#bib.bib1)\)constrains the weight update toΔ​𝐖=𝐁𝐀⊤\\Delta\\mathbf\{W\}=\\mathbf\{B\}\\mathbf\{A\}^\{\\top\}with rankr≪min⁡\(din,dout\)r\\ll\\min\(d\_\{\\text\{in\}\},d\_\{\\text\{out\}\}\), training only𝒪​\(r​\(din\+dout\)\)\\mathcal\{O\}\(r\(d\_\{\\text\{in\}\}\+d\_\{\\text\{out\}\}\)\)parameters per layer\. This low\-rank constraint primarily serves parameter efficiency rather than reflecting task\-specific inductive bias\. Subsequent work improves LoRA along several directions: QLoRADettmerset al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib10)\)adds weight quantization, DoRALiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\)decouples magnitude and direction, AdaLoRAZhanget al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib5)\)adapts rank across layers, and VeRAKopiczkoet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib49)\)shares fixed random bases while training only diagonal scales\. Tensor\-train decompositionOseledets \([2011](https://arxiv.org/html/2605.22869#bib.bib14)\); Novikovet al\.\([2015](https://arxiv.org/html/2605.22869#bib.bib24)\); Qiuet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib33)\)offer another route; adapters such as LoRETTAYanget al\.\([2024b](https://arxiv.org/html/2605.22869#bib.bib50)\)and LoRTAHuet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib51)\)train all tensor cores but leaveΔ​𝐖\\Delta\\mathbf\{W\}low\-rank\. Despite these advances, a performance gap with Full FT remains\.

PEFT with higher or full rank capacity\.A natural remedy is to lift the rank cap\. MoRAJianget al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib48)\)reshapes the adapter into a square matrix to allow higher\-rank updates; RandLoRAAlbertet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib29)\)uses fixed random bases with trainable scales; BOFTLiuet al\.\([2024b](https://arxiv.org/html/2605.22869#bib.bib32)\)applies a butterfly\-factored orthogonal transform, and QuanTAChenet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib30)\), FourierFTGaoet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib52)\), etc\.S2\\text\{S\}^\{2\}FTYanget al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib13)\)and LIFTLiuet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib7)\)train structured\- or unstructured\-sparse subsets of weights, achieving full\-rank updates but requiring gather/scatter operations that are less tensor\-parallel friendly\. These methods achieve nominally full\-rankΔ​𝐖\\Delta\\mathbf\{W\}at the LoRA budget, but have not consistently closed the gap to Full FT on standard benchmarks\. We argue that rank alone is insufficient; the*subspace*in which the update operates is equally important\.

Spectrum\-informed adapters\.A related line of work leverages the pretrained weight’s SVD to guide adaptation\. PiSSAMenget al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib2)\)initializes LoRA from top\-rrsingular directions, MiLoRAWanget al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib3)\)uses bottom\-rr, Spectral AdapterZhang and Pilanci \([2024](https://arxiv.org/html/2605.22869#bib.bib6)\)parameterizes updates in the spectral basis, and SVFTLingamet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib31)\)trains a dense middle matrix between frozen SVD factors\. While these approaches incorporate spectral information, they face two limitations\. First, the low\-rank constraint forces a choice of spectral slice, yet the optimal subspace is task\-dependent: SFT emphasizes principal directions, whereas RLVR favors off\-principal onesZhuet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib28)\)\. Second, SVD\-based initialization is only a soft constraint, allowing updates to drift away from the pretrained subspace during trainingYinet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib64)\)\.

## 3Spectral Preconditioning Improves LLM Fine\-tuning

Throughout this paper,𝐖∈ℝdout×din\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}denotes a pretrained linear layer with forward pass𝐲=𝐖𝐱\\mathbf\{y\}=\\mathbf\{W\}\\mathbf\{x\}\. And we use𝐖′\\mathbf\{W\}^\{\\prime\}to denote the fine\-tuned weight\.

### 3\.1Observations from Full FT Training Dynamics

Both Full FT and existing PEFT methods update weights without considering the spectral structure learned during pretraining, risking disruption of the pretrained feature geometry\. We hypothesize that fine\-tuning should instead preserve and exploit this structure\. To investigate this, we analyze the training dynamics of Full FT on LLaMA\-3\-8B fine\-tuned on Math\-10KHuet al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib20)\)\(Figure[3](https://arxiv.org/html/2605.22869#S3.F3)\)\.

![Refer to caption](https://arxiv.org/html/2605.22869v1/x3.png)Figure 3:\(a\)Gradient lives outside singular basis of pretrained weight, but weight keep stable\.\(b\)Singular values shift selectively, while singular vectors rotate broadly\. \(a\) \(b\) demonstrates layer 15 q\_proj result, the full sweeps are in Figures[6](https://arxiv.org/html/2605.22869#A6.F6)and[7](https://arxiv.org/html/2605.22869#A6.F7)\(Appendix\)\.\(c\)SVD FT outperforms Full FT on both target domain and source domain\.##### Gradients are spectrally diffuse, while weight spectrum remain stable\.

Let𝐖=𝐔​𝚺​𝐕⊤\\mathbf\{W\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\}denote the SVD of a pretrained weight\. We quantify how much a matrix𝐌\\mathbf\{M\}lies in the pretrained column spacecol⁡\(𝐔\)\\operatorname\{col\}\(\\mathbf\{U\}\)using the*column\-space energy ratio*:

ρ​\(𝐌;𝐔\)=‖𝐔⊤​𝐌‖F2‖𝐌‖F2\.\\rho\(\\mathbf\{M\};\\,\\mathbf\{U\}\)\\;=\\;\\frac\{\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{M\}\\\|\_\{F\}^\{2\}\}\{\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\}\\,\.\(1\)We haveρ∈\[0,1\]\\rho\\in\[0,1\]andρ=1\\rho=1indicates full alignment withspan​\(𝐔\)\\mathrm\{span\}\(\\mathbf\{U\}\)\. Gaussian random matrices has expected energy ratiorank⁡\(𝐔\)/dout\\operatorname\{rank\}\(\\mathbf\{U\}\)/d\_\{\\text\{out\}\}\(see Appendix[E\.2](https://arxiv.org/html/2605.22869#A5.SS2)\)\. Tracking the energy ratio of the gradient𝐆𝐖\\mathbf\{G\}\_\{\\mathbf\{W\}\}during Full FT \(Figure[3](https://arxiv.org/html/2605.22869#S3.F3)a\), we observe thatρ​\(𝐆𝐖,𝐔\)\\rho\(\\mathbf\{G\}\_\{\\mathbf\{W\}\},\\mathbf\{U\}\)stays near the random baseline, showing gradients are updating all directions and are not particularly aligned with pretrained directions\. In contrast,ρ​\(𝐖′,𝐔\)≈1\\rho\(\\mathbf\{W\}^\{\\prime\},\\mathbf\{U\}\)\\approx 1throughout training, indicating that the Full FT implicitly preserves the weight’s spectral structure despite diffusive gradient updates\.

##### Singular values shift selectively; singular vectors rotate broadly\.

Comparing the SVDs of𝐖′\\mathbf\{W\}^\{\\prime\}and𝐖\\mathbf\{W\}\(Figure[3](https://arxiv.org/html/2605.22869#S3.F3)b\), we find that only a few singular values change significantly, while most remain close to their pretrained values\. In contrast, the singular vectors exhibit substantial rotation: cosine similarities between columns of𝐔′\\mathbf\{U\}^\{\\prime\}and𝐔\\mathbf\{U\}vary widely, with no clear bias toward principal or off\-principal directions\. The above findings suggest that Full FT, despite being unconstrained, implicitly preserves the pretrained column space, concentrating spectral changes on a few singular values while allowing broad directional rotation\. This raises a natural question:what if we explicitly enforce these patterns as architectural constraints?

### 3\.2Insights from SVD Fine\-Tuning Experiment

We study a controlled SVD Fine\-tuning \(SVD FT\) to validate the above hypothesis\. We decompose each pretrained weight via SVD,𝐖=𝐔​𝚺​𝐕⊤\\mathbf\{W\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\}, freeze the left\-singular basis𝐔\\mathbf\{U\}, and fine\-tune only𝚺\\mathbf\{\\Sigma\}and𝐕⊤\\mathbf\{V\}^\{\\top\}\. This SVD FT has identical parameter count and initialization to full FT, differing only in parameterization\. It induces a*spectral preconditioning*, where the effective update from𝐕⊤\\mathbf\{V\}^\{\\top\}is

Δ​𝐖\|𝐕=−η​𝐔​𝚺2​𝐔⊤​𝐆𝐖,\\Delta\\mathbf\{W\}\\big\|\_\{\\mathbf\{V\}\}\\;=\\;\-\\eta\\;\\mathbf\{U\}\\mathbf\{\\Sigma\}^\{2\}\\mathbf\{U\}^\{\\top\}\\;\\mathbf\{G\}\_\{\\mathbf\{W\}\}\\,,\(2\)with𝐆𝐖=∂ℒ/∂𝐖\\mathbf\{G\}\_\{\\mathbf\{W\}\}=\\partial\\mathcal\{L\}/\\partial\\mathbf\{W\}\. The projection𝐔𝐔⊤\\mathbf\{U\}\\mathbf\{U\}^\{\\top\}restricts updates tocol⁡\(𝐔\)\\operatorname\{col\}\(\\mathbf\{U\}\), formalizing the column\-space preservation observed earlier, while𝚺\\mathbf\{\\Sigma\}separates magnitude from direction, aligning with the distinct spectral dynamics\.

We evaluate LLaMA\-3\-8B trained with Full FT and SVD FT on Math\-10K, measuring both target domain math reasoning \(GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2605.22869#bib.bib18)\)\) and source domain commonsense reasoning performance \(Figure[3](https://arxiv.org/html/2605.22869#S3.F3)c\)\. We observe that: \(1\) SVD FT improves target\-domain performance, indicating that gradients withincol⁡\(𝐔\)\\operatorname\{col\}\(\\mathbf\{U\}\)suffice for effective learning; \(2\) SVD FT improves source\-domain generalization, whereas Full FT degrades relative to the pretrained model\.

This experiment suggests thatspectral preconditioning better preserves pretrained structure, mitigating forgetting and promoting generalizable representations\.

## 4TheFuRAFramework

![Refer to caption](https://arxiv.org/html/2605.22869v1/x4.png)Figure 4:\(a\)Effective rank ofΔ​𝐖\\Delta\\mathbf\{W\}; full sweep is in Figure[8](https://arxiv.org/html/2605.22869#A6.F8)\(Appendix\)\.\(b\)Singular\-direction heatmaps show sharper active/inactive separation, reflecting spectral preconditioning from frozen𝐋\\mathbf\{L\}\.\(c\)FuRAbetter preserves pretrained capabilities\.SVD FT demonstrates the benefits of spectral preconditioning but is impractical: it retains a full\-size trainable factor and doubles forward\-pass cost\. We proposeFuRA, which addresses both issues using a Block Tensor\-Train \(BTT\) parameterization\.

FuRAreparameterizes each linear layer with a 2\-core Block Tensor\-Train \(BTT\) decompositionQiuet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib33)\), which can be interpreted as block\-wise SVD\. The full\-rank 2\-core BTT can be used to represent arbitrary matrix\. By carefully selecting a special block size, we can perform spectral preconditioned update as for the SVD FT, while with a cost on par with LoRA\.

##### Block tensor\-train \(BTT\) decomposition\.

We focus on 2\-core decomposition case\. Given𝐖∈ℝdout×din\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}, choose anm×nm\\times ngrid such thatdout=m​ad\_\{\\text\{out\}\}=maanddin=n​bd\_\{\\text\{in\}\}=nb\. Each block𝐖i​j∈ℝa×b\\mathbf\{W\}\_\{ij\}\\in\\mathbb\{R\}^\{a\\times b\}is factorized into𝐋i​j∈ℝa×r\\mathbf\{L\}\_\{ij\}\\in\\mathbb\{R\}^\{a\\times r\}and𝐑i​j∈ℝr×b\\mathbf\{R\}\_\{ij\}\\in\\mathbb\{R\}^\{r\\times b\}, forming cores𝓛∈ℝm×n×a×r\\bm\{\\mathcal\{L\}\}\\in\\mathbb\{R\}^\{m\\times n\\times a\\times r\}and𝓡∈ℝm×n×r×b\\bm\{\\mathcal\{R\}\}\\in\\mathbb\{R\}^\{m\\times n\\times r\\times b\}\. The matrix\-vector product𝐲=𝐖𝐱\\mathbf\{y\}=\\mathbf\{W\}\\mathbf\{x\}becomes

yα​β=∑γ​σ𝓛α​β​γ​σ​∑δ𝓡σ​β​γ​δ​𝐱γ​δ\.y\_\{\\alpha\\beta\}=\\sum\_\{\\gamma\\sigma\}\\bm\{\\mathcal\{L\}\}\_\{\\alpha\\beta\\gamma\\sigma\}\\sum\_\{\\delta\}\\bm\{\\mathcal\{R\}\}\_\{\\sigma\\beta\\gamma\\delta\}\\,\\mathbf\{x\}\_\{\\gamma\\delta\}\.\(3\)At full rankr=min⁡\(a,b\)r=\\min\(a,b\), this BTT is applying full\-rank SVD for each individual block and hence the representation is equivalent to the original dense matrix\. For a square, full\-rank decomposition \(dout=din=dd\_\{\\text\{out\}\}=d\_\{\\text\{in\}\}=d,m=n=a=b=r=dm=n=a=b=r=\\sqrt\{d\}\), the total parameter count and MVM FLOPs are both2​d22d^\{2\}Qiuet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib33)\), same as SVD\.

##### TheFuRAparameterization\.

We decompose each pretrained weight using a parameter\-efficient BTT configuration and keep the diagonal singular values as separate trainable parameters\. Settingm=1m=1yields blocks with a*large*core𝐋k∈ℝdout×r\\mathbf\{L\}\_\{k\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times r\}and a*small*core𝐑k∈ℝr×b\\mathbf\{R\}\_\{k\}\\in\\mathbb\{R\}^\{r\\times b\}withb≪dinb\\ll d\_\{\\text\{in\}\}\. At full rankr=br=b, this decomposition is lossless while only introducesdin​bd\_\{\\text\{in\}\}badditional parameters\. For each block, full\-rank SVD yields𝐖k=𝐔k​𝚺k​𝐕k⊤\\mathbf\{W\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\mathbf\{\\Sigma\}\_\{k\}\\mathbf\{V\}\_\{k\}^\{\\top\}, and we set𝐋k=𝐔k∈ℝdout×b\\mathbf\{L\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\\!\\in\\\!\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times b\},𝐒k=diag​\(𝚺k\)∈ℝb\\mathbf\{S\}\_\{k\}=\\mathrm\{diag\}\(\\mathbf\{\\Sigma\}\_\{k\}\)\\\!\\in\\\!\\mathbb\{R\}^\{b\},𝐑k=𝐕k⊤∈ℝb×b\\mathbf\{R\}\_\{k\}=\\mathbf\{V\}\_\{k\}^\{\\top\}\\\!\\in\\\!\\mathbb\{R\}^\{b\\times b\}\. Stacking blocks, we got BTT cores𝐋=\[𝐋1,…,𝐋n\]\\mathbf\{L\}=\[\\mathbf\{L\}\_\{1\},\\dots,\\mathbf\{L\}\_\{n\}\],𝐒=\[𝐒1,…,𝐒n\]\\mathbf\{S\}=\[\\mathbf\{S\}\_\{1\},\\dots,\\mathbf\{S\}\_\{n\}\],𝐑=\[𝐑1,…,𝐑n\]\\mathbf\{R\}=\[\\mathbf\{R\}\_\{1\},\\dots,\\mathbf\{R\}\_\{n\}\]\.FuRAfreezes𝐋\\mathbf\{L\}and trains only𝐒\\mathbf\{S\}and𝐑\\mathbf\{R\}\. The fine\-tuned weight is assembled column\-block\-wise:

𝐖k′=𝐋k​diag​\(𝐒k′\)​𝐑k′,𝐖′=\[𝐖1′​\|𝐖2′\|​⋯\|𝐖n′\],\\mathbf\{W\}^\{\\prime\}\_\{k\}\\;=\\;\\mathbf\{L\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}^\{\\prime\}\_\{k\}\)\\,\\mathbf\{R\}^\{\\prime\}\_\{k\},\\qquad\\mathbf\{W\}^\{\\prime\}\\;=\\;\\bigl\[\\,\\mathbf\{W\}^\{\\prime\}\_\{1\}\\,\\big\|\\,\\mathbf\{W\}^\{\\prime\}\_\{2\}\\,\\big\|\\,\\cdots\\,\\big\|\\,\\mathbf\{W\}^\{\\prime\}\_\{n\}\\,\\bigr\],\(4\)where\[⋅\|⋅\]\[\\cdot\|\\cdot\]denotes horizontal \(column\-block\) concatenation\. Algorithm[1](https://arxiv.org/html/2605.22869#alg1)in Appendix summarizes the procedure\.

##### Complexity analysis\.

The block\-SVD initialization is aone\-timecost \(e\.g\.∼1\\sim 1minute for 8B model\)\. During training,FuRAmatches LoRA in both memory and step time \(Table[1](https://arxiv.org/html/2605.22869#S4.T1)\)\. The trainable count is\|𝐑\|\+\|𝐒\|=n​r​b\+n​r≈d3/2\|\\mathbf\{R\}\|\+\|\\mathbf\{S\}\|=nrb\+nr\\approx d^\{3/2\}, roughly2%2\\%of pretrained weight size atd=4096d=4096; the𝐋\\mathbf\{L\}\-stage reduces to a single dense MVM and the𝐑\\mathbf\{R\}\-stage to a batched BMM, both well supported in deep learning frameworks; after training, the cores merge back into a single dense matrix so there is no additional inference overhead\. We defer the full breakdown to Appendix[B](https://arxiv.org/html/2605.22869#A2)\.

Table 1:System overhead comparison\. Avg is the average accuracy on LLaMA\-3\-8B commonsense SFT Detail setups and analysis are in Appendix[B\.1](https://arxiv.org/html/2605.22869#A2.SS1)\.Advantages ofFuRA\.This single design choice delivers the following properties

- •*Full\-rank capacity\.*Each block’s update lies incol⁡\(𝐋k\)\\operatorname\{col\}\(\\mathbf\{L\}\_\{k\}\), and across blocks these subspaces span the full output space, leavingΔ​𝐖\\Delta\\mathbf\{W\}unconstrained in rank\. The optimizer, rather than a preset rank, controls layer capacity\. As shown in Fig\.[4](https://arxiv.org/html/2605.22869#S4.F4)\(a\), the effective rank ofFuRAupdate is significantly higher than LoRA\. The lower effective rank compared to Full FT suggests that the spectral preconditioner focuses capacity on task\-relevant directions rather than distributing it uniformly\. Importantly, the full\-rank block\-wise singular basis is preserved, avoiding commitment to specific spectral subspaces\. We formalize this in Section[5](https://arxiv.org/html/2605.22869#S5), Claim 1\.
- •*Spectral preconditioning\.*The spectral preconditioning from SVD FT \(Section[3\.2](https://arxiv.org/html/2605.22869#S3.SS2), Eq\. \([2](https://arxiv.org/html/2605.22869#S3.E2)\)\) extends naturally toFuRA’s block structure\. The trainable𝐒\\mathbf\{S\}decouples magnitude and direction, enabling finer spectral control, similar in spirit to DoRALiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\)\. As shown in Figure[4](https://arxiv.org/html/2605.22869#S4.F4)\(b\), the optimizer selectively updates important singular directions while preserving others\. The full preconditioner is derived in Section[5](https://arxiv.org/html/2605.22869#S5), Claim 2\.
- •*Better generalization\.*FuRAoutperforms Full FT and LoRA on target\-domain math reasoning using only∼2%\{\\sim\}2\\%trainable parameters, while better preserving \(improving\) source\-domain commonsense reasoning accuracy \[Fig\.[4](https://arxiv.org/html/2605.22869#S4.F4)\(c\)\]\. By restricting updates to the pretrained column space, it favors reweighting existing features over introducing arbitrary directions that may overfit to the fine\-tuning distribution\.

## 5Theoretical Insights

In this section, we present two theoretical claims regarding the theoretical properties ofFuRA\. We support each claim with a short derivation and provide the full proofs to Appendix[E\.2](https://arxiv.org/html/2605.22869#A5.SS2)\.

##### Claim 1 \(Full\-rank update capacity\)\.

AlthoughFuRAtrains only𝐑\\mathbf\{R\}and𝐒\\mathbf\{S\}, the updateΔ​𝐖\\Delta\\mathbf\{W\}has full\-rank capacity: it can reach any rank up tomin⁡\(din,dout\)\\min\(d\_\{\\text\{in\}\},d\_\{\\text\{out\}\}\)whenever the pretrained weight𝐖\\mathbf\{W\}is full rank, in contrast to LoRA\-style adapters that hard\-caprank⁡\(Δ​𝐖\)≤r\\operatorname\{rank\}\(\\Delta\\mathbf\{W\}\)\\leq r\.

###### Proposition 5\.1\(No explicit rank cap\)\.

Under theFuRAparameterization in Eq\. \([4](https://arxiv.org/html/2605.22869#S4.E4)\), with𝐋\\mathbf\{L\}frozen and𝐒,𝐑\\mathbf\{S\},\\mathbf\{R\}trainable, the updateΔ​𝐖=𝐖′−𝐖\\Delta\\mathbf\{W\}=\\mathbf\{W\}^\{\\prime\}\-\\mathbf\{W\}is not rank\-constrained \(Lemma[E\.2](https://arxiv.org/html/2605.22869#A5.Thmtheorem2)\)\.

Sketch of Proof\.Per block,Δ​𝐖k\\Delta\\mathbf\{W\}\_\{k\}lies incol⁡\(𝐋k\)\\operatorname\{col\}\(\\mathbf\{L\}\_\{k\}\)\(Propositions[E\.3](https://arxiv.org/html/2605.22869#A5.Thmtheorem3)and[E\.4](https://arxiv.org/html/2605.22869#A5.Thmtheorem4), appendix\); across thennblocks, these subspaces collectively spancol⁡\(𝐖\)\\operatorname\{col\}\(\\mathbf\{W\}\)\.

##### Claim 2 \(Spectral preconditioning\)\.

The effective update underFuRAis not standard SGD on𝐖\\mathbf\{W\}: it appliescolumn\-space projectionandsingular\-value preconditioning, both induced by the frozen block\-wise SVD basis\. We provide the explanations below\.

Consider a single block with SVD𝐖k=𝐔k​𝚺k​𝐕k⊤\\mathbf\{W\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\mathbf\{\\Sigma\}\_\{k\}\\mathbf\{V\}\_\{k\}^\{\\top\}and thekk\-th block input𝐱k\\mathbf\{x\}\_\{k\}\. Let𝐠=∂ℒ/∂𝐲\\mathbf\{g\}=\\partial\\mathcal\{L\}/\\partial\\mathbf\{y\}and𝐆k=𝐠𝐱k⊤\\mathbf\{G\}\_\{k\}=\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}denote the unconstrained gradient\. Under the defaultFuRAsetting \(train𝐑\\mathbf\{R\}and𝐒\\mathbf\{S\}, freeze𝐋=𝐔\\mathbf\{L\}=\\mathbf\{U\}\), the forward pass is𝐲k=𝐔k​diag​\(𝐒k\)​𝐑k​𝐱k\\mathbf\{y\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{R\}\_\{k\}\\,\\mathbf\{x\}\_\{k\}\. A single gradient step yields

Δ​𝐖k=−η​𝐔k​diag​\(𝐒k\)2​𝐔k⊤​𝐆k⏟from​𝐑​update\+𝐔k​diag​\(Δ​𝐒k\)​𝐕k⊤⏟from​𝐒​update,\\Delta\\mathbf\{W\}\_\{k\}\\;=\\;\\underbrace\{\-\\eta\\,\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{G\}\_\{k\}\}\_\{\\text\{from \}\\mathbf\{R\}\\text\{ update\}\}\\;\+\\;\\underbrace\{\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\Delta\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{V\}\_\{k\}^\{\\top\}\}\_\{\\text\{from \}\\mathbf\{S\}\\text\{ update\}\},\(5\)wherediag​\(𝐒k\)2\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}arises becausediag​\(𝐒k\)\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)appears twice in the chain rule: in∂ℒ/∂𝐑k\\partial\\mathcal\{L\}/\\partial\\mathbf\{R\}\_\{k\}and in𝐔k​diag​\(𝐒k\)​δ​𝐑k\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\delta\\mathbf\{R\}\_\{k\}\. The preconditioner𝐔k​diag​\(𝐒k\)2​𝐔k⊤\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{U\}\_\{k\}^\{\\top\}in the first term combines two effects:

- •Column\-space projection\.Since𝐔k\\mathbf\{U\}\_\{k\}has orthonormal columns \(𝐔k⊤​𝐔k=𝐈\\mathbf\{U\}\_\{k\}^\{\\top\}\\mathbf\{U\}\_\{k\}=\\mathbf\{I\}, but𝐔k​𝐔k⊤≠𝐈\\mathbf\{U\}\_\{k\}\\mathbf\{U\}\_\{k\}^\{\\top\}\\neq\\mathbf\{I\}for rectangular blocks\), the updateΔ​𝐖k\\Delta\\mathbf\{W\}\_\{k\}is restricted tocol​\(𝐔k\)\\mathrm\{col\}\(\\mathbf\{U\}\_\{k\}\), the pretrained left\-singular subspace\. InFuRA,𝐔k\\mathbf\{U\}\_\{k\}is always rectangular, making this projection non\-trivial\.
- •Singular\-value preconditioning\.Withincol​\(𝐔k\)\\mathrm\{col\}\(\\mathbf\{U\}\_\{k\}\), gradients are scaled byσi2\\sigma\_\{i\}^\{2\}along each singular direction, amplifying high\-signal components and suppressing low\-signal ones\. This induces an adaptive, spectrum\-dependent learning rate\.

##### Design choices\.

FuRAexposes two design axes; Table[2](https://arxiv.org/html/2605.22869#S5.T2)enumerates all corners\.

\(1\) Subspace preservation\.Choosing orientationm=1m\{=\}1with𝐋\\mathbf\{L\}frozen \(the default\) preserves the per\-block pretrained*output*subspace: every block update satisfiescol⁡\(Δ​𝐖k\)⊆col⁡\(𝐔k\)\\operatorname\{col\}\(\\Delta\\mathbf\{W\}\_\{k\}\)\\subseteq\\operatorname\{col\}\(\\mathbf\{U\}\_\{k\}\)\. The symmetric choicen=1n\{=\}1with𝐑\\mathbf\{R\}frozen instead preserves the per\-block pretrained*input*subspace,row⁡\(Δ​𝐖k\)⊆row⁡\(𝐕k⊤\)\\operatorname\{row\}\(\\Delta\\mathbf\{W\}\_\{k\}\)\\subseteq\\operatorname\{row\}\(\\mathbf\{V\}\_\{k\}^\{\\top\}\)\. Both orientations share the same trainable parameter count and full\-rank update capacity \(Claim 1\), but bias the optimizer toward different pretrained directions\.

\(2\) Placement of𝐒\\mathbf\{S\}\.Keeping𝐒\\mathbf\{S\}as a separate trainable vector \(default\) yields thediag​\(𝐒k\)2\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}preconditioner of Eq\. \([5](https://arxiv.org/html/2605.22869#S5.E5)\) plus the magnitude term𝐔k​diag​\(Δ​𝐒k\)​𝐕k⊤\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\Delta\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{V\}\_\{k\}^\{\\top\}\. Merging𝐒\\mathbf\{S\}into the trainable core removes spectral reweighting \(“fair” projector𝐔k​𝐔k⊤\\mathbf\{U\}\_\{k\}\\mathbf\{U\}\_\{k\}^\{\\top\}, MiLoRA\-like\); merging it into the frozen core produces a principal\-biaseddiag​\(𝐒k\)2\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\-weighted projector \(PiSSA\-like\)\. Derivations for all four placements: Appendix[E\.3](https://arxiv.org/html/2605.22869#A5.SS3)\.

Table 2:Effective gradient\-inducedΔ​𝐖k\\Delta\\mathbf\{W\}\_\{k\}per design corner ofFuRA\(up to the learning rate−η\-\\eta\)\. Principal\-biased \(PiSSA\-like\) and off\-principal\-biased \(MiLoRA\-like\) updates arise as two corners of the same framework\. Thedefaultconfiguration used in the main experiments is the last row\.

## 6Experiments

##### Setup\.

Tasks\.We evaluateFuRAon two settings of major interest to the LLM community: 1\)*Commonsense reasoning SFT*, in which we fine\-tune on Commonsense\-170KHuet al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib20)\)and evaluate on the standard eight\-task suite \(BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC\-e, ARC\-c, OBQA\)Clarket al\.\([2019](https://arxiv.org/html/2605.22869#bib.bib36)\); Bisket al\.\([2020](https://arxiv.org/html/2605.22869#bib.bib37)\); Sapet al\.\([2019](https://arxiv.org/html/2605.22869#bib.bib38)\); Zellerset al\.\([2019](https://arxiv.org/html/2605.22869#bib.bib39)\); Sakaguchiet al\.\([2020](https://arxiv.org/html/2605.22869#bib.bib40)\); Clarket al\.\([2018](https://arxiv.org/html/2605.22869#bib.bib41)\); Mihaylovet al\.\([2018](https://arxiv.org/html/2605.22869#bib.bib42)\); 2\)*Math reinforcement learning with verifiable rewards*, in which we run GRPOShaoet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib8)\)on a math prompt set and evaluate on four held\-out benchmarks: MATH\-500, AMC23, AIME\-24, and AIME\-25Hendryckset al\.\([2021](https://arxiv.org/html/2605.22869#bib.bib19)\); Team \([2023](https://arxiv.org/html/2605.22869#bib.bib67),[2024](https://arxiv.org/html/2605.22869#bib.bib68),[2025](https://arxiv.org/html/2605.22869#bib.bib69)\)\.Models\.For commonsense SFT we use LLaMA\-2\-7BTouvronet al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib70)\)and LLaMA\-3\-8BGrattafioriet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib17)\); for math RLVR we use Qwen3\-1\.7BQwen Team \([2025](https://arxiv.org/html/2605.22869#bib.bib16)\)and Qwen2\.5\-7BTeamet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib71)\)\. We additionally evaluate 4\-bit quantized fine\-tuning on LLaMA\-3\-8B and LLaMA\-3\-70B\.Baseline methods\.We compare against Full FT and four families of parameter\-efficient methods covering the leading LoRA variants: \(i\)*low\-rank adapters*LoRAHuet al\.\([2022](https://arxiv.org/html/2605.22869#bib.bib1)\)and DoRALiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\); \(ii\)*SVD\-initialized adapters*PiSSAMenget al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib2)\)and MiLoRAWanget al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib3)\); and \(iii\)*full\-rank capacity*RandLoRAAlbertet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib29)\);\. For all baselines we use the rank and other hyperparameters reported as best in the original references\. Full hyperparameters and sweeps are in Appendix[C](https://arxiv.org/html/2605.22869#A3)\.

### 6\.1Commonsense reasoning SFT \(LLaMA\-2\-7B and LLaMA\-3\-8B\)

Table 3:Commonsense reasoning, fine\-tuned on Commonsense\-170K\.Table[3](https://arxiv.org/html/2605.22869#S6.T3)reports per\-task accuracy on the eight\-task commonsense suite for LLaMA\-2\-7B and LLaMA\-3\-8B after training on Commonsense\-170K\.FuRAachieves88\.01%88\.01\\%average accuracy on LLaMA\-3\-8B, outperforming LoRA by\+4\.45\+4\.45, DoRA by\+2\.87\+2\.87, and Full FT by\+1\.27\+1\.27\. Methods that initialize LoRA with principal \(PiSSA,\+2\.44\+2\.44over LoRA\) or off\-principal \(MiLoRA,\+1\.50\+1\.50\) singular directions of the pretrained weight improve over LoRA; full\-rank\-capacity RandLoRA is essentially at LoRA parity \(83\.2883\.28vs\.83\.4683\.46\); while none outperforms Full FT\. In contrast,FuRAcombines full\-rank capacity, preservation of the full\-rank pretrained spectral structure, and spectral preconditioning in a unified design, enabling it to surpass Full FT while tuning only1\.46%1\.46\\%of parameters\.

### 6\.2Math Reinforcement Learning

RL setup\.We run GRPOShaoet al\.\([2024](https://arxiv.org/html/2605.22869#bib.bib8)\)for5050policy\-gradient steps with3232prompts per step and88rollouts per prompt\. The reward is\{0,1\}\\\{0,1\\\}based on symbolic equivalence of the extracted answer\. Rollouts are generated with vLLMKwonet al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib35)\); optimization uses AdamWLoshchilov and Hutter \([2019](https://arxiv.org/html/2605.22869#bib.bib34)\)\. AIME\-24/25 are reported as avg@8 at temperatureT=0\.6T\{=\}0\.6; MATH\-500 and AMC23 use greedy decoding \(T=0T\{=\}0\)\. All runs are on a single NVIDIA H100 \(94GB\) for training\.

Table 4:Math RL with GRPO on Qwen3\-1\.7B and Qwen2\.5\-7B,5050policy\-gradient steps\.Results\.Beyond SFT,FuRAalso outperforms other PEFT methods and matches Full FT in reinforcement learning with verifiable rewards \(RLVR\)\. Table[4](https://arxiv.org/html/2605.22869#S6.T4)reports results on four held\-out mathbenchmarks for Qwen3\-1\.7B and Qwen2\.5\-7B, all results are averaged over 3 independent seeds \(mean±\\pmstd are in Appendix[D\.3](https://arxiv.org/html/2605.22869#A4.SS3)\)\. On*Qwen3\-1\.7B*,FuRAoutperforms Full FT and other PEFT methods on MATH\-500, AMC23, AIME\-24\. PiSSA and MiLoRA underperform in RLVR due to their low\-rank commitment to principal/off\-principal spaces, which does not align with RLVR training dynamicsZhuet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib28)\); Yinet al\.\([2025](https://arxiv.org/html/2605.22869#bib.bib64)\)\.FuRApreserves full\-rank spectrum at a similar parameter budget, thus generalizes better in RLVR\. On*Qwen2\.5\-7B*,FuRAalso matches or outperforms Full FT\.

### 6\.3VLM Visual Instruction Tuning \(LLaVA\-1\.5\-7B\)

We follow the DoRALiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\)setup to perform visual instruction tuning for LLaVA\-1\.5\-7BLiuet al\.\([2023a](https://arxiv.org/html/2605.22869#bib.bib56)\)\. Table[7](https://arxiv.org/html/2605.22869#S6.T7)reports the 7\-task average over VQAv2Goyalet al\.\([2017](https://arxiv.org/html/2605.22869#bib.bib57)\), GQAHudson and Manning \([2019](https://arxiv.org/html/2605.22869#bib.bib58)\), VisWizGurariet al\.\([2018](https://arxiv.org/html/2605.22869#bib.bib59)\), ScienceQALuet al\.\([2022](https://arxiv.org/html/2605.22869#bib.bib60)\), TextVQASinghet al\.\([2019](https://arxiv.org/html/2605.22869#bib.bib61)\), POPELiet al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib62)\), and MMBenchLiuet al\.\([2023b](https://arxiv.org/html/2605.22869#bib.bib63)\)\.FuRAexceeds Full FT by\+1\.1\+1\.1and matches DoRA \(67\.667\.6\) with3\.4×\\mathbf\{3\.4\{\\times\}\}fewer trainable parameters \(1\.37%1\.37\\%vs\.4\.63%4\.63\\%\)\. Per\-task evaluation results are in Appendix[D\.5](https://arxiv.org/html/2605.22869#A4.SS5)\.

### 6\.4QFuRA: 4\-bit quantized fine\-tuning

Following QLoRADettmerset al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib10)\), we construct QFuRA by quantizing the frozen large core𝐋\\mathbf\{L\}in 4\-bit NormalFloat \(NF4\) while keeping the trainable𝐑\\mathbf\{R\}and𝐒\\mathbf\{S\}in bf16\. Tables[7](https://arxiv.org/html/2605.22869#S6.T7)and[7](https://arxiv.org/html/2605.22869#S6.T7)report the 8\-task Commonsense\-170K average on LLaMA\-3\-8B and GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2605.22869#bib.bib18)\)accuracy after fine\-tuning on MetaMathQA\-100KYuet al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib65)\)for LLaMA\-3\-70B\. On LLaMA\-3\-8B, QFuRA reaches87\.30%87\.30\\%, surpassing QLoRADettmerset al\.\([2023](https://arxiv.org/html/2605.22869#bib.bib10)\)by\+3\.41\+3\.41and QDoRALiuet al\.\([2024a](https://arxiv.org/html/2605.22869#bib.bib4)\)by\+0\.96\+0\.96at1\.46%1\.46\\%trainable parameters; on LLaMA\-3\-70B, QFuRA scores83\.7883\.78on GSM8K, beating QLoRA and QDoRA\. The results validate thatFuRAis robust to quantizing the frozen large core\. Per\-task scores are in Appendix[D\.4](https://arxiv.org/html/2605.22869#A4.SS4)\.

Table 5:Visual instruction tuning of LLaVA\-1\.5\-7B\.
Table 6:4\-bit quantized fine\-tuning on LLaMA\-3\-8B\.
Table 7:4\-bit quantized fine\-tuning on LLaMA\-3\-70B\.

### 6\.5Ablations: BlockTT design corners

We ablateFuRAdesign corners on LLaMA\-3\-8B Commonsense\-170K SFT and Qwen3\-1\.7B GRPO math RL\. Table[8](https://arxiv.org/html/2605.22869#S6.T8)comparesFuRAFull \(all BlockTT factors trainable\) and the sixFuRAPEFT corners spanning the two design axes of Section[5](https://arxiv.org/html/2605.22869#S5)\(Claim 2\): block orientation \(m=1m\{=\}1vs\.n=1n\{=\}1\) and placement of𝐒\\mathbf\{S\}\(separate trainable, merged into the frozen core, or merged into the trainable core\)\.

Training every factor \(FuRAFull\) is the best, but the PEFT variant closes nearly the entire gap \(87\.9187\.91vs\.88\.0488\.04SFT; matching RL\), confirming that the gain comes from full\-rank spectral preconditioning rather than additional trainable capacity\. Across the six PEFT corners the default preserving output subspace \(m=1m=1\) and keeping singular values as separate parameters \(𝑳​𝑺¯​𝑹¯\{\\color\[rgb\]\{0\.1171875,0\.3125,0\.78515625\}\\bm\{L\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{S\}\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{R\}\}\}\) wins on both SFT and RL\. Intuitively, freezing𝐋\\mathbf\{L\}retains the pretrained output feature space while letting the trainable𝐑\\mathbf\{R\}remix based on new input patterns in the fine\-tuning data; freezing𝐑\\mathbf\{R\}instead pins the input subspace and could filter out task\-specific input features that the fine\-tuning data carry\. We additionally ablate the BlockTT shape factorization\(n,b\)\(n,b\)of the input dimension and show that balanced factorization achieves better result; results are in Appendix[D\.6](https://arxiv.org/html/2605.22869#A4.SS6)\.

Table 8:Ablation study overFuRAdesign corners\. Color code:blue= frozen,red= trainable\.

## 7Conclusion, Limitation, and Broader Impact

We have identified spectral preconditioning as a missing ingredient in current fine\-tuning, and have proposedFuRA, which delivers full\-rank update capacity, spectral preconditioning, and LoRA\-level parameter efficiency from a single architectural choice\. We have demonstrated that in both SFT and RLVRFuRAmatches or surpasses Full FT with level step time and GPU memory on par with LoRA\. Several directions remain open: a deeper theoretical account of*why*spectral\-preconditioned updates improve generalization; initializations from decompositions beyond SVD; custom kernels to further accelerate the block tensor\-train contraction; and a stronger QFuRAthat exploits the orthonormal rows of the frozen core𝐋\\mathbf\{L\}for more aggressive quantization\. We hope these directions inspire further research on the spectral structure of pretrained models and the design of parameter\-efficient adaptation\.FuRAcould reduce computation and energy cost to access high\-quality model adaptation and avoid excessive energy spent on tuning LoRA ranks\. It inherits the general societal risks of LLM fine\-tuning but introduces no new generative capabilities beyond those of the base checkpoints\.

## References

- \[1\]\(2025\)RandLoRA: full\-rank parameter\-efficient fine\-tuning of large models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p2.2),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.8.8.1)\.
- \[2\]Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi\(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[3\]Z\. Chen, R\. R\. Yang, S\. Singh, and M\. Soljacic\(2024\)QuanTA: efficient high\-rank fine\-tuning of LLMs with quantum\-informed tensor adaptation\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p2.2)\.
- \[4\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[5\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? Try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[6\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§D\.1](https://arxiv.org/html/2605.22869#A4.SS1.p1.23),[§3\.2](https://arxiv.org/html/2605.22869#S3.SS2.p2.1),[§6\.4](https://arxiv.org/html/2605.22869#S6.SS4.p1.8)\.
- \[7\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p4.4),[§2](https://arxiv.org/html/2605.22869#S2.p1.4),[§6\.4](https://arxiv.org/html/2605.22869#S6.SS4.p1.8),[Table 7](https://arxiv.org/html/2605.22869#S6.T7.10.2.2.3),[Table 7](https://arxiv.org/html/2605.22869#S6.T7.16.2.2.3)\.
- \[8\]Z\. Gao, Q\. Wang, A\. Chen, Z\. Liu, B\. Wu, L\. Chen, and J\. Li\(2024\)Parameter\-efficient fine\-tuning with discrete Fourier transform\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p2.2)\.
- \[9\]Y\. Goyal, T\. Khot, D\. Summers\-Stay, D\. Batra, and D\. Parikh\(2017\)Making the V in VQA matter: elevating the role of image understanding in Visual Question Answering\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[10\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[11\]D\. Gurari, Q\. Li, A\. J\. Stangl, A\. Guo, C\. Lin, K\. Grauman, J\. Luo, and J\. P\. Bigham\(2018\)VizWiz grand challenge: answering visual questions from blind people\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[12\]N\. Halko, P\. Martinsson, and J\. A\. Tropp\(2011\)Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions\.SIAM review53\(2\),pp\. 217–288\.Cited by:[§B\.2](https://arxiv.org/html/2605.22869#A2.SS2.SSS0.Px5.p1.3)\.
- \[13\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the MATH dataset\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[14\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[Table 17](https://arxiv.org/html/2605.22869#A4.T17.23.19.10),[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§2](https://arxiv.org/html/2605.22869#S2.p1.4),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.12.12.1),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.4.4.1),[Table 7](https://arxiv.org/html/2605.22869#S6.T7.4.4.4.3)\.
- \[15\]I\. Hu, R\. Cong, A\. Zhang, A\. Shetty, V\. Lingam, W\. Chou, A\. G\. Dimakis, and S\. Sanghavi\(2024\)LoRTA: low rank tensor adaptation of large language models\.arXiv preprint arXiv:2410\.04060\.Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p1.4)\.
- \[16\]Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. K\. Lee\(2023\)LLM\-Adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.arXiv preprint arXiv:2304\.01933\.Cited by:[§D\.2](https://arxiv.org/html/2605.22869#A4.SS2.SSS0.Px1.p1.3),[§3\.1](https://arxiv.org/html/2605.22869#S3.SS1.p1.1),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[17\]D\. A\. Hudson and C\. D\. Manning\(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[18\]T\. Jiang, S\. Huang, S\. Luo, Z\. Zhang, H\. Huang, F\. Wei, W\. Deng, F\. Sun, Q\. Zhang, D\. Wang, and F\. Zhuang\(2025\)MoRA: high\-rank updating for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p2.2)\.
- \[19\]J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu, K\. Milan, J\. Quan, T\. Ramalho, A\. Grabska\-Barwinska,et al\.\(2017\)Overcoming catastrophic forgetting in neural networks\.Proceedings of the national academy of sciences114\(13\),pp\. 3521–3526\.Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p2.4)\.
- \[20\]D\. J\. Kopiczko, T\. Blankevoort, and Y\. M\. Asano\(2024\)VeRA: vector\-based random matrix adaptation\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p1.4)\.
- \[21\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the ACM Symposium on Operating Systems Principles,Cited by:[§6\.2](https://arxiv.org/html/2605.22869#S6.SS2.p1.6)\.
- \[22\]Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen\(2023\)Evaluating object hallucination in large vision\-language models\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[23\]V\. Lingam, A\. Tejaswi, A\. Vavre, A\. Shetty, G\. K\. Gudur, J\. Ghosh, A\. Dimakis, E\. Choi, A\. Bojchevski, and S\. Sanghavi\(2024\)SVFT: parameter\-efficient fine\-tuning with singular vectors\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p3.2)\.
- \[24\]H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee\(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems,Cited by:[§D\.5](https://arxiv.org/html/2605.22869#A4.SS5.p1.9),[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[25\]S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen\(2024\)DoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning,Cited by:[Table 11](https://arxiv.org/html/2605.22869#A3.T11),[§D\.5](https://arxiv.org/html/2605.22869#A4.SS5.p1.9),[§D\.5](https://arxiv.org/html/2605.22869#A4.SS5.p2.4),[Table 17](https://arxiv.org/html/2605.22869#A4.T17),[Table 17](https://arxiv.org/html/2605.22869#A4.T17.32.28.10),[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§1](https://arxiv.org/html/2605.22869#S1.p4.4),[§2](https://arxiv.org/html/2605.22869#S2.p1.4),[2nd item](https://arxiv.org/html/2605.22869#S4.I1.i2.p1.1),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1),[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5),[§6\.4](https://arxiv.org/html/2605.22869#S6.SS4.p1.8),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.13.13.1),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.5.5.1),[Table 7](https://arxiv.org/html/2605.22869#S6.T7.12.4.4.3),[Table 7](https://arxiv.org/html/2605.22869#S6.T7.18.4.4.3),[Table 7](https://arxiv.org/html/2605.22869#S6.T7.6.6.6.3)\.
- \[26\]W\. Liu, Z\. Qiu, Y\. Feng, Y\. Xiu, Y\. Xue, L\. Yu, H\. Feng, Z\. Liu, J\. Heo, S\. Peng, Y\. Wen, M\. J\. Black, A\. Weller, and B\. Schölkopf\(2024\)Parameter\-efficient orthogonal finetuning via butterfly factorization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p2.2)\.
- \[27\]Y\. Liu, H\. Duan, Y\. Zhang, B\. Li, S\. Zhang, W\. Zhao, Y\. Yuan, J\. Wang, C\. He, Z\. Liu, K\. Chen, and D\. Lin\(2023\)MMBench: is your multi\-modal model an all\-around player?\.arXiv preprint arXiv:2307\.06281\.Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[28\]Z\. Liu, T\. Pang, O\. Balabanov, C\. Yang, T\. Huang, L\. Yin, Y\. Yang, and S\. Liu\(2025\)LIFT the veil for the truth: principal weights emerge after rank reduction for reasoning\-focused supervised fine\-tuning\.InInternational Conference on Machine Learning,Cited by:[§B\.2](https://arxiv.org/html/2605.22869#A2.SS2.SSS0.Px3.p1.9),[§D\.1](https://arxiv.org/html/2605.22869#A4.SS1.p1.23),[§D\.2](https://arxiv.org/html/2605.22869#A4.SS2.SSS0.Px1.p1.3),[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§2](https://arxiv.org/html/2605.22869#S2.p2.2)\.
- \[29\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§6\.2](https://arxiv.org/html/2605.22869#S6.SS2.p1.6)\.
- \[30\]P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. Kalyan\(2022\)Learn to explain: multimodal reasoning via thought chains for science question answering\.InAdvances in Neural Information Processing Systems,Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[31\]F\. Meng, Z\. Wang, and M\. Zhang\(2024\)PiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§2](https://arxiv.org/html/2605.22869#S2.p3.2),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.6.6.1)\.
- \[32\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? A new dataset for open book question answering\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[33\]A\. Novikov, D\. Podoprikhin, A\. Osokin, and D\. Vetrov\(2015\)Tensorizing neural networks\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p1.4)\.
- \[34\]I\. V\. Oseledets\(2011\)Tensor\-train decomposition\.SIAM Journal on Scientific Computing33\(5\),pp\. 2295–2317\.Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p1.4)\.
- \[35\]S\. Qiu, A\. Potapczynski, M\. Finzi, M\. Goldblum, and A\. G\. Wilson\(2024\)Compute better spent: replacing dense layers with structured matrices\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p1.4),[§4](https://arxiv.org/html/2605.22869#S4.SS0.SSS0.Px1.p1.14),[§4](https://arxiv.org/html/2605.22869#S4.p2.1)\.
- \[36\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[37\]K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi\(2020\)WinoGrande: an adversarial Winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[38\]M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi\(2019\)Social IQa: commonsense reasoning about social interactions\.InProceedings of the Conference on Empirical Methods in Natural Language Processing,Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[39\]J\. Schulman and Thinking Machines Lab\(2025\)LoRA without regret\.Note:[https://thinkingmachines\.ai/blog/lora/](https://thinkingmachines.ai/blog/lora/)Blog postCited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1)\.
- \[40\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1),[§6\.2](https://arxiv.org/html/2605.22869#S6.SS2.p1.6)\.
- \[41\]A\. Singh, V\. Natarajan, M\. Shah, Y\. Jiang, X\. Chen, D\. Batra, D\. Parikh, and M\. Rohrbach\(2019\)Towards VQA models that can read\.InIEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§6\.3](https://arxiv.org/html/2605.22869#S6.SS3.p1.5)\.
- \[42\]Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[43\]Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[44\]Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[45\]Q\. Team, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang,et al\.\(2025\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[46\]H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.External Links:[Link](https://arxiv.org/abs/2307.09288)Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[47\]H\. Wang, Y\. Li, S\. Wang, G\. Chen, and Y\. Chen\(2024\)MiLoRA: harnessing minor singular components for parameter\-efficient LLM finetuning\.arXiv preprint arXiv:2406\.09044\.Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p3.2),[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1),[Table 4](https://arxiv.org/html/2605.22869#S6.T4.7.7.7.1)\.
- \[48\]X\. Yang, J\. Leng, G\. Guo, J\. Zhao, R\. Nakada, L\. Zhang, H\. Yao, and B\. Chen\(2024\)S2FT: efficient, scalable and generalizable LLM fine\-tuning by structured sparsity\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p2.2)\.
- \[49\]Y\. Yang, K\. Zhen, E\. Banijamali, A\. Mouchtaris, and Z\. Zhang\(2024\)LoRETTA: low\-rank economic tensor\-train adaptation for ultra\-low\-parameter fine\-tuning of large language models\.InProceedings of the Conference of the North American Chapter of the Association for Computational Linguistics,Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p1.4)\.
- \[50\]Q\. Yin, Y\. Wu, Z\. Shen, S\. Li, Z\. Wang, Y\. Li, C\. T\. Leong, J\. Kang, and J\. Gu\(2025\)Evaluating parameter efficient methods for rlvr\.arXiv preprint arXiv:2512\.23165\.Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p3.2),[§6\.2](https://arxiv.org/html/2605.22869#S6.SS2.p2.1)\.
- \[51\]L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. Liu\(2023\)Metamath: bootstrap your own mathematical questions for large language models\.arXiv preprint arXiv:2309\.12284\.Cited by:[§6\.4](https://arxiv.org/html/2605.22869#S6.SS4.p1.8)\.
- \[52\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue,et al\.\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1)\.
- \[53\]R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi\(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the Annual Meeting of the Association for Computational Linguistics,Cited by:[§6](https://arxiv.org/html/2605.22869#S6.SS0.SSS0.Px1.p1.1)\.
- \[54\]F\. Zhang and M\. Pilanci\(2024\)Spectral adapter: fine\-tuning in spectral space\.InAdvances in Neural Information Processing Systems,Cited by:[§E\.2](https://arxiv.org/html/2605.22869#A5.SS2.p1.1),[§2](https://arxiv.org/html/2605.22869#S2.p3.2)\.
- \[55\]Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.22869#S1.p1.1),[§2](https://arxiv.org/html/2605.22869#S2.p1.4)\.
- \[56\]H\. Zhu, Z\. Zhang, H\. Huang, D\. Su, Z\. Liu, J\. Zhao, I\. Fedorov, H\. Pirsiavash, Z\. Sha, J\. Lee, D\. Z\. Pan, Z\. Wang, Y\. Tian, and K\. S\. Tai\(2025\)The path not taken: RLVR provably learns off the principals\.arXiv preprint arXiv:2511\.08567\.Note:NeurIPS 2025 Workshop on Efficient Reasoning \(spotlight\)Cited by:[§2](https://arxiv.org/html/2605.22869#S2.p3.2),[§6\.2](https://arxiv.org/html/2605.22869#S6.SS2.p2.1)\.

## Appendix AFuRAAlgorithm

Algorithm 1FuRA: Full\-Rank Adaptation0:Pretrained weight

𝐖∈ℝdout×din\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}, block count

nn, block width

b=din/nb=d\_\{\\text\{in\}\}/n
1:Reshape

𝐖\\mathbf\{W\}into blocks:

𝓦∈ℝn×dout×b\\bm\{\\mathcal\{W\}\}\\in\\mathbb\{R\}^\{n\\times d\_\{\\text\{out\}\}\\times b\}
2:for

k=1,…,nk=1,\\ldots,ndo

3:

𝐔k,𝚺k,𝐕k←SVD​\(𝓦k\)\\mathbf\{U\}\_\{k\},\\mathbf\{\\Sigma\}\_\{k\},\\mathbf\{V\}\_\{k\}\\leftarrow\\text\{SVD\}\(\\bm\{\\mathcal\{W\}\}\_\{k\}\)⊳\\trianglerightlossless,r=br=b

4:endfor

5:

𝐋k←𝐔k\\mathbf\{L\}\_\{k\}\\leftarrow\\mathbf\{U\}\_\{k\},

𝐒k←diag​\(𝚺k\)\\mathbf\{S\}\_\{k\}\\leftarrow\\mathrm\{diag\}\(\\mathbf\{\\Sigma\}\_\{k\}\),

𝐑k←𝐕k⊤\\mathbf\{R\}\_\{k\}\\leftarrow\\mathbf\{V\}\_\{k\}^\{\\top\}⊳\\trianglerightassign cores

6:Freeze

𝐋\\mathbf\{L\}; set

𝐑\\mathbf\{R\}and

𝐒\\mathbf\{S\}as trainable

7:whiletrainingdo

8:Reshape input:

𝓧∈ℝn×b\\bm\{\\mathcal\{X\}\}\\in\\mathbb\{R\}^\{n\\times b\}
9:Forward:

𝐲=∑k=1n𝐋k​diag​\(𝐒k\)​𝐑k​𝓧k\\mathbf\{y\}=\\sum\_\{k=1\}^\{n\}\\mathbf\{L\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{R\}\_\{k\}\\,\\bm\{\\mathcal\{X\}\}\_\{k\}⊳\\trianglerighttwo batched GEMMs

10:Update

𝐑\\mathbf\{R\}and

𝐒\\mathbf\{S\}via optimizer \(gradient flows through

𝐋\\mathbf\{L\}but

𝐋\\mathbf\{L\}is not updated\)

11:endwhile

12:Deploy:merge cores into

𝐖′∈ℝdout×din\\mathbf\{W\}^\{\\prime\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}via Eq\. \([4](https://arxiv.org/html/2605.22869#S4.E4)\)⊳\\trianglerightno serving overhead

## Appendix BFuRAComplexity analysis

This section expands the brief complexity discussion in Section[4](https://arxiv.org/html/2605.22869#S4)\(and the headline numbers in Table[1](https://arxiv.org/html/2605.22869#S4.T1)\)\.

### B\.1System cost measurement protocol

This section documents the protocol used for the step\-time and peak\-memory numbers in Table[1](https://arxiv.org/html/2605.22869#S4.T1)\.

##### Setup\.

LLaMA\-3\-8B Commonsense\-170K SFT, 300 optimizer steps, single H100 NVL \(95\.895\.8GB\), seed4343\. Per\-device batch88, gradient accumulation22\(effective tokens/step32,76832\{,\}768\), sequence length20482048, bf16 mixed precision, gradient checkpointing on, AdamW withη=2×10−4\\eta\{=\}2\{\\times\}10^\{\-4\}, linear LR schedule with3%3\\%warmup\. Identical for every method; only the adapter recipe varies\.

##### Reported metrics\.

*Step \(s\)*is the median wall\-clock per optimizer step over the final200200steps \(after a100100\-step warmup that excludes adapter\-attach and SVD initialization\)\.*Memory \(GB\)*istorch\.cuda\.max\_memory\_allocatedover the full run, including the warmup window\.\.

### B\.2Results and Analysis

FuRAmatches the per\-step time of LoRA/DoRA at comparable budgets, achieves the*lowest*peak GPU memory, and uniquely enables full\-rank updates\. We detail this below\.

##### Trainable parameters\.

The trainable count is\|𝐑\|\+\|𝐒\|=n​r​b\+n​r\|\\mathbf\{R\}\|\+\|\\mathbf\{S\}\|=nrb\+nr\. For a squared×dd\\times dlayer withn=b=r=dn=b=r=\\sqrt\{d\}, this is≈d3/2\\approx d^\{3/2\}, i\.e\.,𝒪​\(1/d\)\\mathcal\{O\}\(1/\\sqrt\{d\}\)ofd2d^\{2\}\. Atd=4096d=4096,𝐑\\mathbf\{R\}has∼0\.26\{\\sim\}0\.26M parameters, and the overall trainable fraction on LLaMA\-3\-8B is1\.46%1\.46\\%, comparable to LoRA/DoRA \(∼1\.4%\{\\sim\}1\.4\\%\) and far below LIFT’s∼6\.2%\{\\sim\}6\.2\\%\.

##### Forward pass\.

For aFuRAlayer withdin=n⋅bd\_\{\\text\{in\}\}=n\\cdot b, and rankr=br=b, the cores have shapes𝐑∈ℝn×r×b\\mathbf\{R\}\\in\\mathbb\{R\}^\{n\\times r\\times b\},𝐋∈ℝn×dout×r\\mathbf\{L\}\\in\\mathbb\{R\}^\{n\\times d\_\{\\text\{out\}\}\\times r\}, and𝐒=\[𝐒1,…,𝐒n\]∈ℝn×r\\mathbf\{S\}=\[\\mathbf\{S\}\_\{1\},\\dots,\\mathbf\{S\}\_\{n\}\]\\in\\mathbb\{R\}^\{n\\times r\}with each𝐒k∈ℝr\\mathbf\{S\}\_\{k\}\\in\\mathbb\{R\}^\{r\}stored as a vector\. The forward pass applies, for input𝐱∈ℝdin\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{\\text\{in\}\}\}reshaped as𝓧∈ℝn×b\\bm\{\\mathcal\{X\}\}\\in\\mathbb\{R\}^\{n\\times b\},

𝓩k\\displaystyle\\bm\{\\mathcal\{Z\}\}\_\{k\}=𝐑k​𝓧k\\displaystyle=\\mathbf\{R\}\_\{k\}\\bm\{\\mathcal\{X\}\}\_\{k\}\(k=1,…,n\),𝓩∈ℝn×r,\\displaystyle\(k=1,\\dots,n\),\\quad\\bm\{\\mathcal\{Z\}\}\\in\\mathbb\{R\}^\{n\\times r\},𝓩~k\\displaystyle\\widetilde\{\\bm\{\\mathcal\{Z\}\}\}\_\{k\}=diag​\(𝐒k\)​𝓩k,\\displaystyle=\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\bm\{\\mathcal\{Z\}\}\_\{k\},𝐲\\displaystyle\\mathbf\{y\}=∑k=1n𝐋k​𝓩~k,𝐲∈ℝdout\.\\displaystyle=\\sum\_\{k=1\}^\{n\}\\mathbf\{L\}\_\{k\}\\widetilde\{\\bm\{\\mathcal\{Z\}\}\}\_\{k\},\\quad\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\}\.
Table[1](https://arxiv.org/html/2605.22869#S4.T1)reports per\-step wall\-clock, throughput, and peak GPU memory for300300\-step commonsense SFT on LLaMA\-3\-8B\.FuRAruns at0\.0460\.046s/step, matching LoRA/DoRA and4\.3×4\.3\\timesfaster than Full FT\. Peak memory is22\.722\.7GB, lower than LoRA/DoRA and3\.4×3\.4\\timesbelow Full FT\. Efficiency comes from a sequential design: the𝐋\\mathbf\{L\}\-stage reduces to a standard dense MVM, while the𝐑\\mathbf\{R\}\-stage uses fused small MVMs viatorch\.bmm\. Withb=d=64b=\\sqrt\{d\}=64, these align well with tensor cores, improving utilization over LoRA’s tall\-skinny GEMMs\.

##### Memory\.

FuRAachieves lower peak memory than LoRA because its sequential factorization𝐋​𝐒​𝐑​𝐱\\mathbf\{L\}\\,\\mathbf\{S\}\\,\\mathbf\{R\}\\,\\mathbf\{x\}produces a single intermediate activation per layer of sizeℝn×r\\mathbb\{R\}^\{n\\times r\}, whereas LoRA’s parallel residual𝐖𝐱\+𝐁𝐀𝐱\\mathbf\{W\}\\mathbf\{x\}\+\\mathbf\{B\}\\mathbf\{A\}\\mathbf\{x\}stashes*two*additional activations: the down\-projection𝐀𝐱∈ℝT×r\\mathbf\{A\}\\mathbf\{x\}\\in\\mathbb\{R\}^\{T\\times r\}and the input𝐱∈ℝT×d\\mathbf\{x\}\\in\\mathbb\{R\}^\{T\\times d\}feeding𝐁𝐀𝐱\\mathbf\{B\}\\mathbf\{A\}\\mathbf\{x\}\. Compared to sparse fine\-tuning method LIFT\[[28](https://arxiv.org/html/2605.22869#bib.bib7)\],FuRAuses dramatically less memory because LIFT must additionally keep the full bf16 backbone gradient buffer \(∼16\{\\sim\}16GB on88B params\) plus a per\-weight bool mask, whereasFuRAproduces its full\-rank update entirely through the small factored cores𝐑,𝐒\\mathbf\{R\},\\mathbf\{S\}and never materializes a backbone\-sized gradient\.

##### Deployment\.

After training, the three cores𝐋,𝐒,𝐑\\mathbf\{L\},\\mathbf\{S\},\\mathbf\{R\}merge into a single dense matrix𝐖′∈ℝdout×din\\mathbf\{W\}^\{\\prime\}\\in\\mathbb\{R\}^\{d\_\{\\text\{out\}\}\\times d\_\{\\text\{in\}\}\}via Eq\. \([4](https://arxiv.org/html/2605.22869#S4.E4)\), incurring no serving latency relative to the base model\.

##### Initialization cost\.

Block\-SVD initialization is a one\-time cost:52\.952\.9s on LLaMA\-3\-8B \(160 target linear layers\), versus∼\\sim1 s for LoRA/DoRA\. This is<0\.2%<0\.2\\%of full SFT runtime \(10 hours for Commonsense\-170K finetuning\)\. The cost can be further reduced by Fast SVD methods like\[[12](https://arxiv.org/html/2605.22869#bib.bib55)\]\.

## Appendix CHyperparameters

FuRAhyperparameters for Tables[1](https://arxiv.org/html/2605.22869#S4.T1),[3](https://arxiv.org/html/2605.22869#S6.T3),[4](https://arxiv.org/html/2605.22869#S6.T4),[7](https://arxiv.org/html/2605.22869#S6.T7), and[7](https://arxiv.org/html/2605.22869#S6.T7)\. Tables[9](https://arxiv.org/html/2605.22869#A3.T9),[10](https://arxiv.org/html/2605.22869#A3.T10),[11](https://arxiv.org/html/2605.22869#A3.T11), and[13](https://arxiv.org/html/2605.22869#A3.T13)list theFuRAtraining configuration used for the four task suites\. AllFuRAruns use the project default corner:m=1m=1and𝐒\\mathbf\{S\}kept as a separate trainable vector\. The LLaMA\-3\-8B QFuRA row of Table[7](https://arxiv.org/html/2605.22869#S6.T7)uses the same settings as the LLaMA\-3\-8B column of Table[9](https://arxiv.org/html/2605.22869#A3.T9), with the frozen large core𝐋\\mathbf\{L\}NF4\-quantized viabitsandbytesand the optimizer swapped toPagedAdamW8bit\. The LLaMA\-3\-70B QFuRA row of Table[7](https://arxiv.org/html/2605.22869#S6.T7)uses Table[13](https://arxiv.org/html/2605.22869#A3.T13)\.

Table 9:FuRAhyperparameter configuration for the commonsense reasoning tasks \(Table[3](https://arxiv.org/html/2605.22869#S6.T3)\)\.Table 10:FuRAhyperparameter configuration for math GRPO RL \(Table[4](https://arxiv.org/html/2605.22869#S6.T4)\)\. Reward∈\{0,1\}\\in\\\{0,1\\\}viamath\_utils\.is\_equiv; vLLM rollout atgpu\_memory\_utilization=0\.25=0\.25; GRPO clip ratio0\.20\.2, KL coefficient0, no entropy bonus\.Table 11:FuRAhyperparameter configuration for visual instruction tuning of LLaVA\-1\.5\-7B onllava\_v1\_5\_mix665k\(5,197 steps=1=1epoch\)\. Same recipe as the LLaVA\-1\.5 DoRA fine\-tune of\[[25](https://arxiv.org/html/2605.22869#bib.bib4)\]withFuRAsubstituted for DoRA; best LR selected from\{2,3,4\}×10−4\\\{2,3,4\\\}\{\\times\}10^\{\-4\}\.Table 12:Math\-10K SFT hyperparameter configuration for the LR / batch\-size sweep on LLaMA\-3\-8B \(§[D\.1](https://arxiv.org/html/2605.22869#A4.SS1)\)\. All three methods share the same training schedule \(3 epochs on Math\-10K, AdamW, linear LR decay,0\.030\.03warmup ratio, max sequence length20482048, bf16\); only LR and effective batch size vary across the sweep\. Best LRs found per \(method, effective batch size\) are reported in §[D\.1](https://arxiv.org/html/2605.22869#A4.SS1)\.Table 13:QFuRA hyperparameter configuration for math fine\-tuning of LLaMA\-3\-70B on MetaMathQA\-100K \(Table[7](https://arxiv.org/html/2605.22869#S6.T7)\)\. Mirrors the QPiSSA recipe \(η=2×10−5\\eta=2\{\\times\}10^\{\-5\}, batch1×1281\{\\times\}128, sequence length512512\) with the rank\-rrSVD adapter replaced by QFuRA’s quantized BlockTT factorization\. The frozen large core𝐋\\mathbf\{L\}is 4\-bit\-quantized viabitsandbytesstorage layout; the trainable cores𝐑\\mathbf\{R\}and𝐒\\mathbf\{S\}stay in bf16\.
## Appendix DExtended experimental results

This section collects the per\-task scores behind the headline averages reported in the body\. Subsection[D\.3](https://arxiv.org/html/2605.22869#A4.SS3)expands Table[4](https://arxiv.org/html/2605.22869#S6.T4)\(Math RL with GRPO on Qwen3\-1\.7B and Qwen2\.5\-7B\) into per\-seed mean±\\pmstd across 3 seeds\. Subsection[D\.4](https://arxiv.org/html/2605.22869#A4.SS4)expands Table[7](https://arxiv.org/html/2605.22869#S6.T7)\(QFuRA on LLaMA\-3\-8B Commonsense\-170K\) to all eight tasks, Subsection[D\.5](https://arxiv.org/html/2605.22869#A4.SS5)expands Table[7](https://arxiv.org/html/2605.22869#S6.T7)\(FuRAon LLaVA\-1\.5\-7B visual instruction tuning\) to all seven vision\-language benchmarks, and Subsection[D\.1](https://arxiv.org/html/2605.22869#A4.SS1)reports the LR and effective\-batch\-size sweep for LLaMA\-3\-8B SFT on Math\-10K\.

### D\.1LLaMA\-3\-8B Math\-10K SFT: LR and batch\-size sweep

We additionally ran an LR and effective\-batch\-size sweep on LLaMA\-3\-8B SFT on Math\-10K\[[28](https://arxiv.org/html/2605.22869#bib.bib7)\], evaluated on GSM8K\[[6](https://arxiv.org/html/2605.22869#bib.bib18)\]\. Hyperparameters shared across all runs are in Table[12](https://arxiv.org/html/2605.22869#A3.T12)\. We sweep the learning rate at the default effective batch size of1616for each method: Full FT over\{8×10−6,1×10−5,2×10−5,3×10−5\}\\\{8\{\\times\}10^\{\-6\},\\,1\{\\times\}10^\{\-5\},\\,2\{\\times\}10^\{\-5\},\\,3\{\\times\}10^\{\-5\}\\\}, LoRA over\{3×10−5,6×10−5,1×10−4,2×10−4\}\\\{3\{\\times\}10^\{\-5\},\\,6\{\\times\}10^\{\-5\},\\,1\{\\times\}10^\{\-4\},\\,2\{\\times\}10^\{\-4\}\\\}, andFuRAover\{2,3,4,6\}×10−4\\\{2,\\,3,\\,4,\\,6\\\}\{\\times\}10^\{\-4\}\. We then sweep the effective batch size in\{16,64,256\}\\\{16,64,256\\\}, picking the best LR per \(method, batch size\) by the standard square\-root\-rule rescaling around the small\-batch optimum\. Best LRs we found areη=1×10−5\\eta=1\{\\times\}10^\{\-5\}\(Full FT, bsz1616\),η=2×10−5\\eta=2\{\\times\}10^\{\-5\}\(Full FT, bsz6464and256256\);η=6×10−5\\eta=6\{\\times\}10^\{\-5\}\(LoRA, bsz1616\),η=2×10−4\\eta=2\{\\times\}10^\{\-4\}\(LoRA, bsz6464\),η=6×10−4\\eta=6\{\\times\}10^\{\-4\}\(LoRA, bsz256256\);η=3×10−4\\eta=3\{\\times\}10^\{\-4\}\(FuRA, bsz1616\),η=6×10−4\\eta=6\{\\times\}10^\{\-4\}\(FuRA, bsz6464\),η=8×10−4\\eta=8\{\\times\}10^\{\-4\}\(FuRA, bsz256256\)\. Figure[5](https://arxiv.org/html/2605.22869#A4.F5)plots the resulting GSM8K accuracy across the LR sweep at bsz=16=16\(left\) and across the batch\-size sweep with best per\-batch LR \(right\)\.

![Refer to caption](https://arxiv.org/html/2605.22869v1/x5.png)

![Refer to caption](https://arxiv.org/html/2605.22869v1/x6.png)

Figure 5:LLaMA\-3\-8B Math\-10K SFT, GSM8K accuracy\.Left: LR sweep at effective batch size=16=16\.Right: batch\-size sweep\. Peak per method is annotated on the left panel\.
### D\.2LLaMA\-2\-7B and LLaMA\-3\-8B Commonsense\-170K Results

##### Source of the baseline numbers\.

For Tables[3](https://arxiv.org/html/2605.22869#S6.T3), theFuRAresult is the averaged result across 3 random seeds, with specified numbers per seed in Table[14](https://arxiv.org/html/2605.22869#A4.T14)\.FuRAresult has low variance across different random seeds\. The LLaMA\-2\-7B rows and the LLaMA\-3\-8B Full FT / LoRA / DoRA rows, together with the LIFT comparison numbers are reproduced from the hyperparameter setting in LIFT paper\[[28](https://arxiv.org/html/2605.22869#bib.bib7)\], which itself uses the LLM\-Adapters\[[16](https://arxiv.org/html/2605.22869#bib.bib20)\]recipe ofr=64r\{=\}64adapters on five linear modulesq\_proj k\_proj v\_proj up\_proj down\_proj, best learning rate for each method, and others aligned with Tables[9](https://arxiv.org/html/2605.22869#A3.T9)\. Tuning all 7 modules downgrades the LoRA performance\[[16](https://arxiv.org/html/2605.22869#bib.bib20)\]\. We additionally ran the*RandLoRA, MiLoRA, and PiSSA*baselines on LLaMA\-3\-8B in\-house under the same recipe and 8\-task evaluation harness, sweeping the learning rate per method\. RandLoRA and MiLoRA peak atη=1×10−4\\eta=1\{\\times\}10^\{\-4\}; PiSSA peaks atη=2×10−5\\eta=2\{\\times\}10^\{\-5\}\.

Table 14:FuRA \(PEFT\) seed sweep on LLaMA\-3\-8B fine\-tuned on Commonsense\-170K \(3 epochs\)\.

### D\.3Math RL: per\-seed mean±\\pmstd across 3 seeds

This subsection expands Table[4](https://arxiv.org/html/2605.22869#S6.T4)into per\-seed mean±\\pmstd across three independent seeds \(42, 43, 44\) for each \(model, method\) cell\. Sample std is taken acrossn=3n\{=\}3seeds, so the SEM≈0\.6×\\approx 0\.6\\timesstd\. AIME\-24/25 are avg@8 atT=0\.6T\{=\}0\.6; MATH\-500 and AMC23 are greedy@1 atT=0T\{=\}0\. The Paper Table \(Table[4](https://arxiv.org/html/2605.22869#S6.T4)\) reports the per\-seed mean from the same set of runs\.

Table 15:Math RL with GRPO: per\-seed mean±\\pmstd across 3 seeds \(42, 43, 44\)\. The Paper Table \(Table[4](https://arxiv.org/html/2605.22869#S6.T4)\) reports the per\-seed mean of the same runs\.
### D\.4QFuRA: per\-task scores on Commonsense\-170K \(LLaMA\-3\-8B\)

Table 16:4\-bit QFuRA vs\. QLoRA, QDoRA, and bf16 full fine\-tuning on LLaMA\-3\-8B Commonsense\-170K\.Bold= column\-best across the 4\-bit quantized methods\.Table[7](https://arxiv.org/html/2605.22869#S6.T7)\(full results Table[16](https://arxiv.org/html/2605.22869#A4.T16)\) are all our own runs as specified in Tables[9](https://arxiv.org/html/2605.22869#A3.T9)\. QFuRA wins seven of eight per\-task columns among the 4\-bit methods \(all but BoolQ, where QDoRA leads by2\.52\.5pp\), and the gain over QDoRA is concentrated on ARC\-c \(\+1\.4\+1\.4\), ARC\-e \(\+1\.3\+1\.3\), OBQA \(\+2\.6\+2\.6\), and PIQA \(\+1\.9\+1\.9\)\. QFuRA’s training and evaluation hyperparameters are documented in Table[9](https://arxiv.org/html/2605.22869#A3.T9)\(LLaMA\-3\-8B column\) with the only delta being NF4 quantization of𝐋\\mathbf\{L\}viabitsandbytesand thePagedAdamW8bitoptimizer\.

### D\.5VLM visual instruction tuning: per\-task scores \(LLaVA\-1\.5\-7B\)

We follow the exact LLaVA\-1\.5\[[24](https://arxiv.org/html/2605.22869#bib.bib56)\]\+ DoRA\[[25](https://arxiv.org/html/2605.22869#bib.bib4)\]fine\-tuning recipe: 1 epoch onllava\_v1\_5\_mix665k\(5,1975\{,\}197optimizer steps\), per\-device batch44with gradient accumulation44\(effective batch1616\), sequence length20482048, bf16, gradient checkpointing on, AdamW, cosine LR schedule with3%3\\%warmup, andmm\_projector\_lr=2×10−5=2\{\\times\}10^\{\-5\}\.FuRAreplaces DoRA on all seven linear projections of the language tower \(Q, K, V, O, Gate, Up, Down\) under the project default corner \(output\_one\_block/ small core trainable /s\_merged\_to=keep\_trainable/ full rank\), with the CLIP\-L/336 vision encoder and the MLP projector kept in their LLaVA\-1\.5 recipe\. We swept theFuRALR over\{2,3,4\}×10−4\\\{2,3,4\\\}\{\\times\}10^\{\-4\}; LR=3×10−4=3\{\\times\}10^\{\-4\}wins the 7\-task average and is the row reported below\.

Evaluation protocol\.We follow the DoRA paper’s Table 12 conventions: VQAv2 = test\-dev2015 Overall \(server eval\); GQA = testdev\_balanced; VisWiz = test Overall \(server eval\); SQA = full Acc on all2121K questions; VQAT\{\}^\{\\text\{T\}\}= TextVQA val Acc; POPE = mean F1 across\{\\\{random, popular, adversarial\}×100\\\}\{\\times\}100; MMBench = dev split, CircularEval\. Full FT, LoRA, and DoRA columns are quoted from\[[25](https://arxiv.org/html/2605.22869#bib.bib4)\]Table 12\.

Table 17:Per\-task LLaVA\-1\.5\-7B visual instruction tuning results on the 7 vision\-language benchmarks of\[[25](https://arxiv.org/html/2605.22869#bib.bib4)\], Table 12\. Full FT / LoRA / DoRA quoted from the DoRA paper;FuRAis our run at LR=3×10−4=3\{\\times\}10^\{\-4\}\(best of\{2,3,4\}×10−4\\\{2,3,4\\\}\{\\times\}10^\{\-4\}\)\.Bold= column\-best\.Takeaways\.FuRAmatches DoRA’s 7\-task average while training3\.4×\\mathbf\{3\.4\{\\times\}\}fewer parameters \(1\.37%1\.37\\%vs\.4\.63%4\.63\\%of7\.067\.06B model parameters\)\.FuRAwins VisWiz \(\+2\.2\+2\.2\) and TextVQA \(\+1\.1\+1\.1\), ties POPE within0\.60\.6, and trails on the reasoning\-heavy SQA \(−2\.6\-2\.6\) and MMBench \(−1\.6\-1\.6\) splits\. The wall\-clock cost is also lower: at the same effective batch and step count,FuRAtrains in2121h1111m on a single H100 vs\. DoRA’s3535h4646m \(𝟒𝟏%\\mathbf\{41\\%\}faster\)\.

Param count derivation \(1\.37%1\.37\\%\)\.For each linear within\_features=n⋅b=n\\cdot b, the trainable count under the defaultFuRAcorner isbtt\_r\+btt\_s=n​b2\+n​b=in\_features⋅\(b\+1\)\\texttt\{btt\\\_r\}\+\\texttt\{btt\\\_s\}=nb^\{2\}\+nb=\\texttt\{in\\\_features\}\\cdot\(b\{\+\}1\), while the densebtt\_l∈ℝin\_features×out\_features\\texttt\{btt\\\_l\}\\in\\mathbb\{R\}^\{\\texttt\{in\\\_features\}\\times\\texttt\{out\\\_features\}\}core stays frozen\. With3232LLaMA\-2\-7B blocks×7\\times\\,7projections per block,in\_features∈\{4096,11008\}\\texttt\{in\\\_features\}\\in\\\{4096,11008\\\}factor as\(64,64\)\(64,64\)and\(86,128\)\(86,128\)respectively, giving32⋅\(6⋅4096⋅65\+11008⋅129\)≈96\.632\\cdot\(6\\cdot 4096\\cdot 65\+11008\\cdot 129\)\\approx 96\.6M trainable parameters out of LLaVA\-1\.5\-7B’s7\.067\.06B total \(≈95\.4\\approx 95\.4M frombtt\_rplus1\.11\.1M frombtt\_s, with embeddings, lm\_head, norms, the CLIP vision tower, and the MLP projector all kept frozen\):96\.6/7060=1\.37%96\.6/7060=1\.37\\%\.

### D\.6Ablation details

The body of Section[6\.5](https://arxiv.org/html/2605.22869#S6.SS5)reports only the SFT 8\-task average and the MATH\-500/AMC23 RL columns\. Table[19](https://arxiv.org/html/2605.22869#A4.T19)below give the corresponding per\-task SFT scores and Table[19](https://arxiv.org/html/2605.22869#A4.T19)adds the AIME\-24/AIME\-25 RL columns for both ablation slices\. Color code matches the body:blue= frozen,red\+underbar= trainable\.

Table 18:FuRAPEFT design corners, full SFT per\-task results \(LLaMA\-3\-8B Commonsense\-170K\)\.Table 19:FuRAPEFT design corners, full RL benchmarks \(Qwen3\-1\.7B GRPO math\)\.#### Shape factorization ablation

Holding the s\-placement recipe fixed at the headline default𝑳​𝑺¯​𝑹¯\{\\color\[rgb\]\{0\.1171875,0\.3125,0\.78515625\}\\bm\{L\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{S\}\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{R\}\}\}, this sub\-ablation varies how the input dimensiondind\_\{\\text\{in\}\}is split into block sizes\(n,b\)\(n,b\)withn⋅b=dinn\\cdot b=d\_\{\\text\{in\}\}\. We evaluate three regimes on LLaMA\-3\-8B Commonsense\-170K \(1 epoch,η=2×10−4\\eta=2\{\\times\}10^\{\-4\}, seed 43\) covering the full balanced\-to\-extreme axis\.

Table 20:Shape factorization ablation: how the input dimensiondin=n⋅bd\_\{\\text\{in\}\}=n\\cdot bis split into BlockTT block sizes\. LLaMA\-3\-8B, Commonsense\-170K, 1 epoch,η=2×10−4\\eta=2\{\\times\}10^\{\-4\}, seed 43, default s\-placement𝑳​𝑺¯​𝑹¯\{\\color\[rgb\]\{0\.1171875,0\.3125,0\.78515625\}\\bm\{L\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{S\}\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{R\}\}\}\.Bold= column\-best\.Concretely on LLaMA\-3\-8B: Default uses\(n=32,b=128\)\(n\{=\}32,b\{=\}128\)forQ/K/V/OQ/K/V/O\(head\-aligned,n=num\_headsn\{=\}\\text\{num\\\_heads\},b=head\_dimb\{=\}\\text\{head\\\_dim\}\),\(n=64,b=64\)\(n\{=\}64,b\{=\}64\)for Gate/Up, and\(n=112,b=128\)\(n\{=\}112,b\{=\}128\)for Down \(closest factor pair\)\. Unbalanced uses\(n=512,b=8\)\(n\{=\}512,b\{=\}8\)forQ/K/V/OQ/K/V/Oand Gate/Up, and\(n=1792,b=8\)\(n\{=\}1792,b\{=\}8\)for Down\. Extreme usesb=1,n=dinb\{=\}1,n\{=\}d\_\{\\text\{in\}\}on every module\.

A more balanced shape factorization is preferred: pushingbbtoward11\(largenn\) hurts model quality: Avg drops from88\.0688\.06\(balanced\) to79\.9979\.99\(b=8b\{=\}8\) to33\.1833\.18\(b=1b\{=\}1, near chance level on most tasks\)\. Under𝑳​𝑺¯​𝑹¯\{\\color\[rgb\]\{0\.1171875,0\.3125,0\.78515625\}\\bm\{L\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{S\}\}\}\\,\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{R\}\}\}the trainable count is dominated by\|𝐑\|=r⋅din\|\\mathbf\{R\}\|=r\\cdot d\_\{\\text\{in\}\}with effective rankr=min⁡\(a,b\)r=\\min\(a,b\); shrinkingbbshrinks both the achievable rank and the small\-core size, so very smallbbis a double penalty on representational capacity*and*trainable budget\. The Extreme row ends up with more trainable parameters than the Unbalanced row \(0\.14%0\.14\\%vs\.0\.03%0\.03\\%\) because the separately\-held𝑺¯\{\\color\[rgb\]\{0\.78515625,0\.15625,0\.15625\}\\underline\{\\bm\{S\}\}\}vector dominates the parameter count oncer=1r\{=\}1, yet model quality is worse\.

## Appendix EExtended proof

### E\.1Properties of the energy ratio \([1](https://arxiv.org/html/2605.22869#S3.E1)\)

In the following lemma, we summarize the properties of the energy ratio \([1](https://arxiv.org/html/2605.22869#S3.E1)\) to support our observations on the spectral analysis for the Full FT\.

###### Lemma E\.1\(Properties of the Energy Ratio\)\.

Let𝐔∈ℝdout×k\\mathbf\{U\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times k\}have orthonormal columns\. Then:

1. \(i\)\(Boundedness and geometric interpretation\)\.For any𝐌∈ℝdout×din\\mathbf\{M\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}with‖𝐌‖F\>0\\\|\\mathbf\{M\}\\\|\_\{F\}\>0, 0≤ρ​\(𝐌;𝐔\)≤1\.0\\;\\leq\\;\\rho\(\\mathbf\{M\};\\,\\mathbf\{U\}\)\\;\\leq\\;1\\,\.Moreover: - •ρ​\(𝐌;𝐔\)=1\\rho\(\\mathbf\{M\};\\mathbf\{U\}\)=1if and only ifcol​\(𝐌\)⊆col​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{M\}\)\\subseteq\\mathrm\{col\}\(\\mathbf\{U\}\), i\.e\., every column of𝐌\\mathbf\{M\}lies in the column space of𝐔\\mathbf\{U\}\. - •ρ​\(𝐌;𝐔\)=0\\rho\(\\mathbf\{M\};\\mathbf\{U\}\)=0if and only ifcol​\(𝐌\)⟂col​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{M\}\)\\perp\\mathrm\{col\}\(\\mathbf\{U\}\), i\.e\., every column of𝐌\\mathbf\{M\}is orthogonal to the column space of𝐔\\mathbf\{U\}\. - •ρ​\(𝐌;𝐔\)∈\(0,1\)\\rho\(\\mathbf\{M\};\\mathbf\{U\}\)\\in\(0,1\)if and only if𝐌\\mathbf\{M\}spreads its energy betweencol​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{U\}\)and its orthogonal complement\.
2. \(ii\)\(Gaussian baseline\)\.If𝐆∈ℝdout×din\\mathbf\{G\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}has i\.i\.d\. entries with zero mean and varianceσ2\>0\\sigma^\{2\}\>0, then 𝔼​\[ρ​\(𝐆;𝐔\)\]=kdout\.\\mathbb\{E\}\\bigl\[\\rho\(\\mathbf\{G\};\\,\\mathbf\{U\}\)\\bigr\]\\;=\\;\\frac\{k\}\{d\_\{\\mathrm\{out\}\}\}\\,\.\(6\)

###### Proof\.

Part \(i\)\.

Write𝐌=\[𝐦1,…,𝐦din\]\\mathbf\{M\}=\[\\mathbf\{m\}\_\{1\},\\ldots,\\mathbf\{m\}\_\{d\_\{\\mathrm\{in\}\}\}\]column\-wise\. Since𝐔\\mathbf\{U\}has orthonormal columns,𝐏=𝐔𝐔⊤\\mathbf\{P\}=\\mathbf\{U\}\\mathbf\{U\}^\{\\top\}is the orthogonal projector ontocol​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{U\}\)\. Note that‖𝐔⊤​𝐦j‖2=‖𝐏𝐦j‖2\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{m\}\_\{j\}\\\|^\{2\}=\\\|\\mathbf\{P\}\\mathbf\{m\}\_\{j\}\\\|^\{2\}for each column\. We decompose each column as

𝐦j=𝐏𝐦j\+\(𝐈−𝐏\)​𝐦j,\\mathbf\{m\}\_\{j\}=\\mathbf\{P\}\\mathbf\{m\}\_\{j\}\+\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{m\}\_\{j\}\\,,where the two components are orthogonal\. By the Pythagorean theorem:

‖𝐦j‖2=‖𝐏𝐦j‖2\+‖\(𝐈−𝐏\)​𝐦j‖2\.\\\|\\mathbf\{m\}\_\{j\}\\\|^\{2\}=\\\|\\mathbf\{P\}\\mathbf\{m\}\_\{j\}\\\|^\{2\}\+\\\|\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{m\}\_\{j\}\\\|^\{2\}\\,\.Summing over columns:

‖𝐌‖F2=‖𝐔⊤​𝐌‖F2\+‖\(𝐈−𝐏\)​𝐌‖F2\.\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}=\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{M\}\\\|\_\{F\}^\{2\}\+\\\|\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{M\}\\\|\_\{F\}^\{2\}\\,\.Since both terms are non\-negative and their sum equals‖𝐌‖F2\>0\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}\>0, we have0≤‖𝐔⊤​𝐌‖F2≤‖𝐌‖F20\\leq\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{M\}\\\|\_\{F\}^\{2\}\\leq\\\|\\mathbf\{M\}\\\|\_\{F\}^\{2\}, givingρ​\(𝐌;𝐔\)∈\[0,1\]\\rho\(\\mathbf\{M\};\\mathbf\{U\}\)\\in\[0,1\]\.

*Caseρ=1\\rho=1:*ρ​\(𝐌;𝐔\)=1\\rho\(\\mathbf\{M\};\\mathbf\{U\}\)=1iff‖\(𝐈−𝐏\)​𝐌‖F2=0\\\|\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{M\}\\\|\_\{F\}^\{2\}=0iff\(𝐈−𝐏\)​𝐦j=𝟎\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{m\}\_\{j\}=\\mathbf\{0\}for alljjiff𝐦j∈col​\(𝐔\)\\mathbf\{m\}\_\{j\}\\in\\mathrm\{col\}\(\\mathbf\{U\}\)for alljjiffcol​\(𝐌\)⊆col​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{M\}\)\\subseteq\\mathrm\{col\}\(\\mathbf\{U\}\)\.

*Caseρ=0\\rho=0:*ρ​\(𝐌;𝐔\)=0\\rho\(\\mathbf\{M\};\\mathbf\{U\}\)=0iff‖𝐔⊤​𝐌‖F2=0\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{M\}\\\|\_\{F\}^\{2\}=0iff𝐔⊤​𝐦j=𝟎\\mathbf\{U\}^\{\\top\}\\mathbf\{m\}\_\{j\}=\\mathbf\{0\}for alljjiff𝐦j⟂col​\(𝐔\)\\mathbf\{m\}\_\{j\}\\perp\\mathrm\{col\}\(\\mathbf\{U\}\)for alljjiffcol​\(𝐌\)⟂col​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{M\}\)\\perp\\mathrm\{col\}\(\\mathbf\{U\}\)\.

*Caseρ∈\(0,1\)\\rho\\in\(0,1\):*This is the complement of the above two cases:𝐌\\mathbf\{M\}has nonzero projection onto bothcol​\(𝐔\)\\mathrm\{col\}\(\\mathbf\{U\}\)andcol​\(𝐔\)⟂\\mathrm\{col\}\(\\mathbf\{U\}\)^\{\\perp\}, meaning its energy is distributed between the two subspaces\.

Part \(ii\)\.

Let𝐆\\mathbf\{G\}have i\.i\.d\. entries with mean zero and varianceσ2\\sigma^\{2\}\. Denote the columns of𝐔\\mathbf\{U\}by𝐮1,…,𝐮k\\mathbf\{u\}\_\{1\},\\ldots,\\mathbf\{u\}\_\{k\}\.

*Numerator:*

𝔼​\[‖𝐔⊤​𝐆‖F2\]\\displaystyle\\mathbb\{E\}\\bigl\[\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}\\\|\_\{F\}^\{2\}\\bigr\]=∑j=1din∑i=1k𝔼​\[\(𝐮i⊤​𝐠j\)2\]\.\\displaystyle=\\sum\_\{j=1\}^\{d\_\{\\mathrm\{in\}\}\}\\sum\_\{i=1\}^\{k\}\\mathbb\{E\}\\bigl\[\(\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{g\}\_\{j\}\)^\{2\}\\bigr\]\\,\.\(7\)Each𝐮i⊤​𝐠j=∑ℓ=1dout\(𝐮i\)ℓ​Gℓ​j\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{g\}\_\{j\}=\\sum\_\{\\ell=1\}^\{d\_\{\\mathrm\{out\}\}\}\(\\mathbf\{u\}\_\{i\}\)\_\{\\ell\}\\,G\_\{\\ell j\}is a linear combination of independent zero\-mean entries, so

𝔼​\[\(𝐮i⊤​𝐠j\)2\]=Var​\(𝐮i⊤​𝐠j\)=σ2​∑ℓ=1dout\(𝐮i\)ℓ2=σ2​‖𝐮i‖2=σ2\.\\mathbb\{E\}\[\(\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{g\}\_\{j\}\)^\{2\}\]=\\mathrm\{Var\}\(\\mathbf\{u\}\_\{i\}^\{\\top\}\\mathbf\{g\}\_\{j\}\)=\\sigma^\{2\}\\sum\_\{\\ell=1\}^\{d\_\{\\mathrm\{out\}\}\}\(\\mathbf\{u\}\_\{i\}\)\_\{\\ell\}^\{2\}=\\sigma^\{2\}\\\|\\mathbf\{u\}\_\{i\}\\\|^\{2\}=\\sigma^\{2\}\\,\.Therefore𝔼​\[‖𝐔⊤​𝐆‖F2\]=k⋅din⋅σ2\\mathbb\{E\}\[\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}\\\|\_\{F\}^\{2\}\]=k\\cdot d\_\{\\mathrm\{in\}\}\\cdot\\sigma^\{2\}\.

*Denominator:*

𝔼​\[‖𝐆‖F2\]=dout⋅din⋅σ2\.\\mathbb\{E\}\\bigl\[\\\|\\mathbf\{G\}\\\|\_\{F\}^\{2\}\\bigr\]=d\_\{\\mathrm\{out\}\}\\cdot d\_\{\\mathrm\{in\}\}\\cdot\\sigma^\{2\}\\,\.
*Computing𝔼​\[ρ​\(𝐆;𝐔\)\]\\mathbb\{E\}\[\\rho\(\\mathbf\{G\};\\mathbf\{U\}\)\]:*Let𝐏=𝐔𝐔⊤\\mathbf\{P\}=\\mathbf\{U\}\\mathbf\{U\}^\{\\top\}as in Part \(i\)\. By the Pythagorean decomposition established there:

‖𝐆‖F2=‖𝐔⊤​𝐆‖F2\+‖\(𝐈−𝐏\)​𝐆‖F2\.\\\|\\mathbf\{G\}\\\|\_\{F\}^\{2\}=\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}\\\|\_\{F\}^\{2\}\+\\\|\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{G\}\\\|\_\{F\}^\{2\}\\,\.Therefore

ρ​\(𝐆;𝐔\)=‖𝐔⊤​𝐆‖F2‖𝐔⊤​𝐆‖F2\+‖\(𝐈−𝐏\)​𝐆‖F2\.\\rho\(\\mathbf\{G\};\\mathbf\{U\}\)=\\frac\{\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}\\\|\_\{F\}^\{2\}\}\{\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}\\\|\_\{F\}^\{2\}\+\\\|\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{G\}\\\|\_\{F\}^\{2\}\}\\,\.Extend𝐔\\mathbf\{U\}to a full orthonormal basis𝐔¯=\[𝐔∣𝐔⟂\]∈ℝdout×dout\\bar\{\\mathbf\{U\}\}=\[\\mathbf\{U\}\\mid\\mathbf\{U\}\_\{\\perp\}\]\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{out\}\}\}, where𝐔⟂∈ℝdout×\(dout−k\)\\mathbf\{U\}\_\{\\perp\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times\(d\_\{\\mathrm\{out\}\}\-k\)\}\. Define𝐇=𝐔¯⊤​𝐆\\mathbf\{H\}=\\bar\{\\mathbf\{U\}\}^\{\\top\}\\mathbf\{G\}\. Since𝐔¯\\bar\{\\mathbf\{U\}\}is orthogonal and𝐆\\mathbf\{G\}has i\.i\.d\.𝒩​\(0,σ2\)\\mathcal\{N\}\(0,\\sigma^\{2\}\)entries,𝐇\\mathbf\{H\}also has i\.i\.d\.𝒩​\(0,σ2\)\\mathcal\{N\}\(0,\\sigma^\{2\}\)entries \(orthogonal transformations preserve the i\.i\.d\. Gaussian distribution\)\. The firstkkrows of𝐇\\mathbf\{H\}equal𝐔⊤​𝐆\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}and the remainingdout−kd\_\{\\mathrm\{out\}\}\-krows equal𝐔⟂⊤​𝐆=\(𝐈−𝐏\)​𝐆\\mathbf\{U\}\_\{\\perp\}^\{\\top\}\\mathbf\{G\}=\(\\mathbf\{I\}\-\\mathbf\{P\}\)\\mathbf\{G\}expressed in the𝐔⟂\\mathbf\{U\}\_\{\\perp\}basis\. Thus:

‖𝐔⊤​𝐆‖F2=∑i=1k∑j=1dinHi​j2,‖𝐆‖F2=∑i=1dout∑j=1dinHi​j2\.\\\|\\mathbf\{U\}^\{\\top\}\\mathbf\{G\}\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{k\}\\sum\_\{j=1\}^\{d\_\{\\mathrm\{in\}\}\}H\_\{ij\}^\{2\}\\,,\\qquad\\\|\\mathbf\{G\}\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{d\_\{\\mathrm\{out\}\}\}\\sum\_\{j=1\}^\{d\_\{\\mathrm\{in\}\}\}H\_\{ij\}^\{2\}\\,\.LetXℓ=Hℓ2/σ2X\_\{\\ell\}=H\_\{\\ell\}^\{2\}/\\sigma^\{2\}for each entryℓ\\ell\. AllXℓX\_\{\\ell\}are i\.i\.d\.χ2​\(1\)\\chi^\{2\}\(1\)\. Then:

ρ​\(𝐆;𝐔\)=∑ℓ=1k⋅dinXℓ∑ℓ=1dout⋅dinXℓ\.\\rho\(\\mathbf\{G\};\\mathbf\{U\}\)=\\frac\{\\sum\_\{\\ell=1\}^\{k\\cdot d\_\{\\mathrm\{in\}\}\}X\_\{\\ell\}\}\{\\sum\_\{\\ell=1\}^\{d\_\{\\mathrm\{out\}\}\\cdot d\_\{\\mathrm\{in\}\}\}X\_\{\\ell\}\}\\,\.The numerator is a partial sum of the denominator, and all terms are i\.i\.d\. By symmetry, each term contributes equally in expectation to the total sum\. Therefore:

𝔼​\[∑ℓ=1k⋅dinXℓ∑ℓ=1dout⋅dinXℓ\]=k⋅dindout⋅din=kdout,\\mathbb\{E\}\\\!\\left\[\\frac\{\\sum\_\{\\ell=1\}^\{k\\cdot d\_\{\\mathrm\{in\}\}\}X\_\{\\ell\}\}\{\\sum\_\{\\ell=1\}^\{d\_\{\\mathrm\{out\}\}\\cdot d\_\{\\mathrm\{in\}\}\}X\_\{\\ell\}\}\\right\]=\\frac\{k\\cdot d\_\{\\mathrm\{in\}\}\}\{d\_\{\\mathrm\{out\}\}\\cdot d\_\{\\mathrm\{in\}\}\}=\\frac\{k\}\{d\_\{\\mathrm\{out\}\}\}\\,,where the first equality follows from the exchangeability of the\{Xℓ\}\\\{X\_\{\\ell\}\\\}: for i\.i\.d\. positive random variablesX1,…,XNX\_\{1\},\\ldots,X\_\{N\}, the expectation𝔼​\[X1/\(X1\+⋯\+XN\)\]=1/N\\mathbb\{E\}\[X\_\{1\}/\(X\_\{1\}\+\\cdots\+X\_\{N\}\)\]=1/Nby symmetry, so𝔼​\[\(∑ℓ=1nXℓ\)/\(∑ℓ=1NXℓ\)\]=n/N\\mathbb\{E\}\[\(\\sum\_\{\\ell=1\}^\{n\}X\_\{\\ell\}\)/\(\\sum\_\{\\ell=1\}^\{N\}X\_\{\\ell\}\)\]=n/N\. ∎

### E\.2Claim 1: Full\-rank Update Capacity

We follow\[[54](https://arxiv.org/html/2605.22869#bib.bib6)\]to define the rank capacity of an weight updateΔ​W=fθ​\(W\)\\Delta W=f\_\{\\theta\}\(W\)as

ℛ​\(fθ;W\):=maxθ⁡rank​\(fθ​\(W\)\)−minθ⁡rank​\(fθ​\(W\)\)\\mathcal\{R\}\(f\_\{\\theta\};W\):=\\max\_\{\\theta\}\\,\\mathrm\{rank\}\\\!\\left\(f\_\{\\theta\}\(W\)\\right\)\\;\-\\;\\min\_\{\\theta\}\\,\\mathrm\{rank\}\\\!\\left\(f\_\{\\theta\}\(W\)\\right\)\(8\)
which describes the range of matrix ranks the weight update can achieve\.

###### Lemma E\.2\(Rank capacity of slice\-wise spectral\-core adaptation\)\.

Let𝐖∈ℝm×n\\mathbf\{W\}\\in\\mathbb\{R\}^\{m\\times n\}be a full\-rank pretrained weight matrix withrank⁡\(𝐖\)=min⁡\(m,n\)\\operatorname\{rank\}\(\\mathbf\{W\}\)=\\min\(m,n\)\.

\(Column\-sliced form\)\.Supposen=n0​n1n=n\_\{0\}n\_\{1\}, and write

𝐖=\[𝐖1,…,𝐖n1\],𝐖i∈ℝm×n0\.\\mathbf\{W\}=\[\\mathbf\{W\}\_\{1\},\\dots,\\mathbf\{W\}\_\{n\_\{1\}\}\],\\quad\\mathbf\{W\}\_\{i\}\\in\\mathbb\{R\}^\{m\\times n\_\{0\}\}\.For each slice, consider a full\-rank factorization

𝐖i=𝐏i​𝐐i,𝐏i∈ℝm×rc,𝐐i∈ℝrc×n0,rc=min⁡\(m,n0\)\.\\mathbf\{W\}\_\{i\}=\\mathbf\{P\}\_\{i\}\\mathbf\{Q\}\_\{i\},\\quad\\mathbf\{P\}\_\{i\}\\in\\mathbb\{R\}^\{m\\times r\_\{c\}\},\\;\\mathbf\{Q\}\_\{i\}\\in\\mathbb\{R\}^\{r\_\{c\}\\times n\_\{0\}\},\\quad r\_\{c\}=\\min\(m,n\_\{0\}\)\.Fix𝐏i\\mathbf\{P\}\_\{i\}and tune\{𝐐i\}\\\{\\mathbf\{Q\}\_\{i\}\\\}\. Define

fθ​\(𝐖\)=\[𝐏1​𝐐~1,…,𝐏n1​𝐐~n1\]\.f\_\{\\theta\}\(\\mathbf\{W\}\)=\[\\mathbf\{P\}\_\{1\}\\widetilde\{\\mathbf\{Q\}\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\\widetilde\{\\mathbf\{Q\}\}\_\{n\_\{1\}\}\]\.Then

ℛ​\(fθ;𝐖\)=rank⁡\(\[𝐏1,…,𝐏n1\]\)\.\\mathcal\{R\}\(f\_\{\\theta\};\\mathbf\{W\}\)=\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.In particular, ifm≤nm\\leq n, thenℛ​\(fθ;𝐖\)=m\\mathcal\{R\}\(f\_\{\\theta\};\\mathbf\{W\}\)=m\.

\(Row\-sliced form\)\.Supposem=m0​m1m=m\_\{0\}m\_\{1\}, and write

𝐖=\[𝐖1⋮𝐖m1\],𝐖j∈ℝm0×n\.\\mathbf\{W\}=\\begin\{bmatrix\}\\mathbf\{W\}\_\{1\}\\\\ \\vdots\\\\ \\mathbf\{W\}\_\{m\_\{1\}\}\\end\{bmatrix\},\\quad\\mathbf\{W\}\_\{j\}\\in\\mathbb\{R\}^\{m\_\{0\}\\times n\}\.For each slice, consider a full\-rank factorization

𝐖j=𝐔j​𝐕j,𝐔j∈ℝm0×rr,𝐕j∈ℝrr×n,rr=min⁡\(m0,n\)\.\\mathbf\{W\}\_\{j\}=\\mathbf\{U\}\_\{j\}\\mathbf\{V\}\_\{j\},\\quad\\mathbf\{U\}\_\{j\}\\in\\mathbb\{R\}^\{m\_\{0\}\\times r\_\{r\}\},\\;\\mathbf\{V\}\_\{j\}\\in\\mathbb\{R\}^\{r\_\{r\}\\times n\},\\quad r\_\{r\}=\\min\(m\_\{0\},n\)\.Fix𝐕j\\mathbf\{V\}\_\{j\}and tune\{𝐔j\}\\\{\\mathbf\{U\}\_\{j\}\\\}\. Define

gϕ​\(𝐖\)=\[𝐔~1​𝐕1⋮𝐔~m1​𝐕m1\]\.g\_\{\\phi\}\(\\mathbf\{W\}\)=\\begin\{bmatrix\}\\widetilde\{\\mathbf\{U\}\}\_\{1\}\\mathbf\{V\}\_\{1\}\\\\ \\vdots\\\\ \\widetilde\{\\mathbf\{U\}\}\_\{m\_\{1\}\}\\mathbf\{V\}\_\{m\_\{1\}\}\\end\{bmatrix\}\.Then

ℛ​\(gϕ;𝐖\)=rank⁡\(\[𝐕1⊤⋯𝐕m1⊤\]\)\.\\mathcal\{R\}\(g\_\{\\phi\};\\mathbf\{W\}\)=\\operatorname\{rank\}\\\!\\left\(\\begin\{bmatrix\}\\mathbf\{V\}\_\{1\}^\{\\top\}&\\cdots&\\mathbf\{V\}\_\{m\_\{1\}\}^\{\\top\}\\end\{bmatrix\}\\right\)\.In particular, ifn≤mn\\leq m, thenℛ​\(gϕ;𝐖\)=n\\mathcal\{R\}\(g\_\{\\phi\};\\mathbf\{W\}\)=n\.

###### Proof\.

We first prove the column\-sliced case\. By construction,

fθ​\(𝐖\)=\[𝐏1​𝐐~1,…,𝐏n1​𝐐~n1\]\.f\_\{\\theta\}\(\\mathbf\{W\}\)=\[\\mathbf\{P\}\_\{1\}\\widetilde\{\\mathbf\{Q\}\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\\widetilde\{\\mathbf\{Q\}\}\_\{n\_\{1\}\}\]\.For eachii, every column of𝐏i​𝐐~i\\mathbf\{P\}\_\{i\}\\widetilde\{\\mathbf\{Q\}\}\_\{i\}lies incol​\(𝐏i\)\\mathrm\{col\}\(\\mathbf\{P\}\_\{i\}\)\. Hence

col​\(fθ​\(𝐖\)\)⊆col​\(\[𝐏1,…,𝐏n1\]\),\\mathrm\{col\}\(f\_\{\\theta\}\(\\mathbf\{W\}\)\)\\subseteq\\mathrm\{col\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\),which implies

rank⁡\(fθ​\(𝐖\)\)≤rank⁡\(\[𝐏1,…,𝐏n1\]\)\.\\operatorname\{rank\}\(f\_\{\\theta\}\(\\mathbf\{W\}\)\)\\leq\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.Thus

maxθ⁡rank⁡\(fθ​\(𝐖\)\)≤rank⁡\(\[𝐏1,…,𝐏n1\]\)\.\\max\_\{\\theta\}\\operatorname\{rank\}\(f\_\{\\theta\}\(\\mathbf\{W\}\)\)\\leq\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.
On the other hand, since each𝐐~i∈ℝrc×n0\\widetilde\{\\mathbf\{Q\}\}\_\{i\}\\in\\mathbb\{R\}^\{r\_\{c\}\\times n\_\{0\}\}is unconstrained, we can choose𝐐~i\\widetilde\{\\mathbf\{Q\}\}\_\{i\}so that the columns of𝐏i​𝐐~i\\mathbf\{P\}\_\{i\}\\widetilde\{\\mathbf\{Q\}\}\_\{i\}span any subspace contained incol​\(𝐏i\)\\mathrm\{col\}\(\\mathbf\{P\}\_\{i\}\)\. Aggregating across all slices, we can realize any column set in

col​\(\[𝐏1,…,𝐏n1\]\)\.\\mathrm\{col\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.Therefore,

maxθ⁡rank⁡\(fθ​\(𝐖\)\)=rank⁡\(\[𝐏1,…,𝐏n1\]\)\.\\max\_\{\\theta\}\\operatorname\{rank\}\(f\_\{\\theta\}\(\\mathbf\{W\}\)\)=\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.
Moreover, by setting𝐐~i=𝟎\\widetilde\{\\mathbf\{Q\}\}\_\{i\}=\\mathbf\{0\}for allii, we obtainfθ​\(𝐖\)=𝟎f\_\{\\theta\}\(\\mathbf\{W\}\)=\\mathbf\{0\}, hence

minθ⁡rank⁡\(fθ​\(𝐖\)\)=0\.\\min\_\{\\theta\}\\operatorname\{rank\}\(f\_\{\\theta\}\(\\mathbf\{W\}\)\)=0\.Thus,

ℛ​\(fθ;𝐖\)=rank⁡\(\[𝐏1,…,𝐏n1\]\)\.\\mathcal\{R\}\(f\_\{\\theta\};\\mathbf\{W\}\)=\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.
Now assumem≤nm\\leq nandrank⁡\(𝐖\)=m\\operatorname\{rank\}\(\\mathbf\{W\}\)=m\. Since

𝐖=\[𝐏1​𝐐1,…,𝐏n1​𝐐n1\],\\mathbf\{W\}=\[\\mathbf\{P\}\_\{1\}\\mathbf\{Q\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\\mathbf\{Q\}\_\{n\_\{1\}\}\],all columns of𝐖\\mathbf\{W\}lie incol​\(\[𝐏1,…,𝐏n1\]\)\\mathrm\{col\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\), so

m=rank⁡\(𝐖\)≤rank⁡\(\[𝐏1,…,𝐏n1\]\)\.m=\\operatorname\{rank\}\(\\mathbf\{W\}\)\\leq\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)\.Since the latter is at mostmm, we conclude

rank⁡\(\[𝐏1,…,𝐏n1\]\)=m,\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\)=m,and henceℛ​\(fθ;𝐖\)=m\\mathcal\{R\}\(f\_\{\\theta\};\\mathbf\{W\}\)=m\.

The row\-sliced case follows symmetrically\. For

gϕ​\(𝐖\)=\[𝐔~1​𝐕1⋮𝐔~m1​𝐕m1\],g\_\{\\phi\}\(\\mathbf\{W\}\)=\\begin\{bmatrix\}\\widetilde\{\\mathbf\{U\}\}\_\{1\}\\mathbf\{V\}\_\{1\}\\\\ \\vdots\\\\ \\widetilde\{\\mathbf\{U\}\}\_\{m\_\{1\}\}\\mathbf\{V\}\_\{m\_\{1\}\}\\end\{bmatrix\},each row lies inrow​\(𝐕j\)\\mathrm\{row\}\(\\mathbf\{V\}\_\{j\}\)\. Hence

row\(gϕ\(𝐖\)\)⊆span\{row\(𝐕j\)\}j=1m1,\\mathrm\{row\}\(g\_\{\\phi\}\(\\mathbf\{W\}\)\)\\subseteq\\operatorname\{span\}\\\{\\mathrm\{row\}\(\\mathbf\{V\}\_\{j\}\)\\\}\_\{j=1\}^\{m\_\{1\}\},which implies

maxϕ⁡rank⁡\(gϕ​\(𝐖\)\)=rank⁡\(\[𝐕1⊤⋯𝐕m1⊤\]\)\.\\max\_\{\\phi\}\\operatorname\{rank\}\(g\_\{\\phi\}\(\\mathbf\{W\}\)\)=\\operatorname\{rank\}\\\!\\left\(\\begin\{bmatrix\}\\mathbf\{V\}\_\{1\}^\{\\top\}&\\cdots&\\mathbf\{V\}\_\{m\_\{1\}\}^\{\\top\}\\end\{bmatrix\}\\right\)\.Again the minimum rank is0\. Ifn≤mn\\leq mandrank⁡\(𝐖\)=n\\operatorname\{rank\}\(\\mathbf\{W\}\)=n, then the row space must have dimensionnn, yielding

ℛ​\(gϕ;𝐖\)=n\.\\mathcal\{R\}\(g\_\{\\phi\};\\mathbf\{W\}\)=n\.∎

##### Constraints of block\-wise fixed\-subspace adaptation\.

Although the proposed slice\-wise parameterization attains maximal rank capacity, it imposes strong structural constraints on the adapted weight\. We characterize these constraints below\.

###### Proposition E\.3\(Block\-wise fixed\-subspace constraint\)\.

Consider the column\-sliced parameterization

fθ​\(𝐖\)=\[𝐏1​𝐐~1,…,𝐏n1​𝐐~n1\],f\_\{\\theta\}\(\\mathbf\{W\}\)=\[\\mathbf\{P\}\_\{1\}\\widetilde\{\\mathbf\{Q\}\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\\widetilde\{\\mathbf\{Q\}\}\_\{n\_\{1\}\}\],with fixed\{𝐏i\}\\\{\\mathbf\{P\}\_\{i\}\\\}\. Then for anyθ\\theta:

col​\(fθ​\(𝐖\)\(:,ℐi\)\)⊆col​\(𝐏i\),\\mathrm\{col\}\\\!\\left\(f\_\{\\theta\}\(\\mathbf\{W\}\)\_\{\(:,\\mathcal\{I\}\_\{i\}\)\}\\right\)\\subseteq\\mathrm\{col\}\(\\mathbf\{P\}\_\{i\}\),whereℐi\\mathcal\{I\}\_\{i\}indexes columns belonging to sliceii\. Equivalently,

fθ​\(𝐖\)∈𝒮col:=\{\[𝐗1,…,𝐗n1\]\|col​\(𝐗i\)⊆col​\(𝐏i\)\}\.f\_\{\\theta\}\(\\mathbf\{W\}\)\\in\\mathcal\{S\}\_\{\\text\{col\}\}:=\\left\\\{\[\\mathbf\{X\}\_\{1\},\\dots,\\mathbf\{X\}\_\{n\_\{1\}\}\]\\;\\middle\|\\;\\mathrm\{col\}\(\\mathbf\{X\}\_\{i\}\)\\subseteq\\mathrm\{col\}\(\\mathbf\{P\}\_\{i\}\)\\right\\\}\.

###### Proposition E\.4\(Row\-space constraint \(dual form\)\)\.

For the row\-sliced parameterization

gϕ​\(𝐖\)=\[𝐔~1​𝐕1⋮𝐔~m1​𝐕m1\],g\_\{\\phi\}\(\\mathbf\{W\}\)=\\begin\{bmatrix\}\\widetilde\{\\mathbf\{U\}\}\_\{1\}\\mathbf\{V\}\_\{1\}\\\\ \\vdots\\\\ \\widetilde\{\\mathbf\{U\}\}\_\{m\_\{1\}\}\\mathbf\{V\}\_\{m\_\{1\}\}\\end\{bmatrix\},we have

row​\(gϕ​\(𝐖\)\(𝒥j,:\)\)⊆row​\(𝐕j\),\\mathrm\{row\}\\\!\\left\(g\_\{\\phi\}\(\\mathbf\{W\}\)\_\{\(\\mathcal\{J\}\_\{j\},:\)\}\\right\)\\subseteq\\mathrm\{row\}\(\\mathbf\{V\}\_\{j\}\),and hence

gϕ\(𝐖\)∈𝒮row:=\{\[𝐘1⋮𝐘m1\]\|row\(𝐘j\)⊆row\(𝐕j\)\}\.g\_\{\\phi\}\(\\mathbf\{W\}\)\\in\\mathcal\{S\}\_\{\\text\{row\}\}:=\\left\\\{\\begin\{bmatrix\}\\mathbf\{Y\}\_\{1\}\\\\ \\vdots\\\\ \\mathbf\{Y\}\_\{m\_\{1\}\}\\end\{bmatrix\}\\;\\middle\|\\;\\mathrm\{row\}\(\\mathbf\{Y\}\_\{j\}\)\\subseteq\\mathrm\{row\}\(\\mathbf\{V\}\_\{j\}\)\\right\\\}\.

##### Implications\.

The above propositions show that adaptation is restricted to a*product of subspaces*across slices\. This leads to several nontrivial consequences:

\(1\) No cross\-block subspace transfer\.Columns in sliceiicannot utilize directions incol​\(𝐏j\)\\mathrm\{col\}\(\\mathbf\{P\}\_\{j\}\)forj≠ij\\neq i\. Hence global redistribution of information across slices is prohibited\.

\(2\) Restricted perturbation geometry\.LetΔ​𝐖=fθ​\(𝐖\)−𝐖\\Delta\\mathbf\{W\}=f\_\{\\theta\}\(\\mathbf\{W\}\)\-\\mathbf\{W\}\. Then

Δ​𝐖=\[𝐏1​Δ​𝐐1,…,𝐏n1​Δ​𝐐n1\],\\Delta\\mathbf\{W\}=\[\\mathbf\{P\}\_\{1\}\\Delta\\mathbf\{Q\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\\Delta\\mathbf\{Q\}\_\{n\_\{1\}\}\],so each block update lies in a fixed linear subspace\. In particular,

Δ𝐖∈𝒯:=\{\[𝐏1𝐙1,…,𝐏n1𝐙n1\]\},\\Delta\\mathbf\{W\}\\in\\mathcal\{T\}:=\\left\\\{\[\\mathbf\{P\}\_\{1\}\\mathbf\{Z\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\\mathbf\{Z\}\_\{n\_\{1\}\}\]\\right\\\},which is a strict subset ofℝm×n\\mathbb\{R\}^\{m\\times n\}unless allcol​\(𝐏i\)\\mathrm\{col\}\(\\mathbf\{P\}\_\{i\}\)spanℝm\\mathbb\{R\}^\{m\}individually\.

\(3\) Coupled global rank vs local expressivity\.Although the global rank can reachrank⁡\(\[𝐏1,…,𝐏n1\]\)\\operatorname\{rank\}\(\[\\mathbf\{P\}\_\{1\},\\dots,\\mathbf\{P\}\_\{n\_\{1\}\}\]\), each slice individually satisfies

rank⁡\(𝐏i​𝐐~i\)≤rc\.\\operatorname\{rank\}\(\\mathbf\{P\}\_\{i\}\\widetilde\{\\mathbf\{Q\}\}\_\{i\}\)\\leq r\_\{c\}\.Hence expressivity is locally bottlenecked even when global rank is maximal\.

\(4\) Subspace preservation bias\.The adaptation preserves the pretrained column subspaces\{col​\(𝐏i\)\}\\\{\\mathrm\{col\}\(\\mathbf\{P\}\_\{i\}\)\\\}\. Therefore, learning is biased toward*reweighting and recombining existing features*rather than creating new directions\.

##### Comparison to LoRA and spectral adapters\.

Unlike LoRA, which introduces a global low\-rank updateΔ​𝐖=𝐀𝐁⊤\\Delta\\mathbf\{W\}=\\mathbf\{A\}\\mathbf\{B\}^\{\\top\}spanning all columns, the above parameterization enforces block\-wise independence of subspaces\. Compared to spectral adapters that directly modify singular directions, this approach fixes the singular subspaces and only adapts coefficients within them\.

##### When is this favorable?

This constraint is beneficial when pretrained subspaces are already well\-aligned with downstream tasks, as it: \(i\) preserves stable directions, \(ii\) reduces destructive interference, and \(iii\) improves conditioning by restricting updates to structured subspaces\.

### E\.3Preconditioner derivations

Let𝐖k=𝐔k​𝚺k​𝐕k⊤\\mathbf\{W\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\mathbf\{\\Sigma\}\_\{k\}\\mathbf\{V\}\_\{k\}^\{\\top\}be the SVD of one block\. Letℒ\\mathcal\{L\}be the loss,𝐲=𝐖𝐱\\mathbf\{y\}=\\mathbf\{W\}\\mathbf\{x\},𝐠=∂ℒ/∂𝐲\\mathbf\{g\}=\\partial\\mathcal\{L\}/\\partial\\mathbf\{y\}\. Standard identities give the unconstrained per\-block gradient∂ℒ/∂𝐖k=𝐠𝐱k⊤\\partial\\mathcal\{L\}/\\partial\\mathbf\{W\}\_\{k\}=\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\. We specialize to the fourFuRAvariants\.

##### Corner 1: output\-side training,𝐒\\mathbf\{S\}merged to frozen𝐑\\mathbf\{R\}\.

The parameterization is𝐖k=𝐋k​𝐑k\\mathbf\{W\}\_\{k\}=\\mathbf\{L\}\_\{k\}\\mathbf\{R\}\_\{k\}with𝐋k\\mathbf\{L\}\_\{k\}trainable and𝐑k=diag​\(𝐒k\)​𝐕k⊤\\mathbf\{R\}\_\{k\}=\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{V\}\_\{k\}^\{\\top\}frozen\. The gradient on𝐋k\\mathbf\{L\}\_\{k\}is

∂ℒ/∂𝐋k=\(𝐠𝐱k⊤\)​𝐑k⊤=𝐠𝐱k⊤​𝐕k​diag​\(𝐒k\)\.\\partial\\mathcal\{L\}/\\partial\\mathbf\{L\}\_\{k\}\\;=\\;\(\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\)\\mathbf\{R\}\_\{k\}^\{\\top\}\\;=\\;\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\.A GD step𝐋k←𝐋k−η​𝐠𝐱k⊤​𝐕k​diag​\(𝐒k\)\\mathbf\{L\}\_\{k\}\\leftarrow\\mathbf\{L\}\_\{k\}\-\\eta\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)yields

Δ​𝐖k=Δ​𝐋k⋅𝐑k=−η​𝐠𝐱k⊤​𝐕k​diag​\(𝐒k\)2​𝐕k⊤,\\Delta\\mathbf\{W\}\_\{k\}\\;=\\;\\Delta\\mathbf\{L\}\_\{k\}\\cdot\\mathbf\{R\}\_\{k\}\\;=\\;\-\\eta\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{V\}\_\{k\}^\{\\top\},the “diag​\(𝐒\)2\\mathrm\{diag\}\(\\mathbf\{S\}\)^\{2\}\-weighted” row of Table[2](https://arxiv.org/html/2605.22869#S5.T2):diag​\(𝐒k\)\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)enters once via𝐑k⊤\\mathbf\{R\}\_\{k\}^\{\\top\}in the gradient and once via𝐑k\\mathbf\{R\}\_\{k\}in the reconstruction\.

##### Corner 2: output\-side training,𝐒\\mathbf\{S\}merged to trainable𝐋\\mathbf\{L\}\.

The parameterization is𝐖k=𝐋k​𝐑k\\mathbf\{W\}\_\{k\}=\\mathbf\{L\}\_\{k\}\\mathbf\{R\}\_\{k\}with𝐋k=𝐔k​diag​\(𝐒k\)\\mathbf\{L\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)trainable and𝐑k=𝐕k⊤\\mathbf\{R\}\_\{k\}=\\mathbf\{V\}\_\{k\}^\{\\top\}frozen\. The gradient on𝐋k\\mathbf\{L\}\_\{k\}is

∂ℒ/∂𝐋k=\(𝐠𝐱k⊤\)​𝐑k⊤=𝐠𝐱k⊤​𝐕k\.\\partial\\mathcal\{L\}/\\partial\\mathbf\{L\}\_\{k\}\\;=\\;\(\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\)\\mathbf\{R\}\_\{k\}^\{\\top\}\\;=\\;\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}\.A GD step yieldsΔ​𝐋k=−η​𝐠𝐱k⊤​𝐕k\\Delta\\mathbf\{L\}\_\{k\}=\-\\eta\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}, hence

Δ​𝐖k=Δ​𝐋k⋅𝐑k=−η​𝐠𝐱k⊤​𝐕k​𝐕k⊤,\\Delta\\mathbf\{W\}\_\{k\}\\;=\\;\\Delta\\mathbf\{L\}\_\{k\}\\cdot\\mathbf\{R\}\_\{k\}\\;=\\;\-\\eta\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}\\mathbf\{V\}\_\{k\}^\{\\top\},the “fair” projector𝐕k​𝐕k⊤\\mathbf\{V\}\_\{k\}\\mathbf\{V\}\_\{k\}^\{\\top\}row of Table[2](https://arxiv.org/html/2605.22869#S5.T2): when𝐒\\mathbf\{S\}is absorbed into the trainable side, its scaling is freely re\-tuned by the optimizer and drops out of the effective preconditioner, leaving an unweighted projection ontorow⁡\(𝐕k⊤\)\\operatorname\{row\}\(\\mathbf\{V\}\_\{k\}^\{\\top\}\)\.

##### Corner 3 \(default\): input\-side training,𝐒\\mathbf\{S\}kept separate trainable\.

Parameterize𝐖k=𝐋k​diag​\(𝐒k\)​𝐑k\\mathbf\{W\}\_\{k\}=\\mathbf\{L\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{R\}\_\{k\}with𝐋k=𝐔k\\mathbf\{L\}\_\{k\}=\\mathbf\{U\}\_\{k\}frozen and𝐑k=𝐕k⊤\\mathbf\{R\}\_\{k\}=\\mathbf\{V\}\_\{k\}^\{\\top\},𝐒k\\mathbf\{S\}\_\{k\}trainable\. The gradient on𝐑k\\mathbf\{R\}\_\{k\}is∂ℒ/∂𝐑k=diag​\(𝐒k\)​𝐔k⊤​𝐠𝐱k⊤\\partial\\mathcal\{L\}/\\partial\\mathbf\{R\}\_\{k\}=\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}; the gradient on𝐒k\\mathbf\{S\}\_\{k\}is the diagonal of𝐔k⊤​𝐠𝐱k⊤​𝐕k\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\,\\mathbf\{V\}\_\{k\}\. A GD step on𝐑k\\mathbf\{R\}\_\{k\}givesΔ​𝐑k=−η​diag​\(𝐒k\)​𝐔k⊤​𝐠𝐱k⊤\\Delta\\mathbf\{R\}\_\{k\}=\-\\eta\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}, so the effective weight update from𝐑\\mathbf\{R\}is

Δ​𝐖k\|𝐑=𝐔k​diag​\(𝐒k\)​Δ​𝐑k=−η​𝐔k​diag​\(𝐒k\)2​𝐔k⊤​𝐠𝐱k⊤,\\Delta\\mathbf\{W\}\_\{k\}\\big\|\_\{\\mathbf\{R\}\}\\;=\\;\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\Delta\\mathbf\{R\}\_\{k\}\\;=\\;\-\\eta\\,\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\},wherediag​\(𝐒k\)\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)enters twice: once from𝐋k​diag​\(𝐒k\)\\mathbf\{L\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)multiplyingΔ​𝐑k\\Delta\\mathbf\{R\}\_\{k\}, and once insideΔ​𝐑k\\Delta\\mathbf\{R\}\_\{k\}itself\. Combined with the𝐒\\mathbf\{S\}update:

Δ​𝐖k=−η​𝐔k​diag​\(𝐒k\)2​𝐔k⊤​𝐠𝐱k⊤\+𝐔k​diag​\(Δ​𝐒k\)​𝐕k⊤,\\Delta\\mathbf\{W\}\_\{k\}\\;=\\;\-\\eta\\,\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\\;\+\\;\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\Delta\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{V\}\_\{k\}^\{\\top\},the default preconditioner of Equation \([5](https://arxiv.org/html/2605.22869#S5.E5)\)\. The first term combines column\-space projection \(𝐔k​𝐔k⊤\\mathbf\{U\}\_\{k\}\\mathbf\{U\}\_\{k\}^\{\\top\}, non\-trivial since𝐔k\\mathbf\{U\}\_\{k\}is rectangular\) with singular\-value preconditioning \(diag​\(𝐒k\)2\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}scaling\)\.

##### Corner 4: input\-side training,𝐒\\mathbf\{S\}merged to frozen𝐋\\mathbf\{L\}\.

The parameterization is𝐖k=𝐋k​𝐑k\\mathbf\{W\}\_\{k\}=\\mathbf\{L\}\_\{k\}\\mathbf\{R\}\_\{k\}with𝐋k=𝐔k​diag​\(𝐒k\)\\mathbf\{L\}\_\{k\}=\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)frozen and𝐑k=𝐕k⊤\\mathbf\{R\}\_\{k\}=\\mathbf\{V\}\_\{k\}^\{\\top\}trainable\. The gradient on𝐑k\\mathbf\{R\}\_\{k\}is

∂ℒ/∂𝐑k=𝐋k⊤​\(𝐠𝐱k⊤\)=diag​\(𝐒k\)​𝐔k⊤​𝐠𝐱k⊤\.\\partial\\mathcal\{L\}/\\partial\\mathbf\{R\}\_\{k\}\\;=\\;\\mathbf\{L\}\_\{k\}^\{\\top\}\(\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\)\\;=\\;\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}\.A GD step yieldsΔ​𝐑k=−η​diag​\(𝐒k\)​𝐔k⊤​𝐠𝐱k⊤\\Delta\\mathbf\{R\}\_\{k\}=\-\\eta\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\}, hence

Δ​𝐖k=𝐋k⋅Δ​𝐑k=−η​𝐔k​diag​\(𝐒k\)2​𝐔k⊤​𝐠𝐱k⊤,\\Delta\\mathbf\{W\}\_\{k\}\\;=\\;\\mathbf\{L\}\_\{k\}\\cdot\\Delta\\mathbf\{R\}\_\{k\}\\;=\\;\-\\eta\\,\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{U\}\_\{k\}^\{\\top\}\\,\\mathbf\{g\}\\mathbf\{x\}\_\{k\}^\{\\top\},the principal\-biased𝐔k​diag​\(𝐒k\)2​𝐔k⊤\\mathbf\{U\}\_\{k\}\\,\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)^\{2\}\\,\\mathbf\{U\}\_\{k\}^\{\\top\}row of Table[2](https://arxiv.org/html/2605.22869#S5.T2)\(PiSSA\-like\):diag​\(𝐒k\)\\mathrm\{diag\}\(\\mathbf\{S\}\_\{k\}\)enters once via𝐋k⊤\\mathbf\{L\}\_\{k\}^\{\\top\}in the gradient and once via𝐋k\\mathbf\{L\}\_\{k\}in the reconstruction, mirroring Corner 1 with the input/output sides swapped\. The remaining row of Table[2](https://arxiv.org/html/2605.22869#S5.T2)\(input\-side training with𝐒\\mathbf\{S\}merged into the trainable𝐑\\mathbf\{R\}\) follows by the same calculation as Corner 2 with the input/output sides swapped, yielding the unweighted projector𝐔k​𝐔k⊤\\mathbf\{U\}\_\{k\}\\mathbf\{U\}\_\{k\}^\{\\top\}\.

## Appendix FFull per\-layer sweeps for the motivating figures

This section provides the full per\-layer / per\-module versions of the panels shown for a single representative weight matrix in Figures[3](https://arxiv.org/html/2605.22869#S3.F3)and[4](https://arxiv.org/html/2605.22869#S4.F4)of the body\. The body uses layer 15q\_projas a representative slice; the figures here verify that the trends hold across all transformer layers and across the matched set of linear modules\.

![Refer to caption](https://arxiv.org/html/2605.22869v1/x7.png)Figure 6:Full per\-layer / per\-module sweep for Figure[3](https://arxiv.org/html/2605.22869#S3.F3)\(a\): energy\-ratioρ​\(𝐆𝐖;𝐔\)\\rho\(\\mathbf\{G\}\_\{\\mathbf\{W\}\};\\mathbf\{U\}\)andρ​\(𝐖′;𝐔\)\\rho\(\\mathbf\{W\}^\{\\prime\};\\mathbf\{U\}\)tracked through Full FT training on LLaMA\-3\-8B / Math\-10K\. The body figure shows layer 15q\_proj; the same pattern \(gradient near the random baseline, weight near1\.01\.0\) holds across all layers and modules\.![Refer to caption](https://arxiv.org/html/2605.22869v1/x8.png)Figure 7:Full per\-layer / per\-module sweep for Figure[3](https://arxiv.org/html/2605.22869#S3.F3)\(b\): comparison of singular values and singular vectors of𝐖\\mathbf\{W\}versus𝐖′\\mathbf\{W\}^\{\\prime\}after Full FT on LLaMA\-3\-8B / Math\-10K\. The body figure shows layer 15q\_proj; across all layers and modules only a few singular values change significantly while most remain close to their pretrained values\.![Refer to caption](https://arxiv.org/html/2605.22869v1/x9.png)Figure 8:Full per\-layer / per\-module sweep for Figure[4](https://arxiv.org/html/2605.22869#S4.F4)\(a\): effective rank ofΔ​𝐖\\Delta\\mathbf\{W\}on the Qwen3\-1\.7B GRPO math\-RL task\. The body figure shows a representative module; here we report all modules in every transformer layer, confirming thatFuRA’s effective rank closely tracks Full FT layer\-by\-layer while remaining well above LoRA\.