ChainDoRA: 基于Tensor-Train分解的权重分解低秩适应用于参数高效的大语言模型微调

# Tensor-Train Factorized Weight-Decomposed Low-Rank Adaptation for Parameter-Efficient LLM Fine-Tuning
Source: [https://arxiv.org/html/2609.25058](https://arxiv.org/html/2609.25058)
Ashfak YeafiAffiliation:Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology, Khulna, BangladeshEmail:[yeafiashfak@gmail\.com](mailto:[email protected])Mehedi HasanAffiliation:Department of Computer Science and Engineering, Brac University, Dhaka, BangladeshEmail:[mehedi\.hasan1@g\.bracu\.ac\.bd](mailto:)Md Khairul IslamAffiliation:Department of Mathematics and Computer Science, Hobart and William Smith Colleges, Geneva, NY, USAEmail:[khairul\.robotics@gmail\.com](mailto:)

###### Abstract

Parameter\-efficient fine\-tuning \(PEFT\) adapts large language models \(LLMs\) with a small task\-specific parameter budget, but the dense low\-rank factors used by LoRA and retained in DoRA still scale linearly with the dimensions of each adapted layer\. We propose*ChainDoRA*, which preserves DoRA’s magnitude–direction decomposition while constructing its directional low\-rank factors from a connected Tensor\-Train \(TT\) chain whose boundary rank is the adapter rank and whose independent internal TT rank controls capacity and parameter cost\. Under a controlled 15,119\-example response\-only adaptation setting with LLaMA\-7B, ChainDoRA with TT rank 16 achieves 72\.30% average accuracy across seven commonsense reasoning benchmarks, compared with 69\.88% for LoRA and 69\.39% for DoRA\. At the same adapter rank, ChainDoRA uses 5\.35M trainable parameters versus 56\.10M for LoRA and 56\.98M for DoRA, a 90\.62% reduction relative to DoRA\. Ablations over TT rank and adapter placement reveal controllable parameter–accuracy trade\-offs, while cross\-architecture analysis shows that the structural parameter savings persist across representative 3B–72B model configurations\. Code is available at[https://github\.com/AshfakYeafi/Chain\-DoRA](https://github.com/AshfakYeafi/Chain-DoRA)\.

Keywords:parameter\-efficient fine\-tuning; low\-rank adaptation; DoRA; Tensor\-Train decomposition; large language models; model adaptation\.

## 1Introduction

The rapid growth of pretrained language models has made task\-specific adaptation increasingly expensive in terms of memory, optimization, and model storage\. Conventional full\-parameter fine\-tuning updates the entire pretrained model and consequently requires maintaining a complete task\-specific parameter set\. Parameter\-efficient fine\-tuning \(PEFT\) addresses this limitation by keeping most or all pretrained parameters frozen while optimizing a comparatively small set of task\-specific parameters\. Representative approaches include lightweight adapter modules\[[1](https://arxiv.org/html/2609.25058#bib.bib4)\], continuous prefix optimization\[[2](https://arxiv.org/html/2609.25058#bib.bib5)\], and low\-rank reparameterization\[[3](https://arxiv.org/html/2609.25058#bib.bib1)\]\. Among these methods, Low\-Rank Adaptation \(LoRA\) has become particularly attractive because its learned update can be merged into the pretrained weight at inference time without introducing an additional inference branch\.

LoRA parameterizes the update to a frozen weight matrix through two trainable low\-rank factors\[[3](https://arxiv.org/html/2609.25058#bib.bib1)\]\. This substantially reduces the number of optimized parameters relative to full fine\-tuning, but its adapter size remains proportional to the input and output dimensions of every adapted linear transformation\. Moreover, LoRA modifies the pretrained weight additively without explicitly separating changes in weight scale from changes in orientation\. Weight\-Decomposed Low\-Rank Adaptation \(DoRA\) addresses the latter issue by decomposing the pretrained weight into magnitude and directional components and applying LoRA to the directional component while learning the magnitude separately\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]\. DoRA has demonstrated improved adaptation capability over LoRA across language and vision\-language tasks; however, its directional update continues to rely on the same pair of densely parameterized low\-rank matrices used by LoRA\.

Tensor decompositions offer an alternative mechanism for representing high\-dimensional parameters through structured low\-dimensional factors\. The Tensor\-Train \(TT\) format represents a high\-order tensor as a sequence of low\-order cores connected through auxiliary TT ranks\[[5](https://arxiv.org/html/2609.25058#bib.bib8)\]\. TT representations have previously been used to compress neural\-network parameters\[[6](https://arxiv.org/html/2609.25058#bib.bib9)\], and recent work has extended tensorized parameterizations to PEFT\. LoRETTA employs TT representations for ultra\-low\-parameter LLM adaptation\[[7](https://arxiv.org/html/2609.25058#bib.bib21)\], while TT\-LoRA explores TT\-based parameterization for efficient large\-model adaptation\[[8](https://arxiv.org/html/2609.25058#bib.bib10)\]\. More recent approaches have also investigated weight\-decomposed tensor adaptation and unified TT representations for constructing correlated low\-rank factors\. These studies indicate that tensor\-network structure can substantially reduce the number of directly optimized parameters, but they also highlight the importance of how the tensor representation is integrated into the adaptation mechanism\.

Motivated by these observations, we propose*ChainDoRA*, a Tensor\-Train\-factorized extension of magnitude–direction adaptation\. Rather than directly optimizing the two dense low\-rank matrices in the directional branch of DoRA, ChainDoRA represents them through contractions of a single connected sequence of trainable TT cores\. The input\-side cores contract to formATT∈ℝr×dinA\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}, while the output\-side cores formBTT∈ℝdout×rB\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\. The two contractions are connected through a TT bond whose dimension is fixed to the adapter rankrr\. A separate internal TT rankρ\\rhocontrols the capacity of the tensorized representation\. Consequently, the nominal low\-rank adaptation dimension and the internal tensor complexity can be controlled independently\.

![Refer to caption](https://arxiv.org/html/2609.25058v1/ChainDoRA_overall.png)Figure 1:Overview of the proposed ChainDoRA framework\. A connected Tensor\-Train chain constructs the directional factorsATTA\_\{\\mathrm\{TT\}\}andBTTB\_\{\\mathrm\{TT\}\}, which are coupled through the adapter\-rank boundaryrr\. Their product defines the low\-rank directional update, while the row\-wise magnitude vectormmis optimized independently\. The pretrained weightW0W\_\{0\}remains frozen\.As illustrated in Fig\.[1](https://arxiv.org/html/2609.25058#S1.F1), the resulting TT\-derived update is incorporated into the magnitude–direction decomposition rather than used as an independent additive tensor adapter\. This design preserves the row\-wise magnitude control of DoRA while replacing its dense directional factors with a structured tensor representation\. For the LLaMA\-7B configuration studied in this work, adapting the query, key, value, up\-projection, and down\-projection layers with rankr=32r=32requires approximately 56\.98M trainable parameters with DoRA\. ChainDoRA with internal TT rankρ=16\\rho=16reduces this number to 5\.35M, corresponding to a 90\.62% reduction in trainable parameters while retaining the same nominal adapter rank\.

We evaluate ChainDoRA under a controlled 15,119\-example response\-only adaptation setting with LLaMA\-7B as the frozen backbone\. The evaluation covers seven commonsense reasoning benchmarks: BoolQ, PIQA, SocialIQA, WinoGrande, ARC\-Easy, ARC\-Challenge, and OpenBookQA\. Under matched optimization, target\-module, and evaluation settings, ChainDoRA achieves a seven\-task average accuracy of 72\.30%, compared with 69\.88% for LoRA and 69\.39% for DoRA\. We further investigate the effects of TT rank and adapter placement, demonstrating a tunable parameter–accuracy trade\-off\. In addition, we analyze ChainDoRA’s parameter scaling across multiple LLM architectures and model sizes to characterize how the proposed factorization scales beyond LLaMA\-7B\.

The main contributions of this work are summarized as follows:

- •We introduce*ChainDoRA*, a magnitude–direction PEFT framework that replaces the densely parameterized directional factors of DoRA with factors obtained from a connected Tensor\-Train representation\.
- •We formulate the adapter rankrras the boundary rank connecting the input\- and output\-side TT contractions while introducing an independent internal TT rankρ\\rho\. This separates the nominal low\-rank adaptation dimension from the parameter capacity of the tensor representation\.
- •We develop a parameter\-efficient implementation in which the TT\-derived directional update is integrated with row\-wise magnitude normalization, together with a balanced nonzero TT initialization designed for stable optimization of the multiplicative tensor chain\.
- •We conduct a controlled comparison with LoRA and DoRA on seven commonsense reasoning benchmarks using LLaMA\-7B\. ChainDoRA achieves a 72\.30% average accuracy with 5\.35M trainable parameters, improving over both baselines while reducing the trainable parameter count by approximately an order of magnitude\.
- •We provide systematic ablations of TT rank and adaptation location, together with a cross\-architecture parameter\-scaling analysis, to characterize the efficiency–capacity trade\-offs of the proposed parameterization\.

The remainder of this paper is organized as follows\. Section[2](https://arxiv.org/html/2609.25058#S2)reviews low\-rank, weight\-decomposed, and tensor\-based PEFT methods\. Section[3](https://arxiv.org/html/2609.25058#S3)presents the ChainDoRA formulation, TT construction, initialization strategy, and parameter complexity\. Section[4](https://arxiv.org/html/2609.25058#S4)describes the experimental protocol\. The empirical results and ablation studies are then presented and discussed, followed by the conclusion\.

## 2Related Work

### 2\.1Parameter\-Efficient and Low\-Rank Adaptation

Parameter\-efficient fine\-tuning \(PEFT\) adapts pretrained models while updating only a small subset of parameters, thereby reducing the memory and storage requirements associated with conventional full\-parameter fine\-tuning\. Early PEFT approaches introduced lightweight trainable modules between frozen Transformer layers\[[1](https://arxiv.org/html/2609.25058#bib.bib4)\], while prefix tuning optimized continuous task\-specific vectors that are prepended to the hidden states of Transformer layers\[[2](https://arxiv.org/html/2609.25058#bib.bib5)\]\. These approaches established that effective downstream adaptation does not necessarily require updating the entire pretrained parameter set\.

Low\-Rank Adaptation \(LoRA\)\[[3](https://arxiv.org/html/2609.25058#bib.bib1)\]instead reparameterizes the update of a frozen weight matrixW0W\_\{0\}as the product of two trainable low\-rank matrices\. For a rank\-rradapter, the update takes the formB​ABA, whereA∈ℝr×dinA\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}andB∈ℝdout×rB\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\. This formulation substantially reduces the number of trainable parameters relative to full fine\-tuning and allows the learned update to be merged into the pretrained weight for inference\. LoRA has subsequently served as the basis for numerous PEFT variants that modify rank allocation, parameter sharing, initialization, or backbone precision\.

AdaLoRA\[[9](https://arxiv.org/html/2609.25058#bib.bib6)\], for example, parameterizes incremental updates using a singular\-value\-decomposition\-inspired formulation and dynamically reallocates the available rank budget according to parameter importance\. QLoRA\[[10](https://arxiv.org/html/2609.25058#bib.bib7)\]addresses a complementary source of memory cost by backpropagating through a frozen 4\-bit quantized backbone while optimizing LoRA adapters\. VeRA\[[11](https://arxiv.org/html/2609.25058#bib.bib19)\]further reduces task\-specific storage by sharing a pair of frozen random low\-rank matrices across layers and learning comparatively small scaling vectors\. These methods demonstrate that parameter efficiency can be improved through rank allocation, quantization, and parameter sharing; nevertheless, conventional LoRA and many of its derivatives retain an explicitly matrix\-factorized representation of the task\-specific update\.

### 2\.2Magnitude–Direction Adaptation

Weight\-Decomposed Low\-Rank Adaptation \(DoRA\)\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]analyzes fine\-tuning through the decomposition of a pretrained weight into magnitude and directional components\. Instead of applying a low\-rank update directly to the complete weight matrix, DoRA learns the magnitude independently and uses LoRA to adapt the directional component\. The resulting weight can be written conceptually as a learned magnitude multiplied by a normalized, low\-rank\-updated direction\. This decomposition was motivated by observed differences between the magnitude and directional changes produced by full fine\-tuning and conventional LoRA\. DoRA was shown to improve over LoRA across language, vision\-language, and multimodal adaptation tasks while preserving the ability to merge the adapted weights for inference\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]\.

Despite this decomposition, the directional branch of DoRA is still parameterized through the two dense rank\-rrfactors inherited from LoRA\. For an adapted matrix with dimensionsdout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}, these factors requirer⁡\(din\+dout\)r\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)trainable parameters before accounting for the magnitude vector\. Consequently, the directional adapter size continues to grow linearly with the dimensions of the target transformation\. ChainDoRA retains the magnitude–direction decomposition of DoRA but replaces these directly optimized directional matrices with factors derived from a structured Tensor\-Train representation\.

### 2\.3Tensorized Parameter\-Efficient Fine\-Tuning

Tensor decomposition provides a structured alternative to conventional matrix factorization for representing high\-dimensional parameters\. The Tensor\-Train \(TT\) format introduced by Oseledets\[[5](https://arxiv.org/html/2609.25058#bib.bib8)\]represents a high\-order tensor through a sequence of low\-order cores connected by auxiliary ranks\. The number of parameters is determined by the physical mode sizes and TT ranks rather than by explicitly storing the complete tensor\. TT representations were subsequently applied to neural networks by tensorizing dense weight matrices and replacing them with compact TT structures\[[6](https://arxiv.org/html/2609.25058#bib.bib9)\]\.

Recent work has extended tensor\-network representations to PEFT\. LoRETTA\[[7](https://arxiv.org/html/2609.25058#bib.bib21)\]proposes two TT\-based adaptation strategies\. LoRETTAadptensorizes lightweight adapter modules, whereas LoRETTArepperforms weight reparameterization using compact TT factors\. The latter demonstrates that Tensor\-Train structure can provide substantial parameter reductions during LLM fine\-tuning\. TT\-LoRA\[[8](https://arxiv.org/html/2609.25058#bib.bib10)\]further investigates Tensor\-Train parameterization for LLM adaptation and studies the trade\-off among tensorization structure, TT rank, parameter count, and downstream performance\. In contrast to standard LoRA, these approaches use higher\-order tensor structure to reduce the number of explicitly optimized adaptation parameters\.

Other tensorized methods adopt different decompositions or initialization strategies\. DoTA\[[12](https://arxiv.org/html/2609.25058#bib.bib20)\]introduces Weight\-Decomposed Tensor Adaptation based on a Matrix Product Operator \(MPO\) representation and leverages a decomposition of pretrained weights to obtain an informative initialization for tensor adaptation\. Its quantized extension, QDoTA, combines this formulation with low\-bit backbone quantization\. Although DoTA and ChainDoRA both involve weight\-decomposed adaptation, their parameterizations are fundamentally different: DoTA adapts weights through an MPO\-based tensor representation initialized from pretrained weights, whereas ChainDoRA retains the DoRA magnitude–direction formulation and tensorizes the low\-rank directional factors\.

TensorGuide\[[13](https://arxiv.org/html/2609.25058#bib.bib11)\]is particularly related to the present work because it considers the interaction between the two low\-rank matrices rather than treating them as independent objects\. TensorGuide employs a unified TT network driven by controlled Gaussian inputs to jointly generate correlated low\-rank adaptation matrices\. This contrasts with separately tensorizing the two LoRA factors and demonstrates that coupling their parameterization can improve the efficiency and expressivity of tensorized low\-rank adaptation\.

ChainDoRA shares the general motivation of exploiting dependencies between low\-rank factors but differs in both construction and placement of the tensor representation\. Specifically, ChainDoRA forms a single connected sequence of trainable TT cores whose input\-side and output\-side contractions directly produceATT∈ℝr×dinA\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}andBTT∈ℝdout×rB\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\. The TT bond separating the two contractions is explicitly assigned the adapter rankrr, while the remaining bonds are governed by an independent TT rankρ\\rho\. The resulting productBTT​ATTB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}is then used specifically to modify the directional component of a DoRA\-style magnitude–direction decomposition\. Thus, the adapter rank controlling the effective low\-rank update and the TT rank controlling the internal tensor capacity are explicitly separated in the proposed formulation\.

Table 1:Conceptual comparison of representative low\-rank and tensorized parameter\-efficient adaptation methods\. “JointA/BA/Bstructure” indicates whether the two low\-rank factors are generated or parameterized through a shared structured representation\.MethodLow\-Rank UpdateTensor RepresentationMagnitude–DirectionJointA/BA/BStructurePrimary Structural MechanismLoRA\[[3](https://arxiv.org/html/2609.25058#bib.bib1)\]✓–––Dense trainable low\-rank factorsAdaLoRA\[[9](https://arxiv.org/html/2609.25058#bib.bib6)\]✓–––Adaptive rank\-budget allocationDoRA\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]✓–✓–Magnitude–direction decompositionLoRETTA\[[7](https://arxiv.org/html/2609.25058#bib.bib21)\]✓TT––Tensorized adapters / reparameterizationTT\-LoRA\[[8](https://arxiv.org/html/2609.25058#bib.bib10)\]✓TT––TT\-parameterized adaptationDoTA\[[12](https://arxiv.org/html/2609.25058#bib.bib20)\]–MPOWeight\-decomposed–Pretrained\-weight MPO initializationTensorGuide\[[13](https://arxiv.org/html/2609.25058#bib.bib11)\]✓TT–✓Unified TT\-generated correlated factorsChainDoRA✓TT✓✓Connected TT directional factors with boundary rankrr

Table[1](https://arxiv.org/html/2609.25058#S2.T1)summarizes these distinctions\. Rather than treating tensor decomposition as an independent adapter or as a replacement for the complete adapted weight, ChainDoRA uses a connected TT representation specifically within the directional branch of magnitude–direction adaptation\. This construction motivates the formulation developed in Section[3](https://arxiv.org/html/2609.25058#S3)\.

## 3Methodology

The proposed ChainDoRA framework combines the magnitude–direction reparameterization of DoRA\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]with a connected Tensor\-Train \(TT\) representation\[[5](https://arxiv.org/html/2609.25058#bib.bib8)\]of the directional low\-rank factors\. Instead of directly optimizing the two dense low\-rank matrices used by LoRA and DoRA, ChainDoRA constructs these factors from contractions of a single TT chain\. The adapter rankrris assigned to the TT bond separating the input\-side and output\-side contractions, while a distinct internal TT rankρ\\rhocontrols the capacity and parameter cost of the tensorized representation\. The pretrained weight remains frozen, and only the TT cores and the row\-wise magnitude parameters are optimized\.

Fig\.[1](https://arxiv.org/html/2609.25058#S1.F1)provides the overall architecture, while Fig\.[2](https://arxiv.org/html/2609.25058#S3.F2)illustrates the internal TT construction used to generate the two directional factors\.

### 3\.1Preliminaries: LoRA and Weight\-Decomposed Adaptation

Consider a pretrained linear transformation with frozen weight

W0∈ℝdout×din,W\_\{0\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\},\(1\)
wheredind\_\{\\mathrm\{in\}\}anddoutd\_\{\\mathrm\{out\}\}denote the input and output dimensions, respectively\. LoRA\[[3](https://arxiv.org/html/2609.25058#bib.bib1)\]represents a task\-specific weight update as

Δ​WLoRA=s​B​A,s=αr,\\Delta W\_\{\\mathrm\{LoRA\}\}=sBA,\\qquad s=\\frac\{\\alpha\}\{r\},\(2\)
where

A∈ℝr×din,B∈ℝdout×r\.A\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\},\\qquad B\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\.\(3\)
Here,rris the low\-rank adaptation dimension andα\\alphais the LoRA scaling coefficient\. Since

rank⁡\(B​A\)≤r,\\operatorname\{rank\}\(BA\)\\leq r,\(4\)
the update matrix has rank at mostrr, reducing the number of trainable degrees of freedom relative to a dense update while the pretrained matrixW0W\_\{0\}remains unchanged\.

DoRA\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]further decomposes adaptation into magnitude and direction\. For thedout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}weight\-storage convention used in our implementation, the decomposition is performed row\-wise\. The initial magnitude is therefore defined as

mi\(0\)=‖W0,i:‖2,i=1,…,dout,m\_\{i\}^\{\(0\)\}=\\left\\\|W\_\{0,i:\}\\right\\\|\_\{2\},\\qquad i=1,\\ldots,d\_\{\\mathrm\{out\}\},\(5\)
or equivalently,

m\(0\)=‖W0‖row\.m^\{\(0\)\}=\\left\\\|W\_\{0\}\\right\\\|\_\{\\mathrm\{row\}\}\.\(6\)
Given an unnormalized directional matrixVV, the corresponding magnitude–direction reconstruction is

W=m⊙V‖V‖row,W=m\\odot\\frac\{V\}\{\\left\\\|V\\right\\\|\_\{\\mathrm\{row\}\}\},\(7\)
where⊙\\odotdenotes row\-wise broadcasting\. In conventional DoRA, the directional matrix is updated using the LoRA factors,

VDoRA=W0\+s​B​A\.V\_\{\\mathrm\{DoRA\}\}=W\_\{0\}\+sBA\.\(8\)
ChainDoRA preserves the decomposition in \([7](https://arxiv.org/html/2609.25058#S3.E7)\) but replaces the directly parameterized matricesAAandBBwith factors generated from a connected Tensor\-Train representation\.

### 3\.2Connected Tensor\-Train Directional Factorization

Let the input and output dimensions be factorized into smaller physical modes,

din=∏j=1pnjin,dout=∏j=1qnjout,d\_\{\\mathrm\{in\}\}=\\prod\_\{j=1\}^\{p\}n\_\{j\}^\{\\mathrm\{in\}\},\\qquad d\_\{\\mathrm\{out\}\}=\\prod\_\{j=1\}^\{q\}n\_\{j\}^\{\\mathrm\{out\}\},\(9\)
whereppandqqdenote the numbers of input\-side and output\-side modes, respectively\. These modes are concatenated into a single sequence

\(n1,…,nN\)=\(n1in,…,npin,n1out,…,nqout\),N=p\+q\.\(n\_\{1\},\\ldots,n\_\{N\}\)=\(n\_\{1\}^\{\\mathrm\{in\}\},\\ldots,n\_\{p\}^\{\\mathrm\{in\}\},n\_\{1\}^\{\\mathrm\{out\}\},\\ldots,n\_\{q\}^\{\\mathrm\{out\}\}\),\\qquad N=p\+q\.\(10\)
Following the TT representation\[[5](https://arxiv.org/html/2609.25058#bib.bib8)\], ChainDoRA introducesNNtrainable third\-order cores

𝒢j∈ℝRj−1×nj×Rj,j=1,…,N,\\mathcal\{G\}\_\{j\}\\in\\mathbb\{R\}^\{R\_\{j\-1\}\\times n\_\{j\}\\times R\_\{j\}\},\\qquad j=1,\\ldots,N,\(11\)
with boundary ranks

R0=RN=1\.R\_\{0\}=R\_\{N\}=1\.\(12\)
A central design choice of ChainDoRA is to assign the TT bond between the input\-side and output\-side contractions directly to the adapter rank,

The remaining non\-boundary TT ranks are controlled by a separate internal rankρ\\rho,

Rj=\{1,j=0​or​j=N,r,j=p,ρ,otherwise\.R\_\{j\}=\\begin\{cases\}1,&j=0\\ \\text\{or\}\\ j=N,\\\\ r,&j=p,\\\\ \\rho,&\\text\{otherwise\}\.\\end\{cases\}\(14\)
Consequently, the complete rank sequence can be written compactly as

𝐑=\(1,ρ,…,ρ⏟p−1,r,ρ,…,ρ⏟q−1,1\)\.\\mathbf\{R\}=\\left\(1,\\underbrace\{\\rho,\\ldots,\\rho\}\_\{p\-1\},r,\\underbrace\{\\rho,\\ldots,\\rho\}\_\{q\-1\},1\\right\)\.\(15\)
The role ofrris therefore distinct from that ofρ\\rho:rrcontrols the maximum rank of the resulting directional update, whereasρ\\rhocontrols the capacity of the TT representation used to construct its factors\.

Figure 2:Connected Tensor\-Train construction of ChainDoRA\. The input\-side and output\-side TT contractions generateATTA\_\{\\mathrm\{TT\}\}andBTTB\_\{\\mathrm\{TT\}\}, respectively\. The adapter rankrrforms the boundary bond between the two contractions, while the remaining internal TT bonds are controlled by rankρ\\rho\.For a fixed physical indexiji\_\{j\}, define the corresponding TT\-core matrix slice as

Gj\(ij\)=𝒢j\(:,ij,:\)∈ℝRj−1×Rj\.G\_\{j\}^\{\(i\_\{j\}\)\}=\\mathcal\{G\}\_\{j\}\(:,i\_\{j\},:\)\\in\\mathbb\{R\}^\{R\_\{j\-1\}\\times R\_\{j\}\}\.\(16\)
Letϕin​\(⋅\)\\phi\_\{\\mathrm\{in\}\}\(\\cdot\)andϕout​\(⋅\)\\phi\_\{\\mathrm\{out\}\}\(\\cdot\)denote the flattening maps from the tensorized input and output indices to the original matrix coordinates\. The contraction of the firstppcores yields a1×r1\\times rvector for each input coordinate\. The corresponding column ofATTA\_\{\\mathrm\{TT\}\}is

ATT\[:,ϕin\(i1,…,ip\)\]=\(G1\(i1\)G2\(i2\)⋯Gp\(ip\)\)T\.A\_\{\\mathrm\{TT\}\}\\left\[:,\\phi\_\{\\mathrm\{in\}\}\(i\_\{1\},\\ldots,i\_\{p\}\)\\right\]=\\left\(G\_\{1\}^\{\(i\_\{1\}\)\}G\_\{2\}^\{\(i\_\{2\}\)\}\\cdots G\_\{p\}^\{\(i\_\{p\}\)\}\\right\)^\{\\mathrm\{T\}\}\.\(17\)
Similarly, contraction of the output\-side cores yields anr×1r\\times 1vector, and the corresponding row ofBTTB\_\{\\mathrm\{TT\}\}is

BTT\[ϕout\(o1,…,oq\),:\]=\(Gp\+1\(o1\)⋯GN\(oq\)\)T\.B\_\{\\mathrm\{TT\}\}\\left\[\\phi\_\{\\mathrm\{out\}\}\(o\_\{1\},\\ldots,o\_\{q\}\),:\\right\]=\\left\(G\_\{p\+1\}^\{\(o\_\{1\}\)\}\\cdots G\_\{N\}^\{\(o\_\{q\}\)\}\\right\)^\{\\mathrm\{T\}\}\.\(18\)
Therefore,

ATT∈ℝr×din,BTT∈ℝdout×r\.A\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\},\\qquad B\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\.\(19\)
Importantly,ATTA\_\{\\mathrm\{TT\}\}andBTTB\_\{\\mathrm\{TT\}\}are not stored as independent trainable dense matrices\. Their entries are differentiable functions of the TT cores on the two sides of the shared rank\-rrboundary\.

The ChainDoRA directional residual is defined as

Δ​VCD=s​BTT​ATT,s=αr\.\\Delta V\_\{\\mathrm\{CD\}\}=sB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\},\\qquad s=\\frac\{\\alpha\}\{r\}\.\(20\)
Since the intermediate dimension is stillrr,

rank⁡\(Δ​VCD\)≤r,\\operatorname\{rank\}\\left\(\\Delta V\_\{\\mathrm\{CD\}\}\\right\)\\leq r,\(21\)
regardless of the internal TT rankρ\\rho\. Increasingρ\\rhotherefore increases the representational capacity of the TT factors without changing the nominal upper bound on the rank of the final directional update\.

### 3\.3Magnitude–Direction Composition

The unnormalized ChainDoRA directional matrix is

VCD=W0\+s​BTT​ATT\.V\_\{\\mathrm\{CD\}\}=W\_\{0\}\+sB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}\.\(22\)
The effective adapted weight is obtained through row\-wise normalization and magnitude scaling,

WCD=m⊙VCD‖VCD‖row\.W\_\{\\mathrm\{CD\}\}=m\\odot\\frac\{V\_\{\\mathrm\{CD\}\}\}\{\\left\\\|V\_\{\\mathrm\{CD\}\}\\right\\\|\_\{\\mathrm\{row\}\}\}\.\(23\)
Substituting \([22](https://arxiv.org/html/2609.25058#S3.E22)\) gives the complete ChainDoRA reparameterization,

WCD=m⊙W0\+s​BTT​ATT‖W0\+s​BTT​ATT‖row\.W\_\{\\mathrm\{CD\}\}=m\\odot\\frac\{W\_\{0\}\+sB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}\}\{\\left\\\|W\_\{0\}\+sB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}\\right\\\|\_\{\\mathrm\{row\}\}\}\.\(24\)
The set of trainable parameters of an adapted layer is therefore

ΘCD=\{m,𝒢1,…,𝒢N\},\\Theta\_\{\\mathrm\{CD\}\}=\\left\\\{m,\\mathcal\{G\}\_\{1\},\\ldots,\\mathcal\{G\}\_\{N\}\\right\\\},\(25\)
whileW0W\_\{0\}remains frozen throughout fine\-tuning\.

### 3\.4Factored Row\-Norm Computation

Direct evaluation of‖W0\+s​BTT​ATT‖row\\\|W\_\{0\}\+sB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}\\\|\_\{\\mathrm\{row\}\}by first constructing the densedout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}matrixBTT​ATTB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}introduces unnecessary temporary storage\. The required norm can instead be evaluated through low\-rank intermediate quantities\. A related factored\-norm formulation has recently been studied for scaling DoRA implementations\[[14](https://arxiv.org/html/2609.25058#bib.bib3)\]\.

For notational simplicity, let

A=ATT,B=BTT\.A=A\_\{\\mathrm\{TT\}\},\\qquad B=B\_\{\\mathrm\{TT\}\}\.\(26\)
For output rowii,

‖W0,i:\+sBi:A‖22\\displaystyle\\left\\\|W\_\{0,i:\}\+sB\_\{i:\}A\\right\\\|\_\{2\}^\{2\}=‖W0,i:‖22\\displaystyle=\\left\\\|W\_\{0,i:\}\\right\\\|\_\{2\}^\{2\}\(27\)\+2s⟨W0,i:,Bi:A⟩\\displaystyle\+2s\\left\\langle W\_\{0,i:\},B\_\{i:\}A\\right\\rangle\+s2‖Bi:A‖22\.\\displaystyle\+s^\{2\}\\left\\\|B\_\{i:\}A\\right\\\|\_\{2\}^\{2\}\.
Define

qi=‖W0,i:‖22,q\_\{i\}=\\left\\\|W\_\{0,i:\}\\right\\\|\_\{2\}^\{2\},\(28\)
ci=∑k=1rBi​k​\(W0​AT\)i​k,c\_\{i\}=\\sum\_\{k=1\}^\{r\}B\_\{ik\}\\left\(W\_\{0\}A^\{\\mathrm\{T\}\}\\right\)\_\{ik\},\(29\)
and

ui=Bi:\(AAT\)Bi:T\.u\_\{i\}=B\_\{i:\}\\left\(AA^\{\\mathrm\{T\}\}\\right\)B\_\{i:\}^\{\\mathrm\{T\}\}\.\(30\)
The row norm is then

ni=qi\+2​s​ci\+s2​ui\+ϵ,n\_\{i\}=\\sqrt\{q\_\{i\}\+2sc\_\{i\}\+s^\{2\}u\_\{i\}\+\\epsilon\},\(31\)
whereϵ\\epsilonis a small numerical\-stability constant\. The dominant rank\-dependent intermediate matrices are

W0​AT∈ℝdout×rW\_\{0\}A^\{\\mathrm\{T\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\(32\)
and

A​AT∈ℝr×r,AA^\{\\mathrm\{T\}\}\\in\\mathbb\{R\}^\{r\\times r\},\(33\)
rather than the fulldout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}matrixB​ABA\.

The linear action of the low\-rank residual can likewise be evaluated sequentially\. For an inputxx,

z=ATT​x,z∈ℝr,z=A\_\{\\mathrm\{TT\}\}x,\\qquad z\\in\\mathbb\{R\}^\{r\},\(34\)
followed by

Δ​y=BTT​z\.\\Delta y=B\_\{\\mathrm\{TT\}\}z\.\(35\)
Let

gi=mini\.g\_\{i\}=\\frac\{m\_\{i\}\}\{n\_\{i\}\}\.\(36\)
The complete forward transformation can therefore be expressed as

y=g⊙\[W0​x\+s​BTT​\(ATT​x\)\]\.y=g\\odot\\left\[W\_\{0\}x\+sB\_\{\\mathrm\{TT\}\}\\left\(A\_\{\\mathrm\{TT\}\}x\\right\)\\right\]\.\(37\)
This formulation avoids materializingBTT​ATTB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}solely for either the linear residual or the row\-norm computation\.

### 3\.5Detached Direction Normalization

The original DoRA formulation notes that retaining the normalization term in the complete gradient graph introduces additional training\-memory overhead\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]\. Following its low\-memory modification, our implementation treats the dynamically computed directional norm as a constant during backpropagation\.

Let

n¯=stopgrad⁡\(‖VCD‖row\)\.\\bar\{n\}=\\operatorname\{stopgrad\}\\left\(\\left\\\|V\_\{\\mathrm\{CD\}\}\\right\\\|\_\{\\mathrm\{row\}\}\\right\)\.\(38\)
The training\-time reparameterization is therefore

WCD=m⊙VCDn¯\.W\_\{\\mathrm\{CD\}\}=m\\odot\\frac\{V\_\{\\mathrm\{CD\}\}\}\{\\bar\{n\}\}\.\(39\)
The forward numerical value is unchanged by the stop\-gradient operation; only gradient propagation through the normalization denominator is removed\. We use this detached\-denominator configuration for the reported ChainDoRA experiments\.

### 3\.6Balanced Nonzero TT Initialization

Initialization requires additional care for a multiplicative TT parameterization\. Conventional LoRA commonly initializes one low\-rank factor to zero so that the initial residual vanishes\. Directly transferring this strategy to a chain of multiple tensor factors is undesirable because a zero factor can suppress gradients to other multiplicative factors\. A similar zero\-gradient issue for zero\-initialized tensor factors has been reported for TT\-based PEFT in LoRETTA\[[7](https://arxiv.org/html/2609.25058#bib.bib21)\]\.

ChainDoRA therefore initializes every TT core with nonzero, fan\-in\-scaled Gaussian entries\. The magnitude vector is initialized from the frozen pretrained weight according to

m\(0\)=‖W0‖row\.m^\{\(0\)\}=\\left\\\|W\_\{0\}\\right\\\|\_\{\\mathrm\{row\}\}\.\(40\)
Because products of several tensor cores can develop strongly imbalanced scales, we next apply gauge\-preserving balancing between neighboring TT cores\. For any positive scalarλj\\lambda\_\{j\}, the transformation

𝒢j←λj​𝒢j,𝒢j\+1←λj−1​𝒢j\+1\\mathcal\{G\}\_\{j\}\\leftarrow\\lambda\_\{j\}\\mathcal\{G\}\_\{j\},\\qquad\\mathcal\{G\}\_\{j\+1\}\\leftarrow\\lambda\_\{j\}^\{\-1\}\\mathcal\{G\}\_\{j\+1\}\(41\)
leaves their joint contraction unchanged\. We use this freedom to reduce large scale disparities between adjacent cores\.

After contraction, the two directional factors are additionally balanced across the adapter boundary\. Let

arms=RMS⁡\(ATT\),brms=RMS⁡\(BTT\)\.a\_\{\\mathrm\{rms\}\}=\\operatorname\{RMS\}\\left\(A\_\{\\mathrm\{TT\}\}\\right\),\\qquad b\_\{\\mathrm\{rms\}\}=\\operatorname\{RMS\}\\left\(B\_\{\\mathrm\{TT\}\}\\right\)\.\(42\)
Choosing

γ=brmsarms\\gamma=\\sqrt\{\\frac\{b\_\{\\mathrm\{rms\}\}\}\{a\_\{\\mathrm\{rms\}\}\}\}\(43\)
and applying

ATT←γ​ATT,BTT←γ−1​BTTA\_\{\\mathrm\{TT\}\}\\leftarrow\\gamma A\_\{\\mathrm\{TT\}\},\\qquad B\_\{\\mathrm\{TT\}\}\\leftarrow\\gamma^\{\-1\}B\_\{\\mathrm\{TT\}\}\(44\)
preserves the directional product,

\(γ−1​BTT\)​\(γ​ATT\)=BTT​ATT,\\left\(\\gamma^\{\-1\}B\_\{\\mathrm\{TT\}\}\\right\)\\left\(\\gamma A\_\{\\mathrm\{TT\}\}\\right\)=B\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\},\(45\)
while reducing scale imbalance between the two sides of the TT chain\.

Unlike exact zero initialization, ChainDoRA starts with a small but nonzero directional perturbation\. We explicitly calibrate its relative magnitude as

δ0=‖s​BTT\(0\)​ATT\(0\)‖F‖W0‖F\.\\delta\_\{0\}=\\frac\{\\left\\\|sB\_\{\\mathrm\{TT\}\}^\{\(0\)\}A\_\{\\mathrm\{TT\}\}^\{\(0\)\}\\right\\\|\_\{\\mathrm\{F\}\}\}\{\\left\\\|W\_\{0\}\\right\\\|\_\{\\mathrm\{F\}\}\}\.\(46\)
For all experiments, the initialization is rescaled to satisfy

δ0=τ,τ=10−3\.\\delta\_\{0\}=\\tau,\\qquad\\tau=10^\{\-3\}\.\(47\)
Thus, ChainDoRA begins in a controlled neighborhood of the pretrained transformation while keeping all TT factors nonzero and trainable from the first optimization step\.

The calibration does not require materializing the dense productBTT​ATTB\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}\. Its Frobenius norm follows from ther×rr\\times rGram matrices,

‖BTT​ATT‖F2=tr⁡\[\(BTTT​BTT\)​\(ATT​ATTT\)\]\.\\left\\\|B\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}=\\operatorname\{tr\}\\left\[\\left\(B\_\{\\mathrm\{TT\}\}^\{\\mathrm\{T\}\}B\_\{\\mathrm\{TT\}\}\\right\)\\left\(A\_\{\\mathrm\{TT\}\}A\_\{\\mathrm\{TT\}\}^\{\\mathrm\{T\}\}\\right\)\\right\]\.\(48\)
This allows initialization calibration to be performed using low\-dimensional intermediate quantities\.

Algorithm[1](https://arxiv.org/html/2609.25058#alg1)summarizes the complete ChainDoRA training pipeline, including TT\-core initialization, construction of the directional factors, factored row normalization, parameter optimization, and final weight merging\.

Algorithm 1ChainDoRA Training and Weight Merging1:Input:Frozen weights

\{W0\(ℓ\)\}\\\{W\_\{0\}^\{\(\\ell\)\}\\\}, data

𝒟\\mathcal\{D\}, adapter rank

rr, TT rank

ρ\\rho, scale

α\\alpha, target

τ\\tau
2:

s←α/rs\\leftarrow\\alpha/r
3:Initialization

4:foreach adapted layer

ℓ\\elldo

5:Tensorize

din\(ℓ\)d\_\{\\mathrm\{in\}\}^\{\(\\ell\)\}and

dout\(ℓ\)d\_\{\\mathrm\{out\}\}^\{\(\\ell\)\}
6:Set

𝐑=\(1,ρ,…,ρ,r,ρ,…,ρ,1\)\\mathbf\{R\}=\(1,\\rho,\\ldots,\\rho,r,\\rho,\\ldots,\\rho,1\)
7:Initialize all

\{𝒢j\(ℓ\)\}j=1N\\\{\\mathcal\{G\}\_\{j\}^\{\(\\ell\)\}\\\}\_\{j=1\}^\{N\}with nonzero fan\-in\-scaled Gaussian entries

8:

m\(ℓ\)←‖W0\(ℓ\)‖rowm^\{\(\\ell\)\}\\leftarrow\\\|W\_\{0\}^\{\(\\ell\)\}\\\|\_\{\\mathrm\{row\}\}
9:Gauge\-balance neighboring TT cores and contract them into

ATT\(ℓ\)A\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}and

BTT\(ℓ\)B\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}
10:Balance the two factors across the rank\-

rrboundary

11:Calibrate

δ0\(ℓ\)=‖s​BTT\(ℓ\)​ATT\(ℓ\)‖F/‖W0\(ℓ\)‖F≈τ\\delta\_\{0\}^\{\(\\ell\)\}=\\\|sB\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}A\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}\\\|\_\{F\}/\\\|W\_\{0\}^\{\(\\ell\)\}\\\|\_\{F\}\\approx\\tau
12:endfor

13:Fine\-tuning

14:foreach mini\-batch

\(x,y\)∈𝒟\(x,y\)\\in\\mathcal\{D\}do

15:foreach adapted layer

ℓ\\elldo

16:Contract TT cores into

ATT\(ℓ\)A\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}and

BTT\(ℓ\)B\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}
17:

Δ​y\(ℓ\)←s​BTT\(ℓ\)​\(ATT\(ℓ\)​x\)\\Delta y^\{\(\\ell\)\}\\leftarrow sB\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}\(A\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}x\)
18:Compute factored row norm

n\(ℓ\)n^\{\(\\ell\)\}using Eqs\. \([28](https://arxiv.org/html/2609.25058#S3.E28)\)–\([31](https://arxiv.org/html/2609.25058#S3.E31)\)

19:

n¯\(ℓ\)←stopgrad⁡\(n\(ℓ\)\)\\bar\{n\}^\{\(\\ell\)\}\\leftarrow\\operatorname\{stopgrad\}\(n^\{\(\\ell\)\}\)
20:

y\(ℓ\)←\(m\(ℓ\)⊘n¯\(ℓ\)\)⊙\[W0\(ℓ\)​x\+Δ​y\(ℓ\)\]y^\{\(\\ell\)\}\\leftarrow\(m^\{\(\\ell\)\}\\oslash\\bar\{n\}^\{\(\\ell\)\}\)\\odot\[W\_\{0\}^\{\(\\ell\)\}x\+\\Delta y^\{\(\\ell\)\}\]
21:endfor

22:Compute

ℒresp\\mathcal\{L\}\_\{\\mathrm\{resp\}\}and update only

\{m\(ℓ\),𝒢1:N\(ℓ\)\}\\\{m^\{\(\\ell\)\},\\mathcal\{G\}\_\{1:N\}^\{\(\\ell\)\}\\\}with AdamW

23:endfor

24:Weight merging

25:foreach adapted layer

ℓ\\elldo

26:Contract final TT cores into

ATT\(ℓ\)A\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}and

BTT\(ℓ\)B\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}
27:

VCD\(ℓ\)←W0\(ℓ\)\+s​BTT\(ℓ\)​ATT\(ℓ\)V\_\{\\mathrm\{CD\}\}^\{\(\\ell\)\}\\leftarrow W\_\{0\}^\{\(\\ell\)\}\+sB\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}A\_\{\\mathrm\{TT\}\}^\{\(\\ell\)\}
28:

WCD\(ℓ\)←m\(ℓ\)⊙VCD\(ℓ\)/‖VCD\(ℓ\)‖rowW\_\{\\mathrm\{CD\}\}^\{\(\\ell\)\}\\leftarrow m^\{\(\\ell\)\}\\odot V\_\{\\mathrm\{CD\}\}^\{\(\\ell\)\}/\\\|V\_\{\\mathrm\{CD\}\}^\{\(\\ell\)\}\\\|\_\{\\mathrm\{row\}\}
29:endfor

30:Return:

\{WCD\(ℓ\)\}\\\{W\_\{\\mathrm\{CD\}\}^\{\(\\ell\)\}\\\}

### 3\.7Trainable\-Parameter Complexity

For a rank\-rrLoRA adapter, the number of trainable parameters associated with one adapted matrix is

PLoRA=r⁡\(din\+dout\)\.P\_\{\\mathrm\{LoRA\}\}=r\\left\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\\right\)\.\(49\)
DoRA additionally learns a magnitude parameter for each output row, giving

PDoRA=r⁡\(din\+dout\)\+dout\.P\_\{\\mathrm\{DoRA\}\}=r\\left\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\\right\)\+d\_\{\\mathrm\{out\}\}\.\(50\)
For ChainDoRA, the dense low\-rank matrices are replaced by the TT cores\. The number of trainable TT parameters is the standard TT storage cost

PTT=∑j=1NRj−1​nj​Rj\.P\_\{\\mathrm\{TT\}\}=\\sum\_\{j=1\}^\{N\}R\_\{j\-1\}n\_\{j\}R\_\{j\}\.\(51\)
Including the trainable magnitude vector,

PChainDoRA=dout\+∑j=1NRj−1​nj​Rj\.P\_\{\\mathrm\{ChainDoRA\}\}=d\_\{\\mathrm\{out\}\}\+\\sum\_\{j=1\}^\{N\}R\_\{j\-1\}n\_\{j\}R\_\{j\}\.\(52\)
The relative parameter ratio with respect to DoRA is therefore

ηCD/DoRA=dout\+∑j=1NRj−1​nj​Rjr⁡\(din\+dout\)\+dout\.\\eta\_\{\\mathrm\{CD/DoRA\}\}=\\frac\{d\_\{\\mathrm\{out\}\}\+\\sum\_\{j=1\}^\{N\}R\_\{j\-1\}n\_\{j\}R\_\{j\}\}\{r\(d\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}\)\+d\_\{\\mathrm\{out\}\}\}\.\(53\)
Correspondingly, the percentage reduction is

ℛCD/DoRA=100​\(1−ηCD/DoRA\)\.\\mathcal\{R\}\_\{\\mathrm\{CD/DoRA\}\}=100\\left\(1\-\\eta\_\{\\mathrm\{CD/DoRA\}\}\\right\)\.\(54\)
Equation \([52](https://arxiv.org/html/2609.25058#S3.E52)\) also makes the role of the internal TT rank explicit: reducingρ\\rhodecreases the TT\-core parameter count without changing the adapter boundary rankrr\. This provides a direct mechanism for controlling the parameter–capacity trade\-off, which is examined empirically in the TT\-rank ablation study\.

### 3\.8Adapter Merging and Inference

After fine\-tuning, the TT cores are contracted to obtainATTA\_\{\\mathrm\{TT\}\}andBTTB\_\{\\mathrm\{TT\}\}, and the final effective weight is computed using \([24](https://arxiv.org/html/2609.25058#S3.E24)\)\. The resulting matrixWCDW\_\{\\mathrm\{CD\}\}can then replace the corresponding frozen pretrained weight\. Consequently, the TT adapter does not require a separate inference branch after merging, analogous to the mergeable property of LoRA and DoRA\[[3](https://arxiv.org/html/2609.25058#bib.bib1),[4](https://arxiv.org/html/2609.25058#bib.bib2)\]\.

The proposed factorization is primarily intended to reduce the number of trainable and stored task\-specific parameters\. This reduction should not be interpreted as an equivalent reduction in training FLOPs, because contracting TT cores and evaluating the normalized directional update introduce additional operations\. We therefore report parameter efficiency and predictive performance separately in the experimental analysis\.

## 4Experimental Setup

We evaluate ChainDoRA under a controlled parameter\-efficient adaptation setting designed to isolate the effect of the proposed TT\-factorized directional parameterization\. All primary experiments use the same pretrained backbone, adaptation rank, target modules, optimization schedule, training data, and evaluation pipeline for LoRA, DoRA, and ChainDoRA\. Unless otherwise stated, only the adapter parameterization differs between methods\.

### 4\.1Backbone and Training Data

We use LLaMA\-7B\[[15](https://arxiv.org/html/2609.25058#bib.bib18)\]as the frozen pretrained backbone\. LLaMA is a decoder\-only Transformer family containing models ranging from 7B to 65B parameters\[[15](https://arxiv.org/html/2609.25058#bib.bib18)\]\. The 7B configuration is used throughout the primary experiments to maintain a fixed backbone while comparing the parameter efficiency of different adaptation methods\.

Our implementation follows the commonsense\-reasoning adaptation pipeline used in the public DoRA framework\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]\. Rather than training on the complete commonsense instruction collection, we construct a fixed adaptation pool containing 15,119 instruction–response examples\. The adaptation pool is derived from the Commonsense170K collection introduced with LLM\-Adapters\[[16](https://arxiv.org/html/2609.25058#bib.bib22)\]\. A validation subset of 120 examples is held out using a fixed random seed of 42, leaving 14,999 examples for parameter optimization\. All LoRA, DoRA, ChainDoRA, and ChainDoRA ablation experiments use exactly the same split\.

Each training instance consists of an instruction and its target response\. We employ*response\-only supervision*: tokens belonging to the instruction portion of the prompt are masked from the language\-modeling loss, and only response tokens contribute to optimization\. Let𝐲=\(y1,…,yT\)\\mathbf\{y\}=\(y\_\{1\},\\ldots,y\_\{T\}\)denote the tokenized instruction–response sequence, and let𝒮\\mathcal\{S\}denote the token positions corresponding to the target response\. The training objective is

ℒresp=−1\|𝒮\|∑t∈𝒮logpθ\(yt∣y<t\),\\mathcal\{L\}\_\{\\mathrm\{resp\}\}=\-\\frac\{1\}\{\|\\mathcal\{S\}\|\}\\sum\_\{t\\in\\mathcal\{S\}\}\\log p\_\{\\theta\}\\left\(y\_\{t\}\\mid y\_\{<t\}\\right\),\(55\)
where tokens outside𝒮\\mathcal\{S\}are ignored when computing the loss\. The same serialized instruction template and response delimiter are used during fine\-tuning and evaluation to avoid train–test differences caused by prompt formatting\.

### 4\.2Adaptation Configuration

For the main comparison, LoRA, DoRA, and ChainDoRA use the same adapter rank

and scaling coefficient

Adaptation is applied to five linear transformations in every Transformer block,

ℳfull=\{qproj,kproj,vproj,u​pproj,d​o​w​nproj\}\.\\mathcal\{M\}\_\{\\mathrm\{full\}\}=\\\{q\_\{\\mathrm\{proj\}\},k\_\{\\mathrm\{proj\}\},v\_\{\\mathrm\{proj\}\},up\_\{\\mathrm\{proj\}\},down\_\{\\mathrm\{proj\}\}\\\}\.\(58\)
For the 32 Transformer layers of LLaMA\-7B, this corresponds to 160 adapted linear modules\.

The main ChainDoRA configuration uses internal TT rank

The hidden dimension of LLaMA\-7B is tensorized as

4096=16×16×16,4096=16\\times 16\\times 16,\(60\)
whereas the feed\-forward dimension is represented as

11008=16×16×43\.11008=16\\times 16\\times 43\.\(61\)
Table[2](https://arxiv.org/html/2609.25058#S4.T2)summarizes the resulting mode assignments\.

Table 2:Tensorization used for ChainDoRA on LLaMA\-7B\.Moduledind\_\{\\mathrm\{in\}\}doutd\_\{\\mathrm\{out\}\}Input ModesOutput Modesqprojq\_\{\\mathrm\{proj\}\}40964096\(16,16,16\)\(16,16,16\)\(16,16,16\)\(16,16,16\)kprojk\_\{\\mathrm\{proj\}\}40964096\(16,16,16\)\(16,16,16\)\(16,16,16\)\(16,16,16\)vprojv\_\{\\mathrm\{proj\}\}40964096\(16,16,16\)\(16,16,16\)\(16,16,16\)\(16,16,16\)u​pprojup\_\{\\mathrm\{proj\}\}409611008\(16,16,16\)\(16,16,16\)\(16,16,43\)\(16,16,43\)d​o​w​nprojdown\_\{\\mathrm\{proj\}\}110084096\(16,16,43\)\(16,16,43\)\(16,16,16\)\(16,16,16\)

Under this configuration, conventional LoRA contains 56\.10M trainable parameters and DoRA contains 56\.98M trainable parameters\. ChainDoRA withρ=16\\rho=16requires 5\.35M trainable parameters while preserving the same adapter boundary rankr=32r=32\. Exact parameter\-efficiency comparisons are reported in Section[5](https://arxiv.org/html/2609.25058#S5)\.

### 4\.3Optimization and Implementation Details

All methods are fine\-tuned for three epochs using AdamW\. The learning rate is set to

η=2×10−4,\\eta=2\\times 10^\{\-4\},\(62\)
with zero weight decay and 100 warm\-up steps\. The global batch size and micro\-batch size are both 4, resulting in one optimizer update per micro\-batch on the single\-GPU setup\. The maximum tokenized sequence length is 256 and the adapter dropout probability is 0\.05\.

Training uses FP16 precision and gradient checkpointing\. Validation is performed every 80 optimization steps, and adapter checkpoints are saved at the same interval\. At the end of training, the best validation checkpoint is restored\. All experiments are implemented in PyTorch using the PEFT\-compatible DoRA codebase extended with the proposed ChainDoRA module\. Training is conducted on a single NVIDIA GeForce RTX 5090 GPU with 32 GB of memory\.

Table[3](https://arxiv.org/html/2609.25058#S4.T3)summarizes the common training configuration\.

Table 3:Training configuration used for the primary comparison\.ConfigurationValueBackboneLLaMA\-7BAdaptation pool15,119 examplesOptimization examples14,999Validation examples120Training epochs3OptimizerAdamWLearning rate2×10−42\\times 10^\{\-4\}Weight decay0Warm\-up steps100Global batch size4Micro\-batch size4Maximum sequence length256Adapter dropout0\.05Adapter rankrr32Scaling coefficientα\\alpha64Main ChainDoRA TT rankρ\\rho16Target modulesQ, K, V, Up, DownValidation interval80 stepsCheckpoint interval80 stepsPrecisionFP16Gradient checkpointingEnabledSupervisionResponse onlyHardwareNVIDIA RTX 5090 \(32 GB\)

### 4\.4Commonsense Reasoning Benchmarks

We evaluate the adapted models on seven established commonsense reasoning benchmarks covering complementary forms of linguistic, physical, social, coreference, and science\-oriented reasoning\.

BoolQ\[[17](https://arxiv.org/html/2609.25058#bib.bib12)\]consists of naturally occurring yes/no questions and evaluates binary question answering\. PIQA\[[18](https://arxiv.org/html/2609.25058#bib.bib13)\]measures physical commonsense reasoning through two candidate solutions to everyday physical situations\. SocialIQA\[[19](https://arxiv.org/html/2609.25058#bib.bib14)\]evaluates reasoning about social interactions, intentions, and consequences\. WinoGrande\[[20](https://arxiv.org/html/2609.25058#bib.bib15)\]is a large\-scale adversarially constructed Winograd\-style benchmark designed to reduce exploitable annotation biases\.

ARC\[[21](https://arxiv.org/html/2609.25058#bib.bib16)\]consists of natural grade\-school science questions and is divided into ARC\-Easy and ARC\-Challenge\. The Challenge partition was constructed from questions that were incorrectly answered by baseline retrieval and co\-occurrence systems, making it a more difficult subset\. OpenBookQA\[[22](https://arxiv.org/html/2609.25058#bib.bib17)\]evaluates elementary science question answering that may require combining a provided scientific fact with broader common knowledge\.

Table[4](https://arxiv.org/html/2609.25058#S4.T4)lists the seven evaluation sets and the number of examples used in our experiments\.

Table 4:Commonsense reasoning benchmarks used for evaluation\.BenchmarkTest ExamplesPrimary Reasoning TypeBoolQ3,270Yes/no reasoningPIQA1,838Physical commonsenseSocialIQA1,954Social commonsenseWinoGrande1,267Coreference / commonsenseARC\-Easy2,376Science reasoningARC\-Challenge1,172Challenging science reasoningOpenBookQA500Science \+ common knowledgeTotal12,377–

### 4\.5Evaluation Protocol and Metrics

A single evaluation pipeline is used for all methods\. Each test example is converted to an instruction containing the question, candidate answers, and the required benchmark\-specific output identifier\. The same instruction serialization used during fine\-tuning is retained during inference, with the response field left empty for generation\.

Before evaluation, the learned adapter parameters are merged into their corresponding linear transformations\. Generation is deterministic and uses beam search with four beams,

Nbeam=4,N\_\{\\mathrm\{beam\}\}=4,\(63\)
with a maximum of 32 newly generated tokens,

Tgenmax=32\.T\_\{\\mathrm\{gen\}\}^\{\\max\}=32\.\(64\)
Sampling is disabled\. The generated continuation is mapped to the benchmark\-specific answer vocabulary\. Examples includetrue/falsefor BoolQ,solution1/solution2for PIQA,option1/option2for WinoGrande, andanswer1,answer2,…\\ldotsfor the multiple\-choice question\-answering datasets\. The first valid answer identifier appearing in the generated response is used as the prediction\. A generation containing no valid identifier is counted as incorrect\.

The primary metric for benchmarkjjis classification accuracy,

Accj=Ncorrect,jNtotal,j×100\.\\mathrm\{Acc\}\_\{j\}=\\frac\{N\_\{\\mathrm\{correct\},j\}\}\{N\_\{\\mathrm\{total\},j\}\}\\times 100\.\(65\)
To summarize performance across the complete benchmark suite, we report the unweighted macro\-average over the seven tasks,

Avg\.=17​∑j=17Accj\.\\mathrm\{Avg\.\}=\\frac\{1\}\{7\}\\sum\_\{j=1\}^\{7\}\\mathrm\{Acc\}\_\{j\}\.\(66\)
The macro\-average gives equal weight to each benchmark regardless of the number of test examples\.

### 4\.6Ablation Study Design

We conduct two principal ablation studies to examine the effects of TT capacity and adapter placement\.

First, the internal TT rank is varied while keeping the adapter rank fixed atr=32r=32:

ρ∈\{4,8,16\}\.\\rho\\in\\\{4,8,16\\\}\.\(67\)
All five projection types inℳfull\\mathcal\{M\}\_\{\\mathrm\{full\}\}remain adapted in this experiment\. This ablation isolates the effect of TT representation capacity without changing the nominal rank of the directional update\.

Second, we vary the locations at which ChainDoRA is inserted\. The evaluated target sets are

ℳQV=\{qproj,vproj\},\\mathcal\{M\}\_\{\\mathrm\{QV\}\}=\\\{q\_\{\\mathrm\{proj\}\},v\_\{\\mathrm\{proj\}\}\\\},\(68\)
ℳQKV=\{qproj,kproj,vproj\},\\mathcal\{M\}\_\{\\mathrm\{QKV\}\}=\\\{q\_\{\\mathrm\{proj\}\},k\_\{\\mathrm\{proj\}\},v\_\{\\mathrm\{proj\}\}\\\},\(69\)
ℳMLP=\{u​pproj,d​o​w​nproj\},\\mathcal\{M\}\_\{\\mathrm\{MLP\}\}=\\\{up\_\{\\mathrm\{proj\}\},down\_\{\\mathrm\{proj\}\}\\\},\(70\)
and

ℳQV\+MLP=\{qproj,vproj,u​pproj,d​o​w​nproj\}\.\\mathcal\{M\}\_\{\\mathrm\{QV\+MLP\}\}=\\\{q\_\{\\mathrm\{proj\}\},v\_\{\\mathrm\{proj\}\},up\_\{\\mathrm\{proj\}\},down\_\{\\mathrm\{proj\}\}\\\}\.\(71\)
These configurations are compared with full QKV\+MLP adaptation\. Unless otherwise specified, the projection\-location study usesρ=16\\rho=16\.

We additionally evaluate a highly compressed configuration combiningρ=8\\rho=8with MLP\-only adaptation\. This variant is included to examine whether reductions in TT capacity and adaptation coverage can be combined while retaining useful downstream performance\.

Table 5:ChainDoRA configurations used in the ablation studies\.Configurationρ\\rhoTarget SetTrainable Params\.TT16\-Full16QKV\+MLP5,346,816TT8\-Full8QKV\+MLP2,784,000TT4\-Full4QKV\+MLP1,748,352TT16\-QV16QV1,867,776TT16\-QKV16QKV2,801,664TT16\-MLP16MLP2,545,152TT16\-QV\+MLP16QV\+MLP4,412,928TT8\-MLP8MLP1,383,168

### 4\.7Cross\-Architecture Parameter\-Scaling Analysis

To evaluate how the parameterization scales beyond LLaMA\-7B, we additionally perform a configuration\-level parameter analysis across model families and sizes\. This analysis does not require loading pretrained model weights or performing additional fine\-tuning\. Instead, architectural configurations are used to determine the dimensions and number of target linear transformations, after which the trainable parameter counts of LoRA, DoRA, and ChainDoRA are computed analytically\.

The analysis includes Qwen2\.5\-3B and Qwen2\.5\-72B\[[23](https://arxiv.org/html/2609.25058#bib.bib23)\], Mistral\-7B\[[24](https://arxiv.org/html/2609.25058#bib.bib24)\], Phi\-3\-Medium\[[25](https://arxiv.org/html/2609.25058#bib.bib25)\], OPT\-30B\[[26](https://arxiv.org/html/2609.25058#bib.bib26)\], and Falcon\-40B\[[27](https://arxiv.org/html/2609.25058#bib.bib27)\]\. These architectures span approximately 3B to 72B parameters and represent multiple Transformer families with different attention and feed\-forward configurations\. For each architecture, target modules are selected according to their native attention and feed\-forward projection structure\. Full\-target ChainDoRA is used for the primary apples\-to\-apples comparison with LoRA and DoRA, while selective MLP\-only variants are reported separately as additional compression configurations\.

This experiment is intended to characterize parameter scaling rather than downstream accuracy across these model families; no pretrained weights are loaded and no performance claims are made for the configuration\-only models\.

## 5Results and Discussion

This section evaluates ChainDoRA from three complementary perspectives: predictive performance, trainable\-parameter efficiency, and sensitivity to the design of the tensorized adapter\. We first compare ChainDoRA with matched LoRA and DoRA baselines on the seven commonsense reasoning benchmarks\. We then study the effect of the internal TT rank and the placement of ChainDoRA across attention and feed\-forward projections\. Finally, we examine the parameter scaling of the proposed representation across several large\-language\-model architectures\.

### 5\.1Overall Commonsense Reasoning Performance

Table[6](https://arxiv.org/html/2609.25058#S5.T6)reports the primary comparison among LoRA, DoRA, and ChainDoRA\. All three methods use the same LLaMA\-7B backbone, rankr=32r=32, scaling coefficientα=64\\alpha=64, target modules, optimization schedule, response\-only training objective, and evaluation protocol\. ChainDoRA uses the full QKV\+MLP target set with internal TT rankρ=16\\rho=16\.

Table 6:Commonsense reasoning performance of LoRA, DoRA, and ChainDoRA under the controlled 15,119\-example adaptation setting\. Accuracy is reported in percent\. The best accuracy in each benchmark column is shown in bold\.MethodParams\.BoolQPIQASocialIQAWinoGrandeARC\-EARC\-COBQAAvg\.LoRA56\.10M64\.8372\.0972\.4765\.8278\.0362\.8873\.0069\.88DoRA56\.98M65\.6074\.5470\.2165\.5176\.6060\.0773\.2069\.39ChainDoRA5\.35M66\.2477\.6471\.8066\.3082\.3267\.5874\.2072\.30

ChainDoRA achieves the highest macro\-average accuracy, reaching 72\.30% across the seven benchmarks\. This corresponds to improvements of 2\.42 percentage points over LoRA and 2\.91 percentage points over DoRA\. Relative to LoRA, ChainDoRA achieves higher accuracy on six of the seven benchmarks, with particularly notable improvements on PIQA, ARC\-Easy, and ARC\-Challenge\. The respective gains are 5\.55, 4\.29, and 4\.70 percentage points\. LoRA obtains the highest SocialIQA score, exceeding ChainDoRA by 0\.67 percentage points\.

Compared with DoRA, ChainDoRA achieves higher accuracy on all seven tasks\. The largest difference is observed on ARC\-Challenge, where accuracy increases from 60\.07% to 67\.58%, corresponding to a 7\.51\-percentage\-point improvement\. ARC\-Easy increases from 76\.60% to 82\.32%, a gain of 5\.72 percentage points\. These results show that, under the controlled adaptation setting considered here, the connected TT parameterization achieves higher predictive performance while substantially reducing the number of trainable parameters\.

DoRA obtains a slightly lower average accuracy than LoRA in our experiments \(69\.39% versus 69\.88%\)\. This observation is specific to the matched training and evaluation configuration used in this work\. The original DoRA study reports improvements over LoRA across multiple language and multimodal settings\[[4](https://arxiv.org/html/2609.25058#bib.bib2)\]; therefore, the present result should not be interpreted as a general comparison of the learning capabilities of LoRA and DoRA\. Instead, the matched baselines establish the relative performance of ChainDoRA within the controlled response\-only adaptation regime examined in this study\.

### 5\.2Parameter Efficiency

The predictive improvement of ChainDoRA is accompanied by a substantial reduction in trainable parameters\. Table[7](https://arxiv.org/html/2609.25058#S5.T7)summarizes the parameter requirements of the three primary methods\.

Table 7:Trainable\-parameter efficiency of the primary methods\. The compression factor indicates how many times larger the corresponding baseline adapter is than ChainDoRA\.MethodTrainable Params\.CompressionReductionLoRA56,098,81610\.49×10\.49\\times90\.47%DoRA56,975,36010\.66×10\.66\\times90\.62%ChainDoRA5,346,8161\.00×1\.00\\times–

At identical adapter rankr=32r=32, ChainDoRA reduces the trainable parameter count by 90\.47% relative to LoRA and 90\.62% relative to DoRA\. Equivalently, LoRA and DoRA require approximately10\.49×10\.49\\timesand10\.66×10\.66\\timesmore trainable parameters, respectively\. Importantly, this reduction is achieved without decreasing the nominal rank bound of the directional update: all three methods user=32r=32, while ChainDoRA controls the complexity of the factor construction independently through the TT rankρ\\rho\.

The relationship between trainable parameter count and average benchmark accuracy is visualized in Fig\.[3](https://arxiv.org/html/2609.25058#S5.F3)\. ChainDoRA\-TT16 occupies a favorable region of this parameter–accuracy space, combining the highest average accuracy among the primary methods with a substantially smaller task\-specific parameter footprint\.

Figure 3:Parameter–accuracy trade\-off among LoRA, DoRA, and representative ChainDoRA configurations\.
### 5\.3Effect of Tensor\-Train Rank

The internal TT rankρ\\rhodetermines the capacity of the connected tensor representation without altering the adapter boundary rankrr\. To evaluate this effect, we vary

ρ∈\{4,8,16\},\\rho\\in\\\{4,8,16\\\},\(72\)
while keepingr=32r=32and adapting all QKV\+MLP projections\.

Table 8:Effect of the internal TT rank on ChainDoRA under full QKV\+MLP adaptation with a fixed adapter rank ofr=32r=32\. Accuracy is reported in percent\. The best accuracy in each benchmark column is shown in bold\.𝝆\\boldsymbol\{\\rho\}Params\.BoolQPIQASocialIQAWinoGrandeARC\-EARC\-COBQAAvg\.41\.748M57\.1675\.6366\.3359\.9177\.9058\.6266\.8066\.0582\.784M64\.6276\.6167\.9663\.8580\.9862\.2971\.0069\.61165\.347M66\.2477\.6471\.8066\.3082\.3267\.5874\.2072\.30

Table[8](https://arxiv.org/html/2609.25058#S5.T8)shows a monotonic capacity–performance relationship over the evaluated ranks\. Increasingρ\\rhofrom 4 to 8 improves the seven\-task average from 66\.05% to 69\.61%, and increasingρ\\rhoto 16 further raises it to 72\.30%\.

Reducingρ\\rhofrom 16 to 8 lowers the trainable parameter count from 5\.347M to 2\.784M, corresponding to a 47\.93% reduction, while reducing average accuracy by 2\.69 percentage points\. Atρ=4\\rho=4, the adapter uses 1\.748M trainable parameters, a 67\.30% reduction relative to TT16\-Full, with a 6\.25\-percentage\-point decrease in average accuracy\.

These results empirically support the intended separation between the adapter rankrrand the TT capacity parameterρ\\rho\. The former fixes the maximum rank of the resulting directional update, whereas the latter provides an independent mechanism for adjusting the expressiveness and storage cost of its tensorized construction\. Similar rank–compression trade\-offs have been observed in previous TT\-based adaptation studies\[[7](https://arxiv.org/html/2609.25058#bib.bib21),[8](https://arxiv.org/html/2609.25058#bib.bib10)\]; however, in ChainDoRA this trade\-off occurs specifically within the directional component of magnitude–direction adaptation\.

Figure 4:Effect of internal TT rankρ\\rhoon the seven\-task average accuracy and ChainDoRA parameter count\. Increasingρ\\rhoimproves predictive performance while increasing the size of the TT representation\.
### 5\.4Effect of Adaptation Location

We next investigate whether the full set of attention and feed\-forward projections is necessary for effective ChainDoRA adaptation\. Table[9](https://arxiv.org/html/2609.25058#S5.T9)compares full QKV\+MLP adaptation with selective insertion into QV, QKV, MLP, and QV\+MLP projections\. All variants in this experiment useρ=16\\rho=16\.

Table 9:Effect of ChainDoRA adapter placement\.TargetParams\. \(M\)Avg\. \(%\)𝚫\\Delta\(pp\)QV1\.86870\.43−1\.87\-1\.87QKV2\.80268\.60−3\.70\-3\.70MLP2\.54570\.93−1\.37\-1\.37QV\+MLP4\.41370\.93−1\.37\-1\.37QKV\+MLP5\.34772\.30–Full QKV\+MLP adaptation provides the highest overall accuracy\. However, selective adaptation reveals that a substantial portion of the performance can be retained with considerably fewer parameters\. In particular, MLP\-only adaptation achieves 70\.93% average accuracy with 2\.545M trainable parameters, reducing the adapter size by 52\.40% relative to TT16\-Full while losing only 1\.37 percentage points\.

The QV\+MLP configuration also achieves 70\.93% average accuracy, matching the MLP\-only configuration at the reported precision, but requires 4\.413M trainable parameters\. Thus, adding Q and V projections increases the parameter count by approximately 1\.868M without improving the reported macro\-average\. Among the selective configurations, MLP\-only adaptation therefore provides the strongest parameter–accuracy balance\.

Interestingly, QKV adaptation obtains a lower average accuracy than QV adaptation despite using more trainable parameters\. This result indicates that increasing the number of adapted projection types does not necessarily produce monotonic performance improvements under a fixed TT configuration\. The effectiveness of the adapter therefore depends not only on its total parameter count but also on where the structured update is introduced\.

![Refer to caption](https://arxiv.org/html/2609.25058v1/projection_ablation_heatmap.png)Figure 5:Per\-benchmark performance of ChainDoRA under different adapter\-placement strategies\.
### 5\.5Combined Rank and Placement Compression

The TT\-rank and projection\-location experiments suggest two independent ways to reduce ChainDoRA’s parameter count: decreasing the internal TT rank and restricting adaptation to a subset of projections\. We therefore evaluate a combined configuration usingρ=8\\rho=8and MLP\-only adaptation\.

Table[9](https://arxiv.org/html/2609.25058#S5.T9)compares full QKV\+MLP adaptation with selective placement of ChainDoRA\. Here,Δ\\Deltadenotes the change in seven\-task average accuracy, in percentage points, relative to the full QKV\+MLP configuration\.

Table 10:Representative ChainDoRA parameter–accuracy trade\-offs\.ConfigurationParams\. \(M\)Avg\. \(%\)TT16\-Full5\.34772\.30TT16\-MLP2\.54570\.93TT8\-Full2\.78469\.61TT8\-MLP1\.38368\.97TT4\-Full1\.74866\.05The TT8\-MLP variant requires only 1\.383M trainable parameters, representing a 74\.13% reduction relative to TT16\-Full\. Compared with TT8\-Full, MLP\-only placement removes 50\.32% of the trainable parameters while reducing average accuracy by only 0\.64 percentage points\.

Compared with TT4\-Full, TT8\-MLP uses 20\.89% fewer parameters while achieving a 2\.92\-percentage\-point higher average accuracy\. Thus, TT8\-MLP strictly dominates TT4\-Full with respect to both parameter count and average accuracy within the evaluated configuration space\.

### 5\.6Cross\-Architecture Parameter Scaling

The preceding experiments evaluate downstream accuracy on LLaMA\-7B\. To determine whether the parameter savings are specific to this architecture, we additionally perform the configuration\-level analysis described in Section[4\.7](https://arxiv.org/html/2609.25058#S4.SS7)\. This experiment calculates trainable parameter counts from model configurations without loading pretrained weights and therefore measures structural scaling only\.

Table[11](https://arxiv.org/html/2609.25058#S5.T11)compares full\-target DoRA with full\-target ChainDoRA\-TT16 across six representative architectures\. Here, CD\-TT16 denotes the full\-target ChainDoRA configuration with internal TT rankρ=16\\rho=16\.

Table 11:Cross\-architecture trainable\-parameter scaling\.ModelParams\. \(M\)Red\.Comp\.DoRACD\-TT16\(%\)\(×\\times\)Qwen2\.5\-3B40\.674\.9787\.778\.18Mistral\-7B57\.415\.4690\.4810\.51Phi\-3\-Medium100\.256\.0993\.9216\.45OPT\-30B178\.9112\.4293\.0614\.40Falcon\-40B193\.7212\.1293\.7415\.98Qwen2\.5\-72B286\.3019\.7593\.1014\.49ChainDoRA\-TT16 requires fewer trainable parameters than DoRA for every architecture considered\. The compression factor ranges from8\.18×8\.18\\timeson Qwen2\.5\-3B to16\.45×16\.45\\timeson Phi\-3\-Medium, with an average compression factor of approximately13\.34×13\.34\\timesacross the six configurations\. The absolute savings become increasingly large for the larger models; for example, the Qwen2\.5\-72B configuration requires 286\.30M DoRA parameters but only 19\.75M ChainDoRA\-TT16 parameters under the corresponding target\-module mapping\.

Figure 6:Trainable\-parameter scaling of full\-target DoRA and ChainDoRA\-TT16 across representative language\-model architectures\.More aggressive selective configurations provide further reductions\. For example, the TT8\-MLP variant yields compression factors ranging from approximately28\.9×28\.9\\timesto43\.7×43\.7\\timesrelative to full\-target DoRA across the same architecture set\. Because these configurations adapt fewer modules, they are treated separately from the primary full\-target comparison\.

### 5\.7Discussion and Limitations

The experimental results reveal three main properties of ChainDoRA\. First, the connected TT representation can substantially reduce the parameter cost of DoRA\-style directional adaptation without requiring a smaller adapter boundary rank\. The primary TT16 configuration uses only 5\.35M trainable parameters while obtaining the highest average accuracy in the controlled seven\-task comparison\. This suggests that a densely parameterized pair of directional factors is not necessary to obtain an effective rank\-3232update in the studied setting\.

Second, parameter efficiency is controlled by more than a single scalar compression parameter\. The TT\-rank experiment shows thatρ\\rhoprovides a direct capacity–accuracy trade\-off, while the projection study demonstrates that adapter placement also has a substantial effect\. In particular, MLP\-only adaptation retains much of the performance of full adaptation with approximately half of the trainable parameters\. Combining these two design dimensions produces the TT8\-MLP configuration, which improves upon TT4\-Full in both parameter count and accuracy\. These observations motivate treating TT rank and target\-module placement as complementary design variables rather than optimizing either independently\.

Third, the configuration\-level scaling experiment indicates that the parameter savings are not restricted to the LLaMA\-7B dimensions\. The full\-target ChainDoRA construction remains substantially smaller than DoRA across architectures ranging from approximately 3B to 72B parameters\. However, this experiment evaluates parameter counts only and should not be interpreted as evidence that the LLaMA\-7B accuracy gains transfer directly to these other model families\.

Several limitations remain\. The downstream experiments are conducted using a single pretrained backbone and a fixed 15,119\-example adaptation pool rather than exhaustive full\-dataset fine\-tuning\. The reported conclusions should therefore be interpreted as evidence from a controlled parameter\-efficient adaptation setting rather than as claims of state\-of\-the\-art commonsense reasoning performance\. In addition, the cross\-architecture study is analytical and does not include model training\. The primary baselines use matched optimization hyperparameters to isolate adapter parameterization; method\-specific hyperparameter searches could change their absolute performance\. Future work should evaluate ChainDoRA across additional model families, larger adaptation datasets, multiple random seeds, and broader reasoning and generation benchmarks\. Further investigation of TT contraction kernels and training\-time efficiency would also be valuable, since reductions in trainable parameter count do not necessarily translate directly into equivalent reductions in computational cost\.

## 6Conclusion

This work introduced*ChainDoRA*, a parameter\-efficient adaptation framework that combines magnitude–direction weight decomposition with a connected Tensor\-Train representation of the directional low\-rank factors\. Instead of directly optimizing the dense matrices used by LoRA and DoRA, ChainDoRA constructsATT∈ℝr×dinA\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}andBTT∈ℝdout×rB\_\{\\mathrm\{TT\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}from contractions of a shared TT chain\. The adapter rankrris retained as the boundary rank between the input\- and output\-side contractions, while an independent internal TT rankρ\\rhocontrols the capacity and parameter cost of the structured representation\.

Under a controlled 15,119\-example adaptation setting with LLaMA\-7B, ChainDoRA\-TT16 achieved a seven\-task average commonsense reasoning accuracy of 72\.30%, compared with 69\.88% for LoRA and 69\.39% for DoRA\. At the same adapter rankr=32r=32, ChainDoRA required only 5\.35M trainable parameters, compared with 56\.10M for LoRA and 56\.98M for DoRA, corresponding to a 90\.62% reduction relative to DoRA\. The TT\-rank ablation further demonstrated a monotonic capacity–performance trade\-off acrossρ∈\{4,8,16\}\\rho\\in\\\{4,8,16\\\}, while the adapter\-placement study showed that selective MLP adaptation can retain much of the full QKV\+MLP ChainDoRA performance with substantially fewer trainable parameters\. Combining these two compression dimensions produced the TT8\-MLP configuration, which achieved a more favorable parameter–accuracy operating point than TT4\-Full\.

The configuration\-level scaling analysis further showed that the structural parameter savings of ChainDoRA persist across model architectures ranging from approximately 3B to 72B parameters, although downstream performance on these additional architectures remains to be evaluated\. Overall, the results suggest that the dense directional factors used in conventional magnitude–direction adaptation can be replaced by a structured tensor representation while retaining strong downstream performance and substantially reducing the number of task\-specific trainable parameters\.

Future work will investigate ChainDoRA across additional language\-model families, larger and more diverse adaptation datasets, multiple random seeds, and broader reasoning and generation tasks\. Further optimization of TT contraction and normalization kernels may also improve training\-time efficiency, since reductions in trainable parameter count do not necessarily translate directly into proportional reductions in computational cost\.

## References

- \[1\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for nlp\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 2790–2799\.Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.25058#S2.SS1.p1.1)\.
- \[2\]\(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing,pp\. 4582–4597\.External Links:[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.353)Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.25058#S2.SS1.p1.1)\.
- \[3\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p1.1),[§1](https://arxiv.org/html/2609.25058#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.25058#S2.SS1.p2.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.2.1),[§3\.1](https://arxiv.org/html/2609.25058#S3.SS1.p3.1),[§3\.8](https://arxiv.org/html/2609.25058#S3.SS8.p1.1)\.
- \[4\]S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen\(2024\)DoRA: weight\-decomposed low\-rank adaptation\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32100–32121\.External Links:[Link](https://proceedings.mlr.press/v235/liu24bn.html)Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.25058#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.4.1),[§3\.1](https://arxiv.org/html/2609.25058#S3.SS1.p10.1),[§3\.5](https://arxiv.org/html/2609.25058#S3.SS5.p1.1),[§3\.8](https://arxiv.org/html/2609.25058#S3.SS8.p1.1),[§3](https://arxiv.org/html/2609.25058#S3.p1.1),[§4\.1](https://arxiv.org/html/2609.25058#S4.SS1.p2.1),[§5\.1](https://arxiv.org/html/2609.25058#S5.SS1.p4.1)\.
- \[5\]I\. V\. Oseledets\(2011\)Tensor\-train decomposition\.SIAM Journal on Scientific Computing33\(5\),pp\. 2295–2317\.External Links:[Document](https://dx.doi.org/10.1137/090752286)Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.25058#S2.SS3.p1.1),[§3\.2](https://arxiv.org/html/2609.25058#S3.SS2.p5.1),[§3](https://arxiv.org/html/2609.25058#S3.p1.1)\.
- \[6\]A\. Novikov, D\. Podoprikhin, A\. Osokin, and D\. P\. Vetrov\(2015\)Tensorizing neural networks\.InAdvances in Neural Information Processing Systems,Vol\.28\.Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.25058#S2.SS3.p1.1)\.
- \[7\]Y\. Yang, J\. Zhou, N\. Wong, and Z\. Zhang\(2024\)LoRETTA: low\-rank economic tensor\-train adaptation for ultra\-low\-parameter fine\-tuning of large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Mexico City, Mexico,pp\. 3161–3176\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.174)Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.25058#S2.SS3.p2.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.5.1),[§3\.6](https://arxiv.org/html/2609.25058#S3.SS6.p1.1),[§5\.3](https://arxiv.org/html/2609.25058#S5.SS3.p6.1)\.
- \[8\]A\. Anjum, M\. E\. Eren, I\. Boureima, B\. Alexandrov, and M\. Bhattarai\(2024\)Tensor train low\-rank approximation \(TT\-LoRA\): democratizing ai with accelerated llms\.arXiv preprint arXiv:2408\.01008\.Cited by:[§1](https://arxiv.org/html/2609.25058#S1.p3.1),[§2\.3](https://arxiv.org/html/2609.25058#S2.SS3.p2.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.6.1),[§5\.3](https://arxiv.org/html/2609.25058#S5.SS3.p6.1)\.
- \[9\]Q\. Zhang, M\. Chen, A\. Bukharin, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)Adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2609.25058#S2.SS1.p3.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.3.1)\.
- \[10\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 10088–10115\.Cited by:[§2\.1](https://arxiv.org/html/2609.25058#S2.SS1.p3.1)\.
- \[11\]D\. J\. Kopiczko, T\. Blankevoort, and Y\. M\. Asano\(2024\)VeRA: vector\-based random matrix adaptation\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2609.25058#S2.SS1.p3.1)\.
- \[12\]X\. Hu, X\. Cheng, P\. Liu, W\. Liu, J\. Luan, B\. Wang, and Y\. Liu\(2024\)DoTA: weight\-decomposed tensor adaptation for large language models\.arXiv preprint arXiv:2412\.20891\.Cited by:[§2\.3](https://arxiv.org/html/2609.25058#S2.SS3.p3.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.7.1)\.
- \[13\]J\. Qi, C\. Liu, S\. M\. Siniscalchi, C\. H\. Yang, and M\. Hsieh\(2025\)Joint tensor\-train parameterization for efficient and expressive low\-rank adaptation\.arXiv preprint arXiv:2506\.16456\.Cited by:[§2\.3](https://arxiv.org/html/2609.25058#S2.SS3.p4.1),[Table 1](https://arxiv.org/html/2609.25058#S2.T1.2.1.8.1)\.
- \[14\]A\. Zelenin and A\. Zhuravlyova\(2026\)Scaling DoRA: high\-rank adaptation via factored norms and fused kernels\.arXiv preprint arXiv:2603\.22276\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2603.22276)Cited by:[§3\.4](https://arxiv.org/html/2609.25058#S3.SS4.p1.1)\.
- \[15\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§4\.1](https://arxiv.org/html/2609.25058#S4.SS1.p1.1)\.
- \[16\]Z\. Hu, L\. Wang, Y\. Lan, W\. Xu, E\. Lim, L\. Bing, X\. Xu, S\. Poria, and R\. Lee\(2023\)LLM\-adapters: an adapter family for parameter\-efficient fine\-tuning of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 5254–5276\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.319)Cited by:[§4\.1](https://arxiv.org/html/2609.25058#S4.SS1.p2.1)\.
- \[17\]C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova\(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1300)Cited by:[§4\.4](https://arxiv.org/html/2609.25058#S4.SS4.p2.1)\.
- \[18\]Y\. Bisk, R\. Zellers, R\. Le Bras, J\. Gao, and Y\. Choi\(2020\)PIQA: reasoning about physical commonsense in natural language\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 7432–7439\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6239)Cited by:[§4\.4](https://arxiv.org/html/2609.25058#S4.SS4.p2.1)\.
- \[19\]M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi\(2019\)Social IQa: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),Hong Kong, China,pp\. 4463–4473\.External Links:[Document](https://dx.doi.org/10.18653/v1/D19-1454)Cited by:[§4\.4](https://arxiv.org/html/2609.25058#S4.SS4.p2.1)\.
- \[20\]K\. Sakaguchi, R\. Le Bras, C\. Bhagavatula, and Y\. Choi\(2020\)WinoGrande: an adversarial winograd schema challenge at scale\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.34,pp\. 8732–8740\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v34i05.6399)Cited by:[§4\.4](https://arxiv.org/html/2609.25058#S4.SS4.p2.1)\.
- \[21\]P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord\(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.4](https://arxiv.org/html/2609.25058#S4.SS4.p3.1)\.
- \[22\]T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal\(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,Brussels, Belgium,pp\. 2381–2391\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[§4\.4](https://arxiv.org/html/2609.25058#S4.SS4.p3.1)\.
- \[23\]A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§4\.7](https://arxiv.org/html/2609.25058#S4.SS7.p2.1)\.
- \[24\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. Renard Lavaud, M\. Lachaux, P\. Stock, T\. Le Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. El Sayed\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§4\.7](https://arxiv.org/html/2609.25058#S4.SS7.p2.1)\.
- \[25\]M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan, N\. Bach, A\. Bahree, A\. Bakhtiari, J\. Bao, H\. Behl,et al\.\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.Cited by:[§4\.7](https://arxiv.org/html/2609.25058#S4.SS7.p2.1)\.
- \[26\]S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin, T\. Mihaylov, M\. Ott, S\. Shleifer, K\. Shuster, D\. Simig, P\. S\. Koura, A\. Sridhar, T\. Wang, and L\. Zettlemoyer\(2022\)OPT: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§4\.7](https://arxiv.org/html/2609.25058#S4.SS7.p2.1)\.
- \[27\]E\. Almazrouei, H\. Alobeidli, A\. Alshamsi, A\. Cappelli, R\. Cojocaru, M\. Debbah, É\. Goffinet, D\. Hesslow, J\. Launay, Q\. Malartic, D\. Mazzotta, B\. Noune, B\. Pannier, and G\. Penedo\(2023\)The falcon series of open language models\.arXiv preprint arXiv:2311\.16867\.Cited by:[§4\.7](https://arxiv.org/html/2609.25058#S4.SS7.p2.1)\.