CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA

# CARE-LoRA: Compressed Activation REconstruction for Memory-Efficient LoRA
Source: [https://arxiv.org/html/2607.11940](https://arxiv.org/html/2607.11940)
Gengyu Zhang, Haiyin Ran, Zhengbao He, Yuhang Liu, Hanling Tian, Zhehao Huang, and Xiaolin HuangThe authors are with the Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200240, China\. E\-mail: \{zhang2021jiaoda, rhy2006, lstefanie, yuhangliu, hanlingtian, kinght\_h,xiaolinhuang\}@sjtu\.edu\.cn\.\(Corresponding author: Xiaolin Huang\.\)

###### Abstract

As the scale of large pre\-trained models continues to grow, fine\-tuning them under limited memory budgets has become increasingly challenging\. Low\-Rank Adaptation \(LoRA\), currently one of the most widely adopted parameter\-efficient fine\-tuning \(PEFT\) methods, mitigates this challenge by optimizing only low\-rank adaptation matrices, thereby greatly reducing the number of trainable parameters\. With the parameter overhead substantially reduced, the activations retained for backpropagation have emerged as the primary remaining memory bottleneck during LoRA fine\-tuning\. To address this, we propose CARE\-LoRA, a data\-aware Compressed Activation REconstruction framework\. By exploiting the inherent projection structure of LoRA, CARE\-LoRA replaces the full input activation with the low\-rank compressed activation naturally produced by the LoRA branch\. It further computes a lightweight reconstruction matrix during the forward pass with negligible additional computation cost, which is used during backpropagation to reconstruct the gradient signal, thereby keeping LoRA matrices fully trainable\. Extensive experiments across diverse models and downstream tasks demonstrate that, while substantially reducing the overall memory footprint, CARE\-LoRA achieves competitive or even superior performance compared with standard LoRA and representative LoRA variants\. Our code is publicly available athttps://github\.com/fishandyu/CARE\-LoRA\.

###### Index Terms:

Low\-rank adaptation, parameter\-efficient fine\-tuning, activation compression, large language models\.

## 1Introduction

Pretraining followed by downstream fine\-tuning has become a standard recipe for building modern language systems\. However, full\-parameter fine\-tuning remains difficult under limited GPU memory, because it not only updates all pretrained weights, but also requires storing their gradients, optimizer states, and intermediate activations for backpropagation\. Parameter\-efficient fine\-tuning \(PEFT\) mitigates the parameter bottleneck by restricting optimization to a small number of trainable parameters\. In PEFT, Low\-Rank Adaptation \(LoRA\) has become a deployment\-friendly and widely used approach: it freezes the pretrained backbone, learns only low\-rank weight updates, and can merge the learned adaptation into the base model without extra inference latency\[[11](https://arxiv.org/html/2607.11940#bib.bib1)\]\.

In practice, although LoRA substantially reduces trainable parameters, their gradients, and optimizer states, it does not reduce the activation memory required by backpropagation, making activations a more prominent bottleneck\. This has motivated recent work on reducing the activations saved during LoRA\-style fine\-tuning\. HyC\-LoRA targets activations buffered for nonlinear operators in LoRA training and compresses them via outlier\-aware low\-bit quantization\[[29](https://arxiv.org/html/2607.11940#bib.bib2)\]\. LoRAct instead reduces activation memory by decomposing full activation matrices into compact low\-rank factors\[[23](https://arxiv.org/html/2607.11940#bib.bib3)\]\. Gradient checkpointing, a standard memory\-saving technique, reduces activation memory by storing only a subset of intermediate activations and recomputing the rest during backpropagation\[[7](https://arxiv.org/html/2607.11940#bib.bib4)\]\. While effective, these methods mainly reduce activation memory through generic quantization, decomposition, or recomputation, rather than exploiting the structural properties of LoRA factors\.

A representative approach that exploits this factorized structure is LoRA\-FA\[[31](https://arxiv.org/html/2607.11940#bib.bib5)\], which freezesAAand only updatesBB, thereby replacing stored full activations with compressed low\-rank activations\. Although LoRA\-FA largely reduces the activation memory, its performance is far from satisfactory\. The reason is that whenAAis fixed, the weight changeΔ​W=A​B\\Delta W=ABis restricted to the subspace determined by a random, non\-trained initialization matrix\. In other words, the adapter obtained by LoRA\-FA can only learn how to map features from a fixed low\-dimensional subspace, but the subspace itself cannot be adjusted as in standard LoRA\.

The challenge to overcome this limitation is that on the one hand we need to unfreezeAAto pursue adaptability and on the other hand we have to avoid calculating the exact gradient ofAAto keep the memory efficiency\. To achieve this goal, we propose a novel method to approximately reconstruct the backward information\. We name this methodCARE\-LoRA, standing for theCompressedActivationREconstruction framework for memory\-efficient low\-rank adaptation\. It stores the natural compressed activationZ=X​A∈ℝN×rZ=XA\\in\\mathbb\{R\}^\{N\\times r\}together with a lightweight*data\-aware*reconstruction matrixM∈ℝr×mM\\in\\mathbb\{R\}^\{r\\times m\}, wherer≪min⁡\{m,N\}r\\ll\\min\\\{m,N\\\}\. As a result, its memory overhead is marginally higher than LoRA\-FA, while recovering much of the performance of standard LoRA by keepingAAtrainable\.

We evaluate CARE\-LoRA across diverse fine\-tuning settings, including natural language understanding, mathematical reasoning, code generation, instruction following, and image generation\. Across these settings, our method reduces total peak memory by up to roughly20%20\\%relative to standard LoRA\. At the same rank, it substantially improves over LoRA\-FA, raising the average score by 4\.3 points on GLUE\[[27](https://arxiv.org/html/2607.11940#bib.bib6)\]and 7\.4 points on SuperGLUE\[[26](https://arxiv.org/html/2607.11940#bib.bib7)\]at less than1%1\\%additional memory\. Moreover, reducing memory consumption allows CARE\-LoRA to use a higher rank for stronger performance\. By increasing the rank, CARE\-LoRA further surpasses LoRA and representative LoRA variants such as DoRA\[[15](https://arxiv.org/html/2607.11940#bib.bib8)\]and PiSSA\[[18](https://arxiv.org/html/2607.11940#bib.bib9)\]\. Notably, CARE\-LoRA achieves the highest average performance on the Mistral\-7B\-v0\.3\[[12](https://arxiv.org/html/2607.11940#bib.bib10)\]fine\-tuning tasks while requiring only1\.02×1\.02\\timesthe per\-step training time of LoRA and still maintaining a lower memory footprint than LoRA\.

![Refer to caption](https://arxiv.org/html/2607.11940v1/x1.png)Figure 1:Overview of CARE\-LoRA\. CARE\-LoRA stores the compressed activationZZand a lightweight reconstruction matrixMMinstead of the full activationXX, enabling the gradient ofAAto be reconstructed while keeping both LoRA factors trainable\.The main contributions of this work are as follows:

- •We revisit the memory bottleneck of LoRA fine\-tuning and note that, once adapter\-side parameters are sufficiently reduced, activation storage becomes the dominant remaining obstacle\.
- •We propose CARE\-LoRA, a data\-aware framework that stores compressed activations and a lightweight reconstruction matrix, keeping both LoRA factors trainable under a near LoRA\-FA memory budget\. We further prove that its projection\-down subspace can evolve during fine\-tuning\.
- •We show empirically that CARE\-LoRA substantially outperforms LoRA\-FA at the same rank\. By reinvesting the saved memory into a higher rank, it further surpasses standard LoRA and representative variants, achieving a favorable balance of memory efficiency, speed, and accuracy across multiple tasks\.

## 2Related Work

### 2\.1Low\-Rank Adaptation

Low\-Rank Adaptation \(LoRA\) is a widely adopted parameter\-efficient fine\-tuning method that freezes the pretrained weightWWand learns a low\-rank updateΔ​W=A​B\\Delta W=AB, where the projection\-down matrixAAcompresses the input activationXXand the projection\-up matrixBBexpands it back, adding no inference cost once merged\[[11](https://arxiv.org/html/2607.11940#bib.bib1)\]\. Many variants improve LoRA from complementary perspectives\. On the parameter side, AdaLoRA adaptively allocates the rank budget across weight matrices\[[32](https://arxiv.org/html/2607.11940#bib.bib11)\], while VeRA and LoRA\-XS reduce trainable parameters further through shared frozen random projections\[[13](https://arxiv.org/html/2607.11940#bib.bib12)\]or a small trainable core between frozen SVD\-derived factors\[[2](https://arxiv.org/html/2607.11940#bib.bib13)\]\. On the optimization side, LoRA\-GA and PiSSA enhance adapter initialization to speed up convergence\[[28](https://arxiv.org/html/2607.11940#bib.bib14),[18](https://arxiv.org/html/2607.11940#bib.bib9)\], and DoRA decomposes each weight into magnitude and direction\[[15](https://arxiv.org/html/2607.11940#bib.bib8)\]\. QLoRA instead targets the frozen backbone, back\-propagating through a 4\-bit quantized model to cut base\-weight memory\[[9](https://arxiv.org/html/2607.11940#bib.bib15)\]\. Despite these advances, all of them still retain the full input activation for backpropagation, leaving the activation memory of fine\-tuning unaddressed\.

### 2\.2Memory\-Efficient Fine\-Tuning

During fine\-tuning, memory is consumed by the trainable parameters, their gradients, optimizer states, and the activations retained for backpropagation\. Once parameter\-efficient methods reduce the first three terms, activation becomes the dominant term and scales with batch size and sequence length\[[23](https://arxiv.org/html/2607.11940#bib.bib3)\]\. General\-purpose techniques target different parts of this budget, recomputing activations on the backward pass\[[7](https://arxiv.org/html/2607.11940#bib.bib4),[14](https://arxiv.org/html/2607.11940#bib.bib16)\], projecting gradients and optimizer states into a low\-rank subspace\[[33](https://arxiv.org/html/2607.11940#bib.bib17)\], or compressing stored activations through quantization or low\-rank projection\[[5](https://arxiv.org/html/2607.11940#bib.bib18),[22](https://arxiv.org/html/2607.11940#bib.bib19)\], yet none of them is designed for LoRA fine\-tuning\. Closer to our setting, LoRAct compresses the cached activation with an online low\-rank decomposition\[[23](https://arxiv.org/html/2607.11940#bib.bib3)\]\. HyC\-LoRA quantizes the activations buffered for nonlinear operators with an outlier\-aware scheme\[[29](https://arxiv.org/html/2607.11940#bib.bib2)\]\. Although both reduce the activation stored during LoRA fine\-tuning, they do not make good use of the factorized structure of LoRA\. LoRA\-FA instead exploits this, freezing the projection\-down matrix and training only the projection\-up matrix, so it does not need to store the full activation\[[31](https://arxiv.org/html/2607.11940#bib.bib5)\]\. However, this restricts the projection\-down subspace throughout fine\-tuning and thus degrades performance\. To address this limitation, we propose a method that uses the compressed activation naturally produced by LoRA to reconstruct the gradient for training\. Our method not only preserves the memory savings of LoRA\-FA but also retains the performance of standard LoRA, with only negligible computational overhead\.

## 3Method

Algorithm 1CARE\-LoRAForward Pass

Backward Pass

0:Input:activation

X∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}
0:Storage:frozenweight

W∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}, trainable matrices

A∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\}and

B∈ℝr×nB\\in\\mathbb\{R\}^\{r\\times n\}
0:Output:

Y∈ℝN×nY\\in\\mathbb\{R\}^\{N\\times n\}, matrices

Z,MZ,Mto be saved

1:

Z←X​AZ\\leftarrow XA
1:Prepare for backward reconstruction\.

2:

M←Z†​XM\\leftarrow Z^\{\\dagger\}X
2:

Z†Z^\{\\dagger\}implemented as

\(Z⊤​Z\+λ​I\)−1​Z⊤\(Z^\{\\top\}Z\+\\lambda I\)^\{\-1\}Z^\{\\top\}
3:

Δ​Y←Z​B\\Delta Y\\leftarrow ZB
4:

Y←X​W\+Δ​YY\\leftarrow XW\+\\Delta Y
5:Save

ZZand

MM, discard

XX
6:Continue forward propagation with

YY

0:Input:upstreamgradient

G=∂ℒ/∂YG=\\partial\\mathcal\{L\}/\\partial Y
0:Storage:compressedactivation

Z∈ℝN×rZ\\in\\mathbb\{R\}^\{N\\times r\}, reconstruction matrix

M∈ℝr×mM\\in\\mathbb\{R\}^\{r\\times m\}
0:Output:gradients

∇~A​ℒ,∇Bℒ,∇Xℒ\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\},\\nabla\_\{B\}\\mathcal\{L\},\\nabla\_\{X\}\\mathcal\{L\}
1:

∇Xℒ←G​\(W\+A​B\)⊤\\nabla\_\{X\}\\mathcal\{L\}\\leftarrow G\(W\+AB\)^\{\\top\}
2:

∇Bℒ←Z⊤​G\\nabla\_\{B\}\\mathcal\{L\}\\leftarrow Z^\{\\top\}G
2:Exact gradient

3:

X^←Z​M\\widehat\{X\}\\leftarrow ZM
3:Activation reconstruction

4:

∇~A​ℒ←X^⊤​G​B⊤\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}\\leftarrow\\widehat\{X\}^\{\\top\}GB^\{\\top\}
5:Update

AAand

BBwith optimizer

6:Continue backpropagation with

∇Xℒ\\nabla\_\{X\}\\mathcal\{L\}

### 3\.1Preliminaries

Consider a linear layer with a frozen pretrained weightW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}\. LetX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}be the input activation, whereNNdenotes the number of tokens in the current batch, e\.g\., batch size times sequence length\. LoRA introduces two trainable low\-rank matricesA∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\}andB∈ℝr×nB\\in\\mathbb\{R\}^\{r\\times n\}, withr≪min⁡\(m,n,N\)r\\ll\\min\(m,n,N\), and the adapted linear transformation is

Y=X​\(W\+A​B\)\.Y=X\(W\+AB\)\.\(1\)Following common practice, we omit the LoRA scaling factors=α/rs=\\alpha/rfor simplicity, since it can be absorbed into the low\-rank factorsAAandBBwithout affecting the following derivation\.

Denote the upstream gradient by

G=∂ℒ∂Y∈ℝN×n\.G=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial Y\}\\in\\mathbb\{R\}^\{N\\times n\}\.\(2\)The exact LoRA gradients are

∇Bℒ\\displaystyle\\nabla\_\{B\}\\mathcal\{L\}=A⊤​X⊤​G=\(X​A\)⊤​G,\\displaystyle=A^\{\\top\}X^\{\\top\}G=\(XA\)^\{\\top\}G,\(3\)∇Aℒ\\displaystyle\\nabla\_\{A\}\\mathcal\{L\}=X⊤​G​B⊤,\\displaystyle=X^\{\\top\}GB^\{\\top\},\(4\)∇Xℒ\\displaystyle\\nabla\_\{X\}\\mathcal\{L\}=G​\(W\+A​B\)⊤\.\\displaystyle=G\(W\+AB\)^\{\\top\}\.\(5\)Eq\. \([3](https://arxiv.org/html/2607.11940#S3.E3)\) shows that the gradient ofBBonly requires an intermediate matrix

Z=X​A∈ℝN×r\.Z=XA\\in\\mathbb\{R\}^\{N\\times r\}\.\(6\)SinceZZis produced by the projection\-down matrixAAand reduces the activation dimension frommmtorr, it naturally serves as a compressed representation of the full activationXXunder the LoRA parameterization\.

In contrast, Eq\. \([4](https://arxiv.org/html/2607.11940#S3.E4)\) shows that the gradient ofAAstill depends on the full activationX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}\. Therefore, standard LoRA must retainXXduring the forward pass if bothAAandBBare trainable\. LoRA\-FA avoids this activation dependency by freezingAAand updating onlyBB\. In this case, the backward pass only needs to storeZ=X​AZ=XAinstead of the full activationXX\. However, the projection\-down matrixAAthen remains fixed throughout fine\-tuning\. CARE\-LoRA keeps the same compressed activationZZ, but reconstructs the missing activation signal needed to updateAA\.

### 3\.2Compressed Activation REconstruction

CARE\-LoRA aims to update both LoRA matrices while retaining the compressed activationZZinstead of the full activationXX\. The key challenge is computing the gradient ofAA\. Since∇Aℒ=X⊤​G​B⊤\\nabla\_\{A\}\\mathcal\{L\}=X^\{\\top\}GB^\{\\top\}, directly reconstructing this gradient fromZZwould require simultaneous access to the forward activationXXand the backward gradientGG, which is incompatible with our activation\-saving setting\. We therefore reconstructXXinstead and use the reconstructed activation for subsequent gradient computation ofAA\.

SinceZ=X​AZ=XAis already computed in the LoRA branch, we seek a linear reconstruction matrix

M∈ℝr×mM\\in\\mathbb\{R\}^\{r\\times m\}such that the reconstructed activation

approximates the original activationXX\. For a fixed batch activationXXand current LoRA projection\-down matrixAA, we defineM⋆M^\{\\star\}by the least\-squares problem

M⋆\\displaystyle M^\{\\star\}=arg⁡minM⁡‖X−X​A​M‖F2\\displaystyle=\\arg\\min\_\{M\}\\\|X\-XAM\\\|\_\{F\}^\{2\}\(8\)=arg⁡minM⁡‖X−Z​M‖F2\.\\displaystyle=\\arg\\min\_\{M\}\\\|X\-ZM\\\|\_\{F\}^\{2\}\.WhenZZhas full column rank, the closed\-form solution is

M⋆=\(Z⊤​Z\)−1​Z⊤​X=Z†​X\.M^\{\\star\}=\(Z^\{\\top\}Z\)^\{\-1\}Z^\{\\top\}X=Z^\{\\dagger\}X\.\(9\)Thus, under the Frobenius reconstruction loss, CARE\-LoRA uses the current batch itself to compute the best linear decoder from the compressed activationZZback toXX\.

In practice, directly invertingZ⊤​ZZ^\{\\top\}Zis numerically unstable, sinceZZcould be rank\-deficient or ill\-conditioned\. We therefore use a Tikhonov\-regularized version\[[24](https://arxiv.org/html/2607.11940#bib.bib20)\]

Mλ⋆=\(Z⊤​Z\+λ​Ir\)−1​Z⊤​X,λ\>0\.M^\{\\star\}\_\{\\lambda\}=\(Z^\{\\top\}Z\+\\lambda I\_\{r\}\)^\{\-1\}Z^\{\\top\}X,\\qquad\\lambda\>0\.\(10\)The regularizer makes the reconstruction numerically stable and always well\-defined, becauseZ⊤​Z\+λ​IrZ^\{\\top\}Z\+\\lambda I\_\{r\}is positive definite for anyλ\>0\\lambda\>0\. In our implementation,λ\\lambdais set to a very small value, serving only as a numerical safeguard with negligible effect on the least\-squares reconstruction\. For simplicity, we writeMMforMλ⋆M^\{\\star\}\_\{\\lambda\}below\.

During the forward pass, CARE\-LoRA follows standard LoRA, and the difference lies in what is kept for the backward pass\. WhileXXis still available during the forward computation, CARE\-LoRA uses it once to compute the batch\-dependent reconstruction matrixMMas in Eq\. \([10](https://arxiv.org/html/2607.11940#S3.E10)\), then saves only

Z∈ℝN×randM∈ℝr×mZ\\in\\mathbb\{R\}^\{N\\times r\}\\quad\\text\{and\}\\quad M\\in\\mathbb\{R\}^\{r\\times m\}for the LoRA backward pass, and discards the full activationX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}\. Therefore, the saved LoRA activation\-related tensors scale asO​\(N​r\+m​r\)O\(Nr\+mr\)instead ofO​\(N​m\)O\(Nm\)\. Sincer≪min⁡\(N,m\)r\\ll\\min\(N,m\), this is much smaller than storingXX\.

During the backward pass, given the upstream gradientG=∂ℒ/∂YG=\\partial\\mathcal\{L\}/\\partial Y, CARE\-LoRA computes the gradient ofBBexactly using the stored matrixZZ,

∇Bℒ=\(X​A\)⊤​G=Z⊤​G,\\nabla\_\{B\}\\mathcal\{L\}=\(XA\)^\{\\top\}G=Z^\{\\top\}G,\(11\)which is identical to Eq\. \([3](https://arxiv.org/html/2607.11940#S3.E3)\)\. The input gradient∇Xℒ\\nabla\_\{X\}\\mathcal\{L\}is also unchanged from standard LoRA and is computed as in Eq\. \([5](https://arxiv.org/html/2607.11940#S3.E5)\)\. Therefore, CARE\-LoRA does not introduce any reconstruction error into the gradient propagated to previous layers\.

The only approximated term is the gradient ofAA\. Since Eq\. \([4](https://arxiv.org/html/2607.11940#S3.E4)\) requires the discarded activationXX, CARE\-LoRA uses the reconstructed activationX^=Z​M\\widehat\{X\}=ZMin its place:

∇~A​ℒ=X^⊤​G​B⊤=M⊤​\(Z⊤​\(G​B⊤\)\)\.\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=\\widehat\{X\}^\{\\top\}GB^\{\\top\}=M^\{\\top\}\\bigl\(Z^\{\\top\}\(GB^\{\\top\}\)\\bigr\)\.\(12\)In our implementation, we compute the last expression in Eq\. \([12](https://arxiv.org/html/2607.11940#S3.E12)\) directly, without explicitly constructing the large matrixX^=Z​M∈ℝN×m\\widehat\{X\}=ZM\\in\\mathbb\{R\}^\{N\\times m\}\. This avoids forming a full\-size reconstructed activation and further reduces peak memory\.

Thus, CARE\-LoRA reduces activation memory by storingZZandMMinstead of the full activationXX, while the only approximation introduced during training is the local gradient used to updateAA\. The update ofBBand the inter\-layer gradient propagation remain exact as in standard LoRA\. Fig\.[1](https://arxiv.org/html/2607.11940#S1.F1)illustrates the forward and backward computation, and Algorithm[1](https://arxiv.org/html/2607.11940#alg1)formalizes it in detail\.

### 3\.3Adaptation of the Projection\-Down Subspace

The main limitation of LoRA\-FA is not only that one of the two LoRA matrices is frozen, but that the update subspace induced by the projection\-down matrixAAremains fixed throughout fine\-tuning\. In a LoRA layer withΔ​W=A​B\\Delta W=AB, every column of the low\-rank update satisfies

Δ​W:,j=A​B:,j,\\Delta W\_\{:,j\}=AB\_\{:,j\},\(13\)and therefore lies inCol⁡\(A\)\\operatorname\{Col\}\(A\)\. Consequently, once LoRA\-FA freezesAA, the initialization ofAAdetermines a fixedrr\-dimensional subspace in which all subsequent updates toWWmust reside, while trainingBBcan only adjust the coordinates ofΔ​W\\Delta Wwithin this subspace\.

SinceA∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\}withr≪mr\\ll m, genericallyAAhas full column rankrr, so its row space is already the entireℝr\\mathbb\{R\}^\{r\}\. Therefore, the expressive bottleneck lies in the column spaceCol⁡\(A\)\\operatorname\{Col\}\(A\), anrr\-dimensional subspace of the much larger input feature spaceℝm\\mathbb\{R\}^\{m\}, determining which input directions are retained by the compressed activationZ=X​AZ=XA\. To understand whether CARE\-LoRA truly keeps the projection\-down matrix adaptive, we analyze the evolution ofCol⁡\(A\)\\operatorname\{Col\}\(A\)below\.

For the following analysis, we let

U=G​B⊤∈ℝN×r,ΣX=X⊤​X∈ℝm×m\.U=GB^\{\\top\}\\in\\mathbb\{R\}^\{N\\times r\},\\qquad\\Sigma\_\{X\}=X^\{\\top\}X\\in\\mathbb\{R\}^\{m\\times m\}\.HereUUis the upstream gradient after being propagated throughB⊤B^\{\\top\}, andΣX\\Sigma\_\{X\}is the unnormalized second\-moment matrix ofXX, or equivalently the unnormalized batch covariance when the activations are centered\.

Using the regularized reconstruction matrix in Eq\. \([10](https://arxiv.org/html/2607.11940#S3.E10)\),∇~A​ℒ\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}in Eq\. \([12](https://arxiv.org/html/2607.11940#S3.E12)\) can be written as

∇~A​ℒ\\displaystyle\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=M⊤​Z⊤​G​B⊤\\displaystyle=M^\{\\top\}Z^\{\\top\}GB^\{\\top\}=X⊤​Z​\(Z⊤​Z\+λ​Ir\)−1​Z⊤​U\\displaystyle=X^\{\\top\}Z\(Z^\{\\top\}Z\+\\lambda I\_\{r\}\)^\{\-1\}Z^\{\\top\}U=X⊤​X​A​\(A⊤​X⊤​X​A\+λ​Ir\)−1​A⊤​X⊤​U\\displaystyle=X^\{\\top\}XA\(A^\{\\top\}X^\{\\top\}XA\+\\lambda I\_\{r\}\)^\{\-1\}A^\{\\top\}X^\{\\top\}U=ΣX​A​Kλ,\\displaystyle=\\Sigma\_\{X\}AK\_\{\\lambda\},\(14\)where

Kλ=\(A⊤​ΣX​A\+λ​Ir\)−1​A⊤​X⊤​U∈ℝr×r\.K\_\{\\lambda\}=\(A^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}\)^\{\-1\}A^\{\\top\}X^\{\\top\}U\\in\\mathbb\{R\}^\{r\\times r\}\.\(15\)
Eq\. \([14](https://arxiv.org/html/2607.11940#S3.E14)\) exposes the source of subspace adaptation\. The factorΣX​A\\Sigma\_\{X\}Afirst maps the current projection directions through the second\-moment matrix, whileKλK\_\{\\lambda\}only forms linear combinations of the resultingrrdirections\. Therefore, CARE\-LoRA can introduce directions outside the current column space ofAAwhenΣX\\Sigma\_\{X\}maps the current subspace to directions not already contained in it, as long as the backward signal does not cancel these off\-subspace components\.

To clearly state this intuition, we letS=Col⁡\(A\)S=\\operatorname\{Col\}\(A\)be the current projection\-down subspace, and let

PS∈ℝm×mandPS⟂=Im−PSP\_\{S\}\\in\\mathbb\{R\}^\{m\\times m\}\\qquad\\text\{and\}\\qquad P\_\{S\}^\{\\perp\}=I\_\{m\}\-P\_\{S\}denote the orthogonal projectors ontoSSand its complement, respectively\. Then consider a gradient step onAAusing the reconstructed gradient with learning rateη\>0\\eta\>0,

A\+=A−η​∇~A​ℒ=A−η​ΣX​A​Kλ,A^\{\+\}=A\-\\eta\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=A\-\\eta\\Sigma\_\{X\}AK\_\{\\lambda\},\(16\)whereA\+A^\{\+\}denotes the updated projection\-down matrix\. SincePS⟂​A=0P\_\{S\}^\{\\perp\}A=0, we have

PS⟂​A\+=−η​PS⟂​ΣX​A​Kλ\.P\_\{S\}^\{\\perp\}A^\{\+\}=\-\\eta P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}AK\_\{\\lambda\}\.\(17\)Hence, if

PS⟂​ΣX​A​Kλ≠0,P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}AK\_\{\\lambda\}\\neq 0,\(18\)thenA\+A^\{\+\}has at least one column with a nonzero component outsideSS\. Therefore,

Col⁡\(A\+\)⊈Col⁡\(A\),\\operatorname\{Col\}\(A^\{\+\}\)\\nsubseteq\\operatorname\{Col\}\(A\),\(19\)which implies that the column space ofAAhas changed\. IfA\+A^\{\+\}remains full column rank, then this change is a movement of the learnedrr\-dimensional projection\-down subspace\.

The condition in Eq\. \([18](https://arxiv.org/html/2607.11940#S3.E18)\) separates into a data\-dependent part and a gradient\-dependent part\. The data\-dependent part is

PS⟂​ΣX​A≠0\.P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A\\neq 0\.\(20\)This condition fails if and only if

Col⁡\(ΣX​A\)⊆Col⁡\(A\),\\operatorname\{Col\}\(\\Sigma\_\{X\}A\)\\subseteq\\operatorname\{Col\}\(A\),\(21\)or equivalently, whenCol⁡\(A\)\\operatorname\{Col\}\(A\)is an invariant subspace ofΣX\\Sigma\_\{X\}\. However, sinceΣX=X⊤​X\\Sigma\_\{X\}=X^\{\\top\}Xis induced by the heterogeneous input data, a randomly initializedrr\-dimensional subspaceCol⁡\(A\)\\operatorname\{Col\}\(A\)will not in general be invariant underΣX\\Sigma\_\{X\}, except for some special cases such asΣX=c​Im\\Sigma\_\{X\}=cI\_\{m\}\.

The gradient\-dependent part concerns whether the right factorKλK\_\{\\lambda\}cancels the off\-subspace component produced byΣX​A\\Sigma\_\{X\}A\. Let

D=PS⟂​ΣX​A\.D=P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A\.\(22\)When the data\-dependent condition holds, we haveD≠0D\\neq 0\. The outside component disappears only in the degenerate case

D​Kλ=0\.DK\_\{\\lambda\}=0\.\(23\)This requiresKλK\_\{\\lambda\}to be rank\-deficient\. Otherwise, ifKλK\_\{\\lambda\}were nonsingular, right multiplication byKλK\_\{\\lambda\}could not turn a nonzero matrixDDinto zero\. Since

Kλ=\(A⊤​ΣX​A\+λ​Ir\)−1​Z⊤​U,K\_\{\\lambda\}=\(A^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}\)^\{\-1\}Z^\{\\top\}U,\(24\)whereA⊤​ΣX​A\+λ​IrA^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}is invertible forλ\>0\\lambda\>0, this can happen only whenZ⊤​UZ^\{\\top\}Uis rank\-deficient\.

The rank deficiency ofZ⊤​UZ^\{\\top\}Ucan arise in only three cases\. First,ZZis rank\-deficient, i\.e\.,rank⁡\(Z\)<r\\operatorname\{rank\}\(Z\)<r\. Second,UUis rank\-deficient, i\.e\.,rank⁡\(U\)<r\\operatorname\{rank\}\(U\)<r\. These two cases are already degenerate: bothZZandUUare matrices inℝN×r\\mathbb\{R\}^\{N\\times r\}, and in typical fine\-tuning batches we haveN≫rN\\gg r, so they are expected to have full column rank under non\-degenerate activations and backward signals\.

The remaining case is more subtle: bothZZandUUhave rankrr, butZ⊤​UZ^\{\\top\}Uis still singular\. This is a nongeneric event\. In particular, ifZZandUUare viewed as generic full\-column\-rank matrices inℝN×r\\mathbb\{R\}^\{N\\times r\}, then

Pr⁡\(rank⁡\(Z⊤​U\)​<r\|​rank⁡\(Z\)=rank⁡\(U\)=r\)=0\.\\Pr\\\!\\left\(\\operatorname\{rank\}\(Z^\{\\top\}U\)<r\\,\\middle\|\\,\\operatorname\{rank\}\(Z\)=\\operatorname\{rank\}\(U\)=r\\right\)=0\.\(25\)Combining the three cases above, under the generic non\-degenerate setting,Z⊤​UZ^\{\\top\}Uis not rank\-deficient\. Therefore, when the data\-dependent conditionD≠0D\\neq 0holds, we generically haveD​Kλ≠0DK\_\{\\lambda\}\\neq 0, so the gradient\-dependent part retains an off\-subspace component in the update ofAA\.

In summary, CARE\-LoRA avoids the fixed\-subspace limitation of LoRA\-FA\. The reconstructed gradient∇~A​ℒ=ΣX​A​Kλ\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=\\Sigma\_\{X\}AK\_\{\\lambda\}uses the input second\-moment matrix to generate data\-dependent update directions forAA\. Under non\-degenerate activations and backward signals, these directions are not restricted toCol⁡\(A\)\\operatorname\{Col\}\(A\), allowing the projection\-down subspace to evolve while keeping the memory benefit of avoiding storage of the full activationXX\.

### 3\.4Memory and Computation Costs

We next analyze the memory and computation cost introduced by CARE\-LoRA\. The analysis focuses on the tensors and matrix multiplications specific to the LoRA branch of one adapted linear layer\. Computation and storage shared by standard LoRA and CARE\-LoRA, including the frozen\-weight path of the adapted linear layer and intermediate tensors required by non\-LoRA operations, are excluded from this local comparison\.

Memory Cost\.For a LoRA layer with input activationX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}, projection\-down matrixA∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\}, and projection\-up matrixB∈ℝr×nB\\in\\mathbb\{R\}^\{r\\times n\}, standard LoRA must retain the full activationXXin order to compute∇Aℒ=X⊤​G​B⊤\\nabla\_\{A\}\\mathcal\{L\}=X^\{\\top\}GB^\{\\top\}\. Thus, the LoRA\-side saved activation tensor scales as

ℳLoRAact=N​m\.\\mathcal\{M\}\_\{\\mathrm\{LoRA\}\}^\{\\mathrm\{act\}\}=Nm\.\(26\)CARE\-LoRA keeps bothAAandBBtrainable\. It saves the compressed activationZ∈ℝN×rZ\\in\\mathbb\{R\}^\{N\\times r\}and the reconstruction matrixM∈ℝr×mM\\in\\mathbb\{R\}^\{r\\times m\}, but discards the full activationXX\. Therefore,

ℳCAREact=N​r\+r​m=r​\(N\+m\)\.\\mathcal\{M\}\_\{\\mathrm\{CARE\}\}^\{\\mathrm\{act\}\}=Nr\+rm=r\(N\+m\)\.\(27\)The activation\-memory ratio to standard LoRA is

ℳCAREactℳLoRAact=r​\(N\+m\)N​m\.\\frac\{\\mathcal\{M\}\_\{\\mathrm\{CARE\}\}^\{\\mathrm\{act\}\}\}\{\\mathcal\{M\}\_\{\\mathrm\{LoRA\}\}^\{\\mathrm\{act\}\}\}=\\frac\{r\(N\+m\)\}\{Nm\}\.\(28\)Therefore, CARE\-LoRA substantially reduces the LoRA\-side activation storage when the rankr≪min⁡\(m,N\)r\\ll\\min\(m,N\)\.

As a concrete example, consider fine\-tuning T5\-Base on MNLI with sequence length256256and batch size3232, soN=8192N=8192\. For a typical hidden\-size linear layer withm=768m=768and rankr=8r=8, the LoRA\-side activation\-memory ratio of CARE\-LoRA to standard LoRA is

8​\(8192\+768\)8192×768≈1\.14%\.\\frac\{8\(8192\+768\)\}\{8192\\times 768\}\\approx 1\.14\\%\.
Computation Cost\.We count the LoRA\-branch matrix\-multiplication cost for one adapted linear layer up to constant factors, including both the forward and backward passes\. In standard LoRA, the LoRA branch involves three multiplications with costN​m​rNmrand three multiplications with costN​r​nNrn, giving

TLoRA=3​N​m​r\+3​N​r​n=3​N​r​\(m\+n\)\.T\_\{\\mathrm\{LoRA\}\}=3Nmr\+3Nrn=3Nr\(m\+n\)\.\(29\)The additional cost of CARE\-LoRA mainly comes from solving the reconstruction matrixMMand using it to compute the reconstructedAA\-gradient, giving

Textra=2​N​r2\+2​m​r2\+r3\.T\_\{\\mathrm\{extra\}\}=2Nr^\{2\}\+2mr^\{2\}\+r^\{3\}\.\(30\)Thus,

TCARE=TLoRA\+Textra\.T\_\{\\mathrm\{CARE\}\}=T\_\{\\mathrm\{LoRA\}\}\+T\_\{\\mathrm\{extra\}\}\.\(31\)The computation cost ratio to standard LoRA is

TCARETLoRA=1\+r​\(2​N\+2​m\+r\)3​N​\(m\+n\)\.\\frac\{T\_\{\\mathrm\{CARE\}\}\}\{T\_\{\\mathrm\{LoRA\}\}\}=1\+\\frac\{r\(2N\+2m\+r\)\}\{3N\(m\+n\)\}\.\(32\)Sincer≪min⁡\(m,n,N\)r\\ll\\min\(m,n,N\), the additional reconstruction terms do not change the dominant order of LoRA\-branch matrix multiplications\. Therefore, CARE\-LoRA introduces only a small computation overhead compared with standard LoRA\.

For the same T5\-Base setting, takingm=n=768m=n=768andr=8r=8as an example, the additional LoRA\-branch computation cost of CARE\-LoRA over standard LoRA is

8​\(2×8192\+2×768\+8\)3×8192​\(768\+768\)≈0\.38%\.\\frac\{8\(2\\times 8192\+2\\times 768\+8\)\}\{3\\times 8192\(768\+768\)\}\\approx 0\.38\\%\.
TABLE I:Results \(%\) and Memory Usage \(MiB\) on Fine\-Tuning the T5\-Base Model on a Subset of GLUE Datasets\.Avg\. MemDenotes the Average Peak CUDA Memory Usage Across Different Tasks\.TABLE II:Results \(%\) and Memory Usage \(MiB\) on Fine\-Tuning the T5\-Base Model on a Subset of SuperGLUE Datasets\.Avg\. MemDenotes the Average Peak CUDA Memory Usage Across Different Tasks\.

## 4Experiments

We evaluate CARE\-LoRA on three categories of fine\-tuning tasks\. Fornatural language understanding\(NLU\), we fine\-tune T5\-Base\[[20](https://arxiv.org/html/2607.11940#bib.bib21)\]on GLUE\[[27](https://arxiv.org/html/2607.11940#bib.bib6)\]and SuperGLUE\[[26](https://arxiv.org/html/2607.11940#bib.bib7)\]\. Fornatural language generation\(NLG\), we fine\-tune Mistral\-7B\-v0\.3\[[12](https://arxiv.org/html/2607.11940#bib.bib10)\]for mathematical reasoning, code generation, and instruction following\. Fordiffusion\-based image generation, we fine\-tune SD3\-Medium\[[10](https://arxiv.org/html/2607.11940#bib.bib22)\]on DreamBooth\[[21](https://arxiv.org/html/2607.11940#bib.bib23)\]\. Together, these tasks cover diverse domains, model architectures, and sequence lengths ranging from256256to20482048tokens, providing a comprehensive assessment of CARE\-LoRA across a broad range of fine\-tuning scenarios\.

### 4\.1Baselines

We mainly compare the following methods:

- •LoRA\[[11](https://arxiv.org/html/2607.11940#bib.bib1)\]applies low\-rank adaptation to all linear layers of the frozen backbone\.
- •LoRAct\[[23](https://arxiv.org/html/2607.11940#bib.bib3)\]compresses cached activations via low\-rank decomposition\. We apply it only to LoRA layers to match CARE\-LoRA under a comparable memory budget\.
- •LoRA\-FA\[[31](https://arxiv.org/html/2607.11940#bib.bib5)\]freezesAAand updates onlyBBto avoid storing full activations\.
- •CARE\-LoRA\(ours\) keeps bothAAandBBtrainable while avoiding storing full activations\.

For a fair comparison, all methods use the same backbone, training data, target modules, optimizer \(AdamW\[[16](https://arxiv.org/html/2607.11940#bib.bib24)\]\), learning rate, precision, batch size, and training steps within each task, implemented in a unified PEFT\-based codebase\[[17](https://arxiv.org/html/2607.11940#bib.bib25)\]\. In the LLM fine\-tuning experiments, we additionally report CARE\-LoRA atr=16r=16, which is still more memory\-efficient than LoRA and LoRAct atr=8r=8\. The only method\-specific hyperparameter is decomposition rankkkof LoRAct, set to6464for T5\-Base and128128for Mistral\-7B\-v0\.3\. All results are averaged over three common seeds \(0,2121,4242\) with standard deviation reported\. The best result in each column is boldfaced and the second\-best is underlined\. Additional analyses are presented later in this section\.

TABLE III:Results \(%\) and Memory Usage \(MiB\) on Fine\-Tuning the Mistral\-7B\-v0\.3 Model on Three LLM Fine\-Tuning Tasks\.Avg\. MemDenotes the Average Peak CUDA Memory Usage Across Different Tasks\.MethodMathCodeInstructOverallGSM8KHumanEvalPrompt\-StrictInstruction\-StrictAvg\.Avg\. MemLoRA \(r=8r=8\)71\.22±0\.7071\.22\_\{\\pm 0\.70\}42\.89±2\.5442\.89\_\{\\pm 2\.54\}45\.22±1\.2345\.22\_\{\\pm 1\.23\}55\.64±0\.4255\.64\_\{\\pm 0\.42\}53\.74±0\.5053\.74\_\{\\pm 0\.50\}29214\.2729214\.27PiSSA \(r=8r=8\)71\.44¯±0\.19\\underline\{71\.44\}\_\{\\pm 0\.19\}41\.46±2\.4441\.46\_\{\\pm 2\.44\}45\.04±0\.4745\.04\_\{\\pm 0\.47\}55\.92±0\.7055\.92\_\{\\pm 0\.70\}53\.47±0\.7653\.47\_\{\\pm 0\.76\}29208\.5429208\.54DoRA \(r=8r=8\)70\.99±0\.4670\.99\_\{\\pm 0\.46\}43\.90±2\.79\\mathbf\{43\.90\}\_\{\\pm 2\.79\}46\.03¯±0\.92\\underline\{46\.03\}\_\{\\pm 0\.92\}56\.28¯±0\.69\\underline\{56\.28\}\_\{\\pm 0\.69\}54\.30¯±1\.13\\underline\{54\.30\}\_\{\\pm 1\.13\}30701\.6530701\.65LoRAct \(r=8r=8\)70\.91±0\.4370\.91\_\{\\pm 0\.43\}43\.50¯±0\.70\\underline\{43\.50\}\_\{\\pm 0\.70\}44\.24±1\.0544\.24\_\{\\pm 1\.05\}55\.00±0\.5755\.00\_\{\\pm 0\.57\}53\.41±0\.5153\.41\_\{\\pm 0\.51\}25185\.5125185\.51LoRA\-FA \(r=8r=8\)67\.65±0\.5767\.65\_\{\\pm 0\.57\}40\.85±0\.6140\.85\_\{\\pm 0\.61\}39\.49±2\.0339\.49\_\{\\pm 2\.03\}51\.48±2\.0051\.48\_\{\\pm 2\.00\}49\.87±0\.7649\.87\_\{\\pm 0\.76\}24214\.74\\mathbf\{24214\.74\}CARE\-LoRA \(r=8r=8\)70\.68±0\.1270\.68\_\{\\pm 0\.12\}42\.68±2\.2042\.68\_\{\\pm 2\.20\}44\.49±0\.2844\.49\_\{\\pm 0\.28\}54\.64±0\.1454\.64\_\{\\pm 0\.14\}53\.12±0\.4653\.12\_\{\\pm 0\.46\}24469\.27¯\\underline\{24469\.27\}CARE\-LoRA \(r=16r=16\)71\.82±0\.23\\mathbf\{71\.82\}\_\{\\pm 0\.23\}43\.29±1\.6143\.29\_\{\\pm 1\.61\}46\.27±0\.77\\mathbf\{46\.27\}\_\{\\pm 0\.77\}56\.47±0\.52\\mathbf\{56\.47\}\_\{\\pm 0\.52\}54\.47±0\.20\\mathbf\{54\.47\}\_\{\\pm 0\.20\}24948\.6524948\.65

TABLE IV:Results \(%\) and Memory Usage \(MiB\) on Fine\-Tuning Diffusion Model SD3\-Medium on DreamBooth, Averaged Over Ten Subjects and Three Seeds\.Avg\. MemDenotes the Average Peak CUDA Memory Usage\.
### 4\.2Results on Natural Language Understanding

Setting\.We fine\-tune T5\-Base on five widely used tasks from each of GLUE \(MNLI, QNLI, SST\-2, CoLA, MRPC\) and SuperGLUE \(BoolQ, CB, COPA, RTE, WiC\), following the original text\-to\-text formulation\. We report each task’s standard metric, accuracy for most tasks, Matthews correlation for CoLA, and F1 for MRPC and CB\. Following the protocol described in Sec\.[4\.1](https://arxiv.org/html/2607.11940#S4.SS1), all methods use rankr=8r=8\. Results are summarized in Tables[I](https://arxiv.org/html/2607.11940#S3.T1)and[II](https://arxiv.org/html/2607.11940#S3.T2)\.

Results\.At the same rank, CARE\-LoRA achieves strong performance while keeping a memory footprint close to LoRA\-FA\. On GLUE, CARE\-LoRA obtains an average score of 85\.22, essentially matching standard LoRA at 85\.24 and outperforming LoRA\-FA by 4\.25 points\. On SuperGLUE, CARE\-LoRA achieves the best average score of 78\.18, improving over LoRA and LoRAct while surpassing LoRA\-FA by 7\.37 points\. In terms of memory, CARE\-LoRA reduces average peak memory by about20%20\\%on both GLUE and SuperGLUE relative to LoRA, while requiring less than1%1\\%additional memory compared with LoRA\-FA\. These results show that the reconstructed update ofAAeffectively mitigates the fixed\-subspace limitation of LoRA\-FA without giving up its activation\-memory advantage\.

### 4\.3Results on Large Language Models

Setting\.We evaluate CARE\-LoRA on Mistral\-7B\-v0\.3 across three tasks: mathematical reasoning, code generation, and instruction following\. Each task focuses on a specific capability and uses well\-established datasets and metrics for training and evaluation, as detailed below:

- •Mathematical reasoning\.We fine\-tune on a 100k subset of MetaMathQA\[[30](https://arxiv.org/html/2607.11940#bib.bib26)\]with sequence length under512512tokens and report accuracy on GSM8K\[[8](https://arxiv.org/html/2607.11940#bib.bib27)\]\.
- •Code generation\.We fine\-tune on a 100k subset of OpenCodeInstruct\[[1](https://arxiv.org/html/2607.11940#bib.bib28)\]with sequence length under10241024tokens and report pass@1 on HumanEval\[[6](https://arxiv.org/html/2607.11940#bib.bib29)\]\.
- •Instruction following\.We fine\-tune on a 50k subset of SmolTalk\[[3](https://arxiv.org/html/2607.11940#bib.bib30)\]with sequence length under20482048tokens and report prompt\-level and instruction\-level strict accuracy on IFEval\[[34](https://arxiv.org/html/2607.11940#bib.bib31)\]\.

Here, we additionally compare against two widely used LoRA variants that target performance improvement rather than memory efficiency\. DoRA\[[15](https://arxiv.org/html/2607.11940#bib.bib8)\]decomposes pretrained weights into magnitude and direction, and we use its memory\-saving version in our experiments\. PiSSA\[[18](https://arxiv.org/html/2607.11940#bib.bib9)\]initializes the LoRA factors from the principal components of the pretrained weight via SVD\. Since CARE\-LoRA reduces activation memory, the saved memory can be reinvested into a higher rank\. We therefore additionally evaluate CARE\-LoRA atr=16r=16as a performance\-oriented setting for comparison with other baselines\.

Results\.Table[III](https://arxiv.org/html/2607.11940#S4.T3)reports results on Mistral\-7B\-v0\.3\. Atr=8r=8, CARE\-LoRA achieves competitive performance compared with standard LoRA while substantially reducing peak memory\. Atr=16r=16, CARE\-LoRA achieves the highest overall average of 54\.47 and leads all methods on GSM8K, Prompt\-Strict, and Instruction\-Strict, while also outperforming LoRA on HumanEval\. DoRA achieves the second\-best overall average, yet CARE\-LoRA requires 18\.7% less peak memory\. LoRA\-FA attains the lowest memory footprint by freezingAAthroughout fine\-tuning, but CARE\-LoRA uses only 3% more memory while improving the overall average by 4\.6 points\. Excluding LoRA\-FA, CARE\-LoRA atr=16r=16achieves the best accuracy while using less peak memory than every baseline\.

### 4\.4Results on Diffusion Models

Setting\.We fine\-tune SD3\-Medium on DreamBooth personalization\. We select ten subjects fixed prior to training \(dog, dog2, cat, backpack, bear plushie, can, candle, colorful sneaker, duck toy, and teapot\), covering both live subjects and a diverse range of common object categories, so that conclusions are not tied to a single subject type\. A separate adapter is trained per subject, and personalization quality is measured by DINO\[[4](https://arxiv.org/html/2607.11940#bib.bib32)\]and CLIP\-I\[[19](https://arxiv.org/html/2607.11940#bib.bib33)\]for subject fidelity and CLIP\-T for text alignment, averaged across subjects and seeds\. Following the higher\-rank setting studied above, we compare standard LoRA atr=8r=8with CARE\-LoRA atr=16r=16to test whether the saved activation memory can also be used effectively in diffusion\-transformer fine\-tuning\.

Results\.Table[IV](https://arxiv.org/html/2607.11940#S4.T4)shows that CARE\-LoRA atr=16r=16achieves a higher overall average of 46\.89, compared to 46\.79 for LoRA\. Subject fidelity improves on both metrics, with DINO rising from 43\.18 to 43\.40 and CLIP\-I from 69\.18 to 69\.29, while CLIP\-T remains essentially unchanged at 28\.00\. At the same time, peak memory drops from 9882 to 8630 MiB, a reduction of approximately 13%\. These results confirm that the memory benefit of CARE\-LoRA extends to the diffusion\-transformer architecture without sacrificing personalization quality\.

TABLE V:Activation Memory \(MiB\) of LoRA Layers Under Different Batch Sizes When Fine\-Tuning T5\-Base on MNLI\.Memory RatioIs Computed as the Activation Memory of CARE\-LoRA Divided by That of LoRA\. Smaller Values Indicate Lower Activation Memory Cost\.
### 4\.5Memory Analysis

Peak memory\.Across all four benchmarks \(Tables[I](https://arxiv.org/html/2607.11940#S3.T1),[II](https://arxiv.org/html/2607.11940#S3.T2),[III](https://arxiv.org/html/2607.11940#S4.T3), and[IV](https://arxiv.org/html/2607.11940#S4.T4)\), CARE\-LoRA consistently reduces peak memory relative to standard LoRA\. In the T5 experiments, CARE\-LoRA at the same rank reduces the total peak memory by roughly20%20\\%relative to LoRA on both GLUE and SuperGLUE\. In the larger Mistral\-7B\-v0\.3 and SD3\-Medium experiments, CARE\-LoRA uses rankr=16r=16and still requires less peak memory than standard LoRA atr=8r=8\. Since model weights, LoRA parameters, and optimizer states are fixed across methods and occupy a substantial share of total memory, the actual reduction on the activation budget is considerably larger in proportion\.

Activation memory of LoRA layers\.Table[V](https://arxiv.org/html/2607.11940#S4.T5)further isolates the savings on the LoRA layers, reporting their activation memory when fine\-tuning T5\-Base on MNLI under varying batch sizes\. CARE\-LoRA reduces this memory to only 2\.5% to 4\.7% of LoRA’s\. The ratio decreases as batch size grows, because the compressed activationZ∈ℝN×rZ\\in\\mathbb\{R\}^\{N\\times r\}scales withNNwhile the rank\-rrreconstruction matrixMMremains fixed, so the per\-sample overhead of CARE\-LoRA diminishes relative to the full activationX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}retained by LoRA at larger batch sizes\.

TABLE VI:Per\-Step Training Time \(Seconds\) on Fine\-Tuning Mistral\-7B\-v0\.3, Averaged Over Three Seeds on Different Training Tasks\.Time RatioIs Computed as Each Method’s Average Per\-Step Time Divided by That of LoRA\.
### 4\.6Time Analysis

TABLE VII:Comparison With Gradient Checkpointing Under a Similar Memory Budget When Fine\-Tuning Mistral\-7B\-v0\.3 on the Math Task\. Per\-Step Training Time \(Seconds\) and Peak CUDA Memory \(MiB\) Are Averaged Over Three Seeds\.Per\-step training time\.Table[VI](https://arxiv.org/html/2607.11940#S4.T6)compares the average per\-step training time on Mistral\-7B\-v0\.3, normalized by LoRA\. We omit PiSSA, whose per\-step computation matches LoRA once initialized\. For CARE\-LoRA, we report the costlierr=16r=16configuration as a conservative case\. DoRA combines per\-step weight renormalization with backward\-pass recomputation in its memory\-saving implementation, making it3\.39×3\.39\\timesslower than LoRA despite its competitive accuracy\. LoRAct performs an online low\-rank decomposition of the cached activation, raising the per\-step time to1\.38×1\.38\\timesthat of LoRA\. LoRA\-FA is the fastest but least accurate baseline\. CARE\-LoRA atr=16r=16requires only1\.02×1\.02\\timesthe per\-step time of LoRA atr=8r=8, nearly matching it\. Combined with the accuracy and memory results above, these findings show that CARE\-LoRA is the only method that is simultaneously memory\-efficient, fast, and accurate, whereas each baseline sacrifices at least one of these three axes\.

Comparison with gradient checkpointing\.Gradient checkpointing \(GC\)\[[7](https://arxiv.org/html/2607.11940#bib.bib4)\]is a standard approach to activation\-memory reduction, trading compute for memory by recomputing activations during the backward pass\. To enable a direct comparison, we apply GC to the first 25% of Mistral\-7B\-v0\.3’s decoder blocks, i\.e\., 8 of 32, during LoRA fine\-tuning on the Math task, so that its memory budget roughly matches that of CARE\-LoRA atr=16r=16\. Since GC does not alter the optimization, LoRA\+GC retains the accuracy of LoRA, so we focus on memory and time\. As shown in Table[VII](https://arxiv.org/html/2607.11940#S4.T7), under a comparable memory budget, CARE\-LoRA atr=16r=16outperforms LoRA\+GC in both per\-step time \(2\.34 vs\. 2\.46 s\) and peak memory \(25272 vs\. 25992 MiB\)\. The advantage is structural: GC saves memory by recomputing entire blocks, whereas CARE\-LoRA exploits the inherent low\-rank structure of LoRA, substantially reducing compute overhead at the same memory level\.

## 5Conclusion

In this paper, we propose CARE\-LoRA, a data\-aware framework that replaces the full activation retained by standard LoRA with a compressed activation and a lightweight reconstruction matrix, keeping both LoRA matrices trainable at a memory cost close to LoRA\-FA\. Building on this reconstruction, we proved that the projection\-down subspace is not frozen: under non\-degenerate activations and backward signals, it can evolve during training, unlike the fixed subspace that LoRA\-FA imposes\. Extensive experiments across diverse tasks show that CARE\-LoRA closes most of the accuracy gap LoRA\-FA leaves relative to standard LoRA, and that reinvesting the saved memory into a higher rank lets it surpass standard LoRA and representative variants with only a negligible increase in training time, achieving a favorable balance of memory efficiency, speed, and accuracy\.

## References

- \[1\]\(2025\)OpenCodeInstruct: a large\-scale instruction tuning dataset for code LLMs\.arXiv preprint arXiv:2504\.04030\.External Links:2504\.04030Cited by:[2nd item](https://arxiv.org/html/2607.11940#S4.I2.i2.p1.1)\.
- \[2\]K\. Bałazy, M\. Banaei, K\. Aberer, and J\. Tabor\(2025\)LoRA\-XS: low\-rank adaptation with extremely small number of parameters\.InEuropean Conference on Artificial Intelligence,Frontiers in Artificial Intelligence and Applications, Vol\.413,pp\. 3194–3201\.External Links:[Document](https://dx.doi.org/10.3233/FAIA251185)Cited by:[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5)\.
- \[3\]L\. Ben Allalet al\.\(2025\)SmolLM2: when smol goes big—data\-centric training of a fully open small language model\.InConference on Language Modeling,Cited by:[3rd item](https://arxiv.org/html/2607.11940#S4.I2.i3.p1.1)\.
- \[4\]M\. Caronet al\.\(2021\)Emerging properties in self\-supervised vision transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 9650–9660\.Cited by:[§4\.4](https://arxiv.org/html/2607.11940#S4.SS4.p1.2)\.
- \[5\]J\. Chenet al\.\(2021\)ActNN: reducing training memory footprint via 2\-bit activation compressed training\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 1803–1813\.Cited by:[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1)\.
- \[6\]M\. Chenet al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.External Links:2107\.03374Cited by:[2nd item](https://arxiv.org/html/2607.11940#S4.I2.i2.p1.1)\.
- \[7\]T\. Chen, B\. Xu, C\. Zhang, and C\. Guestrin\(2016\)Training deep nets with sublinear memory cost\.arXiv preprint arXiv:1604\.06174\.External Links:1604\.06174Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1),[§4\.6](https://arxiv.org/html/2607.11940#S4.SS6.p2.2)\.
- \[8\]K\. Cobbeet al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:2110\.14168Cited by:[1st item](https://arxiv.org/html/2607.11940#S4.I2.i1.p1.1)\.
- \[9\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 10088–10115\.External Links:[Document](https://dx.doi.org/10.52202/075280-0441)Cited by:[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5)\.
- \[10\]P\. Esseret al\.\(2024\)Scaling rectified flow transformers for high\-resolution image synthesis\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 12606–12633\.Cited by:[§4](https://arxiv.org/html/2607.11940#S4.p1.2)\.
- \[11\]E\. J\. Huet al\.\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5),[1st item](https://arxiv.org/html/2607.11940#S4.I1.i1.p1.1)\.
- \[12\]A\. Q\. Jianget al\.\(2023\)Mistral 7B\.arXiv preprint arXiv:2310\.06825\.External Links:2310\.06825Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p5.3),[§4](https://arxiv.org/html/2607.11940#S4.p1.2)\.
- \[13\]D\. J\. Kopiczko, T\. Blankevoort, and Y\. M\. Asano\(2024\)VeRA: vector\-based random matrix adaptation\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5)\.
- \[14\]V\. A\. Korthikantiet al\.\(2023\)Reducing activation recomputation in large transformer models\.InProceedings of Machine Learning and Systems,Vol\.5,pp\. 341–353\.Cited by:[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1)\.
- \[15\]S\. Liuet al\.\(2024\)DoRA: weight\-decomposed low\-rank adaptation\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 32100–32121\.Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p5.3),[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5),[§4\.3](https://arxiv.org/html/2607.11940#S4.SS3.p2.1)\.
- \[16\]I\. Loshchilov and F\. Hutter\(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§4\.1](https://arxiv.org/html/2607.11940#S4.SS1.p2.8)\.
- \[17\]S\. Mangrulkaret al\.\(2022\)PEFT: state\-of\-the\-art parameter\-efficient fine\-tuning methods\.Note:GitHub repository\[Online\]\. Available:https://github\.com/huggingface/peft\. Accessed: Jul\. 7, 2026Cited by:[§4\.1](https://arxiv.org/html/2607.11940#S4.SS1.p2.8)\.
- \[18\]F\. Meng, Z\. Wang, and M\. Zhang\(2024\)PiSSA: principal singular values and singular vectors adaptation of large language models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 121038–121072\.External Links:[Document](https://dx.doi.org/10.52202/079017-3846)Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p5.3),[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5),[§4\.3](https://arxiv.org/html/2607.11940#S4.SS3.p2.1)\.
- \[19\]A\. Radfordet al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 8748–8763\.Cited by:[§4\.4](https://arxiv.org/html/2607.11940#S4.SS4.p1.2)\.
- \[20\]C\. Raffelet al\.\(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.Cited by:[§4](https://arxiv.org/html/2607.11940#S4.p1.2)\.
- \[21\]N\. Ruiz, Y\. Li, V\. Jampani, Y\. Pritch, M\. Rubinstein, and K\. Aberman\(2023\)DreamBooth: fine tuning text\-to\-image diffusion models for subject\-driven generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 22500–22510\.Cited by:[§4](https://arxiv.org/html/2607.11940#S4.p1.2)\.
- \[22\]Y\. Shamshoum, N\. Hodos, Y\. Sieradzki, and A\. Schuster\(2025\)CompAct: compressed activations for memory\-efficient LLM training\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1: Long Papers,Albuquerque, NM, USA,pp\. 1511–1524\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.71)Cited by:[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1)\.
- \[23\]J\. Shi, W\. Wei, J\. Qi, X\. Chen, T\. Wei, and Y\. Li\(2025\)Memory\-efficient fine\-tuning via low\-rank activation compression\.arXiv preprint arXiv:2509\.23472\.External Links:2509\.23472Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1),[2nd item](https://arxiv.org/html/2607.11940#S4.I1.i2.p1.1)\.
- \[24\]A\. N\. Tikhonov\(1963\)On the regularization of ill\-posed problems\.Doklady Akademii Nauk SSSR153\(1\),pp\. 49–52\.Cited by:[§3\.2](https://arxiv.org/html/2607.11940#S3.SS2.p3.2)\.
- \[25\]R\. Vershynin\(2018\)High\-dimensional probability: an introduction with applications in data science\.Cambridge Series in Statistical and Probabilistic Mathematics,Cambridge University Press,Cambridge, U\.K\.\.External Links:[Document](https://dx.doi.org/10.1017/9781108231596)Cited by:[Lemma 2](https://arxiv.org/html/2607.11940#Thmtheorem2)\.
- \[26\]A\. Wanget al\.\(2019\)SuperGLUE: a stickier benchmark for general\-purpose language understanding systems\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p5.3),[§4](https://arxiv.org/html/2607.11940#S4.p1.2)\.
- \[27\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,Brussels, Belgium,pp\. 353–355\.External Links:[Document](https://dx.doi.org/10.18653/v1/W18-5446)Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p5.3),[§4](https://arxiv.org/html/2607.11940#S4.p1.2)\.
- \[28\]S\. Wang, L\. Yu, and J\. Li\(2024\)LoRA\-GA: low\-rank adaptation with gradient approximation\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 54905–54931\.External Links:[Document](https://dx.doi.org/10.52202/079017-1741)Cited by:[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5)\.
- \[29\]Y\. Wanget al\.\(2025\)HyC\-LoRA: memory efficient LoRA fine\-tuning with hybrid activation compression\.InProceedings of Machine Learning and Systems,Vol\.7\.Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1)\.
- \[30\]L\. Yuet al\.\(2024\)MetaMath: bootstrap your own mathematical questions for large language models\.InInternational Conference on Learning Representations,Cited by:[1st item](https://arxiv.org/html/2607.11940#S4.I2.i1.p1.1)\.
- \[31\]L\. Zhang, L\. Zhang, S\. Shi, X\. Chu, and B\. Li\(2023\)LoRA\-FA: memory\-efficient low\-rank adaptation for large language models fine\-tuning\.arXiv preprint arXiv:2308\.03303\.External Links:2308\.03303Cited by:[§1](https://arxiv.org/html/2607.11940#S1.p3.4),[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1),[3rd item](https://arxiv.org/html/2607.11940#S4.I1.i3.p1.2)\.
- \[32\]Q\. Zhanget al\.\(2023\)AdaLoRA: adaptive budget allocation for parameter\-efficient fine\-tuning\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.11940#S2.SS1.p1.5)\.
- \[33\]J\. Zhao, Z\. Zhang, B\. Chen, Z\. Wang, A\. Anandkumar, and Y\. Tian\(2024\)GaLore: memory\-efficient LLM training by gradient low\-rank projection\.InInternational Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 61121–61143\.Cited by:[§2\.2](https://arxiv.org/html/2607.11940#S2.SS2.p1.1)\.
- \[34\]J\. Zhouet al\.\(2023\)Instruction\-following evaluation for large language models\.arXiv preprint arXiv:2311\.07911\.External Links:2311\.07911Cited by:[3rd item](https://arxiv.org/html/2607.11940#S4.I2.i3.p1.1)\.

## Appendix AProofs for Subspace Adaptation

This appendix gives the detailed derivations for the subspace adaptation analysis in Section III\-C\. We first recall, with explicit intermediate steps, how the reconstructed gradient ofAAyields a condition for the projection\-down subspace to change\. We then give complete proofs of two claims that Section III\-C states without proof, namely that the data\-dependent part is nonzero except on a measure\-zero set of initializations, and that the matrixZ⊤​UZ^\{\\top\}Ugoverning the gradient\-dependent part is generically nonsingular\.

### A\.1Gradient Form and the Movement Condition

Recall that

Z=X​A,U=G​B⊤,ΣX=X⊤​X\.Z=XA,\\qquad U=GB^\{\\top\},\\qquad\\Sigma\_\{X\}=X^\{\\top\}X\.CARE\-LoRA stores the regularized reconstruction matrix

M=\(Z⊤​Z\+λ​Ir\)−1​Z⊤​X\.M=\(Z^\{\\top\}Z\+\\lambda I\_\{r\}\)^\{\-1\}Z^\{\\top\}X\.\(A1\)The reconstructed gradient forAAis

∇~A​ℒ=M⊤​Z⊤​U\.\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=M^\{\\top\}Z^\{\\top\}U\.\(A2\)Taking the transpose ofMMgives

M⊤=X⊤​Z​\(Z⊤​Z\+λ​Ir\)−1\.M^\{\\top\}=X^\{\\top\}Z\(Z^\{\\top\}Z\+\\lambda I\_\{r\}\)^\{\-1\}\.\(A3\)Therefore,

∇~A​ℒ\\displaystyle\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=X⊤​Z​\(Z⊤​Z\+λ​Ir\)−1​Z⊤​U\\displaystyle=X^\{\\top\}Z\(Z^\{\\top\}Z\+\\lambda I\_\{r\}\)^\{\-1\}Z^\{\\top\}U=X⊤​X​A​\(A⊤​X⊤​X​A\+λ​Ir\)−1​A⊤​X⊤​U\\displaystyle=X^\{\\top\}XA\(A^\{\\top\}X^\{\\top\}XA\+\\lambda I\_\{r\}\)^\{\-1\}A^\{\\top\}X^\{\\top\}U=ΣX​A​\(A⊤​ΣX​A\+λ​Ir\)−1​A⊤​X⊤​U\.\\displaystyle=\\Sigma\_\{X\}A\(A^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}\)^\{\-1\}A^\{\\top\}X^\{\\top\}U\.\(A4\)Defining

Kλ=\(A⊤​ΣX​A\+λ​Ir\)−1​A⊤​X⊤​UK\_\{\\lambda\}=\(A^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}\)^\{\-1\}A^\{\\top\}X^\{\\top\}U\(A5\)yields

∇~A​ℒ=ΣX​A​Kλ\.\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=\\Sigma\_\{X\}AK\_\{\\lambda\}\.\(A6\)Becauseλ\>0\\lambda\>0andA⊤​ΣX​AA^\{\\top\}\\Sigma\_\{X\}Ais positive semidefinite,A⊤​ΣX​A\+λ​IrA^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}is positive definite and therefore invertible\. Thus, the regularized form is always well\-defined\.

We now use this form to determine when a gradient step movesAAoutside its current column space\. Let

S=Col⁡\(A\),PS⟂=Im−PS\.S=\\operatorname\{Col\}\(A\),\\qquad P\_\{S\}^\{\\perp\}=I\_\{m\}\-P\_\{S\}\.Since every column ofAAlies inSS, we have

PS⟂​A=0\.P\_\{S\}^\{\\perp\}A=0\.\(A7\)For a gradient step

A\+=A−η​ΣX​A​Kλ,A^\{\+\}=A\-\\eta\\Sigma\_\{X\}AK\_\{\\lambda\},\(A8\)left\-multiplying byPS⟂P\_\{S\}^\{\\perp\}gives

PS⟂​A\+\\displaystyle P\_\{S\}^\{\\perp\}A^\{\+\}=PS⟂​A−η​PS⟂​ΣX​A​Kλ\\displaystyle=P\_\{S\}^\{\\perp\}A\-\\eta P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}AK\_\{\\lambda\}=−η​PS⟂​ΣX​A​Kλ\.\\displaystyle=\-\\eta P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}AK\_\{\\lambda\}\.\(A9\)Therefore, if

PS⟂​ΣX​A​Kλ≠0,P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}AK\_\{\\lambda\}\\neq 0,\(A10\)thenPS⟂​A\+≠0P\_\{S\}^\{\\perp\}A^\{\+\}\\neq 0\. Hence at least one column ofA\+A^\{\+\}has a nonzero component inS⟂S^\{\\perp\}, which implies

Col⁡\(A\+\)⊈S=Col⁡\(A\)\.\\operatorname\{Col\}\(A^\{\+\}\)\\nsubseteq S=\\operatorname\{Col\}\(A\)\.\(A11\)IfA\+A^\{\+\}has full column rank, thenCol⁡\(A\+\)\\operatorname\{Col\}\(A^\{\+\}\)is again anrr\-dimensional subspace, and the learned projection\-down subspace has changed\. As stated in Section III\-C, this condition separates into a data\-dependent part,PS⟂​ΣX​A≠0P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A\\neq 0, and a gradient\-dependent part concerning whetherKλK\_\{\\lambda\}cancels it\. The next two subsections prove that each part holds except on a measure\-zero set of degenerate cases\.

### A\.2Invariance Analysis of the Data\-Dependent Part

Section III\-C states that the data\-dependent conditionPS⟂​ΣX​A≠0P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A\\neq 0fails exactly whenCol⁡\(A\)\\operatorname\{Col\}\(A\)is an invariant subspace ofΣX\\Sigma\_\{X\}, but does not prove this equivalence\. We prove it here, and then justify why such an invariant subspace is nongeneric\.

We show that

PS⟂​ΣX​A=0P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A=0\(A12\)is equivalent toS=Col⁡\(A\)S=\\operatorname\{Col\}\(A\)being invariant underΣX\\Sigma\_\{X\}\.

First, ifPS⟂​ΣX​A=0P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A=0, then every column ofΣX​A\\Sigma\_\{X\}Alies inSS, so

Col⁡\(ΣX​A\)⊆S\.\\operatorname\{Col\}\(\\Sigma\_\{X\}A\)\\subseteq S\.\(A13\)For any vectors∈Ss\\in S, there existsv∈ℝrv\\in\\mathbb\{R\}^\{r\}such thats=A​vs=Av\. Then

ΣX​s=ΣX​A​v∈Col⁡\(ΣX​A\)⊆S\.\\Sigma\_\{X\}s=\\Sigma\_\{X\}Av\\in\\operatorname\{Col\}\(\\Sigma\_\{X\}A\)\\subseteq S\.\(A14\)ThusΣX​S⊆S\\Sigma\_\{X\}S\\subseteq S, i\.e\.,SSis an invariant subspace ofΣX\\Sigma\_\{X\}\.

Conversely, ifΣX​S⊆S\\Sigma\_\{X\}S\\subseteq S, then each column ofAAbelongs toSS, and therefore each column ofΣX​A\\Sigma\_\{X\}Aalso belongs toSS\. Hence

PS⟂​ΣX​A=0\.P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A=0\.\(A15\)This proves

PS⟂​ΣX​A=0⟺ΣX​S⊆S\.P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A=0\\quad\\Longleftrightarrow\\quad\\Sigma\_\{X\}S\\subseteq S\.\(A16\)
It remains to argue that a randomly initializedCol⁡\(A\)\\operatorname\{Col\}\(A\)is essentially never an invariant subspace ofΣX\\Sigma\_\{X\}\. For a fixed non\-scalar symmetric matrixΣX\\Sigma\_\{X\}, a continuously sampledrr\-dimensional subspace is not an invariant subspace with probability one\. Indeed, the invariant subspaces of a symmetric matrix are constrained by its eigenspace decomposition\. Every invariant subspace decomposes as a direct sum of subspaces drawn from the individual eigenspaces ofΣX\\Sigma\_\{X\}, so an invariantrr\-dimensional subspace must align with this eigenstructure rather than sit in a generic position\. UnlessΣX\\Sigma\_\{X\}is proportional to the identity, this alignment requirement confines the invariantrr\-dimensional subspaces to a measure\-zero subset of the Grassmannian of allrr\-dimensional subspaces, so a continuously distributed initialization ofCol⁡\(A\)\\operatorname\{Col\}\(A\)avoids them almost surely\. The isotropic caseΣX=c​Im\\Sigma\_\{X\}=cI\_\{m\}is the degenerate exception, since every subspace is invariant\.

### A\.3Genericity of the Gradient\-Dependent Part

We now turn to the gradient\-dependent part\. Let

D=PS⟂​ΣX​A\.D=P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}A\.\(A17\)Even whenD≠0D\\neq 0, the outside\-subspace component of the CARE\-LoRA gradient is

Thus the outside component vanishes if and only if every column ofKλK\_\{\\lambda\}lies in the null space ofDD

D​Kλ=0⟺Col⁡\(Kλ\)⊆Null⁡\(D\)\.DK\_\{\\lambda\}=0\\quad\\Longleftrightarrow\\quad\\operatorname\{Col\}\(K\_\{\\lambda\}\)\\subseteq\\operatorname\{Null\}\(D\)\.\(A18\)This is a backward\-signal degeneracy\. Since

Kλ\\displaystyle K\_\{\\lambda\}=\(A⊤​ΣX​A\+λ​Ir\)−1​A⊤​X⊤​U\\displaystyle=\(A^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}\)^\{\-1\}A^\{\\top\}X^\{\\top\}U=\(A⊤​ΣX​A\+λ​Ir\)−1​Z⊤​U,\\displaystyle=\(A^\{\\top\}\\Sigma\_\{X\}A\+\\lambda I\_\{r\}\)^\{\-1\}Z^\{\\top\}U,\(A19\)and the first factor is invertible forλ\>0\\lambda\>0, this degeneracy is governed entirely by the rank ofZ⊤​UZ^\{\\top\}U\. Because an invertible factor preserves rank,KλK\_\{\\lambda\}is singular if and only ifZ⊤​UZ^\{\\top\}Uis\. For example, ifU=G​B⊤=0U=GB^\{\\top\}=0, thenZ⊤​U=0Z^\{\\top\}U=0, soKλ=0K\_\{\\lambda\}=0and no update toAAis produced\. This happens at the first optimization step under the common LoRA initializationB=0B=0\.

Section III\-C asserts that, away from such special cases,Z⊤​UZ^\{\\top\}Uis generically nonsingular, without proof\. We prove this claim below\.

###### Lemma 1\(Generic nonsingularity ofZ⊤​UZ^\{\\top\}U\)\.

LetN≥rN\\geq rand letZ,U∈ℝN×rZ,U\\in\\mathbb\{R\}^\{N\\times r\}\. Suppose that the joint distribution of\(Z,U\)\(Z,U\)is absolutely continuous with respect to the Lebesgue measure onℝN×r×ℝN×r\\mathbb\{R\}^\{N\\times r\}\\times\\mathbb\{R\}^\{N\\times r\}\. Then

Pr⁡\(rank⁡\(Z⊤​U\)<r\)=0\.\\Pr\\\!\\left\(\\operatorname\{rank\}\(Z^\{\\top\}U\)<r\\right\)=0\.\(A20\)In particular, sincerank⁡\(Z\)=r\\operatorname\{rank\}\(Z\)=randrank⁡\(U\)=r\\operatorname\{rank\}\(U\)=ralso hold with probability one, we have

Pr⁡\(rank⁡\(Z⊤​U\)​<r\|​rank⁡\(Z\)=rank⁡\(U\)=r\)=0\.\\Pr\\\!\\left\(\\operatorname\{rank\}\(Z^\{\\top\}U\)<r\\,\\middle\|\\,\\operatorname\{rank\}\(Z\)=\\operatorname\{rank\}\(U\)=r\\right\)=0\.\(A21\)

###### Proof\.

The conditionrank⁡\(Z⊤​U\)<r\\operatorname\{rank\}\(Z^\{\\top\}U\)<ris equivalent to

det\(Z⊤​U\)=0\.\\det\(Z^\{\\top\}U\)=0\.\(A22\)The function

p​\(Z,U\)=det\(Z⊤​U\)p\(Z,U\)=\\det\(Z^\{\\top\}U\)\(A23\)is a polynomial in the entries ofZZandUU\. This polynomial is not identically zero\. For example, taking

Z=\[Ir0\],U=\[Ir0\]Z=\\begin\{bmatrix\}I\_\{r\}\\\\ 0\\end\{bmatrix\},\\qquad U=\\begin\{bmatrix\}I\_\{r\}\\\\ 0\\end\{bmatrix\}\(A24\)givesZ⊤​U=IrZ^\{\\top\}U=I\_\{r\}and hencep​\(Z,U\)=1p\(Z,U\)=1\. Therefore, the zero set\{\(Z,U\):p​\(Z,U\)=0\}\\\{\(Z,U\):p\(Z,U\)=0\\\}is a proper algebraic variety and has Lebesgue measure zero\. Since\(Z,U\)\(Z,U\)has an absolutely continuous joint distribution, the probability of this event is zero\.

The same argument also implies thatrank⁡\(Z\)=r\\operatorname\{rank\}\(Z\)=randrank⁡\(U\)=r\\operatorname\{rank\}\(U\)=rhold with probability one, because the vanishing of allr×rr\\times rminors is again a measure\-zero algebraic condition\. This proves the conditional statement\. ∎

Together with Section[A\.2](https://arxiv.org/html/2607.11940#A1.SS2), this shows that, under the assumptions stated above, the set of initializations and backward signals for whichPS⟂​ΣX​A​Kλ=0P\_\{S\}^\{\\perp\}\\Sigma\_\{X\}AK\_\{\\lambda\}=0has measure zero\. Hence, the projection\-down subspace generically evolves under CARE\-LoRA\.

## Appendix BDetailed Memory Analysis

This appendix gives the detailed derivation for the memory analysis of the main paper\. We analyze one adapted linear layer

Y\\displaystyle Y=X​\(W\+A​B\),\\displaystyle=X\(W\+AB\),X\\displaystyle X∈ℝN×m,A∈ℝm×r,B∈ℝr×n\.\\displaystyle\\in\\mathbb\{R\}^\{N\\times m\},\\quad A\\in\\mathbb\{R\}^\{m\\times r\},\\quad B\\in\\mathbb\{R\}^\{r\\times n\}\.The count below only includes tensors specific to the LoRA branch\. The frozen backbone pathX​WXWand other model activations are shared by the compared methods\.

### B\.1Activation Memory

Standard LoRA keepsAAandBBtrainable\. To compute

∇Aℒ=X⊤​G​B⊤,\\nabla\_\{A\}\\mathcal\{L\}=X^\{\\top\}GB^\{\\top\},\(B1\)the full activationX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}must be available in the backward pass\. Following the dominant\-term accounting used in the main text, the LoRA\-side activation count is

ℳLoRAact=N​m\.\\mathcal\{M\}\_\{\\mathrm\{LoRA\}\}^\{\\mathrm\{act\}\}=Nm\.\(B2\)
LoRA\-FA freezesAAand only updatesBB\. Since it does not need∇Aℒ\\nabla\_\{A\}\\mathcal\{L\}, it only saves

Z=X​A∈ℝN×r,Z=XA\\in\\mathbb\{R\}^\{N\\times r\},\(B3\)and therefore

ℳLoRA​\-​FAact=N​r\.\\mathcal\{M\}\_\{\\mathrm\{LoRA\\text\{\-\}FA\}\}^\{\\mathrm\{act\}\}=Nr\.\(B4\)
CARE\-LoRA savesZZand the batch\-dependent reconstruction matrixM∈ℝr×mM\\in\\mathbb\{R\}^\{r\\times m\}, giving

ℳCAREact=N​r\+r​m\.\\mathcal\{M\}\_\{\\mathrm\{CARE\}\}^\{\\mathrm\{act\}\}=Nr\+rm\.\(B5\)It is smaller than the LoRA\-side activation count whenever

N​r\+r​m<N​m⟺r<N​mN\+m\.Nr\+rm<Nm\\quad\\Longleftrightarrow\\quad r<\\frac\{Nm\}\{N\+m\}\.\(B6\)For example, whenN=8192N=8192andm=768m=768, the right\-hand side is approximately702\.2702\.2, which is far larger than common LoRA ranks such as88or1616, so this condition is easily satisfied in typical fine\-tuning settings\.

### B\.2Layer\-Wise Numerical Examples

ForN=8192N=8192, the activation counts for a single adapted layer are as follows\.

TABLE VIII:LoRA\-Side Activation Storage for One Adapted Linear Layer\. Counts Are Reported in Number of Scalar Values\.If LoRA is applied to all attention and feed\-forward linear layers of a T5\-Base\-style encoder\-decoder model, the same calculation can be aggregated over layers\. With1212encoder blocks and1212decoder blocks, an encoder block contains four attention projections and two feed\-forward projections, while a decoder block contains self\-attention, cross\-attention, and two feed\-forward projections\. Usingdmodel=768d\_\{\\mathrm\{model\}\}=768anddff=3072d\_\{\\mathrm\{ff\}\}=3072, the sum of input dimensions over these adapted linear layers is

∑ℓmℓ\\displaystyle\\sum\_\{\\ell\}m\_\{\\ell\}=12​\(5⋅768\+3072\)\+12​\(9⋅768\+3072\)\\displaystyle=2\(5\\cdot 68\+072\)\+2\(9\\cdot 68\+072\)=202,752,\\displaystyle=02\{,\}52,and the number of adapted linear layers is

Ladapt=12⋅6\+12⋅10=192\.L\_\{\\mathrm\{adapt\}\}=12\\cdot 6\+12\\cdot 10=192\.ForN=8192N=8192, standard LoRA stores

N​∑ℓmℓ=1660\.9​MN\\sum\_\{\\ell\}m\_\{\\ell\}=1660\.9\\mathrm\{M\}LoRA\-side activation values, independent ofrrunder this dominant\-term count\. Forr=8r=8, CARE\-LoRA stores

r​\(N​Ladapt\+∑ℓmℓ\)=14\.20​Mr\\left\(NL\_\{\\mathrm\{adapt\}\}\+\\sum\_\{\\ell\}m\_\{\\ell\}\\right\)=14\.20\\mathrm\{M\}values, about0\.86%0\.86\\%of the standard LoRA count\. Assuming two bytes per value in BF16 or FP16, this corresponds to roughly3\.093\.09GiB versus27\.127\.1MiB for these adapter\-side activation buffers\. Forr=16r=16, the corresponding CARE\-LoRA count is28\.41​M28\.41\\mathrm\{M\}values, or about1\.71%1\.71\\%of standard LoRA\.

## Appendix CDetailed Computation Analysis

This appendix gives the detailed derivation for the computation analysis of the main paper, using the same adapted linear layer as in Appendix[B](https://arxiv.org/html/2607.11940#A2), with input activationX∈ℝN×mX\\in\\mathbb\{R\}^\{N\\times m\}, projection\-down matrixA∈ℝm×rA\\in\\mathbb\{R\}^\{m\\times r\}, and projection\-up matrixB∈ℝr×nB\\in\\mathbb\{R\}^\{r\\times n\}\. As in Appendix[B](https://arxiv.org/html/2607.11940#A2), the count below only includes matrix multiplications specific to the LoRA branch, excluding the shared frozen backbone path\.

### C\.1Computation Cost

We count matrix multiplication costs up to constant factors, i\.e\., using multiply\-accumulate order rather than distinguishing a factor of two in FLOPs\. The LoRA branch consists of the following operations\.

Standard LoRA\.In the forward pass, standard LoRA computes

Z=X​A,Δ​Y=Z​B,Z=XA,\\qquad\\Delta Y=ZB,with costsN​m​rNmrandN​r​nNrn, respectively\.

In the backward pass, letG=∂ℒ/∂YG=\\partial\\mathcal\{L\}/\\partial Y\. Standard LoRA computes

U=G​B⊤∈ℝN×r,U=GB^\{\\top\}\\in\\mathbb\{R\}^\{N\\times r\},\(C1\)with costN​r​nNrn, then

∇Aℒ=X⊤​U\\nabla\_\{A\}\\mathcal\{L\}=X^\{\\top\}U\(C2\)with costN​m​rNmr, and

∇Bℒ=Z⊤​G\\nabla\_\{B\}\\mathcal\{L\}=Z^\{\\top\}G\(C3\)with costN​r​nNrn\. The LoRA\-side input\-gradient contribution

has costN​m​rNmr\. Therefore,

TLoRA=N​m​r⏟X​A\+N​r​n⏟Z​B\+N​r​n⏟G​B⊤\+N​m​r⏟X⊤​U\+N​r​n⏟Z⊤​G\+N​m​r⏟U​A⊤,T\_\{\\mathrm\{LoRA\}\}=\\underbrace\{Nmr\}\_\{XA\}\+\\underbrace\{Nrn\}\_\{ZB\}\+\\underbrace\{Nrn\}\_\{GB^\{\\top\}\}\+\\underbrace\{Nmr\}\_\{X^\{\\top\}U\}\+\\underbrace\{Nrn\}\_\{Z^\{\\top\}G\}\+\\underbrace\{Nmr\}\_\{UA^\{\\top\}\},\(C5\)which gives

TLoRA=3​N​m​r\+3​N​r​n\.T\_\{\\mathrm\{LoRA\}\}=3Nmr\+3Nrn\.\(C6\)
CARE\-LoRA\.CARE\-LoRA shares the sameX​AXA,Z​BZB,G​B⊤GB^\{\\top\},Z⊤​GZ^\{\\top\}G, andU​A⊤UA^\{\\top\}computations\. The difference is how theAA\-gradient is obtained\.

In the forward pass, CARE\-LoRA forms

H=Z⊤​Z\+λ​Ir,C=Z⊤​X,H=Z^\{\\top\}Z\+\\lambda I\_\{r\},\\qquad C=Z^\{\\top\}X,and solves

M=solve​\(H,C\)\.M=\\mathrm\{solve\}\(H,C\)\.\(C7\)The cost of formingHHisN​r2Nr^\{2\}\. The cost of formingCCisN​m​rNmr\. ThisN​m​rNmrterm replaces theN​m​rNmrterm used by standard LoRA to computeX⊤​UX^\{\\top\}Uin the backward pass\. Therefore, it is not an additional large\-matrix term relative to standard LoRA\. SinceHHis anr×rr\\times rsymmetric positive definite matrix forλ\>0\\lambda\>0, it can be factorized by Cholesky decomposition and solved againstmmright\-hand sides\. This contributes

O​\(r3\+m​r2\)\.O\(r^\{3\}\+mr^\{2\}\)\.\(C8\)
In the backward pass, CARE\-LoRA computes

with costN​r2Nr^\{2\}, and then

∇~A​ℒ=M⊤​T\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=M^\{\\top\}T\(C10\)with costm​r2mr^\{2\}\. Therefore,

TCARE\\displaystyle T\_\{\\mathrm\{CARE\}\}=3​N​m​r\+3​N​r​n\+2​N​r2\+2​m​r2\+r3\.\\displaystyle=3Nmr\+3Nrn\+2Nr^\{2\}\+2mr^\{2\}\+r^\{3\}\.\(C11\)The ratio between CARE\-LoRA and standard LoRA is

TCARETLoRA\\displaystyle\\frac\{T\_\{\\mathrm\{CARE\}\}\}\{T\_\{\\mathrm\{LoRA\}\}\}=3​N​m​r\+3​N​r​n\+2​N​r2\+2​m​r2\+r33​N​m​r\+3​N​r​n\\displaystyle=\\frac\{3Nmr\+3Nrn\+2Nr^\{2\}\+2mr^\{2\}\+r^\{3\}\}\{3Nmr\+3Nrn\}\(C12\)=1\+2​N​r2\+2​m​r2\+r33​N​r​\(m\+n\)\\displaystyle=1\+\\frac\{2Nr^\{2\}\+2mr^\{2\}\+r^\{3\}\}\{3Nr\(m\+n\)\}=1\+r​\(2​N\+2​m\+r\)3​N​\(m\+n\)\.\\displaystyle=1\+\\frac\{r\(2N\+2m\+r\)\}\{3N\(m\+n\)\}\.Thus the extra arithmetic scales withrrrelative to the large feature dimensions\. For common settings withr≪min⁡\(m,n,N\)r\\ll\\min\(m,n,N\), the additional algebraic cost is small\.

### C\.2Numerical Cost Ratios

For a hidden\-to\-hidden layer withN=8192N=8192,m=n=768m=n=768, andr=8r=8,

TLoRA=3⋅8192⋅768⋅8\+3⋅8192⋅8⋅768=301\.99​M,T\_\{\\mathrm\{LoRA\}\}=3\\cdot 8192\\cdot 768\\cdot 8\+3\\cdot 8192\\cdot 8\\cdot 768=301\.99\\mathrm\{M\},and the reconstruction overhead is

2⋅8192⋅82\+2⋅768⋅82\+83=1\.147​M\.2\\cdot 8192\\cdot 8^\{2\}\+2\\cdot 768\\cdot 8^\{2\}\+8^\{3\}=1\.147\\mathrm\{M\}\.Therefore,

TCARETLoRA≈1\.0038\.\\frac\{T\_\{\\mathrm\{CARE\}\}\}\{T\_\{\\mathrm\{LoRA\}\}\}\\approx 1\.0038\.Forr=16r=16, the ratio becomes approximately1\.00761\.0076\.

For a feed\-forward down\-projection input withm=3072m=3072,n=768n=768, andr=8r=8, the same formula gives

TCARETLoRA≈1\.0019\.\\frac\{T\_\{\\mathrm\{CARE\}\}\}\{T\_\{\\mathrm\{LoRA\}\}\}\\approx 1\.0019\.Forr=16r=16, the ratio is approximately1\.00381\.0038\. These ratios should be understood as algebraic operation counts\. Actual wall\-clock time also depends on kernel fusion, memory access patterns, and the efficiency of the small Cholesky solve\.

## Appendix DDirectional Fidelity of the Reconstructed Gradient

Global Cosine Similarity\.CARE\-LoRA updates the projection\-down matrix with the reconstructed gradient∇~A​ℒ=X^⊤​G​B⊤\\widetilde\{\\nabla\}\_\{A\}\\mathcal\{L\}=\\widehat\{X\}^\{\\top\}GB^\{\\top\}in place of the exact gradient∇Aℒ=X⊤​G​B⊤\\nabla\_\{A\}\\mathcal\{L\}=X^\{\\top\}GB^\{\\top\}\. We quantify the fidelity of this approximation via the global cosine similarity between the exact and reconstructedAA\-gradients across allLLadapted layers:

Cglobal=⟨gall,g~all⟩‖gall‖2​‖g~all‖2,C\_\{\\mathrm\{global\}\}=\\frac\{\\langle g\_\{\\mathrm\{all\}\},\\widetilde\{g\}\_\{\\mathrm\{all\}\}\\rangle\}\{\\\|g\_\{\\mathrm\{all\}\}\\\|\_\{2\}\\,\\\|\\widetilde\{g\}\_\{\\mathrm\{all\}\}\\\|\_\{2\}\},\(D1\)wheregall=\[vec\(∇A1ℒ\)⊤,…,vec\(∇ALℒ\)⊤\]⊤g\_\{\\mathrm\{all\}\}=\\big\[\\operatorname\{vec\}\(\\nabla\_\{A\_\{1\}\}\\mathcal\{L\}\)^\{\\top\},\\dots,\\operatorname\{vec\}\(\\nabla\_\{A\_\{L\}\}\\mathcal\{L\}\)^\{\\top\}\\big\]^\{\\top\}andg~all\\widetilde\{g\}\_\{\\mathrm\{all\}\}is defined analogously\. By constructionCglobal∈\[−1,1\]C\_\{\\mathrm\{global\}\}\\in\[\-1,1\], withCglobal=1C\_\{\\mathrm\{global\}\}=1indicating perfect directional alignment between the two gradients\. In practice,gallg\_\{\\mathrm\{all\}\}need not be explicitly constructed because the inner product and squared norms can be computed and accumulated layer by layer\. We focus on direction rather than magnitude because differences in magnitude can be absorbed by the learning rate and further attenuated by AdamW’s per\-coordinate rescaling\.

Setting\.We measureCglobalC\_\{\\mathrm\{global\}\}along a standard CARE\-LoRA fine\-tuning run of Mistral\-7B\-v0\.3 on the Math task withr=16r=16and seed 0\. The trajectory is driven entirely by the reconstructed gradient, and at diagnostic checkpoints we temporarily retainXXto compute the exact∇Aℒ\\nabla\_\{A\}\\mathcal\{L\}for measurement only\. These checkpoints are placed evenly at5%,10%,…,100%5\\%,10\\%,\\dots,100\\%of the optimizer steps\. The0%0\\%point is excluded because the default initializationB=0B=0forces both gradients to vanish and leaves the cosine undefined\.

Results\.Figure[2](https://arxiv.org/html/2607.11940#A4.F2)plotsCglobalC\_\{\\mathrm\{global\}\}along the training trajectory\.CglobalC\_\{\\mathrm\{global\}\}stays consistently above0\.60\.6from the first valid checkpoint to the last, with no visible degradation across training\. The reconstructedAA\-gradient therefore retains strong directional agreement with the exact∇Aℒ\\nabla\_\{A\}\\mathcal\{L\}throughout fine\-tuning, justifying its use as the surrogate update direction in CARE\-LoRA\.

![Refer to caption](https://arxiv.org/html/2607.11940v1/x2.png)Figure 2:Global cosine similarity between the exact and reconstructedAA\-gradients of CARE\-LoRA along the training trajectory on the Math task\. The dashed line at1\.01\.0marks perfect alignment\.Geometric interpretation\.The observed cosine must be read against the geometry of the ambient space\. The concatenatedAA\-gradient vector in Eq\. \([D1](https://arxiv.org/html/2607.11940#A4.E1)\) lives inℝd\\mathbb\{R\}^\{d\}withd=r​∑ℓmℓd=r\\sum\_\{\\ell\}m\_\{\\ell\}, wheremℓm\_\{\\ell\}is the input width of theℓ\\ell\-th adapted layer\. For Mistral\-7B\-v0\.3 withr=16r=16andL=224L=224adapted linear layers,∑ℓmℓ=1,245,184\\sum\_\{\\ell\}m\_\{\\ell\}=1\{,\}245\{,\}184, givingd≈1\.99×107d\\approx 1\.99\\times 10^\{7\}\. In a space of this dimension, two directions chosen without relation to each other are almost exactly orthogonal, as the following lemma makes precise\.

###### Lemma 2\(Near\-orthogonality in high dimension\[[25](https://arxiv.org/html/2607.11940#bib.bib34)\]\)\.

Letu,vu,vbe independent and uniformly distributed on the unit sphere𝕊d−1⊂ℝd\\mathbb\{S\}^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}\. Then𝔼​⟨u,v⟩=0\\mathbb\{E\}\\langle u,v\\rangle=0,Var⁡⟨u,v⟩=1/d\\operatorname\{Var\}\\langle u,v\\rangle=1/d, and for everyε∈\(0,1\)\\varepsilon\\in\(0,1\),

Pr⁡\[\|⟨u,v⟩\|≥ε\]≤2​exp⁡\(−d​ε22\)\.\\Pr\\\!\\big\[\\,\|\\langle u,v\\rangle\|\\geq\\varepsilon\\,\\big\]\\;\\leq\\;2\\exp\\\!\\Big\(\\\!\-\\tfrac\{d\\varepsilon^\{2\}\}\{2\}\\Big\)\.\(D2\)

Instantiated here, Lemma[2](https://arxiv.org/html/2607.11940#Thmtheorem2)implies that, under the random\-direction null model, an uninformative reconstruction would produce a cosine of typical magnitude1/d≈2\.2×10−41/\\sqrt\{d\}\\approx 2\.2\\times 10^\{\-4\}, and that the probability of reaching even0\.60\.6by chance is at most2​exp⁡\(−3\.6×106\)2\\exp\(\-3\.6\\times 10^\{6\}\), effectively zero\. This provides supporting evidence that the reconstructed gradient retains meaningful directional alignment with the exactAA\-gradient\. We further note that0\.60\.6is an empirical lower bound across our diagnostic checkpoints rather than a typical value, so the observed directional alignment over the bulk of training is generally stronger than this conservative reading suggests\.

TABLE IX:Hyperparameter Configuration for Fine\-Tuning T5\-Base on the GLUE Datasets\.TABLE X:Hyperparameter Configuration for Fine\-Tuning T5\-Base on the SuperGLUE Datasets\.
## Appendix EDetailed Experimental Configurations

This appendix reports the key hyperparameters, evaluation protocols, and hardware used for the three experimental settings in Section IV, covering T5\-Base fine\-tuning on GLUE and SuperGLUE, Mistral\-7B\-v0\.3 fine\-tuning on mathematical reasoning, code generation, and instruction following, and SD3\-Medium fine\-tuning on DreamBooth personalization\.

Unless otherwise noted, all methods across the three settings use the standard LoRA initialization, withAAdrawn from a Kaiming uniform distribution andBBset to zero, the default and most widely used initialization for LoRA\-based fine\-tuning\. The exception is PiSSA, which instead uses the SVD\-based initialization already described in Section IV\-C\.

For CARE\-LoRA, we use a fixed diagonal offsetλ=10−6\\lambda=10^\{\-6\}in all experiments when solving\(Z⊤​Z\+λ​Ir\)​M=Z⊤​X\(Z^\{\\top\}Z\+\\lambda I\_\{r\}\)M=Z^\{\\top\}X\. This value is used for numerical stability and is not tuned across tasks\.

### E\.1Experiments on Natural Language Understanding

Table[IX](https://arxiv.org/html/2607.11940#A4.T9)and Table[X](https://arxiv.org/html/2607.11940#A4.T10)list the hyperparameters used for fine\-tuning T5\-Base on GLUE and SuperGLUE, corresponding to the same\-rank results in Section IV\-B\. All compared methods use rankr=8r=8withα=16\\alpha=16\.

The number of training epochs varies by task, as shown in Table[IX](https://arxiv.org/html/2607.11940#A4.T9)and Table[X](https://arxiv.org/html/2607.11940#A4.T10)\. We evaluate on the validation set after every epoch and report the best\-epoch result for each task\.

All experiments in this subsection are conducted on a single NVIDIA GeForce RTX 2080 Ti 11GB\.

### E\.2Experiments on Large Language Models

Table[XI](https://arxiv.org/html/2607.11940#A5.T11)lists the hyperparameters used for fine\-tuning Mistral\-7B\-v0\.3 on mathematical reasoning, code generation, and instruction following, corresponding to the results in Section IV\-C\. The main comparison uses rankr=8r=8withα=16\\alpha=16for all methods, and CARE\-LoRA is additionally evaluated atr=16r=16withα=32\\alpha=32to study whether the saved activation memory can be reinvested into stronger performance, keepingα/r=2\\alpha/r=2fixed\.

TABLE XI:Hyperparameter Configuration for Fine\-Tuning Mistral\-7B\-v0\.3 on Mathematical Reasoning, Code Generation, and Instruction Following\.TABLE XII:Training and Generation Configuration for Fine\-Tuning SD3\-Medium on DreamBooth Personalization\.All experiments in this subsection are conducted on a single NVIDIA GeForce RTX 5090 32GB\.

### E\.3Experiments on Diffusion Models

Table[XII](https://arxiv.org/html/2607.11940#A5.T12)lists the training and generation configuration used for fine\-tuning SD3\-Medium on DreamBooth personalization, corresponding to the results in Section IV\-D\. Each of the ten subjects is trained independently under this configuration for both standard LoRA and CARE\-LoRA\.

All experiments in this subsection are conducted on a single NVIDIA GeForce RTX 4090 24GB\.

## Appendix FBroader Impacts and Limitations

Broader Impacts\.CARE\-LoRA reduces the activation memory required for LoRA fine\-tuning while keeping both LoRA matrices trainable, lowering the hardware cost of adapting large pretrained models to new tasks\. This can make fine\-tuning large language and diffusion models more accessible to research groups and practitioners with limited computational resources\. As with other efficiency improvements for model fine\-tuning, easier access to adaptation could also lower the barrier for misuse, and we encourage use of CARE\-LoRA consistent with the license and usage terms of the underlying pretrained models\.

Limitations\.While CARE\-LoRA is effective across the settings studied in this paper, several directions remain open\. \(i\) Although the experiments in this paper cover T5\-Base, Mistral\-7B\-v0\.3, and SD3\-Medium, evaluating larger backbones and additional modalities remains to be explored\. \(ii\) The subspace\-movement guarantee proved in this paper holds generically\. This is a common form of guarantee in this line of analysis, though it does not cover atypical worst\-case configurations\. \(iii\) The reconstruction is developed for linear layers adapted by LoRA, and extending a similar idea to other adapter families or to activations retained by nonlinear operators is a direction we leave for future work\.