Same Rank, Different Needs: Rethinking LoRA Redistribution in Federated Learning
Summary
This paper introduces RASA, a receiver-aware framework for redistributing aggregated LoRA updates in federated learning, showing that spectral truncation can cause arbitrarily higher response error than input-aware readouts and achieving gains such as +3.23 points on DomainNet-100 and up to +8.33 ROUGE-L on SuperNI.
View Cached Full Text
Cached at: 11/19/26, 09:02 AM
# Rethinking LoRA Redistribution in Federated Learning
Source: [https://arxiv.org/html/2611.23742](https://arxiv.org/html/2611.23742)
## Same Rank, Different Needs: Rethinking LoRA Redistribution in Federated Learning
Jihua Zhu††thanks:Corresponding author\.Chang LiuAffiliation:School of Software Engineering, Xi’an Jiaotong UniversityAffiliation:chenjinqian@stu\.xjtu\.edu\.cn,zhujh@xjtu\.edu\.cn
###### Abstract
Federated low\-rank adaptation \(LoRA\) enables clients with different resource budgets to collaboratively fine\-tune foundation models\. To distribute an aggregated update within each client’s rank budget, existing methods commonly retain its leading singular directions\. However, directions that best reconstruct the update matrix need not best preserve its responses on a particular client’s inputs: a lower\-energy direction may matter more to that recipient\. We formalize this mismatch and show that matrix\-optimal truncation can incur arbitrarily larger response error than another readout of the same rank\. Motivated by this result, we propose Receiver\-Aware Subspace Adaptation \(RASA\), a plug\-in framework for receiver\-aware knowledge transfer\. RASA uses compact diagonal input second moments to perform uncentered PCA of receiver\-weighted update responses\. The resulting readout is globally optimal for the rank\-constrained receiver objective and preserves at least as much response energy as spectral truncation under the diagonal metric\. RASA also introduces spectral\-reference aggregation, which removes the personalized initialization offset while retaining the locally learned increment\. Experiments on DomainNet\-100, NICO\+\+, GLUE, and SuperNI demonstrate RASA’s effectiveness across methods under resource, task, and data heterogeneity\. In particular, RASA improves DomainNet\-100 accuracy by 3\.23 percentage points with FlexLoRA and achieves gains of up to 8\.33 ROUGE\-L points on seen SuperNI tasks and 4\.37 on unseen tasks with modest overhead\.
## 1Introduction
Federated learning enables clients to acquire knowledge beyond their local data through collaborative model training\[[13](https://arxiv.org/html/2611.23742#bib.bib1)\]\. For foundation models, realizing this promise requires adaptation to heterogeneous device capabilities\. Low\-rank adaptation \(LoRA\) makes local fine\-tuning affordable by learning compact updates to a frozen backbone\[[9](https://arxiv.org/html/2611.23742#bib.bib2)\], while heterogeneous federated LoRA matches each client’s adapter rank to its resource budget\[[4](https://arxiv.org/html/2611.23742#bib.bib4),[1](https://arxiv.org/html/2611.23742#bib.bib3)\]\. This supports collaboration across clients that differ in both capacity and data, including visual devices observing different categories, environments, and acquisition conditions\. A central goal is to let these clients benefit from shared knowledge within the capacity they can afford\.
However, acquiring knowledge at the server does not ensure that this knowledge reaches its recipients\. After aggregation, the server must compress its update into a low\-rank adapter for each client\. A widely used solution retains the largest singular components up to the client’s rank\[[1](https://arxiv.org/html/2611.23742#bib.bib3),[14](https://arxiv.org/html/2611.23742#bib.bib7),[8](https://arxiv.org/html/2611.23742#bib.bib8)\]\. This is optimal for reconstructing the update matrix and naturally accommodates different capacities\. It also makes a consequential choice: directions are ranked by their global matrix energy, independently of the inputs on which a particular recipient will use them\.
*The strongest directions in the matrix need not be the most influential directions for the receiver\.*A matrix direction contributes to a model through its interaction with the input\. A large singular component can have little effect on a client whose inputs barely activate it; a smaller component can be essential for reproducing the server update on another client’s data\. Compression may therefore discard useful responses even when the corresponding directions are already present in the aggregate\. Equal\-rank clients then receive identical views of shared knowledge despite different needs\. The bottleneck lies in which directions survive redistribution, and becomes especially important when only a few can be retained\.
Figure[1](https://arxiv.org/html/2611.23742#S1.F1)contrasts these choices at the same rank\-one budget\. Spectral readout keeps a common matrix direction for both receivers, whereas receiver\-aware readout adapts the retained direction to their response geometry\. More generally, our two\-direction analysis shows that the ratio between spectral and receiver\-optimal response errors can be arbitrarily large\. Thus, allocating sufficient capacity and allocating the right information are separate requirements\. This motivates our central question:*how should a server use each client’s limited rank to deliver the knowledge that matters on its data?*
Figure 1:The same rank preserves different amounts of receiver information\.In this analytical example, dark filled dots are responsesMxMx, hollow dots are their rank\-one projectionsPMxPMx, and dashed segments show discarded responses\. Spectral readout uses a common direction; RASA adapts to each receiver\. Lost response is normalized squared projection error\. Images indicate illustrative domains; the construction is given in the supplement\.Our key insight is that the value of a low\-rank readout should be measured through the responses it preserves for its recipient\. Given server updateMM, a client with inputsxxshould retain a matrixCiC\_\{i\}that keepsCixC\_\{i\}xclose toMxMx\. This suggests selecting the principal directions of the server update’s responses on that receiver\. Compact diagonal input second moments estimate this geometry, while fidelity to the complete aggregate provides a global reference\. The resulting uncentered principal component analysis \(PCA\) gives the global rank\-constrained optimum in server coordinates\. The rank budget determines how many directions survive; the receiver’s data determine which directions they are\.
Receiver\-aware distribution also participates in a feedback loop\. A trained adapter contains both the initialization selected for that client and the increment learned from local data\. Direct aggregation carries both into the shared state, so changing the readout changes the next aggregate even when clients perform no local training\. We connect personalized readout to continued collaboration through a reference\-consistency principle: client\-specific initializations shape local learning, while the server’s spectral transition anchors the returned contributions\. This gives the outgoing readout and the returning update complementary roles within one federated process\.
We developReceiver\-Aware Subspace Adaptation \(RASA\)around these two linked operations\. First, RASA uses compact local input statistics to extract a response\-preserving adapter from the server update at the client’s fixed rank\. Second, RASA aggregates the locally learned increment using the corresponding spectral initialization as a reference\. Clients train from their personalized readouts, while the server removes their direct distribution offsets on return\. This allows personalization to shape what clients learn and pass back to the federation\. RASA operates as a plug\-in to SVD\-based methods, retaining ordinary LoRA training and the assigned client ranks\.
Our contributions are summarized as follows:
- •We identify receiver\-response loss as a redistribution bottleneck, showing why matrix\-optimal truncation can discard client\-relevant directions\.
- •We propose RASA, coupling compact response PCA with spectral\-reference aggregation\. We prove readout optimality under the diagonal receiver metric and reference consistency without local learning, and analyze sensitivity to estimated statistics\.
- •We conduct extensive experiments on visual and language benchmarks, demonstrating the effectiveness of RASA across diverse baselines under data, resource, and task heterogeneity\.
## 2Related Work
Resource adaptation and aggregation\.HetLoRA uses heterogeneous ranks and local pruning\[[4](https://arxiv.org/html/2611.23742#bib.bib4)\], while FlexLoRA redistributes aggregated updates by spectral truncation\[[1](https://arxiv.org/html/2611.23742#bib.bib3)\]\. FLoRA stacks factors to preserve update products\[[22](https://arxiv.org/html/2611.23742#bib.bib5)\]; FedEx\-LoRA corrects factor\-aggregation error\[[16](https://arxiv.org/html/2611.23742#bib.bib6)\]\. ECLoRA combines randomized decomposition with error compensation\[[14](https://arxiv.org/html/2611.23742#bib.bib7)\], and RB\-LoRA balances rank\-wise contributions\[[8](https://arxiv.org/html/2611.23742#bib.bib8)\]\. LoRA\-A2alternates factor freezing and adapts ranks\[[10](https://arxiv.org/html/2611.23742#bib.bib21)\]; LoRA\-FAIR jointly refines aggregation and initialization\[[2](https://arxiv.org/html/2611.23742#bib.bib20)\]\. RASA selects receiver\-dependent responses at fixed ranks and corrects the resulting initialization offset\.
Personalized federated subspaces\.Client\-specific subspaces can exploit historical update geometry\[[3](https://arxiv.org/html/2611.23742#bib.bib9)\], personalized spectral selection\[[19](https://arxiv.org/html/2611.23742#bib.bib10)\], activation\-insensitive directions\[[24](https://arxiv.org/html/2611.23742#bib.bib11)\], or shared input\-aware action geometry\[[6](https://arxiv.org/html/2611.23742#bib.bib12)\]\. RASA studies which responses of a given server update should survive each recipient’s rank budget, and couples this readout with spectral\-reference aggregation\. The supplement provides a mechanism\-level comparison and detailed discussion\.
Activation\-aware compression\.ASVD uses activation information to rescale weights\[[23](https://arxiv.org/html/2611.23742#bib.bib13)\], and SVD\-LLM uses whitening to connect spectral truncation with output reconstruction\[[20](https://arxiv.org/html/2611.23742#bib.bib14)\]\. RASA brings response preservation to repeated, receiver\-specific redistribution under fixed client budgets\. A shared aggregate is read through multiple local metrics, and the resulting initializations participate in subsequent federated optimization\. Compact readout and reference correction provide a unified treatment of these two stages\.
## 3What Survives Low\-Rank Redistribution?
The information retained by a low\-rank matrix and the responses retained for a receiver are different objects\. We formalize this distinction and use it to specify what a client should obtain from the server\.
Federated LoRA\.ConsiderNNclients with local data distributions𝒟i\\mathcal\{D\}\_\{i\}and rank budgetsrir\_\{i\}\. At roundtt, the active set is𝒜t\\mathcal\{A\}\_\{t\}, with normalized aggregation weightspitp\_\{i\}^\{t\}\. For one adapted layer, the frozen weight isW0∈ℝm×dW\_\{0\}\\in\\mathbb\{R\}^\{m\\times d\}and the effective LoRA update issiBiAis\_\{i\}B\_\{i\}A\_\{i\}, whereBi∈ℝm×riB\_\{i\}\\in\\mathbb\{R\}^\{m\\times r\_\{i\}\},Ai∈ℝri×dA\_\{i\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times d\}, andsi\>0s\_\{i\}\>0is the client’s LoRA scale\. Throughout,MM,CiC\_\{i\},SiS\_\{i\}, andWiW\_\{i\}denote*effective update matrices*, including this scale and excludingW0W\_\{0\}\. We suppress the round and layer indices where unambiguous\.
What spectral readout optimizes\.LetM∈ℝm×dM\\in\\mathbb\{R\}^\{m\\times d\}be the server update at the distribution interface\. By the Eckart–Young theorem\[[7](https://arxiv.org/html/2611.23742#bib.bib22)\], spectral readout returns
Si=SVDri\(M\)∈argminrank\(C\)≤ri‖M−C‖F2\.S\_\{i\}=\\operatorname\{SVD\}\_\{r\_\{i\}\}\(M\)\\in\\argmin\_\{\\operatorname\{rank\}\(C\)\\leq r\_\{i\}\}\\left\\\|M\-C\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(1\)For equal ranks and a fixed tie\-breaking rule, this produces identical readouts across clients\. Equation \([1](https://arxiv.org/html/2611.23742#S3.E1)\) depends on the recipient only throughrir\_\{i\}\.
What the recipient experiences\.Letx∈ℝdx\\in\\mathbb\{R\}^\{d\}denote a layer input under a fixed reference forward pass on local data\. The error in reproducing the server update is
ℒiresp\(C\)\\displaystyle\\mathcal\{L\}^\{\\mathrm\{resp\}\}\_\{i\}\(C\)=𝔼x∼𝒟i‖\(M−C\)x‖22\\displaystyle=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{i\}\}\\\|\(M\-C\)x\\\|\_\{2\}^\{2\}=tr\[\(M−C\)Σi\(M−C\)⊤\],\\displaystyle=\\operatorname\{tr\}\\big\[\(M\-C\)\\Sigma\_\{i\}\(M\-C\)^\{\\top\}\\big\],\(2\)whereΣi=𝔼\[xx⊤\]\\Sigma\_\{i\}=\\mathbb\{E\}\[xx^\{\\top\}\]is an*uncentered*second moment\. For vision transformers,xxis a patch or class\-token representation at the adapted projection; for language models, it is a valid token representation\. Holding these inputs fixed isolates the immediate effect of compression at that layer\.
The metric can change the preferred subspace substantially\. ForM=diag\(a,b\)M=\\operatorname\{diag\}\(a,b\),a\>b\>0a\>b\>0, and rank one, spectral readout retains the first direction\. A receiver withΣi=diag\(ϵ,1\)\\Sigma\_\{i\}=\\operatorname\{diag\}\(\\epsilon,1\)incurs response errorb2b^\{2\}\. Retaining the second direction instead givesa2ϵa^\{2\}\\epsilon\. For0<ϵ<b2/a20<\\epsilon<b^\{2\}/a^\{2\}, the latter is response\-optimal, so the ratio isb2/\(a2ϵ\)b^\{2\}/\(a^\{2\}\\epsilon\)and grows without bound asϵ→0\\epsilon\\rightarrow 0\. Even an optimal matrix approximation can therefore be arbitrarily inefficient at preserving a receiver’s responses under the same rank budget\. The direction the receiver needs is present inMM; it is lost at redistribution\.
A regularized receiver objective\.To obtain a compact statistic, RASA usesDi=diag\(𝔼\[x12\],…,𝔼\[xd2\]\)D\_\{i\}=\\operatorname\{diag\}\(\\mathbb\{E\}\[x\_\{1\}^\{2\}\],\\ldots,\\mathbb\{E\}\[x\_\{d\}^\{2\}\]\)\. WriteℒiD\(C\)=tr\[\(M−C\)Di\(M−C\)⊤\]\\mathcal\{L\}\_\{i\}^\{D\}\(C\)=\\operatorname\{tr\}\[\(M\-C\)D\_\{i\}\(M\-C\)^\{\\top\}\]for the resulting diagonal response proxy\. We optimize
minrank\(C\)≤ri𝒥i\(C\),𝒥i\(C\)=tr\[\(M−C\)Gi\(M−C\)⊤\],\\min\_\{\\operatorname\{rank\}\(C\)\\leq r\_\{i\}\}\\mathcal\{J\}\_\{i\}\(C\),\\quad\\mathcal\{J\}\_\{i\}\(C\)=\\operatorname\{tr\}\\big\[\(M\-C\)G\_\{i\}\(M\-C\)^\{\\top\}\\big\],\(3\)whereGi=Di\+λIdG\_\{i\}=D\_\{i\}\+\\lambda I\_\{d\}andλ≥0\\lambda\\geq 0\. TheDiD\_\{i\}term prioritizes active input directions, whileλ‖M−C‖F2\\lambda\\left\\\|M\-C\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}preserves fidelity to the complete server update whenλ\>0\\lambda\>0\. The settingλ=0\\lambda=0, used in our visual experiments, retains the response objective alone\. The diagonal approximation is exact when input cross moments vanish; its effect otherwise is quantified in Sec\.[5](https://arxiv.org/html/2611.23742#S5)\. The target remains the complete aggregateMM, so the feasible readout is not restricted to its leading rank\-rir\_\{i\}spectral subspace\.
## 4Receiver\-Aware Subspace Adaptation
RASA uses each client’s fixed capacity to preserve the aggregate’s responses on its data, then carries the resulting local learning back to the federation\. These two stages are implemented by receiver\-specific readout and spectral\-reference aggregation, respectively\. Figure[2](https://arxiv.org/html/2611.23742#S4.F2)illustrates their interaction\.
Figure 2:The federated loop of RASA\.The server distributes receiver\-specificCiC\_\{i\}from sharedMMand cached statisticsDiD\_\{i\}\. Clients train locally and uploadWi=Ci\+ΔiW\_\{i\}=C\_\{i\}\+\\Delta\_\{i\}\. The method using product averaging corrects each return toSi\+ΔiS\_\{i\}\+\\Delta\_\{i\}and aggregates the corrected updates into the nextMM, preserving learned increments throughout the loop\.### 4\.1Compact Receiver Statistics
Each client estimates the diagonal second moment using at most 32 training examples and 4,096 valid layer\-input tokens per layer\. The token budget is distributed as evenly as possible across examples; valid positions are then sampled within each example, excluding padding\. The estimator is
d^i=1nitok∑x∈𝒳ix⊙x,Di=diag\(d^i\)\.\\widehat\{d\}\_\{i\}=\\frac\{1\}\{n\_\{i\}^\{\\mathrm\{tok\}\}\}\\sum\_\{x\\in\\mathcal\{X\}\_\{i\}\}x\\odot x,\\qquad D\_\{i\}=\\operatorname\{diag\}\(\\widehat\{d\}\_\{i\}\)\.\(4\)Thus each refresh transmitsddscalars per layer\. The server caches the latest vector and its timestamp for subsequent readouts\. A client without a cached statistic receives the server’s spectral readout\.
### 4\.2PCA of Receiver Responses
Factor the distribution target as
M=UT,U⊤U=Ik,M=UT,\\qquad U^\{\\top\}U=I\_\{k\},\(5\)whereU∈ℝm×kU\\in\\mathbb\{R\}^\{m\\times k\}spans the columns ofMMandT∈ℝk×dT\\in\\mathbb\{R\}^\{k\\times d\}\. The server can supply this representation directly, or it can be obtained by reduced QR from stacked low\-rank factors\. All receiver readouts operate on this distribution target\.
An inputxxproduces the server responseMx=U\(Tx\)Mx=U\(Tx\)\. SinceUUpreserves Euclidean lengths, the responses can be studied through their coordinatesTxTx\. Their uncentered second moment isTΣiT⊤T\\Sigma\_\{i\}T^\{\\top\}\. RASA approximates this response geometry withTDiT⊤TD\_\{i\}T^\{\\top\}and adds the matrix\-fidelity term\. Definer¯i=min\(ri,k\)\\bar\{r\}\_\{i\}=\\min\(r\_\{i\},k\)and construct
Hi=TGiT⊤=TDiT⊤\+λTT⊤\.H\_\{i\}=TG\_\{i\}T^\{\\top\}=TD\_\{i\}T^\{\\top\}\+\\lambda TT^\{\\top\}\.\(6\)LetQi∈ℝk×r¯iQ\_\{i\}\\in\\mathbb\{R\}^\{k\\times\\bar\{r\}\_\{i\}\}contain orthonormal eigenvectors associated with its largest eigenvalues\. RASA returns
Ci=UQiQi⊤T\.C\_\{i\}=UQ\_\{i\}Q\_\{i\}^\{\\top\}T\.\(7\)
###### Theorem 1\(Optimal receiver readout\)\.
For anyGi⪰0G\_\{i\}\\succeq 0, Eq\. \([7](https://arxiv.org/html/2611.23742#S4.E7)\) is a global minimizer of Eq\. \([3](https://arxiv.org/html/2611.23742#S3.E3)\) over all matrices of rank at mostrir\_\{i\}\. Ifμ1≥⋯≥μk≥0\\mu\_\{1\}\\geq\\cdots\\geq\\mu\_\{k\}\\geq 0are the eigenvalues ofHiH\_\{i\}, its optimal value is∑j\>r¯iμj\\sum\_\{j\>\\bar\{r\}\_\{i\}\}\\mu\_\{j\}\.
###### Proof\.
LetZ=MGi1/2Z=MG\_\{i\}^\{1/2\}\. Every feasibleCCsatisfiesrank\(CGi1/2\)≤ri\\operatorname\{rank\}\(CG\_\{i\}^\{1/2\}\)\\leq r\_\{i\}, so its objective is bounded below by the best rank\-rir\_\{i\}approximation error ofZZ\. SinceZZ⊤=UHiU⊤ZZ^\{\\top\}=UH\_\{i\}U^\{\\top\}, the projectorP∗=UQiQi⊤U⊤P\_\{\*\}=UQ\_\{i\}Q\_\{i\}^\{\\top\}U^\{\\top\}retains a leading left singular subspace ofZZ\. The choiceC=P∗M=UQiQi⊤TC=P\_\{\*\}M=UQ\_\{i\}Q\_\{i\}^\{\\top\}Tattains the bound becauseCGi1/2=P∗ZCG\_\{i\}^\{1/2\}=P\_\{\*\}Z\. The residual is∑j\>r¯iμj\\sum\_\{j\>\\bar\{r\}\_\{i\}\}\\mu\_\{j\}\. ∎
RASA therefore selects the principal directions of receiver\-weighted update responses\. More precisely,Hi=YiYi⊤H\_\{i\}=Y\_\{i\}Y\_\{i\}^\{\\top\}forYi=\[TDi1/2,λT\]Y\_\{i\}=\[TD\_\{i\}^\{1/2\},\\,\\sqrt\{\\lambda\}T\]: the readout is uncentered PCA with a fidelity reference, evaluated without formingYiY\_\{i\}\. The theorem needs no inverse and covers both positive fidelity regularization and the response\-only settingλ=0\\lambda=0\. Even diagonalDiD\_\{i\}generally yields a denseHiH\_\{i\}, allowing rotated combinations of server directions\. A positive isotropicGiG\_\{i\}recovers spectral readout, and increasingλ\\lambdaapproaches it when the spectral boundary has an eigengap\.
### 4\.3Balanced LoRA Instantiation
Write the small readout coordinates asQi⊤T=LiΓiRi⊤Q\_\{i\}^\{\\top\}T=L\_\{i\}\\Gamma\_\{i\}R\_\{i\}^\{\\top\}\. On its nonzero singular components, set
Bi0=si−1/2UQiLiΓi1/2,Ai0=si−1/2Γi1/2Ri⊤\.B\_\{i\}^\{0\}=s\_\{i\}^\{\-1/2\}UQ\_\{i\}L\_\{i\}\\Gamma\_\{i\}^\{1/2\},\\quad A\_\{i\}^\{0\}=s\_\{i\}^\{\-1/2\}\\Gamma\_\{i\}^\{1/2\}R\_\{i\}^\{\\top\}\.\(8\)ThensiBi0Ai0=Cis\_\{i\}B\_\{i\}^\{0\}A\_\{i\}^\{0\}=C\_\{i\}, and both factors share the singular\-value scale\. The client trains these ordinary LoRA factors using the client’s local optimizer\. IfCiC\_\{i\}is zero or rank\-deficient, unused directions receive a one\-sided nonzero, zero\-product initialization so that they remain trainable; the supplement specifies this edge case\.
### 4\.4Spectral\-Reference Aggregation
For the same server state and client rank, retain bothCiC\_\{i\}and the referenceSi=SVDri\(M\)S\_\{i\}=\\operatorname\{SVD\}\_\{r\_\{i\}\}\(M\)\. Local training returnsWi=Ci\+ΔiW\_\{i\}=C\_\{i\}\+\\Delta\_\{i\}\. Direct averaging obeys
∑ipiWi=∑ipi\(Si\+Δi\)\+∑ipi\(Ci−Si\)⏟direct distribution offset\.\\sum\_\{i\}p\_\{i\}W\_\{i\}=\\sum\_\{i\}p\_\{i\}\(S\_\{i\}\+\\Delta\_\{i\}\)\+\\underbrace\{\\sum\_\{i\}p\_\{i\}\(C\_\{i\}\-S\_\{i\}\)\}\_\{\\text\{direct distribution offset\}\}\.\(9\)The last term is introduced by distribution before any client learns\. RASA removes it by forming
W~i=Wi−Ci\+Si=Si\+Δi\.\\widetilde\{W\}\_\{i\}=W\_\{i\}\-C\_\{i\}\+S\_\{i\}=S\_\{i\}\+\\Delta\_\{i\}\.\(10\)For a method using product\-space weighted averaging,
Mnext=∑i∈𝒜tpit\[Si\+\(Wi−Ci\)\]\.M\_\{\\mathrm\{next\}\}=\\sum\_\{i\\in\\mathcal\{A\}\_\{t\}\}p\_\{i\}^\{t\}\\big\[S\_\{i\}\+\(W\_\{i\}\-C\_\{i\}\)\\big\]\.\(11\)The incrementΔi\\Delta\_\{i\}is still learned fromCiC\_\{i\}\. Therefore, personalization affects the server through the local optimization trajectory, while its direct initialization displacement is removed\. The reference transition is∑ipiSi\\sum\_\{i\}p\_\{i\}S\_\{i\}, which incorporates the server’s rank\-dependent truncation ofMM\.
Correction in shared server coordinates\.Both initializations come from the same target:Ci=UPiTC\_\{i\}=UP\_\{i\}TwithPi=QiQi⊤P\_\{i\}=Q\_\{i\}Q\_\{i\}^\{\\top\}, andSi=UPiSTS\_\{i\}=UP\_\{i\}^\{S\}TwithPiSP\_\{i\}^\{S\}the leading rank\-r¯i\\bar\{r\}\_\{i\}projector ofTT⊤TT^\{\\top\}\. For a spectral fallback, setPi=PiSP\_\{i\}=P\_\{i\}^\{S\}\. Their offsets can therefore be combined before aggregation:
K\\displaystyle K=∑i∈𝒜tpit\(PiS−Pi\),\\displaystyle=\\sum\_\{i\\in\\mathcal\{A\}\_\{t\}\}p\_\{i\}^\{t\}\(P\_\{i\}^\{S\}\-P\_\{i\}\),\(12\)Mnext\\displaystyle M\_\{\\mathrm\{next\}\}=∑i∈𝒜tpitWi\+UKT\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{A\}\_\{t\}\}p\_\{i\}^\{t\}W\_\{i\}\+UKT\.This evaluates the same corrected aggregate as Eq\. \([11](https://arxiv.org/html/2611.23742#S4.E11)\) in common coordinates\. The supplement gives the factorized implementation and caching requirements\.
Integration with existing methods\.FlexLoRA applies Eq\. \([11](https://arxiv.org/html/2611.23742#S4.E11)\) before its next decomposition\. For ECLoRA, the corrected incoming aggregate is passed to ECLoRA’s server compression and error\-compensation procedure; the same\-state spectral reference is defined at its distribution interface\. RB\-LoRA assigns different weights to rank columns\. Letℱi\(B,A\)=siBΩiA\\mathcal\{F\}\_\{i\}\(B,A\)=s\_\{i\}B\\Omega\_\{i\}Adenote its weighted contribution, withΩi\\Omega\_\{i\}computed from the original participating client ranks and data counts\. We use
ℱi\(Bi\+,Ai\+\)−ℱi\(Bi0,Ai0\)\+ℱi\(BiS,AiS\)\.\\mathcal\{F\}\_\{i\}\(B\_\{i\}^\{\+\},A\_\{i\}^\{\+\}\)\-\\mathcal\{F\}\_\{i\}\(B\_\{i\}^\{0\},A\_\{i\}^\{0\}\)\+\\mathcal\{F\}\_\{i\}\(B\_\{i\}^\{S\},A\_\{i\}^\{S\}\)\.\(13\)All three terms use the same column weights\. We store the actual dispatched factors and keep their column identities through local training\. Supplementary Sec\. B gives the weights, the corresponding shared\-core form, and a signed\-factor representation\.
Algorithm 1RASA for a product\-average aggregationInput:Server state, ranks\{ri\}\\\{r\_\{i\}\\\}, cached moments,λ≥0\\lambda\\geq 0\. For each communication roundtt:
1. 1\.Obtain distribution targetMt=UtTtM^\{t\}=U^\{t\}T^\{t\}; select𝒜t\\mathcal\{A\}\_\{t\}\.
2. 2\.For eachi∈𝒜ti\\in\\mathcal\{A\}\_\{t\}, setSit←SVDri\(Mt\)S\_\{i\}^\{t\}\\leftarrow\\operatorname\{SVD\}\_\{r\_\{i\}\}\(M^\{t\}\)\.
3. 3\.If moments are cached, formHiH\_\{i\}by Eq\. \([6](https://arxiv.org/html/2611.23742#S4.E6)\) andCitC\_\{i\}^\{t\}by Eq\. \([7](https://arxiv.org/html/2611.23742#S4.E7)\); otherwise setCit←SitC\_\{i\}^\{t\}\\leftarrow S\_\{i\}^\{t\}\.
4. 4\.Cache the actual dispatched initialization and reference\.
5. 5\.Send factors forCitC\_\{i\}^\{t\}; receive locally trainedWitW\_\{i\}^\{t\}\.
6. 6\.Update the moment cache if a refresh is returned\.
7. 7\.FormM~t←∑i∈𝒜tpit\(Wit−Cit\+Sit\)\\widetilde\{M\}^\{t\}\\leftarrow\\sum\_\{i\\in\\mathcal\{A\}\_\{t\}\}p\_\{i\}^\{t\}\(W\_\{i\}^\{t\}\-C\_\{i\}^\{t\}\+S\_\{i\}^\{t\}\), equivalently by Eq\. \([12](https://arxiv.org/html/2611.23742#S4.E12)\)\.
8. 8\.Apply the baseline server processing toM~t\\widetilde\{M\}^\{t\}\.
Table 1:Visual classification with FlexLoRA\.Final\-round accuracy \(%\), mean±\\pmstd over three runs\. ViT\-B/16,λ=0\\lambda=0; 50/30 rounds on DomainNet\-100/NICO\+\+\. Avg\. weights the six domains equally; shading marks RASA\.Table 2:Language understanding on GLUE\.Accuracy \(%\), mean±\\pmstd over three runs\. MNLI\-mm uses the MNLI\-m checkpoint\. Average weights five scores equally;Δ\\Deltasubtracts displayed means \(pp\)\. Bold marks the better mean\.
## 5Properties and Computational Cost
Response preservation at the same rank\.Optimality for the regularized objective also gives a guarantee for the unregularized response proxyℒiD\\mathcal\{L\}\_\{i\}^\{D\}\.
###### Corollary 1\(Response preservation over spectral readout\)\.
LetCiC\_\{i\}be the RASA readout andSi=SVDri\(M\)S\_\{i\}=\\operatorname\{SVD\}\_\{r\_\{i\}\}\(M\)\. Then
ℒiD\(Si\)−ℒiD\(Ci\)\\displaystyle\\mathcal\{L\}\_\{i\}^\{D\}\(S\_\{i\}\)\-\\mathcal\{L\}\_\{i\}^\{D\}\(C\_\{i\}\)≥λ\[‖M−Ci‖F2−‖M−Si‖F2\]\\displaystyle\\geq\\lambda\\big\[\\left\\\|M\-C\_\{i\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\-\\left\\\|M\-S\_\{i\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\\big\]\(14\)≥0\.\\displaystyle\\geq 0\.
###### Proof\.
CompareCiC\_\{i\}with the feasibleSiS\_\{i\}in Eq\. \([3](https://arxiv.org/html/2611.23742#S3.E3)\), use the Frobenius optimality ofSiS\_\{i\}in Eq\. \([1](https://arxiv.org/html/2611.23742#S3.E1)\), and rearrange\. ∎
Thus the fidelity term does not sacrifice response preservation relative to spectral readout under the same diagonal metric\. Transfer from this proxy to the full receiver response depends on the statistic approximation considered below\.
A measurable advantage over spectral readout\.LetP∗=QiQi⊤P\_\{\*\}=Q\_\{i\}Q\_\{i\}^\{\\top\}and letPSP\_\{S\}be the rank\-rir\_\{i\}projector selected by ordinary SVD in the same server coordinates, assumingri<kr\_\{i\}<k\. For any orthogonal projectorPP,
𝒥i\(UPT\)=tr\(Hi\)−tr\(PHi\)\.\\mathcal\{J\}\_\{i\}\(UPT\)=\\operatorname\{tr\}\(H\_\{i\}\)\-\\operatorname\{tr\}\(PH\_\{i\}\)\.\(15\)Consequently,𝒥i\(Si\)−𝒥i\(Ci\)=tr\[\(P∗−PS\)Hi\]≥0\\mathcal\{J\}\_\{i\}\(S\_\{i\}\)\-\\mathcal\{J\}\_\{i\}\(C\_\{i\}\)=\\operatorname\{tr\}\[\(P\_\{\*\}\-P\_\{S\}\)H\_\{i\}\]\\geq 0\. Ifγi=μri−μri\+1\>0\\gamma\_\{i\}=\\mu\_\{r\_\{i\}\}\-\\mu\_\{r\_\{i\}\+1\}\>0, then
𝒥i\(Si\)−𝒥i\(Ci\)≥γi2‖P∗−PS‖F2\.\\mathcal\{J\}\_\{i\}\(S\_\{i\}\)\-\\mathcal\{J\}\_\{i\}\(C\_\{i\}\)\\geq\\frac\{\\gamma\_\{i\}\}\{2\}\\left\\\|P\_\{\*\}\-P\_\{S\}\\right\\\|\_\{\\mathrm\{F\}\}^\{2\}\.\(16\)Together with Corollary[1](https://arxiv.org/html/2611.23742#Thmcorollary1), the inequality gives strict improvement in the diagonal response proxy whenever the leading subspaces differ and the receiver eigengap is positive\. It supplies a layerwise diagnostic of response preservation\. Section[6\.3](https://arxiv.org/html/2611.23742#S6.SS3)examines response error alongside task NLL to study their empirical relationship\.
Effect of diagonal and stale statistics\.LetGi⋆=Σicurrent\+λIG\_\{i\}^\{\\star\}=\\Sigma\_\{i\}^\{\\mathrm\{current\}\}\+\\lambda Idescribe the current full second moment, and letG^i=D^icached\+λI\\widehat\{G\}\_\{i\}=\\widehat\{D\}\_\{i\}^\{\\mathrm\{cached\}\}\+\\lambda I\. LetC^i\\widehat\{C\}\_\{i\}be the readout obtained withG^i\\widehat\{G\}\_\{i\}, andCi⋆C\_\{i\}^\{\\star\}the optimum underGi⋆G\_\{i\}^\{\\star\}\. Define
Ei=T\(G^i−Gi⋆\)T⊤\.E\_\{i\}=T\(\\widehat\{G\}\_\{i\}\-G\_\{i\}^\{\\star\}\)T^\{\\top\}\.\(17\)The variational characterization of the leading eigenspace yields
𝒥i⋆\(C^i\)−𝒥i⋆\(Ci⋆\)≤2r¯i‖Ei‖2\.\\mathcal\{J\}\_\{i\}^\{\\star\}\(\\widehat\{C\}\_\{i\}\)\-\\mathcal\{J\}\_\{i\}^\{\\star\}\(C\_\{i\}^\{\\star\}\)\\leq 2\\bar\{r\}\_\{i\}\\left\\\|E\_\{i\}\\right\\\|\_\{2\}\.\(18\)Moreover,‖Ei‖2≤‖T‖22‖D^icached−Σicurrent‖2\\left\\\|E\_\{i\}\\right\\\|\_\{2\}\\leq\\left\\\|T\\right\\\|\_\{2\}^\{2\}\\left\\\|\\widehat\{D\}\_\{i\}^\{\\mathrm\{cached\}\}\-\\Sigma\_\{i\}^\{\\mathrm\{current\}\}\\right\\\|\_\{2\}\. The latter difference separates into diagonal sampling error, temporal drift, and omitted cross moments\. Thus a small upload is useful when it preserves the moment geometry relevant toTT; increasing the token count alone cannot remove diagonal\-approximation error\. Proofs and this decomposition are in the supplement\.
Zero\-local\-update consistency\.If clients perform no local update,Wi=CiW\_\{i\}=C\_\{i\}and Eq\. \([10](https://arxiv.org/html/2611.23742#S4.E10)\) givesW~i=Si\\widetilde\{W\}\_\{i\}=S\_\{i\}\. Therefore the corrected incoming aggregate is exactly the baseline spectral\-reference aggregate from the same state\. The claim also holds under Eq\. \([13](https://arxiv.org/html/2611.23742#S4.E13)\), where the trained factors equal the dispatched factors, and with identical downstream server processing\. Without correction, the difference is precisely the last term in Eq\. \([9](https://arxiv.org/html/2611.23742#S4.E9)\)\.
Communication and computation\.Per layer, a refreshed statistic addsddscalars to the usualri\(m\+d\)r\_\{i\}\(m\+d\)adapter\-upload scalars\. If both use the same precision andm=dm=d, the relative upload overhead is1/\(2ri\)1/\(2r\_\{i\}\)per refresh, amortized by the refresh interval\. Downlink rank and the number of locally trained adapter parameters are unchanged\. After sharingUU,TT, andTT⊤TT^\{\\top\}across clients, a dense construction ofHiH\_\{i\}costsO\(k2d\)O\(k^\{2\}d\)and a full eigendecomposition costsO\(k3\)O\(k^\{3\}\)per receiver\. Instantiation costsO\(mkri\+krid\+ri2d\)O\(mkr\_\{i\}\+kr\_\{i\}d\+r\_\{i\}^\{2\}d\)\. The full cost also includes obtaining the server representation and the statistic\-collection forward pass\.
LetRt=∑i∈𝒜triR\_\{t\}=\\sum\_\{i\\in\\mathcal\{A\}\_\{t\}\}r\_\{i\}\. Direct signed stacks use at most3Rt3R\_\{t\}columns\. The shared\-core form in Eq\. \([12](https://arxiv.org/html/2611.23742#S4.E12)\) admits an alternative bound ofRt\+rank\(K\)≤Rt\+min\(k,2Rt\)R\_\{t\}\+\\operatorname\{rank\}\(K\)\\leq R\_\{t\}\+\\min\(k,2R\_\{t\}\)\. Both represent the same aggregate, while each client still trains and transmits rankrir\_\{i\}\.
## 6Experiments
Table 3:SuperNI after 30 federated rounds\.Free\-generation ROUGE\-L, mean±\\pmstd over three runs\. Seen/unseen: 50/20 clients; unseen uses spectral readout\.Δ\\Delta: paired mean gain over Pure\.Our comprehensive evaluation covers visual classification, language understanding, and instruction following under data heterogeneity \(domains and label skew\), resource heterogeneity \(client ranks\), and task heterogeneity \(seen/unseen instructions\)\. We test compatibility with FlexLoRA, ECLoRA, and RB\-LoRA through benchmark, ablation, mechanism, sensitivity, and cost studies\.
### 6\.1Experimental Setup
Our benchmarks are DomainNet\-100\[[15](https://arxiv.org/html/2611.23742#bib.bib18)\], NICO\+\+\[[25](https://arxiv.org/html/2611.23742#bib.bib19)\], GLUE\[[18](https://arxiv.org/html/2611.23742#bib.bib15)\], and SuperNI\[[21](https://arxiv.org/html/2611.23742#bib.bib16)\]\. RASA preserves each method’s client ranks and aggregation rule\. Main visual comparisons use FlexLoRA; language and diagnostic studies also cover ECLoRA and RB\-LoRA\.
The main visual benchmarks use ViT\-B/16\[[5](https://arxiv.org/html/2611.23742#bib.bib17)\],λ=0\\lambda=0, and 50/30 rounds on DomainNet\-100/NICO\+\+, reporting final accuracy over six domains\. GLUE classification uses RoBERTa\-base\[[12](https://arxiv.org/html/2611.23742#bib.bib24)\], three clients, and Dirichletα=0\.1\\alpha=0\.1\. SuperNI instruction generation uses a 1B\-parameterLLaMA\[[17](https://arxiv.org/html/2611.23742#bib.bib25)\]model, reporting round\-30 ROUGE\-L\[[11](https://arxiv.org/html/2611.23742#bib.bib23)\]on 50 seen/20 unseen clients under normal, uniform, and heavy\-tail ranks; unseen tasks use spectral readout\. Tables[1](https://arxiv.org/html/2611.23742#S4.T1)–[3](https://arxiv.org/html/2611.23742#S6.T3)report mean±\\pmstd over seeds 42–44\. Further protocols are in the supplement\.
### 6\.2Performance across Visual and Language Tasks
Visual classification\.Table[1](https://arxiv.org/html/2611.23742#S4.T1)shows gains across all six domains on both benchmarks\. On DomainNet\-100, RASA improves average accuracy from69\.1869\.18to72\.4072\.40, with a paired gain of3\.233\.23points\. Infograph and Sketch benefit most, gaining5\.575\.57and4\.414\.41points\. On NICO\+\+, average accuracy increases from89\.9489\.94to90\.3690\.36, with the largest gain of0\.830\.83points on Rock\. These results support adapting shared updates to heterogeneous visual distributions\.
Language tasks and method compatibility\.On GLUE \(Table[2](https://arxiv.org/html/2611.23742#S4.T2)\), RASA improves the averages of FlexLoRA, ECLoRA, and RB\-LoRA by1\.271\.27,0\.740\.74, and0\.440\.44points; thirteen of fifteen scores improve\. On seen SuperNI tasks \(Table[3](https://arxiv.org/html/2611.23742#S6.T3)\), it improves eight of nine comparisons\. FlexLoRA gains4\.564\.56,8\.338\.33, and4\.584\.58ROUGE\-L under normal, uniform, and heavy\-tail ranks\. ECLoRA improves under every profile, and RB\-LoRA gains up to3\.703\.70\.
RASA also improves eight of nine unseen\-task comparisons, including all profiles for FlexLoRA and RB\-LoRA\. FlexLoRA gains up to4\.374\.37ROUGE\-L\. Since unseen tasks use spectral readout, these gains demonstrate transfer through the shared model learned during RASA training\.
### 6\.3Ablation and Mechanism Analysis
Table 4:Component ablation on SuperNI\.Round\-30 ROUGE\-L \(RL,↑\\uparrow\) and loss \(↓\\downarrow\), normal rank profile\. The two ablated variants report single\-run results; Pure baseline and Full RASA report means across runs\.Table 5:Statistical approximations on GLUE\.Mean±\\pmstd over three seeds\. Diag \+ Mean sends variance and mean vectors\. Extra uplink is per client/layer/refresh, relative to rank\-8 FP32 LoRA \(48 KiB/layer\)\.Table 6:Resource overhead on DomainNet\-100\.ViT\-B/16, six clients, shared A100s,λ=0\\lambda=0; refresh each round\. Time: three\-run mean; memory: mean peak\. Communication is identical across methods\. Changes are relative to Pure\.Figure 3:Effect of receiver statistics on NICO\+\+ at rank eight\.Relative reductions from spectral readout in response error and task NLL, under the same seed\. Positive values indicate improvement\.Component contributions\.Table[4](https://arxiv.org/html/2611.23742#S6.T4)shows that balancing alone \(w/o receiver statistics\) improves over the baseline, while receiver\-aware readout further boosts seen\-task ROUGE\-L\. Update correction improves both seen and unseen ROUGE\-L over the uncorrected variant, with Full RASA achieving the best scores on both splits\.
Statistical approximation\.Table[5](https://arxiv.org/html/2611.23742#S6.T5)shows that Diag improves over Pure across all three methods, attaining the highest mean accuracy for ECLoRA\. Its mean accuracies remain close to Full while requiring384\.5×384\.5\\timesless statistics uplink \(3 vs\. 1153\.5 KiB per layer per refresh\)\. Adding mean statistics doubles the payload without consistent accuracy gains, supporting Diag as a practical accuracy–communication trade\-off\.
Receiver statistics\.Figure[3](https://arxiv.org/html/2611.23742#S6.F3)shows that receiver statistics give the largest reductions in both response error and NLL across all three methods on NICO\+\+ at rank eight\. In contrast, global and shuffled statistics reduce response error but worsen NLL for RB\-LoRA\. This highlights the importance of matching statistics to the recipient: lower response error alone does not guarantee better task performance\.
### 6\.4Sensitivity and Efficiency
Figure 4:Sensitivity on DomainNet\-100\.\(a\) Round\-50 test accuracy change fromλ=0\\lambda=0\(seed 42\)\. \(b\) Validation NLL change from fresh statistics atλ=0\.1\\lambda=0\.1, fixing the round\-50, rank\-32 reference\. Error bars: mean±\\pmstd over three runs\.Fidelity regularization\.Figure[4](https://arxiv.org/html/2611.23742#S6.F4)\(a\) evaluates round\-50 test accuracy on DomainNet\-100 withλ∈\{0,0\.1,1,10\}\\lambda\\in\\\{0,0\.1,1,10\\\}for ECLoRA and RB\-LoRA\. Both peak atλ=0\.1\\lambda=0\.1, gaining approximately0\.180\.18and0\.210\.21accuracy points overλ=0\\lambda=0\. Gains diminish asλ\\lambdaincreases to 1 and 10, while remaining positive\. A moderate fidelity term complements receiver adaptation; stronger regularization increasingly favors the shared spectral geometry\.
Staleness of cached activation statistics\.We fix the round\-50, rank\-32 reference atλ=0\.1\\lambda=0\.1and replay statistics from rounds 50, 40, and 30 on the same validation examples, isolating readout sensitivity to cache age\. Figure[4](https://arxiv.org/html/2611.23742#S6.F4)\(b\) shows nearly unchanged mean NLL for ECLoRA and reductions of approximately0\.5%0\.5\\%and0\.7%0\.7\\%for RB\-LoRA at ages 10 and 20\. Thus, older statistics retain useful receiver information for this fixed\-state readout\. This complements Eq\. \([18](https://arxiv.org/html/2611.23742#S5.E18)\), which relates readout error to drift in response geometry\.
Resource overhead\.Table[6](https://arxiv.org/html/2611.23742#S6.T6)measures DomainNet\-100 costs with six clients and statistics refreshed every round\. Recorded time increases by10\.64%10\.64\\%for FlexLoRA and2\.69%2\.69\\%for RB\-LoRA\. ECLoRA’s−2\.80%\-2\.80\\%difference occurs amid run\-to\-run variability on shared GPUs\. Peak process GPU memory is unchanged at the reported precision\. Statistics add0\.4220\.422MiB per round to the uplink, leaving downlink unchanged; total communication rises by only1\.20%1\.20\\%\. Less frequent refreshes amortize the additional statistic payload\.
## 7Conclusion
RASA couples response PCA with spectral\-reference aggregation to reduce receiver\-response loss in federated LoRA\. Compact statistics guide subspace selection at fixed rank, while reference correction preserves locally learned increments\. Theory establishes receiver\-objective optimality and zero\-local\-update consistency\. Experiments across vision and language tasks show gains under heterogeneity with modest communication overhead\.
## LLM Usage Statement
We used LLM\-based tools, primarily GPT6\-Astra and Codex, to assist with coding, polish the writing, and support self\-review of the manuscript\. We take full responsibility for the content\.
## References
- \[1\]J\. Bai, D\. Chen, B\. Qian, L\. Yao, and Y\. Li\(2024\)Federated fine\-tuning of large language models under heterogeneous tasks and client resources\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://arxiv.org/abs/2402.11505)Cited by:[§1](https://arxiv.org/html/2611.23742#S1.p1.1),[§1](https://arxiv.org/html/2611.23742#S1.p2.1),[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[2\]J\. Bian, L\. Wang, L\. Zhang, and J\. Xu\(2025\)LoRA\-FAIR: federated LoRA fine\-tuning with aggregation and initialization refinement\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3737–3746\.External Links:[Link](https://openaccess.thecvf.com/content/ICCV2025/html/Bian_LoRA-FAIR_Federated_LoRA_Fine-Tuning_with_Aggregation_and_Initialization_Refinement_ICCV_2025_paper.html)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[3\]J\. Chen, C\. Liu, and J\. Zhu\(2026\)Beyond factor aggregation: gauge\-aware low\-rank server representations for federated LoRA\.arXiv preprint arXiv:2605\.06733\.External Links:[Link](https://arxiv.org/abs/2605.06733)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p2.1)\.
- \[4\]Y\. J\. Cho, L\. Liu, Z\. Xu, A\. Fahrezi, and G\. Joshi\(2024\)Heterogeneous LoRA for federated fine\-tuning of on\-device foundation models\.InConference on Empirical Methods in Natural Language Processing,External Links:[Link](https://aclanthology.org/2024.emnlp-main.717/)Cited by:[§1](https://arxiv.org/html/2611.23742#S1.p1.1),[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[5\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby\(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2010.11929)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p2.1)\.
- \[6\]J\. Du, S\. He, and L\. Feng\(2026\)What should federated LoRA share? FedSAIL via input\-aware subspace alignment\.arXiv preprint arXiv:2609\.32485\.External Links:[Link](https://arxiv.org/abs/2609.32485)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p2.1)\.
- \[7\]C\. Eckart and G\. Young\(1936\)The approximation of one matrix by another of lower rank\.Psychometrika1\(3\),pp\. 211–218\.External Links:[Document](https://dx.doi.org/10.1007/BF02288367)Cited by:[§3](https://arxiv.org/html/2611.23742#S3.p3.1)\.
- \[8\]S\. Ha, Y\. Oh, and Y\. Jeon\(2026\)RB\-LoRA: rank\-balanced aggregation for low\-rank adaptation with federated fine\-tuning\.InFindings of the Association for Computational Linguistics: EACL,pp\. 1737–1746\.External Links:[Link](https://aclanthology.org/2026.findings-eacl.88/)Cited by:[§1](https://arxiv.org/html/2611.23742#S1.p2.1),[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[9\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§1](https://arxiv.org/html/2611.23742#S1.p1.1)\.
- \[10\]J\. Koo, M\. Jang, and J\. Ok\(2025\)Towards robust and efficient federated low\-rank adaptation with heterogeneous clients\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics,pp\. 416–429\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.19),[Link](https://aclanthology.org/2025.acl-long.19/)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[11\]C\. Lin\(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p2.1)\.
- \[12\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.External Links:[Link](https://arxiv.org/abs/1907.11692)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p2.1)\.
- \[13\]H\. B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y Arcas\(2017\)Communication\-efficient learning of deep networks from decentralized data\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1273–1282\.External Links:[Link](https://proceedings.mlr.press/v54/mcmahan17a.html)Cited by:[§1](https://arxiv.org/html/2611.23742#S1.p1.1)\.
- \[14\]W\. Ning, J\. Wang, Q\. Qi, H\. Sun, D\. Cheng, C\. Liu, L\. Zhang, Z\. Zhuang, and J\. Liao\(2025\)Federated fine\-tuning on heterogeneous LoRAs with error\-compensated aggregation\.IEEE Transactions on Neural Networks and Learning Systems\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2025.3586545),[Link](https://ieeexplore.ieee.org/document/11083124/)Cited by:[§1](https://arxiv.org/html/2611.23742#S1.p2.1),[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[15\]X\. Peng, Q\. Bai, X\. Xia, Z\. Huang, K\. Saenko, and B\. Wang\(2019\)Moment matching for multi\-source domain adaptation\.InIEEE/CVF International Conference on Computer Vision,pp\. 1406–1415\.External Links:[Link](https://openaccess.thecvf.com/content_ICCV_2019/html/Peng_Moment_Matching_for_Multi-Source_Domain_Adaptation_ICCV_2019_paper.html)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p1.1)\.
- \[16\]R\. Singhal, K\. Ponkshe, and P\. Vepakomma\(2025\)FedEx\-LoRA: exact aggregation for federated and efficient fine\-tuning of large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1316–1336\.External Links:ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[17\]H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar, A\. Rodriguez, A\. Joulin, E\. Grave, and G\. Lample\(2023\)LLaMA: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.External Links:[Link](https://arxiv.org/abs/2302.13971)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p2.1)\.
- \[18\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. Bowman\(2019\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1804.07461)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p1.1)\.
- \[19\]L\. Wang, J\. Bian, L\. Zhang, and J\. Xu\(2026\)Breaking the structural identity: personalized federated LoRA fine\-tuning under rank heterogeneity\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 12816–12832\.External Links:[Link](https://aclanthology.org/2026.findings-emnlp.608/)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p2.1)\.
- \[20\]X\. Wang, Y\. Zheng, Z\. Wan, and M\. Zhang\(2025\)SVD\-LLM: truncation\-aware singular value decomposition for large language model compression\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=LNYIUouhdt)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p3.1)\.
- \[21\]Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap, E\. Pathak, G\. Karamanolakis, H\. Lai, I\. Purohit, I\. Mondal, J\. Anderson, K\. Kuznia, K\. Doshi, K\. K\. Pal, M\. Patel, M\. Moradshahi, M\. Parmar, M\. Purohit, N\. Varshney, P\. R\. Kaza, P\. Verma, R\. S\. Puri, R\. Karia, S\. Doshi, S\. K\. Sampat, S\. Mishra, S\. R\. A, S\. Patro, T\. Dixit, and X\. Shen\(2022\)Super\-NaturalInstructions: generalization via declarative instructions on 1600\+ NLP tasks\.InConference on Empirical Methods in Natural Language Processing,pp\. 5085–5109\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.340),[Link](https://aclanthology.org/2022.emnlp-main.340/)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p1.1)\.
- \[22\]Z\. Wang, Z\. Shen, Y\. He, G\. Sun, H\. Wang, L\. Lyu, and A\. Li\(2024\)FLoRA: federated fine\-tuning large language models with heterogeneous low\-rank adaptations\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://arxiv.org/abs/2409.05976)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p1.1)\.
- \[23\]Z\. Yuan, Y\. Shang, Y\. Song, Q\. Wu, Y\. Yan, and G\. Sun\(2023\)ASVD: activation\-aware singular value decomposition for compressing large language models\.arXiv preprint arXiv:2312\.05821\.External Links:[Link](https://arxiv.org/abs/2312.05821)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p3.1)\.
- \[24\]H\. Zhang, J\. Wang, S\. Rajendran, W\. Pan, L\. Tsoi, Y\. Chen, F\. Wang, and J\. Zhou\(2026\)Dysco: dynamic subspace boosting to mitigate LoRA interference in federated learning\.arXiv preprint arXiv:2607\.14367\.External Links:[Link](https://arxiv.org/abs/2607.14367)Cited by:[§2](https://arxiv.org/html/2611.23742#S2.p2.1)\.
- \[25\]X\. Zhang, Y\. He, R\. Xu, H\. Yu, Z\. Shen, and P\. Cui\(2023\)NICO\+\+: towards better benchmarking for domain generalization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16036–16047\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2023/html/Zhang_NICO_Towards_Better_Benchmarking_for_Domain_Generalization_CVPR_2023_paper.html)Cited by:[§6\.1](https://arxiv.org/html/2611.23742#S6.SS1.p1.1)\.Similar Articles
Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity
FedRoRA is a novel framework for personalized federated LoRA fine-tuning that addresses rank heterogeneity and data heterogeneity in federated learning by decoupling adaptation into shared global directions and personalized magnitudes.
One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning
This paper introduces an adaptive anisotropic learning-rate model for LoRA fine-tuning to address within-module heterogeneity, improving performance and rank capacity utilization across benchmarks.
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
This paper introduces GLoRA, a gauge-aware server representation for Federated LoRA that addresses the semantic mismatch in factor aggregation by estimating a consensus update subspace. Experiments show GLoRA outperforms baselines in performance and efficiency across heterogeneous client scenarios.
Adaptive Phase-Switching for Communication-Efficient Federated LoRA Fine-Tuning
This paper introduces an adaptive phase-switching method for communication-efficient federated LoRA fine-tuning, achieving up to 40.5% round-trip savings in communication costs while maintaining model performance on large language models.
LoRA on a Budget: Function-Aware Post-Hoc Compression for Adapter Fleets
A new arXiv paper introduces Functional Rank Allocation (FRA), an unlabeled method for compressing large fleets of LoRA adapters via functional rank-distortion frontiers and fleet-level minimax allocation under a shared rank budget, improving worst-case and tail utility over uniform spectral compression across 78 adapters from three public fleets.