Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA

# Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
Source: [https://arxiv.org/html/2605.06733](https://arxiv.org/html/2605.06733)
Jinqian Chen School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000 chenjinqian@stu\.xjtu\.edu\.cn &Chang Liu School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000 chang\.liu@stu\.xjtu\.edu\.cn &Jihua Zhu School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000 zhujh@xjtu\.edu\.cn

###### Abstract

Federated LoRA enables parameter\-efficient adaptation of large language models under decentralized data and limited client resources\.However, directly averaging LoRA factors is representation\-dependent: the same intrinsic update admits infinitely many gauge\-equivalent factorizations, so factor\-level aggregation can change under arbitrary coordinate choices while the underlying update remains unchanged\. This reveals a semantic mismatch in existing federated LoRA aggregation rules\. We proposeGLoRA, a gauge\-aware server representation for federated LoRA\.Instead of aggregating raw factors, GLoRA estimates a consensus update subspace from client projectors and aggregates client updates in shared reference coordinates, thereby representing semantic update aggregation entirely in low\-rank form\. To support heterogeneous client capacities, GLoRA further provides a rank\-compatible readout that instantiates adapters of different ranks from the same server state without dense update reconstruction\. Experiments on GLUE and SuperNI show that GLoRA consistently outperforms federated LoRA baselines under data, resource, and task heterogeneity, including heterogeneous client ranks, sparse participation, larger backbones, and unseen\-task evaluation\. GLoRA also achieves a favorable efficiency–performance trade\-off, suggesting that effective federated LoRA requires not merely averaging low\-rank factors, but defining a semantically meaningful server\-side representation for aggregation\.

## 1Introduction

Federated fine\-tuning\[[15](https://arxiv.org/html/2605.06733#bib.bib1),[23](https://arxiv.org/html/2605.06733#bib.bib2)\]has emerged as an increasingly important paradigm for adapting foundation models under decentralized data and client\-side resource constraints\. In this setting, parameter\-efficient fine\-tuning \(PEFT\)\[[8](https://arxiv.org/html/2605.06733#bib.bib3),[13](https://arxiv.org/html/2605.06733#bib.bib4),[24](https://arxiv.org/html/2605.06733#bib.bib5)\]is especially attractive, since only a small fraction of model parameters needs to be updated and communicated\. Among PEFT methods, LoRA\[[8](https://arxiv.org/html/2605.06733#bib.bib3)\]is particularly compelling: it represents adaptation as a low\-rank update and therefore offers the promise of both lightweight local training and lightweight server aggregation\. This makes federated LoRA a natural candidate for scaling LLM adaptation across heterogeneous clients\.

Despite this promise, federated LoRA exposes a fundamental mismatch between*what is cheap to communicate*and*what is meaningful to aggregate*\. A LoRA update is represented asΔ​W=B​A\\Delta W=BA, but this factorization is not unique: the same update matrix admits infinitely many gauge\-equivalent realizations\(B,A\)\(B,A\)and\(B​Q,Q−1​A\)\(BQ,Q^\{\-1\}A\)\. In centralized optimization, this ambiguity is often less exposed because the factors remain internal variables of a single optimizer\. In federated optimization, by contrast, LoRA factors are transmitted across clients and reused as server\-side aggregation objects\. At that point, the ambiguity is no longer merely algebraic; it becomes a semantic issue of the server update rule itself\. If the server aggregates raw factors directly, then identical intrinsic client updates may still induce different server outputs under gauge\-equivalent reparameterizations\. In other words, the aggregation rule is no longer defined on the update matrices themselves, but on arbitrary coordinate realizations\.

A substantial line of recent federated LoRA work\[[17](https://arxiv.org/html/2605.06733#bib.bib6),[16](https://arxiv.org/html/2605.06733#bib.bib8),[6](https://arxiv.org/html/2605.06733#bib.bib9),[2](https://arxiv.org/html/2605.06733#bib.bib7)\]has focused on the*inexact aggregation*problem, namely the mismatch between aggregating products and aggregating factors\. This line of work is important, but it leaves a more basic question unresolved: before asking whether a factor\-level rule recovers the correct aggregate, one must first ask whether raw factors are semantically valid aggregation objects at all\. A straightforward repair for gauge ambiguity in federated LoRA is to aggregate induced updatesΔ​Wi=Bi​Ai\\Delta W\_\{i\}=B\_\{i\}A\_\{i\}directly and then refactorize the result for redistribution\[[1](https://arxiv.org/html/2605.06733#bib.bib10)\]\. This avoids the semantic defect of raw factor aggregation, but it also forces the server back into dense\-update materialization and matrix factorization\. For large transformer layers, this means forming and decomposing matrices inℝdout×din\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}, which is increasingly unattractive at LLM scale and misaligned with the low\-rank efficiency premise that motivates federated PEFT in the first place\.

This paper is built around the following question:*Can we preserve the semantic correctness of update\-level aggregation without collapsing back to dense\-update aggregation?*We answer this question by shifting the focus from aggregation rules to*server representations*\. Our key idea is that the federated server should neither aggregate raw factor coordinates nor materialize full dense updates\. Instead, it should maintain a*gauge\-aware low\-rank representation*that captures the geometry of current\-round client updates and expresses them under a shared reference frame\.

We instantiate this principle inGLoRA, a gauge\-aware low\-rank server representation for federated LoRA\. GLoRA first rewrites each client update in subspace–coordinate form by applying a gauge fixing step:Bi=Ui​RiB\_\{i\}=U\_\{i\}R\_\{i\}andAi′=Ri​AiA^\{\\prime\}\_\{i\}=R\_\{i\}A\_\{i\}, whereUiU\_\{i\}is an orthonormal basis for the client’s update subspace\. The server then aggregates the gauge\-invariant projectorsUi​Ui⊤U\_\{i\}U\_\{i\}^\{\\top\}to estimate a consensus subspaceUrefU\_\{\\mathrm\{ref\}\}\. Finally, each client update is translated into this shared reference frame and aggregated through comparable coordinates\. The resulting server state\(Uref,Aglobal\)\(U\_\{\\mathrm\{ref\}\},A\_\{\\mathrm\{global\}\}\)remains low\-rank, yet corresponds exactly to the projection of the dense update average onto the learned consensus subspace\. For heterogeneous clients, GLoRA reads out adapters of different ranks from the same server state, avoiding separate aggregation objects or dense refactorization\. We evaluate GLoRA on GLUE\[[19](https://arxiv.org/html/2605.06733#bib.bib11)\]and SuperNI\[[20](https://arxiv.org/html/2605.06733#bib.bib12)\]under data heterogeneity, heterogeneous client ranks, sparse participation, larger backbones, and unseen\-task evaluation\. Across these settings, GLoRA consistently improves over federated LoRA baselines based on direct factor aggregation and achieves a favorable efficiency–performance trade\-off relative to dense\-update aggregation methods\. These results suggest that the key design choice in federated LoRA is not merely how to average low\-rank factors, but what server\-side object should represent the aggregate update\. Contributions are summarized as follows:

- •We identify*gauge dependence*as a semantic failure mode of factor\-level federated LoRA aggregation\. Unlike the inexact aggregation problem, which asks whether a factor\-level rule recovers a desired product\-level aggregate, gauge dependence asks a more basic question: whether raw LoRA factors are representation\-invariant aggregation objects at all\.
- •We proposeGLoRA, a gauge\-aware aggregation method for federated LoRA\. GLoRA maintains a compact low\-rank server state, aggregates client updates through a consensus subspace and shared reference coordinates, and instantiates heterogeneous clients via rank\-compatible readout, avoiding both raw factor aggregation and dense\-update materialization\.
- •We conduct extensive experiments on GLUE and SuperNI under data, resource, and task heterogeneity with sparse participation\. Results show that GLoRA consistently improves over federated LoRA baselines, scales to larger backbones, and achieves a favorable efficiency–performance trade\-off compared with dense\-update aggregation\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2605.06733v1/x1.png)Figure 1:Motivation of gauge\-aware LoRA aggregation in FL\. Gauge\-equivalent factors, e\.g\.,\(B,A\)\(B,A\)and\(B​Q,Q−1​A\)\(BQ,Q^\{\-1\}A\), represent the same updateΔ​W=B​A\\Delta W=BAbut can yield different server models under raw factor aggregation\. The validation experiment shows that existing factor\-aggregation methods are gauge\-sensitive, whereas GLoRA achieves near\-zero aggregation inconsistency\.#### Parameter\-Efficient Fine\-Tuning\.

Parameter\-efficient fine\-tuning \(PEFT\) adapts large pre\-trained models by updating only a small subset of parameters\. Representative directions include adapter\-based tuning\[[7](https://arxiv.org/html/2605.06733#bib.bib13)\], prompt/prefix\-based tuning\[[10](https://arxiv.org/html/2605.06733#bib.bib14),[9](https://arxiv.org/html/2605.06733#bib.bib15)\], and multiplicative modulation methods such as IA3\[[12](https://arxiv.org/html/2605.06733#bib.bib16)\]\. Among them, LoRA\[[8](https://arxiv.org/html/2605.06733#bib.bib3)\]has become particularly influential due to its low\-rank parameterization and zero additional inference latency after merging\. Subsequent variants mainly improve LoRA in centralized settings, such as adaptive rank allocation in AdaLoRA\[[24](https://arxiv.org/html/2605.06733#bib.bib5)\], quantization\-aware fine\-tuning in QLoRA\[[5](https://arxiv.org/html/2605.06733#bib.bib17)\], and enhanced decomposition in DoRA\[[13](https://arxiv.org/html/2605.06733#bib.bib4)\]\. These methods substantially improve PEFT efficiency and expressiveness, but they do not address the server\-side semantic validity of aggregating LoRA updates across federated clients\.

Federated LoRA\.Recent work incorporates LoRA into federated fine\-tuning of foundation models\. A first line of research directly aggregates LoRA factors or modifies factor\-level aggregation rules, including FedIT\[[23](https://arxiv.org/html/2605.06733#bib.bib2)\], FFA\-LoRA\[[17](https://arxiv.org/html/2605.06733#bib.bib6)\], FedSA\-LoRA\[[6](https://arxiv.org/html/2605.06733#bib.bib9)\], LoRA\-FAIR\[[2](https://arxiv.org/html/2605.06733#bib.bib7)\], and FedEx\-LoRA\[[16](https://arxiv.org/html/2605.06733#bib.bib8)\]\. These methods mainly focus on reducing aggregation error, improving robustness, or addressing the algebraic*inexact aggregation*problem\. Our work differs in viewpoint: instead of asking how to aggregate LoRA factors more accurately, we ask a logically prior question—whether raw factor coordinates are semantically valid server\-side aggregation objects at all under gauge ambiguity\. A second line of work studies heterogeneity in federated LoRA, especially heterogeneous client ranks and system budgets\. HetLoRA\[[4](https://arxiv.org/html/2605.06733#bib.bib18)\], FLoRA\[[21](https://arxiv.org/html/2605.06733#bib.bib19)\], and FlexLoRA\[[1](https://arxiv.org/html/2605.06733#bib.bib10)\]design rank\-adaptive or resource\-aware redistribution mechanisms for heterogeneous clients\. These methods highlight the practical importance of client heterogeneity, but they do not directly resolve the representation issue raised in this paper\. In contrast, we treat heterogeneous rank not as the starting point of the problem, but as a design constraint that the server\-side representation must naturally support\.

## 3Rethinking LoRA in Distributed Optimization

### 3\.1LoRA Is an Equivalence\-Class Parameterization

LoRA represents an adaptation matrix as a low\-rank productΔ​W=B​A\\Delta W=BA, whereB∈ℝdout×rB\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}andA∈ℝr×dinA\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}withr≪min⁡\(dout,din\)r\\ll\\min\(d\_\{\\mathrm\{out\}\},d\_\{\\mathrm\{in\}\}\)\. This representation is efficient but not unique\. For any invertible matrixQ∈GL​\(r\)Q\\in\\mathrm\{GL\}\(r\),

B​A=\(B​Q\)​\(Q−1​A\)\.BA=\(BQ\)\(Q^\{\-1\}A\)\.\(1\)Hence, a LoRA update is not identified by a unique factor pair, but by an equivalence class of factorizations that induce the same update matrix\.

###### Definition 1\(Gauge\-equivalent LoRA factorizations\)

Two factor pairs\(B,A\)\(B,A\)and\(B~,A~\)\(\\widetilde\{B\},\\widetilde\{A\}\)are*gauge\-equivalent*if there exists an invertible matrixQ∈GL​\(r\)Q\\in\\mathrm\{GL\}\(r\)such thatB~=B​Q,A~=Q−1​A\.\\widetilde\{B\}=BQ,\\widetilde\{A\}=Q^\{\-1\}A\.Their equivalence class is\[\(B,A\)\]=\{\(B​Q,Q−1​A\):Q∈GL​\(r\)\}\.\[\(B,A\)\]\\;=\\;\\\{\(BQ,Q^\{\-1\}A\):Q\\in\\mathrm\{GL\}\(r\)\\\}\.

This ambiguity is often less exposed in centralized training, where the factors remain internal variables of a single optimizer\. In distributed optimization, however, LoRA factors are no longer private coordinates: they are transmitted across clients and must serve as server\-side aggregation objects\. At that point, gauge ambiguity is no longer merely an algebraic nuisance\. It becomes a semantic issue of the server update rule itself\.

### 3\.2Distributed LoRA Requires a Gauge\-Invariant Server Update Rule

Consider roundttwith active client set𝒞t\\mathcal\{C\}\_\{t\}\. Each clienti∈𝒞ti\\in\\mathcal\{C\}\_\{t\}produces a local LoRA updateΔ​Wit=Bit​Ait,\\Delta W\_\{i\}^\{t\}=B\_\{i\}^\{t\}A\_\{i\}^\{t\},where\(Bit,Ait\)\(B\_\{i\}^\{t\},A\_\{i\}^\{t\}\)is only one representative of the equivalence class\[\(Bit,Ait\)\]\[\(B\_\{i\}^\{t\},A\_\{i\}^\{t\}\)\]\. A distributed LoRA method may therefore be abstracted as a server update ruleΩt\+1=Ψ​\(\{\(Bit,Ait\)\}i∈𝒞t,Ωt\),\\Omega^\{t\+1\}=\\Psi\\\!\\left\(\\\{\(B\_\{i\}^\{t\},A\_\{i\}^\{t\}\)\\\}\_\{i\\in\\mathcal\{C\}\_\{t\}\},\\,\\Omega^\{t\}\\right\),where the central question is whetherΨ\\Psidepends on the intrinsic client updates\{Δ​Wit\}\\\{\\Delta W\_\{i\}^\{t\}\\\}, or on the arbitrary factor coordinates used to express them\.

###### Definition 2\(Gauge\-invariant state update\)

A server update ruleΨ\\Psiis*gauge\-invariant*if, for any roundttand any collection of invertible matrices\{Qi\}i∈𝒞t\\\{Q\_\{i\}\\\}\_\{i\\in\\mathcal\{C\}\_\{t\}\},Ψ​\(\{\(Bit,Ait\)\}i∈𝒞t,Ωt\)=Ψ​\(\{\(Bit​Qi,Qi−1​Ait\)\}i∈𝒞t,Ωt\)\.\\Psi\\\!\\left\(\\\{\(B\_\{i\}^\{t\},A\_\{i\}^\{t\}\)\\\}\_\{i\\in\\mathcal\{C\}\_\{t\}\},\\Omega^\{t\}\\right\)=\\Psi\\\!\\left\(\\\{\(B\_\{i\}^\{t\}Q\_\{i\},Q\_\{i\}^\{\-1\}A\_\{i\}^\{t\}\)\\\}\_\{i\\in\\mathcal\{C\}\_\{t\}\},\\Omega^\{t\}\\right\)\.

Definition[2](https://arxiv.org/html/2605.06733#Thmdefinition2)formalizes the minimal requirement for distributed LoRA: the server should respond to the underlying updates themselves, rather than to arbitrary factor coordinates\. This immediately explains why direct factor\-space aggregation is problematic\. Any rule that operates explicitly on\(Bi,Ai\)\(B\_\{i\},A\_\{i\}\), rather than only onΔ​Wi=Bi​Ai\\Delta W\_\{i\}=B\_\{i\}A\_\{i\}, can in general change its output under a gauge\-equivalent reparameterization, even when the underlying client updates remain fixed \(See Fig\.[1](https://arxiv.org/html/2605.06733#S2.F1)panel 3\)\.

#### Discussion: Gauge invariance vs\. inexact aggregation\.

Existing federated LoRA methods often emphasize the*inexact aggregation*issue, i\.e\.,∑iBi​Ai≠\(∑iBi\)​\(∑iAi\)\\sum\_\{i\}B\_\{i\}A\_\{i\}\\neq\\Big\(\\sum\_\{i\}B\_\{i\}\\Big\)\\Big\(\\sum\_\{i\}A\_\{i\}\\Big\), which asks whether a factor\-level rule can recover the desired update average\. We argue that gauge invariance is a more fundamental requirement\. Before asking how to aggregate LoRA factors, one must first ask whether raw factors are valid aggregation objects at all\. Under gauge\-equivalent reparameterizations, the same intrinsic updateΔ​Wi=Bi​Ai\\Delta W\_\{i\}=B\_\{i\}A\_\{i\}can be represented by different coordinates, so aggregating raw factors may yield different server updates for the same client updates\. This is a semantic failure, whereas inexact aggregation is an algebraic approximation error after a representation has already been fixed\.

A straightforward remedy is to aggregate update matrices directlyΔ​Wgt=∑i∈𝒞tpit​Bit​Ait,\\Delta W\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}B\_\{i\}^\{t\}A\_\{i\}^\{t\},and then refactorize the result for redistribution\. While semantically valid, this reduces federated LoRA to dense\-update aggregation, requiring the server to materialize and decompose adout×dind\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}matrix\. Such a solution undermines the efficiency premise of PEFT at LLM scale\.

This motivates our central design question:*Can we build a server\-side low\-rank representation that is gauge\-invariant, avoids dense re\-materialization, and supports heterogeneous client ranks?*GLoRA answers this question by aggregating client updates through a shared consensus subspace and rank\-compatible low\-rank coordinates\.

## 4GLoRA: Gauge\-aware LoRA Aggregation for Distributed Optimization

The discussion above identifies the required server object: it should be defined by the intrinsic client updates, but represented without dense materialization\. GLoRA realizes this object with a subspace–coordinate server state\. At roundtt, the server maintainsΩt=\(Ureft,Zgt\),\\Omega^\{t\}=\(U\_\{\\mathrm\{ref\}\}^\{t\},Z\_\{g\}^\{t\}\),whereUreft∈ℝdout×RU\_\{\\mathrm\{ref\}\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times R\}is an orthonormal reference basis with rank budgetRR, andZgt∈ℝR×dinZ\_\{g\}^\{t\}\\in\\mathbb\{R\}^\{R\\times d\_\{\\mathrm\{in\}\}\}stores the aggregated coordinates in this basis\. Together they induce the low\-rank server updateΔ​Wgt=Ureft​Zgt\.\\Delta W\_\{g\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}Z\_\{g\}^\{t\}\.The role of the two components is deliberately separated:UreftU\_\{\\mathrm\{ref\}\}^\{t\}specifies the common update geometry to which clients are aligned, whileZgtZ\_\{g\}^\{t\}stores the aggregated update after this alignment\. The rest of this section explains how GLoRA constructs and redistributes this state: Sec\.[4\.1](https://arxiv.org/html/2605.06733#S4.SS1)extracts gauge\-invariant update geometry, Sec\.[4\.2](https://arxiv.org/html/2605.06733#S4.SS2)aggregates projected updates in the shared frame, and Sec\.[4\.3](https://arxiv.org/html/2605.06733#S4.SS3)reads out rank\-compatible adapters for heterogeneous clients\.

### 4\.1Gauge\-Invariant Subspace Extraction and Consensus Geometry

Each active clienti∈𝒞ti\\in\\mathcal\{C\}\_\{t\}returns a LoRA updateΔ​Wit=Bit​Ait,Bit∈ℝdout×ri,Ait∈ℝri×din,\\Delta W\_\{i\}^\{t\}=B\_\{i\}^\{t\}A\_\{i\}^\{t\},B\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\},A\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\},whererir\_\{i\}may vary across clients\. Since\(Bit,Ait\)\(B\_\{i\}^\{t\},A\_\{i\}^\{t\}\)is only one representative of the equivalence class inducingΔ​Wit\\Delta W\_\{i\}^\{t\}, GLoRA first converts the update into a subspace–coordinate form and extracts its gauge\-invariant column\-subspace geometry via a reduced QR decomposition111We assume full column rank for notation\. IfBitB\_\{i\}^\{t\}is rank\-deficient, replacerir\_\{i\}withrank⁡\(Bit\)\\operatorname\{rank\}\(B\_\{i\}^\{t\}\); the results still hold\.:

Bit=Uit​Tit,\(Uit\)⊤​Uit=Iri,A^it=Tit​Ait\.B\_\{i\}^\{t\}=U\_\{i\}^\{t\}T\_\{i\}^\{t\},\\qquad\(U\_\{i\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}=I\_\{r\_\{i\}\},\\qquad\\widehat\{A\}\_\{i\}^\{t\}=T\_\{i\}^\{t\}A\_\{i\}^\{t\}\.\(2\)ThenΔ​Wit=Uit​A^it\.\\Delta W\_\{i\}^\{t\}=U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}\.This separates the update into a column subspacespan​\(Uit\)\\mathrm\{span\}\(U\_\{i\}^\{t\}\)and coordinatesA^it\\widehat\{A\}\_\{i\}^\{t\}inside that subspace\. The column subspace is the part of the update geometry that is comparable across clients\. We represent it by the orthogonal projectorPit=Uit​\(Uit\)⊤\.P\_\{i\}^\{t\}=U\_\{i\}^\{t\}\(U\_\{i\}^\{t\}\)^\{\\top\}\.

Unlike raw factors,PitP\_\{i\}^\{t\}is invariant to any invertible reparameterizationBit↦Bit​QiB\_\{i\}^\{t\}\\mapsto B\_\{i\}^\{t\}Q\_\{i\}, because such a transformation does not change the subspace spanned byBitB\_\{i\}^\{t\}\. The server therefore estimates the shared update geometry by

Kt=∑i∈𝒞tpit​Pit=∑i∈𝒞tpit​Uit​\(Uit\)⊤,Kt∈ℝdout×doutK^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}P\_\{i\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}U\_\{i\}^\{t\}\(U\_\{i\}^\{t\}\)^\{\\top\},\\quad K^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{out\}\}\}\(3\)wherepit≥0p\_\{i\}^\{t\}\\geq 0and∑i∈𝒞tpit=1\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}=1\. The reference basis is defined as

Ureft=TopEigR​\(Kt\),\(Ureft\)⊤​Ureft=IR\.U\_\{\\mathrm\{ref\}\}^\{t\}=\\mathrm\{TopEig\}\_\{R\}\(K^\{t\}\),\\qquad\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{\\mathrm\{ref\}\}^\{t\}=I\_\{R\}\.\(4\)Thus,UreftU\_\{\\mathrm\{ref\}\}^\{t\}captures the rank\-RRconsensus subspace most consistently supported by current\-round client updates\. Importantly, this step does not require materializing the dense matrixKtK^\{t\}\. In practice, the same eigenspace can be obtained from the thin matrixMt=\[pit​Uit\]i∈𝒞t,M^\{t\}=\\big\[\\sqrt\{p\_\{i\}^\{t\}\}U\_\{i\}^\{t\}\\big\]\_\{i\\in\\mathcal\{C\}\_\{t\}\},sinceKt=Mt​\(Mt\)⊤K^\{t\}=M^\{t\}\(M^\{t\}\)^\{\\top\}\. The server therefore operates on a matrix with only∑i∈𝒞tri\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}columns\.

###### Proposition 1\(Exactness under sufficient server rank\.\)

LetΔ​Wit=Uit​A^it\\Delta W\_\{i\}^\{t\}=U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}with\(Uit\)⊤​Uit=Iri\(U\_\{i\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}=I\_\{r\_\{i\}\}, and letr∪t=rank​\(\[Uit\]i∈𝒞t\)r\_\{\\cup\}^\{t\}=\\mathrm\{rank\}\\big\(\[U\_\{i\}^\{t\}\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\\big\)be the dimension of the union span of participating client update subspaces\. If the server rank satisfiesR≥r∪tR\\geq r\_\{\\cup\}^\{t\}, then GLoRA recovers the dense update average exactly:Ureft​Zgt=∑i∈𝒞tpit​Δ​Wit\.U\_\{\\mathrm\{ref\}\}^\{t\}Z\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\\Delta W\_\{i\}^\{t\}\.In particular, sincer∪t≤∑i∈𝒞trir\_\{\\cup\}^\{t\}\\leq\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}, the conditionR≥∑i∈𝒞triR\\geq\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}is sufficient for exact aggregation\. The proof is provided in Appendix[1](https://arxiv.org/html/2605.06733#Thmproof1)\.

### 4\.2Projected Update Aggregation in Shared Coordinates

AfterUreftU\_\{\\mathrm\{ref\}\}^\{t\}is fixed, each client update can be expressed in the same reference frame asZit=\(Ureft\)⊤​Uit​A^it=\(Ureft\)⊤​Δ​Wit∈ℝR×dinZ\_\{i\}^\{t\}=\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}=\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}\\Delta W\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{R\\times d\_\{\\mathrm\{in\}\}\}\. Its reconstruction,Ureft​Zit=Ureft​\(Ureft\)⊤​Δ​WitU\_\{\\mathrm\{ref\}\}^\{t\}Z\_\{i\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}\\Delta W\_\{i\}^\{t\}, is exactly the orthogonal projection of clientii’s update onto the consensus subspace\.

Since allZitZ\_\{i\}^\{t\}now share the same basis, they can be aggregated directly asZgt=∑i∈𝒞tpit​ZitZ\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}Z\_\{i\}^\{t\}, yielding the server updateΔ​Wgt=Ureft​Zgt\\Delta W\_\{g\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}Z\_\{g\}^\{t\}\. Combining the two expressions givesZgt=\(Ureft\)⊤​∑i∈𝒞tpit​Δ​WitZ\_\{g\}^\{t\}=\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\\Delta W\_\{i\}^\{t\}, and hence

Δ​Wgt=Ureft​\(Ureft\)⊤​∑i∈𝒞tpit​Δ​Wit\.\\Delta W\_\{g\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\\Delta W\_\{i\}^\{t\}\.\(5\)
Equation \([5](https://arxiv.org/html/2605.06733#S4.E5)\) is the key identity of GLoRA\. It shows that GLoRA performs update\-level aggregation, but only inside the learned consensus subspace\. Therefore, the server update is defined by the intrinsic matrices\{Δ​Wit\}\\\{\\Delta W\_\{i\}^\{t\}\\\}rather than by arbitrary LoRA coordinates, while the server never materializes the dense average∑ipit​Δ​Wit\\sum\_\{i\}p\_\{i\}^\{t\}\\Delta W\_\{i\}^\{t\}\.

###### Proposition 2\(Gauge invariance of GLoRA aggregation\)

For any roundtt, the GLoRA server updateΔ​Wgt\\Delta W\_\{g\}^\{t\}is invariant to client\-side gauge reparameterizations\. That is, replacing each submitted factor pair\(Bit,Ait\)\(B\_\{i\}^\{t\},A\_\{i\}^\{t\}\)by\(Bit​Qi,Qi−1​Ait\)\(B\_\{i\}^\{t\}Q\_\{i\},Q\_\{i\}^\{\-1\}A\_\{i\}^\{t\}\)for any invertibleQi∈GL​\(ri\)Q\_\{i\}\\in\\mathrm\{GL\}\(r\_\{i\}\)leavesΔ​Wgt\\Delta W\_\{g\}^\{t\}unchanged\.

### 4\.3Client\-Aware Readout for Heterogeneous Ranks

The server maintains a single stateΩt=\(Ureft,Zgt\)\\Omega^\{t\}=\(U\_\{\\mathrm\{ref\}\}^\{t\},Z\_\{g\}^\{t\}\), but clientiimay only instantiate a rank\-rir\_\{i\}adapter\. Thus, redistribution requires a rank\-compatible readout from the same server update\.

We first put the server update into an energy\-ordered low\-rank form\. BecauseUreftU\_\{\\mathrm\{ref\}\}^\{t\}is orthonormal, it is sufficient to decompose the small coordinate matrix:

Zgt=Ot​Σt​\(Vt\)⊤,Ust=Ureft​Ot\.Z\_\{g\}^\{t\}=O^\{t\}\\Sigma^\{t\}\(V^\{t\}\)^\{\\top\},\\qquad U\_\{s\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}O^\{t\}\.\(6\)ThenΔ​Wgt=Ust​Σt​\(Vt\)⊤=∑j=1ℓσjt​ujt​\(vjt\)⊤\\Delta W\_\{g\}^\{t\}=U\_\{s\}^\{t\}\\Sigma^\{t\}\(V^\{t\}\)^\{\\top\}=\\sum\_\{j=1\}^\{\\ell\}\\sigma\_\{j\}^\{t\}u\_\{j\}^\{t\}\(v\_\{j\}^\{t\}\)^\{\\top\}, whereℓ=min⁡\(R,din\)\\ell=\\min\(R,d\_\{\\mathrm\{in\}\}\)\. A purely spectral readout would simply send the top\-rir\_\{i\}spectral components to clientii\. This is globally optimal for approximatingΔ​Wgt\\Delta W\_\{g\}^\{t\}in Frobenius norm, but it may be suboptimal under data heterogeneity, where different clients benefit from different directions\.

GLoRA therefore uses a core–tail readout\. The core set𝒢it=\{1,…,⌊γ​ri⌋\}\\mathcal\{G\}\_\{i\}^\{t\}=\\\{1,\\ldots,\\lfloor\\gamma r\_\{i\}\\rfloor\\\}, withγ∈\[0,1\]\\gamma\\in\[0,1\], preserves a common shared update\. The remaining rank budget is assigned according to client\-specific update geometry\. LetHit−H\_\{i\}^\{t\-\}be the latest available orthonormal basis of clientii’s historical update subspace\. For each non\-core component, we compute the alignment scoreai​jt=‖\(Hit−\)⊤​ujt‖22a\_\{ij\}^\{t\}=\\\|\(H\_\{i\}^\{t\-\}\)^\{\\top\}u\_\{j\}^\{t\}\\\|\_\{2\}^\{2\}\. The selected component set isℐit=𝒢it∪TopKri−\|𝒢it\|​\{ai​jt:j∉𝒢it\}\.\\mathcal\{I\}\_\{i\}^\{t\}=\\mathcal\{G\}\_\{i\}^\{t\}\\cup\\mathrm\{TopK\}\_\{r\_\{i\}\-\|\\mathcal\{G\}\_\{i\}^\{t\}\|\}\\\{a\_\{ij\}^\{t\}:j\\notin\\mathcal\{G\}\_\{i\}^\{t\}\\\}\.If no history is available, GLoRA falls back to the global spectral order\. Givenℐit\\mathcal\{I\}\_\{i\}^\{t\}, the server initializes clientiiwith the balanced factorization

Bi,initt=Ust​\[:,ℐit\]​diag​\(σℐitt\),Ai,initt=diag​\(σℐitt\)​\(Vt​\[:,ℐit\]\)⊤\.B\_\{i,\\mathrm\{init\}\}^\{t\}=U\_\{s\}^\{t\}\[:,\\mathcal\{I\}\_\{i\}^\{t\}\]\\mathrm\{diag\}\\\!\\left\(\\sqrt\{\\sigma\_\{\\mathcal\{I\}\_\{i\}^\{t\}\}^\{t\}\}\\right\),\\qquad A\_\{i,\\mathrm\{init\}\}^\{t\}=\\mathrm\{diag\}\\\!\\left\(\\sqrt\{\\sigma\_\{\\mathcal\{I\}\_\{i\}^\{t\}\}^\{t\}\}\\right\)\\left\(V^\{t\}\[:,\\mathcal\{I\}\_\{i\}^\{t\}\]\\right\)^\{\\top\}\.\(7\)This yieldsBi,initt​Ai,initt=∑j∈ℐitσjt​ujt​\(vjt\)⊤B\_\{i,\\mathrm\{init\}\}^\{t\}A\_\{i,\\mathrm\{init\}\}^\{t\}=\\sum\_\{j\\in\\mathcal\{I\}\_\{i\}^\{t\}\}\\sigma\_\{j\}^\{t\}u\_\{j\}^\{t\}\(v\_\{j\}^\{t\}\)^\{\\top\}\. The square\-root split preserves the selected update while balancing magnitude between the two LoRA factors, which provides a better initialization for subsequent local optimization\. This readout does not introduce client\-specific server modules\. All clients receive different rank\-limited views of the same gauge\-aware server state\. The shared core maintains global consistency, while the client\-aware tail uses the remaining rank budget to adapt to local update geometry\.

The pseudo\-code of GLoRA is provided in the Appendix\.

## 5Experiments

### 5\.1Cross\-Device Federated Evaluation Setup

We evaluate GLoRA under three practical sources of cross\-device heterogeneity: data distribution, client resources, and task semantics\.

![Refer to caption](https://arxiv.org/html/2605.06733v1/x2.png)Figure 2:Client types and rank distributions for hetero\-rank experiments\.#### Data heterogeneity\.

For GLUE\[[19](https://arxiv.org/html/2605.06733#bib.bib11)\], we create non\-IID client partitions usingDir​\(α\)\\mathrm\{Dir\}\(\\alpha\)withα∈\{0\.1,0\.5\}\\alpha\\in\\\{0\.1,0\.5\\\}, corresponding to highly and moderately heterogeneous label distributions, respectively\.

#### Resource heterogeneity\.

To simulate resource imbalance across devices, we define five client types with different LoRA ranks, as shown in Figure[2](https://arxiv.org/html/2605.06733#S5.F2)\. Following FlexLoRA\[[1](https://arxiv.org/html/2605.06733#bib.bib10)\], we instantiate three population\-level rank patterns:*normal*,*long\-tail*, and*uniform*\. These settings evaluate whether aggregation methods can handle heterogeneous adapter capacities\.

#### Task heterogeneity\.

We construct a task\-heterogeneous benchmark from SuperNI\[[20](https://arxiv.org/html/2605.06733#bib.bib12)\]\. Specifically, we sample 50 task categories from 76 categories and build 50 client groups from 1,600 tasks, where each client is associated with a distinct task\. We additionally hold out 20 unseen and unrelated tasks for unseen\-client evaluation, measuring both client performance and generalization ability\.

#### Backbone models\.

For GLUE, we follow FedSA\-LoRA\[[6](https://arxiv.org/html/2605.06733#bib.bib9)\]and FFA\-LoRA\[[17](https://arxiv.org/html/2605.06733#bib.bib6)\]and use RoBERTa\-Base\[[14](https://arxiv.org/html/2605.06733#bib.bib20)\]\. For SuperNI, we follow FlexLoRA\[[1](https://arxiv.org/html/2605.06733#bib.bib10)\]and use Data\-Juicer LLaMA\-1B\-dj\-refine\-150B\[[3](https://arxiv.org/html/2605.06733#bib.bib21)\]as the default backbone\. We further evaluate scalability on Qwen2\-7B\[[22](https://arxiv.org/html/2605.06733#bib.bib22)\]and Gemma\-2\-9B\[[18](https://arxiv.org/html/2605.06733#bib.bib23)\]\. Unless otherwise specified, LoRA fine\-tuning is applied only to the query and value projection layers\. All experiments were conducted on dual RTX 5090 GPUs, except for the 7B and 9B models, which were run on an single RTX PRO 6000 GPU\.

### 5\.2Overall Performance with Homogeneous Ranks

#### Implementation details\.

We evaluate on five GLUE tasks: MNLI\-m, MNLI\-mm, SST\-2, QQP, and QNLI\. Following prior federated LoRA studies\[[17](https://arxiv.org/html/2605.06733#bib.bib6),[16](https://arxiv.org/html/2605.06733#bib.bib8)\], for each task, the training set is split across three clients using Dirichlet label skew withα∈\{0\.1,0\.5\}\\alpha\\in\\\{0\.1,0\.5\\\}\. We use full client participation, homogeneous LoRA rankr=8r=8, local batch size 128, 10 local steps per round, and 1,000 communication rounds\. All methods are optimized with Adam\. For fair comparison, we tune the learning rate over\{2×10−4,5×10−4,10−3,2×10−3,5×10−3\}\\\{2\\times 10^\{\-4\},5\\times 10^\{\-4\},10^\{\-3\},2\\times 10^\{\-3\},5\\times 10^\{\-3\}\\\}and report the best result for each method\.

#### Main results\.

As shown in Table[1](https://arxiv.org/html/2605.06733#S5.T1), GLoRA achieves the best average performance under both Dirichlet splits\. Under the more challengingDir​\(0\.1\)\\mathrm\{Dir\}\(0\.1\)setting, it ranks first on four out of five tasks, demonstrating strong robustness to severe client distribution shift, while FedSA fails to converge on most tasks\. UnderDir​\(0\.5\)\\mathrm\{Dir\}\(0\.5\), GLoRA obtains the best result on every task\. These results suggest that its gains stem not only from handling rank mismatch, but also from aggregating LoRA updates in a well\-defined, gauge\-aware representation space\.

Table 1:Performance comparison on GLUE tasks under different Dirichlet splits with homogeneous ranks\. The best results are highlighted inbold, and the second\-best results areunderlined\.

### 5\.3Overall Performance with Heterogeneous Ranks

#### Implementation details\.

We evaluate three representative rank distributions: normal, uniform, and heavy\-tail as described in Section[5\.1](https://arxiv.org/html/2605.06733#S5.SS1)\. For each GLUE task, the training data are partitioned into 50 clients withDir​\(0\.5\)\\mathrm\{Dir\}\(0\.5\)label skew, and 50% of clients participate in each round\. Clients are assigned different LoRA ranks according to their resource types, while other configurations follow Section[5\.2](https://arxiv.org/html/2605.06733#S5.SS2)\.

#### Main results\.

As shown in Table[2](https://arxiv.org/html/2605.06733#S5.T2), GLoRA achieves the best average performance under all three rank distributions and ranks first on 14 out of 15 task\-distribution pairs\. This consistent advantage across normal, uniform, and heavy\-tail rank profiles suggests that GLoRA is robust to heterogeneous adapter capacities\. By aggregating updates in a shared gauge\-aware subspace and redistributing them according to client capacity and update history, GLoRA better preserves intrinsic update information while remaining compatible with resource\-constrained clients\.

Table 2:Performance comparison on GLUE tasks with heterogeneous LoRA ranks underDir​\(0\.5\)\\mathrm\{Dir\}\(0\.5\)label skew\. The best and second\-best results are highlighted inboldandunderlined, respectively\.

### 5\.4Sparse Cross\-Task FL with Heterogeneous Client Resources

#### Implementation details\.

We further evaluate GLoRA on SuperNI in a challenging cross\-task FL setting where task heterogeneity, heterogeneous client resources, and sparse participation coexist\. Following FlexLoRA\[[1](https://arxiv.org/html/2605.06733#bib.bib10)\], we report ROUGE\-L\[[11](https://arxiv.org/html/2605.06733#bib.bib24)\]on both in\-domain training clients and unseen evaluation clients\. We construct 50 training clients and 20 unseen evaluation clients from heterogeneous SuperNI tasks, with only 10% client participation per round\. Each selected client trains for one local epoch over 30 FL rounds\. We use a local batch size of 4 and Adam optimizer, with the learning rate selected from\{10−4,2×10−4,5×10−4,10−3\}\\\{10^\{\-4\},2\\times 10^\{\-4\},5\\times 10^\{\-4\},10^\{\-3\}\\\}\. Rank distributions follow Section[5\.3](https://arxiv.org/html/2605.06733#S5.SS3)\.

#### Main results\.

Table[3](https://arxiv.org/html/2605.06733#S5.T3)shows that GLoRA achieves the best performance across all rank distributions on both in\-domain and unseen clients\. This demonstrates that GLoRA remains effective when task heterogeneity, resource heterogeneity, and sparse participation are jointly present\. Beyond fitting the training clients, GLoRA also improves unseen\-task generalization, suggesting that its gauge\-aware subspace aggregation better preserves transferable update information than rank\-specific factor aggregation or truncation\. Fig\.[3](https://arxiv.org/html/2605.06733#S5.F3)further confirms this trend at the task\-category level, where GLoRA performs strongly across most seen and unseen categories under different rank distributions\.

Table 3:ROUGE\-L comparison on SuperNI under sparse participation and heterogeneous LoRA ranks\. Results are reported on both in\-domain training clients and unseen evaluation clients\.![Refer to caption](https://arxiv.org/html/2605.06733v1/x3.png)Figure 3:Per\-category ROUGE\-L performance on SuperNI across seen and unseen task categories\.

### 5\.5In\-depth Analysis

#### Q1: How does gauge ambiguity affect federated LoRA optimization?

As shown in Fig\.[4](https://arxiv.org/html/2605.06733#S5.F4), methods that aggregate raw LoRA factors exhibit more unstable convergence under strong data heterogeneity, e\.g\.,Dir​\(0\.1\)\\mathrm\{Dir\}\(0\.1\)\. This is because gauge\-equivalent LoRA updates can have different factor coordinates, making factor\-level aggregation ill\-defined\. GLoRA mitigates this issue by aggregating intrinsic low\-rank updates in a gauge\-aware subspace, leading to smoother optimization\. We note that GLoRA does not eliminate the inherent difficulty of data or rank heterogeneity; rather, it removes the additional instability introduced by gauge ambiguity\.

#### Q2: How does the rank budgetRRaffect performance?

RRdetermines the size of the server consensus subspace\. In experiments, we control it by a ratioρ\\rho, i\.e\.,R=ρ​∑i∈𝒞triR=\\rho\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}, whererir\_\{i\}is the rank of participating clientii\. WhenRRcovers the total participating rank, GLoRA recovers exact update aggregation; otherwise, it discards tail update directions\. Fig\.[5](https://arxiv.org/html/2605.06733#S5.F5)shows that GLoRA degrades gracefully asRRdecreases on both GLUE and SuperNI, suggesting that leading consensus directions retain most useful update information and provide a practical efficiency–performance trade\-off\.

#### Q3: How important is client\-aware readout \(CAR\)?

Fig\.[6](https://arxiv.org/html/2605.06733#S5.F6)shows that the core ratio matters more under severe data heterogeneity, where client updates contain stronger client\-specific components\. Nevertheless, GLoRA remains robust across a broad range of ratios\. Under milder heterogeneity, performance becomes less sensitive, indicating that CAR does not require delicate tuning\.

#### Q4: Does GLoRA scale to larger backbones?

Table[6](https://arxiv.org/html/2605.06733#S5.T6)reports additional GLUE results on Qwen2\-7B and Gemma\-2\-9B with the same setting in Sec\.[5\.2](https://arxiv.org/html/2605.06733#S5.SS2)\. GLoRA consistently maintains its advantage, suggesting that gauge\-aware aggregation is not limited to small or medium backbones but remains effective for larger models\.

#### Q5: Are the core designs necessary?

Table[4](https://arxiv.org/html/2605.06733#S5.T4)ablates projected aggregation \(PA\) and client\-aware readout \(CAR\)\. Removing either component degrades performance: projected aggregation provides a well\-defined aggregation space, while client\-aware readout adapts the shared update representation to heterogeneous client capacities\. Their combination yields the strongest overall performance\.

#### Q6: What is the efficiency of GLoRA?

Table[6](https://arxiv.org/html/2605.06733#S5.T6)shows that GLoRA achieves a favorable efficiency–effectiveness trade\-off\. It supports heterogeneous ranks without gauge\-dependent factor aggregation and avoids the dense\-update operations required by FlexLoRA\. Per LoRA layer, its cost isΘ​\(rΣ2​\(dout\+din\)\+rΣ3\)\\Theta\\\!\\left\(r\_\{\\Sigma\}^\{2\}\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+r\_\{\\Sigma\}^\{3\}\\right\), whererΣ=∑i∈𝒞trir\_\{\\Sigma\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}, making it much cheaper than dense\-update aggregation\. The “Rounds” column reports the rounds needed to reach0\.920\.92accuracy on SST\-2 under uniform ranks, where GLoRA withβ=0\.5\\beta=0\.5converges fastest\. Details are in Appendix\.

![[Uncaptioned image]](https://arxiv.org/html/2605.06733v1/x4.png)

Figure 4:Convergence on SST\-2 underDir​\(0\.1\)\\mathrm\{Dir\}\(0\.1\)\.![[Uncaptioned image]](https://arxiv.org/html/2605.06733v1/x5.png)

Figure 5:Sensitivity to rank budgetRR\.

![[Uncaptioned image]](https://arxiv.org/html/2605.06733v1/x6.png)

Figure 6:Hyperparameter landscape over core ratioγ\\gamma\.Table 4:Ablation study of GLoRA\. CAR is not applicable without consensus subspaces and is marked as N/A\.

Table 5:Larger\-backbone performance underDir​\(0\.1\)\\mathrm\{Dir\}\(0\.1\)\.
Table 6:Efficiency comparison of federated LoRA aggregation methods\. Homo: SST\-2; Hetero: SuperNI with normal ranks\.

## 6Conclusion

We presentedGLoRA, a gauge\-aware low\-rank server representation for federated LoRA aggregation\. By separating update geometry from factor coordinates, GLoRA avoids gauge\-dependent factor aggregation, performs semantic aggregation in a shared consensus subspace, and supports heterogeneous clients through rank\-compatible readout without dense\-update materialization\. Experiments on GLUE and SuperNI show that GLoRA consistently improves over federated LoRA baselines under data, resource, and task heterogeneity, while maintaining a favorable efficiency–performance trade\-off\. These results suggest that effective federated LoRA depends not only on averaging low\-rank factors, but also on defining a meaningful server\-side aggregation object\. One limitation is that the current consensus subspace is estimated mainly from client column\-space geometry; incorporating update\-energy or task\-aware weighting into subspace construction is an interesting direction for future work\.

## References

- \[1\]J\. Bai, D\. Chen, B\. Qian, L\. Yao, and Y\. Li\(2024\)Federated fine\-tuning of large language models under heterogeneous tasks and client resources\.Advances in Neural Information Processing Systems37,pp\. 14457–14483\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p3.2),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px2.p1.1),[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1),[§5\.4](https://arxiv.org/html/2605.06733#S5.SS4.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2605.06733#S5.T2.14.12.12.7),[Table 2](https://arxiv.org/html/2605.06733#S5.T2.32.30.30.7),[Table 2](https://arxiv.org/html/2605.06733#S5.T2.50.48.48.7)\.
- \[2\]J\. Bian, L\. Wang, L\. Zhang, and J\. Xu\(2025\)LoRA\-fair: federated lora fine\-tuning with aggregation and initialization refinement\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3737–3746\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p3.2),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1)\.
- \[3\]D\. Chen, Y\. Huang, Z\. Ma, H\. Chen, X\. Pan, C\. Ge, D\. Gao, Y\. Xie, Z\. Liu, J\. Gao,et al\.\(2024\)Data\-juicer: a one\-stop data processing system for large language models\.InCompanion of the 2024 International Conference on Management of Data,pp\. 120–134\.Cited by:[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1)\.
- \[4\]Y\. J\. Cho, L\. Liu, Z\. Xu, A\. Fahrezi, and G\. Joshi\(2024\)Heterogeneous lora for federated fine\-tuning of on\-device foundation models\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 12903–12913\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1),[Table 2](https://arxiv.org/html/2605.06733#S5.T2.26.24.24.8),[Table 2](https://arxiv.org/html/2605.06733#S5.T2.44.42.42.8),[Table 2](https://arxiv.org/html/2605.06733#S5.T2.8.6.6.8)\.
- \[5\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)Qlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[6\]P\. Guo, S\. Zeng, Y\. Wang, H\. Fan, F\. Wang, and L\. QuSelective aggregation for low\-rank adaptation in federated learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p3.2),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.24.24.24.7),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.54.54.54.7)\.
- \[7\]N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. De Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. Gelly\(2019\)Parameter\-efficient transfer learning for nlp\.InInternational conference on machine learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[8\]E\. J\. Hu, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p1.1),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[9\]B\. Lester, R\. Al\-Rfou, and N\. Constant\(2021\)The power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 conference on empirical methods in natural language processing,pp\. 3045–3059\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]X\. L\. Li and P\. Liang\(2021\)Prefix\-tuning: optimizing continuous prompts for generation\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 4582–4597\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]C\. LIN\(2004\)Rouge: a package for automatic evaluation of summaries\.InProc\. Workshop on Text Summariation Branches Out, Post\-Conference Workshop of ACL 2004,Cited by:[§5\.4](https://arxiv.org/html/2605.06733#S5.SS4.SSS0.Px1.p1.1)\.
- \[12\]H\. Liu, D\. Tam, M\. Muqeeth, J\. Mohta, T\. Huang, M\. Bansal, and C\. A\. Raffel\(2022\)Few\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.Advances in Neural Information Processing Systems35,pp\. 1950–1965\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]S\. Liu, C\. Wang, H\. Yin, P\. Molchanov, Y\. F\. Wang, K\. Cheng, and M\. Chen\(2024\)Dora: weight\-decomposed low\-rank adaptation\.InForty\-first International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p1.1),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov\(2019\)Roberta: a robustly optimized bert pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1)\.
- \[15\]B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y Arcas\(2017\)Communication\-efficient learning of deep networks from decentralized data\.InArtificial intelligence and statistics,pp\. 1273–1282\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p1.1)\.
- \[16\]R\. Singhal, K\. Ponkshe, and P\. Vepakomma\(2025\)FedEx\-lora: exact aggregation for federated and efficient fine\-tuning of large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1316–1336\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p3.2),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1),[§5\.2](https://arxiv.org/html/2605.06733#S5.SS2.SSS0.Px1.p1.3),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.18.18.18.7),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.48.48.48.7)\.
- \[17\]Y\. Sun, Z\. Li, Y\. Li, and B\. DingImproving lora in privacy\-preserving federated learning\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p3.2),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1),[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1),[§5\.2](https://arxiv.org/html/2605.06733#S5.SS2.SSS0.Px1.p1.3),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.12.12.12.7),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.42.42.42.7)\.
- \[18\]G\. Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Ramé,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.Cited by:[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1)\.
- \[19\]A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman\(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p5.6),[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px1.p1.2)\.
- \[20\]Y\. Wang, S\. Mishra, P\. Alipoormolabashi, Y\. Kordi, A\. Mirzaei, A\. Naik, A\. Ashok, A\. S\. Dhanasekaran, A\. Arunkumar, D\. Stap,et al\.\(2022\)Super\-naturalinstructions: generalization via declarative instructions on 1600\+ nlp tasks\.InProceedings of the 2022 conference on empirical methods in natural language processing,pp\. 5085–5109\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p5.6),[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px3.p1.1)\.
- \[21\]Z\. Wang, Z\. Shen, Y\. He, G\. Sun, H\. Wang, L\. Lyu, and A\. Li\(2024\)Flora: federated fine\-tuning large language models with heterogeneous low\-rank adaptations\.Advances in Neural Information Processing Systems37,pp\. 22513–22533\.Cited by:[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1)\.
- \[22\]A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Tang, J\. Wang, J\. Yang, J\. Tu, J\. Zhang, J\. Ma, J\. Yang, J\. Xu, J\. Zhou, J\. Bai, J\. He, J\. Lin, K\. Dang, K\. Lu, K\. Chen, K\. Yang, M\. Li, M\. Xue, N\. Ni, P\. Zhang, P\. Wang, R\. Peng, R\. Men, R\. Gao, R\. Lin, S\. Wang, S\. Bai, S\. Tan, T\. Zhu, T\. Li, T\. Liu, W\. Ge, X\. Deng, X\. Zhou, X\. Ren, X\. Zhang, X\. Wei, X\. Ren, X\. Liu, Y\. Fan, Y\. Yao, Y\. Zhang, Y\. Wan, Y\. Chu, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Guo, and Z\. Fan\(2024\)Qwen2 technical report\.External Links:2407\.10671,[Link](https://arxiv.org/abs/2407.10671)Cited by:[§5\.1](https://arxiv.org/html/2605.06733#S5.SS1.SSS0.Px4.p1.1)\.
- \[23\]J\. Zhang, S\. Vahidian, M\. Kuo, C\. Li, R\. Zhang, T\. Yu, G\. Wang, and Y\. Chen\(2024\)Towards building the federatedgpt: federated instruction tuning\.InICASSP 2024\-2024 IEEE international conference on acoustics, speech and signal processing \(ICASSP\),pp\. 6915–6919\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p1.1),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p2.1),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.36.36.36.8),[Table 1](https://arxiv.org/html/2605.06733#S5.T1.6.6.6.8)\.
- \[24\]Q\. Zhang, M\. Chen, A\. Bukharin, N\. Karampatziakis, P\. He, Y\. Cheng, W\. Chen, and T\. Zhao\(2023\)Adalora: adaptive budget allocation for parameter\-efficient fine\-tuning\.arXiv preprint arXiv:2303\.10512\.Cited by:[§1](https://arxiv.org/html/2605.06733#S1.p1.1),[§2](https://arxiv.org/html/2605.06733#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix ANotation and Proofs

### A\.1Proof of Proposition[1](https://arxiv.org/html/2605.06733#Thmproposition1)

###### Proof 1

Recall that each participating client update can be written in the subspace–coordinate form

Δ​Wit=Uit​A^it,\(Uit\)⊤​Uit=Iri\.\\Delta W\_\{i\}^\{t\}=U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\},\\qquad\(U\_\{i\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}=I\_\{r\_\{i\}\}\.Let

𝒮t=span​\(\[Uit\]i∈𝒞t\)\\mathcal\{S\}^\{t\}=\\mathrm\{span\}\\big\(\[U\_\{i\}^\{t\}\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\\big\)denote the union span of all participating client update subspaces, and let

r∪t=dim\(𝒮t\)=rank​\(\[Uit\]i∈𝒞t\)\.r\_\{\\cup\}^\{t\}=\\dim\(\\mathcal\{S\}^\{t\}\)=\\mathrm\{rank\}\\big\(\[U\_\{i\}^\{t\}\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\\big\)\.The matrix used to estimate the consensus subspace is

Kt=∑i∈𝒞tpit​Uit​\(Uit\)⊤\.K^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}U\_\{i\}^\{t\}\(U\_\{i\}^\{t\}\)^\{\\top\}\.Equivalently, define the thin matrix

Mt=\[pit​Uit\]i∈𝒞t\.M^\{t\}=\\big\[\\sqrt\{p\_\{i\}^\{t\}\}U\_\{i\}^\{t\}\\big\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\.Then

Kt=Mt​\(Mt\)⊤\.K^\{t\}=M^\{t\}\(M^\{t\}\)^\{\\top\}\.Sincepit\>0p\_\{i\}^\{t\}\>0for participating clients, the column span ofMtM^\{t\}is exactly𝒮t\\mathcal\{S\}^\{t\}\. Therefore,

range​\(Kt\)=range​\(Mt\)=𝒮t,\\mathrm\{range\}\(K^\{t\}\)=\\mathrm\{range\}\(M^\{t\}\)=\\mathcal\{S\}^\{t\},and hence

rank​\(Kt\)=r∪t\.\\mathrm\{rank\}\(K^\{t\}\)=r\_\{\\cup\}^\{t\}\.
When the server rank satisfiesR≥r∪tR\\geq r\_\{\\cup\}^\{t\}, the reference basis

Ureft=TopEigR​\(Kt\)U\_\{\\mathrm\{ref\}\}^\{t\}=\\mathrm\{TopEig\}\_\{R\}\(K^\{t\}\)contains an orthonormal basis of the entire nonzero eigenspace ofKtK^\{t\}\. Since the nonzero eigenspace ofKtK^\{t\}is precisely𝒮t\\mathcal\{S\}^\{t\}, we have

𝒮t⊆span​\(Ureft\)\.\\mathcal\{S\}^\{t\}\\subseteq\\mathrm\{span\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)\.Thus, for every clienti∈𝒞ti\\in\\mathcal\{C\}\_\{t\},

Ureft​\(Ureft\)⊤​Uit=Uit,U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}=U\_\{i\}^\{t\},because every column ofUitU\_\{i\}^\{t\}lies in𝒮t\\mathcal\{S\}^\{t\}\.

By the definition of the aggregated coordinates,

Zgt=∑i∈𝒞tpit​Zit=∑i∈𝒞tpit​\(Ureft\)⊤​Uit​A^it\.Z\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}Z\_\{i\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}\.Multiplying both sides byUreftU\_\{\\mathrm\{ref\}\}^\{t\}, we obtain

Ureft​Zgt=∑i∈𝒞tpit​Ureft​\(Ureft\)⊤​Uit​A^it\.U\_\{\\mathrm\{ref\}\}^\{t\}Z\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}\.Using

Ureft​\(Ureft\)⊤​Uit=Uit,U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}=U\_\{i\}^\{t\},this becomes

Ureft​Zgt=∑i∈𝒞tpit​Uit​A^it=∑i∈𝒞tpit​Δ​Wit\.U\_\{\\mathrm\{ref\}\}^\{t\}Z\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\\Delta W\_\{i\}^\{t\}\.Therefore, GLoRA exactly recovers the dense update average wheneverR≥r∪tR\\geq r\_\{\\cup\}^\{t\}\.

Finally, since the rank of a concatenation is at most the sum of the ranks of its blocks,

r∪t=rank​\(\[Uit\]i∈𝒞t\)≤∑i∈𝒞tri,r\_\{\\cup\}^\{t\}=\\mathrm\{rank\}\\big\(\[U\_\{i\}^\{t\}\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\\big\)\\leq\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\},the stronger condition

R≥∑i∈𝒞triR\\geq\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}is sufficient for exact aggregation\.

### A\.2Proof of Proposition[2](https://arxiv.org/html/2605.06733#Thmproposition2)

###### Proof 2

Consider arbitrary client\-side gauge transformations

Bit↦B~it=Bit​Qi,Ait↦A~it=Qi−1​Ait,B\_\{i\}^\{t\}\\mapsto\\widetilde\{B\}\_\{i\}^\{t\}=B\_\{i\}^\{t\}Q\_\{i\},\\qquad A\_\{i\}^\{t\}\\mapsto\\widetilde\{A\}\_\{i\}^\{t\}=Q\_\{i\}^\{\-1\}A\_\{i\}^\{t\},whereQi∈GL​\(ri\)Q\_\{i\}\\in\\mathrm\{GL\}\(r\_\{i\}\)is invertible\. The induced LoRA update is unchanged:

B~it​A~it=Bit​Qi​Qi−1​Ait=Bit​Ait=Δ​Wit\.\\widetilde\{B\}\_\{i\}^\{t\}\\widetilde\{A\}\_\{i\}^\{t\}=B\_\{i\}^\{t\}Q\_\{i\}Q\_\{i\}^\{\-1\}A\_\{i\}^\{t\}=B\_\{i\}^\{t\}A\_\{i\}^\{t\}=\\Delta W\_\{i\}^\{t\}\.Thus, the intrinsic client update matrix remains the same\.

Next, we show that the consensus subspace is also unchanged\. Let

Bit=Uit​TitB\_\{i\}^\{t\}=U\_\{i\}^\{t\}T\_\{i\}^\{t\}be the reduced QR decomposition used by GLoRA\. SinceQiQ\_\{i\}is invertible,

span​\(B~it\)=span​\(Bit​Qi\)=span​\(Bit\)=span​\(Uit\)\.\\mathrm\{span\}\(\\widetilde\{B\}\_\{i\}^\{t\}\)=\\mathrm\{span\}\(B\_\{i\}^\{t\}Q\_\{i\}\)=\\mathrm\{span\}\(B\_\{i\}^\{t\}\)=\\mathrm\{span\}\(U\_\{i\}^\{t\}\)\.Therefore, the orthonormal basis obtained fromB~it\\widetilde\{B\}\_\{i\}^\{t\}may differ fromUitU\_\{i\}^\{t\}, but it spans the same column space\. Hence there exists an orthogonal matrixSi∈ℝri×riS\_\{i\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times r\_\{i\}\}such that

U~it=Uit​Si\.\\widetilde\{U\}\_\{i\}^\{t\}=U\_\{i\}^\{t\}S\_\{i\}\.Consequently, the corresponding orthogonal projector is invariant:

P~it=U~it​\(U~it\)⊤=Uit​Si​Si⊤​\(Uit\)⊤=Uit​\(Uit\)⊤=Pit\.\\widetilde\{P\}\_\{i\}^\{t\}=\\widetilde\{U\}\_\{i\}^\{t\}\(\\widetilde\{U\}\_\{i\}^\{t\}\)^\{\\top\}=U\_\{i\}^\{t\}S\_\{i\}S\_\{i\}^\{\\top\}\(U\_\{i\}^\{t\}\)^\{\\top\}=U\_\{i\}^\{t\}\(U\_\{i\}^\{t\}\)^\{\\top\}=P\_\{i\}^\{t\}\.It follows that the weighted projector aggregation matrix is unchanged:

K~t=∑i∈𝒞tpit​P~it=∑i∈𝒞tpit​Pit=Kt\.\\widetilde\{K\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\\widetilde\{P\}\_\{i\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}P\_\{i\}^\{t\}=K^\{t\}\.Therefore, the reference subspace computed fromKtK^\{t\}is unchanged\. In particular,UreftU\_\{\\mathrm\{ref\}\}^\{t\}spans the same consensus subspace before and after the gauge transformation\.

The GLoRA server update can be written as

Δ​Wgt=Ureft​\(Ureft\)⊤​∑i∈𝒞tpit​Δ​Wit\.\\Delta W\_\{g\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}\\Delta W\_\{i\}^\{t\}\.Both terms in this expression are invariant under the gauge transformations: the projector

Ureft​\(Ureft\)⊤U\_\{\\mathrm\{ref\}\}^\{t\}\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}is unchanged because the consensus subspace is unchanged, and each intrinsic client updateΔ​Wit\\Delta W\_\{i\}^\{t\}is unchanged as shown above\. Therefore,

Δ​W~gt=Δ​Wgt\.\\widetilde\{\\Delta W\}\_\{g\}^\{t\}=\\Delta W\_\{g\}^\{t\}\.Hence, the GLoRA server update is invariant to arbitrary client\-side gauge reparameterizations\.

Table 7:Summary of notation\.SymbolMeaningShapettCommunication roundScalariiClient indexScalar𝒞t\\mathcal\{C\}\_\{t\}Set of active clients at roundttSetpitp\_\{i\}^\{t\}Aggregation weight of clientiiat roundttScalardind\_\{\\mathrm\{in\}\}Input dimension of the adapted linear layerScalardoutd\_\{\\mathrm\{out\}\}Output dimension of the adapted linear layerScalarrrLoRA rank in the homogeneous settingScalarrir\_\{i\}LoRA rank of clientiiScalarRRServer\-side rank budgetScalarΔ​W\\Delta WLoRA adaptation matrixℝdout×din\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}BBLoRA left factorℝdout×r\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}AALoRA right factorℝr×din\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\}QQInvertible gauge transformation matrixℝr×r\\mathbb\{R\}^\{r\\times r\}GL​\(r\)\\mathrm\{GL\}\(r\)General linear group of invertibler×rr\\times rmatricesSet\[\(B,A\)\]\[\(B,A\)\]Gauge\-equivalence class of LoRA factorizationsSetΔ​Wit\\Delta W\_\{i\}^\{t\}Local LoRA update of clientiiat roundttℝdout×din\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}BitB\_\{i\}^\{t\}Clientii’s LoRA left factor at roundttℝdout×ri\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\}AitA\_\{i\}^\{t\}Clientii’s LoRA right factor at roundttℝri×din\\mathbb\{R\}^\{r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\}QiQ\_\{i\}Client\-specific gauge transformationℝri×ri\\mathbb\{R\}^\{r\_\{i\}\\times r\_\{i\}\}Ωt\\Omega^\{t\}Server state at roundttTupleΨ\\PsiServer update ruleFunctionUitU\_\{i\}^\{t\}Orthonormal basis of clientii’s update subspaceℝdout×ri\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\}TitT\_\{i\}^\{t\}Upper\-triangular factor from reduced QR decomposition ofBitB\_\{i\}^\{t\}ℝri×ri\\mathbb\{R\}^\{r\_\{i\}\\times r\_\{i\}\}A^it\\widehat\{A\}\_\{i\}^\{t\}Gauge\-fixed coordinate matrix of clientiiℝri×din\\mathbb\{R\}^\{r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\}IriI\_\{r\_\{i\}\}Identity matrix of sizerir\_\{i\}ℝri×ri\\mathbb\{R\}^\{r\_\{i\}\\times r\_\{i\}\}PitP\_\{i\}^\{t\}Orthogonal projector onto clientii’s update subspaceℝdout×dout\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{out\}\}\}KtK^\{t\}Weighted covariance / projector aggregation matrix for subspace estimationℝdout×dout\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{out\}\}\}MtM^\{t\}Thin matrix used to obtain the eigenspace ofKtK^\{t\}efficientlyℝdout×∑i∈𝒞tri\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}\}UreftU\_\{\\mathrm\{ref\}\}^\{t\}Server reference basis / consensus subspaceℝdout×R\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times R\}IRI\_\{R\}Identity matrix of sizeRRℝR×R\\mathbb\{R\}^\{R\\times R\}r∪tr\_\{\\cup\}^\{t\}Dimension of the union span of participating client subspacesScalarZitZ\_\{i\}^\{t\}Clientii’s projected coordinates in the reference basisℝR×din\\mathbb\{R\}^\{R\\times d\_\{\\mathrm\{in\}\}\}ZgtZ\_\{g\}^\{t\}Aggregated server coordinates in the reference basisℝR×din\\mathbb\{R\}^\{R\\times d\_\{\\mathrm\{in\}\}\}Δ​Wgt\\Delta W\_\{g\}^\{t\}Gauge\-aware server update induced by\(Ureft,Zgt\)\(U\_\{\\mathrm\{ref\}\}^\{t\},Z\_\{g\}^\{t\}\)ℝdout×din\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}OtO^\{t\}Left singular vectors ofZgtZ\_\{g\}^\{t\}in the coordinate spaceℝR×ℓ\\mathbb\{R\}^\{R\\times\\ell\}Σt\\Sigma^\{t\}Singular value matrix ofZgtZ\_\{g\}^\{t\}ℝℓ×ℓ\\mathbb\{R\}^\{\\ell\\times\\ell\}VtV^\{t\}Right singular vectors ofZgtZ\_\{g\}^\{t\}ℝdin×ℓ\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\\times\\ell\}UstU\_\{s\}^\{t\}Server spectral basis after mapping back to the original output spaceℝdout×ℓ\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times\\ell\}ℓ\\ellEffective spectral rank,ℓ=min⁡\(R,din\)\\ell=\\min\(R,d\_\{\\mathrm\{in\}\}\)Scalarσjt\\sigma\_\{j\}^\{t\}Thejj\-th singular value of the server updateScalarujtu\_\{j\}^\{t\}Thejj\-th left spectral direction of the server updateℝdout\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\}vjtv\_\{j\}^\{t\}Thejj\-th right spectral direction of the server updateℝdin\\mathbb\{R\}^\{d\_\{\\mathrm\{in\}\}\}γ\\gammaCore ratio for client\-aware readoutScalar in\[0,1\]\[0,1\]𝒢it\\mathcal\{G\}\_\{i\}^\{t\}Core spectral component set for clientiiSetHit−H\_\{i\}^\{t\-\}Latest historical update subspace basis of clientiiℝdout×hi\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times h\_\{i\}\}ai​jta\_\{ij\}^\{t\}Alignment score between clientii’s history and spectral directionjjScalarℐit\\mathcal\{I\}\_\{i\}^\{t\}Selected spectral component set for clientiiSet,\|ℐit\|=ri\|\\mathcal\{I\}\_\{i\}^\{t\}\|=r\_\{i\}Bi,inittB\_\{i,\\mathrm\{init\}\}^\{t\}Redistributed LoRA left factor for clientiiℝdout×ri\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\}Ai,inittA\_\{i,\\mathrm\{init\}\}^\{t\}Redistributed LoRA right factor for clientiiℝri×din\\mathbb\{R\}^\{r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\}

Algorithm 1GLoRA: Gauge\-aware LoRA Aggregation1:Initial model

W0W^\{0\}; total communication rounds

TT; server rank budget

RR; client ranks

\{ri\}\\\{r\_\{i\}\\\}; core ratio

γ\\gamma\.

2:Initialize client adapter initializations and previous\-round subspace bases

\{Hi0−\}\\\{H\_\{i\}^\{0\-\}\\\}\.

3:for

t=0,1,…,T−1t=0,1,\\ldots,T\-1do

4:Sample active client set

𝒞t\\mathcal\{C\}\_\{t\}\.

5:Send the current global model and the latest available adapter initialization to each client

i∈𝒞ti\\in\\mathcal\{C\}\_\{t\}\.

6:for allclient

i∈𝒞ti\\in\\mathcal\{C\}\_\{t\}in paralleldo

7:Perform local training and obtain LoRA factors

Bit∈ℝdout×ri,Ait∈ℝri×din\.B\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\},\\qquad A\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\}\.
8:Compute reduced QR decomposition:

Bit=Uit​Tit,\(Uit\)⊤​Uit=Iri\.B\_\{i\}^\{t\}=U\_\{i\}^\{t\}T\_\{i\}^\{t\},\\qquad\(U\_\{i\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}=I\_\{r\_\{i\}\}\.
9:Compute gauge\-fixed coordinates:

A^it=Tit​Ait\.\\widehat\{A\}\_\{i\}^\{t\}=T\_\{i\}^\{t\}A\_\{i\}^\{t\}\.
10:Send

\(Uit,A^it\)\(U\_\{i\}^\{t\},\\widehat\{A\}\_\{i\}^\{t\}\)to the server\.

11:endfor

12:

\(Ureft,Zgt\)←ProjectedAggregate​\(\{Uit,A^it,pit\}i∈𝒞t,R\)\(U\_\{\\mathrm\{ref\}\}^\{t\},Z\_\{g\}^\{t\}\)\\leftarrow\\textsc\{ProjectedAggregate\}\\big\(\\\{U\_\{i\}^\{t\},\\widehat\{A\}\_\{i\}^\{t\},p\_\{i\}^\{t\}\\\}\_\{i\\in\\mathcal\{C\}\_\{t\}\},R\\big\)\.

13:

\{\(Bi,initt,Ai,initt\)\}i∈𝒟t←ClientAwareReadout​\(Ureft,Zgt,\{ri\}i∈𝒟t,γ,\{Hit−\}i∈𝒟t\)\\\{\(B\_\{i,\\mathrm\{init\}\}^\{t\},A\_\{i,\\mathrm\{init\}\}^\{t\}\)\\\}\_\{i\\in\\mathcal\{D\}\_\{t\}\}\\leftarrow\\textsc\{ClientAwareReadout\}\\big\(U\_\{\\mathrm\{ref\}\}^\{t\},Z\_\{g\}^\{t\},\\\{r\_\{i\}\\\}\_\{i\\in\\mathcal\{D\}\_\{t\}\},\\gamma,\\\{H\_\{i\}^\{t\-\}\\\}\_\{i\\in\\mathcal\{D\}\_\{t\}\}\\big\)\.

14:Store current client subspaces for future rounds:

Hi\(t\+1\)−←Uit,∀i∈𝒞t\.H\_\{i\}^\{\(t\+1\)\-\}\\leftarrow U\_\{i\}^\{t\},\\qquad\\forall i\\in\\mathcal\{C\}\_\{t\}\.
15:endfor

16:Final global model\.

Algorithm 2ProjectedAggregate1:Gauge\-fixed client updates

\{Uit,A^it,pit\}i∈𝒞t\\\{U\_\{i\}^\{t\},\\widehat\{A\}\_\{i\}^\{t\},p\_\{i\}^\{t\}\\\}\_\{i\\in\\mathcal\{C\}\_\{t\}\}; server rank budget

RR\.

2:Construct the thin weighted subspace matrix:

Mt=\[pit​Uit\]i∈𝒞t\.M^\{t\}=\\big\[\\sqrt\{p\_\{i\}^\{t\}\}U\_\{i\}^\{t\}\\big\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\.
3:Compute the consensus reference basis:

Ureft=TopEigR​\(Mt​\(Mt\)⊤\),\(Ureft\)⊤​Ureft=IR\.U\_\{\\mathrm\{ref\}\}^\{t\}=\\mathrm\{TopEig\}\_\{R\}\\big\(M^\{t\}\(M^\{t\}\)^\{\\top\}\\big\),\\qquad\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{\\mathrm\{ref\}\}^\{t\}=I\_\{R\}\.
4:for allclient

i∈𝒞ti\\in\\mathcal\{C\}\_\{t\}do

5:Project the client update into the shared reference frame:

Zit=\(Ureft\)⊤​Uit​A^it\.Z\_\{i\}^\{t\}=\(U\_\{\\mathrm\{ref\}\}^\{t\}\)^\{\\top\}U\_\{i\}^\{t\}\\widehat\{A\}\_\{i\}^\{t\}\.
6:endfor

7:Aggregate shared coordinates:

Zgt=∑i∈𝒞tpit​Zit\.Z\_\{g\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}Z\_\{i\}^\{t\}\.
8:return

UreftU\_\{\\mathrm\{ref\}\}^\{t\}and

ZgtZ\_\{g\}^\{t\}\.

Algorithm 3ClientAwareReadout1:Server state

\(Ureft,Zgt\)\(U\_\{\\mathrm\{ref\}\}^\{t\},Z\_\{g\}^\{t\}\); target client ranks

\{ri\}i∈𝒟t\\\{r\_\{i\}\\\}\_\{i\\in\\mathcal\{D\}\_\{t\}\}; core ratio

γ\\gamma; previous\-round bases

\{Hit−\}i∈𝒟t\\\{H\_\{i\}^\{t\-\}\\\}\_\{i\\in\\mathcal\{D\}\_\{t\}\}\.

2:Compute thin SVD of the server coordinate matrix:

Zgt=Ot​Σt​\(Vt\)⊤,ℓ=min⁡\(R,din\)\.Z\_\{g\}^\{t\}=O^\{t\}\\Sigma^\{t\}\(V^\{t\}\)^\{\\top\},\\qquad\\ell=\\min\(R,d\_\{\\mathrm\{in\}\}\)\.
3:Map the spectral basis back to the original output space:

Ust=Ureft​Ot\.U\_\{s\}^\{t\}=U\_\{\\mathrm\{ref\}\}^\{t\}O^\{t\}\.
4:for allclient

i∈𝒟ti\\in\\mathcal\{D\}\_\{t\}do

5:Define the shared core component set:

𝒢it=\{1,…,⌊γ​ri⌋\}\.\\mathcal\{G\}\_\{i\}^\{t\}=\\\{1,\\ldots,\\lfloor\\gamma r\_\{i\}\\rfloor\\\}\.
6:if

Hit−H\_\{i\}^\{t\-\}is availablethen

7:Compute alignment scores for non\-core components:

ai​jt=‖\(Hit−\)⊤​ujt‖22,j∉𝒢it\.a\_\{ij\}^\{t\}=\\\|\(H\_\{i\}^\{t\-\}\)^\{\\top\}u\_\{j\}^\{t\}\\\|\_\{2\}^\{2\},\\qquad j\\notin\\mathcal\{G\}\_\{i\}^\{t\}\.
8:Select client\-aware components:

ℐit=𝒢it∪TopKri−\|𝒢it\|​\{ai​jt:j∉𝒢it\}\.\\mathcal\{I\}\_\{i\}^\{t\}=\\mathcal\{G\}\_\{i\}^\{t\}\\cup\\mathrm\{TopK\}\_\{r\_\{i\}\-\|\\mathcal\{G\}\_\{i\}^\{t\}\|\}\\\{a\_\{ij\}^\{t\}:j\\notin\\mathcal\{G\}\_\{i\}^\{t\}\\\}\.
9:else

10:Select components by global spectral order:

ℐit=\{1,…,ri\}\.\\mathcal\{I\}\_\{i\}^\{t\}=\\\{1,\\ldots,r\_\{i\}\\\}\.
11:endif

12:Initialize client

iiwith balanced LoRA factors:

Bi,initt=Ust​\[:,ℐit\]​diag​\(σℐitt\),B\_\{i,\\mathrm\{init\}\}^\{t\}=U\_\{s\}^\{t\}\[:,\\mathcal\{I\}\_\{i\}^\{t\}\]\\mathrm\{diag\}\\\!\\left\(\\sqrt\{\\sigma\_\{\\mathcal\{I\}\_\{i\}^\{t\}\}^\{t\}\}\\right\),Ai,initt=diag​\(σℐitt\)​\(Vt​\[:,ℐit\]\)⊤\.A\_\{i,\\mathrm\{init\}\}^\{t\}=\\mathrm\{diag\}\\\!\\left\(\\sqrt\{\\sigma\_\{\\mathcal\{I\}\_\{i\}^\{t\}\}^\{t\}\}\\right\)\\left\(V^\{t\}\[:,\\mathcal\{I\}\_\{i\}^\{t\}\]\\right\)^\{\\top\}\.
13:endfor

14:return

\{\(Bi,initt,Ai,initt\)\}i∈𝒟t\\\{\(B\_\{i,\\mathrm\{init\}\}^\{t\},A\_\{i,\\mathrm\{init\}\}^\{t\}\)\\\}\_\{i\\in\\mathcal\{D\}\_\{t\}\}\.

## Appendix BPseudo\-code of GLoRA

We here present the pseudo\-code of GLoRA\. Here𝒟t\\mathcal\{D\}\_\{t\}denotes the set of clients for which the server prepares rank\-compatible adapter initializations\. In practice, readout can be performed lazily when a client is selected in the next round\.

## Appendix CServer\-side Complexity

We analyze the per\-module server\-side complexity of different federated LoRA aggregation methods\. Consider one LoRA module whose induced update has dimension

Δ​Wit∈ℝdout×din\.\\Delta W\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\}\.Let𝒞t\\mathcal\{C\}\_\{t\}denote the participating client set at roundtt, and letrir\_\{i\}be the LoRA rank of clientii\. Following the notation in the main text, we define the total participating rank as

rΣ=∑i∈𝒞tri\.r\_\{\\Sigma\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}\.For homogeneous\-rank methods,ri=rr\_\{i\}=randrΣ=\|𝒞t\|​rr\_\{\\Sigma\}=\|\\mathcal\{C\}\_\{t\}\|r\. We also denote

m=min⁡\(dout,din\)\.m=\\min\(d\_\{\\mathrm\{out\}\},d\_\{\\mathrm\{in\}\}\)\.For the full model, the total cost is obtained by summing the following per\-module costs over all LoRA modules\.

#### FedIT\.

FedIT directly averages homogeneous LoRA factors

Ait∈ℝr×din,Bit∈ℝdout×r\.A\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{r\\times d\_\{\\mathrm\{in\}\}\},\\qquad B\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\}\.For each client, the server accumulatesr​din\+dout​rrd\_\{\\mathrm\{in\}\}\+d\_\{\\mathrm\{out\}\}rentries\. Therefore, the per\-module server cost is

𝒞FedIT=Θ​\(\|𝒞t\|​r​\(dout\+din\)\)=Θ​\(rΣ​\(dout\+din\)\)\.\\mathcal\{C\}\_\{\\mathrm\{FedIT\}\}=\\Theta\\bigl\(\|\\mathcal\{C\}\_\{t\}\|r\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\\bigr\)=\\Theta\\bigl\(r\_\{\\Sigma\}\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\\bigr\)\.This method is efficient but only supports homogeneous ranks\.

#### HetLoRA\.

HetLoRA supports heterogeneous ranks through truncation\-based factor aggregation\. The factor aggregation and rank readout terms scale linearly with the number of transmitted factor parameters\. In the sparse\-weighted version considered in this paper, the server additionally computes weights based on the dense low\-rank productsBit​AitB\_\{i\}^\{t\}A\_\{i\}^\{t\}\. For clientii, forming

Bit​Ait:\(dout×ri\)​\(ri×din\)→dout×dinB\_\{i\}^\{t\}A\_\{i\}^\{t\}:\(d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\)\(r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\)\\rightarrow d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}costs

Θ​\(dout​din​ri\)\.\\Theta\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}r\_\{i\}\)\.Summing over all participating clients gives

Θ​\(dout​din​∑i∈𝒞tri\)=Θ​\(dout​din​rΣ\)\.\\Theta\\left\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}r\_\{i\}\\right\)=\\Theta\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}r\_\{\\Sigma\}\)\.Sincedoutd\_\{\\mathrm\{out\}\}anddind\_\{\\mathrm\{in\}\}are typically much larger than the LoRA ranks, this dense\-product term dominates the lower\-order factor aggregation and readout costs\. Hence,

𝒞HetLoRA=Θ​\(dout​din​rΣ\)\.\\mathcal\{C\}\_\{\\mathrm\{HetLoRA\}\}=\\Theta\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}r\_\{\\Sigma\}\)\.

#### FlexLoRA\.

FlexLoRA first forms the dense semantic update

Δ​W¯t=∑i∈𝒞tpit​Bit​Ait\.\\bar\{\\Delta W\}^\{t\}=\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}p\_\{i\}^\{t\}B\_\{i\}^\{t\}A\_\{i\}^\{t\}\.Constructing the dense update costs

∑i∈𝒞tΘ​\(dout​din​ri\)=Θ​\(dout​din​rΣ\)\.\\sum\_\{i\\in\\mathcal\{C\}\_\{t\}\}\\Theta\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}r\_\{i\}\)=\\Theta\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}r\_\{\\Sigma\}\)\.It then performs a thin SVD of

Δ​W¯t∈ℝdout×din,\\bar\{\\Delta W\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times d\_\{\\mathrm\{in\}\}\},which costs

Θ​\(dout​din​m\),m=min⁡\(dout,din\)\.\\Theta\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}m\),\\qquad m=\\min\(d\_\{\\mathrm\{out\}\},d\_\{\\mathrm\{in\}\}\)\.Therefore, the dominant per\-module server cost is

𝒞FlexLoRA=Θ​\(dout​din​\(rΣ\+m\)\)\.\\mathcal\{C\}\_\{\\mathrm\{FlexLoRA\}\}=\\Theta\\bigl\(d\_\{\\mathrm\{out\}\}d\_\{\\mathrm\{in\}\}\(r\_\{\\Sigma\}\+m\)\\bigr\)\.This explains the large runtime overhead of FlexLoRA in Table[6](https://arxiv.org/html/2605.06733#S5.T6)\.

Table 8:Per\-module server\-side complexity\. The table reports the dominant terms used in the main text\.
#### GLoRA\.

GLoRA avoids dense\-update construction and performs aggregation in a gauge\-aware low\-rank reference space\. Each client update is first represented in a gauge\-fixed subspace–coordinate form

Bit=Uit​Tit,A^it=Tit​Ait,B\_\{i\}^\{t\}=U\_\{i\}^\{t\}T\_\{i\}^\{t\},\\qquad\\widehat\{A\}\_\{i\}^\{t\}=T\_\{i\}^\{t\}A\_\{i\}^\{t\},where

Uit∈ℝdout×ri,A^it∈ℝri×din\.U\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{i\}\},\\qquad\\widehat\{A\}\_\{i\}^\{t\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times d\_\{\\mathrm\{in\}\}\}\.The QR\-based gauge fixing is performed on the client side and is not counted as server\-side aggregation cost\.

On the server, GLoRA forms the weighted concatenated basis

Mt=\[pit​Uit\]i∈𝒞t∈ℝdout×rΣ\.M^\{t\}=\\big\[\\sqrt\{p\_\{i\}^\{t\}\}U\_\{i\}^\{t\}\\big\]\_\{i\\in\\mathcal\{C\}\_\{t\}\}\\in\\mathbb\{R\}^\{d\_\{\\mathrm\{out\}\}\\times r\_\{\\Sigma\}\}\.The Gram matrix

Gt=\(Mt\)⊤​Mt∈ℝrΣ×rΣG^\{t\}=\(M^\{t\}\)^\{\\top\}M^\{t\}\\in\\mathbb\{R\}^\{r\_\{\\Sigma\}\\times r\_\{\\Sigma\}\}costs

Θ​\(dout​rΣ2\),\\Theta\(d\_\{\\mathrm\{out\}\}r\_\{\\Sigma\}^\{2\}\),and its eigendecomposition costs

Θ​\(rΣ3\)\.\\Theta\(r\_\{\\Sigma\}^\{3\}\)\.The resulting reference basisUreftU\_\{\\mathrm\{ref\}\}^\{t\}is then used to translate client coordinates into the shared server frame\. Under the common settingR=O​\(rΣ\)R=O\(r\_\{\\Sigma\}\), this coordinate aggregation costs

Θ​\(rΣ2​din\)\.\\Theta\(r\_\{\\Sigma\}^\{2\}d\_\{\\mathrm\{in\}\}\)\.Therefore, the dominant per\-module server cost of GLoRA is

𝒞GLoRA=Θ​\(rΣ2​\(dout\+din\)\+rΣ3\)\.\\mathcal\{C\}\_\{\\mathrm\{GLoRA\}\}=\\Theta\\\!\\left\(r\_\{\\Sigma\}^\{2\}\(d\_\{\\mathrm\{out\}\}\+d\_\{\\mathrm\{in\}\}\)\+r\_\{\\Sigma\}^\{3\}\\right\)\.This compact form matches the complexity reported in Table[6](https://arxiv.org/html/2605.06733#S5.T6)\. SincerΣ≪dout,dinr\_\{\\Sigma\}\\ll d\_\{\\mathrm\{out\}\},d\_\{\\mathrm\{in\}\}in LoRA adaptation, GLoRA avoids the dense\-update and full\-SVD operations required by FlexLoRA while still supporting heterogeneous ranks\.