SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

# SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks
Source: [https://arxiv.org/html/2608.10144](https://arxiv.org/html/2608.10144)
Yue Xia Technical University of Munich Munich, 80333 Germany yue1\.xia@tum\.de &Tayyebeh Jahani\-Nezhad Technische Universität Berlin Berlin, 10587 Germany t\.jahani\.nezhad@tu\-berlin\.de &Mayank Bakshi Northern Arizona University Flagstaff, AZ 86011 USA mayank\.bakshi@nau\.edu &Rawad Bitar Technical University of Munich Munich, 80333 Germany rawad\.bitar@tum\.de

###### Abstract

We consider federated parameter efficient fine\-tuning of large neural networks with low\-rank adaptation \(LoRA, Hu et al\. 2022\)\. Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension\-incompatible, and factor\-wise averaging suffers from a*bilinear mismatch*\. We proposeSeFoRA, a sketch\-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator\. As a result,SeFoRAalleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model\. We introduce a rank\-homogeneous version calledSeFoRA\-Howhich allows for direct adapter aggregation in this setting\. We prove convergence to a neighborhood of the first\-order stationary point at rate𝒪​\(1/T\)\\mathcal\{O\}\(1/T\)for the rank\-homogeneous setting\. Numerical experiments on fine\-tuning RoBERTa\-Large on GLUE datasets show how our algorithms outperform the state\-of\-the\-art\.

## 1Introduction

Foundation models such as GPT\-4\[[1](https://arxiv.org/html/2608.10144#bib.bib1)\], BERT\[[2](https://arxiv.org/html/2608.10144#bib.bib2)\], PaLM 2\[[3](https://arxiv.org/html/2608.10144#bib.bib3)\], Llama 2\[[4](https://arxiv.org/html/2608.10144#bib.bib4)\], Claude 3\[[5](https://arxiv.org/html/2608.10144#bib.bib5)\], and ViT\[[6](https://arxiv.org/html/2608.10144#bib.bib6)\]have remarkable learning capabilities and performance across a diverse range of tasks\[[7](https://arxiv.org/html/2608.10144#bib.bib7),[8](https://arxiv.org/html/2608.10144#bib.bib8),[9](https://arxiv.org/html/2608.10144#bib.bib9),[10](https://arxiv.org/html/2608.10144#bib.bib10)\]\. Adapting such foundation models to downstream tasks relies on the availability of vast, potentially heterogeneous datasets and requires tremendous computational resources, especially when fine\-tuning all model parameters\. Since fine\-tuning all model parameters requires substantial computational and memory resources, parameter\-efficient fine\-tuning \(PEFT\) methods that adapt a pretrained model by updating only a small, structured subset of parameters, are employed, see\[[11](https://arxiv.org/html/2608.10144#bib.bib11),[12](https://arxiv.org/html/2608.10144#bib.bib12),[13](https://arxiv.org/html/2608.10144#bib.bib13),[14](https://arxiv.org/html/2608.10144#bib.bib14),[15](https://arxiv.org/html/2608.10144#bib.bib15),[16](https://arxiv.org/html/2608.10144#bib.bib16),[17](https://arxiv.org/html/2608.10144#bib.bib17),[18](https://arxiv.org/html/2608.10144#bib.bib18),[19](https://arxiv.org/html/2608.10144#bib.bib19),[20](https://arxiv.org/html/2608.10144#bib.bib20),[21](https://arxiv.org/html/2608.10144#bib.bib21),[22](https://arxiv.org/html/2608.10144#bib.bib22)\]\.

Training on large amounts of distributed data can be achieved using Federated learning \(FL\)\[[23](https://arxiv.org/html/2608.10144#bib.bib23)\]\. FL is a distributed learning paradigm in which a central server \(federator\) coordinates the training of a neural network on heterogeneous data generated and owned by participating clients\. The clients keep their data, train the neural network locally, and share only their model updates with a central server\. As such, FL enhances data privacy by keeping raw data local, reduces communication and storage costs associated with transferring data to a central server, and scales well due to its ease of implementation at the client\. As such, FL has attracted significant attention from the scientific community and is being applied across several domains\[[24](https://arxiv.org/html/2608.10144#bib.bib24),[25](https://arxiv.org/html/2608.10144#bib.bib25),[26](https://arxiv.org/html/2608.10144#bib.bib26)\]\.

Combining PEFT with FL span adapter\-based methods\[[27](https://arxiv.org/html/2608.10144#bib.bib27),[28](https://arxiv.org/html/2608.10144#bib.bib28)\], prompt tuning\[[29](https://arxiv.org/html/2608.10144#bib.bib29),[30](https://arxiv.org/html/2608.10144#bib.bib30)\], selective parameter optimization\[[31](https://arxiv.org/html/2608.10144#bib.bib31)\], and low\-rank gradient subspace optimization\[[32](https://arxiv.org/html/2608.10144#bib.bib32)\]\. We focus on federated PEFT methods using Low\-Rank Adaptation \(LoRA\)\[[17](https://arxiv.org/html/2608.10144#bib.bib17)\], which has emerged as one of the most popular PEFT techniques, e\.g\.,\[[19](https://arxiv.org/html/2608.10144#bib.bib19),[20](https://arxiv.org/html/2608.10144#bib.bib20),[33](https://arxiv.org/html/2608.10144#bib.bib33),[34](https://arxiv.org/html/2608.10144#bib.bib34),[35](https://arxiv.org/html/2608.10144#bib.bib35),[36](https://arxiv.org/html/2608.10144#bib.bib36),[37](https://arxiv.org/html/2608.10144#bib.bib37)\]and used in FL, e\.g\.,\[[38](https://arxiv.org/html/2608.10144#bib.bib38),[39](https://arxiv.org/html/2608.10144#bib.bib39),[40](https://arxiv.org/html/2608.10144#bib.bib40),[41](https://arxiv.org/html/2608.10144#bib.bib41),[42](https://arxiv.org/html/2608.10144#bib.bib42),[43](https://arxiv.org/html/2608.10144#bib.bib43),[44](https://arxiv.org/html/2608.10144#bib.bib44)\]\. A detailed literature review is provided in Section[6](https://arxiv.org/html/2608.10144#S6)\.

Combining LoRA with FL introduces challenges that do not arise in either centralized LoRA or conventional FL alone\. The main reason is that, unlike standard FL, LoRA updates cannot be averaged directly\. In LoRA, for each layer of the network, each clientiicomputes an update of the formΔ​𝐖i=𝐁i​𝐀i\\Delta\\mathbf\{W\}\_\{i\}=\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}, where the model updateΔ​𝐖i∈ℝd×s\\Delta\\mathbf\{W\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times s\}is represented as a multiplication of two factor matrices called*adapters*𝐁i∈ℝd×r\\mathbf\{B\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times r\},𝐀i∈ℝr×s\\mathbf\{A\}\_\{i\}\\in\\mathbb\{R\}^\{r\\times s\}, for a chosenr≪min⁡\(d,s\)r\\ll\\min\(d,s\)\. This factorization, together with the choice ofrr, reduces the number of trainable parameters per layer fromd​sdsto\(d\+s\)​r\(d\+s\)rand restricts the rank ofΔ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}torr111We omit the layer index as the operation is the same for all layers\. At the end, the clients concatenate its adapters across all layers and send them to the server\.\. In LoRA\-based FL, the server aims to obtainΔ​𝐖≜∑ipi​Δ​𝐖i=∑ipi​𝐁i​𝐀i\\Delta\\mathbf\{W\}\\triangleq\\sum\_\{i\}p\_\{i\}\\Delta\\mathbf\{W\}\_\{i\}=\\sum\_\{i\}p\_\{i\}\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}, where0≤pi≤10\\leq p\_\{i\}\\leq 1and∑ipi=1\\sum\_\{i\}p\_\{i\}=1\. TransmittingΔ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}to the server is generally undesirable, as it incurs the same communication cost as transmitting a full fine\-tuning update\. Instead, the clients can transmit𝐁i\\mathbf\{B\}\_\{i\}and𝐀i\\mathbf\{A\}\_\{i\}separately, and the server can aggregate them\. The server has two options:*\(i\)*compute eachΔ​𝐖i=𝐁i​𝐀i\\Delta\\mathbf\{W\}\_\{i\}=\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}’s and average them to obtainΔ​𝐖\\Delta\\mathbf\{W\}; or*\(ii\)*average the𝐀i\\mathbf\{A\}\_\{i\}’s and𝐁i\\mathbf\{B\}\_\{i\}’s first to obtain\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111=∑ipi​𝐀i\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}=\\sum\_\{i\}p\_\{i\}\\mathbf\{A\}\_\{i\}and\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111=∑ipi​𝐁i\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}=\\sum\_\{i\}p\_\{i\}\\mathbf\{B\}\_\{i\}and compute\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111=\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}=\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\.Each approach introduces its own challenges, affecting either the computational efficiency or the accuracy of the aggregated update, which in turn influences the convergence of the federated fine\-tuning algorithm\. We focus on the following challenges\.

##### Challenge 1: Rank\-heterogeneity

In the rank\-heterogeneous setting, clients use different adapter sizesrir\_\{i\}to match their local compute budgets or the complexity of their data, resulting in matrices with incompatible dimensions in aggregation\. Allowing for rank heterogeneity is crucial\. In the original LoRA work\[[17](https://arxiv.org/html/2608.10144#bib.bib17)\], it is shown that small adapter ranks can be sufficient for certain tasks\. However, this may not always work, especially when the downstream task differs significantly from the pretraining task\. Potential solutions include computing and averaging the𝐖i\\mathbf\{W\}\_\{i\}’s or zero\-padding the matrices to ensure dimension compatibility, both of which introduce approximation errors\.

##### Challenge 2: Bilinear mismatch

Zero\-padding and averaging the adapters first has a lower computational cost, but it does not recover the correct average of the updates, leading to the*bilinear mismatch*problem \(also known as aggregation bias\):\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111≠∑ipi​𝐁i​𝐀i\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\\neq\\sum\_\{i\}p\_\{i\}\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}\. The bilinear mismatch problem is exacerbated when considering differential privacy guarantees\[[40](https://arxiv.org/html/2608.10144#bib.bib40)\]and the non\-linearity of the aggregation does not allow for secure aggregation, typically used in privacy\-preserving FL\[[24](https://arxiv.org/html/2608.10144#bib.bib24)\]\.

##### Closely related work

We summarize the closest work on rank\-heterogeneous LoRA\-based federated PEFT here, provide more details in[Section˜2\.3](https://arxiv.org/html/2608.10144#S2.SS3)and give an extensive review in Section[6](https://arxiv.org/html/2608.10144#S6)\.

FedIT\[[38](https://arxiv.org/html/2608.10144#bib.bib38)\]introduced the use of LoRA without accounting for rank\-heterogeneity and the bilinear mismatch\. Since then, federated LoRA methods aim at addressing these challenges through various trade\-offs\. In FLoRA\[[45](https://arxiv.org/html/2608.10144#bib.bib45)\], the server stacks the adapters, forming larger matrices, and sends them to the clients\. This increases the server\-to\-client communication cost, but allows the clients to compute the full update\. A key difference is that the backbone model is not frozen\. Instead, the clients update the backbone model at every iteration and keep re\-initializing their adapters\. To overcome those limitations, in FlexLora\[[46](https://arxiv.org/html/2608.10144#bib.bib46)\]the server computes and averages the updates𝐖i=𝐁i​𝐀i\\mathbf\{W\}\_\{i\}=\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}’s to obtainΔ​𝐖\\Delta\\mathbf\{W\}and applies SVD onΔ​𝐖\\Delta\\mathbf\{W\}and truncates it to a rank\-rir\_\{i\}update matrix sent to clientii\. The computational overhead per layer at the server is𝒪​\(N​d​r​s\)\\mathcal\{O\}\(Ndrs\)for computing the𝐖i\\mathbf\{W\}\_\{i\}’s and𝒪​\(d2​s\)\\mathcal\{O\}\(d^\{2\}s\)for the SVD\. In\[[41](https://arxiv.org/html/2608.10144#bib.bib41)\], the authors remark that operating in the full model space, i\.e\., as is done in FlexLoRA, loses information on the cross\-relation across clients and only retains the knowledge on the cross\-relation between the adapters𝐁i\\mathbf\{B\}\_\{i\}and𝐀i\\mathbf\{A\}\_\{i\}\. This observation is corroborated in numerical experiments showing the underwhelming performance of such methods\. Therefore, in HetLoRA\[[41](https://arxiv.org/html/2608.10144#bib.bib41)\], the federator zero\-pads the clients’𝐀i\\mathbf\{A\}\_\{i\}’s and𝐁i\\mathbf\{B\}\_\{i\}’s and aggregates them using a careful weighted average\. The computational complexity of this method is𝒪​\(N​d​r​s\)\\mathcal\{O\}\(Ndrs\)for computing the weights and the global update\. Nevertheless, this method does not mitigate the bilinear mismatch problem\. In FSLoRA\[[47](https://arxiv.org/html/2608.10144#bib.bib47)\]the training objective is modified\. Sketching matrices are embedded into the forward pass as follows\. A global rankrris fixed, and random diagonalr×rr\\times rmatrices with onlyrir\_\{i\}non\-zero entries are chosen as sketching matrices called𝐒i\\mathbf\{S\}\_\{i\}\. Altering the update to𝐁i​𝐒i​𝐀i\\mathbf\{B\}\_\{i\}\\mathbf\{S\}\_\{i\}\\mathbf\{A\}\_\{i\}ensures that clientiitrains onlyrir\_\{i\}dimensions\.

To our knowledge, there is no rank\-heterogeneous LoRA\-based federated PEFT that mitigates the bilinear mismatch and does not operate in the full model space\. Besides, only FSLoRA\[[47](https://arxiv.org/html/2608.10144#bib.bib47)\]provides theoretical insights into the convergence of the rank\-heterogeneous algorithm for non\-convex loss functions\. However, this analysis comes at the cost of fixing the global matrix and choosing a random subspace to train per iteration, which reduces the flexibility of allowing the clients to choose their own subspaces to train on\. A convergence analysis for FLoRA is given for strongly convex loss functions, which does not match typical neural network optimization constraints\.

##### Our contributions

We introduceSeFoRA, a rank\-agnostic federated LoRA\-based PEFT method that accounts for rank heterogeneity, alleviates the bilinear mismatch, does not operate in the full model space, and enables linear aggregation at the federator\. Instead of transmitting𝐀i\\mathbf\{A\}\_\{i\}and𝐁i\\mathbf\{B\}\_\{i\}, each client sketches its local updateΔ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}into two compact, fixed\-size matrices using the sketching method proposed in\[[48](https://arxiv.org/html/2608.10144#bib.bib48)\]\. Since the sketching operator in\[[48](https://arxiv.org/html/2608.10144#bib.bib48)\]is linear, aggregating the sketches at the server is equivalent to computing the sketch of the aggregated updateΔ​𝐖\\Delta\\mathbf\{W\}exactly, thereby overcoming the bilinear mismatch problem with low computational overhead\. The main properties and advantages ofSeFoRAcan be summarized as follows:

1. 1\.SeFoRAaggregates fixed\-dimensional sketches, irrespective of clients’ local ranks\. Importantly, it can be seamlessly combined with adaptive rank\-truncation at the clients\.
2. 2\.We give a rank\-homogeneous version calledSeFoRA\-Hoand provide a convergence guarantee on the original LoRA objective\.SeFoRA\-Hoconverges at rate𝒪​\(1/T\+c\)\\mathcal\{O\}\(1/T\+c\), whereccis a sketching\-dependent error floor\. We bound this error using the spectral tail energy of the aggregated client updates\.
3. 3\.Numerical experiments fine\-tuning RoBERTa\-Large on GLUE datasets in the rank\-heterogeneous and rank\-homogeneous settings show thatSeFoRAandSeFoRA\-Hooutperform existing methods\. Interestingly, in some settings FlexLoRA andSeFoRAhave similar accuracies, despite the difference in computation cost at the server\. Similarly, FedIT andSeFoRA\-Hohave comparable accuracies in some settings\. However, FedIT does not allow for rank heterogeneity\.
4. 4\.Although this work is not directly concerned with privacy guarantees, mitigating the bilinear mismatch and allowing for linear aggregation open the door for secure aggregation and more efficient differential privacy mechanisms that we will explore in future work\.

Table 1:Comparison of rank\-heterogeneous federated LoRA aggregation methods\. We report the computation cost at the server in the regimed≫N\+rd\\gg N\+r\. The parameters are: sketch dimension byk=𝒪​\(r\)k=\\mathcal\{O\}\(r\), number of clientsNN, number of roundsTT, local LoRA rankrir\_\{i\}, and sketching\-ratio\-adjusted smoothnessL~\\widetilde\{L\}andccis a non\-negative constant\.MethodBilinearmismatchAggregationin full spaceServer computecostServer\-clientcommunication costConvergence⋆rateHetLoRA\[[41](https://arxiv.org/html/2608.10144#bib.bib41)\]YesNo𝒪​\(N​r​d​s\)\\mathcal\{O\}\(Nrds\)ri​\(d\+s\)r\_\{i\}\(d\+s\)—FlexLoRA\[[46](https://arxiv.org/html/2608.10144#bib.bib46)\]NoYes𝒪​\(N​\(d\+s\)\+d2​s\)\\mathcal\{O\}\(N\(d\+s\)\+d^\{2\}s\)ri​\(d\+s\)r\_\{i\}\(d\+s\)—FLoRA\[[45](https://arxiv.org/html/2608.10144#bib.bib45)\]NoYes𝒪​\(1\)\\mathcal\{O\}\(1\)∑iri​\(d\+s\)\\sum\_\{i\}r\_\{i\}\(d\+s\)𝒪​\(1/T\)\\mathcal\{O\}\(1/T\)FSLoRA\[[47](https://arxiv.org/html/2608.10144#bib.bib47)\]YesNo𝒪​\(N​r​\(d\+s\)\)\\mathcal\{O\}\(Nr\(d\{\+\}s\)\)r​\(d\+s\)r\(d\+s\)𝒪​\(L~/N​T\)\\mathcal\{O\}\(\\widetilde\{L\}/\\sqrt\{NT\}\)SeFoRA\(Ours\)NoNo𝒪​\(\(N\+r\)​r​\(d\+s\)\)\\mathcal\{O\}\\big\(\(N\{\+\}r\)\\,r\\,\(d\{\+\}s\)\\big\)d​r\+k​sdr\+ks𝒪​\(1/T\+c\)\\mathcal\{O\}\(1/T\+c\)⋆\\starFLoRA guarantees hold for strongly convex losses, FSLoRA guarantees hold for the modified training, and our guarantees hold for the rank\-homogeneous setting\.

## 2Setup and Problem Formulation

In this section, we formally define the setting and present the mathematical formulation of the problem\. In the sequel, we provide a detailed review of the literature, summarize our algorithm, and present a formal statement of our main results\.

### 2\.1Notation

For an integerNN, let\[N\]=1,⋯,N\[N\]=\{1,\\cdots,N\}\. Vectors and matrices are denoted by bold lowercase and uppercase letters, respectively\. For a matrix𝐀\\mathbf\{A\}with rankrr, we denote its singular values asσ1​\(𝐀\)≥⋯≥σr​\(𝐀\)≥0\\sigma\_\{1\}\(\\mathbf\{A\}\)\\geq\\cdots\\geq\\sigma\_\{r\}\(\\mathbf\{A\}\)\\geq 0\. Let‖𝐀‖2=σ1​\(𝐀\)\\\|\\mathbf\{A\}\\\|\_\{2\}=\\sigma\_\{1\}\(\\mathbf\{A\}\)denote the spectral norm and‖𝐀‖F=∑i=1rσi​\(𝐀\)2\\\|\\mathbf\{A\}\\\|\_\{F\}=\\sqrt\{\\sum\_\{i=1\}^\{r\}\\sigma\_\{i\}\(\\mathbf\{A\}\)^\{2\}\}the Frobenius norm\. For a matrix𝐀\\mathbf\{A\},𝐀:,1:r\\mathbf\{A\}\_\{:,1:r\}and𝐀1:r,:\\mathbf\{A\}\_\{1:r,:\}denote the submatrices formed by its firstrrcolumns and firstrrrows, respectively, and𝐀1:r,1:r\\mathbf\{A\}\_\{1:r,1:r\}denotes its leadingr×rr\\times rsubmatrix\. For a tuple of matrices𝙱=\(𝐁1,⋯,𝐁L\)\\mathbf\{\\mathtt\{B\}\}=\(\\mathbf\{B\}\_\{1\},\\cdots,\\mathbf\{B\}\_\{L\}\), we let‖𝙱‖2≜maxl∈\[L\]⁡‖𝐁l‖2\\\|\\mathbf\{\\mathtt\{B\}\}\\\|\_\{2\}\\triangleq\\max\_\{l\\in\[L\]\}\\\|\\mathbf\{B\}\_\{l\}\\\|\_\{2\},‖𝙱‖F≜∑l‖𝐁l‖F2\\\|\\mathbf\{\\mathtt\{B\}\}\\\|\_\{F\}\\triangleq\\sqrt\{\\sum\_\{l\}\\\|\\mathbf\{B\}\_\{l\}\\\|\_\{F\}^\{2\}\}, and we say that𝙱\\mathbf\{\\mathtt\{B\}\}has rankrrifrank⁡\(𝐁l\)≤r\\operatorname\{\\mathrm\{rank\}\}\(\\mathbf\{B\}\_\{l\}\)\\leq rfor everyl∈\[L\]l\\in\[L\]\. The transpose of a tuple𝙱\\mathbf\{\\mathtt\{B\}\}is denoted by𝙱⊤≜\(\(𝐁1\)⊤,⋯,\(𝐁L\)⊤\)\\mathbf\{\\mathtt\{B\}\}^\{\\top\}\\triangleq\(\(\\mathbf\{B\}\_\{1\}\)^\{\\top\},\\cdots,\(\\mathbf\{B\}\_\{L\}\)^\{\\top\}\)\. The product and addition of two tuples of matrices𝙰\\mathbf\{\\mathtt\{A\}\}and𝙱\\mathbf\{\\mathtt\{B\}\}are defined as𝙱𝙰=\(𝐁1​𝐀1,⋯,𝐁L​𝐀L\)\\mathbf\{\\mathtt\{B\}\}\\mathbf\{\\mathtt\{A\}\}=\(\\mathbf\{B\}\_\{1\}\\mathbf\{A\}\_\{1\},\\cdots,\\mathbf\{B\}\_\{L\}\\mathbf\{A\}\_\{L\}\)and𝙱\+𝙰=\(𝐁1\+𝐀1,⋯,𝐁L\+𝐀L\)\\mathbf\{\\mathtt\{B\}\}\+\\mathbf\{\\mathtt\{A\}\}=\(\\mathbf\{B\}\_\{1\}\+\\mathbf\{A\}\_\{1\},\\cdots,\\mathbf\{B\}\_\{L\}\+\\mathbf\{A\}\_\{L\}\), respectively\. The inner product of two tuples𝙰\\mathbf\{\\mathtt\{A\}\}and𝙱\\mathbf\{\\mathtt\{B\}\}is⟨𝙰,𝙱⟩≜∑l=1L⟨𝐀l,𝐁l⟩=∑l=1Ltr​\(\(𝐀l\)⊤​𝐁l\)\\langle\\mathbf\{\\mathtt\{A\}\},\\mathbf\{\\mathtt\{B\}\}\\rangle\\triangleq\\sum\_\{l=1\}^\{L\}\\langle\\mathbf\{A\}\_\{l\},\\mathbf\{B\}\_\{l\}\\rangle=\\sum\_\{l=1\}^\{L\}\\mathrm\{tr\}\(\{\(\\mathbf\{A\}\_\{l\}\)\}^\{\\top\}\\mathbf\{B\}\_\{l\}\)\.

### 2\.2Setup

##### Federated learning setup

We consider an FL system consisting of a central server \(the federator\) andNNclients, indexed byi∈\[N\]i\\in\[N\]\. Each clienti∈\[N\]i\\in\[N\]holds a local dataset𝒟i⊆𝒳×𝒴\\mathcal\{D\}\_\{i\}\\subseteq\\mathcal\{X\}\\times\\mathcal\{Y\}, with\|𝒟i\|\|\\mathcal\{D\}\_\{i\}\|being the number of data samples held by clientii\. Let𝒟=⋃i∈\[N\]𝒟i\\mathcal\{D\}=\\bigcup\_\{i\\in\[N\]\}\\mathcal\{D\}\_\{i\}denote the global dataset and let\(𝒙,y\)∈𝒟i\(\\bm\{x\},y\)\\in\\mathcal\{D\}\_\{i\}denote a single data sample in the dataset𝒟i\\mathcal\{D\}\_\{i\}, where𝒙\\bm\{x\}is the feature vector andyyis the label\. Letf​\(𝐖;𝒙\):𝒲×𝒳→𝒴f\(\\mathbf\{W\};\\bm\{x\}\):\\mathcal\{W\}\\times\\mathcal\{X\}\\to\\mathcal\{Y\}denote the output of a neural network parameterized by𝐖∈𝒲\\mathbf\{W\}\\in\\mathcal\{W\}for an input𝒙\\bm\{x\}\. Letℓ:𝒴×𝒴→ℝ\\ell:\\mathcal\{Y\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}be the sample\-wise loss function that measures the difference between the predicted labelf​\(𝐖;𝒙\)f\(\\mathbf\{W\};\\bm\{x\}\)and the true labelyy\. The local empirical riskℒi​\(𝐖\)\\mathcal\{L\}\_\{i\}\(\\mathbf\{W\}\)for clientiiis defined as

ℒi​\(𝐖\)≜1\|𝒟i\|​∑\(𝒙,y\)∈𝒟iℓ​\(f​\(𝐖;𝒙\),y\),\\displaystyle\\mathcal\{L\}\_\{i\}\(\\mathbf\{W\}\)\\triangleq\\frac\{1\}\{\|\\mathcal\{D\}\_\{i\}\|\}\\sum\_\{\(\\bm\{x\},y\)\\in\\mathcal\{D\}\_\{i\}\}\\ell\(f\(\\mathbf\{W\};\\bm\{x\}\),y\),and the global empirical risk is defined asℒ​\(𝐖\)≜∑i=1Npi​ℒi​\(𝐖\)\\mathcal\{L\}\(\\mathbf\{W\}\)\\triangleq\\sum\_\{i=1\}^\{N\}p\_\{i\}\\mathcal\{L\}\_\{i\}\(\\mathbf\{W\}\), where the weightspip\_\{i\}satisfy0≤pi≤10\\leq p\_\{i\}\\leq 1and∑i=1Npi=1\\sum\_\{i=1\}^\{N\}p\_\{i\}=1, and are typically chosen proportional to the local dataset sizes, i\.e\.,pi=\|𝒟i\|∑i=1N\|𝒟i\|p\_\{i\}=\\frac\{\|\\mathcal\{D\}\_\{i\}\|\}\{\\sum\_\{i=1\}^\{N\}\|\\mathcal\{D\}\_\{i\}\|\}\. The goal of FL is to collaboratively learn a model𝐖⋆\\mathbf\{W\}^\{\\star\}that minimizes the global empirical risk, i\.e\., find𝐖⋆∈argmin𝐖∈𝒲⁡ℒ​\(𝐖\)\\mathbf\{W\}^\{\\star\}\\in\\operatorname\{\\mathrm\{argmin\}\}\_\{\\mathbf\{W\}\\in\\mathcal\{W\}\}\\mathcal\{L\}\(\\mathbf\{W\}\)\.

##### Fine\-tuning and LoRA

To fine\-tune the pre\-trained model for a downstream task, we focus on a subset of the parameters\. Let a tuple of matrices𝚆=\(𝐖1,⋯,𝐖L\)\\mathbf\{\\mathtt\{W\}\}=\(\\mathbf\{W\}\_\{1\},\\cdots,\\mathbf\{W\}\_\{L\}\)denote theLLlinear layers selected for fine\-tuning, where for each layer𝐖l∈ℝdl×sl\\mathbf\{W\}\_\{l\}\\in\\mathbb\{R\}^\{d\_\{l\}\\times s\_\{l\}\}\. Let𝚆0=\(𝐖0,1,⋯,𝐖0,L\)\\mathbf\{\\mathtt\{W\}\}\_\{0\}=\(\\mathbf\{W\}\_\{0,1\},\\cdots,\\mathbf\{W\}\_\{0,L\}\)denote the pre\-trained values of𝚆\\mathbf\{\\mathtt\{W\}\}andΔ​𝚆=\(Δ​𝐖1,⋯,Δ​𝐖L\)\\Delta\\mathbf\{\\mathtt\{W\}\}=\(\\Delta\\mathbf\{W\}\_\{1\},\\cdots,\\Delta\\mathbf\{W\}\_\{L\}\)be the change of𝚆\\mathbf\{\\mathtt\{W\}\}during fine\-tuning, such that𝚆=𝚆0\+Δ​𝚆\\mathbf\{\\mathtt\{W\}\}=\\mathbf\{\\mathtt\{W\}\}\_\{0\}\+\\Delta\\mathbf\{\\mathtt\{W\}\}\.

In LoRA,𝚆0\\mathbf\{\\mathtt\{W\}\}\_\{0\}is frozen, i\.e\., it does not receive gradient updates during fine\-tuning\. For each layerl∈\[L\]l\\in\[L\], the weight changeΔ​𝐖l\\Delta\\mathbf\{W\}\_\{l\}is defined to be a low\-rank matrix, represented by a product of two factor matrices of smaller dimensions with a maximum rankrl≪min⁡\(dl,sl\)r\_\{l\}\\ll\\min\(d\_\{l\},s\_\{l\}\), i\.e\.,

𝐖l=𝐖0,l\+Δ​𝐖l=𝐖0,l\+𝐁l​𝐀l,\\displaystyle\\mathbf\{W\}\_\{l\}=\\mathbf\{W\}\_\{0,l\}\+\\Delta\\mathbf\{W\}\_\{l\}=\\mathbf\{W\}\_\{0,l\}\+\\mathbf\{B\}\_\{l\}\\mathbf\{A\}\_\{l\},whereΔ​𝐖l∈ℝdl×sl\\Delta\\mathbf\{W\}\_\{l\}\\in\\mathbb\{R\}^\{d\_\{l\}\\times s\_\{l\}\},𝐁l∈ℝdl×rl\\mathbf\{B\}\_\{l\}\\in\\mathbb\{R\}^\{d\_\{l\}\\times r\_\{l\}\}, and𝐀l∈ℝrl×sl\\mathbf\{A\}\_\{l\}\\in\\mathbb\{R\}^\{r\_\{l\}\\times s\_\{l\}\}\. Following standard practice\[[17](https://arxiv.org/html/2608.10144#bib.bib17)\], for each layerll, we initialize𝐀l\\mathbf\{A\}\_\{l\}with random Gaussian entries and𝐁l\\mathbf\{B\}\_\{l\}as the zero matrix, ensuringΔ​𝐖l=𝟎\\Delta\\mathbf\{W\}\_\{l\}=\\mathbf\{0\}at initialization\. We denote the tuple of low\-rank factor matrices for all the layers as𝙱=\(𝐁1,⋯,𝐁L\)\\mathbf\{\\mathtt\{B\}\}=\(\\mathbf\{B\}\_\{1\},\\cdots,\\mathbf\{B\}\_\{L\}\)and𝙰=\(𝐀1,⋯,𝐀L\)\\mathbf\{\\mathtt\{A\}\}=\(\\mathbf\{A\}\_\{1\},\\cdots,\\mathbf\{A\}\_\{L\}\)\.

##### Federated LoRA

In the federated LoRA setting, each clientiiperforms local LoRA fine\-tuning using its local tuples of low\-rank matrices𝙱i=\(𝐁i,1,⋯,𝐁i,L\)\\mathbf\{\\mathtt\{B\}\}\_\{i\}=\(\\mathbf\{B\}\_\{i,1\},\\cdots,\\mathbf\{B\}\_\{i,L\}\)and𝙰i=\(𝐀i,1,⋯,𝐀i,L\)\\mathbf\{\\mathtt\{A\}\}\_\{i\}=\(\\mathbf\{A\}\_\{i,1\},\\cdots,\\mathbf\{A\}\_\{i,L\}\), where the local rank profile𝐫i=\(ri,1,⋯,ri,L\)\\mathbf\{r\}\_\{i\}=\(r\_\{i,1\},\\cdots,r\_\{i,L\}\)is specific to each clientiiand may vary across iterations due to heterogeneity in data, computational and memory resources, and the relative importance of different layers during training\. A global rank profile𝐫=\(r1,⋯,rL\)\\mathbf\{r\}=\(r\_\{1\},\\cdots,r\_\{L\}\)is defined such thatrl=maxi∈\[N\]⁡ri,l\\displaystyle r\_\{l\}=\\max\_\{i\\in\[N\]\}r\_\{i,l\}for alll∈\[L\]l\\in\[L\]\. We define the local empirical loss for clientiievaluated at the local parameters\(𝙱i,𝙰i\)\(\\mathbf\{\\mathtt\{B\}\}\_\{i\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}\)as:

ℒilora​\(𝙱i,𝙰i;𝒟i\)≜1\|𝒟i\|​∑\(𝒙,y\)∈𝒟iℓ​\(f​\(𝚆0\+𝙱i​𝙰i;𝒙\),y\)\.\\displaystyle\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\},\\mathbf\{\\mathtt\{A\}\}\_\{i\};\\mathcal\{D\}\_\{i\}\)\\triangleq\\frac\{1\}\{\|\\mathcal\{D\}\_\{i\}\|\}\\sum\_\{\(\\bm\{x\},y\)\\in\\mathcal\{D\}\_\{i\}\}\\ell\(f\(\\mathbf\{\\mathtt\{W\}\}\_\{0\}\+\\mathbf\{\\mathtt\{B\}\}\_\{i\}\\mathbf\{\\mathtt\{A\}\}\_\{i\};\\bm\{x\}\),y\)\.
Let the global empirical loss for full fine\-tuning over the general update tupleΔ​𝚆\\Delta\\mathbf\{\\mathtt\{W\}\}be

ℒfull​\(Δ​𝚆;𝒟\)≜1\|𝒟\|​∑\(𝒙,y\)∈𝒟ℓ​\(f​\(𝚆0\+Δ​𝚆;𝒙\),y\)\.\\displaystyle\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\mathcal\{D\}\)\\triangleq\\frac\{1\}\{\|\\mathcal\{D\}\|\}\\sum\_\{\(\\bm\{x\},y\)\\in\\mathcal\{D\}\}\\ell\(f\(\\mathbf\{\\mathtt\{W\}\}\_\{0\}\+\\Delta\\mathbf\{\\mathtt\{W\}\};\\bm\{x\}\),y\)\.Under the LoRA factorization, the global empirical loss evaluated at the global parameters\(𝙱,𝙰\)\(\\mathbf\{\\mathtt\{B\}\},\\mathbf\{\\mathtt\{A\}\}\)is equivalent to the weighted sum of the local empirical losses at\(𝙱,𝙰\)\(\\mathbf\{\\mathtt\{B\}\},\\mathbf\{\\mathtt\{A\}\}\):

ℒlora​\(𝙱,𝙰;𝒟\)≜ℒfull​\(Δ​𝚆;𝒟\)=∑i=1Npi​ℒilora​\(𝙱,𝙰;𝒟i\),\\displaystyle\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\},\\mathbf\{\\mathtt\{A\}\};\\mathcal\{D\}\)\\triangleq\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\mathcal\{D\}\)=\\sum\_\{i=1\}^\{N\}p\_\{i\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\},\\mathbf\{\\mathtt\{A\}\};\\mathcal\{D\}\_\{i\}\),
To minimize the global objectiveℒlora​\(𝙱,𝙰\)\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\},\\mathbf\{\\mathtt\{A\}\}\), in each communication roundt∈\{0,⋯,T−1\}t\\in\\\{0,\\cdots,T\-1\\\}, the federator broadcasts the current global parameters\(𝙱\(t\),𝙰\(t\)\)\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)to the clients\. Each clienti∈\[N\]i\\in\[N\]initializes its local parameters as\(𝙱i\(t,0\),𝙰i\(t,0\)\)=\(𝙱\(t\),𝙰\(t\)\)\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,0\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,0\)\}\)=\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\), and performsEElocal steps of mini\-batch stochastic gradient descent \(SGD\)\. We denote by\(𝙱i\(t,e\),𝙰i\(t,e\)\)\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\)the local parameters at stepe∈\{0,⋯,E−1\}e\\in\\\{0,\\cdots,E\-1\\\}during communication roundtt\.

At each local stepee, clientiisamples a mini\-batch of dataξi\(e\)⊂𝒟i\\xi\_\{i\}^\{\(e\)\}\\subset\\mathcal\{D\}\_\{i\}and computes the stochastic gradient of the local objective\. The local update rules are given by:

𝙱i\(t,e\+1\)\\displaystyle\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\+1\)\}=𝙱i\(t,e\)−η\(t\)​∇𝙱ℒilora​\(𝙱i\(t,e\),𝙰i\(t,e\);ξi\(e\)\),\\displaystyle=\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\eta^\{\(t\)\}\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\),𝙰i\(t,e\+1\)\\displaystyle\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\+1\)\}=𝙰i\(t,e\)−η\(t\)​∇𝙰ℒilora​\(𝙱i\(t,e\),𝙰i\(t,e\);ξi\(e\)\),\\displaystyle=\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\eta^\{\(t\)\}\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\),whereη\(t\)\>0\\eta^\{\(t\)\}\>0is the learning rate at roundtt\. AfterEElocal steps, the resulting updated local parameters𝙱i\(t\+1\)=𝙱i\(t,E\)\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}=\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,E\)\}and𝙰i\(t\+1\)=𝙰i\(t,E\)\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}=\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,E\)\}are sent to the federator for aggregation\.

### 2\.3Problem formulation

##### Challenges of LoRA with FedAvg

Upon receiving the clients’𝙱i\(t\+1\)\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}and𝙰i\(t\+1\)\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}, the federator needs to compute a global𝙱\(t\+1\)\\mathbf\{\\mathtt\{B\}\}^\{\(t\+1\)\}and𝙰\(t\+1\)\\mathbf\{\\mathtt\{A\}\}^\{\(t\+1\)\}\. In rank\-homogeneous settings where all clients have the same rank for each layer, the federator could use vanilla FedAvg as in FedIT\[[38](https://arxiv.org/html/2608.10144#bib.bib38)\], i\.e\., averaging the𝙱i\(t\+1\)\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}and𝙰i\(t\+1\)\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}and sending them back to the clients\. However, this introduces additional errors in the global model update due to the bilinear mismatch, i\.e\., sinceΔ​𝚆\(t\+1\)≜\(∑i∈\[N\]pi​𝙱i\(t\+1\)\)​\(∑i∈\[N\]pi​𝙰i\(t\+1\)\)≠\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111\(t\+1\)≜∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\+1\)\}\\triangleq\\big\(\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\big\)\\big\(\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\\big\)\\neq\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{\(t\+1\)\}\\triangleq\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\. In rank\-heterogeneous settings, the adapters have different dimensions and averaging is not directly possible\. Hence, an alternative solution is to operate in the full model space\. In\[[45](https://arxiv.org/html/2608.10144#bib.bib45)\], the federator horizontally concatenates the𝙱i\(t\+1\)\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}matrices and vertically concatenates the𝙰i\(t\+1\)\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}matrices to form two large matrices sent to the clients\. Each client computes\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111\(t\+1\)\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{\(t\+1\)\}locally and applies it to the backbone model𝚆0\\mathbf\{\\mathtt\{W\}\}\_\{0\}, thereby updating the backbone parameters\. The main drawbacks are an additional communication overhead and not freezing the backbone model\. Hence, in FlexLoRA\[[39](https://arxiv.org/html/2608.10144#bib.bib39)\], the federator computes\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111\(t\+1\)\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{\(t\+1\)\}at the server and, using SVD, truncates it to fit each client’s local rank profile\. However, as discussed and corroborated numerically in\[[41](https://arxiv.org/html/2608.10144#bib.bib41)\], computing and averaging the\\macc@depthΔ\\macc@set@skewchar\\macc@nested@a111=i𝙱i𝙰i\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\_\{i\}=\\mathbf\{\\mathtt\{B\}\}\_\{i\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}loses information on the cross\-relation across clients and only retains the knowledge on the cross\-relation between the adapters𝙱i\\mathbf\{\\mathtt\{B\}\}\_\{i\}and𝙰i\\mathbf\{\\mathtt\{A\}\}\_\{i\}\. Thus, in HetLoRA, the federator zero\-pads the𝙱i\(t\+1\)\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}and𝙰i\(t\+1\)\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}to ensure they have the same dimension, aggregates them to obtainΔ​𝚆\(t\+1\)\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\+1\)\}by determining the weightspip\_\{i\}’s according to the norm of the singular value vector ofΔ​𝚆i\(t\+1\)\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t\+1\)\}\. Then, similarly to FlexLoRA, the federator uses SVD to truncateΔ​𝚆\(t\+1\)\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\+1\)\}to the client’s local rank profile\. On top of the computation overhead at the federator introduced by computing theΔ​𝚆i\(t\+1\)\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t\+1\)\}, this method does not resolve the bilinear mismatch problem\. Furthermore, zero\-padding the matrices is not always faithful to the rank representation of the clients, which is why a careful computation of thepip\_\{i\}’s is needed and why other works, e\.g\.,\[[49](https://arxiv.org/html/2608.10144#bib.bib49),[50](https://arxiv.org/html/2608.10144#bib.bib50)\]replace zero\-padding with column repetition\.

Therefore, the question that remains unanswered is:

*Research question:Is it possible to construct a rank\-heterogeneous LoRA\-based FL method that mitigates the bilinear mismatch and does not operate in the full model space?*

We answer this question in the affirmative as we explain next\.

## 3Our Algorithm and Contributions

We jointly tackle rank heterogeneity and bilinear mismatch without operating in the full model space\. The main ingredient of our algorithm is operating in a sketched\-space of the full model using the linear matrix sketching method introduced in\[[48](https://arxiv.org/html/2608.10144#bib.bib48)\]\. For ease of presentation, we describe the method for a single layer\. The same procedure is applied independently to every layer, and the resulting matrices will be concatenated at the end\. Therefore, we omit the layer indexll\. Each client sketches \(compresses\) its local updateΔ​𝐖i∈ℝd×s\\Delta\\mathbf\{W\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times s\}into two matrices,𝐘i∈ℝd×r\\mathbf\{Y\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times r\}and𝐙i∈ℝk×s\\mathbf\{Z\}\_\{i\}\\in\\mathbb\{R\}^\{k\\times s\}, wherer=rl=maxi∈\[N\]⁡ri,lr=r\_\{l\}=\\max\_\{i\\in\[N\]\}r\_\{i,l\}is the maximum local rank for each layer andkkis a constant satisfyingk\>r\+1k\>r\+1\. By aggregating the𝐘i\\mathbf\{Y\}\_\{i\}and𝐙i\\mathbf\{Z\}\_\{i\}sent by the clients, the federator obtains sketches of the global updateΔ​𝐖=∑i∈\[N\]pi​Δ​𝐖i\\Delta\\mathbf\{W\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\Delta\\mathbf\{W\}\_\{i\}, from which it computes a low\-rank approximation ofΔ​𝐖\\Delta\\mathbf\{W\}, as shown in[Algorithm˜1](https://arxiv.org/html/2608.10144#alg1)\. Therefore, this maps local adapters of rankrir\_\{i\}to sketches of dimensionrrandkk, respectively, thereby enabling linear aggregation and bypassing the bilinear mismatch while avoiding the aggregation in the full spaceΔ​𝐖\\Delta\\mathbf\{W\}\. Similar to FSLoRA, this incurs an additional communication overhead proportional to\(r−ri\)\(r\-r\_\{i\}\)and\(k−ri\)\(k\-r\_\{i\}\)\. Before presenting our theoretical guarantees and demonstrating that our method outperforms the state of the art, we first explain the sketching method introduced in\[[48](https://arxiv.org/html/2608.10144#bib.bib48)\]\.

##### Sketching as Low\-Rank Approximation\[[48](https://arxiv.org/html/2608.10144#bib.bib48)\]

Suppose𝐖∈ℝd×s\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\\times s\}is an arbitrary matrix\. Letr≪min⁡\{d,s\}r\\ll\\min\\\{d,s\\\}be the target rank\. Given a sketch parameterkk, let𝛀∈ℝs×r\\mathbf\{\\Omega\}\\in\\mathbb\{R\}^\{s\\times r\}and𝚿∈ℝk×d\\mathbf\{\\Psi\}\\in\\mathbb\{R\}^\{k\\times d\}be independent random matrices whose entries are drawn independently from the standard normal distribution, two*sketch matrices*of𝐖\\mathbf\{W\}are produced via left and right matrix multiplication as:

𝐘≜𝐖​𝛀and𝐙≜𝚿​𝐖\.\\displaystyle\\mathbf\{Y\}\\triangleq\\mathbf\{W\}\\mathbf\{\\Omega\}\\quad\\text\{and\}\\quad\\mathbf\{Z\}\\triangleq\\mathbf\{\\Psi\}\\mathbf\{W\}\.
Given the sketch matrices,𝐘∈ℝd×r\\mathbf\{Y\}\\in\\mathbb\{R\}^\{d\\times r\}, and𝐙∈ℝk×s\\mathbf\{Z\}\\in\\mathbb\{R\}^\{k\\times s\}, the low\-rank approximation𝐖^\\widehat\{\\mathbf\{W\}\}of𝐖\\mathbf\{W\}with rankrris computed using the following steps:*\(i\)*factorize𝐘\\mathbf\{Y\}into𝐘=𝐁^​𝐑\\mathbf\{Y\}=\\widehat\{\\mathbf\{B\}\}\\mathbf\{R\}via QR decomposition;*\(ii\)*compute𝚿​𝐁^\\mathbf\{\\Psi\}\\widehat\{\\mathbf\{B\}\}and find the QR decomposition\(𝚿​𝐁^\)=𝐔𝐓\(\\mathbf\{\\Psi\}\\widehat\{\\mathbf\{B\}\}\)=\\mathbf\{U\}\\mathbf\{T\};*\(iii\)*compute𝐀^=𝐓−1​\(𝐔T​𝐙\)\\widehat\{\\mathbf\{A\}\}=\\mathbf\{T\}^\{\-1\}\(\\mathbf\{U\}^\{T\}\\mathbf\{Z\}\); and,*\(iv\)*produce the low\-rank approximation𝐖^=𝐁^​𝐀^\\widehat\{\\mathbf\{W\}\}=\\widehat\{\\mathbf\{B\}\}\\widehat\{\\mathbf\{A\}\}\.This sketch\-inversion procedure is summarized in[Algorithm˜1](https://arxiv.org/html/2608.10144#alg1)\.

Algorithm 1Sketch\-inversion Algorithm1:Given two random matrices

𝚿∈ℝk×d\\mathbf\{\\Psi\}\\in\\mathbb\{R\}^\{k\\times d\}and

𝛀∈ℝs×r\\mathbf\{\\Omega\}\\in\\mathbb\{R\}^\{s\\times r\}drawn from a standard normal distribution, and two matrices

𝐘∈ℝd×r\\mathbf\{Y\}\\in\\mathbb\{R\}^\{d\\times r\}and

𝐙∈ℝk×s\\mathbf\{Z\}\\in\\mathbb\{R\}^\{k\\times s\}, where

𝐘=𝐖​𝛀\\mathbf\{Y\}=\\mathbf\{W\}\\mathbf\{\\Omega\}and

𝐙=𝚿​𝐖\\mathbf\{Z\}=\\mathbf\{\\Psi\}\\mathbf\{W\}are two sketch matrices produced from the target large matrix

𝐖∈ℝd×s\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\\times s\}\.

2:Return the low\-rank factors

𝐁^∈ℝd×r\\widehat\{\\mathbf\{B\}\}\\in\\mathbb\{R\}^\{d\\times r\}and

𝐀^∈ℝr×s\\widehat\{\\mathbf\{A\}\}\\in\\mathbb\{R\}^\{r\\times s\}of the target matrix

𝐖\\mathbf\{W\}\.

3:functionUnsketching\(

𝐘,𝐙,𝚿,𝛀\\mathbf\{Y\},\\mathbf\{Z\},\\mathbf\{\\Psi\},\\mathbf\{\\Omega\}\)

4:

\(𝐁^,∼\)←QR​\(𝐘\)\(\\widehat\{\\mathbf\{B\}\},\\sim\)\\leftarrow\\mathrm\{QR\}\(\\mathbf\{Y\}\)
5:

\(𝐔,𝐓\)←QR​\(𝚿​𝐁^\)\(\\mathbf\{U\},\\mathbf\{T\}\)\\leftarrow\\mathrm\{QR\}\(\\mathbf\{\\Psi\}\\widehat\{\\mathbf\{B\}\}\)
6:

𝐀^=𝐓−1​\(𝐔T​𝐙\)\\widehat\{\\mathbf\{A\}\}=\\mathbf\{T\}^\{\-1\}\(\\mathbf\{U\}^\{T\}\\mathbf\{Z\}\)
7:Return

\(𝐁^,𝐀^\)\(\\widehat\{\\mathbf\{B\}\},\\widehat\{\\mathbf\{A\}\}\)
8:endfunction

Algorithm 2SeFoRA1:The pretrained model

𝐖0\\mathbf\{W\}\_\{0\}\. Total rounds

TT, local steps

EE, and local rank updating threshold

τ\\tau\. In total,

NNclients, each client

iiholding their local dataset

𝒟i\\mathcal\{D\}\_\{i\}\. The aggregation weights

pi=\|𝒟i\|/∑i=1N\|𝒟i\|p\_\{i\}=\|\\mathcal\{D\}\_\{i\}\|/\\sum\_\{i=1\}^\{N\}\|\\mathcal\{D\}\_\{i\}\|\. The local rank profiles

𝐫i\(0\)=\(ri,1\(0\),⋯,ri,L\(0\)\)\\mathbf\{r\}\_\{i\}^\{\(0\)\}=\(r\_\{i,1\}^\{\(0\)\},\\cdots,r\_\{i,L\}^\{\(0\)\}\)\. For

l∈\[L\]l\\in\[L\], set

rl≜maxi∈\[N\]⁡ri,l\(0\)r\_\{l\}\\triangleq\\max\_\{i\\in\[N\]\}r\_\{i,l\}^\{\(0\)\}and

kl\>rl\+1k\_\{l\}\>r\_\{l\}\+1\. Set

rmax≜maxl∈\[L\]⁡rlr\_\{\\max\}\\triangleq\\max\_\{l\\in\[L\]\}r\_\{l\}, and

kmax≜maxl∈\[L\]⁡klk\_\{\\max\}\\triangleq\\max\_\{l\\in\[L\]\}k\_\{l\}\. Initialize global matrices

𝐁l\(0\)∈ℝd×rl\\mathbf\{B\}\_\{l\}^\{\(0\)\}\\in\\mathbb\{R\}^\{d\\times r\_\{l\}\}as a full zero matrix and

𝐀l\(0\)∈ℝrl×s\\mathbf\{A\}\_\{l\}^\{\(0\)\}\\in\\mathbb\{R\}^\{r\_\{l\}\\times s\}as a random Gaussian matrix\.

𝙱\(0\)=\(𝐁1\(0\),⋯,𝐁L\(0\)\)\\mathbf\{\\mathtt\{B\}\}^\{\(0\)\}=\(\\mathbf\{B\}\_\{1\}^\{\(0\)\},\\cdots,\\mathbf\{B\}\_\{L\}^\{\(0\)\}\)and

𝙰\(0\)=\(𝐀1\(0\),⋯,𝐀L\(0\)\)\\mathbf\{\\mathtt\{A\}\}^\{\(0\)\}=\(\\mathbf\{A\}\_\{1\}^\{\(0\)\},\\cdots,\\mathbf\{A\}\_\{L\}^\{\(0\)\}\)\. A function

Truncate​\(𝐁,𝐀,ri\)\\textsc\{Truncate\}\(\\mathbf\{B\},\\mathbf\{A\},r\_\{i\}\)that truncates LoRA factors with rank

rrto rank

ri≤rr\_\{i\}\\leq r\([Algorithm˜3](https://arxiv.org/html/2608.10144#alg3)\), a function

UpdateLocalRank​\(𝐁i,𝐀i,τ\)\\textsc\{UpdateLocalRank\}\(\\mathbf\{B\}\_\{i\},\\mathbf\{A\}\_\{i\},\\tau\)that updates the client’s local rank

rir\_\{i\}and truncate the factors to rank

rir\_\{i\}\([Algorithm˜4](https://arxiv.org/html/2608.10144#alg4)\), and a function

Unsketching​\(𝐘,𝐙,𝚿,𝛀\)\\textsc\{Unsketching\}\(\\mathbf\{Y\},\\mathbf\{Z\},\\mathbf\{\\Psi\},\\mathbf\{\\Omega\}\)that unsketch the sketch matrices \([Algorithm˜1](https://arxiv.org/html/2608.10144#alg1)\)\.

2:forround

t=0,1,⋯,T−1t=0,1,\\cdots,T\-1do

3:Federator initializes

𝚿\(t\+1\)∈ℝkmax×d\\mathbf\{\\Psi\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{k\_\{\\max\}\\times d\}and

𝛀\(t\+1\)∈ℝs×rmax\\mathbf\{\\Omega\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{s\\times r\_\{\\max\}\}from a standard normal distribution, and send the shared seed that generates

𝚿\(t\+1\)\\mathbf\{\\Psi\}^\{\(t\+1\)\}and

𝛀\(t\+1\)\\mathbf\{\\Omega\}^\{\(t\+1\)\}to all clients\.

4:Federator broadcasts

𝙱\(t\)\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}and

𝙰\(t\)\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}to all clients\.

5:for allclients

iiin paralleldo

6:for alladapter layer

l∈\[L\]l\\in\[L\]do

7:Extract

𝛀l\(t\+1\)=𝛀:,1:rl\(t\+1\)∈ℝs×rl\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\\\!\\\!=\\\!\\mathbf\{\\Omega\}^\{\(t\+1\)\}\_\{:,1:r\_\{l\}\}\\\!\\in\\\!\\mathbb\{R\}^\{s\\times r\_\{l\}\}and

𝚿l\(t\+1\)=𝚿1:kl,:\(t\+1\)∈ℝkl×d\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\\!\\\!=\\\!\\mathbf\{\\Psi\}^\{\(t\+1\)\}\_\{1:k\_\{l\},:\}\\\!\\in\\\!\\mathbb\{R\}^\{k\_\{l\}\\times d\}from the shared seed\.

8:Obtain

𝐁l\(t\)\\mathbf\{B\}\_\{l\}^\{\(t\)\}and

𝐀l\(t\)\\mathbf\{A\}\_\{l\}^\{\(t\)\}\.

9:Initialize local LoRA factors as:

10:ifUpdating local ranksthen

11:Set local rank

ri,l\(t\)=ri,l\(0\)r\_\{i,l\}^\{\(t\)\}=r\_\{i,l\}^\{\(0\)\}
12:end if

13:iflocal rank

ri,l\(t\)<rlr\_\{i,l\}^\{\(t\)\}<r\_\{l\}then

14:

\(𝐁i,l\(t\),𝐀i,l\(t\)\)←Truncate​\(𝐁l\(t\),𝐀l\(t\),ri,l\(t\)\)\(\\mathbf\{B\}\_\{i,l\}^\{\(t\)\},\\mathbf\{A\}\_\{i,l\}^\{\(t\)\}\)\\leftarrow\\textsc\{Truncate\}\(\\mathbf\{B\}\_\{l\}^\{\(t\)\},\\mathbf\{A\}\_\{l\}^\{\(t\)\},r\_\{i,l\}^\{\(t\)\}\)
15:else

16:Set

𝐁i,l\(t\)=𝐁l\(t\)\\mathbf\{B\}\_\{i,l\}^\{\(t\)\}=\\mathbf\{B\}\_\{l\}^\{\(t\)\}and

𝐀i,l\(t\)=𝐀l\(t\)\\mathbf\{A\}\_\{i,l\}^\{\(t\)\}=\\mathbf\{A\}\_\{l\}^\{\(t\)\}
17:end if

18:endfor

19:Using

𝐁i,l\(t\)\\mathbf\{B\}\_\{i,l\}^\{\(t\)\}and

𝐀i,l\(t\)\\mathbf\{A\}\_\{i,l\}^\{\(t\)\}, perform

EElocal steps to obtain

𝐁i,l\(t\+1\)\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}and

𝐀i,l\(t\+1\)\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}for all

l∈\[L\]l\\in\[L\]\.

20:for alladapter layer

l∈\[L\]l\\in\[L\]do

21:Rank update:

22:ifUpdating local ranksthen

23:

\(𝐁i,l\(t\+1\),𝐀i,l\(t\+1\)\),ri,l\(t\+1\)←UpdateLocalRank​\(𝐁i,l\(t\+1\),𝐀i,l\(t\+1\),τ\)\(\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\},\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\),r\_\{i,l\}^\{\(t\+1\)\}\\leftarrow\\textsc\{UpdateLocalRank\}\(\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\},\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\},\\tau\)
24:end if

25:Compute the sketch matrices

𝐘i,l\(t\+1\)\\displaystyle\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,l\}=Δ​𝐖i,l\(t\+1\)​𝛀l\(t\+1\)=𝐁i,l\(t\+1\)​\(𝐀i,l\(t\+1\)​𝛀l\(t\+1\)\)∈ℝd×rl,\\displaystyle=\\Delta\\mathbf\{W\}^\{\(t\+1\)\}\_\{i,l\}\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}=\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\(\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\)\\in\\mathbb\{R\}^\{d\\times r\_\{l\}\},𝐙i,l\(t\+1\)\\displaystyle\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,l\}=𝚿l\(t\+1\)​Δ​𝐖i,l\(t\+1\)=\(𝚿l\(t\+1\)​𝐁i,l\(t\+1\)\)​𝐀i,l\(t\+1\)∈ℝkl×s\.\\displaystyle=\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\Delta\\mathbf\{W\}^\{\(t\+1\)\}\_\{i,l\}=\(\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\)\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{k\_\{l\}\\times s\}\.
26:endfor

27:Send

𝚈i\(t\+1\)=\(𝐘i,1\(t\+1\),⋯,𝐘i,L\(t\+1\)\)\\mathbf\{\\mathtt\{Y\}\}^\{\(t\+1\)\}\_\{i\}=\(\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,1\},\\cdots,\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,L\}\)and

𝚉i\(t\+1\)=\(𝐙i,1\(t\+1\),⋯,𝐙i,L\(t\+1\)\)\\mathbf\{\\mathtt\{Z\}\}^\{\(t\+1\)\}\_\{i\}=\(\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,1\},\\cdots,\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,L\}\)to federator\.

28:endfor

29:for alladapter layer

l∈\[L\]l\\in\[L\]do

30:Federator aggregates

𝐘l\(t\+1\)=∑i∈\[N\]pi​𝐘i,l\(t\+1\)​and​𝐙l\(t\+1\)=∑i∈\[N\]pi​𝐙i,l\(t\+1\)\\mathbf\{Y\}\_\{l\}^\{\(t\+1\)\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,l\}\\;\\text\{ and \}\\;\\mathbf\{Z\}\_\{l\}^\{\(t\+1\)\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,l\}\.

31:

\(𝐁l\(t\+1\),𝐀l\(t\+1\)\)←Unsketching​\(𝐘l\(t\+1\),𝐙l\(t\+1\),𝚿l\(t\+1\),𝛀l\(t\+1\)\)\(\\mathbf\{B\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{A\}\_\{l\}^\{\(t\+1\)\}\)\\leftarrow\\textsc\{Unsketching\}\(\\mathbf\{Y\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{Z\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\)
32:endfor

33:endfor

34:Return

\(𝙱\(T\),𝙰\(T\)\)\(\\mathbf\{\\mathtt\{B\}\}^\{\(T\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(T\)\}\)

### 3\.1SeFoRA

SeFoRAresolves rank\-heterogeneity and bilinear mismatch as follows\.

##### Client rank

Each client sets its own local rank profile𝐫i=\(ri,1,⋯,ri,L\)\\mathbf\{r\}\_\{i\}=\(r\_\{i,1\},\\cdots,r\_\{i,L\}\)\.SeFoRAis agnostic to how the clients adjust their local ranks\. They may use adaptive techniques such as those proposed in\[[51](https://arxiv.org/html/2608.10144#bib.bib51),[52](https://arxiv.org/html/2608.10144#bib.bib52)\], adapting the ranks based on their computational and memory resources as well as the contribution of each layer to fine\-tuning in terms of effective rank\. Since the local rank may change across rounds and may vary across layers, we formally denote it byri,l\(t\)r\_\{i,l\}^\{\(t\)\}\.

##### Client rank update

Clients can periodically update the rank of their adapters, e\.g\., everyTuT\_\{u\}rounds, to match their computational and memory constraints, and data complexity\. As a concrete example, inSeFoRA, we choose the rank update method shown in[Algorithm˜4](https://arxiv.org/html/2608.10144#alg4)\. This method can be replaced by other rank\-update strategies and is chosen since it allows the local ranks to grow as well as shrink\.

#### 3\.1\.1Algorithm procedure

The procedure is summarized in[Algorithm˜2](https://arxiv.org/html/2608.10144#alg2)and is explained in detail next\. For notational simplicity, we present the algorithm under the assumption that all adapted layers share the same input and output dimensions, i\.e\.,dl=dd\_\{l\}=dandsl=ss\_\{l\}=s\. The method extends directly to heterogeneous dimensions by using compatible sizes\.

##### Initialization

The federator has the pretrained model𝚆0\\mathbf\{\\mathtt\{W\}\}\_\{0\}\. Each client sets an initial local rank profile𝐫i\(0\)=\(ri,1\(0\),⋯,ri,L\(0\)\)\\mathbf\{r\}\_\{i\}^\{\(0\)\}=\(r\_\{i,1\}^\{\(0\)\},\\cdots,r\_\{i,L\}^\{\(0\)\}\)based on local compute and memory constraints, i\.e\.,ri,l\(0\)r\_\{i,l\}^\{\(0\)\}is the maximum rank that clientiican allocate to layerll\. This profile contains the maximum rank each client can allocate per layer and will be communicated to the federator\. The federator creates a global rank profile𝐫\(0\)=\(r1\(0\),⋯,rL\(0\)\)\\mathbf\{r\}^\{\(0\)\}=\(r\_\{1\}^\{\(0\)\},\\cdots,r\_\{L\}^\{\(0\)\}\)whererl\(0\)=maxi∈\[N\]⁡ri,l\(0\)r\_\{l\}^\{\(0\)\}=\\max\_\{i\\in\[N\]\}r\_\{i,l\}^\{\(0\)\}for alll∈\[L\]l\\in\[L\]\. The rank profiles are fixed throughout training, whereas the active local rank of a client may change across rounds\. The federator initializes the adapters𝙱\(0\)\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(0\)\}and𝙰\(0\)\\mathbf\{\\mathtt\{A\}\}^\{\(0\)\}\. A common choice is to initialize𝐁l\(0\)∈ℝd×rl\\mathbf\{B\}\_\{l\}^\{\(0\)\}\\in\\mathbb\{R\}^\{d\\times r\_\{l\}\}as the zero matrix and draw the entries of𝐀l\(0\)∈ℝrl×s\\mathbf\{A\}\_\{l\}^\{\(0\)\}\\in\\mathbb\{R\}^\{r\_\{l\}\\times s\}independently from a Gaussian distribution, whererl≜rl\(0\)=maxi∈\[N\]⁡ri,l\(0\)r\_\{l\}\\triangleq r\_\{l\}^\{\(0\)\}=\\max\_\{i\\in\[N\]\}r\_\{i,l\}^\{\(0\)\}andklk\_\{l\}is a parameter chosen to satisfykl\>rl\+1k\_\{l\}\>r\_\{l\}\+1\. Definermax≜maxl∈\[L\]⁡rlr\_\{\\max\}\\triangleq\\max\_\{l\\in\[L\]\}r\_\{l\}, andkmax≜maxl∈\[L\]⁡klk\_\{\\max\}\\triangleq\\max\_\{l\\in\[L\]\}k\_\{l\}\. The federator draws two independent random matrices𝛀\(1\)∈ℝs×rmax\\mathbf\{\\Omega\}^\{\(1\)\}\\in\\mathbb\{R\}^\{s\\times r\_\{\\max\}\}and𝚿\(1\)∈ℝkmax×d\\mathbf\{\\Psi\}^\{\(1\)\}\\in\\mathbb\{R\}^\{k\_\{\\max\}\\times d\}from the standard normal distribution, and communicates them to the clients via a shared random seed\. For layerll, the clients use the corresponding submatrices𝛀l\(1\)=𝛀:,1:rl\(1\)∈ℝs×rl\\mathbf\{\\Omega\}\_\{l\}^\{\(1\)\}=\\mathbf\{\\Omega\}^\{\(1\)\}\_\{:,1:r\_\{l\}\}\\in\\mathbb\{R\}^\{s\\times r\_\{l\}\}and𝚿l\(1\)=𝚿1:kl,:\(1\)∈ℝkl×d\\mathbf\{\\Psi\}\_\{l\}^\{\(1\)\}=\\mathbf\{\\Psi\}^\{\(1\)\}\_\{1:k\_\{l\},:\}\\in\\mathbb\{R\}^\{k\_\{l\}\\times d\}\.

In the remaining, we focus on one layerlland one roundtt\.

##### Client per\-round computation

At roundtt, clientiireceives the global adapters𝐁l\(t\)∈ℝd×rl\\mathbf\{B\}\_\{l\}^\{\(t\)\}\\in\\mathbb\{R\}^\{d\\times r\_\{l\}\}and𝐀l\(t\)∈ℝrl×s\\mathbf\{A\}\_\{l\}^\{\(t\)\}\\in\\mathbb\{R\}^\{r\_\{l\}\\times s\}for layerll, as well as a shared seed from which it generates the random matrices𝛀\(t\+1\)∈ℝs×rmax\\mathbf\{\\Omega\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{s\\times r\_\{\\max\}\}and𝚿\(t\+1\)∈ℝkmax×d\\mathbf\{\\Psi\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{k\_\{\\max\}\\times d\}from the federator\. It then extracts the layer\-specific matrices𝛀l\(t\+1\)=𝛀:,1:rl\(t\+1\)∈ℝs×rl\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}=\\mathbf\{\\Omega\}^\{\(t\+1\)\}\_\{:,1:r\_\{l\}\}\\in\\mathbb\{R\}^\{s\\times r\_\{l\}\}and𝚿l\(t\+1\)=𝚿1:kl,:\(t\+1\)∈ℝkl×d\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}=\\mathbf\{\\Psi\}^\{\(t\+1\)\}\_\{1:k\_\{l\},:\}\\in\\mathbb\{R\}^\{k\_\{l\}\\times d\}\.

*Fine\-tuning\.*If at this round no rank update is required, the client truncates the received adapters using[Algorithm˜3](https://arxiv.org/html/2608.10144#alg3)to obtain adapters of rankri,l\(t\)r\_\{i,l\}^\{\(t\)\}\. The client proceeds to fine\-tune the rank\-ri,l\(t\)r\_\{i,l\}^\{\(t\)\}adapters forEElocal iterations and sets its local rank tori,l\(t\+1\)=ri,l\(t\)r\_\{i,l\}^\{\(t\+1\)\}=r\_\{i,l\}^\{\(t\)\}\. If local rank update is required, clienti∈\[N\]i\\in\[N\]does the following\. First, before fine\-tuning, the local rank is set to the maximum rank budget the client can compute for this single layer, i\.e\.,ri,l\(0\)r\_\{i,l\}^\{\(0\)\}, according to its local rank profile\. Then, the client truncates the received adapters to rankri,l\(0\)r\_\{i,l\}^\{\(0\)\}via[Algorithm˜3](https://arxiv.org/html/2608.10144#alg3), and fine\-tunes the adapters forEEiterations\. After local fine\-tuning, clientiiapplies[Algorithm˜4](https://arxiv.org/html/2608.10144#alg4)to its locally updated adapters𝐁i,l\(t\+1\)∈ℝd×ri,l\(0\)\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{d\\times r\_\{i,l\}^\{\(0\)\}\}and𝐀i,l\(t\+1\)∈ℝri,l\(0\)×s\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{r\_\{i,l\}^\{\(0\)\}\\times s\}, to determine the new local rankri,l′r^\{\\prime\}\_\{i,l\}\. The client then truncates the locally updated adapters to that rankri,l′r^\{\\prime\}\_\{i,l\}and sets its local rank tori,l\(t\+1\)=ri,l′r\_\{i,l\}^\{\(t\+1\)\}=r^\{\\prime\}\_\{i,l\}before computing the sketches\.

*Adapter sketching\.*After fine\-tuning the adapters, clientiicomputes𝐀i,l\(t\+1\)​𝛀l\(t\+1\)\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}and then𝐘i,l\(t\+1\)=𝐁i,l\(t\+1\)​\(𝐀i,l\(t\+1\)​𝛀l\(t\+1\)\)\\mathbf\{Y\}\_\{i,l\}^\{\(t\+1\)\}=\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\(\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\)\. Similarly, the client computes𝐙i,l\(t\+1\)=\(𝚿l\(t\+1\)​𝐁i,l\(t\+1\)\)​𝐀i,l\(t\+1\)\\mathbf\{Z\}\_\{i,l\}^\{\(t\+1\)\}=\(\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\)\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}also by first computing the multiplication in parentheses to reduce the computation cost\. The client sends𝐘i,l\(t\+1\)\\mathbf\{Y\}\_\{i,l\}^\{\(t\+1\)\}and𝐙i,l\(t\+1\)\\mathbf\{Z\}\_\{i,l\}^\{\(t\+1\)\}to the federator\.

##### Federator aggregation and communication

Due to the linearity of the sketching operation, aggregating𝐘i,l\(t\+1\)\\mathbf\{Y\}\_\{i,l\}^\{\(t\+1\)\}and𝐙i,l\(t\+1\)\\mathbf\{Z\}\_\{i,l\}^\{\(t\+1\)\}is implicitly aggregatingΔ​𝐖i,l\(t\+1\)\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t\+1\)\}but in a smaller subspace\. Hence, the federator computes\\macc@depthΔ\\macc@set@skewchar\\macc@nested@a111=l\(t\+1\)∑i∈\[N\]pi𝐘i,l\(t\+1\)\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\_\{l\}^\{\(t\+1\)\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{Y\}\_\{i,l\}^\{\(t\+1\)\}and\\macc@depthΔ\\macc@set@skewchar\\macc@nested@a111=l\(t\+1\)∑i∈\[N\]pi𝐙i,l\(t\+1\)\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\_\{l\}^\{\(t\+1\)\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{Z\}\_\{i,l\}^\{\(t\+1\)\}and uses them as input for[Algorithm˜1](https://arxiv.org/html/2608.10144#alg1)to obtain𝐁l\(t\+1\)\{\\mathbf\{B\}\}\_\{l\}^\{\(t\+1\)\}and𝐀l\(t\+1\)\{\\mathbf\{A\}\}\_\{l\}^\{\(t\+1\)\}\.

To initiate the next round, the federator transmits the global adapters𝐁l\(t\+1\)\{\\mathbf\{B\}\}\_\{l\}^\{\(t\+1\)\}and𝐀l\(t\+1\)\{\\mathbf\{A\}\}\_\{l\}^\{\(t\+1\)\}and a fresh shared seed for the new random matrices\.

Algorithm 3Truncation to rankri<rr\_\{i\}<r1:Given two global LoRA factors

𝐁∈ℝd×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\\times r\}and

𝐀∈ℝr×s\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times s\}, and the target local rank

rir\_\{i\}\.

2:Return the truncated local factors

𝐁i∈ℝd×ri\\mathbf\{B\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times r\_\{i\}\}and

𝐀i∈ℝri×s\\mathbf\{A\}\_\{i\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times s\}so that

𝐁i​𝐀i\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}is a rank\-

rir\_\{i\}approximations of

𝐁𝐀\\mathbf\{B\}\\mathbf\{A\}\.

3:functionTruncate\(

𝐁,𝐀,ri\\mathbf\{B\},\\mathbf\{A\},r\_\{i\}\)

4:If

𝐁=0\\mathbf\{B\}=0then return

\(𝐁i,𝐀i\)=\(𝐁:,1:ri,𝐀1:ri,:\)\(\\mathbf\{B\}\_\{i\},\\mathbf\{A\}\_\{i\}\)=\(\\mathbf\{B\}\_\{:,1:r\_\{i\}\},\\mathbf\{A\}\_\{1:r\_\{i\},:\}\)
5:Form full SVD of

𝐀=𝐔​𝚺​𝐕⊤\\mathbf\{A\}=\\mathbf\{U\}\\mathbf\{\\Sigma\}\\mathbf\{V\}^\{\\top\}and truncate the SVD matrices to rank

rir\_\{i\}, i\.e\., the resulting

𝐔:,1:ri∈ℝr×ri\\mathbf\{U\}\_\{:,1:r\_\{i\}\}\\in\\mathbb\{R\}^\{r\\times r\_\{i\}\},

𝚺1:ri,1:ri∈ℝri×ri\\mathbf\{\\Sigma\}\_\{1:r\_\{i\},1:r\_\{i\}\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times r\_\{i\}\}, and

𝐕1:ri,:⊤∈ℝri×s\\mathbf\{V\}^\{\\top\}\_\{1:r\_\{i\},:\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times s\}\.

6:

𝐁i←𝐁𝐔:,1:ri​\(𝚺1:ri,1:ri\)1/2\\mathbf\{B\}\_\{i\}\\leftarrow\\mathbf\{B\}\\mathbf\{U\}\_\{:,1:r\_\{i\}\}\(\\mathbf\{\\Sigma\}\_\{1:r\_\{i\},1:r\_\{i\}\}\)^\{1/2\}
7:

𝐀i←\(𝚺1:ri,1:ri\)1/2​𝐕1:ri,:⊤\\mathbf\{A\}\_\{i\}\\leftarrow\(\\mathbf\{\\Sigma\}\_\{1:r\_\{i\},1:r\_\{i\}\}\)^\{1/2\}\\mathbf\{V\}^\{\\top\}\_\{1:r\_\{i\},:\}
8:Return

\(𝐁i,𝐀i\)\(\\mathbf\{B\}\_\{i\},\\mathbf\{A\}\_\{i\}\)
9:endfunction

Algorithm 4Update Local Rank and Return Truncated Updates1:Given the local LoRA factors with rank

rr, i\.e\.,

𝐁i∈ℝd×r\\mathbf\{B\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times r\}and

𝐀i∈ℝr×s\\mathbf\{A\}\_\{i\}\\in\\mathbb\{R\}^\{r\\times s\}\.

2:Return local factors with updated local rank

rir\_\{i\}, i\.e\.,

𝐁i∈ℝd×ri\\mathbf\{B\}\_\{i\}\\in\\mathbb\{R\}^\{d\\times r\_\{i\}\}and

𝐀i∈ℝri×s\\mathbf\{A\}\_\{i\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times s\}\.

3:functionUpdateLocalRank\(

𝐁i,𝐀i\\mathbf\{B\}\_\{i\},\\mathbf\{A\}\_\{i\},

τ\\tau\)

4:Find the QR decompositions of

𝐁i\\mathbf\{B\}\_\{i\}and

𝐀i⊤\\mathbf\{A\}\_\{i\}^\{\\top\}, i\.e\.,

𝐁i=𝐐B​𝐑B\\mathbf\{B\}\_\{i\}=\\mathbf\{Q\}\_\{B\}\\mathbf\{R\}\_\{B\}and

𝐀i⊤=𝐐A​𝐑A\\mathbf\{A\}\_\{i\}^\{\\top\}=\\mathbf\{Q\}\_\{A\}\\mathbf\{R\}\_\{A\}\.

5:Form

𝐌=𝐑B​𝐑A⊤∈ℝr×r\\mathbf\{M\}=\\mathbf\{R\}\_\{B\}\\mathbf\{R\}\_\{A\}^\{\\top\}\\in\\mathbb\{R\}^\{r\\times r\}, and compute its exact SVD:

𝐌=𝐔M​𝚺​𝐕M⊤\\mathbf\{M\}=\\mathbf\{U\}\_\{M\}\\mathbf\{\\Sigma\}\\mathbf\{V\}\_\{M\}^\{\\top\}\.

6:Extract squared singular values from

𝚺\\mathbf\{\\Sigma\}to find the minimal rank

rir\_\{i\}such that the cumulative singular value energy, i\.e\.,

∑j=1riσj2∑j=1rσj2\\frac\{\\sum\_\{j=1\}^\{r\_\{i\}\}\\sigma\_\{j\}^\{2\}\}\{\\sum\_\{j=1\}^\{r\}\\sigma\_\{j\}^\{2\}\}, exceeds threshold

τ\\tau\(e\.g\.,

90%90\\%\)\.

7:Compress the local LoRA factors directly using the truncated singular components:

𝐁i\\displaystyle\\mathbf\{B\}\_\{i\}←𝐐B​\(𝐔M\):,1:ri​\(𝚺1:ri,1:ri\)1/2∈ℝd×ri\\displaystyle\\leftarrow\\mathbf\{Q\}\_\{B\}\(\\mathbf\{U\}\_\{M\}\)\_\{:,1:r\_\{i\}\}\(\\mathbf\{\\Sigma\}\_\{1:r\_\{i\},1:r\_\{i\}\}\)^\{1/2\}\\in\\mathbb\{R\}^\{d\\times r\_\{i\}\}𝐀i\\displaystyle\\mathbf\{A\}\_\{i\}←\(𝚺1:ri,1:ri\)1/2​\(𝐕M\):,1:riT​𝐐AT∈ℝri×s\\displaystyle\\leftarrow\(\\mathbf\{\\Sigma\}\_\{1:r\_\{i\},1:r\_\{i\}\}\)^\{1/2\}\(\\mathbf\{V\}\_\{M\}\)\_\{:,1:r\_\{i\}\}^\{T\}\\mathbf\{Q\}\_\{A\}^\{T\}\\in\\mathbb\{R\}^\{r\_\{i\}\\times s\}
8:Return

\(𝐁i,𝐀i\),ri\(\\mathbf\{B\}\_\{i\},\\mathbf\{A\}\_\{i\}\),r\_\{i\}
9:endfunction

### 3\.2SeFoRA\-Ho: An algorithm for the rank\-homogeneous case

To enable comparison with the state\-of\-the\-art rank\-homogeneous LoRA\-based federated PEFT algorithms and provide theoretical convergence analysis, we provide a rank\-homogeneous version ofSeFoRA\. The main additional benefit ofSeFoRA\-Hocompared with existing rank\-homogeneous algorithms is linearity in aggregation in a small subspace without the bilinear mismatch problem\. This property is essential for enabling secure aggregation\[[24](https://arxiv.org/html/2608.10144#bib.bib24)\]and mitigates the noise factors resulting from multiplying noisy LoRA adapters when employing differential privacy mechanisms, see e\.g\.,\[[40](https://arxiv.org/html/2608.10144#bib.bib40)\]\. We will analyze the privacy benefits ofSeFoRA\-HoandSeFoRAin future work\.

SeFoRA\-Hois obtained by modifying Step 7 in[Algorithm˜2](https://arxiv.org/html/2608.10144#alg2)by setting𝐁i,l\(t\)=𝐁l\(t\)\\mathbf\{B\}\_\{i,l\}^\{\(t\)\}=\{\\mathbf\{B\}\_\{l\}\}^\{\(t\)\}and𝐀i,l\(t\)=𝐀l\(t\)\\mathbf\{A\}\_\{i,l\}^\{\(t\)\}=\{\\mathbf\{A\}\_\{l\}\}^\{\(t\)\}, and removing Step 9\. For completeness,SeFoRA\-Hois summarized in[Appendix˜A](https://arxiv.org/html/2608.10144#A1)\.

We show in[˜1](https://arxiv.org/html/2608.10144#Thmthm1)that this algorithm converges under standard assumptions stated in[Section˜4](https://arxiv.org/html/2608.10144#S4)\.

## 4Theoretical Analysis and Insights

This section analyzes the proposed method theoretically\. Specifically, we analyzeSeFoRA\-Ho, the rank\-homogeneous version ofSeFoRA\. We use the tuple notations introduced in[Section˜2\.1](https://arxiv.org/html/2608.10144#S2.SS1)throughout\. We first state the smoothness and stochastic gradient assumptions used below\.

###### Assumption 1\(Per\-sample Lipschitz smoothness\)\.

Letℓ​\(f​\(𝚆0\+Δ​𝚆;𝐱\),y\)\\ell\(f\(\\mathbf\{\\mathtt\{W\}\}\_\{0\}\+\\Delta\\mathbf\{\\mathtt\{W\}\};\\bm\{x\}\),y\)denote the loss evaluated on a single data sample\(𝐱,y\)\(\\bm\{x\},y\)\. There exists a real valueμ\>0\\mu\>0such that for any two model updatesΔ​𝚆\\Delta\\mathbf\{\\mathtt\{W\}\}andΔ​𝚆′\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}:

‖∇Δ​𝚆ℓ​\(f​\(𝚆0\+Δ​𝚆;𝒙\),y\)−∇Δ​𝚆ℓ​\(f​\(𝚆0\+Δ​𝚆′;𝒙\),y\)‖F≤μ​‖Δ​𝚆−Δ​𝚆′‖F\.\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\ell\(f\(\\mathbf\{\\mathtt\{W\}\}\_\{0\}\+\\Delta\\mathbf\{\\mathtt\{W\}\};\\bm\{x\}\),y\)\-\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\ell\(f\(\\mathbf\{\\mathtt\{W\}\}\_\{0\}\+\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\};\\bm\{x\}\),y\)\\\|\_\{F\}\\leq\\mu\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}\\\|\_\{F\}\.

###### Lemma 1\.

Under[˜1](https://arxiv.org/html/2608.10144#Thmassumption1), the local empirical lossℒifull​\(Δ​𝚆\)\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\), the global empirical lossℒfull​\(Δ​𝚆\)\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\), and the stochastic gradientℒifull​\(Δ​𝚆;ξi\)\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\xi\_\{i\}\)evaluated on mini\-batchξi\\xi\_\{i\}areμ\\mu\-smooth with respect toΔ​𝚆\\Delta\\mathbf\{\\mathtt\{W\}\}, i\.e\.,

‖∇Δ​𝚆ℒifull​\(Δ​𝚆\)−∇Δ​𝚆ℒifull​\(Δ​𝚆′\)‖F≤μ​‖Δ​𝚆−Δ​𝚆′‖F,\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\)\-\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}\)\\\|\_\{F\}\\leq\\mu\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}\\\|\_\{F\},‖∇Δ​𝚆ℒfull​\(Δ​𝚆\)−∇Δ​𝚆ℒfull​\(Δ​𝚆′\)‖F≤μ​‖Δ​𝚆−Δ​𝚆′‖F,\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\)\-\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}\)\\\|\_\{F\}\\leq\\mu\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}\\\|\_\{F\},and

𝔼ξi​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆;ξi\)−∇Δ​𝚆ℒifull​\(Δ​𝚆′;ξi\)‖F\]≤μ​‖Δ​𝚆−Δ​𝚆′‖F\.\\mathbb\{E\}\_\{\\xi\_\{i\}\}\\left\[\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\xi\_\{i\}\)\-\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\};\\xi\_\{i\}\)\\\|\_\{F\}\\right\]\\leq\\mu\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\\prime\}\\\|\_\{F\}\.
###### Proof\.

Each loss in the statement is an average of sample losses\. Averaging the inequality in Assumption[1](https://arxiv.org/html/2608.10144#Thmassumption1)and using the triangle inequality concludes the proof\. ∎

###### Assumption 2\(Unbiased stochastic gradients with bounded moments\)\.

For each clientiiand allΔ​𝚆∈ℝd×s\\Delta\\mathbf\{\\mathtt\{W\}\}\\in\\mathbb\{R\}^\{d\\times s\}, the stochastic gradient∇Δ​𝚆ℒifull​\(Δ​𝚆;ξi\)\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\xi\_\{i\}\)is an unbiased estimator of the local gradient∇Δ​𝚆ℒifull​\(Δ​𝚆\)\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\), i\.e\.,

𝔼ξi​\[∇Δ​𝚆ℒifull​\(Δ​𝚆;ξi\)\]=∇Δ​𝚆ℒifull​\(Δ​𝚆\)\.\\mathbb\{E\}\_\{\\xi\_\{i\}\}\\left\[\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\xi\_\{i\}\)\\right\]=\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\)\.Moreover, there are constantsχ,κ\>0\\chi,\\kappa\>0such that

𝔼ξi​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆;ξi\)‖F2\]≤χ2,and𝔼ξi​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆;ξi\)‖F4\]≤κ4\.\\mathbb\{E\}\_\{\\xi\_\{i\}\}\\left\[\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\xi\_\{i\}\)\\\|\_\{F\}^\{2\}\\right\]\\leq\\chi^\{2\},\\quad\\text\{and\}\\quad\\mathbb\{E\}\_\{\\xi\_\{i\}\}\\left\[\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\};\\xi\_\{i\}\)\\\|\_\{F\}^\{4\}\\right\]\\leq\\kappa^\{4\}\.

To facilitate the convergence analysis, we adopt the following bounded\-factor assumption, which is also used in prior theoretical analyses of federated LoRA\[[43](https://arxiv.org/html/2608.10144#bib.bib43),[54](https://arxiv.org/html/2608.10144#bib.bib54)\]\.

###### Assumption 3\(Bounded LoRA factors\)\.

There exist constantsM𝐀,M𝐁\>0M\_\{\\mathbf\{A\}\},M\_\{\\mathbf\{B\}\}\>0such that, for every layerll, roundtt, clientii, and local stepee, the spectral norms of both the local LoRA factors and the global LoRA factors are bounded, i\.e\.,‖𝐀i,l\(t,e\)‖2≤M𝐀\\\|\\mathbf\{A\}\_\{i,l\}^\{\(t,e\)\}\\\|\_\{2\}\\leq M\_\{\\mathbf\{A\}\},‖𝐁i,l\(t,e\)‖2≤M𝐁\\\|\\mathbf\{B\}\_\{i,l\}^\{\(t,e\)\}\\\|\_\{2\}\\leq M\_\{\\mathbf\{B\}\}and‖𝐀l\(t\)‖2≤M𝐀\\\|\\mathbf\{A\}\_\{l\}^\{\(t\)\}\\\|\_\{2\}\\leq M\_\{\\mathbf\{A\}\}\.

Notice that, we have‖𝐁l\(t\)‖2≤1\\\|\\mathbf\{B\}\_\{l\}^\{\(t\)\}\\\|\_\{2\}\\leq 1as it is theQQ\-factor from the QR step at the federator\.[˜3](https://arxiv.org/html/2608.10144#Thmassumption3)implies that

‖𝙰i\(t,e\)‖F≤L​r​M𝐀and‖𝙱i\(t,e\)‖F≤L​r​M𝐁,\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\\\|\_\{F\}\\leq\\sqrt\{Lr\}M\_\{\\mathbf\{A\}\}\\quad\\text\{and\}\\quad\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\\\|\_\{F\}\\leq\\sqrt\{Lr\}M\_\{\\mathbf\{B\}\},and

‖𝙰\(t\)‖F≤L​r​M𝐀and‖𝙱\(t\)‖F≤L​r\.\\\|\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\\|\_\{F\}\\leq\\sqrt\{Lr\}M\_\{\\mathbf\{A\}\}\\quad\\text\{and\}\\quad\\\|\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\\|\_\{F\}\\leq\\sqrt\{Lr\}\.
To quantify the sketching error incurred by reconstructing the aggregated update from its sketches, we recall the following guarantee\.

###### Lemma 2\(Low\-Rank Approximation \(Sketching\) Error\[[48](https://arxiv.org/html/2608.10144#bib.bib48)\]\)\.

Letr≥2r\\geq 2,ϱ∈\{0,⋯,r−2\}\\varrho\\in\\\{0,\\cdots,r\-2\\\}, and the sketch parameter satisfyk\>r\+1k\>r\+1\. Draw random matrices𝛀∈ℝs×r\\mathbf\{\\Omega\}\\in\\mathbb\{R\}^\{s\\times r\}and𝚿∈ℝk×d\\mathbf\{\\Psi\}\\in\\mathbb\{R\}^\{k\\times d\}independently from the standard normal distribution\. The rank\-rrapproximation𝐖^\\widehat\{\\mathbf\{W\}\}obtained from[Algorithm˜1](https://arxiv.org/html/2608.10144#alg1)satisfies

𝔼​‖𝐖^−𝐖‖F2≤\(1\+rk−r−1\)⋅minϱ<r−1⁡\(1\+ϱr−ϱ−1\)⋅τϱ\+12​\(𝐖\),\\displaystyle\\mathbb\{E\}\\\|\\widehat\{\\mathbf\{W\}\}\-\\mathbf\{W\}\\\|\_\{F\}^\{2\}\\leq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\\min\_\{\\varrho<r\-1\}\(1\+\\frac\{\\varrho\}\{r\-\\varrho\-1\}\)\\cdot\\tau\_\{\\varrho\+1\}^\{2\}\(\\mathbf\{W\}\),where thejj\-th tail energyτj2\\tau\_\{j\}^\{2\}is defined as:

τj2​\(𝐖\)≜minrank​\(𝐖′\)<j⁡‖𝐖−𝐖′‖F2=∑i≥jσi2​\(𝐖\)\.\\displaystyle\\tau\_\{j\}^\{2\}\(\\mathbf\{W\}\)\\triangleq\\min\_\{\\text\{rank\}\(\\mathbf\{W\}^\{\\prime\}\)<j\}\\\|\\mathbf\{W\}\-\\mathbf\{W\}^\{\\prime\}\\\|\_\{F\}^\{2\}=\\sum\_\{i\\geq j\}\\sigma\_\{i\}^\{2\}\(\\mathbf\{W\}\)\.

For a tuple𝚆=\(𝐖1,⋯,𝐖L\)\\mathbf\{\\mathtt\{W\}\}=\(\\mathbf\{W\}\_\{1\},\\cdots,\\mathbf\{W\}\_\{L\}\), use the corresponding tuple tail energyτj2​\(𝚆\)≜∑l=1Lτj2​\(𝐖l\)\\tau\_\{j\}^\{2\}\(\\mathbf\{\\mathtt\{W\}\}\)\\triangleq\\sum\_\{l=1\}^\{L\}\\tau\_\{j\}^\{2\}\(\\mathbf\{W\}\_\{l\}\)\. This definition makes the sketching error and the tail\-energy condition below well defined for all adapted layers\. This ensures that the same inequality in[˜2](https://arxiv.org/html/2608.10144#Thmlem2)holds for the tuple after applying the sketch independently to each layer, using the tuple tail energy defined above\.

Letℱt\\mathcal\{F\}\_\{t\}be the algorithmic history at the beginning of roundtt, and write𝔼t\[⋅\]=𝔼\[⋅∣ℱt\]\\mathbb\{E\}\_\{t\}\[\\cdot\]=\\mathbb\{E\}\[\\cdot\\mid\\mathcal\{F\}\_\{t\}\]\. The next theorem shows thatSeFoRA\-Hoconverges under Assumptions 1–3\.

###### Theorem 1\.

Suppose[˜1](https://arxiv.org/html/2608.10144#Thmassumption1),[˜2](https://arxiv.org/html/2608.10144#Thmassumption2), and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3)hold\. Let the global minimum of the unregularized loss beℒfull,⋆≜minΔ​𝚆⁡ℒfull​\(Δ​𝚆\)\\mathcal\{L\}^\{\\text\{full\},\\star\}\\triangleq\\min\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\)\. Take constant learning rateη\(t\)=η\\eta^\{\(t\)\}=\\eta\. Letℰsketch\(t\+1\)\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}denote the reconstruction error,ℰsketch\(t\+1\)=𝙱\(t\+1\)​𝙰\(t\+1\)−∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}=\\mathbf\{\\mathtt\{B\}\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\+1\)\}\-\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\. LetC1=χ2​M𝐀2\+2​μ2​\(M𝐁4\+M𝐀4\)C\_\{1\}=\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\+2\\mu^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\),C2=χ2​M𝐁2\+2​μ2​M𝐀2​\(M𝐁4\+M𝐀4\)C\_\{2\}=\\chi^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\+2\\mu^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\),C3=M𝐀​M𝐁​κ2C\_\{3\}=M\_\{\\mathbf\{A\}\}M\_\{\\mathbf\{B\}\}\\kappa^\{2\},C4=μ​χ2​\(M𝐁2\+M𝐀4\)C\_\{4\}=\\mu\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{2\}\+M\_\{\\mathbf\{A\}\}^\{4\}\), the sketching floorS1\(t\+1\)=χ⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]S\_\{1\}^\{\(t\+1\)\}=\\chi\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\],\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​S=1T​∑t=0T−1𝔼​\[S1\(t\+1\)\]\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{S\}=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[S\_\{1\}^\{\(t\+1\)\}\\right\], andΔ0=𝔼​\[ℒfull​\(Δ​𝚆\(0\)\)\]−ℒfull,⋆\\Delta\_\{0\}=\\mathbb\{E\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(0\)\}\)\\right\]\-\\mathcal\{L\}^\{\\text\{full\},\\star\}\.SeFoRA\-Hosatisfies:

1T​∑t=0T−1\(𝔼​\[‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\+𝔼​\[‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\)\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\+\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\\right\)≤2​Δ0η​E​T\+2η​E​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​S\+η​E⋅2​\(χ​C3\+2​C4\)\+η2​E2⋅23​χ2​\(C1\+C2\)\+4​η3​E3⋅μ​C32\.\\displaystyle\\leq\\frac\{2\\Delta\_\{0\}\}\{\\eta ET\}\+\\frac\{2\}\{\\eta E\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{S\}\+\\eta E\\cdot 2\\left\(\\chi C\_\{3\}\+2C\_\{4\}\\right\)\+\\eta^\{2\}E^\{2\}\\cdot\\frac\{2\}\{3\}\\chi^\{2\}\\left\(C\_\{1\}\+C\_\{2\}\\right\)\+4\\eta^\{3\}E^\{3\}\\cdot\\mu C\_\{3\}^\{2\}\.

The proof of[˜1](https://arxiv.org/html/2608.10144#Thmthm1)can be found in[Section˜B\.1](https://arxiv.org/html/2608.10144#A2.SS1)\.

When we further assume that the spectral tail is bounded as in[˜4](https://arxiv.org/html/2608.10144#Thmassumption4), we present[Corollary˜1](https://arxiv.org/html/2608.10144#Thmcorollary1)\.

###### Assumption 4\.

There exist a fixed numberϱ∈\{0,⋯,r−2\}\\varrho\\in\\\{0,\\cdots,r\-2\\\}and a constant\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111≥0\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\\geq 0such that, for everyT≥1T\\geq 1,

1T​∑t=0T−1𝔼​\[τϱ\+12​\(∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\)\]≤\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​1112\.\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\tau\_\{\\varrho\+1\}^\{2\}\(\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\)\\right\]\\leq\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{2\}\.

###### Corollary 1\.

Suppose[˜1](https://arxiv.org/html/2608.10144#Thmassumption1),[˜2](https://arxiv.org/html/2608.10144#Thmassumption2),[˜3](https://arxiv.org/html/2608.10144#Thmassumption3), and[˜4](https://arxiv.org/html/2608.10144#Thmassumption4)hold, and define

ϵ2≜\(1\+rk−r−1\)⋅\(1\+ϱr−ϱ−1\)⋅\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​1112\.\\epsilon^\{2\}\\triangleq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\(1\+\\frac\{\\varrho\}\{r\-\\varrho\-1\}\)\\cdot\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{2\}\.Choose

γT≜Δ0T\+χ​ϵ\+2​μ​ϵ2χ​C3\+2​C4,andη=γTE,\\gamma\_\{T\}\\triangleq\\sqrt\{\\frac\{\\frac\{\\Delta\_\{0\}\}\{T\}\+\\chi\\epsilon\+2\\mu\\epsilon^\{2\}\}\{\\chi C\_\{3\}\+2C\_\{4\}\}\},\\qquad\\text\{and\}\\qquad\\eta=\\frac\{\\gamma\_\{T\}\}\{E\},SeFoRA\-Hosatisfies:

1T​∑t=0T−1\(𝔼​\[‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\+𝔼​\[‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\)\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\+\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\\right\)≤4​γT​\(χ​C3\+2​C4\)\+γT2​2​χ2​\(C1\+C2\)3\+4​γT3​μ​C32\.\\displaystyle\\leq 4\\gamma\_\{T\}\(\\chi C\_\{3\}\+2C\_\{4\}\)\+\\gamma\_\{T\}^\{2\}\\frac\{2\\chi^\{2\}\\\!\\left\(C\_\{1\}\+C\_\{2\}\\right\)\\\!\}\{3\}\+4\\gamma\_\{T\}^\{3\}\\mu C\_\{3\}^\{2\}\.

The proof of[Corollary˜1](https://arxiv.org/html/2608.10144#Thmcorollary1)can be found in[Section˜B\.2](https://arxiv.org/html/2608.10144#A2.SS2)\.

## 5Experiments

We conduct experiments to evaluate the performance ofSeFoRA\-Hoin the rank\-homogeneous setting andSeFoRAin the rank\-heterogeneous setting\. We compare the proposed methods against their respective federated LoRA baselines across four natural language understanding datasets from the GLUE benchmark\.

##### Models and datasets

We employ RoBERTa\-Large\[[55](https://arxiv.org/html/2608.10144#bib.bib55)\]as the pretrained backbone, and evaluate it on four datasets from the General Language Understanding Evaluation \(GLUE\)\[[56](https://arxiv.org/html/2608.10144#bib.bib56)\]benchmark, i\.e\., SST\-2, QNLI, QQP, and MNLI\. SST\-2 is a sentiment\-classification task, QNLI identifies whether a sentence contains the answer to a question, QQP evaluates whether two sentences are semantically equivalent, and MNLI is a three\-class classification task, which evaluates the entailment between two sentences\.

The LoRA adapters are inserted into the query, key, and value projection matrices of every self\-attention layer\. The pretrained backbone parameters are frozen during federated fine\-tuning, while the adapters and the classification head are optimized\.

##### Baselines

For the rank\-homogeneous case, we compareSeFoRA\-Howith two baselines, namely, FedIT\[[38](https://arxiv.org/html/2608.10144#bib.bib38)\]and FFA\-LoRA\[[44](https://arxiv.org/html/2608.10144#bib.bib44)\]\. FedIT aggregates the two LoRA factors uploaded by the clients separately\. Although straightforward and communication\-efficient, FedIT suffers from the bilinear mismatch\. FFA\-LoRA eliminates the bilinear mismatch by freezing one of the LoRA factor matrices\.

For the rank\-heterogeneous case, we compareSeFoRAwith the three most relevant approaches, i\.e\., FlexLoRA\[[46](https://arxiv.org/html/2608.10144#bib.bib46)\], FLoRA\[[45](https://arxiv.org/html/2608.10144#bib.bib45)\]and FSLoRA\[[47](https://arxiv.org/html/2608.10144#bib.bib47)\]\. FlexLoRA reconstructs each client’s full update from the LoRA adapters, thereby sacrificing server\-side computation to the bilinear mismatch\. FLoRA concatenates client factors so that the aggregation can be represented exactly, but the effective adapter rank increases with the number of participating clients, and clients re\-initialize the adapters in every round\. FSLoRA communicates selected components of fixed\-rank local adapters, supporting heterogeneous ranks across clients but not rank adaptation during training\. In our FSLoRA experiments, each client is assigned a fixed rank at initialization, which remains unchanged throughout all communication rounds\.

##### Experimental setup

We use 40 clients, all of whom participate in every round\. To simulate data heterogeneity, the data is allocated among clients by sampling from a Dirichlet distribution that determines the portion of samples each client receives for each label\. The parameterα\\alphacontrols the level of heterogeneity\. A smallerα\\alphacorresponds to a higher level of heterogeneity\. We setα=0\.5\\alpha=0\.5to model data heterogeneity\.

Each client performs local stepsE∈\{1,5\}E\\in\\\{1,5\\\}per communication round\. In each local step, clients randomly sample a mini\-batch of100100samples from their local training dataset\. We use Adam with a learning rate ofη=9​e−4\\eta=9e\-4across all experiments\. The adapter contribution is scaled by0\.10\.1\.

Although SeFoRA allows clients to have different maximum rank budgets due to heterogeneous computational and memory resources, in these experiments, all clients are assigned the same maximum rank\. The random matrices used in the simulation for sketching are independently generated in each round\. In the rank updating setting, the rank is updated every100100rounds, andτ=90%\\tau=90\\%energy is preserved\. Although the algorithm allows layer\-specific ranks, in all experiments we use a uniform rank configuration across adapted layers\. Specifically, in theSeFoRA\-Hoexperiments, we setri,l=ri=rr\_\{i,l\}=r\_\{i\}=rfor everyllandii\. In theSeFoRAexperiments, we setri=maxl⁡ri,lr\_\{i\}=\\max\_\{l\}r\_\{i,l\}andr=maxi⁡rir=\\max\_\{i\}r\_\{i\}\. Consequently, each client has a single scalar local rank shared by all adapted layers\. Unless stated otherwise, we set the maximum adapter rank tor=16r=16and the sketch dimensionk=18k=18\. For FlexLoRA and FLoRA, we employ the same rank adaptation mechanism as inSeFoRA\. For FSLoRA, heterogeneous client ranks are sampled from the set\{2,4,⋯,14\}\\\{2,4,\\cdots,14\\\}and remain fixed throughout training\.

We set the total number of rounds to beT=500T=500\. We repeat each experiment using three random seeds and report the mean accuracy and standard deviation\.

##### Rank\-homogeneous results

In[Table˜2](https://arxiv.org/html/2608.10144#S5.T2), we compare the proposed methodSeFoRA\-Howith rank\-homogeneous baselines, usingr=16r=16andk=18k=18\. With one local step,SeFoRA\-Hoobtains results close to FedIT, while substantially outperforming FFA\-LoRA\. Increasing the number of local steps to55improves all three methods\. FFA\-LoRA performs consistently worse thanSeFoRA\-Hoand FedIT, suggesting that freezing one LoRA factor restricts the model’s expressivity, particularly on more challenging tasks\.

##### Rank\-heterogeneous results

[Fig\.˜1](https://arxiv.org/html/2608.10144#S5.F1)presents the convergence behavior ofSeFoRA, FlexLoRA, FLoRA, and FSLoRA on SST\-2 and QNLI\. FLoRA converges considerably more slowly, particularly on QNLI and when the local step is11\. This behavior can be explained by the fact that FLoRA integrates the updated global model with the backbone model and reinitializes the local adapters in each round\. Using55local steps substantially accelerates convergence and improves final performance for all methods\.SeFoRAachieves performance comparable to or slightly better than FlexLoRA, without the need to reconstruct all clients’ updates, which incurs a𝒪​\(d2​s\)\\mathcal\{O\}\(d^\{2\}s\)computational cost at the server\.

[Table˜3](https://arxiv.org/html/2608.10144#S5.T3)studies the influence of the adapter rankrrand the sketch dimensionkkunderE=5E=5\. We considerr∈\{2,4,8,16\}r\\in\\\{2,4,8,16\\\}and comparek=r\+2k=r\+2withk=2​rk=2r\. For a fixed rankrr, increasing the sketch dimension fromk=r\+2k=r\+2to2​r2rproduces little or no improvement\. These results indicate thatk=r\+2k=r\+2is sufficient to preserve the update information while requiring less communication thank=2​rk=2r\. The table also shows that small ranks are sufficient for SST\-2 and QNLI, whereas the more challenging tasks, such as QQP and MNLI, benefit more from larger adapter ranks\.

[Fig\.˜2](https://arxiv.org/html/2608.10144#S5.F2)shows the adaptive rank behavior\. Rank adaptation is performed every100100communication rounds\. For the larger initial ranks, the average local rank decreases substantially across the communication rounds\. This shows that the adapter updates demand a reduced rank as federated fine\-tuning progresses\. Comparing rank evolution across datasets, the results suggest that rank adaptation also reflects differences in task complexity\. As task complexity increases, the adapter requires a higher rank to express the information contained\.

Local StepsMethodSST\-2QNLIQQPMNLIE=1E=1FedIT95\.1±0\.495\.1\\pm 0\.489\.5±0\.889\.5\\pm 0\.883\.8±2\.283\.8\\pm 2\.285\.1±2\.285\.1\\pm 2\.2FFA\-LoRA92\.0±0\.492\.0\\pm 0\.475\.3±2\.875\.3\\pm 2\.876\.6±1\.976\.6\\pm 1\.957\.5±1\.057\.5\\pm 1\.0SeFoRA\-Ho95\.0±0\.295\.0\\pm 0\.288\.8±0\.988\.8\\pm 0\.983\.1±2\.283\.1\\pm 2\.284\.6±0\.584\.6\\pm 0\.5E=5E=5FedIT95\.4±0\.295\.4\\pm 0\.293\.5±0\.193\.5\\pm 0\.188\.1±0\.488\.1\\pm 0\.489\.2±0\.489\.2\\pm 0\.4FFA\-LoRA94\.4±0\.394\.4\\pm 0\.388\.9±0\.788\.9\\pm 0\.783\.5±1\.083\.5\\pm 1\.084\.9±0\.484\.9\\pm 0\.4SeFoRA\-Ho95\.7±0\.595\.7\\pm 0\.593\.7±0\.293\.7\\pm 0\.287\.9±0\.487\.9\\pm 0\.489\.6±0\.389\.6\\pm 0\.3Table 2:We report the accuracy \(%\\%\) in the rank\-homogeneous setting\.SeFoRA\-Housesr=16r=16andk=18k=18\.![Refer to caption](https://arxiv.org/html/2608.10144v1/result_fig/qnli_sst2_local_steps_mean_std.png)Figure 1:We report the accuracy in the rank\-heterogeneous setting\. We user=16r=16andk=18k=18forSeFoRA\. FSLoRA clients use fixed ranks sampled from\{2,4,⋯,14\}\\\{2,4,\\cdots,14\\\}\.rank \(rr\)kkSST\-2QNLIQQPMNLIr=16r=16323295\.5±0\.195\.5\\pm 0\.193\.3±0\.393\.3\\pm 0\.387\.6±0\.487\.6\\pm 0\.489\.5±0\.289\.5\\pm 0\.2181895\.5±0\.195\.5\\pm 0\.193\.2±0\.293\.2\\pm 0\.287\.7±0\.487\.7\\pm 0\.489\.4±0\.389\.4\\pm 0\.3r=8r=8161695\.4±0\.295\.4\\pm 0\.293\.1±0\.293\.1\\pm 0\.287\.4±0\.487\.4\\pm 0\.489\.2±0\.389\.2\\pm 0\.3101095\.5±0\.295\.5\\pm 0\.293\.2±0\.193\.2\\pm 0\.187\.4±0\.487\.4\\pm 0\.489\.1±0\.389\.1\\pm 0\.3r=4r=48895\.4±0\.495\.4\\pm 0\.493\.0±0\.193\.0\\pm 0\.187\.0±0\.587\.0\\pm 0\.588\.9±0\.388\.9\\pm 0\.36695\.5±0\.195\.5\\pm 0\.193\.0±0\.493\.0\\pm 0\.487\.0±0\.587\.0\\pm 0\.588\.9±0\.188\.9\\pm 0\.1r=2r=24495\.3±0\.695\.3\\pm 0\.693\.0±0\.393\.0\\pm 0\.386\.5±0\.686\.5\\pm 0\.688\.3±0\.188\.3\\pm 0\.1Table 3:We report the accuracy \(%\\%\) to illustrate the sensitivity ofSeFoRAto the rankrrand sketch dimensionkk, conducting55local steps\. We varyr∈\{2,4,8,16\}r\\in\\\{2,4,8,16\\\}andk∈\{r\+2,2​r\}k\\in\\\{r\+2,2r\\\}\.![Refer to caption](https://arxiv.org/html/2608.10144v1/result_fig/adaptive_rank.png)Figure 2:We report the evolution of the clients’ average adaptive rank forSeFoRAusingE=5E=5local steps andk=r\+2k=r\+2\. Local ranks are updated every100100communication rounds\.

## 6Related Work

##### Parameter\-efficient fine\-tuning \(PEFT\)\.

PEFT methods adapt a pretrained model𝐖0\\mathbf\{W\}\_\{0\}by updating only a small, structured subset of parameters\. They fall into three families:*additive*\(inserting trainable modules\[[11](https://arxiv.org/html/2608.10144#bib.bib11),[12](https://arxiv.org/html/2608.10144#bib.bib12),[13](https://arxiv.org/html/2608.10144#bib.bib13),[14](https://arxiv.org/html/2608.10144#bib.bib14)\]\),*selective*\(updating a subset of existing weights\[[15](https://arxiv.org/html/2608.10144#bib.bib15),[16](https://arxiv.org/html/2608.10144#bib.bib16)\]\), and*reparametrized*\(imposing low\-dimensional structure on the update\[[17](https://arxiv.org/html/2608.10144#bib.bib17),[18](https://arxiv.org/html/2608.10144#bib.bib18),[19](https://arxiv.org/html/2608.10144#bib.bib19),[20](https://arxiv.org/html/2608.10144#bib.bib20),[21](https://arxiv.org/html/2608.10144#bib.bib21)\]\); see\[[22](https://arxiv.org/html/2608.10144#bib.bib22)\]for a survey\.

Among reparametrized methods, LoRA\[[17](https://arxiv.org/html/2608.10144#bib.bib17)\]is perhaps the most popular method due to its simplicity and negligible inference overhead\. The weight updates are parametrized asΔ​𝐖=𝐁𝐀\\Delta\\mathbf\{W\}=\\mathbf\{B\}\\mathbf\{A\}with*factor matrices*𝐁∈ℝd×r\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\\times r\},𝐀∈ℝr×s\\mathbf\{A\}\\in\\mathbb\{R\}^\{r\\times s\},r≪min⁡\(d,s\)r\\ll\\min\(d,s\), thus reducing the number of trainable parameters fromd​sdsto\(d\+s\)​r\(d\+s\)r\. Beyond the original LoRA formulation\[[17](https://arxiv.org/html/2608.10144#bib.bib17)\], numerous variants have been proposed to improve adaptation efficiency, convergence, and parameter utilization\. LoRA\+\[[33](https://arxiv.org/html/2608.10144#bib.bib33)\]accelerates optimization by assigning different learning rates to the two low\-rank matrices\. DoRA\[[34](https://arxiv.org/html/2608.10144#bib.bib34)\]improves fine\-tuning performance by decomposing pretrained weights into magnitude and directional components, allowing LoRA to optimize only the directional updates\. Other methods focus on improving parameter efficiency through adaptive resource allocation\. LoRA\-drop\[[35](https://arxiv.org/html/2608.10144#bib.bib35)\]prunes redundant LoRA parameters based on their contribution to the model output, whereas AdaLoRA\[[20](https://arxiv.org/html/2608.10144#bib.bib20)\]dynamically reallocates the adaptation rank across layers according to their relative importance under a fixed parameter budget\. HydraLoRA\[[19](https://arxiv.org/html/2608.10144#bib.bib19)\]modifies the factorization structure\. Furthermore, LoRA has been adopted in application\-specific settings, such as automatic prompt engineering for logical reasoning\[[36](https://arxiv.org/html/2608.10144#bib.bib36)\]\. On the theory side,\[[37](https://arxiv.org/html/2608.10144#bib.bib37)\]shows thatℓ2\\ell\_\{2\}\-regularized LoRA is equivalent to nuclear\-norm\-regularized full fine\-tuning subject to a rank constraint\.

##### PEFT and federated learning

Federated learning \(FL\) has become the de facto paradigm for distributed machine learning, enabling collaborative model training while keeping data decentralized\. By allowing clients to train models locally without sharing raw data, FL offers better privacy guarantees and has consequently found widespread adoption in privacy\-sensitive applications, including finance\[[57](https://arxiv.org/html/2608.10144#bib.bib57),[58](https://arxiv.org/html/2608.10144#bib.bib58)\]and healthcare\[[59](https://arxiv.org/html/2608.10144#bib.bib59),[60](https://arxiv.org/html/2608.10144#bib.bib60),[61](https://arxiv.org/html/2608.10144#bib.bib61),[62](https://arxiv.org/html/2608.10144#bib.bib62)\]\. Motivated by the success of large language models \(LLMs\), recent research has extended FL to support their efficient fine\-tuning and deployment\[[63](https://arxiv.org/html/2608.10144#bib.bib63),[64](https://arxiv.org/html/2608.10144#bib.bib64),[38](https://arxiv.org/html/2608.10144#bib.bib38)\]\. Several frameworks have subsequently been proposed to accelerate research in this emerging area\. FederatedScope\-LLM\[[65](https://arxiv.org/html/2608.10144#bib.bib65)\]provides a comprehensive platform for federated fine\-tuning and instruction tuning of LLMs, while OpenFedLLM\[[66](https://arxiv.org/html/2608.10144#bib.bib66)\]offers a unified framework for training LLMs over decentralized private data and systematically studies federated instruction tuning and value alignment\. Complementing these systems, FedLLM\-Bench\[[67](https://arxiv.org/html/2608.10144#bib.bib67)\]introduces realistic benchmarks and extensive empirical evaluations for federated fine\-tuning, enabling standardized assessment of optimization algorithms and communication\-efficient methods\. Together, these works have established the foundational infrastructure, benchmarks, and evaluation methodologies that drive the rapidly growing field of federated fine\-tuning of LLMs\.

Combining PEFT with FL enables for adapting large foundation models on vast amount of data while substantially reducing communication overhead, computational complexity, and privacy risks\. Existing approaches span adapter\-based methods\[[27](https://arxiv.org/html/2608.10144#bib.bib27),[28](https://arxiv.org/html/2608.10144#bib.bib28)\], prompt tuning\[[29](https://arxiv.org/html/2608.10144#bib.bib29),[30](https://arxiv.org/html/2608.10144#bib.bib30)\], selective parameter optimization\[[31](https://arxiv.org/html/2608.10144#bib.bib31)\], and low\-rank gradient subspace optimization\[[32](https://arxiv.org/html/2608.10144#bib.bib32)\]\. More recently, Low\-Rank Adaptation \(LoRA\)\[[17](https://arxiv.org/html/2608.10144#bib.bib17)\]has become a popular PEFT technique on its own and in FL\.

##### Federated LoRA\.

Early works such as FedIT\[[38](https://arxiv.org/html/2608.10144#bib.bib38)\]integrate LoRA into the standard FedAvg framework for federated instruction tuning, while subsequent methods address the limitations of naive LoRA aggregation under heterogeneous data and system conditions\. However, combining LoRA with FL raises several challenges faces challenges that do not exist in centralized LoRA or in federated averaging alone\. Namely: the error introduced by the bilinear mismatch resulting from first aggregating the𝐀i\\mathbf\{A\}\_\{i\}’s and𝐁i\\mathbf\{B\}\_\{i\}’s and then computing\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111=\(∑ipi​𝐁i\)​\(∑ipi​𝐀i\)\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}=\(\\sum\_\{i\}p\_\{i\}\\mathbf\{B\}\_\{i\}\)\(\\sum\_\{i\}p\_\{i\}\\mathbf\{A\}\_\{i\}\)as done in FedIT\[[38](https://arxiv.org/html/2608.10144#bib.bib38)\]; or the rank\-rrapproximation error resulting from first computingΔ​𝐖=∑ipi​𝐁i​𝐀i\\Delta\\mathbf\{W\}=\\sum\_\{i\}p\_\{i\}\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}which is not necessarily rank\-rrand then computing a rank\-rrapproximation, e\.g\., using SVD decomposition as in FlexLoRA\[[46](https://arxiv.org/html/2608.10144#bib.bib46)\]\. In addition, the proposed aggregation rule should account for the*rank heterogeneity*problem: when clients use different ranksrir\_\{i\}, the factor matrices have mismatched dimensions, rendering direct factor averaging infeasible\. To overcome those hurdles, several approaches have been proposed, as we detail in the sequel\. Beyond model parameterization, several recent works have focused on improving the optimization process itself\. FedBCGD\[[68](https://arxiv.org/html/2608.10144#bib.bib68)\]introduces an accelerated block coordinate gradient descent algorithm, FedSWA\[[69](https://arxiv.org/html/2608.10144#bib.bib69)\]employs stochastic weight averaging to improve generalization under highly non\-IID data, FedAdamW\[[70](https://arxiv.org/html/2608.10144#bib.bib70)\]proposes a communication\-efficient AdamW\-inspired optimizer for federated large models, FedNSAM\[[71](https://arxiv.org/html/2608.10144#bib.bib71)\]analyzes the relationship between local and global sharpness to improve optimization consistency, FedMuon\[[72](https://arxiv.org/html/2608.10144#bib.bib72)\]accelerates convergence through matrix orthogonalization, and DP\-FedPGN\[[73](https://arxiv.org/html/2608.10144#bib.bib73)\]encourages globally flatter minima in differentially private FL by penalizing gradient norms\. Client data heterogeneity is tackled in\[[74](https://arxiv.org/html/2608.10144#bib.bib74)\]\. In LA\-LoRA\[[40](https://arxiv.org/html/2608.10144#bib.bib40)\], during local iterations, the clients alternately update𝐁\\mathbf\{B\}and𝐀\\mathbf\{A\}and send both updated adapters to the federator for aggregation\. This alternation is shown to provide stability during training and to allow for differential privacy with lower noise\. FedASK\[[75](https://arxiv.org/html/2608.10144#bib.bib75)\]uses a double\-sketching approach primarily to provide a differential privacy guarantee, but does not account for rank\-heterogeneity\. Robustness to label noise in federated LoRA is studied in\[[76](https://arxiv.org/html/2608.10144#bib.bib76)\]\. Personalized federated adaptation is also considered, e\.g\.,\[[77](https://arxiv.org/html/2608.10144#bib.bib77),[78](https://arxiv.org/html/2608.10144#bib.bib78)\]\.

##### Tackling the bilinear mismatch

The bilinear mismatch introduces an error ofO​\(η2\)O\(\\eta^\{2\}\)per round for a single local step and grows asO​\(η2​E2\)O\(\\eta^\{2\}E^\{2\}\)for multiple local steps\[[45](https://arxiv.org/html/2608.10144#bib.bib45),[79](https://arxiv.org/html/2608.10144#bib.bib79)\]and is more prominent when considering differential privacy guarantees\. In\[[44](https://arxiv.org/html/2608.10144#bib.bib44)\], FFA\-LoRA freezes the initialized adapter𝐀\\mathbf\{A\}and trains only the adapter𝐁\\mathbf\{B\}\. As such, the federator avoids the bilinear mismatch problem\. However, freezing the adapter𝐀\\mathbf\{A\}reduces the expressiveness of the adaptation and affects the performance\. In FedSVD\[[80](https://arxiv.org/html/2608.10144#bib.bib80)\], the clients only train𝐁\\mathbf\{B\}locally, while the global adapter𝐀\(t\)\\mathbf\{A\}^\{\(t\)\}is computed based on an SVD of the matrix𝐁\(t\)​𝐀\(t−1\)\\mathbf\{B\}^\{\(t\)\}\\mathbf\{A\}^\{\(t\-1\)\}; thus enabling the update of the factor𝐀\\mathbf\{A\}while clients still train and aggregate only𝐁\\mathbf\{B\}\. In a similar spirit, FedSA\-LoRA\[[43](https://arxiv.org/html/2608.10144#bib.bib43)\]trains𝐀\\mathbf\{A\}globally while each client trains𝐁\\mathbf\{B\}locally\. A formal convergence analysis; however, since𝐁\\mathbf\{B\}is different for each client, the algorithm is suitable for personalized FL\. The concurrent works introducing LoRA\-A2\[[81](https://arxiv.org/html/2608.10144#bib.bib81)\]and RoLoRA\[[54](https://arxiv.org/html/2608.10144#bib.bib54)\]alternate between freezing𝐀\\mathbf\{A\}training𝐁\\mathbf\{B\}, and training𝐀\\mathbf\{A\}freezing𝐁\\mathbf\{B\}\. In LoRA\-A2, the federator selects an adaptive rank for each client to handle data heterogeneity\. The client updates only a subset of all possible ranks, using a mask to perform the update without modifying the dimensionality of the adapters; however, the paper lacks a theoretical foundation\. In RoLoRA, all clients maintain the same rank, and a convergence guarantee is given\. In\[[82](https://arxiv.org/html/2608.10144#bib.bib82)\], the authors propose FedRot\-LoRA, which takes a different approach: aligning local subspaces via orthogonal rotations at the clients’ side before aggregating\. FedEx\-LoRA\[[79](https://arxiv.org/html/2608.10144#bib.bib79)\]keeps the LoRA factors trainable across rounds and absorbs the mismatch into the frozen pretrained weights as a residual term; however, it increases both computation and communication overheads and does not provide a convergence guarantee\. This reduces the bilinear mismatch error but does not eliminate it\. LoRA\-FAIR\[[83](https://arxiv.org/html/2608.10144#bib.bib83)\]introduces a correction term in𝐁\\mathbf\{B\}to prevent the\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}from deviating too much from the actual update∑i𝐁i​∑i𝐀i\\sum\_\{i\}\\mathbf\{B\}\_\{i\}\\sum\_\{i\}\\mathbf\{A\}\_\{i\}\. Despite their advantages, those methods are not applicable in rank\-heterogeneous settings\.

##### Rank\-heterogeneous federated LoRA

In\[[84](https://arxiv.org/html/2608.10144#bib.bib84)\], the authors investigate methods for finding a good initialization of the adapters via full fine\-tuning and then running federated LoRA, without altering the federated LoRA process\. Adapting the federation of LoRA to account for rank heterogeneity is proving to be more successful\. The authors of\[[45](https://arxiv.org/html/2608.10144#bib.bib45)\]propose FLoRA, in which the federator stacks the received𝐀i\\mathbf\{A\}\_\{i\}’s and𝐁i\\mathbf\{B\}\_\{i\}’s from the clients in a way that the multiplication of the stacked matrices emulates the computation and aggregation of the localΔ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}’s\. As such, the globally updated𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}have larger dimensions, and the resultingΔ​𝐖\\Delta\\mathbf\{W\}has a rank that could be larger thanrr; thus increasing the communication and computation cost at the clients\. Furthermore, in each round, the updated global model is integrated into the base model and the adapters are reinitialized, thereby potentially losing information about the adapters obtained from the previous rounds\. Bai et al\.\[[46](https://arxiv.org/html/2608.10144#bib.bib46)\]took another approach in their FlexLoRA\. The federator computes and aggregates the clients’Δ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}’s, and, for each client, the federator uses SVD to decompose and send a rank\-rir\_\{i\}approximation of the globalΔ​𝐖\\Delta\\mathbf\{W\}\. This method incurs anO​\(d2​s\)O\(d^\{2\}s\)computational cost at the server, and the paper analyzes the number of samples the clients need to have to obtain a good generalization bound, but it does not give a proof of convergence for the algorithm\. Similarly, in FedMomentum\[[85](https://arxiv.org/html/2608.10144#bib.bib85)\], the federator computes and aggregates the localΔ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}’s to obtainΔ​𝐖\\Delta\\mathbf\{W\}and then performs a few mathematical transformations onΔ​𝐖\\Delta\\mathbf\{W\}to efficiently compute an approximation of its SVD decomposition\. A key difference in this work is computing the global adapters as𝐁=𝐔~​Σ1/2\\mathbf\{B\}=\\widetilde\{\\mathbf\{\{U\}\}\}\\Sigma^\{1/2\}and𝐀=Σ1/2​𝐕\\mathbf\{A\}=\\Sigma^\{1/2\}\\mathbf\{\{V\}\}, where𝐔~\\widetilde\{\\mathbf\{\{U\}\}\},Σ\\Sigmaand𝐕\\mathbf\{\{V\}\}are the matrices obtained from the approximation of the SVD ofΔ​𝐖\\Delta\\mathbf\{W\}\. The computation complexity at the federator isO​\(d​N2​r2\)O\(dN^\{2\}r^\{2\}\)\. Also, no convergence guarantee is given\. Similarly, in FedSRD\[[51](https://arxiv.org/html/2608.10144#bib.bib51)\]the clients apply local sparsification and pruning techniques to reduce the ranks of their local adapters, and the federator reconstructs the clients’ full update𝐁i​𝐀i\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}and aggregates in the full\-rank space, which increases the computation at the server\. The server then computes the SVD ofΔ​𝐖\\Delta\\mathbf\{W\}to compute a low\-rank update and uses sparsification techniques to further reduce the communication cost; this work also lacks a convergence analysis\. In HetLoRA\[[41](https://arxiv.org/html/2608.10144#bib.bib41)\], the clients prune the adapters to smaller ranks by adding a regularization factor to the optimization problem\. Then, the federator zero\-pads the clients’𝐀i\\mathbf\{A\}\_\{i\}’s and𝐁i\\mathbf\{B\}\_\{i\}’s to ensure they have the same dimension\. However, when aggregating the zero\-padded adapters, a careful weighted average is performed\. For each adapter, a weightpip\_\{i\}proportional to the norm of the singular value vector ofΔ​𝐖i\\Delta\\mathbf\{W\}\_\{i\}is computed\. The computational complexity of this method isO​\(N​d​r​s\)O\(Ndrs\)for computing the global update\. Similarly, no convergence guarantee is given\. FSLoRA\[[47](https://arxiv.org/html/2608.10144#bib.bib47)\]modifies the update to𝐁𝐒𝐀\\mathbf\{B\}\\mathbf\{S\}\\mathbf\{A\}to embed the sketching matrices§i\\S\_\{i\}at the clients and account for rank heterogeneity\. The matrices𝐒i\\mathbf\{S\}\_\{i\}arer×rr\\times rdiagonal matrices with onlyrir\_\{i\}non\-zero entries chosen at random\. The algorithm can be theoretically proven to converge with a rate ofO​\(L~/N​T\)O\(\\widetilde\{L\}/\\sqrt\{NT\}\)when run forTTiterations\. However, the choice of whichrir\_\{i\}entries are non\-zero is random and not based on the properties of client’s data\. The authors of\[[49](https://arxiv.org/html/2608.10144#bib.bib49)\]replace the zero\-padding by replicating columns of𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}to make the matrices have the same ranks\. In RB\-LoRA\[[50](https://arxiv.org/html/2608.10144#bib.bib50)\], a unifying framework for completing the local adapters \(via replication\) and stacking them for computation at the federator is proposed\. A client’s local adapters are weighted based on the amount of data the client holds, proportional to the total number of data \(for𝐀\\mathbf\{A\}\) and to the total number of data held by clients with ranks less than or equal to the client’s \(for𝐁\\mathbf\{B\}\)\. The method’s performance is evaluated via numerical experiments\. In\[[86](https://arxiv.org/html/2608.10144#bib.bib86)\], the proposed AFLoRA adds a diagonal matrix𝚲\\boldsymbol\{\\Lambda\}of dimensionr×rr\\times rto the clients’ local adapters to haveΔ​𝐖i=𝐁i​𝚲i​𝐀i\\Delta\\mathbf\{W\}\_\{i\}=\\mathbf\{B\}\_\{i\}\\boldsymbol\{\\Lambda\}\_\{i\}\\mathbf\{A\}\_\{i\}\. The diagonal matrix is meant to compute the contribution of each dimension of𝐁i\\mathbf\{B\}\_\{i\}to the localΔ​𝐖\\Delta\\mathbf\{W\}\. Then, the clients accordingly adjust their local ranks\. The adapter𝐀\\mathbf\{A\}is only trained at the federator using a public dataset, and the𝐁i\\mathbf\{B\}\_\{i\}and𝚲i\\boldsymbol\{\\Lambda\}\_\{i\}are trained at the clients\. To aggregate the local adapters, zero padding is used, and aggregation weights proportional to the clients’ data volume and local LoRA ranks are computed\. FedARA\[[52](https://arxiv.org/html/2608.10144#bib.bib52)\]adaptively allocates local ranks to each layer at clients, and layers with small ranks are not updated\. Through a majority vote at the federator, global layers will also be pruned\. However, aggregating client updates still requires zero\-padding and matrices of the same dimension\. Recent work has further examined the biases caused by heterogeneous ranks\[[87](https://arxiv.org/html/2608.10144#bib.bib87),[88](https://arxiv.org/html/2608.10144#bib.bib88)\]\. In\[[87](https://arxiv.org/html/2608.10144#bib.bib87)\], rank collapse is identified, i\.e\., the energy of the aggregated update progressively concentrates on the smallest rank shared by the participating clients\. They propose raFLoRA, which splits each local update into rank partitions and aggregates them separately\. This prevents higher\-rank components from being diluted by clients who do not update them\. Nevertheless, the server reconstructs the partitioned updates in the full model space and applies an SVD to recover global factors, incurring computational overhead at the server\. FedHL\[[89](https://arxiv.org/html/2608.10144#bib.bib89)\]focuses on the truncation\-induced bias arising in federated LoRA with heterogeneous ranks\. The method maintains a full\-rank global model as a calibrated aggregation basis, and the truncation bias can be eliminated by using a compensation mechanism\.

In\[[90](https://arxiv.org/html/2608.10144#bib.bib90)\], the authors introduce a subspace regularization term in the objective, making sure the subspaces spanned by the clients’ low\-rank updates stay aligned\. However, in each round, the clients sendΔ​𝐖i=𝐁i​𝐀i\\Delta\\mathbf\{W\}\_\{i\}=\\mathbf\{B\}\_\{i\}\\mathbf\{A\}\_\{i\}to the federator, increasing the communication overhead between the clients and the federator\.

## References

- Achiam et al\. \[2023\]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al\.GPT\-4 technical report\.*arXiv preprint arXiv:2303\.08774*, 2023\.
- Devlin et al\. \[2019\]Jacob Devlin, Ming\-Wei Chang, Kenton Lee, and Kristina Toutanova\.BERT: Pre\-training of deep bidirectional transformers for language understanding\.In*Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\)*, pages 4171–4186, 2019\.
- Anil et al\. \[2023\]Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al\.PaLM 2 technical report\.*arXiv preprint arXiv:2305\.10403*, 2023\.
- Touvron et al\. \[2023\]Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al\.Llama 2: Open foundation and fine\-tuned chat models\.*arXiv preprint arXiv:2307\.09288*, 2023\.
- \[5\]Antrhopic\.The claude 3 model family: Opus, sonnet, haiku\.online\.URL[https://www\-cdn\.anthropic\.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model\_Card\_Claude\_3\.pdf](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf)\.accessed on 03\.Aug\.2026\.
- Dosovitskiy et al\. \[2020\]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al\.An image is worth 16x16 words: Transformers for image recognition at scale\.*arXiv preprint arXiv:2010\.11929*, 2020\.
- Bill and Eriksson \[2023\]Desirée Bill and Theodor Eriksson\.Fine\-tuning a llm using reinforcement learning from human feedback for a therapy chatbot application, 2023\.
- Dong et al\. \[2023\]Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu\.Towards next\-generation intelligent assistants leveraging llm techniques\.In*Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 5792–5793, 2023\.
- Kelly et al\. \[2023\]Dominique Kelly, Yimin Chen, Sarah E Cornwell, Nicole S Delellis, Alex Mayhew, Sodiq Onaolapo, and Victoria L Rubin\.Bing chat: The future of search engines?*Proceedings of the Association for Information Science and Technology*, 60\(1\):1007–1009, 2023\.
- Singhal et al\. \[2023\]Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole\-Lewis, Stephen Pfohl, et al\.Large language models encode clinical knowledge\.*Nature*, 620\(7972\):172–180, 2023\.
- Houlsby et al\. \[2019\]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly\.Parameter\-Efficient Transfer Learning for NLP\.In*Proceedings of the 36th International Conference on Machine Learning*, pages 2790–2799\. PMLR, May 2019\.
- He et al\. \[2021\]Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg\-Kirkpatrick, and Graham Neubig\.Towards a Unified View of Parameter\-Efficient Transfer Learning\.In*International Conference on Learning Representations*, October 2021\.
- Li and Liang \[2021\]Xiang Lisa Li and Percy Liang\.Prefix\-Tuning: Optimizing Continuous Prompts for Generation\.In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,*Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)*, pages 4582–4597, Online, August 2021\. Association for Computational Linguistics\.doi:10\.18653/v1/2021\.acl\-long\.353\.
- Lester et al\. \[2021\]Brian Lester, Rami Al\-Rfou, and Noah Constant\.The Power of Scale for Parameter\-Efficient Prompt Tuning\.In Marie\-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen\-tau Yih, editors,*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 3045–3059, Online and Punta Cana, Dominican Republic, November 2021\. Association for Computational Linguistics\.doi:10\.18653/v1/2021\.emnlp\-main\.243\.
- Zaken et al\. \[2022\]Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg\.BitFit: Simple Parameter\-efficient Fine\-tuning for Transformer\-based Masked Language\-models, September 2022\.
- Liao et al\. \[2023\]Baohao Liao, Yan Meng, and Christof Monz\.Parameter\-Efficient Fine\-Tuning without Introducing New Latency\.In Anna Rogers, Jordan Boyd\-Graber, and Naoaki Okazaki, editors,*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 4242–4260, Toronto, Canada, July 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.acl\-long\.233\.
- Hu et al\. \[2022\]Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al\.LoRA: Low\-rank adaptation of large language models\.*Iclr*, 1\(2\):3, 2022\.
- Liu et al\. \[2024a\]Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\.DoRA: Weight\-Decomposed Low\-Rank Adaptation\.In*Proceedings of the 41st International Conference on Machine Learning*, pages 32100–32121\. PMLR, July 2024a\.
- Tian et al\. \[2025\]Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu\.HydraLoRA: An Asymmetric LoRA architecture for efficient fine\-tuning\.In*Proceedings of the 38th International Conference on Neural Information Processing Systems*, volume 37 of*NIPS ’24*, pages 9565–9584, Red Hook, NY, USA, June 2025\. Curran Associates Inc\.
- Zhang et al\. \[2023\]Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao\.AdaLoRA: Adaptive budget allocation for parameter\-efficient fine\-tuning\.*arXiv preprint arXiv:2303\.10512*, 2023\.
- Valipour et al\. \[2023\]Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi\.DyLoRA: Parameter\-efficient tuning of pre\-trained models using dynamic search\-free low\-rank adaptation\.In*Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pages 3274–3287, 2023\.
- Han et al\. \[2024\]Zeyu Han, Chao Gao, and Jinyang Liu\.Parameter\-Efficient Fine\-Tuning for Large Models: A Comprehensive Survey\.*Transactions on Machine Learning Research*, October 2024\.
- McMahan et al\. \[2017\]Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas\.Communication\-efficient learning of deep networks from decentralized data\.In*Artificial intelligence and statistics*, pages 1273–1282\. Pmlr, 2017\.
- Kairouz and McMahan \[2021\]Peter Kairouz and H Brendan McMahan\.Advances and open problems in federated learning\.*Foundations and trends in machine learning*, 14\(1\-2\):1–210, 2021\.
- Jiang et al\. \[2025\]Wenhao Jiang, Yuchuan Luo, Guilin Deng, Silong Chen, Xu Yang, Shihong Wu, Xinwen Gao, Lin Liu, and Shaojing Fu\.Federated large language models: Feasibility, robustness, security and future directions\.*arXiv preprint arXiv:2505\.08830*, 2025\.
- Li et al\. \[2021\]Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He\.A survey on federated learning systems: Vision, hype and reality for data privacy and protection\.*IEEE Transactions on Knowledge and Data Engineering*, 35\(4\):3347–3366, 2021\.
- Cai et al\. \[2023\]Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu\.Efficient federated learning for modern NLP\.In*Proceedings of the 29th Annual International Conference on Mobile Computing and Networking \(MobiCom\)*, pages 1–16, 2023\.
- Ghiasvand et al\. \[2024\]Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, and Ramtin Pedarsani\.Communication\-efficient and tensorized federated fine\-tuning of large language models\.*arXiv preprint arXiv:2410\.13097*, 2024\.
- Zhao et al\. \[2023\]Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu\.Fedprompt: Communication\-efficient and privacy\-preserving prompt tuning in federated learning\.In*ICASSP*, pages 1–5, 2023\.
- Qiu et al\. \[2023\]Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan\-Yi Lin\.Text\-driven prompt generation for vision\-language models in federated learning\.*arXiv preprint arXiv:2310\.06123*, 2023\.
- Yu et al\. \[2023\]Sixing Yu, J\. Pablo Muñoz, and Ali Jannesari\.Bridging the gap between foundation models and heterogeneous federated learning\.*arXiv preprint arXiv:2310\.00247*, 2023\.
- Peng et al\. \[2026\]Hongyi Peng, Han Yu, Xiaoxiao Li, and Qiang Yang\.Rethinking LoRA for data heterogeneous federated learning: Subspace and state alignment\.*arXiv preprint arXiv:2602\.01746*, 2026\.
- Hayou et al\. \[2024\]Soufiane Hayou, Nikhil Ghosh, and Bin Yu\.LoRA\+: Efficient low rank adaptation of large models\.In*International Conference on Machine Learning \(ICML\)*, 2024\.
- Liu et al\. \[2024b\]Shih\-Yang Liu, Chien\-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu\-Chiang Frank Wang, Kwang\-Ting Cheng, and Min\-Hung Chen\.DoRA: Weight\-decomposed low\-rank adaptation\.In*International Conference on Machine Learning \(ICML\)*, 2024b\.
- Zhou et al\. \[2025a\]Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang\.LoRA\-drop: Efficient LoRA parameter pruning based on output evaluation\.In*Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2025a\.
- Chen et al\. \[2024a\]Songlin Chen, Weicheng Wang, Xiaoliang Chen, Peng Lu, Zaiyan Yang, and Yajun Du\.LLaMA\-LoRA neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains\.*Data Intelligence*, 2024a\.
- Kim et al\. \[2025\]Junsu Kim, Jaeyeon Kim, and Ernest K Ryu\.LoRA training provably converges to a low\-rank global minimum or it fails loudly \(but it probably won’t fail\)\.In*International Conference on Machine Learning*, volume 2025, 2025\.
- Zhang et al\. \[2024\]Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen\.Towards building the federated GPT: Federated instruction tuning\.In*IEEE international conference on acoustics, speech and signal processing \(ICASSP\)*, pages 6915–6919\. IEEE, 2024\.
- Liu et al\. \[2026a\]Muqing Liu, Chongjie Si, and Yuheng Jia\.Flexlora: Entropy\-guided flexible low\-rank adaptation\.*arXiv preprint arXiv:2601\.22905*, 2026a\.
- Liu et al\. \[2026b\]Jin Liu, Yinbin Miao, Ning Xi, and Junkang Liu\.Rethinking loRA for privacy\-preserving federated learning in large models\.In*The Fourteenth International Conference on Learning Representations*, 2026b\.URL[https://openreview\.net/forum?id=BPzSV4uw0x](https://openreview.net/forum?id=BPzSV4uw0x)\.
- Cho et al\. \[2024\]Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi\.Heterogeneous LoRA for federated fine\-tuning of on\-device foundation models\.In*Proceedings of the 2024 conference on empirical methods in natural language processing*, pages 12903–12913, 2024\.
- Chen et al\. \[2024b\]Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti\.Robust federated finetuning of foundation models via alternating minimization of lora\.*arXiv preprint arXiv:2409\.02346*, 2024b\.
- Guo et al\. \[2025\]Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu\.Selective aggregation for low\-rank adaptation in federated learning\.In*13th International Conference on Learning Representations Iclr 2025*, 2025\.
- Sun et al\. \[2024\]Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding\.Improving loRA in privacy\-preserving federated learning\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=NLPzL6HWNl](https://openreview.net/forum?id=NLPzL6HWNl)\.
- Wang et al\. \[2024\]Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li\.FLoRA: Federated fine\-tuning large language models with heterogeneous low\-rank adaptations\.*Advances in Neural Information Processing Systems*, 37:22513–22533, 2024\.
- Bai et al\. \[2024\]Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li\.Federated fine\-tuning of large language models under heterogeneous tasks and client resources\.*Advances in Neural Information Processing Systems*, 37:14457–14483, 2024\.
- Fang et al\. \[2026\]Wenzhi Fang, Dong\-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G Brinton\.Federated sketching LoRA: On\-device collaborative fine\-tuning of large language models\.In*International Conference on Machine Learning \(ICML\)*, 2026\.
- Tropp et al\. \[2017\]Joel A Tropp, Alp Yurtsever, Madeleine Udell, and Volkan Cevher\.Practical sketching algorithms for low\-rank matrix approximation\.*SIAM Journal on Matrix Analysis and Applications*, 38\(4\):1454–1485, 2017\.
- Byun and Lee \[2025\]Yuji Byun and Jaeho Lee\.Towards federated low\-rank adaptation of language models with rank heterogeneity\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\)*, pages 356–362, 2025\.
- Ha et al\. \[2026\]Sihyeon Ha, Yongjeong Oh, and Yo\-Seb Jeon\.RB\-LoRA: Rank\-balanced aggregation for low\-rank adaptation with federated fine\-tuning\.In*Findings of the Association for Computational Linguistics: EACL 2026*, pages 1737–1746, 2026\.
- Yan et al\. \[2026a\]Guochen Yan, Luyuan Xie, Qingni Shen, Yuejian Fang, and Zhonghai Wu\.Fedsrd: Sparsify\-reconstruct\-decompose for communication\-efficient federated large language models fine\-tuning\.In*Proceedings of the ACM Web Conference 2026*, pages 5087–5098, 2026a\.
- Wu et al\. \[2026a\]Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang\.Adaptive rank allocation for federated parameter\-efficient fine\-tuning of language models\.*IEEE Transactions on Computers*, 2026a\.
- Ding et al\. \[2023\]Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun\.Sparse low\-rank adaptation of pre\-trained language models\.In*Proceedings of the 2023 conference on empirical methods in natural language processing*, pages 4133–4145, 2023\.
- Chen et al\. \[2026\]Shuangyi Chen, Yuanxin Guo, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti\.Robust federated finetuning of LLMs via alternating optimization of LoRA\.*Advances in Neural Information Processing Systems*, 38:120038–120090, 2026\.
- Liu et al\. \[2019\]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov\.Roberta: A robustly optimized bert pretraining approach\.*arXiv preprint arXiv:1907\.11692*, 2019\.
- Wang et al\. \[2018\]Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman\.GLUE: A multi\-task benchmark and analysis platform for natural language understanding\.In*Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP*, pages 353–355, 2018\.
- Long et al\. \[2020\]Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang\.Federated learning for open banking\.In*Federated Learning: Privacy and Incentive*, pages 240–254\. Springer, 2020\.
- Chatterjee et al\. \[2023\]Pushpita Chatterjee, Debashis Das, and Danda B\. Rawat\.Federated learning empowered recommendation model for financial consumer services\.*IEEE Transactions on Consumer Electronics*, 2023\.
- Feng et al\. \[2022\]Chun\-Mei Feng, Yunlu Yan, Shanshan Wang, Yong Xu, Ling Shao, and Huazhu Fu\.Specificity\-preserving federated learning for mr image reconstruction\.*IEEE Transactions on Medical Imaging*, 42\(7\):2010–2021, 2022\.
- Jiang et al\. \[2023\]Meirui Jiang, Holger R\. Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu\.Fair federated medical image segmentation via client contribution estimation\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 16302–16311, 2023\.
- Feng et al\. \[2023\]Chun\-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo\.Learning federated visual prompt in null space for mri reconstruction\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 8064–8073, 2023\.
- Yan et al\. \[2024\]Yunlu Yan, Hong Wang, Yawen Huang, Nanjun He, Lei Zhu, Yong Xu, Yuexiang Li, and Yefeng Zheng\.Cross\-modal vertical federated learning for mri reconstruction\.*IEEE Journal of Biomedical and Health Informatics*, 2024\.
- Qu \[2024\]Youyang Qu\.Federated learning driven large language models for swarm intelligence: A survey\.*arXiv preprint arXiv:2406\.09831*, 2024\.
- Fan et al\. \[2023\]Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang\.FATE\-LLM: An industrial grade federated learning framework for large language models\.*arXiv preprint arXiv:2310\.10049*, 2023\.
- Kuang et al\. \[2024\]Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou\.FederatedScope\-LLM: A comprehensive package for fine\-tuning large language models in federated learning\.In*Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 5260–5271, 2024\.
- Ye et al\. \[2024a\]Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen\.OpenFedLLM: Training large language models on decentralized private data via federated learning\.In*Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pages 6137–6147, 2024a\.
- Ye et al\. \[2024b\]Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen\.FedLLM\-Bench: Realistic benchmarks for federated learning of large language models\.*Advances in Neural Information Processing Systems*, 37:111106–111130, 2024b\.
- Liu et al\. \[2024c\]Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YunXiang Gong\.FedBCGD: Communication\-efficient accelerated block coordinate gradient descent for federated learning\.In*Proceedings of the 32nd ACM International Conference on Multimedia*, pages 2955–2963, 2024c\.
- Liu et al\. \[2024d\]Junkang Liu, Yuanyuan Liu, Fanhua Shang, Hongying Liu, Jin Liu, and Wei Feng\.Improving generalization in federated learning with highly heterogeneous data via momentum\-based stochastic controlled weight averaging\.In*International Conference on Machine Learning \(ICML\)*, 2024d\.
- Liu et al\. \[2025a\]Junkang Liu, Fanhua Shang, Kewen Zhu, Hongying Liu, Yuanyuan Liu, and Jin Liu\.FedAdamW: A communication\-efficient optimizer with convergence and generalization guarantees for federated large models\.*arXiv preprint arXiv:2510\.27486*, 2025a\.
- Liu et al\. \[2025b\]Junkang Liu, Fanhua Shang, Yuxuan Tian, Hongying Liu, and Yuanyuan Liu\.Consistency of local and global flatness for federated learning\.In*Proceedings of the 33rd ACM International Conference on Multimedia*, pages 3875–3883, 2025b\.
- Liu et al\. \[2025c\]Junkang Liu, Fanhua Shang, Junchao Zhou, Hongying Liu, Yuanyuan Liu, and Jin Liu\.FedMuon: Accelerating federated learning with matrix orthogonalization\.*arXiv preprint arXiv:2510\.27403*, 2025c\.
- Liu et al\. \[2025d\]Junkang Liu, Yuxuan Tian, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Junchao Zhou, and Daorui Ding\.DP\-FedPGN: Finding global flat minima for differentially private federated learning via penalizing gradient norm\.*arXiv preprint arXiv:2510\.27504*, 2025d\.
- Yan et al\. \[2025\]Yunlu Yan, Chun\-Mei Feng, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu, and Lei Zhu\.Federated residual low\-rank adaptation of large language models\.In*The Thirteenth International Conference on Learning Representations*, 2025\.
- Wen et al\. \[2025\]Ming Wen, Jiaqi Zhu, Yuedong Xu, Yipeng Zhou, and Dingding Han\.Differentially private federated low rank adaptation beyond fixed\-matrix\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2025\.URL[https://openreview\.net/forum?id=TecJ926Vgn](https://openreview.net/forum?id=TecJ926Vgn)\.
- Fang and Ye \[2026\]Xiuwen Fang and Mang Ye\.Towards robust parameter\-efficient fine\-tuning for federated learning\.*Advances in Neural Information Processing Systems*, 38:141777–141800, 2026\.
- Zhang et al\. \[2026a\]Yicheng Zhang, Zhen Qin, Zhaomin Wu, Jian Hou, and Shuiguang Deng\.Personalized federated fine\-tuning for LLMs via data\-driven heterogeneous model architectures\.In*Proceedings of the ACM Web Conference 2026*, pages 5099–5110, 2026a\.
- Lu et al\. \[2026\]Yao Lu, Jiaxing QI, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian\.FDLoRA: Personalized federated learning of large language model via dual lora tuning\.*arXiv preprint arXiv:2406\.07925*, 2026\.URL[https://arxiv\.org/abs/2406\.07925](https://arxiv.org/abs/2406.07925)\.
- Singhal et al\. \[2024\]Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma\.FedEx\-LoRA: Exact aggregation for federated and efficient fine\-tuning of foundation models\.*arXiv preprint arXiv:2410\.09432*, 2024\.
- Lee et al\. \[2026\]Seanie Lee, Sangwoo Park, Dong Bok Lee, Dominik Wagner, Haebin Seong, Tobias Bocklet, Juho Lee, and Sung Ju Hwang\.Fedsvd: Adaptive orthogonalization for private federated learning with lora\.*Advances in Neural Information Processing Systems*, 38:119733–119757, 2026\.
- Koo et al\. \[2025\]Jabin Koo, Minwoo Jang, and Jungseul Ok\.Towards robust and efficient federated low\-rank adaptation with heterogeneous clients\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 416–429, 2025\.
- Zhang et al\. \[2026b\]Haoran Zhang, Dongjun Kim, Seohyeon Cha, and Haris Vikalo\.FedRot\-LoRA: Mitigating rotational misalignment in federated lora\.*arXiv preprint arXiv:2602\.23638*, 2026b\.
- Bian et al\. \[2024\]Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu\.LoRA\-FAIR: Federated LoRA fine\-tuning with aggregation and initialization refinement\.*arXiv preprint arXiv:2411\.14961*, 2024\.
- Babakniya et al\. \[2023\]Sara Babakniya, Ahmed Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee\-Bong Song, MOSTAFA EL\-Khamy, and Salman Avestimehr\.SLoRA: Federated parameter efficient fine\-tuning of language models\.In*International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023*, 2023\.URL[https://openreview\.net/forum?id=06quMTmtRV](https://openreview.net/forum?id=06quMTmtRV)\.
- Yan et al\. \[2026b\]Peishen Yan, Yang Hua, Hao Wang, Jiaru Zhang, Xiaoyu Wu, Tao Song, and Haibing Guan\.FedMomentum: Preserving lora training momentum in federated fine\-tuning\.*arXiv preprint arXiv:2603\.08014*, 2026b\.
- Zhou et al\. \[2025b\]Yajie Zhou, Xiaoyi Pang, and Zhibo Wang\.AFLoRA: Adaptive federated fine\-tuning of large language models with resource\-aware low\-rank adaption\.*arXiv preprint arXiv:2505\.24773*, 2025b\.
- Wu et al\. \[2026b\]Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang\.Preventing rank collapse in federated low\-rank adaptation with client heterogeneity\.*arXiv preprint arXiv:2602\.13486*, 2026b\.
- Peng et al\. \[2025\]Zihao Peng, Jiandian Zeng, Boyuan Li, Guo Li, Shengbo Chen, and Tian Wang\.Fedhl: Federated learning for heterogeneous low\-rank adaptation via unbiased aggregation\.*arXiv preprint arXiv:2505\.18494*, 2025\.
- Meng et al\. \[2026\]Chuiyang Meng, Ming Tang, and Vincent Wong\.Florg: Federated fine\-tuning with low\-rank gram matrices and procrustes alignment\.In*International Conference on Learning Representations*, volume 2026, pages 82804–82824, 2026\.
- Senarath et al\. \[2026\]Neranjan Senarath, Rohit Muralitharan, and Sadia Asif\.Subspace\-constrained federated learning with low\-rank adaptation\.*arXiv preprint arXiv:2606\.22724*, 2026\.

## Appendix AAlgorithmic representation ofSeFoRA\-Ho

SeFoRA\-Hois summarized in[Algorithm˜5](https://arxiv.org/html/2608.10144#alg5)\.

Algorithm 5SeFoRA\-Ho1:The pretrained model

𝐖0\\mathbf\{W\}\_\{0\}\. Total rounds

TTand local steps

EE\. In total,

NNclients, each holding their local dataset

𝒟i\\mathcal\{D\}\_\{i\}\. The aggregation weights

pi=\|𝒟i\|/∑i=1N\|𝒟i\|p\_\{i\}=\|\\mathcal\{D\}\_\{i\}\|/\\sum\_\{i=1\}^\{N\}\|\\mathcal\{D\}\_\{i\}\|\. The global rank profile

𝐫=\(r1,⋯,rL\)\\mathbf\{r\}=\(r\_\{1\},\\cdots,r\_\{L\}\)and

kl\>rl\+1k\_\{l\}\>r\_\{l\}\+1for every

l∈\[L\]l\\in\[L\]\. Set

rmax≜maxl∈\[L\]⁡rlr\_\{\\max\}\\triangleq\\max\_\{l\\in\[L\]\}r\_\{l\}, and

kmax≜maxl∈\[L\]⁡klk\_\{\\max\}\\triangleq\\max\_\{l\\in\[L\]\}k\_\{l\}\. Initialize global matrices

𝐁l\(0\)∈ℝd×rl\\mathbf\{B\}\_\{l\}^\{\(0\)\}\\in\\mathbb\{R\}^\{d\\times r\_\{l\}\}as a full zero matrix and

𝐀l\(0\)∈ℝrl×s\\mathbf\{A\}\_\{l\}^\{\(0\)\}\\in\\mathbb\{R\}^\{r\_\{l\}\\times s\}as a random Gaussian matrix\.

𝙱\(0\)=\(𝐁1\(0\),⋯,𝐁L\(0\)\)\\mathbf\{\\mathtt\{B\}\}^\{\(0\)\}=\(\\mathbf\{B\}\_\{1\}^\{\(0\)\},\\cdots,\\mathbf\{B\}\_\{L\}^\{\(0\)\}\)and

𝙰\(0\)=\(𝐀1\(0\),⋯,𝐀L\(0\)\)\\mathbf\{\\mathtt\{A\}\}^\{\(0\)\}=\(\\mathbf\{A\}\_\{1\}^\{\(0\)\},\\cdots,\\mathbf\{A\}\_\{L\}^\{\(0\)\}\)\. A function

Unsketching​\(𝐘,𝐙,𝚿,𝛀\)\\textsc\{Unsketching\}\(\\mathbf\{Y\},\\mathbf\{Z\},\\mathbf\{\\Psi\},\\mathbf\{\\Omega\}\)that unsketch the sketch matrices \([Algorithm˜1](https://arxiv.org/html/2608.10144#alg1)\)\.

2:forround

t=0,1,⋯,T−1t=0,1,\\cdots,T\-1do

3:Federator initializes

𝚿\(t\+1\)∈ℝkmax×d\\mathbf\{\\Psi\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{k\_\{\\max\}\\times d\}and

𝛀\(t\+1\)∈ℝs×rmax\\mathbf\{\\Omega\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{s\\times r\_\{\\max\}\}from a standard normal distribution, and send the shared seed that generates

𝚿\(t\+1\)\\mathbf\{\\Psi\}^\{\(t\+1\)\}and

𝛀\(t\+1\)\\mathbf\{\\Omega\}^\{\(t\+1\)\}to all clients\.

4:Federator broadcasts

𝙱\(t\)\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}and

𝙰\(t\)\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}to all clients\.

5:for allclients

iiin paralleldo

6:for alladapted layers

l∈\[L\]l\\in\[L\]do

7:Extract

𝛀l\(t\+1\)=𝛀:,1:rl\(t\+1\)∈ℝs×rl\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\\\!\\\!=\\\!\\mathbf\{\\Omega\}^\{\(t\+1\)\}\_\{:,1:r\_\{l\}\}\\\!\\in\\\!\\mathbb\{R\}^\{s\\times r\_\{l\}\}and

𝚿l\(t\+1\)=𝚿1:kl,:\(t\+1\)∈ℝkl×d\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\\!\\\!=\\\!\\mathbf\{\\Psi\}^\{\(t\+1\)\}\_\{1:k\_\{l\},:\}\\\!\\in\\\!\\mathbb\{R\}^\{k\_\{l\}\\times d\}from the shared seed\.

8:Obtain

𝐁l\(t\)\\mathbf\{B\}\_\{l\}^\{\(t\)\}and

𝐀l\(t\)\\mathbf\{A\}\_\{l\}^\{\(t\)\}\.

9:Initialize local LoRA factors

𝐁i,l\(t\)=𝐁l\(t\)\\mathbf\{B\}\_\{i,l\}^\{\(t\)\}=\\mathbf\{B\}\_\{l\}^\{\(t\)\}and

𝐀i,l\(t\)=𝐀l\(t\)\\mathbf\{A\}\_\{i,l\}^\{\(t\)\}=\\mathbf\{A\}\_\{l\}^\{\(t\)\}\.

10:endfor

11:Using

𝐁i,l\(t\)\\mathbf\{B\}\_\{i,l\}^\{\(t\)\}and

𝐀i,l\(t\)\\mathbf\{A\}\_\{i,l\}^\{\(t\)\}, perform

EElocal steps to obtain

𝐁i,l\(t\+1\)\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}and

𝐀i,l\(t\+1\)\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}for all

l∈\[L\]l\\in\[L\]\.

12:for alladapter layer

l∈\[L\]l\\in\[L\]do

13:Compute the sketch matrices

𝐘i,l\(t\+1\)\\displaystyle\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,l\}=Δ​𝐖i,l\(t\+1\)​𝛀l\(t\+1\)=𝐁i,l\(t\+1\)​\(𝐀i,l\(t\+1\)​𝛀l\(t\+1\)\)∈ℝd×rl,\\displaystyle=\\Delta\\mathbf\{W\}^\{\(t\+1\)\}\_\{i,l\}\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}=\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\(\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\)\\in\\mathbb\{R\}^\{d\\times r\_\{l\}\},𝐙i,l\(t\+1\)\\displaystyle\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,l\}=𝚿l\(t\+1\)​Δ​𝐖i,l\(t\+1\)=\(𝚿l\(t\+1\)​𝐁i,l\(t\+1\)\)​𝐀i,l\(t\+1\)∈ℝkl×s\\displaystyle=\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\Delta\\mathbf\{W\}^\{\(t\+1\)\}\_\{i,l\}=\(\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\}\\mathbf\{B\}\_\{i,l\}^\{\(t\+1\)\}\)\\mathbf\{A\}\_\{i,l\}^\{\(t\+1\)\}\\in\\mathbb\{R\}^\{k\_\{l\}\\times s\}
14:endfor

15:Send

𝚈i\(t\+1\)=\(𝐘i,1\(t\+1\),⋯,𝐘i,L\(t\+1\)\)\\mathbf\{\\mathtt\{Y\}\}^\{\(t\+1\)\}\_\{i\}=\(\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,1\},\\cdots,\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,L\}\)and

𝚉i\(t\+1\)=\(𝐙i,1\(t\+1\),⋯,𝐙i,L\(t\+1\)\)\\mathbf\{\\mathtt\{Z\}\}^\{\(t\+1\)\}\_\{i\}=\(\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,1\},\\cdots,\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,L\}\)to federator\.

16:endfor

17:for alladapter layer

l∈\[L\]l\\in\[L\]do

18:Federator aggregates

𝐘l\(t\+1\)=∑i∈\[N\]pi​𝐘i,l\(t\+1\)​and​𝐙l\(t\+1\)=∑i∈\[N\]pi​𝐙i,l\(t\+1\)\\mathbf\{Y\}\_\{l\}^\{\(t\+1\)\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{Y\}^\{\(t\+1\)\}\_\{i,l\}\\;\\text\{ and \}\\;\\mathbf\{Z\}\_\{l\}^\{\(t\+1\)\}=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{Z\}^\{\(t\+1\)\}\_\{i,l\}\.

19:

\(𝐁l\(t\+1\),𝐀l\(t\+1\)\)←Unsketching​\(𝐘l\(t\+1\),𝐙l\(t\+1\),𝚿l\(t\+1\),𝛀l\(t\+1\)\)\(\\mathbf\{B\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{A\}\_\{l\}^\{\(t\+1\)\}\)\\leftarrow\\textsc\{Unsketching\}\(\\mathbf\{Y\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{Z\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{\\Psi\}\_\{l\}^\{\(t\+1\)\},\\mathbf\{\\Omega\}\_\{l\}^\{\(t\+1\)\}\)
20:endfor

21:endfor

22:Return

\(𝙱\(T\),𝙰\(T\)\)\(\\mathbf\{\\mathtt\{B\}\}^\{\(T\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(T\)\}\)

## Appendix BProof of[˜1](https://arxiv.org/html/2608.10144#Thmthm1)and[Corollary˜1](https://arxiv.org/html/2608.10144#Thmcorollary1)

In the proofs below, we adopt the following notations:

Letℱt\\mathcal\{F\}\_\{t\}represent all algorithmic history up to the beginning of roundtt\. We define𝔼t\[⋅\]≜𝔼\[⋅∣ℱt\]\\mathbb\{E\}\_\{t\}\[\\cdot\]\\triangleq\\mathbb\{E\}\[\\cdot\\mid\\mathcal\{F\}\_\{t\}\]as the conditional expectation taking into account all sources of randomness occurring during roundtt\. Let𝔼ξi​\[⋅\]\\mathbb\{E\}\_\{\\xi\_\{i\}\}\[\\cdot\]denote the expectation taken with respect to the local mini\-batch sampling at roundtt, conditioned on the model state at the moment of sampling\.

We define

g𝙰\(t\)\\displaystyle g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}=∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\),\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\),g𝙱\(t\)\\displaystyle\\quad g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}=∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)g𝙰,i\(t\)\\displaystyle g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}=∇𝙰ℒilora​\(𝙱\(t\),𝙰\(t\)\),\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\),g𝙱,i\(t\)\\displaystyle\\quad g\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}=∇𝙱ℒilora​\(𝙱\(t\),𝙰\(t\)\)\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)g^𝙰,i\(t,e\)\\displaystyle\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}=∇𝙰ℒilora​\(𝙱i\(t,e\),𝙰i\(t,e\);ξi\(e\)\),\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\),g^𝙱,i\(t,e\)\\displaystyle\\quad\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\}=∇𝙱ℒilora​\(𝙱i\(t,e\),𝙰i\(t,e\);ξi\(e\)\)\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)g𝙰,i\(t,e\)\\displaystyle g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}=∇𝙰ℒilora​\(𝙱i\(t,e\),𝙰i\(t,e\)\),\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\),g𝙱,i\(t,e\)\\displaystyle\\quad g\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\}=∇𝙱ℒilora​\(𝙱i\(t,e\),𝙰i\(t,e\)\)\\displaystyle=\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\)\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\\displaystyle\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}=1E​∑e=0E−1g^𝙰,i\(t,e\),\\displaystyle=\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\},\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\\displaystyle\\quad\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}=1E​∑e=0E−1g^𝙱,i\(t,e\)\\displaystyle=\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\}D𝙰\(t\)\\displaystyle D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}=∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]−g𝙰\(t\),\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\-g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\},D𝙱\(t\)\\displaystyle\\quad D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}=∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\]−g𝙱\(t\)\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\right\]\-g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}=∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]−g𝙰,i\(t\),\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\-g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\},=∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\]−g𝙱,i\(t\)​\(Drift Terms\)\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\right\]\-g\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\text\{\\quad\(Drift Terms\)\}
###### Lemma 3\.

Under[˜2](https://arxiv.org/html/2608.10144#Thmassumption2)and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3), the local drift of the LoRA factors𝙱\\mathbf\{\\mathtt\{B\}\}and𝙰\\mathbf\{\\mathtt\{A\}\}, and the updateΔ​𝚆\\Delta\\mathbf\{\\mathtt\{W\}\}at the local stepe∈\{0,⋯,E−1\}e\\in\\\{0,\\cdots,E\-1\\\}, and for any roundt∈\{0,⋯,T−1\}t\\in\\\{0,\\cdots,T\-1\\\}and clienti∈\[N\]i\\in\[N\]inSeFoRA\-Hoare bounded as:

𝔼​‖𝙱i\(t,e\)−𝙱\(t\)‖F2\\displaystyle\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤\(η\(t\)\)2​e2​χ2​M𝐀2,\\displaystyle\\leq\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\},𝔼​‖𝙰i\(t,e\)−𝙰\(t\)‖F2\\displaystyle\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤\(η\(t\)\)2​e2​χ2​M𝐁2,and\\displaystyle\\leq\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}M\_\{\\mathbf\{B\}\}^\{2\},\\quad\\text\{and\}𝔼​‖Δ​𝚆i\(t,e\)−Δ​𝚆\(t\)‖F2\\displaystyle\\mathbb\{E\}\\left\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤2​\(η\(t\)\)2​e2​χ2​\(M𝐁4\+M𝐀4\)\.\\displaystyle\\leq 2\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\.
###### Proof\.

The local fine\-tuning starts from\(𝙱i\(t,0\),𝙰i\(t,0\)\)=\(𝙱\(t\),𝙰\(t\)\)\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,0\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,0\)\}\)=\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)and updates

𝙱i\(t,e\+1\)\\displaystyle\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\+1\)\}=𝙱i\(t,e\)−η\(t\)​g^𝙱,i\(t,e\),\\displaystyle=\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\eta^\{\(t\)\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\},𝙰i\(t,e\+1\)\\displaystyle\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\+1\)\}=𝙰i\(t,e\)−η\(t\)​g^𝙰,i\(t,e\),\\displaystyle=\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\eta^\{\(t\)\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\},where we defineg^𝙰,i\(t,e\)=∇𝙰ℒifull​\(𝙱i\(t,e\),𝙰i\(t,e\);ξi\(e\)\)=\(𝙱i\(t,e\)\)⊤⋅∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\);ξi\(e\)\)\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\\\!=\\\!\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)\\\!=\\\!\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\)^\{\\top\}\\cdot\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)andg^𝙱,i\(t,e\)=∇𝙱ℒifull​\(𝙱i\(t,e\),𝙰i\(t,e\);ξi\(e\)\)=∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\);ξi\(e\)\)⋅\(𝙰i\(t,e\)\)⊤\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\}\\\!=\\\!\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)\\\!=\\\!\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)\\cdot\(\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\)^\{\\top\}\. Thus, fore∈\{0,⋯,E−1\}e\\in\\\{0,\\cdots,E\\\!\-\\\!1\\\},

𝔼​‖𝙱i\(t,e\)−𝙱\(t\)‖F2\\displaystyle\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=𝔼​‖−η\(t\)​∑j=0e−1g^𝙱,i\(t,j\)‖F2\\displaystyle=\\mathbb\{E\}\\left\\\|\-\\eta^\{\(t\)\}\\sum\_\{j=0\}^\{e\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,j\)\}\\right\\\|\_\{F\}^\{2\}=𝔼​\[\(η\(t\)\)2⋅‖∑j=0e−1g^𝙱,i\(t,j\)‖F2\]\\displaystyle=\\mathbb\{E\}\\left\[\(\\eta^\{\(t\)\}\)^\{2\}\\cdot\\left\\\|\\sum\_\{j=0\}^\{e\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,j\)\}\\right\\\|\_\{F\}^\{2\}\\right\]≤\(a\)\(η\(t\)\)2​e⋅∑j=0e−1𝔼​‖g^𝙱,i\(t,j\)‖F2\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{\\leq\}\}\(\\eta^\{\(t\)\}\)^\{2\}e\\cdot\\sum\_\{j=0\}^\{e\-1\}\\mathbb\{E\}\\left\\\|\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,j\)\}\\right\\\|\_\{F\}^\{2\}≤\(b\)\(η\(t\)\)2​e⋅∑j=0e−1𝔼​‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,j\);ξi\(j\)\)‖F2⋅‖𝙰i\(t,j\)‖22\\displaystyle\\stackrel\{\{\\scriptstyle\(b\)\}\}\{\{\\leq\}\}\(\\eta^\{\(t\)\}\)^\{2\}e\\cdot\\sum\_\{j=0\}^\{e\-1\}\\mathbb\{E\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,j\)\};\\xi\_\{i\}^\{\(j\)\}\)\\right\\\|\_\{F\}^\{2\}\\cdot\\left\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,j\)\}\\right\\\|\_\{2\}^\{2\}≤\(c\)\(η\(t\)\)2​e2​χ2​M𝐀2\\displaystyle\\stackrel\{\{\\scriptstyle\(c\)\}\}\{\{\\leq\}\}\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}where\(a\)\(a\)holds due to the Jensen’s inequality,\(b\)\(b\)employs‖X​Y‖F2≤‖X‖22⋅‖Y‖F2\\\|XY\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{2\}^\{2\}\\cdot\\\|Y\\\|\_\{F\}^\{2\}, and\(c\)\(c\)utilizes[˜2](https://arxiv.org/html/2608.10144#Thmassumption2)and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3)\. Similarly,𝔼​‖𝙰i\(t,e\)−𝙰\(t\)‖F2≤\(η\(t\)\)2​e2​χ2​M𝐁2\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\leq\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\. Then,

𝔼​‖Δ​𝚆i\(t,e\)−Δ​𝚆\(t\)‖F2\\displaystyle\\mathbb\{E\}\\left\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=𝔼​‖𝙱i\(t,e\)​𝙰i\(t,e\)−𝙱\(t\)​𝙰\(t\)‖F2\\displaystyle=\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=𝔼​‖𝙱i\(t,e\)​\(𝙰i\(t,e\)−𝙰\(t\)\)\+\(𝙱i\(t,e\)−𝙱\(t\)\)​𝙰\(t\)‖F2\\displaystyle=\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\(\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\+\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\)\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤2​𝔼​‖𝙱i\(t,e\)‖F2​‖𝙰i\(t,e\)−𝙰\(t\)‖F2\+2​𝔼​‖𝙱i\(t,e\)−𝙱\(t\)‖F2​‖𝙰\(t\)‖F2\\displaystyle\\leq 2\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\\right\\\|\_\{F\}^\{2\}\\left\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\+2\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\left\\\|\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤2​M𝐁2​𝔼​‖𝙰i\(t,e\)−𝙰\(t\)‖F2\+2​M𝐀2​𝔼​‖𝙱i\(t,e\)−𝙱\(t\)‖F2\\displaystyle\\leq 2M\_\{\\mathbf\{B\}\}^\{2\}\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\+2M\_\{\\mathbf\{A\}\}^\{2\}\\mathbb\{E\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤2​\(η\(t\)\)2​e2​χ2​\(M𝐁4\+M𝐀4\)\\displaystyle\\leq 2\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)∎

###### Lemma 4\.

For roundt∈\{0,⋯,T−1\}t\\in\\\{0,\\cdots,T\-1\\\}, under[˜1](https://arxiv.org/html/2608.10144#Thmassumption1),[˜2](https://arxiv.org/html/2608.10144#Thmassumption2)and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3), the following bounds hold inSeFoRA\-Ho:

‖D𝙰\(t\)‖F2≤\(E−1\)​\(2​E−1\)3​χ2​\(η\(t\)\)2​\(χ2​M𝐀2\+2​μ2​\(M𝐁4\+M𝐀4\)\),\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\leq\\frac\{\(E\-1\)\(2E\-1\)\}\{3\}\\chi^\{2\}\(\\eta^\{\(t\)\}\)^\{2\}\\left\(\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\+2\\mu^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\),and

‖D𝙱\(t\)‖F2≤\(E−1\)​\(2​E−1\)3​χ2​\(η\(t\)\)2​\(χ2​M𝐁2\+2​μ2​M𝐀2​\(M𝐁4\+M𝐀4\)\)\.\\\|D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\leq\\frac\{\(E\-1\)\(2E\-1\)\}\{3\}\\chi^\{2\}\(\\eta^\{\(t\)\}\)^\{2\}\\left\(\\chi^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\+2\\mu^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\.
###### Proof\.

We prove the bound on𝙰\\mathbf\{\\mathtt\{A\}\}; the𝙱\\mathbf\{\\mathtt\{B\}\}bound follows by the symmetric argument\. The Frobenius norm of the drifts‖D𝙰\(t\)‖F2\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}is bounded as:

‖D𝙰\(t\)‖F2=‖∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]−g𝙰\(t\)‖F2\\displaystyle\\left\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\-g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=‖∑i∈\[N\]pi​1E​∑e=0E−1𝔼t​\[g^𝙰,i\(t,e\)\]−∑i∈\[N\]pi​1E​∑e=0E−1g𝙰,i\(t\)‖F2\\displaystyle=\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\[\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\\right\]\-\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=‖∑i∈\[N\]pi​1E​∑e=0E−1𝔼t​\[g𝙰,i\(t,e\)−g𝙰,i\(t\)\]‖F2\\displaystyle=\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\[g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\-g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\\right\\\|\_\{F\}^\{2\}≤\(a\)∑i∈\[N\]pi​1E​∑e=0E−1𝔼t​‖g𝙰,i\(t,e\)−g𝙰,i\(t\)‖F2\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{\\leq\}\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\-g\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=∑i∈\[N\]pi​1E​∑e=0E−1𝔼t​‖\(𝙱i\(t,e\)\)⊤⋅∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\)\)−\(𝙱\(t\)\)⊤​∇Δ​𝚆ℒifull​\(Δ​𝚆\(t\)\)‖F2\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\)^\{\\top\}\\cdot\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\)\-\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\)^\{\\top\}\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}=∑i∈\[N\]pi1E∑e=0E−1𝔼t∥\(\(𝙱i\(t,e\)\)⊤−\(𝙱\(t\)\)⊤\)⋅∇Δ​𝚆ℒifull\(Δ𝚆i\(t,e\)\)\+\(𝙱\(t\)\)⊤⋅\(∇Δ​𝚆ℒifull\(Δ𝚆i\(t,e\)\)−∇Δ​𝚆ℒifull\(Δ𝚆\(t\)\)∥F2\)\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\left\(\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\)^\{\\top\}\-\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\)^\{\\top\}\\right\)\\cdot\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\)\+\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\)^\{\\top\}\\cdot\\left\(\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\)\-\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\)≤\(b\)2​∑i∈\[N\]pi​1E​∑e=0E−1𝔼t​‖𝙱i\(t,e\)−𝙱\(t\)‖F2​‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\)\)‖F2\\displaystyle\\stackrel\{\{\\scriptstyle\(b\)\}\}\{\{\\leq\}\}2\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\)\\right\\\|\_\{F\}^\{2\}\+2​∑i∈\[N\]pi​1E​∑e=0E−1𝔼t​‖𝙱\(t\)‖22​‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\)\)−∇Δ​𝚆ℒifull​\(Δ​𝚆\(t\)\)‖F2\\displaystyle\\quad\+2\\sum\_\{i\\in\[N\]\}p\_\{i\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{2\}^\{2\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\)\-\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}≤\(c\)2​∑i∈\[N\]pi​χ2​1E​∑e=0E−1𝔼t​‖𝙱i\(t,e\)−𝙱\(t\)‖F2\+2⋅∑i∈\[N\]pi​μ2​1E​∑e=0E−1𝔼t​‖Δ​𝚆i\(t,e\)−Δ​𝚆\(t\)‖F2\\displaystyle\\stackrel\{\{\\scriptstyle\(c\)\}\}\{\{\\leq\}\}2\\sum\_\{i\\in\[N\]\}p\_\{i\}\\chi^\{2\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\+2\\cdot\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mu^\{2\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤\(d\)2​∑i∈\[N\]pi​χ2​1E​∑e=0E−1\(η\(t\)\)2​e2​χ2​M𝐀2\+2⋅∑i∈\[N\]pi​μ2​1E​∑e=0E−12​\(η\(t\)\)2​e2​χ2​\(M𝐁4\+M𝐀4\)\\displaystyle\\stackrel\{\{\\scriptstyle\(d\)\}\}\{\{\\leq\}\}2\\sum\_\{i\\in\[N\]\}p\_\{i\}\\chi^\{2\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\+2\\cdot\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mu^\{2\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}2\(\\eta^\{\(t\)\}\)^\{2\}e^\{2\}\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)=2​χ2​\(η\(t\)\)2​\(χ2​M𝐀2\+2​μ2​\(M𝐁4\+M𝐀4\)\)⋅1E​∑e=0E−1e2\\displaystyle=2\\chi^\{2\}\(\\eta^\{\(t\)\}\)^\{2\}\\left\(\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\+2\\mu^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\\cdot\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}e^\{2\}=\(E−1\)​\(2​E−1\)3​χ2​\(η\(t\)\)2​\(χ2​M𝐀2\+2​μ2​\(M𝐁4\+M𝐀4\)\)\\displaystyle=\\frac\{\(E\-1\)\(2E\-1\)\}\{3\}\\chi^\{2\}\(\\eta^\{\(t\)\}\)^\{2\}\\left\(\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\+2\\mu^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)where\(a\)\(a\)holds by applying Jensen’s inequality three times,\(b\)\(b\)holds since‖X​Y‖F2≤‖X‖22⋅‖Y‖F2≤‖X‖F2⋅‖Y‖F2\\\|XY\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{2\}^\{2\}\\cdot\\\|Y\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{F\}^\{2\}\\cdot\\\|Y\\\|\_\{F\}^\{2\},\(c\)\(c\)holds since‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\)\)‖F2=‖𝔼ξi​\[∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\);ξi\)\]‖F2≤𝔼ξi​‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\);ξi\)‖F2≤χ2\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\)\\right\\\|\_\{F\}^\{2\}=\\left\\\|\\mathbb\{E\}\_\{\\xi\_\{i\}\}\\left\[\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}\)\\right\]\\right\\\|\_\{F\}^\{2\}\\leq\\mathbb\{E\}\_\{\\xi\_\{i\}\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}\)\\right\\\|\_\{F\}^\{2\}\\leq\\chi^\{2\}\([˜2](https://arxiv.org/html/2608.10144#Thmassumption2)\),[˜1](https://arxiv.org/html/2608.10144#Thmassumption1)and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3), and\(d\)\(d\)holds due to[˜3](https://arxiv.org/html/2608.10144#Thmlem3)and∑e=1E−1e2=\(E−1\)​E​\(2​E−1\)6\\sum\_\{e=1\}^\{E\-1\}e^\{2\}=\\frac\{\(E\-1\)E\(2E\-1\)\}\{6\}\. Similarly,

‖D𝙱\(t\)‖F2\\displaystyle\\\|D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}≤2​∑i∈\[N\]pi​χ2​1E​∑e=0E−1𝔼t​‖𝙰i\(t,e\)−𝙰\(t\)‖F2\+2​M𝐀2⋅∑i∈\[N\]pi​μ2​1E​∑e=0E−1𝔼t​‖Δ​𝚆i\(t,e\)−Δ​𝚆\(t\)‖F2\\displaystyle\\leq 2\\sum\_\{i\\in\[N\]\}p\_\{i\}\\chi^\{2\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\+2M\_\{\\mathbf\{A\}\}^\{2\}\\cdot\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mu^\{2\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}≤\(E−1\)​\(2​E−1\)3​χ2​\(η\(t\)\)2​\(χ2​M𝐁2\+2​μ2​M𝐀2​\(M𝐁4\+M𝐀4\)\)\.\\displaystyle\\leq\\frac\{\(E\-1\)\(2E\-1\)\}\{3\}\\chi^\{2\}\(\\eta^\{\(t\)\}\)^\{2\}\\left\(\\chi^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\+2\\mu^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\.∎

###### Lemma 5\.

For each clienti∈\[N\]i\\in\[N\]and roundt∈\{0,⋯,T−1\}t\\in\\\{0,\\cdots,T\-1\\\}, under[˜2](https://arxiv.org/html/2608.10144#Thmassumption2)and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3), the following bounds hold inSeFoRA\-Ho:𝔼t​‖\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)‖F2≤M𝐀2​M𝐁2​κ4;\\mathbb\{E\}\_\{t\}\\left\\\|\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\,\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\;\\leq\\;M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\};

###### Proof\.

𝔼t∥g¯𝙱,i\(t\)⋅\\displaystyle\\mathbb\{E\}\_\{t\}\\Big\\\|\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\cdotg¯𝙰,i\(t\)∥F2=𝔼t​\[‖1E2​∑e=0E−1g^𝙱,i\(t,e\)⋅∑e=0E−1g^𝙰,i\(t,e\)‖F2\]\\displaystyle\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\Big\\\|\_\{F\}^\{2\}=\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\frac\{1\}\{E^\{2\}\}\\sum\_\{e=0\}^\{E\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\}\\cdot\\sum\_\{e=0\}^\{E\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\\right\\\|\_\{F\}^\{2\}\\right\]≤\(a\)1E4​E2⋅∑e1,e2𝔼t​\[‖g^𝙱,i\(t,e1\)⋅g^𝙰,i\(t,e2\)‖F2\]\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{\\leq\}\}\\frac\{1\}\{E^\{4\}\}E^\{2\}\\cdot\\sum\_\{e\_\{1\},e\_\{2\}\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\_\{1\}\)\}\\cdot\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\_\{2\}\)\}\\right\\\|\_\{F\}^\{2\}\\right\]=1E2⋅∑e1,e2𝔼t​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e1\);ξi\(e1\)\)⋅\(𝙰i\(t,e1\)\)⊤⋅\(𝙱i\(t,e2\)\)⊤⋅∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e2\);ξi\(e2\)\)‖F2\]\\displaystyle=\\frac\{1\}\{E^\{2\}\}\\cdot\\sum\_\{e\_\{1\},e\_\{2\}\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\_\{1\}\)\};\\xi\_\{i\}^\{\(e\_\{1\}\)\}\)\\cdot\(\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\_\{1\}\)\}\)^\{\\top\}\\cdot\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\_\{2\}\)\}\)^\{\\top\}\\cdot\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\_\{2\}\)\};\\xi\_\{i\}^\{\(e\_\{2\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]=1E2​∑e1,e2𝔼t​\[∑l=1L‖∇Δ​𝐖ℒifull​\(Δ​𝐖i,l\(t,e1\);ξi\(e1\)\)​\(𝐀i,l\(t,e1\)\)⊤⋅\(𝐁i,l\(t,e2\)\)⊤​∇Δ​𝐖ℒifull​\(Δ​𝐖i,l\(t,e2\);ξi\(e2\)\)‖F2\]\\displaystyle=\\frac\{1\}\{E^\{2\}\}\\sum\_\{e\_\{1\},e\_\{2\}\}\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{l=1\}^\{L\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t,e\_\{1\}\)\};\\xi\_\{i\}^\{\(e\_\{1\}\)\}\)\(\\mathbf\{A\}\_\{i,l\}^\{\(t,e\_\{1\}\)\}\)^\{\\top\}\\cdot\(\\mathbf\{B\}\_\{i,l\}^\{\(t,e\_\{2\}\)\}\)^\{\\top\}\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t,e\_\{2\}\)\};\\xi\_\{i\}^\{\(e\_\{2\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]≤\(b\)1E2​∑e1,e2𝔼t​\[∑l=1L‖∇Δ​𝐖ℒifull​\(Δ​𝐖i,l\(t,e1\);ξi\(e1\)\)‖F2⋅‖𝐀i,l\(t,e1\)‖22⋅‖𝐁i,l\(t,e2\)‖22⋅‖∇Δ​𝐖ℒifull​\(Δ​𝐖i,l\(t,e2\);ξi\(e2\)\)‖F2\]\\displaystyle\\stackrel\{\{\\scriptstyle\(b\)\}\}\{\{\\leq\}\}\\frac\{1\}\{E^\{2\}\}\\sum\_\{e\_\{1\},e\_\{2\}\}\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{l=1\}^\{L\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t,e\_\{1\}\)\};\\xi\_\{i\}^\{\(e\_\{1\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\cdot\\left\\\|\\mathbf\{A\}\_\{i,l\}^\{\(t,e\_\{1\}\)\}\\right\\\|\_\{2\}^\{2\}\\cdot\\left\\\|\\mathbf\{B\}\_\{i,l\}^\{\(t,e\_\{2\}\)\}\\right\\\|\_\{2\}^\{2\}\\cdot\\left\\\|\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t,e\_\{2\}\)\};\\xi\_\{i\}^\{\(e\_\{2\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]≤\(c\)M𝐀2​M𝐁2E2​∑e1,e2𝔼t​\[∑l=1L‖∇Δ​𝐖ℒifull​\(Δ​𝐖i,l\(t,e1\);ξi\(e1\)\)‖F2⋅‖∇Δ​𝐖ℒifull​\(Δ​𝐖i,l\(t,e2\);ξi\(e2\)\)‖F2\]\\displaystyle\\stackrel\{\{\\scriptstyle\(c\)\}\}\{\{\\leq\}\}\\frac\{M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\}\{E^\{2\}\}\\sum\_\{e\_\{1\},e\_\{2\}\}\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{l=1\}^\{L\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t,e\_\{1\}\)\};\\xi\_\{i\}^\{\(e\_\{1\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\cdot\\left\\\|\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{i,l\}^\{\(t,e\_\{2\}\)\};\\xi\_\{i\}^\{\(e\_\{2\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]≤\(d\)M𝐀2​M𝐁2E2​∑e1,e2𝔼t​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e1\);ξi\(e1\)\)‖F2​‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e2\);ξi\(e2\)\)‖F2\]\\displaystyle\\stackrel\{\{\\scriptstyle\(d\)\}\}\{\{\\leq\}\}\\frac\{M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\}\{E^\{2\}\}\\sum\_\{e\_\{1\},e\_\{2\}\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\_\{1\}\)\};\\xi\_\{i\}^\{\(e\_\{1\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\_\{2\}\)\};\\xi\_\{i\}^\{\(e\_\{2\}\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]≤\(e\)M𝐀2​M𝐁2E2​∑e1,e2𝔼t​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e1\);ξi\(e1\)\)‖F4\]​𝔼t​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e2\);ξi\(e2\)\)‖F4\]\\displaystyle\\stackrel\{\{\\scriptstyle\(e\)\}\}\{\{\\leq\}\}\\frac\{M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\}\{E^\{2\}\}\\sum\_\{e\_\{1\},e\_\{2\}\}\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\_\{1\}\)\};\\xi\_\{i\}^\{\(e\_\{1\}\)\}\)\\right\\\|\_\{F\}^\{4\}\\right\]\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\_\{2\}\)\};\\xi\_\{i\}^\{\(e\_\{2\}\)\}\)\\right\\\|\_\{F\}^\{4\}\\right\]\}≤\(f\)M𝐀2​M𝐁2E2​∑e1,e2κ4⋅κ4\\displaystyle\\stackrel\{\{\\scriptstyle\(f\)\}\}\{\{\\leq\}\}\\frac\{M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\}\{E^\{2\}\}\\sum\_\{e\_\{1\},e\_\{2\}\}\\sqrt\{\\kappa^\{4\}\\cdot\\kappa^\{4\}\}≤M𝐀2​M𝐁2​κ4,\\displaystyle\\leq M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\},where in\(a\)\(a\)holds due to Jensen’s inequality, in\(b\)\(b\)we employ‖X​Y‖F2≤‖X‖22⋅‖Y‖F2≤‖X‖F2⋅‖Y‖F2\\\|XY\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{2\}^\{2\}\\cdot\\\|Y\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{F\}^\{2\}\\cdot\\\|Y\\\|\_\{F\}^\{2\},\(c\)\(c\)holds due to[˜3](https://arxiv.org/html/2608.10144#Thmassumption3),\(d\)\(d\)holds because∑xl2​yl2≤∑xl2​∑yl2≤\(∑xl\)2​\(∑yl\)2\\sum x\_\{l\}^\{2\}y\_\{l\}^\{2\}\\leq\\sum x\_\{l\}^\{2\}\\sum y\_\{l\}^\{2\}\\leq\(\\sum x\_\{l\}\)^\{2\}\(\\sum y\_\{l\}\)^\{2\}forxl≥0x\_\{l\}\\geq 0,\(e\)\(e\)holds due to Cauchy\-Schwarz inequality, and\(f\)\(f\)uses[˜2](https://arxiv.org/html/2608.10144#Thmassumption2)\. ∎

###### Lemma 6\.

For each clienti∈\[N\]i\\in\[N\]and roundt∈\{0,⋯,T−1\}t\\in\\\{0,\\cdots,T\-1\\\}inSeFoRA\-Ho, under[˜2](https://arxiv.org/html/2608.10144#Thmassumption2)and[˜3](https://arxiv.org/html/2608.10144#Thmassumption3),𝔼t​‖\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)‖F2≤M𝐁2​χ2\\mathbb\{E\}\_\{t\}\\left\\\|\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\leq M\_\{\\mathbf\{B\}\}^\{2\}\\chi^\{2\}and𝔼t​‖\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)‖F2≤M𝐀2​χ2\.\\mathbb\{E\}\_\{t\}\\left\\\|\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\leq M\_\{\\mathbf\{A\}\}^\{2\}\\chi^\{2\}\.

###### Proof\.

We prove𝔼t​‖\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)‖F2≤M𝐁2​χ2\\mathbb\{E\}\_\{t\}\\\|\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\leq M\_\{\\mathbf\{B\}\}^\{2\}\\chi^\{2\}in the rank\-homogeneous case\. The bound on\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}is symmetric\.

𝔼t​‖g¯𝙰,i\(t\)‖F2\\displaystyle\\mathbb\{E\}\_\{t\}\\left\\\|\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}=1E2​𝔼t​‖∑e=0E−1g^𝙰,i\(t,e\)‖F2\\displaystyle=\\frac\{1\}\{E^\{2\}\}\\mathbb\{E\}\_\{t\}\\left\\\|\\sum\_\{e=0\}^\{E\-1\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\\right\\\|\_\{F\}^\{2\}≤\(a\)1E2​E​∑e=0E−1𝔼t​‖g^𝙰,i\(t,e\)‖F2\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{\\leq\}\}\\frac\{1\}\{E^\{2\}\}E\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\\right\\\|\_\{F\}^\{2\}=1E​∑e=0E−1𝔼t​‖\(𝙱i\(t,e\)\)⊤⋅∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\);ξi\(e\)\)‖F2\\displaystyle=\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\mathbb\{E\}\_\{t\}\\left\\\|\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\)^\{\\top\}\\cdot\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)\\right\\\|\_\{F\}^\{2\}≤\(b\)1E​∑e=0E−1‖𝙱i\(t,e\)‖22​𝔼t​‖∇Δ​𝚆ℒifull​\(Δ​𝚆i\(t,e\);ξi\(e\)\)‖F2\\displaystyle\\stackrel\{\{\\scriptstyle\(b\)\}\}\{\{\\leq\}\}\\frac\{1\}\{E\}\\sum\_\{e=0\}^\{E\-1\}\\\|\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\\\|\_\{2\}^\{2\}\\mathbb\{E\}\_\{t\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{i\}^\{\(t,e\)\};\\xi\_\{i\}^\{\(e\)\}\)\\right\\\|\_\{F\}^\{2\}≤M𝐁2​χ2,\\displaystyle\\leq M\_\{\\mathbf\{B\}\}^\{2\}\\chi^\{2\},where\(a\)\(a\)is Jensen’s inequality and\(b\)\(b\)uses‖X​Y‖F2≤‖X‖22​‖Y‖F2\\\|XY\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{2\}^\{2\}\\\|Y\\\|\_\{F\}^\{2\}\. The bound on\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}is identical with the roles of𝙰\\mathbf\{\\mathtt\{A\}\}and𝙱\\mathbf\{\\mathtt\{B\}\}swapped\. ∎

### B\.1Proof of[˜1](https://arxiv.org/html/2608.10144#Thmthm1)

###### Proof\.

Since the full objective isμ\\mu\-smooth with respect toΔ​𝚆\\Delta\\mathbf\{\\mathtt\{W\}\}, we have

𝔼t​\[ℒfull​\(Δ​𝚆SeFoRA\-Ho\(t\+1\)\)\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\)\\right\]\(1\)≤\\displaystyle\\leq𝔼t​\[ℒfull​\(Δ​𝚆\(t\)\)\]\+𝔼t​\[⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)⟩\]⏟Term 1\+μ2​𝔼t​\[‖Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)‖F2\]⏟Term 2,\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\]\+\\underbrace\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\rangle\\right\]\}\_\{\\text\{Term 1\}\}\+\\underbrace\{\\frac\{\\mu\}\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\]\}\_\{\\text\{Term 2\}\},Suppose the global LoRA factors at roundttare𝙰\(t\)\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}and𝙱\(t\)\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}, the global model obtained bySeFoRA\-Hoat roundt\+1t\+1is a low\-rank approximation of the true aggregation result∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\. The local fine\-tuning starts from\(𝙱i\(t,0\),𝙰i\(t,0\)\)=\(𝙱\(t\),𝙰\(t\)\)\(\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,0\)\},\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,0\)\}\)=\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)and updates

𝙱i\(t,e\+1\)\\displaystyle\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\+1\)\}=𝙱i\(t,e\)−η\(t\)​g^𝙱,i\(t,e\),\\displaystyle=\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,e\)\}\-\\eta^\{\(t\)\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t,e\)\},𝙰i\(t,e\+1\)\\displaystyle\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\+1\)\}=𝙰i\(t,e\)−η\(t\)​g^𝙰,i\(t,e\)\.\\displaystyle=\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,e\)\}\-\\eta^\{\(t\)\}\\hat\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t,e\)\}\.The global model at roundt\+1t\+1can be expressed as

Δ​𝚆SeFoRA\-Ho\(t\+1\)\\displaystyle\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}=𝙱SeFoRA\-Ho\(t\+1\)​𝙰SeFoRA\-Ho\(t\+1\)\\displaystyle=\\mathbf\{\\mathtt\{B\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}=∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\+ℰsketch\(t\+1\)\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\+\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}=∑i∈\[N\]pi​𝙱i\(t,E\)​𝙰i\(t,E\)\+ℰsketch\(t\+1\)\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t,E\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t,E\)\}\+\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}=∑i∈\[N\]pi​\(𝙱\(t\)−η\(t\)​E​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\)​\(𝙰\(t\)−η\(t\)​E​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\)\+ℰsketch\(t\+1\)\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\left\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\right\)\\left\(\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\)\+\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}=𝙱\(t\)​𝙰\(t\)−η\(t\)​E​∑i∈\[N\]pi​𝙱\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)−η\(t\)​E​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​𝙰\(t\)\+\(η\(t\)\)2​E2​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\+ℰsketch\(t\+1\),\\displaystyle=\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\+\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\},where𝔼t​‖ℰsketch\(t\+1\)‖F2≤\(1\+rk−r−1\)⋅minϱ<r−1⁡\(1\+ϱr−ϱ−1\)⋅𝔼t​\[τϱ\+12​\(∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\)\]\\mathbb\{E\}\_\{t\}\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\leq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\\min\_\{\\varrho<r\-1\}\(1\+\\frac\{\\varrho\}\{r\-\\varrho\-1\}\)\\cdot\\mathbb\{E\}\_\{t\}\\left\[\\tau\_\{\\varrho\+1\}^\{2\}\(\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\)\\right\]\([˜2](https://arxiv.org/html/2608.10144#Thmlem2)\)\. Thus, the model update

Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)=−η\(t\)​E​∑i∈\[N\]pi​𝙱\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)−η\(t\)​E​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​𝙰\(t\)\+\(η\(t\)\)2​E2​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\+ℰsketch\(t\+1\)\.\\displaystyle\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}=\-\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\+\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\.\(2\)

##### For Term 1:

From[Eq\.˜2](https://arxiv.org/html/2608.10144#A2.E2), we can write

𝔼t​\[Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\]=\\displaystyle=−𝔼t​\[η\(t\)​E​∑i∈\[N\]pi​𝙱\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]−𝔼t​\[η\(t\)​E​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​𝙰\(t\)\]\+𝔼t​\[\(η\(t\)\)2​E2​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]\+𝔼t​\[ℰsketch\(t\+1\)\]\\displaystyle\-\\mathbb\{E\}\_\{t\}\\left\[\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\-\\mathbb\{E\}\_\{t\}\\left\[\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\]\+\\mathbb\{E\}\_\{t\}\\left\[\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\+\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]=\\displaystyle=−η\(t\)​E⋅𝙱\(t\)⋅∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]−η\(t\)​E⋅∑i∈\[N\]pi​𝔼t​\[\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)\]⋅𝙰\(t\)\+\(η\(t\)\)2​E2​𝔼t​\[∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]\+𝔼t​\[ℰsketch\(t\+1\)\]\\displaystyle\-\\eta^\{\(t\)\}E\\cdot\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\cdot\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\-\\eta^\{\(t\)\}E\\cdot\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\right\]\\cdot\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\+\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]=\\displaystyle=−η\(t\)​E⋅𝙱\(t\)⋅\(D𝙰\(t\)\+g𝙰\(t\)\)−η\(t\)​E⋅\(D𝙱\(t\)\+g𝙱\(t\)\)⋅𝙰\(t\)\+\(η\(t\)\)2​E2​𝔼t​\[∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]\+𝔼t​\[ℰsketch\(t\+1\)\]\\displaystyle\-\\eta^\{\(t\)\}E\\cdot\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\cdot\(D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\+g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\)\-\\eta^\{\(t\)\}E\\cdot\(D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\+g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\)\\cdot\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\+\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]=\\displaystyle=−η\(t\)​E​𝙱\(t\)​g𝙰\(t\)−η\(t\)​E​g𝙱\(t\)​𝙰\(t\)−η\(t\)​E​𝙱\(t\)​D𝙰\(t\)−η\(t\)​E​D𝙱\(t\)​𝙰\(t\)\+\(η\(t\)\)2​E2​𝔼t​\[∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\]\+𝔼t​\[ℰsketch\(t\+1\)\]\.\\displaystyle\-\\eta^\{\(t\)\}E\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\-\\eta^\{\(t\)\}Eg\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\-\\eta^\{\(t\)\}ED\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\+\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]\.Thus,

𝔼t​\[⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)⟩\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\rangle\\right\]=\\displaystyle=⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝔼t​\[Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)\]⟩\\displaystyle\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbb\{E\}\_\{t\}\\left\[\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\]\\right\\rangle=\\displaystyle=−η\(t\)​E⋅⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝙱\(t\)​g𝙰\(t\)⟩−η\(t\)​E⋅⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),g𝙱\(t\)​𝙰\(t\)⟩\\displaystyle\-\\eta^\{\(t\)\}E\\cdot\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\rangle\-\\eta^\{\(t\)\}E\\cdot\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\rangle−η\(t\)​E⋅⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝙱\(t\)​D𝙰\(t\)⟩−η\(t\)​E⋅⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),D𝙱\(t\)​𝙰\(t\)⟩\\displaystyle\-\\eta^\{\(t\)\}E\\cdot\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\rangle\-\\eta^\{\(t\)\}E\\cdot\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\rangle\+\(η\(t\)\)2​E2​⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝔼t​\[∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)\]⟩\+⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝔼t​\[ℰsketch\(t\+1\)\]⟩\\displaystyle\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\\right\\rangle\+\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]\\right\\rangle=\(a\)\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{=\}\}−η\(t\)​E⋅\(‖g𝙰\(t\)‖F2\+‖g𝙱\(t\)‖F2\)\+η\(t\)​E⋅⟨g𝙰\(t\),−D𝙰\(t\)⟩\+η\(t\)​E⋅⟨g𝙱\(t\),−D𝙱\(t\)⟩\\displaystyle\-\\eta^\{\(t\)\}E\\cdot\\left\(\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\)\+\\eta^\{\(t\)\}E\\cdot\\left\\langle g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\},\-D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\rangle\+\\eta^\{\(t\)\}E\\cdot\\left\\langle g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\},\-D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\right\\rangle\+\(η\(t\)\)2​E2​⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝔼t​\[∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)\]⟩\+⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝔼t​\[ℰsketch\(t\+1\)\]⟩\\displaystyle\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\\right\\rangle\+\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]\\right\\rangle≤\(b\)\\displaystyle\\stackrel\{\{\\scriptstyle\(b\)\}\}\{\{\\leq\}\}−η\(t\)​E⋅\(‖g𝙰\(t\)‖F2\+‖g𝙱\(t\)‖F2\)\+η\(t\)​E⋅\(12​‖g𝙰\(t\)‖F2\+12​‖D𝙰\(t\)‖F2\)\+η\(t\)​E⋅\(12​‖g𝙱\(t\)‖F2\+12​‖D𝙱\(t\)‖F2\)\\displaystyle\-\\eta^\{\(t\)\}E\\cdot\\left\(\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\)\+\\eta^\{\(t\)\}E\\cdot\(\\frac\{1\}\{2\}\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\frac\{1\}\{2\}\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\)\+\\eta^\{\(t\)\}E\\cdot\(\\frac\{1\}\{2\}\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\frac\{1\}\{2\}\\\|D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\)\+\(η\(t\)\)2​E2⋅‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F⋅‖𝔼t​\[∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)\]‖F\+‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F⋅‖𝔼t​\[ℰsketch\(t\+1\)\]‖F\\displaystyle\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\cdot\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\\cdot\\left\\\|\\mathbb\{E\}\_\{t\}\\left\[\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\]\\right\\\|\_\{F\}\+\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\\cdot\\left\\\|\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\]\\right\\\|\_\{F\}≤\(c\)\\displaystyle\\stackrel\{\{\\scriptstyle\(c\)\}\}\{\{\\leq\}\}−12​η\(t\)​E⋅\(‖g𝙰\(t\)‖F2\+‖g𝙱\(t\)‖F2\)\+η\(t\)​E2​‖D𝙰\(t\)‖F2\+η\(t\)​E2​‖D𝙱\(t\)‖F2\\displaystyle\-\\frac\{1\}\{2\}\\eta^\{\(t\)\}E\\cdot\\left\(\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\)\+\\frac\{\\eta^\{\(t\)\}E\}\{2\}\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\frac\{\\eta^\{\(t\)\}E\}\{2\}\\\|D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\(η\(t\)\)2​E2⋅‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F⋅𝔼t​\[‖∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)‖F2\]\+‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\],\\displaystyle\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\cdot\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\},where\(a\)\(a\)holds since the definition of the tuple inner product and the cyclic property of the trace, i\.e\.,

⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝙱\(t\)​g𝙰\(t\)⟩\\displaystyle\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\rangle=∑i=1Ltr​\(\(g𝐀l\(t\)\)⊤​\(𝐁l\(t\)\)⊤​∇Δ​𝐖ℒfull​\(Δ​𝐖l\(t\)\)\)=∑i=1Ltr​\(\(g𝐀l\(t\)\)⊤​g𝐀l\(t\)\)=‖g𝙰\(t\)‖F2,\\displaystyle=\\sum\_\{i=1\}^\{L\}\\mathrm\{tr\}\\left\(\{\(g\_\{\\mathbf\{A\}\_\{l\}\}^\{\(t\)\}\)\}^\{\\top\}\{\(\\mathbf\{B\}\_\{l\}^\{\(t\)\}\)\}^\{\\top\}\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{l\}^\{\(t\)\}\)\\right\)=\\sum\_\{i=1\}^\{L\}\\mathrm\{tr\}\\left\(\{\(g\_\{\\mathbf\{A\}\_\{l\}\}^\{\(t\)\}\)\}^\{\\top\}g\_\{\\mathbf\{A\}\_\{l\}\}^\{\(t\)\}\\right\)=\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\},⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),g𝙱\(t\)​𝙰\(t\)⟩\\displaystyle\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\rangle=∑i=1Ltr​\(\(g𝐁l\(t\)\)⊤​∇Δ​𝐖ℒfull​\(Δ​𝐖l\(t\)\)​\(𝐀l\(t\)\)⊤\)=∑i=1Ltr​\(\(g𝐁l\(t\)\)⊤​g𝐁l\(t\)\)=‖g𝙱\(t\)‖F2,\\displaystyle=\\sum\_\{i=1\}^\{L\}\\mathrm\{tr\}\\left\(\{\(g\_\{\\mathbf\{B\}\_\{l\}\}^\{\(t\)\}\)\}^\{\\top\}\\nabla\_\{\\Delta\\mathbf\{W\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{W\}\_\{l\}^\{\(t\)\}\)\{\(\\mathbf\{A\}\_\{l\}^\{\(t\)\}\)\}^\{\\top\}\\right\)=\\sum\_\{i=1\}^\{L\}\\mathrm\{tr\}\\left\(\{\(g\_\{\\mathbf\{B\}\_\{l\}\}^\{\(t\)\}\)\}^\{\\top\}g\_\{\\mathbf\{B\}\_\{l\}\}^\{\(t\)\}\\right\)=\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\},and similarly,⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),𝙱\(t\)​D𝙰\(t\)⟩=⟨g𝙰\(t\),D𝙰\(t\)⟩\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\rangle=\\left\\langle g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\},D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\right\\rangle,⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),D𝙱\(t\)​𝙰\(t\)⟩=⟨g𝙱\(t\),D𝙱\(t\)⟩\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\rangle=\\left\\langle g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\},D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\right\\rangle,\(b\)\(b\)holds since Young’s inequality and⟨X,Y⟩≤\|⟨X,Y⟩\|≤‖X‖F​‖Y‖F\\langle X,Y\\rangle\\leq\|\\langle X,Y\\rangle\|\\leq\\\|X\\\|\_\{F\}\\\|Y\\\|\_\{F\}, and\(c\)\(c\)holds since Jensen’s inequality\. Since‖D𝙰\(t\)‖F\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}and‖D𝙱\(t\)‖F\\\|D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}are bounded by[˜4](https://arxiv.org/html/2608.10144#Thmlem4), and according to[˜5](https://arxiv.org/html/2608.10144#Thmlem5), we know

𝔼t​\[‖∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)‖F2\]≤∑i∈\[N\]pi​𝔼t​\[‖g¯𝙱,i\(t\)⋅g¯𝙰,i\(t\)‖F2\]≤M𝐀2​M𝐁2​κ4,\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\\leq\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\cdot\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\\leq M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\},we only need to derive the bounds for‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\. Due to Jensen’s inequality and the inequality that‖𝔼​\[X\]‖2≤𝔼​\[‖X‖2\]\\\|\\mathbb\{E\}\[X\]\\\|^\{2\}\\leq\\mathbb\{E\}\[\\\|X\\\|^\{2\}\], we have

‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F2\\displaystyle\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}=‖∑i∈\[N\]pi​∇Δ​𝚆ℒifull​\(Δ​𝚆\(t\)\)‖F2\\displaystyle=\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}≤∑i∈\[N\]pi​‖∇Δ​𝚆ℒifull​\(Δ​𝚆\(t\)\)‖F2\\displaystyle\\leq\\sum\_\{i\\in\[N\]\}p\_\{i\}\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}=∑i∈\[N\]pi​‖𝔼t​\[∇Δ​𝚆ℒifull​\(Δ​𝚆\(t\);ξi\)\]‖F2\\displaystyle=\\sum\_\{i\\in\[N\]\}p\_\{i\}\\left\\\|\\mathbb\{E\}\_\{t\}\\left\[\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\};\\xi\_\{i\}\)\\right\]\\right\\\|\_\{F\}^\{2\}≤∑i∈\[N\]pi​𝔼t​\[‖∇Δ​𝚆ℒifull​\(Δ​𝚆\(t\);ξi\)‖F2\]\\displaystyle\\leq\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}\_\{i\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\};\\xi\_\{i\}\)\\right\\\|\_\{F\}^\{2\}\\right\]≤χ2\.\\displaystyle\\leq\\chi^\{2\}\.\(3\)
Hence, by combining the previous bounds and utilizing[˜5](https://arxiv.org/html/2608.10144#Thmlem5), Term 1 is bounded as

𝔼t​\[⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)⟩\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\rangle\\right\]≤−12​η\(t\)​E⋅\(‖g𝙰\(t\)‖F2\+‖g𝙱\(t\)‖F2\)\+η\(t\)​E2​‖D𝙰\(t\)‖F2\+η\(t\)​E2​‖D𝙱\(t\)‖F2\\displaystyle\\leq\-\\frac\{1\}\{2\}\\eta^\{\(t\)\}E\\cdot\\left\(\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\)\+\\frac\{\\eta^\{\(t\)\}E\}\{2\}\\\|D\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\frac\{\\eta^\{\(t\)\}E\}\{2\}\\\|D\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\(η\(t\)\)2​E2⋅‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F⋅𝔼t​\[‖∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)‖F2\]\+‖∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\)‖F⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\\displaystyle\\quad\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\cdot\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+\\left\\\|\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}≤−12​η\(t\)​E⋅\(‖g𝙰\(t\)‖F2\+‖g𝙱\(t\)‖F2\)\+\(η\(t\)\)3​\(E−1\)​E​\(2​E−1\)6​χ2⋅\(χ2​\(M𝐀2\+M𝐁2\)\+2​μ2​\(1\+M𝐀2\)​\(M𝐁4\+M𝐀4\)\)\\displaystyle\\leq\-\\frac\{1\}\{2\}\\eta^\{\(t\)\}E\\cdot\\left\(\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\)\+\(\\eta^\{\(t\)\}\)^\{3\}\\frac\{\(E\-1\)E\(2E\-1\)\}\{6\}\\chi^\{2\}\\cdot\\left\(\\chi^\{2\}\(M\_\{\\mathbf\{A\}\}^\{2\}\+M\_\{\\mathbf\{B\}\}^\{2\}\)\+2\\mu^\{2\}\(1\+M\_\{\\mathbf\{A\}\}^\{2\}\)\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\+\(η\(t\)\)2​E2​χ⋅M𝐀​M𝐁​κ2\+χ⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\.\\displaystyle\\quad\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\chi\\cdot M\_\{\\mathbf\{A\}\}M\_\{\\mathbf\{B\}\}\\kappa^\{2\}\+\\chi\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\.

##### For Term 2:

Using[Eq\.˜2](https://arxiv.org/html/2608.10144#A2.E2), we have

𝔼t​\[‖Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)‖F2\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\]=\\displaystyle=𝔼t​\[‖−η\(t\)​E​∑i∈\[N\]pi​𝙱\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)−η\(t\)​E​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​𝙰\(t\)\+\(η\(t\)\)2​E2​∑i∈\[N\]pi​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙱,i\(t\)​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​g𝙰,i\(t\)\+ℰsketch\(t\+1\)‖F2\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\-\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\-\\eta^\{\(t\)\}E\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\+\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]≤\(a\)\\displaystyle\\stackrel\{\{\\scriptstyle\(a\)\}\}\{\{\\leq\}\}4​\(η\(t\)\)2​E2​𝔼t​\[‖∑i∈\[N\]pi​𝙱\(t\)​g¯𝙰,i\(t\)‖F2\]\+4​\(η\(t\)\)2​E2​𝔼t​\[‖∑i∈\[N\]pi​g¯𝙱,i\(t\)​𝙰\(t\)‖F2\]\+4​\(η\(t\)\)4​E4​𝔼t​\[‖∑i∈\[N\]pi​g¯𝙱,i\(t\)​g¯𝙰,i\(t\)‖F2\]\+4​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\\displaystyle 4\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\+4\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\+4\(\\eta^\{\(t\)\}\)^\{4\}E^\{4\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\sum\_\{i\\in\[N\]\}p\_\{i\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\+4\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]≤\(b\)\\displaystyle\\stackrel\{\{\\scriptstyle\(b\)\}\}\{\{\\leq\}\}4​\(η\(t\)\)2​E2​‖𝙱\(t\)‖22⋅∑i∈\[N\]pi​𝔼t​‖g¯𝙰,i\(t\)‖F2\+4​\(η\(t\)\)2​E2​∑i∈\[N\]pi​𝔼t​‖g¯𝙱,i\(t\)‖F2⋅‖𝙰\(t\)‖22\\displaystyle 4\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\\|\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\}\\\|\_\{2\}^\{2\}\\cdot\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\\\|\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\+4\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\\\|\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\cdot\\\|\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\\\|\_\{2\}^\{2\}\+4​\(η\(t\)\)4​E4​∑i∈\[N\]pi​𝔼t​\[‖g¯𝙱,i\(t\)⋅g¯𝙰,i\(t\)‖F2\]\+4​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\\displaystyle\+4\(\\eta^\{\(t\)\}\)^\{4\}E^\{4\}\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\bar\{g\}\_\{\\mathbf\{\\mathtt\{B\}\},i\}^\{\(t\)\}\\cdot\\bar\{g\}\_\{\\mathbf\{\\mathtt\{A\}\},i\}^\{\(t\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\+4\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]≤\(c\)\\displaystyle\\stackrel\{\{\\scriptstyle\(c\)\}\}\{\{\\leq\}\}4​\(η\(t\)\)2​E2​χ2​\(M𝐁2\+M𝐀4\)\+4​\(η\(t\)\)4​E4​M𝐀2​M𝐁2​κ4\+4​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\],\\displaystyle 4\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{2\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\+4\(\\eta^\{\(t\)\}\)^\{4\}E^\{4\}M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\}\+4\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\],where\(a\)\(a\)holds by grouping the first two terms and using‖W\+X\+Y\+Z‖F2≤4​\(‖W‖F2\+‖X‖F2\+‖Y‖F2\+‖Z‖F2\)\\\|W\+X\+Y\+Z\\\|\_\{F\}^\{2\}\\leq 4\(\\\|W\\\|\_\{F\}^\{2\}\+\\\|X\\\|\_\{F\}^\{2\}\+\\\|Y\\\|\_\{F\}^\{2\}\+\\\|Z\\\|\_\{F\}^\{2\}\),\(b\)\(b\)holds due to‖X​Y‖F2≤‖X‖22⋅‖Y‖F2\\\|XY\\\|\_\{F\}^\{2\}\\leq\\\|X\\\|\_\{2\}^\{2\}\\cdot\\\|Y\\\|\_\{F\}^\{2\}and Jensen’s inequality, and\(c\)\(c\)holds from[˜2](https://arxiv.org/html/2608.10144#Thmassumption2),[˜3](https://arxiv.org/html/2608.10144#Thmassumption3),[˜5](https://arxiv.org/html/2608.10144#Thmlem5), and[˜6](https://arxiv.org/html/2608.10144#Thmlem6)\.

By combining the bounds on Term 1 and Term 2, we have

𝔼t​\[ℒfull​\(Δ​𝚆SeFoRA\-Ho\(t\+1\)\)\]−𝔼t​\[ℒfull​\(Δ​𝚆\(t\)\)\]\\displaystyle\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\)\\right\]\-\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\]≤𝔼t​\[⟨∇Δ​𝚆ℒfull​\(Δ​𝚆\(t\)\),Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)⟩\]\+μ2​𝔼t​\[‖Δ​𝚆SeFoRA\-Ho\(t\+1\)−Δ​𝚆\(t\)‖F2\]\\displaystyle\\leq\\mathbb\{E\}\_\{t\}\\left\[\\left\\langle\\nabla\_\{\\Delta\\mathbf\{\\mathtt\{W\}\}\}\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\),\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\right\\rangle\\right\]\+\\frac\{\\mu\}\{2\}\\mathbb\{E\}\_\{t\}\\left\[\\\|\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\-\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\]≤−12​η\(t\)​E⋅\(‖g𝙰\(t\)‖F2\+‖g𝙱\(t\)‖F2\)\+\(η\(t\)\)2​E2⋅\(χ​M𝐀​M𝐁​κ2\+2​μ​χ2​\(M𝐁2\+M𝐀4\)\)\\displaystyle\\leq\-\\frac\{1\}\{2\}\\eta^\{\(t\)\}E\\cdot\\left\(\\\|g\_\{\\mathbf\{\\mathtt\{A\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\+\\\|g\_\{\\mathbf\{\\mathtt\{B\}\}\}^\{\(t\)\}\\\|\_\{F\}^\{2\}\\right\)\+\(\\eta^\{\(t\)\}\)^\{2\}E^\{2\}\\cdot\\left\(\\chi M\_\{\\mathbf\{A\}\}M\_\{\\mathbf\{B\}\}\\kappa^\{2\}\+2\\mu\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{2\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\+\(η\(t\)\)3​\(E−1\)​E​\(2​E−1\)6⋅χ2​\(χ2​\(M𝐀2\+M𝐁2\)\+2​μ2​\(1\+M𝐀2\)​\(M𝐁4\+M𝐀4\)\)\\displaystyle\\quad\+\(\\eta^\{\(t\)\}\)^\{3\}\\frac\{\(E\-1\)E\(2E\-1\)\}\{6\}\\cdot\\chi^\{2\}\\left\(\\chi^\{2\}\(M\_\{\\mathbf\{A\}\}^\{2\}\+M\_\{\\mathbf\{B\}\}^\{2\}\)\+2\\mu^\{2\}\(1\+M\_\{\\mathbf\{A\}\}^\{2\}\)\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\+\(η\(t\)\)4​E4⋅2​μ​M𝐀2​M𝐁2​κ4\+χ⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\.\\displaystyle\\quad\+\(\\eta^\{\(t\)\}\)^\{4\}E^\{4\}\\cdot 2\\mu M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\}\+\\chi\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]\.
Therefore,

‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\+‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2≤2​\(𝔼t​\[ℒfull​\(Δ​𝚆\(t\)\)\]−𝔼t​\[ℒfull​\(Δ​𝚆SeFoRA\-Ho\(t\+1\)\)\]\)η\(t\)​E\\displaystyle\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\+\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\leq\\frac\{2\\left\(\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(t\)\}\)\\right\]\-\\mathbb\{E\}\_\{t\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(t\+1\)\}\)\\right\]\\right\)\}\{\\eta^\{\(t\)\}E\}\+2​η\(t\)​E⋅\(χ​M𝐀​M𝐁​κ2\+2​μ​χ2​\(M𝐁2\+M𝐀4\)\)\+2​\(η\(t\)\)2​\(E−1\)​\(2​E−1\)6⋅χ2​\(χ2​\(M𝐀2\+M𝐁2\)\+2​μ2​\(1\+M𝐀2\)​\(M𝐁4\+M𝐀4\)\)\\displaystyle\\quad\\\!\+\\\!2\\eta^\{\(t\)\}E\\\!\\cdot\\\!\\left\(\\chi M\_\{\\mathbf\{A\}\}M\_\{\\mathbf\{B\}\}\\kappa^\{2\}\+2\\mu\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{2\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\+2\(\\eta^\{\(t\)\}\)^\{2\}\\frac\{\(E\-1\)\(2E\-1\)\}\{6\}\\\!\\cdot\\\!\\chi^\{2\}\\\!\\left\(\\chi^\{2\}\(M\_\{\\mathbf\{A\}\}^\{2\}\\\!\+\\\!M\_\{\\mathbf\{B\}\}^\{2\}\)\\\!\+\\\!2\\mu^\{2\}\(1\\\!\+\\\!M\_\{\\mathbf\{A\}\}^\{2\}\)\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\+4​\(η\(t\)\)3​E3⋅μ​M𝐀2​M𝐁2​κ4\+2η\(t\)​E​\(χ⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\)\.\\displaystyle\\quad\+4\(\\eta^\{\(t\)\}\)^\{3\}E^\{3\}\\cdot\\mu M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\}\+\\frac\{2\}\{\\eta^\{\(t\)\}E\}\\left\(\\chi\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]\\right\)\.
By taking constant learning rateη\(t\)=η\\eta^\{\(t\)\}=\\eta, taking total expectation𝔼​\[⋅\]\\mathbb\{E\}\[\\cdot\]on both sides, and summing this inequality fromt=0t=0toT−1T\-1and computing the average, we have

1T​∑t=0T−1\(𝔼​\[‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\+𝔼​\[‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\)≤𝔼​\[ℒfull​\(Δ​𝚆\(0\)\)\]−𝔼​\[ℒfull​\(Δ​𝚆SeFoRA\-Ho\(T\)\)\]12​T​η​E\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\+\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\\right\)\\leq\\frac\{\\mathbb\{E\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(0\)\}\)\\right\]\-\\mathbb\{E\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(T\)\}\)\\right\]\}\{\\frac\{1\}\{2\}T\\eta E\}\+2​η​E⋅\(χ​M𝐀​M𝐁​κ2\+2​μ​χ2​\(M𝐁2\+M𝐀4\)\)\+23​η2​E2⋅χ2​\(χ2​\(M𝐀2\+M𝐁2\)\+2​μ2​\(1\+M𝐀2\)​\(M𝐁4\+M𝐀4\)\)\\displaystyle\\quad\+2\\eta E\\\!\\cdot\\\!\\left\(\\chi M\_\{\\mathbf\{A\}\}M\_\{\\mathbf\{B\}\}\\kappa^\{2\}\+2\\mu\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{2\}\+M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\\\!\+\\\!\\frac\{2\}\{3\}\\eta^\{2\}E^\{2\}\\\!\\cdot\\\!\\chi^\{2\}\\\!\\left\(\\chi^\{2\}\(M\_\{\\mathbf\{A\}\}^\{2\}\\\!\+\\\!M\_\{\\mathbf\{B\}\}^\{2\}\)\\\!\+\\\!2\\mu^\{2\}\(1\+M\_\{\\mathbf\{A\}\}^\{2\}\)\(M\_\{\\mathbf\{B\}\}^\{4\}\\\!\+\\\!M\_\{\\mathbf\{A\}\}^\{4\}\)\\right\)\+4​η3​E3⋅μ​M𝐀2​M𝐁2​κ4\+1T​∑t=0T−12η​E​𝔼​\[χ⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\]\.\\displaystyle\\quad\+4\\eta^\{3\}E^\{3\}\\cdot\\mu M\_\{\\mathbf\{A\}\}^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\\kappa^\{4\}\+\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{2\}\{\\eta E\}\\mathbb\{E\}\\left\[\\chi\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]\\right\]\.
Next, we rearrange the terms in the above bound\. DefineC1=χ2​M𝐀2\+2​μ2​\(M𝐁4\+M𝐀4\)C\_\{1\}=\\chi^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\+2\\mu^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\),C2=χ2​M𝐁2\+2​μ2​M𝐀2​\(M𝐁4\+M𝐀4\)C\_\{2\}=\\chi^\{2\}M\_\{\\mathbf\{B\}\}^\{2\}\+2\\mu^\{2\}M\_\{\\mathbf\{A\}\}^\{2\}\(M\_\{\\mathbf\{B\}\}^\{4\}\+M\_\{\\mathbf\{A\}\}^\{4\}\),C3=M𝐀​M𝐁​κ2C\_\{3\}=M\_\{\\mathbf\{A\}\}M\_\{\\mathbf\{B\}\}\\kappa^\{2\},C4=μ​χ2​\(M𝐁2\+M𝐀4\)C\_\{4\}=\\mu\\chi^\{2\}\(M\_\{\\mathbf\{B\}\}^\{2\}\+M\_\{\\mathbf\{A\}\}^\{4\}\), the sketching floorS1\(t\+1\)=χ⋅𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ​𝔼t​\[‖ℰsketch\(t\+1\)‖F2\]S\_\{1\}^\{\(t\+1\)\}=\\chi\\cdot\\sqrt\{\\mathbb\{E\}\_\{t\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\mathbb\{E\}\_\{t\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\],\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​S=1T​∑t=0T−1𝔼​\[S1\(t\+1\)\]\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{S\}=\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[S\_\{1\}^\{\(t\+1\)\}\\right\], andΔ0=𝔼​\[ℒfull​\(Δ​𝚆\(0\)\)\]−ℒfull,⋆\\Delta\_\{0\}=\\mathbb\{E\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(0\)\}\)\\right\]\-\\mathcal\{L\}^\{\\text\{full\},\\star\}\. We have

1T​∑t=0T−1\(𝔼​\[‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\+𝔼​\[‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\)≤𝔼​\[ℒfull​\(Δ​𝚆\(0\)\)\]−𝔼​\[ℒfull​\(Δ​𝚆SeFoRA\-Ho\(T\)\)\]12​T​η​E\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\+\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\\right\)\\leq\\frac\{\\mathbb\{E\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}^\{\(0\)\}\)\\right\]\-\\mathbb\{E\}\\left\[\\mathcal\{L\}^\{\\text\{full\}\}\(\\Delta\\mathbf\{\\mathtt\{W\}\}\_\{\\texttt\{\{SeFoRA\-Ho\}\}\}^\{\(T\)\}\)\\right\]\}\{\\frac\{1\}\{2\}T\\eta E\}\+η​E⋅2​\(χ​C3\+2​C4\)\+η2​E2⋅23​χ2​\(C1\+C2\)\+4​η3​E3⋅μ​C32\+2η​E⋅1T​∑t=0T−1𝔼​\[S1\(t\+1\)\]\\displaystyle\\quad\+\\eta E\\cdot 2\\left\(\\chi C\_\{3\}\+2C\_\{4\}\\right\)\+\\eta^\{2\}E^\{2\}\\cdot\\frac\{2\}\{3\}\\chi^\{2\}\\left\(C\_\{1\}\+C\_\{2\}\\right\)\+4\\eta^\{3\}E^\{3\}\\cdot\\mu C\_\{3\}^\{2\}\+\\frac\{2\}\{\\eta E\}\\cdot\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[S\_\{1\}^\{\(t\+1\)\}\\right\]=2​Δ0η​E​T\+2η​E​\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​111​S\+η​E⋅2​\(χ​C3\+2​C4\)\+η2​E2⋅23​χ2​\(C1\+C2\)\+4​η3​E3⋅μ​C32\\displaystyle=\\frac\{2\\Delta\_\{0\}\}\{\\eta ET\}\+\\frac\{2\}\{\\eta E\}\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{S\}\+\\eta E\\cdot 2\\left\(\\chi C\_\{3\}\+2C\_\{4\}\\right\)\+\\eta^\{2\}E^\{2\}\\cdot\\frac\{2\}\{3\}\\chi^\{2\}\\left\(C\_\{1\}\+C\_\{2\}\\right\)\+4\\eta^\{3\}E^\{3\}\\cdot\\mu C\_\{3\}^\{2\}This concludes the proof of[˜1](https://arxiv.org/html/2608.10144#Thmthm1)\. ∎

### B\.2Proof of[Corollary˜1](https://arxiv.org/html/2608.10144#Thmcorollary1)

###### Proof\.

According to[˜2](https://arxiv.org/html/2608.10144#Thmlem2), the error between the aggregated client update∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}and its low\-rank approximation can be bounded as:

𝔼t​‖ℰsketch\(t\+1\)‖F2\\displaystyle\\mathbb\{E\}\_\{t\}\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}≤\(1\+rk−r−1\)⋅minϱ′<r−1⁡\(1\+ϱ′r−ϱ′−1\)⋅𝔼​\[τϱ′\+12​\(∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\)\]\\displaystyle\\leq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\\min\_\{\\varrho^\{\\prime\}<r\-1\}\(1\+\\frac\{\\varrho^\{\\prime\}\}\{r\-\\varrho^\{\\prime\}\-1\}\)\\cdot\\mathbb\{E\}\\left\[\\tau\_\{\\varrho^\{\\prime\}\+1\}^\{2\}\(\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\)\\right\]≤\(1\+rk−r−1\)⋅\(1\+ϱr−ϱ−1\)⋅𝔼​\[τϱ\+12​\(∑i∈\[N\]pi​𝙱i\(t\+1\)​𝙰i\(t\+1\)\)\],\\displaystyle\\leq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\(1\+\\frac\{\\varrho\}\{r\-\\varrho\-1\}\)\\cdot\\mathbb\{E\}\\left\[\\tau\_\{\\varrho\+1\}^\{2\}\(\\sum\_\{i\\in\[N\]\}p\_\{i\}\\mathbf\{\\mathtt\{B\}\}\_\{i\}^\{\(t\+1\)\}\\mathbf\{\\mathtt\{A\}\}\_\{i\}^\{\(t\+1\)\}\)\\right\],where the second inequality holds by evaluating at the fixedϱ\\varrhofrom[˜4](https://arxiv.org/html/2608.10144#Thmassumption4)\.

Defineϵ2≜\(1\+rk−r−1\)⋅\(1\+ϱr−ϱ−1\)⋅\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​1112\\epsilon^\{2\}\\triangleq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\(1\+\\frac\{\\varrho\}\{r\-\\varrho\-1\}\)\\cdot\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{2\}\. Averaging over theTTcommunication rounds and invoking[˜4](https://arxiv.org/html/2608.10144#Thmassumption4), we obtain

1T​∑t=0T−1𝔼​\[‖ℰsketch\(t\+1\)‖F2\]≤\(1\+rk−r−1\)⋅\(1\+ϱr−ϱ−1\)⋅\\macc@depth​Δ​\\macc@set@skewchar​\\macc@nested@a​1112=ϵ2\.\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]\\leq\(1\+\\frac\{r\}\{k\-r\-1\}\)\\cdot\(1\+\\frac\{\\varrho\}\{r\-\\varrho\-1\}\)\\cdot\{\\macc@depth\\@ne\\macc@set@skewchar\\macc@nested@a 111\{\}\}^\{2\}=\\epsilon^\{2\}\.
Thus, the sketching floor is

1T​∑t=0T−1𝔼​\[S1\(t\+1\)\]=χ⋅1T​∑t=0T−1𝔼​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ⋅1T​∑t=0T−1𝔼​\[‖ℰsketch\(t\+1\)‖F2\]\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[S\_\{1\}^\{\(t\+1\)\}\\right\]=\\chi\\cdot\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sqrt\{\\mathbb\{E\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\cdot\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]≤χ⋅1T​∑t=0T−1𝔼​\[‖ℰsketch\(t\+1\)‖F2\]\+2​μ⋅1T​∑t=0T−1𝔼​\[‖ℰsketch\(t\+1\)‖F2\]\\displaystyle\\leq\\chi\\cdot\\sqrt\{\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\left\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\right\\\|\_\{F\}^\{2\}\\right\]\}\+2\\mu\\cdot\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\left\[\\\|\\mathcal\{E\}\_\{\\text\{sketch\}\}^\{\(t\+1\)\}\\\|\_\{F\}^\{2\}\\right\]=χ​ϵ\+2​μ​ϵ2,\\displaystyle=\\chi\\epsilon\+2\\mu\\epsilon^\{2\},where the inequality holds due to Jensen’s inequality\.

By substituting this bound into the result of[˜1](https://arxiv.org/html/2608.10144#Thmthm1), we obtain

1T​∑t=0T−1\(𝔼​\[‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\+𝔼​\[‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\)\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\+\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\\right\)≤1η​E⋅2​Δ0T\+2η​E​\(χ​ϵ\+2​μ​ϵ2\)\+η​E⋅2​\(χ​C3\+2​C4\)\+η2​E2⋅23​χ2​\(C1\+C2\)\+4​η3​E3⋅μ​C32\\displaystyle\\leq\\frac\{1\}\{\\eta E\}\\cdot\\frac\{2\\Delta\_\{0\}\}\{T\}\+\\frac\{2\}\{\\eta E\}\(\\chi\\epsilon\+2\\mu\\epsilon^\{2\}\)\+\\eta E\\cdot 2\\left\(\\chi C\_\{3\}\+2C\_\{4\}\\right\)\+\\eta^\{2\}E^\{2\}\\cdot\\frac\{2\}\{3\}\\chi^\{2\}\\left\(C\_\{1\}\+C\_\{2\}\\right\)\+4\\eta^\{3\}E^\{3\}\\cdot\\mu C\_\{3\}^\{2\}
DefineX=2​Δ0T\+2​\(χ​ϵ\+2​μ​ϵ2\)X=\\frac\{2\\Delta\_\{0\}\}\{T\}\+2\(\\chi\\epsilon\+2\\mu\\epsilon^\{2\}\)andY=2​\(χ​C3\+2​C4\)Y=2\\left\(\\chi C\_\{3\}\+2C\_\{4\}\\right\), the functionf​\(η​E\)=1η​E⋅X\+η​E⋅Yf\(\\eta E\)=\\frac\{1\}\{\\eta E\}\\cdot X\+\\eta E\\cdot Yis minimized at\(η​E\)⋆=XY\(\\eta E\)^\{\\star\}=\\sqrt\{\\frac\{X\}\{Y\}\}, andf​\(\(η​E\)⋆\)=2​X​Yf\(\(\\eta E\)^\{\\star\}\)=2\\sqrt\{XY\}\. Thus, by choosing

η=1E​XY=1E​Δ0T\+χ​ϵ\+2​μ​ϵ2χ​C3\+2​C4,\\eta=\\frac\{1\}\{E\}\\sqrt\{\\frac\{X\}\{Y\}\}=\\frac\{1\}\{E\}\\sqrt\{\\frac\{\\frac\{\\Delta\_\{0\}\}\{T\}\+\\chi\\epsilon\+2\\mu\\epsilon^\{2\}\}\{\\chi C\_\{3\}\+2C\_\{4\}\}\},we have

1T​∑t=0T−1\(𝔼​\[‖∇𝙰ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\+𝔼​\[‖∇𝙱ℒlora​\(𝙱\(t\),𝙰\(t\)\)‖F2\]\)\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{A\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\+\\mathbb\{E\}\\left\[\\left\\\|\\nabla\_\{\\mathbf\{\\mathtt\{B\}\}\}\\mathcal\{L\}^\{\\text\{lora\}\}\(\\mathbf\{\\mathtt\{B\}\}^\{\(t\)\},\\mathbf\{\\mathtt\{A\}\}^\{\(t\)\}\)\\right\\\|\_\{F\}^\{2\}\\right\]\\right\)≤2​X​Y\+XY⋅23​χ2​\(C1\+C2\)\+4​\(XY\)32⋅μ​C32\\displaystyle\\leq 2\\sqrt\{XY\}\+\\frac\{X\}\{Y\}\\cdot\\frac\{2\}\{3\}\\chi^\{2\}\\left\(C\_\{1\}\+C\_\{2\}\\right\)\+4\\left\(\\frac\{X\}\{Y\}\\right\)^\{\\frac\{3\}\{2\}\}\\cdot\\mu C\_\{3\}^\{2\}=4​\(Δ0T\+χ​ϵ\+2​μ​ϵ2\)​\(χ​C3\+2​C4\)\+Δ0T\+χ​ϵ\+2​μ​ϵ2χ​C3\+2​C4⋅2​χ2​\(C1\+C2\)3\+4​\(Δ0T\+χ​ϵ\+2​μ​ϵ2χ​C3\+2​C4\)32⋅μ​C32\.\\displaystyle=4\\sqrt\{\\\!\\left\(\\frac\{\\Delta\_\{0\}\}\{T\}\\\!\+\\\!\\chi\\epsilon\\\!\+\\\!2\\mu\\epsilon^\{2\}\\right\)\\\!\\left\(\\chi C\_\{3\}\\\!\+\\\!2C\_\{4\}\\right\)\}\+\\frac\{\\frac\{\\Delta\_\{0\}\}\{T\}\\\!\+\\\!\\chi\\epsilon\+2\\mu\\epsilon^\{2\}\}\{\\chi C\_\{3\}\+2C\_\{4\}\}\\\!\\cdot\\\!\\frac\{2\\chi^\{2\}\\\!\\left\(C\_\{1\}\\\!\+\\\!C\_\{2\}\\right\)\\\!\}\{3\}\\\!\+\\\!4\\\!\\left\(\\\!\\frac\{\\frac\{\\Delta\_\{0\}\}\{T\}\\\!\+\\\!\\chi\\epsilon\\\!\+\\\!2\\mu\\epsilon^\{2\}\}\{\\chi C\_\{3\}\+2C\_\{4\}\}\\\!\\right\)^\{\\frac\{3\}\{2\}\}\\\!\\\!\\\!\\\!\\\!\\cdot\\mu C\_\{3\}^\{2\}\.∎