Federated Compositional Muon Optimizer for Matrix-Wise Models

# Federated Compositional Muon Optimizer for Matrix-Wise Models
Source: [https://arxiv.org/html/2608.12710](https://arxiv.org/html/2608.12710)
Wang YanThanks:Wang Yan is with College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China\.Feihu HuangThanks:Feihu Huang is with College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China; and also with MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China\. Email: huangfeihu2018@gmail\.com

###### Abstract

Muon, a more recently developed optimizer, is useful for matrix\-wise models in AI areas\. Although many works have studied Muon and its variants, these methods are still not particularly well\-suited for hierarchical structured problems\. To fill this gap, we propose an effective federated compositional Muon \(FedCoMuon\) optimizer to solve distributed matrix\-wise compositional optimization problems\. Specifically, our FedCoMuon optimizer builds on compositional gradient tracking and orthogonalized momentum\. Moreover, we propose a variance reduced variant of FedCoMuon \(FedCoMuon\-VR\) based on a momentum\-based variance reduced technique\. In theory, we analyze the convergence properties of our algorithms under the non\-i\.i\.d\. and non\-convex settings\. In particular, we prove that our FedCoMuon\-VR obtains a lower sample complexity ofO⁡\(ϵ−3\)O\(\\epsilon^\{\-3\}\)for finding anϵ\\epsilon\-stationary solution than the existing FedMuon algorithms\. Extensive numerical experiments on robust federated learning and task\-distributed risk\-sensitive meta learning show that our proposed methods are competitive with existing compositional baselines and achieve the best reported accuracy in several settings\.

## 1Introduction

Federated learning \(FL\)\([McMahan et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib29);[Kairouz et al\. 2019](https://arxiv.org/html/2608.12710#bib.bib15);[Li et al\. 2021](https://arxiv.org/html/2608.12710#bib.bib25)\)enables multiple clients to collaboratively train a shared model without centralizing their local data, making it attractive for privacy\-sensitive and resource\-constrained applications such as mobile intelligence, healthcare, recommendation systems, and edge learning\. The first federated learning algorithm, i\.e\., FedAvg\([McMahan et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib29)\), established the standard paradigm of alternating local stochastic updates with periodic model aggregation, while subsequent methods such as FedProx\([Li et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib26)\), SCAFFOLD\([Karimireddy et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib16)\), and FedOpt\([Reddi et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib34)\)further improved federated training under client heterogeneity and limited communication\. Despite their success, these methods were designed for standard federated empirical risk minimization, where the global objective is typically a single\-level weighted average of client losses\.

Beyond this standard single\-level formulation, many federated applications involve more complex nested objectives\([Yang 2026](https://arxiv.org/html/2608.12710#bib.bib45)\), especially when the learned model is expected to perform reliably across heterogeneous or adverse client distributions\. In distributionally robust FL, for example, a nonlinear risk measure is applied to client\-level expected losses, resulting in a nested stochastic compositional objective\([Huang and Li 2021](https://arxiv.org/html/2608.12710#bib.bib9)\)\. Similar nested structures arise in the task\-distributed meta learning problem\([Huang and Li 2021](https://arxiv.org/html/2608.12710#bib.bib9);[Wang et al\. 2023](https://arxiv.org/html/2608.12710#bib.bib42)\)and federated reinforcement learning\([Jin et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib13);[Tarzanagh et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib39)\)\. Since the outer gradient depends on an inner stochastic expectation, directly applying local SGD generally yields biased gradient estimates\. Several federated compositional optimization methods correct this bias, including ComFedL\([Huang and Li 2021](https://arxiv.org/html/2608.12710#bib.bib9)\), Local\-MOML\([Wang et al\. 2023](https://arxiv.org/html/2608.12710#bib.bib42)\), Local\-SCGDM\([Gao et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib7)\), and FedNest\([Tarzanagh et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib39)\)\. Although these methods address the nested stochastic structure, their algorithmic formulations do not explicitly exploit the matrix structure of model parameters\.

In fact, modern neural networks such as Convolutional Neural Networks \(CNN\)\([LeCun et al\. 2015](https://arxiv.org/html/2608.12710#bib.bib23)\)and Transformers\([Vaswani et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib41)\)are matrix\-wise models with numerous matrix parameter blocks in attention and linear layers\. Since training such models is computationally expensive, designing optimizers that can effectively exploit their matrix structure has attracted increasing attention\. Muon\([Jordan et al\. 2024](https://arxiv.org/html/2608.12710#bib.bib14)\)updates the matrix parameter blocks of matrix\-wise models using an orthogonalized momentum direction, which is efficiently computed through Newton–Schulz iterations\. It has demonstrated competitive empirical performance across a variety of large language models \(LLMs\)\([Liu et al\. 2025a](https://arxiv.org/html/2608.12710#bib.bib27)\)\. More recently, several studies have extended Muon to federated learning\([Takezawa et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib38);[Liu et al\. 2025b](https://arxiv.org/html/2608.12710#bib.bib28);[Zhang and Gao 2025](https://arxiv.org/html/2608.12710#bib.bib47)\)\. However, the existing Muon\-type federated algorithms are primarily developed for single\-level stochastic objectives, and it remains unclear how Muon can be effectively applied to federated stochastic compositional optimization, where the nested expectation introduces additional gradient estimation errors under heterogeneous client data\. This raises a natural question:Could we develop effective Muon\-type federated algorithms for matrix\-wise compositional optimization under heterogeneous data?

In this paper, we provide an affirmative answer to the above question and develop two effective Muon\-type federated learning algorithms \(i\.e\., FedCoMuon and FedCoMuon\-VR\) to solve distributed matrix\-wise stochastic compositional problems\. In summary, our main contributions are summarized as follows:

- •We develop a class of effective Muon\-type federated compositional algorithms \(i\.e\., FedCoMuon and FedCoMuon\-VR\) for distributed matrix\-wise stochastic compositional optimization\. Specifically, our FedCoMuon combines compositional gradient tracking with orthogonalized matrix momentum, while FedCoMuon\-VR further incorporates a momentum\-based variance\-reduction technique\([Cutkosky and Orabona 2019](https://arxiv.org/html/2608.12710#bib.bib4)\)\.
- •We provide a solid convergence analysis for the proposed algorithms under non\-convex and non\-i\.i\.d\. settings\. To find anϵ\\epsilon\-stationary point, our FedCoMuon requires a sample complexity of𝒪⁡\(ϵ−4\)\\mathcal\{O\}\(\\epsilon^\{\-4\}\)and a communication complexity of𝒪⁡\(ϵ−3\)\\mathcal\{O\}\(\\epsilon^\{\-3\}\), while our FedCoMuon\-VR has a lower sample complexity of𝒪⁡\(ϵ−3\)\\mathcal\{O\}\(\\epsilon^\{\-3\}\)while retaining the same communication complexity\. In particular, our FedCoMuon\-VR has a lower sample complexity than the existing Federated Muon algorithms\([Takezawa et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib38);[Liu et al\. 2025b](https://arxiv.org/html/2608.12710#bib.bib28);[Zhang and Gao 2025](https://arxiv.org/html/2608.12710#bib.bib47);[Qian et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib33)\)\.
- •Experiments on the task\-distributed meta learning problem and robust federated learning demonstrate effectiveness of the proposed algorithms\.

## 2Related Work

### 2\.1Federated Learning

Federated learning\([McMahan et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib29)\)is a popular distributed learning paradigm in machine learning\. FedAvg\([McMahan et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib29)\)is the first federated learning algorithm, which reduces communication frequency by allowing each client to perform multiple local SGD steps before server aggregation\. Under heterogeneous client data, however, repeated local updates may drift from the global descent direction and slow convergence\. Subsequently, some effective variants of FedAvg have been developed\. For example, FedProx\([Li et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib26)\)and SCAFFOLD\([Karimireddy et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib16)\)mitigate this issue through proximal regularization and control variances, respectively\. Meanwhile, momentum\-based, adaptive, and variance\-reduced extensions\([Khanduri et al\. 2021](https://arxiv.org/html/2608.12710#bib.bib17);[Cheng et al\. 2024](https://arxiv.org/html/2608.12710#bib.bib3);[Yan et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib44)\)further improve convergence and communication efficiency\.

### 2\.2Compositional Optimization

Compositional optimization\([Wang et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib43)\)is a class of effective nested structural optimization problems in machine learning\. Since compositional optimization is widely used in many machine learning tasks such as robust learning and federated learning, many algorithms\([Wang et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib43);[Ghadimi et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib8);[Chen et al\. 2021](https://arxiv.org/html/2608.12710#bib.bib2);[Zhang and Xiao 2021](https://arxiv.org/html/2608.12710#bib.bib46)\)have recently been developed\. For example, SCGD\([Wang et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib43)\)controls this bias by tracking the inner mapping with a moving average\. Subsequently, several accelerated algorithms based on momentum or variance\-reduction techniques have been developed for non\-convex stochastic compositional optimization\([Ghadimi et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib8);[Chen et al\. 2021](https://arxiv.org/html/2608.12710#bib.bib2);[Zhang and Xiao 2021](https://arxiv.org/html/2608.12710#bib.bib46);[Jiang et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib12)\)\.

To solve distributed compositional optimization, some federated compositional algorithms have been developed\. For example, ComFedL\([Huang and Li 2021](https://arxiv.org/html/2608.12710#bib.bib9)\)introduced a federated compositional framework for distributionally robust learning and meta learning\. Subsequently, Local\-MOML\([Wang et al\. 2023](https://arxiv.org/html/2608.12710#bib.bib42)\)and Local\-SCGDM\([Gao et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib7)\)improved sample and communication efficiency through local updates and momentum\-based tracking\. FedDRO\([Khanduri et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib18)\)further studies the interaction between compositional gradient bias and client heterogeneity in distributionally robust federated learning\.[Tarzanagh et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib39);[Huang et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib11);[Gao 2024](https://arxiv.org/html/2608.12710#bib.bib6)proposed variance\-reduced federated compositional algorithms for distributed non\-convex stochastic composition optimization based on variance\-reduction techniques\. In fact, the federated setting is more challenging because compositional gradient bias and client drift should be controlled simultaneously\.

### 2\.3Muon\-Based Optimization

In the last two years, Muon\([Jordan et al\. 2024](https://arxiv.org/html/2608.12710#bib.bib14)\)has emerged as a promising optimizer, which directly updates the matrix parameter blocks of matrix\-wise models\. Specifically, it uses orthogonalization of the matrix\-valued momentum by a few Newton–Schulz iterations, which can be interpreted as an LMO\-based optimizer over a spectral\-norm ball\([Pethick et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib31)\)\. More recently, its convergence properties have been studied in non\-convex stochastic optimization\([Shen et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib37);[Li and Hong 2025](https://arxiv.org/html/2608.12710#bib.bib24);[Riabinin et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib35);[Kovalev 2025](https://arxiv.org/html/2608.12710#bib.bib20);[Kim and Oh 2026](https://arxiv.org/html/2608.12710#bib.bib19)\)\. Subsequently, its variance\-reduced variants\([Sfyraki and Wang 2025](https://arxiv.org/html/2608.12710#bib.bib36);[Huang et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib10);[Qian et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib32);[Chang et al\. 2025](https://arxiv.org/html/2608.12710#bib.bib1)\)have also been studied\.

More recently, several works[Takezawa et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib38);[Liu et al\. 2025b](https://arxiv.org/html/2608.12710#bib.bib28);[Zhang and Gao 2025](https://arxiv.org/html/2608.12710#bib.bib47)have begun to study Federated Muon \(i\.e\., FedMuon\) algorithms for distributed matrix\-wise optimization\. Specifically,[Takezawa et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib38)proposed the FedMuon algorithm based on bias\-correction mechanism to address the bias induced by local linear minimization oracles\.[Liu et al\. 2025b](https://arxiv.org/html/2608.12710#bib.bib28)presented the FedMuon algorithm by using momentum aggregation and local\-global alignment to mitigate client drift under heterogeneous data\. Meanwhile,[Zhang and Gao 2025](https://arxiv.org/html/2608.12710#bib.bib47)developed the FedMuon algorithm with hyper\-parameter choices independent of problem\-specific constants, and established its convergence under both bounded\-variance and heavy\-tailed stochastic noise\. Subsequently,[Qian et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib33)proposed communication\-efficient federated Gluon algorithm based on gradient compression and error feedback\. However, these methods focus on standard single\-level objectives and do not provide algorithms or convergence guarantees for stochastic compositional optimization under heterogeneous client data\.

##### Notation\.

Let\[K\]=\{1,2,…,K\}\[K\]=\\\{1,2,\\ldots,K\\\}denote the set of clients\.∥⋅∥\\\|\\cdot\\\|denotes Euclidean and spectral norm for vector and matrix, respectively\. For matricesA,B∈ℝm×nA,B\\in\\mathbb\{R\}^\{m\\times n\}, we use⟨A,B⟩=tr⁡\(A⊤​B\)\\langle A,B\\rangle=\\operatorname\{tr\}\(A^\{\\top\}B\)to denote the Frobenius inner product, and use‖A‖F\\\|A\\\|\_\{F\}to denote the Frobenius norm\.A⊗BA\\otimes Bdenotes the Kronecker product of matricesAAandBB\.

## 3Preliminaries

In this paper, we study the Muon optimizer to solve the following distributed matrix\-wise compositional optimization problem:

minW∈ℝm×n⁡1K​∑k=1K𝔼ζ∼𝒟fk​\[fk​\(𝔼ξ∼𝒟gk​\[gk​\(W,ξ\)\],ζ\)\],\\displaystyle\\min\_\{W\\in\\mathbb\{R\}^\{m\\times n\}\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\_\{\\zeta\\sim\\mathcal\{D\}\_\{f\}^\{k\}\}\\left\[f^\{k\}\\left\(\\mathbb\{E\}\_\{\\xi\\sim\\mathcal\{D\}\_\{g\}^\{k\}\}\[g^\{k\}\(W;\\xi\)\];\\zeta\\right\)\\right\],\(1\)where the inner and outer expected mappings are defined asgk​\(W\)≜𝔼ξ∼𝒟gk​\[gk​\(W,ξ\)\]:ℝm×n→ℝdg^\{k\}\(W\)\\triangleq\\mathbb\{E\}\_\{\\xi\\sim\\mathcal\{D\}\_\{g\}^\{k\}\}\[g^\{k\}\(W;\\xi\)\]:\\mathbb\{R\}^\{m\\times n\}\\to\\mathbb\{R\}^\{d\}andfk​\(y\)≜𝔼ζ∼𝒟fk​\[fk​\(y,ζ\)\]:ℝd→ℝf^\{k\}\(y\)\\triangleq\\mathbb\{E\}\_\{\\zeta\\sim\\mathcal\{D\}\_\{f\}^\{k\}\}\[f^\{k\}\(y;\\zeta\)\]:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}, respectively\. We writeF⁡\(W\)=K−1​∑k=1KFk​\(W\)F\(W\)=K^\{\-1\}\\sum\_\{k=1\}^\{K\}F^\{k\}\(W\), whereFk​\(W\)=fk​\(gk​\(W\)\)F^\{k\}\(W\)=f^\{k\}\(g^\{k\}\(W\)\)\. The inner and outer data distributions,𝒟gk\\mathcal\{D\}\_\{g\}^\{k\}and𝒟fk\\mathcal\{D\}\_\{f\}^\{k\}, may differ across clients\. This formulation therefore captures both the nested compositional structure and the data heterogeneity inherent in federated settings\.

Next, we introduce several mild assumptions for Problem \([1](https://arxiv.org/html/2608.12710#S3.E1)\)\.

###### Assumption 1\(Lower\-bounded objective\)\.

The global objectiveF⁡\(W\)F\(W\)has a lower bound, i\.e\.,F∗:=infW∈ℝm×nF⁡\(W\)\>−∞F\_\{\*\}:=\\inf\_\{W\\in\\mathbb\{R\}^\{m\\times n\}\}F\(W\)\>\-\\infty\.

###### Assumption 2\(Unbiased stochastic oracles and bounded variances\)\.

For any clientk∈\[K\]k\\in\[K\], the stochastic oracles are unbiased:

𝔼ξk​\[gk​\(W,ξk\)\]=gk​\(W\),𝔼ξk​\[∇gk​\(W,ξk\)\]=∇gk​\(W\),𝔼ζk​\[∇yfk​\(y,ζk\)\]=∇yfk​\(y\)\.\\displaystyle\\mathbb\{E\}\_\{\\xi^\{k\}\}\[g^\{k\}\(W;\\xi^\{k\}\)\]=g^\{k\}\(W\),\\ \\mathbb\{E\}\_\{\\xi^\{k\}\}\[\\nabla g^\{k\}\(W;\\xi^\{k\}\)\]=\\nabla g^\{k\}\(W\),\\ \\mathbb\{E\}\_\{\\zeta^\{k\}\}\[\\nabla\_\{y\}f^\{k\}\(y;\\zeta^\{k\}\)\]=\\nabla\_\{y\}f^\{k\}\(y\)\.The samples are independent across clients and iterations, andξk\\xi^\{k\}is independent ofζk\\zeta^\{k\}\. In addition, there exist constantsσg,σ∇g,σf\>0\\sigma\_\{g\},\\sigma\_\{\\nabla g\},\\sigma\_\{f\}\>0such that, for anyW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}andy∈ℝdy\\in\\mathbb\{R\}^\{d\},

𝔼ξk​‖gk​\(W,ξk\)−gk​\(W\)‖2≤σg2,\\displaystyle\\mathbb\{E\}\_\{\\xi^\{k\}\}\\\|g^\{k\}\(W;\\xi^\{k\}\)\-g^\{k\}\(W\)\\\|^\{2\}\\leq\\sigma\_\{g\}^\{2\},𝔼ξk​‖∇gk​\(W,ξk\)−∇gk​\(W\)‖F2≤σ∇g2,\\displaystyle\\mathbb\{E\}\_\{\\xi^\{k\}\}\\\|\\nabla g^\{k\}\(W;\\xi^\{k\}\)\-\\nabla g^\{k\}\(W\)\\\|\_\{F\}^\{2\}\\leq\\sigma\_\{\\nabla g\}^\{2\},𝔼ζk​‖∇yfk​\(y,ζk\)−∇yfk​\(y\)‖2≤σf2\.\\displaystyle\\mathbb\{E\}\_\{\\zeta^\{k\}\}\\\|\\nabla\_\{y\}f^\{k\}\(y;\\zeta^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(y\)\\\|^\{2\}\\leq\\sigma\_\{f\}^\{2\}\.

###### Assumption 3\(Bounded gradient moments\)\.

For any clientk∈\[K\]k\\in\[K\], there exist constantsCg,Cf\>0C\_\{g\},C\_\{f\}\>0such that, for anyW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}andy∈ℝdy\\in\\mathbb\{R\}^\{d\}, the stochastic Jacobians and outer gradients have bounded second moments:

𝔼ξk​‖∇gk​\(W,ξk\)‖F2≤Cg2,𝔼ζk​‖∇yfk​\(y,ζk\)‖2≤Cf2\.\\displaystyle\\mathbb\{E\}\_\{\\xi^\{k\}\}\\\|\\nabla g^\{k\}\(W;\\xi^\{k\}\)\\\|\_\{F\}^\{2\}\\leq C\_\{g\}^\{2\},\\ \\mathbb\{E\}\_\{\\zeta^\{k\}\}\\\|\\nabla\_\{y\}f^\{k\}\(y;\\zeta^\{k\}\)\\\|^\{2\}\\leq C\_\{f\}^\{2\}\.

Assumptions[1](https://arxiv.org/html/2608.12710#Thmassumption1)–[3](https://arxiv.org/html/2608.12710#Thmassumption3)have been commonly used in the convergence analysis of stochastic compositional optimization algorithms\([Tarzanagh et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib39);[Gao et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib7);[Jiang et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib12);[Gao 2024](https://arxiv.org/html/2608.12710#bib.bib6)\)\.

Algorithm 1FedCoMuon Algorithm1:Input:

η\>0\\eta\>0,

α∈\[0,1\)\\alpha\\in\[0,1\),

β∈\[0,1\)\\beta\\in\[0,1\), and

τ\>0\\tau\>0\.

2:Initialize:For allk∈\[K\]k\\in\[K\], setW0k=W0∈ℝm×nW\_\{0\}^\{k\}=W\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\},u0k=gk​\(W0k,ξ0k\)u\_\{0\}^\{k\}=g^\{k\}\(W\_\{0\}^\{k\};\\xi\_\{0\}^\{k\}\), andM0k=∇gk​\(W0k,ξ0k\)​\(∇yfk​\(u0k,ζ0k\)⊗In\)M\_\{0\}^\{k\}=\\nabla g^\{k\}\(W\_\{0\}^\{k\};\\xi\_\{0\}^\{k\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(u\_\{0\}^\{k\};\\zeta\_\{0\}^\{k\}\)\\otimes I\_\{n\}\\bigr\)forξ0k∼𝒟gk\\xi\_\{0\}^\{k\}\\sim\\mathcal\{D\}\_\{g\}^\{k\}andζ0k∼𝒟fk\\zeta\_\{0\}^\{k\}\\sim\\mathcal\{D\}\_\{f\}^\{k\};

3:for

t=0,1,…,T−1t=0,1,\\dots,T\-1do

4:foreach client

k∈\[K\]k\\in\[K\]\(in parallel\)do

5:\(Utk,Σtk,Vtk\)=SVD⁡\(Mtk\)\(U\_\{t\}^\{k\},\\Sigma\_\{t\}^\{k\},V\_\{t\}^\{k\}\)=\\mathrm\{SVD\}\(M\_\{t\}^\{k\}\); // OrthonormalizeMtkM\_\{t\}^\{k\}with the Newton–Schulz approach

6:

Wt\+1k=Wtk−η​Utk​\(Vtk\)⊤W\_\{t\+1\}^\{k\}=W\_\{t\}^\{k\}\-\\eta U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\};

7:Draw two independent samples

ξt\+1k∼𝒟gk\\xi\_\{t\+1\}^\{k\}\\sim\\mathcal\{D\}\_\{g\}^\{k\}and

ζt\+1k∼𝒟fk\\zeta\_\{t\+1\}^\{k\}\\sim\\mathcal\{D\}\_\{f\}^\{k\};

8:

ut\+1k=α​gk​\(Wt\+1k,ξt\+1k\)\+\(1−α\)​utku\_\{t\+1\}^\{k\}=\\alpha g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\+\(1\-\\alpha\)u\_\{t\}^\{k\};

9:

Zt\+1k=∇gk​\(Wt\+1k,ξt\+1k\)​\(∇yfk​\(ut\+1k,ζt\+1k\)⊗In\)Z\_\{t\+1\}^\{k\}=\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\\otimes I\_\{n\}\\bigr\);

10:

Mt\+1k=β​Zt\+1k\+\(1−β\)​MtkM\_\{t\+1\}^\{k\}=\\beta Z\_\{t\+1\}^\{k\}\+\(1\-\\beta\)M\_\{t\}^\{k\};

11:endfor

12:if

mod⁡\(t\+1,τ\)=0\\operatorname\{mod\}\(t\+1,\\tau\)=0then

13:Receive

\{Wt\+1k,Mt\+1k\}k=1K\\\{W\_\{t\+1\}^\{k\},M\_\{t\+1\}^\{k\}\\\}\_\{k=1\}^\{K\}from all clients;

14:

W¯t\+1=K−1​∑kWt\+1k\\bar\{W\}\_\{t\+1\}=K^\{\-1\}\\sum\_\{k\}W\_\{t\+1\}^\{k\};

15:

M¯t\+1=K−1​∑kMt\+1k\\bar\{M\}\_\{t\+1\}=K^\{\-1\}\\sum\_\{k\}M\_\{t\+1\}^\{k\};

16:Send

W¯t\+1\\bar\{W\}\_\{t\+1\}and

M¯t\+1\\bar\{M\}\_\{t\+1\}to each client;

17:endif

18:endfor

19:Output:Sampling uniformly from

\{W¯t\}t=0T−1\\\{\\bar\{W\}\_\{t\}\\\}\_\{t=0\}^\{T\-1\}\(in theory\), and

W¯T\\bar\{W\}\_\{T\}\(in practice\)\.

## 4Federated Compositional Muon Methods

In this section, we propose a class of efficient federated compositional Muon algorithms \(i\.e\., FedCoMuon and FedCoMuon\-VR\) for large matrix\-valued models\. Specifically, our FedCoMuon builds on compositional gradient tracking and Muon, while our FedCoMuon\-VR further builds on a momentum\-based variance\-reduction technique\.

Algorithm 2FedCoMuon\-VR Algorithm1:Input:

η\>0\\eta\>0,

α,β,γ,ρ∈\[0,1\)\\alpha,\\beta,\\gamma,\\rho\\in\[0,1\),

τ\>0\\tau\>0, and

b\>0b\>0\.

2:Initialize:For allk∈\[K\]k\\in\[K\], setW0k=W0∈ℝm×nW\_\{0\}^\{k\}=W\_\{0\}\\in\\mathbb\{R\}^\{m\\times n\}, and drawbbi\.i\.d\. samples\{ξ0,jk\}j=1b\\\{\\xi\_\{0,j\}^\{k\}\\\}\_\{j=1\}^\{b\}from𝒟gk\\mathcal\{D\}\_\{g\}^\{k\}andbbi\.i\.d\. samples\{ζ0,jk\}j=1b\\\{\\zeta\_\{0,j\}^\{k\}\\\}\_\{j=1\}^\{b\}from𝒟fk\\mathcal\{D\}\_\{f\}^\{k\}, and setu0k=1b​∑j=1bgk​\(W0k,ξ0,jk\)u\_\{0\}^\{k\}=\\frac\{1\}\{b\}\\sum\_\{j=1\}^\{b\}g^\{k\}\(W\_\{0\}^\{k\};\\xi\_\{0,j\}^\{k\}\),H0k=1b​∑j=1b∇gk​\(W0k,ξ0,jk\)H\_\{0\}^\{k\}=\\frac\{1\}\{b\}\\sum\_\{j=1\}^\{b\}\\nabla g^\{k\}\(W\_\{0\}^\{k\};\\xi\_\{0,j\}^\{k\}\), andv0k=1b​∑j=1b∇yfk​\(u0k,ζ0,jk\)v\_\{0\}^\{k\}=\\frac\{1\}\{b\}\\sum\_\{j=1\}^\{b\}\\nabla\_\{y\}f^\{k\}\(u\_\{0\}^\{k\};\\zeta\_\{0,j\}^\{k\}\),M0k=H0k​\(v0k⊗In\)M\_\{0\}^\{k\}=H\_\{0\}^\{k\}\(v\_\{0\}^\{k\}\\otimes I\_\{n\}\)\. SetM¯0=K−1​∑k=1KM0k\\bar\{M\}\_\{0\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{0\}^\{k\}andM0k=M¯0M\_\{0\}^\{k\}=\\bar\{M\}\_\{0\}for allk∈\[K\]k\\in\[K\];

3:for

t=0,1,…,T−1t=0,1,\\ldots,T\-1do

4:foreach client

k∈\[K\]k\\in\[K\]\(in parallel\)do

5:

\(Utk,Σtk,Vtk\)=SVD⁡\(Mtk\)\(U\_\{t\}^\{k\},\\Sigma\_\{t\}^\{k\},V\_\{t\}^\{k\}\)=\\operatorname\{SVD\}\(M\_\{t\}^\{k\}\);

6:

Wt\+1k=Wtk−η​Utk​\(Vtk\)⊤W\_\{t\+1\}^\{k\}=W\_\{t\}^\{k\}\-\\eta U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\};

7:endfor

8:if

mod⁡\(t\+1,τ\)=0\\operatorname\{mod\}\(t\+1,\\tau\)=0then

9:Receive

\{Wt\+1k\}k=1K\\\{W\_\{t\+1\}^\{k\}\\\}\_\{k=1\}^\{K\}from all clients;

10:W¯t\+1=1K​∑k=1KWt\+1k\\bar\{W\}\_\{t\+1\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}W\_\{t\+1\}^\{k\};

11:Send

W¯t\+1\\bar\{W\}\_\{t\+1\}to each client;

12:endif

13:foreach client

k∈\[K\]k\\in\[K\]\(in parallel\)do

14:Draw

ξt\+1k∼𝒟gk\\xi\_\{t\+1\}^\{k\}\\sim\\mathcal\{D\}\_\{g\}^\{k\}and

ζt\+1k∼𝒟fk\\zeta\_\{t\+1\}^\{k\}\\sim\\mathcal\{D\}\_\{f\}^\{k\};

15:ut\+1k=gk​\(Wt\+1k,ξt\+1k\)\+\(1−α\)​\(utk−gk​\(Wtk,ξt\+1k\)\)u\_\{t\+1\}^\{k\}=g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\+\(1\-\\alpha\)\\bigl\(u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\\bigr\);

16:vt\+1k=ΠCf​\[∇yfk​\(ut\+1k,ζt\+1k\)\+\(1−β\)​\(vtk−∇yfk​\(utk,ζt\+1k\)\)\]v\_\{t\+1\}^\{k\}=\\Pi\_\{C\_\{f\}\}\\bigl\[\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\+\(1\-\\beta\)\\bigl\(v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\\bigr\)\\bigr\];

17:Ht\+1k=ΠCg​\[∇gk​\(Wt\+1k,ξt\+1k\)\+\(1−γ\)​\(Htk−∇gk​\(Wtk,ξt\+1k\)\)\]H\_\{t\+1\}^\{k\}=\\Pi\_\{C\_\{g\}\}\\bigl\[\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\+\(1\-\\gamma\)\\bigl\(H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\\bigr\)\\bigr\];

18:

Mt\+1k=\(1−ρ\)​Mtk\+ρ​Ht\+1k​\(vt\+1k⊗In\)M\_\{t\+1\}^\{k\}=\(1\-\\rho\)M\_\{t\}^\{k\}\+\\rho H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\);

19:endfor

20:if

mod⁡\(t\+1,τ\)=0\\operatorname\{mod\}\(t\+1,\\tau\)=0then

21:Receive

\{Mt\+1k\}k=1K\\\{M\_\{t\+1\}^\{k\}\\\}\_\{k=1\}^\{K\}from all clients;

22:M¯t\+1=1K​∑k=1KMt\+1k\\bar\{M\}\_\{t\+1\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}M\_\{t\+1\}^\{k\};

23:Send

M¯t\+1\\bar\{M\}\_\{t\+1\}to each client;

24:endif

25:endfor

26:Output:Sampling uniformly from

\{W¯t\}t=0T−1\\\{\\bar\{W\}\_\{t\}\\\}\_\{t=0\}^\{T\-1\}\(in theory\), and

W¯T\\bar\{W\}\_\{T\}\(in practice\)\.

### 4\.1FedCoMuon Algorithm

In this subsection, we provide an efficient federated compositional Muon \(i\.e\., FedCoMuon\) algorithm to solve the problem \([1](https://arxiv.org/html/2608.12710#S3.E1)\)\. Algorithm[1](https://arxiv.org/html/2608.12710#alg1)shows the algorithmic framework for FedCoMuon\.

In our FedCoMuon algorithm, at thett\-th iteration, each clientkkuses its local stochastic samples to update the matrix momentum as

Mt\+1k=β​Zt\+1k\+\(1−β\)​Mtk,M\_\{t\+1\}^\{k\}=\\beta Z\_\{t\+1\}^\{k\}\+\(1\-\\beta\)M\_\{t\}^\{k\},\(2\)whereβ∈\[0,1\)\\beta\\in\[0,1\)is the momentum parameter\. HereZt\+1kZ\_\{t\+1\}^\{k\}denotes the stochastic compositional gradient, defined as

Zt\+1k=∇gk​\(Wt\+1k,ξt\+1k\)​\(∇yfk​\(ut\+1k,ζt\+1k\)⊗In\),Z\_\{t\+1\}^\{k\}=\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\\otimes I\_\{n\}\\bigr\),\(3\)whereξt\+1k\\xi\_\{t\+1\}^\{k\}andζt\+1k\\zeta\_\{t\+1\}^\{k\}are fresh stochastic samples\. In the above estimator,ut\+1ku\_\{t\+1\}^\{k\}tracks the inner function valuegk​\(Wt\+1k\)g^\{k\}\(W\_\{t\+1\}^\{k\}\), thereby enablingZt\+1kZ\_\{t\+1\}^\{k\}to form a matrix\-form stochastic compositional gradient estimate\. The parameterα∈\[0,1\)\\alpha\\in\[0,1\)controls the moving\-average update ofut\+1ku\_\{t\+1\}^\{k\}\.

Following the Muon algorithm\([Jordan et al\. 2024](https://arxiv.org/html/2608.12710#bib.bib14)\), we use Newton–Schulz iterations instead of an expensive exact SVD to obtain the orthogonalized momentum direction\.

Everyτ\\taulocal iterations, clients communicate with the server\. In each communication round, the server receives\{Wt\+1k,Mt\+1k\}k=1K\\\{W\_\{t\+1\}^\{k\},M\_\{t\+1\}^\{k\}\\\}\_\{k=1\}^\{K\}, averages them to obtainW¯t\+1\\bar\{W\}\_\{t\+1\}andM¯t\+1\\bar\{M\}\_\{t\+1\}, and sends the averaged variables back to all clients\.

### 4\.2FedCoMuon\-VR Algorithm

In this subsection, we provide an efficient federated compositional variance\-reduced Muon \(i\.e\., FedCoMuon\-VR\) algorithm to solve the problem \([1](https://arxiv.org/html/2608.12710#S3.E1)\) based on the momentum\-based variance\-reduction technique\. Algorithm[2](https://arxiv.org/html/2608.12710#alg2)provides the algorithmic framework for FedCoMuon\-VR\.

In Algorithm[2](https://arxiv.org/html/2608.12710#alg2), after obtainingWt\+1kW\_\{t\+1\}^\{k\}, each clientkkdraws fresh samplesξt\+1k\\xi\_\{t\+1\}^\{k\}andζt\+1k\\zeta\_\{t\+1\}^\{k\}, and recursively updatesut\+1ku\_\{t\+1\}^\{k\},vt\+1kv\_\{t\+1\}^\{k\}, andHt\+1kH\_\{t\+1\}^\{k\}to track the inner function value, the outer gradient, and the inner Jacobian, respectively\. Each recursive estimator evaluates its function on the same fresh sample at two consecutive iterates, which reduces the stochastic estimation error\.

Letℬf:=\{v:‖v‖≤Cf\}\\mathcal\{B\}\_\{f\}:=\\\{v:\\\|v\\\|\\leq C\_\{f\}\\\}andℬg:=\{H:‖H‖F≤Cg\}\\mathcal\{B\}\_\{g\}:=\\\{H:\\\|H\\\|\_\{F\}\\leq C\_\{g\}\\\}, and letΠCf\\Pi\_\{C\_\{f\}\}andΠCg\\Pi\_\{C\_\{g\}\}denote the Euclidean projections ontoℬf\\mathcal\{B\}\_\{f\}andℬg\\mathcal\{B\}\_\{g\}, respectively\. These projection operators keep the outer\-gradient estimator and the Jacobian estimator bounded\. The resulting estimators form the variance\-reduced compositional gradient estimateHt\+1k​\(vt\+1k⊗In\)H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\), which is used to update the matrix momentum with parameterρ∈\[0,1\)\\rho\\in\[0,1\)\. Everyτ\\taulocal iterations, the server averages the local model parameters and momentum matrices, and broadcasts the averaged variables back to all clients\.

## 5Convergence Analysis

In this section, we study the convergence properties of our FedCoMuon and FedCoMuon\-VR algorithm under some mild assumptions\. All related proofs are provided in the Appendix\.

### 5\.1Convergence Properties of Our FedCoMuon

###### Assumption 4\(Smoothness\)\.

For any clientk∈\[K\]k\\in\[K\], the population mappingsgkg^\{k\}andfkf^\{k\}areLgL\_\{g\}\- andLfL\_\{f\}\-smooth, respectively\. That is, for anyW1,W2∈ℝm×nW\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{m\\times n\}andy1,y2∈ℝdy\_\{1\},y\_\{2\}\\in\\mathbb\{R\}^\{d\},

‖∇gk​\(W1\)−∇gk​\(W2\)‖F\\displaystyle\\\|\\nabla g^\{k\}\(W\_\{1\}\)\-\\nabla g^\{k\}\(W\_\{2\}\)\\\|\_\{F\}≤Lg​‖W1−W2‖F,‖∇yfk​\(y1\)−∇yfk​\(y2\)‖≤Lf​‖y1−y2‖\.\\displaystyle\\leq L\_\{g\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\},\\ \\\|\\nabla\_\{y\}f^\{k\}\(y\_\{1\}\)\-\\nabla\_\{y\}f^\{k\}\(y\_\{2\}\)\\\|\\leq L\_\{f\}\\\|y\_\{1\}\-y\_\{2\}\\\|\.

###### Assumption 5\(Gradient heterogeneity\)\.

There exists a constantδ≥0\\delta\\geq 0such that, for allW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\},

1K​∑k=1K‖∇Fk​\(W\)−∇F​\(W\)‖F2≤δ2\.\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F^\{k\}\(W\)\-\\nabla F\(W\)\\\|\_\{F\}^\{2\}\\leq\\delta^\{2\}\.

###### Theorem 1\.

Suppose Assumptions[1](https://arxiv.org/html/2608.12710#Thmassumption1)–[5](https://arxiv.org/html/2608.12710#Thmassumption5)hold\. Let\{Wtk,utk,Mtk\}\\\{W\_\{t\}^\{k\},u\_\{t\}^\{k\},M\_\{t\}^\{k\}\\\}be generated by Algorithm[1](https://arxiv.org/html/2608.12710#alg1)with0<α,β<10<\\alpha,\\beta<1, andα​τ≤1\\alpha\\tau\\leq 1\. Then, we have

1T​∑t=0T−1𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+2​n​Cg​Lf​σgβ​T\+2​n​\(1β​T\+2​β​τ\)​\(Cg​σf\+Cf​σ∇g\)\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{2\\sqrt\{n\}C\_\{g\}L\_\{f\}\\sigma\_\{g\}\}\{\\beta T\}\+2\\sqrt\{n\}\\bigl\(\\frac\{1\}\{\\beta T\}\+2\\beta\\tau\\bigr\)\\bigl\(C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\bigr\)\+2​n​Cg​Lf​\(1\+2​β​τ\)​\(4​σg2α​T\+86​Cg2​nα2​η2\+86​α​σg2\)1/2\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}C\_\{g\}L\_\{f\}\(1\+2\\beta\\tau\)\\biggl\(\\frac\{4\\sigma\_\{g\}^\{2\}\}\{\\alpha T\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\\biggr\)^\{1/2\}\+η​n​LF​\(12\+4​τ\+6β\+8​β​τ2\)\+2​n​β​τ​δ\+2​n​βK​Cg2​σf2\+Cf2​σ∇g2\.\\displaystyle\\hskip 9\.24994pt\+\\eta nL\_\{F\}\\bigl\(\\frac\{1\}\{2\}\+4\\tau\+\\frac\{6\}\{\\beta\}\+8\\beta\\tau^\{2\}\\bigr\)\+2\\sqrt\{n\}\\,\\beta\\tau\\delta\+\\frac\{2\\sqrt\{n\\beta\}\}\{\\sqrt\{K\}\}\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\.\(4\)

### 5\.2Convergence Properties of Our FedCoMuon\-VR

###### Assumption 6\(Mean\-square sample smoothness\)\.

For any clientk∈\[K\]k\\in\[K\], the sample mappings are mean\-square Lipschitz\. That is, for anyW1,W2∈ℝm×nW\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{m\\times n\}andy1,y2∈ℝdy\_\{1\},y\_\{2\}\\in\\mathbb\{R\}^\{d\},

𝔼ξk​‖gk​\(W1,ξk\)−gk​\(W2,ξk\)‖2≤Cg2​‖W1−W2‖F2,\\displaystyle\\mathbb\{E\}\_\{\\xi^\{k\}\}\\\|g^\{k\}\(W\_\{1\};\\xi^\{k\}\)\-g^\{k\}\(W\_\{2\};\\xi^\{k\}\)\\\|^\{2\}\\leq C\_\{g\}^\{2\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\}^\{2\},𝔼ξk​‖∇gk​\(W1,ξk\)−∇gk​\(W2,ξk\)‖F2≤Lg2​‖W1−W2‖F2,\\displaystyle\\mathbb\{E\}\_\{\\xi^\{k\}\}\\\|\\nabla g^\{k\}\(W\_\{1\};\\xi^\{k\}\)\-\\nabla g^\{k\}\(W\_\{2\};\\xi^\{k\}\)\\\|\_\{F\}^\{2\}\\leq L\_\{g\}^\{2\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\}^\{2\},𝔼ζk​‖∇yfk​\(y1,ζk\)−∇yfk​\(y2,ζk\)‖2≤Lf2​‖y1−y2‖2\.\\displaystyle\\mathbb\{E\}\_\{\\zeta^\{k\}\}\\\|\\nabla\_\{y\}f^\{k\}\(y\_\{1\};\\zeta^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(y\_\{2\};\\zeta^\{k\}\)\\\|^\{2\}\\leq L\_\{f\}^\{2\}\\\|y\_\{1\}\-y\_\{2\}\\\|^\{2\}\.

###### Assumption 7\(Client heterogeneity\)\.

There exist constantsΔg,Δ∇g,Δf≥0\\Delta\_\{g\},\\Delta\_\{\\nabla g\},\\Delta\_\{f\}\\geq 0such that, for anyk,j∈\[K\]k,j\\in\[K\],W∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}andy∈ℝdy\\in\\mathbb\{R\}^\{d\},

‖∇yfk​\(y\)−∇yfj​\(y\)‖≤Δf,\\displaystyle\\\|\\nabla\_\{y\}f^\{k\}\(y\)\-\\nabla\_\{y\}f^\{j\}\(y\)\\\|\\leq\\Delta\_\{f\},‖∇gk​\(W\)−∇gj​\(W\)‖F≤Δ∇g,\\displaystyle\\\|\\nabla g^\{k\}\(W\)\-\\nabla g^\{j\}\(W\)\\\|\_\{F\}\\leq\\Delta\_\{\\nabla g\},‖gk​\(W\)−gj​\(W\)‖≤Δg\.\\displaystyle\\\|g^\{k\}\(W\)\-g^\{j\}\(W\)\\\|\\leq\\Delta\_\{g\}\.

Assumption[7](https://arxiv.org/html/2608.12710#Thmassumption7)imposes a common bounded\-heterogeneity condition for compositional federated learning under non\-i\.i\.d\. setting\([Tarzanagh et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib39);[Huang et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib11)\)\.

###### Theorem 2\.

Suppose Assumptions[1](https://arxiv.org/html/2608.12710#Thmassumption1)–[3](https://arxiv.org/html/2608.12710#Thmassumption3),[6](https://arxiv.org/html/2608.12710#Thmassumption6), and[7](https://arxiv.org/html/2608.12710#Thmassumption7)hold\. Let\{Wtk,utk,vtk,Htk,Mtk\}\\\{W\_\{t\}^\{k\},u\_\{t\}^\{k\},v\_\{t\}^\{k\},H\_\{t\}^\{k\},M\_\{t\}^\{k\}\\\}be generated by Algorithm[2](https://arxiv.org/html/2608.12710#alg2)withη\>0\\eta\>0,0<α,β,γ,ρ<10<\\alpha,\\beta,\\gamma,\\rho<1,b≥1b\\geq 1, andτ\>0\\tau\>0\. Then, for anyT≥1T\\geq 1, we have

1T​∑t=0T−1𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+LF​n​η2\+2n\[4​Cg​Cfρ​T\+LF​n​ηρ\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{L\_\{F\}n\\eta\}\{2\}\+2\\sqrt\{n\}\\biggl\[\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\(2\+6​ρ​τ\)​\(Cf2​σ∇g2​\(1γ​T​b\+2​γ\)\+2​Cg2​σf2​\(1β​T​bCLOSECLOSE\\displaystyle\\hskip 18\.49988pt\+\(2\+\\sqrt\{6\}\\,\\rho\\tau\)\\biggl\(C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+2C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\Bigl\(\\frac\{1\}\{\\beta Tb\}OPEN\+2​β\)\+2​Cg2​Lf2​σg2​\[1α​T​b\+2​α\+8​αβ​T​b\+8​α2β​\(1\+2​α\)\]\\displaystyle\\hskip 18\.49988pt\+2\\beta\\Bigr\)\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+4​n​η2​τ​\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβCLOSE\\displaystyle\\hskip 18\.49988pt\+4n\\eta^\{2\}\\tau\\Bigl\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}OPENOPEN\+Cf2​Lg2γ\)\+n​η2​τ2​\(Cf2​Lg2\+Cg4​Lf2\+LF22\)\)1/2\\displaystyle\\hskip 18\.49988pt\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\Bigr\)\+n\\eta^\{2\}\\tau^\{2\}\\left\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\+\\frac\{L\_\{F\}^\{2\}\}\{2\}\\right\)\\biggr\)^\{1/2\}\+6ρτ\(Cf2Δ∇g2\+2Cg2Δf2\+2Cg2Lf2Δg2\)1/2\]\.\\displaystyle\\hskip 18\.49988pt\+\\sqrt\{6\}\\,\\rho\\tau\\left\(C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+2C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\right\)^\{1/2\}\\biggr\]\.\(5\)

Figure 1:Training and test performance on the imbalanced MNIST robust federated learning task\.Figure 2:Effect of the synchronization gapτ\\tauon our algorithm for the robust federated learning task\.

## 6Numerical Experiments

In the section, we evaluate FedCoMuon and FedCoMuon\-VR on robust federated learning and task\-distributed meta learning\. In the experiment, we compare our methods with task\-specific standard federated baselines: FedAvg\([McMahan et al\. 2017](https://arxiv.org/html/2608.12710#bib.bib29)\)for robust federated learning and FedMAML\([Fallah et al\. 2020](https://arxiv.org/html/2608.12710#bib.bib5)\)for task\-distributed meta learning\. We also consider three recent Muon\-based federated methods\. Since all three methods are named FedMuon in their original papers, we distinguish them as FedMuon\([Zhang and Gao 2025](https://arxiv.org/html/2608.12710#bib.bib47)\), FedMuon\-LGA\([Liu et al\. 2025b](https://arxiv.org/html/2608.12710#bib.bib28)\), and FedMuon\-BC\([Takezawa et al\. 2026](https://arxiv.org/html/2608.12710#bib.bib38)\)\. In addition, we include the federated compositional methods ComFedL\([Huang and Li 2021](https://arxiv.org/html/2608.12710#bib.bib9)\)and Local\-SCGDM\([Gao et al\. 2022](https://arxiv.org/html/2608.12710#bib.bib7)\)\. For all methods, the learning rates and method\-specific hyper\-parameters are selected via grid search, and we report the best\-performing configurations\.

### 6\.1Robust Federated Learning

In this experiment, we evaluate FedCoMuon and FedCoMuon\-VR on robust federated learning, which can be formulated as the following distributed compositional optimization problem:

minW∈ℝm×n⁡1K​∑k=1Kf⁡\(gk​\(W\)/λ\),\\min\_\{W\\in\\mathbb\{R\}^\{m\\times n\}\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}f\\left\(g^\{k\}\(W\)/\\lambda\\right\),\(6\)wheref⁡\(⋅\)=exp⁡\(⋅\)f\(\\cdot\)=\\exp\(\\cdot\)andλ\>0\\lambda\>0is a regularization parameter\. Other monotonically increasing functions may also be used asff\. We implement image classification on the MNIST\([LeCun et al\. 1998](https://arxiv.org/html/2608.12710#bib.bib22)\)dataset and language modeling on the WikiText\-2\([Merity et al\. 2016](https://arxiv.org/html/2608.12710#bib.bib30)\)dataset\. Specifically, we train a 4\-layer CNN on MNIST and a Transformer on WikiText\-2\.

Figure 3:Test loss and perplexity of each method on the WikiText\-2 language\-modeling task\.#### 6\.1\.1Image Classification on MNIST

For image classification, we train a 4\-layer CNN on MNIST in a federated system with 10 clients\. To create an imbalanced partition, we assign 5000 training images to one client and 20 images to each of the remaining clients\. For FedCoMuon and FedCoMuon\-VR, the learning rate is set to0\.010\.01\. The total number of training iterations is set to500500, and the synchronization gap is set toτ=5\\tau=5unless otherwise specified\. Additional implementation details and complete hyperparameter settings are provided in the Appendix[C](https://arxiv.org/html/2608.12710#A3)\.

As shown in Figure[1](https://arxiv.org/html/2608.12710#S5.F1), FedCoMuon and FedCoMuon\-VR outperform the baseline methods in terms of both training and test performance under the highly imbalanced data partition\. Our methods outperform the federated compositional baselines, demonstrating the effectiveness of Muon\-based updates for robust compositional optimization\. In particular, FedCoMuon\-VR exhibits more stable convergence and achieves the best overall performance, showing the benefit of the variance\-reduction mechanism\. Figure[2](https://arxiv.org/html/2608.12710#S5.F2)further shows that our methods perform consistently under different synchronization gaps, withτ=1\\tau=1yielding the best performance\.

#### 6\.1\.2Language Modeling on WikiText\-2

For language modeling, we conduct experiments on WikiText\-2 with an 8\-layer Transformer language model\. The Transformer has a hidden dimension of 768, 8 attention heads, a feed\-forward dimension of 1024, sinusoidal positional encodings, and a sequence length of 128\. We split the training data across 10 clients in an imbalanced manner, where one client holds about 50% of the training blocks and the remaining clients equally share the rest\. For FedCoMuon and FedCoMuon\-VR, the learning rates are set to0\.020\.02and0\.030\.03, respectively\. The total number of training iterations is set to500500\. We report test loss and perplexity \(PPL\) as the evaluation metrics\. Additional implementation details and complete hyper\-parameter settings are provided in Appendix[C](https://arxiv.org/html/2608.12710#A3)\.

As shown in Figure[3](https://arxiv.org/html/2608.12710#S6.F3), both FedCoMuon and FedCoMuon\-VR converge faster and achieve lower test loss and perplexity than the baseline methods\. These results demonstrate the effectiveness of the proposed Muon\-based compositional updates for Transformer language models with many matrix\-valued parameters\. In particular, FedCoMuon\-VR achieves the best overall performance\.

\(a\)χ=0\.3\\chi=0\.3\(b\)χ=0\.5\\chi=0\.5\(c\)χ=0\.7\\chi=0\.7
Figure 4:Test accuracy and loss of each method on the task\-distributed CNN meta\-learning task with heterogeneous CIFAR\-10 data forχ∈\{0\.3,0\.5,0\.7\}\\chi\\in\\\{0\.3,0\.5,0\.7\\\}\.Figure 5:Effect of the synchronization gapτ\\tauon task\-distributed CNN meta learning with heterogeneous CIFAR\-10 data\.Figure 6:Test accuracy and loss of each method on the task\-distributed ViT\-Tiny meta learning task with heterogeneous CIFAR\-10 forχ=0\.3\\chi=0\.3\.

### 6\.2Task\-Distributed Meta Learning Problem

In this experiment, we conduct task\-distributed meta learning experiments on the CIFAR\-10\([Krizhevsky 2009](https://arxiv.org/html/2608.12710#bib.bib21)\)dataset\. We first consider a 7\-layer CNN as a standard setting for comparing federated compositional optimization methods under different levels of data heterogeneity\. We further extend the evaluation to ViT\-Tiny\([Touvron et al\. 2021](https://arxiv.org/html/2608.12710#bib.bib40)\), a matrix\-wise vision model based on the Transformer architecture and containing many matrix\-valued parameter blocks\. Specifically, we optimize the following risk\-sensitive compositional MAML objective:

minW∈ℝm×n1K∑k=1Kexp\(ℓk\(W−η∇ℓk\(W\)\)/λ\),\\min\_\{W\\in\\mathbb\{R\}^\{m\\times n\}\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\exp\\left\(\\ell^\{k\}\\left\(W\-\\eta\\nabla\\ell^\{k\}\(W\)\\right\)/\\lambda\\right\),\(7\)whereℓk\\ell^\{k\}denotes the loss function on clientkk,η\>0\\eta\>0is the inner\-loop learning rate, andλ\>0\\lambda\>0is a regularization parameter controlling the degree of risk sensitivity\.

#### 6\.2\.1CNN\-Based Meta Learning

For CNN\-based meta learning, we conduct experiments on CIFAR\-10 using a 7\-layer CNN in a federated system with 10 clients and one central server\. Each client is assigned a distinct dominant class, where aχ\\chifraction of its local samples belongs to the dominant class and a\(1−χ\)/9\(1\-\\chi\)/9fraction belongs to each of the remaining classes\. We evaluateχ∈\{0\.3,0\.5,0\.7\}\\chi\\in\\\{0\.3,0\.5,0\.7\\\}\. For FedCoMuon and FedCoMuon\-VR, the inner\- and outer\-loop learning rates are set to0\.030\.03and0\.10\.1respectively\. The regularization parameter is set toλ=0\.5\\lambda=0\.5, the synchronization gap is set toτ=5\\tau=5, and the total number of training iterations is set to600600\. Additional implementation details and complete hyperparameter settings are provided in the Appendix[C](https://arxiv.org/html/2608.12710#A3)\.

As shown in Figure[4](https://arxiv.org/html/2608.12710#S6.F4), FedCoMuon and FedCoMuon\-VR achieve better overall performance than the baseline methods under different levels of data heterogeneity\. Asχ\\chiincreases, the optimization problem becomes more challenging, while the relative advantage of our methods over the baseline methods becomes more pronounced\. Figure[5](https://arxiv.org/html/2608.12710#S6.F5)further shows the sensitivity of our methods to different synchronization gaps, withτ=1\\tau=1yielding the best performance\.

#### 6\.2\.2ViT\-Tiny\-Based Meta Learning

For task\-distributed meta learning with ViT\-Tiny, we conduct experiments on CIFAR\-10 using a ViT\-Tiny model with 12 Transformer blocks, a hidden dimension of 192, three attention heads, and a patch size of 4\. We adopt the same federated setting and dominant\-class data partition as in the CNN experiments and setχ=0\.3\\chi=0\.3\. For both FedCoMuon and FedCoMuon\-VR, the inner\- and outer\-loop learning rates are set to0\.0050\.005and0\.010\.01respectively\. The total number of training iterations is set to15001500\. Additional implementation details and complete hyperparameter settings are provided in the Appendix[C](https://arxiv.org/html/2608.12710#A3)\.

As shown in Figure[6](https://arxiv.org/html/2608.12710#S6.F6), FedCoMuon and FedCoMuon\-VR achieve substantially higher test accuracy and lower test loss than FedMAML and the federated compositional baselines\. They also remain competitive with the three FedMuon baselines throughout training\. These results demonstrate the effectiveness of our compositional Muon methods for task\-distributed meta learning with the Transformer\-based ViT\-Tiny model\.

## 7Conclusion

In this paper, we studied the matrix\-wise composition optimization, and proposed a class of effective federated compositional Muon algorithms \(i\.e\., FedCoMuon and FedCoMuon\-VR\), which build on compositional gradient tracking and orthogonalized momentum\. In theory, we established convergence guarantees under non\-i\.i\.d\. and non\-convex settings\. In particular, our FedCoMuon\-VR algorithm achieves a lower sample complexity ofO⁡\(ϵ−3\)O\(\\epsilon^\{\-3\}\)for finding anϵ\\epsilon\-stationary solution than the existing FedMuon algorithms\. Extensive experiments on robust federated learning and the task\-distributed meta learning problem demonstrate the effectiveness of our proposed algorithms\.

## References

- Chang et al\. \[2025\]Da Chang, Yongxiang Liu, and Ganzhao Yuan\.On the convergence of muon and beyond\.*arXiv preprint arXiv:2509\.15816*, 2025\.
- Chen et al\. \[2021\]Tianyi Chen, Yuejiao Sun, and Wotao Yin\.Solving stochastic compositional optimization is nearly as easy as solving stochastic optimization\.*IEEE Transactions on Signal Processing*, 69:4937–4948, 2021\.
- Cheng et al\. \[2024\]Ziheng Cheng, Xinmeng Huang, Pengfei Wu, and Kun Yuan\.Momentum benefits non\-IID federated learning simply and provably\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Cutkosky and Orabona \[2019\]Ashok Cutkosky and Francesco Orabona\.Momentum\-based variance reduction in non\-convex SGD\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2019\.
- Fallah et al\. \[2020\]Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar\.Personalized federated learning: A meta\-learning approach\.*arXiv preprint arXiv:2002\.07948*, 2020\.
- Gao \[2024\]Hongchang Gao\.A doubly recursive stochastic compositional gradient descent method for federated multi\-level compositional optimization\.In*Proceedings of the 41st International Conference on Machine Learning*, pages 14540–14610, 2024\.
- Gao et al\. \[2022\]Hongchang Gao, Junchi Li, and Heng Huang\.On the convergence of local stochastic compositional gradient descent with momentum\.In*International Conference on Machine Learning \(ICML\)*, 2022\.
- Ghadimi et al\. \[2020\]Saeed Ghadimi, Andrzej Ruszczynski, and Mengdi Wang\.A single timescale stochastic approximation method for nested stochastic optimization\.*SIAM Journal on Optimization*, 30\(1\):960–979, 2020\.
- Huang and Li \[2021\]Feihu Huang and Jian Li\.Compositional federated learning: Applications in distributionally robust averaging and meta learning\.*arXiv preprint arXiv:2106\.11264*, 2021\.
- Huang et al\. \[2025\]Feihu Huang, Yuning Luo, and Songcan Chen\.LiMuon: Light and fast muon optimizer for large models\.*arXiv preprint arXiv:2509\.14562*, 2025\.
- Huang et al\. \[2026\]Feihu Huang, Xinrui Wang, Siqi Zhang, and Soncan Chen\.Faster adaptive momentum\-based federated methods for distributed composition optimization\.*Machine Learning*, 115\(5\):106, 2026\.
- Jiang et al\. \[2022\]Wei Jiang, Bokun Wang, Yibo Wang, Lijun Zhang, and Tianbao Yang\.Optimal algorithms for stochastic multi\-level compositional optimization\.In*International Conference on Machine Learning*, pages 10195–10216\. PMLR, 2022\.
- Jin et al\. \[2022\]Hao Jin, Yang Peng, Wenhao Yang, Shusen Wang, and Zhihua Zhang\.Federated reinforcement learning with environment heterogeneity\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, volume 151, pages 18–37\. PMLR, 2022\.
- Jordan et al\. \[2024\]Keller Jordan, Yuchen Jin, Vlad Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein\.Muon: An optimizer for hidden layers in neural networks\.[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/), 2024\.
- Kairouz et al\. \[2019\]Peter Kairouz, H\. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al\.Advances and open problems in federated learning\.*arXiv preprint arXiv:1912\.04977*, 2019\.
- Karimireddy et al\. \[2020\]Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh\.SCAFFOLD: Stochastic controlled averaging for federated learning\.In*International Conference on Machine Learning \(ICML\)*, pages 5132–5143\. PMLR, 2020\.
- Khanduri et al\. \[2021\]Prashant Khanduri, Pranay Sharma, Haibo Yang, Mingyi Hong, Jia Liu, Ketan Rajawat, and Pramod K\. Varshney\.STEM: A stochastic two\-sided momentum algorithm achieving near\-optimal sample and communication complexities for federated learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 34, pages 6050–6061, 2021\.
- Khanduri et al\. \[2026\]Prashant Khanduri, Chengyin Li, Rafi Ibn Sultan, Aditi Sarker, Yao Qiang, Joerg Kliewer, and Dongxiao Zhu\.On federated compositional optimization: Algorithms, analysis, and guarantees\.*Transactions on Machine Learning Research*, 2026\.ISSN 2835\-8856\.URL[https://openreview\.net/forum?id=4uRlbSNevR](https://openreview.net/forum?id=4uRlbSNevR)\.
- Kim and Oh \[2026\]Gyu Yeol Kim and Min\-hwan Oh\.Convergence of muon with newton\-schulz\.*arXiv preprint arXiv:2601\.19156*, 2026\.
- Kovalev \[2025\]Dmitry Kovalev\.Understanding gradient orthogonalization for deep learning via non\-euclidean trust\-region optimization\.*arXiv preprint arXiv:2503\.12645*, 2025\.
- Krizhevsky \[2009\]Alex Krizhevsky\.Learning multiple layers of features from tiny images\.Technical report, University of Toronto, 2009\.
- LeCun et al\. \[1998\]Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner\.Gradient\-based learning applied to document recognition\.*Proceedings of the IEEE*, 86\(11\):2278–2324, 1998\.
- LeCun et al\. \[2015\]Yann LeCun, Yoshua Bengio, and Geoffrey Hinton\.Deep learning\.*nature*, 521\(7553\):436–444, 2015\.
- Li and Hong \[2025\]Jiaxiang Li and Mingyi Hong\.A note on the convergence of muon and further\.*arXiv preprint arXiv:2502\.02900*, 2025\.
- Li et al\. \[2021\]Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He\.A survey on federated learning systems: Vision, hype and reality for data privacy and protection\.*IEEE Transactions on Knowledge and Data Engineering*, 35\(4\):3347–3366, 2021\.
- Li et al\. \[2020\]Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith\.Federated optimization in heterogeneous networks\.*Proceedings of Machine learning and systems*, 2:429–450, 2020\.
- Liu et al\. \[2025a\]Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, et al\.Muon is scalable for LLM training\.*arXiv preprint arXiv:2502\.16982*, 2025a\.
- Liu et al\. \[2025b\]Junkang Liu, Fanhua Shang, Junchao Zhou, Hongying Liu, Yuanyuan Liu, and Jin Liu\.FedMuon: Accelerating federated learning with matrix orthogonalization\.*arXiv preprint arXiv:2510\.27403*, 2025b\.
- McMahan et al\. \[2017\]Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas\.Communication\-efficient learning of deep networks from decentralized data\.In*Artificial Intelligence and Statistics \(AISTATS\)*, pages 1273–1282\. PMLR, 2017\.
- Merity et al\. \[2016\]Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher\.Pointer sentinel mixture models\.*arXiv preprint arXiv:1609\.07843*, 2016\.
- Pethick et al\. \[2025\]Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti\-Falls, and Volkan Cevher\.Training deep learning models with norm\-constrained LMOs\.In*International Conference on Machine Learning \(ICML\)*, 2025\.
- Qian et al\. \[2025\]Xun Qian, Hussein Rammal, Dmitry Kovalev, and Peter Richtarik\.Muon is provably faster with momentum variance reduction\.*arXiv preprint arXiv:2512\.16598*, 2025\.
- Qian et al\. \[2026\]Xun Qian, Alexander Gaponov, Grigory Malinovsky, and Peter Richtárik\.Communication\-efficient gluon in federated learning\.*arXiv preprint arXiv:2604\.10689*, 2026\.
- Reddi et al\. \[2020\]Sashank J\. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H\. Brendan McMahan\.Adaptive federated optimization\.*arXiv preprint arXiv:2003\.00295*, 2020\.
- Riabinin et al\. \[2025\]Artem Riabinin, Egor Shulgin, Kaja Gruntkowska, and Peter Richtárik\.Gluon: Making muon & scion great again\!\(bridging theory and practice of lmo\-based optimizers for llms\)\.*arXiv preprint arXiv:2505\.13416*, 2025\.
- Sfyraki and Wang \[2025\]Maria\-Eleni Sfyraki and Jun\-Kun Wang\.Lions and muons: Optimization via stochastic frank\-wolfe\.*arXiv preprint arXiv:2506\.04192*, 2025\.
- Shen et al\. \[2025\]Wei Shen, Ruichuan Huang, Minhui Huang, Cong Shen, and Jiawei Zhang\.On the convergence analysis of muon\.*arXiv preprint arXiv:2505\.23737*, 2025\.
- Takezawa et al\. \[2026\]Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang, and Sebastian U\. Stich\.FedMuon: Federated learning with bias\-corrected LMO\-based optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.
- Tarzanagh et al\. \[2022\]Davoud Ataee Tarzanagh, Mingchen Li, Christos Thrampoulidis, and Samet Oymak\.FedNest: Federated bilevel, minimax, and compositional optimization\.In*International Conference on Machine Learning \(ICML\)*, 2022\.
- Touvron et al\. \[2021\]Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou\.Training data\-efficient image transformers & distillation through attention\.In*Proceedings of the 38th International Conference on Machine Learning*, volume 139, pages 10347–10357\. PMLR, 2021\.
- Vaswani et al\. \[2017\]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention is all you need\.*Advances in neural information processing systems*, 30, 2017\.
- Wang et al\. \[2023\]Bingcong Wang, Zezhou Yuan, Yiming Ying, and Tianbao Yang\.Memory\-based optimization methods for model\-agnostic meta\-learning and personalized federated learning\.*Journal of Machine Learning Research*, 24\(145\):1–46, 2023\.
- Wang et al\. \[2017\]Mengdi Wang, Ethan X Fang, and Han Liu\.Stochastic compositional gradient descent: algorithms for minimizing compositions of expected\-value functions\.*Mathematical Programming*, 161\(1\):419–449, 2017\.
- Yan et al\. \[2025\]Wenjing Yan, Kai Zhang, Xiaolu Wang, and Xuanyu Cao\.Problem\-parameter\-free federated learning\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Yang \[2026\]Tianbao Yang\.Compositional optimization for advanced machine learning, 2026\.URL[https://opt4ml\.org/](https://opt4ml.org/)\.Book preprint\.
- Zhang and Xiao \[2021\]Junyu Zhang and Lin Xiao\.MultiLevel composite stochastic optimization via nested variance reduction\.*SIAM Journal on Optimization*, 31\(2\):1131–1157, 2021\.
- Zhang and Gao \[2025\]Xinwen Zhang and Hongchang Gao\.On provable benefits of muon in federated learning\.*arXiv preprint arXiv:2510\.03866*, 2025\.

## Appendix AConvergence Analysis of our FedCoMuon Algorithm

In this section, we provide the detailed convergence analysis of FedCoMuon under the assumptions stated in the main paper\. We first introduce the following notation:W¯t=1K​∑k=1KWtk\\bar\{W\}\_\{t\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}W\_\{t\}^\{k\}andM¯t=1K​∑k=1KMtk\\bar\{M\}\_\{t\}=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}M\_\{t\}^\{k\}\. Moreover,F⁡\(W\)=1K​∑k=1Kfk​\(gk​\(W\)\)F\(W\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}f^\{k\}\(g^\{k\}\(W\)\), and

∇F​\(W\)=1K​∑k=1K∇gk​\(W\)​\(∇yfk​\(gk​\(W\)\)⊗In\)\.\\nabla F\(W\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\nabla g^\{k\}\(W\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\\otimes I\_\{n\}\\bigr\)\.We next establish several auxiliary lemmas used in the convergence analysis\.

###### Lemma 1\.

Given Assumptions[3](https://arxiv.org/html/2608.12710#Thmassumption3),[2](https://arxiv.org/html/2608.12710#Thmassumption2), and[4](https://arxiv.org/html/2608.12710#Thmassumption4), for each clientk∈\[K\]k\\in\[K\], the local compositional objectiveFk​\(W\)=fk​\(gk​\(W\)\)F^\{k\}\(W\)=f^\{k\}\(g^\{k\}\(W\)\)isLFL\_\{F\}\-smooth, i\.e\., for anyW1,W2∈ℝm×nW\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{m\\times n\},

∥∇Fk​\(W1\)−∇Fk​\(W2\)∥F≤LF​‖W1−W2‖F,\\lVert\\nabla F^\{k\}\(W\_\{1\}\)\-\\nabla F^\{k\}\(W\_\{2\}\)\\rVert\_\{F\}\\leq L\_\{F\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\},\(8\)whereLF=Cf​Lg\+Cg2​LfL\_\{F\}=C\_\{f\}L\_\{g\}\+C\_\{g\}^\{2\}L\_\{f\}\. Consequently,F⁡\(W\)=K−1​∑k=1KFk​\(W\)F\(W\)=K^\{\-1\}\\sum\_\{k=1\}^\{K\}F^\{k\}\(W\)is alsoLFL\_\{F\}\-smooth\.

###### Proof\.

Jensen’s inequality and Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[3](https://arxiv.org/html/2608.12710#Thmassumption3)imply‖∇gk​\(W\)‖F≤Cg\\\|\\nabla g^\{k\}\(W\)\\\|\_\{F\}\\leq C\_\{g\}and‖∇yfk​\(y\)‖≤Cf\\\|\\nabla\_\{y\}f^\{k\}\(y\)\\\|\\leq C\_\{f\}\. Consequently,gkg^\{k\}isCgC\_\{g\}\-Lipschitz, i\.e\.,‖gk​\(W1\)−gk​\(W2\)‖≤Cg​‖W1−W2‖F\\\|g^\{k\}\(W\_\{1\}\)\-g^\{k\}\(W\_\{2\}\)\\\|\\leq C\_\{g\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\}\. For anyW1,W2∈ℝm×nW\_\{1\},W\_\{2\}\\in\\mathbb\{R\}^\{m\\times n\}, we have

∥∇Fk​\(W1\)−∇Fk​\(W2\)∥F\\displaystyle\\lVert\\nabla F^\{k\}\(W\_\{1\}\)\-\\nabla F^\{k\}\(W\_\{2\}\)\\rVert\_\{F\}=∥∇gk​\(W1\)​\(∇yfk​\(gk​\(W1\)\)⊗In\)−∇gk​\(W2\)​\(∇yfk​\(gk​\(W2\)\)⊗In\)∥F\\displaystyle=\\lVert\\nabla g^\{k\}\(W\_\{1\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{1\}\)\)\\otimes I\_\{n\}\\right\)\-\\nabla g^\{k\}\(W\_\{2\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}=∥∇gk\(W1\)\(∇yfk\(gk\(W1\)\)⊗In\)−∇gk\(W1\)\(∇yfk\(gk\(W2\)\)⊗In\)\\displaystyle=\\lVert\\nabla g^\{k\}\(W\_\{1\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{1\}\)\)\\otimes I\_\{n\}\\right\)\-\\nabla g^\{k\}\(W\_\{1\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\otimes I\_\{n\}\\right\)\+∇gk\(W1\)\(∇yfk\(gk\(W2\)\)⊗In\)−∇gk\(W2\)\(∇yfk\(gk\(W2\)\)⊗In\)∥F\\displaystyle\\hskip 9\.24994pt\+\\nabla g^\{k\}\(W\_\{1\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\otimes I\_\{n\}\\right\)\-\\nabla g^\{k\}\(W\_\{2\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}≤∥∇gk​\(W1\)​\(\(∇yfk​\(gk​\(W1\)\)−∇yfk​\(gk​\(W2\)\)\)⊗In\)∥F\\displaystyle\\leq\\lVert\\nabla g^\{k\}\(W\_\{1\}\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{1\}\)\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\right\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}\+∥\(∇gk​\(W1\)−∇gk​\(W2\)\)​\(∇yfk​\(gk​\(W2\)\)⊗In\)∥F\\displaystyle\\hskip 9\.24994pt\+\\lVert\\left\(\\nabla g^\{k\}\(W\_\{1\}\)\-\\nabla g^\{k\}\(W\_\{2\}\)\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}≤‖∇gk​\(W1\)‖F​∥∇yfk​\(gk​\(W1\)\)−∇yfk​\(gk​\(W2\)\)∥\\displaystyle\\leq\\\|\\nabla g^\{k\}\(W\_\{1\}\)\\\|\_\{F\}\\lVert\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{1\}\)\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\rVert\+‖∇gk​\(W1\)−∇gk​\(W2\)‖F​∥∇yfk​\(gk​\(W2\)\)∥\\displaystyle\\hskip 9\.24994pt\+\\\|\\nabla g^\{k\}\(W\_\{1\}\)\-\\nabla g^\{k\}\(W\_\{2\}\)\\\|\_\{F\}\\lVert\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{2\}\)\)\\rVert≤Cg​Lf​‖gk​\(W1\)−gk​\(W2\)‖\+Cf​Lg​‖W1−W2‖F\\displaystyle\\leq C\_\{g\}L\_\{f\}\\\|g^\{k\}\(W\_\{1\}\)\-g^\{k\}\(W\_\{2\}\)\\\|\+C\_\{f\}L\_\{g\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\}≤\(Cg2​Lf\+Cf​Lg\)​‖W1−W2‖F\.\\displaystyle\\leq\\left\(C\_\{g\}^\{2\}L\_\{f\}\+C\_\{f\}L\_\{g\}\\right\)\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\}\.\(9\)
Moreover,

‖∇F​\(W1\)−∇F​\(W2\)‖F\\displaystyle\\\|\\nabla F\(W\_\{1\}\)\-\\nabla F\(W\_\{2\}\)\\\|\_\{F\}=∥1K​∑k=1K\(∇Fk​\(W1\)−∇Fk​\(W2\)\)∥F\\displaystyle=\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{1\}\)\-\\nabla F^\{k\}\(W\_\{2\}\)\\right\)\\rVert\_\{F\}≤1K​∑k=1K∥∇Fk​\(W1\)−∇Fk​\(W2\)∥F\\displaystyle\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\nabla F^\{k\}\(W\_\{1\}\)\-\\nabla F^\{k\}\(W\_\{2\}\)\\rVert\_\{F\}≤LF​‖W1−W2‖F,\\displaystyle\\leq L\_\{F\}\\\|W\_\{1\}\-W\_\{2\}\\\|\_\{F\},\(10\)where we used Assumptions[3](https://arxiv.org/html/2608.12710#Thmassumption3)and[4](https://arxiv.org/html/2608.12710#Thmassumption4), the triangle inequality, and the submultiplicativity of matrix norms\. ∎

###### Lemma 2\.

LetM∈ℝm×nM\\in\\mathbb\{R\}^\{m\\times n\}have compact singular value decompositionM=U​Σ​V⊤M=U\\Sigma V^\{\\top\}\. Then

‖U​V⊤‖F≤n,⟨M,U​V⊤⟩=‖M‖∗\.\\\|UV^\{\\top\}\\\|\_\{F\}\\leq\\sqrt\{n\},\\hskip 18\.49988pt\\langle M,UV^\{\\top\}\\rangle=\\\|M\\\|\_\{\*\}\.\(11\)Moreover, for anyA,B∈ℝm×nA,B\\in\\mathbb\{R\}^\{m\\times n\}, ifB=UB​ΣB​VB⊤B=U\_\{B\}\\Sigma\_\{B\}V\_\{B\}^\{\\top\}is a compact singular value decomposition, then

⟨A,UB​VB⊤⟩≥‖A‖F−2​n​‖A−B‖F\.\\langle A,U\_\{B\}V\_\{B\}^\{\\top\}\\rangle\\geq\\\|A\\\|\_\{F\}\-2\\sqrt\{n\}\\,\\\|A\-B\\\|\_\{F\}\.\(12\)

###### Proof\.

Letr=rank⁡\(M\)r=\\operatorname\{rank\}\(M\)\. Then

‖U​V⊤‖F2\\displaystyle\\\|UV^\{\\top\}\\\|\_\{F\}^\{2\}=tr⁡\(\(U​V⊤\)⊤​U​V⊤\)=tr⁡\(V​U⊤​U​V⊤\)=tr⁡\(V​V⊤\)=tr⁡\(V⊤​V\)=r≤n,\\displaystyle=\\operatorname\{tr\}\\\!\\left\(\(UV^\{\\top\}\)^\{\\top\}UV^\{\\top\}\\right\)=\\operatorname\{tr\}\\\!\\left\(VU^\{\\top\}UV^\{\\top\}\\right\)=\\operatorname\{tr\}\\\!\\left\(VV^\{\\top\}\\right\)=\\operatorname\{tr\}\\\!\\left\(V^\{\\top\}V\\right\)=r\\leq n,\(13\)and

⟨M,U​V⊤⟩\\displaystyle\\langle M,UV^\{\\top\}\\rangle=tr⁡\(M⊤​U​V⊤\)\\displaystyle=\\operatorname\{tr\}\\\!\\left\(M^\{\\top\}UV^\{\\top\}\\right\)=tr⁡\(\(U​Σ​V⊤\)⊤​U​V⊤\)\\displaystyle=\\operatorname\{tr\}\\\!\\left\(\(U\\Sigma V^\{\\top\}\)^\{\\top\}UV^\{\\top\}\\right\)=tr⁡\(V​Σ​U⊤​U​V⊤\)=tr⁡\(V​Σ​V⊤\)\\displaystyle=\\operatorname\{tr\}\\\!\\left\(V\\Sigma U^\{\\top\}UV^\{\\top\}\\right\)=\\operatorname\{tr\}\\\!\\left\(V\\Sigma V^\{\\top\}\\right\)=tr⁡\(Σ​V⊤​V\)=tr⁡\(Σ\)=‖M‖∗\.\\displaystyle=\\operatorname\{tr\}\\\!\\left\(\\Sigma V^\{\\top\}V\\right\)=\\operatorname\{tr\}\(\\Sigma\)=\\\|M\\\|\_\{\*\}\.\(14\)For anyA,B∈ℝm×nA,B\\in\\mathbb\{R\}^\{m\\times n\},

⟨A,UB​VB⊤⟩\\displaystyle\\langle A,U\_\{B\}V\_\{B\}^\{\\top\}\\rangle=⟨B,UB​VB⊤⟩\+⟨A−B,UB​VB⊤⟩\\displaystyle=\\langle B,U\_\{B\}V\_\{B\}^\{\\top\}\\rangle\+\\langle A\-B,U\_\{B\}V\_\{B\}^\{\\top\}\\rangle≥‖B‖∗−‖A−B‖F​‖UB​VB⊤‖F\\displaystyle\\geq\\\|B\\\|\_\{\*\}\-\\\|A\-B\\\|\_\{F\}\\\|U\_\{B\}V\_\{B\}^\{\\top\}\\\|\_\{F\}≥‖B‖F−n​‖A−B‖F\\displaystyle\\geq\\\|B\\\|\_\{F\}\-\\sqrt\{n\}\\,\\\|A\-B\\\|\_\{F\}≥‖A‖F−‖A−B‖F−n​‖A−B‖F\\displaystyle\\geq\\\|A\\\|\_\{F\}\-\\\|A\-B\\\|\_\{F\}\-\\sqrt\{n\}\\,\\\|A\-B\\\|\_\{F\}=‖A‖F−\(1\+n\)​‖A−B‖F\\displaystyle=\\\|A\\\|\_\{F\}\-\(1\+\\sqrt\{n\}\)\\\|A\-B\\\|\_\{F\}≥‖A‖F−2​n​‖A−B‖F,\\displaystyle\\geq\\\|A\\\|\_\{F\}\-2\\sqrt\{n\}\\,\\\|A\-B\\\|\_\{F\},\(15\)where we used the compact SVD, the Cauchy–Schwarz inequality, and the standard nuclear–Frobenius norm relations\. ∎

###### Lemma 3\.

Let\{Wtk,utk,Mtk\}\\\{W\_\{t\}^\{k\},u\_\{t\}^\{k\},M\_\{t\}^\{k\}\\\}be generated by Algorithm[1](https://arxiv.org/html/2608.12710#alg1)\. Given Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[3](https://arxiv.org/html/2608.12710#Thmassumption3), for anyT,τ\>0T,\\tau\>0and0<α<10<\\alpha<1satisfyingα​τ≤1\\alpha\\tau\\leq 1, we have

1T​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2≤4α​T​1K​∑k=1K𝔼​‖u0k−gk​\(W0k\)‖2\+86​Cg2​nα2​η2\+86​α​σg2\.\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\\leq\\frac\{4\}\{\\alpha T\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\.\(16\)

###### Proof\.

Within each communication block, the update rule, Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[3](https://arxiv.org/html/2608.12710#Thmassumption3), Lemma[2](https://arxiv.org/html/2608.12710#Thmlemma2), and Young’s inequality give

𝔼​∥ut\+1k−gk​\(Wt\+1k\)∥2\\displaystyle\\mathbb\{E\}\\lVert u\_\{t\+1\}^\{k\}\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\rVert^\{2\}=𝔼​∥\(1−α\)​\(utk−gk​\(Wtk\)\)\+\(1−α\)​\(gk​\(Wtk\)−gk​\(Wt\+1k\)\)\+α⁡\(gk​\(Wt\+1k,ξt\+1k\)−gk​\(Wt\+1k\)\)∥2\\displaystyle=\\mathbb\{E\}\\lVert\(1\-\\alpha\)\\left\(u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\right\)\+\(1\-\\alpha\)\\left\(g^\{k\}\(W\_\{t\}^\{k\}\)\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\right\)\+\\alpha\\left\(g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\right\)\\rVert^\{2\}≤\(1−α\)​𝔼​∥utk−gk​\(Wtk\)∥2\+1α​𝔼​∥gk​\(Wtk\)−gk​\(Wt\+1k\)∥2\+α2​σg2\\displaystyle\\leq\(1\-\\alpha\)\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\+\\frac\{1\}\{\\alpha\}\\mathbb\{E\}\\lVert g^\{k\}\(W\_\{t\}^\{k\}\)\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\rVert^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}≤\(1−α\)​𝔼​∥utk−gk​\(Wtk\)∥2\+Cg2α​𝔼​∥Wt\+1k−Wtk∥F2\+α2​σg2\\displaystyle\\leq\(1\-\\alpha\)\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\+\\frac\{C\_\{g\}^\{2\}\}\{\\alpha\}\\mathbb\{E\}\\lVert W\_\{t\+1\}^\{k\}\-W\_\{t\}^\{k\}\\rVert\_\{F\}^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}=\(1−α\)​𝔼​∥utk−gk​\(Wtk\)∥2\+Cg2​η2α​𝔼​∥Utk​\(Vtk\)⊤∥F2\+α2​σg2\\displaystyle=\(1\-\\alpha\)\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\+\\frac\{C\_\{g\}^\{2\}\\eta^\{2\}\}\{\\alpha\}\\mathbb\{E\}\\lVert U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}≤\(1−α\)​𝔼​∥utk−gk​\(Wtk\)∥2\+Cg2​nα​η2\+α2​σg2\.\\displaystyle\\leq\(1\-\\alpha\)\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\+\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\.\(17\)Lets=q​τs=q\\taube the beginning of a block\. Iterating \([17](https://arxiv.org/html/2608.12710#A1.E17)\) within the block gives

∑ℓ=0τ−11K​∑k=1K𝔼​‖us\+ℓk−gk​\(Ws\+ℓk\)‖2\\displaystyle\\sum\_\{\\ell=0\}^\{\\tau\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\+\\ell\}^\{k\}\-g^\{k\}\(W\_\{s\+\\ell\}^\{k\}\)\\\|^\{2\}≤∑ℓ=0τ−1\[\(1−α\)ℓ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+∑i=0ℓ−1\(1−α\)i​\(Cg2​nα​η2\+α2​σg2\)\]\\displaystyle\\leq\\sum\_\{\\ell=0\}^\{\\tau\-1\}\\Bigg\[\(1\-\\alpha\)^\{\\ell\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\sum\_\{i=0\}^\{\\ell\-1\}\(1\-\\alpha\)^\{i\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\\Bigg\]≤τ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+τ2​\(Cg2​nα​η2\+α2​σg2\),\\displaystyle\\leq\\tau\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\tau^\{2\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\),\(18\)1K​∑k=1K𝔼​‖us\+τk−gk​\(Ws\+τk\)‖2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\+\\tau\}^\{k\}\-g^\{k\}\(W\_\{s\+\\tau\}^\{k\}\)\\\|^\{2\}≤\(1−α\)τ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+∑i=0τ−1\(1−α\)i​\(Cg2​nα​η2\+α2​σg2\)\\displaystyle\\leq\(1\-\\alpha\)^\{\\tau\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\sum\_\{i=0\}^\{\\tau\-1\}\(1\-\\alpha\)^\{i\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)≤\(1−α\)τ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+τ⁡\(Cg2​nα​η2\+α2​σg2\)\.\\displaystyle\\leq\(1\-\\alpha\)^\{\\tau\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\tau\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\.\(19\)At the block end, the server replacesWs\+τkW\_\{s\+\\tau\}^\{k\}withW¯s\+τ\\bar\{W\}\_\{s\+\\tau\}while leavingus\+τku\_\{s\+\\tau\}^\{k\}unchanged\. Young’s inequality with parameterα​τ/2\\alpha\\tau/2gives

1K​∑k=1K𝔼​‖us\+τk−gk​\(W¯s\+τ\)‖2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\+\\tau\}^\{k\}\-g^\{k\}\(\\bar\{W\}\_\{s\+\\tau\}\)\\\|^\{2\}≤\(1\+α​τ2\)​1K​∑k=1K𝔼​‖us\+τk−gk​\(Ws\+τk\)‖2\+\(1\+2α​τ\)​1K​∑k=1K𝔼​‖gk​\(Ws\+τk\)−gk​\(W¯s\+τ\)‖2\\displaystyle\\leq\\left\(1\+\\frac\{\\alpha\\tau\}\{2\}\\right\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\+\\tau\}^\{k\}\-g^\{k\}\(W\_\{s\+\\tau\}^\{k\}\)\\\|^\{2\}\+\\left\(1\+\\frac\{2\}\{\\alpha\\tau\}\\right\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|g^\{k\}\(W\_\{s\+\\tau\}^\{k\}\)\-g^\{k\}\(\\bar\{W\}\_\{s\+\\tau\}\)\\\|^\{2\}≤\(1\+α​τ2\)​\(1−α\)τ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+32​τ​\(Cg2​nα​η2\+α2​σg2\)\\displaystyle\\leq\\left\(1\+\\frac\{\\alpha\\tau\}\{2\}\\right\)\(1\-\\alpha\)^\{\\tau\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\frac\{3\}\{2\}\\tau\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\+\(1\+2α​τ\)Cg2η2K∑k=1K𝔼∥∑i=ss\+τ−1\(Uik\(Vik\)⊤−1K∑j=1KUij\(Vij\)⊤\)∥F2\\displaystyle\\hskip 9\.24994pt\+\\left\(1\+\\frac\{2\}\{\\alpha\\tau\}\\right\)C\_\{g\}^\{2\}\\frac\{\\eta^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\sum\_\{i=s\}^\{s\+\\tau\-1\}\\left\(U\_\{i\}^\{k\}\(V\_\{i\}^\{k\}\)^\{\\top\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\right\)\\rVert\_\{F\}^\{2\}≤\(1\+α​τ2\)​\(1−α\)τ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+32​τ​\(Cg2​nα​η2\+α2​σg2\)\+4​\(1\+2α​τ\)​Cg2​η2​τ2​n\\displaystyle\\leq\\left\(1\+\\frac\{\\alpha\\tau\}\{2\}\\right\)\(1\-\\alpha\)^\{\\tau\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\frac\{3\}\{2\}\\tau\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\+4\\left\(1\+\\frac\{2\}\{\\alpha\\tau\}\\right\)C\_\{g\}^\{2\}\\eta^\{2\}\\tau^\{2\}n≤\(1\+α​τ2\)​e−α​τ​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+32​τ​\(Cg2​nα​η2\+α2​σg2\)\+12​τ​Cg2​nα​η2\\displaystyle\\leq\\left\(1\+\\frac\{\\alpha\\tau\}\{2\}\\right\)e^\{\-\\alpha\\tau\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+\\frac\{3\}\{2\}\\tau\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\+\\frac\{12\\tau C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}≤\(1−α​τ3\)​1K​∑k=1K𝔼​‖usk−gk​\(Wsk\)‖2\+14​τ​\(Cg2​nα​η2\+α2​σg2\),\\displaystyle\\leq\\left\(1\-\\frac\{\\alpha\\tau\}\{3\}\\right\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{s\}^\{k\}\-g^\{k\}\(W\_\{s\}^\{k\}\)\\\|^\{2\}\+14\\tau\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\),\(20\)LetQ=⌊T/τ⌋Q=\\left\\lfloor T/\\tau\\right\\rfloorandR=T−Q​τ<τR=T\-Q\\tau<\\tau\. Since∑q=0Q−1\(1−α​τ3\)q≤3α​τ\\sum\_\{q=0\}^\{Q\-1\}\\left\(1\-\\frac\{\\alpha\\tau\}\{3\}\\right\)^\{q\}\\leq\\frac\{3\}\{\\alpha\\tau\}, combining \([18](https://arxiv.org/html/2608.12710#A1.E18)\), \([20](https://arxiv.org/html/2608.12710#A1.E20)\), and \([17](https://arxiv.org/html/2608.12710#A1.E17)\), we have

1T​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}=1T​∑q=0Q−1∑ℓ=0τ−11K​∑k=1K𝔼​‖uq​τ\+ℓk−gk​\(Wq​τ\+ℓk\)‖2\+1T​∑ℓ=0R−11K​∑k=1K𝔼​‖uQ​τ\+ℓk−gk​\(WQ​τ\+ℓk\)‖2\\displaystyle=\\frac\{1\}\{T\}\\sum\_\{q=0\}^\{Q\-1\}\\sum\_\{\\ell=0\}^\{\\tau\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{q\\tau\+\\ell\}^\{k\}\-g^\{k\}\(W\_\{q\\tau\+\\ell\}^\{k\}\)\\\|^\{2\}\+\\frac\{1\}\{T\}\\sum\_\{\\ell=0\}^\{R\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{Q\\tau\+\\ell\}^\{k\}\-g^\{k\}\(W\_\{Q\\tau\+\\ell\}^\{k\}\)\\\|^\{2\}≤1T\[τ∑q=0Q−11K∑k=1K𝔼∥uq​τk−gk\(Wq​τk\)∥2\+Qτ2\(Cg2​nαη2\+α2σg2\)\\displaystyle\\leq\\frac\{1\}\{T\}\\Bigg\[\\tau\\sum\_\{q=0\}^\{Q\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{q\\tau\}^\{k\}\-g^\{k\}\(W\_\{q\\tau\}^\{k\}\)\\\|^\{2\}\+Q\\tau^\{2\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\+R1K∑k=1K𝔼∥uQ​τk−gk\(WQ​τk\)∥2\+R2\(Cg2​nαη2\+α2σg2\)\]\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+R\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{Q\\tau\}^\{k\}\-g^\{k\}\(W\_\{Q\\tau\}^\{k\}\)\\\|^\{2\}\+R^\{2\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\\Bigg\]≤1T\[τ\(3α​τ1K∑k=1K𝔼∥u0k−gk\(W0k\)∥2\+42​Qα\(Cg2​nαη2\+α2σg2\)\)\+Qτ2\(Cg2​nαη2\+α2σg2\)\\displaystyle\\leq\\frac\{1\}\{T\}\\Bigg\[\\tau\\Bigg\(\\frac\{3\}\{\\alpha\\tau\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+\\frac\{42Q\}\{\\alpha\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\\Bigg\)\+Q\\tau^\{2\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\+R\(1K∑k=1K𝔼∥u0k−gk\(W0k\)∥2\+42α\(Cg2​nαη2\+α2σg2\)\)\+R2\(Cg2​nαη2\+α2σg2\)\]\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+R\\Bigg\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+\\frac\{42\}\{\\alpha\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\\Bigg\)\+R^\{2\}\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha\}\\eta^\{2\}\+\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\\right\)\\Bigg\]≤1T\[3α1K∑k=1K𝔼∥u0k−gk\(W0k\)∥2\+43T\(Cg2​nα2η2\+ασg2\)\\displaystyle\\leq\\frac\{1\}\{T\}\\Bigg\[\\frac\{3\}\{\\alpha\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+43T\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+\\alpha\\sigma\_\{g\}^\{2\}\\right\)\+1α1K∑k=1K𝔼∥u0k−gk\(W0k\)∥2\+43T\(Cg2​nα2η2\+ασg2\)\]\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+\\frac\{1\}\{\\alpha\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+43T\\left\(\\frac\{C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+\\alpha\\sigma\_\{g\}^\{2\}\\right\)\\Bigg\]=4α​T​1K​∑k=1K𝔼​‖u0k−gk​\(W0k\)‖2\+86​Cg2​nα2​η2\+86​α​σg2\.\\displaystyle=\\frac\{4\}\{\\alpha T\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\.\(21\)The second inequality follows by iterating \([20](https://arxiv.org/html/2608.12710#A1.E20)\), while the last inequality usesQ​τ≤TQ\\tau\\leq T,R<τ≤1/αR<\\tau\\leq 1/\\alpha, andR≤TR\\leq T\.

∎

###### Lemma 4\.

Let\{Wtk,utk,Mtk\}\\\{W\_\{t\}^\{k\},u\_\{t\}^\{k\},M\_\{t\}^\{k\}\\\}be generated by Algorithm[1](https://arxiv.org/html/2608.12710#alg1)\. For each clientk∈\[K\]k\\in\[K\]and iterationt≥0t\\geq 0, define the stochastic compositional gradient estimator

Ztk=∇gk​\(Wtk,ξtk\)​\(∇yfk​\(utk,ζtk\)⊗In\)\.Z\_\{t\}^\{k\}=\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\}^\{k\}\)\\otimes I\_\{n\}\\right\)\.\(22\)Given Assumptions[4](https://arxiv.org/html/2608.12710#Thmassumption4),[3](https://arxiv.org/html/2608.12710#Thmassumption3), and[2](https://arxiv.org/html/2608.12710#Thmassumption2), for anyk∈\[K\]k\\in\[K\]andt≥0t\\geq 0, we have

𝔼​∥Ztk−∇Fk​\(Wtk\)∥F≤Cg​Lf​𝔼​∥utk−gk​\(Wtk\)∥2\+Cg​σf\+Cf​σ∇g\.\\mathbb\{E\}\\lVert Z\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\rVert\_\{F\}\\leq C\_\{g\}L\_\{f\}\\sqrt\{\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\.\(23\)

###### Proof\.

For anyk∈\[K\]k\\in\[K\], we have

𝔼​∥Ztk−∇Fk​\(Wtk\)∥F\\displaystyle\\mathbb\{E\}\\lVert Z\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\rVert\_\{F\}=𝔼​∥∇gk​\(Wtk,ξtk\)​\(∇yfk​\(utk,ζtk\)⊗In\)−∇gk​\(Wtk\)​\(∇yfk​\(gk​\(Wtk\)\)⊗In\)∥F\\displaystyle=\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\}^\{k\}\)\\otimes I\_\{n\}\\right\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}≤𝔼​∥∇gk​\(Wtk,ξtk\)​\(\(∇yfk​\(utk,ζtk\)−∇yfk​\(utk\)\)⊗In\)∥F\\displaystyle\\leq\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}\+𝔼​∥∇gk​\(Wtk,ξtk\)​\(\(∇yfk​\(utk\)−∇yfk​\(gk​\(Wtk\)\)\)⊗In\)∥F\\displaystyle\\hskip 9\.24994pt\+\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\right\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}\+𝔼​∥\(∇gk​\(Wtk,ξtk\)−∇gk​\(Wtk\)\)​\(∇yfk​\(gk​\(Wtk\)\)⊗In\)∥F\\displaystyle\\hskip 9\.24994pt\+\\mathbb\{E\}\\lVert\\left\(\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}≤𝔼​∥∇gk​\(Wtk,ξtk\)∥F2​𝔼​∥∇yfk​\(utk,ζtk\)−∇yfk​\(utk\)∥2\\displaystyle\\leq\\sqrt\{\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\\rVert\_\{F\}^\{2\}\}\\sqrt\{\\mathbb\{E\}\\lVert\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\rVert^\{2\}\}\+Lf​𝔼​∥∇gk​\(Wtk,ξtk\)∥F2​𝔼​∥utk−gk​\(Wtk\)∥2\\displaystyle\\hskip 9\.24994pt\+L\_\{f\}\\sqrt\{\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\\rVert\_\{F\}^\{2\}\}\\sqrt\{\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\}\+Cf​𝔼​∥∇gk​\(Wtk,ξtk\)−∇gk​\(Wtk\)∥F2\\displaystyle\\hskip 9\.24994pt\+C\_\{f\}\\sqrt\{\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert\_\{F\}^\{2\}\}≤Cg​σf\+Cg​Lf​𝔼​∥utk−gk​\(Wtk\)∥2\+Cf​σ∇g,\\displaystyle\\leq C\_\{g\}\\sigma\_\{f\}\+C\_\{g\}L\_\{f\}\\sqrt\{\\mathbb\{E\}\\lVert u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\rVert^\{2\}\}\+C\_\{f\}\\sigma\_\{\\nabla g\},\(24\)where we used Assumptions[4](https://arxiv.org/html/2608.12710#Thmassumption4),[2](https://arxiv.org/html/2608.12710#Thmassumption2), and[3](https://arxiv.org/html/2608.12710#Thmassumption3), together with the Cauchy–Schwarz inequality\. ∎

###### Lemma 5\.

LetW¯t=K−1​∑k=1KWtk\\bar\{W\}\_\{t\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}W\_\{t\}^\{k\}andM¯t=K−1​∑k=1KMtk\\bar\{M\}\_\{t\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{t\}^\{k\}, and letst=τ⁡⌊t/τ⌋s\_\{t\}=\\tau\\lfloor t/\\tau\\rfloordenote the beginning of the communication block containing iterationtt\. Then, for everyt≥0t\\geq 0,

1K​∑k=1K‖Wtk−W¯t‖F≤2​η​τ​n\.\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}\\leq 2\\eta\\tau\\sqrt\{n\}\.\(25\)Moreover, under Assumptions[4](https://arxiv.org/html/2608.12710#Thmassumption4),[2](https://arxiv.org/html/2608.12710#Thmassumption2),[3](https://arxiv.org/html/2608.12710#Thmassumption3), and[5](https://arxiv.org/html/2608.12710#Thmassumption5), for anyT≥1T\\geq 1,

1T​∑t=0T−11K​∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}≤β​τ​\[2​Cg​Lf​\(1T​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\)1/2\+2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\]\.\\displaystyle\\leq\\beta\\tau\\Bigg\[2C\_\{g\}L\_\{f\}\\left\(\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\\right\)^\{1/2\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\Bigg\]\.\(26\)

###### Proof\.

For anyt≥0t\\geq 0, since the last synchronization occurs atst=τ⁡⌊t/τ⌋s\_\{t\}=\\tau\\lfloor t/\\tau\\rfloor, we haveWstk=W¯stW\_\{s\_\{t\}\}^\{k\}=\\bar\{W\}\_\{s\_\{t\}\}and

1K​∑k=1K‖Wtk−W¯t‖F\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}=1K​∑k=1K∥W¯st−η​∑i=stt−11K​∑j=1KUij​\(Vij\)⊤−Wstk\+η​∑i=stt−1Uik​\(Vik\)⊤∥F\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\bar\{W\}\_\{s\_\{t\}\}\-\\eta\\sum\_\{i=s\_\{t\}\}^\{t\-1\}\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\-W\_\{s\_\{t\}\}^\{k\}\+\\eta\\sum\_\{i=s\_\{t\}\}^\{t\-1\}U\_\{i\}^\{k\}\(V\_\{i\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}≤η​1K​∑k=1K∥∑i=stt−1Uik​\(Vik\)⊤−∑i=stt−11K​∑j=1KUij​\(Vij\)⊤∥F\\displaystyle\\leq\\eta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\sum\_\{i=s\_\{t\}\}^\{t\-1\}U\_\{i\}^\{k\}\(V\_\{i\}^\{k\}\)^\{\\top\}\-\\sum\_\{i=s\_\{t\}\}^\{t\-1\}\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\rVert\_\{F\}≤η​1K​∑k=1K∥∑i=stt−1Uik​\(Vik\)⊤∥F\+η​1K​∑k=1K∥∑i=stt−11K​∑j=1KUij​\(Vij\)⊤∥F\\displaystyle\\leq\\eta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\sum\_\{i=s\_\{t\}\}^\{t\-1\}U\_\{i\}^\{k\}\(V\_\{i\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}\+\\eta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\sum\_\{i=s\_\{t\}\}^\{t\-1\}\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\rVert\_\{F\}≤η​1K​∑k=1K∑i=stt−1‖Uik​\(Vik\)⊤‖F\+η​1K​∑k=1K∑i=stt−11K​∑j=1K‖Uij​\(Vij\)⊤‖F\\displaystyle\\leq\\eta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=s\_\{t\}\}^\{t\-1\}\\\|U\_\{i\}^\{k\}\(V\_\{i\}^\{k\}\)^\{\\top\}\\\|\_\{F\}\+\\eta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sum\_\{i=s\_\{t\}\}^\{t\-1\}\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\\|U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\\|\_\{F\}≤2​η​\(t−st\)​n≤2​η​τ​n\.\\displaystyle\\leq 2\\eta\(t\-s\_\{t\}\)\\sqrt\{n\}\\leq 2\\eta\\tau\\sqrt\{n\}\.\(27\)
Under the synchronization convention above,Mstk=M¯stM\_\{s\_\{t\}\}^\{k\}=\\bar\{M\}\_\{s\_\{t\}\}\. Fort=stt=s\_\{t\}, the momentum disagreement is zero\. Fort\>stt\>s\_\{t\}, averaging the momentum recursion over the clients and unrolling it fromsts\_\{t\}give

Mtk−M¯t\\displaystyle M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}=\(1−β\)t−st​\(Mstk−M¯st\)\+∑i=st\+1tβ​\(1−β\)t−i​\(Zik−1K​∑j=1KZij\)\\displaystyle=\(1\-\\beta\)^\{t\-s\_\{t\}\}\\left\(M\_\{s\_\{t\}\}^\{k\}\-\\bar\{M\}\_\{s\_\{t\}\}\\right\)\+\\sum\_\{i=s\_\{t\}\+1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\left\(Z\_\{i\}^\{k\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}Z\_\{i\}^\{j\}\\right\)=∑i=st\+1tβ​\(1−β\)t−i​\(Zik−1K​∑j=1KZij\)\.\\displaystyle=\\sum\_\{i=s\_\{t\}\+1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\left\(Z\_\{i\}^\{k\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}Z\_\{i\}^\{j\}\\right\)\.\(28\)Consequently, the triangle inequality gives, for everyt≥0t\\geq 0,

1K​∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}≤1K​∑k=1K𝔼​∥∑i=st\+1tβ​\(1−β\)t−i​\(Zik−1K​∑j=1KZij\)∥F\\displaystyle\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\sum\_\{i=s\_\{t\}\+1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\left\(Z\_\{i\}^\{k\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}Z\_\{i\}^\{j\}\\right\)\\rVert\_\{F\}≤∑i=st\+1tβ​\(1−β\)t−i​1K​∑k=1K𝔼​∥Zik−1K​∑j=1KZij∥F\.\\displaystyle\\leq\\sum\_\{i=s\_\{t\}\+1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert Z\_\{i\}^\{k\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}Z\_\{i\}^\{j\}\\rVert\_\{F\}\.\(29\)For anyii,

1K​∑k=1K𝔼​∥Zik−1K​∑j=1KZij∥F\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert Z\_\{i\}^\{k\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}Z\_\{i\}^\{j\}\\rVert\_\{F\}≤2K​∑k=1K𝔼​∥Zik−∇Fk​\(Wik\)∥F\+1K​∑k=1K𝔼​∥∇Fk​\(Wik\)−1K​∑j=1K∇Fj​\(Wij\)∥F\\displaystyle\\leq\\frac\{2\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert Z\_\{i\}^\{k\}\-\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\\rVert\_\{F\}\+\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\nabla F^\{j\}\(W\_\{i\}^\{j\}\)\\rVert\_\{F\}≤2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+2​Cg​σf\+2​Cf​σ∇g\+1K​∑k=1K𝔼​∥∇Fk​\(Wik\)−1K​∑j=1K∇Fj​\(Wij\)∥F\\displaystyle\\leq 2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\nabla F^\{j\}\(W\_\{i\}^\{j\}\)\\rVert\_\{F\}≤2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+2​Cg​σf\+2​Cf​σ∇g\+1K​∑k=1K𝔼​∥∇Fk​\(Wik\)−∇Fk​\(W¯i\)∥F\\displaystyle\\leq 2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\-\\nabla F^\{k\}\(\\bar\{W\}\_\{i\}\)\\rVert\_\{F\}\+1K∑k=1K𝔼∥∇Fk\(W¯i\)−∇F\(W¯i\)∥F\+𝔼∥∇F\(W¯i\)−1K∑j=1K∇Fj\(Wij\)∥F\\displaystyle\\hskip 9\.24994pt\+\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\nabla F^\{k\}\(\\bar\{W\}\_\{i\}\)\-\\nabla F\(\\bar\{W\}\_\{i\}\)\\rVert\_\{F\}\+\\mathbb\{E\}\\lVert\\nabla F\(\\bar\{W\}\_\{i\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\nabla F^\{j\}\(W\_\{i\}^\{j\}\)\\rVert\_\{F\}≤2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+2​Cg​σf\+2​Cf​σ∇g\+LF​1K​∑k=1K𝔼​‖Wik−W¯i‖F\\displaystyle\\leq 2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+L\_\{F\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\}^\{k\}\-\\bar\{W\}\_\{i\}\\\|\_\{F\}\+\(1K​∑k=1K𝔼​‖∇Fk​\(W¯i\)−∇F​\(W¯i\)‖F2\)1/2\+LF​1K​∑j=1K𝔼​‖W¯i−Wij‖F\\displaystyle\\hskip 9\.24994pt\+\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|\\nabla F^\{k\}\(\\bar\{W\}\_\{i\}\)\-\\nabla F\(\\bar\{W\}\_\{i\}\)\\\|\_\{F\}^\{2\}\\right\)^\{1/2\}\+L\_\{F\}\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\mathbb\{E\}\\\|\\bar\{W\}\_\{i\}\-W\_\{i\}^\{j\}\\\|\_\{F\}≤2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+2​Cg​σf\+2​Cf​σ∇g\\displaystyle\\leq 2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+2LF1K∑k=1K𝔼∥Wik−W¯i∥F\+\(1K∑k=1K𝔼∥∇Fk\(W¯i\)−∇F\(W¯i\)∥F2\)1/2\\displaystyle\\hskip 9\.24994pt\+2L\_\{F\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\}^\{k\}\-\\bar\{W\}\_\{i\}\\\|\_\{F\}\+\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|\\nabla F^\{k\}\(\\bar\{W\}\_\{i\}\)\-\\nabla F\(\\bar\{W\}\_\{i\}\)\\\|\_\{F\}^\{2\}\\right\)^\{1/2\}≤2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\.\\displaystyle\\leq 2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\.\(30\)Combining \([29](https://arxiv.org/html/2608.12710#A1.E29)\) and \([30](https://arxiv.org/html/2608.12710#A1.E30)\), we obtain

1T​∑t=0T−11K​∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}≤1T​∑t=0T−1∑i=st\+1tβ​\(1−β\)t−i​\[2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\]\\displaystyle\\leq\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{i=s\_\{t\}\+1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\Bigg\[2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\Bigg\]≤βT​∑t=0T−1∑i=st\+1t2​Cg​Lf​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+βT​∑t=0T−1∑i=st\+1t\(2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\)\\displaystyle\\leq\\frac\{\\beta\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{i=s\_\{t\}\+1\}^\{t\}2C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+\\frac\{\\beta\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{i=s\_\{t\}\+1\}^\{t\}\\left\(2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\right\)≤2​β​Cg​LfT​∑i=0T−1∑t=imin⁡\{T−1,τ⁡⌊i/τ⌋\+τ−1\}1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+βT​∑t=0T−1\(t−st\)​\(2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\)\\displaystyle\\leq\\frac\{2\\beta C\_\{g\}L\_\{f\}\}\{T\}\\sum\_\{i=0\}^\{T\-1\}\\sum\_\{t=i\}^\{\\min\\\{T\-1,\\tau\\lfloor i/\\tau\\rfloor\+\\tau\-1\\\}\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+\\frac\{\\beta\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\(t\-s\_\{t\}\)\\left\(2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\right\)≤2​β​τ​Cg​LfT​∑i=0T−11K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\+β​τ​\(2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\)\\displaystyle\\leq\\frac\{2\\beta\\tau C\_\{g\}L\_\{f\}\}\{T\}\\sum\_\{i=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\+\\beta\\tau\\left\(2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\right\)=β​τ​\[2​Cg​LfT​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\+2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\]\\displaystyle=\\beta\\tau\\Bigg\[\\frac\{2C\_\{g\}L\_\{f\}\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\Bigg\]≤β​τ​\[2​Cg​Lf​\(1T​∑t=0T−1\(1K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\)2\)1/2\+2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\]\\displaystyle\\leq\\beta\\tau\\Bigg\[2C\_\{g\}L\_\{f\}\\left\(\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\}\\right\)^\{2\}\\right\)^\{1/2\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\Bigg\]≤β​τ​\[2​Cg​Lf​\(1T​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\)1/2\+2​Cg​σf\+2​Cf​σ∇g\+4​η​τ​n​LF\+δ\],\\displaystyle\\leq\\beta\\tau\\Bigg\[2C\_\{g\}L\_\{f\}\\left\(\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\\right\)^\{1/2\}\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\Bigg\],\(31\)where we used Lemmas[4](https://arxiv.org/html/2608.12710#Thmlemma4)and[5](https://arxiv.org/html/2608.12710#Thmlemma5), Assumption[5](https://arxiv.org/html/2608.12710#Thmassumption5), and the Cauchy–Schwarz inequality\. ∎

###### Lemma 6\.

LetW¯t=K−1​∑k=1KWtk\\bar\{W\}\_\{t\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}W\_\{t\}^\{k\}andM¯t=K−1​∑k=1KMtk\\bar\{M\}\_\{t\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{t\}^\{k\}\. Given Assumptions[4](https://arxiv.org/html/2608.12710#Thmassumption4),[2](https://arxiv.org/html/2608.12710#Thmassumption2), and[3](https://arxiv.org/html/2608.12710#Thmassumption3), for everyt≥0t\\geq 0, we have

F⁡\(W¯t\+1\)\\displaystyle F\(\\bar\{W\}\_\{t\+1\}\)≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+η2​n​LF2\+2​η​n​1K​∑k=1K‖Mtk−M¯t‖F\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}\+2\\eta\\sqrt\{n\}\\,\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}\+2ηnLF1K∑k=1K∥Wtk−W¯t∥F\+2ηn∥1K∑k=1K\(∇Fk\(Wtk\)−Mtk\)∥F\.\\displaystyle\\hskip 9\.24994pt\+2\\eta\\sqrt\{n\}\\,L\_\{F\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}\+2\\eta\\sqrt\{n\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\-M\_\{t\}^\{k\}\\right\)\\rVert\_\{F\}\.\(32\)

###### Proof\.

F⁡\(W¯t\+1\)\\displaystyle F\(\\bar\{W\}\_\{t\+1\}\)≤F⁡\(W¯t\)\+⟨∇F​\(W¯t\),W¯t\+1−W¯t⟩\+LF2​‖W¯t\+1−W¯t‖F2\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\+\\left\\langle\\nabla F\(\\bar\{W\}\_\{t\}\),\\bar\{W\}\_\{t\+1\}\-\\bar\{W\}\_\{t\}\\right\\rangle\+\\frac\{L\_\{F\}\}\{2\}\\\|\\bar\{W\}\_\{t\+1\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}^\{2\}=F⁡\(W¯t\)−η⁡⟨∇F​\(W¯t\),1K​∑k=1KUtk​\(Vtk\)⊤⟩\+η2​LF2​∥1K​∑k=1KUtk​\(Vtk\)⊤∥F2\\displaystyle=F\(\\bar\{W\}\_\{t\}\)\-\\eta\\left\\langle\\nabla F\(\\bar\{W\}\_\{t\}\),\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\right\\rangle\+\\frac\{\\eta^\{2\}L\_\{F\}\}\{2\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}≤F⁡\(W¯t\)−η​1K​∑k=1K⟨∇F​\(W¯t\),Utk​\(Vtk\)⊤⟩\+η2​n​LF2\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\\langle\\nabla F\(\\bar\{W\}\_\{t\}\),U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\right\\rangle\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+2​η​n​1K​∑k=1K‖∇F​\(W¯t\)−Mtk‖F\+η2​n​LF2\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+2\\eta\\sqrt\{n\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\-M\_\{t\}^\{k\}\\\|\_\{F\}\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+η2​n​LF2\+2​η​n​\(1K​∑k=1K‖M¯t−Mtk‖F\+∥1K​∑k=1K\(∇Fk​\(W¯t\)−Mtk\)∥F\)\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}\+2\\eta\\sqrt\{n\}\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\bar\{M\}\_\{t\}\-M\_\{t\}^\{k\}\\\|\_\{F\}\+\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(\\bar\{W\}\_\{t\}\)\-M\_\{t\}^\{k\}\\right\)\\rVert\_\{F\}\\right\)≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+η2​n​LF2\+2​η​n​1K​∑k=1K‖M¯t−Mtk‖F\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}\+2\\eta\\sqrt\{n\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\bar\{M\}\_\{t\}\-M\_\{t\}^\{k\}\\\|\_\{F\}\+2​η​n​∥1K​∑k=1K\(∇Fk​\(W¯t\)−∇Fk​\(Wtk\)\+∇Fk​\(Wtk\)−Mtk\)∥F\\displaystyle\\hskip 9\.24994pt\+2\\eta\\sqrt\{n\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(\\bar\{W\}\_\{t\}\)\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\+\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\-M\_\{t\}^\{k\}\\right\)\\rVert\_\{F\}≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+η2​n​LF2\+2​η​n​1K​∑k=1K‖M¯t−Mtk‖F\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}\+2\\eta\\sqrt\{n\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\bar\{M\}\_\{t\}\-M\_\{t\}^\{k\}\\\|\_\{F\}\+2ηn1K∑k=1K∥∇Fk\(W¯t\)−∇Fk\(Wtk\)∥F\+2ηn∥1K∑k=1K\(∇Fk\(Wtk\)−Mtk\)∥F\\displaystyle\\hskip 9\.24994pt\+2\\eta\\sqrt\{n\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\nabla F^\{k\}\(\\bar\{W\}\_\{t\}\)\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\rVert\_\{F\}\+2\\eta\\sqrt\{n\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\-M\_\{t\}^\{k\}\\right\)\\rVert\_\{F\}≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+η2​n​LF2\+2​η​n​1K​∑k=1K‖M¯t−Mtk‖F\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{\\eta^\{2\}nL\_\{F\}\}\{2\}\+2\\eta\\sqrt\{n\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\bar\{M\}\_\{t\}\-M\_\{t\}^\{k\}\\\|\_\{F\}\+2ηnLF1K∑k=1K∥Wtk−W¯t∥F\+2ηn∥1K∑k=1K\(∇Fk\(Wtk\)−Mtk\)∥F,\\displaystyle\\hskip 9\.24994pt\+2\\eta\\sqrt\{n\}\\,L\_\{F\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}\+2\\eta\\sqrt\{n\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\-M\_\{t\}^\{k\}\\right\)\\rVert\_\{F\},\(33\)where we used theLFL\_\{F\}\-smoothness ofFF, Lemma[2](https://arxiv.org/html/2608.12710#Thmlemma2), and the triangle inequality\. ∎

###### Lemma 7\.

Under Assumptions[4](https://arxiv.org/html/2608.12710#Thmassumption4),[2](https://arxiv.org/html/2608.12710#Thmassumption2), and[3](https://arxiv.org/html/2608.12710#Thmassumption3), let\{Wtk,utk,Mtk\}\\\{W\_\{t\}^\{k\},u\_\{t\}^\{k\},M\_\{t\}^\{k\}\\\}be generated by Algorithm[1](https://arxiv.org/html/2608.12710#alg1)\. For any integerT≥1T\\geq 1,0<β<10<\\beta<1, andτ∈ℕ\+\\tau\\in\\mathbb\{N\}\_\{\+\}, we have

1T​∑t=0T−1𝔼​∥1K​∑k=1K\(Mtk−∇Fk​\(Wtk\)\)∥F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)\\rVert\_\{F\}≤1β​T​\[Cg​Lf​σg\+Cg​σf\+Cf​σ∇g\]\+3​η​n​LFβ\\displaystyle\\leq\\frac\{1\}\{\\beta T\}\\Bigg\[C\_\{g\}L\_\{f\}\\sigma\_\{g\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\Bigg\]\+\\frac\{3\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta\}\+Cg​Lf​\(1T​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\)1/2\+Cg2​σf2\+Cf2​σ∇g2​βK\.\\displaystyle\\hskip 9\.24994pt\+C\_\{g\}L\_\{f\}\\left\(\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\\right\)^\{1/2\}\+\\frac\{\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\\,\\sqrt\{\\beta\}\}\{\\sqrt\{K\}\}\.\(34\)

###### Proof\.

For each clientkkand iterationi≥1i\\geq 1, the stochastic compositional gradient error admits the exact decomposition

Zik−∇Fk​\(Wik\)\\displaystyle Z\_\{i\}^\{k\}\-\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)=∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)​\(\(∇yfk​\(uik\)−∇yfk​\(gk​\(Wik\)\)\)⊗In\)\\displaystyle=\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\right\)\\otimes I\_\{n\}\\right\)\+∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)​\(\(∇yfk​\(uik,ζik\)−∇yfk​\(uik\)\)⊗In\)\\displaystyle\\hskip 9\.24994pt\+\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\+\(∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)−∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤\)\)×\(∇yfk\(gk\(Wik\)\)⊗In\)\\displaystyle\\hskip 9\.24994pt\+\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\right\)\\times\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\+\(∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤\)−∇gk​\(Wik\)\)​\(∇yfk​\(gk​\(Wik\)\)⊗In\)\.\\displaystyle\\hskip 9\.24994pt\+\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\-\\nabla g^\{k\}\(W\_\{i\}^\{k\}\)\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\.\(35\)
Letℱi−1\\mathcal\{F\}\_\{i\-1\}denote the history beforeξik\\xi\_\{i\}^\{k\}andζik\\zeta\_\{i\}^\{k\}are drawn\. By the unbiasedness and independence of the stochastic oracles, the two centered stochastic components in \([35](https://arxiv.org/html/2608.12710#A1.E35)\) are orthogonal in expectation, and the cross terms between distinct clients vanish\. Hence,

𝔼∥1K∑k=1K\[∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)\(\(∇yfk\(uik;ζik\)−∇yfk\(uik\)\)⊗In\)\\displaystyle\\mathbb\{E\}\\Bigg\\\|\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigg\[\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\+\(∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)−∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤\)\)\(∇yfk\(gk\(Wik\)\)⊗In\)\]∥F2\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\]\\Bigg\\\|\_\{F\}^\{2\}=1K2​∑k=1K𝔼\|∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)​\(\(∇yfk​\(uik,ζik\)−∇yfk​\(uik\)\)⊗In\)\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\Bigg\\\|\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\+\(∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)−∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤\)\)\(∇yfk\(gk\(Wik\)\)⊗In\)∥F2\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\\\|\_\{F\}^\{2\}=1K2​∑k=1K𝔼​‖∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)​\(\(∇yfk​\(uik,ζik\)−∇yfk​\(uik\)\)⊗In\)‖F2\\displaystyle=\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\Bigg\\\|\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\\Bigg\\\|\_\{F\}^\{2\}\+1K2∑k=1K𝔼∥\(∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)−∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤\)\)\(∇yfk\(gk\(Wik\)\)⊗In\)∥F2\\displaystyle\\hskip 9\.24994pt\+\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\Bigg\\\|\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\\\|\_\{F\}^\{2\}≤1K2∑k=1K𝔼\[∥∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)∥F2𝔼\[∥∇yfk\(uik;ζik\)−∇yfk\(uik\)∥2\|ℱi−1,ξik\]\]\\displaystyle\\leq\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\left\[\\lVert\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\rVert\_\{F\}^\{2\}\\mathbb\{E\}\\left\[\\lVert\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\rVert^\{2\}\\,\\middle\|\\,\\mathcal\{F\}\_\{i\-1\},\\xi\_\{i\}^\{k\}\\right\]\\right\]\+1K2∑k=1K𝔼\[∥∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)−∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤\)∥F2∥∇yfk\(gk\(Wik\)\)∥2\]\\displaystyle\\hskip 9\.24994pt\+\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\left\[\\lVert\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\rVert\_\{F\}^\{2\}\\lVert\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\rVert^\{2\}\\right\]≤1K2​∑k=1K\(Cg2​σf2\+Cf2​σ∇g2\)\\displaystyle\\leq\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\left\(C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\right\)=Cg2​σf2\+Cf2​σ∇g2K\.\\displaystyle=\\frac\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{K\}\.\(36\)The last inequality follows from Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[3](https://arxiv.org/html/2608.12710#Thmassumption3), together with Jensen’s inequality, which gives‖∇yfk​\(y\)‖≤Cf\\\|\\nabla\_\{y\}f^\{k\}\(y\)\\\|\\leq C\_\{f\}\.

SynchronizingMtkM\_\{t\}^\{k\}does not change its client average\. According to the momentum update,

1K​∑k=1K\(Mtk−∇Fk​\(Wtk\)\)\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)=1K​∑k=1K\(\(1−β\)​Mt−1k\+β​Ztk−∇Fk​\(Wtk\)\)\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\(1\-\\beta\)M\_\{t\-1\}^\{k\}\+\\beta Z\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)=\(1−β\)​1K​∑k=1K\(Mt−1k−∇Fk​\(Wt−1k\)\)\+\(1−β\)​1K​∑k=1K\(∇Fk​\(Wt−1k\)−∇Fk​\(Wtk\)\)\\displaystyle=\(1\-\\beta\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{t\-1\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\-1\}^\{k\}\)\\right\)\+\(1\-\\beta\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{t\-1\}^\{k\}\)\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)\+β1K∑k=1K\(Ztk−∇Fk\(Wtk\)\)\\displaystyle\\hskip 9\.24994pt\+\\beta\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(Z\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)=\(1−β\)t​1K​∑k=1K\(M0k−∇Fk​\(W0k\)\)\+∑i=1t\(1−β\)t−i\+1​1K​∑k=1K\(∇Fk​\(Wi−1k\)−∇Fk​\(Wik\)\)\\displaystyle=\(1\-\\beta\)^\{t\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{0\}^\{k\}\-\\nabla F^\{k\}\(W\_\{0\}^\{k\}\)\\right\)\+\\sum\_\{i=1\}^\{t\}\(1\-\\beta\)^\{t\-i\+1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{i\-1\}^\{k\}\)\-\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\\right\)\+∑i=1tβ\(1−β\)t−i1K∑k=1K\(Zik−∇Fk\(Wik\)\)\.\\displaystyle\\hskip 9\.24994pt\+\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(Z\_\{i\}^\{k\}\-\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\\right\)\.\(37\)
Substituting \([35](https://arxiv.org/html/2608.12710#A1.E35)\) into \([37](https://arxiv.org/html/2608.12710#A1.E37)\) and applying the triangle inequality yield

𝔼​∥1K​∑k=1K\(Mtk−∇Fk​\(Wtk\)\)∥F\\displaystyle\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)\\rVert\_\{F\}≤\(1−β\)t​𝔼​∥1K​∑k=1K\(M0k−∇Fk​\(W0k\)\)∥F⏟𝒯0\\displaystyle\\leq\(1\-\\beta\)^\{t\}\\underbrace\{\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{0\}^\{k\}\-\\nabla F^\{k\}\(W\_\{0\}^\{k\}\)\\right\)\\rVert\_\{F\}\}\_\{\\mathcal\{T\}\_\{0\}\}\+𝔼​∥∑i=1t\(1−β\)t−i\+1​1K​∑k=1K\(∇Fk​\(Wi−1k\)−∇Fk​\(Wik\)\)∥F⏟𝒯1\\displaystyle\\hskip 9\.24994pt\+\\underbrace\{\\mathbb\{E\}\\lVert\\sum\_\{i=1\}^\{t\}\(1\-\\beta\)^\{t\-i\+1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla F^\{k\}\(W\_\{i\-1\}^\{k\}\)\-\\nabla F^\{k\}\(W\_\{i\}^\{k\}\)\\right\)\\rVert\_\{F\}\}\_\{\\mathcal\{T\}\_\{1\}\}\+𝔼​∥∑i=1tβ​\(1−β\)t−i​1K​∑k=1K∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)​\(\(∇yfk​\(uik\)−∇yfk​\(gk​\(Wik\)\)\)⊗In\)∥F⏟𝒯2\\displaystyle\\hskip 9\.24994pt\+\\underbrace\{\\mathbb\{E\}\\lVert\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\right\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}\}\_\{\\mathcal\{T\}\_\{2\}\}\+𝔼∥∑i=1tβ\(1−β\)t−i1K∑k=1K\[∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)\(\(∇yfk\(uik;ζik\)−∇yfk\(uik\)\)⊗In\)\+\(∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)−∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤\)\)×\(∇yfk\(gk\(Wik\)\)⊗In\)\]∥F⏟𝒯3\\displaystyle\\hskip 9\.24994pt\+\\underbrace\{\\begin\{aligned\} &\\mathbb\{E\}\\Bigg\\\|\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigg\[\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\\\\\[\-0\.77498pt\] &\\hskip 28\.45274pt\+\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\right\)\\times\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\]\\Bigg\\\|\_\{F\}\\end\{aligned\}\}\_\{\\mathcal\{T\}\_\{3\}\}\+𝔼​‖∑i=1tβ​\(1−β\)t−i​1K​∑k=1K\(∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤\)−∇gk​\(Wik\)\)×\(∇yfk​\(gk​\(Wik\)\)⊗In\)‖F⏟𝒯4\.\\displaystyle\\hskip 9\.24994pt\+\\underbrace\{\\begin\{aligned\} &\\mathbb\{E\}\\Bigg\\\|\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\-\\nabla g^\{k\}\(W\_\{i\}^\{k\}\)\\right\)\\times\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\\\|\_\{F\}\\end\{aligned\}\}\_\{\\mathcal\{T\}\_\{4\}\}\.\(38\)
For𝒯0\\mathcal\{T\}\_\{0\}, sinceK−1​∑k=1KM0k=K−1​∑k=1KZ0kK^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{0\}^\{k\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}Z\_\{0\}^\{k\}, we have

𝒯0\\displaystyle\\mathcal\{T\}\_\{0\}=𝔼​∥1K​∑k=1K\(M0k−∇Fk​\(W0k\)\)∥F\\displaystyle=\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{0\}^\{k\}\-\\nabla F^\{k\}\(W\_\{0\}^\{k\}\)\\right\)\\rVert\_\{F\}=𝔼​∥1K​∑k=1K\(Z0k−∇Fk​\(W0k\)\)∥F\\displaystyle=\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(Z\_\{0\}^\{k\}\-\\nabla F^\{k\}\(W\_\{0\}^\{k\}\)\\right\)\\rVert\_\{F\}≤1K​∑k=1K𝔼​∥Z0k−∇Fk​\(W0k\)∥F\\displaystyle\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert Z\_\{0\}^\{k\}\-\\nabla F^\{k\}\(W\_\{0\}^\{k\}\)\\rVert\_\{F\}≤Cg​Lf​1K​∑k=1K𝔼​∥u0k−gk​\(W0k\)∥2\+Cg​σf\+Cf​σ∇g\\displaystyle\\leq C\_\{g\}L\_\{f\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\lVert u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\rVert^\{2\}\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}≤Cg​Lf​\(1K​∑k=1K𝔼​∥u0k−gk​\(W0k\)∥2\)1/2\+Cg​σf\+Cf​σ∇g\\displaystyle\\leq C\_\{g\}L\_\{f\}\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\rVert^\{2\}\\right\)^\{1/2\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}≤Cg​Lf​σg\+Cg​σf\+Cf​σ∇g\.\\displaystyle\\leq C\_\{g\}L\_\{f\}\\sigma\_\{g\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\.\(39\)The second inequality follows from Lemma[4](https://arxiv.org/html/2608.12710#Thmlemma4), the third follows from the Cauchy–Schwarz inequality, and the last usesu0k=gk​\(W0k,ξ0k\)u\_\{0\}^\{k\}=g^\{k\}\(W\_\{0\}^\{k\};\\xi\_\{0\}^\{k\}\)and Assumption[2](https://arxiv.org/html/2608.12710#Thmassumption2)\.

For𝒯1\\mathcal\{T\}\_\{1\}, the triangle inequality and Lemma[1](https://arxiv.org/html/2608.12710#Thmlemma1)give

1T​∑t=0T−1𝒯1\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathcal\{T\}\_\{1\}≤LFT​∑t=0T−1∑i=1t\(1−β\)t−i\+1​1K​∑k=1K𝔼​‖Wik−Wi−1k‖F\\displaystyle\\leq\\frac\{L\_\{F\}\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{i=1\}^\{t\}\(1\-\\beta\)^\{t\-i\+1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\}^\{k\}\-W\_\{i\-1\}^\{k\}\\\|\_\{F\}=LFT​∑i=1T−1\(∑t=iT−1\(1−β\)t−i\+1\)​1K​∑k=1K𝔼​‖Wik−Wi−1k‖F\\displaystyle=\\frac\{L\_\{F\}\}\{T\}\\sum\_\{i=1\}^\{T\-1\}\\left\(\\sum\_\{t=i\}^\{T\-1\}\(1\-\\beta\)^\{t\-i\+1\}\\right\)\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\}^\{k\}\-W\_\{i\-1\}^\{k\}\\\|\_\{F\}≤LFβ​T​∑i=1T−11K​∑k=1K𝔼​‖Wik−Wi−1k‖F\\displaystyle\\leq\\frac\{L\_\{F\}\}\{\\beta T\}\\sum\_\{i=1\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\}^\{k\}\-W\_\{i\-1\}^\{k\}\\\|\_\{F\}≤η​LFβ​T\{n\[T−1−⌊T−1τ⌋\]\+∑1≤i≤T−1mod⁡\(i,τ\)=0\(1K∑k=1K𝔼∥∑ℓ=i−τi−2\(Uℓk\(Vℓk\)⊤−1K∑j=1KUℓj\(Vℓj\)⊤\)∥F2\)1/2\\displaystyle\\leq\\frac\{\\eta L\_\{F\}\}\{\\beta T\}\\Bigg\\\{\\sqrt\{n\}\\left\[T\-1\-\\left\\lfloor\\frac\{T\-1\}\{\\tau\}\\right\\rfloor\\right\]\+\\sum\_\{\\begin\{subarray\}\{c\}1\\leq i\\leq T\-1\\\\ \\operatorname\{mod\}\(i,\\tau\)=0\\end\{subarray\}\}\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\sum\_\{\\ell=i\-\\tau\}^\{i\-2\}\\left\(U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{\\ell\}^\{j\}\(V\_\{\\ell\}^\{j\}\)^\{\\top\}\\right\)\\rVert\_\{F\}^\{2\}\\right\)^\{1/2\}\+n⌊T−1τ⌋\}\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\\hskip 18\.49988pt\+\\sqrt\{n\}\\left\\lfloor\\frac\{T\-1\}\{\\tau\}\\right\\rfloor\\Bigg\\\}≤η​LFβ​T​\{n​\[T−1−⌊T−1τ⌋\]\+∑1≤i≤T−1mod⁡\(i,τ\)=0\(1K​∑k=1K𝔼​∥∑ℓ=i−τi−2Uℓk​\(Vℓk\)⊤∥F2\)1/2\+n​⌊T−1τ⌋\}\\displaystyle\\leq\\frac\{\\eta L\_\{F\}\}\{\\beta T\}\\Bigg\\\{\\sqrt\{n\}\\left\[T\-1\-\\left\\lfloor\\frac\{T\-1\}\{\\tau\}\\right\\rfloor\\right\]\+\\sum\_\{\\begin\{subarray\}\{c\}1\\leq i\\leq T\-1\\\\ \\operatorname\{mod\}\(i,\\tau\)=0\\end\{subarray\}\}\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\sum\_\{\\ell=i\-\\tau\}^\{i\-2\}U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}\\right\)^\{1/2\}\+\\sqrt\{n\}\\left\\lfloor\\frac\{T\-1\}\{\\tau\}\\right\\rfloor\\Bigg\\\}≤η​n​LFβ​T​\[T−1\+\(τ−1\)​⌊T−1τ⌋\]\\displaystyle\\leq\\frac\{\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta T\}\\left\[T\-1\+\(\\tau\-1\)\\left\\lfloor\\frac\{T\-1\}\{\\tau\}\\right\\rfloor\\right\]<2​η​n​LFβ\.\\displaystyle<\\frac\{2\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta\}\.\(40\)The equality exchanges the sums, and the second inequality uses∑t=iT−1\(1−β\)t−i\+1≤β−1\\sum\_\{t=i\}^\{T\-1\}\(1\-\\beta\)^\{t\-i\+1\}\\leq\\beta^\{\-1\}\. The next three inequalities use the local and synchronized model updates, the variance identity, and Lemma[2](https://arxiv.org/html/2608.12710#Thmlemma2)\. The last inequality usesτ⁡⌊\(T−1\)/τ⌋≤T−1<T\\tau\\lfloor\(T\-1\)/\\tau\\rfloor\\leq T\-1<T\.

For𝒯2\\mathcal\{T\}\_\{2\}, we have

𝒯2\\displaystyle\\mathcal\{T\}\_\{2\}=𝔼​∥∑i=1tβ​\(1−β\)t−i​1K​∑k=1K∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)​\(\(∇yfk​\(uik\)−∇yfk​\(gk​\(Wik\)\)\)⊗In\)∥F\\displaystyle=\\mathbb\{E\}\\lVert\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\right\)\\otimes I\_\{n\}\\right\)\\rVert\_\{F\}≤∑i=1tβ​\(1−β\)t−i​1K​∑k=1K𝔼⁡\[∥∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)∥F×‖∇yfk​\(uik\)−∇yfk​\(gk​\(Wik\)\)‖\]\\displaystyle\\leq\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\Big\[\\lVert\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\rVert\_\{F\}\\times\\\|\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\\|\\Big\]≤Lf​∑i=1tβ​\(1−β\)t−i​1K​∑k=1K𝔼⁡\[∥∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)∥F​‖uik−gk​\(Wik\)‖\]\\displaystyle\\leq L\_\{f\}\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\Big\[\\lVert\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\rVert\_\{F\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|\\Big\]≤Cg​Lf​∑i=1tβ​\(1−β\)t−i​1K​∑k=1K𝔼​‖uik−gk​\(Wik\)‖2\.\\displaystyle\\leq C\_\{g\}L\_\{f\}\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{i\}^\{k\}\-g^\{k\}\(W\_\{i\}^\{k\}\)\\\|^\{2\}\}\.\(41\)The third step follows from theLfL\_\{f\}\-smoothness offkf^\{k\}, and the last step follows from Assumption[3](https://arxiv.org/html/2608.12710#Thmassumption3)and the Cauchy–Schwarz inequality\.

For𝒯3\\mathcal\{T\}\_\{3\}, we have

𝒯32\\displaystyle\\mathcal\{T\}\_\{3\}^\{2\}=\(𝔼∥∑i=1tβ\(1−β\)t−i1K∑k=1K\[∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)\(\(∇yfk\(uik;ζik\)−∇yfk\(uik\)\)⊗In\)\\displaystyle=\\Bigg\(\\mathbb\{E\}\\Bigg\\\|\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigg\[\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\+\(∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)−∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤\)\)\(∇yfk\(gk\(Wik\)\)⊗In\)\]∥F\)2\\displaystyle\\hskip 128\.0374pt\+\\left\(\\begin\{aligned\} &\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\\\\[\-0\.77498pt\] &\\hskip 9\.24994pt\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\end\{aligned\}\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\]\\Bigg\\\|\_\{F\}\\Bigg\)^\{2\}≤𝔼∥∑i=1tβ\(1−β\)t−i1K∑k=1K\[∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)\(\(∇yfk\(uik;ζik\)−∇yfk\(uik\)\)⊗In\)\\displaystyle\\leq\\mathbb\{E\}\\Bigg\\\|\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigg\[\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\+\(∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)−∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤\)\)\(∇yfk\(gk\(Wik\)\)⊗In\)\]∥F2\\displaystyle\\hskip 128\.0374pt\+\\left\(\\begin\{aligned\} &\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\\\\[\-0\.77498pt\] &\\hskip 9\.24994pt\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\end\{aligned\}\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\]\\Bigg\\\|\_\{F\}^\{2\}=∑i=1tβ2\(1−β\)2​\(t−i\)𝔼∥1K∑k=1K\[∇gk\(Wi−1k−ηUi−1k\(Vi−1k\)⊤;ξik\)\(\(∇yfk\(uik;ζik\)−∇yfk\(uik\)\)⊗In\)\\displaystyle=\\sum\_\{i=1\}^\{t\}\\beta^\{2\}\(1\-\\beta\)^\{2\(t\-i\)\}\\mathbb\{E\}\\Bigg\\\|\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigg\[\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\\left\(\\left\(\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\};\\zeta\_\{i\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{i\}^\{k\}\)\\right\)\\otimes I\_\{n\}\\right\)\+\(∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤,ξik\)−∇gk​\(Wi−1k−η​Ui−1k​\(Vi−1k\)⊤\)\)\(∇yfk\(gk\(Wik\)\)⊗In\)\]∥F2\\displaystyle\\hskip 128\.0374pt\+\\left\(\\begin\{aligned\} &\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\};\\xi\_\{i\}^\{k\}\\right\)\-\\\\\[\-0\.77498pt\] &\\hskip 9\.24994pt\\nabla g^\{k\}\\left\(W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\\right\)\\end\{aligned\}\\right\)\\left\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{i\}^\{k\}\)\)\\otimes I\_\{n\}\\right\)\\Bigg\]\\Bigg\\\|\_\{F\}^\{2\}≤Cg2​σf2\+Cf2​σ∇g2K​∑i=1tβ2​\(1−β\)2​\(t−i\)\\displaystyle\\leq\\frac\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{K\}\\sum\_\{i=1\}^\{t\}\\beta^\{2\}\(1\-\\beta\)^\{2\(t\-i\)\}≤Cg2​σf2\+Cf2​σ∇g2K​β21−\(1−β\)2\\displaystyle\\leq\\frac\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{K\}\\frac\{\\beta^\{2\}\}\{1\-\(1\-\\beta\)^\{2\}\}=Cg2​σf2\+Cf2​σ∇g2K​β2−β\\displaystyle=\\frac\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{K\}\\frac\{\\beta\}\{2\-\\beta\}≤\(Cg2​σf2\+Cf2​σ∇g2\)​βK\.\\displaystyle\\leq\\frac\{\\left\(C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\right\)\\beta\}\{K\}\.\(42\)Therefore,

𝒯3\\displaystyle\\mathcal\{T\}\_\{3\}≤Cg2​σf2\+Cf2​σ∇g2​βK\.\\displaystyle\\leq\\frac\{\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\\sqrt\{\\beta\}\}\{\\sqrt\{K\}\}\.\(43\)The first inequality follows from Jensen’s inequality, the second equality follows from the martingale difference property, and the first inequality after that follows from \([36](https://arxiv.org/html/2608.12710#A1.E36)\)\. The last inequality uses0<β<10<\\beta<1\.

For𝒯4\\mathcal\{T\}\_\{4\}, the synchronization rule, the variance identity, and Lemma[2](https://arxiv.org/html/2608.12710#Thmlemma2)give

1K∑k=1K𝔼∥Wi−1k−ηUi−1k\(Vi−1k\)⊤−Wik∥F≤ητn1\{mod\(i,τ\)=0\}\.\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\-W\_\{i\}^\{k\}\\rVert\_\{F\}\\leq\\eta\\tau\\sqrt\{n\}\\,\\mathbf\{1\}\_\{\\\{\\operatorname\{mod\}\(i,\\tau\)=0\\\}\}\.\(44\)Therefore, Assumption[4](https://arxiv.org/html/2608.12710#Thmassumption4)and‖∇yfk​\(y\)‖≤Cf\\\|\\nabla\_\{y\}f^\{k\}\(y\)\\\|\\leq C\_\{f\}imply

1T​∑t=0T−1𝒯4\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathcal\{T\}\_\{4\}≤Cf​LgT​∑t=0T−1∑i=1tβ​\(1−β\)t−i​1K​∑k=1K𝔼​∥Wi−1k−η​Ui−1k​\(Vi−1k\)⊤−Wik∥F\\displaystyle\\leq\\frac\{C\_\{f\}L\_\{g\}\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{i=1\}^\{t\}\\beta\(1\-\\beta\)^\{t\-i\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\-W\_\{i\}^\{k\}\\rVert\_\{F\}≤Cf​LgT​∑i=1T−11K​∑k=1K𝔼​∥Wi−1k−η​Ui−1k​\(Vi−1k\)⊤−Wik∥F\\displaystyle\\leq\\frac\{C\_\{f\}L\_\{g\}\}\{T\}\\sum\_\{i=1\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert W\_\{i\-1\}^\{k\}\-\\eta U\_\{i\-1\}^\{k\}\(V\_\{i\-1\}^\{k\}\)^\{\\top\}\-W\_\{i\}^\{k\}\\rVert\_\{F\}≤η​n​Cf​Lg≤η​n​LFβ\.\\displaystyle\\leq\\eta\\sqrt\{n\}\\,C\_\{f\}L\_\{g\}\\leq\\frac\{\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta\}\.\(45\)
Combining \([38](https://arxiv.org/html/2608.12710#A1.E38)\), \([39](https://arxiv.org/html/2608.12710#A1.E39)\), \([40](https://arxiv.org/html/2608.12710#A1.E40)\), \([41](https://arxiv.org/html/2608.12710#A1.E41)\), and \([43](https://arxiv.org/html/2608.12710#A1.E43)\)–\([45](https://arxiv.org/html/2608.12710#A1.E45)\), and averaging overt=0,…,T−1t=0,\\ldots,T\-1, we obtain

1T​∑t=0T−1𝔼​∥1K​∑k=1K\(Mtk−∇Fk​\(Wtk\)\)∥F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(M\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\right\)\\rVert\_\{F\}≤1β​T​\[Cg​Lf​σg\+Cg​σf\+Cf​σ∇g\]\+3​η​n​LFβ\\displaystyle\\leq\\frac\{1\}\{\\beta T\}\\Bigg\[C\_\{g\}L\_\{f\}\\sigma\_\{g\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\Bigg\]\+\\frac\{3\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta\}\+Cg​LfT∑t=0T−11K∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\+Cg2​σf2\+Cf2​σ∇g2​βK\\displaystyle\\hskip 9\.24994pt\+\\frac\{C\_\{g\}L\_\{f\}\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\sqrt\{\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\}\+\\frac\{\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\\,\\sqrt\{\\beta\}\}\{\\sqrt\{K\}\}≤1β​T​\[Cg​Lf​σg\+Cg​σf\+Cf​σ∇g\]\+3​η​n​LFβ\\displaystyle\\leq\\frac\{1\}\{\\beta T\}\\Bigg\[C\_\{g\}L\_\{f\}\\sigma\_\{g\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\Bigg\]\+\\frac\{3\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta\}\+Cg​Lf​\(1T​∑t=0T−11K​∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\)1/2\+Cg2​σf2\+Cf2​σ∇g2​βK\.\\displaystyle\\hskip 9\.24994pt\+C\_\{g\}L\_\{f\}\\left\(\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\\right\)^\{1/2\}\+\\frac\{\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\\,\\sqrt\{\\beta\}\}\{\\sqrt\{K\}\}\.\(46\)The first inequality uses∑t=0T−1\(1−β\)t≤β−1\\sum\_\{t=0\}^\{T\-1\}\(1\-\\beta\)^\{t\}\\leq\\beta^\{\-1\}and∑t=iT−1β​\(1−β\)t−i≤1\\sum\_\{t=i\}^\{T\-1\}\\beta\(1\-\\beta\)^\{t\-i\}\\leq 1, and the second follows from the Cauchy–Schwarz inequality\. This proves \([34](https://arxiv.org/html/2608.12710#A1.E34)\)\. ∎

###### Theorem 1\(Convergence of FedCoMuon\)\.

Under Assumptions[1](https://arxiv.org/html/2608.12710#Thmassumption1)–[5](https://arxiv.org/html/2608.12710#Thmassumption5), let\{Wtk,utk,Mtk\}\\\{W\_\{t\}^\{k\},u\_\{t\}^\{k\},M\_\{t\}^\{k\}\\\}be generated by Algorithm[1](https://arxiv.org/html/2608.12710#alg1)\. Under the synchronization convention stated above, for any integerT≥1T\\geq 1,η\>0\\eta\>0,0<α,β<10<\\alpha,\\beta<1, andτ\>0\\tau\>0satisfyingα​τ≤1\\alpha\\tau\\leq 1, we have

1T​∑t=0T−1𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+2​n​Cg​Lf​σgβ​T\+2​n​\(1β​T\+2​β​τ\)​\(Cg​σf\+Cf​σ∇g\)\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{2\\sqrt\{n\}C\_\{g\}L\_\{f\}\\sigma\_\{g\}\}\{\\beta T\}\+2\\sqrt\{n\}\\left\(\\frac\{1\}\{\\beta T\}\+2\\beta\\tau\\right\)\\left\(C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\right\)\+2​n​Cg​Lf​\(1\+2​β​τ\)​\(4​σg2α​T\+86​Cg2​nα2​η2\+86​α​σg2\)1/2\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}C\_\{g\}L\_\{f\}\(1\+2\\beta\\tau\)\\left\(\\frac\{4\\sigma\_\{g\}^\{2\}\}\{\\alpha T\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\\right\)^\{1/2\}\+η​n​LF​\(12\+4​τ\+6β\+8​β​τ2\)\+2​n​βK​Cg2​σf2\+Cf2​σ∇g2\\displaystyle\\hskip 9\.24994pt\+\\eta nL\_\{F\}\\left\(\\frac\{1\}\{2\}\+4\\tau\+\\frac\{6\}\{\\beta\}\+8\\beta\\tau^\{2\}\\right\)\+\\frac\{2\\sqrt\{n\\beta\}\}\{\\sqrt\{K\}\}\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\+2​n​β​τ​δ\.\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}\\,\\beta\\tau\\delta\.\(47\)

###### Proof\.

Summing Lemma[6](https://arxiv.org/html/2608.12710#Thmlemma6)overt=0,…,T−1t=0,\\ldots,T\-1and usingF⁡\(W¯T\)≥F∗F\(\\bar\{W\}\_\{T\}\)\\geq F\_\{\*\}gives the first inequality below\. Applying Lemmas[3](https://arxiv.org/html/2608.12710#Thmlemma3),[5](https://arxiv.org/html/2608.12710#Thmlemma5), and[7](https://arxiv.org/html/2608.12710#Thmlemma7)gives the second\.

1T​∑t=0T−1𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+η​n​LF2\+2​n​1T​∑t=0T−11K​∑k=1K𝔼​‖Mtk−M¯t‖F\+2​n​LF​1T​∑t=0T−11K​∑k=1K𝔼​‖Wtk−W¯t‖F\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{\\eta nL\_\{F\}\}\{2\}\+2\\sqrt\{n\}\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}\+2\\sqrt\{n\}\\,L\_\{F\}\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}\+2n1T∑t=0T−1𝔼∥1K∑k=1K\(∇Fk\(Wtk\)−Mtk\)∥F\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\bigl\(\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\-M\_\{t\}^\{k\}\\bigr\)\\rVert\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+η​n​LF2\+4ητnLF\+2nβτ\[2CgLf\(4α​T1K∑k=1K𝔼∥u0k−gk\(W0k\)∥2\+86​Cg2​nα2η2\+86ασg2\)1/2\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{\\eta nL\_\{F\}\}\{2\}\+4\\eta\\tau nL\_\{F\}\+2\\sqrt\{n\}\\,\\beta\\tau\\Bigg\[2C\_\{g\}L\_\{f\}\\left\(\\frac\{4\}\{\\alpha T\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\\right\)^\{1/2\}\+2Cgσf\+2Cfσ∇g\+4ητnLF\+δ\]\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\\hskip 18\.49988pt\+2C\_\{g\}\\sigma\_\{f\}\+2C\_\{f\}\\sigma\_\{\\nabla g\}\+4\\eta\\tau\\sqrt\{n\}\\,L\_\{F\}\+\\delta\\Bigg\]\+2n\[1β​T\(CgLfσg\+Cgσf\+Cfσ∇g\)\+3​η​n​LFβ\+Cg2​σf2\+Cf2​σ∇g2​βK\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}\\Bigg\[\\frac\{1\}\{\\beta T\}\\Bigg\(C\_\{g\}L\_\{f\}\\sigma\_\{g\}\+C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\Bigg\)\+\\frac\{3\\eta\\sqrt\{n\}\\,L\_\{F\}\}\{\\beta\}\+\\frac\{\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\\,\\sqrt\{\\beta\}\}\{\\sqrt\{K\}\}\+CgLf\(4α​T1K∑k=1K𝔼∥u0k−gk\(W0k\)∥2\+86​Cg2​nα2η2\+86ασg2\)1/2\]\\displaystyle\\hskip 9\.24994pt\+C\_\{g\}L\_\{f\}\\left\(\\frac\{4\}\{\\alpha T\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\\right\)^\{1/2\}\\Bigg\]≤F⁡\(W¯0\)−F∗η​T\+2​n​Cg​Lf​σgβ​T\+2​n​\(1β​T\+2​β​τ\)​\(Cg​σf\+Cf​σ∇g\)\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{2\\sqrt\{n\}C\_\{g\}L\_\{f\}\\sigma\_\{g\}\}\{\\beta T\}\+2\\sqrt\{n\}\\left\(\\frac\{1\}\{\\beta T\}\+2\\beta\\tau\\right\)\\left\(C\_\{g\}\\sigma\_\{f\}\+C\_\{f\}\\sigma\_\{\\nabla g\}\\right\)\+2​n​Cg​Lf​\(1\+2​β​τ\)​\(4​σg2α​T\+86​Cg2​nα2​η2\+86​α​σg2\)1/2\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}C\_\{g\}L\_\{f\}\(1\+2\\beta\\tau\)\\left\(\\frac\{4\\sigma\_\{g\}^\{2\}\}\{\\alpha T\}\+\\frac\{86C\_\{g\}^\{2\}n\}\{\\alpha^\{2\}\}\\eta^\{2\}\+86\\alpha\\sigma\_\{g\}^\{2\}\\right\)^\{1/2\}\+η​n​LF​\(12\+4​τ\+6β\+8​β​τ2\)\+2​n​βK​Cg2​σf2\+Cf2​σ∇g2\+2​n​β​τ​δ\.\\displaystyle\\hskip 9\.24994pt\+\\eta nL\_\{F\}\\left\(\\frac\{1\}\{2\}\+4\\tau\+\\frac\{6\}\{\\beta\}\+8\\beta\\tau^\{2\}\\right\)\+\\frac\{2\\sqrt\{n\\beta\}\}\{\\sqrt\{K\}\}\\sqrt\{C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\+C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\+2\\sqrt\{n\}\\,\\beta\\tau\\delta\.The last inequality uses the initializationu0k=gk​\(W0k,ξ0k\)u\_\{0\}^\{k\}=g^\{k\}\(W\_\{0\}^\{k\};\\xi\_\{0\}^\{k\}\)and Assumption[2](https://arxiv.org/html/2608.12710#Thmassumption2), which implyK−1​∑k=1K𝔼​‖u0k−gk​\(W0k\)‖2≤σg2K^\{\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}^\{k\}\)\\\|^\{2\}\\leq\\sigma\_\{g\}^\{2\}\. ∎

Forη=T−3/4\\eta=T^\{\-3/4\},α=β=T−1/2\\alpha=\\beta=T^\{\-1/2\}, andτ=T1/4\\tau=T^\{1/4\}, we haveατ=T−1/4≤1\\alpha\\tau=T^\{\-1/4\}\\leq 1\. The non\-square\-root terms in Theorem[1](https://arxiv.org/html/2608.12710#Thmtheorem1a)are at mostO\(T−1/4\)O\(T^\{\-1/4\}\), while every term inside the square root is at mostO\(T−1/2\)O\(T^\{\-1/2\}\)\. Hence,T−1∑t=0T−1𝔼∥∇F\(W¯t\)∥F=O\(T−1/4\)T^\{\-1\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}=O\(T^\{\-1/4\}\)\.

## Appendix BConvergence Analysis of our FedCoMuon\-VR Algorithm

In this section, we provide the detailed convergence analysis of FedCoMuon\-VR\. We writeW¯t=K−1​∑k=1KWtk\\bar\{W\}\_\{t\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}W\_\{t\}^\{k\}andM¯t=K−1​∑k=1KMtk\\bar\{M\}\_\{t\}=K^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{t\}^\{k\}\. For vectors,∥⋅∥\\\|\\cdot\\\|denotes the Euclidean norm\. For matrices,∥⋅∥F\\\|\\cdot\\\|\_\{F\},∥⋅∥\\\|\\cdot\\\|, and∥⋅∥∗\\\|\\cdot\\\|\_\{\*\}denote the Frobenius, spectral, and nuclear norms, respectively\. The local data distributions may be non\-i\.i\.d\. across clients\.

###### Lemma 8\(Local model drift\)\.

For everyt=0,…,Tt=0,\\ldots,T,

1K​∑k=1K𝔼​‖Wtk−W¯t‖F2≤n​η2​τ2\.\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}^\{2\}\\leq n\\eta^\{2\}\\tau^\{2\}\.\(48\)Moreover,

1K​T​∑i=0T−1∑k=1K𝔼​‖Wi\+1k−Wik‖F2≤2​n​η2​τ\.\\frac\{1\}\{KT\}\\sum\_\{i=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\+1\}^\{k\}\-W\_\{i\}^\{k\}\\\|\_\{F\}^\{2\}\\leq 2n\\eta^\{2\}\\tau\.\(49\)

###### Proof\.

Lets⁡\(t\)=τ​⌊t/τ⌋s\(t\)=\\tau\\lfloor t/\\tau\\rfloorbe the latest communication index not larger thantt\. Since‖Uℓk​\(Vℓk\)⊤‖F2=rank⁡\(Uℓk​\(Vℓk\)⊤\)≤n\\\|U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\\\|\_\{F\}^\{2\}=\\operatorname\{rank\}\(U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\)\\leq nfor every update indexℓ\\ell,Ws⁡\(t\)k=W¯s⁡\(t\)W\_\{s\(t\)\}^\{k\}=\\bar\{W\}\_\{s\(t\)\}, and0≤t−s⁡\(t\)≤τ−10\\leq t\-s\(t\)\\leq\\tau\-1, we have

1K​∑k=1K𝔼​‖Wtk−W¯t‖F2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}^\{2\}=1K​∑k=1K𝔼​∥η​∑ℓ=s⁡\(t\)t−1Uℓk​\(Vℓk\)⊤−η​∑ℓ=s⁡\(t\)t−11K​∑j=1KUℓj​\(Vℓj\)⊤∥F2\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\eta\\sum\_\{\\ell=s\(t\)\}^\{t\-1\}U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\-\\eta\\sum\_\{\\ell=s\(t\)\}^\{t\-1\}\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{\\ell\}^\{j\}\(V\_\{\\ell\}^\{j\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}≤1K​∑k=1K𝔼​∥η​∑ℓ=s⁡\(t\)t−1Uℓk​\(Vℓk\)⊤∥F2\\displaystyle\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\eta\\sum\_\{\\ell=s\(t\)\}^\{t\-1\}U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}≤1K​∑k=1K𝔼​\(∑ℓ=s⁡\(t\)t−1η​‖Uℓk​\(Vℓk\)⊤‖F\)2\\displaystyle\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\left\(\\sum\_\{\\ell=s\(t\)\}^\{t\-1\}\\eta\\\|U\_\{\\ell\}^\{k\}\(V\_\{\\ell\}^\{k\}\)^\{\\top\}\\\|\_\{F\}\\right\)^\{2\}≤n​η2​\(t−s⁡\(t\)\)2≤n​η2​τ2\.\\displaystyle\\leq n\\eta^\{2\}\(t\-s\(t\)\)^\{2\}\\leq n\\eta^\{2\}\\tau^\{2\}\.\(50\)Moreover, averaging the local update givesW¯i\+1=W¯i−η​K−1​∑j=1KUij​\(Vij\)⊤\\bar\{W\}\_\{i\+1\}=\\bar\{W\}\_\{i\}\-\\eta K^\{\-1\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}at every iteration\. Ifmod⁡\(i\+1,τ\)≠0\\operatorname\{mod\}\(i\+1,\\tau\)\\neq 0, then

1K​∑k=1K𝔼​‖Wi\+1k−Wik‖F2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\+1\}^\{k\}\-W\_\{i\}^\{k\}\\\|\_\{F\}^\{2\}=η2K​∑k=1K𝔼​‖Uik​\(Vik\)⊤‖F2≤n​η2\.\\displaystyle=\\frac\{\\eta^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|U\_\{i\}^\{k\}\(V\_\{i\}^\{k\}\)^\{\\top\}\\\|\_\{F\}^\{2\}\\leq n\\eta^\{2\}\.\(51\)Ifmod⁡\(i\+1,τ\)=0\\operatorname\{mod\}\(i\+1,\\tau\)=0, synchronization and the averaged update give

1K​∑k=1K𝔼​‖Wi\+1k−Wik‖F2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\+1\}^\{k\}\-W\_\{i\}^\{k\}\\\|\_\{F\}^\{2\}=1K​∑k=1K𝔼​∥W¯i−Wik−η​1K​∑j=1KUij​\(Vij\)⊤∥F2\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\bar\{W\}\_\{i\}\-W\_\{i\}^\{k\}\-\\eta\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}≤2K​∑k=1K𝔼​‖Wik−W¯i‖F2\+2​η2​𝔼​∥1K​∑j=1KUij​\(Vij\)⊤∥F2\\displaystyle\\leq\\frac\{2\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\}^\{k\}\-\\bar\{W\}\_\{i\}\\\|\_\{F\}^\{2\}\+2\\eta^\{2\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}≤2​n​η2​\(τ−1\)2\+2​η2K​∑j=1K𝔼​‖Uij​\(Vij\)⊤‖F2\\displaystyle\\leq 2n\\eta^\{2\}\(\\tau\-1\)^\{2\}\+\\frac\{2\\eta^\{2\}\}\{K\}\\sum\_\{j=1\}^\{K\}\\mathbb\{E\}\\\|U\_\{i\}^\{j\}\(V\_\{i\}^\{j\}\)^\{\\top\}\\\|\_\{F\}^\{2\}≤2​n​η2​\(\(τ−1\)2\+1\)\.\\displaystyle\\leq 2n\\eta^\{2\}\\bigl\(\(\\tau\-1\)^\{2\}\+1\\bigr\)\.\(52\)Consequently,

1K​T​∑i=0T−1∑k=1K𝔼​‖Wi\+1k−Wik‖F2\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{i=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{i\+1\}^\{k\}\-W\_\{i\}^\{k\}\\\|\_\{F\}^\{2\}≤T−⌊T/τ⌋T​n​η2\+⌊T/τ⌋T​2​n​η2​\(\(τ−1\)2\+1\)\\displaystyle\\leq\\frac\{T\-\\lfloor T/\\tau\\rfloor\}\{T\}n\\eta^\{2\}\+\\frac\{\\lfloor T/\\tau\\rfloor\}\{T\}2n\\eta^\{2\}\\bigl\(\(\\tau\-1\)^\{2\}\+1\\bigr\)≤n​η2​\(2​τ−3\+3τ\)≤2​n​η2​τ,\\displaystyle\\leq n\\eta^\{2\}\\left\(2\\tau\-3\+\\frac\{3\}\{\\tau\}\\right\)\\leq 2n\\eta^\{2\}\\tau,\(53\)where we used the synchronization rule,‖Utk​\(Vtk\)⊤‖F2≤n\\\|U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\\|\_\{F\}^\{2\}\\leq n, and the number of communication steps\. ∎

###### Lemma 9\(Tracker recursions\)\.

If0<α,β,γ<10<\\alpha,\\beta,\\gamma<1andb≥1b\\geq 1, then

1K​T​∑t=0T−1∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2≤σg2α​T​b\+2​σg2​α\+4​Cg2​n​η2​τα\.\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\\leq\\frac\{\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+2\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{4C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\.\(54\)1K​T​∑t=0T−1∑k=1K𝔼​‖Htk−∇gk​\(Wtk\)‖F2≤σ∇g2γ​T​b\+2​σ∇g2​γ\+4​Lg2​n​η2​τγ\.\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}\\leq\\frac\{\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+2\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{4L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\.\(55\)1K​T​∑t=0T−1∑k=1K𝔼​‖vtk−∇yfk​\(gk​\(Wtk\)\)‖2\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\\|^\{2\}≤2​σf2β​T​b\+2​Lf2​σg2α​T​b\+16​Lf2​σg2​αβ​T​b\+4​Lf2​σg2​α\+16​Lf2​σg2​α2β\+32​Lf2​σg2​α3β\\displaystyle\\leq\\frac\{2\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{2L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+4L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{32L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\(56\)\+4​σf2​β\+8​Lf2​Cg2​n​η2​τα\+16​Lf2​Cg2​n​η2​τβ\+64​Lf2​Cg2​n​η2​α​τβ\.\\displaystyle\+4\\sigma\_\{f\}^\{2\}\\beta\+\\frac\{8L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{16L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{64L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\.

###### Proof\.

Condition on the history before drawing the fresh samples\. For theuu\-tracker,

𝔼​‖ut\+1k−gk​\(Wt\+1k\)‖2\\displaystyle\\mathbb\{E\}\\\|u\_\{t\+1\}^\{k\}\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\\|^\{2\}=𝔼∥\(1−α\)\(utk−gk\(Wtk\)\)\+\[gk\(Wt\+1k;ξt\+1k\)−gk\(Wtk;ξt\+1k\)−gk\(Wt\+1k\)\+gk\(Wtk\)\]\\displaystyle=\\mathbb\{E\}\\lVert\(1\-\\alpha\)\(u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\)\+\\bigl\[g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\+g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\+α\[gk\(Wtk;ξt\+1k\)−gk\(Wtk\)\]∥2\\displaystyle\\hskip 18\.49988pt\+\\alpha\\bigl\[g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\\rVert^\{2\}=\(1−α\)2𝔼∥utk−gk\(Wtk\)∥2\+𝔼∥\[gk\(Wt\+1k;ξt\+1k\)−gk\(Wtk;ξt\+1k\)−gk\(Wt\+1k\)\+gk\(Wtk\)\]\\displaystyle=\(1\-\\alpha\)^\{2\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\+\\mathbb\{E\}\\lVert\\bigl\[g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\+g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\+α\[gk\(Wtk;ξt\+1k\)−gk\(Wtk\)\]∥2\\displaystyle\\hskip 18\.49988pt\+\\alpha\\bigl\[g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\\rVert^\{2\}≤\(1−α\)​𝔼​‖utk−gk​\(Wtk\)‖2\+2​𝔼​‖gk​\(Wt\+1k,ξt\+1k\)−gk​\(Wtk,ξt\+1k\)−gk​\(Wt\+1k\)\+gk​\(Wtk\)‖2\\displaystyle\\leq\(1\-\\alpha\)\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\+2\\mathbb\{E\}\\\|g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\+1\}^\{k\}\)\+g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\+2​α2​𝔼​‖gk​\(Wtk,ξt\+1k\)−gk​\(Wtk\)‖2\\displaystyle\\hskip 18\.49988pt\+2\\alpha^\{2\}\\mathbb\{E\}\\\|g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}≤\(1−α\)​𝔼​‖utk−gk​\(Wtk\)‖2\+2​Cg2​𝔼​‖Wt\+1k−Wtk‖F2\+2​α2​σg2,\\displaystyle\\leq\(1\-\\alpha\)\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\+2C\_\{g\}^\{2\}\\mathbb\{E\}\\\|W\_\{t\+1\}^\{k\}\-W\_\{t\}^\{k\}\\\|\_\{F\}^\{2\}\+2\\alpha^\{2\}\\sigma\_\{g\}^\{2\},\(57\)where we used Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[6](https://arxiv.org/html/2608.12710#Thmassumption6)\. Averaging \([57](https://arxiv.org/html/2608.12710#A2.E57)\) overt=0,…,T−1t=0,\\ldots,T\-1andk∈\[K\]k\\in\[K\], and then using Lemma[8](https://arxiv.org/html/2608.12710#Thmlemma8), gives

1K​T​∑t=0T−1∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}≤1K​α​T​∑k=1K𝔼​‖u0k−gk​\(W0\)‖2\+2​Cg2α​K​T​∑t=0T−1∑k=1K𝔼​‖Wt\+1k−Wtk‖F2\+2​α​σg2\\displaystyle\\leq\\frac\{1\}\{K\\alpha T\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{0\}^\{k\}\-g^\{k\}\(W\_\{0\}\)\\\|^\{2\}\+\\frac\{2C\_\{g\}^\{2\}\}\{\\alpha KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{t\+1\}^\{k\}\-W\_\{t\}^\{k\}\\\|\_\{F\}^\{2\}\+2\\alpha\\sigma\_\{g\}^\{2\}≤1K​α​T​∑k=1K𝔼​∥1b​∑j=1b\(gk​\(W0,ξ0,jk\)−gk​\(W0\)\)∥2\+4​Cg2​n​η2​τα\+2​α​σg2\\displaystyle\\leq\\frac\{1\}\{K\\alpha T\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\frac\{1\}\{b\}\\sum\_\{j=1\}^\{b\}\(g^\{k\}\(W\_\{0\};\\xi\_\{0,j\}^\{k\}\)\-g^\{k\}\(W\_\{0\}\)\)\\rVert^\{2\}\+\\frac\{4C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+2\\alpha\\sigma\_\{g\}^\{2\}≤σg2α​T​b\+4​Cg2​n​η2​τα\+2​α​σg2\.\\displaystyle\\leq\\frac\{\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{4C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+2\\alpha\\sigma\_\{g\}^\{2\}\.\(58\)
For theHH\-tracker, Assumption[3](https://arxiv.org/html/2608.12710#Thmassumption3)and Jensen’s inequality imply‖∇gk​\(Wt\+1k\)‖F≤Cg\\\|\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\\|\_\{F\}\\leq C\_\{g\}, and henceΠCg​\[∇gk​\(Wt\+1k\)\]=∇gk​\(Wt\+1k\)\\Pi\_\{C\_\{g\}\}\[\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\]=\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\. The non\-expansiveness of the projection therefore gives

𝔼​‖Ht\+1k−∇gk​\(Wt\+1k\)‖F2\\displaystyle\\mathbb\{E\}\\\|H\_\{t\+1\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\\|\_\{F\}^\{2\}=𝔼​∥ΠCg​\[∇gk​\(Wt\+1k,ξt\+1k\)\+\(1−γ\)​\(Htk−∇gk​\(Wtk,ξt\+1k\)\)\]−ΠCg​\[∇gk​\(Wt\+1k\)\]∥F2\\displaystyle=\\mathbb\{E\}\\lVert\\Pi\_\{C\_\{g\}\}\\\!\\Bigl\[\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\+\(1\-\\gamma\)\\bigl\(H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\\bigr\)\\Bigr\]\-\\Pi\_\{C\_\{g\}\}\\\!\\bigl\[\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\bigr\]\\rVert\_\{F\}^\{2\}≤𝔼​∥∇gk​\(Wt\+1k,ξt\+1k\)\+\(1−γ\)​\(Htk−∇gk​\(Wtk,ξt\+1k\)\)−∇gk​\(Wt\+1k\)∥F2\\displaystyle\\leq\\mathbb\{E\}\\lVert\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\+\(1\-\\gamma\)\\bigl\(H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\\bigr\)\-\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\\rVert\_\{F\}^\{2\}=𝔼∥\(1−γ\)\(Htk−∇gk\(Wtk\)\)\+\[∇gk\(Wt\+1k;ξt\+1k\)−∇gk\(Wtk;ξt\+1k\)−∇gk\(Wt\+1k\)\+∇gk\(Wtk\)\]\\displaystyle=\\mathbb\{E\}\\lVert\(1\-\\gamma\)\(H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\)\+\\bigl\[\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\+\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\+γ\[∇gk\(Wtk;ξt\+1k\)−∇gk\(Wtk\)\]∥F2\\displaystyle\\hskip 18\.49988pt\+\\gamma\\bigl\[\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\\rVert\_\{F\}^\{2\}=\(1−γ\)2𝔼∥Htk−∇gk\(Wtk\)∥F2\+𝔼∥\[∇gk\(Wt\+1k;ξt\+1k\)−∇gk\(Wtk;ξt\+1k\)−∇gk\(Wt\+1k\)\+∇gk\(Wtk\)\]\\displaystyle=\(1\-\\gamma\)^\{2\}\\mathbb\{E\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}\+\\mathbb\{E\}\\lVert\\bigl\[\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\+1\}^\{k\}\)\+\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\+γ\[∇gk\(Wtk;ξt\+1k\)−∇gk\(Wtk\)\]∥F2\\displaystyle\\hskip 18\.49988pt\+\\gamma\\bigl\[\\nabla g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\bigr\]\\rVert\_\{F\}^\{2\}≤\(1−γ\)​𝔼​‖Htk−∇gk​\(Wtk\)‖F2\+2​Lg2​𝔼​‖Wt\+1k−Wtk‖F2\+2​γ2​σ∇g2,\\displaystyle\\leq\(1\-\\gamma\)\\mathbb\{E\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}\+2L\_\{g\}^\{2\}\\mathbb\{E\}\\\|W\_\{t\+1\}^\{k\}\-W\_\{t\}^\{k\}\\\|\_\{F\}^\{2\}\+2\\gamma^\{2\}\\sigma\_\{\\nabla g\}^\{2\},\(59\)where we used the non\-expansiveness of projection and Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[6](https://arxiv.org/html/2608.12710#Thmassumption6)\. Averaging \([59](https://arxiv.org/html/2608.12710#A2.E59)\) overt=0,…,T−1t=0,\\ldots,T\-1andk∈\[K\]k\\in\[K\], and then using Lemma[8](https://arxiv.org/html/2608.12710#Thmlemma8), gives

1K​T​∑t=0T−1∑k=1K𝔼​‖Htk−∇gk​\(Wtk\)‖F2\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}≤1K​γ​T​∑k=1K𝔼​‖H0k−∇gk​\(W0\)‖F2\+2​Lg2γ​K​T​∑t=0T−1∑k=1K𝔼​‖Wt\+1k−Wtk‖F2\+2​γ​σ∇g2\\displaystyle\\leq\\frac\{1\}\{K\\gamma T\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|H\_\{0\}^\{k\}\-\\nabla g^\{k\}\(W\_\{0\}\)\\\|\_\{F\}^\{2\}\+\\frac\{2L\_\{g\}^\{2\}\}\{\\gamma KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|W\_\{t\+1\}^\{k\}\-W\_\{t\}^\{k\}\\\|\_\{F\}^\{2\}\+2\\gamma\\sigma\_\{\\nabla g\}^\{2\}≤1K​γ​T​∑k=1K𝔼​∥1b​∑j=1b\(∇gk​\(W0,ξ0,jk\)−∇gk​\(W0\)\)∥F2\+4​Lg2​n​η2​τγ\+2​γ​σ∇g2\\displaystyle\\leq\\frac\{1\}\{K\\gamma T\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\frac\{1\}\{b\}\\sum\_\{j=1\}^\{b\}\\bigl\(\\nabla g^\{k\}\(W\_\{0\};\\xi\_\{0,j\}^\{k\}\)\-\\nabla g^\{k\}\(W\_\{0\}\)\\bigr\)\\rVert\_\{F\}^\{2\}\+\\frac\{4L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+2\\gamma\\sigma\_\{\\nabla g\}^\{2\}≤σ∇g2γ​T​b\+4​Lg2​n​η2​τγ\+2​γ​σ∇g2\.\\displaystyle\\leq\\frac\{\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{4L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+2\\gamma\\sigma\_\{\\nabla g\}^\{2\}\.\(60\)
For thevv\-tracker, Assumption[3](https://arxiv.org/html/2608.12710#Thmassumption3)and Jensen’s inequality imply‖∇yfk​\(ut\+1k\)‖≤Cf\\\|\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\\\|\\leq C\_\{f\}, and henceΠCf​\[∇yfk​\(ut\+1k\)\]=∇yfk​\(ut\+1k\)\\Pi\_\{C\_\{f\}\}\[\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\]=\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\. Thus, non\-expansiveness gives

𝔼​‖vt\+1k−∇yfk​\(ut\+1k\)‖2\\displaystyle\\mathbb\{E\}\\\|v\_\{t\+1\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\\\|^\{2\}=𝔼​∥ΠCf​\[∇yfk​\(ut\+1k,ζt\+1k\)\+\(1−β\)​\(vtk−∇yfk​\(utk,ζt\+1k\)\)\]−ΠCf​\[∇yfk​\(ut\+1k\)\]∥2\\displaystyle=\\mathbb\{E\}\\lVert\\Pi\_\{C\_\{f\}\}\\\!\\Bigl\[\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\+\(1\-\\beta\)\\bigl\(v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\\bigr\)\\Bigr\]\-\\Pi\_\{C\_\{f\}\}\\\!\\bigl\[\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\\bigr\]\\rVert^\{2\}≤𝔼​∥∇yfk​\(ut\+1k,ζt\+1k\)\+\(1−β\)​\(vtk−∇yfk​\(utk,ζt\+1k\)\)−∇yfk​\(ut\+1k\)∥2\\displaystyle\\leq\\mathbb\{E\}\\lVert\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\+\(1\-\\beta\)\\bigl\(v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\\bigr\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\\rVert^\{2\}=𝔼∥\(1−β\)\(vtk−∇yfk\(utk\)\)\+\[∇yfk\(ut\+1k;ζt\+1k\)−∇yfk\(utk;ζt\+1k\)−∇yfk\(ut\+1k\)\+∇yfk\(utk\)\]\\displaystyle=\\mathbb\{E\}\\lVert\(1\-\\beta\)\(v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\)\+\\bigl\[\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\+1\}^\{k\}\)\+\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\bigr\]\+β\[∇yfk\(utk;ζt\+1k\)−∇yfk\(utk\)\]∥2\\displaystyle\\hskip 18\.49988pt\+\\beta\\bigl\[\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\};\\zeta\_\{t\+1\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\bigr\]\\rVert^\{2\}≤\(1−β\)​𝔼​‖vtk−∇yfk​\(utk\)‖2\+2​Lf2​𝔼​‖ut\+1k−utk‖2\+2​β2​σf2,\\displaystyle\\leq\(1\-\\beta\)\\mathbb\{E\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\\|^\{2\}\+2L\_\{f\}^\{2\}\\mathbb\{E\}\\\|u\_\{t\+1\}^\{k\}\-u\_\{t\}^\{k\}\\\|^\{2\}\+2\\beta^\{2\}\\sigma\_\{f\}^\{2\},\(61\)𝔼​‖ut\+1k−utk‖2\\displaystyle\\mathbb\{E\}\\\|u\_\{t\+1\}^\{k\}\-u\_\{t\}^\{k\}\\\|^\{2\}=𝔼​∥gk​\(Wt\+1k,ξt\+1k\)−gk​\(Wtk,ξt\+1k\)\+α⁡\(gk​\(Wtk,ξt\+1k\)−utk\)∥2\\displaystyle=\\mathbb\{E\}\\lVert g^\{k\}\(W\_\{t\+1\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\+\\alpha\(g^\{k\}\(W\_\{t\}^\{k\};\\xi\_\{t\+1\}^\{k\}\)\-u\_\{t\}^\{k\}\)\\rVert^\{2\}≤2​Cg2​𝔼​‖Wt\+1k−Wtk‖F2\+4​α2​𝔼​‖utk−gk​\(Wtk\)‖2\+4​α2​σg2\.\\displaystyle\\leq 2C\_\{g\}^\{2\}\\mathbb\{E\}\\\|W\_\{t\+1\}^\{k\}\-W\_\{t\}^\{k\}\\\|\_\{F\}^\{2\}\+4\\alpha^\{2\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}\+4\\alpha^\{2\}\\sigma\_\{g\}^\{2\}\.\(62\)Combining the last two displays, averaging overttandkk, and substituting \([58](https://arxiv.org/html/2608.12710#A2.E58)\) gives

1K​T​∑t=0T−1∑k=1K𝔼​‖vtk−∇yfk​\(utk\)‖2\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\\|^\{2\}≤1K​β​T​∑k=1K𝔼​∥1b​∑j=1b\(∇yfk​\(u0k,ζ0,jk\)−∇yfk​\(u0k\)\)∥2\+8​Lf2​Cg2​n​η2​τβ\\displaystyle\\leq\\frac\{1\}\{K\\beta T\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\frac\{1\}\{b\}\\sum\_\{j=1\}^\{b\}\\bigl\(\\nabla\_\{y\}f^\{k\}\(u\_\{0\}^\{k\};\\zeta\_\{0,j\}^\{k\}\)\-\\nabla\_\{y\}f^\{k\}\(u\_\{0\}^\{k\}\)\\bigr\)\\rVert^\{2\}\+\\frac\{8L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+8​Lf2​α2β​\(σg2α​T​b\+2​σg2​α\+4​Cg2​n​η2​τα\)\+8​Lf2​σg2​α2β\+2​σf2​β\\displaystyle\\hskip 18\.49988pt\+\\frac\{8L\_\{f\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\\left\(\\frac\{\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+2\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{4C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\\right\)\+\\frac\{8L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+2\\sigma\_\{f\}^\{2\}\\beta≤σf2β​T​b\+8​Lf2​σg2​αβ​T​b\+8​Lf2​σg2​α2β\+16​Lf2​σg2​α3β\+2​σf2​β\\displaystyle\\leq\\frac\{\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{8L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+\\frac\{8L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+2\\sigma\_\{f\}^\{2\}\\beta\+8​Lf2​Cg2​n​η2​τβ\+32​Lf2​Cg2​n​η2​α​τβ\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{8L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{32L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\.\(63\)Averaging the Lipschitz bound‖vtk−∇yfk​\(gk​\(Wtk\)\)‖2≤2​‖vtk−∇yfk​\(utk\)‖2\+2​Lf2​‖utk−gk​\(Wtk\)‖2\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\\|^\{2\}\\leq 2\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\\|^\{2\}\+2L\_\{f\}^\{2\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}and using \([58](https://arxiv.org/html/2608.12710#A2.E58)\) and \([63](https://arxiv.org/html/2608.12710#A2.E63)\) yields

1K​T​∑t=0T−1∑k=1K𝔼​‖vtk−∇yfk​\(gk​\(Wtk\)\)‖2\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\\|^\{2\}≤2​1K​T​∑t=0T−1∑k=1K𝔼​‖vtk−∇yfk​\(utk\)‖2\+2​Lf2​1K​T​∑t=0T−1∑k=1K𝔼​‖utk−gk​\(Wtk\)‖2\\displaystyle\\leq 2\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(u\_\{t\}^\{k\}\)\\\|^\{2\}\+2L\_\{f\}^\{2\}\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|u\_\{t\}^\{k\}\-g^\{k\}\(W\_\{t\}^\{k\}\)\\\|^\{2\}≤2​σf2β​T​b\+2​Lf2​σg2α​T​b\+16​Lf2​σg2​αβ​T​b\+4​Lf2​σg2​α\+16​Lf2​σg2​α2β\+32​Lf2​σg2​α3β\+4​σf2​β\\displaystyle\\leq\\frac\{2\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{2L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+4L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{32L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+4\\sigma\_\{f\}^\{2\}\\beta\+8​Lf2​Cg2​n​η2​τα\+16​Lf2​Cg2​n​η2​τβ\+64​Lf2​Cg2​n​η2​α​τβ,\\displaystyle\\hskip 18\.49988pt\\hskip 9\.24994pt\+\\frac\{8L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{16L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{64L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\},\(64\)where we used the non\-expansiveness of projection, theLfL\_\{f\}\-smoothness, and the precedinguu\-tracker bound\. ∎

###### Lemma 10\(Product estimator at the averaged model\)\.

1T​∑t=0T−1𝔼​∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F2\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\\bigl\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\\bigr\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}^\{2\}≤4​Cf2​σ∇g2γ​T​b\+8​Cg2​σf2β​T​b\+8​Cg2​Lf2​σg2α​T​b\+64​Cg2​Lf2​σg2​αβ​T​b\+16​Cg2​Lf2​σg2​α\+64​Cg2​Lf2​σg2​α2β\\displaystyle\\hskip 9\.24994pt\\leq\\frac\{4C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{8C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{8C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+16C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+128​Cg2​Lf2​σg2​α3β\+16​Cg2​σf2​β\+8​Cf2​σ∇g2​γ\+32​Cg4​Lf2​n​η2​τα\+64​Cg4​Lf2​n​η2​τβ\\displaystyle\\hskip 18\.49988pt\+\\frac\{128C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+16C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+8C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{32C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{64C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+256​Cg4​Lf2​n​η2​α​τβ\+16​Cf2​Lg2​n​η2​τγ\+4​n​\(Cf2​Lg2\+Cg4​Lf2\)​η2​τ2\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{256C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+\\frac\{16C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+4n\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\)\\eta^\{2\}\\tau^\{2\}\.\(65\)

###### Proof\.

The projection steps imply‖Htk‖F≤Cg\\\|H\_\{t\}^\{k\}\\\|\_\{F\}\\leq C\_\{g\}and‖vtk‖≤Cf\\\|v\_\{t\}^\{k\}\\\|\\leq C\_\{f\}for allt≥0t\\geq 0andk∈\[K\]k\\in\[K\]\. Since∇F​\(W¯t\)=K−1​∑k=1K∇gk​\(W¯t\)​\(∇yfk​\(gk​\(W¯t\)\)⊗In\)\\nabla F\(\\bar\{W\}\_\{t\}\)=K^\{\-1\}\\sum\_\{k=1\}^\{K\}\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(\\bar\{W\}\_\{t\}\)\)\\otimes I\_\{n\}\\bigr\), we have the expanded product\-estimator error

∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F2\\displaystyle\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}^\{2\}=∥1K​∑k=1KHtk​\(vtk⊗In\)−1K​∑k=1K∇gk​\(W¯t\)​\(∇yfk​\(gk​\(W¯t\)\)⊗In\)∥F2\\displaystyle=\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(\\bar\{W\}\_\{t\}\)\)\\otimes I\_\{n\}\\bigr\)\\rVert\_\{F\}^\{2\}=∥1K​∑k=1K\[Htk​\(vtk⊗In\)−∇gk​\(W¯t\)​\(vtk⊗In\)\]\+1K​∑k=1K\[∇gk​\(W¯t\)​\(vtk⊗In\)−∇gk​\(W¯t\)​\(∇yfk​\(gk​\(W¯t\)\)⊗In\)\]∥F2\\displaystyle=\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigl\[H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\\Bigr\]\+\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\Bigl\[\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(\\bar\{W\}\_\{t\}\)\)\\otimes I\_\{n\}\\bigr\)\\Bigr\]\\rVert\_\{F\}^\{2\}≤2K​∑k=1K∥\(Htk−∇gk​\(W¯t\)\)​\(vtk⊗In\)∥F2\+2K​∑k=1K∥∇gk​\(W¯t\)​\(\(vtk−∇yfk​\(gk​\(W¯t\)\)\)⊗In\)∥F2\\displaystyle\\leq\\frac\{2\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\bigl\(H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\\bigr\)\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\\rVert\_\{F\}^\{2\}\+\\frac\{2\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\\bigl\(\(v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(\\bar\{W\}\_\{t\}\)\)\)\\otimes I\_\{n\}\\bigr\)\\rVert\_\{F\}^\{2\}≤2​Cf2K​∑k=1K‖Htk−∇gk​\(W¯t\)‖F2\+2​Cg2K​∑k=1K‖vtk−∇yfk​\(gk​\(W¯t\)\)‖2\\displaystyle\\leq\\frac\{2C\_\{f\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}^\{2\}\+\\frac\{2C\_\{g\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(\\bar\{W\}\_\{t\}\)\)\\\|^\{2\}≤4​Cf2K​∑k=1K‖Htk−∇gk​\(Wtk\)‖F2\+4​Cf2​Lg2K​∑k=1K‖Wtk−W¯t‖F2\+4​Cg2K​∑k=1K‖vtk−∇yfk​\(gk​\(Wtk\)\)‖2\\displaystyle\\leq\\frac\{4C\_\{f\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}\+\\frac\{4C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}^\{2\}\+\\frac\{4C\_\{g\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\\|^\{2\}\+4​Cg4​Lf2K∑k=1K∥Wtk−W¯t∥F2\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}^\{2\}\.\(66\)Averaging this display overttand using Lemmas[8](https://arxiv.org/html/2608.12710#Thmlemma8)and[9](https://arxiv.org/html/2608.12710#Thmlemma9)gives

1T​∑t=0T−1𝔼​∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F2\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\\bigl\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\\bigr\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}^\{2\}≤4​Cf2​σ∇g2γ​T​b\+8​Cg2​σf2β​T​b\+8​Cg2​Lf2​σg2α​T​b\+64​Cg2​Lf2​σg2​αβ​T​b\+16​Cg2​Lf2​σg2​α\+64​Cg2​Lf2​σg2​α2β\\displaystyle\\hskip 9\.24994pt\\leq\\frac\{4C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{8C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{8C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+16C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+128​Cg2​Lf2​σg2​α3β\+16​Cg2​σf2​β\+8​Cf2​σ∇g2​γ\+32​Cg4​Lf2​n​η2​τα\+64​Cg4​Lf2​n​η2​τβ\\displaystyle\\hskip 18\.49988pt\+\\frac\{128C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+16C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+8C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{32C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{64C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+256​Cg4​Lf2​n​η2​α​τβ\+16​Cf2​Lg2​n​η2​τγ\+4​n​\(Cf2​Lg2\+Cg4​Lf2\)​η2​τ2,\\displaystyle\\hskip 18\.49988pt\+\\frac\{256C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+\\frac\{16C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+4n\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\)\\eta^\{2\}\\tau^\{2\},\(67\)where we used the projection bounds and Lemmas[8](https://arxiv.org/html/2608.12710#Thmlemma8)and[9](https://arxiv.org/html/2608.12710#Thmlemma9)\. ∎

###### Lemma 11\(Client product disagreement under non\-i\.i\.d\. data\)\.

LetFk​\(W\):=fk​\(gk​\(W\)\)F^\{k\}\(W\):=f^\{k\}\(g^\{k\}\(W\)\)andF⁡\(W\):=K−1​∑k=1KFk​\(W\)F\(W\):=K^\{\-1\}\\sum\_\{k=1\}^\{K\}F^\{k\}\(W\), and setLF:=Cf​Lg\+Cg2​LfL\_\{F\}:=C\_\{f\}L\_\{g\}\+C\_\{g\}^\{2\}L\_\{f\}\.

1T​∑t=0T−11K​∑k=1K𝔼​∥Htk​\(vtk⊗In\)−1K​∑j=1KHtj​\(vtj⊗In\)∥F2\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}H\_\{t\}^\{j\}\(v\_\{t\}^\{j\}\\otimes I\_\{n\}\)\\rVert\_\{F\}^\{2\}≤6​Cf2​σ∇g2γ​T​b\+12​Cg2​σf2β​T​b\+12​Cg2​Lf2​σg2α​T​b\+96​Cg2​Lf2​σg2​αβ​T​b\+24​Cg2​Lf2​σg2​α\+96​Cg2​Lf2​σg2​α2β\\displaystyle\\hskip 9\.24994pt\\leq\\frac\{6C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{12C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+24C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+192​Cg2​Lf2​σg2​α3β\+24​Cg2​σf2​β\+12​Cf2​σ∇g2​γ\+48​Cg4​Lf2​n​η2​τα\+96​Cg4​Lf2​n​η2​τβ\\displaystyle\\hskip 18\.49988pt\+\\frac\{192C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+24C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+12C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{48C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{96C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+384​Cg4​Lf2​n​η2​α​τβ\+24​Cf2​Lg2​n​η2​τγ\+3​LF2​n​η2​τ2\+6​Cf2​Δ∇g2\+12​Cg2​Δf2\+12​Cg2​Lf2​Δg2\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{384C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+\\frac\{24C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+3L\_\{F\}^\{2\}n\\eta^\{2\}\\tau^\{2\}\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\.\(68\)

###### Proof\.

By Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[3](https://arxiv.org/html/2608.12710#Thmassumption3), Jensen’s inequality gives

‖∇gk​\(W\)‖F≤Cg,‖∇yfk​\(y\)‖≤Cf\.\\\|\\nabla g^\{k\}\(W\)\\\|\_\{F\}\\leq C\_\{g\},\\hskip 18\.49988pt\\\|\\nabla\_\{y\}f^\{k\}\(y\)\\\|\\leq C\_\{f\}\.Moreover, Assumptions[2](https://arxiv.org/html/2608.12710#Thmassumption2)and[6](https://arxiv.org/html/2608.12710#Thmassumption6), together with Jensen’s inequality, imply

‖gk​\(W\)−gk​\(W′\)‖\\displaystyle\\\|g^\{k\}\(W\)\-g^\{k\}\(W^\{\\prime\}\)\\\|≤Cg​‖W−W′‖F,\\displaystyle\\leq C\_\{g\}\\\|W\-W^\{\\prime\}\\\|\_\{F\},\(69\)‖∇gk​\(W\)−∇gk​\(W′\)‖F\\displaystyle\\\|\\nabla g^\{k\}\(W\)\-\\nabla g^\{k\}\(W^\{\\prime\}\)\\\|\_\{F\}≤Lg​‖W−W′‖F,\\displaystyle\\leq L\_\{g\}\\\|W\-W^\{\\prime\}\\\|\_\{F\},‖∇yfk​\(y\)−∇yfk​\(y′\)‖\\displaystyle\\\|\\nabla\_\{y\}f^\{k\}\(y\)\-\\nabla\_\{y\}f^\{k\}\(y^\{\\prime\}\)\\\|≤Lf​‖y−y′‖\.\\displaystyle\\leq L\_\{f\}\\\|y\-y^\{\\prime\}\\\|\.Therefore, for anyW,W′∈ℝm×nW,W^\{\\prime\}\\in\\mathbb\{R\}^\{m\\times n\},

‖∇Fk​\(W\)−∇Fk​\(W′\)‖F\\displaystyle\\\|\\nabla F^\{k\}\(W\)\-\\nabla F^\{k\}\(W^\{\\prime\}\)\\\|\_\{F\}=∥∇gk​\(W\)​\(∇yfk​\(gk​\(W\)\)⊗In\)−∇gk​\(W′\)​\(∇yfk​\(gk​\(W′\)\)⊗In\)∥F\\displaystyle=\\lVert\\nabla g^\{k\}\(W\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\\otimes I\_\{n\}\\bigr\)\-\\nabla g^\{k\}\(W^\{\\prime\}\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W^\{\\prime\}\)\)\\otimes I\_\{n\}\\bigr\)\\rVert\_\{F\}≤‖∇gk​\(W\)−∇gk​\(W′\)‖F​‖∇yfk​\(gk​\(W\)\)‖\+‖∇gk​\(W′\)‖F​‖∇yfk​\(gk​\(W\)\)−∇yfk​\(gk​\(W′\)\)‖\\displaystyle\\leq\\\|\\nabla g^\{k\}\(W\)\-\\nabla g^\{k\}\(W^\{\\prime\}\)\\\|\_\{F\}\\\|\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\\\|\+\\\|\\nabla g^\{k\}\(W^\{\\prime\}\)\\\|\_\{F\}\\\|\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W^\{\\prime\}\)\)\\\|≤Cf​Lg​‖W−W′‖F\+Cg​Lf​‖gk​\(W\)−gk​\(W′\)‖\\displaystyle\\leq C\_\{f\}L\_\{g\}\\\|W\-W^\{\\prime\}\\\|\_\{F\}\+C\_\{g\}L\_\{f\}\\\|g^\{k\}\(W\)\-g^\{k\}\(W^\{\\prime\}\)\\\|≤Cf​Lg​‖W−W′‖F\+Cg2​Lf​‖W−W′‖F=LF​‖W−W′‖F,\\displaystyle\\leq C\_\{f\}L\_\{g\}\\\|W\-W^\{\\prime\}\\\|\_\{F\}\+C\_\{g\}^\{2\}L\_\{f\}\\\|W\-W^\{\\prime\}\\\|\_\{F\}=L\_\{F\}\\\|W\-W^\{\\prime\}\\\|\_\{F\},\(70\)where we used Assumptions[3](https://arxiv.org/html/2608.12710#Thmassumption3)and[6](https://arxiv.org/html/2608.12710#Thmassumption6)\. Averaging over clients also gives

‖∇F​\(W\)−∇F​\(W′\)‖F≤1K​∑k=1K‖∇Fk​\(W\)−∇Fk​\(W′\)‖F≤LF​‖W−W′‖F\.\\\|\\nabla F\(W\)\-\\nabla F\(W^\{\\prime\}\)\\\|\_\{F\}\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F^\{k\}\(W\)\-\\nabla F^\{k\}\(W^\{\\prime\}\)\\\|\_\{F\}\\leq L\_\{F\}\\\|W\-W^\{\\prime\}\\\|\_\{F\}\.\(71\)The non\-i\.i\.d\. term is controlled once for later substitution\. For anyW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}andk,j∈\[K\]k,j\\in\[K\], the above population bounds together with Assumption[7](https://arxiv.org/html/2608.12710#Thmassumption7)give

∥∇Fk​\(W\)−∇Fj​\(W\)∥F2\\displaystyle\\lVert\\nabla F^\{k\}\(W\)\-\\nabla F^\{j\}\(W\)\\rVert\_\{F\}^\{2\}≤2​∥\(∇gk​\(W\)−∇gj​\(W\)\)​\(∇yfk​\(gk​\(W\)\)⊗In\)∥F2\+2​∥∇gj​\(W\)​\(\(∇yfk​\(gk​\(W\)\)−∇yfj​\(gj​\(W\)\)\)⊗In\)∥F2\\displaystyle\\leq 2\\lVert\\bigl\(\\nabla g^\{k\}\(W\)\-\\nabla g^\{j\}\(W\)\\bigr\)\\bigl\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\\otimes I\_\{n\}\\bigr\)\\rVert\_\{F\}^\{2\}\+2\\lVert\\nabla g^\{j\}\(W\)\\bigl\(\(\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\-\\nabla\_\{y\}f^\{j\}\(g^\{j\}\(W\)\)\)\\otimes I\_\{n\}\\bigr\)\\rVert\_\{F\}^\{2\}≤2​Cf2​Δ∇g2\+2​Cg2​‖∇yfk​\(gk​\(W\)\)−∇yfj​\(gj​\(W\)\)‖2\\displaystyle\\leq 2C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+2C\_\{g\}^\{2\}\\\|\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\)\)\-\\nabla\_\{y\}f^\{j\}\(g^\{j\}\(W\)\)\\\|^\{2\}≤2​Cf2​Δ∇g2\+4​Cg2​Δf2\+4​Cg2​Lf2​Δg2,\\displaystyle\\leq 2C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+4C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+4C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\},\(72\)and Jensen’s inequality yields

1K​∑k=1K‖∇Fk​\(W\)−∇F​\(W\)‖F2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F^\{k\}\(W\)\-\\nabla F\(W\)\\\|\_\{F\}^\{2\}=1K​∑k=1K∥1K​∑j=1K\(∇Fk​\(W\)−∇Fj​\(W\)\)∥F2\\displaystyle=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\lVert\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\bigl\(\\nabla F^\{k\}\(W\)\-\\nabla F^\{j\}\(W\)\\bigr\)\\rVert\_\{F\}^\{2\}≤1K2​∑k=1K∑j=1K‖∇Fk​\(W\)−∇Fj​\(W\)‖F2\\displaystyle\\leq\\frac\{1\}\{K^\{2\}\}\\sum\_\{k=1\}^\{K\}\\sum\_\{j=1\}^\{K\}\\\|\\nabla F^\{k\}\(W\)\-\\nabla F^\{j\}\(W\)\\\|\_\{F\}^\{2\}≤2​Cf2​Δ∇g2\+4​Cg2​Δf2\+4​Cg2​Lf2​Δg2,\\displaystyle\\leq 2C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+4C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+4C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\},\(73\)where we used Assumption[7](https://arxiv.org/html/2608.12710#Thmassumption7), theLfL\_\{f\}\-smoothness, and the bounded\-gradient assumptions\.

LetZtk=Htk​\(vtk⊗In\)Z\_\{t\}^\{k\}=H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)andZ¯t=K−1​∑j=1KZtj\\bar\{Z\}\_\{t\}=K^\{\-1\}\\sum\_\{j=1\}^\{K\}Z\_\{t\}^\{j\}\. SinceZ¯t\\bar\{Z\}\_\{t\}is the Euclidean mean of\{Ztk\}k=1K\\\{Z\_\{t\}^\{k\}\\\}\_\{k=1\}^\{K\}, the previous bounds imply, for everytt,

1K​∑k=1K‖Ztk−Z¯t‖F2\\displaystyle\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|Z\_\{t\}^\{k\}\-\\bar\{Z\}\_\{t\}\\\|\_\{F\}^\{2\}≤1K​∑k=1K‖Ztk−∇F​\(W¯t\)‖F2\\displaystyle\\leq\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|Z\_\{t\}^\{k\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}^\{2\}≤3K​∑k=1K‖Ztk−∇Fk​\(Wtk\)‖F2\+3K​∑k=1K‖∇Fk​\(Wtk\)−∇Fk​\(W¯t\)‖F2\\displaystyle\\leq\\frac\{3\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|Z\_\{t\}^\{k\}\-\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}\+\\frac\{3\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F^\{k\}\(W\_\{t\}^\{k\}\)\-\\nabla F^\{k\}\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}^\{2\}\+3K∑k=1K∥∇Fk\(W¯t\)−∇F\(W¯t\)∥F2\\displaystyle\\hskip 18\.49988pt\+\\frac\{3\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F^\{k\}\(\\bar\{W\}\_\{t\}\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}^\{2\}≤6​Cf2K​∑k=1K‖Htk−∇gk​\(Wtk\)‖F2\+6​Cg2K​∑k=1K‖vtk−∇yfk​\(gk​\(Wtk\)\)‖2\\displaystyle\\leq\\frac\{6C\_\{f\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|H\_\{t\}^\{k\}\-\\nabla g^\{k\}\(W\_\{t\}^\{k\}\)\\\|\_\{F\}^\{2\}\+\\frac\{6C\_\{g\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|v\_\{t\}^\{k\}\-\\nabla\_\{y\}f^\{k\}\(g^\{k\}\(W\_\{t\}^\{k\}\)\)\\\|^\{2\}\+3​LF2K∑k=1K∥Wtk−W¯t∥F2\+6Cf2Δ∇g2\+12Cg2Δf2\+12Cg2Lf2Δg2,\\displaystyle\\hskip 18\.49988pt\+\\frac\{3L\_\{F\}^\{2\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|W\_\{t\}^\{k\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}^\{2\}\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\},\(74\)where we used the optimality of the Euclidean mean and the preceding smoothness and heterogeneity bounds\. Averaging this inequality overtt, taking expectations, and applying Lemmas[8](https://arxiv.org/html/2608.12710#Thmlemma8)and[9](https://arxiv.org/html/2608.12710#Thmlemma9), we obtain

1T​∑t=0T−11K​∑k=1K𝔼​‖Ztk−Z¯t‖F2\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|Z\_\{t\}^\{k\}\-\\bar\{Z\}\_\{t\}\\\|\_\{F\}^\{2\}≤6​Cf2​\(σ∇g2γ​T​b\+2​σ∇g2​γ\+4​Lg2​n​η2​τγ\)\+6​Cg2​\(2​σf2β​T​b\+2​Lf2​σg2α​T​b\+16​Lf2​σg2​αβ​T​b\+4​Lf2​σg2​αCLOSE\\displaystyle\\leq 6C\_\{f\}^\{2\}\\left\(\\frac\{\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+2\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{4L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\\right\)\+6C\_\{g\}^\{2\}\\Bigg\(\\frac\{2\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{2L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+4L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alphaOPEN\+16​Lf2​σg2​α2β\+32​Lf2​σg2​α3β\+4​σf2​β\+8​Lf2​Cg2​n​η2​τα\+16​Lf2​Cg2​n​η2​τβ\+64​Lf2​Cg2​n​η2​α​τβ\)\\displaystyle\\hskip 18\.49988pt\+\\frac\{16L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{32L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+4\\sigma\_\{f\}^\{2\}\\beta\+\\frac\{8L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{16L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{64L\_\{f\}^\{2\}C\_\{g\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\\Bigg\)\+3​LF2​n​η2​τ2\+6​Cf2​Δ∇g2\+12​Cg2​Δf2\+12​Cg2​Lf2​Δg2\\displaystyle\\hskip 18\.49988pt\+3L\_\{F\}^\{2\}n\\eta^\{2\}\\tau^\{2\}\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}=6​Cf2​σ∇g2γ​T​b\+12​Cg2​σf2β​T​b\+12​Cg2​Lf2​σg2α​T​b\+96​Cg2​Lf2​σg2​αβ​T​b\+24​Cg2​Lf2​σg2​α\+96​Cg2​Lf2​σg2​α2β\+192​Cg2​Lf2​σg2​α3β\\displaystyle=\\frac\{6C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{12C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+24C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{192C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+24​Cg2​σf2​β\+12​Cf2​σ∇g2​γ\+48​Cg4​Lf2​n​η2​τα\+96​Cg4​Lf2​n​η2​τβ\+384​Cg4​Lf2​n​η2​α​τβ\\displaystyle\\hskip 18\.49988pt\+24C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+12C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{48C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{96C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{384C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+24​Cf2​Lg2​n​η2​τγ\+3​LF2​n​η2​τ2\+6​Cf2​Δ∇g2\+12​Cg2​Δf2\+12​Cg2​Lf2​Δg2\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{24C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+3L\_\{F\}^\{2\}n\\eta^\{2\}\\tau^\{2\}\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\.\(75\)Substituting the definition ofZtkZ\_\{t\}^\{k\}gives the stated bound\. ∎

###### Lemma 12\(Average product tracker error\)\.

Assume0<ρ<10<\\rho<1and letM¯t:=K−1​∑k=1KMtk\\bar\{M\}\_\{t\}:=K^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{t\}^\{k\}\. Then

1T​∑t=0T−1𝔼​‖M¯t−∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤4​Cg​Cfρ​T\+LF​n​ηρ\+\[4​Cf2​σ∇g2γ​T​b\+8​Cg2​σf2β​T​b\+8​Cg2​Lf2​σg2α​T​b\+64​Cg2​Lf2​σg2​αβ​T​b\+16Cg2Lf2σg2α\+64​Cg2​Lf2​σg2​α2β\\displaystyle\\leq\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\\Bigg\[\\frac\{4C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{8C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{8C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+16C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+128​Cg2​Lf2​σg2​α3β\+16​Cg2​σf2​β\+8​Cf2​σ∇g2​γ\+32​Cg4​Lf2​n​η2​τα\+64​Cg4​Lf2​n​η2​τβ\+256​Cg4​Lf2​n​η2​α​τβ\\displaystyle\\hskip 18\.49988pt\+\\frac\{128C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+16C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+8C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{32C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{64C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{256C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+16​Cf2​Lg2​n​η2​τγ\+4n\(Cf2Lg2\+Cg4Lf2\)η2τ2\]1/2\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{16C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+4n\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\)\\eta^\{2\}\\tau^\{2\}\\Bigg\]^\{1/2\}\.\(76\)

###### Proof\.

‖M¯t\+1−∇F​\(W¯t\+1\)‖F\\displaystyle\\\|\\bar\{M\}\_\{t\+1\}\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\\|\_\{F\}=∥\(1−ρ\)​\(M¯t−∇F​\(W¯t\)\)\+ρ⁡\[1K​∑k=1KHt\+1k​\(vt\+1k⊗In\)−∇F​\(W¯t\+1\)\]\+\(1−ρ\)​\(∇F​\(W¯t\)−∇F​\(W¯t\+1\)\)∥F\\displaystyle=\\lVert\(1\-\\rho\)\\bigl\(\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\bigr\)\+\\rho\\left\[\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\right\]\+\(1\-\\rho\)\\bigl\(\\nabla F\(\\bar\{W\}\_\{t\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\bigr\)\\rVert\_\{F\}≤\(1−ρ\)​‖M¯t−∇F​\(W¯t\)‖F\+ρ​∥1K​∑k=1KHt\+1k​\(vt\+1k⊗In\)−∇F​\(W¯t\+1\)∥F\+\(1−ρ\)​‖∇F​\(W¯t\)−∇F​\(W¯t\+1\)‖F\\displaystyle\\leq\(1\-\\rho\)\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\rho\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\rVert\_\{F\}\+\(1\-\\rho\)\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\\|\_\{F\}≤\(1−ρ\)​‖M¯t−∇F​\(W¯t\)‖F\+ρ​∥1K​∑k=1KHt\+1k​\(vt\+1k⊗In\)−∇F​\(W¯t\+1\)∥F\+LF​‖W¯t\+1−W¯t‖F\\displaystyle\\leq\(1\-\\rho\)\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\rho\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\rVert\_\{F\}\+L\_\{F\}\\\|\\bar\{W\}\_\{t\+1\}\-\\bar\{W\}\_\{t\}\\\|\_\{F\}≤\(1−ρ\)​‖M¯t−∇F​\(W¯t\)‖F\+ρ​∥1K​∑k=1KHt\+1k​\(vt\+1k⊗In\)−∇F​\(W¯t\+1\)∥F\+LF​ηK​∑k=1K‖Q⁡\(Mtk\)‖F\\displaystyle\\leq\(1\-\\rho\)\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\rho\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\rVert\_\{F\}\+\\frac\{L\_\{F\}\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|Q\(M\_\{t\}^\{k\}\)\\\|\_\{F\}≤\(1−ρ\)​‖M¯t−∇F​\(W¯t\)‖F\+ρ​∥1K​∑k=1KHt\+1k​\(vt\+1k⊗In\)−∇F​\(W¯t\+1\)∥F\+LF​η​n,\\displaystyle\\leq\(1\-\\rho\)\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\rho\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\rVert\_\{F\}\+L\_\{F\}\\eta\\sqrt\{n\},\(77\)where we used theLFL\_\{F\}\-smoothness ofFF, the triangle inequality, and‖Utk​\(Vtk\)⊤‖F≤n\\\|U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\\|\_\{F\}\\leq\\sqrt\{n\}\. Consequently,

1T​∑t=0T−1𝔼​‖M¯t−∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤1ρ​T​∑t=0T−1𝔼​‖M¯t−∇F​\(W¯t\)‖F−1ρ​T​∑t=0T−1𝔼​‖M¯t\+1−∇F​\(W¯t\+1\)‖F\\displaystyle\\leq\\frac\{1\}\{\\rho T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\-\\frac\{1\}\{\\rho T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\+1\}\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\\|\_\{F\}\+1T∑t=0T−1𝔼∥1K∑k=1KHt\+1k\(vt\+1k⊗In\)−∇F\(W¯t\+1\)∥F\+LF​η​nρ\\displaystyle\\hskip 9\.24994pt\+\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\+1\}^\{k\}\(v\_\{t\+1\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\+1\}\)\\rVert\_\{F\}\+\\frac\{L\_\{F\}\\eta\\sqrt\{n\}\}\{\\rho\}≤𝔼​‖M¯0−∇F​\(W0\)‖Fρ​T\+1T​∑t=1T𝔼​∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F\+LF​η​nρ\\displaystyle\\leq\\frac\{\\mathbb\{E\}\\\|\\bar\{M\}\_\{0\}\-\\nabla F\(W\_\{0\}\)\\\|\_\{F\}\}\{\\rho T\}\+\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}\+\\frac\{L\_\{F\}\\eta\\sqrt\{n\}\}\{\\rho\}≤2​Cg​Cfρ​T\+2​Cg​CfT\+1T​∑t=0T−1𝔼​∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F\+LF​η​nρ\\displaystyle\\leq\\frac\{2C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{2C\_\{g\}C\_\{f\}\}\{T\}\+\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}\+\\frac\{L\_\{F\}\\eta\\sqrt\{n\}\}\{\\rho\}≤4​Cg​Cfρ​T\+LF​η​nρ\+1T​∑t=0T−1𝔼​∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F\.\\displaystyle\\leq\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\eta\\sqrt\{n\}\}\{\\rho\}\+\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}\.\(78\)By Cauchy–Schwarz and Lemma[10](https://arxiv.org/html/2608.12710#Thmlemma10), the remaining average satisfies

1T​∑t=0T−1𝔼​∥1K​∑k=1KHtk​\(vtk⊗In\)−∇F​\(W¯t\)∥F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}H\_\{t\}^\{k\}\(v\_\{t\}^\{k\}\\otimes I\_\{n\}\)\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\rVert\_\{F\}≤\\displaystyle\\leq\{\}\[4​Cf2​σ∇g2γ​T​b\+8​Cg2​σf2β​T​b\+8​Cg2​Lf2​σg2α​T​b\+64​Cg2​Lf2​σg2​αβ​T​b\+16Cg2Lf2σg2α\\displaystyle\\Bigg\[\\frac\{4C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{8C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{8C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+16C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+64​Cg2​Lf2​σg2​α2β\+128​Cg2​Lf2​σg2​α3β\+16​Cg2​σf2​β\+8​Cf2​σ∇g2​γ\+32​Cg4​Lf2​n​η2​τα\\displaystyle\\hskip 18\.49988pt\+\\frac\{64C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{128C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+16C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+8C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{32C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+64​Cg4​Lf2​n​η2​τβ\+256​Cg4​Lf2​n​η2​α​τβ\+16​Cf2​Lg2​n​η2​τγ\+4n\(Cf2Lg2\+Cg4Lf2\)η2τ2\]1/2\.\\displaystyle\\hskip 18\.49988pt\+\\frac\{64C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{256C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+\\frac\{16C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+4n\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\)\\eta^\{2\}\\tau^\{2\}\\Bigg\]^\{1/2\}\.\(79\)Combining the preceding two displays proves \([76](https://arxiv.org/html/2608.12710#A2.E76)\)\. ∎

###### Lemma 13\(Client momentum disagreement\)\.

Assume0<ρ<10<\\rho<1\. LetM¯t:=K−1​∑k=1KMtk\\bar\{M\}\_\{t\}:=K^\{\-1\}\\sum\_\{k=1\}^\{K\}M\_\{t\}^\{k\}ands⁡\(t\):=τ​⌊t/τ⌋s\(t\):=\\tau\\lfloor t/\\tau\\rfloor\. Then

1K​T​∑t=0T−1∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}≤ρτ\[6​Cf2​σ∇g2γ​T​b\+12​Cg2​σf2β​T​b\+12​Cg2​Lf2​σg2α​T​b\\displaystyle\\leq\\rho\\tau\\Bigg\[\\frac\{6C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{12C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+96​Cg2​Lf2​σg2​αβ​T​b\+24​Cg2​Lf2​σg2​α\+96​Cg2​Lf2​σg2​α2β\+192​Cg2​Lf2​σg2​α3β\+24​Cg2​σf2​β\+12​Cf2​σ∇g2​γ\\displaystyle\\hskip 18\.49988pt\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+24C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{192C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+24C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+12C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+48​Cg4​Lf2​n​η2​τα\+96​Cg4​Lf2​n​η2​τβ\+384​Cg4​Lf2​n​η2​α​τβ\+24​Cf2​Lg2​n​η2​τγ\+3​LF2​n​η2​τ2\\displaystyle\\hskip 18\.49988pt\+\\frac\{48C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{96C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+\\frac\{384C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+\\frac\{24C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+3L\_\{F\}^\{2\}n\\eta^\{2\}\\tau^\{2\}\+6Cf2Δ∇g2\+12Cg2Δf2\+12Cg2Lf2Δg2\]1/2\.\\displaystyle\\hskip 18\.49988pt\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\Bigg\]^\{1/2\}\.\(80\)

###### Proof\.

Averaging the momentum recursion over the clients, subtracting the resulting identity from the local recursion, and using the synchronization conditionMs⁡\(t\)k=M¯s⁡\(t\)M\_\{s\(t\)\}^\{k\}=\\bar\{M\}\_\{s\(t\)\}, we obtain

1K​T​∑t=0T−1∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}=1K​T​∑t=0T−1∑k=1K𝔼​∥∑ℓ=s⁡\(t\)\+1tρ​\(1−ρ\)t−ℓ​\[Hℓk​\(vℓk⊗In\)−1K​∑j=1KHℓj​\(vℓj⊗In\)\]∥F\\displaystyle=\\frac\{1\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert\\sum\_\{\\ell=s\(t\)\+1\}^\{t\}\\rho\(1\-\\rho\)^\{t\-\\ell\}\\left\[H\_\{\\ell\}^\{k\}\(v\_\{\\ell\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}H\_\{\\ell\}^\{j\}\(v\_\{\\ell\}^\{j\}\\otimes I\_\{n\}\)\\right\]\\rVert\_\{F\}≤ρK​T​∑t=0T−1∑k=1K∑ℓ=s⁡\(t\)\+1t\(1−ρ\)t−ℓ​𝔼​∥Hℓk​\(vℓk⊗In\)−1K​∑j=1KHℓj​\(vℓj⊗In\)∥F\\displaystyle\\leq\\frac\{\\rho\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\sum\_\{\\ell=s\(t\)\+1\}^\{t\}\(1\-\\rho\)^\{t\-\\ell\}\\mathbb\{E\}\\lVert H\_\{\\ell\}^\{k\}\(v\_\{\\ell\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}H\_\{\\ell\}^\{j\}\(v\_\{\\ell\}^\{j\}\\otimes I\_\{n\}\)\\rVert\_\{F\}≤ρT​∑t=0T−1∑ℓ=s⁡\(t\)\+1t1K​∑k=1K𝔼​∥Hℓk​\(vℓk⊗In\)−1K​∑j=1KHℓj​\(vℓj⊗In\)∥F\\displaystyle\\leq\\frac\{\\rho\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{\\ell=s\(t\)\+1\}^\{t\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert H\_\{\\ell\}^\{k\}\(v\_\{\\ell\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}H\_\{\\ell\}^\{j\}\(v\_\{\\ell\}^\{j\}\\otimes I\_\{n\}\)\\rVert\_\{F\}≤ρ​τ​1T​∑ℓ=0T−11K​∑k=1K𝔼​∥Hℓk​\(vℓk⊗In\)−1K​∑j=1KHℓj​\(vℓj⊗In\)∥F\.\\displaystyle\\leq\\rho\\tau\\frac\{1\}\{T\}\\sum\_\{\\ell=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert H\_\{\\ell\}^\{k\}\(v\_\{\\ell\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}H\_\{\\ell\}^\{j\}\(v\_\{\\ell\}^\{j\}\\otimes I\_\{n\}\)\\rVert\_\{F\}\.\(81\)The inequalities above use the triangle inequality,\(1−ρ\)t−ℓ≤1\(1\-\\rho\)^\{t\-\\ell\}\\leq 1, and the fact that eachℓ\\elloccurs at mostτ\\tautimes\. Lemma C\.4 then bounds the remaining term:

1T​∑ℓ=0T−11K​∑k=1K𝔼​∥Hℓk​\(vℓk⊗In\)−1K​∑j=1KHℓj​\(vℓj⊗In\)∥F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{\\ell=0\}^\{T\-1\}\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\lVert H\_\{\\ell\}^\{k\}\(v\_\{\\ell\}^\{k\}\\otimes I\_\{n\}\)\-\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}H\_\{\\ell\}^\{j\}\(v\_\{\\ell\}^\{j\}\\otimes I\_\{n\}\)\\rVert\_\{F\}≤\\displaystyle\\leq\{\}\[6​Cf2​σ∇g2γ​T​b\+12​Cg2​σf2β​T​b\+12​Cg2​Lf2​σg2α​T​b\+96​Cg2​Lf2​σg2​αβ​T​b\+24Cg2Lf2σg2α\\displaystyle\\Bigg\[\\frac\{6C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\}\{\\gamma Tb\}\+\\frac\{12C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\}\{\\beta Tb\}\+\\frac\{12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\}\{\\alpha Tb\}\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\}\{\\beta Tb\}\+24C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha\+96​Cg2​Lf2​σg2​α2β\+192​Cg2​Lf2​σg2​α3β\+24​Cg2​σf2​β\+12​Cf2​σ∇g2​γ\+48​Cg4​Lf2​n​η2​τα\+96​Cg4​Lf2​n​η2​τβ\\displaystyle\\hskip 9\.24994pt\+\\frac\{96C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{2\}\}\{\\beta\}\+\\frac\{192C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\alpha^\{3\}\}\{\\beta\}\+24C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\beta\+12C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\gamma\+\\frac\{48C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\alpha\}\+\\frac\{96C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\tau\}\{\\beta\}\+384​Cg4​Lf2​n​η2​α​τβ\+24​Cf2​Lg2​n​η2​τγ\+3LF2nη2τ2\+6Cf2Δ∇g2\+12Cg2Δf2\+12Cg2Lf2Δg2\]1/2\.\\displaystyle\\hskip 9\.24994pt\+\\frac\{384C\_\{g\}^\{4\}L\_\{f\}^\{2\}n\\eta^\{2\}\\alpha\\tau\}\{\\beta\}\+\\frac\{24C\_\{f\}^\{2\}L\_\{g\}^\{2\}n\\eta^\{2\}\\tau\}\{\\gamma\}\+3L\_\{F\}^\{2\}n\\eta^\{2\}\\tau^\{2\}\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\Bigg\]^\{1/2\}\.\(82\)Combining the last two displays proves \([80](https://arxiv.org/html/2608.12710#A2.E80)\)\. ∎

###### Lemma 14\(Muon descent\)\.

For everytt,

𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤𝔼⁡\[F⁡\(W¯t\)−F⁡\(W¯t\+1\)\]η\+2​nK​∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\leq\\frac\{\\mathbb\{E\}\[F\(\\bar\{W\}\_\{t\}\)\-F\(\\bar\{W\}\_\{t\+1\}\)\]\}\{\\eta\}\+\\frac\{2\\sqrt\{n\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}\(83\)\+2​n​𝔼​‖M¯t−∇F​\(W¯t\)‖F\+LF​n​η2\.\\displaystyle\+2\\sqrt\{n\}\\,\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{L\_\{F\}n\\eta\}\{2\}\.

###### Proof\.

By theLFL\_\{F\}\-smoothness ofFF,

F⁡\(W¯t\+1\)\\displaystyle F\(\\bar\{W\}\_\{t\+1\}\)≤F⁡\(W¯t\)−η⁡⟨∇F​\(W¯t\),1K​∑k=1KUtk​\(Vtk\)⊤⟩\+LF​η22​∥1K​∑k=1KUtk​\(Vtk\)⊤∥F2\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\left\\langle\\nabla F\(\\bar\{W\}\_\{t\}\),\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\right\\rangle\+\\frac\{L\_\{F\}\\eta^\{2\}\}\{2\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}=F⁡\(W¯t\)−ηK​∑k=1K⟨∇F​\(W¯t\),Utk​\(Vtk\)⊤⟩\+LF​η22​∥1K​∑k=1KUtk​\(Vtk\)⊤∥F2\\displaystyle=F\(\\bar\{W\}\_\{t\}\)\-\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\\langle\\nabla F\(\\bar\{W\}\_\{t\}\),U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\right\\rangle\+\\frac\{L\_\{F\}\\eta^\{2\}\}\{2\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}=F⁡\(W¯t\)−ηK​∑k=1K⟨Mtk,Utk​\(Vtk\)⊤⟩−ηK​∑k=1K⟨∇F​\(W¯t\)−Mtk,Utk​\(Vtk\)⊤⟩\+LF​η22​∥1K​∑k=1KUtk​\(Vtk\)⊤∥F2\\displaystyle=F\(\\bar\{W\}\_\{t\}\)\-\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\\langle M\_\{t\}^\{k\},U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\right\\rangle\-\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\\langle\\nabla F\(\\bar\{W\}\_\{t\}\)\-M\_\{t\}^\{k\},U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\right\\rangle\+\\frac\{L\_\{F\}\\eta^\{2\}\}\{2\}\\lVert\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\rVert\_\{F\}^\{2\}≤F⁡\(W¯t\)−ηK​∑k=1K‖Mtk‖∗\+ηK​∑k=1K‖∇F​\(W¯t\)−Mtk‖∗\+LF​η22​K​∑k=1K‖Utk​\(Vtk\)⊤‖F2\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|M\_\{t\}^\{k\}\\\|\_\{\*\}\+\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\-M\_\{t\}^\{k\}\\\|\_\{\*\}\+\\frac\{L\_\{F\}\\eta^\{2\}\}\{2K\}\\sum\_\{k=1\}^\{K\}\\\|U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\\|\_\{F\}^\{2\}≤F⁡\(W¯t\)−ηK​∑k=1K\(‖∇F​\(W¯t\)‖∗−‖∇F​\(W¯t\)−Mtk‖∗\)\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\left\(\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{\*\}\-\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\-M\_\{t\}^\{k\}\\\|\_\{\*\}\\right\)\+ηK∑k=1K∥∇F\(W¯t\)−Mtk∥∗\+LF​η22​K∑k=1K∥Utk\(Vtk\)⊤∥F2\\displaystyle\\hskip 9\.24994pt\+\\frac\{\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\-M\_\{t\}^\{k\}\\\|\_\{\*\}\+\\frac\{L\_\{F\}\\eta^\{2\}\}\{2K\}\\sum\_\{k=1\}^\{K\}\\\|U\_\{t\}^\{k\}\(V\_\{t\}^\{k\}\)^\{\\top\}\\\|\_\{F\}^\{2\}≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖∗\+2​ηK​∑k=1K‖Mtk−∇F​\(W¯t\)‖∗\+LF​n​η22\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{\*\}\+\\frac\{2\\eta\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|M\_\{t\}^\{k\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{\*\}\+\\frac\{L\_\{F\}n\\eta^\{2\}\}\{2\}≤F⁡\(W¯t\)−η​‖∇F​\(W¯t\)‖F\+2​η​nK​∑k=1K‖Mtk−M¯t‖F\+2​η​n​‖M¯t−∇F​\(W¯t\)‖F\+LF​n​η22,\\displaystyle\\leq F\(\\bar\{W\}\_\{t\}\)\-\\eta\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{2\\eta\\sqrt\{n\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}\+2\\eta\\sqrt\{n\}\\,\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{L\_\{F\}n\\eta^\{2\}\}\{2\},\(84\)where we used Lemma[2](https://arxiv.org/html/2608.12710#Thmlemma2), the triangle inequality, and the standard nuclear–Frobenius norm relations\. Rearranging and taking expectation proves the claim\. ∎

###### Theorem 1\(Convergence of FedCoMuon\-VR with first\-order product tracker\)\.

Suppose Assumptions[1](https://arxiv.org/html/2608.12710#Thmassumption1)–[3](https://arxiv.org/html/2608.12710#Thmassumption3),[6](https://arxiv.org/html/2608.12710#Thmassumption6), and[7](https://arxiv.org/html/2608.12710#Thmassumption7)hold\. LetLF=Cf​Lg\+Cg2​LfL\_\{F\}=C\_\{f\}L\_\{g\}\+C\_\{g\}^\{2\}L\_\{f\}\. Assumeη\>0\\eta\>0,0<α,β,γ,ρ<10<\\alpha,\\beta,\\gamma,\\rho<1,b≥1b\\geq 1, andτ\>0\\tau\>0\. Then, for anyT≥1T\\geq 1,

1T​∑t=0T−1𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+LF​n​η2\+2n\[4​Cg​Cfρ​T\+LF​n​ηρ\+\(2\+6ρτ\)\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{L\_\{F\}n\\eta\}\{2\}\+2\\sqrt\{n\}\\Bigg\[\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\(2\+\\sqrt\{6\}\\,\\rho\\tau\)×\(Cf2​σ∇g2​\(1γ​T​b\+2​γ\)\+2​Cg2​σf2​\(1β​T​b\+2​β\)CLOSE\\displaystyle\\hskip 18\.49988pt\\hskip 9\.24994pt\\times\\Bigg\(C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+2C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+2Cg2Lf2σg2\[1α​T​b\+2α\\displaystyle\\hskip 9\.24994pt\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\\right\.\+8​αβ​T​b\+8​α2β\(1\+2α\)\]\+4nη2τ\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\\left\.\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+4n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+nη2τ2\(Cf2Lg2\+Cg4Lf2\+LF22\)\)1/2\+6ρτ\(Cf2Δ∇g2\+2Cg2Δf2\+2Cg2Lf2Δg2\)1/2\]\.\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+n\\eta^\{2\}\\tau^\{2\}\\left\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\+\\frac\{L\_\{F\}^\{2\}\}\{2\}\\right\)\\Bigg\)^\{1/2\}\+\\sqrt\{6\}\\,\\rho\\tau\\left\(C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+2C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\right\)^\{1/2\}\\Bigg\]\.\(85\)

###### Proof\.

By Lemma[14](https://arxiv.org/html/2608.12710#Thmlemma14), for everytt,

𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤𝔼⁡\[F⁡\(W¯t\)−F⁡\(W¯t\+1\)\]η\+2​nK​∑k=1K𝔼​‖Mtk−M¯t‖F\\displaystyle\\leq\\frac\{\\mathbb\{E\}\[F\(\\bar\{W\}\_\{t\}\)\-F\(\\bar\{W\}\_\{t\+1\}\)\]\}\{\\eta\}\+\\frac\{2\\sqrt\{n\}\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}\+2​n​𝔼​‖M¯t−∇F​\(W¯t\)‖F\+LF​n​η2\.\\displaystyle\\hskip 9\.24994pt\+2\\sqrt\{n\}\\,\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}\+\\frac\{L\_\{F\}n\\eta\}\{2\}\.\(86\)After summing this inequality overt=0,…,T−1t=0,\\ldots,T\-1, dividing byTT, and usingF⁡\(W¯T\)≥F∗F\(\\bar\{W\}\_\{T\}\)\\geq F\_\{\*\}, it remains to bound the two averaged error terms\. Applying Lemmas[12](https://arxiv.org/html/2608.12710#Thmlemma12)and[13](https://arxiv.org/html/2608.12710#Thmlemma13)gives

2​nK​T​∑t=0T−1∑k=1K𝔼​‖Mtk−M¯t‖F\+2​nT​∑t=0T−1𝔼​‖M¯t−∇F​\(W¯t\)‖F\\displaystyle\\frac\{2\\sqrt\{n\}\}\{KT\}\\sum\_\{t=0\}^\{T\-1\}\\sum\_\{k=1\}^\{K\}\\mathbb\{E\}\\\|M\_\{t\}^\{k\}\-\\bar\{M\}\_\{t\}\\\|\_\{F\}\+\\frac\{2\\sqrt\{n\}\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\bar\{M\}\_\{t\}\-\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤\\displaystyle\\leq\{\}2n\[4​Cg​Cfρ​T\+LF​n​ηρ\+\[4Cf2σ∇g2\(1γ​T​b\+2γ\)\+8Cg2σf2\(1β​T​b\+2β\)\\displaystyle 2\\sqrt\{n\}\\Bigg\[\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\\Bigg\[4C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+8C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+8​Cg2​Lf2​σg2​\[1α​T​b\+2​α\+8​αβ​T​b\+8​α2β​\(1\+2​α\)\]\\displaystyle\\hskip 9\.24994pt\+8C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+16nη2τ\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\+4n\(Cf2Lg2\+Cg4Lf2\)η2τ2\]1/2\+ρτ\[\\displaystyle\\hskip 9\.24994pt\+16n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+4n\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\)\\eta^\{2\}\\tau^\{2\}\\Bigg\]^\{1/2\}\{\}\+\\rho\\tau\\Bigg\[6​Cf2​σ∇g2​\(1γ​T​b\+2​γ\)\+12​Cg2​σf2​\(1β​T​b\+2​β\)\+12​Cg2​Lf2​σg2​\[1α​T​b\+2​α\+8​αβ​T​b\+8​α2β​\(1\+2​α\)\]\\displaystyle\\hskip 9\.24994pt6C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+12C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+24nη2τ\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\+3LF2nη2τ2\+6Cf2Δ∇g2\+12Cg2Δf2\+12Cg2Lf2Δg2\]1/2\]\\displaystyle\\hskip 9\.24994pt\+24n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+3L\_\{F\}^\{2\}n\\eta^\{2\}\\tau^\{2\}\+6C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+12C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+12C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\Bigg\]^\{1/2\}\\Bigg\]≤\\displaystyle\\leq\{\}2n\[4​Cg​Cfρ​T\+LF​n​ηρ\+\[4\(Cf2σ∇g2\(1γ​T​b\+2γ\)\+2Cg2σf2\(1β​T​b\+2β\)\\displaystyle 2\\sqrt\{n\}\\Bigg\[\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\\Bigg\[4\\Bigg\(C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+2C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+2​Cg2​Lf2​σg2​\[1α​T​b\+2​α\+8​αβ​T​b\+8​α2β​\(1\+2​α\)\]\+4​n​η2​τ​\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\\displaystyle\\hskip 9\.24994pt\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+4n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+nη2τ2\(Cf2Lg2\+Cg4Lf2\+LF22\)\)\]1/2\+ρτ\[6\(Cf2σ∇g2\(1γ​T​b\+2γ\)\+2Cg2σf2\(1β​T​b\+2β\)\\displaystyle\\hskip 9\.24994pt\+n\\eta^\{2\}\\tau^\{2\}\\left\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\+\\frac\{L\_\{F\}^\{2\}\}\{2\}\\right\)\\Bigg\)\\Bigg\]^\{1/2\}\{\}\+\\rho\\tau\\Bigg\[6\\Bigg\(C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+2C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+2​Cg2​Lf2​σg2​\[1α​T​b\+2​α\+8​αβ​T​b\+8​α2β​\(1\+2​α\)\]\+4​n​η2​τ​\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\\displaystyle\\hskip 9\.24994pt\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+4n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+nη2τ2\(Cf2Lg2\+Cg4Lf2\+LF22\)\)\+6\(Cf2Δ∇g2\+2Cg2Δf2\+2Cg2Lf2Δg2\)\]1/2\]\\displaystyle\\hskip 9\.24994pt\+n\\eta^\{2\}\\tau^\{2\}\\left\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\+\\frac\{L\_\{F\}^\{2\}\}\{2\}\\right\)\\Bigg\)\+6\\left\(C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+2C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\right\)\\Bigg\]^\{1/2\}\\Bigg\]≤\\displaystyle\\leq\{\}2n\[4​Cg​Cfρ​T\+LF​n​ηρ\\displaystyle 2\\sqrt\{n\}\\Bigg\[\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\(2\+6​ρ​τ\)​\(Cf2​σ∇g2​\(1γ​T​b\+2​γ\)\+2​Cg2​σf2​\(1β​T​b\+2​β\)CLOSE\\displaystyle\\hskip 9\.24994pt\+\(2\+\\sqrt\{6\}\\,\\rho\\tau\)\\Bigg\(C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+2C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+2​Cg2​Lf2​σg2​\[1α​T​b\+2​α\+8​αβ​T​b\+8​α2β​\(1\+2​α\)\]\+4​n​η2​τ​\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\\displaystyle\\hskip 18\.49988pt\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+4n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+nη2τ2\(Cf2Lg2\+Cg4Lf2\+LF22\)\)1/2\+6ρτ\(Cf2Δ∇g2\+2Cg2Δf2\+2Cg2Lf2Δg2\)1/2\]\.\\displaystyle\\hskip 18\.49988pt\+n\\eta^\{2\}\\tau^\{2\}\\left\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\+\\frac\{L\_\{F\}^\{2\}\}\{2\}\\right\)\\Bigg\)^\{1/2\}\+\\sqrt\{6\}\\,\\rho\\tau\\left\(C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+2C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\right\)^\{1/2\}\\Bigg\]\.\(87\)Substituting the last display into the averaged descent inequality gives

1T​∑t=0T−1𝔼​‖∇F​\(W¯t\)‖F\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}≤F⁡\(W¯0\)−F∗η​T\+LF​n​η2\+2n\[4​Cg​Cfρ​T\+LF​n​ηρ\+\(2\+6ρτ\)\\displaystyle\\leq\\frac\{F\(\\bar\{W\}\_\{0\}\)\-F\_\{\*\}\}\{\\eta T\}\+\\frac\{L\_\{F\}n\\eta\}\{2\}\+2\\sqrt\{n\}\\Bigg\[\\frac\{4C\_\{g\}C\_\{f\}\}\{\\rho T\}\+\\frac\{L\_\{F\}\\sqrt\{n\}\\,\\eta\}\{\\rho\}\+\(2\+\\sqrt\{6\}\\,\\rho\\tau\)×\(Cf2σ∇g2\(1γ​T​b\+2γ\)\+2Cg2σf2\(1β​T​b\+2β\)\+2Cg2Lf2σg2\[1α​T​b\+2α\\displaystyle\\hskip 18\.49988pt\\hskip 9\.24994pt\\times\\Bigg\(C\_\{f\}^\{2\}\\sigma\_\{\\nabla g\}^\{2\}\\left\(\\frac\{1\}\{\\gamma Tb\}\+2\\gamma\\right\)\+2C\_\{g\}^\{2\}\\sigma\_\{f\}^\{2\}\\left\(\\frac\{1\}\{\\beta Tb\}\+2\\beta\\right\)\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\sigma\_\{g\}^\{2\}\\left\[\\frac\{1\}\{\\alpha Tb\}\+2\\alpha\\right\.\+8​αβ​T​b\+8​α2β\(1\+2α\)\]\+4nη2τ\(2​Cg4​Lf2α\+4​Cg4​Lf2β\+16​Cg4​Lf2​αβ\+Cf2​Lg2γ\)\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\\left\.\+\\frac\{8\\alpha\}\{\\beta Tb\}\+\\frac\{8\\alpha^\{2\}\}\{\\beta\}\(1\+2\\alpha\)\\right\]\+4n\\eta^\{2\}\\tau\\left\(\\frac\{2C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\alpha\}\+\\frac\{4C\_\{g\}^\{4\}L\_\{f\}^\{2\}\}\{\\beta\}\+\\frac\{16C\_\{g\}^\{4\}L\_\{f\}^\{2\}\\alpha\}\{\\beta\}\+\\frac\{C\_\{f\}^\{2\}L\_\{g\}^\{2\}\}\{\\gamma\}\\right\)\+nη2τ2\(Cf2Lg2\+Cg4Lf2\+LF22\)\)1/2\+6ρτ\(Cf2Δ∇g2\+2Cg2Δf2\+2Cg2Lf2Δg2\)1/2\]\.\\displaystyle\\hskip 18\.49988pt\\hskip 18\.49988pt\+n\\eta^\{2\}\\tau^\{2\}\\left\(C\_\{f\}^\{2\}L\_\{g\}^\{2\}\+C\_\{g\}^\{4\}L\_\{f\}^\{2\}\+\\frac\{L\_\{F\}^\{2\}\}\{2\}\\right\)\\Bigg\)^\{1/2\}\+\\sqrt\{6\}\\,\\rho\\tau\\left\(C\_\{f\}^\{2\}\\Delta\_\{\\nabla g\}^\{2\}\+2C\_\{g\}^\{2\}\\Delta\_\{f\}^\{2\}\+2C\_\{g\}^\{2\}L\_\{f\}^\{2\}\\Delta\_\{g\}^\{2\}\\right\)^\{1/2\}\\Bigg\]\.\(88\)This proves the theorem\. ∎

Forη=α=β=γ=T−2/3\\eta=\\alpha=\\beta=\\gamma=T^\{\-2/3\},ρ=T−1/3\\rho=T^\{\-1/3\},b=T2/3b=T^\{2/3\}, andτ=O⁡\(1\)\\tau=O\(1\), the non\-square\-root terms in Theorem[1](https://arxiv.org/html/2608.12710#Thmtheorem1b)are at mostO\(T−1/3\)O\(T^\{\-1/3\}\), while every term inside the square root is at mostO\(T−2/3\)O\(T^\{\-2/3\}\)\. Moreover,2\+6​ρ​τ=O⁡\(1\)2\+\\sqrt\{6\}\\,\\rho\\tau=O\(1\)andρτ=O\(T−1/3\)\\rho\\tau=O\(T^\{\-1/3\}\)\. Hence,T−1∑t=0T−1𝔼∥∇F\(W¯t\)∥F=O\(T−1/3\)T^\{\-1\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\|\\nabla F\(\\bar\{W\}\_\{t\}\)\\\|\_\{F\}=O\(T^\{\-1/3\}\)\.

## Appendix CDetailed Experimental Settings

All experiments were repeated using two random seeds, 42 and 43, and we report the average results across the two runs\. For all Muon\-based methods, we use five Newton–Schulz iterations to approximate the orthogonalization of matrix\-valued parameters\. All learning rates and method\-specific hyperparameters are selected through grid search based on validation performance\.

### C\.1Robust Federated Learning

#### C\.1\.1Image Classification on MNIST

The CNN architecture used for MNIST is summarized in Table[1](https://arxiv.org/html/2608.12710#A3.T1)\.

Table 1:CNN architecture used for MNIST\.For the MNIST experiments, we search the learning rate over\{0\.005,0\.01,0\.02,0\.03,0\.05,0\.1,0\.2,0\.5,1\}\\\{0\.005,0\.01,0\.02,0\.03,0\.05,0\.1,0\.2,0\.5,1\\\}\. We set the batch size to 20 andλ=0\.5\\lambda=0\.5for all algorithms\. FedAvg and ComFedL use a learning rate of0\.020\.02, while all other methods use a learning rate of0\.010\.01\. For FedMuon, we setβ=0\.1\\beta=0\.1\. FedMuon\-LGA usesα=0\.2\\alpha=0\.2andβ=0\.8\\beta=0\.8, while FedMuon\-BC usesα=0\.1\\alpha=0\.1\. For Local\-SCGDM, we setα=0\.2\\alpha=0\.2andγ=0\.3\\gamma=0\.3\. FedCoMuon usesα=0\.2\\alpha=0\.2andβ=0\.1\\beta=0\.1\. For FedCoMuon\-VR, we setα=0\.2\\alpha=0\.2,β=0\.8\\beta=0\.8,γ=0\.9\\gamma=0\.9, andρ=0\.2\\rho=0\.2\.

#### C\.1\.2Language Modeling on WikiText\-2

We use an 8\-layer Transformer encoder with an embedding dimension of 768 and eight attention heads per layer\. Each Transformer block employs a feed\-forward network with a hidden dimension of 1024, together with sinusoidal positional encodings\. A dropout rate of0\.10\.1is applied throughout the network\. The final output layer projects the hidden representations to the vocabulary size for next\-token prediction\.

The learning rates are searched over\{0\.005,0\.01,0\.02,0\.03,0\.05,0\.1,0\.2,0\.5,1\}\\\{0\.005,0\.01,0\.02,0\.03,0\.05,0\.1,0\.2,0\.5,1\\\}\. In the experiments, we set the batch size to 32 for all algorithms\. We set the learning rate to0\.10\.1for FedAvg\. FedMuon and FedMuon\-BC use a learning rate of0\.030\.03, withβ=0\.3\\beta=0\.3for FedMuon andα=0\.1\\alpha=0\.1for FedMuon\-BC\. FedMuon\-LGA uses a learning rate of0\.010\.01, withα=0\.5\\alpha=0\.5andβ=0\.8\\beta=0\.8\. ComFedL and Local\-SCGDM use a learning rate of0\.20\.2, while Local\-SCGDM additionally usesα=0\.2\\alpha=0\.2andγ=0\.1\\gamma=0\.1\. For FedCoMuon, we set the learning rate to0\.020\.02and useα=β=0\.2\\alpha=\\beta=0\.2\. For FedCoMuon\-VR, we set the learning rate to0\.030\.03, withα=0\.3\\alpha=0\.3,β=0\.8\\beta=0\.8,γ=0\.6\\gamma=0\.6, andρ=0\.2\\rho=0\.2\.

### C\.2Task\-Distributed Meta Learning

#### C\.2\.1CNN\-Based Meta Learning

The architecture of the 7\-layer CNN used for CIFAR\-10 is summarized in Table[2](https://arxiv.org/html/2608.12710#A3.T2)\.

Table 2:Architecture of the 7\-layer CNN used for CIFAR\-10\.Both the inner and outer learning rates are searched over\{0\.005,0\.01,0\.02,0\.03,0\.05,0\.1,0\.2,0\.5,1\}\\\{0\.005,0\.01,0\.02,0\.03,0\.05,0\.1,0\.2,0\.5,1\\\}\. We set the batch size to 64 for all algorithms\. We searchλ\\lambdaover\{0\.2,0\.5,1,2\}\\\{0\.2,0\.5,1,2\\\}and selectλ=0\.5\\lambda=0\.5\. The same hyperparameter search spaces are used for all heterogeneity levels\.

For the 7\-layer CNN experiments on CIFAR\-10, the outer learning rate is set to0\.10\.1for all methods\. FedMAML uses an inner learning rate of0\.030\.03forχ=0\.3\\chi=0\.3and0\.50\.5, and0\.050\.05forχ=0\.7\\chi=0\.7\. FedMuon, FedMuon\-LGA, FedMuon\-BC, and ComFedL use an inner learning rate of0\.050\.05, whereas Local\-SCGDM, FedCoMuon, and FedCoMuon\-VR use0\.010\.01,0\.030\.03, and0\.030\.03, respectively\. For FedMuon, we setβ=0\.4\\beta=0\.4\. For FedMuon\-LGA, we setα=0\.5\\alpha=0\.5andβ=0\.8\\beta=0\.8, while FedMuon\-BC usesα=0\.1\\alpha=0\.1\. Local\-SCGDM usesα=γ=0\.9\\alpha=\\gamma=0\.9\. For FedCoMuon, we setα=β=0\.7\\alpha=\\beta=0\.7across all heterogeneity settings\. For FedCoMuon\-VR, we setα=0\.1\\alpha=0\.1,β=0\.8\\beta=0\.8, andρ=0\.2\\rho=0\.2for all heterogeneity settings, whileγ\\gammais set to0\.70\.7,0\.70\.7, and0\.90\.9forχ=0\.3\\chi=0\.3,0\.50\.5, and0\.70\.7, respectively\.

#### C\.2\.2ViT\-Tiny\-Based Meta Learning

For ViT\-Tiny\-based meta learning, we conduct experiments on CIFAR\-10 using a ViT\-Tiny model with 12 Transformer blocks, a hidden dimension of 192, three attention heads, and a patch size of 4\. We adopt the same federated setting and dominant\-class data partition as in the CNN experiments and setχ=0\.3\\chi=0\.3\.

We set the batch size to 32 for all algorithms\. Both the inner and outer learning rates are searched over\{0\.001,0\.003,0\.005,0\.01,0\.02,0\.05,0\.1,0\.2,0\.5\}\\\{0\.001,0\.003,0\.005,0\.01,0\.02,0\.05,0\.1,0\.2,0\.5\\\}\. FedMAML uses inner and outer learning rates of0\.050\.05and0\.10\.1, respectively\. FedMuon, FedMuon\-LGA, and FedMuon\-BC use an inner learning rate of0\.0050\.005and an outer learning rate of0\.010\.01\. We setβ=0\.3\\beta=0\.3for FedMuon,α=0\.5\\alpha=0\.5andβ=0\.8\\beta=0\.8for FedMuon\-LGA, andα=0\.1\\alpha=0\.1for FedMuon\-BC\. ComFedL uses inner and outer learning rates of0\.0030\.003and0\.10\.1, respectively\. Local\-SCGDM uses inner and outer learning rates of0\.0030\.003and0\.050\.05, withα=0\.3\\alpha=0\.3andγ=0\.9\\gamma=0\.9\. FedCoMuon uses inner and outer learning rates of0\.0050\.005and0\.010\.01, withα=0\.9\\alpha=0\.9andβ=0\.3\\beta=0\.3\. FedCoMuon\-VR also uses inner and outer learning rates of0\.0050\.005and0\.010\.01, respectively, withα=0\.8\\alpha=0\.8,β=0\.8\\beta=0\.8,γ=0\.9\\gamma=0\.9, andρ=0\.2\\rho=0\.2\.