FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA

# FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA
Source: [https://arxiv.org/html/2607.26618](https://arxiv.org/html/2607.26618)
###### Abstract

Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples\. However, task heterogeneity across clients can cause cross\-task interference and gradient conflicts during aggregation\. Federated MoE\-LoRA addresses this challenge through specialized LoRA experts and conditional routing\. Yet existing methods typically specialize at client granularity, implicitly assuming task\-coherent clients\. Our core insight is that experts need*purity*—pattern\-coherent updates that preserve specialization—whereas routers need*contrast*—mixed\-task observations that support expert comparison\. We proposeFedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization to meet these two requirements\.FedWeaveuses unsupervised prototype discovery to form local buckets and align them across clients, enabling prototype\-level expert aggregation while retaining mixed\-task client trajectories for router training\. At inference,FedWeaveperforms sparse inference with one active expert while preserving nearly all soft\-routing performance\. Our theoretical analysis explains why asymmetric aggregation is advantageous: it controls expert convergence in stationarity through off\-pattern contamination, identifies the consensus error induced by fragmented router trajectories, and bounds sparse\-inference risk\. On a heterogeneous multi\-task benchmark with mainstream LLM backbones,FedWeaveconsistently outperforms strong baselines, while ablations verify the effectiveness of our design\.

## Introduction

Federated learning \(FL\) enables multiple clients to train models collaboratively without sharing their private data\(McMahanet al\.[2017](https://arxiv.org/html/2607.26618#bib.bib16); Kairouzet al\.[2021](https://arxiv.org/html/2607.26618#bib.bib10)\)\. This paradigm is particularly attractive for large language models \(LLMs\), whose instruction data may be distributed across users and organizations, with direct sharing restricted by policies and regulations on data privacy\. However, full\-parameter LLM fine\-tuning is often impractical in federated environments because of its substantial computational and communication costs\. Parameter\-efficient fine\-tuning \(PEFT\), especially low\-rank adaptation \(LoRA\)\(Huet al\.[2022](https://arxiv.org/html/2607.26618#bib.bib9)\), addresses this constraint by updating lightweight adapters while keeping the backbone frozen\. Recent work has therefore explored federated instruction tuning and federated LoRA as scalable strategies for decentralized LLM adaptation\(Zhanget al\.[2023](https://arxiv.org/html/2607.26618#bib.bib35); Sunet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib26); Guoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib8); Bianet al\.[2026a](https://arxiv.org/html/2607.26618#bib.bib28),[b](https://arxiv.org/html/2607.26618#bib.bib29)\)\.

Despite this progress, federated LLM instruction tuning remains fundamentally challenged by task heterogeneity\. Clients may differ in their domains, task distributions, and instruction styles; more importantly, a single client may itself mix several latent instruction patterns\(Marfoqet al\.[2021](https://arxiv.org/html/2607.26618#bib.bib15); Fenget al\.[2023](https://arxiv.org/html/2607.26618#bib.bib6); Talassoet al\.[2026](https://arxiv.org/html/2607.26618#bib.bib27)\)\. These patterns may demand different reasoning skills, output structures, or adaptation directions, despite residing on the same client\. When these patterns are compressed into a single client update, their adaptation signals lose the pattern coherence required for specialization before cross\-client aggregation begins\.

Nevertheless, current efforts on federated PEFT mainly organize heterogeneity at the granularity of clients\. Client correction, clustering, and personalization improve how client updates are optimized or shared\(Liet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib13); Sattleret al\.[2021](https://arxiv.org/html/2607.26618#bib.bib23); Sunet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib26); Guoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib8)\)\. Mixture\-of\-Experts \(MoE\) and multi\-expert LoRA add specialization capacity by routing inputs to lightweight experts\(Shazeeret al\.[2017](https://arxiv.org/html/2607.26618#bib.bib24); Feduset al\.[2022](https://arxiv.org/html/2607.26618#bib.bib5); Liaoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib14)\)\. Federated expert models bring this idea to decentralized settings\(Zecet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib34); Reisseret al\.[2021](https://arxiv.org/html/2607.26618#bib.bib21); Wanget al\.[2025](https://arxiv.org/html/2607.26618#bib.bib32)\)\. Yet when expert allocation, training, and aggregation remain tied to client identity, incompatible patterns within one client are mixed into the same update and related patterns across clients remain unaligned\.

The central challenge is a component\-dependent mismatch in aggregation granularity\. Put simply, experts need*purity*—pattern\-coherent updates that preserve specialization—whereas routers need*contrast*—cross\-pattern observations that enable expert comparison\. Consequently, client\-level aggregation is too coarse for experts, while independently training and aggregating a router for each pattern fragments the cross\-pattern trajectory required for routing\. No single aggregation granularity serves both components\. Moreover, realizing such mixed\-granularity aggregation is non\-trivial: training budgets are limited and unevenly distributed across patterns, while experts and routers exhibit distinct convergence dynamics\.

FedWeaveoperationalizes this purity–contrast insight through asymmetric aggregation as illustrated in Figure[1](https://arxiv.org/html/2607.26618#Sx2.F1)\. Each client performs unsupervised prototype discovery without requiring task\-identity labels and derives an adaptation signature for each local bucket\. The server uses these signatures to align related buckets across clients into global expert groups\. Local training interleaves bucket\-homogeneous mini\-batches, after which the server aggregates expert updates among matched prototypes\. In contrast, the broadcast global router is updated across each client’s complete bucket trajectory and aggregated from one client\-level delta, preserving cross\-pattern contrast for expert selection\.FedWeavetrains with soft routing and supports both soft\-mixture and sparse inference; in the sparse mode, only one LoRA expert is activated\.

Our component\-wise theory makes this asymmetry explicit: prototype\-level aggregation givesO​\(1/U\)O\(1/U\)expert convergence to a contamination\-controlled stationary neighborhood under non\-vanishing routed weight; the router result gives a PL\-convergence bound separating interleaving\-order bias from reset consensus error; gap calibration then transfers these bounds to sparse\-inference risk\. On a multi\-task benchmark comprising CoEdIT, GSM8K, TweetEval, and ARC\-C,FedWeaveachieves the best performance among the evaluated baselines on Llama3\.2\-3B and Gemma\-2\-2B and outperforms the closest client\-level expert baseline across the tested heterogeneity levels\(Meta AI[2024](https://arxiv.org/html/2607.26618#bib.bib17); Gemma Teamet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib18)\), while ablations verify the effectiveness of our design\. Sparse inference preserves nearly all soft\-routing quality while activating only one expert per input\.

Our contributions are summarized as follows:

- •We identify a purity–contrast principle for federated MoE\-LoRA under intra\-client task heterogeneity: experts require pattern\-coherent updates for specialization, whereas routers require client\-level trajectories that retain cross\-pattern contrast\.
- •We proposeFedWeave, a federated MoE\-LoRA framework which refines the granularity to the prototype level and aggregates expert and router in an asymmetric way\.
- •We derive component\-wise convergence guarantees: contamination\-controlled expert stationarity, PL router convergence separating order bias from reset consensus error, and sparse\-risk transfer\.

## Related Work

##### Federated learning under heterogeneity\.

FedProx and SCAFFOLD correct client\-level optimization\(Liet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib13); Karimireddyet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib11)\), while clustered FL learns models for latent client groups\(Sattleret al\.[2021](https://arxiv.org/html/2607.26618#bib.bib23); Ghoshet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib7)\); both retain the client\-level update structure of FedAvg\(McMahanet al\.[2017](https://arxiv.org/html/2607.26618#bib.bib16); Kairouzet al\.[2021](https://arxiv.org/html/2607.26618#bib.bib10)\)\. Mixture\-based FL instead models each client through latent sources\(Marfoqet al\.[2021](https://arxiv.org/html/2607.26618#bib.bib15)\), and FedIns performs instance\-adaptive inference under intra\-client heterogeneity\(Fenget al\.[2023](https://arxiv.org/html/2607.26618#bib.bib6)\)\.FedWeaveinstead turns aligned within\-client prototypes into shared expert\-update units\.

##### Federated PEFT for LLMs\.

LoRA and QLoRA restrict adaptation to low\-rank residuals\(Huet al\.[2022](https://arxiv.org/html/2607.26618#bib.bib9); Dettmerset al\.[2023](https://arxiv.org/html/2607.26618#bib.bib4)\), enabling federated instruction tuning\(Zhanget al\.[2023](https://arxiv.org/html/2607.26618#bib.bib35); Yanget al\.[2025](https://arxiv.org/html/2607.26618#bib.bib39)\)\. Federated LoRA methods address factor\-aggregation bias through fixed or selective factors, server correction, alternating updates, exact inner\-matrix aggregation, or gauge\-aware consensus subspaces\(Sunet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib26); Guoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib8); Bianet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib40); Kooet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib42); Singhalet al\.[2026](https://arxiv.org/html/2607.26618#bib.bib43); Chenet al\.[2026](https://arxiv.org/html/2607.26618#bib.bib44)\)\. Others accommodate heterogeneous ranks or structures, privacy noise, personalization, or layer\-wise sharing\(Choet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib30); Wanget al\.[2024](https://arxiv.org/html/2607.26618#bib.bib33); Fanet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib41); Leeet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib31); Bianet al\.[2026a](https://arxiv.org/html/2607.26618#bib.bib28); Haoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib19); Shenet al\.[2026](https://arxiv.org/html/2607.26618#bib.bib25)\)\. FedTreeLoRA varies layer\-wise sharing depth through a client hierarchy to couple statistical and functional heterogeneity\(Bianet al\.[2026b](https://arxiv.org/html/2607.26618#bib.bib29)\)\. These methods modify adapter parameterization or client\-level sharing, whereasFedWeavechanges the within\-client data unit that defines expert specialization\.

##### Expert models and sparse routing\.

Sparse MoE models\(Shazeeret al\.[2017](https://arxiv.org/html/2607.26618#bib.bib24); Feduset al\.[2022](https://arxiv.org/html/2607.26618#bib.bib5)\)and MoE\-LoRA\(Liaoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib14)\)provide conditional specialization; LLaVA\-MoLE and HotMoE address conflicts through sparse or hybrid routing\(Chenet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib37); Huanget al\.[2026](https://arxiv.org/html/2607.26618#bib.bib38)\), while sparse\-and\-orthogonal federated LoRA targets multi\-task interference through implicit expert separation\(Yanget al\.[2026](https://arxiv.org/html/2607.26618#bib.bib45)\)\. Federated MoE methods personalize experts by client\(Zecet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib34); Reisseret al\.[2021](https://arxiv.org/html/2607.26618#bib.bib21)\): FedMoE selects client\-specific sub\-MoEs from a pretrained sparse model\(Meiet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib36)\), while FedLEASE allocates LoRA experts to client clusters\(Wanget al\.[2025](https://arxiv.org/html/2607.26618#bib.bib32)\)\. LoMo\-Fed separates global and local experts in personalized vision heads\(Sanget al\.[2026](https://arxiv.org/html/2607.26618#bib.bib46)\), whereas FedRouter associates local clusters with task\-centric adapters but uses nearest\-centroid selection at evaluation\(Talassoet al\.[2026](https://arxiv.org/html/2607.26618#bib.bib27)\)\. In contrast,FedWeaveaggregates expert updates over aligned within\-client prototypes while training a single global router from complete client trajectories, matching expert purity with router contrast\.

![Refer to caption](https://arxiv.org/html/2607.26618v1/x1.png)Figure 1:FedWeave overview\.Aligned local prototypes provide pattern\-coherent batch supervision for experts, whereas the single global router retains cross\-pattern contrast by training over each client’s complete bucket trajectory\. The server aggregates the two components at different granularities\.

## Method

### Problem Setup

We considerNNclients with local datasets\{𝒟i\}i=1N\\\{\\mathcal\{D\}\_\{i\}\\\}\_\{i=1\}^\{N\}and a shared frozen LLM backboneW0W\_\{0\}\. The trainable model comprises LoRA expertsΘ=\{θm\}m=1M⋆\\Theta=\\\{\\theta\_\{m\}\\\}\_\{m=1\}^\{M^\{\\star\}\}and a global routerφ\\varphi\. We call a coherent local sub\-distribution that induces similar adaptation behavior a*latent instruction pattern*\. Task labels are used only to construct and audit the benchmark; they are not provided toFedWeave\.

Figure[1](https://arxiv.org/html/2607.26618#Sx2.F1)summarizes the pipeline\. Each client first discovers local buckets\{ℬi,c\}c=1Ci\\\{\\mathcal\{B\}\_\{i,c\}\\\}\_\{c=1\}^\{C\_\{i\}\}, the server then aligns buckets into global expert groups, and local training interleaves bucket\-homogeneous mini\-batches\. The central asymmetry is that aligned\-prototype sample counts weight expert updates to preserve pattern purity, whereas the global router spans each client’s complete trajectory to retain cross\-pattern contrast\. Implementation details, diagnostics, and proofs are in Appendices[A](https://arxiv.org/html/2607.26618#A1)–[C](https://arxiv.org/html/2607.26618#A3)\.

### Local Prototype Discovery

Each client embeds its local examples with the frozen backboneW0W\_\{0\}and partitions𝒟i\\mathcal\{D\}\_\{i\}into local prototype buckets\{ℬi,c\}c=1Ci\\\{\\mathcal\{B\}\_\{i,c\}\\\}\_\{c=1\}^\{C\_\{i\}\}\. We call the conceptual specialization unit a*prototype*and its concrete local implementation a*bucket*\. This exposes intra\-client structure without sharing raw examples\.

Semantic proximity identifies local modes but does not determine whether their updates are compatible\. Each bucket therefore performs a short LoRA warmup from the same initialization and records an adaptation signaturesi,cs\_\{i,c\}from the resultingBBmatrices, following prior observations of functional asymmetry between the LoRA factors\(Guoet al\.[2025](https://arxiv.org/html/2607.26618#bib.bib8); Sunet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib26)\)\. The signature enables cross\-client matching of buckets that can share an expert while preserving expert\-update purity\.

### Cross\-Client Prototype Alignment

The alignment must decide both how many specialization units exist and which units should share parameters across clients\. Each client runskk\-means on frozen\-backbone embeddings and selectsCiC\_\{i\}by maximizing the average silhouette coefficient\(Rousseeuw[1987](https://arxiv.org/html/2607.26618#bib.bib22)\)\. The server formsD=\(du​v\)D=\(d\_\{uv\}\)from the mean layer\-wise cosine distance acrossLBL\_\{B\}LoRA\-BBblocks\. LetAggM​\(D\)\\mathrm\{Agg\}\_\{M\}\(D\)denote agglomerative clustering intoMMgroups, where𝒦g\\mathcal\{K\}\_\{g\}is the feasible candidate set\. We selectM∈𝒦gM\\in\\mathcal\{K\}\_\{g\}by silhouette score:

du​v\\displaystyle d\_\{uv\}=1LB​∑ℓ=1LB\[1−cos⁡\(su\(ℓ\),sv\(ℓ\)\)\],\\displaystyle=\\frac\{1\}\{L\_\{B\}\}\\sum\_\{\\ell=1\}^\{L\_\{B\}\}\\\!\\left\[1\-\\cos\(s\_\{u\}^\{\(\\ell\)\},s\_\{v\}^\{\(\\ell\)\}\)\\right\],\(1\)M⋆\\displaystyle M^\{\\star\}=arg⁡maxM∈𝒦g⁡Sil​\(AggM​\(D\)\)\.\\displaystyle=\\arg\\max\_\{M\\in\\mathcal\{K\}\_\{g\}\}\\mathrm\{Sil\}\(\\mathrm\{Agg\}\_\{M\}\(D\)\)\.The one\-time alignment fixesM⋆M^\{\\star\}anda​\(i,c\)∈\{1,…,M⋆\}a\(i,c\)\\in\\\{1,\\ldots,M^\{\\star\}\\\}throughout federated training, mapping bucketccof clientiito one global expert\. With𝒰m=\{\(i,c\):a​\(i,c\)=m\}\\mathcal\{U\}\_\{m\}=\\\{\(i,c\):a\(i,c\)=m\\\}and the warmup adapterθ~i,c\\widetilde\{\\theta\}\_\{i,c\}, the implementation initializes

θm0=1\|𝒰m\|​∑\(i,c\)∈𝒰mθ~i,c\.\\theta\_\{m\}^\{0\}=\\frac\{1\}\{\|\\mathcal\{U\}\_\{m\}\|\}\\sum\_\{\(i,c\)\\in\\mathcal\{U\}\_\{m\}\}\\widetilde\{\\theta\}\_\{i,c\}\.\(2\)Semantic similarity exposes local modes, while adaptation similarity determines which modes share an expert without benchmark task labels\. K\-means and agglomerative clustering are default implementations; the framework requires only coherent local buckets and cross\-client alignment, and Table[3](https://arxiv.org/html/2607.26618#Sx4.T3)evaluates alternatives\.

### Local Training over Interleaved Buckets

After prototype alignment, each communication round trains experts and the router with different local granularities\. For an examplex∈ℬi,cx\\in\\mathcal\{B\}\_\{i,c\}, the router producesrm​\(x;φ\)≥0r\_\{m\}\(x;\\varphi\)\\geq 0with∑mrm​\(x;φ\)=1\\sum\_\{m\}r\_\{m\}\(x;\\varphi\)=1, and the bucket objective is

ℒi,c​\(Θ,φ\)=1\|ℬi,c\|​∑\(x,y\)∈ℬi,cℓ​\(fW0,Θ,r​\(x;φ\)​\(x\),y\)\.\\mathcal\{L\}\_\{i,c\}\(\\Theta,\\varphi\)=\\frac\{1\}\{\|\\mathcal\{B\}\_\{i,c\}\|\}\\sum\_\{\(x,y\)\\in\\mathcal\{B\}\_\{i,c\}\}\\ell\\\!\\left\(f\_\{W\_\{0\},\\Theta,r\(x;\\varphi\)\}\(x\),y\\right\)\.\(3\)Soft routing keeps router\-expert training differentiable and supplies comparative signals to non\-anchor experts while early assignments remain uncertain, reducing premature expert starvation\. It can also transmit off\-pattern gradients; the theory records this routing leakage in each expert’s off\-target contribution weight\.

Each client retains the same fixedEE\-step mini\-batch budget, allocates it across buckets approximately in proportion to bucket size with a small minimum for every nonempty bucket, and then interleaves the scheduled bucket identifiers\. At each scheduled step, the client draws a mini\-batch only from the selected bucket, preserving pattern\-coherent expert supervision\. The broadcast global\-router state persists across the entire interleaved schedule, accumulating cross\-pattern contrast across the client’s buckets\.

### Asymmetric Federated Aggregation

LetΔ​θi,mt\\Delta\\theta\_\{i,m\}^\{t\}andΔ​φit\\Delta\\varphi\_\{i\}^\{t\}denote clientii’s local expert and router deltas at roundtt, and letni,m=∑c:a​\(i,c\)=m\|ℬi,c\|n\_\{i,m\}=\\sum\_\{c:a\(i,c\)=m\}\|\\mathcal\{B\}\_\{i,c\}\|be the local sample count aligned to expertmmfor FedAvg\-style weighting\(McMahanet al\.[2017](https://arxiv.org/html/2607.26618#bib.bib16)\)\. The server updates

θmt\+1\\displaystyle\\theta\_\{m\}^\{t\+1\}=θmt\+∑i∈𝒮t:ni,m\>0ni,m∑j∈𝒮tnj,m​Δ​θi,mt,\\displaystyle=\\theta\_\{m\}^\{t\}\+\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}:n\_\{i,m\}\>0\}\\frac\{n\_\{i,m\}\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{t\}\}n\_\{j,m\}\}\\Delta\\theta\_\{i,m\}^\{t\},\(4\)φt\+1\\displaystyle\\varphi^\{t\+1\}=φt\+∑i∈𝒮t\|𝒟i\|∑j∈𝒮t\|𝒟j\|​Δ​φit\.\\displaystyle=\\varphi^\{t\}\+\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}\\frac\{\|\\mathcal\{D\}\_\{i\}\|\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{t\}\}\|\\mathcal\{D\}\_\{j\}\|\}\\Delta\\varphi\_\{i\}^\{t\}\.
The expert rule weights each delta by matched\-pattern supervision to preserve expert\-update purity, whereas the router rule contributes one delta from each client’s full interleaved trajectory to retain cross\-bucket contrast\.

Algorithm 1FedWeavetraining and sparse inference0:Client datasets

\{𝒟i\}\\\{\\mathcal\{D\}\_\{i\}\\\}, frozen backbone

W0W\_\{0\}, rounds

TT, local\-step budget

EE
1:for allclients

iido

2:Embed

𝒟i\\mathcal\{D\}\_\{i\}, select

CiC\_\{i\}, and form buckets

\{ℬi,c\}\\\{\\mathcal\{B\}\_\{i,c\}\\\}
3:Warm up one LoRA per bucket and send signatures

\{si,c\}\\\{s\_\{i,c\}\\\}
4:endfor

5:Cluster signatures into

M⋆M^\{\\star\}experts and obtain

a​\(i,c\)a\(i,c\)
6:Initialize global experts

\{θm0\}\\\{\\theta\_\{m\}^\{0\}\\\}and router

φ0\\varphi^\{0\}
7:for

t=0t=0to

T−1T\-1do

8:Sample

𝒮t\\mathcal\{S\}\_\{t\}; broadcast

\{θmt\}\\\{\\theta\_\{m\}^\{t\}\\\},

φt\\varphi^\{t\}, and assignments

9:for all

i∈𝒮ti\\in\\mathcal\{S\}\_\{t\}in paralleldo

10:Allocate

EEsteps by bucket sample counts and interleave bucket IDs

11:Update soft\-routed experts and the broadcast global\-router state

12:Return

Δ​θi,m\\Delta\\theta\_\{i,m\}when

ni,m\>0n\_\{i,m\}\>0and one

Δ​φi\\Delta\\varphi\_\{i\}
13:endfor

14:Aggregate expert and router deltas using Eq\. \([4](https://arxiv.org/html/2607.26618#Sx3.E4)\)

15:endfor

16:At inference, activate

θarg⁡maxm⁡rm​\(x;φ\)\\theta\_\{\\arg\\max\_\{m\}r\_\{m\}\(x;\\varphi\)\}for each input

xx

### Sparse Inference

Training uses the full softmax mixture so that each example supplies comparative supervision across experts\. At inference,FedWeaveselectsm^​\(x\)=arg⁡maxm⁡rm​\(x;φ\)\\widehat\{m\}\(x\)=\\arg\\max\_\{m\}r\_\{m\}\(x;\\varphi\)and activates onlyθm^​\(x\)\\theta\_\{\\widehat\{m\}\(x\)\}; the analysis below connects router optimization to the risk of this sparse decision\.

### Theory of Asymmetric Aggregation

The analysis formalizes three advantages ofFedWeave\. First, prototype conditioning givesO​\(1/U\)O\(1/U\)expert convergence to a stationary neighborhood controlled by routed off\-pattern weight\. Second, persistent router training contracts under the PL condition while exposing deterministic interleaving bias instead of incurring the reset consensus residual\. Third, gap calibration transfers router suboptimality to Top\-1 sparse\-inference risk\. The full assumptions, the mixed\-gradient conflict calculation, and all proofs are given in Appendix[C](https://arxiv.org/html/2607.26618#A3)\.

##### Expert convergence: stationarity under routed contamination\.

The first advantage is expert convergence: prototype\-conditioned aggregation makes the deviation from an expert’s target gradient depend on the off\-pattern weight routed to that expert, rather than the full client mixture\. To quantify it, fix the router and the other experts, let expertmmtarget patternκm\\kappa\_\{m\}, and letχm​\(θm\)=∑i,c,k≠κmωi,c,k,m​\(θm\)\\chi\_\{m\}\(\\theta\_\{m\}\)=\\sum\_\{i,c,k\\neq\\kappa\_\{m\}\}\\omega\_\{i,c,k,m\}\(\\theta\_\{m\}\)be its off\-target routed contribution weight, whereωi,c,k,m\\omega\_\{i,c,k,m\}is the normalized weight of pattern\-kkexamples in clientii’s bucketccrouted to expertmm\. Under bounded routed gradients and within\-pattern shiftδpat\\delta\_\{\\mathrm\{pat\}\}, the directional error is at mostεmu=2​Bg​χm​\(θmu\)\+δpat\\varepsilon\_\{m\}^\{u\}=2B\_\{g\}\\chi\_\{m\}\(\\theta\_\{m\}^\{u\}\)\+\\delta\_\{\\mathrm\{pat\}\}\. HereFκm,mF\_\{\\kappa\_\{m\},m\}is expertmm’s clean loss on patternκm\\kappa\_\{m\}; assume it isLEL\_\{E\}\-smooth and lower bounded, and that the routed stochastic direction has conditional variance at most\(ρmu\)2​σE2\(\\rho\_\{m\}^\{u\}\)^\{2\}\\sigma\_\{E\}^\{2\}\. Letgmu=∇Fκm,m​\(θmu\)g\_\{m\}^\{u\}=\\nabla F\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{u\}\),ρmu=ρm​\(θmu\)\\rho\_\{m\}^\{u\}=\\rho\_\{m\}\(\\theta\_\{m\}^\{u\}\),amu=ηE​ρmu≤1/LEa\_\{m\}^\{u\}=\\eta\_\{E\}\\rho\_\{m\}^\{u\}\\leq 1/L\_\{E\},Am,U=∑u<U𝔼​\[amu\]\>0A\_\{m,U\}=\\sum\_\{u<U\}\\mathbb\{E\}\[a\_\{m\}^\{u\}\]\>0, andΔE,m0=Fκm,m​\(θm0\)−Fκm,m⋆\\Delta\_\{E,m\}^\{0\}=F\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{0\}\)\-F\_\{\\kappa\_\{m\},m\}^\{\\star\}\.

###### Theorem 1\(Expert convergence under routed contamination\)\.

1Am,U​∑u<U𝔼​\[amu​‖gmu‖2\]\\displaystyle\\tfrac\{1\}\{A\_\{m,U\}\}\\\!\\sum\_\{u<U\}\\mathbb\{E\}\[a\_\{m\}^\{u\}\\\|g\_\{m\}^\{u\}\\\|^\{2\}\]≤2​ΔE,m0Am,U\+1Am,U​∑u<U𝔼​\[amu​\(εmu\)2\]\\displaystyle\\leq\\tfrac\{2\\Delta\_\{E,m\}^\{0\}\}\{A\_\{m,U\}\}\+\\tfrac\{1\}\{A\_\{m,U\}\}\\\!\\sum\_\{u<U\}\\mathbb\{E\}\[a\_\{m\}^\{u\}\(\\varepsilon\_\{m\}^\{u\}\)^\{2\}\]\(5\)\+LE​σE2Am,U​∑u<U𝔼​\[\(amu\)2\]\.\\displaystyle\\quad\+\\tfrac\{L\_\{E\}\\sigma\_\{E\}^\{2\}\}\{A\_\{m,U\}\}\\\!\\sum\_\{u<U\}\\mathbb\{E\}\[\(a\_\{m\}^\{u\}\)^\{2\}\]\.

With non\-vanishing routed weight and bounded contamination, this isO​\(1/U\)O\(1/U\)convergence to a contamination–noise stationary neighborhood;Am,UA\_\{m,U\}records the effective optimization budget received by expertmmand thus exposes expert starvation\. With other terms matched, the prototype bound is no looser ifχm​\(θmu\)≤αmclient,u\\chi\_\{m\}\(\\theta\_\{m\}^\{u\}\)\\leq\\alpha\_\{m\}^\{\\mathrm\{client\},u\}at every active step, and strictly tighter if the inequality is strict with positive probability\.

##### Router convergence: persistent versus reset\-and\-average\.

The second advantage is router convergence: FedWeave keeps one broadcast router state across interleaved buckets, so all bucket updates follow a single trajectory\. For comparison, reset\-and\-average maintains one state per bucket and averages them at the end, mirroring the router\-state reset in the*w/o client\-router*ablation\. With experts fixed, letRi,cR\_\{i,c\}be the router loss on bucketcc, and letRi​\(φ\)=∑cπi,c​Ri,c​\(φ\)R\_\{i\}\(\\varphi\)=\\sum\_\{c\}\\pi\_\{i,c\}R\_\{i,c\}\(\\varphi\),gi​\(φ\)=∇Ri​\(φ\)g\_\{i\}\(\\varphi\)=\\nabla R\_\{i\}\(\\varphi\), andgi,c​\(φ\)=∇Ri,c​\(φ\)g\_\{i,c\}\(\\varphi\)=\\nabla R\_\{i,c\}\(\\varphi\), whereπi,c=\|ℬi,c\|/\|𝒟i\|\\pi\_\{i,c\}=\|\\mathcal\{B\}\_\{i,c\}\|/\|\\mathcal\{D\}\_\{i\}\|is the bucket sample fraction\. AssumeLRL\_\{R\}\-smooth bucket losses, aμ\\mu\-PL client objective\(Karimiet al\.[2016](https://arxiv.org/html/2607.26618#bib.bib12)\), conditional variance at mostσR2\\sigma\_\{R\}^\{2\}, and0<ηR≤1/LR0<\\eta\_\{R\}\\leq 1/L\_\{R\}\. OverSSlocal steps, letg^i,c\\widehat\{g\}\_\{i,c\}be the stochastic gradient on bucketcc; for the persistent schedule and reset states, defineqs​\(φs\)=𝔼​\[g^i,cs​\(φs\)∣φs\]−gi​\(φs\)q\_\{s\}\(\\varphi\_\{s\}\)=\\mathbb\{E\}\[\\widehat\{g\}\_\{i,c\_\{s\}\}\(\\varphi\_\{s\}\)\\mid\\varphi\_\{s\}\]\-g\_\{i\}\(\\varphi\_\{s\}\),φ¯s=∑cπi,c​φc,s\\bar\{\\varphi\}\_\{s\}=\\sum\_\{c\}\\pi\_\{i,c\}\\varphi\_\{c,s\}, andes=∑cπi,c​\[gi,c​\(φc,s\)−gi,c​\(φ¯s\)\]e\_\{s\}=\\sum\_\{c\}\\pi\_\{i,c\}\[g\_\{i,c\}\(\\varphi\_\{c,s\}\)\-g\_\{i,c\}\(\\bar\{\\varphi\}\_\{s\}\)\]\. Let

ℰi,S\\displaystyle\\mathcal\{E\}\_\{i,S\}=\(1−μ​ηR\)S​\(Ri​\(φ0\)−Ri⋆\)\+LR​ηR​σR22​μ,\\displaystyle=\(1\-\\mu\\eta\_\{R\}\)^\{S\}\(R\_\{i\}\(\\varphi\_\{0\}\)\-R\_\{i\}^\{\\star\}\)\+\\tfrac\{L\_\{R\}\\eta\_\{R\}\\sigma\_\{R\}^\{2\}\}\{2\\mu\},\(6\)𝒪i,S\\displaystyle\\mathcal\{O\}\_\{i,S\}=ηR2​∑s<S\(1−μ​ηR\)S−1−s​𝔼​‖qs​\(φs\)‖2,\\displaystyle=\\tfrac\{\\eta\_\{R\}\}\{2\}\\sum\_\{s<S\}\(1\-\\mu\\eta\_\{R\}\)^\{S\-1\-s\}\\mathbb\{E\}\\\|q\_\{s\}\(\\varphi\_\{s\}\)\\\|^\{2\},𝒞i,S\\displaystyle\\mathcal\{C\}\_\{i,S\}=ηR2​∑s<S\(1−μ​ηR\)S−1−s​𝔼​‖es‖2\.\\displaystyle=\\tfrac\{\\eta\_\{R\}\}\{2\}\\sum\_\{s<S\}\(1\-\\mu\\eta\_\{R\}\)^\{S\-1\-s\}\\mathbb\{E\}\\\|e\_\{s\}\\\|^\{2\}\.
###### Theorem 2\(Persistent versus reset\-and\-average router convergence\)\.

Under matched initialization, depth, and effective pattern weights,

𝔼​\[Ri​\(φS\)−Ri⋆\]\\displaystyle\\mathbb\{E\}\[R\_\{i\}\(\\varphi\_\{S\}\)\-R\_\{i\}^\{\\star\}\]≤ℰi,S\+𝒪i,S,\\displaystyle\\leq\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{O\}\_\{i,S\},\(7\)𝔼​\[Ri​\(φ¯S\)−Ri⋆\]\\displaystyle\\mathbb\{E\}\[R\_\{i\}\(\\bar\{\\varphi\}\_\{S\}\)\-R\_\{i\}^\{\\star\}\]≤ℰi,S\+𝒞i,S\.\\displaystyle\\leq\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{C\}\_\{i,S\}\.

Both procedures share the same PL contraction and stochastic\-noise floor\. Sample\-count\-proportional randomized bucket sampling gives𝒪i,S=0\\mathcal\{O\}\_\{i,S\}=0;FedWeave’s deterministic interleaving incurs𝒪i,S\\mathcal\{O\}\_\{i,S\}, whereas reset\-and\-average incurs𝒞i,S\\mathcal\{C\}\_\{i,S\}from bucket\-local dispersion\.

##### Sparse inference: router error controls Top\-1 risk\.

Finally, although training uses a soft mixture, inference activates only one expert; insufficient router probability on the latent oracle expert can therefore incur excess loss\. In the clean one\-pattern\-per\-expert regime, letm†​\(x\)m^\{\\dagger\}\(x\)be the oracle expert,Ri,oracleR\_\{i,\\mathrm\{oracle\}\}its risk, andSi​\(φ\)=𝔼​\[1−rm†​\(x\)​\(x;φ\)\]S\_\{i\}\(\\varphi\)=\\mathbb\{E\}\[1\-r\_\{m^\{\\dagger\}\(x\)\}\(x;\\varphi\)\]\. Assume the expert\-loss\-gap calibrationcgap​Si​\(φ\)≤Ri​\(φ\)−Ri,oracle\+βmixc\_\{\\mathrm\{gap\}\}S\_\{i\}\(\\varphi\)\\leq R\_\{i\}\(\\varphi\)\-R\_\{i,\\mathrm\{oracle\}\}\+\\beta\_\{\\mathrm\{mix\}\}withcgap\>0c\_\{\\mathrm\{gap\}\}\>0, and letAi=\[Ri⋆−Ri,oracle\]\+A\_\{i\}=\[R\_\{i\}^\{\\star\}\-R\_\{i,\\mathrm\{oracle\}\}\]\_\{\+\}\. If non\-oracle excess loss is bounded byΔmax\\Delta\_\{\\max\}, then the Top\-1 riskRsparse,iR\_\{\\mathrm\{sparse\},i\}obeys

Rsparse,i−Ri,oracle≤2​Δmaxcgap​\[Ri​\(φ\)−Ri⋆\+Ai\+βmix\]\.R\_\{\\mathrm\{sparse\},i\}\-R\_\{i,\\mathrm\{oracle\}\}\\leq\\frac\{2\\Delta\_\{\\max\}\}\{c\_\{\\mathrm\{gap\}\}\}\[R\_\{i\}\(\\varphi\)\-R\_\{i\}^\{\\star\}\+A\_\{i\}\+\\beta\_\{\\mathrm\{mix\}\}\]\.\(8\)Taking expectations, Theorem[2](https://arxiv.org/html/2607.26618#Thmtheorem2)suppliesℰi,S\+𝒪i,S\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{O\}\_\{i,S\}forφS\\varphi\_\{S\}andℰi,S\+𝒞i,S\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{C\}\_\{i,S\}forφ¯S\\bar\{\\varphi\}\_\{S\}\.

Table 1:Main results atα=0\.3\\alpha=0\.3\(mean±\\pmstandard deviation over seeds 42/43/44\)\. “Macro Avg\.” is the unweighted mean across tasks within each panel\. Best and second\-best results areboldandunderlined\.

## Experiments

### Experimental Setup

##### Benchmark and Datasets\.

Our controlled heterogeneous multi\-task benchmark combines CoEdIT\(Rahejaet al\.[2023](https://arxiv.org/html/2607.26618#bib.bib20)\), GSM8K\(Cobbeet al\.[2021](https://arxiv.org/html/2607.26618#bib.bib3)\), TweetEval sentiment classification \(TweetEval\)\(Barbieriet al\.[2020](https://arxiv.org/html/2607.26618#bib.bib1)\), and ARC\-Challenge \(ARC\-C\)\(Clarket al\.[2018](https://arxiv.org/html/2607.26618#bib.bib2)\)\. For each seed, we sample disjoint per\-task subsets of 2,000/100/400 train/validation/test examples from the pooled sources, yielding 1,600 test examples in total\. Equal task budgets keep client mixtures and macro\-averages comparable across tasks\.

##### Federated training\.

We partition training data among 20 clients by task\-identity Dirichlet sampling; smallerα\\alphameans stronger client imbalance\. The main setting isα=0\.3\\alpha=0\.3; mean±\\pmstandard deviation over seeds 42/43/44 jointly reflects data subsampling, client partitioning, and training randomness\. Task labels construct and audit this controlled benchmark but are never exposed toFedWeave\. The benchmark contains distinct instruction patterns that can be recovered from the frozen representations; Table[3](https://arxiv.org/html/2607.26618#Sx4.T3)reports the resulting alignment quality\. We evaluate Llama3\.2\-3B\(Meta AI[2024](https://arxiv.org/html/2607.26618#bib.bib17)\)and Gemma\-2\-2B\(Gemma Teamet al\.[2024](https://arxiv.org/html/2607.26618#bib.bib18)\)with all 20 clients participating for 20 rounds and 10 local optimizer steps per round\. All methods use the same local\-step budget;FedWeaveredistributes its 10 steps across buckets according to their sample counts\. All runs use AdamW with learning rate 1e\-4, effective batch size 8, gradient clipping at 1\.0, and maximum length 512\. Single\-adapter baselines use one rank\-32 LoRA, whereasFedWeaveand FedLEASE use rank 8 per expert; all LoRAs use scaling factor 16, dropout 0\.05, and query/value projections\.FedWeaveand FedLEASE select the expert count adaptively by silhouette search over prototype and client signatures, respectively; the reported main runs selected four experts for both methods\. For either backbone, this realized count matches the total LoRA capacity of its rank\-32 single\-adapter counterpart\.FedWeavetrains a hidden\-size\-512 global router with learning rate 5e\-5, zero dropout\. Experts average their aligned warmup adapters, whereas the two\-layer router uses default random initialization and initially distributes weight across all experts via soft routing\. At test time, sparse inference activates one expert\.

##### Evaluation\.

CoEdIT uses mean ROUGE\-1/2/Lsum F1; GSM8K and ARC\-C use exact\-answer accuracy; and TweetEval uses accuracy\. Greedy decoding permits 64/192/4/4 new tokens for CoEdIT/GSM8K/TweetEval/ARC\-C; full settings are in Appendix[A](https://arxiv.org/html/2607.26618#A1)\.

### Baselines and Comparison Scope

Holding the backbone, LoRA capacity, data, and global\-model target fixed, we compare four representative federated LoRA approaches\.FedITjointly trains both factors of one shared LoRA and aggregates client deltas by sample\-weighted FedAvg\.FFA\-LoRAfreezes the randomly initializedAAfactor and trains and aggregates only the zero\-initializedBBfactor\.FedSA\-LoRAaggregatesAAglobally while retaining client\-localBBstates across rounds; global evaluation pairs the sharedAAwith their sample\-weighted mean\.FedLEASE, the closest empirical baseline, selects the client\-cluster count by silhouette score and maintains one expert and router for each selected cluster\. Backbone\-specific trainable\-parameter counts are tabulated in Table[5](https://arxiv.org/html/2607.26618#A1.T5)of Appendix[A](https://arxiv.org/html/2607.26618#A1)\.FedWeaveremains in the same trainable\-parameter range as the full\-adapter baselines FedIT/FedSA and smaller than FedLEASE\.

### Overall Effectiveness

Table[1](https://arxiv.org/html/2607.26618#Sx3.T1)shows that, atα=0\.3\\alpha=0\.3,FedWeaveraises the best\-baseline macro\-average score from 0\.5673 to 0\.5872 on Llama3\.2\-3B and from 0\.4839 to 0\.5163 on Gemma\-2\-2B, while reducing macro\-average loss from 0\.7496 to 0\.7264 and from 0\.7443 to 0\.6954, respectively\. On Llama3\.2\-3B it leads on GSM8K, TweetEval, and ARC\-C, while FedIT remains strongest on CoEdIT; on Gemma\-2\-2B it ranks first on all four tasks\. The gain therefore persists across two architectures without requiring a task\-specific expert assignment\. Figure[2](https://arxiv.org/html/2607.26618#Sx4.F2)shows thatFedWeaveassigns the four tasks to distinct dominant experts \(E1/E4/E3/E2\), whereas FedLEASE reuses E3 for CoEdIT and GSM8K and distributes the remaining tasks more broadly\.

![Refer to caption](https://arxiv.org/html/2607.26618v1/x2.png)Figure 2:Row\-normalized task–expert routing weights on Llama3\.2\-3B atα=0\.3\\alpha=0\.3\.FedWeavelearns a distinct dominant expert for each task compared to FedLEASE\.
### Robustness to Client Heterogeneity

Figure[3](https://arxiv.org/html/2607.26618#Sx4.F3)shows that, against FedLEASE on Llama3\.2\-3B,FedWeaveimproves macro\-average score by 0\.0150, 0\.0223, and 0\.0106 forα=0\.1,0\.3,0\.5\\alpha=0\.1,0\.3,0\.5, respectively, while reducing macro\-average loss by 0\.0256, 0\.0336, and 0\.0245\. The advantage remains positive at all three heterogeneity levels and is largest atα=0\.3\\alpha=0\.3\.

![Refer to caption](https://arxiv.org/html/2607.26618v1/x3.png)Figure 3:Macro\-average score and teacher\-forced loss on Llama3\.2\-3B across Dirichlet heterogeneity levels\. Shaded regions show one standard deviation over three seeds\.
### Ablation Study

#### Aggregation Granularity

With the same backbone, data, and expert capacity,*w/o prototype\-expert*uses Client/Client aggregation, whereas*w/o client\-router*uses Prototype/Prototype aggregation with per\-bucket router resets and sample\-weighted endpoints\.

Table 2:Aggregation\-granularity ablation\.*w/o prototype\-expert*: Client/Client;*w/o client\-router*: Prototype/Prototype with per\-bucket router resets and sample\-weighted endpoints;FedWeave: Prototype/Client with one global router following complete client trajectories\.Table 3:Prototype alignment quality under different local clusterers\. Purity measures dominant\-label agreement, NMI measures normalized information agreement, and ARI measures chance\-corrected pairwise agreement; exact definitions are in Appendix[A](https://arxiv.org/html/2607.26618#A1.SSx1)\. Complexities are per client, whereni=\|𝒟i\|n\_\{i\}=\|\\mathcal\{D\}\_\{i\}\|,CiC\_\{i\}is the selected local bucket count,ded\_\{e\}is the frozen embedding dimension, andIIis the number ofkk\-means iterations; server\-side signature alignment is fixed\.Table[2](https://arxiv.org/html/2607.26618#Sx4.T2)shows macro\-average gains of 0\.0223 and 0\.0294 over*w/o prototype\-expert*and*w/o client\-router*, with loss reductions of 0\.0336 and 0\.0428\.FedWeaveleads every task score and loss; against*w/o client\-router*, its largest score gains on TweetEval and ARC\-C support preserving cross\-bucket contrast within one router trajectory\. The*w/o prototype\-expert*comparison supports forming experts from coherent bucket contexts\. Because the*w/o client\-router*implementation has effective weightπ~c∝ac​Hc\\widetilde\{\\pi\}\_\{c\}\\propto a\_\{c\}H\_\{c\}, whereaca\_\{c\}is the endpoint aggregation coefficient andHcH\_\{c\}is the reset depth for bucketcc, its comparison withFedWeavecaptures both router\-state fragmentation and bucket reweighting, as predicted by the asymmetric analysis\. These comparisons support the component\-dependent purity–contrast design\.

#### Prototype Discovery

Table[3](https://arxiv.org/html/2607.26618#Sx4.T3)compares client\-side clusterers\.kk\-means and spectral clustering achieve nearly identical global purity \(0\.9515 versus 0\.9499\) and similar macro scores \(0\.5872 versus 0\.5889\)\. However,kk\-means scales asO​\(I​ni​Ci​de\)O\(In\_\{i\}C\_\{i\}d\_\{e\}\), whereas spectral clustering costsO​\(ni3\)O\(n\_\{i\}^\{3\}\)\. We therefore adoptkk\-means as the default because it preserves high\-quality prototype discovery at substantially lower asymptotic cost\.

#### Sparse Inference

Using the same trained checkpoints over three seeds, Figure[4](https://arxiv.org/html/2607.26618#Sx4.F4)shows that sparse inference \(activating the highest\-scoring expert\) reaches 0\.5872 versus 0\.5883 for soft routing and lowers mean latency from 3177 to 2147 ms \(32\.4%\)\. Activating two experts reaches 0\.5890 at 2587 ms, only 0\.0018 above sparse inference at higher latency\.

![Refer to caption](https://arxiv.org/html/2607.26618v1/x4.png)Figure 4:Sparse\-inference quality–latency trade\-off over three seeds\. Sparse inference activates the Top\-1 expert at lower latency but nearly preserves soft\-routing quality\.

## Limitations

##### Evaluation scope\.

Due to the limited computational resources, our study focuses on 2B–3B backbones and controlled four\-task tests\. This scope enables matched comparisons across methods, backbones, and heterogeneity levels, while broader validation on larger models and naturally occurring mixtures remains future work\.

##### Recoverable specialization\.

FedWeaveassumes that local patterns are sufficiently separable in representation and adaptation\-signature space for prototype recovery and cross\-client alignment\. The high alignment purity in Table[3](https://arxiv.org/html/2607.26618#Sx4.T3)supports this condition in the studied regime, where pronounced task heterogeneity makes specialization most relevant\. When heterogeneity is weak and adaptation signatures become less distinguishable, the benefit of specialization may diminish\.

##### Privacy and discovery overhead\.

FedWeavecurrently does not incorporate differential privacy or secure aggregation for its update\-derived signatures; integrating these mechanisms is an important extension\. Prototype discovery also introduces a one\-time first\-stage overhead from embedding, short LoRA warmups, and signature upload, which is amortized across subsequent communication rounds\. Detailed computation, communication, and parameter accounting is provided in Appendix[B](https://arxiv.org/html/2607.26618#A2)\.

## Conclusion

FedWeaveaddresses intra\-client task heterogeneity by assigning specialization and routing to different aggregation units: aligned local prototypes form coherent expert updates, while a single global router trained from complete client trajectories retains cross\-pattern contrast\. The component\-wise theory mirrors this design: prototype conditioning controls routing\-weighted expert contamination, persistent trajectories replace the reset consensus residual with an explicit schedule\-order residual, and gap calibration connects router optimization to sparse\-inference risk\. Across two backbones, it outperforms the evaluated baselines on the controlled benchmark, and its advantage over FedLEASE persists across three heterogeneity levels\. The granularity ablation supports the asymmetric design, while sparse inference preserves nearly all soft\-routing quality with one active expert\. These results identify aggregation granularity, rather than expert count alone, as a central design choice for federated MoE\-LoRA\. Future work should test natural, drifting streams with privacy\-preserving online alignment\.

## References

- F\. Barbieri, J\. Camacho\-Collados, L\. Espinosa Anke, and L\. Neves \(2020\)TweetEval: unified benchmark and comparative evaluation for tweet classification\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1644–1650\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.148)Cited by:[Benchmark and Datasets\.](https://arxiv.org/html/2607.26618#Sx4.SSx1.SSS0.Px1.p1.1)\.
- J\. Bian, L\. Wang, L\. Zhang, and J\. Xu \(2025\)LoRA\-FAIR: federated LoRA fine\-tuning with aggregation and initialization refinement\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3737–3746\.Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- J\. Bian, L\. Wang, L\. Zhang, and J\. Xu \(2026a\)FedALT: federated fine\-tuning through adaptive local training with rest\-of\-world LoRA\.Proceedings of the AAAI Conference on Artificial Intelligence40\(24\),pp\. 19728–19736\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i24.39054)Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- J\. Bian, L\. Wang, L\. Zhang, and J\. Xu \(2026b\)FedTreeLoRA: reconciling statistical and functional heterogeneity in federated LoRA fine\-tuning\.arXiv preprint arXiv:2603\.13282\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- J\. Chen, C\. Liu, and J\. Zhu \(2026\)Beyond factor aggregation: gauge\-aware low\-rank server representations for federated LoRA\.arXiv preprint arXiv:2605\.06733\.Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- S\. Chen, Z\. Jie, and L\. Ma \(2024\)LLaVA\-MoLE: sparse mixture of LoRA experts for mitigating data conflicts in instruction finetuning MLLMs\.arXiv preprint arXiv:2401\.16160\.Cited by:[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. J\. Cho, L\. Liu, Z\. Xu, A\. Fahrezi, and G\. Joshi \(2024\)Heterogeneous LoRA for federated fine\-tuning of on\-device foundation models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 12903–12913\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.717)Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[Benchmark and Datasets\.](https://arxiv.org/html/2607.26618#Sx4.SSx1.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[Benchmark and Datasets\.](https://arxiv.org/html/2607.26618#Sx4.SSx1.SSS0.Px1.p1.1)\.
- T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer \(2023\)QLoRA: efficient finetuning of quantized LLMs\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- B\. Fan, X\. Su, S\. Tarkoma, and P\. Hui \(2025\)HeLoRA: LoRA\-heterogeneous federated fine\-tuning for foundation models\.ACM Transactions on Internet Technology25\(2\),pp\. 11:1–11:22\.External Links:[Document](https://dx.doi.org/10.1145/3723877)Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- C\. Feng, K\. Yu, N\. Liu, X\. Xu, S\. Khan, and W\. Zuo \(2023\)Towards instance\-adaptive inference for federated learning\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 23287–23296\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p2.1),[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- Gemma Team, M\. Rivière, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard, B\. Shahriari, A\. Rame,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2408.00118)Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p6.1),[Federated training\.](https://arxiv.org/html/2607.26618#Sx4.SSx1.SSS0.Px2.p1.3)\.
- A\. Ghosh, J\. Chung, D\. Yin, and K\. Ramchandran \(2020\)An efficient framework for clustered federated learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 19586–19597\.Cited by:[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- P\. Guo, S\. Zeng, Y\. Wang, H\. Fan, F\. Wang, and L\. Qu \(2025\)Selective aggregation for low\-rank adaptation in federated learning\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1),[Local Prototype Discovery](https://arxiv.org/html/2607.26618#Sx3.SSx2.p2.2)\.
- J\. Hao, Y\. Wu, A\. Payani, M\. Lee, and M\. Liu \(2025\)Personalized federated fine\-tuning for heterogeneous data: an automatic rank learning approach via two\-level LoRA\.arXiv preprint arXiv:2503\.03920\.Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Huang, Z\. Yang, Z\. Wang, J\. Qi, R\. Yu, X\. Fan, and C\. Wang \(2026\)Hybrid routing for a mixture of LoRA experts\.Proceedings of the AAAI Conference on Artificial Intelligence40\(37\),pp\. 31211–31219\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i37.40383)Cited by:[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- P\. Kairouz, H\. B\. McMahan, B\. Avent, A\. Bellet, M\. Bennis, A\. N\. Bhagoji, K\. Bonawitz, Z\. Charles, G\. Cormode, R\. Cummings,et al\.\(2021\)Advances and open problems in federated learning\.Foundations and Trends in Machine Learning14\(1–2\),pp\. 1–210\.External Links:[Document](https://dx.doi.org/10.1561/2200000083)Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- H\. Karimi, J\. Nutini, and M\. Schmidt \(2016\)Linear convergence of gradient and proximal\-gradient methods under the polyak–Łojasiewicz condition\.InMachine Learning and Knowledge Discovery in Databases,pp\. 795–811\.Cited by:[Router convergence: persistent versus reset\-and\-average\.](https://arxiv.org/html/2607.26618#Sx3.SSx7.SSS0.Px2.p1.16)\.
- S\. P\. Karimireddy, S\. Kale, M\. Mohri, S\. Reddi, S\. Stich, and A\. T\. Suresh \(2020\)SCAFFOLD: stochastic controlled averaging for federated learning\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 5132–5143\.Cited by:[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- J\. Koo, M\. Jang, and J\. Ok \(2025\)Towards robust and efficient federated low\-rank adaptation with heterogeneous clients\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 416–429\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.19)Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- S\. Lee, S\. Park, D\. B\. Lee, D\. Wagner, H\. Seong, T\. Bocklet, J\. Lee, and S\. J\. Hwang \(2025\)FedSVD: adaptive orthogonalization for private federated learning with LoRA\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- T\. Li, A\. K\. Sahu, M\. Zaheer, M\. Sanjabi, A\. Talwalkar, and V\. Smith \(2020\)Federated optimization in heterogeneous networks\.InProceedings of Machine Learning and Systems,Vol\.2,pp\. 429–450\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- M\. Liao, W\. Chen, J\. Shen, S\. Guo, and H\. Wan \(2025\)HMoRA: making LLMs more effective with hierarchical mixture of LoRA experts\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- O\. Marfoq, G\. Neglia, A\. Bellet, L\. Kameni, and R\. Vidal \(2021\)Federated multi\-task learning under a mixture of distributions\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 15434–15447\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p2.1),[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- H\. B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. Aguera y Arcas \(2017\)Communication\-efficient learning of deep networks from decentralized data\.InProceedings of the 20th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.54,pp\. 1273–1282\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1),[Asymmetric Federated Aggregation](https://arxiv.org/html/2607.26618#Sx3.SSx5.p1.6)\.
- H\. Mei, D\. Cai, A\. Zhou, S\. Wang, and M\. Xu \(2024\)FedMoE: personalized federated learning via heterogeneous mixture of experts\.arXiv preprint arXiv:2408\.11304\.Cited by:[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- Meta AI \(2024\)Llama 3\.2 model card\.Note:https://github\.com/meta\-llama/llama\-models/blob/main/models/llama3\_2/MODEL\_CARD\.mdCited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p6.1),[Federated training\.](https://arxiv.org/html/2607.26618#Sx4.SSx1.SSS0.Px2.p1.3)\.
- V\. Raheja, D\. Kumar, R\. Koo, and D\. Kang \(2023\)CoEdIT: text editing by task\-specific instruction tuning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 5274–5291\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.350)Cited by:[Benchmark and Datasets\.](https://arxiv.org/html/2607.26618#Sx4.SSx1.SSS0.Px1.p1.1)\.
- M\. Reisser, C\. Louizos, E\. Gavves, and M\. Welling \(2021\)Federated mixture of experts\.arXiv preprint arXiv:2107\.06724\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- P\. J\. Rousseeuw \(1987\)Silhouettes: a graphical aid to the interpretation and validation of cluster analysis\.Journal of Computational and Applied Mathematics20,pp\. 53–65\.External Links:[Document](https://dx.doi.org/10.1016/0377-0427%2887%2990125-7)Cited by:[Cross\-Client Prototype Alignment](https://arxiv.org/html/2607.26618#Sx3.SSx3.p1.9)\.
- T\. Sang, Z\. Chu, J\. Xuan, and X\. Zhang \(2026\)A communication\-efficient personalized federated learning framework driven by parameter decoupling\.Neurocomputing692,pp\. 133790\.External Links:[Document](https://dx.doi.org/10.1016/j.neucom.2026.133790)Cited by:[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- F\. Sattler, K\. Müller, and W\. Samek \(2021\)Clustered federated learning: model\-agnostic distributed multi\-task optimization under privacy constraints\.IEEE Transactions on Neural Networks and Learning Systems32\(8\),pp\. 3710–3722\.External Links:[Document](https://dx.doi.org/10.1109/TNNLS.2020.3015958)Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Federated learning under heterogeneity\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px1.p1.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- Z\. Shen, J\. Lu, H\. Wan, and J\. Chen \(2026\)SDFLoRA: selective decoupled federated LoRA for privacy\-preserving fine\-tuning with heterogeneous clients\.arXiv preprint arXiv:2601\.11219\.Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- R\. Singhal, K\. Ponkshe, R\. Vartak, L\. R\. Varshney, and P\. Vepakomma \(2026\)Fed\-SB: a silver bullet for extreme communication efficiency and performance in \(private\) federated LoRA fine\-tuning\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=87UyFEhzyP)Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- Y\. Sun, Z\. Li, Y\. Li, and B\. Ding \(2024\)Improving LoRA in privacy\-preserving federated learning\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1),[Local Prototype Discovery](https://arxiv.org/html/2607.26618#Sx3.SSx2.p2.2)\.
- G\. U\. Talasso, M\. Kurmanji, A\. M\. de Souza, N\. D\. Lane, and L\. A\. Villas \(2026\)Task\-centric personalized federated fine\-tuning of language models\.arXiv preprint arXiv:2604\.00050\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p2.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- L\. Wang, J\. Bian, L\. Zhang, and J\. Xu \(2025\)Adaptive LoRA experts allocation and selection for federated fine\-tuning\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- Z\. Wang, Z\. Shen, Y\. He, G\. Sun, H\. Wang, L\. Lyu, and A\. Li \(2024\)FLoRA: federated fine\-tuning large language models with heterogeneous low\-rank adaptations\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Document](https://dx.doi.org/10.52202/079017-0708)Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- N\. Yang, S\. Wang, O\. Huan, M\. Chen, T\. Q\. S\. Quek, and C\. Yin \(2026\)Wireless federated multi\-task LLM fine\-tuning via sparse\-and\-orthogonal LoRA\.arXiv preprint arXiv:2602\.20492\.Cited by:[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- Y\. Yang, G\. Long, Q\. Lu, L\. Zhu, J\. Jiang, and C\. Zhang \(2025\)Federated low\-rank adaptation for foundation models: a survey\.InProceedings of the Thirty\-Fourth International Joint Conference on Artificial Intelligence,pp\. 10779–10787\.External Links:[Document](https://dx.doi.org/10.24963/ijcai.2025/1196)Cited by:[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.
- E\. L\. Zec, O\. Mogren, J\. Martinsson, L\. R\. Sütfeld, and D\. Gillblad \(2020\)Specialized federated learning using a mixture of experts\.arXiv preprint arXiv:2010\.02056\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p3.1),[Expert models and sparse routing\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px3.p1.1)\.
- J\. Zhang, S\. Vahidian, M\. Kuo, C\. Li, R\. Zhang, T\. Yu, Y\. Zhou, G\. Wang, and Y\. Chen \(2023\)Towards building the federated GPT: federated instruction tuning\.arXiv preprint arXiv:2305\.05644\.Cited by:[Introduction](https://arxiv.org/html/2607.26618#Sx1.p1.1),[Federated PEFT for LLMs\.](https://arxiv.org/html/2607.26618#Sx2.SS0.SSS0.Px2.p1.1)\.

## Appendix AImplementation and Reproducibility Details

Table[4](https://arxiv.org/html/2607.26618#A1.T4)collects the resolved settings used by the reported runs\. Local discovery usesℓ2\\ell\_\{2\}\-normalized, mean\-pooled last\-layer prompt representations from the frozen backbone\. For a client withnnexamples, the candidate counts are2,…,min⁡\(8,n−1\)2,\\ldots,\\min\(8,n\-1\); the default iskk\-means with 10 restarts, selected by Euclidean silhouette score\. Each bucket then warms up a rank\-8 LoRA from the same initialization for 10 steps with batch size 8\. Its signature concatenates the LoRA\-BBfactors on query/value projections across layers\. The server computes the mean layer\-wise cosine distance between signatures and applies silhouette\-selected agglomerative clustering over the same feasible count range, so the number of global experts is data\-adaptive; this procedure selects four experts in the reported main runs\. FedLEASE independently searches from 2 to at most 8 client clusters by the same silhouette criterion and likewise selects four experts in the reported runs\.

All 20 clients participate in every communication round\. The router receives the attention\-mask\-weighted mean of the frozen backbone’s last hidden states, computed with LoRA adapters disabled, and uses a two\-layer MLP with hidden size 512 followed by a full softmax over experts\. The single global router is broadcast to each client, updated across its interleaved bucket schedule, and returned as one delta per round\. Generation uses greedy decoding \(do\_sample=False,num\_beams=1\)\. The code package accompanying the paper contains training and evaluation scripts, deterministic data\-construction code, and a manifest recording source dataset identifiers, split pools, sample budgets, and seed protocols\. When executed, the generated data cache records the source split and row index of every sampled example\.

Table 4:Resolved implementation settings for the reported experiments\. “Effective batch” counts gradient accumulation; only the memory\-bound Gemma FedLEASE runs use micro\-batch 4 with accumulation 2\. Full\-softmax training uses every selected expert, so the stored cumulative\-probability threshold is inactive in these runs\.Table 5:Trainable\-parameter budget for the two backbones\. Counts exclude the frozen backbone, optimizer state, and replicated client\-local copies\. “Adaptive \(4\)” means that silhouette selection determined the expert count and selected four in the reported runs\.### Alignment Metrics

LetΩ=\{ω1,…,ωR\}\\Omega=\\\{\\omega\_\{1\},\\ldots,\\omega\_\{R\}\\\}be the discovered clusters and𝒴=\{y1,…,yK\}\\mathcal\{Y\}=\\\{y\_\{1\},\\ldots,y\_\{K\}\\\}the reference pattern labels used only for evaluation\. Fornnevaluated examples or prototypes, purity is the fraction assigned consistently with the dominant reference label in each cluster:

Purity⁡\(Ω,𝒴\)=1n​∑r=1Rmaxk⁡\|ωr∩yk\|\.\\operatorname\{Purity\}\(\\Omega,\\mathcal\{Y\}\)=\\frac\{1\}\{n\}\\sum\_\{r=1\}^\{R\}\\max\_\{k\}\|\\omega\_\{r\}\\cap y\_\{k\}\|\.\(9\)Normalized mutual information measures the information shared by the two partitions:

NMI⁡\(Ω,𝒴\)=2​I​\(Ω;𝒴\)H​\(Ω\)\+H​\(𝒴\)\.\\operatorname\{NMI\}\(\\Omega,\\mathcal\{Y\}\)=\\frac\{2I\(\\Omega;\\mathcal\{Y\}\)\}\{H\(\\Omega\)\+H\(\\mathcal\{Y\}\)\}\.\(10\)Fornr​k=\|ωr∩yk\|n\_\{rk\}=\|\\omega\_\{r\}\\cap y\_\{k\}\|,ar=∑knr​ka\_\{r\}=\\sum\_\{k\}n\_\{rk\}, andbk=∑rnr​kb\_\{k\}=\\sum\_\{r\}n\_\{rk\}, the adjusted Rand index is

ARI=∑r,k\(nr​k2\)−∑r\(ar2\)​∑k\(bk2\)\(n2\)12​\[∑r\(ar2\)\+∑k\(bk2\)\]−∑r\(ar2\)​∑k\(bk2\)\(n2\)\.\\operatorname\{ARI\}=\\frac\{\\sum\_\{r,k\}\\binom\{n\_\{rk\}\}\{2\}\-\\dfrac\{\\sum\_\{r\}\\binom\{a\_\{r\}\}\{2\}\\sum\_\{k\}\\binom\{b\_\{k\}\}\{2\}\}\{\\binom\{n\}\{2\}\}\}\{\\dfrac\{1\}\{2\}\\left\[\\sum\_\{r\}\\binom\{a\_\{r\}\}\{2\}\+\\sum\_\{k\}\\binom\{b\_\{k\}\}\{2\}\\right\]\-\\dfrac\{\\sum\_\{r\}\\binom\{a\_\{r\}\}\{2\}\\sum\_\{k\}\\binom\{b\_\{k\}\}\{2\}\}\{\\binom\{n\}\{2\}\}\}\.\(11\)Purity emphasizes dominant\-label correctness, NMI measures global information agreement, and ARI measures pairwise agreement after correcting for chance\.

## Appendix BCost Accounting

Prototype discovery runs once\. For clientii, it embeds\|𝒟i\|\|\\mathcal\{D\}\_\{i\}\|examples, clusters them, and performs10​Ci10C\_\{i\}bucket warmup steps\. Federated training schedules20×10=20020\\times 10=200local steps per client, so warmup alone adds aCi/20C\_\{i\}/20step ratio \(20% for four buckets\), excluding embedding and clustering\. ForP=∑iCiP=\\sum\_\{i\}C\_\{i\}uploaded prototypes, the server computes pairwise layer\-wise cosine distances and silhouette\-based agglomerative groups\.

For rankrroverLLlayers, query/value outputsol,q,ol,vo\_\{l,q\},o\_\{l,v\}give signature dimensiondB=r​∑l=1L\(ol,q\+ol,v\)d\_\{B\}=r\\sum\_\{l=1\}^\{L\}\(o\_\{l,q\}\+o\_\{l,v\}\)\. For Llama3\.2\-3B at rank 8,dB=917,504d\_\{B\}=917\{,\}504FP32 values, or 3\.50 MiB per bucket; clientiisends3\.50​Ci3\.50C\_\{i\}MiB once\. During each communication round,FedWeaveand FedLEASE use rank 8 per adaptively selected expert\. For the four\-expert selections on Llama3\.2\-3B, both instantiate 9\.18M LoRA parameters, matching the total LoRA capacity of rank\-32 FedIT\. Including routing modules, the corresponding worst\-case trainable\-state payload contains 10\.75M scalars forFedWeaveand 15\.48M for FedLEASE; Table[5](https://arxiv.org/html/2607.26618#A1.T5)reports the backbone\-specific totals for both models\. Inactive expert uploads are omitted; bytes depend on precision\. These counts expose deterministic overhead but do not replace end\-to\-end wall\-clock and peak\-memory profiling\.

## Appendix CTheoretical Analysis

This section analyzes the two mechanisms behindFedWeave\. We first analyze gradient conflict and routing\-weighted expert convergence with the router and other experts fixed\. We then fix the experts, compare persistent and reset\-and\-average router optimization, and transfer the bounds to sparse\-inference risk\. Randomized sample\-count\-proportional bucket sampling defines the clean router reference, while an explicit order\-bias term covers deterministic balanced interleaving\. Task identities are used only as observable proxies for latent patterns in the analysis and diagnostics;FedWeavenever observes them\.

### Gradient Conflict under Pattern Mixtures

LetFk​\(θ\)F\_\{k\}\(\\theta\)be the expected loss of latent patternkkfor the component under study and letgk​\(θ\)=∇Fk​\(θ\)g\_\{k\}\(\\theta\)=\\nabla F\_\{k\}\(\\theta\)\. If clientiicontains pattern proportionspi,kp\_\{i,k\}, then its mixed objective and gradient are

Fi​\(θ\)=∑k=1Kpi,k​Fk​\(θ\),gi​\(θ\)=∑k=1Kpi,k​gk​\(θ\)\.F\_\{i\}\(\\theta\)=\\sum\_\{k=1\}^\{K\}p\_\{i,k\}F\_\{k\}\(\\theta\),\\qquad g\_\{i\}\(\\theta\)=\\sum\_\{k=1\}^\{K\}p\_\{i,k\}g\_\{k\}\(\\theta\)\.\(12\)The squared norm of the mixed direction expands as

‖gi‖2=∑kpi,k2​‖gk‖2\+2​∑k<ℓpi,k​pi,ℓ​⟨gk,gℓ⟩\.\\\|g\_\{i\}\\\|^\{2\}=\\sum\_\{k\}p\_\{i,k\}^\{2\}\\\|g\_\{k\}\\\|^\{2\}\+2\\sum\_\{k<\\ell\}p\_\{i,k\}p\_\{i,\\ell\}\\langle g\_\{k\},g\_\{\\ell\}\\rangle\.\(13\)Thus a negative cross\-pattern inner product cancels part of the usable descent direction before server aggregation\. Define the corresponding conflict measure

Γi​\(θ\)=∑k<ℓpi,k​pi,ℓ​\[−⟨gk​\(θ\),gℓ​\(θ\)⟩\]\+\.\\Gamma\_\{i\}\(\\theta\)=\\sum\_\{k<\\ell\}p\_\{i,k\}p\_\{i,\\ell\}\\bigl\[\-\\langle g\_\{k\}\(\\theta\),g\_\{\\ell\}\(\\theta\)\\rangle\\bigr\]\_\{\+\}\.\(14\)
The role of local purity can be stated without assuming that every pattern pair conflicts\. For bucketccof clientii, letqi,c,kq\_\{i,c,k\}be its pattern proportions andk​\(i,c\)=arg⁡maxk⁡qi,c,kk\(i,c\)=\\arg\\max\_\{k\}q\_\{i,c,k\}\. If its off\-dominant probability is at mostϵ\\epsilonand‖gk‖≤G\\\|g\_\{k\}\\\|\\leq G, then its conflict measure satisfies

Γi,cbucket:=∑k<ℓqi,c,k​qi,c,ℓ​\[−⟨gk,gℓ⟩\]\+≤\(ϵ\+ϵ22\)​G2\.\\Gamma\_\{i,c\}^\{\\mathrm\{bucket\}\}:=\\sum\_\{k<\\ell\}q\_\{i,c,k\}q\_\{i,c,\\ell\}\\bigl\[\-\\langle g\_\{k\},g\_\{\\ell\}\\rangle\\bigr\]\_\{\+\}\\leq\\left\(\\epsilon\+\\frac\{\\epsilon^\{2\}\}\{2\}\\right\)G^\{2\}\.\(15\)Indeed, pairs involving the dominant pattern have total probability weight at mostϵ\\epsilon, pairs among off\-dominant patterns have weight at mostϵ2/2\\epsilon^\{2\}/2, and each negative inner product is at mostG2G^\{2\}in magnitude\. Equation \([15](https://arxiv.org/html/2607.26618#A3.E15)\) shows that coherent buckets bound worst\-case within\-update cancellation through their off\-dominant probability\. The reported purity scores diagnose this condition but do not directly measure gradient inner products\.

### Routing\-Weighted Expert Contamination

There areKKlatent patterns andM⋆M^\{\\star\}global experts\. For the clean comparison, assumeK=M⋆K=M^\{\\star\}and a bijectionψ:\{1,…,K\}→\{1,…,M⋆\}\\psi:\\\{1,\\ldots,K\\\}\\to\\\{1,\\ldots,M^\{\\star\}\\\}, and writeκm=ψ−1​\(m\)\\kappa\_\{m\}=\\psi^\{\-1\}\(m\)\. WhenK≠M⋆K\\neq M^\{\\star\}, the fidelity bound still applies to experts with a designated target pattern after unmatched or merged pattern weight is included in the contamination term, but the one\-to\-one oracle\-routing statements below require the clean regime\.

For expertmm, letρm​\(θm\)\>0\\rho\_\{m\}\(\\theta\_\{m\}\)\>0denote its total routed contribution weight and decompose the raw expected soft\-routed gradient by clientii, local bucketcc, and latent patternkk:

hmraw​\(θm\)\\displaystyle\\smash\{h\_\{m\}^\{\\mathrm\{raw\}\}\(\\theta\_\{m\}\)\}=ρm​\(θm\)​h¯m​\(θm\),\\displaystyle=\\rho\_\{m\}\(\\theta\_\{m\}\)\\bar\{h\}\_\{m\}\(\\theta\_\{m\}\),\(16\)h¯m​\(θm\)\\displaystyle\\bar\{h\}\_\{m\}\(\\theta\_\{m\}\)=∑i,c,kωi,c,k,m​\(θm\)​gi,c,k,m​\(θm\),\\displaystyle=\\sum\_\{i,c,k\}\\omega\_\{i,c,k,m\}\(\\theta\_\{m\}\)g\_\{i,c,k,m\}\(\\theta\_\{m\}\),ωi,c,k,m\\displaystyle\\omega\_\{i,c,k,m\}≥0,∑i,c,kωi,c,k,m=1\.\\displaystyle\\geq 0,\\qquad\\sum\_\{i,c,k\}\\omega\_\{i,c,k,m\}=1\.We condition on positive total routed weight when definingh¯m\\bar\{h\}\_\{m\}; an inactive expert has zero raw update\. Hereωi,c,k,m\\omega\_\{i,c,k,m\}is the normalized nonnegative contribution weight after accounting for bucket sampling, prototype\-conditioned upload weighting, and input\-dependent routing gates\. The conditional directiongi,c,k,mg\_\{i,c,k,m\}retains the corresponding loss sensitivity, so Eq\. \([16](https://arxiv.org/html/2607.26618#A3.E16)\) does not factor a mean routing coefficient from an unweighted bucket gradient\.

###### Assumption 1\(Bounded routed gradients and within\-pattern shift\)\.

For everyi,c,k,mi,c,k,mand iterate,

‖gi,c,k,m​\(θm\)‖\\displaystyle\\\|g\_\{i,c,k,m\}\(\\theta\_\{m\}\)\\\|≤Bg,\\displaystyle\\leq B\_\{g\},‖gk,m​\(θm\)‖\\displaystyle\\\|g\_\{k,m\}\(\\theta\_\{m\}\)\\\|≤Bg,\\displaystyle\\leq B\_\{g\},\(17\)‖gi,c,k,m​\(θm\)−gk,m​\(θm\)‖\\displaystyle\\\|g\_\{i,c,k,m\}\(\\theta\_\{m\}\)\-g\_\{k,m\}\(\\theta\_\{m\}\)\\\|≤δpat\.\\displaystyle\\leq\\delta\_\{\\mathrm\{pat\}\}\.wheregk,m=∇Fk,mg\_\{k,m\}=\\nabla F\_\{k,m\}is the reference gradient for patternkkand expertmm\.

Define the effective routed contamination of expertmmas

χm​\(θm\)=∑i,c,k≠κmωi,c,k,m​\(θm\)\.\\chi\_\{m\}\(\\theta\_\{m\}\)=\\sum\_\{i,c,k\\neq\\kappa\_\{m\}\}\\omega\_\{i,c,k,m\}\(\\theta\_\{m\}\)\.\(18\)Letk​\(i,c\)k\(i,c\)be the dominant pattern of clientii’s bucketcc, and leta​\(i,c\)a\(i,c\)be its aligned anchor expert\. Using the same routed contribution weights, define

ϵmloc\\displaystyle\\epsilon\_\{m\}^\{\\mathrm\{loc\}\}=∑i,c,kωi,c,k,m​𝟏​\{k≠k​\(i,c\)\},\\displaystyle=\\sum\_\{i,c,k\}\\omega\_\{i,c,k,m\}\\mathbf\{1\}\\\{k\\neq k\(i,c\)\\\},\(19\)ϵmalign\\displaystyle\\epsilon\_\{m\}^\{\\mathrm\{align\}\}=∑i,c,kωi,c,k,m​𝟏​\{k​\(i,c\)≠κa​\(i,c\)\},\\displaystyle=\\sum\_\{i,c,k\}\\omega\_\{i,c,k,m\}\\mathbf\{1\}\\\{k\(i,c\)\\neq\\kappa\_\{a\(i,c\)\}\\\},ϵmleak\\displaystyle\\epsilon\_\{m\}^\{\\mathrm\{leak\}\}=∑i,c,kωi,c,k,m​𝟏​\{a​\(i,c\)≠m\}\.\\displaystyle=\\sum\_\{i,c,k\}\\omega\_\{i,c,k,m\}\\mathbf\{1\}\\\{a\(i,c\)\\neq m\\\}\.
###### Lemma 1\(Contamination decomposition\)\.

In the clean one\-pattern\-per\-expert regime,

χm≤ϵmloc\+ϵmalign\+ϵmleak\.\\chi\_\{m\}\\leq\\epsilon\_\{m\}^\{\\mathrm\{loc\}\}\+\\epsilon\_\{m\}^\{\\mathrm\{align\}\}\+\\epsilon\_\{m\}^\{\\mathrm\{leak\}\}\.\(20\)

###### Proof\.

Ifk≠κmk\\neq\\kappa\_\{m\}, then at least one of the following must hold:k≠k​\(i,c\)k\\neq k\(i,c\),k​\(i,c\)≠κa​\(i,c\)k\(i,c\)\\neq\\kappa\_\{a\(i,c\)\}, ora​\(i,c\)≠ma\(i,c\)\\neq m\. Otherwisek=k​\(i,c\)=κa​\(i,c\)k=k\(i,c\)=\\kappa\_\{a\(i,c\)\}anda​\(i,c\)=ma\(i,c\)=m, which impliesk=κmk=\\kappa\_\{m\}\. Applying this indicator union bound inside Eq\. \([18](https://arxiv.org/html/2607.26618#A3.E18)\) proves the result\. ∎

The local and global purity values in the main text diagnose the first two terms, while routing concentration qualitatively diagnoses the third\. Equation \([20](https://arxiv.org/html/2607.26618#A3.E20)\) connects these diagnostics toχm\\chi\_\{m\}; numerical equality would require additional assumptions\.

###### Proposition 1\(Routing\-weighted expert fidelity\)\.

Under Assumption[1](https://arxiv.org/html/2607.26618#Thmassumption1),

‖h¯m​\(θm\)−gκm,m​\(θm\)‖≤2​Bg​χm​\(θm\)\+δpat\.\\\|\\bar\{h\}\_\{m\}\(\\theta\_\{m\}\)\-g\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}\)\\\|\\leq 2B\_\{g\}\\chi\_\{m\}\(\\theta\_\{m\}\)\+\\delta\_\{\\mathrm\{pat\}\}\.\(21\)

###### Proof\.

Insertgk,mg\_\{k,m\}into Eq\. \([16](https://arxiv.org/html/2607.26618#A3.E16)\)\. The weighted within\-pattern shift is at mostδpat\\delta\_\{\\mathrm\{pat\}\}\. Fork=κmk=\\kappa\_\{m\}, the remaining difference is zero; fork≠κmk\\neq\\kappa\_\{m\}, boundedness gives‖gk,m−gκm,m‖≤2​Bg\\\|g\_\{k,m\}\-g\_\{\\kappa\_\{m\},m\}\\\|\\leq 2B\_\{g\}\. Weighting the off\-target terms proves Eq\. \([21](https://arxiv.org/html/2607.26618#A3.E21)\)\. ∎

Combining Proposition[1](https://arxiv.org/html/2607.26618#Thmproposition1)with Lemma[1](https://arxiv.org/html/2607.26618#Thmlemma1)yields the interpretable bound

‖h¯m−gκm,m‖≤2​Bg​\(ϵmloc\+ϵmalign\+ϵmleak\)\+δpat\.\\\|\\bar\{h\}\_\{m\}\-g\_\{\\kappa\_\{m\},m\}\\\|\\leq 2B\_\{g\}\\bigl\(\\epsilon\_\{m\}^\{\\mathrm\{loc\}\}\+\\epsilon\_\{m\}^\{\\mathrm\{align\}\}\+\\epsilon\_\{m\}^\{\\mathrm\{leak\}\}\\bigr\)\+\\delta\_\{\\mathrm\{pat\}\}\.\(22\)
For a client\-level expert unit, define the analogous off\-target contribution weightαmclient\\alpha\_\{m\}^\{\\mathrm\{client\}\}from its routing\-weighted update decomposition\.

###### Corollary 1\(Direct granularity comparison\)\.

The client\-level direction obeys

‖h¯mclient−gκm,m‖≤2​Bg​αmclient\+δpat\.\\\|\\bar\{h\}\_\{m\}^\{\\mathrm\{client\}\}\-g\_\{\\kappa\_\{m\},m\}\\\|\\leq 2B\_\{g\}\\alpha\_\{m\}^\{\\mathrm\{client\}\}\+\\delta\_\{\\mathrm\{pat\}\}\.\(23\)Consequently, prototype conditioning gives a strictly tighter worst\-case target\-direction bound wheneverχm<αmclient\\chi\_\{m\}<\\alpha\_\{m\}^\{\\mathrm\{client\}\}\.

###### Assumption 2\(Smooth target objective and routed stochastic direction\)\.

For expertmm,Fκm,mF\_\{\\kappa\_\{m\},m\}isLEL\_\{E\}\-smooth and lower bounded byFκm,m⋆F\_\{\\kappa\_\{m\},m\}^\{\\star\}\. At active expert\-update stepuu, letℱu\\mathcal\{F\}\_\{u\}containθmu\\theta\_\{m\}^\{u\}, the total routed weightρmu\>0\\rho\_\{m\}^\{u\}\>0, and the routing history before stochastic gradient noise is drawn\. The raw stochastic direction satisfies

𝔼​\[dmu∣ℱu\]\\displaystyle\\mathbb\{E\}\[d\_\{m\}^\{u\}\\mid\\mathcal\{F\}\_\{u\}\]=ρmu​\(gκm,m​\(θmu\)\+bmu\),\\displaystyle=\\rho\_\{m\}^\{u\}\\bigl\(g\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{u\}\)\+b\_\{m\}^\{u\}\\bigr\),\(24\)𝔼\[∥dmu−𝔼\[dmu∣ℱu\]∥2∣ℱu\]\\displaystyle\\mathbb\{E\}\[\\\|d\_\{m\}^\{u\}\-\\mathbb\{E\}\[d\_\{m\}^\{u\}\\mid\\mathcal\{F\}\_\{u\}\]\\\|^\{2\}\\mid\\mathcal\{F\}\_\{u\}\]≤\(ρmu\)2​σE2,\\displaystyle\\leq\(\\rho\_\{m\}^\{u\}\)^\{2\}\\sigma\_\{E\}^\{2\},where Proposition[1](https://arxiv.org/html/2607.26618#Thmproposition1)gives‖bmu‖≤2​Bg​χm​\(θmu\)\+δpat\\\|b\_\{m\}^\{u\}\\\|\\leq 2B\_\{g\}\\chi\_\{m\}\(\\theta\_\{m\}^\{u\}\)\+\\delta\_\{\\mathrm\{pat\}\}\. Inactive steps have effective step size zero\.

###### Theorem 3\(Per\-expert convergence in stationarity under routed contamination\)\.

Letθmu\+1=θmu−ηE​dmu\\theta\_\{m\}^\{u\+1\}=\\theta\_\{m\}^\{u\}\-\\eta\_\{E\}d\_\{m\}^\{u\},amu=ηE​ρmu≤1/LEa\_\{m\}^\{u\}=\\eta\_\{E\}\\rho\_\{m\}^\{u\}\\leq 1/L\_\{E\}, andAm,U=∑u<U𝔼​\[amu\]\>0A\_\{m,U\}=\\sum\_\{u<U\}\\mathbb\{E\}\[a\_\{m\}^\{u\}\]\>0\. DefineΔE,m0=Fκm,m​\(θm0\)−Fκm,m⋆\\Delta\_\{E,m\}^\{0\}=F\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{0\}\)\-F\_\{\\kappa\_\{m\},m\}^\{\\star\}\. Writegmu=gκm,m​\(θmu\)g\_\{m\}^\{u\}=g\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{u\}\)andεmu=2​Bg​χm​\(θmu\)\+δpat\\varepsilon\_\{m\}^\{u\}=2B\_\{g\}\\chi\_\{m\}\(\\theta\_\{m\}^\{u\}\)\+\\delta\_\{\\mathrm\{pat\}\}\. Then

1Am,U​∑u<U𝔼​\[amu​‖gmu‖2\]\\displaystyle\\frac\{1\}\{A\_\{m,U\}\}\\sum\_\{u<U\}\\mathbb\{E\}\\\!\\left\[a\_\{m\}^\{u\}\\\|g\_\{m\}^\{u\}\\\|^\{2\}\\right\]≤2​ΔE,m0Am,U\\displaystyle\\leq\\frac\{2\\Delta\_\{E,m\}^\{0\}\}\{A\_\{m,U\}\}\(25\)\+1Am,U​∑u<U𝔼​\[amu​\(εmu\)2\]\\displaystyle\\quad\+\\frac\{1\}\{A\_\{m,U\}\}\\sum\_\{u<U\}\\mathbb\{E\}\\\!\\left\[a\_\{m\}^\{u\}\(\\varepsilon\_\{m\}^\{u\}\)^\{2\}\\right\]\+LE​σE2Am,U​∑u<U𝔼​\[\(amu\)2\]\.\\displaystyle\\quad\+\\frac\{L\_\{E\}\\sigma\_\{E\}^\{2\}\}\{A\_\{m,U\}\}\\sum\_\{u<U\}\\mathbb\{E\}\[\(a\_\{m\}^\{u\}\)^\{2\}\]\.

###### Proof\.

Letgu=gκm,m​\(θmu\)g^\{u\}=g\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{u\}\),μu=gu\+bmu\\mu^\{u\}=g^\{u\}\+b\_\{m\}^\{u\}, andau=amua\_\{u\}=a\_\{m\}^\{u\}\. Smoothness and the conditional variance bound give

𝔼​\[Fκm,m​\(θmu\+1\)∣ℱu\]\\displaystyle\\mathbb\{E\}\[F\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{u\+1\}\)\\mid\\mathcal\{F\}\_\{u\}\]≤Fκm,m​\(θmu\)−au​⟨gu,μu⟩\\displaystyle\\leq F\_\{\\kappa\_\{m\},m\}\(\\theta\_\{m\}^\{u\}\)\-a\_\{u\}\\langle g^\{u\},\\mu^\{u\}\\rangle\(26\)\+LE​au22​\(‖μu‖2\+σE2\)\.\\displaystyle\\quad\+\\frac\{L\_\{E\}a\_\{u\}^\{2\}\}\{2\}\(\\\|\\mu^\{u\}\\\|^\{2\}\+\\sigma\_\{E\}^\{2\}\)\.Using−⟨gu,μu⟩=−12​‖gu‖2−12​‖μu‖2\+12​‖bmu‖2\-\\langle g^\{u\},\\mu^\{u\}\\rangle=\-\\frac\{1\}\{2\}\\\|g^\{u\}\\\|^\{2\}\-\\frac\{1\}\{2\}\\\|\\mu^\{u\}\\\|^\{2\}\+\\frac\{1\}\{2\}\\\|b\_\{m\}^\{u\}\\\|^\{2\}andau≤1/LEa\_\{u\}\\leq 1/L\_\{E\}yields one\-step descent with residualau​‖bmu‖2/2\+LE​au2​σE2/2a\_\{u\}\\\|b\_\{m\}^\{u\}\\\|^\{2\}/2\+L\_\{E\}a\_\{u\}^\{2\}\\sigma\_\{E\}^\{2\}/2\. Taking total expectation, telescoping, and applying Proposition[1](https://arxiv.org/html/2607.26618#Thmproposition1)proves Eq\. \([25](https://arxiv.org/html/2607.26618#A3.E25)\)\. ∎

###### Corollary 2\(Matched\-effective\-weight convergence comparison\)\.

Consider prototype\- and client\-level expert updates with the same initialization, effective weights\{amu\}u<U\\\{a\_\{m\}^\{u\}\\\}\_\{u<U\}, smoothness, and noise terms\. Let their nonnegative directional\-error bounds beεm,protou=2​Bg​χmu\+δpat\\varepsilon\_\{m,\\mathrm\{proto\}\}^\{u\}=2B\_\{g\}\\chi\_\{m\}^\{u\}\+\\delta\_\{\\mathrm\{pat\}\}andεm,clientu=2​Bg​αmclient,u\+δpat\\varepsilon\_\{m,\\mathrm\{client\}\}^\{u\}=2B\_\{g\}\\alpha\_\{m\}^\{\\mathrm\{client\},u\}\+\\delta\_\{\\mathrm\{pat\}\}\. Ifχmu≤αmclient,u\\chi\_\{m\}^\{u\}\\leq\\alpha\_\{m\}^\{\\mathrm\{client\},u\}at every active step, the prototype\-conditioned right\-hand side of Eq\. \([25](https://arxiv.org/html/2607.26618#A3.E25)\) is no larger, and is strictly smaller whenever the inequality is strict with positive probability at an active step\.

###### Proof\.

Under the matched quantities, only the directional\-error term differs; the claim follows from\(εm,protou\)2≤\(εm,clientu\)2\(\\varepsilon\_\{m,\\mathrm\{proto\}\}^\{u\}\)^\{2\}\\leq\(\\varepsilon\_\{m,\\mathrm\{client\}\}^\{u\}\)^\{2\}\. ∎

Equation \([25](https://arxiv.org/html/2607.26618#A3.E25)\) is a fixed\-context expert convergence guarantee\. It separates two failure modes of the actual raw update:χm\\chi\_\{m\}controls directional contamination, whileAm,UA\_\{m,U\}records the effective optimization budget received by expertmm\. If an expert receives negligible routed weight, the first term remains large even when its normalized direction is pure\.

### Persistent and Reset\-and\-Average Router Optimization

Freeze the experts and writeφ\\varphifor the router parameters\. For clientii, letπi,c=\|ℬi,c\|/\|𝒟i\|\\pi\_\{i,c\}=\|\\mathcal\{B\}\_\{i,c\}\|/\|\\mathcal\{D\}\_\{i\}\|be bucketcc’s sample fraction and let

Ri​\(φ\)\\displaystyle R\_\{i\}\(\\varphi\)=∑cπi,c​Ri,c​\(φ\),\\displaystyle=\\sum\_\{c\}\\pi\_\{i,c\}R\_\{i,c\}\(\\varphi\),\(27\)gi​\(φ\)\\displaystyle g\_\{i\}\(\\varphi\)=∇Ri​\(φ\),\\displaystyle=\\nabla R\_\{i\}\(\\varphi\),gi,c​\(φ\)\\displaystyle g\_\{i,c\}\(\\varphi\)=∇Ri,c​\(φ\)\.\\displaystyle=\\nabla R\_\{i,c\}\(\\varphi\)\.
The persistent router performs SGD on one shared state\. Randomized sample\-count\-proportional sampling drawscsc\_\{s\}according toPr⁡\(cs=c\)=πi,c\\Pr\(c\_\{s\}=c\)=\\pi\_\{i,c\}and is conditionally unbiased\. For the implemented deterministic sample\-count\-balanced schedule, define its conditional order bias

qs​\(φs\)=𝔼​\[g^i,cs​\(φs\)∣φs\]−gi​\(φs\)\.q\_\{s\}\(\\varphi\_\{s\}\)=\\mathbb\{E\}\[\\widehat\{g\}\_\{i,c\_\{s\}\}\(\\varphi\_\{s\}\)\\mid\\varphi\_\{s\}\]\-g\_\{i\}\(\\varphi\_\{s\}\)\.\(28\)Thusqs=0q\_\{s\}=0for the randomized reference, while deterministic interleaving is covered through an explicit residual\. The reset alternative instead maintains bucket\-local states, evaluates subsequent gradients at those dispersed states, and averages them after the local block\. The gradient\-dominance condition below is a standard route to linear SGD convergence, while the resulting dispersion term is analogous to client drift in local federated optimization; the main text provides the relevant citations\.

###### Assumption 3\(Router smoothness, gradient dominance, and noise\)\.

EachRi,cR\_\{i,c\}isLRL\_\{R\}\-smooth, andRiR\_\{i\}satisfies theμ\\mu\-Polyak–Łojasiewicz condition

‖∇Ri​\(φ\)‖2≥2​μ​\(Ri​\(φ\)−Ri⋆\)\.\\\|\\nabla R\_\{i\}\(\\varphi\)\\\|^\{2\}\\geq 2\\mu\(R\_\{i\}\(\\varphi\)\-R\_\{i\}^\{\\star\}\)\.\(29\)Persistent gradients have conditional meangi​\(φs\)\+qs​\(φs\)g\_\{i\}\(\\varphi\_\{s\}\)\+q\_\{s\}\(\\varphi\_\{s\}\), and their centered conditional variance is at mostσR2\\sigma\_\{R\}^\{2\}\. For reset updates, the weighted aggregate of bucket\-gradient noise has conditional second moment at mostσR2\\sigma\_\{R\}^\{2\}\.

###### Theorem 4\(Persistent shared\-state router convergence\)\.

Forφs\+1=φs−ηR​g^i,cs​\(φs\)\\varphi\_\{s\+1\}=\\varphi\_\{s\}\-\\eta\_\{R\}\\widehat\{g\}\_\{i,c\_\{s\}\}\(\\varphi\_\{s\}\)with0<ηR≤1/LR0<\\eta\_\{R\}\\leq 1/L\_\{R\},

𝔼​\[Ri​\(φS\)−Ri⋆\]\\displaystyle\\mathbb\{E\}\[R\_\{i\}\(\\varphi\_\{S\}\)\-R\_\{i\}^\{\\star\}\]≤ℰi,S\+𝒪i,S,\\displaystyle\\leq\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{O\}\_\{i,S\},\(30\)ℰi,S\\displaystyle\\mathcal\{E\}\_\{i,S\}:=\(1−μ​ηR\)S​\(Ri​\(φ0\)−Ri⋆\)\+LR​ηR​σR22​μ,\\displaystyle=\(1\-\\mu\\eta\_\{R\}\)^\{S\}\(R\_\{i\}\(\\varphi\_\{0\}\)\-R\_\{i\}^\{\\star\}\)\+\\frac\{L\_\{R\}\\eta\_\{R\}\\sigma\_\{R\}^\{2\}\}\{2\\mu\},𝒪i,S\\displaystyle\\mathcal\{O\}\_\{i,S\}:=ηR2​∑s=0S−1\(1−μ​ηR\)S−1−s​𝔼​‖qs​\(φs\)‖2\.\\displaystyle=\\frac\{\\eta\_\{R\}\}\{2\}\\sum\_\{s=0\}^\{S\-1\}\(1\-\\mu\\eta\_\{R\}\)^\{S\-1\-s\}\\mathbb\{E\}\\\|q\_\{s\}\(\\varphi\_\{s\}\)\\\|^\{2\}\.

###### Proof\.

Letμs=gi​\(φs\)\+qs​\(φs\)\\mu\_\{s\}=g\_\{i\}\(\\varphi\_\{s\}\)\+q\_\{s\}\(\\varphi\_\{s\}\)\. Smoothness, the centered variance bound, and−⟨gi,μs⟩=−12​‖gi‖2−12​‖μs‖2\+12​‖qs‖2\-\\langle g\_\{i\},\\mu\_\{s\}\\rangle=\-\\frac\{1\}\{2\}\\\|g\_\{i\}\\\|^\{2\}\-\\frac\{1\}\{2\}\\\|\\mu\_\{s\}\\\|^\{2\}\+\\frac\{1\}\{2\}\\\|q\_\{s\}\\\|^\{2\}imply

𝔼​\[Ri​\(φs\+1\)∣φs\]\\displaystyle\\mathbb\{E\}\[R\_\{i\}\(\\varphi\_\{s\+1\}\)\\mid\\varphi\_\{s\}\]≤Ri​\(φs\)−ηR2​‖gi​\(φs\)‖2\+ηR2​‖qs​\(φs\)‖2\\displaystyle\\leq R\_\{i\}\(\\varphi\_\{s\}\)\-\\frac\{\\eta\_\{R\}\}\{2\}\\\|g\_\{i\}\(\\varphi\_\{s\}\)\\\|^\{2\}\+\\frac\{\\eta\_\{R\}\}\{2\}\\\|q\_\{s\}\(\\varphi\_\{s\}\)\\\|^\{2\}\(31\)\+LR​ηR22​σR2\.\\displaystyle\\quad\+\\frac\{L\_\{R\}\\eta\_\{R\}^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\.Here the nonpositive coefficient of‖μs‖2\\\|\\mu\_\{s\}\\\|^\{2\}was dropped usingηR≤1/LR\\eta\_\{R\}\\leq 1/L\_\{R\}\. Applying Eq\. \([29](https://arxiv.org/html/2607.26618#A3.E29)\) and unrolling proves Eq\. \([30](https://arxiv.org/html/2607.26618#A3.E30)\)\. ∎

For the clean matched\-weight reset abstraction, setqs=0q\_\{s\}=0, initialize every bucket state at the same routerφc,0=φ¯0\\varphi\_\{c,0\}=\\bar\{\\varphi\}\_\{0\}, and take local SGD stepsφc,s\+1=φc,s−ηR​g^i,c​\(φc,s\)\\varphi\_\{c,s\+1\}=\\varphi\_\{c,s\}\-\\eta\_\{R\}\\widehat\{g\}\_\{i,c\}\(\\varphi\_\{c,s\}\)\. This equal\-depth construction isolates state fragmentation:ssindexes depth along each bucket\-local path, so the comparison matches depth rather than total gradient evaluations\. Define the virtual weighted mean, local\-model dispersion, and consensus\-gradient error

φ¯s\\displaystyle\\bar\{\\varphi\}\_\{s\}=∑cπi,c​φc,s,\\displaystyle=\\sum\_\{c\}\\pi\_\{i,c\}\\varphi\_\{c,s\},\(32\)Ds\\displaystyle D\_\{s\}=∑cπi,c​‖φc,s−φ¯s‖2,\\displaystyle=\\sum\_\{c\}\\pi\_\{i,c\}\\\|\\varphi\_\{c,s\}\-\\bar\{\\varphi\}\_\{s\}\\\|^\{2\},es\\displaystyle e\_\{s\}=∑cπi,c​\(gi,c​\(φc,s\)−gi,c​\(φ¯s\)\)\.\\displaystyle=\\sum\_\{c\}\\pi\_\{i,c\}\\bigl\(g\_\{i,c\}\(\\varphi\_\{c,s\}\)\-g\_\{i,c\}\(\\bar\{\\varphi\}\_\{s\}\)\\bigr\)\.The mean reset trajectory therefore follows a biased client\-objective direction:

𝔼\[∑cπi,cg^i,c\(φc,s\)\|\{φc,s\}c\]=gi\(φ¯s\)\+es\.\\mathbb\{E\}\\\!\\left\[\\sum\_\{c\}\\pi\_\{i,c\}\\widehat\{g\}\_\{i,c\}\(\\varphi\_\{c,s\}\)\\,\\middle\|\\,\\\{\\varphi\_\{c,s\}\\\}\_\{c\}\\right\]=g\_\{i\}\(\\bar\{\\varphi\}\_\{s\}\)\+e\_\{s\}\.\(33\)
###### Theorem 5\(Reset\-and\-average router convergence with consensus error\)\.

Under Assumption[3](https://arxiv.org/html/2607.26618#Thmassumption3)and0<ηR≤1/LR0<\\eta\_\{R\}\\leq 1/L\_\{R\},

𝔼​\[Ri​\(φ¯S\)−Ri⋆\]\\displaystyle\\mathbb\{E\}\[R\_\{i\}\(\\bar\{\\varphi\}\_\{S\}\)\-R\_\{i\}^\{\\star\}\]≤\(1−μ​ηR\)S​\(Ri​\(φ¯0\)−Ri⋆\)\+LR​ηR​σR22​μ\\displaystyle\\leq\(1\-\\mu\\eta\_\{R\}\)^\{S\}\(R\_\{i\}\(\\bar\{\\varphi\}\_\{0\}\)\-R\_\{i\}^\{\\star\}\)\+\\frac\{L\_\{R\}\\eta\_\{R\}\\sigma\_\{R\}^\{2\}\}\{2\\mu\}\(34\)\+ηR2​∑s=0S−1\(1−μ​ηR\)S−1−s​𝔼​‖es‖2,\\displaystyle\\quad\+\\frac\{\\eta\_\{R\}\}\{2\}\\sum\_\{s=0\}^\{S\-1\}\(1\-\\mu\\eta\_\{R\}\)^\{S\-1\-s\}\\mathbb\{E\}\\\|e\_\{s\}\\\|^\{2\},where

‖es‖2≤LR2​Ds\.\\\|e\_\{s\}\\\|^\{2\}\\leq L\_\{R\}^\{2\}D\_\{s\}\.\(35\)Thus reset\-and\-average can incur an optimization term generated by bucket\-local model dispersion; the persistent shared\-state bound in Eq\. \([30](https://arxiv.org/html/2607.26618#A3.E30)\) has no such consensus term\.

###### Proof\.

The weighted mean update isφ¯s\+1=φ¯s−ηR​∑cπi,c​g^i,c​\(φc,s\)\\bar\{\\varphi\}\_\{s\+1\}=\\bar\{\\varphi\}\_\{s\}\-\\eta\_\{R\}\\sum\_\{c\}\\pi\_\{i,c\}\\widehat\{g\}\_\{i,c\}\(\\varphi\_\{c,s\}\)\. Equation \([33](https://arxiv.org/html/2607.26618#A3.E33)\) and the same biased\-descent identity used in Theorem[4](https://arxiv.org/html/2607.26618#Thmtheorem4)give

𝔼​\[Ri​\(φ¯s\+1\)−Ri⋆\]\\displaystyle\\mathbb\{E\}\[R\_\{i\}\(\\bar\{\\varphi\}\_\{s\+1\}\)\-R\_\{i\}^\{\\star\}\]≤\(1−μ​ηR\)​𝔼​\[Ri​\(φ¯s\)−Ri⋆\]\\displaystyle\\leq\(1\-\\mu\\eta\_\{R\}\)\\mathbb\{E\}\[R\_\{i\}\(\\bar\{\\varphi\}\_\{s\}\)\-R\_\{i\}^\{\\star\}\]\(36\)\+ηR2​𝔼​‖es‖2\+LR​ηR22​σR2\.\\displaystyle\\quad\+\\frac\{\\eta\_\{R\}\}\{2\}\\mathbb\{E\}\\\|e\_\{s\}\\\|^\{2\}\+\\frac\{L\_\{R\}\\eta\_\{R\}^\{2\}\}\{2\}\\sigma\_\{R\}^\{2\}\.Unrolling proves Eq\. \([34](https://arxiv.org/html/2607.26618#A3.E34)\)\. For Eq\. \([35](https://arxiv.org/html/2607.26618#A3.E35)\), Jensen’s inequality andLRL\_\{R\}\-smoothness give

‖es‖2\\displaystyle\\\|e\_\{s\}\\\|^\{2\}≤∑cπi,c​‖gi,c​\(φc,s\)−gi,c​\(φ¯s\)‖2\\displaystyle\\leq\\sum\_\{c\}\\pi\_\{i,c\}\\\|g\_\{i,c\}\(\\varphi\_\{c,s\}\)\-g\_\{i,c\}\(\\bar\{\\varphi\}\_\{s\}\)\\\|^\{2\}\(37\)≤LR2​∑cπi,c​‖φc,s−φ¯s‖2=LR2​Ds\.\\displaystyle\\leq L\_\{R\}^\{2\}\\sum\_\{c\}\\pi\_\{i,c\}\\\|\\varphi\_\{c,s\}\-\\bar\{\\varphi\}\_\{s\}\\\|^\{2\}=L\_\{R\}^\{2\}D\_\{s\}\.∎

For the randomized referenceqs=0q\_\{s\}=0, matched initialization, step size, depth, and noise, the certificate gap is the nonnegative consensus sum in Eq\. \([34](https://arxiv.org/html/2607.26618#A3.E34)\), and is strict when that sum is positive\. For the deterministic persistent schedule, the comparison is between𝒪i,S\\mathcal\{O\}\_\{i,S\}and the reset residual; their magnitudes determine the bound ordering\.

The dispersion term is directly tied to heterogeneity across bucket router objectives\. At a common initializationφ\\varphi, define

ζi,within2​\(φ\)=∑cπi,c​‖gi,c​\(φ\)−gi​\(φ\)‖2\.\\zeta\_\{i,\\mathrm\{within\}\}^\{2\}\(\\varphi\)=\\sum\_\{c\}\\pi\_\{i,c\}\\\|g\_\{i,c\}\(\\varphi\)\-g\_\{i\}\(\\varphi\)\\\|^\{2\}\.\(38\)
###### Corollary 3\(First\-step reset dispersion\)\.

For deterministic full gradients and one reset step from a common initialization,

D1=ηR2​ζi,within2​\(φ\)\.D\_\{1\}=\\eta\_\{R\}^\{2\}\\zeta\_\{i,\\mathrm\{within\}\}^\{2\}\(\\varphi\)\.\(39\)Hence heterogeneous bucket gradients create the consensus\-error source that enters Eq\. \([34](https://arxiv.org/html/2607.26618#A3.E34)\) from subsequent local steps onward\.

###### Proof\.

After one step,φc,1=φ−ηR​gi,c​\(φ\)\\varphi\_\{c,1\}=\\varphi\-\\eta\_\{R\}g\_\{i,c\}\(\\varphi\)andφ¯1=φ−ηR​gi​\(φ\)\\bar\{\\varphi\}\_\{1\}=\\varphi\-\\eta\_\{R\}g\_\{i\}\(\\varphi\)\. Substitution intoD1D\_\{1\}proves the result\. ∎

Theorem[5](https://arxiv.org/html/2607.26618#Thmtheorem5)isolates state fragmentation under matched effective pattern weights\. If reset endpoints trained forHcH\_\{c\}steps are aggregated with coefficientsaca\_\{c\}, their first\-order normalized direction instead uses

π~c=ac​Hc∑jaj​Hj,bwt​\(φ\)=∑c\(π~c−πi,c\)​gi,c​\(φ\)\.\\widetilde\{\\pi\}\_\{c\}=\\frac\{a\_\{c\}H\_\{c\}\}\{\\sum\_\{j\}a\_\{j\}H\_\{j\}\},\\qquad b\_\{\\mathrm\{wt\}\}\(\\varphi\)=\\sum\_\{c\}\(\\widetilde\{\\pi\}\_\{c\}\-\\pi\_\{i,c\}\)g\_\{i,c\}\(\\varphi\)\.\(40\)This bias precedes dispersion; a multi\-step error also contains aπ~\\widetilde\{\\pi\}\-weighted consensus term\. If bothHc∝πi,cH\_\{c\}\\propto\\pi\_\{i,c\}andac∝πi,ca\_\{c\}\\propto\\pi\_\{i,c\}, thenπ~c∝πi,c2\\widetilde\{\\pi\}\_\{c\}\\propto\\pi\_\{i,c\}^\{2\}; step\-normalized endpoint deltas or any rule enforcingπ~c=πi,c\\widetilde\{\\pi\}\_\{c\}=\\pi\_\{i,c\}restore matched weighting\. More explicitly, if‖gi,c​\(φ\)‖≤GR\\\|g\_\{i,c\}\(\\varphi\)\\\|\\leq G\_\{R\}, then

‖bwt​\(φ\)‖≤GR​‖π~−πi‖1\.\\\|b\_\{\\mathrm\{wt\}\}\(\\varphi\)\\\|\\leq G\_\{R\}\\\|\\widetilde\{\\pi\}\-\\pi\_\{i\}\\\|\_\{1\}\.\(41\)For an equal\-depth reset using fixed weightsπ~\\widetilde\{\\pi\}, define

φ¯sπ~\\displaystyle\\bar\{\\varphi\}\_\{s\}^\{\\widetilde\{\\pi\}\}=∑cπ~c​φc,s,\\displaystyle=\\sum\_\{c\}\\widetilde\{\\pi\}\_\{c\}\\varphi\_\{c,s\},\(42\)esπ~\\displaystyle e\_\{s\}^\{\\widetilde\{\\pi\}\}=∑cπ~c​\(gi,c​\(φc,s\)−gi,c​\(φ¯sπ~\)\)\.\\displaystyle=\\sum\_\{c\}\\widetilde\{\\pi\}\_\{c\}\\bigl\(g\_\{i,c\}\(\\varphi\_\{c,s\}\)\-g\_\{i,c\}\(\\bar\{\\varphi\}\_\{s\}^\{\\widetilde\{\\pi\}\}\)\\bigr\)\.Its expected aggregate direction isgi​\(φ¯sπ~\)\+bwt​\(φ¯sπ~\)\+esπ~g\_\{i\}\(\\bar\{\\varphi\}\_\{s\}^\{\\widetilde\{\\pi\}\}\)\+b\_\{\\mathrm\{wt\}\}\(\\bar\{\\varphi\}\_\{s\}^\{\\widetilde\{\\pi\}\}\)\+e\_\{s\}^\{\\widetilde\{\\pi\}\}\. Thus the biased\-descent proof of Theorem[5](https://arxiv.org/html/2607.26618#Thmtheorem5)replacesese\_\{s\}bybwt\+esπ~b\_\{\\mathrm\{wt\}\}\+e\_\{s\}^\{\\widetilde\{\\pi\}\}, so weight mismatch enters the same geometrically weighted optimization residual before the gap\-calibration transfer\. In the evaluated*w/o client\-router*ablation,HcH\_\{c\}is allocated approximately by bucket size \(with minimum\-step and rounding corrections\) and endpoint coefficients are again proportional to bucket sample count\. Thus the aggregation\-granularity ablation in Table[2](https://arxiv.org/html/2607.26618#Sx4.T2)evaluates the implemented reset\-and\-reweight procedure, whose residual includes both weight mismatch and consensus error\.

### Router Optimization and Sparse\-Inference Risk

Letz​\(x\)∈\{1,…,K\}z\(x\)\\in\\\{1,\\ldots,K\\\}be the latent pattern ofxx, letm†​\(x\)=ψ​\(z​\(x\)\)m^\{\\dagger\}\(x\)=\\psi\(z\(x\)\)be its oracle expert, and letℓm​\(x,y\)\\ell\_\{m\}\(x,y\)be the loss using only expertmm\. DefineRi,oracle=𝔼\(x,y\)∼𝒟i​\[ℓm†​\(x\)​\(x,y\)\]R\_\{i,\\mathrm\{oracle\}\}=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{i\}\}\[\\ell\_\{m^\{\\dagger\}\(x\)\}\(x,y\)\], letrm​\(x;φ\)r\_\{m\}\(x;\\varphi\)be the router probability, and define the oracle\-probability deficit

Si​\(φ\)=𝔼x∼𝒟i​\[1−rm†​\(x\)​\(x;φ\)\]\.S\_\{i\}\(\\varphi\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{i\}\}\[1\-r\_\{m^\{\\dagger\}\(x\)\}\(x;\\varphi\)\]\.\(43\)
###### Assumption 4\(Gap\-calibrated task loss\)\.

The fixed experts have an identifiable oracle assignment, and the soft\-routed task objective is calibrated to that assignment:

cgap​Si​\(φ\)≤Ri​\(φ\)−Ri,oracle\+βmix,cgap\>0,βmix≥0\.\\begin\{gathered\}c\_\{\\mathrm\{gap\}\}S\_\{i\}\(\\varphi\)\\leq R\_\{i\}\(\\varphi\)\-R\_\{i,\\mathrm\{oracle\}\}\+\\beta\_\{\\mathrm\{mix\}\},\\\\ c\_\{\\mathrm\{gap\}\}\>0,\\qquad\\beta\_\{\\mathrm\{mix\}\}\\geq 0\.\\end\{gathered\}\(44\)Define the nonnegative router\-class approximation gapAi:=\[Ri⋆−Ri,oracle\]\+A\_\{i\}:=\[R\_\{i\}^\{\\star\}\-R\_\{i,\\mathrm\{oracle\}\}\]\_\{\+\}\. For a linear mixture of fixed expert losses with pointwise non\-oracle loss gap at leastΔ\\Delta, Eq\. \([44](https://arxiv.org/html/2607.26618#A3.E44)\) holds withcgap=Δc\_\{\\mathrm\{gap\}\}=\\Deltaandβmix=0\\beta\_\{\\mathrm\{mix\}\}=0\.

Letsm​\(x;φ\)s\_\{m\}\(x;\\varphi\)denote router logits and

marφ⁡\(x\)=sm†​\(x\)​\(x;φ\)−maxj≠m†​\(x\)⁡sj​\(x;φ\)\.\\operatorname\{mar\}\_\{\\varphi\}\(x\)=s\_\{m^\{\\dagger\}\(x\)\}\(x;\\varphi\)\-\\max\_\{j\\neq m^\{\\dagger\}\(x\)\}s\_\{j\}\(x;\\varphi\)\.\(45\)DefineRτ,i​\(φ\)=Prx∼𝒟i⁡\[marφ⁡\(x\)≤τ\]R\_\{\\tau,i\}\(\\varphi\)=\\Pr\_\{x\\sim\\mathcal\{D\}\_\{i\}\}\[\\operatorname\{mar\}\_\{\\varphi\}\(x\)\\leq\\tau\]\.

###### Proposition 2\(Gap\-calibrated margin risk\)\.

Under Assumption[4](https://arxiv.org/html/2607.26618#Thmassumption4), for everyτ∈ℝ\\tau\\in\\mathbb\{R\},

Rτ,i​\(φ\)≤1\+eτcgap​\(Ri​\(φ\)−Ri⋆\+Ai\+βmix\)\.R\_\{\\tau,i\}\(\\varphi\)\\leq\\frac\{1\+e^\{\\tau\}\}\{c\_\{\\mathrm\{gap\}\}\}\\bigl\(R\_\{i\}\(\\varphi\)\-R\_\{i\}^\{\\star\}\+A\_\{i\}\+\\beta\_\{\\mathrm\{mix\}\}\\bigr\)\.\(46\)

###### Proof\.

Ifmarφ⁡\(x\)≤τ\\operatorname\{mar\}\_\{\\varphi\}\(x\)\\leq\\tau, then one competing logit is at leastsm†​\(x\)−τs\_\{m^\{\\dagger\}\(x\)\}\-\\tau, sorm†​\(x\)​\(x;φ\)≤\(1\+e−τ\)−1r\_\{m^\{\\dagger\}\(x\)\}\(x;\\varphi\)\\leq\(1\+e^\{\-\\tau\}\)^\{\-1\}and1−rm†​\(x\)​\(x;φ\)≥\(1\+eτ\)−11\-r\_\{m^\{\\dagger\}\(x\)\}\(x;\\varphi\)\\geq\(1\+e^\{\\tau\}\)^\{\-1\}\. Therefore𝟏​\{marφ⁡\(x\)≤τ\}≤\(1\+eτ\)​\(1−rm†​\(x\)​\(x;φ\)\)\\mathbf\{1\}\\\{\\operatorname\{mar\}\_\{\\varphi\}\(x\)\\leq\\tau\\\}\\leq\(1\+e^\{\\tau\}\)\(1\-r\_\{m^\{\\dagger\}\(x\)\}\(x;\\varphi\)\)\. Taking expectation, applying Eq\. \([44](https://arxiv.org/html/2607.26618#A3.E44)\), and usingRi​\(φ\)−Ri,oracle≤Ri​\(φ\)−Ri⋆\+AiR\_\{i\}\(\\varphi\)\-R\_\{i,\\mathrm\{oracle\}\}\\leq R\_\{i\}\(\\varphi\)\-R\_\{i\}^\{\\star\}\+A\_\{i\}proves the result\. ∎

Letℰi,S\\mathcal\{E\}\_\{i,S\}and𝒪i,S\\mathcal\{O\}\_\{i,S\}be defined in Eq\. \([30](https://arxiv.org/html/2607.26618#A3.E30)\), and define𝒞i,S=ηR2​∑s=0S−1\(1−μ​ηR\)S−1−s​𝔼​‖es‖2\\mathcal\{C\}\_\{i,S\}=\\frac\{\\eta\_\{R\}\}\{2\}\\sum\_\{s=0\}^\{S\-1\}\(1\-\\mu\\eta\_\{R\}\)^\{S\-1\-s\}\\mathbb\{E\}\\\|e\_\{s\}\\\|^\{2\}\.

###### Corollary 4\(Persistent and reset oracle\-routing risk\)\.

Under Assumptions[3](https://arxiv.org/html/2607.26618#Thmassumption3)–[4](https://arxiv.org/html/2607.26618#Thmassumption4), a common initialization, and the matched\-weight reset construction of Theorem[5](https://arxiv.org/html/2607.26618#Thmtheorem5), for everyτ∈ℝ\\tau\\in\\mathbb\{R\},

𝔼​Rτ,i​\(φS\)\\displaystyle\\mathbb\{E\}R\_\{\\tau,i\}\(\\varphi\_\{S\}\)≤1\+eτcgap​\(ℰi,S\+𝒪i,S\+Ai\+βmix\),\\displaystyle\\leq\\frac\{1\+e^\{\\tau\}\}\{c\_\{\\mathrm\{gap\}\}\}\(\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{O\}\_\{i,S\}\+A\_\{i\}\+\\beta\_\{\\mathrm\{mix\}\}\),\(47\)𝔼​Rτ,i​\(φ¯S\)\\displaystyle\\mathbb\{E\}R\_\{\\tau,i\}\(\\bar\{\\varphi\}\_\{S\}\)≤1\+eτcgap​\(ℰi,S\+𝒞i,S\+Ai\+βmix\)\.\\displaystyle\\leq\\frac\{1\+e^\{\\tau\}\}\{c\_\{\\mathrm\{gap\}\}\}\(\\mathcal\{E\}\_\{i,S\}\+\\mathcal\{C\}\_\{i,S\}\+A\_\{i\}\+\\beta\_\{\\mathrm\{mix\}\}\)\.

This follows from Proposition[2](https://arxiv.org/html/2607.26618#Thmproposition2)and Theorems[4](https://arxiv.org/html/2607.26618#Thmtheorem4)–[5](https://arxiv.org/html/2607.26618#Thmtheorem5)\. Hence persistent shared\-state SGD has low oracle\-routing surrogate risk whenℰi,S\\mathcal\{E\}\_\{i,S\},𝒪i,S\\mathcal\{O\}\_\{i,S\},AiA\_\{i\}, andβmix\\beta\_\{\\mathrm\{mix\}\}are small\. In the randomized reference𝒪i,S=0\\mathcal\{O\}\_\{i,S\}=0, whereas matched reset additionally propagates𝒞i,S\\mathcal\{C\}\_\{i,S\}\. The weight\-mismatched extension above replaces this clean consensus residual by one containingbwt\+esπ~b\_\{\\mathrm\{wt\}\}\+e\_\{s\}^\{\\widetilde\{\\pi\}\}\.

Finally, letm^​\(x\)=arg⁡maxm⁡sm​\(x;φ\)\\widehat\{m\}\(x\)=\\arg\\max\_\{m\}s\_\{m\}\(x;\\varphi\)andRsparse,i=𝔼\(x,y\)∼𝒟i​\[ℓm^​\(x\)​\(x,y\)\]R\_\{\\mathrm\{sparse\},i\}=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{i\}\}\[\\ell\_\{\\widehat\{m\}\(x\)\}\(x,y\)\], and assume

\[ℓm^​\(x\)​\(x,y\)−ℓm†​\(x\)​\(x,y\)\]\+≤Δmax\.\[\\ell\_\{\\widehat\{m\}\(x\)\}\(x,y\)\-\\ell\_\{m^\{\\dagger\}\(x\)\}\(x,y\)\]\_\{\+\}\\leq\\Delta\_\{\\max\}\.\(48\)Since the mismatch event is contained in\{marφ⁡\(x\)≤0\}\\\{\\operatorname\{mar\}\_\{\\varphi\}\(x\)\\leq 0\\\},

###### Corollary 5\(Optimization\-controlled sparse\-routing risk\)\.

Rsparse,i−Ri,oracle\\displaystyle R\_\{\\mathrm\{sparse\},i\}\-R\_\{i,\\mathrm\{oracle\}\}≤Δmax​R0,i​\(φ\)\\displaystyle\\leq\\Delta\_\{\\max\}R\_\{0,i\}\(\\varphi\)\(49\)≤2​Δmaxcgap​\(Ri​\(φ\)−Ri⋆\+Ai\+βmix\)\.\\displaystyle\\leq\\frac\{2\\Delta\_\{\\max\}\}\{c\_\{\\mathrm\{gap\}\}\}\\bigl\(R\_\{i\}\(\\varphi\)\-R\_\{i\}^\{\\star\}\+A\_\{i\}\+\\beta\_\{\\mathrm\{mix\}\}\\bigr\)\.

For bounded metrics one may take a globalΔmax\\Delta\_\{\\max\}; for unbounded token\-level cross\-entropy, Eq\. \([48](https://arxiv.org/html/2607.26618#A3.E48)\) applies to clipped loss or requires a finite conditional excess\-loss moment\. The routing heatmap and same\-checkpoint sparse\-inference ablation empirically probe the reliable\-routing regime captured by the bound\.