A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization

# A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilization
Source: [https://arxiv.org/html/2609.25471](https://arxiv.org/html/2609.25471)
Zakaria AldenehMartin PelikanJan “Honza” SilovskyTatiana LikhomanenkoSheikh Shams AzamAffiliation:Apple

September 21, 2026

###### Abstract

Semi\-supervised federated learning \(SSFL\) trains models on clients’ unlabeled data using a teacher to generate pseudo\-labels, with a small labeled seed dataset on the server\. Automatic Speech Recognition \(ASR\) is particularly fragile here: pseudo\-label errors compound across the output sequence and across training rounds into divergence, leaving a large gap to fully\-supervised FL\. We show that closing this gap turns on two coupled design axes—the teacher\(which model generates the pseudo\-labels\) andthe anchor\(the server\-side updates on labeled data that stabilize training\)\. On the teacher axis, a per\-client online teacher \(each client’s own evolving model\) diverges on its own, but once stabilized it matches or beats the broadcast global teacher \(one server model, fixed within a round\)—decisively in\-domain and competitively under domain shift\. As the seed grows stronger and the online teacher’s advantage narrows, a transitioning teacher \(global→\\rightarrowonline at roundrr\) matches or beats both\. On the anchor axis, the server must keep training on labeled data between rounds—otherwise the online teacher drifts—and this interleaving, more than the seed model, governs convergence\. The two axes are inseparable: aggressive teacher choices pay off only once the anchor stabilizes training, which is highly sensitive to data augmentation and batch size—the settings that govern how much input and gradient noise the server injects\. How much stabilization is needed is domain\-dependent, governed by the dispersion of the seed data and its overlap with client data\. These findings yield guidelines for SSFL in ASR training, improving over the strongest prior method on 9 of 11 pairs, by20\.8%20\.8\\%on average in\-domain and10\.0%10\.0\\%cross\-domain, narrowing the gap to fully\-supervised FL\.

††We study semi\-supervised federated ASR entirely in simulation on public datasets, spanning a wide range of speaking styles and domain\-shift conditions along with a broad sweep of training configurations, to distill practical guidelines\. No real user data or production telemetry informed this work\. All reported system parameters are choices made for these simulations and do not represent any currently deployed system or plan thereof\.## 1Introduction

Safeguarding privacy is a foundational commitment when building modern machine learning systems, especially when training on data that may be considered personal\([Truong et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib45)\)\. One option for alleviating such concerns is using synthetic data, which can be generated at scale with little to no privacy concerns\. However, synthetic alternatives still face challenges and do not yet close the gap to real user data\. For example, in speech, Text\-to\-Speech \(TTS\) substitutes struggle to replicate personalized speech patterns, diverse acoustic environments, and natural device characteristics, leaving distributional mismatch with real\-world conditions\([Hilmes et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib17);[Su et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib43);[Ogun et al\., 2025](https://arxiv.org/html/2609.25471#bib.bib33)\)\. An alternative remedy—and the focus of this work—is federated learning\([Truong et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib45);[Jeong et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib19)\)\.

Federated learning \(FL\)\([McMahan et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib30)\)inverts the pipeline—instead of moving data to the model, the model moves to the data, with user data never leaving the device\. Each device computes a model update locally on its own data and communicates only that update to the server, which aggregates the updates from many devices into a new global model\. Speech is a natural fit for this framework: user audio is abundant on consumer devices, privacy\-sensitive when centralized, and not yet replaceable by synthetic alternatives\. Recent work has demonstrated that FL for end\-to\-end Automatic Speech Recognition \(ASR\) is feasible, with FL models approaching centrally\-trained baselines\([Pelikan et al\., 2025](https://arxiv.org/html/2609.25471#bib.bib36);[Guliani et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib15)\)\.

To strengthen privacy further, differential privacy \(DP\) adds calibrated noise to the communicated updates so that information unique to any single user is masked, bounding how much the shared updates can reveal about that user’s data\([Abadi et al\., 2016](https://arxiv.org/html/2609.25471#bib.bib1)\)\. DP has been combined with FL for ASR via per\-layer clipping techniques that mitigate DP noise’s disproportionate impact on attention layers, achieving strong privacy guarantees at modestword error rate \(WER\)cost\([Pelikan et al\., 2025](https://arxiv.org/html/2609.25471#bib.bib36)\)\. While this private deployment scenario motivates our work, we do not add DP noise in this first study: our recipe depends on large, low\-variance server\-side updates on labeled data to stabilize training against pseudo\-label noise, and DP\-SGD’s per\-layer clipping and noise injection may work against that variance reduction in ways that require dedicated tuning and analysis \([Section8](https://arxiv.org/html/2609.25471#S8)\); we therefore leave a joint study of DP and pseudo\-labeling to future work\.

Beyond privacy, a separate obstacle limits FL for ASR in practice: most work assumes that clients hold labeled data\([Li et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib25);[Reddi et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib38);[Azam, 2022](https://arxiv.org/html/2609.25471#bib.bib5)\)—an assumption that rarely holds, as transcription is expensive and time consuming on device\.*Semi\-supervised federated learning*\(SSFL\) addresses this gap: the server holds a small labeled corpus—the*seed data*—while clients contribute only unlabeled audio—the*client data*\. The two may also differ in distribution, with the seed often drawn from a curated corpus and client data reflecting diverse real\-world speaking styles and acoustic conditions, making domain shift both between seed and clients, and across clients themselves a central concern\.

This domain mismatch is further amplified by the nature of semi\-supervised learning:automated transcripts produced for a client’s unlabeled audio—pseudo\-labels—inject additional noise into training, compounding the instability that domain shift already creates\. Therefore, for SSFL for ASR, two design choices are central:which model generates the pseudo\-labelsandhow the server “anchors” training using the labeled seed data\.111We use standard teacher–student terminology: the model being optimized is the*student*\(each client’s local model\)and the model that generates its pseudo\-labels is the*teacher*\. The two may or may not be the same model\.For pseudo\-label generation, existing SSFL for ASR works use either a*static teacher*\(frozen at the seed model\)\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)or a*global teacher*\(the broadcast model held fixed within each round but refreshed across rounds\)\([Rao et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib37)\), and combines server\-side gradients with the aggregated client pseudo\-gradients to reduce drift away from the labeled distribution\. The alternative, an*online teacher*\(each client’s evolving local model, which is the student itself\), has not been examined in the SSFL for ASR literature, as[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)argue that the online teacher would drift from the global model and destabilize training\. Whether these defaults are optimal, or even adequate for ASR tasks, has not been systematically studied\.

Across both axes, we observe that the conventional defaults turn out to be suboptimal, leaving a substantial WER gap to fully\-supervised FL\. On the pseudo\-label\-source axis \(described in[Section5](https://arxiv.org/html/2609.25471#S5)\), the online teacher matches or beats the global teacher in\-domain and on most cross\-domain settings\. As the strength of the seed model grows, the online–global gap shrinks, and a*transitioning*teacher \(global→\\rightarrowonline at roundrr\) matches or beats both pure strategies when the global is better than the online teacher\. However, the online and transitioning teachers can break asymmetrically: swapping which corpus serves as the seed and which as the client can flip training from diverging to converging—evidence that the domain gap acts asymmetrically rather than as a symmetric distance between corpora\.

In[Section6](https://arxiv.org/html/2609.25471#S6), we trace this failure to two server\-update factors—the SpecAugment\([Park et al\., 2019](https://arxiv.org/html/2609.25471#bib.bib34)\)\(a data augmentation that masks blocks of time and frequency in the input features\) strength and batch size used during server updates on the seed data\. Strong SpecAug causes outright divergence, with a deletion\-only error signature in the middle of training\. Small batch does not necessarily diverge but destabilizes training by injecting noise into the server update, converging to a significantly higher WER than large batch\. Tuning both factors stabilizes SSFL training and substantially improves WER for both teachers, most dramatically for the online teacher, which it rescues from divergence\. In[Section7](https://arxiv.org/html/2609.25471#S7), we provide a holistic comparison against three existing SSFL for ASR methods across several source×\\timestarget pairs, showing that the resulting recipe closes the gap to fully\-supervised FL while providing practical guidance for both design choices\. Our contributions are summarized as follows:

1. 1\.We analyze the choice of teacher for pseudo\-labeling in SSFL for ASR—global, online, and transitioning\. Contrary to the prevailing assumption that the online \(per\-client evolving\) teacher is unstable, we show that it matches or beats the global teacher on most data pairs, and a transitioning teacher \(global→\\rightarrowonline at roundrr\) extends this advantage as the seed model grows stronger\([Section5](https://arxiv.org/html/2609.25471#S5)\)\.
2. 2\.We identify frequent enough server updates on the seed data as a key stability condition for the online and transitioning teachers\([Section6](https://arxiv.org/html/2609.25471#S6)\)\.
3. 3\.We characterize an asymmetric breakage of the online and transitioning teachers under cross\-domain shift: swapping which corpus serves as the seed and which as the client can flip training from diverging to converging—evidence that the domain gap acts asymmetrically rather than as a symmetric distance between corpora\([Section6\.1](https://arxiv.org/html/2609.25471#S6.SS1)\)\.
4. 4\.We analyze the two server\-update factors that govern stability—SpecAug strength and batch size\. Together they set the input and stochastic gradient noise each server update injects, and thus the cross\-client consistency of the pseudo\-labels; strong SpecAug causes training to diverge and a small batch destabilizes it, whereas tuning both stabilizes SSFL training \([Section6\.3](https://arxiv.org/html/2609.25471#S6.SS3)\)\.
5. 5\.We compare the resulting recipe against three existing SSFL for ASR methods \(Static PL, FedNST\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\),[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)\) and a strong global teacher baseline derived from[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)across severalsource \(server data\)→\\rightarrowtarget \(client data\) pairs, showing that it closes the gap to fully\-supervised FL\([Section7\.1](https://arxiv.org/html/2609.25471#S7.SS1)\)\.
6. 6\.We provide extensive ablation studiescharacterizing howcohort size, local steps, and thechoice of pseudo\-label generator, i\.e\., teacher \([Table2](https://arxiv.org/html/2609.25471#S3.T2)\) affect WER and training stability \([Section7\.3](https://arxiv.org/html/2609.25471#S7.SS3)\)\.

## 2Related Work

### 2\.1Federated Learning

Federated learning \(FL\) enables collaborative model training across distributed clients without sharing raw data\([McMahan et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib30)\), withFedAvgserving as the foundational algorithm that performs multiple local SGD steps before server\-side aggregation\. A central challenge is*client drift*: when client data is non\-IID, local models diverge from the global optimum, slowing convergence\([Kairouz et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib20)\)\. A broad line of work addresses this through proximal regularization\([Li et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib25)\), control variates\([Karimireddy et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib21)\), dynamic regularization\([Acar et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib2)\), adaptive server\-side optimization\([Reddi et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib38);[Wang et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib48)\), and personalization\([Fallah et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib11)\), all in the supervised setting\. Our work targets the semi\-supervised FL setting, where the dominant source of instability is not client drift but*pseudo\-label drift*—the divergence of client\-generated pseudo labels from the global model’s pseudo labels—which we study along two axes: how pseudo labels are generated \([Section2\.2](https://arxiv.org/html/2609.25471#S2.SS2)\) and how the server stabilizes training \([Section2\.3](https://arxiv.org/html/2609.25471#S2.SS3)\)\.

### 2\.2Pseudo\-labeling: from Centralized SSL to SSFL

Pseudo\-labeling—training on model\-generated labels for unlabeled data—is the dominant approach to semi\-supervised learning\([Lee, 2013](https://arxiv.org/html/2609.25471#bib.bib23)\)\. In image classification, FixMatch combines confidence\-thresholded pseudo\-labels with consistency regularization\([Sohn et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib41)\)\. Follow\-up work refines the thresholding strategy via curriculum learning\([Zhang et al\., 2021a](https://arxiv.org/html/2609.25471#bib.bib55)\)and self\-adaptive schedules\([Wang et al\., 2023b](https://arxiv.org/html/2609.25471#bib.bib50)\), and recent analysis explains FixMatch’s generalization advantage over supervised training\([Li et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib24)\)\.

For speech recognition, noisy student training generates pseudo\-labels from a teacher model and iteratively retrains a larger student\([Park et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib35)\); SlimIPL\([Likhomanenko et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib26)\)and momentum pseudo\-labeling\([Higuchi et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib16)\)replace the costly iterative regime with on\-the\-fly pseudo\-label regeneration during training, and continuous pseudo\-labeling extends this across the full training trajectory\([Likhomanenko et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib27)\)\. A common thread is that the*choice of pseudo\-label source*—a fixed seed teacher, an iterative EMA teacher \(which we call theglobal teacher\), or the evolving student itself \(theonline teacher\)—is a central design axis\.

In federated SSL, pseudo\-labels are typically generated by the global server model broadcast each round, as in SemiFL\([Diao et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib10)\)and most subsequent image\-classification SSFL work\. FedSwitch\([Zhao et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib58)\)is the closest prior work to study the choice of pseudo\-label source: it adaptively switches between a global teacher and a teacher\-student EMA based on relative quality, reporting that the choice substantially affects convergence\. BSemiFL\([Wang et al\., 2025](https://arxiv.org/html/2609.25471#bib.bib47)\)weighs the two sources via a Bayesian density estimator\. However, both approaches target image classification and assume discrete label spaces; FedSwitch additionally assumes clients are selected repeatedly across rounds so that per\-client teacher state persists\. Neither assumption holds for sequence\-output ASR with large, non\-persistent client populations\.

Other SSFL directions address orthogonal design choices: parameter decomposition for disjoint labeled/unlabeled aggregation\([Zhang et al\., 2021b](https://arxiv.org/html/2609.25471#bib.bib57)\), consistency regularization\([Yang et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib53);[Malaviya et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib29)\), and knowledge\-enhanced or prototype\-based aggregation\([Wang et al\., 2023a](https://arxiv.org/html/2609.25471#bib.bib49);[Hu et al\., 2026](https://arxiv.org/html/2609.25471#bib.bib18)\)\. These directions are complementary to the pseudo\-labeling and anchoring questions studied here\.

### 2\.3Server Training and Anchoring

A parallel concern in semi\-supervised learning is how to prevent the pseudo\-label source from drifting as the model updates on its own predictions\. Mean Teacher maintains an exponential moving average of the student’s weights as the teacher, anchoring pseudo\-label generation to a slower\-moving checkpoint\([Tarvainen & Valpola, 2017](https://arxiv.org/html/2609.25471#bib.bib44)\)\. Temporal ensembling achieves a similar effect by ensembling past predictions\([Laine & Aila, 2017](https://arxiv.org/html/2609.25471#bib.bib22)\)\. Noisy Student iteratively re\-initializes a larger student from a frozen teacher, which acts as a coarse\-grained anchor between rounds of self\-training\([Xie et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib51)\)\. In the federated setting, the analogous question is how the server anchors training against noisy client pseudo\-labels\.

Federated SSL approaches vary in how the server stabilizes training\. FedNST freezes pseudo\-labels generated once by the seed model, so the anchor is the seed itself\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)\. SemiFL alternates client pseudo\-labeled rounds with server\-side fine\-tuning on the labeled seed data, using the labeled pass as a periodic anchor\([Diao et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib10)\)\.[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)augment each round with a supervised gradient computed on a small held\-out set of labeled server clients\. GDST integrates server\-side fine\-tuning to stabilize the global model against noisy client updates with client\-side training on unlabeled data via pseudo\-labeling and global distillation for image classification\([Liu et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib28)\)\. Each of these strategies implicitly makes a different choice about when and how the labeled signal re\-anchors training, but the implications for ASR—where pseudo\-label quality is harder to assess and token\-level errors compound—have not been systematically studied\.

### 2\.4Federated Learning in ASR

Supervised FL for ASR is well\-established\.[Guliani et al\. \(2021\)](https://arxiv.org/html/2609.25471#bib.bib15)demonstrates FedAvg\-style training for end\-to\-end ASR at scale, and[Pelikan et al\. \(2025\)](https://arxiv.org/html/2609.25471#bib.bib36)establish the first benchmark for differentially private FL\-ASR using per\-layer clipping to handle transformer gradient heterogeneity\. All of these assume fully labeled clients, which is impractical for on\-device transcription\. The semi\-supervised setting we study—where labels reside only at the server—is formalized next \([Section3](https://arxiv.org/html/2609.25471#S3)\), alongside the pseudo\-labeling and server\-anchoring questions introduced above\.

## 3Semi\-Supervised FL for ASR: Problem Setting and Practical Constraints

Semi\-supervised FL admits four canonical configurations depending on where labeled data resides\([Song et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib42)\):*labels\-at\-all\-clients*\(every client is labeled\),*labels\-at\-partial\-clients*\(only a subset of clients are labeled\),*unlabeled\-at\-server*\(clients are labeled, the server is not\), and*labels\-at\-server*\(the server is labeled, clients are not\)\. Of these,*labels\-at\-server*setting is the closest to theconstraints that motivatedeployed ASR: transcription is expensive and infeasible on\-device, whereas service providers can curate a small, high\-quality labeled corpus centrally\. The other three presuppose client\-side labeling at a scale that is not available in practice\. Accordingly,labels\-at\-server is the focus of this paper, and it is also the setting adopted by all existing SSFL for ASR work\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31);[Rao et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib37)\)\.

A second distinction from centralized semi\-supervised ASR is that the objective here is*twofold*: \(i\) reduce WER on the target client domain by exploiting the unlabeled client audio, and \(ii\) avoid regressing on the server\-side seed distribution, which represents the curated baseline the service is already committed to\. A strategy that improves client\-domain WER at the cost of significant seed\-corpus regression is not a practical win, so we report WER on both the target and the seed test sets throughout\.

Compared to image classification, ASR also poses challenges that are specific to the output space: predictions are variable\-length sequences rather than fixed\-label classes, pseudo\-label quality is harder to assess, and token\-level errors compound at the sequence level\. As a result, only a handful of works have addressed SSFL for ASR\. FedNST\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)extends noisy student training\([Park et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib35)\)to the federated setting, generating pseudo\-labels once with a seed model and never updating them throughout training\.[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)employ the global model to generate pseudo\-labels for local updates as in\([Diao et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib10)\), with server\-side gradients computed on a small held\-out labeled subset\. Both approaches rely on a fixed or global teacher and do not explore locally updated teachers, leaving open the question of whether adaptive local pseudo\-labeling can improve performance—the central question of our work\. We next formalize the training procedure and identify the practical constraints that shape the design space\.

Algorithm 1Semi\-Supervised Federated Learning1:Model parameter

θ\\theta, set of clients

𝒮\\mathcal\{S\}with per\-round cohort

𝒮t\\mathcal\{S\}\_\{t\}, labeled server data

ℒ\\mathcal\{L\}, unlabeled client data

𝒰i\\mathcal\{U\}\_\{i\}of client

ii, communication rounds

TT, server training probability

pp, server optimizerServerOpt

2:Trained global model parameters

3:Seed Training Stage

4:Train a model on

ℒ\\mathcal\{L\}until convergence to obtain

θ0\\theta\_\{0\}, used to initialize teacher

ϕ\\phiand student models

θ\\theta
5:

6:Federated Training Stage

7:foreach communication round

t=1,2,…,Tt=1,2,\\dots,Tdo

8:Server selects a subset of clients

𝒮t\\mathcal\{S\}\_\{t\}from

𝒮\\mathcal\{S\}
9:for allclient

i∈𝒮ti\\in\\mathcal\{S\}\_\{t\}in paralleldo

10:PL generation:Generate pseudo\-labels for

𝒰i\\mathcal\{U\}\_\{i\}using teacher

ϕt\\phi\_\{t\}\(global\) or

θt,τ\\theta\_\{t,\\tau\}\(online\)

11:Data Filtering:Remove samples with high uncertainty before local training

12:Local Training:Optimize locally to

θti\\theta\_\{t\}^\{i\}and send pseudo\-gradient

gti=θt−θtig\_\{t\}^\{i\}=\\theta\_\{t\}\-\\theta\_\{t\}^\{i\}to the server

13:endfor

14:FL update:Aggregate

g¯t=1\|𝒮t\|​∑i∈𝒮tgti\\bar\{g\}\_\{t\}=\\frac\{1\}\{\|\\mathcal\{S\}\_\{t\}\|\}\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}g\_\{t\}^\{i\}\(FedAvg\) and update

θt\+1=ServerOpt​\(θt,g¯t\)\\theta\_\{t\+1\}=\\textsc\{ServerOpt\}\(\\theta\_\{t\},\\bar\{g\}\_\{t\}\)
15:Server update:With probability

pp, further update

θt\+1\\theta\_\{t\+1\}with a supervised step on

ℒ\\mathcal\{L\}
16:Teacher update:Update the teacher model

ϕt\\phi\_\{t\}if applicable, e\.g\., EMA

17:endfor

Static PLFedNSTRao*et al*\.Diao*et al*\. for ASRPL generationone\-timeθ0\\theta\_\{0\}one\-timeθ0\\theta\_\{0\}EMA teacherϕt\\phi\_\{t\}EMA teacherϕt\\phi\_\{t\}Server updatesno server updatesfull server dataheld\-out server clientsa batch of server dataTable 1:Comparison of SSFL methods along the pseudo\-label generation and server training axes\.TeacherPseudo\-label source \(at roundtt, local stepτ\\tau\)GlobalEMA of the broadcast model,ϕt=λ​ϕt−1\+\(1−λ\)​θt\\phi\_\{t\}=\\lambda\\phi\_\{t\-1\}\+\(1\-\\lambda\)\\theta\_\{t\}\(λ=0\.99\\lambda\{=\}0\.99\), held fixed within the roundOnlineclient’s local modelθt,τ\\theta\_\{t,\\tau\}, evolving within the roundLocal EMAθ¯t,τ=γ​θ¯t,τ−1\+\(1−γ\)​θt,τ\\bar\{\\theta\}\_\{t,\\tau\}=\\gamma\\,\\bar\{\\theta\}\_\{t,\\tau\-1\}\+\(1\-\\gamma\)\\,\\theta\_\{t,\\tau\}\(γ=1\\gamma\{=\}1: global,γ=0\\gamma\{=\}0: online\)Transitioningglobal teacher for the firstrrrounds, online teacher thereafterTable 2:The pseudo\-label teachers compared in this paper\. The global and online teachers are the two primary sources; the local EMA teacher \([Section5\.1](https://arxiv.org/html/2609.25471#S5.SS1)\) interpolates between them, and the transitioning teacher \([Section5\.4](https://arxiv.org/html/2609.25471#S5.SS4)\) switches from global to online at roundrr\.Note that the global teacher’s cross\-round decayλ\\lambdaand the local EMA teacher’s within\-round decayγ\\gamma\(reset each round\) are independent parameters on different timescales\.### 3\.1Problem Setting

##### Training template\.

Algorithm[1](https://arxiv.org/html/2609.25471#alg1)summarizes the SSFL training procedure under the labels\-at\-server setting\. Training proceeds in two stages\. In the*seed stage*, the server trains a model on the labeled corpusℒ=\{\(𝐱j,𝐲j\)\}j=1Ns\\mathcal\{L\}=\\\{\(\\mathbf\{x\}\_\{j\},\\mathbf\{y\}\_\{j\}\)\\\}\_\{j=1\}^\{N\_\{s\}\}until convergence, obtaining a model parameterθ0\\theta\_\{0\}; this seed model initializes both the student \(the model being trained\) and the teacher \(the model that generates pseudo\-labels\)\. In the*federated stage*, each communication roundttproceeds as follows: \(i\) server broadcasts the current global model to a cohort𝒮t⊂𝒮\\mathcal\{S\}\_\{t\}\\subset\\mathcal\{S\}, \(ii\) each clientiigenerates pseudo\-labels for its unlabeled audio𝒰i=\{𝐱i,j\}j=1NC\\mathcal\{U\}\_\{i\}=\\\{\\mathbf\{x\}\_\{i,j\}\\\}\_\{j=1\}^\{N\_\{C\}\}using some teacher, optionally filters uncertain samples, applies augmentation, and performs several local SGD steps, \(iii\) clients send their resulting pseudo\-gradients back to the server, which aggregates them \(e\.g\., with FedAvg\([McMahan et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib30)\)\) and applies a server update, and \(iv\) model is updated further with the labeled seed data before updating the teacher\.

##### Design axes\.

Existing SSFL methods differ along two axes within this template: thepseudo\-label source\([Section2\.2](https://arxiv.org/html/2609.25471#S2.SS2)\)—which model generates the pseudo\-labels for client training—and theserver training strategy\([Section2\.3](https://arxiv.org/html/2609.25471#S2.SS3)\)—how and when the server re\-injects the labeled seed signal into the global model\. Table[1](https://arxiv.org/html/2609.25471#S3.T1)summarizes how existing methods instantiate these two axes\.

##### Prior methods\.

Along the*pseudo\-label source*axis, Static PL and FedNST\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)generate pseudo\-labels once from the initial seed modelθ0\\theta\_\{0\}and keep them fixed throughout training, whereas[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)and[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)regenerate pseudo\-labels each round from an exponential moving average \(EMA\) teacherϕt\\phi\_\{t\}that evolves with the global model, defined asϕt=λ​ϕt−1\+\(1−λ\)​θt\\phi\_\{t\}=\\lambda\\,\\phi\_\{t\-1\}\+\(1\-\\lambda\)\\,\\theta\_\{t\}, whereλ\\lambdais the EMA decay rate\. Along the*server training*axis, Static PL performs no server\-side updates; FedNST computes a weighted average of gradients from the full server update \(gradient descent\) and client pseudo\-gradients; Rao*et al*\. augment each FL round with supervised gradients from held\-out labeled server clients; and Diao*et al*\. alternate FL aggregation rounds with a server\-side supervised pass on a single batch\.222FL update and server update are merged together to update the student modelθ\\thetaonce in FedNST and Rao*et al*\.We provide a per\-method description of how FedNST and[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)instantiate these two axes in[Section11](https://arxiv.org/html/2609.25471#S11)\.

Note that[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)validate their proposed method on image classification tasks \(not ASR tasks\)\. However, as it is one of the most effective methods in SSFL for image classification and less bound to image classification tasks333[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)employ FixMatch\([Li et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib24)\)but it is not straightforward to apply such algorithms to ASR tasks due to variable\-length sequences and non\-unique alignment of CTC\., we modify and employ it as a baseline for comparison\. Interestingly,[Figure1\(a\)](https://arxiv.org/html/2609.25471#S4.F1.sf1)shows that[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)for ASR significantly outperform other baselines: Static PL, FedNST and[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)whenℒ:\\mathcal\{L\}:LS100 and𝒰\\mathcal\{U\}: LS860\(LibriSpeech subsets;[Section4\.1](https://arxiv.org/html/2609.25471#S4.SS1)\)\. Therefore, we focus our analysis on comparison with the tailored[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10)for ASR\.

##### Global vs\. online teacher\.

Letf^​\(⋅,θ\)\\hat\{f\}\(\\cdot;\\theta\)denote the decoding function parameterized byθ\\thetagenerating pseudo\-labels\. At roundtt, local stepτ\\tau, on clientiiand sample𝐱i,j\\mathbf\{x\}\_\{i,j\}, the pseudo\-label𝐲^i,j\\hat\{\\mathbf\{y\}\}\_\{i,j\}is produced from one of two sources:

Global teacher:𝐲^i,j=f^​\(𝐱i,j,ϕt\),\\displaystyle\\hat\{\\mathbf\{y\}\}\_\{i,j\}=\\hat\{f\}\\\!\\left\(\\mathbf\{x\}\_\{i,j\};\\ \\phi\_\{t\}\\right\),\(1\)Online teacher:𝐲^i,j=f^​\(𝐱i,j,θt,τ\),\\displaystyle\\hat\{\\mathbf\{y\}\}\_\{i,j\}=\\hat\{f\}\\\!\\left\(\\mathbf\{x\}\_\{i,j\};\\ \\theta\_\{t,\\tau\}\\right\),\(2\)whereϕt\\phi\_\{t\}isthe teacher model, maintained as an exponential moving average of the broadcast global model across rounds \(ϕt=λ​ϕt−1\+\(1−λ\)​θt\\phi\_\{t\}=\\lambda\\,\\phi\_\{t\-1\}\+\(1\-\\lambda\)\\,\\theta\_\{t\},λ=0\.99\\lambda=0\.99;[Section10\.3](https://arxiv.org/html/2609.25471#S10.SS3)\), held fixed across all local steps within the round, andθt,τ\\theta\_\{t,\\tau\}is the client’s local parameters at stepτ∈\{0,1,…,K−1\}\\tau\\in\\\{0,1,\\ldots,K\-1\\\}, which evolve during local training\.

The two teachers therefore differ only when clients take more than one local step: with a single step \(K=1K=1\) the local model has not yet moved, soθt,τ=ϕt\\theta\_\{t,\\tau\}=\\phi\_\{t\}and every client is labeled by the same global model\. Multiple local steps are what let the online teacher adapt to each client within the round, but they trade off against stability—too many steps let the local model drift and its pseudo\-labels grow stale or inconsistent, which we study in[Section7\.3](https://arxiv.org/html/2609.25471#S7.SS3)\.More fundamentally, multiple local steps are standard in FL because communication is the bottleneck—each round pays network latency and a synchronous aggregation barrier, whereas local computation is cheap—so more local work per round reduces the number of communication rounds\([McMahan et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib30)\), and can even accelerate convergence\([Mishchenko et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib32)\)\.[Table2](https://arxiv.org/html/2609.25471#S3.T2)summarizes these two sources together with the local EMA and transitioning teachers introduced in[Section5](https://arxiv.org/html/2609.25471#S5)\. The remainder of the paper analyzes this choice—and its combination with server update—across progressively harder domain\-shift regimes\.

### 3\.2Practical Constraints and Scope

##### Practical constraints\.

We adopt three simplifying constraints that shape our experimental design; they are not intended to characterize any particular production FL system:

1. 1\.No labeled data exists on clients; only the server holds a labeled seed dataset\.
2. 2\.Clients are*non\-persistent*across rounds—a client selected in one round may never be selected again—so strategies that accumulate per\-client state \(e\.g\., a local model checkpoint\) across rounds are fragile\.
3. 3\.Communication overhead must remain minimal: only a single model can be broadcast per round, ruling out approaches that require transmitting separate teacher and student snapshots\.

Together, these constraints narrow the design space to methods that differ in how pseudo\-labels are generated each round and how the server re\-injects the labeled signal into the global model—the two axes of Table[1](https://arxiv.org/html/2609.25471#S3.T1)\.

##### Scope\.

Several techniques have been shown to improve pseudo\-label quality in centralized semi\-supervised ASR but are orthogonal to the two axes studied here\. Language\-model\-guided decoding during pseudo\-label generation—such as LM beam search\([Xu et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib52)\)or LM fusion\([Park et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib35);[Zhang et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib56)\)—can improve transcription quality regardless of whether the pseudo\-label source is a static seed or an evolving EMA teacher\. Similarly, data filtering based on confidence or uncertainty, as employed by[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37), can be layered on top of any pseudo\-label source and server training strategy\. Because these mechanisms can be independently combined with the choices we study, we hold them fixed \(no LM, no filtering\) throughout our experiments to isolate the effect of the pseudo\-label source and server training strategy\.

## 4Experimental Setup

All configurations described below and in[Section10](https://arxiv.org/html/2609.25471#S10)reflect choices for our public\-dataset simulations\.

### 4\.1Datasets and Domain\-Shift Pairs

We evaluate methods on four popular English ASR corpora spanning a range of speaking styles and recording conditions: LibriSpeech \(LS, read audiobooks\), TED\-LIUM \(Ted, TED talks\), Common Voice \(CV, crowd\-sourced read speech\) and Fisher \(telephone conversations\)\.[Table5](https://arxiv.org/html/2609.25471#S10.T5)summarizes each corpus \(hours, number of speakers, and utterance durations\)\. Throughout, LSxxdenotes anxx\-hour labeled subset of LS, and CV10 and CV90 a10%10\\%/90%90\\%split of CV; subsets of the same corpus \(e\.g\., LS100 and LS860\) are drawn from the same distribution and use disjoint speakers, differing only in size\.

Experiments are conducted on server \(source\)→\\rightarrowclient \(target\) pairs, for example, LS100→\\rightarrowLS860 denotesℒ:\\mathcal\{L\}:LS100 and𝒰:\\mathcal\{U\}:LS860\. These pairs range from in\-domain \(server and client drawn from the same corpus\) to cross\-domain \(different corpora\)\. As one possible way to measure this domain shift, an embedding\-based similarity measure \([Section12](https://arxiv.org/html/2609.25471#S12)\) cleanly separates the in\-domain pairs from the cross\-domain ones, and can serve as a quantitative proxy for the server–client overlap\.

For client datasets, we split datasets into small per speaker datasets, treating each speaker as asimulatedclient for federated learning\. For server datasets, we do not split them and sample batches uniformly randomly after filtering out audio samples where their duration is too long, e\.g\.,3030sec\. Across every pair, the server \(seed\) and client speaker sets are disjoint—no speaker appears in both—so clients never observe the labeled seed\.In every pair the server holds a small amount of labeled seed data and the clients hold a larger, disjoint pool of unlabeled data\. For the in\-domain pairs the seed and client data are non\-overlapping splits of a single corpus \(LS100→\\rightarrowLS860:100100h of labeled seed data vs\.860860h of unlabeled client data; CV10→\\rightarrowCV90:10%10\\%labeled seed data vs\.90%90\\%unlabeled data distributed across clients\); for the cross\-domain pairs a small labeled seed corpus is paired with a larger unlabeled client corpus from a different domain \(e\.g\.,100100h of LS against∼1,593\{\\sim\}1\{,\}593h of CV\)\. Only the server data is labeled\.

For every pair, we report WER on both the server and client datasets, since a strategy that improves target WER at the cost of significant regression on server data is not a practical win\. All results are reported on the dev sets, as they form part of our analysis and hyperparameter tuning, except for the final comprehensive comparison in[Table3](https://arxiv.org/html/2609.25471#S7.T3), which we report on the test sets\. Per\-dataset statistics and train, dev, and test splits are deferred to[Section10\.1](https://arxiv.org/html/2609.25471#S10.SS1)\.

### 4\.2Model Architecture and Training Configuration

##### Model\.

All experiments use a Transformer\([Vaswani et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib46)\)encoder trained with CTC loss\([Graves et al\., 2006](https://arxiv.org/html/2609.25471#bib.bib13)\)on 80\-dim log\-mel filterbank features, a character\-level tokenizer, and greedy decoding\. Despite its simplicity, greedy decoding is shown to be as effective as more costly decoding algorithms such as beam search for semi\-supervised learning for ASR tasks\([Likhomanenko et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib26)\)\.

As stated in[section3\.2](https://arxiv.org/html/2609.25471#S3.SS2), we deliberately omit both language\-model\-guided decoding and confidence\-based data filtering: these are orthogonal to our two axes of study—pseudo\-label generation and server training—and either could be added on top of any configuration we evaluate\.

##### Training\.

For training, audio feature inputs are augmented with SpecAugment\([Park et al\., 2019](https://arxiv.org/html/2609.25471#bib.bib34)\)although we use clean features without augmentation to generate pseudo\-labels\. We use a simple SGD\([Robbins & Monro, 1951](https://arxiv.org/html/2609.25471#bib.bib40)\)without learning rate scheduler for local updates on the client side and LAMB\([You et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib54)\)on the server side for both FL updates and server updates on the labeled server dataℒ\\mathcal\{L\}\. Note that we deliberately choose a layer\-wise adaptive optimizer, e\.g\., LAMB, following[Azam et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib6)\.

The seed model—trained on the server’s labeled dataℒ\\mathcal\{L\}—initializes both the studentθ\\thetaand the EMA teacherϕ\\phiused for pseudo\-labeling\. In addition, we reduce both dropout and layer dropout rate from0\.30\.3\(for seed model training\) to0\.10\.1during SSFL to increase the model capacity required to learn from large unlabeled data, following[Likhomanenko et al\. \(2021\)](https://arxiv.org/html/2609.25471#bib.bib26)\. More detailed architecture and training hyperparameters are provided in[Section10\.2](https://arxiv.org/html/2609.25471#S10.SS2)\.

### 4\.3Semi\-Supervised Federated Learning Configuration

We aggregate client updates with FedAvg\([McMahan et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib30)\)\. Each sampled clienti∈𝒮ti\\in\\mathcal\{S\}\_\{t\}trains locally from the broadcast global modelθt\\theta\_\{t\}and returns a pseudo\-gradientgti=θt−θtig\_\{t\}^\{i\}=\\theta\_\{t\}\-\\theta\_\{t\}^\{i\}, whereθti\\theta\_\{t\}^\{i\}are its locally updated parameters\. FedAvg takes their unweighted mean, which the server optimizer then applies to obtain the next global model:

g¯t=1\|𝒮t\|​∑i∈𝒮tgti,θt\+1=ServerOpt​\(θt,g¯t\),\\bar\{g\}\_\{t\}=\\frac\{1\}\{\|\\mathcal\{S\}\_\{t\}\|\}\\sum\_\{i\\in\\mathcal\{S\}\_\{t\}\}g\_\{t\}^\{i\},\\qquad\\theta\_\{t\+1\}=\\textsc\{ServerOpt\}\\\!\\left\(\\theta\_\{t\},\\,\\bar\{g\}\_\{t\}\\right\),\(3\)so every sampled client contributes equally regardless of how much data it holds\. The server optimizerServerOptis LAMB \([Section4\.2](https://arxiv.org/html/2609.25471#S4.SS2)\)\.

The server applies a supervised update onℒ\\mathcal\{L\}each round with probabilitypp\(*server training probability*\) as specified in line 12 of[Algorithm1](https://arxiv.org/html/2609.25471#alg1)\. For example,p=0\.5p=0\.5corresponds in expectation to alternating training—a supervised update every other round on average—whilep=1p=1applies server training every round andp=0p=0disables it entirely\. This server update onℒ\\mathcal\{L\}is crucial to reduce instability of SSFL as will be shown in[Section5](https://arxiv.org/html/2609.25471#S5)\.

To reflect the slow training process of FL, we constrain FL steps to be no more than66k\. Also, due to large discrepancy in audio duration per sample, instead of running local updates to the fixed number of epochs, we run it to the fixed number of local steps:160160for𝒰:\\mathcal\{U\}:LS, and2020for the other datasets unless specified otherwise\. Other hyperparameters such as communication rounds, cohort size, local steps, EMA decay, and the exact batch\-size values are listed in[Section10\.3](https://arxiv.org/html/2609.25471#S10.SS3)\.

\(a\)Baseline comparison\.
\(b\)LS dev\-clean and dev\-other\.

Figure 1:Panel \(a\): comparison of the online teacher against existing SSFL baselines—Static PL, FedNST, Rao et al\., and Diao et al\. for ASR—on LS dev\-clean and dev\-other\. Panel \(b\): online\-teacher advantage holds on both LS dev\-clean \(left\) and dev\-other \(right\) test sets, with online consistently outperforming global throughout training\.

## 5Pseudo\-Label Generation

This section compares the performance of four pseudo\-label teachers—global, online, local EMA, and transitioning; their definitions are summarized in[Table2](https://arxiv.org/html/2609.25471#S3.T2)\.

### 5\.1Global vs\. Online Teacher

Prior SSFL work generates pseudo\-labels from either a fixed initial teacherϕ0\\phi\_\{0\}\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)or a global EMA teacherϕt\\phi\_\{t\}updated each round\([Diao et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib10);[Rao et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib37)\); none consider an online teacher whose state evolves within the round, reflecting a prevailing assumption—made explicit by[Diao et al\. \(2022\)](https://arxiv.org/html/2609.25471#bib.bib10);[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.25471#bib.bib58)—that local updates would drift from the global model and destabilize training\. We revisit this choice first in the in\-domain regime \(LS100→\\rightarrowLS860\), where the seed and target share the same distribution and any local drift moves the model along in\-distribution directions\. All experiments in this section use SpecAugment on client audio and a periodic server\-side supervised pass onℒ\\mathcal\{L\}determined by server training probabilityp=0\.5p=0\.5\. We defer analysis of the server training frequency to[Section5\.2](https://arxiv.org/html/2609.25471#S5.SS2)\.

The right panel of[Figure1\(b\)](https://arxiv.org/html/2609.25471#S4.F1.sf2)plots target\-domain WER over FL rounds for both global and online teachers \(Eqs\.[1](https://arxiv.org/html/2609.25471#E1),[2](https://arxiv.org/html/2609.25471#E2); defined in[Section3\.1](https://arxiv.org/html/2609.25471#S3.SS1)\)\. The online teacher reaches lower WER than the global teacher from the first few FL rounds onward, and the gap persists rather than vanishing as training proceeds \(Δ​WER=2\.88\\Delta\\text\{WER\}=2\.88in the final round\)\. This improvement is not at the cost of regression on the source domain\. WER on dev\-clean \(left in[Figure1\(b\)](https://arxiv.org/html/2609.25471#S4.F1.sf2)\) stays withinΔ​WER=0\.5\\Delta\\text\{WER\}=0\.5of the global teacher\.

The mechanism is direct: the client’s local updates refine the model along useful in\-distribution directions, so the online teacher’s pseudo\-labels reflect what the student has just learned on the current client’s audio, while the global teacher’s pseudo\-labels remain stale for the rest of the round\. Unlike FedSwitch\([Zhao et al\., 2024](https://arxiv.org/html/2609.25471#bib.bib58)\)—which switches adaptively between a global teacher and a teacher\-student EMA but relies on per\-client state—the unconditional online teacher already suffices in\-domain under non\-persistent clients\.

\(a\)Intermediate local steps\.\(b\)Local EMA decay\.\(c\)Server training probability\.
Figure 2:Mechanism validations on LS100→\\rightarrowLS860\. \(a\) Local\-model WER at intermediate local steps\{0,10,40,160\}\\\{0,10,40,160\\\}\. Step 0 corresponds to the global teacher \(no local updates yet\) and the last step to the online teacher; WER drops monotonically as local training proceeds, validating that within\-round adaptation drives the online teacher’s gain\. \(b\) WER over FL rounds for the global, online, and local\-EMA teachers across EMA decay rates \(LS dev\-other\)\. Local EMA underperforms online at every decay rate, suggesting the latest adaptation step matters more than smoothed local state\. \(c\) Bar chart comparing online\-teacher WER across server training probabilitiespp\. Sufficient probability \(p≥0\.2p\\geq 0\.2\) keeps the online teacher stable\. Lower \(or zero\) probability causes divergence as the global model drifts from the labeled distribution\.##### Local adaptation drives the gain\.

We investigate why the online teacher performs better than the global teacher by tracking how a client’s predictions improve over its local training within a single FL round\. At local SGD steps\{0,10,40,160\}\\\{0,10,40,160\\\}—where step00is the server parameters at the start of the round before any local update—we take each client’s local model and decode a shared client validation set, computing the WER against its references\. The local\-trajectory WER at stepssis the mean of these per\-client WERs across the cohort\. Step00therefore corresponds to the global teacher and the final step to the online teacher\.[Figure2\(a\)](https://arxiv.org/html/2609.25471#S5.F2.sf1)shows that this WER drops monotonically as local training proceeds \(00step→\\rightarrow160160step\), so the local model adapts to the client distribution within the round\. This adaptation is exactly why the online teacher outperforms the global teacher: it generates pseudo\-labels from the adapted local model, whereas the global teacher uses the unadapted server model from the start of the round\.

##### The latest local step matters most\.

We investigate further if interpolation between global and online teachers improves WER compared to the online teacher\. To this end, we consider a more generic teacher definition based on EMA decay, which we call the local EMA teacher\. More formally, the local EMA teacher is updated at local stepτ\\tauas:θ¯t,τ=γ​θ¯t,τ−1\+\(1−γ\)​θt,τ\\bar\{\\theta\}\_\{t,\\tau\}=\\gamma\\bar\{\\theta\}\_\{t,\\tau\-1\}\+\(1\-\\gamma\)\\theta\_\{t,\\tau\}\. Whenγ=0\.0\\gamma=0\.0, it is equivalent to the online teacher whereas whenγ=1\.0\\gamma=1\.0, it is equivalent to the global teacher\.

[Figure2\(b\)](https://arxiv.org/html/2609.25471#S5.F2.sf2)provides the comparison betweenγ=\{0\.0,0\.9,0\.99,0\.999,1\.0\}\\gamma=\\\{0\.0,0\.9,0\.99,0\.999,1\.0\\\}\. It demonstrates that decreasingγ\\gamma\(closer to the online teacher\) gradually improves WER, showing that fast adaptivity of the teacher seems to be an important factor\.

### 5\.2Online Teacher Requires Frequent Server Training

The online teacher’s stability hinges on*frequent*server updates: without periodic re\-injection of supervised signal from the labeled server datasetℒ\\mathcal\{L\}, the global model drifts from the labeled distribution across rounds, pseudo\-label quality degrades cumulatively, and the online teacher eventually diverges\.

[Figure2\(c\)](https://arxiv.org/html/2609.25471#S5.F2.sf3)sweeps the server training probabilitypp\(defined in[Section4\.3](https://arxiv.org/html/2609.25471#S4.SS3)\) on LS100→\\rightarrowLS860 acrossp∈\{0\.0,0\.1,…,0\.5\}p\\in\\\{0\.0,0\.1,\\ldots,0\.5\\\}: atp=0p=0training diverges, atp=0\.1p=0\.1it remains unstable, resulting in WER=12\.8=12\.8\. However, atp≥0\.2p\\geq 0\.2the online teacher converges to the1010–1111range in WER, withp=0\.3p=0\.3giving the best converged WER of10\.2710\.27\. A moderately frequent server update thus suffices: once pseudo\-label drift is suppressed, anyp≥0\.2p\\geq 0\.2works with no significant additional benefit from raisingppfurther\. Therefore, we fixp=0\.5p=0\.5throughout the paper unless stated otherwise\. This sensitivity toppmotivates the deeper investigation of server update strategy in[Section6](https://arxiv.org/html/2609.25471#S6)\.

### 5\.3Cross\-Domain Robustness

To check whether the in\-domain finding of[Section5\.1](https://arxiv.org/html/2609.25471#S5.SS1)extends beyond LS100→\\rightarrowLS860, we evaluate the same global vs\. online comparison on two cross\-domain targets: LS100→\\rightarrowCV \(read audiobooks→\\rightarrowcrowd\-sourced read speech\) and LS100→\\rightarrowTed \(read audiobooks→\\rightarrowprepared talks\)\.

[Figure3](https://arxiv.org/html/2609.25471#S5.F3)shows that the online teacher continues to outperform the global teacher on both pairs of server and client datasets\. On the target validation sets in the right figures of[Figure3](https://arxiv.org/html/2609.25471#S5.F3)\(a\)\(b\), global vs\. online show WER on CV\-dev:39\.939\.9vs\.37\.737\.7and WER on Ted\-dev:12\.612\.6vs\.11\.611\.6\. Note that this improvement is not at the cost of regression on the source data as shown in the left figures\. The advantage is therefore not specific to the in\-domain regime—it persists across the kinds of cross\-domain shifts that arise when the server’s labeled seed and the clients’ unlabeled data are drawn from different speaking styles and recording conditions\.

\(a\)LS100→\\rightarrowCV: dev\-clean and en\-dev\.

\(b\)LS100→\\rightarrowTed: dev\-clean and ted\-dev\.

Figure 3:Online vs\. global teacher at fixed seed \(LS100\) on two cross\-domain targets\. Panel \(a\): LS100→\\rightarrowCV \(read audiobooks→\\rightarrowcrowd\-sourced read speech\), evaluated on the source\-domain \(LS dev\-clean, left\) and target\-domain \(CV en\-dev, right\) test sets\. Panel \(b\): LS100→\\rightarrowTed \(read audiobooks→\\rightarrowprepared talks\), evaluated on the source\-domain \(LS dev\-clean, left\) and target\-domain \(Ted ted\-dev, right\) test sets\. Online continues to outperform global on both pairs, confirming that the in\-domain finding of[Section5\.1](https://arxiv.org/html/2609.25471#S5.SS1)extends to cross\-domain shifts at fixed seed\.
### 5\.4Seed Strength and Transitioning Teacher

##### Seed strength\.

Stronger seeds are intended to approximate more mature hypothetical deployment settings, where the labeled server dataset would have accumulated over time\.A seed\-strength sweep both tests how the benefit of online teacher generalizes to less\-favorable conditions for online, and characterizes the regime where online teacher’s local adaptation still pays off\.

[Figure4](https://arxiv.org/html/2609.25471#S5.F4)shows that as the seed grows stronger \(LS100, 360, 600, 960→\\rightarrowCV\), the gap between online and global narrows further:Δ​WER\\Delta\\text\{WER\}shrinks from2\.522\.52at LS100 to0\.890\.89at LS360 \(34\.2734\.27vs\.33\.3833\.38\),1\.161\.16at LS600 \(30\.0530\.05vs\.28\.8928\.89\), and−0\.61\-0\.61at LS960 \(20\.9420\.94vs\.21\.5521\.55\)\. A strong seed already produces good pseudo\-labels, leaving less room for the online teacher’s local adaptation to improve them\. With the strongest seed \(LS960→\\rightarrowCV\), the online teacher can even underperform the global teacher in early rounds, before the model has had time to adapt to the target domain\. In those early rounds, the strong seed’s global predictions on the unseen target distribution still beat the online teacher’s, which relies on a model that has barely begun to adapt\.

##### Transitioning teacher\.

This early\-round disadvantage motivates the*transitioning teacher*, which uses the global teacher for the firstrrrounds and switches to the online teacher thereafter\. The right panel of[Figure5](https://arxiv.org/html/2609.25471#S5.F5)\(a\) \(CV en\-dev, target\-domain\) shows that the transitioning teacher is almost identical untilr=2r=2k and switches to the online teacher afterwards, and this transition provides significant improvement over the global teacher byΔ​WER=1\.20\\Delta\\text\{WER\}=1\.20\(global:20\.920\.9vs\. transition:19\.719\.7\)\.

As stated before, this is because in early rounds the model is far from the target distribution, so the online teacher would produce unreliable pseudo\-labels while the global teacher provides stable ones\. Once the model has adapted enough to the target domain, the online teacher’s local adaptivity becomes more beneficial\. The optimal crossover roundr∗r^\{\*\}depends on seed strength and domain gap\. For example, with a weaker seed model \(LS100→\\rightarrowCV\), this transition is never beneficial, i\.e\.,r∗=0r^\{\*\}=0\. Also, when there is little to no domain gap, e\.g\., LS100→\\rightarrowLS860, the transition is again not beneficial\.

Figure 4:Best WER across seed strengths \(LS100/360/600/960→\\rightarrowCV\) for the global, online, and transitioning teacher\. The online teacher’s advantage over global narrows as the seed grows stronger; the transitioning teacher \(global→\\rightarrowonline at roundrr\) recovers the gap and matches or beats the better of the two pure strategies at every seed strength\.
\(a\)LS960→\\rightarrowCV: dev\-other and en\-dev\.
\(b\)LS→\\rightarrowTed\.\(c\)Ted→\\rightarrowLS\.
Figure 5:Panel \(a\): training curves on LS960→\\rightarrowCV for global, online, and transitioning teachers on the source\-domain \(LS dev\-other, left\) and target\-domain \(CV en\-dev, right\) test sets: at this strong seed, online slightly underperforms global, while the transitioning teacher \(global→\\rightarrowonline at roundrr\) improves over both pure strategies on the target\. Panels \(b,c\): direction\-asymmetric breakage of the online teacher: LS→\\rightarrowTed converges with online outperforming global, whereas Ted→\\rightarrowLS diverges rapidly\. The asymmetry motivates the investigation in[Section6\.3](https://arxiv.org/html/2609.25471#S6.SS3)\.Take\-aways•Contrary to the common assumption that a per\-client*online*teacher is too unstable to use, it matches or beats the broadcast*global*teacher both in\-domain and under domain shift\.•The online teacher’s advantage comes from its within\-round adaptation to each client’s distribution, but the same adaptivity makes it prone to instability, so it holds only when paired with frequent enough server training on the seed data\.•The online teacher’s advantage over the global teacher shrinks as the seed grows stronger \(Δ​WER=2\.52\\Delta\\text\{WER\}=2\.52at LS100→\\rightarrowCV down to−0\.61\-0\.61at LS960→\\rightarrowCV\), but a*transitioning*teacher \(global→\\rightarrowonline at roundrr\) recovers the gap and matches or beats both at every seed strength\.

## 6Server\-Update Strategy

[Section5](https://arxiv.org/html/2609.25471#S5)established that the online teacher matches or beats the global teacher across both in\-domain and cross\-domain settings, including significant domain shift \(e\.g\., LS→\\rightarrowCV\)\. However, the online teacher does not always work\. This section analyzes its failure modes and how to mitigate them\.

\(a\)Audio duration distribution\.
\(b\)Ted/Ted\-Long→\\rightarrowLS\.

Figure 6:Audio duration does not explain the asymmetric breakage\. \(a\) Ted\-Long matches LS in audio duration distribution, while original Ted is markedly shorter\. \(b\) When the seed is trained on Ted\-Long instead of Ted, the Ted\-Long→\\rightarrowLS direction still diverges similarly to Ted→\\rightarrowLS—ruling out audio duration as the driver of the asymmetry\.### 6\.1Online Teacher Breaks Asymmetrically

##### Asymmetric breakage\.

We extend the online\-vs\.\-global comparison from[Section5](https://arxiv.org/html/2609.25471#S5)to two new pairs\. On LS→\\rightarrowTed \([Figure5\(b\)](https://arxiv.org/html/2609.25471#S5.F5.sf2)\), the online teacher outperforms the global teacher as expected\. On the reverse Ted→\\rightarrowLS \([Figure5\(c\)](https://arxiv.org/html/2609.25471#S5.F5.sf3)\), however, the online teacher diverges while the global teacher converges normally\. The same asymmetric pattern holds for LS→\\rightarrowCV vs\. CV→\\rightarrowLS\.

The asymmetry rules out a simple “large domain gap = failure” intuition\. In fact, the larger gap \(LS→\\rightarrowCV\) still converges with online beating global, while the smaller gap \(Ted→\\rightarrowLS\) diverges\. The cause must lie in the*direction*—specifically, in the properties of the source data used for server training\. We first rule out seed\-training\-side factors before tracing the failure to the FL\-stage server\-training step itself\. We share these negative results to help readers avoid similar pitfalls\.

### 6\.2Investigating Instability: Seed Training

The server data participates in training at two stages—seed training before FL and server\-training updates during FL—so we first check whether the seed model itself is the source of instability\. There are three aspects of seed training we validate as potential cause of instability of the online teacher in the Ted→\\rightarrowLS direction: i\) audio duration of the seed corpus, ii\) SpecAug strength, and iii\) overfitting of the seed model\.

##### Audio duration\.

[Figure6\(a\)](https://arxiv.org/html/2609.25471#S6.F6.sf1)shows that Ted’s audio duration distribution \(mean: 6\.07 s, median: 5\.94 s\) is markedly shorter than LS’s \(mean: 12\.30 s, median: 13\.79 s\)\. We hypothesize this gap drives the asymmetric breakage: a seed model trained on Ted’s short clips may struggle when further trained on LS’s longer clips during FL with pseudo\-labels\. Because online\-teacher pseudo\-labels are noisier than global\-teacher ones, the combined effect of pseudo\-label noise and longer audio could destabilize the online teacher even where the global teacher remains stable\. To validate this, we construct Ted\-Long, a Ted variant that concatenates up to two clips while keeping each example under 40 seconds, yielding a duration distribution \(mean: 12\.08 s, median: 11\.79 s\) close to LS’s\.

[Figure6\(b\)](https://arxiv.org/html/2609.25471#S6.F6.sf2)shows that Ted\-Long→\\rightarrowLS still diverges similarly to Ted→\\rightarrowLS, even though the Ted\-Long seed performs better on Ted dev\. This rules out audio duration as the driver of the instability\.

##### SpecAug strength\.

We hypothesize that strong SpecAug during seed training compounds the instability in the Ted→\\rightarrowLS direction: aggressive masking on the source corpus can cause improper feature learning, producing significantly wrong predictions on out\-of\-domain data such as LS\. Because online\-teacher pseudo\-labels are noisier than global\-teacher ones, the combined effect of wrong seed predictions and pseudo\-label noise could destabilize the online teacher even where the global teacher remains stable\.

By default, we use the standard SpecAugment recipe: 2 frequency masks of maximum widthWf=30W\_\{f\}=30bins and 10 time masks of maximum widthWt=50W\_\{t\}=50frames \(capped at0\.1×L0\.1\\times Lfor utterance lengthLL\)\. To probe this hypothesis, we sweep the maximum mask widths\(Wf,Wt\)∈\{\(30,30\),\(10,30\),\(30,10\),\(10,10\)\}\(W\_\{f\},W\_\{t\}\)\\in\\\{\(30,30\),\(10,30\),\(30,10\),\(10,10\)\\\}during Ted seed training and evaluate Ted→\\rightarrowLS in FL with each resulting seed as shown in[Figure7\(a\)](https://arxiv.org/html/2609.25471#S6.F7.sf1)\.

[Figure7\(a\)](https://arxiv.org/html/2609.25471#S6.F7.sf1)shows an unexpected pattern: stronger SpecAug during seed training is actually beneficial to the global teacher, while the effect on the online teacher is less clear\. A narrower time mask \(Wt=10W\_\{t\}=10\) appears to help: the online teacher does not diverge at\(Wf,Wt\)=\(30,10\)\(W\_\{f\},W\_\{t\}\)=\(30,10\)\. However, it still diverges at\(10,10\)\(10,10\), and at\(10,30\)\(10,30\)divergence is avoided but WER does not improve over the initial seed model\. Overall, this rules out strong SpecAug during seed training as the major driver of the instability\.

##### Overfitting of the seed model\.

Lastly, we check whether overfitting of the seed model to the source corpus contributes to the instability\. We train seed models with varying numbers of steps∈\{400​k,500​k,600​k\}\\in\\\{400\\text\{k\},500\\text\{k\},600\\text\{k\}\\\}on Ted and evaluate Ted→\\rightarrowLS in FL with each seed\.[Figure7\(b\)](https://arxiv.org/html/2609.25471#S6.F7.sf2)shows that the seed model trained for fewer steps \(400k\) diverges later than the default 800k checkpoint, but it does not improve over the initial seed model and still diverges eventually\. This suggests that while overfitting may exacerbate the instability, it is not the root cause of the online teacher’s divergence in the Ted→\\rightarrowLS direction\.

In the end, none of these factors explains the asymmetric breakage: in every configuration, Ted→\\rightarrowLS continues to diverge while LS→\\rightarrowTed converges\. The cause must therefore lie in the FL\-stage server update, which we examine in[Section6\.3](https://arxiv.org/html/2609.25471#S6.SS3)\.

\(a\)SpecAug strength\.\(b\)Early seed checkpoint\.\(c\)Server update on LS→\\rightarrowCV\.
Figure 7:*Seed\-training and server\-update factors in the Ted→\\rightarrowLS instability\.*\(a\) SpecAug strength sweep over\(Wf,Wt\)∈\{\(30,30\),\(10,30\),\(30,10\),\(10,10\)\}\(W\_\{f\},W\_\{t\}\)\\in\\\{\(30,30\),\(10,30\),\(30,10\),\(10,10\)\\\}during Ted seed training, evaluated as Ted→\\rightarrowLS: stronger SpecAug benefits the global teacher, while the online teacher shows no clear pattern\. \(b\) Seed checkpoint sweep over\{400​k,500​k,600​k\}\\\{400\\text\{k\},500\\text\{k\},600\\text\{k\}\\\}training steps: the 400k checkpoint delays but does not avoid divergence\. \(c\) With the seed fixed on LS→\\rightarrowCV, the online teacher diverges when the server update uses Ted but converges when it uses LS\.

### 6\.3Investigating Instability: Server Update

An evidence we have for the server update being the source of instability is that the divergence happens depending on which source dataset is used for server update, even when the seed model is fixed\. For example,[Figure7\(c\)](https://arxiv.org/html/2609.25471#S6.F7.sf3)shows that when we fix the seed model to be the same checkpoint trained on LS and client data is CV, the online teacher diverges when the server update is performed on Ted but converges when it is performed on LS\. It is worth noting that the global teacher converges in both cases, which further supports the hypothesis that the server update step is the source of instability for the online teacher\.

Now, the question is what about the server update step causes the instability and why it only affects the online teacher but not the global teacher\. To investigate this, we analyze the server update step in detail and identify two levers that govern its stability: the SpecAug strength and batch size applied during the server update\.

\(a\)WER on LS dev\-other\.\(b\)Errors: Strong SpecAug\.\(c\)PL Sensitivity \(CER\)\.\(d\)Manifold Dist\. \(FID\)\.
Figure 8:Comparison of strong and moderate SpecAug on Ted→\\rightarrowLS\. \(a\) WER on LS dev\-other over FL rounds for moderate vs\. strong SpecAug\. \(b\) Insertion and deletion error rates over FL rounds\. Strong SpecAug yields high deletion with near\-zero insertion at some FL step due to instability of training\. \(c\) PL sensitivity, measured as the FID between pseudo\-label distributions on original vs\. slightly\-perturbed inputs\. Strong SpecAug increases sensitivity, evidencing the input\-level ill\-conditioning of the server pass\. \(d\) Manifold distance, measured as the FID between the model’s hidden\-state distribution and the seed\-trained reference\. Strong SpecAug pushes the model further off\-manifold\.#### 6\.3\.1SpecAug Strength

##### Setup\.

As a reminder, the default SpecAug parameters are 2 frequency masks of maximum widthWf=30W\_\{f\}=30bins and 10 time masks of maximum widthWt=50W\_\{t\}=50frames \(capped at0\.1×L0\.1\\times Lfor utterance lengthLL\)\. We refer to this default setting as*strong*SpecAug, and we refer to a milder setting of\(Wf,Wt\)=\(15,25\)\(W\_\{f\},W\_\{t\}\)=\(15,25\)as*moderate*SpecAug\.

##### WER and error\-type signature\.

We compare moderate and strong SpecAug on the server update for Ted→\\rightarrowLS in[Figure8\(a\)](https://arxiv.org/html/2609.25471#S6.F8.sf1), which shows strong SpecAug diverges while moderate SpecAug converges to a better WER than the seed model\. We further investigate the mechanism of this divergence by analyzing the error types\.[Figure8\(b\)](https://arxiv.org/html/2609.25471#S6.F8.sf2)shows that strong SpecAug yields high deletion error and near\-zero insertion error starting from around3\.83\.8k FL steps\. This error signature implies that after around 3\.8k FL steps, the model outputs nothing or very little meaningless transcription\.

##### Pseudo\-label sensitivity\.

We conjecture that this is because client model predictions become highly inconsistent across clients under strong SpecAug, which produces noisy pseudo\-labels that are dominated by the randomness of mask sampling rather than the content of the input\. This inconsistency is evidenced by the pseudo\-label sensitivity to small input perturbations as shown in[Figure8\(c\)](https://arxiv.org/html/2609.25471#S6.F8.sf3)\. We measure “pseudo\-label \(PL\) sensitivity” as the average pairwise character error rate \(CER\) betweenkkindependently SpecAug\-perturbed decodings of the same utterance, averaged over 256 fixed evaluation utterances\. Lower values indicate more stable predictions under augmentation noise\. We observe that the PL sensitivity suddenly increases around 3\.8k FL steps under strong SpecAug, coinciding with the onset of high deletion error\.

##### Manifold drift\.

However, we still do not understand how strong SpecAug leads to this inconsistency\. We hypothesize that strong SpecAug adds strong noise to server updates, which pushes the model off the manifold of the seed data and into a regime where the model’s predictions are highly unstable under input perturbations\. To validate this, we measure the distance between the model’s hidden\-state distribution at the current step and at initialization using the Fréchet distance, shown in[Figure8\(d\)](https://arxiv.org/html/2609.25471#S6.F8.sf4)\. Specifically, we pass 512 fixed samples from the server dataset through the model, extract penultimate\-layer features \(mean\-pooled over time\), and fit a Gaussian\(μt,Σt\)\(\\mu\_\{t\},\\Sigma\_\{t\}\)at steptt\. The manifold distance at stepttis the Fréchet distance to the reference distribution\(μ0,Σ0\)\(\\mu\_\{0\},\\Sigma\_\{0\}\)at initialization:

FIDt=‖μ0−μt‖2\+Tr⁡\(Σ0\+Σt−2​\(Σ0​Σt\)1/2\)\.\\mathrm\{FID\}\_\{t\}=\\\|\\mu\_\{0\}\-\\mu\_\{t\}\\\|^\{2\}\+\\mathrm\{Tr\}\\\!\\left\(\\Sigma\_\{0\}\+\\Sigma\_\{t\}\-2\(\\Sigma\_\{0\}\\Sigma\_\{t\}\)^\{1/2\}\\right\)\.\(4\)Higher values indicate larger representation drift from the seed\-trained reference\.[Figure8\(d\)](https://arxiv.org/html/2609.25471#S6.F8.sf4)shows that the manifold of strong SpecAug drifts much further from the seed model reference than that of moderate SpecAug, supporting the hypothesis that strong SpecAug pushes the model into a regime of unstable predictions\.

\(a\)WER on LS dev\-other\.\(b\)PL Sensitivity \(CER\)\.\(c\)Manifold Distance \(FID\)\.
Figure 9:*Batch size sweep at fixed moderate SpecAug on Ted→\\rightarrowLS*\(dev\-other\)\. \(a\) WER over FL rounds on LS dev\-other\. Small batch converges along a noisier trajectory to a higher WER, while larger batches converge to a low, stable WER\. \(b\) PL sensitivity, measured as the FID between pseudo\-label distributions on original vs\. slightly\-perturbed inputs\. Small batch yields higher sensitivity, evidencing the optimizer\-level ill\-conditioning of the server pass\. \(c\) Manifold distance, measured as the FID between the model’s hidden\-state distribution and the seed\-trained reference\. Small batch pushes the model further off\-manifold\.

#### 6\.3\.2Batch Size

##### Setup\.

Now that we know strong SpecAug causes server\-training instability, we fix SpecAug at moderate and sweep the server\-training batch size from small \(120\) to large \(3840\)\. Throughout the paper, the batch size is measured in seconds of audio rather than number of samples: a batch of 120 means we select samples whose total duration sums to approximately 120 seconds per server\-update step\. We use duration\-based batching—rather than a fixed number of samples—because audio length varies substantially across utterances \(typically 2–30 seconds\), and this keeps per\-step compute consistent\. When batch size is too large to fit in memory, we accumulate gradients over multiple forward passes before performing the optimizer step\. Again, we analyze the effect of batch size on the Ted→\\rightarrowLS setting\.

##### WER signature\.

Unlike strong SpecAug, small batch does not cause divergence—training converges in all cases \([Figure9\(a\)](https://arxiv.org/html/2609.25471#S6.F9.sf1)\)\. Instead, it converges along a noisier trajectory to a noticeably higher final WER, and increasing the batch progressively recovers stable training, with batch 3840 fully recovering under moderate SpecAug\. This is initially counterintuitive: because we clip the gradient to magnitude 1, a larger batch does not anchor the update with a larger step—it only sharpens the*direction*of the update\. The benefit of large batch is therefore entirely a reduction in gradient\-direction noise, which points to an optimizer\-level mechanism rather than the input\-level one behind strong SpecAug\.

##### Pseudo\-label sensitivity\.

As with SpecAug, we probe this instability through PL sensitivity \([Figure9\(b\)](https://arxiv.org/html/2609.25471#S6.F9.sf2)\)\. Small batch yields higher sensitivity to input perturbations, evidencing that the noisy server gradient leaves client predictions less stable—the optimizer\-level counterpart of the input\-level ill\-conditioning caused by strong SpecAug\.

##### Manifold drift\.

The same noise pushes the model off the seed\-data manifold\. Using the manifold distance defined above,[Figure9\(c\)](https://arxiv.org/html/2609.25471#S6.F9.sf3)shows that small batch drifts further from the seed\-trained reference than large batch, mirroring the SpecAug signature but driven at the optimizer level rather than the input level\.

![Refer to caption](https://arxiv.org/html/2609.25471v1/figures/ablation_batch/speaker_tsne_ls_vs_ted.png)\(a\)Speaker\-embedding t\-SNE\.
\(b\)Batch\-size ablation\.
Figure 10:\(a\) t\-SNE of speaker embeddings for LS860 and Ted samples, visualizing the relative dispersion of the two corpora\. \(b\) Batch\-size ablation on Ted→\\rightarrowLS \(LS dev\-other\)\. Each panel fixes the server batch size \(120120,480480,38403840\) and overlays WER over FL rounds for three sampling schemes:*random*,*mutually exclusive*, and*overlapped*\. Random diverges or performs worse at every batch size, mutually exclusive diverges only at the smallest batch, and overlapped converges throughout\.
##### Dispersion, not overlap\.

A larger batch could stabilize training for either of two reasons: it lowers the variance of the server gradient by averaging over more samples \(a*dispersion*effect\), or it is more likely to include samples that resemble the client data, making the supervised update a better proxy for the client distribution \(an*overlap*effect\)\. To separate the two, we vary a batch’s dispersion and its overlap with the client distribution independently\. This separation is possible because the seed \(Ted\) and client \(LS860\) data form distinct clusters in a t\-SNE of their speaker embeddings as shown in[Figure10\(a\)](https://arxiv.org/html/2609.25471#S6.F10.sf1), which lets us construct batches that either avoid or include client\-like samples\. Concretely, we draw server batches in three ways:*random*\(high dispersion, some overlap\),*mutually exclusive*\(low dispersion, no overlap with the client distribution\), and*overlapped*\(low dispersion, high overlap\)\. The two hypotheses make opposite predictions\. If overlap were the driver, the mutually exclusive scheme—which contains no client\-like samples—should diverge while the random and overlapped schemes converge\. If dispersion were the driver, the two low\-dispersion schemes should converge while the high\-dispersion random scheme diverges\.

[Figure10\(b\)](https://arxiv.org/html/2609.25471#S6.F10.sf2)compares the three schemes at batch sizes120120,480480, and38403840\. The random scheme diverges or trains noisily at every batch size, the mutually exclusive scheme diverges only at the smallest batch and converges at the moderate and large batches, and the overlapped scheme converges at every batch size\. Because random is the only high\-dispersion scheme and the only one to fail across the board, dispersion is the primary driver of instability\. Overlap plays a secondary role: among the two low\-dispersion schemes, the overlapped one is more stable, so the absence of client\-like samples does contribute to instability—but only at the smallest batch, and it is recovered by a moderately larger batch\. Overlap therefore matters less than dispersion and can be compensated by batch size\.

Both levers: data augmentation \(SpecAug\) and batch size feed the same server\-update noise, so neither alone suffices: the online teacher matches or exceeds global only when both are jointly in their stable range—which we verify across domain\-shift pairs next\.

Figure 11:Cross\-dataset WER comparison across seed→\\rightarrowclient grid\. Each cell reports \{online, global\} teacher with and without stabilization applied\. The stabilized transitioning teacher \(large\-batch, moderate\-SpecAug server training\) wins on every converging pair\.

### 6\.4Stabilizers Generalize Across Domain\-Shift Pairs

We now test whether the two stabilizers identified on Ted→\\rightarrowLS in[Section6\.3](https://arxiv.org/html/2609.25471#S6.SS3)generalize across multiple cross\-domain pairs\. We extend the analysis to four cross\-domain pairs: Ted→\\rightarrowLS, Ted→\\rightarrowCV, CV10→\\rightarrowLS, and CV10→\\rightarrowTed in[Figure11](https://arxiv.org/html/2609.25471#S6.F11)\. For each pair we cross the pseudo\-label source \(global or online\) with a stabilizer \(a48004800batch with SpecAug scale0\.50\.5\) or without a stabilizer \(a480480batch with SpecAug scale1\.01\.0\), yielding four configurations per pair\.

Without the stabilizers, the online teacher diverges except for CV10→\\rightarrowTed, while the global teacher stays stable but plateaus at a higher WER than the stabilized online teacher\. With the stabilizers, all online teachers converge\. Interestingly, the global teachers also benefit from the stabilizers, with significant WER improvements over the non\-stabilized global teacher across all pairs\.

Together, these results show that the two stabilizers are not specific to the Ted→\\rightarrowLS direction\. They generalize across all four cross\-domain pairs, restoring convergence to the online teacher wherever it previously diverged and lowering WER for the global teacher as well\. Stabilized server training is thus a prerequisite for the online teacher under domain shift, regardless of the particular source and client corpora\. We carry this stabilized configuration into the cross\-dataset comparison against existing SSFL methods in[Section7](https://arxiv.org/html/2609.25471#S7)\.

Take\-aways•The online teacher can break, and the failure is*asymmetric*: Ted→\\rightarrowLS diverges while the reverse LS→\\rightarrowTed converges, so it is directional rather than a matter of domain\-gap magnitude\.•The breakage stems not from seed training but from the server update: the noise it injects—input\-level from strong augmentation, optimizer\-level from a small batch—pushes the model off\-manifold and makes pseudo\-labels inconsistent across clients, degrading WER or, in the extreme, diverging\.•This instability is stabilized simply, by reducing the data augmentation strength and increasing the batch size, which helps both the online and global teachers and restores convergence even on the pairs where the online teacher previously diverged\.

## 7Comprehensive Analysis and Practical Guidance

### 7\.1Comprehensive Comparison

##### Experimental setting\.

[Table3](https://arxiv.org/html/2609.25471#S7.T3)reports the best WER each method reaches across the seed→\\rightarrowclient grid \(11 pairs in total\): the supervised upper bound \(GT labels\), three existing SSFL methods \(Static PL, FedNST\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)444FedNST trains on the full labeled server data each round, making it1\.31\.3–2\.3×2\.3\\timesslower than the other methods \([Section11\.3](https://arxiv.org/html/2609.25471#S11.SS3)\)\. Within the shared77\-day wall\-clock limit it may not complete the full step budget\.[Table3](https://arxiv.org/html/2609.25471#S7.T3)entries reflect truncated runs\., and Rao et al\.\([Rao et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib37)\)\) as well as the global and online teachers, each with and without the stabilizer\. To track whether the trained model forgets the seed domain, we also report the source\-domain WER in parentheses, which is meaningful only for cross\-domain pairs\. For the in\-domain pairs, the seed and client are disjoint subsets of the same corpus so the client never sees the labeled seed: LS100 \(seed\) with LS860 \(client\) for LS, and CV10 \(seed\) with CV90 \(client\) for CV\.

To mimic realistic FL training scenarios, we set the cohort size to roughly 1–3% of the clients in each dataset: 64 for LS and Ted, 1024 for CV, and 256 for Fisher, whose client pools number about 2k, 2k, 35k, and 11k respectively\. We use 20 or 40 local steps, choosing the better per pair on the validation set since the optimum varies across methods, except for LS100→\\rightarrowLS860 where we use 160\.

The stabilizer is a large\-batch server update \(gradient accumulation over 10 default batches\)\. For the online teacher we pair it with moderate SpecAugment \(scale 0\.4 or 0\.5\), whereas for the global teacher we keep the default mask widthsWf=30W\_\{f\}=30andWt=50W\_\{t\}=50, which we find works best for it\.

##### Results\.

We observe that[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)is the strongest existing SSFL method, but a large gap to the GT\-label upper bound remains, e\.g\.,7\.187\.18vs\.14\.3414\.34on CV10→\\rightarrowTed and10\.5310\.53vs\.42\.5642\.56on CV10→\\rightarrowFisher\. With the stabilizer, both teachers improve over[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)on most pairs—the global teacher on 8 of 11 and the online teacher on 9 of 11\. Without the stabilizer, the online teacher diverges on 5 of 11 pairs, but with it the online teacher converges on all 11, with the largest gains on high\-dispersion \(Ted\) sources where it recovers from divergence \(Ted→\\rightarrowLS: Div\. vs\.11\.3411\.34, Ted→\\rightarrowFisher: Div\. vs\.18\.5118\.51\)\.

Overall, both the stabilized online and global teachers substantially outperform existing SSFL methods, closing much of the gap to the GT\-label upper bound\. The stabilized online teacher is the strongest configuration for in\-domain pairs, where it adapts quickly to the client distribution and its added noise matters little given the heavy source–client overlap\. Under domain shift, the two teachers reach similar WER, but we favor the global teacher for its robustness: its pseudo\-labels stay fixed within an FL round, making it less sensitive to the SpecAugment strength and batch size than the online teacher\.

Method*Client: LS**Client: Ted**Client: CV**Client: Fisher*TedLS100CV10LSCV10LSTedCV10LSTedCV10GT Labels6\.73\(7\.72\)7\.397\.56\(24\.64\)6\.46\(8\.00\)7\.18\(29\.31\)11\.39\(8\.85\)14\.60\(8\.68\)18\.2810\.06\(8\.21\)10\.22\(7\.58\)10\.53\(30\.57\)Static PL14\.92\(7\.36\)17\.7130\.11\(25\.14\)10\.22\(7\.63\)33\.57\(28\.66\)21\.25\(8\.86\)24\.90\(8\.64\)23\.4431\.35\(7\.90\)29\.98\(7\.78\)Div\.\(28\.68\)FedNST16\.59\(7\.34\)18\.4330\.70\(19\.54\)11\.26\(6\.86\)36\.68\(24\.75\)22\.49\(6\.39\)23\.12\(6\.83\)23\.8231\.58\(6\.58\)28\.61\(6\.81\)Div\.\(24\.55\)Rao*et al*\.11\.12\(7\.50\)15\.2617\.91\(24\.97\)8\.21\(7\.61\)14\.34\(28\.76\)16\.12\(8\.28\)19\.80\(8\.58\)22\.3219\.82\(7\.95\)19\.35\(7\.81\)42\.56\(28\.72\)Global teacher12\.62\(7\.91\)12\.5514\.84\(21\.59\)8\.82\(7\.31\)13\.88\(25\.84\)17\.55\(8\.60\)20\.45\(7\.75\)24\.7821\.06\(7\.42\)23\.10\(7\.59\)33\.02\(27\.13\)\+ stabilizer11\.42\(6\.70\)13\.1513\.40\(18\.43\)8\.08\(7\.34\)11\.09\(22\.27\)15\.82\(6\.88\)17\.89\(6\.88\)22\.2120\.78\(6\.98\)25\.58\(6\.68\)29\.44\(22\.64\)Online teacherDiv\.9\.44Div\.9\.12\(7\.54\)15\.84\(27\.18\)17\.24\(8\.48\)Div\.25\.6524\.18\(7\.39\)Div\.Div\.\+ stabilizer11\.34\(6\.75\)9\.2513\.50\(18\.53\)7\.79\(7\.40\)11\.07\(22\.53\)15\.58\(6\.79\)18\.85\(6\.92\)21\.8421\.45\(7\.19\)18\.51\(6\.90\)27\.43\(26\.05\)Table 3:WER \(%\) on the test sets across eleven \(seed, client\) pairs, grouped by client dataset \(top header\), with each column a seed dataset\. Each cell shows the target\-domain WER on top and, where available, the source\-domain WER in parentheses below\.*Div\.*denotes divergence\.

### 7\.2Transitioning Teacher

Figure 12:Teachers with the stabilizerWe further validate the effectiveness of the transitioning teacher, which switches from the global to the online teacher at a fixed roundrr, on the LS960→\\rightarrowCV pair with the stabilizers applied\. Unlike[Figure5](https://arxiv.org/html/2609.25471#S5.F5)where the stabilizers are not applied, here we apply them to all teachers\.

As[Figure12](https://arxiv.org/html/2609.25471#S7.F12)shows, the stabilizers shrink the gap between the global and online teachers considerably \(without the stabilizers, global:20\.9420\.94vs\. online:21\.5521\.55, and with the stabilizers, global:18\.6518\.65vs\. online:18\.7318\.73WER\)\. The transitioning teacher \(switching atr=2000r=2000\) still outperforms both, though by a smaller margin than in[Figure5](https://arxiv.org/html/2609.25471#S5.F5)because the stabilizers already improve the global and online teachers significantly \(transitioning:19\.719\.7without the stabilizers vs\.18\.2518\.25with them\)\. In summary, the transitioning teacher yields a small but consistent improvement by recovering the online teacher’s early\-round disadvantage, capturing the global teacher’s early stability and the online teacher’s late adaptivity\. Since this margin is modest once the stabilizer is applied, we present it as an analysis of that early\-round effect rather than as a general recommendation\.

\(a\)Cohort size\.\(b\)Local steps\.
Figure 13:Ablation studies on LS→\\rightarrowCV for the global and online teachers, both with stabilizers\. \(a\) Effect of cohort size\{64,256,1024\}\\\{64,256,1024\\\}at 20 local steps\. \(b\) Effect of the number of local steps\{20,40,80,160\}\\\{20,40,80,160\\\}at cohort size 64\.
### 7\.3Ablation Studies

##### Cohort size\.

Cohort size—the number of clients sampled per round—is a primary knob of FL, and its effect on SSFL may not match its effect on supervised FL, since there is additional noise coming from pseudo\-labeling and server updates\. We ablate it for the global and online teachers, both with stabilizers, over cohort sizes\{64,256,1024\}\\\{64,256,1024\\\}on LS→\\rightarrowCV at 20 local steps in[Figure13\(a\)](https://arxiv.org/html/2609.25471#S7.F13.sf1)\. We observe that as cohort size increases, the performance increases for both global and online teachers\. There is no significant difference in trend between global and online teachers\. It is generally advisable to maximize cohort size if possible\.

##### Local training steps\.

Another important knob for FL training is the number of local steps each client takes per round\. It governs the communication–computation trade\-off, but in SSFL, it also controls how much the online teacher adapts within a round, and hence the quality of the pseudo\-labels it produces\.

We ablate it for the global and online teachers, both with stabilizers, over\{20,40,80,160\}\\\{20,40,80,160\\\}local steps on LS→\\rightarrowCV at cohort size 64 in[Figure13\(b\)](https://arxiv.org/html/2609.25471#S7.F13.sf2)\. For both global and online teachers, 40 local steps are the best choice\. If the number of local steps is too large, the performance degrades for both teachers, for different reasons\. As the number of local steps increases, pseudo\-labels produced by the global teacher become stale, which leads the local model to update in a suboptimal direction\. On the other hand, the online teacher, although it does not produce stale pseudo\-labels, overfits to the client data too much, which induces severe drift between client models and destabilizes aggregation\.

##### Data augmentation and dropout\.

We generate pseudo\-labels without SpecAugment or dropout to give the teacher a clean signal\. During local training, however, we apply both SpecAugment and dropout to prevent overfitting to the pseudo\-labels, following common practice\. It is unclear how SpecAugment and dropout interact with the server update probabilityppand affect the online teacher, so we ablate them in[Table4](https://arxiv.org/html/2609.25471#S7.T4)on both in\-domain \(LS100→\\rightarrowLS860\) and cross\-domain \(LS→\\rightarrowCV\) settings at cohort size 64 with 20 local steps\.

As in[Figure2\(c\)](https://arxiv.org/html/2609.25471#S5.F2.sf3), atp=0\.0p=0\.0the online teacher diverges in both settings\. Atp=0\.2p=0\.2it tends to converge in both, though it still diverges in some cases, and when it does converge its WER is lower than atp=0\.5p=0\.5\. This is because a lowerppmakes training rely more on the client data than the server data, which improves WER\. Loweringpptoo far, however, can cause divergence due to client drift\. Dropout as regularization partly offsets this risk: althoughp=0\.2p=0\.2generally diverges without dropout in\-domain, it converges with dropout\.

We observe a similar trend for SpecAugment strength in\-domain: a higher scale improves WER but raises the chance of divergence, which dropout again mitigates\. For cross\-domain, the effect of SpecAugment is less pronounced as the optimal scale is lower than in\-domain\. When SpecAugment is too strong, the online teacher diverges in both settings whereas when it is too weak, the WER can be higher\. Therefore, we recommend that practitioners jointly tune SpecAugment strength and the server update probabilityppalong with dropout as regularization to minimize WER in their own applications\.

without dropout ppScale0\.50\.50\.20\.20\.00\.01\.01\.0Div\.Div\.Div\.0\.30\.314\.1Div\.Div\.0\.00\.015\.311\.7Div\.

with dropout ppScale0\.50\.50\.20\.20\.00\.01\.01\.011\.010\.3Div\.0\.30\.312\.310\.8Div\.0\.00\.012\.910\.4Div\.

\(a\)In\-domain: LS100→\\rightarrowLS860\.
without dropout ppScale0\.50\.50\.20\.20\.00\.01\.01\.029\.8Div\.Div\.0\.30\.324\.621\.7Div\.0\.00\.024\.921\.7Div\.

with dropout ppScale0\.50\.50\.20\.20\.00\.01\.01\.026\.2Div\.Div\.0\.30\.324\.521\.8Div\.0\.00\.024\.721\.6Div\.

\(b\)Cross\-domain: LS→\\rightarrowCV\.

Table 4:Scale for SpecAugment strength \(1\.0/0\.3/0\.01\.0/0\.3/0\.0, rows\)×\\timesserver update probabilitypp\(0\.5/0\.2/0\.00\.5/0\.2/0\.0, columns\) grids of final WER \(%\), for in\-domain and cross\-domain settings, without and with dropout\.*Div\.*denotes divergence\.

### 7\.4When to Use What: Practical Guidelines

These guidelines are derived from the public\-benchmark experiments of[Section4](https://arxiv.org/html/2609.25471#S4);[Section8](https://arxiv.org/html/2609.25471#S8)discusses their scope and limitations\.Our findings translate into a simple recipe that requires no per\-pair hyperparameter tuning, driven by two properties a practitioner can assess up front: whether the setting is in\-domain or cross\-domain, and the dispersion and audio duration of the server \(seed\) data\.

##### Pseudo\-label source\.

The choice of teacher follows from the overlap between the server and client data, which can usually be judged from domain knowledge\.555When domain knowledge is insufficient, this overlap can be estimated directly from data: the embedding\-based similarity measure of[Section12](https://arxiv.org/html/2609.25471#S12)separates in\-domain from cross\-domain pairs and serves as a quantitative proxy for it\.When the two are highly overlapped \(in\-domain\), we recommend the online teacher, whose within\-round adaptation to the client distribution improves the pseudo\-labels while its added noise matters little\. Under domain shift, the online and global teachers reach similar WER on most pairs, but we recommend the global teacher as the safer default\. Because its pseudo\-labels stay fixed within a round, it is more robust to the choice of SpecAugment strength and batch size, whereas the online teacher requires careful tuning of these settings to avoid divergence\.

##### Server update settings\.

We recommend keeping the server update as a simple supervised batch step on the seed data, applied on its own between FL rounds, rather than folding it into the client aggregation as FedNST\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)and Rao*et al*\.\([Rao et al\., 2023](https://arxiv.org/html/2609.25471#bib.bib37)\)do \([Section11](https://arxiv.org/html/2609.25471#S11)\)\. This plain update is simpler to implement and tune\. It is also no less effective in our comparison, where neither folded\-update baseline reaches a lower WER \([Table3](https://arxiv.org/html/2609.25471#S7.T3)\)\.

Two further settings then control its stability and should be matched to the source data\. First, the server batch size should scale with the dispersion of the server data: a higher\-dispersion source needs a larger batch to keep the variance of the server gradient low enough to avoid divergence \([Section6\.3](https://arxiv.org/html/2609.25471#S6.SS3)\)\. This adjustment helps both teachers\. Second, the SpecAugment strength interacts with the teacher choice\. For the online teacher, it should be calibrated to the audio duration of the source, since longer clips tolerate stronger masking whereas shorter clips require weaker masking to keep the effective mask coverage below the divergence threshold\. The global teacher, by contrast, is far less sensitive to SpecAugment strength and generally benefits from strong masking\.

Take\-aways•Across the 11 seed–client pairs, the global and online teachers*with the stabilizer*substantially outperform existing SSFL methods \(Static PL, FedNST, and Rao et al\.\), closing much of the gap to fully\-supervised FL\.•No single teacher dominates: the online teacher wins in\-domain and on most cross\-domain pairs, but under domain shift the two are close, so the more robust global teacher—less sensitive to the data augmentation and batch\-size settings—is the safer choice\.•These findings yield a simple recipe that needs no per\-pair tuning: \(1\) use the online teacher in\-domain and the global teacher under domain shift, \(2\) keep a plain supervised batch update on the server, \(3\) scale the server batch size with the dispersion of the server data, and \(4\) calibrate the SpecAugment strength \(to audio duration for the online teacher\)\.

## 8Discussion

### 8\.1Limitations

This work demonstrates the advantage of different pseudo\-label teachers and the impact of server\-side stabilizers\. However, these findings may not transfer to settings that differ from our experimental setup\. We outline the main limitations in scope below so that practitioners can judge how well our conclusions apply to their own settings\.

##### Language and dataset scope\.

All four corpora we study \(LS, Ted, CV, and Fisher\) are English\. Languages that are tonal, low\-resource, or written in non\-Latin scripts may exhibit different SpecAugment\-coverage thresholds and a different online\-vs\.\-global trade\-off\.

##### Architecture and training scope\.

Our experiments use a single architecture, a Transformer encoder with a CTC head\. Other families—RNN\-T\([Graves, 2012](https://arxiv.org/html/2609.25471#bib.bib12)\), attention encoder–decoder\([Chan et al\., 2016](https://arxiv.org/html/2609.25471#bib.bib7)\), and Conformer\([Gulati et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib14)\)—may express the two server\-training failure modes \(divergence under strong SpecAugment versus higher\-WER instability under small batches\) differently\. We also decode greedily without a language model, whereas beam search with an external LM is standard in deployment, so whether the online\-teacher advantage compounds or diminishes under LM\-fused decoding is untested\. Finally, we fix the optimizer pair to SGD on clients and LAMB\([You et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib54)\)on the server\. Adam\-class server optimizers such as FedAdam\([Reddi et al\., 2021](https://arxiv.org/html/2609.25471#bib.bib38)\)may interact differently with the SpecAugment strength\.

##### Federated learning setup\.

All experiments use public benchmark corpora \(LibriSpeech, TED\-LIUM, Common Voice, Fisher\); no real user data, production traffic, or production\-system telemetry informed any part of this study, and the reported system parameters \(cohort sizes, compute budgets, communication\-round counts, optimizer settings\) are not intended to characterize, and have not been validated against, any production configuration\.We simulate clients by sampling cohorts from static datasets, so the non\-stationary data, mid\-round dropout, network failures, and continuous churn of real deployments are not modeled\. In addition, the recommended stabilizer configuration with a large batch, e\.g\., a batch of 480 accumulated over 10 steps, requires substantial server\-side compute\. Although server compute is often not a bottleneck in practice, practitioners with constrained server budgets may not be able to apply the recipe directly\.

##### Baseline comparison scope\.

Our comparison against FedNST and Rao*et al*\. \([Table3](https://arxiv.org/html/2609.25471#S7.T3)\) is end\-to\-end rather than a controlled ablation of the server\-update mechanism, since these baselines also differ from our configuration in their pseudo\-label teacher\. We therefore recommend keeping the server update as a simple supervised batch step applied on its own between FL rounds \([Section7\.4](https://arxiv.org/html/2609.25471#S7.SS4)\) primarily for its simplicity, and leave a controlled comparison to future work\.

### 8\.2Future Work

##### Privacy guarantee\.

Our recipe leaves the FL privacy guarantee intact: the online teacher generates pseudo\-labels locally with no extra communication, and the larger server batches use only the server’s own labeled data\. We have not, however, tested it under differential privacy\. Since DP\-FL adds per\-layer gradient clipping and noise while our stabilizer relies on large, low\-variance server updates, the two may work against each other—DP noise could undo the variance reduction the stabilizer provides, so batch size and SpecAugment strength may need re\-tuning under a DP budget\. Quantifying this interaction, and establishing formal guarantees under a specified threat model, is a natural next step\.

##### Better decoding and data filtering\.

We decode greedily and apply no pseudo\-label filtering—two axes orthogonal to the teacher and stabilizer choices studied here that could be layered on top\. Beam search with an external language model is standard in deployment, and confidence\-based filtering of pseudo\-labels \(held off throughout,[Section3\.2](https://arxiv.org/html/2609.25471#S3.SS2)\) is a common SSL ingredient\. Whether the online\-teacher advantage compounds with LM\-fused decoding, and whether confidence thresholds compose with the transitioning teacher and the stabilizers, are open questions worth pursuing\.

## 9Conclusion

We studied semi\-supervised federated learning \(SSFL\) for automatic speech recognition \(ASR\), a setting in which pseudo\-label errors compound across the output sequence and across training rounds into divergence and leave a large gap to fully\-supervised FL\. We showed that closing this gap turned on two coupled design axes—the teacher that generates the pseudo\-labels and the anchor that stabilizes training through server\-side updates on labeled data\.

On theteacher axis, the best choice was dynamic rather than fixed: a per\-client online teacher matched or beat the broadcast global teacher once stabilized, and a transitioning teacher \(global→\\rightarrowonline\) outperformed either as the seed model grew stronger\. On theanchor axis, interleaving server training on the labeled seed between FL rounds was a prerequisite for stability, and this anchor—more than the seed model itself—governed convergence\. The two axes proved inseparable: the aggressive teacher choices paid off only once the anchor stabilized training, which we found was highly sensitive to SpecAugment strength and batch size, the two levers that control server\-update gradient noise and hence pseudo\-label consistency\. How much stabilization was needed proved domain\-dependent, governed by the dispersion of the seed data and its overlap with the client data\.

Together, these findings yielded practical guidelines that stabilized an otherwise brittle procedure and closed much of the gap to fully\-supervised FL: applied correctly, the recipe improved WER over the strongest prior SSFL method by20\.8%20\.8\\%in\-domain and10\.0%10\.0\\%cross\-domain on average \(on 9 of 11 pairs\), and by roughly48%48\\%over a naive static pseudo\-labeling baseline in\-domain \(17\.7→9\.317\.7\\rightarrow 9\.3WER\)\.

## Acknowledgments

We thank David Grangier, Skyler Seto, Amar Subramanya, and Russ Web for essential feedback on the paper and Apple infrastructure team for assistance with developing scalable, fault tolerant code\.

## References

- Abadi et al\. \(2016\)Martin Abadi, Andy Chu, Ian Goodfellow, H\. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang\.Deep Learning with Differential Privacy\.In*ACM SIGSAC Conference on Computer and Communications Security \(CCS\)*, 2016\.
- Acar et al\. \(2021\)Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N\. Whatmough, and Venkatesh Saligrama\.Federated Learning Based on Dynamic Regularization\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Aldeneh et al\. \(2025\)Zakaria Aldeneh, Takuya Higuchi, Jee\-Weon Jung, Li\-Wei Chen, Stephen Shum, Ahmed Hussen Abdelaziz, Shinji Watanabe, Tatiana Likhomanenko, and Barry\-John Theobald\.Speaker\-IPL: Unsupervised Learning of Speaker Characteristics with i\-Vector based Pseudo\-Labels\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2025\.[10\.1109/ICASSP49660\.2025\.10887848](https://doi.org/10.1109/ICASSP49660.2025.10887848)\.
- Aldeneh et al\. \(2026\)Zakaria Aldeneh, Skyler Seto, Maureen de Seyssel, Jie Chi, Zijin Gu, Takuya Higuchi, Jee\-weon Jung, Shinji Watanabe, David Grangier, Barry\-John Theobald, and Tatiana Likhomanenko\.Which Data Matter? Embedding\-Based Data Selection for Speech Recognition\.*arXiv preprint arXiv:2603\.05819*, 2026\.
- Azam \(2022\)Sheikh Shams Azam\.*Towards Privacy and Communication Efficiency in Distributed Representation Learning*\.PhD thesis, Purdue University, 2022\.
- Azam et al\. \(2023\)Sheikh Shams Azam, Tatiana Likhomanenko, Martin Pelikan, and Jan Silovsky\.Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End\-to\-End ASR\.In*IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\)*, 2023\.
- Chan et al\. \(2016\)William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals\.Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2016\.
- Chen et al\. \(2022\)Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Michael Zeng, Xiangzhan Yu, and Furu Wei\.WavLM: Large\-Scale Self\-Supervised Pre\-Training for Full Stack Speech Processing\.*IEEE Journal of Selected Topics in Signal Processing*, 16\(6\):1505–1518, 2022\.
- Desplanques et al\. \(2020\)Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck\.ECAPA\-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2020\.
- Diao et al\. \(2022\)Enmao Diao, Jie Ding, and Vahid Tarokh\.SemiFL: Semi\-Supervised Federated Learning for Unlabeled Clients with Alternate Training\.In*Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- Fallah et al\. \(2020\)Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar\.Personalized Federated Learning: A Meta\-Learning Approach\.In*Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Graves \(2012\)Alex Graves\.Sequence Transduction with Recurrent Neural Networks\.*arXiv preprint arXiv:1211\.3711*, 2012\.
- Graves et al\. \(2006\)Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber\.Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks\.In*International Conference on Machine Learning \(ICML\)*, 2006\.
- Gulati et al\. \(2020\)Anmol Gulati, James Qin, Chung\-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang\.Conformer: Convolution\-Augmented Transformer for Speech Recognition\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2020\.
- Guliani et al\. \(2021\)Dhruv Guliani, Francoise Beaufays, and Giovanni Motta\.Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2021\.
- Higuchi et al\. \(2021\)Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori\.Momentum Pseudo\-Labeling for Semi\-Supervised Speech Recognition\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2021\.
- Hilmes et al\. \(2024\)Benedikt Hilmes, Nick Rossenbach, and Ralf Schlüter\.On the Effect of Purely Synthetic Training Data for Different Automatic Speech Recognition Architectures\.In*Proc\. Synthetic Data’s Transformative Role in Foundational Speech Models*, pp\. 46–50, 2024\.[10\.21437/SynData4GenAI\.2024\-10](https://doi.org/10.21437/SynData4GenAI.2024-10)\.
- Hu et al\. \(2026\)Bo Hu, Xingjian Lu, Luhan Wang, and Xiaoying Tu\.Fed\-ARPL: Adaptive and Reciprocal Prototype Learning for Semi\-Supervised Federated Learning, 2026\.URL[https://openreview\.net/forum?id=JcFyHkngnr](https://openreview.net/forum?id=JcFyHkngnr)\.Withdrawn submission to ICLR 2026\.
- Jeong et al\. \(2024\)Hyejun Jeong, Shiqing Ma, and Amir Houmansadr\.A Survey on Federated Unlearning: Challenges and Opportunities\.*arXiv preprint arXiv:2403\.02437*, 2024\.
- Kairouz et al\. \(2021\)Peter Kairouz, H\. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al\.Advances and Open Problems in Federated Learning\.*Foundations and Trends in Machine Learning*, 14\(1–2\):1–210, 2021\.
- Karimireddy et al\. \(2020\)Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh\.SCAFFOLD: Stochastic Controlled Averaging for Federated Learning\.In*International Conference on Machine Learning \(ICML\)*, 2020\.
- Laine & Aila \(2017\)Samuli Laine and Timo Aila\.Temporal Ensembling for Semi\-Supervised Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2017\.
- Lee \(2013\)Dong\-Hyun Lee\.Pseudo\-Label: The Simple and Efficient Semi\-Supervised Learning Method for Deep Neural Networks\.In*ICML Workshop on Challenges in Representation Learning*, 2013\.
- Li et al\. \(2024\)Jingyang Li, Jiachun Pan, Vincent Y\. F\. Tan, Kim\-Chuan Toh, and Pan Zhou\.Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning\.*arXiv preprint arXiv:2410\.11206*, 2024\.
- Li et al\. \(2020\)Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith\.Federated Optimization in Heterogeneous Networks\.In*Conference on Machine Learning and Systems \(MLSys\)*, 2020\.
- Likhomanenko et al\. \(2021\)Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert\.slimIPL: Language\-Model\-Free Iterative Pseudo\-Labeling\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2021\.
- Likhomanenko et al\. \(2023\)Tatiana Likhomanenko, Ronan Collobert, Navdeep Jaitly, and Samy Bengio\.Continuous Soft Pseudo\-Labeling in ASR\.In*International Conference on Learning Representations \(ICLR\)*, 2023\.
- Liu et al\. \(2021\)Xinyi Liu, Linghui Zhu, Shu\-Tao Xia, Yong Jiang, and Xue Yang\.GDST: Global Distillation Self\-Training for Semi\-Supervised Federated Learning\.In*IEEE Global Communications Conference \(GLOBECOM\)*, 2021\.
- Malaviya et al\. \(2023\)Shubham Malaviya, Manish Shukla, Pratik Korat, and Sachin Lodha\.FedFAME: A Data Augmentation Free Framework Based on Model Contrastive Learning for Federated Semi\-Supervised Learning\.In*ACM Symposium on Applied Computing \(SAC\)*, pp\. 1114–1121, 2023\.[10\.1145/3555776\.3577613](https://doi.org/10.1145/3555776.3577613)\.
- McMahan et al\. \(2017\)Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas\.Communication\-Efficient Learning of Deep Networks from Decentralized Data\.In*International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, 2017\.
- Mehmood et al\. \(2022\)Haaris Mehmood, Agnieszka Dobrowolska, Karthikeyan Saravanan, and Mete Ozay\.FedNST: Federated Noisy Student Training for Automatic Speech Recognition\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2022\.
- Mishchenko et al\. \(2022\)Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtárik\.ProxSkip: Yes\! Local Gradient Steps Provably Lead to Communication Acceleration\! Finally\!In*International Conference on Machine Learning \(ICML\)*\. PMLR, 2022\.
- Ogun et al\. \(2025\)Sewade Ogun, Vincent Colotte, and Emmanuel Vincent\.An Exhaustive Evaluation of TTS\- and VC\-Based Data Augmentation for ASR\.*arXiv preprint arXiv:2503\.08954*, 2025\.
- Park et al\. \(2019\)Daniel S\. Park, William Chan, Yu Zhang, Chung\-Cheng Chiu, Barret Zoph, Ekin D\. Cubuk, and Quoc V\. Le\.SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2019\.
- Park et al\. \(2020\)Daniel S\. Park, Yu Zhang, Ye Jia, Wei Han, Chung\-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V\. Le\.Improved Noisy Student Training for Automatic Speech Recognition\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2020\.
- Pelikan et al\. \(2025\)Martin Pelikan, Sheikh Shams Azam, Vitaly Feldman, Jan Silovsky, Kunal Talwar, Christopher G\. Brinton, and Tatiana Likhomanenko\.Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping\.In*Neural Information Processing Systems \(NeurIPS\)*, 2025\.
- Rao et al\. \(2023\)Milind Rao, Gopinath Chennupati, Gautam Tiwari, Anit Kumar Sahu, Anirudh Raju, Ariya Rastrow, and Jasha Droppo\.Federated Self\-Learning with Weak Supervision for Speech Recognition\.In*IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, 2023\.
- Reddi et al\. \(2021\)Sashank J\. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H\. Brendan McMahan\.Adaptive Federated Optimization\.In*International Conference on Learning Representations \(ICLR\)*, 2021\.
- Reimers & Gurevych \(2019\)Nils Reimers and Iryna Gurevych\.Sentence\-BERT: Sentence Embeddings Using Siamese BERT\-Networks\.In*Empirical Methods in Natural Language Processing \(EMNLP\)*, 2019\.
- Robbins & Monro \(1951\)Herbert Robbins and Sutton Monro\.A Stochastic Approximation Method\.*The Annals of Mathematical Statistics*, 1951\.
- Sohn et al\. \(2020\)Kihyuk Sohn, David Berthelot, Chun\-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D\. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel\.FixMatch: Simplifying Semi\-Supervised Learning with Consistency and Confidence\.In*Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Song et al\. \(2024\)Zixing Song, Xiangli Yang, Yifei Zhang, Xinyu Fu, Zenglin Xu, and Irwin King\.A Systematic Survey on Federated Semi\-Supervised Learning\.In*International Joint Conference on Artificial Intelligence \(IJCAI\)*, 2024\.
- Su et al\. \(2024\)Hsuan Su, Hua Farn, Fan\-Yun Sun, Shang\-Tse Chen, and Hung\-yi Lee\.Task Arithmetic Can Mitigate Synthetic\-to\-Real Gap in Automatic Speech Recognition\.In*Empirical Methods in Natural Language Processing \(EMNLP\)*, pp\. 8905–8915, 2024\.
- Tarvainen & Valpola \(2017\)Antti Tarvainen and Harri Valpola\.Mean Teachers Are Better Role Models: Weight\-Averaged Consistency Targets Improve Semi\-Supervised Deep Learning Results\.In*Neural Information Processing Systems \(NeurIPS\)*, 2017\.
- Truong et al\. \(2021\)Nguyen Truong, Kai Sun, Siyao Wang, Florian Guitton, and Yike Guo\.Privacy Preservation in Federated Learning: An Insightful Survey from the GDPR Perspective\.*Computers & Security*, 110:102402, 2021\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N\. Gomez, Łukasz Kaiser, and Illia Polosukhin\.Attention Is All You Need\.In*Neural Information Processing Systems \(NeurIPS\)*, 2017\.
- Wang et al\. \(2025\)Haozhao Wang, Shengyu Wang, Jiaming Li, Hao Ren, Xingshuo Han, Wenchao Xu, Shangwei Guo, Tianwei Zhang, and Ruixuan Li\.BSemiFL: Semi\-Supervised Federated Learning via a Bayesian Approach\.In*International Conference on Machine Learning \(ICML\)*, 2025\.
- Wang et al\. \(2020\)Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H\. Vincent Poor\.Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization\.In*Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- Wang et al\. \(2023a\)Jiaqi Wang, Shenglai Zeng, Zewei Long, Yaqing Wang, Houping Xiao, and Fenglong Ma\.Knowledge\-Enhanced Semi\-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoT\.In*SIAM International Conference on Data Mining \(SDM\)*, 2023a\.
- Wang et al\. \(2023b\)Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie\.FreeMatch: Self\-Adaptive Thresholding for Semi\-Supervised Learning\.In*International Conference on Learning Representations \(ICLR\)*, 2023b\.
- Xie et al\. \(2020\)Qizhe Xie, Minh\-Thang Luong, Eduard Hovy, and Quoc V\. Le\.Self\-Training with Noisy Student Improves ImageNet Classification\.In*IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 2020\.
- Xu et al\. \(2020\)Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert\.Iterative Pseudo\-Labeling for Speech Recognition\.In*Conference of the International Speech Communication Association \(INTERSPEECH\)*, 2020\.
- Yang et al\. \(2023\)Nan Yang, Dong Yuan, Charles Z\. Liu, Yongkun Deng, and Wei Bao\.FedIL: Federated Incremental Learning from Decentralized Unlabeled Data with Convergence Analysis\.*arXiv preprint arXiv:2302\.11823*, 2023\.
- You et al\. \(2020\)Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho\-Jui Hsieh\.Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes\.In*International Conference on Learning Representations \(ICLR\)*, 2020\.
- Zhang et al\. \(2021a\)Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki\.FlexMatch: Boosting Semi\-Supervised Learning with Curriculum Pseudo Labeling\.In*Neural Information Processing Systems \(NeurIPS\)*, 2021a\.
- Zhang et al\. \(2020\)Yu Zhang, James Qin, Daniel S\. Park, Wei Han, Chung\-Cheng Chiu, Ruoming Pang, Quoc V\. Le, and Yonghui Wu\.Pushing the Limits of Semi\-Supervised Learning for Automatic Speech Recognition\.In*NeurIPS SAS Workshop*, 2020\.
- Zhang et al\. \(2021b\)Zhengming Zhang, Yaoqing Yang, Zhewei Yao, Yujun Yan, Joseph E\. Gonzalez, Kannan Ramchandran, and Michael W\. Mahoney\.Improving Semi\-Supervised Federated Learning by Reducing the Gradient Diversity of Models\.In*IEEE International Conference on Big Data \(BigData\)*, 2021b\.
- Zhao et al\. \(2024\)Jessica Zhao, Sayan Ghosh, Akash Bharadwaj, and Chih\-Yao Ma\.When Does the Student Surpass the Teacher? Federated Semi\-Supervised Learning with Teacher\-Student EMA\.*arXiv preprint arXiv:2301\.10114*, 2024\.

\\beginappendix

## 10Experimental Details

This appendix lists the full dataset, model, and federated\-learning configuration summarized in[Section4](https://arxiv.org/html/2609.25471#S4)\. Defaults apply throughout the paper unless a specific section notes that a value is varied\.These values reflect choices made for our public\-dataset simulations and should not be read as characterizing any production system’s configuration\.

### 10\.1Datasets and Splits

[Table5](https://arxiv.org/html/2609.25471#S10.T5)summarizes the four English ASR corpora used in our experiments, chosen to span a broad range of speaking styles and recording conditions—from clean read audiobooks to conversational telephone speech—so that our conclusions do not hinge on a single acoustic domain\. This appendix describes each corpus, how it is partitioned into server \(seed\) and client data, the seed and client subsets and domain\-shift pairs we evaluate, and the train/dev/test protocol\.

Table 5:Per\-corpus statistics\. Hours are computed from the per\-utterance durations of the audio used in our pipeline\. Speaker counts are approximate client\-pool sizes, since each speaker is treated as one federated client\. Min, median, and max utterance durations are in seconds\.CorpusSpeaking styleHoursSpeakersMin \(s\)Median \(s\)Max \(s\)LibriSpeech \(LS\)Read audiobooks960∼\\sim2k0\.813\.829\.7TED\-LIUM \(Ted\)TED talks452∼\\sim2k0\.15\.930\.3Common Voice \(CV\)Crowd\-sourced read1593∼\\sim35k1\.45\.613\.0FisherTelephone conversation1928∼\\sim11k0\.32\.4203\.0##### Corpora\.

LibriSpeech \(LS\)consists of read English audiobooks sampled at 16 kHz, yielding long, fluent utterances with few disfluencies\.TED\-LIUM \(Ted\)contains prepared but spontaneously delivered TED\-talk speech, with hesitations, restarts, and a single speaker per talk\.Common Voice \(CV\)is a large crowd\-sourced corpus of short read sentences recorded by many volunteers on heterogeneous consumer devices, giving wide accent and channel diversity\.Fishercomprises two\-party conversational English telephone speech sampled at 8 kHz, with short, disfluent, and often overlapping turns\. Together they cover read, spontaneous, crowd\-sourced, and telephone speech, which lets us study domain shift across genuinely different acoustic and linguistic conditions\.

##### Server and client partitioning\.

Each corpus is used either as a labeled server \(seed\) corpus or as an unlabeled client corpus, depending on the pair\. For client corpora, we partition the data by speaker and treat each speaker as a separatesimulatedfederated client\. For server corpora, we do not partition the data666The exception is our reproduction of[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37), for which we need rehearsal data partitioned by speaker ID\.and instead sample training batches uniformly at random\. In both roles, utterances longer than 30 s are filtered out before training \(max\_audio\_len\_s=30\.0\)\.

##### Seed and client subsets\.

We denote by LSxxanxx\-hour labeled subset of LS used as the seed corpus, with the disjoint remainder serving as the unlabeled client corpus\. For example, LS100→\\rightarrowLS860 pairs a 100 h seed with the remaining 860 h as clients, and the seed\-strength sweep in[Section5\.4](https://arxiv.org/html/2609.25471#S5.SS4)uses LS100, LS360, LS600, and LS960 seeds\. Similarly, CV10 and CV90 denote a 10%/90% split of CV into seed and client partitions\.

##### Domain\-shift pairs\.

We evaluate the eleven server→\\rightarrowclient pairs listed in[Table3](https://arxiv.org/html/2609.25471#S7.T3)\. They range from*in\-domain*pairs, where the seed and client are disjoint splits of the same corpus \(LS100→\\rightarrowLS860 and CV10→\\rightarrowCV90\), through*moderate*shifts between read and spontaneous speech \(LS↔\\leftrightarrowTed\), to*large*shifts onto crowd\-sourced or telephone speech \(e\.g\., LS→\\rightarrowCV at several seed strengths, CV10→\\rightarrowTed, and pairs involving Fisher\)\. Where possible we evaluate both directions of a shift \(e\.g\., LS→\\rightarrowTed and Ted→\\rightarrowLS\), which lets us test whether the domain gap acts symmetrically\.

##### Train/dev/test splits\.

We use the standard benchmark splits for each corpus\. For LS we use dev\-clean and dev\-other \(and test\-clean and test\-other for[Table3](https://arxiv.org/html/2609.25471#S7.T3)\), for CV the English dev and test sets, and for Ted its dev and test sets\. As noted in[Section4\.1](https://arxiv.org/html/2609.25471#S4.SS1), all analysis and hyperparameter\-tuning results are reported on the dev sets, and only the final comprehensive comparison in[Table3](https://arxiv.org/html/2609.25471#S7.T3)uses the held\-out test sets, which keeps the test data untouched during model development\.

### 10\.2Model and Training Hyperparameters

#### Model Parameters

Architecture\.The acoustic model is a Transformer encoder\([Vaswani et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib46)\)with 36 blocks, a hidden/embedding dimension of 768, 4 attention heads, and a feed\-forward \(MLP\) dimension of 3072, preceded by a convolutional front\-end \(kernel 7, stride 3\) that subsamples the input by a factor of 3\. The model has approximately 255M parameters\.

Audio features\.We extract 80\-dimensional log\-mel filterbank features using a 25 ms window and a 10 ms stride\. Utterances whose duration falls outside\[0​s,30​s\]\[0\\,\\text\{s\},30\\,\\text\{s\}\]are discarded \(max\_audio\_len\_s=30\.0\), and the maximum target length is capped at 400 characters\.

Loss\.The model is trained with the connectionist temporal classification \(CTC\) loss\([Graves et al\., 2006](https://arxiv.org/html/2609.25471#bib.bib13)\)\.

Tokenizer\.We use a character\-level tokenizer with an output vocabulary of 28 characters plus the CTC blank symbol\.

Decoding\.Decoding is greedy \(argmax over the CTC posteriors\) and uses no external language model\.

#### Training Parameters

Augmentation\.We apply SpecAugment\([Park et al\., 2019](https://arxiv.org/html/2609.25471#bib.bib34)\)with 2 frequency masks of maximum widthWf=30W\_\{f\}=30and 10 time masks of maximum widthWt=50W\_\{t\}=50, with the time\-mask width capped at a ratio of0\.10\.1of the utterance length and no mask averaging\. Augmentation is enabled from the first training step \(start\_saug=0\)\.

Dropout\.We use dropout rate of 0\.3 to train the seed model in the separate central run\. During FL training, Dropout is fixed to 0\.1 throughout, withdropout=layer\_dropout= 0\.1 to increase the capacity of the model following[Likhomanenko et al\. \(2021\)](https://arxiv.org/html/2609.25471#bib.bib26)\.

Client optimizer\.Clients optimize with SGD at a learning rate of 0\.2 for Ted sources and 0\.4 for LS sources, without any learning\-rate schedule\.

Server optimizer\.The server optimizes with LAMB\([You et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib54)\)using an exponential\-decay schedule with decay rate 0\.6\. The server learning rate is 0\.004 for most source→\\totarget pairs, with the exceptions of Ted→\\toLS and LS\-100→\\toLS\-860 \(0\.003\) and CV\-10→\\toCV\-90 \(0\.001\)\. The schedule is set relative to the total number of FL stepsTT: for most pairs \(T=3000T=3000\), decay begins atT/3T/3and the learning rate is scaled by0\.60\.6for every subsequentT/6T/6steps\. The two exceptions useT=2000T=2000: LS\-100→\\toLS\-860 begins decay atT/2T/2and is scaled by0\.60\.6everyT/8T/8steps, whereas CV\-10→\\toCV\-90 begins decay only at5​T/85T/8\.

Seed training\.The model is seeded by pretraining on the labeled server corpus in a separate central run\. This checkpoint is loaded to initialize both the student and the EMA teacher\.

### 10\.3Semi\-Supervised Federated Learning Hyperparameters

#### Federated Learning Parameters

Aggregation\.Throughout all experiments, client updates are aggregated using FedAvg\([McMahan et al\., 2017](https://arxiv.org/html/2609.25471#bib.bib30)\)with equal weights to all clients that participate\.

FL steps\.We runT=3000T=3000FL steps for most pairs, andT=2000T=2000for the in\-domain LS100→\\rightarrowLS860 and CV10→\\rightarrowCV90 pairs, where one FL step corresponds to a single round of client aggregation and, with probabilitypp, a server update\. The server learning\-rate schedule scales withTTas described above\.

Cohort size\.For the comprehensive comparison in[Table3](https://arxiv.org/html/2609.25471#S7.T3), we fix the cohort size per client corpus to roughly 1–3% of its client pool: 64 for LS and Ted, 1024 for CV, and 256 for Fisher\. We separately ablate the cohort size over\{64,256,1024\}\\\{64,256,1024\\\}in[Section7\.3](https://arxiv.org/html/2609.25471#S7.SS3)\.

Local steps\.Each client performsKKlocal steps per round\. By default we useK=20K=20or4040, whichever performs better on the validation set, except when the client corpus is LS where we useK=160K=160\. We ablateKKover\{20,40,80,160\}\\\{20,40,80,160\\\}\.

Server training\.In each round, the server performs a supervised pass over the seed corpus with probabilityp=0\.2p=0\.2\. The default server batch size is 120 s of audio for LS sources and 480 s for Ted sources, and high\-dispersion sources use10×10\\timesgradient accumulation \(server\_grad\_accum\_steps=10\)\.

#### Pseudo\-Labeling Parameters

EMA teacher\.The EMA teacher uses a decay rate ofλ=0\.99\\lambda=0\.99\(the global\-EMA default, which the student configs do not override\)\.

PL source\.Pseudo\-labels are produced by a global or online teacher \(cache\_update\_method=pre withuse\_ema\)\. When a transitioning teacher is used, the transition round isr≈2000r\\approx 2000\.

Unless otherwise stated, all experiments use the defaults above\. Each section in the main text explicitly notes which hyperparameters are varied\.

## 11Baseline Method Details

This appendix expands on how the existing SSFL for ASR baselines instantiate the two design axes of[Section3\.1](https://arxiv.org/html/2609.25471#S3.SS1): the pseudo\-label source and the server training strategy\. We writeℒ\\mathcal\{L\}for the labeled server corpus,𝒰i\\mathcal\{U\}\_\{i\}for the unlabeled data of clientii, andθ0\\theta\_\{0\}for the seed model trained onℒ\\mathcal\{L\}\.

### 11\.1FedNST

FedNST\([Mehmood et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib31)\)extends noisy student training\([Park et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib35)\)to the federated setting\. It first trains a seed modelθ0\\theta\_\{0\}on the labeled server dataℒ\\mathcal\{L\}, then runsTTfederated rounds\.

##### Pseudo\-label generation\.

Before federated training begins, every client generates pseudo\-labels once from the seed modelθ0\\theta\_\{0\}, decoding its unlabeled audio𝒰i\\mathcal\{U\}\_\{i\}with an external language model and beam search777As mentioned in[Section3](https://arxiv.org/html/2609.25471#S3), we do not apply any language model fusion or rescoring\., and caches the resulting transcripts on\-device for reuse in every subsequent round\. The teacher is therefore frozen at the seed, and no confidence filtering or class balancing is applied by default\. FedNST also considers regenerating pseudo\-labels each round from the latest global model, but this raises the wall\-clock cost of a run by roughly10×10\\timesfor a negligible change in WER, so the one\-time variant is used throughout\.

##### Server update\.

FedNST merges the federated aggregation and the server\-side supervised update into a single global step\. In roundtt, each sampled clientiitrains for several local epochs on its cached pseudo\-labels and returns a pseudo\-gradientgti=θt−θtig\_\{t\}^\{i\}=\\theta\_\{t\}\-\\theta\_\{t\}^\{i\}, whereθti\\theta\_\{t\}^\{i\}are its locally updated parameters\. The server aggregates these FedAvg\-style, weighted by the per\-client sample countnin\_\{i\},

gC=∑inin​gti,n=∑ini,g\_\{C\}=\\sum\_\{i\}\\frac\{n\_\{i\}\}\{n\}\\,g\_\{t\}^\{i\},\\qquad n=\\sum\_\{i\}n\_\{i\},\(5\)and in parallel performs a supervised pass on the labeled server dataℒ\\mathcal\{L\}to obtain a server pseudo\-gradientgSg\_\{S\}\. The two are combined by a weighted average and applied as one update,

g=α​gS\+\(1−α\)​gC,θt\+1=ServerOpt​\(θt,g\),g=\\alpha\\,g\_\{S\}\+\(1\-\\alpha\)\\,g\_\{C\},\\qquad\\theta\_\{t\+1\}=\\textsc\{ServerOpt\}\(\\theta\_\{t\},g\),\(6\)with mixing weightα=0\.5\\alpha=0\.5in their experiments \(we also use this value throughout the experiments\)\. Because the labeled pass contributes to every round, the seed signal re\-anchors training continuously rather than periodically\.

### 11\.2Rao et al\.

[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)study federated continual learning for an RNN\-T ASR model, where a paired teacher labels on\-device audio and the model is updated without ground\-truth transcripts\.

##### Pseudo\-label generation\.

Pseudo\-labels are produced each round by a paired teacherϕt\\phi\_\{t\}that is an exponential moving average \(EMA\) of the global student, refreshed everyuurounds,

ϕt=λ​ϕt−1\+\(1−λ\)​θtwhen​t≡0\(modu\),\\phi\_\{t\}=\\lambda\\,\\phi\_\{t\-1\}\+\(1\-\\lambda\)\\,\\theta\_\{t\}\\quad\\text\{when \}t\\equiv 0\\pmod\{u\},\(7\)and held fixed otherwise\. We setu=1u=1in our experiments, so the teacher is refreshed every round\. Unlike FedNST’s frozen seed teacher, this teacher tracks the evolving global model\. Each sampled device transcribes its unlabeled audio𝒰i\\mathcal\{U\}\_\{i\}withϕt\\phi\_\{t\}, filters out utterances of very low or very high confidence, and trains on the retained samples with audio augmentation\. Optionally, weak\-supervision signals such as alternate\-system NLU semantics or user feedback scores are folded in through a policy\-gradient loss, but these are orthogonal to the pseudo\-label source itself\.888In our comparison we apply neither the confidence filtering nor the weak supervision, isolating[Rao et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib37)’s pseudo\-label source and server training strategy\.

##### Server update\.

The labeled server data enters training as*rehearsal*on a set of cloud “pseudo\-devices”𝒮C\\mathcal\{S\}\_\{C\}\. These pseudo\-devices draw ground\-truth\-labeled dataℒ\\mathcal\{L\}and compute local updates exactly like real clients\. In roundtt, every participantkkin the union of the sampled real devices𝒮t\\mathcal\{S\}\_\{t\}and the cloud pseudo\-devices𝒮C\\mathcal\{S\}\_\{C\}returns a pseudo\-gradientgtk=θt−wtkg\_\{t\}^\{k\}=\\theta\_\{t\}\-w\_\{t\}^\{k\}, wherewtkw\_\{t\}^\{k\}are its locally updated parameters, and the server averages them in a single aggregation step,

θt\+1=ServerOpt​\(θt,1\|𝒮t∪𝒮C\|​∑k∈𝒮t∪𝒮Cgtk\)\.\\theta\_\{t\+1\}=\\textsc\{ServerOpt\}\\\!\\left\(\\theta\_\{t\},\\;\\frac\{1\}\{\|\\mathcal\{S\}\_\{t\}\\cup\\mathcal\{S\}\_\{C\}\|\}\\sum\_\{k\\in\\mathcal\{S\}\_\{t\}\\cup\\mathcal\{S\}\_\{C\}\}g\_\{t\}^\{k\}\\right\)\.\(8\)The supervised signal is therefore not a separate server pass but an additional group of clients folded into the same FedAvg aggregation, acting as a regularizer that mitigates catastrophic forgetting on the seed distribution\. In their experiments,\|𝒮C\|=40\|\\mathcal\{S\}\_\{C\}\|=40cloud pseudo\-devices are used alongside\|𝒮t\|=400\|\\mathcal\{S\}\_\{t\}\|=400sampled real devices per round, a rehearsal cohort of10%10\\%of the sampled clients\. Following this setup, we set the rehearsal cohort to10%10\\%of the FL cohort each round\.

Table 6:Per\-round, per\-GPU server\-update compute relative to a global/online \(ours\) step, for LS100→\\toLS860 \(C=32,K=160,G=16C\{=\}32,K\{=\}160,G\{=\}16\) and LS→\\toCV \(C=1024,K=20,G=64C\{=\}1024,K\{=\}20,G\{=\}64\)\.nS=\|ℒ\|/bSn\_\{S\}=\|\\mathcal\{L\}\|/b\_\{S\}is the batch count of one full source sweep,GGthe number of GPUs,CCthe cohort size, andKKthe local steps per client\. Batching is by audio duration, so batch count is a FLOP proxy; a shared server batch sizebSb\_\{S\}is assumed\. The per\-GPU client load isC​K/G=320C\\,K/G=320batches for both pairs\.Rel\. compute/stepMethodServer updateBatches/GPULS100→\\toLS860LS→\\toCVStatic PLnone00––Global/Online \(ours\)kksupervised stepskk1\.001\.001\.001\.00Rao*et al*\.rehearsal onρ​C\\rho Cclientsρ​nS/G\\rho\\,n\_\{S\}/G1\.001\.001\.111\.11FedNSTfull source sweepnS/Gn\_\{S\}/G1\.251\.252\.342\.34

### 11\.3Server\-Update Compute

Both client training and the server update are sharded across theGGGPUs, so the relevant quantity is the*per\-GPU*batch count: each round a GPU processesC​K/GC\\,K/Gclient batches \(KKlocal steps per client\) plus its share of the server update, at a cost of\(\#​batches\)×3​bS\(\\\#\\,\\text\{batches\}\)\\times 3b\_\{S\}for a common batch sizebSb\_\{S\}\. Methods differ only in the server term \([Table6](https://arxiv.org/html/2609.25471#S11.T6)\)\. FedNST sweeps the*entire*labeled source,nS/G=\|ℒ\|/\(bS​G\)n\_\{S\}/G=\|\\mathcal\{L\}\|/\(b\_\{S\}G\)batches per GPU; Rao*et al*\. rehearseρ​C\\rho Cclients,ρ​nS/G\\rho\\,n\_\{S\}/G; and our global/online teachers takekkdata\-parallel supervised steps \(kkper GPU, independent ofGG\)\. Because client and FedNST\-server work both scale as1/G1/G, the relative slowdown is GPU\-count\-independent for a fixed pair\.

Crucially, FedNST’s server term grows with the*source*size, while every other method’s stays tied to the cohort: the per\-GPU client load is320320batches for both pairs, but FedNST addsnS/G=1500/16≈94n\_\{S\}/G=1500/16\\approx 94for LS100→\\rightarrowLS860 \(100100h source\) versus28800/64=45028800/64=450for LS→\\rightarrowCV \(960960h source\), while our teachers add onlykk\. A FedNST step therefore costs≈1\.25×\\approx 1\.25\\times\(LS100→\\rightarrowLS860\) and≈2\.34×\\approx 2\.34\\times\(LS→\\rightarrowCV\) a global/online step \([Table6](https://arxiv.org/html/2609.25471#S11.T6)\)\. All runs share a77\-day wall\-clock limit\. The other methods reach20002000FL steps for LS100→\\rightarrowLS860 and30003000for LS→\\rightarrowCV within it, whereas FedNST’s higher per\-step cost prevents it from completing these budgets within the limit, so its[Table3](https://arxiv.org/html/2609.25471#S7.T3)entries reflect truncated runs\.

## 12Quantifying Domain Shift

The pseudo\-label\-source guideline in[Section7\.4](https://arxiv.org/html/2609.25471#S7.SS4)turns on whether the server and client data are in\-domain \(highly overlapping\) or cross\-domain\. That judgment is usually made from domain knowledge; here we make it quantitative with a speaker\-level similarity measure between two corpora, adapting the embedding\-based data\-selection representation of[Aldeneh et al\. \(2026\)](https://arxiv.org/html/2609.25471#bib.bib4)\.

##### Per\-speaker embeddings\.

Following[Aldeneh et al\. \(2026\)](https://arxiv.org/html/2609.25471#bib.bib4), we represent each speaker by a single13441344\-dimensional vector formed by concatenating three per\-utterance embeddings and averaging over that speaker’s utterances: a192192\-dim speaker\-verification embedding\([Desplanques et al\., 2020](https://arxiv.org/html/2609.25471#bib.bib9)\)\(voice timbre and speaker identity\), a768768\-dim WavLM\([Chen et al\., 2022](https://arxiv.org/html/2609.25471#bib.bib8)\)embedding \(phoneme\-level acoustic content\), and a384384\-dim sentence\-BERT\([Reimers & Gurevych, 2019](https://arxiv.org/html/2609.25471#bib.bib39)\)embedding of the transcript \(lexical and topic content\)\.

##### Pairwise similarity\.

Given per\-speaker embedding matricesEA∈ℝNA×1344E\_\{A\}\\in\\mathbb\{R\}^\{N\_\{A\}\\times 1344\}andEB∈ℝNB×1344E\_\{B\}\\in\\mathbb\{R\}^\{N\_\{B\}\\times 1344\}for corporaAAandBB, we L2\-normalize each row and, for each speakera∈Aa\\in A, take its nearest neighbor inBB,sa=maxb∈B⁡cos⁡\(ea,eb\)s\_\{a\}=\\max\_\{b\\in B\}\\cos\(e\_\{a\},e\_\{b\}\)\. TheA→BA\\rightarrow Bscore is the mean ofsas\_\{a\}over speakers inAA, and we symmetrize a pair by averagingA→BA\\rightarrow BandB→AB\\rightarrow A\. Higher values indicate more similar speaker populations, i\.e\., smaller domain shift\. The three modalities are concatenated without per\-modality normalization, so the score is dominated by the speaker and semantic components\.

##### Results and use as a guideline\.

[Table7](https://arxiv.org/html/2609.25471#S12.T7)reports the measure for the corpora we study\. The two in\-domain pairs \(LS100↔\\leftrightarrowLS860 and CV10↔\\leftrightarrowCV90\) score0\.560\.56and0\.590\.59, both well above the cross\-domain pairs \(0\.380\.38–0\.440\.44\)\. The gap from the lowest in\-domain score \(0\.560\.56\) to the highest cross\-domain score \(LS↔\\leftrightarrowTed,0\.440\.44\) exceeds the spread among the cross\-domain pairs themselves\. The measure therefore separates in\-domain from cross\-domain and can serve as a quantitative proxy for the server–client overlap that the guideline in[Section7\.4](https://arxiv.org/html/2609.25471#S7.SS4)uses to choose the teacher: a high score indicates an in\-domain pair \(favoring the online teacher\), and a low score a cross\-domain pair \(favoring the more robust global teacher\)\. Being symmetric, the measure captures the overall distance between two speaker populations rather than the direction of transfer; the direction\-asymmetric instability of[Section6](https://arxiv.org/html/2609.25471#S6)is governed instead by the dispersion of the server data\.

Table 7:Speaker\-level similarity between corpora, computed as the symmetrized nearest\-neighbor cosine over the13441344\-dim concatenated embeddings\. Higher means more similar speaker populations, i\.e\., smaller domain shift\. The in\-domain pairs \(top\) score well above the cross\-domain pairs \(bottom\)\.PairSimilarityLS100↔\\leftrightarrowLS860 \(in\-domain\)0\.560\.56CV10↔\\leftrightarrowCV90 \(in\-domain\)0\.590\.59LS↔\\leftrightarrowTed0\.440\.44LS↔\\leftrightarrowCV0\.400\.40Ted↔\\leftrightarrowCV0\.380\.38

## 13Contributions

The overall vision for studying semi\-supervised learning for federated learning in ASR was conceived by Wonho Bae, Sheikh Shams Azam, Tatiana Likhomanenko, Martin Pelikan, and Jan “Honza” Silovsky, who identified the gap in current research and defined the problem scope\. This work builds on motivating techniques from prior co\-authored works published by the group: \(i\) private federated learning for ASR[Pelikan et al\. \(2025\)](https://arxiv.org/html/2609.25471#bib.bib36);[Azam et al\. \(2023\)](https://arxiv.org/html/2609.25471#bib.bib6)led by Martin Pelikan, Sheikh Shams Azam and Tatiana Likhomanenko and \(ii\) Iterative Pseudo\-Labeling[Xu et al\. \(2020\)](https://arxiv.org/html/2609.25471#bib.bib52);[Aldeneh et al\. \(2025\)](https://arxiv.org/html/2609.25471#bib.bib3)led by Zakaria Aldeneh\. Specific contributions of the authors can be attributed as:

- •Algorithm Design\.The design of the algorithm and ablations was led by Wonho Bae, Jan “Honza” Silovsky, Tatiana Likhomanenko, and Sheikh Shams Azam in consultation with Martin Pelikan and Zakaria Aldeneh\.
- •Implementation and Experimental Results\.Wonho Bae developed the Semi\-Supervised FL for ASR training pipeline by adapting the PFL codebase from Martin Pelikan and Tatiana Likhomanenko, conducted the domain shift analysis in[Section12](https://arxiv.org/html/2609.25471#S12)using code and guidance from Zakaria Aldeneh, and led all experiments\. The code and data analysis were further reviewed by Tatiana Likhomanenko and Sheikh Shams Azam\. Sheikh Shams Azam also contributed to the evaluations and ablation studies\. All work was done in consultation with the other authors\.
- •Writing and Paper Preparation\.The manuscript was written by Wonho Bae and Sheikh Shams Azam\. It was edited and reviewed by all other authors\.

††Apple and the Apple logo are trademarks of Apple Inc\., registered in the U\.S\. and other countries and regions\.