Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs

# Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs
Source: [https://arxiv.org/html/2609.11762](https://arxiv.org/html/2609.11762)
1stJordi LuqueAffiliation:Scientific Research Telefónica Innovación Digital Barcelona, Spain jordi\.luque@telefonica\.com2ndFernando LópezAffiliation:Telefónica Innovación Digital Universidad Autónoma de Madrid Madrid, Spain fernando\.lopez@telefonica\.com3rdAleix SantAffiliation:Scientific Research Telefónica Innovación Digital Universitat Politècnica de Catalunya Barcelona, Spain aleix\.santsavall@telefonica\.comAffiliation:

###### Abstract

Per\-layer differential privacy \(DP\) clipping improves gradient fidelity in federated learning by allocating per\-matrix clipping budgets proportional to parameter count\. We show that this recipe breaks for speech large language models \(speech\-LLMs\), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm\. Single\-pool per\-layer methods suffer*cross\-component budget collapse*, dragging word error rate \(WER\) far from flat global clipping or collapsing training entirely\. When the norm imbalance is milder, adaptive single\-pool methods partially recover, confirming that collapse severity scales with the inter\-component norm ratio\. We empirically diagnose the root cause across six per\-layer methods and three speech\-LLM architectures\. We then propose*α\\alpha\-split*, a two\-pool allocation that normalises encoder and LLM parameters into independent pools, and show that jointℓ2\\ell\_\{2\}sensitivity and the original\(ε,δ\)\(\\varepsilon,\\delta\)\-DP guarantee are unchanged\. At architecture\-calibratedα\\alpha, our method recovers WER utility compared to flat DP, while granting the encoder4\.47×4\.47\{\\times\}tighter per\-component noise protection against speaker voice\-based gradient\-inversion attacks at only\+2\.6%\+2\.6\\%LLM noise overhead\.

###### Index Terms:

federated learning, differential privacy, speech recognition, large language models, per\-layer clipping, LoRA

## IIntroduction

Large language models \(LLMs\) and speech foundation models are increasingly combined to build end\-to\-end speech\-LLM systems for automatic speech recognition \(ASR\), spoken understanding, and conversational speech applications\[[1](https://arxiv.org/html/2609.11762#bib.bib1),[2](https://arxiv.org/html/2609.11762#bib.bib2)\]\. These architectures typically couple three heterogeneous components: an acoustic encoder, a cross\-modal connector, and a language decoder\[[3](https://arxiv.org/html/2609.11762#bib.bib3),[4](https://arxiv.org/html/2609.11762#bib.bib4)\]\. While this modular design improves transferability and downstream performance, it also introduces optimization asymmetries across components, especially in distributed and privacy\-sensitive training settings\.

Federated learning \(FL\)\[[5](https://arxiv.org/html/2609.11762#bib.bib5)\]is a natural paradigm for speech applications because raw audio is privacy\-critical, bandwidth\-heavy, and often constrained by data governance policies\. However, practical FL deployments for speech\-LLMs face two challenges\. First, client data are strongly non\-IID across speakers, accents, microphones, and acoustic environments, which induces unstable update distributions\[[6](https://arxiv.org/html/2609.11762#bib.bib6)\]\. Second, differential privacy \(DP\) mechanisms\[[7](https://arxiv.org/html/2609.11762#bib.bib7),[8](https://arxiv.org/html/2609.11762#bib.bib8),[9](https://arxiv.org/html/2609.11762#bib.bib9)\], particularly clipping\-based methods, can alter optimization dynamics in ways that are not yet well understood for multimodal, multi\-component models\.

To achieve formal\(ε,δ\)\(\\varepsilon,\\delta\)\-DP\[[8](https://arxiv.org/html/2609.11762#bib.bib8)\]in federated learning,*gradient clipping*serves as a fundamental mathematical requirement: before the server injects calibrated Gaussian noise, it must strictly bound theℓ2\\ell\_\{2\}sensitivity of each client update to a clipping budgetCC\. Standard DP\-FL applies a single*global*CCto the full concatenated update\. While most prior FL\+DP studies report aggregate utility\-privacy trade\-offs using global hyperparameters\[[6](https://arxiv.org/html/2609.11762#bib.bib6)\], this approach is architecturally oblivious—encoder LoRA adapters, connector projections, and LLM adapters are clipped identically despite having fundamentally different update magnitudes and sensitivity requirements\. Recent work\[[10](https://arxiv.org/html/2609.11762#bib.bib10)\]demonstrated that*per\-layer*clipping, where each parameter matrix receives an individual budget proportional to its size, substantially outperforms flat global clipping for homogeneous single\-component ASR\. We show this recipe breaks for multimodal speech\-LLMs\.

We demonstrate that this state\-of\-the\-art recipe\[[10](https://arxiv.org/html/2609.11762#bib.bib10)\]systematically*breaks down*in modular Speech\-LLMs when the encoder/LLM update norm ratio is large enough\. In such situations, the encoder’s vast number of parameters structurally dilutes the LLM’s clipping budget\. We investigate six single\-pool per\-layer methods, reporting that all of them suffer severe cross\-component budget collapse, driving the utility Word Error Rate \(WER\) either far from the flat global clipping ceiling or collapsing training entirely\. To resolve this bottleneck, we formalize the necessity of component\-aware privacy allocation\. Our key contributions are:

1. 1\.A precise diagnosis of*cross\-component budget collapse*: We empirically demonstrate that when encoder/LLM norm imbalances are extreme \(≥10×\\geq 10\\times\), single\-pool methods suffer severe budget collapse during critical early warmup rounds, necessitating a structural solution\.
2. 2\.An adaptive single\-pool formulation: We adapt the size\-proportional per\-layer clipping concept\[[10](https://arxiv.org/html/2609.11762#bib.bib10)\]to parameterised LoRA matrices and introduce a dynamic gradient norms tracking mechanism\. We show that this adaptive formulation dynamically corrects mild encoder/LLM norm imbalances in Speech\-LLMs\.
3. 3\.The structural*α\\alpha\-split*DP design: We propose a component\-aware strategy that decouples gradient normalisation into two independent pools, allocating budgets ofC​αC\\sqrt\{\\alpha\}to the acoustic encoder andC​1−αC\\sqrt\{1\-\\alpha\}to the LLM decoder\. We prove that this separation entirely prevents budget collapse while strictly preserving the jointℓ2\\ell\_\{2\}sensitivity and the global\(ε,δ\)\(\\varepsilon,\\delta\)\-DP mathematical guarantees\.
4. 4\.Delineation of operating regimes and asymmetric biometric privacy: We map out clear deployment guidelines: Adaptive layer\-based is optimal for mild imbalances \(measured≈1\.7×\\approx\\\!1\.7\{\\times\}for Voxtral\-Mini\-3B\), whileα\\alpha\-split is strictly required for extreme imbalances \(measured≈12×\\approx\\\!12\{\\times\}for Whisper\+TinyLlama/EuroLLM\)\. Furthermore, we show that theα\\alpha\-split formulation uniquely grants the acoustic encoder tighter effective noise multiplier, achieving stronger acoustic protection for sensitive biometric data at a negligible noise overhead to the LLM\.

## IISpeech\-LLM Framework and Private Federated Setup

### II\-ASpeech\-LLM Architecture

*Speech\-LLMs*couple three heterogeneous components: an acoustic encoder \(ℰ\\mathcal\{E\}\), a cross\-modal connector \(𝒞\\mathcal\{C\}\), and a language decoder \(ℒ\\mathcal\{L\}\), producing a transcript as:

𝐲^=ℒ⁡\(\[𝒞⁡\(ℰ⁡\(𝐱\)\);𝐄text\]\)\.\\hat\{\\mathbf\{y\}\}=\\mathcal\{L\}\\left\(\[\\mathcal\{C\}\(\\mathcal\{E\}\(\\mathbf\{x\}\)\);\\,\\mathbf\{E\}\_\{\\mathrm\{text\}\}\]\\right\)\.\(1\)
Figure[1](https://arxiv.org/html/2609.11762#S2.F1)summarises the full pipeline\. All three components are trained jointly by minimising the standard autoregressive cross\-entropy loss over the ground\-truth transcript tokens𝐲=\(y1,…,yS\)\\mathbf\{y\}=\(y\_\{1\},\\ldots,y\_\{S\}\):

ℒCE\(𝜽\)=−∑s=1Slogp𝜽\(ys∣𝒞\(ℰ\(𝐱\)\),𝐄text,y<s\),\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(\\boldsymbol\{\\theta\}\)=\-\\sum\_\{s=1\}^\{S\}\\log p\_\{\\boldsymbol\{\\theta\}\}\\left\(y\_\{s\}\\mid\\mathcal\{C\}\(\\mathcal\{E\}\(\\mathbf\{x\}\)\),\\,\\mathbf\{E\}\_\{\\mathrm\{text\}\},\\,y\_\{<s\}\\right\),\(2\)where𝜽\\boldsymbol\{\\theta\}collects all trainable parameters \(LoRA adapters and connector projection\), andy<sy\_\{<s\}denotes the preceding tokens supplied via teacher forcing\. The loss is computed over transcript tokens; audio tokens and prompt embeddings appear as conditioning context\. We study a three\-component*Speech\-LLMs*: a Whisper large\-v3\-turbo encoder\[[11](https://arxiv.org/html/2609.11762#bib.bib11)\]ℰ\\mathcal\{E\}, a linear connector𝒞\\mathcal\{C\}, and a TinyLlama\-1\.1B\[[12](https://arxiv.org/html/2609.11762#bib.bib12)\]\(or EuroLLM\-1\.7B\-Instruct\[[13](https://arxiv.org/html/2609.11762#bib.bib13)\]\) decoderℒ\\mathcal\{L\}; and additionally evaluate*Voxtral\-Mini\-3B*\[[14](https://arxiv.org/html/2609.11762#bib.bib14)\]an end\-to\-end multimodal Speech\-LLM, composed of a Whisper\-large\-v3\-based audio encoder and a Ministral\-3B\[[15](https://arxiv.org/html/2609.11762#bib.bib15)\]LLM, a 30\-layer Llama\-style text decoder; jointly pretrained on audio understanding and ASR tasks\. Voxtral uses a different connector𝒞\\mathcal\{C\}, downsampling the audio by a factor of 4, followed by a 2\-layer MLP projector; a deeper, higher\-compression connector\.

### II\-BCentral DP\-FL Trust Model

We adopt the*central DP*trust model\[[8](https://arxiv.org/html/2609.11762#bib.bib8),[9](https://arxiv.org/html/2609.11762#bib.bib9)\], i\.e\. the aggregation server is trusted, each client clips its own update to boundℓ2\\ell\_\{2\}sensitivity, and the server adds calibrated Gaussian noise to the aggregate, thus privacy is guaranteed against any external observer of the released model sequence\. Each client clips its update before transmission:

Δ​𝜽~\(i\)=Δ​𝜽\(i\)⋅min⁡\(1,C‖Δ​𝜽\(i\)‖2\),\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}^\{\(i\)\}=\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}\\cdot\\min\\\!\\left\(1,\\;\\frac\{C\}\{\\left\\\|\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}\\right\\\|\_\{2\}\}\\right\),\(3\)and the server aggregates with Gaussian noise:

𝜽\(t\+1\)=𝜽\(t\)\+1n​∑i=1nΔ​𝜽~\(i\)\+𝒩⁡\(𝟎,σ2​C2n2​𝐈\)\.\\boldsymbol\{\\theta\}^\{\(t\+1\)\}=\\boldsymbol\{\\theta\}^\{\(t\)\}\+\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}^\{\(i\)\}\+\\mathcal\{N\}\\\!\\left\(\\mathbf\{0\},\\,\\frac\{\\sigma^\{2\}C^\{2\}\}\{n^\{2\}\}\\,\\mathbf\{I\}\\right\)\.\(4\)Theℓ2\\ell\_\{2\}sensitivity of the clipped aggregate isC/nC/n, soσ\\sigmais the noise multiplier relative to the sensitivity\. All experiments useC=1\.0C\{=\}1\.0,σ=0\.1\\sigma\{=\}0\.1; privacy accounting uses Rényi DP\[[16](https://arxiv.org/html/2609.11762#bib.bib16)\]\. Note that in the case of FedAvg\[[5](https://arxiv.org/html/2609.11762#bib.bib5)\]for Speech\-LLMs, each clientiicomputes the full parameter delta as a flat concatenation:

Δ​𝜽\(i\)=\[Δ​𝜽ℰ\(i\),Δ​𝜽𝒞\(i\),Δ​𝜽ℒ\(i\)\]\.\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}=\\bigl\[\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{E\}\}^\{\(i\)\},\\;\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{C\}\}^\{\(i\)\},\\;\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{L\}\}^\{\(i\)\}\\bigr\]\.\(5\)
![Refer to caption](https://arxiv.org/html/2609.11762v1/figures/arch_diagram.png)Fig\. 1:Speech\-LLM architecture\. The audio encoderℰ\\mathcal\{E\}maps raw speech𝐱\\mathbf\{x\}to frame\-level representations𝐇\\mathbf\{H\}\. The connector𝒞\\mathcal\{C\}downsamples frames, reducing sequence length bykk\. The LLM decoderℒ\\mathcal\{L\}generates the transcript𝐲^\\hat\{\\mathbf\{y\}\}conditioned on the projected tokens and a task prompt\.TABLE I:LoRA parameterppusingr​a​n​k=8rank\{=\}8for Whisper \+ TinyLlama Speech\-LLM, applied to queries \(qq\), keys \(OPENk\)k\)and values \(vv\) in attention heads\.ComponentMatricesAvg\. paramsTotalEncoder \(ℰ\\mathcal\{E\}, q/k/v, 32L\)6410,2401,966,080LLM \(ℒ\\mathcal\{L\}, q/v, 22L\)4416,384720,896Connector \(𝒞\\mathcal\{C\}, weight\+bias\)22,0484,096Total \(H\)110—2,691,072Table[I](https://arxiv.org/html/2609.11762#S2.T1)summarises the LoRA parameter counts\. The encoder represents 73\.1% of all LoRA parameters, yet each encoder matrix is considerably*smaller*than an LLM attention matrix\.

### II\-CHeterogeneous Dataset and Speaker Partitioning

#### II\-C1Corpus

All experiments use the Multilingual LibriSpeech \(MLS\) corpus\[[17](https://arxiv.org/html/2609.11762#bib.bib17)\], an audiobook corpus covering 8 European languages derived from LibriVox recordings\. We use the official MLStrainsplits as the federated training pool \(685\.7 h total\), the MLSdevsplit for validation during training, and the MLStestsplit as the held\-out evaluation benchmark \(138 h, 19,492 samples\)\.

#### II\-C2Speaker Data Partition \(non\-IID\)

We perform a stratified speaker\-based partition that assigns all utterances of a single MLS speaker to one client\. WithK=316K\{=\}316clients, this creates the strongest possible non\-IID distribution: each client’s data is drawn from a single acoustic identity, language, and recording environment, producing simultaneous*linguistic*\(each client speaks at most one language\) and*acoustic*\(microphone, room, speaking rate\) heterogeneity\. Due to the LibriVox origin of MLS, 8 of 316 clients \(2\.5%\) correspond to speakers also present in the MLS test split, accounting for 4,747 of 169,586 training samples \(2\.8%\)\. All FL experiments in this work use this partition\.

### II\-DFederated Optimization Configuration

All FL experiments use the Flower simulation framework with Ray as the backend\[[18](https://arxiv.org/html/2609.11762#bib.bib18)\]\. All FL experiments train forT=40T\{=\}40global rounds across a total population ofK=316K=316speaker\-partitioned clients; in each round, a random cohort ofn≈94n\\approx 94clients \(sampling rateq=0\.3q=0\.3\)\. Local clients performE=10E\{=\}10local epochs of fine\-tuning before aggregation, using AdamW optimiser with maximum learning rateη=10−4\\eta\{=\}10^\{\-4\}, cosine decay, batch size 16\. Evaluation uses the MLS test split with overall WER reported across all 8 languages combined\. All models are fine\-tuned with LoRA\[[19](https://arxiv.org/html/2609.11762#bib.bib19)\]adapters applied to all query, key, value for encoder and query, value for LLM decoder heads\. Unlike the encoders and LLM decoders, which carry rich pretrained representations, we initialise the multimodal connector from scratch, except for Voxtral, and must learn to bridge audio and text modalities entirely from the federated fine\-tuning data\.

![Refer to caption](https://arxiv.org/html/2609.11762v1/figures/norm_heterogeneity.png)Fig\. 2:Mean update norm‖Δ​θh‖2\\\|\\Delta\\theta\_\{h\}\\\|\_\{2\}per\-Trainable Grouphh, i\.e\. LoRA matrix \(log scale, sorted descending, with no\-DP and all components unfrozen\) for the Whisper \+ TinyLlama Speech\-LLM\. Green bars stand for LLM LoRA matrices\. Blue correspond to encoder LoRA matrices\. Orange \(index 0\) to connector \(single full\-rank matrix\)\. Pink band: Gaussian noise floorσ​C/n≈0\.01\\sigma C/\\sqrt\{n\}\{\\approx\}0\.01\.

## IIICross\-Component Budget Collapse

### III\-AEmpirical Gradient\-Norm Asymmetries

With flat or global clipping, see Eq\. \([3](https://arxiv.org/html/2609.11762#S2.E3)\), every LoRA matrix is scaled by the*same*ratio scalarrr:

Δ​𝜽~k\(i\)=r\(i\)​Δ​𝜽k\(i\),\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}\_\{k\}^\{\(i\)\}=r^\{\(i\)\}\\Delta\\boldsymbol\{\\theta\}\_\{k\}^\{\(i\)\},\(6\)withr\(i\)=min⁡\(1,C‖Δ​𝜽\(i\)‖2\)r^\{\(i\)\}=\\min\\\!\\left\(1,\\,\\frac\{C\}\{\\\|\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}\\\|\_\{2\}\}\\right\), the per\-clientiiratio\.

Fig\.[2](https://arxiv.org/html/2609.11762#S2.F2)depicts the empirical per\-matrix update norms from a federated training non\-DP with all components unfrozen\. Two clusters are clearly separated by an order of magnitude: LLM matrices \(green,≈0\.1\{\\approx\}0\.1–0\.20\.2\) and encoder LoRA matrices \(blue,≈0\.01\{\\approx\}0\.01–0\.020\.02\)\. The pink band marks the Gaussian noise floorσ​C/n≈0\.01\\sigma C/\\sqrt\{n\}\\approx 0\.01: encoder updates already sit at the noise level under flat DP, while LLM updates remain well above it\. When the LLM pool dominates‖Δ​𝜽\(i\)‖2\\\|\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}\\\|\_\{2\},r\(i\)r^\{\(i\)\}is driven towardC/‖Δ​𝜽ℒ\(i\)‖2C/\\\|\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{L\}\}^\{\(i\)\}\\\|\_\{2\}, and the encoder pool, with norms already10×10\\timessmaller, is suppressed further toward the noise floor\. This∼10×\{\\sim\}10\\timesper\-matrix norm gap drives the cross\-component budget collapse, where encoder matrices collectively absorb a disproportionate share of theC2C^\{2\}budget\.

### III\-BSNR Suppression under Global Clipping

We can quantify this suppression via the per\-component*signal\-to\-noise ratio*\(SNR\)\. After aggregatingnnclients, the per\-parameter signal of componenthhis‖Δ​𝜽~h‖2/ph\\\|\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}\_\{h\}\\\|\_\{2\}/\\sqrt\{p\_\{h\}\}, while the per\-parameter noise standard deviation from the Gaussian mechanism \([4](https://arxiv.org/html/2609.11762#S2.E4)\) isσ​C/n\\sigma C/n\. Thus:

SNRh=‖Δ​𝜽~h‖2/phσ​C/n\.\\mathrm\{SNR\}\_\{h\}=\\frac\{\\displaystyle\\left\\\|\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}\_\{h\}\\right\\\|\_\{2\}/\\,\\sqrt\{p\_\{h\}\}\}\{\\sigma C/n\}\.\(7\)Thus the*ratio*between two components depends only on their clipped norms and parameter counts:

SNRℰSNRℒ=‖Δ​𝜽~ℰ‖2/pℰ‖Δ​𝜽~ℒ‖2/pℒ\.\\frac\{\\mathrm\{SNR\}\_\{\\mathcal\{E\}\}\}\{\\mathrm\{SNR\}\_\{\\mathcal\{L\}\}\}=\\frac\{\\\|\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}\_\{\\mathcal\{E\}\}\\\|\_\{2\}\\,/\\,\\sqrt\{p\_\{\\mathcal\{E\}\}\}\}\{\\\|\\widetilde\{\\Delta\\boldsymbol\{\\theta\}\}\_\{\\mathcal\{L\}\}\\\|\_\{2\}\\,/\\,\\sqrt\{p\_\{\\mathcal\{L\}\}\}\}\.\(8\)Under LLM\-dominant global clipping, every client’s update is scaled byr\(i\)≈C/‖Δ​𝜽ℒ\(i\)‖2r^\{\(i\)\}\\approx C/\\\|\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{L\}\}^\{\(i\)\}\\\|\_\{2\}, so both components share the samer\(i\)r^\{\(i\)\}and the ratio reduces to a norm\-and\-size comparison\. With per\-matrix norms differing one order of magnitude and similar per\-matrix parameter counts \(see Table[I](https://arxiv.org/html/2609.11762#S2.T1)\), the per\-layer encoder signal is≈10×\{\\approx\}10\\timesweaker than the LLM’s\. Since the noise floor is identical for both pools, the encoder operates atSNRℰ≈0\.1​SNRℒ\\mathrm\{SNR\}\_\{\\mathcal\{E\}\}\\approx 0\.1\\,\\mathrm\{SNR\}\_\{\\mathcal\{L\}\}under global clipping\. Fig\.[3](https://arxiv.org/html/2609.11762#S3.F3)confirms that this structural imbalance persists dynamically under flat DP with all components unfrozen, LLM\+connector\-dominated global norms keep the clipping rate high across all 40 rounds, leaving the encoder signal consistently noise\-dominated during training\.

![Refer to caption](https://arxiv.org/html/2609.11762v1/figures/flat_dp_clipping_dynamics.png)Fig\. 3:Flat DP with all unfrozen components \(Flat\-ebaseline, 40 rounds,C=1\.0C\{=\}1\.0, averaging 94 clients/round\)\. Main panel \(left axis, dashed\): mean total joint update norm‖Δ​𝜽\(i\)‖2\\\|\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}\\\|\_\{2\}across clients\. Right axis: client clipping rate \(red, fraction of clients whose update norm exceedsCC\) and mean clip ratior¯\(t\)=1nt​∑imin⁡\(1,C/‖Δ​𝜽\(i\)‖2\)\\bar\{r\}^\{\(t\)\}\\\!=\\\!\\frac\{1\}\{n\_\{t\}\}\\sum\_\{i\}\\min\(1,C/\\\|\\Delta\\boldsymbol\{\\theta\}^\{\(i\)\}\\\|\_\{2\}\)\(purple\), i\.e\. the average fraction of each update that survives clipping before aggregation;r¯\(t\)=1\\bar\{r\}^\{\(t\)\}\\\!=\\\!1means no client is clipped, whiler¯\(t\)≪1\\bar\{r\}^\{\(t\)\}\\\!\\ll\\\!1means updates are severely truncated\. Inset: mean per\-component norm budget per round \(encoder blue, connector orange, LLM green\)\. LLM and connector norms dominate throughout, sustaining a high global clip rate \(\>\>80%\) that suppresses the encoder signal to the noise floor\.

## IVEvaluated DP Methods

We perform FL\-DP experiments using different strategies to allocate theCCbudget across the layers of a Speech\-LLM\. All methods share commonC=1\.0C\{=\}1\.0budget,σ=0\.1\\sigma\{=\}0\.1, and server Gaussian noise𝒩⁡\(0,\(σ​C/n\)2​𝐈\)\\mathcal\{N\}\(0,\(\\sigma C/n\)^\{2\}\\mathbf\{I\}\)\. The methods in\[[10](https://arxiv.org/html/2609.11762#bib.bib10)\]replace the single global clip in \([3](https://arxiv.org/html/2609.11762#S2.E3)\) with per\-layerℓ\\ellbudgets that partitionC2C^\{2\}proportionally to layer \(LoRA rank matrix\) size,pℓp\_\{\\ell\}:

Cℓ=C⋅pℓ∑kpk,so that∑ℓCℓ2=C\.C\_\{\\ell\}=C\\cdot\\sqrt\{\\frac\{p\_\{\\ell\}\}\{\\sum\_\{k\}p\_\{k\}\}\},\\quad\\text\{so that\}\\quad\\sqrt\{\\sum\_\{\\ell\}C\_\{\\ell\}^\{2\}\}=C\.\(9\)This*Euclidean\-tight*allocation\[[10](https://arxiv.org/html/2609.11762#bib.bib10)\]was originally designed for homogeneous single\-component models with full\-rank updates and is the basis for all methods we study:

- •Flat: refers to DP\-FedAvg baseline\[[9](https://arxiv.org/html/2609.11762#bib.bib9)\], i\.e\. a single normCCis applied to the full concatenated update in Eq\.[5](https://arxiv.org/html/2609.11762#S2.E5), so no per\-layer structure\.
- •PFL\-Uniform: we split uniformly the budget among LoRA matrices:Cℓ=C/HC\_\{\\ell\}=C/\\sqrt\{H\}, regardless of size\. WhereHHis the total number of matrices, i\.e\. adapted layers\.
- •PFL\-Dim: size\-proportional allocation via Eq\. \([9](https://arxiv.org/html/2609.11762#S4.E9)\); the “dim\-prop” variant from\[[10](https://arxiv.org/html/2609.11762#bib.bib10)\], adapted for LoRA rank matrices\.
- •PFL\-Unif\+EMAandPFL\-Dim\+EMA\(adaptive\): we implemented original*exponential moving average*\(EMA\) variants that dynamically reweight per\-layer budgets from observed gradient norms\. At each FL roundtt, a smoothed per\-layer norm estimate is updated as ν^ℓ\(t\)=\(1−β\)​ν^ℓ\(t−1\)\+β​νℓ\(t\),\\hat\{\\nu\}\_\{\\ell\}^\{\(t\)\}=\(1\{\-\}\\beta\)\\,\\hat\{\\nu\}\_\{\\ell\}^\{\(t\-1\)\}\+\\beta\\,\\nu\_\{\\ell\}^\{\(t\)\},\(10\)whereνℓ\(t\)\\nu\_\{\\ell\}^\{\(t\)\}is the mean client delta\-norm for layerℓ\\ellin roundtt,β=0\.2\\beta\{=\}0\.2is the decay rate \(higherβ\\beta= faster adaptation\)\. Allocation weights and per\-layer norms are then: wℓtarget∝\(ν^ℓ\)γ⋅wℓbase,Cℓ=Ct​wℓ,w\_\{\\ell\}^\{\\text\{target\}\}\\propto\\bigl\(\\hat\{\\nu\}\_\{\\ell\}\\bigr\)^\{\\gamma\}\\cdot w\_\{\\ell\}^\{\\text\{base\}\},\\qquad C\_\{\\ell\}=C\_\{t\}\\sqrt\{w\_\{\\ell\}\},\(11\)withγ=0\.75\\gamma\{=\}0\.75andwℓbasew\_\{\\ell\}^\{\\text\{base\}\}equal to the PFL\-Dim or PFL\-Unif initialisation for each respective variant\. Weights are frozen atwℓbasew\_\{\\ell\}^\{\\text\{base\}\}for the first three warmup rounds\.
- •α\\alpha\-Split\(static\): encoder and LLM\+\+connector treated as independent pools with budgetsC​αC\\sqrt\{\\alpha\}andC​1−αC\\sqrt\{1\{\-\}\\alpha\}\. Fully described in following Section[IV\-A](https://arxiv.org/html/2609.11762#S4.SS1)\.

The\-esuffix \(e\.g\.PFL\-Dim\-e\) denotes the encoder\-unfrozen variant; bare names denote encoder\-frozen training\.

### IV\-ATheα\\alpha\-Split Design

#### IV\-A1Formulation

Letα∈\(0,1\)\\alpha\\in\(0,1\)be a*component budget fraction*hyperparameter controlling what share ofC2C^\{2\}is allocated to the encoder\. Encoder and LLM\+connector parameters are clipped with*independent*per\-layer budgets:

Cℓe​n​c=C​α​pℓ∑k∈ℰpk,ℓ∈ℰ\\displaystyle C^\{enc\}\_\{\\ell\}=C\\sqrt\{\\alpha\}\\sqrt\{\\frac\{p\_\{\\ell\}\}\{\\sum\_\{k\\in\\mathcal\{E\}\}p\_\{k\}\}\},\\quad\\ell\\in\\mathcal\{E\}\(12\)CℓL​L​M=C​1−α​pℓ∑k∈ℒpk,ℓ∈ℒ\.\\displaystyle C^\{LLM\}\_\{\\ell\}=C\\sqrt\{1\-\\alpha\}\\sqrt\{\\frac\{p\_\{\\ell\}\}\{\\sum\_\{k\\in\\mathcal\{L\}\}p\_\{k\}\}\},\\quad\\ell\\in\\mathcal\{L\}\.\(13\)Each pool is internally Euclidean\-tight\. Note that the connector is included inℒ\\mathcal\{L\}, because assigning it toℰ\\mathcal\{E\}would incorrectly inflate the encoder budget\. For the linear connector used in Whisper \+ \{TinyLlama,EuroLLM\} the two matrices total∼4,096\{\\sim\}4\{,\}096parameters and are indeed negligible withinℒ\\mathcal\{L\}\.

TABLE II:Encoder and LLM \(Whisper \+ TinyLlama\) per\-layer clip norms vs\.α\\alpha\.Flat\-ereference:C¯ℓLLM=0\.1474\\bar\{C\}\_\{\\ell\}^\{\\text\{LLM\}\}=0\.1474\.α\\alphaC¯ℓenc\\bar\{C\}\_\{\\ell\}^\{\\text\{enc\}\}C¯ℓLLM\\bar\{C\}\_\{\\ell\}^\{\\text\{LLM\}\}LLM / Flat\-eNote0\.000\.0000\.15031\.020Enc\. frozen0\.010\.0130\.14951\.014Min\. enc\. signal0\.050\.0280\.14650\.994<<1% LLM loss0\.100\.0400\.14260\.967<<5% LLM loss0\.200\.0560\.13450\.912PFL\-Dim\+EMA\-e steady\-state0\.500\.0880\.10630\.721Half budget each
#### IV\-A2Sensitivity and Privacy Preservation

Letcℰ=‖clippedℰ‖2c\_\{\\mathcal\{E\}\}=\\\|\\text\{clipped\}\_\{\\mathcal\{E\}\}\\\|\_\{2\}andcℒ=‖clippedℒ‖2c\_\{\\mathcal\{L\}\}=\\\|\\text\{clipped\}\_\{\\mathcal\{L\}\}\\\|\_\{2\}\. Then theℓ2\\ell\_\{2\}sensitivity is:

Δ=cℰ2\+cℒ2≤α​C2\+\(1−α\)​C2=C\.\\Delta=\\sqrt\{c\_\{\\mathcal\{E\}\}^\{2\}\+c\_\{\\mathcal\{L\}\}^\{2\}\}\\leq\\sqrt\{\\alpha C^\{2\}\+\(1\-\\alpha\)C^\{2\}\}=C\.\(14\)The sensitivity is unchanged atCC\. The server Gaussian noise𝒩⁡\(0,\(σ​C/n\)2​𝐈\)\\mathcal\{N\}\(0,\(\\sigma C/n\)^\{2\}\\mathbf\{I\}\)requires no modification, and the Rényi DP accountant,ε\\varepsilon, andδ\\deltaare all identical toFlat\-DP\. Thusα\\alphais a pure utility\-vs\-component\-privacy parameter\.

#### IV\-A3Choosingα\\alpha

Table[II](https://arxiv.org/html/2609.11762#S4.T2)shows the LLM per\-layer budget as a function ofα\\alpha\. We selectα=0\.05\\alpha=0\.05because \(i\) the LLM retains 99\.4% of itsFlat\-ebudget with negligible regression risk; \(ii\) the encoder clip norm \(0\.0280\.028\) is≈2×\{\\approx\}2\\timesthe observed encoder delta\-norm \(0\.0130\.013\), providing sufficient gradient signal without near\-100% clipping\. Note that both criteria depend solely on parameter counts and the encoder/LLM update\-norm ratio, which can be measured on any public corpus before deployment, makingα\\alphaselection fully privacy\-compatible\.

Fig\. 4:Effective per\-component noise multiplier vs\. encoder budget fractionα\\alpha\. Solid lines:α\\alpha\-Split\-e\(encoder red, LLM blue\)\. Dashed:Flat\-ereference \(σeff=σ\\sigma\_\{\\mathrm\{eff\}\}\{=\}\\sigmafor both\)\. Atα∗=0\.05\\alpha^\{\*\}\{=\}0\.05\(vertical dotted line\) the encoder receives4\.47×4\.47\\timestighter per\-component privacy at only\+2\.6%\+2\.6\\%LLM overhead, with no change to the joint\(ε,δ\)\(\\varepsilon,\\delta\)guarantee\.
#### IV\-A4Per\-Component Privacy Advantage

Note thatFlat\-DP should achieve the utility ceiling \(lower WER for ASR task\) for any per\-layer strategy at the same\(ε,δ\)\(\\varepsilon,\\delta\)\. Any per\-layer method at the same global sensitivityCCcalibrates noise toCC, somin⁡\(σeffℰ,σeffℒ\)≥σ\\min\(\\sigma\_\{\\mathrm\{eff\}\}^\{\\mathcal\{E\}\},\\,\\sigma\_\{\\mathrm\{eff\}\}^\{\\mathcal\{L\}\}\)\\geq\\sigma\. The genuine advantage ofα\\alpha\-split is*asymmetric per\-component privacy*\. Because the encoder pool is bounded atC​αC\\sqrt\{\\alpha\}, its effective noise multiplier isσeffℰ=σα=0\.447\\sigma\_\{\\mathrm\{eff\}\}^\{\\mathcal\{E\}\}=\\frac\{\\sigma\}\{\\sqrt\{\\alpha\}\}=0\.447, which is4\.47×4\.47\\timesthat ofFlat\-e, while the LLM pays onlyσeffℒ=σ1−α=0\.103\\sigma\_\{\\mathrm\{eff\}\}^\{\\mathcal\{L\}\}=\\frac\{\\sigma\}\{\\sqrt\{1\-\\alpha\}\}=0\.103, which is only\+2\.6%\+2\.6\\%vs\.Flat\-e\. Fig\.[4](https://arxiv.org/html/2609.11762#S4.F4)visualisesσeff\\sigma\_\{\\mathrm\{eff\}\}as a function ofα\\alpha, confirming the steep encoder privacy gain and flat LLM overhead for smallα\\alpha\. Since the encoder pool is bounded atC​αC\\sqrt\{\\alpha\}, an adversary performing gradient inversion learns4\.47×4\.47\\timesless about acoustic encoder updates, directly guarding speaker biometric attributes \(accent, prosody, voice identity\)\. Note that whatα\\alpha\-splitchanges is the*per\-component*interpretation: standard global clipping mixes acoustic and linguistic gradients into a single pool, whereasα\\alpha\-split structurally isolates the encoder’s biometric representations, providing a privacy guarantee that per\-layer single\-pool methods cannot achieve regardless of their budget allocation\.

Fig\. 5:Top:Per\-component client clipping rate per FL round underα\\alpha\-Split\-e\(α=0\.05\\alpha\{=\}0\.05,C=1\.0C\{=\}1\.0, 94 clients/round\)\. Red: encoder pool \(C​α=0\.224C\\sqrt\{\\alpha\}\{=\}0\.224\); blue: LLM\+connector pool \(C​1−α=0\.975C\\sqrt\{1\{\-\}\\alpha\}\{=\}0\.975\)\.Bottom:Mean update norm‖Δ​𝜽‖2\\\|\\Delta\\boldsymbol\{\\theta\}\\\|\_\{2\}per component—encoder‖Δ​𝜽ℰ‖2\\\|\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{E\}\}\\\|\_\{2\}\(red\) and LLM\+connector‖Δ​𝜽ℒ‖2\\\|\\Delta\\boldsymbol\{\\theta\}\_\{\\mathcal\{L\}\}\\\|\_\{2\}\(blue\)—with dashed lines at the respective pool clip thresholds\.
#### IV\-A5Training Dynamics and Clipping Rate Analysis

Fig\.[5](https://arxiv.org/html/2609.11762#S4.F5)shows the per\-component clipping rate vs\. FL round for theα\\alpha\-Split\-erun \(α=0\.05\\alpha\{=\}0\.05,C=1\.0C\{=\}1\.0, averaging 94 clients/round\)\. Two distinct regimes emerge: the encoder clip rate starts at≈51%\{\\approx\}51\\%\(rounds 1–6\) and quickly falls to55–14%14\\%once its norm stabilises well below the0\.2240\.224threshold; the LLM\+connector starts at≈93%\{\\approx\}93\\%clip rate \(norm≈4\.0\{\\approx\}4\.0\) and converges gradually to2525–40%40\\%by round 40\. The low encoder clipping rate after round 8 confirms thatα\\alpha\-Splitis structurally non\-destructive\. Because DP guarantees depend on bounding worst\-case sensitivity rather than active clip frequency, this mechanism enforces the biometric privacy bound while keeping clipping bias minimal once training stabilises\.

## VExperimental Results

Table[III](https://arxiv.org/html/2609.11762#S5.T3)compares the utility of all methods on the MLS test split\. Statistical significance of WER differences is assessed using speaker\-level percentile bootstrap test speakers \(B=10,000B\{=\}10\{,\}000, 95% CI\), with all utterances from each sampled speaker included to preserve within\-speaker correlations\[[20](https://arxiv.org/html/2609.11762#bib.bib20)\]\.

### V\-AEncoder\-Frozen Variants

When the encoder is frozen, cross\-component coupling is absent and results are consistent across all three architectures\.PFL\-Dim\+EMAachieves the best DP result in every group \(W\+TinyLlama:0\.16570\.1657; W\+EuroLLM:0\.13620\.1362; Voxtral:0\.14500\.1450\), outperformingFlatby0\.00280\.0028–0\.00820\.0082\.PFL\-Uniformdegrades severely in all frozen groups \(0\.46430\.4643for W\+TinyLlama,0\.35000\.3500for W\+EuroLLM\), confirming equal\-budget\-per\-layer is harmful regardless of architecture\. Notably,α\\alpha\-Splitslightly*underperforms*Flatfor EuroLLM frozen \(\+0\.0128\+0\.0128vs\.\+0\.0114\+0\.0114\): with the encoder inactive the entire budget falls on LLM layers under any method, so the two\-pool design yields no structural advantage\.

### V\-BEncoder\-Unfrozen Variants and Cross\-Component Collapse

Activating the encoder exposes the cross\-component budget collapse across all single\-pool methods\. On Whisper\+TinyLlama,α\\alpha\-Split\-eremains the*only*per\-layer method that avoids catastrophic degradation\.PFL\-Uniform\-e/PFL\-Unif\+EMA\-ecollapse \(0\.9944†0\.9944^\{\\dagger\}/0\.8243†0\.8243^\{\\dagger\}\);PFL\-Dim\-eandPFL\-Dim\+EMA\-eregress\+63%\+63\\%and\+30%\+30\\%vs\.Flat\-e\. ThePFL\-Uniform\-e→\\toPFL\-Unif\+EMA\-ecomparison suggests that EMA cannot rescue a structurally broken base allocation\.α\\alpha\-Split\-estays within0\.0030\.003absolute of theFlat\-e\(0\.19590\.1959\), the utility ceiling for any per\-layer method at the same\(ε,δ\)\(\\varepsilon,\\delta\)\(Section[IV\-A4](https://arxiv.org/html/2609.11762#S4.SS1.SSS4)\)\. RepeatingFlat\-e,PFL\-Dim\+EMA\-e, andα\\alpha\-Split\-ewith two additional random seeds yields std≤0\.02\\leq 0\.02for all three, confirming the rankings are stable across training runs\.

On Whisper\+EuroLLM the same replicates on a stronger multilingual backbone, confirming that cross\-component degradation is architectural\.PFL\-Dim\-e\(0\.19750\.1975\) andPFL\-Dim\+EMA\-e\(0\.18450\.1845\) regress vs\.Flat\-e\(0\.14570\.1457\);PFL\-Unif\+EMA\-ecollapses to0\.47690\.4769\.α\\alpha\-Split\-e\(0\.16000\.1600\) is again the only per\-layer method avoiding regression, reducing the gap toFlat\-efrom\+0\.0388\+0\.0388to\+0\.0143\+0\.0143\. BothFlat\-e\(−0\.0300\-0\.0300\) andα\\alpha\-Split\-e\(−0\.0157\-0\.0157\)*outperform*the no\-DP FedAvg baseline \(0\.17570\.1757\), consistent with gradient clipping acting as implicit regularisation under the unstable default encoder lr\.

TABLE III:WER on MLStestsplit \(19,492 samples,C=1\.0C\{=\}1\.0,σ=0\.1\\sigma\{=\}0\.1, 40 FL rounds\)\. Gap = WER−\-FedAvg \(no DP\) per group\.†\\dagger: collapsed \(WER\>0\.5\\mathrm\{WER\}\{\>\}0\.5\)\.∗\*: mean over 3 seeds; std≤0\.02\\leq\\\!0\.02\.Bold: best DP result per group/column; —: not evaluated\.Enc\. frozenEnc\. unfrozenMethodWERGapWERGapWhisper\+TinyLlama\-1\.1BFedAvg \(no DP\)0\.1415—0\.1842—Flat0\.1685\+0\.0270\+0\.02700\.1959∗0\.1959^\{\*\}\+0\.0117\+0\.0117PFL\-Uniform0\.4643\+0\.3228\+0\.32280\.9944†0\.9944^\{\\dagger\}\+0\.8102†\+0\.8102^\{\\dagger\}PFL\-Dim0\.1711\+0\.0296\+0\.02960\.3051\+0\.1209\+0\.1209PFL\-Dim\+EMA0\.1657\+0\.0242\+0\.02420\.2599∗0\.2599^\{\*\}\+0\.0757\+0\.0757PFL\-Unif\+EMA0\.2360\+0\.0945\+0\.09450\.8243†0\.8243^\{\\dagger\}\+0\.6401†\+0\.6401^\{\\dagger\}α\\alpha\-Split0\.1677\+0\.0262\+0\.02620\.1986∗\\mathbf\{0\.1986\}^\{\*\}\+0\.0144\+0\.0144Whisper\+EuroLLM\-1\.7B\-InstructFedAvg \(no DP\)0\.1330—0\.1757—Flat0\.1444\+0\.0114\+0\.01140\.1457−0\.0300\-0\.0300PFL\-Uniform0\.3500\+0\.2170\+0\.2170——PFL\-Dim0\.1381\+0\.0051\+0\.00510\.1975\+0\.0218\+0\.0218PFL\-Dim\+EMA0\.1362\+0\.0032\+0\.00320\.1845\+0\.0088\+0\.0088PFL\-Unif\+EMA0\.1780\+0\.0450\+0\.04500\.4769\+0\.3012\+0\.3012α\\alpha\-Split0\.1458\+0\.0128\+0\.01280\.1600−0\.0157\-0\.0157Voxtral\-Mini\-3BFedAvg \(no DP\)0\.1442—0\.1362—Flat0\.1478\+0\.0036\+0\.00360\.1467\+0\.0105\+0\.0105PFL\-Dim\+EMA0\.1450\+0\.0008\+0\.00080\.1413\+0\.0051\+0\.0051α\\alpha\-Split——0\.1495\+0\.0133\+0\.0133On Voxtral\-Mini\-3B the regime differs: a dedicated no\-DP profiling run confirms a steady\-state LLM/encoder norm ratio of only≈1\.7×\{\\approx\}1\.7\{\\times\}\(vs\.≈12×\{\\approx\}12\{\\times\}for Whisper\+TinyLlama\)\. At this milder imbalance,PFL\-Dim\+EMA\-edoes*not*collapse \(0\.14130\.1413,\+0\.0051\+0\.0051\); EMA adaptation suffices without structural pool separation\.α\\alpha\-Split\-e\(0\.14950\.1495\) is the*worst*DP method: the fixedα=0\.05\\alpha\{=\}0\.05over\-clips the encoder \(clip/norm≈0\.04\{\\approx\}0\.04\) whose gradient magnitude is comparable to the LLM at this ratio\. Together, the three architectures delineate the operating regime:α=0\.05\\alpha\{=\}0\.05is optimal when the encoder/LLM norm ratio is≥12×\{\\geq\}12\{\\times\};PFL\-Dim\+EMA\-eis the safer default when the ratio is unknown or substantially below12×12\{\\times\}\.

## VIDiscussion

Althoughα=0\.05\\alpha=0\.05is well\-calibrated for the extreme≈12×\{\\approx\}12\{\\times\}norm imbalance in Whisper \+ TinyLlama architecture, it should not be applied blindly\. In practice,α\\alphacan be selected dynamically by observing component delta\-norms during a few non\-DP warmup rounds, as shown in Table[II](https://arxiv.org/html/2609.11762#S4.T2)\. Architectures with milder imbalances \(e\.g\., Voxtral at≈1\.7×\{\\approx\}1\.7\{\\times\}\) do not require a structural pool split, and adaptive single\-pool methods likePFL\-Dim\+EMAsuffice\. Blindly imposingα=0\.05\\alpha=0\.05in such regimes over\-clips the encoder, as confirmed by our Voxtral results in Table[III](https://arxiv.org/html/2609.11762#S5.T3)\.

While unfreezing the encoder under non\-private FedAvg degrades performance due to default learning rate sensitivity in Whisper \+ TinyLlama \(yet improves Voxtral\-Mini\-3B\), DP clipping actually outperforms this baseline on EuroLLM\. This might appear to indicate a poorly tuned baseline, but controlled learning\-rate sweep shows that optimising the encoder lr \(×0\.02\{\\times\}0\.02\) brings no\-DP FedAvg to WER0\.11960\.1196, far below every DP result, see Appendix for further details\. The DP\-over\-FedAvg result at default lr reflects that DP sensitivity bounds act as an implicit regularizer, dampening divergent encoder updates under unstable optimisation scales\[[21](https://arxiv.org/html/2609.11762#bib.bib21)\]\. The Table[III](https://arxiv.org/html/2609.11762#S5.T3)comparison is internally consistent: all DP methods use the same default lr as the FedAvg baseline, so the relative DP–FedAvg gap is meaningful; only the absolute WER level would shift under a tuned lr\. Crucially, encoder learning rate tuning andα\\alpha\-split are complementary, not interchangeable\. Reducing the encoder learning rate \(e\.g\., by50×50\{\\times\}\) shrinks the cross\-component norm gap without altering the DP mechanism, but it lowers the encoder’s gradient signal\-to\-noise ratio under DP noise\. By contrast, our proposedα\\alpha\-split formulation preserves the encoder’s full learning rate while granting a4\.47×4\.47\{\\times\}tighter component\-specific privacy protection, structurally resolving the bottleneck of excessive gradient truncation\.

## VIIConclusion

We identified*cross\-component budget collapse*as a failure of single\-pool per\-layer DP clipping in Speech\-LLMs, where encoder updates dilute the LLM clipping budget and cause severe utility regressions\. Our proposedα\\alpha\-Split\-edesign resolves this pathology by decoupling the encoder and LLM updates into independent pools\. At an architecture\-calibratedα=0\.05\\alpha=0\.05,α\\alpha\-Split\-eachieves utility close to both flat DP and non\-DP baselines on Whisper\+TinyLlama, while providing4\.47×4\.47\\timestighter biometric privacy protection for the speech encoder\. On the Whisper\+EuroLLM architecture,α\\alpha\-Split\-eyields the best per\-layer DP results, outperforming our adaptivePFL\-Dim\+EMA\-ebaseline by2\.45%2\.45\\%absolute—confirming that cross\-component budget collapse is structural and independent of the LLM backbone\. Conversely, on Voxtral\-Mini\-3B, which exhibits a mild gradient\-norm imbalance \(≈1\.7×\{\\approx\}1\.7\{\\times\}\), our adaptivePFL\-Dim\+EMA\-eformulation achieves the best overall DP utility\. Our findings establish clear deployment boundaries: a structuralα\\alpha\-splitis optimal for extreme component imbalances \(≥12×\\geq 12\{\\times\}\), whereas the dynamicPFL\-Dim\+EMA\-eserves as the more robust default for balanced architectures\.

## AI\-Generated Content Disclosure

We used a generative AI to assist in paraphrasing, improving clarity and grammar in parts of the manuscript and coding assistance\. All generated content was reviewed and validated by the authors\.

## Acknowledgment

This work has received funding from the European Union’s Horizon Europe research and innovation programme under the project ELOQUENCE \(Grant Agreement No\. 101135916\)\. This work was supported by computational resources from the EuroHPC Joint Undertaking under the EuroHPC AI Factory grant EHPC\-AIF\-2026LS01\-004\.

## References

- \[1\]W\. Cui, D\. Yu, X\. Jiao, Z\. Meng, G\. Zhang, Q\. Wang, S\. Y\. Guo, and I\. King, “Recent advances in speech language models: A survey,” in*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar, Eds\. Vienna, Austria: Association for Computational Linguistics, Jul\. 2025, pp\. 13 943–13 970\. \[Online\]\. Available:[https://aclanthology\.org/2025\.acl\-long\.682/](https://aclanthology.org/2025.acl-long.682/)
- \[2\]X\. Fan, Z\. Sun, Gao*et al\.*, “What makes a good speech tokenizer for llm\-centric speech generation? a systematic study,” in*Proceedings of the Fortieth AAAI Conference on Artificial Intelligence and Thirty\-Eighth Conference on Innovative Applications of Artificial Intelligence and Sixteenth Symposium on Educational Advances in Artificial Intelligence*, ser\. AAAI’26/IAAI’26/EAAI’26\. AAAI Press, 2026\. \[Online\]\. Available:[https://doi\.org/10\.1609/aaai\.v40i36\.40318](https://doi.org/10.1609/aaai.v40i36.40318)
- \[3\]Z\. Ma, G\. Yang, Y\. Yang, Z\. Gao, J\. Wang, Z\. Du, F\. Yu, Q\. Chen, S\. Zheng, S\. Zhang, and X\. Chen, “An embarrassingly simple approach for llm with strong asr capacity,”*CoRR*, vol\. abs/2402\.08846, 2024\. \[Online\]\. Available:[https://doi\.org/10\.48550/arXiv\.2402\.08846](https://doi.org/10.48550/arXiv.2402.08846)
- \[4\]X\. Zhang, D\. Zhang, S\. Li, Y\. Zhou, and X\. Qiu, “Speechtokenizer: Unified speech tokenizer for speech language models,” in*The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024*\. OpenReview\.net, 2024\. \[Online\]\. Available:[https://openreview\.net/forum?id=AF9Q8Vip84](https://openreview.net/forum?id=AF9Q8Vip84)
- \[5\]B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. Agüera y Arcas, “Communication\-efficient learning of deep networks from decentralized data,” in*Proceedings of the 20th International Conference on Artificial Intelligence and Statistics*, ser\. Proceedings of Machine Learning Research, vol\. 54\. PMLR, 2017, pp\. 1273–1282\.
- \[6\]P\. Kairouz, H\. B\. McMahan, B\. Avent, A\. Bellet, M\. Bennis, A\. N\. Bhagoji, K\. Bonawitz, Z\. Charles, G\. Cormode, R\. Cummings*et al\.*, “Advances and open problems in federated learning,”*Foundations and Trends in Machine Learning*, vol\. 14, no\. 1–2, pp\. 1–210, 2021\.
- \[7\]C\. Dwork, “Differential privacy,” in*Proceedings of the 33rd International Conference on Automata, Languages and Programming \- Volume Part II*, ser\. ICALP’06\. Berlin, Heidelberg: Springer\-Verlag, 2006, p\. 1–12\. \[Online\]\. Available:[https://doi\.org/10\.1007/11787006\_1](https://doi.org/10.1007/11787006_1)
- \[8\]C\. Dwork and A\. Roth,*The Algorithmic Foundations of Differential Privacy*\. Now Publishers, 2014, vol\. 9, no\. 3–4\.
- \[9\]H\. B\. McMahan, D\. Ramage, K\. Talwar, and L\. Zhang, “Learning differentially private recurrent language models,” in*International Conference on Learning Representations*, 2018\. \[Online\]\. Available:[https://openreview\.net/forum?id=BJ0hF1Z0b](https://openreview.net/forum?id=BJ0hF1Z0b)
- \[10\]M\. Pelikan, S\. S\. Azam, V\. Feldman, J\. H\. Silovsky, K\. Talwar, C\. Brinton, and T\. Likhomanenko, “Enabling differentially private federated learning for speech recognition: Benchmarks, adaptive optimizers, and gradient clipping,” in*Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diago, CA, USA, December 2\-7*, 2025\. \[Online\]\. Available:[https://papers\.nips\.cc/paper\_files/paper/2025/hash/a0f30168b3acb17101cbb1ea46e76aac\-Abstract\-Conference\.html](https://papers.nips.cc/paper_files/paper/2025/hash/a0f30168b3acb17101cbb1ea46e76aac-Abstract-Conference.html)
- \[11\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. McLeavey, and I\. Sutskever, “Robust speech recognition via large\-scale weak supervision,” in*Proceedings of the 40th International Conference on Machine Learning*, ser\. Proceedings of Machine Learning Research, vol\. 202\. PMLR, 2023, pp\. 28 492–28 518\.
- \[12\]P\. Zhang, G\. Zeng, T\. Wang, and W\. Lu, “TinyLlama: An open\-source small language model,”*arXiv preprint arXiv:2401\.02385*, 2024\. \[Online\]\. Available:[https://arxiv\.org/abs/2401\.02385](https://arxiv.org/abs/2401.02385)
- \[13\]P\. H\. Martins, P\. Fernandes, J\. Alves, A\. Agrawal, M\. A\. Farajian, R\. Lopes, R\. Rei, P\. Zaremoodi, N\. M\. Guerreiro, A\. F\. T\. Martins*et al\.*, “EuroLLM: Multilingual language models for europe,”*arXiv preprint arXiv:2409\.16235*, 2024\. \[Online\]\. Available:[https://arxiv\.org/abs/2409\.16235](https://arxiv.org/abs/2409.16235)
- \[14\]H\. Liu*et al\.*, “Voxtral,”*arXiv preprint arXiv:2507\.13264*, 2025\. \[Online\]\. Available:[https://arxiv\.org/abs/2507\.13264](https://arxiv.org/abs/2507.13264)
- \[15\]——, “Ministral 3,” 2026\. \[Online\]\. Available:[https://arxiv\.org/abs/2601\.08584](https://arxiv.org/abs/2601.08584)
- \[16\]I\. Mironov, “Rényi differential privacy,” in*Proceedings of the 30th IEEE Computer Security Foundations Symposium*\. IEEE, 2017, pp\. 263–275\.
- \[17\]V\. Pratap, Q\. Xu, A\. Sriram, G\. Synnaeve, and R\. Collobert, “MLS: A large\-scale multilingual dataset for speech research,” in*Proceedings of Interspeech*, 2020, pp\. 2757–2761\.
- \[18\]D\. J\. Beutel, T\. Topal, A\. Mathur, X\. Qiu, T\. Parcollet, and N\. D\. Lane, “Flower: A friendly federated learning research framework,” in*arXiv preprint arXiv:2007\.14390*, 2020\. \[Online\]\. Available:[https://arxiv\.org/abs/2007\.14390](https://arxiv.org/abs/2007.14390)
- \[19\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen, “LoRA: Low\-rank adaptation of large language models,” in*International Conference on Learning Representations*, 2022\. \[Online\]\. Available:[https://openreview\.net/forum?id=nZeVKeeFYf9](https://openreview.net/forum?id=nZeVKeeFYf9)
- \[20\]M\. Bisani and H\. Ney, “Bootstrap estimates for confidence intervals in ASR performance evaluation,” in*Proceedings of ICASSP*, vol\. 1, 2004, pp\. I–409–I–412\.
- \[21\]Z\. Bu, U\. Muthukumar, and S\. Xu, “Automatic clipping: Differentially private deep learning made easier and stronger,” in*Advances in Neural Information Processing Systems*, vol\. 35, 2022, pp\. 17 343–17 356\.

## Appendix AEuroLLM Encoder Learning\-Rate Sensitivity

The EuroLLM unfrozen results in Table[III](https://arxiv.org/html/2609.11762#S5.T3)use a default encoder learning rate equal to the LLM’s \(multiplier×1\.0\\times 1\.0\)\. Because EuroLLM\-1\.7B produces stronger LLM gradient signals than TinyLlama\-1\.1B, the encoder is dominated in training unless its learning rate \(lr\) is reduced\. Table[IV](https://arxiv.org/html/2609.11762#A1.T4)shows WER as a function of the encoder learning rate \(enc\-lr\) multiplier for FedAvg \(no DP, client local epochsE=10E=10, over 40 FL rounds\)\. The best non\-DP result, enc\-lr×0\.02\\times 0\.02/ le5 \(WER0\.11960\.1196\), closely matches the frozen EuroLLM baseline \(0\.13300\.1330\) and substantially improves over the default lr baseline \(0\.17570\.1757, Table[IV](https://arxiv.org/html/2609.11762#A1.T4)\)\.

TABLE IV:Non\-DP FedAvg WER vs\. encoder learning rate multiplier, Whisper\+EuroLLM unfrozen \(MLS test partition, 40 rounds\)\.Enc\. LR mult\.Local ep\.WER×0\.02\\times 0\.0250\.1196×0\.02\\times 0\.02100\.1264×0\.05\\times 0\.0550\.1325×0\.10\\times 0\.1050\.1343×0\.20\\times 0\.2050\.1598×0\.02\\times 0\.02\(enc\+llm\)100\.1393×1\.00\\times 1\.00\(default\)100\.1757×1\.00\\times 1\.00\(default\)50\.1861Table[V](https://arxiv.org/html/2609.11762#A1.T5)reports DP results at enc\-lr×0\.02\\times 0\.02\(encoder only\), using the matched no\-DP baseline of WER0\.12640\.1264\(enc\-lr×0\.02\\times 0\.02, E=10\)\. PFL\-Dim\+EMA\-e improves dramatically from0\.18450\.1845\(default lr, Table[III](https://arxiv.org/html/2609.11762#S5.T3)\) to0\.1440\\mathbf\{0\.1440\}\(\+0\.0176\+0\.0176\), now*outperforming*bothα\\alpha\-Split\-e\(0\.15110\.1511,\+0\.0247\+0\.0247\) andFlat\-e\(0\.15250\.1525,\+0\.0261\+0\.0261\)\.

TABLE V:DP WER at encoder learning rate×0\.02\\times 0\.02\(encoder only\), Whisper\+EuroLLM unfrozen \(C=1\.0C\{=\}1\.0,σ=0\.1\\sigma\{=\}0\.1, E=10\)\. Gap vs\. matched no\-DP baseline \(enc\-lr×0\.02\\times 0\.02, le10, WER0\.12640\.1264\)\.Bold: best DP result\.MethodWERGapFedAvg \(no DP\)0\.1264—Flat\-e0\.1525\+0\.0261\+0\.0261PFL\-Dim\+EMA\-e0\.1440\+0\.0176\+0\.0176α\\alpha\-Split\-e0\.1511\+0\.0247\+0\.0247This reversal result directly confirms the cross\-component collapse diagnosis\. At the default encoder lr \(×1\.0\\times 1\.0\), encoder update norms are∼\\sim10×\\timeslarger than individual LLM\-layer norms; the 64 encoder matrices monopolise clip\-budget slots in the single shared pool, leaving the LLM under\-served\. Reducing the encoder lr by×50\\times 50shrinks encoder update norms proportionally, eliminating the inter\-component norm imbalance*without any change to the DP mechanism*\. With the imbalance removed,PFL\-Dim\+EMA\-e’s global adaptive allocation can concentrate the full budgetCCon the LLM layers where gradients are large, recovering near\-flat utility\. By contrast,α\\alpha\-Split\-eenforces a fixed5%5\\%/95%95\\%\(tuned on Whisper \+ TinyLlama architecture\) encoder/LLM pool split regardless of actual norms; with encoder updates already attenuated by the reduced lr, the encoder pool is structurally under\-utilised and the LLM pool is marginally tighter than a globally adaptive method would choose—givingPFL\-Dim\+EMA\-ea small but consistent edge\.

Importantly, lr tuning andα\\alpha\-split are complementary, not interchangeable: the×50\\times 50lr reduction attenuates the encoder’s gradient signal \(lower signal\-to\-noise ratio under DP noise\), whereasα\\alpha\-split preserves the encoder’s full lr while granting it4×4\\timestighter\(ε,δ\)\(\\varepsilon,\\delta\)\-DP protection\. The enc\-lr×0\.02\\times 0\.02result therefore serves as a controlled ablation: it proves that the collapse observed at default lr is driven by the encoder/LLM gradient\-norm imbalance\.

## Appendix BMLS Corpus Granular Statistics and Speaker Overlap Analysis

This appendix provides the granular structural details of the Multilingual LibriSpeech \(MLS\) speaker\-based client partition \(K=316K=316clients\) to complement the high\-level dataset overview presented in Section II\-C\.

### B\-AClient Volume and Linguistic Imbalances

The stratified speaker partition induces an extreme, multi\-dimensional data imbalance across both languages and individual clients\. While English clients dominate the network numerically, they represent a small fraction of the total training hours\. Conversely, Continental European languages are represented by a few highly active clients with massive local datasets\.

Specifically, the partition exhibits the following structural characteristics:

- •High\-Client, Low\-Volume Regimes \(English\):English represents81\.0%81\.0\\%of the total client population \(256256out of316316clients\) but accounts for only15\.4%15\.4\\%\(105\.3105\.3hours\) of the total training data\. This yields an average training volume of only0\.410\.41hours \(24\.724\.7minutes\) per English client\.
- •Low\-Client, High\-Volume Regimes \(Polish\):Polish represents exactly0\.31%0\.31\\%of the client population \(11client\) but holds3\.7%3\.7\\%of the total training data \(25\.725\.7hours\)\. The single Polish client has over62×62\\timesmore training audio than the average English client\.
- •Highly Symmetric Clusters \(German/Spanish\):German and Spanish exhibit a moderate balance between client counts and data volume, with Spanish clients averaging9\.319\.31hours and German clients averaging8\.478\.47hours\.

This severe volume disparity \(ranging from2424minutes to over2525hours per client\) provides a challenging non\-IID optimization landscape\. Standard FedAvg updates are weighted by the local client sample countnkn\_\{k\}, meaning a tiny fraction of highly active European clients can disproportionately influence global gradient updates during a given round\. Table[VI](https://arxiv.org/html/2609.11762#A2.T6)outlines the complete language\-by\-language breakdown of training hours, client counts, and average local data volumes\.

TABLE VI:Granular breakdown of the MLS training pool under the speaker\-based partition \(K=316K=316clients\)\.LanguageTraining HoursClients \(KK\)Avg\. Hours/ClientFrench251\.61516\.77German160\.9198\.47English105\.32560\.41Spanish83\.899\.31Italian27\.373\.90Polish25\.7125\.70Portuguese18\.553\.70Dutch12\.743\.18Total685\.73162\.17
### B\-BSpeaker Overlap Mechanics in audiobook Corpora

Because the MLS corpus is derived from public\-domain LibriVox audiobooks, the training, validation, and evaluation splits are subject to the inherent constraints of audiobook recording structures\. In LibriVox, a single volunteer reader \(speaker\) frequently contributes to multiple books, or records independent chapters across different volumes\.

To maximize the acoustic diversity of the dataset, established central ASR benchmarks split these recorded chapters across training, validation, and test sets\. When simulating a speaker\-based cross\-device FL network, where each unique speaker is mapped to a single client node, this underlying splits design introduces a2\.5%2\.5\\%speaker overlap\. Specifically,88out of the316316clients \(2\.8%2\.8\\%of total training samples\) represent speakers who also appear in the test split\.

We explicitly maintain this standard partition for three critical methodological reasons:

1. 1\.Generalization vs\. Personalization:It allows us to evaluate the model’s generalized performance on entirely unseen speakers, while simultaneously observing how well the model adapts to unseen utterances from speakers who were present in the training set\.
2. 2\.Disjoint Utterances:No individual audio sample \(utterance\) is shared between splits\. The validation and test utterances are entirely disjoint from the training data, preventing direct memorization or data leakage\.
3. 3\.Benchmark Alignment:Preserving this speaker allocation ensures our FL results are directly comparable to existing centralized and federated benchmarks in the ASR literature, maintaining empirical continuity\.