RAPTOR: Role-Aware Private Training for Mixture-of-Experts

# RAPTOR: Role-Aware Private Training for Mixture-of-Experts
Source: [https://arxiv.org/html/2609.05770](https://arxiv.org/html/2609.05770)
Khai Le\-DucAffiliation:Vector InstituteAffiliation:Knovel Engineering LabNguyen DoMinh Son Hoang, Florent Draye, Thai Hoang, Hoang Phuong Dam, Jiarui LiuAffiliation:KAISTAffiliation:MPI for Intelligent Systems, TübingenAffiliation:Salesforce AI ResearchAffiliation:Carnegie Mellon UniversityChris Ngo, Terry Jingchen Zhang, Anh Le Duc Tran, Nhat Do Minh, Minh Ngoc LeAffiliation:University of TorontoAffiliation:Vector InstituteAffiliation:Knovel Engineering LabAffiliation:University of OxfordAffiliation:Jinesis Lab, University of Toronto & Vector InstituteAffiliation:Hanoi University of Science and TechnologyAffiliation:Vietnam National University, HanoiMy T\. Thai, Ran Xu, Silvio Savarese, Mona DiabAffiliation:University of FloridaAffiliation:Salesforce AI ResearchAffiliation:Carnegie Mellon UniversityAffiliation:Stanford UniversityBernhard Schölkopf, Zhijing Jin, Huy L\. Nguyen, Daeyoung KimAffiliation:KAISTAffiliation:MPI for Intelligent Systems, TübingenAffiliation:Jinesis Lab, University of Toronto & Vector InstituteAffiliation:ELLIS Institute TübingenAffiliation:Northeastern UniversityAffiliation:Co\-first authorsducdm200158@kaist\.ac\.krduckhai\.le@mail\.utoronto\.ca[leduckhai/RAPTOR](https://github.com/leduckhai/RAPTOR)

###### Abstract

Differentially private \(DP\) fine\-tuning methods treat sparse Mixture\-of\-Experts \(MoE\) models as a single dense block, ignoring that shared layers see all data while experts only see routed records\. We identify and formally characterize three resulting failure modes: global clipping suppresses expert gradients, batch\-level normalization dilutes sparse expert updates, and fixed privacy noise degrades signal\-to\-noise ratio on low\-load experts\. We introduce![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR\- aRole\-Aware Private Trainingframework, which alternates shared and expert optimization and targets each failure directly, using expert\-specific clipping and noise together with a public expected\-owner denominator and a count\-independent update schedule that avoids conditioning on private, realized expert counts\. We prove the resulting mechanism satisfies\(ε,δ\)\(\\varepsilon,\\delta\)\-DP: because each record is assigned to exactly one owner expert, per\-expert mechanisms within a layer compose in parallel, so updating allEEexperts costs no more, in privacy terms, than updating one, with shared and expert streams composing sequentially across training\. We further derive a bias\-variance decomposition of the public\-denominator estimator showing its bias grows predictably with routing imbalance, yielding a privacy\-free rule for selecting which layer to protect from routing entropy measured on a small public corpus\. Experiments on Switch Transformer and OLMoE fine\-tuning across GLUE tasks, and on the DeepSeek\-VL2\-Tiny, show consistent gains over standard DP baselines across several privacy levels \(ε\\varepsilon\), with the largest margins typically at the tightest budgets\.

![Refer to caption](https://arxiv.org/html/2609.05770v1/images/teaser-v5.png)Figure 1:Applying differential privacy to sparse MoE is not straightforward: treating the model as dense leads to three structural failures \- clipping interference, update\-scale dilution, and load\-dependent signal degradation\. Our role\-aware method is designed to address these mismatches\.## 1Introduction

Sparse Mixture\-of\-Experts \(MoE\) architectures decouple model capacity from per\-record computation, making conditional sparsity a central strategy for scaling foundation models[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.05770#bib.bib6);[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib7);[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib15)\. Yet many of their highest\-value adaptations \- on medical records, enterprise documents, personal interactions, and other proprietary corpora \- require formal protection against memorization, for which differential privacy \(DP\)[Dwork and Roth \(2014\)](https://arxiv.org/html/2609.05770#bib.bib2), enforced via per\-record gradient clipping and Gaussian noise[Abadi et al\. \(2016\)](https://arxiv.org/html/2609.05770#bib.bib1), is the prevailing standard\. Combining the two is not a straightforward application of existing tools: DP optimizers were designed for dense models, where every parameter receives signal from every record, so one clipping bound, one noise scale, and one normalization denominator suffice\. Sparse routing breaks this premise \- shared components still see every record, while each expert sees only the records routed to it \- and the only prior work on private MoE training[Tholoniat et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib8)sidesteps the asymmetry by treating the model as a single dense block\. We show this mismatch is a structural incompatibility, not an implementation detail \(Fig\.[1](https://arxiv.org/html/2609.05770#S0.F1), left\): monolithic DP training suppresses expert gradients dominated by denser blocks \(*clipping interference*, panel 1\), dilutes sparse expert updates through full\-batch normalization \(*update\-scale dilution*, panel 2\), and concentrates fixed privacy noise on low\-load experts \(*load\-dependent SNR degradation*, panel 3\) \- together undermining the expert specialization that makes MoEs useful \(Sec\.[4](https://arxiv.org/html/2609.05770#S4)\)\.

To address these limitations, we propose![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR\(Role\-Aware Private Training\), which aligns the DP mechanism with the shared/expert role structure rather than ignoring it \(Fig\.[1](https://arxiv.org/html/2609.05770#S0.F1), right\)\. The*shared stream*updates dense components on all records via standard per\-record clipping and noise\. The*expert stream*updates each expert only on its owner records, with expert\-specific clippingCeC\_\{e\}and noiseσe\\sigma\_\{e\}\(resolving clipping interference, panel 1\), a public expected\-owner denominatorDe=B/ED\_\{e\}=B/Ethat never touches realized counts \(resolving dilution without leaking routing decisions, panel 2\), a count\-independent schedule that updates every expert at every step, and privacy\-free entropy\-based layer selection \(resolving load\-dependent SNR degradation, panel 3\)\. Fig\.[2](https://arxiv.org/html/2609.05770#S1.F2)gives an overview\.

![Refer to caption](https://arxiv.org/html/2609.05770v1/images/overview_arch2.png)Figure 2:Overview of![Refer to caption](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORat the selected sparse layerl⋆l^\{\\star\}\.\(a\)The frozen router performs the base MoE model’s standard token\-level top\-kkrouting\. Our framework aggregates gate weights across all tokens of recordxix\_\{i\}and selects one deterministic*owner expert*al⋆​\(xi\)a\_\{l^\{\\star\}\}\(x\_\{i\}\)\. The forward pass is unchanged: all activated experts contribute, but only the owner receives the expert\-parameter gradient; activated non\-owner experts use stop\-gradient\. This assignment partitions𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}into disjoint owner groups𝒟l⋆,e=\{\(xi,yi\):al⋆​\(xi\)=e\}\\mathcal\{D\}\_\{l^\{\\star\},e\}=\\\{\(x\_\{i\},y\_\{i\}\):a\_\{l^\{\\star\}\}\(x\_\{i\}\)=e\\\}\.\(b\)Each cycle draws independent Poisson subsamples for the shared and expert streams\. Shared updates use clippingCsC\_\{s\}, noise multiplierσs\\sigma\_\{s\}, and denominatorBB\. Expert records are partitioned by owner and updated usingCeC\_\{e\},σe\\sigma\_\{e\}, and the public denominatorDe=B/ED\_\{e\}=B/E, with noise\-only updates for empty subsets\. Expert mechanisms compose in parallel withinl⋆l^\{\\star\}, while shared and expert releases compose sequentially, withεsh=ρ​ε\\varepsilon\_\{\\mathrm\{sh\}\}=\\rho\\varepsilonandεexp=\(1−ρ\)​ε\\varepsilon\_\{\\mathrm\{exp\}\}=\(1\-\\rho\)\\varepsilon\.#### Contributions\.

1. 1\.Diagnostic analysis\.We identify and formally characterize three structural failure modes of monolithic DP\-MoE training \- clipping interference, update\-scale dilution, and load\-dependent SNR degradation \(Sec\.[4](https://arxiv.org/html/2609.05770#S4)\)\.
2. 2\.Role\-aware optimization\.We propose Role\-Aware Private Training, resolving all three failures via alternating shared/expert streams, expert\-specific clipping and noise, and a count\-independent update schedule \(Sec\.[5](https://arxiv.org/html/2609.05770#S5)\)\.
3. 3\.Privacy guarantee\.We prove\(ε,δ\)\(\\varepsilon,\\delta\)\-DP via parallel composition over routing\-induced owner groups: updating allEEexperts within a layer costs no more than updating one \(Sec\.[6](https://arxiv.org/html/2609.05770#S6)\)\.
4. 4\.Utility analysis and layer selection\.We bound the public\-denominator estimator’s bias by routing imbalance, yielding the privacy\-free rulel⋆=arg⁡maxl⁡H⁡\(q\(l\)\)l^\{\\star\}=\\arg\\max\_\{l\}H\(q^\{\(l\)\}\)on a small public corpus, and show this bias is stable under bounded representation drift via a Voronoi abstraction \(Sec\.[7](https://arxiv.org/html/2609.05770#S7)\)\.
5. 5\.Empirical validation\.Gains up to\+3\.24\+3\.24points over monolithic and matched\-scope DP baselines on Switch Transformer \(four GLUE tasks,ε∈\{1,4,8\}\\varepsilon\\in\\\{1,4,8\\\}\), with consistent improvements on OLMoE and on the vision\-language DeepSeek\-VL2\-Tiny model \(Sec\.[8](https://arxiv.org/html/2609.05770#S8)\)\.

## 2Related Work

Private fine\-tuning of pretrained LMs \- full\-parameter[Li et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib3)or parameter\-efficient via LoRA[Yu et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib4);[Hu et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib5)\- applies one clipping bound, noise scale, and denominator uniformly across all parameters, a reasonable simplification for dense models where every parameter sees every record\. This breaks under MoE routing, where shared components see every record but each expert sees only its routed subset; to our knowledge, no DP fine\-tuning method distinguishes these exposure patterns\. Separately, non\-private MoE training stabilizes routing via batch\-level load\-balancing losses, which do not translate to record\-level DP: their gradients break per\-record sensitivity, and they depend on realized expert counts, which are private and cannot enter normalization or control flow without leaking routing decisions\. The only prior work on private MoE training,[Tholoniat et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib8), sidesteps this by treating MoE as a single dense block \- feasible, but leaving shared, routing, and expert parameters structurally indistinguishable, which suppresses expert gradients, dilutes sparse updates, and concentrates noise on low\-load experts\. We address this by freezing the router, replacing realized counts with a public expected\-owner denominator, and analyzing expert updates via parallel composition over disjoint owner groups \- closing the gap left by dense DP methods[Li et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib3);[Yu et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib4);[Hu et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib5), which ignore routing entirely, and by[Tholoniat et al\.](https://arxiv.org/html/2609.05770#bib.bib8), which retains a monolithic optimizer despite it\. Additional related work is in Appendix[A](https://arxiv.org/html/2609.05770#A1)\.

## 3Preliminaries

### 3\.1Setup and Notation

Let𝒟priv=\{\(xi,yi\)\}i=1n\\mathcal\{D\}\_\{\\mathrm\{priv\}\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}be the private dataset\. A sparse MoE model hasEEexperts per sparse layer \(\[E\]=\{1,…,E\}\[E\]=\\\{1,\\ldots,E\\\}\) over layersℒ\\mathcal\{L\}, with parametersθ=\(θsh,θrt,\{θl,e\}\)\\theta=\(\\theta\_\{\\mathrm\{sh\}\},\\theta\_\{\\mathrm\{rt\}\},\\\{\\theta\_\{l,e\}\\\}\)\. Each expert carries a roler⁡\(e\)∈\{routed,shared\}r\(e\)\\in\\\{\\textsc\{routed\},\\textsc\{shared\}\\\}: shared experts \(e\.g\., DeepSeek\-MoE’s always\-active experts[Dai et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib29)\) see every token and fold intoθsh\\theta\_\{\\mathrm\{sh\}\}; routed experts see only assigned tokens\. Unless stated otherwise,r⁡\(e\)=routedr\(e\)=\\textsc\{routed\}for allee\. At steptt,ℬtexp⊆𝒟priv\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\\subseteq\\mathcal\{D\}\_\{\\mathrm\{priv\}\}is Poisson\-subsampled, so its realized size is random and private; we writeB=𝔼⁡\[\|ℬtexp\|\]B=\\mathbb\{E\}\[\|\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\|\]for the fixed, public target size and normalize byBB\- never the realized count \- the principle underlyingDe=B/ED\_\{e\}=B/E\(Sec\.[5](https://arxiv.org/html/2609.05770#S5)\)\. For gradientggand radiusC\>0C\{\>\}0,clip⁡\(g,C\)=g⋅min⁡\{1,C/‖g‖2\}\\mathrm\{clip\}\(g,C\)=g\\cdot\\min\\\{1,C/\\\|g\\\|\_\{2\}\\\}; subscriptss,es,edistinguish shared\-/expert\-stream quantities, e\.g\.\(Cs,σs,ηs\)\(C\_\{s\},\\sigma\_\{s\},\\eta\_\{s\}\)vs\.\(Ce,σe,ηe\)\(C\_\{e\},\\sigma\_\{e\},\\eta\_\{e\}\)\.

### 3\.2Differential Privacy

###### Definition 1\.

[Dwork and Roth \(2014\)](https://arxiv.org/html/2609.05770#bib.bib2)𝒜\\mathcal\{A\}is\(ε,δ\)\(\\varepsilon,\\delta\)\-DP if for all neighboring𝒟,𝒟′\\mathcal\{D\},\\mathcal\{D\}^\{\\prime\}and output sets𝒮\\mathcal\{S\},Pr\[𝒜\(𝒟\)∈𝒮\]≤eεPr\[𝒜\(𝒟′\)∈𝒮\]\+δ\\Pr\[\\mathcal\{A\}\(\\mathcal\{D\}\)\\in\\mathcal\{S\}\]\\leq e^\{\\varepsilon\}\\Pr\[\\mathcal\{A\}\(\\mathcal\{D\}^\{\\prime\}\)\\in\\mathcal\{S\}\]\+\\delta\.

DP\-SGD[Abadi et al\. \(2016\)](https://arxiv.org/html/2609.05770#bib.bib1)privatizes each step’s gradient via per\-record clipping and Gaussian noise: forℬt\\mathcal\{B\}\_\{t\}of expected sizeBB,

g¯t=1B​\(∑i∈ℬtclip⁡\(gi,t,C\)\+𝒩⁡\(0,σ2​C2​I\)\)\.\\bar\{g\}\_\{t\}=\\frac\{1\}\{B\}\\\!\\left\(\\sum\_\{i\\in\\mathcal\{B\}\_\{t\}\}\\mathrm\{clip\}\(g\_\{i,t\},C\)\+\\mathcal\{N\}\(0,\\sigma^\{2\}C^\{2\}I\)\\right\)\.\(1\)We privatize every release this way, but optimize with Adam \-g¯t\\bar\{g\}\_\{t\}feeds Adam’s moment estimates in place of the raw gradient, giving*DP\-Adam*\. This is post\-processing of the already\-released sequence and adds no privacy cost beyond theTT\-fold composition already charged \(Sec\.[6\.2](https://arxiv.org/html/2609.05770#S6.SS2)\)\.

### 3\.3Sparse MoE Models and Load Balancing

Sparse MoE Transformers[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.05770#bib.bib6);[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib7)replace dense FFNs withEEexperts and a router assigning each token tokkof them; shared experts fold into the shared stream, routed experts into the expert stream, and sparse activation causes load imbalance\. Our method assigns each record deterministically to one routed expert per layer, its*owner expert*\- the expert receiving the most of the record’s tokens across thekkassignments, reducing to the standard case atk=1k\{=\}1\- the structure our parallel\-composition analysis relies on \(Sec\.[6](https://arxiv.org/html/2609.05770#S6)\)\. Auxiliary load\-balancing lossesℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}depend on batch\-level routing statistics, giving per\-record gradients that reflect other records’ decisions and break the per\-record sensitivity DP\-SGD requires; we therefore freeze the router and omitℒaux\\mathcal\{L\}\_\{\\mathrm\{aux\}\}\.

## 4Diagnosing Monolithic Private Optimization Failures

Monolithic private optimization applies one clipping bound, one noise scale, and one normalization denominator to all MoE parameters \- as if the model were dense\. This ignores that shared layers, routers, and task heads receive signal from every record, while each expert receives signal only from its routed records\. Under DP, this asymmetry causes three structural failures, each a property of the mechanism definition rather than any hyperparameter choice\. Appendix[G](https://arxiv.org/html/2609.05770#A7)formalizes this claim algebraically and reports empirical role\-mismatch and per\-expert SNR diagnostics \(Tab\.[8](https://arxiv.org/html/2609.05770#A7.T8)\) that corroborate it directionally\.

### 4\.1Clipping Interference

The per\-record gradientgi,tfull=\[gsh,grt,ghead,gexp\]i,tg\_\{i,t\}^\{\\mathrm\{full\}\}=\[g^\{\\mathrm\{sh\}\},g^\{\\mathrm\{rt\}\},g^\{\\mathrm\{head\}\},g^\{\\mathrm\{exp\}\}\]\_\{i,t\}is clipped by a single factorαi,tfull=min⁡\(1,C/‖gi,tfull‖2\)\\alpha\_\{i,t\}^\{\\mathrm\{full\}\}=\\min\(1,C/\\\|g\_\{i,t\}^\{\\mathrm\{full\}\}\\\|\_\{2\}\), dominated by the three dense blocks, so it suppresses the expert component even when‖gi,texp‖2\\\|g\_\{i,t\}^\{\\mathrm\{exp\}\}\\\|\_\{2\}alone is withinCC\. RelaxingCCdoes not resolve the coupling: noise scales withCC, so any value permissive enough for experts inflates dense\-block noise\. The natural resolution is a separate clipping bound for the expert stream, as in our method\.

### 4\.2Update\-Scale Dilution

Letmt,l,e=\|ℬt,l,e\|m\_\{t,l,e\}=\|\\mathcal\{B\}\_\{t,l,e\}\|be the number of records routed to experteein batchℬtexp\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\. The monolithic update

g^t,l,emono=1B​\(∑i∈ℬt,l,eg~i,t,l,eexp\+ξt,l,e\)\\widehat\{g\}\_\{t,l,e\}^\{\\mathrm\{mono\}\}=\\frac\{1\}\{B\}\\\!\\left\(\\sum\_\{i\\in\\mathcal\{B\}\_\{t,l,e\}\}\\widetilde\{g\}\_\{i,t,l,e\}^\{\\mathrm\{exp\}\}\+\\xi\_\{t,l,e\}\\right\)\(2\)normalizes byBBdespite onlymt,l,em\_\{t,l,e\}records contributing signal, rescaling expert gradients toward zero bymt,l,e/Bm\_\{t,l,e\}/B\. Normalizing by the realized countmt,l,em\_\{t,l,e\}is not DP\-free: it is a private quantity whose use in normalization or control flow leaks routing information\. We instead use the public expected\-owner denominatorDe=B/ED\_\{e\}=B/E, which is data\-independent and unbiased under uniform routing\.

### 4\.3Load\-Dependent Signal Degradation

Privacy noise per expert is fixed regardless of load; useful signal scales withmt,l,em\_\{t,l,e\}\. Uneven routing therefore concentrates noise on low\-load experts\. We track this with the routing entropy:

Ht,l\\displaystyle H\_\{t,l\}=−1log⁡E∑e=1Eqt,l,elogqt,l,e,\\displaystyle=\-\\frac\{1\}\{\\log E\}\\sum\_\{e=1\}^\{E\}q\_\{t,l,e\}\\log q\_\{t,l,e\},\(3\)whereqt,l,e=mt,l,e/Bq\_\{t,l,e\}=m\_\{t,l,e\}/B,τ=10−8\\tau=10^\{\-8\}denotes a small constant added for numerical stability in ratio denominators\. Sec\.[7](https://arxiv.org/html/2609.05770#S7)connectsHt,lH\_\{t,l\}and the entropy deficitlog⁡E−Ht,l\\log E\-H\_\{t,l\}to the imbalance bias of the public\-denominator expert estimator, motivating routing entropy as a candidate, privacy\-free layer\-selection diagnostic, whose relationship to Adam\-based training utility Sec\.[7](https://arxiv.org/html/2609.05770#S7)examines directly \(Remark[2](https://arxiv.org/html/2609.05770#Thmremark2)\);

## 5Role\-Aware Private Training

### 5\.1Public Layer Selection

We protect a single sparse layerl⋆l^\{\\star\}per stage\. This choice keeps simple the parallel\-composition argument we prove later \(Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1), Sec\.[6](https://arxiv.org/html/2609.05770#S6)\): updating allEEexperts within a protected layer costs no more, in privacy terms, than updating one, provided the*owner map*\- the deterministic assignment of each private record to a single owner expert at the protected layer, formalized in Sec\.[5\.2](https://arxiv.org/html/2609.05770#S5.SS2)\(Eq\.[5](https://arxiv.org/html/2609.05770#S5.E5)\) \- is fixed before training and does not change during the stage\. This condition holds trivially when only one layer is trained\. Multi\-layer extension would require a staged scheme \- train and freeze one layer’s owner map before the next, composing privacy costs sequentially \- left to future work; the main paper scopes toL=\{l⋆\}L=\\\{l^\{\\star\}\\\}\. Before private training, we select one sparse layer using external public unlabeled data𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}\(disjoint from𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}, format\-matched to the task, e\.g\. unlabeled IMDb reviews for sentiment, SQuAD contexts for entailment\)\. For each candidate layerll, we apply the frozen router to𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}, compute the empirical owner distributionq\(l\)q^\{\(l\)\}, and set

l⋆=arg⁡maxl⁡H⁡\(q\(l\)\)\.l^\{\\star\}=\\arg\\max\_\{l\}H\\\!\\left\(q^\{\(l\)\}\\right\)\.\(4\)Layer selection \(Eq\.[4](https://arxiv.org/html/2609.05770#S5.E4)\) requires a disjoint, format\-matched public corpus; without one, we default to the last sparse layer\. Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)shows first\-/last\-sparse trailing entropy\-selection on less balanced tasks \- but in one setting lacking a public corpus it still improved over the monolithic DP baseline, suggesting the core mechanism does not strictly require entropy\-based selection\. Since𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}is public and the router is frozen, this consumes*no privacy budget*and does not touch𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\. Sec\.[7](https://arxiv.org/html/2609.05770#S7)shows entropy provably shrinks the public\-denominator estimator’s biasβe\\beta\_\{e\}\(Theorem[2](https://arxiv.org/html/2609.05770#Thmtheorem2)\); Remark[2](https://arxiv.org/html/2609.05770#Thmremark2)clarifies this does not extend to a guarantee on per\-expert signal\-to\-noise ratio, whose benefit for Adam\-based training we support empirically \(Sec\.[8](https://arxiv.org/html/2609.05770#S8)\) rather than by an analogous bound\.

### 5\.2Record\-Level Expert Ownership

At layerll, the frozen router gives each tokent∈Tok⁡\(xi\)t\\in\\mathrm\{Tok\}\(x\_\{i\}\)a top\-kkexpert set with gate weightsgl,e​\(xi,t\)≥0g\_\{l,e\}\(x\_\{i\},t\)\\geq 0\(gl,e=0g\_\{l,e\}=0off the top\-kk,∑egl,e=1\\sum\_\{e\}g\_\{l,e\}=1\)\. We lift this to a deterministic record\-level owner map by gate\-weighted plurality,

al\(xi\)=arg​maxe∈\[E\]∑t∈Tok⁡\(xi\)gl,e\(xi,t\),a\_\{l\}\(x\_\{i\}\)=\\operatorname\*\{arg\\,max\}\_\{e\\in\[E\]\}\\sum\_\{t\\in\\mathrm\{Tok\}\(x\_\{i\}\)\}g\_\{l,e\}\(x\_\{i\},t\),\(5\)ties broken by lowest index; replacinggl,eg\_\{l,e\}by𝟙\[e∈top\-k\]\\mathbb\{1\}\[e\\in\\text\{top\-\}k\]recovers the majority\-vote rule atk=1k=1\. This induces a deterministic partition of𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}into*owner groups*𝒟l,e\\mathcal\{D\}\_\{l,e\}, one per experte∈\[E\]e\\in\[E\], with⋃e𝒟l,e=𝒟priv\\bigcup\_\{e\}\\mathcal\{D\}\_\{l,e\}=\\mathcal\{D\}\_\{\\mathrm\{priv\}\}for everyk≥1k\\geq 1\. The forward pass uses standard top\-kkrouting, but the*expert stream*deposits gradient only into the owner expert’s parameters via a stop\-gradientsg⁡\(⋅\)\\mathrm\{sg\}\(\\cdot\)on all non\-owner experts:

ai\\displaystyle a\_\{i\}≔al​\(xi\),\\displaystyle\\coloneqq a\_\{l\}\(x\_\{i\}\),\(6\)gi,lexp\\displaystyle g^\{\\exp\}\_\{i,l\}=∇θl,aiℓexp​\(xi,θl,ai,θl,¬aisg\),\\displaystyle=\\nabla\_\{\\theta\_\{l,a\_\{i\}\}\}\\ell\_\{\\mathrm\{exp\}\}\\\!\\left\(x\_\{i\};\\theta\_\{l,a\_\{i\}\},\\theta^\{\\mathrm\{sg\}\}\_\{l,\\neg a\_\{i\}\}\\right\),so each record contributes to exactly one owner expert; only the attribution of private expert updates is restricted\. Under add/remove adjacency, one record changes only group𝒟l,al​\(x0\)\\mathcal\{D\}\_\{l,a\_\{l\}\(x\_\{0\}\)\}, by a single clipped contribution of norm≤Ce\\leq C\_\{e\}\. HenceΔl,e=Ce/De=E​Ce/B\\Delta\_\{l,e\}=C\_\{e\}/D\_\{e\}=EC\_\{e\}/B, independent of\|𝒟l,e\|\|\\mathcal\{D\}\_\{l,e\}\|andkk, and conditions \(i\)\-\(iii\) of Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1)\(Sec\.[6\.1](https://arxiv.org/html/2609.05770#S6.SS1)\) hold verbatim for anykk\. The owner map and realized counts\|𝒟l,e\|\|\\mathcal\{D\}\_\{l,e\}\|are never released;al⋆a\_\{l^\{\\star\}\}is computed once from the frozen router before training and fixed thereafter\.

### 5\.3Shared and Expert Stream Updates

#### Shared stream\.

Every record contributes to the shared loss, so the shared stream computes a standard privatized estimate over its own subsample:

g¯tsh=1B​\(∑i∈ℬtshclip⁡\(gi,tsh,Cs\)\+𝒩⁡\(0,σs2​Cs2​I\)\)\.\\bar\{g\}\_\{t\}^\{\\mathrm\{sh\}\}=\\frac\{1\}\{B\}\\\!\\left\(\\sum\_\{i\\in\\mathcal\{B\}\_\{t\}^\{\\mathrm\{sh\}\}\}\\mathrm\{clip\}\(g\_\{i,t\}^\{\\mathrm\{sh\}\},C\_\{s\}\)\+\\mathcal\{N\}\(0,\\sigma\_\{s\}^\{2\}C\_\{s\}^\{2\}I\)\\right\)\.\(7\)

#### Residual objective\.

Letzish∈ℝ\|𝒴\|z\_\{i\}^\{\\mathrm\{sh\}\}\\in\\mathbb\{R\}^\{\|\\mathcal\{Y\}\|\}be the output logits for recordiiunder the current shared parametersθsh\\theta\_\{\\mathrm\{sh\}\}\(expert parameters\{θl⋆,e\}\\\{\\theta\_\{l^\{\\star\},e\}\\\}held fixed, no gradient this pass\)\. Define the residual weight

wi=1−softmax⁡\(zish\)yi∈\[0,1\],w\_\{i\}=1\-\\operatorname\{softmax\}\(z\_\{i\}^\{\\mathrm\{sh\}\}\)\_\{y\_\{i\}\}\\;\\in\[0,1\],\(8\)the shared stream’s predicted error probability on the true class:wi≈0w\_\{i\}\\approx 0when confidently correct,wi≈1w\_\{i\}\\approx 1when not\. The expert lossℓexp=wi​ℓcls\\ell\_\{\\mathrm\{exp\}\}=w\_\{i\}\\,\\ell\_\{\\mathrm\{cls\}\}concentrates expert capacity on records the shared stream has not yet classified confidently\. Sincewiw\_\{i\}enters before clipping toCeC\_\{e\}, sensitivity is unaffected\.

#### Privatized expert update\.

For owner subsetℬt,l,e=\{i∈ℬtexp:al​\(xi\)=e\}\\mathcal\{B\}\_\{t,l,e\}=\\\{i\\in\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}:a\_\{l\}\(x\_\{i\}\)=e\\\}and public denominatorDe=B/ED\_\{e\}=B/E, the*expert stream*computes its own privatized estimate over the owner subset:

g¯t,l,eexp=1De​\(CLOSE\\displaystyle\\bar\{g\}\_\{t,l,e\}^\{\\mathrm\{exp\}\}=\\frac\{1\}\{D\_\{e\}\}\\Bigg\(∑i∈ℬt,l,eclip⁡\(gi,t,l,eexp,Ce\)\\displaystyle\\sum\_\{i\\in\\mathcal\{B\}\_\{t,l,e\}\}\\mathrm\{clip\}\\\!\\left\(g\_\{i,t,l,e\}^\{\\mathrm\{exp\}\},C\_\{e\}\\right\)\(9\)OPEN\+𝒩⁡\(0,σe2​Ce2​I\)\)\\displaystyle\+\\mathcal\{N\}\(0,\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}I\)\\Bigg\)Three properties follow: \(i\) expert clipping decouples from the shared stream, resolving the suppression of Sec\.[4\.1](https://arxiv.org/html/2609.05770#S4.SS1); \(ii\)DeD\_\{e\}depends only on the publicB,EB,E, so realized counts are never revealed; \(iii\) the normalized sensitivityCe/De=E​Ce/BC\_\{e\}/D\_\{e\}=EC\_\{e\}/Bfeeds directly into the accountant of Sec\.[6](https://arxiv.org/html/2609.05770#S6)\.

#### Empty owner subsets\.

Ifℬt,l,e=∅\\mathcal\{B\}\_\{t,l,e\}=\\emptyset, the update reduces to noise alone,g¯t,l,eexp=De−1​𝒩​\(0,σe2​Ce2​I\)\\bar\{g\}\_\{t,l,e\}^\{\\mathrm\{exp\}\}=D\_\{e\}^\{\-1\}\\mathcal\{N\}\(0,\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}I\): every expert updates at every step with no count\-dependent skipping or logging, which would leak realized counts and require additional accounting\.

### 5\.4Alternating Optimization and Layer Scope

Each outer cycle \- one iteration of theforttloop in Algorithm[1](https://arxiv.org/html/2609.05770#alg1)\(lines[5](https://arxiv.org/html/2609.05770#algx1.l5)\-[7](https://arxiv.org/html/2609.05770#algx1.l7)and the nested expert loop that follows\) \- draws two independent Poisson subsamples,ℬtsh\\mathcal\{B\}\_\{t\}^\{\\mathrm\{sh\}\}andℬtexp\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\- rather than reusing one sample for both streams, which costs no additional data access since the expert stream already needs its own forward pass to computewiw\_\{i\}\- then performs one shared\-stream update overℬtsh\\mathcal\{B\}\_\{t\}^\{\\mathrm\{sh\}\}followed by one expert\-stream update overℬtexp\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}for allEEexperts\. Alternating lets shared representations and expert residual corrections co\-adapt: each shared step updates thewiw\_\{i\}seen by the next expert step, and each expert step changes the residual for the next shared step\. Privacy accounting for the two streams remains independent of the downstream optimizer, since the independent subsamples make their per\-step releases independent random mechanisms \(Sec\.[6](https://arxiv.org/html/2609.05770#S6)\)\.

Algorithm 1![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR1:

𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\(

nnrecords\),

𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}, frozen router, candidate layers,

B,E,p=B/n,Cs,Ce,σs,σe,ρB,E,p\{=\}B/n,C\_\{s\},C\_\{e\},\\sigma\_\{s\},\\sigma\_\{e\},\\rho
2:

l⋆←arg⁡maxl⁡H⁡\(q\(l\)\)l^\{\\star\}\\leftarrow\\arg\\max\_\{l\}H\(q^\{\(l\)\}\)on

𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}⊳\\trianglerightno privacy cost

3:

al⋆​\(xi\)←arg⁡max⁡∑t∈Tok⁡\(xi\)e∈\[E\]⁡gl⋆,e​\(xi,t\)a\_\{l^\{\\star\}\}\(x\_\{i\}\)\\leftarrow\\arg\\max\_\{e\\in\[E\]\}\\sum\_\{t\\in\\mathrm\{Tok\}\(x\_\{i\}\)\}g\_\{l^\{\\star\},e\}\(x\_\{i\},t\),

∀xi∈𝒟priv\\forall x\_\{i\}\\in\\mathcal\{D\}\_\{\\mathrm\{priv\}\}⊳\\trianglerightonce; never released

4:for

t=1,…,Tt=1,\\dots,Tdo

5:

ℬtsh←\{i:uish=1\}\\mathcal\{B\}\_\{t\}^\{\\mathrm\{sh\}\}\\leftarrow\\\{i:u\_\{i\}^\{\\mathrm\{sh\}\}\{=\}1\\\},

uish​∼iid​Bern​\(p\)u\_\{i\}^\{\\mathrm\{sh\}\}\\overset\{\\text\{iid\}\}\{\\sim\}\\mathrm\{Bern\}\(p\)⊳\\trianglerightshared subsample

6:

g¯tsh←1B​\(∑i∈ℬtshclip⁡\(gi,tsh,Cs\)\+𝒩⁡\(0,σs2​Cs2​I\)\)\\bar\{g\}\_\{t\}^\{\\mathrm\{sh\}\}\\leftarrow\\tfrac\{1\}\{B\}\\big\(\\sum\_\{i\\in\\mathcal\{B\}\_\{t\}^\{\\mathrm\{sh\}\}\}\\mathrm\{clip\}\(g\_\{i,t\}^\{\\mathrm\{sh\}\},C\_\{s\}\)\+\\mathcal\{N\}\(0,\\sigma\_\{s\}^\{2\}C\_\{s\}^\{2\}I\)\\big\); DP\-Adam on

θsh\\theta\_\{\\mathrm\{sh\}\}
7:

ℬtexp←\{i:uiexp=1\}\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\\leftarrow\\\{i:u\_\{i\}^\{\\mathrm\{exp\}\}\{=\}1\\\},

uiexp​∼iid​Bern​\(p\)u\_\{i\}^\{\\mathrm\{exp\}\}\\overset\{\\text\{iid\}\}\{\\sim\}\\mathrm\{Bern\}\(p\),

\{uiexp\}⟂\{uish\}\\\{u\_\{i\}^\{\\mathrm\{exp\}\}\\\}\\perp\\\{u\_\{i\}^\{\\mathrm\{sh\}\}\\\}⊳\\trianglerightindependent subsample

8:for

e∈\[E\]e\\in\[E\]do

9:

ℬt,e←\{i∈ℬtexp:al⋆​\(xi\)=e\}\\mathcal\{B\}\_\{t,e\}\\leftarrow\\\{i\\in\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}:a\_\{l^\{\\star\}\}\(x\_\{i\}\)\{=\}e\\\}
10:

g¯t,eexp←1De​\(∑i∈ℬt,eclip⁡\(gi,t,eexp,Ce\)\+𝒩⁡\(0,σe2​Ce2​I\)\)\\bar\{g\}\_\{t,e\}^\{\\mathrm\{exp\}\}\\leftarrow\\tfrac\{1\}\{D\_\{e\}\}\\big\(\\sum\_\{i\\in\\mathcal\{B\}\_\{t,e\}\}\\mathrm\{clip\}\(g\_\{i,t,e\}^\{\\mathrm\{exp\}\},C\_\{e\}\)\+\\mathcal\{N\}\(0,\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}I\)\\big\); DP\-Adam on

θl⋆,e\\theta\_\{l^\{\\star\},e\}⊳\\trianglerightDe=B/ED\_\{e\}\{=\}B/E; noise\-only if empty

11:endfor

12:endfor

13:

θT\\theta\_\{T\}

## 6Privacy Analysis

We analyze our framework under\(ε,δ\)\(\\varepsilon,\\delta\)\-DP with add/remove adjacency; only the final model is released; owner assignments, expert counts, residual weights, and intermediate gradients are never exposed\.

### 6\.1Parallel Composition Across Experts

###### Lemma 1\(Parallel composition across owner groups\)\.

Fix layerll\. Suppose\(i\)ala\_\{l\}is deterministic and stage\-fixed;\(ii\)every record belongs to exactly one owner group𝒟l,e\\mathcal\{D\}\_\{l,e\}\(the partition cell of records assigned to experteebyala\_\{l\}, Sec\.[5\.2](https://arxiv.org/html/2609.05770#S5.SS2)\); and\(iii\)Ml,eM\_\{l,e\}depends only on𝒟l,e\\mathcal\{D\}\_\{l,e\}and public quantities\. If eachMl,eM\_\{l,e\}is\(εl,e,δl,e\)\(\\varepsilon\_\{l,e\},\\delta\_\{l,e\}\)\-DP on its owner group, the joint mechanismMl=\(Ml,1,…,Ml,E\)M\_\{l\}=\(M\_\{l,1\},\\dots,M\_\{l,E\}\)is\(maxe⁡εl,e,maxe⁡δl,e\)\(\\max\_\{e\}\\varepsilon\_\{l,e\},\\,\\max\_\{e\}\\delta\_\{l,e\}\)\-DP\.

All three conditions hold by construction:ala\_\{l\}is stage\-fixed \(Sec\.[5\.2](https://arxiv.org/html/2609.05770#S5.SS2)\); the public denominatorDe=B/ED\_\{e\}=B/Eand count\-independent schedule makeMl,eM\_\{l,e\}depend only on𝒟l,e\\mathcal\{D\}\_\{l,e\}and public quantities; and per\-group DP follows from subsampled\-Gaussian accounting with sensitivityΔl,e\\Delta\_\{l,e\}\(Eq\. \([20](https://arxiv.org/html/2609.05770#A2.E20)\)\)\. Full proofs are in Appendix[B\.3](https://arxiv.org/html/2609.05770#A2.SS3)\.

### 6\.2Composition Across Streams and Layers

Interleaved shared/expert updates compose adaptively under the PRV framework[Gopi et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib11): sinceℬtsh\\mathcal\{B\}\_\{t\}^\{\\mathrm\{sh\}\}andℬtexp\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}are independent Poisson subsamples drawn afresh each cycle, per\-step mechanisms are conditionally independent givenθt\\theta\_\{t\}, so standard PRV convolution applies without a joint privacy\-loss variable\. Writingω⊛T\\omega^\{\\circledast T\}forTT\-fold self\-convolution: the shared stream has per\-step PRVωsh\\omega\_\{\\mathrm\{sh\}\}set by\(Cs,σs,γ\)\(C\_\{s\},\\sigma\_\{s\},\\gamma\); each layer’s expert mechanisms, by Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1), compose in*parallel*\(disjoint owner groups, no realized counts released\), giving a single layer\-level PRVωl=ωl,e\\omega\_\{l\}=\\omega\_\{l,e\}\(homogeneousCe,σe,De=B/EC\_\{e\},\\sigma\_\{e\},D\_\{e\}\{=\}B/Eacross experts\) rather than anEE\-fold convolution\. Sequential composition across shared updates and trained layers then gives

ωtot=ωsh⊛Tsh⊛⊛l∈ℒtrainωl⊛Texp,l,\\omega\_\{\\mathrm\{tot\}\}=\\omega\_\{\\mathrm\{sh\}\}^\{\\circledast T\_\{\\mathrm\{sh\}\}\}\\circledast\\mathop\{\\circledast\}\\limits\_\{l\\in\\mathcal\{L\}\_\{\\mathrm\{train\}\}\}\\omega\_\{l\}^\{\\circledast T\_\{\\mathrm\{exp\},l\}\},\(10\)withℒtrain⊆ℒ\\mathcal\{L\}\_\{\\mathrm\{train\}\}\\subseteq\\mathcal\{L\}the privately\-updated layers andε\(δ\)=inf\{ε≥0:Pr\[ωtot\>ε\]≤δ\}\\varepsilon\(\\delta\)=\\inf\\\{\\varepsilon\\geq 0:\\Pr\[\\omega\_\{\\mathrm\{tot\}\}\>\\varepsilon\]\\leq\\delta\\\}\.

#### Budget split\.

A fractionρ∈\(0,1\)\\rho\\in\(0,1\)ofε\\varepsilongoes to the shared stream,1−ρ1\{\-\}\\rhoto the experts; we solve forσs,σe\\sigma\_\{s\},\\sigma\_\{e\}jointly via binary search over the PRV accountant, usingρ=0\.9\\rho=0\.9atE=8E=8\(Appendix[H](https://arxiv.org/html/2609.05770#A8)\); sensitivity toρ\\rhoand largerEEis in Appendix[E\.3](https://arxiv.org/html/2609.05770#A5.SS3)\.

## 7Utility Analysis of the Expert Estimator

### 7\.1Bias\-Variance Decomposition and Layer Selection

#### Setup\.

Fix one selected layer \(suppress layer index\)\. The owner map assigns each record to exactly one owner expert via gate\-weighted plurality over its tokens \(Eq\. \([5](https://arxiv.org/html/2609.05770#S5.E5)\)\), so it partitions𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}into disjoint groups𝒟e\\mathcal\{D\}\_\{e\}for anyk≥1k\\geq 1\- the forward pass may still route a record’s tokens tokkexperts, but only the owner expert receives its gradient\. Groups have sizesnen\_\{e\}, empirical frequenciesqe=ne/nq\_\{e\}=n\_\{e\}/n, and target meansμe=ne−1​∑i∈𝒟ehiexp\\mu\_\{e\}=n\_\{e\}^\{\-1\}\\sum\_\{i\\in\\mathcal\{D\}\_\{e\}\}h\_\{i\}^\{\\mathrm\{exp\}\}, wherehiexp=clip⁡\(giexp,Ce\)h\_\{i\}^\{\\mathrm\{exp\}\}=\\mathrm\{clip\}\(g\_\{i\}^\{\\mathrm\{exp\}\},C\_\{e\}\)\. Writeu=\(1/E,…,1/E\)u=\(1/E,\\dots,1/E\)for the uniform distribution,Σe\\Sigma\_\{e\}for the within\-group covariance; under Poisson subsampling at ratep=B/np=B/n,𝔼​\|ℬt,e\|=B​qe\\mathbb\{E\}\|\\mathcal\{B\}\_\{t,e\}\|=Bq\_\{e\}\.

###### Theorem 1\(Bias\-variance decomposition\)\.

Under Poisson subsampling with ratep=B/np=B/n, lettingdddenote the parameter dimension,𝔼​‖g¯t,eexp−μe‖22=βe2\+Vesamp\+VeDP\\mathbb\{E\}\\\|\\bar\{g\}\_\{t,e\}^\{\\mathrm\{exp\}\}\-\\mu\_\{e\}\\\|\_\{2\}^\{2\}=\\beta\_\{e\}^\{2\}\+V\_\{e\}^\{\\mathrm\{samp\}\}\+V\_\{e\}^\{\\mathrm\{DP\}\}, where

βe2\\displaystyle\\beta\_\{e\}^\{2\}=\(E​qe−1\)2​‖μe‖22,\\displaystyle=\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\},\(11\)Vesamp\\displaystyle V\_\{e\}^\{\\mathrm\{samp\}\}=E2​p​ne​\(1−p\)B2​\(‖μe‖22\+tr⁡Σe\),\\displaystyle=\\frac\{E^\{2\}p\\,n\_\{e\}\(1\-p\)\}\{B^\{2\}\}\\bigl\(\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+\\operatorname\{tr\}\\Sigma\_\{e\}\\bigr\),\(12\)VeDP\\displaystyle V\_\{e\}^\{\\mathrm\{DP\}\}=E2​σe2​Ce2​dB2\.\\displaystyle=\\frac\{E^\{2\}\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}\\,d\}\{B^\{2\}\}\.\(13\)

The squared biasβe2=\(E​qe−1\)2​‖μe‖22\\beta\_\{e\}^\{2\}=\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}vanishes exactly whenqe=1/Eq\_\{e\}=1/E\- the public denominatorDe=B/ED\_\{e\}=B/Eis unbiased under uniform routing, with bias growing quadratically under imbalance – and theE2E^\{2\}factor inVeDPV\_\{e\}^\{\\mathrm\{DP\}\}reflects the normalized sensitivityE​Ce/BEC\_\{e\}/B\. Proofs are in Appendix[B\.5](https://arxiv.org/html/2609.05770#A2.SS5)\.

###### Corollary 1\(Balanced routing\)\.

Ifqe=1/Eq\_\{e\}=1/E, thenβe=0\\beta\_\{e\}=0and

𝔼​‖g¯t,eexp−μe‖22≤Ce​EB\+E​σe​Ce​dB,\\sqrt\{\\mathbb\{E\}\\bigl\\\|\\bar\{g\}\_\{t,e\}^\{\\mathrm\{exp\}\}\-\\mu\_\{e\}\\bigr\\\|\_\{2\}^\{2\}\}\\;\\leq\\;C\_\{e\}\\\!\\sqrt\{\\frac\{E\}\{B\}\}\+\\frac\{E\\sigma\_\{e\}C\_\{e\}\\sqrt\{d\}\}\{B\},\(14\)separating non\-private sampling error from the DP cost; scaling and comparison to the standard DP mean\-estimation rate are in Appendix[B\.8](https://arxiv.org/html/2609.05770#A2.SS8)\.

#### Routing imbalance and chi\-squared divergence\.

Aggregatingβe2\\beta\_\{e\}^\{2\}across experts via∑e\(Eqe−1\)2=Eχ2\(q∥u\)\\sum\_\{e\}\(Eq\_\{e\}\-1\)^\{2\}=E\\chi^\{2\}\(q\\\|u\)\(Appendix[B\.6](https://arxiv.org/html/2609.05770#A2.SS6)\) gives:

###### Theorem 2\(Aggregate imbalance bias\)\.

LetG¯2=maxe⁡‖μe‖22\\bar\{G\}^\{2\}=\\max\_\{e\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\. Then

1E​∑e=1Eβe2\\displaystyle\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\\beta\_\{e\}^\{2\}≤G¯2χ2\(q∥u\),\\displaystyle\\leq\\bar\{G\}^\{2\}\\chi^\{2\}\(q\\\|u\),\(15\)∑e=1Eqe​βe2\\displaystyle\\sum\_\{e=1\}^\{E\}q\_\{e\}\\beta\_\{e\}^\{2\}≤G¯2Eχ2\(q∥u\)\.\\displaystyle\\leq\\bar\{G\}^\{2\}E\\chi^\{2\}\(q\\\|u\)\.\(16\)

#### Entropy as a layer\-selection proxy\.

Chi\-squared divergence is harder to estimate stably on small public samples than entropy; via Pinsker’s inequality andχ2\(q∥u\)=E∥q−u∥22≤E∥q−u∥12\\chi^\{2\}\(q\\\|u\)=E\\\|q\-u\\\|\_\{2\}^\{2\}\\leq E\\\|q\-u\\\|\_\{1\}^\{2\},

χ2\(q∥u\)≤2E\(logE−H\(q\)\),\\chi^\{2\}\(q\\,\\\|\\,u\)\\;\\leq\\;2E\\,\\bigl\(\\log E\-H\(q\)\\bigr\),\(17\)combining with Eq\. \([16](https://arxiv.org/html/2609.05770#S7.E16)\) gives

∑eqe​βe2≤2​G¯2​E2​\(log⁡E−H⁡\(q\)\)\.\\sum\_\{e\}q\_\{e\}\\,\\beta\_\{e\}^\{2\}\\;\\leq\\;2\\bar\{G\}^\{2\}E^\{2\}\\,\\bigl\(\\log E\-H\(q\)\\bigr\)\.\(18\)The right\-hand side of Eq\. \([18](https://arxiv.org/html/2609.05770#S7.E18)\) is the only lever we can pull without touching private data:G¯2\\bar\{G\}^\{2\}andEEare fixed by the model, so the bound on the aggregate bias∑eqe​βe2\\sum\_\{e\}q\_\{e\}\\beta\_\{e\}^\{2\}shrinks exactly aslog⁡E−H⁡\(q\)\\log E\-H\(q\)shrinks\. Minimizing the guaranteed bias therefore reduces to maximizingH⁡\(q\)H\(q\)over candidate layers, which is computable from a public corpus alone: this is the entire justification for the criterionl⋆=arg⁡maxl⁡H⁡\(q\(l\)\)l^\{\\star\}=\\arg\\max\_\{l\}H\(q^\{\(l\)\}\)\. A layer with higherH⁡\(q\)H\(q\)thus has a smaller imbalance\-bias guarantee\. This gives a privacy\-free, computable criterion:l⋆=arg⁡maxl⁡H⁡\(q\(l\)\)l^\{\\star\}=\\arg\\max\_\{l\}H\(q^\{\(l\)\}\)needs only𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}and the frozen router, with its connection to DP\-Adam utility scoped in Remark[2](https://arxiv.org/html/2609.05770#Thmremark2)\.

### 7\.2Stability of Expert Ownership

The owner mapal⋆a\_\{l^\{\\star\}\}is frozen before training and never recomputed asθsh\\theta\_\{\\mathrm\{sh\}\}updates; this does not affect privacy \(Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1)needs only stage\-fixedness\), but bears on whether it stays a faithful specialization proxy as the representation drifts\. Under a Voronoi abstraction, we bound this drift \(proofs in Appendix[B\.9](https://arxiv.org/html/2609.05770#A2.SS9)–[B\.10](https://arxiv.org/html/2609.05770#A2.SS10)\):

###### Theorem 3\(Ownership stability, informal\)\.

If the representation moves by at mostrr, only records with margin≤2​Δ𝒞​r\\leq 2\\Delta\_\{\\mathcal\{C\}\}rcan change owner \(Δ𝒞\\Delta\_\{\\mathcal\{C\}\}: max prototype distance\); under a mild boundary\-mass condition \(rateκ\\kappa\), both reassignment probability and routing\-mass drift scale*linearly*inrr\- only near\-boundary records can flip, and how many there are grows gracefully with drift, not catastrophically\.

###### Theorem 4\(Estimation error under drift, informal\)\.

Assuming anLgL\_\{g\}\-Lipschitz gradient field,

𝔼​‖g¯e−νe‖22\\displaystyle\\mathbb\{E\}\\\|\\bar\{g\}\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}≤Vesamp\+VeDP\\displaystyle\\leq V\_\{e\}^\{\\mathrm\{samp\}\}\+V\_\{e\}^\{\\mathrm\{DP\}\}\+2​\(E​qe−1\)2​‖μe‖22\+2​Lg2​re2,\\displaystyle\+2\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+2L\_\{g\}^\{2\}r\_\{e\}^\{2\},re2r\_\{e\}^\{2\}the mean squared distance from a record to its owning prototype\. The first three terms match Theorem[1](https://arxiv.org/html/2609.05770#Thmtheorem1); the last is the extra cost of ownership being a coarse proxy rather than exact, vanishing as owner groups tighten around their prototype\.

## 8Experiments

### 8\.1Setup

#### Model and tasks\.

We fine\-tunegoogle/switch\-base\-8[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib7)\(E=8E=8, six encoder MoE layers at blocks\{1,3,5,7,9,11\}\\\{1,3,5,7,9,11\\\}\) on four GLUE tasks[Wang et al\. \(2018\)](https://arxiv.org/html/2609.05770#bib.bib9)\(SST\-2, MNLI, QNLI, QQP\), reporting dev\-set accuracy\. We additionally evaluateOLMoE\-1B\-7B[Muennighoff et al\. \(2025\)](https://arxiv.org/html/2609.05770#bib.bib14)\(E=64E=64, top\-88routing\) to exercise the primary\-expert ownership rule fork\>1k\>1\(Sec\.[5\.2](https://arxiv.org/html/2609.05770#S5.SS2)\)\. We additionally testDeepSeek\-VL2\-Tiny[Wu et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib41), a vision\-language MoE with both shared and routed experts \(Sec\.[3\.3](https://arxiv.org/html/2609.05770#S3.SS3)\), fine\-tuned on ScienceQA[Lu et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib42), to test generalization beyond text\-only models \(Appendix[D](https://arxiv.org/html/2609.05770#A4)\)\. Hyperparameter grids are in Appendix[C](https://arxiv.org/html/2609.05770#A3); runtime/memory in Appendix[I](https://arxiv.org/html/2609.05770#A9); ablations in Appendix[E](https://arxiv.org/html/2609.05770#A5)\- component ablation \(clipping, denominator, alternating schedule, residual objective\), matched\-scope diagnostics \(shared\-/expert\-only, global DP at Ours’s scope\), and budget\-splitρ\\rhosensitivity\.

#### Privacy\.

All private methods useε∈\{1,4,8\}\\varepsilon\\in\\\{1,4,8\\\}under\(ε,δ\)\(\\varepsilon,\\delta\)\-DP withδ=1/\|𝒟priv\|1\.1\\delta=1/\|\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\|^\{1\.1\}, matched across methods\.

#### Layer selection\.

l⋆=arg⁡maxl⁡H⁡\(q\(l\)\)l^\{\\star\}=\\arg\\max\_\{l\}H\(q^\{\(l\)\}\), computed from the frozen router and external public data only \(no privacy cost\), yields block 1 \(SST\-2\), block 7 \(MNLI\), block 11 \(QNLI\), block 9 \(QQP\) \- see Appendix[F](https://arxiv.org/html/2609.05770#A6)for the full sweep\. We also report first\- and last\-sparse\-layer variants to make the entropy prediction falsifiable\.

#### Baselines\.

Non\-private upper bounds:LoRA, full fine\-tuning without noise\.Monolithic DP:DP\-Adam LoRA[Yu et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib4)and DP\-Adam[Tholoniat et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib8), both treating all parameters as one dense block\.

### 8\.2Results on Switch Transformer

Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)reports results atε∈\{1,4,8\}\\varepsilon\\in\\\{1,4,8\\\}\.

#### ![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORvs\. monolithic DP\.

Ours attains the highest accuracy on all twelve task\-budget cells\. Gains over Monolithic DP\-Adam LoRA range\+0\.23\+0\.23to\+3\.24\+3\.24pts \(SST\-2/MNLI/QNLI/QQP\) atε=8\\varepsilon=8and\+1\.51\+1\.51to\+2\.92\+2\.92atε=1\\varepsilon=1, growing at tighter budgets on SST\-2/QNLI but not MNLI/QQP \- the high\-noise advantage of Corollary[1](https://arxiv.org/html/2609.05770#Thmcorollary1)is baseline\- and task\-dependent, not uniform\.

#### Entropy criterion\.

Selection is non\-trivially falsifiable on MNLI \(block 7\) and QQP \(block 9\), neither first nor last; entropy\-selected matches or exceeds both variants in every cell, with the largest gains on MNLI and QQP, where chosen layers differ most from the runner\-up\.

ε\\varepsilonMethodSST\-2MNLIQNLIQQP∞\\inftyNon\-priv\. LoRA94\.6986\.5591\.2090\.96Non\-priv\. FFN94\.5086\.3691\.0991\.2811Monolith\. DP\-Adam LoRA90\.9476\.9180\.3881\.19Monolith\. DP\-Adam87\.0472\.7281\.4481\.84Ours, first sparse92\.9077\.4082\.6083\.20Ours, last sparse92\.6677\.8783\.3083\.80Ours, entropy\-sel\.92\.9078\.4283\.3084\.0844Monolith\. DP\-Adam LoRA92\.6678\.3381\.9582\.26Monolith\. DP\-Adam90\.4875\.9082\.4183\.32Ours, first sparse94\.0578\.1383\.6084\.50Ours, last sparse93\.4678\.5084\.7084\.78Ours, entropy\-sel\.94\.0579\.0384\.7084\.7888Monolith\. DP\-Adam LoRA93\.9279\.3682\.4882\.68Monolith\. DP\-Adam90\.3776\.7082\.6184\.09Ours, first sparse94\.1579\.4383\.6085\.39Ours, last sparse93\.5080\.5885\.2685\.73Ours, entropy\-sel\.94\.1581\.3285\.2685\.92Table 1:Switch\-base\-8under\(ε,δ\)\(\\varepsilon,\\delta\)\-DP withδ=1/\|𝒟priv\|1\.1\\delta=1/\|\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\|^\{1\.1\}\. Entropy\-selected layers fixed from public routing statistics before training: block 1 \(SST\-2\), block 7 \(MNLI\), block 11 \(QNLI\), block 9 \(QQP\)\.Bold: best per task–budget;underline: second\-best distinct method\.

### 8\.3Results on OLMoE

Tab\.[2](https://arxiv.org/html/2609.05770#S8.T2)reportsOLMoE\-1B\-7Bresults atε∈\{1,4,8\}\\varepsilon\\in\\\{1,4,8\\\}, with records assigned via majority\-vote ownership fork\>1k\>1\(Sec\.[5\.2](https://arxiv.org/html/2609.05770#S5.SS2)\)\. Ours \(entropy\-selected\) attains the highest accuracy in all twelve task–budget cells\. Atε=8\\varepsilon=8, gains over Monolithic DP\-Adam LoRA are\+0\.69\+0\.69\(SST\-2\),\+2\.18\+2\.18\(MNLI\),\+1\.74\+1\.74\(QNLI\),\+0\.32\+0\.32\(QQP\); atε=1\\varepsilon=1,\+2\.63\+2\.63,\+0\.84\+0\.84,\+4\.50\+4\.50,\+0\.70\+0\.70respectively\. SST\-2, QNLI, and QQP gains grow at tighter budgets, consistent with the high\-noise regime prediction of Corollary[1](https://arxiv.org/html/2609.05770#Thmcorollary1)and mirroring Sec\.[8\.2](https://arxiv.org/html/2609.05770#S8.SS2); the MNLI gain instead shrinks from\+2\.18\+2\.18to\+0\.84\+0\.84\.

ε\\varepsilonMethodSST\-2MNLIQNLIQQP∞\\inftyNon\-priv\. LoRA96\.6790\.1994\.6592\.03Non\-priv\. FFN95\.1889\.7293\.1391\.0711Monolith\. DP\-Adam LoRA92\.7885\.3184\.2485\.89Ours, entropy\-sel\.95\.4186\.1588\.7486\.5944Monolith\. DP\-Adam LoRA95\.0785\.7488\.5686\.43Ours, entropy\-sel\.95\.3087\.4490\.4486\.8988Monolith\. DP\-Adam LoRA95\.0786\.0889\.0086\.62Ours, entropy\-sel\.95\.7688\.2690\.7486\.94Table 2:OLMoE\-1B\-7B \(E=64E=64, top\-88routing\) under\(ε,δ\)\(\\varepsilon,\\delta\)\-DP,δ=1/\|𝒟priv\|1\.1\\delta=1/\|\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\|^\{1\.1\}\. Entropy\-selected layers fixed from public routing statistics: block 1 \(SST\-2, QNLI, QQP\), block 15 \(MNLI\)\.
### 8\.4Empirical Privacy Auditing via Membership Inference

Methodε\\varepsilonAcc\.Loss attackLiRAAUCTPR@0\.1%FPRAUCTPR@0\.1%FPRNon\-priv\. LoRA∞\\infty91\.200\.5680\.45%0\.6203\.00%Monolithic DP\-LoRA180\.380\.5010\.10%0\.5040\.13%Matched\-scope Global DP182\.900\.5020\.11%0\.5050\.14%![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR183\.300\.5020\.10%0\.5060\.15%Monolithic DP\-LoRA481\.950\.5050\.12%0\.5130\.23%Matched\-scope Global DP484\.680\.5050\.13%0\.5140\.26%![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR484\.700\.5060\.13%0\.5160\.28%Monolithic DP\-LoRA882\.480\.5100\.16%0\.5240\.45%Matched\-scope Global DP884\.900\.5100\.16%0\.5270\.50%![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR885\.260\.5120\.18%0\.5300\.58%Table 3:MIA audit on QNLI \(Switch\-base\-8\)\.6464shadow models;neval=20,000n\_\{\\mathrm\{eval\}\}=20\{,\}000\(1010k/1010k members/non\-members\)\. Single run per cell\.Bold: highest attack success \(AUC/TPR\) among DP methods at thatε\\varepsilon\- the*least*favorable privacy result, not the best method; underline: second\-highest/tied\. Acc\. bolding follows the opposite convention \(highest accuracy\), as in Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)\.We audit QNLI \(Switch\-base\-8\) with a loss\-threshold attack and shadow\-model LiRA[Carlini et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib43)\(6464shadow models;neval=20,000n\_\{\\mathrm\{eval\}\}=20\{,\}000:1010k members,1010k non\-members\), reportingTPR​@​0\.1%​FPR\\mathrm\{TPR\}@0\.1\\%\\mathrm\{FPR\}alongside AUC since average\-case AUC can mask worst\-case success \(Tab\.[3](https://arxiv.org/html/2609.05770#S8.T3)\)\. All DP methods sharply reduce LiRA TPR vs\. non\-private training, with attack success growing monotonically inε\\varepsilon\. Our framework attains the highest TPR/AUC among DP methods on both LiRA metrics and loss\-attack AUC at everyε\\varepsilon\(tying Monolithic DP\-LoRA on loss\-attack TPR atε=1\\varepsilon\{=\}1\), with the gap widening asε\\varepsilongrows \- not a DP violation, but a signature of the same higher per\-expert SNR \(Theorem[1](https://arxiv.org/html/2609.05770#Thmtheorem1)\) driving its accuracy gains\.

## 9Summary

Monolithic DP mismatches sparse MoE’s shared/expert structure\.![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORfixes this via alternating streams, expert\-specific clipping/noise, public normalization, and entropy\-based layer selection, under a parallel\-composition DP guarantee\. Gains hold across Switch, OLMoE, and DeepSeek\-VL2\-Tiny, typically largest at tight budgets\.

## Limitations

Frozen router\.Fixing owner groups requires freezing the router, which precludes privately updating routing parameters within a stage\. Adaptive router training could improve utility but would require re\-accounting for ownership changes\.

Public denominator\.De=B/ED\_\{e\}=B/Eis privacy\-safe but not oracle\-optimal: under heavy imbalance it may over\- or under\-scale expert updates\. Adaptive private normalization is left to future work\.

Residual weighting and outlier amplification\.The residual weightwi=1−softmax​\(zish\)yiw\_\{i\}=1\-\\mathrm\{softmax\}\(z\_\{i\}^\{\\mathrm\{sh\}\}\)\_\{y\_\{i\}\}\(Eq\. \([8](https://arxiv.org/html/2609.05770#S5.E8)\)\) up\-weights low\-confidence records – a superset of genuinely hard subpopulations and atypical or mislabeled ones, which tend to retain highwiw\_\{i\}once the shared stream reliably disagrees with their label\. Clipping is applied after thewiw\_\{i\}multiplication \(Lemma[2](https://arxiv.org/html/2609.05770#Thmlemma2)\), so the sensitivity boundCeC\_\{e\}and the\(ε,δ\)\(\\varepsilon,\\delta\)\-DP guarantee hold regardless ofwiw\_\{i\}\- a signal\-composition effect, not a sensitivity violation\. But a small owner group’s clipped sum is then more likely dominated by its highest\-wiw\_\{i\}member, which may fit label noise over genuine structure, compounds with the SNR degradation of Sec\.[4](https://arxiv.org/html/2609.05770#S4)on low\-load experts, and concentrates signal on records associated with elevated memorization risk even under a fixed nominalε\\varepsilon\. We did not run label\-noise or subgroup diagnostics to quantify this; a bounded variant \(wiγw\_\{i\}^\{\\gamma\},γ<1\\gamma<1, or an EMA\-discountedwiw\_\{i\}\) costs no extra privacy budget and is a natural robustness knob for future work\.

Extensions\.Future work includes private router adaptation, adaptive normalization, and extension to generative and larger\-scale multimodal MoEs\.

## References

- Abadiet al\.\(2016\)M\. Abadi, A\. Chu, I\. J\. Goodfellow, H\. B\. McMahan, I\. Mironov, K\. Talwar, and L\. ZhangDeep learning with differential privacy\.InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24\-28, 2016,E\. R\. Weippl, S\. Katzenbeisser, C\. Kruegel, A\. C\. Myers, and S\. Halevi \(Eds\.\),pp\. 308–318\.External Links:[Link](https://doi.org/10.1145/2976749.2978318),[Document](https://dx.doi.org/10.1145/2976749.2978318)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1),[§1](https://arxiv.org/html/2609.05770#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.05770#S3.SS2.p1.1)\.
- Carliniet al\.\(2022\)N\. Carlini, S\. Chien, M\. Nasr, S\. Song, A\. Terzis, and F\. TramerMembership inference attacks from first principles\.In2022 IEEE symposium on security and privacy \(SP\),pp\. 1897–1914\.Cited by:[§8\.4](https://arxiv.org/html/2609.05770#S8.SS4.p1.1)\.
- Chiet al\.\(2022\)Z\. Chi, L\. Dong, S\. Huang, D\. Dai, S\. Ma, B\. Patra, S\. Singhal, P\. Bajaj, X\. Song, X\. Mao, H\. Huang, and F\. WeiOn the representation collapse of sparse mixture of experts\.External Links:2204\.09179,[Link](https://arxiv.org/abs/2204.09179)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1),[§3\.1](https://arxiv.org/html/2609.05770#S3.SS1.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Zhang, H\. Ding, H\. Xin, H\. Gao, H\. Li, H\. Qu, J\. L\. Cai, J\. Liang, J\. Guo, J\. Ni, J\. Li, J\. Wang, J\. Chen, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. Song, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, L\. Zhao, L\. Wang, L\. Zhang, M\. Li, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, N\. Tian, P\. Huang, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Chen, Q\. Du, R\. J\. Chen, R\. L\. Jin, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Xu, R\. Zhang, R\. Chen, S\. S\. Li, S\. Lu, S\. Zhou, S\. Chen, S\. Wu, S\. Ye, S\. Ye, S\. Ma, S\. Wang, S\. Zhou, S\. Yu, S\. Zhou, S\. Pan, T\. Wang, T\. Yun, T\. Pei, T\. Sun, W\. L\. Xiao, W\. Zeng, W\. Zhao, W\. An, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, X\. Q\. Li, X\. Jin, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Shen, X\. Chen, X\. Zhang, X\. Chen, X\. Nie, X\. Sun, X\. Wang, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yu, X\. Song, X\. Shan, X\. Zhou, X\. Yang, X\. Li, X\. Su, X\. Lin, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. X\. Zhu, Y\. Zhang, Y\. Xu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Li, Y\. Wang, Y\. Yu, Y\. Zheng, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Tang, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Wu, Y\. Ou, Y\. Zhu, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Zha, Y\. Xiong, Y\. Ma, Y\. Yan, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Z\. F\. Wu, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Huang, Z\. Zhang, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Xu, Z\. Wu, Z\. Zhang, Z\. Li, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Gao, and Z\. PanDeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1)\.
- Duet al\.\(2022\)N\. Du, Y\. Huang, A\. M\. Dai, S\. Tong, D\. Lepikhin, Y\. Xu, M\. Krikun, Y\. Zhou, A\. W\. Yu, O\. Firat, B\. Zoph, L\. Fedus, M\. Bosma, Z\. Zhou, T\. Wang, Y\. E\. Wang, K\. Webster, M\. Pellat, K\. Robinson, K\. Meier\-Hellstern, T\. Duke, L\. Dixon, K\. Zhang, Q\. V\. Le, Y\. Wu, Z\. Chen, and C\. CuiGLaM: efficient scaling of language models with mixture\-of\-experts\.External Links:2112\.06905,[Link](https://arxiv.org/abs/2112.06905)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1)\.
- Dworket al\.\(2006\)C\. Dwork, K\. Kenthapadi, F\. McSherry, I\. Mironov, and M\. NaorOur data, ourselves: privacy via distributed noise generation\.InAnnual international conference on the theory and applications of cryptographic techniques,pp\. 486–503\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p2.1)\.
- Dwork and Roth \(2014\)C\. Dwork and A\. RothThe algorithmic foundations of differential privacy\.Foundations and trends® in theoretical computer science9\(3\-4\),pp\. 211–487\.Cited by:[§B\.4](https://arxiv.org/html/2609.05770#A2.SS4.p3.1.1),[§1](https://arxiv.org/html/2609.05770#S1.p1.1),[Definition 1](https://arxiv.org/html/2609.05770#Thmdefinition1.p1.1)\.
- Dworket al\.\(2010\)C\. Dwork, G\. N\. Rothblum, and S\. VadhanBoosting and differential privacy\.In2010 IEEE 51st annual symposium on foundations of computer science,pp\. 51–60\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1),[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1),[§1](https://arxiv.org/html/2609.05770#S1.p1.1),[§3\.3](https://arxiv.org/html/2609.05770#S3.SS3.p1.1),[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px1.p1.1)\.
- Gopiet al\.\(2021\)S\. Gopi, Y\. T\. Lee, and L\. WutschitzNumerical composition of differential privacy\.Advances in Neural Information Processing Systems34,pp\. 11631–11642\.Cited by:[Appendix C](https://arxiv.org/html/2609.05770#A3.SS0.SSS0.Px1.p1.1),[Appendix C](https://arxiv.org/html/2609.05770#A3.SS0.SSS0.Px2.p1.1),[§6\.2](https://arxiv.org/html/2609.05770#S6.SS2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§2](https://arxiv.org/html/2609.05770#S2.p1.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. de Las Casas, E\. B\. Hanna, F\. Bressand, G\. Lengyel, G\. Bour, G\. Lample, L\. R\. Lavaud, L\. Saulnier, M\. Lachaux, P\. Stock, S\. Subramanian, S\. Yang, S\. Antoniak, T\. L\. Scao, T\. Gervet, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMixtral of experts\.CoRRabs/2401\.04088\.External Links:[Link](https://doi.org/10.48550/arXiv.2401.04088),[Document](https://dx.doi.org/10.48550/ARXIV.2401.04088),2401\.04088Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1),[§1](https://arxiv.org/html/2609.05770#S1.p1.1)\.
- Kasiviswanathanet al\.\(2011\)S\. P\. Kasiviswanathan, H\. K\. Lee, K\. Nissim, S\. Raskhodnikova, and A\. SmithWhat can we learn privately?\.SIAM Journal on Computing40\(3\),pp\. 793–826\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p1.1)\.
- Lepikhinet al\.\(2020\)D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. ChenGShard: scaling giant models with conditional computation and automatic sharding\.External Links:2006\.16668,[Link](https://arxiv.org/abs/2006.16668)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1)\.
- Lewiset al\.\(2021\)M\. Lewis, S\. Bhosale, T\. Dettmers, N\. Goyal, and L\. ZettlemoyerBASE layers: simplifying training of large, sparse models\.External Links:2103\.16716,[Link](https://arxiv.org/abs/2103.16716)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1)\.
- Liet al\.\(2021\)X\. Li, F\. Tramer, P\. Liang, and T\. HashimotoLarge language models can be strong differentially private learners\.arXiv preprint arXiv:2110\.05679\.Cited by:[§2](https://arxiv.org/html/2609.05770#S2.p1.1)\.
- Luet al\.\(2022\)P\. Lu, S\. Mishra, T\. Xia, L\. Qiu, K\. Chang, S\. Zhu, O\. Tafjord, P\. Clark, and A\. KalyanLearn to explain: multimodal reasoning via thought chains for science question answering\.Advances in neural information processing systems35,pp\. 2507–2521\.Cited by:[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px1.p1.1)\.
- Maaset al\.\(2011\)A\. Maas, R\. E\. Daly, P\. T\. Pham, D\. Huang, A\. Y\. Ng, and C\. PottsLearning word vectors for sentiment analysis\.InProceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies,pp\. 142–150\.Cited by:[Appendix F](https://arxiv.org/html/2609.05770#A6.SS0.SSS0.Px1.p1.1)\.
- McMahanet al\.\(2017\)B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y ArcasCommunication\-efficient learning of deep networks from decentralized data\.InArtificial intelligence and statistics,pp\. 1273–1282\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p1.1)\.
- McSherry and Talwar \(2007\)F\. McSherry and K\. TalwarMechanism design via differential privacy\.In48th Annual IEEE Symposium on Foundations of Computer Science \(FOCS’07\),pp\. 94–103\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p1.1)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, E\. P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, and et al\.OLMoE: open mixture\-of\-experts language models\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=xXTkbTBmqq)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1),[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px1.p1.1)\.
- Nieet al\.\(2020\)Y\. Nie, A\. Williams, E\. Dinan, M\. Bansal, J\. Weston, and D\. KielaAdversarial nli: a new benchmark for natural language understanding\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 4885–4901\.Cited by:[Appendix F](https://arxiv.org/html/2609.05770#A6.SS0.SSS0.Px1.p1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga, A\. Desmaison, A\. Köpf, E\. Z\. Yang, Z\. DeVito, M\. Raison, A\. Tejani, S\. Chilamkurthy, B\. Steiner, L\. Fang, J\. Bai, and S\. ChintalaPyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8\-14, 2019, Vancouver, BC, Canada,H\. M\. Wallach, H\. Larochelle, A\. Beygelzimer, F\. d’Alché\-Buc, E\. B\. Fox, and R\. Garnett \(Eds\.\),pp\. 8024–8035\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html)Cited by:[Appendix C](https://arxiv.org/html/2609.05770#A3.SS0.SSS0.Px1.p1.1)\.
- Rajbhandariet al\.\(2022\)S\. Rajbhandari, C\. Li, Z\. Yao, M\. Zhang, R\. Y\. Aminabadi, A\. A\. Awan, J\. Rasley, and Y\. HeDeepSpeed\-moe: advancing mixture\-of\-experts inference and training to power next\-generation ai scale\.External Links:2201\.05596,[Link](https://arxiv.org/abs/2201.05596)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSquad: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 conference on empirical methods in natural language processing,pp\. 2383–2392\.Cited by:[Appendix F](https://arxiv.org/html/2609.05770#A6.SS0.SSS0.Px1.p1.1)\.
- Riquelmeet al\.\(2021\)C\. Riquelme, J\. Puigcerver, B\. Mustafa, M\. Neumann, R\. Jenatton, A\. S\. Pinto, D\. Keysers, and N\. HoulsbyScaling vision with sparse mixture of experts\.External Links:2106\.05974,[Link](https://arxiv.org/abs/2106.05974)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1)\.
- Rogerset al\.\(2016\)R\. Rogers, A\. Roth, A\. Smith, and O\. ThakkarMax\-information, differential privacy, and post\-selection hypothesis testing\.In2016 IEEE 57th Annual Symposium on Foundations of Computer Science \(FOCS\),pp\. 487–494\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p2.1)\.
- Rolleret al\.\(2021\)S\. Roller, S\. Sukhbaatar, A\. Szlam, and J\. WestonHash layers for large sparse models\.External Links:2106\.04426,[Link](https://arxiv.org/abs/2106.04426)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1)\.
- Saikhet al\.\(2022\)T\. Saikh, T\. Ghosal, A\. Mittal, A\. Ekbal, and P\. BhattacharyyaScienceqa: a novel resource for question answering on scholarly articles\.International Journal on Digital Libraries23\(3\),pp\. 289–301\.Cited by:[Appendix D](https://arxiv.org/html/2609.05770#A4.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1),[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1),[§1](https://arxiv.org/html/2609.05770#S1.p1.1),[§3\.3](https://arxiv.org/html/2609.05770#S3.SS3.p1.1)\.
- Smith \(2011\)A\. SmithPrivacy\-preserving statistical estimation with optimal convergence rates\.InProceedings of the forty\-third annual ACM symposium on Theory of computing,pp\. 813–822\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p2.1)\.
- Thakkaret al\.\(2019\)O\. Thakkar, G\. Andrew, and H\. B\. McMahanDifferentially private learning with adaptive clipping\.arXiv e\-prints,pp\. arXiv–1905\.Cited by:[§A\.1](https://arxiv.org/html/2609.05770#A1.SS1.p1.1)\.
- Tholoniatet al\.\(2024\)P\. Tholoniat, H\. A\. Inan, J\. Kulkarni, and R\. SimDifferentially private training of mixture of experts models\.arXiv preprint arXiv:2402\.07334\.Cited by:[§1](https://arxiv.org/html/2609.05770#S1.p1.1),[§2](https://arxiv.org/html/2609.05770#S2.p1.1),[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px4.p1.1)\.
- Vadhan \(2017\)S\. VadhanThe complexity of differential privacy\.InTutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich,pp\. 347–450\.Cited by:[§A\.2](https://arxiv.org/html/2609.05770#A1.SS2.p1.1)\.
- Wanget al\.\(2018\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)L\. Wang, H\. Gao, C\. Zhao, X\. Sun, and D\. DaiAuxiliary\-loss\-free load balancing strategy for mixture\-of\-experts\.External Links:2408\.15664,[Link](https://arxiv.org/abs/2408.15664)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p2.1)\.
- Wuet al\.\(2024\)Z\. Wu, X\. Chen, Z\. Pan, X\. Liu, W\. Liu, D\. Dai, H\. Gao, Y\. Ma, C\. Wu, B\. Wang,et al\.Deepseek\-vl2: mixture\-of\-experts vision\-language models for advanced multimodal understanding\.arXiv preprint arXiv:2412\.10302\.Cited by:[Appendix D](https://arxiv.org/html/2609.05770#A4.p1.1),[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px1.p1.1)\.
- Yousefpouret al\.\(2021\)A\. Yousefpour, I\. Shilov, A\. Sablayrolles, D\. Testuggine, K\. Prasad, M\. Malek, J\. Nguyen, S\. Ghosh, A\. Bharadwaj, J\. Zhao, G\. Cormode, and I\. MironovOpacus: user\-friendly differential privacy library in pytorch\.CoRRabs/2109\.12298\.External Links:[Link](https://arxiv.org/abs/2109.12298),2109\.12298Cited by:[Appendix C](https://arxiv.org/html/2609.05770#A3.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2021\)D\. Yu, S\. Naik, A\. Backurs, S\. Gopi, H\. A\. Inan, G\. Kamath, J\. Kulkarni, Y\. T\. Lee, A\. Manoel, L\. Wutschitz,et al\.Differentially private fine\-tuning of language models\.arXiv preprint arXiv:2110\.06500\.Cited by:[§A\.1](https://arxiv.org/html/2609.05770#A1.SS1.p1.1),[§2](https://arxiv.org/html/2609.05770#S2.p1.1),[§8\.1](https://arxiv.org/html/2609.05770#S8.SS1.SSS0.Px4.p1.1)\.
- Zhanget al\.\(2025\)X\. Zhang, Z\. Bu, B\. Balle, M\. Hong, M\. Razaviyayn, and V\. MirrokniDisk: differentially private optimizer with simplified kalman filter for noise reduction\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 80289–80316\.Cited by:[§A\.1](https://arxiv.org/html/2609.05770#A1.SS1.p1.1)\.
- Zhenget al\.\(2026\)L\. Zheng, X\. Wang, T\. Zhang, Y\. Cao, K\. Cheng, and Y\. ShenDifferentially private subspace fine\-tuning for large language models\.InFortieth AAAI Conference on Artificial Intelligence, Thirty\-Eighth Conference on Innovative Applications of Artificial Intelligence, Sixteenth Symposium on Educational Advances in Artificial Intelligence, AAAI 2026, Singapore, January 20\-27, 2026,S\. Koenig, C\. Jenkins, and M\. E\. Taylor \(Eds\.\),pp\. 28830–28838\.External Links:[Link](https://doi.org/10.1609/aaai.v40i34.40117),[Document](https://dx.doi.org/10.1609/AAAI.V40I34.40117)Cited by:[§A\.1](https://arxiv.org/html/2609.05770#A1.SS1.p1.1)\.
- Zhouet al\.\(2022\)Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Zhao, A\. Dai, Z\. Chen, Q\. Le, and J\. LaudonMixture\-of\-experts with expert choice routing\.External Links:2202\.09368,[Link](https://arxiv.org/abs/2202.09368)Cited by:[§A\.3](https://arxiv.org/html/2609.05770#A1.SS3.p1.1)\.

![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORSupplementary

###### Contents

1. [1Introduction](https://arxiv.org/html/2609.05770#S1)
2. [2Related Work](https://arxiv.org/html/2609.05770#S2)
3. [3Preliminaries](https://arxiv.org/html/2609.05770#S3)1. [3\.1Setup and Notation](https://arxiv.org/html/2609.05770#S3.SS1) 2. [3\.2Differential Privacy](https://arxiv.org/html/2609.05770#S3.SS2) 3. [3\.3Sparse MoE Models and Load Balancing](https://arxiv.org/html/2609.05770#S3.SS3)
4. [4Diagnosing Monolithic Private Optimization Failures](https://arxiv.org/html/2609.05770#S4)1. [4\.1Clipping Interference](https://arxiv.org/html/2609.05770#S4.SS1) 2. [4\.2Update\-Scale Dilution](https://arxiv.org/html/2609.05770#S4.SS2) 3. [4\.3Load\-Dependent Signal Degradation](https://arxiv.org/html/2609.05770#S4.SS3)
5. [5Role\-Aware Private Training](https://arxiv.org/html/2609.05770#S5)1. [5\.1Public Layer Selection](https://arxiv.org/html/2609.05770#S5.SS1) 2. [5\.2Record\-Level Expert Ownership](https://arxiv.org/html/2609.05770#S5.SS2) 3. [5\.3Shared and Expert Stream Updates](https://arxiv.org/html/2609.05770#S5.SS3) 4. [5\.4Alternating Optimization and Layer Scope](https://arxiv.org/html/2609.05770#S5.SS4)
6. [6Privacy Analysis](https://arxiv.org/html/2609.05770#S6)1. [6\.1Parallel Composition Across Experts](https://arxiv.org/html/2609.05770#S6.SS1) 2. [6\.2Composition Across Streams and Layers](https://arxiv.org/html/2609.05770#S6.SS2)
7. [7Utility Analysis of the Expert Estimator](https://arxiv.org/html/2609.05770#S7)1. [7\.1Bias\-Variance Decomposition and Layer Selection](https://arxiv.org/html/2609.05770#S7.SS1) 2. [7\.2Stability of Expert Ownership](https://arxiv.org/html/2609.05770#S7.SS2)
8. [8Experiments](https://arxiv.org/html/2609.05770#S8)1. [8\.1Setup](https://arxiv.org/html/2609.05770#S8.SS1) 2. [8\.2Results on Switch Transformer](https://arxiv.org/html/2609.05770#S8.SS2) 3. [8\.3Results on OLMoE](https://arxiv.org/html/2609.05770#S8.SS3) 4. [8\.4Empirical Privacy Auditing via Membership Inference](https://arxiv.org/html/2609.05770#S8.SS4)
9. [9Summary](https://arxiv.org/html/2609.05770#S9)
10. [References](https://arxiv.org/html/2609.05770#bib)
11. [AMore Related Works](https://arxiv.org/html/2609.05770#A1)1. [A\.1Structured Private Optimization](https://arxiv.org/html/2609.05770#A1.SS1) 2. [A\.2Partition\-Based DP Composition](https://arxiv.org/html/2609.05770#A1.SS2) 3. [A\.3Sparse Mixture\-of\-Expert Models](https://arxiv.org/html/2609.05770#A1.SS3)
12. [BTheory Analysis](https://arxiv.org/html/2609.05770#A2)1. [B\.1Public and Private Quantities](https://arxiv.org/html/2609.05770#A2.SS1) 2. [B\.2Per\-Update Sensitivity](https://arxiv.org/html/2609.05770#A2.SS2) 3. [B\.3Proof of Lemma](https://arxiv.org/html/2609.05770#A2.SS3) 4. [B\.4Proof of Lemma](https://arxiv.org/html/2609.05770#A2.SS4) 5. [B\.5Proof of Theorem](https://arxiv.org/html/2609.05770#A2.SS5) 6. [B\.6Proof of Theorem](https://arxiv.org/html/2609.05770#A2.SS6) 7. [B\.7SGD\-Surrogate Implication](https://arxiv.org/html/2609.05770#A2.SS7) 8. [B\.8Scaling of the DP Term](https://arxiv.org/html/2609.05770#A2.SS8) 9. [B\.9Stability of expert ownership](https://arxiv.org/html/2609.05770#A2.SS9) 10. [B\.10Voronoi\-aware expert estimator Error](https://arxiv.org/html/2609.05770#A2.SS10) 11. [B\.11Scope of the Bias\-SNR Connection \(Remark 2 details\)](https://arxiv.org/html/2609.05770#A2.SS11)
13. [CExperiment Details](https://arxiv.org/html/2609.05770#A3)
14. [DMore Expriments on Other Models](https://arxiv.org/html/2609.05770#A4)
15. [EAblation Study](https://arxiv.org/html/2609.05770#A5)1. [E\.1Component Ablation](https://arxiv.org/html/2609.05770#A5.SS1) 2. [E\.2Matched\-Scope Diagnostics](https://arxiv.org/html/2609.05770#A5.SS2) 3. [E\.3Sensitivity to the Shared/Expert Budget Splitρ\\rho](https://arxiv.org/html/2609.05770#A5.SS3)
16. [FEmpirical Routing Entropy Analysis](https://arxiv.org/html/2609.05770#A6)
17. [GMechanism Diagnostics](https://arxiv.org/html/2609.05770#A7)
18. [HBudget Allocation: Surrogate Analysis and Empirical Sensitivity](https://arxiv.org/html/2609.05770#A8)
19. [IRuntime and Memory](https://arxiv.org/html/2609.05770#A9)

## Appendix AMore Related Works

### A\.1Structured Private Optimization

A growing body of work exploits structure in the private update\. Gradient filtering and noise\-shaping methods reduce effective noise variance by suppressing or temporally smoothing noise components[Zhang et al\. \(2025\)](https://arxiv.org/html/2609.05770#bib.bib16)\. Parameter\-efficient methods shrink the noised dimension via adapters[Yu et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib4)or restrict updates to low\-rank or learned subspaces[Zheng et al\. \(2026\)](https://arxiv.org/html/2609.05770#bib.bib13)\. Group\-wise clipping[Thakkar et al\. \(2019\)](https://arxiv.org/html/2609.05770#bib.bib20)calibrates the clipping bound to parameter subsets with related gradient scale \- the closest prior idea to our expert\-specific clipping, but applied to layer\-wise or client\-wise groups in a dense setting rather than to routing\-induced sparse owner partitions\.

These methods operate on the temporal, low\-rank, or layer\-wise structure of the gradient\. We exploit a different axis: the sparse routing structure of MoE computation\. Rather than reshaping the gradient subspace or noise trajectory, we align clipping, normalization, and privacy*composition*with the shared\-expert role decomposition\. Critically, our parallel\-composition analysis \(Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1)\) requires disjoint, deterministic, data\-independent owner partitions \- a structural property not present in group\-wise clipping for dense models, where groups are fixed by architecture rather than by private routing decisions\. Subspace, low\-rank, and noise\-shaping methods are orthogonal to our framework and could in principle be applied within each role\-aware stream; we leave this to future work\.

### A\.2Partition\-Based DP Composition

Parallel composition over disjoint data partitions is a classical result in differential privacy[McSherry and Talwar \(2007\)](https://arxiv.org/html/2609.05770#bib.bib21), and has been applied in federated learning[McMahan et al\. \(2017\)](https://arxiv.org/html/2609.05770#bib.bib22), local DP[Kasiviswanathan et al\. \(2011\)](https://arxiv.org/html/2609.05770#bib.bib23), and multi\-party computation[Dwork et al\. \(2010\)](https://arxiv.org/html/2609.05770#bib.bib24)\. Our use of parallel composition differs in two respects\. First, the partition is induced by a private routing mechanism: owner groups\{𝒟l,e\}\\\{\\mathcal\{D\}\_\{l,e\}\\\}are determined by the frozen router applied to private inputs, raising the question of whether the partition itself leaks information\. We sidestep this by fixing the router before private training and computing the owner map once from the frozen pretrained router, so the partition is a deterministic function of public model weights and private inputs \- analogous to a data\-independent hash partition conditioned on the \(public\) router[Vadhan \(2017\)](https://arxiv.org/html/2609.05770#bib.bib28)\. Second, the partition changes across layers; we handle this via sequential composition across layers with stage\-fixed owner maps \(Sec\.[6\.2](https://arxiv.org/html/2609.05770#S6.SS2)\)\.

Data\-dependent partitions \- where the partition itself depends on the private dataset \- require additional privacy accounting for the partitioning step[Dwork et al\. \(2006\)](https://arxiv.org/html/2609.05770#bib.bib27);[Rogers et al\. \(2016\)](https://arxiv.org/html/2609.05770#bib.bib25)\. Our design avoids this: the owner mapala\_\{l\}is a deterministic function of the*frozen public router*and the private inputs, and the router parameters are public\. The privacy cost of computingala\_\{l\}is zero \(it is post\-processing of the public router applied to private inputs, producing internal bookkeeping that is never released\)\. This is stated formally in Appendix[B\.1](https://arxiv.org/html/2609.05770#A2.SS1)and is the key distinction from settings where the partition must itself be privately released[Smith \(2011\)](https://arxiv.org/html/2609.05770#bib.bib26)\.

### A\.3Sparse Mixture\-of\-Expert Models

Sparse MoE layers replace a dense feed\-forward block with a collection ofEEexperts and a router that activates only a small and token\-dependent subset, separating parameter count from per\-token compute[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.05770#bib.bib6)\. The recipe scales through GShard[Lepikhin et al\. \(2020\)](https://arxiv.org/html/2609.05770#bib.bib30), the Switch Transformer[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib7), and GLaM[Du et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib31), and is employed by open flagship LLMs such as Mixtral[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib15), DeepSeek\-V3[DeepSeek\-AI et al\. \(2025\)](https://arxiv.org/html/2609.05770#bib.bib32), and OLMoE[Muennighoff et al\. \(2025\)](https://arxiv.org/html/2609.05770#bib.bib14), the last of which we fine\-tune in Sec\.[8\.3](https://arxiv.org/html/2609.05770#S8.SS3)\. A central design axis is token\-to\-expert assignment: token\-choice top\-kkgating[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.05770#bib.bib6);[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib7)leaves per\-expert load uncontrolled, whereas expert\-choice routing[Zhou et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib33)and deterministic schemes such as BASE layers[Lewis et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib34)and Hash layers[Roller et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib35)enforce balance by changing the assignment mechanism\.

Learned routing tends to collapse onto a handful of experts[Chi et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib36), which has motivated remedies from auxiliary losses that balance load across the batch[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.05770#bib.bib6);[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib7)to the bias adjustment of DeepSeek[Wang et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib37), which balances load without any auxiliary loss\. Each of these depends on routing statistics aggregated over the batch and on the realized load of each expert\. These quantities depend on private data and thus break the sensitivity of individual records that DP\-SGD[Abadi et al\. \(2016\)](https://arxiv.org/html/2609.05770#bib.bib1)requires\. For this reason, we freeze the router, dropℒaux\\mathcal\{L\}\_\{\\text\{aux\}\}\(Sec\.[3\.3](https://arxiv.org/html/2609.05770#S3.SS3)\), and normalize by the public denominatorDe=B/ED\_\{e\}=B/Erather than the realized count \(Sec\.[5\.3](https://arxiv.org/html/2609.05770#S5.SS3)\)\. A complementary line of work isolates shared knowledge in experts that are always active so that the routed experts can specialize[Dai et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib29);[Rajbhandari et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib38)\. This is exactly the asymmetry our framework makes explicit: experts that see every record join the shared stream, while only the conditionally routed experts enter the expert stream \(Sec\.[3\.3](https://arxiv.org/html/2609.05770#S3.SS3)\)\. Routing entropy has also been a way to measure how evenly experts are used in language and vision MoEs[Riquelme et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib39)\. We reuse it to select a layer from public data at no privacy cost \(Sec\.[7](https://arxiv.org/html/2609.05770#S7)\) and connect it to the bias of the estimator based on the public denominator \(Theorem[2](https://arxiv.org/html/2609.05770#Thmtheorem2)\)\.

## Appendix BTheory Analysis

### B\.1Public and Private Quantities

The privacy guarantee is with respect to𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\. Public quantities include: the pretrained checkpoint, frozen router, candidate layer set,BB,EE,CsC\_\{s\},CeC\_\{e\},σs\\sigma\_\{s\},σe\\sigma\_\{e\}, the alternating update schedule, and𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}\(disjoint from𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\)\. All DP statements are conditioned on these and onl⋆l^\{\\star\}\.

The owner mapal:𝒳→\[E\]a\_\{l\}:\\mathcal\{X\}\\to\[E\]and the realized owner counts\|𝒟l,e\|\|\\mathcal\{D\}\_\{l,e\}\|are*private internal quantities*:ala\_\{l\}is computed once before private training using only the frozen public router, is fixed for the duration of the stage, and is never released\. Realized counts\|ℬt,l,e\|\|\\mathcal\{B\}\_\{t,l,e\}\|are never read by the update rule, never released, and never used in branching logic\. Under add/remove adjacency, a one\-record change affects exactly one owner group, which is the structural fact that enables Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1)\. If the router or checkpoint were trained on𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}without DP, that step would incur additional privacy cost\. Our protocol avoids this via a frozen public router and public\-only layer selection\.

### B\.2Per\-Update Sensitivity

#### Shared stream\.

Each per\-record gradient is clipped toCsC\_\{s\}, so the clipped sum changes by at mostCsC\_\{s\}under add/remove\. Normalizing byBB:

Δsh=CsB\.\\Delta\_\{\\mathrm\{sh\}\}=\\frac\{C\_\{s\}\}\{B\}\.\(19\)

#### Expert stream\.

Each per\-record contribution is clipped toCeC\_\{e\}; normalizing byDe=B/ED\_\{e\}=B/E:

Δl,e=CeDe=E​CeB\.\\Delta\_\{l,e\}=\\frac\{C\_\{e\}\}\{D\_\{e\}\}=\\frac\{EC\_\{e\}\}\{B\}\.\(20\)This is independent of the realized count\|ℬt,l,e\|\|\\mathcal\{B\}\_\{t,l,e\}\|\. Residual weightswiw\_\{i\}do not affect sensitivity since they enter before clipping; the clipped contribution is at mostCeC\_\{e\}regardless ofwi∈\[0,1\]w\_\{i\}\\in\[0,1\]\.

#### Adam\.

Moment updates are deterministic functions of privatized gradients and the public schedule, hence post\-processing that consumes no additional budget\.

### B\.3Proof of Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1)

###### Proof\.

Let𝒟,𝒟′\\mathcal\{D\},\\mathcal\{D\}^\{\\prime\}differ by one record\(x0,y0\)\(x\_\{0\},y\_\{0\}\), which belongs to exactly one groupe0=al​\(x0\)e\_\{0\}=a\_\{l\}\(x\_\{0\}\)by conditions \(i\)–\(ii\)\. For alle≠e0e\\neq e\_\{0\}, the group𝒟l,e\\mathcal\{D\}\_\{l,e\}is identical under both datasets, so by condition \(iii\) the output distribution ofMl,eM\_\{l,e\}is unchanged\. The jointMlM\_\{l\}therefore differs only throughMl,e0M\_\{l,e\_\{0\}\}, which is\(εl,e0,δl,e0\)\(\\varepsilon\_\{l,e\_\{0\}\},\\delta\_\{l,e\_\{0\}\}\)\-DP\. Taking the worst case overe0e\_\{0\}gives the stated bound\. We additionally verify thatMl,eM\_\{l,e\}satisfies condition \(iii\) under alternating optimization\. The expert mechanism at steptttakes as input: \(a\) the owner subsetℬt,l,e⊆𝒟l,e\\mathcal\{B\}\_\{t,l,e\}\\subseteq\\mathcal\{D\}\_\{l,e\}, \(b\) the current model stateθt\\theta\_\{t\}, and \(c\) public hyperparameters\. Sinceθt\\theta\_\{t\}is a deterministic post\-processing of previously privatized gradients \(Remark[3](https://arxiv.org/html/2609.05770#Thmremark3)\), it constitutes a public quantity conditioned on the released mechanism outputs\. No private quantity outside𝒟l,e\\mathcal\{D\}\_\{l,e\}entersMl,eM\_\{l,e\}directly; the only cross\-group information flows through the DP\-protectedθt\\theta\_\{t\}, which is already accounted for in the sequential composition across steps \(Sec\.[6\.2](https://arxiv.org/html/2609.05770#S6.SS2)\)\. ∎

### B\.4Proof of Lemma[2](https://arxiv.org/html/2609.05770#Thmlemma2)

###### Lemma 2\(Residual weighting does not break parallel composition\)\.

Fix stepttand layerℓ⋆\\ell^\{\\star\}\. Letθt\\theta\_\{t\}be the model state aftert−1t\-1complete alternating cycles, and let

wi=1−softmax​\(fsh​\(xi,θt\)\)yi∈\[0,1\]w\_\{i\}\\;=\\;1\-\\mathrm\{softmax\}\\\!\\bigl\(f\_\{\\mathrm\{sh\}\}\(x\_\{i\};\\,\\theta\_\{t\}\)\\bigr\)\_\{y\_\{i\}\}\\;\\in\\;\[0,1\]\(21\)be the residual weight for recordii\. Define the expert mechanism at stepttas

Mℓ,e\(t\)​\(Dℓ,e\)=\\displaystyle M\_\{\\ell,e\}^\{\(t\)\}\(D\_\{\\ell,e\}\)\\;=1De​\(∑i∈Bt,ℓ,eclip⁡\(wi⋅gi,t,ℓ,eexp,Ce\)CLOSE\\displaystyle\\frac\{1\}\{D\_\{e\}\}\\Biggl\(\\sum\_\{i\\in B\_\{t,\\ell,e\}\}\\mathrm\{clip\}\\\!\\bigl\(w\_\{i\}\\cdot g^\{\\mathrm\{exp\}\}\_\{i,t,\\ell,e\},\\;C\_\{e\}\\bigr\)\(22\)OPEN\+𝒩⁡\(0,σe2​Ce2​I\)\)\.\\displaystyle\+\\;\\mathcal\{N\}\\\!\\left\(0,\\,\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}\\,I\\right\)\\Biggr\)\.ThenMℓ,e\(t\)M\_\{\\ell,e\}^\{\(t\)\}satisfies condition \(iii\) of Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1): it depends onDℓ,eD\_\{\\ell,e\}and*effectively public*quantities, enabling parallel composition across owner groups\.

###### Proof\.

We proceed in three parts\.

Part A:θt\\theta\_\{t\}is a releaseddpoutput, not a free cross\-group channel\.

By the post\-processing property of differential privacy\([Dwork and Roth, 2014](https://arxiv.org/html/2609.05770#bib.bib2)\), any deterministic function of an\(ε,δ\)\(\\varepsilon,\\delta\)\-DP output is itself\(ε,δ\)\(\\varepsilon,\\delta\)\-DP\. The sequence of released privatized gradients up to stepttis

τ<t=\{\\displaystyle\\tau\_\{<t\}\\;=\\;\\bigl\\\{g¯1sh,\{g¯1,ℓ,eexp\}e∈\[E\],\\displaystyle\\bar\{g\}^\{\\mathrm\{sh\}\}\_\{1\},\\;\\\{\\bar\{g\}^\{\\mathrm\{exp\}\}\_\{1,\\ell,e\}\\\}\_\{e\\in\[E\]\},\(23\)…,g¯sht−1,\{g¯expt−1,ℓ,e\}e∈\[E\]\}\.\\displaystyle\\ldots,\\;\\bar\{g\}^\{\\mathrm\{sh\}\}\_\{t\-1\},\\;\\\{\\bar\{g\}^\{\\mathrm\{exp\}\}\_\{t\-1,\\ell,e\}\\\}\_\{e\\in\[E\]\}\\;\\bigr\\\}\.By sequential composition \(Sec\.[6\.2](https://arxiv.org/html/2609.05770#S6.SS2)\),τ<t\\tau\_\{<t\}is\(ε<t,δ<t\)\(\\varepsilon\_\{<t\},\\delta\_\{<t\}\)\-DP, and this cost is already fully accounted for inωtot\\omega\_\{\\mathrm\{tot\}\}\(Eq\.[10](https://arxiv.org/html/2609.05770#S6.E10)\)\. The model stateθt=optimizer⁡\(τ<t,public schedule\)\\theta\_\{t\}=\\mathrm\{optimizer\}\(\\tau\_\{<t\},\\,\\text\{public schedule\}\)is a deterministic function ofτ<t\\tau\_\{<t\}and public hyperparameters; by post\-processing it is therefore\(ε<t,δ<t\)\(\\varepsilon\_\{<t\},\\delta\_\{<t\}\)\-DP and carries*no additional privacy cost*beyond what sequential composition already charges\.

Crucially,θt\\theta\_\{t\}is a function of*already\-released*mechanism outputs, not a new query intoDprivD\_\{\\mathrm\{priv\}\}\. Conditioning onθt\\theta\_\{t\}in the definition ofMℓ,e\(t\)M\_\{\\ell,e\}^\{\(t\)\}is therefore equivalent to conditioning on a public side\-channel whose privacy cost is already captured\.

Part B:wiw\_\{i\}does not introduce cross\-group record dependencies at steptt\.

We decompose the dependence ofMℓ,e\(t\)M\_\{\\ell,e\}^\{\(t\)\}onDprivD\_\{\\mathrm\{priv\}\}into two channels\.

*\(i\) Direct channel\.*Mℓ,e\(t\)M\_\{\\ell,e\}^\{\(t\)\}reads records inBt,ℓ,e⊆Dℓ,eB\_\{t,\\ell,e\}\\subseteq D\_\{\\ell,e\}directly to compute per\-record gradients\. By the disjoint ownership condition \(Sec\. 5\.2\), no record outsideDℓ,eD\_\{\\ell,e\}appears inBt,ℓ,eB\_\{t,\\ell,e\}\.

*\(ii\) Indirect channel viaθt\\theta\_\{t\}\.*The residual weightwiw\_\{i\}depends onθt\\theta\_\{t\}, which was influenced by records outsideDℓ,eD\_\{\\ell,e\}through prior shared\-stream steps\. However, this influence has already been privatized: every recordxj∉Dℓ,ex\_\{j\}\\notin D\_\{\\ell,e\}that contributed toθt\\theta\_\{t\}did so only through the released privatized gradientsτ<t\\tau\_\{<t\}, whose privacy cost is captured inε<t\\varepsilon\_\{<t\}\. The weight

wi=1−softmax​\(fsh​\(xi,θt\)\)yiw\_\{i\}\\;=\\;1\-\\mathrm\{softmax\}\\\!\\bigl\(f\_\{\\mathrm\{sh\}\}\(x\_\{i\};\\,\\theta\_\{t\}\)\\bigr\)\_\{y\_\{i\}\}\(24\)is therefore a post\-processing of previously released DP outputs composed with the private record\(xi,yi\)\(x\_\{i\},y\_\{i\}\), where\(xi,yi\)∈Dℓ,e\(x\_\{i\},y\_\{i\}\)\\in D\_\{\\ell,e\}by construction\. No recordxj∉Dℓ,ex\_\{j\}\\notin D\_\{\\ell,e\}is read afresh at stepttthroughwiw\_\{i\}; its influence is confined to the already\-accountedθt\\theta\_\{t\}\.

Part C: Sensitivity is unaffected bywiw\_\{i\}\.

Sincewi∈\[0,1\]w\_\{i\}\\in\[0,1\]and clipping is applied*after*multiplication,

‖clip⁡\(wi⋅gi,t,ℓ,eexp,Ce\)‖2≤Ce∀wi∈\[0,1\]\.\\bigl\\\|\\mathrm\{clip\}\\\!\\bigl\(w\_\{i\}\\cdot g^\{\\mathrm\{exp\}\}\_\{i,t,\\ell,e\},\\;C\_\{e\}\\bigr\)\\bigr\\\|\_\{2\}\\;\\leq\\;C\_\{e\}\\qquad\\forall\\,w\_\{i\}\\in\[0,1\]\.\(25\)The per\-update sensitivityΔℓ,e=E​Ce/B\\Delta\_\{\\ell,e\}=EC\_\{e\}/B\(Eq\. \(25\)\) holds regardless ofwiw\_\{i\}\. The noise magnitudeσe​Ce\\sigma\_\{e\}C\_\{e\}is calibrated toΔℓ,e\\Delta\_\{\\ell,e\}, so the Gaussian mechanism guarantees\(εe,δe\)\(\\varepsilon\_\{e\},\\delta\_\{e\}\)\-DP onDℓ,eD\_\{\\ell,e\}with the same parameters as thewi=1w\_\{i\}=1case\.

Conclusion\.Mℓ,e\(t\)M\_\{\\ell,e\}^\{\(t\)\}readsDℓ,eD\_\{\\ell,e\}directly \(Parts A–B\), has bounded sensitivity \(Part C\), and its dependence on records outsideDℓ,eD\_\{\\ell,e\}is fully mediated through the DP\-protectedθt\\theta\_\{t\}whose cost is already accounted for in sequential composition\. Condition \(iii\) of Lemma[1](https://arxiv.org/html/2609.05770#Thmlemma1)is therefore satisfied, and parallel composition applies\. ∎

### B\.5Proof of Theorem[1](https://arxiv.org/html/2609.05770#Thmtheorem1)

###### Proof\.

WriteSe=∑i∈𝒟ebi​hiexpS\_\{e\}=\\sum\_\{i\\in\\mathcal\{D\}\_\{e\}\}b\_\{i\}h\_\{i\}^\{\\mathrm\{exp\}\}withbi​∼iid​Bernoulli​\(p\)b\_\{i\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\mathrm\{Bernoulli\}\(p\)\. Then𝔼​Se=p​ne​μe=B​qe​μe\\mathbb\{E\}S\_\{e\}=pn\_\{e\}\\mu\_\{e\}=Bq\_\{e\}\\mu\_\{e\}, so

𝔼​g¯t,eexp=EB​𝔼​Se=E​qe​μe,\\mathbb\{E\}\\bar\{g\}\_\{t,e\}^\{\\mathrm\{exp\}\}=\\tfrac\{E\}\{B\}\\mathbb\{E\}S\_\{e\}=Eq\_\{e\}\\mu\_\{e\},\(26\)giving bias\(E​qe−1\)​μe\(Eq\_\{e\}\-1\)\\mu\_\{e\}andβe2=\(E​qe−1\)2​‖μe‖22\\beta\_\{e\}^\{2\}=\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\. Bernoulli sampling givesCov⁡\(Se\)=p⁡\(1−p\)​ne​\(μe​μe⊤\+Σe\)\\mathrm\{Cov\}\(S\_\{e\}\)=p\(1\-p\)n\_\{e\}\(\\mu\_\{e\}\\mu\_\{e\}^\{\\top\}\+\\Sigma\_\{e\}\)\. Sinceξt,e⟂Se\\xi\_\{t,e\}\\perp S\_\{e\},

Cov⁡\(g¯t,eexp\)=E2B2​\[p⁡\(1−p\)​ne​\(μe​μe⊤\+Σe\)\+σe2​Ce2​Id\]\.\\mathrm\{Cov\}\(\\bar\{g\}\_\{t,e\}^\{\\mathrm\{exp\}\}\)=\\frac\{E^\{2\}\}\{B^\{2\}\}\\bigl\[p\(1\-p\)n\_\{e\}\(\\mu\_\{e\}\\mu\_\{e\}^\{\\top\}\+\\Sigma\_\{e\}\)\+\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}I\_\{d\}\\bigr\]\.\(27\)Taking traces and applying the bias\-variance identity yields Eqs\. \([11](https://arxiv.org/html/2609.05770#S7.E11)\)–\([13](https://arxiv.org/html/2609.05770#S7.E13)\)\. ∎

### B\.6Proof of Theorem[2](https://arxiv.org/html/2609.05770#Thmtheorem2)

###### Proof\.

From Theorem[1](https://arxiv.org/html/2609.05770#Thmtheorem1),βe2≤G¯2​\(E​qe−1\)2\\beta\_\{e\}^\{2\}\\leq\\bar\{G\}^\{2\}\(Eq\_\{e\}\-1\)^\{2\}\.

#### Eq\. \([15](https://arxiv.org/html/2609.05770#S7.E15)\) \(unweighted average\)\.

Recallχ2\(q∥u\)=E∑e\(qe−1/E\)2=E−1∑e\(Eqe−1\)2\\chi^\{2\}\(q\\\|u\)=E\\sum\_\{e\}\(q\_\{e\}\-1/E\)^\{2\}=E^\{\-1\}\\sum\_\{e\}\(Eq\_\{e\}\-1\)^\{2\}, so∑e\(Eqe−1\)2=Eχ2\(q∥u\)\\sum\_\{e\}\(Eq\_\{e\}\-1\)^\{2\}=E\\chi^\{2\}\(q\\\|u\)\. Averaging:

1E∑eβe2≤G¯2E∑e\(Eqe−1\)2=G¯2χ2\(q∥u\)\.\\tfrac\{1\}\{E\}\\\!\\sum\_\{e\}\\beta\_\{e\}^\{2\}\\leq\\tfrac\{\\bar\{G\}^\{2\}\}\{E\}\\\!\\sum\_\{e\}\(Eq\_\{e\}\-1\)^\{2\}=\\bar\{G\}^\{2\}\\chi^\{2\}\(q\\\|u\)\.

#### Eq\. \([16](https://arxiv.org/html/2609.05770#S7.E16)\) \(frequency\-weighted\)\.

Becauseβe2≤G¯2​\(E​qe−1\)2\\beta\_\{e\}^\{2\}\\leq\\bar\{G\}^\{2\}\(Eq\_\{e\}\-1\)^\{2\},

∑eqe​βe2\\displaystyle\\sum\_\{e\}q\_\{e\}\\beta\_\{e\}^\{2\}≤G¯2​∑eqe​\(E​qe−1\)2\.\\displaystyle\\leq\\bar\{G\}^\{2\}\\\!\\sum\_\{e\}q\_\{e\}\(Eq\_\{e\}\-1\)^\{2\}\.\(28\)Expanding\(E​qe−1\)2=E2​qe2−2​E​qe\+1\(Eq\_\{e\}\-1\)^\{2\}=E^\{2\}q\_\{e\}^\{2\}\-2Eq\_\{e\}\+1:

∑eqe​\(E​qe−1\)2\\displaystyle\\sum\_\{e\}q\_\{e\}\(Eq\_\{e\}\\\!\-\\\!1\)^\{2\}=E2​∑eqe3−2​E​∑eqe2\+1\.\\displaystyle=E^\{2\}\\\!\\sum\_\{e\}q\_\{e\}^\{3\}\-2E\\\!\\sum\_\{e\}q\_\{e\}^\{2\}\+1\.\(29\)Applyingqe3≤qe2q\_\{e\}^\{3\}\\leq q\_\{e\}^\{2\}\(sinceqe≤1q\_\{e\}\\leq 1\):

\([29](https://arxiv.org/html/2609.05770#A2.E29)\)≤\(E2−2​E\)​∑eqe2\+1\.\\displaystyle\\leq\(E^\{2\}\\\!\-\\\!2E\)\\\!\\sum\_\{e\}q\_\{e\}^\{2\}\+1\.\(30\)The identityχ2\(q∥u\)=E∑eqe2−1\\chi^\{2\}\(q\\\|u\)=E\\sum\_\{e\}q\_\{e\}^\{2\}\-1gives∑eqe2=\(χ2\(q∥u\)\+1\)/E\\sum\_\{e\}q\_\{e\}^\{2\}=\(\\chi^\{2\}\(q\\\|u\)\+1\)/E, so:

\([30](https://arxiv.org/html/2609.05770#A2.E30)\)=\(E−2\)χ2\(q∥u\)\+\(E−1\)\.\\displaystyle=\(E\\\!\-\\\!2\)\\chi^\{2\}\(q\\\|u\)\+\(E\\\!\-\\\!1\)\.\(31\)Combining \([28](https://arxiv.org/html/2609.05770#A2.E28)\)–\([31](https://arxiv.org/html/2609.05770#A2.E31)\):

∑eqeβe2≤G¯2\[\(E−2\)χ2\(q∥u\)\+\(E−1\)\]\.\\sum\_\{e\}q\_\{e\}\\beta\_\{e\}^\{2\}\\leq\\bar\{G\}^\{2\}\\\!\\left\[\(E\\\!\-\\\!2\)\\chi^\{2\}\(q\\\|u\)\+\(E\\\!\-\\\!1\)\\right\]\.\(32\)SinceE≥2E\\geq 2both coefficients are non\-negative\. A coarser path viaqe≤1q\_\{e\}\\leq 1directly gives:

∑eqe​\(E​qe−1\)2\\displaystyle\\sum\_\{e\}q\_\{e\}\(Eq\_\{e\}\\\!\-\\\!1\)^\{2\}≤∑e\(Eqe−1\)2=Eχ2\(q∥u\),\\displaystyle\\leq\\sum\_\{e\}\(Eq\_\{e\}\\\!\-\\\!1\)^\{2\}=E\\chi^\{2\}\(q\\\|u\),\(33\)recovering∑eqeβe2≤G¯2Eχ2\(q∥u\)\\sum\_\{e\}q\_\{e\}\\beta\_\{e\}^\{2\}\\leq\\bar\{G\}^\{2\}E\\chi^\{2\}\(q\\\|u\), i\.e\. Eq\. \([16](https://arxiv.org/html/2609.05770#S7.E16)\)\. Bound \([32](https://arxiv.org/html/2609.05770#A2.E32)\) is tighter for smallχ2\\chi^\{2\}; bound \([33](https://arxiv.org/html/2609.05770#A2.E33)\) is simpler and suffices for the entropy criterion in Sec\.[7](https://arxiv.org/html/2609.05770#S7)\.∎

### B\.7SGD\-Surrogate Implication

LetFbalF\_\{\\mathrm\{bal\}\}beLL\-smooth and PL with parameterβ\>0\\beta\>0\. For biasbtb\_\{t\}and noiseztz\_\{t\}with𝔼​‖zt‖22≤V\\mathbb\{E\}\\\|z\_\{t\}\\\|\_\{2\}^\{2\}\\leq V,𝔼​‖bt‖22≤Bimb\\mathbb\{E\}\\\|b\_\{t\}\\\|\_\{2\}^\{2\}\\leq B\_\{\\mathrm\{imb\}\}, andη≤1/L\\eta\\leq 1/L:

𝔼⁡\[Fbal​\(θT\)\]−Fbal∗\\displaystyle\\mathbb\{E\}\[F\_\{\\mathrm\{bal\}\}\(\\theta\_\{T\}\)\]\-F\_\{\\mathrm\{bal\}\}^\{\*\}≤\(1−η​β\)T​Δ0\+O⁡\(L​η​Vβ\)\+O⁡\(L​Bimbβ2\),\\displaystyle\\quad\\leq\(1\-\\eta\\beta\)^\{T\}\\Delta\_\{0\}\+O\\\!\\left\(\\tfrac\{L\\eta V\}\{\\beta\}\\right\)\+O\\\!\\left\(\\tfrac\{LB\_\{\\mathrm\{imb\}\}\}\{\\beta^\{2\}\}\\right\),\(34\)whereΔ0=Fbal​\(θ0\)−Fbal∗\\Delta\_\{0\}=F\_\{\\mathrm\{bal\}\}\(\\theta\_\{0\}\)\-F\_\{\\mathrm\{bal\}\}^\{\*\}\. BoundingBimbB\_\{\\mathrm\{imb\}\}via Theorem[2](https://arxiv.org/html/2609.05770#Thmtheorem2)and Eq\. \([17](https://arxiv.org/html/2609.05770#S7.E17)\) shows routing imbalance contributes an error vanishing at uniform routing\.*This is an explanatory bound within a surrogate model, not a convergence guarantee for DP\-Adam on non\-convex MoE networks\.*

### B\.8Scaling of the DP Term

Under balanced routing, Corollary[1](https://arxiv.org/html/2609.05770#Thmcorollary1)gives𝔼​‖g¯t,eexp−μe‖22≤Ce​E/B\+E​σe​Ce​d/B\\sqrt\{\\mathbb\{E\}\\\|\\bar\{g\}\_\{t,e\}^\{\\mathrm\{exp\}\}\-\\mu\_\{e\}\\\|\_\{2\}^\{2\}\}\\leq C\_\{e\}\\sqrt\{E/B\}\+E\\sigma\_\{e\}C\_\{e\}\\sqrt\{d\}/B\. Withσe=Θ⁡\(log⁡\(1/δ\)/εstep\)\\sigma\_\{e\}=\\Theta\(\\sqrt\{\\log\(1/\\delta\)\}/\\varepsilon\_\{\\mathrm\{step\}\}\), the privacy term scales as

E​Ce​d​log⁡\(1/δ\)B​εstep,\\frac\{EC\_\{e\}\\sqrt\{d\\log\(1/\\delta\)\}\}\{B\\,\\varepsilon\_\{\\mathrm\{step\}\}\},\(35\)matching the standard per\-expert DP mean\-estimation rate up to log factors whenB=Θ⁡\(n\)B=\\Theta\(n\)\. This is a single\-estimator result, not a convergence guarantee for DP\-Adam\.

### B\.9Stability of expert ownership

This appendix formalizes the stability claim of Sec\.[7\.2](https://arxiv.org/html/2609.05770#S7.SS2): because the owner mapal⋆a\_\{l^\{\\star\}\}is frozen before training and never recomputed, we ask how far the induced ownership partition can drift as the upstream representationϕ=θsh​\(⋅\)\\phi=\\theta\_\{\\mathrm\{sh\}\}\(\\cdot\)evolves, under a Voronoi abstraction of ownership\.

###### Definition 2\(Voronoi expert ownership\)\.

Letϕ:𝒳→ℝm\\phi:\\mathcal\{X\}\\to\\mathbb\{R\}^\{m\}be a frozen representation map and letC=\{ce\}e=1E⊂ℝmC=\\\{c\_\{e\}\\\}\_\{e=1\}^\{E\}\\subset\\mathbb\{R\}^\{m\}be expert prototypes\. The Voronoi cell of experteeis𝒱e=\{z:∥z−ce∥22≤∥z−cj∥22,∀j∈\[E\]\}\\mathcal\{V\}\_\{e\}=\\\{z:\\\|z\-c\_\{e\}\\\|\_\{2\}^\{2\}\\leq\\\|z\-c\_\{j\}\\\|\_\{2\}^\{2\},\\forall j\\in\[E\]\\\}\. The induced owner map isaC​\(x\)=arg⁡mine∈\[E\]⁡‖ϕ⁡\(x\)−ce‖22a\_\{C\}\(x\)=\\arg\\min\_\{e\\in\[E\]\}\\\|\\phi\(x\)\-c\_\{e\}\\\|\_\{2\}^\{2\}, with deterministic tie\-breaking\. The routing mass isqe=ℙ⁡\(aC​\(X\)=e\)q\_\{e\}=\\mathbb\{P\}\(a\_\{C\}\(X\)=e\)\.

###### Definition 3\(Voronoi margin and boundary regularity\)\.

Forz=ϕ⁡\(x\)z=\\phi\(x\), leta=aC​\(x\)a=a\_\{C\}\(x\)and define the marginγC​\(z\)=minj≠a⁡\{‖z−cj‖22−‖z−ca‖22\}\\gamma\_\{C\}\(z\)=\\min\_\{j\\neq a\}\\\{\\\|z\-c\_\{j\}\\\|\_\{2\}^\{2\}\-\\\|z\-c\_\{a\}\\\|\_\{2\}^\{2\}\\\}\. LetΔC=maxj,k⁡‖cj−ck‖2\\Delta\_\{C\}=\\max\_\{j,k\}\\\|c\_\{j\}\-c\_\{k\}\\\|\_\{2\}\. We say that the representation distribution satisfies\(κ,C\)\(\\kappa,C\)\-boundary regularity ifℙ⁡\(γC​\(ϕ⁡\(X\)\)≤t\)≤κ​t\\mathbb\{P\}\(\\gamma\_\{C\}\(\\phi\(X\)\)\\leq t\)\\leq\\kappa tfor allt\>0t\>0\.

###### Definition 4\(Perturbed ownership\)\.

Given another representation mapϕ~:𝒳→ℝm\\tilde\{\\phi\}:\\mathcal\{X\}\\to\\mathbb\{R\}^\{m\}, definea~C​\(x\)=arg⁡mine∈\[E\]⁡‖ϕ~​\(x\)−ce‖22\\tilde\{a\}\_\{C\}\(x\)=\\arg\\min\_\{e\\in\[E\]\}\\\|\\tilde\{\\phi\}\(x\)\-c\_\{e\}\\\|\_\{2\}^\{2\}andq~e=ℙ⁡\(a~C​\(X\)=e\)\\tilde\{q\}\_\{e\}=\\mathbb\{P\}\(\\tilde\{a\}\_\{C\}\(X\)=e\), with the same deterministic tie\-breaking\.

###### Theorem 5\(Voronoi stability of expert ownership\)\.

If‖ϕ~​\(x\)−ϕ⁡\(x\)‖2≤r\\\|\\tilde\{\\phi\}\(x\)\-\\phi\(x\)\\\|\_\{2\}\\leq rfor allxx, thena~C​\(x\)=aC​\(x\)\\tilde\{a\}\_\{C\}\(x\)=a\_\{C\}\(x\)for everyxxsuch thatγC​\(ϕ⁡\(x\)\)\>2​ΔC​r\\gamma\_\{C\}\(\\phi\(x\)\)\>2\\Delta\_\{C\}r\. Moreover, under\(κ,C\)\(\\kappa,C\)\-boundary regularity,ℙ⁡\(a~C​\(X\)≠aC​\(X\)\)≤2​κ​ΔC​r\\mathbb\{P\}\(\\tilde\{a\}\_\{C\}\(X\)\\neq a\_\{C\}\(X\)\)\\leq 2\\kappa\\Delta\_\{C\}rand‖q~−q‖1≤4​κ​ΔC​r\\\|\\tilde\{q\}\-q\\\|\_\{1\}\\leq 4\\kappa\\Delta\_\{C\}r\.

###### Proof\.

Fixx∈𝒳x\\in\\mathcal\{X\}and writez=ϕ⁡\(x\)z=\\phi\(x\),z~=ϕ~​\(x\)\\tilde\{z\}=\\tilde\{\\phi\}\(x\), anda=aC​\(x\)a=a\_\{C\}\(x\)\. For eachj≠aj\\neq a, define

ha,j​\(u\)=‖u−cj‖22−‖u−ca‖22\.h\_\{a,j\}\(u\)=\\\|u\-c\_\{j\}\\\|\_\{2\}^\{2\}\-\\\|u\-c\_\{a\}\\\|\_\{2\}^\{2\}\.Sincea=arg⁡mine⁡‖z−ce‖22a=\\arg\\min\_\{e\}\\\|z\-c\_\{e\}\\\|\_\{2\}^\{2\},ha,j​\(z\)≥0h\_\{a,j\}\(z\)\\geq 0for allj≠aj\\neq a, and

γC​\(z\)=minj≠a⁡ha,j​\(z\)\.\\gamma\_\{C\}\(z\)=\\min\_\{j\\neq a\}h\_\{a,j\}\(z\)\.Expanding the squared norms gives

ha,j​\(u\)\\displaystyle h\_\{a,j\}\(u\)=−2​⟨u,cj⟩\+‖cj‖22\+2​⟨u,ca⟩−‖ca‖22\\displaystyle=\-2\\langle u,c\_\{j\}\\rangle\+\\\|c\_\{j\}\\\|\_\{2\}^\{2\}\+2\\langle u,c\_\{a\}\\rangle\-\\\|c\_\{a\}\\\|\_\{2\}^\{2\}=2​⟨u,ca−cj⟩\+‖cj‖22−‖ca‖22\.\\displaystyle=2\\langle u,c\_\{a\}\-c\_\{j\}\\rangle\+\\\|c\_\{j\}\\\|\_\{2\}^\{2\}\-\\\|c\_\{a\}\\\|\_\{2\}^\{2\}\.Therefore

ha,j​\(z~\)−ha,j​\(z\)\\displaystyle h\_\{a,j\}\(\\tilde\{z\}\)\-h\_\{a,j\}\(z\)=2​⟨z~−z,ca−cj⟩\.\\displaystyle=2\\langle\\tilde\{z\}\-z,c\_\{a\}\-c\_\{j\}\\rangle\.By Cauchy–Schwarz andΔC=maxp,q⁡‖cp−cq‖2\\Delta\_\{C\}=\\max\_\{p,q\}\\\|c\_\{p\}\-c\_\{q\}\\\|\_\{2\},

ha,j​\(z~\)\\displaystyle h\_\{a,j\}\(\\tilde\{z\}\)=ha,j​\(z\)\+2​⟨z~−z,ca−cj⟩\\displaystyle=h\_\{a,j\}\(z\)\+2\\langle\\tilde\{z\}\-z,c\_\{a\}\-c\_\{j\}\\rangle≥ha,j​\(z\)−2​‖z~−z‖2​‖ca−cj‖2\\displaystyle\\geq h\_\{a,j\}\(z\)\-2\\\|\\tilde\{z\}\-z\\\|\_\{2\}\\\|c\_\{a\}\-c\_\{j\}\\\|\_\{2\}≥ha,j​\(z\)−2​r​ΔC\.\\displaystyle\\geq h\_\{a,j\}\(z\)\-2r\\Delta\_\{C\}\.Taking the minimum overj≠aj\\neq ayields

minj≠a⁡ha,j​\(z~\)\\displaystyle\\min\_\{j\\neq a\}h\_\{a,j\}\(\\tilde\{z\}\)≥minj≠a⁡ha,j​\(z\)−2​r​ΔC\\displaystyle\\geq\\min\_\{j\\neq a\}h\_\{a,j\}\(z\)\-2r\\Delta\_\{C\}=γC​\(z\)−2​r​ΔC\.\\displaystyle=\\gamma\_\{C\}\(z\)\-2r\\Delta\_\{C\}\.Thus, ifγC​\(z\)\>2​r​ΔC\\gamma\_\{C\}\(z\)\>2r\\Delta\_\{C\}, thenha,j​\(z~\)\>0h\_\{a,j\}\(\\tilde\{z\}\)\>0for everyj≠aj\\neq a\. Equivalently,

‖z~−ca‖22<‖z~−cj‖22,∀j≠a\.\\\|\\tilde\{z\}\-c\_\{a\}\\\|\_\{2\}^\{2\}<\\\|\\tilde\{z\}\-c\_\{j\}\\\|\_\{2\}^\{2\},\\qquad\\forall j\\neq a\.Hencea~C​\(x\)=aC​\(x\)\\tilde\{a\}\_\{C\}\(x\)=a\_\{C\}\(x\)\. Therefore

\{a~C\(X\)≠aC\(X\)\}⊆\{γC\(ϕ\(X\)\)≤2rΔC\}\.\\\{\\tilde\{a\}\_\{C\}\(X\)\\neq a\_\{C\}\(X\)\\\}\\subseteq\\\{\\gamma\_\{C\}\(\\phi\(X\)\)\\leq 2r\\Delta\_\{C\}\\\}\.By\(κ,C\)\(\\kappa,C\)\-boundary regularity,

ℙ⁡\(a~C​\(X\)≠aC​\(X\)\)\\displaystyle\\mathbb\{P\}\(\\tilde\{a\}\_\{C\}\(X\)\\neq a\_\{C\}\(X\)\)≤ℙ⁡\(γC​\(ϕ⁡\(X\)\)≤2​r​ΔC\)\\displaystyle\\leq\\mathbb\{P\}\(\\gamma\_\{C\}\(\\phi\(X\)\)\\leq 2r\\Delta\_\{C\}\)≤2​κ​r​ΔC\.\\displaystyle\\leq 2\\kappa r\\Delta\_\{C\}\.
It remains to control the routing mass\. LetA=aC​\(X\)A=a\_\{C\}\(X\)andA~=a~C​\(X\)\\tilde\{A\}=\\tilde\{a\}\_\{C\}\(X\)\. For each expertee,

\|q~e−qe\|\\displaystyle\|\\tilde\{q\}\_\{e\}\-q\_\{e\}\|=\|ℙ⁡\(A~=e\)−ℙ⁡\(A=e\)\|\\displaystyle=\\left\|\\mathbb\{P\}\(\\tilde\{A\}=e\)\-\\mathbb\{P\}\(A=e\)\\right\|=\|𝔼\[𝟏\{A~=e\}−𝟏\{A=e\}\]\|\\displaystyle=\\left\|\\mathbb\{E\}\[\\mathbf\{1\}\\\{\\tilde\{A\}=e\\\}\-\\mathbf\{1\}\\\{A=e\\\}\]\\right\|≤𝔼\|𝟏\{A~=e\}−𝟏\{A=e\}\|\.\\displaystyle\\leq\\mathbb\{E\}\\left\|\\mathbf\{1\}\\\{\\tilde\{A\}=e\\\}\-\\mathbf\{1\}\\\{A=e\\\}\\right\|\.The two indicators differ only when the assignments disagree, and more explicitly,

\|𝟏\{A~=e\}−𝟏\{A=e\}\|\\displaystyle\\left\|\\mathbf\{1\}\\\{\\tilde\{A\}=e\\\}\-\\mathbf\{1\}\\\{A=e\\\}\\right\|≤𝟏​\{A~=e,A≠e\}\\displaystyle\\leq\\mathbf\{1\}\\\{\\tilde\{A\}=e,A\\neq e\\\}\+𝟏​\{A=e,A~≠e\}\.\\displaystyle\+\\mathbf\{1\}\\\{A=e,\\tilde\{A\}\\neq e\\\}\.Summing overeegives

‖q~−q‖1\\displaystyle\\\|\\tilde\{q\}\-q\\\|\_\{1\}=∑e=1E\|q~e−qe\|\\displaystyle=\\sum\_\{e=1\}^\{E\}\|\\tilde\{q\}\_\{e\}\-q\_\{e\}\|≤∑e=1Eℙ⁡\(A~=e,A≠e\)\\displaystyle\\leq\\sum\_\{e=1\}^\{E\}\\mathbb\{P\}\(\\tilde\{A\}=e,A\\neq e\)\+∑e=1Eℙ\(A=e,A~≠e\)\.\\displaystyle\+\\sum\_\{e=1\}^\{E\}\\mathbb\{P\}\(A=e,\\tilde\{A\}\\neq e\)\.Since exactly one value ofeesatisfiesA~=e\\tilde\{A\}=eand exactly one value satisfiesA=eA=e,

∑e=1Eℙ⁡\(A~=e,A≠e\)=ℙ⁡\(A~≠A\),\\sum\_\{e=1\}^\{E\}\\mathbb\{P\}\(\\tilde\{A\}=e,A\\neq e\)=\\mathbb\{P\}\(\\tilde\{A\}\\neq A\),and

∑e=1Eℙ⁡\(A=e,A~≠e\)=ℙ⁡\(A≠A~\)\.\\sum\_\{e=1\}^\{E\}\\mathbb\{P\}\(A=e,\\tilde\{A\}\\neq e\)=\\mathbb\{P\}\(A\\neq\\tilde\{A\}\)\.Consequently,

‖q~−q‖1\\displaystyle\\\|\\tilde\{q\}\-q\\\|\_\{1\}≤2​ℙ​\(A~≠A\)\\displaystyle\\leq 2\\mathbb\{P\}\(\\tilde\{A\}\\neq A\)≤4​κ​r​ΔC\.\\displaystyle\\leq 4\\kappa r\\Delta\_\{C\}\.The theorem follows\. ∎

### B\.10Voronoi\-aware expert estimator Error

###### Definition 5\(Voronoi cell radius and gradient field\)\.

LetZi=ϕ⁡\(xi\)Z\_\{i\}=\\phi\(x\_\{i\}\)andAi=aC​\(xi\)A\_\{i\}=a\_\{C\}\(x\_\{i\}\)\. For each expertee, define

ℐe=\{i:Ai=e\},ne=\|ℐe\|,qe=ne/n\.\\mathcal\{I\}\_\{e\}=\\\{i:A\_\{i\}=e\\\},\\qquad n\_\{e\}=\|\\mathcal\{I\}\_\{e\}\|,\\qquad q\_\{e\}=n\_\{e\}/n\.Lethi=clip\(giexp,Ce\)∈ℝdh\_\{i\}=\\clip\(g\_\{i\}^\{\\exp\},C\_\{e\}\)\\in\\mathbb\{R\}^\{d\}be the clipped expert gradient, and define

μe=1ne​∑i∈ℐehi,\\mu\_\{e\}=\\frac\{1\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}h\_\{i\},Σe=1ne​∑i∈ℐe\(hi−μe\)​\(hi−μe\)⊤\.\\Sigma\_\{e\}=\\frac\{1\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\(h\_\{i\}\-\\mu\_\{e\}\)\(h\_\{i\}\-\\mu\_\{e\}\)^\{\\top\}\.Assume there exists anLgL\_\{g\}\-Lipschitz fieldψ:ℝm→ℝd\\psi:\\mathbb\{R\}^\{m\}\\to\\mathbb\{R\}^\{d\}such thathi=ψ⁡\(Zi\)h\_\{i\}=\\psi\(Z\_\{i\}\)\. Define

νe=ψ⁡\(ce\),re2=1ne​∑i∈ℐe‖Zi−ce‖22\.\\nu\_\{e\}=\\psi\(c\_\{e\}\),\\qquad r\_\{e\}^\{2\}=\\frac\{1\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\\|Z\_\{i\}\-c\_\{e\}\\\|\_\{2\}^\{2\}\.

###### Definition 6\(Public\-denominator expert estimator\)\.

Under Poisson subsamplingSi∼Bernoulli⁡\(p\)S\_\{i\}\\sim\{\\rm Bernoulli\}\(p\)withB=p​nB=pn, define

g¯e=EB​\(∑i∈ℐeSi​hi\+ξe\),\\bar\{g\}\_\{e\}=\\frac\{E\}\{B\}\\left\(\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}S\_\{i\}h\_\{i\}\+\\xi\_\{e\}\\right\),where

ξe∼𝒩⁡\(0,σe2​Ce2​Id\)\.\\xi\_\{e\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}I\_\{d\}\)\.

###### Theorem 6\(Voronoi\-aware expert estimation error\)\.

For each expertee,

𝔼​‖g¯e−νe‖22≤Vesamp\+VeDP\+2​\(E​qe−1\)2​‖μe‖22\+2​Lg2​re2,\\mathbb\{E\}\\\|\\bar\{g\}\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}\\leq V\_\{e\}^\{\\rm samp\}\+V\_\{e\}^\{\\rm DP\}\+2\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+2L\_\{g\}^\{2\}r\_\{e\}^\{2\},where

Vesamp=E2​p​\(1−p\)​neB2​\(‖μe‖22\+tr⁡Σe\),V\_\{e\}^\{\\rm samp\}=\\frac\{E^\{2\}p\(1\-p\)n\_\{e\}\}\{B^\{2\}\}\\left\(\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+\\tr\\Sigma\_\{e\}\\right\),and

VeDP=E2​σe2​Ce2​dB2\.V\_\{e\}^\{\\rm DP\}=\\frac\{E^\{2\}\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}d\}\{B^\{2\}\}\.Consequently, ifG¯2=maxe⁡‖μe‖22\\bar\{G\}^\{2\}=\\max\_\{e\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}, then

1E​∑e=1E𝔼​‖g¯e−νe‖22\\displaystyle\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\\mathbb\{E\}\\\|\\bar\{g\}\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}≤1E​∑e=1E\(Vesamp\+VeDP\)\\displaystyle\\leq\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\\left\(V\_\{e\}^\{\\rm samp\}\+V\_\{e\}^\{\\rm DP\}\\right\)\+2G¯2χ2\(q∥u\)\+2​Lg2E∑e=1Ere2\.\\displaystyle\+2\\bar\{G\}^\{2\}\\chi^\{2\}\(q\\\|u\)\+\\frac\{2L\_\{g\}^\{2\}\}\{E\}\\sum\_\{e=1\}^\{E\}r\_\{e\}^\{2\}\.

###### Proof\.

Fix an expertee\. All expectations below are taken over the Poisson subsampling variables\{Si\}\\\{S\_\{i\}\\\}and the Gaussian noiseξe\\xi\_\{e\}, conditional on the dataset\. By Definition 6,

g¯e=EB​∑i∈ℐeSi​hi\+EB​ξe\.\\bar\{g\}\_\{e\}=\\frac\{E\}\{B\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}S\_\{i\}h\_\{i\}\+\\frac\{E\}\{B\}\\xi\_\{e\}\.SinceSi∼Bernoulli⁡\(p\)S\_\{i\}\\sim\{\\rm Bernoulli\}\(p\)and𝔼⁡\[ξe\]=0\\mathbb\{E\}\[\\xi\_\{e\}\]=0,

𝔼⁡\[g¯e\]\\displaystyle\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]=EB​∑i∈ℐe𝔼⁡\[Si\]​hi\\displaystyle=\\frac\{E\}\{B\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\mathbb\{E\}\[S\_\{i\}\]h\_\{i\}=E​pB​∑i∈ℐehi\.\\displaystyle=\\frac\{Ep\}\{B\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}h\_\{i\}\.UsingB=p​nB=pnand∑i∈ℐehi=ne​μe\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}h\_\{i\}=n\_\{e\}\\mu\_\{e\}, we obtain

𝔼⁡\[g¯e\]=E​nen​μe=E​qe​μe\.\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]=\\frac\{En\_\{e\}\}\{n\}\\mu\_\{e\}=Eq\_\{e\}\\mu\_\{e\}\.Hence the bias relative to the cell mean is

𝔼⁡\[g¯e\]−μe=\(E​qe−1\)​μe\.\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]\-\\mu\_\{e\}=\(Eq\_\{e\}\-1\)\\mu\_\{e\}\.
We next compute the variance\. Independence of the Bernoulli variables gives

Var⁡\(EB​∑i∈ℐeSi​hi\)\\displaystyle\{\\rm Var\}\\left\(\\frac\{E\}\{B\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}S\_\{i\}h\_\{i\}\\right\)=E2B2​∑i∈ℐeVar⁡\(Si​hi\)\.\\displaystyle=\\frac\{E^\{2\}\}\{B^\{2\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\{\\rm Var\}\(S\_\{i\}h\_\{i\}\)\.Sincehih\_\{i\}is fixed conditional on the dataset,

Var⁡\(Si​hi\)=p⁡\(1−p\)​hi​hi⊤\.\{\\rm Var\}\(S\_\{i\}h\_\{i\}\)=p\(1\-p\)h\_\{i\}h\_\{i\}^\{\\top\}\.Therefore,

tr​Var​\(EB​∑i∈ℐeSi​hi\)\\displaystyle\{\\rm tr\}\\,\{\\rm Var\}\\left\(\\frac\{E\}\{B\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}S\_\{i\}h\_\{i\}\\right\)=E2​p​\(1−p\)B2​∑i∈ℐe‖hi‖22\.\\displaystyle=\\frac\{E^\{2\}p\(1\-p\)\}\{B^\{2\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\\|h\_\{i\}\\\|\_\{2\}^\{2\}\.By the empirical second\-moment identity,

1ne​∑i∈ℐe‖hi‖22=‖μe‖22\+tr​Σe\.\\frac\{1\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\\|h\_\{i\}\\\|\_\{2\}^\{2\}=\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+\{\\rm tr\}\\,\\Sigma\_\{e\}\.Thus

tr​Var​\(EB​∑i∈ℐeSi​hi\)=Vesamp\.\{\\rm tr\}\\,\{\\rm Var\}\\left\(\\frac\{E\}\{B\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}S\_\{i\}h\_\{i\}\\right\)=V\_\{e\}^\{\\rm samp\}\.The DP noise is independent of the sampling term, and

tr​Var​\(EB​ξe\)\\displaystyle\{\\rm tr\}\\,\{\\rm Var\}\\left\(\\frac\{E\}\{B\}\\xi\_\{e\}\\right\)=E2B2​tr​\(σe2​Ce2​Id\)\\displaystyle=\\frac\{E^\{2\}\}\{B^\{2\}\}\{\\rm tr\}\\left\(\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}I\_\{d\}\\right\)=E2​σe2​Ce2​dB2=VeDP\.\\displaystyle=\\frac\{E^\{2\}\\sigma\_\{e\}^\{2\}C\_\{e\}^\{2\}d\}\{B^\{2\}\}=V\_\{e\}^\{\\rm DP\}\.Therefore,

𝔼​‖g¯e−𝔼⁡\[g¯e\]‖22=Vesamp\+VeDP\.\\mathbb\{E\}\\left\\\|\\bar\{g\}\_\{e\}\-\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]\\right\\\|\_\{2\}^\{2\}=V\_\{e\}^\{\\rm samp\}\+V\_\{e\}^\{\\rm DP\}\.
Now decompose the error relative to the prototype gradientνe\\nu\_\{e\}:

𝔼​‖g¯e−νe‖22\\displaystyle\\mathbb\{E\}\\\|\\bar\{g\}\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}=𝔼​‖g¯e−𝔼⁡\[g¯e\]‖22\+‖𝔼⁡\[g¯e\]−νe‖22\.\\displaystyle=\\mathbb\{E\}\\left\\\|\\bar\{g\}\_\{e\}\-\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]\\right\\\|\_\{2\}^\{2\}\+\\left\\\|\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]\-\\nu\_\{e\}\\right\\\|\_\{2\}^\{2\}\.Using𝔼⁡\[g¯e\]=E​qe​μe\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]=Eq\_\{e\}\\mu\_\{e\},

𝔼⁡\[g¯e\]−νe=\(E​qe−1\)​μe\+\(μe−νe\)\.\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]\-\\nu\_\{e\}=\(Eq\_\{e\}\-1\)\\mu\_\{e\}\+\(\\mu\_\{e\}\-\\nu\_\{e\}\)\.Hence, by‖a\+b‖22≤2​‖a‖22\+2​‖b‖22\\\|a\+b\\\|\_\{2\}^\{2\}\\leq 2\\\|a\\\|\_\{2\}^\{2\}\+2\\\|b\\\|\_\{2\}^\{2\},

‖𝔼⁡\[g¯e\]−νe‖22\\displaystyle\\left\\\|\\mathbb\{E\}\[\\bar\{g\}\_\{e\}\]\-\\nu\_\{e\}\\right\\\|\_\{2\}^\{2\}≤2​\(E​qe−1\)2​‖μe‖22\+2​‖μe−νe‖22\.\\displaystyle\\leq 2\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+2\\\|\\mu\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}\.
It remains to control‖μe−νe‖22\\\|\\mu\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}\. Sincehi=ψ⁡\(Zi\)h\_\{i\}=\\psi\(Z\_\{i\}\)andνe=ψ⁡\(ce\)\\nu\_\{e\}=\\psi\(c\_\{e\}\),

μe−νe=1ne​∑i∈ℐe\(ψ⁡\(Zi\)−ψ⁡\(ce\)\)\.\\mu\_\{e\}\-\\nu\_\{e\}=\\frac\{1\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\left\(\\psi\(Z\_\{i\}\)\-\\psi\(c\_\{e\}\)\\right\)\.By Jensen’s inequality,

‖μe−νe‖22\\displaystyle\\\|\\mu\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}≤1ne​∑i∈ℐe‖ψ⁡\(Zi\)−ψ⁡\(ce\)‖22\.\\displaystyle\\leq\\frac\{1\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\\|\\psi\(Z\_\{i\}\)\-\\psi\(c\_\{e\}\)\\\|\_\{2\}^\{2\}\.Sinceψ\\psiisLgL\_\{g\}\-Lipschitz,

‖ψ⁡\(Zi\)−ψ⁡\(ce\)‖22≤Lg2​‖Zi−ce‖22\.\\\|\\psi\(Z\_\{i\}\)\-\\psi\(c\_\{e\}\)\\\|\_\{2\}^\{2\}\\leq L\_\{g\}^\{2\}\\\|Z\_\{i\}\-c\_\{e\}\\\|\_\{2\}^\{2\}\.Therefore,

‖μe−νe‖22\\displaystyle\\\|\\mu\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}≤Lg2ne​∑i∈ℐe‖Zi−ce‖22\\displaystyle\\leq\\frac\{L\_\{g\}^\{2\}\}\{n\_\{e\}\}\\sum\_\{i\\in\\mathcal\{I\}\_\{e\}\}\\\|Z\_\{i\}\-c\_\{e\}\\\|\_\{2\}^\{2\}=Lg2​re2\.\\displaystyle=L\_\{g\}^\{2\}r\_\{e\}^\{2\}\.Combining the preceding bounds gives

𝔼​‖g¯e−νe‖22≤Vesamp\+VeDP\+2​\(E​qe−1\)2​‖μe‖22\+2​Lg2​re2\.\\mathbb\{E\}\\\|\\bar\{g\}\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}\\leq V\_\{e\}^\{\\rm samp\}\+V\_\{e\}^\{\\rm DP\}\+2\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\+2L\_\{g\}^\{2\}r\_\{e\}^\{2\}\.
We now prove the aggregate bound\. LetG¯2=maxe⁡‖μe‖22\\bar\{G\}^\{2\}=\\max\_\{e\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\. Then

1E​∑e=1E\(E​qe−1\)2​‖μe‖22\\displaystyle\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}≤G¯2E​∑e=1E\(E​qe−1\)2\.\\displaystyle\\leq\\frac\{\\bar\{G\}^\{2\}\}\{E\}\\sum\_\{e=1\}^\{E\}\(Eq\_\{e\}\-1\)^\{2\}\.Sinceue=1/Eu\_\{e\}=1/E,

χ2\(q∥u\)\\displaystyle\\chi^\{2\}\(q\\\|u\)=∑e=1E\(qe−ue\)2ue\\displaystyle=\\sum\_\{e=1\}^\{E\}\\frac\{\(q\_\{e\}\-u\_\{e\}\)^\{2\}\}\{u\_\{e\}\}=E​∑e=1E\(qe−1E\)2\.\\displaystyle=E\\sum\_\{e=1\}^\{E\}\\left\(q\_\{e\}\-\\frac\{1\}\{E\}\\right\)^\{2\}\.Moreover,

1E​∑e=1E\(E​qe−1\)2\\displaystyle\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\(Eq\_\{e\}\-1\)^\{2\}=E​∑e=1E\(qe−1E\)2\\displaystyle=E\\sum\_\{e=1\}^\{E\}\\left\(q\_\{e\}\-\\frac\{1\}\{E\}\\right\)^\{2\}=χ2\(q∥u\)\.\\displaystyle=\\chi^\{2\}\(q\\\|u\)\.Therefore,

1E∑e=1E\(Eqe−1\)2∥μe∥22≤G¯2χ2\(q∥u\)\.\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}\\leq\\bar\{G\}^\{2\}\\chi^\{2\}\(q\\\|u\)\.Averaging the per\-expert inequality overeeyields

1E​∑e=1E𝔼​‖g¯e−νe‖22\\displaystyle\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\\mathbb\{E\}\\\|\\bar\{g\}\_\{e\}\-\\nu\_\{e\}\\\|\_\{2\}^\{2\}≤1E​∑e=1E\(Vesamp\+VeDP\)\\displaystyle\\leq\\frac\{1\}\{E\}\\sum\_\{e=1\}^\{E\}\\left\(V\_\{e\}^\{\\rm samp\}\+V\_\{e\}^\{\\rm DP\}\\right\)\+2G¯2χ2\(q∥u\)\+2​Lg2E∑e=1Ere2\.\\displaystyle\+2\\bar\{G\}^\{2\}\\chi^\{2\}\(q\\\|u\)\+\\frac\{2L\_\{g\}^\{2\}\}\{E\}\\sum\_\{e=1\}^\{E\}r\_\{e\}^\{2\}\.This proves the theorem\. ∎

### B\.11Scope of the Bias\-SNR Connection \(Remark 2 details\)

This appendix expands the two claims of Remark[2](https://arxiv.org/html/2609.05770#Thmremark2)\.

Non\-monotonicity\.βe2=\(E​qe−1\)2​‖μe‖22\\beta\_\{e\}^\{2\}=\(Eq\_\{e\}\-1\)^\{2\}\\\|\\mu\_\{e\}\\\|\_\{2\}^\{2\}is symmetric aroundqe=1/Eq\_\{e\}=1/E: an expert atqe=2/Eq\_\{e\}=2/Ehas the sameβe2\\beta\_\{e\}^\{2\}as one atqe=0q\_\{e\}=0, though these are opposite outcomes for training\.βe\\beta\_\{e\}measures deviation from a normalization convention, not training quality\.

Impossibility of an entropy bound on SNR\.No functionffsatisfies1/SNRe≤f⁡\(H⁡\(q\)\)1/\\mathrm\{SNR\}\_\{e\}\\leq f\(H\(q\)\)for all valid\(q,e\)\(q,e\)\. Sinceχ2\(q∥u\)∈\[0,E−1\]\\chi^\{2\}\(q\\\|u\)\\in\[0,E\{\-\}1\]for every validqq, takeqe=δq\_\{e\}=\\deltafor one expert andqe′=\(1−δ\)/\(E−1\)q\_\{e^\{\\prime\}\}=\(1\-\\delta\)/\(E\{\-\}1\)for the rest\. Asδ→0\+\\delta\\to 0^\{\+\},χ2\(q∥u\)→1/\(E−1\)\\chi^\{2\}\(q\\\|u\)\\to 1/\(E\{\-\}1\)\- near its own*minimum*\- whileH⁡\(q\)→log⁡\(E−1\)H\(q\)\\to\\log\(E\{\-\}1\), near\-maximal, andSNRe∝qe2→0\\mathrm\{SNR\}\_\{e\}\\propto q\_\{e\}^\{2\}\\to 0\. A starved expert’s contribution to the aggregate divergence is onlyO⁡\(1/E\)O\(1/E\), so it can coexist with near\-maximal entropy while its own SNR vanishes; no finiteffcan bound1/SNRe1/\\mathrm\{SNR\}\_\{e\}byH⁡\(q\)H\(q\)\.

## Appendix CExperiment Details

#### Code and reproducibility\.

All experiments use PyTorch[Paszke et al\. \(2019\)](https://arxiv.org/html/2609.05770#bib.bib10)with PRV accounting[Gopi et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib11)via Opacus[Yousefpour et al\. \(2021\)](https://arxiv.org/html/2609.05770#bib.bib12)v1\.6\.0 \(Python 3\.12, CUDA 12\.6, PyTorch 2\.9\)\.

#### Hardware and training\.

Each trial runs on a single B200 \(196 GB\)\. At each optimiser step, every recordi∈𝒟privi\\in\\mathcal\{D\}\_\{\\mathrm\{priv\}\}is independently included inℬtexp\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}with probabilityp=B/np=B/n, giving an*expected*batch size ofB=p​nB=pn\. Because the realised\|ℬtexp\|\|\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\|may be large, we accumulate per\-record clipped gradients across memory\-efficient microbatches*within*a single optimiser step; no gradients are accumulated*across*steps\. The denominatorsBB\(shared stream\) andDe=B/ED\_\{e\}=B/E\(expert stream\) are fixed expected values independent of the realised\|ℬtexp\|\|\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}\|\. Privacy budgets are computed via theprvaccountant\([Gopi et al\., 2021](https://arxiv.org/html/2609.05770#bib.bib11)\)with Poisson rateppand add/remove adjacency\.

#### Hyperparameters and tuning\.

Primary hyperparameters are epochs, batch sizeBB, learning rateη\\eta, clipping thresholdCC, and noise multiplierσDP\\sigma\_\{\\mathrm\{DP\}\}\(set by the PRV accountant\)\. Each method\-including Baselines A, B, and C\-is tuned*independently*on SST\-2 using the same search grid \(Tab\.[4](https://arxiv.org/html/2609.05770#A3.T4)\) and the same budget ofNtotal=100N\_\{\\mathrm\{total\}\}=100trials; the best\-performing configuration per method is then frozen for MNLI, QNLI, and QQP\. SST\-2 results therefore reflect in\-task tuning performance, while the remaining three tasks measure cross\-task transfer under a fixed, task\-agnostic configuration\. Tab\.[4](https://arxiv.org/html/2609.05770#A3.T4)lists the search grid; selected values are bolded\.

Specifically, Baseline A treatsCCandσDP\\sigma\_\{\\mathrm\{DP\}\}as free variables over the full grid and does*not*inheritCsC\_\{s\}from the proposed method; the PRV accountant targets the full budgetε\\varepsilon\(i\.e\.ρ=1\\rho=1\) and solves forσDP\\sigma\_\{\\mathrm\{DP\}\}jointly withCCvia binary search, matching the accounting procedure of the proposed method\. Baselines B and C likewise search over all grid values ofη\\eta,CC,BB, and epochs independently\. The proposed method applies LoRA to the shared streamθsh\\theta\_\{\\mathrm\{sh\}\}and each expertθl⋆,e\\theta\_\{l^\{\\star\},e\}; the role\-aware mechanism \(ownership map, public denominatorDeD\_\{e\}, residual objective\) is independent of this parameterisation choice\.

HyperparameterSearch gridEpochs\{10,𝟐𝟎,30\}\\\{10,\\mathbf\{20\},30\\\}Batch sizeBB\{500,𝟏𝟎𝟎𝟎,2000,3000\}\\\{500,\\mathbf\{1000\},2000,3000\\\}Learning rateη\\eta\{10,𝟓,3,1,0\.5\}×10−4\\\{10,\\mathbf\{5\},3,1,0\.5\\\}\\times 10^\{\-4\}ClippingCC\{0\.1,1\.0,10\}\\\{0\.1,\\mathbf\{1\.0\},10\\\}Budget splitρ\\rho\{0\.7,0\.8,0\.9,0\.95\}\\\{0\.7,0\.8,\\mathbf\{0\.9\},0\.95\\\}Table 4:Hyperparameter search grid \(SST\-2\)\. Selected values bolded; configuration frozen for all other tasks\.
#### Clipping andδ\\delta\.

We use per\-recordℓ2\\ell\_\{2\}clippingclip⁡\(g,C\)=g⋅min⁡\(1,C/‖g‖2\)\\mathrm\{clip\}\(g,C\)=g\\cdot\\min\(1,C/\\\|g\\\|\_\{2\}\)and setδ=1/N1\.1\\delta=1/N^\{1\.1\}for each task\.

## Appendix DMore Expriments on Other Models

![Refer to caption](https://arxiv.org/html/2609.05770v1/images/deepseek.png)Figure 3:Validation accuracy on ScienceQA forDeepSeek\-VL2\-Tinyacrossε∈\{1,4,8\}\\varepsilon\\in\\\{1,4,8\\\}\. Single run perε\\varepsilon\.As a secondary check of generalization beyond text\-only encoder MoEs, we fine\-tuneDeepSeek\-VL2\-Tiny[Wu et al\. \(2024\)](https://arxiv.org/html/2609.05770#bib.bib41)on ScienceQA[Saikh et al\. \(2022\)](https://arxiv.org/html/2609.05770#bib.bib40), a vision\-language task\.DeepSeek\-VL2\-Tinyhas both always\-active shared experts and routed experts \(Sec\.[3](https://arxiv.org/html/2609.05770#S3)\); shared experts fold into the shared stream, and![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORis applied to the last routed\-expert layer as in the main experiments\. No disjoint, format\-matched public corpus was available for this setting, so we protect the last sparse layer by fixed default rather than via entropy\-based selection \(Sec\.[5\.4](https://arxiv.org/html/2609.05770#S5.SS4)\)\. Fig\.[3](https://arxiv.org/html/2609.05770#A4.F3)reports validation accuracy atε∈\{1,4,8\}\\varepsilon\\in\\\{1,4,8\\\}\. Atε=1\\varepsilon=1, Ours reaches86\.98%86\.98\\%vs\.85\.12%85\.12\\%for Monolithic DP\-Adam LoRA \(\+1\.86\+1\.86\), consistent with the high\-noise pattern on Switch and OLMoE; the gap closes withε\\varepsilon, tying at44\(87\.17%87\.17\\%\) and marginally exceeding non\-private LoRA at88\(87\.27%87\.27\\%vs\.87\.22%87\.22\\%\) \- a single\-run artifact \(Reporting, Sec\.[8\.1](https://arxiv.org/html/2609.05770#S8.SS1)\) rather than evidence that private training exceeds its non\-private upper bound\. That gains persist under a fixed\-layer fallback, without entropy\-based selection, suggests![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR’s core mechanism does not strictly depend on that selection step \(Sec\.[5\.1](https://arxiv.org/html/2609.05770#S5.SS1)\)\.

## Appendix EAblation Study

### E\.1Component Ablation

Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)attributesOurs’s overall gains to the update rule as a whole, but does not show which of the four components introduced in Sec\.[5](https://arxiv.org/html/2609.05770#S5)\- expert\-specific clipping, the public denominatorDe=B/ED\_\{e\}\{=\}B/E, the alternating shared/expert schedule, and the residual objective \- is doing the work\. To isolate each one’s contribution, we ablateOursonSwitch\-base\-8by removing or replacing exactly one component at a time while holding the entropy\-selected sparse layer fixed:*w/De=BD\_\{e\}\{=\}B*replaces the public denominator with the realized\-count\-free but naive full\-batch denominator;*w/ full\-model clip*replaces separate shared/expert clipping with a single global boundCC;*w/o alternating*updates shared and expert parameters jointly in one step rather than in alternation \(Sec\.[5\.4](https://arxiv.org/html/2609.05770#S5.SS4)\); and*w/o residual obj\.*drops the residual weightingwiw\_\{i\}\(Eq\. \([8](https://arxiv.org/html/2609.05770#S5.E8)\)\), training the expert stream on the raw classification loss instead\. Tab\.[5](https://arxiv.org/html/2609.05770#A5.T5)reports this ablation atε≈8\\varepsilon\\approx 8andε=1\\varepsilon=1side by side, so that each component’s contribution can be compared directly across privacy budgets\.

The*alternating schedule*is the largest contributor: removing it costs up to1\.971\.97pts atε≈8\\varepsilon\\approx 8\(QNLI\) and2\.092\.09pts atε=1\\varepsilon=1\(QQP\), and on QNLI/QQP atε≈8\\varepsilon\\approx 8the no\-alternating variant falls*below*the matched\-scope global baseline \(Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)\), showing that role\-aware clipping and normalization alone are not sufficient without the two\-stream schedule\. The public denominatorDe=B/ED\_\{e\}=B/Econtributes0\.050\.05\-0\.980\.98pts, with the largest single gain on QNLI; consistent with Corollary[1](https://arxiv.org/html/2609.05770#Thmcorollary1), this gain is proportionally larger relative to the other components atε=1\\varepsilon=1, where the bias fromDe=BD\_\{e\}=Bunder high noise is more damaging\. Expert\-specific clipping contributes0\.060\.06\-0\.620\.62pts, and the residual objective contributes0\.040\.04\-0\.660\.66pts, smallest on SST\-2, where the shared stream alone suffices for sentiment classification\. Relative toε≈8\\varepsilon\\approx 8, component contributions grow atε=1\\varepsilon=1on SST\-2, MNLI, and QQP but shrink on QNLI: the high\-noise amplification suggested by Corollary[1](https://arxiv.org/html/2609.05770#Thmcorollary1)is task\-dependent, not uniform\. Together, all components yield0\.230\.23\-3\.243\.24pts over Monolithic DP\-Adam LoRA \(Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)\)\.

ε≈8\\varepsilon\\approx 8ε=1\\varepsilon=1MethodSST\-2MNLIQNLIQQPSST\-2MNLIQNLIQQPMonolithic DP\-Adam LoRA93\.9293\.9279\.3679\.3682\.4882\.4882\.6882\.6890\.9490\.9476\.9176\.9180\.3880\.3881\.1981\.19![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR\(full\)94\.15\\mathbf\{94\.15\}81\.32\\mathbf\{81\.32\}85\.26\\mathbf\{85\.26\}85\.92\\mathbf\{85\.92\}92\.90\\mathbf\{92\.90\}78\.42\\mathbf\{78\.42\}83\.30\\mathbf\{83\.30\}84\.08\\mathbf\{84\.08\}w/De=BD\_\{e\}=B94\.1094\.1080\.9680\.9684\.2884\.2885\.5585\.5592\.8392\.8377\.9877\.9882\.7982\.7983\.5583\.55w/ full\-model clip94\.0994\.0981\.2581\.2584\.6484\.6485\.8485\.8492\.8292\.8278\.3478\.3483\.2283\.2283\.9983\.99w/o alternating94\.0294\.0280\.2180\.2183\.2983\.2984\.0984\.0992\.7592\.7577\.1577\.1581\.6881\.6881\.9981\.99w/o residual obj\.94\.0794\.0781\.2281\.2284\.6084\.6085\.8285\.8292\.8692\.8678\.3078\.3083\.1883\.1883\.9683\.96Table 5:Component ablation onSwitch\-base\-8atε≈8\\varepsilon\\approx 8andε=1\\varepsilon=1\. Bold: full![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR\. Each row removes or replaces one component; all variants use the entropy\-selected sparse layer\.
### E\.2Matched\-Scope Diagnostics

The gains reported in Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)compare![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORagainst Monolithic DP\-Adam and Monolithic DP\-Adam LoRA, both of which differ fromOursalong*two*axes at once: they use a different update rule \(one clipping bound, one noise scale, one denominator, applied to the full model\)*and*a different training scope \(all trainable parameters, rather than the shared stream plus one selected expert layer\)\. This leaves an open question: how much of Ours’s advantage comes from the role\-aware update rule itself \- separate clipping, the public denominatorDe=B/ED\_\{e\}\{=\}B/E, and the alternating schedule \- versus simply from training a smaller, more targeted set of parameters under a matched privacy budget? To isolate this, we construct three matched\-scope baselines that fix Ours’s layer, total privacy budget, and step count, and vary*only*the update rule applied within that scope: Baseline A trains the same shared and expert parameters as Ours but with one global clipping bound, one noise scale, and one denominator \(i\.e\. monolithic DP applied to Ours’s reduced scope\); Baseline B trains only the shared stream, with the selected expert layer frozen; and Baseline C trains only the selected expert layer, with the shared stream frozen\. Comparing Ours against these three, rather than against the full\-scope monolithic baselines alone, lets us attribute the accuracy gains to the update rule specifically, rather than to scope or parameter count\.

Tab\.[6](https://arxiv.org/html/2609.05770#A5.T6)reports Baselines A\-C onSwitch\-base\-8\(same layer, scope, budget, and steps as Ours; update rule only varies; Sec\.[8\.1](https://arxiv.org/html/2609.05770#S8.SS1)\) alongside Monolithic DP\-Adam LoRA and Ours \(entropy\-selected\) for reference, across all three privacy budgets\. Baseline A \(global DP\) isolates whether matching training scope to Ours alone explains the gains; it trails Ours by0\.360\.36–3\.853\.85points on MNLI, QNLI, QQP atε=8\\varepsilon=8, and by comparable margins atε∈\{1,4\}\\varepsilon\\in\\\{1,4\\\}\- confirming the remaining gap is attributable to role\-aware clipping, normalization, and the alternating schedule \(Tab\.[5](https://arxiv.org/html/2609.05770#A5.T5)\), not scope alone\. Baseline B \(shared\-only, expert parameters frozen\) is competitive with Ours on SST\-2 but trails on QNLI and QQP at every budget, showing expert updates contribute beyond what the shared stream alone achieves\. Baseline C \(expert\-only, shared stream frozen\) collapses toward random accuracy on MNLI \(3232–36%36\\%\) and QNLI \(≈50%\\approx\\\!50\\%\) across all budgets, confirming the shared stream is necessary, not merely helpful\.

ε\\varepsilonMethodSST\-2MNLIQNLIQQP11Monolith\. DP\-Adam LoRA90\.9476\.9180\.3881\.19Baseline A \(global\)92\.5576\.8082\.9083\.10Baseline B \(shared\-only\)91\.8076\.4082\.5082\.90Baseline C \(expert\-only\)67\.7832\.7549\.7263\.66Ours, entropy\-sel\.92\.9078\.4283\.3084\.0844Monolith\. DP\-Adam LoRA92\.6678\.3381\.9582\.26Baseline A \(global\)93\.3277\.1084\.6884\.40Baseline B \(shared\-only\)93\.2077\.7084\.6884\.78Baseline C \(expert\-only\)68\.6934\.9149\.7564\.20Ours, entropy\-sel\.94\.0579\.0384\.7084\.7888Monolith\. DP\-Adam LoRA93\.9279\.3682\.4882\.68Baseline A \(global\)93\.4677\.4784\.9085\.20Baseline B \(shared\-only\)93\.5078\.1084\.8385\.50Baseline C \(expert\-only\)68\.8135\.5749\.7364\.47Ours, entropy\-sel\.94\.1581\.3285\.2685\.92Table 6:Matched\-scope diagnostics onSwitch\-base\-8: Baselines A–C share Ours’s layer, scope, budget, and step count, varying only the update rule \(Sec\.[8\.1](https://arxiv.org/html/2609.05770#S8.SS1)\)\. Monolithic DP\-Adam LoRA and Ours \(entropy\-selected\) shown for reference; full results including first\-/last\-sparse variants and non\-private upper bounds are in Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)\. Single run per cell\.
### E\.3Sensitivity to the Shared/Expert Budget Splitρ\\rho

![Refer to caption](https://arxiv.org/html/2609.05770v1/images/sst2_acc_vs_epsilon_ratio.png)Figure 4:SST\-2 accuracy \(%\) of![Refer to caption](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORonSwitch\-base\-8acrossε∈\{1,2,4,8\}\\varepsilon\\in\\\{1,2,4,8\\\}andρ∈\{0\.7,0\.8,0\.9,0\.95\}\\rho\\in\\\{0\.7,0\.8,0\.9,0\.95\\\}\.![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORintroduces one hyperparameter with no analogue in monolithic DP fine\-tuning:ρ∈\(0,1\)\\rho\\in\(0,1\), which splits the total privacy budgetε\\varepsilonbetween the shared stream \(ρ​ε\\rho\\varepsilon\) and the expert streams \(\(1−ρ\)​ε\(1\{\-\}\\rho\)\\varepsilon, Sec\.[6](https://arxiv.org/html/2609.05770#S6)\)\. Because the two streams have different sensitivities and noise requirements \- the expert stream’s noise variance carries an extraE2E^\{2\}factor from the normalized sensitivityE​Ce/BEC\_\{e\}/B\(Theorem[1](https://arxiv.org/html/2609.05770#Thmtheorem1)\) \- this split is not a cosmetic tuning knob: an unfavorable choice ofρ\\rhocould starve either stream of budget and erase the method’s advantage entirely\. Proposition[1](https://arxiv.org/html/2609.05770#Thmproposition1)gives qualitative guidance \(shared\-dominantρ\\rhounder the surrogate analysis of Appendix[H](https://arxiv.org/html/2609.05770#A8)\), but does not certify a specific value; we therefore validate the choice ofρ=0\.9\\rho=0\.9empirically here by sweepingρ\\rhoagainstε\\varepsilonand checking that accuracy is not unduly sensitive to this choice within a reasonable range\.

Fig\.[4](https://arxiv.org/html/2609.05770#A5.F4)shows SST\-2 accuracy acrossρ∈\{0\.7,0\.8,0\.9,0\.95\}\\rho\\in\\\{0\.7,0\.8,0\.9,0\.95\\\}andε∈\{1,2,4,8\}\\varepsilon\\in\\\{1,2,4,8\\\}\. Accuracy increases monotonically withε\\varepsilonforρ≤0\.9\\rho\\leq 0\.9; atρ=0\.95\\rho=0\.95accuracy instead drops fromε=4\\varepsilon=4toε=8\\varepsilon=8\(93\.46→92\.0993\.46\\to 92\.09\), an inversion that only appears at the most shared\-dominant split tested\. Sensitivity toρ\\rhois highest atε=1\\varepsilon=1, where the tightest overall budget makes misallocation between streams most costly, and shrinks at largerε\\varepsilon, where both streams have enough budget that the split matters less\. Settingρ=0\.95\\rho=0\.95hurts atε=8\\varepsilon=8\(92\.0992\.09vs\.94\.1594\.15forρ=0\.9\\rho=0\.9\), showing that under\-budgeting the expert stream degrades expert\-specific learning even when the total budget is generous\. We useρ=0\.9\\rho=0\.9as the default throughout, as it is uniformly at or near the best setting across all four values ofε\\varepsilontested\.

## Appendix FEmpirical Routing Entropy Analysis

We verify whether routing entropy, estimated on external public corpora, provides a reliable privacy\-free signal for layer selection\. Per Sec\.[7](https://arxiv.org/html/2609.05770#S7), higherH⁡\(q\(l\)\)H\(q^\{\(l\)\}\)implies smaller public\-denominator bias at layerll\.

#### Calibration corpora\.

For each task we use an external unlabeled corpus disjoint from all GLUE splits and from𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}: IMDb reviews for SST\-2[Maas et al\. \(2011\)](https://arxiv.org/html/2609.05770#bib.bib19), ANLI R2 pairs for MNLI[Nie et al\. \(2020\)](https://arxiv.org/html/2609.05770#bib.bib18), SQuAD contexts for QNLI[Rajpurkar et al\. \(2016\)](https://arxiv.org/html/2609.05770#bib.bib17)\. For QQP, we use PAWS\-Wiki sentence pairs from the PAWSlabeled\_finaltraining split, with labels ignored\. Exact string\-match verification finds zero pair overlap with GLUE QQP, QNLI, MNLI, SST\-2 train\.

#### Protocol\.

For each candidate layerl∈\{0,…,5\}l\\in\\\{0,\\dots,5\\\}\(encoder blocks\{1,3,5,7,9,11\}\\\{1,3,5,7,9,11\\\}\), we apply the frozenSwitch\-base\-8router to𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}and compute the empirical owner distribution

q\(l\)e=\|\{xi∈𝒞pub:al​\(xi\)=e\}\|\|𝒞pub\|\.q^\{\(l\)\}\_\{e\}=\\frac\{\|\\\{x\_\{i\}\\in\\mathcal\{C\}\_\{\\mathrm\{pub\}\}:a\_\{l\}\(x\_\{i\}\)=e\\\}\|\}\{\|\\mathcal\{C\}\_\{\\mathrm\{pub\}\}\|\}\.\(36\)Tab\.[7](https://arxiv.org/html/2609.05770#A6.T7)reportsH⁡\(q\)/log⁡EH\(q\)/\\log E,log⁡E−H⁡\(q\)\\log E\-H\(q\), andχ2\(q∥u\)\\chi^\{2\}\(q\\\|u\)for all 24 layer\-task combinations\.

#### Results\.

Routing balance varies substantially across tasks and layers\. On SST\-2, block 1 is by far the most balanced \(H/log⁡E=0\.972H/\\log E=0\.972,χ2=0\.118\\chi^\{2\}=0\.118\), while block 9 collapses \(H/log⁡E=0\.304H/\\log E=0\.304\)\. On MNLI, block 1 collapses \(χ2=4\.946\\chi^\{2\}=4\.946\) and block 7 is most balanced \(H/log⁡E=0\.682H/\\log E=0\.682,χ2=1\.262\\chi^\{2\}=1\.262\)\. On QNLI, block 11 leads \(H/log⁡E=0\.902H/\\log E=0\.902,χ2=0\.393\\chi^\{2\}=0\.393\)\. On QQP, blocks 5 and 7 collapse \(χ2\>2\.9\\chi^\{2\}\>2\.9\) while block 9 is most balanced \(H/log⁡E=0\.837H/\\log E=0\.837,χ2=0\.683\\chi^\{2\}=0\.683\)\.

Applyingl⋆=arg⁡maxl⁡H⁡\(q\(l\)\)l^\{\\star\}=\\arg\\max\_\{l\}H\(q^\{\(l\)\}\)selects block 1 \(SST\-2\), block 7 \(MNLI\), block 11 \(QNLI\), and block 9 \(QQP\) \- all unambiguous, with margins of at least0\.0290\.029nats over the runner\-up\. Within each task, entropy andχ2\\chi^\{2\}criteria agree, consistent with the Pinsker bound \(Eq\. \([17](https://arxiv.org/html/2609.05770#S7.E17)\)\)\. A fixed “last\-layer” heuristic would miss the optimal layer for SST\-2 \(by0\.3110\.311nats\) and QQP\. These selections were fixed before any private training; Sec\.[8\.2](https://arxiv.org/html/2609.05770#S8.SS2)tests them directly\.

TaskCalibration sourcennSparselayerEncoderblockH⁡\(q\)/log⁡EH\(q\)/\\\!\\log E↑\\uparrowlog⁡E−H⁡\(q\)\\log E\\\!\-\\\!H\(q\)↓\\downarrowχ2\(q∥u\)\\chi^\{2\}\(q\\\|u\)↓\\downarrowMNLIANLI R245,460010\.3071\.4424\.946MNLIANLI R245,460130\.6740\.6771\.580MNLIANLI R245,460250\.6410\.7461\.617MNLIANLI R245,460370\.6820\.6601\.262MNLIANLI R245,460490\.6430\.7431\.356MNLIANLI R245,4605110\.6460\.7361\.381QNLISQuAD context87,599010\.7610\.4981\.257QNLISQuAD context87,599130\.8810\.2460\.387QNLISQuAD context87,599250\.8730\.2640\.505QNLISQuAD context87,599370\.8520\.3070\.543QNLISQuAD context87,599490\.8070\.4010\.851QNLISQuAD context87,5995110\.9020\.2040\.393QQPPAWS\-Wiki49,289010\.7270\.5681\.481QQPPAWS\-Wiki49,289130\.8140\.3881\.008QQPPAWS\-Wiki49,289250\.5740\.8852\.922QQPPAWS\-Wiki49,289370\.4981\.0443\.379QQPPAWS\-Wiki49,289490\.8370\.3380\.683QQPPAWS\-Wiki49,2895110\.8250\.3640\.721SST\-2IMDb unsup\.50,000010\.9720\.0580\.118SST\-2IMDb unsup\.50,000130\.8280\.3580\.876SST\-2IMDb unsup\.50,000250\.7670\.4850\.972SST\-2IMDb unsup\.50,000370\.8760\.2590\.405SST\-2IMDb unsup\.50,000490\.3041\.4474\.774SST\-2IMDb unsup\.50,0005110\.6610\.7061\.763Table 7:Routing statistics for all six sparse encoder layers ofSwitch\-base\-8on external public corpora \(labels ignored; disjoint from GLUE splits and𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\)\.H⁡\(q\)/log⁡EH\(q\)/\\log E\(↑\\uparrow\), entropy deficitlog⁡E−H⁡\(q\)\\log E\-H\(q\)in nats \(↓\\downarrow\), andχ2\(q∥u\)\\chi^\{2\}\(q\\\|u\)\(↓\\downarrow\) forE=8E=8experts\.Bold: entropy\-selected layerl⋆l^\{\\star\}per task\. All selections are unambiguous; no privacy budget is consumed\.
#### Corpus\-size sensitivity\.

Fig\.[5](https://arxiv.org/html/2609.05770#A6.F5)and Fig\.[6](https://arxiv.org/html/2609.05770#A6.F6)show that the minimumnnfor stable layer selection correlates inversely with the entropy margin between the top two layers \(Tab\.[7](https://arxiv.org/html/2609.05770#A6.T7)\): SST\-2 stabilises atn=10n=10\(margin0\.1440\.144nats\), QNLI atn≥100n\\geq 100\(0\.0210\.021nats\), MNLI atn≥500n\\geq 500\(0\.0080\.008nats\), and QQP atn≥1,000n\\geq 1\{,\}000\(0\.0230\.023nats, three\-way near\-tie\)\. All four tasks select the correct layer atn=1,000n=1\{,\}000, under2\.2%2\.2\\%of the smallest calibration corpus used\. We recommendn≥1,000n\\geq 1\{,\}000as a conservative default; when the entropy margin at that size exceeds0\.050\.05nats, increasingnnfurther changes neither the ranking nor the selected block\.

![Refer to caption](https://arxiv.org/html/2609.05770v1/images/calibration_entropy_vs_n.png)Figure 5:Normalised routing entropyH⁡\(q\(l\)\)/log⁡EH\(q^\{\(l\)\}\)/\\log Eas a function of calibration examplesnn\(log scale\) for all six sparse encoder layers ofSwitch\-base\-8, across four tasks\. Each line is one candidate layer; the entropy\-selected layer at the full corpus size is marked in bold in Tab\.[7](https://arxiv.org/html/2609.05770#A6.T7)\. Entropy estimates stabilise byn≈100n\\approx 100–500500for most layers; the main exception is MNLI block 1 \(blue\), which is heavily collapsed and grows slowly even at largenn\. The frozen router is applied to unlabelled inputs only; no private data are accessed\.![Refer to caption](https://arxiv.org/html/2609.05770v1/images/calibration_best_block_vs_n.png)Figure 6:Public\-calibration vs\. GLUE\-development normalised routing entropyH⁡\(q\)/log⁡EH\(q\)/\\log Efor all 24 layer\-task combinations \(six sparse encoder layers×\\timesfour tasks\)\. Each point is labelled with its encoder block index\. The dashed line is the diagonaly=xy=x\. Transfer is closest for QNLI; MNLI shows a systematic gap, with all six layers above the diagonal\. The largest deviations are MNLI block 1 and SST\-2 blocks 7 and 9, none of which is selected by the entropy criterion; entropy\-selected layers sit closer to the diagonal than collapsed, non\-selected ones\.
#### Public\-to\-private entropy transfer\.

The entropy\-based layer\-selection rule \(Sec\.[5\.1](https://arxiv.org/html/2609.05770#S5.SS1)\) relies on routing statistics computed on a public corpus𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}to proxy the routing balance on the private training set𝒟priv\\mathcal\{D\}\_\{\\mathrm\{priv\}\}\. Fig\.[7](https://arxiv.org/html/2609.05770#A6.F7)checks this transfer empirically by plotting, for each of the 24 layer\-task combinations, the normalised routing entropyH⁡\(q\)/log⁡EH\(q\)/\\log Eestimated on𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}against the corresponding entropy estimated on the GLUE development set\. Transfer is close for QNLI and, more loosely, for SST\-2 and QQP; MNLI is the exception, with all six layers sitting visibly above the diagonal \(dev\-set entropy exceeding public\-corpus entropy throughout\), a systematic gap rather than an isolated outlier\. The largest single deviation is MNLI block 1 \(low public entropy but high dev\-set entropy\), followed by SST\-2 blocks 9 and 7; none of these three is selected byarg⁡maxl⁡H⁡\(q\(l\)\)\\arg\\max\_\{l\}H\(q^\{\(l\)\}\)for its task\. Entropy\-selected layers sit closer to the diagonal than collapsed, non\-selected ones across all four tasks \- e\.g\. SST\-2’s selected block 1 \(H/log⁡E=0\.972H/\\log E=0\.972publicly\) deviates far less than its own block 9\. On SST\-2, QNLI, and QQP, the public\-corpus argmax also matches the dev\-set ranking among top layers\. On MNLI, blocks 3 and 7 are separated by only0\.0080\.008nats on𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}\(Tab\.[7](https://arxiv.org/html/2609.05770#A6.T7)\), a margin narrow enough that the dev\-set ranking between these two layers may not agree with the public\-corpus one; Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)shows the publicly\-selected block 7 nonetheless yields the best downstream accuracy on MNLI\. We read this as evidence that public\-corpus entropy is a directionally reliable but imperfect proxy for private routing balance, consistent with Remark[2](https://arxiv.org/html/2609.05770#Thmremark2)’s broader caution that entropy is a validated heuristic rather than a certified optimum\.

![Refer to caption](https://arxiv.org/html/2609.05770v1/images/public_glue_dev_entropy_scatter.png)Figure 7:Public\-calibration vs\. GLUE\-development normalised routing entropyH⁡\(q\)/log⁡EH\(q\)/\\log Efor all 24 layer–task combinations \(six sparse encoder layers×\\timesfour tasks\)\. Each point is labelled with its encoder block index\. The dashed line is the diagonaly=xy=x\. Points near the diagonal indicate that entropy estimated on the public corpus transfers to the private training distribution\. The two largest off\-diagonal deviations \(SST\-2 blocks 7 and 9, blue circles\) are collapsed layers not selected by the entropy criterion; the entropy\-selected layer for each task lies near or above the diagonal\.

## Appendix GMechanism Diagnostics

The structural failures diagnosed in Sec\.[4](https://arxiv.org/html/2609.05770#S4)follow from the algebraic form of monolithic private optimization: a single clipping factor couples dense and sparse roles, the full\-batch denominator dilutes expert updates, and fixed privacy noise interacts unfavorably with routing imbalance\. These failures are properties of the mechanism definition and do not require empirical validation to exist\. Empirical diagnostics are nevertheless useful for measuring how strongly they manifest under the actual gradient distribution, routing frequencies, and noise realizations of a trained model\.

Tab\.[8](https://arxiv.org/html/2609.05770#A7.T8)reports three such diagnostics atε≈8\\varepsilon\\approx 8across GLUE tasks\. These results should be read as mechanism\-level evidence that is directionally informative but not statistically powered: diagnostic runs were completed for a limited number of configurations per task, and the table is intended to corroborate the algebraic arguments of Sec\.[4](https://arxiv.org/html/2609.05770#S4)rather than to constitute a statistically controlled comparison\.

TaskMethodMrole↓M\_\{\\mathrm\{role\}\}\\downarrowSNR¯↑\\overline\{\\mathrm\{SNR\}\}\\uparrowSNR​\-​CV↓\\mathrm\{SNR\}\\text\{\-\}\\mathrm\{CV\}\\downarrowSST\-2Monolithic DP\-Adam LoRA0\.3200\.3201\.08×10−41\.08\{\\times\}10^\{\-4\}2\.352\.35Monolithic DP\-Adam0\.5910\.5916\.38×10−66\.38\{\\times\}10^\{\-6\}1\.201\.20![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR0\.4010\.4011\.80×10−51\.80\{\\times\}10^\{\-5\}1\.691\.69MNLIMonolithic DP\-Adam LoRA0\.4940\.4949\.65×10−59\.65\{\\times\}10^\{\-5\}3\.543\.54Monolithic DP\-Adam0\.5600\.5603\.04×10−53\.04\{\\times\}10^\{\-5\}1\.241\.24![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR0\.3370\.3372\.25×10−42\.25\{\\times\}10^\{\-4\}1\.861\.86QNLIMonolithic DP\-Adam LoRA0\.7230\.7232\.64×10−52\.64\{\\times\}10^\{\-5\}1\.691\.69Monolithic DP\-Adam0\.6190\.6199\.58×10−69\.58\{\\times\}10^\{\-6\}1\.171\.17![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR0\.3970\.3977\.19×10−57\.19\{\\times\}10^\{\-5\}1\.621\.62QQPMonolithic DP\-Adam LoRA0\.7010\.7015\.98×10−55\.98\{\\times\}10^\{\-5\}2\.032\.03Monolithic DP\-Adam0\.5730\.5731\.81×10−51\.81\{\\times\}10^\{\-5\}1\.311\.31![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR0\.3240\.3241\.14×10−41\.14\{\\times\}10^\{\-4\}2\.532\.53Table 8:Mechanism diagnostics onSwitch\-base\-8atε≈8\\varepsilon\\approx 8\.MroleM\_\{\\mathrm\{role\}\}: total variation distance between gradient\-norm share and parameter\-count share across roles \(lower = more proportionate signal allocation\)\.SNR¯\\overline\{\\mathrm\{SNR\}\}: mean per\-expert signal\-to\-noise ratio \(higher = stronger expert update signal\)\.SNR\\mathrm\{SNR\}\-CV: coefficient of variation of per\-expert SNR \(lower = more uniform update quality across experts\)\.#### Role mismatch\.

For each top\-level rolerr\(shared parameters, router, classifier, and experts\), letsrnorms\_\{r\}^\{\\mathrm\{norm\}\}be the share of gradient norm assigned to that role andsrparams\_\{r\}^\{\\mathrm\{param\}\}be its share of trainable parameters\. We define

Mrole=12​∑r\|srnorm−srparam\|\.M\_\{\\mathrm\{role\}\}=\\frac\{1\}\{2\}\\sum\_\{r\}\\bigl\|s\_\{r\}^\{\\mathrm\{norm\}\}\-s\_\{r\}^\{\\mathrm\{param\}\}\\bigr\|\.\(37\)Lower values indicate that gradient signal is more proportionate to the trainable parameter mass across roles\. Across all tasks with available Monolithic DP\-Adam diagnostics,![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORsubstantially reducesMroleM\_\{\\mathrm\{role\}\}relative to Monolithic DP\-Adam: from0\.5910\.591to0\.4010\.401on SST\-2, from0\.5600\.560to0\.3370\.337on MNLI, and from0\.6190\.619to0\.3970\.397on QNLI\. Relative to Monolithic DP\-Adam LoRA, role mismatch is reduced on MNLI, QNLI, and QQP, though not on SST\-2\. The role\-aware decomposition therefore does not uniformly minimize this diagnostic, but it consistently avoids the pronounced role mismatch observed under full monolithic DP\-Adam\.

#### Mean expert SNR\.

The columnSNR¯\\overline\{\\mathrm\{SNR\}\}measures the average per\-expert signal\-to\-noise ratio, capturing the absolute strength of the expert update signal after accounting for privacy noise\. On MNLI, QNLI, and QQP,![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORachieves the largest mean expert SNR among the available configurations:2\.25×10−42\.25\{\\times\}10^\{\-4\}on MNLI,7\.19×10−57\.19\{\\times\}10^\{\-5\}on QNLI, and1\.14×10−41\.14\{\\times\}10^\{\-4\}on QQP\. On SST\-2, Monolithic DP\-Adam LoRA records the highest mean SNR, while our framework still improves over full Monolithic DP\-Adam\. This pattern is consistent with the intended effect of role\-aware expert updates: the method is most beneficial when monolithic optimization gives experts weak or diluted signal, whereas simpler tasks such as SST\-2 can already be served adequately by a strong LoRA baseline\.

#### Expert SNR variation\.

The SNR\-CV column reports the coefficient of variation of per\-expert SNR; lower values indicate more uniform update quality across experts\.![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORreduces SNR\-CV relative to Monolithic DP\-Adam LoRA on SST\-2, MNLI, and QNLI, but does not always achieve the lowest SNR\-CV overall: full Monolithic DP\-Adam has lower SNR\-CV on SST\-2, MNLI, and QNLI, and Monolithic DP\-Adam LoRA has lower SNR\-CV than our framework on QQP\. This does not contradict the utility results: a low SNR\-CV can arise from uniformly*weak*expert updates rather than uniformly useful ones\. Full Monolithic DP\-Adam exhibits low SNR\-CV on SST\-2 and QNLI precisely because its mean expert SNR is far lower than that of our framework\. We therefore interpret SNR\-CV jointly withSNR¯\\overline\{\\mathrm\{SNR\}\}: role\-aware training primarily improves absolute expert signal strength and role proportionality, while only partially reducing cross\-expert SNR variation\.

#### Summary\.

Tab\.[8](https://arxiv.org/html/2609.05770#A7.T8)provides directional support for the mechanism account developed in Sec\.[4](https://arxiv.org/html/2609.05770#S4)\.![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORis not uniformly best on every diagnostic, and the limited number of completed runs means that the reported values should be treated as indicative rather than conclusive\. Nevertheless, the method consistently reduces the large role mismatch of full monolithic DP\-Adam and substantially increases mean expert SNR on the harder GLUE tasks\. These trends are directionally consistent with the downstream accuracy gains in Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1): the advantage of role\-aware private training stems less from eliminating all expert imbalance and more from preventing expert updates from being suppressed or diluted by a dense monolithic private optimizer\.

#### Corpus\-Level Routing Diagnostics

Replacing the training batchℬtexp\\mathcal\{B\}\_\{t\}^\{\\mathrm\{exp\}\}with the full calibration corpus𝒞pub\\mathcal\{C\}\_\{\\mathrm\{pub\}\}\(Sec\.[5\.1](https://arxiv.org/html/2609.05770#S5.SS1)\) gives corpus\-level estimatesCV^l\\widehat\{\\mathrm\{CV\}\}\_\{l\}andH^l\\widehat\{H\}\_\{l\}, computed once from the frozen router before any private training begins\. Tab\.[9](https://arxiv.org/html/2609.05770#A7.T9)reports these for the entropy\-selected and most\-collapsed layers across all four tasks\. In every case the selected layer has lowerCV^l\\widehat\{\\mathrm\{CV\}\}\_\{l\}and higherH^l\\widehat\{H\}\_\{l\}than the collapsed layer, confirming that the two statistics are consistent and that high routing entropy coincides with more uniform load\.

TaskPublic corpusLayer \(block\)CV^l\\widehat\{\\mathrm\{CV\}\}\_\{l\}H^l/log⁡E\\widehat\{H\}\_\{l\}/\\log ESST\-2IMDb unsup\.selected \(1\)0\.3440\.972collapsed \(9\)2\.1850\.304MNLIANLI R2selected \(7\)1\.1230\.682collapsed \(1\)2\.2240\.307QNLISQuAD contextselected \(11\)0\.6270\.902collapsed \(1\)1\.1210\.761QQPPAWS\-Wikiselected \(9\)0\.7000\.837collapsed \(7\)1\.7630\.498Table 9:Corpus\-level routing diagnostics for the entropy\-selected and most\-collapsed sparse layers ofSwitch\-base\-8, computed on external public calibration corpora with labels ignored \(see Appendix[F](https://arxiv.org/html/2609.05770#A6)for corpus details\)\. HigherCV^l\\widehat\{\\mathrm\{CV\}\}\_\{l\}and lowerH^l\\widehat\{H\}\_\{l\}indicate stronger routing imbalance; the two statistics are consistent across all tasks\. Values are identical across private training methods because the router is frozen\.

## Appendix HBudget Allocation: Surrogate Analysis and Empirical Sensitivity

#### Surrogate analysis\.

The bias\-variance decomposition of Theorem[1](https://arxiv.org/html/2609.05770#Thmtheorem1)shows that privacy noise enters the shared and expert streams through separate variance terms,VsDP∝Cs2/\(ρ​ε\)2V^\{\\mathrm\{DP\}\}\_\{s\}\\propto C\_\{s\}^\{2\}/\(\\rho\\varepsilon\)^\{2\}andVeDP∝E2​Ce2/\(\(1−ρ\)​ε\)2V^\{\\mathrm\{DP\}\}\_\{e\}\\propto E^\{2\}C\_\{e\}^\{2\}/\(\(1\-\\rho\)\\varepsilon\)^\{2\}, whose relative magnitudes depend on the budget splitρ\\rho\. To obtain a tractable criterion for choosingρ\\rho, we weight these terms by the downstream sensitivity of the task loss to each stream\.

###### Proposition 1\(Budget allocation under shared\-dominant routing\)\.

Under balanced routing \(qe=1/Eq\_\{e\}=1/E\), define the single\-step surrogate

ℒ⁡\(ρ\)=λs​Cs2ρ2\+λe​E2​Ce2\(1−ρ\)2,ρ∈\(0,1\),\\mathcal\{L\}\(\\rho\)=\\frac\{\\lambda\_\{s\}C\_\{s\}^\{2\}\}\{\\rho^\{2\}\}\+\\frac\{\\lambda\_\{e\}E^\{2\}C\_\{e\}^\{2\}\}\{\(1\-\\rho\)^\{2\}\},\\qquad\\rho\\in\(0,1\),\(38\)whereλs,λe\>0\\lambda\_\{s\},\\lambda\_\{e\}\>0weight the task loss sensitivity to shared\- and expert\-stream perturbations\. The unique minimizer is

ρ∗=11\+\(λe​E2​Ce2λs​Cs2\)1/3\.\\rho^\{\*\}=\\frac\{1\}\{1\+\\left\(\\dfrac\{\\lambda\_\{e\}E^\{2\}C\_\{e\}^\{2\}\}\{\\lambda\_\{s\}C\_\{s\}^\{2\}\}\\right\)^\{1/3\}\}\.\(39\)

###### Proof\.

SettingA=λs​Cs2A=\\lambda\_\{s\}C\_\{s\}^\{2\}andBc=λe​E2​Ce2B\_\{c\}=\\lambda\_\{e\}E^\{2\}C\_\{e\}^\{2\}, differentiatingℒ⁡\(ρ\)=A/ρ2\+Bc/\(1−ρ\)2\\mathcal\{L\}\(\\rho\)=A/\\rho^\{2\}\+B\_\{c\}/\(1\-\\rho\)^\{2\}and equating to zero gives\(\(1−ρ\)/ρ\)3=Bc/A\(\(1\-\\rho\)/\\rho\)^\{3\}=B\_\{c\}/A, yielding Eq\. \([39](https://arxiv.org/html/2609.05770#A8.E39)\)\. Strict convexity \(d2​ℒ/d​ρ2=6​A/ρ4\+6​Bc/\(1−ρ\)4\>0d^\{2\}\\mathcal\{L\}/d\\rho^\{2\}=6A/\\rho^\{4\}\+6B\_\{c\}/\(1\-\\rho\)^\{4\}\>0\) confirms uniqueness\. ∎

#### Interpretation\.

Eq\. \([39](https://arxiv.org/html/2609.05770#A8.E39)\) gives two qualitative directions\. First,ρ∗\\rho^\{\*\}increases withλs/λe\\lambda\_\{s\}/\\lambda\_\{e\}: higher shared\-stream sensitivity concentrates the budget on the shared stream\. Second,ρ∗\\rho^\{\*\}*decreases*withEE: theE2E^\{2\}amplification of expert\-stream noise makes expert\-side budget more valuable per unit of noise reduction under the surrogate, pullingρ∗\\rho^\{\*\}toward00asEEgrows\. The shared\-dominant regimeρ∗→1\\rho^\{\*\}\\\!\\to\\\!1we adopt therefore does not come fromEE\- it comes from the sensitivity asymmetry: the empirical estimateλe/λs≲6\.3×10−3\\lambda\_\{e\}/\\lambda\_\{s\}\\lesssim 6\.3\\times 10^\{\-3\}\(Tab\.[10](https://arxiv.org/html/2609.05770#A8.T10)\) shows the task loss is far more sensitive to shared\-stream perturbations, which must outweigh theE2E^\{2\}factor forρ∗\\rho^\{\*\}to sit near11\. Proposition[1](https://arxiv.org/html/2609.05770#Thmproposition1)provides qualitative guidance, not a certified value ofρ∗\\rho^\{\*\}: the sensitivity ratioλe/λs\\lambda\_\{e\}/\\lambda\_\{s\}required to formally concludeρ∗≥0\.9\\rho^\{\*\}\\geq 0\.9\(Eq\. \(41\)\) is stringent and, per Tab\.[10](https://arxiv.org/html/2609.05770#A8.T10), does not hold for this model\. We therefore treatρ=0\.9\\rho=0\.9as an empirically validated shared\-dominant choice \(Sec\.[8](https://arxiv.org/html/2609.05770#S8)\), not a certified optimum\.

#### Empirical sensitivity estimate\.

For each roler∈\{shared,experts\}r\\in\\\{\\text\{shared\},\\,\\text\{experts\}\\\}, we inject Gaussian noiseξr∼𝒩⁡\(0,σr2​I\)\\xi\_\{r\}\\sim\\mathcal\{N\}\(0,\\,\\sigma\_\{r\}^\{2\}I\)matched to the DP noise magnitude atε≈8\\varepsilon\\approx 8, apply it toθr\\theta\_\{r\}, and recordΔ​ℒval\\Delta\\mathcal\{L\}\_\{\\mathrm\{val\}\}\. We define the one\-sided sensitivity estimate

λ\+=max⁡\(Δ​ℒval,0\)𝔼​‖ξr‖22,\\lambda\_\{\+\}=\\frac\{\\max\(\\Delta\\mathcal\{L\}\_\{\\mathrm\{val\}\},\\;0\)\}\{\\mathbb\{E\}\\\|\\xi\_\{r\}\\\|\_\{2\}^\{2\}\},\(40\)flooring non\-positive values at zero: a negative or near\-zeroΔ​ℒval\\Delta\\mathcal\{L\}\_\{\\mathrm\{val\}\}indicates that the role lies below the measurement noise floor\.

RoleParamsΔ​ℒval\\Delta\\mathcal\{L\}\_\{\\mathrm\{val\}\}λ\+\\lambda\_\{\+\}Shared9,438,7229\{,\}438\{,\}7222\.175×10−4\\phantom\{\-\}2\.175\\times 10^\{\-4\}2\.305×10−52\.305\\times 10^\{\-5\}Experts7,864,3207\{,\}864\{,\}320−1\.361×10−6\-1\.361\\times 10^\{\-6\}00Table 10:Noise\-injection sensitivity on SST\-2 atε≈8\\varepsilon\\approx 8\.λ\+\\lambda\_\{\+\}floors non\-positiveΔ​ℒval\\Delta\\mathcal\{L\}\_\{\\mathrm\{val\}\}at zero \(Eq\. \([40](https://arxiv.org/html/2609.05770#A8.E40)\)\)\. A non\-positive expert value indicates sensitivity below the measurement noise floor\.Tab\.[10](https://arxiv.org/html/2609.05770#A8.T10)shows clear shared\-stream sensitivity \(λs=2\.305×10−5\\lambda\_\{s\}=2\.305\\times 10^\{\-5\}\) and negligible expert sensitivity\. Using the magnitude of the signed expert estimate as a conservative upper bound,

λeλs≲1\.361×10−62\.175×10−4≈6\.3×10−3\.\\frac\{\\lambda\_\{e\}\}\{\\lambda\_\{s\}\}\\;\\lesssim\\;\\frac\{1\.361\\times 10^\{\-6\}\}\{2\.175\\times 10^\{\-4\}\}\\;\\approx\\;6\.3\\times 10^\{\-3\}\.\(41\)ForE=8E=8andCs=CeC\_\{s\}=C\_\{e\}, Eq\. \([41](https://arxiv.org/html/2609.05770#A8.E41)\) requiresλe/λs≤2\.1×10−5\\lambda\_\{e\}/\\lambda\_\{s\}\\leq 2\.1\\times 10^\{\-5\}\. The empirical bound exceeds this threshold, so Proposition[1](https://arxiv.org/html/2609.05770#Thmproposition1)does not formally certifyρ∗=0\.9\\rho^\{\*\}=0\.9for this model\. We therefore treatρ=0\.9\\rho=0\.9as a shared\-dominant initialization: motivated by the surrogate and consistent with the near\-zero measured expert sensitivity, but validated primarily by the empirical sweep in Sec\.[E\.3](https://arxiv.org/html/2609.05770#A5.SS3)\.

## Appendix IRuntime and Memory

Tab\.[11](https://arxiv.org/html/2609.05770#A9.T11)reports implementation\-level runtime and memory for fine\-tuningSwitch\-base\-8on SST\-2 atε≈8\\varepsilon\\approx 8, comparing![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTORagainst the two Monolithic DP baselines from Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)\. We include this because Role\-Aware changes training scope as well as the DP mechanism, so accuracy gains alone don’t show whether they come at a compute cost\. Global DP\-Adam trains the full model \(307\.85307\.85M params,100%100\\%\); Global DP\-Adam LoRA trains only adapters \(14\.1614\.16M,4\.60%4\.60\\%\); our framework trains the shared stream plus one expert layer \(17\.3017\.30M,5\.62%5\.62\\%\) \- comparable in scope to LoRA,∼18×\\sim\\\!18\\timessmaller than the full model\. Our framework’s gains over Monolithic DP\-Adam \(Tab\.[1](https://arxiv.org/html/2609.05770#S8.T1)\) are thus not explained by training more of the model\.

Peak memory is lowest for![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR\(112\.16112\.16GB\), below both LoRA \(130\.74130\.74GB\) and the full model \(165\.51165\.51GB\), plausibly because expert\-stream clipping operates on owner subsetsℬt,l,e\\mathcal\{B\}\_\{t,l,e\}rather than the full batch at once\. Per\-step time and total GPU\-hours are highest for the full model \(7\.237\.23s,2\.612\.61GPU\-hrs\) and lowest for LoRA \(4\.074\.07s,1\.471\.47\); our framework sits close to LoRA \(4\.514\.51s,1\.651\.65GPU\-hrs\) despite the extra forward pass for the residual weightwiw\_\{i\}and the alternating schedule \(Sec\.[5\.4](https://arxiv.org/html/2609.05770#S5.SS4)\) \- about37%37\\%cheaper in total compute than the full\-model baseline, at higher accuracy\.

MethodTrainable params% paramsPeak GBSec\./stepGPU\-hoursGlobal DP\-Adam LoRA14\.1614\.16M4\.60%4\.60\\%130\.74130\.744\.074\.071\.471\.47Global DP\-Adam307\.85307\.85M100\.00%100\.00\\%165\.51165\.517\.237\.232\.612\.61![[Uncaptioned image]](https://arxiv.org/html/2609.05770v1/images/RAPTOR_icon.png)RAPTOR17\.3017\.30M5\.62%5\.62\\%112\.16\\mathbf\{112\.16\}4\.514\.511\.651\.65Table 11:Implementation footprint of fine\-tuningSwitch\-base\-8on SST\-2 atε≈8\\varepsilon\\approx 8\. Raw implementation\-level costs, not a controlled same\-scope benchmark—scope differs by design\. Percentages relative to the full trainableSwitch\-base\-8model used by Global DP\-Adam\. Bold: lowest peak memory\.