When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry

# When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry
Source: [https://arxiv.org/html/2610.10616](https://arxiv.org/html/2610.10616)
Jiayang LiuAffiliation:Nanyang Technological UniversityLu SunAffiliation:Tohoku UniversityYuke HuAffiliation:King Abdullah University of Science and TechnologyZheng LiAffiliation:Shandong UniversityRui WenAffiliation:Institute of Science Tokyo

###### Abstract

Mixture\-of\-Experts \(MoE\) language models produce routing information during inference that may be logged or exposed for monitoring, debugging, load analysis, and safety auditing\. Unlike ordinary model outputs, this telemetry reveals a view of the model’s internal computation, raising a privacy question: can it reveal whether an example was used to fine\-tune the deployed model? We introduce a router\-augmented membership inference attack that combines conventional output\-side signals with aggregated routing features and applies a membership classifier learned from independently fine\-tuned shadow models to the target model\. Across three MoE architectures and three data domains, router telemetry consistently improves membership inference over a strong output\-signal ensemble, increasing TPR at 1% FPR by 2\.7–9\.4 percentage points across all nine settings\. The leakage persists across full fine\-tuning, frozen\-router training, LoRA, and instruction tuning, and remains observable with only discrete expert selections, restricted telemetry, or a single shadow model\. Mechanistic analysis further shows that the leakage does not require router\-specific memorization: fine\-tuning introduces membership information into hidden representations, while the router exposes a projection of this signal even when its parameters are frozen\. Perturbing the telemetry reduces this additional leakage only as its fidelity degrades\. Our results show that router telemetry can turn an operational signal into an additional privacy surface for fine\-tuned MoE models\.

## 1Introduction

Mixture\-of\-Experts \(MoE\) language models route each token through a small subset of expert networks selected by a learned router, producing routing decisions throughout inference\([Jiang et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib27);[Muennighoff et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib2)\)\. Pretrained MoE models are commonly adapted to downstream tasks through fine\-tuning on task\- or domain\-specific data and subsequently deployed for inference\. During deployment, the routing process also produces information about how tokens are assigned across experts, which may be retained or surfaced as telemetry for monitoring, debugging, load analysis, or safety auditing\([Hwang et al\., 2023](https://arxiv.org/html/2610.10616#bib.bib34);[Lv et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib28)\)\. Such telemetry exposes signals derived from intermediate representations that are not available from model outputs alone\. This raises a natural privacy question: can exposed router telemetry enable membership inference and reveal whether a candidate input was used to fine\-tune the deployed model? In this work, we investigate the membership\-inference risk associated with router telemetry in deployed MoE language models\.

Existing membership inference attacks on language models primarily rely on signals available from model outputs, including likelihoods, confidence statistics, and changes relative to a reference checkpoint\([Shi et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib6);[Zhang et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib7);[Wen et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib1);[Chen et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib12);[Ilic et al\., 2026b](https://arxiv.org/html/2610.10616#bib.bib13)\)\. However, membership information may be only weakly reflected in final outputs, limiting the effectiveness of such attacks, particularly at stringent low\-false\-positive operating points\([Das et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib29);[Duan et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib30)\)\. Recent studies have therefore explored membership signals encoded in intermediate representations, including hidden states, attention patterns, and gradient\-induced feature drift\([Makhija et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib19);[Zaree et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib20);[Ranjan et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib21)\)\. Exploiting these signals, however, typically requires direct access to internal model states, which is unavailable in standard deployment settings\. Router telemetry occupies an intermediate regime: it is derived from internal representations, yet may be exposed as an operational signal during inference\. It may therefore reveal membership information beyond that available from model outputs, without requiring direct access to hidden states\. Whether this creates a meaningful membership\-inference risk remains largely unexplored\.

Figure[1](https://arxiv.org/html/2610.10616#S1.F1)illustrates the deployment setting and router\-augmented membership inference pipeline considered in this work\. We consider an attacker who can query a fine\-tuned MoE model, observe its outputs and corresponding router telemetry, and access the public pretrained checkpoint as a reference, but has no access to the target model’s parameters, hidden states, or private fine\-tuning data\. For each candidate input, the attacker combines output\-side membership signals with a fixed\-dimensional router representation obtained by aggregating the exposed routing information across tokens, experts, and layers\. Since membership labels for the victim’s fine\-tuning data are unavailable, the attacker independently fine\-tunes shadow MoE models on auxiliary data from the same domain and uses their training and held\-out examples as labeled members and non\-members\. A membership classifier is then trained on the combined output\-side and router representations and applied directly to the target model to produce a membership score\.

![Refer to caption](https://arxiv.org/html/2610.10616v1/MoE-MIA.png)Figure 1:Overview of the threat model and router\-augmented membership inference pipeline\.Across three MoE architectures and three data domains, our experiments show that exposed router telemetry consistently increases membership leakage beyond output\-side signals, with particularly clear gains at the low\-FPR operating point\. The effect persists across different fine\-tuning regimes, including full fine\-tuning, frozen\-router training, LoRA, and instruction tuning\. We further examine how the leakage changes under restricted telemetry exposure and limited shadow\-model access, and find that the membership signal remains observable even with partial routing information or only a single shadow model\. Our mechanistic analysis shows that the leakage is not specific to router memorization: membership information is already encoded in fine\-tuned hidden representations, while the router exposes a projection of this signal even when its parameters are frozen\. Finally, we evaluate telemetry perturbations as potential defenses and find that reducing the additional leakage comes with a corresponding degradation in telemetry fidelity\.

Contributions\.Our contributions are as follows\.

- •Router\-telemetry membership leakage in deployed MoE models\.We identify and systematically study membership leakage through router telemetry exposed during MoE inference, revealing an additional privacy surface beyond conventional model outputs\.
- •Router\-augmented membership inference\.We develop an attack that combines output\-side membership signals with aggregated router telemetry and learns the membership classifier from independently fine\-tuned shadow models on same\-domain auxiliary data\.
- •Comprehensive empirical characterization\.Across three MoE architectures and three data domains, router telemetry consistently adds membership signal beyond output\-side attacks\. We further show that the leakage persists across fine\-tuning regimes and restricted attacker access, and reflects membership information encoded in hidden representations rather than router\-specific memorization\.

## 2Related Work

Membership inference in language models\.Membership inference attacks \(MIAs\) aim to determine whether an example was used to train or fine\-tune a model\([Shokri et al\., 2017](https://arxiv.org/html/2610.10616#bib.bib35)\)\. Most existing attacks derive membership signals from model outputs\. Reference\-free methods use likelihood\- or confidence\-based statistics, including Min\-K%\([Shi et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib6)\), Min\-K%\+\+\([Zhang et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib7)\), ReCaLL\([Xie et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib11)\), prediction confidence\([Song and Mittal, 2021](https://arxiv.org/html/2610.10616#bib.bib14)\), and in\-context optimization signals such as ICP\([Lu et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib16)\)\. Reference\-based methods instead calibrate target\-model outputs using auxiliary models or checkpoints\. LiRA\([Carlini et al\., 2022](https://arxiv.org/html/2610.10616#bib.bib10)\)estimates member and non\-member score distributions with shadow models,[Mireshghallah et al\. \(2022\)](https://arxiv.org/html/2610.10616#bib.bib17)compare candidate likelihoods under target and reference models, and SPV\([Fu et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib18)\)builds a reference model from self\-prompted data\. For fine\-tuned language models, the pretrained checkpoint provides a natural reference\. WBC\([Chen et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib12)\)compares localized token\-loss patterns between the fine\-tuned and pretrained models, EZ\([Ilic et al\., 2026b](https://arxiv.org/html/2610.10616#bib.bib13)\)measures probability shifts at prediction\-error positions, and LTsig\([Ilic et al\., 2026a](https://arxiv.org/html/2610.10616#bib.bib32)\)learns membership signatures from token\-level distributional differences between the two models\. Despite these advances, output\-based methods are limited to information reflected in the model’s predictions and may not capture all membership\-related signals encoded during fine\-tuning\.

A complementary line of work studies membership signals in internal representations\.[Makhija et al\. \(2026\)](https://arxiv.org/html/2610.10616#bib.bib19)exploit hidden\-state and attention features for membership inference, while[Zaree et al\. \(2026\)](https://arxiv.org/html/2610.10616#bib.bib20)use attention patterns to distinguish members from non\-members\.[Ranjan et al\. \(2026\)](https://arxiv.org/html/2610.10616#bib.bib21)further exploit gradient\-induced changes in internal representations and model outputs to infer membership\. These studies show that internal model states can contain membership information beyond that captured by output statistics\. However, exploiting such signals requires access to hidden states, attention patterns, or other internal model information, which is typically unavailable in deployment settings\.

Privacy risks of MoE routing\.Existing work has shown that routing behavior in MoE models can reveal sensitive information about inference\-time inputs\.[Yona et al\. \(2024\)](https://arxiv.org/html/2610.10616#bib.bib22)exploit tie\-handling behavior in Expert\-Choice Routing to recover victim prompts, while[Nuriyev and Kulp \(2026\)](https://arxiv.org/html/2610.10616#bib.bib23)show that expert selection traces can reveal substantial information about input tokens\.[Avinash \(2026\)](https://arxiv.org/html/2610.10616#bib.bib24)further demonstrates that routing signatures can predict the task category of an input, and[Ding et al\. \(2025\)](https://arxiv.org/html/2610.10616#bib.bib25)exploit hardware side channels induced by expert execution to infer information about prompts and responses\. These studies establish that routing\-related signals can expose properties of the input being processed, but do not investigate whether router telemetry can reveal information about the model’s fine\-tuning data\. More relevant to our setting, GateBleed\([Kalyanapu et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib26)\)demonstrates membership inference through a hardware timing side channel and, separately, shows that MoE expert choices can be inferred from the same hardware signal, which requires hardware\-level side\-channel access that is typically unavailable to external attackers\. In contrast, we study the membership\-inference risk introduced when router telemetry is exposed in deployed MoE systems\.

## 3Problem Statement

Mixture\-of\-Experts models\.A Mixture\-of\-Experts \(MoE\) language model replaces some or all dense feed\-forward blocks with a set ofEEexpert networks and a learned router\. For tokenttat MoE layerℓ\\ell, lethℓ,t∈ℝdh\_\{\\ell,t\}\\in\\mathbb\{R\}^\{d\}denote its hidden representation\. The router computes

zℓ,t=Wℓ​hℓ,t∈ℝE,pℓ,t=softmax⁡\(zℓ,t\),z\_\{\\ell,t\}=W\_\{\\ell\}h\_\{\\ell,t\}\\in\\mathbb\{R\}^\{E\},\\qquad p\_\{\\ell,t\}=\\operatorname\{softmax\}\(z\_\{\\ell,t\}\),whereWℓW\_\{\\ell\}is the router weight matrix andpℓ,tp\_\{\\ell,t\}gives the routing distribution over experts\. The router selects the top\-kkexperts,

Sℓ,t=TopK⁡\(pℓ,t,k\),S\_\{\\ell,t\}=\\operatorname\{TopK\}\(p\_\{\\ell,t\},k\),and routes the token to the corresponding expert networks\.

For an input sequencex=\(x1,…,xT\)x=\(x\_\{1\},\\ldots,x\_\{T\}\), we useR⁡\(x\)R\(x\)to denote the routing information produced during inference\. Depending on the exposure granularity,R⁡\(x\)R\(x\)may contain continuous routing scores or probabilities, discrete selected\-expert identities, or aggregated expert\-usage statistics derived from them\. We collectively refer to such information as*router telemetry*\.

Membership inference attacks\.LetMpreM\_\{\\mathrm\{pre\}\}denote a pretrained model, and letMftM\_\{\\mathrm\{ft\}\}denote the model obtained by fine\-tuningMpreM\_\{\\mathrm\{pre\}\}on a private datasetDftD\_\{\\mathrm\{ft\}\}\. Given a candidate examplexx, membership inference aims to determine whetherxxwas included inDftD\_\{\\mathrm\{ft\}\}\. We define the membership label as

m⁡\(x\)=\{1,x∈Dft,0,x∉Dft\.m\(x\)=\\begin\{cases\}1,&x\\in D\_\{\\mathrm\{ft\}\},\\\\ 0,&x\\notin D\_\{\\mathrm\{ft\}\}\.\\end\{cases\}
A membership inference attack produces a scores⁡\(x\)∈ℝs\(x\)\\in\\mathbb\{R\}, where a larger value indicates stronger evidence thatxxis a member\. Given a decision thresholdτ\\tau, the attack predicts

m^\(x\)=𝕀\[s\(x\)≥τ\]\.\\hat\{m\}\(x\)=\\mathbb\{I\}\\\!\\left\[s\(x\)\\geq\\tau\\right\]\.
Metrics\.For a decision thresholdτ\\tau, the true\-positive rate \(TPR\) and false\-positive rate \(FPR\) are defined as

TPR⁡\(τ\)=Pr⁡\[s⁡\(x\)≥τ∣m⁡\(x\)=1\],\\operatorname\{TPR\}\(\\tau\)=\\Pr\\\!\\left\[s\(x\)\\geq\\tau\\mid m\(x\)=1\\right\],and

FPR⁡\(τ\)=Pr⁡\[s⁡\(x\)≥τ∣m⁡\(x\)=0\]\.\\operatorname\{FPR\}\(\\tau\)=\\Pr\\\!\\left\[s\(x\)\\geq\\tau\\mid m\(x\)=0\\right\]\.
We report the area under the receiver operating characteristic curve \(*AUROC*\) as a threshold\-independent measure of membership separability\. Because membership inference is particularly relevant at low false\-positive rates, we additionally report

TPR⁡@​α%​FPR,\\operatorname\{TPR\}@\\alpha\\%\\operatorname\{FPR\},defined as the TPR achieved at an operating threshold corresponding to an FPR ofα%\\alpha\\%\. We useTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}as our primary low\-FPR metric\.

### 3\.1Threat Model

Victim’s behavior\.The victim fine\-tunes a pretrained MoE modelMpreM\_\{\\mathrm\{pre\}\}on a private datasetDftD\_\{\\mathrm\{ft\}\}and deploys the resulting modelMftM\_\{\\mathrm\{ft\}\}as an inference service\. During inference, the serving infrastructure may retain or expose per\-request router telemetryR⁡\(⋅\)R\(\\cdot\)for operational purposes such as monitoring, debugging, or expert\-load analysis\.

Attacker’s knowledge\.The attacker knows the publicly available pretrained checkpointMpreM\_\{\\mathrm\{pre\}\}from which the target model is fine\-tuned, as well as the domain of the downstream fine\-tuning data\. However, the attacker does not know the victim’s private fine\-tuning datasetDftD\_\{\\mathrm\{ft\}\}\. The attacker has no access to the parameters, intermediate hidden states, or gradients ofMftM\_\{\\mathrm\{ft\}\}\.

Attacker’s capability\.The attacker observes token\-level output probabilities \(or logits\) returned by the deployed model together with the corresponding router telemetry\. The attacker can also query the public checkpointMpreM\_\{\\mathrm\{pre\}\}to construct reference\-based attack signals\. In addition, the attacker can independently fine\-tune one or more copies ofMpreM\_\{\\mathrm\{pre\}\}on auxiliary data from the same domain and use these shadow models to train a membership classifier that is applied to the target model\.

Attacker’s goal\.Given a candidate examplexx, the attacker’s goal is to determine whether

x∈Dft,x\\in D\_\{\\mathrm\{ft\}\},i\.e\., whetherxxwas used to fine\-tune the deployed target model\.

## 4Router\-Augmented Membership Inference

Our attack augments conventional output\-side membership inference with router telemetry exposed by the target MoE model\. For each candidate example, we combine output\-derived membership signals with a fixed\-dimensional representation of its routing behavior\. The membership classifier is learned from independently fine\-tuned shadow models constructed using auxiliary data from the same domain, and is then applied to the target model\.

#### Output\-side membership signals\.

We construct an ensemble of existing output\-side membership signals to characterize the membership information available from conventional model outputs\. The ensemble combines both target\-model signals derived fromMftM\_\{\\mathrm\{ft\}\}and reference\-based signals obtained by comparingMftM\_\{\\mathrm\{ft\}\}with the public pretrained checkpointMpreM\_\{\\mathrm\{pre\}\}\. For an inputxx, we denote the resulting output\-side representation as

ϕout​\(x\)=Φout​\(Mft,Mpre,x\)\.\\phi\_\{\\mathrm\{out\}\}\(x\)=\\Phi\_\{\\mathrm\{out\}\}\(M\_\{\\mathrm\{ft\}\},M\_\{\\mathrm\{pre\}\},x\)\.

#### Router\-telemetry features\.

We summarize the per\-token routing information into a fixed\-dimensional representation for each input\. Specifically, we use the average routing probability assigned to each expert\. For experteeat MoE layerℓ\\ell, we define

rℓ,e​\(x\)=1T​∑t=1Tpℓ,t,e,r\_\{\\ell,e\}\(x\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}p\_\{\\ell,t,e\},wherepℓ,t,ep\_\{\\ell,t,e\}is the routing probability assigned to experteefor tokentt\. Concatenating these statistics across all MoE layers and experts gives the router representation

ϕR​\(x\)=\[rℓ,e​\(x\)\]ℓ,e\.\\phi\_\{R\}\(x\)=\\left\[r\_\{\\ell,e\}\(x\)\\right\]\_\{\\ell,e\}\.

#### Router\-augmented membership classifier\.

Since the victim’s private fine\-tuning data are unknown, the attacker constructs shadow models to learn membership patterns applicable to the target model\. For eachj∈\{1,…,K\}j\\in\\\{1,\\ldots,K\\\}, the attacker constructs a fine\-tuning setD~jin\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}and a held\-out setD~jout\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j\}from auxiliary data in the same domain, and fine\-tunesMpreM\_\{\\mathrm\{pre\}\}onD~jin\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}to obtain a shadow modelM~j\\widetilde\{M\}\_\{j\}\. For thejj\-th shadow model, we define

m~j​\(x\)=\{1,x∈D~jin,0,x∈D~jout\.\\widetilde\{m\}\_\{j\}\(x\)=\\begin\{cases\}1,&x\\in\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\},\\\\ 0,&x\\in\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j\}\.\\end\{cases\}
For each shadow modelM~j\\widetilde\{M\}\_\{j\}, the attacker extracts the corresponding output\-side and router representationsϕout\(j\)​\(x\)\\phi\_\{\\mathrm\{out\}\}^\{\(j\)\}\(x\)andϕR\(j\)​\(x\)\\phi\_\{R\}^\{\(j\)\}\(x\)\. The representations from allKKshadow models are pooled to form the attack\-training dataset

𝒟attack=⋃j=1K\{\(\[ϕout\(j\)​\(x\);ϕR\(j\)​\(x\)\],m~j​\(x\)\):x∈D~jin∪D~jout\}\.\\mathcal\{D\}\_\{\\mathrm\{attack\}\}=\\bigcup\_\{j=1\}^\{K\}\\left\\\{\\left\(\[\\phi\_\{\\mathrm\{out\}\}^\{\(j\)\}\(x\);\\phi\_\{R\}^\{\(j\)\}\(x\)\],\\widetilde\{m\}\_\{j\}\(x\)\\right\):x\\in\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}\\cup\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j\}\\right\\\}\.
The attacker trains a membership classifierggon𝒟attack\\mathcal\{D\}\_\{\\mathrm\{attack\}\}\. For a candidate examplexxqueried against the target model, the membership score is

s⁡\(x\)=g⁡\(\[ϕout​\(x\);ϕR​\(x\)\]\)\.s\(x\)=g\\\!\\left\(\[\\phi\_\{\\mathrm\{out\}\}\(x\);\\phi\_\{R\}\(x\)\]\\right\)\.The classifier learned from the shadow models is then applied directly to the independently fine\-tuned target modelMftM\_\{\\mathrm\{ft\}\}\.

## 5Experiments

In this section, we empirically evaluate the additional membership leakage introduced by exposed router telemetry\. We first examine whether router telemetry provides additional leakage beyond output\-side membership signals, and whether this effect persists across architectures, domains, adaptation regimes, and restricted attacker access\. We then examine whether the leakage is specific to the router or instead reflects membership information encoded more broadly in the hidden representations during fine\-tuning\. Finally, we evaluate telemetry perturbations as potential mitigations and examine the resulting privacy–utility trade\-off\.

### 5\.1Experimental Setup

Models and datasets\.We consider three open\-weight MoE models: OLMoE\-1B\-7B\-0924 \(OLMoE\-1B\-7B\)\([Muennighoff et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib2)\), Qwen1\.5\-MoE\-A2\.7B\-Chat \(Qwen1\.5\-MoE\)\([Team, 2024](https://arxiv.org/html/2610.10616#bib.bib3)\), and LLaMA\-MoE\-v2\-3\.8B\-2\.8\-sft \(LLaMA\-MoE\-v2\)\([Qu et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib4)\)\. We evaluate each model on three domains: arXiv, GitHub, and Wikipedia\. More details are provided in Appendix[A\.1](https://arxiv.org/html/2610.10616#A1.SS1)\.

Shadow\-model construction\.For each model–domain pair, we partition the corresponding domain corpus into four mutually disjoint subsets\. For each subsetjj, we construct a fine\-tuning setD~jin\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}and a held\-out setD~jout\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j\}\(with text\-only blind classifiers\([Das et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib29)\)remaining near chance in distinguishing the two sets\)\. We then fine\-tune the corresponding pretrained model onD~jin\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}to obtain an independently fine\-tuned modelM~j\\widetilde\{M\}\_\{j\}\. Full construction, validation, and fine\-tuning details are provided in Appendix[A\.2](https://arxiv.org/html/2610.10616#A1.SS2)\.

MIA baselines\.We consider a set of output\-side membership inference signals, including NLL\([Yeom et al\., 2018](https://arxiv.org/html/2610.10616#bib.bib5)\), Min\-K%\([Shi et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib6)\), Min\-K%\+\+\([Zhang et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib7)\), zlib\-normalized likelihood\([Carlini et al\., 2021](https://arxiv.org/html/2610.10616#bib.bib8)\), confidence\-derived logit margin\([Song and Mittal, 2021](https://arxiv.org/html/2610.10616#bib.bib14)\), log\-probability percentile statistics\([Shi et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib6);[Zhang and Wu, 2024](https://arxiv.org/html/2610.10616#bib.bib15)\), reference loss drop\([Watson et al\., 2022](https://arxiv.org/html/2610.10616#bib.bib9);[Carlini et al\., 2022](https://arxiv.org/html/2610.10616#bib.bib10)\), WBC\([Chen et al\., 2026](https://arxiv.org/html/2610.10616#bib.bib12)\), EZ\([Ilic et al\., 2026b](https://arxiv.org/html/2610.10616#bib.bib13)\), ReCaLL\([Xie et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib11)\), and LTsig\([Ilic et al\., 2026a](https://arxiv.org/html/2610.10616#bib.bib32)\)\. We evaluate these signals individually and also concatenate them into an ensemble output\-side representation, denoted byAoutA\_\{\\mathrm\{out\}\}, which is used by the membership classifier\. Detailed definitions of the individual signals are provided in Appendix[A\.3](https://arxiv.org/html/2610.10616#A1.SS3)\.

Evaluation protocol\.For each model–domain pair, we perform leave\-one\-out evaluation over the four independently fine\-tuned models: one is treated as the target and the remaining three as shadow models\. Both the output\-only and router\-augmented classifiers are trained on the pooled shadow\-model data and applied directly to the held\-out target, with the latter additionally incorporating router telemetry\. We repeat this procedure over all four held\-out targets and pool the resulting predictions for evaluation\. We use histogram gradient boosting as the membership classifier and continuous expert\-load telemetry as the main router signal\.

### 5\.2Router Telemetry Exposure Amplifies Membership Leakage

We first examine whether exposing router telemetry increases the membership inference risk of fine\-tuned MoE language models\. To this end, we evaluate router\-augmented membership inference across three MoE architectures and three data domains, covering all nine model–domain combinations\. As shown in Table[1](https://arxiv.org/html/2610.10616#S5.T1), augmenting the output\-signal ensembleAoutA\_\{\\mathrm\{out\}\}with router telemetry consistently improvesTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}in all nine settings, with absolute gains ranging from 0\.0267 to 0\.0937\. These results show that exposing router telemetry introduces additional membership leakage beyond that already captured by conventional output\-side signals, and that this effect generalizes across different MoE architectures and data domains\.

Router telemetry also improves AUROC across all nine settings, although the gains are smaller becauseAoutA\_\{\\mathrm\{out\}\}is already close to saturation in several cases\. In comparison, router telemetry yields more pronounced gains at the stricter low\-FPR operating point measured byTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}\. The same trend also holds for individual output\-side MIA baselines, with detailed per\-baseline results reported in Appendix[B](https://arxiv.org/html/2610.10616#A2)\.

Table 1:Membership inference across all combinations of three MoE architectures and three data domains\.Δ\\Deltadenotes the absolute improvement after incorporating router telemetry into the output\-signal ensembleAoutA\_\{\\mathrm\{out\}\}\.
### 5\.3Leakage Persists Across Fine\-Tuning Regimes

Table 2:Membership inference under different fine\-tuning regimes on OLMoE\-1B\-7B with arXiv, measured byTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}\.Δ\\Deltadenotes the improvement after incorporating router telemetry\.Having established that router telemetry provides additional membership signal beyond model outputs, we next ask whether this effect depends on the fine\-tuning procedure\. Using OLMoE\-1B\-7B on arXiv, we compare standard full fine\-tuning with frozen\-router fine\-tuning, LoRA adaptation\([Hu et al\., 2022](https://arxiv.org/html/2610.10616#bib.bib31)\), and instruction tuning\([Wei et al\., 2022](https://arxiv.org/html/2610.10616#bib.bib33)\)with response\-only loss\. As shown in Table[2](https://arxiv.org/html/2610.10616#S5.T2), incorporating router telemetry improvesTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}under all four regimes, with gains ranging from 0\.0305 under LoRA to 0\.0904 with a frozen router\.

The leakage therefore persists across substantially different adaptation settings\. In particular, freezing the router does not reduce the additional leakage, with a gain comparable to that under full fine\-tuning\. The effect also remains under LoRA and instruction tuning, showing that it is not specific to full\-parameter adaptation\. More detailed settings and results are provided in Appendix[C](https://arxiv.org/html/2610.10616#A3)\.

### 5\.4Leakage Under Restricted Telemetry and Shadow\-Model Access

The experimental results in Section[5\.2](https://arxiv.org/html/2610.10616#S5.SS2)assume access to the full continuous router\-telemetry representation\. We next examine whether the observed membership leakage remains exploitable under more restricted access conditions that may arise in practical deployment settings\. Specifically, we consider three settings: replacing continuous routing probabilities with discrete expert selections, restricting the amount of continuous telemetry exposed to the attacker, and reducing the number of shadow models available for attack training\.

Discrete expert\-selection telemetry\.The experimental results in Section[5\.2](https://arxiv.org/html/2610.10616#S5.SS2)use the continuous expert\-load representationϕR\\phi\_\{R\}, which retains the routing probabilities assigned to individual experts\. We first consider a more restricted representation that exposes only which experts are selected by the router\. For each layerℓ\\elland expertee, we define

rℓ,ecount\(x\)=1T∑t=1T𝕀\[e∈ℰℓ,t\(k\)\],r^\{\\mathrm\{count\}\}\_\{\\ell,e\}\(x\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbb\{I\}\\left\[e\\in\\mathcal\{E\}\_\{\\ell,t\}^\{\(k\)\}\\right\],\(1\)whereℰℓ,t\(k\)\\mathcal\{E\}\_\{\\ell,t\}^\{\(k\)\}denotes the set of experts selected for tokenttat layerℓ\\ell\. Flattening these layer–expert selection frequencies gives the discrete representationϕRcount​\(x\)\\phi\_\{R\}^\{\\mathrm\{count\}\}\(x\), which contains no continuous routing probabilities\.

As shown in Table[3](https://arxiv.org/html/2610.10616#S5.T3), augmentingAoutA\_\{\\mathrm\{out\}\}with the discrete expert\-selection representationϕRcount\\phi\_\{R\}^\{\\mathrm\{count\}\}improvesTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}in all nine model–domain settings, with gains ranging from 0\.0283 to 0\.1043\. These improvements are comparable in magnitude to those obtained with the continuous router representationϕR\\phi\_\{R\}, and in several settings the discrete representation achieves a similar or slightly larger gain\. This indicates that the additional membership leakage does not depend on access to precise routing probabilities, as discrete expert\-selection information alone is sufficient to expose a comparable membership signal\.

Table 3:Membership inference with discrete expert\-selection telemetry\.Δ⁡\(ϕRcount\)\\Delta\(\\phi\_\{R\}^\{\\mathrm\{count\}\}\)denotes the improvement inTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}obtained by incorporating the discrete expert\-selection representation, whileΔ⁡\(ϕR\)\\Delta\(\\phi\_\{R\}\)reports the corresponding improvement with the continuous expert\-load representation\.Restricted continuous telemetry\.We next ask how the leakage changes when only a subset of the continuous expert\-load telemetry is exposed\. Using OLMoE\-1B\-7B on arXiv, we retain only the top\-KKexperts ranked by document\-level routing load to restrict the amount of continuous router telemetry exposed to the attacker\.

Figure 2:Effect of restricted continuous router telemetry on OLMoE\-1B\-7B with arXiv\. The dashed line denotes the full continuous telemetry setting\.As shown in Figure[2](https://arxiv.org/html/2610.10616#S5.F2), even top\-1 telemetry provides an additionalΔ​TPR⁡@​1%​FPR\\Delta\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}of 0\.0175 overAoutA\_\{\\mathrm\{out\}\}\. The gain increases monotonically with top\-KK, reaching 0\.0564 at top\-K=8K=8and 0\.0641 at top\-K=32K=32, with diminishing improvements as more experts are exposed\. Thus, membership leakage remains observable even when only a small fraction of the continuous routing information is available\.

Limited shadow\-model access\.Finally, we consider limited shadow\-model access\. Using only a single shadow model still improvesTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}overAoutA\_\{\\mathrm\{out\}\}by 0\.0493, showing that the router\-induced membership signal remains exploitable even with minimal shadow\-model resources\. Additional results are reported in Appendix[D\.2](https://arxiv.org/html/2610.10616#A4.SS2)\.

### 5\.5Router Telemetry Exposes Membership Signals in Hidden Representations

Figure 3:Mechanistic analysis on OLMoE\-1B\-7B with arXiv\. Membership AUROC of matched\-rank random, router, and frozen\-router projections across fine\-tuning epochs\. The dashed line denotes chance\-level AUROC of 0\.5\.Having established that router telemetry amplifies membership leakage, we next investigate whether this leakage is specific to the router or instead reflects membership information encoded more broadly in the hidden representations during fine\-tuning\. Using OLMoE\-1B\-7B on arXiv, we compare the router projection with a matched\-rank random projection under full fine\-tuning and with the corresponding router projection under frozen\-router fine\-tuning to examine whether the observed membership signal is specific to the router subspace or depends on router\-parameter updates\.

As shown in Figure[3](https://arxiv.org/html/2610.10616#S5.F3), all three conditions are near the chance\-level AUROC of 0\.5 before fine\-tuning\. After one epoch, the router and random projections reach AUROCs of 0\.522 and 0\.515, respectively, increasing to 0\.661 and 0\.594 after two epochs\. The router projection therefore retains more membership information than a matched\-rank random subspace\. Freezing the router produces a similar trend, with the router projection reaching an AUROC of 0\.661 after two epochs under both full and frozen\-router fine\-tuning\.

Together, these results suggest that the observed leakage does not require router\-specific memorization\. Instead, fine\-tuning introduces membership\-related information into the hidden representations, and router telemetry exposes a projection of this signal during inference\.

### 5\.6Mitigating Router\-Telemetry Leakage Trades Off Utility

Having established that exposed router telemetry introduces additional membership leakage, we next examine whether this risk can be mitigated by perturbing the telemetry before it is exposed\. Using OLMoE\-1B\-7B on arXiv, we consider three post\-processing mechanisms applied to the continuous expert\-load representation: Gaussian noise, random feature dropout, and quantization\. These perturbations are applied after inference and do not modify the model or its inference procedure\. We measure the remaining leakage by the improvement inTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}overAoutA\_\{\\mathrm\{out\}\}, and telemetry utility by its Pearson correlation with the original expert loads\. We use correlation as the primary utility metric in the main analysis, while top\-1 expert agreement, which shows the same qualitative trend, is reported in Appendix[F](https://arxiv.org/html/2610.10616#A6)\.

As shown in Figure[4](https://arxiv.org/html/2610.10616#S5.F4), all three perturbation mechanisms reduce the additional membership leakage as the perturbation becomes stronger, while simultaneously degrading telemetry utility\. For Gaussian noise, theTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}gain decreases from 0\.0841 without perturbation to 0\.0563 atσ=1\\sigma=1and 0\.0159 atσ=4\\sigma=4, while telemetry correlation decreases from 1\.0 to 0\.924 and 0\.588, respectively\. Feature dropout produces a stronger reduction in both quantities, with the gain and correlation decreasing to 0\.0190 and 0\.153 atp=0\.9p=0\.9, and to 0\.0040 and 0\.047 atp=0\.99p=0\.99\. Similarly, quantization reduces the gain from 0\.0719 atB=32B=32to 0\.0121 atB=8B=8and 0\.0079 atB=4B=4, while correlation decreases from 0\.750 to 0\.584 and 0\.045\.

Overall, stronger telemetry perturbation can substantially suppress the additional membership leakage, but doing so also reduces the fidelity of the exposed telemetry\. In particular, the strongest suppression under dropout and quantization occurs only when telemetry correlation has already fallen close to zero\. Within the evaluated settings, we do not observe a configuration that suppresses the router\-induced leakage while preserving high\-fidelity telemetry\.

\(a\)Gaussian noise\.\(b\)Feature dropout\.\(c\)Quantization\.
Figure 4:Router\-telemetry perturbation on OLMoE\-1B\-7B with arXiv\. In each panel, the left y\-axis reports the improvement inTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}overAoutA\_\{\\mathrm\{out\}\}, and the right y\-axis reports telemetry utility measured by Pearson correlation with the original expert loads\. The x\-axis shows the corresponding perturbation parameter, with stronger perturbation given by largerσ\\sigmaorppand smallerBB\.

## 6Conclusion

We study membership\-inference risk introduced by router telemetry in fine\-tuned Mixture\-of\-Experts language models\. Our results show that routing information exposed during inference provides membership signal beyond that available from conventional output\-side attacks across multiple MoE architectures, data domains, and fine\-tuning regimes\. This additional leakage does not require access to precise routing probabilities: discrete expert selections, restricted top\-KKtelemetry, and even limited shadow\-model access remain sufficient to improve membership inference\. Our mechanistic analysis further shows that the effect is not explained by router\-specific memorization\. Instead, fine\-tuning induces membership\-related structure in hidden representations, and the router exposes a projection of this information even when its parameters are frozen during fine\-tuning\. Finally, post\-processing router telemetry can reduce the additional leakage, but stronger perturbations also degrade the fidelity of the telemetry itself\. Taken together, these findings show that router telemetry should not be treated as an innocuous operational signal: exposing routing information can reveal properties of a model’s private fine\-tuning data and therefore introduces an additional privacy consideration when deploying MoE language models\.

## AI Use Statement

We used generative AI tools to assist with polishing the writing of this manuscript, including improving grammar, clarity, and readability\. All AI\-assisted text was reviewed and revised by the authors\. Generative AI was not used to generate research ideas, experimental results, or scientific claims\. The authors take full responsibility for the final content of this work\.

## Ethics Statement

This work studies a privacy risk associated with exposing router telemetry from fine\-tuned Mixture\-of\-Experts language models\. As with other membership inference research, the techniques studied in this paper have potential dual\-use implications, since they could be used to infer whether particular examples were included in a model’s fine\-tuning data\. Our experiments are conducted in controlled settings using open\-weight models and constructed membership instances from arXiv, GitHub, and Wikipedia, and do not involve human subjects or access to private user data from deployed systems\. The purpose of this work is to characterize an underexplored privacy surface in MoE deployment and to inform safer telemetry\-exposure practices\. We also evaluate several telemetry\-perturbation mechanisms to quantify the trade\-off between reducing membership leakage and preserving the operational utility of the exposed telemetry\. We hope these findings encourage model providers to consider routing information as potentially privacy\-sensitive when designing logging, monitoring, and telemetry interfaces\.

## References

- Avinash \(2026\)M\. S\. R\. AvinashTask\-conditioned routing signatures in sparse mixture\-of\-experts transformers\.CoRRabs/2603\.11114\.External Links:[Link](https://doi.org/10.48550/arXiv.2603.11114),[Document](https://dx.doi.org/10.48550/ARXIV.2603.11114),2603\.11114Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p3.1)\.
- Carliniet al\.\(2022\)N\. Carlini, S\. Chien, M\. Nasr, S\. Song, A\. Terzis, and F\. TramèrMembership inference attacks from first principles\.In43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22\-26, 2022,pp\. 1897–1914\.External Links:[Link](https://doi.org/10.1109/SP46214.2022.9833649),[Document](https://dx.doi.org/10.1109/SP46214.2022.9833649)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Carliniet al\.\(2021\)N\. Carlini, F\. Tramèr, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. B\. Brown, D\. Song, Ú\. Erlingsson, A\. Oprea, and C\. RaffelExtracting training data from large language models\.In30th USENIX Security Symposium, USENIX Security 2021, August 11\-13, 2021,M\. D\. Bailey and R\. Greenstadt \(Eds\.\),pp\. 2633–2650\.External Links:[Link](https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting)Cited by:[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Chenet al\.\(2026\)Y\. Chen, Y\. Du, K\. Zhang, A\. Kundu, C\. Fleming, B\. Ribeiro, and N\. LiWindow\-based membership inference attacks against fine\-tuned large language models\.CoRRabs/2601\.02751\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.02751),[Document](https://dx.doi.org/10.48550/ARXIV.2601.02751),2601\.02751Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Daset al\.\(2025\)D\. Das, J\. Zhang, and F\. TrantèrBlind baselines beat membership inference attacks for foundation models\.In2025 IEEE Security and Privacy, SP 2025 \- Workshops, San Francisco, CA, USA, May 15, 2025,M\. Blanton, W\. Enck, and C\. Nita\-Rotaru \(Eds\.\),pp\. 118–125\.External Links:[Link](https://doi.org/10.1109/SPW67851.2025.00016),[Document](https://dx.doi.org/10.1109/SPW67851.2025.00016)Cited by:[§A\.1](https://arxiv.org/html/2610.10616#A1.SS1.p3.1),[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p2.1)\.
- Dinget al\.\(2025\)R\. Ding, T\. Xu, X\. Shen, A\. A\. Ding, and Y\. FeiMoEcho: exploiting side\-channel attacks to compromise user privacy in mixture\-of\-experts llms\.InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, CCS 2025, Taipei, Taiwan, October 13\-17, 2025,C\. Huang, J\. Chen, S\. Shieh, D\. Lie, and V\. Cortier \(Eds\.\),pp\. 2159–2173\.External Links:[Link](https://doi.org/10.1145/3719027.3765174),[Document](https://dx.doi.org/10.1145/3719027.3765174)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p3.1)\.
- Duanet al\.\(2024\)M\. Duan, A\. Suri, N\. Mireshghallah, S\. Min, W\. Shi, L\. Zettlemoyer, Y\. Tsvetkov, Y\. Choi, D\. Evans, and H\. HajishirziDo membership inference attacks work on large language models?\.CoRRabs/2402\.07841\.External Links:[Link](https://doi.org/10.48550/arXiv.2402.07841),[Document](https://dx.doi.org/10.48550/ARXIV.2402.07841),2402\.07841Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1)\.
- Fuet al\.\(2024\)W\. Fu, H\. Wang, C\. Gao, G\. Liu, Y\. Li, and T\. JiangMembership inference attacks against fine\-tuned large language models via self\-prompt calibration\.InAdvances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024,A\. Globersons, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. M\. Tomczak, and C\. Zhang \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper/_files/paper/2024/hash/f36ad694188bb4c4bbbd61e2038e069e-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=nZeVKeeFYf9)Cited by:[Appendix C](https://arxiv.org/html/2610.10616#A3.SS0.SSS0.Px3.p1.1),[§5\.3](https://arxiv.org/html/2610.10616#S5.SS3.p1.1)\.
- Hwanget al\.\(2023\)C\. Hwang, W\. Cui, Y\. Xiong, Z\. Yang, Z\. Liu, H\. Hu, Z\. Wang, R\. Salas, J\. Jose, P\. Ram, J\. Chau, P\. Cheng, F\. Yang, M\. Yang, and Y\. XiongTutel: adaptive mixture\-of\-experts at scale\.InProceedings of the Sixth Conference on Machine Learning and Systems, MLSys 2023, Miami, FL, USA, June 4\-8, 2023,D\. Song, M\. Carbin, and T\. Chen \(Eds\.\),External Links:[Link](https://proceedings.mlsys.org/paper/_files/paper/2023/hash/5616d34cf8ff73942cfd5aa922842556-Abstract-mlsys2023.html)Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p1.1)\.
- Ilicet al\.\(2026a\)D\. Ilic, K\. Cvejoski, D\. Stanojevic, and E\. GrigorenkoLearning the signature of memorization in autoregressive language models\.CoRRabs/2604\.03199\.External Links:[Link](https://doi.org/10.48550/arXiv.2604.03199),[Document](https://dx.doi.org/10.48550/ARXIV.2604.03199),2604\.03199Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Ilicet al\.\(2026b\)D\. Ilic, D\. Stanojevic, and K\. CvejoskiPowerful training\-free membership inference against fine\-tuned autoregressive language models\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\), ACL 2026, San Diego, California, United States, July 2\-7, 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),pp\. 14077–14093\.External Links:[Link](https://doi.org/10.18653/v1/2026.acl-long.640),[Document](https://dx.doi.org/10.18653/V1/2026.ACL-LONG.640)Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. de Las Casas, E\. B\. Hanna, F\. Bressand, G\. Lengyel, G\. Bour, G\. Lample, L\. R\. Lavaud, L\. Saulnier, M\. Lachaux, P\. Stock, S\. Subramanian, S\. Yang, S\. Antoniak, T\. L\. Scao, T\. Gervet, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMixtral of experts\.CoRRabs/2401\.04088\.External Links:[Link](https://doi.org/10.48550/arXiv.2401.04088),[Document](https://dx.doi.org/10.48550/ARXIV.2401.04088),2401\.04088Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p1.1)\.
- Kalyanapuet al\.\(2025\)J\. Kalyanapu, F\. Dizani, D\. Asher, A\. Ghanbari, R\. Cammarota, A\. Aysu, and S\. M\. AjorpazGateBleed: exploiting on\-core accelerator power gating for high performance and stealthy attacks on AI\.InProceedings of the 58th IEEE/ACM International Symposium on Microarchitecture, MICRO 2025, Seoul, Republic of Korea, October 18\-22, 2025,pp\. 308–325\.External Links:[Link](https://doi.org/10.1145/3725843.3756097),[Document](https://dx.doi.org/10.1145/3725843.3756097)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p3.1)\.
- Luet al\.\(2026\)Z\. Lu, H\. Chi, N\. Baracaldo, S\. R\. Kadhe, Y\. Jeon, and L\. YuIn\-context probing for membership inference in fine\-tuned language models\.In33rd Annual Network and Distributed System Security Symposium, NDSS 2026, San Diego, California, USA, February 23\-27, 2026,External Links:[Link](https://www.ndss-symposium.org/ndss-paper/in-context-probing-for-membership-inference-in-fine-tuned-language-models/)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1)\.
- Lvet al\.\(2026\)B\. Lv, Z\. Xu, K\. Xiu, R\. Ding, T\. Zheng, Z\. Wang, and K\. RenRouteScan: A non\-intrusive approach to auditing moe llms safety via expert routing telemetry\.CoRRabs/2605\.24817\.External Links:[Link](https://doi.org/10.48550/arXiv.2605.24817),[Document](https://dx.doi.org/10.48550/ARXIV.2605.24817),2605\.24817Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p1.1)\.
- Makhijaet al\.\(2026\)D\. Makhija, M\. G\. Arivazhagan, V\. B\. Kumar, and R\. GangadharaiahNeural breadcrumbs: membership inference attacks on llms through hidden state and attention pattern analysis\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2026 \- Volume 1: Long Papers, Rabat, Morocco, March 24\-29, 2026,V\. Demberg, K\. Inui, and L\. Marquez \(Eds\.\),pp\. 5608–5620\.External Links:[Link](https://doi.org/10.18653/v1/2026.eacl-long.262),[Document](https://dx.doi.org/10.18653/V1/2026.EACL-LONG.262)Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p2.1)\.
- Mireshghallahet al\.\(2022\)F\. Mireshghallah, K\. Goyal, A\. Uniyal, T\. Berg\-Kirkpatrick, and R\. ShokriQuantifying privacy risks of masked language models using membership inference attacks\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\-11, 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 8332–8347\.External Links:[Link](https://doi.org/10.18653/v1/2022.emnlp-main.570),[Document](https://dx.doi.org/10.18653/V1/2022.EMNLP-MAIN.570)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1)\.
- Muennighoffet al\.\(2025\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, E\. P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, and et al\.OLMoE: open mixture\-of\-experts language models\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=xXTkbTBmqq)Cited by:[§A\.2](https://arxiv.org/html/2610.10616#A1.SS2.p1.1),[§1](https://arxiv.org/html/2610.10616#S1.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p1.1)\.
- Nuriyev and Kulp \(2026\)A\. Nuriyev and G\. KulpExpert selections in moe models reveal \(almost\) as much as text\.CoRRabs/2602\.04105\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.04105),[Document](https://dx.doi.org/10.48550/ARXIV.2602.04105),2602\.04105Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p3.1)\.
- Quet al\.\(2024\)X\. Qu, D\. Dong, X\. Hu, T\. Zhu, W\. Sun, and Y\. ChengLLaMA\-moe v2: exploring sparsity of llama from perspective of mixture\-of\-experts with post\-training\.CoRRabs/2411\.15708\.External Links:[Link](https://doi.org/10.48550/arXiv.2411.15708),[Document](https://dx.doi.org/10.48550/ARXIV.2411.15708),2411\.15708Cited by:[§A\.2](https://arxiv.org/html/2610.10616#A1.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p1.1)\.
- Ranjanet al\.\(2026\)R\. Ranjan, U\. Grover, X\. Lin, and A\. PolyzouG\-drift MIA: membership inference via gradient\-induced feature drift in llms\.InPattern Recognition \- 28th International Conference, ICPR 2026, Lyon, France, August 17\-22, 2026, Proceedings, Part IV,M\. D\. Marsico, T\. K\. Ho, F\. Jurie, C\. Liu, D\. P\. Lopresti, I\. Nyström, J\. Ogier, A\. Ross, and L\. Wang \(Eds\.\),Lecture Notes in Computer Science, Vol\.16815,pp\. 359–374\.External Links:[Link](https://doi.org/10.1007/978-3-032-31663-9/_24),[Document](https://dx.doi.org/10.1007/978-3-032-31663-9%5F24)Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p2.1)\.
- Shiet al\.\(2024\)W\. Shi, A\. Ajith, M\. Xia, Y\. Huang, D\. Liu, T\. Blevins, D\. Chen, and L\. ZettlemoyerDetecting pretraining data from large language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=zWqr3MQuNs)Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Shokriet al\.\(2017\)R\. Shokri, M\. Stronati, C\. Song, and V\. ShmatikovMembership inference attacks against machine learning models\.In2017 IEEE Symposium on Security and Privacy, SP 2017, San Jose, CA, USA, May 22\-26, 2017,pp\. 3–18\.External Links:[Link](https://doi.org/10.1109/SP.2017.41),[Document](https://dx.doi.org/10.1109/SP.2017.41)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1)\.
- Song and Mittal \(2021\)L\. Song and P\. MittalSystematic evaluation of privacy risks of machine learning models\.In30th USENIX Security Symposium, USENIX Security 2021, August 11\-13, 2021,M\. D\. Bailey and R\. Greenstadt \(Eds\.\),pp\. 2615–2632\.External Links:[Link](https://www.usenix.org/conference/usenixsecurity21/presentation/song)Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Team \(2024\)Q\. TeamQwen1\.5\-moe: matching 7b model performance with 1/3 activated parameters”\.External Links:[Link](https://qwenlm.github.io/blog/qwen-moe/)Cited by:[§A\.2](https://arxiv.org/html/2610.10616#A1.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p1.1)\.
- Watsonet al\.\(2022\)L\. Watson, C\. Guo, G\. Cormode, and A\. SablayrollesOn the importance of difficulty calibration in membership inference attacks\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=3eIrli0TwQ)Cited by:[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Weiet al\.\(2022\)J\. Wei, M\. Bosma, V\. Y\. Zhao, K\. Guu, A\. W\. Yu, B\. Lester, N\. Du, A\. M\. Dai, and Q\. V\. LeFinetuned language models are zero\-shot learners\.InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25\-29, 2022,External Links:[Link](https://openreview.net/forum?id=gEZrGCozdqR)Cited by:[§5\.3](https://arxiv.org/html/2610.10616#S5.SS3.p1.1)\.
- Wenet al\.\(2024\)R\. Wen, Z\. Li, M\. Backes, and Y\. ZhangMembership Inference Attacks Against In\-Context Learning\.InACM SIGSAC Conference on Computer and Communications Security \(CCS\),Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1)\.
- Xieet al\.\(2024\)R\. Xie, J\. Wang, R\. Huang, M\. Zhang, R\. Ge, J\. Pei, N\. Gong, and B\. DhingraReCaLL: membership inference via relative conditional log\-likelihoods\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12\-16, 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),pp\. 8671–8689\.External Links:[Link](https://doi.org/10.18653/v1/2024.emnlp-main.493),[Document](https://dx.doi.org/10.18653/V1/2024.EMNLP-MAIN.493)Cited by:[§A\.3](https://arxiv.org/html/2610.10616#A1.SS3.SSS0.Px5.p1.1),[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Yeomet al\.\(2018\)S\. Yeom, I\. Giacomelli, M\. Fredrikson, and S\. JhaPrivacy risk in machine learning: analyzing the connection to overfitting\.In31st IEEE Computer Security Foundations Symposium, CSF 2018, Oxford, United Kingdom, July 9\-12, 2018,pp\. 268–282\.External Links:[Link](https://doi.org/10.1109/CSF.2018.00027),[Document](https://dx.doi.org/10.1109/CSF.2018.00027)Cited by:[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Yonaet al\.\(2024\)I\. Yona, I\. Shumailov, J\. Hayes, and N\. CarliniStealing user prompts from mixture of experts\.CoRRabs/2410\.22884\.External Links:[Link](https://doi.org/10.48550/arXiv.2410.22884),[Document](https://dx.doi.org/10.48550/ARXIV.2410.22884),2410\.22884Cited by:[§2](https://arxiv.org/html/2610.10616#S2.p3.1)\.
- Zareeet al\.\(2026\)P\. Zaree, M\. A\. A\. Mamun, Y\. Dong, I\. Alouani, and N\. B\. Abu\-GhazalehAttenMIA: LLM membership inference attack through attention signals\.CoRRabs/2601\.18110\.External Links:[Link](https://doi.org/10.48550/arXiv.2601.18110),[Document](https://dx.doi.org/10.48550/ARXIV.2601.18110),2601\.18110Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p2.1)\.
- Zhang and Wu \(2024\)A\. Zhang and C\. WuAdaptive pre\-training data detection for large language models via surprising tokens\.CoRRabs/2407\.21248\.External Links:[Link](https://doi.org/10.48550/arXiv.2407.21248),[Document](https://dx.doi.org/10.48550/ARXIV.2407.21248),2407\.21248Cited by:[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.
- Zhanget al\.\(2025\)J\. Zhang, J\. Sun, E\. C\. Yeats, Y\. Ouyang, M\. Kuo, J\. Zhang, H\. \(\. Yang, and H\. LiMin\-k%\+\+: improved baseline for pre\-training data detection from large language models\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=ZGkfoufDaU)Cited by:[§1](https://arxiv.org/html/2610.10616#S1.p2.1),[§2](https://arxiv.org/html/2610.10616#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.10616#S5.SS1.p3.1)\.

## Appendix AExperimental Setup

### A\.1Dataset Construction and Validation

We evaluate membership inference on three domains: arXiv, GitHub, and Wikipedia\. For each domain, we construct four independent membership instances from a common source pool\. Each instancej∈\{1,…,4\}j\\in\\\{1,\\ldots,4\\\}consists of a fine\-tuning setD~jin\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}and a held\-out setD~jout\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j\}, corresponding to member and non\-member examples, respectively\. The source corpus is shuffled before construction, and candidate documents are assigned so that no document is shared across different instances:

\(D~jin∪D~jout\)∩\(D~j′in∪D~j′out\)=∅,j≠j′\.\\left\(\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}\\cup\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j\}\\right\)\\cap\\left\(\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j^\{\\prime\}\}\\cup\\widetilde\{D\}^\{\\mathrm\{out\}\}\_\{j^\{\\prime\}\}\\right\)=\\varnothing,\\qquad j\\neq j^\{\\prime\}\.\(2\)This prevents the membership classifier from benefiting from repeated documents across shadow and target models\.

For arXiv, each instance contains 2,500 members and 2,500 non\-members, drawn from a source pool of 24,606 documents\. For GitHub and Wikipedia, each instance contains 1,500 members and 1,500 non\-members, with a source pool of 12,000 documents per domain\. The same four data partitions are reused across all three model architectures within a domain\. Thus, when comparing architectures, the underlying membership instances are held fixed and only the model changes\. Dataset statistics are summarized in Table[4](https://arxiv.org/html/2610.10616#A1.T4)\.

Table 4:Dataset statistics used for the main experiments\.NinN\_\{\\mathrm\{in\}\}andNoutN\_\{\\mathrm\{out\}\}denote the number of member and non\-member examples, respectively, in each of the four independent membership instances\.We additionally verify that the member and non\-member partitions do not contain readily exploitable surface\-text differences\. Such differences can allow model\-free ”blind” classifiers to outperform membership inference attacks, making attack performance difficult to interpret\([Das et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib29)\)\. Following this evaluation practice, for each of the 12 domain–instance pairs, we train four text\-only classifiers that use no information from either the pretrained or fine\-tuned model: word TF–IDF features with 1–2\-grams, character TF–IDF features with 3–5\-grams, bag\-of\-words features, and hand\-crafted surface statistics\. Each classifier is evaluated using five\-fold out\-of\-fold prediction\. Across all 48 checks, AUROC ranges from 0\.4817 to 0\.5329, with no systematic deviation from chance\. This sanity check indicates that the membership labels are not readily predictable from surface properties of the constructed datasets alone\.

### A\.2Models and Fine\-Tuning

We consider three open\-weight MoE language models: OLMoE\-1B\-7B\-0924\([Muennighoff et al\., 2025](https://arxiv.org/html/2610.10616#bib.bib2)\), Qwen1\.5\-MoE\-A2\.7B\-Chat\([Team, 2024](https://arxiv.org/html/2610.10616#bib.bib3)\), and LLaMA\-MoE\-v2\-3\.8B\-2\.8\-sft\([Qu et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib4)\)\. Combining the three architectures with the three domains above yields the3×33\\times 3model–domain grid used in the main experiments\.

For each model–domain pair and each membership instancejj, we fine\-tune the corresponding pretrained model onD~jin\\widetilde\{D\}^\{\\mathrm\{in\}\}\_\{j\}to obtain an independently adapted modelM~j\\widetilde\{M\}\_\{j\}\. Unless otherwise specified, the main experiments use full\-parameter fine\-tuning, including the router parameters\. We use Adafactor with a learning rate of8×10−68\\times 10^\{\-6\}, a training batch size of 16, and a maximum sequence length of 256 tokens\. All main results are evaluated after one epoch of fine\-tuning, and the same configuration is used across all nine model–domain settings\.

The four models within each model–domain setting are trained independently on mutually disjoint fine\-tuning sets\. In the main leave\-one\-out evaluation, each model is in turn treated as the target model while the remaining three act as shadow models\.

### A\.3MIA Baseline Implementation

We evaluate output\-side membership signals derived from the fine\-tuned target model and, when required, the public reference checkpoint\. Unless otherwise specified, the resulting feature representations are provided to the same histogram gradient boosting classifier used throughout our experiments\.

#### Reference\-free likelihood and confidence features\.

We first construct a 13\-dimensional feature block, denotedϕstrong\\phi\_\{\\mathrm\{strong\}\}, from token\-level statistics of the fine\-tuned model\. Letℓt\\ell\_\{t\}denote the log\-probability assigned to the true token at positiontt\.

NLL\.We use the mean negative log\-likelihood over all valid tokens as a one\-dimensional feature\.

Min\-K%\.We sort the token log\-probabilities in ascending order and compute the mean over the lowestK%K\\%of tokens forK∈\{10,20,30\}K\\in\\\{10,20,30\\\}\. The resulting three values form a three\-dimensional Min\-K% feature\.

Min\-K%\+\+\.For each token, we normalize the target\-token logit using the mean and standard deviation of the full vocabulary\-logit distribution at that position:

zt=ut−μtσt,z\_\{t\}=\\frac\{u\_\{t\}\-\\mu\_\{t\}\}\{\\sigma\_\{t\}\},\(3\)whereutu\_\{t\}is the true\-token logit andμt\\mu\_\{t\}andσt\\sigma\_\{t\}are the mean and standard deviation of the vocabulary logits\. We sort\{zt\}\\\{z\_\{t\}\\\}in ascending order and average the lowestK%K\\%forK∈\{10,20\}K\\in\\\{10,20\\\}, yielding a two\-dimensional feature\.

zlib\-normalized likelihood\.Let

ρzlib​\(x\)=\|zlib⁡\(x\)\|\|x\|\\rho\_\{\\mathrm\{zlib\}\}\(x\)=\\frac\{\|\\operatorname\{zlib\}\(x\)\|\}\{\|x\|\}\(4\)denote the compression ratio of the UTF\-8 representation of the document\. We use bothρzlib​\(x\)\\rho\_\{\\mathrm\{zlib\}\}\(x\)and

NLL⁡\(x\)ρzlib​\(x\)\+ϵ\\frac\{\\operatorname\{NLL\}\(x\)\}\{\\rho\_\{\\mathrm\{zlib\}\}\(x\)\+\\epsilon\}\(5\)as features, giving a two\-dimensional zlib block\.

Logit margin\.At each valid token position, we compute the difference between the largest and second\-largest vocabulary logits and average this quantity over the document, yielding a single scalar feature\.

Log\-probability percentile statistics\.From the sequence of true\-token log\-probabilities, we compute the 10th, 50th, and 90th percentiles together with the standard deviation\. These four statistics form a four\-dimensional feature block\.

The six feature groups above contain 13 dimensions in total and constitute the reference\-free feature block used as part ofAoutA\_\{\\mathrm\{out\}\}\.

#### Reference loss drop\.

Letℓtft\\ell\_\{t\}^\{\\mathrm\{ft\}\}andℓtpre\\ell\_\{t\}^\{\\mathrm\{pre\}\}denote the true\-token log\-probabilities under the fine\-tuned and reference models, respectively\. We compute

ϕrefdrop​\(x\)=1T​∑t=1T\(ℓtft−ℓtpre\),\\phi\_\{\\mathrm\{refdrop\}\}\(x\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\left\(\\ell\_\{t\}^\{\\mathrm\{ft\}\}\-\\ell\_\{t\}^\{\\mathrm\{pre\}\}\\right\),\(6\)using the positions available under both models\.

#### WBC\.

Following the window\-based comparison between the fine\-tuned and reference models, we consider ten window sizes

𝒲=\{2,3,4,5,8,11,15,21,29,40\}\.\\mathcal\{W\}=\\\{2,3,4,5,8,11,15,21,29,40\\\}\.\(7\)For each window sizeww, we slide a window across the document and compute the fraction of windows for which the summed log\-probability under the fine\-tuned model exceeds that under the reference model\. We denote this vote fraction byvwv\_\{w\}and compute

SWBC=1\|𝒲\|​∑w∈𝒲vw\.S\_\{\\mathrm\{WBC\}\}=\\frac\{1\}\{\|\\mathcal\{W\}\|\}\\sum\_\{w\\in\\mathcal\{W\}\}v\_\{w\}\.\(8\)The individual WBC baseline uses onlySWBCS\_\{\\mathrm\{WBC\}\}\. ForAoutA\_\{\\mathrm\{out\}\}, we additionally retain the vote fractions\{vw\}w∈𝒲\\\{v\_\{w\}\\\}\_\{w\\in\\mathcal\{W\}\}\.

#### EZ\.

Letℰ\\mathcal\{E\}denote positions where the fine\-tuned model’s top\-1 prediction differs from the ground\-truth token, and define

dt=ℓtft−ℓtpre,t∈ℰ\.d\_\{t\}=\\ell\_\{t\}^\{\\mathrm\{ft\}\}\-\\ell\_\{t\}^\{\\mathrm\{pre\}\},\\qquad t\\in\\mathcal\{E\}\.\(9\)We compute

P=∑t∈ℰmax⁡\(dt,0\),N=∑t∈ℰmax⁡\(−dt,0\)\.P=\\sum\_\{t\\in\\mathcal\{E\}\}\\max\(d\_\{t\},0\),\\qquad N=\\sum\_\{t\\in\\mathcal\{E\}\}\\max\(\-d\_\{t\},0\)\.\(10\)The individual EZ baseline uses the ratio

SEZ=PN\+ϵ\.S\_\{\\mathrm\{EZ\}\}=\\frac\{P\}\{N\+\\epsilon\}\.\(11\)WithinAoutA\_\{\\mathrm\{out\}\}, we use the four\-dimensional block

\[PN\+ϵ,log⁡\(1\+P\),log⁡\(1\+N\),\|ℰ\|/T\]\.\\left\[\\frac\{P\}\{N\+\\epsilon\},\\log\(1\+P\),\\log\(1\+N\),\|\\mathcal\{E\}\|/T\\right\]\.\(12\)

#### ReCaLL\.

We use a fixed prefix constructed from eight held\-out non\-member documents\. The concatenated prefix is tokenized and truncated to at most 128 tokens and is reused for all evaluated candidates\. For candidatexx, ReCaLL is computed as the difference between its mean token log\-probability under the fine\-tuned model when conditioned on this prefix and its mean token log\-probability without the prefix\. The original ReCaLL score is defined as the ratio of the conditional to the unconditional log\-likelihood\([Xie et al\., 2024](https://arxiv.org/html/2610.10616#bib.bib11)\); we use their difference as a simpler variant that captures the same prefix\-induced change\. This signal uses only the fine\-tuned model and does not require the reference checkpoint\.

#### LTsig\.

We construct a signature from token\-level differences between the fine\-tuned and reference models\. Let

dt=ℓtft−ℓtpre\.d\_\{t\}=\\ell\_\{t\}^\{\\mathrm\{ft\}\}\-\\ell\_\{t\}^\{\\mathrm\{pre\}\}\.\(13\)The signature contains 16 uniformly spaced quantiles of\{dt\}\\\{d\_\{t\}\\\}, eight uniformly spaced quantiles of\{ℓtft\}\\\{\\ell\_\{t\}^\{\\mathrm\{ft\}\}\\\}, and eight summary statistics:

\[mean⁡\(d\),std⁡\(d\),Pr⁡\(dt\>0\),∑tdt,mean\(rft\),mean\(rpre\),mean\(rft−rpre\),T\],\\begin\{split\}\[&\\operatorname\{mean\}\(d\),\\operatorname\{std\}\(d\),\\Pr\(d\_\{t\}\>0\),\\sum\_\{t\}d\_\{t\},\\\\ &\\operatorname\{mean\}\(r^\{\\mathrm\{ft\}\}\),\\operatorname\{mean\}\(r^\{\\mathrm\{pre\}\}\),\\operatorname\{mean\}\(r^\{\\mathrm\{ft\}\}\-r^\{\\mathrm\{pre\}\}\),T\],\\end\{split\}\(14\)whererftr^\{\\mathrm\{ft\}\}andrprer^\{\\mathrm\{pre\}\}denote the corresponding token\-rank statistics\. This produces a 32\-dimensional feature representation\. Our implementation follows the token\-level target–reference signature idea of LTsig, but uses this fixed\-dimensional summary with the common histogram gradient boosting classifier rather than reproducing the original sequence\-based classifier\.

#### Output\-signal ensemble\.

Our primary output\-side baseline, denoted byAoutA\_\{\\mathrm\{out\}\}, concatenates the 13\-dimensional likelihood/confidence block with reference loss drop, the full WBC feature block, the full EZ feature block, ReCaLL, and the 32\-dimensional LTsig\-style signature:

ϕout​\(x\)=concatb∈ℬ⁡ϕb​\(x\)\.\\phi\_\{\\mathrm\{out\}\}\(x\)=\\operatorname\{concat\}\_\{b\\in\\mathcal\{B\}\}\\phi\_\{b\}\(x\)\.\(15\)A histogram gradient boosting classifier is then trained onϕout​\(x\)\\phi\_\{\\mathrm\{out\}\}\(x\)\.

For the individual\-baseline analysis in Appendix[B](https://arxiv.org/html/2610.10616#A2), we evaluate each feature family separately\. The standalone WBC and EZ rows useSWBCS\_\{\\mathrm\{WBC\}\}andSEZS\_\{\\mathrm\{EZ\}\}, respectively, whileAoutA\_\{\\mathrm\{out\}\}retains their full feature blocks as described above\.

### A\.4Router\-Telemetry Representation

Our primary telemetry representation is the continuous expert\-load feature introduced in Section[4](https://arxiv.org/html/2610.10616#S4)\. For each MoE layerℓ\\elland expertee, we average the softmax routing probability across all tokens in the candidate sequence:

rℓ,e​\(x\)=1T​∑t=1Tpℓ,t,e\.r\_\{\\ell,e\}\(x\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}p\_\{\\ell,t,e\}\.\(16\)We then flatten the per\-layer expert loads into

ϕR​\(x\)=\[rℓ,e​\(x\)\]ℓ,e\.\\phi\_\{R\}\(x\)=\\left\[r\_\{\\ell,e\}\(x\)\\right\]\_\{\\ell,e\}\.\(17\)This continuous representation is used by default throughout the main experiments\.

We separately evaluate more restricted forms of telemetry\. In particular,ϕRcount\\phi\_\{R\}^\{\\mathrm\{count\}\}records the frequency with which each expert is selected by the router rather than its continuous routing probability, thereby retaining only discrete expert\-selection information\.

### A\.5Attack Training and Evaluation

We use histogram gradient boosting as the default membership classifier\. Both output\-only and router\-augmented attacks follow the same leave\-one\-out shadow\-model evaluation protocol\.

#### Leave\-one\-out shadow\-model evaluation\.

For each model–domain setting, we construct four independently fine\-tuned models\. In each evaluation round, one modelM~j\\widetilde\{M\}\_\{j\}is treated as the held\-out target, while the remaining three models serve as shadow models\. The attack classifier is trained only on the labeled member and non\-member examples from these three shadow models and is then applied directly to the held\-out target, without using any target membership labels or target\-specific refitting\. We repeat this procedure four times so that each independently fine\-tuned model serves once as the target\.

#### Output\-only attack\.

For each shadow model, we extract the output\-side representationϕout​\(x\)\\phi\_\{\\mathrm\{out\}\}\(x\)for its member and non\-member examples\. The representations from the three shadow models are pooled to train the output\-only membership classifier\. For a candidatexxqueried against the held\-out target, the output\-only membership score is

sout​\(x\)=gout​\(ϕout​\(x\)\)\.s\_\{\\mathrm\{out\}\}\(x\)=g\_\{\\mathrm\{out\}\}\\\!\\left\(\\phi\_\{\\mathrm\{out\}\}\(x\)\\right\)\.\(18\)

#### Router\-augmented attack\.

The router\-augmented attack uses the identical shadow\-model training and target evaluation protocol, but augments the output\-side representation with router telemetry:

ϕaug​\(x\)=\[ϕout​\(x\);ϕR​\(x\)\]\.\\phi\_\{\\mathrm\{aug\}\}\(x\)=\[\\phi\_\{\\mathrm\{out\}\}\(x\);\\phi\_\{R\}\(x\)\]\.\(19\)After feature standardization, the membership classifier is trained on the pooled shadow\-model examples and applied directly to the held\-out target:

saug​\(x\)=gaug​\(ϕaug​\(x\)\)\.s\_\{\\mathrm\{aug\}\}\(x\)=g\_\{\\mathrm\{aug\}\}\\\!\\left\(\\phi\_\{\\mathrm\{aug\}\}\(x\)\\right\)\.\(20\)
Because the output\-only and router\-augmented attacks use the same shadow models, membership labels, classifier family, and held\-out target, their performance difference isolates the additional membership signal provided by router telemetry\.

## Appendix BResults Across MIA Baselines

The main experiments use the output\-signal ensembleAoutA\_\{\\mathrm\{out\}\}as the primary output\-side baseline\. To examine whether the membership signal provided by router telemetry depends on a particular MIA baseline, we augment each individual output\-side baseline with the continuous expert\-load representationϕR\\phi\_\{R\}and evaluate it over the full3×33\\times 3model–domain grid\.

Table[5](https://arxiv.org/html/2610.10616#A2.T5)reports the improvements in AUROC andTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}from adding router telemetry\. Each entry is the difference between the router\-augmented attack and the corresponding output\-side baseline\.

Across the 108 baseline–model–domain combinations, router telemetry improves AUROC in 107 settings andTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}in 104 settings\. Positive gains are observed across likelihood\-based, confidence\-based, and reference\-based MIA baselines\. The few cases with negligible or negative gains are concentrated in logit margin and ReCaLL\. Baselines with strong output\-only performance, such as LTsig andAoutA\_\{\\mathrm\{out\}\}, generally exhibit smaller gains, leaving less headroom for additional improvement\.

Overall, router telemetry provides membership information that complements a broad range of output\-side MIA baselines\. This pattern holds for both AUROC andTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}, indicating that the additional membership signal is not specific to the output\-signal ensemble or to a particular evaluation metric\.

Table 5:Improvements from augmenting individual MIA baselines with continuous expert\-load telemetryϕR\\phi\_\{R\}across all model–domain settings\. Each entry reports the difference between the router\-augmented attack and its output\-only counterpart\.
## Appendix CFine\-Tuning Regimes

We provide the implementation details for the fine\-tuning regimes evaluated in Section[5\.3](https://arxiv.org/html/2610.10616#S5.SS3)\. All experiments use OLMoE\-1B\-7B on arXiv with the same four disjoint membership instances and are evaluated after one epoch of fine\-tuning\.

#### Full fine\-tuning\.

We update all model parameters, including the router gates, using Adafactor with a learning rate of8×10−68\\times 10^\{\-6\}, a batch size of 16, and a maximum sequence length of 256\. This configuration is identical to the main fine\-tuning setup described in Appendix[A\.2](https://arxiv.org/html/2610.10616#A1.SS2)\.

#### Frozen\-router fine\-tuning\.

We use the same configuration as full fine\-tuning but keep the router gate parameters fixed throughout training\. All other model parameters are updated normally\. This setting separates the membership signal exposed through router telemetry from changes to the router parameters themselves\.

#### LoRA\.

We apply LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.10616#bib.bib31)\)to the attention and expert\-MLP projection layers with rankr=16r=16and scaling factorα=32\\alpha=32\. We use a learning rate of2×10−42\\times 10^\{\-4\}, with approximately 2\.2% of the model parameters trainable\. The router gates are not included in the LoRA target modules and therefore remain fixed during fine\-tuning\.

#### Instruction tuning\.

We perform full\-parameter fine\-tuning using an instruction–response format\. Each arXiv document is converted into the following training example:

\#\#\# Instruction: Write an academic abstract\. \#\#\# Response: <document\>

The training loss is applied only to the response tokens, while the instruction tokens are masked from the loss\. During feature extraction, we use the same instruction–response format so that the output\-side signals and router telemetry are computed under the same input context used during fine\-tuning\.

## Appendix DLimited Router Telemetry and Shadow\-Model Access

We provide the implementation details for the restricted router\-telemetry and shadow\-model settings evaluated in Section[5\.4](https://arxiv.org/html/2610.10616#S5.SS4)\.

### D\.1Top\-K Router Telemetry

To restrict the amount of exposed router telemetry, we construct a top\-KKvariant ofϕR\\phi\_\{R\}on OLMoE\-1B\-7B with arXiv\. For each candidate, experts are ranked according to their aggregated document\-level routing load\. We retain the dimensions corresponding to the top\-KKexpert loads and mask the remaining dimensions\. We evaluate

top\-​K∈\{1,2,4,8,16,32\}\.\\text\{top\-\}K\\in\\\{1,2,4,8,16,32\\\}\.\(21\)The restricted representation is then used with the same attack training and evaluation protocol as the full expert\-load representation\.

This setting is distinct fromϕRcount\\phi\_\{R\}^\{\\mathrm\{count\}\}\. The top\-KKrepresentation restricts the amount of continuous document\-level load information available to the attacker, whereasϕRcount\\phi\_\{R\}^\{\\mathrm\{count\}\}is constructed directly from discrete expert selections\.

### D\.2Number of Shadow Models

The main attack uses three shadow models for each held\-out target\. To evaluate the effect of reducing shadow\-model access, we repeat the OLMoE\-1B\-7B on arXiv experiment withKshadow∈\{1,2,3\}K\_\{\\mathrm\{shadow\}\}\\in\\\{1,2,3\\\}\.

Figure 5:Effect of the number of shadow models on OLMoE\-1B\-7B with arXiv\. The y\-axis reports the improvement inTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}overAoutA\_\{\\mathrm\{out\}\}\.For each held\-out target and eachKshadowK\_\{\\mathrm\{shadow\}\}, we enumerate all subsets of sizeKshadowK\_\{\\mathrm\{shadow\}\}from the remaining three independently fine\-tuned models\. The membership classifier is trained on the pooled member and non\-member examples from each selected subset and evaluated directly on the held\-out target\. Predictions are pooled across all shadow\-model subsets and all four held\-out targets before computing the evaluation metrics\. The settingKshadow=3K\_\{\\mathrm\{shadow\}\}=3is identical to the leave\-one\-out setting used in the main experiments\.

As shown in Figure[5](https://arxiv.org/html/2610.10616#A4.F5), a single shadow model already yields an improvement of 0\.0493 inTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}overAoutA\_\{\\mathrm\{out\}\}\. The gain increases to 0\.0696 with two shadow models and 0\.0841 with three, showing that the router\-augmented attack remains effective even with limited shadow\-model access\.

## Appendix EMechanism Analysis

We provide implementation details for the mechanism analysis in the main text\. The analysis is conducted on OLMoE\-1B\-7B with arXiv using the same four disjoint membership instances as the main experiments\. We analyze the pretrained model and the models after one and two epochs of fine\-tuning\. Hidden states and router weight matrices are recorded at each MoE layer for both member and non\-member examples\.

### E\.1Router\-Subspace Projection

At MoE layerℓ\\ell, the router maps the hidden statehℓ,t∈ℝdh\_\{\\ell,t\}\\in\\mathbb\{R\}^\{d\}of tokenttto router logits through

zℓ,t=Wℓ​hℓ,t,z\_\{\\ell,t\}=W\_\{\\ell\}h\_\{\\ell,t\},\(22\)whereWℓ∈ℝE×dW\_\{\\ell\}\\in\\mathbb\{R\}^\{E\\times d\}is the router weight matrix\. For OLMoE\-1B\-7B,d=2048d=2048andE=64E=64\. The row space ofWℓW\_\{\\ell\}therefore defines a low\-dimensional subspace of the hidden representation that is directly observed by the router\.

To measure the membership information contained in this subspace, we project the hidden states onto the row space ofWℓW\_\{\\ell\}at each MoE layer and aggregate the projected representations over the input sequence\. We then train a membership classifier on the resulting features using five\-fold stratified out\-of\-fold prediction separately for each independently fine\-tuned model\. The reported AUROC is averaged over the four models\.

#### Matched\-rank random projection\.

As a control, we construct a random subspace with the same rank as the router subspace at each layer and project the same hidden states onto this subspace\. The router and random projections therefore use the same layers and the same projection dimensionality\. This comparison controls for the amount of information retained solely due to dimensionality and tests whether the router subspace captures more membership information than an equally sized generic projection of the hidden state\.

### E\.2Frozen\-Router Control

We repeat the complete projection analysis with a frozen\-router fine\-tuning run\. The data partitions, training configuration, and evaluation procedure are identical to those of the corresponding full fine\-tuning run, except that all router gate parameters are held fixed throughout fine\-tuning\. The remaining model parameters are updated normally\.

For both full fine\-tuning and frozen\-router fine\-tuning, we extract the hidden states and router weight matrices at the pretrained checkpoint and after each fine\-tuning epoch\. We then apply the same router\-subspace and matched\-rank random\-subspace evaluations\. Because the router weights do not change in the frozen\-router setting, this comparison separates membership information exposed through the router subspace from membership information that could arise from direct updates to the router parameters\.

## Appendix FTelemetry Perturbation Defenses

We provide the implementation details for the telemetry perturbation defenses evaluated in the main experiments and report top\-1 expert agreement as an additional measure of telemetry utility\. All experiments use OLMoE\-1B\-7B on arXiv\. The perturbations are applied directly to the continuous expert\-load representationϕR\\phi\_\{R\}before it is exposed to the attacker, while the model and output\-side signals remain unchanged\.

### F\.1Perturbation Mechanisms

#### Gaussian noise\.

For each telemetry dimensionjj, we add zero\-mean Gaussian noise scaled by the empirical standard deviation of that dimension:

r~j=max⁡\(0,rj\+ϵj\),ϵj∼𝒩⁡\(0,σ2​sj2\),\\widetilde\{r\}\_\{j\}=\\max\\left\(0,r\_\{j\}\+\\epsilon\_\{j\}\\right\),\\qquad\\epsilon\_\{j\}\\sim\\mathcal\{N\}\\left\(0,\\sigma^\{2\}s\_\{j\}^\{2\}\\right\),\(23\)whererjr\_\{j\}is the original expert\-load feature andsjs\_\{j\}is its empirical standard deviation\. We evaluateσ∈\{0,0\.1,0\.25,0\.5,1,2,4\}\\sigma\\in\\\{0,0\.1,0\.25,0\.5,1,2,4\\\}\.

#### Feature dropout\.

For each candidate input, we independently mask each telemetry dimension with probabilitypp:

r~j=mj​rj,mj∼Bernoulli⁡\(1−p\)\.\\widetilde\{r\}\_\{j\}=m\_\{j\}r\_\{j\},\\qquad m\_\{j\}\\sim\\operatorname\{Bernoulli\}\(1\-p\)\.\(24\)We evaluatep∈\{0,0\.25,0\.5,0\.75,0\.9,0\.99\}p\\in\\\{0,0\.25,0\.5,0\.75,0\.9,0\.99\\\}\. The dropout mask is sampled independently for each candidate input\.

#### Quantization\.

We reduce the precision of the exposed expert loads by quantizing each dimension into a fixed quantization resolution:

r~j=1B​round⁡\(B​rj\),\\widetilde\{r\}\_\{j\}=\\frac\{1\}\{B\}\\operatorname\{round\}\\left\(Br\_\{j\}\\right\),\(25\)where smallerBBcorresponds to coarser telemetry\. We evaluateB∈\{256,32,16,8,4\}B\\in\\\{256,32,16,8,4\\\}\.

For each perturbation level, the perturbed telemetry replacesϕR\\phi\_\{R\}in the same leave\-one\-out attack pipeline described in Appendix[A\.5](https://arxiv.org/html/2610.10616#A1.SS5)\. The output\-signal ensembleAoutA\_\{\\mathrm\{out\}\}is kept unchanged\.

### F\.2Top\-1 Expert Agreement

In addition to the load\-correlation metric reported in the main text, we measure whether the perturbed telemetry preserves the identity of the most heavily loaded expert\. For each candidate inputiiand MoE layerℓ\\ell, top\-1 expert agreement is defined as

Agreetop1=1N​L∑i=1N∑ℓ=1L𝕀\[argmaxeri,ℓ,e=argmaxer~i,ℓ,e\],\\operatorname\{Agree\}\_\{\\mathrm\{top1\}\}=\\frac\{1\}\{NL\}\\sum\_\{i=1\}^\{N\}\\sum\_\{\\ell=1\}^\{L\}\\mathbb\{I\}\\left\[\\arg\\max\_\{e\}r\_\{i,\\ell,e\}=\\arg\\max\_\{e\}\\widetilde\{r\}\_\{i,\\ell,e\}\\right\],\(26\)whereNNis the number of candidate inputs andLLis the number of MoE layers\. Higher agreement indicates that the perturbed telemetry better preserves expert\-selection information from the original router loads\.

\(a\)Gaussian noise\.\(b\)Feature dropout\.\(c\)Quantization\.
Figure 6:Membership signal and telemetry utility under router\-telemetry perturbation on OLMoE\-1B\-7B with arXiv\. The left axis reports the additionalTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}from router telemetry, while the right axis reports top\-1 expert agreement with the original telemetry\.Figure[6](https://arxiv.org/html/2610.10616#A6.F6)shows that reducing the membership signal is accompanied by a loss of expert\-selection information\. Under Gaussian noise, both quantities decrease gradually\. Atσ=4\\sigma=4, the additionalTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}decreases from 0\.0841 to 0\.0159, while top\-1 expert agreement decreases from 1\.000 to 0\.407\.

Feature dropout produces a sharper reduction in telemetry utility\. Atp=0\.9p=0\.9, the additionalTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}is 0\.0190 and top\-1 agreement is 0\.101\. Increasing the dropout rate top=0\.99p=0\.99further reduces these values to 0\.0040 and 0\.050, respectively\.

Quantization shows a similar trade\-off\. WithB=8B=8, the additionalTPR⁡@​1%​FPR\\operatorname\{TPR\}@1\\%\\operatorname\{FPR\}decreases to 0\.0121 while top\-1 agreement decreases to 0\.316\. AtB=4B=4, the corresponding values are 0\.0079 and 0\.076\.

Overall, the top\-1 agreement results are consistent with the privacy–utility trade\-off observed using load correlation in the main text\. Stronger perturbations reduce the membership signal, but the settings that suppress most of the additional signal also substantially degrade the expert\-selection information retained by the exposed telemetry\.