刻画微调混合专家大语言模型中可恢复的专家利用特征

# Exploiting Structural Memorization for Expert Pathway Recovery in Distributed Fine-Tuning of Mixture-of-Experts LLMs
Source: [https://arxiv.org/html/2610.41039](https://arxiv.org/html/2610.41039)
CCS:Security and privacyCCS:Computing methodologies Machine learningFei WangAffiliation:University of Toronto,Department of Electrical and Computer Engineering,Toronto,Ontario,Canadaemail:[silviafey\.wang@utoronto\.ca](mailto:[email protected])Baochun LiAffiliation:University of Toronto,Department of Electrical and Computer Engineering,Toronto,Ontario,Canadaemail:[bli@ece\.toronto\.edu](mailto:[email protected])andBo LiAffiliation:Hong Kong University of Science and Technology,Department of Computer Science and Engineering,Hong Kong,Chinaemail:[bli@cse\.ust\.hk](mailto:[email protected])

###### Abstract\.

As Mixture\-of\-Experts \(MoE\) large language models become increasingly deployed in distributed computing environments—from federated learning to edge\-cloud systems—their unique architectural properties harbor a new, underexplored landscape of security vulnerabilities\. In this work, we move beyond traditional textual analysis to uncover*structural memorization*: a novel vulnerability where a fine\-tuned model memorizes not just its training data, but also the specific internal pathway used to process it, creating a consistent and characteristic*routing fingerprint*unique to the training process\. We provide comprehensive empirical evidence that*structural memorization*is a tangible phenomenon, demonstrating how it emerges consistently even from just a few dataset samples and how its intensity is directly influenced by the selection of trainable modules\.

We show that these memorized expert pathways can be stealthily recovered by an adversary through the model’s unconditional generation process, with only access to the model’s routing and output logits, requiring no prior knowledge of the fine\-tuning data or training procedures\. In distributed deployments, this vulnerability is amplified as routing information becomes exposed at multiple trust boundaries, including federated aggregation points, edge\-cloud interfaces, and multi\-tenant monitoring systems\. This recovery process exposes a critical blind spot: traditional membership inference metrics adopted by the literature to measure model memorization, when viewed through the lens of expert routing, fail to effectively identify memorized data corresponding to these structural pathways\. To better characterize this detection gap, we devise composite metrics that combine reference model tail likelihood with complexity\-aware normalization, significantly improving the detection of memorized routing structures\. This discovery enables adversaries to pinpoint critical experts shaped by the fine\-tuning process, whether dominant or dormant, and execute sophisticated targeted attacks like router manipulation or backdoor injection under realistic distributed threat scenarios where attackers lack training data access but can observe routing patterns across system components\.

###### Keywords:

Mixture\-of\-Experts, Large Language Models, Distributed AI Systems, Federated Learning, Membership Inference, Fine\-Tuning, Security, Privacy

## 1\.Introduction

The deployment of large language models \(LLMs\) in distributed computing environments has become increasingly prevalent as organizations seek to balance computational efficiency with data privacy\. Federated learning enables collaborative model training across geographically distributed data silos without centralizing sensitive information\([McMahan et al\., 2016](https://arxiv.org/html/2610.41039#bib.bib1);[Kairouz et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib2)\), while edge computing deployments partition models between resource\-constrained devices and cloud infrastructure to minimize latency\([Zhou et al\., 2019](https://arxiv.org/html/2610.41039#bib.bib3);[Deng et al\., 2020](https://arxiv.org/html/2610.41039#bib.bib4)\)\. Multi\-tenant cloud platforms offer model fine\-tuning as a service, allowing multiple organizations to customize shared base models on proprietary datasets\([Gunawi et al\., 2016](https://arxiv.org/html/2610.41039#bib.bib5);[Verma et al\., 2015](https://arxiv.org/html/2610.41039#bib.bib6)\)\. These distributed AI systems introduce complex trust boundaries where model internals, including intermediate activations and routing decisions, become exposed across multiple nodes\. This distributed exposure creates novel privacy risks that extend beyond traditional centralized training paradigms\.

Among modern architectures deployed in these distributed settings, Mixture\-of\-Experts \(MoE\) models have emerged as a particularly attractive solution due to their computational efficiency and scalability\. MoE addresses the prohibitive inference costs associated with scaling large language models \(LLMs\) through a distinctive approach\. In contrast to dense models that process all inputs monolithically, MoE leverages sparse activation, using a routing network to dynamically assign each token to expert sub\-networks, each implemented as a feed\-forward network\. This approach not only yields a superior performance\-to\-compute ratio, but also allows experts to develop specialized knowledge, enabling the deployment of larger, more capable language models that would not otherwise be feasible with dense architectures\. The sparse activation property makes MoE particularly well\-suited for distributed deployments: expert parallelism allows different experts to reside on different compute nodes\([Rajbhandari et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib7);[Fedus et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib8)\), reducing per\-node memory requirements while enabling horizontal scaling across clusters\. This distributed nature has accelerated MoE adoption in federated learning scenarios\([Shen et al\., 2024a](https://arxiv.org/html/2610.41039#bib.bib9)\)and other distributed deployment contexts\. As examples, Mixtral 8×\\times7B\([Jiang et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib29)\)surpasses the performance of much larger dense models such as Llama 2 \(70B\) and GPT\-3\.5 \(175B\) on most benchmarks, while using only a fraction of the parameters during inference \(13B active out of 47B total\)\([Jiang et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib29)\)\. DeepSeek\-V2\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.41039#bib.bib30)\)is also renowned for its exceptional performance in highly technical domains, such as mathematics and code generation\.

However, the sparse activation patterns that make MoE architectures efficient also create novel attack vectors that exploit the specialized nature of expert routing\. Recent work has demonstrated various ways adversaries can compromise model security, ranging from degrading model performance by injecting malicious tokens that over\-occupy expert buffer capacity\([Hayes et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib25)\)to extracting user prompts by exploiting expert buffers in expert choice routing\([Zhou et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib33);[Yona et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib24)\)\. Most alarmingly, BadMoE\([Wang et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib26)\)demonstrates that attackers can inject backdoors into dormant experts, which are consistently under\-utilized for particular tasks, and optimize routing triggers to activate them during inference\. A critical prerequisite underlying these expert\-targeting attacks is identifying which experts to target\. Once adversaries know the dominant or dormant experts, they can manipulate them for performance degradation or stealthy backdoor implantation to steer the model toward generating malicious or harmful outputs\.

While identifying idle experts in attacks like BadMoE assumes knowledge of the target task and dataset, a more realistic and concerning threat emerges in practical distributed deployment scenarios\. Organizations routinely fine\-tune MoE models on proprietary datasets for domain specialization across distributed infrastructure\. However, model weights are often more accessible than training data through leaks, collaborations, or open\-source releases\. In distributed settings, routing information becomes inherently exposed: federated participants observe routing decisions during local training, edge devices handle routing computations before cloud offloading, and multi\-tenant platforms log routing statistics for resource management\. This distributed exposure of routing patterns creates a fundamentally different attack surface than centralized deployments\. When adversaries have access to deployed models and can observe their interactions, but lack knowledge of the fine\-tuning process, training data, and target tasks, it raises a critical question:*Can adversaries identify vulnerable experts that are the most or least utilized without any prior knowledge of the fine\-tuning data or procedures?*

In this paper, we answer this question affirmatively by introducing a novel attack that recovers the*internal expert pathways*in fine\-tuned MoE models\. Building on the well\-established understanding that large language models memorize training data during both pre\-training\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14);[Carlini et al\., 2023](https://arxiv.org/html/2610.41039#bib.bib12)\)and fine\-tuning\([Zeng et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib10);[Mireshghallah et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib11)\), and drawing from the key\-value memory interpretation of feed\-forward networks \(FFNs\)\([Geva et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib35)\), we hypothesize that fine\-tuning memorization manifests distinctively through expert utilization patterns in MoE blocks\. To the best of our knowledge, we are the first to introduce a new and important observation: that data memorization in MoE architectures extends beyond generated text to*structural memorization*\. We demonstrate that fine\-tuning on a specific corpus imprints a characteristic fingerprint on the model’s expert utilization patterns, where certain experts become consistently more active than others and more associated with memorizing the fine\-tuning data\. These memorized routing signatures remain accessible through ordinary model interactions, enabling adversaries to map the model’s expert landscape without any knowledge of the original training data\. In distributed deployments, this vulnerability becomes even more serious\. Adversaries can observe routing patterns at multiple points in the system, whether at federated aggregation points, edge\-cloud interfaces, or multi\-tenant monitoring systems\. By aggregating these observations, they can reconstruct fine\-tuning signatures even when each individual observation is incomplete\.

Our investigation reveals several key properties of these structural memorization patterns\. We show that expert utilization patterns form a stable*signature*characteristic of a particular dataset when accumulated over the fine\-tuning corpus, with this distinctive pattern emerging consistently even from just a few member samples\. The strength and detectability of these signatures depend critically on the fine\-tuning configuration\. We explore how different target linear layers in parameter\-efficient fine\-tuning,*i\.e\.*, those in attention blocks, those in FFNs, or both, create distinct routing signature profiles with varying degrees of structural memorization\. We find that attention\-involved fine\-tuning creates more pronounced and recoverable structural changes, while FFN\-only fine\-tuning produces more subtle modifications\.

These findings enable us to develop a practical recovery attack against fine\-tuned MoE models\. We demonstrate that adversaries with gray\-box access to routing and output logits can recover routing fingerprints from the fine\-tuned model\. Unlike traditional memorization studies that extract training data from generations\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14);[Kassem et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib16);[Nasr et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib13);[Zhang et al\., 2023](https://arxiv.org/html/2610.41039#bib.bib15)\), we investigate how memorization manifests in the internal expert utilization patterns during token processing\. While membership inference attacks \(MIAs\)\([Hu et al\., 2022b](https://arxiv.org/html/2610.41039#bib.bib18)\)are commonly used to identify whether model outputs correspond to memorized samples from the fine\-tuning corpus, we discover that existing MIA metrics\([Yeom et al\., 2018](https://arxiv.org/html/2610.41039#bib.bib21);[Shi et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib22);[Zhang et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib23);[Mireshghallah et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib11)\)fail to effectively capture the complex routing\-based memorization signatures in MoE architectures\. Through systematic experimentation, we find that hybrid metrics combining tail likelihood analysis with pre\-trained model calibration and text complexity measures significantly improve the detection of memorized routing structures\. These combinations are particularly effective at distinguishing fine\-tuning memorization from both pre\-training and same\-domain routing patterns\.

Through this methodology, we can profile memorized routing signatures and identify dominant experts for the fine\-tuning dataset using remarkably few samples\. We validate these findings across a diverse range of open\-source MoE LLMs, from small\-scale community models to large\-scale commercial architectures\. The ability to recover these routing fingerprints creates important security and privacy implications for distributed AI deployments\. First, our signature recovery approach serves as a validation framework for membership inference attacks\. Higher recovery accuracy directly reflects more effective MIA metrics that successfully filter member data, providing a novel evaluation methodology for assessing membership inference performance in MoE architectures\. Second, this capability enables adversaries to identify dominant or dormant experts for specific fine\-tuning tasks without accessing any actual training text\. Such knowledge facilitates targeted attacks like backdoor injection into dormant experts, as demonstrated in BadMoE, but under more realistic threat scenarios where attackers lack knowledge of the original training data\. Third, in distributed settings where routing information is exposed across multiple system components, our attack reveals how seemingly innocuous operational data can be exploited to breach data privacy guarantees, highlighting fundamental tensions between distributed system transparency and privacy protection\.

## 2\.Preliminaries

### 2\.1\.Mixture\-of\-Experts Large Language Models

The sparse MoE architecture has recently gained significant traction in the development of modern LLMs\. MoE decomposes a FFN layer within a dense model into multiple individual expert sub\-networks of the same size as the original layer\. For each token’s forward pass, a subset of experts is dynamically selected to be activated by a*routing mechanism*, creating sparse activation patterns that require the same memory capacity but much less compute compared to using all experts, or equivalently, a dense model of the same size\.[Table1](https://arxiv.org/html/2610.41039#S2.T1)details the specifications of the MoE models considered in this paper, includingTinyMistral\([M4\-ai, 2024](https://arxiv.org/html/2610.41039#bib.bib27)\),PhiMoE\([Abdin et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib28)\),OLMoE\([Muennighoff et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib31)\), andMixtral\([Jiang et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib29)\)\.

Table 1\.Comparison of MoE model architectures investigated in this paper\.Model\# ParamsMoE ShapeTop\-KHighlightsActive / TotalL×NL\\times NTinyMistral: M4\-ai/TinyMistral\-6x248M0\.5B / 1\.5B12×\\times62Small\-scale community\-built model\.OLMoE: allenai/OLMoE\-1B\-7B\-09241\.3B / 6\.9B16×\\times648Fully open\-source for MoE research\.PhiMoE: microsoft/Phi\-3\.5\-MoE\-instruct6\.6B / 41\.9B32×\\times162On par with Gemini\-1\.5\-Flash and GPT\-4o\-mini\.Mixtral: mistralai/Mixtral\-8x7B\-v0\.112\.9B / 46\.7B32×\\times82Matches Llama\-2\-70B performance\.The routing mechanism in MoE models typically maps each input token to a set ofKKselected experts\. Specifically, a gating functiong⁡\(⋅\)g\(\\cdot\), usually implemented as a learned linear layer, computes a score for each expert based on the token’s hidden representation\. These scores, referred to as*router logits*, are used to select the top\-KKexperts with the highest logit values\. A softmax function is then applied to these selected logits to produce normalized router weights, which represent the probability of selecting each expert among the chosen set\. The output of the MoE layer is computed as a weighted sum of the outputs from the selected expertsEl,iE\_\{l,i\}processing the token\. Formally, the output𝐨lt\\mathbf\{o\}^\{t\}\_\{l\}of MoE layerllis given by

\(1\)𝐨lt=∑i∈Top\-K​\(g⁡\(𝐡lt\)\)softmax​\(g⁡\(𝐡lt\)\)i⋅El,i​\(𝐡lt\),\\mathbf\{o\}^\{t\}\_\{l\}=\\sum\_\{i\\in\\text\{Top\-$K$\}\(g\(\\mathbf\{h\}^\{t\}\_\{l\}\)\)\}\\text\{softmax\}\(g\(\\mathbf\{h\}^\{t\}\_\{l\}\)\)\_\{i\}\\cdot E\_\{l,i\}\(\\mathbf\{h\}^\{t\}\_\{l\}\),where𝐡lt\\mathbf\{h\}^\{t\}\_\{l\}is the hidden state of tokenttentering MoE layerll, andEl,iE\_\{l,i\}denotes the expert sub\-network corresponding to expertiiin layerll\.

Previous work has shown that feed\-forward layers actually operate as neural memories\([Geva et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib35)\), where the first layer parameter matrix acts as keys and the second parameter matrix acts as values\. Applied to MoE models, the projection matrices of the experts can be regarded as keys and values as well\([Lo et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib34)\):

\(2\)El,i​\(𝐡lt\)=\(σS​i​L​U​\(𝐡lt​Wg​a​t​el,i\)⊙\(𝐡lt​Wu​pl,i\)\)​Wd​o​w​nl,i,E\_\{l,i\}\(\\mathbf\{h\}^\{t\}\_\{l\}\)=\(\\sigma\_\{SiLU\}\(\\mathbf\{h\}^\{t\}\_\{l\}W^\{l,i\}\_\{gate\}\)\\odot\(\\mathbf\{h\}^\{t\}\_\{l\}W^\{l,i\}\_\{up\}\)\)W^\{l,i\}\_\{down\},whereσS​i​L​U\\sigma\_\{SiLU\}denotes the SiLU activation function used in modern MoE models\. In this formulation, the gate and up projections jointly function as*keys*that determine which memory patterns to activate, withWg​a​t​el,iW^\{l,i\}\_\{gate\}controlling neuron\-level gating andWu​pl,iW^\{l,i\}\_\{up\}producing activation weights, while the down projectionWd​o​w​nl,iW^\{l,i\}\_\{down\}serves as*values*storing the actual output patterns\. This key\-value memory interpretation helps explain*structural memorization*in MoE models: fine\-tuning imprints characteristic expert utilization patterns that reflect the training data’s structure, creating distinctive routing signatures that persist during inference\.

![Refer to caption](https://arxiv.org/html/2610.41039v1/moe.png)Figure 1\.Illustration of a decoder layer in an MoE model, highlighting the routing mechanism and LoRA\-tunable linear layers\. \(![Refer to caption](https://arxiv.org/html/2610.41039v1/figures/fire.png)\): Trainable; \(![Refer to caption](https://arxiv.org/html/2610.41039v1/figures/snowflake.png)\): Frozen\.
### 2\.2\.Fine\-Tuning MoE LLMs

Fine\-tuning modern LLMs, including MoE architectures, is highly accessible through open\-source frameworks such as LLaMA\-Factory\([Zheng et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib36)\)and OpenRLHF\([Hu et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib37)\)\. These platforms support a comprehensive suite of algorithms and accommodate both full\-parameter fine\-tuning and parameter\-efficient approaches like LoRA\([Hu et al\., 2022a](https://arxiv.org/html/2610.41039#bib.bib38)\), which learns two low\-rank matricesAAandBBand adds their product to the original weights, and QLoRA\([Dettmers et al\., 2023](https://arxiv.org/html/2610.41039#bib.bib39)\), a LoRA variant with quantized weights that has become the*de facto*standard for efficient LLM fine\-tuning\. In this work, we employ the LLaMA\-Factory framework to fine\-tune MoE models using QLoRA with 4\-bit quantization\.

A common strategy for efficiently fine\-tuning large MoE models is to update only a subset of parameters, typically the linear layers within the attention and MoE blocks\. A key aspect of this approach is freezing the router, which, as recent work shows, increases stability by preserving sophisticated pre\-trained routing policies that could otherwise be catastrophically disrupted on smaller, specialized datasets\([Shen et al\., 2024b](https://arxiv.org/html/2610.41039#bib.bib32)\)\. However, while this general practice is established, the literature lacks clear understanding of how parameter\-efficient fine\-tuning affects structural memorization when applied to specific linear layer configurations\. These LoRA target modules include attention blocks alone with query, key, value, and output projections, MoE blocks alone containing feed\-forward networks with gate, up, and down projections \(also known as w1, w2, w3 in models likeMixtral\) or both blocks combined in each decoder layer\. We refer to these three target module sets asAttn,FFN, andBothrespectively\. As a wrap\-up,[Fig\.1](https://arxiv.org/html/2610.41039#S2.F1)illustrates a decoder layer in an MoE LLM, highlighting the routing mechanism and LoRA\-tunable linear layers within these blocks\.

### 2\.3\.Memorization and Membership Inference in Fine\-Tuned LLMs

Memorization in LLMs is a well\-known phenomenon that enables data extraction attacks\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14);[Nasr et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib13);[Kassem et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib16);[Zhang et al\., 2023](https://arxiv.org/html/2610.41039#bib.bib15)\)\. One category of extraction is based on “extractable memorization,” which attempts to elicit training data by constructing specific prompts, such as short strings sampled from the public Internet\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14)\)or prompts iteratively refined by another LLM\([Kassem et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib16)\)\. However, this approach is often limited by the challenges of effective prompt design and verifying if the output is a true memorized sample\. A more direct approach, termed “discoverable memorization” by\([Carlini et al\., 2023](https://arxiv.org/html/2610.41039#bib.bib12);[Nasr et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib13)\), explicitly prompts the model with prefixes taken from the training dataset to determine if the model’s generated text reveals the corresponding suffix\. While easier to design and evaluate, and substantially more effective\([Nasr et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib13)\), this approach requires adversaries to have access to the model’s training dataset to compose prefixes of sufficient length for extraction\. Although these studies have been extended to the fine\-tuning setting\([Zeng et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib10);[Mireshghallah et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib11);[Hou et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib17)\), the unique memorization behaviors of MoE models remain not well understood\. In particular, how memorization manifests in the internal routing patterns of fine\-tuned MoE models remains an open question\.

Frequently used as a proxy for measuring memorization, MIAs are used to determine whether a generated sample is included in a model’s training dataset by exploiting differences in model prediction behavior\. Existing MIAs adopted by the literature for text memorization evaluation span from reference\-free metrics such as LOSS\([Yeom et al\., 2018](https://arxiv.org/html/2610.41039#bib.bib21)\), Min\-K%\([Shi et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib22)\), and Min\-K%\+\+\([Zhang et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib23)\)that achieve state\-of\-the\-art results, to reference\-based metrics that compare likelihood differences with reference LLMs\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14);[Carlini et al\., 2023](https://arxiv.org/html/2610.41039#bib.bib12)\), particularly using pre\-trained models as references\([Mireshghallah et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib11)\)\. These existing approaches do not require training auxiliary models, making them computationally efficient for our setting\. In contrast, recent reference\-based MIA methods for fine\-tuned LLMs train dedicated auxiliary models on constructed datasets\([Fu et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib19);[Huang et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib20)\), which is prohibitively expensive for efficiently filtering members from generated samples and thus not considered in our work\. However, as we discover, even the training\-free baseline metrics are sometimes not effective enough to identify samples that correspond to memorized expert pathways from the repeated unconditional generation process needed to capture the*structural signatures*embedded in MoE routing patterns\. We detail the specific formulations for these baselines in[Section4](https://arxiv.org/html/2610.41039#S4), where they are evaluated against our new metrics for recovering memorized expert pathways in MoE LLMs\.

## 3\.Impact of Fine\-Tuning on Expert Utilization

We begin by examining the internal routing patterns and expert utilization of MoE models, which constitute key components of structural memorization\. We first define expert utilization and dominant experts for our analysis, then demonstrate that fine\-tuning datasets leave distinctive fingerprints on the model through expert utilization signatures — stable and robust patterns characteristic of both data distribution and fine\-tuning modules\. Across all experiments, we train the larger models \(MixtralandPhiMoE\) on an NVIDIA GH200 Grace Hopper Superchip with 80GB memory, while the smaller models \(OLMoEandTinyMistral\) utilize an NVIDIA GeForce RTX 4090 with 24GB memory\. For consistency across all experiments, we use 4\-bit quantization and a LoRA configuration with rank88, alpha1616, and dropout0\.10\.1, along with a batch size of1616\.

### 3\.1\.Expert Utilization and Specialization

To quantify the internal routing patterns of MoE models, we extend the concept of domain specialization\([Muennighoff et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib31)\)to define expert utilization: the proportion of tokens routed to a particular expert during inference\. We measure this by tracking which of the top\-KKexperts are selected by the router for each token and aggregating these selections\.

###### Definition 0 \(Expert Utilization\)\.

For an MoE modelMMwithLLexpert layers andNNexperts per layer, theExpert Utilization MatrixUM∈ℝL×NU\_\{M\}\\in\\mathbb\{R\}^\{L\\times N\}is a matrix where each elementul,iu\_\{l,i\}represents the proportion of a set ofTTtotal tokens that are forwarded to expertiiat layerll\.

\(3\)ul,i=1T​∑t=1T𝟙​\(El,i​is selected for token​t\),u\_\{l,i\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbbm\{1\}\(E\_\{l,i\}\\text\{ is selected for token \}t\),where𝟙​\(⋅\)\\mathbbm\{1\}\(\\cdot\)is the indicator function\.

Applying this definition to a datasetDDyields a corpus\-specific utilization matrix,UM​\(D\)U\_\{M\}\(D\), which aggregates the expert selections for all tokens within that corpus\. Analyzing this matrix reveals the model’s learned specialization by identifying which experts are consistently utilized for that particular domain or task\. Furthermore, we define dominant experts as the subset of most frequently utilized experts within each layer\.

###### Definition 0 \(Dominant Expert\)\.

Letul,iu\_\{l,i\}be the utilization of expertiiat layerll, andϕ∈\[0,1\]\\phi\\in\[0,1\]be the threshold\. An expertiiis consideredϕ\\phi\-dominant if its utilization satisfies:

ul,i\>ϕ×maxi′∈\{1,…,N\}⁡ul,i′\.u\_\{l,i\}\>\\phi\\times\\max\_\{i^\{\\prime\}\\in\\\{1,\\dots,N\\\}\}u\_\{l,i^\{\\prime\}\}\.

Figure 2\.Dominant experts for different data domains, whose utilization is greater than 80% of the maximum utilization within each layer \(ϕ=0\.8\\phi=0\.8\)\.[Fig\.2](https://arxiv.org/html/2610.41039#S3.F2)visualizes the expert specialization\([Muennighoff et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib31)\)ofOLMoEandPhiMoE, by plotting the dominant experts as defined above for several distinct data domains, including C4, Books, Wikipedia, arXiv, and GitHub, investigated in\([Muennighoff et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib31)\)\. The figure reveals that expert allocation is highly domain\-specific, with minimal overlap between unrelated domains across layers\. For instance, inOLMoE, the set of dominant experts for Books is largely separate from that of GitHub\. The most significant overlap occurs between domains that share structural or syntactic similarities\. For example, arXiv and GitHub share several dominant experts in multiple layers, likely because both scientific papers and source code are rich in highly structured text, such as mathematical formulations and formal syntax\. Furthermore, the concentration of dominance can vary significantly by domain\. General domains like C4\([Raffel et al\., 2020](https://arxiv.org/html/2610.41039#bib.bib40)\), a web crawl containing diverse data, exhibit a sparse and distributed pattern of dominance, with some layers activating up to eight different experts out of 64\. In contrast, specialized domains like arXiv show a much more concentrated pattern, often relying on only one to three dominant experts per layer, indicating a more focused computational pathway\. The two MoE architectures also exhibit distinct dominance patterns\.OLMoE, with 64 experts per layer, shows relatively concentrated expert usage with sparser activation patterns\.PhiMoE, with only 16 experts per layer, displays more distributed dominance, as evidenced by the higher density of activated experts in the figure\. This is a natural consequence of its smaller expert pool requiring broader utilization to cover diverse domain characteristics\.

### 3\.2\.Stability of Expert Utilization Signatures

The stability of expert utilization signatures is essential for the feasibility of our recovery attack, as it ensures that the signatures we recover through unconditional generation represent genuine model fingerprints rather than sampling artifacts\. To investigate this stability, we compute the expert utilization matrixUM​\(Dk\)U\_\{M\}\(D\_\{k\}\)by processing all tokens from random subsetsDkD\_\{k\}ofkksamples, using a batch size of20482048\. We then assess convergence of the utilization pattern by measuring the Pearson correlation between eachUM​\(Dk\)U\_\{M\}\(D\_\{k\}\)and a reference signatureUM​\(Dkmax\)U\_\{M\}\(D\_\{k\_\{\\max\}\}\)derived from a sufficiently large subset of sizekmaxk\_\{\\max\}\.[Fig\.3](https://arxiv.org/html/2610.41039#S3.F3)shows the results across all MoE models, fine\-tuned with all linear layers and two corresponding fine\-tuning datasets: CodeAlpaca111https://huggingface\.co/datasets/codealpaca/CodeAlpaca\-20K, a specialized dataset of 20K code generation samples with focused distribution, and MedGuide222https://huggingface\.co/datasets/epfl\-llm/guidelines, an integrated dataset of 37K clinical practice guidelines from nine high\-quality online medical sources across different medical domains and geographical locations\.

Figure 3\.Correlation between expert utilization signatures with increasing sample sizes\. Each sample in CodeAlpaca has an average length of8989tokens, and each sample in MedGuide has an average length of29392939tokens\.The robustness of routing signatures varies significantly based on model characteristics, with model scale emerging as the primary factor\. For the same CodeAlpaca dataset,Mixtralrequires around10001000samples to achieve a stable signature, while the other three models produce reliable signatures with fewer than100100samples, a10×10\\timesdifference in data requirements\. This suggests that larger models exhibit more complex routing behaviors that require more extensive sampling to accurately capture their true utilization patterns\. The underlying expert utilization signature remains consistent, but larger models need more observations to overcome the natural variance in routing decisions during inference\. Interestingly, specific MoE architectural differences, such as the number of expert layers, experts per layer, or top\-KKpolicy, have negligible impact on signature stability compared to overall model scale\.

The nature of the training data profoundly impacts signature robustness\. The more homogeneous CodeAlpaca dataset shows smoother convergence and consistent upward trends compared to the heterogeneous MedGuide corpus, which exhibits much higher variance in correlation scores, with some points dropping significantly even between100100and10001000samples\. This makes intuitive sense, as focused datasets with consistent patterns should yield more stable signatures\. Remarkably, this occurs despite the fact that MedGuide samples are about33×33\\timeslonger than those in CodeAlpaca on average, providing substantially more tokens per sample for signature formation\. This demonstrates that data consistency matters more than token density for creating robust signature patterns\.

All combinations plateau beyond10001000samples at very high correlations \(≥0\.98\\geq 0\.98\), demonstrating that expert utilization signatures stabilize robustly with remarkably few samples,*i\.e\.*, less than5%5\\%of the total dataset size for CodeAlpaca and less than3%3\\%for MedGuide, representing a consistent characteristic of how the model processes data from a particular distribution\. This efficiency provides crucial foundational support for our signature recovery methodology, as it ensures that adversaries can reliably extract meaningful routing fingerprints with minimal data collection and consistent results\.

\(a\)Mixtral& CodeAlpaca\(b\)OLMoE& MedGuide![Refer to caption](https://arxiv.org/html/2610.41039v1/similarity_matrix_mixtral_codealpaca_pearson.png)\(c\)Mixtral& CodeAlpaca
![Refer to caption](https://arxiv.org/html/2610.41039v1/similarity_matrix_olmoe_guidelines_2000_iters2594_pearson.png)\(d\)OLMoE& MedGuide

Figure 4\.Expert utilization signature changes when fine\-tuning different LoRA target modules\. The upper panel shows the magnitude of expert utilization differences before and after fine\-tuning specific target modules across all layers\. The lower panel shows the similarity matrix of expert utilization signatures between models fine\-tuned with different target modules, where “None” represents the pre\-trained model\.
### 3\.3\.Impact of Fine\-tuning Target Modules

To understand how fine\-tuning different LoRA trainable modules affect expert utilization patterns, we look into the expert utilization changes before and after fine\-tuning\. A key finding emerges: the same training data creates subtly but consistently different expert utilization fingerprints depending on which modules are trained\. Here, we present results from fine\-tuningMixtralon CodeAlpaca in the left panel andOLMoEon MedGuide in the right panel of[Fig\.4](https://arxiv.org/html/2610.41039#S3.F4), both trained for one epoch using supervised fine\-tuning with a learning rate of5​e−55e^\{\-5\}\.

As shown in[Figs\.4\(a\)](https://arxiv.org/html/2610.41039#S3.F4.sf1)and[4\(b\)](https://arxiv.org/html/2610.41039#S3.F4.sf2), the spatial distribution in the expert utilization patterns is particularly telling: when attention layers participate in fine\-tuning \(AttnorBoth\), changes concentrate dramatically in early and final layers, specifically the first 4 layers and last 8 layers inMixtraland first 3 layers and last 5 layers inOLMoE, creating a distinctive U\-shaped signature\. In contrast, FFN\-only fine\-tuning distributes changes more evenly across layers with a slight bias toward later layers, while consistently producing the most conservative routing changes compared to attention\-involved configurations\.

The similarity matrix in[Figs\.4\(c\)](https://arxiv.org/html/2610.41039#S3.F4.sf3)and[4\(d\)](https://arxiv.org/html/2610.41039#S3.F4.sf4)quantifies how these different approaches relate to each other\. Across both models, the pre\-trained model shows noticeably lower correlations with all fine\-tuned versions compared to the correlations among fine\-tuned models themselves\. Among the fine\-tuning approaches,Attnfine\-tuning produces the most distinct signatures, showing the lowest correlation with the pre\-trained model, whileFFNfine\-tuning maintains the highest similarity to the pre\-trained model, consistent with the spatial patterns observed above\. Comparing between architectures,OLMoEexhibits larger separation between different fine\-tuning configurations thanMixtral\.

This hierarchy aligns with the key\-value memory interpretation that we extended to MoE architectures in[Section2\.1](https://arxiv.org/html/2610.41039#S2.SS1)\. When attention layers are modified, they alter the hidden representations feeding into expert keys, fundamentally changing routing decisions throughout the network\. This cascading effect explains why attention modifications create more dramatic routing changes: each altered hidden state propagates through subsequent layers, amplifying routing disruptions as information flows through the network\. FFN modifications, while directly changing the values stored in expert memories, have minimal impact on the routing logic itself, resulting in more localized effects\. Consequently, attention\-based methods produce the most distinctive routing patterns\.

## 4\.Data\-Free Profiling of Expert Utilization Signature

Having established that expert utilization signatures are both consistent and characteristic of fine\-tuned MoE models, we now demonstrate how adversaries can exploit these properties to recover routing fingerprints through access to routing and output logits\. This creates a unique attack surface: while previous work has focused on extracting memorized text, we pivot from analyzing*what*the model generates to analyzing*how*it generates\. We demonstrate that an adversary can exploit the model’s “idle state” to recover routing fingerprints without any knowledge of the fine\-tuning data\. By examining how experts are utilized during confident, unconditional generation, we demonstrate that the resulting routing patterns closely reflect those observed in the original fine\-tuning corpus\. This means that even without access to training data or public datasets, an adversary can recover a high\-fidelity proxy for the routing signature left by fine\-tuning, simply by observing the model’s unconditional generation behavior — transforming the model’s own inference process into a side\-channel for privacy attacks\.

![Refer to caption](https://arxiv.org/html/2610.41039v1/attack.png)Figure 5\.Data\-free recovery of a fine\-tuned MoE model’s expert utilization signature: an illustration\.Table 2\.The membership inference metrics used to filter the candidate expert utilization patterns from the generated samples\.MetricFormulationLOSS\([Yeom et al\., 2018](https://arxiv.org/html/2610.41039#bib.bib21)\)L⁡\(x,Mf​t\)L\(x,M\_\{ft\}\)The negative average log\-likelihood of the sampleLower\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14)\)L\(x\.lower\(\),Mf​t\)L⁡\(x,Mf​t\)\\frac\{L\(x\.lower\(\),M\_\{ft\}\)\}\{L\(x,M\_\{ft\}\)\}The ratio of loss on the original vs\. lowercased sampleRef\([Mireshghallah et al\., 2022](https://arxiv.org/html/2610.41039#bib.bib11)\)L⁡\(x,Mp​r​e\)L⁡\(x,Mf​t\)\\frac\{L\(x,M\_\{pre\}\)\}\{L\(x,M\_\{ft\}\)\}The ratio of loss on the fine\-tuned vs\. pre\-trained modelZlib\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14)\)z​l​i​b​\(x\)L⁡\(x,Mf​t\)\\frac\{zlib\(x\)\}\{L\(x,M\_\{ft\}\)\}The sample’s complexity normalized by its loss on the fine\-tuned modelMin\-K%1\|min\-​k%\|​∑xi∈min\-​k%log⁡p⁡\(xi\|x1,x2,…,xi−1\)\\frac\{1\}\{\|\\text\{min\-\}k\\%\|\}\\sum\\limits\_\{x\_\{i\}\\in\\text\{min\-\}k\\%\}\\log p\(x\_\{i\}\|x\_\{1\},x\_\{2\},\\ldots,x\_\{i\-1\}\)\([Shi et al\., 2024](https://arxiv.org/html/2610.41039#bib.bib22)\)The average log\-likelihood of the lowestk%k\\%token probabilitiesMin\-K%\+\+1\|min\-​k%\|​∑\(x<t,xt\)∈min\-​k%Min\-K%\+\+token seq\.​\(x<t,xt\)\\frac\{1\}\{\|\\text\{min\-\}k\\%\|\}\\sum\\limits\_\{\(x\_\{<t\},x\_\{t\}\)\\in\\text\{min\-\}k\\%\}\\text\{Min\-K\\%\+\+\}\_\{\\text\{token seq\.\}\}\(x\_\{<t\},x\_\{t\}\),\([Zhang et al\., 2025](https://arxiv.org/html/2610.41039#bib.bib23)\)Min\-K%\+\+token seq\.​\(x<t,xt\)=log⁡p⁡\(xt\|x<t\)−μ⋅\|x<tσ⋅\|x<t\\text\{Min\-K\\%\+\+\}\_\{\\text\{token seq\.\}\}\(x\_\{<t\},x\_\{t\}\)=\\frac\{\\log p\(x\_\{t\}\|x\_\{<t\}\)\-\\mu\_\{\\cdot\|x\_\{<t\}\}\}\{\\sigma\_\{\\cdot\|x\_\{<t\}\}\}The average log\-likelihood of the lowestk%k\\%token probabilities, normalized by the vocabulary distribution\(a\)TinyMistral& CodeAlpaca\(b\)OLMoE& CodeAlpaca
Figure 6\.Comparison of recovered vs\. ground\-truth expert utilization matrices, filtered using various membership metrics and evaluated through matrix similarity and F1 scores of dominant experts\. Results are ranked by descending cosine similarity between recovered and ground\-truth matrices\. \(⧫\\blacklozenge\): Comparisons against the pre\-trained model’s signature on identical fine\-tuning data; \(⧫\\blacklozenge\): Comparisons against the same fine\-tuned model’s signature on GitHub data\. These two baselines should theoretically produce lower similarity values\.### 4\.1\.Recovering the Expert Utilization Signature

Following the approach in\([Carlini et al\., 2021](https://arxiv.org/html/2610.41039#bib.bib14)\)for extracting memorized training data from GPT\-2, we prompt the MoE model with a single special end\-of\-sequence token \(*e\.g\.*,<\|endoftext\|\>forOLMoEor</s\>forMixtral\) to indicate the end of a text sequence, then allow the model to generate512512tokens using top\-nnsampling withn=40n=40\. This produces sequences that are highly likely according to the model’s own likelihood estimates\. We repeat this process50005000times and compute the expert utilization for the generated tokens in each sample\.

The critical challenge lies in identifying which generated samples best reflect the fine\-tuning signature rather than pre\-training patterns or same\-domain patterns, while having zero knowledge of the fine\-tuning data\. To best approximate the signature, we need to filter the most representative expert utilization patterns from the generated samples that best capture the memorization of fine\-tuning data\. We adapt various MIA metrics from the literature to the fine\-tuning scenario, as summarized in[Table2](https://arxiv.org/html/2610.41039#S4.T2)\. However, our analysis reveals fundamental limitations in these existing approaches: they either fail to differentiate fine\-tuning memorization from pre\-training patterns \(*e\.g\.*, LOSS, Lower, Zlib, Min\-K%, and Min\-K%\+\+\), or cannot effectively filter out trivial high\-likelihood tokens that dominate unconditional generation \(*e\.g\.*, Ref\)\. To address this limitation, we devise three complementary hybrid metrics that synergistically combine the strengths of existing approaches, though their performance does not follow the anticipated sequence as our later analysis reveals:

- •Baseline hybrid combining compression\-based complexity with pre\-trained model likelihood ratios\. Zlib\-Ref​\(x\)=z​l​i​b​\(x\)L⁡\(x,Mf​t\)×L⁡\(x,Mp​r​e\)L⁡\(x,Mf​t\)\\textbf\{Zlib\-Ref\}\(x\)=\\frac\{zlib\(x\)\}\{L\(x,M\_\{ft\}\)\}\\times\\frac\{L\(x,M\_\{pre\}\)\}\{L\(x,M\_\{ft\}\)\}
- •Enhanced version combining compression and pre\-trained model calibration with low\-probability tail token filtering\. Zlib\-Ref\-Min\-K%\(x\)=z​l​i​b​\(x\)L⁡\(x,Mf​t\)×\\displaystyle\\textbf\{Zlib\-Ref\-Min\-K\\%\}\(x\)=\\frac\{zlib\(x\)\}\{L\(x,M\_\{ft\}\)\}\\times\(Min\-K%​\(x,Mf​t\)−Min\-K%​\(x,Mp​r​e\)\)\\displaystyle\(\\text\{Min\-K\\%\}\(x,M\_\{ft\}\)\-\\text\{Min\-K\\%\}\(x,M\_\{pre\}\)\)
- •Vocabulary\-normalized version combining compression and pre\-trained model calibration with vocabulary\-relative token probability scoring\. Zlib\-Ref\-Min\-K%\+\+\(x\)=z​l​i​b​\(x\)L⁡\(x,Mf​t\)×\\displaystyle\\textbf\{Zlib\-Ref\-Min\-K\\%\+\+\}\(x\)=\\frac\{zlib\(x\)\}\{L\(x,M\_\{ft\}\)\}\\times\(Min\-K%\+\+​\(x,Mf​t\)−Min\-K%\+\+​\(x,Mp​r​e\)\)\\displaystyle\(\\text\{Min\-K\\%\+\+\}\(x,M\_\{ft\}\)\-\\text\{Min\-K\\%\+\+\}\(x,M\_\{pre\}\)\)

After ranking the generated samples with one of these metrics, we filter the topNm​e​mN\_\{mem\}samples with the highest scores and aggregate the expert utilization matrices from forwarding the newly generated512512tokens in each sample to reconstruct the routing signature and extract dominant experts\. This aggregation transforms scattered routing observations into a coherent fingerprint that mirrors the original fine\-tuning data distribution\. The complete routing signature recovery process is illustrated in[Fig\.5](https://arxiv.org/html/2610.41039#S4.F5)\.

![Refer to caption](https://arxiv.org/html/2610.41039v1/heatmap_diff_CodeAlpaca-20k_1000.png)\(a\)TinyMistral& CodeAlpaca
![Refer to caption](https://arxiv.org/html/2610.41039v1/heatmap_diff_codealpaca.png)\(b\)OLMoE& CodeAlpaca
\(c\)TinyMistral\(d\)OLMoE
Figure 7\.Comparison of model’s routing signatures\. Subfigures \(a\) and \(b\) show the difference in the model’s expert utilization matrices on the fine\-tuning corpus before and after fine\-tuning; subfigures \(c\) and \(d\) show the correlation between the fine\-tuned model’s expert utilization matrices on fine\-tuning corpus and same\-domain corpus\.Table 3\.Matrix cosine similarity, Pearson correlation, and F1 scores of dominant experts between recovered and ground\-truth expert utilization forMixtralon CodeAlpaca, with best and second\-best results highlighted in bold and underlined, respectively\.ModulesMetricLOSSLowerRefZlibMin\-K%Min\-K%\+\+Zlib\-RefZlib\-Ref\-Min\-K%Zlib\-Ref\-Min\-K%\+\+BothCosSim0\.9170\.8080\.7110\.9880\.9250\.9160\.9860\.9010\.909Pearson0\.193\-0\.094\-0\.0620\.4740\.1770\.2070\.449\-0\.004\-0\.045F10\.4640\.3380\.3270\.8770\.4650\.4520\.8500\.4010\.393AttnCosSim0\.7370\.8570\.7710\.9820\.7400\.7440\.9840\.9840\.972Pearson0\.002\-0\.0740\.0050\.413\-0\.0050\.0020\.5010\.5580\.333F10\.3500\.3500\.2870\.7860\.3530\.3640\.7870\.7860\.586FFNCosSim0\.9310\.9380\.7850\.9760\.9430\.9370\.9790\.9360\.960Pearson0\.181\-0\.093\-0\.0080\.2020\.2240\.0930\.2580\.1520\.127F10\.5420\.4440\.3640\.7170\.5660\.5280\.7550\.4320\.522
### 4\.2\.Experimental Results

To validate the effectiveness of the recovery under, we conduct a comprehensive evaluation across multuple model architectures\.[Fig\.6](https://arxiv.org/html/2610.41039#S4.F6), onTinyMistralandOLMoE, compares the recovered signature against the ground\-truth signature of the fine\-tuned model on the fine\-tuning corpus, alongside two critical baseline signatures: the pre\-trained model’s signature on the same corpus used for fine\-tuning and the same fine\-tuned model’s signature on a different corpus within the same domain\. To further analyze these results, we provide additional visualizations that illuminate the underlying patterns, including the differences in expert utilization matrices between the pre\-trained and fine\-tuned models in heatmaps[Figs\.7\(a\)](https://arxiv.org/html/2610.41039#S4.F7.sf1)and[7\(b\)](https://arxiv.org/html/2610.41039#S4.F7.sf2), and the correlation between the expert utilization matrices in the fine\-tuning corpus and the same\-domain corpus in[Figs\.7\(c\)](https://arxiv.org/html/2610.41039#S4.F7.sf3)and[7\(d\)](https://arxiv.org/html/2610.41039#S4.F7.sf4)\. Another set of results regarding different fine\-tuning target modules on more models are provided in[Tables3](https://arxiv.org/html/2610.41039#S4.T3)and[8](https://arxiv.org/html/2610.41039#S4.F8)\.

We use three metrics to evaluate the similarity between the recovered signature and the ground\-truth signature: cosine similarity that measures the angular similarity between two flattened matrix vectors, Pearson correlation that measures linear co\-variation between corresponding expert utilization values, and F1 scores ofϕ\\phi\-dominant \(ϕ=0\.8\\phi=0\.8\) experts that evaluate the accuracy of identifying which specific experts were most important in the model for the fine\-tuning data\. The K values in Min\-K% and Min\-K%\+\+ shown in the figures represent the optimal threshold percentages selected from extensive testing across a range of 5% to 60%\. We setNm​e​mN\_\{mem\}to only 10, which is a total of51205120forwarded tokens, roughly equivalent to the context of 50 samples in CodeAlpaca, based on our preliminary findings in[Section3\.2](https://arxiv.org/html/2610.41039#S3.SS2)\.

Memorized fine\-tuning signatures can be effectively recovered when using appropriate MIA metrics for filtering\.The main blue bars in[Fig\.6](https://arxiv.org/html/2610.41039#S4.F6)reveal remarkably high cosine similarity scores and Pearson correlations between recovered and ground\-truth expert utilization signatures when filtered with specific MIA metrics, with our hybrid metrics Zlib\-Ref\-Min\-K% and Zlib\-Ref leading the pack\. This success indicates that combining compression\-based detection with pre\-trained model references creates synergistic effects that enhance membership inference capabilities during fine\-tuning analysis\. Interestingly, Zlib\-Ref\-Min\-K%\+\+ delivers mediocre or even the worst performance despite Min\-K%\+\+ being an improved variant of Min\-K% when used individually\. However, LOSS, Lower, and Ref consistently underperform across both models\. This degraded performance stems from their shared reliance on sample\-level averaging of negative log\-likelihood, which suffers from high false positive rates because trivial elements like common words and repetitive phrases can also yield high likelihood, obscuring the meaningful patterns in the generated samples that characterize fine\-tuning signatures\.

The F1 scores for dominant experts serve as particularly discriminative indicators of MIA metric effectiveness\. These scores exhibit greater variance across metrics because they focus on the subset of experts most affected by fine\-tuning — the primary locus where learned specializations become apparent in routing patterns — making them sensitive markers of signature recovery quality\. Unlike broader similarity measures that consider all experts equally, F1 scores can reveal that metrics with comparable cosine similarity show dramatically different performance\.

Figure 8\.Upper panel:OLMoEon MedGuide; Lower panel:PhiMoEon CodeAlpaca\. From left to right:Both,Attn,FFN\.Signatures in larger, more complex MoEs are harder to recover but create more distinctive fingerprints\.Recovery performance exhibits a clear relationship with model complexity\. Near\-complete signature recovery proves more achievable for the simplerTinyMistralcompared to the more complexOLMoEarchitecture\. This disparity can be attributed toOLMoE’s substantially larger expert pool \(14×14\\timesmore experts\) and its higher top\-K routing policy, which creates more intricate activation patterns that are inherently harder to reconstruct from observations of the model’s routing and output logits\.

WhileOLMoE’s fine\-tuning process induces more diverse and dramatic changes in expert utilization across layers,TinyMistral’s simpler 2\-out\-of\-6 expert selection mechanism creates inherent limitations in routing discrimination\. Certain experts \(notably experts 1 and 4\) maintain consistent activation across most layers, causing fine\-tuning specialization to concentrate on these already\-dominant experts while leaving others relatively unchanged, as demonstrated in[Figs\.7\(a\)](https://arxiv.org/html/2610.41039#S4.F7.sf1)and[7\(b\)](https://arxiv.org/html/2610.41039#S4.F7.sf2)\. This architectural constraint also results in less discriminative routing patterns between pre\-trained and fine\-tuned states — a limitation that arises from the model design rather than MIA metric failures, as evidenced by the consistently small differences between the blue and red bars across various metrics\.

Several metrics, including Min\-K%\+\+, Lower, and Ref, demonstrate problematic behavior by incorrectly identifying dominant experts, paradoxically yielding higher F1 scores on pre\-trained signatures than on fine\-tuned ones\. This occurs because these metrics’ approximated expert utilization matrices predominantly capture experts that were already dominant in the pre\-trained model, rather than those newly specialized through fine\-tuning\. This highlights the limitations of these metrics in capturing the nuanced changes induced by fine\-tuning, particularly in simpler MoE architectures likeTinyMistral\.

Signature recovery may yield misleading intelligence when MoEs exhibit similar routing patterns across same\-domain datasets\.A counterintuitive finding emerges when comparing blue and yellow bars: numerous MIA metrics produce higher similarity scores for same\-domain data than for actual fine\-tuning data\. This unexpected result occurs because routing patterns in same\-domain datasets closely mirror those in fine\-tuning data, creating high correlations between their respective signatures as demonstrated in[Figs\.7\(c\)](https://arxiv.org/html/2610.41039#S4.F7.sf3)and[7\(d\)](https://arxiv.org/html/2610.41039#S4.F7.sf4)\.

This phenomenon presents significant implications for adversarial applications\. While recovered signatures might successfully identify the domain of fine\-tuning data, they could simultaneously mislead toward different datasets within the same domain, generating false positives in fine\-tuning data identification\. An adversary might correctly infer that a model was fine\-tuned on code\-related content but incorrectly attribute the training to a different coding dataset entirely\. Nevertheless, Zlib\-Ref\-Min\-K% and Zlib\-Ref consistently outperform other metrics by minimizing these misleading effects and providing more reliable signature recovery\.

Pearson correlation presents a more stringent challenge for large\-scale MoE models\.The results in[Table3](https://arxiv.org/html/2610.41039#S4.T3)show a clear winner: Zlib\-based metrics such as pure Zlib, Zlib\-Ref, and Zlib\-Ref\-Min\-K% outperform other approaches\. What’s particularly striking is that cosine similarity stays reliably above 0\.9 for these compression\-based methods, while Pearson correlations struggle significantly, often dropping to near\-zero or negative values with simpler metrics like LOSS, Lower, and Ref\. The hybrid Zlib\-Ref combinations prove especially effective, indicating that pairing compression\-based detection with reference model comparisons creates a more robust signature recovery approach\. This disparity suggests that signature recovery effectiveness drops significantly in large\-scale MoE models when measured by Pearson correlation, especially when comparing to results on smaller models likeTinyMistralandOLMoE\.

Attention\-involved fine\-tuning creates more detectable structural memorization\.The choice of LoRA target modules makes a substantial difference in how well signatures can be recovered\. As further visualized in[Fig\.8](https://arxiv.org/html/2610.41039#S4.F8)across more models and datasets, fine\-tuningBothorAttndelivers stronger performance, whileFFN\-only fine\-tuning consistently struggles with Pearson correlation of utilization matrices and F1 scores of dominant experts\. This finding reinforces our earlier observations in[Section3\.3](https://arxiv.org/html/2610.41039#S3.SS3): when attention blocks participate in fine\-tuning, they create more pronounced structural changes that adversaries can exploit, whereas isolated FFN training leaves behind subtler fingerprints that prove much harder to detect and recover\.

## 5\.Concluding Remarks

This work reveals a hidden vulnerability in MoE architectures: expert utilization patterns become persistent fingerprints of training data that adversaries can extract through ordinary model interactions, requiring no access to training data or model parameters\. Our signature recovery approach enables adversaries to map expert utilization patterns and identify vulnerable experts for targeted manipulation, including degrading downstream performance through dominant expert disruption, hijacking model behavior through backdoor injection into dormant experts, and enhancing data extraction by exploiting expert specialization\. Beyond demonstrating the attack, our hybrid metrics provide a diagnostic framework for evaluating membership inference effectiveness in MoE models, revealing that existing metrics fail to capture routing\-based memorization\. In distributed deployments, this vulnerability is amplified as routing information becomes exposed at multiple trust boundaries, enabling adversaries to aggregate partial observations into complete signatures\. Our findings underscore that routing\-level privacy must be considered a first\-class concern in distributed AI system design, as sparse activation mechanisms create unintended side channels that illuminate which experts process sensitive training data\.

## References

- Abdinet al\.\(2024\)M\. Abdin, J\. Aneja, H\. Awadalla, A\. Awadallah, A\. A\. Awan,et al\.Phi\-3 Technical Report: A Highly Capable Language Model Locally on Your Phone\.arXiv preprint arXiv:2404\.14219\.Cited by:[§2\.1](https://arxiv.org/html/2610.41039#S2.SS1.p1.1)\.
- Carliniet al\.\(2023\)N\. Carlini, D\. Ippolito, M\. Jagielski, K\. Lee, F\. Tramer, and C\. ZhangQuantifying Memorization Across Neural Language Models\.InProc\. the Eleventh International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p5.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1)\.
- Carliniet al\.\(2021\)N\. Carlini, F\. Tramèr, E\. Wallace, M\. Jagielski, A\. Herbert\-Voss, K\. Lee, A\. Roberts, T\. Brown, D\. Song, Ú\. Erlingsson, A\. Oprea, and C\. RaffelExtracting Training Data from Large Language Models\.InProc\. the 30th USENIX Security Symposium \(USENIX Security\),pp\. 2633–2650\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p5.1),[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1),[§4\.1](https://arxiv.org/html/2610.41039#S4.SS1.p1.1),[Table 2](https://arxiv.org/html/2610.41039#S4.T2.5.4.1.1.1),[Table 2](https://arxiv.org/html/2610.41039#S4.T2.5.8.1.1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-V2: A Strong, Economical, and Efficient Mixture\-of\-Experts Language Model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p2.1)\.
- Denget al\.\(2020\)S\. Deng, H\. Zhao, W\. Fang, J\. Yin, S\. Dustdar, and A\. Y\. ZomayaEdge Intelligence: The Confluence of Edge Computing and Artificial Intelligence\.Proc\. the IEEE Internet of Things Journal7\(8\),pp\. 7457–7469\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p1.1)\.
- Dettmerset al\.\(2023\)T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. ZettlemoyerQLoRA: Efficient Finetuning of Quantized LLMs\.InProc\. the Thirty\-Seventh Annual Conference on Neural Information Processing Systems \(NeurIPS\),Vol\.36,pp\. 10088–10115\.Cited by:[§2\.2](https://arxiv.org/html/2610.41039#S2.SS2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\.Journal of Machine Learning Research23\(1\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p2.1)\.
- Fuet al\.\(2024\)W\. Fu, H\. Wang, C\. Gao, G\. Liu, Y\. Li, and T\. JiangMembership Inference Attacks against Fine\-tuned Large Language Models via Self\-prompt Calibration\.InProc\. the Thirty\-Eighth Annual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1)\.
- Gevaet al\.\(2021\)M\. Geva, R\. Schuster, J\. Berant, and O\. LevyTransformer Feed\-Forward Layers Are Key\-Value Memories\.InProc\. the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 5484–5495\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p5.1),[§2\.1](https://arxiv.org/html/2610.41039#S2.SS1.p3.1)\.
- Gunawiet al\.\(2016\)H\. S\. Gunawi, M\. Hao, R\. O\. Suminto, A\. Laksono, A\. D\. Satria, J\. Adityatama, and K\. J\. EliazarWhy Does the Cloud Stop Computing? Lessons from Hundreds of Service Outages\.InProc\. the Seventh ACM Symposium on Cloud Computing \(SoCC\),pp\. 1–16\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p1.1)\.
- Hayeset al\.\(2024\)J\. Hayes, I\. Shumailov, and I\. YonaBuffer Overflow in Mixture of Experts\.InProc\. the Thirty\-Eighth Annual Conference on Neural Information Processing Systems \(NeurIPS\) Safe Generative AI Workshop,Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p3.1)\.
- Houet al\.\(2025\)J\. Hou, C\. Wu, L\. Luo, and Q\. ZengImpact of Fine\-Tuning Methods on Memorization in Large Language Models\.arXiv preprint arXiv:2507\.00258\.Cited by:[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1)\.
- Huet al\.\(2022a\)E\. J\. Hu, yelong shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: Low\-Rank Adaptation of Large Language Models\.InProc\. the Tenth International Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2610.41039#S2.SS2.p1.1)\.
- Huet al\.\(2022b\)H\. Hu, Z\. Salcic, L\. Sun, G\. Dobbie, P\. S\. Yu, and X\. ZhangMembership Inference Attacks on Machine Learning: A Survey\.ACM Computing Surveys54\(11s\),pp\. 1–37\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1)\.
- Huet al\.\(2024\)J\. Hu, X\. Wu, Z\. Zhu, Xianyu, W\. Wang, D\. Zhang, and Y\. CaoOpenRLHF: An Easy\-to\-use, Scalable and High\-performance RLHF Framework\.arXiv preprint arXiv:2405\.11143\.Cited by:[§2\.2](https://arxiv.org/html/2610.41039#S2.SS2.p1.1)\.
- Huanget al\.\(2025\)Z\. Huang, Y\. Liu, D\. He, and Y\. LiDF\-MIA: A Distribution\-Free Membership Inference Attack on Fine\-Tuned Large Language Models\.InProc\. the Thirty\-Ninth AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 343–351\.Cited by:[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.Mixtral of Experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.41039#S2.SS1.p1.1)\.
- Kairouzet al\.\(2021\)P\. Kairouz, H\. B\. McMahan, B\. Avent, A\. Bellet, M\. Bennis, A\. Nitin Bhagoji, K\. Bonawitz, Z\. Charles, G\. Cormode, R\. Cummings, R\. G\. L\. D’Oliveira, H\. Eichner, S\. El Rouayheb, D\. Evans, J\. Gardner, Z\. Garrett, A\. Gascón, B\. Ghazi, P\. B\. Gibbons, M\. Gruteser, Z\. Harchaoui, C\. He, L\. He, Z\. Huo, B\. Hutchinson, J\. Hsu, M\. Jaggi, T\. Javidi, G\. Joshi, M\. Khodak, J\. Konecný, A\. Korolova, F\. Koushanfar, S\. Koyejo, T\. Lepoint, Y\. Liu, P\. Mittal, M\. Mohri, R\. Nock, A\. Özgür, R\. Pagh, H\. Qi, D\. Ramage, R\. Raskar, M\. Raykova, D\. Song, W\. Song, S\. U\. Stich, Z\. Sun, A\. T\. Suresh, F\. Tramèr, P\. Vepakomma, J\. Wang, L\. Xiong, Z\. Xu, Q\. Yang, F\. X\. Yu, H\. Yu, and S\. ZhaoAdvances and Open Problems in Federated Learning\.Foundations and Trends in Machine Learning14\(1–2\),pp\. 1–210\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p1.1)\.
- Kassemet al\.\(2024\)A\. M\. Kassem, O\. Mahmoud, N\. Mireshghallah, H\. Kim, Y\. Tsvetkov, Y\. Choi, S\. Saad, and S\. RanaAlpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs\.arXiv preprint arXiv:2403\.04801\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1)\.
- Loet al\.\(2025\)K\. M\. Lo, Z\. Huang, Z\. Qiu, Z\. Wang, and J\. FuA Closer Look into Mixture\-of\-Experts in Large Language Models\.InFindings of the Association for Computational Linguistics,pp\. 4427–4447\.Cited by:[§2\.1](https://arxiv.org/html/2610.41039#S2.SS1.p3.1)\.
- M4\-ai \(2024\)M4\-aiTinyMistral\-6x248M Model on Hugging Face\.Note:[https://huggingface\.co/M4\-ai/TinyMistral\-6x248M](https://huggingface.co/M4-ai/TinyMistral-6x248M)Accessed: 2025\-06\-15Cited by:[§2\.1](https://arxiv.org/html/2610.41039#S2.SS1.p1.1)\.
- McMahanet al\.\(2016\)H\. B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y ArcasCommunication\-Efficient Learning of Deep Networks from Decentralized Data\.InProc\. the 19th International Conference on Artificial Intelligence and Statistics \(AISTATS\),Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p1.1)\.
- Mireshghallahet al\.\(2022\)F\. Mireshghallah, A\. Uniyal, T\. Wang, D\. Evans, and T\. Berg\-KirkpatrickAn Empirical Analysis of Memorization in Fine\-tuned Autoregressive Language Models\.InProc\. Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1816–1826\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p5.1),[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1),[Table 2](https://arxiv.org/html/2610.41039#S4.T2.5.6.1.1.1)\.
- Muennighoffet al\.\(2024\)N\. Muennighoff, L\. Soldaini, D\. Groeneveld, K\. Lo, J\. Morrison, S\. Min, W\. Shi, P\. Walsh, O\. Tafjord, N\. Lambert, Y\. Gu, S\. Arora, A\. Bhagia, D\. Schwenk, D\. Wadden, A\. Wettig, B\. Hui, T\. Dettmers, D\. Kiela, A\. Farhadi, N\. A\. Smith, P\. W\. Koh, A\. Singh, and H\. HajishirziOLMoE: Open Mixture\-of\-Experts Language Models\.arXiv preprint arXiv:2409\.02060\.Cited by:[§2\.1](https://arxiv.org/html/2610.41039#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.41039#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.41039#S3.SS1.p3.1)\.
- Nasret al\.\(2025\)M\. Nasr, J\. Rando, N\. Carlini, J\. Hayase, M\. Jagielski, A\. F\. Cooper, D\. Ippolito, C\. A\. Choquette\-Choo, F\. Tramèr, and K\. LeeScalable Extraction of Training Data from Aligned, Production Language Models\.InProc\. the Thirteenth International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring The Limits of Transfer Learning with a Unified Text\-to\-Text Transformer\.The Journal of Machine Learning Research21\(1\)\.Cited by:[§3\.1](https://arxiv.org/html/2610.41039#S3.SS1.p3.1)\.
- Rajbhandariet al\.\(2022\)S\. Rajbhandari, C\. Li, Z\. Yao, M\. Zhang, R\. Y\. Aminabadi, A\. A\. Awan, J\. Rasley, and Y\. HeDeepSpeed\-MoE: Advancing Mixture\-of\-Experts Inference and Training to Power Next\-Generation AI Scale\.arXiv preprint arXiv:2201\.05596\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p2.1)\.
- Shenet al\.\(2024a\)S\. Shen, L\. Hou, Y\. Zhou, N\. Du, S\. Longpre, J\. Wei, H\. W\. Chung, B\. Zoph, W\. Fedus, X\. Chen, T\. Vu, Y\. Wu, W\. Chen, A\. Webson, Y\. Li, V\. Y\. Zhao, H\. Yu, K\. Keutzer, T\. Darrell, and D\. ZhouMixture\-of\-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models\.pp\. 1–17\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p2.1)\.
- Shenet al\.\(2024b\)S\. Shen, L\. Hou, Y\. Zhou, N\. Du, S\. Longpre, J\. Wei, H\. W\. Chung, B\. Zoph, W\. Fedus, X\. Chen, T\. Vu, Y\. Wu, W\. Chen, A\. Webson, Y\. Li, V\. Y\. Zhao, H\. Yu, K\. Keutzer, T\. Darrell, and D\. ZhouMixture\-of\-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models\.InProc\. the Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2\.2](https://arxiv.org/html/2610.41039#S2.SS2.p2.1)\.
- Shiet al\.\(2024\)W\. Shi, A\. Ajith, M\. Xia, Y\. Huang, D\. Liu, T\. Blevins, D\. Chen, and L\. ZettlemoyerDetecting Pretraining Data from Large Language Models\.InProc\. the Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1),[Table 2](https://arxiv.org/html/2610.41039#S4.T2.5.11.1.1.1)\.
- Vermaet al\.\(2015\)A\. Verma, L\. Pedrosa, M\. Korupolu, D\. Oppenheimer, E\. Tune, and J\. WilkesLarge\-Scale Cluster Management at Google with Borg\.InProc\. the Tenth European Conference on Computer Systems \(EuroSys\),pp\. 1–17\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p1.1)\.
- Wanget al\.\(2025\)Q\. Wang, Q\. Pang, X\. Lin, S\. Wang, and D\. WuBadMoE: Backdooring Mixture\-of\-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Experts\.arXiv preprint arXiv:2504\.18598\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p3.1)\.
- Yeomet al\.\(2018\)S\. Yeom, I\. Giacomelli, M\. Fredrikson, and S\. JhaPrivacy Risk in Machine Learning: Analyzing the Connection to Overfitting\.InProc\. the 31st Computer Security Foundations Symposium \(CSF\),pp\. 268–282\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1),[Table 2](https://arxiv.org/html/2610.41039#S4.T2.5.2.1.1.1)\.
- Yonaet al\.\(2024\)I\. Yona, I\. Shumailov, J\. Hayes, and N\. CarliniStealing User Prompts from Mixture of Experts\.arXiv preprint arXiv:2410\.22884\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p3.1)\.
- Zenget al\.\(2024\)S\. Zeng, Y\. Li, J\. Ren, Y\. Liu, H\. Xu, P\. He, Y\. Xing, S\. Wang, J\. Tang, and D\. YinExploring Memorization in Fine\-tuned Language Models\.InProc\. the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 3917–3948\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p5.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1)\.
- Zhanget al\.\(2025\)J\. Zhang, J\. Sun, E\. Yeats, Y\. Ouyang, M\. Kuo, J\. Zhang, H\. F\. Yang, and H\. LiMin\-K%\+\+: Improved Baseline for Pre\-Training Data Detection from Large Language Models\.InProc\. the Thirteenth International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p2.1),[Table 2](https://arxiv.org/html/2610.41039#S4.T2.5.13.1.1.1)\.
- Zhanget al\.\(2023\)Z\. Zhang, J\. Wen, and M\. HuangETHICIST: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimation\.InProc\. the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 12674–12687\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p7.1),[§2\.3](https://arxiv.org/html/2610.41039#S2.SS3.p1.1)\.
- Zhenget al\.\(2024\)Y\. Zheng, R\. Zhang, J\. Zhang, Y\. Ye, Z\. Luo, Z\. Feng, and Y\. MaLlamaFactory: Unified Efficient Fine\-Tuning of 100\+ Language Models\.InProc\. the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§2\.2](https://arxiv.org/html/2610.41039#S2.SS2.p1.1)\.
- Zhouet al\.\(2022\)Y\. Zhou, T\. Lei, H\. Liu, N\. Du, Y\. Huang, V\. Y\. Zhao, A\. M\. Dai, Z\. Chen, Q\. V\. Le, and J\. LaudonMixture\-of\-Experts with Expert Choice Routing\.InProc\. the Thirty\-Sixth Annual Conference on Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p3.1)\.
- Zhouet al\.\(2019\)Z\. Zhou, X\. Chen, E\. Li, L\. Zeng, K\. Luo, and J\. ZhangEdge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing\.Proc\. the IEEE107\(8\),pp\. 1738–1762\.Cited by:[§1](https://arxiv.org/html/2610.41039#S1.p1.1)\.

## Appendix AOpen Science

The artifacts required to evaluate the core contributions of this paper are the source code implementing \(i\) the fine\-tuning pipeline for the evaluated MoE LLMs, \(ii\) the extraction of expert utilization profiles and routing fingerprints from unconditional generations, \(iii\) the baseline and composite membership inference metrics, and \(iv\) the scripts and configuration files needed to reproduce all figures and tables\. All models \(TinyMistral,PhiMoE,OLMoE, andMixtral\) and datasets used in our evaluation are publicly available from their original sources\. Our code and experiment configurations are available at[REPOSITORY\-URL](https://repository-url/)\.