Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking
Summary
This paper presents a systematic sensitivity analysis of Mixture-of-Experts models using magnitude-based expert masking, finding that late layers are more resilient to masking, which provides a practical path for model compression.
View Cached Full Text
Cached at: 08/17/26, 09:36 AM
# Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking Source: [https://arxiv.org/abs/2608.13565](https://arxiv.org/abs/2608.13565) [View PDF](https://arxiv.org/pdf/2608.13565) > Abstract:Mixture\-of\-Experts \(MoE\) architectures scale large language models \(LLMs\) while preserving computational efficiency through sparse activation\. Despite their widespread adoption, the relative importance of individual MoE layers remains insufficiently characterized, particularly for model compression\. This paper presents a systematic layer\-wise sensitivity analysis of the Qwen3\.6\-35B\-A3B model \(40 MoE layers, 256 experts per layer, top\-8 routing\) using magnitude\-based expert masking on the XLCoST cross\-lingual code translation benchmark\. We conduct a multi\-phase study spanning 100, 300, and 500 prompt evaluation scales across three H100 GPU servers\. Our central finding is that layer sensitivity is strongly depth\-dependent: early layers \(0\-9\) and middle layers \(10\-29\) are highly fragile to expert masking, while late layers \(30\-39\), and especially very\-late layers \(35\-39\), tolerate aggressive masking of low\-magnitude experts\. Flat all\-layer masking at 30% retains only 150/300 Good\+Similar outputs at 300\-prompt scale, whereas late\-focused policies retain 249\-255/300 while masking 640\-1,145 experts\. On a later 500\-prompt held\-out validation slice, the narrow very\-late policy \(layers 35\-39 @ 50%\) achieves the strongest quality/masked\-expert tradeoff among tested candidates, retaining 419/500 Good\+Similar outputs while masking only 640 of 10,240 total experts\. We additionally characterize top\-k routing width reduction from 8 to 6 active experts per token, which shows a large observed wall\-clock reduction on a 100\-prompt probe with no Good\+Similar loss, though it does not yet compose cleanly with aggressive expert masking\. These findings provide an empirical foundation for depth\-aware MoE expert masking and establish a practical path toward physical weight surgery, activation\-based expert scoring, and training\-based recovery\. ## Submission history From: Hritvik Shrivastava \[[view email](https://arxiv.org/show-email/ccc7963c/2608.13565)\] **\[v1\]**Thu, 25 Jun 2026 05:00:34 UTC \(277 KB\)
Similar Articles
PuzzleMoE: Efficient Compression of Large Mixture-of-Experts Models via Sparse Expert Merging and Bit-packed inference
PuzzleMoE introduces a pairwise dual-mask expert merging algorithm and bit-level packing technique to compress large Mixture-of-Experts models, maintaining performance while reducing storage and accelerating inference.
Output Dilution: Redundant but Fragile Representations in MoE Models
This paper reveals that Mixture-of-Experts models encode moral content as robustly as dense models in probing but are far more fragile due to output dilution, where aggregated expert outputs dilute the signal, making representations vulnerable to noise.
Pruning and Distilling Mixture-of-Experts into Dense Language Models
A systematic framework converts mixture-of-experts models into dense architectures through expert scoring, selection, grouping, and knowledge distillation, achieving superior performance and efficiency compared to traditional pruning methods.
Emergent Modularity in Mixture-of-Experts Models (8 minute read)
Ai2 releases EMO, a 14B-parameter mixture-of-experts language model trained to develop emergent modularity. It allows using a small subset of experts for specific tasks while maintaining near full-model performance.
Shape Mutating Expert Compression:LorExperts and BTExperts
This paper introduces LorExperts and BTExperts, router-preserving compression methods for Mixture-of-Experts LLMs that cluster experts and represent non-dominant members as low-rank corrections, improving compression quality over prior methods like D2-MoE.