TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts

# TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
Source: [https://arxiv.org/html/2606.09885](https://arxiv.org/html/2606.09885)
Jiangyang He,Shaolin Zhu,Deyi Xiong111Corresponding author\. TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China \{jiangyanghe, zhushaolin, dyxiong\}@tju\.edu\.cn

###### Abstract

Mixture\-of\-Experts large language models \(LLMs\) scale efficiently through sparse activation, yet their deployment is fundamentally constrained by the large static parameter footprint of experts\. Existing compression approaches either remove entire experts, disrupting routing topology and harming performance, or rely on unstructured weight pruning with limited practical efficiency\. To address the limitations, we propose TENP, a structuredTrapezoidalExpertNeuronPruning framework\. Using a few samples, we identify and retain important experts, while applying expert neuron pruning \(ENP\) to less important experts, preserving model parameters in a trapezoidal pattern from shallow to deep layers\. When evaluating expert importance, we jointly consider both the magnitude of the expert output and its ability to change the direction of the input vector\. For ENP, we measure each neuron’s projected contribution to the expert output to identify and retain important neurons\. We conduct extensive experiments on the Qwen and DeepSeek models\. Under a routing expert sparsity of 40% and an average of 63\.76% activated expert parameters, the DeepSeek model suffers only a 1\-point drop in accuracy compared to the full\-parameter model\. Moreover, it outperforms the full\-parameter model by 10% on code generation tasks\.

TENP: Trapezoidal Expert Neuron Pruning For Mixture\-of\-Experts

Jiangyang He, Shaolin Zhu, Deyi Xiong111Corresponding author\.TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China\{jiangyanghe, zhushaolin, dyxiong\}@tju\.edu\.cn

## 1Introduction

The paradigm shift towards Mixture\-of\-Experts \(MoE\) architectures has become a cornerstone for scaling, as it effectively decouples model capacity from computational costOpenAI \([2025](https://arxiv.org/html/2606.09885#bib.bib22)\)\. By conditionally activating a sparse subset of parameters for each token, models such as DeepSeek\-V3\.2DeepSeek\-AIet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib23)\)and Qwen3Yanget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib13)\)achieve state\-of\-the\-art performance with significantly reduced FLOPs compared to their dense counterpartsSunet al\.\([2024a](https://arxiv.org/html/2606.09885#bib.bib46)\); Panet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib47)\); Zhuet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib45)\)\. However, this efficiency comes with a substantial trade\-off that the massive static parameter footprint required to host the full set of experts creates a severe bottleneck for deployment, particularly in memory\-constrained environmentsBaiet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib1)\)\.

To address the memory bottleneck, existing compression methodologies primarily bifurcate into expert pruning, weight pruningBaiet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib1)\)or quantizationDuet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib44)\); Jinet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib43)\)\. Expert pruning methods attempt to permanently remove less significant experts based on activation frequency or router gradientsMuzioet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib29)\); Donget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib32)\)\. However, these approaches are constrained by a fundamental limitation: the coarse\-grained removal of entire experts disrupts the model’s original routing topology\. As indicated in recent analysesChenet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib9)\), altering the routing path compels forces tokens to be dispatched to sub\-optimal experts, which can precipitate significant performance degradation on domain\-specific tasks\. Alternatively, unstructured weight pruning methodsSunet al\.\([2024b](https://arxiv.org/html/2606.09885#bib.bib30)\)target individual parameters but consequently yield irregular sparsity patterns that require specialized hardware kernels for acceleration\.

Recent empirical analyses indicate that for compressed models, restoring their original routing paths can recover model performanceChenet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib9)\)\. However, experts that have been removed can no longer be routed to\. We therefore prune redundant parameters within experts while keeping the experts intact and routable\. Prior workSunet al\.\([2024b](https://arxiv.org/html/2606.09885#bib.bib30)\); Chenget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib33)\)has found that a large amount of redundancy exists at the microscopic, neuron\-level within these experts\. Furthermore, redundancy is non\-uniformly distributed in the layer\-wise representational capacity of LLMs\. Shallow layers primarily process local syntactic features and exhibit high redundancy, whereas deep layers encapsulate complex semantic reasoningYanget al\.\([2024a](https://arxiv.org/html/2606.09885#bib.bib34)\); Gaoet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib27)\)\. This implies that a uniform pruning ratio is sub\-optimal for MoE architectures and motivates a depth\-aware allocation of parameter budgets\.

To address these limitations, we proposeTENP\(Trapezoidal Expert Neuron Pruning\), a structured pruning framework tailored for MoE LLMs\. Unlike expert\-level pruning, which alters where a token goes, TENP focuses on slimming down what the expert computes by pruning neurons within experts to maintain the validity of the pre\-trained router’s decisions\. In particular, TENP introduces a depth\-aware Trapezoidal sparsity strategy\. It applies aggressive pruning to shallow, high\-redundancy layers and progressively retains more capacity in deep layers to preserve reasoning capabilities\. To accurately identify redundant neurons without costly retraining, we design a dual\-metric evaluation that combines the magnitude contribution with a directional impact score to distinguish between essential transformations and redundant identity\-like mappings\.

Our contributions are summarized as follows:

- •We proposeTENP, astructured expert\-neuron pruningthat reduces memory usage while strictly preserving the original MoE routing topology\.
- •We introducean Trapezoidal sparsity distribution strategy\. We empirically demonstrate that allocating high parameter budgets to deep layers while aggressively compressing shallow layers yields a superior trade\-off between model size and performance\.
- •Experiments demonstrate that, under routing expert sparsities of 40% and 70%, our method consistently outperforms existing expert pruning approaches across a wide range of reasoning tasks and benchmarks, achieving an average improvement of approximately 16%\.

## 2Related Work

#### MoE LLMs

MoE models have gained significant attention in recent years due to their unique capability of expanding model capacity without proportionally increasing computational costs\. MoE architectures partition a large neural network \(or specific components\) into multiple expert sub\-networks, where only a subset is activated for each input token based on routing decisionsShazeeret al\.\([2017](https://arxiv.org/html/2606.09885#bib.bib15)\); Feduset al\.\([2022](https://arxiv.org/html/2606.09885#bib.bib16)\)\. GShardLepikhinet al\.\([2020](https://arxiv.org/html/2606.09885#bib.bib17)\)pioneers trillion\-parameter models by distributing parameters across multiple devices\. DeepSeekMoEDaiet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib18)\)presents a shared expert mechanism to reduce communication overhead and computational cost\. The effectiveness of MoE architectures has been validated at the 16 billion parameter scaleTeam \([2024](https://arxiv.org/html/2606.09885#bib.bib19)\); DeepSeek\-AIet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib20)\)\. More recently, MixtralJianget al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib21)\), GPT\-OSSOpenAI \([2025](https://arxiv.org/html/2606.09885#bib.bib22)\), DeepSeekV3\.2DeepSeek\-AIet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib23)\), and KimiK2Teamet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib24)\)have demonstrated MoE’s efficacy at the hundred\-billion parameter scale\. Advanced routing strategies have also emerged, with DA\-MoEYaoet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib25)\)and XMoEYanget al\.\([2024b](https://arxiv.org/html/2606.09885#bib.bib26)\)implementing dynamic expert selection mechanisms that allocate more computational resources to challenging tokens\.Gaoet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib27)\)propose a pyramid\-shaped architecture where layers closer to the output employ more parameters\. Meanwhile, GroveMoEWuet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib28)\)adopts heterogeneous experts with dynamic parameter activation to optimize performance\.

![Refer to caption](https://arxiv.org/html/2606.09885v1/x1.png)Figure 1:Schematic illustration of the Trapezoidal Expert Neuron Pruning \(TENP\) framework\. \(a\) TENP preserves the complete routing structure, allowing even pruned experts to remain routable\. \(b\) Comparison between an expert before pruning and after pruning: neurons inside the expert are removed, while the dimensionality of the expert’s output remains unchanged\. \(c\) After pruning, the parameter distribution of the model changes from a rectangular structure, where each layer has an identical number of parameters, to a trapezoidal structure with fewer parameters in shallow layers and more parameters in deep layers\.
#### Pruning and Compression of MoE Models

As scaling laws continue to drive exponential growth in MoE model sizes, numerous techniques have emerged to reduce parameter counts while preserving performance\. SEER\-MoEMuzioet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib29)\)prunes less important experts based on their activation frequency or gating score\.Luet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib31)\)introduces expert\-level pruning combined with dynamic skipping mechanisms\.Donget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib32)\)concentrates model capabilities in specific domains through few\-shot expert localization\. While these methods reduce parameter counts, they generally fail to decrease computational requirements proportionally\.Chenget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib33)\)achieve computation reduction through fine\-grained neuron activation within selected experts, without reducing the overall parameter count\. More comprehensive approaches include MoE\-I2Yanget al\.\([2024a](https://arxiv.org/html/2606.09885#bib.bib34)\), which proposes a three\-stage pruning methodology requiring subsequent fine\-tuning to recover performance, and Task\-Specific Expert PruningChenet al\.\([2022](https://arxiv.org/html/2606.09885#bib.bib35)\), which integrates pruning with task\-specific training\. SparseGPTFrantar and Alistarh \([2023](https://arxiv.org/html/2606.09885#bib.bib36)\), MoE\-PrunerXieet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib37)\), and WandaSunet al\.\([2024b](https://arxiv.org/html/2606.09885#bib.bib30)\)employ an unstructured pruning method that imposes specific hardware requirements\. Alternative approaches likeLiuet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib38)\)consolidate important neurons across experts, though this compromises the original routing mechanism\. Similarly,Liet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib39)\)andLiet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib40)\)propose expert merging strategies that face routing challenges\. The methods compress all experts into a single dense model, although they have reduced many parameters, have changed the architecture of the modelHeet al\.\([2023](https://arxiv.org/html/2606.09885#bib.bib41)\); Caoet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib42)\)\.

## 3Method

Our approach consists of two stages\. In the first stage, we follow and modify EASY\-EP to identify important experts\. We conduct experiments with different important expert retention ratios, as detailed in Appendix[D](https://arxiv.org/html/2606.09885#A4)\. Ultimately, we determine that under a routed expert sparsity of 40%, retaining 20%–30% of important experts yields optimal results, while under a routed expert sparsity of 70%, retaining 5% of important experts is sufficient\. This approach not only preserves the complete routing topology but also reduces the average number of activated parameters per expert\. Moreover, the number of important experts preserved in each layer gradually increases with depth\. As illustrated in Figure[1](https://arxiv.org/html/2606.09885#S2.F1), higher layers retain more parameters, resulting in a trapezoidal parameter distribution\. In the second stage, we perform neuron pruning on the less important experts identified in the first stage\. Alternatively, the first stage can be skipped, and neuron pruning can be directly applied to all experts; we refer to this approach as Expert Neuron Pruning \(ENP\)\. We evaluate the contribution of each intermediate dimension of an expert to the output using the method described below, where each intermediate dimension corresponds to specific rows and columns of the expert’s parameters\. For dimensions deemed unimportant, we remove the corresponding rows and columns of parameters to eliminate that intermediate dimension\.

### 3\.1Retaining Important Experts

To evaluate expert importance, we compute an importance score from the experts’ outputs\. Suppose the input to the MoE block at layerllis𝐡tl\\mathbf\{h\}\_\{t\}^\{l\}\. Let the output of theii\-th routed expertEil\\mathrm\{E\}\_\{i\}^\{l\}for tokenttbe𝐡¯i,tl\\mathbf\{\\overline\{h\}\}\_\{i,t\}^\{l\}, and the routing weight be𝐠i,tl\\mathbf\{g\}\_\{i,t\}^\{l\}\. The MoE output of all routed experts at layerllis denoted as𝐡~tl\\mathbf\{\\tilde\{h\}\}\_\{t\}^\{l\}, which is the weighted sum of theNNexperts’ outputs, as follows:

𝐡¯i,tl=Eil​\(𝐡tl\),\\mathbf\{\\overline\{h\}\}\_\{i,t\}^\{l\}=\\mathrm\{E\}\_\{i\}^\{l\}\(\\mathbf\{h\}\_\{t\}^\{l\}\),\(1\)𝐡~tl=∑i=1N𝐠i,tl⋅𝐡¯i,tl\.\\mathbf\{\\tilde\{h\}\}\_\{t\}^\{l\}=\\sum\_\{i=1\}^\{N\}\\mathbf\{g\}\_\{i,t\}^\{l\}\\cdot\\mathbf\{\\overline\{h\}\}\_\{i,t\}^\{l\}\.\(2\)
We define the length\-based contribution of expert outputs as𝐜i,tl\\mathbf\{c\}\_\{i,t\}^\{l\}\. Using∥⋅∥\\\|\\cdot\\\|to denote theℓ2\\ell\_\{2\}norm, for each tokentt, we quantify the contribution of expertiito the layer output by the product of the routing weight and the output norm:

𝐜i,tl=𝐠i,tl​‖𝐡¯i,tl‖,∀𝐠i,tl\>0\.\\mathbf\{c\}\_\{i,t\}^\{l\}=\\mathbf\{g\}\_\{i,t\}^\{l\}\\,\\\|\\mathbf\{\\overline\{h\}\}\_\{i,t\}^\{l\}\\\|,\\quad\\forall\\,\\mathbf\{g\}\_\{i,t\}^\{l\}\>0\.\(3\)
Inspired byMenet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib48)\)andDonget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib32)\), beyond the output magnitude and routing weight, we consider each expert’s ability to alter the direction of the input vector\.Menet al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib48)\)suggest that when a layer behaves closer to an identity mapping, it tends to be more redundant\. However,𝐜i,tl\\mathbf\{c\}\_\{i,t\}^\{l\}only reflects the magnitude of an expert’s output vector and the weight assigned to the expert by the router; it does not capture the expert’s ability to change the direction of the input vector\. In other words, relying solely on𝐜i,tl\\mathbf\{c\}\_\{i,t\}^\{l\}does not allow us to determine whether an expert is performing an identity mapping, since an expert that implements an identity mapping can also have a large𝐜i,tl\\mathbf\{c\}\_\{i,t\}^\{l\}value\. Therefore, we introduce𝐬i,tl\\mathbf\{s\}\_\{i,t\}^\{l\}to quantify the directional change induced by the expert, defined as one minus the cosine similarity between the expert’s input and output\. Values close to zero correspond to near\-identity behavior, whereas larger values indicate more substantial angular deviations\.

𝐬i,tl=1−Sim​\(𝐡tl,𝐡¯i,tl\),\\mathbf\{s\}\_\{i,t\}^\{l\}=1\-\\mathrm\{Sim\}\(\\mathbf\{h\}\_\{t\}^\{l\},\\mathbf\{\\overline\{h\}\}\_\{i,t\}^\{l\}\),\(4\)whereSim​\(⋅,⋅\)\\mathrm\{Sim\}\(\\cdot,\\cdot\)denotes cosine similarity\.

Finally, we jointly consider the magnitude of the expert output vector, the weight assigned by the router, and the expert’s ability to alter the vector direction, and average these factors over all tokensTTto comprehensively evaluate the importance of each expert, as shown in the following formula:

𝐈​\(Eil\)=∑t=1T𝐜i,tl⋅𝐬i,tl\.\\mathbf\{I\}\(\\mathrm\{E\}\_\{i\}^\{l\}\)=\\sum\_\{t=1\}^\{T\}\\mathbf\{c\}\_\{i,t\}^\{l\}\\cdot\\mathbf\{s\}\_\{i,t\}^\{l\}\.\(5\)
For each domain, we use 128 validation samples to evaluate expert and neuron importance\. We also investigate the effect of different numbers of samples on the results, as reported in[4\.8](https://arxiv.org/html/2606.09885#S4.SS8)\. For aggregation across domainsτ\\tau, we apply anℓ2\\ell\_\{2\}\-norm\-based normalization \(regularization\) as follows:

𝐈mix​\(Eil\)=∑τ∈𝒯𝐈τ​\(Eil\)∑j=1N𝐈τ​\(Ejl\)2\.\\mathbf\{I\}\_\{\\mathrm\{mix\}\}\(\\mathrm\{E\}\_\{i\}^\{l\}\)=\\sum\_\{\\tau\\in\\mathcal\{T\}\}\\frac\{\\mathbf\{I\}\_\{\\tau\}\(\\mathrm\{E\}\_\{i\}^\{l\}\)\}\{\\sqrt\{\\sum\_\{j=1\}^\{N\}\\mathbf\{I\}\_\{\\tau\}\(\\mathrm\{E\}\_\{j\}^\{l\}\)^\{2\}\}\}\.\(6\)

### 3\.2Expert Neuron Pruning

After selecting important experts, we perform neuron pruning on the remaining experts\. As noted above, neuron pruning can also be applied to all experts directly\. For a single expert, its output can be written as:

𝐡¯tl=𝐖down​SwiGLU​\(𝐖up​𝐡tl,𝐖gate​𝐡tl\),\\overline\{\\mathbf\{h\}\}\_\{t\}^\{l\}=\\mathbf\{W\}\_\{\\mathrm\{down\}\}\\,\\mathrm\{SwiGLU\}\(\\mathbf\{W\}\_\{\\mathrm\{up\}\}\\mathbf\{h\}\_\{t\}^\{l\},\\;\\mathbf\{W\}\_\{\\mathrm\{gate\}\}\\mathbf\{h\}\_\{t\}^\{l\}\),\(7\)where𝐖gate\\mathbf\{W\}\_\{\\mathrm\{gate\}\}is the gating matrix,𝐖up\\mathbf\{W\}\_\{\\mathrm\{up\}\}is the first linear projection,𝐖down\\mathbf\{W\}\_\{\\mathrm\{down\}\}is the second linear projection, andSwiGLU​\(⋅,⋅\)\\mathrm\{SwiGLU\}\(\\cdot,\\cdot\)denotes the SiLU\-gated activation\.\*\*\*If using a different FFN variant, the formulation can be adjusted accordingly\.

We extract thekk\-th row of𝐖up\\mathbf\{W\}\_\{\\mathrm\{up\}\}and𝐖gate\\mathbf\{W\}\_\{\\mathrm\{gate\}\}, and thekk\-th column of𝐖down\\mathbf\{W\}\_\{\\mathrm\{down\}\}, denoted by𝐰up,k\\mathbf\{w\}\_\{\\mathrm\{up\},k\},𝐰gate,k\\mathbf\{w\}\_\{\\mathrm\{gate\},k\}, and𝐰down,k\\mathbf\{w\}\_\{\\mathrm\{down\},k\}, respectively\. Substituting them into the above equation yields the expert output when only thekk\-th neuron is retained:

𝐡¯t,kl=𝐰down,k​SwiGLU​\(𝐰up,k​𝐡tl,𝐰gate,k​𝐡tl\)\.\\overline\{\\mathbf\{h\}\}\_\{t,k\}^\{l\}=\\mathbf\{w\}\_\{\\mathrm\{down\},k\}\\,\\mathrm\{SwiGLU\}\(\\mathbf\{w\}\_\{\\mathrm\{up\},k\}\\mathbf\{h\}\_\{t\}^\{l\},\\;\\mathbf\{w\}\_\{\\mathrm\{gate\},k\}\\mathbf\{h\}\_\{t\}^\{l\}\)\.\(8\)
Both𝐡¯t,kl\\overline\{\\mathbf\{h\}\}\_\{t,k\}^\{l\}and𝐡¯tl\\overline\{\\mathbf\{h\}\}\_\{t\}^\{l\}share the same output dimension, i\.e\.,𝐡¯t,kl,𝐡¯tl∈ℝd\\overline\{\\mathbf\{h\}\}\_\{t,k\}^\{l\},\\;\\overline\{\\mathbf\{h\}\}\_\{t\}^\{l\}\\in\\mathbb\{R\}^\{d\}\. We then quantify the importance of neuronkkby either the magnitude of its projection onto the full expert output or by theℓ2\\ell\_\{2\}norm of𝐡¯t,kl\\overline\{\\mathbf\{h\}\}\_\{t,k\}^\{l\}\. Using the projection magnitude, we define:

𝐩k=⟨𝐡¯t,kl,𝐡¯tl⟩‖𝐡¯tl‖\.\\mathbf\{p\}\_\{k\}=\\frac\{\\langle\\overline\{\\mathbf\{h\}\}\_\{t,k\}^\{l\},\\;\\overline\{\\mathbf\{h\}\}\_\{t\}^\{l\}\\rangle\}\{\\\|\\overline\{\\mathbf\{h\}\}\_\{t\}^\{l\}\\\|\}\.\(9\)
A larger projection magnitude \(orℓ2\\ell\_\{2\}norm\) indicates a more important neuron\. We aggregate neuron importance by averaging across tokens for thekk\-th neuron in expertiiat layerll\(As in Appendix in Section[B](https://arxiv.org/html/2606.09885#A2)\):

𝐏i,kl=1T​∑t=1T𝐏i,kl\.\\mathbf\{P\}\_\{i,k\}^\{l\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbf\{P\}\_\{i,k\}^\{l\}\.\(10\)
We prune each expert by keeping its top\-KKmost important neurons\. LetTopK​\(𝐏il\)idx\\mathrm\{TopK\}\(\\mathbf\{P\}\_\{i\}^\{l\}\)\_\{\\mathrm\{idx\}\}denote the indices of the top\-KKvalues in𝐏il\\mathbf\{P\}\_\{i\}^\{l\}\. For theii\-th expert at layerll, the pruned parameters are:

ModelMethodE↓\\downarrowA↓\\downarrowGSM8KMBPPHumanevalARC\-EARC\-CAvg\.Qwen1\.5MoE\-A2\.7BFull100%100%61\.547\.634\.286\.476\.161\.16Random30%100%1\.70\.00\.025\.825\.710\.64Frequency30%100%1\.60\.00\.052\.644\.119\.66Gating Score30%100%2\.30\.41\.256\.343\.220\.68EASY\-EP30%100%3\.40\.41\.855\.344\.821\.14ENP\(Ours\)30%30%23\.125\.117\.178\.266\.041\.90TENP\(Ours\)30%34\.51%25\.725\.318\.978\.266\.943\.00Random60%100%19\.61\.20\.077\.864\.532\.62Frequency60%100%30\.914\.66\.780\.066\.739\.78Gating Score60%100%30\.818\.98\.584\.573\.043\.14EASY\-EP60%100%36\.835\.419\.580\.168\.348\.02ENP\(Ours\)60%60%51\.340\.628\.084\.674\.955\.88TENP\(Ours\)60%61\.38%58\.345\.731\.185\.175\.159\.06DeepSeek\-V2\-LiteFull100%100%41\.143\.226\.284\.170\.352\.98Random30%100%1\.10\.00\.024\.524\.29\.96Frequency30%100%1\.90\.00\.024\.324\.010\.04Gating Score30%100%1\.90\.00\.026\.727\.811\.28EASY\-EP30%100%2\.84\.31\.238\.429\.615\.26ENP\(Ours\)30%30%3\.79\.10\.059\.648\.424\.16TENP\(Ours\)30%36\.38%21\.133\.114\.073\.357\.139\.72Random60%100%1\.80\.00\.034\.331\.713\.56Frequency60%100%32\.824\.411\.675\.361\.841\.18Gating Score60%100%21\.530\.711\.674\.057\.539\.06EASY\-EP60%100%34\.842\.114\.676\.762\.746\.18ENP\(Ours\)60%60%22\.133\.919\.577\.865\.743\.80TENP\(Ours\)60%63\.76%38\.445\.729\.979\.166\.851\.98

Table 1:Comparison of our method with other expert\-pruning approaches across all benchmarks\.EEdenotes the equivalent total parameter count of the routed experts, andAAdenotes the average activated parameter count of the routed experts\.𝐖~i,upl\\displaystyle\\tilde\{\\mathbf\{W\}\}\_\{i,\\mathrm\{up\}\}^\{l\}=𝐖i,upl​\[TopK​\(𝐏il\)idx,:\],\\displaystyle=\\mathbf\{W\}\_\{i,\\mathrm\{up\}\}^\{l\}\\big\[\\mathrm\{TopK\}\(\\mathbf\{P\}\_\{i\}^\{l\}\)\_\{\\mathrm\{idx\}\},:\\big\],\(11\)𝐖~i,gatel\\displaystyle\\tilde\{\\mathbf\{W\}\}\_\{i,\\mathrm\{gate\}\}^\{l\}=𝐖i,gatel​\[TopK​\(𝐏il\)idx,:\],\\displaystyle=\\mathbf\{W\}\_\{i,\\mathrm\{gate\}\}^\{l\}\\big\[\\mathrm\{TopK\}\(\\mathbf\{P\}\_\{i\}^\{l\}\)\_\{\\mathrm\{idx\}\},:\\big\],\(12\)𝐖~i,downl\\displaystyle\\tilde\{\\mathbf\{W\}\}\_\{i,\\mathrm\{down\}\}^\{l\}=𝐖i,downl​\[:,TopK​\(𝐏il\)idx\]\.\\displaystyle=\\mathbf\{W\}\_\{i,\\mathrm\{down\}\}^\{l\}\\big\[:,\\mathrm\{TopK\}\(\\mathbf\{P\}\_\{i\}^\{l\}\)\_\{\\mathrm\{idx\}\}\\big\]\.\(13\)
By replacing the original expert’s weight matrix𝐖i,upl,𝐖i,gatel,𝐖i,downl\\mathbf\{W\}\_\{i,\\mathrm\{up\}\}^\{l\},\\mathbf\{W\}\_\{i,\\mathrm\{gate\}\}^\{l\},\\mathbf\{W\}\_\{i,\\mathrm\{down\}\}^\{l\}with the pruned𝐖~i,upl,𝐖~i,gatel​𝐖~i,downl\\tilde\{\\mathbf\{W\}\}\_\{i,\\mathrm\{up\}\}^\{l\},\\tilde\{\\mathbf\{W\}\}\_\{i,\\mathrm\{gate\}\}^\{l\}\\tilde\{\\mathbf\{W\}\}\_\{i,\\mathrm\{down\}\}^\{l\}, we obtain the neuron\-pruned expert\. The forward computation of the pruned expert can be formally formulated as:

𝐡¯tl=𝐖~down​SwiGLU​\(𝐖~up​𝐡tl,𝐖~gate​𝐡tl\),\\overline\{\\mathbf\{h\}\}\_\{t\}^\{l\}=\\tilde\{\\mathbf\{W\}\}\_\{\\mathrm\{down\}\}\\,\\mathrm\{SwiGLU\}\(\\tilde\{\\mathbf\{W\}\}\_\{\\mathrm\{up\}\}\\mathbf\{h\}\_\{t\}^\{l\},\\;\\tilde\{\\mathbf\{W\}\}\_\{\\mathrm\{gate\}\}\\mathbf\{h\}\_\{t\}^\{l\}\),\(14\)
Neuron pruning provides an additional benefit: it not only reduces the total number of parameters, but also decreases the number of parameters that are activated in the routed experts\.

## 4Experiment

MethodE↓\\downarrowA↓\\downarrowMBPPHumanevalARC\-EARC\-CAvg\.Full100%100%43\.226\.284\.270\.355\.98Random60%100%0\.00\.034\.331\.716\.50TENP w/o Both \(Random Select Both\)60%60\.73%1\.50\.041\.037\.019\.88TENP w/o ENP \(Random Select ENP\)60%60\.87%6\.70\.659\.845\.728\.20TENP w/o TE \(Random Select TE\)60%61\.00%34\.720\.776\.864\.449\.15Only EP60%100%45\.78\.577\.863\.648\.90Only ENP\-L260%60%32\.318\.977\.565\.948\.65Only ENP\-COS60%60%41\.717\.778\.765\.951\.00TENP60%63\.55%47\.229\.879\.466\.755\.78

Table 2:Results of the ablation study\. Only EP denotes applying expert pruning only\. ENP\-L2 and ENP\-COS denote neuron pruning based on neuron importance measured by theℓ2\\ell\_\{2\}norm or by the projection length, respectively\. Random denotes randomly selected groups used as a control baseline\.We conducted experiments on two MoE models with distinct architectures: Qwen1\.5\-MoE\-A2\.7BTeam \([2024](https://arxiv.org/html/2606.09885#bib.bib19)\)and DeepSeek\-V2\-LiteDeepSeek\-AIet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib20)\)\. Detailed model descriptions are provided in Appendix[A](https://arxiv.org/html/2606.09885#A1)\. We also report experiments on Qwen3\-Next\-80B\-A3B\-InstructYanget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib13)\)in Appendix[F](https://arxiv.org/html/2606.09885#A6)\.

### 4\.1Experimental Setup

#### Evaluation\.

We evaluated the proposed method on a diverse set of benchmarks spanning multiple domains\. For challenging mathematical reasoning, we used GSM8KCobbeet al\.\([2021](https://arxiv.org/html/2606.09885#bib.bib2)\)with 8\-shot prompting\. For code generation, we reported results on MBPPAustinet al\.\([2021](https://arxiv.org/html/2606.09885#bib.bib3)\)with 3\-shot prompting and HumanEvalChenet al\.\([2021](https://arxiv.org/html/2606.09885#bib.bib4)\)under zero\-shot evaluation\. For science question answering, we used ARC\-Easy and ARC\-ChallengeClarket al\.\([2018](https://arxiv.org/html/2606.09885#bib.bib5)\), both under 25\-shot prompting\. Following standard protocols, we used dataset\-specific few\-shot settings on open\-source datasets and reported results averaged over three runs\.

#### Baselines\.

We compared against four representative expert\-pruning approaches for MoE models\. As a lower bound, we included a random expert selection baseline to quantify performance when no preference is given to expert selection\. We further evaluated the frequency\-based pruning and gating\-score\-based pruning strategies proposed in SEER\-MoEMuzioet al\.\([2024](https://arxiv.org/html/2606.09885#bib.bib29)\)\. In addition, we compared with the recently proposed EASY\-EPDonget al\.\([2025](https://arxiv.org/html/2606.09885#bib.bib32)\)\. These baselines retained only the experts that are ranked highest according to statistics estimated from a small number of samples per dataset\.†††We do not included comparisons to methods that require unstructured pruning \(e\.g\., Wanda\), additional fine\-tuning \(e\.g\., MoE\-I2\), or approaches that modify the model architecture\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2606.09885#S3.T1)reported a comprehensive comparison between our approach and a variety of baselines across multiple datasets, model backbones, and pruning ratios\. When we applied TENP to prune DeepSeek\-V2\-Lite, the resulting model achieves better performance on both mathematical reasoning and knowledge\-intensive QA tasks than other expert\-pruning methods such as SEER\-MoE and EASY\-EP\. On code\-generation benchmarks \(e\.g\., MBPP and HumanEval\), the pruned model yields more than one point improvement over the full model\. Similar trends are observed on Qwen1\.5MoE\-A2\.7B, where our method consistently surpassed competing approaches across all evaluated domains\. For relatively simple QA\-style tasks such as ARC, pruned models generally preserved their original performance well\. In contrast, on more challenging reasoning\-heavy tasks \(e\.g\., GSM8K\), pruning induced some degradation; nevertheless, our approach still outperforms baselines\. Notably, for code generation, the performance drop is often small and can even exceed the full model, as highlighted by the underlined entries in Table[1](https://arxiv.org/html/2606.09885#S3.T1)\. We also conducted experiments at a higher sparsity level \(70%\), where our method consistently outperformed other approaches across all benchmarks\. Additional comparisons under different sparsity settings are provided in Appendix[C](https://arxiv.org/html/2606.09885#A3)\.

#### Activated\-Parameter Efficiency\.

Most existing expert\-pruning methods can remove experts, yet keep the number of experts selected by the router unchanged\. As a result, although the total parameter count decreases, the number of activated parameters remained the same \(normalized to100%100\\%\)\. In contrast, our approach pruned neurons within experts, thereby reducing not only the number of experts but also the activated parameters of routed experts accordingly\. With neuron pruning alone, our ENP variant achieved the lowest activated\-parameter footprint and, in most cases, still outperforms prior expert\-pruning methods\.

As indicated by metricAAin Table[1](https://arxiv.org/html/2606.09885#S3.T1), if the router selected a preserved \(unpruned\) expert, its activated parameters remain at100%100\\%\. If it selected a neuron\-pruned expert, the activated parameters are reduced to50%50\\%or lower of the original\. Since the routing probability of preserved experts is higher than that of neuron\-pruned experts, the overall activated parameters are only slightly higher than the fraction of preserved experts\. Consequently, the activated\-parameter cost of our routed experts is substantially lower than that of existing expert\-pruning approaches\.

![Refer to caption](https://arxiv.org/html/2606.09885v1/x2.png)Figure 2:Expert selection frequencies under different pruning methods, and their differences compared to the expert selection frequencies of the full\-parameter model\.![Refer to caption](https://arxiv.org/html/2606.09885v1/x3.png)Figure 3:Layer\-wise error of the pruned models, measured as the Euclidean distance between the output vectors of each layer and those of the full\-parameter model\.

### 4\.3Ablation Study

As shown in Table[2](https://arxiv.org/html/2606.09885#S4.T2), applying either expert pruning alone \(Only EP\) or expert neuron pruning alone \(ENP\) already yields a certain level of pruning effectiveness to validate the effectiveness of the two core components of our method\.

When ENP is applied, measuring neuron importance by the projection length of a neuron’s output vector onto the final output vector of the corresponding expert consistently outperformed the method that used theℓ2\\ell\_\{2\}norm of the neuron output vector\. This result indicated that projection\-based importance better captured a neuron’s contribution to the expert output\. When experts are selected randomly, the model performance is noticeably affected\. This observation demonstrated the validity of the experts we have retained in a trapezoidal structure\. Furthermore, when both experts and neurons are selected randomly, our method still outperformed the baseline that randomly selected experts \(Random\), even though both configurations have the same number of parameters\. This advantage arose from the different parameter distributions: our method allocated more parameters to higher layers through the trapezoidal structure and preserved the complete routing architecture\. We also compare other layer selection methods in Appendix[E](https://arxiv.org/html/2606.09885#A5)\. By combining important experts with neurons selected based on projection\-length importance, we obtained TENP, which achieved the best overall performance\. Notably, TENP incurred almost no accuracy loss compared to the full\-parameter model\.

MethodDataGSM8KMBPPHumanevalARC\-EARC\-CInAvg\.OutAvgInAvg\.OutAvgEASY\-EPMath38\.328\.013\.464\.455\.638\.3040\.3547\.4531\.81Code33\.043\.718\.956\.344\.231\.3044\.50Science23\.66\.31\.879\.166\.472\.7510\.57TENPMath40\.920\.510\.466\.851\.240\.9037\.2351\.3733\.06Code29\.550\.827\.469\.954\.139\.1051\.17Science20\.29\.82\.480\.767\.574\.1010\.80

Table 3:Generalization performance of the pruned models\. Data denotes the domain of the data used for pruning\.ModelMethodEMMLUQwen1\.5MoE\-A2\.7BFull100%61\.05GatingScore60%49\.13EASY\-EP60%47\.49TENP\(Ours\)60%54\.81

Table 4:The generalization performance of three pruning methods on the MMLU dataset, where no MMLU data is used during the pruning process\.
### 4\.4Routing Analysis

We characterized routing changes by measuring the difference in expert selection counts before and after pruning\. As shown in Figure[2](https://arxiv.org/html/2606.09885#S4.F2), we analyzed and compared the expert selection statistics of the full\-parameter model, the expert\-pruned model, and our proposed pruning method on DeepSeek\. We focus on the changes in expert selection frequencies induced by pruning\. When we retained only a subset of experts, it induced substantial changes in expert selection patterns\. Notably, the selection frequency of remaining experts does not uniformly increase after pruning\. Instead, while some experts experience a significant increase in selection frequency, others are selected less frequently, or even less than before pruning\. This observation indicates that routing behavior changes drastically\. More importantly, some experts that were not pruned and were previously routable are no longer selected by the router after pruning\. This led to a significant shift in the output representations\. Furthermore, we observed a trend that the magnitude of changes in expert selection frequency increased in deep layers, suggesting that routing behavior in deep layers is more severely affected by expert pruning\. In contrast, under our proposed method, the selection frequency of each expert remained almost unchanged, as illustrated in the bottom\-right figure in Figure[2](https://arxiv.org/html/2606.09885#S4.F2)\. This indicated that our approach largely preserved the original routing behavior\. Maintaining stable expert routing is one of the key reasons for the effectiveness of our method\.

### 4\.5Error Analysis

Pruning the FFN layers inevitably caused discrepancies between the outputs of the pruned model and those of the full\-parameter model\. In general, pruning more parameters led to larger output deviations\. Since the output of one layer serves as the input to the next, these deviations further influence subsequent routing decisions, causing errors to accumulate progressively across layers, as shown in Figure[3](https://arxiv.org/html/2606.09885#S4.F3)\. To quantify this effect, we computed the difference between the output vectors of the pruned model and the full model at each layer, and used theℓ2\\ell\_\{2\}norm \(i\.e\., Euclidean distance\) of the difference vector as a measure of layer\-wise error\. Our method exhibited a similar overall error trend to the expert pruning method EASY\-EP\. When ENP is applied in isolation, relatively large errors can be observed even in the shallow layers\. This behavior arose because certain experts play a disproportionately important role; uniform neuron pruning removes neurons indiscriminately from both critical and less critical experts, which introduced a substantial error at early stages\. Nevertheless, because ENP preserved the original routing structure, error accumulation across layers proceeds at a slower rate than with direct expert pruning\. Consequently, in the middle and deep layers of the model, the error introduced by expert pruning exceeded that caused by neuron pruning\. Interestingly, although the error gradually accumulates across layers, it drops sharply at the final layer\. This observation highlights the strong representational capacity of high\-level experts\. Motivated by this phenomenon, our model adopts a trapezoidal parameter distribution, allocating more parameters to higher layers\.

### 4\.6Generalization Ability

In this section, we designed a set of generalization experiments and compared our method with other approaches\. We performed pruning using data from a single domain,e\.g\., mathematics, code, or science, and then evaluated the pruned models on both in domain and out of domain benchmarks\. The experimental results are summarised in Table[3](https://arxiv.org/html/2606.09885#S4.T3)and Table[4](https://arxiv.org/html/2606.09885#S4.T4)\. For in\-domain pruning, our method consistently outperforms expert pruning approaches\. More importantly, our method achieves the highest average performance among the compared methods on out of domain evaluation\. We attribute this strong generalization capability to the fact that our approach better preserves the original routing behavior and the overall structural integrity of the model\.

MethodParamMem\(GB\)InputOutputTotalRateFull100%32\.952790\.432793\.085583\.511\.0Expert Pruning50%18\.553941\.073944\.817885\.881\.41ENP\(Ours\)50%16\.374094\.034097\.918191\.941\.47

Table 5:A comparison of throughput and static memory consumption among the full\-parameter model, expert pruning methods, and our method\. Param denotes the routed expert retention ratio, and Mem represents the static GPU memory consumption of the Qwen model on a single A100\-SXM\-80GB GPU\. Input denotes the input token throughput \(tok/s\), Output denotes the output token throughput \(tok/s\), Total denotes the total token throughput \(tok/s\), and Rate represents the ratio of the total throughput of each method relative to that of the full\-parameter model\.
### 4\.7Static Memory Consumption and Throughput

We further compare our ENP method with expert pruning approaches in terms of memory consumption and throughput\. As shown in Table[5](https://arxiv.org/html/2606.09885#S4.T5), under the same parameter scale \(50% routed expert sparsity\), ENP yields smaller individual experts, making it more friendly to GPU memory allocation, and significantly reduces static memory consumption from 32\.95 GB to 16\.37 GB\. With fewer activated parameters, ENP achieves higher input, output, and total throughput, increasing the total throughput to 147% relative to the full\-parameter model, and outperforming the expert pruning model with the same parameter scale by 6%\.

Unlike unstructured pruning methods, both ENP and TENP do not impose any special hardware requirements\. TENP requires modifications to the inference framework \(e\.g\., vLLM, SGLang\) to accommodate experts of different sizes, whereas ENP can be deployed without modifying the inference framework\.

![Refer to caption](https://arxiv.org/html/2606.09885v1/x4.png)Figure 4:The impact of different data scales on pruning performance\.
### 4\.8The Impact of Different Data Scales on Pruning Performance

Following the data selection strategy in EASY\-EP and considering practical scenarios where sufficient samples may not be available, we use only the prompts from the dataset and the responses generated by the full\-parameter model itself, thereby minimizing human involvement\. We further investigate the impact of different data scales on model performance, as shown in Figure[4](https://arxiv.org/html/2606.09885#S4.F4)\. For a single application scenario, under a routed expert sparsity of 60%, using only one sample is sufficient for the model to retain most of its performance\. With two samples, the pruned model can already surpass the performance of the full\-parameter model\. Performance plateaus at eight samples, and further scaling brings no significant improvement\.

## 5Conclusion

In this paper, we have presented TENP, a pruning method for MoE models that preserves important experts while structurally pruning unimportant neurons within experts\. Extensive experiments demonstrate that our approach better maintains the original routing behavior of the model, induces smaller intermediate\-layer errors, and achieves superior generalization performance\. Moreover, TENP consistently performs well across different sparsity levels, model architectures, and benchmarks\.

## Limitations

Although TENP has been shown to be highly effective on Qwen and DeepSeek models, and larger models are expected to contain more redundant parameters suggesting that TENP could yield even greater pruning benefits we have not yet conducted experiments on extremely large scale mixture\-of\-experts models such as DeepSeek\-V3\.2\. We leave the evaluation of TENP on such large\-scale models to future work\.

## Acknowledgments

The present research was supported by the National Key Research and Development Program of China \(Grant No\. 2023YFE0116400\)\. We would like to thank the anonymous reviewers for their insightful comments\.

## References

- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le, and C\. Sutton \(2021\)Program synthesis with large language models\.External Links:2108\.07732,[Link](https://arxiv.org/abs/2108.07732)Cited by:[§4\.1](https://arxiv.org/html/2606.09885#S4.SS1.SSS0.Px1.p1.1)\.
- DiEP: adaptive mixture\-of\-experts compression through differentiable expert pruning\.External Links:2509\.16105,[Link](https://arxiv.org/abs/2509.16105)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1),[§1](https://arxiv.org/html/2606.09885#S1.p2.1)\.
- M\. Cao, G\. Li, J\. Ji, J\. Zhang, X\. Ma, S\. Liu, and L\. Yin \(2025\)Condense, don’t just prune: enhancing efficiency and performance in moe layer pruning\.External Links:2412\.00069,[Link](https://arxiv.org/abs/2412.00069)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§4\.1](https://arxiv.org/html/2606.09885#S4.SS1.SSS0.Px1.p1.1)\.
- T\. Chen, S\. Huang, Y\. Xie, B\. Jiao, D\. Jiang, H\. Zhou, J\. Li, and F\. Wei \(2022\)Task\-specific expert pruning for sparse mixture\-of\-experts\.External Links:2206\.00277,[Link](https://arxiv.org/abs/2206.00277)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Chen, Y\. Shao, P\. Wang, and J\. Cheng \(2025\)EAC\-MoE: expert\-selection aware compressor for mixture\-of\-experts large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12942–12963\.External Links:[Link](https://aclanthology.org/2025.acl-long.633/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.633),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p2.1),[§1](https://arxiv.org/html/2606.09885#S1.p3.1)\.
- R\. Cheng, Y\. Guan, Y\. Ding, Q\. Hu, Y\. Wei, C\. Yuan, Y\. Shen, W\. Chen, and Y\. Gong \(2025\)Mixture of neuron experts\.External Links:2510\.05781,[Link](https://arxiv.org/abs/2510.05781)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p3.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§4\.1](https://arxiv.org/html/2606.09885#S4.SS1.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§4\.1](https://arxiv.org/html/2606.09885#S4.SS1.SSS0.Px1.p1.1)\.
- D\. Dai, C\. Deng, C\. Zhao, R\. X\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu, Z\. Xie, Y\. K\. Li, P\. Huang, F\. Luo, C\. Ruan, Z\. Sui, and W\. Liang \(2024\)DeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.External Links:2401\.06066,[Link](https://arxiv.org/abs/2401.06066)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- DeepSeek\-AI, A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Yang, H\. Zhang, H\. Ding, H\. Xin, H\. Gao, H\. Li, H\. Qu, J\. L\. Cai, J\. Liang, J\. Guo, J\. Ni, J\. Li, J\. Chen, J\. Yuan, J\. Qiu, J\. Song, K\. Dong, K\. Gao, K\. Guan, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, L\. Zhao, L\. Zhang, M\. Li, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, N\. Tian, P\. Huang, P\. Wang, P\. Zhang, Q\. Zhu, Q\. Chen, Q\. Du, R\. J\. Chen, R\. L\. Jin, R\. Ge, R\. Pan, R\. Xu, R\. Chen, S\. S\. Li, S\. Lu, S\. Zhou, S\. Chen, S\. Wu, S\. Ye, S\. Ma, S\. Wang, S\. Zhou, S\. Yu, S\. Zhou, S\. Zheng, T\. Wang, T\. Pei, T\. Yuan, T\. Sun, W\. L\. Xiao, W\. Zeng, W\. An, W\. Liu, W\. Liang, W\. Gao, W\. Zhang, X\. Q\. Li, X\. Jin, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Shen, X\. Chen, X\. Chen, X\. Nie, X\. Sun, X\. Wang, X\. Liu, X\. Xie, X\. Yu, X\. Song, X\. Zhou, X\. Yang, X\. Lu, X\. Su, Y\. Wu, Y\. K\. Li, Y\. X\. Wei, Y\. X\. Zhu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Li, Y\. Wang, Y\. Zheng, Y\. Zhang, Y\. Xiong, Y\. Zhao, Y\. He, Y\. Tang, Y\. Piao, Y\. Dong, Y\. Tan, Y\. Liu, Y\. Wang, Y\. Guo, Y\. Zhu, Y\. Wang, Y\. Zou, Y\. Zha, Y\. Ma, Y\. Yan, Y\. You, Y\. Liu, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Huang, Z\. Zhang, Z\. Xie, Z\. Hao, Z\. Shao, Z\. Wen, Z\. Xu, Z\. Zhang, Z\. Li, Z\. Wang, Z\. Gu, Z\. Li, and Z\. Xie \(2024\)DeepSeek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.External Links:2405\.04434,[Link](https://arxiv.org/abs/2405.04434)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.09885#S4.p1.1)\.
- DeepSeek\-AI, A\. Liu, A\. Mei, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Lu, C\. Zhao, C\. Deng, C\. Xu, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, E\. Li, F\. Zhou, F\. Lin, F\. Dai, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Xu, H\. Li, H\. Liang, H\. Wei, H\. Zhang, H\. Luo, H\. Ji, H\. Ding, H\. Tang, H\. Cao, H\. Gao, H\. Qu, H\. Zeng, J\. Huang, J\. Li, J\. Xu, J\. Hu, J\. Chen, J\. Xiang, J\. Yuan, J\. Cheng, J\. Zhu, J\. Ran, J\. Jiang, J\. Qiu, J\. Li, J\. Song, K\. Dong, K\. Gao, K\. Guan, K\. Huang, K\. Zhou, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Wang, L\. Zhao, L\. Yin, L\. Guo, L\. Luo, L\. Ma, L\. Wang, L\. Zhang, M\. S\. Di, M\. Y\. Xu, M\. Zhang, M\. Zhang, M\. Tang, M\. Zhou, P\. Huang, P\. Cong, P\. Wang, Q\. Wang, Q\. Zhu, Q\. Li, Q\. Chen, Q\. Du, R\. Xu, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Yin, R\. Xu, R\. Shen, R\. Zhang, S\. H\. Liu, S\. Lu, S\. Zhou, S\. Chen, S\. Cai, S\. Chen, S\. Hu, S\. Liu, S\. Hu, S\. Ma, S\. Wang, S\. Yu, S\. Zhou, S\. Pan, S\. Zhou, T\. Ni, T\. Yun, T\. Pei, T\. Ye, T\. Yue, W\. Zeng, W\. Liu, W\. Liang, W\. Pang, W\. Luo, W\. Gao, W\. Zhang, X\. Gao, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Chen, X\. Zhang, X\. Nie, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yu, X\. Li, X\. Yang, X\. Li, X\. Chen, X\. Su, X\. Pan, X\. Lin, X\. Fu, Y\. Q\. Wang, Y\. Zhang, Y\. Xu, Y\. Ma, Y\. Li, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Wang, Y\. Qian, Y\. Yu, Y\. Zhang, Y\. Ding, Y\. Shi, Y\. Xiong, Y\. He, Y\. Zhou, Y\. Zhong, Y\. Piao, Y\. Wang, Y\. Chen, Y\. Tan, Y\. Wei, Y\. Ma, Y\. Liu, Y\. Yang, Y\. Guo, Y\. Wu, Y\. Wu, Y\. Cheng, Y\. Ou, Y\. Xu, Y\. Wang, Y\. Gong, Y\. Wu, Y\. Zou, Y\. Li, Y\. Xiong, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Z\. F\. Wu, Z\. Z\. Ren, Z\. Zhao, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Huang, Z\. Wu, Z\. Li, Z\. Zhang, Z\. Xu, Z\. Wang, Z\. Gu, Z\. Zhu, Z\. Li, Z\. Zhang, Z\. Xie, Z\. Gao, Z\. Pan, Z\. Yao, B\. Feng, H\. Li, J\. L\. Cai, J\. Ni, L\. Xu, M\. Li, N\. Tian, R\. J\. Chen, R\. L\. Jin, S\. S\. Li, S\. Zhou, T\. Sun, X\. Q\. Li, X\. Jin, X\. Shen, X\. Chen, X\. Song, X\. Zhou, Y\. X\. Zhu, Y\. Huang, Y\. Li, Y\. Zheng, Y\. Zhu, Y\. Ma, Z\. Huang, Z\. Xu, Z\. Zhang, D\. Ji, J\. Liang, J\. Guo, J\. Chen, L\. Xia, M\. Wang, M\. Li, P\. Zhang, R\. Chen, S\. Sun, S\. Wu, S\. Ye, T\. Wang, W\. L\. Xiao, W\. An, X\. Wang, X\. Sun, X\. Wang, Y\. Tang, Y\. Zha, Z\. Zhang, Z\. Ju, Z\. Zhang, and Z\. Qu \(2025\)DeepSeek\-v3\.2: pushing the frontier of open large language models\.External Links:2512\.02556,[Link](https://arxiv.org/abs/2512.02556)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Dong, H\. Peng, P\. Liu, W\. X\. Zhao, D\. Wu, F\. Xiao, and Z\. Wang \(2025\)Domain\-specific pruning of large mixture\-of\-experts models with few\-shot demonstrations\.External Links:2504\.06792,[Link](https://arxiv.org/abs/2504.06792)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p2.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2606.09885#S3.SS1.p3.4),[§4\.1](https://arxiv.org/html/2606.09885#S4.SS1.SSS0.Px2.p1.1)\.
- J\. Du, R\. Jin, W\. Huang, W\. Liu, J\. Luan, and D\. Xiong \(2025\)Optimize quantization for large language models via progressive training\.InCompanion Proceedings of the ACM on Web Conference 2025,WWW ’25,New York, NY, USA,pp\. 2474–2483\.External Links:ISBN 9798400713316,[Link](https://doi.org/10.1145/3701716.3717578),[Document](https://dx.doi.org/10.1145/3701716.3717578)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p2.1)\.
- W\. Fedus, B\. Zoph, and N\. Shazeer \(2022\)Switch transformers: scaling to trillion parameter models with simple and efficient sparsity\.External Links:2101\.03961,[Link](https://arxiv.org/abs/2101.03961)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Frantar and D\. Alistarh \(2023\)SparseGPT: massive language models can be accurately pruned in one\-shot\.External Links:2301\.00774,[Link](https://arxiv.org/abs/2301.00774)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Gao, K\. Chen, J\. Rao, B\. Sun, R\. Liu, D\. Peng, Y\. Zhang, X\. Guo, J\. Yang, and V\. Subrahmanian \(2024\)Higher layers need more lora experts\.External Links:2402\.08562,[Link](https://arxiv.org/abs/2402.08562)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p3.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- S\. He, R\. Fan, L\. Ding, L\. Shen, T\. Zhou, and D\. Tao \(2023\)Merging experts into one: improving computational efficiency of mixture of experts\.External Links:2310\.09832,[Link](https://arxiv.org/abs/2310.09832)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, E\. B\. Hanna, F\. Bressand, G\. Lengyel, G\. Bour, G\. Lample, L\. R\. Lavaud, L\. Saulnier, M\. Lachaux, P\. Stock, S\. Subramanian, S\. Yang, S\. Antoniak, T\. L\. Scao, T\. Gervet, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2024\)Mixtral of experts\.External Links:2401\.04088,[Link](https://arxiv.org/abs/2401.04088)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Jin, J\. Du, W\. Huang, W\. Liu, J\. Luan, B\. Wang, and D\. Xiong \(2024\)A comprehensive evaluation of quantization strategies for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 12186–12215\.External Links:[Link](https://aclanthology.org/2024.findings-acl.726/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.726)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p2.1)\.
- D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2020\)GShard: scaling giant models with conditional computation and automatic sharding\.External Links:2006\.16668,[Link](https://arxiv.org/abs/2006.16668)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Li, N\. Li, X\. Yuan, W\. Xu, Q\. Chen, S\. Guo, and H\. Zhang \(2025\)CoMoE: collaborative optimization of expert aggregation and offloading for moe\-based llms at edge\.External Links:2508\.09208,[Link](https://arxiv.org/abs/2508.09208)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Li, Z\. Zhang, P\. Yadav, Y\. Sung, Y\. Cheng, M\. Bansal, and T\. Chen \(2024\)Merge, then compress: demystify efficient smoe with hints from its routing policy\.External Links:2310\.01334,[Link](https://arxiv.org/abs/2310.01334)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- E\. Liu, J\. Zhu, Z\. Lin, X\. Ning, M\. B\. Blaschko, S\. Yan, G\. Dai, H\. Yang, and Y\. Wang \(2024\)Efficient expert pruning for sparse mixture\-of\-experts language models: enhancing performance and reducing inference costs\.External Links:2407\.00945,[Link](https://arxiv.org/abs/2407.00945)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Lu, Q\. Liu, Y\. Xu, A\. Zhou, S\. Huang, B\. Zhang, J\. Yan, and H\. Li \(2024\)Not all experts are equal: efficient expert pruning and skipping for mixture\-of\-experts large language models\.External Links:2402\.14800,[Link](https://arxiv.org/abs/2402.14800)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Men, M\. Xu, Q\. Zhang, Q\. Yuan, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. Chen \(2025\)ShortGPT: layers in large language models are more redundant than you expect\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 20192–20204\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1035/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1035),ISBN 979\-8\-89176\-256\-5Cited by:[§3\.1](https://arxiv.org/html/2606.09885#S3.SS1.p3.4)\.
- A\. Muzio, A\. Sun, and C\. He \(2024\)SEER\-moe: sparse expert efficiency through regularization for mixture\-of\-experts\.External Links:2404\.05089,[Link](https://arxiv.org/abs/2404.05089)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p2.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2606.09885#S4.SS1.SSS0.Px2.p1.1)\.
- OpenAI \(2025\)Gpt\-oss\-120b & gpt\-oss\-20b model card\.External Links:2508\.10925,[Link](https://arxiv.org/abs/2508.10925)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Pan, B\. Xiong, L\. Yang, R\. Jin, S\. Zhang, Y\. Chen, L\. Shi, J\. Zhou, J\. Wu, Z\. D\. Wang, J\. Peng, J\. Xiao, T\. Dong, Z\. Han, Z\. Chen, Y\. Ren, and D\. Xiong \(2025\)Advancing large language models for tibetan with curated data and continual pre\-training\.ArXivabs/2507\.09205\.External Links:[Link](https://api.semanticscholar.org/CorpusID:280137725)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1)\.
- N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.External Links:1701\.06538,[Link](https://arxiv.org/abs/1701.06538)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Sun, R\. Jin, S\. Xu, L\. Pan, Supryadi, M\. Cui, J\. Du, Y\. Lei, L\. Yang, L\. Shi, J\. Xiao, S\. Zhu, and D\. Xiong \(2024a\)FuxiTranyu: a multilingual large language model trained with balanced data\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track,F\. Dernoncourt, D\. Preoţiuc\-Pietro, and A\. Shimorina \(Eds\.\),Miami, Florida, US,pp\. 1499–1522\.External Links:[Link](https://aclanthology.org/2024.emnlp-industry.110/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-industry.110)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1)\.
- M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. Kolter \(2024b\)A simple and effective pruning approach for large language models\.External Links:2306\.11695,[Link](https://arxiv.org/abs/2306.11695)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p2.1),[§1](https://arxiv.org/html/2606.09885#S1.p3.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Team, Y\. Bai, Y\. Bao, G\. Chen, J\. Chen, N\. Chen, R\. Chen, Y\. Chen, Y\. Chen, Y\. Chen, Z\. Chen, J\. Cui, H\. Ding, M\. Dong, A\. Du, C\. Du, D\. Du, Y\. Du, Y\. Fan, Y\. Feng, K\. Fu, B\. Gao, H\. Gao, P\. Gao, T\. Gao, X\. Gu, L\. Guan, H\. Guo, J\. Guo, H\. Hu, X\. Hao, T\. He, W\. He, W\. He, C\. Hong, Y\. Hu, Z\. Hu, W\. Huang, Z\. Huang, Z\. Huang, T\. Jiang, Z\. Jiang, X\. Jin, Y\. Kang, G\. Lai, C\. Li, F\. Li, H\. Li, M\. Li, W\. Li, Y\. Li, Y\. Li, Z\. Li, Z\. Li, H\. Lin, X\. Lin, Z\. Lin, C\. Liu, C\. Liu, H\. Liu, J\. Liu, J\. Liu, L\. Liu, S\. Liu, T\. Y\. Liu, T\. Liu, W\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Y\. Liu, Z\. Liu, E\. Lu, L\. Lu, S\. Ma, X\. Ma, Y\. Ma, S\. Mao, J\. Mei, X\. Men, Y\. Miao, S\. Pan, Y\. Peng, R\. Qin, B\. Qu, Z\. Shang, L\. Shi, S\. Shi, F\. Song, J\. Su, Z\. Su, X\. Sun, F\. Sung, H\. Tang, J\. Tao, Q\. Teng, C\. Wang, D\. Wang, F\. Wang, H\. Wang, J\. Wang, J\. Wang, J\. Wang, S\. Wang, S\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Y\. Wang, Z\. Wang, Z\. Wang, Z\. Wang, C\. Wei, Q\. Wei, W\. Wu, X\. Wu, Y\. Wu, C\. Xiao, X\. Xie, W\. Xiong, B\. Xu, J\. Xu, J\. Xu, L\. H\. Xu, L\. Xu, S\. Xu, W\. Xu, X\. Xu, Y\. Xu, Z\. Xu, J\. Yan, Y\. Yan, X\. Yang, Y\. Yang, Z\. Yang, Z\. Yang, Z\. Yang, H\. Yao, X\. Yao, W\. Ye, Z\. Ye, B\. Yin, L\. Yu, E\. Yuan, H\. Yuan, M\. Yuan, H\. Zhan, D\. Zhang, H\. Zhang, W\. Zhang, X\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Y\. Zhang, Z\. Zhang, H\. Zhao, Y\. Zhao, H\. Zheng, S\. Zheng, J\. Zhou, X\. Zhou, Z\. Zhou, Z\. Zhu, W\. Zhuang, and X\. Zu \(2025\)Kimi k2: open agentic intelligence\.External Links:2507\.20534,[Link](https://arxiv.org/abs/2507.20534)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- Q\. Team \(2024\)Qwen1\.5\-moe: matching 7b model performance with 1/3 activated parameters"\.External Links:[Link](https://qwenlm.github.io/blog/qwen-moe/)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.09885#S4.p1.1)\.
- H\. Wu, H\. Chen, X\. Chen, Z\. Zhou, T\. Chen, Y\. Zhuang, G\. Lu, Z\. Huang, J\. Zhao, L\. Liu, Z\. Lan, B\. Yu, and J\. Li \(2025\)Grove moe: towards efficient and superior moe llms with adjugate experts\.External Links:2508\.07785,[Link](https://arxiv.org/abs/2508.07785)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Xie, Z\. Zhang, D\. Zhou, C\. Xie, Z\. Song, X\. Liu, Y\. Wang, X\. Lin, and A\. Xu \(2024\)MoE\-pruner: pruning mixture\-of\-experts large language model using the hints from its router\.External Links:2410\.12013,[Link](https://arxiv.org/abs/2410.12013)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1),[§4](https://arxiv.org/html/2606.09885#S4.p1.1)\.
- C\. Yang, Y\. Sui, J\. Xiao, L\. Huang, Y\. Gong, Y\. Duan, W\. Jia, M\. Yin, Y\. Cheng, and B\. Yuan \(2024a\)MoE\-i2: compressing mixture of experts models through inter\-expert pruning and intra\-expert low\-rank decomposition\.External Links:2411\.01016,[Link](https://arxiv.org/abs/2411.01016)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p3.1),[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Yang, S\. Qi, W\. Gu, C\. Wang, C\. Gao, and Z\. Xu \(2024b\)XMoE: sparse models with fine\-grained and adaptive expert selection\.External Links:2403\.18926,[Link](https://arxiv.org/abs/2403.18926)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Yao, C\. Liu, X\. Meng, Y\. Zhan, J\. Wu, S\. Pan, and W\. Hu \(2024\)DA\-moe: addressing depth\-sensitivity in graph\-level analysis through mixture of experts\.External Links:2411\.03025,[Link](https://arxiv.org/abs/2411.03025)Cited by:[§2](https://arxiv.org/html/2606.09885#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Zhu, Supryadi, S\. Xu, H\. Sun, L\. Pan, M\. Cui, J\. Du, R\. Jin, A\. Branco, and D\. Xiong \(2024\)Multilingual large language models: a systematic survey\.ArXivabs/2411\.11072\.External Links:[Link](https://api.semanticscholar.org/CorpusID:274131470)Cited by:[§1](https://arxiv.org/html/2606.09885#S1.p1.1)\.

ModelRetainGSM8KMBPPHumanevalARC\-EARC\-CAvg\.Qwen1\.5MoE\-A2\.7B0%51\.340\.628\.084\.674\.955\.8810%52\.942\.332\.385\.175\.057\.5220%53\.842\.932\.385\.174\.657\.7430%58\.345\.731\.185\.175\.159\.0640%57\.544\.530\.583\.073\.557\.8050%54\.342\.326\.881\.271\.055\.12DeepSeek\-V2\-Lite0%22\.133\.919\.577\.865\.743\.8010%36\.848\.028\.778\.164\.751\.2620%38\.445\.729\.979\.166\.851\.9830%38\.744\.526\.278\.566\.750\.9240%38\.640\.923\.876\.462\.648\.4650%25\.334\.613\.470\.956\.240\.08

Table 6:The performance of our TENP method on different datasets under varying important expert retention ratios, where Retain denotes the important expert retention ratio\.## Appendix AModel Detail

Qwen1\.5\-MoE\-A2\.7B contains 14 billion parameters and 24 layers\. Each layer consists of 60 routed experts and 4 shared experts\. For each token, the router selects the top 4 experts with the highest scores in each layer to perform the forward computation\.DeepSeek\-V2\-Lite contains 16 billion parameters and 27 layers, where the first layer is a dense layer\. Starting from the second layer, each layer includes 64 routed experts and 2 shared experts\. For each token, the router selects the top 6 experts with the highest scores in each layer to perform the forward computation\.

## Appendix BExpert Neuron Importance Algorithm

Algorithm[1](https://arxiv.org/html/2606.09885#alg1)describes in detail our method for selecting important neurons within an expert\. Given only the parameters of a specific expert and the input vectors, we can compute the importance of each neuron based on this information\. By leveraging PyTorch’s broadcasting mechanism in matrix multiplication, we are able to compute the average importance of all neurons in an expert across all tokens using a single forward pass\.

The algorithm illustrates that neuron importance is determined by computing the projection length of the output vector produced independently by each neuron onto the expert output, which is the superposition of the outputs of all neurons\. A simpler alternative is to directly evaluate the magnitude of each neuron’s output using itsL2L\_\{2\}norm\. In this case, it suffices to directly compute theL2L\_\{2\}norm ofCCin the algorithm\.

Algorithm 1Expert Neuron Importance Algorithmx←Input hidden statesx\\leftarrow\\textit\{Input hidden states\}⊳\\trianglerightx:L×dx:L\\times d

K←Neuron Number Of One ExpertK\\leftarrow\\textit\{Neuron Number Of One Expert \}

WG←Gate MatrixW\_\{G\}\\leftarrow\\textit\{Gate Matrix\}⊳\\trianglerightWG:K×dW\_\{G\}:K\\times d

WU←Up MatrixW\_\{U\}\\leftarrow\\textit\{Up Matrix\}⊳\\trianglerightWU:K×dW\_\{U\}:K\\times d

WD←Down MatrixW\_\{D\}\\leftarrow\\textit\{Down Matrix\}⊳\\trianglerightWD:d×KW\_\{D\}:d\\times K

y←Output hidden statesy\\leftarrow\\textit\{Output hidden states\}⊳\\trianglerighty:L×dy:L\\times d

P←Neuron ImportanceP\\leftarrow\\textit\{Neuron Importance\}⊳\\trianglerightP:KP:K

m←act​\(WG​x⊙WU​x\)m\\leftarrow\\mathrm\{act\}\(W\_\{G\}x\\odot W\_\{U\}x\)⊳\\trianglerightm:L×Km:L\\times K

y←WD​my\\leftarrow W\_\{D\}m

M←m⊤\.unsqueeze​\(−1\)M\\leftarrow m^\{\\top\}\.\\mathrm\{unsqueeze\}\(\-1\)⊳\\trianglerightM:K×L×1M:K\\times L\\times 1

𝐖D​2←WD⊤\.unsqueeze​\(1\)\\mathbf\{W\}\_\{D2\}\\leftarrow W\_\{D\}^\{\\top\}\.\\mathrm\{unsqueeze\}\(1\)⊳\\triangleright𝐖D​2:K×1×d\\mathbf\{W\}\_\{D2\}:K\\times 1\\times d

C←M​@​𝐖D​2C\\leftarrow M\\ @\\ \\mathbf\{W\}\_\{D2\}⊳\\trianglerightC:K×L×dC:K\\times L\\times d

Y←y\.unsqueeze​\(0\)Y\\leftarrow\\textit\{y\}\.\\mathrm\{unsqueeze\}\(0\)⊳\\trianglerightY:1×L×dY:1\\times L\\times d

s←\(C⊙Y\)\.sum​\(dim=−1\)s\\leftarrow\(C\\odot Y\)\.\\mathrm\{sum\}\(\\mathrm\{dim\}=\-1\)⊳\\trianglerights:K×Ls:K\\times L

r←Y\.norm​\(p=2,dim=−1\)r\\leftarrow Y\.\\mathrm\{norm\}\(p=2,\\mathrm\{dim\}=\-1\)⊳\\trianglerightr:K×Lr:K\\times L

ε←10−8\\varepsilon\\leftarrow 10^\{\-8\}

P←s/\(r\+ε\)P\\leftarrow s\\ /\\ \(r\+\\varepsilon\)⊳\\trianglerightP:K×LP:K\\times L

P←P\.mean​\(dim=1\)P\\leftarrow P\.\\mathrm\{mean\}\(\\mathrm\{dim\}=1\)⊳\\trianglerightP:KP:K

return

y,Py,P

![Refer to caption](https://arxiv.org/html/2606.09885v1/x5.png)Figure 5:The performance of different model pruning methods across various benchmarks under different sparsity levels\.
## Appendix CPruning at different levels of sparsification

To more comprehensively reflect the effectiveness of our method under different sparsity levels of routed experts, we design experiments with scales of 15%, 30%, 45%, 60%, 75%, and 90%\. As shown in Figure[5](https://arxiv.org/html/2606.09885#A2.F5), the experimental curves indicate that our method achieves the best performance on average across different sparsity levels\. At the scale of 90%, our method outperforms the full\-parameter model on nearly all benchmarks, with particularly significant improvements on code generation and mathematical reasoning tasks\. At the scale of 75%, our method performs on par with the full\-parameter model\. At the scale of 60%, the model performance is slightly lower than that of the full\-parameter model, yet still superior to other expert pruning methods\. At the scale of 45%, 30%, and 15%, our method continues to preserve the core capabilities of the model\.

## Appendix DThe Impact of Different Important Expert Retention Ratios on Pruning Performance

To evaluate the effect of different important expert retention ratios on pruning performance, we conduct experiments under a routed expert sparsity of 60% with varying retention ratios to determine the optimal setting\. As shown in Table[7](https://arxiv.org/html/2606.09885#A4.T7), the optimal important expert retention ratio for the Qwen model is 30%, corresponding to a retention ratio of unimportant expert neurons of\(60%−30%\)/\(100%−30%\)≈42\.86%\(60\\%\-30\\%\)/\(100\\%\-30\\%\)\\approx 42\.86\\%\. For the DeepSeek model, the optimal important\-expert retention ratio is 20%, with the corresponding retention ratio of unimportant expert neurons being\(60%−20%\)/\(100%−20%\)=50\.00%\(60\\%\-20\\%\)/\(100\\%\-20\\%\)=50\.00\\%\. When the important\-expert retention ratio is 0%, the TENP method degenerates into the ENP method\. From the table, we observe that as the retention ratio increases, accuracy first improves and then degrades, with the optimal retention ratios concentrated in the middle range\. By default, setting the parameter budget of important experts equal to that of unimportant experts yields favorable performance\.

ModelMethodETotal SparsityGSM8KMBPPHumanEvalARC\-EARC\-CAvg\.Qwen1\.5MoE\-A2\.7BENP60%35%51\.340\.628\.084\.674\.955\.88TENP\+BI\(MoDeGPT\)60%35%49\.139\.430\.585\.275\.355\.90TENP\(Rectange\)60%35%51\.139\.129\.285\.275\.355\.98TENP\+Rank\(ShortGPT\)60%35%49\.741\.328\.085\.775\.356\.00TENP \(Ours\)60%35%58\.345\.731\.185\.175\.159\.06

Table 7:A comparison of other layer selection methods with our methodModelMethodEGSM8KMBPPHumanEvalARC\-EARC\-CAvg\.Qwen3\-Next\-80B\-A3B\-InstructFull100\.00%93\.776\.784\.194\.893\.988\.64Random60\.00%85\.863\.364\.689\.988\.178\.34Frequency60\.00%87\.872\.273\.194\.091\.583\.72GatingScore60\.00%92\.674\.079\.894\.593\.186\.80EASY\-EP60\.00%93\.573\.381\.194\.593\.287\.12TENP60\.00%93\.675\.181\.194\.493\.387\.50

Table 8:The evaluation results of our method on Qwen3\-Next\-80B\-A3B\-Instruct at 60% expert sparsity\.
## Appendix EOther layer selection methods

We reproduced both methods on our model using their official open\-source implementations\. For MoDeGPT \(BI\), we computed BI scores with the released code and allocated the retained experts to each layer proportionally; when the allocated number exceeded a layer’s capacity, the overflow experts were re\-assigned to the layers with the smallest retention\. For ShortGPT ranking, we followed the open\-source procedure to estimate layer importance and then used the resulting ranking to reorder our trapezoidal per\-layer expert allocation\. The results indicate that, although layer evaluation criteria developed for dense models can affect expert allocation in MoE models, they do not lead to consistent gains under the same sparsity budget\. Specifically, the BI and ShortGPT based variants show slight advantages on language understanding tasks such as ARC\-E and ARC\-C, but perform noticeably worse on reasoning and code generation benchmarks, including GSM8K, MBPP, and HumanEval\. By contrast, our allocation strategy achieves the best overall average performance, while also delivering the strongest results on reasoning and code generation under the same sparsity constraint\.

## Appendix FPerformance under the 80B parameter setting

In addition to the two 14B and 16B models used in the main experiments, we also conducted experiments on the Qwen3\-Next\-80B\-A3B\-Instruct model\. Table[8](https://arxiv.org/html/2606.09885#A4.T8)further supplements the validation of the effectiveness of our method on large\-scale models\. As can be observed from the experimental results, as the number of model parameters increases, the level of redundancy also grows, leading to a smaller loss in model accuracy after pruning\. Notably, even random pruning achieves relatively strong performance, with an average score decrease of only about 10%\. In contrast, our methods, TENP and EASY\-EP, incur almost no degradation in model accuracy\.