MoRA:通过路由偏置学习与专家近似实现 MoE 剪枝

arXiv cs.LG 论文

摘要

MoRA 提出了一种结构化的 MoE 专家剪枝框架,使用可学习的路由偏置,结合语言模型损失与路由多样性正则项进行优化,并引入剪枝后的专家近似机制。在移除 25-50% 专家的设置下,在 Qwen3-30B-A3B、DeepSeek-V2-Lite 和 Moonlight-16B-A3B 上超越了当前最先进的剪枝方法。

arXiv:2610.00367v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) models enable parameter scaling with limited per-token computation by activating only a small subset of experts for each token, but deploying them still requires loading the complete expert pool into memory. Structured expert pruning can effectively reduce the memory usage by removing experts. However, existing pruning methods either use expert ranking criteria that are not well aligned with model performance or rely on effective expert subset searching that is computationally expensive. Moreover, these methods typically overlook the routing-behavior redundancy among the retained experts. In this paper, we propose MoE Pruning via Router Bias Learning and Expert Approximation (MoRA), a framework for structured MoE expert pruning. We introduce a learnable router bias for each expert and optimize these biases by minimizing the language-modeling loss and a routing-diversity regularizer. The learned router biases sharpen the routing probability distributions to identify experts critical to model performance while encouraging the selection of experts with diverse routing preferences. In addition, we introduce an expert approximation mechanism as a post-pruning enhancement. It leverages the remaining experts to approximate the outputs of pruned experts by affine transformation, further improving the performance of the pruned model. We evaluate MoRA on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B, removing 25\% and 50\% of the routed experts in each MoE layer. Extensive experiments on nine zero-shot benchmarks show that MoRA outperforms state-of-the-art pruning algorithms. Our code will be released.
查看原文
查看缓存全文

缓存时间: 2026/10/03 09:52

# MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Source: [https://arxiv.org/html/2610.00367](https://arxiv.org/html/2610.00367)
###### Abstract

Mixture\-of\-Experts \(MoE\) models enable parameter scaling with limited per\-token computation by activating only a small subset of experts for each token, but deploying them still requires loading the complete expert pool into memory\. Structured expert pruning can effectively reduce the memory usage by removing experts\. However, existing pruning methods either use expert ranking criteria that are not well aligned with model performance or rely on effective expert subset searching that is computationally expensive\. Moreover, these methods typically overlook the routing\-behavior redundancy among the retained experts\. In this paper, we propose MoE Pruning via Router Bias Learning and Expert Approximation \(MoRA\), a framework for structured MoE expert pruning\. We introduce a learnable router bias for each expert and optimize these biases by minimizing the language\-modeling loss and a routing\-diversity regularizer\. The learned router biases sharpen the routing probability distributions to identify experts critical to model performance while encouraging the selection of experts with diverse routing preferences\. In addition, we introduce an expert approximation mechanism as a post\-pruning enhancement\. It leverages the remaining experts to approximate the outputs of pruned experts by affine transformation, further improving the performance of the pruned model\. We evaluate MoRA on Qwen3\-30B\-A3B, DeepSeek\-V2\-Lite, and Moonlight\-16B\-A3B, removing 25% and 50% of the routed experts in each MoE layer\. Extensive experiments on nine zero\-shot benchmarks show that MoRA outperforms state\-of\-the\-art pruning algorithms\. Our code will be released\.

## 1Introduction

Mixture\-of\-Experts \(MoE\) models activate only a small subset of experts for each token, enabling model scaling with limited per\-token computation\([Shazeer et al\., 2017](https://arxiv.org/html/2610.00367#bib.bib1);[Fedus et al\., 2022](https://arxiv.org/html/2610.00367#bib.bib3)\)\. Although sparse activation reduces inference computation, deploying MoE models still requires loading the full expert pool, resulting in substantial memory usage\. Structured expert pruning offers a direct way to reduce this cost by removing less important experts while retaining the sparse routing structure\([Lee et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib10);[Xie et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib11);[Zhang et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib12)\)\. Hence, the core challenge is to determine which experts can be removed while preserving the predictive behavior of the model\.

Early expert pruning methods rank experts using aggregate routing statistics, such as activation frequency and average gate values\([Koishekenov et al\., 2023](https://arxiv.org/html/2610.00367#bib.bib17);[Xie et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib11)\)\. However, these expert ranking criteria may not directly align with the model predictive performance\. For example, frequency\-based pruning may assign higher scores to experts that frequently process common function words with less semantic information\([Dong et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib26)\), as illustrated in Figure[1](https://arxiv.org/html/2610.00367#S1.F1)\(a\)\. Recent methods select an expert subset by minimizing layer\-wise output reconstruction error on a calibration dataset under a prescribed retention budget\([Lu et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib14)\), as illustrated in Figure[1](https://arxiv.org/html/2610.00367#S1.F1)\(b\)\. However, these methods require evaluating candidate expert subsets through a set\-level search, whose cost grows rapidly with the number of experts\. Moreover, the above\-mentioned methods generally do not consider the diversity in the routing behavior of retained experts\. Consequently, they may retain experts with highly similar routing patterns, leading to redundancy within the retained set\.

Figure 1:Comparison of three ways to select retained MoE experts\. \(a\) Statistic\-based pruning assigns each expert a fixed score based on routing statistics and retains the highest\-scoring experts\. \(b\) Reconstruction\-based pruning evaluates candidate expert subsets using layer\-output reconstruction error and selects the subset with the lowest reconstruction error\. \(c\) MoRA adds a router bias to each expert to sharpen the routing probability distribution\. These biases are optimized to establish an expert ranking for subsequent selection\.To address these issues, we propose MoE pruning via Router bias learning and expert Approximation \(MoRA\), a framework for structured MoE expert pruning\. As shown in Figure[1](https://arxiv.org/html/2610.00367#S1.F1)\(c\), we first introduce a learnable router bias for each expert\. These experts are originally pretrained under the load\-balancing constraints\. The biases can sharpen routing probability distributions over experts and serve as scores for selecting experts\. To learn effective biases, we optimize them with the language\-modeling \(LM\) loss and a routing\-diversity regularizer\. The former encourages higher biases for experts critical to predictive performance, while the latter discourages experts with similar routing preferences from receiving high biases simultaneously\. Herein, similar routing preference means that experts tend to receive similar routing weights for the same tokens\.

We also introduce an expert approximation mechanism in our MoRA as a post\-pruning enhancement\. Specifically, we assign a retained expert as a proxy for each pruned expert and apply an affine transformation to approximate the output of the pruned expert with the retained experts, thereby mitigating the performance degradation caused by pruning\. We evaluate MoRA on Qwen3\-30B\-A3B[Yang et al\. \(2025\)](https://arxiv.org/html/2610.00367#bib.bib6), DeepSeek\-V2\-Lite[Liu et al\. \(2024\)](https://arxiv.org/html/2610.00367#bib.bib5), and Moonlight\-16B\-A3B[Liu et al\. \(2025\)](https://arxiv.org/html/2610.00367#bib.bib7)under two deployment budgets that remove 25% and 50% of the routed experts in each MoE layer\. We evaluate zero\-shot downstream performance on nine benchmarks\. Further analysis reveals that the association between the learned router biases and predictive performance is stronger in models with more experts and in deeper MoE layers\.

Our main contributions are as follows:

- •We propose MoRA, a structured MoE expert pruning framework that learns router biases using the LM loss and a routing\-diversity regularizer\. The learned biases serve as scores for selecting experts with high predictive importance and diverse routing preferences\.
- •We propose expert approximation as a post\-pruning enhancement, which uses lightweight affine transformations to approximate the outputs of pruned experts with the retained experts\. It effectively mitigates the performance degradation caused by pruning\.
- •We evaluate MoRA on Qwen3\-30B\-A3B, DeepSeek\-V2\-Lite, and Moonlight\-16B\-A3B at 25% and 50% routed\-expert pruning\. Results on nine zero\-shot benchmarks show that MoRA outperforms other pruning baselines while better preserving the predictive performance\.

## 2Related Work

Sparse MoE models\.Sparse MoE models increase model capacity while limiting per\-token computation by activating only a subset of experts\([Shazeer et al\., 2017](https://arxiv.org/html/2610.00367#bib.bib1)\)\. Large\-scale systems such as GShard\([Lepikhin et al\., 2020](https://arxiv.org/html/2610.00367#bib.bib2)\)and Switch Transformers\([Fedus et al\., 2022](https://arxiv.org/html/2610.00367#bib.bib3)\)established routing and training recipes for this regime\. Despite sparse activation, storing all expert parameters remains memory\-intensive\. This motivates expert pruning to reduce memory requirements while preserving predictive performance\.

MoE Pruning\.Weight\-level pruning methods, such as SparseGPT, Wanda, and MoE\-Pruner, can reduce MoE parameters by pruning individual weights within experts\([Frantar and Alistarh, 2023](https://arxiv.org/html/2610.00367#bib.bib28);[Sun et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib27);[Xie et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib11)\)\. However, these unstructured pruning methods directly reduce storage requirements and typically require specialized hardware support to achieve inference speedup\. Expert\-level pruning instead removes entire experts, directly reducing storage requirements\. Early methods rank experts using activation frequency and gating scores\([Koishekenov et al\., 2023](https://arxiv.org/html/2610.00367#bib.bib17)\)\. NAEE\([Lu et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib14)\)selects expert subsets that minimize the reconstruction loss between the original and pruned layer outputs\. However, enumerating the subsets leads to expensive computation as the number of experts and the pruning budget increase\. STUN\([Lee et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib10)\)avoids exhaustive subset evaluation through expert clustering and parameter\-space approximations\. Complementary approaches compress MoE models through expert merging\([Li et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib13);[Chen et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib15)\), or replacement with lightweight modules\([Zhang et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib12);[Hao et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib16)\)\. However, modifying or replacing expert functions can introduce additional approximation errors\.

## 3MoRA

In this section, we propose MoE pruning via Router bias learning and expert approximation \(MoRA\), a framework for structured MoE pruning\. We introduce a learnable router bias for each expert to sharpen the routing probability distributions, and optimize these biases using the LM loss and a routing\-diversity regularizer\. The resulting biases serve as expert selection scores that balance predictive importance and routing diversity\.

Then, MoRA performs expert approximation based on affine transformation as a post\-pruning enhancement\. Specifically, MoRA estimates the directed proxy fidelity between retained and removed experts\. It fits a channel\-wise affine transformation from each candidate retained\-expert output to the corresponding removed\-expert output and evaluates the resulting approximation quality\. For each removed expert, MoRA selects the retained expert with the highest proxy fidelity as its proxy\. MoRA then reuses the fitted affine map to approximate the output of pruned expert during inference\.

### 3\.1Preliminary

Consider an MoE layerℓ\\ellwithEℓE\_\{\\ell\}routed experts\. For a token representationhh, the router selects a top\-kℓk\_\{\\ell\}expert set𝒯ℓ​\(h\)\\mathcal\{T\}\_\{\\ell\}\(h\)\. The routed\-expert output is:

yℓ​\(h\)=∑e∈𝒯ℓ​\(h\)pℓ,e​\(h\)​fℓ,e​\(h\),y\_\{\\ell\}\(h\)=\\sum\_\{e\\in\\mathcal\{T\}\_\{\\ell\}\(h\)\}p\_\{\\ell,e\}\(h\)f\_\{\\ell,e\}\(h\),\(1\)wherefℓ,ef\_\{\\ell,e\}is the function implemented by expertee\.pℓ,e​\(h\)p\_\{\\ell,e\}\(h\)denotes the mixing weight applied to selected expertee\.

### 3\.2MoE pruning: router bias learning

The key challenge of MoE pruning is to retain an expert set that preserves the performance of the original MoE model\. A desirable retained set should satisfy two requirements\. First, it should preserve as much of the next\-token prediction capability as possible\. Second, the retained experts should exhibit diverse routing preferences, allowing a limited number of retained experts to cover complementary routing behaviors\.

Unlike methods pruning based on statistics or reconstruction, we propose a learning\-based expert pruning mechanism to efficiently identify experts that are critical to model performance\. During pretraining, sparse MoE models commonly regulate expert traffic through auxiliary load\-balancing objectives\([Yang et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib6);[Liu et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib5)\)or expert\-level correction biases\([Liu et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib7)\)to mitigate routing collapse and encourage more balanced expert utilization\. Building on this, we introduce an additional learnable router bias for each expert to sharpen routing probability distributions\. These biases are optimized with the LM loss and a routing\-diversity regularizer\. The learned biases therefore can serve as retention scores that balance predictive performance and routing diversity\.

#### 3\.2\.1Learnable Router Biases

MoRA augments the frozen pretrained router with one learnable router bias for each routed expert\. For experteein MoE layerℓ\\ell, letsℓ,e∈ℝs\_\{\\ell,e\}\\in\\mathbb\{R\}denote its router bias\. Given the hidden statehhof a token, we modify the corresponding pretrained logit as:

rℓ,e′​\(h\)=rℓ,e​\(h\)\+αℓ​sℓ,eτ,r^\{\\prime\}\_\{\\ell,e\}\(h\)=\\frac\{r\_\{\\ell,e\}\(h\)\+\\alpha\_\{\\ell\}s\_\{\\ell,e\}\}\{\\tau\},\(2\)whererℓ,e​\(h\)r\_\{\\ell,e\}\(h\)is the original router logit,αℓ\>0\\alpha\_\{\\ell\}\>0controls the strength of the bias, andτ\>0\\tau\>0is a calibration temperature\. The original logitrℓ,e​\(h\)r\_\{\\ell,e\}\(h\)remains dependent on the current token, whereassℓ,es\_\{\\ell,e\}provides a global expert preference across all tokens\. For a softmax router, letps​\(e∣h\)p\_\{s\}\(e\\mid h\)denote the normalized routing probability induced by the adjusted logitsrℓ,e′​\(h\)r^\{\\prime\}\_\{\\ell,e\}\(h\)before top\-kkselection\. For any two expertseeande′e^\{\\prime\}, their routing odds satisfy:

ps​\(e∣h\)ps​\(e′∣h\)=exp⁡\(rℓ,e​\(h\)−rℓ,e′​\(h\)\+αℓ​\(sℓ,e−sℓ,e′\)τ\)\.\\frac\{p\_\{s\}\(e\\mid h\)\}\{p\_\{s\}\(e^\{\\prime\}\\mid h\)\}=\\exp\\left\(\\frac\{r\_\{\\ell,e\}\(h\)\-r\_\{\\ell,e^\{\\prime\}\}\(h\)\+\\alpha\_\{\\ell\}\\left\(s\_\{\\ell,e\}\-s\_\{\\ell,e^\{\\prime\}\}\\right\)\}\{\\tau\}\\right\)\.\(3\)The learned bias therefore adjusts the router probability distributions\. A largersℓ,es\_\{\\ell,e\}leads to a higher routing probability for experteerelative to other experts\.

In contrast, existing statistic\-based pruning methods commonly estimate expert importance using activation frequency or average routing probability\([Koishekenov et al\., 2023](https://arxiv.org/html/2610.00367#bib.bib17);[Xie et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib11)\)\. These methods can result in a high importance score when the expert is associated with common and easily predictable tokens\. However, such experts may not be critical to the model’s predictive performance since these easily predictable tokens generally produce smaller language\-modeling losses and weaker gradient signals during training\.

#### 3\.2\.2Optimizing the Biases for Predictive Utility and Routing Diversity

MoRA learns the router biases using the LM loss on the calibration corpus\. The gradient of LM loss aggregates the effect of each expert on predictive performance across calibration tokens\.

Specifically, consider a softmax router with normalized selected weights, and a locally fixed top\-kkset𝒯ℓ​\(h\)\\mathcal\{T\}\_\{\\ell\}\(h\)\. For a selected expertee, the contribution through token representationhhto the gradient of LM loss with respect to its bias is:

Gℓ,e​\(h\)=αℓτ​pℓ,e​\(h\)​⟨gℓ​\(h\),fℓ,e​\(h\)−yℓ​\(h\)⟩,G\_\{\\ell,e\}\(h\)=\\frac\{\\alpha\_\{\\ell\}\}\{\\tau\}p\_\{\\ell,e\}\(h\)\\left\\langle g\_\{\\ell\}\(h\),f\_\{\\ell,e\}\(h\)\-y\_\{\\ell\}\(h\)\\right\\rangle,\(4\)wherepℓ,e​\(h\)p\_\{\\ell,e\}\(h\)is the routing weight derived from the adjusted logitsrℓ′​\(h\)r^\{\\prime\}\_\{\\ell\}\(h\),fℓ,e​\(h\)f\_\{\\ell,e\}\(h\)is the output of expertee, andyℓ​\(h\)y\_\{\\ell\}\(h\)is the routed\-expert mixture defined in Equation[1](https://arxiv.org/html/2610.00367#S3.E1)\. The vectorgℓ​\(h\)=∂ℒLM/∂yℓ​\(h\)g\_\{\\ell\}\(h\)=\\partial\\mathcal\{L\}\_\{\\mathrm\{LM\}\}/\\partial y\_\{\\ell\}\(h\)is computed from the full batch LM objective\.

A negative inner product indicates that moving this routed output toward expertee’s output locally decreases the LM loss, yielding a gradient contribution that favors increasing its bias under gradient descent\. A positive inner product gives the opposite contribution\. The actual bias update depends on the contributions across all tokens, the diversity regularizer, and the optimizer\.

Experts with similar routing preferences may receive high biases simultaneously, limiting the diversity of routing preferences represented in the retained set under a fixed expert budget\. To address this, we introduce a routing\-diversity regularizer based on the geometry of the pretrained router\. The regularizer suppresses the expert with the lower bias in each similar pair, encouraging the retained set to cover more distinct routing preferences\. Specifically, we measure the similarity between different experts using the cosine similarity of their corresponding pretrained router rows:

Cℓ,e,e′=\[max⁡\(0,cos⁡\(wℓ,e,wℓ,e′\)\)\]2\.C\_\{\\ell,e,e^\{\\prime\}\}=\\left\[\\max\\left\(0,\\cos\\left\(w\_\{\\ell,e\},w\_\{\\ell,e^\{\\prime\}\}\\right\)\\right\)\\right\]^\{2\}\.\(5\)Here,wℓ,ew\_\{\\ell,e\}denotes the pretrained router row associated with experteein layerℓ\\ell\. For each pair with different biases, we apply the similarity penalty only to the lower\-bias expert\. We define the diversity loss as:

ℒDiv=∑ℓ∈ℳ∑e≠e′Cℓ,e,e′​m​i​n​\(σ⁡\(sℓ,e\),σ⁡\(sℓ,e′\)\),\\mathcal\{L\}\_\{\\mathrm\{Div\}\}=\\sum\_\{\\ell\\in\\mathcal\{M\}\}\\sum\_\{e\\neq e^\{\\prime\}\}C\_\{\\ell,e,e^\{\\prime\}\}min\(\\sigma\(s\_\{\\ell,e\}\),\\sigma\(s\_\{\\ell,e^\{\\prime\}\}\)\),\(6\)whereℳ\\mathcal\{M\}denotes the set of MoE layer indices\. For a similar expert pair satisfyingσ⁡\(sℓ,e\)<σ⁡\(sℓ,e′\)\\sigma\(s\_\{\\ell,e\}\)<\\sigma\(s\_\{\\ell,e^\{\\prime\}\}\), minimizing this loss suppressesσ⁡\(sℓ,e\)\\sigma\(s\_\{\\ell,e\}\)without pushing downσ⁡\(sℓ,e′\)\\sigma\(s\_\{\\ell,e^\{\\prime\}\}\)through the same pair\. This encourages priority separation among experts with overlapping routing preferences\.

We optimize the learnable biasess=\{sℓ,e\}s=\\\{s\_\{\\ell,e\}\\\}using a joint objective that combines the LM loss with the diversity regularization:

ℒbias=ℒLM\+λd​ℒDiv,\\mathcal\{L\}\_\{\\mathrm\{bias\}\}=\\mathcal\{L\}\_\{\\mathrm\{LM\}\}\+\\lambda\_\{\\mathrm\{d\}\}\\mathcal\{L\}\_\{\\mathrm\{Div\}\},\(7\)whereλd≥0\\lambda\_\{\\mathrm\{d\}\}\\geq 0controls the strength of the diversity regularization\.

#### 3\.2\.3Expert Selection from Learned Router Biases

After calibration, we rank the routed experts in each MoE layer according to their learned router biases to jointly consider the predictive performance and routing diversity\. Letℰℓ=\{1,…,Eℓ\}\\mathcal\{E\}\_\{\\ell\}=\\\{1,\\ldots,E\_\{\\ell\}\\\}denote the original expert set in layerℓ\\ell, and letBℓB\_\{\\ell\}denote the number of experts to retain\. The retained and removed sets are:

Sℓ=TopBℓ⁡\(sℓ,1,…,sℓ,Eℓ\),S¯ℓ=ℰ−Sℓ,S\_\{\\ell\}=\\operatorname\{Top\}\_\{B\_\{\\ell\}\}\\left\(s\_\{\\ell,1\},\\ldots,s\_\{\\ell,E\_\{\\ell\}\}\\right\),\\qquad\\overline\{S\}\_\{\\ell\}=\\mathcal\{E\}\-S\_\{\\ell\},\(8\)whereTopBℓ\\operatorname\{Top\}\_\{B\_\{\\ell\}\}returns the indices of theBℓB\_\{\\ell\}largest values\. Experts inS¯ℓ\\overline\{S\}\_\{\\ell\}are physically removed\. The learned biases are then discarded and introduce no inference overhead\.

### 3\.3Post\-pruning Enhancement: Expert Approximation

Although expert pruning reduces physical storage, removing experts discards their output contributions and may degrade model performance\. To mitigate this degradation, MoRA fits a channel\-wise affine map from each retained candidate to each removed target and evaluates its output\-approximation fidelity\. Each removed expert is assigned the retained candidate with the highest fidelity, and the same fitted affine map is then used to transform the selected proxy output during inference\.

#### 3\.3\.1Directed Proxy Fidelity Estimation

We divide the calibration corpus into two non\-overlapping subsets: a fit split𝒟fit\\mathcal\{D\}\_\{\\mathrm\{fit\}\}and a scoring split𝒟score\\mathcal\{D\}\_\{\\mathrm\{score\}\}\. The fit split is used to estimate the pairwise affine maps, while the scoring split is used to evaluate their approximation fidelity\. For a removed target experti∈S¯ℓi\\in\\overline\{S\}\_\{\\ell\}and a retained candidate expertj∈Sℓj\\in S\_\{\\ell\}, MoRA predicts the target output using:

f^ℓ,i←j​\(h\)=a^ℓ,i,j⊙fℓ,j​\(h\)\+b^ℓ,i,j,\\widehat\{f\}\_\{\\ell,i\\leftarrow j\}\(h\)=\\widehat\{a\}\_\{\\ell,i,j\}\\odot f\_\{\\ell,j\}\(h\)\+\\widehat\{b\}\_\{\\ell,i,j\},\(9\)wherea^ℓ,i,j\\widehat\{a\}\_\{\\ell,i,j\}andb^ℓ,i,j\\widehat\{b\}\_\{\\ell,i,j\}are the fitted channel\-wise scale and bias\. These parameters are estimated on𝒟fit\\mathcal\{D\}\_\{\\mathrm\{fit\}\}using weighted ridge regression\.

We derive target\-conditioned token weightsωs,t,i\\omega\_\{s,t,i\}from the pretrained router, assigning larger weights to tokens that are more relevant to target expertii\. Then we evaluate the affine predictor on𝒟score\\mathcal\{D\}\_\{\\mathrm\{score\}\}using the weighted coefficient of determination:

Rℓ,i,j2=1−∑\(s,t\)∈𝒟scoreωs,t,i​‖fℓ,i​\(hs,t\)−f^ℓ,i←j​\(hs,t\)‖22∑\(s,t\)∈𝒟scoreωs,t,i​‖fℓ,i​\(hs,t\)−μℓ,iscore‖22,qℓ,i,j=max⁡\(0,Rℓ,i,j2\),R^\{2\}\_\{\\ell,i,j\}=1\-\\frac\{\\sum\_\{\(s,t\)\\in\\mathcal\{D\}\_\{\\mathrm\{score\}\}\}\\omega\_\{s,t,i\}\\left\\\|f\_\{\\ell,i\}\(h\_\{s,t\}\)\-\\widehat\{f\}\_\{\\ell,i\\leftarrow j\}\(h\_\{s,t\}\)\\right\\\|\_\{2\}^\{2\}\}\{\\sum\_\{\(s,t\)\\in\\mathcal\{D\}\_\{\\mathrm\{score\}\}\}\\omega\_\{s,t,i\}\\left\\\|f\_\{\\ell,i\}\(h\_\{s,t\}\)\-\\mu\_\{\\ell,i\}^\{\\mathrm\{score\}\}\\right\\\|\_\{2\}^\{2\}\},\\qquad q\_\{\\ell,i,j\}=\\max\\left\(0,R^\{2\}\_\{\\ell,i,j\}\\right\),\(10\)whereμℓ,iscore\\mu\_\{\\ell,i\}^\{\\mathrm\{score\}\}is the weighted mean output of target expertiion the scoring split\. A largerqℓ,i,jq\_\{\\ell,i,j\}indicates that the affine\-transformed output of retained expertjjmore accurately approximates the output variation of removed expertii\. The relation is directional because the affine map predictsiifromjj, and both the predictor and token weights are conditioned on target expertii\.

Each removed expert is assigned the retained candidate with the highest proxy fidelity:

jℓ​\(i\)=arg⁡maxj∈Sℓ​qℓ,i,j\.j\_\{\\ell\}\(i\)=\\underset\{j\\in S\_\{\\ell\}\}\{\\arg\\max\}\\;q\_\{\\ell,i,j\}\.\(11\)

#### 3\.3\.2Expert Approximation based on Affine Transformation

After the proxy assignment is fixed, MoRA performs the expert approximation based on affine transformation\. For a removed expertiiand its assigned retained proxyjℓ​\(i\)j\_\{\\ell\}\(i\), MoRA replaces the removed expert output withf^ℓ,i←jℓ​\(i\)​\(h\)\\widehat\{f\}\_\{\\ell,i\\leftarrow j\_\{\\ell\}\(i\)\}\(h\)from Equation[9](https://arxiv.org/html/2610.00367#S3.E9)\. To limit the effect of unreliable affine approximations, we enable this proxy only when the selected proxy fidelity satisfiesqℓ,i,jℓ​\(i\)\>qminq\_\{\\ell,i,j\_\{\\ell\}\(i\)\}\>q\_\{\\mathrm\{min\}\}, whereqminq\_\{\\mathrm\{min\}\}is a prescribed threshold\.

No additional optimization is required after proxy assignment\. The exported model stores only the retained experts, the target\-proxy mappings, and the fitted channel\-wise affine parameters\. The removed expert parameters and proxy fidelity scores are discarded\. Each pruned expert requires only2​do2d\_\{o\}affine parameters, wheredod\_\{o\}is the expert output dimension, corresponding to one scale and one bias per output channel\. This is far fewer than the number of parameters in the original expert\.

## 4Experiments

### 4\.1Experimental Setup

Models and pruning budgets\.We evaluate Qwen3\-30B\-A3B\([Yang et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib6)\), DeepSeek\-V2\-Lite\([Liu et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib5)\), and Moonlight\-16B\-A3B\([Liu et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib7)\)\. Qwen3 has 48 MoE layers with 128 routed experts per layer and activates eight experts per token\. DeepSeek\-V2\-Lite and Moonlight each have 26 MoE layers with 64 routed experts, two shared experts, and six activated routed experts per token\. We remove either 25% or 50% of the routed experts in every MoE layer, retaining 96/64 experts for Qwen3 and 48/32 experts for the other two models\. Shared experts and all non\-expert parameters are preserved, and the original routing top\-kkis unchanged\.

Baselines\.Activation Countranks routed experts according to how often they are selected by the pretrained router on the calibration corpus and retains the highest\-ranked experts in each layer\.Average Gate Massranks experts by their normalized top\-kkrouting weights averaged over all calibration tokens, incorporating both selection frequency and routing confidence\([Koishekenov et al\., 2023](https://arxiv.org/html/2610.00367#bib.bib17);[Xie et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib11)\)\.MoNEcombines expert access frequency with output fluctuation to identify important experts and replaces less important experts with lightweight approximations\([Zhang et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib12)\)\.MC\-SMoEgroups experts according to the similarity of their router\-logit patterns and constructs representative experts through activation\-frequency\-weighted merging\([Li et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib13)\)\.HC\-SMoEhierarchically clusters experts using their output features and merges the experts within each cluster\([Chen et al\., 2025](https://arxiv.org/html/2610.00367#bib.bib15)\)\.

Implementation Details\.Following the common practice in existing MoE pruning methods\([Ling et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib18);[Zhang et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib12);[Lu et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib14)\), we conduct experiments using C4\([Raffel et al\., 2020](https://arxiv.org/html/2610.00367#bib.bib8)\)as the calibration dataset\. The router biases are learned on C4 with 1000 iterations for Qwen3\-30B\-A3B, DeepSeek\-V2\-Lite and Moonlight\-16B\-A3B\. For post\-pruning enhancement, we divide the sampled C4 calibration corpus into two non\-overlapping subsets of equal size: Each subset contains 64 sequences of 1024 tokens\. We set the hyperparametersλd=0\.8\\lambda\_\{\\mathrm\{d\}\}=0\.8andqmin=0\.4q\_\{\\mathrm\{min\}\}=0\.4\.

Evaluation benchmarks\.Following the previous MoE pruning methods\([Zhang et al\., 2026](https://arxiv.org/html/2610.00367#bib.bib12);[Lu et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib14);[Ling et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib18);[An et al\., 2024](https://arxiv.org/html/2610.00367#bib.bib4)\), we use lm\-evaluation\-harness\([Gao et al\., 2021](https://arxiv.org/html/2610.00367#bib.bib9)\)for zero\-shot evaluation on BoolQ\([Clark et al\., 2019](https://arxiv.org/html/2610.00367#bib.bib20)\), RTE\([Wang et al\., 2019](https://arxiv.org/html/2610.00367#bib.bib21)\), HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.00367#bib.bib22)\), WinoGrande\([Sakaguchi et al\., 2021](https://arxiv.org/html/2610.00367#bib.bib25)\), PIQA\([Bisk et al\., 2020](https://arxiv.org/html/2610.00367#bib.bib23)\), OpenBookQA\([Mihaylov et al\., 2018](https://arxiv.org/html/2610.00367#bib.bib24)\), ARC\-Easy\([Clark et al\., 2018](https://arxiv.org/html/2610.00367#bib.bib19)\), ARC\-Challenge\([Clark et al\., 2018](https://arxiv.org/html/2610.00367#bib.bib19)\), and MMLU\. HellaSwag, PIQA, OpenBookQA, ARC\-Easy, and ARC\-Challenge useacc\_norm; the other tasks useacc\. Avg\. is the mean of these nine percentages\.

### 4\.2Main Results

Comparison with baselines\.Tables[1](https://arxiv.org/html/2610.00367#S4.T1),[2](https://arxiv.org/html/2610.00367#S4.T2), and[3](https://arxiv.org/html/2610.00367#S4.T3)compare MoRA with five pruning baselines on nine zero\-shot benchmarks\. MoRA achieves the highest average score across all three architectures at both 25% and 50% pruning ratios\. Compared with the strongest baseline in each setting, MoRA improves the average score by 1\.14, 0\.43, and 1\.01 percentage points at 25% pruning, and by 6\.43, 0\.56, and 0\.97 points at 50% pruning, on Qwen3\-30B\-A3B, DeepSeek\-V2\-Lite, and Moonlight\-16B\-A3B, respectively\. We attribute the weaker performance of HC\-SMoE on Qwen3 to a potential mismatch between its clustering criterion, based on average output similarity, and the high degree of expert specialization in Qwen3\. Merging experts under this aggregate criterion may reduce the diversity of expert outputs\.

Table 1:Zero\-shot results \(%\) on Qwen3\-30B\-A3B with C4 as the calibration dataset\. Bold marks the best result among MoRA and the baselines for each column at each pruning ratio\.Table 2:Zero\-shot results \(%\) on DeepSeek\-V2\-Lite with C4 as the calibration dataset\. Bold marks the best result among MoRA and the baselines for each column at each pruning ratio\.Table 3:Zero\-shot results \(%\) on Moonlight\-16B\-A3B with C4 as the calibration dataset\. Bold marks the best result among MoRA and the baselines for each column at each pruning ratio\.Compression efficiency\.As shown in Table[4](https://arxiv.org/html/2610.00367#S4.T4), MoRA preserves most of the original performance at 25% expert pruning, with average score drops of only 0\.60, 2\.37, and 2\.59 percentage points on Qwen3\-30B\-A3B, DeepSeek\-V2\-Lite, and Moonlight\-16B\-A3B, respectively\. The affine transformations require less than 0\.08% of the retained parameter count across all evaluated settings\. Their additional inference FLOPs are bounded by0\.0005%0\.0005\\%of those of the original model, enabling post\-pruning enhancement with negligible parameter and computational overhead\.

Table 4:Compression efficiency of MoRA\. We report the model parameter counts and storage, and the additional overhead of the affine transformations\.
### 4\.3Ablation Study

Table 5:Component ablation on Qwen3\-30B\-A3B\. Avg\. denotes the mean score across nine benchmarks\.##### Effect of the key components\.

We evaluate the effect of the key components of MoRA, containing two objectives and expert approximation\. As shown in Table[5](https://arxiv.org/html/2610.00367#S4.T5), jointly optimizing the LM and diversity losses achieves average scores of 72\.17 and 65\.30 at 25% and 50% pruning, respectively\. Removing the diversity loss reduces these scores by 4\.84 and 7\.22 points, while removing the LM loss causes drops of 4\.69 and 10\.73 points\. The larger degradation without the LM loss at 50% pruning suggests that predictive utility becomes more important under a tighter retention budget\. The diversity objective discourages overlapping routing preferences but does not assess the contribution to next\-token prediction of each expert\. Consequently, selecting experts solely for diversity may discard predictively useful experts, with fewer retained experts available to compensate for their removal at higher pruning ratios\.

Pruning with learned biases versus adjusted routing probabilities\.

Table[6](https://arxiv.org/html/2610.00367#S4.T6)shows that computing gate mass from the bias\-adjusted router improves average performance over the original gate mass in all six settings\. However, ranking experts by adjusted gate mass still underperforms direct bias\-based selection, particularly at 50% pruning on Qwen3 and DeepSeek\-V2\-Lite, where the gaps reach 6\.49 and 2\.64 percentage points, respectively\. These results indicate that aggregating routing probabilities after bias calibration remains less effective than directly ranking experts by the learned biases, particularly at high pruning ratios\. This suggests that expert utilization, even under the calibrated router, does not fully capture predictive importance\.

Table 6:Mean scores \(%\) across nine zero\-shot benchmarks\. Average Gate Mass uses the pretrained router, whereasr′r^\{\\prime\}Gate Mass uses the bias\-adjusted routing probabilities\. We report the average score of MoRA after pruning without expert approximation\.

### 4\.4Discussion

#### 4\.4\.1Learned Router Biases and Expert Deletion Sensitivity

We investigate whether learned router biases reflect the model’s sensitivity to expert removal\. On Qwen3\-30B\-A3B, we use a randomly sampled 10% subset of the C4 calibration tokens and measure the increase in next\-token prediction loss after masking one expert at a time\. All pretrained parameters remain fixed, and the router recomputes top\-kkselection over the remaining experts\.

Within each layer, we analyze the top 50% of experts ranked by learned bias and, separately, by activation frequency\. Figure[2](https://arxiv.org/html/2610.00367#S4.F2)presents scatter plots and Spearman correlations between each score and the loss increase caused by expert deletion\. Learned biases exhibit a clear positive correlation with deletion sensitivity, with stronger correlations in deeper MoE layers, whereas activation frequency shows a less consistent relationship\.

![Refer to caption](https://arxiv.org/html/2610.00367v1/figure3.png)Figure 2:Relationship between expert scores and deletion\-induced loss increases on Qwen3\-30B\-A3B\. Each analysis uses the top 50% of experts within a layer ranked by the corresponding score\. Spearman correlations for learned biases and activation frequency are denoted byρs\\rho\_\{s\}andρa\\rho\_\{a\}, respectively\.
#### 4\.4\.2Token\-category routing analysis

We analyze token\-category routing patterns in Qwen3\-30B\-A3B using 128 samples from C4\. Specifically, we record expert assignments under the pretrained router and visualize token\-category routing probabilities with experts ordered by learned bias or activation frequency\. Figure[3](https://arxiv.org/html/2610.00367#S4.F3)shows that content words tend to be routed to experts with higher learned biases, whereas numbers and punctuation tend to be routed to experts with higher activation frequencies\.

![Refer to caption](https://arxiv.org/html/2610.00367v1/figure2.png)Figure 3:Heatmap visualization of token\-category routing probabilities in Qwen3 at Layers 0, 24, and 47\. Rows correspond to word categories, and darker cells indicate larger routing probabilities\. Experts are ordered from left to right by decreasing activation frequency \(a\) and learned biases \(b\)\.

## 5Conclusion

In this paper, we propose MoRA, a framework for structured MoE expert pruning\. We introduce a learnable router bias for each expert that sharpens the routing probability distributions\. Experts that are critical to the predictive performance tend to receive higher learned biases and are therefore prioritized for retention\. The retained experts are then used to approximate the outputs of the pruned experts through channel\-wise affine transformations\. Extensive experiments across three MoE architectures demonstrate that MoRA outperforms the evaluated baselines and preserves predictive performance with minimal additional parameter overhead\.

## References

- Anet al\.\(2024\)Y\. An, X\. Zhao, T\. Yu, M\. Tang, and J\. WangFluctuation\-based adaptive structured pruning for large language models\.InProceedings of the AAAI conference on artificial intelligence,Vol\.38,pp\. 10865–10873\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Chenet al\.\(2025\)I\. Chen, H\. Liu, W\. Sun, C\. Chao, Y\. Hsu, and C\. LeeRetraining\-free merging of sparse moe via hierarchical clustering\.InProceedings of the 42nd International Conference on Machine Learning,Vol\.267,pp\. 8594–8620\.Cited by:[§2](https://arxiv.org/html/2610.00367#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p2.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolq: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),pp\. 2924–2936\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Donget al\.\(2026\)Z\. Dong, H\. Peng, P\. Liu, X\. Zhao, D\. Wu, F\. Xiao, and Z\. WangDomain\-specific pruning of large mixture\-of\-experts models with few\-shot demonstrations\.Advances in Neural Information Processing Systems38,pp\. 103552–103577\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p2.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p1.1),[§2](https://arxiv.org/html/2610.00367#S2.p1.1)\.
- Frantar and Alistarh \(2023\)E\. Frantar and D\. AlistarhSparsegpt: massive language models can be accurately pruned in one\-shot\.InInternational conference on machine learning,pp\. 10323–10337\.Cited by:[§2](https://arxiv.org/html/2610.00367#S2.p2.1)\.
- Gaoet al\.\(2021\)L\. Gao, J\. Tow, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, K\. McDonell, N\. Muennighoff,et al\.A framework for few\-shot language model evaluation\.Zenodo\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Haoet al\.\(2026\)J\. Hao, Z\. Hao, J\. Guo, L\. Shen, Y\. Luo, H\. Hu, and D\. ZengLightMoE: reducing mixture\-of\-experts redundancy through expert replacing\.arXiv preprint arXiv:2603\.12645\.Cited by:[§2](https://arxiv.org/html/2610.00367#S2.p2.1)\.
- Koishekenovet al\.\(2023\)Y\. Koishekenov, A\. Berard, and V\. NikoulinaMemory\-efficient nllb\-200: language\-specific expert pruning of a massively multilingual machine translation model\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3567–3585\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p2.1),[§2](https://arxiv.org/html/2610.00367#S2.p2.1),[§3\.2\.1](https://arxiv.org/html/2610.00367#S3.SS2.SSS1.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p2.1)\.
- Leeet al\.\(2025\)J\. Lee, S\. Hwang, A\. Qiao, D\. F\. Campos, Z\. Yao, and Y\. HeStun: structured\-then\-unstructured pruning for scalable moe pruning\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 13660–13676\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p1.1),[§2](https://arxiv.org/html/2610.00367#S2.p2.1)\.
- Lepikhinet al\.\(2020\)D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. ChenGshard: scaling giant models with conditional computation and automatic sharding\.arXiv preprint arXiv:2006\.16668\.Cited by:[§2](https://arxiv.org/html/2610.00367#S2.p1.1)\.
- Liet al\.\(2024\)P\. Li, Z\. Zhang, P\. Yadav, Y\. Sung, Y\. Cheng, M\. Bansal, and T\. ChenMerge, then compress: demystify efficient smoe with hints from its routing policy\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 14234–14256\.Cited by:[§2](https://arxiv.org/html/2610.00367#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p2.1)\.
- Linget al\.\(2024\)G\. Ling, Z\. Wang, Y\. Yan, and Q\. LiuSlimgpt: layer\-wise structured pruning for large language models\.Advances in Neural Information Processing Systems37,pp\. 107112–107137\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Liuet al\.\(2024\)A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo,et al\.Deepseek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.00367#S3.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p1.1)\.
- Liuet al\.\(2025\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.00367#S3.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p1.1)\.
- Luet al\.\(2024\)X\. Lu, Q\. Liu, Y\. Xu, A\. Zhou, S\. Huang, B\. Zhang, J\. Yan, and H\. LiNot all experts are equal: efficient expert pruning and skipping for mixture\-of\-experts large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6159–6172\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p2.1),[§2](https://arxiv.org/html/2610.00367#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Mihaylovet al\.\(2018\)T\. Mihaylov, P\. Clark, T\. Khot, and A\. SabharwalCan a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2381–2391\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p3.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p1.1),[§2](https://arxiv.org/html/2610.00367#S2.p1.1)\.
- Sunet al\.\(2024\)M\. Sun, Z\. Liu, A\. Bair, and Z\. KolterA simple and effective pruning approach for large language models\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 4942–4964\.Cited by:[§2](https://arxiv.org/html/2610.00367#S2.p2.1)\.
- Wanget al\.\(2019\)A\. Wang, Y\. Pruksachatkun, N\. Nangia, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. BowmanSuperglue: a stickier benchmark for general\-purpose language understanding systems\.Advances in neural information processing systems32\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Xieet al\.\(2024\)Y\. Xie, Z\. Zhang, D\. Zhou, C\. Xie, Z\. Song, X\. Liu, Y\. Wang, X\. Lin, and A\. XuMoe\-pruner: pruning mixture\-of\-experts large language model using the hints from its router\.arXiv preprint arXiv:2410\.12013\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p1.1),[§1](https://arxiv.org/html/2610.00367#S1.p2.1),[§2](https://arxiv.org/html/2610.00367#S2.p2.1),[§3\.2\.1](https://arxiv.org/html/2610.00367#S3.SS2.SSS1.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p4.1),[§3\.2](https://arxiv.org/html/2610.00367#S3.SS2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaswag: can a machine really finish your sentence?\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 4791–4800\.Cited by:[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.
- Zhanget al\.\(2026\)G\. Zhang, H\. Yuxuan, Y\. Lou, Y\. Zhang, W\. Zhao, and Y\. YouMone: replacing redundant experts with lightweight novices for structured pruning of moe\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 39983–40009\.Cited by:[§1](https://arxiv.org/html/2610.00367#S1.p1.1),[§2](https://arxiv.org/html/2610.00367#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2610.00367#S4.SS1.p4.1)\.

相似文章

面向混合专家语言模型的重构优化专家剪枝

arXiv cs.LG

论文提出 OptEP,一种面向混合专家(MoE)语言模型的后训练专家剪枝方法,通过连续松弛优化专家选择变量并结合重构驱动的权重更新,在 Qwen3、GLM-4.5-Air、Mixtral 等模型上显著优于现有剪枝基线,激进剪枝下精度提升最高达 16.20%。