BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
Summary
BEAM introduces binary expert activation masking for dynamic routing in Mixture-of-Experts LLMs, achieving up to 85% FLOPs reduction with minimal performance loss and up to 2.5× faster decoding.
arXiv:2605.14438v1 Announce Type: new
Abstract: Mixture-of-Experts (MoE) architectures enhance the efficiency of large language models by activating only a subset of experts per token. However, standard MoE employs a fixed Top-K routing strategy, leading to redundant computation and suboptimal inference latency. Existing acceleration methods either require costly retraining with architectural changes or suffer from severe performance drop at high sparsity due to train-inference mismatch. To address these limitations, we propose BEAM (Binary Expert Activation Masking), a novel method that learns token-adaptive expert selection via trainable binary masks. With a straight-through estimator and an auxiliary regularization loss, BEAM induces dynamic expert sparsity through end-to-end training while maintaining model capability. We further implement an efficient custom CUDA kernel for BEAM, ensuring seamless integration with the vLLM inference framework. Experiments show that BEAM retains over 98\% of the original model's performance while reducing MoE layer FLOPs by up to 85\%, achieving up to 2.5$\times$ faster decoding and 1.4$\times$ higher throughput, demonstrating its effectiveness as a practical, plug-and-play solution for efficient MoE inference.
View Cached Full Text
Cached at: 05/15/26, 06:24 AM
# BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE Source: [https://arxiv.org/html/2605.14438](https://arxiv.org/html/2605.14438) Juntong Wu1,2,\*, Jialiang Cheng1,\*, 🖂, Qishen Yin2, Yue Dai1, & Yuliang Yan1, Fuyu Lv1, Ou Dan1, Li Yuan2, 🖂 1Taobao & Tmall Group of Alibaba 2Shenzhen Graduate School, Peking University Correspondence:[jichen\.cjl@alibaba\-inc\.com](https://arxiv.org/html/2605.14438v1/mailto:[email protected]),[yuanli\-ece@pku\.edu\.cn](https://arxiv.org/html/2605.14438v1/mailto:[email protected]) ###### Abstract Mixture\-of\-Experts \(MoE\) architectures enhance the efficiency of large language models by activating only a subset of experts per token\. However, standard MoE employs a fixed Top\-K routing strategy, leading to redundant computation and suboptimal inference latency\. Existing acceleration methods either require costly retraining with architectural changes or suffer from severe performance drop at high sparsity due to train\-inference mismatch\. To address these limitations, we proposeBEAM\(Binary Expert Activation Masking\), a novel method that learns token\-adaptive expert selection via trainable binary masks\. With a straight\-through estimator and an auxiliary regularization loss, BEAM induces dynamic expert sparsity through end\-to\-end training while maintaining model capability\. We further implement an efficient custom CUDA kernel for BEAM, ensuring seamless integration with the vLLM inference framework\. Experiments show that BEAM retains over 98% of the original model’s performance while reducing MoE layer FLOPs by up to 85%, achieving up to 2\.5×\\timesfaster decoding and 1\.4×\\timeshigher throughput, demonstrating its effectiveness as a practical, plug\-and\-play solution for efficient MoE inference\. Code implementation of BEAM can be found in[https://github\.com/Time\-Rune/BEAM](https://github.com/Time-Rune/BEAM)\. ## 1Introduction Mixture\-of\-Experts \(MoE\) enables efficient scaling through sparse activation, where each token is processed by only a small subset of specialized feed\-forward network \(FFN\) experts\(Yanget al\.,[2025a](https://arxiv.org/html/2605.14438#bib.bib1); Liuet al\.,[2024a](https://arxiv.org/html/2605.14438#bib.bib2); Jianget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib3)\)\. Figure 1:Performance–sparsity trade\-off of BEAM and baselines on Qwen3\-30B\-A3B\.The dominant paradigm for expert selection is the fixed Top\-K routing mechanism, which selects the K experts with the highest router logits for each token\(Shazeeret al\.,[2017](https://arxiv.org/html/2605.14438#bib.bib4); Lepikhinet al\.,[2020](https://arxiv.org/html/2605.14438#bib.bib5)\)\. While simple and widely adopted, it ignores token\-level complexity, leading to redundant computation for simple tokens\(Huanget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib6); Zenget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib7)\)\. This inefficiency ultimately limits the potential for faster MoE model inference\. To address the inefficiency of fixed Top\-K routing, recent work has explored dynamic expert activation, falling into three categories\. The first modifies the routing logits to enable token\-adaptive expert counts\(Huanget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib6); Luet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib8); Yanget al\.,[2024b](https://arxiv.org/html/2605.14438#bib.bib9); Aghdamet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib10); Guoet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib11)\), but fails to skip redundant high\-weight experts and enforces a minimum activation floor, limiting achievable sparsity\. The second introduces special experts such as zero\-computation null experts to control sparsity\(Zenget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib7); Jinet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib12); Guiet al\.,[2025](https://arxiv.org/html/2605.14438#bib.bib13)\), yet requires additional hyperparameters and complicated fine\-tuning process, and only enables passive, indirect sparsity control\. The third merges or prunes experts statically\(Chenet al\.,[2025](https://arxiv.org/html/2605.14438#bib.bib14); Liuet al\.,[2024b](https://arxiv.org/html/2605.14438#bib.bib15); Yanget al\.,[2024a](https://arxiv.org/html/2605.14438#bib.bib16)\), but cannot adapt to input complexity at inference time and often suffers from severe performance degradation at high sparsity levels\. Figure 2:Vanilla Top\-KKvs\. BEAM: BEAM learns a binary mask over Top\-KKcandidates for token\-adaptive activation\.In this work, we proposeBEAM\(BinaryExpertActivationMasking\), a novel dynamic routing framework designed to achieve extreme expert sparsity and inference speedups in MoE models\. As shown in Figure[2](https://arxiv.org/html/2605.14438#S1.F2), BEAM introduces a lightweight, learnable mask router that generates a binary mask applied to the top\-K candidate experts from the primary router, selectively deactivating redundant ones\. Sparsity is encouraged via an auxiliary regularization loss, and gradients are propagated through the binary mask using the straight\-through estimator \(STE\)\(Bengioet al\.,[2013](https://arxiv.org/html/2605.14438#bib.bib18)\)\. Crucially, BEAM decouples sparsity control from expert selection\. The primary router still handles load balancing and expert choice, while the mask router solely determines activation count\. This separation avoids conflicts and enables more activation patterns within the Top\-K candidate set, providing fine\-grained, token\-adaptive sparsity control that fixed Top\-K or logits\-based methods cannot express\. To demonstrate the practical impact, we integrate BEAM into vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib17)\)through a custom CUDA kernel, requiring only a single\-line change and delivering significant real\-world speedups, which makes BEAM a practical, plug\-and\-play solution for efficient MoE deployment\. Our contributions are summarized as follows: - •We propose BEAM, a novel dynamic routing framework that achieves extreme expert sparsity via a learnable mask router\. It directly prunes redundant experts from the Top\-K set for token\-adaptive computation, in contrast to existing indirect or post\-hoc approaches\. - •We provide a practical, plug\-and\-play deployment solution by integrating BEAM into vLLM through a custom CUDA kernel, requiring minimal code changes\. - •Extensive experiments show BEAM preserves over 98% of performance while reducing MoE layer FLOPs by up to 85% \(Figure[1](https://arxiv.org/html/2605.14438#S1.F1)\), yielding 1\.4×\\timeshigher throughput and 2\.5×\\timesfaster decoding\. ## 2Related Work Routing Logits ModificationThese methods modify routing logits to enable token\-adaptive expert counts\. MoE\-Dynamic\(Huanget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib6)\)and XMoE\(Yanget al\.,[2024b](https://arxiv.org/html/2605.14438#bib.bib9)\)activate experts until the cumulative probability exceeds a threshold\. DTop\-p\(Jinet al\.,[2025](https://arxiv.org/html/2605.14438#bib.bib20)\)improves MoE\-Dynamic by replacing the fixed threshold with a learnable sparsity controller\. Adaptive Gating\(Liet al\.,[2023b](https://arxiv.org/html/2605.14438#bib.bib19)\)and NAEE\(Luet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib8)\)dynamically switches between Top\-1 and Top\-2 based on the gap between the top two logits\. DA\-MoE\(Aghdamet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib10)\)computes token importance from attention scores to allocate a dynamic Top\-K\. DynMoE\(Guoet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib11)\)replaces the softmax router with per\-expert sigmoid gates\. MaskMoE\(Suet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib37)\)employs static vocabulary\-based masks derived from pretraining data distributions to improve rare\-token expert assignment\. However, most of them rely on the unverified heuristic that low entropy of routing logits implies fewer needed experts, fail to skip redundant high\-weight experts, and require at least one active expert, preventing acceleration\. Special ExpertsThese methods reduce FLOPs by routing tokens to experts that incur no computation\. AdaMoE\(Zenget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib7)\)introduces null experts that outputs zero\. LongCat\(Guiet al\.,[2025](https://arxiv.org/html/2605.14438#bib.bib13)\)uses zero\-computation experts that return the input as their output\. MoE\+\+\(Jinet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib12)\)extended this idea with three types of zero\-computation experts\. However, these methods introduce extra hyperparameters and achieve sparsity indirectly via passive placeholder routing rather than explicit expert minimization, undermining plug\-and\-play usability\. Static Expert Merging and PruningThese training\-free methods reduce redundancy by merging or pruning experts\. DEK\(Zhanget al\.,[2025](https://arxiv.org/html/2605.14438#bib.bib21)\)groups similar experts in feature space and merges experts in each group\. EEP\(Liuet al\.,[2024b](https://arxiv.org/html/2605.14438#bib.bib15)\)utilizes a gradient\-free evolutionary search to determine pruning and merging patterns\. MC\-SMoE\(Liet al\.,[2023c](https://arxiv.org/html/2605.14438#bib.bib22)\)leverages routing statistics to guide expert merging and decomposes the merged experts into low\-rank and structural sparse alternatives\. HC\-SMoE\(Chenet al\.,[2025](https://arxiv.org/html/2605.14438#bib.bib14)\)applies hierarchical clustering on expert outputs to merge experts\. However, these methods cannot adapt to the varying complexity of input tokens at inference time and often suffer performance degradation under high compression\. ## 3Method ### 3\.1Preliminaries and Motivation MoE replaces dense FFN layers withNNexpert networks\{ℰ1,…,ℰN\}\\\{\\mathcal\{E\}\_\{1\},\\dots,\\mathcal\{E\}\_\{N\}\\\}and a routerℛ\\mathcal\{R\}\. Given an input token𝐱∈ℝdh\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{h\}\}, the router computes logits𝐫=ℛ\(𝐱\)∈ℝN\\mathbf\{r\}=\\mathcal\{R\}\(\\mathbf\{x\}\)\\in\\mathbb\{R\}^\{N\}, which are converted into routing weights via softmax\. Under standard Top\-K routing, only the K experts with the largest routing logits are activated\. Specifically, theTop\-K\(⋅\)\\mathrm\{Top\}\\text\{\-\}K\(\\cdot\)operator retains the K largest values in𝐫\\mathbf\{r\}and sets the remaining entries to−∞\-\\infty, yielding routing weights: 𝐠i=Softmax\(Top\-K\(𝐫\)\)i\.\\mathbf\{g\}\_\{i\}=\\mathrm\{Softmax\}\(\\mathrm\{Top\}\\text\{\-\}K\(\\mathbf\{r\}\)\)\_\{i\}\.\(1\)The MoE output is a weighted sum of expert outputs: 𝐲=∑i=1N𝐠i⋅ℰi\(𝐱\),\\mathbf\{y\}=\\sum\_\{i=1\}^\{N\}\\mathbf\{g\}\_\{i\}\\cdot\\mathcal\{E\}\_\{i\}\(\\mathbf\{x\}\),\(2\)where each expertℰi\\mathcal\{E\}\_\{i\}typically follows a Gated Linear Unit \(GLU\) structure: ℰi\(𝐱\)=\(δ\(𝐱𝐖gate\(i\)\)⊙\(𝐱𝐖up\(i\)\)\)𝐖down\(i\)\.\\mathcal\{E\}\_\{i\}\(\\mathbf\{x\}\)=\\left\(\\delta\(\\mathbf\{x\}\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{\(i\)\}\)\\odot\(\\mathbf\{x\}\\mathbf\{W\}\_\{\\mathrm\{up\}\}^\{\(i\)\}\)\\right\)\\mathbf\{W\}\_\{\\mathrm\{down\}\}^\{\(i\)\}\.\(3\) Although Top\-K routing enables scalable training, it assigns a uniform computational budget to all tokens, causing redundancy for simple ones\. Existing dynamic routing methods attempt to address this problem but remain limited in practice\. First, these approaches implicitly treat routing rank as a proxy for expert importance\. However, a lower\-ranked expert can still be critical for a given token while a high\-weight one may be redundant, which is empirically validated in Section[5\.2](https://arxiv.org/html/2605.14438#S5.SS2)and Appendix[B\.4](https://arxiv.org/html/2605.14438#A2.SS4)\. Second, cumulative probability thresholds and null experts cannot actively prune redundant experts, limiting compression ratios \(Section[4\.2](https://arxiv.org/html/2605.14438#S4.SS2)\)\. Third, these methods entangle expert selection, load balancing, and sparsity control in a single router, creating inherent gradient conflicts, thereby degrading model capacity \(Section[4\.2](https://arxiv.org/html/2605.14438#S4.SS2)\)\. ### 3\.2BEAM: Binary Expert Activation Masking Figure 3:The illustration of our proposed BEAM method with 4 experts and K=3 as an example\.The above limitations motivate BEAM, which enables token\-adaptive expert activation by introducing a lightweight and learnable mask router that generates a binary mask to selectively deactivate redundant experts from the standard Top\-K candidate set, as shown in Figure[3](https://arxiv.org/html/2605.14438#S3.F3)\. Formally, given an input token embedding𝐱∈ℝdh\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\_\{h\}\}, BEAM operates in four steps\. Step 1: Standard Top\-K Routing\.The primary routerℛ\\mathcal\{R\}computes logits𝐫=ℛ\(𝐱\)∈ℝN\\mathbf\{r\}=\\mathcal\{R\}\(\\mathbf\{x\}\)\\in\\mathbb\{R\}^\{N\}, whereNNis the total number of experts\. TheTop\-K\(⋅\)\\mathrm\{Top\}\\text\{\-\}K\(\\cdot\)operator retains the K largest values and sets the rest to−∞\-\\infty\. The normalized routing weights are computed as: 𝐠i=Softmax\(Top\-K\(𝐫\)\)i,i=1,…,N,\\mathbf\{g\}\_\{i\}=\\mathrm\{Softmax\}\(\\mathrm\{Top\}\\text\{\-\}K\(\\mathbf\{r\}\)\)\_\{i\},\\quad i=1,\\dots,N,\(4\)where𝐠i\>0\\mathbf\{g\}\_\{i\}\>0only for the top K experts and∑i=1N𝐠i=1\\sum\_\{i=1\}^\{N\}\\mathbf\{g\}\_\{i\}=1\. Step 2: Raw Mask Generation\.A lightweight auxiliary mask router, parameterized by𝐖m∈ℝdh×N\\mathbf\{W\}\_\{m\}\\in\\mathbb\{R\}^\{d\_\{h\}\\times N\}, processes the same input token𝐱\\mathbf\{x\}to generate a raw mask𝐦^\\hat\{\\mathbf\{m\}\}\. We apply a Sigmoid activationσ\\sigmato constrain the raw mask values to the range\(0,1\)\(0,1\): 𝐦^=σ\(𝐱𝐖m\)\.\\hat\{\\mathbf\{m\}\}=\\sigma\(\\mathbf\{x\}\\mathbf\{W\}\_\{m\}\)\.\(5\)𝐦^\\hat\{\\mathbf\{m\}\}reflects the model’s confidence in the necessity of each expert for the current token\. Step 3: Binary Masking\.We binarize the raw mask𝐦^\\hat\{\\mathbf\{m\}\}using a fixed threshold ofτ=0\.5\\tau=0\.5to obtain a discrete mask𝐦∈\{0,1\}N\\mathbf\{m\}\\in\\\{0,1\\\}^\{N\}: 𝐦i=\{1,if𝐦^i≥0\.5,0,otherwise,\.\\mathbf\{m\}\_\{i\}=\\begin\{cases\}1,&\\text\{if \}\\hat\{\\mathbf\{m\}\}\_\{i\}\\geq 0\.5,\\\\ 0,&\\text\{otherwise\},\\end\{cases\}\.\(6\)Since𝐦i=0\\mathbf\{m\}\_\{i\}=0disables expertiiregardless of its Top\-K status, the number of activated experts per token can be possibly reduced to0\. Step 4: Masked Aggregation\.The final routing weights𝐠^\\hat\{\\mathbf\{g\}\}are obtained by performing an element\-wise multiplication between the Top\-K weights𝐠\\mathbf\{g\}and the binary mask𝐦\\mathbf\{m\}: 𝐠^=𝐠⊙𝐦,\\hat\{\\mathbf\{g\}\}=\\mathbf\{g\}\\odot\\mathbf\{m\},\(7\)and the layer output is computed by aggregating the masked activations: 𝐲=∑i=1N𝐠^i⋅ℰi\(𝐱\)\.\\mathbf\{y\}=\\sum\_\{i=1\}^\{N\}\\hat\{\\mathbf\{g\}\}\_\{i\}\\cdot\\mathcal\{E\}\_\{i\}\(\\mathbf\{x\}\)\.\(8\) This design provides three key advantages\. First, it decouples routing and sparsification,i\.e\., the primary router handles expert selection and load balancing, while the mask router focuses exclusively on redundancy elimination, avoiding conflicting optimization objectives\. Second, expert sparsity is learned end\-to\-end without manual tuning, enabling aggressive expert reduction while preserving model capability\. Third, the binary mask provides a hardware\-friendly signal that can be directly leveraged by custom CUDA kernels, facilitating efficient real\-world deployment\. ### 3\.3Training Strategy BEAM is trained end\-to\-end using two key components\. The first is the Straight\-Through Estimator \(STE\) to handle the non\-differentiable binarization operation\. The second is an auxiliary sparsity regularization loss added to the standard MoE objective to jointly optimize task performance, expert load balancing, and computational efficiency\. #### 3\.3\.1Straight\-Through Estimator The binary mask𝐦\\mathbf\{m\}is generated via a non\-differentiable hard thresholding function as defined in Equation[6](https://arxiv.org/html/2605.14438#S3.E6)\. To enable the mask router to be trained via backpropagation, we adopt the STE method\(Bengioet al\.,[2013](https://arxiv.org/html/2605.14438#bib.bib18)\)to approximate the gradient\. Specifically, during the backward pass, the threshold function is treated as an identity mapping, allowing the gradient of the lossℒ\\mathcal\{L\}with respect to𝐦\\mathbf\{m\}to be propagated directly to the raw mask𝐦^\\hat\{\\mathbf\{m\}\}: ∂ℒ∂𝐦^≈∂ℒ∂𝐦\.\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\hat\{\\mathbf\{m\}\}\}\\approx\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\mathbf\{m\}\}\.\(9\)This allows the mask router to be trained with backpropagation despite the discrete nature of𝐦\\mathbf\{m\}\. Note that all Top\-K experts are computed regardless of𝐦\\mathbf\{m\}to ensure proper gradient flow during training\. To ensure stable training, we initialize the mask router parameters to zero\. This yields𝐦^=0\.5\\hat\{\\mathbf\{m\}\}=0\.5and𝐦=1\\mathbf\{m\}=1for all experts at the start of training, which preserves the original Top\-K behavior and allows sparsity to emerge gradually as training proceeds\. #### 3\.3\.2Sparsity\-Guided Optimization The total training loss combines three terms\. In addition to the standard language modeling lossℒlm\\mathcal\{L\}\_\{lm\}and the expert load\-balancing lossℒbal\\mathcal\{L\}\_\{bal\}, we introduce an auxiliary sparsity regularization lossℒreg\\mathcal\{L\}\_\{reg\}, defined as theL1L\_\{1\}norm of the raw mask restricted to the Top\-K candidate set𝒯K\\mathcal\{T\}\_\{K\}: ℒreg=1K∑i∈𝒯K\|𝐦^i\|\.\\mathcal\{L\}\_\{reg\}=\\frac\{1\}\{K\}\\sum\_\{i\\in\\mathcal\{T\}\_\{K\}\}\|\\hat\{\\mathbf\{m\}\}\_\{i\}\|\.\(10\)ℒreg\\mathcal\{L\}\_\{reg\}directly encourages the mask router to suppress redundant experts among selected candidates without introducing extraneous gradients for non\-selected experts\. The overall objective is a weighted sum: ℒ=ℒlm\+αℒbal\+βℒreg,\\mathcal\{L\}=\\mathcal\{L\}\_\{lm\}\+\\alpha\\mathcal\{L\}\_\{bal\}\+\\beta\\mathcal\{L\}\_\{reg\},\(11\)whereα\\alphaandβ\\betaare hyperparameters that control the balance between expert utilization and computational efficiency\. Through this sparsity\-guided optimization, BEAM learns to activate only the necessary experts for each token, achieving high inference speed without compromising performance\. ### 3\.4Theoretical Analysis We provide a theoretical analysis of BEAM’s training dynamics\. The core operation is the masked routing weight𝐠^=𝐠⊙𝐦\\hat\{\\mathbf\{g\}\}=\\mathbf\{g\}\\odot\\mathbf\{m\}, where𝐠\\mathbf\{g\}is the output of the primary router and𝐦=𝕀\(𝐦^≥0\.5\)\\mathbf\{m\}=\\mathbb\{I\}\(\\hat\{\\mathbf\{m\}\}\\geq 0\.5\)is the binary mask derived from𝐦^=σ\(𝐚\)\\hat\{\\mathbf\{m\}\}=\\sigma\(\\mathbf\{a\}\), with𝐚=𝐱𝐖m\\mathbf\{a\}=\\mathbf\{x\}\\mathbf\{W\}\_\{m\}being the mask router pre\-activation\. The mask router receives gradients from two sources: the task lossℒlm\\mathcal\{L\}\_\{lm\}propagated through the masked routing weights𝐠^\\hat\{\\mathbf\{g\}\}via STE, and the sparsity regularizationℒreg\\mathcal\{L\}\_\{reg\}applied directly to the Top\-K mask values\. The load\-balancing lossℒbal\\mathcal\{L\}\_\{bal\}is computed solely from the primary router’s weights𝐠\\mathbf\{g\}before masking and does not produce gradients for the mask router\. ###### Definition 3\.1\(Gradient for Mask Router\)\. Under STE, the full gradient ofℒ\\mathcal\{L\}with respect to the mask router pre\-activation𝐚\\mathbf\{a\}is: \(∇𝐚ℒ\)i=\(∂ℒlm∂g^i⋅𝐠i\+βK⋅𝟏\[i∈𝒯K\]\)σ′\(ai\),\\left\(\\nabla\_\{\\mathbf\{a\}\}\\mathcal\{L\}\\right\)\_\{i\}=\\left\(\\frac\{\\partial\\mathcal\{L\}\_\{lm\}\}\{\\partial\\hat\{g\}\_\{i\}\}\\cdot\\mathbf\{g\}\_\{i\}\+\\frac\{\\beta\}\{K\}\\cdot\\mathbf\{1\}\_\{\[i\\in\\mathcal\{T\}\_\{K\}\]\}\\right\)\\sigma^\{\\prime\}\(a\_\{i\}\),\(12\)where𝒯K\\mathcal\{T\}\_\{K\}denotes the Top\-K candidate set and𝟏\[i∈𝒯K\]\\mathbf\{1\}\_\{\[i\\in\\mathcal\{T\}\_\{K\}\]\}is its indicator function\. ###### Theorem 3\.2\(Selective Gradient Propagation\)\. The gradient in Equation[12](https://arxiv.org/html/2605.14438#S3.E12)satisfies: 𝐠i=0⇒\(∇𝐚ℒ\)i=0,\\displaystyle\\mathbf\{g\}\_\{i\}=0\\quad\\Rightarrow\\quad\\left\(\\nabla\_\{\\mathbf\{a\}\}\\mathcal\{L\}\\right\)\_\{i\}=0,\(13\)𝐠i\>0⇒\(∇𝐚ℒ\)i=\(∂ℒlm∂g^i⋅𝐠i\+βK\)σ′\(ai\)\.\\displaystyle\\mathbf\{g\}\_\{i\}\>0\\quad\\Rightarrow\\quad\\left\(\\nabla\_\{\\mathbf\{a\}\}\\mathcal\{L\}\\right\)\_\{i\}=\\left\(\\frac\{\\partial\\mathcal\{L\}\_\{lm\}\}\{\\partial\\hat\{g\}\_\{i\}\}\\cdot\\mathbf\{g\}\_\{i\}\+\\frac\{\\beta\}\{K\}\\right\)\\sigma^\{\\prime\}\(a\_\{i\}\)\.\(14\) ###### Proof\. Since𝐦^i=σ\(ai\)∈\(0,1\)\\hat\{\\mathbf\{m\}\}\_\{i\}=\\sigma\(a\_\{i\}\)\\in\(0,1\), the L1 gradient simplifies to∂\|𝐦^i\|/∂𝐦^i=1\\partial\|\\hat\{\\mathbf\{m\}\}\_\{i\}\|/\\partial\\hat\{\\mathbf\{m\}\}\_\{i\}=1\. Note thatσ′\(ai\)\>0\\sigma^\{\\prime\}\(a\_\{i\}\)\>0for allai∈ℝa\_\{i\}\\in\\mathbb\{R\}\. Case 1:If𝐠i=0\\mathbf\{g\}\_\{i\}=0, theni∉𝒯Ki\\notin\\mathcal\{T\}\_\{K\}\. The task\-loss term vanishes because𝐠i=0\\mathbf\{g\}\_\{i\}=0, and the regularization term vanishes becauseℒreg\\mathcal\{L\}\_\{reg\}is restricted to𝒯K\\mathcal\{T\}\_\{K\}\. Hence\(∇𝐚ℒ\)i=0\\left\(\\nabla\_\{\\mathbf\{a\}\}\\mathcal\{L\}\\right\)\_\{i\}=0, and the mask router receives no learning signal for non\-selected experts\. Case 2:If𝐠i\>0\\mathbf\{g\}\_\{i\}\>0, theni∈𝒯Ki\\in\\mathcal\{T\}\_\{K\}and both terms contribute\. The gradient direction is determined by the sign of∂ℒlm∂g^i⋅𝐠i\+βK\\frac\{\\partial\\mathcal\{L\}\_\{lm\}\}\{\\partial\\hat\{g\}\_\{i\}\}\\cdot\\mathbf\{g\}\_\{i\}\+\\frac\{\\beta\}\{K\}: the task\-loss term drivesaia\_\{i\}toward values that reduceℒlm\\mathcal\{L\}\_\{lm\}, while the constantβK\\frac\{\\beta\}\{K\}consistently pushesaia\_\{i\}downward to encourage sparsity\. Expertiiis retained when its task contribution outweighs the sparsity pressure \(∂ℒlm∂g^i⋅𝐠i<−βK\\frac\{\\partial\\mathcal\{L\}\_\{lm\}\}\{\\partial\\hat\{g\}\_\{i\}\}\\cdot\\mathbf\{g\}\_\{i\}<\-\\frac\{\\beta\}\{K\}\), and pruned otherwise\. The hyperparameterβ\\betadirectly controls this trade\-off\. ∎ Further analysis of full expert masking behaviour and details of the efficient BEAM implementation in vLLM are provided in Appendix[A\.3](https://arxiv.org/html/2605.14438#A1.SS3)and Appendix[A\.4](https://arxiv.org/html/2605.14438#A1.SS4), respectively\. ## 4Experiments ### 4\.1Experimental Setup Models and Training DataWe evaluate BEAM on three representative MoE models: Qwen1\.5‑MoE‑A2\.7B\(Baiet al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib23)\), DeepSeekV2‑Lite\(Liuet al\.,[2024a](https://arxiv.org/html/2605.14438#bib.bib2)\), and Qwen3‑30B‑A3B\(Yanget al\.,[2025a](https://arxiv.org/html/2605.14438#bib.bib1)\)\. We conduct supervised fine\-tuning using the Tulu 3 SFT Mixture Dataset\(Lambertet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib24)\), which covers reasoning, coding, and general knowledge tasks\. All baselines and BEAM are fine\-tuned on the same dataset with identical training configurations to ensure fair comparison\. BaselinesWe compare against five methods: \(1\)Top\-K Reducedtrains with a smaller Top\-K\. \(2\)Top\-K Pruningtrains with the original Top\-K and reduces Top\-K at inference\. \(3\)MoE\-Dynamic\(Huanget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib6)\)activates experts until cumulative routing probability exceeds thresholdϕ\\phi\. \(4\)AdaMoE\(Zenget al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib7)\)adds null experts with zero computation\. \(5\)DynMoE\(Guoet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib11)\)uses sigmoid router to adaptively determine activated experts\. Evaluation BenchmarksFor accuracy evaluation, we use eight benchmarks from OpenCompass\(Contributors,[2023](https://arxiv.org/html/2605.14438#bib.bib25)\)across three domains:Reasoning\(Math\(Hendryckset al\.,[2021b](https://arxiv.org/html/2605.14438#bib.bib26)\), GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2605.14438#bib.bib28)\)\), HumanEval\(H\_Eval\)\(Chenet al\.,[2021](https://arxiv.org/html/2605.14438#bib.bib27)\)\),Knowledge\(MMLU\(Hendryckset al\.,[2021a](https://arxiv.org/html/2605.14438#bib.bib29)\), CEVAL\(Huanget al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib35)\), CMMLU\(Liet al\.,[2023a](https://arxiv.org/html/2605.14438#bib.bib34)\)\), andCommon Sense\(CommonsenseQA\(CSQA\)\(Talmoret al\.,[2019](https://arxiv.org/html/2605.14438#bib.bib31)\), BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2605.14438#bib.bib32)\)\)\. For acceleration evaluation, we report Time per Output Token \(TPOT\), Time to First Token \(TTFT\), and throughput under varying QPS using vLLM\(Kwonet al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib17)\)\. All models run on a single NVIDIA H20 GPU with fixed input/output lengths of128/32128/32tokens and50005000test samples\. HyperparametersFor MoE\-Dynamic, AdaMoE, and BEAM, we tune their respective hyperparameters,i\.e\., cumulative probability thresholdϕ\\phi, null expert count, and L1 loss coefficientβ\\beta, to match comparable sparsity levels with other methods at each setting\. All experiments are conducted on NVIDIA H20 GPUs under identical hyperparameter settings, as detailed in Appendix[B\.1](https://arxiv.org/html/2605.14438#A2.SS1)\. ### 4\.2Performance Comparison Table 1:Performance comparison on Qwen1\.5\-MoE\-A2\.7B under different sparsity levels\. Best results within each sparsity group are marked inbold\.ReasoningKnowledgeCommonSenseMethods \\TasksAvg\. KMATHGSM8KH\_EvalMMLUCEVALCMMLUBoolQCSQAAvg\.\(Acc\.↑\\uparrow\)Qwen1\.5\-MoEK=4\{\}\_\{~\\text\{K=4\}\}4\.0023\.0457\.4750\.6159\.2874\.1575\.1872\.6381\.3361\.71Mid SparsityTop\-K PruningK=2\{\}\_\{~\\text\{K=2\}\}2\.0022\.4049\.3643\.9058\.6970\.7671\.4062\.9780\.5157\.50Top\-K ReducedK=2\{\}\_\{~\\text\{K=2\}\}2\.0021\.9853\.6851\.8358\.8170\.5371\.5077\.5280\.3460\.77MoE\-Dynamicϕ=0\.4\{\}\_\{~\\text\{$\\phi$=0\.4\}\}2\.2020\.9451\.8647\.5657\.8567\.3567\.8473\.8280\.3458\.45AdaMoENull=60\{\}\_\{~\\text\{Null=60\}\}1\.5317\.9253\.9846\.9547\.8946\.9547\.8272\.0562\.9849\.57BEAM\(β=0\.01\)\{\}\_\{~\(\\beta=0\.01\)\}1\.5624\.7855\.5053\.0558\.7570\.4769\.1878\.3280\.8461\.36High SparsityTop\-K PruningK=1\{\}\_\{~\\text\{K=1\}\}1\.009\.7834\.1225\.6153\.8460\.6259\.4652\.4574\.1246\.25Top\-K ReducedK=1\{\}\_\{~\\text\{K=1\}\}1\.0018\.8249\.2041\.4653\.9760\.7060\.7872\.2676\.4154\.20MoE\-Dynamicϕ=0\.2\{\}\_\{~\\text\{$\\phi$=0\.2\}\}1\.4717\.2245\.6442\.0753\.2958\.4658\.8574\.2574\.6153\.05AdaMoENull=120\{\}\_\{~\\text\{Null=120\}\}1\.2615\.7647\.3842\.3246\.5541\.1143\.5764\.7164\.3745\.72BEAM\(β=0\.1\)\{\}\_\{~\(\\beta=0\.1\)\}0\.5623\.5455\.0449\.3958\.0570\.2267\.5272\.6979\.7759\.53Extreme SparsityBEAM\(β=1\.0\)\{\}\_\{~\(\\beta=1\.0\)\}0\.1118\.7251\.1842\.0754\.1758\.9757\.1569\.1169\.9452\.66 Table 2:Performance comparison on Qwen3\-30B\-A3B under different sparsity levels\. Best results within each sparsity group are marked inbold\.ReasoningKnowledgeCommonSenseMethods \\TasksAvg\. KMATHGSM8KH\_EvalMMLUCEVALCMMLUBoolQCSQAAvg\.\(Acc\.↑\\uparrow\)Qwen3\-30B\-A3BK=8\{\}\_\{~\\text\{K=8\}\}8\.0058\.2888\.0282\.9381\.8083\.5683\.6986\.7686\.2481\.41Mid SparsityTop\-K PruningK=4\{\}\_\{~\\text\{K=4\}\}4\.0048\.7649\.5176\.8373\.4975\.8576\.2781\.0475\.0269\.60Top\-K ReducedK=4\{\}\_\{~\\text\{K=4\}\}4\.0056\.4484\.4680\.4978\.2780\.4078\.2787\.6885\.1878\.90MoE\-Dynamicϕ=0\.3\{\}\_\{~\\text\{$\\phi$=0\.3\}\}5\.0454\.2882\.0376\.2277\.8678\.1778\.9387\.2285\.1877\.49AdaMoENull=128\{\}\_\{~\\text\{Null=128\}\}4\.0241\.6862\.4469\.9372\.5162\.7163\.3984\.7177\.1566\.81BEAM\(β=0\.01\)\{\}\_\{~\(\\beta=0\.01\)\}4\.2355\.1685\.5281\.7180\.0981\.4681\.5388\.0786\.4079\.99High SparsityTop\-K PruningK=2\{\}\_\{~\\text\{K=2\}\}2\.000\.681\.140\.0025\.3917\.5116\.4217\.3116\.8711\.92Top\-K ReducedK=2\{\}\_\{~\\text\{K=2\}\}2\.0044\.6077\.7972\.5672\.9972\.6372\.9983\.8580\.0272\.18MoE\-Dynamicϕ=0\.1\{\}\_\{~\\text\{$\\phi$=0\.1\}\}1\.7440\.1075\.3672\.3867\.5964\.3863\.6781\.0778\.8767\.93AdaMoENull=256\{\}\_\{~\\text\{Null=256\}\}2\.6434\.0673\.0158\.5444\.5038\.7537\.7661\.0460\.4451\.01BEAM\(β=0\.1\)\{\}\_\{~\(\\beta=0\.1\)\}1\.2355\.4485\.9081\.1076\.0674\.0474\.5485\.7584\.2877\.14Extreme SparsityTop\-K ReducedK=1\{\}\_\{~\\text\{K=1\}\}1\.0027\.9663\.0851\.2258\.2652\.9753\.0556\.2168\.8853\.95BEAM\(β=1\.0\)\{\}\_\{~\(\\beta=1\.0\)\}0\.5652\.2081\.9677\.4469\.4466\.2869\.3678\.4780\.1071\.91 Table 3:Performance comparison on DeepSeekV2\-Lite under different sparsity levels\. Best results within each sparsity group are marked inbold\.ReasoningKnowledgeCommonSenseMethods \\TasksAvg\. KMATHGSM8KH\_EvalMMLUCEVALCMMLUBoolQCSQAAvg\.\(Acc\.↑\\uparrow\)DeepSeekV2\-LiteK=6\{\}\_\{~\\text\{K=6\}\}6\.0020\.0262\.7043\.9055\.0455\.2660\.9375\.2068\.1455\.15Mid SparsityTop\-K PruningK=4\{\}\_\{~\\text\{K=4\}\}4\.0015\.1057\.0937\.8046\.6853\.2560\.0069\.6967\.4050\.88Top\-K ReducedK=4\{\}\_\{~\\text\{K=4\}\}4\.0016\.5857\.2440\.8554\.7055\.8960\.4076\.3972\.4854\.32MoE\-Dynamicϕ=0\.3\{\}\_\{~\\text\{$\\phi$=0\.3\}\}4\.3119\.0835\.6338\.3542\.9553\.3658\.1263\.5570\.6047\.70AdaMoENull=64\{\}\_\{~\\text\{Null=64\}\}3\.2512\.0037\.7625\.6753\.0157\.2858\.8562\.3269\.8647\.09BEAM\(β=0\.01\)\{\}\_\{~\(\\beta=0\.01\)\}2\.6120\.3660\.2746\.9556\.6554\.1260\.4276\.5765\.1155\.06High SparsityTop\-K PruningK=2\{\}\_\{~\\text\{K=2\}\}2\.0013\.3846\.0228\.6643\.2551\.0636\.9158\.9058\.6442\.10Top\-K ReducedK=2\{\}\_\{~\\text\{K=2\}\}2\.0015\.1851\.1034\.7651\.6553\.3560\.3968\.8766\.8350\.27MoE\-Dynamicϕ=0\.1\{\}\_\{~\\text\{$\\phi$=0\.1\}\}3\.9015\.9656\.5639\.0028\.5255\.1058\.4334\.5671\.0944\.90AdaMoENull=128\{\}\_\{~\\text\{Null=128\}\}2\.119\.2428\.9620\.7339\.7054\.8352\.2954\.5070\.2741\.31BEAM\(β=0\.1\)\{\}\_\{~\(\\beta=0\.1\)\}1\.0817\.1859\.2143\.2948\.0856\.1560\.6771\.0770\.9353\.32Extreme SparsityTop\-K ReducedK=1\{\}\_\{~\\text\{K=1\}\}1\.007\.9031\.9925\.0028\.7541\.4645\.0153\.1452\.9135\.77BEAM\(β=1\.0\)\{\}\_\{~\(\\beta=1\.0\)\}0\.4811\.7245\.1942\.0738\.3350\.1154\.4869\.6667\.5747\.39 Table[1](https://arxiv.org/html/2605.14438#S4.T1), Table[2](https://arxiv.org/html/2605.14438#S4.T2), and Table[3](https://arxiv.org/html/2605.14438#S4.T3)summarize the performance and sparsity results of BEAM and baselines across multiple MoE models, organized by mid, high, and extreme sparsity levels\. We report average activated experts per token \(Avg\-K\) and downstream task scores\. Comparisons with DynMoE are provided in Appendix[B\.3](https://arxiv.org/html/2605.14438#A2.SS3)\. BEAM achieves extreme sparsity with minimal performance loss\.BEAM consistently preserves over98%of original accuracy at mid sparsity across all three models while reducing Avg\-K by 47%–61%\. At high sparsity, Avg\-K drops to as low as14%of the original \(e\.g\.,0\.56/40\.56/4on Qwen1\.5\) with over 95% accuracy retained\. The advantage of BEAM is most evident under extreme sparsity\. On DeepSeekV2, BEAM \(K=0\.48K=0\.48\) outperforms Top\-K Reduced \(K=1K=1\) by32\.49%, while on Qwen3 the margin reaches33\.29%\. On Qwen1\.5, BEAM reaches Avg\-K=0\.11=0\.11, indicating that most tokens completely bypass routed experts, while still retaining 85% of the original performance, which demonstrates effective token\-adaptive redundancy removal\. Existing dynamic routing methods underperform in post\-training settings\.Top\-K Pruning degrades sharply at higher sparsity, while Top\-K Reduced is more stable but its fixed per\-token budget consistently underperforms BEAM\. Even at extreme sparsity, BEAM with fewer average experts outperforms Top\-K Reduced \(K=1K=1\) on both Qwen3 and DeepSeek\. MoE\-Dynamic and AdaMoE also fall short: the former requires model\-specific threshold tuning without competitive trade\-offs, while the latter suffers from performance degradation due to null\-expert interference\. BEAM avoids these issues by decoupling sparsification from expert selection via a lightweight mask router, enabling stable training and preserving the original expert load balance \(Appendix[B\.5](https://arxiv.org/html/2605.14438#A2.SS5)\)\. β\\betaprovides smooth control over the sparsity–accuracy trade\-off\.Increasingβ\\betaconsistently improves sparsity with gradual accuracy loss \(Tables[1](https://arxiv.org/html/2605.14438#S4.T1)–[3](https://arxiv.org/html/2605.14438#S4.T3)\), making it straightforward to adapt the method to deployment constraints via a single parameter\. Atβ=0\.1\\beta=0\.1, BEAM preserves over 95% accuracy across all models, offering a good trade\-off\. ### 4\.3Acceleration Comparison We evaluate inference acceleration under both online and offline settings\. In the online setting, models are deployed as services and we measure TTFT and TPOT across varying QPS to simulate real\-world serving\. In the offline setting, we use a large fixed batch size to maximize GPU utilization and report throughput, reflecting scenarios like large\-scale LLM knowledge distillation\. For fair comparison with performance\-efficiency tradeoff, we tested the inference speed of all baseline methods and BEAM underHigh Sparsity\. As shown in Figure[4](https://arxiv.org/html/2605.14438#S4.F4), BEAM achieves consistent speedups across all models and settings\. It achieves at least1\.3×1\.3\\timesimprovement in TPOT and over1\.1×1\.1\\timesgains in both TTFT and throughput\. Notably, on DeepSeek\-V2\-Lite at QPS=24, BEAM reaches up to2\.5×2\.5\\timesdecoding acceleration\. The achievable speedup is limited by model architecture\. For example, Qwen1\.5\-MoE\-A2\.7B contains 4 shared experts out of 8 total, limiting their MoE layer FLOPs reduction to at most 50%\. In contrast, Qwen3\-30B\-A3B has no shared experts, enabling an 85% FLOPs reduction and substantially higher throughput gains\. In comparison, MoE\-Dynamic and AdaMoE achieve limited sparsity and introduce extra overhead, yielding negligible or no acceleration benefits\. Figure 4:Comparison of TPOT, TTFT, and throughput across different methods\. ### 4\.4Ablation Study Table 4:Binary thresholdτ\\tauablation on Qwen1\.5\.ThresholdAvg\. KReason\.Know\.Common\.Avg\.↑\\uparrowτ=0\.1\\tau=0\.10\.7843\.9166\.8066\.4358\.12τ=0\.3\\tau=0\.30\.7044\.4166\.3770\.8259\.25τ=0\.5\\tau=0\.50\.5643\.0165\.1276\.2359\.61τ=0\.7\\tau=0\.70\.4241\.9262\.2369\.7256\.49τ=0\.9\\tau=0\.90\.2841\.0960\.0059\.2652\.72 Table 5:Training configuration ablation on Qwen3\.Configs\.Avg\. KReason\.Know\.Common\.Avg\.↑\\uparrowBEAM1\.2374\.1574\.8885\.0277\.14\- w/o\.ℒreg\\mathcal\{L\}\_\{reg\}6\.3175\.4276\.5583\.2377\.80\(0\.9%↑\\uparrow\)\-ℒ1\\mathcal\{L\}\_\{1\}toℒ2\\mathcal\{L\}\_\{2\}2\.0171\.3473\.2184\.2975\.28\(2\.4%↓\\downarrow\)\- Soft1\.3412\.7021\.9442\.3023\.56\(69\.5%↓\\downarrow\)\- Soft w/\. Temp\.1\.7865\.3072\.3582\.2973\.31\(5\.0%↓\\downarrow\) Ablation on Binary ThresholdWe evaluate binarization thresholdsτ∈\{0\.1,0\.3,0\.5,0\.7,0\.9\}\\tau\\in\\\{0\.1,0\.3,0\.5,0\.7,0\.9\\\}on Qwen1\.5\-MoE\-A2\.7B, as shown in Table[4](https://arxiv.org/html/2605.14438#S4.T4)\. Increasingτ\\taumonotonically reduces Avg\-K and thus increases sparsity\. We find thatτ=0\.5\\tau=0\.5achieves the best overall performance, largely driven by stronger commonsense results\. A plausible explanation is thatτ=0\.5\\tau=0\.5offers the greatest gradient sensitivity around the decision boundary while maintaining a stable Top\-K initialization\. Based on this result, we fixτ=0\.5\\tau=0\.5and vary only the regularization coefficientβ\\betato control sparsity\. Ablation on Training ApproachWe evaluate several training variants on Qwen3\-30B\-A3B, including removingℒreg\\mathcal\{L\}\_\{\\text\{reg\}\}, replacing L1 with L2 regularization, and replacing STE\-based binary masking with soft\-mask training\. For the latter, we consider both plain sigmoid gating \(Soft\) and a temperature\-scaled sigmoid that gradually sharpens the mask \(Soft w/\. Temp\.\)\. As shown in Table[5](https://arxiv.org/html/2605.14438#S4.T5), removingℒreg\\mathcal\{L\}\_\{\\text\{reg\}\}slightly improves reasoning performance but substantially increases expert activation\. L2 regularization is inferior to L1 in both sparsity and accuracy\. Both soft\-mask variants also underperform binary\-mask training, where plain sigmoid gating fails severely because of the train\-inference mismatch, while temperature scaling only partially mitigates this issue\. Overall, the results support the use ofℒreg\\mathcal\{L\}\_\{\\text\{reg\}\}, L1 regularization, and STE\-based binary masking in BEAM\. ## 5Analysis ### 5\.1Token\-wise Sparsity Analysis Figure 5:The average number of activated experts per token in BEAM: Qwen3\-30B\-A3B\.To understand how BEAM adapts computation per token, we visualize the average number of activated experts across tokens, as shown in Figure[5](https://arxiv.org/html/2605.14438#S5.F5)\. We obtain several key findings\.1\. Expert activation varies across tokens\.The most demanding tokens activate up to4\.654\.65experts on average, versus only0\.60\.6for the least demanding\.2\. Activation aligns with semantic richness\.Content words \(e\.g\., nouns, verbs\) consistently trigger more experts than function words \(e\.g\., prepositions\) and punctuation\.3\. Chat template tokens are highly redundant\.Fixed prompts like “You are a helpful assistant” activate few experts yet maintain performance, suggesting minimal informational value\. These findings show that BEAM dynamically allocates computation based on token informativeness\. ### 5\.2Layer\-wise and Position\-wise Analysis \(a\)Layer\-wise activated experts\. \(b\)Position\-wise masking probability\. Figure 6:Layer\-wise sparsity and position\-wise masking analysis\.We measure the average number of activated experts per layer during prefill and decode on 1,000 randomly sampled inputs \(Figure[6\(a\)](https://arxiv.org/html/2605.14438#S5.F6.sf1)\)\. DeepSeek exhibits nearly identical expert usage in both phases, while Qwen1\.5 and Qwen3 consistently use more experts during decoding\. These models also develop an encoder\-decoder\-like pattern: shallower layers primarily support knowledge storage, while deeper layers allocate more expert capacity to decoding and reasoning, meaning that BEAM adapts layer\-wise sparsity to functional roles\. We further compare the per\-position masking probability of BEAM, MoE\-Dynamic, and AdaMoE on Qwen3 Under similar sparsity conditions\. \(Figure[6\(b\)](https://arxiv.org/html/2605.14438#S5.F6.sf2)\)\. MoE\-Dynamic exhibits strong position bias, never masking Top\-1 \(0\.00\) and applying extremely high masking beyond Top\-6 \(0\.79∼\\sim0\.94\)\. AdaMoE shows a monotonic increase from Top\-1 \(0\.06\) to Top\-8 \(0\.66\), indicating moderate but still rank\-dependent bias\. In contrast, BEAM shows only a mild increase from Top\-1 \(0\.43\) to Top\-8 \(0\.53\), demonstrating that it evaluates expert relevance based on token\-specific features rather than routing rank\. Another layer\-wise rank masking analysis is provided in Appendix[B\.4](https://arxiv.org/html/2605.14438#A2.SS4)\. We also provide task\-specific acceleration analysis, expert load balancing analysis, and token\-layer sparsity visualizations\. Details can be found in Appendix[B\.6](https://arxiv.org/html/2605.14438#A2.SS6),[B\.5](https://arxiv.org/html/2605.14438#A2.SS5), and[B\.7](https://arxiv.org/html/2605.14438#A2.SS7)\. ## 6Conclusion We propose BEAM, a plug\-and\-play dynamic routing framework that introduces a lightweight mask router to selectively deactivate redundant experts within the Top\-K set, enabling token\-adaptive sparsity without modifying the model architecture\. Integrated into vLLM via an efficient CUDA kernel, BEAM delivers up to 2\.5×\\timesfaster decoding and 1\.4×\\timeshigher throughput with over 98% accuracy retention, demonstrating that decoupling sparsity control from routing enables stable and practical MoE inference acceleration\. ## References - M\. A\. Aghdam, H\. Jin, and Y\. Wu \(2024\)Da\-moe: towards dynamic expert allocation for mixture\-of\-experts models\.arXiv preprint arXiv:2409\.06669\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p1.1)\. - W\. Amer, F\. Kurdahi,et al\.\(2026\)ConfLayers: adaptive confidence\-based layer skipping for self\-speculative decoding\.arXiv preprint arXiv:2604\.14612\.Cited by:[§A\.3](https://arxiv.org/html/2605.14438#A1.SS3.p2.1)\. - J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§B\.1\.1](https://arxiv.org/html/2605.14438#A2.SS1.SSS1.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p1.1)\. - Y\. Bengio, N\. Léonard, and A\. Courville \(2013\)Estimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p4.1),[§3\.3\.1](https://arxiv.org/html/2605.14438#S3.SS3.SSS1.p1.4)\. - I\. Chen, H\. Liu, W\. Sun, C\. Chao, Y\. Hsu, and C\. Lee \(2025\)Retraining\-free merging of sparse moe via hierarchical clustering\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=hslOzRxzXL)Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p3.1)\. - M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.External Links:2107\.03374Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.6.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 2924–2936\.External Links:[Link](https://aclanthology.org/N19-1300/),[Document](https://dx.doi.org/10.18653/v1/N19-1300)Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.14.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.4.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - O\. Contributors \(2023\)OpenCompass: a universal evaluation platform for foundation models\.Note:[https://github\.com/open\-compass/opencompass](https://github.com/open-compass/opencompass)Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - A\. Gui, B\. Li, B\. Tao, B\. Zhou, B\. Chen, C\. Zhang, C\. Han, C\. Yang, C\. Zhang,et al\.\(2025\)Introducing longcat\-flash\-thinking: a technical report\.arXiv preprint arXiv:2509\.18883\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p2.1)\. - Y\. Guo, Z\. Cheng, X\. Tang, Z\. Tu, and T\. Lin \(2024\)Dynamic mixture of experts: an auto\-tuning approach for efficient transformer models\.arXiv preprint arXiv:2405\.14297\.Cited by:[§B\.3](https://arxiv.org/html/2605.14438#A2.SS3.p1.1),[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p2.1)\. - D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021a\)Measuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021b\)Measuring mathematical problem solving with the math dataset\.NeurIPS\.Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.2.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - Q\. Huang, Z\. An, N\. Zhuang, M\. Tao, C\. Zhang, Y\. Jin, K\. Xu, L\. Chen, S\. Huang, and Y\. Feng \(2024\)Harder tasks need more experts: dynamic routing in moe models\.arXiv preprint arXiv:2403\.07652\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p2.1),[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p2.1)\. - Y\. Huang, Y\. Bai, Z\. Zhu, J\. Zhang, J\. Zhang, T\. Su, J\. Liu, C\. Lv, Y\. Zhang, J\. Lei, Y\. Fu, M\. Sun, and J\. He \(2023\)C\-eval: a multi\-level multi\-discipline chinese evaluation suite for foundation models\.InAdvances in Neural Information Processing Systems,Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.10.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.\(2024\)Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p1.1)\. - C\. Jin, H\. Peng, M\. Xiang, Q\. Zhang, X\. Yuan, A\. Hasan, O\. Dibua, Y\. Gong, Y\. Kang, and D\. N\. Metaxas \(2025\)Sparsity\-controllable dynamic top\-p moe for large foundation model pre\-training\.arXiv preprint arXiv:2512\.13996\.Cited by:[§2](https://arxiv.org/html/2605.14438#S2.p1.1)\. - P\. Jin, B\. Zhu, L\. Yuan, and S\. Yan \(2024\)Moe\+\+: accelerating mixture\-of\-experts methods with zero\-computation experts\.arXiv preprint arXiv:2410\.07348\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p2.1)\. - W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p2.2),[§1](https://arxiv.org/html/2605.14438#S1.p4.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p4.2)\. - N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.\(2024\)Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p1.1)\. - T\. Lawson and L\. Aitchison \(2025\)Learning to skip the middle layers of transformers\.arXiv preprint arXiv:2506\.21103\.Cited by:[§A\.3](https://arxiv.org/html/2605.14438#A1.SS3.p2.1)\. - D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen \(2020\)Gshard: scaling giant models with conditional computation and automatic sharding\.arXiv preprint arXiv:2006\.16668\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p2.1)\. - H\. Li, Y\. Zhang, F\. Koto, Y\. Yang, H\. Zhao, Y\. Gong, N\. Duan, and T\. Baldwin \(2023a\)CMMLU: measuring massive multitask language understanding in chinese\.External Links:2306\.09212Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.12.1.1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.8.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - J\. Li, Q\. Su, Y\. Yang, Y\. Jiang, C\. Wang, and H\. Xu \(2023b\)Adaptive gating in mixture\-of\-experts based language models\.arXiv preprint arXiv:2310\.07188\.Cited by:[§2](https://arxiv.org/html/2605.14438#S2.p1.1)\. - P\. Li, Z\. Zhang, P\. Yadav, Y\. Sung, Y\. Cheng, M\. Bansal, and T\. Chen \(2023c\)Merge, then compress: demystify efficient smoe with hints from its routing policy\.arXiv preprint arXiv:2310\.01334\.Cited by:[§2](https://arxiv.org/html/2605.14438#S2.p3.1)\. - A\. Liu, B\. Feng, B\. Wang, B\. Wang, B\. Liu, C\. Zhao, C\. Dengr, C\. Ruan, D\. Dai, D\. Guo,et al\.\(2024a\)Deepseek\-v2: a strong, economical, and efficient mixture\-of\-experts language model\.arXiv preprint arXiv:2405\.04434\.Cited by:[§B\.1\.1](https://arxiv.org/html/2605.14438#A2.SS1.SSS1.p1.1),[§1](https://arxiv.org/html/2605.14438#S1.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p1.1)\. - E\. Liu, J\. Zhu, Z\. Lin, X\. Ning, M\. B\. Blaschko, S\. Yan, G\. Dai, H\. Yang, and Y\. Wang \(2024b\)Efficient expert pruning for sparse mixture\-of\-experts language models: enhancing performance and reducing inference costs\.arXiv preprint arXiv:2407\.00945\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p3.1)\. - X\. Lu, Q\. Liu, Y\. Xu, A\. Zhou, S\. Huang, B\. Zhang, J\. Yan, and H\. Li \(2024\)Not all experts are equal: efficient expert pruning and skipping for mixture\-of\-experts large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 6159–6172\.External Links:[Link](https://aclanthology.org/2024.acl-long.334/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.334)Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p1.1)\. - N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean \(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.arXiv preprint arXiv:1701\.06538\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p2.1)\. - Z\. Su, Z\. Lin, X\. Bai, X\. Wu, Y\. Xiong, H\. Lian, G\. Ma, H\. Chen, G\. Ding, W\. Zhou,et al\.\(2024\)Maskmoe: boosting token\-level learning via routing mask in mixture\-of\-experts\.arXiv preprint arXiv:2407\.09816\.Cited by:[§2](https://arxiv.org/html/2605.14438#S2.p1.1)\. - A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421),[Document](https://dx.doi.org/10.18653/v1/N19-1421),1811\.00937Cited by:[§B\.1\.3](https://arxiv.org/html/2605.14438#A2.SS1.SSS3.p1.1),[Table 7](https://arxiv.org/html/2605.14438#A2.T7.4.16.1.1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p3.1)\. - A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025a\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§B\.1\.1](https://arxiv.org/html/2605.14438#A2.SS1.SSS1.p1.1),[§1](https://arxiv.org/html/2605.14438#S1.p1.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p1.1)\. - C\. Yang, Y\. Sui, J\. Xiao, L\. Huang, Y\. Gong, Y\. Duan, W\. Jia, M\. Yin, Y\. Cheng, and B\. Yuan \(2024a\)MoE\-i2: compressing mixture of experts models through inter\-expert pruning and intra\-expert low\-rank decomposition\.arXiv preprint arXiv:2411\.01016\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1)\. - N\. Yang, F\. Liu, J\. Wang, T\. Yang, K\. Liu, H\. Guan, and L\. Jiang \(2025b\)DASH: input\-aware dynamic layer skipping for efficient llm inference with markov decision policies\.arXiv preprint arXiv:2505\.17420\.Cited by:[§A\.3](https://arxiv.org/html/2605.14438#A1.SS3.p2.1)\. - Y\. Yang, S\. Qi, W\. Gu, C\. Wang, C\. Gao, and Z\. Xu \(2024b\)Xmoe: sparse models with fine\-grained and adaptive expert selection\.arXiv preprint arXiv:2403\.18926\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p1.1)\. - Z\. Zeng, Y\. Miao, H\. Gao, H\. Zhang, and Z\. Deng \(2024\)Adamoe: token\-adaptive routing with null experts for mixture\-of\-experts language models\.arXiv preprint arXiv:2406\.13233\.Cited by:[§1](https://arxiv.org/html/2605.14438#S1.p2.1),[§1](https://arxiv.org/html/2605.14438#S1.p3.1),[§2](https://arxiv.org/html/2605.14438#S2.p2.1),[§4\.1](https://arxiv.org/html/2605.14438#S4.SS1.p2.1)\. - Z\. Zhang, X\. Liu, H\. Cheng, C\. Xu, and J\. Gao \(2025\)Diversifying the expert knowledge for task\-agnostic pruning in sparse mixture\-of\-experts\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 86–102\.Cited by:[§2](https://arxiv.org/html/2605.14438#S2.p3.1)\. ## Appendix AAppendix on Method ### A\.1Impact Statements This work focuses on improving the computational efficiency of Mixture\-of\-Experts models\. We do not identify any societal impacts specific to the proposed method beyond those already associated with the general use and deployment of large language models\. ### A\.2Limitations Our work has several limitations\. First, BEAM is evaluated on three MoE architectures; its effectiveness on other MoE designs \(e\.g\., with different gating mechanisms or expert granularities\) remains to be validated\. Second, BEAM requires a post\-training SFT phase to learn the mask router, which incurs additional training cost proportional to the model size\. Third, the achievable inference speedup depends on the model’s shared\-expert ratio\. For example, architectures with a large proportion of shared experts may benefit less, as shared\-expert computation cannot be reduced by BEAM\. Finally, our acceleration benchmarks are conducted on single\-GPU settings, and the interaction between BEAM’s dynamic sparsity and multi\-GPU expert parallelism strategies needs further investigation\. ### A\.3Behaviors under Zero Activation Since BEAM permits between0andKKactivated experts, the zero\-activation case requires special consideration\. In a modern Transformer MoE block, the hidden state update follows: 𝐡′=𝐡\+∑i=1N𝐠^iℰi\(𝒩\(𝐡\)\)\+δsh𝐠shℰsh\(𝒩\(𝐡\)\),\\mathbf\{h\}^\{\\prime\}=\\mathbf\{h\}\+\\sum\\nolimits\_\{i=1\}^\{N\}\\hat\{\\mathbf\{g\}\}\_\{i\}\\,\\mathcal\{E\}\_\{i\}\(\\mathcal\{N\}\(\\mathbf\{h\}\)\)\+\\delta\_\{\\mathrm\{sh\}\}\\,\\mathbf\{g\}\_\{\\mathrm\{sh\}\}\\,\\mathcal\{E\}\_\{\\mathrm\{sh\}\}\(\\mathcal\{N\}\(\\mathbf\{h\}\)\),\(15\)where𝒩\(⋅\)\\mathcal\{N\}\(\\cdot\)denotes the normalization function,ℰi\(⋅\)\\mathcal\{E\}\_\{i\}\(\\cdot\)denotes theii\-th routed expert,𝐠^i∈ℝ\\hat\{\\mathbf\{g\}\}\_\{i\}\\in\\mathbb\{R\}denotes the normalized routing weight assigned to expertii,ℰsh\(⋅\)\\mathcal\{E\}\_\{\\mathrm\{sh\}\}\(\\cdot\)denotes the shared expert,𝐠sh\\mathbf\{g\}\_\{\\mathrm\{sh\}\}denotes the routing weight assigned to the shared expert, andδsh∈\{0,1\}\\delta\_\{\\mathrm\{sh\}\}\\in\\\{0,1\\\}is an indicator variable specifying whether a shared expert is present\. When all routed experts are skipped,i\.e\.,𝐠^i=0\\hat\{\\mathbf\{g\}\}\_\{i\}=0for alli∈\{1,…,N\}i\\in\\\{1,\\dots,N\\\}, then the update becomes: 𝐡′=𝐡\+δsh𝐠shℰsh\(𝒩\(𝐡\)\)\.\\mathbf\{h\}^\{\\prime\}=\\mathbf\{h\}\+\\delta\_\{\\mathrm\{sh\}\}\\,\\mathbf\{g\}\_\{\\mathrm\{sh\}\}\\,\\mathcal\{E\}\_\{\\mathrm\{sh\}\}\(\\mathcal\{N\}\(\\mathbf\{h\}\)\)\.\(16\)Therefore, ifδsh=1\\delta\_\{\\mathrm\{sh\}\}=1, as in architectures with shared experts such as Qwen1\.5\-MoE and DeepSeek, the layer reduces to shared\-expert\-only computation\. Ifδsh=0\\delta\_\{\\mathrm\{sh\}\}=0, as in architectures without shared experts such as Qwen3\-MoE, the token bypasses the entire MoE layer through the residual path\. Under zero activation, BEAM degenerates into a form of dynamic layer skipping, which has also been studied in prior work as an efficient inference acceleration mechanism\[Yanget al\.,[2025b](https://arxiv.org/html/2605.14438#bib.bib40), Lawson and Aitchison,[2025](https://arxiv.org/html/2605.14438#bib.bib38), Ameret al\.,[2026](https://arxiv.org/html/2605.14438#bib.bib39)\]\. Empirically, as shown in Section[4\.2](https://arxiv.org/html/2605.14438#S4.SS2), BEAM maintains strong model performance even when the average number of activated experts is below11, implying that zero\-activation cases occur frequently in practice\. This observation suggests that substantial layer computation in LLMs is redundant\. Moreover, the analysis in Section[5](https://arxiv.org/html/2605.14438#S5)shows that zero activation arises more often in deeper layers during prefill and for tokens with limited semantic content\. ### A\.4Key Modifications for BEAM We implement BEAM in vLLM by minimally extending its standard MoE CUDA pipeline with two kernel\-level changes\. First,mask\_route\_kernelwrites the expert index as−1\-1whenever the corresponding mask logit is non\-positive\. Second,moe\_align\_block\_size\_kernelignores all−1\-1entries during expert\-wise token grouping and block alignment, thereby removing masked experts from subsequent computation\. This modification is lightweight, preserves compatibility with vLLM’s existing optimizations such as operator fusion and memory coalescing, and introduces negligible integration overhead\. The core code is provided below, and the full implementation will be released upon acceptance\. 1template<typenamescalar\_t\> 2\_\_global\_\_voidmask\_route\_kernel\( 3constint64\_t\*\_\_restrict\_\_topk\_ids, 4constscalar\_t\*\_\_restrict\_\_mask\_logits, 5int64\_t\*\_\_restrict\_\_output\_ids, 6constintnum\_tokens,constinttop\_k,constintnum\_experts\)\{ 7 8intidx=blockIdx\.x\*blockDim\.x\+threadIdx\.x; 9if\(idx\>=num\_tokens\*top\_k\)return; 10inttoken\_idx=idx/top\_k; 11intslot\_idx=idx%top\_k; 12intinput\_idx=token\_idx\*top\_k\+slot\_idx; 13int64\_toriginal\_expert=topk\_ids\[input\_idx\]; 14 15if\(token\_idx\>=num\_tokens\|\|slot\_idx\>=top\_k\|\| 16original\_expert<0\|\|original\_expert\>=num\_experts\)\{ 17output\_ids\[input\_idx\]=\-1; 18return; 19\} 20 21 22intexpert\_idx=token\_idx\*num\_experts\+original\_expert; 23scalar\_tlogit=mask\_logits\[expert\_idx\]; 24output\_ids\[input\_idx\]=\(logit\>0\)?original\_expert:\-1; 25\} 26 27template<typenamescalar\_t,typenametoken\_cnts\_t\> 28\_\_global\_\_voidmoe\_align\_block\_size\_kernel\(\)\{ 29 30for\(inti=start\_idx;i<end\_idx;\+\+i\)\{ 31int64\_texpert\_id=topk\_ids\[i\]; 32if\(expert\_id\!=\-1\)\{ 33\+\+tokens\_cnts\[index\(num\_experts,threadIdx\.x\+1,expert\_id\)\]; 34\} 35\} 36 37\} ## Appendix BAppendix on Experiment ### B\.1Experimental Setup #### B\.1\.1Models Table 6:Main hyperparameters for each model\.Model ConfigQwen1\.5\-MoE\-A2\.7BDeepSeekV2\-LiteQwen3\-30B\-A3BTotal Params \(B\)14\.31630Activated Params \(B\)2\.72\.43MoE Layers / Total Layers24/2426/2748/48Experts per MoE Layer6064128Activated Experts per Token4 \(selected\) \+ 4 \(shared\)6 \(selected\) \+ 2 \(shared\)8hidden size256020482048intermediate size5632109446144Vocabulary Size151936102400151936Inference SettingQwen1\.5\-MoE\-A2\.7BDeepSeekV2\-LiteQwen3\-30B\-A3BTemperature0\.70\.30\.7Top\-pp0\.80\.950\.8Top\-kk205020Repetition Penalty1\.051\.001\.00Max Output Tokens102410242048Batch Size161616Training SettingQwen1\.5\-MoE\-A2\.7BDeepSeekV2\-LiteQwen3\-30B\-A3BLearning Rate5×10−55\\times 10^\{\-5\}5×10−55\\times 10^\{\-5\}5×10−55\\times 10^\{\-5\}Learning Rate ScheduleLinearLinearLinearLoad Balancing Loss Coefficient \(α\\alpha\)1×10−31\\times 10^\{\-3\}1×10−31\\times 10^\{\-3\}1×10−31\\times 10^\{\-3\}Per Device Batch Size323232Number of GPUs323264Max Token Length409640964096Warm up ratio0\.030\.030\.03Number of Epochs222 We do experiments on three representative MoE models: Qwen1\.5‑MoE‑A2\.7B\[Baiet al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib23)\], DeepSeekV2‑Lite\[Liuet al\.,[2024a](https://arxiv.org/html/2605.14438#bib.bib2)\], and Qwen3‑30B‑A3B\[Yanget al\.,[2025a](https://arxiv.org/html/2605.14438#bib.bib1)\]\. - •Qwen1\.5\-MoE\-A2\.7B:Each token activates44shared experts and44routed experts \(out of6060\) in each layer\. - •DeepSeekV2\-Lite:Each token activates22shared experts and66routed experts \(out of6464\) in each layer\. - •Qwen3\-30B\-A3B:Each token activates88routed experts \(out of128128\) in each layer\. More details can be found in Table[6](https://arxiv.org/html/2605.14438#A2.T6)\. #### B\.1\.2Hyper\-Parameters Tables[6](https://arxiv.org/html/2605.14438#A2.T6)summarize the main configurations for all MoE models studied in this work\. All trainings and evaluations are performed on NVIDIA H20 GPUs #### B\.1\.3Benchmarks Table 7:Overview of OpenCompass tasks used for evaluation\.TaskDomain/FormatDescription / ExampleMath\[Hendryckset al\.,[2021b](https://arxiv.org/html/2605.14438#bib.bib26)\]Reasoning / Open\-EndedA dataset of high school\-level mathematical problems requiring step\-by\-step solutions\. Example:A positive multiple of 45 less than 1000 is randomly selected\. What is the probability that it is a two\-digit integer? Express your answer as a common fraction\.GSM8K\[Cobbeet al\.,[2021](https://arxiv.org/html/2605.14438#bib.bib28)\]Reasoning / Open\-EndedGrade school math word problems with a focus on multi\-step reasoning\. Example:Shiloh is 44 years old today\. In 7 years, he will be three times as old as his nephew\. How old is his nephew today?HumanEval\[Chenet al\.,[2021](https://arxiv.org/html/2605.14438#bib.bib27)\]Reasoning / Open\-EndedPython programming problems requiring function implementation based on a natural language description\. Example:Write a function that returns the sum of two numbers\.MMLU\[Liet al\.,[2023a](https://arxiv.org/html/2605.14438#bib.bib34)\]Knowledge / Multiple\-ChoiceA massive multitask test consisting of multiple\-choice questions from various branches of knowledge\. Example:Who set the world record for the mile race in 1886? A\. R Bannister, B\. S Coe, C\. J DiMaggio, D\. WG GeorgeCEVAL\[Huanget al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib35)\]Knowledge / Multiple\-ChoiceA comprehensive Chinese evaluation suite for foundation models\. Example:下列各项中，应征收资源税的是\_\_\_\_\_。 A\. 人造石油 B\. 某商贸企业零售的煤炭 C\. 开采铁矿石同时开采的锰矿 D\. 某联合企业进口的石油CMMLU\[Liet al\.,[2023a](https://arxiv.org/html/2605.14438#bib.bib34)\]Knowledge / Multiple\-ChoiceA comprehensive Chinese multi\-subject exam benchmark with 57 subjects\. Example:关系数据库中数据的逻辑结构是（A）树结构（B）维度表（C）层次结构（D）形状结构BoolQ\[Clarket al\.,[2019](https://arxiv.org/html/2605.14438#bib.bib32)\]CommonSense / Multiple\-Choice \(Yes/No\)Reading comprehension questions with yes/no answers based on a passage\. Example:Property tax – Property tax or ‘house tax’ is a local tax … Is house tax and property tax are same?CommonSenseQA\[Talmoret al\.,[2019](https://arxiv.org/html/2605.14438#bib.bib31)\]CommonSense / Multiple\-ChoiceA new multiple\-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers \. Example:Sammy wanted to go to where the people were\. Where might he go? A\. race track, B\. populated areas, C\. the desert, D\. apartment, E\. roadblock\."For accuracy comparison, we select a diverse set of tasks from the OpenCompass\[Contributors,[2023](https://arxiv.org/html/2605.14438#bib.bib25)\]benchmark, covering multiple domains:Reasoning\(MATH\[Hendryckset al\.,[2021b](https://arxiv.org/html/2605.14438#bib.bib26)\], GSM8K\[Cobbeet al\.,[2021](https://arxiv.org/html/2605.14438#bib.bib28)\], and Human Eval\[Chenet al\.,[2021](https://arxiv.org/html/2605.14438#bib.bib27)\]\);Knowledge\(MMLU\[Hendryckset al\.,[2021a](https://arxiv.org/html/2605.14438#bib.bib29)\], CEVAL\[Huanget al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib35)\], and CMMLU\[Liet al\.,[2023a](https://arxiv.org/html/2605.14438#bib.bib34)\]\); andCommonSense\(CommonsenseQA\[Talmoret al\.,[2019](https://arxiv.org/html/2605.14438#bib.bib31)\], BoolQ\[Clarket al\.,[2019](https://arxiv.org/html/2605.14438#bib.bib32)\]\)\. Details and examples of these tasks are provided in Table[7](https://arxiv.org/html/2605.14438#A2.T7)\. For acceleration comparison, we use vLLM\[Kwonet al\.,[2023](https://arxiv.org/html/2605.14438#bib.bib17)\]as the inference framework\. Each model is deployed on a single GPU, and we record theTime per Output Token\(TPOT, in ms\) across differentQueries per Second\(QPS\), theTime To First Token\(TTFT, in ms\) in 32 QPS \(high\-computing scenarios\), and the offlineThroughput\(samples/s\)\. The input and output sequence lengths are fixed at128128and3232tokens, respectively, and each test processes a total of 5,000 samples\. ### B\.2Training Dynamics Figure 7:Training curves of BEAM \(β=0\.1\\beta=0\.1\) on three MoEs\. Blue: language modeling loss \(left axis\)\. Orange: expert active rate \(right axis\)\. Gray dashed line: SFT baseline loss without BEAM\.Figure[7](https://arxiv.org/html/2605.14438#A2.F7)shows the language modeling loss and expert active rate during BEAM training \(β=0\.1\\beta=0\.1\) on all three models\. The gray dashed line indicates the converged loss of standard SFT without BEAM\. Two observations emerge\. First, BEAM’s language modeling loss converges to a level comparable to the SFT baseline across all models, confirming that the mask router and sparsity regularization do not compromise model capacity\. Second, expert sparsification concentrates in the first∼\\sim0\.5 epoch, where the active rate drops sharply from near 100% to a stable plateau\. The remaining training focuses on optimizing the language modeling objective under the learned sparsity pattern\. ### B\.3More Baseline Comparison We additionally compare BEAM with DynMoE\[Guoet al\.,[2024](https://arxiv.org/html/2605.14438#bib.bib11)\], which replaces hard Top\-K routing with sigmoid\-gated expert selection based on token\-expert affinity\. Table[8](https://arxiv.org/html/2605.14438#A2.T8)summarizes the results on all three evaluated models\. Table 8:Performance comparisons of DynMoE and BEAM relative to the original models\.ReasoningKnowledgeCommonSenseMethods \\TasksAvg\. KMATHGSM8KH\_EvalMMLUCEVALCMMLUBoolQCSQAAvg\.\(Acc\.↑\\uparrow\)Qwen1\.5\-MoE\-A2\.7BQwen1\.5\-MoEK=4\{\}\_\{~\\text\{K=4\}\}4\.0023\.0457\.4750\.6159\.2874\.1575\.1872\.6381\.3361\.71DynMoE30\.0610\.7245\.0331\.1042\.9437\.6039\.3560\.8961\.5141\.14BEAM\(β=0\.01\)\{\}\_\{~\(\\beta=0\.01\)\}1\.5624\.7855\.5053\.0558\.7570\.4769\.1878\.3280\.8461\.36Qwen3\-30B\-A3BQwen3\-30B\-A3BK=8\{\}\_\{~\\text\{K=8\}\}8\.0058\.2888\.0282\.9381\.8083\.5683\.6986\.7686\.2481\.41DynMoE61\.6619\.4666\.4939\.0240\.6436\.4236\.5158\.5051\.1143\.52BEAM\(β=0\.01\)\{\}\_\{~\(\\beta=0\.01\)\}4\.2355\.1685\.5281\.7180\.0981\.4681\.5388\.0786\.4079\.99DeepSeekV2\-LiteDeepSeekV2\-LiteK=6\{\}\_\{~\\text\{K=6\}\}6\.0020\.0262\.7043\.9055\.0455\.2660\.9375\.2068\.1455\.15DynMoE30\.500\.041\.360\.005\.116\.630\.418\.816\.393\.59BEAM\(β=0\.01\)\{\}\_\{~\(\\beta=0\.01\)\}2\.6120\.3660\.2746\.9556\.6554\.1260\.4276\.5765\.1155\.06 DynMoE exhibits substantial instability in the post\-training setting, as its learned gating tends to over\-activate experts far beyond the original Top\-K budget\. Specifically, the average number of activated experts rises to 61\.66 on Qwen3\-30B\-A3B \(vs\. Top\-K = 8\), 30\.06 on Qwen1\.5\-MoE\-A2\.7B \(vs\. Top\-K = 4\), and 30\.50 on DeepSeekV2\-Lite \(vs\. Top\-K = 6\)\. This instability also leads to severe accuracy degradation, most notably on DeepSeekV2\-Lite, where performance collapses from 55\.15 to 3\.59 average accuracy\. This is likely because DynMoE completely replaces the original router architecture, making it ill\-suited for post\-training scenarios where preserving the pretrained routing structure is critical\. In contrast, BEAM achieves substantially higher sparsity while retaining over 98% of the original model’s performance across all three models\. ### B\.4Layer\-wise Masking Rank Analysis Figure 8:Layer\-wise masking rank analysis across three MoE models\. The shaded region between the min masked rank and max kept rank indicates the overlap zone where BEAM’s masking decisions are token\-dependent\.To further investigate whether BEAM’s masking decisions follow routing rank, we record the minimum masked rank and maximum kept rank per layer across all three models \(Figure[8](https://arxiv.org/html/2605.14438#A2.F8)\)\. Across all layers and models, the min masked rank stays as low as 1–3, meaning that even highly\-ranked experts are frequently pruned when redundant for a given token\. Meanwhile, the max kept rank extends to the lower end of the Top\-K range, confirming that low\-ranked experts can be retained when critical\. The wide overlap between masked and kept ranks demonstrates that BEAM’s decisions are driven by token\-expert relevance rather than routing position\. ### B\.5Expert Load\-Balance Analysis To investigate BEAM’s impact on load balancing in MoE models, we visualize the utilization rates of experts for both original model and BEAM\-augmented model during inference in our studied MoE models, as shown in Figure[9](https://arxiv.org/html/2605.14438#A2.F9)\. The results demonstrate that the BEAM method performs uniform masking across experts, maintaining relatively balanced expert loads even on models such as Qwen3\-30B\-A3B that contains 128 experts\. This finding highlights the applicability of the BEAM method to large\-scale expert\-parallel MoE models\. Figure 9:Expert load balance visualization of MoE models before and after BEAM fine\-tuning\. ### B\.6Task\-specific Inference Speed Analysis Table 9:Task\-specific acceleration comparisons on Qwen3\-30B\-A3B\.ModelMATHGSM8KH\_EvalMMLUCEVALCMMLUBoolQCSQAAllQwen3\-30B\-A3B2667s329s43s980s77s460s89s46s4691sBEAM2058s247s28s709s54s330s81s36s3543sSpeedup1\.30x1\.33x1\.53x1\.38x1\.42x1\.39x1\.10x1\.27x1\.32x To further evaluate BEAM’s acceleration across tasks, we measure the inference speed of the BEAM\-augmented Qwen3\-MoE model \(β=0\.1\\beta=0\.1\) and its baseline on several evaluation benchmarks using vLLM on a single NVIDIA H20 GPU\. The results are summarized in Table[9](https://arxiv.org/html/2605.14438#A2.T9)\. BEAM achieves consistent speedups across all tasks, indicating efficiency improvements in both prefill and decoding\. ### B\.7Token\-Layer Sparsity Visualization We visualize the per\-token and per\-layer expert activation patterns of Qwen1\.5\-MoE\-A2\.7B, DeepSeekV2\-Lite, and Qwen3\-30B\-A3B on the same input prompt: “In only one sentence, what do you think of the future of AI?”\. The results are shown in Figures[10](https://arxiv.org/html/2605.14438#A2.F10),[11](https://arxiv.org/html/2605.14438#A2.F11), and[12](https://arxiv.org/html/2605.14438#A2.F12), which show significant differences in activation patterns across models and layers\. Chat template tokens such as ”You are a helpful assistant” activate almost no experts across all models\. Notably, the degree of prefill\-decode divergence varies by model: Qwen3 exhibits substantially more expert activation during decoding than prefill, Qwen1\.5\-MoE shows a moderate increase, while DeepSeekV2\-Lite maintains largely consistent activation across both phases\. For Qwen models, we also observe that prefill tokens mainly activate experts in shallow layers whereas decode tokens make stronger use of deeper layers, developing an encoder\-decoder\-like pattern where shallow layers handle knowledge encoding and deeper layers focus on reasoning\. Figure 10:Per\-token and per\-layer expert activation heatmap for DeepSeekV2\-Lite\. Each cell indicates the number of activated experts for a token \(vertical axis\) at a given layer \(horizontal axis\)\.Figure 11:Per\-token and per\-layer expert activation heatmap for Qwen1\.5\-MoE\-A2\.7B\. Each cell indicates the number of activated experts for a token \(vertical axis\) at a given layer \(horizontal axis\)\.Figure 12:Per\-token and per\-layer expert activation heatmap for Qwen3\-30B\-A3B\. Each cell indicates the number of activated experts for a token \(vertical axis\) at a given layer \(horizontal axis\)\.
Similar Articles
dMoE: dLLMs with Learnable Block Experts
dMoE proposes block-level expert routing for diffusion LLMs, reducing the number of uniquely activated experts from 69.5 to 14.6 while retaining 99.11% performance and achieving 76-80% memory reduction with 1.14-1.66× speedup.
EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs
EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free LLMs, using dynamic byte patches as routing units to enable sparse conditional computation. Experiments show it achieves the lowest held-out bits-per-byte among baselines while maintaining downstream accuracy.
ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
ELDR is an expert-locality-aware decode router for prefill-decode disaggregated Mixture-of-Experts serving that predicts expert activations from prefill signatures and routes requests to minimize latency, implemented in vLLM and achieving 5.9-13.9% reduction in median TPOT on up to 40 GPUs.
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
This paper proposes UniF-MoE, a unified framework for token-adaptive Mixture-of-Experts computation that first shares reusable computation across experts and then routes the remaining residual demand, improving performance while reducing activated computation, latency, and memory on DomainBed and GLUE benchmarks.
Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations
This paper investigates whether layer-wise differences exist in Mixture-of-Experts (MoE) models and proposes EAACD, an expert-aware adaptive contrast decoding method that leverages expert activation patterns in higher layers to reduce hallucinations in LLMs for QA tasks.