[Paper] ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning

# Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Source: [https://arxiv.org/html/2501.15316](https://arxiv.org/html/2501.15316)
Shangqian Gaosg24bi@fsu\.eduThanks:Corresponding author\.Affiliation:Department of Computer Science, Florida State UniversityHua Tingthua@nd\.eduReza Shirkavandrezashkv@cs\.umd\.eduAffiliation:Department of Computer Science, University of Maryland, College ParkChi\-Heng Linchiheng\.lin@samsung\.comAffiliation:Samsung Research AmericaZheng Tangzheng\.tang@samsung\.comAffiliation:Samsung Research AmericaZhengao Lizl23i@fsu\.eduAffiliation:Department of Computer Science, Florida State UniversityLongge Yuanly23a@fsu\.eduAffiliation:Department of Computer Science, Florida State UniversityFangyi Lifangyili@seas\.upenn\.eduAffiliation:School of Engineering and Applied Science, University of PennsylvaniaZeyu Zhangzeyzhan@amazon\.comAffiliation:Amazon AGIAlireza Ganjdaneshaliganj@umd\.eduAffiliation:Department of Computer Science, University of Maryland, College ParkLou Qianqian\.lou@ucf\.eduAffiliation:Department of Computer Science, University of Central FloridaXu Jiexujie@ufl\.eduAffiliation:Department of Health Outcomes and Biomedical Informatics, University of FloridaYen\-Chang Hsuyenchang\.hsu@samsung\.comAffiliation:Samsung Research America

###### Abstract

Large Language Models \(LLMs\) demonstrate remarkable capabilities but face deployment challenges due to their high computational demands\. Traditional pruning methods reduce these costs by permanently removing parameters, which inevitably leads to performance degradation\. To mitigate this issue, we propose ToMoE, a method that transforms dense LLMs into Mixture\-of\-Experts \(MoE\) models by uncovering experts inherently present within dense models, without requiring any weight updates\. ToMoE leverages dynamic structural pruning to unify expert construction and router training in a single stage, achieving consistently strong performance\. Remarkably, even without fine\-tuningthe model weights, ToMoE consistently outperforms state\-of\-the\-art pruning and MoE techniques across Phi\-2, LLaMA\-2, LLaMA\-3, and Qwen\-2\.5 models\. The code for this paper is available at[https://github\.com/gaosh/ToMoE](https://github.com/gaosh/ToMoE)\.

## 1Introduction

Although LLMs demonstrate remarkable capacity to perform diverse tasks\([8](https://arxiv.org/html/2501.15316#bib.bib29);[34](https://arxiv.org/html/2501.15316#bib.bib30);[56](https://arxiv.org/html/2501.15316#bib.bib31);[50](https://arxiv.org/html/2501.15316#bib.bib32);[3](https://arxiv.org/html/2501.15316#bib.bib33);[16](https://arxiv.org/html/2501.15316#bib.bib36);[55](https://arxiv.org/html/2501.15316#bib.bib34);[33](https://arxiv.org/html/2501.15316#bib.bib35)\), their huge model size often limits their usability on devices with limited resources\. As a result, considerable efforts\([45](https://arxiv.org/html/2501.15316#bib.bib12);[5](https://arxiv.org/html/2501.15316#bib.bib11);[21](https://arxiv.org/html/2501.15316#bib.bib37)\)are focused on minimizing the computational and memory costs of these models\. Structural pruning\([45](https://arxiv.org/html/2501.15316#bib.bib12)\)has emerged as a promising solution to this challenge because, unlike unstructured pruning, it achieves compression without the need for specialized implementations\. However, the problem with structural pruning methods is that they will substantially reduce the model capacity, resulting in an obvious performance gap compared to the dense model\. The fine\-tuning cost for even partially recovering this gap is tremendous\.

To achieve a better trade\-off between the number of parameters and performance, sparse Mixture of Experts \(MoE\) models\([58](https://arxiv.org/html/2501.15316#bib.bib38);[38](https://arxiv.org/html/2501.15316#bib.bib39)\)are designed to activate only a subset of the model’s parameters, corresponding to the selected experts\. Recently proposed MoE models, such as DeepseekMoE\([13](https://arxiv.org/html/2501.15316#bib.bib40)\), demonstrated that they can match the performance of dense models with a similar total parameter count while using a small number of active parameters\. Following this lead, transforming dense models into MoE models could offer a promising approach to bridging the performance gap left by structural pruning methods\. Unlike prior efforts to construct MoE models from dense models\([71](https://arxiv.org/html/2501.15316#bib.bib61);[37](https://arxiv.org/html/2501.15316#bib.bib60);[73](https://arxiv.org/html/2501.15316#bib.bib41)\), our findings reveal that MoE inherently exists within dense models and can be uncovered without updating model weights \(continue pretraining\)\. Specifically, we show that these experts can be identified through dynamic structural pruning\. These results represent a novel contribution that has not been demonstrated in previous studies\.

The core idea of MoE models is conditional computation, where experts are dynamically selected based on input tokens\. This concept aligns closely with dynamic pruning methods\([25](https://arxiv.org/html/2501.15316#bib.bib15)\), which make pruning decisions given input features\. Leveraging this connection, we propose to construct MoE models from dense models by using dynamic structural pruning\. Specifically, for Multi\-Head self\-Attention \(MHA\) layers, we apply top\-K routing and static pruning for compression, while for MLP layers, we transform them into MoE layers using top\-1 expert routing\. The routing mechanism learned for dynamic structural pruning can be directly applied to serve as the routing module for MoE layers\. With differentiable discrete operations, the MoE conversion process can be formulated as a differentiable dynamic pruning problem\. With this formulation, we can efficiently convert a dense model to an MoE model at a cost similar to or lower than regular structural pruning methods\. The comparison between our method, static pruning, and the original LLM is shown in Fig\.[1](https://arxiv.org/html/2501.15316#S2.F1)\.

Built upon the above findings and techniques, we proposed ToMoE to effectively convert dense LLMs to MoE models with dynamic pruning\. The contributions of this work can be summarized as follows:

- •Dense\-to\-MoE Conversion Through Dynamic Pruning:We introduce a novel approach to convert dense models into MoE models through dynamic pruning\. Specifically, we implement top\-K routing and static pruning for MHA layers along the head dimension and top\-1 routing for MLP layers across the learned experts\. This formulation ensures sparse and efficient computation while retaining model capacity\.
- •Joint Optimization for Routing and Experts:The proposed method involves jointly optimizing routing modules and expert configurations by solving a regularized optimization problem\. Our approach leverages differentiable operations to enable efficient and flexible MoE constructions\.
- •Consistent Performance Improvements:Our method consistently outperforms state\-of\-the\-art structural pruning and MoE construction techniques on various taskswhile training only the router, without fine\-tuning the model weights\. This performance improvement is demonstrated across widely used public models such as Phi\-2, LLaMA\-2, LLaMA\-3, and Qwen\-2\.5\.
- •Detailed Analysis:We extensively analyze the resulting model from ToMoE across multiple perspectives, including parameter allocation, router behavior, and the ablation of different design components\. We hope these analyses provide valuable insights and guidance for future research in this area\.

## 2Related Works

\(a\)
\(b\)
\(c\)

Figure 1:\(a\):The original LLM uses all parameters to process all the input text\.\(b\):The static pruning for LLMs permanently removes model parameters, and the rest of the parameters are used to process all the input text\.Our approach \(c\):LLMs with dynamic pruning use different sub\-networks \(illustrated by different colors\) to process different tokens\. We incorporate MoE to achieve a fixed expected budget in inference\.Pruning:Structural pruning\([40](https://arxiv.org/html/2501.15316#bib.bib59);[35](https://arxiv.org/html/2501.15316#bib.bib1);[45](https://arxiv.org/html/2501.15316#bib.bib12)\)is an attractive technique for real\-world deployments since it removes redundant parameters to reduce model size without requiring specialized implementations\. Structural pruning methods fall into two main categories: static pruning\([4](https://arxiv.org/html/2501.15316#bib.bib2);[48](https://arxiv.org/html/2501.15316#bib.bib6);[18](https://arxiv.org/html/2501.15316#bib.bib20)\)and dynamic pruning\([25](https://arxiv.org/html/2501.15316#bib.bib15);[10](https://arxiv.org/html/2501.15316#bib.bib19);[2](https://arxiv.org/html/2501.15316#bib.bib3);[14](https://arxiv.org/html/2501.15316#bib.bib4)\)\. Static pruning removes parameters based on input\-agnostic importance metrics\. For example, LLM\-Pruner\([45](https://arxiv.org/html/2501.15316#bib.bib12)\)eliminates non\-essential coupled structures using gradient\-based criteria\. The problem with structural pruning is that it often creates a noticeable performance gap relative to dense models\([45](https://arxiv.org/html/2501.15316#bib.bib12);[5](https://arxiv.org/html/2501.15316#bib.bib11)\)\. In contrast, dynamic pruning removes weights based on input\-dependent metrics\. Early attempts for dynamic pruning\([25](https://arxiv.org/html/2501.15316#bib.bib15);[10](https://arxiv.org/html/2501.15316#bib.bib19)\)focus on Convolutional Neural Networks, where channels are selectively activated for input samples\. Recent works, such as D\-LLM\([67](https://arxiv.org/html/2501.15316#bib.bib5)\), incorporate the concept of conditional computation into LLMs by selectively skipping layers based on input tokens\. The problem with dynamic pruning methods is that they do not have a fixed budget given different inputs, which creates problems when serving LLMs in a mini\-batch setting or in the prefilling stage\. Our method, on the other hand, converts the dense LLM to a sparse MoE model with a fixed per\-token budget\.

Another line of research applies contextual sparsity for LLMs\([43](https://arxiv.org/html/2501.15316#bib.bib62);[72](https://arxiv.org/html/2501.15316#bib.bib63);[36](https://arxiv.org/html/2501.15316#bib.bib64)\), where neurons are selectively activated given certain conditions\. Although there are some promising results, they are generally more difficult to achieve better inference efficiency due to their irregular sparsity patterns\. In contrast, MoE models have more comprehensive support from the system side, making them a popular choice for scaling up the model\. Thus, our method mainly focuses on converting dense models to MoE models\.

MoE:Sparse Mixture\-of\-Experts \(MoE\) models improve upon pure structural pruning by maintaining or even enhancing model capacity without a proportional increase in computational cost\. For instance, Sparsely\-Gated MoE\([58](https://arxiv.org/html/2501.15316#bib.bib38)\)employs a trainable gating network to select a small subset of experts for each input, enabling the model to scale to thousands of experts efficiently\([38](https://arxiv.org/html/2501.15316#bib.bib39)\)\. More recent methods like DeepSeekMoE\([13](https://arxiv.org/html/2501.15316#bib.bib40)\)further address expert specialization, matching dense\-model performance with a similar number of activated parameters\. Previous methods constructing MoE from the dense model\([71](https://arxiv.org/html/2501.15316#bib.bib61);[37](https://arxiv.org/html/2501.15316#bib.bib60);[73](https://arxiv.org/html/2501.15316#bib.bib41)\)separate the expert construction and router training into two distinct stages, often leading to sub\-optimal performance\. In contrast, our method integrates expert construction directly into the pruning process, treating it as a unified step with router learning and thereby largely improving the performance without fine\-tuning\.

## 3ToMoE

Figure 2:ToMoEuses top\-1 routing for MLP layers, and static and dynamic pruning along the head dimension for MHA layers\.Most recent LLMs, like GPT\([54](https://arxiv.org/html/2501.15316#bib.bib42)\), LLaMA\([64](https://arxiv.org/html/2501.15316#bib.bib9)\), etc\., adapt decoder\-only architectures and thus our method focuses on decoder\-only architectures\. A typical decoder block consists of Multi\-Head Attention \(MHA\) and Multi\-Layer Perceptron \(MLP\) layers\. For clarity, we denote the sequence length byTT, the hidden dimension bydd, the MLP intermediate dimension bydmidd\_\{\\text\{mid\}\}, and the number of attention heads byHH\.

To reduce the computational costs of the decoder\-only architecture, we propose to convert the original model into MoE models\. For MHA layers, we utilize top\-K routing and static pruning along the head dimensiondH\\frac\{d\}\{H\}\. Top\-K routing and static pruning for MHA layers ensure that, during prefilling or model serving, all tokens maintain the same head dimension, enabling parallel processing\. For MLP layers, our approach transforms them into MoE layers along the MLP middle dimensiondmidd\_\{\\text\{mid\}\}and employs top\-1 routing\. A key distinction between our method and previous dynamic pruning approaches is that the converted model maintains consistent computational costs for all inputs\. This property could be crucial for efficient processing\.

### 3\.1Expert Embeddings

Inspired by the recent success of using hypernetworks\([26](https://arxiv.org/html/2501.15316#bib.bib17);[22](https://arxiv.org/html/2501.15316#bib.bib56);[24](https://arxiv.org/html/2501.15316#bib.bib43)\)to generate pruning decisions, we adopt a hypernetwork to generate expert embeddings:

𝐄all=HN\(z\),~\\mathbf\{E\}\_\{\\text\{all\}\}=\\text\{HN\(z\)\},\(1\)wherezzis the input to the hypernetwork drawn from a random distribution, and𝐄all=\[𝐄1,⋯,𝐄l,⋯,𝐄L\]\\mathbf\{E\}\_\{\\text\{all\}\}=\[\\mathbf\{E\}\_\{1\},\\cdots,\\mathbf\{E\}\_\{l\},\\cdots,\\mathbf\{E\}\_\{L\}\]contains embeddings for all layers and𝐄l∈ℝN×de\\mathbf\{E\}\_\{l\}\\in\\mathbb\{R\}^\{N\\times d\_\{e\}\}, whereNNis the number of experts andded\_\{e\}is the expert embedding dimension\. Each embedding𝐄l,i\\mathbf\{E\}\_\{l,i\}will then be used to generate the configurations of experts\. The purpose of having the hypernetwork to generate𝐄all\\mathbf\{E\}\_\{\\text\{all\}\}is to introduce inter\-layer dependencies across different layers and operations\. This design has been shown to accelerate the learning process in practice\([24](https://arxiv.org/html/2501.15316#bib.bib43)\)\. More details are given in the Appendix[A](https://arxiv.org/html/2501.15316#A1)\.

### 3\.2Expert Construction

In this section, we will talk about how to construct experts from MLP layers\. In a decoder layer, the formulation of MLP is:fMLP​\(𝐗\)=σ⁡\(𝐗𝐖G\)⊙\(𝐗𝐖U\)​𝐖Df\_\{\\text\{MLP\}\}\(\\mathbf\{X\}\)=\\sigma\(\\mathbf\{X\}\\mathbf\{W\}\_\{G\}\)\\odot\(\\mathbf\{X\}\\mathbf\{W\}\_\{U\}\)\\mathbf\{W\}\_\{D\}, where matrices𝐖U∈ℝd×dmid\\mathbf\{W\}\_\{U\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\text\{mid\}\}\},𝐖G∈ℝd×dmid\\mathbf\{W\}\_\{G\}\\in\\mathbb\{R\}^\{d\\times d\_\{\\text\{mid\}\}\}and𝐖D∈ℝdmid×d\\mathbf\{W\}\_\{D\}\\in\\mathbb\{R\}^\{d\_\{\\text\{mid\}\}\\times d\}denote up, gated, and down projection matrices\. In addition,σ\\sigmadenotes nonlinear activation functions and⊙\\odotdenotes the Hadamard product \(element\-wise product\)\.

Assume the target is to useNNexperts, under the setting of structural pruning, each expert can be represented by:

fMLPi​\(𝐗t\)=σ⁡\(𝐗t​𝐖G​𝐒i\)⊙\(𝐗t​𝐖U​𝐒i\)​𝐒i⊤​𝐖D,~f\_\{\\text\{MLP\}\}^\{i\}\(\\mathbf\{X\}\_\{t\}\)=\\sigma\(\\mathbf\{X\}\_\{t\}\\mathbf\{W\}\_\{G\}\\mathbf\{S\}\_\{i\}\)\\odot\(\\mathbf\{X\}\_\{t\}\\mathbf\{W\}\_\{U\}\\mathbf\{S\}\_\{i\}\)\\mathbf\{S\}\_\{i\}^\{\\top\}\\mathbf\{W\}\_\{D\},\(2\)wherei=1,⋯,Ni=1,\\cdots,N, and𝐒i=Diag​\(𝐬i\)\\mathbf\{S\}\_\{i\}=\\text\{Diag\}\(\\mathbf\{s\}\_\{i\}\)\(𝐬i∈ℝdmid\\mathbf\{s\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\text\{mid\}\}\},𝐒i∈ℝdmid×dmid\\mathbf\{S\}\_\{i\}\\in\\mathbb\{R\}^\{d\_\{\\text\{mid\}\}\\times d\_\{\\text\{mid\}\}\}\), is a binary diagonal matrix selecting a subset of weight vectors for theiith expert\.𝐗t\\mathbf\{X\}\_\{t\}is thettth token, which is assumed to be routed to theiith expert\. Once each expert is formulated, its configuration is learned as follows:

𝐬=ST\-GSig​\(ProjDMLP​\(𝐆𝐄\)\),𝐆=ST\-GSmax​\(Router​\(𝐗\)\),~\\mathbf\{s\}=\{\\text\{ST\\mbox\{\-\}GSig\}\}\(\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}\(\\mathbf\{G\}\\mathbf\{E\}\)\),\\ \\mathbf\{G\}=\{\\text\{ST\\mbox\{\-\}GSmax\}\}\(\\text\{Router\}\(\\mathbf\{X\}\)\),\(3\)where𝐆∈ℝT×N\\mathbf\{G\}\\in\\mathbb\{R\}^\{T\\times N\}is the output of the router module,𝐄\\mathbf\{E\}\(llis omitted for clarity\) is the expert embeddings,ProjDMLP:ℝde→ℝdmid\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}:\\mathbb\{R\}^\{d\_\{e\}\}\\to\\mathbb\{R\}^\{d\_\{\\text\{mid\}\}\}is a projection module to project the latent embedding to the MLP middle dimension,Router​\(⋅\):ℝde→ℝN\\text\{Router\}\(\\cdot\):\\mathbb\{R\}^\{d\_\{e\}\}\\to\\mathbb\{R\}^\{N\}is the router module that maps the inputs to anNN\-dimensional routing score vector for expert selection, andST\-GSigandST\-GSmaxare Straight\-Through Gumbel\-Sigmoid and Gumbel\-Softmax functions respectively\([29](https://arxiv.org/html/2501.15316#bib.bib7)\)\. Under this setting,𝐬i\\mathbf\{s\}\_\{i\}will contain retained positions \(represented by11\) for theiith expert, and𝐆\\mathbf\{G\}contains one\-hot routing decisions for tokens in𝐗\\mathbf\{X\}\.

### 3\.3MHA top\-K Routing

An MHA layer can be represented asfMHA​\(𝐗\)=∑i=1Hσs​\(e⁡\(𝐗𝐖Q,i\)​e⊤​\(𝐗𝐖K,i\)\)​𝐗𝐖V,i​𝐖O,if\_\{\\text\{MHA\}\}\(\\mathbf\{X\}\)=\\sum\_\{i=1\}^\{H\}\\sigma\_\{s\}\\left\(e\(\\mathbf\{X\}\\mathbf\{W\}\_\{Q,i\}\)e^\{\\top\}\(\\mathbf\{X\}\\mathbf\{W\}\_\{K,i\}\)\\right\)\\mathbf\{X\}\\mathbf\{W\}\_\{V,i\}\\mathbf\{W\}\_\{O,i\}, where𝐖Q,i,𝐖K,i,𝐖V,i∈ℝd×dH\\mathbf\{W\}\_\{Q,i\},\\mathbf\{W\}\_\{K,i\},\\mathbf\{W\}\_\{V,i\}\\in\\mathbb\{R\}^\{d\\times\\frac\{d\}\{H\}\},𝐖O,i∈ℝdH×d\\mathbf\{W\}\_\{O,i\}\\in\\mathbb\{R\}^\{\\frac\{d\}\{H\}\\times d\}are the query, key, value, and output matrices for each attention head, and𝐗∈ℝT×d\\mathbf\{X\}\\in\\mathbb\{R\}^\{T\\times d\}is the input hidden states\.e\{e\}andσs\\sigma\_\{s\}denote positional embedding and the softmax function\.

For MHA layers, we perform two kinds of pruning: dynamic top\-K pruning and static pruning, both along the head dimension\. Justifications regarding the design choice are provided in the Appendix[F](https://arxiv.org/html/2501.15316#A6)\. Like MLP layers, we also insert selection matrices:

fMHA​\(𝐗t\)=∑i=1H\[σs​\(e⁡\(𝐗t​𝐖Q,i\)​𝐒0​𝐒0⊤​e⊤​\(𝐗t​𝐖K,i\)\)​𝐗t​𝐖V,i​𝐒t\]​𝐒t⊤​𝐖O,i,f\_\{\\text\{MHA\}\}\(\\mathbf\{X\}\_\{t\}\)=\\sum\_\{i=1\}^\{H\}\\left\[\\sigma\_\{s\}\\left\(e\(\\mathbf\{X\}\_\{t\}\\mathbf\{W\}\_\{Q,i\}\)\\mathbf\{S\}\_\{0\}\\mathbf\{S\}\_\{0\}^\{\\top\}e^\{\\top\}\(\\mathbf\{X\}\_\{t\}\\mathbf\{W\}\_\{K,i\}\)\\right\)\\mathbf\{X\}\_\{t\}\\mathbf\{W\}\_\{V,i\}\\mathbf\{S\}\_\{t\}\\right\]\\mathbf\{S\}\_\{t\}^\{\\top\}\\mathbf\{W\}\_\{O,i\},\(4\)where𝐒0,𝐒t∈ℝdH×dH\\mathbf\{S\}\_\{0\},\\mathbf\{S\}\_\{t\}\\in\\mathbb\{R\}^\{\\frac\{d\}\{H\}\\times\\frac\{d\}\{H\}\}are selection matrices\.𝐒0\\mathbf\{S\}\_\{0\}is the shared selection matrix for static pruning of query and key matrices, while𝐒t\\mathbf\{S\}\_\{t\}is the token\-specific selection matrix for the value and output matrices of thett\-th token\. We apply the same selection matrix across all heads, ensuring that all heads have the same head dimensions at inference time\. To generate the selection matrix, we calculate its diagonal vector𝐬t\\mathbf\{s\}\_\{t\}as:

𝐬t=ST\-GSig​\(ProjDMHA​\(ProjEMHA​\(𝐗t\)\+1N​𝟏⊤​𝐄\)\),~\\mathbf\{s\}\_\{t\}=\{\\text\{ST\\mbox\{\-\}GSig\}\}\(\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MHA\}\}\(\\text\{Proj\}\_\{\\text\{E\}\}^\{\\text\{\\tiny MHA\}\}\(\\mathbf\{X\}\_\{t\}\)\+\\frac\{1\}\{N\}\\mathbf\{1\}^\{\\top\}\\mathbf\{E\}\)\),\(5\)where𝟏∈ℝN\\mathbf\{1\}\\in\\mathbb\{R\}^\{N\}is an all\-one vector,1N​𝟏⊤​𝐄\\frac\{1\}\{N\}\\mathbf\{1\}^\{\\top\}\\mathbf\{E\}represents the average expert embedding of sizeded\_\{e\},ProjDMHA:ℝde→ℝdH\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MHA\}\}:\\mathbb\{R\}^\{d\_\{e\}\}\\to\\mathbb\{R\}^\{\\frac\{d\}\{H\}\}is a projection module to map the latent embedding to the head dimension andProjEMHA:ℝd→ℝde\\text\{Proj\}\_\{\\text\{E\}\}^\{\\text\{\\tiny MHA\}\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{e\}\}is also a projection module to project input tokens to the space of expert embeddings, andST\-GSigis defined in Sec\.[3\.2](https://arxiv.org/html/2501.15316#S3.SS2)\. Whent=0t=0, we initialize𝐗=0\\mathbf\{X\}=0, and set𝐬0=ST\-GSig​\(ProjDMHA​\(1N​𝟏⊤​𝐄\)\)\\mathbf\{s\}\_\{0\}=\{\\text\{ST\\mbox\{\-\}GSig\}\}\(\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MHA\}\}\(\\frac\{1\}\{N\}\\mathbf\{1\}^\{\\top\}\\mathbf\{E\}\)\), since it is input independent\.

During training, the number of ones in𝐬\\mathbf\{s\}can vary freely\. After training is complete, we computeK=round​\(1T​∑t=1T∑i=1dH𝐬t,i\)K=\\text\{round\}\(\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\sum\_\{i=1\}^\{\\frac\{d\}\{H\}\}\\mathbf\{s\}\_\{t,i\}\)for a subset of tokens and use it for top\-K routing during inference\. Note that thekkin top\-K for𝐬0\\mathbf\{s\}\_\{0\}and𝐬t\\mathbf\{s\}\_\{t\}\(t≥1t\\geq 1\) can be different, allowing for larger flexibility\. Also, note that𝐬0\\mathbf\{s\}\_\{0\}must follow specific structural constraints to be compatible with the position embeddinge⁡\(⋅\)e\(\\cdot\), and more details can be found in Appendix[A\.2](https://arxiv.org/html/2501.15316#A1.SS2)\.

### 3\.4Regularizations for MoE Constructions

In Sec\.[3\.2](https://arxiv.org/html/2501.15316#S3.SS2)and Sec\.[3\.3](https://arxiv.org/html/2501.15316#S3.SS3), we briefly introduced the design space for constructing MoE models using dynamic structural pruning\. In this subsection, we will introduce regularizations customized to the characteristics of MoE models\.

Union of Experts Regularization\.An ideal sparse MoE model converted from a dense model should maximize parameter utilization, which means that the total number of parameters in the MoE model should closely approximate the dense model\. To add this regularization to our learning process, we push the union of experts to be closer to the original model\. More specifically, we use MHA layers as an example:

𝐮=⋃i=1T𝐬i=1−∏i=1T\(1−𝐬i\),~\\mathbf\{u\}=\\bigcup\_\{i=1\}^\{T\}\\mathbf\{s\}\_\{i\}=1\-\\prod\_\{i=1\}^\{T\}\(1\-\\mathbf\{s\}\_\{i\}\),\(6\)where⋃\\bigcupis the union operator, and𝐮\\mathbf\{u\}is the union of all kept positions for each token\. For MLP layers, it can be calculated similarly\. We then push∑𝐮\|𝐮\|\\frac\{\\sum\\mathbf\{u\}\}\{\|\\mathbf\{u\}\|\}\(\|𝐮\|\|\\mathbf\{u\}\|represents the size of𝐮\\mathbf\{u\}\) to 1:

ℛU=1L​∑l=1Lfreg​\(∑𝐮l\|𝐮l\|,1\),~\\mathcal\{R\}\_\{\\text\{U\}\}=\\frac\{1\}\{L\}\\sum\_\{l=1\}^\{L\}f\_\{\\text\{reg\}\}\(\\frac\{\\sum\\mathbf\{u\}\_\{l\}\}\{\|\\mathbf\{u\}\_\{l\}\|\},1\),\(7\)wherefreg​\(⋅,⋅\)f\_\{\\text\{reg\}\}\(\\cdot,\\cdot\)can be any regression loss functions, and we will choosefregf\_\{\\text\{reg\}\}later\.

Parameter Regularization\.For a sparse MoE model, we also need to control the number of active parameters given the provided budget\. To achieve this goal, we can directly penalize the maximum width across different experts\. We choose the maximum width over experts instead of the mean, median, or other alternatives because the maximum provides precise control over the upper bound of the number of active parameters\.

Denote the width of a layer asdl∗d\_\{l\}^\{\*\}, where∗∈\{MLP,MHA\}\*\\in\\\{\\text\{MLP\},\\text\{MHA\}\\\}\. For MLP layers, it can be calculated bydlMLP=max⁡\(𝐬𝟏dmid\)d\_\{l\}^\{\\text\{ MLP\}\}=\\max\(\\mathbf\{s\}\\mathbf\{1\}\_\{d\_\{\\text\{mid\}\}\}\), where𝟏dmid∈ℝdmid\\mathbf\{1\}\_\{d\_\{\\text\{mid\}\}\}\\in\\mathbb\{R\}^\{d\_\{\\text\{mid\}\}\}is an all\-one vector of sizedmidd\_\{\\text\{mid\}\}\.𝐬𝟏mid\\mathbf\{s\}\\mathbf\{1\}\_\{\\text\{mid\}\}produces the width of all experts, anddlMLPd\_\{l\}^\{\\text\{ MLP\}\}represents the maximum width across all experts\. The width of MHA layers can be calculated similarly\. Based ondl∗d\_\{l\}^\{\*\}, we can calculate the number of active parameters in the modelT​\(𝐝MoE\)\\text\{T\}\(\\mathbf\{d\_\{\\text\{MoE\}\}\}\), where𝐝MoE=\[d1∗,⋯,dL∗\]\\mathbf\{d\_\{\\text\{MoE\}\}\}=\[d\_\{1\}^\{\*\},\\cdots,d\_\{L\}^\{\*\}\]\. To push the number of active parameters to a predefined ratepp, the following objective is applied:

ℛP=freg​\(T​\(𝐝MoE\),p​Ttotal\),~\\mathcal\{R\}\_\{\\text\{P\}\}=f\_\{\\text\{reg\}\}\(\\text\{T\}\(\\mathbf\{d\_\{\\text\{MoE\}\}\}\),p\\text\{T\}\_\{\\text\{total\}\}\),\(8\)whereTtotal\\text\{T\}\_\{\\text\{total\}\}is the total number of parameters, andp∈\(0,1\]p\\in\(0,1\]represents the ratios of the active parameters\. Forfregf\_\{\\text\{reg\}\}in Eq\.[8](https://arxiv.org/html/2501.15316#S3.E8)and Eq\.[7](https://arxiv.org/html/2501.15316#S3.E7), the following functionfregf\_\{\\text\{reg\}\}is used:

freg​\(x,y\)=log⁡\(max⁡\(x,y\)/min⁡\(x,y\)\)\.f\_\{\\text\{reg\}\}\(x,y\)=\\log\(\\max\(x,y\)/\\min\(x,y\)\)\.
Load Balancing Regularization\.When determining the configurations of experts, we also apply the load balancing regularization to encourage a balanced load across experts\([39](https://arxiv.org/html/2501.15316#bib.bib45);[19](https://arxiv.org/html/2501.15316#bib.bib44)\)\. The load balancing loss from the Switch Transformer\([19](https://arxiv.org/html/2501.15316#bib.bib44)\)is adopted:

ℛL=N​∑i=1NFi​Pi,~\\mathcal\{R\}\_\{\\text\{L\}\}=N\\sum\_\{i=1\}^\{N\}F\_\{i\}P\_\{i\},\(9\)whereFi=1T​∑t=1T𝟙​\(𝐆t,i=1\)F\_\{i\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\mathbbm\{1\}\(\\mathbf\{G\}\_\{t,i\}=1\)\. The indicator function𝟙​\(⋅\)\\mathbbm\{1\}\(\\cdot\)returns11if the condition is true and00otherwise\.FiF\_\{i\}represents the fraction of tokens assigned to theii\-th expert\.Pi=1T​∑GSmax​\(Gt,i\)P\_\{i\}=\\frac\{1\}\{T\}\\sum\\text\{GSmax\}\(\{G\}\_\{t,i\}\), whereG=Router​\(𝐗\)\{G\}=\\text\{Router\}\(\\mathbf\{X\}\)is the router outputbefore ST\-GSmax\.PiP\_\{i\}is the fraction of the router probability allocated for theii\-th expert\.ℛL\\mathcal\{R\}\_\{\\text\{L\}\}will encourages uniform routing across different experts as shown in\([19](https://arxiv.org/html/2501.15316#bib.bib44)\)\. The combination of Eq\.[8](https://arxiv.org/html/2501.15316#S3.E8)and Eq\.[9](https://arxiv.org/html/2501.15316#S3.E9)creates an interesting phenomenon where they encourage uniform allocation of width among experts\.

Table 1:Perplexity comparisons of structured pruning methods and ToMoE for LLaMA\-2 7B and 13B models on WikiText\-2\.
### 3\.5Learning to Construct MoEs

Based on the aforementioned techniques, MoEs can be constructed from the dense LLMs by training router parameters, projection parameters, and hypernetwork parameters, while keeping all original model parameters frozen\. This approach enables the rapid construction of an effective MoE model with a resource budget comparable to that of structural pruning\. The overall framework of our method is shown in Fig\.[2](https://arxiv.org/html/2501.15316#S3.F2), and the corresponding training objective function can be formulated as:

minΘ⁡ℒ⁡\(f′​\(x,𝐄all\),f⁡\(x\)\)\+α​ℛ𝐏\+β​ℛ𝐔\+γ​ℛ𝐋,~\\min\_\{\\Theta\}\\ \\mathcal\{L\}\(f^\{\\prime\}\(x;\\mathbf\{E\}\_\{\\text\{all\}\}\),f\(x\)\)\+\\alpha\\mathcal\{R\}\_\{\\mathbf\{P\}\}\+\\beta\\mathcal\{R\}\_\{\\mathbf\{U\}\}\+\\gamma\\mathcal\{R\}\_\{\\mathbf\{L\}\},\(10\)
Table 2:Comparisons with semi\-structured pruning on LLaMA\-2\.whereΘ=\[ΘHN,ΘRouter,ΘProj\-MHA,ΘProj\-MLP\]\\Theta=\[\\Theta\_\{\\text\{HN\}\},\\Theta\_\{\\text\{Router\}\},\\Theta\_\{\\text\{Proj\-MHA\}\},\\Theta\_\{\\text\{Proj\-MLP\}\}\],ΘHN\\Theta\_\{\\text\{HN\}\}is trainable parameters for the hypernetwork in Eq\.[1](https://arxiv.org/html/2501.15316#S3.E1),ΘRouter\\Theta\_\{\\text\{Router\}\}andΘProj\-MLP\\Theta\_\{\\text\{Proj\-MLP\}\}are trainable parameters for the router and the project module in Eq\.[3](https://arxiv.org/html/2501.15316#S3.E3),ΘProj\-MHA\\Theta\_\{\\text\{Proj\-MHA\}\}is the trainable parameters of the projection modules given in Eq\.[5](https://arxiv.org/html/2501.15316#S3.E5),ℛ𝐏\\mathcal\{R\}\_\{\\mathbf\{P\}\},ℛ𝐔\\mathcal\{R\}\_\{\\mathbf\{U\}\}, andℛ𝐋\\mathcal\{R\}\_\{\\mathbf\{L\}\}are regularization terms defined in Sec\.[3\.4](https://arxiv.org/html/2501.15316#S3.SS4)\. Andα\\alpha,β\\beta, andγ\\gammaare hyperparameters to control the strength of these regularization terms\. Here,ffrepresents the original dense model, andf′f^\{\\prime\}is the model equipped with our designed modules for MoE construction\. Under this setting, we useℒ⁡\(⋅,⋅\)\\mathcal\{L\}\(\\cdot,\\cdot\)to calculate the KL divergence between the logits offfandf′f^\{\\prime\}, which is used as the guidance to preserve the capacity of the dense model\([28](https://arxiv.org/html/2501.15316#bib.bib46)\)\. We also found that using the KL divergence alone can lead to the best performance, and this observation complies with the experimental setup in\([49](https://arxiv.org/html/2501.15316#bib.bib47)\)\. Also, note that we performin\-placeknowledge distillation since the original model weights are frozen\. Thus, the knowledge distillation process does not introduce overheads in terms of GPU memory\.

After learning how to construct the MoE, we convert the MLP layer toNNexperts with shared weights\. After pruning the MLP layer, we save1N​𝟏⊤​𝐄\\frac\{1\}\{N\}\\mathbf\{1\}^\{\\top\}\\mathbf\{E\}for MHA layers as the bias of theProjEMHA\\text\{Proj\}\_\{\\text\{E\}\}^\{\\text\{\\tiny MHA\}\}, and we dropProjDMLP\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}and convert Eq\.[5](https://arxiv.org/html/2501.15316#S3.E5)into a Top\-K routing function as well as use𝐬0\\mathbf\{s\}\_\{0\}for pruning𝐖Q\\mathbf\{W\}\_\{Q\}and𝐖K\\mathbf\{W\}\_\{K\}\. Our construction also enables converting the MoE back into a pseudo\-MoE model\. The MoE model and the pseudo\-MoE model are equivalent, and more details can be found in Appendix[B\.3](https://arxiv.org/html/2501.15316#A2.SS3)\.

Table 3:Zero\-shot task performance of compressed LLaMA\-2 7B, LLaMA\-3 8B, Qwen\-2\.5 7B\.

## 4Experiments

### 4\.1Settings

Models\.Our ToMoE method is evaluated using several LLMs with decoder blocks\. Specifically, we choose the following models: LLaMA\-2\([65](https://arxiv.org/html/2501.15316#bib.bib22)\): LLaMA\-2 7B and LLaMA\-2 13B; LLaMA\-3 8B\([17](https://arxiv.org/html/2501.15316#bib.bib48)\); Phi\-2\([30](https://arxiv.org/html/2501.15316#bib.bib21)\); Qwen\-2\.5\([69](https://arxiv.org/html/2501.15316#bib.bib49)\): Qwen\-2\.5 7B and Qwen\-2\.5 14B\. Results for LLaMA\-2 13B and Qwen\-2\.5 14B are presented in the Appendix[D](https://arxiv.org/html/2501.15316#A4)\.

Implementations\.ToMoE is implemented by Pytorch\([51](https://arxiv.org/html/2501.15316#bib.bib18)\)and Hugging Face transformer library\([68](https://arxiv.org/html/2501.15316#bib.bib8)\)\. The model weights are frozen when training the modules with learnable parametersΘ\\Thetain Obj\.[10](https://arxiv.org/html/2501.15316#S3.E10)\. We use the AdamW\([44](https://arxiv.org/html/2501.15316#bib.bib23)\)optimizer to optimizeΘ\\Theta, which is trained for 10,000 iterations for all models\. For all experiments, we setα=16\\alpha=16,β=2\.0\\beta=2\.0, andγ=1\.0\\gamma=1\.0, whereα\\alpha,β\\beta, andγ\\gammaare defined in Obj\.[10](https://arxiv.org/html/2501.15316#S3.E10)\. Without specific descriptions, the number of experts for ToMoE is88across all settings\. Depending on the size of the base model, 1 to 4 NVIDIA A100 GPUs are used to trainΘ\\Theta\. More implementation details can be found in Appendix[C](https://arxiv.org/html/2501.15316#A3)\.

Datasets\.Two training settings are provided for all modules with learnable parametersΘ\\Theta: \(1\) using WikiText[47](https://arxiv.org/html/2501.15316#bib.bib65), and \(2\) using a mixed dataset comprising WikiText, Alpaca[63](https://arxiv.org/html/2501.15316#bib.bib66), and Code\-Alpaca[9](https://arxiv.org/html/2501.15316#bib.bib67)\(mixing ratio: 1:1:1\)\. Based on our observation, ToMoE benefits from a diverse mixture of datasets to effectively construct experts\. Following previous methods\([5](https://arxiv.org/html/2501.15316#bib.bib11);[24](https://arxiv.org/html/2501.15316#bib.bib43)\), our method and other methods are evaluated on five well\-known zero\-shot tasks: PIQA\([7](https://arxiv.org/html/2501.15316#bib.bib24)\); WinoGrande\([57](https://arxiv.org/html/2501.15316#bib.bib26)\); HellaSwag\([70](https://arxiv.org/html/2501.15316#bib.bib25)\); ARC\-e and ARC\-c\([12](https://arxiv.org/html/2501.15316#bib.bib27)\)\.We further evaluate our method on the following tasks and configurations to ensure consistency with comparison baselines: 32\-shot BoolQ\([11](https://arxiv.org/html/2501.15316#bib.bib70)\), SciQ\([32](https://arxiv.org/html/2501.15316#bib.bib71)\), 5\-shot WinoGrande, 25\-shot ARC\-c, 10\-shot HellaSwag, TruthfulQA[42](https://arxiv.org/html/2501.15316#bib.bib72), and 5\-shot MMLU\([27](https://arxiv.org/html/2501.15316#bib.bib73)\)\.We use llm\-eval\-harness\([23](https://arxiv.org/html/2501.15316#bib.bib10)\)to evaluate the compressed models\.

Baselines\.ToMoE is compared to baselines from structural pruning methods\([45](https://arxiv.org/html/2501.15316#bib.bib12);[5](https://arxiv.org/html/2501.15316#bib.bib11);[46](https://arxiv.org/html/2501.15316#bib.bib52);[60](https://arxiv.org/html/2501.15316#bib.bib51);[66](https://arxiv.org/html/2501.15316#bib.bib13);[41](https://arxiv.org/html/2501.15316#bib.bib50);[24](https://arxiv.org/html/2501.15316#bib.bib43)\), semi\-structural pruning methods\([20](https://arxiv.org/html/2501.15316#bib.bib14);[62](https://arxiv.org/html/2501.15316#bib.bib28);[15](https://arxiv.org/html/2501.15316#bib.bib53)\)and MoE constructionmethods\([73](https://arxiv.org/html/2501.15316#bib.bib41);[37](https://arxiv.org/html/2501.15316#bib.bib60);[53](https://arxiv.org/html/2501.15316#bib.bib68);[52](https://arxiv.org/html/2501.15316#bib.bib69)\)\.

Table 4:Compassion against MoE construction methods\.Table 5:Comparison against MoE construction methods on LLaMA\-2 7B\.Table 6:Comparison against LLaMA\-MoE\-v2 on LLaMA\-3 8B\.Table 7:ToMoE Visualization for the last layer of the LLaMA\-2 7B model with 50% active parametersExpert Color:Expert 1Expert 2Expert 3Expert 4Expert 5Expert 6Expert 7Expert 8<s\>GrandTheftAutoVIisanupcomingvideogameindevelopmentbyRockstarGames\.ItisduetobetheeighthmainGrandTheftAutogame,followingGrandTheftAutoV\(2013\),andthesixteenthentryoverall\.SetwithinthefictionalopenworldstateofLeonidabasedonFloridaanditsMiami\-inspiredViceCity,thestoryisexpectedtofollowthecriminalduoofLuciaandhermalepartner\.\\nFollowingyearsofspeculationandanticipation,RockstarconfirmedinFebruary2022thatthegamewasindevelopment\.ThatSeptember,footagefromunfinishedversionswasleakedonlineinwhatjournalistsdescribedasoneofthebiggestleaksinthehistoryofthevideogameindustry\.ThegamewasformallyrevealedinDecember2023andisscheduledtobereleasedinlate2025forthePlayStation5andXboxSeriesX/S\.\\nGrandTheftAutoVIissetinthefictionalopenworldstateofLeonidabasedonFloridawhichincludesViceCity,afictionalisedversionofMiami\.ViceCitywaspreviouslyfeaturedinGrandTheftAuto\(1997\)andasthemainsettingofGrandTheftAuto:ViceCity\(2002\)andGrandTheftAuto:ViceCityStories\(2006\)\.Thegameworldparodies2020sAmericanculture,withsatiricaldepictionsofsocialmediaandinfluencerculture,andreferencestoInternetmemessuchasFloridaMan\.Thestoryfollowsacriminalduo:Lucia,theseries’firstfemaleprotagonistsince2000,andhermalepartner;thefirsttrailerdepictsLuciaasaprisoninmate,andlaterevadingcustodywithherpartner\.

![Refer to caption](https://arxiv.org/html/2501.15316v2/l_loss.png)

\(a\)
![Refer to caption](https://arxiv.org/html/2501.15316v2/lres_loss.png)

\(b\)
![Refer to caption](https://arxiv.org/html/2501.15316v2/lu_loss.png)

\(c\)
![Refer to caption](https://arxiv.org/html/2501.15316v2/lb_loss.png)

\(d\)

Figure 3:The training dynamics give different ratiosppof active parameters on the Qwen\-2\.5 7B model\.
### 4\.2Language Modeling

![Refer to caption](https://arxiv.org/html/2501.15316v2/routing_heatmap.png)Figure 4:Experts token allocation of ToMoE for the LLaMA\-3 8B model collected on the WikiText dataset\.![Refer to caption](https://arxiv.org/html/2501.15316v2/model_arch_qwen25_7b.png)Figure 5:Model width after ToMoE for the Qwen\-2\.5 7B model when the number of active parameters equals 50%\.![Refer to caption](https://arxiv.org/html/2501.15316v2/model_arch_qwen25_7b_uline.png)\(a\)
![Refer to caption](https://arxiv.org/html/2501.15316v2/training_costs.png)\(b\)
\(c\)
Figure 6:\(a\) Model width and union of experts\. \(b\) Costs of different learning\-based methods\. \(c\) Ablation study on Qwen\-2\.5 7B\.Tab\.[1](https://arxiv.org/html/2501.15316#S3.T1)presents the perplexity results of structured pruning methods applied to LLaMA\-2 models of sizes 7B and 13B on the WikiText\-2 dataset, comparing various methods with 70%, 60%, and 50% of active parameters setting—corresponding to pruning ratios of 30%, 40%, and 50% for pruning, respectively\. Across all pruning ratios, ToMoE consistently achieves the lowest perplexity compared to other methods, even outperforming many approaches with significantly larger numbers of active parameters\. For instance, ToMoE with 50% active parameters achieves a perplexity of 8\.36, which is superior to LLM\-Pruner, ShortGPT, SLEB, and SliceGPT at a 30% pruning ratio\. Furthermore, ToMoE with 50% active parameters surpasses ModeGPT and LLM Surgeon at a 40% pruning ratio\. While the gap between ToMoE and DISP\-LLM is smaller, it is still obvious at a 50% pruning ratio: ToMoE achieves a perplexity that is 1\.48 points lower than DISP\-LLM\. ToMoE also exhibits superior performance with the LLaMA\-2 13B model, maintaining a similar advantage over other methods as observed with the LLaMA\-2 7B model\. This demonstrates the effectiveness of ToMoE in maintaining strong language modeling performance, even with much fewer active parameters\. Tab\.[2](https://arxiv.org/html/2501.15316#S3.T2)presents a comparison of our method against semi\-structural pruning techniques\. Our approach consistently achieves the lowest perplexity while retaining 50% of the active parameters\. Moreover, the performance gap between our method and the semi\-structural pruning methods is also obvious\. On the LLaMA\-2 7B model, SparseGPT achieves the second\-best performance, with our method improving upon it by 1\.81 in terms of perplexity\. For the LLaMA\-2 13B model, Pruner\-Zero shows the second\-best performance, while ToMoE further reduces the perplexity by 0\.63\. The comparison against semi\-structural pruning methods further demonstrates the advantage of our method on the language modeling task\.

### 4\.3Zero\-Shot and Few\-Shot Performance

In Tab\.[3](https://arxiv.org/html/2501.15316#S3.T3), we present the zero\-shot performance of various methods on LLaMA\-2 7B, LLaMA\-3 8B, and Qwen\-2\.5 7B\. Our method consistently achieves the best average performance across all models\. ForLLaMA\-2 7B, compared to weaker methods like ShortGPT and SliceGPT, our approach demonstrates significant advantages \(ToMoE 50%: 60\.72 vs\. SliceGPT 40\.84 and ShortGPT 70%: 47\.07\)\. The advantage against stronger baselines is also obvious\. Although ModeGPT performs closer to ToMoE, the gap remains significant\. With 60% active parameters, ToMoE is 3\.14 times better than ModeGPT\. ForLLaMA\-3 8B, the performance advantage of ToMoE is even larger, where it reduces 5% more active parameters than ModeGPT while still achieving a 2\.71 performance gain\. Furthermore, when removing 15% more active parameters compared to ShortGPT and SliceGPT, ToMoE exceeds their average performance by 18\.93 and 16 points, respectively\. ForQwen\-2\.5 7B, ToMoE significantly outperforms DISP\-LLM, consistent with previous findings on other models\. We further investigate the effect of the number of expertsNNwhen 40% to 50% of the parameters are active\. The results indicate that increasing the number of experts to 16 is beneficial\. However, further increasingNNto 24 provides only marginal or no improvement, likely because a too\-large number of experts burdens the learning process\.Thus, we recommend choosing the number of expertsNNto be smaller than 16\.

In Tab\.[4](https://arxiv.org/html/2501.15316#S4.T4), our method demonstrated superb advantages compared to existing MoE construction methods\. In “\+fine\-tuning" setting of LLaMA\-MoE, the resulting model is trained for the same number of iterations as ToMoE for updating model weights\.In Tab\.[5](https://arxiv.org/html/2501.15316#S4.T5)and Tab\.[6](https://arxiv.org/html/2501.15316#S4.T6), we further compare our method with CMoE, LLaMA\-MoE, and LLaMA\-MoE\-v2, following the experimental settings in their papers\. Our approach consistently surpasses all baselines while requiring significantly fewer tokens\. Notably, in Tab\.[6](https://arxiv.org/html/2501.15316#S4.T6), when compared against the fully trained LLaMA\-MoE\-v2, our method achieves comparable performance even without additional fine\-tuning of the model weights\. In summary, our method shows that learning routers and experts together is a more promising solution compared to existing works\.

### 4\.4Analysis of ToMoE

Training Dynamics\.In Fig\.[3](https://arxiv.org/html/2501.15316#S4.F3), we visualize the training dynamics under different values ofpp\. Across allpp, the knowledge distillation lossℒ\\mathcal\{L\}\(Fig\.[3](https://arxiv.org/html/2501.15316#S4.F3)\(a\)\), the parameter regularization lossℛP\\mathcal\{R\}\_\{\\text\{P\}\}, and the union of experts regularization lossℛU\\mathcal\{R\}\_\{\\text\{U\}\}decrease over the course of training\. Notably, the parameter regularization loss quickly drops to00in the early stages of training, while using a smallerpprequires more iterations\. The peak of the union of experts regularization loss increases when using smaller values ofpp, indicating that the initial solution tends to only cover a small portion of the dense model\. Regarding the load balancing loss, it oscillates around 0\.15, demonstrating that ToMoE maintains a relatively balanced load distribution during the training process\.

Ablation Study\.In Fig\.[6\(c\)](https://arxiv.org/html/2501.15316#S4.F6.sf3), we present the average zero\-shot task performance under different settings\. Forp=0\.4p=0\.4andp=0\.5p=0\.5, replacing the knowledge distillation loss with the language modeling loss significantly impacts performance\. Atp=0\.4p=0\.4, removingℛU\\mathcal\{R\}\_\{\\text\{U\}\}also results in a substantial performance drop, whereas the impact is much smaller atp=0\.5p=0\.5\. We hypothesize that this difference arises because reducingppmakes the learning process more challenging\. Without the guidance provided byℛU\\mathcal\{R\}\_\{\\text\{U\}\}, the model struggles to effectively utilize the parameters of the original model\. Additionally, the choice of dataset affects performance, particularly when switching to the WikiText dataset\. This demonstrates that a mixing dataset is beneficial to the overall performance\.

Other Analysis\.\(1\)\.Fig\.[5](https://arxiv.org/html/2501.15316#S4.F5)presents the width of our ToMoE model for Qwen\-2\.5 7B, which shows the layer\-wise configuration is highly non\-uniform\. It demonstrates that our method can flexibly set the width of different layers and operations\.\(2\)\.Fig\.[6\(a\)](https://arxiv.org/html/2501.15316#S4.F6.sf1)shows that the union of experts is close to the full model capacity, even though the width of experts across different layers is highly non\-uniform, demonstrating the effectiveness of our loss design\.\(3\)\.Fig\.[6\(b\)](https://arxiv.org/html/2501.15316#S4.F6.sf2)plots the costs of different learning\-based methods in terms of US dollars\. ToMoE costs similarly compared to DISP\-LLM with LLaMA\-2 7B and 13B models, and both of them are much cheaper than LLM Surgeon\.\(4\)\.Fig\.[4](https://arxiv.org/html/2501.15316#S4.F4)shows the token allocation across experts on the Wikitext dataset\. We observe that the early and late layers exhibit relatively balanced expert utilization, while the middle layers have certain experts activated more frequently\.\(5\)\.Finally, we visualize the expert selection for LLaMA\-2 7B in Tab\.[7](https://arxiv.org/html/2501.15316#S4.T7)\. We can observe that each expert aligns syntax rather than semantic meanings, resembling the observations in\([31](https://arxiv.org/html/2501.15316#bib.bib54)\)\.

## 5Conclusion

In this paper, we propose a novel algorithm, ToMoE, for converting dense models into MoE models through dynamic structural pruning\. The resulting MoE models significantly outperform state\-of\-the\-art structural pruning methods while using similar or lower training costs compared to other learning\-based pruning methods\. Our findings reveal the presence of meaningful experts within the MLP layers of dense models, even without fine\-tuning the model weights\. ToMoE serves as a powerful tool for uncovering these experts within the original dense LLM\.

## References

- Ainslieet al\.\(2023\)J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. SanghaiGQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 4895–4901\.Cited by:[§A\.2](https://arxiv.org/html/2501.15316#A1.SS2.p2.1)\.
- Anagnostidiset al\.\(2023\)S\. Anagnostidis, D\. Pavllo, L\. Biggio, L\. Noci, A\. Lucchi, and T\. HofmannDynamic context pruning for efficient and interpretable autoregressive transformers\.InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),External Links:[Link](http://papers.nips.cc/paper%5C_files/paper/2023/hash/cdaac2a02c4fdcae77ba083b110efcc3-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Anthropic \(2023\)AnthropicClaude: a safer conversational ai assistant\.Note:[https://www\.anthropic\.com/index/2023/02/claude](https://www.anthropic.com/index/2023/02/claude)Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Anwaret al\.\(2017\)S\. Anwar, K\. Hwang, and W\. SungStructured pruning of deep convolutional neural networks\.ACM J\. Emerg\. Technol\. Comput\. Syst\.13\(3\),pp\. 32:1–32:18\.External Links:[Link](https://doi.org/10.1145/3005348),[Document](https://dx.doi.org/10.1145/3005348)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Ashkbooset al\.\(2024\)S\. Ashkboos, M\. L\. Croci, M\. G\. do Nascimento, T\. Hoefler, and J\. HensmanSliceGPT: compress large language models by deleting rows and columns\.InThe Twelfth International Conference on Learning Representations,Cited by:[Table 10](https://arxiv.org/html/2501.15316#A1.T10.5.1.10.2),[Table 10](https://arxiv.org/html/2501.15316#A1.T10.5.1.4.2),[Table 10](https://arxiv.org/html/2501.15316#A1.T10.5.1.7.2),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.10.2),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.4.2),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.12.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.5.1),[§1](https://arxiv.org/html/2501.15316#S1.p1.1),[§2](https://arxiv.org/html/2501.15316#S2.p1.1),[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.7.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.12.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.5.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Bengioet al\.\(2013\)Y\. Bengio, N\. Léonard, and A\. CourvilleEstimating or propagating gradients through stochastic neurons for conditional computation\.arXiv preprint arXiv:1308\.3432\.Cited by:[§A\.3](https://arxiv.org/html/2501.15316#A1.SS3.p1.2)\.
- Bisket al\.\(2020\)Y\. Bisk, R\. Zellers, J\. Gao, Y\. Choi,et al\.Piqa: reasoning about physical commonsense in natural language\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 7432–7439\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Brown \(2020\)T\. B\. BrownLanguage models are few\-shot learners\.arXiv preprint arXiv:2005\.14165\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Chaudhary \(2023\)S\. ChaudharyCode alpaca: an instruction\-following llama model for code generation\.GitHub\.Note:[https://github\.com/sahil280114/codealpaca](https://github.com/sahil280114/codealpaca)Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Chenet al\.\(2020\)Y\. Chen, X\. Dai, M\. Liu, D\. Chen, L\. Yuan, and Z\. LiuDynamic convolution: attention over convolution kernels\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11030–11039\.Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Clarket al\.\(2019\)C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. ToutanovaBoolQ: exploring the surprising difficulty of natural yes/no questions\.InNAACL,Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1.2)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.arXiv preprint arXiv:1803\.05457\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu,et al\.Deepseekmoe: towards ultimate expert specialization in mixture\-of\-experts language models\.arXiv preprint arXiv:2401\.06066\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p2.1),[§2](https://arxiv.org/html/2501.15316#S2.p3.1)\.
- Donget al\.\(2024a\)H\. Dong, X\. Yang, Z\. Zhang, Z\. Wang, Y\. Chi, and B\. ChenGet more with LESS: synthesizing recurrence with KV cache compression for efficient LLM inference\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,External Links:[Link](https://openreview.net/forum?id=uhHDhVKFMW)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Donget al\.\(2024b\)P\. Dong, L\. Li, Z\. Tang, X\. Liu, X\. Pan, Q\. Wang, and X\. ChuPruner\-zero: evolving symbolic pruning metric from scratch for large language models\.InForty\-first International Conference on Machine Learning,Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Donget al\.\(2022\)Q\. Dong, L\. Li, D\. Dai, C\. Zheng, J\. Ma, R\. Li, H\. Xia, J\. Xu, Z\. Wu, T\. Liu,et al\.A survey on in\-context learning\.arXiv preprint arXiv:2301\.00234\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Dubeyet al\.\(2024\)A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p1.1)\.
- Fanget al\.\(2023\)G\. Fang, X\. Ma, M\. Song, M\. B\. Mi, and X\. WangDepgraph: towards any structural pruning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16091–16101\.Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§B\.3](https://arxiv.org/html/2501.15316#A2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2501.15316#S3.SS4.p5.1),[§3\.4](https://arxiv.org/html/2501.15316#S3.SS4.p5.2)\.
- Frantar and Alistarh \(2023\)E\. Frantar and D\. AlistarhSparsegpt: massive language models can be accurately pruned in one\-shot\.InInternational Conference on Machine Learning,pp\. 10323–10337\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Frantaret al\.\(2022\)E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. AlistarhGptq: accurate post\-training quantization for generative pre\-trained transformers\.arXiv preprint arXiv:2210\.17323\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Ganjdaneshet al\.\(2024\)A\. Ganjdanesh, R\. Shirkavand, S\. Gao, and H\. HuangNot all prompts are made equal: prompt\-based pruning of text\-to\-image diffusion models\.arXiv preprint arXiv:2406\.12042\.Cited by:[§3\.1](https://arxiv.org/html/2501.15316#S3.SS1.p1.1)\.
- Gaoet al\.\(2021\)A framework for few\-shot language model evaluationExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.5371628),[Link](https://doi.org/10.5281/zenodo.5371628)Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Gaoet al\.\(2024\)S\. Gao, C\. Lin, T\. Hua, Z\. Tang, Y\. Shen, H\. Jin, and Y\. HsuDISP\-llm: dimension\-independent structural pruning for large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[Table 10](https://arxiv.org/html/2501.15316#A1.T10.5.1.12.1),[Table 10](https://arxiv.org/html/2501.15316#A1.T10.5.1.6.1),[Table 10](https://arxiv.org/html/2501.15316#A1.T10.5.1.9.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.12.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.13.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.17.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.18.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.21.2),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.23.2),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.6.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.7.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.18.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.19.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.7.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.8.1),[§3\.1](https://arxiv.org/html/2501.15316#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2501.15316#S3.SS1.p1.2),[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.9.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.17.2),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.21.2),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Gaoet al\.\(2019\)X\. Gao, Y\. Zhao, Ł\. Dudziak, R\. Mullins, and C\. XuDynamic channel pruning: feature boosting and suppression\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=BJxh2j0qYm)Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p3.1),[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Haet al\.\(2016\)D\. Ha, A\. Dai, and Q\. V\. LeHypernetworks\.arXiv preprint arXiv:1609\.09106\.Cited by:[§3\.1](https://arxiv.org/html/2501.15316#S3.SS1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.Proceedings of the International Conference on Learning Representations \(ICLR\)\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1.2)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[§3\.5](https://arxiv.org/html/2501.15316#S3.SS5.p2.1)\.
- Janget al\.\(2016\)E\. Jang, S\. Gu, and B\. PooleCategorical reparameterization with gumbel\-softmax\.arXiv preprint arXiv:1611\.01144\.Cited by:[§A\.3](https://arxiv.org/html/2501.15316#A1.SS3.p1.1),[§3\.2](https://arxiv.org/html/2501.15316#S3.SS2.p2.3)\.
- Javaheripiet al\.\(2023\)M\. Javaheripi, S\. Bubeck, M\. Abdin, J\. Aneja, S\. Bubeck, C\. C\. T\. Mendes, W\. Chen, A\. Del Giorno, R\. Eldan, S\. Gopi,et al\.Phi\-2: the surprising power of small language models\.Microsoft Research Blog\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p1.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, E\. B\. Hanna, F\. Bressand,et al\.Mixtral of experts\.arXiv preprint arXiv:2401\.04088\.Cited by:[§4\.4](https://arxiv.org/html/2501.15316#S4.SS4.p3.1)\.
- Johannes Welbl \(2017\)M\. G\. Johannes WelblCrowdsourcing multiple choice science questions\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1.2)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.External Links:[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Kenton and Toutanova \(2019\)J\. D\. M\. C\. Kenton and L\. K\. ToutanovaBert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of naacL\-HLT,Vol\.1,pp\. 2\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Kurticet al\.\(2022\)E\. Kurtic, D\. Campos, T\. Nguyen, E\. Frantar, M\. Kurtz, B\. Fineran, M\. Goin, and D\. AlistarhThe optimal BERT surgeon: scalable and accurate second\-order pruning for large language models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7\-11, 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),pp\. 4163–4181\.External Links:[Link](https://doi.org/10.18653/v1/2022.emnlp-main.279),[Document](https://dx.doi.org/10.18653/V1/2022.EMNLP-MAIN.279)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Leeet al\.\(2024a\)D\. Lee, J\. Lee, G\. Zhang, M\. Tiwari, and A\. MirhoseiniCATS: context\-aware thresholding for sparsity in large language models\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=v3w2a7EInO)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p2.1)\.
- Leeet al\.\(2024b\)J\. Lee, S\. Hwang, W\. Park, and M\. JiBreaking relu barrier: generalized moefication for dense pretrained models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 10097–10107\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p2.1),[§2](https://arxiv.org/html/2501.15316#S2.p3.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1.2),[Table 4](https://arxiv.org/html/2501.15316#S4.T4.5.1.10.2),[Table 4](https://arxiv.org/html/2501.15316#S4.T4.5.1.11.2)\.
- Lepikhinet al\.\(2021a\)D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. ChenGShard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p2.1),[§2](https://arxiv.org/html/2501.15316#S2.p3.1)\.
- Lepikhinet al\.\(2021b\)D\. Lepikhin, H\. Lee, Y\. Xu, D\. Chen, O\. Firat, Y\. Huang, M\. Krikun, N\. Shazeer, and Z\. Chen\{GS\}hard: scaling giant models with conditional computation and automatic sharding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qrwe7XHTmYb)Cited by:[§3\.4](https://arxiv.org/html/2501.15316#S3.SS4.p5.1)\.
- Liet al\.\(2017\)H\. Li, A\. Kadav, I\. Durdanovic, H\. Samet, and H\. P\. GrafPruning filters for efficient convnets\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJqFGTslg)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Linet al\.\(2024\)C\. Lin, S\. Gao, J\. S\. Smith, A\. Patel, S\. Tuli, Y\. Shen, H\. Jin, and Y\. HsuModegpt: modular decomposition for large language model compression\.arXiv preprint arXiv:2408\.09632\.Cited by:[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.14.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.8.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.10.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.14.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.15.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.9.1),[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.8.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.13.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.7.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.8.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulqa: measuring how models mimic human falsehoods\.InProceedings of the 60th annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 3214–3252\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1.2)\.
- Liuet al\.\(2023\)Z\. Liu, J\. Wang, T\. Dao, T\. Zhou, B\. Yuan, Z\. Song, A\. Shrivastava, C\. Zhang, Y\. Tian, C\. Ré, and B\. ChenDeja vu: contextual sparsity for efficient LLMs at inference time\.InProceedings of the 40th International Conference on Machine Learning,pp\. 22137–22176\.Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p2.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p2.1)\.
- Maet al\.\(2023\)X\. Ma, G\. Fang, and X\. WangLlm\-pruner: on the structural pruning of large language models\.Advances in neural information processing systems36,pp\. 21702–21720\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1),[§2](https://arxiv.org/html/2501.15316#S2.p1.1),[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.3.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Menet al\.\(2024\)X\. Men, M\. Xu, Q\. Zhang, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. ChenShortgpt: layers in large language models are more redundant than you expect\.arXiv preprint arXiv:2403\.03853\.Cited by:[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.11.2),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.4.2),[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.5.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.11.2),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.4.2),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Merityet al\.\(2016\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.External Links:1609\.07843Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Molchanovet al\.\(2019\)P\. Molchanov, A\. Mallya, S\. Tyree, I\. Frosio, and J\. KautzImportance estimation for neural network pruning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 11264–11272\.Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Muralidharanet al\.\(2024\)S\. Muralidharan, S\. T\. Sreenivas, R\. B\. Joshi, M\. Chochowski, M\. Patwary, M\. Shoeybi, B\. Catanzaro, J\. Kautz, and P\. MolchanovCompact language models via pruning and knowledge distillation\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§3\.5](https://arxiv.org/html/2501.15316#S3.SS5.p2.1)\.
- OpenAI \(2022\)OpenAIChatGPT: optimizing language models for dialogue\.Note:[https://openai\.com/chatgpt](https://openai.com/chatgpt)Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Paszkeet al\.\(2019\)A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, L\. Antiga,et al\.PyTorch: an imperative style, high\-performance deep learning library\.InAdvances in Neural Information Processing Systems,pp\. 8024–8035\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p2.1)\.
- Peiet al\.\(2025\)Z\. Pei, L\. Zou, H\. Zhen, X\. Yu, W\. Liu, S\. J\. Pan, M\. Yuan, and B\. YuCMoE: converting mixture\-of\-experts from dense to accelerate llm inference\.arXiv preprint arXiv:2502\.04416\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1.2),[Table 5](https://arxiv.org/html/2501.15316#S4.T5.6.1.6.1)\.
- Quet al\.\(2024\)X\. Qu, D\. Dong, X\. Hu, T\. Zhu, W\. Sun, and Y\. ChengLlama\-moe v2: exploring sparsity of llama from perspective of mixture\-of\-experts with post\-training\.arXiv preprint arXiv:2411\.15708\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1.2),[Table 5](https://arxiv.org/html/2501.15316#S4.T5.6.1.5.1),[Table 6](https://arxiv.org/html/2501.15316#S4.T6.6.1.4.2),[Table 6](https://arxiv.org/html/2501.15316#S4.T6.6.1.5.1)\.
- Radfordet al\.\(2018\)A\. Radford, K\. Narasimhan, T\. Salimans, and I\. SutskeverImproving language understanding by generative pre\-training\.Note:Technical report, OpenAIExternal Links:[Link](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)Cited by:[§3](https://arxiv.org/html/2501.15316#S3.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, D\. Amodei, J\. Clark, M\. Brundage, and I\. SutskeverLanguage models are unsupervised multitask learners\.Note:[https://cdn\.openai\.com/better\-language\-models/language\_models\_are\_unsupervised\_multitask\_learners\.pdf](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)OpenAI Technical ReportCited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Raffelet al\.\(2020\)C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. LiuExploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of machine learning research21\(140\),pp\. 1–67\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p1.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinogrande: an adversarial winograd schema challenge at scale\.Communications of the ACM64\(9\),pp\. 99–106\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p2.1),[§2](https://arxiv.org/html/2501.15316#S2.p3.1)\.
- Shazeer \(2019\)N\. ShazeerFast transformer decoding: one write\-head is all you need\.arXiv preprint arXiv:1911\.02150\.Cited by:[§A\.2](https://arxiv.org/html/2501.15316#A1.SS2.p2.1)\.
- Songet al\.\(2024\)J\. Song, K\. Oh, T\. Kim, H\. Kim, Y\. Kim,et al\.SLEB: streamlining llms through redundancy verification and elimination of transformer blocks\.InForty\-first International Conference on Machine Learning,Cited by:[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.6.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§A\.2](https://arxiv.org/html/2501.15316#A1.SS2.p1.1)\.
- Sunet al\.\(2024\)M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. KolterA simple and effective pruning approach for large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PxoFut3dWW)Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Taoriet al\.\(2023\)R\. Taori, I\. Gulrajani, T\. Zhang, Y\. Dubois, X\. Li, C\. Guestrin, P\. Liang, and T\. B\. HashimotoStanford alpaca: an instruction\-following llama model\.GitHub\.Note:[https://github\.com/tatsu\-lab/stanford\_alpaca](https://github.com/tatsu-lab/stanford_alpaca)Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Touvronet al\.\(2023a\)H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§3](https://arxiv.org/html/2501.15316#S3.p1.1)\.
- Touvronet al\.\(2023b\)H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§A\.2](https://arxiv.org/html/2501.15316#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p1.1)\.
- van der Ouderaaet al\.\(2024\)T\. F\. van der Ouderaa, M\. Nagel, M\. Van Baalen, and T\. BlankevoortThe llm surgeon\.InThe Twelfth International Conference on Learning Representations,Cited by:[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.11.1),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.16.2),[Table 9](https://arxiv.org/html/2501.15316#A1.T9.5.1.5.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.13.1),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.17.2),[Table 12](https://arxiv.org/html/2501.15316#A2.T12.5.1.6.1),[Table 1](https://arxiv.org/html/2501.15316#S3.T1.5.1.4.1),[Table 3](https://arxiv.org/html/2501.15316#S3.T3.5.1.6.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1)\.
- Wanget al\.\(2024\)H\. Wang, L\. Xie, H\. Zhao, C\. Zhang, H\. Qian, J\. C\. Lui,et al\.D\-llm: a token adaptive computing resource allocation strategy for large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p1.1)\.
- Wolfet al\.\(2020\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. RushTransformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Link](https://www.aclweb.org/anthology/2020.emnlp-demos.6)Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p2.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei,et al\.Qwen2\. 5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p1.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.Cited by:[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p3.1)\.
- Zhanget al\.\(2022\)Z\. Zhang, Y\. Lin, Z\. Liu, P\. Li, M\. Sun, and J\. ZhouMoEfication: transformer feed\-forward layers are mixtures of experts\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 877–890\.Cited by:[§1](https://arxiv.org/html/2501.15316#S1.p2.1),[§2](https://arxiv.org/html/2501.15316#S2.p3.1)\.
- Zhenget al\.\(2024\)H\. Zheng, X\. Bai, X\. Liu, Z\. Mao, B\. Chen, F\. Lai, and A\. PrakashLearn to be efficient: build structured sparsity in large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=iSfCWhvEGA)Cited by:[§2](https://arxiv.org/html/2501.15316#S2.p2.1)\.
- Zhuet al\.\(2024\)T\. Zhu, X\. Qu, D\. Dong, J\. Ruan, J\. Tong, C\. He, and Y\. ChengLlama\-moe: building mixture\-of\-experts from llama with continual pre\-training\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15913–15923\.Cited by:[Table 15](https://arxiv.org/html/2501.15316#A4.T15.3.1.3.1),[§1](https://arxiv.org/html/2501.15316#S1.p2.1),[§2](https://arxiv.org/html/2501.15316#S2.p3.1),[§4\.1](https://arxiv.org/html/2501.15316#S4.SS1.p4.1.2),[Table 4](https://arxiv.org/html/2501.15316#S4.T4.5.1.4.2),[Table 4](https://arxiv.org/html/2501.15316#S4.T4.5.1.6.1),[Table 5](https://arxiv.org/html/2501.15316#S4.T5.6.1.4.2)\.

## Appendix ADetails of trainable modules

### A\.1Module Configurations

We present the details of trainable modules in Tab\.[8](https://arxiv.org/html/2501.15316#A1.T8)\. In short, we project the input tokens to a low\-dimensional space and add them to the output of the HyperNetwork\. The inputszzto the HyperNetwork are fixed random vectors of sizeN×32N\\times 32sampled from a Normal Distribution\. Except for the HyperNetwork, other individual trainable modules are created for each MHA and MLP layer\. If we haveLLblocks, then we will haveLLProjEMHA\\text\{Proj\}\_\{\\text\{E\}\}^\{\\text\{\\tiny MHA\}\},LLProjDMHA\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MHA\}\}with output size ofdH\\frac\{d\}\{H\},LLProjDMHA\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MHA\}\}with output size ofd2​H\\frac\{d\}\{2H\}LL,ProjDMLP\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}andLLRouterlayers\. Notations ofdd,HH,dmidd\_\{\\text\{mid\}\}, andNNare already defined in Sec\.[3](https://arxiv.org/html/2501.15316#S3)\.

Table 8:Detailed configuration of trainable modules\.After we complete the training of ToMoE, we do not have to preserve all modules\. The embeddings from the HyperNetwork will be saved, so the HyperNetwork can be removed without impacting the model\.ProjDMLP\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}brings most additional parameters, fortunately, it can also be removed\. After the training of ToMoE,𝐄\\mathbf\{E\}andProjDMLP\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}can be used to directly generate experts:

𝐬𝐞=ST\-GSig​\(ProjDMLP​\(𝐄\)\),~\\mathbf\{s\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}=\\text\{ST\-GSig\}\(\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}\(\\mathbf\{E\}\)\),\(11\)whereST\-GSigagain is the Straight\-Through Gumbel\-Sigmoid function\.𝐬𝐞∈\{0,1\}N×dmid\\mathbf\{s\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}\\in\\\{0,1\\\}^\{N\\times d\_\{\\text\{mid\}\}\}is the resulting binary vectors to select experts from the dense model\. Once𝐬𝐞\\mathbf\{s\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}is generated, it can be reused, and thus we no longer needProjDMLP\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}\. Let𝐒𝐞i=Diag\(𝐬𝐞i\),i=1,⋯,N\\mathbf\{S\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}=\\text\{Diag\}\(\\mathbf\{s\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}\),\\ i=1,\\cdots,N\. Similarly, we use𝐬^𝐞i∈ℜdmid×dmid′\\hat\{\\mathbf\{s\}\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}\\in\\Re^\{d\_\{\\text\{mid\}\}\\times d\_\{\\text\{mid\}\}^\{\\prime\}\}to represent the actual column or row selection matrix by removing zero columns or rows, wheredmid′<dmidd\_\{\\text\{mid\}\}^\{\\prime\}<d\_\{\\text\{mid\}\}and it is the width of each expert\. Theiith expert can be represented as:

fMLPi​\(𝐗\)=σ⁡\(𝐗𝐖G​𝐒^𝐞i\)⊙\(𝐗𝐖U​𝐒^𝐞i\)​𝐒^𝐞i​𝐖D\.f\_\{\\text\{MLP\}\}^\{i\}\(\\mathbf\{X\}\)=\\sigma\(\\mathbf\{X\}\\mathbf\{W\}\_\{G\}\\hat\{\\mathbf\{S\}\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}\)\\odot\(\\mathbf\{X\}\\mathbf\{W\}\_\{U\}\\hat\{\\mathbf\{S\}\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}\)\\hat\{\\mathbf\{S\}\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}\\mathbf\{W\}\_\{D\}\.\\\\\(12\)After ToMoE, given the result of the routing function𝐆=ST\-GSmax​\(Router​\(𝐗\)\)\\mathbf\{G\}=\{\\text\{ST\-GSmax\}\}\(\\text\{Router\}\(\\mathbf\{X\}\)\), the MLP calculation with MoE can be written as:

𝐘t=𝐆t,i​fMLPi​\(𝐗t\),\\mathbf\{Y\}\_\{t\}=\\mathbf\{G\}\_\{t,i\}f\_\{\\text\{MLP\}\}^\{i\}\(\\mathbf\{X\}\_\{t\}\),~\(13\)where𝐗t\\mathbf\{X\}\_\{t\}is the feature map ofttth token, andiirepresents the index where𝐆t,i=1\\mathbf\{G\}\_\{t,i\}=1\. Note that Eq\.[13](https://arxiv.org/html/2501.15316#A1.E13)is still differentiable with respect to the parameters of theRouter\.

Another question is how many parameters we need after introducing Top\-K routing for MHA layers and Top\-1 routing for MLP layers\. Analytically, the additional parameters can be calculated byL×d×128\+L×128×\(dH\)\+L×d×NL\\times d\\times 128\+L\\times 128\\times\(\\frac\{d\}\{H\}\)\+L\\times d\\times N\. Let’s use LLaMA\-2 7B as an example,L=32L=32,d=4096d=4096,N=8N=8,H=32H=32, the additional parameters are1×32×4096×128\+32×128×\(128\)\+32×4096×8=0\.01841\\times 32\\times 4096\\times 128\+32\\times 128\\times\(128\)\+32\\times 4096\\times 8=0\.0184B\. This is equivalent to0\.27%0\.27\\%of the total parameters of the LLaMA\-2 7B model, and thus, the additional parameter is not significant compared to the original number of parameters\.

### A\.2Head Dimension Pruning vs\. RoPE

Rotary Position Embedding \(RoPE\)\([61](https://arxiv.org/html/2501.15316#bib.bib55)\)is a popular positional encoding method, and it is regularly used in LLMs like LLaMA\([65](https://arxiv.org/html/2501.15316#bib.bib22)\)\. RoPE divides thedH\\frac\{d\}\{H\}dimensional space intod2​H\\frac\{d\}\{2H\}sub\-spaces, and they are applied on query and key\. This means that if we want to perform head dimension pruning for query and key matrices, we need to follow the sub\-spaces resulting from RoPE and make these two sub\-spaces share the same pruning mask𝐬0\\mathbf\{s\}\_\{0\}, and the final pruning mask for query and key is𝐬0′=\[𝐬0\[1:d2​H\],𝐬0\[1:d2​H\]\]\\mathbf\{s\}\_\{0\}^\{\\prime\}=\[\{\\mathbf\{s\}\_\{0\}\}\_\{\[1:\\frac\{d\}\{2H\}\]\},\{\\mathbf\{s\}\_\{0\}\}\_\{\[1:\\frac\{d\}\{2H\}\]\}\], and clearly the size of𝐬0\[1:d2​H\]\{\\mathbf\{s\}\_\{0\}\}\_\{\[1:\\frac\{d\}\{2H\}\]\}isd2​H\\frac\{d\}\{2H\}\. In short, we simply select the first half of elements from𝐬0\\mathbf\{s\}\_\{0\}and repeat it twice to make the final pruning decision\. We also found that applying dynamic pruning for query and key matrices along the head dimension is difficult and unreasonable since different tokens may have different positions after pruning\. It becomes a problem when calculating the inner product between the query and key matrices given different tokens\.

By applying head dimension pruning, our method also does not need to be modified when facing different attention mechanisms like GQA \(Grouped\-Query Attention\)\([1](https://arxiv.org/html/2501.15316#bib.bib57)\)and MQA \(Multi\-Query Attention\)\([59](https://arxiv.org/html/2501.15316#bib.bib58)\)\.

Table 9:Zero\-shot task performance of compressed LLaMA\-2 13B and Qwen\-2\.5 14B\.
### A\.3Details of Gumbel\-Softmax and Gumbel\-Sigmoid

The Gumbel\-Softmax function\([29](https://arxiv.org/html/2501.15316#bib.bib7)\)allows for differentiable sampling from a categorical distribution\. Given logits𝐱\\mathbf\{x\}, the Gumbel\-Softmax sample𝐲\\mathbf\{y\}is computed as:

𝐲=softmax​\(𝐱\+𝐠τ\),\\mathbf\{y\}=\\text\{softmax\}\\left\(\\frac\{\\mathbf\{x\}\+\\mathbf\{g\}\}\{\\tau\}\\right\),where each element of𝐠\\mathbf\{g\}is drawn fromGumbel​\(0,1\)\\text\{Gumbel\}\(0,1\), andτ\\tauis the temperature parameter that controls the smoothness of the distribution\. Combining Gumbel\-Softmax with the Straight\-Through gradient Estimator\([6](https://arxiv.org/html/2501.15316#bib.bib16)\), we have the following equation:

ST\-GSmax​\(𝐱\)=one\-hot​\(arg⁡maxi∈D​\[xi\+giτ\]\)~\\text\{ST\-GSmax\}\(\\mathbf\{x\}\)=\\text\{one\-hot\}\\left\(\\arg\\max\_\{i\\in D\}\\left\[\\frac\{x\_\{i\}\+g\_\{i\}\}\{\\tau\}\\right\]\\right\)\(14\)whereD=\{1,2,⋯,N\}D=\\\{1,2,\\cdots,N\\\},NNagain is the number of experts in our setting, andone\-hotwill assign11corresponding to the position of the maximum value in𝐱\+𝐠τ\\frac\{\\mathbf\{x\}\+\\mathbf\{g\}\}\{\\tau\}and assign00to other positions\.

The Gumbel\-Sigmoid function is a special case of the Gumbel\-Softmax function, designed for binary distributions\. Given logits𝐱\\mathbf\{x\}, the Gumbel\-Sigmoid sample𝐲\\mathbf\{y\}is computed as:

𝐲=sigmoid​\(𝐱\+𝐠τ\),\\mathbf\{y\}=\\text\{sigmoid\}\\left\(\\frac\{\\mathbf\{x\}\+\\mathbf\{g\}\}\{\\tau\}\\right\),where𝐠\\mathbf\{g\}is sampled fromGumbel​\(0,1\)\\text\{Gumbel\}\(0,1\)andτ\\tauagain is the temperature parameter\. Combining with the Straight\-Through gradient Estimator, we have the following equation:

ST\-GSig​\(𝐱\)=round​\(sigmoid​\(𝐱\+𝐠\+bτ\)\),~\\text\{ST\-GSig\}\(\\mathbf\{x\}\)=\\text\{round\}\(\\text\{sigmoid\}\\left\(\\frac\{\\mathbf\{x\}\+\\mathbf\{g\}\+b\}\{\\tau\}\\right\)\),\(15\)wherebbis a constant bias in our implementation and it ensures that all experts start from the whole model,round​\(⋅\)\\text\{round\}\(\\cdot\)will round the input values to the nearest integer, and in our case, it rounds inputs to00or11\. For all experiments, we setb=3\.0b=3\.0in Eq\.[15](https://arxiv.org/html/2501.15316#A1.E15), and we setτ=0\.4\\tau=0\.4for Eq\.[14](https://arxiv.org/html/2501.15316#A1.E14)and Eq\.[15](https://arxiv.org/html/2501.15316#A1.E15)\.

Table 10:Zero\-shot task performance of the compressed Phi\-2\.Listing 1:Pseudo\-code for self\-knowledge distillation\.1withtorch\.no\_grad\(\):

2

3helper\.set\_module\_status\(model,False\)

4

5

6teacher\_output=model\(inputs\)

7teacher\_logits=teacher\_output\.logits

8

9

10helper\.set\_module\_status\(model,True\)

11

12

13model\_output=model\(inputs\)

14logits=model\_output\.logits

## Appendix BMore Details of the Loss Design

### B\.1Implementation of the Self\-Knowledge Distillation

During the ToMoE learning process, we freeze the parameters of the original model\. This approach offers the additional benefit of enabling self\-knowledge distillation without the need to load an extra model\. In Lst\.[1](https://arxiv.org/html/2501.15316#LST1), we present the pseudo\-code for the self\-knowledge distillation process\. In summary, we first disable the trainable modules associated with ToMoE and compute the output logits from the original model\. Next, we re\-enable the trainable modules for ToMoE and perform a regular forward pass\. The logits from the original model are then used to guide the learning of ToMoE\.

Table 11:ToMoE Visualization of LLaMA\-2 7B with 50% active parametersTable 12:Zero\-shot task performance of compressed LLaMA\-2 7B with more settings\.
### B\.2Efficient Implementation ofℛ𝐮\\mathcal\{R\}\_\{\\mathbf\{u\}\}

Recall from Eq\.[6](https://arxiv.org/html/2501.15316#S3.E6)that the union regularization for MLP and MHA layers is defined as:

ℛ𝐮=⋃i=1T𝐬i=1−∏i=1T\(1−𝐬i\)\.\\mathcal\{R\}\_\{\\mathbf\{u\}\}=\\bigcup\_\{i=1\}^\{T\}\\mathbf\{s\}\_\{i\}=1\-\\prod\_\{i=1\}^\{T\}\(1\-\\mathbf\{s\}\_\{i\}\)\.For MLP layers, this equation incurs high computational costs since𝐬∈ℝT×dmid\\mathbf\{s\}\\in\\mathbb\{R\}^\{T\\times d\_\{\\text\{mid\}\}\}, whereas for MHA layers, the cost is significantly lower becausedH≪dmid\\frac\{d\}\{H\}\\ll d\_\{\\text\{mid\}\}\. To simplify Eq\.[6](https://arxiv.org/html/2501.15316#S3.E6), note that all𝐬i\\mathbf\{s\}\_\{i\}\(i=1,…,Ti=1,\\ldots,T\) are derived fromNNexperts\. Using embeddings from the hypernetwork, we calculate the configuration ofNNexperts as:

𝐬𝐞=ST\-GSig​\(ProjDMLP​\(𝐄\)\),\\mathbf\{s\}\_\{\\mathbf\{e\}\}=\\text\{ST\-GSig\}\(\\text\{Proj\}\_\{\\text\{D\}\}^\{\\text\{\\tiny MLP\}\}\(\\mathbf\{E\}\)\),and substitute𝐬𝐞\\mathbf\{s\}\_\{\\mathbf\{e\}\}into Eq\.[6](https://arxiv.org/html/2501.15316#S3.E6):

ℛ𝐮MLP=⋃i=1N𝐬𝐞i=1−∏i=1N\(1−𝐬𝐞i\)\.~\\mathcal\{R\}\_\{\\mathbf\{u\}\}^\{\\text\{\\tiny MLP\}\}=\\bigcup\_\{i=1\}^\{N\}\\mathbf\{s\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}=1\-\\prod\_\{i=1\}^\{N\}\(1\-\\mathbf\{s\}\_\{\\scriptstyle\{\\mathbf\{e\}\}\}^\{i\}\)\.\(16\)This reduces computation by a factor ofTN\\frac\{T\}\{N\}\. For example, in LLaMA\-2, the computational cost is reduced by20488=256\\frac\{2048\}\{8\}=256times\.

Table 13:Ablation study on design choices of ToMoE and the impact of temperatureτ\\tauon performance\.
### B\.3Equivalence of MoE and pseudo\-MoE

One major challenge when training MoE models is maintaining an appropriate expert capacity, defined as the number of tokens each expert processes\([19](https://arxiv.org/html/2501.15316#bib.bib44)\)\. This is typically addressed using a load balancing loss\. Without this loss, some experts may become overloaded while others remain underutilized, leading to bottlenecks where a few experts dominate the computation\.

Although ToMoE also requires load balancing loss, the potential overhead introduced by load balancing is mitigated by the pseudo\-MoE approach after ToMoE\. After applying ToMoE, the resulting model can be trained using pseudo\-MoE, which resembles the training of a dense model\. This is straightforward to implement as follows:

fMLP​\(𝐗\)=σ⁡\(𝐗𝐖G\)​𝐒⊙\(𝐗𝐖U​𝐒\)​𝐒𝐖D,f\_\{\\text\{MLP\}\}\(\\mathbf\{X\}\)=\\sigma\(\\mathbf\{X\}\\mathbf\{W\}\_\{G\}\)\{\\mathbf\{S\}\}\\odot\(\\mathbf\{X\}\\mathbf\{W\}\_\{U\}\{\\mathbf\{S\}\}\)\{\\mathbf\{S\}\}\\mathbf\{W\}\_\{D\},\(17\)where𝐒i\\mathbf\{S\}\_\{i\}in𝐒\\mathbf\{S\}represents the routed expert from𝐒e\\mathbf\{S\}\_\{e\}in Eq\.[11](https://arxiv.org/html/2501.15316#A1.E11), as determined by the router\. The pseudo\-MoE is useful when the active number of parameters is relatively large\. In such cases, pseudo\-MoE training can be more time\-efficient than conventional MoE training\.

## Appendix CMore Implementation Details

During training the modules of ToMoE, we use AdamW optimizer to optimize it with a constant learning rate10−310^\{\-3\}and weight decay0\.050\.05\. For different models, we always set the mini\-batchsize to11on each GPU\. For LLaMA\-2 7B, and Qwen\-2\.5 7B models, we use 2 NVIDIA A100 GPUs, For LLaMA\-3 8B, we use 3 NVIDIA A100 GPUs\. For LLaMA\-2 13B and Qwen\-2\.5 14B models, we use 4 NVIDIA A100 GPUs\. For all the rest models, we use 1 NVIDIA A100 GPU\. We setp=\{0\.6,0\.5,0\.4,0\.3\}p=\\\{0\.6,0\.5,0\.4,0\.3\\\}when the ratios of active parameters equals to\{40%,50%,60%,70%\}\\\{40\\%,50\\%,60\\%,70\\%\\\}\.

For the Alpaca dataset111https://huggingface\.co/datasets/tatsu\-lab/alpaca, we use the ‘text’ column within the dataset, which combines the columns of ‘instruction’ and ‘output’\. For the Code Alpaca dataset222https://github\.com/sahil280114/codealpaca, we combine the ‘instruction’, ‘input’, and ‘output’ columns as one training sample\.

Table 14:ToMoE Visualization of LLaMA\-2 7B with 50% active parameters \(continued\)\.
## Appendix DMore Experimental Results

![Refer to caption](https://arxiv.org/html/2501.15316v2/llama2_arch.png)Figure 7:Model width after ToMoE for the LLaMA\-2 7B model when the number of active parameters equals 50%\.![Refer to caption](https://arxiv.org/html/2501.15316v2/experts_size_std.png)Figure 8:Box plot of widths across different experts for the LLaMA\-2 7B model when the number of active parameters equals 50%\.![Refer to caption](https://arxiv.org/html/2501.15316v2/expert_similarity.png)Figure 9:The similarity of different experts from different layers of ToMoE of the LLaMA\-2 7B model\.Table 15:Inference throughput \(tokens per second\) under different mini\-batchsizes\.In Tab\.[9](https://arxiv.org/html/2501.15316#A1.T9)and Tab\.[10](https://arxiv.org/html/2501.15316#A1.T10), we present the zero\-shot performance of various methods on LLaMA\-2 13B, Qwen\-2\.5 14B, and Phi\-2 models\. From Tab\.[9](https://arxiv.org/html/2501.15316#A1.T9), it is evident that ToMoE consistently outperforms other methods\. Compared to 7B or 8B models, the performance gap between our method and other approaches is smaller, which also holds for the differences between baseline methods\. This is likely due to the larger model sizes\. Table[12](https://arxiv.org/html/2501.15316#A2.T12)presents the zero\-shot performance of the LLaMA\-2 7B model across more baselines and active parameters\. As shown in the table, ToMoE consistently achieves significantly better performance than all competing methods\.

On the LLaMA\-2 13B model, ToMoE surpasses structural pruning methods even with smaller compression rates\. For instance, ToMoE with 50% active parameters performs better than MoDeGPT and LLM Surgeon with a 40% compression rate\. The performance gap becomes even more obvious when comparing methods with the same number of active parameters\. Similarly, from Tab\.[10](https://arxiv.org/html/2501.15316#A1.T10), ToMoE demonstrates superior performance compared to SliceGPT and DISP\-LLM\. Specifically, ToMoE with 70% active parameters achieves better results than all three compression levels of SliceGPT and DISP\-LLM\.

Figure 10:MMLU accuracy vs\. active parameters\.In Tab\.[15](https://arxiv.org/html/2501.15316#A4.T15), we report the inference throughput \(measured in tokens per second\) of different models under varying batch sizes\. Compared to the dense LLaMA\-2 7B baseline, both converted MoE models achieve higher throughput due to reduced active parameters\. Our resulting model has a similar throughput to LLaMA\-MoE when the batch size is large enough\.

In Fig\.[10](https://arxiv.org/html/2501.15316#A4.F10), we present the accuracy–parameter trade\-off on the more challenging MMLU dataset with the LLaMA\-3 8B model\. The results show that our method can still provide meaningful results when activating only 50% of the model parameters\.

In Fig\.[7](https://arxiv.org/html/2501.15316#A4.F7), we illustrate the width of ToMoE for the LLaMA\-2 7B model\. A highly non\-uniform pattern emerges in the allocation of active parameters, indicating that ToMoE can effectively determine the ideal distribution of active parameters, even when the allocation is highly non\-uniform\.

In Tab\.[13](https://arxiv.org/html/2501.15316#A2.T13), we present an ablation study to analyze several design choices in ToMoE, focusing on architectural components and the effect of the temperature parameterτ\\tau\. Although ToMoE applies contextual sparsity to the value and output \(VO\) projections within the attention layer, their contribution to overall performance is relatively minor due to the small head dimension \(128, in the case of LLaMA\-2 7B\)\. To validate this, we disable dynamic attention sparsity and instead apply only static structural pruning to the attention layer \(denoted as "w/o VO routing"\)\. This leads to only a modest performance drop of about 1% using 50% active parameters\.

We also examine the role of global expert embeddings with GRU in conveying cross\-layer architectural information\. Specifically, we compare the default global expert embedding with a local\-only variant \("local emb"\), where expert embeddings are used only in MLP layers and removed from attention layers\. Results show a slight decrease in performance, suggesting that global expert embeddings contribute to better coordination across layers\.

Additionally, we evaluate the sensitivity of ToMoE to the temperatureτ\\tauin the routing mechanism\. The results withτ∈\{0\.3,0\.4,0\.5\}\\tau\\in\\\{0\.3,0\.4,0\.5\\\}show that performance remains relatively stable, indicating robustness to the choice ofτ\\tauwithin a reasonable range\.

Finally, we explore head pruning in the early stages of ToMoE development\. However, this approach yielded significantly lower performance\. This may be due to the distortion of attention feature maps when heads are removed, which makes it more difficult to train effective MLP experts\.

These results highlight the effectiveness of ToMoE in preserving the capacity of LLMs compared to structural pruning methods\. Additionally, they demonstrate that ToMoE performs robustly across various scales and types of LLMs\.

## Appendix EVisualization of Experts

In this section, we analyze the properties of the experts produced by our method\. Tab\.[11](https://arxiv.org/html/2501.15316#A2.T11)and Tab\.[14](https://arxiv.org/html/2501.15316#A3.T14)present visualizations of the routed tokens among experts across different layers and input texts\.

In Tab\.[11](https://arxiv.org/html/2501.15316#A2.T11), we observe no distinct patterns in the allocation of tokens to specific experts, which aligns with our observations in Tab\.[7](https://arxiv.org/html/2501.15316#S4.T7)\. An interesting trend emerges when comparing layers: the first layer exhibits a more diverse token distribution, while subsequent layers prefer to assign continuous tokens to the same expert\. Tab\.[14](https://arxiv.org/html/2501.15316#A3.T14)focuses on inputs related to a math problem\. Unlike the visualization in Tab\.[11](https://arxiv.org/html/2501.15316#A2.T11), the MoE routing for the math problem reveals clearer semantic patterns\. For instance,Expert 2in MLP 16 is predominantly activated by numbers and mathematical notations, and a similar behavior is observed forExpert 8in MLP 32\. This suggests that the experts in ToMoE may encode more distinct semantic meanings compared to MoE models trained from scratch\. Further investigation is required to fully understand the precise semantic roles of ToMoE experts\.

In Fig\.[8](https://arxiv.org/html/2501.15316#A4.F8), we present a box plot showing the expert sizes across different layers\. The figure reveals that the maximum and minimum expert sizes are closely aligned across layers\. This outcome is a direct result of applying constraints from Eq\.[8](https://arxiv.org/html/2501.15316#S3.E8)and Eq\.[9](https://arxiv.org/html/2501.15316#S3.E9), as well as only penalizing the largest expert in Eq\.[8](https://arxiv.org/html/2501.15316#S3.E8)\. During training, minimizing the task loss \(self\-knowledge distillation loss\) encourages experts to grow in size\. Consequently, smaller experts do not remain small due to the task loss and they are not penalized by the parameter regularization loss\. This iterative process leads to all experts eventually converging to similar sizes\. After completing the ToMoE training process, we adjust the width of all experts tomatch the maximum size among them\. This ensures uniform computational cost across all experts\.

In Fig\.[9](https://arxiv.org/html/2501.15316#A4.F9), we present a visualization of the similarity between different experts across all layers of the LLaMA\-2 7B model\. Within the same layer, experts generally exhibit comparable similarity values, indicating that while the experts share the same size, their weights remain distinct\. Notably, certain layers, such as layer 1 and layer 30, show lower similarity values\. This observation aligns with expectations, as the expert sizes in these layers are smaller\.

## Appendix FDesign Choice for the MHA

Ideally, to achieve maximum flexibility, one might consider applying dynamic pruning to all projection matrices in the MHA layer, including the query \(WQW\_\{Q\}\), key \(WKW\_\{K\}\), value \(WVW\_\{V\}\), and output \(WOW\_\{O\}\) matrices\. However, there is a fundamental limitation when attempting to apply dynamic pruning along the head dimension for the query and key matrices\.

Suppose we generate pruning masksSt∈\{0,1\}dS\_\{t\}\\in\\\{0,1\\\}^\{d\}at each time stepttbased on the inputXt∈ℝ1×dX\_\{t\}\\in\\mathbb\{R\}^\{1\\times d\}, and consider two distinct time steps,aaandbb\. For theii\-th attention head, the attention score between queries and keys is influenced by the pruning masks\. Specifically, the effective attention score between theaa\-th query and theaa\-th key is given by:

e⁡\(Xa​WQ,i\)​Sa​Sa⊤​e​\(Xa​WK,i\)⊤,e\(X\_\{a\}W\_\{Q,i\}\)S\_\{a\}S\_\{a\}^\{\\top\}e\(X\_\{a\}W\_\{K,i\}\)^\{\\top\},while the attention score between theaa\-th query and thebb\-th key is:

e⁡\(Xa​WQ,i\)​Sa​Sb⊤​e​\(Xb​WK,i\)⊤\.e\(X\_\{a\}W\_\{Q,i\}\)S\_\{a\}S\_\{b\}^\{\\top\}e\(X\_\{b\}W\_\{K,i\}\)^\{\\top\}\.
The*effective width*—that is, the dimensionality over which attention is computed—between theaa\-th query and theaa\-th key is‖Sa​Sa⊤‖0=∑Sa=K\\\|S\_\{a\}S\_\{a\}^\{\\top\}\\\|\_\{0\}=\\sum S\_\{a\}=K, assuming the mask has exactlyKKactive elements\. However, for cross\-position pairs like\(a,b\)\(a,b\), the effective width becomes‖Sa​Sb⊤‖0=‖Sa⊙Sb‖0≤min⁡\(∑Sa,∑Sb\)=K\\\|S\_\{a\}S\_\{b\}^\{\\top\}\\\|\_\{0\}=\\\|S\_\{a\}\\odot S\_\{b\}\\\|\_\{0\}\\leq\\min\(\\sum S\_\{a\},\\sum S\_\{b\}\)=K\. The equality holds only whenSa=SbS\_\{a\}=S\_\{b\}, which generally does not hold for arbitrarya≠ba\\neq b\.

This observation implies that dynamically pruned query and key matrices fail to fully utilize the allocated capacityKKunless the pruning masks are identical across all positions\. Moreover, the variability of the effective width across different query\-key pairs introduces instability and inconsistent capacity utilization, making this approach less favorable compared to static pruning for the query and key matrices\.

In contrast, dynamic pruning does not encounter this issue when applied to the value and output matrices, as these are not involved in pairwise comparisons like the query\-key dot products\. Therefore, we adopt dynamic pruning only for the value and output projections, while keeping the query and key projections pruned statically to maintain stable and full\-capacity attention computation\.