Accelerating Dense LLMs via L0-regularized Mixture-of-Experts

arXiv cs.AI Papers

Summary

This paper proposes L0-MoE, a lightweight Mixture-of-Experts approach using L0-regularization to accelerate dense Large Language Models with up to 2.5x speedup while maintaining competitive performance.

arXiv:2609.21672v1 Announce Type: new Abstract: Large language models (LLMs) achieve strong performance but suffer from slow and costly inference. Existing acceleration methods often lead to noticeable performance degradation, while Mixture-of-Experts (MoE) models require extensive computational resources. In this paper, we propose L0-MoE, a lightweight MoE approach using L0-regularization to accelerate dense LLMs nearly without performance loss. Our method introduces a cluster confusion matrix for domain-aware dataset curation and applies dynamic batching for efficient training. Experiments show that L0-MoE achieves up to 2.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines.
Original Article
View Cached Full Text

Cached at: 09/21/26, 09:30 AM

# Accelerating Dense LLMs via L0-regularized Mixture-of-Experts
Source: [https://arxiv.org/html/2609.21672](https://arxiv.org/html/2609.21672)
Jiudong YangAffiliation:FuTu AI, Shenzhen, ChinaEmail:[simonyang@futunn\.com](mailto:)Zhaowen TaoAffiliation:YZW, Chengdu, ChinaEmail:[taozhaowen@gmail\.com](mailto:)Meng Chen††thanks:˜ Corresponding author\.Affiliation:Wise AI, Melbourne, AustraliaEmail:[chenmengdx@gmail\.com](mailto:)

###### Abstract

Large language models \(LLMs\) achieve strong performance but suffer from slow and costly inference\. Existing acceleration methods often lead to noticeable performance degradation, while Mixture\-of\-Experts \(MoE\) models require extensive computational resources\. In this paper, we propose L0\-MoE, a lightweight MoE approach using L0\-regularization to accelerate dense LLMs nearly without performance loss\. Our method introduces a cluster confusion matrix for domain\-aware dataset curation and applies dynamic batching for efficient training\. Experiments show that L0\-MoE achieves up to 2\.5x speedup over dense models while maintaining competitive performance, outperforming existing LLM acceleration baselines\. Code is now released111https://github\.com/zhangzhenyu13/L0\-MOE\.

## 1Introduction

Large language models \(LLMs\) have demonstrated remarkable intelligence across various tasks[OpenAI et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib22);[Gemini\-Team et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib23);[Dubey et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib13);[Jiang et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib15);[DeepSeek\-AI et al\. \(2025\)](https://arxiv.org/html/2609.21672#bib.bib24);[Yang et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib14), including question answering, mathematics, coding, and content generation\. A key insight into their success is the parameter scaling law[Kaplan et al\. \(2020\)](https://arxiv.org/html/2609.21672#bib.bib25), which suggests that increasing model size enhances performance across diverse tasks, potentially advancing artificial general intelligence \(AGI\)[Bubeck et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib26)\. However, larger LLMs incur high inference costs, leading to slower generation speeds and increased computational expenses\. Thus, optimizing LLM inference efficiency has become a critical challenge for both academia and industry\.

Various approaches have been proposed to accelerate LLM inference, which can be categorized into three main techniques: \(1\)Quantization, including GPTQ[Frantar et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib8), SmoothQuant[Xiao et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib11), AWQ[Lin et al\. \(2024b\)](https://arxiv.org/html/2609.21672#bib.bib12)and DuQuant[Lin et al\. \(2024a\)](https://arxiv.org/html/2609.21672#bib.bib39), reduces precision by converting weights and activations from floating\-point to lower\-bit integer formats, significantly improving efficiency\. \(2\)Model pruning, such as LLM\-Pruner[Ma et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib2)and LLM\-Shearing[Xia et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib1), removes redundant parameters based on predefined criteria to compress models and accelerate inference\. \(3\)Knowledge distillation[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib17);[Feng et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib16), like reverse\-KD[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib17)and Chain\-of\-Thought \(CoT\) Distillation[Feng et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib16), transfers knowledge from large LLMs to smaller ones using distillation techniques[Hinton et al\. \(2015\)](https://arxiv.org/html/2609.21672#bib.bib27), reducing computational demands\. While these methods achieve substantial speedup, they often come at the cost of performance degradation, posing challenges for real\-world deployment\.

![Refer to caption](https://arxiv.org/html/2609.21672v1/architecture.png)Figure 1:Overview of the L0\-MoE Architecture, which includes three main stages: \(1\) cluster confusion matrix based sampling, \(2\) expert formation using L0 regularization, and \(3\) dynamic batching for MoE training\. The figure above illustrates the process of building an L0\-MoE with four experts overnniterations of dataset sampling\.Recently, sparsely gated Mixture\-of\-Experts \(MoE\) models[Cai et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib20), particularly in transformer\-based large language models, have significantly improved inference speed optimization\. MoE operates on a simple yet effective principle: different model components, known as experts, specialize in distinct tasks or data aspects\. For a given input, only relevant experts are activated, reducing computational costs while leveraging a vast pool of specialized knowledge\. This scalable and flexible approach aligns with the scaling law, enabling larger model capacities without proportional computational overhead\. However, current MoE training focuses on training from scratch or upcycling dense LLMs, both requiring vast computational resources and high\-quality corpora\. For instance, DeepSeek\-V3[DeepSeek\-AI et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib28)and Qwen2\.5\-Max[Yang et al\. \(2025\)](https://arxiv.org/html/2609.21672#bib.bib29)were pre\-trained on 14\.8T and 20T tokens, respectively, with additional fine\-tuning, making them costly and less accessible\. In contrast,little research has explored leveraging MoE to accelerate inference using a small\-scale training corpus \(e\.g\., tens of billions of tokens\) while maintaining performance comparable to dense LLMs\. This direction is particularly appealing for large\-scale industrial applications with cost\-sensitive deployment constraints\.

To address this issue, we proposeL0\-MoE, a mixture\-of\-experts \(MoE\) model built via L0\-regularization[Louizos et al\. \(2018\)](https://arxiv.org/html/2609.21672#bib.bib5)using a small, curated30B\-token corpus\. Our approach has two key components: \(1\) L0\-regularization selects critical hidden dimensions in transformer MLPs to form experts\. \(2\) A cluster confusion matrix \(CCM\)\-based sampling method curates the training corpus and schedules dynamic batching\. Using the BGE\-M3 encoder[Chen et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib4)and K\-means clustering[Jin and Han \(2010\)](https://arxiv.org/html/2609.21672#bib.bib30), we extract diverse semantic domains from RedPajama[Weber et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib7)to construct expert\-relevant sub\-datasets\. A gating mechanism and dynamic batching optimize training\. L0\-MoE achieves2\.5×inference speedup with no obvious performance loss across four benchmarks\. Our contributions are as follows: 1\) We introduce a novel MoE building method leveraging L0\-regularization, enabling efficient LLM inference acceleration with minimal training cost\. 2\) We propose a CCM\-based corpus curation and dynamic batching strategy for effective MoE training\. 3\) Extensive experiments validate the efficiency of our method in achieving inference speedup while maintaining performance\.

## 2Preliminary

### 2\.1L0\-regularization

L0\-regularization[Louizos et al\. \(2018\)](https://arxiv.org/html/2609.21672#bib.bib5)is a powerful technique for feature selection and parameter pruning in neural networks\. It imposes a penalty on parameters that deviate from zero, without additional constraints\. This approach enhances model efficiency by eliminating unnecessary computations and resources, as irrelevant parameters are pruned and thus not computed\. For a given weight matrixW∈Rm×nW\\in R^\{m\\times n\}, a mask matrixZ∈0,1nZ\\in\{0,1\}^\{n\}is employed to derive a reduced weightg⁡\(W,Z\)∈Rm×n​0g\(W,Z\)\\in R^\{m\\times n0\}, whereggselectsn​0<nn0<ncolumns fromWWusingZZ\. Due to the non\-differentiable nature ofZZ, optimizing it is challenging\. To address this, the binary hard concrete function is introduced for L0\-regularization, as shown in Equation[1](https://arxiv.org/html/2609.21672#S2.E1)\.

u∼𝒰s=S​i​g​m​o​i​d​\(\(l​o​g​\(u\)−l​o​g​\(1−u\)\+l​o​g​a\)/b\)s¯=s⁡\(ζ−γ\)\+γz=m​i​n​\(1,m​a​x​\(0,s¯\)\)\\begin\{matrix\}u\\sim\\mathcal\{U\}\\\\ s=Sigmoid\(\(log\(u\)\-log\(1\-u\)\+log$a$\)/$b$\)\\\\ \\bar\{s\}=s\(\\zeta\-\\gamma\)\+\\gamma\\\\ z=min\(1,max\(0,\\bar\{s\}\)\)\\end\{matrix\}\(1\)
The uniform distribution𝒰\\mathcal\{U\}is defined over the interval \[0,1\]\. We set the hyper\-parameters asb=0\.83b=0\.83,ζ=1\.1\\zeta=1\.1, andγ=−0\.1\\gamma=\-0\.1by following[Louizos et al\. \(2018\)](https://arxiv.org/html/2609.21672#bib.bib5)\. Using the learnedzz, we estimate the proportion of retained weights asr^=s​u​m​\(z\)m∗n\\hat\{r\}=\\frac\{sum\(z\)\}\{m\*n\}\. To effectively control the desired retention ratiorrfor a given weight matrixWW, we employ a Lagrangian multiplier[Wang et al\. \(2019\)](https://arxiv.org/html/2609.21672#bib.bib6), as described in Equation[2](https://arxiv.org/html/2609.21672#S2.E2)\.

ℒl0=λ1​\(r^−r\)\+λ2​\(r^−r\)2\\mathcal\{L\}\_\{l\_\{0\}\}=\\lambda\_\{1\}\(\\hat\{r\}\-r\)\+\\lambda\_\{2\}\(\\hat\{r\}\-r\)^\{2\}\(2\)
We initialize the learnable parametersλ1\\lambda\_\{1\}andλ2\\lambda\_\{2\}to 0 in our experiments\. In our approach,rrrepresents the retention ratio of the feed\-forward network \(FFN\) up\-projection dimension\.

### 2\.2Mixture of Expert

Mixture of Experts \(MoE\)[Cai et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib20)employs a modular architecture comprising a gating network and multiple expert networks to enhance efficiency and performance through parameter scaling\. This architecture partitions the model into several experts, each specializing in specific subsets of input data\. MoE utilizes a gating mechanism with a router to dynamically select the appropriate experts for processing incoming inputs, allowing the model to concentrate on relevant features while minimizing unnecessary computations\. In our approach, the router is implemented as a linear projection layerWr​o​u​t​e​r∈Rd×NW\_\{router\}\\in R^\{d\\times N\}\. MoE incorporates two auxiliary losses \(Equation[3](https://arxiv.org/html/2609.21672#S2.E3)\), such as the load balancing lossℒb​a​l​a​n​c​e\\mathcal\{L\}\_\{balance\}[Fedus et al\. \(2022\)](https://arxiv.org/html/2609.21672#bib.bib18)and the routerZZ\-Lossℒ​z\\mathcal\{L\}\{z\}[Zoph et al\. \(2022\)](https://arxiv.org/html/2609.21672#bib.bib19), to promote a balanced distribution of inputs among experts\. These losses penalize high values in the logits produced by the gating network, encouraging a more even allocation of tokens to experts\.

ℒa​u​x=ℒb​a​l​a​n​c​e\+λ​ℒzℒb​a​l​a​n​c​e=∑i=1i=N\(ciB−1N\)2ℒz=1B​∑1B\(l​o​g​\(∑iNexi\(j\)\)\)\\begin\{matrix\}\\mathcal\{L\}\_\{aux\}=\\mathcal\{L\}\_\{balance\}\+\\lambda\\mathcal\{L\}\_\{z\}\\\\ \\mathcal\{L\}\_\{balance\}=\\sum\_\{i=1\}^\{i=N\}\(\\frac\{c\_\{i\}\}\{B\}\-\\frac\{1\}\{N\}\)^\{2\}\\\\ \\mathcal\{L\}\_\{z\}=\\frac\{1\}\{B\}\\sum\_\{1\}^\{B\}\(log\(\\sum\_\{i\}^\{N\}e^\{x\_\{i\}^\{\(j\)\}\}\)\)\\end\{matrix\}\(3\)
Herecic\_\{i\}represents the tokens of theit​hi^\{th\}expert, andNNdenotes the number of experts\. The batch containsBBtokens\. The logit for thejt​hj^\{th\}token from theit​hi^\{th\}expert, as determined by the router module, is denoted asxi\(j\)x\_\{i\}^\{\(j\)\}\.

## 3Approach

### 3\.1Cluster Confusion Matrix based Sampling

Given a pretraining corpus, we construct training datasets via the following steps: 1\) Randomly sample a small subset without replacement and use the BGE\-M3 encoder[Chen et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib4)to extractds​vd\_\{sv\}\-dimensional semantic vectors for each sample\. 2\) Apply the K\-means clustering algorithm[Jin and Han \(2010\)](https://arxiv.org/html/2609.21672#bib.bib30)to the semantic vectors to identifyKKcentersC∈ℝK×ds​vC\\in\\mathbb\{R\}^\{K\\times d\_\{sv\}\}\. Divide the small subset intoKKfolds and samplemminstances from each fold to form a datasetDs​l=\{Ds1,…,DsK\}D\_\{sl\}=\\\{D\_\{s\_\{1\}\},\\ldots,D\_\{s\_\{K\}\}\\\}for domain semantic learning, where\|Dsk\|=m\|D\_\{s\_\{k\}\}\|=mfor1≤k≤K1\\leq k\\leq K\. 3\) Repeat steps 1 and 2 forQQiterations to obtainQ×KQ\\times Kcenters andQQdatasets\. For thelt​hl^\{th\}iteration \(l=\{1,2,…,Q\}l=\\\{1,2,\\ldots,Q\\\}\), the cluster centers areC\(l\)∈ℝK×ds​vC^\{\(l\)\}\\in\\mathbb\{R\}^\{K\\times d\_\{sv\}\}and the constructed dataset isDs​l\(l\)D\_\{sl\}^\{\(l\)\}\.

We define the clustering confusion matrix \(CCM\) as per Equation[4](https://arxiv.org/html/2609.21672#S3.E4), whereδ=0\.1\\delta=0\.1is a hyperparameter,CiC\_\{i\}represents theit​hi^\{th\}center vector, andC​C​M​\[i,l\]CCM\[i,l\]denotes the clustering confusion value for theit​hi^\{th\}center at iterationll\. The hypothesis posits that the semantic domain distance for theit​hi^\{th\}center betweenCiC\_\{i\}andCi\(l\)C\_\{i\}^\{\(l\)\}can be assessed using bidirectional inter\-clustering \(f1f\_\{1\}andf2f\_\{2\}\) and intra\-clustering \(f3f\_\{3\}\) cosine similarity\. A larger semantic domain distance indicates thatDs​l\(l\)D\_\{sl\}^\{\(l\)\}from thelt​hl^\{th\}iteration divides domains more distinctly\. We compute the domain semantic distance using Equation[5](https://arxiv.org/html/2609.21672#S3.E5)and reorder theQQdatasets based ondd​s\(l\)d\_\{ds\}^\{\(l\)\}\. In addition to the initialDs​l\(0\)D\_\{sl\}^\{\(0\)\}, our datasets now includeQ−1Q\-1ordered datasetsDs​lo​r​d​\(l\)D\_\{sl\}^\{ord\(l\)\}, whereo​r​d​\(l\)ord\(l\)is the order index\.

C​C​M​\[i,l\]=\(f1\+f2\)∗f3f1​\(i,l\)=1K​∑k=1k=Ke1−s​i​m​\(Ci,Ck\(l\)\)f2​\(i,l\)=1K​∑k=1k=Ke1−s​i​m​\(Ck,Ci\(l\)\)f3​\(i,l\)=δ​∑k=1k=Kes​i​m​\(Ck\(l\),Ci\(l\)\)∑k=1k=Kes​i​m​\(Ck,Ci\)\\begin\{matrix\}CCM\[i,l\]=\(f\_\{1\}\+f\_\{2\}\)\*f\_\{3\}\\\\ f\_\{1\}\(i,l\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{k=K\}e^\{1\-sim\(C\_\{i\},C\_\{k\}^\{\(l\)\}\)\}\\\\ f\_\{2\}\(i,l\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{k=K\}e^\{1\-sim\(C\_\{k\},C\_\{i\}^\{\(l\)\}\)\}\\\\ f\_\{3\}\(i,l\)=\\delta\\frac\{\\sum\_\{k=1\}^\{k=K\}e^\{sim\(C\_\{k\}^\{\(l\)\},C\_\{i\}^\{\(l\)\}\)\}\}\{\\sum\_\{k=1\}^\{k=K\}e^\{sim\(C\_\{k\},C\_\{i\}\)\}\}\\end\{matrix\}\(4\)dd​s\(l\)=max\(CCM\[:,l\]\)\+βK∑i=1i=KCCM\[i,l\]d\_\{ds\}^\{\(l\)\}=max\(CCM\[:,l\]\)\+\\frac\{\\beta\}\{K\}\\sum\_\{i=1\}^\{i=K\}CCM\[i,l\]\(5\)

### 3\.2Expert Construction via L0\-regularization

We construct the experts using pretrained checkpoints of dense LLMs\. The intermediate size of the feed forward network \(FFN\) layer isdi​n​td\_\{int\}, and we apply a maskZ∈Rdi​n​tZ\\in R^\{d\_\{int\}\}\. For each domain subsetDskD\_\{s\_\{k\}\}\(k∈\{1,2,…,K\}k\\in\\\{1,2,\\ldots,K\\\}\) derived from the initialDs​l\(0\)D\_\{sl\}^\{\(0\)\}, we employ the LLM pretraining lossℒl​l​m\\mathcal\{L\}\_\{llm\}along with the L0\-regularization loss, as specified in Equation[1](https://arxiv.org/html/2609.21672#S2.E1), to selectr∈\(0,1\)r\\in\(0,1\)\*100% of the dimensions fromdi​n​td\_\{int\}, following Equation[6](https://arxiv.org/html/2609.21672#S3.E6)\.

ℒe​x​p=ℒl​l​m\+ℒl​0\\mathcal\{L\}\_\{exp\}=\\mathcal\{L\}\_\{llm\}\+\\mathcal\{L\}\_\{l0\}\(6\)To ensure stable training, we gradually adjustrrfrom 100% to the target ratiort​a​r​g​e​tr^\{target\}\. We freeze all non\-MLP parameters of dense LLMs, and the L0\-regularization\-based training yieldsKKexperts, each specialized for distinct semantic domains\.

### 3\.3Dynamic Batching for MoE Training

To train the MoE to effectively select appropriate experts based on inputs, we follow Equation[7](https://arxiv.org/html/2609.21672#S3.E7), whereℒa​u​x\\mathcal\{L\}\_\{aux\}is defined in Equation[3](https://arxiv.org/html/2609.21672#S2.E3)\. The MoE is initialized withKKpre\-trained experts and a router for each MoE layer\.

ℒ=ℒl​l​m\+α​ℒa​u​x\\mathcal\{L\}=\\mathcal\{L\}\_\{llm\}\+\\alpha\\mathcal\{L\}\_\{aux\}\(7\)
We employ a two\-loop batch construction strategy during training: 1\) domain semantic distance scheduling, where we begin withDs​lo​r​d​\(l\)D\_\{sl\}^\{ord\(l\)\}having a lowerdd​sd\_\{ds\}; 2\) multi\-domain gathering scheduling, where samples inDs​lo​r​d​\(l\)D\_\{sl\}^\{ord\(l\)\}are arranged in a cyclic sequence orderxi1,xi2,…,xiKx\_\{i\}^\{1\},x\_\{i\}^\{2\},\\ldots,x\_\{i\}^\{K\}, and we selectp∗Kp\*K\(p=\{1,2,3,…\}p=\\\{1,2,3,\\ldots\\\}\) samples to form a batch\. This scheduling offers two advantages: 1\) In the initial iterations, the MoE rapidly learns to select appropriate experts since the domain samples inDs​lD\_\{sl\}have been previously encountered by the experts\. Consequently, the sequence\-level selection capabilities of routers are effectively initialized\. 2\) As training progresses, the domains inDs​lo​r​d​\(l\)D\_\{sl\}^\{ord\(l\)\}gradually transition to different semantic spaces, encouraging routers to select multiple experts for each input sample\. This enhances the token\-level selection capabilities of the routers\.

## 4Experiments

### 4\.1Experimental Setup

ModelMMLUGSM8KHumanEvalBBHAverageSpeedupLlama\-3\-8B66\.656\.033\.557\.753\.5Llama\-3\-8B w/ L0\-MoE66\.355\.933\.757\.253\.32\.0xMistral\-7B64\.152\.229\.356\.150\.4Mistral\-7B w/ L0\-MoE64\.853\.631\.155\.951\.42\.1xQwen2\-7B70\.379\.951\.262\.666\.0Qwen2\-7B w/ L0\-MoE70\.480\.552\.061\.566\.12\.5xTable 1:Evaluation of different LLMs on MMLU, GSM8K, HumanEval and BBH benchmarks\.Dataset\.We train on the RedPajama dataset[Weber et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib7), a replicated pre\-training corpus for LLaMA models, following prior work[Xia et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib1)\. Evaluation is conducted on four public benchmarks: MMLU[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib31), GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib33), HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib32), and BigBench Hard \(BBH\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib34)\. Each benchmark evaluates distinct aspects of model performance, offering insights into the strengths and limitations of LLMs\.

Baselines\.To assess effectiveness and versatility, we evaluate our method on three open\-source LLMs: Llama\-3\-8B[Dubey et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib13), Mistral\-7B[Jiang et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib15), and Qwen2\-7B[Yang et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib14)\. Comparisons include L0\-regularized MoEs, original LLMs, and inference optimization techniques such as GPTQ quantization[Frantar et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib8), LLM Shearing pruning[Xia et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib1), and RKD \+ CoT knowledge distillation[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib17);[Feng et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib16)\. For CCM, we run 21 iterations, collecting 30B tokens\. Experiments use a cluster/expert size ofK=64K=64with linear warmup, annealing, and a peak learning rate of 1e\-4\. Further details are in Appendix[A\.2](https://arxiv.org/html/2609.21672#A1.SS2)\.

Implementation Details\.We train our model using the FSDP framework222[https://pytorch\.org/docs/stable/fsdp\.html](https://pytorch.org/docs/stable/fsdp.html), employing a layer\-wise wrapping policy with the Zero\-3 parameter sharding strategy, without CPU offloading\. For inference during evaluation, we utilize the SGlang framework333[https://github\.com/sgl\-project/sglang](https://github.com/sgl-project/sglang), which is highly optimized for the efficient execution of both dense LLMs and MoEs\. All baseline models in our experiments utilize the same SGlang inference framework, ensuring a fair and consistent comparison of inference speeds\. Our method is framework\-agnostic and can similarly be implemented using other inference frameworks \(e\.g\., vLLM444[https://github\.com/vllm\-project/vllm](https://github.com/vllm-project/vllm)\)\. The primary source of inference acceleration in our work is the proposed L0\-regularization\-based MoE architecture, not the inference framework itself\. To ensure a fair comparison, we strictly adhere to the original evaluation settings for each benchmark\. To support future research, we will release our curated dataset and code to enhance the reproducibility of our work\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.21672#S4.T1)presents the model with the highest performance under our settings\. The L0\-MoE consistently achieves a2​\-​2\.5​x2\\text\{\-\}2\.5\\text\{x\}inference speedup across all base LLMs\. Additionally, L0\-MoE maintains performance comparable to the base LLMs across four benchmarks, with the L0\-MoE variant of Mistral even demonstrating a 1% average performance improvement\. Table[2](https://arxiv.org/html/2609.21672#S4.T2)compares these results with other inference acceleration baselines, which, despite achieving some speedup, exhibit noticeable performance degradation\.

ModelMMLUGSM8KSpeedupQwen2\-7B70\.379\.9\-L0\-MoE70\.480\.52\.5xGPTQ67\.873\.81\.8xLLM Shearing68\.275\.52\.6xRKD \+ CoT61\.260\.25\.1xTable 2:Comparison with other inference acceleration baselines\. We employ Qwen2\-7B as the base LLM\.ModelMMLUGSM8KL0\-MoE70\.480\.5CCM w/o K\-means68\.278\.1w/ random order batching68\.275\.5w/ random batch batching66\.677\.1Random MoE48\.169\.6Magnitude52\.669\.1OBS68\.474\.1SVD55\.273\.8Table 3:Ablation study of CCM and L0\-regularization\. We conduct experiments on MMLU and GSM8K datasets with Qwen2\-7B\.
### 4\.3Ablation Study

Table[3](https://arxiv.org/html/2609.21672#S4.T3)presents the ablation study on the CCM module, dynamic batching, and L0\-regularization\. Removing the K\-means clustering from the CCM module results in a performance decline, underscoring the importance of effective sub\-dataset curation\. For dynamic batching, substituting it with random order or random batch scheduling also leads to degraded performance\.

In the context of MoE expert construction, we replace L0\-regularization with four alternative methods: 1\)Random MoE[Zhu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib40): Selects MLP dimensions randomly, serving as a baseline to assess the necessity and effectiveness of dimension selection in expert construction, 2\)Magnitude[Sun et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib3)\): Selects the most influential elements in the weight matrix, improving upon traditional magnitude pruning by considering both the weights and their corresponding input activations using the L2 norm, 3\)OBS[Frantar et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib9);[Frantar and Alistarh \(2022\)](https://arxiv.org/html/2609.21672#bib.bib10)\): Identifies the most critical dimensions using the OBS Hessian matrix, which encapsulates second\-order derivative information of the loss function with respect to model parameters\. This approach is crucial for both pruning and quantization, as it helps retain the weights that most significantly impact model performance, 4\)SVD[Wang et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib21)\): Decomposes the weight matrix using singular value decomposition and selects the most significant columns\. By retaining the largest singular values, it reduces parameter count while preserving essential information\. This truncation minimizes compression loss, and layer\-wise updates further fine\-tune the model to maintain accuracy\. The results demonstrate the superiority of L0\-MoE over them\.

ModelMMLU\#Para\.\(B\)SpeedupL0\-MoE70\.423\.32\.5xCCM Iter\. \(Q=2Q=2\)68\.223\.32\.5xCCM Iter\. \(Q=5Q=5\)68\.823\.32\.5xCCM Iter\. \(Q=10Q=10\)69\.723\.32\.5xExpert size \(K=8K=8\)50\.94\.84\.6xExpert size \(K=32K=32\)69\.212\.73\.2xTable 4:Hyper\-parameter tuning of sampling iterations \(QQ\) and cluster size \(KK\), keeping 2\.8B activated parameters for L0\-MoE\. Qwen2\-7B is the base LLM\. \#Para\.\(B\) is the number of model parameters\.
### 4\.4Discussion

Our approach involves two critical hyperparameters: the sampling iterationsQQin CCM curated datasets and the expert sizeKKin MoE\. Table[4](https://arxiv.org/html/2609.21672#S4.T4)provides a detailed overview of hyperparameter tuning\. Increasing the number of iterations for CCM enhances performance but also demands greater computational resources\. We find that an initial iteration plus 20 additional iterations suffice to optimize model performance\. While increasing the number of experts improves performance, it also reduces inference speed\. Therefore, we select an appropriate expert size to balance performance enhancement and LLM acceleration\.

Besides, the optimal number of clusters \(KK\) primarily depends on the characteristics of the pre\-training corpus; thus, it may not directly transfer to other experiments if the corpus differs significantly\. We recommend applying our method to pre\-training corpora with abundant topical diversity, such as RedPajama, which contains millions of domains\. In such corpora, a largerKKcan effectively cluster more specialized subsets, enabling CCM to be more effectively applied when constructing L0\-MoE models\.

To assess model size impact, we conducted further experiments using a 1\.5B\-parameter Qwen2 model with 64 experts, achieving a 2x speedup without performance degradation\. However, due to computational resource constraints, we have not yet experimented with larger models \(e\.g\., 70B parameters\)\. We hypothesize that larger LLMs could potentially achieve even greater speedups\. We leave the verification of this hypothesis for larger\-scale models as future work\.

## 5Conclusion and Future Work

In this paper, we propose a novel Mixture\-of\-Experts based approach to accelerate LLM inference, leveraging clustering confusion matrix for dataset curation, L0\-regularization for expert selection, and dynamic batching for efficient training with only 30B tokens\. Our method achieves a 2\.5× speedup over dense LLMs, outperforming strong baselines nearly without performance loss\. Future work will explore scaling our approach to larger LLMs and expanding the corpus size to further enhance L0\-MoE performance beyond dense LLMs\.

## Limitations

We did not compare our method with MoEs such as DeepSeek\-MoE[Dai et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib35), Qwen\-MoE[Qwen\-Team \(2024\)](https://arxiv.org/html/2609.21672#bib.bib36), and Mixtral[Jiang et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib37), which scale up model parameters in dense models with immense computational costs, processing trillions of tokens\. In contrast, our approach utilizes only 30B tokens, making it more comparable to baseline post\-training inference speedup methods\.

Despite the promising results, several limitations remain: \(1\) The dataset for training each expert is selected via sequence\-level semantic clustering, introducing exposure bias since MoE expert selection is performed at the token level\. \(2\) The method does not explicitly measure inter\-expert differences, potentially leading to redundant parameters that hinder L0\-MoE’s inference acceleration\. Future work should explore token\-level dataset partitioning to mitigate exposure bias\. Additionally, novel learning paradigms are needed to reduce parameter redundancy and enhance expert routing efficiency\.

## References

- Ainslieet al\.\(2023\)J\. Ainslie, J\. Lee\-Thorp, M\. de Jong, Y\. Zemlyanskiy, F\. Lebron, and S\. SanghaiGQA: training generalized multi\-query transformer models from multi\-head checkpoints\.InThe 2023 Conference on Empirical Methods in Natural Language Processing,External Links:[Link](https://openreview.net/forum?id=hmOwOZWzYE)Cited by:[§A\.1](https://arxiv.org/html/2609.21672#A1.SS1.p1.1)\.
- Bubecket al\.\(2023\)S\. Bubeck, V\. Chandrasekaran, R\. Eldan, J\. Gehrke, E\. Horvitz, E\. Kamar, P\. Lee, Y\. T\. Lee, Y\. Li, S\. Lundberg, H\. Nori, H\. Palangi, M\. T\. Ribeiro, and Y\. ZhangSparks of artificial general intelligence: early experiments with gpt\-4\.External Links:2303\.12712,[Link](https://arxiv.org/abs/2303.12712)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1)\.
- Caiet al\.\(2024\)W\. Cai, J\. Jiang, F\. Wang, J\. Tang, S\. Kim, and J\. HuangA survey on mixture of experts\.arXiv preprint arXiv:2407\.06204\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p3.1),[§2\.2](https://arxiv.org/html/2609.21672#S2.SS2.p1.1)\.
- Chenet al\.\(2024\)J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. LiuBge m3\-embedding: multi\-lingual, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.arXiv preprint arXiv:2402\.03216\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p4.1),[§3\.1](https://arxiv.org/html/2609.21672#S3.SS1.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[3rd item](https://arxiv.org/html/2609.21672#A1.I1.i3.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[2nd item](https://arxiv.org/html/2609.21672#A1.I1.i2.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p1.1),[§A\.3](https://arxiv.org/html/2609.21672#A1.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p1.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\. X\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu, Z\. Xie, Y\. K\. Li, P\. Huang, F\. Luo, C\. Ruan, Z\. Sui, and W\. LiangDeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.External Links:2401\.06066,[Link](https://arxiv.org/abs/2401.06066)Cited by:[Limitations](https://arxiv.org/html/2609.21672#Sx1.p1.1)\.
- DeepSeek\-AIet al\.\(2025\)DeepSeek\-AI, D\. Guo, D\. Yang, H\. Zhang, J\. Song,et al\.DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1)\.
- DeepSeek\-AIet al\.\(2024\)DeepSeek\-AI, A\. Liu, B\. Feng, B\. Xue, B\. Wang,et al\.DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p3.1)\.
- Dubeyet al\.\(2024\)A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§2\.2](https://arxiv.org/html/2609.21672#S2.SS2.p1.1)\.
- Fenget al\.\(2024\)T\. Feng, Y\. Li, L\. Chenglin, H\. Chen, F\. Yu, and Y\. ZhangTeaching small language models reasoning through counterfactual distillation\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 5831–5842\.Cited by:[3rd item](https://arxiv.org/html/2609.21672#A1.I2.i3.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p3.1),[§1](https://arxiv.org/html/2609.21672#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Frantar and Alistarh \(2022\)E\. Frantar and D\. AlistarhOptimal brain compression: a framework for accurate post\-training quantization and pruning\.Advances in Neural Information Processing Systems35,pp\. 4475–4488\.Cited by:[§4\.3](https://arxiv.org/html/2609.21672#S4.SS3.p2.1)\.
- Frantaret al\.\(2023\)E\. Frantar, S\. Ashkboos, T\. Hoefler, and D\. AlistarhOPTQ: accurate quantization for generative pre\-trained transformers\.InThe Eleventh International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=tcbBPnfwxS)Cited by:[1st item](https://arxiv.org/html/2609.21672#A1.I2.i1.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p3.1),[§1](https://arxiv.org/html/2609.21672#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Frantaret al\.\(2021\)E\. Frantar, E\. Kurtic, and D\. AlistarhM\-fac: efficient matrix\-free approximations of second\-order information\.Advances in Neural Information Processing Systems34,pp\. 14873–14886\.Cited by:[§4\.3](https://arxiv.org/html/2609.21672#S4.SS3.p2.1)\.
- Gemini\-Teamet al\.\(2024\)Gemini\-Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai,et al\.Gemini: a family of highly capable multimodal models\.External Links:2312\.11805,[Link](https://arxiv.org/abs/2312.11805)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1)\.
- Guet al\.\(2024\)Y\. Gu, L\. Dong, F\. Wei, and M\. HuangMiniLLM: knowledge distillation of large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[3rd item](https://arxiv.org/html/2609.21672#A1.I2.i3.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p3.1),[§1](https://arxiv.org/html/2609.21672#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[1st item](https://arxiv.org/html/2609.21672#A1.I1.i1.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p1.1),[§A\.3](https://arxiv.org/html/2609.21672#A1.SS3.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.External Links:1503\.02531,[Link](https://arxiv.org/abs/1503.02531)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p2.1)\.
- Jianget al\.\(2023\)A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. d\. l\. Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier,et al\.Mistral 7b\.arXiv preprint arXiv:2310\.06825\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Jianget al\.\(2024\)A\. Q\. Jiang, A\. Sablayrolles, A\. Roux, A\. Mensch, B\. Savary, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, E\. B\. Hanna, F\. Bressand, G\. Lengyel, G\. Bour, G\. Lample, L\. R\. Lavaud, L\. Saulnier, M\. Lachaux, P\. Stock, S\. Subramanian, S\. Yang, S\. Antoniak, T\. L\. Scao, T\. Gervet, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. SayedMixtral of experts\.External Links:2401\.04088,[Link](https://arxiv.org/abs/2401.04088)Cited by:[Limitations](https://arxiv.org/html/2609.21672#Sx1.p1.1)\.
- Jin and Han \(2010\)X\. Jin and J\. HanK\-means clustering\.InEncyclopedia of Machine Learning,pp\. 563–564\.External Links:ISBN 978\-0\-387\-30164\-8,[Document](https://dx.doi.org/10.1007/978-0-387-30164-8%5F425),[Link](https://doi.org/10.1007/978-0-387-30164-8_425)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p4.1),[§3\.1](https://arxiv.org/html/2609.21672#S3.SS1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.External Links:2001\.08361,[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1)\.
- Linet al\.\(2024a\)H\. Lin, H\. Xu, Y\. Wu, J\. Cui, Y\. Zhang, L\. Mou, L\. Song, Z\. Sun, and Y\. WeiDuquant: distributing outliers via dual transformation makes stronger quantized llms\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,Cited by:[§A\.3](https://arxiv.org/html/2609.21672#A1.SS3.p1.1),[§1](https://arxiv.org/html/2609.21672#S1.p2.1)\.
- Linet al\.\(2024b\)J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. HanAWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of Machine Learning and Systems6,pp\. 87–100\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p2.1)\.
- Louizoset al\.\(2018\)C\. Louizos, M\. Welling, and D\. P\. KingmaLearning sparse neural networks through l\_0 regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1Y8hhg0b)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.21672#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2609.21672#S2.SS1.p2.1)\.
- Maet al\.\(2023\)X\. Ma, G\. Fang, and X\. WangLLM\-pruner: on the structural pruning of large language models\.InThirty\-seventh Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=J8Ajf9WfXP)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p2.1)\.
- OpenAIet al\.\(2024\)OpenAI, J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman,et al\.GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1)\.
- Qwen\-Team \(2024\)Qwen\-TeamQwen1\.5\-moe: matching 7b model performance with 1/3 activated parameters”\.External Links:[Link](https://qwenlm.github.io/blog/qwen-moe/)Cited by:[Limitations](https://arxiv.org/html/2609.21672#Sx1.p1.1)\.
- Sunet al\.\(2023\)M\. Sun, Z\. Liu, A\. Bair, and J\. Z\. KolterA simple and effective pruning approach for large language models\.arXiv preprint arXiv:2306\.11695\.Cited by:[§4\.3](https://arxiv.org/html/2609.21672#S4.SS3.p2.1)\.
- Suzgunet al\.\(2023\)M\. Suzgun, N\. Scales, N\. Schärli, S\. Gehrmann, Y\. Tay, H\. W\. Chung, A\. Chowdhery, Q\. Le, E\. Chi, D\. Zhou, and J\. WeiChallenging BIG\-bench tasks and whether chain\-of\-thought can solve them\.InFindings of the Association for Computational Linguistics: ACL 2023,A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 13003–13051\.External Links:[Link](https://aclanthology.org/2023.findings-acl.824/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.824)Cited by:[4th item](https://arxiv.org/html/2609.21672#A1.I1.i4.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p1.1)\.
- Wanget al\.\(2024\)X\. Wang, Y\. Zheng, Z\. Wan, and M\. ZhangSvd\-llm: truncation\-aware singular value decomposition for large language model compression\.arXiv preprint arXiv:2403\.07378\.Cited by:[§4\.3](https://arxiv.org/html/2609.21672#S4.SS3.p2.1)\.
- Wanget al\.\(2019\)Z\. Wang, J\. Wohlwend, and T\. LeiStructured pruning of large language models\.arXiv preprint arXiv:1910\.04732\.Cited by:[§2\.1](https://arxiv.org/html/2609.21672#S2.SS1.p2.1)\.
- Weberet al\.\(2024\)M\. Weber, D\. Fu, Q\. Anthony, Y\. Oren, S\. Adams, A\. Alexandrov, X\. Lyu, H\. Nguyen, X\. Yao, V\. Adams,et al\.Redpajama: an open dataset for training large language models\.arXiv preprint arXiv:2411\.12372\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p4.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p1.1)\.
- Xiaet al\.\(2024\)M\. Xia, T\. Gao, Z\. Zeng, and D\. ChenSheared LLaMA: accelerating language model pre\-training via structured pruning\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=09iOdaeOzp)Cited by:[2nd item](https://arxiv.org/html/2609.21672#A1.I2.i2.p1.1),[§A\.2](https://arxiv.org/html/2609.21672#A1.SS2.p3.1),[§1](https://arxiv.org/html/2609.21672#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Xiaoet al\.\(2023\)G\. Xiao, J\. Lin, M\. Seznec, H\. Wu, J\. Demouth, and S\. HanSmoothquant: accurate and efficient post\-training quantization for large language models\.InInternational Conference on Machine Learning,pp\. 38087–38099\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p2.1)\.
- Yanget al\.\(2024\)A\. Yang, B\. Yang, B\. Hui, B\. Zheng, B\. Yu, C\. Zhou, C\. Li, C\. Li, D\. Liu, F\. Huang,et al\.Qwen2 technical report\.arXiv preprint arXiv:2407\.10671\.Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p1.1),[§4\.1](https://arxiv.org/html/2609.21672#S4.SS1.p2.1)\.
- Yanget al\.\(2025\)A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. QiuQwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§1](https://arxiv.org/html/2609.21672#S1.p3.1)\.
- Zhuet al\.\(2024\)T\. Zhu, X\. Qu, D\. Dong, J\. Ruan, J\. Tong, C\. He, and Y\. ChengLLaMA\-MoE: building mixture\-of\-experts from LLaMA with continual pre\-training\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 15913–15923\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.890/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.890)Cited by:[§4\.3](https://arxiv.org/html/2609.21672#S4.SS3.p2.1)\.
- Zophet al\.\(2022\)B\. Zoph, I\. Bello, S\. Kumar, N\. Du, Y\. Huang, J\. Dean, N\. Shazeer, and W\. FedusSt\-moe: designing stable and transferable sparse expert models\.arXiv preprint arXiv:2202\.08906\.Cited by:[§2\.2](https://arxiv.org/html/2609.21672#S2.SS2.p1.1)\.

## Appendix AAppendix

This section provides further details on the model architecture, experimental setup \(including evaluation tasks, baselines, and hyperparameter settings\), and implementation details\.

### A\.1Model Architecture

Table[5](https://arxiv.org/html/2609.21672#A1.T5)presents the detailed architecture of the baseline models and L0\-MoE\. All models incorporate group query attention \(GQA\)[Ainslie et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib38)within the self\-attention layer\. For the L0\-MoE models, the bottom 4 layers \(for Qwen2\) and 8 layers \(for Mistral and Llama3\) are configured as dense layers, while the remaining layers are transformed into MoE layers\. We select the top\-2 experts for each input token\.

### A\.2Experimental Setups

Evaluation Tasks\.We assess performance on four public benchmarks: MMLU[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib31), GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib33), HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib32), and BigBench Hard \(BBH\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib34)\.

- •MMLU\(Massive Multitask Language Understanding\)[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib31)comprises 57 tasks spanning diverse subjects, including STEM \(Science, Technology, Engineering, and Mathematics\), humanities, social sciences, and specialized domains such as law and ethics\.
- •GSM8K\(Grade School Math 8K\)[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib33)is a benchmark designed to assess the mathematical reasoning capabilities of LLMs, containing 8,500 high\-quality elementary math word problems\.
- •HumanEval[Chen et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib32)evaluates the code generation capabilities of LLMs through 164 programming tasks, each requiring the model to generate a function that satisfies a given set of test cases\.
- •BBH\(BIG\-Bench Hard\)[Suzgun et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib34)is a subset of the larger BIG\-Bench dataset, consisting of 23 highly challenging tasks designed to exceed the capabilities of current LLMs\. These tasks demand creative problem\-solving and deep domain expertise\.

Baselines\.We compare the L0\-regularized MoEs with the original LLMs and other LLM inference optimization methods, including the quantization baseline GPTQ[Frantar et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib8), the model pruning baseline LLM Shearing[Xia et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib1), and the knowledge distillation baseline RKD \+ CoT[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib17);[Feng et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib16)\.

- •GPTQ[Frantar et al\. \(2023\)](https://arxiv.org/html/2609.21672#bib.bib8)is a block\-wise quantization method that extends traditional power\-of\-two quantization by allowing non\-uniform bin widths, enabling a better approximation of the original floating\-point value distribution\.
- •LLM\-Shearing[Xia et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib1)employs structured pruning to construct lightweight, structured LLMs from pretrained checkpoints\. It jointly removes attention heads, layers, feed\-forward networks \(FFNs\), and hidden dimensions in an end\-to\-end manner to optimize efficiency\.
- •RKD \+ CoT: We apply RKD[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib17)to distill the CoT[Feng et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib16)capabilities of Qwen2\-7B into Qwen2\-1\.5B\. RKD[Gu et al\. \(2024\)](https://arxiv.org/html/2609.21672#bib.bib17)aligns the student model with the teacher’s distribution using reverse Kullback\-Leibler divergence \(KLD\), encouraging the student to focus on the most probable outcomes\. This helps preserve the quality of the student model’s predictions by distilling Chain\-of\-Thought \(CoT\) reasoning from the teacher model\.

Hyper\-parameter Setting\.The detailed hyper\-parameter settings are presented in Table[7](https://arxiv.org/html/2609.21672#A1.T7)\. This includes the hyper\-parameters for the clustering confusion matrix \(CCM\) as well as those for MoE training\.

### A\.3Comparison with DuQuant

To further validate our method, we compare it with DuQuant[Lin et al\. \(2024a\)](https://arxiv.org/html/2609.21672#bib.bib39), a recent quantization technique targeting outlier activations in large language models \(LLMs\)\. DuQuant uses rotation and permutation to redistribute outliers, aiming to simplify quantization and improve robustness\. We evaluate it on the LLama\-3\-8B model using MMLU[Hendrycks et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib31)and GSM8K[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.21672#bib.bib33)benchmarks\. As shown in Table[6](https://arxiv.org/html/2609.21672#A1.T6), DuQuant suffers noticeable performance degradation, highlighting its limitations\. In contrast, our L0\-MoE method performs better under the same setting, demonstrating superior accuracy preservation\.

ModelParameters\(B\)LayerHiddenQ/KVFFNMoE FFNExpertsLlama\-3\-8B7\.532409632/814336Llama\-3\-8B w/ L0\-MoE25\.1/3\.232/24409632/814336102464:2Mistral\-7B7\.132409632/814336Mistral\-7B w/ L0\-MoE24\.7/2\.932/24409632/814336102464:2Qwen2\-7B728358428/418944Qwen2\-7B w/ L0\-MoE23\.3/2\.828/24358428/418944128064:2Table 5:Detailed model architecture parameters\. We denote the total and activated parameters of the MoEs, as well as the total layers and MoE layers, using the format “32/24”, etc\. All models utilize GQA, and we present the query/key\-value heads\. “FFN” refers to the dense decoder MLP size, while “MoE FFN” indicates the intermediate size of the expert for the MoE layer\. The total and activated experts are represented as “64:2”, etc\.ModelMMLUGSM8KSpeedUpLlama\-3\-8B66\.656\.0\-Llama\-3\-8B w/ L0\-MoE66\.355\.92\.0×\\timesDeQuant/W4A457\.951\.60\.46×\\timesDeQuant \+ LWC/W4A462\.251\.10\.46×\\timesTable 6:Performance comparison of L0\-MoE and DuQuant on MMLU and GSM8K benchmarks\. Llama\-3\-8B is the base LLM\. Due to the lack of DuQuant support in SGlang, we tested inference speed using naive PyTorch transformers \(batch size = 64, sequence length = 1200\)\. Without optimized kernels, DuQuant is slow \(0\.46x speedup\), but future SGlang support could make it comparable to GPTQ \(1\.8x speedup\)\.CCM Hyper\-parametersQQ21KK64ds​vd\_\{sv\}1024Ds​l\(0\)D\_\{sl\}^\{\(0\)\}12B tokens;\|Dsi\(0\)\|≈0\.15​B\|D\_\{s\_\{i\}\}^\{\(0\)\}\|\\approx 0\.15BDs​l\(l\)D\_\{sl\}^\{\(l\)\},l≥1l\\geq 10\.9B tokens;\|Dsi\(0\)\|≈0\.007​B\|D\_\{s\_\{i\}\}^\{\(0\)\}\|\\approx 0\.007BMoE Training Hyper\-parametersSequence length4096Learning rate1e\-4Warmup ratio \(expert\)0\.2Warmup ratio \(MoE\)0\.06Warmup typeLinearAnnealing ratio0\.1Annealing typeCosineBatch tokens512Kα\\alphain Eq\.[7](https://arxiv.org/html/2609.21672#S3.E7)0\.01β\\betain Equation[5](https://arxiv.org/html/2609.21672#S3.E5)0\.5λ\\lambdain Eq\.[3](https://arxiv.org/html/2609.21672#S2.E3)0\.3Training epoch1Table 7:Hyper\-parameters for CCM and MoE training\.

Similar Articles

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

Hugging Face Daily Papers

论文提出 E-MoE(Enhanced Mixture-of-Experts),用 MoE 路由决策构成的离散共享隐变量来建模非因子化的反向过程,缓解连续隐变量的后验坍缩问题,并在无需增加活跃参数的情况下提升掩码扩散语言模型在少步生成场景下的样本质量。

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Hugging Face Daily Papers

The paper proposes LOOM, a scalable recipe for Looped Mixture-of-Experts models that stabilizes recurrence via residual scaling and input re-injection while diversifying expert usage with per-loop routers, achieving stable scaling to 9-12 loops and improved perplexity and zero-shot accuracy on 100M-1.7B models.

EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

arXiv cs.AI

EntropyMoE introduces an entropy-aware Mixture-of-Experts architecture for tokenizer-free LLMs, using dynamic byte patches as routing units to enable sparse conditional computation. Experiments show it achieves the lowest held-out bits-per-byte among baselines while maintaining downstream accuracy.