The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Summary
A technical survey of Mixture-of-Experts architectures in LLMs, organizing evolution along expert granularity, topology, routing, load balancing, and execution, and proposing complementary views of architectural milestones and control planes.
arXiv:2608.08650v1 Announce Type: new
Abstract: Mixture-of-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone. This technical survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure. We describe eight architectural milestones as a dependency graph with six mainline developments and two orthogonal branches, rather than as eight successive generations. We then analyze individual systems through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism. These planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation is mapped onto physical devices. The framework connects algorithmic choices such as Top-k routing, shared experts, fine-grained experts, and dynamic expert composition with systems concerns including token dispatch, device placement, all-to-all communication, and communication-computation overlap. We conclude with equal-budget pretraining experiments, quality and systems metrics, and open research questions. The main trend is a shift from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.
View Cached Full Text
Cached at: 08/11/26, 08:09 AM
# Contents Source: [https://arxiv.org/html/2608.08650](https://arxiv.org/html/2608.08650) \\setmathfont STIXTwoMath\-Regular\.otf\\setCJKmainfont\[AutoFakeBold=2,ItalicFont=gkai00mp\.ttf\]gbsn00lp\.ttf\\setCJKsansfont\[AutoFakeBold=2\]gbsn00lp\.ttf\\setCJKmonofontgbsn00lp\.ttf\\EdefEscapeHexbilingual\.english\.0bilingual\.english\.0\\EdefEscapeHexEnglish VersionEnglish Version\\hyper@anchorstartbilingual\.english\.0\\hyper@anchorend \\zihao 1The Evolution of Mixture\-of\-Experts Architectures in Large Language Models \\zihao 3Routing, Topology, Load Balancing, and Expert Parallelism Jiguo Li111This report was completed with the assistance of Codex\. [jiguolee@gmail\.com](https://arxiv.org/html/2608.08650v1/mailto:[email protected]) August 2026 ###### Abstract Mixture\-of\-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone\. This technical survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture\-of\-Experts systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure\. We describe eight architectural milestones as a dependency graph with six mainline developments and two orthogonal branches, rather than as eight successive generations\. We then analyze individual systems through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism\. These planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation is mapped onto physical devices\. The framework connects algorithmic choices such as Top\-k routing, shared experts, fine\-grained experts, and dynamic expert composition with systems concerns including token dispatch, device placement, all\-to\-all communication, and communication\-computation overlap\. We conclude with equal\-budget pretraining experiments, quality and systems metrics, and open research questions\. The main trend is a shift from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution\. ###### 摘要 Mixture\-of\-Experts（MoE）已经成为大语言模型在固定单 token 计算预算下扩展参数容量的核心架构之一。然而，若只按模型发布时间罗列 GShard、Switch、Mixtral、DeepSeekMoE、DeepSeek\-V3 与新近动态 MoE，会掩盖真正推动结构变化的瓶颈迁移。本文综合算法、推理系统和高效架构三类 Survey，并回到代表性原始论文与官方技术报告，提出一个由五条耦合路线构成的统一框架：expert 粒度、expert 拓扑、路由自由度、负载均衡作用域和执行结构。基于该框架，本文把 MoE 演进归纳为八个演进节点；它们不是八代模型的线性年表，而是由六个主干节点与两个正交分支构成的依赖图。在历史时间轴之外，本文进一步用 Expert topology、Routing、Balance 和 Expert Parallel 四个控制面剖解任一代 MoE 的内部工作机制，分别回答“有哪些专家、每个 token 选谁、群体负载如何受控、选中的计算如何映射到设备”。最后给出适用于基座预训练的等预算实验设计、系统指标和仍待解决的研究问题。核心结论是：现代 MoE 的竞争焦点已从“稀疏激活更多参数”转向“让语义路由、计算预算和物理执行解耦”。 Keywords:large language models; Mixture\-of\-Experts; sparse routing; load balancing; Expert Parallelism; dynamic computation; ScMoE Key point:The eight milestones describe historical bottleneck migration, whereas the four control planes provide a structural view of an individual MoE system\. They are complementary views, not two competing stage taxonomies\. ## 1 Problem Definition and Analysis Scope Classic MoE research asks how a gating network allocates examples to local experts\. Sparse MoE for large language models adds a stricter requirement: total parameter capacity should grow without a proportional increase in the parameters and computation activated by each token\. Existing surveys provide complementary perspectives: Cai et al\. organize algorithms, systems, and applications into a full\-stack taxonomy\[[1](https://arxiv.org/html/2608.08650#bib.bib1)\]; Liu et al\. analyze model, system, and hardware optimizations for inference\[[2](https://arxiv.org/html/2608.08650#bib.bib2)\]; and Zhu et al\. place MoE within a broader landscape of sparse attention, state\-space models, and hybrid architectures\[[3](https://arxiv.org/html/2608.08650#bib.bib3)\]\. Rather than repeating a model\-by\-model chronology, this report asks a structural question:which bottleneck does each architectural change remove, and where does the bottleneck move next? The scope of the discussion is limited to sparse MoE in decoder\-only LLM pre\-training, focusing on covering FFN MoE\. Expert Parallel \(EP\), All\-to\-All, token capacity and expert placement, which will adversely affect structure selection, are also discussed\. Multimodal MoE, MoE\-LoRA, external model ensemble and pure post\-training expert fusion are outside the scope of this article\. Model capability numbers are only used to illustrate architecture scalability and cannot be used as causal comparisons across papers: training data, number of tokens, optimizers, context length, post\-training and evaluation pollution control are often not consistent\. ## 2 Unified formalization: What does MoE simultaneously optimize? ### 2\.1 Token\-choice MoE Given the hidden representationxt∈𝐑dx\_\{t\}\\in\\mathbf\{R\}^\{d\}of thettth token, Router calculates expert affinity: si,t=ϕ\(xt,ei\),𝒦t=TopKi∈\{1,…,N\}\(si,t\),s\_\{i,t\}=\\phi\(x\_\{t\},e\_\{i\}\),\\qquad\\mathcal\{K\}\_\{t\}=\\operatorname\{TopK\}\_\{i\\in\\\{1,\\ldots,N\\\}\}\(s\_\{i,t\}\),\(1\)Among them,NNis the number of routed experts, andeie\_\{i\}is the expert routing embedding or Router weight\. The output is yt=xt\+∑i∈𝒦tgi,tEi\(xt\),gi,t=exp\(si,t\)∑j∈𝒦texp\(sj,t\)\.y\_\{t\}=x\_\{t\}\+\\sum\_\{i\\in\\mathcal\{K\}\_\{t\}\}g\_\{i,t\}E\_\{i\}\(x\_\{t\}\),\\qquad g\_\{i,t\}=\\frac\{\\exp\(s\_\{i,t\}\)\}\{\\sum\_\{j\\in\\mathcal\{K\}\_\{t\}\}\\exp\(s\_\{j,t\}\)\}\.\(2\)If there are always\-on shared experts, add∑jEjshared\(xt\)\\sum\_\{j\}E^\{\\mathrm\{shared\}\}\_\{j\}\(x\_\{t\}\)to the formula \([2](https://arxiv.org/html/2608.08650#S2.E2)\)\. From a formula perspective, MoE is just a sparse function combination; from a system perspective,𝒦t\\mathcal\{K\}\_\{t\}determines which devices the token spans, how many tokens each card gets, and the shape of each local GEMM\. ### 2\.2 Three goals of capacity, quality and system efficiency MoE is not a single\-objective optimization\. Roughly speaking, the model hopes to maximize the total capacityPtotalP\_\{\\mathrm\{total\}\}while controlling the single token activation amountPactiveP\_\{\\mathrm\{active\}\}: Pactive≈Pdense\+kNProuted\+Pshared,P\_\{\\mathrm\{active\}\}\\approx P\_\{\\mathrm\{dense\}\}\+\\frac\{k\}\{N\}P\_\{\\mathrm\{routed\}\}\+P\_\{\\mathrm\{shared\}\},\(3\)However, the formula \([3](https://arxiv.org/html/2608.08650#S2.E3)\) is only an approximation of the parameter caliber: attention, embedding, shared experts, different expert widths and frame statistics methods will all cause deviations\. On the other hand, let the number of tokens received by expertiiin a statistical window benin\_\{i\}, then the load variation coefficient is CVexpert=1N∑i\(ni−n¯\)2n¯\.\\operatorname\{CV\}\_\{\\mathrm\{expert\}\}=\\frac\{\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i\}\(n\_\{i\}\-\\bar\{n\}\)^\{2\}\}\}\{\\bar\{n\}\}\.\(4\)Low CV benefits Expert Parallel throughput but does not necessarily indicate useful knowledge specialization\. The architectural tension is thatsemantic specialization may be uneven, while physical execution must avoid severe stragglers\. Table 1:Six questions that determine MoE architectural differences ### 2\.3 Four control planes: object, decision, control and execution The design questions in Table[1](https://arxiv.org/html/2608.08650#S2.T1)operate at different levels\. To separate historical architectural evolution from the internal operation of one MoE system, we organize the system cross\-section bystate variables, decision granularity, and update timescaleinto four control planes\. Topology is a slowly changing model structure; Routing makes discrete token\-level decisions; Balance aggregates statistics and applies feedback over groups of tokens; and Expert Parallelism \(EP\) realizes logical decisions as communication and kernels on physical devices\. Chapter 3 follows bottleneck migration over time, whereas Chapters 4–7 analyze these four control planes\.The control planes are not four additional stages of evolution\. Formally, the four\-level relationship can be written as ℰ\\displaystyle\\mathcal\{E\}=T\(θtopo\),\\displaystyle=T\(\\theta\_\{\\mathrm\{topo\}\}\),Topology: construct the expert set, groups, and sharing relations;\\displaystyle\\text\{Topology: construct the expert set, groups, and sharing relations\};𝒦t\\displaystyle\\mathcal\{K\}\_\{t\}=R\(xt,ℰ;θroute,b\),\\displaystyle=R\(x\_\{t\},\\mathcal\{E\};\\theta\_\{\\mathrm\{route\}\},b\),Routing: select an expert subset for tokent;\\displaystyle\\text\{Routing: select an expert subset for token \}t;ni\\displaystyle n\_\{i\}=∑t𝟏\[i∈𝒦t\],\(b,α,c\)←C\(\{ni\},π\),\\displaystyle=\\sum\_\{t\}\\mathbf\{1\}\[i\\in\\mathcal\{K\}\_\{t\}\],\\quad\(b,\\alpha,c\)\\leftarrow C\(\\\{n\_\{i\}\\\},\\pi\),Balance: update bias, loss, or capacity from aggregate load;\\displaystyle\\text\{Balance: update bias, loss, or capacity from aggregate load\};yt\\displaystyle y\_\{t\}=EP\(xt,𝒦t,π,σ\),\\displaystyle=\\operatorname\{EP\}\\\!\\left\(x\_\{t\},\\mathcal\{K\}\_\{t\},\\pi,\\sigma\\right\),EP: execute dispatch and combine under placementπand scheduleσ\.\\displaystyle\\text\{EP: execute dispatch and combine under placement \}\\pi\\text\{ and schedule \}\\sigma\.\(5\)Among them,ℰ\\mathcal\{E\}is the logical expert set,𝒦t\\mathcal\{K\}\_\{t\}is the routing result of the token,nin\_\{i\}is the load of expertiiin the statistics window,bb,α\\alpha, andccrepresent Router bias, auxiliary loss intensity and capacity respectively\. Control volume,π\\piis the mapping from expert to device, andσ\\sigmais communication and computing scheduling\. Formula \([5](https://arxiv.org/html/2608.08650#S2.E5)\) explains: The four layers are not independent modules, but a closed loop\. Topology object layerdefinitionℰ\\mathcal\{E\}with candidate constraintsRouting decision\-making leveloutput𝒦t\\mathcal\{K\}\_\{t\}Expert Parallel Execution layeraccording toπ,σ\\pi,\\sigmaoutputyty\_\{t\}Balance control layerstatisticsnin\_\{i\}and device/node/communication loadfeedback:loss / bias / capacity→\\rightarrowRouting;placement / replication→\\rightarrowEPFigure 1:The closed\-loop relationship between Topology, Routing, Balance and Expert Parallel\. The first three solid lines form the forward execution chain, and the dashed lines represent the feedback of statistics and system costs on routing and placement\.Table 2:Criteria for the division of the four control planesThere are two types of reverse dependencies between the four layers that cannot be ignored\. First, the physical topology will constrain semantic routing: in order to reduce fan\-out, device\-limited routing actively reduces the candidate set visible to a certain token inℰ\\mathcal\{E\}\. Second, Balance does not just tune the Router: runtime replication or expert placement can improve the physical load without changing𝒦t\\mathcal\{K\}\_\{t\}\. Therefore, ”load balancing” cannot only be understood as an auxiliary loss, and ”Expert Parallel” is not a passive implementation that is intervened after the model structure is determined\. ## 3 Eight Evolutionary Milestones: Criteria, Mainline, and Branches The eight milestones are neither a fixed taxonomy copied from one survey nor a mechanical division by year\. They are an analytical summary based onmigration of the dominant bottleneck\. A change qualifies as a milestone only if it satisfies three criteria\. First, it introduces an independently adjustable design variable, such as Top\-kk, expert granularity, a shared path, or a dynamic active budget\. Second, it moves the system’s principal bottleneck, for example from compute growing linearly with the number of experts, to discrete routing and load imbalance, and then to All\-to\-All communication, small GEMMs, or weight I/O\. Third, later architectures inherit the change, so it is more than a one\-off implementation detail\. According to this standard, node 1–6 forms a clearer historical backbone: statistical division of labor→\\rightarrowsparse activation→\\rightarrowTransformer scale→\\rightarrowdecoder\-only productization→\\rightarrowfine\-grained knowledge organization→\\rightarrowultra\-sparse capacity expansion\. Node 7 and node 8 are not simply new ”generations” that follow node 6: Node 7 relaxes the assumption that ”each token has a fixed amount of calculation”; node 8 relaxes the assumption that ”semantic routing, intra\-layer expert topology and physical communication must be bound”\. Both can be combined with the expert structure of node 5 or 6\. Figure[2](https://arxiv.org/html/2608.08650#S3.F2)shows this inheritance relationship\. 1 Statistical division of laborDense/Soft MoE2 sparse activationTop\-kkconditional compute3 Cluster scalingTransformer \+ EP4 Productizationopen decoder MoE5 Knowledge segmentationFine\-grained \+ shared6 Capacity continues to expandUltra\-sparse scaling7 Calculating Budget Dynamicsadaptive\-kk/ zero\-compute experts8 Semantic and physical decouplingScMoE / Heterogeneous / Cross\-layer / structured communicationFigure 2:The relationship between eight evolution nodes\. Solid lines represent primary inheritance on the historical trunk, and dashed lines represent stackable orthogonal branches; node numbers do not represent strict generational replacement\.Table 3:Division basis of eight nodes and bottleneck migration### 3\.1 Dense/Soft MoE: Statistical division of labor rather than computational sparsity Jacobs et al\. proposed the basic form\[[4](https://arxiv.org/html/2608.08650#bib.bib4)\]of gating network and local experts in 1991\. What this node establishes isstatistical division of labor: gate generates mixed weights according to the input, different sub\-networks fit different areas of the input space, and the training goals can promote specialization\. Since all experts usually participate in weighting, routing is continuously differentiable, requiring neither token capacity nor sparse dispatch in the All\-to\-All sense\. Why is this a separate milestone?It established three roles retained by later MoE systems: a router or gate, a set of experts, and a weighted combination operator\.Why is it not yet a modern sparse MoE?Increasing the number of experts still increases compute approximately linearly, so model capacity and FLOPs per token remain coupled\. The next milestone preserves statistical specialization while executing only a small subset of experts, enabling conditional scaling but introducing optimization and systems problems associated with discrete selection\. ### 3\.2 Sparse conditional computation: Top\-k establishes capacity leverage Shazeer et al\. applied noisy Top\-kkrouting to very large sparse networks\[[5](https://arxiv.org/html/2608.08650#bib.bib5)\], so that only a few experts execute for each input\. If the total number of experts isNNand each input activates onlyk≪Nk\\ll Nexperts, total capacity can grow withNNwhile the dominant expert FLOPs scale approximately withkk\. This establishes the central capacity lever of modern MoE:parameter capacity is decoupled from per\-token compute\. This decoupling is not free\. Top\-kkcauses unselected experts to have no gradient from this token; popular experts will overflow capacity, and unpopular experts may not be trained for a long time; dispatch/combine must also be added for cross\-device execution\. Therefore, noisy routing, importance/load auxiliary loss, capacity factor and token drop are not peripheral techniques, but supporting mechanisms induced by sparse execution itself\. Node 3 does not change this basic algorithm, but answers: how to make MoE stable and executable when it is repeatedly embedded in Transformer layers and scaled to thousands of devices\. ### 3\.3 Transformer MoE and Expert Parallel GShard systematically embeds Top\-2 MoE FFN into Transformer and relies on automatic sharding to trainon 2048 TPU multi\-language model\[[6](https://arxiv.org/html/2608.08650#bib.bib6)\]with more than 600B\. The classic execution chain is thus fixed as Attention→\\rightarrowRouter→\\rightarrowDispatch All\-to\-All→\\rightarrowExpert FFN→\\rightarrowCombine All\-to\-All\. The new variable of this node is not ”more experts”, butexpert parallelism and cluster execution semantics: how tokens are rearranged across devices, how each expert is batch\-processed, how overflows are handled, and how communication and calculation are synchronized\. Switch Transformer further adopts Top\-1, trading lower communication and simpler execution for expert combination capabilities and routing fault tolerance reduction\[[7](https://arxiv.org/html/2608.08650#bib.bib7)\]\. ST\-MoE elevates stability to a first\-class design goal and introduces Router z\-loss constraint logits numerical scale\[[8](https://arxiv.org/html/2608.08650#bib.bib8)\]\. It needs to be distinguished: z\-loss controls numerical stability, and balance loss controls usage distribution\. The two are not the same mechanism\. During the same period, BASE Layers wrote the training route as a strictly balanced linear allocation problem\[[9](https://arxiv.org/html/2608.08650#bib.bib9)\], while Expert Choice allowed experts to choose fixed capacity tokens\[[10](https://arxiv.org/html/2608.08650#bib.bib10)\]\. They can directly guarantee equilibrium, but batch\-level joint allocation and expert\-side capacity are not as natural as token\-choice for online autoregressive decoding, so they did not replace the Top\-kkbackbone of decoder\-only LLM\. The legacy of node 3 is the complete system contract of ”Router \+ EP \+ capacity/balance \+ All\-to\-All”; nodes 4 and 5 both follow it, but shift the focus from ”can large\-scale training” to model quality, deployment availability and expert internal organization\. ### 3\.4 Open\-Weight Coarse\-Grained MoE Mixtral uses8×\\times7B parameters with Top\-2 routing per token\[[11](https://arxiv.org/html/2608.08650#bib.bib11)\]\. It did not introduce a new router family and therefore is not a milestone purely in terms of algorithmic novelty\. Its importance is different: it demonstrated thata conventional coarse\-grained MoE can support pretraining, instruction tuning, inference deployment, and community reproductionin an open\-weight decoder\-only LLM\. Why is this aseparate milestone? Node 3 demonstrates scalable training on very large clusters; node 4 establishes an end\-to\-end path from pretraining and instruction tuning to practical deployment for a general\-purpose decoder model\. It inherits Top\-kk, isomorphic experts, and EP without changing the basic execution chain\. Once the system can run reliably, the main bottleneck shifts to knowledge organization: a small number of large experts tend to relearn common capabilities, while the Router can compose only coarse knowledge blocks\. Node 5 addresses this bottleneck directly\. ### 3\.5 Shared \+ fine\-grained experts DeepSeekMoE makes two key changes to coarse\-grained experts\[[12](https://arxiv.org/html/2608.08650#bib.bib12)\]\. First, it splits a large FFN into multiple smaller experts, allowing the Router to compose more knowledge units under the same active budget\. Second, shared\-expert isolation moves knowledge needed by all tokens onto an always\-on path, reducing redundancy among routed experts\. The former increasescomposition resolution, while the latter explicitly separates common and conditional capabilities\. DeepSeek\-V2 scales this structure to236B total parameters and 21B active parameters, and adds device\-limited routing: each token’s candidate experts are restricted to a small number of devices, thereby controlling cross\-device fan\-out\[[13](https://arxiv.org/html/2608.08650#bib.bib13)\]\. Shared or dense paths can also provide computational windows that hide communications\. DeepSeek\-V2 combines shared expert computation with EP communication overlap; Snowflake Arctic’s dense residual path also reflects the similar algorithm – system collaboration\[[14](https://arxiv.org/html/2608.08650#bib.bib14)\]\. Butshared expert is not a necessary condition for fine\-grained MoE\. Qwen3\-235B\-A22B uses128 fine\-grained experts and Top\-8, but removes the shared expert and relies on global\-batch balance to allow commonly used capabilities to naturally form\[[15](https://arxiv.org/html/2608.08650#bib.bib15)\]among routed experts\. Therefore, node 5 actually contains two divisible axes: expert granularity and shared path\. They are often combined, but there is no logical binding\. Node 5 inherits the EP system of node 3, but increases the number of experts, Top\-kk, and device fan\-out at the same time, making small GEMM, routing fragmentation, and communication topology new bottlenecks\. Node 6 continues to expand along the lines of ”more and smaller experts”; Node 8 attempts to change the physical implementation of fan\-out and communication\. ### 3\.6 Ultra\-sparse scaling The question at this stage is:*with active parameters fixed, does increasing the total number of experts continue to reduce loss?*PEER uses product\-key retrieval to select a small subset frommore than one million tiny experts\[[16](https://arxiv.org/html/2608.08650#bib.bib16)\]\. Kimi K2 scales an engineering\-oriented ultra\-sparse design to1\.04T total parameters, 32\.6B active parameters, 384 routed experts, and Top\-8 routing, and reports scaling gains from higher sparsity at a fixed active budget\[[17](https://arxiv.org/html/2608.08650#bib.bib17)\]\. GPT\-OSS\-120B uses128 experts with Top\-4 routingand a relatively small active footprint, illustrating an ultra\-sparse design intended for local deployment\[[18](https://arxiv.org/html/2608.08650#bib.bib18)\]\. Why separate ultra\-sparse scaling from fine\-grained experts?Fine\-grained experts primarily reduce knowledge redundancy and increase composition resolution\. Ultra\-sparse scaling instead asks whether a larger candidate expert set continues to improve loss when active parameters and training FLOPs are fixed\. Both increase expert count, but they optimize different objectives\. Sparsity cannot increase without bound\.AsNNgrows, router scoring, expert\-weight I/O, All\-to\-All metadata, and small\-GEMM fragmentation all increase\. When each expert receives too few tokens per step,theoretical FLOP savings no longer translate into wall\-clock gains\. Ultra\-sparse scaling therefore depends more heavily on fine\-grained organization and routing control rather than replacing them\. The next two milestones relax different constraints of the fixed Top\-kkmainline\. ### 3\.7 Dynamic compute: from fixed Top\-k to token\-dependent budget Fixed Top\-kkimplies that every token receives the same number of expert FLOPs even when tokens differ in semantic difficulty and required capacity\. LongCat\-Flash introduces zero\-computation experts: the router still selects a fixed number of slots, but some slots are identities, so actual FFN compute varies by token\[[19](https://arxiv.org/html/2608.08650#bib.bib19)\]\. Adaptive\-kk, threshold routing, and zero\-compute slots share the objective ofdecoupling which experts are selected from how much computation is allocated\. Relationship to ultra\-sparse scaling\.Ultra\-sparse scaling expands the candidate capacityNNwhile usually keepingkkfixed; dynamic computation changes the actualktk\_\{t\}, or the effective FFN compute, for each token\. The two approaches can be combined: the candidate pool remains large, while easy tokens receive less compute and difficult tokens receive more\. The control objective consequently expands from expert load alone to the mean active budget, budget variance, per\-expert load, and P99 latency\. Constraining average training FLOPs does not guarantee tail latency because difficult tokens may cluster within a micro\-batch or request window\. The remaining issue of dynamic computing is closed\-loop control: if the Router determines both the semantic expert and the computing budget, the balance bias affects both specialization and throughput; if only the long\-term average is constrained, short\-term bursts may occur\. Therefore, adaptive bias/PID feedback, budget loss and runtime admission control are supporting issues and cannot be solved by simply modifying Top\-kk\. ### 3\.8 Semantic routing decoupled from physical execution Node 1–7 mainly changes Router’s optional expert or active budget; node 8 changes the lower\-level assumption:semantically selecting an expert does not mean that it must be executed immediately along the fixed intra\-layer All\-to\-All path\. This node is not a single model family, but four frontier branches with a common goal\. Execution\-dependency rearrangement\.ScMoE does not change which experts a token selects; it changes the dependency graph\. Its cross\-layer shortcut lets routed experts consume an intermediate representation from the preceding layer while the current dense/shared path overlaps the dispatch/combine communication window\[[20](https://arxiv.org/html/2608.08650#bib.bib20)\]\. It reduces exposed communication but does not directly solve routing skew\. Figure[3](https://arxiv.org/html/2608.08650#S3.F3)gives an abstract comparison\. Attention / shortcutDispatchRouted expertsCombineMergeDense/shared branchcommunication path with dense Computational parallelismFigure 3:At its core, ScMoE is about rearranging execution dependencies to expand the communication hidden window; it is not a new routing algorithm\.LongCat\-Flash combines ScMoE with token\-dimension chunking and Single Batch Overlap; the official LongCat\-2\.0 report further adopts fully parallel dense/MoE execution on each core of a dedicated accelerator\[[21](https://arxiv.org/html/2608.08650#bib.bib21)\]\. These developments reduce exposed communication,but they do not automatically eliminate severe routing skew, heterogeneous expert runtimes, or GPU\-memory pressure\. Heterogeneous experts\.MoHGE uses two\-level routing: it first selects an expert group and then experts of different sizes within that group, together with group\-wise and intra\-group balancing constraints\[[22](https://arxiv.org/html/2608.08650#bib.bib22)\]\. Equal expert width becomes a design variable rather than a default constraint, allowing different tokens to receive different forms of compute\. The cost is greater complexity in heterogeneous GEMMs, capacity planning, and load metrics\. Cross\-layer sharing\.GMoE designates some experts as cross\-layer Global Experts while retaining layer\-local experts, reducing redundant capabilities across layers\[[23](https://arxiv.org/html/2608.08650#bib.bib23)\]\. This differs from a conventional shared expert in scope: the former is reused across layers, whereas the latter is typically an always\-on path within one layer\. Cross\-layer sharing introduces additional caching, scheduling, and inter\-layer interference concerns\. Structured communication\.Multi\-Head LatentMoE and Head Parallel seek a topology whose communication volume isO\(1\)O\(1\)with respect to the number of activated expertskk, yielding more predictable traffic\[[24](https://arxiv.org/html/2608.08650#bib.bib24)\]\. Whereas ScMoE hides dynamic communication, this route attempts to reduce or structure the communication itself\. In the absence of public validation at trillion\-parameter and ten\-thousand\-accelerator scale,it should be treated as a research frontier, not as a mainstream replacement for Top\-kkexpert parallelism\. The four branches respectively optimize temporal overlap, computation structure, parameter reuse, and data movement\. They are complementary and can be combined\. ## 4 Expert topology: object layer In the four\-layer framework introduced in Chapter 2, Topology is the object layer\. It defines the expert setℰ\\mathcal\{E\}, the granularity of each knowledge unit, the sharing scope, and computational heterogeneity before Routing makes token\-level selections\. The central question is therefore not simply how many experts exist, butwhat kinds of computational units the router can compose\. It is not sufficient to understand the evolution of MoE only as the growth of expert count\. Table[4](https://arxiv.org/html/2608.08650#S4.T4)shows that what really changes is the granularity, sharing scope and computing heterogeneity of knowledge units\. Table 4:The main evolution route of Expert topologyAnother dimension that is often overlooked is the initialization method\. Sparse Upcycling copies the existing dense FFN into multiple experts, and then adds a new Router to continue training\[[25](https://arxiv.org/html/2608.08650#bib.bib25)\]\. It reuses dense checkpoints, but brings expert symmetry\. It requires routing noise, data differences and subsequent training to differentiate the experts\. Further Expert Upcycling is extended from the existingEE\-expert MoE tomEmEexperts, while keeping Top\-kkand active cost unchanged\[[26](https://arxiv.org/html/2608.08650#bib.bib26)\]\. Upcycling is not a new Router family, but it will change the formation path and training cost of specialization\. ## 5 Routing: decision\-making layer In the closed loop of Figure[1](https://arxiv.org/html/2608.08650#S2.F1), Routing receives the expert set defined by Topology and the candidate constraints imposed by the system, and generates𝒦t\\mathcal\{K\}\_\{t\}for each token\. It optimizeslocal semantic matching; it does not directly guarantee a reasonable aggregate load over a sequence, batch, or device\. The latter belongs to the Balance control plane discussed in Section 6\. Table 5:Main Router families and their applicable boundariesThe fact that Token\-choice has become mainstream does not mean that it is optimal in terms of optimization, but that it is the most consistent with the autoregressive online constraints: the routing of the current token only relies on the current state, and there is no need to wait for future tokens or solve the global distribution of the entire batch\. Continuous routing schemes such as Soft MoE and Lory avoid discrete Top\-kk, but the system advantages have not yet been proven on ultra\-large\-scale decoder and low\-latency EP\[[27](https://arxiv.org/html/2608.08650#bib.bib27),[28](https://arxiv.org/html/2608.08650#bib.bib28)\]\. Routing freedom is also constrained by network topology\. Device\-limited or node\-limited routing limits candidate experts to a small number of devices/nodes to reduce single\-token communication fan\-out\. The trade\-off is that the semantically optimal expert may be excluded by topological constraints, so this type of design is inherently an explicit trade\-off between quality and communication\. ## 6 Balance: control layer Balance does not redefine experts, nor does it independently generate semantic routes for tokens\. It observes the\{ni\}\\\{n\_\{i\}\\\}and equipment, nodes and communication loads accumulated from many routing decisions, and then feeds back to the decision\-making and execution layers through loss, bias, capacity or runtime placement\. The core problem is: whileallows uneven semantic specialization, how to avoid training collapse and physical straggler\. ### 6\.1 Sequence/batch\-wise expert balance The classic auxiliary loss can be written as ℒbal=αN∑i=1NfiPi,\\mathcal\{L\}\_\{\\mathrm\{bal\}\}=\\alpha N\\sum\_\{i=1\}^\{N\}f\_\{i\}P\_\{i\},\(6\)Among them,fif\_\{i\}is the token fraction actually assigned to expertiiin the statistical window,PiP\_\{i\}is the average routing probability, andα\\alphacontrols the equalization intensity\. When all experts use approximately uniform values, the formula \([6](https://arxiv.org/html/2608.08650#S6.E6)\) is smaller\. This loss is responsible for both preventing expert collapse and avoiding EP straggler, but they are not equivalent: the model may require uneven specialization, but the hardware hopes to have an even load\. Whenα\\alphais too large, the hardware target directly interferes with the language modeling gradient\. ### 6\.2 Device/node/communication\-aware balance DeepSeek\-V2 splits the balance into expert\-level, device\-level and communication balance, and controls fan\-out\[[13](https://arxiv.org/html/2608.08650#bib.bib13)\]through device\-limited routing\. The key to this change is not to add a few more losses, but to acknowledge thatlogical expert balance is not equal to physical device or network balance: If there are multiple experts on a device, as long as the total tokens between devices are close, it is not necessarily necessary that all experts are completely equal\. ### 6\.3 Auxiliary\-Loss\-Free Load Balancing The Auxiliary\-Loss\-Free \(ALF\) strategy adds non\-gradient routing biasbib\_\{i\}to each expert: 𝒦t=TopKi\(si,t\+bi\),bi←bi\+ηsign\(n¯−ni\),\\mathcal\{K\}\_\{t\}=\\operatorname\{TopK\}\_\{i\}\(s\_\{i,t\}\+b\_\{i\}\),\\qquad b\_\{i\}\\leftarrow b\_\{i\}\+\\eta\\,\\operatorname\{sign\}\(\\bar\{n\}\-n\_\{i\}\),\(7\)Whereη\\etais the bias update speed\. The bias affects the Top\-kkselection, but does not participate in the LM gradient as an affinity weight, thereby reducing the direct interference of the equilibrium target on the model parameter update direction\[[29](https://arxiv.org/html/2608.08650#bib.bib29)\]\. DeepSeek\-V3 uses batch\-wise ALF as the main balancing mechanism, while retaining a very weak sequence\-wise auxiliary loss to prevent extreme single\-sequence imbalance\[[30](https://arxiv.org/html/2608.08650#bib.bib30)\]\. Therefore,”DeepSeek\-V3 has no balance loss at all” is not accurate\. Global\-batch statistics are more stable than single micro\-batch, and also allow some experts to carry more specific domain tokens on local batches; the cost is that load needs to be aggregated across DP/EP ranges, and expanding the statistical range may introduce additional communication\. ### 6\.4 Runtime expert placement Balance on the training distribution cannot guarantee true inference traffic\. Domain shift, SFT/RL and off balance updates all change expert popularity\. DeepSeek\-V3 uses redundant expert deployment, replicates hotspot experts and adjusts placement\[[30](https://arxiv.org/html/2608.08650#bib.bib30)\]at runtime\. This trend re\-partitions the goals: Routers can preserve semantically non\-uniform specialization, and runtimes avoid device stragglers by copying, migrating, or relocating them\. Sequence/Batchexpert balanceDevice/Node/CommunicationGlobal\-batchALF biasRuntime replicationand placementFigure 4:The evolution of Balance is not to simply weaken the constraints, but to gradually separate the physical execution goals from the LM gradient\. ## 7 Expert Parallel: Execution Layer EP consumes the token–expert assignment generated by Routing, and combines it with placementπ\\pito convert the logical expert into dispatch, local GEMM and combine on the device\. It usually does not change ”who is semantically selected”, but determines the wall\-clock cost of this selection; when the system cost is too high, the first three layers will be reversely modified through device\-limited routing, placement or architectural reconstruction\. The algorithmic benefits of MoE only hold true if the sparse execution is efficient\. The capacity\-based system reserves fixed token slots for each expert, and overflow tokens are discarded or used as residual, which can easily cause inconsistency between training and inference\. MegaBlocks uses block\-sparse kernels to implement dropless MoE, so that dynamic token shapes no longer require token dropping\[[31](https://arxiv.org/html/2608.08650#bib.bib31)\]\. Systems such as Tutel dynamically select parallel and pipeline strategies to adapt to different cluster sizes\.\[[32](https://arxiv.org/html/2608.08650#bib.bib32)\]\. As experts become thinner and Top\-kkbecomes larger, communication gradually becomes a first\-class architectural constraint, driving three processing methods: 1. 1\.Limited communication topology: device/node\-limited routing reduces fan\-out; 2. 2\.Hidden communication: shared computation overlap, DualPipe, ScMoE and token chunking expand the overlap window; 3. 3\.Reconstruct communication: Head Parallel and other solutions turn dynamic traffic into a more deterministic communication mode\. Heremust distinguish between ”total All\-to\-All time” and ”exposed communication time”\. If the communication is completely covered by the dense path, the marginal benefit of continuing to optimize the link bandwidth on the end\-to\-end step time will decrease; conversely, even if the total communication volume remains unchanged, as long as the critical path is shortened, the throughput may be significantly improved\. Therefore, the training report should at least give thetraffic volume, overlap ratio, exposed dispatch/combine, MFU and step\-time P99\. ## 8 The Modern Mainline and Frontier Branches As of 2026, the most common combinations of large\-scale decoder MoE can be summarized as: Token\-choice Top\-kk\+ fine\-grained experts \+ optional shared path \+ global/batch\-wise balance or ALF \+ topology\-limited routing \+ dropless kernels/communication overlap \+ runtime expert placement DeepSeek\-V3, Qwen3 and Kimi K2 differ in shared expert, balance mechanism and EP size, but they are all located on this backbone\. At the same time, the following branches have not yet reached a unified conclusion: - •MoE Attention: JetMoE sparses Attention and FFN at the same time, further reducing active FLOPs, but KV/cache and routing are more complex\[[33](https://arxiv.org/html/2608.08650#bib.bib33)\]; - •Hybrid mixer \+ MoE: Jamba alternates between Transformer and Mamba layers, indicating that MoE is a parameter expansion method and does not bind the full Attention backbone\[[34](https://arxiv.org/html/2608.08650#bib.bib34)\]; - •Million experts: High theoretical composability, but weighted I/O, small GEMM and retrieval overhead still limit industrial deployment; - •Heterogeneous experts: Choose different capacities according to token difficulty, with natural expression, but physical placement and tail delay are more difficult; - •Cross\-layer global experts: The goal is to eliminate inter\-layer duplication capabilities, but there is still a lack of evidence for very large\-scale training; - •Structured communication: Head Parallel may reduce the pain point of dynamic EP, but it is still in the early verification stage\. Table[6](https://arxiv.org/html/2608.08650#S8.T6)compares representative models based on architecture selection rather than benchmark ranking\. Table 6:Architectural location of representative LLM MoE; active parameter size varies by report ## 9 Experimental Design for Foundation\-Model Pretraining cross\-model benchmark cannot identify architectural contributions\.To determine whether fine\-grained, shared, ALF or ScMoE is effective, you should at least fix thetraining token, data ratio, active parameters and optimizerat the same time, and reporttraining\-FLOPs\-matched and wall\-clock\-matchedresult\. Table 7:Proposed equal\-budget MoE ablation matrixFor code pre\-training scenarios, it is recommended to additionally bucket statistics by language, repo/file/function granularity, code and natural language token routing\. Key observations include: expert co\-activation in different languages, whether cross\-file reference tokens are concentrated in a small number of experts, whether code and code\-related documents share experts, and whether long repo\-level sequences cause sequence\-wise load peaks\. Only in this way can we distinguish between ”experts forming beneficial specialization” and ”data matching causing accidental traffic skew”\. The system side should at least record at the same time: - •max/mean, CV and Gini at the three levels of expert, device and node; Load under the three statistical windows of - •sequence, micro\-batch, and global batch; - •dispatch/combine exposed time, not just the total All\-to\-All time; - •token drop, capacity overflow, redundant expert hit rate; - •training FLOPs, active parameters, wall\-clock and cluster network topology\. ## 10 Open Questions ### 10\.1 What is the correct unit of analysis for Specialization? ”Mathematical expert” or ”code expert” are intuitive narratives, but actual routing is often represented by a combination of token morphology, syntax, location, language, and hierarchy\. A single expert may not be a stable semantic unit, and a cross\-layer expert group or co\-activation pattern may be more interpretable\. In the future, routing mutual information, expert representation similarity and intervention\-based causal test should be combined instead of just showing token word cloud\. ### 10\.2 Where Is the Systems Optimum at Higher Sparsity? Increasing total experts under fixed active FLOPs may reduce model loss, but system benefits are limited by network bandwidth, expert weight I/O, batch size and kernel arithmetic intensity\. The optimal sparsity is not a pure model constant, but the model – hardware joint scaling law\. The EP size, network topology, and number of tokens per expert must be given when reporting this conclusion\. ### 10\.3 Can Load Balancing Move Primarily to Runtime? ALF reduces the gradient interference of the auxiliary loss, but still changes the Top\-kkselection\. Runtime replication and placement can allow more freedom in semantic routing, but will bring parameter copy memory, migration costs and cache consistency issues\. The optimal boundary between weak constraints in training and strong equilibrium at runtime remains undetermined\. ### 10\.4 How Dynamic Compute Meets Tail Latency SLAs Zero experts or adaptive\-kkcan reduce average FLOPs, but difficult tokens will create new stragglers if they are in the same micro\-batch or request window\. The model needs to control average budget, budget variance, and P99 simultaneously, rather than just reporting average active parameters\. ### 10\.5 How Can Post\-Training Preserve Router Stability? SFT/RL data will change the domain distribution and sequence shape, thereby changing the expert load\. Whether to retain balance updates, routing replay, and freeze Routers during the training phase cannot be directly extrapolated from the pre\-training settings\. Routing drift and specialization retention should be measured separately in the pretrain, SFT and RL stages\. ## 11 Summary and Outlook The architectural history ofLLM MoE is not a single\-line expansion of ”more and more experts”, but the co\-evolution of five routes:expert has changed from a small number of large modules to fine\-grained, ultra\-sparse and heterogeneous computing units; the topology has developed from layer\-local routed experts to shared, cross\-layer global/local and latent\-head structures; Router has evolved from simple Top\-kkgradually adds global statistics, bias control and dynamic computing budget; balance moves from sequence\-wise auxiliary loss to device/communication\-aware, ALF and runtime placement; execution develops from serial All\-to\-All to dropless kernels, topology\-limited routing, ScMoE overlap and structured communication\. Therefore, there is no unique ”next\-generation structure” after ScMoE\. A more credible direction is a combination of four: freer semantic routing, explicit average computing budget control, topology awareness or deterministic communication, and runtime expert placement\. Architecture evaluation must also be expanded from a single validation loss to a joint analysis of quality, specialization, active budget, exposed communication, and tail latency\. For the next round of large\-scale pretraining, the most informative experiment is not a direct comparison of final benchmark scores from two different recipes\. Instead, construct a controlled matrix withthe same data, token budget, and active FLOPs, then vary expert granularity, the shared path, balancing scope, and execution topology separately\.This separation is necessary to distinguish gains from parameter capacity, routing specialization, and systems throughput, and to derive MoE design conclusions that transfer to new hardware\. ## Appendix AArchitecture Evolution Cheat Sheet Table 8:Bottleneck migration of MoE architecture evolutionPeriodRepresentative workMain problems solvedNew bottlenecks transferred out1991–2016Adaptive MixturesLearning soft division of laborComputation increases linearly with the number of experts2017–2019Sparse MoEParameter capacity and single token FLOPs decouplingcollapse, capacity, All\-to\-All2020–2022GShard, Switch, ST\-MoETransformer Large\-scale sparse trainingStability, token drop, routing quality2023–2024MixtralOpen decoder MoE availabilityCoarse\-grained and knowledge redundant2024–2025DeepSeekMoE/V2, Qwen3Fine\-grained combination and public knowledge processingCommunication fan\-out, small GEMM, balance scope2024–2025PEER, Kimi K2, gpt\-ossultra\-sparse capacity expansionretrieval, I/O, low token/expert2024–2026ALF, LongCat, ScMoEReduce gradient interference and hidden communicationRuntime dynamic load and tail delay2025–2026MoHGE, GMoE, Head ParallelHeterogeneous computing, cross\-layer sharing, structured communicationUltra\-large\-scale quality and system verification ## References - \[1\]Cai W, et al\. A Survey on Mixture of Experts in Large Language Models\. IEEE Transactions on Knowledge and Data Engineering, 2025\.[https://arxiv\.org/abs/2407\.06204](https://arxiv.org/abs/2407.06204)\. - \[2\]Liu J, et al\. A Survey on Inference Optimization Techniques for Mixture of Experts Models\. arXiv:2412\.14219, 2024\.[https://arxiv\.org/abs/2412\.14219](https://arxiv.org/abs/2412.14219)\. - \[3\]Zhu X, et al\. Speed Always Wins: A Survey on Efficient Architectures for Large Language Models\. arXiv:2508\.09834, 2025\.[https://arxiv\.org/abs/2508\.09834](https://arxiv.org/abs/2508.09834)\. - \[4\]Jacobs R A, Jordan M I, Nowlan S J, Hinton G E\. Adaptive Mixtures of Local Experts\. Neural Computation, 3\(1\):79–87, 1991\.[doi:10\.1162/neco\.1991\.3\.1\.79](https://doi.org/10.1162/neco.1991.3.1.79)\. - \[5\]Shazeer N, et al\. Outrageously Large Neural Networks: The Sparsely\-Gated Mixture\-of\-Experts Layer\. ICLR, 2017\.[https://arxiv\.org/abs/1701\.06538](https://arxiv.org/abs/1701.06538)\. - \[6\]Lepikhin D, et al\. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\. ICLR, 2021\.[https://arxiv\.org/abs/2006\.16668](https://arxiv.org/abs/2006.16668)\. - \[7\]Fedus W, Zoph B, Shazeer N\. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\. Journal of Machine Learning Research, 23\(120\):1–39, 2022\.[https://arxiv\.org/abs/2101\.03961](https://arxiv.org/abs/2101.03961)\. - \[8\]Zoph B, et al\. ST\-MoE: Designing Stable and Transferable Sparse Expert Models\. arXiv:2202\.08906, 2022\.[https://arxiv\.org/abs/2202\.08906](https://arxiv.org/abs/2202.08906)\. - \[9\]Lewis M, et al\. BASE Layers: Simplifying Training of Large, Sparse Models\. ICML, 2021\.[https://arxiv\.org/abs/2103\.16716](https://arxiv.org/abs/2103.16716)\. - \[10\]Zhou Y, et al\. Mixture\-of\-Experts with Expert Choice Routing\. NeurIPS, 2022\.[https://arxiv\.org/abs/2202\.09368](https://arxiv.org/abs/2202.09368)\. - \[11\]Jiang A Q, et al\. Mixtral of Experts\. arXiv:2401\.04088, 2024\.[https://arxiv\.org/abs/2401\.04088](https://arxiv.org/abs/2401.04088)\. - \[12\]Dai D, et al\. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture\-of\-Experts Language Models\. ACL, 2024\.[https://arxiv\.org/abs/2401\.06066](https://arxiv.org/abs/2401.06066)\. - \[13\]DeepSeek\-AI\. DeepSeek\-V2: A Strong, Economical, and Efficient Mixture\-of\-Experts Language Model\. arXiv:2405\.04434, 2024\.[https://arxiv\.org/abs/2405\.04434](https://arxiv.org/abs/2405.04434)\. - \[14\]Snowflake\. Arctic: Snowflake’s Open\-Source LLM\. Official Technical Blog, 2024\. Accessed 2026\-08\-07\.[https://www\.snowflake\.com/en/blog/arctic\-open\-efficient\-foundation\-language\-models\-snowflake/](https://www.snowflake.com/en/blog/arctic-open-efficient-foundation-language-models-snowflake/)\. - \[15\]Qwen Team\. Qwen3 Technical Report\. arXiv:2505\.09388, 2025\.[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\. - \[16\]He X\. Mixture of A Million Experts\. arXiv:2407\.04153, 2024\.[https://arxiv\.org/abs/2407\.04153](https://arxiv.org/abs/2407.04153)\. - \[17\]Moonshot AI\. Kimi K2: Open Agentic Intelligence\. arXiv:2507\.20534, 2025\.[https://arxiv\.org/abs/2507\.20534](https://arxiv.org/abs/2507.20534)\. - \[18\]OpenAI\. gpt\-oss Model Card: Architecture\. 2025\. Accessed 2026\-08\-07\.[https://deploymentsafety\.openai\.com/gpt\-oss/architecture](https://deploymentsafety.openai.com/gpt-oss/architecture)\. - \[19\]LongCat Team\. LongCat\-Flash Technical Report\. arXiv:2509\.01322, 2025\.[https://arxiv\.org/abs/2509\.01322](https://arxiv.org/abs/2509.01322)\. - \[20\]Cai W, et al\. Shortcut\-connected Expert Parallelism for Accelerating Mixture\-of\-Experts\. arXiv:2404\.05019, 2024\.[https://arxiv\.org/abs/2404\.05019](https://arxiv.org/abs/2404.05019)\. - \[21\]LongCat Team\. LongCat 2\.0 Technical Blog\. 2026\. Accessed 2026\-08\-07\.[https://longcat\.ai/blog/longcat\-2\.0/](https://longcat.ai/blog/longcat-2.0/)\. - \[22\]MoHGE Authors\. MoHGE: Mixture of Heterogeneous Grouped Experts\. ACL Industry Track, 2026\.[https://aclanthology\.org/2026\.acl\-industry\.20/](https://aclanthology.org/2026.acl-industry.20/)\. - \[23\]GMoE Authors\. GMoE: Global Mixture\-of\-Experts\. ACL, 2026\.[https://aclanthology\.org/2026\.acl\-long\.2065/](https://aclanthology.org/2026.acl-long.2065/)\. - \[24\]Cui Y, et al\. Multi\-Head LatentMoE and Head Parallel\. arXiv:2602\.04870, 2026\.[https://arxiv\.org/abs/2602\.04870](https://arxiv.org/abs/2602.04870)\. - \[25\]Komatsuzaki A, et al\. Sparse Upcycling: Training Mixture\-of\-Experts from Dense Checkpoints\. arXiv:2212\.05055, 2022\.[https://arxiv\.org/abs/2212\.05055](https://arxiv.org/abs/2212.05055)\. - \[26\]Expert Upcycling Authors\. Expert Upcycling: Training Larger MoE Models from Smaller MoE Checkpoints\. arXiv:2604\.19835, 2026\.[https://arxiv\.org/abs/2604\.19835](https://arxiv.org/abs/2604.19835)\. - \[27\]Puigcerver J, et al\. From Sparse to Soft Mixtures of Experts\. ICLR, 2024\.[https://arxiv\.org/abs/2308\.00951](https://arxiv.org/abs/2308.00951)\. - \[28\]Zhong Z, et al\. Lory: Fully Differentiable Mixture\-of\-Experts for Autoregressive Language Model Pre\-training\. COLM, 2024\.[https://arxiv\.org/abs/2405\.03133](https://arxiv.org/abs/2405.03133)\. - \[29\]Wang Q, et al\. Auxiliary\-Loss\-Free Load Balancing Strategy for Mixture\-of\-Experts\. arXiv:2408\.15664, 2024\.[https://arxiv\.org/abs/2408\.15664](https://arxiv.org/abs/2408.15664)\. - \[30\]DeepSeek\-AI\. DeepSeek\-V3 Technical Report\. arXiv:2412\.19437, 2024\.[https://arxiv\.org/abs/2412\.19437](https://arxiv.org/abs/2412.19437)\. - \[31\]Gale T, et al\. MegaBlocks: Efficient Sparse Training with Mixture\-of\-Experts\. MLSys, 2023\.[https://arxiv\.org/abs/2211\.15841](https://arxiv.org/abs/2211.15841)\. - \[32\]Hwang C, et al\. Tutel: Adaptive Mixture\-of\-Experts at Scale\. MLSys, 2023\.[https://arxiv\.org/abs/2206\.03382](https://arxiv.org/abs/2206.03382)\. - \[33\]Shen Y, et al\. JetMoE: Reaching Llama2 Performance with 0\.1M Dollars\. arXiv:2404\.07413, 2024\.[https://arxiv\.org/abs/2404\.07413](https://arxiv.org/abs/2404.07413)\. - \[34\]Lieber O, et al\. Jamba: A Hybrid Transformer–Mamba Language Model\. arXiv:2403\.19887, 2024\.[https://arxiv\.org/abs/2403\.19887](https://arxiv.org/abs/2403.19887)\. \\EdefEscapeHex bilingual\.chinese\.0bilingual\.chinese\.0\\EdefEscapeHex中文版中文版\\hyper@anchorstartbilingual\.chinese\.0\\hyper@anchorend \\zihao 1LLM MoE 架构演进：从稀疏路由到动态计算与通信结构重构 \\zihao 3Expert Topology、Routing、Load Balance 与 Expert Parallel 的协同演化 Jiguo Li111本文在Codex协助下完成 [jiguolee@gmail\.com](https://arxiv.org/html/2608.08650v1/mailto:[email protected]) 2026 年 8 月 关键词：大语言模型；Mixture\-of\-Experts；稀疏路由；负载均衡；Expert Parallel；动态计算；ScMoE 阅读提要：第三章的八个节点描述历史上的瓶颈迁移；第四至七章则从 topology、routing、balance 和 Expert Parallel 四个控制面剖解单个 MoE 系统。前者是时间轴，后者是系统剖面，不能视为两套并列的“发展阶段”。 ## 目录 ## 附录 A问题定义与分析范围 经典 MoE 研究关注如何让 gating network 将样本分配给不同局部专家；LLM 时代的稀疏 MoE 则额外要求：总参数容量增长时，每个 token 的激活参数和计算量不能同比增长。现有三类综述分别提供互补视角：Cai 等人的 Survey 按算法、系统和应用建立全栈 taxonomy\[[1](https://arxiv.org/html/2608.08650#biba.bib1)\]；Liu 等人重点拆解推理阶段的模型、系统与硬件优化\[[2](https://arxiv.org/html/2608.08650#biba.bib2)\]; Zhu 等人则把 MoE 放回 sparse attention、SSM 和 hybrid architecture 的大图景中\[[3](https://arxiv.org/html/2608.08650#biba.bib3)\]。本文不重复逐篇枚举，而是追问：每一次结构变化消除了什么瓶颈，又把瓶颈转移到哪里？ 讨论范围限定为 decoder\-only LLM 预训练中的稀疏 MoE，重点覆盖 FFN MoE，同时讨论会反向影响结构选择的 Expert Parallel（EP）、All\-to\-All、token capacity 与 expert placement。多模态 MoE、MoE\-LoRA、外部模型 ensemble 和纯后训练专家融合不在本文范围内。模型能力数字只用于说明架构可扩展性，不能作为跨论文的因果比较：训练数据、token 数、优化器、上下文长度、post\-training 和评测污染控制往往并不一致。 ## 附录 B统一形式化：MoE 同时优化什么 ### B\.1 Token\-choice MoE 给定第tt个 token 的隐藏表示xt∈𝐑dx\_\{t\}\\in\\mathbf\{R\}^\{d\}，Router 计算 expert affinity： si,t=ϕ\(xt,ei\),𝒦t=TopKi∈\{1,…,N\}\(si,t\),s\_\{i,t\}=\\phi\(x\_\{t\},e\_\{i\}\),\\qquad\\mathcal\{K\}\_\{t\}=\\operatorname\{TopK\}\_\{i\\in\\\{1,\\ldots,N\\\}\}\(s\_\{i,t\}\),\(1\)其中NN是 routed experts 数，eie\_\{i\}是 expert routing embedding 或 Router 权重。输出为 yt=xt\+∑i∈𝒦tgi,tEi\(xt\),gi,t=exp\(si,t\)∑j∈𝒦texp\(sj,t\)\.y\_\{t\}=x\_\{t\}\+\\sum\_\{i\\in\\mathcal\{K\}\_\{t\}\}g\_\{i,t\}E\_\{i\}\(x\_\{t\}\),\\qquad g\_\{i,t\}=\\frac\{\\exp\(s\_\{i,t\}\)\}\{\\sum\_\{j\\in\\mathcal\{K\}\_\{t\}\}\\exp\(s\_\{j,t\}\)\}\.\(2\)若存在 always\-on shared experts，则在式 \([2](https://arxiv.org/html/2608.08650#A2.E2)\)中增加∑jEjshared\(xt\)\\sum\_\{j\}E^\{\\mathrm\{shared\}\}\_\{j\}\(x\_\{t\}\)。从公式看，MoE 只是稀疏函数组合；从系统看，𝒦t\\mathcal\{K\}\_\{t\}决定 token 要跨越哪些设备、每张卡获得多少 token，以及每个局部 GEMM 的形状。 ### B\.2 容量、质量与系统效率的三目标 MoE 并不是单目标优化。粗略地，模型希望最大化总容量PtotalP\_\{\\mathrm\{total\}\}，同时控制单 token 激活量PactiveP\_\{\\mathrm\{active\}\}： Pactive≈Pdense\+kNProuted\+Pshared,P\_\{\\mathrm\{active\}\}\\approx P\_\{\\mathrm\{dense\}\}\+\\frac\{k\}\{N\}P\_\{\\mathrm\{routed\}\}\+P\_\{\\mathrm\{shared\}\},\(3\)但式 \([3](https://arxiv.org/html/2608.08650#A2.E3)\)只是参数口径近似：attention、embedding、shared experts、不同 expert 宽度及框架统计方式都会造成偏差。另一方面，令一个统计窗口内 expertii接收的 token 数为nin\_\{i\}，则负载变异系数为 CVexpert=1N∑i\(ni−n¯\)2n¯\.\\operatorname\{CV\}\_\{\\mathrm\{expert\}\}=\\frac\{\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i\}\(n\_\{i\}\-\\bar\{n\}\)^\{2\}\}\}\{\\bar\{n\}\}\.\(4\)低 CV 有利于 EP 吞吐，却不必然有利于知识 specialization。架构设计的根本张力是：语义上允许不均匀，物理执行上又不能出现严重 straggler。 表 1:决定 MoE 架构差异的六个问题 ### B\.3 四个控制面：对象、决策、控制与执行 表[1](https://arxiv.org/html/2608.08650#A2.T1)中的设计问题并不处于同一层次。为了区分“历史上架构如何演进”和“一个具体 MoE 系统内部如何工作”，本文进一步按状态变量、决策粒度和更新时间尺度把系统横截面分为四个控制面：Topology 是较慢变化的模型结构；Routing 是逐 token 的离散决策；Balance 在一组 tokens 上统计并施加反馈；Expert Parallel（EP）把逻辑决策落实为设备上的通信与 kernel。第三章将沿历史时间轴讨论瓶颈迁移，第四至七章则分别展开这四个控制面；后者不是四个新的发展阶段。 形式上，可以把四层关系写成 ℰ\\displaystyle\\mathcal\{E\}=T\(θtopo\),\\displaystyle=T\(\\theta\_\{\\mathrm\{topo\}\}\),Topology：构造 expert 集合、分组和共享关系;\\displaystyle\\text\{Topology：构造 expert 集合、分组和共享关系\};𝒦t\\displaystyle\\mathcal\{K\}\_\{t\}=R\(xt,ℰ;θroute,b\),\\displaystyle=R\(x\_\{t\},\\mathcal\{E\};\\theta\_\{\\mathrm\{route\}\},b\),Routing：为 tokent选择 expert 子集;\\displaystyle\\text\{Routing：为 token $t$ 选择 expert 子集\};ni\\displaystyle n\_\{i\}=∑t𝟏\[i∈𝒦t\],\(b,α,c\)←C\(\{ni\},π\),\\displaystyle=\\sum\_\{t\}\\mathbf\{1\}\[i\\in\\mathcal\{K\}\_\{t\}\],\\quad\(b,\\alpha,c\)\\leftarrow C\(\\\{n\_\{i\}\\\},\\pi\),Balance：依据聚合负载更新 bias、loss 或 capacity;\\displaystyle\\text\{Balance：依据聚合负载更新 bias、loss 或 capacity\};yt\\displaystyle y\_\{t\}=EP\(xt,𝒦t,π,σ\),\\displaystyle=\\operatorname\{EP\}\\\!\\left\(x\_\{t\},\\mathcal\{K\}\_\{t\},\\pi,\\sigma\\right\),EP：按 placementπ和 scheduleσ执行 dispatch/combine\.\\displaystyle\\text\{EP：按 placement $\\pi$ 和 schedule $\\sigma$ 执行 dispatch/combine\}\.\(5\)其中ℰ\\mathcal\{E\}是逻辑 expert 集合，𝒦t\\mathcal\{K\}\_\{t\}是 token 的路由结果，nin\_\{i\}是统计窗口内 expertii的负载，bb、α\\alpha、cc分别代表 Router bias、辅助损失强度和 capacity 控制量，π\\pi是 expert 到设备的映射，σ\\sigma是通信与计算调度。式\([5](https://arxiv.org/html/2608.08650#A2.E5)\)说明：四层并非独立模块，而是一个闭环。 Topology 对象层定义ℰ\\mathcal\{E\}与候选约束Routing 决策层输出𝒦t\\mathcal\{K\}\_\{t\}Expert Parallel 执行层按π,σ\\pi,\\sigma输出yty\_\{t\}Balance 控制层统计nin\_\{i\}与 device/node/communication load反馈：loss / bias / capacity→\\rightarrowRouting；placement / replication→\\rightarrowEP图 1:Topology、Routing、Balance 与 Expert Parallel 的闭环关系。前三条实线形成前向执行链，虚线表示统计与系统代价对路由和放置的反馈。表 2:四个控制面的划分标准四层之间存在两类不能忽略的反向依赖。第一，物理拓扑会约束语义路由：device\-limited routing 为降低 fan\-out，主动缩小ℰ\\mathcal\{E\}中对某个 token 可见的候选集合。第二，Balance 不只调 Router：runtime replication 或 expert placement 可以在不改变𝒦t\\mathcal\{K\}\_\{t\}的情况下改善物理负载。因此“负载均衡”不能只理解为一个 auxiliary loss，“Expert Parallel”也不是模型结构确定后才介入的被动实现。 ## 附录 C八个演进节点：划分标准、主干与分支 这里的“八个”不是从某篇 Survey 直接摘录的固定 taxonomy，也不是按年份机械切段，而是本文依据主导瓶颈迁移做出的分析性归纳。一个变化只有同时满足以下三点，才被单列为演进节点：第一，它引入了可独立调节的新设计变量，例如 Top\-kk、expert 粒度、shared path 或动态 active budget；第二，它使系统的主要矛盾发生迁移，例如从计算随 expert 数线性增长，转为离散路由与负载不均，再转为 All\-to\-All、小 GEMM 或权重 I/O；第三，它被后续架构继承，因而不是某个模型的一次性实现细节。 按这一标准，节点 1–6 构成较清晰的历史主干：统计分工→\\rightarrow稀疏激活→\\rightarrowTransformer 规模化→\\rightarrowdecoder\-only 产品化→\\rightarrow细粒度知识组织→\\rightarrowultra\-sparse 容量扩展。节点 7 和节点 8 则不是简单接在节点 6 之后的新“代际”：节点 7 放松“每个 token 固定计算量”的假设；节点 8 放松“语义路由、层内 expert 拓扑和物理通信必须绑定”的假设。二者都可与节点 5 或 6 的 expert 结构组合。图[2](https://arxiv.org/html/2608.08650#A3.F2)给出这种继承关系。 1 统计分工Dense/Soft MoE2 稀疏激活Top\-kkconditional compute3 集群规模化Transformer \+ EP4 产品化开放 decoder MoE5 知识细分Fine\-grained \+ shared6 容量继续扩展Ultra\-sparse scaling7 计算预算动态化adaptive\-kk/ zero\-compute experts8 语义与物理解耦ScMoE / 异构 / 跨层 / 结构化通信图 2:八个演进节点的关系。实线表示历史主干上的主要继承，虚线表示可叠加的正交分支；节点编号不代表严格代际替换。表 3:八个节点的划分依据与瓶颈迁移### C\.1 Dense/Soft MoE：统计分工而非计算稀疏 Jacobs 等人在 1991 年提出 gating network 与 local experts 的基本形式\[[4](https://arxiv.org/html/2608.08650#biba.bib4)\]。这一节点建立的是统计分工：gate 根据输入产生混合权重，不同子网络拟合输入空间的不同区域，训练目标可以推动 specialization。由于所有 experts 通常都参与加权，路由是连续可导的，既不需要 token capacity，也不存在 All\-to\-All 意义上的稀疏 dispatch。 为什么单列。它定义了后来 MoE 一直保留的三个角色：Router/gate、experts 和 weighted combine。为何还不是现代稀疏 MoE。增加 expert 数会近似线性增加计算，因此“模型容量”和“每 token FLOPs”尚未解耦。节点 2 继承其统计分工目标，但将 dense mixture 改为只执行少数 experts；这一步才产生 conditional scaling，也同时产生离散选择带来的训练与系统问题。 ### C\.2 Sparse conditional computation：Top\-k 建立容量杠杆 Shazeer 等人将 noisy Top\-kkrouting 用于超大稀疏网络\[[5](https://arxiv.org/html/2608.08650#biba.bib5)\]，使每个输入只执行少数 experts。若总 expert 数为NN、每个输入只激活k≪Nk\\ll N个 experts，那么 total capacity 可以随NN增长，而主要 expert FLOPs 近似只随kk增长。这里第一次形成现代 MoE 的容量杠杆：参数容量与单 token 计算量解耦。 这一解耦并非免费。Top\-kk使未选 expert 没有来自该 token 的梯度；热门 experts 会溢出 capacity，冷门 experts 可能长期得不到训练；跨设备执行还必须加入 dispatch/combine。因而 noisy routing、importance/load auxiliary loss、capacity factor 和 token drop 不是外围技巧，而是稀疏执行本身诱发的配套机制。节点 3 并不改变这一基本算法，而是回答：当 MoE 被反复嵌入 Transformer 层、扩展到数千设备时，如何让它稳定且可执行。 ### C\.3 Transformer MoE 与 Expert Parallel GShard 将 Top\-2 MoE FFN 系统化嵌入 Transformer，并依靠自动 sharding 在2048 个 TPU上训练超过 600B的多语言模型\[[6](https://arxiv.org/html/2608.08650#biba.bib6)\]。经典执行链由此固定为 Attention→\\rightarrowRouter→\\rightarrowDispatch All\-to\-All→\\rightarrowExpert FFN→\\rightarrowCombine All\-to\-All\. 这一节点的新增变量不是“更多 experts”，而是expert parallelism 与集群执行语义：token 如何跨设备重排、每个 expert 如何批处理、溢出如何处理、通信与计算如何同步。Switch Transformer 进一步采用 Top\-1，以更低通信和更简单的执行换取专家组合能力与路由容错下降\[[7](https://arxiv.org/html/2608.08650#biba.bib7)\]。ST\-MoE 则把稳定性提升为一等设计目标，引入 Router z\-loss 约束 logits 数值尺度\[[8](https://arxiv.org/html/2608.08650#biba.bib8)\]。需要区分：z\-loss 控制数值稳定，balance loss 控制使用分布，两者不是同一机制。 同期，BASE Layers 把训练路由写成严格均衡的线性分配问题\[[9](https://arxiv.org/html/2608.08650#biba.bib9)\]，Expert Choice 则让 experts 选择固定容量 tokens\[[10](https://arxiv.org/html/2608.08650#biba.bib10)\]。它们能够直接保证均衡，但 batch\-level 联合分配和 expert\-side capacity 对在线自回归 decoding 不如 token\-choice 自然，因此没有取代 decoder\-only LLM 的 Top\-kk主干。节点 3 的遗产是“Router \+ EP \+ capacity/balance \+ All\-to\-All”这一完整系统契约；节点 4 和 5 都沿用它，只是把重点从“能否大规模训练”转向模型质量、部署可用性和 expert 内部组织。 ### C\.4 开放权重 coarse\-grained MoE Mixtral 采用8×\\times7B、每 token Top\-2\[[11](https://arxiv.org/html/2608.08650#biba.bib11)\]。它没有发明新的 Router family，因此若只按算法新颖性看，似乎不应成为独立节点。但它补齐了另一条关键证据链：经典 coarse\-grained MoE 可以在开放权重 decoder\-only LLM 中贯通预训练、指令微调、推理部署与社区复现。 为什么单列。节点 3 证明的是超大集群上的训练可扩展性；节点 4 证明的是通用 decoder 模型的可用性和部署闭环。它继承 Top\-kk、同构 experts 和 EP，不改变基本执行链。也正因为系统已能稳定运行，主要矛盾才显露为知识组织问题：少量大 experts 容易重复学习公共能力，Router 只能在很粗的知识块之间组合。节点 5 正面处理这一瓶颈。 ### C\.5 Shared \+ fine\-grained experts DeepSeekMoE 对粗粒度专家做了两项关键改造\[[12](https://arxiv.org/html/2608.08650#biba.bib12)\]：其一，把一个大 FFN 切成多个较小 experts，使 Router 在相同 active budget 下组合更多知识单元；其二，加入 shared expert isolation，把所有 token 都需要的公共知识交给 always\-on 路径，减少 routed experts 的知识冗余。前者提高组合分辨率，后者把“公共能力”和“条件能力”显式拆开。DeepSeek\-V2 将这一结构扩大到236B/21B active，并加入 device\-limited routing：每个 token 的候选 experts 被限制在少量设备内，从而控制跨设备 fan\-out\[[13](https://arxiv.org/html/2608.08650#biba.bib13)\]。 共享或稠密路径还能提供隐藏通信的计算窗口。DeepSeek\-V2 将 shared expert computation 与 EP communication overlap；Snowflake Arctic 的 dense residual path 也体现了同类算法–系统协同\[[14](https://arxiv.org/html/2608.08650#biba.bib14)\]。但shared expert 不是 fine\-grained MoE 的必要条件。Qwen3\-235B\-A22B 采用128 个细粒度 experts、Top\-8，却移除 shared expert，并依靠 global\-batch balance 允许常用能力在 routed experts 中自然形成\[[15](https://arxiv.org/html/2608.08650#biba.bib15)\]。 因此，节点 5 实际包含两个可拆分的轴：expert granularity 与 shared path。它们经常组合，但不存在逻辑绑定。节点 5 继承节点 3 的 EP 系统，却把 expert 数、Top\-kk和设备 fan\-out 同时推高，使小 GEMM、路由碎片化和通信拓扑成为新瓶颈。节点 6 沿着“更多、更小 experts”继续扩展；节点 8 则试图改变 fan\-out 与通信的物理实现。 ### C\.6 Ultra\-sparse scaling 这一阶段的问题变为：*固定 active parameters 时，继续增加 total experts 是否仍能降低 loss？*PEER 用 product\-key retrieval 从超过百万个 tiny experts中选择少数 experts\[[16](https://arxiv.org/html/2608.08650#biba.bib16)\]；Kimi K2 将工程可落地的 ultra\-sparse 结构扩展到1\.04T/32\.6B active、384 routed experts、Top\-8，并报告固定 active budget 下更高 sparsity 的 scaling 收益\[[17](https://arxiv.org/html/2608.08650#biba.bib17)\]；gpt\-oss\-120b 则采用128 experts、Top\-4和较小 active footprint，体现面向本地部署的高稀疏路线\[[18](https://arxiv.org/html/2608.08650#biba.bib18)\]。 为什么从 fine\-grained 单列。fine\-grained 的首要目标是减少知识冗余、提高组合分辨率；ultra\-sparse 的首要问题则是 scaling：固定 active parameters 和训练 FLOPs 时，扩大候选 expert 集是否持续改善 loss。二者都增加 expert count，但优化目标不同。 稀疏度不能无限增加。随着NN增大，Router scoring、expert weight I/O、All\-to\-All metadata 和小 GEMM 碎片化都会上升；当每个 expert 每步获得的 tokens 太少时，理论 FLOPs 优势不能转化为 wall\-clock 收益。这意味着节点 6 没有终结节点 5，反而更依赖其 fine\-grained 组织和路由控制。节点 7 与 8 则从两个正交方向绕开固定 Top\-kk主干的限制。 ### C\.7 动态计算：从固定 Top\-k 到 token\-dependent budget 固定 Top\-kk隐含所有 token 使用相同数量的 expert FLOPs，即便 token 的语义难度和所需容量不同。LongCat\-Flash 加入 zero\-computation experts：Router 仍选择固定 slots，但部分 slots 是 identity，因此真实执行的 FFN 数量随 token 变化\[[19](https://arxiv.org/html/2608.08650#biba.bib19)\]。adaptive\-kk、threshold routing 与 zero\-compute slots 的共同目标，是把“选哪个 expert”和“到底计算多少”从一个固定超参数中拆开。 与节点 6 的关系。ultra\-sparse 扩大的是候选容量NN，但通常仍保持固定kk；动态计算改变的是每个 token 的实际ktk\_\{t\}或有效 FFN 数。两者可以组合：候选池很大，同时简单 token 少算、困难 token 多算。控制目标也由单一 expert load 扩展为“平均 active budget \+ budget variance \+ real\-expert load \+ P99 latency”。训练时的平均 FLOPs 约束并不能自动保证在线尾延迟，因为困难 tokens 可能在同一 micro\-batch 或请求窗口内聚集。 动态计算的遗留问题是闭环控制：若 Router 同时决定语义 expert 和计算预算，balance bias 既影响 specialization，又影响吞吐；若只约束长期均值，又可能出现短时 burst。因而 adaptive bias/PID 式反馈、budget loss 与 runtime admission control 是配套问题，而非单纯修改 Top\-kk即可解决。 ### C\.8 语义路由与物理执行解耦 节点 1–7 主要改变 Router 可选的专家或 active budget；节点 8 改变的是更底层的假设：语义上选中了某个 expert，并不意味着必须沿固定的层内 All\-to\-All 路径立即执行它。这一节点不是单一模型家族，而是四条具有共同目标的前沿分支。 执行依赖重排。ScMoE 不改变 token 选哪些 experts，而是改变依赖图。其跨层 shortcut 让 routed experts 消费前层中间表示，同时用当前 dense/shared path 覆盖 dispatch/combine 的通信窗口\[[20](https://arxiv.org/html/2608.08650#biba.bib20)\]。它解决的是 exposed communication，不直接解决 routing skew。图[3](https://arxiv.org/html/2608.08650#A3.F3)给出抽象比较。 Attention / shortcutDispatchRouted expertsCombineMergeDense/shared branch通信路径与 dense 计算并行图 3:ScMoE 的核心是重排执行依赖以扩大通信隐藏窗口；它不是新的路由算法。LongCat\-Flash 将 ScMoE 与 token\-dimension chunking、Single Batch Overlap 结合；LongCat\-2\.0 的官方报告进一步在专用 accelerator 上采用 dense/MoE per\-core full parallel execution\[[21](https://arxiv.org/html/2608.08650#biba.bib21)\]。这些进展降低 exposed communication，但不会自动解决严重 routing skew、expert 内部计算差异和显存压力。 异构 expert。MoHGE 采用两级路由：先选 expert group，再在组内选择不同大小 experts，并结合 group\-wise 与 intra\-group balance 约束部署\[[22](https://arxiv.org/html/2608.08650#biba.bib22)\]。它把“expert 等宽”从默认约束变成设计变量，使不同 token 可获得不同形态的计算能力；代价是异构 GEMM、容量规划和负载指标都更复杂。 跨层共享。GMoE 把一部分 experts 设为跨层共享的 Global Experts，同时保留 layer\-local experts，以减少跨层能力冗余\[[23](https://arxiv.org/html/2608.08650#biba.bib23)\]。这与 shared expert 的区别在于共享作用域：前者跨 layer，后者通常是在单层内对所有 tokens always\-on。跨层共享会引入缓存、调度和层间干扰问题。 结构化通信。Multi\-Head LatentMoE 与 Head Parallel 尝试改变通信拓扑，使通信量相对 activated expert 数kk为O\(1\)O\(1\)，并获得更确定的流量\[[24](https://arxiv.org/html/2608.08650#biba.bib24)\]。与 ScMoE“隐藏动态通信”不同，这条路线试图“减少或结构化通信本身”。由于尚缺少 trillion\-parameter、万卡级公开验证，它应被视为研究前沿，而非已经替代 Top\-kkEP 的主流。 四条分支分别优化时间重叠、计算形态、参数复用和数据移动，可组合而非替代。 ## 附录 DExpert topology：对象层 在第二章的四层框架中，Topology 是对象层：它先定义 expert 集合ℰ\\mathcal\{E\}、知识单元的粒度、共享范围和计算异质性，Routing 才能在其上做逐 token 选择。因而，Topology 的核心不只是 expert count，而是Router 可以组合什么样的计算单元。 把 MoE 演进只理解为 expert count 增长是不充分的。表[4](https://arxiv.org/html/2608.08650#A4.T4)显示，真正变化的是知识单元的粒度、共享范围和计算异质性。 表 4:Expert topology 的主要演进路线另一个经常被忽略的维度是初始化方式。Sparse Upcycling 将已有 dense FFN 复制成多个 experts，再新增 Router 继续训练\[[25](https://arxiv.org/html/2608.08650#biba.bib25)\]。它复用 dense checkpoint，却带来 expert symmetry，需要路由噪声、数据差异和后续训练使 experts 分化。更进一步的 Expert Upcycling 从已有EE\-expert MoE 扩展到mEmEexperts，同时保持 Top\-kk与 active cost 不变\[[26](https://arxiv.org/html/2608.08650#biba.bib26)\]。upcycling 不是新的 Router family，但会改变 specialization 的形成路径和训练成本。 ## 附录 ERouting：决策层 在图[1](https://arxiv.org/html/2608.08650#A2.F1)的闭环中，Routing 接收 Topology 定义的 expert 集合与系统施加的候选约束，对每个 token 产生𝒦t\\mathcal\{K\}\_\{t\}。它优化的是局部语义匹配，并不直接保证一个 sequence、batch 或 device 上的总体负载合理；后者属于第六章的控制层。 表 5:主要 Router family 及其适用边界Token\-choice 成为主流并不表示它在优化上最优，而是它最符合自回归在线约束：当前 token 的路由只依赖当前状态，不需要等待未来 tokens 或求解整个 batch 的全局分配。Soft MoE 和 Lory 等连续路由方案避免离散 Top\-kk，但尚未在超大规模 decoder 与低延迟 EP 上证明系统优势\[[27](https://arxiv.org/html/2608.08650#biba.bib27),[28](https://arxiv.org/html/2608.08650#biba.bib28)\]。 路由自由度还受到网络拓扑约束。device\-limited 或 node\-limited routing 将候选 expert 限制在少量设备/节点集合，降低单 token 的通信 fan\-out。代价是语义最优 expert 可能被拓扑约束排除，因此这类设计本质上是质量与通信之间的显式折中。 ## 附录 FBalance：控制层 Balance 不重新定义 experts，也不独立生成 token 的语义路由。它观察许多次 Routing 决策累积出的\{ni\}\\\{n\_\{i\}\\\}及设备、节点和通信负载，再通过 loss、bias、capacity 或 runtime placement 反馈给决策层和执行层。其核心问题是：允许语义 specialization 不均匀的同时，如何避免训练 collapse 和物理 straggler。 ### F\.1 Sequence/batch\-wise expert balance 经典辅助损失可写为 ℒbal=αN∑i=1NfiPi,\\mathcal\{L\}\_\{\\mathrm\{bal\}\}=\\alpha N\\sum\_\{i=1\}^\{N\}f\_\{i\}P\_\{i\},\(6\)其中fif\_\{i\}是统计窗口内实际分配到 expertii的 token fraction，PiP\_\{i\}是平均 routing probability，α\\alpha控制均衡强度。当所有 experts 使用近似均匀时，式 \([6](https://arxiv.org/html/2608.08650#A6.E6)\)较小。该损失同时承担防止 expert collapse 和避免 EP straggler 两项职责，但二者并不等价：模型可能需要不均匀 specialization，硬件却希望均匀负载。α\\alpha过大时，硬件目标会直接干扰语言建模梯度。 ### F\.2 Device/node/communication\-aware balance DeepSeek\-V2 把 balance 拆为 expert\-level、device\-level 与 communication balance，并通过 device\-limited routing 控制 fan\-out\[[13](https://arxiv.org/html/2608.08650#biba.bib13)\]。这一变化的关键不是多加几个 loss，而是承认逻辑 expert 均衡不等于物理设备或网络均衡：若一个 device 上有多个 experts，只要 device 间总 token 接近，未必需要所有 experts 完全等量。 ### F\.3 Auxiliary\-Loss\-Free Load Balancing Auxiliary\-Loss\-Free（ALF）策略给每个 expert 增加非梯度 routing biasbib\_\{i\}： 𝒦t=TopKi\(si,t\+bi\),bi←bi\+ηsign\(n¯−ni\),\\mathcal\{K\}\_\{t\}=\\operatorname\{TopK\}\_\{i\}\(s\_\{i,t\}\+b\_\{i\}\),\\qquad b\_\{i\}\\leftarrow b\_\{i\}\+\\eta\\,\\operatorname\{sign\}\(\\bar\{n\}\-n\_\{i\}\),\(7\)其中η\\eta是 bias 更新速度。bias 影响 Top\-kk选择，但不作为 affinity 权重参与 LM 梯度，从而减少均衡目标对模型参数更新方向的直接干扰\[[29](https://arxiv.org/html/2608.08650#biba.bib29)\]。DeepSeek\-V3 使用 batch\-wise ALF 作为主均衡机制，同时保留很弱的 sequence\-wise auxiliary loss，防止单序列极端失衡\[[30](https://arxiv.org/html/2608.08650#biba.bib30)\]。因此，“DeepSeek\-V3 完全没有任何 balance loss”并不准确。 global\-batch 统计比单 micro\-batch 更稳定，也允许某些 experts 在局部 batch 上承载更多特定 domain token；代价是需要跨 DP/EP 范围聚合负载，扩大统计范围可能引入额外通信。 ### F\.4 Runtime expert placement 训练分布上的均衡不能保证真实推理流量。domain shift、SFT/RL 和关闭 balance 更新都会改变 expert popularity。DeepSeek\-V3 使用冗余 expert deployment，在运行时复制热点 experts 并调整 placement\[[30](https://arxiv.org/html/2608.08650#biba.bib30)\]。这一趋势把目标重新分工：Router 可以保留语义上的非均匀 specialization，runtime 通过复制、迁移或重新放置避免设备 straggler。 Sequence/Batchexpert balanceDevice/Node/CommunicationGlobal\-batchALF biasRuntime replicationand placement图 4:Balance 的演进不是简单减弱约束，而是逐步把物理执行目标从 LM 梯度中剥离。 ## 附录 GExpert Parallel：执行层 EP 消费 Routing 产生的 token–expert assignment，并结合 placementπ\\pi把逻辑 expert 转化为设备上的 dispatch、局部 GEMM 与 combine。它通常不改变“语义上选中了谁”，却决定这次选择的 wall\-clock 成本；当系统代价过高时，又会通过 device\-limited routing、placement 或架构重构反向修改前三层。 MoE 的算法收益只有在稀疏执行高效时才成立。capacity\-based 系统为每个 expert 预留固定 token slots，溢出 token 被丢弃或走 residual，容易造成训练与推理不一致。MegaBlocks 使用 block\-sparse kernels 实现 dropless MoE，使动态 token shape 不再要求 token dropping\[[31](https://arxiv.org/html/2608.08650#biba.bib31)\]。Tutel 等系统则动态选择并行和 pipeline 策略，以适配不同集群规模\[[32](https://arxiv.org/html/2608.08650#biba.bib32)\]。 随着专家更细、Top\-kk更大，通信逐渐成为一等架构约束，推动出三种处理方式： 1. 1\.限制通信拓扑：device/node\-limited routing 降低 fan\-out； 2. 2\.隐藏通信：shared computation overlap、DualPipe、ScMoE 和 token chunking 扩大 overlap window； 3. 3\.重构通信：Head Parallel 等方案把动态流量变为更确定的通信模式。 这里必须区分“总 All\-to\-All 时间”和“exposed communication time”。若通信完全被 dense path 覆盖，继续优化链路带宽对端到端 step time 的边际收益会下降；反之，即使总通信量不变，只要关键路径缩短，吞吐也可能显著改善。因此训练报告至少应同时给出通信量、overlap 比例、exposed dispatch/combine、MFU 和 step\-time P99。 ## 附录 H现代主干与前沿分支 截至 2026 年，大规模 decoder MoE 最常见的组合可概括为： Token\-choice Top\-kk\+ fine\-grained experts \+ 可选 shared path \+ global/batch\-wise balance 或 ALF \+ topology\-limited routing \+ dropless kernels/communication overlap \+ runtime expert placement DeepSeek\-V3、Qwen3 和 Kimi K2 在 shared expert、balance 机制和 EP size 上不同，但都位于这条主干。与此同时，以下分支仍未形成统一结论： - •MoE Attention：JetMoE 同时稀疏化 Attention 与 FFN，进一步降低 active FLOPs，但 KV/cache 和路由更复杂\[[33](https://arxiv.org/html/2608.08650#biba.bib33)\]； - •Hybrid mixer \+ MoE：Jamba 在 Transformer 与 Mamba layers 间交替，说明 MoE 是参数扩容方式，不绑定全 Attention backbone\[[34](https://arxiv.org/html/2608.08650#biba.bib34)\]； - •Million experts：理论组合性高，但权重 I/O、小 GEMM 和检索开销仍限制工业部署； - •Heterogeneous experts：按 token 难度选择不同容量，表达自然，但物理放置和尾延迟更难； - •Cross\-layer global experts：目标是消除层间重复能力，仍缺少超大规模训练证据； - •Structured communication：Head Parallel 可能降低动态 EP 痛点，但尚处早期验证阶段。 表[6](https://arxiv.org/html/2608.08650#A8.T6)以架构选择而非 benchmark 排名对代表性模型进行对照。 表 6:代表性 LLM MoE 的架构位置；active 参数口径因报告而异 ## 附录 I面向基座预训练的实验设计 跨模型 benchmark 无法识别架构贡献。若要判断 fine\-grained、shared、ALF 或 ScMoE 是否有效，应至少同时固定训练 token、数据配比、active parameters 和优化器，并分别报告training\-FLOPs\-matched 与 wall\-clock\-matched结果。 表 7:建议的等预算 MoE 消融矩阵对代码预训练场景，建议额外按语言、repo/file/function 粒度、代码与自然语言 token 分桶统计 routing。关键观测包括：不同语言的 expert co\-activation、跨文件引用 token 是否集中到少数专家、代码与 code\-related 文档是否共享 experts，以及 repo\-level 长序列是否造成 sequence\-wise load 峰值。这样才能区分“专家形成了有益 specialization”和“数据配比造成偶然流量偏斜”。 系统侧至少应同时记录： - •expert、device、node 三个层级的 max/mean、CV 与 Gini； - •sequence、micro\-batch、global batch 三个统计窗口下的负载； - •dispatch/combine exposed time，而不只看总 All\-to\-All time； - •token drop、capacity overflow、redundant expert hit rate； - •training FLOPs、active parameters、wall\-clock 和集群网络拓扑。 ## 附录 J开放问题 ### J\.1 Specialization 的正确分析单位是什么 “数学专家”或“代码专家”是直观叙事，但实际路由常表现为 token 形态、语法、位置、语言和层级组合。单 expert 未必是稳定的语义单位，跨层 expert group 或 co\-activation pattern 可能更可解释。未来应把 routing mutual information、expert representation similarity 和 intervention\-based causal test 结合起来，而不只展示 token word cloud。 ### J\.2 更高 sparsity 的系统最优点在哪里 在固定 active FLOPs 下增加 total experts 可能降低模型 loss，但系统收益受网络带宽、expert weight I/O、batch size 与 kernel arithmetic intensity 限制。最优 sparsity 不是纯模型常数，而是模型–硬件联合 scaling law。报告该结论时必须给出 EP size、网络拓扑和每 expert token 数。 ### J\.3 Balance 能否主要下沉到 runtime ALF 减少了辅助 loss 的梯度干扰，却仍会改变 Top\-kk选择。runtime replication 和 placement 可以允许语义路由更自由，但会带来参数副本显存、迁移成本和缓存一致性问题。训练弱约束与运行时强均衡之间的最优边界仍未确定。 ### J\.4 动态计算如何满足尾延迟 SLA zero experts 或 adaptive\-kk可以降低平均 FLOPs，但困难 tokens 若在同一 micro\-batch 或请求窗口集中，会制造新的 straggler。模型需要同时控制平均预算、预算方差和 P99，而不能只报告平均 active parameters。 ### J\.5 Post\-training 如何保持 Router 稳定 SFT/RL 数据会改变领域分布和序列形态，从而改变 expert load。训练阶段是否保留 balance 更新、是否 routing replay、是否冻结 Router，不能从预训练设置直接外推。应分别测量 pretrain、SFT 和 RL 阶段的 routing drift 与 specialization retention。 ## 附录 K总结与展望 LLM MoE 的架构史不是“expert 越来越多”的单线扩张，而是五条路线的共同演化：expert 从少量大模块变成细粒度、超稀疏和异构计算单元；拓扑从 layer\-local routed experts 发展到 shared、cross\-layer global/local 和 latent\-head 结构；Router 从简单 Top\-kk逐步加入全局统计、bias 控制和动态计算预算；balance 从 sequence\-wise 辅助损失走向 device/communication\-aware、ALF 与 runtime placement；执行则从串行 All\-to\-All 发展到 dropless kernels、topology\-limited routing、ScMoE overlap 和结构化通信。 因此，ScMoE 之后并不存在唯一的“下一代结构”。更可信的方向是四者组合：更自由的语义路由、显式的平均计算预算控制、拓扑感知或确定性通信，以及运行时 expert placement。架构评估也必须从单一 validation loss 扩展为质量、specialization、active budget、exposed communication 和尾延迟的联合分析。 对下一轮大规模预训练，最具信息量的实验不是直接比较两个不同训练配方的终局 benchmark，而是构造同数据、同 token、同 active FLOPs的受控矩阵，分别扫描 expert granularity、shared path、balance scope 和 execution topology。只有这样，才能区分参数容量收益、路由专业化收益与系统吞吐收益，并建立可迁移到下一代硬件的 MoE 设计结论。 ## 附录 L架构演进速查表 表 8:MoE 架构演进的瓶颈迁移时期代表工作解决的主要问题转移出的新瓶颈1991–2016Adaptive Mixtures学习软分工计算随 expert 数线性增长2017–2019Sparse MoE参数容量与单 token FLOPs 解耦collapse、capacity、All\-to\-All2020–2022GShard、Switch、ST\-MoETransformer 大规模稀疏训练稳定性、token drop、路由质量2023–2024Mixtral开放 decoder MoE 可用性粗粒度与知识冗余2024–2025DeepSeekMoE/V2、Qwen3细粒度组合与公共知识处理通信 fan\-out、小 GEMM、balance scope2024–2025PEER、Kimi K2、gpt\-ossultra\-sparse 容量扩展检索、I/O、低 token/expert2024–2026ALF、LongCat、ScMoE减少梯度干扰与隐藏通信运行时动态负载与尾延迟2025–2026MoHGE、GMoE、Head Parallel异构计算、跨层共享、结构化通信超大规模质量和系统验证 ## 参考文献 - \[1\]Cai W, et al\. A Survey on Mixture of Experts in Large Language Models\. IEEE Transactions on Knowledge and Data Engineering, 2025\.[https://arxiv\.org/abs/2407\.06204](https://arxiv.org/abs/2407.06204)\. - \[2\]Liu J, et al\. A Survey on Inference Optimization Techniques for Mixture of Experts Models\. arXiv:2412\.14219, 2024\.[https://arxiv\.org/abs/2412\.14219](https://arxiv.org/abs/2412.14219)\. - \[3\]Zhu X, et al\. Speed Always Wins: A Survey on Efficient Architectures for Large Language Models\. arXiv:2508\.09834, 2025\.[https://arxiv\.org/abs/2508\.09834](https://arxiv.org/abs/2508.09834)\. - \[4\]Jacobs R A, Jordan M I, Nowlan S J, Hinton G E\. Adaptive Mixtures of Local Experts\. Neural Computation, 3\(1\):79–87, 1991\.[doi:10\.1162/neco\.1991\.3\.1\.79](https://doi.org/10.1162/neco.1991.3.1.79)\. - \[5\]Shazeer N, et al\. Outrageously Large Neural Networks: The Sparsely\-Gated Mixture\-of\-Experts Layer\. ICLR, 2017\.[https://arxiv\.org/abs/1701\.06538](https://arxiv.org/abs/1701.06538)\. - \[6\]Lepikhin D, et al\. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\. ICLR, 2021\.[https://arxiv\.org/abs/2006\.16668](https://arxiv.org/abs/2006.16668)\. - \[7\]Fedus W, Zoph B, Shazeer N\. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\. Journal of Machine Learning Research, 23\(120\):1–39, 2022\.[https://arxiv\.org/abs/2101\.03961](https://arxiv.org/abs/2101.03961)\. - \[8\]Zoph B, et al\. ST\-MoE: Designing Stable and Transferable Sparse Expert Models\. arXiv:2202\.08906, 2022\.[https://arxiv\.org/abs/2202\.08906](https://arxiv.org/abs/2202.08906)\. - \[9\]Lewis M, et al\. BASE Layers: Simplifying Training of Large, Sparse Models\. ICML, 2021\.[https://arxiv\.org/abs/2103\.16716](https://arxiv.org/abs/2103.16716)\. - \[10\]Zhou Y, et al\. Mixture\-of\-Experts with Expert Choice Routing\. NeurIPS, 2022\.[https://arxiv\.org/abs/2202\.09368](https://arxiv.org/abs/2202.09368)\. - \[11\]Jiang A Q, et al\. Mixtral of Experts\. arXiv:2401\.04088, 2024\.[https://arxiv\.org/abs/2401\.04088](https://arxiv.org/abs/2401.04088)\. - \[12\]Dai D, et al\. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture\-of\-Experts Language Models\. ACL, 2024\.[https://arxiv\.org/abs/2401\.06066](https://arxiv.org/abs/2401.06066)\. - \[13\]DeepSeek\-AI\. DeepSeek\-V2: A Strong, Economical, and Efficient Mixture\-of\-Experts Language Model\. arXiv:2405\.04434, 2024\.[https://arxiv\.org/abs/2405\.04434](https://arxiv.org/abs/2405.04434)\. - \[14\]Snowflake\. Arctic: Snowflake’s Open\-Source LLM\. Official Technical Blog, 2024\. Accessed 2026\-08\-07\.[https://www\.snowflake\.com/en/blog/arctic\-open\-efficient\-foundation\-language\-models\-snowflake/](https://www.snowflake.com/en/blog/arctic-open-efficient-foundation-language-models-snowflake/)\. - \[15\]Qwen Team\. Qwen3 Technical Report\. arXiv:2505\.09388, 2025\.[https://arxiv\.org/abs/2505\.09388](https://arxiv.org/abs/2505.09388)\. - \[16\]He X\. Mixture of A Million Experts\. arXiv:2407\.04153, 2024\.[https://arxiv\.org/abs/2407\.04153](https://arxiv.org/abs/2407.04153)\. - \[17\]Moonshot AI\. Kimi K2: Open Agentic Intelligence\. arXiv:2507\.20534, 2025\.[https://arxiv\.org/abs/2507\.20534](https://arxiv.org/abs/2507.20534)\. - \[18\]OpenAI\. gpt\-oss Model Card: Architecture\. 2025\. Accessed 2026\-08\-07\.[https://deploymentsafety\.openai\.com/gpt\-oss/architecture](https://deploymentsafety.openai.com/gpt-oss/architecture)\. - \[19\]LongCat Team\. LongCat\-Flash Technical Report\. arXiv:2509\.01322, 2025\.[https://arxiv\.org/abs/2509\.01322](https://arxiv.org/abs/2509.01322)\. - \[20\]Cai W, et al\. Shortcut\-connected Expert Parallelism for Accelerating Mixture\-of\-Experts\. arXiv:2404\.05019, 2024\.[https://arxiv\.org/abs/2404\.05019](https://arxiv.org/abs/2404.05019)\. - \[21\]LongCat Team\. LongCat 2\.0 Technical Blog\. 2026\. Accessed 2026\-08\-07\.[https://longcat\.ai/blog/longcat\-2\.0/](https://longcat.ai/blog/longcat-2.0/)\. - \[22\]MoHGE Authors\. MoHGE: Mixture of Heterogeneous Grouped Experts\. ACL Industry Track, 2026\.[https://aclanthology\.org/2026\.acl\-industry\.20/](https://aclanthology.org/2026.acl-industry.20/)\. - \[23\]GMoE Authors\. GMoE: Global Mixture\-of\-Experts\. ACL, 2026\.[https://aclanthology\.org/2026\.acl\-long\.2065/](https://aclanthology.org/2026.acl-long.2065/)\. - \[24\]Cui Y, et al\. Multi\-Head LatentMoE and Head Parallel\. arXiv:2602\.04870, 2026\.[https://arxiv\.org/abs/2602\.04870](https://arxiv.org/abs/2602.04870)\. - \[25\]Komatsuzaki A, et al\. Sparse Upcycling: Training Mixture\-of\-Experts from Dense Checkpoints\. arXiv:2212\.05055, 2022\.[https://arxiv\.org/abs/2212\.05055](https://arxiv.org/abs/2212.05055)\. - \[26\]Expert Upcycling Authors\. Expert Upcycling: Training Larger MoE Models from Smaller MoE Checkpoints\. arXiv:2604\.19835, 2026\.[https://arxiv\.org/abs/2604\.19835](https://arxiv.org/abs/2604.19835)\. - \[27\]Puigcerver J, et al\. From Sparse to Soft Mixtures of Experts\. ICLR, 2024\.[https://arxiv\.org/abs/2308\.00951](https://arxiv.org/abs/2308.00951)\. - \[28\]Zhong Z, et al\. Lory: Fully Differentiable Mixture\-of\-Experts for Autoregressive Language Model Pre\-training\. COLM, 2024\.[https://arxiv\.org/abs/2405\.03133](https://arxiv.org/abs/2405.03133)\. - \[29\]Wang Q, et al\. Auxiliary\-Loss\-Free Load Balancing Strategy for Mixture\-of\-Experts\. arXiv:2408\.15664, 2024\.[https://arxiv\.org/abs/2408\.15664](https://arxiv.org/abs/2408.15664)\. - \[30\]DeepSeek\-AI\. DeepSeek\-V3 Technical Report\. arXiv:2412\.19437, 2024\.[https://arxiv\.org/abs/2412\.19437](https://arxiv.org/abs/2412.19437)\. - \[31\]Gale T, et al\. MegaBlocks: Efficient Sparse Training with Mixture\-of\-Experts\. MLSys, 2023\.[https://arxiv\.org/abs/2211\.15841](https://arxiv.org/abs/2211.15841)\. - \[32\]Hwang C, et al\. Tutel: Adaptive Mixture\-of\-Experts at Scale\. MLSys, 2023\.[https://arxiv\.org/abs/2206\.03382](https://arxiv.org/abs/2206.03382)\. - \[33\]Shen Y, et al\. JetMoE: Reaching Llama2 Performance with 0\.1M Dollars\. arXiv:2404\.07413, 2024\.[https://arxiv\.org/abs/2404\.07413](https://arxiv.org/abs/2404.07413)\. - \[34\]Lieber O, et al\. Jamba: A Hybrid Transformer–Mamba Language Model\. arXiv:2403\.19887, 2024\.[https://arxiv\.org/abs/2403\.19887](https://arxiv.org/abs/2403.19887)\.
Similar Articles
A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
This paper investigates whether bilingual Mixture-of-Experts (MoE) language models develop linguistically structured expert routing. It finds that interpretable linguistic organization emerges within MoE routing patterns, and that curriculum training influences specialization in language balance.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.
@_avichawla: https://x.com/_avichawla/status/2100876555409039605
The article explains the engineering aspects of Mixture-of-Experts (MoE) inference, detailing token routing, expert batching, GPU distribution, and performance trade-offs for efficient serving.
Emergent Modularity in Mixture-of-Experts Models (8 minute read)
Ai2 releases EMO, a 14B-parameter mixture-of-experts language model trained to develop emergent modularity. It allows using a small subset of experts for specific tasks while maintaining near full-model performance.
Pruning and Distilling Mixture-of-Experts into Dense Language Models
A systematic framework converts mixture-of-experts models into dense architectures through expert scoring, selection, grouping, and knowledge distillation, achieving superior performance and efficiency compared to traditional pruning methods.