Distributionally Robust Mixture-of-Experts Training

Hugging Face Daily Papers Papers
mixture-of-experts routing-robustness distributionally-robust-training transformers sparse-models

Summary

提出DRMoET——一种将各层专家视为内生鲁棒性分组、优化高损失路由结果的即插即用MoE训练目标，在10.3B总参数规模上显著提升下游任务平均表现与路由鲁棒性。

Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability problem: when routing is imperfect, load-balanced models may send tokens to experts that are insufficiently trained for the assigned inputs. We propose Distributionally Robust MoE Training (DRMoET), a drop-in objective that treats layer-wise experts as endogenous robustness groups and optimizes high-loss routing outcomes rather than merely equalizing traffic. DRMoET updates a per-layer expert distribution by an entropy-regularized softmax rule on EMA-smoothed, activation-weighted expert losses, strengthening plausible non-top routing paths while preserving standard MoE computation. Under the FLAME-MoE recipe at 746M-total and 10.3B-total scales, DRMoET improves downstream averages over both standard FLAME-MoE and auxiliary-loss-free balancing. At 10.3B total parameters and 67B training tokens, DRMoET improves the seven-task average from 0.6625 to 0.6767, while the auxiliary-loss-free baseline achieves 0.6431. Mechanistic analyses show lower expert-loss variance with nearly unchanged mean loss, 4.3% lower excess loss under forced mid-k misrouting, and improved domain-expert specialization. These results position routing robustness-not only utilization balance-as a practical objective for reliable sparse MoE scaling. Project page and code are available at: https://drmoet.github.io/.
Original Article
View Cached Full Text

Cached at: 10/12/26, 06:41 PM

Paper page - Distributionally Robust Mixture-of-Experts Training

Source: https://huggingface.co/papers/2610.07207 Published on Oct 5

·

Submitted byhttps://huggingface.co/TonyTeng

xt2251on Oct 12

Abstract

Mixture-of-Experts(MoE)transformersscalecapacitybyactivatingonlyafewexpertspertoken,butthissparsitycreatesahiddenreliabilityproblem:whenroutingisimperfect,load-balancedmodelsmaysendtokenstoexpertsthatareinsufficientlytrainedfortheassignedinputs.WeproposeDistributionallyRobustMoETraining(DRMoET),adrop-inobjectivethattreatslayer-wiseexpertsasendogenousrobustnessgroupsandoptimizeshigh-lossroutingoutcomesratherthanmerelyequalizingtraffic.DRMoETupdatesaper-layerexpertdistributionbyanentropy-regularizedsoftmaxruleonEMA-smoothed,activation-weightedexpertlosses,strengtheningplausiblenon-toproutingpathswhilepreservingstandardMoEcomputation.UndertheFLAME-MoErecipeat746M-totaland10.3B-totalscales,DRMoETimprovesdownstreamaveragesoverbothstandardFLAME-MoEandauxiliary-loss-freebalancing.At10.3Btotalparametersand67Btrainingtokens,DRMoETimprovestheseven-taskaveragefrom0.6625to0.6767,whiletheauxiliary-loss-freebaselineachieves0.6431.Mechanisticanalysesshowlowerexpert-lossvariancewithnearlyunchangedmeanloss,4.3%lowerexcesslossunderforcedmid-kmisrouting,andimproveddomain-expertspecialization.Theseresultspositionroutingrobustness-notonlyutilizationbalance-asapracticalobjectiveforreliablesparseMoEscaling.Projectpageandcodeareavailableat:https://drmoet.github.io/.

View arXiv pageView PDFProject pageGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.07207

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### TonyTeng/DRMoET Text Generation• Updated5 days ago • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.07207 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.07207 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

arXiv cs.CL

该论文提出 Elastic Expert Routing，通过从以 k 为中心的局部离散分布中随机采样激活专家数量，平滑传统 top-k 路由的硬边界，在保持计算成本不变的前提下提升稀疏 MoE 模型在微调与从头预训练场景下的下游表现。

Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

arXiv cs.LG

论文提出 Spectral-MoE，一种在稠密模型转 MoE 时为不同层动态分配专家利用率（utilization budget）的方法，包括静态和动态两种分配器，在 Llama-2-7B、Llama-3-8B、Qwen2.5-7B 上显著降低困惑度并提升下游任务准确率，确立了逐层专家利用率这一新的优化方向。

E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

Hugging Face Daily Papers

论文提出 E-MoE（Enhanced Mixture-of-Experts），用 MoE 路由决策构成的离散共享隐变量来建模非因子化的反向过程，缓解连续隐变量的后验坍缩问题，并在无需增加活跃参数的情况下提升掩码扩散语言模型在少步生成场景下的样本质量。

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

arXiv cs.LG

论文提出 OptEP，一种面向混合专家（MoE）语言模型的后训练专家剪枝方法，通过连续松弛优化专家选择变量并结合重构驱动的权重更新，在 Qwen3、GLM-4.5-Air、Mixtral 等模型上显著优于现有剪枝基线，激进剪枝下精度提升最高达 16.20%。