EMO: Pretraining Mixture of Experts for Emergent Modularity

Hugging Face Daily Papers Papers
mixture-of-experts modular-deployment emergent-modularity pretraining sparse-models composable-architectures

Summary

EMO is a Mixture-of-Experts model that enables modular deployment by grouping similar domain tokens with shared experts, achieving performance comparable to standard MoEs while allowing significant expert pruning (25% experts retain 99% performance) without performance degradation.

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge. Mixture-of-Experts (MoEs) seemingly offer a potential alternative by activating only a subset of experts per input, but in practice, restricting inference to a subset of experts for a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encourage tokens from similar domains to rely on similar experts. Since tokens within a document often share a domain, EMO restricts them to select experts from a shared pool, while allowing different documents to use different pools. This simple constraint enables coherent expert groupings to emerge during pretraining using document boundaries alone. We pretrain a 1B-active, 14B-total EMO on 1T tokens. As a full model, it matches standard MoE performance. Crucially, it enables selective expert use: retaining only 25% (12.5%) of experts incurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting. We further find that expert subsets in EMO specialize at semantic levels (e.g., domains such as math or code), in contrast to the low-level syntactic specialization observed in standard MoEs. Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large, sparse models and open new opportunities for composable architectures.
Original Article
View Cached Full Text

Cached at: 05/09/26, 02:30 AM

Paper page - EMO: Pretraining Mixture of Experts for Emergent Modularity

Source: https://huggingface.co/papers/2605.06663

Abstract

EMO is a Mixture-of-Experts model that enables modular deployment by grouping similar domain tokens with shared experts, achieving performance comparable to standard MoEs while allowing significant expert pruning without performance degradation.

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities, e.g., code, math, or domain-specific knowledge.Mixture-of-Experts(MoEs) seemingly offer a potential alternative by activating only a subset ofexpertsper input, but in practice, restricting inference to a subset ofexpertsfor a given domain leads to severe performance degradation. This limits their practicality in memory-constrained settings, especially as models grow larger and sparser. We introduce EMO, an MoE designed for modularity-the independent use and composition of expert subsets-without requiring human-defined priors. Our key idea is to encouragetokens from similar domains to rely on similarexperts. Sincetokens within a document often share a domain, EMO restricts them to selectexpertsfrom a shared pool, while allowing different documents to use different pools. This simple constraint enables coherent expert groupings to emerge duringpretrainingusingdocument boundariesalone. We pretrain a 1B-active, 14B-total EMO on 1Ttokens. As a full model, it matches standard MoE performance. Crucially, it enablesselective expert use: retaining only 25% (12.5%) ofexpertsincurs just a 1% (3%) absolute drop, whereas standard MoEs break under the same setting. We further find that expert subsets in EMO specialize at semantic levels (e.g., domains such as math or code), in contrast to the low-level syntactic specialization observed in standard MoEs. Altogether, our results demonstrate a path toward modular, memory-efficient deployment of large,sparse modelsand open new opportunities forcomposable architectures.

View arXiv pageView PDFProject pageGitHub7Add to collection

Get this paper in your agent:

hf papers read 2605\.06663

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper10

#### allenai/Emo_1b14b_1T Text Generation• 14B• Updatedabout 11 hours ago • 31 • 6 #### allenai/EMO Text Generation• 14B• Updatedabout 10 hours ago • 3 #### allenai/Emo_1b14b_130B Text Generation• 14B• Updatedabout 11 hours ago • 68 • 2 #### allenai/StdMoE_1b4b_130B Text Generation• 4B• Updatedabout 11 hours ago • 23 • 2 Browse 10 models citing this paper## Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.06663 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.06663 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Mixture of Experts (MoEs) in Transformers

Hugging Face Blog

Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.

new MoE from ai2, EMO

Reddit r/LocalLLaMA

AI2 released EMO, a Mixture of Experts language model with 1B active parameters out of 14B total, trained on 1 trillion tokens and featuring document-level routing where experts cluster around domains.