How to Loop MoE: Flatten the Experts, Untie the Attention
Summary
A preprint introducing Foil, which flattens MoE experts into fewer layers with more experts per layer and more recurrent passes, plus untied attention per pass, achieving lower pretraining loss at matched parameters and compute.
View Cached Full Text
Cached at: 10/06/26, 12:54 PM
Paper page - How to Loop MoE: Flatten the Experts, Untie the Attention
Source: https://huggingface.co/papers/2609.35751 Looped Transformers reuse the same block across multiple passes, while sparse Mixture-of-Experts models store many experts but activate only a few for each token. These two ideas are naturally complementary: every new pass gives a token another routing decision, allowing it to reach different experts and expert combinations without adding expert parameters. But this raises an open design question: under a fixed parameter and compute budget, how should experts be distributed across layers and passes; and which components should be shared across passes?
Our new preprint, โHow to Loop MoE: Flatten the Experts, Untie the Attention,โ answers this question with Foil.
๐ข๐๐ฟ ๐ถ๐ฑ๐ฒ๐ฎ: Flatten the experts. Loop more. Untie the attention.
At each flattening step, Foil uses: โ half as many expert layers โ twice as many experts per layer โ twice as many recurrent passes โ separate attention parameters for each pass while keeping total parameters, expert compute per token, and effective depth fixed.
We go from: 8 experts ร 8 layers ร 2 passes to: 64 experts ร 1 layer ร 16 passes
๐ฅ๐ฒ๐๐๐น๐๐: โ Every Foil configuration achieves lower pretraining loss than the baseline at 20B tokens. โ ๐๐ ๐ญ๐ฌ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐, ๐น๐ผ๐๐ ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ฒ๐ ๐บ๐ผ๐ป๐ผ๐๐ผ๐ป๐ถ๐ฐ๐ฎ๐น๐น๐ ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐ฑ๐ฒ๐ด๐ฟ๐ฒ๐ฒ ๐ผ๐ณ ๐ณ๐น๐ฎ๐๐๐ฒ๐ป๐ถ๐ป๐ด ๐ฎ๐ ๐บ๐ฎ๐๐ฐ๐ต๐ฒ๐ฑ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐ ๐ฎ๐ป๐ฑ ๐ฐ๐ผ๐บ๐ฝ๐๐๐ฒ. The fully flattened Foil ends 0.012 nat below the baseline, with downstream accuracy on par or better. โ Untying attention becomes increasingly valuable as the model gets flatter. At the fully flattened shape, it lowers loss by 0.049 nat and improves mean accuracy on three representative downstream tasks by 3.3 points over tied attention at the same shape, with no additional compute.
โ More experts per layer make additional loops more useful; and more loops make additional experts more useful. The two amplify each other. โ Load balance alone is not enough; it should be read together with routing confidence. In our experiments, routing confidence moves with looping gain, and its per-pass peak may signal diminishing returns from further looping or flattening.
Similar Articles
ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration
ExFold is a unified training-free framework that accelerates MoE model inference by folding excluded expert contributions into retained experts, achieving up to 1.41ร speedup while maintaining high quality.
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
FlexMoE proposes a one-for-all nested intra-expert pruning method for MoE language models, enabling multiple deployable subnetworks from a single training run with minimal performance loss.
Post-Trained MoE Can Skip Half Experts via Self-Distillation
ZEDA is a low-cost framework that converts post-trained static MoE models into dynamic ones by injecting zero-output experts and using self-distillation, achieving over 50% expert FLOP reduction with marginal accuracy loss on benchmarks.
Less is MoE: Trimming Experts in Domain-Specialist Language Models
This paper introduces Fisher-MoE, a method that compresses Mixture-of-Experts models by trimming intermediate dimensions within FFN layers using Fisher importance, achieving 45% weight memory reduction and 21% throughput improvement without significant capability loss.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.