@serendip410: New blog on saving experts from an untold, silent failure mode in ultra-sparse MoEs : While pursuing ultra-sparse MoE t…
Summary
The blog post identifies a silent expert death failure mode in ultra-sparse MoE training and introduces LM Loss as Auxiliary Loss (LLAL) to rescue lower-layer experts, improving validation loss and downstream performance.
View Cached Full Text
Cached at: 08/25/26, 08:11 PM
New blog on saving experts from an untold, silent failure mode in ultra-sparse MoEs :
While pursuing ultra-sparse MoE training (top-8/768), we found that lower-layer experts can become functionally useless while training/validation loss and load balance remain normal. All experts in these layers have much smaller weight norms than those in healthy layers. We call this silent expert death[Figure 1].
And it wasn’t just us. Auditing public MoEs, we found similar early-layer collapse signatures. For example, MiMo-v2.5-Pro and Qwen3.5-397B-A17B [Figure 2]. Benchmarks can hide unused capacity.
Why? This is not merely a load-balancing failure. We found the learning signal itself disappears. Shallow-expert momentum (the accumulated gradients), as well as the square root of second moment, fell by up to ~3 orders of magnitude while deeper layers stayed stable[Figure 1]. By the time norms collapse, the model has already largely stopped relying on the lower pathway.
We introduce LM Loss as Auxiliary Loss (LLAL): temporarily connect an early MoE layer to the LM head during the first few thouands steps then remove it. On 60B /180B models, this simple strategy rescued the lower layers—and the gain persisted after removal. It improved nearly every aspect of expert learning: • Better validation loss [Figure3] • No silent expert collapse [Figure 4] • Better downstream metrics [Figure 4] • Markedly better load balance
Check out our blog for more details! A bunch of experiments inside—enjoy the 40-minute read :)
https://alltoall.notion.site/save-lower-layer-moe-experts-llal…
Similar Articles
Post-Trained MoE Can Skip Half Experts via Self-Distillation
ZEDA is a low-cost framework that converts post-trained static MoE models into dynamic ones by injecting zero-output experts and using self-distillation, achieving over 50% expert FLOP reduction with marginal accuracy loss on benchmarks.
ACE: Adapter Consolidation across Experts for Parameter-Efficient Fine-Tuning of MoE LLMs
ACE introduces a method to consolidate redundant adapters across experts in MoE large language models for more efficient parameter-efficient fine-tuning, achieving up to 1.48× training speedup without increasing peak memory.
@rohanpaul_ai: A large MoE model may be wasting half its expert compute on tokens that barely need expert help. In this paper 50% of e…
A new method called Zero-Expert Self-Distillation Adaptation (ZEDA) allows MoE models like Qwen3 and GLM to skip half their expert computations on easy tokens with minimal accuracy loss, achieving ~20% inference speedup by adding dummy experts that output nothing.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.
Proposed architecture for inferencing sparse MOE models increasing Active parameters using layered + linear decay. Succinct reasoning without any model training or fine tune. [p]
A llama.cpp implementation that expands MoE expert routing beyond native top-K during inference using adaptive thresholds and layer-specific linear decay, without requiring model retraining or fine-tuning.