Is there a limit on the number of active parameters in an MoE model?
Summary
Discussion on the limit of active parameters in Mixture-of-Experts (MoE) models, questioning whether there is a cap on active parameter count beyond which quality doesn't improve.
Similar Articles
What is the point of MoE models, beyond being faster?
A discussion about the advantages of Mixture of Experts (MoE) models over dense models beyond speed, considering RAM constraints and scaling limits.
124B total but only ~5B active-this is exactly the shape I want for my box. Are low-active MoEs just the local sweet spot now?
A discussion on the merits of Mixture of Experts models with 124B total parameters but only ~5B active, suggesting that low-active MoEs may be the optimal local sweet spot.
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
This paper develops a principled scaling theory for Mixture-of-Experts (MoE) architectures, introducing the Maximally Scale-Stable Parameterization (MSSP) that ensures stable training and hyperparameter transfer across width, depth, expert width, and number of experts, validated by experiments.
Emergent Modularity in Mixture-of-Experts Models (8 minute read)
Ai2 releases EMO, a 14B-parameter mixture-of-experts language model trained to develop emergent modularity. It allows using a small subset of experts for specific tasks while maintaining near full-model performance.
Why are MoE models so belittled?
Discusses the common perception that MoE models with low active parameters are inferior to dense models, arguing that router effectiveness and architecture nuances matter.