Is there a limit on the number of active parameters in an MoE model?
Summary
Discussion on the limit of active parameters in Mixture-of-Experts (MoE) models, questioning whether there is a cap on active parameter count beyond which quality doesn't improve.
Similar Articles
What is the point of MoE models, beyond being faster?
A discussion about the advantages of Mixture of Experts (MoE) models over dense models beyond speed, considering RAM constraints and scaling limits.
124B total but only ~5B active-this is exactly the shape I want for my box. Are low-active MoEs just the local sweet spot now?
A discussion on the merits of Mixture of Experts models with 124B total parameters but only ~5B active, suggesting that low-active MoEs may be the optimal local sweet spot.
Output Dilution: Redundant but Fragile Representations in MoE Models
This paper reveals that Mixture-of-Experts models encode moral content as robustly as dense models in probing but are far more fragile due to output dilution, where aggregated expert outputs dilute the signal, making representations vulnerable to noise.
How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization
This paper develops a principled scaling theory for Mixture-of-Experts (MoE) architectures, introducing the Maximally Scale-Stable Parameterization (MSSP) that ensures stable training and hyperparameter transfer across width, depth, expert width, and number of experts, validated by experiments.
Emergent Modularity in Mixture-of-Experts Models (8 minute read)
Ai2 releases EMO, a 14B-parameter mixture-of-experts language model trained to develop emergent modularity. It allows using a small subset of experts for specific tasks while maintaining near full-model performance.