@charles_irl: did you know that MoEs like ChatGPT only use 10% of their brain
Summary
A tweet notes that Mixture-of-Experts models like ChatGPT only use a fraction of their parameters at a time, humorously comparing it to the '10% of brain' myth.
Similar Articles
@_avichawla: https://x.com/_avichawla/status/2100876555409039605
The article explains the engineering aspects of Mixture-of-Experts (MoE) inference, detailing token routing, expert batching, GPU distribution, and performance trade-offs for efficient serving.
ChatGPT's image model is better at math than most people
The article highlights that ChatGPT's image model demonstrates superior mathematical reasoning capabilities compared to most humans.
Are the rich RAM /poor GPU people wrong here?
Discusses the trade-off between dense and Mixture-of-Experts (MoE) models for local AI, noting that high-RAM users have limited MoE options beyond Qwen 3.5 122B, and questioning if large GPU is the only viable path.
@rohanpaul_ai: A large MoE model may be wasting half its expert compute on tokens that barely need expert help. In this paper 50% of e…
A new method called Zero-Expert Self-Distillation Adaptation (ZEDA) allows MoE models like Qwen3 and GLM to skip half their expert computations on easy tokens with minimal accuracy loss, achieving ~20% inference speedup by adding dummy experts that output nothing.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.