AMD Instella-MoE-16B-A3B
Summary
AMD released an open-source Mixture-of-Experts model Instella-MoE-16B-A3B on HuggingFace.
Similar Articles
amd/Instella-MoE-16B-A3B-Think
AMD releases Instella-MoE, a fully open 16B-parameter Mixture-of-Experts language model with 2.8B active parameters, trained from scratch on AMD Instinct GPUs and released across all training stages from pre-training to reinforcement learning.
tencent/Hy3
Tencent released Hy3, a 295B-parameter Mixture-of-Experts model with 21B active parameters, under Apache 2.0 license, outperforming similar-size models and rivaling larger open-source models with 2-5x parameters.
MobileMoE: Scaling On-Device Mixture of Experts
MobileMoE introduces efficient on-device mixture-of-experts language models with sub-billion parameters, achieving better performance and efficiency than dense baselines and existing MoE models. The models are trained on open-source datasets and demonstrate significant speedups on commodity smartphones.
Emergent Modularity in Mixture-of-Experts Models (8 minute read)
Ai2 releases EMO, a 14B-parameter mixture-of-experts language model trained to develop emergent modularity. It allows using a small subset of experts for specific tasks while maintaining near full-model performance.
Mixture of Experts (MoEs) in Transformers
Hugging Face blog post explaining Mixture of Experts (MoEs) architecture in Transformers, covering the shift from dense to sparse models, weight loading optimizations, expert parallelism, and training techniques for MoE-based language models.