@samsja19: we made our training stack significantly faster by fusing computation and communication in the MoE layer, adding backwa…
Summary
Prime Intellect made DeepSeek's MegaMoE kernels trainable, fusing computation and communication in the MoE layer to achieve up to 55% faster MoE training than TorchTitan, with support for BF16, MXFP8, and FP8 on Blackwell and Hopper GPUs.
View Cached Full Text
Cached at: 10/29/26, 06:03 PM
we made our training stack significantly faster by fusing computation and communication in the MoE layer, adding backward kernels to deepseek MegaMoE
available for BF16, MXFP8, and FP8 on Blackwell and Hopper GPUs
Prime Intellect (@PrimeIntellect): Introducing Prime Mega MoE.
We made DeepSeek’s Mega MoE kernel trainable. The results on two Qwen3 models:
• Up to 55% faster MoE training than TorchTitan • Up to 34% faster hosted training with prime-rl
Similar Articles
40% speedup of MoE training with faster megakernel, by cursor, of all people (for B200s)
Cursor open-sources Mixture-of-Kittens (MoK), a deterministic MoE training megakernel for NVIDIA Blackwell GPUs that fuses computation and communication, delivering up to 2.37x speedup over baseline implementations.
DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism
This paper introduces DisagMoE, a system for MoE training that optimizes computation-communication overlap by disaggregating attention and FFN layers across GPU groups. Implemented on Megatron-LM, it achieves up to 1.8x speedup on H800 clusters by addressing inter-node communication bottlenecks.
@techNmak: The smartest way to run a giant MoE model is not to add more GPUs. It is to stop treating every expert as GPU-worthy. L…
KTransformers is a framework that optimizes inference and fine-tuning of large Mixture-of-Experts models by dynamically placing only active experts on the GPU while keeping the rest in CPU memory, enabling large models like DeepSeek-V3 to run on limited consumer GPU memory.
@samsja19: prime-rl can now train 1T parameters MoE blazingly fast, under 5 minutes per step, or 1k steps in ~3 days To achieve th…
Prime Intellect released prime-rl v0.6.0, enabling reinforcement learning at trillion-parameter MoE scale with sub-5-minute step times and optimized inference, training, and rollout.
Inside Microsoft's Frontier Pre-Training Stack (18 minute read)
微软公布了其最新的 MoE 基础模型 MAI-Base-2（80B 激活参数 / 2.8T 总参数），在自研评测中超越所有开源权重模型，预训练 FLOPs 效率比 DeepSeek-V4-Pro-Base 高 4-12 倍，使用 16k 张 GB200 GPU 在 38 天内完成训练。