hopper-gpu

FlashMask-3 introduces a Hopper-optimized mask-aware sparse attention kernel and zero-communication distributed mask localization for long-context training, achieving major TFLOPs and throughput gains over FlashMask, FlexAttention, MagiAttention, and Megatron-Core baselines, including 7.3-11.0% end-to-end speedup on a 118B MoE trained on 128 H100 GPUs at 128K context.