Tag
FlashMask-3 introduces a Hopper-optimized mask-aware sparse attention kernel and zero-communication distributed mask localization for long-context training, achieving major TFLOPs and throughput gains over FlashMask, FlexAttention, MagiAttention, and Megatron-Core baselines, including 7.3-11.0% end-to-end speedup on a 118B MoE trained on 128 H100 GPUs at 128K context.
Jetha Chan disassembled the Attention kernel for the SM100 data center GPU, identified and fixed inefficiencies, resulting in significant performance improvements.