Tag
Meta Engineering extends FlashAttention-4 with MXFP8 support for NVIDIA Blackwell, achieving up to 2.85 PFLOP/s forward performance and integrating into production training workflows like GEM.
The article details optimization techniques for FlashAttention-4, including an S/P ping-pong method for decode to overlap operations, achieving up to 16% performance gain on NVIDIA Blackwell GPUs.