New set of FP4 attention kernels for B300, achieving up to 1.69x speedup over FA4
Summary
The FastVideo team releases new FP4 attention kernels for B300, achieving up to 1.69x speedup over FlashAttention 4.
View Cached Full Text
Cached at: 07/14/26, 04:24 AM
(1/6) The FastVideo team is excited to release a new set of FP4 attention kernels for B300, achieving up to 1.69x speedup over FA4!
(2/6) On B300, the MUFU (exp2) throughput doubles, removing softmax from the bottleneck for BF16, but it remains a bottleneck for FP4. We use the new http://tcgen05.ld.red instructions to compute row max during TMEM -> registers load instead of the slow CUDA core instructions to alleviate the bottleneck, leading to 8%/12% perf gain for FP4/FP8 PV.
(3/6) We also spent time fixing the precision of our RTX 5090 kernel in causal mode (https://github.com/hao-ai-lab/FastVideo/pull/1506…, based on SageAttention 3), which enables end-to-end FP4 LLM serving on SM120!
(4/6) Currently, we are actively working on QAD (Quantization-aware Distillation) for larger diffusion models such as Wan2.1-14B and NVIDIA Cosmos.
(5/6) Attn-QAT paper: https://arxiv.org/abs/2603.00040 Attn-QAT Blog: https://haoailab.com/blogs/attn-qat/ B300 kernel: https://github.com/hao-ai-lab/flash-attention-fp4/blob/fp4_B300/flash_attn/cute/flash_fwd_sm100_fp4.py… FastWan-QAD Blog: https://haoailab.com/blogs/fastwan-qad/…
(6/6) We want to thank to the entire FastVideo team as well as @NVIDIAAI for sponsoring the FastVideo project!
@EdenTan20 will be at ICML this week–welcome to chat if interested!
This will be the location: 7.9 10:30am poster session HALL A #2800
Can Attention-FFN Disaggregation still win on the newest rack-scale GPU systems? We built FastAFD, an open-source AFD runtime for GB200 NVL72: 72 Blackwell GPUs in one NVLink domain. It improves per-GPU decode throughput by 1.35-1.45×. Code: https://github.com/hao-ai-lab/FastAFD… Blog: https://haoailab.com/blogs/fastafd/
Similar Articles
FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation
FVAttn is a training-free sparse attention system that uses runtime load balancing to improve distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism for video generation, achieving up to 4.41x attention speedup and 2.11x inference speedup over FlashAttention on step-distilled Wan2.2 I2V while maintaining competitive video quality.
Anyone using Flash Attention 2 (ai-bond) on their V100's? How is the performance?
A user benchmarks a V100-compatible port of Flash Attention 2, reporting 3x-17x speedups and up to 94% memory reduction over default PyTorch attention.
@xenovacom: Opus 4.7 just wrote a custom WebGPU kernel that runs Qwen3.5 up to 13x faster using a fused LinearAttention op! Agentic…
Opus 4.7 auto-generated a custom WebGPU kernel that accelerates Qwen3.5 inference up to 13× via fused LinearAttention, now shipping in Transformers.js v4.2.0.
@AaronWeiHuang: Our new blog looks at how FP4 is moving beyond compression into a practical primitive for training and inference across…
NVIDIA's blog details how FP4, with the NVFP4 format and Blackwell hardware, has evolved from a compression trick to a practical primitive for training and inference across LLMs and diffusion models, achieving near 16-bit accuracy.
LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation
LongLive-2.0 introduces an NVFP4-based parallel infrastructure for long video generation, achieving up to 2.15x training speedup and 1.84x inference speedup with a 5B model reaching 45.7 FPS.