attention-kernels

FLAMIA 提出了一族面向 NVIDIA Blackwell GPU 的混合精度注意力内核，利用 MXFP8 与 NVFP4/FP8 路径，在避免因果信息泄漏的同时加速 Transformer 训练，在 B200/B300 上实现约 1.06–1.42 倍的前向与反向注意力加速。

The FastVideo team releases new FP4 attention kernels for B300, achieving up to 1.69x speedup over FlashAttention 4.

This article explores whether FlashAttention-3/4 optimizations benefit RTX GPUs, concluding that FA-2 is the ceiling due to hardware limitations on consumer cards.

A detailed blog post dissecting ThunderKittens, a compact DSL for high-performance AI kernels, including a bottom-up analysis of its abstractions and a benchmark implementing a non-causal attention prefill kernel that outperforms FlashAttention-2 by ~1.55x and matches FlashAttention-3.

Moonshot AI releases FlashKDA, an open-source CUTLASS-based implementation of Kimi Delta Attention kernels that delivers 1.72×–2.22× prefill speedup on H20 GPUs.