flashattention

FlashPrefill V2 improves long-context LLM serving through mean-corrected sparse attention and optimized GPU operators, delivering substantial speedups over FlashAttention-2 and dense baselines.

The article clarifies that GPU performance in AI inference is limited by memory bandwidth rather than compute power, using the NVIDIA H100 as an example to explain GPU architecture and its effect on token generation rates.

This article explores whether FlashAttention-3/4 optimizations benefit RTX GPUs, concluding that FA-2 is the ceiling due to hardware limitations on consumer cards.

Profiles Dan Fu, a key contributor to high-performance kernels like FlashAttention, Hyena, Monarch Mixer, and ThunderKittens, now a distinguished researcher at Together AI whose work is used in ChatGPT, Claude, and Gemini.

A visual deep dive into FlashAttention, explaining memory optimization and operator fusion for efficient attention computation in language model training.