Tag
A highly recommended 4.5-hour GPU programming lesson on CUDA and ThunderKittens by Ben Spector, offering an in-depth, behind-the-scenes look at kernel optimization.
QuixiAI ported ThunderKittens to Metal, enabling kernel support on MPS and MLX for training models on Mac.
Profiles Dan Fu, a key contributor to high-performance kernels like FlashAttention, Hyena, Monarch Mixer, and ThunderKittens, now a distinguished researcher at Together AI whose work is used in ChatGPT, Claude, and Gemini.
A detailed blog post dissecting ThunderKittens, a compact DSL for high-performance AI kernels, including a bottom-up analysis of its abstractions and a benchmark implementing a non-causal attention prefill kernel that outperforms FlashAttention-2 by ~1.55x and matches FlashAttention-3.