@h100envy: Dan Fu co-wrote FlashAttention with Tri Dao. Then he co-built Hyena, Monarch Mixer, and ThunderKittens. Now he's distin…
Summary
Profiles Dan Fu, a key contributor to high-performance kernels like FlashAttention, Hyena, Monarch Mixer, and ThunderKittens, now a distinguished researcher at Together AI whose work is used in ChatGPT, Claude, and Gemini.
View Cached Full Text
Cached at: 06/28/26, 06:13 PM
Dan Fu co-wrote FlashAttention with Tri Dao. Then he co-built Hyena, Monarch Mixer, and ThunderKittens. Now he’s distinguished researcher at Together AI.
Four kernels you can find inside the inference stack running ChatGPT, Claude, and Gemini. One person, all in your hot path.
His obsession: squeeze more tokens per second out of the same GPU. ThunderKittens is a 100-line CUDA DSL that beats hand-written kernels.
Everyone is renting more GPUs. He keeps making the ones you have count for more.
Similar Articles
@ekzhang1: me looking at people like this guy who write real gpu kernels :)
AI model Claude was used to write a FlashAttention forward kernel using the pyptx DSL, achieving near-parity performance with hand-tuned FlashAttention-4 on NVIDIA B200 hardware.
@hamzaelshafie: New in-depth blog post: "Dissecting ThunderKittens: Anatomy of a Compact DSL for High-Performance AI Kernels" This post…
A detailed blog post dissecting ThunderKittens, a compact DSL for high-performance AI kernels, including a bottom-up analysis of its abstractions and a benchmark implementing a non-causal attention prefill kernel that outperforms FlashAttention-2 by ~1.55x and matches FlashAttention-3.
@h100envy: CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $12…
A CMU PhD who developed the kernels now used by NVIDIA in TensorRT-LLM explains fast attention, covering fused CUDA kernels, FlashInfer, Triton, and paged-KV attention, enabling more tokens per second on the same GPU.
@ariG23498: I was fascinated when I first heard about kernel fusion from @cHHillee's blog post "Making Deep Learning Go Brrrr From …
The author shares excitement about kernel fusion and demonstrates using HuggingFace's kernels project to profile a GeGLU FFN fused Liger kernel, noting the profile's beauty.
@h100envy: Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs. She also built Fle…
Ying Sheng co-wrote SGLang, the inference engine now serving Grok at xAI on a hundred thousand GPUs, achieving 5x cost cuts over DeepSeek's API; she also built FlexGen and helped build Chatbot Arena.