@h100envy: CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $12…
Summary
A CMU PhD who developed the kernels now used by NVIDIA in TensorRT-LLM explains fast attention, covering fused CUDA kernels, FlashInfer, Triton, and paged-KV attention, enabling more tokens per second on the same GPU.
View Cached Full Text
Cached at: 07/02/26, 04:25 PM
CMU PhD who built the kernels NVIDIA now ships in TensorRT-LLM explained fast attention in 68 minutes - better than $1200 GPU programming courses.
pick the attention pattern -> generate a fused CUDA kernel -> drop it into vLLM/SGLang -> same GPU, way more tokens per second.
That loop is why FlashInfer now powers inference at NVIDIA, vLLM, SGLang, and half the serving stacks you use.
FlashInfer + Triton + JIT-compiled kernels + paged-KV attention - that’s the stack.
PyTorch core engineer at Meta turned CUDA kernel writing into a sport in 13 minutes - better than $1500 GPU programming bootcamps.
profile the kernel -> find the bottleneck -> rewrite -> benchmark -> merge the winning code into PyTorch.
That loop is how the open community now beats hand-tuned vendor kernels.
GPU MODE community + KernelBot competition + winning kernel merged into the framework - that’s the stack.
Watch it, then steal the loop below.
Similar Articles
@ekzhang1: me looking at people like this guy who write real gpu kernels :)
AI model Claude was used to write a FlashAttention forward kernel using the pyptx DSL, achieving near-parity performance with hand-tuned FlashAttention-4 on NVIDIA B200 hardware.
@charles_irl: Somehow missed this one in the hustle and bustle. Very cool demo!
A developer built a 12M parameter LLM using a custom ML framework with a Rust backend and CUDA kernels, including Flash Attention and AdamW, and trained it from scratch.
@Alacritic_Super: If you are serious about LLM inference, study FlashAttention. It's one of the most important optimizations behind moder…
A tweet recommending studying FlashAttention for LLM inference, highlighting its importance in optimizing GPU memory traffic and speeding up attention mechanisms, with links to the GitHub repository and papers for FlashAttention, FlashAttention-2, and FlashAttention-3.
@ZhihuFrontier: GPU programming changed because Tensor Cores became too fast to feed Zhihu contributor THU-PACMAN实验室 shared a sharp bre…
A detailed analysis of how NVIDIA GPU programming evolved from Volta to Blackwell, highlighting the shift from synchronous thread models to asynchronous dataflow and the challenges of feeding Tensor Cores. The article discusses new hardware features like TMA, TMEM, and tcgen05 MMA, and shows how modern kernels like FlashAttention-3 and FlashMLA exploit these changes for higher utilization.
@maxxfuu: Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unopti…
A developer shares their day 6 of inference engineering, writing a CUDA kernel for 1D convolution, explaining PagedAttention's memory efficiency, and providing an overview of GPU memory hierarchy (global, register, local, constant, shared).