@charles_irl: Rewriting parallelism is a big move and it'd be nice to make it even faster than we can do with CuTe DSL. FA4 is a very…
Summary
Discussion about rewriting parallelism to improve kernel performance using CuTe DSL and tile programming models for the FA4 (FlashAttention 4) kernel.
View Cached Full Text
Cached at: 06/12/26, 10:58 AM
Rewriting parallelism is a big move and it’d be nice to make it even faster than we can do with CuTe DSL.
FA4 is a very tile-pilled, Tensor Core-maxxing kernel. We’d love to write (and repeatedly rewrite) such kernels with @blelbach and team’s new tile programming models. https://t.co/91fKeCAYYt
Similar Articles
@charles_irl: Last fall, we shared our deep dive on FA4 internals. But we didn't stop at grokking the kernel. Since then, we've been …
A blog post details contributions to FlashAttention-4 to improve its performance for large language model inference, especially for decode-heavy workloads, by adjusting parallelism strategies and supporting irregular memory accesses.
@PatrickToulme: This exercise makes me believe the future of DSLs and compilers is very much agentic. Programming languages and DSLs bu…
Claude Fable used the pyptx DSL to write a FlashAttention forward kernel for NVIDIA B200 that achieves near-parity performance with the hand-tuned CUTLASS kernel, demonstrating the potential for AI agents in compiler and DSL design.
@ekzhang1: me looking at people like this guy who write real gpu kernels :)
AI model Claude was used to write a FlashAttention forward kernel using the pyptx DSL, achieving near-parity performance with hand-tuned FlashAttention-4 on NVIDIA B200 hardware.
@charles_irl: https://x.com/charles_irl/status/2069113412869914944
详细介绍了针对语音克隆模型的W4A4 CUDA内核优化，通过INT4量化和融合LoRA，实现了比FP16快2.6倍的推理速度。
@charles_irl: ^That's a sample of CuTe DSL, which is used in, among others, the FlashAttention-4 kernel. Below is the sample CuTe ker…
A tweet showcasing a CuTe DSL kernel sample that uses layouts to express transposition, part of the FlashAttention-4 kernel.