Tag
A detailed analysis of how NVIDIA GPU programming evolved from Volta to Blackwell, highlighting the shift from synchronous thread models to asynchronous dataflow and the challenges of feeding Tensor Cores. The article discusses new hardware features like TMA, TMEM, and tcgen05 MMA, and shows how modern kernels like FlashAttention-3 and FlashMLA exploit these changes for higher utilization.
This article explains the entire process of compiling and launching a CUDA kernel, from source code to hardware execution, using a simple vector addition example and detailing the role of nvcc, PTX, SASS, and ioctls.
A thread recommending the optimal order to read CUDA books, starting with CUDA by Example to build intuition before diving into more advanced texts.
GPU Mode is a learning resource featuring a YouTube series, GitHub repo with slides/notebooks, and a practice website for mastering CUDA programming.
A new book from CMU's Machine Learning Systems course teaches modern GPU programming for ML systems, covering Blackwell architecture, GEMM, and FlashAttention using the TIRx Python DSL.
Modular announces a free 4-week Mojo course taught by the Mojo team, covering language fundamentals to GPU programming, starting July 9th on YouTube.
A tweet sharing a naive CUDA softmax implementation using shared memory reduction, noting that reduction is straightforward.
NVIDIA Labs releases cuTile Rust, a tile-based system for writing memory-safe, data-race-free GPU kernels in idiomatic Rust. It extends Rust's ownership model to GPU kernels, JIT-compiles Rust AST to GPU code, and achieves performance close to native CUDA.
A tweet discussing two agentic GPU kernel optimization systems: Auto GPU Kernel by @dogacel0 and Kernel Design Agents from @songhan_mit's lab, both winners at the MLSys Sparse Attention FlashInfer competition. The thread highlights different approaches using subagents and Claude skills for GPU programming.
Implemented a top-k kernel from the kernel design section of the MSA paper, using exp-free comparison and warp-level tree merging with CUDA shuffles. The code is available on GitHub.
The author documents their progress in learning GPU programming, focusing on understanding the high-level differences between FlashAttention 2, 3, and 4 forward passes, and lists several low-level concepts they need to explore further.
A daily GPU programming thread highlights a talk by Charles_irl that reverse-engineers FlashAttention4 code before the paper release, praising the Modal team's deep code dissection and inferences about the forward pass.
This article examines the history of CUDA alternatives like OpenCL and SYCL, explaining why they failed to become dominant in AI compute due to slow committee-driven development and the challenges of open coopetition.
Kazuki Fujii announces the first installment of a blog series on CUDA Programming basics, written in an accessible way, essential for understanding FlashAttention and hardware-aware acceleration techniques.
CUDA 13.3 introduces significant enhancements including Tile C++ support, C++23 standard, improved NVRTC, stable CUDA Python 1.0 APIs, and PTX 9.3 with new fabric instructions and async multimem operations, targeting kernel developers and runtime engineers.
New articles in the GPU Glossary cover CuTe DSL, CUTLASS, and CuTe – tools for writing high-performance GPU kernels on data center GPUs, with examples in Python.
A learner shares enthusiasm for Stanford CS336 lecture 7 on GPU parallelism, which covers fundamental operations and connects them to multi-GPU setups and parallelism techniques like tensor, data, and pipeline parallelism.
Promotional post for the book 'CUDA for Deep Learning' by Elliot Arledge, offering a first chapter summary video that explains GPU performance, the CUDA programming model, and when to write custom CUDA kernels.
A curated list of major books on CUDA programming covering beginner to advanced topics, including C++ and Python, with focus on practical resources for NVIDIA GPU parallel computing.
CUDA-oxide is an experimental Rust-to-CUDA compiler developed by NVIDIA that enables writing safe GPU kernels in idiomatic Rust, compiling directly to PTX without requiring domain-specific languages or foreign bindings.