@gpuwaster: 61/100 of GPU Grind going through this GTC 2020 lecture: Developing CUDA kernels to push Tensor Cores to the Absolute L…
Summary
The author, as part of a 100-part GPU learning series, reviews a GTC 2020 lecture by Andrew Kerr on developing high-performance CUDA kernels for Tensor Cores on NVIDIA A100, discussing techniques and the trade-off between raw CUDA and CUTLASS.
View Cached Full Text
Cached at: 07/12/26, 10:53 AM
61/100 of GPU Grind
going through this GTC 2020 lecture: Developing CUDA kernels to push Tensor Cores to the Absolute Limit on NVIDIA A100
definitely an interesting title since that’s exactly what i’m trying to do, it’s a talk given by Andrew Kerr from the CUTLASS team that recaps the strategies behind optimizing GEMM on Ampere, explaining how mma instructions, ldmatrix instructions and cp.async works and what it takes to create a good memory pipeline
I was looking for new tricks when starting the video, but it’s all things i’ve already been doing (well they might be doing it better though; especially for the swizzling part)
the main conclusion i draw from this is that writing such a kernel would be so much easier using cutlass.. it’s kinda the point of the lecture i mean it’s given by the cutlass team but i feel like i wouldn’t be learning as much if i wasn’t doing it in raw cuda, am i torturing myself for nothing?
link: https://developer.nvidia.com/gtc/2020/video/s21745-vid…
GTC 2020: Developing CUDA kernels to push Tensor Cores to the Absolute Limit on NVIDIA A100
Source: https://developer.nvidia.com/gtc/2020/video/s21745-vid
After clicking “Watch Now” you will be prompted to login or join.
Click “Watch Now” to login or join the NVIDIA Developer Program.
Developing CUDA kernels to push Tensor Cores to the Absolute Limit on NVIDIA A100
Andrew Kerr, NVIDIA
GTC 2020
NVIDIA Ampere GPU Architecture pushes the performance envelope by doubling the math throughput of Tensor Cores for mixed precision and also adds support for double precision, Tensor Float 32, and bfloat16 data types. We’ll describe how to implement high-performance CUDA kernels using Tensor Cores on A100, applying techniques such as register blocking, software pipelining, and carefully constructed memory layouts to avoid bank conflicts. Then we’ll describe abstractions for programming Tensor Cores available in CUTLASS, as well as other new features. This talk is intended for advanced CUDA C++ programmers who are eager to write kernels pushing Tensor Cores to peak performance. We recommend that you review previous presentations on this topic such as the introduction to CUTLASS (GTC 2018) and Programming Volta Tensor Cores in CUTLASS (GTC 2019).
thom✨ (@gpuwaster): 60/100 of GPU Grind
following stanford cs149 with lecture 4 covering the basics on creating parallel programs, going from amdahl’s law, problem decomposition into tasks (by the programmer), task assignment (typically by the compiler), and how that works in ISPC which is the
Similar Articles
@levidiamode: 183/365 of GPU Programming This 4.5 hour lesson on CUDA + ThunderKittens by @bfspector (TK co-author, Stanford PhD stud…
A highly recommended 4.5-hour GPU programming lesson on CUDA and ThunderKittens by Ben Spector, offering an in-depth, behind-the-scenes look at kernel optimization.
@levidiamode: Day 138/365 of GPU Programming One of my favorite lectures I've watched this year is Stanford's CS336 lecture 7 on GPU …
A learner shares enthusiasm for Stanford CS336 lecture 7 on GPU parallelism, which covers fundamental operations and connects them to multi-GPU setups and parallelism techniques like tensor, data, and pipeline parallelism.
@ZhihuFrontier: GPU programming changed because Tensor Cores became too fast to feed Zhihu contributor THU-PACMAN实验室 shared a sharp bre…
A detailed analysis of how NVIDIA GPU programming evolved from Volta to Blackwell, highlighting the shift from synchronous thread models to asynchronous dataflow and the challenges of feeding Tensor Cores. The article discusses new hardware features like TMA, TMEM, and tcgen05 MMA, and shows how modern kernels like FlashAttention-3 and FlashMLA exploit these changes for higher utilization.
@Jolyne_AI: GitHub Open-Source CUDA System Tutorial: LeetCUDA (From Beginner to Advanced, All in One) 200+ Progressive CUDA Kernel Practice Problems, with a companion HGEMM library achieving 98%–100% of cuBLAS performance. Plus 100+ articles on high-performance computing...
LeetCUDA is an open-source CUDA system tutorial on GitHub, featuring over 200 progressive CUDA Kernel practice problems and 100+ high-performance computing blog posts. Its companion HGEMM library achieves 98%–100% of cuBLAS performance, making it ideal for CUDA beginners and AI engineers to systematically master CUDA optimization.
@maxxfuu: Day 6/90 of Inference Engineering I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unopti…
A developer shares their day 6 of inference engineering, writing a CUDA kernel for 1D convolution, explaining PagedAttention's memory efficiency, and providing an overview of GPU memory hierarchy (global, register, local, constant, shared).