@levidiamode: Day 138/365 of GPU Programming One of my favorite lectures I've watched this year is Stanford's CS336 lecture 7 on GPU …
Summary
A learner shares enthusiasm for Stanford CS336 lecture 7 on GPU parallelism, which covers fundamental operations and connects them to multi-GPU setups and parallelism techniques like tensor, data, and pipeline parallelism.
View Cached Full Text
Cached at: 05/22/26, 05:56 PM
Day 138/365 of GPU Programming
One of my favorite lectures I’ve watched this year is Stanford’s CS336 lecture 7 on GPU parallelism.
It builds up the foundations by teaching the fundamental operations like broadcasting, scattering, gathering, reducing, all-gathering and all-to-all from first principles and gives you an intuition for how these connect to the multi GPU setup in part 1 of the lecture.
It then connects these principles to tensor, data and pipeline parallelism in part 2.
The reason why I love the teaching methodology of this particular lecture is that it goes into the practical details right away and hands you the code for what this conceptually would look like if you were to see it in a codebase.
levi (@levidiamode): Day 137/365 of GPU Programming
Haven’t done a review day in a while, so taking some time looking at my notes over the last few days and rereading various papers I am still wrapping my head around.
The last week has been really useful in gaining an initial understanding of
Similar Articles
@levidiamode: 183/365 of GPU Programming This 4.5 hour lesson on CUDA + ThunderKittens by @bfspector (TK co-author, Stanford PhD stud…
A highly recommended 4.5-hour GPU programming lesson on CUDA and ThunderKittens by Ben Spector, offering an in-depth, behind-the-scenes look at kernel optimization.
@levidiamode: 157/365 of GPU Programming Another FlashAttention4 resource that's been really helpful for me is the talk @charles_irl …
A daily GPU programming thread highlights a talk by Charles_irl that reverse-engineers FlashAttention4 code before the paper release, praising the Modal team's deep code dissection and inferences about the forward pass.
@Hesamation: Stanford dropped their latest course on Parallel Programming, GPU, and CUDA. 24 hours, 19 lessons. this is one of the h…
Stanford has released a free course on Parallel Programming, GPU, and CUDA, consisting of 24 hours of content across 19 lessons, covering GPU architecture, CUDA, and performance optimization.
@levidiamode: 163/365 of GPU Programming Looking at a few different agentic GPU kernel optimization systems today. The two I'm most i…
A tweet discussing two agentic GPU kernel optimization systems: Auto GPU Kernel by @dogacel0 and Kernel Design Agents from @songhan_mit's lab, both winners at the MLSys Sparse Attention FlashInfer competition. The thread highlights different approaches using subagents and Claude skills for GPU programming.
@levidiamode: 158/365 of GPU Programming I think I understand the high level differences between the FlashAttention 2, 3 and 4 forwar…
The author documents their progress in learning GPU programming, focusing on understanding the high-level differences between FlashAttention 2, 3, and 4 forward passes, and lists several low-level concepts they need to explore further.