@matthewjgunton: There are 3 basic levels in the NVIDIA software stack CUDA C++, PTX, and SASS Understanding all 3 helps you know why CU…

There are 3 basic levels in the NVIDIA software stack CUDA C++, PTX, and SASS Understanding all 3 helps you know why CUDA built a trillion dollar moat CUDA C++ - It gives you NVIDIA’s programming model (threads, blocks, shared memory, synchronization) without requiring you to write instructions for an exact GPU architecture. In fact, a good CUDA kernel is often used across multiple generations of NVIDIA hardware. This means the newer chips ship day 1 with useful software ready to go CUDA C++ is lowered into PTX PTX (Parallel Thread Execution) - This is a virtual instruction set. Because they are virtual machine instructions, they allow one CUDA kernel to run across different NVIDIA devices without needing to be rewritten. That being said when we want to use a specific command on a specific part of a chip (like a WGMMA), we will specify it in PTX so that there is no confusion during compilation about what we want to use. Naturally, this choice comes with the trade-off of portability. Once we specify exactly what hardware component we want, we cannot port that kernel to an NVIDIA device that doesn't have that component PTX is lowered to SASS SASS - NVIDIA's native machine code. This is the end of the road. We take our virtual instruction set and bring it down into the exact executable code that will run on our specific machine. This is the least portable version of the code, but of course the most complete. Coding at this level is very painful, so very few people work here CUDA's great advantage is that you can take a kernel written for Ampere and move it to Hopper on launch day. This lets NVIDIA's software stack compound The challenge is the abstraction. By giving this distance, you rely heavily on your NVIDIA compilers to find fast ways to run your code on different hardware In practice, a developer will try many different strategies to find the fastest way to run their kernel In other words, you have to do a manual search At Luminal, we're creating a compiler that will handle this search entirely automatically and deterministically