@akshay_pachaar: How work is organized inside a GPU. A GPU does not treat a large computation as one job. It keeps dividing that job int…

How work is organized inside a GPU. A GPU does not treat a large computation as one job. It keeps dividing that job into smaller pieces until thousands of simple operations can run together. The easiest way to understand this is to follow the work from the program you launch to the arithmetic the chip performs. 1) A kernel creates the full workload. A kernel is a function meant to run repeatedly across different pieces of data. When you launch one, it creates a grid. The grid represents all the work required for that launch. 2) The grid is divided into thread blocks. Each block owns one portion of the workload. Its threads stay together on the same streaming multiprocessor, or SM, so they can coordinate and exchange values through fast shared memory. The GPU schedules blocks independently. This allows blocks from a large grid to spread across many SMs without needing to coordinate with one another. 3) Each block is divided into warps. A thread is one lane performing the kernel’s instructions on its own data. The hardware collects threads into fixed groups of 32 called warps. A block containing 256 threads therefore becomes eight warps. These warps are the units the SM actually chooses between during execution. All 32 threads in a warp receive the same instruction. They perform it together, but on different values. This is where the GPU gets its width. It also explains why branching can hurt performance. If threads in one warp choose different paths, the GPU must run each path separately while temporarily disabling the threads that took the other one. 4) Warps execute inside an SM. An SM contains compute units, warp schedulers, registers, and shared memory. It can keep many warps resident at once, even though only some execute during a given clock tick. When one warp requests data from slower memory and has to wait, the scheduler selects another ready warp. Switching is extremely cheap because every resident warp already has its state stored on the SM. The GPU does not eliminate memory delays. It hides them by always having another warp ready to run. This hierarchy also explains why workload size matters. Too few blocks leave SMs unused. Too few resident warps leave the scheduler with nothing to execute during a memory stall. The complete path is simple. A kernel creates a grid → The grid contains blocks → Blocks contain warps → Warps contain 32 threads → Blocks are assigned to SMs → Their warps are scheduled onto compute units. That structure is the foundation behind GPU parallelism, latency hiding, and the need for large batches of similar work. I wrote the full breakdown on how GPUs work, why they are organized this way, and what that design means for LLM performance. The article is quoted below.