@akshay_pachaar: how do you know whether your GPU is compute-bound or memory-bound? here's a simple explanation: your model weights sit …

how do you know whether your GPU is compute-bound or memory-bound? here's a simple explanation: your model weights sit in HBM (high bandwidth memory), the large pool of memory on the GPU board. no arithmetic happens there. before any multiplication can run, those weights have to travel from HBM into the arithmetic units, and that trip is the slowest thing the chip does. so the real question is how much work you get out of each weight once it has made the trip. that gives you one ratio. count how many arithmetic operations something performs, then count how many bytes it pulled out of HBM to perform them. divide the first by the second and you get its work per byte, which is the horizontal axis in the visual. every chip has a break-even value for that ratio, and it is just peak arithmetic divided by peak memory bandwidth. take an H100. it performs 989 trillion operations per second at 16-bit precision, and it can pull 3.35 trillion bytes per second out of memory. divide one by the other and you get 295 operations per byte, which is the 300 marked on the chart below. that number says the chip can afford about 300 operations for every byte it fetches. anything cheaper leaves the arithmetic units idle, waiting on the next delivery. generating one token for a single request sits at the far left of the red line. every weight is fetched from HBM, gets one multiply and one add (2 operations), and is then dropped. each weight takes two bytes at 16-bit precision, so that is one operation per byte, roughly 300 times below break-even. this is why the red segment slopes. along it your speed is set entirely by how fast bytes arrive, so a chip with more arithmetic capability buys you nothing. batching is how you move right. run 32 requests together and each weight, fetched exactly once, does its multiply and add against 32 different values before being dropped. the traffic out of HBM is unchanged, and the work you got from it went up 32 times. a long prompt does the same thing, except the many values come from the tokens of one sequence instead of separate requests. past the break-even point the line goes flat, which is the green region. processing a long prompt and training both land there, since each fetched byte now feeds enough operations that the arithmetic units become the slow stage and the memory path has capacity to spare. so the two halves ask for opposite fixes. left of the line you cut bytes moved or reuse each fetch harder, and right of it you need faster arithmetic or better algorithms. the threshold belongs to the hardware. where your workload sits relative to it belongs to you. I wrote the full breakdown of how a modern GPU works, and the article is quoted below. stay tuned for more on this!