Tag
一篇技术博客深入解释了投机解码为什么快：并非因为总FLOPs减少（实际上反而增加），而是因为它改变了GPU的工作类型——在低批量场景下把原本空闲的算力从内存瓶颈（memory-bound）转为计算瓶颈（compute-bound），用单序列的'深度'换取了批量解码的'广度'。
The article explains how to determine if a GPU workload is compute-bound or memory-bound by analyzing operations per byte fetched from HBM, using NVIDIA's H100 as an example, and discusses how batching and prompt length affect performance.
DiffusionGemma is out; it's compute-bound and 4x faster than other Gemma-4 models with 1k tok/s on H100, and excels at coding tasks including 3D generation and front-end.
A comprehensive blog post explaining how to optimize deep learning performance by understanding three key components: compute, memory bandwidth, and overhead, using first principles to identify the performance regime and focus on effective optimizations.