Tag
一篇技术博客深入解释了投机解码为什么快:并非因为总FLOPs减少(实际上反而增加),而是因为它改变了GPU的工作类型——在低批量场景下把原本空闲的算力从内存瓶颈(memory-bound)转为计算瓶颈(compute-bound),用单序列的'深度'换取了批量解码的'广度'。
This paper compares tensor parallelism and KV-cache compression techniques for memory-bound LLM serving, finding that compression is generally cheaper and discusses decision rules based on model size relative to device memory.
The article explains how to determine if a GPU workload is compute-bound or memory-bound by analyzing operations per byte fetched from HBM, using NVIDIA's H100 as an example, and discusses how batching and prompt length affect performance.
An investigation into why dual RTX 5060 Ti GPUs max out at ~50% utilization when running large LLMs like Qwen 27B reveals that memory bandwidth is the bottleneck and layer-by-layer splitting causes idle time, making it a relay race rather than parallel computation.
Explains the two phases of LLM inference - prefill and decode - detailing how GPU bottlenecks shift from compute-bound during prefill to memory-bound during decode, and the importance of KV caching.
CODA reparameterizes memory-bound operations in LLM training to fuse them into the matmul epilogue, achieving near state-of-the-art performance with LLM-generated kernels.
A Stanford lecture on AI inference emphasizes practical bottlenecks like KV-cache and techniques like speculative decoding and continuous batching, offering more real-world insight than typical ML courses.