@smolix: Here's part 1 (of 5) of my short course on efficient LLM inference that I taught at Columbia University. Slides are hea…

Here’s part 1 (of 5) of my short course on efficient LLM inference that I taught at Columbia University. Slides are heavily updated from two weeks ago.

https://t.co/WVCf7mUdkY

TL;DR: The core bottleneck in efficient LLM inference is the mismatch between memory bandwidth and compute capability. This lecture starts from hardware architecture, analyzes how GPUs limit inference speed, and introduces approaches like model compression and KV cache optimization to reduce costs.

Lecture Overview

Hi everyone, I’m Alex. This is a lecture I recorded at the Columbia University Machine Learning Summer School (about a week ago). There are five lectures total. The first gives an overview of how LLM inference works and its parameters; the subsequent ones dive into hardware, serving deployment, model compression, and KV cache compression. We won’t discuss training or architecture design—only how to make already-trained models run faster.

Why Inference is a Memory Problem

GPU Physical Constraints: Blackwell Ultra Example

One of the most powerful consumer GPUs currently is NVIDIA’s Blackwell Ultra. It’s actually two chips glued together (because a single die would be too large due to reticle size limits). The two chips are connected via 18 NVLink links, each at 100 GB/s, and communicate with NVSwitch. HPM controllers connect to high-bandwidth memory (HBM3E standard). CPU interconnects (like Grace Hopper) are much faster than PCIe.

Key numbers: B200’s FP8 compute is 4.5 petaflops (half for FP16, double to 9 petaflops for FP4). Main memory bandwidth is 8 TB/s. Ratio: 4500 petaflops ÷ 8 TB/s ≈ 500 floating-point operations per byte needed to keep the processor fed. This means data movement is the bottleneck.

• Generation (decoding): memory bandwidth is key.
• Prefill (input prompt): compute capability is key.

In online inference, small batch sizes and fast turnaround further amplify the memory bottleneck. Course core: avoid unnecessary data movement between hardware components.

Learning Objectives

• Compute KV cache bytes per token given model shape.
• Distinguish compute costs of prefill vs. decoding.
• Estimate decoding throughput as a function of bandwidth (within 10-30% error).

Running example: Quinn 38B model. Ultimately discuss how to make it work with a million-token context (a bit large for an 8B parameter model, but works as an example).

How to Make LLM Inference Cheaper and Faster

Core idea: without changing the model’s answers, turn a large bitstream into a smaller one.

• Don’t care about prompt optimization or response processing.
• Only care about increasing throughput (tokens per second).
• Other techniques like RAG, prompt engineering, diffusion models are for another course.

Cost Reduction Trends

Taking GPT-3.5-class models as an example, cost per million tokens has dropped significantly. Expected as low as $0.01 per million tokens by 2026. Possible sources:

1. Hardware improvements: NVIDIA and others provide better architectures, cheaper operations per dollar (roughly 2-4×).
2. Post-training optimization: better training within the same architecture boosts performance (roughly 2×).
3. Architecture design improvements: speculative decoding, compression, mixture of experts, mixed block attention, native sparse attention, etc. (roughly 2×).

Combined, this yields 8-10× improvement per year. More powerful models follow a similar trajectory.

Overview of This Lecture’s Content

1. Hardware: HBM, Tensor Core, numerical formats (e.g., NVFP4 requires specific GPU), local serving ecosystem.
2. Serving: VLM vs. SGLang comparison (no deep-dive into partisan debates), caching, KV cache-aware scheduling, model-hardware alignment.
3. Model side: weight compression, KV cache compression (MLA, sparsity, Turbo Quant, etc.).
4. New architectures (briefly mentioned).

Skipped Topics

• Training efficiency, audio/real-time streaming, video large token streams, multi-node distributed, multi-LoRA adapter serving. These each have dedicated tutorials.

Transformer Architecture Review

Classic diagram from the “Attention is All You Need” paper, still largely applicable nine years later. Consider Quinn 8B: 36 actual layers (not 32). Flow: token input → tokenizer → token embedding (ID to vector) → transformer layers (attention + feedforward + normalization) → output.

Source: @smolix: Here’s part 1 (of 5) of my short course on efficient LLM inference that I taught at Columbia University. (https://www.youtube.com/watch?v=3ggYI8Osgss)