Inference Engineering: A Deep Dive into the Free Book “Inference Engineering” — Prefill is Compute-Bound, Decode is Memory-Bound, Explaining Inference Optimization from CUDA to Production Deployment

Inference Engineering

A systematic engineering book published by Baseten on “how to run AI models fast, cost-effectively, and reliably in production.” Authored by @philipkiely, it is available for free online reading. It covers the complete technology stack from CUDA kernels to Kubernetes, targeting engineers and technical decision-makers who need to self-host open-source models (such as DeepSeek, Qwen, Kimi, Llama, etc.). The writing context is: After the release of DeepSeek V3/R1, the capability gap between open-source and closed-source models has been essentially eliminated, and self-hosted inference can achieve over 80% cost savings and four-nines availability; however, the trade-off is that you must understand every layer from hardware to production.

Inference Engineering by Philip Kiely | Read the Full Book OnlineFrom baseten.co

Book Structure: The Three-Layer Inference Stack

The framework introduced in Chapter 0 runs throughout the book: effective inference requires coordination across three layers:

• Runtime Layer (how fast a single model instance runs): CUDA → PyTorch → Inference Engine, along with six key techniques: quantization, batching, caching, speculative decoding, parallelism, and disaggregated serving.

• Infrastructure Layer (handling scale): Auto-scaling → Capacity planning → Unified multi-region, multi-cloud resource pools.

• Tool Layer (abstractions for development): Finding the middle ground between “black-box APIs” and “bare primitives.”

The chapter order follows this logic: First, define goals (Chapter 1) → Understand the model (Chapter 2) → Understand the hardware (Chapter 3) → Understand the software stack (Chapter 4) → Apply optimization techniques (Chapter 5) → Extend to non-text modalities (Chapter 6) → Deploy to production (Chapter 7).

Chapter-by-Chapter Interpretation

Chapter 1: Prerequisites: Define “Better” Before Optimizing

The core argument is optimization is trade-off management. The author uses a fitting analogy: NFL players are not the biggest, fastest, or strongest; they are athletes specialized for specific positions. The same applies to inference systems: the more constraints, the higher the achievable performance. Before starting, you must answer five questions: which model, what application interface, latency budget, unit economics, and traffic patterns.

Key Practical Takeaways:

• Three triggers for switching from shared APIs to dedicated deployment: Usage volume makes per-GPU billing more cost-effective, need for fine-tuning or specific SLAs, or a multi-model pipeline requiring reduced network overhead. Otherwise, continue using APIs.

• Model selection is the biggest performance decision: Under equivalent optimization, smaller models are always faster and cheaper. The book’s text-to-SQL example is compelling—SQL is a constrained language, and a few-billion-parameter fine-tuned model can match the performance of a hundreds-of-billions-parameter general model.

• Metrics framework: TTFT (determined by compute-bound prefill) and TPS (determined by memory-bound decode) are the two core metrics. It’s crucial to distinguish between single-user “perceived TPS” and system “total TPS.” LLM latency is right-skewed, so averages are misleading—focus on P90/P99 tails. A diagnostic rule: if inference is fast but end-to-end is slow, the issue lies in infrastructure, not the model.

Chapter 2: Models: All Optimizations Start with Understanding the Compute Mechanics

Two generation paradigms cover all modalities: autoregressive token generation (LLMs, VLMs, embeddings, ASR, TTS) and iterative denoising (image/video diffusion models). The technical depth of this chapter centers on two areas:

Bottleneck Analysis Framework (the book’s most important analytical tool): A GPU’s two key resources are compute (ops/s) and memory bandwidth (bytes/s). The H100’s ops-to-byte ratio is about 295; for perfect balance, every byte read should involve 295 operations. Arithmetic intensity (total compute / total memory access) higher than this ratio indicates compute-bound; lower indicates memory-bound. This leads to three iron laws: prefill is compute-bound, decode is memory-bound, image/video generation is compute-bound. This judgment dictates the direction of all subsequent optimizations and hardware selection.

Attention Optimization: Attention scales quadratically with sequence length. Decode uses KV cache to reduce this to linear, but it’s still expensive. Two paths: lossless implementation optimizations (FlashAttention eliminates redundant reads/writes, PagedAttention resolves memory fragmentation without changing complexity) and lossy algorithmic optimizations (sliding window, linear/compressed attention, MLA—breaking the quadratic barrier at the cost of quality).

Another easily overlooked detail: MoE models (e.g., Qwen3-235B-A22B, activating 22B/235B per token) save compute per request, but in batch serving, different requests activate different experts, causing nearly all parameters to be read; the sparsity benefit is discounted in production and requires expert parallelism to recover.

Chapter 3: Hardware: Choose the Card Based on the Bottleneck

Expands on the iron laws: prefill/video generation favors high FLOPS, decode favors high bandwidth. The H100 (80GB/3.35TB/s) and H200 (141GB/4.8TB/s) have identical compute, but H200 is clearly superior for decode-heavy workloads. Precision is a compute lever—halving precision doubles FLOPS. When comparing GPUs, use the same precision and non-sparse numbers.

Practical points: Memory must fit weights + at least 50% headroom for KV cache; interconnects determine parallelism limits (NVLink 900–1800 GB/s ≫ InfiniBand ≫ Ethernet); small models (~under 2B) benefit from MIG hardware partitioning (H100 can split up to 7 instances) to avoid wasting a full card; NVIDIA’s real moat isn’t hardware but the CUDA software stack—competitors (AMD, TPU, Cerebras, Groq) compete on memory bandwidth, power efficiency, or platform integration, but all are constrained by the software ecosystem.

Chapter 4: Software: Abstraction Level Dictates Leverage, Extreme Performance Requires Going Deeper

The software stack from bottom to top: CUDA kernels (cuBLAS/CUTLASS/FlashInfer—kernel selection matters more than writing, kernel fusion is key to decode speedup) → PyTorch (torch.compile, but cannot fuse plugin kernels like FlashAttention) → Inference Engines → Dynamo.

Three major engine choices are the practical core of this chapter:

• vLLM: The default choice, with the broadest hardware coverage, best day-zero model support, and best usability.

• SGLang: Rising with the Chinese open-source model ecosystem (DeepSeek/Qwen), focusing on large-scale MoE multi-node deployment (GB200 NVL72), highly customizable, the first choice at xAI.

• TensorRT-LLM: Highest performance ceiling, but limited to NVIDIA and complex configuration; the V1 version now runs standalone on PyTorch, with usability close to competitors.

Dynamo doesn’t replace engines; it orchestrates at scale: cross-node KV cache reuse and routing, prefill/decode disaggregation, and multi-node expert parallelism. Finally, it emphasizes benchmarking methodology: the gold standard is shadow traffic (replicating real production requests for load testing), changing one variable at a time—benchmarks measure “how it performs,” profiling measures “why it performs that way.”

Chapter 5: Techniques: Five Acceleration Methods and Their Interactions

This is the technical heart of the book. Each of the five sections could stand alone, but their true value lies in how they interact:

• Quantization: On the prefill side, lower precision doubles compute; on the decode side, it effectively doubles memory bandwidth. In practice, 16→8 bit yields a 30–50% speedup. In production, stick to floating-point formats (FP8/MXFP8 is the sweet spot). Sensitivity order: linear weights < activations < KV cache < attention (softmax almost always retains original precision). The acceptance standard is “zero perceivable loss.”

• Speculative Decoding: Uses idle compute to produce N+1 tokens per forward pass (draft + verification). It only improves TPS, not TTFT; it must be dynamically disabled when compute is saturated at large batch sizes. Among variants, EAGLE (a dedicated draft head, up to 8 draft tokens per pass) is the general-purpose default, while n-gram lookup easily outperforms EAGLE in code completion, where output regenerates input.

• Caching: The hard limit of prefix caching is that the prefix is truncated at the first differing token. This directly spawns “context engineering”: keeping volatile content as late as possible in the context. Storage has four tiers (GPU → host RAM → local SSD → network SSD), and Dynamo’s KVBM enables cross-tier migration.

• Parallelism: Memory math (roughly 1GB per 1 billion parameters at FP8) determines the minimum GPU count; TP (tensor parallelism) improves single-user TPS but consumes NVLink bandwidth; EP (expert parallelism) improves total throughput and can span nodes; PP (pipeline parallelism) is only a fallback for multi-node setups.

• Disaggregated Serving: Splits prefill and decode into separate engines, each optimized independently (e.g., prefill can use lower parallelism). Dynamo provides a production-grade implementation (dynamic xPyD ratio adjustment). The barrier to entry is high: it requires daily token volume of 1–10 billion + model ≥100 billion parameters + prefill-heavy traffic; missing any one of these wastes hardware.

The overarching meta-principle: Constraints trade for performance, traffic determines optimization depth, and techniques must be evaluated in combination. Baseten engineers tested 77 configurations for a single client model to double TPS—tuning is a continuous empirical process, not a one-time setup.

Chapter 6: Modalities: Extension and Divergence of Two Architectural Archetypes

VLMs, embeddings, ASR, and TTS are essentially variants of the LLM tech stack (TTS is even fine-tuned Llama with an expanded audio token vocabulary), and most optimizations transfer directly. Image/video diffusion models take a different path: no KV cache, no autoregression, compute-bound rather than memory-bound, shifting the optimization target from memory bandwidth to FLOPS and attention itself.

Each modality has unique engineering considerations: A high-resolution VLM image is worth about 1000 tokens; downsampling is its specific quality-speed dial; embedding models have no decode, so prefix caching and disaggregated serving are irrelevant—they rely on massive batch sizes + horizontal scaling; Whisper can achieve a real-time factor of 1000x for long-file transcription (1 hour of audio in 4 seconds), but at the cost of losing prefix continuity, requiring hallucination detection; TTS only needs 80–100 tokens per second for real-time use; faster is pointless, so all optimization shifts to maximizing concurrent streams per GPU; video generation is the most compute-intensive (attention accounts for 70–80%, full-node 8 GPUs at batch 1), relying on per-step/per-layer selective quantization + Context Parallelism (not TP) + ring attention.

Chapter 7: Production: Where Value is Realized, and Where New Problems Emerge

A single instance, no matter how fast, will be overwhelmed by traffic—this is an infrastructure problem, not a PyTorch or CUDA problem. The key contribution of this chapter is breaking down “latency” into a full-stack ledger: container image size, cold start phases (GPU provisioning, image loading, weight loading, engine compilation—TensorRT-LLM compilation can take minutes), cross-cluster 10ms vs. 50ms gaps, and even client TLS handshakes consuming 10% of a 300ms P95 budget.

Hard data and conclusions for production: Llama 3 training data shows about one hardware failure per 50,000 GPU hours—failures are normal, requiring active-active and self-healing scheduling; canary deployments combined with auto-scaling incur almost no extra cost at scale (old versions auto-scale down), while blue-green deployments require double the GPUs; procurement should use “low-cost reserved capacity as a base + on-demand/Spot for peaks”; cost accounting should not reverse-engineer token costs from GPU spend, but positively compare total costs and include self-hosted engineering time in the TCO.

Synthesis: Fundamental Principles from the Book

• Identify the bottleneck before selecting the method: Whether compute-bound or memory-bound—this judgment runs through all decisions in hardware selection, kernel optimization, parallelism strategy, and disaggregated serving.

• Constraints and traffic are performance allies: Vertical applications should proactively add constraints; high traffic unlocks advanced optimizations (disaggregated serving, KV-aware routing, multi-node parallelism)—many techniques are net-negative at low traffic.

• Evaluate techniques in combination, not in isolation: Quantization is an enabler (saved bandwidth makes disaggregated serving and caching more effective); speculative decoding and large batches are mutually exclusive; tuning is a continuous empirical process.

• Latency is end-to-end and full-stack: From kernel fusion to client connection reuse, every millisecond counts; averages are misleading, focus on percentiles.

• Failures are normal; production is the value realization point: All single-component optimizations are meaningless if they don’t pass the infrastructure test.