Machine Learning Systems: The Physics of AI Engineering (Harvard University · 2 Volumes, 33 Chapters · 1.46 Million Words · From Single Sensors to 10,000-GPU Clusters)

Harvard University’s “Machine Learning Systems”

Harvard / Vijay Janapa Reddi · MIT Press 2026 · Open-Source Textbook (CC-BY-NC-SA) · mlsysbook.ai Subtitle: The Physics of AI Engineering

What Kind of Book Is This?

One-line positioning: It aims to do for ML systems engineering what Hennessy & Patterson’s Computer Architecture: A Quantitative Approach did for computer architecture—transforming a field driven by intuition and experience into an engineering discipline with teachable, quantifiable laws.

This isn’t just rhetoric. The foreword is written by David Patterson (Turing Award winner and pioneer of quantitative architecture analysis), who explicitly draws parallels between the two books. Author Vijay Janapa Reddi is a Harvard professor, and this book originated from his course CS249r and the TinyML edX project. The statement in the author’s description captures the essence of the entire book:

“The world is rushing to build AI systems; it is not yet engineering them.”

The author’s core judgment: the past decade of ML research has been dominated by a model-centric mindset, but the model is not a system. A neural network that performs well in a research notebook is useless if it fails to meet latency constraints, if the data pipeline can’t handle traffic, or if energy consumption exceeds economic value. The gap between a “working model” and a “usable system” isn’t about implementation details—it’s about missing engineering principles. He calls this ML’s “Garmisch moment” (referring to the 1968 NATO conference that sparked the software engineering discipline).

The book offers a very clear three-part disciplinary positioning:

• Computer science researches what is computable;

• Machine learning researches what is learnable;

• AI engineering researches what is buildable, under the physical constraints of real hardware and real data.

The full work spans two volumes, 33 chapters, from individual chips to 10,000-GPU clusters, from microwatt sensors to megawatt data centers. The accompanying ecosystem includes TinyTorch (20 modules for building an ML framework from scratch), interactive Labs, hardware kits, and the StaffML exercise platform—reflecting the author’s belief that the only way to understand a system is to build it yourself.

Associated open-source project: GitHub - harvard-edge/cs249r_book: Machine Learning Systems: Foundations, Scaling, Agentic AI, and… From github.com

The “Physics” of the Book: A Few Laws That Run Through It

The most unique aspect of this book is that, like a physics textbook, it organizes all content around a few quantitative laws.

1. The Iron Law of ML Systems

T=BWDvol+Rpeak⋅ηhwO+LlatT= BW D vol

• R peak

⋅η

hw

O

+L lat

The time for any ML workload is decomposed into three terms: data movement (data volume divided by bandwidth), computation (operations divided by peak throughput and hardware efficiency), and latency overhead (scheduling, queuing, synchronization, etc.). The second volume extends this to the distributed form, the Fleet Law:

Tstep(N)=NTcompute+Tcomm(N)+Tsync(N)−ToverlapT

step

(N)= N T compute

+T

comm

(N)+T

sync

(N)−T

overlap

Adding GPUs linearly reduces the compute term, but communication and synchronization terms grow with N—this is the entire math behind why “10,000 GPUs don’t give 10,000x speedup.” All chapters in both volumes essentially teach you how to diagnose which term you are currently “bound by” (the binding constraint) and then move it intentionally.

2. The Roofline Model and the Memory Wall

Rattain=min(Rpeak,BW×I),Iridge=Rpeak/BWR

attain

=min(R peak

, BW×I),I

ridge

=R peak

/BW

This model from the Hennessy & Patterson tradition appears repeatedly because it reveals the cruelest physical fact of modern ML: compute grows far faster than bandwidth (H100’s ridge point is now around 295 FLOP/byte). Compute capacity improves by orders of magnitude every 2–3 years, while DRAM bandwidth has only increased by single-digit multiples over a decade. The consequence: during the decode phase of an LLM, generating each token requires moving all weights, resulting in an arithmetic intensity of only about 1 FLOP/byte—two orders of magnitude below the ridge point. Tensor cores spend 99% of their cycles waiting for data. Adding more compute doesn’t help; the only solutions are adding bandwidth or reducing data movement.

3. The Energy Scaling Ladder (Horowitz Data)

At the 45nm process node: INT8 addition costs 0.03 pJ, FP32 multiplication costs 3.7 pJ, SRAM access costs 0.5 pJ (per byte), and DRAM access costs 640 pJ. One DRAM access uses roughly the same energy as 100,000 INT8 additions. This explains the common root of a series of engineering choices throughout the book: kernel fusion, FlashAttention, systolic arrays, 2:4 sparsity—they aren’t about “calculating faster,” but about “not moving data at all.” The author repeatedly emphasizes: the energy consumption and latency of modern ML chips are dominated by data movement, not computation.

4. Amdahl’s Law and “Overhead Can Only Be Moved”

The speedup limit is capped by the serial portion. In a system where preprocessing takes 45% of latency, a 10x model speedup only yields a 1.8x end-to-end improvement. From this, the author extracts the book’s meta-principle (which he calls the complexity conservation heuristic): overhead cannot be eliminated, only moved. Quantization moves the cost to accuracy verification, distributed training moves memory pressure to communication volume, and automation moves human effort to maintenance. A good engineer isn’t someone who eliminates overhead, but someone who knows precisely where each overhead is moved and ensures it lands somewhere it gets measured.

5. The Drift Equation

Accuracy(t)≈Accuracy0−λ⋅D(Pt∥P0)Accuracy(t)≈Accuracy

0

−λ⋅D(P t

∥P

0

)

Post-deployment accuracy decays with distribution divergence. This is the mathematical formalization of Chapter 14’s “week-two problem” in the first volume (a model with 94% accuracy in week 1 drops to 88% in week 4, while all infrastructure metrics are green—“perfectly available and perfectly wrong”). It also provides the theoretical basis for the book’s emphasis that ML systems fail silently.

6. Two Major Classification Systems

• D·A·M (Volume 1, single node): Data (fuel) × Algorithm (blueprint) × Machine (engine). Any problem can be diagnosed along these three axes, and the bottleneck will shift between them.

• C3 (Volume 2, clusters): Compute × Communication × Coordination. Frontier models like Llama first bind Communication, recommendation systems first bind Coordination, and federated learning first binds Compute. The same cluster theory, three different binding orders—the engineer’s skill lies in “reading it out,” not assuming.

Volume 1: Introduction — Everything on a Single Node

Volume 1 covers the complete ML system stack on a single machine (1–8 GPUs), structured in four parts:

Foundations (Chapters 1–4). The introduction establishes the “dual mission”: statistically trustworthy model behavior + machine production of that behavior within time/memory/energy budgets. Chapter 2 is the book’s core architectural viewpoint: constraints define architecture. Three physical walls—speed of light (36ms round trip across the US), the power wall (P ∝ V²f, doubling frequency ≈ 8x power), and the memory wall—slice the deployment world into four paradigms: Cloud/Edge/Mobile/TinyML, with power spanning roughly nine orders of magnitude (MW to mW). The same model requires fundamentally different engineering, not scaling, across these paradigms. Chapter 3 uses a diabetic retinopathy screening case to thoroughly explain the “lab-to-field gap”: a model with 0.99 AUC in the lab drops to 78% sensitivity in a Thai clinic due to old cameras and lighting issues; the failure isn’t a model failure, but a workflow failure. Chapter 4 proposes “Data is code”: a dataset is compiled source code, so changing data changes program logic, and data must be versioned, tested, and reviewed like code.

Development (Chapters 5–8). This section carries through the idea that “neural networks are computable”: the MAC count for a fully connected layer is input dimensions × output dimensions; backward pass is 2x the forward pass; one training step ≈ 3x forward FLOPs; training memory comprises four components—weights, gradients, optimizer states, and activations (under Adam, optimizer states alone are 16 bytes/parameter; training a 175B-parameter model requires ~2.8TB, the starting point for all distributed memory optimizations). The architecture chapter offers a valuable systems perspective: five “lighthouse models” each lock down a resource constraint (ResNet-50=computation, GPT-2=bandwidth, DLRM=memory capacity, MobileNetV2=latency/power, KWS=milliwatt power), and it points out that FLOPs ≠ speed—MobileNetV2 with 10x fewer FLOPs than ResNet-50 might be slower on bandwidth-constrained hardware. The framework chapter defines frameworks as “compilers for silicon contracts,” using a compilation net benefit formula to answer “when is it worth using torch.compile.” The training chapter has the strongest discipline: memory feasibility first, then throughput; saturate the data pipeline before tuning kernels—“a hungry accelerator looks just like an inefficient one.”

Optimization (Chapters 9–12). The data selection chapter introduces ICR (Information-to-Compute Ratio): under fixed compute, selecting the right data yields higher returns than just adding more data; EL2N scoring can prune half the CIFAR-10 training set without losing accuracy. The compression chapter’s “complexity conservation” is highly insightful: pruning costs capacity, quantization costs accuracy, distillation costs compute—“hardware sets the exchange rate.” Furthermore, compression techniques form a combinatorial optimization problem, not independent functions; pruning changes the distribution of surviving weights, disrupting quantization calibration. The correct sequence is structured pruning → distillation recovery → QAT locking. The hardware chapter is the book’s most physically dense: TPU’s systolic array (weight-stationary, each data element entering the array is reused by all units, reducing memory accesses from 4 per operation to 2/N) is explained very thoroughly. The benchmarking chapter introduces the MLPerf ecosystem and establishes three measurement disciplines: benchmarks are proxies not truth; Goodhart’s Law (once a metric becomes a goal, it ceases to be a good metric); and end-to-end metrics beat component metrics.

Deployment (Chapters 13–15). The serving chapter discusses “serving inversion”: training maximizes throughput, while serving must answer individual requests within latency SLOs, shifting the bottleneck from compute to data movement, queuing, and scheduling. Queuing theory is applied here: at utilization ρ=0.7, latency already reaches 3.3x the mean; p99 is 4.6x the mean. Capacity planning must include headroom; high utilization itself is a warning sign. Technical cores of LLM serving (continuous batching, PagedAttention reducing KV cache waste from 60–80% to under 4%, speculative decoding) are quantitatively explained. The MLOps chapter centers on silent failure, covering retraining economics (optimal retraining interval T* ∝ √(C/QVγ)), PSI/KS drift detection, and statistical power calculations for canary/A/B testing. The responsible engineering chapter takes a very engineering-oriented stance: MLOps is the control loop for reliability; responsible engineering is the control loop for safety. A system can pass verification (meeting training objectives) yet fail validation (violating operational/legal requirements) because the specification itself is incomplete. Fairness isn’t a tuning problem but a mathematical multi-objective constraint (Kleinberg/Chouldechova impossibility theorem: when base rates differ, calibration and error rate balance cannot be achieved simultaneously).

Volume 2: Scaling — The Cluster as Computer

Volume 2’s stance is sharply established from the outset: The three classic constraints of a single node evolve at cluster scale into “three walls” (memory wall, bandwidth wall, energy wall) plus an additional chasm: the reliability chasm. The anchor data is Llama 3’s war story: 16,384 H100 GPUs, 54 days of continuous training, 419 interruptions (about once every 3 hours), yet maintaining over 90% effective training time. Scale turns exceptions into the norm and scheduling from a convenience into productivity itself.

Cluster (Chapters 2–4) describes the data center as “assembling a physical computer”: the bandwidth hierarchy (on-chip 3.3TB/s → NVLink 900GB/s → PCIe 64GB/s → IB 50GB/s) shows the ~9x cliff between NVLink and IB, which is the physical reason tensor parallelism doesn’t extend beyond a node; the network is essentially a “gradient bus” (serving AllReduce, AllToAll, P2P collective communication modes); storage has all ML’s assumptions inverted—checkpoints write 2.45TB every 10 minutes, naive data parallelism creates a “checkpoint storm” (64 replicas ≈ 157TB), ZeRO sharding plus tiered staging compresses this to under 25 seconds.

Distributed Training (Chapters 5–7) forms the theoretical core. The 3D parallelism cube N = d×p×t (data×pipeline×tensor) is paired with a compiler mental model: DP=parallel for, TP=SIMD vectorization, PP=instruction pipelining. Each strategy’s communication fingerprint determines placement rules (TP within nodes, PP within racks, DP across clusters). The collective communication chapter uses the α-β model and energy ladder (SRAM 0.5 pJ/bit to optical IB 1000 pJ/bit, a 2000x difference) to thoroughly explain the algorithm zoo of Ring/Tree/Butterfly AllReduce, and the worldview that “every byte has a travel cost.” The fault tolerance chapter’s Young-Daly law (optimal checkpoint interval τ* = √(2·T_write·MTBF)) is one of the most elegant results in the volume—it’s an expected-value optimum, while the true contract in production is RPO/RTO.

Deployment and Responsibility (Chapters 9–16). The inference chapter presents the book’s most important economic fact: the inference-to-training cost ratio for recommendation systems can reach 100–1000x; serving is the dominant cost in the lifecycle, so “serving inverts every assumption of training.” The edge chapter returns to Volume 1’s physical tradition: GB/s, not TOPS, is the selection metric (LPDDR5X vs. H100 bandwidth differs 30–50x). The sustainable AI chapter takes a clear stance: energy consumption is a physical limit, not environmental sentiment, and provides a complete carbon accounting framework (PUE, grid carbon intensity with 8–40x siting differences, embodied carbon, the AI Jevons paradox—efficiency gains must be coupled with absolute carbon budgets, otherwise they’re consumed by demand growth). The responsible AI chapter treats governance as engineering: six types of versioned artifacts (Claim/Argument/Evidence/Assumptions/Owner/Invalidation rule) in an assurance lifecycle, turning “compliance” into auditable software engineering.

What This Book Truly Teaches

After reading all 33 chapters, you’ll find that specific techniques (NCCL environment variables, TensorRT tactics, K8s operators) are merely examples. The book consistently drills a diagnostic discipline, and both volumes’ conclusion chapters converge to the same statement:

Find the current binding constraint, quantify the cost it imposes, and design operational paths to keep that constraint visible, rather than shifting it to another layer.

Supporting this methodology are several recurring worldviews:

1. Constraint-driven, not goal-driven. First ask “what is binding,” then optimize. Adding compute doesn’t solve bandwidth-bound problems (LLM decode), and increasing batch size doesn’t solve latency-bound problems (p99).

2. Everything has a price. Mixed precision trades numerical risk for throughput; checkpointing trades 33% recomputation for 71% activation memory savings; DP-SGD trades 2–5x computation for privacy guarantees; fairness criteria are mutually exclusive and require explicit trade-offs. Engineering isn’t about finding the optimal solution—it’s about consciously selecting a point on the Pareto frontier and being able to defend it.

3. Silent failure is the default mode. ML systems don’t crash; they silently degrade—accuracy decays with distribution drift, feature implementations inevitably diverge between training and serving, HTTP 200 but features are stale. Thus, monitoring, drift detection, and observability aren’t operational add-ons; they are part of the system’s essence.

4. “The system is the model.” The first volume’s conclusion: the real model isn’t a 500MB weight file, but the sum of the data pipeline + training infrastructure + serving system + operational loops. Optimize the system, and the model improves.

5. The tail is the truth. p99 can be dozens of times the mean; a 100-shard fan-out has a 63% chance of hitting the 1% tail; a serving system’s capacity planning, load balancing, and cache design are all dominated by tail latency. The mean is for reports; quantiles are for user experience.

The Fallacies and Pitfalls at the end of each chapter are the book’s most experience-dense section (inheriting the genre from H&P): roughly a hundred “fallacies and pitfalls,” almost each corresponding to an expensive real-world incident—Knight Capital losing $460 million in 45 minutes, Zillow’s $304 million write-down, the OVH data center fire, Facebook’s “size zero” file, the Uber Tempe incident… These cases ground the laws in hard-won lessons.

What Makes This Book Unique

• Rare positioning. Existing ML systems books either lean toward algorithm theory, tool operation (K8s/PyTorch manuals), or are collections of papers. A complete textbook with “quantitative principles” as its main thread, building from physical constraints to governance obligations, currently has no peer.

• Solid physical intuition. Concepts like the energy scaling ladder, bandwidth cliffs, and roofline ridge movement (“rising ridge”) are scattered across dozens of papers elsewhere; here they are organized into a coherent causal chain. After reading, you’ll see optimizations for LLM inference (PagedAttention, speculative decoding, quantization) with a sense of clarity: “it’s all the same thing.”

• Mature instructional design. Lighthouse models as “constraint probes” run through both volumes; D·A·M/C3 provide retrieval structures; Fallacies and Pitfalls offer negative example learning; accompanying TinyTorch/Labs provide hands-on loops. This is a textbook polished in real classrooms (CS249r), not a pile of papers.

• Approach to responsible AI. Not preachy, but mathematical (impossibility theorems), economic (fairness tax), and engineering-focused (assurance lifecycle)—turning ethics into measurable, auditable constraints. This may be the most effective way to convey it to practitioners.

Who Should Read It, and How?

• ML Engineers/Infra Engineers: This is currently the most systematic “catch-up” material. The author’s suggested ML Engineer path is to skim Volume 1, then closely read the chapters on frameworks, training, hardware, benchmarking, and serving.

• Algorithm Researchers: Chapters 2, 5, and 11 of Volume 1 are worth reading in full—they explain what your methods will encounter on real hardware.

• Technical Managers/Entrepreneurs: The introduction + conclusion chapters from both volumes, plus the MLOps and sustainability chapters—about 6 chapters total—are enough to build correct intuitions on cost and risk (inference: training cost ratio, tail latency, silent failure, carbon budgets).

• Students: Reading Volume 1 sequentially is a complete course; TinyTorch is the accompanying “build from scratch” path.

Finally, worth remembering is the author’s engineering restatement of Sutton’s “Bitter Lesson”: long-term sustainable advantage doesn’t belong to the cleverest architectures, but to the systems engineering that can absorb compute gains. This statement is verified throughout both volumes’ 33 chapters: from AlexNet’s dual-GPU model parallelism to Llama 3’s 419 interruptions while maintaining 90% effective compute, in a decade of AI, it has always been the systems that make the difference.