Why AI needs a new kind of supercomputer network — the OpenAI Podcast Ep. 18

# Why AI Needs a New Kind of Supercomputer Network — OpenAI Podcast Ep. 18 Andrew Mayne, Mark Handley, Greg Steinbrecher 2025-03-21 TL;DR: Training large AI models requires tens of thousands of GPUs to work synchronously. Traditional data center networks cannot handle such highly synchronized, tail-latency-sensitive workloads. OpenAI is developing new protocols like Multi-Path Reliable Connection (MP-RC) to eliminate bandwidth bottlenecks and the impact of failures. ## Guest Background: From Quantum Computing to AI Networks Host Andrew Mayne invited Mark Handley from the core networking team and Greg Steinbrecher from the workload systems team. Mark studied physics and math in undergrad, attempted to build a quantum computer during his PhD, later researched optical chips, and eventually shifted to data center networks. He previously researched realistic models of data center workloads in academia, finding that traditional simplified models lacked information. After joining OpenAI, he was responsible for building a simulator of GPU clusters, and later directly wrote GPU communication software. Greg is a professor at University College London, has researched networking for decades, participated in setting video conferencing standards (later used in 4G/5G), and now focuses on innovation in data center networking. ## Why AI Training Requires a Whole New Networking Mindset Traditional data center network architectures originate from internet design — many independent conversations, leveraging the law of large numbers for statistical multiplexing. But AI training is the opposite: **Let tens of thousands of GPUs collaborate on a single task**, and all GPUs must synchronously wait for the slowest one. - **Synchronous workload**: Inter-GPU communication is part of the computation; all GPUs must agree on the result of each step. If a GPU is hit by a cosmic ray causing a bit flip, the entire step may be wasted. - **Tail latency determines everything**: The problem isn't average speed; it's the **worst case** (P100, i.e., 100th percentile). Congestion or failure on any single link in the network becomes a global bottleneck. - **Scale amplifies failures**: As GPU count grows from a thousand to a hundred thousand, device failures are no longer "accidents" but "the norm." Mean time between failures decreases linearly with scale. The number of network components (optical links, switches) is orders of magnitude larger than GPUs, meaning failures are almost inevitable. ## Multi-Path Reliable Connection: A Necessary Breakthrough To solve these problems, you can't simply modify existing protocols; you must redesign the way network communication works. The team that Mark and Greg are on works closely with researchers in co-design, participates in on-call support for large training runs, and extracts new requirements from pain points. Their proposed "Multi-Path Reliable Connection" (MP-RC) scheme aims to: - **Eliminate congestion blind spots**: Traditional networks choose paths randomly; when multiple flows collide, speed drops sharply. MP-RC uses intelligent path selection to avoid collisions, no longer relying on statistical multiplexing. - **Tolerate device failures**: When a link or switch fails, the protocol can quickly reroute without affecting ongoing training, even making the failure imperceptible. - **Enable scaling**: By concurrently utilizing thousands of paths in the network, minimize tail latency so that "P100" is no longer a limiting factor. “When researchers no longer need to know which network protocol this particular cluster uses, we know we've won.” — This is the team's key metric for success. ## Conclusion AI breaks the traditional data center's compute ocean model. OpenAI's cutting-edge insight is that system design itself is an integral part of model training. Through co-design of the network and workloads, the team is removing key obstacles to continued scaling, enabling everyone to get smarter models faster. Source: Why AI needs a new kind of supercomputer network — the OpenAI Podcast Ep. 18 (https://www.youtube.com/watch?v=TiW96H5HmAw)