@rickawsb: NVIDIA also believes storage is a bigger bottleneck than GPUs — Decoding NVIDIA's latest article. NVIDIA's newly released 'AI Model Co-Design' is a technical article introducing TensorRT-LLM and Blackwell, but also a roadmap for large model design and AI infrastructure in the coming years...

NVIDIA also believes storage is a bigger bottleneck than GPU
— Analysis of NVIDIA’s latest article

NVIDIA’s latest release, AI Model Co-Design, is a technical article introducing TensorRT-LLM and Blackwell, but it also serves as a roadmap for large model design and AI infrastructure development over the next few years.

The real discussion in the article is about how to make models adapt to GPUs from the initial design stage. This means AI has entered a new phase: models are being designed around hardware, rather than hardware being optimized around models.

In the past, the focus of large model competition was always Accuracy, but this article redefines the goal of AI as a balance among Accuracy, Throughput, and Interactivity. Accuracy is no longer the sole determinant; greater commercial value comes from generating more Tokens per unit time and lower response latency. The entire article revolves almost entirely around Inference, with little discussion of training—this itself reflects NVIDIA’s judgment on the future market focus.

The key to understanding the article is the Roofline Model: any program is ultimately limited by one of two bottlenecks—Compute or Memory. Which category a program falls into depends on how much computation the GPU can complete per Byte of data moved: high arithmetic intensity means the program is compute-bound; low arithmetic intensity means it is memory-bound.
“Latency-sensitive decoding runs at low concurrency and is memory-bound.”

Why is inference naturally prone to being memory-bound? The reason lies in its two completely distinct phases. Prefill processes the entire prompt, computing thousands of tokens at once, with large matrices providing high Weight Reuse—so it’s typically compute-bound. Decode, on the other hand, must generate tokens one by one; the next token must wait for the previous one, making it inherently incapable of parallelization along the time dimension, relying only on batching across different users. As the batch size shrinks, generating each token still requires reading almost the entire model weights, sharply reducing Weight Reuse, and the GPU spends more time waiting for data than computing.

The article gives a typical FFN example. The GEMM in FFN can be expressed as M×K multiplied by K×N, where M corresponds to the number of tokens, and N and K correspond to model dimensions. During inference, as concurrency drops, M shrinks rapidly, while N and K remain largely unchanged. This means the actual computation amount decreases, but the N×K weight matrix that needs to be read barely reduces. The article thus concludes: “FFN latency becomes memory-bound because the GEMM-N and GEMM-K dimensions remain large as GEMM-M shrinks.” In other words, the GPU is no longer waiting for computation to complete, but for Weight Read.

This is why the article repeatedly emphasizes Memory, with almost no discussion of FLOPS. The three most representative sentences in the Roofline section are: “Workloads with low arithmetic intensity are capped by memory bandwidth.” “Latency-sensitive decoding is memory-bound.” “Memory time exceeds math time at every token count.” All three essentially say the same thing: for modern inference, what truly limits system performance is no longer GPU compute power, but Memory bandwidth and Weight Read.

MoE can alleviate this problem but cannot completely solve it. MoE transforms every layer of a dense model from full computation to activating only a few experts, significantly reducing Compute. However, as Compute decreases, new bottlenecks emerge. First, All-to-All communication from Expert Parallelism; second, load balancing issues from the Router; third, Weight Read itself still exists. Each expert still needs to read a large number of parameters, so MoE essentially shifts the bottleneck from Compute to Memory and Communication, rather than eliminating it.

This is why NVIDIA highlights Wide Expert Parallelism. By distributing experts across more GPUs, NVIDIA aims to increase the system’s Aggregate Memory Bandwidth while reducing the number of experts stored on each GPU, thereby decreasing the latency caused by Weight Read. Note that the gain here does not come from more FLOPS, but from more Memory Bandwidth. This is the first time the article explicitly frames GPU scaling as Memory scaling, rather than Compute scaling.

NVIDIA’s Helix Parallel, announced in 2025, addresses another type of Memory issue—KV Cache. Traditional Tensor Parallelism, once the number of GPUs exceeds the number of KV Heads, must replicate the entire KV Cache, causing HBM usage to spike. Helix splits the KV Cache by Sequence rather than by Head, with each GPU storing only a portion of the KV Cache, then exchanging Sequences during the Attention phase, before switching back to TP×EP for the FFN. This means Attention and FFN no longer use the same parallelism strategy; each chooses the optimal one. As Long Context grows from tens of thousands of tokens to millions, Helix’s value will only increase.

The article proposes seven design principles, but they all revolve around the same goal. Near-square Matrix improves Arithmetic Intensity; alignment to 128, 256, 512 improves Tensor Core utilization; preferring Width over Depth increases Weight Reuse; NVFP4 reduces Memory Traffic; Wide EP increases Aggregate Memory Bandwidth; Chunked Pipeline reduces Pipeline Bubble; Helix decouples Attention from FFN, using different Parallel Strategies. Although they appear as seven independent suggestions, they all center on the Roofline Model, with a single objective: reduce Data Movement and improve GPU utilization.

What is truly worth noting, however, is what the article does not explicitly state. First, GPU Compute growth has outpaced the compute actually needed by models, and the marginal benefit of further increasing FLOPS is diminishing. Second, the bottleneck in the inference era has shifted from Compute to Memory—HBM, KV Cache, Weight Read, and Memory Traffic will become increasingly important. Third, GPU competition is evolving into system-level competition, encompassing HBM, NVLink, TensorRT, parallelism strategies, and the software stack, rather than just the performance of a single GPU. Fourth, the real optimization target for future AI infrastructure is no longer computation, but Data Movement.

In summary: In more and more AI inference scenarios, storage (more precisely, Memory, Memory Bandwidth, Weight Read, and Data Movement) has become a more likely system bottleneck than GPU compute power.

Author background: Almost all seven authors of this article come from the GPU Architecture, TensorRT-LLM, Distributed Inference, Model-Hardware Co-design, and Compute Architecture teams.

NVIDIA AI (@NVIDIAAI):
As AI models continue to grow in scale and capability, shaping a model matters just as much as its size.

We’re introducing a new series on AI Model Co-Design exploring the synergy between models and hardware. The first post focuses on how model dimensions influence GPU