Systolic arrays handle over 95% of modern AI-chip compute by moving matrix data locally between processing elements, reducing memory traffic. Larger arrays improve peak throughput but are harder to saturate, so compiler scheduling often determines whether chips reach useful utilization.

The Powerhouse of the AI Chip

Systolic arrays are the powerhouse of every single modern AI chip, representing over 95% of their total compute. Without them, AI workloads run 20x-100x slower. In this post we’re going to discuss their design and what makes them so efficient.

GIF

An AI model is made up of dense linear algebra: elementwise activation functions, transposes, scatters and gathers, and most importantly, matrix multiplies (or matmuls). Matmuls make up the vast majority of the compute in AI models, so they are the most important operation to implement efficiently in hardware. Without fast matrix multiplies, even if every other operation in the model ran instantly, your inference would slow to a crawl.

Surprisingly despite overall architecture, memory systems, power delivery, and networking varying vastly from chip to chip, the hardware powering matrix multiplies have converged to nearly the exact same design across every vendor on the market: the systolic array.

This is because matrix multiplication is an operation that requires the input matrices to interact in a geometric pattern. If you look at this diagram Nvidia provides for their tensor core, it illustrates this interaction geometry.

The two input matrices are shown on the outside of the rank-3 tensor. Each element in the rank-3 tensor is the product of a single element from each input matrix. Specifically the (M,N,K) element in the rank 3 tensor is (M,K) * (N,K). The final output matrix is the result of performing a sum reduction on all the vertical elements of the rank-3 tensor, along the K dimension. This is a familiar definition, but focus on the visualization and what it communicates about the geometry of the communication pattern required to perform matrix multiplication. The two inputs matrices have a pair of orthogonal directions and a single shared reduction dimensions.

Flavors of Systolic Arrays

Systolic arrays are technically more general than the specific subtype used to accelerate matrix multiplication. The word systolic is a reference to the pumping motion of the heart, a systolic array is a system which pumps data to where it is needed. Systolic arrays which accelerate matrix multiply are rectangular grid shaped, often square, reflecting the orthogonal communication pattern. In each node some storage and compute lives. The nodes can communicate to their nearest neighbors in the cardinal directions. There are two modes of operation for systolic arrays:

**Accumulator Stationary
**

Each node in the grid is associated with an element in the output matrix. Each node in the grid is initialized with an accumulator. The input matrices are streamed from the top and side, cycle-by-cycle introducing a new k index each cycle. The (m,n) element of the grid receives a new (m,k) and (n,k) element each cycle. It locally performs a fused multiply add, and keeps track of the reduction with the accumulator. At the end of the streamed inputs, the final output matrix is offloaded from the array.

**
Weight Stationary**

Each node in the grid is associated with on of the input matrix elements. In this case the other input matrix is streamed from one side, and at each cycle, the partial reduction is calculated and passed downwards. This mode requires even less communication but requires the weights to be shared across a large input matrix to be practical.

The Systolic Array is a special purpose hardware unit for executing matrix multiplies extremely efficiently. The “systolic” in the name comes from the medical field, referring to how data is pumped through the array in a pulsed manner, similar to how the heart pumps blood. Input matrices are streamed in row-by-row and column-by-column, and output matrices are streamed out iteratively.

Elements flow directly from the top and left side to the bottom and right side, directly flowing to neighboring processing elements each step, removing the need for global data movement over long distances. Each processing element takes an input from the left, multiplies it with a stationary value held in memory, and adds it to an input from the top before emitting the result to the processing element directly below. This is a typical “weight stationary” setup, where weights live in the processing element’s memory, activations flow from the left and accumulators flow top down. Another formulation flows weights from the top and accumulators live stationary in processing element memory.

The core principle behind a systolic array that gives it excellent efficiency is in minimizing data movement. Since computation is strictly local, the array can run large amounts of fused-multiply-adds entirely in parallel. The array can also scale out directly by simply placing more processing elements in the same pattern, allowing larger tiles to be computed at once.

Systolic arrays are so efficient that often overall efficiency of an accelerator directly correlates to A) how large you can make your systolic arrays and B) if you can keep those arrays fed at all times. There exists a tradeoff here: larger arrays allow for larger tiles and more parallel computation, therefore higher overall efficiency, but are also less flexible and harder to fully saturate. Smaller arrays are easier to saturate and more flexible to real-world dynamically sized workloads, but the data locality is limited to the smaller tile size, and therefore less peak efficiency is achievable.

Real World Tradeoffs

We can see this tradeoff play out in the real world. Nvidia GPUs started with a large number of smaller systolic arrays and have transitioned to fewer larger arrays. Mirroring the transition of their chips from, initially 8x8, 32x32 and now up to 128x128 (Nvidia calls their systolic arrays “TensorCores”). Google’s TPUs on the other hand have opted for fewer, larger systolic arrays ranging from 128x128 on TPU v2 to 256x256 on TPU v6 and v7.

This difference shows a deeper split in ideologies: TPU relies on more ahead-of-time compiler scheduling to lay out data and schedule movements and instructions required to fully saturate the systolic arrays, which if saturated can hit very high throughput. GPU on the other hand relies less on ahead of time scheduling and allows kernels to runtime-schedule data movements to saturate the relatively smaller arrays.

On the GPU side, we have seen a movement towards TPU ideology over time. Older GPUs contained smaller (8x8) systolic arrays and data always came from fully dynamically programmable CUDA cores. However with more recent generations we’ve seem the arrays (TensorCores) progressively get larger (128x128 on Blackwell). Other special function units have been introduced such as the TMA to take pressure off of the general-purpose CUDA cores. We’ll likely continue to see this trend on Rubin and Feynman, with fewer, larger cores and more special function units that if driven correctly by complex software can achieve ever higher throughput.

Why Systems Software Matters

Software to drive all of these elements in harmony means the difference between 10% and 80% utilization. Poorly scheduled data movements or unnecessary synchronization can bring everything to a halt.

This is why the Luminal compiler is so focused on minimizing data movements and maximizing overlapping in instruction scheduling. By operating each hardware unit in a pipeline, we can make each part of the chip operate as independently as possible and minimize idle time.

Often times human-written and agent-written kernels suffer from “bubbles” where the systolic array sits idle waiting on data. Large-scale search allows Luminal to try out many different instruction orderings and overlappings to find schedules that keep these hungry arrays fed.

Having a deep understanding of the hardware that runs these models lets us attack the right bottlenecks and reach speed-of-light performance!