@YuvrajS9886: After Data Parallelism we move to Model Parallelism starting with Pipeline Parallelism The paper: GPipe: Easy Scaling w…

X AI KOLs Timeline Papers
pipeline-parallelism model-parallelism gpipe training deep-learning micro-batching checkpointing

Summary

This thread explains GPipe, a paper on pipeline parallelism for scaling large models across multiple GPUs using micro-batching and activation checkpointing.

After Data Parallelism we move to Model Parallelism starting with Pipeline Parallelism The paper: GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism The What? When models can no longer fit on a device (GPU) then we have to dissect the model itself across the GPUs and do the training, it's called model parallelism. Now we can do it some ways, like splitting the layers of a model or splitting the tensors itself! In this paper, authors propose effective methods to perform pipeline parallelism across N GPUs. The Why? Since the layers are split across the GPUs, it will be sequential operation which makes it too slow because we cant move forward if the previous layers hasn't finished its "working" which means the rest of GPUs have to wait - VERY BAD. The How? They proposed two techniques to make it work: 1) Micro batching: split the mini batches of data on each node into M microbatches and this means we wont be passing the whole mini batch at once to any GPU (because it will take some time and others will be waiting for data ) but rather process the said microbatch instead and quickly pass the output to the next node, while the current node can process the next micro batch and this goes on for all the nodes. 2) Checkpointing: Makes use of the activation checkpointing concept, where you sacrifice computation for memory - you discard the activation throughout a particular layer of the model and only save the final output and when its needed during backward pass you recompute those activations. Authors use this to save the final activations at partitions - when there is movement from one GPU to another, beaus ethane its as the input to the next layer. Now one more thing that could have been done here is the concept of pipelining, a tradeoff between computation and communication. Went eh final layer is done with the particular computation it can, though a separate thread, send the output for reduce scatter in case we integrate it with DP too! The peak activation memory requirements reduces from O(N*L*k) to O(N + N*L*k / M * K), K nodes, L layers , M micro batches and N mini batches with a variable k. A few things: 1) increasing M helps 2) can work with slow inter node links like PCIe Cheers
Original Article
View Cached Full Text

Cached at: 07/06/26, 06:18 PM

After Data Parallelism we move to Model Parallelism starting with Pipeline Parallelism

The paper: GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism

The What? When models can no longer fit on a device (GPU) then we have to dissect the model itself across the GPUs and do the training, it’s called model parallelism. Now we can do it some ways, like splitting the layers of a model or splitting the tensors itself!

In this paper, authors propose effective methods to perform pipeline parallelism across N GPUs.

The Why? Since the layers are split across the GPUs, it will be sequential operation which makes it too slow because we cant move forward if the previous layers hasn’t finished its “working” which means the rest of GPUs have to wait - VERY BAD.

The How? They proposed two techniques to make it work:

  1. Micro batching: split the mini batches of data on each node into M microbatches and this means we wont be passing the whole mini batch at once to any GPU (because it will take some time and others will be waiting for data ) but rather process the said microbatch instead and quickly pass the output to the next node, while the current node can process the next micro batch and this goes on for all the nodes.

  2. Checkpointing: Makes use of the activation checkpointing concept, where you sacrifice computation for memory - you discard the activation throughout a particular layer of the model and only save the final output and when its needed during backward pass you recompute those activations.

Authors use this to save the final activations at partitions - when there is movement from one GPU to another, beaus ethane its as the input to the next layer.

Now one more thing that could have been done here is the concept of pipelining, a tradeoff between computation and communication. Went eh final layer is done with the particular computation it can, though a separate thread, send the output for reduce scatter in case we integrate it with DP too!

The peak activation memory requirements reduces from O(NLk) to O(N + NLk / M * K), K nodes, L layers , M micro batches and N mini batches with a variable k.

A few things:

  1. increasing M helps

  2. can work with slow inter node links like PCIe Cheers

Good pointers ! I’ll read that paper too soon

Thanks

Similar Articles