Tag
This thread explains GPipe, a paper on pipeline parallelism for scaling large models across multiple GPUs using micro-batching and activation checkpointing.