PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Papers with Code Trending Papers
pytorch distributed-training data-parallelism deep-learning gpu-scalability

Summary

This paper details the design and optimization of PyTorch's distributed data parallel module, highlighting techniques like gradient bucketing and computation-communication overlap that enable near-linear scalability across 256 GPUs.

This paper presents the design, implementation, and evaluation of the PyTorch distributed data parallel module. PyTorch is a widely-adopted scientific computing package used in deep learning research and applications. Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. In general, the technique of distributed data parallelism replicates the model on every computational resource to generate gradients independently and then communicates those gradients at each iteration to keep model replicas consistent. Despite the conceptual simplicity of the technique, the subtle dependencies between computation and communication make it non-trivial to optimize the distributed training efficiency. As of v1.5, PyTorch natively provides several techniques to accelerate distributed data parallel, including bucketing gradients, overlapping computation with communication, and skipping gradient synchronization. Evaluations show that, when configured appropriately, the PyTorch distributed data parallel module attains near-linear scalability using 256 GPUs.
Original Article
View Cached Full Text

Cached at: 05/08/26, 08:57 AM

Paper page - PyTorch Distributed: Experiences on Accelerating Data Parallel Training

Source: https://huggingface.co/papers/2006.15704 Published on Jun 28, 2020

Abstract

The PyTorch distributed data parallel module optimizes large-scale model training using techniques like gradient bucketing, computation-communication overlap, and selective synchronization to achieve near-linear scalability.

This paper presents the design, implementation, and evaluation of thePyTorchdistributed data parallelmodule.PyTorchis a widely-adopted scientific computing package used in deep learning research and applications. Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. In general, the technique ofdistributed data parallelism replicates the model on every computational resource to generate gradients independently and then communicates those gradients at each iteration to keep model replicas consistent. Despite the conceptual simplicity of the technique, the subtle dependencies between computation and communication make it non-trivial to optimize the distributed training efficiency. As of v1.5,PyTorchnatively provides several techniques to accelerate distributed data parallel, including bucketing gradients, overlapping computation with communication, and skipping gradient synchronization. Evaluations show that, when configured appropriately, thePyTorchdistributed data parallelmodule attains near-linear scalability using 256 GPUs.

View arXiv pageView PDFGitHub99.7kautoAdd to collection

Get this paper in your agent:

hf papers read 2006\.15704

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2006.15704 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2006.15704 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2006.15704 in a Space README.md to link it from this page.

Collections including this paper2

Similar Articles

Training Variable Long Sequences with Data-Centric Parallel

arXiv cs.AI

Introduces Data-Centric Parallel (DCP), a method for training deep learning models on variable long sequences by dynamically adjusting runtime settings per batch, achieving up to 2.88x speedup on 32 H200 GPUs with only 10 lines of code integration.

Techniques for training large neural networks

OpenAI Blog

OpenAI presents comprehensive techniques for training large neural networks across distributed GPU clusters, covering data parallelism, pipeline parallelism, tensor parallelism, and mixture-of-experts approaches to overcome engineering and scalability challenges.