Tag
The Futhark team explains why their compiler's GPU pipelines force all arrays into GPU memory, making sequential CPU-side work (like greedy distance-2 graph colouring for sparse Jacobians) three orders of magnitude slower, and describes a hack that allows keeping some functions off the GPU while still exploiting GPU parallelism for the rest of the program.
SlideDP presents an asynchronous data-parallel runtime for host-resident layer streaming that fine-tunes LLMs beyond GPU memory across multiple GPUs, achieving 1.46–2.64x throughput over SlideFormer, MegaTrain, and ZeRO-Offload and even exceeding FSDP2 peak throughput while fine-tuning Qwen2.5-72B on four RTX 4090s.
The author shares six months of measurements on a four-RTX 3090 local setup, revealing that data parallelism often outperforms tensor parallelism for models fitting on fewer cards, with up to 3.4x throughput difference.
Summary of Lecture 19 on efficient AI distributed training, covering data, pipeline, tensor, and sequence parallelism methods with notes on memory and communication bottlenecks.
This paper details the design and optimization of PyTorch's distributed data parallel module, highlighting techniques like gradient bucketing and computation-communication overlap that enable near-linear scalability across 256 GPUs.
This lecture introduces the flexible evolution of GPU architecture as a SIMD (vector/array) processor, discusses data parallelism, memory bank grouping, bank conflicts, serial bottlenecks, and the history of SIMD instructions (such as MMX), emphasizing how GPUs leverage data parallelism and deal with serial bottlenecks.