Tag
This paper presents ZeroLock, a backpropagation-free algorithm for concurrent memory-efficient LLM training that decouples model updates into independent chunk updates, reducing memory usage by 26.5% and improving throughput by 4.9% compared to BP-based baselines.
Agora enables collective, permissionless internet-scale pretraining of large language models using heterogeneous, preemptible consumer GPUs connected via internet, demonstrated by the successful Pluralis-8B training run with 330 nodes.
This paper explores using retired GPUs to build low-cost clusters for serving LLaMA-70B, finding economic viability in regions with cheap electricity but highlighting potential high carbon emissions without clean energy sources.
A guide on using Kaggle's free dual Tesla T4 GPUs (32GB VRAM) to run large LLMs with massive context windows, covering multi-GPU parallelism strategies in llama.cpp.
This thread explains GPipe, a paper on pipeline parallelism for scaling large models across multiple GPUs using micro-batching and activation checkpointing.
Detailed findings on PCIe bifurcation and P2P performance issues with 4x GPU setups, including workarounds and alternatives for tensor and pipeline parallelism.
Summary of Lecture 19 on efficient AI distributed training, covering data, pipeline, tensor, and sequence parallelism methods with notes on memory and communication bottlenecks.
Testing shows that default pipeline parallelism in llama.cpp wastes VRAM with no speed benefit; compiling with GGML_SCHED_MAX_COPIES=1 saves significant VRAM while maintaining identical inference speed.
This paper introduces MAPL, a method for learned orthogonal compression of activations in pipeline parallelism, reducing communication overhead while maintaining performance via Stiefel manifold constraints and per-stage factorized anchor embeddings.
This paper proposes Speculative Pipeline Decoding (SPD), a framework that uses pipeline parallelism within a single LLM to enable parallel token speculation, avoiding the latency bubbles and accuracy degradation of multi-token prediction in traditional speculative decoding.
SpaceX is finalizing a custom AI training stack written in C, utilizing pipeline parallelism and 220k GB300 GPUs to achieve over an order of magnitude speed improvement, with plans to develop an inference stack for reinforcement learning.
A learner shares enthusiasm for Stanford CS336 lecture 7 on GPU parallelism, which covers fundamental operations and connects them to multi-GPU setups and parallelism techniques like tensor, data, and pipeline parallelism.
This article benchmarks vLLM, SGLang, and llama.cpp on a mixed Blackwell/Ada GPU cluster for long context prefill, finding vLLM significantly outperforms others on heterogeneous setups while SGLang crashes with Ada cards due to FP4 support limitations.
ResBM introduces a transformer-based architecture with residual encoder-decoder bottlenecks for pipeline-parallel training, achieving 128× activation compression while maintaining convergence. The work advances decentralized, internet-grade distributed training by reducing inter-stage communication overhead.