Tag
PyTorch is reportedly adding a fast Thunderbolt communication backend for distributed model training on Macs.
FeLiX is a new federated learning orchestration framework that optimizes time-to-target accuracy on live interaction streams by handling transient client availability, dynamic data heterogeneity, and outcome delays. It introduces streaming-aware availability tiers, fresh-utility selection, and delay-robust aggregation, reducing wall-clock time by up to 2.37x and communication bandwidth by 1.30x versus state-of-the-art baselines.
AMD and Meta contributors ported PyTorch Monarch to AMD Instinct GPUs with ROCm, enabling fault-tolerant distributed training at scale. The blog details the engineering work and validation on large clusters.
This paper proposes using model merging techniques, specifically Iso-C aggregation, to improve the aggregation step in DiLoCo distributed training, resulting in a new method called IsoLoCo that outperforms DiLoCo on language model pre-training.
The article describes the porting of PyTorch Monarch, a distributed training runtime, to AMD GPUs with ROCm, enabling single-controller fault-tolerant training at scale and addressing reliability challenges in large-scale LLM training.
Nous Research introduces Psyche, a decentralized infrastructure for training large language models on distributed heterogeneous hardware, using novel optimizers DeMo and DisTrO to dramatically reduce communication overhead.
The author shares experiences and insights from nnScaler to large-scale distributed training systems, discussing correctness, flexibility, boundary expansion, and the challenges brought by post-training and reinforcement learning.
A curated list of resources for mastering GPU engineering for AI systems, covering CUDA, ROCm, optimization tools, multi-GPU orchestration, and distributed training.
A tweet listing essential skills for AI infrastructure engineers, covering GPU fundamentals, inference optimization, distributed training, and production deployment.
Miles is an open source framework from RadixArk for large-scale LLM reinforcement learning post-training, integrating PyTorch, Ray, SGLang, and NVIDIA Megatron-LM with support for MoE, low-precision, and fault tolerance.
Harvard has open-sourced a comprehensive two-volume Machine Learning Systems textbook that covers engineering AI systems for real-world constraints, including distributed training, production inference, edge deployment, and governance, with hands-on components like TinyTorch, hardware kits, and interactive tools.
DataStates-LLM introduces a scalable checkpointing architecture for transformer models using composable state providers, achieving up to 4x higher throughput and reducing training time by 2.2x compared to existing solutions.
Sharing a machine learning systems notes repo on GitHub, covering distributed computing, parallelization, quantization, and PyTorch internals related to LLM training and inference. Suitable for learners interested in ML systems.
A discussion about pooling GPUs from a community to train a massive AI model, questioning the feasibility and existing projects despite known bottlenecks like latency and weight poisoning.
A new in-depth blog post explains collective communication for multiple GPUs, covering primitives like broadcast and reduce, and helps beginners understand how to scale experiments.
Summary of Lecture 19 on efficient AI distributed training, covering data, pipeline, tensor, and sequence parallelism methods with notes on memory and communication bottlenecks.
The author announces a new blog post on clustering three Jetson Nano Orin Supers for distributed training and inference, continuing a series to help people build small compute clusters with accessible hardware.
Anyscale demonstrates a 20x speedup in cross-region training data reads by using Alluxio NVMe caching with Ray Data, showing warm cache reads drop from 4,241 to 208 seconds for 1TB.
AgentJet is a distributed swarm training framework for LLM agent reinforcement learning that decouples agent rollouts from model optimization, enabling heterogeneous multi-agent RL, multi-task training, fault tolerance, and live code iteration with 1.5-10x training speedup. It also introduces an automated research system capable of autonomously conducting multi-day RL studies on large-scale clusters.
A deep dive into the internals of ByteDance's verl RL post-training framework, including orchestration, single-controller pattern, and a tricky NCCL bug fix. The author shares lessons from forking the framework and building custom tooling.