Tag
Introduces Data-Centric Parallel (DCP), a method for training deep learning models on variable long sequences by dynamically adjusting runtime settings per batch, achieving up to 2.88x speedup on 32 H200 GPUs with only 10 lines of code integration.
Presents SNI-GNN, a SmartNIC-assisted full-graph GNN training system that reduces inter-node communication by predicting remote embeddings in-network, achieving 1.3–3.6x speedups with negligible accuracy loss.
Presents DG-FedReuse, a federated learning mechanism that reuses age-decayed cached client updates under a proxy-gradient threshold to reduce uplink communication, achieving significant modeled savings with minimal accuracy loss on image classification benchmarks.
This paper proposes FedSLM, a parameter-centric framework for federated fine-tuning of foundation models with heterogeneous compressed clients, using SVD-based decomposition and a weak-to-strong elicitation step to handle resource asymmetry. Experiments show it outperforms existing federated baselines while reducing client GPU memory by ~50%.
PyTorch 2.13 release brings FlexAttention to Apple Silicon with up to 12x speedups and 4x peak memory reduction for large-vocabulary models via fused LinearCrossEntropyLoss, alongside updates to distributed training, compilation, and on-device inference.
This paper presents SLAI T-Rex, a full-parameter post-training optimization framework for trillion-parameter MoE models on Ascend NPU SuperPOD, achieving 34.22% MFU and outperforming GPT-5.4-Mini on Operations Research tasks by 3.98 percentage points.
Proposes DiLoCo, a distributed optimization algorithm that enables training large language models across poorly connected devices with 500x less communication while matching fully synchronous performance.
PyTorch is reportedly adding a fast Thunderbolt communication backend for distributed model training on Macs.
FeLiX is a new federated learning orchestration framework that optimizes time-to-target accuracy on live interaction streams by handling transient client availability, dynamic data heterogeneity, and outcome delays. It introduces streaming-aware availability tiers, fresh-utility selection, and delay-robust aggregation, reducing wall-clock time by up to 2.37x and communication bandwidth by 1.30x versus state-of-the-art baselines.
AMD and Meta contributors ported PyTorch Monarch to AMD Instinct GPUs with ROCm, enabling fault-tolerant distributed training at scale. The blog details the engineering work and validation on large clusters.
This paper proposes using model merging techniques, specifically Iso-C aggregation, to improve the aggregation step in DiLoCo distributed training, resulting in a new method called IsoLoCo that outperforms DiLoCo on language model pre-training.
The article describes the porting of PyTorch Monarch, a distributed training runtime, to AMD GPUs with ROCm, enabling single-controller fault-tolerant training at scale and addressing reliability challenges in large-scale LLM training.
Nous Research introduces Psyche, a decentralized infrastructure for training large language models on distributed heterogeneous hardware, using novel optimizers DeMo and DisTrO to dramatically reduce communication overhead.
The author shares experiences and insights from nnScaler to large-scale distributed training systems, discussing correctness, flexibility, boundary expansion, and the challenges brought by post-training and reinforcement learning.
A curated list of resources for mastering GPU engineering for AI systems, covering CUDA, ROCm, optimization tools, multi-GPU orchestration, and distributed training.
A tweet listing essential skills for AI infrastructure engineers, covering GPU fundamentals, inference optimization, distributed training, and production deployment.
Miles is an open source framework from RadixArk for large-scale LLM reinforcement learning post-training, integrating PyTorch, Ray, SGLang, and NVIDIA Megatron-LM with support for MoE, low-precision, and fault tolerance.
Harvard has open-sourced a comprehensive two-volume Machine Learning Systems textbook that covers engineering AI systems for real-world constraints, including distributed training, production inference, edge deployment, and governance, with hands-on components like TinyTorch, hardware kits, and interactive tools.
DataStates-LLM introduces a scalable checkpointing architecture for transformer models using composable state providers, achieving up to 4x higher throughput and reducing training time by 2.2x compared to existing solutions.
Sharing a machine learning systems notes repo on GitHub, covering distributed computing, parallelization, quantization, and PyTorch internals related to LLM training and inference. Suitable for learners interested in ML systems.