Tag
At PyTorchCon 2026, Zhipeng Wang of Google shared DeepSpeed technical advances for large-scale model training, including AutoSP automatic sequence parallelism, Muon optimizer support, and AutoTP/AutoEP for Hugging Face models, alongside ongoing open-source community collaboration.
The paper introduces Allspark, a training and inference framework for transferring reasoning improvements from a weak model to a strong model via alternating chains of thought, with experiments demonstrating accuracy gains in math, reasoning, and ARC-AGI-2 tasks.
This paper improves higher-order optimizers SOAP and Muon for large-scale LLM pretraining, addressing instabilities at large batch sizes and introducing a layer-wise distributed optimizer compatible with Megatron-LM. Experiments show they consistently outperform AdamW at billion-parameter scales.
The developer of a 35B-A6B model reports gaining access to a powerful server with 768GB VRAM and 1.5TB RAM, enabling full-scale evaluations and large-scale tuning. The breakthrough accelerates development of a high-quality AI model.
The author shares experiences and insights from nnScaler to large-scale distributed training systems, discussing correctness, flexibility, boundary expansion, and the challenges brought by post-training and reinforcement learning.
OmniOpt presents a unified framework for optimizer selection in large-scale model training, combining meta-pipeline transformations, norm-constrained linear minimization oracles, and a cross-domain benchmark to systematically analyze optimizer families and their trade-offs.
Prime Intellect releases prime-rl v0.6.0, enabling efficient reinforcement learning at trillion-parameter scale on large Mixture-of-Experts models, with sub-5-minute step times and optimizations for asynchronous RL.
Miles is an enterprise-grade reinforcement learning framework for large-scale model post-training, featuring high-performance rollout with SGLang and scalable training with Megatron-LM.