Tag
该论文提出 LeMA，一种用于多 GPU 大规模神经网络训练的分布式 Levenberg-Marquardt 算法，通过精确对偶公式与 Jacobian 分片策略，将内存开销从参数量平方降低到批量大小平方，并在约 10 亿参数 CNN 上实现 8 GPU 近线性扩展。
该论文研究了分布式训练中外部Nesterov动量在随机伪梯度上的稳定性，推导了随机二次模型上的精确均方稳定性判据，揭示了内层批量大小、本地步数与工作节点平均对训练稳定性的联合影响，并在神经网络上验证了理论结论。
TorchNeuron 作为 PyTorch 的原生 Trainium 后端发布公测，Hugging Face 的 Transformers、Diffusers 等库可直接将其作为普通设备使用，显著提升了模型支持率并取代 Optimum Neuron 的复杂封装。
PyTorch 博客介绍 Zephon 与 TorchTitan 的集成，实现弹性确定性数据加载：训练从 16 GPU 检查点在 8、32、64 GPU 上恢复时，每个全局批次的 SHA-256 哈希与原始运行完全一致。
论文提出 Optimal Hydra，一种面向 LLM 微调的模型异构联邦学习框架，通过全局层级选择与基于客户端数据分布的注意力头选择构建客户端专属子网络（Hydra-nets），在三个数据集上显著优于基线方法并降低本地资源需求。
Chef Robotics 训练其食品组装机器人背后的 Vision-Language-Action 模型速度提升 10 倍，在 16 块 A100 上实现 92% 的多 GPU 扩展效率，依托 Anyscale 与 Ray 平台加速分布式训练。
该论文研究了分布式LLM训练中内部优化器如何影响副本扩展（replica scaling），识别出伪梯度协方差结构与噪声尺度两个关键因素，并从理论和实证上证明基于白化的预条件优化器能实现最强的worker扩展性能。
We trained a model on agentic coding tasks using fully asynchronous RL with mixed-precision, mixed-accelerator samplers, achieving training stability on par with homogeneous clusters at lower cost by applying off-policy correction.
DisaggKD 提出将教师与学生模型分离到不同节点上进行知识蒸馏，并设计一种融合损失在行窗口与词汇表分片上等价于全词汇表损失，从而突破单 GPU 内存限制，在 Qwen3.5 等大模型对上实现与同位部署相当的损失且训练更快。
ByteDance and Fudan University researchers present ByteExpress, a software communication layer over RDMA for large-scale parameter-server inference and training, reporting up to 74% lower P99 push latency and significant CPU savings versus TCP in production deployments.
The paper introduces correlated expert placement and token shuffling to exploit router-learned expert correlations, cutting all-to-all communication time by 1.16–2.63× and end-to-end MoE pretraining step time by up to 1.41× in Megatron-LM without altering routing decisions or model parameters.
PyTorch's Accelerator Integration Working Group published a technical recap of H1 2026 progress, including the Cross-Repository CI Relay (CRCR), refactored device-agnostic test suites, maturation of the OpenReg reference backend, distributed and torch.compile support, and the new Additional Compute Platforms portal to standardize vendor-neutral hardware enablement.
Aleph Alpha describes how they scaled pre-training of their 30B-A3B MoE model (revealed to be Kolibri Origin) from 16 to 512 NVIDIA B200 GPUs using a hierarchical hyperparameter search approach, achieving 35.3% MFU and near-linear scaling just 6% below ideal, trained with a custom fork of torchtitan over InfiniBand/NVLink.
A developer shares lessons learned while building Apex-2, a small open LLM trained on ~80B tokens for about $2,000, covering GPU bottlenecks, DiLoCo-style distributed training with GH200 instances, and aggressive MinHash deduplication of FineWeb-Edu and DCLM.
SkyRL v0.4.0 releases with support for GLM 5.3 Flash/5.3, Kimi K2.7, Qwen3.8 and Nemotron 3.5-Lightning, alongside full FP8/MXFP8 RL, Top-P sampler replay, and sharded P2P weight transfer for large-scale post-training.
Ai2 发布 Olmo-core 3，一款面向大规模 MoE 训练的开源框架升级，采用 DDP-based 路由设计与多种并行技术，可在保持约 3.2B 激活参数的情况下将总参数扩展至万亿级别，吞吐量较旧实现提升约 2.7 倍。
AllenAI released Olmo-core 3, an open, scalable training framework redesigned for large mixture-of-experts LLMs, benchmarked at over one trillion total parameters with minimal throughput loss as the expert pool grows.
The paper introduces FLIP (Federated Learning Interoperability Platform), an open-source multi-application platform that makes cross-continental federated learning in healthcare repeatable, demonstrated via federated fine-tuning and evaluation on synthetic chest X-ray cohorts across UK and Thailand nodes.
SlideDP presents an asynchronous data-parallel runtime for host-resident layer streaming that fine-tunes LLMs beyond GPU memory across multiple GPUs, achieving 1.46–2.64x throughput over SlideFormer, MegaTrain, and ZeRO-Offload and even exceeding FSDP2 peak throughput while fine-tuning Qwen2.5-72B on four RTX 4090s.
Nereus is a cost-aware runtime that adapts reinforcement learning post-training jobs for large language models to efficient execution plans, reducing latency by 27.7% and improving throughput up to 7.27 times over existing tools.