Tag
OxCaml, Jane Street's fork of the OCaml compiler, introduces compile-time guarantees against data races, enabling sequential consistency without runtime overhead. The blog post explains the new mode axes and their implications for parallel programming.
This paper introduces DisagMoE, a system for MoE training that optimizes computation-communication overlap by disaggregating attention and FFN layers across GPU groups. Implemented on Megatron-LM, it achieves up to 1.8x speedup on H800 clusters by addressing inter-node communication bottlenecks.
OpenAI presents comprehensive techniques for training large neural networks across distributed GPU clusters, covering data parallelism, pipeline parallelism, tensor parallelism, and mixture-of-experts approaches to overcome engineering and scalability challenges.