Tag
A tweet reply to a blog post about collective communication in TPU/GPU clusters, suggesting a follow-up to compute expected compute/communication duration for a 1T MoE on Blackwell NVL72 and estimate theoretical MFU and bottlenecks.
Explains the communication model for multi-GPU systems, covering the trade-off between latency and bandwidth, and compares MST and Ring algorithms for collective operations like broadcast.
A new in-depth blog post explains collective communication for multiple GPUs, covering primitives like broadcast and reduce, and helps beginners understand how to scale experiments.