@nrehiew_: This was great! A fun followup is given some arbitrary topology and architecture, compute what is the expected compute/…
Summary
A tweet reply to a blog post about collective communication in TPU/GPU clusters, suggesting a follow-up to compute expected compute/communication duration for a 1T MoE on Blackwell NVL72 and estimate theoretical MFU and bottlenecks.
View Cached Full Text
Cached at: 07/16/26, 12:01 AM
This was great! A fun followup is given some arbitrary topology and architecture, compute what is the expected compute/comm duration in a training step?
Say a 1T Dsv3 style MoE on Blackwell NVL72s, what sort of theoretical MFU can one expect? What is the largest bottleneck? https://t.co/l1WZIVh3oe
Aleksa Gordić (水平问题) (@gordic_aleksa): New in-depth blog post time: “Inside TPU and GPU Clusters: The Anatomy of Collective Communication”.
If you want to deeply understand the core primitives behind scaling the training / inference for MoEs and dense transformers, going a level below FSDP, expert parallelism, data
Similar Articles
@akshay_pachaar: https://x.com/akshay_pachaar/status/2087928032904523980
An educational thread explaining how GPUs work, focusing on the memory-compute asymmetry that dominates LLM serving performance, and demonstrating how techniques like quantization, speculative decoding, and continuous batching follow from that fundamental constraint.
@vivekgalatage: It's super interesting to know the system architecture of the TPUs. https://henryhmko.github.io/posts/tpu/tpu.html…
A deep dive into Google's TPU architecture, explaining the design philosophy of systolic arrays, pipelining, and ahead-of-time compilation that enables high throughput and energy efficiency.
@SzymonOzog_: Saturday reading: "What happens when you run a CUDA kernel" - very cool blogpost on the details about the CPU<->GPU com…
Tweet recommending a blog post that explains the CPU-GPU communication details required when launching a CUDA kernel.
@jerryjliu0: An interesting thing I'm observing from the blog/system card is that on a good chunk of the reported benchmarks (~20-30…
An observation that Claude Opus 5's max thinking leads to performance degradation on ~20-30% of benchmarks compared to xhigh, contrary to the expectation that more test-time compute improves performance.
@jino_rohit: https://x.com/jino_rohit/status/2067620031517860243
Explains the communication model for multi-GPU systems, covering the trade-off between latency and bandwidth, and compares MST and Ring algorithms for collective operations like broadcast.