@gpusteve: you're interviewing for an ml performance role at anthropic and they ask: "you're serving a 70b transformer model on 8 …
Summary
A tweet explains the correct answer to an ML performance interview question at Anthropic about the latency tradeoffs of splitting tensor-parallel linear layers by columns vs. rows when serving a 70B transformer model on 8 GPUs, highlighting that performance is not similar despite equal per-GPU weights.
View Cached Full Text
Cached at: 06/08/26, 05:19 AM
you’re interviewing for an ml performance role at anthropic and they ask:
“you’re serving a 70b transformer model on 8 gpus. what are the latency tradeoffs of splitting tensor-parallel linear layers by columns vs rows?”
you say: “either way each gpu gets 1/8 of the weights, so performance should be similar.” wrong.
here’s how you answer:
start with what the linear layer is doing.
it takes the current hidden states X and multiplies them by a weight matrix W to produce Y.
tensor parallelism shards W across gpus. the important detail is which dimension you shard, because that decides the shape of the partial outputs.
real
real
Similar Articles
@_avichawla: A tricky LLM interview question: You're serving a reasoning model on vLLM, and it keeps running out of GPU memory on lo…
Explains why evicting 90% of KV cache tokens fails to free GPU memory when serving reasoning models on vLLM, due to paged attention fragmentation, and introduces NVIDIA's TriAttention as a solution that achieves 2.5x speedup and 10.7x memory reduction.
Optimizing Transformer model size & inference beyond FP16 + ONNX (pruning/graph opt didn’t help much) [P]
Author shares experience hitting diminishing returns with FP16 + ONNX + pruning on 162 MB transformer, seeks advice on next best steps among quantization, distillation, low-rank factorization, or hardware-specific tricks.
@TheNoise2Signal: How does frontier training use 2,048 GPUs? Because there are five dimensions you can split work across - and at scale, …
Explains how frontier AI training uses up to 2,048 GPUs by splitting work across five dimensions, demystifying model training frameworks.
I compared all specs of the major GPUs/machines that are being used here, because bandwidth is not everything. Some of ya'll need a reality check.
The author compares various GPUs for LLM inference, critiquing common benchmarks and emphasizing the importance of prefill performance over generation speed, offering recommendations for different budgets and use cases.
@TheAhmadOsman: Why do I focus on Inference Engines/Software Stacks for your hardware? - 2x RTX 3090s: ~14.5 tok/s → ~64 tok/s moving t…
Comparison of inference engine performance on different hardware: moving from baseline to vLLM with TP=2 on 2x RTX 3090s improves from ~14.5 tok/s to ~64 tok/s, and on RTX PRO 6000 moving to Sglang improves from ~32 tok/s to ~110 tok/s. Recommends vLLM/Sglang for CUDA/multi-GPU and llama.cpp for edge devices.