ml-performance

A tweet explains the correct answer to an ML performance interview question at Anthropic about the latency tradeoffs of splitting tensor-parallel linear layers by columns vs. rows when serving a 70B transformer model on 8 GPUs, highlighting that performance is not similar despite equal per-GPU weights.