tensor-parallelism

A guide on using Kaggle's free dual Tesla T4 GPUs (32GB VRAM) to run large LLMs with massive context windows, covering multi-GPU parallelism strategies in llama.cpp.

Detailed findings on PCIe bifurcation and P2P performance issues with 4x GPU setups, including workarounds and alternatives for tensor and pipeline parallelism.

This pull request makes tensor parallelism (TP) viable in llama.cpp when using the Vulkan backend, enabling distributed inference across multiple GPUs.

Summary of Lecture 19 on efficient AI distributed training, covering data, pipeline, tensor, and sequence parallelism methods with notes on memory and communication bottlenecks.

A user reports near-linear performance scaling when adding a second RTX 3090 for inference with a Qwen model, achieving roughly 1.8x decode TPS improvement without NVLink.

A tweet explains the correct answer to an ML performance interview question at Anthropic about the latency tradeoffs of splitting tensor-parallel linear layers by columns vs. rows when serving a 70B transformer model on 8 GPUs, highlighting that performance is not similar despite equal per-GPU weights.

llama.cpp maintainers and NVIDIA engineers collaborated to significantly improve multi-GPU performance in ggml, enabling hardware-agnostic tensor parallelism and major performance gains on RTX systems.

This paper proposes PAT, an adaptive tensor parallelism method that dynamically reconfigures TP during the generation stage of synchronous RLHF training to mitigate long-tail generation bottlenecks. Evaluations on LLaMA3.1-8B and Qwen3-14B show reductions in generation latency by up to 34.6% and end-to-end iteration latency by up to 27.2%.

A user reflects on the complexity and fascination of running local LLMs, touching on hardware selection, quantization, and tensor parallelism.

A learner shares enthusiasm for Stanford CS336 lecture 7 on GPU parallelism, which covers fundamental operations and connects them to multi-GPU setups and parallelism techniques like tensor, data, and pipeline parallelism.

A fork of llama.cpp fixes the --split-mode tensor issue with quantized KV caches, achieving up to 40% speed improvement on dual GPU setups without quality loss.

At PyTorchCon Europe 2026, Edward Yang explains PyTorch's new pre-compilation support for distributed training and SPMD type system to help developers write correct tensor parallelism code, addressing common pitfalls in gradient correctness.

llama.cpp build b9095 introduces NCCL-free tensor parallelism for dual Blackwell PCIe GPUs, enabling efficient multi-GPU inference without relying on NCCL.