@ggerganov: Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp m…
Summary
llama.cpp maintainers and NVIDIA engineers collaborated to significantly improve multi-GPU performance in ggml, enabling hardware-agnostic tensor parallelism and major performance gains on RTX systems.
Similar Articles
vulkan: make TP viable by pwilkin · Pull Request #25051 · ggml-org/llama.cpp
This pull request makes tensor parallelism (TP) viable in llama.cpp when using the Vulkan backend, enabling distributed inference across multiple GPUs.
@analogalok: Stop blindly trusting the default multi GPU settings for your Local LLMs. You are literally leaving 25% performance on …
Benchmark results comparing layer vs. tensor parallelism in llama.cpp for dual GPU setups: layer mode is 25% faster for prefill (RAG pipelines), while tensor mode is 16% faster for decode (interactive chat).
NCCL-Free Tensor Parallelism on Dual Blackwell PCIe llama.cpp b9095 released!
llama.cpp build b9095 introduces NCCL-free tensor parallelism for dual Blackwell PCIe GPUs, enabling efficient multi-GPU inference without relying on NCCL.
vulkan: int8 coopmat1 matmul implementation for AMD RDNA3 and RDNA4 (… · ggml-org/llama.cpp@70c4e15
This update to llama.cpp introduces a Vulkan-based implementation of int8 cooperative matrix multiplication for AMD RDNA3 and RDNA4 GPUs, enhancing inference performance on these hardware platforms.
Comparing dual-GPU inference speed between llama.cpp row/tensor split and ik_llama graph split
A user benchmarks dual-GPU inference speed on two RTX 3080 20GB using llama.cpp (row/tensor split) and ik_llama (graph split) with a Qwen3.6-27B GGUF model, comparing token generation and prompt processing speeds.