@ggerganov: Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp m…
Summary
llama.cpp maintainers and NVIDIA engineers collaborated to significantly improve multi-GPU performance in ggml, enabling hardware-agnostic tensor parallelism and major performance gains on RTX systems.
Similar Articles
vulkan: make TP viable by pwilkin · Pull Request #25051 · ggml-org/llama.cpp
This pull request makes tensor parallelism (TP) viable in llama.cpp when using the Vulkan backend, enabling distributed inference across multiple GPUs.
@analogalok: Stop blindly trusting the default multi GPU settings for your Local LLMs. You are literally leaving 25% performance on …
Benchmark results comparing layer vs. tensor parallelism in llama.cpp for dual GPU setups: layer mode is 25% faster for prefill (RAG pipelines), while tensor mode is 16% faster for decode (interactive chat).
NCCL-Free Tensor Parallelism on Dual Blackwell PCIe llama.cpp b9095 released!
llama.cpp build b9095 introduces NCCL-free tensor parallelism for dual Blackwell PCIe GPUs, enabling efficient multi-GPU inference without relying on NCCL.
Comparing dual-GPU inference speed between llama.cpp row/tensor split and ik_llama graph split
A user benchmarks dual-GPU inference speed on two RTX 3080 20GB using llama.cpp (row/tensor split) and ik_llama (graph split) with a Qwen3.6-27B GGUF model, comparing token generation and prompt processing speeds.
CUDA: add fast walsh-hadamard transform by am17an · Pull Request #23615 · ggml-org/llama.cpp
This pull request adds a fast Walsh-Hadamard transform implementation for CUDA in llama.cpp, a popular open-source LLM inference engine. The optimization enhances performance for certain computational operations on NVIDIA GPUs.