Another big tensor fix b9820
Summary
Introduces performance improvements by reducing synchronizations during split compute in the ggml backend, adds async CUDA copy capability, and makes sync relaxation more general across backends.
Similar Articles
ICYM: llama.cpp b9455 --SM Tensor KV Cache Fix is MERGED
llama.cpp version b9455 merges a fix for `-sm tensor` KV cache quantization on multi-GPU setups, addressing a shape information loss issue when flattening tensors.
@ggerganov: Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp m…
llama.cpp maintainers and NVIDIA engineers collaborated to significantly improve multi-GPU performance in ggml, enabling hardware-agnostic tensor parallelism and major performance gains on RTX systems.
NCCL-Free Tensor Parallelism on Dual Blackwell PCIe llama.cpp b9095 released!
llama.cpp build b9095 introduces NCCL-free tensor parallelism for dual Blackwell PCIe GPUs, enabling efficient multi-GPU inference without relying on NCCL.
ggml-webgpu: Improve prefill speeds for k-quants + refactor matmul for Q4/Q5/Q8 and k-quants by yomaytk · Pull Request #24225 · ggml-org/llama.cpp
Improves prefill speeds for k-quants and refactors matrix multiplication for Q4/Q5/Q8 and k-quants in llama.cpp's WebGPU backend.
Dual GPU llama.cpp speedup
A fork of llama.cpp fixes the --split-mode tensor issue with quantized KV caches, achieving up to 40% speed improvement on dual GPU setups without quality loss.