@binsquares: omg, GPU acceleration on smolvm works way better than I thought. can run llama.cpp inside the smol machine with close t…
Summary
User @binsquares reports that GPU acceleration on smolvm achieves nearly 90% of host performance when running llama.cpp via the Vulkan backend.
View Cached Full Text
Cached at: 05/11/26, 04:43 PM
omg, GPU acceleration on smolvm works way better than I thought.
can run llama.cpp inside the smol machine with close to ~90% of performance of the host using vulkan backend.
~127 t/s on qwen-0.5b. Min hassle, max ease of setup. https://t.co/SQzj7PTs9P
Similar Articles
@ggerganov: Highlighting recent advances in multi-GPU and tensor parallel support in llama.cpp Over the last few months llama.cpp m…
llama.cpp maintainers and NVIDIA engineers collaborated to significantly improve multi-GPU performance in ggml, enabling hardware-agnostic tensor parallelism and major performance gains on RTX systems.
I didn't know it was possible to compile llamacpp to run cuda + vulkan at the same time..
The author discovered that compiling llama.cpp with both CUDA and Vulkan backends simultaneously is possible, yielding a ~10% improvement in tokens/sec for decoding. They plan to run further benchmarks to assess the benefits.
sycl : port multi-column MMVQ from CUDA backend (~45% speculative decoding speedup on Intel Arc) by masonmilby · Pull Request #21845 · ggml-org/llama.cpp
A pull request for llama.cpp ports multi-column MMVQ from CUDA to SYCL, achieving approximately 45% speculative decoding speedup on Intel Arc GPUs.
Dual GPU llama.cpp speedup
A fork of llama.cpp fixes the --split-mode tensor issue with quantized KV caches, achieving up to 40% speed improvement on dual GPU setups without quality loss.
vulkan: make TP viable by pwilkin · Pull Request #25051 · ggml-org/llama.cpp
This pull request makes tensor parallelism (TP) viable in llama.cpp when using the Vulkan backend, enabling distributed inference across multiple GPUs.