Tag
This article evaluates the AI inference performance of dual AMD Radeon AI PRO R9700 GPUs, comparing them to Intel Arc Pro B70 and NVIDIA RTX 5090, highlighting cost-effectiveness and software challenges.
A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.
An investigation into why dual RTX 5060 Ti GPUs max out at ~50% utilization when running large LLMs like Qwen 27B reveals that memory bandwidth is the bottleneck and layer-by-layer splitting causes idle time, making it a relay race rather than parallel computation.
An analysis of PCIe transfer performance when running llama.cpp with dual GPUs using pipeline and tensor parallelism.
Discusses the potential value of using two Radeon RX 9060 XT 16GB graphics cards in a CrossFire configuration, likely weighing performance vs. cost.
A comparison of AI inference frameworks ROCm, Vulkan, and vLLM running on dual AMD Radeon 9700 GPUs, likely benchmarking performance for large language models.
A user benchmarks dual-GPU inference speed on two RTX 3080 20GB using llama.cpp (row/tensor split) and ik_llama (graph split) with a Qwen3.6-27B GGUF model, comparing token generation and prompt processing speeds.
A user successfully set up a dual-GPU llama-cpp server with 48GB VRAM using an AMD Radeon PRO and 7800 XT via Vulkan in Docker on Kubuntu 24.04.
A fork of llama.cpp fixes the --split-mode tensor issue with quantized KV caches, achieving up to 40% speed improvement on dual GPU setups without quality loss.