multi-gpu

The article details a validated hardware configuration using 16 RTX 5060 Ti GPUs with PLX switches to run the Deepseek V4 Flash model, achieving specific performance metrics for context handling and throughput.

The author benchmarked llama.cpp flags on a hybrid GPU setup with an RTX 4090 laptop and AMD XTX 7900 eGPU, achieving 70% faster generation, 40% faster prefill, and discovering a bug related to MTP in multi-GPU configurations.

A user seeks advice on building a budget home AI server with 32-48GB VRAM, debating between AMD RX 9060 XT and Nvidia RTX 5060 Ti GPUs, and whether to use AM5 or used EPYC platforms for local LLM inference and large MoE model offloading.

Technical post sharing performance stats for running Qwen 27B on 2x RTX 5070 Ti GPUs with vLLM cu129-nightly, achieving up to 94-87 tps decode and 170k GPU KV cache.

Y Combinator hosted a Paper Club where researchers presented innovations in multi-GPU kernel optimization, including ParallelKittens, a CUDA framework that simplifies development of overlapped multi-GPU kernels and achieves significant speedups across workloads.

Testing reveals that Intel consumer platforms like Z890 with Arrow Lake CPUs have hardware/firmware limitations that prevent proper PCIe Peer-to-Peer (P2P) communication between multiple GPUs, making them unsuitable for multi-GPU AI workloads despite adequate lane counts.

Describes running the GLM-5.2 model with 4-bit quantization at 12.2 tokens per second on a cluster of 16 AMD MI50 GPUs using llama.cpp's RPC, achieving coherent long-form generation at 10.7k context.

User shares benchmark results running the 550B Nemotron Ultra model across two machines using RPC, achieving impressive throughput on older AMD MI50 and Nvidia P40 GPUs.

A user benchmarks AMD MI50 GPUs across different PCIe configurations on an older X99 motherboard, comparing direct PCIe connections vs using a PEX8749 switch. Results show minimal performance difference with slight improvement in token generation speed.

Benchmark results of the unsloth/Qwen3.6-27B-NVFP4 model running on 4x RTX 5060 Ti GPUs with peer-to-peer and pipeline parallelism at various concurrency levels.

Benchmark results comparing layer vs. tensor parallelism in llama.cpp for dual GPU setups: layer mode is 25% faster for prefill (RAG pipelines), while tensor mode is 16% faster for decode (interactive chat).

A guide on using Kaggle's free dual Tesla T4 GPUs (32GB VRAM) to run large LLMs with massive context windows, covering multi-GPU parallelism strategies in llama.cpp.

Wan-Streamer v0.2 is a latency-preserving upgrade to an end-to-end audio-visual interaction model, increasing output resolution from 192x336 to 640x368 while maintaining ~200 ms model-side latency via a multi-GPU thinker-performer architecture.

A report on running the GLM5.2 language model across 5 AMD Radeon Pro 6000 GPUs and an NVIDIA RTX 5090, detailing the high cost and technical challenges.

Ahmad (@TheAhmadOsman) announces a blogpost covering inference engines like llama.cpp, vLLM, and ExLlamaV2, focusing on multi-GPU setups, tensor parallelism, and batch inference for optimized AI model performance.

Technical report on running Qwen 3.6 27B Q8 model on a dual AMD Radeon R9700 setup using llama.cpp with ROCm, including performance benchmarks and configuration details.

A user shares optimized settings for running Qwen3.6 27B (Q8_0) on a dual GPU setup (RTX 4090 + RTX 3090) with llama.cpp, achieving 75-100 t/s and 1500 pp with 250k context.

A discussion on the cheapest local hardware setups for running GLM 5.x and similarly sized models at 4-bit quantization, including CPU-only and multi-GPU options, with a user sharing their experience running Minimax 2.7 and Qwen 3.6 on a 5900X + 128GB DDR4 + 7900XT setup.

The author discovered that compiling llama.cpp with both CUDA and Vulkan backends simultaneously is possible, yielding a ~10% improvement in tokens/sec for decoding. They plan to run further benchmarks to assess the benefits.

A user discovered that a hidden PCIe 2.0 x4 electrical limitation on a Threadripper workstation board was crippling one of four RTX 3090s, causing poor multi-GPU LLM inference performance. Fixing the slot layout and switching to tensor split mode doubled Mistral 128B throughput from ~11 to ~24.7 tok/s.