multi-gpu

Y Combinator hosted a Paper Club where researchers presented innovations in multi-GPU kernel optimization, including ParallelKittens, a CUDA framework that simplifies development of overlapped multi-GPU kernels and achieves significant speedups across workloads.

Testing reveals that Intel consumer platforms like Z890 with Arrow Lake CPUs have hardware/firmware limitations that prevent proper PCIe Peer-to-Peer (P2P) communication between multiple GPUs, making them unsuitable for multi-GPU AI workloads despite adequate lane counts.

Describes running the GLM-5.2 model with 4-bit quantization at 12.2 tokens per second on a cluster of 16 AMD MI50 GPUs using llama.cpp's RPC, achieving coherent long-form generation at 10.7k context.

User shares benchmark results running the 550B Nemotron Ultra model across two machines using RPC, achieving impressive throughput on older AMD MI50 and Nvidia P40 GPUs.

A user benchmarks AMD MI50 GPUs across different PCIe configurations on an older X99 motherboard, comparing direct PCIe connections vs using a PEX8749 switch. Results show minimal performance difference with slight improvement in token generation speed.

Benchmark results of the unsloth/Qwen3.6-27B-NVFP4 model running on 4x RTX 5060 Ti GPUs with peer-to-peer and pipeline parallelism at various concurrency levels.

Benchmark results comparing layer vs. tensor parallelism in llama.cpp for dual GPU setups: layer mode is 25% faster for prefill (RAG pipelines), while tensor mode is 16% faster for decode (interactive chat).

A guide on using Kaggle's free dual Tesla T4 GPUs (32GB VRAM) to run large LLMs with massive context windows, covering multi-GPU parallelism strategies in llama.cpp.

Wan-Streamer v0.2 is a latency-preserving upgrade to an end-to-end audio-visual interaction model, increasing output resolution from 192x336 to 640x368 while maintaining ~200 ms model-side latency via a multi-GPU thinker-performer architecture.

A report on running the GLM5.2 language model across 5 AMD Radeon Pro 6000 GPUs and an NVIDIA RTX 5090, detailing the high cost and technical challenges.

Ahmad (@TheAhmadOsman) announces a blogpost covering inference engines like llama.cpp, vLLM, and ExLlamaV2, focusing on multi-GPU setups, tensor parallelism, and batch inference for optimized AI model performance.

Technical report on running Qwen 3.6 27B Q8 model on a dual AMD Radeon R9700 setup using llama.cpp with ROCm, including performance benchmarks and configuration details.

A user shares optimized settings for running Qwen3.6 27B (Q8_0) on a dual GPU setup (RTX 4090 + RTX 3090) with llama.cpp, achieving 75-100 t/s and 1500 pp with 250k context.

A discussion on the cheapest local hardware setups for running GLM 5.x and similarly sized models at 4-bit quantization, including CPU-only and multi-GPU options, with a user sharing their experience running Minimax 2.7 and Qwen 3.6 on a 5900X + 128GB DDR4 + 7900XT setup.

The author discovered that compiling llama.cpp with both CUDA and Vulkan backends simultaneously is possible, yielding a ~10% improvement in tokens/sec for decoding. They plan to run further benchmarks to assess the benefits.

A user discovered that a hidden PCIe 2.0 x4 electrical limitation on a Threadripper workstation board was crippling one of four RTX 3090s, causing poor multi-GPU LLM inference performance. Fixing the slot layout and switching to tensor split mode doubled Mistral 128B throughput from ~11 to ~24.7 tok/s.

llama.cpp maintainers and NVIDIA engineers collaborated to significantly improve multi-GPU performance in ggml, enabling hardware-agnostic tensor parallelism and major performance gains on RTX systems.

User reports a CUDA error when using tensor split mode with the latest llama.cpp and Qwen-3.6-27b model on dual RTX 3090s with Ubuntu Server 24.04 and Docker.

llama.cpp version b9455 merges a fix for `-sm tensor` KV cache quantization on multi-GPU setups, addressing a shape information loss issue when flattening tensors.

Llama.cpp is expected to receive a fix for split mode tensor crashes on multi-GPU setups, which currently cause VRAM exhaustion every 90-120 minutes. The fix also reportedly brings a ~35% throughput improvement over layer mode.