Tag
Testing reveals that Intel consumer platforms like Z890 with Arrow Lake CPUs have hardware/firmware limitations that prevent proper PCIe Peer-to-Peer (P2P) communication between multiple GPUs, making them unsuitable for multi-GPU AI workloads despite adequate lane counts.
The author shares six months of measurements on a four-RTX 3090 local setup, revealing that data parallelism often outperforms tensor parallelism for models fitting on fewer cards, with up to 3.4x throughput difference.
An analysis of PCIe transfer performance when running llama.cpp with dual GPUs using pipeline and tensor parallelism.
A user asks whether running dual GPUs in PCIe 5.0 x8/x4 vs x8/x8 significantly impacts LLM inference speeds.
A user discovered that a hidden PCIe 2.0 x4 electrical limitation on a Threadripper workstation board was crippling one of four RTX 3090s, causing poor multi-GPU LLM inference performance. Fixing the slot layout and switching to tensor split mode doubled Mistral 128B throughput from ~11 to ~24.7 tok/s.
A developer documents the extensive hardware and firmware hacking required to run an NVIDIA RTX Pro 6000 Blackwell GPU in a legacy Dell PowerEdge R730 server, achieving 650K context length for local AI inference.
AMD is set to release new slottable PCIe-based Instinct GPUs aimed at the enterprise AI market, offering a potential new hardware option for local LLM deployment.