Detailed findings on PCIe bifurcation and P2P performance issues with 4x GPU setups, including workarounds and alternatives for tensor and pipeline parallelism.
I dumped the last week deep diving this and I’m I’ve been using Linux for 14 years and am a cloud systems engineer with a focus on supported Linux infrastructure for a private cloud provider. Essentially, if you are using a single 4x4 bifurcation pcie x16 card inserted into your x16 slot on your mobo and you have 4x gpus connected to it. Regardless of pcie generation that card that does the bifurcation is the choke point for p2p communication. It acts as the pcie bridge that connects the gpus and with TP=4 the bandwidth of that fabric that connects the 4 cards on that pci E bridge will become saturated and yield worse performance than with p2p off. The ways to deal with this would be to either: Don’t run p2p. It’s only a 10 to 15% gain and may not justify the cost and effort of having a setup where p2p gets you that 10% performance. Pick up a Chinese slimsas bifurcation bridge. Supposedly you might not encounter it with those. They run between 150 to 250 Buy a 1200 gen 4 pcie bridge from Cpayne. These devices are specifically made for this use case. But 1200 expense for 10% performance gain probably isn’t worth it Don’t use tensor parallelism. Use pipeline parallelism. The downside with this is pipeline parallelism in my benchmarks yielded worse performance at low concurrency than TP=4 + P2P off. PP=4 only yields better performance if you have significant enough concurrency where all the gpus have something they can be working on where none of them are waiting on another GPU to finish their work There are used PLX switches on eBay. But with these you run a risk of them not supporting a multi GPU setup with P2P due to firmware restrictions that limit non storage devices being used with them. Have a motherboard and cpu combo that provides a dedicated x16 lanes to both the primary and secondary x16 slot. You could have both of these with 8i bifurcation with 2 gpus on each. But if that setup requires a retimer to get gen4 or gen 5 then you are talking 130+ for each of these two retimer bifurcation cards. If there is a solution to this that I didn’t list, please let me know and I’ll update this post.
A user discovered that a hidden PCIe 2.0 x4 electrical limitation on a Threadripper workstation board was crippling one of four RTX 3090s, causing poor multi-GPU LLM inference performance. Fixing the slot layout and switching to tensor split mode doubled Mistral 128B throughput from ~11 to ~24.7 tok/s.
A user shares a configuration of 4x RTX 5060 Ti 16GB with P2P to run Qwen3.6-27B-FP8 at 55 tok/s with 262K context, highlighting the low cost of about $1800 for single-user inference.
A benchmark analysis of Qwen 3.6 27B MTP on 4x RTX 3090 GPUs, demonstrating that using NVLink for tensor parallelism yields significant throughput improvements (up to +53%) over PCIe configurations.
A grad student shares their experience building a multi-GPU workstation with 4x3090 Ti running on a single US wall outlet, detailing constraints, power-limiting challenges, and component choices.
A user shares power limit testing on a 4x RTX 3090 setup running Qwen3.6-27B with vLLM, finding 220W as the sweet spot for peak efficiency with minimal throughput loss.