enabling PCI-E p2p for consumer Nvidia cards will yield you more than you think

Reddit r/LocalLLaMA News

Summary

Enabling PCI-E peer-to-peer (P2P) for consumer Nvidia GPUs with patched drivers and vLLM environment variables yields roughly 25% prefill throughput improvement for free, as demonstrated by benchmarks.

Disclaimer - no LLM was used to write this post/note As larger post about my setup will come later, want to give heads-up to folks who use VLLM and >= 2 GPUs. So I have pretty meaty server (8 channel AMD EPYC, ~150GB/s RAM bw) and 4x5060Ti 16GB in PCI-E 4.0 8x mode. Considering CPU is meaty, high RAM bw, no bottlenecks, messing with p2p should be minuscule, right? That's what I thought as well, now see results of two identical tests: Test prompt: llama-benchy --base-url http://localllm/v1 --model localllm --depth 0 4096 8192 16384 32768 --latency-mode generation 1) no p2p: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------|----------------:|----------------:|---------------:|-----------------:|-----------------:|-----------------:| | localllm | pp2048 | 1648.96 ± 8.49 | | 1241.77 ± 8.66 | 1131.60 ± 8.66 | 1241.77 ± 8.66 | | localllm | tg32 | 89.86 ± 7.05 | 92.76 ± 7.28 | | | | | localllm | pp2048 @ d4096 | 1654.51 ± 6.44 | | 3504.72 ± 11.71 | 3394.55 ± 11.71 | 3504.72 ± 11.71 | | localllm | tg32 @ d4096 | 102.98 ± 3.95 | 106.30 ± 4.08 | | | | | localllm | pp2048 @ d8192 | 1631.74 ± 10.72 | | 5821.87 ± 82.46 | 5711.71 ± 82.46 | 5821.87 ± 82.46 | | localllm | tg32 @ d8192 | 109.27 ± 4.42 | 112.79 ± 4.56 | | | | | localllm | pp2048 @ d16384 | 1601.33 ± 3.18 | | 10502.40 ± 46.13 | 10392.23 ± 46.13 | 10502.94 ± 46.27 | | localllm | tg32 @ d16384 | 98.61 ± 5.01 | 116.87 ± 5.94 | | | | | localllm | pp2048 @ d32768 | 1544.66 ± 1.10 | | 20598.44 ± 65.95 | 20488.27 ± 65.95 | 20598.44 ± 65.95 | | localllm | tg32 @ d32768 | 93.24 ± 15.89 | 122.15 ± 15.63 | | 2) p2p enabled: | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:---------|----------------:|----------------:|---------------:|-----------------:|-----------------:|-----------------:| | localllm | pp2048 | 2305.20 ± 49.73 | | 913.50 ± 27.47 | 808.72 ± 27.47 | 913.50 ± 27.47 | | localllm | tg32 | 97.92 ± 8.26 | 101.07 ± 8.53 | | | | | localllm | pp2048 @ d4096 | 2268.40 ± 6.48 | | 2615.57 ± 28.00 | 2510.79 ± 28.00 | 2615.57 ± 28.00 | | localllm | tg32 @ d4096 | 103.60 ± 9.04 | 106.95 ± 9.33 | | | | | localllm | pp2048 @ d8192 | 2214.54 ± 8.39 | | 4307.55 ± 49.46 | 4202.77 ± 49.46 | 4312.27 ± 55.30 | | localllm | tg32 @ d8192 | 122.88 ± 5.89 | 126.84 ± 6.09 | | | | | localllm | pp2048 @ d16384 | 2164.10 ± 8.34 | | 7809.37 ± 73.29 | 7704.59 ± 73.29 | 7809.37 ± 73.29 | | localllm | tg32 @ d16384 | 111.51 ± 7.92 | 125.93 ± 3.14 | | | | | localllm | pp2048 @ d32768 | 2082.02 ± 3.86 | | 15224.08 ± 21.55 | 15119.30 ± 21.55 | 15224.08 ± 21.55 | | localllm | tg32 @ d32768 | 96.02 ± 7.55 | 136.62 ± 24.75 | | | | Welp, ~25% of PP for free. HF model used: Qwen/Qwen3.6-27B-FP8 KV quant - not used, so F16 Parallelism mode - tensor-parallelism How to enable P2P Your hardware should support ReBAR, enable it in bios. Install patched drivers (read readme carefully!) from https://github.com/aikitoria/open-gpu-kernel-modules Add following environment variables to VLLM startup: NCCL_P2P_DISABLE=0 VLLM_SKIP_P2P_CHECK=1 NCCL_P2P_LEVEL=SYS That's it!
Original Article

Similar Articles

PSA: DO NOT use Intel consumer platforms for multi-GPU setups

Reddit r/LocalLLaMA

Testing reveals that Intel consumer platforms like Z890 with Arrow Lake CPUs have hardware/firmware limitations that prevent proper PCIe Peer-to-Peer (P2P) communication between multiple GPUs, making them unsuitable for multi-GPU AI workloads despite adequate lane counts.

Blackwell and PDL performance increase

Reddit r/LocalLLaMA

Llama.cpp now supports Nvidia's Programmatic Dependent Launch (PDL) for Blackwell GPUs, offering a 5-10% performance boost on token generation. The feature is not enabled by default and requires a build flag.

Ultra budget 20GB vram with 448GB/s for $100 bucks.

Reddit r/LocalLLaMA

Demonstrates achieving 20GB VRAM and 448GB/s bandwidth for around $100 using two NVIDIA P102-100 cards, running a llama.cpp server with a Qwen model and supporting 3 concurrent users with large context.