PSA：请勿使用 Intel 消费级平台进行多 GPU 配置

由于越来越多的人尝试搭建自己的多GPU机器，我觉得应该帮助大家避免一个常见错误——在搭建多GPU系统时使用Intel消费级平台（如Z890）。尽管CPU提供了24条PCIe 5.0通道，高端主板上的两个PCIe x16插槽可以分叉为8x8x，但这对于需要GPU间P2P的AI推理/训练负载来说完全无用。在我的测试中，我使用了Intel Core Ultra 7 270K Plus搭配华硕Z890 Apex主板（BIOS版本3202，这通常是我的超频测试平台）。我最初想看看，与基于Epyc的服务器相比，单核性能更强、更偏向游戏优化的CPU是否能提升GPU推理性能——因为我正在尝试优化推理服务中现有GPU的吞吐量，毕竟所有东西都涨价了。不幸的是，如果想要桌面Intel CPU更强的单核性能，我可能需要使用PCIe交换板来连接GPU，尤其是在Intel平台上，主PCIe插槽本身就被人为限制为只能8x4x4x分叉。我发现Arrow Lake CPU的PCIe根联合体似乎存在某种硬件/固件限制，阻止PCIe P2P正常工作。正如这个issue所示，它甚至无法正确允许PCIe设备间的数据包传输：https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1253 它不仅无法正常工作，即使能工作，带宽也会减半: [P2P (Peer-to-Peer) GPU Bandwidth Latency Test] Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0 Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0 Device=0 CAN Access Peer Device=1 Device=1 CAN Access Peer Device=0 ***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure. So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases. P2P Connectivity Matrix D\D 0 1 0 1 1 1 1 1 Unidirectional P2P=Disabled Bandwidth Matrix (GB/s) D\D 0 1 0 674.36 11.89 1 11.90 677.65 Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s) D\D 0 1 0 619.30 5.48 1 5.48 678.46 Bidirectional P2P=Disabled Bandwidth Matrix (GB/s) D\D 0 1 0 649.01 16.60 1 16.74 680.68 Bidirectional P2P=Enabled Bandwidth Matrix (GB/s) D\D 0 1 0 649.28 10.96 1 10.97 680.37 P2P=Disabled Latency Matrix (us) GPU 0 1 0 1.59 17.11 1 18.39 1.61 CPU 0 1 0 1.29 4.40 1 4.39 1.24 P2P=Enabled Latency (P2P Writes) Matrix (us) GPU 0 1 0 1.59 1.02 1 1.03 1.60 CPU 0 1 0 1.27 1.09 1 1.12 1.27 实际上，这个P2P结果只有在我安装了来自 github.com/aikitoria/open-gpu-kernel-modules 的修补版P2P开放内核驱动后才可能实现——尽管我使用的RTX A6000默认应该支持PCIe P2P（使用NVIDIA官方驱动）。我发现NVIDIA似乎在Intel消费级平台上故意屏蔽了PCIe P2P，原因正是这些兼容性问题。因此你需要使用修补版内核来启用它。 起初我觉得带宽减半也勉强能接受，毕竟延迟大幅降低了。但如果我用强制启用PCIe P2P的修改版驱动运行VLLM（跨2块GPU使用张量并行），模型的输出就会变成一堆乱码"!!!!!!!!!!!!"。 { "id": "chatcmpl-96088a9dde7122ae", "object": "chat.completion", "created": 1784949145, "model": "Qwen3.5-27B-Derestricted", "choices": [ { "index": 0, "message": { "role": "assistant", "content": null, "refusal": null, "annotations": null, "audio": null, "function_call": null, "reasoning": "!!!!!!!!!!!!!!!!!!!!" }, "logprobs": null, "finish_reason": "length", "stop_reason": null, "token_ids": null, "routed_experts": null } ], "service_tier": null, "system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-8736f542", "usage": { "prompt_tokens": 32, "total_tokens": 52, "completion_tokens": 20, "prompt_tokens_details": null }, "prompt_logprobs": null, "prompt_token_ids": null, "prompt_text": null, "kv_transfer_params": null, "metrics": null } 这不是GPU不支持REBAR之类的问题，因为如果检查lspci -v，GPU显示的BAR大小为64G，理论上这就满足PCIe P2P的需求了。BIOS中REBAR已启用，IOMMU已禁用，grub配置如下： GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt" GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt" 02:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GA102GL [RTX A6000] Flags: bus master, fast devsel, latency 0, IRQ 219 Memory at 8f000000 (32-bit, non-prefetchable) [size=16M] Memory at c000000000 (64-bit, prefetchable) [size=64G] Memory at d000000000 (64-bit, prefetchable) [size=32M] I/O ports at a000 [size=128] Expansion ROM at 90000000 [virtual] [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 02:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1) Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 17 Memory at 90080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 03:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GA102GL [RTX A6000] Flags: bus master, fast devsel, latency 0, IRQ 222 Memory at 8d000000 (32-bit, non-prefetchable) [size=16M] Memory at a000000000 (64-bit, prefetchable) [size=64G] Memory at b000000000 (64-bit, prefetchable) [size=32M] I/O ports at 9000 [size=128] Expansion ROM at 8e000000 [virtual] [disabled] [size=512K] Capabilities: <access denied> Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia 03:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1) Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 18 Memory at 8e080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel nvidia-smi输出似乎也表明PCIe P2P应该是可能的： GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PHB 0-23 0 N/A GPU1 PHB X 0-23 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks 如果我使用屏蔽Intel消费级平台上PCIe P2P的官方NVIDIA驱动，可以看到RTX A6000（本应支持P2P）的PCIe P2P现在被禁用了： [P2P (Peer-to-Peer) GPU Bandwidth Latency Test] Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0 Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0 Device=0 CANNOT Access Peer Device=1 Device=1 CANNOT Access Peer Device=0 ***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure. So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases. P2P Connectivity Matrix D