The boring way to run Deepseek V4 Flash-0731 130-150 tks - 16x5060ti 16GB over 2 PLX88096 switches

Reddit r/LocalLLaMA News

Summary

The article details a validated hardware configuration using 16 RTX 5060 Ti GPUs with PLX switches to run the Deepseek V4 Flash model, achieving specific performance metrics for context handling and throughput.

Component Validated configuration Motherboard ASRock Rack SPC621D8U-2T/OVH CPU Xeon Gold 6330 (Get gold/platinum if interested in Optane Pmem gimmicks) GPU fabric Two Broadcom/PLX PEX88096 islands, eight GPUs per island GPUs 16 x RTX 5060 Ti 16 GB OS Ubuntu 22.04.5 LTS Kernel 6.8.0-106-generic NVIDIA driver Aikitoria patched open driver 610.43.02-p2p Required BAR1 16,384 MiB on every GPU UEFI boot enabled; CSM disabled. Secure Boot disabled. The locally built EFI application and patched NVIDIA modules are unsigned. Above 4G Decoding enabled. MMIO High Granularity set to 1024G. MMIO High Base set around 56T. SR-IOV disabled on this machine. intel_iommu=off pci=realloc=on,hpmmioprefsize=512G in GRUB; NVreg_EnableResizableBar=1 for the NVIDIA module; Sets size code 14 → 16 GiB BAR1 on each of the 16 GPUs Temporarily disables PCI memory decoding and clears the old BAR1 address so Linux can reallocate it. PLX switch ACS control register: For every PLX/PEX bridge, writes: ECAP_ACS+0x6.w = 0000 After that, a little vibe coding to make custom all-reduce work within each PLX cluster and make DSpark work for pipeline parallel. For tensor parallel 8, pipeline parallel 2: 500k context available. Around 4000 pp up to 500k context, tg 100-150 (Averaging 140 in DeepSeek Harness) For tensor parallel 4, pipeline parallel 4: Full 1M context available. Around 7000 pp up to 500k context, tg 80 Paid 0.6 x RTX6000 Pro for the whole setup.
Original Article

Similar Articles

Deepseek V4 Flash running on RTX 5090 MoE

Reddit r/LocalLLaMA

User shares optimization benchmarks for DeepSeek-V4-Flash (Q2_K) running on an RTX 5090 using a fork of llama.cpp, achieving 21.3 tokens/s generation and 1 million context size.