Dailychained PLX 88096 switches, Quad RTX 5070 Ti + Quad RTX 5060 Ti

Left: Quad RTX 5060 Ti, Middle: Quad RTX 5070 Ti, both PLX 88096 switch, Right = rehomed host Edit: Benchmarsk were 1 line = fixed WHY: -->> DATA SOVEREIGNITY / PRIVACY<<-- hey, this is (localllama right?), this makes no less sense than my dropping the same $$$ on a motorbike I want but don't need so no Triumph Rocket III motorbike for me boo hoo, is for SOHO Anyway, a bit of a journey, a few hundreds of $$ wasted on power adaptors / pci risers that are not suitable and a small fortune in RTX 50xx GPU that will be obsolete eventually Now I'm still buried in the steep learn to use linux / docker / vllm / models / setup clients learning curve (I am windows since Win 3.1). I am yet to learn to relove the CLI (not since ICL/IBM MFs in the 80's) All setup and running to the point vLLM NCCL messages report P2P enabled within each node, (yet to resolve getting P2P across nodes). A little more work on the cooling (more fans coming)/ best orientation etc to do I expect to have these for a while, hence no loose mining frames etc each of these nodes is self contained built up hardware (prototype quality, a few rough edges here and there), If I can get the GPUs just build another one (I have spare V21 case + 88096 PCB) Note I am in NZ so all I can buy locally is regular basic PC parts, pretty much everything else is overseas import eg even the Thermalrake Core V21 cases had to come from Australia, most everything else is from China 2-4 weeks shipping, if a cable or adapter doesn't work then more delays and i pay sales tax 15% at the border Also these cases party trick is they can be bolted vertically so assuming I can manage rising heat that option saves a bit of space on the desk GPUs are mixed brands/models, a couple I already had, I was incrementally ( every local seller is '1 GPU per customer') collecting 8 x RTX 5060 Ti initially for this build, but at the point I got the the 5th one the price delta between 5060 Ti 16GB and 5070 Ti 16GB got close enough I returned that 5th one, added 3 x RTX 5070 Ti 16GB to one I had already. Note there really is no cost effective used GPU market here so I was only able to buy 1 5060 ti used and I only paid him NZ$150 over what he paid for it in Nov 2025 ;) (Nice for him but even at that markup it was still a score) Looking forward to getting 3.8 Flash Next working too fingers crossed is usable Benchmarks Run command per node below Ubuntu 24.04, NVidia 575.something, CUDA 13.2 vllm 0.30 - so you can see what is enabled (you are correct and thank you for noticing, yes I really do not know what I am doing on the software side, I am just a very old script kiddy) no spec decode etc to keep it reproducable docker run --rm -it \ --name vllm-node1x4 \ --ipc=host \ --gpus '"device=GPU-dd49c72c-4273-9016-aaad-8883c553b0da,GPU-1ef97316-f1c1-3c15-64db-fbd6373179e5,GPU-ab8220cb-5ab1-82a6-8aec-26454657e215,GPU-6ce2d880-240f-18be-ffe0-96ab3d0eede8"' \ -e NCCL_DEBUG=INFO \ -e NCCL_P2P_DISABLE=0 \ -e NCCL_P2P_LEVEL=SYS \ -e VLLM_SKIP_P2P_CHECK=1 \ -e NCCL_BUFFSIZE=16777216 \ -e NCCL_MIN_NCHANNELS=8 \ -v /mnt/ai-assets/huggingface:/root/.cache/huggingface \ -v /mnt/ai-assets/vllm-cache:/root/.cache/vllm \ -v /mnt/ai-assets/models:/models:ro \ -p 8005:8005 \ vllm/vllm-openai:latest \ /models/Qwen/Qwen3.8-27B-FP8 \ --served-model-name qwen3.8-27b-fp8 \ --quantization fp8 \ --tensor-parallel-size 4 \ --max-model-len 65536 \ --max-num-seqs 10 \ --gpu-memory-utilization 0.85 \ --kv-cache-dtype auto \ --host 0.0.0.0 --port 8005 Tests command llama-benchy \ --base-url http://localhost:8005/v1 \ --model qwen3.8-27b-fp8 \ --tokenizer /models/Qwen/Qwen3.8-27B-FP8 \ --depth 0 4096 8192 16384 32768 \ --latency-mode generation Results - No overclock/undervolt etc stock GPU settings (Todo: NVOC overclock VRAM) - look very linear to me, basically 2 to 1 - results seem ok Quad 5070 Ti 16GB | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:----------------|----------------:|---------------:|-------------:|---------------:|---------------:|----------------:| | qwen3.8-27b-fp8 | pp2048 | 6488.44 ± 3.99 | | 363.10 ± 0.19 | 315.79 ± 0.19 | 363.10 ± 0.19 | | qwen3.8-27b-fp8 | tg32 | 82.21 ± 0.06 | 84.86 ± 0.06 | | | | | qwen3.8-27b-fp8 | pp2048 @ d4096 | 5831.49 ± 4.69 | | 1100.96 ± 0.70 | 1053.65 ± 0.70 | 1100.96 ± 0.70 | | qwen3.8-27b-fp8 | tg32 @ d4096 | 81.56 ± 0.09 | 84.19 ± 0.09 | | | | | qwen3.8-27b-fp8 | pp2048 @ d8192 | 5642.19 ± 3.35 | | 1862.27 ± 1.10 | 1814.96 ± 1.10 | 1862.27 ± 1.10 | | qwen3.8-27b-fp8 | tg32 @ d8192 | 81.00 ± 0.01 | 83.61 ± 0.01 | | | | | qwen3.8-27b-fp8 | pp2048 @ d16384 | 5431.23 ± 5.45 | | 3441.21 ± 3.25 | 3393.89 ± 3.25 | 3442.26 ± 3.26 | | qwen3.8-27b-fp8 | tg32 @ d16384 | 80.63 ± 0.14 | 83.23 ± 0.14 | | | | | qwen3.8-27b-fp8 | pp2048 @ d32768 | 5059.44 ± 0.49 | | 6928.84 ± 0.58 | 6881.52 ± 0.58 | 6929.81 ± 1.32 | | qwen3.8-27b-fp8 | tg32 @ d32768 | 79.43 ± 0.26 | 81.99 ± 0.27 | | | | Quad RTX 5060 Ti 16GB | model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) | |:----------------|----------------:|---------------:|-------------:|----------------:|----------------:|----------------:| | qwen3.8-27b-fp8 | pp2048 | 3218.53 ± 1.05 | | 689.86 ± 0.35 | 636.52 ± 0.35 | 689.86 ± 0.35 | | qwen3.8-27b-fp8 | tg32 | 43.35 ± 0.02 | 44.74 ± 0.02 | | | | | qwen3.8-27b-fp8 | pp2048 @ d4096 | 3005.17 ± 0.55 | | 2097.92 ± 0.37 | 2044.59 ± 0.37 | 2097.92 ± 0.37 | | qwen3.8-27b-fp8 | tg32 @ d4096 | 42.85 ± 0.01 | 44.23 ± 0.01 | | | | | qwen3.8-27b-fp8 | pp2048 @ d8192 | 2927.62 ± 1.87 | | 3551.28 ± 2.59 | 3497.95 ± 2.59 | 3551.28 ± 2.59 | | qwen3.8-27b-fp8 | tg32 @ d8192 | 42.66 ± 0.06 | 44.03 ± 0.06 | | | | | qwen3.8-27b-fp8 | pp2048 @ d16384 | 2821.32 ± 1.44 | | 6586.68 ± 3.60 | 6533.34 ± 3.60 | 6587.62 ± 3.66 | | qwen3.8-27b-fp8 | tg32 @ d16384 | 42.25 ± 0.05 | 43.61 ± 0.05 | | | | | qwen3.8-27b-fp8 | pp2048 @ d32768 | 2654.29 ± 0.46 | | 13170.73 ± 2.42 | 13117.40 ± 2.42 | 13171.73 ± 2.56 | | qwen3.8-27b-fp8 | tg32 @ d32768 | 41.30 ± 0.07 | 42.63 ± 0.07 | | | | --- My plan more or less from a while back, with hardware notes pretty much up to date My justification to target 128GB/ All Blackwell: - 128GB = DGX Spark, RTX Spark And Strix 128GB AIOs so will be relevant for a couple of years - All Blackwell = FP8 fast now NVFP4 = faster once mature/production ready? (late 2026?) - Hopefully significantly faster than say DGX Spark - Need 192GB VRAM?: -- Build another Node etc (assuming RTX GPUs still relevant to AI inference, one day used will be < $$) - if not, easier to sell 1 x Node (or worse case 4 x GPUs individually per Node) than 1 x nonolithic DGX Spark or whatever once future wonder AI execution chips exist My justification to target PEX 88096 Backplanes - GPU P2P within each node with patched Nvidia drivers on Linux - Maximise capabilities of (relative to Node1) constrained RTX 5060 Ti 16GB PCIe x8 - Backends e.g. vLLM with say TP=4, PP=2 hopefully maximize architecture - PCIe4 = less bandwidth BUT: -- SO VERY much more forgiving re interference -- MUCH less $$ than anything PCIe5 -- NVidia p2pbandwidthlatencytest shows < 1us latency GPU P2P within each switch Each Node - Modular/ self contained, just chuck a spare SFF-8654 PCI host card into any PC and go AI LLM Inference Tiers - <= 64GB -- Performance / production tier: Node1 (GPU 0-3): vLLM TP=4 = FAST -- Experimentation / second model tier: Node2 (GPU 0-3): vLLM TP=4: Fast enough?? - > 64GB and <= 128GB = Node1 + Node2: Capacity tier: -- Node1 (GPU 0-3) + Node2 (GPU 0-3): vLLM TP=4 PP=2: Constrained to at best Node2 speed, good enough? - Other: -- Host RTX 5080: Embedding eg Qwen3-VL-Embedding-8B watever -- Node2 GPU4 RTX 3080: STT/TTS whatever Host GPU RTX 5080 - Use standalone for utility eg Embedding / Vision / Spec Decoding etc Host (Host 128GB DDR5-6000) - Ryzen 5 9600X - MSI MPG B850 Edge TI WiFi - Jonsbo D41 Mesh Black - XPG Core Reactor II VE 850W - iGPU only to Monitor - SATA SSD for each of WIN / Linux OS - Gen5 M2 SSD on PCIe5 x4 (CPU): -- 2TB = Docker -- 4TB = AI Assets HOT - SATA 2 x 28TB Barracuda HDD Mirrored (Linux) -- AI Assets COLD - RTX 5080 16GB in PCI_E3 (PCIe4 x4 Chipset) Node1 Performance node - 64GB VRAM (TP=4 Parallel) - Chassis: ThermalTake Core V21 - PSU: MSI MEG Ai1600T - PLX/PEX 88096 PCI 4 slot switch (with downstream SFF-8654 ports) -- Slot 1/4: RTX 5070 Ti -- Slot 2/4: RTX 5070 Ti -- Slot 3/4: RTX 5070 Ti -- Slot 4/4: RTX 5070 Ti -- All GPU PCIe4 x16 within PEX 88096 Node2 Capacity / Secondary node - 64GB VRAM Secondary node (TP=4) - 16GB VRAM Utility (RTX 3080) - Chassis: ThermalTake Core V21 - PSU: DeepCool PN1200M - PLX/PEX 88096 PCI 5 slot switch = Tensor Parallel 64GB VRAM (Secondary/capacity node) -- Slot 1/5: RTX 5060 Ti 16GB -- Slot 2/5: RTX 5060 Ti 16GB -- Slot 3/5: RTX 5060 Ti 16GB -- Slot 4/5: RTX 5060 Ti 16GB -- Slot 5/5: Alienware RTX 3080 OEM 10GB (I have a 5th slot and a spare 3080 so..) -- 4 x RTX 5060 Ti 16GB GPU PCIe4 x8 (Due to 5060 x8 electrically) within PEX 88096 HOST <-- SlimSAS PCIe4 x16 --> Node1 <-- SlimSAS PCIe4 x8 --> Node2 Hardware porn before transitioing to the PCI switch approach I started this build Nov 2025 slowly sourcing the parts for a new standalone PC meant as my triple 4k gaming + AI experimentation rig, then I found I wasnt gaming and I kept adding GPUs (well, VRAM really) V1: RTX 5080 + RTX 5070 Ti 16GB (early build photo, missing a few other parts) V2: RTX 5090 + RTX 5070 Ti V2: RTX 5090 + RTX 5080 + RTX 5070 Ti 16GB Then I picked up a 5060 Ti 16GB and was thinking how the hell do I squeeze this in, looked at M2 to what ever adaptors etc etc, yes my MB has bifurication etc etc, ordered a couple then thought nah thats all getting pretty manky, hence the pivot to the current approach, more or less homogenous nodes re generation / vram, plug any node into any PC with spare PCI slot, easy to move and so on Random POC / mid build photos 5 way 88096 PCB will it Post? = YES Host to 4 slot switch daisychain to 5 way switch, will they post/can I see GPUs? = YES - 4 GPU in let downstream green (5 way switch) 2 in upstream black 4 way switch at which point I ran out of room / power cables / risers / bits of wood, but hey, I could see all bits in linux in a massive pci tree Host to 4 slot switch daisychain to 5 way switch, will they post/can I see GPUs? = YES How to mount GPU array in these cases? Was looking for a blade asthetic RTX 5060 pretty easy 5070 a little tighter thats a lot of transistors PCB adapter plates in progress this one got moved about 4 times due to cable constraints etc Work with the fragile risers, dont fight the bends they came with Now you get the appraoch 4 x RTX 5060 Ti + 1 x RTX 3080 10GB Part populated just 2 gpus in each, 88096 80mm cooling fans installed, running ok for some early work patching for P2P etc etc 88096 80mm cooling fans installed (Middle) Quad 5060 Ti populated and running, left still waiting parts Now.. should I sell the 5090 that is now in my older rescurected gaming PC? ( 5600x PCIe4 32Gb DDR4). Probably yes... Note: For a forum dedicated to AI there sure seems a 'wierd 'I hate AI managed posts/slop' herein, so you guys relax I personally fingered every word above (except for some of the vLLM command env vars/switches) Laters