An investigation into why dual RTX 5060 Ti GPUs max out at ~50% utilization when running large LLMs like Qwen 27B reveals that memory bandwidth is the bottleneck and layer-by-layer splitting causes idle time, making it a relay race rather than parallel computation.
So I've been running Qwen 3.6 27B (Q6, ~22GB) across two 5060 Tis for a while now and kept assuming something in my config was off because neither card ever really goes above ~50%. spent way too long last night actually figuring out why and honestly it's kind of a cool rabbit hole. turns out generating literally one token means the GPU has to pull the entire model's weights through memory. not compute, memory. so a 22GB model = 22GB read for a single token, every time. and my cards top out around 448GB/s bandwidth, which caps you at roughly 25-30 tok/s no matter what settings you touch. that part alone explained a lot honestly. but here's the thing that actually made me go "oh" out loud — since the model doesn't fit on one 16GB card, it splits across both. default splitting method does it layer by layer, which means card 1 does its chunk, then hands off to card 2, and card 1 just... sits there. waiting. so you average that out over time and yeah, ~50% is basically the ceiling, not a bug. it's a relay race, not a team lift. apparently there's a row-split mode where both cards chew on the same layer at once instead of taking turns, but that needs constant back-and-forth between the cards, and without NVLink (which these don't have) whether that's actually faster depends completely on your PCIe lanes. gonna have to just benchmark it myself, no universal answer online for this combo. also stumbled on the fact that Google's DiffusionGemma thing generates a whole block of 256 tokens at once instead of one by one, basically to sidestep this exact problem for single-user setups — and they straight up admit in their own release notes that quality takes a hit for it. nothing here is a free lunch apparently, every architecture just picks its poison. anyway if your local rig feels "stuck" at half utilization on a split model, it's probably not you, it's just what happens when two GPUs take turns instead of working together.
Technical post sharing performance stats for running Qwen 27B on 2x RTX 5070 Ti GPUs with vLLM cu129-nightly, achieving up to 94-87 tps decode and 170k GPU KV cache.
A user on a dual RTX 3090 setup discovers that llama.cpp's --split-mode tensor runs prompt processing on CPU (~400 t/s), while switching to --split-mode layer unlocks GPU prompt processing over 1600 t/s, with only a modest drop in token generation speed.
A GitHub repository providing practical configurations and benchmarks for running local LLMs (like Qwen3.6 27B) on dual RTX 5060 Ti 16GB cards using vLLM and llama.cpp.
A user shares their setup using two modded RTX 2080 Ti GPUs with 22GB VRAM each to run Qwen 3.6 27B at 38 tokens/s with llama.cpp, including tips on power limiting, tensor split mode, and KV cache settings.
A user shares power limit testing on a 4x RTX 3090 setup running Qwen3.6-27B with vLLM, finding 220W as the sweet spot for peak efficiency with minimal throughput loss.