A user details the process of running the Qwen 3.8 Next model on a multi-GPU V100 setup, sharing troubleshooting experiences, performance benchmarks, and thermal test results.
Hey everyone, After wrestling with hardware and engine issues for days, I finally got Qwen 3.8 Next running properly on my multi-GPU rig. Thought I’d share the setup journey, benchmarks, and thermal results for anyone trying something similar. Seeing all the ongoing memes on Reddit about multi-GPU setups turning into absolute space heaters and catching fire, I decided to run some rigorous thermal tests to see for myself. he Troubleshooting Odyssey PCIe Link Speed Issue: Right after installation, one of the cards dropped to PCIe Gen 1 x16. Spent about 8 hours over two days diagnosing and fixing it. Finding the Right Engine: Started with sglang-v100, but kept hitting continuous OOM crashes. Someone on Reddit previously suggested the pxa engine, but that threw errors as well. Eventually tried 1cat-vllm, spent some time tweaking it, and finally hit a stable run! Configuration: Running with TP2 PP3. Currently, speculative decoding is limited to speculative=1. Setting it to 2 throws an OOM due to memory constraints (might look into optimizing this later, but for now, it works). Context & Memory Stats Plaintext INFO: Available KV cache memory: 8.78 GiB INFO: GPU KV cache size: 531,288 tokens INFO: Maximum concurrency for 262,144 tokens per request: 2.03x Performance Benchmarks 1. Prompt Processing (Prefill) Input Length (Tokens) Speed (tok/s) 1,024 (1K) 1,389 2,048 (2K) 2,536 4,096 (4K) 3,210 8,192 (8K) 4,336 16,384 (16K) 4,679 32,768 (32K) 4,470 65,536 (64K) 3,820 131,072 (131K) 2,759 2. Text Generation (MTP Comparison) Output Length (Tokens) Base Speed (No MTP, tok/s) Optimized Speed (MTP Enabled, tok/s) 128 22.23 41.34 256 22.32 42.48 512 22.70 43.04 1024 22.86 43.28 2048 22.83 43.38 Average 22.59 42.70 MTP nearly doubles generation throughput across the board. Thermals & Acoustics People often meme about multi-GPU rigs turning into space heaters or jet engines, so I ran a thorough thermal/stress test: Stress Test: Ran gpu-burn continuously for 20 minutes. Thermal Equilibrium: Temperatures peaked at 65°C and stabilized right around 64°C. Fan Curve: Based on my fan control script, the fans were only running at around 76% at 64°C. The cards stay well under 65°C without even needing full blast. Pretty happy with how stable, cool, and quiet this system turned out.
A user successfully runs the Qwen 3.8 27B AI model on a mixed setup of RTX 3060 and 5060 Ti GPUs using tensor parallelism with exllamav3, achieving around 50 tokens per second with MTP enabled.
A Reddit user shares how they successfully ran the Qwen 3.8 Next MoE model on a system with 16GB VRAM and 32GB RAM using aggressive quantization and specific llama.cpp settings, achieving usable performance for large models on limited hardware.
A user shares kernel optimizations for Tesla P100 GPUs to improve performance when running the Qwen 3.8 model with llama.cpp, achieving significant speedups in inference.
Demonstrates running a custom Qwen model (Qwen3.6-35B-A3B-Claude-4.7-Opus-abliterated-MTP-GGUF) on dual Nvidia RTX PRO 6000 Blackwell GPUs at 195 tokens per second using Hugging Face Inference.