A user benchmarks Qwen 35B-A3B (a 35B MoE model) on a 12GB RTX 3060, finding that 12GB VRAM is a practical sweet spot for running the model with 32k context, achieving ~47 t/s generation.
Hardware: RTX 3060 12GB 32GB DDR4-3200 Windows CUDA 13.x Model: Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf The model is a 35B MoE, so `-ncmoe` matters a lot. Lower `-ncmoe` means more MoE blocks stay on GPU. # Main takeaway **12GB VRAM feels like a very practical size for this model.** It lets you keep enough MoE blocks on GPU that plain decoding becomes quite strong, while still leaving room for useful context sizes like 16k/32k. For prompt processing / prefill, I trust the `llama-bench` numbers more than `llama-cli`’s interactive `Prompt:` line, because `llama-bench` gives a cleaner `pp512` measurement. Best plain `llama-bench` result: -ncmoe 18 -t 9 -ctk q8_0 -ctv q8_0 pp512: ~914 t/s tg128: ~46.8 t/s So raw prefill is very fast on this setup. # Best practical coding profile For daily coding, I would use this: llama-cli.exe ^ -m "Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf" ^ -p "..." ^ -n 512 ^ -c 32768 ^ --temp 0 --top-k 1 ^ -ngl 999 -ncmoe 20 ^ -fa on ^ -ctk q8_0 -ctv q8_0 ^ --no-mmap ^ --no-jinja ^ -t 9 ^ --perf Result: Context: 32k Prompt: ~88.9 t/s in llama-cli Generation: ~43.4 t/s VRAM free: ~273 MiB This is a nice balance: large enough context for coding, still fast, and not completely out of VRAM. # Faster 16k profile -c 16384 -ncmoe 19 -ctk q8_0 -ctv q8_0 -t 9 Result: Prompt: ~91.5 t/s in llama-cli Generation: ~44.5 t/s VRAM free: ~37 MiB This is slightly faster, but very close to the VRAM edge. # MoE offload sweep Plain decoding, q4 KV, `-t 11`: -ncmoe 22: tg128 ~41.6 t/s -ncmoe 20: tg128 ~41.7 t/s -ncmoe 19: tg128 ~44.2 t/s -ncmoe 18: tg128 ~45.9 t/s -ncmoe 17: tg128 ~46.6 t/s -ncmoe 16: tg128 ~25.8 t/s <-- cliff / too aggressive So for plain decoding: safe: -ncmoe 18 edge: -ncmoe 17 avoid: -ncmoe 16 # KV cache sweep At `-ncmoe 18`, `-t 11`: q4_0 KV: pp512 ~913 t/s, tg128 ~45.8 t/s q8_0 KV: pp512 ~915 t/s, tg128 ~45.9 t/s q5_0 KV: much slower mixed q8 K + q4/q5 V: much slower So on this GPU, q8 KV is basically free and preferable: -ctk q8_0 -ctv q8_0 # MTP / speculative decoding I also tested MTP with the llama.cpp MTP branch. Best MTP command: llama-cli.exe ^ -m "Qwen3.6-35B-A3B-MTP-IQ4_XS.gguf" ^ --spec-type mtp ^ -p "..." ^ -n 512 ^ --spec-draft-n-max 2 ^ -c 4096 ^ --temp 0 --top-k 1 ^ -ngl 999 -ncmoe 19 ^ -fa on ^ -ctk q4_0 -ctv q4_0 ^ --no-mmap ^ --no-jinja ^ -t 11 ^ --perf Result: Generation: ~47.7 t/s MTP sweep: -ncmoe 24, depth 2: ~43.8 t/s -ncmoe 20, depth 2: ~46.6 t/s -ncmoe 19, depth 2: ~47.7 t/s -ncmoe 18: failed / invalid vector subscript -ncmoe 16: failed / invalid vector subscript Depth 3 was worse: depth 3, -ncmoe 20: ~39.8 t/s So the MTP sweet spot was: --spec-draft-n-max 2 # Conclusion With 12GB VRAM, plain decoding is already very strong: Plain llama-bench: ~914 t/s pp512, ~46.8 t/s tg128 Best MTP observed: ~47.7 t/s generation So MTP only gave about a **2% generation speedup** over well-tuned plain decoding. For coding, I would personally use plain decoding with 32k context: -c 32768 -ncmoe 20 -ctk q8_0 -ctv q8_0 -t 9 The big lesson: for this MoE model, **12GB VRAM is a very practical sweet spot**. It keeps enough experts on GPU that plain decoding becomes fast, q8 KV is usable, and 32k context is realistic.
A user benchmarks three Qwen models (Qwen3.5-27B dense, Qwen3.5-122B-A10B MoE, Qwen3.6-35B-A3B MoE) on 4x RTX 3090 GPUs under real agentic workloads, finding that MoE models consistently underperform the dense 27B at following strict global rules despite speed advantages, with the Qwen3.6-35B leading in generation throughput.
A technique called luce spark allows Qwen 35B-a3B MoE model to run on a 16GB GPU (like RTX 3090) by learning which experts are frequently used and streaming the rest from RAM, achieving ~100 tok/s without VRAM bottleneck.
A user shares optimized settings for running Qwen3.6 27B (Q8_0) on a dual GPU setup (RTX 4090 + RTX 3090) with llama.cpp, achieving 75-100 t/s and 1500 pp with 250k context.
Qwen 3.6 27B runs fast on 16 GB VRAM thanks to 'Pure Quant' technology, achieving 40 tokens/s with MTP and supporting 64k contexts, enabling local AI on consumer GPUs like RTX 4060 Ti.
Detailed benchmarks of Qwen3.6 35B MoE on RTX 5080 16GB show that MTP (Multi-Token Prediction) does not improve inference speed at 128k context due to VRAM constraints; the best configuration is Q4_K_XL without MTP, achieving ~56 tok/s generation at 128k context.