An experiment running the Qwen3.8-2.4T-A95B MoE model locally on dual consumer GPUs (RTX 5090 + 5060 Ti) with llama.cpp, achieving ~0.8 tok/s with MTP speculative decoding enabled.
I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not. I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397 GiB. The model uses 512 routed experts, with 10 active per token. My hardware: CPU: AMD Ryzen 9 9950X3D 16 cores / 32 threads GPU 0: NVIDIA RTX 5090 32 GB VRAM GPU 1: NVIDIA RTX 5060Ti 16 GB VRAM RAM: 128 GB DDR5 5600 CL36 Swap: 350 GB OS: Arch Linux llama.cpp: Unsloth build 10360 The interesting part is that I also managed to get the model’s native MTP speculative decoding working. After testing several configurations, the best settings I found were: --tensor-split 4,1 --n-cpu-moe 91 MTP n_max = 3 MTP p_min = 0.5 MTP routed experts from block 92 forced to CPU The block 92 expert tensors had to stay on CPU because otherwise MTP would run out of VRAM. Performance For a controlled 32-token generation test: Prompt tokens: 16 Generated tokens: 32 Prompt speed: 0.8299 tok/s Generation speed: 0.8033 tok/s Decode time: 39.84 s Total wall time: 59.14 s MTP draft tokens: 21 MTP accepted: 19 MTP acceptance: 90.48% Without MTP, the same test gave: Generation speed: 0.7750 tok/s Decode time: 41.29 s Total wall time: 60.81 s So my final MTP configuration gives roughly: +3.64% generation throughput -2.74% total wall time Not a huge speedup, but considering this is a 2.45T parameter model running locally on consumer hardware, I’ll take it. I also tested different MTP settings: n_max=2 -> too conservative n_max=3 -> best n_max=4 -> much slower p_min=0.0 -> too many bad speculative tokens p_min=0.5 -> best result p_min=0.7 -> too aggressive Interestingly, p_min=0.7 reached 100% draft acceptance, but it generated too few useful speculative tokens and ended up slower than p_min=0.5. Final llama-server command MODEL="/home/marcin/Pobrane/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf" CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 LLAMA_ATTN_ROT_DISABLE=1 \ /home/marcin/Pobrane/unsloth-qwen38-runtime/llama.cpp/build/bin/llama-server \ --model "$MODEL" \ --alias Qwen3.8-2.4T-A95B \ --device CUDA0,CUDA1 \ --main-gpu 0 \ --split-mode layer \ --tensor-split 4,1 \ --n-gpu-layers all \ --n-cpu-moe 91 \ -ot 'blk\.92\.ffn_(down|gate|up)_exps\.weight=CPU' \ --fit on \ --fit-target 2048,1536 \ --no-op-offload \ --ctx-size 512 \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --kv-unified \ --batch-size 128 \ --ubatch-size 32 \ --threads 16 \ --threads-batch 16 \ --parallel 1 \ --spec-type draft-mtp \ --spec-draft-device CUDA0 \ --spec-draft-ngl all \ --spec-draft-cpu-moe \ --spec-draft-n-max 3 \ --spec-draft-p-min 0.5 \ --temp 0 \ --top-k 0 \ --top-p 0.95 \ --min-p 0.0 \ --cache-ram 0 \ --ctx-checkpoints 0 \ --load-mode mmap \ --no-warmup \ --no-ui \ --host 127.0.0.1 \ --port 8200 \ --metrics With MTP enabled, VRAM usage after generation is roughly: RTX 5090: 29.6 GB used / ~2.6 GB free RTX 5060 Ti: 12.3 GB used / ~3.6 GB free One important info: 0.80 tok/s is from a controlled 32-token test, not a large multi-prompt benchmark, so I wouldn’t present it as a universal average for every workload. Longer prompts, context size, output patterns and expert routing can change the result. Still, seeing a 2.4T MoE model actually generating locally on a 5090 + 5060 Ti at around 0.8 tok/s is pretty wild.
Detailed breakdown of running Qwen 3.5 122B MoE on a single RTX 3090 at 35 t/s using a custom llama.cpp fork (ik_llama.cpp) with fused MoE operations and expert offloading to CPU RAM, significantly outperforming stock llama.cpp MTP.
A developer showcases a custom `--moe-direct-io` feature for llama.cpp that achieves 10–15x speedup (20–24 tok/s) running the 125B MoE Qwen3.5-Flash Next (IQ2_XS, 68GB) on just an RTX 3060 12GB with 16GB RAM, while keeping output bit-exact to stock.
A user successfully runs the Qwen 3.8 27B AI model on a mixed setup of RTX 3060 and 5060 Ti GPUs using tensor parallelism with exllamav3, achieving around 50 tokens per second with MTP enabled.
A developer demonstrates running MoE models like Qwen 3.6 35B-A3B and Gemma 4 26B-A4B at 24+ tok/s on an old GTX 1080 (8GB VRAM) with 128k context using llama.cpp with MoE offloading and TurboQuant KV cache quantization, revealing optimization tricks for Gemma's MTP speculative decoding.