I benchmarked Unsloth's Qwen3.6-27B NVFP4 on 1x/2x 5090s. MTP is great until it really isn't.

I've been using the GGUF version of Qwen3.6-27B for a while, so when Unsloth released an NVFP4 version I wanted to see what it could do in vLLM. Just to avoid confusion: this is Unsloth's Qwen3.6-27B NVFP4 release, not NVIDIA's separate NVFP4 release. The main thing I wanted to figure out was how to set num_speculative_tokens. I also wanted to know whether splitting the model across two 5090s actually helps generation speed, and whether MTP still works well once you add concurrency or a large context. The short answer: nspec=3 is very good for one user at short context. Once the GPU is busy with batching, or the context gets large, the advantage mostly disappears and can turn into a pretty nasty slowdown. https://preview.redd.it/dht5v418ileh1.png?width=2400&format=png&auto=webp&s=f4517815a4db1324b2ba16a4c5bf27ff1c0364aa Setup Model: Unsloth Qwen3.6-27B-NVFP4 (compressed-tensors; NVFP4 MLP, FP8 attention) GPUs: 2x RTX 5090, 32 GB each vLLM: 0.25.1 PyTorch: 2.11.0+cu130 Driver: 580.159.03 Attention backend: TRITON_ATTN Max model length: 65,536 1 GPU: tensor_parallel_size=1, max_num_seqs=16 2 GPUs: tensor_parallel_size=2, max_num_seqs=32 MTP method: qwen3_5_mtp For the MTP and concurrency tests, I used the Spec-Bench prompt dataset rather than a synthetic prompt like "count to 100." Single-request test: 24 prompts, 512 requested output tokens Concurrency tests: 256 output tokens, with 8/16/32/48/64 prompts at concurrency 1/4/8/12/16 Context tests: four fixed-length random prompts per context size, 256 output tokens The context sample is small, so I would treat those numbers as a strong signal for this setup, not a universal law. I left sampling at the model defaults: temperature=1.0, top_k=20, and top_p=0.95. That means the exact acceptance rate and MTP speed can move around between runs. For sections 1 and 3, decode speed is 1000 / median TPOT. In normal language, that is generation speed after the first token, with prompt processing excluded. Section 2 is different: it reports the combined output throughput of the whole server across every active request. 1. One GPU with MTP was faster than two GPUs GPUs MTP Decode tok/s Change vs same-GPU baseline 1x 5090 off 66 baseline 1x 5090 nspec=3 120 +82% 2x 5090 off 99 baseline 2x 5090 nspec=3 108 +9% This was the first result that surprised me. One 5090 with nspec=3 reached 120 tok/s, while two 5090s with the same MTP setting reached 108 tok/s. That does not mean the second card is pointless. It gives you much more room for KV cache, context, batching, and prefill. It just did not help single-request decode speed in this test. My guess is that the tensor-parallel communication cost changes the MTP tradeoff quite a bit. There was also more run-to-run movement on TP=2. My earlier run was 117 tok/s on one GPU and 118 tok/s on two. The one-GPU result was very consistent; the two-GPU result was not. I would not treat 108 as some fixed number everyone should expect. 2. MTP falls over once batching takes over These are total server output tokens per second on one RTX 5090. The number is combined across all active requests, not what each individual user sees. Concurrent requests MTP off nspec=3 Change 1 64 121 +87% 4 228 414 +81% 8 453 475 +5% 12 638 501 -22% 16 789 498 -37% At one request, MTP nearly doubled throughput. At eight requests it was barely helping. At 12 and 16 requests it was actively making the server slower. The interesting part is that acceptance did not collapse under load. It was about 73% at concurrency 1 and 71% at concurrency 16. My read is that normal batching is already keeping the GPU busy, so speculative verification becomes extra work that no longer saves enough decode steps to justify itself. The weird point here is concurrency 4. This run measured 414 tok/s with MTP, but an earlier run measured only 250 tok/s and had terrible tail latency. The baseline numbers and the concurrency 1/8/12/16 behavior reproduced pretty closely. I would want more repetitions before putting much faith in that +81% at concurrency 4. 3. Long context eventually flips MTP from faster to slower One RTX 5090, one active request Input length MTP off nspec=3 Change 2k 66 107 +62% 8k 65 100 +54% 32k 61 62 +2% 60k 57 46 -20% Two RTX 5090s with tensor parallelism, one active request Input length MTP off nspec=3 Change 2k 99 105 +6% 8k 96 117 +22% 32k 89 72 -19% 60k 81 45 -44% The crossover was different depending on the GPU setup. On one GPU, MTP was basically even at 32k and 20% slower at 60k. With TP=2, it was already 19% slower at 32k and 44% slower at 60k. On two GPUs, acceptance went from about 71% at 8k to 60% at 60k. At the same time, every verification pass gets more expensive as the KV cache grows. That combination seems to kill the benefit pretty quickly. So I would not use a blanket rule like "always disable MTP past 16k." On this machine, the crossover was somewhere between 8k and 32k with TP=2, and between 32k and 60k on one GPU. Your prompts and acceptance rates will move that point around. What I am actually using now One interactive user with short context: num_speculative_tokens=3 Around eight concurrent requests: test both; MTP was only +5% here 12+ concurrent requests on one 5090: MTP off 32k context with TP=2: MTP off 60k context on either setup: MTP off One extra warning from the larger sweep: TP=2 with nspec=8 at concurrency 16 crashed the vLLM server twice with CUDA error: an illegal memory access was encountered. That was 2/2 attempts on this exact model/backend combination. I am not claiming nspec=8 is broken everywhere. Quick quality sanity check Benchmark numbers are not very useful if the quantized model cannot do anything interesting, so I also gave it one agentic task in a desktop app I built for personal use: "build Flappy Bird." No follow-up prompts, no corrections, and no hints. It planned the task, wrote the code, and launched a working game in one pass. https://reddit.com/link/1v2l1zi/video/lfslklsukleh1/player Obviously, one successful demo is not a quality benchmark and does not prove parity with BF16 or the GGUF version. I am only including it as a sanity check that the model was still usable for a reasonably involved agentic workflow. Reproduction command This is the one-GPU nspec=3 server config: vllm serve /path/to/Qwen3.6-27B-NVFP4 \ --served-model-name qwen \ --tensor-parallel-size 1 \ --attention-backend TRITON_ATTN \ --max-model-len 65536 \ --gpu-memory-utilization 0.90 \ --max-num-seqs 16 \ --reasoning-parser qwen3 \ --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' And this is the single-request Spec-Bench client: vllm bench serve \ --backend openai-chat \ --endpoint /v1/chat/completions \ --model /path/to/model \ --served-model-name qwen \ --tokenizer /path/to/model \ --dataset-name spec_bench \ --dataset-path spec_bench_question.jsonl \ --spec-bench-output-len 512 \ --num-prompts 24 \ --max-concurrency 1 \ --ignore-eos \ --percentile-metrics ttft,tpot,itl,e2el Has anyone else tested this Unsloth release on two 5090s? I would be curious whether the weak TP=2 MTP scaling reproduces with another attention backend. I would also like to see the long-context sweep repeated with deterministic sampling and more prompts.