@Snixtp: The RTX Pro 6000 has two matmul engines, and vLLM was only using one of them when I ran Qwen3.6 27B NVFP4 Marlin is nic…
Summary
A developer discovered that vLLM only used one of two matmul engines on the RTX Pro 6000 for Qwen3.6 27B models. A plugin by Fable 5 selects the right engine per call, nearly doubling prefill performance.
View Cached Full Text
Cached at: 07/05/26, 12:55 AM
The RTX Pro 6000 has two matmul engines, and vLLM was only using one of them when I ran Qwen3.6 27B NVFP4
Marlin is nice for decoding, but CUTLASS is 3.5x faster processing large token prompts. For some reason, vLLM ran Marlin for both, because NVIDIA’s checkpoint files mislabel what the model can actually do.
Fable 5 found out about all this and wrote a vLLM plugin that looks at the batch size and picks the right engine per call.
Results is almost 2x increase in prefill at all context lengths:
Similar Articles
Qwen 3.6 benchmarks on 2x RTX PRO 6000
Benchmarks for Qwen 3.6 27B and 35B models on dual RTX PRO 6000 GPUs using VLLM, showing generation throughput up to 3500 tokens per second.
Qwen3.8FN vLLM vs. llama.cpp vs Ninfer6000
A community member benchmarks Ninfer6000 (a Ninfer fork tuned for RTX Pro 6000 Blackwell and Qwen3.8-Flash-Next) against vLLM and llama.cpp, finding ~1.6x decode speedup over vLLM and 2.2x over llama.cpp with near-identical accuracy, but reports crippling real-world tool-calling malformations that keep them on vLLM.
RTX Pro 4500 Blackwell - Qwen 3.6 27B?
A developer shares local inference benchmarks and systemd configurations for running the Qwen3.6-27B model on an NVIDIA RTX Pro 4500 Blackwell GPU using llama.cpp. The post requests optimization tips for throughput and explores potential use cases for larger models.
@Snixtp: https://x.com/Snixtp/status/2055734339346768225
A user benchmarks the MTP variant of Qwen3.6 27B against the normal version on a single RTX 3090 using llama.cpp, finding MTP offers up to 2.37x faster generation at long contexts (32k-64k) but with slower prefill and no concurrency support yet.
Qwen3.8-Flash-Next (UD-IQ4_XS) on 2x RTX 3060 + 7800X3D, from initial 36 tps prefill to 400 tps and other benchmarks (-sm tensor trap) + VRAM/RAM usage
The article benchmarks llama.cpp and ik_llama.cpp for running the Qwen3.8-Flash-Next model on dual RTX 3060 hardware, showing that optimizing tensor splitting and ubatch settings can dramatically improve prefill speed from 36 t/s to 400 t/s, with comparisons of RAM usage.