What is the meta for running Qwen 3.6 27B at 5-10 tok/s as cheap as possible (without speculative decoding)?
Summary
A question about the cheapest way to run Qwen 3.6 27B at 5-10 tok/s without speculative decoding, focusing on inference optimization and cost reduction.
Similar Articles
Qwen3.6-27B speculative decoding gets better on heavier quants
Qwen3.6-27B model shows improved speculative decoding performance when using heavier quantization levels, enhancing inference efficiency.
Qwen 3.6 27B Speculative Decoding Bench: Pushing ~100 TPS on a single RTX 3090
A detailed benchmark comparing speculative decoding engines for Qwen 3.6 27B on a single RTX 3090, showing ik_llama achieving ~100 tokens per second in code generation. Results include decode TPS, TTFT, VRAM usage, and context degradation across 5 engine variants.
Qwen 27B
A user reports that Qwen 27B at q6kxl quantization with multi-token prediction achieves 50-90 token/s decode and 1500-2200 token/s pre-fill on a 4090+3090 system using LCPP, noting it is reliably coherent and fast for various coding tasks.
Qwen3.6 27B on a 5090, 6.4k sample tok/s distribution after tuning MTP/cache settings
Running Qwen3.6 27B on an RTX 5090, achieving 6.4k tokens per second after tuning MTP and cache settings, demonstrating optimization techniques for inference.
Running Qwen3.6 35b a3b on 8gb vram and 32gb ram ~190k context
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.