This guide explains how to configure DFlash2 speculative decoding with llama.cpp to achieve up to 1.72x faster inference for the Qwen3.8-27B model on an RTX 4080 GPU with 16GB VRAM.
I saw that Q2 is actually very good and produce real good results and I also saw how dflash2 make its running at generating >60 t/s with a 120k context lenght. And I like what its doing!! Heres how to set it up (ai wrote this) DFlash2 speculative decoding on 16GB VRAM — 1.72x faster (setup guide) DFlash2 pairs a small drafter model with your big model. The drafter guesses a block of tokens in parallel, the target verifies them in one pass. Lossless — identical output quality, just faster. Result on RTX 4080 16GB with Qwen3.8-27B: 50.6 → 86.7 tok/s (1.72x), costs 561MB VRAM. --- Build llama.cpp from the PR branch DFlash2 is not in master (only DFlash 1 is). It lives in unmerged PR #27342. If you use a release build you'll get wrong number of tensors; expected 81, got 58 on every drafter quant. git clone --depth 1 https://github.com/ggml-org/llama.cpp.git llama.cpp_new cd llama.cpp_new git remote add zlab https://github.com/z-lab/llama.cpp-fork.git git fetch --depth 1 zlab dflash2 git checkout FETCH_HEAD Verify before building: grep -c "LLM_KV_DFLASH_SELECTOR_RANK" src/llama-arch.cpp # must print 1 Build into a separate directory from any existing llama.cpp. Get the models - Target: your Qwen3.8-27B GGUF (mine: Q2_K_P, 9.94GB) - Drafter: HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF (561MB) Drafters are model-pair-specific — trained for one exact target. Run it llama-server.exe \ -m Qwen3.8-27B-Q2_K_P.gguf \ --model-draft Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf \ --spec-type draft-dflash \ --spec-draft-n-max 3 \ -ngl 99 -c 122880 -fa on \ --cache-type-k q4_0 --cache-type-v q4_0 \ --host 0.0.0.0 --port 8080 --jinja --spec-type draft-dflash is required — without it the drafter loads but never engages. Note it's --model-draft (not --draft-model) and -ngl (not --ngl). --- Throughput Use Q2_K_S-MIX at n_max=3. It's faster than the official Q4_K_M at half the size. n_max=5 is worse across the board — acceptance drops faster than parallelism gains. Context vs speed 4k • KV: q4_0 • tok/s: 86.7 • Speedup: 1.72x 120k • KV: q4_0 • tok/s: 66.1 • Speedup: 1.31x Max stable context with the drafter is 122880. Above that you hit a PR bug (not OOM): - 131072 without drafter → works - 131072 with drafter → invalid vector subscript - 126976 → flaky (loads with --no-warmup, crashes with warmup)
The author shares a high-performance local inference configuration for running Qwen3.6 35B A3B on limited hardware (8GB VRAM, 32GB RAM) using a modified llama.cpp with TurboQuant support, achieving ~37-51 tok/sec with ~190k context.
A user benchmarks the newly merged DFlash speculative decoding method in llama.cpp on Qwen 3.6 27B, achieving up to 4.44x speedup at 36K context compared to baseline, with detailed leaderboard and quality tests.
Benchmarks of DFlash speculative decoding combined with KV cache compression on RTX 5090 show up to 3.26x speedup on Qwen3.6-27B with minimal perplexity degradation, with q4_0/turbo4 providing the best balance.
A detailed account of running the Qwen3.6-35B-A3B MoE model on an 8GB laptop GPU, covering effective optimizations like --no-mmap and VRAM headroom, unexpected findings where speculative decoding improved speed by 26% contrary to benchmarks, and pitfalls with Windows and CPU bottlenecks.
A user shares optimized settings for running Qwen3.6 27B (Q8_0) on a dual GPU setup (RTX 4090 + RTX 3090) with llama.cpp, achieving 75-100 t/s and 1500 pp with 250k context.