User benchmarks and details a GPU-resident expert cache PR in llama.cpp that boosts decode speed for Qwen3.8-Flash-Next on a dual RTX 3090 system from 17 to 25-29 tokens per second.
Sharing some numbers because most posts on this model are either using a single 3090 or unified systems from what I've seen. My current setup: 2x RTX 3090 (PCIe 3.0), dual Xeon E5-2696 v4, 188 GB usable (192GB) DDR4-2133 LRDIMM, llama.cpp, unsloth UD-Q6_K_XL. All 48 expert layers pinned in host RAM, everything else on the GPUs. Full 261k context, f16 KV. Before: ~17 t/s decode, ~350 t/s prefill on a 26k prompt, 12 t/s decode at 131k depth. Now: 25-29 t/s decode short and mid context, ~17 at 131k, prefill is still about the same. Measured this with a python coding prompt. Now as for what I did: PR #27861, the GPU-resident LRU expert cache. Instead of parking whole expert layers in VRAM, it caches recently used experts per layer. The experts this model picks for one token are mostly the same ones it picked for the last few dozen tokens. so the hit rate is 80-85% on code and higher on prose. The trick that made it pay was giving the cache VRAM, also dropping ubatch from 2048 to 512 frees ~5 GB per GPU (compute buffers scale with ubatch), which went from 80 to 135 slots per layer at full context. The cost here is slower prefill on long prompts, short prompts are kind of unaffected. Also, the things that did nothing or hurt on my box were: thread count, poll, CPU masks, q8 KV, lazy PLE, n-gram drafts on prose, MTP at temp 0.7 (verify batches re-read experts from host RAM, it only wins greedy or at deep context), and more than 2 cache uploads per step (saturates PCIe 3.0, hit rate collapses). To replicate you don't need my fork, it's just master plus the PR: git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp git fetch origin pull/27861/head:pr-27861 && git merge pr-27861 cmake -B build -DGGML_CUDA=ON && cmake --build build -j LLAMA_ATTN_ROT_DISABLE=1 numactl --interleave=all build/bin/llama-server -m Qwen3.8-Flash-Next-UD-Q6_K_XL-00001-of-00006.gguf \ -ngl 99 -c 261888 --parallel 1 -fa on \ -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" \ --numa distribute -t 16 -tb 44 -b 4096 -ub 512 -ctk f16 -ctv f16 \ --moe-expert-cache 135 Size the cache to whatever VRAM you have left after the KV and compute buffers, about 100 MB per slot per GPU on Q6. Testing UD-Q4_K_XL is the next item on my list and I'm going to revisit MTP again if it makes sense, still closely following Daniel's PR. Doubt there's much people with a similar setup to mine out there but this helps anyone or if you have questions on approaches to try, let me know.
A detailed benchmark comparing speculative decoding engines for Qwen 3.6 27B on a single RTX 3090, showing ik_llama achieving ~100 tokens per second in code generation. Results include decode TPS, TTFT, VRAM usage, and context degradation across 5 engine variants.
The article benchmarks llama.cpp and ik_llama.cpp for running the Qwen3.8-Flash-Next model on dual RTX 3060 hardware, showing that optimizing tensor splitting and ubatch settings can dramatically improve prefill speed from 36 t/s to 400 t/s, with comparisons of RAM usage.
The article reports that the Qwen3.8-Flash-Next model achieves 120 tokens/second generation speed and 12k tokens/second prefill on a system with 4x AMD R9700 GPUs using optimized vLLM and a custom Docker image.
This guide explains how to configure DFlash2 speculative decoding with llama.cpp to achieve up to 1.72x faster inference for the Qwen3.8-27B model on an RTX 4080 GPU with 16GB VRAM.
A user shares their optimized configuration for deploying the Qwen3.8-Flash-Next model on dual DGX Spark hardware, achieving up to 50t/s decode and 2,900t/s prefill speeds with technical patches and setup details.