More 4x Intel B65 test results - Performance improvements - ~30% faster prefill and TTFT in flash-next vllm by increasing chunk size. & MPT benchmarks - and a video showing me actually using it in LibreChat.
A hobbyist shares test results showing ~30% faster prefill and TTFT in vLLM Flash-Next by increasing the chunk size to 8192 on a 4x Intel B65 build, with detailed Qwen3.8-Flash-Next benchmark tables and a LibreChat demo.
edit: video clip is of me is using Flash-Next on Libre LLM with: Model: devan-carlin/Qwen3.8-Flash-Next-W4A16, INT4 weights with 16-bit activations (W4A16). The BF16 n-gram embedding table is pinned in host RAM. Engine: vLLM 0.30.0 with the Lumnus b70.2 patches and full XPU graphs. Tensor parallel across all 4 cards, with expert parallel. GPU memory: 0.92, which leaves about 350K tokens of KV cache in BF16. Speculative decoding: MTP-3, three draft tokens from the model's own MTP head. Prefill chunk: 8,192 tokens (--max-num-batched-tokens). Context and batching: 262K-token window, up to 16 requests at once, normal scheduling (async off). Howdy folks! some of you have been following my 4x b65 build, i’m sure you’ll be happy to hear that I’m focusing on performance improvements now! So far my testing has been using official vllm guidance so that I can have a baseline and control to test against. I’m documenting my journey on my site here: https://b65.thekaiser.us/by-model/qwen38-flash-next/prefill/ I’m not sponsored, just doing this for the love of the game and to learn and better understand the technology for my own selfish business gains. As I’m starting to turn dials, chunk size stands out to me. I’m used to CUDA, where I’m using 4 to 8 times the chunk size that VLLM defaults to. So naturally I started tweaking that dial and here are some of the results! tldr: looks like VLLM defaults to 1024 chunk sizes, larger chunk sizes can cause pauses, especially when running multiple sessions. It also reduces the total number of sessions that can run at one time. But you will still be able to run multiple sessions before you hit that wall. Are those pauses actually noticeable? Do they matter in a single user environment, or with less than six agents at a time? In my testing so far, the pauses are not noticeable when I’m using chat, and when I’m using agents in the background - all their tasks complete faster end to end. So right now I’m going to keep running it at 8192. if you’ve found any downsides that I haven’t ran into yet, please share and let me know! i’m guessing the default is 1024 because they’re intended for many more simultaneous streams? But I’m just speculating here. How was AI used in this post? I wrote everything except for what’s below. This is an AI generated results overview based on my data: Qwen3.8-Flash-Next: Prefill Prefill speed by chunk size, and throughput and pauses with 1 to 8 users while an agent reads 8K-128K prompts. All-time best of Qwen3.8-Flash-Next decode · 1 user: 84.5 tok/s (graph build · MTP-3 · 4 cards) aggregate · 4 users: 243.2 tok/s (MTP-3) 8 users + agent: 169.1 tok/s (MTP-3 · agent reading 8K-128K) prefill · 32K prompt: 5,142 tok/s (chunk 16,384) first token · 261K: 97.6 s (chunk 16,384) longest prompt: 1,039,940 tok (YaRN ×4 · FP8 KV) Each user, 8 at once: 12.1 tok/s (6 users: 16.7) Longest pause: 2.89 s, chunk 8,192 (1,024: 0.61 s) Prefill by chunk size One request, no MTP, 262K window. Prefill in tok/s. Campaign 2026-10-20-b65-flashnext-prefill. Prefill speed by prompt length, Prefill (tok/s) by Prompt length (tokens) Setup 1 2 3 4 1,024 3,559 3,161 2,035 1,372 2,048 3,962 3,730 2,598 1,851 4,096 4,440 4,695 3,411 2,479 8,192 (default) 4,823 5,068 3,577 2,530 16,384 4,888 5,142 3,770 2,673 p1 / p3: chunk size and memory, no MTP, 262K window Chunk Memory KV pool (tokens) 8K 32K 128K 261K TTFT 261K 1,024 (control) 0.88 587,416 3,559 3,161 2,035 1,372 190.2 s 2,048 0.88 573,990 3,962 3,730 2,598 1,851 140.9 s 4,096 0.88 534,187 4,440 4,695 3,411 2,479 105.2 s 8,192 0.88 461,091 4,844 5,066 3,575 2,530 103.1 s 16,384 0.88 337,952 4,888 5,142 3,770 2,673 97.6 s 8,192 0.92 531,136 4,823 5,068 3,577 2,530 103.2 s 16,384 0.92 441,359 4,858 4,960 3,769 2,674 97.6 s Full page, with interactive charts and the raw data behind every number: https://b65.thekaiser.us/by-model/qwen38-flash-next/prefill/
A hobbyist benchmarks running Qwen 3.6/3.8 MoE models on a £300 Snapdragon X mini PC (optionally dual-boxed via USB4), comparing llama.cpp, Janas, and their own unreleased ToRc engine, hitting 315 tok/s read on the 35B A3B model. The post details hardware setup, MTP head usage, and NPU-assisted CPU inference results.
A user benchmarks the newly merged DFlash speculative decoding method in llama.cpp on Qwen 3.6 27B, achieving up to 4.44x speedup at 36K context compared to baseline, with detailed leaderboard and quality tests.
A hobbyist reports stabilizing a dual CMP 170HX (64GB each) rig running GLM-5.3-Flash via EXL3 3.05bpw at 384K context and ~90 tok/s, and compares it against Qwen3.8-Flash-Next on AWQ INT4 + FP8 PLE under vLLM, including agent-task latency. The post also analyzes trellis-based EXL3 quantization fidelity versus common UD-Q4/Q5 GGUF variants.