DFlash 2: Keep Drafting Parallel

# DFlash 2: Keep Drafting Parallel Source: [https://inco.ai/blog/dflash2/](https://inco.ai/blog/dflash2/) August 18, 2026 Inference is the bottleneck of the agent era\. Agents read, plan, and call tools, often for hours or days\. They consume tokens at a rate chat never approached\. Every one of those tokens takes a full forward pass over the model\. At Inco AI, we are building the inference stack scaled to the token economics of tomorrow\. This post is a sneak peek\. Our team released[DFlash](https://arxiv.org/abs/2602.06036)in January; it now runs in SGLang, vLLM, TensorRT\-LLM, and llama\.cpp\. NVIDIA measured[up to 15× throughput](https://developer.nvidia.com/blog/boost-inference-performance-up-to-15x-on-nvidia-blackwell-using-dflash-speculative-decoding/)with it on Blackwell GPUs; Google reported[3× more tokens per second](https://developers.googleblog.com/supercharging-llm-inference-on-google-tpus-achieving-3x-speedups-with-diffusion-style-speculative-decoding/)on TPUs; CoreWeave's production Kimi K2\.7 Code endpoint,[the fastest for that model on Artificial Analysis](https://www.coreweave.com/blog/kimi-k2-7-code-now-available-on-serverless-inference-with-leading-benchmark-price-performance), runs DFlash by default\. The ecosystem now builds on it:[NVIDIA](https://huggingface.co/nvidia/Kimi-K2.6-DFlash),[Red Hat](https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.dflash), and[Modal](https://huggingface.co/modal-labs/Kimi-K3-DFlash)have all published DFlash drafters; Meta \([Muse Glimmer](https://huggingface.co/meta-models/Muse-Glimmer-30B-assistant)\), Poolside \([Laguna](https://huggingface.co/poolside/Laguna-S-2.1-DFlash)\), Xiaomi \([MiMo\-V2\.5\-Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro-FP4-DFlash)\), and NVIDIA \([Nemotron 3\.5 Lightning](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash)\) ship official drafters with their own models\. On Hugging Face, DFlash models have been downloaded**more than 3\.5 million times**\(as of August 2026\)\. Speculative decoding is a core piece of the modern inference stack\.[1](https://inco.ai/blog/dflash2/#user-content-fn-modal)A small draft model guesses a block of tokens, and the target model verifies the whole block in one forward pass\. Good guesses turn one pass into several tokens; bad ones just get thrown away\. For years, though, the draft itself stayed***autoregressive***: one token at a time\. DFlash made it one\-pass too: the entire block, every position, predicted***in parallel***\. DFlash 2 drafting for Qwen3\.8\-27B on an Apple M5 Max with oMLX, side by side with autoregressive decoding\.DFlash 2 pushes parallel drafting one step further:**over 20% more output from every verification pass, for around 1% added cycle latency**, with the output provably unchanged\. Across benchmarks the gain runs 16–25%\. With the Qwen3\.8\-27B drafter released today, SGLang serves at**2\.7–3\.4× the throughput of autoregressive decoding**at batch size 1\. Predicting every position independently leaves headroom in two places: choosing the right tokens and holding accuracy to the end of the block\. DFlash 2 recovers both without giving up the one\-pass design\. ## [Run It Now](https://inco.ai/blog/dflash2/#run-it-now) DFlash 2 already runs in the mainstream inference engines: ``` pip install -U "sglang[all] @ git+https://github.com/sgl-project/sglang.git@refs/pull/35371/head#subdirectory=python" python -m sglang.launch_server \ --model-path Qwen/Qwen3.8-27B \ --speculative-algorithm DFLASH \ --speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2 \ --speculative-num-draft-tokens 8 ``` ``` pip install -U "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/52816/head" vllm serve Qwen/Qwen3.8-27B \ --speculative-config '{ "method": "dflash", "model": "incoai/Qwen3.8-27B-DFlash2", "num_speculative_tokens": 7 }' ``` ``` git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp git fetch origin pull/27342/head:pr-27342 git switch pr-27342 # NVIDIA CUDA cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build -j # Apple Silicon cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_METAL=ON cmake --build build -j ./build/bin/llama-server \ -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M \ -hfd incoai/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M \ --spec-type draft-dflash \ --spec-draft-n-max 7 ``` Download and install the[prebuilt oMLX with DFlash 2 support](https://github.com/z-lab/omlx-fork/releases/download/0.6.2-dflash2/oMLX-0.6.2-zlab-dflash2-arm64-signed.dmg)\. To run Qwen3\.8\-27B with DFlash 2: 1. Open the oMLX[Model Downloader](http://127.0.0.1:8891/admin/dashboard?tab=models&modelsTab=downloader)and download: - [`mlx\-community/Qwen3\.8\-27B\-4bit`](https://huggingface.co/mlx-community/Qwen3.8-27B-4bit) - [`incoai/Qwen3\.8\-27B\-DFlash2`](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2) 2. Open the[Model Manager](http://127.0.0.1:8891/admin/dashboard?tab=models&modelsTab=manager)and edit`mlx\-community/Qwen3\.8\-27B\-4bit`\. Configure DFlash with the following settings: - **DFlash**: enabled - **Draft model**:`incoai/Qwen3\.8\-27B\-DFlash2` - **Draft quantization**: enabled - **Runtime block size**:`5` - **Verify mode**:`dflash` 3. Save the settings and load the target model\. ## [The Right Tokens Are Already There](https://inco.ai/blog/dflash2/#the-right-tokens-are-already-there) DFlash predicts every position independently, in parallel\. Each pick is plausible on its own\. Yet nothing makes them fit together, and an incoherent block is cut short at verification\. Recent methods such as[Domino](https://arxiv.org/abs/2605.29707)and[DSpark](https://arxiv.org/abs/2607.05147)buy coherence with sequential Markov heads that rewrite each position's full\-vocabulary distribution\. But is that costly autoregressive correction really necessary? No\. The evidence is already in DFlash's own candidate lists\. Take the first position: DFlash's top pick is right 85\.4% of the time, but the right token is in its top 16 candidates 99\.5% of the time\. Even when the top pick is wrong, the right token is usually on the list\. Metric0123456Acceptance lengthRecall@185\.4%80\.3%79\.4%78\.3%77\.5%75\.9%72\.9%4\.27Recall@1699\.5%97\.3%94\.8%92\.6%90\.8%89\.4%87\.8%6\.79 Table 1\. Recall@1 \(how often the top pick is right\) and Recall@16 \(how often the right token is in the top 16\) at each draft position, conditioned on every earlier position being right\. Five\-layer Qwen3\-4B DFlash on GSM8K\. Acceptance length includes the verifier's next token\.An oracle that always picks the right candidate from the top 16 would lift the acceptance length from 4\.27 to 6\.79\.**That gap is pure selection headroom\.**We just need to select the right path through the candidates\. Diffusionisgood⟨mask⟩⟨mask⟩⟨mask⟩Independent Top\-1 Picksfor✗same word, twiceposition1decodingspeculativeslow⋮position2decodingthinkingmodels⋮position3⟨eos⟩again…⋮all adjacent pairs scored at once → one path keptaccepted outputfor target\-decoded tokenmask tokenaccepted draftselected path Figure 1\. The selector in one cycle\. With DFlash alone, each position keeps its top pick; here two neighbors both pick the same word, and the stutter dies at verification\. DFlash 2 keeps each position's top candidates, and the selector traces one coherent path through them; here, the whole block survives\.### [A Lightweight Path Selector](https://inco.ai/blog/dflash2/#a-lightweight-path-selector) Coherence is mostly local: a candidate's fit depends mainly on the token just before it, so scoring neighboring pairs should be enough\. DFlash 2 keeps the top 16 candidates at each position and scores every adjacent pair: for predecessoraaand current candidatebb, St\(a,b\)=Ut\(b\)\+⟨A\(a\)⊙H\(ht\),B\(b\)⟩\.S\_t\(a,b\)=U\_t\(b\)\+\\langle A\(a\)\\odot H\(h\_t\),B\(b\)\\rangle\.The score has two parts\. The first,Ut\(b\)U\_t\(b\), is DFlash's own logit: how much the drafter already likedbbon its own\. The second asks how wellbbfollowsaa:AAandBBgive each token a compact 256\-dimensional embedding, and the two embeddings are matched under a context gateH\(ht\)H\(h\_t\)that decides which parts of the match count\. In essence, this is a low\-rank bilinear attention over adjacent candidates\. Scoring stays fully parallel\. Every adjacent pair at every position is scored in one shot, with no extra backbone or LM\-head pass\. The only sequential work is the final walk over precomputed scores: starting from the last verified token, greedy follows the best successor at each step, sampling draws from the same scores, and rejection sampling restores the exact target distribution\. MethodParamsLatencyT = 0T = 1DFlash——4\.273\.78\+ DSpark correction\+77\.8M\+9\.6%4\.494\.08\+ path selection \(ours\)\+2\.0M\+0\.6%**4\.61****4\.25** Table 2\. Acceptance length with path selection alone \(no convolution\), for five\-layer Qwen3\-4B on GSM8K\. Overheads are relative to plain DFlash: parameters added to the drafter, and added draft–verify cycle latency\.The selector improves DFlash by**0\.34**tokens atT=0T=0and**0\.47**atT=1T=1\. It beats the DSpark correction in both settings with roughly 40× fewer parameters and 16× lower latency overhead\. Choosing is cheaper than predicting\. And there is still room: the oracle reaches 6\.79\. Pairwise scoring is the simplest selector we could think of, and we believe there is plenty to explore\. ## [Suffix Decay Is a Local Problem](https://inco.ai/blog/dflash2/#suffix-decay-is-a-local-problem) We also noticed[both recall rows above](https://inco.ai/blog/dflash2/#table-1)decline toward the end of the block\. Even the oracle decays: with perfect selection, accuracy still falls from 99\.5% at the first position to 87\.8% by the last\. No selector can fix that, because the candidates themselves are running out\. We call this**suffix decay**, and it is a backbone problem\. One suspect is capacity: a five\-layer backbone may be too small to preserve dependencies across the block\. If that is right, depth should help most at later positions\. And it does\! 3\-, 5\-, and 15\-layer DFlash models are almost identical at the first position, and fan apart down the block\. But depth is indiscriminate: ten extra attention blocks add capacity everywhere, even at the early positions that had little left to gain, and erase much of the efficiency that makes DFlash attractive\. Figure 2\. Qwen3\-4B Recall@1 on GSM8K at T=0, conditioned on every earlier position being right\. All drafters are trained under the same setup; the convolutional model is evaluated without the selector\. Its convolutions add 3% parameters and 0\.7% cycle latency; the ten extra layers of 15L add 15\.2%\.Draft position0123456DFlash 3L85\.21%79\.26%77\.18%75\.75%73\.96%70\.4%64\.97%DFlash 5L85\.39%80\.31%79\.39%78\.27%77\.39%76\.03%72\.86%DFlash 15L \(3× more params\)86\.42%81\.61%80\.68%80\.34%80\.59%79\.66%78\.73%DFlash 5L \+ conv \(\+3% params\)85\.83%80\.94%79\.98%79\.68%79\.73%79\.43%77\.61% Figure 2\. Qwen3\-4B Recall@1 on GSM8K at T=0, conditioned on every earlier position being right\. All drafters are trained under the same setup; the convolutional model is evaluated without the selector\. Its convolutions add 3% parameters and 0\.7% cycle latency; the ten extra layers of 15L add 15\.2%\.We want a targeted fix, and DFlash's attention shows where\. It has two jobs: read the context before the block, and model the dependencies inside\. But it spends less and less on the second: the block's share of attention falls from**30% in Layer 1 to 8% in Layer 5**, and what remains concentrates in[a shrinking handful of heads](https://inco.ai/blog/dflash2/#figure-3)\. So we split the jobs: a dedicated module takes the within\-block work, and attention keeps reading the context\. 1 4 8 12 16 20 24 28 32 Layer 1 Layer 2 Layer 3 Layer 4 Layer 5 Attention head 0%90%within\-block mass Heatmap dataAttention head1234567891011121314151617181920212223242526272829303132Layer 117\.6%2\.9%41\.4%50\.8%29\.2%50\.5%44\.6%5\.9%44\.5%11\.3%17\.9%36\.7%0\.0%14\.2%0\.1%0\.0%13\.3%1\.5%18\.7%7\.6%45\.0%33\.5%53\.1%42\.2%64\.3%60\.1%32\.8%47\.7%49\.6%57\.0%26\.0%52\.9%Layer 220\.8%26\.4%39\.6%18\.9%8\.9%22\.6%13\.1%32\.1%22\.9%25\.1%24\.2%28\.6%36\.6%26\.1%41\.0%36\.1%17\.8%25\.5%25\.7%25\.6%4\.3%21\.8%23\.3%22\.1%15\.6%70\.9%58\.0%2\.7%28\.3%38\.5%20\.3%33\.5%Layer 31\.8%11\.0%9\.5%5\.2%34\.8%8\.4%12\.1%14\.4%11\.8%22\.0%8\.8%3\.7%4\.9%10\.6%17\.7%52\.0%4\.4%19\.0%13\.1%9\.9%61\.3%76\.1%47\.0%60\.3%1\.4%8\.9%6\.0%64\.1%9\.4%3\.3%8\.3%8\.3%Layer 40\.4%37\.7%28\.3%85\.5%0\.3%1\.5%0\.4%0\.5%1\.2%12\.5%36\.6%1\.2%1\.7%0\.6%2\.5%1\.3%7\.2%3\.1%48\.9%3\.8%3\.2%1\.0%23\.8%1\.0%0\.1%0\.1%0\.2%0\.3%2\.8%6\.7%12\.9%12\.3%Layer 51\.5%0\.2%0\.6%0\.1%60\.2%76\.0%0\.9%0\.0%0\.2%12\.3%32\.3%0\.1%15\.8%0\.5%0\.5%0\.5%0\.2%0\.1%0\.6%0\.2%0\.3%28\.1%0\.2%1\.3%0\.1%0\.1%0\.2%29\.9%0\.1%0\.1%0\.1%1\.2% Figure 3\. Within\-block attention by head in five\-layer Qwen3\-4B DFlash\. Brighter cells mark heads that spend more attention on the draft block; in later layers the within\-block mass shrinks and concentrates in a few heads\.### [A Lightweight Local Convolution](https://inco.ai/blog/dflash2/#a-lightweight-local-convolution) The within\-block work is short\-range to begin with: a block spans only 4 to 16 tokens, and the tightest dependencies sit between neighbors\. The natural operator is a short convolution: two taps, one on the current position and one reaching one position back, with weights that adapt to the content\. Following[Canon Layers](https://arxiv.org/abs/2512.17351),[Dynamic Short Convolutions](https://arxiv.org/abs/2606.03825), and[Convolution for Large Language Models](https://arxiv.org/abs/2607.18413), we insert this two\-tap dynamic depthwise convolution before and after each attention and feed\-forward sublayer: Conv⁡k\(x\)t=kt,0⊙xt\+kt,1⊙xt−1\.\\operatorname\{Conv\}\_\{k\}\(x\)\_t =k\_\{t,0\}\\odot x\_t\+k\_\{t,1\}\\odot x\_\{t\-1\}\.Each coefficient combines a learned base kernel with a small correction computed from the current hidden state; every 16 channels share one correction\. The first position reads the last verified token's representation, and every later position reads its predecessor's\. Information crosses the block while all positions still compute in parallel\. ×5 layersAttentionMLP…Inside one conv— every position takes two tapsxConv\(x\)verifiedpos1pos2pos3pos1pos2pos3k₁ · the predecessork₀ · itselftwo\-tap convlast verified tokendraft positions Figure 4\. The two\-tap dynamic convolution\. One sits before and after each attention and MLP sublayer of every drafter layer\. Inside it, each position mixes its own representation with its predecessor's, and the first position reads the last verified token\.The convolution is block\-local and stateless, so it drops into DFlash without changing attention, the LM head, or verification\. With only**16\.5M added parameters \(3%\)**, five\-layer DFlash with convolution[comes close to 15\-layer DFlash](https://inco.ai/blog/dflash2/#figure-2), substantially reducing suffix decay\. The convolutions add**0\.7%**to draft–verify cycle latency; ten more Transformer layers add 15\.2%\. Average within\-block attention across Layers 4 and 5 also falls from**9\.4% to 0\.5%**, consistent with the convolution absorbing the local work while attention goes back to reading the context\. A kernel reaching one position back recovers most of what ten extra layers buy: suffix decay is mostly a*local*problem\. ## [Putting It Together](https://inco.ai/blog/dflash2/#putting-it-together) So far, the selector and the convolution have been measured separately;[the full comparison below](https://inco.ai/blog/dflash2/#table-3)puts them together\. We trained the DFlash and DSpark drafters ourselves under matched setups, while MTP ships with the model\. Qwen3\.5\-4B DatasetMTPDFlashDSparkDFlash 2GSM8K4\.784\.995\.69**6\.20**MATH\-5005\.045\.426\.20**6\.76**HumanEval4\.845\.435\.80**6\.28**MBPP4\.164\.494\.96**5\.41**MT\-Bench3\.904\.264\.77**5\.20**Mean4\.544\.925\.49**5\.97** Table 3\. Qwen3\.5\-4B per\-request mean acceptance length\. Sampling: thinking enabled, temperature 1\.0, top\-p 0\.95, top\-k 20, presence penalty 1\.5, with lossless rejection sampling\.DFlash 2 leads on every benchmark\. Averaged across them, it gains**1\.05 tokens over DFlash \(21%\)**and**0\.48 over DSpark**\. The upgrade stays cheap: the selector and the convolution together add only**1\.3%**to the five\-layer DFlash draft–verify cycle latency\. On MATH\-500,[the gain is visible position by position](https://inco.ai/blog/dflash2/#figure-5): DFlash 2 holds steady near 86% to the last position, and every baseline ends the block 6 to 9 points below it\. Figure 5\. Qwen3\.5\-4B conditional acceptance rate on MATH\-500, same sampling as above\.Draft position01234567891011121314MTP84\.57%80\.23%79%78\.42%78\.63%78\.17%77\.36%77\.74%77\.91%76\.96%78\.06%77\.4%77\.49%77\.48%77\.85%DFlash88\.35%77\.7%77\.8%79\.45%80\.3%81\.12%81\.22%81\.07%81\.29%80\.28%80\.64%80\.29%79\.56%78\.77%77\.48%DSpark87\.24%84\.59%83\.79%83\.63%83\.6%83\.27%82\.97%82\.54%82\.21%82\.39%81\.58%80\.7%81\.35%80\.57%79\.86%DFlash 288\.3%85\.3%84\.98%84\.88%85\.41%85\.3%85\.36%85\.13%85\.95%85\.99%86\.41%86\.46%86\.43%86\.02%86\.48% Figure 5\. Qwen3\.5\-4B conditional acceptance rate on MATH\-500, same sampling as above\.## [Two Drafters, Out Today](https://inco.ai/blog/dflash2/#two-drafters-out-today) We are releasing two DFlash 2 drafters today:[one for Qwen3\.8\-27B](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2)and[one for Meta's Muse Glimmer](https://huggingface.co/incoai/Muse-Glimmer-30B-DFlash2)\. For Qwen3\.8\-27B, we compare against the model's native MTP path and a[community DSpark drafter](https://huggingface.co/RadixArk/Qwen3.8-27B-DSpark)\. Qwen3\.8\-27B DatasetMTPDSparkDFlash 2GSM8K5\.024\.36**5\.46**MATH\-5004\.723\.92**5\.28**HumanEval3\.913\.30**4\.39**MBPP3\.993\.51**4\.79**MT\-Bench3\.743\.01**4\.10**Mean4\.283\.62**4\.80** Table 4\. Qwen3\.8\-27B per\-request mean acceptance length with the model's default sampling and a block size of 8, against its native MTP path and a community DSpark drafter\.For Meta's Muse Glimmer, we compare against the official DFlash drafter shipped with the model and a[community DSpark drafter](https://huggingface.co/DaoCloud/Muse-Glimmer-30B-DSpark)\. Muse Glimmer DatasetDFlashDSparkDFlash 2GSM8K5\.435\.45**6\.57**MATH\-5005\.395\.01**6\.56**HumanEval4\.114\.33**5\.66**MBPP3\.744\.02**5\.30**MT\-Bench3\.523\.59**4\.42**Mean4\.444\.48**5\.70** Table 5\. Muse Glimmer per\-request mean acceptance length with the model's default sampling and a block size of 16\. DFlash is the official drafter Meta ships with the model; DSpark is a community drafter\.The margins are wide: on both models, DFlash 2 averages more than**a full token**ahead of DSpark\. It also beats each model's official drafter, MTP on Qwen3\.8\-27B and DFlash on Muse Glimmer\. That translates into**2\.7–3\.4×**the throughput of autoregressive decoding on Qwen3\.8\-27B, and**3\.1–4\.6×**on Muse Glimmer\. The[model cards](https://huggingface.co/collections/incoai/dflash-2-6a8432273c9998ce1685d4c5)break the speedups down by task and concurrency\. ## [The Bottom Line](https://inco.ai/blog/dflash2/#the-bottom-line) An agent writes in an afternoon what a chatbot writes in a month, and decoding sits under every one of those tokens\. DFlash 2 decodes at**close to 3× the speed of autoregressive decoding, about a third of the compute per token**, with the same output\. In seven months, DFlash went from our paper to an industry standard, with more than 3\.5 million downloads\. Inside the same design, DFlash 2 decodes one more full token per pass, for free\. That is only one component of the serving stack\. Inference is nowhere near its floor\. At Inco AI, we are building an end\-to\-end serving stack to keep pushing that floor lower\. DFlash 2 is the first piece\. Two drafters are out today[on Hugging Face](https://huggingface.co/collections/incoai/dflash-2-6a8432273c9998ce1685d4c5)\. If you serve agents at scale and want to evaluate DFlash 2 in your stack, or want a drafter for a model you run, including your own fine\-tunes, write to us:[contact@inco\.ai](mailto:[email protected])\. We are also hiring\. If you want to help build this stack, reach out to us\. **Connect the candidates\. Keep drafting parallel\.** Get updates One email when we ship something new\. We will never share your email address\. ## [Citation](https://inco.ai/blog/dflash2/#citation) Please cite this post as: ``` @misc{inco2026dflash2, title = {{DFlash 2: Keep Drafting Parallel}}, author = {{Inco AI}}, year = {2026}, month = {August}, url = {https://inco.ai/blog/dflash2/} } ``` 1. Modal's["Speculation Is All You Need"](https://modal.com/blog/spec-is-all-u-need)points out that speculative decoding is the optimization that matters for low\-latency serving\. We are huge fans of their work and appreciate their support and discussions since DFlash's release\.[↩](https://inco.ai/blog/dflash2/#user-content-fnref-modal)