Speculative decoding, from zero to DSpark

Written with help from Muse Spark for drafting, editing, and figures. All the mistakes are its.

A new paper called DSpark came out recently from the DeepSeek team, and I wanted to understand what it adds beyond regular speculative decoding. The basic trick for speculative decoding is to draft several tokens cheaply, then verify them with the big model in one pass. DSpark builds on that with two ideas: better long drafts, and a smarter way to decide how many drafted tokens to verify in production.

1. Why decoding is slow

An autoregressive model generates one token per forward pass, and each pass has to stream every weight matrix out of HBM (High Bandwidth Memory) into the arithmetic units. It’s about 140 GB of reads per token for a 70B model in 16-bit. The arithmetic is tiny by comparison: roughly one multiply-add per weight. An H100 does ~10¹⁵ multiply-adds per second but reads only ~3 TB/s. Decoding is memory-bound; the multipliers sit idle waiting for weights.

The loophole is that the weight reads are paid per pass, not per token. If you push eight token positions through one pass, the weights are read once, multiplied against eight vectors. Memory traffic barely changes; the extra arithmetic was free capacity anyway. This is the familiar prefill/decode gap. Decoding can’t batch like prefill because token t+1 doesn’t exist until the pass for token t finishes, but if someone guesses the next eight tokens, the big model can check them prefill-style in one pass. Verification is prefill-shaped while generation is decode-shaped. This is the idea behind speculative decoding: You convert slow generation into fast verification plus a cheap guess.

2. The loop, and why it’s lossless

A small draft model proposes a block of γ tokens. The target verifies the whole block in one pass, walking left to right: accept, accept, … until the first disagreement. The nice think is that a rejection isn’t wasted because the target already computed a distribution at that position and samples a correction token from it. If everything is accepted, the round earns a bonus token from the position after the block.

The output is exactly distributed as if the target had generated alone, thanks to a rejection-sampling rule: accept drafted token xₖ with probability min(1, pᵗ(xₖ)/pᵈ(xₖ)), and on rejection resample from norm(max(0, pᵗ − pᵈ)). A nice consequence is that thethe per-position acceptance probability is 1 minus the total-variation distance between drafter and target - a quantity DSpark later reuses as a free training label.

Latency per token is (T_draft + T_verify)/τ, where τ is tokens gained per round. So you can draft faster, draft better (raise τ), or verify smarter. DSpark goes after two of the three.

3. Two families drafter and where the parallel one breaks

Autoregressive drafters (EAGLE, DeepSeek’s MTP) generate the draft token by token. Coherent, but T_draft ∝ γ, so they must stay shallow (EAGLE is one layer) with small blocks. Parallel drafters (Medusa, DFlash) fill all γ positions in one pass. They anchor token plus mask tokens in and logits everywhere out. This is why they can afford depth (DFlash uses 5 layers) and long blocks.

DFlash’s central trick is KV injection: hidden states from several target layers are saved at prefill, projected into the drafter’s width, and prepended to every drafter layer’s keys and values. The drafter isn’t understanding the conversation with 5 layers of its own, but it’s reading the big model’s notes!

But parallel drafters predict positions independently. From the context “Sure, “ the target likes of course and no problem; sampling each position from its marginal can produce a problem where fragments of two valid answers stitched into garbage. This multi-modal collision (known since non-autoregressive MT, Gu et al. 2018) shows up as suffix decay: conditional acceptance falls off rapidly with position in the block.

4. DSpark idea #1: bolt a bigram onto the parallel drafter

DSpark keeps the expensive 5-layer parallel backbone and adds autoregression only where it costs almost nothing: a small sequential head walks the block left to right and adds a logit correction that depends on the token just sampled before it. The default head is a learned bigram: a V×V correction table stored factorized at rank 256, so each position costs one lookup plus one small matrix-vector product. Now when position 1 samples of, the bigram boosts course and suppresses problem at position 2.

The authors also try an RNN head carrying the whole within-block prefix. It helps only marginally. I think that near-null result is the most useful thing in the paper: suffix decay is mostly adjacent-token incoherence, not missing long-range information. The KV injection already gives every position the whole conversation.

Across Qwen3-4B/8B/14B, DSpark’s accepted length comes out 27–31% above Eagle3 and 16–18% above DFlash, and a 2-layer DSpark already beats the 5-layer DFlash.

DSpark keeps the expensive drafter computation parallel, then adds a cheap sequential Markov head to improve within-block coherence.

DSpark keeps the expensive drafter computation parallel, then adds a cheap sequential Markov head to improve within-block coherence.

5. The serving problem

Everything above is a single-user story. In production one target model serves hundreds of concurrent requests, and each pass processes a batch. The batch is a shared, finite resource: while it’s small the pass is memory-bound and extra tokens ride along free; past some size the pass goes compute-bound and every extra token slows everyone down. Each engine has a characteristic curve SPS(B): steps per second versus batch size, flat then falling.

Every draft token submitted for verification takes a batch slot, and a rejected token wastes its slot. Worse, acceptance depends on content. Roughly 5.6 accepted tokens per round on math, 5.1 on code, 3.5 on chat (Qwen3-4B). That means there is no fixed verification length. This is why DeepSeek’s production system ran MTP-1, one draft token per round, despite having multi-token drafters ready: longer static drafts hurt total throughput at production concurrency. At γ = 1 the ceiling is roughly a 2× speedup.

Extra verification tokens are nearly free while the batch is memory-bound, but costly once the system becomes compute-bound.

Extra verification tokens are nearly free while the batch is memory-bound, but costly once the system becomes compute-bound.

6. DSpark idea #2: schedule verification by expected throughpu

Instead of one length for everyone, the scheduler picks a verification length per request, per step, to maximize expected tokens per second: expected output tokens of the batch times SPS at the resulting batch size.

The missing ingredient is each draft token’s survival probability. For that, they use a confidence head - a linear projection over the backbone hidden state plus the Markov embedding to predicts each position’s conditional acceptance; the chain rule turns conditionals into survival probabilities. Its training label is free: the exact acceptance probability (1 − TV distance) is computable at every training step. The scores are then calibrated with per-position temperature scaling, shrinking the predicted-vs-observed gap from 3–8% to ~1%.

Because survival probabilities are monotonically decreasing within a request, a greedy algorithm works: You need to pool all candidate extensions across all requests, sort by survival probability, admit from the top while throughput improves. At light load the scheduler verifies long (4–6 tokens per request); as the batch fills, budgets shrink. A code request at 0.9 confidence keeps a long verification while a chat request whose confidence collapses gets cut at position 2.

One subtlety: losslessness requires non-anticipation - whether token k is admitted must not depend on the value of token k. A naive argmax over admission paths violates it, because the confidence at position k+1 depends on which xₖ was drawn, biasing the output toward tokens with confident continuations.

7. Making it run in a real engine

Two things break in DeepSeek’s actual engine. The SPS curve is jagged (kernel tile boundaries), so early stopping strands the search at local optima. And with CUDA-graph replay, the next batch size must be fixed before the current pass finishes, but the scheduler needs confidence scores that don’t exist yet.

They fix it by compute the batch’s token capacity K two steps ahead, from old confidence scores, then fill those K slots at the last moment by ranking the live candidates. This also restores losslessness by construction because the cut-off was fixed before any of this step’s tokens existed, and the ranking never consults token k or anything after it. The same design choice fixed the pipeline stall and the exactness proof; my guess is the pipeline constraint came first and the losslessness argument was noticed after the fact.

Deployed on DeepSeek-V4-Flash and V4-Pro under real traffic against the incumbent MTP-1, the verification budget sits at 4–6 tokens per request at moderate concurrency and shrinks automatically as load rises.

8. What to actually remember

Earlier work mostly asked how to build a better drafter: cheaper, deeper, more accurate. DSpark shows that’s half the story. In a real serving system the question is not “how many tokens can I draft?” but “which drafted tokens are worth spending target-model batch capacity on right now?” A semi-autoregressive Markov head makes long parallel drafts coherent; a throughput-aware scheduler decides how much of each draft to verify under the current load.

Open questions: the SPS table ignores context-length mix; the confidence head is trained on teacher-forced prefixes but deployed on sampled ones; whether calibration survives live traffic drift isn’t reported; and the drafter still burns a full γ-block forward on every request.

References

• Cheng et al. (2026), DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

• Chen et al. (2026), DFlash: Block Diffusion for Flash Speculative Decoding

• Leviathan et al. (2023); Chen et al. (2023) — speculative sampling

• Li et al. (2024, 2025) — EAGLE, EAGLE-3

• Cai et al. (2024) — Medusa

• DeepSeek-AI (2024), DeepSeek-V3 Technical Report (MTP objective)

• Gu et al. (2018) — non-autoregressive NMT, multi-modal collision

• Kwon et al. (2023) — vLLM; Zheng et al. (2023) — SGLang

• DeepSeek-AI DeepSpec repository