@dzhulgakov: DSpark from @deepseek_ai ingeniously integrates many speculative decoding ideas to achieve 1.5x to 5x higher throughput…

DSpark from @deepseek_ai ingeniously integrates many speculative decoding ideas to achieve 1.5x to 5x higher throughput in a real production system

Let’s understand it with 10 ideas, starting from the very basics

1. Batching in LLM decoding

Generating tokens is bound on reading weights from memory. So decoding 10 tokens in parallel is only slightly slower than generating 1. Continuous batching leverages this insight

2. Speculative decoding

For the same request, token N+1 depends on token N, so we can’t decode them in parallel. But if we can guess what the tokens are (“speculate”), we can quickly verify which prefix of them is correct from the main model perspective

3. Draft model

How to speculate? With a model, of course. The simplest is to run a smaller model trained on the same distribution, e.g. Qwen 0.8B for Qwen 397B

4. Speculation is not free

time_per_token = (num_tokens_drafted * drafter_time + verify_time(num_tokens_drafted)) / num_tokens_accepted

Slow to run speculator or drafting too many tokens with a low guess rate can be hurtful. The right balance is needed

5. EAGLE and MTP

Make drafter an extra transformer layer of the main model, i.e. it consumes rich latent representation (last activation) in addition to the previous token. Allows to get away with 1-2 layers instead of the full model

Much faster and more accurate speculator

6. DFlash

MTP needs to take N steps to generate N draft tokens. DFlash uses diffusion ideas to produce all N tokens in one forward pass

Much faster speculation, but draft quality sometimes better, sometimes worse than MTP/Eagle

7. DSpark ~= Eagle + MTP

DFlash is stronger at initial positions Eagle3 is more coherent on long drafting but has lower quality guess in the beginning

DSpark combines both parallel block and autoregressive ideas, beating either approach

8. Cheaper sequential block

Eagle3/MTP run full attention at each drafting position. Since DFlash has parallel block to capture previous context, sequential step can be much cheaper with RNN or even Markov model. All leads to an even faster but still accurate drafter!

9. Variable length drafting and hardware-aware scheduler

What num_draft_tokens should be? It varies:

• some requests (e.g. coding) are easier to predict than others
• optimal length depends on server load (batch size). Speculate more with low load when GPU compute is free, speculate less when running with high throughput

Some engines have a static –num-draft-tokens argument. It’s much better to vary it dynamically based on the drafter’s confidence and the current server load.

Implementing it is tricky. CPU/GPU are heavily pipelined in inference engines, and varying length usually requires changing tensor sizes and going back to the CPU, slowing things down.

With careful implementation (and a lot of cool CUDA kernels), everything can be done on GPU without extra overhead

10. Online drafter calibration

Models tend to be overconfident in predicting the next token making it hard to get threshold for stopping drafting

But we can look at the runtime drafter performance and adjust (“calibrate”) thresholds on the fly

Putting it together

The magic of DeepSeek is in excellent system engineering with close model co-design. Many of these ideas were published before. It’s very impressive how they integrate them together to deliver huge e2e improvements with auto-adapting system

We do tons of speculative decoding research and adaptive system engineering at @FireworksAI_HQ, including continuously training and calibrating drafters for customer workloads.

Looking forward to building DSpark ideas and sharing better inference with the world.