Tag
A detailed benchmark comparing speculative decoding engines for Qwen 3.6 27B on a single RTX 3090, showing ik_llama achieving ~100 tokens per second in code generation. Results include decode TPS, TTFT, VRAM usage, and context degradation across 5 engine variants.
BlockPilot proposes an instance-adaptive policy that predicts the optimal block size for diffusion-based speculative decoding, achieving significant speedup with minimal overhead.
DeepSeek open-sourced DSpark, an MIT-licensed framework using speculative decoding to accelerate LLM inference by up to 85%, with support for multiple model families including its own DeepSeek-V4, Alibaba's Qwen, and Google's Gemma.
DeepSeek proposes the DSpark technique, which implements speculative decoding by inserting a mini Transformer after the Final RMSNorm, boosting large model output speed by 60%-85%.
Sharing a new recipe for dflash speculative decoding in llama.cpp, achieving ~70 TPS on a single RTX 3090 using Qwen3.6-27B GGUF with a draft model.
Ornith 35B shows 30-40% token generation speedup when paired with Qwen3.6 35B DFlash speculative model in llama-server, achieving 80% acceptance rate on mixed code and text, though prompt processing suffers.
Yong Quan highlights that better speculative decoders can unlock near-linear throughput gains in LLM inference, as presented at a Modal workshop by Charles.
Proposes EntMTP, a training-free scheduler that adapts tree-based attention topologies for speculative decoding based on local entropy estimates, achieving 1.09-1.15x speedup over Hydra and up to 1.36x over Medusa.
Google Research introduces a new architecture using frozen Multi-Token Prediction to accelerate Gemini Nano models on Pixel devices, significantly improving speed and energy efficiency for on-device AI features.
Progress update on DSpark: training of DFlash backbone and markov head is complete, enabling use on 27B. Next is training the confidence head for adaptive drafting, expected 8-14% speed improvement over DFlash.
An update on the Ornith-1.0-35B GGUF model introduces a native MTP speculative-decode graft for faster inference on a single GPU, achieving ~1.3-1.35x decode speedup while maintaining near-identical token distribution. Benchmark numbers for throughput, TTFT, and long-context performance across multiple quants are provided.
OpenInfer, a pure Rust+CUDA LLM inference engine, quickly added support for DeepSeek's DSpark speculative decoding technique on RTX 5090, achieving nearly 500 tok/s per user and scaling to ~2.4K aggregate tok/s, outperforming DFlash on non-random workloads.
DeepSeek AI released the DeepSpec collection on Hugging Face, featuring speculative decoding models (dspark, dflash, eagle3) based on Qwen3 and Gemma4 in various sizes (1B-3B).
DSpark from DeepSeek AI integrates speculative decoding ideas to achieve 1.5x to 5x higher throughput in production systems. This thread explains 10 key ideas from the basics.
DeepSeek has open-sourced DeepSpec, a full-stack codebase for training and evaluating speculative decoding models.
DeepSeek released a paper and MIT-licensed open-source implementation of speculative decoding (DSpark) that speeds up LLM responses by up to 80% by using a small 'guess' model and a large 'check' model, achieving both speed and accuracy without tradeoffs.
DeepSeek open-sourced DeepSpec, a full-stack codebase for training and evaluating draft models for speculative decoding, enabling 60-85% faster generation. It includes data preparation, training, and evaluation scripts with support for multiple draft model algorithms (DSpark, DFlash, Eagle3).
DeepSeek released DSpark, a speculative decoding method that boosts throughput by 51% to 400% for V4 Flash & Pro, along with the open-source DeepSpec codebase for training and evaluating draft models.
DeepSeek releases preview versions of its V4 series, including DeepSeek-V4-Pro (1.6T parameters, 49B activated) and DeepSeek-V4-Flash (284B parameters, 13B activated), both supporting a one-million-token context and featuring hybrid attention, manifold-constrained hyper-connections, and a Muon optimizer.
An interactive guide explaining speculative decoding and multi-token prediction in LLMs, covering techniques from rejection sampling to MTP used in Qwen 3.6 and Gemma 4, with live diagrams and sliders.