speculative-decoding

DeLS-Spec decouples long- and short-context modeling in speculative decoding by adding a lightweight local head to DFlash, achieving consistent speedups without full retraining. It requires only standard next-token prediction training for the local head and improves acceptance length on Qwen3 benchmarks.

This paper introduces Weaver, a lightweight autoregressive adapter that constructs proposal trees from factorized drafters' top-K marginals, achieving a 4.37x speedup over autoregressive decoding and outperforming the DFlash baseline by 24.7%.

Novita releases Kimi-K2.6 DSpark, a speculator model that uses speculative decoding to accelerate inference of the base Kimi-K2.6 model, achieving higher accepted token lengths.

A user benchmarks the newly merged DFlash speculative decoding method in llama.cpp on Qwen 3.6 27B, achieving up to 4.44x speedup at 36K context compared to baseline, with detailed leaderboard and quality tests.

mlx-dspark brings DeepSeek's DSpark and z-lab's DFlash speculative decoding drafters to Apple Silicon via MLX, enabling lossless speedup (~1.4–1.6×, up to 2× on code/math) and an OpenAI-compatible API for local inference.

A roofline model from the LLM Engineer's Almanac estimates speedups from speculative decoding for different draft lengths across models and hardware, with a note that it may underestimate benefits when overhead is significant.

DSpark is a speculative decoding framework that combines semi-autoregressive draft generation with confidence-scheduled verification to accelerate LLM inference and improve throughput in high-concurrency settings.

A new article by @junupark_ provides an end-to-end code walkthrough of nano-vLLM, an LLM inference engine, with plans to cover speculative decoding techniques like Eagle3 and dFlash.

A detailed recipe for running the unpruned GLM-5.2 model (744B parameters, 256 experts) across 4 NVIDIA DGX Spark nodes with 200K context, achieving up to 60.5 tok/s aggregate. Includes performance benchmarks, credits, and patches.

Bonsai-27B is a 1-bit binary transformer model that achieves full 27B-class reasoning on a phone (iPhone 17 Pro Max) with ~3.9 GB footprint and ~11 tok/s, retaining ~90% of FP16 intelligence.

Explores using Google's Gemma diffusion model as a speculative model for efficient large language model inference.

A bug in vLLM's speculative decoding configuration for GLM-5.2 NVFP4 on four DGX Sparks was fixed, resolving a performance tradeoff and achieving ~24 tok/s at 128K context with MTP4.

The paper introduces AUF (Accept-Until-Fail), a simple modification to the cross-entropy loss for masked block drafters in speculative decoding that restricts supervision to the prefix up to the first predicted failure, improving average emitted length across benchmarks without changing inference.

RedHatAI releases a preview DSpark speculator for GLM-5.2-FP8, the first DSpark draft model for a non-DeepSeek frontier model, achieving ~1.5× faster decode on 4×B300 via vLLM nightly. The checkpoint is a work-in-progress, with training details and acceptance metrics provided.

Introduces Multi-Token Residual Prediction (MRP), a technique that accelerates diffusion language model inference by predicting residuals between adjacent denoising steps, achieving up to 1.56x speedup in SGLang and recovering up to +16 accuracy points in aggressive decoding settings.

Introduced MTP speculative decoding to the Ornith 35B coding model in FP8 precision, achieving approximately 18% faster inference with minimal extra VRAM.

A detailed benchmark comparing speculative decoding engines for Qwen 3.6 27B on a single RTX 3090, showing ik_llama achieving ~100 tokens per second in code generation. Results include decode TPS, TTFT, VRAM usage, and context degradation across 5 engine variants.

BlockPilot proposes an instance-adaptive policy that predicts the optimal block size for diffusion-based speculative decoding, achieving significant speedup with minimal overhead.

DeepSeek open-sourced DSpark, an MIT-licensed framework using speculative decoding to accelerate LLM inference by up to 85%, with support for multiple model families including its own DeepSeek-V4, Alibaba's Qwen, and Google's Gemma.

DeepSeek proposes the DSpark technique, which implements speculative decoding by inserting a mini Transformer after the Final RMSNorm, boosting large model output speed by 60%-85%.