speculative-decoding

DogukanUrker demonstrates running Gemma 4 12B with full 262,144 context at ~100 tok/s on a single RTX 3060 using speculative decoding and KV cache splitting, achieving nearly full GPU utilization without CPU offload.

SpecLA proposes a speculative decoding runtime tailored for stateful linear-attention models, achieving up to 1.70x end-to-end speedup over autoregressive decoding on an NVIDIA H100 with a GDN-1.3B target.

This paper proposes SpecVocab, a method for selecting a per-step vocabulary subset for the draft model in speculative decoding, achieving higher acceptance length and up to 8.1% throughput improvement over EAGLE-3.

Comprehensive benchmarks of llama.cpp's speculative decoding methods on Qwen 3.6 27B show n-gram stacking on DFlash achieves up to 6x speedup on iterative coding tasks, with ngram-mod providing most of the gain and zero VRAM cost.

MiaAI Lab publishes a recipe to serve Google's Gemma 4 31B IT in NVFP4 quantization via vLLM, enabling 256k context, MTP speculative decoding, agentic reasoning, native tool calling, and image/video support.

MiaAI Lab released an upgraded recipe for serving DeepSeek V4 Flash on two DGX Spark nodes using vLLM with DSpark speculative decoding and NVFP4 KV-cache, achieving up to 153.7 tokens per second with six concurrent sessions.

This paper proposes EcoSpec, a cost-aware speculative decoding framework for Mixture-of-Experts models that accounts for expert activation costs during draft selection. It achieves up to 1.62× speedup on large-scale MoE models like DeepSeek-V3.1, Qwen3-235B-A22B, and GPT-OSS-120B by reducing expert footprint without modifying the target model's verification rule.

This paper studies PEFT-BD, a speculative decoding method using a LoRA-like adapter as a block-diffusion drafter, and finds that despite nontrivial accepted prefixes, it does not yield speedup because the drafter still requires a full-backbone pass, making it not compute-efficient.

A developer compares two inference stacks (production build vs SignalNine's q27) on the same Qwen model and finds they produce different honesty under pressure, with one fabricating progress and the other refusing appropriately, suggesting inference engines can affect model behavior beyond speed and quality metrics.

Spatially Speculative Decoding (SSD) accelerates autoregressive image models by predicting entire rows in parallel using small helper networks, achieving up to 13.28x speedup while maintaining benchmark performance.

Proposes Progressive Tree Drafting (PTD), a training-free, model-agnostic speculative decoding method that uses progressive tree structures and stepwise pruning to enable parallel generation of multiple draft paths, achieving up to 2x speedup on various benchmarks.

This pull request adds support for the Hy3 (hy_v3) model with MTP speculative decoding to llama.cpp, enabling efficient inference for this architecture.

AngelSlim/Hy3-GGUF is a toolkit for quantizing and deploying Hy3 large language models on llama.cpp, featuring MTP self-speculative decoding and a thinking/tool-call parser for efficient inference.

Poolside releases GGUF quantizations of the Laguna S 2.1 AI model, including a DFlash speculative decoding draft model, enabling efficient local inference with llama.cpp.

This paper introduces a self-supervised early exit method for LLMs, allowing computation to stop early at intermediate layers when confidence is high, thereby reducing inference cost. It also presents Dynamic Self-Speculative Decoding (DSSD) which achieves higher token acceptance than existing baselines.

A detailed breakdown of why uploading a large PDF to ChatGPT doesn't significantly slow response times, covering techniques like streaming, KV caching, and parallel prefill phase.

Explains the DSpark paper's improvements to speculative decoding for faster LLM inference, focusing on long draft generation and adaptive verification.

Reame is an LLM inference server built on llama.cpp that optimizes for CPU hardware by caching prompt prefixes and generated n-grams, becoming faster with repeated use. It is designed for cheap hardware like shared vCPUs and free tiers, targeting repetitive AI workloads such as document extraction and batch pipelines.

A new implementation of the fastest speculative decoding for Qwen models is available, with support in sglang via a Hugging Face fork and upcoming support in the uzu engine.

DominoTree introduces a training-free best-first draft tree for speculative decoding that uses conditional (non-factorized) correction from Domino to achieve up to 6.6x speedup over autoregressive decoding and the highest mean accept length across evaluated methods on Qwen3 models.