VIA-SD: Verification via Intra-Model Routing for Speculative Decoding
Summary
VIA-SD introduces a multi-tier speculative decoding framework using intra-model routing to reduce verification costs, achieving significant speedups over traditional approaches.
View Cached Full Text
Cached at: 06/12/26, 10:52 AM
Paper page - VIA-SD: Verification via Intra-Model Routing for Speculative Decoding
Source: https://huggingface.co/papers/2606.12243
Abstract
VIA-SD introduces a multi-tier speculative decoding framework that uses intra-model routing to reduce verification costs by employing slim submodels for medium-confidence token validation, achieving significant speedups over traditional approaches.
Speculative decoding(SD) addresses the high inference costs of LLMs by having lightweightdraftersgenerate candidates for largeverifiersto validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier viaintra-model routing, instead of the full verifier. This motivates ourslim-verifierto handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification viaIntra-Model RoutingforSpeculative Decoding(VIA-SD), amulti-tier frameworkusing a routedslim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases,slim-verifierregeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20%speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/
View arXiv pageView PDFProject pageAdd to collection
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.12243 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.12243 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.12243 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
What is Speculative Decoding? (trending on paperswithco.de) [R]
Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.
Speculative Decoding for Autoregressive Video Generation
SDVG adapts speculative decoding to autoregressive video diffusion, using an image-quality router to achieve up to 2.09× speed-up with 95.7% quality retention on MovieGenVideoBench.
Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
Dustin introduces a sparse verification framework for speculative decoding that leverages draft model signals and sparse attention head scoring to overcome the KV cache verification bottleneck, achieving up to 27.85x speedup in self-attention and 9.17x end-to-end decoding speedup on long-context tasks with negligible accuracy loss.
BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
BudgetDraft proposes a multi-view training method for speculative decoding that aligns a sparse-KV drafter with a full-KV verifier, achieving significant speedups for mid-to-long context inference.
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
AdaPLD is a training-free method that improves model-free speculative decoding by using adaptive retrieval combining lexical and semantic similarity, and constructing branched reuse hypotheses to handle continuation uncertainty, achieving up to 3.10x decoding speedup.