SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

Hugging Face Daily Papers Papers
speculative-decoding low-rank-parameterization llm-optimization model-efficiency inference-acceleration language-models draft-model

Summary

SlimSpec introduces a low-rank parameterization for drafter LM-heads to accelerate speculative decoding in LLMs, achieving 4-5x speedup while maintaining full vocabulary support.

Speculative decoding speeds up autoregressive generation in Large Language Models (LLMs) through a two-step procedure, where a lightweight draft model proposes tokens which the target model then verifies in a single forward pass. Although the drafter network is small in modern architectures, its LM-head still performs projection to a large vocabulary, becoming one of the major computational bottlenecks. In prior work this issue has been predominantly addressed via static or dynamic vocabulary truncation. Yet mitigating the bottleneck, these methods bring in extra complexity, such as special vocabulary curation, sophisticated inference-time logic or modifications of the training setup. In this paper, we propose SlimSpec, a low-rank parameterization of the drafter's LM-head that compresses the inner representation rather than the output, preserving full vocabulary support. We evaluate our method with EAGLE-3 drafter across three target models and diverse benchmarks in both latency- and throughput-bound inference regimes. SlimSpec achieves 4-5times acceleration over the standard LM-head architecture while maintaining a competitive acceptance length, surpassing existing methods by up to 8-9% of the end-to-end speedup. Our method requires minimal adjustments of training and inference pipelines. Combined with the aforementioned speedup improvements, it makes SlimSpec a strong alternative across wide variety of draft LM-head architectures.
Original Article
View Cached Full Text

Cached at: 05/12/26, 10:52 AM

Paper page - SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decoding

Source: https://huggingface.co/papers/2605.10453

Abstract

SlimSpec improves speculative decoding efficiency by using low-rank parameterization to compress the drafter’s language model head while maintaining full vocabulary support and achieving significant speedup with minimal pipeline changes.

Speculative decodingspeeds upautoregressive generationinLarge Language Models(LLMs) through a two-step procedure, where a lightweightdraft modelproposes tokens which thetarget modelthen verifies in a single forward pass. Although the drafter network is small in modern architectures, itsLM-headstill performs projection to a large vocabulary, becoming one of the major computational bottlenecks. In prior work this issue has been predominantly addressed via static or dynamicvocabulary truncation. Yet mitigating the bottleneck, these methods bring in extra complexity, such as special vocabulary curation, sophisticated inference-time logic or modifications of the training setup. In this paper, we propose SlimSpec, alow-rank parameterizationof the drafter’sLM-headthat compresses theinner representationrather than the output, preserving full vocabulary support. We evaluate our method with EAGLE-3 drafter across threetarget models and diverse benchmarks in both latency- and throughput-bound inference regimes. SlimSpec achieves 4-5times acceleration over the standardLM-headarchitecture while maintaining a competitive acceptance length, surpassing existing methods by up to 8-9% of theend-to-end speedup. Our method requires minimal adjustments of training and inference pipelines. Combined with the aforementioned speedup improvements, it makes SlimSpec a strong alternative across wide variety of draftLM-headarchitectures.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.10453

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.10453 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.10453 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.10453 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting

arXiv cs.CL

DeLS-Spec decouples long- and short-context modeling in speculative decoding by adding a lightweight local head to DFlash, achieving consistent speedups without full retraining. It requires only standard next-token prediction training for the local head and improves acceptance length on Qwen3 benchmarks.

What is Speculative Decoding? (trending on paperswithco.de) [R]

Reddit r/MachineLearning

Speculative decoding is an inference optimization technique that uses a fast draft model to propose future tokens verified in parallel by a larger model, improving LLM generation speed. The article highlights its trending status on Papers with Code and a recent SGLang blog post about state-of-the-art latencies using DFlash models.

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

arXiv cs.CL

This paper introduces SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference that uses a dynamically sparsified and recallable KV cache along with an entropy-based controller for speculation length, achieving up to 2.79x speedup over autoregressive decoding.