Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

Hugging Face Daily Papers Papers
speculative-decoding large-language-models inference-speedup draft-tree pruning retrieval training-free

Summary

Graft is a training-free framework that enhances speculative decoding by combining pruning and retrieval to improve acceptance rates and inference speed, achieving up to 5.41x speedup on short-context benchmarks and up to 21.8% improvement over EAGLE-3 on Qwen3-235B.

Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41times speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.
Original Article
View Cached Full Text

Cached at: 05/20/26, 02:35 AM

Paper page - Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding

Source: https://huggingface.co/papers/2605.20104 Authors:

,

,

,

,

,

,

,

,

,

,

Abstract

Graft is a training-free framework that enhances speculative decoding by dynamically combining pruning and retrieval operations to improve acceptance rates and inference speed without sacrificing accuracy.

Speculative decoding(SD) accelerates large language model inference by leveraging adraft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansivedraft trees, which unfortunately incur severeVRAM bandwidthandcomputational overheads that bottleneck end-to-end speedups. Whiledynamic-depth pruningcan reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break thisPareto tradeoff, we introduce Graft, acompensation frameworkthat couplespruningandretrievalas mutually reinforcing operations.Pruningsupplies sufficient budget forretrieval, whileretrievalcompensates forpruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft’ mechanism, Graft attaches highly predictive retrieved tokens into positions opened bypruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41times speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to theDFlash-style block draftingparadigm, offering initial evidence and insights for extending grafting beyondautoregressive draft trees.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.20104

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.20104 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.20104 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.20104 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles