A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

# A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Source: [https://arxiv.org/html/2607.27735](https://arxiv.org/html/2607.27735)
Yuesong Liu1\\equalcontrib, Yuan Zeng1\\equalcontrib, Min Lyu1\\corresponding, Ruilin Liu1, Yu Guo1, Yinlong Xu1

###### Abstract

Speculative decoding alleviates the memory\-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length\. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost\. Guided by this analysis, we introduceSparseSpec\-L, a training\-free self\-speculative decoding framework for long\-context inference\. SparseSpec\-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache\. It recycles per\-head attention statistics produced during full\-context verification as a no\-extra\-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache\. An online entropy\-based controller further selects the speculation length according to expected step\-wise efficiency\. Experiments across multiple long\-context tasks and model scales show consistent end\-to\-end acceleration, with up to2\.79×2\.79\\timesspeedup over autoregressive decoding while preserving the target model’s output distribution\.

## Introduction

![Refer to caption](https://arxiv.org/html/2607.27735v1/x1.png)Figure 1:Comparison of existing speculative decoding paradigms \(α\\alphais the token acceptance rate,γ\\gammadenotes drafting overhead,kkrepresents speculation length, andCtC\_\{t\}indicates retraining cost\)\. Auxiliary models suffer from structural gaps that depress acceptance rates; Medusa fixes speculation length k at the architecture level; EAGLE\-3 exhibits acceptance rate decay at large k; static compression permanently discards context, causing information loss\. Empirical evidence for each baseline’s limitations is provided in:†Tab\.[1](https://arxiv.org/html/2607.27735#Sx3.T1),§Appendix, and‡Fig\.[2](https://arxiv.org/html/2607.27735#Sx2.F2)\.Large language models \(LLMs\) demonstrate strong capabilities across generation and reasoning tasks, but their autoregressive decoding process imposes a substantial inference\-latency bottleneck\. Specifically, predicting each subsequent token requires loading the entire key\-value \(KV\) cache of the preceding context into the processor, making decoding heavily memory\-bandwidth bound\(Vaswaniet al\.[2017](https://arxiv.org/html/2607.27735#bib.bib16); Kwonet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib18)\)\. This challenge is further exacerbated in long\-context scenarios, where linear memory growth and quadratic attention complexity\(Dao[2023](https://arxiv.org/html/2607.27735#bib.bib40)\)severely degrade real\-time responsiveness\.

To mitigate this inefficiency, speculative decoding has emerged as a promising paradigm to break this bottleneck\. By decoupling generation into a computationally cheap drafting stage and a parallel verification stage, it iteratively proposes and simultaneously verifies multiple candidate tokens\. This effectively unlocks near\-multiplicative speedups while mathematically guaranteeing the exact target distribution\(Xiaet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib41)\)\.

Despite its theoretical elegance, fully realizing the acceleration potential of speculative decoding remains a practical challenge\. As categorized in Figure[1](https://arxiv.org/html/2607.27735#Sx1.F1), representative approaches expose four common limitations: \(1\)Auxiliary Models\(Chenet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib19); Leviathanet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib20); Miaoet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib21)\)rely on an independent small drafter, but the inherent structural gap between the drafter and target model fundamentally depresses the acceptance rate\. \(2\)Prediction Headslike Medusa\(Caiet al\.[2024a](https://arxiv.org/html/2607.27735#bib.bib32)\)mitigate this gap by attaching drafting heads directly to the target model, but their speculation lengthkkis rigidly fixed at the architecture level, placing a hard mathematical ceiling on achievable speedups\. \(3\)Lightweight Layerslike Eagle\-3\(Liet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib30)\)enable flexible speculation lengths, but their auto\-regressive approximation suffers a steep drop in acceptance rate for tail tokens, severely decaying performance at largekkand exhibiting extreme vulnerability to dataset dependency\. \(4\)Static Compressionmethods\(Sadhukhanet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib35); Chenet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib34)\)offer a training\-free alternative by aggressively discarding historical context to minimize drafting overhead\. However, this irreversible pruning permanently destroys context fidelity, causing persistent information loss and low acceptance rates in long\-context scenarios\. These representative paradigms expose a recurring trade\-off among drafting overhead, speculation length, and acceptance rate\. Although different methods occupy different operating points, none of the evaluated approaches simultaneously maintains low draft cost, robust long\-context acceptance, and an adaptive speculation horizon\.

To address these limitations, we introduceSparseSpec\-L, a training\-free self\-speculative decoding framework for long\-context inference\. SparseSpec\-L uses the target model itself as both drafter and verifier, avoiding the structural mismatch of an auxiliary draft model\. During drafting, the target model accesses a dynamically sparsified KV cache, while verification retains the complete KV cache and therefore preserves the target distribution\. Instead of permanently evicting historical context, SparseSpec\-L recycles per\-head attention statistics produced by the previous full\-context verification pass to reconstruct a recallable sparse index for the next drafting round\. This requires no additional model forward pass, although the current non\-fused attention implementation still incurs kernel\-level overhead\. An online entropy\-based controller further adjusts the speculation length according to real\-time drafting confidence and measured drafting and verification costs\.

Sparse\-context target\-model execution has also been explored by TriForce\(Sunet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib53)\), prior SparseSpec\(Zhaoet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib54)\), and Vegas\(Yueet al\.[2026](https://arxiv.org/html/2607.27735#bib.bib55)\)\. SparseSpec\-L shares their sparse\-to\-full principle but adopts a single\-model, single\-verification\-stage pipeline, refreshes per\-head indices from verification attention statistics, and selects the speculation horizon by expected step\-wise efficiency\. We therefore position it as an efficiency\-oriented design for long\-input workloads, rather than the first sparse\-KV speculative decoding framework\.

Our contributions are threefold\.\(1\)Efficiency analysis\.We formulate speculative decoding speedup as a joint function of drafting cost, prefix acceptance, and speculation length, and characterize the efficiency inversion that occurs when extending the drafting horizon no longer amortizes its marginal cost\. \(2\)Recallable sparse\-context self\-speculation\.We develop a training\-free, single\-model sparse\-to\-full decoding pipeline that refreshes per\-head sparse KV indices using attention statistics from the preceding full\-context verification pass, without permanently discarding historical KV states\. \(3\)Cost\-aware adaptive speculation\.We estimate token\-level acceptance from online entropy statistics and select the speculation length that maximizes expected step\-wise efficiency\. Experiments across long\-context retrieval, reasoning, and synthesis tasks demonstrate consistent end\-to\-end acceleration of up to2\.79×2\.79\\times\.

## Rethinking Speculative Decoding

To fundamentally understand the limitations of current acceleration algorithms and motivate the design of SparseSpec\-L, we first abstract speculative decoding into a generalized framework\. By quantifying the efficiency bottlenecks within this process, we can outline the exact blueprint required for an optimal solution\.

### The Draft\-Verification Paradigm

Speculative decoding accelerates auto\-regressive generation\(Brownet al\.[2020](https://arxiv.org/html/2607.27735#bib.bib44)\)by decoupling the sequential process into a two\-stage pipeline: \(1\)Drafting stage\.A lightweight approximation mechanism generateskkcandidate tokens,𝒳draft\\mathcal\{X\}\_\{\\text\{draft\}\}= \{x^n\+1,…,x^n\+k\\hat\{x\}\_\{n\+1\},\\ldots,\\hat\{x\}\_\{n\+k\}\}, aiming to mimic the target model’s distribution with minimal compute\. \(2\)Verification stage\.The full target model evaluates𝒳draft\\mathcal\{X\}\_\{\\text\{draft\}\}in a single parallel forward pass\. It identifies the longest prefix that satisfies the verification criteria:𝒳prefix=\{x^n\+1,…,x^n\+m\}\\mathcal\{X\}\_\{\\text\{prefix\}\}=\\\{\\hat\{x\}\_\{n\+1\},\\ldots,\\hat\{x\}\_\{n\+m\}\\\}\. The system then commits thesemmtokens and generates an additional corrected tokenxn\+m\+1x\_\{n\+m\+1\}\. Consequently, each step producesm\+1m\+1tokens with only a single target model forward pass\. This framework ensures mathematical equivalence to standard decoding while potentially providing multi\-token speedup per step\(Chenet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib19); Leviathanet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib20)\)\.

### Unified Speedup Analysis

To quantify the performance of speculative decoding, we define the speedup ratioSSas the throughput relative to standard decoding\. Letkkdenote the speculation length, which represents the number of candidate tokens generated by the drafting component in each step\. Letα∈\[0,1\]\\alpha\\in\[0,1\]be the expected acceptance rate, representing the expected proportion of draft tokens accepted per iteration \(α=𝐄​\[m/k\]\\alpha=\\mathbf\{E\}\[m/k\]\)\. On average, the system producesα​k\+1\\alpha k\+1tokens in a single step\.

The time cost of a parallel verification pass forkktokens is approximately equivalent to that of a standard step, both of which we denote asCvC\_\{v\}\. LetCdC\_\{d\}be the per\-token cost of the drafting component\. The speedupSScan be formulated as:

S=\(α​k\+1\)​Cvk​Cd\+Cv=α​k\+1k​\(Cd/Cv\)\+1S=\\frac\{\(\\alpha k\+1\)C\_\{v\}\}\{kC\_\{d\}\+C\_\{v\}\}=\\frac\{\\alpha k\+1\}\{k\(C\_\{d\}/C\_\{v\}\)\+1\}\(1\)
To understand the fundamental impact of the speculation lengthkk, we first define the relative drafting overhead asγ=Cd/Cv\\gamma=C\_\{d\}/C\_\{v\}\. In an idealized scenario where the acceptance rateα\\alphais treated as a constant, the impact ofkkon the speedup ratioSScan be quantified by its partial derivative:

∂S∂k=α−γ\(k​γ\+1\)2\\frac\{\\partial S\}\{\\partial k\}=\\frac\{\\alpha\-\\gamma\}\{\(k\\gamma\+1\)^\{2\}\}\(2\)
Equation[2](https://arxiv.org/html/2607.27735#Sx2.E2)demonstrates that the benefit of extending the speculation length depends fundamentally on the magnitude ofα\\alpharelative toγ\\gamma\. Ifα\>γ\\alpha\>\\gamma, the speedupSSis a monotonically increasing function ofkk, suggesting that a longer speculation length always yields higher efficiency\. Conversely, ifα<γ\\alpha<\\gamma, the derivative∂S∂k\\frac\{\\partial S\}\{\\partial k\}remains negative, indicating that the drafting process is too costly relative to its accuracy and any speculation will degrade performance compared to standard decoding\.

![Refer to caption](https://arxiv.org/html/2607.27735v1/x2.png)Figure 2:Performance analysis of Eagle\-3 and MagicDec\(SnapKV\) with variable speculation steps\. For each method, theleft y\-axisdenotes token acceptance \(hit\) rate and theright y\-axisdenotes speedup over auto\-regressive decoding\. Results are reported on LongBench v2 and XSum with Llama3\.1\-8B\-Instruct\.While the constantα\\alphacase provides a theoretical threshold, in practice, the expected acceptance rateα\\alphaconsistently decays askkscales\. As the speculation length increases, the compounding uncertainty of auto\-regressive generation causes the acceptance rate for tail tokens to decay\. Eventually, this triggers anefficiency inversion, a tipping point where the marginal cost of drafting an additional token eclipses its expected contribution to the speedup \(αtail<γ\\alpha\_\{\\text\{tail\}\}<\\gamma\)\. Consequently, fixingkkacross different contexts is inherently sub\-optimal\.

This theoretical inversion and the limitations of existing paradigms are empirically verified in Figure[2](https://arxiv.org/html/2607.27735#Sx2.F2)\. First, both training\-based models and training\-free static compression methods exhibit a severe speedup crater askkincreases, directly caused by the steep decay ofα\\alpha\. For instance, on LongBench v2, the speedup of MagicDec early peaks at1\.74×1\.74\\timeswhenk=2k=2\. However, pushing the speculation length tok=16k=16triggers a catastrophic decline inα\\alphadown to6\.3%6\.3\\%, ultimately degrading the overall speedup to a sub\-baseline0\.36×0\.36\\times\. Similarly, Eagle\-3 sees its speedup regress from2\.24×2\.24\\timesto1\.83×1\.83\\timeson the XSum dataset under the same length extension\. Second, Figure[2](https://arxiv.org/html/2607.27735#Sx2.F2)exposes the critical dataset dependency of training\-based methods\. While Eagle\-3 achieves a robust acceptance rate of approximately67\.2%67\.2\\%on the XSum dataset \(commonly used in its original evaluation\), its performance collapses drastically to a mere8\.4%8\.4\\%when shifted to the long\-context LongBench scenario\.

Therefore, maximizing acceleration is fundamentally a multidimensional optimization problem\. As summarized in Figure[1](https://arxiv.org/html/2607.27735#Sx1.F1), an ideal framework must simultaneously guarantee an acceptable drafting costγ\\gamma, a robust acceptance rateα\\alpha, and an optimal speculation lengthkk\. However, existing paradigms fail to achieve this\. As demonstrated empirically, simply forcing a longer speculation length to chase higher speedups leads to a steep decline inα\\alpha, ultimately causing an efficiency inversion where actual acceleration drops\. This acceleration bottleneck necessitates SparseSpec\-L\. Our approach pushes the practical speedup ceiling forward through two mechanisms: first, recallable self\-speculation intrinsically guarantees a high and generalizableα\\alpha; second, an entropy\-guided adaptive policy dynamically adjustskkto constantly maintain generation within the peak efficiency zone, effectively mitigating the efficiency inversion\.

## Methodology

![Refer to caption](https://arxiv.org/html/2607.27735v1/x3.png)Figure 3:Overview of SparseSpec\-L’s sparse\-to\-full self\-speculative decoding pipeline\.Left: the recallable sparse KV\-cache index is constructed per attention head by retaining sink, important historical, and recent tokens, guided by attention scores from the previous verification pass\.Right: draft tokens are generated via sparse attention, then verified in a single parallel full\-attention forward pass; accepted tokens are committed, attention weights refresh the sparse index, and the adaptive module selects the optimalk∗k^\{\*\}based on per\-token draft entropy\.Guided by the unified speedup analysis above, an ideal speculative decoding framework must concurrently maintain an acceptable drafting overheadγ\\gamma, guarantee a robust and generalizable acceptance rateα\\alpha, and dynamically calibrate the speculation lengthkk\. To fulfill these criteria, we proposeSparseSpec\-L, a training\-free, self\-speculative decoding framework\. It elegantly resolves these bottlenecks through two complementary mechanisms: recallable sparse\-attention drafting to maximizeα\\alphawhile boundingγ\\gamma, and entropy\-guided adaptive speculation to dynamically optimizekk\.

### Overview of SparseSpec\-L

SparseSpec\-L fundamentally adopts a sparse\-to\-full self\-speculative framework, wherein a single target model acts as both the drafter and the verifier, inherently eliminating any structural mismatch\. Figure[3](https://arxiv.org/html/2607.27735#Sx3.F3)illustrates the two\-stage workflow executed at every decoding iteration\.

Sparse Drafting Stage\.The target model autoregressively generateskkspeculative tokens using a recallable sparse KV cache\. Instead of static pruning, the retained KV positions are dynamically guided by importance scores inherited from the previous verification pass, ensuring the drafter attends only to the most context\-critical subset of history\.

Full Verification Stage\.Thekkdraft tokens are verified in one parallel forward pass using the complete KV cache\. In our current implementation, verification uses dense attention that materializes per\-head attention statistics\. SparseSpec\-L recycles these statistics to refresh token importance for the next drafting round\. This introduces no additional model forward pass, but the use of non\-fused dense attention incurs implementation overhead relative to an optimized FlashAttention\-based verifier\.

We next describe the recallable sparse\-index construction and the entropy\-guided policy for adaptive speculation lengths\.

### Sparse Attention Drafting

Attention\-Based Importance Estimation\.We estimate the importance of historical tokens from the query–key attention statistics materialized during full\-context verification\. The statistics are reused rather than recomputed through an additional forward pass\. Importance is maintained independently for every layerlland attention headhh, allowing the sparse draft cache to preserve heterogeneous head\-specific retrieval patterns\(Michelet al\.[2019](https://arxiv.org/html/2607.27735#bib.bib45)\)\.

For each token positioni∈\{1,…,T\}i\\in\\\{1,\\ldots,T\\\}, its importance scoreSl,h\(i\)S^\{\(i\)\}\_\{l,h\}is calculated by aggregating the attention it received from the most recentWWquery positions within that specific head:

Sl,h\(i\)=∑q=T−W\+1Tal,h\(q→i\)S^\{\(i\)\}\_\{l,h\}=\\sum\_\{q=T\-W\+1\}^\{T\}a^\{\(q\\to i\)\}\_\{l,h\}\(3\)whereal,h\(q→i\)a^\{\(q\\to i\)\}\_\{l,h\}denotes the attention weight from queryqqto keyii, and the sliding windowWWeffectively controls the recency bias of the signal\.

Recallable Index Construction\.Based onSl,h\(i\)S^\{\(i\)\}\_\{l,h\}, we independently construct a sparse index setℐl,h\\mathcal\{I\}\_\{l,h\}for each head\. To ensure comprehensive context fidelity under a total budgetKtotalK\_\{\\text\{total\}\}, each index set retains three complementary groups of KV positions: \(1\)Sink tokens:the firstKsinkK\_\{\\text\{sink\}\}positions, encoding essential global context\(Xiaoet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib25)\); \(2\)Recent tokens:the most recentKrecentK\_\{\\text\{recent\}\}positions, supplying indispensable local context for autoregression; and \(3\)Important historical tokens:the topKmidK\_\{\\text\{mid\}\}positions ranked bySl,h\(i\)S^\{\(i\)\}\_\{l,h\}, drawn exclusively from the remaining non\-sink, non\-recent span to capture critical long\-range dependencies\. Formally, the retained index set is defined as:

ℐl,h=\{1,…,Ksink\}∪TopKKmid​\(Sl,h\)∪\{T−Krecent\+1,…,T\}\\begin\{split\}\\mathcal\{I\}\_\{l,h\}=&\\\{1,\\ldots,K\_\{\\text\{sink\}\}\\\}\\cup\\text\{TopK\}\_\{K\_\{\\text\{mid\}\}\}\(S\_\{l,h\}\)\\\\ &\\cup\\\{T\-K\_\{\\text\{recent\}\}\+1,\\ldots,T\\\}\\end\{split\}\(4\)whereTopKexcludes sink and recent positions\. The sparse KV cache used for drafting is then instantiated via a low\-overhead index gathering:

K~l,h=Gather​\(Kl,h,ℐl,h\),V~l,h=Gather​\(Vl,h,ℐl,h\)\\begin\{split\}\\tilde\{K\}\_\{l,h\}&=\\text\{Gather\}\(K\_\{l,h\},\\mathcal\{I\}\_\{l,h\}\),\\\\ \\tilde\{V\}\_\{l,h\}&=\\text\{Gather\}\(V\_\{l,h\},\\mathcal\{I\}\_\{l,h\}\)\\end\{split\}\(5\)Because the original dense KV tensors remain resident, the sparse indexℐl,h\\mathcal\{I\}\_\{l,h\}can be reconstructed after every verification step without irreversible KV eviction\.

Hyperparameter Selection\.The total budgetKtotal=Ksink\+Kmid\+KrecentK\_\{\\text\{total\}\}=K\_\{\\text\{sink\}\}\+K\_\{\\text\{mid\}\}\+K\_\{\\text\{recent\}\}directly dictates the drafting compression ratio\. In our implementation,KsinkK\_\{\\text\{sink\}\}andKrecentK\_\{\\text\{recent\}\}follow established empirical constants that demonstrate robustness across model families\(Xiaoet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib25); Liet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib26)\), whileKmidK\_\{\\text\{mid\}\}dynamically fills the remaining budget\. The aggregation window is fixed atW=16W=16, which empirically yields stable importance signals across all evaluations\.

### Adaptive Speculation Length

To avoid the efficiency inversion caused by a rigid speculation length, SparseSpec\-L employs a lightweight, training\-free policy that dynamically modulateskkbased on the drafter’s real\-time confidence\.

Entropy\-Based Acceptance Prediction\.During the sparse drafting stage, we actively record the output entropyHiH\_\{i\}of each generated candidate token\. Following the subsequent verification step, every drafted token is categorically labeled as accepted or rejected\. This allows us to track the empirical mean entropy of the accepted class \(H¯acc\\bar\{H\}\_\{\\text\{acc\}\}\) and the rejected class \(H¯rej\\bar\{H\}\_\{\\text\{rej\}\}\) via exponential moving averages with a smoothing coefficientβ\\beta:

H¯acc←β​H¯acc\+\(1−β\)​Hacc,H¯rej←β​H¯rej\+\(1−β\)​Hrej\\begin\{split\}\\bar\{H\}\_\{\\text\{acc\}\}&\\leftarrow\\beta\\bar\{H\}\_\{\\text\{acc\}\}\+\(1\-\\beta\)H\_\{\\text\{acc\}\},\\\\ \\bar\{H\}\_\{\\text\{rej\}\}&\\leftarrow\\beta\\bar\{H\}\_\{\\text\{rej\}\}\+\(1\-\\beta\)H\_\{\\text\{rej\}\}\\end\{split\}\(6\)
For any newly drafted tokeniiwith entropyHiH\_\{i\}, we estimate its soft acceptance probabilitypip\_\{i\}by evaluating its proximity to these tracked class centers\. We apply a normalized softmax over their negative L1 distances:

pi=e−\|Hi−H¯acc\|e−\|Hi−H¯acc\|\+e−\|Hi−H¯rej\|p\_\{i\}=\\frac\{e^\{\-\|H\_\{i\}\-\\bar\{H\}\_\{\\text\{acc\}\}\|\}\}\{e^\{\-\|H\_\{i\}\-\\bar\{H\}\_\{\\text\{acc\}\}\|\}\+e^\{\-\|H\_\{i\}\-\\bar\{H\}\_\{\\text\{rej\}\}\|\}\}\(7\)
Optimal Length Selection\.Because speculative decoding enforces a strict prefix\-matching criterion, the probability that the firstmmdrafted tokens are consecutively accepted is given by the joint probability∏i=1mpi\\prod\_\{i=1\}^\{m\}p\_\{i\}, assuming conditional independence\. Therefore, for a given speculation lengthkk, the expected number of accepted draft tokens is the sum of these prefix probabilities:∑m=1k∏i=1mpi\\sum\_\{m=1\}^\{k\}\\prod\_\{i=1\}^\{m\}p\_\{i\}\.

Table 1:End\-to\-end results on LongBench v2 and∞\\infty\-Bench \(α\\alpha: acceptance rate, Aver\.kk: average accepted tokens, T: throughput \(tokens/s\), S: speedup over autoregressive decoding\)\. Auxiliary Model is not applicable to Qwen2\.5\-7B\-Instruct due to vocabulary mismatch\. Eagle\-3 uses the official released checkpoint which lacks Qwen support\.By adding the single correction token guaranteed during the verification pass and substituting this expected total length back into our unified speedup formulation \(Eq\.[1](https://arxiv.org/html/2607.27735#Sx2.E1)\), we dynamically select the optimal speculation lengthk∗k^\{\*\}from a pre\-defined candidate set𝒦\\mathcal\{K\}that strictly maximizes the expected step\-wise efficiency:

k∗=arg⁡maxk∈𝒦⁡1\+∑m=1k∏i=1mpik​Cd\+Cvk^\{\*\}=\\arg\\max\_\{k\\in\\mathcal\{K\}\}\\frac\{1\+\\sum\_\{m=1\}^\{k\}\\prod\_\{i=1\}^\{m\}p\_\{i\}\}\{kC\_\{d\}\+C\_\{v\}\}\(8\)whereCdC\_\{d\}andCvC\_\{v\}represent the drafting and verification cost, respectively\.

This dynamic modulation acts as an online feedback controller\. When generation is confident \(low entropy, yielding highpip\_\{i\}\), the policy aggressively extendsk∗k^\{\*\}to harvest more accepted tokens per step\. Conversely, when uncertainty spikes, the policy preemptively contractsk∗k^\{\*\}to avoid costly rejections\. Consequently, SparseSpec\-L seamlessly transforms the notorious efficiency inversion from a hard bottleneck into a dynamically managed operating point, optimizing acceleration throughout the decoding process\.

## Experiments

### Experimental Setup

#### Models & Datasets\.

Our primary controlled comparison uses Llama\-3\.1\-8B\-Instruct \(L3\) and Qwen2\.5\-7B\-Instruct \(Q2\) on LongBench v2\(Baiet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib36)\)and InfiniteBench\(Zhanget al\.[2024](https://arxiv.org/html/2607.27735#bib.bib38)\)\. To evaluate generalization across model scales and task types, we additionally evaluate Llama\-3\.2\-3B\-Instruct and Qwen2\.5\-14B\-Instruct on LongBench v2, RULER\(Hsiehet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib59)\)QA1, RULER NIAH, and InfiniteBench\. These workloads cover multi\-task reasoning and synthesis, long\-context question answering, and needle retrieval\.

#### Baselines\.

We compare against an auxiliary drafter\(Miaoet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib21)\), EAGLE\-3\(Liet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib30)\), LayerSkip, MagicDec with StreamingLLM or SnapKV\(Sadhukhanet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib35); Xiaoet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib25); Liet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib26)\), and RAPID\(Chenet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib34)\)\. We use official checkpoints and recommended configurations when available, and report each fixed\-length baseline under its best\-performing speculation length\. Medusa\(Caiet al\.[2024a](https://arxiv.org/html/2607.27735#bib.bib32)\)is omitted from the main table because no aligned checkpoint is available for our target models; an evaluation using its released Vicuna\-7B checkpoint is provided in the supplementary material\.

#### Implementation Details\.

All primary experiments are conducted on a single NVIDIA A40 GPU with 48 GB memory using Hugging Face Transformers\. Prefill is identical for SparseSpec\-L and the autoregressive baseline; SparseSpec\-L modifies only the decoding phase and therefore does not change time to first token under the same prefill implementation\. To materialize per\-head attention statistics, the current verifier uses dense non\-FlashAttention kernels\. These statistics require no additional forward pass, but the non\-fused implementation increases verification latency relative to an optimized autoregressive attention kernel\. The sparse\-index update costs 41\.1 ms per decoding iteration in the primary 60K setting, while the entropy controller contributes less than 0\.5% of total step latency\. Unless otherwise specified, the context length is capped at 64K and the generation length at 128 tokens\.

### Main Results

Table[1](https://arxiv.org/html/2607.27735#Sx3.T1)reports the performance of all baselines on L3 and Q2 across two benchmarks\. Note that while SparseSpec\-L utilizes an adaptive speculation length, all other baselines are reported under their optimal fixed speculation length that yields the highest speedup\. Due to hardware memory constraints, we strictly cap the maximum context length at 64K tokens and the maximum generation length at 128 tokens across all evaluations\.

Generalization across models and tasks\.Table[2](https://arxiv.org/html/2607.27735#Sx4.T2)extends the evaluation beyond the two primary configurations\. SparseSpec\-L provides1\.55×1\.55\\times–2\.79×2\.79\\timesspeedup across 3B–14B models and across multi\-task reasoning, question answering, needle retrieval, and long\-context synthesis\. Acceptance remains between 72\.9% and 84\.6% in these settings\. These results indicate that the acceleration is not restricted to a single model size or benchmark\.

Comparison with layer\-skipping self\-speculation\.We additionally evaluate LayerSkip using its official configuration\. It achieves0\.80×0\.80\\timesspeedup with 41\.4% acceptance on normal\-length inputs and0\.77×0\.77\\timeswith 38\.4% acceptance on the 60K LongBench setting\. Under this evaluated configuration, the limited number of accepted draft tokens is insufficient to amortize the verification cost\. We report the complete setup and throughput results in the supplementary material\.

Batch scaling\.At 16K context, SparseSpec\-L achieves1\.10×1\.10\\times,1\.52×1\.52\\times, and2\.45×2\.45\\timesspeedup at batch sizes 1, 2, and 4, respectively, compared with0\.91×0\.91\\times,1\.24×1\.24\\times, and1\.92×1\.92\\timesfor MagicDec\. Its relative throughput advantage therefore increases from1\.21×1\.21\\timesto1\.28×1\.28\\times\. Full throughput, acceptance, and memory statistics are provided in the supplementary material\.

Table 2:Cross\-model and cross\-task generalization of SparseSpec\-L\. Speedup is measured against autoregressive decoding under the same hardware and sequence configuration\.Table 3:Ablation on speculation length\.Primary comparison\.SparseSpec\-L achieves2\.79×2\.79\\times/2\.60×2\.60\\timesspeedup on LongBench v2/InfiniteBench with L3, and1\.95×1\.95\\times/1\.86×1\.86\\timeswith Q2\. On LongBench v2, it combines72\.9%72\.9\\%acceptance with 11\.26 accepted draft tokens per iteration\. In comparison, MagicDec–SnapKV reaches80\.1%80\.1\\%acceptance only under the short fixed horizonk=2k=2, yielding 1\.59 accepted draft tokens and1\.74×1\.74\\timesspeedup\.

The auxiliary drafter attains a lower acceptance rate than SparseSpec\-L in the same setting \(50\.3% versus 72\.9%\)\. The official EAGLE\-3 checkpoint performs well on its in\-domain XSum setting but transfers poorly to 60K LongBench v2, where acceptance falls below 3%\. SparseSpec\-L also maintains1\.95×1\.95\\timesacceleration on Q2 without model\-specific training\.

![Refer to caption](https://arxiv.org/html/2607.27735v1/x4.png)Figure 4:Sensitivity analysis on LongBench v2 with Llama\-3\.1\-8B\-Instruct model\. The performance \(Top: Speedup; Bottom: Acceptance rate\) are evaluated against two critical dimensions: context length \(Left\) and KV\-cache compression ratio \(Right\)\.
### Ablation Study and Sensitivity Analysis

#### Contribution of Adaptive Speculation Policy\.

Table[3](https://arxiv.org/html/2607.27735#Sx4.T3)compares our entropy\-guided adaptive policy against fixed speculation lengths \(k∈\{4,8,12,16,20\}k\\in\\\{4,8,12,16,20\\\}\) on L3 \(LongBench v2\)\. Pushing a fixed horizon inevitably degrades acceptance rate \(93\.4% atk=4k\{=\}4to 67\.9% atk=20k\{=\}20\), empirically confirming the efficiency inversion in Eq\.[1](https://arxiv.org/html/2607.27735#Sx2.E1): throughput peaks atk=12k\{=\}12\(2\.69×\\times\) then falls\. Our adaptive policy avoids this inversion and outperforms all fixed\-kkconfigurations without manual tuning, reaching2\.79×2\.79\\timesspeedup\. Per\-example speedup distributions are reported in the supplementary material\.

The benefit of adaptivity depends on workload heterogeneity\. On RULER NIAH, adaptive control improves token acceptance from 68\.6% to 79\.9% and throughput from 17\.11 to 20\.07 tokens/s relative to fixedk=10k=10, corresponding to a 17\.3% throughput improvement\. On InfiniteBench, adaptive control closely matches the best fixed setting \(14\.81 versus 14\.75 tokens/s\), while avoiding task\-specific manual selection ofkk\. Thus, adaptivity is particularly useful when token difficulty varies throughout generation, but may provide little additional gain on more uniform workloads\.

As a sanity check, negative draft entropy obtains an acceptance AUC of 0\.638 on RULER QA1\. Tokens in the low\-entropy bin are accepted at 86\.3%, compared with 62\.1% in the high\-entropy bin\. This modest but consistent separation supports entropy as a useful online control signal; it does not imply that entropy alone is a fully calibrated acceptance predictor\.

#### Scaling with Context Length\.

Figure[4](https://arxiv.org/html/2607.27735#Sx4.F4)traces performance as context length scales from 10K to 60K tokens\. While all methods exhibit stable acceptance rates and increasing speedups at longer contexts, SparseSpec\-L consistently achieves the highest acceleration\. Crucially, the competitive acceptance rate of MagicDec \(SnapKV\) is strictly conditioned on its optimal fixed speculation length ofk=2k=2\. When forced to a longer horizon \(k=10k=10\), its acceptance rate degrades below that of StreamingLLM due to permanent information loss\. In contrast, SparseSpec\-L robustly sustains both a high acceptance rate and a large effective speculation length across all context lengths\.

#### Sensitivity to Compression Ratio\.

Modulating the KV\-cache compression ratio exposes the fundamental trade\-off between drafting overhead \(γ\\gamma\) and acceptance rate \(α\\alpha\)\. Retaining more KV caches increasesα\\alphabut incurs higher drafting costs, eventually degrading the overall speedup\. As shown in Figure[4](https://arxiv.org/html/2607.27735#Sx4.F4), SparseSpec\-L consistently dominates all baselines in both speedup and acceptance rate across various compression regimes \(with the sole exception of MagicDec \(SnapKV\) operating at a heavily restrictedk=2k=2\)\. The empirical optimum lies at a compression ratio of∼10%\{\\sim\}10\\%, balancing context fidelity and drafting overhead\.

## Related Work

Speculative decoding\.Speculative decoding accelerates autoregressive generation through cheap drafting and parallel verification\(Chenet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib19); Leviathanet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib20); Miaoet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib21)\)\. Auxiliary drafters incur additional memory and model mismatch, while Medusa, multi\-token prediction, EAGLE\-3, and layer\-skipping methods require model\-specific parameters or sacrifice representational capacity\(Caiet al\.[2024a](https://arxiv.org/html/2607.27735#bib.bib32); Gloeckleet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib31); Liet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib30)\)\.

#### KV cache compression\.

StreamingLLM, H2O, ScissorHands, Quest, and PyramidKV reduce KV access through eviction or sparse selection\(Xiaoet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib25); Zhanget al\.[2023](https://arxiv.org/html/2607.27735#bib.bib22); Liuet al\.[2023](https://arxiv.org/html/2607.27735#bib.bib51); Tanget al\.[2024](https://arxiv.org/html/2607.27735#bib.bib23); Caiet al\.[2024b](https://arxiv.org/html/2607.27735#bib.bib49)\)\. MagicDec uses compressed KV caches for speculative decoding, while InfiniGen\(Leeet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib27)\)and ArkVale\(Chenet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib28)\)support recallable KV recovery\. SparseSpec\-L retains dense KV states for verification and uses a recallable sparse view only for drafting\.

Sparse\-context and adaptive speculation\.TriForce uses hierarchical sparse\-KV target execution before full\-KV verification, while prior SparseSpec and Vegas also perform sparse\-context self\-speculation\(Sunet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib53); Zhaoet al\.[2025](https://arxiv.org/html/2607.27735#bib.bib54); Yueet al\.[2026](https://arxiv.org/html/2607.27735#bib.bib55)\)\. SparseSpec\-L differs in its single\-stage pipeline, per\-head verification\-guided index, and cost\-aware length objective\. Dynamic draft lengths have also been studied by DISCO, AdaEDL, and SVIP\(Mamouet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib58); Agrawalet al\.[2024](https://arxiv.org/html/2607.27735#bib.bib57); Zhanget al\.[2025](https://arxiv.org/html/2607.27735#bib.bib56)\); our controller directly maximizes estimated step efficiency using online entropy and measured latency\.

## Conclusion

We introduced SparseSpec\-L, a training\-free sparse\-to\-full self\-speculative framework for long\-context inference\. It combines a recallable per\-head sparse KV index with cost\-aware adaptive speculation, achieving up to2\.79×2\.79\\timesend\-to\-end speedup across multiple tasks and model scales while preserving full\-KV verification\.

## Limitations

Our implementation uses non\-fused sparse\-KV gathering and dense non\-FlashAttention verification, leaving room for fused kernel optimization\. Experiments are limited to one A40, contexts up to 64K, and 128 generated tokens\. Evaluation on optimized serving engines and longer outputs remains future work\. The entropy controller is only moderately predictive and provides task\-dependent gains\.

## References

- Adaedl: early draft stopping for speculative decoding of large language models via an entropy\-based lower bound on token acceptance probability\.arXiv preprint arXiv:2410\.18351\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p2.1)\.
- Y\. Bai, S\. Tu, J\. Zhang, H\. Peng, X\. Wang, X\. Lv, S\. Cao, J\. Xu, L\. Hou, Y\. Dong,et al\.\(2025\)Longbench v2: towards deeper understanding and reasoning on realistic long\-context multitasks\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3639–3664\.Cited by:[Models & Datasets\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px1.p1.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[The Draft\-Verification Paradigm](https://arxiv.org/html/2607.27735#Sx2.SSx1.p1.8)\.
- T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. Dao \(2024a\)Medusa: simple llm inference acceleration framework with multiple decoding heads\.arXiv preprint arXiv:2401\.10774\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1),[Related Work](https://arxiv.org/html/2607.27735#Sx5.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024b\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[The Draft\-Verification Paradigm](https://arxiv.org/html/2607.27735#Sx2.SSx1.p1.8),[Related Work](https://arxiv.org/html/2607.27735#Sx5.p1.1)\.
- G\. Chen, Q\. Feng, J\. Ni, X\. Li, and M\. Q\. Shieh \(2025\)RAPID: long\-context inference with retrieval\-augmented speculative decoding\.arXiv preprint arXiv:2502\.20330\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1)\.
- R\. Chen, Z\. Wang, B\. Cao, T\. Wu, S\. Zheng, X\. Li, X\. Wei, S\. Yan, M\. Li, and Y\. Liang \(2024\)Arkvale: efficient generative llm inference with recallable key\-value eviction\.Advances in Neural Information Processing Systems37,pp\. 113134–113155\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- T\. Dao \(2023\)Flashattention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p1.1)\.
- F\. Gloeckle, B\. Y\. Idrissi, B\. Rozière, D\. Lopez\-Paz, and G\. Synnaeve \(2024\)Better & faster large language models via multi\-token prediction\.arXiv preprint arXiv:2404\.19737\.Cited by:[Related Work](https://arxiv.org/html/2607.27735#Sx5.p1.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, Y\. Zhang, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.arXiv preprint arXiv:2404\.06654\.Cited by:[Models & Datasets\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px1.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p1.1)\.
- W\. Lee, J\. Lee, J\. Seo, and J\. Sim \(2024\)\{\\\{infinigen\}\\\}: Efficient generative inference of large language models with dynamic\{\\\{kv\}\\\}cache management\.In18th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 24\),pp\. 155–172\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning,pp\. 19274–19286\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[The Draft\-Verification Paradigm](https://arxiv.org/html/2607.27735#Sx2.SSx1.p1.8),[Related Work](https://arxiv.org/html/2607.27735#Sx5.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[Sparse Attention Drafting](https://arxiv.org/html/2607.27735#Sx3.SSx2.p4.5),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1)\.
- Y\. Li, F\. Wei, C\. Zhang, and H\. Zhang \(2025\)Eagle\-3: scaling up inference acceleration of large language models via training\-time test\.arXiv preprint arXiv:2503\.01840\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1),[Related Work](https://arxiv.org/html/2607.27735#Sx5.p1.1)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.Advances in Neural Information Processing Systems36,pp\. 52342–52364\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- J\. Mamou, O\. Pereg, D\. Korat, M\. Berchansky, N\. Timor, M\. Wasserblat, and R\. Schwartz \(2024\)Dynamic speculation lookahead accelerates speculative decoding of large language models\.arXiv preprint arXiv:2405\.04304\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p2.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi,et al\.\(2024\)Specinfer: accelerating large language model serving with tree\-based speculative inference and verification\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 932–949\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1),[Related Work](https://arxiv.org/html/2607.27735#Sx5.p1.1)\.
- P\. Michel, O\. Levy, and G\. Neubig \(2019\)Are sixteen heads really better than one?\.Advances in neural information processing systems32\.Cited by:[Sparse Attention Drafting](https://arxiv.org/html/2607.27735#Sx3.SSx2.p1.2)\.
- R\. Sadhukhan, J\. Chen, Z\. Chen, V\. Tiwari, R\. Lai, J\. Shi, I\. E\. Yen, A\. May, T\. Chen, and B\. Chen \(2024\)Magicdec: breaking the latency\-throughput tradeoff for long context generation with speculative decoding\.arXiv preprint arXiv:2408\.11049\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p3.2),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1)\.
- H\. Sun, Z\. Chen, X\. Yang, Y\. Tian, and B\. Chen \(2024\)Triforce: lossless acceleration of long sequence generation with hierarchical speculative decoding\.arXiv preprint arXiv:2404\.11912\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p5.1),[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p2.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)Quest: query\-aware sparsity for efficient long\-context llm inference\.arXiv preprint arXiv:2406\.10774\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p1.1)\.
- H\. Xia, Z\. Yang, Q\. Dong, P\. Wang, Y\. Li, T\. Ge, T\. Liu, W\. Li, and Z\. Sui \(2024\)Unlocking efficiency in large language model inference: a comprehensive survey of speculative decoding\.Findings of the Association for Computational Linguistics: ACL 2024,pp\. 7655–7671\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p2.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2023\)Efficient streaming language models with attention sinks\.arXiv preprint arXiv:2309\.17453\.Cited by:[Sparse Attention Drafting](https://arxiv.org/html/2607.27735#Sx3.SSx2.p3.7),[Sparse Attention Drafting](https://arxiv.org/html/2607.27735#Sx3.SSx2.p4.5),[Baselines\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px2.p1.1),[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- Y\. Yue, Y\. Xue, and J\. Huang \(2026\)Vegas: self\-speculative decoding with verification\-guided sparse attention\.arXiv preprint arXiv:2602\.07223\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p5.1),[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p2.1)\.
- X\. Zhang, Y\. Chen, S\. Hu, Z\. Xu, J\. Chen, M\. Hao, X\. Han, Z\. Thai, S\. Wang, Z\. Liu,et al\.\(2024\)∞ Bench: extending long context evaluation beyond 100k tokens\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15262–15277\.Cited by:[Models & Datasets\.](https://arxiv.org/html/2607.27735#Sx4.SSx1.SSS0.Px1.p1.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p1.1)\.
- Z\. Zhang, J\. Xu, T\. Liang, X\. Chen, Z\. He, R\. Wang, and Z\. Tu \(2025\)Draft model knows when to stop: self\-verification speculative decoding for long\-form generation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 16696–16708\.Cited by:[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p2.1)\.
- Y\. Zhao, J\. Tang, K\. Zhu, Z\. Ye, C\. Chang, C\. Lin, J\. Park, G\. Xiao, M\. S\. Abdelfattah, M\. Gao,et al\.\(2025\)Accelerating large\-scale reasoning model inference with sparse self\-speculative decoding\.arXiv preprint arXiv:2512\.01278\.Cited by:[Introduction](https://arxiv.org/html/2607.27735#Sx1.p5.1),[KV cache compression\.](https://arxiv.org/html/2607.27735#Sx5.SSx3.SSS0.Px1.p2.1)\.