Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding

# Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decoding
Source: [https://arxiv.org/html/2606.24957](https://arxiv.org/html/2606.24957)
Jian\-Jia ChenXiaolin LinPei\-Shuo WangChi\-Chih ChangChun\-Che YangNing\-Chi HuangGrace Li ZhangKai\-Chiang Wu

###### Abstract

While speculative decoding improves inference throughput for multi\-batch long\-context Large Language Models \(LLMs\), its efficiency is often limited by a verification bottleneck where Key\-Value \(KV\) cache loading dominates latency\. Existing compression methods fail in this regime: static eviction incurs accuracy loss due to saliency shift, while dynamic selection introduces prohibitive computational overhead during the verification path\. We propose Dustin, a sparse verification framework designed for long\-context speculative decoding\. Dustin integrates lookahead signals from the draft model with historical attention from the target model to identify critical tokens with high fidelity across multi\-step verification windows\. To reduce recomputation latency, this approach further employs a sparse estimation scheme that restricts importance scoring to a minimal subset of attention heads\. Evaluations on PG\-19 and LongBench with Qwen2\.5\-72B demonstrate that Dustin achieves a 27\.85× speedup in self\-attention and a 9\.17× end\-to\-end decoding speedup at a 32k sequence length, all with negligible accuracy degradation\.

Machine Learning, ICML

## 1Introduction

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/Introduction/speedup_comparison.png)Figure 1:Latency breakdown of a single speculative decoding step\.Experiments are measured with a 32k input length and batch size 16\. We compare classic Speculative Decoding \(SD\), MagicDec \(MDec\)\(Sadhukhanet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib17)\), and our proposedDustin\. The x\-axis notationTarget\(Draft\)simply indicates the specific target and draft model pair used\.Large language models \(LLMs\)\(Achiamet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib6); Yanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib20); AI@Meta,[2024](https://arxiv.org/html/2606.24957#bib.bib21)\)address the rising demand for long\-context tasks but face severe memory\-bandwidth bottlenecks during auto\-regressive decoding\(Yuanet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib12); Popeet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib4)\)\. As context length grows, the linear expansion of Key\-Value \(KV\) caches increases the memory footprint to hundreds of gigabytes, making memory access the primary factor in latency\(Kwonet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib5); Dao,[2023](https://arxiv.org/html/2606.24957#bib.bib3)\)\.

Recent studies\(Sunet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib16); Sadhukhanet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib17); Yanget al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib26)\)indicate that speculative decoding effectively improves throughput in multi\-batch long\-context settings because the computation cost in verification is lower than the substantial overhead of loading the full KV cache\. However, the challenge of increasing KV cache loading costs persists\. As illustrated in Fig\.[1](https://arxiv.org/html/2606.24957#S1.F1), verification accounts for up to 87\.5% of the decoding latency at a 32k input length and batch size of 16, which limits the potential for end\-to\-end acceleration\.

Integrating KV cache compression strategies offers a promising solution, but standard methods are sub\-optimal for speculative decoding\. Static eviction methods\(Xiaoet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib7); Zhanget al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib9)\)permanently discard the context, leading to loss of precision due to saliency shift\(Zhaoet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib28)\), a phenomenon in which token sets with high attention change over time\. Conversely, dynamic selection methods retain the full KV cache but must re\-score token importance at every step, adding computation on the verification path\. The cost hinges on how importance is estimated: a naïve estimator that materializes attention scores over all heads and layers is prohibitively expensive—our analysis in Sec\.[5\.4\.1](https://arxiv.org/html/2606.24957#S5.SS4.SSS1)shows its overhead exceeds the latency of full\-cache self\-attention beyond 4k tokens\. Page\-level schemes such as Quest\(Tanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib15)\)avoid exactQ​K⊤QK^\{\\top\}via cheap min/max key statistics, yet still incur non\-negligible scoring cost and, as we show in Sec\.[5\.4\.1](https://arxiv.org/html/2606.24957#S5.SS4.SSS1), remain slower than our sparse estimator\. Furthermore, we found that solely relying on historical attention scores often results in substantial accuracy drops, as the verification stage processes multiple future steps concurrently\.

To address these challenges, we introduce Dustin, a sparse verification approach for multi\-batch long\-context speculative decoding\. By fusing lookahead signals from the draft model with historical attention from the target model, Dustin identifies critical tokens with negligible accuracy loss\. To minimize latency, we use a sparse estimation scheme that limits importance scoring to a subset of attention heads, enabling high\-speed verification without compromising generation quality\.

We evaluate Dustin on LongBench\(Baiet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib22)\)and PG\-19\(Raeet al\.,[2019](https://arxiv.org/html/2606.24957#bib.bib1)\)for the Llama3\(AI@Meta,[2024](https://arxiv.org/html/2606.24957#bib.bib21)\)and Qwen2\.5\(Yanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib20)\)families, achieving up to a9\.17×9\.17\\timesdecoding speedup on Qwen2\.5\-72B at a 32k context length with minimal accuracy loss\.

In summary, our key contributions are as follows:

- •We analyzed the limitations of predicting important tokens solely based on either historical attention scores or lookahead attention scores from the draft model\.
- •We designed a hybrid token selection policy and combined it with a search algorithm to find a minimal set of attention heads in order to reduce overhead while preserving accuracy\.
- •Our approach, Dustin, accelerates self\-attention computation by27\.85×27\.85\\times, resulting in a9\.17×9\.17\\timesdecoding phase speedup on PG\-19 benchmarks at a 32k input length and batch size 16 on Qwen2\.5\-72B\.

## 2Background and Related Work

### 2\.1Speculative Decoding

Speculative decoding \(SD\)\(Leviathanet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib10); Chenet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib11),[2024](https://arxiv.org/html/2606.24957#bib.bib24); Miaoet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib25)\)accelerates autoregressive generation by adopting a faster drafter to draft multiple tokens that are then verified in parallel by the target model\. Recent work revisits SD for long\-context and multi\-batch regimes, where inference becomes increasingly memory\-traffic dominated\. TriForce\(Sunet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib16)\)improves scalability via hierarchical speculation\. MagicDec\(Sadhukhanet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib17)\)shows SD can still yield gains in long\-context, large\-batch settings by amortizing full\-KV target verification over multiple drafted tokens and using sparse\-KV drafting to reduce the KV\-cache bottleneck\. QuantSpec\(Tiwariet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib27)\)reduces drafting overhead with quantized weights/KV for self\-speculation, while LongSpec\(Yanget al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib26)\)designs a long\-context\-oriented drafter with a constant\-sized KV cache, together with position\-index and attention\-aggregation mechanisms for efficient long\-context speculative decoding\.

### 2\.2KV Cache Eviction

Long\-context inference is often bottlenecked by KV cache memory and attention cost, motivating methods that reduce computation by retaining or accessing only a subset of cached tokens\. A common line of work performs*attention\-guided eviction*, using recent attention weights as a proxy for token importance to decide which KV entries to be kept\(Xiaoet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib7); Liuet al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib8); Zhanget al\.,[2023](https://arxiv.org/html/2606.24957#bib.bib9); Liet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib13); Caiet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib23); Orenet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib14); Linet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib30)\)\. However, the token set with high attention changes during decoding\. SmallKV\(Zhaoet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib28)\)highlights this as the “saliency shift issue” and mitigates the problem with the help of a small model\.

In contrast to irreversible eviction, Quest\(Tanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib15)\)retains the full KV cache and performs*query\-aware*selection at each decoding step\. It scores KV pages using inexpensive min/max key statistics \(avoiding exactQ​K⊤QK^\{\\top\}\), and attends only to the top\-ranked pages, thereby enabling sparse attention without permanent removal\.

### 2\.3Target\-Side KV Cache Compression in Speculative Decoding

Complementary to compressing the drafter\-side state, another line of work reduces*target\-side verification*cost by verifying draft tokens using only a sparse subset of the target KV cache\. SpecAttn\(Shah,[2025](https://arxiv.org/html/2606.24957#bib.bib29)\)instantiates this idea by using a small draft model to estimate token importance and enabling the target model to verify with token\-level sparse KV access, thereby reducing verification\-time attention cost in long contexts\. This direction is most relevant to our focus on accelerating the*target\-model verification*phase under long\-context inference\.

## 3Observation

While speculative decoding \(SD\) accelerates inference for long sequences in large batches, the challenge of the increasing KV cache loading cost persists\. Consequently, KV cache compression remains essential\. Previous research indicates that permanent KV cache eviction can result in significant information loss due to thesaliency shift issue\(Zhaoet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib28)\), caused by dynamic changes in token importance during decoding\.

The verification phase involves processing multiple draft tokens simultaneously\. This section evaluates two primary sources to predict and select the most significantkktokens for the sampling range\[i,i\+w−1\]\[i,i\+w\-1\], considering futurewwtokens:

1. 1\.Historical Attention Scoresextracted from preceding forward passes of the target model\.
2. 2\.Lookahead Attention Scoresgenerated by the draft model during the speculation of future tokens\.

We show the limitations of predicting important tokens solely based on historical or lookahead attention scores, and propose a hybrid approach that leverages both strengths to preserve generation quality\.

### 3\.1Evaluation Framework: Attention Recovery Rate

We measure how well a KV\-selection policyπ\\pipreserves attention usingAttention Recovery Rate \(ARR\)\. At decoding stepii, let𝒱i\\mathcal\{V\}\_\{i\}be the valid KV cache positions andAi,jA\_\{i,j\}represent the normalized attention weight onj∈𝒱ij\\in\\mathcal\{V\}\_\{i\}such that∑j∈𝒱iAi,j=1\\sum\_\{j\\in\\mathcal\{V\}\_\{i\}\}A\_\{i,j\}=1\. Given a subsetKiπ⊆𝒱iK\_\{i\}^\{\\pi\}\\subseteq\\mathcal\{V\}\_\{i\}selected by the policy, the ARR is defined as:

ARRi​\(π\)≜∑j∈KiπAi,j\.\\mathrm\{ARR\}\_\{i\}\(\\pi\)\\triangleq\\sum\_\{j\\in K\_\{i\}^\{\\pi\}\}A\_\{i,j\}\.\(1\)
##### Windowed ARR\.

To align with the SD verification of a block of tokens, we calculate the average ARR over a forward window of lengthww:

ARRi\(w\)​\(π\)≜1w​∑s=0w−1ARRi\+s​\(π\)\.\\mathrm\{ARR\}^\{\(w\)\}\_\{i\}\(\\pi\)\\triangleq\\frac\{1\}\{w\}\\sum\_\{s=0\}^\{w\-1\}\\mathrm\{ARR\}\_\{i\+s\}\(\\pi\)\.\(2\)

##### SD oracle \(reference upper bound\)\.

The optimal subset for maximizingARRi\(w\)\\mathrm\{ARR\}^\{\(w\)\}\_\{i\}requires access to future attention weights that are unavailable during selection\. Therefore, an oracle policy serves as a theoretical upper bound by selectingKiK\_\{i\}with full access to future attention:

A¯i,j\(w\)≜1w​∑s=0w−1Ai\+s,j,Kiπorc≜TopKk⁡\(A¯i,⋅\(w\)\),\\bar\{A\}\_\{i,j\}^\{\(w\)\}\\triangleq\\frac\{1\}\{w\}\\sum\_\{s=0\}^\{w\-1\}A\_\{i\+s,\\,j\},\\qquad K\_\{i\}^\{\\pi\_\{\\mathrm\{orc\}\}\}\\triangleq\\operatorname\{TopK\}\_\{k\}\\\!\\big\(\\bar\{A\}\_\{i,\\cdot\}^\{\(w\)\}\\big\),\(3\)which yields

ARRi\(w\)​\(πorc\)=∑j∈KiπorcA¯i,j\(w\)\.\\mathrm\{ARR\}^\{\(w\)\}\_\{i\}\(\\pi\_\{\\mathrm\{orc\}\}\)=\\sum\_\{j\\in K\_\{i\}^\{\\pi\_\{\\mathrm\{orc\}\}\}\}\\bar\{A\}\_\{i,j\}^\{\(w\)\}\.\(4\)
Intuitively, ARR is the fraction of the original attention mass preserved by the selected KV tokens: an ARR of 1 means that the selected positions cover all attention mass, while a lower ARR indicates that more attention is lost due to compression\. Therefore, ARR provides a direct measure of how much attention information is retained under a fixed KV cache budget\. We further show in Appendix[B](https://arxiv.org/html/2606.24957#A2)that ARR is strongly negatively correlated with output\-logit KL divergence, suggesting that ARR is a meaningful proxy for sparse\-forward fidelity\.

Experiments utilize theQwen2\.5\-Instructseries on theLongRewarddataset\(Zhanget al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib31)\)which contains an average context of 13\.5k tokens\.Qwen2\.5\-32Bserves as the primary target model for temporal and layer\-wise analysis, while the0\.5Bvariant acts as the draft model\. For clarity in the following plots, the “KV Cache Budget” refers to the fixed capacitykkallocated for the compressed context\.

### 3\.2Temporal Decay of Target Historical Signals

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/Observation/3_1_Target_vs_Target_past/32B_attention_recovery_total.png)Figure 2:Attention recovery rate analysis \(Historical Score\)\. Comparison of attention recovery rates using the future average attention \(Oracle\) versus historical attention scores on the Qwen2\.5\-32B model\. The minimal gap indicates high temporal stability\.This section investigates the validity ofHistorical Attention Scoresas a low\-overhead proxy for future token importance\. Specifically, the analysis evaluates whether token saliency remains consistent across thew=4w=4verification steps and considers a history\-based policy utilizing the past attention of the target model\. Context tokens are ranked byAi−δ,⋅A\_\{i\-\\delta,\\cdot\}, whereδ\\deltadenotes the look\-back distance in decoding steps\. The top\-kktokens are retained and compared against the SD Oracle \(Eq\.[3](https://arxiv.org/html/2606.24957#S3.E3)\)\.

As illustrated in Fig\.[2](https://arxiv.org/html/2606.24957#S3.F2), the results indicate strong short\-term consistency\. With a KV cache budget ofk=512k=512, utilizing the most recent historical attention \(δ=1\\delta=1\) achieves an ARR within1\.04%1\.04\\%of the Oracle\. This suggests thatAi−1,⋅A\_\{i\-1,\\cdot\}serves as a robust proxy for near\-future relevance\. However, the verification phase requires processing multiple draft tokens simultaneously, leading to accuracy degradation as draft depth increases\. The limits of purely historical strategies are further quantified in the ablation study in Section[5\.4\.2](https://arxiv.org/html/2606.24957#S5.SS4.SSS2)\.

### 3\.3Inconsistency of Draft Lookahead Signals

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation2_Sensitivity/200/attention_recovery_comparison_total.png)Figure 3:Attention recovery rate analysis \(Lookahead Scores\)\. Comparison of attention recovery rates on target models \(7B\-72B\) using their own future attention scores \(Oracle\) versus lookahead scores predicted by a 0\.5B draft model\. The significant gap in 14B/32B reveals the inconsistency of cross\-model prediction\.To mitigate the decay inherent in historical signals, an alternative approach utilizeslookahead attention scores\. These scores are attention distributions computed directly by the draft model during speculation and serve as real\-time indicators of immediate saliency shifts\.

While prior methodologies\(Zhaoet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib28); Shah,[2025](https://arxiv.org/html/2606.24957#bib.bib29)\)rely exclusively on lookahead attention scores from the draft model, this study demonstrates that such policies are prone to inconsistency and risk significant ARR degradation\. To evaluate this limitation, a lightweight draft model \(0\.5B\) ranks KV entries via lookahead scores\. This policy is measured against the SD Oracle \(Eq\.[3](https://arxiv.org/html/2606.24957#S3.E3)\) to quantify the discrepancy between predicted and ground\-truth attention\.

As shown in Fig\.[3](https://arxiv.org/html/2606.24957#S3.F3), the ARR exhibits significant variation across different target and draft model pairs\. Although the 7B and 72B models align relatively well with the 0\.5B draft model, the intermediate 14B and 32B variants suffer from substantial information loss\. These findings suggest that selection policies relying solely on signals from draft models lack reliability across diverse model scales\. We provide additional evidence across more draft scales and model families in Appendix[J](https://arxiv.org/html/2606.24957#A10)\.

### 3\.4Hybrid Selection: Integrating Historical and Lookahead Signals

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation3_Sensitivity/200/Observation3_LayerWise_Recovery_LongRange_Qwen2.5-32B+0.5B_MaxPooled.png)Figure 4:Layer\-wise attention recovery analysis\. Comparison of target\-history, draft\-lookahead, and hybrid strategies on Qwen2\.5\-32B\. The results highlight structural complementarity: target\-historical signals dominate in deeper layers, while draft\-lookahead signals excel in early layers \(via lookahead\)\.The results in Sec\.[3\.3](https://arxiv.org/html/2606.24957#S3.SS3)indicate that draft\-lookahead importance is unreliable as a standalone global signal\. However, these signals can enhance the ARR when utilized as an augmentation to the historical signals of the target model\. Fig\.[4](https://arxiv.org/html/2606.24957#S3.F4)provides a layer\-wise analysis to identify where each signal is most informative, motivating a hybrid construction that leverages their synergistic strengths\.

To ensure performance in long\-context scenarios, attention scores are computed using only the designatedSemantic Retrieval Heads \(SRHs\)across specific layers\. As detailed in Sec\.[4\.2](https://arxiv.org/html/2606.24957#S4.SS2), this selective approach filters noise and aligns with retention quality while significantly reducing computational overhead\.

The layer\-wise comparative analysis reveals a clear divergence in policy performance:

- •Draft\-based Lookahead:Achieves high ARR in the initial layers but exhibits performance degradation as network depth increases\.
- •Target\-based History:Demonstrates an inverse trend, maintaining robustness in deeper layers while performing sub\-optimally in the early stages\.

This fundamental trade\-off between the foresight of the draft model and the reliability of the target model history motivates the proposed hybrid selection strategy\. By integrating both attention sources, the hybrid approach maximizes the ARR across the entire architecture\. This configuration follows the optimal search parameters identified in Sec\.[4\.2](https://arxiv.org/html/2606.24957#S4.SS2), ensuring that the distinct strengths of both signals are utilized to maintain high fidelity in the KV cache\.

## 4Methodology

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/overview.png)Figure 5:Overview of our sparse verification approach\. The process begins with hybrid attention aggregation \(Eq\.[5](https://arxiv.org/html/2606.24957#S4.E5)\) to compute a global importance map, followed by Top\-K selection \(Eq\.[6](https://arxiv.org/html/2606.24957#S4.E6)\) to determine the final verification setℐv​e​r​i​f​y\\mathcal\{I\}\_\{verify\}\.To reduce verification overhead during Speculative Decoding \(SD\), we introduce Dustin, a sparse verification approach for efficient Large Language Model \(LLM\) inference\. Dustin identifies critical Key\-Value \(KV\) pairs by integrating target\-historical and draft\-lookahead attention signals\. This approach stems from observations in Sec\.[3\.4](https://arxiv.org/html/2606.24957#S3.SS4)indicating that combining signals from both models maximizes the Attention Recovery Rate \(ARR\)\.

This approach consists of two core components: \(1\)hybrid attention aggregation, which computes a global relevance map for token selection, and \(2\)efficient estimation via Semantic Retrieval Heads \(SRHs\), which reduces computational costs by utilizing a small subset of attention heads\.

### 4\.1Sparse Verification via Hybrid Attention Aggregation

Fig\.[5](https://arxiv.org/html/2606.24957#S4.F5)illustrates the overall sparse verification workflow in Dustin\. During draft generation, Dustin collects attention signals from both the target model and the draft model to estimate the relevance of historical context tokens\. These signals are then aggregated into a global importance map, from which Dustin selects a fixed\-size verification set\. During verification, only the selected KV entries are retrieved to form a compressed target KV cache, which is used to verify the speculative draft tokens\. After verification, the accepted tokens are appended to the target KV cache, while rejected draft tokens are discarded\.

Dustin determines token relevance by aggregating attention from both the last target token and the speculative draft tokens, providing a more comprehensive signal than the heuristic proxies used in prior works like Quest\(Tanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib15)\)\. In addition, unlike methods such as SpecAttn\(Shah,[2025](https://arxiv.org/html/2606.24957#bib.bib29)\), which estimate token relevance separately for each layer, Dustin uses a global index set across all layers\. This layer\-invariant approach simplifies KV retrieval and reduces extra computation overhead\.

Formally, let𝒜Draft∈ℝHd×Ld×N×Γ\\mathcal\{A\}^\{\\text\{Draft\}\}\\in\\mathbb\{R\}^\{H\_\{d\}\\times L\_\{d\}\\times N\\times\\Gamma\}denote the draft attention tensor acrossHdH\_\{d\}heads,LdL\_\{d\}layers,NNcontext tokens, andΓ\\Gammadraft tokens, and let𝒜Target∈ℝHt×Lt×N×1\\mathcal\{A\}^\{\\text\{Target\}\}\\in\\mathbb\{R\}^\{H\_\{t\}\\times L\_\{t\}\\times N\\times 1\}denote the target attention tensor of the last generated token\. We compute the global importance vectors by summing attention scores across heads and layers, and also across theΓ\\Gammadraft tokens for the draft model, yieldingSDraft∈ℝNS\_\{\\text\{Draft\}\}\\in\\mathbb\{R\}^\{N\}andSTarget∈ℝNS\_\{\\text\{Target\}\}\\in\\mathbb\{R\}^\{N\}:

SDraft\\displaystyle S\_\{\\text\{Draft\}\}=∑h=1Hd∑l=1Ld∑γ=1Γ𝒜h,l,n,γDraft,\\displaystyle=\\sum\_\{h=1\}^\{H\_\{d\}\}\\sum\_\{l=1\}^\{L\_\{d\}\}\\sum\_\{\\gamma=1\}^\{\\Gamma\}\\mathcal\{A\}^\{\\text\{Draft\}\}\_\{h,l,n,\\gamma\},\(5\)STarget\\displaystyle S\_\{\\text\{Target\}\}=∑h=1Ht∑l=1Lt∑γ=11𝒜h,l,n,γTarget\.\\displaystyle=\\sum\_\{h=1\}^\{H\_\{t\}\}\\sum\_\{l=1\}^\{L\_\{t\}\}\\sum\_\{\\gamma=1\}^\{1\}\\mathcal\{A\}^\{\\text\{Target\}\}\_\{h,l,n,\\gamma\}\.
As shown in Fig\.[5](https://arxiv.org/html/2606.24957#S4.F5), the final verification setℐverify\\mathcal\{I\}\_\{\\text\{verify\}\}is constructed through a multi\-stage selection process under a fixed budgetkk\. Dustin first protects a small prefix of attention sinks \(ℐsink\\mathcal\{I\}\_\{\\text\{sink\}\}\) and a local window of recent tokens \(ℐwindow\\mathcal\{I\}\_\{\\text\{window\}\}\)\. It then fills the remaining budget by selecting the top\-mmtokens according to the draft signalSDraftS\_\{\\text\{Draft\}\}, followed by the most relevant remaining tokens according to the target signalSTargetS\_\{\\text\{Target\}\}\. This tiered allocation ensures that the most critical historical and lookahead information is preserved for verification:

ℐprot\\displaystyle\\mathcal\{I\}\_\{\\text\{prot\}\}=ℐsink∪ℐwindow,\\displaystyle=\\mathcal\{I\}\_\{\\text\{sink\}\}\\cup\\mathcal\{I\}\_\{\\text\{window\}\},\(6\)ℐdraft\\displaystyle\\mathcal\{I\}\_\{\\text\{draft\}\}=TopK​\(SDraft∖ℐprot,m\),\\displaystyle=\\text\{TopK\}\(S\_\{\\text\{Draft\}\}\\setminus\\mathcal\{I\}\_\{\\text\{prot\}\},m\),ℐtarget\\displaystyle\\mathcal\{I\}\_\{\\text\{target\}\}=TopK​\(STarget∖\(ℐprot∪ℐdraft\),k−m−\|ℐprot\|\),\\displaystyle=\\text\{TopK\}\(S\_\{\\text\{Target\}\}\\setminus\(\\mathcal\{I\}\_\{\\text\{prot\}\}\\cup\\mathcal\{I\}\_\{\\text\{draft\}\}\),k\-m\-\|\\mathcal\{I\}\_\{\\text\{prot\}\}\|\),ℐverify\\displaystyle\\mathcal\{I\}\_\{\\text\{verify\}\}=ℐprot∪ℐdraft∪ℐtarget\.\\displaystyle=\\mathcal\{I\}\_\{\\text\{prot\}\}\\cup\\mathcal\{I\}\_\{\\text\{draft\}\}\\cup\\mathcal\{I\}\_\{\\text\{target\}\}\.

### 4\.2Efficient Estimation via Semantic Retrieval Heads

Full reconstruction of attention tensors for both models imposes heavy computational demands, specifically𝒪​\(Hd⋅Ld⋅N⋅Γ\)\\mathcal\{O\}\(H\_\{d\}\\cdot L\_\{d\}\\cdot N\\cdot\\Gamma\)for the draft model and𝒪​\(Ht⋅Lt⋅N⋅1\)\\mathcal\{O\}\(H\_\{t\}\\cdot L\_\{t\}\\cdot N\\cdot 1\)for the target model\. We minimize this overhead by adopting Semantic Retrieval Heads \(SRHs\)\(Linet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib30)\)as an efficient estimator\. We identify a small subset of attention heads in each layer that capture the most significant semantic dependencies, allowing us to estimate token relevance without calculating the full attention map\.

#### 4\.2\.1Semantic Retrieval Head Scoring

Fig\.[6](https://arxiv.org/html/2606.24957#S4.F6)illustrates our SRH selection pipeline\. Following CompressKV\(Linet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib30)\), we identify SRHs using a layer\-wise selection strategy\. Specifically, based on profiling scores, only a small subset of heads is retained in each layer to provide their attention scores for KV selection\.

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/head_select.png)Figure 6:Selection process of SRHs\. We leverage offline profiling to isolate heads that capture semantic dependencies, enabling efficient online estimation of token relevance using only a fraction of the total attention heads\.
#### 4\.2\.2Sparse Estimation Configuration Search

To determine the optimal layer subset and budget split between models, we utilize a multi\-stage optimization approach, summarized in Algorithm[1](https://arxiv.org/html/2606.24957#alg1)\. Crucially, to ensure alignment with the findings in Sec\.[3\.4](https://arxiv.org/html/2606.24957#S3.SS4), the search utilizes validation setΩ\\Omegawithteacher\-forced trajectories\. Instead of generic accuracy metrics, we employ the attention recovery rate \(ARR\) defined in Eq\.[2](https://arxiv.org/html/2606.24957#S3.E2)as the primary optimization objective\. These metrics are computed exclusively over the identified SRHs to filter noise\. The process decomposes the search space into three phases:

1. \(1\)Greedy target layer search:We first identify the single target layerlt∗l\_\{t\}^\{\*\}that maximizes ARR via grid search, establishing a robust cornerstone for online estimation\.
2. \(2\)Greedy draft layer search:Fixinglt∗l\_\{t\}^\{\*\}, we iteratively append draft layers that increase ARR\. To maintain computational efficiency during this combinatorial search, we employ a few fixed heuristic budgetsBs​e​a​r​c​hB\_\{search\}\(e\.g\., 128, 192, 256 tokens\) instead of full grid search\.
3. \(3\)Bayesian budget tuning:Finally, Optuna\(Akibaet al\.,[2019](https://arxiv.org/html/2606.24957#bib.bib2)\)is employed for Bayesian optimization on the budget parametermm, fine\-tuning the trade\-off between draft\-lookahead and target\-historical attention scores\.

Algorithm[1](https://arxiv.org/html/2606.24957#alg1)is run only once per target–draft model pair after SRH identification, and the resulting configuration is reused across tasks\. Appendix[I](https://arxiv.org/html/2606.24957#A9)shows that this Dustin\-specific offline cost is only 11\.9–35\.4 minutes for the evaluated 7B–72B model pairs\.

Algorithm 1Sparse Verification Configuration Search0:Models

ℳTarget,ℳDraft\\mathcal\{M\}^\{\\text\{Target\}\},\\mathcal\{M\}^\{\\text\{Draft\}\}
0:Scores

𝒜Target,𝒜Draft\\mathcal\{A\}^\{\\text\{Target\}\},\\mathcal\{A\}^\{\\text\{Draft\}\}from validation set

Ω\\Omega
0:Selected layers

ℒs​e​l\\mathcal\{L\}\_\{sel\}, optimal draft split

mm
1:Phase 1: Greedy Target Layer Search

2:

lt∗←arg⁡maxl∈ℒt⁡ARR​\(TargetHistorical​\(l\)\)l^\{\*\}\_\{t\}\\leftarrow\\arg\\max\_\{l\\in\\mathcal\{L\}\_\{t\}\}\\text\{ARR\}\(\\text\{TargetHistorical\}\(l\)\)
3:

ℒs​e​l←\{lt∗\}\\mathcal\{L\}\_\{sel\}\\leftarrow\\\{l^\{\*\}\_\{t\}\\\}
4:Phase 2: Greedy Draft Layer Search

5:for

r=1r=1to

ND​r​a​f​tN\_\{Draft\}do

6:

Js​t​e​p​\(d\)≜ARR​\(Hybrid​\(ℒs​e​l∪\{d\}\),Bs​e​a​r​c​h\)J\_\{step\}\(d\)\\triangleq\\text\{ARR\}\\left\(\\text\{Hybrid\}\(\\mathcal\{L\}\_\{sel\}\\cup\\\{d\\\}\),B\_\{search\}\\right\)
7:

d∗←arg⁡maxd∈ℒd∖ℒs​e​l⁡Js​t​e​p​\(d\)d^\{\*\}\\leftarrow\\arg\\max\_\{d\\in\\mathcal\{L\}\_\{d\}\\setminus\\mathcal\{L\}\_\{sel\}\}J\_\{step\}\(d\)
8:

ℒs​e​l←ℒs​e​l∪\{d∗\}\\mathcal\{L\}\_\{sel\}\\leftarrow\\mathcal\{L\}\_\{sel\}\\cup\\\{d^\{\*\}\\\}
9:endfor

10:Phase 3: Budget Optimization \(Bayesian\)

11:Let

Jf​i​n​a​l​\(m\)≜ARR​\(Hybrid​\(ℒs​e​l\),m\)J\_\{final\}\(m\)\\triangleq\\text\{ARR\}\(\\text\{Hybrid\}\(\\mathcal\{L\}\_\{sel\}\),m\)
12:

m←OptunaOptimize​\(Jf​i​n​a​l​\(m\)\)m\\leftarrow\\text\{OptunaOptimize\}\(J\_\{final\}\(m\)\)

#### 4\.2\.3Online Sparse Estimation and Complexity Analysis

During inference, Dustin recomputes attention scores only for a small subset of the identified Semantic Retrieval Heads \(SRHs\)\. This recomputation is necessary for two reasons\. First, optimized attention kernels such as FlashAttention\(Dao,[2023](https://arxiv.org/html/2606.24957#bib.bib3)\)avoid writing attention matrices to global memory, making attention weights inaccessible from the normal forward path\. Second, Dustin’s target verification forward is itself sparse: it attends only to the selected KV entries inIverifyI\_\{\\mathrm\{verify\}\}\. As a result, this forward pass only computes attention scores for the selected KV subset, while scores for unselected context tokens are never produced\. Since the next verification step requires ranking tokens over the full candidate context, these sparse\-forward attention scores cannot be directly reused for online importance estimation\.

By restricting this recomputation to a sparse subset of heads and layers, Dustin minimizes overhead while acquiring the global signals required for token selection\. For instance, in a Qwen2\.5\-72B/0\.5B setup, restricting the estimator to minimal SRHs reduces the theoretical computational cost to approximately0\.8%0\.8\\%relative to the full hybrid attention\-score calculation\. We provide detailed empirical analysis of latency and overhead in Sec\.[5\.4\.1](https://arxiv.org/html/2606.24957#S5.SS4.SSS1)and Appendix[A](https://arxiv.org/html/2606.24957#A1)\.

## 5Experiment

In this section, we evaluate Dustin under long\-context and multi\-batch inference along two axes: \(i\) generation quality, measured by long\-context task accuracy, and \(ii\) efficiency, measured by a self\-attention latency breakdown and decode\-stage throughput \(tokens/s\)\. We further conduct ablations to quantify the overhead of online token importance estimation and the impact of budget tuning\.

Table 1:Accuracy evaluation on LongBench under two strict KV cache budgets \(512 and 128 tokens\)\.Boldindicates the best result, andunderlineindicates the second\-best result among compressed methods under the same target model and KV budget\.Target ModelCompression MethodKVBudgetSingle\-doc QAMulti\-doc QASummarizationFew\-shotSyntheticCodeAvg\.Qwen2\.5\-72B\-InstructVanilla / Lossless SD—44\.2357\.0027\.5472\.5560\.0066\.7055\.81StreamingLLM51227\.6133\.8221\.7054\.5739\.8435\.0235\.25SnapKV42\.8058\.7723\.0271\.2155\.0057\.2951\.90Quest\*36\.9141\.8025\.9264\.4055\.0857\.2447\.69Dustin44\.2356\.3527\.1071\.2360\.0065\.9755\.23StreamingLLM12823\.2732\.4917\.5848\.4842\.6933\.6832\.72SnapKV36\.5756\.8919\.8063\.7151\.6351\.5447\.08Quest\*23\.1225\.2121\.6752\.8149\.7046\.3737\.01Dustin44\.5955\.3126\.1468\.3552\.1361\.3052\.41Llama\-3\.3\-70B\-InstructVanilla / Lossless SD—45\.4858\.8828\.5870\.7553\.6349\.6450\.94StreamingLLM51232\.3639\.3523\.0151\.5551\.0541\.9839\.61SnapKV44\.4559\.5824\.8768\.7553\.3958\.8252\.41Quest\*40\.3452\.2427\.1866\.6354\.6151\.9548\.96Dustin45\.2759\.1027\.7670\.1453\.4059\.1553\.23StreamingLLM12827\.1339\.7819\.0246\.4352\.4439\.4936\.94SnapKV39\.1758\.0222\.0764\.5554\.6358\.5350\.32Quest\*23\.6627\.9321\.0947\.9444\.2035\.6733\.21Dustin45\.5559\.0226\.7568\.1853\.9662\.7853\.81The symbol \* denotes that Quest does not apply KV compression to decoder layers 0–1, which slightly exceeds KV budget\.

### 5\.1Setup

##### Models

We evaluate Dustin on two model families: Llama3\(AI@Meta,[2024](https://arxiv.org/html/2606.24957#bib.bib21)\)and Qwen2\.5\(Yanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib20)\), each paired with a lightweight same\-family draft model\. In the main text, we report efficiency results on Qwen2\.5\-72B and accuracy results on Qwen2\.5\-72B and Llama\-3\.3\-70B\.

##### Tasks and Benchmarks

We use PG\-19\(Raeet al\.,[2019](https://arxiv.org/html/2606.24957#bib.bib1)\)to measure long\-context generation throughput and latency, and LongBench\(Baiet al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib22)\)to evaluate long\-context task accuracy across question answering, summarization, few\-shot learning, and code\-related tasks\.

##### Implementation Details

All experiments run on NVIDIA H200 GPUs with bfloat16 precision\. For the main\-text efficiency experiments, Qwen2\.5\-72B and Llama\-3\.3\-70B are deployed with pipeline parallelism across 4 H200 GPUs\. Detailed experimental configurations are provided in Appendix[C](https://arxiv.org/html/2606.24957#A3)\.

##### Accuracy Baselines

Under constrained KV\-cache budgets, we compare against: \(i\)Vanilla / Lossless SDmethods that preserve the target\-model computation without KV compression \(including lossless speculative decoding variants such asMagicDec\); \(ii\)StreamingLLM, a streaming\-retention baseline applied to the target model that keeps only sink tokens and a sliding window of recent tokens; \(iii\)SnapKV, a training\-free KV\-cache compression method that uses attention patterns observed near the end of the prompt to select important prompt KV positions for each attention head, and then keeps the compressed prompt KV cache fixed during decode; and \(iv\)Quest, a block/page\-level KV selection method applied to the target model\.

##### Efficiency Baselines

All efficiency baselines use FlashAttention v2 to ensure a fair comparison\. We compare against \(i\)Vanilla, autoregressive decoding without speculative decoding; \(ii\)ClassicSD, speculative decoding with a same\-family draft model and a full KV cache; and \(iii\)MagicDec, speculative decoding with a sparse\-KV draft model\. To isolate the effect of the verification strategy, all speculative\-decoding methods use the same target–draft model pair, sequential draft blocks, and speculative depth for each target model\. Detailed configurations, including draft model size and draft length, are provided in Appendix[C](https://arxiv.org/html/2606.24957#A3)\.

### 5\.2Accuracy Evaluation

We benchmark the accuracy of the sparse verification process against multiple KV cache compression baselines on LongBench under restricted budgets\. As detailed in Table[1](https://arxiv.org/html/2606.24957#S5.T1), Dustin maintains near\-lossless performance across most categories and consistently outperforms existing methods\. These results indicate that the hybrid selection strategy of this approach effectively preserves critical information, ensuring high fidelity during the verification phase\.

### 5\.3Efficiency Evaluation

#### 5\.3\.1Self\-Attention Latency Breakdown

Fig\.[7](https://arxiv.org/html/2606.24957#S5.F7)breaks down theself\-attention cost in Dustin’s target\-model verificationinto*online importance estimation*and*sparse verification attention*\. With a fixed KV budget of 512 tokens, the speedup scales with verification workload:9\.35×9\.35\\times\(16K, batch 8\), and27\.85×27\.85\\times\(32K, batch 16\)\. More detailed results are provided in Appendix[G](https://arxiv.org/html/2606.24957#A7.SS0.SSS0.Px2)\.

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/Experiment/latency_figure_fixed100_2panels.png)Figure 7:Self\-attention speedup evaluation on Qwen2\.5\-72B\.
#### 5\.3\.2End\-to\-End Decode\-Stage Throughput

To evaluate decoding efficiency, we sample passages from PG\-19 and construct a long\-context story continuation task with a fixed instruction prompt\. Table[2](https://arxiv.org/html/2606.24957#S5.T2)reports results for batch sizes of 8 and 16 and context lengths from 8K to 32K tokens\. In this evaluation, throughput during the decoding stage \(tokens/s\) is denoted astput, speedup relative to the Vanilla baseline asα\\alpha, and the average number of accepted tokens per step asτ\\tau\. Detailed performance metrics for additional experimental configurations are provided in Appendix[H](https://arxiv.org/html/2606.24957#A8)\.

Dustin attains the highest throughput, and the advantage grows with both batch size and context length\. OnQwen2\.5\-72B, for batch size 8, speedup increases from3\.01×3\.01\\times\(8K\) to6\.61×6\.61\\times\(32K\)\. For batch size 16, speedup increases from4\.76×4\.76\\times\(8K\) to9\.17×9\.17\\times\(32K\)\. The same trend holds onLlama\-3\.3\-70B, where Dustin reaches 6\.28× \(batch 8\) and 7\.18× \(batch 16\) at 32K, consistently outperforming ClassicSD and MagicDec\.

Table 2:Efficiency evaluation across different context lengths and batch sizes on Qwen2\.5\-72B\-Instruct and Llama\-3\.3\-70B\-InstructMethodBatch SizeContext Length8K16K24K32Ktputτ\\tauα\\alphatputτ\\tauα\\alphatputτ\\tauα\\alphatputτ\\tauα\\alphaQwen2\.5\-72B\-InstructVanilla858\.06\-1\.00×\\times38\.75\-1\.00×\\times29\.05\-1\.00×\\times23\.26\-1\.00×\\timesClassicSD101\.822\.21\.75×\\times67\.662\.21\.75×\\times50\.472\.21\.74×\\times40\.832\.21\.76×\\timesMagicDec98\.902\.11\.70×\\times65\.672\.01\.69×\\times49\.912\.11\.72×\\times39\.632\.01\.70×\\timesDustin174\.862\.23\.01×\\times172\.372\.34\.45×\\times162\.122\.25\.58×\\times153\.672\.26\.61×\\timesVanilla1676\.23\-1\.00×\\times46\.03\-1\.00×\\times32\.95\-1\.00×\\times25\.70\-1\.00×\\timesClassicSD145\.402\.31\.91×\\times89\.262\.21\.94×\\times63\.002\.21\.91×\\times48\.682\.11\.89×\\timesMagicDec143\.672\.11\.88×\\times89\.272\.11\.94×\\times64\.632\.11\.96×\\times50\.382\.01\.96×\\timesDustin362\.502\.24\.76×\\times324\.862\.27\.06×\\times276\.342\.28\.39×\\times235\.812\.29\.17×\\timesLlama\-3\.3\-70B\-InstructVanilla859\.01\-1\.00×\\times39\.44\-1\.00×\\times29\.52\-1\.00×\\times23\.61\-1\.00×\\timesClassicSD115\.912\.71\.96×\\times75\.312\.61\.91×\\times55\.142\.81\.87×\\times43\.412\.91\.84×\\timesMagicDec122\.802\.72\.08×\\times79\.282\.52\.01×\\times60\.672\.72\.06×\\times48\.482\.82\.05×\\timesDustin237\.722\.94\.03×\\times205\.662\.75\.21×\\times175\.432\.95\.94×\\times148\.202\.86\.28×\\timesVanilla1677\.38\-1\.00×\\times46\.69\-1\.00×\\times33\.45\-1\.00×\\times26\.00\-1\.00×\\timesClassicSD151\.332\.71\.96×\\times87\.682\.61\.88×\\times61\.732\.81\.85×\\times47\.422\.91\.82×\\timesMagicDec162\.472\.72\.10×\\times98\.792\.62\.12×\\times71\.202\.82\.13×\\times55\.472\.82\.13×\\timesDustin407\.962\.95\.27×\\times291\.872\.86\.25×\\times226\.852\.86\.78×\\times186\.692\.97\.18×\\times

### 5\.4Ablation Study

To quantify the contribution of each attention source, we evaluate three Dustin variants\.Dustin\-Hdenotes the full hybrid design, which combines*target\-based historical*attention with*draft\-based lookahead*attention to identify important context tokens\. This variant uses 4 selected heads from the target model and 12 selected heads from the draft model\.Dustin\-Tdisables the*draft\-based lookahead*component and performs token selection using only*target\-based historical*attention, with 4 selected target heads\.Dustin\-Ddisables the*target\-based historical*component and performs token selection using only*draft\-based lookahead*attention, with 12 selected draft heads\.

#### 5\.4\.1Analysis of Importance Estimation Overhead

We evaluate online importance estimation latency across varying input lengths using Qwen2\.5\-72B \(target\) and Qwen2\.5\-0\.5B \(draft\), with a batch size of 8 and a KV budget of 512\. Fig\.[8](https://arxiv.org/html/2606.24957#S5.F8)reports normalized latency relative to a full\-cache forward pass, denoted asflash\-attn\. All values are normalized by the full\-cache self\-attention latency\. Quest\-16 denotes Quest with page size 16\. Naïve materializes attention scores from all heads in both the target and draft models\. Dustin\-H introduces only a negligible overhead compared to flash\-attn, while being much faster than Quest\-16 across all context lengths\. In contrast, Naïve materializes attention from all heads and layers and becomes cost\-prohibitive, exceeding full\-cache self\-attention beyond 4K tokens, underscoring the necessity of attention head selection for efficient online estimation\. Across the three head\-selection variants \(Dustin\-H, Dustin\-T, Dustin\-D\), the estimator remains lightweight, with Dustin\-H staying near∼\\sim1% of flash\-attn at 16K–32K\. Compared with Dustin\-T and Dustin\-D, Dustin incurs only a small additional cost to combine both signals; later results show this marginal overhead yields higher accuracy than either single\-source variant\.

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/Experiment/normalized_latency.png)Figure 8:Normalized Latency of Online Importance Estimation
#### 5\.4\.2Historical and Lookahead policy Comparison

Table[3](https://arxiv.org/html/2606.24957#S5.T3)compares the performance of our hybrid strategy against single\-source baselines\. By integrating complementary signals from both target\-historical and draft\-lookahead attention, Dustin\-H consistently obtains relatively good accuracy across diverse benchmarks\. In contrast, single\-source variants \(Dustin\-T and Dustin\-D\) exhibit performance fluctuations depending on the task nature\. This result demonstrates that the hybrid approach provides a stable mechanism for importance estimation, ensuring robust performance preservation where isolated signals might fail\. Results on additional datasets are provided in Appendix[K](https://arxiv.org/html/2606.24957#A11)\.

Table 3:Accuracy comparison of three Dustin variants \(Dustin\-T, Dustin\-D, and Dustin\-H\) on TriviaQA and MultiNews benchmarks\.Tri\. QAM\.NewsMethodAcc\.Δ\\DeltaAcc\.Δ\\DeltaDustin\-T77\.58%\-6\.41%23\.95%\-1\.12%Dustin\-D80\.51%\-3\.48%24\.84%\-0\.23%Dustin\-H83\.63%\-0\.36%25\.19%\+0\.12%

## 6Conclusion

We presentedDustin, a sparse verification framework that addresses the KV\-cache loading bottleneck limiting speculative decoding in long\-context, multi\-batch regimes\. Motivated by the observation that neither target\-historical nor draft\-lookahead attention is individually reliable across model scales and verification depths, Dustin fuses both signals to identify critical tokens with high fidelity, while restricting importance scoring to a small set of Semantic Retrieval Heads to keep online estimation overhead near0\.8%0\.8\\%of a full hybrid computation\. Evaluations on PG\-19 and LongBench across the Llama3 and Qwen2\.5 families show that Dustin delivers up to a27\.85×27\.85\\timesself\-attention speedup and a9\.17×9\.17\\timesend\-to\-end decoding speedup on Qwen2\.5\-72B at a 32k context length, with near\-lossless accuracy that consistently surpasses streaming and page\-level baselines, and an advantage that widens as batch size and context grow\. Two limitations point to future work: the budget\-allocation parameter is fixed via offline profiling, so an input\-aware dynamic scheme could further close the gap to lossless verification; and because Dustin indexes the full history rather than evicting tokens, it does not reduce the KV\-cache memory footprint, and therefore—unlike eviction methods—does not by itself allow longer sequences to fit within a fixed GPU memory budget\. We hope Dustin offers a practical foundation for high\-throughput long\-context inference\.

## Impact Statement

This paper presents work whose goal is to advance the field of machine learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1)\.
- AI@Meta \(2024\)Llama 3 model card\.External Links:[Link](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md)Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1),[§1](https://arxiv.org/html/2606.24957#S1.p5.1),[§5\.1](https://arxiv.org/html/2606.24957#S5.SS1.SSS0.Px1.p1.1)\.
- T\. Akiba, S\. Sano, T\. Yanase, T\. Ohta, and M\. Koyama \(2019\)Optuna: a next\-generation hyperparameter optimization framework\.InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining,pp\. 2623–2631\.Cited by:[item \(3\)](https://arxiv.org/html/2606.24957#S4.I1.i3.p1.1)\.
- Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.\(2024\)Longbench: a bilingual, multitask benchmark for long context understanding\.InProceedings of the 62nd annual meeting of the association for computational linguistics \(volume 1: Long papers\),pp\. 3119–3137\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p5.1),[§5\.1](https://arxiv.org/html/2606.24957#S5.SS1.SSS0.Px2.p1.1)\.
- Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1)\.
- C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. Jumper \(2023\)Accelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- Z\. Chen, A\. May, R\. Svirschevski, Y\. Huang, M\. Ryabinin, Z\. Jia, and B\. Chen \(2024\)Sequoia: scalable, robust, and hardware\-aware speculative decoding\.arXiv preprint arXiv:2402\.12374\.Cited by:[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- T\. Dao \(2023\)Flashattention\-2: faster attention with better parallelism and work partitioning\.arXiv preprint arXiv:2307\.08691\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1),[§4\.2\.3](https://arxiv.org/html/2606.24957#S4.SS2.SSS3.p1.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1)\.
- Y\. Leviathan, M\. Kalman, and Y\. Matias \(2023\)Fast inference from transformers via speculative decoding\.InInternational Conference on Machine Learning,pp\. 19274–19286\.Cited by:[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1)\.
- X\. Lin, J\. Wang, O\. Kondrateva, Y\. Shi, B\. Li, and G\. L\. Zhang \(2025\)CompressKV: semantic retrieval heads know what tokens are not important before generation\.arXiv preprint arXiv:2508\.02401\.Cited by:[Appendix I](https://arxiv.org/html/2606.24957#A9.SS0.SSS0.Px1.p1.1),[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1),[§4\.2\.1](https://arxiv.org/html/2606.24957#S4.SS2.SSS1.p1.1),[§4\.2](https://arxiv.org/html/2606.24957#S4.SS2.p1.2)\.
- Z\. Liu, A\. Desai, F\. Liao, W\. Wang, V\. Xie, Z\. Xu, A\. Kyrillidis, and A\. Shrivastava \(2023\)Scissorhands: exploiting the persistence of importance hypothesis for llm kv cache compression at test time\.Advances in Neural Information Processing Systems36,pp\. 52342–52364\.Cited by:[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1)\.
- X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi,et al\.\(2024\)Specinfer: accelerating large language model serving with tree\-based speculative inference and verification\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 932–949\.Cited by:[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- M\. Oren, M\. Hassid, N\. Yarden, Y\. Adi, and R\. Schwartz \(2024\)Transformers are multi\-state rnns\.arXiv preprint arXiv:2401\.06104\.Cited by:[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1)\.
- R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. Dean \(2023\)Efficiently scaling transformer inference\.Proceedings of machine learning and systems5,pp\. 606–624\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1)\.
- J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, and T\. P\. Lillicrap \(2019\)Compressive transformers for long\-range sequence modelling\.arXiv preprint arXiv:1911\.05507\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p5.1),[§5\.1](https://arxiv.org/html/2606.24957#S5.SS1.SSS0.Px2.p1.1)\.
- R\. Sadhukhan, J\. Chen, Z\. Chen, V\. Tiwari, R\. Lai, J\. Shi, I\. E\. Yen, A\. May, T\. Chen, and B\. Chen \(2024\)Magicdec: breaking the latency\-throughput tradeoff for long context generation with speculative decoding\.arXiv preprint arXiv:2408\.11049\.Cited by:[Figure 1](https://arxiv.org/html/2606.24957#S1.F1),[Figure 1](https://arxiv.org/html/2606.24957#S1.F1.8.2.1),[§1](https://arxiv.org/html/2606.24957#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- H\. Shah \(2025\)SpecAttn: speculating sparse attention\.arXiv preprint arXiv:2510\.27641\.Cited by:[§2\.3](https://arxiv.org/html/2606.24957#S2.SS3.p1.1),[§3\.3](https://arxiv.org/html/2606.24957#S3.SS3.p2.1),[§4\.1](https://arxiv.org/html/2606.24957#S4.SS1.p2.1)\.
- H\. Sun, Z\. Chen, X\. Yang, Y\. Tian, and B\. Chen \(2024\)Triforce: lossless acceleration of long sequence generation with hierarchical speculative decoding\.arXiv preprint arXiv:2404\.11912\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- J\. Tang, Y\. Zhao, K\. Zhu, G\. Xiao, B\. Kasikci, and S\. Han \(2024\)Quest: query\-aware sparsity for efficient long\-context llm inference\.arXiv preprint arXiv:2406\.10774\.Cited by:[Appendix A](https://arxiv.org/html/2606.24957#A1.p6.5),[§1](https://arxiv.org/html/2606.24957#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2606.24957#S4.SS1.p2.1)\.
- R\. Tiwari, H\. Xi, A\. Tomar, C\. Hooper, S\. Kim, M\. Horton, M\. Najibi, M\. W\. Mahoney, K\. Keutzer, and A\. Gholami \(2025\)QuantSpec: self\-speculative decoding with hierarchical quantized kv cache\.arXiv preprint arXiv:2502\.10424\.Cited by:[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- G\. Xiao, Y\. Tian, B\. Chen, S\. Han, and M\. Lewis \(2023\)Efficient streaming language models with attention sinks\.arXiv preprint arXiv:2309\.17453\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1)\.
- P\. Yang, C\. Du, F\. Zhang, H\. Wang, T\. Pang, C\. Du, and B\. An \(2025\)LongSpec: long\-context lossless speculative decoding with efficient drafting and verification\.InES\-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models,Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.24957#S2.SS1.p1.1)\.
- Q\. A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Qiu, S\. Quan, and Z\. Wang \(2024\)Qwen2\.5 technical report\.ArXivabs/2412\.15115\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1),[§1](https://arxiv.org/html/2606.24957#S1.p5.1),[§5\.1](https://arxiv.org/html/2606.24957#S5.SS1.SSS0.Px1.p1.1)\.
- Z\. Yuan, Y\. Shang, Y\. Zhou, Z\. Dong, Z\. Zhou, C\. Xue, B\. Wu, Z\. Li, Q\. Gu, Y\. J\. Lee,et al\.\(2024\)Llm inference unveiled: survey and roofline model insights\.arXiv preprint arXiv:2402\.16363\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p1.1)\.
- J\. Zhang, Z\. Hou, X\. Lv, S\. Cao, Z\. Hou, Y\. Niu, L\. Hou, Y\. Dong, L\. Feng, and J\. Li \(2025\)Longreward: improving long\-context large language models with ai feedback\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3718–3739\.Cited by:[§3\.1](https://arxiv.org/html/2606.24957#S3.SS1.SSS0.Px2.p3.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett,et al\.\(2023\)H2o: heavy\-hitter oracle for efficient generative inference of large language models\.Advances in Neural Information Processing Systems36,pp\. 34661–34710\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1)\.
- Y\. Zhao, Y\. Peng, C\. Nguyen, Z\. Li, X\. Wang, H\. Zhao, and X\. Fu \(2025\)Smallkv: small model assisted compensation of kv cache compression for efficient llm inference\.arXiv preprint arXiv:2508\.02751\.Cited by:[§1](https://arxiv.org/html/2606.24957#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.24957#S2.SS2.p1.1),[§3\.3](https://arxiv.org/html/2606.24957#S3.SS3.p2.1),[§3](https://arxiv.org/html/2606.24957#S3.p1.1)\.

## Appendix ADetailed online sparse estimation and complexity analysis

After identifying SRHs via offline profiling, we efficiently estimate token importance by aggregating attention scores solely from this small subset on\-the\-fly\. In practice, only the attention scores required by this subset are materialized and accumulated during inference, while all remaining heads and layers are processed normally\.

This sparsity is critical for maintaining high inference speed\. Since efficient attention kernels like FlashAttention utilize online softmax to avoid materializing the full attention matrix to global memory, explicitly storing token estimation scores fromalllayers and heads would induce significant I/O overhead\. Our method preserves the throughput advantages of these optimized kernels for the majority of the computation by restricting score materialization to selected SRHs\.

To quantify this benefit, consider a Speculative Decoding \(SD\) configuration with Qwen2\.5\-72B as the target model \(Ht=64,Lt=80H\_\{t\}=64,L\_\{t\}=80\) and Qwen2\.5\-0\.5B as the draft model \(Hd=14,Ld=24H\_\{d\}=14,L\_\{d\}=24\) speculating four tokens \(Γ=4\\Gamma=4\)\. By downsizing the estimator to 1 target layer with 4 heads and 3 draft layers with 4 heads \(denoted asL∗,H∗L^\{\*\},H^\{\*\}\), the overhead ratio relative to the full hybrid calculation is derived as follows:

Overhead Ratio=\(Hd∗⋅Ld∗⋅Γ\)\+\(Ht∗⋅Lt∗⋅1\)\(Hd⋅Ld⋅Γ\)\+\(Ht⋅Lt⋅1\)=\(4⋅3⋅4\)\+\(4⋅1⋅1\)\(14⋅24⋅4\)\+\(64⋅80⋅1\)=48\+41344\+5120≈0\.8%\\begin\{split\}\\text\{Overhead Ratio\}&=\\frac\{\(H\_\{d\}^\{\*\}\\cdot L\_\{d\}^\{\*\}\\cdot\\Gamma\)\+\(H\_\{t\}^\{\*\}\\cdot L\_\{t\}^\{\*\}\\cdot 1\)\}\{\(H\_\{d\}\\cdot L\_\{d\}\\cdot\\Gamma\)\+\(H\_\{t\}\\cdot L\_\{t\}\\cdot 1\)\}\\\\ &=\\frac\{\(4\\cdot 3\\cdot 4\)\+\(4\\cdot 1\\cdot 1\)\}\{\(14\\cdot 24\\cdot 4\)\+\(64\\cdot 80\\cdot 1\)\}\\\\ &=\\frac\{48\+4\}\{1344\+5120\}\\approx 0\.8\\%\\end\{split\}\(7\)
This minimal computational footprint ensures that the sparse verification process introduces negligible latency\.

Furthermore, it is important to distinguish our optimization strategy from methods that target the sequence length dimension \(NN\)\. While our work reduces computational overhead by sparsifying the model architecture dimensions—specifically the number of heads \(HH\) and layers \(LL\)—other recent works, such as Quest\(Tanget al\.,[2024](https://arxiv.org/html/2606.24957#bib.bib15)\), focus on compressing the context dimension during the importance estimation phase\. Specifically, Quest aggregates token\-level Key statistics \(e\.g\., minimum and maximum values\) into page\-level metadata\. This allows it to estimate the relevance of a block of tokens using a single computation, effectively reducing the sequence length dimension for estimation fromNNtoN/page sizeN/\\text\{page size\}\. Our Semantic Retrieval Head \(SRH\) method is orthogonal to such dimension\-reduction techniques\. Whereas Quest compresses the effective sequence length for estimation, we minimize the number of attention mechanisms required\. Consequently, our approach focuses on lightweight architectural profiling rather than dynamic context compression, though both directions could potentially be combined for further efficiency\.

## Appendix BRelationship Between ARR and Output Distribution Distortion

To further validate whether ARR reflects the fidelity of sparse KV selection, we conduct an additional controlled experiment that measures the relationship between ARR and output distribution distortion\. Instead of evaluating a specific KV\-selection policy, we sample KV subsets with different retained attention mass and compare the resulting sparse forward pass against a full\-cache reference\. This removes the bias introduced by any particular selection algorithm and isolates the intrinsic relationship between retained attention mass and output distribution distortion\.

For each model and input sample, we first run a full\-cache forward pass and greedily generate a fixed sequence of tokens\. During this reference run, we store the full\-cache logits and the corresponding attention distribution at each decoding step\. We then replay the same generated token sequence under sparse KV caches with sampled KV subsets spanning different ARR ranges\. For each sparse forward pass, we compute two metrics: ARR, which measures the fraction of full\-cache attention mass retained by the selected KV tokens, and the KL divergence between the next\-token distributions induced by the sparse\-cache and full\-cache logits\.

Formally, letpifullp\_\{i\}^\{\\mathrm\{full\}\}andpisparsep\_\{i\}^\{\\mathrm\{sparse\}\}denote the next\-token probability distributions produced by the full\-cache and sparse\-cache forward passes at decoding stepii, respectively\. We measure output distribution distortion using

DKL​\(pifull∥pisparse\)\.D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\_\{i\}^\{\\mathrm\{full\}\}\\,\\\|\\,p\_\{i\}^\{\\mathrm\{sparse\}\}\\right\)\.\(8\)For each sampled KV subset, we average ARR and KL divergence across decoding steps, and then compute the Pearson correlation between the averaged ARR and averaged KL divergence\.

We run this experiment on a GovReport sample using bothLlama\-3\.1\-8B\-InstructandQwen2\.5\-7B\-Instruct\. The KV budget is fixed to 512 tokens, with 4 sink tokens and 16 recent tokens always preserved\. The resulting ARR–KL correlations are shown in Table[4](https://arxiv.org/html/2606.24957#A2.T4)\. ARR exhibits a consistent negative correlation with KL divergence across both models, indicating that higher ARR generally corresponds to lower output distribution distortion\.

Table 4:Correlation between ARR and output\-logit KL divergence\. Higher ARR is associated with lower output distribution distortion\.ModelARR–KL CorrelationLlama\-3\.1\-8B\-Instruct\-0\.9490Qwen2\.5\-7B\-Instruct\-0\.8079![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/arr_kld.png)Figure 9:Relationship between Attention Recovery Rate \(ARR\) and output\-logit KL divergence\. The left and right panels show results forLlama\-3\.1\-8B\-InstructandQwen2\.5\-7B\-Instruct, respectively\. Each point corresponds to one sampled KV subset, with ARR and KL divergence averaged across decoding steps\. The negative trend indicates that higher ARR generally leads to lower output distribution distortion\.These results support ARR as a meaningful proxy for sparse\-forward fidelity\. Although ARR is computed purely from attention mass, it is predictive of output distribution distortion: preserving more attention mass leads to logits that are closer to the full\-cache reference\. Fig\.[9](https://arxiv.org/html/2606.24957#A2.F9)visualizes this relationship, while Table[4](https://arxiv.org/html/2606.24957#A2.T4)reports the corresponding Pearson correlations\.

## Appendix CDetailed Experimental Setup

### C\.1Models

##### Extending Qwen2\.5 Context Length to 64K via YaRN

All Qwen2\.5 models are configured to support up to 64K context length by enabling YaRN\-based RoPE scaling, following the official recommendation for length extrapolation beyond the default 32,768\-token setting\. Concretely, YaRN is activated through the model configuration \(e\.g\., a RoPE scaling factor of 2 with the original maximum position embeddings set to 32,768 and scaling type set toyarn\), and all Qwen2\.5 experiments that require long contexts use this setup\.

### C\.2Implementation Details

##### FlashAttention\-v2 MagicDec for Fair Efficiency Comparison

To ensure a fair and controlled efficiency comparison, a FlashAttention v2 version of MagicDec is implemented in\-house\. This removes kernel\-level confounders and aligns the attention backend across efficiency baselines, so the measured throughput and latency differences primarily reflect algorithmic design \(e\.g\., speculation and KV handling\) rather than disparate attention implementations\.

##### Hardware Setup and Pipeline Parallelism

All efficiency\-evaluation experiments run on NVIDIA H200 GPUs with bfloat16 precision and pipeline parallelism\. Model deployment follows a size\-dependent GPU allocation: \(i\)4×\\timesH200forQwen2\.5\-72B,Qwen2\.5\-32B, andLlama\-3\.3\-70B; \(ii\)2×\\timesH200forQwen2\.5\-14B,Qwen2\.5\-7B, andLlama\-3\.1\-8B\. This hardware policy is applied consistently across the corresponding efficiency baselines\.

##### Dustin KV Selection Configuration

For Dustin, the protected token set consists of 4 attention sink tokens and a recent window of 16 tokens\. These protected tokens are always retained and are counted as part of the total KV budgetkk\. The remaining budget is allocated to tokens selected by the hybrid draft\-lookahead and target\-historical attention signals following Eq\.[6](https://arxiv.org/html/2606.24957#S4.E6)\.

### C\.3Accuracy baselines setup

##### StreamingLLM Configuration

For baselines utilizing the StreamingLLM strategy, we configure the KV cache to maintain a fixed budgetkk\. This policy strictly retains the first 4 tokens as attention sinks and the most recentk−4k\-4tokens as a sliding local window\.

##### Quest Configuration for Accuracy Evaluation

For LongBench accuracy evaluation, Quest is configured with a page size of16\(i\.e\., block granularity of 16 tokens\) for its block\-level selection\. Following the official Quest practice,*layers 0–1 do not apply KV compression/selection*; block\-level selection is only enabled for higher layers\.

### C\.4Efficiency baselines setup

##### Speculative Decoding Configuration\.

For all speculative\-decoding efficiency baselines, we pair each target model with a lightweight draft model from the same model family\. For a given target model, the same draft model is used across Classic SD, MagicDec, and Dustin, so that the comparison isolates the effect of the verification strategy rather than differences in drafting quality or draft\-model cost\.

All methods use standard sequential draft blocks rather than tree\-based speculation\. At each speculative step, the draft model generates a block ofΓ\\Gammaconsecutive draft tokens, which are then verified in parallel by the target model\. The speculative depthΓ\\Gammais fixed for each target model and shared across Classic SD, MagicDec, and Dustin\. The complete per\-model draft configuration is summarized in Table[5](https://arxiv.org/html/2606.24957#A3.T5)\.

Table 5:Speculative decoding configurations used in efficiency experiments\.Target modelDraft modelDraft structureDepthΓ\\GammaQwen2\.5\-72B\-InstructQwen2\.5\-0\.5B\-InstructSequential4Qwen2\.5\-32B\-InstructQwen2\.5\-0\.5B\-InstructSequential4Qwen2\.5\-14B\-InstructQwen2\.5\-0\.5B\-InstructSequential3Qwen2\.5\-7B\-InstructQwen2\.5\-0\.5B\-InstructSequential3Llama\-3\.3\-70B\-InstructLlama\-3\.2\-1B\-InstructSequential4Llama\-3\.1\-8B\-InstructLlama\-3\.2\-1B\-InstructSequential3
##### Draft KV\-cache Configuration for Classic SD and MagicDec\.

Classic SD follows the standard speculative decoding procedure, where the draft model maintains a full KV cache over the entire input context\. In contrast, MagicDec reduces draft\-stage latency by applying StreamingLLM\-style KV retention to the draft model\. Specifically, MagicDec keeps the first 4 attention\-sink tokens and the most recentk−4k\-4tokens as a sliding local window in the draft\-model KV cache\. Dustin follows the same draft\-model configuration as the corresponding speculative\-decoding baseline for each target model, while applying sparse verification on the target\-model side\.

##### Target\-side Verification Configuration\.

Classic SD and MagicDec verify draft tokens using the full target\-model KV cache and therefore preserve lossless target\-side verification\. Dustin instead performs target\-side sparse verification using the selected verification setℐverify\\mathcal\{I\}\_\{\\text\{verify\}\}described in Sec\.[4\.1](https://arxiv.org/html/2606.24957#S4.SS1)\. Unless otherwise specified, the target\-side KV budget is set tok=512k=512in efficiency experiments\. This setting is shared across Dustin’s sparse verification experiments and is independent of the speculative depthΓ\\Gamma\.

## Appendix DAdditional Accuracy Results on Llama3 and Qwen2\.5 Model Series

Table[6](https://arxiv.org/html/2606.24957#A4.T6)and Table[7](https://arxiv.org/html/2606.24957#A4.T7)report detailed LongBench accuracy results on Llama\-3\.1\-8B\-Instruct and additional Qwen2\.5 models \(32B, 14B, and 7B\) under multiple constrained KV\-cache budgets \(512/256/128\)\. These experiments complement the main\-text evaluation on Llama\-3\.3\-70B\-Instruct and Qwen2\.5\-72B\-Instruct models and assess whether the effectiveness of Dustin generalizes across model scales and model families\.

Across all sizes and budgets, Dustin consistently maintains accuracy close to the FullKV oracle while substantially outperforming streaming baselines and block\-level selection \(Quest\)\. The advantage becomes more pronounced as the KV budget tightens, where competing methods suffer noticeable degradation, whereas Dustin preserves most task performance\.

This trend is stable across diverse task categories, including single\- and multi\-document QA, summarization, few\-shot learning, synthetic retrieval, and code generation, indicating that the proposed hybrid criticality estimation remains effective under both capacity\-constrained and scale\-varying regimes\.

Overall, the results indicate that the benefits of Dustin are not limited to a specific model size or model family, but transfer reliably to Llama\-3\.1\-8B\-Instruct and across Qwen2\.5 models \(7B–32B\), confirming the robustness and scalability of the proposed verification strategy\.

Table 6:Performance comparison of Dustin with StreamingLLM, Quest and FullKV on LongBench for Llama3 series models \(Llama\-3\.1\-8B\-Instruct\)Target ModelCompression MethodKVBudgetSingle\-doc QAMulti\-doc QASummarizationFew\-shotSyntheticCodeAvg\.Llama\-3\.1\-8B\-InstructClassicSD / MagicDec—42\.3444\.1529\.1169\.3852\.7559\.7250\.58StreamingLLM51228\.3530\.1123\.1650\.5147\.1138\.3136\.00Quest33\.6034\.7027\.2155\.9451\.3354\.9243\.90Dustin42\.2644\.2628\.6369\.0852\.7559\.6850\.45StreamingLLM25625\.8429\.7321\.1647\.6047\.0636\.8334\.39Quest27\.6124\.6424\.4548\.6748\.3749\.8638\.15Dustin42\.1544\.3127\.7268\.7752\.5958\.1149\.81StreamingLLM12825\.0229\.4619\.2543\.9445\.2335\.7932\.86Quest18\.5318\.3219\.9634\.1941\.4340\.5429\.54Dustin41\.9543\.9925\.0368\.4252\.8457\.1549\.02

Table 7:Performance comparison of Dustin with StreamingLLM, Quest and FullKV on LongBench for Qwen2\.5 series models \(Qwen2\.5\-32B\-Instruct, Qwen2\.5\-14B\-Instruct, Qwen2\.5\-7B\-Instruct\)Target ModelCompression MethodKVBudgetSingle\-doc QAMulti\-doc QASummarizationFew\-shotSyntheticCodeAvg\.Qwen2\.5\-32B\-InstructClassicSD / MagicDec—42\.4354\.5327\.3767\.5756\.0042\.7847\.52StreamingLLM51224\.3227\.7920\.9045\.7512\.5027\.4727\.33Quest31\.3036\.8824\.1659\.5554\.9740\.0040\.41Dustin41\.3754\.5026\.6066\.7556\.2541\.7346\.85StreamingLLM25620\.8727\.1318\.5742\.5113\.0026\.2525\.56Quest23\.5831\.5521\.0052\.7350\.6439\.5436\.25Dustin41\.2954\.1025\.7164\.8753\.5040\.5345\.72StreamingLLM12819\.9426\.3615\.9340\.3513\.2525\.8924\.46Quest18\.4922\.3616\.8743\.3846\.2234\.1529\.97Dustin41\.1253\.7024\.7662\.8137\.9239\.7743\.28Qwen2\.5\-14B\-InstructClassicSD / MagicDec—42\.1652\.9527\.3671\.6455\.8859\.2952\.27StreamingLLM51223\.7928\.3620\.2547\.985\.3632\.3828\.15Quest29\.2730\.5125\.0460\.6649\.7343\.6739\.83Dustin41\.1152\.3626\.6870\.8855\.7058\.1551\.46StreamingLLM25620\.9126\.2718\.2444\.458\.7532\.1226\.80Quest21\.6423\.0223\.2753\.3743\.8838\.2434\.00Dustin40\.2151\.1926\.1869\.8854\.4256\.4350\.31StreamingLLM12819\.6826\.0616\.0041\.4513\.5030\.3625\.78Quest15\.4114\.3420\.4744\.2233\.0835\.5727\.92Dustin40\.5949\.7525\.5769\.4938\.8653\.7247\.59Qwen2\.5\-7B\-InstructClassicSD / MagicDec—40\.5444\.2827\.7669\.9253\.5064\.1151\.46StreamingLLM51223\.0324\.2121\.9845\.219\.5034\.5628\.23Quest23\.6121\.7124\.8352\.6140\.1547\.9336\.39Dustin40\.2143\.8327\.4568\.7553\.5061\.4550\.40StreamingLLM25619\.0522\.9819\.4742\.8911\.5032\.7426\.40Quest19\.9715\.9322\.5545\.0534\.2442\.2631\.21Dustin40\.4143\.1727\.3867\.8949\.2560\.4249\.44StreamingLLM12819\.3223\.1417\.1941\.0312\.2531\.4425\.54Quest14\.5410\.2919\.3534\.6126\.0235\.7924\.73Dustin39\.8742\.7526\.7667\.8435\.5055\.0446\.29

## Appendix EAdditional Accuracy Results Compared with SmallKV

Given the difference in base models, we focus our comparison on the accuracy drop relative to the respective FullKV baselines rather than absolute scores\. As shown in Table[8](https://arxiv.org/html/2606.24957#A5.T8), our method outperforms SmallKV in four out of five benchmarks, demonstrating significantly smaller performance degradation\. Notably, in Single\-doc QA, our method achieves near\-lossless performance with a negligible drop of 0\.33, whereas SmallKV suffers a 5\.9\-point decline\. Similar trends are observed in Multi\-doc QA, Summarization, and Few\-shot tasks, where our method consistently maintains higher fidelity to the original baseline\. The only exception is the Code benchmark, where SmallKV retains better performance\.

Table 8:Performance comparison of Dustin with SmallKV and FullKV on LongBench for Qwen2 series models \(Qwen2\-7B, Qwen2\.5\-7B\-Instruct\)Target CompressionMethodKVBudgetSingle\-doc QAMulti\-doc QASummarizationFew\-shotCodeQwen2\-7BVanilla / Lossless SD—39\.0440\.7822\.3170\.0756\.95SmallKV≈\\approx51233\.14\(\-5\.9\)37\.54\(\-3\.24\)21\.55\(\-0\.76\)68\.39\(\-1\.68\)56\.83\(\-0\.12\)Qwen2\.5\-7B\-InstructVanilla / SD Methods—40\.5444\.2827\.7669\.9264\.11Dustin51240\.21\(\-0\.33\)43\.83\(\-0\.45\)27\.45\(\-0\.31\)68\.75\(\-1\.17\)61\.45\(\-2\.66\)

## Appendix FAdditional Efficiency Evaluation Compared with SpecAttn

We further compare Dustin with SpecAttn, a sparse verification method that selects KV tokens using attention scores computed during speculative decoding\. Since SpecAttn does not provide an official open\-source implementation, we implement a SpecAttn\-style baseline based on our Dustin codebase\. Specifically, we modify Dustin’s selection procedure to compute token importance using attention scores from every layer and every head of the draft model, following the per\-layer and per\-head relevance estimation strategy of SpecAttn, instead of using Dustin’s hybrid global aggregation design\.

Table[9](https://arxiv.org/html/2606.24957#A6.T9)reports the measured speedups onQwen2\.5\-72B\-Instructwith batch size 16\. Dustin consistently outperforms SpecAttn across all evaluated context lengths, achieving higher speedups at 8K, 16K, and 32K contexts\.

Table 9:Efficiency comparison between SpecAttn and Dustin across different context lengths\. Results are measured on Qwen2\.5\-72B\-Instruct with batch size 16\.MethodBatch SizeContext Length8K16K32KSpecAttn163\.91×\\times5\.62×\\times7\.11×\\timesDustin164\.76×\\times7\.06×\\times9\.17×\\times
## Appendix GMore Self\-Attention Latency Breakdown

Figure[10](https://arxiv.org/html/2606.24957#A7.F10)presents the latency decomposition across four distinct workload configurations on Qwen2\.5\-72B\. We isolate theCriticality Estimation Overhead—which encompasses the time spent onSRH scoring\(utilizing the configuration determined viaoffline layer search\) to identify the optimal verification set—from theSparse Verification Attention, which represents the actual computation of the target model attending to the selected 512 tokens\.

##### Speedup Analysis

Dustin significantly accelerates thefull cache speculative verification self\-attentionbaseline\. Using a fixed 512\-token budget, speedups scale with workload:9\.35×9\.35\\times\(Batch 8\) and15\.58×15\.58\\times\(Batch 16\) at 16k context, rising to16\.57×16\.57\\timesand27\.85×27\.85\\timesrespectively at 32k\. This confirms that sparse verification effectively alleviates memory bandwidth bottlenecks in long\-context, large\-batch scenarios\.

##### Criticality Estimation Overhead

The online estimation overhead is minimal, constituting only0\.46%0\.46\\%–0\.75%0\.75\\%of the full cache latency\. Even relative to the sparse verification step, the overhead remains moderate \(7\.53%7\.53\\%–14\.88%14\.88\\%\)\. This demonstrates that the cost of SRH scoring and layer search is negligible compared to the massive savings in attention computation\.

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/Experiment/latency_figure_fixed120_4panels.png)Figure 10:Detailed latency breakdown of the target model verification phase on Qwen2\.5\-72B\. The charts compare the latency ofFull Cache\(Baseline\) againstDustin, decomposing the latter intoCriticality Estimation\(light blue\) andApproximate Attention\(dark blue\)\. Our estimation overhead remains negligible across all settings, while the sparse attention yields massive latency reductions, particularly at longer context lengths \(32k\) and larger batch sizes \(16\)\.

## Appendix HAdditional End\-to\-End Decode\-Stage Throughput on the Llama3 and Qwen2\.5 Model Series

To extensively validate the scalability of our approach, we report end\-to\-end decode\-stage throughput on the Llama3 and Qwen2\.5 model families across various scales \(70B, 32B, 14B, 8B, 7B\)\. Table[10](https://arxiv.org/html/2606.24957#A8.T10)and Table[11](https://arxiv.org/html/2606.24957#A8.T11)present the results for the Llama3 series and Qwen2\.5 series, respectively\.

##### Performance on Llama3 Series

As shown in Table[10](https://arxiv.org/html/2606.24957#A8.T10), Dustin consistently outperforms all baselines onLlama\-3\.1\-8B\-Instruct, where it achieves up to3\.17×3\.17\\timesspeedup at 32K context with batch size 16, surpassing MagicDec \(1\.89×1\.89\\times\)\. Notably, as the context length increases from 8K to 32K, the throughput of the Vanilla baseline drops significantly \(e\.g\., from 215\.30 to 67\.42 tokens/s at batch 16\), reflecting the severe KV cache bottleneck\. In contrast, Dustin maintains significantly higher throughput, demonstrating its ability to mitigate long\-context overhead\. Compared to ClassicSD and MagicDec, which plateau below1\.9×1\.9\\timeseven at 32K, Dustin’s advantage becomes increasingly prominent at longer contexts\.

##### Performance on Qwen2\.5 Series

Table[11](https://arxiv.org/html/2606.24957#A8.T11)details the results forQwen2\.5\-32B,14B, and7B\. Dustin demonstrates strong scalability across all sizes\. ForQwen2\.5\-32B, the speedup reaches an impressive7\.81×7\.81\\timesat 32K context \(batch 16\), far exceeding the∼2\.2×\\sim 2\.2\\timesspeedup of MagicDec\. Even for the smallerQwen2\.5\-7B, where the compute ratio between the target and draft models is less favorable for speculative decoding, Dustin still achieves a substantial2\.33×2\.33\\timesspeedup at 32K context \(batch 16\)\. This contrasts with ClassicSD and MagicDec, which struggle to exceed1\.4×1\.4\\timesin the same setting\.

##### Impact of Batch Size on Scalability

A consistent finding across both tables is the positive correlation between batch size and speedup\. Increasing the batch size from 8 to 16 amplifies the relative speedup of Dustin \(e\.g\., on Qwen2\.5\-32B at 32K,α\\alphaimproves from5\.85×5\.85\\timesto7\.81×7\.81\\times\)\. This phenomenon confirms that our sparse verification strategy is particularly effective in bandwidth\-constrained regimes \(large batch, long context\), where reducing the volume of KV cache transfers yields the highest returns\.

Table 10:Efficiency evaluation on Llama\-3\.1\-8B\-Instruct across different context lengths and batch sizesMethodBatch SizeContext Length8K16K24K32Ktputτ\\tauα\\alphatputτ\\tauα\\alphatputτ\\tauα\\alphatputτ\\tauα\\alphaLlama\-3\.1\-8B\-InstructVanilla8169\.70\-1\.00×\\times110\.40\-1\.00×\\times80\.31\-1\.00×\\times63\.13\-1\.00×\\timesClassicSD215\.112\.61\.27×\\times149\.502\.71\.35×\\times107\.712\.81\.34×\\times84\.802\.61\.34×\\timesMagicDec236\.272\.51\.39×\\times171\.802\.61\.56×\\times133\.372\.71\.66×\\times110\.052\.51\.74×\\timesDustin313\.302\.71\.85×\\times264\.582\.72\.40×\\times214\.072\.82\.67×\\times180\.142\.72\.85×\\timesVanilla16215\.30\-1\.00×\\times124\.43\-1\.00×\\times87\.21\-1\.00×\\times67\.42\-1\.00×\\timesClassicSD289\.062\.61\.34×\\times168\.332\.71\.35×\\times117\.922\.71\.35×\\times91\.362\.71\.36×\\timesMagicDec337\.752\.51\.57×\\times217\.742\.61\.75×\\times160\.962\.61\.85×\\times127\.112\.61\.89×\\timesDustin510\.882\.72\.37×\\times351\.172\.72\.82×\\times266\.082\.83\.05×\\times213\.642\.83\.17×\\timesTable 11:Efficiency evaluation on Qwen2\.5\-32B, 14B, and 7B\-Instruct across different context lengths and batch sizesMethodBatch SizeContext Length8K16K24K32Ktputτ\\tauα\\alphatputτ\\tauα\\alphatputτ\\tauα\\alphatputτ\\tauα\\alphaQwen2\.5\-32B\-InstructVanilla880\.62\-1\.00×\\times52\.17\-1\.00×\\times38\.56\-1\.00×\\times30\.49\-1\.00×\\timesClassicSD125\.092\.31\.55×\\times88\.452\.31\.70×\\times68\.152\.31\.77×\\times56\.042\.31\.84×\\timesMagicDec128\.092\.21\.59×\\times90\.542\.11\.74×\\times69\.522\.21\.80×\\times57\.612\.21\.89×\\timesDustin205\.002\.32\.54×\\times200\.552\.33\.84×\\times186\.892\.24\.85×\\times178\.352\.25\.85×\\timesVanilla16102\.24\-1\.00×\\times60\.32\-1\.00×\\times42\.73\-1\.00×\\times33\.14\-1\.00×\\timesClassicSD192\.572\.21\.88×\\times122\.972\.42\.04×\\times85\.882\.32\.01×\\times68\.162\.42\.06×\\timesMagicDec193\.292\.11\.89×\\times127\.522\.32\.11×\\times92\.102\.32\.16×\\times74\.702\.32\.25×\\timesDustin426\.062\.34\.17×\\times379\.822\.46\.30×\\times309\.212\.27\.24×\\times258\.692\.27\.81×\\timesQwen2\.5\-14B\-InstructVanilla8113\.99\-1\.00×\\times73\.24\-1\.00×\\times53\.33\-1\.00×\\times41\.96\-1\.00×\\timesClassicSD146\.872\.11\.29×\\times96\.792\.01\.32×\\times77\.292\.21\.45×\\times61\.232\.11\.46×\\timesMagicDec147\.942\.11\.30×\\times101\.752\.01\.39×\\times79\.722\.11\.49×\\times63\.522\.01\.51×\\timesDustin233\.192\.22\.05×\\times221\.472\.13\.02×\\times208\.282\.13\.91×\\times194\.362\.14\.63×\\timesVanilla16142\.80\-1\.00×\\times82\.77\-1\.00×\\times58\.14\-1\.00×\\times44\.87\-1\.00×\\timesClassicSD212\.732\.21\.49×\\times130\.652\.01\.58×\\times94\.752\.11\.63×\\times72\.652\.11\.62×\\timesMagicDec218\.782\.11\.53×\\times137\.912\.01\.67×\\times102\.092\.11\.76×\\times80\.342\.11\.79×\\timesDustin443\.912\.13\.11×\\times397\.192\.14\.80×\\times330\.632\.25\.69×\\times278\.112\.16\.20×\\timesQwen2\.5\-7B\-InstructVanilla8255\.05\-1\.00×\\times206\.22\-1\.00×\\times157\.37\-1\.00×\\times126\.35\-1\.00×\\timesClassicSD244\.142\.30\.96×\\times194\.802\.40\.94×\\times158\.032\.41\.00×\\times132\.622\.31\.05×\\timesMagicDec247\.252\.10\.97×\\times202\.912\.30\.98×\\times171\.162\.31\.09×\\times145\.942\.11\.16×\\timesDustin300\.732\.31\.18×\\times280\.572\.41\.36×\\times257\.392\.31\.64×\\times239\.762\.31\.90×\\timesVanilla16402\.94\-1\.00×\\times249\.36\-1\.00×\\times180\.80\-1\.00×\\times141\.47\-1\.00×\\timesClassicSD385\.722\.40\.96×\\times270\.662\.41\.09×\\times198\.472\.41\.10×\\times155\.422\.31\.10×\\timesMagicDec402\.302\.21\.00×\\times296\.982\.31\.19×\\times235\.082\.31\.30×\\times194\.382\.11\.37×\\timesDustin553\.792\.31\.37×\\times486\.502\.31\.95×\\times393\.902\.32\.18×\\times329\.612\.22\.33×\\times

## Appendix IPorting overhead to new models

Dustin requires a small one\-time offline cost when porting to a new target–draft model pair\. This cost comes from two stages: \(i\) identifying Semantic Retrieval Heads \(SRHs\) for the target model, and \(ii\) collecting full cache attention traces and running the sparse verification configuration search in Algorithm[1](https://arxiv.org/html/2606.24957#alg1)\. Importantly, this procedure is performed once per model pair and does not require per\-task or per\-dataset recalibration\.

##### SRH identification cost\.

We follow the SRH identification procedure of CompressKV\(Linet al\.,[2025](https://arxiv.org/html/2606.24957#bib.bib30)\), where retrieval\-oriented heads are identified once for a model and then reused across downstream tasks\. To quantify the practical cost, we re\-measured the SRH identification pipeline on Llama\-3\.1\-8B\-Instruct using a single A100 GPU\. As shown in Table[12](https://arxiv.org/html/2606.24957#A9.T12), the profiling time remains within tens of minutes even for long calibration contexts\. Specifically, the identification takes 13 minutes at 16K input length, 19 minutes at 32K, and 27 minutes at 50K\. This indicates that SRH profiling is a lightweight offline preparation step rather than a deployment\-time bottleneck\.

Table 12:Offline SRH identification time on Llama\-3\.1\-8B\-Instruct using a single A100 GPU\.Max Input LengthIdentification Time16K13 min32K19 min50K27 min
##### Dustin configuration search cost\.

After SRHs are identified, Dustin performs a one\-time model\-pair\-specific configuration search\. This stage consists of a full cache run to collect target\-historical and draft\-lookahead attention traces, followed by the search procedure described in Algorithm[1](https://arxiv.org/html/2606.24957#alg1)\. Table[13](https://arxiv.org/html/2606.24957#A9.T13)reports the measured overhead across several target–draft pairs\. On a single H100 GPU, the total cost is 11\.9 minutes for Llama\-3\.1\-8B with a 1B draft, 13\.7 minutes for Qwen2\.5\-7B with a 0\.5B draft, and 18\.1 minutes for Qwen2\.5\-14B with a 0\.5B draft\. For larger models evaluated with two H100 GPUs, the total cost is 24\.1 minutes for Qwen2\.5\-32B and 35\.4 minutes for Qwen2\.5\-72B\. These results show that even for 70B\-scale models, the additional Dustin\-specific porting overhead remains below one hour\.

Table 13:One\-time Dustin porting overhead after SRH identification\. The overhead consists of full cache attention collection and Algorithm[1](https://arxiv.org/html/2606.24957#alg1)configuration search\.Target \+ Draft Model PairHardwareAttention CollectionSearchTotalLlama\-3\.1\-8B \+ 1BH100×\\times17\.5 min4\.4 min11\.9 minQwen2\.5\-7B \+ 0\.5BH100×\\times16\.6 min7\.1 min13\.7 minQwen2\.5\-14B \+ 0\.5BH100×\\times110\.6 min7\.5 min18\.1 minQwen2\.5\-32B \+ 0\.5BH100×\\times215\.8 min8\.3 min24\.1 minQwen2\.5\-72B \+ 0\.5BH100×\\times226\.6 min8\.8 min35\.4 min
##### No per\-task recalibration\.

The above cost is amortized across all future inference workloads using the same target–draft pair\. Once the SRH set, selected layers, and draft–target budget split are obtained, Dustin keeps them fixed during deployment\. This is consistent with our robustness analysis, where SRH selection remains stable under changes in calibration size, haystack source, and context length\. Therefore, porting Dustin to a new model mainly requires a short offline profiling phase, while downstream tasks can directly reuse the same configuration without additional task\-specific tuning\.

## Appendix JAdditional Generalization Analysis of Draft\-Lookahead Signals

This appendix provides additional evidence for the observations in Sec\.[3\.3](https://arxiv.org/html/2606.24957#S3.SS3)and Sec\.[3\.4](https://arxiv.org/html/2606.24957#S3.SS4)\. In the main text, we use Qwen2\.5\-0\.5B as the draft model to analyze the inconsistency of draft\-lookahead signals across Qwen2\.5 target models\. Here, we extend the analysis to additional draft scales and model families to better characterize when draft\-lookahead attention is reliable and when the hybrid target\-history/draft\-lookahead strategy is needed\.

### J\.1Experimental Setup

We follow the same ARR\-based evaluation framework as Sec\.[3\.1](https://arxiv.org/html/2606.24957#S3.SS1)\. For each target–draft pair, the draft model generates speculative tokens and provides lookahead attention scores, which are used to rank context KV positions\. The selected top\-kkpositions are then evaluated against the SD oracle defined in Eq\.[4](https://arxiv.org/html/2606.24957#S3.E4)\.

For this appendix analysis, we use 200 evaluation samples from thezai\-org/LongReward\-10kdataset\. The average context length is approximately 13\.5K tokens, and the maximum output length is set to 512 tokens\. We set the verification window length tow=4w=4and use the same KV\-cache budgets as in the main observation experiments\.

We evaluate the following target–draft model pairs:

- •Qwen2\.5 pairs:We use Qwen2\.5\-0\.5B and Qwen2\.5\-1\.5B as draft models, each paired with Qwen2\.5\-7B, Qwen2\.5\-14B, Qwen2\.5\-32B, and Qwen2\.5\-72B as target models\.
- •Llama3 pairs:We use Llama\-3\.2\-1B and Llama\-3\.2\-3B as draft models, each paired with Llama\-3\.1\-8B and Llama\-3\.3\-70B as target models\.

The goal of this analysis is not to exhaustively tune every target–draft pair, but to examine whether the non\-monotonic behavior observed in Fig\.[4](https://arxiv.org/html/2606.24957#S3.F4)persists across different draft scales and architectures\.

### J\.2Cross\-Model ARR with Additional Draft Scales

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation2_Sensitivity/200/attention_recovery_comparison_all12.png)Figure 11:Additional cross\-model ARR analysis across Qwen2\.5 and Llama3 target–draft pairs\. We compare oracle ARR with draft\-lookahead ARR using Qwen2\.5\-0\.5B, 1\.5B and Llama\-3\.2\-1B, 3B as draft models across their corresponding larger target models\. The results further characterize the model\-pair\-dependent reliability of draft\-lookahead signals\.Fig\.[11](https://arxiv.org/html/2606.24957#A10.F11)reports the ARR of draft\-lookahead selection across all target–draft pairs in our appendix setup, covering both Qwen2\.5 and Llama3 families\. The results show that draft\-lookahead reliability varies significantly across model pairs\. In the Qwen2\.5 family, Qwen2\.5\-14B and Qwen2\.5\-32B still show a clear gap from the oracle even with a larger 1\.5B draft, while Qwen2\.5\-72B aligns much more closely with the draft signal\. In contrast, the Llama3 pairs maintain consistently small gaps between draft\-lookahead ARR and oracle ARR\. These results indicate that draft\-lookahead attention can provide useful future\-aware signals, but its reliability is not uniform across model families or target–draft pairs\. This supports the need for a hybrid strategy that incorporates target\-side history when draft\-side signals are unreliable\.

### J\.3Layer\-Wise Complementarity on High\-Gap Model Pairs

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation3_Sensitivity/200/Observation3_LayerWise_Recovery_LongRange_Qwen2.5-14B+0.5B_MaxPooled_y01.png)\(a\)Qwen2\.5\-14B & 0\.5B
![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation3_Sensitivity/200/Observation3_LayerWise_Recovery_LongRange_Qwen2.5-32B+0.5B_MaxPooled_y01.png)\(b\)Qwen2\.5\-32B & 0\.5B
![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation3_Sensitivity/200/Observation3_LayerWise_Recovery_LongRange_Qwen2.5-14B+1.5B_MaxPooled_y01.png)\(c\)Qwen2\.5\-14B & 1\.5B
![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/Observation3_Sensitivity/200/Observation3_LayerWise_Recovery_LongRange_Qwen2.5-32B+1.5B_MaxPooled_y01.png)\(d\)Qwen2\.5\-32B & 1\.5B

Figure 12:Layer\-wise comparison of target\-history, draft\-lookahead, and hybrid selection on high\-gap Qwen2\.5 target–draft pairs\. These results show that draft\-lookahead and target\-history provide complementary signals across layers, motivating the hybrid construction\.We next focus on the high\-gap Qwen2\.5 pairs identified in Fig\.[11](https://arxiv.org/html/2606.24957#A10.F11), namely Qwen2\.5\-14B and Qwen2\.5\-32B paired with Qwen2\.5\-0\.5B, 1\.5B drafts\. For these pairs, we apply the head and layer selection procedure described in Sec\.[4\.2](https://arxiv.org/html/2606.24957#S4.SS2), and compare target\-history, draft\-lookahead, and hybrid selection layer by layer\.

Fig\.[12](https://arxiv.org/html/2606.24957#A10.F12)shows that draft\-lookahead and target\-history provide complementary layer\-wise signals\. The hybrid selection therefore tracks the oracle more closely than either signal alone, supporting Dustin’s hybrid construction\.

## Appendix KAdditional historical and lookahead comparison policy

In this section, we present a fine\-grained analysis of the impact of our hybrid selection strategy compared to single\-source baselines across all 16 LongBench tasks\. We focus on the robustness of the selected budget configuration and the stability of performance across diverse datasets\. Furthermore, we investigate the theoretical upper bound of our framework by assuming an optimal dynamic selection of the mixing parameter\.

### K\.1Configuration of the Mixing Parametermm

As visualized in Fig\.[13](https://arxiv.org/html/2606.24957#A11.F13), the performance of the verification framework is sensitive to the budget allocation parametermm, which controls the trade\-off between Draft\-Lookahead and Target\-Historical signals\. Specifically, the blue lines represent theTarget\-Historical onlypolicy \(wherem=0m=0\), the green lines represent theDraft\-Lookahead onlypolicy \(wheremmis maximized\), and the orange/red curves illustrate theHybridperformance asmmvaries\.

![Refer to caption](https://arxiv.org/html/2606.24957v1/figure/APPENDIX/accuracy_trends_all_benchmarks_fixed_6pct_v2.png)Figure 13:Impact of the budget tuning parametermmon accuracy recovery across different benchmarks\. Blue indicates Target\-Historical only, green indicates Draft\-Lookahead only, and the orange/red curve represents the Hybrid approach\.It is important to note that the optimalmmused in our main experiments \(Dustin\-H\) was determined via Bayesian optimization using Optuna on theLongRewarddataset, acting as a proxy for general long\-context capability\. While this global parameter setting may not correspond to the global maximum for every individual benchmark in LongBench, the consistent performance observed highlights the generalization capability of Dustin, as it functions effectively without requiring dataset\-specific hyperparameter tuning for each downstream task\.

### K\.2Performance Stability and Robustness Analysis

Table[14](https://arxiv.org/html/2606.24957#A11.T14)details the accuracy gap \(Δ\\Delta, in %\) relative to the FullKV baseline\. We compare three variations along with an Oracle baseline:

- •Dustin\-T: Relies solely on Target\-Historical attention\.
- •Dustin\-D: Relies solely on Draft\-Lookahead attention\.
- •Dustin\-H: Our proposed Hybrid method using the fixed Optuna\-tunedmm\.
- •Oracle \(Best\): A theoretical upper bound that dynamically selects the best policy \(T, D, or H\) for each specific benchmark\.

##### Average Performance and Win Rate

Dustin\-H achieves the smallest average accuracy degradation among the fixed policies, with only a0\.59%0\.59\\%drop from the lossless baseline\. It outperforms Dustin\-T and Dustin\-D, which incur average accuracy drops of1\.91%1\.91\\%and0\.71%0\.71\\%, respectively\. In head\-to\-head comparisons across the 16 benchmarks, Dustin\-H further demonstrates superior versatility:

- •vs\. Dustin\-T:Dustin\-H wins in9tasks, loses in 4, and ties in 3\.
- •vs\. Dustin\-D:Dustin\-H wins in9tasks, loses in 6, and ties in 1\.

This indicates that combining signals allows the model to adapt to tasks where one signal type might be insufficient \(e\.g\., retrieval tasks favoring draft lookahead vs\. reasoning tasks favoring historical context\)\.

##### Robustness and Potential of Dynamic Allocation

A critical advantage of the Hybrid approach is its stability\. We calculated the standard deviation of accuracy penalties \(defined as the magnitude of negative drops\)\. The results highlight the robustness of our method compared to single\-source strategies:

- •Dustin\-T:σdrop=1\.86%\\sigma\_\{\\text\{drop\}\}=1\.86\\%
- •Dustin\-D:σdrop=1\.03%\\sigma\_\{\\text\{drop\}\}=1\.03\\%
- •Dustin\-H:σdrop=0\.83%\\sigma\_\{\\text\{drop\}\}=\\mathbf\{0\.83\\%\}
- •Oracle:σdrop=0\.52%\\sigma\_\{\\text\{drop\}\}=0\.52\\%

Comparing Dustin\-H to theOracle\(Dynamicmm\), which achieves an average drop of just−0\.27%\-0\.27\\%and a standard deviation of0\.52%0\.52\\%, we observe that our fixed\-parameter approach captures a significant portion of the recoverable accuracy\. The gap between Dustin\-H \(−0\.59%\-0\.59\\%\) and the Oracle \(−0\.27%\-0\.27\\%\) suggests that while Dustin\-H is highly effective and safe, future work exploring task\-aware dynamicmmselection could further close the gap to lossless verification\.

Table 14:Detailed accuracy comparison \(Δ\\Deltarelative to FullKV, %\)\. Negative values indicate accuracy loss\.Oraclerepresents the best result selected from T, D, and H for each task\.BenchmarkDustin\-TDustin\-DDustin\-HOracleNarrativeQA\-1\.84\-0\.25\-2\.17\-0\.25Qasper\-2\.97\-0\.35\-0\.68\-0\.35MultiFieldQA\-3\.93\-0\.25\-0\.08\-0\.08HotpotQA\-0\.63\+0\.05\+0\.03\+0\.052WikiMQA\-0\.91\-0\.74\-0\.54\-0\.54Musique\+0\.54\+0\.90\+0\.42\+0\.90GovReport\-1\.63\-1\.26\-1\.95\-1\.26QMSum\-4\.33\-1\.83\-0\.49\-0\.49MultiNews\-1\.12\-0\.23\+0\.12\+0\.12TREC\+0\.00\+0\.00\+0\.00\+0\.00TriviaQA\-6\.41\-3\.48\-0\.36\-0\.36SAMSum\-1\.88\-2\.63\-2\.08\-1\.88PassageCount\+0\.50\+0\.00\+0\.50\+0\.50PassageRet\+0\.00\-0\.50\+0\.00\+0\.00LCC\-3\.48\-0\.32\-1\.77\-0\.32RepoBench\-2\.55\-0\.51\-0\.33\-0\.33Average\-1\.91\-0\.71\-0\.59\-0\.27

## Appendix LLimitations

While Dustin effectively reduces verification latency and improves throughput in long\-context scenarios, we identify two primary limitations in the current framework that point towards future research directions\.

##### Static and Dynamic Budget Allocation

As analyzed in the ablation study \(Appendix[K](https://arxiv.org/html/2606.24957#A11)\), our current approach relies on a fixed budget allocation parametermm\(determining the ratio between draft\-lookahead and target\-historical\)\. Although optimizingmmvia offline profiling yields robust performance across averaged benchmarks, the comparison with the “Oracle” policy reveals that different tasks exhibit distinct preferences for validation signals\. The use of a static, globally optimized scalar inevitably results in a performance gap compared to an ideal dynamic strategy\. Future work could explore a lightweight, input\-aware mechanism to dynamically adjustmmon\-the\-fly, potentially closing the gap to near\-lossless verification\.

##### Computation and Memory Capacity

Dustin is primarily designed as a sparse verification framework to mitigate the computational overhead and memory bandwidth bottleneck during the speculative decoding verification phase\. While it significantly reduces the number of active tokens loaded during attention computation, it currently employs a global indexing mechanism that selects from the full history\. Unlike permanent token eviction methods which physically remove tokens to reduce the VRAM footprint, Dustin does not inherently expand the maximum supportable context length constrained by GPU memory capacity\.