TSS：目标端稀疏化在特定领域大型语言模型推测解码中的应用

# TSS: Target-Side Sparsification for Speculative Decoding in Domain-Specific Large Language Models
Source: [https://arxiv.org/html/2609.26100](https://arxiv.org/html/2609.26100)
Lianming HuangThanks:Corresponding author:lmhuang8\-c@my\.cityu\.edu\.hkAffiliation:City University of Hong KongQiao LiAffiliation:Mohamed bin Zayed University of Artificial IntelligenceNan GuanAffiliation:City University of Hong KongChun Jason XueAffiliation:Mohamed bin Zayed University of Artificial Intelligence

###### Abstract

Speculative decoding accelerates large language model inference through collaboration between a lightweight draft model and a target verifier\. Existing methods mainly improve the draft side, while the target model is typically kept dense and unchanged\. We show that, under domain\-specific inference, full\-depth target verification is not always the optimal choice\. Counter\-intuitively, skipping selected target layers can reduce verification cost while simultaneously increasing draft acceptance and preserving, or even improving, downstream task performance\. Based on this observation, we proposeTSS, a target\-side sparsification framework for speculative decoding\. TSS employs an acceptance\- and metric\-aware breadth search to explore multi\-layer skip configurations without imposing a fixed priority between the two objectives\. The selected configurations are stored in a domain\-to\-configuration mapping and applied by a lightweight skip controller, allowing one complete target model to support multiple sparse verification paths without retraining or permanent parameter pruning\. Experiments on Spec\-Bench across multiple domains, model scales, and speculative decoding methods show consistent improvements in draft acceptance and downstream task performance\. In Translation setting, TSS increases the average accept length from 2\.70 to 4\.53 \(\+67\.8%\+67\.8\\%\), improves BLEU from 0\.131 to 0\.237 \(\+80\.9%\+80\.9\\%\), and raises end\-to\-end throughput from 75\.6 to 127\.3 tokens/s, corresponding to a1\.68×1\.68\\timesspeedup\.

## Introduction

Large language models \(LLMs\) have become the foundation of modern AI applications, yet their deployment remains constrained by the cost of autoregressive decoding, where a Transformer decoder typically generates only one token per forward pass\([23](https://arxiv.org/html/2609.26100#bib.bib1);[16](https://arxiv.org/html/2609.26100#bib.bib3);[3](https://arxiv.org/html/2609.26100#bib.bib4)\)\. This sequential dependency has motivated a broad range of acceleration methods, including draft\-based speculative decoding\([16](https://arxiv.org/html/2609.26100#bib.bib3);[3](https://arxiv.org/html/2609.26100#bib.bib4)\), sequence\-to\-sequence speculation\([30](https://arxiv.org/html/2609.26100#bib.bib5)\), big\-little model collaboration\([14](https://arxiv.org/html/2609.26100#bib.bib6)\), tree\-based verification\([21](https://arxiv.org/html/2609.26100#bib.bib7);[28](https://arxiv.org/html/2609.26100#bib.bib8)\), multi\-head candidate generation\([2](https://arxiv.org/html/2609.26100#bib.bib9)\), feature\-level speculative sampling\([17](https://arxiv.org/html/2609.26100#bib.bib10)\), and draft\-free parallel decoding\([7](https://arxiv.org/html/2609.26100#bib.bib11);[33](https://arxiv.org/html/2609.26100#bib.bib12)\)\. As illustrated in Figure[1](https://arxiv.org/html/2609.26100#Sx1.F1), these methods largely follow a draft\-target pipeline in which candidate generation is optimized while the full target model remains densely activated during verification\. Our work starts from a different observation: in domain\-specific speculative decoding, full\-depth target verification may over\-process representations that are already sufficient for the downstream task\. Prior studies describe this behavior as*overthinking*, where correct intermediate predictions are later revised into incorrect outputs\([13](https://arxiv.org/html/2609.26100#bib.bib27);[8](https://arxiv.org/html/2609.26100#bib.bib28)\), while layer\-wise decoding studies show that intermediate and final Transformer layers expose different predictive information\([4](https://arxiv.org/html/2609.26100#bib.bib29)\)\. Motivated by this observation, we introduce a domain\-aware skip controller that selects a target\-layer skip policy without modifying the draft model or candidate proposal process\. During verification, the target follows a domain\-specific sparse path and bypasses layers likely to induce unnecessary refinement\. Empirically, this design not only reduces verification cost, but also increases draft acceptance while preserving or improving downstream task metrics\.

Figure 1:Comparison between conventional speculative decoding and TSS\.However, finding an effective skip configuration for speculative decoding is challenging\. First, conventional speculative decoding treats the dense target model as the reference distribution, and many compression or sparsification methods therefore rely on loss, perplexity, or distributional similarity to preserve the original model behavior\. Once the target verifier is sparsified, however, its token distribution inevitably changes\. For domain\-specific generation, preserving the dense target distribution is not necessarily equivalent to preserving the downstream task metric, making loss\-based layer selection a suboptimal objective\. Second, speculative decoding introduces an additional system\-level requirement: the sparse verifier must maintain or improve draft acceptance\. If the accept length becomes too low, speculative decoding degenerates into ordinary autoregressive decoding with extra draft overhead\. Therefore, target\-side sparsification requires a new skip\-layer search criterion that jointly considers downstream metrics and draft acceptance\.

Based on these observations and challenges, we proposeTSS, a target\-side block\-layer sparsification framework for speculative decoding in domain\-specific LLM inference\. TSS searches for sparse target configurations offline on a calibration set, with downstream task metrics as the primary quality constraint and draft acceptance as a key speculative decoding objective\. Since the skip pattern is selected offline, TSS can afford a broader search over multi\-layer skip combinations rather than relying only on local single\-layer scores\. The final sparse verifier is chosen to optimize a dual objective: preserving or improving domain\-level metrics while increasing accepted draft tokens and reducing target\-side computation\. Our contributions are summarized as follows\.

- •Target\-side sparsification for speculative decoding\.To our knowledge, TSS is the first framework that treats the deployed target verification path itself—rather than a layer\-skipped self\-draft—as the domain\-specific sparsification target\. TSS directly skips selected target layers to reduce verification cost while preserving, and sometimes improving, downstream task performance\.
- •Acceptance\- and metric\-aware skip selection\.We propose a skip\-layer search algorithm tailored to speculative decoding\. Instead of selecting layers according to language\-modeling loss, perplexity, or distributional similarity, TSS uses downstream task metrics and draft acceptance as the core objectives\.
- •Efficient domain\-aware skip control and broad validation\.TSS searches skip configurations offline and applies them through a lightweight domain\-aware controller\. We evaluate TSS on Spec\-Bench across multiple domains, model scales, and speculative decoding methods\. The results show substantial throughput gains, higher accept length, and stable or improved task metrics using directly reproducible skip configurations\.

## Related Work

### Speculative Decoding and Draft\-Side Optimization

Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose candidate tokens and a stronger target model to verify them in parallel\([16](https://arxiv.org/html/2609.26100#bib.bib3);[3](https://arxiv.org/html/2609.26100#bib.bib4)\)\. Most subsequent work focuses on improving the draft or proposal process\. SpecDec trains an independent drafter\([30](https://arxiv.org/html/2609.26100#bib.bib5)\), BiLD uses big\-little model collaboration\([14](https://arxiv.org/html/2609.26100#bib.bib6)\), and SpecInfer and staged speculative decoding organize candidates into trees or stages for parallel verification\([21](https://arxiv.org/html/2609.26100#bib.bib7);[28](https://arxiv.org/html/2609.26100#bib.bib8)\)\. Medusa and EAGLE improve proposals through multiple decoding heads or feature prediction\([2](https://arxiv.org/html/2609.26100#bib.bib9);[17](https://arxiv.org/html/2609.26100#bib.bib10)\), while Lookahead decoding explores draft\-free parallel generation\([7](https://arxiv.org/html/2609.26100#bib.bib11)\)\.KNN\-SSD further classifies incoming requests by domain and matches prompts to domain\-specific skip configurations, demonstrating the feasibility of prompt\-level domain identification\([27](https://arxiv.org/html/2609.26100#bib.bib32)\)\. Despite their differences, these methods generally keep the target model dense and fixed\. Instead, TSS treats the target verifier itself as an optimization target\.

### Layer Skipping and Sparse LLM Inference

Another line of work reduces Transformer computation through layer dropping, early exiting, or direct layer removal\. LayerDrop enables subnetworks of different depths\([6](https://arxiv.org/html/2609.26100#bib.bib14)\), while DeeBERT, FastBERT, and PABEE use intermediate confidence signals for early exit\([32](https://arxiv.org/html/2609.26100#bib.bib15);[19](https://arxiv.org/html/2609.26100#bib.bib16);[34](https://arxiv.org/html/2609.26100#bib.bib24)\)\. SkipBERT skips shallow layers using approximate representations\([29](https://arxiv.org/html/2609.26100#bib.bib25)\), CALM and RAEE dynamically adjusts the computation during generation\([25](https://arxiv.org/html/2609.26100#bib.bib17);[12](https://arxiv.org/html/2609.26100#bib.bib2)\)\. Recent LLM\-oriented methods, including LayerSkip, FREE, and ShortGPT, further suggest that full\-depth generation may involve unnecessary late\-layer computation and overthinking\([5](https://arxiv.org/html/2609.26100#bib.bib13);[1](https://arxiv.org/html/2609.26100#bib.bib26);[20](https://arxiv.org/html/2609.26100#bib.bib18)\)\. These methods mainly optimize perplexity, confidence, representation similarity, or standalone accuracy\. In contrast, TSS mitigates domain\-conditioned overthinking by selecting target layers according to both speculative acceptance and downstream task metrics\.

## Motivation

![Refer to caption](https://arxiv.org/html/2609.26100v1/domain_acc_new.png)Figure 2:Mean accepted tokens per verification step for EAGLE and SAMD on Vicuna\-7B and Llama\-2\-13B across four domains\.### Motivation I: Speculative Acceptance Is Domain\-Dependent

We first observe that speculative acceptance varies substantially across domains, model scales, and speculative decoding methods\. Figure[2](https://arxiv.org/html/2609.26100#Sx3.F2)reports the mean number of accepted tokens per verification step for EAGLE and SAMD on Vicuna\-7B and Llama\-2\-13B\. On Vicuna\-7B, SAMD consistently outperforms EAGLE, but its accept length still varies from 4\.30 on translation to 5\.85 on MMLU\. EAGLE ranges similarly from 2\.70 in translation to 3\.93 in summarization\. On Llama\-2\-13B, the relative ordering reverses: EAGLE achieves higher acceptance across all domains, reaching 5\.66 on MMLU but only 3\.46 on QA, while SAMD ranges from 2\.53 to 3\.96\. These results show that acceptance is determined not only by the speculative method, but also by the interaction among the draft model, target model, and domain\-specific generation task\.

This variation motivates domain\-specific optimization of target verification\. A fixed dense verifier applies the same full\-depth computation to all requests, even though different domains may require different levels of refinement\. In some domains, later target transformations may unnecessarily modify task\-sufficient predictions, reducing alignment with the draft model and causing earlier rejection\. We therefore ask whether selectively shortening the target verification path can mitigate such domain\-conditioned overthinking\. This leads to our next observation: skipping selected target layers can sometimes improve both speculative acceptance and downstream task performance\.

![Refer to caption](https://arxiv.org/html/2609.26100v1/Figures/skip_one_new.png)Figure 3:Single\-layer skip ablation on Llama\-2\-13B with SAMD across four domains\.![Refer to caption](https://arxiv.org/html/2609.26100v1/overview_new_new_new.png)Figure 4:Overview of TSS\. \(a\) Offline domain\-wise profiling searches target\-layer skip configurations using small calibration sets\. \(b\) At runtime, a domain\-aware controller retrieves the matched configuration and executes the corresponding sparse target\-verification path, while leaving the draft model unchanged\. \(c\) A representative case shows that TSS accepts more draft tokens and improves the final output over native dense verification\.
### Motivation II: Target Layers Exhibit Domain\-Specific Overthinking

To examine whether full\-depth target verification may unnecessarily revise task\-sufficient representations, we perform a single\-layer skip ablation on Llama\-2\-13B with SAMD\. For each target layerll, we skip only that layer during verification and report the change in speculative accept length and downstream task metric relative to the dense baseline\. As shown in Figure[3](https://arxiv.org/html/2609.26100#Sx3.F3), skipping many individual layers increases accept length, particularly for translation and QA\. However, the magnitude of this improvement varies substantially across layers and domains\. Some configurations provide little benefit, while skipping sensitive layers, such as the first target layer, can noticeably reduce acceptance in certain domains\.

The downstream metrics exhibit a different and more selective pattern\. Most single\-layer skips produce only small metric changes, whereas a few layers improve task performance and others cause clear degradation\. Moreover, the beneficial positions differ across translation, summarization, QA, and MMLU\. These results suggest that full\-depth verification can induce domain\-conditioned overthinking: additional transformations may reduce draft–target alignment or revise predictions that are already sufficient for a particular task\. At the same time, the contrasting acceptance and metric patterns show that a layer cannot be selected according to either objective alone\. Effective target sparsification must jointly consider speculative acceptance and downstream task quality under each domain\.

## Method

### Problem Formulation

We consider speculative decoding with a draft modelDDand a target modelTTserving requests from multiple domains\. Given an input prefix,DDproposesγ\\gammacandidate tokens, which are verified in parallel byTT\. Let the target model containLLTransformer blocks\. For domaindd, a target\-layer skip configuration is denoted by

Sd⊆\{1,…,L\},S\_\{d\}\\subseteq\\\{1,\\ldots,L\\\},where layers inSdS\_\{d\}are bypassed during verification, producing the sparse target verifierTSdT\_\{S\_\{d\}\}\. The dense verifier corresponds toSd=∅S\_\{d\}=\\emptyset\.

Using a small domain\-specific calibration set𝒞d\\mathcal\{C\}\_\{d\}, we evaluate each configuration according to three quantities:

Ad​\(Sd\)\\displaystyle A\_\{d\}\(S\_\{d\}\)=AvgAccept⁡\(D,TSd,𝒞d\),\\displaystyle=\\operatorname\{AvgAccept\}\(D,T\_\{S\_\{d\}\};\\mathcal\{C\}\_\{d\}\),Md​\(Sd\)\\displaystyle M\_\{d\}\(S\_\{d\}\)=Metric⁡\(D,TSd,𝒞d\),\\displaystyle=\\operatorname\{Metric\}\(D,T\_\{S\_\{d\}\};\\mathcal\{C\}\_\{d\}\),Cd​\(Sd\)\\displaystyle C\_\{d\}\(S\_\{d\}\)=Cost⁡\(TSd;𝒞d\),\\displaystyle=\\operatorname\{Cost\}\(T\_\{S\_\{d\}\};\\mathcal\{C\}\_\{d\}\),whereAdA\_\{d\}is the average number of accepted draft tokens per verification step,MdM\_\{d\}is the downstream task metric, andCdC\_\{d\}is the target verification cost\. Their dense\-baseline values are denoted byAd0=Ad​\(∅\)A\_\{d\}^\{0\}=A\_\{d\}\(\\emptyset\)andMd0=Md​\(∅\)M\_\{d\}^\{0\}=M\_\{d\}\(\\emptyset\)\.

TSS seeks the lowest\-cost domain\-specific verification path that preserves both speculative acceptance and task performance:

Sd⋆=arg​minSd⊆\{1,…,L\}\\displaystyle S\_\{d\}^\{\\star\}=\\operatorname\*\{arg\\,min\}\_\{S\_\{d\}\\subseteq\\\{1,\\ldots,L\\\}\}Cd​\(Sd\)\\displaystyle C\_\{d\}\(S\_\{d\}\)s\.t\.\\displaystyle\\text\{s\.t\.\}Ad​\(Sd\)≥Ad0−ϵA,\\displaystyle A\_\{d\}\(S\_\{d\}\)\\geq A\_\{d\}^\{0\}\-\\epsilon\_\{A\},Md​\(Sd\)≥Md0−ϵM,\\displaystyle M\_\{d\}\(S\_\{d\}\)\\geq M\_\{d\}^\{0\}\-\\epsilon\_\{M\},whereϵA\\epsilon\_\{A\}andϵM\\epsilon\_\{M\}tolerate small calibration fluctuations\. This formulation jointly constrains draft\-target alignment and domain\-level quality, preventing acceleration from being achieved at the expense of either objective\.

As illustrated in Figure[4](https://arxiv.org/html/2609.26100#Sx3.F4), TSS consists of an offline configuration stage and an online execution stage\. Offline, a skip\-layer profiler evaluates the target model on small domain\-specific calibration sets and stores the selected skip masks in a domain\-to\-configuration mapping\. Online, a lightweight controller routes each request to its matched configuration, while the draft model remains unchanged and the target verifier bypasses the selected layers\. The qualitative example further shows that the resulting sparse verifier can accept more draft tokens and produce a better task\-level output than native dense verification\.

Algorithm 1Acceptance\- and Metric\-Aware Breadth Search1:

D,T,𝒞d,L,KD,T,\\mathcal\{C\}\_\{d\},L,Kand thresholds

ϵA,ϵM,τA,τM\\epsilon\_\{A\},\\epsilon\_\{M\},\\tau\_\{A\},\\tau\_\{M\}
2:Best configuration

Sd⋆S\_\{d\}^\{\\star\}and Pareto set

𝒫d\\mathcal\{P\}\_\{d\}
3:Evaluate dense baseline

Ad0,Md0,Cd0A\_\{d\}^\{0\},M\_\{d\}^\{0\},C\_\{d\}^\{0\}
4:

ℒd←∅\\mathcal\{L\}\_\{d\}\\leftarrow\\emptyset
5:for

l=1l=1to

LLdo

6:Evaluate

S=\{l\}S=\\\{l\\\}
7:if

Ad​\(S\)≥Ad0−τA∨Md​\(S\)≥Md0−τMA\_\{d\}\(S\)\\geq A\_\{d\}^\{0\}\-\\tau\_\{A\}\\lor M\_\{d\}\(S\)\\geq M\_\{d\}^\{0\}\-\\tau\_\{M\}then

8:

ℒd←ℒd∪\{l\}\\mathcal\{L\}\_\{d\}\\leftarrow\\mathcal\{L\}\_\{d\}\\cup\\\{l\\\}
9:endif

10:endfor

11:

ℱd←\{∅\}\\mathcal\{F\}\_\{d\}\\leftarrow\\\{\\emptyset\\\},

𝒫d←\{\(∅,Ad0,Md0,Cd0\)\}\\mathcal\{P\}\_\{d\}\\leftarrow\\\{\(\\emptyset,A\_\{d\}^\{0\},M\_\{d\}^\{0\},C\_\{d\}^\{0\}\)\\\}
12:for

k=1k=1to

KKdo

13:

𝒮k←\{S⊆ℒd:\|S\|=k\}\\mathcal\{S\}\_\{k\}\\leftarrow\\\{S\\subseteq\\mathcal\{L\}\_\{d\}:\|S\|=k\\\}
14:

ℱd\(k\)←∅\\mathcal\{F\}\_\{d\}^\{\(k\)\}\\leftarrow\\emptyset
15:for all

S∈𝒮kS\\in\\mathcal\{S\}\_\{k\}do

16:Evaluate

Ad​\(S\),Md​\(S\),Cd​\(S\)A\_\{d\}\(S\),M\_\{d\}\(S\),C\_\{d\}\(S\)
17:if

Ad​\(S\)≥Ad0−ϵA∧Md​\(S\)≥Md0−ϵMA\_\{d\}\(S\)\\geq A\_\{d\}^\{0\}\-\\epsilon\_\{A\}\\land M\_\{d\}\(S\)\\geq M\_\{d\}^\{0\}\-\\epsilon\_\{M\}then

18:

ℱd\(k\)←ℱd\(k\)∪\{S\}\\mathcal\{F\}\_\{d\}^\{\(k\)\}\\leftarrow\\mathcal\{F\}\_\{d\}^\{\(k\)\}\\cup\\\{S\\\}
19:Add

\(S,Ad​\(S\),Md​\(S\),Cd​\(S\)\)\(S,A\_\{d\}\(S\),M\_\{d\}\(S\),C\_\{d\}\(S\)\)to

𝒫d\\mathcal\{P\}\_\{d\}
20:endif

21:endfor

22:if

ℱd\(k\)=∅\\mathcal\{F\}\_\{d\}^\{\(k\)\}=\\emptysetthen

23:break

24:endif

25:

ℱd←ℱd∪ℱd\(k\)\\mathcal\{F\}\_\{d\}\\leftarrow\\mathcal\{F\}\_\{d\}\\cup\\mathcal\{F\}\_\{d\}^\{\(k\)\}
26:endfor

27:Remove dominated configurations from

𝒫d\\mathcal\{P\}\_\{d\}
28:

Sd⋆←arg⁡minS∈ℱd​Cd​\(S\)S\_\{d\}^\{\\star\}\\leftarrow\\arg\\min\_\{S\\in\\mathcal\{F\}\_\{d\}\}C\_\{d\}\(S\)
29:return

Sd⋆,𝒫dS\_\{d\}^\{\\star\},\\mathcal\{P\}\_\{d\}

### Acceptance\- and Metric\-Aware Breadth\-First Search

Selecting multiple target layers is challenging because speculative acceptance and downstream task performance do not always change in the same direction\. A layer that improves the task metric may slightly reduce acceptance, while another layer may improve acceptance but provide little task\-level benefit\. Consequently, assigning a fixed priority or combining the two objectives into a weighted score can prematurely bias the search toward one signal\. Conventional greedy search is particularly vulnerable to this issue: once it commits to a locally favorable layer, all subsequent configurations are restricted to descendants of that choice, potentially missing combinations formed by individually moderate but jointly beneficial layers\. Moreover, the effect of skipping multiple Transformer blocks is not necessarily additive, since removing one block can alter the representations received by all subsequent blocks\.

Algorithm[1](https://arxiv.org/html/2609.26100#alg1)presents our acceptance\- and metric\-aware breadth\-first search\. The algorithm first evaluates the native dense verifier to establish the baseline acceptance, downstream metric, and verification cost\. It then independently evaluates every single\-layer skip and constructs a broad candidate pool:

ℒd=\{l\|Ad​\(\{l\}\)≥Ad0−τA∨Md​\(\{l\}\)≥Md0−τM\}\.\\mathcal\{L\}\_\{d\}=\\left\\\{l\\;\\middle\|\\;A\_\{d\}\(\\\{l\\\}\)\\geq A\_\{d\}^\{0\}\-\\tau\_\{A\}\\;\\lor\\;M\_\{d\}\(\\\{l\\\}\)\\geq M\_\{d\}^\{0\}\-\\tau\_\{M\}\\right\\\}\.The logical*or*retains a layer when it is promising under either acceptance or the downstream metric\. This permissive initial filtering is important because a layer that appears unfavorable under one objective in isolation may still become useful when combined with other layers\. Unlike greedy ranking, this stage does not collapse acceptance and task performance into a single scalar score or discard candidates according to a fixed objective priority\.

After constructingℒd\\mathcal\{L\}\_\{d\}, TSS explores multi\-layer configurations level by level\. At search depthkk, the algorithm enumerates all unique configurations that skip exactlykkcandidate layers:

𝒮k=\{S⊆ℒd\|\|S\|=k\}\.\\mathcal\{S\}\_\{k\}=\\left\\\{S\\subseteq\\mathcal\{L\}\_\{d\}\\;\\middle\|\\;\|S\|=k\\right\\\}\.Thus, all configurations with the same sparsity level are evaluated before proceeding to configurations with more skipped layers\. This breadth\-first organization allows combinations originating from different single\-layer choices to compete at the same search depth, avoiding the early commitment of greedy search\. For every configuration, TSS measures acceptance, downstream performance, and target verification cost on the domain\-specific calibration set\.

A configuration is retained only when it jointly satisfies both feasibility requirements:

ℱd\(k\)=\{S∈𝒮k\|Ad​\(S\)≥Ad0−ϵA,Md​\(S\)≥Md0−ϵM\}\.\\mathcal\{F\}\_\{d\}^\{\(k\)\}=\\left\\\{S\\in\\mathcal\{S\}\_\{k\}\\;\\middle\|\\;\\begin\{aligned\} A\_\{d\}\(S\)&\\geq A\_\{d\}^\{0\}\-\\epsilon\_\{A\},\\\\ M\_\{d\}\(S\)&\\geq M\_\{d\}^\{0\}\-\\epsilon\_\{M\}\\end\{aligned\}\\right\\\}\.The initial candidate construction therefore uses a permissive*or*condition to preserve search diversity, whereas the multi\-layer evaluation uses a strict*and*condition to prevent acceleration from sacrificing either speculative acceptance or task quality\. The search continues until the maximum depthKKis reached or the current depth contains no feasible configuration\.

Finally, TSS removes configurations dominated in both acceptance and downstream metric and selects the feasible configuration with the lowest measured verification cost:

Sd⋆=arg⁡minS∈⋃k=0Kℱd\(k\)​Cd​\(S\)\.S\_\{d\}^\{\\star\}=\\arg\\min\_\{S\\in\\bigcup\_\{k=0\}^\{K\}\\mathcal\{F\}\_\{d\}^\{\(k\)\}\}C\_\{d\}\(S\)\.The selected configuration becomes the default skip policy for domaindd\. Other Pareto\-optimal configurations are also retained, allowing the domain\-to\-configuration mapping to support alternative operating points when different efficiency or quality requirements are preferred\. Since the search is performed offline, its additional evaluation cost does not affect the runtime speculative decoding process\.

### Domain\-to\-Configuration Mapping

The optimal target\-layer skip pattern varies across domains, because different tasks exhibit different acceptance behavior and layer redundancy\. Therefore, TSS does not use a single global skip mask\. Instead, after offline breadth search, it constructs a domain\-specific configuration map:

Φ:𝒟→2\{1,…,L\},Φ⁡\(d\)=Sd⋆,\\Phi:\\mathcal\{D\}\\rightarrow 2^\{\\\{1,\\ldots,L\\\}\},\\qquad\\Phi\(d\)=S\_\{d\}^\{\\star\},where𝒟\\mathcal\{D\}is the set of supported domains andSd⋆S\_\{d\}^\{\\star\}is the selected skip\-layer set for domaindd\.

Each configuration stores the information required for runtime execution:

𝒢d=\(Sd⋆,Ad​\(Sd⋆\),Md​\(Sd⋆\),Rd\),Rd=\|Sd⋆\|L,\\mathcal\{G\}\_\{d\}=\\left\(S\_\{d\}^\{\\star\},\\,A\_\{d\}\(S\_\{d\}^\{\\star\}\),\\,M\_\{d\}\(S\_\{d\}^\{\\star\}\),\\,R\_\{d\}\\right\),\\qquad R\_\{d\}=\\frac\{\|S\_\{d\}^\{\\star\}\|\}\{L\},whereAd​\(Sd⋆\)A\_\{d\}\(S\_\{d\}^\{\\star\}\)andMd​\(Sd⋆\)M\_\{d\}\(S\_\{d\}^\{\\star\}\)are the calibrated accept length and task metric, andRdR\_\{d\}is the target\-layer sparsity ratio\. The resulting mapping is saved as a lightweight local configuration file\. At runtime, the controller retrieves𝒢d\\mathcal\{G\}\_\{d\}according to the request domain and applies the corresponding skip set to the target verifier\. This allows one complete target model to support multiple domain\-specific sparse paths without storing separate pruned model copies\.

Table 1:The results on Spec\-Bench\.
### Domain\-Aware Skip Controller

At runtime, TSS retains the complete target model and applies domain\-specific sparsity through a lightweight skip controller rather than permanent parameter pruning\. Given an input requestxx, the controller first identifies its domaind⁡\(x\)d\(x\)and queries the offline domain\-to\-configuration mapping:

S⁡\(x\)=\{Φ⁡\(d⁡\(x\)\),d⁡\(x\)∈𝒟,∅,d⁡\(x\)∉𝒟,S\(x\)=\\begin\{cases\}\\Phi\(d\(x\)\),&d\(x\)\\in\\mathcal\{D\},\\\\ \\emptyset,&d\(x\)\\notin\\mathcal\{D\},\\end\{cases\}where𝒟\\mathcal\{D\}denotes the set of supported domains\. For a supported domain, the retrieved skip configuration remains fixed throughout the request\. For an unseen or unsupported domain, the controller falls back toS⁡\(x\)=∅S\(x\)=\\emptyset, and the target model executes its original full\-depth verification path\. The draft model and candidate proposal process remain unchanged in both cases\.

Before executing each Transformer block, the target forward function checks whether its layer index belongs to the selected skip set:

hl\+1=\{fl​\(hl\),l∉S⁡\(x\),hl,l∈S⁡\(x\)\.h\_\{l\+1\}=\\begin\{cases\}f\_\{l\}\(h\_\{l\}\),&l\\notin S\(x\),\\\\ h\_\{l\},&l\\in S\(x\)\.\\end\{cases\}A skipped block therefore forwards its input hidden state directly to the next active block, avoiding its attention and feed\-forward computation\. Since all target parameters remain loaded, requests can switch among domain\-specific sparse paths or the dense fallback path without loading separate pruned models\. The controller only performs a simple domain lookup followed by a match check on each layer index, making its runtime overhead practically negligible compared with the computation saved by bypassing Transformer blocks\. Thus, TSS provides efficient domain\-aware execution while preserving the original dense target behavior for requests outside the configured domain set\.

## Experiments

### Experimental Setup

We evaluate TSS on the domain\-specific tasks provided by Spec\-Bench\. For each domain, we split the corresponding data into a calibration set and a test set, using 20% of the prompts for target\-layer skip selection and the remaining 80% for held\-out evaluation\. The calibration set is used only to determine the domain\-specific sparse target configuration, and no model weights are updated\. In all experiments, we set bothϵ\\epsilonandτ\\tauto5%5\\%\. Experimental costs and hardware configurations are detailed in technical supplement\.

#### Models and speculative decoding methods\.

We evaluate TSS on two representative speculative decoding settings: Vicuna\-7B with EAGLE\([17](https://arxiv.org/html/2609.26100#bib.bib10)\)and Llama\-2\-13B with SAMD Token\-Recycle\([10](https://arxiv.org/html/2609.26100#bib.bib34)\)\. Vicuna\-7B containsL=32L=32target layers, while Llama\-2\-13B containsL=40L=40target layers\. For each setting,Nativedenotes the original speculative decoding method with the dense target verifier, and\+TSSdenotes the same method equipped with target\-layer skipping\. TSS keeps the draft model, candidate generation procedure, and accept/reject logic unchanged; only the target verifier is replaced by a domain\-calibrated sparse verifier\.

#### Domains and metrics\.

We evaluate on four tasks from Spec\-Bench\([31](https://arxiv.org/html/2609.26100#bib.bib33)\)—translation, summarization, QA, and RAG—and additionally include MMLU as a multiple\-choice knowledge benchmark\. For translation, we report BLEU, which measuresnn\-gram precision between the generated translation and reference translations with a brevity penalty\([22](https://arxiv.org/html/2609.26100#bib.bib19)\)\. For summarization, we report ROUGE\-L F1, which evaluates the longest common subsequence overlap between the generated summary and the reference summary\([18](https://arxiv.org/html/2609.26100#bib.bib20)\)\. For RAG and QA, we follow the standard open\-domain question answering evaluation and report token\-level F1 after answer normalization, measuring the overlap between the predicted answer and the reference answer\([15](https://arxiv.org/html/2609.26100#bib.bib21);[24](https://arxiv.org/html/2609.26100#bib.bib22)\)\. For MMLU, we report multiple\-choice accuracy over the predicted answer option, following the original MMLU evaluation protocol\([9](https://arxiv.org/html/2609.26100#bib.bib23)\)\.

### Main Results

Table[1](https://arxiv.org/html/2609.26100#Sx4.T1)reports the main results on Spec\-Bench\. Across all evaluated domains and model settings, TSS consistently improves throughput over the native speculative decoding baseline while maintaining or improving the downstream task metric\. On Vicuna\-7B with EAGLE, TSS achieves speedups from 1\.07×\\timesto 1\.68×\\timesacross five domains\. The largest gain appears on translation, where TSS increases the accept length from 2\.70 to 4\.53 and improves throughput from 75\.6 to 127\.3 tok/s, corresponding to a 1\.68×\\timesspeedup\. Similar improvements are observed on RAG, QA, and MMLU, where TSS improves both accept length and task metric while skipping 15\.6% of target layers\.

TSS also generalizes to larger target models and different speculative decoding methods\. On Llama\-2\-13B with SAMD Token\-Recycle, TSS improves throughput by 1\.16×\\times–1\.30×\\timesacross translation, summarization, QA, and MMLU\. Importantly, these speedups do not come at the cost of task performance\. For example, on MMLU, TSS improves the task metric from 0\.359 to 0\.422 while increasing throughput from 58\.6 to 70\.8 tok/s\. On QA, TSS improves accept length from 2\.53 to 2\.74 and task metric from 0\.120 to 0\.129, with a 1\.30×\\timesthroughput gain\. These results show that target\-side sparsification is not limited to a specific draft mechanism or model size\.

The magnitude of TSS gains varies across speculative decoding architectures\. Vicuna\-7B with EAGLE exhibits larger improvements, while Llama\-2\-13B with SAMD Token\-Recycle shows more moderate but consistent gains\. This difference suggests that the benefit of target\-side skipping depends on the interaction among the draft mechanism, target model, and domain, rather than model scale alone\. Nevertheless, TSS improves throughput across both architectures while preserving or improving downstream metrics, demonstrating that target\-side optimization is broadly applicable to heterogeneous speculative decoding pipelines\.

Table 2:Comparison of target\-layer search strategies on MMLU and Translation, SLEB\([26](https://arxiv.org/html/2609.26100#bib.bib30)\)and GM\-Skip\([11](https://arxiv.org/html/2609.26100#bib.bib31)\)\.MethodAccept Len\.MetricSkip LayersThroughputTranslationSLEB3\.4770\.129\{2,3,4,9\}\\\{2,3,4,9\\\}101\.5Random3\.3820\.140\{2,3,4,5\}\\\{2,3,4,5\\\}100\.3GM\-Skip2\.9950\.109\{12,14,22,23\}\\\{12,14,22,23\\\}91\.1Accept\-only3\.7690\.126\{3,15\}\\\{3,15\\\}107\.9Metric\-only3\.0290\.139\{12,18,28\}\\\{12,18,28\\\}91\.1TSS4\.5280\.237\{3,25,30\}\\\{3,25,30\\\}127\.3MMLUSLEB3\.4880\.296\{2,3,4,6\}\\\{2,3,4,6\\\}84\.0Random3\.5390\.132\{2,11,17\}\\\{2,11,17\\\}84\.9GM\-Skip2\.8890\.311\{3,8,19\}\\\{3,8,19\\\}70\.4Accept\-only4\.0230\.219\{4,6,16\}\\\{4,6,16\\\}88\.0Metric\-only3\.0010\.265\{3\}\\\{3\\\}74\.1TSS3\.9550\.328\{3,7,9,14,20\}\\\{3,7,9,14,20\\\}90\.8Overall, TSS achieves simultaneous gains in speculative acceptance, task quality, and end\-to\-end generation speed\. Averaged over all settings in Table[1](https://arxiv.org/html/2609.26100#Sx4.T1), TSS improves throughput by approximately1\.31×1\.31\\times, and increases accept length by about 20% over the native baseline\. These results support our central claim that full\-depth verification can induce domain\-conditioned overthinking, and that selectively bypassing unnecessary target transformations can improve both draft–target alignment and inference efficiency\. All results can be reproduced on Spec\-Bench by directly applying the reported skip\-layer configurations without additional training or model modification\.

Figure 5:Representative translation and MMLU cases\. TSS produces an output closer to the translation reference and corrects an incorrect MMLU answer, while simultaneously improving accept length and throughput\.
### Search Algorithm Ablation

Table[2](https://arxiv.org/html/2609.26100#Sx5.T2)compares TSS with several alternative target\-layer selection strategies\. On Translation, TSS Breadth Search achieves the best results across all three objectives, increasing the accept length to 4\.528, the task metric to 0\.237, and throughput to 127\.3 tokens/s\. In contrast, GM\-Skip\(Greedy search\) and random search select configurations that degrade both acceptance and task quality, while the single\-objective variants improve only one aspect\.

On MMLU, accept\-only search obtains the highest accept length, but substantially reduces accuracy from 0\.328 to 0\.219\. Accept\-only search matches the best task accept, yet produces much lower metrics and throughput\. TSS reaches the same best accuracy while maintaining an accept length of 3\.955 and achieving the highest throughput of 90\.8 tokens/s\. These results show that optimizing acceptance or task quality independently is insufficient, and that breadth search provides a better balance among acceptance, downstream performance, and inference efficiency\.

### Case Study

Figure[5](https://arxiv.org/html/2609.26100#Sx5.F5)presents two representative examples where TSS improves both generation quality and speculative decoding efficiency\. In the translation example, the native model produces a verbose sentence with several expressions that deviate from the reference\. After applying TSS, the output becomes more concise and semantically closer to the reference translation\. Consequently, sentence\-level BLEU increases from 0\.100 to 0\.328, while the average accept length improves from 2\.69 to 2\.89 and throughput increases from 39\.4 to 49\.9 tokens/s\.

The MMLU example shows a stronger effect on task correctness\. The native speculative model selects option C, whereas TSS produces option B, which matches the reference answer\. This changes the sample\-level accuracy from 0 to 1 and is consistent with the overall MMLU score improvement from 0\.359 to 0\.422\. At the same time, TSS increases the accept length from 3\.96 to 4\.08 and throughput from 60\.2 to 72\.3 tokens/s\. These examples suggest that target\-layer skipping does not simply weaken verification\. By removing domain\-redundant computation, TSS can avoid unnecessary output refinement, improve draft\-target alignment, and produce outputs that better match downstream task objectives\.

## Conclusion

We presentedTSS, a target\-side sparsification framework for domain\-specific speculative decoding\. TSS uses acceptance\- and metric\-aware search to identify domain\-specific skip configurations and applies them through a lightweight controller without retraining or separate pruned models\. Experiments on Spec\-Bench show consistent gains in accept length and throughput while preserving or improving task metrics across multiple domains and speculative decoding methods\. These results demonstrate that full\-depth target verification is not always optimal and establish target\-side optimization as a complementary direction for speculative decoding\. Additional ablations and qualitative analyses for all five domains are provided in the technical supplement\.

## References

- Baeet al\.\(2023\)S\. Bae, J\. Ko, H\. Song, and S\. YunFast and robust early\-exiting framework for autoregressive language models with synchronized parallel decoding\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 5910–5924\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.362)Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Caiet al\.\(2024\)T\. Cai, Y\. Li, Z\. Geng, H\. Peng, J\. D\. Lee, D\. Chen, and T\. DaoMedusa: simple llm inference acceleration framework with multiple decoding heads\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Chenet al\.\(2023\)C\. Chen, S\. Borgeaud, G\. Irving, J\. Lespiau, L\. Sifre, and J\. JumperAccelerating large language model decoding with speculative sampling\.arXiv preprint arXiv:2302\.01318\.Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Chuanget al\.\(2024\)Y\. Chuang, Y\. Xie, H\. Luo, Y\. Kim, J\. Glass, and P\. HeDoLa: decoding by contrasting layers improves factuality in large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1)\.
- Elhoushiet al\.\(2024\)M\. Elhoushi, A\. Shrivastava, D\. Liskovich, B\. Hosmer, B\. Wasti, L\. Lai, A\. Mahmoud, B\. Acun, S\. Agarwal, A\. Roman, A\. A\. Aly, B\. Chen, and C\. WuLayerSkip: enabling early exit inference and self\-speculative decoding\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Fanet al\.\(2020\)A\. Fan, E\. Grave, and A\. JoulinReducing transformer depth on demand with structured dropout\.InProceedings of the 8th International Conference on Learning Representations,Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Fuet al\.\(2024\)Y\. Fu, P\. Bailis, I\. Stoica, and H\. ZhangBreak the sequential dependency of llm inference using lookahead decoding\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Halawiet al\.\(2024\)D\. Halawi, J\. Denain, and J\. SteinhardtOverthinking the truth: understanding how language models process false demonstrations\.InThe Twelfth International Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.InProceedings of the 9th International Conference on Learning Representations,Cited by:[Domains and metrics\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Huet al\.\(2025\)Y\. Hu, K\. Wang, X\. Zhang, F\. Zhang, C\. Li, H\. Chen, and J\. ZhangSAM decoding: speculative decoding via suffix automaton\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12187–12204\.External Links:[Link](https://aclanthology.org/2025.acl-long.595/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.595),ISBN 979\-8\-89176\-251\-0Cited by:[Models and speculative decoding methods\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px1.p1.1)\.
- Huanget al\.\(2026\)L\. Huang, H\. Hu, Q\. Li, X\. He, N\. Guan, and C\. J\. XueGM\-Skip: metric\-guided transformer block skipping for efficient vision\-language models\.InFindings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[Table 2](https://arxiv.org/html/2609.26100#Sx5.T2)\.
- Huanget al\.\(2024\)L\. Huang, S\. Wu, Y\. Cui, Y\. Xiong, X\. Liu, T\. Kuo, N\. Guan, and C\. J\. XueRAEE: a robust retrieval\-augmented early exiting framework for efficient inference\.arXiv preprint arXiv:2405\.15198\.Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Kayaet al\.\(2019\)Y\. Kaya, S\. Hong, and T\. DumitrasShallow\-deep networks: understanding and mitigating network overthinking\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 3301–3310\.Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1)\.
- Kimet al\.\(2023\)S\. Kim, K\. Mangalam, S\. Moon, J\. Malik, M\. W\. Mahoney, A\. Gholami, and K\. KeutzerSpeculative decoding with big little decoder\.InAdvances in Neural Information Processing Systems,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Kwiatkowskiet al\.\(2019\)T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. PetrovNatural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[Domains and metrics\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Leviathanet al\.\(2023\)Y\. Leviathan, M\. Kalman, and Y\. MatiasFast inference from transformers via speculative decoding\.InProceedings of the 40th International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Liet al\.\(2024\)Y\. Li, F\. Wei, C\. Zhang, and H\. ZhangEAGLE: speculative sampling requires rethinking feature uncertainty\.InProceedings of the 41st International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1),[Models and speculative decoding methods\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px1.p1.1)\.
- Lin \(2004\)C\. LinROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,pp\. 74–81\.Cited by:[Domains and metrics\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Liuet al\.\(2020\)W\. Liu, P\. Zhou, Z\. Zhao, Z\. Wang, Q\. Deng, Q\. Ju, and H\. WangFastBERT: a self\-distilling bert with adaptive inference time\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Menet al\.\(2025\)X\. Men, M\. Xu, Q\. Zhang, B\. Wang, H\. Lin, Y\. Lu, X\. Han, and W\. ChenShortGPT: layers in large language models are more redundant than you expect\.InFindings of the Association for Computational Linguistics: ACL 2025,Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Miaoet al\.\(2024\)X\. Miao, G\. Oliaro, Z\. Zhang, X\. Cheng, Z\. Wang, Z\. Zhang, R\. Y\. Y\. Wong, A\. Zhu, L\. Yang, X\. Shi, C\. Shi, Z\. Chen, D\. Arfeen, R\. Abhyankar, and Z\. JiaSpecInfer: accelerating generative large language model serving with tree\-based speculative inference and verification\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Papineniet al\.\(2002\)K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBleu: a method for automatic evaluation of machine translation\.InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics,pp\. 311–318\.External Links:[Document](https://dx.doi.org/10.3115/1073083.1073135)Cited by:[Domains and metrics\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Popeet al\.\(2023\)R\. Pope, S\. Douglas, A\. Chowdhery, J\. Devlin, J\. Bradbury, J\. Heek, K\. Xiao, S\. Agrawal, and J\. DeanEfficiently scaling transformer inference\.InProceedings of Machine Learning and Systems,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1)\.
- Rajpurkaret al\.\(2016\)P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. LiangSQuAD: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 2383–2392\.External Links:[Document](https://dx.doi.org/10.18653/v1/D16-1264)Cited by:[Domains and metrics\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Schusteret al\.\(2022\)T\. Schuster, A\. Fisch, J\. Gupta, M\. Dehghani, D\. Bahri, V\. Q\. Tran, Y\. Tay, and D\. MetzlerConfident adaptive language modeling\.InAdvances in Neural Information Processing Systems,Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Songet al\.\(2024\)J\. Song, K\. Oh, T\. Kim, H\. Kim, Y\. Kim, and J\. KimSLEB: streamlining LLMs through redundancy verification and elimination of transformer blocks\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 46136–46155\.Cited by:[Table 2](https://arxiv.org/html/2609.26100#Sx5.T2)\.
- Songet al\.\(2026\)M\. Song, H\. Xia, J\. Zhang, C\. T\. Leong, Q\. Xu, W\. Li, and S\. LiKNN\-SSD: enabling dynamic self\-speculative decoding via nearest neighbor layer set optimization\.InFindings of the Association for Computational Linguistics: EACL 2026,Rabat, Morocco,pp\. 641–655\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-eacl.31)Cited by:[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Spector and Re \(2023\)B\. Spector and C\. ReAccelerating llm inference with staged speculative decoding\.arXiv preprint arXiv:2308\.04623\.Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Wanget al\.\(2022\)J\. Wang, K\. Chen, G\. Chen, L\. Shou, and J\. McAuleySkipBERT: efficient inference with shallow layer skipping\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7287–7301\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.503)Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Xiaet al\.\(2023\)H\. Xia, T\. Ge, P\. Wang, S\. Chen, F\. Wei, and Z\. SuiSpeculative decoding: exploiting speculative execution for accelerating seq2seq generation\.InFindings of the Association for Computational Linguistics: EMNLP 2023,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1),[Speculative Decoding and Draft\-Side Optimization](https://arxiv.org/html/2609.26100#Sx2.SSx1.p1.1)\.
- Xiaet al\.\(2024\)H\. Xia, Z\. Yang, Q\. Dong, P\. Wang, Y\. Li, T\. Ge, T\. Liu, W\. Li, and Z\. SuiUnlocking efficiency in large language model inference: a comprehensive survey of speculative decoding\.InFindings of the Association for Computational Linguistics ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand and virtual meeting,pp\. 7655–7671\.External Links:[Link](https://aclanthology.org/2024.findings-acl.456),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.456)Cited by:[Domains and metrics\.](https://arxiv.org/html/2609.26100#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Xinet al\.\(2020\)J\. X\. Xin, R\. Tang, J\. Lee, Y\. Yu, and J\. LinDeeBERT: dynamic early exiting for accelerating bert inference\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.
- Zhaoet al\.\(2024\)Y\. Zhao, Z\. Xie, C\. Liang, C\. Zhuang, and J\. GuLookahead: an inference acceleration framework for large language model with lossless generation accuracy\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining,Cited by:[Introduction](https://arxiv.org/html/2609.26100#Sx1.p1.1)\.
- Zhouet al\.\(2020\)W\. Zhou, C\. Xu, T\. Ge, J\. McAuley, K\. Xu, and F\. WeiBERT loses patience: fast and robust inference with early exit\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[Layer Skipping and Sparse LLM Inference](https://arxiv.org/html/2609.26100#Sx2.SSx2.p1.1)\.

## Appendix AExperimental Setup

### Models

We evaluate target\-layer skipping \(TSS\) on two speculative\-decoding stacks that cover both mid\-size and larger chat models\.

#### Vicuna\-7B \+ EAGLE\.

The target model islmsys/vicuna\-7b\-v1\.3, a LLaMA\-based chat model withL=32L\{=\}32Transformer layers, hidden size40964096,3232attention heads, intermediate size1100811008, vocabulary size32,00032\{,\}000, and context length20482048\. Weights are loaded infloat16\. The draft model is the official EAGLE checkpointEAGLE\-Vicuna\-7B\-v1\.3\(a one\-layer autoregressive draft head aligned to the Vicuna\-7B representation\), used in the original EAGLE \(non\-EAGLE\-3\) configuration\. Prompts follow the Vicuna conversation template\.

#### Llama\-2\-13B \+ SAMD\.

The target model ismeta\-llama/Llama\-2\-13b\-chat\-hfwithL=40L\{=\}40layers, hidden size51205120,4040attention heads, intermediate size1382413824, vocabulary size32,00032\{,\}000, and context length40964096\(float16\)\. Speculative decoding uses SAMD with*Token Recycle*drafting and tree configurationtoken\_recycle\_4\_15\(Spec\-Bench setting\)\. Prompts use the Llama\-2 chat template \(\[INST\]…\[/INST\]\)\.

### Hardware and Software

All measurements are obtained on 8\*NVIDIA GeForce RTX 4090 GPU with2424GB GDDR6X memory \(driver595\.71595\.71, compute capability8\.98\.9\)\. The host has dual\-socket Intel Xeon Platinum 8470Q CPUs \(2×522\{\\times\}52cores /208208threads\) and754754GB system RAM, running Ubuntu22\.0422\.04LTS\. We use PyTorch2\.112\.11with CUDA12\.812\.8/13\.013\.0and Hugging Face Transformers \(4\.464\.46for the EAGLE stack;4\.574\.57for the SAMD stack\)\. models are loaded infloat16on device\.

### Evaluation Protocol

- •Datasets\.Spec\-Bench\-style domains: translation \(BLEU\), summarization \(ROUGE\-L\), open\-domain QA \(token F1\), RAG \(token F1\), and MMLU \(accuracy\)\. \(*test*\)\. Exception: Llama\-2\-13B QA selection uses3232train prompts\.
- •Generation\.Maximum new tokens per requestT=96T\{=\}96; decoding temperatureτ=0\\tau\{=\}0\(greedy\); EAGLE draft budgettotal\_token=60\{=\}60\.
- •Metrics\.Mean accepted length per verify step, task metric, end\-to\-end throughput \(tok/s\), skip set𝒮\\mathcal\{S\}and sparsity\|𝒮\|/L\|\\mathcal\{S\}\|/L\.
- •Native vs\. TSS\.*Native*:𝒮=∅\\mathcal\{S\}\{=\}\\emptyset\.*\+TSS*: selected target\-layer skip set during verification\.

Table 3:Default hyperparameters for TSS selection and decoding\.CategorySettingValueDecodingPrecisionfloat16Temperatureτ\\tau00Max new tokensTT9696EAGLEtotal\_token6060DataRandom seed4242Selection train size1616\(QA\-13B:3232\)Held\-out test size6464Prompt pool8080/ domainTSS searchBeam width33Early barrier22Latter barrier00Accept constraint \(explore\)accept≥\\geqnative \(hard\)Score tolerance \(relative\)drop≤5%\\leq 5\\%Min skips \(final prefer\)min⁡\(3,\|𝒮\|max\)\\min\(3,\|\\mathcal\{S\}\|\_\{\\max\}\)Selection ruleTriple\-win \(main\)score, accept, tok/s≥\\geqnativeFinal rankingaccept↑\\uparrow, then\|𝒮\|\|\\mathcal\{S\}\|↑\\uparrow, then tok/s↑\\uparrowLayers preferredlatter→\\rightarrowearlyTable 4:Sensitivity of TSS to the acceptance and metric tolerances on Vicuna\-7B across four domains\. We jointly varyϵA\\epsilon\_\{A\}andϵM\\epsilon\_\{M\}and report the resulting accept length, downstream metric, skip configuration, sparsity, throughput, and search evaluations\. The−5%\-5\\%setting provides the best overall balance between task quality and speculative acceptance\.
### Sensitivity to Acceptance and Metric Tolerances

We study the sensitivity of TSS to the feasibility thresholdsϵA\\epsilon\_\{A\}andϵM\\epsilon\_\{M\}by varying them jointly across five settings, as reported in Table[4](https://arxiv.org/html/2609.26100#A1.T4)\. The score tolerance denotes the permitted change relative to the dense baseline:0%0\\%requires both acceptance and the task metric to remain no lower than the baseline, while−5%\-5\\%,−10%\-10\\%, and−20%\-20\\%allow progressively larger decreases\. In contrast,\+5%\+5\\%requires both quantities to improve by at least5%5\\%\. Different tolerances lead to different skip configurations, illustrating that the thresholds directly control the balance between acceptance, task quality, and acceleration\.

The−5%\-5\\%setting provides the most consistent balance across domains\. On Translation, it achieves the highest accept length of 4\.528 and the highest metric of 0\.2370, together with the best throughput of 127\.3 tokens/s\. On QA, it similarly obtains the best acceptance, metric, and throughput, reaching 3\.970, 0\.0640, and 113\.7 tokens/s, respectively\. On RAG, more permissive settings can produce higher acceptance; for example,−20%\-20\\%reaches 4\.428 accepted tokens\. However, its task metric is only 0\.0826, compared with 0\.1091 under−5%\-5\\%\. This result demonstrates that maximizing acceptance alone may select configurations that accept more draft tokens but provide weaker downstream quality\. On MMLU,−10%\-10\\%and−20%\-20\\%slightly increase accept length from 3\.955 to 3\.972 and throughput from 90\.8 to 92\.5 tokens/s, but reduce accuracy from 0\.3281 to 0\.2969\. The small efficiency gain therefore comes at a clear cost in task performance\.

Overall, the−5%\-5\\%tolerance achieves the highest downstream metric in all four evaluated domains and the highest mean accept length across the tested settings\. Its average throughput is also nearly identical to the most permissive setting, while avoiding the metric degradation observed under larger tolerances\. We therefore useϵA=ϵM=5%\\epsilon\_\{A\}=\\epsilon\_\{M\}=5\\%throughout the main experiments, corresponding to the−5%\-5\\%row in the table\. This setting allows small calibration fluctuations while maintaining stable task performance and selecting configurations with strong speculative acceptance\.

### Qualitative Case Analysis

Table[5](https://arxiv.org/html/2609.26100#A1.T5)summarizes representative held\-out cases across domains and model scales, while Table[6](https://arxiv.org/html/2609.26100#A1.T6)provides detailed comparisons for Vicuna\-7B with EAGLE\. The examples exhibit three outcomes of domain\-specific target skipping: TSS may improve the task\-level output, preserve the original task decision while changing the generation behavior, or slightly degrade an individual response\. This variation is expected because each skip configuration is selected according to aggregate acceptance and task performance on a domain\-level calibration set, rather than optimized separately for every prompt\. The cases therefore illustrate both the potential benefits and the request\-level limitations of TSS\.

The Translation examples demonstrate this variability clearly\. In the 7B case, the native output closely matches the reference, whereas TSS omits the final word “value\.” Although the meaning remains largely unchanged, this omission decreases the case\-level BLEU score from 0\.142 to 0\.101\. The degradation is relatively small and represents a case where several skipped layers remain useful for the precise surface realization of an individual prompt\. In contrast, the 13B Translation example exhibits the opposite behavior: TSS produces an output closer to the reference and increases BLEU from 0\.100 to 0\.328\. Together, these cases show that a domain\-level skip configuration does not improve every translation instance uniformly, but can provide a favorable aggregate trade\-off even when a small number of local degradations occur\.

The Summarization cases show more consistent improvements\. For the 7B example, both Native and TSS capture the main event involving Arsenal and Petr Cech, but the TSS output preserves more reference\-relevant content and avoids part of the unnecessary continuation produced by Native\. Consequently, ROUGE\-L increases from 0\.281 to 0\.430\. The corresponding 13B example similarly improves from 0\.143 to 0\.280\. These cases suggest that selected target layers may introduce additional wording or refinements that reduce overlap with concise reference summaries\. Bypassing these layers can preserve a generation path that better matches the summarization objective\.

A similar pattern appears in open\-domain QA\. In the 7B example, Native produces a broad explanation stating that the minimum age for purchasing a BB gun varies across states, whereas the reference answer is simply “18\.” TSS generates the more direct response “18 years,” increasing token\-level F1 from 0\.043 to 0\.143\. The 13B QA case exhibits a larger improvement from 0\.333 to 0\.750\. These examples indicate that TSS can reduce unnecessary elaboration and retain concise task\-relevant content, which is particularly beneficial when evaluation emphasizes normalized answer overlap\.

The RAG example demonstrates a partial rather than complete correction\. Native generates several irrelevant filming locations and receives an F1 score of 0\.000\. TSS shifts the response toward the reference by mentioning locations such as Whitby, Scarborough, the City of Bradford, and the North Riding of Yorkshire, increasing F1 to 0\.310\. Nevertheless, the TSS output still contains additional locations and incomplete phrasing\. This case shows that target skipping can preserve a better\-aligned draft branch, but it does not introduce knowledge that is absent from the prompt context or candidate generation process\. TSS improves the answer direction without fully resolving all factual or formatting errors\.

The MMLU examples distinguish improvements in generation behavior from improvements in the final task decision\. In the 7B case, both Native and TSS select option A, while the correct answer is D, so both receive an accuracy of zero\. However, Native continues with an unrelated off\-task passage, whereas TSS terminates with a concise answer\. TSS therefore improves response stability and removes off\-task drift without correcting the selected option\. In the 13B case, target skipping changes an incorrect native prediction to the correct option, increasing case\-level accuracy from 0 to 1\. These two examples show that reducing full\-depth processing may sometimes only clean the generated response, while in other cases it can also prevent later transformations from altering a correct candidate decision\.

The slightly degraded 7B Translation case also highlights why TSS jointly constrains acceptance and downstream metrics during configuration search\. A configuration that performs well on average may still be suboptimal for an individual request, since some prompts benefit from computations performed by the skipped layers\. TSS therefore does not claim universal per\-request improvement\. Instead, it selects a domain\-level operating point that maintains aggregate task quality while improving speculative acceptance and verification efficiency\. Retaining the complete target model additionally allows unsupported domains or conservative deployments to use the original dense verification path\.

Overall, the qualitative results indicate that full\-depth target verification is not always optimal for domain\-specific speculative decoding\. TSS is particularly effective when later target transformations introduce verbosity, off\-task continuation, or unnecessary revisions that reduce agreement with the task objective\. At the same time, the unchanged and slightly degraded cases demonstrate that the effect is not uniform across all requests\. The benefit of TSS should therefore be understood at the domain level: it improves aggregate acceptance, task performance, and efficiency, rather than assuming that every skipped layer is unnecessary for every input\.

Table 5:Index of representative held\-out qualitative cases across domains and model scales\. The examples include improved, unchanged, and unsuccessful outcomes, illustrating both the benefits and limitations of domain\-specific target\-layer skipping\.Table 6:Qualitative held\-out examples for Vicuna\-7B \+ EAGLE\. Each block is one request from the official test split \(n=64n\{=\}64\), comparing Native speculative decoding with the TSS skip set\. Case\-level task scores appear in each block header\.Table 7:Qualitative held\-out examples for Llama\-2\-13B \+ SAMD \(Token Recycle\)\. Each block is one request from the official test split \(n=64n\{=\}64\), comparing Native speculative decoding with the TSS skip set\. Case\-level task scores appear in each block header\.